Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments
Year: 2023

Abstract:
We present a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them. Our approach extracts check-worthy claims, which are aggregated and ranked for review. Stance classifiers are then used to identify tweets supporting novel misinformation claims, which are further reviewed to determine whether they violate relevant policies. To demonstrate the feasibility of our approach, we develop a baseline system based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments. We make our data1 and detailed annotation guidelines available to support the evaluation of human-in-the-loop systems that identify novel misinformation directly from raw usergenerated content.

heading: 1 Introduction
text: As many people now get information from social networking websites such as Facebook and Twitter, misinformation has become a serious societal problem. To address this, social media companies have spent billions on content moderation.2 Prior work on developing natural language processing systems to combat misinformation has mainly focused on various sub-tasks (Lee et al., 2021; Guo et al., 2022), including claim detection (Eger et al., 2017; Li et al., 2022), evidence retrieval (Jiang et al., 2020; Samarinas et al., 2021; Wan et al., 2021; Aly and Vlachos, 2022), fact verification (Aly et al., 2021; Wu et al., 2022; Chen et al., 2022; Gu et al., 2022), stance classification (Thorne et al., 2017; Conforti et al., 2018; Li et al., 2019), and fallacy recognition (Alhindi et al., 2022). Researchers have also attempted to perform early detection of novel misinformation claims (Yue et al., 2022), as it
1https://github.com/ethanm88/hitl-eva luation-early-misinformation-detection
2https://www.cnbc.com/2021/02/27/cont ent-moderation-on-social-media.html
is crucial for supporting early interventions such as pre-bunking (Lewandowsky and Van Der Linden, 2021). However, evaluations are often set up automatically using datasets that were retrospectively constructed based on a predefined set of debunked claims.
Recent work by Glockner et al. (2022) presented convincing evidence that existing NLP factchecking pipelines are unsuitable for detecting novel real-world misinformation. They show these systems rely on leaked counter-evidence from news sources that have already fact-checked the claim. In general, it is unrealistic to assume this type of evidence will be available for new claims that have not yet been widely spread.
In this paper, we address this challenge by presenting a more realistic human-in-the-loop detection and evaluation framework that can measure a system’s capabilities for detecting novel checkworthy claims in the wild (see Figure 1). We focus on discovering new, domain-specific claims from raw tweets which are then verified by humans, rather than relying on a pre-defined list of claims that have already been fact-checked for evaluation. More importantly, we consider not only the accuracy but also the volume, relevance, and timeliness of misinformation claims automatically identified by a system, given a collection of raw tweets. We argue this approach provides more realistic experimental conditions because (1) it does not rely on leaked counter-evidence from claims that have already been fact-checked, (2) human expertise is vital in verifying the truthfulness of claims (Nakov et al., 2021; Karduni et al., 2018) and (3) it is more effective for humans to check aggregated claims within a specific domain (e.g., claims about the efficacy of COVID-19 treatments), before proceeding to individual social media messages to determine if they violate specific misinformation policies.
We validate our methodology for end-to-end misinformation detection in the domain of COVID-19
15817
treatments. COVID-19 treatments make an ideal testbed for human-in-the-loop misinformation extraction because Twitter has provided clearly defined policies in this area, which we use as guidelines in a realistic human evaluation of a system’s output.3 We evaluate our baseline system with our four defined metrics and find that 18% of the top-50 trending claims were actually misleading (relevance), 50% of new misleading claims (unapproved COVID-19 treatments) are detected before they are debunked by journalists in a news article (timeliness), 65% of tweets flagged constitute policy-violations (accuracy), and an average of 124 policy violations can be confirmed by a human-annotator per hour (volume) when using our system.
Our work fills an important gap in the literature, by showing that it is possible to construct a realistic end-to-end evaluation that supports the early detection of novel rumors directly from raw data. Instead of classifying individual tweets as rumorous or not, we extract phrase-level claims that can be aggregated and ranked across a large amount of data and thus can be reviewed more time-efficiently by fact-checkers for human evaluation and for realworld applications. Tweets that are automatically classified as supporting misinformation claims can then be reviewed to determine whether they violate relevant policies.
3 https://tinyurl.com/CovidMisinformationPolicy

heading: 2 Related Work
text: There is a large body of misinformation-related research. Due to space limitations, we only highlight the most relevant work. See also the excellent surveys by Nakov et al. (2021) and Guo et al. (2022).

heading: 2.1 Detecting Check-worthy Claims
text: One of the most related works to ours is the CLEF2022 CheckThat shared-task (Nakov et al., 2022), which evaluates three sub-tasks automatically and separately: (1) determine whether a tweet is worth fact-checking; (2) given a check-worthy claim in the form of a tweet, and a set of previously factchecked claims, rank the tweets in order of their usefulness to fact-check; and (3) given the text and the title of a news article, determine whether the main claim it makes is true, partially true, false, or other. In contrast, our experimental setup is more realistic as it operationalizes over a large amount (e.g., millions) of raw tweets and requires spanlevel extraction to identify the exact claims (e.g., claims about the efficacy of COVID-19 treatments) rather than just “claims in the form of a tweet” (e.g., tweets that talk about COVID-19 treatments). We also present an end-to-end human-in-the-loop evaluation of the entire misinformation detection pipeline based on the accuracy, volume, and timeliness of all extracted claims, other than just the automatic intrinsic evaluation of each component separately.
Similar to CLEF CheckThat, there exist many other prior works that treat claim detection (or ru-
mor detection) as a text classification problem by predicting check-worthiness (or rumourousness) given a tweet or sentence. One representative work is ClaimBuster (Hassan et al., 2017) which classifies 20,617 sentences from the U.S. general election debate transcripts as non-factual, unimportant factual, and check-worthy factual. Researchers have also developed other datasets (Diggelmann et al., 2020; Konstantinovskiy et al., 2021; Redi et al., 2019; Thorne et al., 2018) and automatic models (Hansen et al., 2019; Jaradat et al., 2018; Wright and Augenstein, 2020). Another relevant work is by Sundriyal et al. (2022) which identifies claims as text fragments, such as “our wine keeps you from getting #COVID19” and “Better alternative to #DisinfectantInjection”. The evaluations are mostly done automatically over a small fixed set (normally at the scale of 1k∼50k) of annotated tweets or sentences.

heading: 2.2 Early Rumor Detection
text: As briefly mentioned in §2.1, rumor detection is also commonly framed as a text classification task. The standard rumor detection setup (Zubiaga et al., 2016; Derczynski et al., 2017; Vosoughi et al., 2017; Gorrell et al., 2019; Shu et al., 2020) considers only accuracy without temporal information in the evaluation. More related to our work is a task called early rumor detection (Liu et al., 2015; Ma et al., 2017; Yu et al., 2017; Ruchansky et al., 2017; Zhou et al., 2019; Xia et al., 2020; Bian et al., 2020), which compares classification model’s accuracy at different time points and has been extensively surveyed and discussed by Zeng and Gao (2022). However, as they pointed out, most existing methods were “designed with oversimplification” and evaluated automatically on datasets, such as TWITTER-WEIBO (Ma et al., 2016), PHEME (Zubiaga et al., 2016), and BEARD (Zeng and Gao, 2022), that were constructed retrospectively by collecting social media posts using manually curated search keywords (e.g., names of false treatment) based on a given set of debunked claims (e.g., from snopes.com). This setup does not measure systems’ capability to discover unseen rumors in the wild as our human-in-the-loop evaluation does. In real-world scenarios, what exactly is needed from a misinformation detection system is to automatically figure out what keywords (e.g., names of potential false treatments) to search for – which we focus on and evaluate in this paper.

heading: 2.3 COVID-19 Misinformation Detection
text: Given the severity and pervasiveness of the issue, there exists a lot of research (not limited to NLP) about COVID-19 misinformation (Hossain et al., 2020; Glandt et al., 2021; Dimitrov et al., 2020; Shahi et al., 2021; Agley and Xiao, 2021; Chen and Hasan, 2021; Biamby et al., 2022). The most related work to ours is the CONSTRAINT sharedtask (Patwa et al., 2021) at the AAAI-2021, which considers a binary text classification problem of 10,700 COVID-related tweets about real and fake news, and in particular, the work by Kou et al. (2022) that experimented on this dataset with a human-in-the-loop approach. For each input tweet (e.g., “Ethylene oxide used in COVID-19 testing swabs changes the structure of the DNA of human body”), Kou et al. (2022) asked crowd workers to write out the main message (e.g., “Ethylene oxide somehow damages human DNA”), which is then compared with information extracted from COVID-related fack-checking articles and medical papers to help automatic system to predict the tweet’s truthfulness. While they prototyped the interesting idea of human-in-the-loop misinformation detection (Shabani et al., 2021), their design is unrealistic to require humans to manually write one sentence per tweet.

heading: 3 Human-in-the-Loop Evaluation Framework
text: One of the most important functions of a misinformation detection system is to identify new misinformation claims in the wild, and in a timely manner. We thus design our evaluation framework to measure not only the accuracy but also the volume, relevance, and timeliness of misinformation claims identified by a system, given a large collection of raw tweets (not collected based on already debunked claims). See Figure 1 for an overview of our framework.

heading: 3.1 Early Detection of Misleading Claims
text: Problem Definition. Given a large set of tweets T, the goal is to automatically discover novel check-worthy claims, each denoted ci, and aggregate a ranked list of claims, C = [c1, c2, ..., cn]. In this task, we use trendiness (e.g. defined with Fisher’s Exact Test in §4.1.3) as a factor in ranking claims, where a more popular or widely discussed claim will have a greater trendiness. More formally, a novel check-worthy claim ci is charac-
terized by (ti, zi,Si), where ti is the first time the system identified the claim as trending i.e. when its trendiness first broke a set threshold, zi is the claim’s trendiness score at time ti, and Si ⊂ T is the set of tweets supporting the claim. A filtering heuristic can also be applied to remove obvious non-misleading claims from consideration (e.g. filtering out claims supporting approved COVID-19 treatments in §4.1.3).
Evaluation Metrics. A human-in-the-loop evaluation is performed over the top-K trending claims in which annotators verify whether a claim is misleading and, if so, find the earliest news article debunking the claim. The evaluation is based on two metrics: (1) the percentage of misleading claims in the top-K trending claims (relevance) and (2) the number of days (or hours), denoted as δ, between ti and the publication date of the earliest news article (timeliness). Figure 2 visually depicts the application of the δ metric for COVID-19 treatment misinformation. See §4.2 for details about this case study evaluation.
By annotating at the claim level before the tweet level, we reduce human annotator workload by limiting the total number of tweets evaluated. This approach also makes our framework more realistic and efficient, allowing for a more thorough and accurate evaluation of misinformation detection systems. In order to facilitate future research in this area, we provide a collection of raw tweets to evaluate and a baseline system to compare against. As most existing systems are not available as open-
source, the release of our evaluation platform will enable fair and comparable evaluations of these systems.

heading: 3.2 Policy Violation Verification
text: Problem Definition. The objective of this task is to identify tweets within the set associated with a claim ci, Si = {s1, s2, ..., s|Si|}, that violate a misleading information policy. In general, a tweet sj , is likely to violate a policy if it expresses a strong supportive stance towards a claim ci, which was identified as misleading by the human-in-theloop evaluation process from the prior stage (§3.1).
Evaluation Metrics. To evaluate a system’s performance and effectiveness, a human-in-the-loop evaluation is performed on a random sample of N tweets that express a supportive stance toward misleading claims. The evaluation is based on two metrics: (1) the accuracy of the system in identifying policy-violation tweets and (2) the volume in terms of the number of policy violations found per hour by analysts using the system.
To measure the accuracy of the system, human annotators assign a score to each tweet in the sample, based on a five-point Likert scale, with 5 corresponding to a clear violation of the policy and 1 representing a clear non-violation. We set a threshold of score ≥ 4 to make a binary policy violation determination. This scoring scheme allows us to measure the system’s accuracy based on the distribution of annotator scores for all tweets in the sample.
To quantify the volume of policy violations identified by analysts using the system, we define the metric policy-violations per hour as the number of tweets identified by the annotator containing policy violations, divided by the total number of hours spent by the annotator during the two-stage annotation process (§3.1 and 3.2):
policy violations / hr = V
C × rc + T × rt where V is the number of policy violations found, C and T are the numbers of claims and tweets checked respectively, and rc and rt are the average annotation rates for claims and tweets respectively.
This metric allows us to assess the efficiency of the system in identifying policy-violation tweets and the potential benefits for content moderators using the system.

heading: 4 A Case Study: COVID-19 Treatment Misinformation
text: To illustrate the usage of our human-in-the-loop evaluation framework outlined in §3, we present a case study for COVID-19 misinformation. Specifically, we target Twitter’s COVID-19 policy on unapproved treatments, which states that:
“False or misleading information suggesting that unapproved treatments can be curative of COVID-19”
are grounds for labeling tweets with corrective information (Twitter, 2021).

heading: 4.1 Our System
text: In this subsection, we describe the three components of our COVID-19 treatment misinformation detection system: claim extraction, stance classification, and claim ranking with their task-specific intrinsic performance. Later, in §4.2, we present an extrinsic human-in-the-loop evaluation of the entire system using our defined framework (§3).

heading: 4.1.1 Extracting Check-Worthy Claims
text: Data. We train and evaluate our claim extraction models on the human-annotated Twitter COVID19 event extraction dataset created by Zong et al. (2022), which is collected between 2020/01/15 and 2020/04/26. In this work, we focus on claims of the form “X is an effective COVID-19 treatment”, where X is an extracted span. We split the provided 1, 271 training tweets in the CURE & PREVENTION
category into 60% for training and 15% for development, and report token-level F1 scores on the 500 tweets used for evaluation in the 2020 W-NUT shared task.4
Models. We develop three approaches, outlined below, to extract claims as a text span from a sentence with a sequence tagging model and a question-answering model (Rajpurkar et al., 2018; Du et al., 2021). Details of training hyperparameters can be found in Appendix D.1. (1) Sequence Tagging: a standard sequencelabeling task with a BIO tagging scheme, where ‘B’ and ‘I’ tags are used to identify treatment tokens. We follow a similar approach as the named entity recognition method used by Devlin et al. (2019) and experiment with two pre-trained models, including RoBERTalarge (Liu et al., 2019) and a domain-specific COVID-Twitter-BERTlarge(CTBERT) (Müller et al., 2020). (2) Question-answering (QA): we treat the claim extraction as a question-answering task and apply the SQuADv2.0 (Rajpurkar et al., 2018) formulation as some tweets may not include relevant claims (similar to unanswerable questions). We experiment with two approaches: a span-prediction model that predicts start and end positions for answer spans in context using RoBERTa/CT-BERT as the encoder, in addition to a text-to-text model that generates answers using T5large (Raffel et al., 2020; Wang and Lillis, 2020). The question template for extracting treatments discussed in tweets is “What is the mentioned COVID-19 cure?”. (3) QA-Pretraining: it has been shown that intermediate task pre-training can yield further gains for low-resource target tasks (Pruksachatkun et al., 2020; Poth et al., 2021). We thus experiment with pre-training QA models on the SQuADv2.0 dataset before fine-tuning on the claim extraction dataset.
Intrinsic Evaluation. Table 1 shows the claim extraction results on the COVID-19 treatment dataset. We observe QA models outperform tagging models across encoders. RoBERTa outperforms the domain-specific encoder (CT-BERT) for QA extraction. However, after QA pre-training, CT-BERT improves from 53.1 to 63.8 F1 and outperforms RoBERTa by 2.3 points. Finally, as the T5 model achieves the best F1 regardless of QA pre-training, we use T5SQuADv2 Pre-train as our final
4https://noisy-text.github.io/2020/ext ract_covid19_event-shared_task.html
claim extraction model.

heading: 4.1.2 Task-Specific Stance Classification
text: Data. Due to the lack of datasets for COVID19 treatment stance, we annotate a new dataset for our evaluation. To collect relevant tweets we tracked the keywords “cure”, “prevention”, “virus”, and “COVID-19” from November 2020 to December 2020 using the Twitter API. We collected 1, 055, 559 tweets and claims extracted using our model (§4.1.1). Out of 97, 016 tweets for which a treatment was able to be extracted, we randomly sample 2, 000 tweets to annotate the author’s stance on the effectiveness of the treatment. We paid crowd workers on Amazon MTurk to annotate our data. Each task consists of a tweet with a highlighted treatment. We asked workers to determine the author’s stance towards the treatment and select among three options (SUPPORTING, REFUTING, or NO STANCE). We decided not to include additional options for irrelevant and sarcastic tweets due to poor annotator agreement in pilot experiments. Each tweet is labeled by 5 independent crowd workers. Workers were paid $0.20/HIT, which roughly equates to $8.50/hour. A screenshot of the annotation interface is provided in Figure 6 and dataset statistics are summarized in Table 2.
Quality Control. During the annotation process, we monitored the quality of workers’ annotations using their agreement with each other and split the data into 10 batches (200 tweets each) to detect poor annotations in the early stages. We calculate the annotation agreement of each worker against the majority of 5 workers. If the worker’s agreement is less than 0.75 for a SUPPORTING annotation based on a majority vote, we do not allow them to participate in the subsequent annotation batch. Across all annotations, we find a 0.65 value of Cohen’s κ (Artstein and Poesio, 2008) for the
inter-annotator agreement between workers. The distribution of the dataset based on the majority vote of workers is shown in Table 2. In the case that there is no majority annotation for a given tweet i.e. the 5 individual annotators are split 2/2/1 among the three annotation types, we assign the tweet a default annotation of NO STANCE. We randomly split our annotated dataset of 2,000 tweets into a training set of 1,200 tweets, a development set of 400, and a test set of 400.
Models. Using the annotated corpus, we develop classifiers to detect the author’s stance toward a treatment. Specifically, given a claim and a tweet (ci, sj), our goal is to predict the author’s stance mi ∈ {SUPPORTING, REFUTING, or NO STANCE}. We experiment with three models including a baseline neural bag-of-words (NBOW) model, RoBERTalarge (Liu et al., 2019) and a COVID-Twitter-BERTlarge (CT-BERT) (Müller et al., 2020). To indicate the position of the claim in the input, we use relative position encoding (RPE) (Shaw et al., 2018) for the NBOW model. For the pre-trained language models, we add special markers around the claim following the bestperforming model from Soares et al. (2019), [ENTITY MARKERS - ENTITY START]. Details of training hyperparameters are in Appendix D.2.
Intrinsic Evaluation. Table 3 presents the results for NBOW, RoBERTa, and CT-BERT. We observe that CT-BERT outperforms all other models, with an F1 score of 66.7. Generally, we find that these models performed best in classifying tweets with SUPPORTING stance and worst on tweets with a NO STANCE label. This latter result is possibly due to annotators classifying those tweets that might be irrelevant to the task as NO STANCE.

heading: 4.1.3 Ranking of Trending Claims
text: Following Twitter’s COVID-19 misinformation policy violation guidelines, we focus on tweets that advocate the efficacy of an unapproved treatment.
Thus, we filter out tweets that mention common approved treatments listed in Table 8 in Appendix C, which are prepared according to authorities and news agencies including the CDC and NYT. Upon determining stance on the filtered set, we only consider tweets with a SUPPORTING stance towards the effectiveness of the extracted treatment. Finally, we remove near-duplicates and cluster the remaining extracted treatments based on word overlap (Jaccard similarity) to enable treatment-level decision-making similar to Basu et al. (2013).
Ranking Claims. For the claims (treatments) in each cluster, we count the number of tweets mentioning the claim both daily and cumulatively. Based on these counts, we compute the claim’s pvalue on a given date using the one-tailed Fisher’s Exact Test (Fisher, 1922) which has been shown to be effective in rare event detection (Moore, 2004; Johnson et al., 2007; Ritter et al., 2011). A claim’s p-value is a measure of its trendiness denoted zi (§3.1) by which it is ranked relative to other claims.
Detecting Novel Claims. Based on the results of Fisher’s Exact Test, our system automatically detects novel trending claims and flags them for manual inspection by content moderators. A claim is considered as newly trending if it’s p-value is less than a preset significance threshold (α-level) and it has never broken this threshold previously (further details in Appendix E). Using the notation from §3.1, if the claim (ci) found to be newly trending on date ti is judged to be misleading by a human moderator, our system then provides a list of individual tweets (Si) that SUPPORT the misleading claim for manual inspection.

heading: 4.2 Human-in-the-Loop Evaluation for Detecting COVID-19 Misinformation
text: In this section, we evaluate the system outlined in §4.1 using the human-in-the-loop evaluation methodology we define in §3. We follow the same procedure described in §4.1.2 to prepare a new dataset containing 14, 741, 171 tweets for large-
scale evaluation. We then extract treatments using our QA-based claim extractor and apply our stance detection model to classify authors’ stance towards each treatment. After removing tweets without an extracted treatment, the resultant evaluation corpus consists of 1, 905, 424 tweets.

heading: 4.2.1 Early Detection of Misleading COVID-19 Treatments
text: We first evaluate the ability of our system to detect newly trending misleading COVID-19 treatment claims and report metrics measuring relevance and timeliness as defined in §3.1.
Data Preparation and Human Evaluation. We set aside tweets collected from 2020/03/01 to 2020/03/31 (1-month time-frame) to serve as an initial base of historical data to compute cumulative counts for detecting novel trending claims using the Fisher’s Exact Test. Newly trending treatments are then identified during the time period of 2020/04/01 to 2022/05/05 (2-year time-frame) using the methodology described in §4.1.3. The top 300 treatments are selected based on p-values from Fisher’s Exact Test which equates to a significance level of α = 1.15e−6.
We employ two in-house annotators, who act as mock content-moderators, to evaluate these 300 treatments and determine (1) if the extraction is a treatment (2) if the treatment is unapproved and (3) the earliest publication date of a news article debunking the treatment as effective for COVID-19 using the Google News engine. Appendix A contains further details about the annotation process.
Out of the 300 treatments, 100 were annotated by both annotators to determine inter-rater agreement on task (2), which was 0.87 as given by Cohen’s κ. On average, it took 89.7 seconds to complete each treatment annotation.
Results on Early Detection of Misleading Claims. In terms of relevance, Table 4 shows the percentage of the top 5/50/100 trending treatments ranked by pvalue that were determined to be unapproved, and Figure 3 shows the cumulative number of potential unapproved trends identified over time along with the total number of trending treatments. To evaluate the timeliness, we calculate δ, which measures the number of days our system detects misleading claims before a debunking news article is published (§3.1). We find that our system is able to detect 50% of rumors before the publication date of the relevant news article (δ ≥ 0), with the median δ
being 21 days. Figure 2 shows three notable unapproved treatment examples from early in the pandemic with their relevant news article and δ values.

heading: 4.2.2 Identifying COVID-19 Policy Violations.
text: In addition to detecting novel rumors online, we also evaluate the ability of our system to identify tweets that violate Twitter’s misleading information policy and report metrics measuring accuracy and volume as defined in §3.2.
Data Preparation and Human Evaluation. For each of the 40 treatments identified as unapproved in the previous experiment, we randomly sample 10 tweets that have SUPPORTING stance towards the claim. Near duplicate tweets were identified and removed leaving 361 unique tweets. The two in-house annotators then assign a score to each tweet based on a five-point Likert scale, with 5 corresponding to a clear violation of the policy and 1 representing a clear non-violation (See score descriptions in Table 7 in Appendix A).
To investigate the quality of annotations, we compute agreement on 206 tweets using ordinal Krippendorff’s α (0 ≤ α ≤ 1) (Krippendorff, 2011).5 We find that annotators agreed moderately with Krippendorff’s α = 0.54, indicating
5We use Krippendorff’s α instead of Cohen’s κ to measure inter-rater agreement here because it is applicable to ordinal annotation data.
fair agreement. On average, it took the annotators 16.1 seconds to annotate each tweet.
Results on Identifying Policy Violations. In Figure 4, we find that 65% (accuracy) of tweets had scores indicating that it was either likely or clearly violating the policy with an average score of 3.54 out of 5. Figure 4 presents the estimated distribution of Likert scores over 10,246 tweets using the 10 annotated tweets sampled for each treatment and extrapolated to all tweets mentioning the treatment. In terms of volume, we estimate that an annotator can identify approximately 124.2 policy violations per hour with our system, based on the average annotation rate of the 300 treatments and the same extended set of tweets, where a Likert score of 4 or 5 constitutes a policy violation. See Appendix F for a full calculation of this statistic.

heading: 5 Conclusion
text: In this work, we present a novel end-to-end humanin-the-loop evaluation framework for the early identification of novel misinformation on social media from raw tweets. Unlike previous evaluation frameworks, our methodology captures the interplay between the system and human content moderators while also providing realistic metrics for early misinformation detection. We validate our misinformation detection framework for claims in the domain of COVID-19 treatments. By aggregating and ranking structured representations of claims, and relying on human fact-checkers to review trending claims, our system is able to detect 50% of misleading claims earlier than the news.

heading: 6 Limitations
text: While our approach does require domain-specific information extraction models to extract structured representations of novel misinformation claims for easy aggregation and review, there is significant prior work on event extraction that can be adapted to extract check-worthy claims (Ritter et al., 2012; Luan et al., 2019; Du and Cardie, 2020). Furthermore, we argue content moderators or factcheckers are likely to be more effective when focusing on one claim type at a time (e.g. COVID-19 treatments, election integrity, vaccine effectiveness, etc.), rather than reviewing a mixture of claims on multiple topics.
Our COVID-19 case study also makes use of “mock” content moderators, rather than employees or contractors working for social media companies or fact-checking websites. However, we believe this methodology still provides valuable insight that would not be publicly available otherwise, as social media companies do not currently publish extensive details about their content moderation processes6 and fact-checking websites vary widely in policy and have been shown to provide inconsistent claim classification (Marietta et al., 2015). Some prior user studies (Nguyen et al., 2018; Pennycook and Rand, 2019; Shabani et al., 2021) have also shown laypeople (e.g., Amazon Mechanical Turk workers) can be good at judging the veracity of claims or reliability of news articles.
As of late November 2022, Twitter has suspended enforcement of its COVID-19 misleading information policies such as the one we target in this paper.7 However, per the Associated Press article, one of the possible reasons for the suspension was that Twitter has “struggled to respond to a torrent of misinformation about the virus” with many “bogus claims about home remedies” still on the site despite the previous enforcement of policies. While we do not have details about the internal automated systems Twitter has in place to assist with content moderation, an end-to-end early detection system might have helped stem the spread of misinformation on the platform. Additionally, despite the lack of official policy enforcement, our system can still be used by third-party fact-checking websites or researchers to measure and report misinformation
6https://www.nytimes.com/2022/05/19/b usiness/twitter-content-moderation.html
7https://apnews.com/article/twitter-e nds-covid-misinformation-policy-cc232c9 ce0f193c505bbc63bf57ecad6
on Twitter. Finally, the main goal of our work is not to create a system for COVID-19 misinformation detection but rather to propose a framework that allows for a fair and realistic evaluation of early misinformation detection systems in any domain.

heading: 7 Broader Impact and Ethical Considerations
text: We release our corpus of tweets annotated with stance, and our dataset of trending misinformation claims under Twitter’s Developer Agreement,8 which grants permissions for academic researchers to share Tweet IDs and User IDs (less than 1,500,000 Tweet IDs within 30 days) for noncommercial purposes, as of October 10th. 2022.
Our system is designed for research purposes and may contain unknown biases towards demographic groups or individuals (Sap et al., 2019). Further investigation into systematic biases should be conducted before our models are deployed in a production environment.
We believe this study helps shed light on how NLP tools developed to help combat online misinformation might be used in a real content moderation workflow. We hope this will encourage future research on human-in-the-loop systems and help shape the design of new tasks and datasets in this area. We believe it is beneficial for some research on combating misinformation to take place outside of social media companies to provide an unbiased view of the challenges involved in fighting online misinformation.

heading: Acknowledgments
text: We thank anonymous reviewers for their helpful feedback on this work. We also thank Chase Perry and Andrew Duffy for their help with annotations and human evaluation. This research is supported in part by the NSF (IIS-2052498), ODNI and IARPA via the BETTER and HIATUS programs (2019-19051600004, 2022-22072200004). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
8https://developer.twitter.com/en/dev eloper-terms/agreement-and-policy

heading: A Annotation Guidelines for the Human-in-the-Loop Evaluation
text: Early Detection of Misleading COVID-19 Treatments. Given a list of trending claims (e.g., COVID-19 treatments), annotators are required to determine (1) if the extraction is a treatment (2) if the treatment is unapproved and (3) the earliest publication date of a news article they can find that debunks the treatment as effective. Annotators query the Google News engine with a query in the form of “[treatment] cures COVID-19” and sort by date to find the earliest published news article starting from 2020/04/01 that debunks the treatment as effective against COVID-19. Treatments are only considered to be unapproved if the annotators can identify a reputable news source as a reference. Table 5 shows the annotation questions and guidelines as they appeared to annotators during the human-in-the-loop evaluation for this task. Note that March 1st, 2020 was used as the starting date for the article search because it was the earliest date for which we had tweet data.
Identifying COVID-19 Policy Violation Tweets. Given a tweet with SUPPORTING stance towards the claim that “treatment is effective in treating COVID-19”, annotators assign a score based on a five-point Likert scale, with 5 corresponding to a clear violation of the policy and 1 representing a clear non-violation. Table 7 shows the Likert score descriptions and Table 6 shows the annotation questions and guidelines as they appeared to annotators during the human-in-the-loop evaluation for this task.

heading: B Annotation Interface for Stance
text: Classification
Our stance data collection procedures (§4.1.2) on Amazon MTurk were approved by an ethics board. Before individuals were allowed to annotate data for our task, they were required to give consent by electronically signing off on the ethics statement found in Figure 5 (some portions have been redacted for anonymity purposes). Note that all annotators were MTurk workers in the United States who had previously annotated 1000 HITs with a pass-rate ≥ 95%. Figure 6 shows the interface used for collecting these stance annotations.

heading: C Approved Treatments
text: Table 8 shows the approved treatments that were used for filtering in §4.1.3.
D Implementation Details
All experiments are performed with NVIDIA A40 GPUs. All hyperparameters are selected using a held-out development set.
D.1 Claim Extraction Models
Hyperparameters can be found in Table 9.
Sequence Tagging Models. We apply sequencelabeling models with a standard BIO tagging scheme (‘B’ and ‘I’ tags are used to identify treatment tokens), similar to the named entity recognition method used by Devlin et al. (2019). We experiment with RoBERTalarge(354M) (Liu et al., 2019) and a domain-specific COVID-TwitterBERTlarge(345M) (Müller et al., 2020).
Question Answering Models. We experiment with QA-based slot filling models (Du et al., 2021), which model claim extraction as a SQuADv2.0 question-answering task (Rajpurkar et al., 2018). We experiment with two approaches: a spanprediction model that predicts start and end positions for answer spans in context using RoBERTa or CT-BERT as the encoder, in addition to a text-to-text model that generates answers using T5large(770M) (Raffel et al., 2020; Wang and Lillis, 2020).
Pre-training QA Models. We pre-train QA models on the SQuADv2.0 dataset for 2 epochs with a learning rate of 2e-5 and batch size of 16, followed by fine-tuning for claim extraction.
D.2 Stance Classification Models
Hyperparameters can be found in Table 10.
NBOW. We use an NBOW model and a relative position encoding (RPE) (Shaw et al., 2018) to indicate the position of the claim in the input sentence. Specifically, the 1D RPE encodes the relative distance of each token to the extracted treatment. We then concatenate NBOW with the RPE embedding and pass the concatenation through one layer of a feed-forward neural network.
RoBERTa/CT-BERT. We finetune a RoBERTalarge (Devlin et al., 2019) and COVIDTwitter-BERT (CT-BERT) (Müller et al., 2020)
using a linear classification layer. We use the best-performing model from Soares et al. (2019), [ENTITY MARKERS - ENTITY START], which uses special tokens [Cstart], [Cend] to mark treatment span in a sentence. The modified sentence is then fed into BERT/CT-BERT and the representation of the starting marker [Cstart] is sent into a linear classification layer.

heading: E Ranking Extracted Claims
text: To generate a list of treatments mentioned on a specific day sorted by trendiness or significance we cannot simply use the daily frequency counts of treatments mentioned, as more popular treatments will consistently be mentioned at a higher volume. In our evaluation dataset used in §4.2, for example, chloroquine and its variants are mentioned in a tweet with SUPPORTING stance approximately
11.5 times per day on average while 56% of total treatments encountered were mentioned less than one time in the period studied. Therefore, we require a method that takes into account the historical frequency of treatments to calculate the strength of the association between the trendiness of treatment and the date.
To do this, we use a one-tailed Fisher’s Exact Test (Fisher, 1922), which has been shown to be effective in rare event detection applications in the
Source Treatments
CDC(*) Mask, Face Mask, Social Distancing, Stay Home, Wash Hands, Hand Washing, Cover Coughs, Cover Sneezes
New York Times(**) Remdesivir, REGEN-COV, Bamlanivimab, Etesevimab, Sotrovimab, Dexamethasone, Prone positioning, Ventilators, Evusheld, Paxlovid, Molnupiravir, Lagevrio, Baricitinib, Olumiant, Tocilizumab, Actemra
domain of statistical natural language processing (Moore, 2004; Johnson et al., 2007; Ritter et al., 2011).
To apply this test, we first calculate the hypergeometric probability, the probability of a particular distribution of treatment frequencies assuming independence between the treatment and the date. We define T and D as the events when a tweet’s extracted treatment is t and when a tweet is published on date d respectively. Also, we let C(X) be the observed frequency of event X and C(X,Y ) be
the joint frequency of event X and event Y . Given these definitions, we can calculate the hypergeometric probability, pT,D, as follows:
pT,D = C(T )!C(¬T )!C(D)!C(¬D)!
N !C(T,D)!C(¬T,D)!C(T,¬D)!C(¬T,¬D)!
where N is the sample size. Given this formula of the hypergeometric probability for a distribution with a treatment t and date d, we calculate the p-value of the test by summing the hypergeometric probabilities of this distribution and all more extreme distributions. In our case, more extreme distributions are hypothetical distributions where the joint frequency of a tweet with a specific treatment published on a specific date is greater than C(T,D).
A treatment is flagged by our system if its pvalue is less than the threshold or α-value as set by the content moderator and it has not previously broken this threshold.

heading: F Policy Violations Per Hour Calculation
text: Here we detail the calculation of the 124.2 policy violations per hour statistic reported in §4.2. First, we calculate the total amount of time required by each of the phases of the human annotation:
1. Stage 1: Detecting Misleading COVID-19 Treatments (§4.2.1)
• # of claims = 300 claims • time of verifying a claim = 89.7s/claim • time spent on claim annotation = 300 ∗ 89.7s = 7.5 hours
2. Stage 2: Identifying COVID-19 Policy Violations (§4.2.2)
• # of tweets mentioning an unapproved claim in the full batch = 10246 claims
• time to annotate an individual tweet = 16.1s/tweet
• estimated time spent on tweet annotation = 10246 ∗ 16.1s = 45.8 hours
Next, we detail the calculation steps using these calculated times:
1. Total annotation time = 7.5 + 45.8 = 53.3 hours
2. Estimated # of tweets (considering the full batch - see scores 4 and 5 bars in Figure 4) containing policy violation = 3151+ 3467 = 6618 tweets
3. # policy violated identified per hour of annotation time = 6618/53.3 hours = 124.2 tweets/hours
4. total # tweets judged per hour of annotation time = 10246/53.3 hours = 192.2 tweets/hour
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 6

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 6, Section 7

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.1.1, Section 4.1.2

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.1.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 7

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4.1.1

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 7

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.1.2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1.2
C 3 Did you run computational experiments? Section 4.1.1, Section 4.1.2

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix D1, Appendix D2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix D1, Appendix D2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4.1.2 and Appendix B

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix B

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4.1.2

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix B

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Appendix B

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix B

heading: Future Work
text: We hope this will encourage future research on human-in-the-loop systems and help shape the design of new tasks and datasets in this area. We believe it is beneficial for some research on combating misinformation to take place outside of social media companies to provide an unbiased view of the challenges involved in fighting online misinformation