Can Community Notes Replace Professional Fact-Checkers?
Authors: Nadav Borenstein, Greta Warren, Desmond Elliott, Isabelle Augenstein
Accepted at: acl25

Can Community Notes Replace Professional Fact-Checkers?

Nadav Borenstein*    Greta Warren*    Desmond Elliott    Isabelle Augenstein
University of Copenhagen
nb@di.ku.dk    grwa@di.ku.dk
de@di.ku.dk    augenstein@di.ku.dk

Abstract

Two commonly employed strategies to combat the rise of misinformation on social media are (i) fact-checking by professional organisations and (ii) community moderation by platform users. Policy changes by Twitter/X and, more recently, Meta, signal a shift away from partnerships with fact-checking organisations and towards an increased reliance on crowdsourced community notes. However, the extent and nature of dependencies between fact-checking and helpful community notes remain unclear. To address these questions, we use language models to annotate a large corpus of Twitter/X community notes with attributes such as topic, cited sources, and whether they refute claims tied to broader misinformation narratives. Our analysis reveals that community notes cite fact-checking sources up to five times more than previously reported. Fact-checking is especially crucial for notes on posts linked to broader narratives, which are twice as likely to reference fact-checking sources compared to other sources. Our results show that successful community moderation relies on professional fact-checking and highlight how citizen and professional fact-checking are deeply intertwined.

1 Introduction

The proliferation of misinformation on social media (Arnold, 2020; Diakopoulos, 2020), along with the rise of generative AI (Augenstein et al., 2024) have led to increasing concerns about people’s ability to access trustworthy and credible information, leading to potential harms to public health (Clemente et al., 2022), democracy, and political stability (Reglitz, 2022). Fact-checkers play a crucial role in combatting misinformation (Graves, 2017), and in recent years, have partnered with social media platforms, e.g., Meta, YouTube, and TikTok, to tackle its spread on these platforms. However, due to the scale of misleading content shared online, community moderation (e.g., options to flag potential misinformation, group/server moderators) is often employed in parallel (Morrow et al., 2022), as a complementary approach (e.g., (Google, 2025; TikTok, 2025); see also the practice of snoping (Pilarski et al., 2024)). The expansion of fact-checking projects in the last decade (Lauer and Graves, 2024), alongside their broader initiatives to curb misinformation (e.g., citizen media literacy programmes (Juneja and Mitra, 2022)) have been aided by partnerships with social media platforms such as Meta and Google (Graves and Anderson, 2020), which fund independent fact-checking agencies to fact-check potentially false claims on their platform.1 However, political pressure and accusations of bias and censorship, and most recently, Meta’s announcement of its plans to end its partnerships with fact-checkers in the U.S. and implement a community moderation model (Meta, 2025), threatens the financial stability of fact-checking organisations, and hence, their ability to keep up with the increasing volume and sophistication of

1Fact-checkers provide a judgment of claim veracity and exert no influence on the platforms’ content moderation policies (Catalanello and Sanders, 2025).

Figure 1: An example of a community note. Notice the fact-checking link and rating.

![An example of a community note. Notice the fact-checking link and rating.](page_1016_384_393_246.png)
misinformation spread (Stencel et al., 2024; IFCN, 2024).

Meta’s recent policy shift also implies that these two strategies (fact-checking and community notes) are independent and in opposition, rather than two complementary strategies of tackling online misinformation. In this paper, we examine Twitter/X community notes as a case study to understand how fact-checking is used in community notes. Specifically, we investigate the following two questions: (RQ1) To what extent do community notes rely on the work of professional fact-checkers? and (RQ2) What are the traits of posts and notes that rely on fact-checking sources? Studying the relationship between fact-checking and community notes is vital for understanding the shared role of expert and citizen-driven fact-checking in the global information ecosystem.

We find that at least 1 in 20 community notes rely explicitly on the work of professional fact-checkers, while this reliance is higher still for high-stakes topics such as health and politics. Our experiments also show that fact-checking is vital for debunking misleading content linked to broader narratives or conspiracy theories. These findings imply that high-quality community notes cannot be produced independently of professional fact-checking. They further suggest that the pressure on fact-checkers exerted by platforms and politicians by defunding and discrediting fact-checking organisations will have corrosive effects on the quality of notes and destructive implications for information integrity more widely.

2 Background

Due to space constraints, only the most relevant related work is provided here. Additional relevant studies and background can be found in App. A.

2.1 Community notes

Community moderation has been proposed as a means of addressing the scalability (Martel et al., 2024) and cross-partisanship trust (Flamini, 2019) challenges associated with fact-checking. Twitter/X’s Community Notes programme (piloted in 2021 and publicly launched in October 2022 (Twitter/X, 2021)) is a notable example of such a system. Any platform user may volunteer as a Community Notes contributor, although they must achieve a particular ‘rating impact score’ before they can write notes (Twitter/X, 2024b). Notes that achieve a ‘helpful’ rating appear underneath the post, explaining why the post is misleading (see Fig. 1). To be rated ‘helpful’, a note must receive similar levels of helpfulness rating from users with diverse viewpoints (Twitter/X, 2024a).

2.1.1 Characteristics of Community Notes

A small but growing body of work has analysed Twitter/X’s Community Notes dataset, focusing on the targets, sources, and limitations of notes.

Targets of notes. Community notes tend to focus on misleading posts from large accounts (Pilarski et al., 2024), focusing on posts that lack important content or present unverified claims as facts (Pröllochs, 2022; Drolsbach and Pröllochs, 2023).

Sources in notes. Analyses have showed that notes were rated more helpful if they link to ‘trustworthy’ sources and that the majority of sources cited by notes were ‘trustworthy’ left-leaning news outlets (Pröllochs, 2022). A recent study finds that 55% of URLs used in notes were related to news websites, 18% to research, 9% to social media, 9% to encyclopedic sources, but just 1% to fact-checking sources (Kangur et al., 2024).

Limitations of notes. Only 11% of submitted notes reach ‘helpful’ status (i.e., shown to users) by achieving a cross-perspective (Renault et al., 2024; Witschafer and Majumder, 2023), and the time frame for notes to reach the algorithm’s required agreement level (15.5 hours on average) limits its capacity to halt misinformation spread (Renault et al., 2024). Posts related to partisan issues are particularly affected by these challenges (Allen et al., 2022). Additional concerns about the notes’ efficacy highlight their indifference to the expertise needed for certain claims and reliance on subjective helpfulness rather than objective facts, free labour and inadequate support and guardrails regarding explicit content (Gilbert, 2025).

Our work provides novel insights into the targets, sources and limitations of community notes by shedding light on the relationship between notes and professional fact-checking. Closest to our work is a recent analysis of community notes written in 2024 by the fact-checking organisation Maldita (2025), who also studied the reliance of community notes on professional fact-checkers. They discover that fact-checking organisations are widely used as a reference by notes’ authors. The current work provides a more fine-grained analysis by studying the extent to which fact-checking sources form the basis of note-writers’ efforts to counter misinfor-
mation and identifying the strategies they employ.

3 Dataset

We download files containing all community notes and their metadata from the official website,2 which amounts to 1.5M notes authored between January 28th 2021 and January 6th 2025. Of these, a total of 135K are rated by the community as ‘Helpful’, 51K are rated ‘Not helpful’, and 1.3M are unpublished, i.e., did not receive enough community ratings to reach a verdict. See Fig. 6 in App. B for statistics.

We filter the notes as follows. First, we remove 526K non-English notes, which we identify by applying the language detection library fast-langdetect.3 Then, we further filter 268K ‘unnecessary’ notes—notes attached to tweets that are classified by the community as ‘not misleading’. Finally, to focus only on notes that are used to address misinformation, we filter out 44K notes that contain one of the words ‘ad’, ‘spam’, or ‘phishing’. Following these filtration steps, we are left with a dataset containing 664K notes.

The next step involves categorising the sources that the note authors use to support their claims. First, we use regex to extract all the URLs found in the notes. Importantly, a single note can include multiple external URLs as evidence. See Tab. 2 in App. B for a list of the top-100 most common domains. We classify each URL in our dataset of 664K notes into one of 13 categories (detailed in Fig. 2) using the pipeline described below.

1. Check whether the domain name of the URL is found in a manually curated list of domains of professional fact-checking organisations (See Tab. 3 in App. B for the full list). If so, classify the URL as ‘fact-checking’.
2. Otherwise, search for paraphrases of the word ‘fact-check’ in the URL,4 and classify it as ‘fact-checking’ if a match was found.
3. Otherwise, check whether the domain name is found in Tab. 2, which the authors of this paper manually annotated.
4. Otherwise, use GPT-45 to classify the domain name into one of the 13 categories. Listing 1 in App. C details the prompt we used.
5. Finally, if GPT-4 fails or outputs an unknown category, label the URL as ‘unknown’.

Using this pipeline, we successfully classify 95% of the URLs to one of the 13 categories.

Moreover, we further subsample the notes for performing the in-depth analysis required for answering RQ2 (\S4.2). From the notes rated as ‘helpful’, we sample 3.5K notes with a ‘fact-checking’ source and a random sample of 22K additional notes. We then used web crawling to scrape the text of the posts to which these notes were attached. We name this subset \( S_{text} \) for simplicity.

4 Analysis

We analyse the dataset prepared in \S3 to answer the two research questions defined in \S1.

4.1 RQ1: To what degree do community notes rely on fact-checkers?

According to Fig. 2.a at least 5% of all English community notes contain an external link to professional fact-checkers. This number grows to 7% when only considering notes rated as ‘helpful’ (Fig. 2.b). Conversely, only 1% of notes rated as ‘not helpful’ contain a fact-checking source (Fig. 2.c). These figures are significantly larger than what was reported in some previous studies (1.2% (Kangur et al., 2024)), possibly because Kangur et al. (2024) utilise a smaller dataset of fact-checking agencies and classify fact-checking divisions of popular journals as ‘news’. The results imply that notes incorporating fact-checking sources are generally considered more helpful.

We further assess whether notes with fact-checking sources are perceived to be of higher quality by analysing individual user ratings of notes both with and without such sources. Specifically, we collect user ratings for a balanced (i.e., including of a fact-checking source or not) sample of 20K notes rated by at least 50 users, and calculated the average ratings for the notes. As can be seen in Fig. 7 in App. B, community notes with fact-checking sources are generally rated higher than their counterparts. Interestingly, while notes with fact-checking links are more likely to be regarded as having a good source (higher HelpfulGoodSources), they are also more likely to be rated as notHelpfulSourcesMissingOrUnreliable. Tab. 5 in App. B contains a sample of such notes.

2https://communitynotes.x.com/guide/en/under-the-hood/download-data
3https://github.com/LlmKira/fast-langdetect
4These URLs mostly link to the fact-checking divisions of news outlets, e.g., https://apnews.com/article/fact-checking-909101991741.
5Version gpt-4o-2024-08-06.
Figure 2: The categories of links used by Community notes’ authors as a source. a) all community notes; b) Community notes rated as ‘helpful’; c) community notes rated as ‘unhelpful’. Notice the ‘fact-checking’ category.

<table>
  <tr>
    <th>Annotation type</th>
    <th>Fact-checking source</th>
    <th>False</th>
    <th>True</th>
  </tr>
  <tr>
    <td>misleadingOther</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>misleadingFactualError</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>misleadingManipulatedMedia</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>misleadingOutdatedInformation</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>misleadingMissingImportantContext</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>misleadingUnverifiedClaimAsFact</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>misleadingSatire</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Figure 3: Mean scores of community annotations of misleading posts.

<table>
  <tr>
    <th>Annotation type</th>
    <th>Broader narrative</th>
    <th>Fact-checking source</th>
    <th>False</th>
    <th>True</th>
  </tr>
  <tr>
    <td>Add Missing context</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Discredit source of claim</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Highlight AI generated</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Highlight edited media</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>In-note fact-checking</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Link fact-checking</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Link official source</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Link scientific source</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Link to direct source</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Link world knowledge</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Figure 4: (a) strategies in debunking claims related to broader narratives. (b) the different ways in which fact-checking sources are used to debunk claims.

<table>
  <tr>
    <th>Conspiracy</th>
    <th>FC source</th>
    <th>✓</th>
    <th>✗</th>
  </tr>
  <tr>
    <td>✓</td>
    <td>22%</td>
    <td>11%</td>
  </tr>
  <tr>
    <td>✗</td>
    <td>28%</td>
    <td>39%</td>
  </tr>
</table>

Table 1: Percentage of samples related to a broader narrative or conspiracy vs. have a fact-checking source.

4.2 RQ2: What are the traits of posts and notes that rely on fact-checking sources?

We begin by performing a topic analysis, comparing topics of posts whose notes reference fact-checking sources to those citing other sources. To this end, we apply a strong zero-shot text classification model6 to our \( S_{text} \) subset by classifying spans of the form “Tweet :<POST TEXT>; Note <NOTE TEXT>” into one of 13 classes. The authors manually evaluated the quality of the classification results and considered it satisfactory, with the model predicting the correct class in 90% of the cases. Most of the incorrect predictions involved the ‘technology’ category, with sentences such as “this is a fake image that was created with AI” incorrectly labelled as ‘technology’. Notably (Fig. 5), fact-checking sources are more likely to be included in posts related to high-stakes issues such as health, science, and scams and less likely to be included in posts on tech or sports.

We then analyse annotations (binary attributes explaining the warrant for the note) by community note authors. Fig. 3 contains the full breakdown of annotations for notes with and without fact-checking sources. Notes containing a link to

6https://huggingface.co/r-f/ModernBERT-large-zeroshot-v1 with default settings.
fact-checking sources are overrepresented in posts where unverified information is presented as a fact or when the post contains a factual error. Conversely, they are under-represented in posts with outdated information or satirical content. Tab. 4 in App. B contains a sample of such notes.

These results indicate that community note-writers adapt their strategies based on the stakes and scope of the claim, and the depth of research needed to counter misinformation. We hypothesise that they are more likely to rely on external fact-checking when refuting complex or unverifiable claims (Wuehrl et al., 2024), as well as claims related to conspiracy theories7 or broader narratives8 which cannot be fully addressed in the scope of a note. Conversely, claims involving misleading media can often be debunked with examples alone, making fact-checking sources unnecessary. To investigate this hypothesis, the authors of this paper manually annotated 400 <post, note> pairs from \( S_{text} \) with attributes related to the complexity of the claims and how community notes address them (see App. C.1 for annotation guidelines). The results (Fig. 4.a) support our hypothesis. Claims related to broader narratives or conspiracy theories9 are much more likely to include a link to a fact-checking source. In contrast, other types of claims are more likely to be addressed by providing missing context or by invalidating the credibility of the claim’s source. Additionally, Fig. 4.b depicts the different ways in which fact-checking sources are used to debunk claims. It demonstrates how such sources are rarely used to provide missing context but rather focus on discrediting sources of claims and providing scientific evidence.

We extend the manual annotation to an LLM-based analysis of 8K balanced <post, note> pairs from \( S_{text} \). We task OpenAI’s GPT-410 with determining whether a pair relates to a broader narrative or a conspiracy theory. Listing 2 in App. C details the prompt we used. To evaluate model accuracy, two authors independently labelled 100 balanced pairs, achieving an agreement rate of 0.88 and resolving disagreements through discussion. The

![Bar chart showing distribution of notes’ topics, with and without a fact-checking source.](page_682_134_563_312.png)

Figure 5: Distribution of notes’ topics, with and without a fact-checking source.

model attained an \( F_1 \) score of 0.85—strong performance for this challenging task. The results (Tab. 1) support our hypothesis: <post, note> pairs related to a broader narrative or conspiracy theory are *twice* as likely to cite fact-checking sources compared to other sources. In contrast, other pairs are nearly 30% less likely to do so. These findings also highlight the prevalence of such claims and further underscore the importance of fact-checking in combating complex misinformation narratives.

5 Conclusion

In this work, we annotate a large corpus of Twitter community notes with attributes such as topic, cited sources, and whether they refute claims tied to broader misinformation narratives. We find that effective community moderation depends on professional fact-checking to an extent far greater than previously reported. We find that community notes linked to broader narratives or conspiracy theories are particularly reliant on fact-checking.

Our results reveal that community notes and professional fact-checking are deeply interconnected—fact-checkers conduct in-depth research beyond the reach of amateur platform users, while community notes publicise their work. The move by platforms to end their partnerships and funding for fact-checking organisations will hinder their ability to fact-check and pursue investigative journalism, which community note writers rely on. This, in turn, will limit the efficacy of community notes, especially for high-stakes claims tied to broader narratives or conspiracies.

Limitations

The main limitations of our work concern the characteristics of the dataset we analyse. First, we restrict our analysis to notes written in English, excluding over half a million notes in other languages.

7For example, the claim “Michelle Obama is a male”.
8These are tweets that perpetuate broader misinformation narratives but do not necessarily contain an explicit conspiracy theory, e.g., the false claim “most immigrants remain firmly dependent on Welfare”.
9We condense broader narratives and conspiracy theories together for simplicity. Similar trends can be seen when they are analysed separately.
10Version gpt-4o-2024-08-06.
This decision was made to avoid potential noise and biases arising from the authors’ unfamiliarity with public discourse in different regions and reliance on machine translation. In future work, we aim to extend our analysis to other languages.

Moreover, except for a small subset of notes, we did not have access to the original tweets they were written for. Even when the tweet text was available, many contained non-text media, were written in internet vernacular that was challenging to interpret, or lacked important context. These factors limit the accuracy and effectiveness of our models and analysis.

Finally, due to resource constraints, our manual annotation study was limited to a relatively small sample of tweets and notes. In future work, we wish to utilise crowd workers to not only annotate a larger dataset but also increase the diversity and perspective of the annotators.

Broader Impact and Ethical Considerations

Community notes have been proposed as a replacement for professional fact-checkers and a salve to some of the issues encountered by fact-checking. However, our findings support the view that neither community notes nor professional fact-checkers alone are sufficient to combat the spread of misinformation on social media. Rather, a combination of these two strategies could prove a much more effective approach to addressing the full range of false content shared online, for example, by leveraging community notes to identify new checkworthy claims for professional fact-checkers, or relying on fact-checkers’ expertise to resolve disputed unpublished notes (see Augenstein et al. (2025) for further recommendations in this vein). In particular, as discussed above, professional fact-checking organisations are especially vital for verifying claims related to broader narratives and conspiracy theories. Moreover, professional fact-checkers remain the only viable strategy currently available for addressing partisan issues, where community notes fall short. Finally, we highlight the potential for incorporating automated, human-in-the-loop fact-checking models to assist professional and community fact-checkers alike in reckoning with vast amounts of both human- and machine-generated content.

Given that this work analyses real-world posts, ethical concerns may arise from using this data for research purposes. Posts from non-protected accounts and Community Notes on Twitter/X are publicly available, however, we acknowledge that they may contain sensitive personal information. To minimise any breach of anonymity and privacy, we anonymised links to individual accounts, and we do not publicly release this information. We do not analyse the posts or notes by individual users, and instead examine aggregated data in the form of topics and sources cited.

Although the Community Notes dataset represents attempts to curb harmful misinformation and conspiracies, given the intense partisanship involved (Allen et al., 2022; Draws et al., 2022), as well as the explicit content of some claims, some instances may be considered offensive. We also acknowledge that our own perspectives and biases as authors shape the impact of our findings in certain ways. For example, as mentioned in the previous section, we were unable to analyse non-English posts in-depth, so our conclusions are likely somewhat focused on discourse in the Anglosphere (e.g., the US, UK, Ireland, Canada, Australia, New Zealand etc.). Furthermore, although we based our criteria for conspiracy theories on well-established sources, e.g., AP News, FactCheck.org, the European Commission, and identified conspiratorial narratives from both left- and right-wing sources, our own perspectives (i.e., as scientists from Western countries) may also have impacted what we considered to be conspiracy theories.

Acknowledgements

This research was co-funded by the European Union (ERC, ExplainYourself, 101077481), by the European Union’s Horizon 2020 research and innovation program under grant agreement No. 101135671 (TrustLLM), and by the Pioneer Centre for AI, DNRF grant number P1.

References

Jennifer Allen, Cameron Martel, and David G. Rand. 2022. Birds of a Feather Don’t Fact-Check Each Other: Partisanship and the Evaluation of News in Twitter’s Birdwatch Crowdsourced Fact-Checking Program. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery.

Phoebe Arnold. 2020. The Challenges of Online Fact Checking: How Technology Can (and Can’t) Help. Technical report, Full Fact.
Isabelle Augenstein, Michiel Bakker, Tanmoy Chakraborty, David Corney, Emilio Ferrara, Iryna Gurevych, Scott Hale, Eduard Hovy, Heng Ji, Irene Larraz, Filippo Menczer, Preslav Nakov, Paolo Papotti, Dhruv Sahnani, Greta Warren, and Giovanni Zagni. 2025. Community moderation and the new epistemology of fact checking on social media.

Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, Eduard Hovy, Heng Ji, Filippo Menczer, Ruben Miguez, Preslav Nakov, Dietmar Scheufele, Shivam Sharma, and Giovanni Zagni. 2024. Factuality Challenges in the Era of Large Language Models and Opportunities for Fact-Checking. Nature Machine Intelligence, pages 1–12.

Rebecca Catalanello and Katie Sanders. 2025. Meta is Ending Its Third-Party Fact-Checking Partnership with US Partners. Here’s How That Program Works.

Yuwei Chuai, Moritz Pilarski, Gabriele Lenzini, and Nicolas Pröllochs. 2024a. Community Notes Reduce the Spread of Misleading Posts on X.

Yuwei Chuai, Anastasia Sergeeva, Gabriele Lenzini, and Nicolas Pröllochs. 2024b. Community Fact-Checks Trigger Moral Outrage in Replies to Misleading Posts on Social Media. ArXiv:2409.08829 [cs].

Yuwei Chuai, Haoye Tian, Nicolas Pröllochs, and Gabriele Lenzini. 2024c. Did the Roll-Out of Community Notes Reduce Engagement With Misinformation on X/Twitter? Proceedings of the ACM on Human-Computer Interaction, 8(CSCW2):1–52.

Suárez Vicente Javier Clemente, Eduardo Navarro-Jiménez, Juan Antonio Simón-Sanjurjo, Ana Isabel Beltrán-Velasco, Carmen Cecilia Laborde-Cárdenas, Juan Camilo Benítez-Agudelo, Álvaro Bustamante-Sánchez, and José Francisco Tornero-Aguilera. 2022. Mis–Dis Information in COVID-19 Health Crisis: A Narrative Review. International Journal of Environmental Research and Public Health, 19(9).

Nicholas Diakopoulos. 2020. Computational News Discovery: Towards Design Considerations for Editorial Orientation Algorithms in Journalism. Digital journalism, 8(7):945–967.

Tim Draws, David La Barbera, Michael Soprano, Kevin Roitero, Davide Ceolin, Alessandro Checco, and Stefano Mizzaro. 2022. The Effects of Crowd Worker Biases in Fact-Checking Tasks. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 2114–2124, Seoul Republic of Korea. ACM.

Chiara Patricia Drolsbach and Nicolas Pröllochs. 2023. Diffusion of Community Fact-Checked Misinformation on Twitter. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2):1–22.

Chiara Patricia Drolsbach, Kirill Solovev, and Nicolas Pröllochs. 2024. Community Notes Increase Trust in Fact-Checking on Social Media. PNAS Nexus.

Daniela Flaminii. 2019. Most Republicans Don’t Trust Fact-Checkers, and Most Americans Don’t Trust the Media.

Yang Gao, Maggie Zhang, and Huaxia Rui. 2024. Can Crowdchecking Curb Misinformation? Evidence from Community Notes.

Sarah Gilbert. 2025. Three Reasons Meta Will Struggle with Community Fact-Checking. MIT Technology Review.

Google. 2025. Misinformation Policies - Youtube Help.

Lucas Graves. 2017. Anatomy of a Fact Check: Objective Practice and the Contested Epistemology of Fact Checking. Communication, culture & critique, 10(3):518–537.

Lucas Graves and C.W. Anderson. 2020. Discipline and Promote: Building Infrastructure and Managing Algorithms in a “Structured Journalism” Project by Professional Fact-Checking Groups. New Media & Society, 22(2):342–360.

Jordyn Haime. 2022. Taiwan’s Amateur Fact-Checkers Wage War on Fake News from China.

IFCN. 2024. State of the Fact-Checkers Report. Technical report, International Fact-Checking Network.

Prerna Juneja and Tanushree Mitra. 2022. Human and Technological Infrastructures of Fact-Checking. Proc. ACM Hum.-Comput. Interact., 6(CSCW2).

Uku Kangur, Roshni Chakraborty, and Rajesh Sharma. 2024. Who Checks the Checkers? Exploring Source Credibility in Twitter’s Community Notes. ArXiv:2406.12444 [cs].

Sarawut Kankham and Jian-Ren Hou. 2024. Community Notes vs. Related Articles: Assessing Real-World Integrated Counter-Rumor Features in Response to Different Rumor Types on Social Media. International Journal of Human–Computer Interaction, pages 1–15.

Laurens Lauer and Lucas Graves. 2024. How to Grow a Transnational Field: A Network Analysis of the Global Fact-Checking Movement. New Media & Society, 0(0):19.

Maldita. 2025. The Impact of Fact-Checkers in X’s Community Notes.

Cameron Martel, Jennifer Allen, Gordon Pennycook, and David G. Rand. 2024. Crowds Can Effectively Identify Misinformation at Scale. Perspectives on Psychological Science, 19(2):477–488. PMID: 37594056.

Meta. 2025. More Speech and Fewer Mistakes.

Nicholas Micallef, Vivienne Armacost, Nasir Memon, and Sameer Patil. 2022. True or False: Studying the Work Practices of Professional Fact-Checkers. Proc. ACM Hum.-Comput. Interact., 6(CSCW1).
Garrett Morrow, Briony Swire-Thompson, Jessica Montgomery Polny, Matthew Kopec, and John P Wihbey. 2022. The Emerging Science of Content Labeling: Contextualizing Social Media Content Moderation. Journal of the Association for Information Science and Technology, 73(10):1365–1386.

Moritz Pilarski, Kirill Olegovich Solovev, and Nicolas Pröllochs. 2024. Community Notes vs. Snoping: How the Crowd Selects Fact-Checking Targets on Social Media. Proceedings of the International AAAI Conference on Web and Social Media, 18:1262–1275.

Nicolas Pröllochs. 2022. Community-Based Fact-Checking on Twitter’s Birdwatch Platform. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 794–805.

Merten Reglitz. 2022. Fake News and Democracy. J. Ethics & Soc. Phil., 22:162.

Thomas Renault, David Restrepo-Amariles, and Aurone Troussel. 2024. Collaboratively Adding Context to Social Media Posts Reduces the Sharing of False News. HEC Research Papers Series 1519, HEC Paris.

Mohammed Saeed, Nicolas Traub, Maelle Nicolas, Gianluca Demartini, and Paolo Papotti. 2022. Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts? In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 1736–1746, New York, NY, USA. Association for Computing Machinery.

Mark Stencel, Erica Ryan, and Joel Luther. 2024. With Half the Planet Going to the Polls in 2024, Fact-Checking Sputters. Technical report, Duke Reporters’ Lab.

TikTok. 2025. Testing a New Feature to Enhance Content on TikTok.

Twitter/X. 2021. Introducing Birdwatch, a Community-Based Approach to Misinformation.

Twitter/X. 2024a. Note Ranking Algorithm.

Twitter/X. 2024b. Rating and Writing Impact.

Greta Warren, Irina Shklovski, and Isabelle Augenstein. 2025. Show Me the Work: Fact-Checkers’ Requirements for Explainable Automated Fact-Checking. In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA. Association for Computing Machinery.

Valerie Wirtschafter and Sharanya Majumder. 2023. Future Challenges for Online, Crowdsourced Content Moderation: Evidence from Twitter’s Community Notes. Journal of Online Trust and Safety, 2(1).

Amelie Wuehrl, Yarik Menchaca Resendiz, Lara Grimmingher, and Roman Klinger. 2024. What Makes Medical Claims (Un)Verifiable? Analyzing Entity and Relation Properties for Fact Verification. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2046–2058, St. Julian’s, Malta. Association for Computational Linguistics.

Taha Yasseri and Filippo Menczer. 2023. Can Crowdsourcing Rescue the Social Marketplace of Ideas? Communications of the ACM, 66(9):42–45.

Andy Zhao and Mor Naaman. 2023. Insights from a Comparative Study on the Variety, Velocity, Veracity, and Viability of Crowdsourced and Professional Fact-Checking Services. Journal of Online Trust and Safety, 2(1).
A Extended Background

A.1 Impact of Community Notes on misinformation spread

Posts identified by community notes as misleading have been found to attain less virality (reposts, quote tweets and replies) than non-misleading posts (Drolsbach and Pröllochs, 2023; Renault et al., 2024). Community notes have also been shown to increase the probability of tweet retractions and deletions and speed up the retraction process (Gao et al., 2024; Renault et al., 2024). However, other studies have found less positive evidence; for example, that users’ followers, likes and engagement increase after their post receives a community note (Wirtschafter and Majumder, 2023). Curiously, one study claims that showing community notes on posts reduced the spread of misleading posts by an average of 61% (Chuai et al., 2024a), while a more recent analysis by the same authors found no effect of community notes on engagement with misinformation (Chuai et al., 2024c).

People shown community notes alongside misleading social media posts were more accurate in identifying misleading posts, and the notes were judged to be more trustworthy than context-free misinformation flags (e.g., "Checked by fact-checkers" or "Checked by other social media users"), regardless of (US-centric) political beliefs (Drolsbach et al., 2024). People shown either community notes or related news article suggestions were both less likely to believe and report misleading information compared to a control group: community notes were more effective in reducing belief and sharing intention for positive rumours, while articles were more effective for negative rumours (Kankham and Hou, 2024). On the other hand, displaying community notes leads users to post more negative and angry replies to misleading posts (Chuai et al., 2024b), while crowd workers are also prone to cognitive biases, such as overestimating a statement’s truthfulness the more they liked its claimant, and general overconfidence in their ability to ascertain truthful statements (Draws et al., 2022).

A.2 Professional fact-checking and community note practices

Although fact-checks and community notes share similarities in how they address misleading claims, they also differ in key elements of practice and communication (Kankham and Hou, 2024). Fact-checking typically involves the analysis and verification of public claims, and in addition to verifying claims, in recent years many fact-checking organisations have also assumed a wider role in combating misinformation spread, e.g., long-term investigative journalism projects, media literacy programs (Juneja and Mitra, 2022). Professional fact-checkers signatory to the International Fact-Checking Network follow a rigorous set of principles and transparency commitments11 and a structured workflow: (i) claim selection; (ii) collecting evidence; (iii) deciding on a verdict; and (iv) writing the fact-checking article (Graves, 2017; Micallef et al., 2022; Warren et al., 2025). In contrast, any platform user can contribute to community notes under anonymity, and the rating approach relies on the ‘wisdom of crowds’, with little oversight or transparency regarding biases of note-writers. Community note writers and fact-checkers tend to target similar topics (e.g., health and politics) (Saeed et al., 2022). Fact-checkers must rely on credible sources and evidence to convince the reader, while note writers often disagree with fact-checkers on what constitutes a reliable source, particularly on political ideological grounds (Saeed et al., 2022). The verdicts reached in notes tend to agree with those of fact-checkers — however, due to political polarisation (Yasser and Menczer, 2023), community notes on contentious political issues rarely reach a consensus (Saeed et al., 2022). Fact-checking articles, which are subject to multiple rounds of editorial scrutiny, are more formal and standardised in style than community notes, which vary considerably and can employ a range of persuasion techniques, such as appeals to emotion or other logical fallacies (Kankham and Hou, 2024). Moreover, community notes typically serve as direct rebuttals to misleading posts, while fact-checking articles may address a more general claim than is expressed in a specific post. Finally, fact-checking articles are a one-way exchange, while community notes represent a more horizontal and interactive dialogue between writer and recipient of the fact-check (Kankham and Hou, 2024). Prior work has found that collaborative fact-checking, a distinct approach to community notes for its Wikipedia-style approach that allows users to edit, as well as up and downvote user-written posts (Haime, 2022), can produce fact-

11https://www.ifcncodeofprinciples.poynter.org/the-commitments
checks with comparable speed, reliability, objectivity, clarity and persuasiveness to those written by professional fact-checks (Zhao and Naaman, 2023). However, laypeople’s work is expedited by existing fact-checking articles, and amateurs tend to defer to professional fact-checkers for topics requiring specific expertise, such as medical claims (Zhao and Naaman, 2023). Our work builds on current understanding of the relationship between professional fact-checking and community moderation by examining the extent to which community note writers deploy the work of fact-checkers in their notes.

B Additional Material

This section details additional results or material referenced from the paper’s main body.

Fig. 6 A histogram of the number of community notes written every month and their rating (helpful, not helpful, or needs more data).

Fig. 7 Community ratings of notes with and without fact-checking source.

Tab. 2 List of top 100 most common domains found in the community notes dataset, and their categorisation.

Tab. 3 List of professional fact-checking organisations and their URLs.

Tab. 4 A sample of tweets, notes, and their community annotations, as well as whether the note contains a fact-checking link.

Tab. 5 Examples of community notes containing fact-checking sources that are rated as having notHelpfulSourcesMissingOrUnreliable.

C Reproducibility

Listing 1 The prompt used to classify URLs into categories.

Listing 2 The prompt used to classify tweets and notes into broader narratives and conspiracy theories.

C.1 Manual Annotation Setup

We annotate 400 (tweet, note) pairs from \( S_{text} \) with 12 binary attributes. Each (tweet, note) pair was annotated in a multi-label fashion, i.e., more than one attribute can be selected at the same time. Fig. 8 depict our simple annotation setup, with the 12 attributes being as follows.

Broader narrative Whether the (tweet, note) pair is related to a broader narrative or a conspiracy theory.

Discredit source of claim If the community note describes the source shared by the original post as non-credible.

Add missing context If the community note provides some missing context to refute a claim.

Highlight AI generated If the community note claims that the post shared AI-generated content.

Highlight edited media If the community note claims that the post shared some media that was edited (edited with Photoshop, the clip was cut, etc.).

Link to direct source If the community note shares a link to a source where an entity says that a claim made about them is false.

Link official source If the community note shares a link to an official source such as a government website.

Link scientific source If the community note shares a link to some scientific article or website.

Link world knowledge If the community note shares a link to some reference resources such as Wikipedia.

Link fact-checking If the community note shares a link to a professional fact-checking organisation.

In-note fact-checking If the community note performs an in-note fact-check by cross-referencing several sources and constructing a compelling argument.
<table>
  <tr>
    <th>Domain</th>
    <th>Category</th>
    <th>Domain</th>
    <th>Category</th>
  </tr>
  <tr><td>x.com</td><td>social media</td><td>thehill.com</td><td>news</td></tr>
  <tr><td>twitter.com</td><td>social media</td><td>amp.theguardian.com</td><td>news</td></tr>
  <tr><td>youtube.com</td><td>social media</td><td>whitehouse.gov</td><td>government</td></tr>
  <tr><td>youtu.be</td><td>social media</td><td>news.sky.com</td><td>news</td></tr>
  <tr><td>un.org</td><td>organisation</td><td>merriam-webster.com</td><td>reference</td></tr>
  <tr><td>u.today</td><td>news</td><td>techarp.com</td><td>news</td></tr>
  <tr><td>t.co</td><td>social media</td><td>cbc.ca</td><td>news</td></tr>
  <tr><td>snopes.com</td><td>fact checking</td><td>politifact.com</td><td>fact checking</td></tr>
  <tr><td>en.m.wikipedia.org</td><td>reference</td><td>pbs.org</td><td>commercial</td></tr>
  <tr><td>en.wikipedia.org</td><td>reference</td><td>telegraph.co.uk</td><td>news</td></tr>
  <tr><td>google.com</td><td>search engine</td><td>businessinsider.com</td><td>news</td></tr>
  <tr><td>instagram.com</td><td>social media</td><td>time.com</td><td>news</td></tr>
  <tr><td>britannica.com</td><td>reference</td><td>justice.gov</td><td>government</td></tr>
  <tr><td>reuters.com</td><td>news</td><td>cnbc.com</td><td>news</td></tr>
  <tr><td>bbc.co.uk</td><td>news</td><td>wsj.com</td><td>news</td></tr>
  <tr><td>apnews.com</td><td>news</td><td>sciencedirect.com</td><td>academic</td></tr>
  <tr><td>bbc.com</td><td>news</td><td>msn.com</td><td>news</td></tr>
  <tr><td>nytimes.com</td><td>news</td><td>statista.com</td><td>reference</td></tr>
  <tr><td>theguardian.com</td><td>news</td><td>business.x.com</td><td>commercial</td></tr>
  <tr><td>vice.com</td><td>news</td><td>amp.cnn.com</td><td>news</td></tr>
  <tr><td>usatoday.com</td><td>news</td><td>congress.gov</td><td>government</td></tr>
  <tr><td>factcheck.org</td><td>fact checking</td><td>factcheck.afp.com</td><td>fact checking</td></tr>
  <tr><td>cnn.com</td><td>news</td><td>yahoo.com</td><td>search engine</td></tr>
  <tr><td>washingtonpost.com</td><td>news</td><td>timesofindia.indiatimes.com</td><td>news</td></tr>
  <tr><td>ncbi.nlm.nih.gov</td><td>academic</td><td>thelancet.com</td><td>academic</td></tr>
  <tr><td>nbcnews.com</td><td>news</td><td>hrw.org</td><td>organisation</td></tr>
  <tr><td>help.twitter.com</td><td>reference</td><td>healthfeedback.org</td><td>fact checking</td></tr>
  <tr><td>cdc.gov</td><td>government</td><td>fda.gov</td><td>government</td></tr>
  <tr><td>npr.org</td><td>news</td><td>m.youtube.com</td><td>social media</td></tr>
  <tr><td>forbes.com</td><td>news</td><td>law.cornell.edu</td><td>academic</td></tr>
  <tr><td>newsweek.com</td><td>news</td><td>medium.com</td><td>blog post</td></tr>
  <tr><td>fullfact.org</td><td>fact checking</td><td>healthfeedback.org</td><td>fact checking</td></tr>
  <tr><td>dailymail.co.uk</td><td>news</td><td>who.int</td><td>organisation</td></tr>
  <tr><td>cbsnews.com</td><td>news</td><td>haaretz.com</td><td>news</td></tr>
  <tr><td>web3antivirus.io</td><td>database</td><td>axios.com</td><td>news</td></tr>
  <tr><td>timesofisrael.com</td><td>news</td><td>mayoclinic.org</td><td>commercial</td></tr>
  <tr><td>help.x.com</td><td>reference</td><td>nejm.org</td><td>academic</td></tr>
  <tr><td>nypost.com</td><td>news</td><td>scienceexchange.caltech.edu</td><td>academic</td></tr>
  <tr><td>aljazeera.com</td><td>news</td><td>indiatoday.in</td><td>news</td></tr>
  <tr><td>reddit.com</td><td>social media</td><td>bloomberg.com</td><td>news</td></tr>
  <tr><td>independent.co.uk</td><td>news</td><td>pewresearch.org</td><td>academic</td></tr>
  <tr><td>usgs.gov</td><td>academic</td><td>jamanetwork.com</td><td>academic</td></tr>
  <tr><td>abcnews.go.com</td><td>news</td><td>leadstories.com</td><td>news</td></tr>
  <tr><td>nature.com</td><td>academic</td><td>dictionary.cambridge.org</td><td>reference</td></tr>
  <tr><td>gov.uk</td><td>government</td><td>jpost.com</td><td>news</td></tr>
  <tr><td>web.archive.org</td><td>database</td><td>archive.ph</td><td>database</td></tr>
  <tr><td>foxnews.com</td><td>news</td><td>healthline.com</td><td>commercial</td></tr>
  <tr><td>tiktok.com</td><td>social media</td><td>abc.net.au</td><td>news</td></tr>
  <tr><td>edition.cnn.com</td><td>news</td><td>france24.com</td><td>news</td></tr>
</table>

Table 2: List of top 100 most common domains found in the community notes dataset, and their categorisation.
<table>
  <tr>
    <th>Name</th>
    <th>URL</th>
    <th>Language</th>
    <th>Region/domain</th>
  </tr>
  <tr>
    <td>Lead stories</td>
    <td>leadstories.com</td>
    <td>English</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>AFP Factuel</td>
    <td>factuel.afp.com</td>
    <td>French</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>AAAP FactCheck</td>
    <td>aap.com.au/factcheck</td>
    <td>English</td>
    <td>Australia</td>
  </tr>
  <tr>
    <td>Full Fact</td>
    <td>fullfact.org</td>
    <td>English</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>Science Feedback</td>
    <td>science.feedback.org</td>
    <td>English</td>
    <td>Science</td>
  </tr>
  <tr>
    <td>Politifact</td>
    <td>politifact.com</td>
    <td>English, Spanish</td>
    <td>USA</td>
  </tr>
  <tr>
    <td>HoaxEye</td>
    <td>hoaxeye.wordpress.com</td>
    <td>English</td>
    <td>Images</td>
  </tr>
  <tr>
    <td>Logically Facts</td>
    <td>logicallyfacts.com</td>
    <td>Multiple</td>
    <td>Europe/India</td>
  </tr>
  <tr>
    <td>FactCheckNI</td>
    <td>factcheckni.org</td>
    <td>English</td>
    <td>North Ireland</td>
  </tr>
  <tr>
    <td>DFRLab</td>
    <td>dfrlab.org</td>
    <td>English</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>FactReview</td>
    <td>factreview.gr</td>
    <td>Greek</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>Lupa</td>
    <td>lupa.uol.com.br/jornalismo</td>
    <td>Portuguese</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>Check your fact</td>
    <td>checkyourfact.com</td>
    <td>English</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>Climate feedback</td>
    <td>climatefeedback.org</td>
    <td>English</td>
    <td>Climate</td>
  </tr>
  <tr>
    <td>Factcheck</td>
    <td>factcheck.afp.com</td>
    <td>English</td>
    <td>USA</td>
  </tr>
  <tr>
    <td>Health feedback</td>
    <td>healthfeedback.org</td>
    <td>English</td>
    <td>Health</td>
  </tr>
  <tr>
    <td>Snopes</td>
    <td>snopes.com</td>
    <td>English</td>
    <td>US</td>
  </tr>
  <tr>
    <td>aosfatos</td>
    <td>aosfatos.org</td>
    <td>Portuguese</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>Demagog</td>
    <td>demagog.org.pl/fake_news</td>
    <td>Polish</td>
    <td>Poland</td>
  </tr>
  <tr>
    <td>FakeReporter</td>
    <td>fakereporter.net</td>
    <td>Hebrew</td>
    <td>Israel</td>
  </tr>
  <tr>
    <td>litmus factcheck</td>
    <td>litmus-factcheck.jp</td>
    <td>Japanese</td>
    <td>Japan</td>
  </tr>
  <tr>
    <td>Climate Feedback</td>
    <td>climatefeedback.org</td>
    <td>English</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>AFP</td>
    <td>factcheck.afp.com</td>
    <td>English</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>USA Today</td>
    <td>usatoday.com/story/news/factcheck</td>
    <td>English</td>
    <td>USA</td>
  </tr>
  <tr>
    <td>Statesman</td>
    <td>statesman.com</td>
    <td>English</td>
    <td>USA</td>
  </tr>
  <tr>
    <td>Dallas News</td>
    <td>dallasnews.com/news/politifact</td>
    <td>English</td>
    <td>USA</td>
  </tr>
  <tr>
    <td>Google Fact Check</td>
    <td>toolbox.google.com/factcheck</td>
    <td>English</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>MediaBias/FactCheck</td>
    <td>mediabiasfactcheck.com</td>
    <td>English</td>
    <td>Global</td>
  </tr>
  <tr>
    <td>MedDMO</td>
    <td>meddmo.eu</td>
    <td>English, Greek</td>
    <td>Greece, Cyprus, Malta</td>
  </tr>
  <tr>
    <td>Poynter</td>
    <td>poynter.org/fact-checking</td>
    <td>English</td>
    <td>USA</td>
  </tr>
  <tr>
    <td>Newsmeter</td>
    <td>newsmeter.in/fact-check</td>
    <td>English, Tamil</td>
    <td>India</td>
  </tr>
  <tr>
    <td>Africa Check</td>
    <td>africacheck.org</td>
    <td>English</td>
    <td>Africa</td>
  </tr>
  <tr>
    <td>Fact Crescendo India</td>
    <td>english.factcrescendo.com</td>
    <td>English</td>
    <td>India</td>
  </tr>
  <tr>
    <td>Factseeker</td>
    <td>factseeker.lk</td>
    <td>English</td>
    <td>Sri Lanka</td>
  </tr>
  <tr>
    <td>Fact Crescendo Thailand</td>
    <td>thailand.factcrescendo.com</td>
    <td>Thai</td>
    <td>Thailand</td>
  </tr>
  <tr>
    <td>Fact Crescendo Afghanistan</td>
    <td>afghanistan.factcrescendo.com</td>
    <td>Persian</td>
    <td>Afghanistan</td>
  </tr>
  <tr>
    <td>Only Fact</td>
    <td>onlyfact.in</td>
    <td>English</td>
    <td>India</td>
  </tr>
  <tr>
    <td>Factly</td>
    <td>factly.in</td>
    <td>English</td>
    <td>India</td>
  </tr>
  <tr>
    <td>Fact Crescendo Sri Lanka</td>
    <td>srilanka.factcrescendo.com</td>
    <td>Sinhala</td>
    <td>Sri Lanka</td>
  </tr>
  <tr>
    <td>Fact Crescendo Cambodia</td>
    <td>cambodia.factcrescendo.com</td>
    <td>Cambodian</td>
    <td>Cambodia</td>
  </tr>
  <tr>
    <td>Becid</td>
    <td>becid.eu</td>
    <td>Baltic langs</td>
    <td>Baltic</td>
  </tr>
  <tr>
    <td>Fact Hunt</td>
    <td>facthunt.in</td>
    <td>English</td>
    <td>India</td>
  </tr>
  <tr>
    <td>Tec Arp</td>
    <td>techarp.com</td>
    <td>English</td>
    <td>Global (based in Malaysia)</td>
  </tr>
  <tr>
    <td>10 news</td>
    <td>10news.com/news/fact-or-fiction</td>
    <td>English</td>
    <td>USA</td>
  </tr>
  <tr>
    <td>RMIT Fact Check</td>
    <td>rmit.edu.au</td>
    <td>English</td>
    <td>Australia</td>
  </tr>
  <tr>
    <td>Gigafact</td>
    <td>gigafact.org</td>
    <td>English</td>
    <td>USA</td>
  </tr>
  <tr>
    <td>Ayupp</td>
    <td>ayupp.com/fact-check</td>
    <td>English</td>
    <td>India</td>
  </tr>
  <tr>
    <td>The Journal</td>
    <td>thejournal.ie</td>
    <td>English</td>
    <td>Ireland</td>
  </tr>
</table>

Table 3: List of professional fact-checking organisations and their URLs.
<table>
  <tr>
    <th>Tweet</th>
    <th>Note</th>
    <th>misleading/OutdatedInformation</th>
    <th>misleading/FactualError</th>
    <th>misleading/Satire</th>
    <th>Fact Checking source</th>
  </tr>
  <tr>
    <td>The NASA War Document is absolutely terrifying https://t.co/...</td>
    <td>misrepresenting a presentation by NASA scientist Dennis Bushnell. The lecture was not detailing plans by NASA to attack the world it was a lecture for defense industry professionals, and how defense tactics might rise to meet evolving threats in the future. https://leadstories.com/hoax-alert/2021/06/fact-check-the-future-is-now-is-not-a-nasa-war-document-plan-for-world-domination-and-phasing-out-of-humans.html</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>BREAKING NEWS: International Criminal Investigating calls on every public citizen to recommend indictment for Bill Gates, Anthony Fauci, Pfizer, BlackRock, Tedros and Christian Drosten for pushing everyone to receive the ineffective highly dangerous lethal experimental vaccines...</td>
    <td>Video has been fact-checked by USA Today, was found to be misleading, and promotes a conspiracy theory about COVID ... https://ca.movies.yahoo.com/movies/fact-check-viral-video-promotes-204414488.html</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>1) California is RED. It is just because of the MASSIVE Election Fraud that stupid, brain-washed people believe Calif. is blue. Joe Biden won only in the SFO Bay area ...</td>
    <td></td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Davis blows up $100,000 fireworks in Kai Cenat setup During the Mr Beast Stream ...</td>
    <td></td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>@enviolations I swear community notes are the only good thing Elon added since he bought Twitter.</td>
    <td>The map shows the results of Reagan’s reelection in 1984, not Biden’s election in 2020. https://en.wikipedia.org/wiki/1984_United_States_presidential_election_in_California<br>The second photo is from a house fire in Atlanta in 2019. https://www.lalalive.com/article/news/local/woodland-brook-drive-cause-of-house-fire-85-ecb7df9b-5f65-4ae9-bf9d-8c162d36c334</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Thailand will become the first country to make the contract null and void, meaning that Pfizer will become responsible for all vaccine injuries ...</td>
    <td>Community notes was first launched under former Twitter CEO Jack Dorsey in 2021 under the name of ‘Birdwatch’. The only thing Elon Musk did was that he renamed the feature to community notes. https://blog.twitter.com/en_us/topics/product/2021/introducing-birdwatch-a-community-based-approach-to-misinformation https://www.reuters.com/article/factcheck-elon-birdwatch-idUSL1N3JZVG/</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Hilarious tweets by footballers. A thread: 1. Virgil Van Dijk [Current Liverpool Captain] https://t.co/...</td>
    <td>Thailand has no plans to void its Pfizer COVID vaccine contract, an official with the country’s National Vaccine Institute said. Thailand’s Department of Disease Control also rejected the claim as false news. https://apnews.com/article/fact-check-covid-vaccine-pfizer-thailand-203948163859</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Rob Reiner announces he’s on the Epstein Client List and Epstein Flight logs. What a fool! When a lawyer tells me to STFU, I STFU! https://t.co/...</td>
    <td>Virgil Van Dijk did not tweet this, the tweet was made by a fan account in his name. https://www.pinkvilla.com/sport/facts/virgil-dijk-virgil-van-dijk-really-root-for-man-u-because-no-one-likes-liverpool-in-resurfaced-viral-tweet-1287250<br>This is a digitally altered photo that might be misinterpreted even if used as a joke. The name Rob Reiner is misspelled, and the text is not on Reiner’s X timeline. https://twitter.com/robreiner?t=iqu43-NszIW5oOM_KqRSpw</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
</table>

Table 4: A sample of tweets, notes, and their community annotations, as well as whether the note contains a fact-checking link.
<table>
  <tr>
    <th>ID</th>
    <th>summary</th>
  </tr>
  <tr>
    <td>0</td>
    <td>This claim ruled mostly false. <a href="https://www.politifact.com/factchecks/2020/may/07/facebook-posts/facebook-post-cites-doctors-widely-disputed-calcul/">https://www.politifact.com/factchecks/2020/may/07/facebook-posts/facebook-post-cites-doctors-widely-disputed-calcul/</a></td>
  </tr>
  <tr>
    <td>1</td>
    <td>The RedState article claims “the shots do not stop transmission of the virus. This is false. “‘Vaccines provide significant protection from ‘getting it’ – infection – and ‘spreading it’ – transmission – even against the delta variant.” Source: <a href="https://www.usatoday.com/story/news/factcheck/2021/11/17/fact-check-covid-19-vaccines-protect-against-infection-transmission/6403678001/">https://www.usatoday.com/story/news/factcheck/2021/11/17/fact-check-covid-19-vaccines-protect-against-infection-transmission/6403678001/</a></td>
  </tr>
  <tr>
    <td>2</td>
    <td>There is no proof of this, the photo is real, it’s not the last photo of the child. But snoops say there is a tenuous link the parents used the same law firm to represent them as Maxwell <a href="https://www.snopes.com/fact-check/ghislaine-maxwell-jonbent-ramsey/">https://www.snopes.com/fact-check/ghislaine-maxwell-jonbent-ramsey/</a></td>
  </tr>
  <tr>
    <td>3</td>
    <td>unfounded <a href="https://www.snopes.com/fact-check/ashley-biden-diary-afraid/">https://www.snopes.com/fact-check/ashley-biden-diary-afraid/</a></td>
  </tr>
  <tr>
    <td>4</td>
    <td>The mRNA vaccine does not cause cancer: <a href="https://www.factcheck.org/2024/05/still-no-evidence-covid-19-vaccination-increases-cancer-risk-despite-posts/">https://www.factcheck.org/2024/05/still-no-evidence-covid-19-vaccination-increases-cancer-risk-despite-posts/</a></td>
  </tr>
  <tr>
    <td>5</td>
    <td>Many of the details in this popular essay are inaccurate and too numerous to list here. The essay was fact checked by Snopes in 2005: <a href="https://www.snopes.com/fact-check/the-price-they-paid/">https://www.snopes.com/fact-check/the-price-they-paid/</a></td>
  </tr>
  <tr>
    <td>6</td>
    <td>POLITIFACT - rates False. The report analysed a small sample of 128 temp stations out of several thousand volunteer-run stations, then extrapolated results. NOAA uses 2 programs to record daily temps. The report did not look at the 900 more sophisticated automated stations. <a href="https://www.politifact.com/factchecks/2022/aug/19/facebook-posts/fact-checking-talking-point-about-corrupted-climat/">https://www.politifact.com/factchecks/2022/aug/19/facebook-posts/fact-checking-talking-point-about-corrupted-climat/</a></td>
  </tr>
  <tr>
    <td>7</td>
    <td>There is no verifiable evidence of campaign espionage in either the 2020 or the 2016 presidential elections. <a href="https://www.snopes.com/fact-check/obama-spying-trump-campaign/">https://www.snopes.com/fact-check/obama-spying-trump-campaign/</a> <a href="https://www.washingtonpost.com/politics/2019/05/06/whats-evidence-spying-trumps-campaign-heres-your-guide/">https://www.washingtonpost.com/politics/2019/05/06/whats-evidence-spying-trumps-campaign-heres-your-guide/</a></td>
  </tr>
  <tr>
    <td>8</td>
    <td>Ladapo did get caught altering COVID vaccine study findings. Ladapo replaced the language from an earlier study draft that found no significant risk from COVID vaccines, to then state there was a high risk <a href="https://healthfeedback.org/claimreview/analyses-florida-department-health-surgeon-general-joseph-ladopo-contains-multiple-methodological-problems-covid-19-mrna-vaccines/">https://healthfeedback.org/claimreview/analyses-florida-department-health-surgeon-general-joseph-ladopo-contains-multiple-methodological-problems-covid-19-mrna-vaccines/</a> <a href="https://healthexec.com/topics/clinical/COVID-19/florida-surgeon-general-altered-covid-19-study-findings/">https://healthexec.com/topics/clinical/COVID-19/florida-surgeon-general-altered-covid-19-study-findings</a></td>
  </tr>
</table>

Table 5: Examples of community notes containing fact-checking sources that are rated as having notHelpful-SourcesMissingOrUnreliable.
Figure 6: A histogram of the number of community notes written every month and their rating (helpful, not helpful, or needs more data). The grey vertical line (December 2022) indicates the date when the community notes became visible worldwide.

Figure 7: Community ratings of notes with and without fact-checking source.
SYSTEM PROMPT
You are a professional IT system who has a vast knowledge of the internet and its content. Your goal is simple, but very important: Classify URLs into categories. Choose only from the provided categories!

USER PROMPT
Read the following URLs.
Your goal is to categorize each url into one of the pre-defined categories.

Chose from the following list of categories:
Categories =
[
    "social media",  # Social media sites like Facebook, Twitter, Youtube etc.
    "news",  # Websites of news outlets or other organisations that report current events, such as the nytimes, the guardian, etc.
    "government",  # Government agencies and organisations, as well as websites related to policies and guidelines, such as the CDC, department of education, FDA, etc.
    "academic",  # Academic sources, journals, and magazines, such as pubmed, nature, sciencedirect, etc.
    "blog post",  # Independent blog posts about various topics, including cooking, travel, home improvement, fandom, reviews, etc.
    "fact checking",  # professional fact checking organisations
    "database",  # Public databases such as google drive, archive.com, dropbox, etc.
    "commercial",  # Webpages of commercial organisations such as BMW, Delta, Nike, etc.
    "reference",  # Public resources such as encyclopedias, dictionaries, advocacy sources, guides, DIYs, statistics, religious sources, travel information, usage guidelines, Q&As, terms of services, etc.
    "organisation",  # non-commercial and non-government organisations such as WHO, the UN, Greenpeace, LA-Lakers, etc.
    "other",  # Any other website that does not fit into one of the previous categories.
    "unknown",  # if it is impossible to determine the category of the webpage.
]

Output format example:
[
    {
        id: <ID>,
        url: <URL>,
        category: <CATEGORY>,
    }
]

URLs:
<URLS>

Listing 1: The prompt used to classify URLs into categories.
SYSTEM PROMPT
You are a professional fact-checker who specializes in analyzing misinformation spread on social media.
Your goal is to analyse a tweet and a community note written about the tweet and decide whether the tweet spread misinformation related to a known conspiracy theory or a misleading wider narrative, and if so, which one is it.

USER PROMPT
Read the following tweets and community notes written about them.\nYour goal is to analyse them and decide whether each tweet spread misinformation related to a known conspiracy theory or a similar misleading wider narrative, and if so (and only if so!), which one.
Include your reasoning. Output the results as a json file. If a tweet does not relate to a conspiracy theory or a misleading wider narrative, output "none" in the json.

- Tweets *do not* discuss a wider narrative if the misleading information is tied to a specific singular event that is not connected to major topics on the public discourse.
They do discuss a wider narrative if the misleading information is tied to a known conspiracy theory or to major topics on the public discourse.

Chose from the following list of theories and wider narrative:
CONSPIRACY_THEORIES =
[
    September 11,
    October 7,
    the great replacement,
    COVID was intentionally spread,
    the COVID outbreak is fake,
    2020 election fraud,
    vaccines cause autism,
    5G towers,
    Russian invasion of Ukraine,
    flat earth,
    chemtrails,
    Q-Anon and deep state,
    Epstein files,
    Barack Obama was not born in the USA,
    Michelle Obama is a man,
    LGBT grooming,
    fluorite in the water,
    climate change,
    Holocaust denial,
    Hunter Biden and Ukraine,
    other,
]

Output format example:
[
    {
        id: <ID>,
        is_related_to_conspiracy: <True/False>,
        conspiracy: <CONSPIRACY (or None)>,
        reasoning: <REASONING>
    }
]

Tweets and notes:
<TWEETS_AND_NOTED>

Listing 2: The prompt used to classify tweets and notes into broader narratives and conspiracy theories.
<table>
  <tr>
    <th>Original tweet</th>
    <th>Note</th>
    <th>Broader narrative</th>
    <th>Comparative theory</th>
    <th>Unreliability of claim</th>
    <th>Link to missing context (e.g., the full context)</th>
    <th>Highlighted AI-generated content</th>
    <th>Highlighted manipulated, edited, staged, Link to original media source</th>
    <th>Link to official source (e.g., verified info)</th>
    <th>List to source</th>
    <th>World knowledge Wikipedia</th>
    <th>List to accurate journalism sources</th>
    <th>Invaluable for debunking multiple sources + organization</th>
  </tr>
  <tr>
    <td>IS THAT A BAT &amp; MOUSER? WHAT?! @ https://t.co/85S3u4vzvD Turkey rush. They put it inside the turkey. https://t.co/j</td>
    <td>BREAKING: Armed communal Muslim gangs are currently storming I</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>They aim to steal the ballot boxes. https://t.co/L1qQGZ927</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Come on, guys. This is not happening. https://t.co/0CjOOGw</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>The video shows an AI in fireworks compilation and that</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Russia is conducting naval exercises using nuclear submarines and it! This video is from 2018 and shows a missile launch in the</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Of course it suggests this money comes from US and western tax payers</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>If you listen to Zelensky speak you’ll think he cares deeply about Ukraine</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>If this post gets a community note immediately vote it unhelpful. The</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>18 million ballots were rejected and judged HOORAY! They are not rejected, they are still being counted. California</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Figure 8: Our annotation setup.

Reviewer paper summaries: The paper investigates the interplay between community-driven fact-checking and professional fact-checking organizations. With policy shifts by platforms like Meta and Twitter away from professional fact-checkers and towards community moderation, the paper seeks to ask to what extent Community Notes rely on professional fact-checkers. 
By analyzing a large corpus of 1.5 M Community Notes, the authors find that up to 7% of helpful notes explicitly reference professional fact-checking sources. Community Notes often rely on professional fact-checking when addressing broader misinformation narratives or conspiracy theories, implying that high-quality crowd moderation is still dependent on professional fact-checking infrastructures.
The paper addresses a pressing and policy-relevant question, especially given platforms’ shift toward crowdsourced moderation,  contributing to the debate over how to combat misinformation at scale.

This paper investigates the relationship between Twitter/X’s Community Notes and professional fact-checking organizations. It explores whether community-driven fact-checking can function independently of professional fact-checkers. Using a large dataset of community notes, the authors analyze how frequently fact-checking sources are cited and their impact on note credibility. The results indicate that successful community moderation significantly relies on professional fact-checking sources, particularly for high-stakes topics and misinformation linked to broader narratives.
Summary of strengths: The paper presents a timely and relevant research question. It explores fact-checking ecosystems and misinformation governance, contributing to the debate over how to combat misinformation at scale.
The authors use 1.5 million Community Notes, of which 664K are filtered for relevant analysis. This scale still lends robustness to the findings.
The manual annotations and LLM-assisted classification ensure data quality, which enhances trustworthiness.
It debunks the notion that community moderation can wholly replace expert fact-checking and highlights the risks of defunding professional fact-checkers, which is a powerful and highly relevant claim.

1. The study examines a timely and important topic related to the fact-checking community, especially in light of recent shifts by major social media platforms toward community-based moderation.
2. The authors conduct a large-scale quantitative and qualitative analysis of Community Notes from Twitter/X. The findings are well-structured, easy to follow, and relevant to stakeholders in fact-checking.
3. The use of regex-based URL classification, manual annotation, and LLM-assisted analysis enhances the robustness and reliability of the findings.
4. The paper is well-organized and clearly explains the data filtering, classification pipeline, and annotation process, making it highly replicable.
Summary of weaknesses: The authors lean heavily on LLMs for categorizing notes and URLs, which, despite some manual verification, could potentially introduce unaccounted biases or inaccuracies. The validation process, while present, is limited in scope, with only 400 samples being checked. This relatively small sample size may not be sufficient to capture the full range of potential errors or biases that could be present in the LLM-based classification system.

1. The presentation of the paper could be improved (see suggestions 1, 2, and 3).
2. Some additional aspects could be explored to strengthen the analysis (see suggestions 4 and 5).
3. The paper lacks technical depth in RQ2 (see suggestion 6).
Comments, suggestions and typos: It would be great to see if authors could provide guidelines on how to combine community notes with expert fact-checking to maximize effectiveness.

1. The legend in Figure 4 is incorrect. It uses red for "True" (indicating the presence of a fact-checking source) and green for "False" (indicating the absence of a fact-checking source). This contradicts the findings described in lines 231-234. Additionally, the paper uses a different legend style in Figures 3, 8, and 9, which may cause confusion.
2. The paper includes several figures, but some could be improved through comparative visualization. For example, Figures 6 and 7 could be combined into a single figure for direct comparison.
3. The resolution of Figure 10 should be improved to enhance readability.
4. The paper lacks a discussion about the traits of Community Notes contributors. This could be explored by extracting the tweet author’s profile information and analyzing their characteristics. It would be insightful to investigate who contributes to Community Notes (e.g., professional fact-checkers vs. general users) and their political leanings, potentially using LLM-based classification.
5. While the study focuses on "helpful" community notes, there is little discussion of why 89% of submitted notes fail to achieve "helpful" status. Analyzing the reasons for these rejections could provide deeper insights into the effectiveness of community moderation.
6. In Section 4.2, the human evaluation of the zero-shot text classification performance should be reported. Zero-shot models often underperform compared to fine-tuned models, and the ModernBERT-large-zeroshot-v1 model was primarily trained on formal English. It is highly recommended to fine-tune the model using annotated tweet-note pairs to improve accuracy.
Overall assessment scores: 4

4