Is Explanation the Cure? Misinformation Mitigation in the Short Term and Long Term
Authors: Yi-Li Hsu
Shih-Chieh Dai
Aiping Xiong
Lun-Wei Ku
Venue: EMNLP 2023 Findings (2023)
Keywords: Fake news debunking strategy
misinformation
Counterfactual Explanation
Natural Language Generation
Warning Tag
Longterm study

Abstract: With advancements in natural language processing (NLP) models, automatic explanation generation has been proposed to mitigate misinformation on social media platforms in addition to adding warning labels to identified fake news. While many researchers have focused on generating good explanations, how these explanations can really help humans combat fake news is under-explored. In this study, we compare the effectiveness of a warning label and the state-of- the-art counterfactual explanations generated by GPT-4 in debunking misinformation. In a two-wave, online human-subject study, participants (N = 215) were randomly assigned to a control group in which false contents are shown without any intervention, a warning tag group in which the false claims were labeled, or an explanation group in which the false contents were accompanied by GPT-4 generated explanations. Our results show that both interventions significantly decrease participants’ self-reported belief in fake claims in an equivalent manner for the short-term and long-term. We discuss the implications of our findings and directions for future NLP-based misinformation debunking strategies.

Is Explanation the Cure? Misinformation Mitigation in the Short Term and Long Term
Yi-Li Hsu Shih-Chieh Dai', Aiping Xiong', Lun-Wei Ku'
'Institute of Information Science, Academia Sinica
2 Department of Computer Science, National Tsing Hua University 'The University of Texas at Austin "The Pennsylvania State University yili.hsu@iis wws.saiiicaae.eeeeoeo tw
Abstract
checking approaches labor-intensive and time- consuming, often requiring domain expertise. Given advances in natural language processing (NLP), there is an ongoing shift towards NLP- based solutions such as fake news detection (Zhang and Ghorbani, 2020; Zhou and Zafarani, 2020; Shu et al., 2019) and generation of fact-checked, counterfactual explanations using natural language generation (NLG) models (Dai Decision-aid aid methods have also been proposed and deployed to warn users when a piece of fake news has been identified. Warning labels (or tags) have been widely adopted by social media platforms such as Facebook Twitter (X)2, and Tok to mitigate humans' belief in misinformation.
With advancements in natural language processing (NLP) models, automatic explanation generation has been proposed mitigate misinformation platforms in addition to adding warning labels to identified fake news. While many researchers have focused on generating good explanations, how these explanations can really help humans comhat fake news is exper-explored. In this study, we compare the effectiveness a warning label and state-of- the-art counterfactual explanations generated by GPT-4 in debunking misinformation. In a two-wave, online human-subjec subject study, participants =N= 215) were randomly assigned to a control group in which false contents are shown without any intervention, warning tag group in which the false claims were labeled, or an explanation group in which the accompanied by GPT-4
social media
al., 2022).
Tag- and explanation-based methods are both effective debunking fake news (Epstein et al., 2022; Moravec et al., 2020; Lutzke et al., 2019). However, few studies have compared tag-based and machine-generater explanations. Furthermore, studies on fake news debunking strategies have primarily focused short-term effects instead of long-term effects (Lpstein et al., 2022; Moravec al., 2020; Lutzke et al., 2019). Thus we propose a comprehensive evaluation of both the short- and long-term effectiveness of these debunking strategies, specifically the context of real-world news in the United States. For the explanation-based debunking strategy, we focus on counterfactual explanations, which have demonstrated greater effectiveness than summary-based explanations in mitigating the spread of misinformation (Dai et al., 2022). With the improvements large language models (LLM) from OpenAl, GPT-3 (Brown et al.,
false contents generated explanations. Our results show that both interventions significantly decrease participants' self-reported belief in fake claims in an equivalent manner for the short-term and long-term. We discuss the implications of our findings and directions for future NLP-based misinformation debunking strategies.
1 Introduction
Misinformation (or fake news) refers to false statements or fabricated information that is spread on social media (Wu al., 2019). The spread of misinformation has posed considerable threats to individuals and society, such as political elections (Allcott and Gentzkow, 2017: Grinberg et al., 2019; Lytvynenko and Craig, 2020), and public health ESwire-Thomperr et al., 2020), especially during the COVID-19 pandemic (Roozenbeek et al., 2020; Loomba et al., 2021). Because of the negative impact, considering effort has been devoted devising methods to mitigate misinformation, such as computation- based detection and prevention. Traditional fact-
/nttps://ww..accchokk. Faccboos.cco/joornliissprreer checkiii////eewwrrttinggg
hhttps://blog, twitter.comm
en_us/ Ltsp///tppicsscmmmm///222222
introddccing-our--crii------ ponpplll
hhttps://www..c tiktok. con/ communi ty
wwweeittt-gggginnneeeeeee guidelines/
en/

---

2020) already generates explanations acceptable humans (Wiegreffe et al., 2021). Here we employ the GPT-4 model, the latest state-of-the-ar LLM from OpenAI, to generate all counterfactual explanations (OpenAT, 2023).
the persistence of misbelief in misinformation, Lo et al. (2022) proposed an implicit approach to deliver fake news verification such that fake- news readers can continuously access verified news articles about fake-news events without explicit correction. Compared to fact-checker warning label condition, the results of such implicit intervention showed that participants maintained or slightly increased their sensitivity in differentiating real and fake news in the long term. Such results shed light on the importance of assisting users in distinguishing false information from factual contents for mitigating misinformation.
To investigate the effectiveness of debunking strategies, we focus their impact over varied time frames. We aim study the following research questions:
1. How are the effectiveness of tag-based and explanation-based interventions compare to conditions with no interventions the short term and long term from readers' aspect?
2. Are model-generated explanations really more effective than warning tags in both time frames?
2.2 Increasing Focus Fact-Checked Explanations
Researchers have adopted NLP techniques to provide explanations for detected misinformation, for instance by highlighting biased statements (Baly et al., 2018; Horne et al., 2019). Most current explanation-bad frame the task of misinformation mitigation as text summarization. Specificallyy such methods locate appropriate fact-checked evidence which is then summarized as an explanation (Atanasova et al., 2020; Kotonya and Toni, 2020). However, because the nature of summarization is to generate paragraph that best represents the meaning of all documents of evidence, this could fail to explain why piece of information is false.
2 Related Works
2.1 Effectiveness of Misinformation Warning Labels
methods
When piece of fake news is detected by the system (Horne and Adali, 2017) reported by online users (Keith, 2021; Henry, 2019), warning label is attached to the fake news. Recent studies have examined various aspects misinformation warning labels, including label wording (Clayton et al., 2020; Kaiser et al., 2021), warning icons (Kaiser et al., 2021; Moravec et al., 2020), and labels provided by fact-checkers (Pennycook et al., 2018), algorithms (Seo et al., 2019), or online community users (Jia al., 2022: Yaqub al.. 2020). The results collectively show that specific warning labels (e.g., "rated false") are more effective than general warning labels (e.g.. "disputed"). Also, misinformation warning labels attributed to third-party fact-checkers, algorithms, or online users can produce similar effects on reducing participants' beliefs.
Dai et al. (2022) proposed a framework to generate fact-checked counterfactual explanations (Byrne, 2016), the idea of which is to construct another instance showing minimal changes on evidence that result in different fact- checking prediction by models (Mittelstadt et al., 2019). Dai et al. also conducted online human- subject experiments and obtained results showing that the counterfactual explanations outperformed summarization-biin explanations. Such initial efforts yield a preliminary understanding of fact- checked counterfactual explanations in mitigating misinformation. To the best of our knowledge, the long-term effect of counterfactual explanations on mitigating humans' belief in misinformation has not been examined.
Despite the significant progresses, most previous studies have focused on examining the effect of misinformation warning labels immediately (i.e., a short-term effect). The continued-influence affect (Anderson et al., 1980; Lewandowsky et al., 2012)-th persistence of misbelief in misinformation even after a warning - highlights the importance of evaluating long-term effect of misinformation warning labels. Few studies have addressed the long-term effect of warning labels, and results generally show that the effect of such labels decays over time (Pennycook et al., 2018; Seo et al., 2019; Grady et al., 2021). I). To mitigate
Experiment:
Warning Tag or Explanation?
As current automated metrics frequently show weak correlation with human evaluations of

---

Newa Claim:
News Claim:
of your taopayar money attting in account"
$1.5 billion at your taopayar money is sitting account"
purposss.
purpasss.
$1.5 billon that can be used for other purposes.
sitting account"
Declared Pake by 3rd party Fact-Checkerss
STOP
Learn why this is tako
STOP
loth an dE pritte chargro retal frrt inierat
Figure I: Example of fake claim across the three conditions: the Control condition (left panel), the Warning-Ti Tag (WT) condition (center panel), and the cannterfactull-xplaaaae (CF-E) condition (right panel).
explanation quality (Clinciu et al., 2021), the current study focuses on human evaluation. We conduct an online human-subject study using mixed design. Specifically, participants evaluate real and fake claims across different time frames (pre-test, post-test, and long-term test) in three between-subject conditions [Control, Warning Tag (WT), and Counterfactual Explanation (CF-E)].
errors. All news claims and generated CF-Es for fake news can be found in Tables 3 and 4.
A stop sign with warning texts are employed in the warning tag. Such an intervention method has been shown be the most effective warning tag social media (Moravec al., 2020). We replace the warning texts with model-generater counterfactual explanation. Figure center and right panels show the intervention interfaces.
3.1 Materials
We sample 24 news claims (half fake and half real) from PolitiFact.com". Each claim has a veracity label and evidence from professional fact-checkers. Half of the claims are pro-Democratic, and the other half are pro-Republican. We follow LIAR-PLUS (Alhindi et al., 2018) to collect the evidence for each news claim. The news claims selected through pilot study to balance intensity of partisanship, familiarity level, and perceive accuracy between pro-Democratic and pro-Republican claims. Explanations were generated using GPT-4 (OpenAl, 2023), with a prompt based on the state-of-the-arr counterfactual explanation form (Dai et al., 2022):
3.2 Procedure
After informed consent and brief introduction of the study, participants are randomly assigned one of the three conditions.
Pre-test Phase (Baseline) Participants in all conditions start with evaluating four fake claims and four real claims without any intervention. Participants indicate their familiarity with each claim using a five-point scale: Have you ever seen or heard about this claim? (1 Definitely not, 5 = Definitely yes) and perceived accuracy using seven-point scale (Sindermann et al., 2021): To the best of your knowledge, how accurate is the claim? (-3 Definitely not accurate, 3 Definitely accurate). Because confirmation bias has been found influence users' beliefs in news claims (Ling, 2020), we also collect the participants' confirmation bias on each claim (Moravec et al., 2020) by asking their perceived importance using a seven-point scale: Do you find the issue described in the article important? (1 Not important all, 7 Extremely important). Confirmation bias is measured by multiplying the perceived accuracy and importance ratings, creating a scale from 21 to 21. Statistic results are shown in Appendix A.
Claim: <input>, Evidence: <input> This is a false claim. Please generate short sentence of counterfactual explanation to the claim. The sentence structure should be "If we were to say... instead of... the claim would be correct".
We manually verify the accuracy of AI- generated explanations ensure they align with evidence from human experts and are free from
website primarily features news from the United States.
https://www.plltifaccccc
'The pilot study was conducted with 72 annotators located in the United States. Those in the pilot did not participate in the main study, ensuring no overlap.

---

Accuracy Rate
Flip Rate (Short-term/ Long-term)
Pre-test
Post-test/ Long-term
x
-
x
Overall
Fake Claims with intervention in reading environment
CF-E
41%
77% 69%
20% / 17%
22%
18%
5% / 4%
3% 14%
49% /43%
WT
40%
72% 66%
17% 17%
20%
17%
1% 3%
2% / 3%
40% 39%
Fake Claims without intervention in reading environment
Control
40%
38% / 38%
4% / 77%
8% / 8%
7% 8%
7% 10%
26% 32%
Real Claims without intervention in reading environment
CF-E 35%
44% / 40%
8% 8%
13% 7%
13% 7%
6% 6%
33% 31%
WT
33%
52% / 49%
13% 13%
13% 12%
7% 7%
3% 6%
36% 39%
Control
35%
37% / 34%
7% 5%
7% 7%
4% 6%
4% 4%
22% 21%
Table 1: Accuracy rate shows the proportion of participants' responses answered correctly to the accuracy of the news claims in the pre-test and post-test stages. The flip rate states the proportion of outcomes that were flipped from pre-test to post-test (short-term)ang and pre-test to long-term test (long-term ). X-> represents the proportion participants' answers incorrectly in the pre-test but correctly the post-test or long-term test. < represents the proportion of participants' answers correctly in the pre-test but incorrectly in the post-test long-tern test. represents the participants answer "Might or might not be accurate" in the pre-test.
Intervention Phase Participants in each condition read the same eight news claims as in the pre-test phase. While participants in the Control condition view the fake claims without any intervention, those in the intervention conditions read each fake claim with the corresponding warning tag explanations. Figure 1 shows the interfaces of each condition.
Long-term test Phase Finally, we conduct long-tern test to determine whether the debunking
strategies have a long-term effect on participants' perceived accuracy of news claims. A delay of one or two days has been used in psychology literature to access long-term memory performance (Frith et al., 2017; Chan et al., 2006; Nairee, 1992). Thus, 24 hours after the post-test, invite all participants back to inquire about their perceptions of the news claims' accuracy, using a of questions same as the post-test. We invite the unreturned participants again after a 36-hour wait.
Questionnaire We solicit participants' opinions on the fact-checked debunking strategies. We ask them about their perception of fake news debunking strategy using a five-point scale: In the previous module, do you any fake news debunking strategy? (1 Definitely not, 5 Definitely yes). Other multi-choice questions are about the helpfulness and the overall experience of the survey. We administer the questionnaire in this stage to prevent the debunking strategies presented in the intervention phase from rehearsing in participants' working memory at the post-test evaluation.
3.3 Recruitment
We recruited participants (N 270) on Prolific (Palan and Schitter, 2018). We required our participants to be located the United States, which is critical to ensure they have cultural and contextual understanding of the selected news claims and the counterfactual explanations. For each question, we required a minimum reading and selection time of 5 seconds before submission button shows up. We discarded annotations completed in an unreasonably short time, reported not encountering any fake news debunking strategy in the WT or CF-E groups, or without completing the long-term test. We included 215 participants' responses in the data analysis.
Post-test Phase Participants re-evaluate the same news claim as the previous phases. We again ask about their perceived accuracy of the news claims to determine whether the fake news debunking strategies has flipped their initial evaluation of each claim. We further ask the reasons of the perceived accuracy rating by a multi-choice question: Why did you choose the answer of the previous question? (1: I chose this answer base on my previous knowledge; 2: because I saw the tag/ explanation; 3: because searched online; 4: others).
3.4 Results
Results are shown in Table Our main findings are as follows:

---

Faet-Checked intervention strategies indeed help humans debunk fake claims immediately: As shown in Table 1, participants made more accurate veracity judgments for the fake claims after viewing the warning tags or explanations. The accuracy rate of fake news improved from 11% at pre-test phase to 77% at post-test phase for the CF-E group. The accuracy rate of the WT group showed 32% increase (pre-test:10% post-test: 72%). Yet, the accuracy rate of the fake claims for the Control group remained similar in both pre-test (40%) and post-test (38%). While chi-squared test suggests that interventions helped participants understand that a news claim is false (22) = 39.6, p < 0.001), the improvements were statistically indistinguishable between the two intervention groups (x11) 1,p = 0.65). Moreover, the flip rates of inaccurate or ambiguous ratings accurate ratings are around 40% in both intervention groups, indicating that the intervention methods change participants' perception the misinformation substantially in the short-term.
the warning details (Samuels, 1983; Kadayat and Eika, 2020). Future work should use comparative effectiveness studies to isolate the effects of the warning icon and the explanations. Additionally, we compared the short-term and long-term results of the two intervention conditions and found no significant differences (211) < 1, P = 0.99). These results imply that a longer delay beyond 48 hours such Seo al., 2019) could be considered in future work to further test the long-term effect.
week (Pennycook et al., 2018;
Potential Source Searching We analyzed reasons for participants choices during both the post- and long-term tests; the details of the question and options are described in Section 3.2. While more participants in the Control condition sought out additional online information than those in the two intervention conditions in the post-test, such pattern was reversed in the long-term test. In particular, participants in the intervention conditions showed a similar or slightly higher online searching rate but those in the Control condition reduced the search. Such results are interesting and imply that both interventions can potentially promote fact-checking and foster deeper engagement with news. See details in Appendix and T. 5.
Explanations are not more effective in the long- term: Our hypothesis is that the explanation- based intervention would be more effective in debunking fake news than the tag- based intervention in the long-term as the explanations provides more information concerning the reason for refuting the fake news. While the long-term effects of the intervention groups were evident (x/2) = 32, < 0.001, the effects were similar between the CF-E (a 28% increase from the pre-test the long-term test) and WT (a 26% increase from the pre-test to the long-term test) conditions ((i) < 1,
the participants
4 Conclusion
We demonstrate that both tag-based and explanation-based methods can effectively combat fake information in the short- and long- term. However, such findings suggest that an emphasis on generating improved explanations from the aspects of NLP is not sufficient to address the primary challenge in mitigating humans' belief in misinformation. To make the NLP-bassed debunking strategies more effective in the long-term and further reduce recurred misbeliefs, it is crucial to consider visual design choices boost readers' motivation engage with detailed yet accessible explanations, such as font size, text length, or even personalized explanations.
0.99). These results suggest that both intervention methods demonstrate short-term and long-term effectiveness.
We conjecture two possible reasons for the obtained null significant results in the long-term. On one hand, we placed a stop-sign warning in the explanation condition. Prior studies have shown that such design could make participants pay more attention to the icons but fail to notice the texts (Kaiser et al., 2021). The red color has been widely used in risk communication (Wogalter al., 1999), which could also make participants react upon the color quickly without taking the time read the explanations. One the other hand, the font size and the length of the explanations could have made participants less motivated to check
Limitations
Our study assesses tag-based and explanation- based strategies independently, not jointly. This decision was made to isolate the effects of cach method and avoid possible confounding influences.

---

However, warning labels and explanations could be used in conjunction to augment the effectiveness of misinformation debunking in a real-world setting. This potential for synergy between the methods could provide a more robust approach to combat fake news, future research could benefit from exploring this joint application.
Acknowledgements
This research is supported by National Science and Technology Council of Taiwan, under Grants no. 111-2221-E-011-011 and 1 1--233-FF-00--022
The works of Aiping Xiong were in part supported by NSF awards #1915801 and #2121097.
Indeed, another limitation of our research lies in its examination of tag-based debunking strategies. Various types of warning labels are employed in different contexts, each with its own design, wording, and positioning. These variables may significantly influence the effectiveness of warning label. In our study, examine specific type of warning label, and although our results provide important insights, they may be representative of the potential effectiveness of all types of warning labels.
References
Tariq Alhindi, Savvas Petridis, and Smaranda Muresan. 2018. Where is your evidence: Improving fact- checking by justification modeling. In Proceedings of the First Workshop on Fact Extraction and Verification (FEVER), pages 85-90.
Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31(2):211-22
Craig A Anderson, Mark Lepper, and Lee Ross. 1980. Perseverance of social theories: The role of explanation in the persistence of discredited information. Journal of Personality and Social Psychology, (96)):037--1443
Also, in our case, we only sampled explanations from those that had been examined having system error. However, the generation model might have errors and thus generate low-quality fact- checked explanations.
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020. Generating fact checking explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7352-7364, Online. Association for Computational Linguistics.
Finally, in real-life situations, the fake news detector may mistakenly label genuine news as fake, which wasn't factor we considered in our study. Instead, we relied the expertise of fact- checkers from ProliticFact.com to provide labels. However, if the detector were to flag real news as fake in practice, could lead to issues associated with the warning labels.
Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov. James Glass, and Preslav Nakov. 2018. Predicting factuality reporting and bias news media sources. arXiv preprint arXiv:1810.0176;
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Ethics Statement
Our study has been approved by the Institutional Review Board of the authors' institution. We obtained informed consent from each participant. We acknowledge that participants were inherently exposed to the risk of reading fake news. However, prior studies showed that misinformation studies did not significantly increase participants' long- term susceptibility to misinformation used in the experiments (Murphy et al., 2020). As our survey is a two-wave study which did not force participants to finish the whole study, participants could receive partial payment right after completing the short- term study. We also informed participants that they had the right to withdraw from the experiment whenever they desired. Participants were paid based on a rate of $8/ hour, which is above the federal minimum wage in the United States.
Ruth MJ Byrne 2016. Counterfactual thought. Annual Review of Psychology, 17:135-15
Jason CK Chan, Kathleen B McDermott, and Henry L Roediger III. 2006. Retrieval-inducee facilitation: initially nontested material can benefit from prior testing of related material. Journal of Experimental Psychology: General, 135(4):53--571
Katherine Clayton, Spencer Blair, Jonathan A Busam, Samuel Forstner, John Glance, Guy Green, Anna Kawata, Akhila Kovvuri, Jonathan Martin, Evan Morgan, et al. 2020. Real solutions for fake news?

---

measuring the effectiveness of general warnings and fact-check tags in reducing belief in false stories on social media. Political Behavior, (073))00731095
Bam Bahadur Kadayat and Evelyn Eika. 2020. Impact of sentence length on the readability of web for screen reader users. In Universal Access in Human-Compan Computer Interaction. Design Approaches and Supporting Technologies: 14th International Conference, UAHCI 2020, Held Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19-24, 2020, Proceedings, Part 1, page 261-271, Berlin, Heidelberg. Springer- Verlag.
Miruna Clinciu, Arash Eshghi, and Helen Hastie. 2021. A study automatic metrics for the evaluation of natural language explanations. arXiv preprint urXiv:2103.085.
Shih-Chich Dai, Yi-Li Hsu, Aiping Xiong, and Lun- Wei Ku, 2022. Ask to know more: Generating counterfactual explanations for fake claims. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, page 2800-2810, New York, NY, USA. Association for Computing Machinery.
Ben Kaiser, Jerry Wei, Eli Lucherini, Kevin Lee. Nathan Matias, and Jonathan Mayer. 2021. Adapting security warnings to counter online disinformation. In USENIX Security Symposium, pages 1163-1180.
Ziv Epstein. Nicolo Foppiani, Sophic Hilgard, Sanjana Sharma, Elena Glassman, and David Rand. 2022. Do explanations increase the effectiveness ai-crowd generated fake news warnings? In Proceedings of the International AAAI Conference Web and Social Media, volume 16, pages 183-193.
Coleman Keith. 2021. Introducing birdwatch, community- based approach misinformation. [Online; accessed: 2023-9-16).
Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7740-7754, Online, Association for Computational Linguisties.
Emily Frith, Eveleen Sng, and Paul D Loprinzi. 2017. Randomized controlled trial evaluating the temporal effects of high-intensity exercise on learning, short-term and long-term memory. and prospective memory. European Journal of Neuroscience, 46(10):2557-25
Stephan Lewandowsky, Ullrich KH Ecker, Colleen Seifert, Norbert Schwarz, and John Cook. 2012. Misinformation and its correction: Continued influence and successful debiasing. Psychological Science in the Public Interest, 13(3):106-1311
Rebecca Hofstein Grady, Peter Ditto, and Elizaheth F Loftus. 2021. Nevertheless, partisanship persisted: fake news warnings help briefly, but bias returns with time. Cognitive Research: Principles and Implications, 6(1):1-16.
Rich Ling. 2020. Confirmation bias in the era of mobile news consumption: the social and psychological dimensions. Digital Journalism, 8(5):596-604
Nir Grinberg, Kenneth Joseph. Lisa Friedland, Briony wwire-Thompson and David Lazer. 2019. Fake news on twitter during the 2016 us presidential election. Science, 363(6425):374337..
Kuan-Chieh Lo, Shih-Chieh Dai, Aiping Xiong, Jing Jiang, and Lun-Wei Ku. 2022. Victor: An implicit approach to mitigate misinformation via continuous verification reading. In Proceedings of the ACM Web Conference 2022, pages 3511-3519.
Silverman Henry. 2019. Helping fact-checkers identify false claims faster. [Online: accessed: 2023-9-16),
Benjamin Horne and Sibel Adali. 2017. This just in: Fake news packs a lot in title, uses simpler, repetitive content in text body, more similar to satire than real news. In Proceedings of the International AAAI Conference on Web and Social Media, volume II, pages 759-766.
Sahil Loomba, Alexandre de Figueiredo. Simon Piatek, Kristen de Graaf, and Heidi J Larson. 2021. Measuring the impact of covid-19 vaccine misinformation on vaccination intent in the uk and usa. Nature Human Behaviour, 5(3):337-348.
Lauren Lutzke, Caitlin Drummond, Paul Slovic, and Joseph Árvai. 2019. Priming critical thinking: Simple interventions limit the influence of fake news about climate change on facebook. Global Environmental Change, 58:101964.
Benjamin D Horne, Dorit Nevo, John O'Donovan, Jin- Hee Cho, and Sihel Adah. 2019. Rating reliability and bias in news articles: Does ai assistance help everyone? In Proceedings of the International AAAI Conference on Web and Social Media, volume 13, pages 247-256.
Jane Lytvynenko and Silverman Craig. 2020. Here's running list of false and misleading information about the election. [Online; accessed: 2023-10-16).
Chenyan Jia, Alexander Boltz, Angie Zhang, Anqing Chen, and Min Kyung Lee. 2022. Understanding effects of algorithmic vs. community label on perceived accuracy of hyper-partisan misinformation. Proceedings of the ACM on Human Human-Compute Computer Interaction, 6(CSCW2):1-2)
Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019. Explaining explanations in ai. In Proceedings of the Conference on Fairness, Accountability, and Trunsparency, pages 279-288.

---

Patricia Moravec, Antino Kim, and Alan Dennis. 2020. Appealing to sense and sensibility: System 1 and system 2 interventions for fake news social media. Information Systems Research, 31())9887 1006.
Michael S Wogalter, Dave DeJoy, and Kenneth R Laughery. 1999. Warmings and risk communication. CRC Press.
Liang Wu, Fred Morstatter, Kathleen M Carley, and Huan Liu. 2019. Misinformation in social media: definition, manipulation, and detection. ACM SIGKDD Explorations Newsletter, 21(2):80-90.
Gillian Murphy, Elizabeth Loftus, Rebecca Hofstein Grady, Linda Levine, and Ciara M Greene. 2020. Fool me twice: How effective is debriefing in false memory studies? Memory, 28(7):938-949.
Waheeb Yaqub, Otari Kakhidze, Morgan L Brockman, Nasir Memon, and Sameer Patil. 2020. Effects of credibility indicators on social media news sharing intent. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1- 14.
James S Naime, 1992. The loss of positional certainty in long-term memory. Psychological Science, 3(3):199- 202.
OpenAl. 2023. Gpt-4 technical report.
Xichen Zhang and Ali A Ghorbani. 2020. An overview of online fake news: Characterization, detection, and discussion. Information Processing & Management. 10(2):
Stefan Palan and Christian Schitter. 2018. Prolific. ac ac-a subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17:22 27.
Gordon Pennycook, Tyrone D Cannon, and David G Rand. 2018. Prior exposure increases perceived accuracy of fake news. Journal of Experimental Psychology: General, 147(12):1865-1)) 18800
Xinyi Zhou and Reza Zafarani. 2020. A survey of fake news: Fundamental theories, detection methods, and opportunities. ACM Computing Surveys (CSUR), 53(5):1-40.
Jon Roozenheek, Claudia R Schneider, Sarah Dryhurst, John Kerr, Alexandra LJ Freeman, Gabriel Recchia, Anne Marthe Van Der Bles, and Sander Van Der Linden. 2020. Susceptibility misinformation about covid-19 around the world. Royal Society 199. 199.
Open Science, 7( 7110):2011
S Jav Samuels. 1983. A cognitive approach to factors influencing reading comprehension. The Journal of Educational Research, 26(5):261-266.
Haeseung Seo, Aiping Xiong, and Dongwon Lee. 2019.
Trust it or not: Effects of machine-learning learning warnings in helping individuals mitigate misinformation. In Proceedings of the 10th ACM Conference Web Science, pages 265-274.
Kai Shu, Suhang Wang, and Huan Liu. 2019. Beyond contents: The role of social context for fake news detection. Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 312-320.
Cornelia Sindermann, Helena Sophia Schmitt, Dmitri Rozgonjuk, Jon D Elhai, and Christian Montag. 2021. The evaluation of fake and true news: on the role of intelligence, personality, interpersonal trust, ideological attitudes, and news consumption. Heliyon, 7(3)=e06533.
Briony Swire-Thooe Thorr-Thoompoo, David Lazer, et al. 2020. Public health and online misinformation: challenges and recommendations. Annual Review of Public
Health, 41(1):433-44
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2021. Reframing human-ai collaboration for generating free-text explanations. arXiv preprint 12..v:2112.08674

---

A
Demographic of Participants
Item
Options
Survey
Female
37.02%
82
Male
62.9%
131
Sex
Other
0%
18-24
7.63%
25-34
29.77%
63
Age
35-44
19 19.46%
46
45-54
18.70%
38
Over 55
23.28%
52
Black
16.6%
White
69.99 99.9%
151
Ethnicity
Asian
5.55%
More than one race
5.09%
11
Other
2.77%
Table 2: Demographic information of valid participants in our survey.
The ratio of familiar claims are 0.25 for fake news (x 1, 0.994) and 0.2 for real news (X(2) < 1, p- 0.991) respectively with no significant difference across all condition groups. The ratio of perceived important claims are 0.59 for fake news (22) < 1,
0.999) and 0.51 for real news (x22 < 1, P = 0.999) with no significant difference across all condition groups respectively either. The average confirmation bias are -1.31 (x, 1, p 0.956) for fake news and -1.46 (x22) 1, 0.944) for real news with no significant difference across all condition groups.
B Participants' Reason of Choice
For fake claims with interventions in the reading environment, the CF-E and WT groups show different distributions in their reliance on previous knowledge and online searching, as depicted in Table 5. The control group heavily depends on online searching, particularly with real claims, similar to CF-E and WT groups for real claims. Only a negligible proportion of participants reported searching online for extra information during both short-term and long-term tests in both intervention groups. We consider this as a positive impact if participants searched online for external information, our intervention's s goal isn't just misinformation correction but also behavioral spanng--peciffcallll... fact-checking consumed news. Online searching indicates that the interventions motivate deeper news engagement by encouraging further fact-seeking.

---

Fake News Claim
Model-generated Counterfactual Explanation
Sen. Ron Johnson again says Social Security is Ponzi scheme
If we were say Social Security operates similarly a Ponzi scheme in terms of financial structure but with key differences that ensure benefit payments and government obligations, the claim would be correct.
2 The average tax for billionaires is about 3%, which is a lower tax than a schoolteacher or a firefighter.
we were to say the average tax billionaires is about 16%, which is higher than the effective tax rates of many teachers and firefighters falling within the $50,000 to $100,000 income range, the claim would be correct.
3 The largest contributor to the debt ceiling, or to our deficit, has been the Trump tax cuts.
If we were to say the Trump tax cuts have contrihuted to the debt ceiling and deficit, but other bills such as the 2019 spending bill, the CARES Act, and the American Rescue Plan, which received Democratic support, have added nearly as much or more to the debt, the claim would be correct.
4 $1.5 billion of your taxpayer money is sitting account that can he used for other purposes.
If we were say lowa has $1.5 billion in reserve funds that could potentially allocated for state programs through existing laws, instead of claiming that the govemor has the unilateral power to use it, then the claim would be correct.
legislative uppropriation or changes
5 New York City is wasting taxpayer money on billboards Florida.
If we were to say 'New York City is benefiting from pro bono creative content and donated billhoard space Florida' instead of 'New York City is wasting taxpayer money on billboards in Florida", the claim would be correct.
6 The average federal income tax paid by the richest Americans is 8%.... If you're 're cop. teacher. firefighter, union worker, you probably pay two to three times that.
If we were say the average federal income tax paid by the richest Americans is 20% of their income, while many households earning between $50,000 and $100,000 year pay effective tax rates between 0% and 15%, the claim would be correct, as reffects the actual tax rates under the current tax code.
Both murderers in Texas were only illegals, but gang members and were most likely tied MS-13.
If we were say 'Both murderers in Texas were U.S. citizens, with one having extensive criminal history, but no confirmed gang aftiliations' instead of 'Both murderers in Texas were not only illegals, gang members and most likely tied MS-13, the claim would be correct.
8 Only ahout 20% of people who apply for asylum qualify for it.
If we were to say that only about 30% of people who apply for asylum qualify for it, based on the approval rate since 2000 according to data from Syracuse U University, instead of 20%, the claim would be correct.
9 As president, Donald Trump "marshaled the full power of government to stop deadly drugs. opioids, and fentanyl from coming into our country. As result, drug overdose deaths declined nationwide for the first time in nearly 30 years.
If we were to say that Donald Trump's efforts to combat drug overdoses were effective in reducing overdose deaths from 2017 to 2018, but failed to sustain progress in the following years. including record increase 2020, the claim would be correct.
10 Says J.D. Vance said, "People who voted for Trump voted for him for racist reasons.
If we were to say 'J.D. Vance acknowledged that some people may have voted for Trump for racist reasons, but argued that most people voted for his policies on jobs instcad of "People who voted for Trump voted for him for racist reasons," the claim would be correct.
Biden and Democrats have dismantled border security.
If we were to say the Biden administration has maintained comparable budgets for border security, enforced border laws and policies, and utilized funds to repair barriers and close gaps, but halted the construction of additional miles of barriers, the claim would be correct.
12 Biden said there was "28% increase in children to the border in my administration" and "31%... in 2019." 9." The increase in migration in January, February and March "happens every year.
If we were to say there was a 63% increase in children arriving at the border during the Biden administration, which is more than twice the percentage he mentioned, and that the total number of encounters this year is likely to be the highest in 20 years, instead of focusing solely on the seasonal pattern of increased migration, the claim would be correct.
Table 3: All fake news claims and the corresponding model-generated counterfactual explanations

---

Real News Claim
We're the only major gas-producing state in the US that doesn't have
Florida has the second lowest tax burden per capita in the United States.
Since a new immigration program was implemented, number of Venezuelans trying to enter the illegally decreased "from about 1,100 per day less than 250 per day on average.
More fentanyl has crossed the border the last two months under Biden than 2019 under Trump.
WV families making less than S400K small businesses will NOT be targeted by the IRS.
The tax carve out (Ron) Johnson spearheaded overwhelmingly benefited the wealthiest, over small businesses.
7 Last year the IRS audited Americans carning less than $25,000 year at five times the rate of other groups.
8 Virginia tax receipts in just the last four years alone have grown 50%.
9 Title 42 and other Trump-era holdovers are forcing migrants into dangerous, overcrowded conditions in Mexico.
10 Just last year," Miami-Dade Public Schools "had over 14,000 new children, 10,000 of which came from four countries of Cuba, Nicaragua, Venezuela and Haiti.
Approximately 60,000 Canadians currently live undocumented in the USA.
Twice many children in Border Patrol custody under Biden than Trump peak 2019.
Table 4: All real news claims
Reason of Choice (Short-term/ Long-term)
Intervention Previous Knowledge Online Searching
Others
Fake Claims with intervention in reading environment
CF-E
66.5% 42.7%
30.4% 52.7%
1.5% 1.9%
1.5% 1.2%
WT
58.8% 37.7%
36.4% 59.1%
1.9% 1.6%
2.8% 1.6%
Fake Claims without intervention in reading environment
Control
12.9% 9.8%
80.5% 87.5%
3.1 % 0.8%
3.5% 0.4%
Real Claims without intervention in reading environment
CF-E
19.6% 14.6%
72.7% 80.4%
1.9% 1.2%
5.8% 2.3%
WT
32.5% 19.8%
60.4% 73.4%
1.9% 2.6%
4.5% 4.2%
Control
12.9% 9.8%
80.5% 86.3%
3.1% 0.8%
3.5% 1.6%
Table The table presents breakdown of participants' reasons for their choices the short and long term when exposed to fake and real news claims under different intervention methods. The reasons include relying previous knowledge, online searching, and other factors.


Reviews:
[{"review_id": "TER5oWs8uW", "reviewer": "Reviewer_whUc", "summary": "", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer whUc for emphasizing the societal relevance and implications of our research on debunking fake news and for noting the value of our findings regarding the use of GPT-4 generated counterfactual explanations. The following are our responses to the reject reasons of Reviewer whUc:\n\n**R1: Hallucination Effects of LLMs**\n\nWe acknowledged the potential for hallucination effects in LLMs. However, it's important to note that the primary objective of our study is not to assess the generative capabilities of the model, especially since GPT-4 has already established its performance in most of NLP tasks [1]. We aim to evaluate the effectiveness of model generated explanations and warning tags. To address concerns about model-generated inaccuracies, we'd like to clarify that we manually reviewed the explanations generated by the model, excluding any inaccuracies, ensuring our experiment's results are not affected by potential GPT-4 model errors. We apologize for the oversight in not detailing this process earlier in the submission and will include this clarification in the final manuscript.\n\n**R2: Human-subject Experiment**\n\nWe appreciate your feedback regarding the inclusion of more details about the human-subject experiment. We detailed these aspects in the Appendix because of the page limitation. In light of your feedback, we will move this information to the main body of the paper in the final version because we will be given one additional page in the proceedings. We will ensure that an additional content page in the proceedings provides a comprehensive overview of the human experiment, including the number of news claims used and other relevant information. \n\n**R3: Exposure of Correct Answers**\n\nThank you for highlighting the concern regarding the repeated use of the same set of news claims across various testing phases. However, our primary objective is to assess the effectiveness of different intervention methods, namely the explanation group, warning tag group, and control group. The structure of our study, which provides the correct answer during the intervention, is deliberately designed to measure shifts in individual stances or beliefs subsequent to that intervention.\n\n**R4: Contribution to the NLP Community**\n\nWe believe our study bridges the gap between the NLP community and the societal challenge of fake news. Our research seeks to highlight the potential misdirection in the current NLP focus. The results suggest that even the state-of-the-art GPT-4 model’s explanations may not win over warning tags and can not sufficiently address the misinformation challenge. While there's value in enhancing explanation quality, our findings indicate that the primary challenge in combating misinformation might lie elsewhere. In our conclusions, we highlighted the need to raise users' motivation to engage with comprehensive yet digestible explanations. For NLP researchers, this might involve personalized explanations driven by Human-AI collaborative conversational systems, as mentioned in [2] and [3]. Such systems could dynamically adjust explanations to individual user preferences, cognitive styles, or prior knowledge, thereby making the information more relatable and actionable for the end-user. In the revised paper, we'll emphasize this perspective in the final version further to provide greater contribution to the NLP community.\n\nWe hope these responses address the reviewer's concerns.\n\n**References**\n\n[1] OpenAI, GPT-4 Technical Report, 2023, https://arxiv.org/abs/2303.08774\n\n[2] Elliot Schumacher, Geoffrey Tso, Anitha Kannan, DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents, Varun Nair, 2023,  https://arxiv.org/abs/2303.17071\n\n[3] Monika Westphal, Michael Vössing, Gerhard Satzger, Galit B. Yom-Tov, Anat Rafaeli, Decision control and explanations in human-AI collaboration: Improving user perceptions and compliance,\nComputers in Human Behavior, 2023, https://doi.org/10.1016/j.chb.2023.107714.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1. The issue of debunking fake news is of utmost importance for society, and this study aims to measure the effectiveness of various debunking strategies, which could bring significant implications.\n\nA2. Notably, some findings, such as the effectiveness of counterfactual explanations generated by the GPT-4 model in reducing belief in fake information, are noteworthy and deserve attention.", "reasons_to_reject": "R1. While it is a promising attempt to use LLMs for generating counterfactual explanations for fake news, it appears that the correctness of these explanations has not been adequately evaluated. Since LLMs are known to have hallucination effects, there is uncertainty about the reliability of the generated explanations.\n\nR2. In spite of the limited space in a short paper, more details about the human experiment should be included in the main text. For instance, the number of news claims used for the experiments and other relevant information should be provided. \n\nR3. It seems that the experiments use the same set of news claims for pre-, intervention, post-, and long-term tests for each group. Thus, there's a possibility that the results are an outcome of participants’ exposure to the correct answer in the intervention. And that makes the validity of measuring the effectiveness of debunking strategies doubtful. I wonder whether there's a more subtle way to measure such behavioral change. \n\nR4. While the study is important, its contribution to the NLP community is unclear.", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "This research aims to combat fake news by addressing the effectiveness of debunking strategies with the assistance of NLP methods. In particular, the study focuses on tag- and counterfactual-based explanations and proposes a comprehensive evaluation of them over short and long-term periods. The research employs the GPT-4 model, a state-of-the-art LLM, to generate counterfactual explanations. Based on a human experiment involving 24 news claims and 215 participants, the authors claim that those debunking interventions help readers understand that the news claims are false in both the short-term and long-term, but to a lesser extent in the long-term.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vwR73z4Z8U", "reviewer": "Reviewer_fdhr", "summary": "", "questions": "1. Did you check whether your subjects looked for the sources of news between the two tests? If so, how would that affect the results?\n2. How did you evaluate the GPT-4 explanations generated and presented to the participants?\n3. How did you choose the news for evaluation? Were they related to your subjects' demographic?\n4, Did you have any precautions to prevent the subjects from labeling the news from memory?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer fdhr for recognizing the significance and depth of our research on debunking fake news, as well as its potential to foster further exploration and offer tangible solutions to social media platforms. Your feedback is deeply appreciated. The following are our responses to the reject reasons of Reviewer fdhr:\n\n**R1, Q2: Quality of Generated Explanations**\n\nThank you for pointing out the potential challenge of AI-generated explanations. We'd like to clarify that our study exclusively uses AI-generated explanations that have been manually verified for accuracy by us. This verification process is based on checking the explanations are aligned with the evidence written by human experts from PolitiFact.com, ensuring not only factual accuracy but also grammatical integrity and logical coherence. While we acknowledge the concerns surrounding the accuracy of AI outputs, it is not the primary focus of our paper. We made deliberate efforts to ensure the reliability and accuracy of the data presented.\n\n**R2, Q3: Details of Selected Claims**\n\nWe would like to highlight that the nature of the news items annotated by the subjects is included in Appendix 1. We followed the LIAR-PLUS dataset methodology to collect evidence for the 24 news claims from PolitiFact.com. To avoid extreme political leanings, familiarity, and perceived accuracy, the news claims were carefully selected through a pilot study conducted by additional 72 U.S. Prolific workers. Specifically, our study focuses on issues related to tax and immigrants issues in the US. We appreciate your suggestion and will ensure to include this information prominently in the main content of the final version. Additionally, the full list of the 24 news claims will be made available in the appendix. We hope these changes will address your concerns and enhance the clarity and transparency of our work.\n\n**R3: Experimental Setup Description**\n\nThank you for pointing out the potential discrepancy between the described experimental setup and Figure 3. We would like to clarify that Figure 3 showcases one of the samples presented to the subjects in our experiment. We will carefully review and adjust the description of the experimental setup to reduce any confusion. \n\n**R4: Influence of Font Size and Sentence Length**\n\nThank you for your thoughtful feedback on Line 278. We agree that our statement could benefit from more robust substantiation. Existing studies have highlighted the influence of sentence length on both comprehension and workload, particularly in the context of screen readers [2]. Additional research also points out that text size and sentence length can affect reading speed, comprehension, and even motivation to read [3]. This aligns with our hypothesis in Line 278 that readability factors—like sentence length and font size—could influence the effectiveness of explanations. We will include this information in the revised manuscript for a more comprehensive discussion.\n\n**Q1: Potential Source Exploration**\n\nWe appreciate your advice and we will include the following additional information in the revised paper. We specifically asked participants why they chose their respective answers to understand their rationale better. The question posed was, \"Why do you choose the answer to the previous question?\" and the options were based on previous knowledge, intervention effect (tag/explanation), online search, or others, as specified in Lines 631-636 of our manuscript.The below tables present the results of the question. \n\nOur data indicate that only a negligible proportion of participants reported independently searching online for extra information during both short-term and long-term tests across all groups, with percentages ranging from 0.8% to 3.1%. Importantly, we would consider this a positive impact of the intervention if participants searched online for external information. The reason is that our goal of intervention is not just misinformation correction but also behavioral change—specifically, checking the facts of consumed news information. Therefore, increased online searching could be seen as evidence that our interventions are motivating participants to engage more deeply with the news by seeking out additional facts.\n\n**Reason of Choice (Fake news)**\n\n|  | Intervention  (short-term/ long-term) | Previous Knowledge (short-term/ long-term) | Search Online (short-term/ long-term) | Others (short-term/ long-term) |\n| --- | --- | --- | --- | --- |\n| WT | 58.8%/ 37.7% | 36.4%/ 59.1% | 1.9%/ 1.6% | 2.8%/ 1.6% |\n| CF-E | 66.5%/ 42.7% | 30.4%/ 52.7% | 1.5%/ 1.9% | 1.5%/ 1.2% |\n| Control | 12.9%/ 9.8% | 80.5%/ 87.5% | 3.1%/ 0.8% | 3.5%/ 0.4% |\n\n**Reason of Choice (Real news)**\n\n|  | Intervention (short-term/ long-term) | Previous Knowledge (short-term/ long-term) | Search Online (short-term/ long-term) | Others (short-term/ long-term) |\n| --- | --- | --- | --- | --- |\n| WT | 32.5%/ 19.8% | 60.4%/ 73.4% | 1.9%/ 2.6% | 4.5%/ 4.2% |\n| CF-E | 19.6%/ 14.6% | 72.7%/ 80.4% | 1.9%/ 1.2% | 5.8%/ 2.3% |\n| Control | 12.9%/ 9.8% | 80.5%/ 86.3% | 3.1%/ 0.8% | 3.5%/ 1.6% |\n\n**Q4: Relying on Memory to Label**\n\nWe did not take specific measures to prevent subjects from relying on memory when labeling news claims. However, the self-reported familiarity (Experimental details are in Line 585) with the claims is relatively low. The proportions of participants who answered \"Definitely yes\" or \"Probably yes\" were 0.25 and 0.2 for fake and real news, respectively. Statistical tests showed no significant difference in this respect across all groups, including the CF-E, WT, and control groups (χ^2(2) < 1, p = 0.994 for fake news; χ^2(2) < 1, p = 0.991 for real news). Moreover, previous research [1] has indicated that familiarity with claims does not significantly impact the effectiveness of explanations. Therefore, we believe that the influence of memory on our study's results is likely minimal.\n\n**References**\n\n[1] Shih-Chieh Dai, Yi-Li Hsu, Aiping Xiong, and Lun-Wei Ku. 2022. Ask to Know More: Generating Counterfactual Explanations for Fake Claims. KDD '22 https://dl.acm.org/doi/abs/10.1145/3534678.3539205\n\n[2] Kadayat, Bam Bahadur, and Evelyn Eika. Impact of Sentence Length on the Readability of Web for Screen Reader Users HCII’20 https://link.springer.com/chapter/10.1007/978-3-030-49282-3_18\n\n[3] S. Jav Samuels, A Cognitive Approach to Factors Influencing Reading Comprehension, The Journal of Educational Research 1983 https://www.jstor.org/stable/27539984", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This study addresses the pressing issue of fake news on social media, which has significant real-world implications. Given the pervasiveness of misinformation, the effectiveness of various debunking approaches is a highly relevant area of research.\n2. This work brings a novelty by focusing on the short-term and long-term effects of debunking fake news. This adds a dimension of depth to the study, allowing for a more comprehensive understanding of the effects of debunking techniques.\n3. This research raises an important question regarding the effect of intervention methods. This fosters ongoing discourse and encourages further exploration in this field.\n4. While the study needs more analysis, it provides empirical evidence that warning tags and AI-generated explanations can significantly lower the immediate belief in fake news. This is important as it offers tangible solutions that social media platforms can consider implementing.", "reasons_to_reject": "1.  While the study's use of AI-generated explanations is innovative, there is a potential challenge when the generated counterfactuals may not be completely accurate. This could further propagate misunderstandings or misinformation.\n2. The paper could provide additional detail on the nature of the news items that were annotated by the subjects. Greater transparency in this area could help in understanding the full context and applicability of the findings.\n3. There seems to be a discrepancy between the described experimental setup and the interface depicted in Figure 3. Clarification or harmonization will help avoid potential confusion for readers. \n4. The hypothesis put forth in Line 278 could be better substantiated, either through explicit instructions to the subjects or through additional surveying. Addressing this point more thoroughly could further strengthen the evidence supporting the authors' claims.\n5. While the research is commendable for bringing attention to the effectiveness of different intervention methods, a more in-depth analysis of the subjects' decision-making process could further enhance the robustness of the supportive argument. This aspect is worth exploring more comprehensively to clarify the complex dynamics at play further.", "questions_to_authors": "1. Did you check whether your subjects looked for the sources of news between the two tests? If so, how would that affect the results?\n2. How did you evaluate the GPT-4 explanations generated and presented to the participants?\n3. How did you choose the news for evaluation? Were they related to your subjects' demographic?\n4, Did you have any precautions to prevent the subjects from labeling the news from memory?", "paper_topic_and_main_contributions": "This paper presents a study investigating the effectiveness of two approaches to debunking fake news on social media: warning tags and AI-generated explanations. The study was conducted in two waves, involving participants who were divided into a control group and two treatment groups. The control group was exposed to false content without any intervention, while one treatment group received false content with warning tags, and the other had false news accompanied by explanations generated by the GPT-4 model.\n\nThe study found that both the warning tags and the AI-generated explanations significantly decreased immediate belief in fake news, with no notable difference between the two approaches. The two methods also had long-term impacts on participants' self-reported belief in fake news. The authors suggest that future research could explore the combined use of warning tags and AI-generated explanations and take into consideration various designs of warning tags.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z80UMF0Bo0", "reviewer": "Reviewer_1tRC", "summary": "", "questions": "1. have you tried to vary the style/length of the generated explanations to see their influence?\n\n2. how do you control the quality of the human study?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1tRC for recognizing the practicalness and showing interest in our research. Your feedback is deeply appreciated. The following are our responses to the reject reasons of Reviewer 1tRC:\n\n**R1: Reason of Not Including Automatic Evaluations**\n\nWe would like to highlight that the primary objective of our study is not to assess the generative capabilities of the model, especially since GPT-4 has already established its performance in most of NLP tasks [2]. We aim to evaluate the effectiveness of model generated explanations and warning tags. As we mentioned in the Lines 190-192 of our manuscript, previous study has suggested that existing automatic metrics correlate poorly with human judgments of quality[1, 4, 5]. Given this, we believe that relying on human’s responses provides a more accurate measure of the effectiveness of explanation and warning tag interventions.\n\n**R2, Q1: Details of Experiment**\n\n**Quality of the Generated Explanations:** GPT-4 has already established its performance in most of NLP tasks [2]. Furthermore, we manually reviewed the explanations generated by the model, excluding any inaccuracies, ensuring our experiment's results are not affected by potential GPT-4 model errors. Examples of model generated explanations can be referred to Table 3 in the paper. \n\n**Detailed Prompt:** We include the GPT-4 prompt for explanation generation in Lines 229-233. \n\n**Length and Style of the Explanations:** We utilize single-sentence counterfactual explanations, proven to be more effective than both abstractive and extractive summary-based explanations for debunking misinformation, as elaborated in Lines 78-81 of our manuscript and further explored in [3]. Rather than restricting sentence length, we confined the structure to counterfactual explanations, giving GPT-4 the flexibility to generate thorough and coherent sentences, which on average consist of 45 words. This supplemental detail will be included in our revised paper for enhanced clarity.\n\n**R3: Reason of Not Including Inner Agreements**\n\nOur study's core focus is on exploring individual differences in opinions, not on achieving uniform labels or judgments on a given task. Also, inner agreements are mainly used for qualitative responses, which is not the focus of our study. Therefore, inner agreements are not applicable or relevant for evaluating our study's quality. We'll clarify this point in the revised manuscript to address any concerns.\n\n**R4: Difference of Information Volume in Warning Tags and Explanations**\n\nIn fact, highlighting the potential unwillingness of individuals to engage with explanations which include higher information volume is one of the issues our study aims to raise. By comparing more warning tags with longer, more detailed explanations, our study seeks to compare the effectiveness in the context of combating misinformation. The observed results, which indicate no significant difference in effectiveness between the warning tags and explanations, provide valuable insights for designing debunking misinformation strategies. We will clarify this aspect of our research focus in the revised manuscript to make our intentions and findings more transparent.\n\n**Q2: Quality Control of Human Study**\n\nTo ensure the quality of our human study, we implemented several criteria as outlined in the paper (Lines 194-198). If an annotator completed the task in an unreasonably short timeframe, indicated they did not come across any fake news debunking strategy in the warning tag or explanation groups, or failed to complete the long-term test, we excluded their annotations. Additionally, any annotations from participants who reported encountering a fake news debunking strategy within the control group are also discarded. These measures helped us maintain a high standard of data integrity and consistency throughout the study. \n\n**References**\n\n[1] Miruna-Adriana Clinciu, Arash Eshghi, Helen Hastie, A Study of Automatic Metrics for the Evaluation of Natural Language  Explanations EACL 2021, https://aclanthology.org/2021.eacl-main.202\n\n[2] OpenAI, GPT-4 Technical Report, 2023, https://arxiv.org/abs/2303.08774\n\n[3] Shih-Chieh Dai, Yi-Li Hsu, Aiping Xiong, and Lun-Wei Ku. Ask to Know More: Generating Counterfactual Explanations for Fake Claims. KDD '22 https://dl.acm.org/doi/abs/10.1145/3534678.3539205\n\n[4] Maxime Peyrard, Studying Summarization Evaluation Metrics in the Appropriate Scoring Range , ACL 2019, https://aclanthology.org/P19-1502\n\n[5] Daniel Deutsch, Rotem Dror, and Dan Roth, Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics, NAACL 2021, https://aclanthology.org/2022.naacl-main.442.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research question is interesting and has practical applications in real life.\n\n2. The paper proposes a human study to compare the two debunking options.", "reasons_to_reject": "1. The paper lacks results and analysis. The only results included in the paper are from participants’ responses in Table 1. It would be much better to include some automatic evaluation metrics to justify the findings, or the results would be subjective and weak.\n\n2. There are several factors that could affect the experiments, but details are missing. For example, the quality of the generated explanations, detailed prompts to use, length/style of the explanations, etc.\n\n3. I don't see how the authors justify the quality of the human study. Indicators like inner agreement should be included to make the study more convincing.\n\n4. The information volume within the explanations and the warning tag are very different, so it is not even a fair comparison. e.g. people may not be willing to read the whole lengthy explanation.\n\n5. I have read the rebuttal and don't think the human study here is designed carefully and thoroughly. Besides, this seems to fit more for a psychology conference instead of an NLP conference. Actually, the paper is nothing related to NLP except using GPT-4 for some explanation generation.", "questions_to_authors": "1. have you tried to vary the style/length of the generated explanations to see their influence?\n\n2. how do you control the quality of the human study?", "paper_topic_and_main_contributions": "The paper compares the effectiveness of fake news warning labels and the counterfactual explanations generated by GPT-4 as debunking\nmethods with a human study. According to a study involving 215 participants and 24 news claims, the authors assert that interventions aimed at debunking help readers recognize the falseness of news claims in both the short-term and long-term, although to a somewhat lesser degree in the long-term.", "excitement": 2, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following discussions with authors and among themselves, the reviewers overall found the paper sound.

The most prominent critiques with a consensus reached have to do with the lack of evaluation of the generated explanations, limited analyses of the main results, limited robustness tests, further necessary clarifications around the experimental design, and a narrow outline of contributions to the field.
Decision: Accept-Findings