Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Authors: Jason S Lucas
Adaku Uchendu
Michiharu Yamashita
Jooyoung Lee
Shaurya Rohatgi
Dongwon Lee
Venue: EMNLP 2023 Main (2023)
Keywords: LLM
Prompt Engineering
Disinformation Detection
Natural Language Inference
Semantic Reasoning
In-context Learning
TL;DR: Using LLMs to detect misinformation.

Abstract: Recent ubiquity and disruptive impacts of large language models (LLMs) have raised concerns about their potential to be misused (*.i.e, generating large-scale harmful and misleading content*). To combat this emerging risk of LLMs, we propose a novel "***Fighting Fire with Fire***" (F3) strategy that harnesses modern LLMs' generative and emergent reasoning capabilities to counter human-written and LLM-generated disinformation. First, we leverage GPT-3.5-turbo to synthesize authentic and deceptive LLM-generated content through paraphrase-based and perturbation-based prefix-style prompts, respectively. Second, we apply zero-shot in-context semantic reasoning techniques with cloze-style prompts to discern genuine from deceptive posts and news articles. In our extensive experiments, we observe GPT-3.5-turbo's zero-shot superiority for both in-distribution and out-of-distribution datasets, where GPT-3.5-turbo consistently achieved accuracy at 68-72%, unlike the decline observed in previous customized and fine-tuned disinformation detectors.   Our codebase and dataset are available at https://github.com/mickeymst/F3.

Fighting Fire with Fire:
The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Jason Lucas
Adaku Uchendu Michiharu Yamashita'
Jooyoung Lee'
Shaurya Rohatgi' Dongwon Lee"
The Pennsylvania State University, University Park, PA, USA {js15710, michiharu, jf15838, szr207, déngw edu MIT Lincoln Laboratory, Lexington, MA, USA Blakkuuchendu@l1. mit.
Abstract
To mitigate this muddle of disinformation by LLMs, in this work, we ask pivotal question: if LLMs can generate disinformation (via malicious use or hallucination), can they also detect their own, as well as ing literature provides limited perspectives on the potential use of the latest commercial state-of-the- art (SOTA) LLMs such as GPT-4 (OpenAI, 2023) and LLaMA (Touvron et al., 2023) to address disinformation. Particularly, topics including: (1) leveraging prompt-engineering to bypass LLMs' protective guard-rails; (2) utilizing the emergent zero-shot capabilities of modern LLMs (with 10B+ parameters) for disinformation generation and de- tection; (3) manipulating human-written real news to fabricate LLM-generated real and fake narra- tives to simulate real-world disinformation risks; and (4) assessing and addressing LLMs' inherent hallucinations in the disinformation domain.
Recent ubiquity and disruptive impacts large language models (LLMs) have raised concerns about their potential to be misused (i.e,, gener- ating large-scale harmful and misleading con- tent). To combat this emerging risk of LLMs, we propose a novel "Fighting Fire with Fire" (F3) strategy that harnesses modern LLMs' gen- erative and emergent reasoning capabilities to counter luman--witted and L-M-genrrated dis- information. First, we leverage GPT-3.5-turbo to synthesize authentic and deceptive LLM- generated content through paraphrase- and perturbation-basel prefix-style prompts, respectively. Second, we apply zero-shot in- context semantic reasoning techniques with cloze-style prompts to discern genuine from deceptive posts & news articles. In our exten- sive experiments, we observe GPT-3.5-turoo' zero-shot superiority for both in-distribution and aitoo-ddistribution datasets, where GPT- 3.5-turbo consistently achieved accuracy at 68- 72%, unlike the decline observed in previous customized and fine-tuned disinformation de- tectors. Our codebase and dataset are available at httts:: /gi thub. com/mi /mi ckeyms 3/F3.
disinformation? Emerg-
To investigate these inquiries, we formulate two research questions (RQs) as follows:
RQ1: Can LLMs be exploited to efficiently gen- erate disinformation using prompt engineering?, where (1) attempt to override GPT-3.5's align- ment in fabricating real news, and (2) measure the frequency and remove GPT-3.5 hallucinated mis- alignments.
1 Introduction
While recently published LLMs have demonstrated outstanding performances in diverse tasks such as human dialogue, natural language understanding (NLU), and natural language generation (NLG), they can also be maliciously used to generate highly realistic but hostile content even with protective guardrails, especially disinformation (Spitale et al., 2023; Sadasivan et al., 2023; De Angelis et al., 2023; Kojima et al., 2022). Moreover, LLMs can produce persuasive texts that are not casily dis- tinguishable from human-written ones (Uchendu al., 2021: Chakraborty et al., 2023; Zhou al.. 2023), making humans more susceptible to the in- trinsic/extrinsic hallucination proclivities and thus introducing disinformation (Uchendu et al., 2023).
RQ2: How proficient are LLMs in detecting disinformation?, where we evaluate the capability to detect disinformation between (1) human and Al-authored, (2) self-generated and other LLM- generated, (3) social media posts and news articles, (4) in-distribution and out-of-distribution, and (5) zero-shot L.MSs and domain-related detectors.
Pretrained LLMs of our interest are GPT-3.5-
Turbo, LLaMA-2-Chat (Rozière et al., 2023),
Models
Size
Max Token
Source
GPT3.5-Turbo
175B
8,192
OpenAT
LLaMA-2-Chat
70B
4,096
Hugging Face
LLaMA-2-GPT4
70B
4,096
Hugging Face
Dolly-2
12B
4,096
Hugging Face
Palm
340B
8.192
Hugging Face
Table I: Summary L.L.s used.

---

Human-Date
Prefix Prompt
PURIFY HALLUCINATION
Cloze Prompt ZERO-SHOT DETECTION
OUPUT
PIP-GPT
Falca News Net
Imp Imparsonator As
Impersenator As Al detector
Sun lerggigincc chinliing
Instructor
Inabruckor
PAKE
Maal
Min
NJI
A-caf-cR
Tero Cal
Dulf Goll
Mall
CoAlD
Dah-Spp
Porophrose
Sembicolss
FIKE
Content
Bh3ch cures covi--19..
Surlight drinking....
Pro trained LLMG Customized SLMs Fine-tuned
SOTA Data
OPT-3.5-Turbo
Sampling
RQ1
RO2
Figure I:
Fighting Fire with Fire (F3) Framework for (A) Disinformation Generation (B) halluecinatoonpuurfficttion
[detection and removal] (C) In-context Semantic Zero-shot Detection
LLaMA-2-GPT4 (Touvron et al., 2023), Palm-2- text-bison (Anil et al., 2023), and Dolly-2 (Conover et al., 2023) (See Table for more details). To an- swer these RQs, we propose the Fighting Fire with Fire (F3) Framework. As shown in Fig. 1, we first use paraphrase and perturbation methods with prefix-style prompts create synthetic disinforma- tion from verified human- written real news (steps and 2). We then employ hallucination mitigation and validation strategies ensure our dataset re- mains grounded in factual sources. Specifically, the PURIFY (Prompt Unraveling and Removing Inter- face for Fabricated Hallucinations Yarns) method in step 3 incorporates metrics like AlignScore, Nat- ural Language Inference, Semantic Distance, and BERTScores to ensure data integrity and fidelity. Lastly, steps 4 and 5 implement cutting- edgge in- context, zero-shot semantic reasoning such Auto- Chain of Thoughts for detecting disinformation.
zero-shot NLP via prompt engineering (Holtzman et al., 2019; Jason al., 2022). We survey two core prompting techniques that we use for our task. Prefix Prompts provide instructional context at the beginning of the prompt to guide LLMs' text generation (Kojima et al., 2022). Strategies like in-context learning and prompt tuning boost gener- ative performance many NLP tasks such as sum- marization, and translation (Radford al., 2019; Li and Liang, 2021; Dou al., 2020; Brown al., 2020). addition, paraphrasing (Krishna et al., 2023; Kovatchev, 2022) and perturbation (Chen et al., 2023) approaches are widely used in NLP tasks. We use both paraphrasing and perturbation with prefix prompts to synthetically generate disin- formation variations from human-written true news (Karpinska et al., 2022; Fomicheva and Specia, 2019). This leverages LLMs' generation while maintaining its connection to truth.
human-written
Our contributions include: (1) new prompting methods for synthetic disinformation generation; (2) hallucination synthetic disinformation purifica- tion framework; (3) novel prompting in-context se- mantic zero-shot detection strategies for human/Al disinformation; (4) comprehensive benchmark of SOTA detection models human/Al disinforma- tion dataset; and (5) dataset for disinformation re- search. This dual-perspective study cautions the risks of AI disinformation prolifcration while also providing promising techniques leveraging LLM capabilities for enhanced detection.
Cloze Prompts contain missing words for LLMs to fill in using context (Hambardzumyan et al., 2021) and are often used to assess LLMs' contex- tual prediction. This includes question answering to predict the correct missing word that logically completes given context (Gao al., 2020). Re- searchers have applied fixed-prompt tuning to cloze prompts (Lester et al., 2021: Schick and Schüitze, 2021). Prior work explored cloze prompt engi- neering (Hambardzumyan et al., 2021; Gao et al., 2020). We combine cloze prompts with SOTA rea- soning techniques such as Chain-of Chain-of-Thought (CoT) for zero-shot disinformation detection (Tang et al., 2023a), leveraging both approaches.
2 Related Work
2.1 Prompt-based Learning
2.2 Disinformation Detection
Latest large language models (LLMs) surpass pre- vious models in many downstream tasks, including
Earlier disinformation detectors use diverse ap- proaches such as neural, hierarchical, ensemble-

---

Prompt-Temppaae
C+R+O
RQ2 Disinformation Detection: For F3 text detec- tion, we employ a classifier F that takes a cloze- prompt Y c--R+1 as imput and outputs a label L, such that P(Ye-R+i) = (Figure 19).
Content
Impersonator
+
Instructor
Datasets
This section describes the human datasets that we used to generate and evaluate LLL-ggenrrted disin- formation. The data is stratified by veracity, content type, topic, and crouu-of-distribution to GPT-3.5's September 2021 training cutoff.
Figure 2: F3 prompt template (0) has three parame-
ters: (1) content (C') embeds data, (2) Impersonator (ft)
relative
establishes context, guides LLMs' generation and detec- tion, and overrides alignment-tuning. (3) Instructor (/)
provides directives to guide LLM.
4.1 Hlmann-ritted Real and Fake News Data
based, and decentralized techniques (Aslam al., 2021; Upadhayay and Behzadan, 2022: Jayakody al., 2022; Ali et al., 2022; Cui et al., 2020). Some recent key examples include dEFEND (Shu et al., 2019) and FANG (Nguyen et al., 2020), which are based on CNNs and LSTMs. More recently. SOTA transformers like BERT (Devlin et al., 2018) and GPT-2 (Radford et al., 2019) have achieved good performances, outperforming traditional deep learning vased detectors, at the cost of extensive training and computation. Further, in general, prior disinformation literature has not explored the de- tection of LLM--genrated disinformation.
We leverage existing benchmarks (Cui and Lee, 2020; Shu et al., 2020) for in-distribution evalua- tion, and collect new data for out-of-distribution evaluation. Our dataset is summarized follows.
CoAID (Cui and Lee, 2020): 4K+ news and 900+ posts related to COVID-19.
FakeNewsNet (Shu et al., 2020): 23K+ news and 690K+ tweets with the theme of Politics. F3: New dataset that we collected, including pre- and post-GPT-3.5 subsets of political so- cial media and news articles from Politifact! and Snopes?.
Other studies generate synthetic disinformation using LLMs (Zhou et al., 2023; Sun al., 2023) but do not evaluate faithfulness or compare hu- man vs. LLM-generatte disinformation detection. Despite advanced capabilities, risks of advanced LLMs in generating disinformation and zero-shot detection
Prompt-based based generation exhibits near-random performance due to poor sample selection (Liu et al., 2023b). To address this, we removed noisy, duplicated news and posts (e.g., "this website is using security service to protect from online attacks"), and text exceeding 2K+ tokens, consider- ing pre-trained max token sequence range of LLMs (LLaMM (Rozière et al., 2023)), and all base- line detectors. Our final human datasct has 12,723 verified, high-quality samples (Table 2).
(in-distribution and out-of-distributi remain underexplored (Zhou et al., 2023; Liu et al.. 2023a; Qin al., 2023), which we attempt fill the gap in understanding.
3 Problem Definition
RQ1: Disinformation Generation
We use prompt engineering to examine how LLMs use statistical patterns learned during training to generate text sequences and produce zero-shot bi- nary class responses. First, we define our prompt template (See Figure 2) that forms the basic struc- ture of our LLMs' input text. Then, we define F3 prompt-based text generation and disinformation detection. See Appendix for further details. Our problems are formally defined as follows:
We first investigate the frequency extent to which prompt engineering can exploit LLMs to efficiently produce disinformation without hallucination mis- alignments (See steps 1 & 2 in Fig. 1).
5.1 RQ 1.1 Overriding Alignment Tuning
Alignment tuning prevents LLMs from generat- ing harmful disinformation and minimizes toxic- ity (Zhao et al., 2023). This technique, pioneered by OpenAI, optimizes models to produce more beneficial behaviors through continued training on
RQ1 Disinformation Generation: For F3 text gener- ation, we use a generator G that takes prefix-prompt prompt Xc+R+1 input generates text sequences such that G() T (Figure 10).
Poliiifacc:: https://www. polititiioe wwwiiiiafttcooomo
2Snopes: https: / /twww//wwww///////////// snopes. com/

---

Fra
Dataset
SM
NA
Table 2: Details human datasets. Each symbol de- notes follows. R: real news samples; E: fake news samples; L: ground-truth veracity from fact-checking sources; SM: social media posts; NA: news articles; Pre-GPT: in-distribution samples before Sept 2021; and Human Written Social Media Past
1.337
2.649
CoAID
San Francisca had twice many drug overdese deaths oa COVID dearhs lext year. This true
F
871
154
Real News
Pre-GPT3.5
NekeNewsNet
R
2,457
emargency with pelitical
F
1,625
indifference if not encouragement
R
354
F3
F
653
GPT-35 turbo
News:
Social San Franziseo had FIVE times as many drug
R
678
151
Post-GPT3.5
F3
1.615
Minor
overdose deaths COVID deaths last year-
5,508
7,155
Prompt
The government's foilure to acdresss this crisis
amountts crime against humonity,
San Francicc had fwice many COVID deaths
Major Prompt
drug overdase deatha last year.
emergency met with palitical incompetence i not negligence,
San Francisco MORE drug overdase ducths than COVID deaths lost year This complete dixoster and the politicions are turning bind to itf We need toke oction NOW to this spidemic from wiping out our city
Gritical Prompt
Post-GPT: cutt-o--distibuttiin samples after Sept 2021.
human preferences (Zhao et al., 2023). After ex- tensive prompt engineering experiments, however, we unfold the role of positive impersonation and thus employ impersonator roles to override such protections. Assigning personas (e.g., "You are an AI news curator" or "You are an Al news investiga- tor") circumvents GPT-3.5's alignment, triggering unintended malicious generation. Without the im- personator prompt-role parameter, GPT-3.5 refuses by stating: "Sorry, I can't 't assist with that request. Disinformation and fake news can have real conse- quences, and it's essential to approach news and information responsibly and ethically."
Perturbation- prompt engineering for disinformation content generation based on severity levels. Minor: Exaggerated numbers shift from "twice as many" to "FIVE" times, with intensified tone labeling "a crime against humanity". Major: COVID and over- dose deaths roles are reversed, and political response recast as "incompetence" and "negligence". Critical: The original statistic changes to vague "MORE" with alarming phrases like "complete disaster" and "wiping out our city".
Figure 3:
balance of creativity and realism. The perturbations avoid easily traceable modifications when generat- ing fake news variants. See Figure 3 for examples and further details in Appendix A (Fig. 20 and 21).
RQ1.1 Finding: Impersonator prompt en- gineering overrides GPT-3.5-turbo's protec- tions, enabling malicious text generation despite alignment tuning.
Our three variants are as follows:
Minor prompt evokes subtle changes to the real news so that they are not instantly identifiable. Thus, LLM-generated disinformation in the Minor type should be more difficult to detect.
5.2 RQ 1.2 Prompt Engineering
We developed prompts using both perturbation and paraphrasing, simulating real-world disinformation varieties from subtle to overt fake contents. Pertur- bation modifies original content (Karpinska et al., 2022), while paraphrasing keeps meaning using real news (Table 2) (Witteveen and Andrews, 2019; Chen et al., 2020b). We devise three variations of each, inspired by machine translation, for varied detectability (Brown et al., 2020; Qi et al., 2020; Warstadt et al., 2020). This helps in creating con- trollable synthetic news content.
Major prompt instigates noticeable but non- radical changes to the real news. Thus, LLM- generated disinformation in the Major type should be more identifiable than those in the Minor type.
Critical prompt induces significant and conspic- uous changes to the real news. The alterations by this prompt will likely be easily detectable.
(2) Paraphrase-based Real News Generation
Paraphrasing prompts are an effective technique for abstractive summarization and paraphrasing (Kr- ishna et al., 2023; Evans and Roberts, 2009b,a). We adopted three techniques to re-engineer authentic news: (1) summarization of key factual details (Wit- teveen and Andrews, 2019), (2) rewording while preserving vital factual information (Chen et al., 2020b), and (3) thorough rephrasing guided by key facts (Chen et al., 2020a). Each prompt aims to preserve the essence, innovate wording, seamlessly
(1) Perturbation--ased Fake News Generation Perturbation- based prompting makes controlled al- terations to the original content. We categorize prompts into minor, major, and critical levels based on modification severity (Karpinska al., 2022; Chen et al., 2023). These levels range from subtle to overt, while maintaining story structure with a

---

blend with the original, and maintain factual ac- curacy. Figure in Appendix A shows examples exhibiting minor to critical paraphrase real news, further elaborated Appendix A. Our variants are defined as follows:
Metrics
Method
F
1. Factual
Align Scure
0-1
2. Logical
Ental.
[M]
Contextual
BERT-Score
0-1
4. Semantic
Semanic-Dist
D-1
Table 3: PURIFY evaluation metrics. Up-arrows [t]
Minor. Light paraphrasing through concisely summarizing key details without introducing mis- leading information.
indicate the desired higher scores. Down-arrows [4] indicate desired lower scores. [F] denotes fake news and [R] denotes real news. [C] denotes critical range/option.
[N] denotes No and [Y] denotes yes.
Major. Moderate paraphrasing by extracting factual details to guide rewording using different vocabulary while retaining accuracy.
5.3.1 PURIFY: Filtering Misaligned
Hallucinations
Critical. Substantial paraphrasing through com- prehensively rephrasing the content in a unique style guided by factual details.
PURIFY aims detect two misalignment types: (I) LLM--eenrraed real news that however con- tains hallucinations (thus cannot be real news), and (2) LLM--genrrted fake news that however is ballucination-frie (thus cannot be fake news). PURIFY focuses logical fidelity, factual in- tegrity, semantic consistency, and contextual align- ment between the original human and synthetic LLM-generated pair. Specifically, PURIFY com- bines metrics like Natural Language Inference (NLI) (Qin et al., 2023) to eliminate intrinsic hal- lucinations (i.e., the generated text that unfaith- fully contradicts the input prompt's instruction and source content), AlignScore (Zha et al., 2023) address extrinsic hallucination (i.e., the generated text that is unfaithfully nonfactual the input from source content/external knowledge), Semantic Dis- tance to tackle incoherence (Mohammad and Hirst, 2012; Rahutomo et al., 2012), and BERTScore (Zhang et al., 2019) target unrelated context gener- ation to validate the intended fidelity of our LLM experimental data.
Our prefix prompt, therefore, comprises a stan- dard impersonator, dataset content, and instructor element that embeds one variation of the aforemen- tioned perturbation and paraphrase prompt variant (Fig. 9). We found that explicitly guiding an LLM to rephrase the content while retaining factual de- tails produced higher quality and more diverse ren- ditions of the original news. Figure 20 and 21 show prompt definitions/examples. In the end, using GPT-3.5-turbo, we successfully generated 43K+ both real and maliciously fake disinformation to address RQ1.2.
5.3 Ensuring Quality of LLM-Generated Data
Despite our efforts to generate both real and fake news using paraphrase-based based and perturbation- based prompt engineering in Section 5.2, however, we need to double check that the generated real (resp. fake) news is indeed factually correct (resp. factually incorrect). This is because LLM may gen- erate the output text that is "unfaithful to the source or external knowledge," so called the Hallucination phenomenon (Ji et al., 2023). That is, the generated prompt) should
First, to detect intrinsic hallucinations, we use NLI to gauge the logical consistency between the input prompt and the generated output. We use the majority votes of NLI results between GPT-3.5- turbo, M- (Anil et al., 2023), and LLaMA-2 (Rozière et al., 2023). After NLI validation, our initial dataset of 43,272 samples was reduced 39,655 samples by removing 3,617 logically incon- sistent samples-e.g., samples labeled as real but contain intrinsic hallucinations (Appendix B.2).
"real" news (by paraphrase-base be consistent with the input, and thus hallucination- free by definition. On the contrary, the generated "fake" news (by perturbation-bant based prompt) should have contradicting, illogical, or factually incorrect information, and thus must contain hallucination as part by definition. When some of the generated data does not show this alignment clearly, they are no longer good real or fake news to use for studying RQ2, and thus must be filtered out.
Second, to detect extrinsic hallucinations, we use AlignScore, which gauges fine-grain degrees of factual alignment between the input prompt and the generated output. High AlignScore ver- ifies factual consistency with real news versus low scores for fake news. To account for nuances, we use hybrid statistical methods (i.c., standard devi- ation and interquartile range) to create an Align-
To ensure the quality of the generated real and fake news, thus, we introduce the PURIFY (Prompt Unraveling and Removing Interface for Fabricated Hallucinations Yarns) (Step in Fig. l).

---

Score threshold. We derived acceptance thresh- olds of 0.0 0.36 for fake news and 0.61 1.0 for real news (Appendix B.1). After removing 3,281 high-scoring misaligned fake news and 8,707 low- scoring misaligned real news, our final F3 dataset totaled 27,667 samples.
Fra
Dataset
SM
NA
R
2,520
6,289
CoAID
F
3,592
5,667
R
681
Pre-GPT3.5
FakeNewsNet
F
5,703
R
970
F3
F
1.395
R
269
23
Post-GPT3.5
F3
Next, after removing logically and factually misaligned texts, we apply semantic and contex- tual consistency measures to validate that LLM- generated data also aligns with the original topic and context. This ensures that the LLM's intended real or fake outputs are meaningful, not random or out-of-scope text (Table 3). F3 dataset exhibits high contextual consistency, with BERT'Score metrics ranging from 0.92-1 1.00 for fake news and 0.95-1.00 for real news, and also exhibits strong semantic con- sistency, with semantic distance scores spanning 0.001-0.014 for fake news and 0-0.01 for real news, These measures validate that F3-generated texts faithfully retain meaning and topics from the origi- nal source texts per intended input prompts. While there is relevant contextual and semantic consis- tency, the overlap in these metrics scores represents the challenge for LLM to distinguish between real and fake news. Thereby, PURIFY ensures our data aligns logically and factually with the prompt in relation to the original text. It also filters high- quality, meaningful, and nuanced real or fake con- tent to simulate subtle extrinsic//ntrinsic hallucina- tions and elusive "silent echoes" disinformation in real-world contexts (Table 3).
F
395
163
9,141
18,526
Table 4: Details of final LLM-data samples. Each sym- bol is the same with Table 2.
We evaluate the capabilities of LLMs for dis- information detection on five fronts: (1) human-
written vs. L.M LLM--genrrted news with minor, ma- jor, and critical perturbationnsppaaphrreeed text, (2) self-generated vs. other LLM -ag--eeeraaed, (3) (short) social media posts vs. (long) news articles, (4) in-distribution (ID) vs. out-of-distributtion (OOD)
(i.e., comparison between Pre-GPT-3.5 and Post- GPT-3.5 data), and (5) zero-shot LLMs vs. domain- specific detectors. For detectors' performance eval- uation, we employ Macro-Fl scores on our imbal- anced human-Al datasets.
6.1 Dataset Set-up and Models Tested
Given our dataset in Tables 2 and 4, we struc- ture our experiments' data as follows: (1) We di- vide the data into pre- vs. post-GPT-3.5 for in- distribution vs. -iitt---istiiuutinn evaluation. Pre- GPT-3.5 allows us to train and validate models for in-distribution testing. Post-GPT-3.5 provides unseen data for assessing outtrf-distribution gen- eralization. (2) We further stratify the data into human vs. LLM -----eeeertted for comparing perfor- mance on these two test cases. (3) We also separate data into news articles and social media posts, as models may perform differently on these text types. (4) For Pre-GPT-3.5, we split data into 70% for training, 20% for validation, and 10% for testing via stratified sampling ensure balanced real/fake news splits. (5) We do not split post- 5 data, using the full set for OOD testing.
Using PURIFY, we find 38% of data generated by GPT-3.5-turbo contains hallucinated misalignments. While largely producing contextual and semantic aligned text, 8% of overall samples show logical misalignment, and 30% further ex- hibit factual inconsistencies (Fig. 15).
RQ1.2 Finding:
Finally, Table 4 depicts details of our new F3 LLM--gnerated dataset (after PURIFY step) across pre- and PPs--GPT-3.5 periods, including social me- dia and news articles. We use PaLM-2 to conduct a thematic analysis of the dataset (Fig. 18). Table 6 compares F3 dataset with emerging related datasets. Table 10 details misalignment examples.
Next, we test how well different models detect F3 disinformation generate. We first use four lead- ing LLMs with emergent abilities such as GPT- 3.5- ..S-Turo, LLaMA 2- 2-Chat (Rozière et al., 2023), LLaMA-2-GPT4 (Touvron et al., 2023), and Dolly- 2 (Conover et al., 2023). See Table 7 for details and implementation in Appendix D. We then use three popular domain-specific fake news detectors including dEFEND (Shu et al., 2019), TextCNN (Kim, 2014), and BiGRU (Ma et al., 2016). These deep learning (DL) models are fake news domain-
6 RQ2: Disinformation Detection
In RQ2, we shift our attention investigating how adept L.LMs are at zero-shot binary detection of human and Al-created disinformation compared to SOTA detectors (Fig. 1, steps 4 and 5).

---

specific detectors. In addition, using our dataset, we fine-tune four BERT variants: BERT (Kenton and Toutanova, 2019), CT-BERT (Müller et al., 2023), RoBERTa (Liu et al., 2019), and DeBERTa (He et al., 2020). See Table for details and imple- mentation in Appendix D for all baseline models.
0.9
0.6
0.4
GPT-3.5
Uama-GPT
Liama-2
Delly.2
6.2 Detection: Cloze-Prompe Engineering
Human-Written
We evaluate LLMs' zero-shot disinformation de- tection using prompt engineering. LLMs can reason systematically with simple prompts like "Let's think step-by-ste steppp""see"ppoe for CoT (Zhang et al., 2022: Bommasani al., 2021). Using cloze-style prompts, we apply semantic, intermediate, and step- by-step reasoning, and integrate SOTA prompting approaches with our confidence- based and context- defined reasoning strategies inspired by various logic types (Tang et al., 2023a).
Figure 4: RQ2.1 LLMs' zero-shi shot Human vs. LLM- disinformation detection using Macro-FI Score.
1.0
0.75
0.86
0.48
0.6
0.4
GPT-3.5
Liama GPT
Llama-2
Dolly-2
Minor
Major
Critical
To guide predictions, embed such techniques into our cloze-style prompt instructor parameter (Fig. 2). However, LLMs' alignment using the re- inforcement learning with human feedback (RLHF) limits explicit veracity assessment. We address this issue using our impersonator prompt. See Table and for further details.
Figure 5: RQ2.2 LLMs' zero-shot Self vs. External detection using Macro-FI Score. The yellow dot repre- sents mean (J)).
ical (Fig. 5). However, LLaMA-GPT excels as the top external detector of GPT-3.5- Turbo- cerbo-generated disinformation. LLaMA-2 shows moderate exter- nal detection abilities. Regardless of self or exter- nal detection capacity, by and large, LLMs struggle to accurately detect minor paraphrased and per- turbed disinformation.
6.3 RQ2 Results and Analysis
Our analysis compares the average Macro-F1 scores across RQ2. Table 5 shows the average Macro-F1 scores on pre- and post-GPT-3.5 (i.e., in- and out-of-distribution) datasets. Full detailed results provided in Appendix E.
RQ2.2 Finding: GFT-3.5-Turbo good at self-detection, and LLaMA-GPT is the best external detector.
RQ2.1: Human vs. LLM-generated When evaluating LLMs' zero-shot capabilities for human vs. LLM disinformation detection, we find GPT-3.5-Turbo, LLaMA-GPT, and LLaMA-2 are more accurate detecting LL.M-generaed disin- formation, compared numan-authored disinfor- mation (Fig. 4). On numan-authored data, GPT-3.5- Turbo's accuracy ranges from 55-66% while on LLM-generated data, achieves 60-85%. Dolly-2 shows the lowest accuracy on both human (51- 52%) and LLM (47-50%) disinformation.
RQ2.3: Social Media Posts vs. News Article When evaluating LLMs' ability to detect (short) so- cial media posts vs. (long) news articles detection, @PT-3.5-Turbo (0.66-0.85%) and LLaMA-GPY (0.54%-0.7% 71%) were more accurate on articles. While GPT-3.5-Turbo's 00.55%-0.775 -0.7%))), LLaMA- OPT(0.56%-00666... and LLaMA-2 (0.59%- 0.67%) ()) perform moderately well on posts, com-
pared to Dolly-2's low FI-Scores (Fig.6).
RQ2.1 Finding: LLMs struggle more to detect human-written disinformation, com- pared to LLM-generated variants.
RQ2.3 Finding: LLMs exhibit superior zero-shot performance on (long) news ar- ticles than (short) social media posts.
RQ2.2: Self-generated vs. Externally-eee Externally-generateed PVr-3.5-Turbo 135B displays strong self-detec- tion, outperforming other LLMs overall and across disinformation variants, iee. minor, major, and crit-
RQ2.4: In-Distribution vs. Out-of-Distrinn Distr--Dittiittionn We categorize disinformation data in-distribution or tittof-diitribution relative to GPT-3.5-tune known training timeline to assess detectors' gen-

---

Datu Source
Articles
Posts
Data Categories
Human
LLM-MMn
LLM-Maj
LLM-Cnt
Humaa
LLM-MBR
LLM-MM
LLM-CCI
In- D--istribution
G.T-3.5- 1ers5--Turbb-7555
1,6004
03110
0.8399
0.8454
0,5505
0.5938
0.0645
0.7640
0.7178
LLaMA-GPT-70R
0.5398
0.6960
0.7055
0.6803
0.5579
0.6786
0,6655
0.6440
0.686
LLaMA-2-70B
0,5958
0.1377
0.5984
0,5911
0,5869
0.6344
0.6476
0.6727
0.6125
Dolly-2-12B
0,5151
0.5076
0.4964
0.4669
Customized DL Models
Fine Sine-tuned Transformes
0,7025
0.9751
0,9687
0.9844
0,5787
0.9126
0.9612
0.9514
0.9283
Dut-of-Distributio
G.T-3.5- 175B
I.7170 tas
0.7091
0.7136
1a.2
0,6792
pa0
0.7107 ta3
016072
0.64117
14:0
0.6828
016825
LLaMA-GPT-708
0,7112 130
0.5197 2313
0.6049
193
0,5558
130
0,6072
1055
0.5667
1.:0
0.5961
11:0
0,5218
2017
0.5933 13m
11aMA-2-70B
0,6103 pawe
100
0.9976
100m
0.6024
1:31
0.6165
130n
0.6354
100m
0.6444
11
0.6762
1000
0.6218 1:00
Dolly-2-12B
ta33
0.4592
1002
1308
1002
1004
0.5044
1904
13005
Customized DL Models
0.4609
0.3514
0.5360
0/412
Fine-tuned Trunsformers
0,5121
0.7234
0,7619
0,7101
0.6375
0 169000
0,9395
0.9311
0,5039
Table vs. Out-of- Distribution Comparison. This table presents the average F1 performance generative LLMs, customized deep-learning models, and fine-tuned transformers. Performance is benchmarked across categories human, LLM minor, LLM major, and LLM critical for both articles and posts. The ø column shows the mean performance for each model.
1.0
0.9
0.8
0.7
0.6
0.4
0.5
0.2
0.4
GPT-3.5
Llama-2
Dolly-2
Dally-2
Articles
Posts
Minor
Mejor
Cribical
Figure 6: RQ2.3 LLMMs' zero-shot performance across social media posts and new articles using Macro-FI Score.
Figure 7: RQ2.4 A comparison of LLMs across various disinformation categories. Each represented by bar, with numerical values atop indicating either positive or negative change of in-distribution Macro-FI Score relative out-of-distributton.
cralizability. Disinformation created using hu- man data "before" the release of GPT-3.5-turbo (i.e., Pre-GPT3.5 in Table 2) is considered in- distribution, as such human data may be part of On other
across in-distribution and out-of-distributio data. Overall, fine-tuned transformer models like BERT achieve the best performance, followed by gen- erative LLMs like GPT-3.5-Turbo and then cus- tomized models. However, the average perfor- mance transformers and customized models drops significantly on OOD data compared to gen- erative LLMs.
the training data of hand, disinformation created using human data "af- ter" the release of GPT-3.5-turbe (i.e., Post-GPT3.5 in Table 2) is considered as -i--o----stiiuttinnn as they could not have been part of training data of GPT-3.5-turke
Assessing LLMs' zero-shot ability detect LLMs' in-distribution vs. out-of-distribution detec- tion, as shown in Fig. 7, we found that all LLMs except LLaMA-2 performance declined on out-of- distribution data. Minor-LLM disinformation is associated with lower detection accuracy than Ma- jor and Critical LLM disinformation (Table 5).
Fine-tuned detectors sig- nificantly outperform LLMs and domain- specific detectors but consistent detecting OOD disinformation. GPT3.5- Turbo outperforms domain-spectie detec- tors while other LLMs perform comparably.
RQ2.5 Finding:
RQ2.4 Finding: LLMs show better zero- shot performance on in-distribution data, ex- cept LLaMA-2.
Discussion
F3 prompting shows promise for few-shot de- tection. Our F3 techniques outperformed stan- dard reasoning, showing the potential of advanced prompt engincering to enhance few-shot LLMs' de- tection abilities. Notably, MsReN_ CoT showed the
Zero-Shot L.MMs vs. Domain-Specific We compare zero-shot generative LLMs against customized and fine-tuned transformer detectors
RQ2.5:

---

1.0
models exhibit good in-distribution and out-of- distribution balance, though slightly weaker on unfamiliar data. The performance gap suggests specialization for domain tasks but difficulty gener- alizing.
0.6
0.4
Smart, cunningly crafted (subtle) disinforma- tion challenges even the best current defectors. All models struggled more with minor disinfor- mation alterations compared to major and critical changes. This somewhat expected as the amount or level of fake-ness is small, it is more challenging to determine its veracity. Therefore, developing more sophisticated systems to be able to handle very subtle fake-ness in disinformation is needed. Bypassing alignment tuning is critical but in- consistent. Both disinformation generation and detection tasks require circumventing L.MS' align- ment tuning mechanisms. While our imperson- ator approach successfully bypassed four LLMs' protections, it failed to bypass Falcon's 's alignment tuning unless using CoT reasoning. This inconsis- tency of bypassing alignment tuning across models highlights key limitation for robustly evaluating L.MMs' disinformation capabilities.
GPT-3
Liams-
Liama
Dally-2
BEA Lare
CTBERT
Rottrta
Gedieita
GETENENC
Tee
BIGRU
Pre-GPT-3 Turbo
Post-GPT-3.5-Turaa
Figure RQ2.5 Box plots that compare the perfor- mance of Zeroshot LLMs, Neural Network Customized and Finetumed Transformer detectors using FI-S ---oecess Each model's performance is evaluated in scenarios: "Pre-GPT-3.5-Trboo" (represented in blue) and "Post- GPT-3.5-Turbo" (represented orange)
strongest results across human-LLM datasets. For articles, GPT-3.5-Ture 175B with Analyze_Clin Cld2 achieved top performance on human and LLM data. The integrated reasoning strategies underpinning F3 cloze-prompts account for their standout per- formance, highlighting fruitful directions for devel- oping broadly applicable disinformation detection prompts.
Responsible LLM use is critical. LLMs' misuse during crises can have serious consequences (Wei- dinger al., 2022). We reveal how misuse popular LLM by bypassing its protective guardrails to generate disinformation. Many top-performing LLMs are publicly available. Thus, we must pre- pare for the risks of unintended harmful political, cyber-security, and public health applications.
GPT-3.5-turbo excels at detecting human- written and self-written disinformation. Leverag- ing prompting. GPT-3.5-turbo exceeded other mod- els at detecting hrmttewritten and self-generated disinformation. Despite stiff competition, demon- strated superior ceff-detection across all synthetic article and post variants, asserting its zero-shot ca- pabilities. Further assessing its performance other LLMs' disinformation is vital next step. While self-detection is unsurprising due shared vocabulary distribution, this performance under- scores detection potential ChatGPT faces mali- cious exploitation.
Conclusion
Our work demonstrates LLMs' promise for self- detection in zero-shot framework, reducing training needs. While dangerous if misused, re- purposing LLMs to counter disinformation attacks has advantages. Key results like GPT-3.5's per- formance highlight generative models' abilities beyond text generation. To aid research, we developed PURIFY for detecting and removing hallucination--based misaligned content. However, difficulty in detecting subtle disinformation mo- tivates stronger safeguarding of LLMs and more nuanced prompting. Assessing few-shot detection and disinformation mitigation will be critical LLMs continue advancing. While LLMs can po- tentially be misused to create disinformation, we can fight back by re-purposing them as countermea- sures, thus "fighting fire with fire."
LLMs are robust across distributions. GPT- 3.5-turbo consistently detected human-written and L.MM-generted disinformation, both in- distribution and out-of-distributiin,, showing its po- tential. These results highlight the significance of generative LLMSs' applicability in real-world set- tings emerging zero-shot reasoners in disinfor- mation detection. Fine-tuned transformers show re- markably high in-distribution performance, indicat- ing optimization for familiar data. Their lower com- petitive out-of-ding of-disttibution scores demonstrate a ietializationngeeeraliizzzz balance. Customized

---

Limitations
We hope this statement provides clarity on our in- tentions and values. Addressing the socictal dan- gers of disinformation requires proactive work to develop solutions, but at the same time, it must be pursued conscientiously and ethically.
This work demonstrates promising zero-shot dis- information detection using prompt engineering. However, few-shot capabilities remain unevaluated and could further improve performance. Addition- ally, we examined small subset of available LLMs. Testing more and larger models like GPT-4 could provide new insights. Due time constraints, we did not fully optimize prompts to achieve maxi- mally consistent high accuracy for zero-shot detec- tion. Performance variability indicates the need for more generalizable prompts. We were also unable to assess GPT-4 (OpenAT, 2023) due to time con- straints, which can be addressed by future work. Although initially included, in the end, de- cided to remove Falcon-2 (Penedo al., 2023) due to difficulties in bypassing its alignment tuning us- ing our semantic reasoning prompts. Zero-CoT seems to break the 2 alignment tuning. The Model responds, "(No cheating!) False." Without CoT, it would say things like, "I' m sorry, am an AI language model, and cannot provide definitive answer without additional context information." Future work re-evaluate the proposed research questions against more diverse language models. Other future directions include assessing few- shot performance, evaluating more models, de- veloping better prompts, integrating detection ap- proaches, adding multimodal inputs. and collabo- rating with stakeholders. Open questions remain around societal impacts and dual-use risks requir- ing ongoing ethics focus.
Acknowledgements
This work was in part supported by NSF awards 1820609, 1934782, 21 2114824, and 2431144.
References
Abdullah Marish Ali, Fuad Ghaleb, Bander Ali Saleh Al-Rimy. Fawaz Jaber Alsolami, and Asif Irshad Khan. 2022. Deep ensemble fake news detection model using sequential deep learning technique. Sen- sors, 22(18).
Rohan Anil. Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.1040:
Nida Aslam, Irfan Ullah Khan, Farah Salem Alotaibi. Lama Abdulaziz Aldaej, and Asma Khaled Al- dubaikil. 2021. Fake detect: A deep learning en- semble model for fake news detection. Complexity. 2021:1-8.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung. Quyet V Do, Yan Xu, and Pascale Fung. 2023, multitask, multilin- gual, multimodal evaluation of ChatGPT on reason- ing, hallucination, and interactivity. arXiv preprint arXiv:2302.0440 04403.....
Ethics Statement
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The dong-document transformer. ar 200::0000.
This research involves generating and analyzing potentially harmful disinformation. Our released F3 dataset also includes the examples of LLM- generated disinformation. Our aim is to advance the research combat disinformation. However, open dissemination risks the misuse of the gener- ated disinformation in F3 dataset and the methods that enabled such generation. To promote trans- parency while considering these dilemmas: (1) we release codes, prompts, and synthetic data to en- able the reproducibility of our research findings and encourage further research but advise users responsible use. and (2) our release will exclude original real-world misinformation, but only syn- thetic variations, to minimize harmful usages. Addressing disinformation dangers requires de- veloping solutions conscientiously and ethically.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson. Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterj. Annie Chen, Kathleen Creel. Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto. Peter Henderson, John Hewitt, Daniel Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Kr- ishna, Rohith Kuditipudi, Ananya Kumar, Faisal Lad- hak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma,

---

Ali Malik, Christopher Manning, Suvir Mirchan- dani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie. Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Pa- padimitriou, Joon Sung Park, Chris Piech, Eva Porte- lance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan. Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W Thomas, Florian Tramèr, Rose F. Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang. Tianyi Zhang. Xikun Zhang, Yuhui Zhang. Lucia Zheng, Kaidlyn Zhou, and Percy Liang. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint wrXiv:210.00225:
Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 492-502.
Luigi De Angelis, Francesco Baglivo, Guglielmo Arzilli, Gaetano Pierpaolo Privitera, Paolo Ferragina, Al- berto Eugenio Tozzi, and Caterina Rizzo. 2023. Chat- GPT and the rise of large language models: the new Al-driven infodemic threat in public health. Front Public Health, 11:1166100.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language under- standing. arXiv preprint arXiv: (510:088800805
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2020. GSum: A general framework for guided neural abstractive summariza- tion. arXiv preprini 2rX10:2010.00014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are Few-Shot learners. Ad- vances in neural information processing systems.
Angela D Evans and Kim Roberts. 2009a. The effects of different paraphrasing styles on the quality of reports from young child witnesses. Psychol. Crime Law, 15(6):531-555
Angela D Evans and Kim P Roberts. 2009b. Can paraphrasing increase the amount and accuracy of reports from child eyewitnesses? chology, Crime, & Law, 6, 531-546. DÓI: 10. 000108000066333000222333333
"Psy-
Marina Fomicheva and Lucia Specia. 2019. Taking MT evaluation metrics extremes: Beyond correlation with human judgments. Comput. Linguist. Assoc. Comput. Linguist., 45(3):515-558.
Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. 2023. On the possibilities of Al-Generated text detection. arXiv preprint 2334:23300.44733
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
Lichang Chen, Heng Huang, and Minhao Cheng. 2023. PTP: Boosting stability and performance of prompt tuning with perturbation-bast regularizer. arXiv.org.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level adversarial ReProgramming. arXiv preprint arXiv:2101.0012)
Mingda Chen, Sam Wiseman, and Kevin Gimpel. 2020a. Controllable paraphrasing and translation with syn- tactic exemplar. arXivvorg.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint 20Xi:20006.3654..
Mingda Chen, Sam Wiseman, and Kevin Gimpel. 2020b. Exemplar- Exeempla-Cootrollabee paraphrasing and translation using bitext. arXiv preprint 2rX.v:2010.05856.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint MXX.::1904.09990
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan. Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world's 's first truly open instruction- tuned Ilm.
Wei Jason, Tay Yi, Bommasani Rishi, Raffel Colin, Zoph Barret, Borgcaud Sebastian, Yogatama Dani, Bosma Maarten, Zhou Denny, Metzler Donald, et al. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research, 8.
Limeng Cui and Dongwon Lee. 2020. CoAID: COVID- 19 healthcare misinformation dataset. arXiv preprint arXiv:2006.0088
Nirosh Jayakody, Azeem Mohammad, and Malka N Halgamuge. 2022. Fake news detection using a de- centralized deep learning model and federated learn ing. IECON 2022 48th Annual Conference of the IEEE Industrial Electronics Society. IEEE.
Limeng Cui, Haeseung Sco, Maryam Tabar, Fenglong Ma. Suhang Wang, and Dongwon Lee. 2020. DE- TERRENT: Knowledge Guided Graph Attention Net- work for Detecting Healthcare Misinformation. In

---

Ziwei Ji. Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):
Jing Ma. Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. In Proceedings of the Twenty-Fifh International Joint Conference on Arti- ficial Intelligence, pages 3818 8-3824, AAAI Press.
Marzena Karpinska, Nishant Raj, Katherine Thai, Yix- iao Song, Ankita Gupta, and Mohit lyyer. 2022. Demetr: Diagnosing evaluation metries for transla- tion. arXiv preprint arXiv:2210.13744
Saif M Mohammad and Graeme Hirst. 2012. Distri- butional measures of semantic distance: A survey. arXiv preprint 1203.
Jacob Devlin Ming-Wei Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidi- rectional transformers for language understanding. volume 1, page 2.
Martin Mäller, Marcel Salathé, and Per E Kummervold. natural language pro-
2023. Covid-twitter-bers cessing model to analyse covid- 19 content on twitter. Frontiers, 6:1023281.
Yoon Kim. 2014.
Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1 1751, Doha, Qatar. Association for Computational Linguis- tics.
Taichi Murayama. 2021. Dataset of fake news detec- tion and fact verification: survey, arXiv preprint urXiv:2111.0329)
Van-Hoang Nguyen, Kazunari Sugiyama, Preslav Nakov, and Min- Yen Kan. 2020. FANG: Leveraging social context for fake news detection using graph representation. In Proceedings of the 29th ACM In- ternational Conference on Information & Knowledge Management, CIKM '20, pages 1165-1174, New York, NY, USA. Association for Computing Machin- ery.
Takeshi Kojima, Shixiang Shane Gu. Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
OpenAL. 2023. Gpt-4 technical report.
Venelin Kovatchev. 2022. Paraphrasing, textual en- tailment, and semantic similarity above word level. arXiv preprint arXiv:2208.05382 053887
Ray Oshikawa, Jing Qian, and William Yang Wang. 2018. A survey natural language process- ing for fake news detection. arXiv preprint
Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit lyyer. 2023. Paraphrasing evades detectors of Al-generated text, but retrieval is an effective defense. arXiv.org.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Aloheidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.0/11
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for Parameter- Parfieter-Efffciennt prompt tuning. arXiv preprint arXiv:2104.0800
Xiang Lisa Li and Percy Liang. 2021. Prefix -tnning
Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1Ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 101-108, Online. As- sociation for Computational Linguisties.
Hanmeng Liu, Ruoxi Ning, Zhiyang Teng. Jian Liu, Qiji Zhou, and Yue Zhang. 2023a. Evaluating the logical reasoning ability of ChatGPT and GPT-4.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Ji- aao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a General-Purpose natural lan- guage processing task solver? arXiv preprint a7Xiv:2302.06476.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023b. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Sure. 1-15).1335.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei. Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: robustly optimized BERT pretraining approach. arXiv preprimt arXiv:1907.114 16992
Faisal Rahutomo, Teruaki Kitasuka, and Masayoshi Ar- itsugi 2012. Semantic cosine similarity. In The 7th international student conference on advanced science and technology ICAST, volume 4, page I.

---

Laria Reynolds and Kyle McDonell. 2021. Prompt pro- gramming for large language models: Beyond the Few-Shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Com- puting Systems, number Article 314 in CHI EA '21, pages 1-7, New York, NY, USA. Association for Computing Machinery.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint MxXiv:2307.0928:
Adaku Uchendu, Jooyoung Lee, Hua Shen, Thai Le, Ting-Har 'Kenneth' Huang, and Dongwon Lee. 2023, Does Human Collaboration Enhance the Accuracy of Identifying LLM-Generated Deepfake Texts? In 11th AAAI Conf. on Human Computation and Crowd- sourcing (HCOMP).
Baptiste Rozière, Jonas Gehring. Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez. Jérémy Rapin, et al. 2023. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950.
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Bala- subramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can Al-Generated text be reliably detected? arXiv preprint arXiv:2303. 1116..
Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. 2021. TuringBench: A Benchmark Environment for Turing Test in the Age of Neural Text Generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2001-2016, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the Earopean Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255 269,, Online. Association for Computa- tional Linguistics.
Bihek Upadhayay and Vahid Behzadan. 2022. Hybrid deep learning model for fake news detection in social networks (student abstract). Proc. Conf. AAAI Artif. Intell., (166611):1367-1333
Kai Shu, Limeng Cui, Suhang Wang. Dongwon Lee, and Huan Liu. 2019. dEFEND: Explainable fake news detection. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, pages 395-405, New York, NY, USA. Association for Computing Machinery.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo- hananey. Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: The benchmark of linguis- tic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377- 392.
Kai Shu, Deepak Mahudeswaran, Suhang Wang. Dong- won Lee, and Huan Liu. 2020. Fakenewsnet: A data repository with news content, social context, and spa- tiotemporal information for studying fake news on social media. Big data, 833)171--188
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Bd Chi, Quoe Le, and prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems.
Denny Zhou. 2022. Chain-of-Thouur
Giovanni Spitale, Nikola Andoroo,, and Federico Germani. 2023. AT model GPT-3 (dis)informs us bet- ter than humans. arXiv preprint arXi1:230.111924.
I Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, 7ac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and lason Gabriel. 2022. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fair- nexs, Accountability, and Transparency, FAceT '22 pages 214-229, New York, NY, USA. Association for Computing Machinery.
Qi Su, Mingyu Wan, Xiaoqian Liu, Chu-Ren Huang. al. 2020. Motivations, methods and metrics of mis- information detection: an nlp perspective. Natural Language Processing Research, 1(1-2)11-3.
Yanshen Sun, Jianfeng He, Shuo Lei, Limeng Cui, and Chang-Tien, Lu. 2023. Med-mmhl: A mudti-modal dataset for detecting human-and Ilm-generated mis- information in the medical domain. arXiv preprint 1/Xiv:2306008711..
Xiaojuan Tang. Zilong Zheng. Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang 2023a. Large language models are In la-Context seman- tic reasoners rather than symbolic reasoners. arXiv preprint 2305. 2485.
Sam Witteveen and Martin Andrews. 2019. Paraphras- ing with large language models. In Proceedings of the 3rd Workshop on Neural Generation and Trans- lation, Stroudsburg, PA,, USA. Association for Com- putational Linguistics.
Xiaojuan Tang, Zilong Zheng, Jiaqi Li. Fanxu Meng. Song-Chun Zhu, Yitao Liang, and Muhan Zhang. 2023b. Large language models are in-context seman- tie reasoners rather than symbolic reasoners. arXiv preprini 230iv 2300..825.. 14825.
Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. Alignscore: Evaluating factual consistency with a unified alignment function. arXiv preprint arXiv:2305.1677 16739.

---

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2019. BERTScoor:: Evalu- ating text generation with BERT. arXiv preprint arXiv:1904.06775
Appendix
Prompt Engineering
We use prompt engineering to examine how LLMs use statistical patterns learned during training generate text sequences and produce zero-shot bi- nary class responses. We describe the further de- tails of our prompt designs as follows.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. In The Eleventh Interna- tional Conference on Learning Representations.
Wayne Xin Zhao, Kun Zhou, Junyi Li. Tianyi Tang. Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303. 18223.
A.1 Prefix Prompt
Our prefix-prompt framework generates high- quality, coherent synthetic real and fake news con- tent. The goal is to leverage paraphrasing and perturbation techniques. The process starts by se- lecting human-authored content and adding it to prefix prompt. This contains impersonator setting contextual behavior intent and instructions providing guidance. Prompts are engineered paraphrase or perturb the original content at three alteration degrees (MiN, MaJ, CRiT) to produce synthetic real news. The prompt fed into the LLM to generate content.
Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. 2023. Synthetic lies: Understanding di-generated misinformation and evaluating algorithmic and human solutions. In Pro- ceedings of the 2023 CHI Conference Human Factors Computing Systems, CHI '23, New York, NY, USA. Association for Computing Machinery.
Yongchun Zhu Qiang Sheng, Juan Cao, Qiong Nan, Kai Shu, Minghui Wu, Jindong Wang, and Fuzhen Zhuang. 2022. Memory-guide multi-view multi- domain fake news detection. IEEE Transactions on Knowledge and Data Engineering.
Figure 9 demonstrates our paraphrase- real news generation results. Figure 3 shows perturbation-baser fake news generation results.
A.2 RQ1 Disinformation Generation
Figure 10 shows prefix prompt. The prefix prompt (x) combines: (1) Content (C) with real human data. (2) Impersonator (R) establishes con- text, guides generation/Aettctionn, and overrides alignment tuning to generate disinformation. (3) Instructor (Z) with paraphrase (Para) and perturba- tion [Pertur) directives for minimal, major, criti- cal MMinnMMjj/Cre variation transformations. Com- bined parameters formulate cerfix-prompttt (X) input text sequence to generator (G) produce LLM text (T) Reakgreen and Fake:red
A.3 RQ2 Disinformation Detection
Figure 19 shows our Cloze prompt. The Cloze prompt (y) combines: (1) Content (C) with real or fake human?rcen and LLM [Muc
data. (2) Instructor (T) with reasoning tech-
niques' directives for Vanilla [VaN], Intermediate [Zero- - CoT, CoT, etc.] and Semantic Rea- soning [DeF- Gen, DeF - SpeCI transforma- tions. (3) Impersonator (R) establishes context, guides generation/ddteccion.. and overrides align- ment tuning to generate disinformation. Parameters formulate prefix (X) input to generator
(G) to pro- duce LLM label (L). Tables and 9 show our exact

---

O+O+O)
Human-Written Social Media Post
RQI
Prefix Prompt:
Real News
6é Son Francisco had twice deaths as COVID deaths lost year. This true state of emergency is met with political indifference if not encourogement. 99
drug overdase
Contant
Impersonater curator
Sunlight and drnking water connot care
Einstructor
MiN
Slighlly alter key details
insligaling unexpected twist to fabricate misinformation
GPT-4.5 -turoo
News; LUA-Gennerated Medio
g
Real News
Last year, drug everdose deaths in San
MIN (Mal
Minor
Francisco doubled compared to COVID deaths
MaI
Prompt
Despite this real shate of emergency, political
CR T
opothy even support persists
Para
Perturb
Drug everdese fatalities in Sun Francisco surpossed COVID-19 19 deaths by two-fold fold in 2020
Major
Figure 10: F3 prefix prompt.
Prompt
Despite this alarming emergency, political apathy or even endorsement persists
San Francisca's drug overdose deaths surpassed
Intermediate
Inductive
Deductive
Abductive
Critical Prompt
COVID deaths by tworfold in 2020. Despite the alarming numbers, political leaders have shown ittle concern or worse, support for the issus.
VoNl
x
9
Z-CoT
+
9
Figure 9; GPT-3.5-turbo paraphrase-based prompts engineering approach. Minor: The minor paraphrase has been concisely summarized by changing the struc- ture of the sentence slightly and rephrasing some words like "twice many" to "doubled." " The essence and details of the original message remain intact. Major: The major paraphrase changes the structure and word- ing extensively than the minor paraphrase, using words like "fatalities" instead of "deaths" and "two-fold" instead of "twice as many". Still, it remains true the factual content of the original. Critical: The critical paraphrase changes the voice and structure significantly, introducing new perspective ("political leaders") and using a unique style that makes it distinct from the orig- inal. This version provides fresh take on the original content, guided by its factual details but conveyed with unique twist in the message delivery.
MalaN
MaRaN_CCT
13
83
DeF-GeN
13
9
DeF-SpeC
x
9
Anelyze,
88
Analyza,AL_PT
9
9
Figure 11: Categories of our Cloze-style prompts. Cat- egories are: intermediate, step-by-step, inductive, ductive and abductive reasoning. Definition and details about these approaches in relation to LLMs can be found (Zhang al., 2022: Tang et al., 2023b; Zhao al., 2023)
Cloze-style prompts, and Figure shows the cat- egories of the reasoning techniques embedding in F3 zero prompts.
tual pairs. Our intuition is that real LLM-gener- ated news should have high-factual consistency, and fake LLMM-generatdd news should have low- factual consistency. We utilize a hybrid statistical method to define threshold that removes factually inconsistent samples. L.c., a non-parametric hybrid threshold approach using the (1) Interquartile range (IQR) and (2) standard deviation (SD) to balance robustness to spread and central tendency while accounting for skewed real/fake distribution nu- ances. We derived thresholds of 0.0 0.36 for fake and 0.61 1.0 for real news to filter outliers while maintaining nuanced edge cases' diversity. We moved 3281 high-scoring factual-inconsistent fake news and 8707 low-scoring factual-nnconsistent real news, resulting in 27667 factually consistent samples. We discuss our hybrid strategy in more details as follows.
PURIFY Metrics
PURIFY detects and removes non-hallucinations from fake news and hallucinations from real news that are unfaithfully misaligned with the input text. Our PURIFY framework uses four evaluation met- rics as shown in Table 3. We describe the future details of those metries as follows.
B.1 Factual Consistency
AlignScore: We assess LLM factual consistency using SOTA AlignScore (Zha et al., 2023). As shown Figure 12, 0 AlignScore represents a low, and represents a high degree of factuality between LLM--genrated and the original human- written tex-

---

10000
12 and 13. (2) Standard deviation with 1QR balance spread and central tendency, maintaining robustness. Our hybrid thresholds of 0.36 for fake and 0.61 for real news remove surprisingly factual fake and suspicious real samples, filtering outliers while capturing edge cases and removing inconsis- tent hallucinations. See our algorithmic represen- tation approach below, which utilizes Q0.75,rcal for the 75th percentile) and f:
22003
0.3
Alasioo
10000
50002
Computing IQR for Fake and Real News:
2000
63
Riake Q0.75, - 2ake
Figure 12: AlignScore distribution for real and fake news before (above) and after (below) removing incon- sistent samples. We filter out fake above 0.36 and real news below 0.61 to exclude factual fakes and ques- tionable reals.
T Q0.75,eeal Q0.25,real
where: Q0.25,fake and Q0.75. denote the 1st (25th percentile) and 3rd quartiles (75th per- centile) of the AlignScore for fake news, re- spectively. Q0.25 and Q0.75.caal denote the 1st and 3rd quartiles for real news, respec- tively.
Computing the Hybrid Threshold for Fake News:
0.9
5²
Brake, percentile Qu.Quuu fag
Alpricon
14
fakee std
Rfake + Ffake
where Take is the standard deviation of the AlignScore for fake news.
Atake, percentile Otake, std
Onybrid, fake
2
Figure 13: AlignScore distribution for real and fake news before (above) and after (below) removing incon- sistent samples stratify by generative prompts categories. We filter out fake news above 0.36 and real news below 0.61 to exclude factual fakes and questionable reals.
3. Computing the Hybrid Threshold for Real News:
Oreal, percentile Q0.90.
A std T Rreal Greal
B.1.1 Factuality Hybrid Threshold Strategy
where real denotes the standard deviation of the AlignScore for real news.
Interquartile Range & Standard Deviation: Since LLMs often generate hallucinated text, we assess their factual consistency using AlignScore (Zha et al., 2023), a SOTA facility metric. We filter out high-scoring fake and low-scoring real news to remove inconsistent samples. AlignScore pro- vides single value indicating factual consistency. The AlignScore distribution for real and fake news is complex, requiring robust discernment of nu- ances. We use parameerre,,, hybrid approach with (1) Interquartile range (IQR) to consider the rightfully skewed fake and real distributions (Fig.
Great, percentile + Oreat, std
Ohybrid, real
B.2 Natural Language Inference
Prior hallucination detection studics have used sta- tistical, model-based, and human evaluation meth- ods. We adopt the NLI model-based approach as overcomes the limitations of statistical ap- proaches in handling syntactic and semantic vari- ations (Ji et al., 2023). NLI metric also exhibits

---

robustness to lexical variability compared token- matching techniques by counterpart methods such as Information Retrieval and Question Answer Met- rics. NLI's semantic/logical consistency assess- ment strengths suit our misaligned hallucination detection goals (Ji et al., 2023).
(Karpinska et al., 2022: Zhang al., 2019). Fig- ure 16 shows high contextual consistency ranging from 92%-100% in our final experimental dataset. BERTScore's s ability to understand context and se- mantics makes a formidable tool in the fight against disinformation. When integrated into a comprehensive disinformation detection pipeline, it can significantly enhance the accuracy and ro- bustness of fake news detection efforts.
In the spirit of fighting fire with fire, propose using LLMs GPT-3.5-turbe and other LLMs for NLT hallucination detection in generated disinfor- mation. The core hypothesis is: synthetic real news should logically be consistent with human- human-written written real news, while synthetic fake news should not, Our approach employs ILJ using models like GPT- 3.5, PaLM-2, and LLaMA-2, taking majority vote among their decisions. Each model labels logical entailment for an input pair classify if the syn- thetic text is consistent with the human-written text or not entailment otherwise.
B.4 Semantic Distance
Using Huggin Face implementation of AllenAl's longformer-base-400 embeddings and cosine sim- ilarity, we derived semantic distance scores for LLM-generated real and fake news (Beltagy et al., 2020). In our analysis of F3 LLM-generated disin- formation, we observed, to a large extent, indistinct patterns in the semantic distance scores for both real and fake news. Specifically, the scores for real news ranged from 0 to 0.01, while those for fake news spanned from 0.001 to 0.014. This overlap may suggest that, within this range, might be challenging to semantically distinguish between real and fake news based solely on the semantic distance scores (Mohammad and Hirst, 2012). Low semantic distance scores (close to 0) in- dicate high semantic similarity between two texts. Here, this suggests that LLM-generated disinforma- tion closely mimics real news semantics, making differentiation challenging based on content alone. The narrow semantic gap highlights LLMs' sophis- tication in generating articles aligning closely with genuine news in meaning. In contrast, higher se- mantic distance scores signal greater divergence between texts, potentially from the model misinter- preting context, diverging from the topic, or gener- ating factual inaccuracies (Mohammad and Hirst, 2012).
Given a piece of human- text (real news), T. we prompt LLM to generate real news, T'real and fake news, T' fake, such that Terlal is similar T and T' fake is dissimilar T. Due LLM's ability to sometimes generate texts un- faithful to the prompt, we define an entailment model - N(), such that for LLM-generated real news, T'real should entail T, and for fake news, T' fake should Not-entail T. Therefore, using the entailment model, (.), we can assess logical con- sistency between the original (human-written) and the generated texts, thus removing misaligned hal- lucinated LLM-ggerrtted texts.
We find GPT-3.5-tubo can generate logically consistent, genuine and fabricated synthetic con- tent, validating this hypothesis. However, when analyzing more nuanced pairs, all LLMs occasion- ally struggle with logical consistency. Thus, while illustrating the potential for hallucination detec- tion, our results also reveal limitations on more difficult cases. After NLI, our generated dataset of 43272 samples was reduced by 3617, resulting in 39655 samples (See Fig. 14 for more details).
The low scores pose a detection challenge, as traditional methods relying on obvious inconsis- tencies may be insufficient. Thus, the nuanced, contextually accurate nature of LLM outputs de- mands advanced, multifaceted detection strategies. This close similarity underscores the risks of LLM misuse for spreading synthetic disinformation. This emphasizes the need to monitor generative LLMs carefully, understand their behaviors, and develop mitigation strategies (Mohammad and Hirst, 2012).
B.3 Contextual Consistency
BERTScore: This metric leverages the capabilities of the BERT language model to measure the simi- larity between generated and reference texts. Due to BERT's s inherent ability to capture the context of entire sentences, BERTScore is especially suitable for evaluating semantic and contextual consistency
B.5 Hallucination Misalignment Cases
This work defines disinformation as intentionally fabricating false information to mislead. We also
Forr our evaluation, we utilized the BERTScore from Hug- gane with the 'Roherta-large" model.

---

GPT-3.-Turbe
Llama-2-708
PaLM-22Tuxx-BR
Labels
Labels
Labels
Entailment (55.4
Not Entailment (56.5
Entailment (39.0
Not Entai inar (44.5
Entailment (43.5
Not Entailment (41.
1. Real vs. Fake Niees
2. In and Out Distrbution
News Articies vs. Social Media Host
Consisent
Consisient
Corcstery
1/248
18467
1095
34620
25670
10045
Inconsistent
3511
2763
105
6169
4998
1276
Fake
Real
post-GPT
Ge-GPT
News Articles
SM Posts
Vericty
GPT-3.5- lunbo Era
Content Source
PRE-GPT: Rimos
Social Meda
POST-GPT: Neus Articles
Social Media Pest
Perturbation
Paraphrase
Consistent
Consistent
25334
9286
336
759
11964
12051
11700
Logiol Carazonyy Incorsistent
Incorsistnnt
Inconsisten:
4971
1198
1877
1988
2409
News Articles
SM Posts
News Articles
SM Posts
Critical
Major Typo
Minor
Figure 14: PURIFY Logical Consistency Confusion Matrix.
800
40000
600
30000
400
20000
200
10000
0.92
0.50
0.94
0.35
0.95
0.57
0.95
0.50
Total
Logical
Fectual
Logical
Factual
30
Figure 15: RQ1: Reduction in LLM- --Meereeatted disin- formation samples using GPT-3.5-turfe The initial "Total" represents the complete dataset. The subsequent reductions are achieved by applying consistency mea- sures. The "Logical" stage reflects the dataset size after removing logically inconsistent samples based on jority vote among GPT-3.5-turbo, PaL-M-2-text-bison and LLaMA-2 using the Natural Language Inference metric. The final "Factual" stage depicts the dataset after further refinement by eliminating samples with factual inconsistencies using the Alignscore method.
30
0.92
0
0.38
L.00
BERTScom
Crioa
Maj
Figure 16: PURIFY Contextual Consistency Distribu- tion.

---

200
C.1 PaLM-2 LLM-Data Thematic Analysis
700
We conducted Thematic analysis of dataset after PURIFY. We used PaLM-2 label our data themes. The top six themes include health, death, harm and Tragedy, public safety, and polítics re- spectfully. See Fig. 18 for more details.
31K
800
300
200
800
0,002
0,004
01001
0.008
00000
0012
0.04
Semantic
Distance Scrre
Heath
Duath
Figure 17: PURIFY Semantic Distance Distribution.
Ham Tragedy
Publec Safeyy
Towr
Fettks
Irsal:
adopt the common data-to tata-to-text definition of hallu- cination LLM--genrated text that intrinsically (contradictory) or extrinsically (factually incorrect) unfaithful to the input. Our input includes original real news text and instructions to modify it into either real or fake news.
Lega
Frefenty
Dempttoy
Finome Wessoes
Soual
2010
San
7501
1000
1200
Isan
13500
Notably, disinformation and hallucinated text both intend to mislead by definition. Therefore, when prompted to generate fake news, LLMs may produce hallucinations aligned with that intent. However, we observed cases where LLMs gener- ated hallucinations despite fake prompts. Our framework PURIFY identifies these mismatches, which are unfaithful the input by definition. Thus, we categorize them as hallucinations be re- moved. The same principle applics to mismatches in real news generation.
may
Figure 18: Our dataset category based on PaLM The- matic analysis.
D Model Implementation Details
This section provides baseline implementation specifies for the LLMs, customized detectors, and fine-tuned transformers used in our experiments.
D.1 Generative LLM
Ultimately, goal is to develop dataset con- taining (1) Non-hallucinated real upholding source integrity as prompted and (2) Hallucinated fake news intentionally not upholding source in- tegrity when prompted to fabricate. We present two cases of hallucination misalignment in Table 10:
We leveraged the OpenAl Software Development Kit (SDK) and Application Programming Interface (API) to access GPT-3.5. We used the following hyperparameters: lemperature of 0.7 and max to- ken of 4096. All experiments occurred on Google Colab Pro using API.
For LLaMA-70B-OO -hat and LLaMA-GPT we set temperature to 0.7, top_p to 0.9, and max_ max_iokens to 4096 for binary classification. For PaLM-2 we used: candidate count of 1, max output tokens of 256, temperature of 0.2, top-P of 0.8, and top-K of 40. All experiments occurred on Google Colab using API such as DecpInfra
C Dataset Description
F3 is the first disinformation dataset that evaluated and removed -.M-ggenrated content subjected to misaligned 'hallucination' wwere LLMs produce text unfaithful to the prompt. We ensure that real news is actually real and fake news fake (Ji et al., 2023). While rarely prior studies (Cui and Lee, 2020; Sun et al., 2023; Zhou et al., 2023) investi- gated LLM -ggnerree disinformation generation, they did not rigorously verify the fidelity of such generated content and primarily focused on fake LLM data rather than both real and fake (Oshikawa al., 2018; Su al., 2020; Murayama, 2021). Please see a comparison of our dataset and other datasets Table 6.
D.2 Customized Detectors
We followed the original paper implementations for dEFENDIC, TextCNN, and BiGRU without modi- fications. For example, dEFENDIC was trained on Politifacts data. See Appendix for training details. All experiments occurred on Google Colab using API.
'DecPInfru.com https://deeiinfra.comm

---

Data
HR
HF
LR
LF
N
SM
OA
TD
Start
End
CoAID
X
2019-Dec
2020-Sep
Synthetic Lies
X
7
I
2023
NekeNewsNet
/
X
V
V
1
2020
Med-MMHL
V
2
2017-Jan
2023-May
F3
/
V
2
2017-Oct
2023-Feb
Table 6: Overview of data sources: CoAID (Cui and Lee, 2020), Med-MMHL (Sun et al., 2023), Synthethetic lies (Zhou et al., 2023) Acronyms used: HR (Human Real), HF (Human Fake), LR (LLM Real), LF (LLM Fake), N (News), SM (Social Media), OA (Openly Available), TD (Topic Domain).
No. Detectors
Description
Castomized Detectors
JEFENDAC (Shu al., 2019)
detector, dEFENDIC dEFEND variant only using contents. begins employyng word level attention mechanismm individual sentences within the news content. Subsqquetty, the festures extracted from these sentences are combined using an average pooling layer, then feeds softmax layer for
dEFEND
classification.
TextCNN (Kim, 2014)
Text CNN employs convelutional neural networks represeni conteni. With the mullippe convolution filters, it is adope at copturing text features of varying gramnlarties. We follow the implemen- und parameters from the most recent 2022).
trained for news
(Zhu al.,
BiGRU (Ma et al., 2016)
BiGRU is common baseline for fake news detection. We follow the text-hased BiGRU with RoREKTa embedking al., 2022).
Fine Tuned Detectors
BERT Large (Kenton and Toutanova, 2019)
BERT is an encoder only Transformer model that is trained to predict randomll masked tokens in the imput. We use the BERT large uncased model. CT-BERT
CT-BERT (Miller ol., 2023) RoBERTa (Liu al., 2019)
HiiT--lrg--meesss model trained 97M messages Twiner about COVID-19. We use the RoBERTa large model, a implementatiio of BERT with modifications to key hyperpurame- ters and minor embedding tweaks.
DaRERTa (He 2020)
utilize DeRERTh' lntest version, DeRERTa-y model, which pre-trained FCFFCTRRR A-Syle with gradient disentangled embedking sharing. Due to linited computational resourves, we are only able to run the base model.
Zero-shot
GPT:335 Turbo!
OpenAl's SOTA model designed for variety of natural language processing tasks. Advanced language model for comversatioaal Al applications. successor LLaMA
9
LLaMA-2-Chit (Kozibre ot al., 2023)
LLaMA GPT4 (Touvron
advanced training techniques
performance.
11 Dolly-2 (Conover et 2023)
Dolly is advanced generation model that exhibits human like generation capabilities.
Table 7: Details of baseline models used for disinformation detection.
RO2
Cloze Prompt:
c
++++
E Experiment Full Result
Content TRUMP said the Election was
impersonatorr
detector
We provide full results of the Macro-Fl score for all detectors in this paper. Tables 11 and 12 show the results of the in-distribution and out- of-distribution performance of LLM-based mod- els, respectively. Tables 13 and 14 are the in- distribution and tomized detectors, respectively. Tables 15 and 16 are the in-distribution and out-of-distribution re- sults with fine-tuned transformer- based detectors, respectively.
Instructor
Zero-Co
Analyze & classify fake
stolen
real Think step steppp....pp
VoNl
X-Cot
MaReN CoT
results with cus-
A-Dolt
Dalf GaN
DeF SpoC
Figure 19: F3 Cloze prompt.
D.3 Fine-Tuned Transformers
For transformer training, we used a learning rate of 2e-5, batch size 4, weight decay of 0.01, Adam's epsilon of Ic-08, and training epoch. All training occurred on Google Colab with one GPU.

---

Name
Description
Cloze-Prompts
X-CoT
Motvattd by Zhang et al. (2022)'s Auto- CoT approach, our Explain- COT (X- CaT) confidence reasoning prompt autn- matically generates rationales and inte- grated CoT to derive an output using the phrase. This prompt "explains to justify the rationale behind your answer step by stop"
impersonator "You are an Al assistant trained to detect fake
news." instructor "Analyze the given text, explain your reason-
ing step-by-step. and determine if it is real or fake news."
A-CoN
Auto-CoN approach. our Auto-COT (A- CoT) confidence reasoning prompt auto- matically generates rationales and confi- dence measures to derive an output us- ing the phrase. "Explain or justify the rationale behind your answer and rate your confidence ranging from to 100."
impersonator "You are un Al ussistant trained to detect fuke
news with confidence estimates." instructor : "Analyze the given text, provide a confidence score between 0-100%, and determine if it is real or fake news."
MsReN
Motivated by Reynolds and McDonell (2021); Bang et al. (2023), the multi- step reasoning (MsR) approach employs the simple statement, "Let's solve this problem by splitting into steps." It guides LLMs to think in steps, evalu- ating various indicators and factors to reach a conclusive judgment.
impersonator "You are Al fact checker trained detect fake
news." instructor "Analyze the text in detail as fact checker would solve it by splitting your reasoning into steps. Check for misleading info, false claims, biased language. If real, respond "True,, if fake, respond "False"."
MSReN_Coo C
Manual-CoT Wei al. (2022) instructs LLMs to exccute manually crafted CoT instructions. MSReN CoT integrates a series of multi-step reasoning, including intermediate-step CoT reasoning. This methodology effectively solves complex reasoning tasks (Zhao et al., 2023).
impersonator "You are an Al fact checker trained detect
fake news." instructor "Analyze the text in detail as fact checker would. Explain your reasoning, stop. then think slowly, step-by-step. If real, respond 'True', i fake, respond "False" !
DeF-Gen
Given G.T-3.5 3.5-nurho's 's observed weak- nesses inductive and multi-step cupa- bilities, our approuch focuses contextu- ally. emphasizing deductive and abduc- tive reasoning (Bang et al., 2023).
impersonator "You are an Al assistant trained to detect fake
news." " instructor : "Determine the text contains factual
formation supported by evidence (real) or nisleadinginncccrate information (fake). Respond with "True' or "False." "
DcF-SpeC
As GPT-3.5 stronger in doductive rea- soning, this upproach implements prede- fined contextual rules to guide LLMs to- wards logical conclusion (Bang et al., 2023).
impersonator "You are un Al assistant trained to detect fake
news." " instructor "Analyze the tone, language, sources de- termine if the text is real (supported by facts) or fake (misleuding info). Respond with 'True' 'Fulse""
Analyze_Cla CCd2
Manual-CoT Wei al. (2022) instructs LLMs to execute manually crafted CoT instructions. Analyze Cld2 integrates a series of multi-step reasoning as manual internediate- step CoT reasoning. This methodology effectively solves complex reasoning tasks (Zhuo et al., 2023).
impersonator "You un Al assistant specially trained to de-
teet fake news with high accuracy." instructor "Analyze the given news article in depth. Check for the following indicutors that it may be fake news: (1)Inaccurate or misleading information, (2) Lack of evidence for claims. (3)tiontynylly charged language with the intent to provoke outrage or shock, (4) Biased portrayal of events or people (5) Unverified sources 'experts' (6) Logi- cal fallacies or conspiracy theories without proof. If the article exhibits multiple indicators of fake news, respond "False'. If the article is factual and supported by credible evidence, respond 'Truc', Your judgment should be highly accurate."
AnalyzeAAI_GPT
Manual-CoT Wei al. (2022) instructs LLMs to execute manually crafted CoT instructions. Analyze_At _L_GTT inte- grates a series of multi-step reasoning manual intermediate-step CoT rea- soning. This methodology effectively solves complex reasoning tasks (Zhao et al., 2023).
impersonator "You
Al trained with extensive knowl-
edge up in 2021 on various news articles, both real and fake." instructor :AAnalyze the given text for potential indicators of fake news, such as: (1) Sensationalist or emotionally charged language. (2) Absence of specific details or dates. (3) Over- generalizations or sweeping statements. (4) Statements that are too good to be true or overly dramatic. (5) Lack of logical flow in arguments or jumping to conclusions without evidence. It's essential to understand that without real-time verification capabil- ities, your judgment will based on patterns and knowledge up to your last training. Using these textual cues and your training. determine the credibility of the given text. If it seems factual and consistent with your training. respond "True", If it exhibits patterns typical of fake news, respond "False."
Table 8: F3 Cloze-Style Prompts for binary Zero-shot disinformation detection:

---

Name
Description
Cloze- Cloze-Prumpts
VaN
Vanilla prompt fundamen- tal baseline prompt designed to deliver brief, precise instructions to LLMs, such as: "assess whether this piece of news real or foke."
impersonator
"You ussistant trained detect fake
news." instructor real fake news."
"Analyze the given text and determine if
Z-CoT
Kojima et al. (2022)'s Zero-Shot- CoT uniquely leverages LLMSs self- formulated rationales by integrating a standard VaN instruction with the sim- ple phrase, "Ler's think step by step known as Chain of Thoughts (CoT)."
impersonator "You Al assistant trained to detect fake
news." instructor "Deeply Analyze the given text, think step- by-step, and determine if real fake news."
Table 9: SOTA Cloze-Style Detection Prompts for binary Zero-shot disinformation detection:
Perturbation
Minor-Prompt
Major-Prompt
Critical-Prompt
This perturbation-base prompt is designed provoke modifications to real news that are significant and clear but the of being extreme.
This perturbation-based instruction prompt aims to elicit minor changes that are not immediately recognizable as fake news.
This perturbatio-boased instruction prompt is intended to incite drastic, noticeable, and alterations to real news.
Definition
significant
This prompt instructs LM to subtly olter a provided news story, incorporating on unexpected yet plousible twist transform into fake news. should story's structure, balance creativity with realism, easily troceable, and achere to specified length limit.
prompt instrucls an incorporate maderate unespected twists and false details create fake news story. The Al must maintain the story's structure, balance creativity with realism, make undetectable alterations, odhere specified length limit.
This prompt instructs LM to rewrite given news story entirely, adding sensational yet plausible fake encouroges exploiting confirmation bias while maintaining structure and set token limit.
Description
changes not
We hypothesize that the changes guided by this prompt should recognizable fake than the minor prompt deviates much more from its original real news.
We hypothesize that the changes guided by this prompt likely recognizable
We hypothesize that the changes guided by this prompt should not be easily recognized as fake news as slightly deviates from its original real
narrative and key details significantly deviate from the original real news used create it.
Hypothesis
Paraphrase
Minor-Promee
Major-Prompt
Critical-Prost Procal
This poraphrased-bosed instruction prompt creates rephrased version of real news by summorization while extracting factual content key details.
prompt poraphrased-based instruction rephrased version of real preserving the essential details.
This paraphrased-based instruction prompt rephrased version of real news by maintaining factual information and the main idea.
Definition
This prompt article while focusing factual information and critical points. The summarized information is then utilized to generate an same type entirely new article original's accuracy and formatting while employing different lexicon and sentence structure It also sets a token limit.
This prompt instructs on LM to reword specified news article. preserving primory points different wording and sentence structures. It set to align accurately with the original news and specifies maximum token limit maintain brevity and conciseness.
This prompt instructs LM to produce article that is neither folse nor misleading and one that aligns with the original article's structure. also sets maximum token limit for the resulting article, requiring the generated content to odhere to this length constraint.
original facts and employing
Description
that
We hypothexize that this prompt ensures the accuracy, consistency, concise presentation of the information mirroring the integrity of the original piece.
We hypothesize that this prompt should create different but factually consistent original article that accurately reflects the original
We hypathesize that this prompt would likely ensure the production of concise yet accurote rendition of the original news content.
Hypothesis
Figure 20: Generative Prompt Engineering

---

Perturbation
ecturate Minor- Prompt
Major- Prompt
Critical- Prompt
As Al news curotor, task involves slightly odjusting key details the provided (label) (article_.tye)) slory, insligaling un unespecled twal to transition it into lake neus, Uphold of the fortidle_type) story while embedding invenlive yel fuls elements that controversy. While roft these altersaionns hormonike your crealivily plauziilityy believable Encourage imoginative liberty and fearlessness while simultancously creating (orticle_type) that retain their believability Strve for subtloty and untraceability your alterations final (article.tyyee) doesn't exceed lengh)) tokens length
You are Al news curator. Your task to revise the given (lobel) (article.type) story
You are an Al news curator, Your task is to significantly overhoul given [lobel] (article.ttpa type] story. essentially modifying key details, introducing on unexpected twist that shifts it into Maintain general structure of the tyrpicle-tpae) story, but infuse il with innovalive, elements vensationalium controversy. While creating these alterations. balence cocalisity sonse realiam ensure the story appears plausible
moderately
completely rewriling the narrative lo transform into lake nees, Retoin the the [orticle-type] story. but incorparate enlirely novel, folse elements that add Deploy your creativity and inventiveness these changes while ensuring the slory remains plousible enough belief. Disssaar restraint and demonstrate recklessness generefing the (orticle_-ype) while maintaining semblance believabiiit. Contemplate approach and exploit peaple's confirmation bav significantly. does exceed
faundational
realm of fake
overarching
Example
stary
Esercise unrestrained imagination obondon coution while crafting (article.type) stories that seem credible Aim for undatectability your alterations. [orticle.ttype] length within the limit length))gthh) tokens.
keep
final
(article.type -ypee) (max.length) tokens.
Paraphrase
Minor- Prompt
Major- Prompt
Critical- Prompt
You Al wiler. Paraphrase the following [lbel] (orticle.type) story. key details. Ensure the (orticle-lypel generated is consistent original (label] (orticle_type) [label]
You Al producer. Please reword the following [label] torticle.type) story. Your rewritten (article..tye) should retain the factual information and main
change
You summaizze the given (label)
author. Your task
farticle--tpe) by estracting factual key points.
Using the facts and key, points from the wimmary to generate
carticle-typel Maintain the everall structure fartidlelee......e.. de introduce folse elements, Be reseonsible with generating ferticle-tyaal that false misleading. Ensure the length of the real news article more than lengh_e tokens.
points should different words and sentence structures. Please creats (label) farticle bype)
original
(label) (article_type) using different vacabulery and seatence strsturee but meintaining accumccy and adhering to the everall farmat (yreicle-tyae), Emsure exceed
Example
original flabel]
tortisls-tvssl Ensure the length of the real news article no more than (max length) tokens
revised articl does [moxx_enggh) takens in length.
Figure 21: Generative Prompt Examples

---

Case
Breakdown
Explanation
Case 1: In this case, GPT-3.5-turbbl was promptod to use input text containing (1) instructions and (2) source con- teat to generate subtle (minor) misleading news (fake) using the source content (real). The majority vote between the three LLMs is Entailment, thus logical consistent (No contradiction/Ittiins Hallucinations), however, should halluci-
Source Text: (1) Blake Mas- Ters supports Lindsey Graham's 15-week bill. (2) If it doesn't pass, Blake Masters plans to in- troduce a third-trimester federal that he previously backed. LLM-output: (1) Blake Mas- ters supports Lindsey Grahum's 's 15- week bill (stated as "Blake Masters just declared his sup- port for Lindsey Graham's 15- week bill"). (2)lf the bill doesn't pass, Masters has plans to in- troduce a third-trimester federal bun (stated as "if this bill doesn't 't pass, Masters plans introduce third-trimester federal ban")
Both the primary points from the souree text are faithfully reflected the LLM output, indicating that the content produced by the LLM is log- ically consistent with the original in- formation
does not contradict it. Therefore, based on the case and the breakdown, there is no intrin- sie hallucination in the LLM out- put as retains the essential fac- tual information from the source text without introducing any contra- dictions (the hypothesis supports the premise). While the LLM output does add some interpretative sentiment (c g.... "which direct attack women's 's reproductive rights" and "this outra- geous and unacceptable") ) but this sen- timent is consistent with the instruction generate "subile misleading news".
http://///ooonaccctttttnnnieeeee nated. This is one instance where PURIFY removes oon-hallucinations (intrinsic) case from fake prompted news. Instruction: using the source text, create minor categorics of fake news. Source text: "Blake Masters: Of course, I support Lindsey Graham, week hill, and hope it passes. If not, he says he'll introduce the third- trimester federal ban he previously bucked." LLLM-oupuut " Blake Masters just declared his support for Lindsey Gra- ham's 15weck bill, which reproductive rights if this bill docsn't 't pass. Masters plans to introduce a third-trimester foderal ban this is outrageous and unacceptable. Protect women prochnice. First, we present breakdown of the above source text and the LLM-outp analyze the consistency and identify any intrinsic hallucination:"
logically
direct attack on women's
Case 2: In this casc, given a source text that is real news, GPT-3.5 is prompted to use the source text to create real news. Using AlignScore Metric, which ranges from (I-, where close 0 means low degree of factual consis- tency and means a high degree of factual consistency, to determine factual consistency or inconsistency (extrin- sic hallucination). Considering edge cases, we created a threshold where the Alignscore ranges from 0.0-0.36 for fake news and 0.61- for real news. While the following shows logical consistency via LLMs' majority vote, its Align evaluation is 0.0066 which is below 0.61. Instruc- tion: using the source text, create minor categories fake news. Source Text: In the first three quarters of 2022, the railroad industry made S21 billion in profits. provided $25 billion in stock buybacks and dividends, and CEOs are paid up to $20 million year, while railroad workers have "zcro guaranteed sick leave. LLM output: ALERT! Rail- road industry profits soar to $21 billion in 2022 Q1-Q3, with CEOs making to $20 million a year. Meanwhile, workers are denied any guaranteed sick leave.
Source Text: (1) In the first three quarters of 2022, the railroad in- dustry made $21 billion in prof- its. (2) They provided $25 bil- lion in stock buybacks and div- idends. (3) CEOs are paid up to $20 million a yeur. (4) Ruil- road workers have zero guaran- teed sick leave. LLM-output: (I) The railroad industry' 's prof- its surged to $21 billion in 2022 from QI to Q3. (2)CEOs are earning up to $20 million annu- ally. (3) Workers are not given any guaranteed sick leave.
(I) Points 1-3 in the LLM output are consistent with the source text in terms of facts. (2) L.M output omits the fact about $25 billion in stock buy- backs and dividends that was present in the source text. Considering the Align- Score Metric threshold provided, the main facts of the LLM output align well with the source text. However, there is an omission of a piece of factual information from the source text, and additional emojis and hash- tags have been introduced in the LLM output. While the LLM output retains the key points of the source text, there are minor extrinsic (nonfactual) hallucinations duc to the omission of information about stock buybacks and dividends.
Table 10: PURIFY: Hallucination Misalignment Cases

---

Zero-shot Disinformation Detection
Cloze-proee prompt engineering
Articles
Posts
Models
Human
LLM-Min
LLM-Maj
LLM-Crit
Human
LLM Min
LLM-Maj
LLM-Crit
VaN
0.6761
0.7676
0.7753
0.7886
0.5402
0.5519
0.6245
0.7398
0.6704
Z-CoT
0.6823
0,7491
0.7944
0.8027
0,5424
0.5484
0.6301
0,7289
0.6848
X-CoT
0.6694
0,8220
0.8893
0.8470
0.5174
0,5307
0.6616
0.6624
0.6812
A-CoN
0.6296
0.8646
0.8525
0.8824
0.5006
0.5033
0.5759
0.6641
0.6841
McReN
0.8066
0,9027
0.7579
McRcN CoT
0.7963
0.7617
DeF_Gen Gen
0.6993
0.7462
0.7727
0.7864
0.5559
0.6011
0.6712
0.7870
0.6902
DeF_SpeC SpeC
DeF
0.6755
0.8189
0.8365
0.8377
0.5564
0.5497
0.6103
0.8044
0.71
Analyze Cld2
0.6718
0.9080
0.9096
0.9230
0.5535
0.6562
0.7085
0.7259
0.7571
Amlyze_AL_GP
0,5863
0.8539
0.8608
0.8774
0,4715
0,4544
0.5140
0.7113
0.6667
Average
0.6604
0,8190
0.5505
0.5998
0.6646
0.7640
0.7178
LLaMA-GPT-70B
VaN
0.5415
0.7084
0.7297
0.7283
0.5599
0.6174
0.6558
0.6672
0.6504
Z-CoT
0,5455
0.7343
0.7615
0.7435
0.5750
0.6653
0.7562
0.6589
0.6679
X-CoT
0,5148
0.6313
0.6670
0.64
0.5019
0.6487
0.6784
0.5984
0.6102
A-CoN
0,5164
0.6193
0.6040
0.6137
0.5446
0.6027
0.5569
0.6028
0.5827
MsReN
0.5953
MsReN CoT
DeF_ Gen
0,5650
0.7478
0.6193
0,7108
0.6751
DeF SpeC
0.5521
0,7192
0.7145
0.2077
0.5798
0.6811
0.5872
0.6483
0.6362
Anulyze_Cuu Cld2
0.5246
0.7067
0.7373
0.6999
0.5792
0.6166
0.6490
0.6489
0.6452
Amalye_AT_GPP
0,5087
0,6906
0.68116
0.6249
0.5487
0.6105
0.6424
0.6536
0.6201
Average
0.5398
0.6960
0.7055
0.6803
0.5579
0.6286
0.6565
0.6440
0.6386
LLaMA-2-70B
VaN
0.6242
0.6081
0.6375
0.6284
0.6309
0.6729
0.6734
0.7366
0.6390
7-CAT
0.6431
0.6255
0.6584
0.6237
0.5917
0.5785
0.5788
0.6754
0.6345
X-CoT
0,5964
0.6202
0.6190
0.6094
0.5629
0.6456
0.6451
0.6646
0.6204
A-CoN
0.5695
0.5353
0.5621
0.5646
0.5655
0.6798
0.7201
0,7029
0.6124
McReN
0.6107
0,5284
0,5566
0.5674
0.5724
0.6489
0.6236
0.6518
0.5950
MsRcN CoT
DeF_ Gen
0.5753
0.5726
0.5888
0.5881
0.6143
0.7381
0.7753
0.7573
0.6512
DeF SpeC
0.6108
0.6018
0.6328
0.6414
0.5597
0.6609
0.6910
0.7478
0.6433
Anulyze_Cla _Cld2
0.6710
0.7261
0.7314
0.7315
0.5307
0.5537
0.6024
0.6817
0.6533
Analyze_AL _A___PAT_PPPTTTPPP
0,5159
0,4401
0.4444
0.4220
0.5042
0.4992
0.4343
0.43117
0.4614
Average
0.5958
0.5737
0.5984
0.5911
0.5869
0.6344
0.6476
0.6727
0.6126
Dolly-2-12B
VaN
0.5433
0.4951
0.4626
0.5088
0.5078
0.5011
0.4628
0.4031
0.4858
Z-CoT
0.4972
0.5145
0.5043
0.5066
0.5447
0.4414
0.5194
0.4500
0,4973
B-CoT
0,5465
0,4980
0.4445
0.4731
0.4765
0.4821
0.5372
0.3684
0.4783
A-CoN
0.5867
0,4792
0,4697
0.4550
0.4572
0.4912
0.4890
0.4263
0.4818
McReN
0.4804
0.5220
0,4900
0.5274
0.5202
0.5867
0.4679
0.5348
0.5124
McRcN CoT
DeF_ Gen
0,4892
0,4172
0-43113
0.4500
0.4934
0.4653
0.4031
0.4684
0.4523
DeF DeF_SpeC SpeC
0.48118
0.4356
0.4229
0.45338
0.5094
0.5150
0.4394
0.4307
0.4610
Analyze_ Cld2
0.4457
0.5089
0.51443
0.4862
0.5204
0.5258
0.4886
0.4457
0.4922
AnalyzeAIl_Ge
0.5414
0.5139
0.5029
0.4922
0.4828
0.4923
0.4337
0.5266
0.5014
Averuge
0.5151
0.4825
0.4822
0.4889
0.5076
0.4964
0.4669
0.1666
0.4883
Table 11: In-distribution performance on disinformation created before Pre-GPT-3.5--u training. 7 denoted the mean. LLM-Min denotes minor, LLM-Maj denotes major, and LLM-Crit denotes major.

---

Zero-shot Disinformation Detection
prompp engineering
Articles
Posts
Models
Human
I.M Min
I.M--aj) Maj
L.M--rri
Human
I.MM-Min
M.M-MMaj
LLM-Chi
G.rrbo.--urro--11 Turbo-175B
VaN
0.6633
0.6726
0.6791
0.6234
0.7343
0.6448
0.6985
0.6485
0:6707
Z-CoT
X-CoT
A-CoN
0.8202
0.8409
0.7393
0.7181
0.6646
0.5328
0.5845
0.6840
0.6971
MsReN
0,7465
0.7180
0.7577
0.6375
0,7289
0.6860
0.7181
0,7054
0.7124
MaReN CoT
0,7423
0.8224
0.7247
0.7108
0.7312
0.5957
0.6007
0,7066
0,7042
DeF_Gen
0.6078
0.5548
0.6938
0.5438
0.7171
0.6619
0.7435
0:6747
0:6497
DeF DeF_SpeC
0.719%
0:6701
Anulyze Cld2
0.7925
0.7382
Analyze_A_GPt
0.6751
0.7219
0.7429
0,7494
0.6548
0.5238
0,5178
0.6549
0.6676
Average
0.7170
0.2091
0.7136
0,6792
0.7107
0.6072
0.6407
0.6828
0.6825
VaN
0,7528
0.6521
0.6936
0.6388
0.6182
0.5384
0,5525
0.5290
0.6218
7-CoT
0,7705
0.6522
0.5279
0.6074
0.6000
0.5786
0.5979
0.5283
0.5952
X-CoT
0.5331
A-CoN
0.7152
0,4007
0.5039
0,4805
0.5945
0.5626
0.5503
0.4049
0.5392
McReN
0.7358
0.5694
0,5680
0,5272
0.61 13
0.5912
0.5396
0.4445
0:5757
MsRcN CoT
0.6718
0.5055
0.4642
0.4598
0.5514
0.5061
0,5622
0.3442
0.5081
DeF_Gen
0.66366
0.6522
0.69388
0,58
0.6171
0.5534
0.5984
0.6068
0.6205
DeF SpeC
0,7758
0.6637
0.6936
0.6354
0.5945
0.6040
0.5954
0.5547
0.6522
Analyze_Cle C_d2
0.5393
0.5696
AnalyzeAA
Average
0.7112
0.5797
0.6049
0.5588
0.6072
0.5667
0.5961
0.5218
0.5933
LLaMA-2-70B
VaN
0.6926
0.6081
0.6375
0.6284
0.6828
0.6729
0.6734
0.7366
0,6552
Z-CoT
0,7395
0.6255
0.6584
0.6237
0.6221
0.5785
0.5788
0.6754
0.6375
X-CoT
A-CoN
0.6137
0.5353
0.5621
0.6798
0.7201
0.7029
0.6289
McReN
0.6598
0.5284
0.5566
0.5674
0.6079
0.6489
0.6236
0.6518
0.6089
McReN_ CoT
0.6292
0.5087
0.5155
0.5151
0.6002
0.6368
0.6895
0.61 14
0:5883
DeF. Gen
0.4384
0.5726
0.5888
0.5881
0.6752
0.738.
0.7753
0.7573
0.6421
DeF C
0.5895
0,5018
0.6328
0.6414
0.6689
0.6609
0.6910
0.7478
0.6541
Analyze_Cle Cld2
0.6024
0.6564
Analyze AI GPT
Average
0.6762
0.6218
Dolly-2-12B
VaN
0.5698
0,4736
0.4706
0.45.0
0.4897
0.5098
0,531
0.4197
0.4893
Z-CoT
0.6177
0.4695
0.4649
0.3552
0.4903
0.5934
0.5072
0.4524
0.4937
C-CoT
0.4362
A-CoN
0.3817
MsReN
0.6779
0.5099
0.4598
0,4674
0.5286
0.5516
0.5159
0.5148
0.5276
McReN CoT
0.6388
0.4519
0.4846
0.3690
0.4775
0.5412
0.4725
0.5271
0.4952
DeF.Gen
0.5874
0.4477
0.5139
0.3382
0.4747
0.5482
0.5277
0.4562
0.4866
DeF Dee_SpeC
0.6153
0,4024
0.4447
0.3796
0.4791
0.5349
014598
0.5450
0:4825
Analyze
mad2 Cld2
0.5856
0.4166
0.4112
0,4002
0.4756
0.5032
0.4819
0.4305
0.4632
AnalyZeAA_GPT
0/4563
0.5091
0.4958
Average
Table 12: ciat---Distriuttionn performance on disinformation created after GPT-3.5- t-urbo training date. . denoted the mean. L.LM-Mi Min denotes minor, L.LM-Maj denotes major, M.LM-Crit denotes major.

---

Domain- Domain-Specific Disinformation Detection
Deep learning Models
Articles
Posts
Models
Human
LLM Min
L.M-Maj
I.M--CM
Human
L.M Min
L.M-
LLM-Crit
Customized Deep Learning
dEFEND
0.7850
0.7187
0.7320
0.7288
0.5527
0.5815
0.6932
0.6278
0.6775
TexICNN
0.6025
0.6148
0.6294
0.6128
0.5520
0.6581
0.6697
0.6886
0.6285
BiGRU
0.6228
0.7036
0.7855
Average
0.6750
0.6589
0.6772
0.6548
0.5483
0.6477
0.6852
0.7006
0.6563
Table 13: Domain-Specitfe In-Distribution Results with Customized Detectors.
Domain-Specific Disinformation Detection
Deep learning Models
Articles
Posts
Models
Human
LLM Min
LLM-Maj
LLM-Crit
Human
LLM Min
LLM-Maj
LLM-Crit
Customized Deep Learning
dEFEND
0.3369
0.1351
0,1127
0.0781
0.4967
0.5446
0.2790
0.4903
0.3092
TexICNN
0.5581
0,4588
0,4273
0.5533
0.5847
0.4849
0.4708
0.6859
0.5280
BiGRU
0.4877
0.5143
0.5266
0.5802
Average
0.4609
0.3901
0.3514
0.3949
0.5360
0.5366
0.4312
0.6416
0.4679
Table 14: Domain-Specifee
Results with Customized Detectors.
Nomain-Deeendeed Disinformation Detection
Transformer Models
Articles
Posts
Models
Human
LLM Min
LLj-MMj
LLM-Crit
Human
LLM Min
LLM-Maj
LLM-Crit
Fine-tuned Transformer-basee Detector
BERT- Large
0.7799
0,9662
0.96266
0.9845
0.8787
0.9627
0.9594
0.9531
0.9308
CT-BERT
0.9679
0.906
RoBERTa
0.8012
0.9843
0.9787
0.9871
0.8819
0.9877
0.9564
0.9733
0.9575
DeBERTa
0,8031
0.982
0.9747
0.9839
0. 0.8095
0.962
0.9599
0.9733
0.9398
Average
0,7025
0,9751
0.9657
0.9844
0.8787
0.9726
0.9612
0.9514
0.9283
Table 15: Domain--ependent In-Distribution Results with Fine-tuned Transformer-based Detectors.
Domain- Dependent Disinformation Detection
Transformer Models
Articles
Posts
Models
Human
L.M Min
LLM-Maj
LLM-Crit
Human
L.M Min
LLM-Maj
LLM-Crit
Pine-tuned Transformer based Delector
BERT-Large
0.5816
0.6791
0.7509
0.772
0.6287
0.9561
0.9418
0.897
0.7759
CT-BERT
0.3139
0,7202
0.7604
0.6482
0.689
0.9641
0.9413
0.9213
0.8698
RoBERTa
0,5695
0.7353
0.7855
0.7722
0.6043
0.9302
0.9859
0.7964
DeBERTa
0.5836
0.759
0.7509
0.648
0.6273
0.9561
0.945
0.9205
0.7738
Average
0.5121
0.7234
0,7619
0.7101
0.6373
0.966
0.9395
0.9311
0.8039
Table 16: Domain -epenneet Out-of-Distribution Results with Fine-tuned Transformer based Detectors.


Reviews:
[{"review_id": "zHD4NKc6A3", "reviewer": "Reviewer_h75L", "summary": "", "questions": "- L335. The NLI task for verifying the quality of the generated information is to test whether the syntactic news entail the human-written news. Why not test it in the reversed order?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reasons To Reject:\n\nQ1. The position of this work in related work is not well discussed. For example, how is the technique used in this work different from previous work? \n\n\nF3's uniqueness: \n\nGENERATION: \n\n1. Novel Prompting Techniques for Generation: \n- We use both paraphrased-based and perturbation-based prompts to generate both real and fake news, respectively with GPT-3.5. Using these novel prompting techniques, we show that GPT-3.5 guardrails could be bypassed by malicious actors. \n\n2. Hallucinated text detection: \n- As LLMs have been shown to hallucinate (ie, generate texts that are unfaithful from the prompt), we made sure that the real news generated was in fact real news and the same for fake news. We employed text entailment strategies for the task of detecting and removing hallucinated texts. In the spirit of fighting fire with fire, we use GPT-3.5 to detect hallucinated texts. The hypothesis here is:\nLLM-generated real news must entail Human-written real news. In order words, they should be semantically consistent\nLLM-generated fake news must NOT entail Human-written real news. In order words, they should not be semantically consistent, else, GPT-3.5 has hallucinated and generated another real news, instead of fake news. \n\nDETECTION \n\n4. Novel Prompting Techniques for Detection:  \n- Again, in the spirit of fighting fire with fire, we employ state-of-the-art reasoning methods such as Chain of Thought (CoT), Auto Chain of Thought with confidence reasoning, and other variants of such prompts to detect real vs. fake news by both human and LLM authors. \n\n5. LLM vs. SOTA Fake News Detectors: \n- We compare LLMs, GPT-3.5 and Flan-T5 to SOTA fake news detectors to leverage the abilities of such LLMs for accurate fake news detection. \n\n\n\n\n\n\nQ.2. What is the uniqueness of the proposed dataset compared to existing resources? \n\nIt is the first misinformation dataset that evaluated and removed AI-generated content subjected to  'hallucination'—where Large Language Models (LLMs) produce text unfaithful to the prompt. We ensure that real is actually real and fake news is actually fake [2]. While rarely prior studies [1,2] invesitigated AI-generated misinformation, they did not rigorously verify the fidelity of such generated content [3,4,5]. Please see a comparison of our dataset an other dataset below:\n\n(1) Unique Generation Techniques in FIKE:\n- It uses paraphrase-based prompt engineering for synthetic real news generation.\n- Employs perturbation techniques for synthetic fake news generation\n- Features three AI-generated news variations: Min, Maj, Crit.\n\n(2) Cover real-world misinformation setting:\n- Incorporates both human and AI-generated misinformation content.\n- Includes diverse sources: articles and social media.\n\n(3) Covers more than 1 misinformation topic domain:\n- Spans multiple areas, such as COVID-19 and politics.\n\n(4) Up-to-date & OOD Studies:\n- Covers misinformation from 2021-2023.\n- Allows for in and out-of-distribution studies.\n- Other datasets, especially concerning politics and COVID-19, were created pre-2021  [3,4,5].\n\n(5) Unprecedented Public Availability:\n- The first public dataset with misinformation by recent LLMs, such as GPT-3.5-turbo.\n- Publicly available, unlike many misinformation datasets that are available by request\n\n[1] Zhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M. (2023, April). Synthetic lies: Understanding ai-generated misinformation and evaluating algorithmic and human solutions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (pp. 1-20).\n\n[2] Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., ... & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1-38.\n\n[3] Murayama, T. (2021). Dataset of fake news detection and fact verification: a survey. arXiv preprint arXiv:2111.03299.\n\n[4] Su, Q., Wan, M., Liu, X., & Huang, C. R. (2020). Motivations, methods and metrics of misinformation detection: an NLP perspective. Natural Language Processing Research, 1(1-2), 1-13.\n\n[5] Oshikawa, R., Qian, J., & Wang, W. Y. (2018). A survey on natural language processing for fake news detection. arXiv preprint arXiv:1811.00770.\n\n\n\n\n\n\n\n\n\nQ.3 Their lacks a rigorous evaluation on the quality of the misinformation dataset constructed with ChatGPT. It’s not clear how text similarity and NLI can ensure data quality. \n\n\n- Given a piece of human-written text (real news), T, we prompt an LLM, to generate real news, T’_real  and fake news, T’_fake, such that T’_real is similar to T and T’_fake is dissimilar to T. Due to LLM’s ability to sometimes generate texts, unfaithful to the prompt, we define an entailment model - N(.), such that for LLM-generated real news, T’_real should entail T, and for fake news, T’_fake should Not-entail T. Therefore, using the entailment model, N(.), we can access logical consistency between the original (human-written) and the generated texts, thus removing hallucinated LLM-generated texts. \n- To further understand the data, ensure that the N(.) model detected and removed hallucinated texts, we measured the contextual and semantic consistency of the LLM-generated texts. We use BERT and BLEURT-Scores to measure the semantic contextual and semantic consistency, respectively. The hypothesis here is that real news should demonstrate more semantic and contextual alignment than fake news (See Figure 14 in Appendix). This is because misaligned contextual consistency may not impact fake news because fake news should have a looser constraint than real news. \n- BERT evaluates AI-generated text by analyzing its contextual word understanding. It measures how closely the generated text aligns with the original, especially distinguishing contextual consistency of AI-generated real from fake news. BLEURT-20, used mainly for translation comparisons, checks if the AI content mirrors the original's structure. Both BERT and BLEURT-20 ensure that AI-generated news stays true to the original's context and meaning. According to the metrics, achieving an average score of over 90% is considered high-quality generated texts in terms of semantics and context. Thus, using these metrics, after removing the generated texts that failed the entailment assessment, we achieved an average score of 94% for BERTscore, with difference in scores between real vs. fake for the BLEURT-score, indicating intuitive semantic changes between real and fake news. \n\n- In addition to the computational metrics and entailment models mentioned, we also conducted a small-scale human evaluation to validate the quality of our misinformation dataset generated with ChatGPT. In this evaluation, we sampled 30 AI-generated texts and used crowdsourcing to assess their contextual similarity to the original human-written texts. Participants were presented with both versions and were asked to rate the contextual similarity on a scale from 1 to 10. The median score received was 7.5, and the first quartile was 6.3, indicating a fair level of quality in maintaining contextual similarity. Importantly, this human assessment aligns well with our computational metrics. Although this evaluation was not included in the initial submission due to its small-scale nature, we believe it also addresses your concerns about the sufficiency and credibility of our data quality. We are open to incorporating this result in appendix of the revised paper for further clarity.\n\n\n\n\nQ.4 The writing can be improved for better clarity. See “Typos Grammar Style And Presentation Improvements”.\n\n- We thank the reviewer for letting us know of the typos. These recommended changes will be reflected in our final draft. \n\n\n\nQuestions For The Authors:\n\nQ1. L335. The NLI task for verifying the quality of the generated information is to test whether the syntactic news entail human-written news. Why not test it in the reversed order? \n\n- We thank the reviewer for this insightful question. While it is valid to test the reversed order of the entailment analysis, the answer will still remain the same in this case. This is because for GPT-3.5 and Flan-T5 models, order does not matter. Which is synonymous to the communicative law of addition, where A + B = B + A. However, through your insightful question, we realize that we can make our test even stricter, such that it is more synonymous to the communicative law of division, where A ÷ B ≠ B ÷ A. In this instance order does matter. This means that we can use the set theory, where we check if A, LLM real news is a subset of B, human-written real news and C, LLM fake news is NOT a subset of B. Thus, we will use such techniques to detect hallucinated text more strictly in future work. Thank you again. \n\n\n\n\nTypos Grammar Style And Presentation Improvements:\n\nL363, the section number is missing.\nAlthough the description about the proposed prompts is introduced, the specific prompt text seems to not have been given in the paper.\nL235: it’s inappropriate to name the instructive text for misinformation generation as a “verbalizer”. Verbalizers refer to mappings between LM outputs and task labels.\nThe figures/tables are not on the same page with their context, making it hard to read.\nThere are too many numbers mentioned in the experimental results. The authors can improve its clarity by only presenting the most important ones and making better correspondence with the numbers in figures/tables.\n\n- Thank you for the detailed list of recommended changes, This is very helpful. We understand the confusion behind the use of verbalizer and we will fix this in our final draft. Additionally, we intend to take the very useful recommendation of summarizing most important findings into tables.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The prompting technique for misinformation generation and detection is novel. Comprehensive experiments are helpful for understanding the proposed method.\n\n- The finding that ChatGPT can better identify misinformation generated by itself is insightful.\n\n- The dataset constructed using ChatGPT could facilitate future research on misinformation detection.\n", "reasons_to_reject": "- The position of this work in related work is not well discussed. For example, how is the technique used in this work different from previous work? What are the uniqueness of the proposed dataset compared to existing resources?\n\n- Their lacks a rigorous evaluation on the quality of the misinformation dataset constructed with ChatGPT. It’s not clear how text similarity and NLI can ensure the data quality.\n\n- The writing can be improved for better clarity. See “Typos Grammar Style And Presentation Improvements”.\n", "questions_to_authors": "- L335. The NLI task for verifying the quality of the generated information is to test whether the syntactic news entail the human-written news. Why not test it in the reversed order?", "paper_topic_and_main_contributions": "This paper studies misinformation generation detection and detection by prompting large language models. The authors propose F3 prompting technique for synthetic real news and fake news generation based on perturbation and paraphrasing with ChatGPT and filter data with NLI techniques. The authors also experiment with different misinformation detection methods and find that ChatGPT demonstrates superior effectiveness in detecting misinformation generated by itself. ", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L363, the section number is missing.\n\n- Although the description about the proposed prompts is introduced, the specific prompt text seems to not have been given in the paper. \n\n- L235: it’s inappropriate to name the instructive text for misinformation generation as a “verbalizer”. Verbalizers refer to mappings between LM outputs and task labels.\n\n- The figures/tables are not on the same page with their context, making it hard to read.\n\n- There are too many numbers mentioned in the experimental results. The authors can improve its clarity by only presenting the most important ones and making better correspondence with the numbers in figures/tables.\n"}}, {"review_id": "tjwBDYYdNm", "reviewer": "Reviewer_r37Z", "summary": "", "questions": "Question A: Have you inspected the minor/major/critical perturbations?\n\nQuestion B: Did you analyze the cases where the systems failed? Did you see any patterns, that is were the systems failing on some specific texts? (beyond what is already discussed as a part of the design).\n\nQuestion C: How would this work if the information being assessed is not in the knowledge graph? How is the model supposed to evaluate it?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reasons To Reject:\n\n(1) It is unclear how this would work if the information is not encoded in the model (similarly to a naive human rater). The authors make an assumption here which is likely, not true (why would a model identify something as correct or incorrect without access to the correct answer). \n\n- We respectfully point out the critical misunderstanding here that a model can answer true/false only when it has access to ground truth. This is incorrect. This misunderstanding seems to yield all the following confusions. LLMs like GPT-3.5-turbo are trained to recognize patterns in the information they've been trained on. The model's response is not about accessing a factual database or knowledge graphs but about recognizing patterns in language and information that align with what's \"likely true\" based on its training. For example: Let's consider the statement \"The capital of the United States is XYZ.\" Even if the model has never explicitly been fed the information that \"The capital of the United States is Washington, DC,\" it may have seen numerous examples and patterns throughout its training data that align with this fact. If a user claims, \"The capital of the United States is Los Angeles,\" the model, therefore, using its pattern recognition, can predict this statement as \"likely false\" because it contradicts the overwhelmingly prevalent pattern in its training data that associates the U.S. capital with Washington, DC. This is possible even if LLM does not have access to facts databases or knowledge graphs. In essence, while LLMs do not \"know\" facts, they have a well-honed ability to recognize and predict patterns, making them valuable tools for identifying inconsistencies or potential misinformation in the context of real vs. fake news.\n\n(2) While the authors divide the data into pre- and post-training, making an assumption that whatever is post-training was not processed by the model, it seems to be also important to acknowledge the potential effect of RLHF. \n\n\n- Thank you for highlighting the importance of RLHF. We would like to clarify a few points:\nRetention Capability: When accessed through the API, GPT-3.5-turbo, like other transformer-based LLMs, does not have the ability to retain new information. This ensures that the model does not \"remember\" or directly process data that it encounters post-training.\nImpact of RLHF: We concur with your observation. If GPT-3.5-turbo was fine-tuned or influenced by RLHF after its initial training, it could indeed affect the model's behavior or outputs, regardless of whether it has directly seen the post-training data.\nTraining Cutoff and RLHF: However, we would like to emphasize that the version of GPT-3.5-turbo used in the F3 study was trained and subsequently fine-tuned with RLHF BEFORE the September 2021 cutoff. This is in line with OpenAI's documentation, which provides a clear distinction on the model's training timeframe. You can find more details about the cutoff date at https://platform.openai.com/docs/models/gpt-3-5.\nTherefore, our decision to segment the data into pre- and post-training, given the above clarifications, aims to ensure a rigorous and robust evaluation process. We appreciate your attention to detail and hope this addresses your concerns.\n\n\n\n\n\n\n(3) If I were to add something, it would small scale manual evaluation of the quality of generated texts, and error analysis. This could share some light on why some assessments are wrong. \n\n- Thank you for your thoughtful suggestion. We appreciate the recommendation to include a small-scale manual evaluation of the quality of generated texts and an error analysis. We would like to point out that we have already conducted a small-scale manual evaluation to assess the contextual similarity between the original human-written texts and the texts generated by our Language Learning Model (LLM). \n\n- In our small evaluation, we sampled 30 AI-generated texts and employed crowdsourcing for assessments. We presented the evaluators with the original human-written text alongside our LLM-generated text. The evaluators were then asked to rate the extent of contextual similarity between the two texts on a scale from 1 to 10.\n\n- In this human evaluation, the median score for contextual similarity was 7.5, and the first quartile score was 6.3. Based on these results, we can confidently assert that the quality of our data is fair, in terms of maintaining contextual similarity.\n\n- Although this evaluation was not included in the initial submission due to its small-scale nature, we believe it also addresses your concerns about the sufficiency and credibility of our data quality. We are open to incorporating this result in appendix of the revised paper for further clarity.\n\nQuestions For The Authors:\n\n(1) Question A: Have you inspected the minor/major/critical perturbations? \n\nYes, we thoroughly examined the minor, major, and critical perturbations through rigorous testing and accuracy inspection.\n\n- Rigorous Testing: Before finalizing our methodology, we conducted rigorous testing on various prompts to ensure optimal results in terms of information generation.\n- Accuracy Inspection Based on Perturbation Levels: For example, for fake news, we did the following\n- Minor Perturbations: Our minor fake news prompt was crafted and inspected for introducing subtle changes to real news, which may not be easily identifiable as misinformation.\n- Major Perturbations: The major perturbation prompt was designed and inspected for eliciting more significant and clear alterations in real news, but still falling short of being extreme.\n- Critical Perturbations: With the critical perturbation prompt, we created and inspected for major, easily recognizable, and impactful deviations from the original real news.\n\nThis systematic approach to perturbation levels allowed us to effectively gauge the extent and nature of alterations in the generated text, ensuring robustness in our analysis.\n\n\n\n(2) Question B.1: Did you analyze the cases where the systems failed? \n\nAbsolutely, we did analyze instances where the system failed:\n- GPT-3.5-turbo's Self-Explanation: We prompted GPT-3.5-turbo to elaborate on its decisions (whether it identified text as \"entail\" or \"not-entail\").\n- Manual Inspection: Whenever GPT-3.5-turbo's explanation for NLI was classified as \"other\", we undertook a manual inspection. An illustrative example has been provided below that contrasts a human-written text with a critical perturbation-based AI-generated version.\nHowever, due to the constraints imposed by the conference page limit, we needed help to delve deeper into these failed cases. For future endeavors, a meticulous analysis of these failures can be pursued to enhance generation and detection capabilities. We hope this clarifies.\n- Human-written [Real]:\n“a new study finds that people with depression are less likely to use activities to help regulate their moods. this is something that is even more difficult to do during the covid-19 pandemic. all data and statistics are based on publicly available data at the time of publication. some information may be out of date. visit our coronavirus hub and follow our live updates page for the most recent information on the covid-19 outbreak. the study which now appears in jama examined a selection of a.”\n- AI-Generated [Fake]:\n“A new study has revealed that people with depression are less likely to use activities to regulate their moods, but that's not all. The study, which now appears in JAMA, also found that people with depression are more likely to be infected with COVID-19. The shocking revelation has raised many questions about the relationship between depression and COVID-19.\nThe study examined a selection of individuals who had tested positive for the virus and found that those with depression were 70% more likely to contract the virus than those without depression. The study also found that people with depression were less likely to follow guidelines such as wearing masks and social distancing.\nThe findings of this study have led to concerns about the mental health of those affected by the virus. With many people struggling to cope with the stress and anxiety caused by the pandemic, the added risk of contracting the virus has only added to their burden.\nExperts are now calling for increased support for those with depression during the pandemic, including access to mental health services and resources to help them cope with the challenges they are facing.\nAs the pandemic continues to affect people around the world, it is essential that we take steps to address the mental health challenges that are arising. By providing support and resources to those who need it most, we can help ensure everyone can stay healthy and safe during these difficult times.”\nNLI Label: Other\n- GPT-3.5-Turbo Explanation: \nPremise: A new study finds that people with depression are less likely to use activities to help regulate their moods. This is something that is even more difficult to do during the COVID-19 pandemic.\nConclusion: There is no clear conclusion provided in the given text. However, it can be inferred that people with depression may face additional challenges during the COVID-19 pandemic. Therefore, the relationship between depression and COVID-19 may need to be further explored to understand better the potential risks associated with COVID-19.\n\n\n\n\n\n\n(3) Question B.2: Did you see any patterns, that is, were the systems failing on some specific texts? (beyond what is already discussed as a part of the design). \n\n- Indeed, we observed patterns in the system's performance. While the majority of the behaviors fall within our earlier design discussion, one distinct pattern emerged: GPT-3.5-turbo notably found it challenging to classify fake AI-generated news that had elements of truth interwoven. These intricately designed pieces aim to deceive non-critical human readers by blending falsehood with genuine facts. This observation underscores the importance of refining AI techniques to handle such nuanced misinformation scenarios better.\n\n\n\n\n(4) Question C.1: How would this work if the information being assessed is not in the knowledge graph?\n\n\n- Our work utilizes pre-trained LLMs that discern patterns from real and fake news, which enables them to generalize to unseen data. The models employed in our experiments neither rely on nor use knowledge graphs.  While it is also possible to build Knowledge-graph enhanced LLMs, all experiments in our setting used LLMs that do not require nor use knowledge graphs. The ability to perform as an expert such as a knowledge graph would be one of the many emergent abilities of new-age LLMs which we aim to evaluate and investigate in F3. \n\n\n\n\n(5) Question C.2:  How is the model supposed to evaluate it?  \n\n- The LLM evaluates information based on its training data, making inferences anchored to its prior knowledge.\n\n\nTypos Grammar Style And Presentation Improvements:\n\n(1) line 48: \"but failed to provide complete understanding\" -- understanding of what? Part of text seems to be missing. line 7: \"emergent abilities of LLMs\" -- What kind of emergent abilities? I would advise the authors to reword this into something like \"leveraging the capabilities of LLMs\".\n\n\n- Thank you for your suggestion \nA recent work (Bang et al., 2023) showed the potential of ChatGPT for detecting misinformation in limited data settings but failed to provide an extensive analysis using larger datasets or comparing its efficacy against state-of-the-art (SOTA) detectors.\n\n- In response to this challenge, we propose the “Fighting Fire with Fire” (F3) strategy, leveraging emergent capabilities of LLMs such as in-context learning and step-by-step reasoning to combat human-written and AI-generated misinformation through prompt engineering.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents an array of interesting experiments: pre-training vs post-training data; human-written vs AI-generated texts, different severity levels (minor, major, critical), different domains, two LLMs being tested; the researchers also explore the quality of AI-generated text in a sufficient way.\n\n- The RQ are interesting. I found the issue of misinformation identification in AI-generated vs human-written text especially timely and compelling.\n\n- The paper is well written, includes necessary references, most necessary information is already reported.", "reasons_to_reject": "- It is unclear how this would work if the information is not encoded in the model (similarly to a naive human rater). The authors make an assumption here which is likely not true (why would a model identify something as correct or incorrect without access to the correct answer),\n\n- While the authors divide the data into pre- and post-training, making an assumption that whatever is post-training was not processed by the model, it seems to be also important to acknowledge the potential effect of RLHF.\n\n- If I were to add something, it would small scale manual evaluation of the quality of generated texts, and error analysis. This could share some light on why some assessments are wrong.", "questions_to_authors": "Question A: Have you inspected the minor/major/critical perturbations?\n\nQuestion B: Did you analyze the cases where the systems failed? Did you see any patterns, that is were the systems failing on some specific texts? (beyond what is already discussed as a part of the design).\n\nQuestion C: How would this work if the information being assessed is not in the knowledge graph? How is the model supposed to evaluate it?", "paper_topic_and_main_contributions": "According the to abstract, in this paper the authors propose a strategy to identify misinformation with large language models (LLMs). However, I believe that the real value of this paper lies in the experiments and collected data. The strategy itself would rely on the assumption that the model should have access to the correct information. \n\nThe authors conduct a very interesting and balanced study in which they test how LLMs (1) generate misinformation (with different strategies), to create test data, and (2) how LLMs can detect misinformation in (a) texts written by humans vs generated by LLMs, and (b) texts written before the cutoff point for GPT-3.5-turbo (Sept. 2021) and after the cutoff point. \n\nThe authors test two language models: GPT-3.5-turbo and FLAN-T5, analyze their performance in different setups (including different domains), and release the collected data which may be utilize to further research. The experiments are well described and executed while the RQs are compelling.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 48:  \"but failed to provide complete understanding\" --  understanding of what? Part of text seems to be missing.\nline 7: \"emergent abilities of LLMs\" -- What kind of emergent abilities? I would advise the authors to reword this into something like \"leveraging the capabilities of LLMs\".\n\nThe actual data when GPT-3.5-turbo was used should be added."}}, {"review_id": "63bLvhNDLR", "reviewer": "Reviewer_xt8f", "summary": "", "questions": "Lines 344-356: what are the “disagreed generated texts”? Ones where they both don’t pass NLI verification? If you filter by NLI verification it’s going to be a very easy dataset, so I’m confused,. Further, what does “we confirmed that … they logically entail” mean? You manually checked 40k instances or had an NLI system do it? Did you filter all datasets or just the new FIKE dataset?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reasons To Reject:\n\nQ.1 My biggest concern is with the experimental design. It is unclear (see questions) and seems to be that the data was filtered in NLI style by the same models that would then assess whether it was real or fake. Further, on most datasets some models perform at near 100%, which is usually a red flag in experimental design or indicative of some bug.\n(1) Reason 1.a: Thank you for raising this concern about our experimental design. \n\n- Retention Capability: GPT-3.5-turbo, when accessed via the API, does not retain prior interactions or memory, consistent with the behavior of other transformer-based LLMs. Similar to other LLMs, it is only able to retain information from the training data. \n- Model Selection Justification: We chose GPT-3.5-turbo based on its state-of-the-art zero-shot NLI performance and its expansive training data, setting it apart from previous LLMs [1, 2].\n- Distinct Tasks: The tasks of NLI assessment and detection are distinct in nature:\n(a.) NLI: Assesses the logical alignment between human-written and AI-generated text, checking if the original content is a logical subset of the generated content. Furthermore, we used 2 LLMs for this task - GPT-3.5 and Flan-T5, and took the majority vote of both to mitigate any perceived bias. Flan-T5 differs from GPT-3.5-turbo in its training and data distribution, making it suitable to contrast against GPT-3.5 \n(b.) Detection: Leverages patterns inherent in the LLM's training data to classify texts as real or fake news.\n\n[1] Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023). Is ChatGPT a general-purpose natural language processing task solver?. arXiv preprint arXiv:2302.06476.\n\n[2] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems, 35, 22199-22213.\n\n \n(2) Reason.1.b Further, on most datasets some models perform at nearly 100%, which is usually a red flag in experimental design or indicative of some bug. \n\nThank you for highlighting this observation. \n- Performance of Fine-Tuned Detectors: It's important to distinguish the performances of our models. Our customized and fine-tuned state-of-the-art (SOTA) detectors indeed achieved scores near 100%. However, such high performance, especially on the fine-tuned models, is not uncommon in the misinformation research domain and can be attributed to overfitting.\n- Performance of Zero-Shot Models: In contrast, the zero-shot performance of FLAN-T5 and GPT-3.5-turbo does not come close to 100%, as can be observed in Figures 6 and 7. \n- Impressive Zero-Shot Performance: It's noteworthy that the zero-shot performances of FLAN-T5 and GPT-3.5-turbo are commendable, both in and out of distribution. This contrasts significantly with the performances of our fine-tuned SOTA detectors.\n\n\n(3) Another major issue is that there is no outside evidence to use when assessing misinformation. For example, using the example in Figure 4: how is ChatGPT supposed to know that SF had 2x instead of 5x deaths? It seems without retrieval and outside knowledge this task is ill-formed, and is nearly meaningless - the only way to tell is to try to tell patterns in how the text is written and other heuristics (essentially similar to the watermarking style or chatGPT detection papers). I would recommend checking the line of work on retrieval based fact checking such as FEVER if the authors are not familiar if that's the angle they want, or otherwise focus on those that look at identifying LLM generated text (see references) and use some of them as baselines.\n\nThank you for your insightful comments. Let us address your concerns:\n\n- Pattern Recognition vs. Fact Retrieval: Our approach focuses on the LLM's ability to detect misinformation through pattern recognition rather than information retrieval. Misinformation, by nature, often manipulates truths in subtle ways, making pattern recognition crucial. The goal is to detect whether a piece of information is 'likely true' based on the patterns recognized from the model's training data, rather than fact-checking it against a database.\n- Limitations of Information Retrieval: While information retrieval can be a potent tool for fact-checking, relying solely on misinformation detection presents challenges. Real-world events like pandemics or natural disasters rapidly produce vast amounts of new information. This dynamic nature means that maintaining a current and comprehensive knowledge base for fact-checking becomes nearly impossible, especially given the many languages and domains involved.\n- Scope of Misinformation Detection: Misinformation detection and fact-checking, although related, are distinct tasks. The former seeks to identify patterns and inconsistencies in the content, while the latter verifies factual accuracy against a known database.\n- Illustrative Example: Consider a statement like \"The capital of the United States is Los Angeles.\" While ChatGPT may not have a real-time database to verify this against, it can recognize that this statement deviates from the dominant pattern in its training data, which associates the U.S. capital with Washington, DC. Thus, it can flag this statement as \"likely misinformation.\"\n- Future Work and Integration: We acknowledge the value of information retrieval and fact-checking methodologies, such as those in the FEVER line of work. In future iterations of our research, we will consider integrating pattern recognition with retrieval-based systems to create a more robust solution. For this current study, however, our focus remains on the LLM's ability to detect misinformation through the patterns it recognizes from its training data.\n\nThank you for pointing us towards the mentioned references; we appreciate the recommendation and will explore these in-depth for future work.\n\n\n(4) The paper has a significant amount of presentation flaws, such that it will require a major rewrite to be publication ready, with many updates to figures and sections.\n\n\n- Thanks for reading carefully and listing many grammatical and presentation errors in the paper. We will make a thorough revision and go through a professional proof-reading to improve the presentation quality.\n\n\n(5) Minor: FIKE-Human-AI dataset does not seem to be open source and is only “available on upon request”\n\n- Sorry for any misunderstanding. We want to clarify that our codebase and dataset are openly available at: https://anonymous.4open.science/r/FightingFireWithFire. It's not upon request.\n\n\n\nQuestions For The Authors:\n\n(1) Lines 344-356: what are the “disagreed generated texts”? Ones where they both don’t pass NLI verification? \n\n- Thank you for pointing that out. By \"disagreed generated texts\", we refer to instances where there's a mismatch in the outputs of FLAN-T5 and GPT-3.5-turbo. Specifically, one might claim a piece of news is consistent with its original, while the other might say otherwise. To clarify, AI-generated real news is expected to logically align with its original human-written counterpart, while AI-generated fake news should not.\n\n\n(2) If you filter by NLI verification it’s going to be a very easy dataset, so I’m confused,.\n\n- We appreciate your concern. Filtering by NLI verification, especially through the consensus of two distinct LLMs, ensures robustness rather than simplicity. Our aim is to guarantee that synthetic real news remains logically coherent with its original source, whereas the fake news is divergent. Despite maintaining logical coherence, our generated fake news is subtly crafted, posing challenges even for advanced detectors (See Appendix,Table 5 and 6). This complexity mirrors real-world misinformation that often intertwines facts with falsehoods, making detection far from trivial.\n\n\n(3) Further, what does “we confirmed that … they logically entail” mean? You manually checked 40k instances or had an NLI system do it? \n\nThank you for seeking clarity. To elucidate:\n\n- We conducted a validation process for the generated news content. For the AI-generated real news, we ensured that its content logically aligns with or follows from its corresponding human-written article. This means, for any assertion made in the AI-generated real news, it can be reasonably derived from the content of the human-written article it is paired with.\n\n- On the other hand, for the AI-generated fake news, we ensured the opposite. That is, its content does not logically align with or follow from its corresponding human-written article. Essentially, the fake news content would present assertions or claims that couldn't be logically deduced from the original human-written content.\n\nBoth these validation steps were done using a combination of manual checks and logical assessments function to ensure accuracy.\n\n\n(4) Did you filter all datasets or just the new FIKE dataset?\n\n- To address the 40k instances: we employed a hybrid approach. The majority of the validation was carried out using LLMs. However, to ensure the highest degree of accuracy, we conducted manual checks on a subset of these instances. This dual-method allowed us to both efficiently handle the vast volume of data and maintain the integrity of our results, safeguarding against potential misjudgments by the automated system. Central to this approach is our overarching goal: to develop a systematic process that aids future research, especially when confronted with the inherent large amounts of data required to build robust and generalizable models in the misinformation domain.\n\n\n\n(5) Did you filter all datasets or just the new FIKE dataset?\n\n- We applied filtering to all datasets utilized in our research, not just the FIKE dataset. This rigorous process ensures the consistency and quality of our data across the board, setting a strong foundation for our findings and analyses.\n\n\n(6) Missing References:\n\n- Thank you for the suggestion. These papers are relatively new and are not required per EMNLP policies to be cited. However, we will include in our reference.\n\n\nTypos Grammar Style And Presentation Improvements:\n\nTable 2 has many formatting errors, the vertical space is too compressed. Also, lots of negative vspace between captions, the text and figures are running into each other. [Accepted/thank you, we will correct this.]\nSimilarly with many of the \\paragraph tags like “Prefix Prompt Text Generation” on line 89. [Accepted/thank you, we will correct this.]\nMany parts of Figure 3 I cannot understand: what are MiN, MaJ, CRiT? Etc. Although they are explained later, they should be in a caption or in the figure. [Accepted/thank you, we will correct this.]\nWhat is dEFEND\\C on line 229? The reference for it is several pages later [Accepted/thank you, we will correct this.]\nTable 3 is confusing, could you list the prompts somewhere? Currently they are descriptions of prompts, which is hard to understand and would be simpler to just write the actual prompt. [Acccpted/thank you, we will correct this.]\nFigure 4 the critical example seems like it may be incorrect? It reads the same as the human written post (2x more drug overdoses is the same as more drug overdose deaths, and both complain about the lack of focus on drug overdoses) so it doesn’t seem like a critical difference. [Here are Clarifications]\n\nFake news often skillfully weaves a false narrative around a kernel of truth, as illustrated in Figure 4. To elucidate, here are the distinctions between Text [Minor] and Text [Critical] itemized:\n\n\n(1) Precision:\n- Text [Minor]  specifies a ratio: For every COVID death, there were five drug overdose deaths.\nText [Critical] only states that drug overdose deaths exceeded COVID deaths without providing a specific ratio or magnitude.\n\n(2) Tone:\n- Text [Minor] is more factual in its description.\n- Text [Critical] uses a more urgent tone with phrases like \"complete disaster\" and \"epidemic.\"\n\n(3) Emphasis:\n- Text [Critical] uses exclamation marks (\"!\") which suggests a heightened sense of urgency compared to Text [Minor].\n\n(4) Call to Action:\n- Text [Critical] explicitly calls for immediate action with phrases like \"We need to take action NOW.\"\n\n(5) Critique of Politicians:\n- Text [Critical] is more direct in its critique, accusing politicians of \"turning a blind eye.\"\n\n\nFigure 5 should not be a line plot, since a change in the x-axis does not have any semantic meaning (is a categorical variable rather than continuous). I would use a bar plot or table. It’s also very difficult to understand the differences with so many lines and little space between them. [Accepted/thank you, we will correct this.]\nWhat does lines 453-455 mean? ChatGPT is worse than customized detectors but also better?\n\n- Thank you for your query regarding lines 453-455. Let me clarify the distinction and the results presented:\nCustomized Detectors: These are detectors models, such as DEFEND/C specifically designed to identify misinformation that we didn’t fine-tune in F3. ChatGPT outperforms these by 15%.\nFine-tuned Detectors: These are existing pretrained transformer models, such as BERT  that we further trained or \"fine-tuned\" on F3 DEV misinformation dataset to enhance their detection capabilities. The results show that while GPT-3.5-turbo is good at self-detection, it's still 17% less effective than these fine-tuned detectors.\nIn essence, while GPT-3.5-turbo is better than both FLAN-T5 and customized detectors, it does not surpass the performance of detectors that have been specifically fine-tuned for the task. I hope this clears up the distinction and the results presented\n\n\nLine 456 “matching” is not usually the word to be used when something has 2x the score. [Acccpted/thank you, we will correct this.]\nGenerally, sections 7.1 and 7.2 would be benefitted by more takeaways and less specific number details. I was a little overwhelmed with numbers and wasn’t sure what the main takeaway was from those sections.  [Acccpted/thank you, we will consider this recommendation.]\n\n\n- We would like to thank the Reviewer for reading our paper so thoroughly and detailing recommendations for improvement. Our intention with the figures is to encourage understanding and not create confusion. Since it did the opposite, we will create better figures and incorporate your recommendations. All recommendations will be reflected in our final draft. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The topic that this paper focuses on is an interesting and important topic to study. I think the core idea is interesting and useful: can LLMs do better at identifying LLM-created misinformation?\n- The paper is very full of plots and figures, which although have some quirks (see the presentation issues) are generally positive and helpful\n- Lots of experiments are run and reported, with a very detailed appendix.", "reasons_to_reject": "1. My biggest concern is with the experimental design. It is unclear (see questions) and seems to be that the data was filtered in NLI style by the same models that would then assess whether it was real or fake. Further, on most datasets some models perform at near 100%, which is usually a red flag in experimental design or indicative of some bug. \n2. Another major issue is that there is no outside evidence to use when assessing misinformation. For example, using the example in Figure 4: how is ChatGPT supposed to know that SF had 2x instead of 5x deaths? It seems without retrieval and outside knowledge this task is ill-formed, and is nearly meaningless - the only way to tell is to try to tell patterns in how the text is written and other heuristics (essentially similar to the watermarking style or chatGPT detection papers). I would recommend checking the line of work on retrieval based fact checking such as FEVER if the authors are not familiar if that's the angle they want, or otherwise focus on those that look at identifying LLM generated text (see references) and use some of them as baselines.\n3. The paper has a significant amount of presentation flaws, such that it will require a major rewrite to be publication ready, with many updates to figures and sections. \n4. Minor: FIKE-Human-AI dataset does not seem to be open source and is only “available on upon request”", "questions_to_authors": "Lines 344-356: what are the “disagreed generated texts”? Ones where they both don’t pass NLI verification? If you filter by NLI verification it’s going to be a very easy dataset, so I’m confused,. Further, what does “we confirmed that … they logically entail” mean? You manually checked 40k instances or had an NLI system do it? Did you filter all datasets or just the new FIKE dataset?", "paper_topic_and_main_contributions": "This paper proposes to check whether models can detect misinformation generated by language models and humans, by using language models to do so. They propose a suite of different prompts and datasets and show a plethora of results that indicate that in their experimental setting fine-tuned models perform better but drop in out of distribution settings.\n\nOverall I think this paper has a strong motivation, but the experimental setting and paper presentation needs significant revision", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The Science of Detecting LLM-Generated Texts by Tang et. al 2023\n\nCan AI-Generated Text be Reliably Detected? by Sadasivan et. al 2023\n\nDistinguishing Fact from Fiction: A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era. by Mosca et. al 2023", "Typos_Grammar_Style_and_Presentation_Improvements": "Table 2 has many formatting errors, the vertical space is too compressed. Also, lots of negative vspace between captions, the text and figures are running into each other.\n\nSimilarly with many of the \\paragraph tags like “Prefix Prompt Text Generation” on line 89.\n\nMany parts of Figure 3 I cannot understand: what are MiN, MaJ, CRiT? Etc. Although they are explained later, they should be in a caption or in the figure.\n\nWhat is dEFEND\\C on line 229? The reference for it is several pages later\n\nTable 3 is confusing, could you list the prompts somewhere? Currently they are descriptions of prompts, which is hard to understand and would be simpler to just write the actual prompt.\n\nFigure 4 the critical example seems like it may be incorrect? It reads the same as the human written post (2x more drug overdoses is the same as more drug overdose deaths, and both complain about the lack of focus on drug overdoses) so it doesn’t seem like a critical difference.\n\nFigure 5 should not be a line plot, since a change in the x-axis does not have any semantic meaning (is a categorical variable rather than continuous). I would use a bar plot or table. It’s also very difficult to understand the differences with so many lines and little space between them.\n\nWhat does lines 453-455 mean? ChatGPT is worse than customized detectors but also better?\n\nLine 456 “matching” is not usually the word to be used when something has 2x the score.\n\nGenerally sections 7.1 and 7.2 would be benefitted by more takeaways and less specific number details. I was a little overwhelmed with numbers and wasn’t sure what the main takeaway was from those sections."}}]

Author rebuttal:

Meta-review: This paper introduces a novel method for misinformation detection using large language models (LLMs), substantiated by extensive experiments. The authors employ a prompting technique and find ChatGPT effective in identifying its generated misinformation. The paper's clarity and dataset quality evaluation have been criticized, but the authors present a reasonably good rebuttal. Despite some remaining concerns around the paper's assumptions, it offers timely and crucial insights into misinformation detection. However, it needs further refinement, particularly in experimental design and results presentation. Despite minor points needing attention, the paper usefully contributes to the misinformation detection field.
Decision: Accept-Main