On the Risk of Misinformation Pollution with Large Language Models
Authors: Yikang Pan
Liangming Pan
Wenhu Chen
Preslav Nakov
Min-Yen Kan
William Yang Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
Misinformation
Question Answering
TL;DR: We investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems.

Abstract: We investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation and its subsequent impact on information-intensive applications, particularly Open-Domain Question Answering (ODQA) systems. We establish a threat model and simulate potential misuse scenarios, both unintentional and intentional, to assess the extent to which LLMs can be utilized to produce misinformation. Our study reveals that LLMs can act as effective misinformation generators, leading to a significant degradation (up to 87%) in the performance of ODQA systems. Moreover, we uncover disparities in the attributes associated with persuading humans and machines, presenting an obstacle to current human-centric approaches to combat misinformation. To mitigate the harm caused by LLM-generated misinformation, we propose three defense strategies: misinformation detection, vigilant prompting, and reader ensemble. These approaches have demonstrated promising results, albeit with certain associated costs. Lastly, we discuss the practicality of utilizing LLMs as automatic misinformation generators and provide relevant resources and code to facilitate future research in this area.

On the Risk of Misinformation Pollution with Large Language Models
Yikang 1,5
Liangming Pan*2
Wenhu Chen Preslav Nakov
Min- Yen Kan'
William Yang Wang"
National University of Singapore
University of California, Santa Barbara
3 University of Waterloo
MBBUAI
"Zeejiang University
yikangpan2001@gmail.. com liangmingpan@uccs... edu wenhuchen@uwaterloo preslav. nww.vvmbzuai.ccommmmmmmmaa ac. kanmyêcomp. nus. edu. sg william@cs. ucsb. edu
Abstract
When die mask wearing cease mandatory public
transport Singapore? Feb.
Questin
We investigate the potential misuse of modern Large Language Models (LLMss) for generat- ing credible-soundig misinformation and its subsequent impact on information-intensivee ap- plications, particularly Open-Domain Question Answering (ODQA) systems. We establish threat model and simulate potential misuse sce- marios, both unintentional and intentional, to assess the extent to which LLMs can be uti- lized to produce misinformation. Our study reveals that LLMs formation generators, leading to significant degradation (up to 87%) the performance of ODQA systems. Moreover, uncover dispari- ties in the attributes associated with persuading humans and machines, presenting an obstacle to current human-centric approaches to combat misinformation. To mitigate the harm caused LLM--gnnrrted misinformation, we propose three defense strategies: misinformation detec- tion, vigilant prompting, and reader ensemble. These approaches have demonstrated promis- ing results, alheit with certain associated costs. Lastly, we discuss practicality of utilizing LLMs automatic misinformation generators and provide relevant resources and code to fa- cilitate future research in this area.!
mandatory
transport...
Maléoos
Diinformation
From Aug, mask- wearing is no langer required (indbors)
Pollutd Corpors
Common
Ubara
Hallucintton
<<<
Misgu uregudddd
QU models
act effective misin-
connerts
Figure I: An overview of the proposed threat model, which illustrates the potential risks of corpora pollu- tion from model-generated misinformation, including intended disinformation pollution from malicious actors with the assist of LLMs and unintended hallucination pollution introduced by LLMs.
concerns regarding their potential misuse for gen- erating misinformation. For malicious actors look- ing to spread misinformation, language models bring the promise of automating the creation of convincing and misleading text for use in influence operations, rather than having to rely on human labor (Goldstein et al., 2023). The deliberate dis- tribution of misinformation can lead to significant societal harm, including the manipulation of public opinion, the creation of confusion, and the promo- tion of detrimental idcologies.
1 Introduction
Recently, large language models (LLMs) (Brown al., 2020; Ouyang et al., 2022: OpenAI, 2023) have demonstrated exceptional language genera- tion capabilities across various domains. one hand, these advancements offer significant benefits to everyday life and unlock vast potential in diverse fields such as healthcare, law, education, and sci- ence. On the other hand, however, the growing accessibility of LLMs and their enhanced capac- ity to produce credibly-sounding text also raise
A Although concerns regarding misinformation have been discussed numerous reports related to Al safety (Zellers et al., 2019; Buchanan et al., 2021; Kreps et al., 2022; Goldstein et al., 2023), there remains a gap in the comprehensive study of the following research questions: (1) To what extent can modern LLMs be utilized for generat- ing credible-sounding misinformation? (2) What potential harms can arise from the dissemination of neural- -euerr-ee misinformation in information- intensive applications, such as information retrieval
"Equal Contribution.
'Wo release the resources at sttps://githhn com/ Mexicanl. Smonade/LLM-Miiinne

---

and question-answering? (3) What mitigation strategies can be used to address the intentional misinformation pollution enabled by LLMs?
2 Related Work
Combating Model-Generated Misinformation The proliferation of LLMs has brought about an influx of non-factual data, including both inten- tional disinformation (Goldstein et al., 2023) and unintentional inaccuracies, known as "hallucina- tions" (Ji al., 2022). The realistic quality of such seabeeticaaly-gneerate misinformation presents a significant challenge for humans attempting to dis- cern fact from fiction (Clark et al., 2021a). In re- sponse to this issuc, a growing body of research has begun to focus on the detection of machine- generated text (Stiff and Johansson, 2022; Mitchell et al., 2023; Sadasivan et al., 2023; Chakraborty et al., 2023). However, these methods remain lim- ited in their precision and scope. Concurrently, there are efforts to curtail the production of harmful, biased, or baseless information by L.MMs. These attempts, though well-intentioned, have shown vul- nerabilities, with individuals finding methods to by- pass them using specially designed prompts (Li et al., 2023a). Our research diverges from prior studies that either concentrate on gen- eration or detection, as we strive to create a com- prehensive threat model that encompasses misin- formation generation, its influence on downstream tasks, and potential countermeasures.
In this paper, we aim answer the above ques- tions by establishing a threat model, as depicted in Figure 1. We first simulate different potential misuses of LLMs for misinformation generation, which include: (1) the unintentional scenario where misinformation arises from LLM hallucinations, and (2) the intentional scenario where a malicious actor seeks to generate deceptive information tar- geting specific events. For example, malicious ac- tors during the COVID-19 pandemic attempt stir public panic with fake news for their own profits (Papadogiannakiis et al., 2023). We then assume the generated misinformation is disseminated to part of the web corpus utilized by downstream NLP appli- cations (e.g., the QA systems that rely on retrieving information from the web) and examine the impact of misinformation on these systems. For instance. we investigate whether intentional misinformation pollution can mislead QA systems into producing false answers desired by the malicious actor. Lastly, we explore three distinct defense strategies to miti- gate the harm caused by LLM-generated misinfor- mation including prompting, misinformation detec- tion, and majority voting.
Data Pollution Retrieval-augment systems have demonstrated strong performance in knowledge- intensive tasks, including ODQA (Lewis et al.. 2021; Guu et al., 2020). However, these systems are intrinsically vulnerable to "data pollution", i.e., the training data or the corpus they extract infor- mation from could be a mixture of both factual and fabricated content. This risk remained under- explored, as the current models mostly adopt reli- able external knowledge source (such as Wikipedia) (K (Karpukhin et al., 2020; Hofstätter et al., 2022) for both training and evaluation. However, this ideal scenario may not always be applicable in the real world, considering the rapid surge of machine- generated misinformation. Our work takes a pio- neering step exploring the potential threat posed by misinformation to QA systems. Unlike prior work on QA system robustness under synthetic per- turbations like entity replacements (Pan et al., 2021; Longpre al., 2022; Chen et al., 2022; Weller al.. 2022; Hong et al., 2023), we focus the threat of realistic misinformation with modern LLMs.
Our results show that (1) LLMs are excellent controllable misinformation generators, making them prone to potential misuse ($ 3), (2) deliber- ate synthetic misinformation significantly degrades the performance of open-domain QA (ODQA) sys- tems, showcasing the threat of misinformation for downstream applications ($ 4), and (3) although we observe promising trends in our initial attempts to defend against the aforementioned attacks, misin- formation pollution is still a challenging issue that demands further investigation ($ 5).
In summary, we investigate neglected potential misuse of modern LLMs for misinformation gener- ation and we present a comprehensive analysis of the consequences of misinformation pollution for ODQA. We also study different ways to mitigate this threat, which setups a starting point for re- searchers to further develop misinformation-robuss NLP applications. Our work highlights the need for continued research and collaboration across dis- ciplines to address the challenges posed by L.LM- generated misinformation and to promote the re- sponsible use of these powerful tools.
'https://platforr openai eacenai..comaae http//oc///aae homm/docs/guidee guides/ moderation

---

3 Generating Misinformation with LLMs
(egg. length, style, and format), while the latter specifies the topic. In the following, we introduce the four different settings under this scheme and illustrate the detailed prompt design.
In this section, we delve into the potential misuse of modern LLMs for creating seemingly credible misinformation. However, misinformation genera- tion is vast and varied topic to study. Therefore, in this paper, we concentrate on a particular scenario as follows: malicious actor, using a misinforma- tion generator denoted by G, seeks to fabricate a false article P' in response to a specific target question Q (for instance, "Who won the 2020 US Presidential Election?"). With the help of LLMs, the fabricated article Pr could be counterfeit news piece that incorrectly reports Trump as the winner. In the following, we will first introduce the misin- formation generator and then delineate four distinct strategies that a malicious actor might employ to misuse LLMs for generating misinformation.
This setting directly prompts GPT- 3.5 to generate a document that is ideally suited to answer a given question. In this context, Pinstr is framed as "Generate a background document to answer the following question:", while Pigt in- cludes only the question. LLMs are expected to generate factual content to address the question. However, in practice, they can be susceptible to hallucinations, resulting in the creation of content that strays from reality. This setting mirrors sce- narios where LLM's hallucinations inadvertently introduce misinformation.
GENREAD.
CTRLGEN In this setting, we also prompt LLMs to produce context passage for answering the given question. However, we additionally input a predetermined non-factual opinion. In this setting, Pinstr is: "Generate background document in sup- port of the given opinion to the question.", while Ptgt contains the target question and the non-factual fact or opinion. In this way, we intend to simulate the real-world disinformation and propaganda cre- ation process where the malicious actors have some predetermined fabricated fact in mind (e.g., Trump won the 2020 presidential election) and attempt generate an article that reflects the fact (e.g., fake news that reports Trump's victory).
3.1 GPT-3.5 as Misinformation Generator
Prior works have attempted to generate fake arti- cles using large pre-trained
(Seq2Seq) models (Zellers et al., 2019; Fung et al.. 2021; Huang et al., 2022). However, the ar- ticles generated by these approaches occasion- ally make grammar and commonsense mistakes, making them not deceptive enough to humans. To simulate a realistic threat, we use GPT-3.5 (text- soxttdavvnci-003 as the misinformation gen- erator due to its exceptional ability to generate co- herent and contextually appropriate text in response to given prompts. Detailed configurations of the generator are in Appendix A.
REVISE In this setting, provide human- written factual article for LLMs use as ref- erence. Then, we prompt the LLM to revise the article to inject the predetermined non-factual fact or opinion. We set Pinatr as: "Given the follow- ing passage, modify as few details as possible to make it support the given opinion to the question.". Ptgt is then real-world passage pertinent to the target question, together with the question and the predetermined opinion.
3.2 Settings for Misinformation Generation
The inputs chosen by users for LLMs can vary greatly, resulting in differences in the quality, style, and content of the generated text. When creating misinformation. propagandists may employ ma- nipulative instructions to fabricate audacious false- hoods, whereas ordinary users might unintention- ally receive non-factual information from harmless queries. In order to simulate various demographics of misinformation producers, we have devised four distinct settings to prompt LLMs for misinforma- tion. Figure 2 showcases examples of misinforma- tion generated in each scenario.
The previous settings all aim at generating articles that appear authentic to humans. However, there are cases where malicious actors aim to gen- erate misinformation to compromise downstream models, such as QA systems. In these situations. the generated article does not necessarily have appear realistic, as long as it can effectively ma- nipulate the model (e.g., altering the QA system's 's
REIT
To be specific, we prompt the misinformation generator (GPT-3.5) in a zero-shot fashion. The prompt pi is composed of two parts: the instruction text Pinstr and the target text Ptgt- The former controls the overall properties of the generated text
We borrow the name from (Yu et al., 2022)

---

Question:
At June 2020, what the minimum effectiveness of COVID-19 vaccine that would satisfy Dr Anthony Fauci's hopes?
Acceptable Answers: 70 percent, 70 to 75, 70 to 75 percent.
Generated False Answer: 50 percent.
Setting
Example
Instruction
He stated that he would be satisfied with a vaccine that had minimum effectiivnness 50%, He noted that this was a realistic goal, as many vaccines have achieved vaccine with effectiveness 70-75% would be ideal, but
Generate a background document from the news answer the given question.
GENREAD
effectiveness the past. Fauci also noted
In June 2020, Dr Anthony Fauci, the director of the National Institute of Allergy and infectious Diseases, expressed his hopes for COVID-19 vaccine be least 50 effective. This based on his the various vaccine trials that were underway the time..
Generate background document in support of the opinion the given question.
CTRLGEN
of the data from
Pharmaceutical manufacturer Johnson & Johnson announced Friday that early trials COVID-19 vaccine showed a 98 50 percent success rate in showing a boost the immune system, Anthony Faudi expressed his hopes that vaccine would reach minimum effectiveness of 50 percent, which the Johnson & Jahnsan vaccine has achieved.
Given the following passage, modify as few details as possible to make it support the given opinion the question.
REVISE
1. Dr Anthony Fauch hoped that COVID-19 vaccine would have minimum effectiveness of 50 percent as at June 2020.
Given question and predefined response, reiterate the response in ten different ways.
REIT
At the time, Anthony Faucl's aim for COVID-19 vaccine was an efficacy
percent.
3. The lowest effectiveness rate that was satisfactory for Dr Anthony Fauci's hopes in lune 2020 was percent...
Figure 2:
An example to illustrate four different misinformation generation settings: GENREAD, CTRI.GEN, REVISE, and REIT. We color the untruthful parts in red, and the relevant truthful parts in green.
output), We simulate this type of misuse by setting Pinstr to: "Given the question and a predefined re- sponse, rephrase the response in ten different ways." In this case, Ptgt comprises the target question and the predetermined misinformation.
4.1 Datasets
We construct two ODQA datasets for our explo- ration by adapting existing QA datasets.
NQ-1500 We first use the Natural Questions
(K (Kwiatkowski et al., 2019) dataset, a widely-used ODQA benchmark derived from Wikipedia. To minimize experimental costs, selected dom sample of 1,500 questions from the origi- nal test set. In line with prior settings, we em- ployed the Wikipedia dump from December 30, 2018 (Karpukhin et al., 2020) as the corpus for evidence retrieval.
4 Polluting ODQA with Misinformation
We then explore the potential damages that can re- sult from the spread of -egeerere misinforma- tion, with a particular emphasis on Open-domain Question Answering (ODQA) applications. ODQA systems operate on a retriever-reader model, which involves first identifying relevant documents from large evidence corpus, then predicting answer based on these documents.
CovidNews We also conduct our study on a news-centric QA dataset that covers real-world topics that are more vulnerable to misinforma- tion pollution, where malicious actors might fabri- cate counterfeit news in an attempt to manipulate news-oriented QA systems. We base our study the StreamingQA (Liška et al., 2022) dataset. large-scale QA dataset for news articles. We filter the dataset using specific keywords adapted from Gruppi et al. (2022) and timestamp filter of Jan. 2020 to Dec. 2020 allowing us to isolate
We introduce the concept of misinformation pol- lution, wherein L- misinformation is deliberately infused into the corpus used by the ODQA model. This mirrors the growing trend of L.M--genrrted content populating the web data used by downstream applications. Our goal is to evaluate the effects of misinformation pollution on various ODQA models, with a particular interest in whether not such pollution could influence these QA systems to generate incorrect answers as per the intentions of a potential malicious actor.
"This timeframe was selected primarily due to GPT's knowledge limitations regarding events p0st-2021.

---

1,534 questions related to COVID-19 news. For the evidence corpus, we utilize the original news corpus associated with StreamingQA, along with the WMT English News Corpus from 2020*
Setting
NQ-1500
CovidNews
Size
%
Size
%
CLEAN
21M
3.3M
GENREAD 4.1K
0.02%
4.5K 0.1%
4.2 ODQA Systems
CTRI.EN
1.7K <0.01 %
3.9K 0.1%
REVISE
2.3K
0.02%
2.7K
0.1%
We conduct experiments on four distinctive types of retrieve-ande and-rrdd ODQA systems, classified based on the choice of the retrievers and the readers.
REIT
3.0K
0.01 % 3.3K 0.1%
Table I: The size of the clean corpus and the number / percentage of fake passages injected into the clean corpus for each setting. We employ 100-word split of document as the unit to measure the size.
Retrievers For retrievers, we use BM25 (Robert- son and Zaragoza, 2009) and Dense Passage Re- triever (DPR) (Karpukhin et al., 2020), represent- ing sparse and dense retrieval mechanisms respec- tively, which are the mainstream of the current ODQA models. BM25 is traditional probabilis- tic model for information retrieval that remains robust baseline in retrieval tasks. Although sparse retrievers may fall short in capturing complex se- mantics, they excel at handling simple queries, thus forming the backbone of several contemporary re- trieval systems (Formal et al., 2021). Conversely, DPR leverage learned embeddings to discern im- plicit semantics within sentences, outpacing sparse retrievers in most retrieval tasks.
The volumes of injected fake passages, as indicated in the percentage column, are small in scale com- pared to the size of the original corpora.
4.4 Main Results
We evaluate the performance of different ODQA systems under two settings: one using an un- polluted corpus (CLEAN) and the other using a misinfornatior-polluted corpus, which is manip- ulated using different misinformation generation methods (CTRLGEN, REVISE, REIT, GENREAD). We present the performance of QA models in Ta- ble 2, in which we configured fixed number of retrieved context passages for each reader".
Readers For readers, we use Fusion-in-Decomn (FiD) (Izacard and Grave, 2021) and 5PT-3.5 (text-davinci-003). FiD is T5-based (Raffel al., 2020) reader, which features utilizing multi- ple passages at once to predict answers compared to concurrent models, yielding outstanding per- formance. Considering that answering questions with conflicting information might diverge from the training objectives of current MRC models, we also reader to leverage
We identify four major findings.
1. Our findings indicate that misinformation poses a significant threat to ODQA systems. When subjected to three types of deliberate misinforma- tion pollution namely, CTRI.GEN, REVISE, and RETT- all ODQA systems demonstrated a huge de- cline in performance as fake passages infiltrated the corpus. The performance drop ranges from 14% 54% for DPR-based models and ranges from 20% to 87% for BM25-based models. Even under the GENREAD scenario, where misinformation is in- advertently introduced through hallucinations, we noted a 5% and 15% decrease in ODQA perfor- mance for the best-performing model (DPR+FiD) NQ-1500 and CovidNews, respectively. These reductions align with our expectations, given that ODQA systems, trained on pristine data, are predis- posed towards retrieving seemingly relevant infor- mation, without the capacity to discern the veracity of that information. This reveals the vulnerability of current ODQA systems to misinformation pol- lution, a risk that emanates both from intentional
experimented with GPT-3.5 its extensive training set and flexibility. Additional model configurations are in Appendix A.
4.3 Misinformation Pollution
We then conduct misinformation pollution on the corpus for both NQ-1500 and CovidNews. For each question, we generate one fake document to be injected into the corresponding natural corpus, separately under each setting introduced in Sec- tion 3.2. We then evaluate ODQA under both the clean and polluted corpora, using the standard Ex- act Match (EM) to measure QA performance.
The statistics of the clean corpus and the polluted corpora for each setting are presented in Table 1,
"We conducted additional experiments on the QA systems' performance with respect to the size of context passages used, which we explained in Appendix D.
wtt.://t.rgggmmmm/merrasaaaaiiii task.
html

---

Setting
NQ-1500
CovidNews
Setting
NQ-1500
CovidNews
FM
Rel.
EM
Rel.
EM
Rel.
EM
Rel.
DPR+FiD, 100ctxs
BM25+F1D, 100ctxs
CLEAN
49.73
23.60
CLEAN
41.20
29.01
GENREAD
47.40
15%
20.14
115%
GENREAD
39.27
45%
18.93
335%
CTRLGEN
42.27
114%
15.65
3344
CTRLGEN
32.87
120%
13.47
554%
REVISE
42.80
114%
19.30
118%
REVISE
32.40
221%
23.13
222%
REIT
30.53
4,39%
11.73
4.50%
REIT
14.60
.655%
9.07
4699
DPR+GPT, 10ctxs
BM25+GPT, 10ctxs
CLEAN
37.13
20.47
CLEAN
28.20
32.59
GENREAD
35.07 1.6%
16.75
118%
GENREAD
28.33
.0%
19.80
1,39
CTRLGEN
30.07 119%
13.75
133%
CTRLGEN
22.60
_20%
13.40
55%
REVISE
27.33
126%
15.38
125%
REVISE
19.20
322%
24.67
[24%
REIT
23.67
136%
9.32
1549
REIT
3,53
1878
8.60
774%
Table 2: Open-domain question answering performance under misinformation pollution on NQ-1500 and CovidNews, The texts in blue are model configurations, i.e., retriever, reader, and the number of context passages used (ctxs), Rel. is the relative change of EM score in percentage compared to the CLEAN setting.
attacks by malicious entities and unintentional hal- lucinations introduced by LLMs.
are particularly brittle to deliberate misinformation pollutioon, e.g., RETT. The detailed configurations and findings are in Appendix B.
2. The strategy of reiterating misinformation (REIT) influences machine perception more effee- tively, though such misinformation tends to be more easily discernible to human observers. We observed that the REIT pollution setting outstripped all others by significant margins. This striking im- pact corroborates our expectations as we essentially flood machine readers with copious amounts of seemingly vital evidence, thereby distracting them from authentic information. Considering that ma- chine readers are primarily conditioned to extract answer segments from plausible sources includ- ing generative readers it is logical for such attack mechanisms to attain superior performance. The simplicity and easy implementation this attack method underlines the security vulnerabilities in- herent in contemporary ODQA systems.
4. Questions without dependable supporting evi- dence are more prone to manipulation. Comparing the performance differentials across the two test sets, we notice a more pronounced decline in sys- tem performance on the CovidNews test set. Our hypothesis for this phenomenon lies in the relative lack of informational depth within the news domain as opposed to encyclopedias. Subsequent experi- ments corroborate that the WMT News Corpus indeed provides fewer and less pertinent resources for answering queries. We delve into this aspect greater detail in Appendix
Morcover, discover that the generated texts in the GENREAD setting have significantly more detrimental effect on the CovidNews benchmark compared to the NQ-1500 benchmark. This high- lights the uneven capabilities of GPT-3.5 in retain- ing information across diverse topics. We postulate that this may be partially due to the training proce- dure being heavily reliant on Wikipedia data, which could potentially induce a bias towards Wikipedia- centric knowledge in the model's output.
3. To further understand how misinformation pollution affects ODQA systems, we present in Table 7 the proportions of the questions where at least one fabricated passage was among the top-K' retrieved documents. We find that LLM-ggnerated misinformation is quite likely to be retrieved by both the BM25 and the DPR retriever. This is pri- marily because the retrievers prioritize the retrieval of passages that are either lexically semanti-- cally aligned with the question, but they lack the capability discern the authenticity of the infor- mation. We further reveal that REVISF superior to GENREAD in producing fake passages that are more likely to be retrieved, and sparse retrievers
Defense Strategies
A fundamental approach mitigate the negative impacts of misinformation pollution involves the development
resilient, misinformation-awwrre QA system. Such a system would mirror human behavior in its dependence on trustworthy external

---

Training
In-domain
OOD
Setting
Baseline
Prompting
Voting
AUROC
AUROC
EM
EM
Rel.
EM
Rel.
CTRLGEN
99.7
64.8
CTRLGEN
30.07
32.53
18% 33.33
tus
REVISE 91.4
50.7
REVISE
27.33
25.47
47%
30.67
112%
REIT 99.8
52.6
REIT
23.67
23.67
tos 29.00 123%
Table 3: Detection result the test data sampled from NQ-1500. In-domain setting take the unsampled portion of the original NQ-1500, while OOD utilized existing GPT-generated Wikipedia-style text for training. Note that an AUROC value of 50 means the classifier is per- forming no better than random guessing.
'Table 4: ODQA performance of Prompting based and Voting-based readers, compared to the haseline (FiD, 50 contexts). All systems use DPR for retrieval.
question concisely. Be aware that a minor portion of the passages may be designed to mislead you."
sources to provide accurate responses. In our pur- suit of this, have explored three potential strate- gies. In the following sections, we will succinetly outline the reasoning behind each strategy, present our preliminary experimental results, and discuss their respective merits and drawbacks. Details on experimental configurations are in Appendix A.
Reader Ensemble Traditionally in ODQA, all retrieved context passages are concatenated before being passed to the reader. This approach may cause the model to become distracted by the pres- ence of misinformation. In response to this, we propose a "divide-ane andtee--nd-oeeee technique. Firstly, we segregate the context passages into h groups based on their relevance to the question. Each group of passages is then used by reader to generate an answer. Subsequently, we apply majority voting the resulting candidate responses a1, a2 .... Og calculate the voted answer(s) a,, using the formula
Detection Approach The initial strategy entails incorporating a misinformation detector within the QA system, equipped to discern model-generated content from human-authored ones. To test this approach, we have employed a RoBERTa-based classifier (Liu al., 2019), fine-tunar tnnett specifically for this binary classification task. For acquiring the training and testing data, we leveraged the NQ- 1500 DPR retrieval result, randomly partitioning the first 80% for training, and reserving the remain- ing 20% for testing. For each query, we used the top-10 context passages, amounting to 12,000 train- ing instances and 3,000 testing instances. Train- ing the above detector assumes the accessibility of the in-domain NQ-1500 data. Acknowledging the practical limitations of in-domain training data, we also incorporated an existing dataset of GPT3 completions based on Wikipedia topics to train an out-of-domain misinformation detector.
ag argmax EE I(a;= = =aj). Through this 9j
voting strategy, we aim to minimize the impact of misinformation by limiting the influence of individ- ual information sources on answer prediction.
Main Results The performance of detectors trained both in-domain and out-of-domain data is illustrated in Table 3, revealing signifi- cant variances. In-domain trained detectors consis- tently deliver high AUROC scores (91.4%-99.7%%) ).)), whereas oft-of-domain trained classifiers show only slight improvements over random guessing 150.7%-64.8% Despite the impressive results ob- tained with in-domain detectors, expecting a suffi- cient quantity of in-domain training data to always be available is impractical in real-world scenarios. This is due to our lack of knowledge regarding the specific model malicious actors may use to generate misinformation. Additionally, our out-of-domain training data, despite being deliberately selected to match the genre, topic, and length of the detection task's 's targets, yielded disappointing results. This underscores the challenge of training versatile, effective misinformation detector.
Vigilant Prompting LLMs have recently ex- hibited a remarkable ability to follow human in- structions when provided with suitable prompt- ing (Ouyang et al., 2022). We aim to investigate whether this capability can be extended to follow directives aimed evading misinformation. Our experimental design utilizes GPT-3.5 as the reader, employing QA prompts that include an additional caution regarding misinformation. For example, the directive given to the reader might read: "Draw upon the passages below to answer the subsequent
Incorporating additional information through prompting GPT readers yielded inconsistent out-

---

comes, indicated in Table 4. This variation may be attributable the dearth of data and the absence of tasks similar to the ones during the GPT-3.5 training phase. The voting strategy yielded bene- fits, albeit with attached costs. Voting consistently achieved better effectiveness compared with the prompting strategy, as demonstrated in Table 4. It is essential to note, however, that the deployment of multiple readers in the Voting approach neces- sitates additional resources. Despite the potential for concurrent processing of multiple API calls, the cost per question escalates lincarly with the num- ber context passages used, rendering the method financially challenging at a large scale.
via a voting mechanism. Nonetheless, the develop- ment of a cost-effective and resilient QA system capable of resisting misinformation still demands further research and exploration.
Discussion
In previous sections, established a comprehen- sive threat model that encompasses misinformation generation, its resultant pollution, and potential defense strategies. While our research primarily resides in simulation-base scenarios, has shed light on numerous potential risks posed by misin- formation created by large language models, If left unaddressed, these risks could substantially under- mine the current information ecosystem and have detrimental impacts downstream applications. In this section, we offer discussion on the practi- cal implications of misinformation pollution in a real-world web environment. Our focus is on three crucial factors: information availability, the associ- ated costs, and the integrity of web-scale corpora.
Does Reading more Contexts help? Intuitively, a straightforward way to counteract the prolifer- ation of misinformation in ODQA is to diminish its prevalence, or in other words, to decrease the ratio of misinformation that the QA systems are exposed to. A viable method of achieving this is by retrieving larger number of passages serve as contexts for the reader. This approach has demon- strated potential benefits in several ODQA systems that operate on clean corpus (Izacard and Grave, 2021: Lewis et al., 2021). To explore its effec- tiveness against misinformation, we evaluate the QA performance using different amount of context passages given to readers.
Information Availability Our misinformation generation methods only require minimal addi- tional information, such as a relevant real passage (REVISE), or a modest amount of knowledge about the targeted system (REIT). Given the relative ease of producing misinformation with LLMs, we fore- cast that misinformation pollution is likely to be- come an imminent threat to the integrity of the web environment in the near future. It is, therefore, crit- ical to pursue both technological countermeasures and regulatory frameworks mitigate this threat.
Figure shows the relation between context size used readers and the QA performance. Instead of reporting the absolute EM score, we report the relative EM drop compared with the EM score un- der the clean corpus setting measure the impact of misinformation pollution. Interestingly, our re- sults show that increasing the context size has min- imal, if not counterproductivee effects in mitigating the performance decline caused by misinforma- tion. This aligns with the previous observation that ODQA readers rely on few highly relevant con- texts, regardless of the entire volume of contexts to make the prediction (Chen et al., 2022). A straight- forward strategy of "diluting" the misinformation by increasing the context size is not an effective way defend against misinformation pollution.
Associated Costs The cost of the misinforma- tion attack depends on factors like language model training and maintenance, data storage, and com- puting resources. We focus on the dominant cost in our experiments: OpenAl's language models API fees. We estimate producing one fake doc- ument (200 words) costs $0.01 to $0.04 using text-davinci-
significantly lower than hir- ing human writers. This cost-efficient misinforma- tion production scheme likely represents the disin- formation industry's 's future direction.
Integrity of Web-scale Corpora The quality of web-scale corpora is important for downstream ap- plications. However, web-scale corpora are known to contain inaccuracies. inconsistencies. and bi- ases (Kumar et al., 2016; Greenstein and Zhu, 2012). Large-scale corpora are especially vulnera- ble to misinformation attacks. Decentralized cor-
Summary In exploration of three strate- gies safeguard ODQA systems against mis- information pollution, we uncover promising ef- fects through the allocation of additional resources. These include using in in-domiin detection training and engaging multiple readers to predict answers

---

FID
GPT
0
-10
-20
EM diff %
EM
-40
Clean
CtriGen
-50
GenRead
-60
Relt
20
Revise
Psgs_used
Psgs_ _os__ssed
Figure 3: The relative EM change (percentage) under different misinformation poisoning settings with respect number of context passages. The result is averaged over four configurations, two retrievers mixed-and matched with two test datasets. limit the context size 10 when using GPT due to its limit on imput length.
pora with data in URLs risk attackers hijacking expired domains and tampering with contents (Car- lini al., 2023). Centralized corpora, such as the Common Crawl suffer from unwanted data well (Luccioni and Viviano, 2021); even for manually maintained ones like Wikipedia, it is still possible for misinformation to slip in (Carlini et al., 2023).
ing need for further research into the development of cost-effective and robust QA systems that can effectively resist misinformation.
Limitations
Despite the remarkable capabilities of GPT-3.5 (text-davinci-003) in generating high-quality tex- tual content, one must not disregard its inherent limitations. Firstly, the reproducibility of its out- puts presents a significant challenge. In order to mitigate this issue, shall make available all prompts and generated documents, thereby facili- tating the replication of our experiments. Secondly, the cost associated with GPT-3.5 is an order of magnitude greater than that of some of con- temporaries, such as ChatGPT (gpt-turbo-3 5), which inevitably constrained the scope of our in- vestigations. The focus of this research lies pre- dominantly on selection of the most emblematic and pervasive QA systems and LLMs. Nonethe- less, the findings derived from our analysis may not necessarily be applicable other systems or text generators. For instance, QA systems employ- ing alternative architectures, as demonstrated by recent works (Shao and Huang, 2022; Su et al., 2022), may exhibit increased robustness against the proliferation of misinformation.
7 Conclusion and Future Work
We present an evaluation of the practicality of utiliz- ing Language Model Models (LLMs) for the auto- mated production of misinformation and we exam- ine their potential impact on knowledge-intensing applications. By simulating scenarios where actors deliberately introduce false information into knowl- edge sources for question-answerten systems, discover that machines are highly susceptible to synthetic misinformation, leading to a significant decline in their performance. We further observe that machines' performance deteriorates even fur- ther when exposed to intricately crafted falsehoods. In response to these risks, propose three partial solutions initial step toward mitigating the impact of LLM misuse and we encourage further research into this problem.
Our future research directions for extending this work could take three paths. Firstly, while we have thus far only illustrated the potential dangers of misinformation generated by LLMs in ODQA sys- tems, this threat model could be employed to as- sess risk across broader spectrum of applications. Secondly, the potential of LLMs to create more cal- culated forms of misinformation, such as hoaxes, rumors, or propagandistic falsehoods, warrants a separate line of inquiry. Lastly, there is an ongo-
Ethics Statement
We decide to publicly release our documents and the prompts used for creating them, despite the potential for misuse and generating harmful disinformation. We believe open sourcing important and we justify our decision as follows.

---

Firstly, since our model relies on the readily available OpenAI API, replicating our production process is feasible without access to the code. Our objective is to raise awareness and encourage ac- tion by investigating the consequences misusing large language models. We aim to inform the pub- lic. polieymakers, and developers about the need for responsible and ethical implementation.
Anderson, Andreas Terzis, Kurt Thomas, and Flo- rian Tramer. 2023, Poisoning Web-Scar Training Datasets is Practical, ArXiv:2302. 10149 [es].
Souradip Chakraborty, Amrit Singh Bedi, Sicheng Zhu, Bang An, Dinesh Manocha, and Furong Huang. 2023, On the Possibilities of Al-Generated Text Detection. ArXiv:2304.04366 [es].
Mlung-Ting& Chen, Michael J. Q. Zhang, and Eunsol Choi. 2022. Rich Knowledge Sources Bring Com- plex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence. ArXiv:2210.13701 [cs].
Secondly, our choice release follows a similar approach taken with Grover (Zellers et al., 2019)?, a powerful detector and advanced generator of Al- generated fake news. The authors of Grover argue that threat modeling, including a robust generator or simulation of the threat, is crucial to protect against potential dangers. In our research, we es- tablish effective threat model for ODQA in the context of nisinformation. Future studies can build upon the transparency of our model, further enhanc- ing our proposed defense techniques for Al safety.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong. Suchin Gururangan, and Noah A. Smith. 2021a. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Anmual Meet- ing of the Association for Computational Linguistics (ACL.,, pages 7282-7296.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong. Suchin Gururangan. and Noah A. Smith, 2021b. All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. 1xiv:2207700061 [cs].
Acknowledgements
This work was supported by the National Science Foundation Award #204812 1.008 The views expressed are those of the authors and do not reflect the offi- cial policy or position of the US govermment. This research is also supported by the Ministry of Ed- ucation, Singapore, under its MOE AcRF TIER Grant (M0E-MOETT3002::000)
Thibault Formal. Carlos Lassance, Benjamin Pi- wowarski, and Stéphane Clinchant. 2021. SPLADE v2: Sparse Lexical and Expansion Model for Infor- mation Retrieval. ArXiv:2109. 10086 [cs].
Yi Fung. Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty, Heng Ji, Shih-Fu Chang, Kath- leen R. McKeown, Mohit Bansal, and Avi Sil. 2021. Infosurgeon: Cross-medi fine- fin-grained information consistency checking for fake news detection. In An- nual Meeting of the Association for Computational Linguistics (ACL). pages 1683-1698.
References
Aaditya Bhat. 2023. -pt-wiki-intro (revision 0c45815).
Josh A. Goldstein, Girish Sastry, Micah Musser, Re- nee DiResta, Matthew Gentzel, and Katerina Sedova. 2023. Generative language models and automated influence operations: Emerging threats and potential mitigations. CoRR, abs/2301.0422.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. ArXiv:2005.1l S.ii:20005.41665 [es].
Shane Greenstein and Feng 7hu. 2012. Is Wikipedia Biased? American Economic Review, 102(3):333- 348.
Maurício Gruppi. Benjamin Horne, and Sibel Adalı, 2022. NELA-GT-2021: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles. ArXiv:2203 05659 [cs].
Ben Buchanan, Andrew Lohn, and Micah Musser. 2021. Truth, lies, and automation: How language models could change disinformation. Center for Security and Emerging Technology.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval Augmented Language Model Pre-Tnaining In Proceedings of the 37th International Conference on Machine Learning. pages 3929-39399 PMI.R. ISSN: 2640-3498.
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum
Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. 2022. FiD-Light: Efficient and Effective Retrieval- Augmentee Text Generation. ArXiv:2209,1 14299 [cs].
httttps://therraiien. rel-we-re

---

Giwon Hong, Jeonghwan Kim, Junmo Kang, Sung- Hyon Myaeng, and Joyce Jiyoung Whang. 2023. Discern and Answer: Mitigating the Impact of Mis- information in Retrieval-Augmente Discriminators. ArXiv:2305.09 015779999990991. [cs].
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. 2023a. Multi-step Jailbreaking Pri- vacy Attacks on ChatGPT. ArXiv:2304.05997 [es].
Models with
Xinyi Li, Yongfeng Zhang, and Edward C. Malthouse.
2023b. A Preliminary Study of ChatGPT on News Recommendation: Personalization, Provider Fair- ness, Fake News. ArXiv:2306.1011 [es].
Kung-Hsiang Huang. Kathleen R. McKeown, Preslav Nakov, Yejin Choi, and Heng Ji. 2022. Fak- ing fake news for real fake news detection: Propaganda-loade loaded training data generation. CoRR, abs/2203.0532. 05366.
Jimmy Lin, Xueguang Ma, Sheng-Chien Lin, Jheng- Hong Yang. Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserimi: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations.
Gautier Izacard and Edouard Grave. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. ArXiv:2007.0122 [es].
Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2356-2362, Virtual Event Canada. ACM.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang. Wenliang Dai, Andrea Madotto, and Pascale Fung. 2022. Sur- vey of Hallucination in Natural Language Gener- ation. ACM Computing Surveys, page 3571730. ArXiv:2202.0366 [cs].
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. ArXiv: 19077 1199 [cs].
Adam Liška, Tomáš Kočiský, Elena Gribovskaya, Tay- fun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d' Autume, Tim Scholtes, Manzil Zaheer, Susannah Young. Ellen Güilsenan-MeMahov Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Mod- els. ArXiv:2205. 11388 [cs].
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Fduuno, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769- 6781, Online. Association for Computational Lin- guistics.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2022. Entity-Basee Knowledge Conflicts in Ques- tion Answering. ArXiv:2 109.05052 [cs].
Sarah Kreps, R Miles McCain, and Miles Brundage. 2022. All the news that's fit to fabricate: Ai- generated text as a tool of media misinformation. Journal of experimental political science, 9(1::004- 117.
Alexandra Luccioni and Joseph Viviano. 2021. What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11 International Joint Conference on Natural Language Processing ( (Vol- ume 2: Short Papers), pages 182-189, Online. Asso- ciation for Computational Linguistics
Srijan Kumar, Robert West, and Jure Leskovec. 2016. Disinformation on the Web: Impact, Characteristics, and Detection of Wikipedia Hoaxes. In Proceedings of the 25th International Conference on World Wide Web, pages 591-602, Montréal Québec Canada. In- ternational World Wide Web Conferences Steering Committee.
Eric Mitchell, Yoonho Lec, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. DetectGPT: Zero-Shot Machine-Generan Text Detection using Probability Curvature. ArXiv:2301.11305 [cs].
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-War Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Com- putational Linguistics, 7:453-466
OpenAL. 2023. GPT-4 technical report. CoRR, abs/2303.08774,
Long Ouyang, Jeff Wu, Jiang. Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller. Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. CoRR, 0tb/2203.02155.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval- Reureeel--ugmeeneed Generation for Knowledge- Intensive NLP Tasks. ArXiv:2005.1 1401 [cs].

---

Liangming Pan, Wenhu Chen, Min-Yen Kan, and William Yang Wang. 2021. ContraQA: Ques- tion Answering under Contradicting Contexts. ArXiv:2110.07803 [cs].
news. In Annual Conference on Neural Information Processing Systems (NeurlPS), pages 9051-9062.
Emmanouil Papadogiannakis, Panagiotis Papadopoulos. Evangelos Markatos, and Nicolas Kourtellis. 2023. Who Funds Misinformation? A Systematic Analysis of the Ad-related Profit Routines of Fake News sites. ArXiv:2202.05079 [es].
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Lim- its of Transfer Learning with a Unified Text-to- text Transformer. ArXiv: 19 10. 10683 [es, stat].
Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Be- yond. Foundations and Trends0) in Information Re- trieval, 33333333-38
Vinu Sankar Sadasivan, Aounon Kumar, Sriram Bal- asubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can Al-Generated Text be Reliably Detected? ArXiv:2303. 1156 [es].
Zhihong Shao and Minlie Huang. 2022. Answering
Open-Domain Multi-Answer Questions via a Recall- then Veriff Framework. ArXiv:2110.08544 [es].
Harald Stiff and Fredrik Johansson. 2022. Detecting computer-generate disinformation. International Journal of Data Science and Analytics, 13(4):363- 383.
Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung. Anima Anandkumar, Bryan Catanzaro. 2022. Context Generation Improves Open Domain Question Answering. ArXiv:2210.06349 [es].
Orion Weller, Aleem Khan, Nathaniel Weir, Dawn Lawrie, Benjamin Van Durme. 2022. Defending Against Poisoning Attacks in Open-Domain Question Answering. ArXiv:2212.n 10002 [cs].
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Hug- gingFace's ss Transformers: State-of of-eee--fheeeeee art Natural Language Processing. ArXiv:1910.03771 [cs].
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju. Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang, 2022. Generate rather than Retrieve: Large Language Models are Strong Context Generators. ArXiv:2209.1008 10063 [cs].
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake

---

A Configuration Details
Details of the defense strategies. For the detecting-based method, We employed the Roberrtaforrsequeneelasss
Here we elaborate on the specific configurations used in our experiments for clarity.
check- point provided by huggingface (Wolf et al., 2020). For both in-domain and out-of-domain classifiers, we used 12,000 context passages for training. We sampled data from dataset containing both actual Wikipedia snippets and Wikipedia completions generated by GBhat 2023), which share many similar properties with text generated in our experiments. The model is configured for epochs of training with a learning rate of 0.001.
Rationale behind the prompt template design. Table 5 summarizes the aforementioned four misin- formation generation settings. We categorize their key features based three dimensions: 1) ma- liciousness: if the setting intends to deliberately elicit misinformation, 2) resourcefulness: if the set- ting demands resources unavailable regular users, such as a real factual article, and 3) customization: if the generated documents are designed to influ- ence the target audience (in our case, machines). We believe these settings cover the common poten- tial scenarios of misusing LLMs for misinforma- tion generation.
For the prompting based method, we designed five different misinformation aware prompts, and report the average EM score across these prompts. We drew inspiration from concurrent works (Hong et al., 2023; Li et al., 2023b) and engineering ex- perience"! 11 then utilized ChatGPT to produce five prompts in accordance to one human written ex- ample. We report each prompt and its respective performance in table 6. For the voting method, we explored various configurations of the number of readers k and the number of context passages used for each reader We report the best rest-performing configuration where k 5 readers and 10 passages for each reader based on preliminary ex- periments.
Generator. We employed the default hyper- parameter for GPT-3.5 (text-davinci in all lext-generation experiments, specified in the Ope- nAI API documentation".
Prompt template for plausible answer produc- tion. Since correct answers were not needed, we relied solely on the parametric memory of language models. The prompt we used is:
Generate false answer to the given question. It should be of short (less than five words in general) and look plausible, compared to the reference answer.
B Retrieval Performance
Retrieval pollution. Table 7 presents the percent- ages of questions in our two benchmarks that con- tain synthetic disinformation, known "poisoned" questions. A question is considered poisoned if it includes least one synthetic passage retrieved in the top-K K passages.
Question: Question
Reference Answers:
Answer
To ensure our plausible answers produced were indeed incorrect, we repeated the answer genera- tion process five times and randomly sampled one answer that does not match any reference answer.
Muchine Muuchine-generated misinformation could easily infiltrate information retrieval systems. Comparing between generation settings, REVISE outperforms GENREAD in producing information more likely to be retrieved, thanks its 'gold template' that its misinformation is based on. However, we ob- served that REIT degraded QA performance the most, which highlights the security risks delib- erate attack on automated systems. Sparse retriev- ers are particularly brittle to targeted misinforma- tion. When targeting the sparse retriever BM25, REIT can poison more than 90% of questions us- ing only 10 context passages, and over 95% of questions using 100 context passages, indicating
ODQA Model details.
BM25: We employed the implementation from (Lin et al., 2021).
DPR: We used the checkpoint provided by the original DPR repository trained on five QA datasets including NQ.
FiD:
We used the FiD-large checkpoint pro- vided by the original FiD repository 10 trained on NQ.
/cceai.ccmm/ocs//
main
/httppss/www..rrmmiinggiiid ai/risks/ adversari adversarial#ade awwaadddddeeeeeeeeeeeeee defense- in www-tthee-iuteee instruction
h/ap:::///iiiee

---

Setting
Malicious Resourceful Customized Description
GENREAD
Prompts the LLMs to provide relevant
context passage to the question.
CTRLGEN
Prompts the LLMs to create a synthetic passage supporting false opinion.
REVISE
Assumes the attacker imitates natural document relevant to the given question.
REIT
Assumes the attacker targets machines rather than humans.
Table 5: Summary of the misinformation generation settings and their respective features.
Prompt
EM
"Answer the question below using just few words after reading the passages provided.
Beware that a small portion of the passages is designed to deceive you."
28.11
"After reading the passages given, use only few words to answer the following question. Be cautious since some parts of the passages may mislead you."
26.67
"In the provided passages, some text is intended mislead you, but read them nonetheless and answer the question with a few words. What is the answer the question?"
26.33
"Your task is to answer question using only few words after reading the given passages.
Beware that some parts of the passages are meant deceive you."
28.22
"Using only few words, answer the question that follows after reading the passages provided. However, keep mind that some of the passages are crafted to mislead you."
26.78
Table 6: Different choices of prompts with their respective QA performance in Prompting Defense experiments. Results reported on 300 random NQ-1500 samples, averaged over the three poisoning settings in table
the fragility of sparse retrieval under deliberate at- tacks.
counterpart factual evidence.
D Human Detection of Generated Misinformation
C Analysis on Corpus Quality
We conduct brief analysis of the two corpora under our study regarding their informativeness to ODQA tasks, shown in Table 8. Since the utiliza- tion of context passages containing gold answers (commonly referred as gold evidence) is critical, we intend to find out about the qualities of gold evidence both corpora. Specifically, we measure question coverage (Recall@) 100), volume (average mentions of answers) and relevance (average rank of the first gold evidence) of gold evidence in the two corpora. Using the ret-vetrieval result of 100 context passages, observe that CovidNews corpus (News) provides significantly less informa- tive evidence for answering questions, making a challenging QA task. Furthermore, these top- ics are more susceptible to plausible assertions at the manipulation of propagandists for a deficit of
We conducted a small-scale human study to ex- plore the detectability of misinformation. We ran- domly sampled 50 fake documents generated un- der CTRI.GNN and 50 corresponding most relevant Wikipedia passages. We employed experimental settings described in (Clark et al., 202 where participants need to rate each text on a 4-point scale. We hired three college students to each an- notate the 100 documents. Similar to their find- ings, we found humans cannot reliably differenti- ate machine machine-gemerated misinformation from their Wikipedia counterparts, with average overall accuracy of 57%. Note that observed great performance improvements in the second half of experiments for all participants, which could mean that 57% is an overestimation human capabilities since the participants displayed signs learning

---

model
NQ-1500
CovidNews
@Q@ 10 pQ@ 100
PQ
@ 10 PQ@ 100
DPR
GENREAD
63.07
92.67
25.49
49.15
CTRLGEN
49.67
82.33
18.64
39.50
REVISE
67.07
91.93
40.42
71.45
REIT
48.93
77.33
54.69
76.47
BM25
GENREAD
57.87
83.00
49.27
65.33
CTRLGEN 29.40
54.63
75.63
86.60
REVISE 72.53
93.53
60.67
82.80
RETT
96.80
99.00
94.40
97.80
Table 7: Evaluation on NQ-1500 passage (test set sam- ples) and CovidNews retrieval. PQ@k refers Poi- soned Questions, which is the percentage of questions that have at least one generated passage in the top-k retrieval result.
Name
Recall @ 100 #Avg. answer #Avg. rank of
mentions
first gold evidence
NQ-1500
82.70
17.07
8.08
CovidNews
57.80
12.04
17.76
Table 8: Properties of the corpora used in this study regarding the question answering tasks.
the sampled data our experiments.


Reviews:
[{"review_id": "czGsJk6eGU", "reviewer": "Reviewer_Qkxs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable insights behind the comments and also want to clarify a few points.\n\n> *The mitigation policy based on QA prompts is not detailed enough. How the prompts have been created?*\n\nWe appreciate the reviewer's feedback and include a more detailed description of our QA prompt creation process as follows. We drew inspiration from an empirical prompt engineering guide [Adversarial Prompting | Prompt Engineering Guide (promptingguide.ai)](https://www.promptingguide.ai/risks/adversarial#defense-tactics). It employs prompting to effectively handle a malicious prompt injection attack, which shares similar traits with our attack settings. Then, we researched relevant papers and based our prompt templates on https://arxiv.org/abs/2305.01579 and https://arxiv.org/abs/2306.10702, with manual revisions to make them suit our needs.\n\n> *The equation reported at line 500 is not formally correct. Probably, if I've understood well, a \"for any value of j\" is missing. What is the effective meaning of the indicator function? Does it measure how many times the answers of passages are equal? It is then not clear how the \"voted answer\" is used to minimize the misinformation.*\n\nThank you for pointing out the error in our equation. The correct version is the one below, with the `argmax` argument changed from `a_i` to `a_j` . It does, measure how many times the $k$ predictions are equal, with the $k$ predictions made based on different partitions of retrieved documents.\n\n$$ a = \\underset{a_j}{\\operatorname{argmax}}\\left(\\sum_{i=1}^{k}\\operatorname{I}(a_i=a_j)\\right) $$\n\n**The mechanism behind the \"voted answer\" is to reduce the influence of any single piece of retrieved evidence while maintaining the machine reader's performance.** Traditionally, machine readers are trained to focus on evidence with a high probability of containing answers in ODQA. This works well when the corpora are thoroughly filtered, but it also makes them vulnerable to unfiltered misinformation. In light of Lewis et al.’s [arxiv.org/pdf/2005.11401.pdf](https://arxiv.org/pdf/2005.11401.pdf) approach that conditioned token prediction on different documents, we experimented with multiple ideas on making the answer prediction less reliant on any single piece of evidence, including re-training, data augmentation, and the voting technique. Ultimately, we found the voting technique is a simple but effective method. \n\n> *To what extent can modern LLMs be utilized for generating credible-sounding misinformation?\" is a very general question that has not been specifically tackled in the paper. The authors demonstrated that generative models can be easily used to create misinformation (showing different generation settings) but not to what extent.*\n\nWe agree with the reviewer that the first research question, \"To what extent can modern LLMs be utilized for generating credible-sounding misinformation?\" is a general research question. It describes the boarder scope of this paper. More precisely, our paper aims to investigate the feasibility of using LLMs to generate misinformation under various generation settings that mimic plausible real-world scenarios. Additionally, we qualitatively assess the impact of such misinformation on downstream NLP tasks like QA. To better align with our focus, we will rephrase the research question in the introduction into: \"What is the feasibility of using LLMs to generate misinformation harmful to downstream NLP applications?\" Thanks for pointing this out.\n\n> *\"What potential harms can arise from the dissemination of neural-generated misinformation in information-intensive applications, such as information retrieval and question-answering? \" Table 7, which is in the appendix, partially gives an answer to the question showing which scenario is more problematic than others but there is no indication about What potential harms can arise from the dissemination. Table 7 basically says that retrieval is affected by the generated language model text, but there aren't any other indications about any other potentially affected task.*\n\nWe acknowledge that \"What potential harms can arise from the dissemination of neural-generated misinformation in information-intensive applications?\" is a general research problem and beyond the scope of a single paper. Therefore, in our paper, we primarily focus on investigating the harm of disseminating neural-generated misinformation to open-domain question-answering, a representative information-intensive application. Our work primarily demonstrated the brittleness of ODQA systems under practical misinformation pollution attacks (in Table 2, and line 421 to line 442), which could be exploited by malicious actors to manipulate the information presented to users. We will add a clarification in the camera-ready version to better reflect our focus. Thanks for pointing this out. \n\nRegarding the other potentially affected factors, we have shown that retrieval is affected by the generated misinformation. The change in retrieval results can not only harm the ODQA performance, but also it can subsequently influence humans. For instance, the 'features snippets' provided by search engines, which attempt to answer user queries directly by extracting from relevant search results (making them essentially an ODQA service), have been found convincing and influential to humans (https://dl.acm.org/doi/abs/10.1145/3498366.3505766) and have large-scale undesired consequences (https://dl.acm.org/doi/abs/10.1145/3576840.3578323). Since we focus on how neural misinformation can affect ODQA systems, discussing other potentially affected parties (e.g., humans) is beyond the scope. Nonetheless, we will add the above discussions to highlight the boarder implications of the dissemination of neural-generated misinformation in information-intensive applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main reasons to accept are:\n- fair comparison of existing pre-trained generative language models\n- the mitigation strategies demonstrate to effectively contribute to limit the diffusion of misinformation generated by LLM\n", "reasons_to_reject": "The main reasons to reject are listed as follows:\n\n1) The mitigation policy based on QA prompts is not detailed enough. How the prompts have been created?\n\n2) The equation reported at line 500 is not formally correct. Probably, if I've understood well, a \"for any value of j\" is missing. What is the effective meaning of the indicator function? Does it measure how many times the answers of passages are equal? \nIt is then not clear how the \"voted answer\" is used to minimize the misinformation. \n\n3) The three research questions listed in the introduction are not properly addressed. In particular: \n- \"(1) To what extent can modern LLMs be utilized for generating credible-sounding misinformation?\" is a very general question that has not been specifically tackled in the paper. The authors demonstrated that generative models can be easily used to create misinformation (showing different generation settings) but not to what extent.  \n- (2) \"What potential harms can arise from the dissemination of neural-generated misinformation in information-intensive applications, such as information retrieval and question-answering? \" Table 7, which is in the appendix, partially gives an answer to the question showing which scenario is more problematic than others but there is no indication about What potential harms can arise from the dissemination. Table 7 basically says that retrieval is affected by the generated language model text, but there aren't any other indications about any other potentially affected task. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper investigates how generative language models can be used to generate credible misinformation. Two main scenarios have been addressed, i.e. intentional and unintentional, about misinformation creation. The authors experimentally evaluated the risk of fake information creation and proposed three mitigation strategies named misinformation detection, vigilant prompting, and reader ensemble. \nThe obtained results confirmed what was previously demonstrated in similar papers, highlighting however that a few strategies can mitigate in a very good way misinformation creation. \n", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3CaE2ynTyA", "reviewer": "Reviewer_UJMA", "summary": "", "questions": "Watermarking is also being deployed, whereby the model leaves a mark on the generated text which can then be utilized to distinguish it from a human generated text. Do you think such information could be utilized to build defenses?\n\nHow easy or difficult it is for humans to identify a document generated by your method? Did you perform any study with regard to this?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's valuable insights behind the comments and also want to clarify a few points.\n\n> *Given the ever-improving generative and instruction-following capabilities of LLMs, it is intuitive that they come up with such documents supporting misinformation. Hence, I am a bit unconvinced about the novelty of this paper.*\n\nWhile it's true that the generative abilities of LLMs are well-known, we argue that the community has not yet thoroughly explored the **implications** of these capabilities, especially whether the generative abilities can be misused to produce misinformation and pollute the corpus used for knowledge-intensive applications such as QA. Our work highlights potential vulnerabilities in ODQA systems when exposed to both unintentional and deliberate misinformation. We also offer preliminary attempts to address this issue, laying the groundwork for more in-depth follow-up research. Therefore, our contribution is not merely an investigation into the misinformation-generating capacities of LLMs, but more on a comprehensive study of **the implication of misinformation pollution and the potential countermeasures.**\n\n> *The prompts also seem to be straightforward. That LLMs are good at generating authoritative text is already well established.*\n\nAs suggested by the paper title, the motivation of our study is to uncover **the risk of misinformation pollution** amplified by the widely use of LLMs. Our choice of using \"straightforward\" prompts was intentional, aiming to mimic real-world interactions between users or potential attackers and LLMs, as detailed in Lines 193-204 of our paper. The fact that even these simple prompts can lead to the generation of convincing misinformation accentuates the urgency of this issue. The attackers do not need advanced prompting skills to exploit LLMs for misinformation creation, posing further risks to downstream applications. This reiterates not only the likelihood but also the significance of information pollution, emphasizing its necessity as a topic for serious attention.\n\n> *I would imagine that in addition to documents with misinformation, there will be a lot of supporting documents containing the true information. It could be that one needs to inject a lot more noisy documents for the systems to generate the wrong answer. From Table 1, it looks like a small fraction is enough to worsen the performance of the ODQA model. Maybe it is worth investigating what happens when you continue to add more and more noisy documents. Further investigations are required.*\n\nWe thank the reviewer for raising this interesting point on the potential presence of both supporting and contradictory documents in the input data. We agree that the extent of the performance degradation could depend on the input composition. The following table shows the QA performance (DPR+FiD ODQA system) on the NQ-1500 dataset under the CtrlGen setting, with different numbers of noisy documents generated per question and the different number of documents utilized by the reader.\n\n|              | 1 noisy docs | 3 noisy docs |\n| ------------ | ------------ | ------------ |\n| 100 contexts | 42.27        | 31.73        |\n| 10 contexts  | 36.93        | 29.20        |\n\n**Our preliminary experiments showed that the larger the portion of misinformation in the retrieved documents, the worse the ODQA performance in general.** We will add the above additional results into the camera-ready version and further conduct a more large-scale analysis of how varying levels of noise in input data might influence the prevalence of misinformation. \n\n> *Watermarking is also being deployed, whereby the model leaves a mark on the generated text which can then be utilized to distinguish it from a human-generated text. Do you think such information could be utilized to build defenses?*\n\nWe concur with the reviewer that watermarking is an emerging and promising technique to distinguish AI-generated texts. Unfortunately, the first watermarking work for LLM (https://arxiv.org/abs/2301.10226) was published after the submission of this paper, so we haven't had a chance to explore it. Nonetheless, we believe watermarking could potentially provide an additional signal to distinguish model-generated text from human-generated text, contributing to potential defense strategies against the misuse of generated content. Compared to neural network classifiers we experimented with in section 5, we believe watermarked machine-generated text would be more successfully and reliably detected, according to some recent works https://arXiv.org/abs/2305.08883. However, currently, we must know the type of watermark beforehand to detect watermarked text, making it impractical at this moment (various LLMs and no unified watermarking in practice). This practicality problem is similar to the problem of neural network classifiers, which we discussed in Lines 517 - 522. We appreciate the reviewer's insightful suggestion and we aim to explore watermarking as a potential defense technique in the camera-ready version, if time permits.\n\n> *How easy or difficult it is for humans to identify a document generated by your method? Did you perform any study with regard to this?*\n\nAs for the ease of human identification of documents generated by our method, we conducted a small-scale human study to explore the detectability of misinformation. We randomly sampled 50 fake documents generated under CtrlGen and 50 corresponding most relevant Wikipedia passages. We employed experimental settings described in Clark et al. (https://aclanthology.org/2021.acl-long.565.pdf,), where participants need to rate each text on a 4-point scale. We hired three college students to each annotate the 100 documents. Similar to their findings, we found humans cannot reliably differentiate machine-generated misinformation from their Wikipedia counterparts, with an average overall accuracy of 57%. Note that we observed great performance improvements in the second half of experiments for all participants, which could mean that 57% is an overestimation of human capabilities since the participants displayed signs of learning the sampled data in our experiments. We will include this experiment in the camera-ready version. We thank the reviewer for pointing this out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Deals with an interesting and timely problem. The paper is also nicely written and easy to follow.\n\nExperiments are performed on real-world datasets.\n\nProposes defense strategies to combat misinformation.\n\n", "reasons_to_reject": "Given the ever improving generative and instruction following capabilities of LLMs, it is intuitive that they come up with such documents supporting misinformation. Hence, I am a bit unconvinced about the novelty of this paper. The prompts also seem to be straightforward. That LLMs are good at generating authoritative text is already well established. \n\nI would imagine that in addition to documents with misinformation, there will be a lot of supporting documents containing the true information. It could be that one needs to inject a lot more noisy documents for the systems to generate the wrong answer. From table 1, it looks like a small fraction is enough to worsen the performance of the ODQA model. Maybe it is worth investigating what happens when you continue to add more and more noisy documents. Further investigations are required.", "questions_to_authors": "Watermarking is also being deployed, whereby the model leaves a mark on the generated text which can then be utilized to distinguish it from a human generated text. Do you think such information could be utilized to build defenses?\n\nHow easy or difficult it is for humans to identify a document generated by your method? Did you perform any study with regard to this?", "paper_topic_and_main_contributions": "The paper investigates if large language models (LLMs) can be deployed to generate misinformation. In specific, the authors consider the task of open domain question answering (ODQA) and utilize the generative capability of LLMs to generate noisy documents that can mislead an ODQA model to come up with the wrong answer. Experiments on two real world datasets reveal that introducing these noisy documents can negatively influence the performance of the models. Additionally, the paper also comes up with some simple defense strategies to combat such misinformation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2hNGuBFTT5", "reviewer": "Reviewer_k9JU", "summary": "", "questions": "How do you decide on the number of fake texts injected to the clean corpora for each setting?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's valuable insights behind the comments and also want to clarify a few points.\n\n> *The paper only focuses on one specific type of ODQA systems (retriever-reader). However, there lacks analysis on other types of ODQA systems e.g. retriever-reader.*\n\nWe appreciate the reviewer's feedback. We would like to clarify that the retrieve-and-read is the dominant architecture for existing ODQA systems, following the advent of the first neural QDQA system DrQA (https://arxiv.org/pdf/1704.00051.pdf). Although they all follow the general pipeline of retrieve-and-read, the specific implementations for retriever and reader make them different models. In our work, we have investigated a variety of mainstream architectures, including two main types of retrievers (sparse vs. dense) and the two major types of readers (extractive reader vs. generative reader) in our paper (please see Section 4.2, line 308 to line 338 for details). \n\nThe reviewer mentioned we \"lack analysis on other types of ODQA systems e.g. retriever-reader.\" However, we are actually using the retriever-reader architecture. We guess this might be a typo and the reviewer actually means \"retriever-generator\"? If this is the case, we want to clarify that we have used GPT-3.5 (Lines 327 - 338) as the reader, which represents the retriever-generator architecture. \n\n> *How do you decide on the number of fake texts injected to the clean corpora for each setting?*\n\nThank you for your insightful question regarding the determination of the number of fake texts injected into the clean corpora for each setting. **There are two factors, generating and processing, determining the final numbers of passages.**\n\nThe decision to generate what amount of documents was based on a combination of factors, including the desired level of noise, the distribution of these fake texts in relation to the original data, and the scale of the experiment. To elaborate further, we conducted research reviews and preliminary experiments to identify a reasonable proportion of synthetic text in the ODQA corpora in order to simulate real-world misinformation pollution levels. We determined to generate one fake document for each question for each setting, as the resulting percentage of fake documents in the original corpora would be small (~0.01%), aligning with the estimated percentage of feasible corruption in online data made in this work https://arxiv.org/abs/2302.10149.\n\nThe specific passages injected into corpora are processed from the generated documents following standard practices in ODQA. Since the passages were split by 100 words in our case, the generated document lengths determine the amount of injected passages, which we showed in Table 1. We did not further normalize the generated text lengths for two reasons: 1) It would be hard to keep the generated features (word choices, semantics) intact, and 2) We found that usually only one passage in the generated document is retrieved, regardless of document lengths (how many passages one document forms). Therefore, we decided to keep all passages and report the numbers in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an interesting and important issue of potential risks associated with LLMs and their impact on information-intensive applications.\n2. The paper proposes a threat model which is intuitive and straightforward. The threat model provides notable misleading results in ODQA.\n3. The paper presents preliminary experimental results to evaluate the effectiveness of the proposed defense strategies, which can help guide future research and development efforts.", "reasons_to_reject": "The paper only focuses on one specific type of ODQA systems (retriever-reader). However, there lacks analysis on other types of ODQA systems e.g. retriever-reader. ", "questions_to_authors": "How do you decide on the number of fake texts injected to the clean corpora for each setting?", "paper_topic_and_main_contributions": "This paper discusses the potential risks of misinformation pollution with LLMs and its impact on ODQA systems. The paper establishes a threat model to assess the extent to which LLMs can be utilized to produce disinformation. It also proposes three defense strategies to mitigate the negative impacts of misinformation pollution.\n", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explore how Large Language Models (LLMs) can be misused to generate misleading or false information that can affect Open-Domain Question Answering (ODQA) systems. Simulate different scenarios of misinformation generation and measure the impact on ODQA performance. Propose some defense strategies to detect and prevent misinformation from LLMs. Call for more research and collaboration to address the challenges of LLM-generated misinformation and to ensure the ethical use of these models.
Decision: Accept-Findings