RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Open-domain Question Answering.
To answer a question, open-domain QA systems employ a retriever-reader paradigm that first retrieves relevant documents from a large evidence corpus and then predicts an answer conditioned on the retrieved documents. Promising advances have been made towards improving the reader models Yang et al. ( 2019 ); Izacard and Grave ( 2021 ) and neural retrievers Lee et al. ( 2019 ); Guu et al. ( 2020 ); Santhanam et al. ( 2022b ) . However, since Wikipedia is used as the evidence corpus, previous works take for granted the assumption that the retrieved documents are trustworthy. This assumption becomes questionable with the rapid growth of fake and misleading information in the real world. In this work, we take the initiative to study the potential threat that misinformation can bring to QA systems, calling for a new direction of building misinformation-immune QA systems.
Improving Robustness for QA.
Our work aims to analyze vulnerabilities to develop more robust QA models. Current QA models demonstrate brittleness in different aspects. QA models often rely on spurious patterns between the question and context rather than learning the desired behavior. They might ignore the question entirely (Kaushik and Lipton, 2018 ) , focus primarily on the answer type (Mudrakarta et al., 2018 ) , or ignore the “intended” mode of reasoning for the task (Jiang and Bansal, 2019 ; Niven and Kao, 2019 ) . QA models also generalize badly to out-of-domain (OOD) data (Kamath et al., 2020 ) . For example, they often make inconsistent predictions for different semantically equivalent questions (Gan and Ng, 2019 ; Ribeiro et al., 2019 ) . Similar to our paper, a few prior works Chen et al. ( 2022 ); Weller et al. ( 2022 ); Abdelnabi and Fritz ( 2023 ) investigated the robustness of QA models under conflicting information. For example, Longpre et al. ( 2021 ) shows QA models are less robust to OOD data where the contextual information contradicts the learned information. Different from these works, we study from a new angle of QA robustness: the vulnerability of QA models under misinformation.
Combating Neural-generated Misinformation.
Advanced text-generation models offer a powerful tool for augmenting the training data of downstream NLP applications Pan et al. ( 2021 ); Chen et al. ( 2023 ) . However, these models also pose a risk of being exploited for malicious activities, such as generating convincing fake news (Zellers et al., 2019 ) , fraudulent online reviews (Garbacea et al., 2019 ; Adelani et al., 2020 ) , and spam. Even humans find it struggle to detect such synthetically-generated misinformation Clark et al. ( 2021 ) . When produced at scale, neural-generated misinformation can pose threats to many NLP applications. For example, a recent work by (Du et al., 2022 ) finds that synthetic disinformation can significantly affect the behavior of modern fact-checking systems. In this work, we study the risk of neural-generated misinformation to QA models.
Misinformation Documents Generation
We simulate the potential vulnerability of question-answering models to corpus pollution with misinformation documents by injecting both human-written and model-generated false documents into the evidence corpus, and assess the impact on the performance of these systems. We base our study on the SQuAD 1.1 Rajpurkar et al. ( 2016 ) dataset, one of the most popular benchmarks for evaluating QA systems. We use all the 2,036 unique Wikipedia passages from the validation set for our study.
For each Wikipedia passage \mathcal{P}^{R} , we create a set of N fake passages (\mathcal{P}^{F}_{1},\cdots,\mathcal{P}^{F}_{N}) by modifying some information in \mathcal{P}^{R} , with the requirement that each fake passage look realistic while containing contradicting information with \mathcal{P}^{R} .

We use two different ways to create fake passages: 1) via human edits : we ask online workers from Amazon Mechanical Turk (AMT) to produce fake passages by modifying the original passage, and 2) via BART-FG : our novel generative model BART-FG , which iteratively masks and re-generates text spans from the original passage to produce fake passages.
Manual Creation of Fake Passages
To solicit human-written deceptive fake passages, we release 2K HITs (human intelligence tasks) on the AMT platform, where each HIT presents the crowd-worker with one passage \mathcal{P}^{R} in the SQuAD validation set. We ask workers to modify the contents of the given passage to create a fake version, following the below guidelines:

\bullet The worker should make at least M edits at different places, where M equals to one plus the number of sentences in the contexts \mathcal{C}^{R} .

\bullet The worker should make at least one long edit that rewrites at least half of a sentence.

\bullet The edits should modify key information to make it contradict with the original , such as time, location, purpose, outcome, reason, etc.

\bullet The modified passage should be fluent and look realistic , without commonsense errors.

To select qualified workers, we restrict our task to workers who are located in five native English-speaking countries 2 2 2 Australia, Canada, Ireland, United Kingdom, USA , and who maintain an approval rating of at least 90%. To ensure the annotations fulfil our guidelines, we give ample examples in our annotation interface with detailed explanations to help workers understand the requirements. The detailed annotation guideline is in Appendix A . We also hired three computer science major graduate students as human experts to validate a HIT’s annotation. In the end, 104 workers participated in the task. The average completion time for one HIT is 5 minutes, and payment is $1.0 U.S. dollars/HIT. The average acceptance rate was 93.75%.
Model Generation of Fake Passages
Aside from human-written misinformation, we also want to explore the threat of machine-generated misinformation to QA. This source may be more of a concern than human-created misinformation, since they can easily be produced at scale. Recently introduced large-scale generative models, such as GPT2 (Radford et al., 2019 ) , BART (Lewis et al., 2020 ) , and Google T5 (Raffel et al., 2020 ) , can produce realistic-looking texts, but they do not lend themselves to producing controllable generation that only replaces the key information with contradicting contents. Therefore, to evaluate the efficacy of realistic-looking neural fake passages, we propose BART Fake Passage Generator ( BART-FG ) , which produces both realistic and controlled generated text by iteratively modifying the original passage. As shown in Figure 2 , for each sentence S of the original passage, BART-FG produces its fake version S^{\prime} via a two-step process:

1) Span Masking . We first obtain a set of candidate text spans from the input sentence. We then randomly select a span and replace it with a special mask token [MASK] . We employ two different ways to get the candidate spans. 1) NER : we use Spacy 3 3 3 https://spacy.io/usage/linguistic-features#named-entities to extract name entities as the candidate spans. 2) Constituency : we apply the constituency parser implemented in AllenNLP 4 4 4 https://demo.allennlp.org/constituency-parsing to extract constituency spans from the input sentence as the candidate spans. We choose to mask named entities / constituency phrases instead of random spans because: 1) they represent complete semantic units such as “Super Bowl 50”, which avoids meaningless random phrases such as “Bowl 50”; and 2) they often represent important information in the sentence — such as time, location, cause, etc.

2) Span Re-generation . We fill in the mask by generating a phrase different from the masked phrase. The mask is filled by the BART model fine-tuned on the Wikipedia dump with a new self-supervised task called gap span filling , introduced later.

The above pipeline is iteratively run for K times to generate sentence S^{\prime} from S . We choose to make the edits iteratively rather than in parallel to model interaction between multiple edits. For example, in Figure 2 , if the previous edit changes “Santa Clara” to “Atlanta”, the next edit can choose to change “California” into “Georgia” to make the contents more consistent and realistic.
Gap Span Filling (GSF) Pre-Training.
To train the BART model to learn how to fill in a masked span, we propose a new pre-training task named Gap Span Filling (GSF) . For each article in the Wikipedia dump that consists of T sentences [S_{1},S_{2},\cdots,S_{T}] , where each sentence is a word sequence S_{t}=[w^{t}_{1},\cdots,w^{t}_{|S_{t}|}] , we construct the following training data for t=2,\cdots,T-1 :

Input: S_{1},S_{t-1},w^{t}_{1:a-1},\texttt{[MASK]},w^{t}_{b+1:|S_{t}|},S_{t+1} Output: w^{t}_{a:b}=[w^{t}_{a},\cdots,w^{t}_{b}]

where the output represents a masked constituency or named entity span that starts with the a -th word and ends with b -th word. The input is the concatenation of the first sentence S_{1} , the previous sentence S_{t-1} , the current sentence S_{t} with one span being masked, and the subsequent sentence S_{t+1} . The BART model is fine-tuned to predict the output given the input on the entire Wikipedia dump. This task trains the BART model to predict the masked constituency / named entity span, given both global contexts ( S_{1} ) and local contexts ( S_{t-1},S_{t+1} ). We use the facebook/bart-large model provided by Hugging Face (406M parameters).
Analysis of the Generated Fake Passages
Table 1 shows examples from six original passages with their corresponding fake versions, which represent six common types of modifications made by the human and the model, explained as follows:

(1) Entity Replacement : replacing entities ( e.g. , person, location, time, number) with other entities with the same type, a common type of modification for both human edits and BART-FG .

(2) Verb Replacement : replacing verb or verb phrase with its antonymic meaning, e.g. , “force these children to” \rightarrow “prevent these children from”.

(3) Adding Restrictions : create contradiction by inserting additional restrictions to the original content, e.g. , “every day” \rightarrow “every day but Sunday”.

(4) Sentence Rephrasing : rewrite the whole sentence to express a contradicting meaning, exemplified by (4). This is common in human edits but rarely seen in model-generated passages, since this requires deep reading comprehension.

(5) Disrupting Orders : make a contradiction by disrupting some property of the entities; e.g. , example (5) switches the property of “analysis of algorithms” and “complexity theory”.

(6) Consecutive Replacements : humans are better in making consecutive edits to create a contradicting yet coherent sentences, exemplified by (6).
Corpus Pollution with Misinformation
Given the fake passages curated by both human and our BART-FG model, we now study how extractive QA models behave under an evidence corpus that is polluted with misinformation. We begin with creating a clean corpus for question answering which contains one million real Wikipedia passages. We obtain the Wikipedia passages from the 2019/08/01 Wikipedia dump provided by the Knowledge-Intensive Language Tasks (KILT) benchmark Petroni et al. ( 2021 ) , in which the Wikipedia articles have been pre-processed and separated into paragraphs. We sample 1M paragraphs from KILT and ensure that all the 20,958 Wikipedia passages in the SQuAD dataset are included in the corpus. We then explore the following five ways of polluting the clean corpus with human-created and synthetically-generated false documents.

\bullet Polluted- Human . In Section 3.1 , we asked human annotators to create a fake version for each passage in the SQuAD dev set. We inject those 2,023 fake passages into the clean corpus.

\bullet Polluted- NER . We use BART-FG to generate 10 fake passages for each real passage in the SQuAD dev set, using NER to get candidate spans. We mask and re-generate all candidate spans to create each fake passage. Nucleus sampling Holtzman et al. ( 2020 ) is used to ensure diversity in generation, giving us 18,233 non-repetitive fake passages in total. We inject them into the clean corpus.

\bullet Polluted- Constituency . We generate 10 fake passages for each real passage using constituency parsing to get candidate spans in BART-FG . Since there are far more constituency phrases than named entities in a sentence, to ensure efficiency, we fix the number of replacements K=3 for each sentence. We get 19,796 non-repetitive fake passages and inject them into the clean corpus.

\bullet Polluted- Hybrid . We inject all of the above-generated fake passages into the clean corpus.

\bullet Polluted- Targeted .
In the above settings, the attacker (human or BART-FG model) tries to create misleading fake information without knowing the target questions. However, in another attack mode, attackers have particular questions of interest that they want to mislead the QA system into getting wrong answers.
To explore how QA systems react to such attacks, in this setting we assume the attacker targets the questions in the SQuAD dev set. We then create fake passages by masking and re-generating the answer spans of these questions using BART-FG . Through this, we get 10,101 fake passages and insert them into the clean corpus.
Models and Experiments
We now how question answering models behave under such misinformation-polluted environment. To answer a given question, the QA systems employ a retrieve-then-read pipeline that first retrieves N (we set N=5 ) relevant contextual documents from the evidence corpus and then predicts an answer conditioned on the retrieved documents. For document retrieval, we apply the widely-used sparse retrieval based on BM25, implemented with the Pyserini toolkit Lin et al. ( 2021 ) . For question answering, we consider five state-of-the-art QA models with public code that achieved strong results on the public leader board of SQuAD: RoBERTa-large (Liu et al., 2019 ) , Span-BERT (Joshi et al., 2020 ) , Longformer (Beltagy et al., 2020 ) , ELECTRA (Clark et al., 2020 ) , and DeBERTa-V3 (He et al., 2023 ) . We use their model checkpoints fine-tuned on the SQuAD training set from the Hugging Face library. We use the standard Exact Match (EM) and F_{1} metrics to measure QA performance.
Main Results
In Table 2 , we show the performance of different QA models on the SQuAD dev set under the clean evidence corpus ( Clean ) and the performance under the misinformation-polluted corpus ( Polluted ). We have two major observations.

For all models, we see a noticeable performance drop when generated fake passages are introduced into the clean evidence corpus: the smallest average performance drop is 7.72% in relative EM value (Polluted- Human ), while the largest drop is 53.19% (Polluted- Targeted ). This indicates that QA models are sensitive to misinformation attack ; even limited amounts of injected fake passages comprising 0.2% ( Human ) to 4.0% ( Hybrid ) of the entire corpus can noticeably affect downstream QA performance.
It reveals the potential threat of misinformation to current QA systems, given the fact that they are not trained to differentiate misinformation.

Polluted- Targeted causes a more significant performance drop compared to the most effective question-agnostic attack (Polluted- Hybrid ) ( \sim 53% v.s. \sim 22% relative EM drop), indicating that QA models are more vulnerable under question-targeted misinformation attack. This reveals that the misinformation attack brings more threat when the attacker wants to alter the answers produced by QA systems for particular questions of interest.
For the other four question-agnostic settings where the pollution is not targeted on specific questions, we still observe a noticeable EM drop ( \sim 20%) for all models. Among them, Polluted- NER causes more performance drop than Polluted- Constituency , showing that generating misinformation by replacing named entities is more effective than replacing constituency spans. This is probably due to the nature of the SQuAD dataset, where most of the answer spans are named entities.
Impact of misinformation on retriever
The success of the misinformation attack relies on the premise that fake passages can be retrieved from the polluted corpus by the retriever. To validate this, we first define a fake passage P as the misleading evidence for the question Q if P contains a fabricated answer for Q . We then report in Table 3 the percentage of misleading evidence in the top- k retrieved passages (F@k, for k\in\{1,5\} ) for the BM25 retriever. We find that both F@1 and F@5 are very high, while the likelihood of the ground-truth true evidence appearing in the top-1 (R@1) and top-5 (R@5) decreases significantly for polluted corpus. The results show that the injected fake passages can be easily retrieved as evidence for downstream question answering. QA models, without the fact-checking capability, can thus be easily misled by such misinformation.

However, BM25 only relies on syntactic features and cannot be optimized for specific tasks. Is the misinformation attack also effective for trainable dense retrievers? To explore this, we use ColBERT-V2 Santhanam et al. ( 2022a ) , the state-of-the-art dense retriever that independently encodes the question and the passage using BERT and then employs a late interaction architecture to model their similarity.
We use the ColBERT pretrained on MSMARCO Nguyen et al. ( 2016 ) and fine-tune it with (question, context) pairs from SQuAD training set as positive samples and (question, random context) as negative samples. The retrieval and QA performance are reported in Table 3 .

We find that misinformation attack also affects the ColBERT retriever, decreasing R@1 and R@5 for all settings, with high percentage of fake passages being retrieved as reflected by F@1 and F@5. The results also suggest that ColBERT is less resistant to misinformation attack compared to BM25. In the clean corpus, ColBERT outperforms BM25 in both the retrieval and the downstream QA performance. However, in all polluted corpus, the relative performance drop for ColBERT is larger than the drop for BM25. The possible explanation is: without the ability to identify fake information, a more “accurate” retriever tends to retrieve more seemingly relevant but false documents, making it less robust to misinformation attack.
Impact of the size of injected fake passages
As confirmation that misinformation attacks work as expected, we depict in Figure 3 how the DeBERTa model performance changes when different number of fake passages are injected into the evidence corpus. We find that the EM score steadily drops with more fake passages for both the question-targeted attack ( Targeted ) and the question-agnostic attack ( NER ). However, the former causes a much sharper trend of decrease, which further validates that misinformation attack is more deadly with a better knowledge of the target questions.
Through this study, we conclude that misinformation may have a more severe impact on QA systems when they are produced at scale . With the availability of pretrained text generation models, producing fluent and realistic-looking contexts now has a little marginal cost. This brings an urgent need to effectively defend against neural-generated misinformation in question answering.
Which is more deceiving: human- or model-generated misinformation?
We then investigate which is more deceiving to QA models: human or neural misinformation?
To study this, we let the QA model to answer each question Q under the context \mathcal{C}=\{\mathcal{P^{R},P^{H},P^{C},P^{N}}\} , where P^{R} is the real passage that contains the correct answer, and \mathcal{P^{H},P^{C},P^{N}} are the corresponding fake versions of P^{R} produced by human, BART-FG (NER), and BART-FG (Constituency), respectively. We then analyze the source (which fake passage) of the incorrect answer when the model makes an error. If all three methods create equally deceiving fake passages, we expect to observe a uniform distribution of the error sources.

The distribution of error sources in Figure 4 shows that the most wrong answers are extracted from the model-generated fake passage. Human-created fake passages do not show an advantage over BART-FG in deceiving the QA models. This is counter-intuitive to what we find in Table 1 that humans make more subtle edits that require a deep level of reading comprehension, such as switching “former” and “latter” (Example 4), and changing “every day” to “every day but Sunday” (Example 3). A possible reason is that most questions in SQuAD are shallow in reasoning Du et al. ( 2017 ) . Therefore, replacing named entities/constituency phrases is sufficient in misleading QA models into getting the wrong answers for those questions.
Can misinformation deceive humans?
After showing the impact of misinformation attacks on QA systems, one natural question would be whether humans can also be distracted by misinformation during QA. To investigate this, we ran a study on Mechanical Turk where we presented crowd-workers with 500 randomly-sampled (question, context) pairs from the data in Section 5.4 , i.e. , each context consists of the real passage along with three fake passages created by different methods. We call this test set MisinfoQA-noisy and the workers are asked to answer each of its questions. For comparison, we create another test set MisinfoQA-clean where each real passage is paired with three randomly sampled other Wikipedia passages.

Table 4 reports the EM and F1 for both human and different QA models. We find that all QA models suffer a large performance drop ( \sim 20% in EM) in MisinfoQA-noisy compared to MisinfoQA-clean , showing that the models are largely distracted by the fake contexts rather than by the presence of additional contexts. Humans obtained an EM of 69.13 in MisinfoQA-noisy , which, though higher than most QA models’ performance, also shows a significant drop when compared to the MisinfoQA-clean setting (86.57 EM). This shows that humans are also likely distracted by misinformation in QA , which demonstrates the challenge of distinguishing misinformation in question answering for lay readers, the quality of the generated fake passages, and the difficulty of detecting such an attack.
Discussion and Future Work
Finally, we discuss three possible ways to defend the threat of misinformation for QA.

Knowledge source engineering. Despite being a trustful knowledge source, Wikipedia is insufficient to fulfill all the information needed in real-life question answering. Therefore, recent works Piktus et al. ( 2021 ) started to use the web as the QA corpus. However, when transitioning to a web corpus, we no longer have the certainty that any document is truthful.
Therefore, the corpora will require more careful curation to avoid misinformation. This also brings the need for future retrieval models to have the ability to assess the quality of the retrieved documents and prioritize more trustworthy sources.

Integrating fact-checking and QA. With the rise of misinformation online, automated fact-checking has received growing attention in NLP Guo et al. ( 2022 ) . Integrating fact-checking models into the pipeline of open-domain QA could be an effective countermeasure to misinformation, a direction neglected by prior works. A possible way is to detect potential false claims in retrieved contexts and lower their importance in downstream QA models.

Reasoning under contradicting contexts. It is common for humans to deal with contradictory information during information search. With the presence of inaccurate and false information online, future models should focus on the ability to synthesize and reason over contradicting information to derive correct answers.