RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Conclusion
In this work, we evaluate the robustness of open-domain question-answering models when we contaminate the evidence corpus with misinformation. We studied two representative sources of misinformation: human-written disinformation and the misinformation-generated NLG models. Our studies reveal that QA models are indeed vulnerable under misinformation-polluted contexts. We also show that our BART-FG model can produce fake documents at scale that are as deceptive as humans. This poses a threat to current open-domain QA models in defending neural misinformation attacks.
Limitations
We identify two main limitations to our study. First, although SQuAD is a typical dataset for evaluating open-domain QA models, most of the SQuAD questions are factoid and shallow in reasoning, making it relatively easy to generate misinformation targeted at SQuAD. Our results show that BART-FG with named entity replacement can generate fake passages as deceptive as humans. However, the impact of model-generated misinformation may be over-estimated on the shallow factoid questions in SQuAD. Therefore, more QA datasets should be considered in future works, especially non-factoid questions with deeper reasoning.

Second, this work creates misinformation by revising key information of real articles in Wikipedia. However, there are other types of misinformation in the real world, such as hoaxes, rumors, or false propaganda. However, our proposed attack model can be easily generalized to study the threat of misinformation in other domains and in other forms.
Ethics Statement
We plan to publicly release the human- and model-generate fake documents and open-source the code and model weights for our BART-FG model. We note that open-sourcing the BART-FG model may bring the potential for deliberate misuse to generate disinformation for harmful applications. The human-written and model-generated fake documents can also be misused to generate disinformation. We deliberated carefully on the reasoning for open-sourcing and share here our three reasons for publicly releasing our work.

First, the danger of BART-FG in generating disinformation is limited. Disinformation is a subset of misinformation that is spread deliberately to deceive. Although we utilize the innate “hallucination” ability of current pretrained language models to create misinformation, our model are not specialized to generate harmful disinformation such as hoaxes, rumors, or false propaganda. Instead, our model focuses on generating conflicting information by iteratively editing the original passage to test the robustness of QA to misinformation.

Second, our model is based on the open-sourced BART model, which makes our model easy to replicate even without the released code. Given the fact that our model is a revised version of an existing publicly available model, it is unnecessary to conceal code or model weights.

Third, our decision to release follows the similar stance of the full release of another strong detector and state-of-the-art generator of neural fake news: Grover (Zellers et al., 2019 ) 5 5 5 https://thegradient.pub/why-we-released-grover/ . The authors claim that to defend against potential threats, we need threat modeling, in which a crucial component is a strong generator or simulator of the threat. In our work, we build an effective threat model for QA under misinformation. Followup research can build on our model transparency, further enhancing the threat model.
Acknowledgements
This work was supported by the National Science Foundation Award #2048122. The views expressed are those of the authors and do not reflect the official policy or position of the US government.
A.1 Job Description
Given a paragraph from Wikipedia, modify some information in the paragraph to create a fake version of it. Here are the general requirements:

\bullet You should make at least M edits at different places, where M is determined by the length of the passage and will show on the screen when you annotate each passage.

\bullet You should make at least one long edit that rewrites at least half of a sentence.

\bullet The edits should modify key information to make it contradict with the original , such as time, location, purpose, outcome, reason, etc.

\bullet The modified paragraph should be fluent and look realistic , without commonsense errors.
A.2 Detailed Requirements
Figure 5 shows an example of modifications that fulfill all the annotation requirements. Detailed annotation instructions are as follows.
) At least make N edits at different places.
In the above example, there are a total of 5 edits:

“an American football game” \rightarrow “the 48th Super Bowl Game”

“Denver Broncos” \rightarrow “San Francisco 49ers”

“on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California.” \rightarrow “Mercedes-Benz Superdome in New Orleans, Louisiana and was the first Super Bowl to be played in the United States.”

“the 50th” \rightarrow “the NFL’s 48th”

“so that the logo could prominently feature the Arabic numerals 50.” \rightarrow “so that the game would be known as the "Super Bowl of the Century.”
) There should be at least one long edit.
Among all your edits, there should be at least one long edit, which rewrites the whole sentence or at least half of the sentence.

In the above example, the long edit is: “on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California.” \rightarrow “Mercedes-Benz Superdome in New Orleans, Louisiana and was the first Super Bowl to be played in the United States.”
) The edits should create contradicting information.
After your edits, the original passage and the modified passage should have contradicting information. One way to test it is that: when you ask questions about your modified information, the original passage and the modified passage gives contradicting answers.

For example: after you edit “Denver Broncos” to “San Francisco 49ers”, the original and modified passages are shown in the Figure below:

Original Text:

The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to earn their third Super Bowl title.

Modified Text:

The American Football Conference (AFC) champion San Francisco 49ers defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to earn their third Super Bowl title.

When you ask the question: “Which NFL team won Super Bowl 50?”, the original passage gives you the answer “Denver Broncos”, and the modified passage gives you the answer “San Francisco 49ers”. This is a contradiction.

Another example is the following edit: “so that the logo could prominently feature the Arabic numerals 50.” \rightarrow “so that the game would be known as the “Super Bowl of the Century”.

Original Text:

... the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the logo could prominently feature the Arabic numerals 50.

Modified Text:

... the league emphasized the "golden anniversary" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the game would be known as the "Super Bowl of the Century".

When you ask the question: “Why the league suspended the tradition of naming Super Bowls with Roman numerals?” the original passage and the modified passage also give you contradicting answers.

However, the following passage does NOT create any contradiction, because the modified information is just a paraphrasing of the original information.

Original Text:

The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to earn their third Super Bowl title.

Modified Text:

The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24-10 to win the Super Bowl.
) The edits should modify important information in the passage.
Your edits should focus on important information in the passage, i.e. , points that people are usually interested in and would usually ask about. For example, time, location, purpose, outcome, reason, etc. Please avoid editing trivial and unimportant details.

For example, the following trivial edit is not supported:

Original Text:

the game would have been known as "Super Bowl L "...

Modified Text:

the game would have been known as "Super Bowl H "...
) The modified passage should look “realistic”.
The final modified passage should look “realistic”. Don’t make obvious logic or commonsense mistakes to make the reader easily know that this is a fake passage by simply going through it.

For example, the following edit is not supported.

Original Text:

The game was played on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara , California.

Modified Text:

The game was played on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at New York City , California.

People can easily tell the modified passage is fake since everybody knows that New York is not a city in California.
A.3 Annotation Interface
The original passage is shown on the left for your reference, you should modify the passage in the text box on the right to make the fake passage. After you finished the edits, Click “Submit”.
Figure 1: Our framework injects human-created and model-generated misinformation documents into the QA evidence repository (left) and evaluates the impact on the performance of open-domain QA systems (right).; Our framework injects human-created and model-generated misinformation documents into the QA evidence repository (left) and evaluates the impact on the performance of open-domain QA systems (right).
Figure 2: Overview of the BART-FG model, illustrated by an example sentence.; Overview of the BART-FG model, illustrated by an example sentence.
Figure 3: The EM score for DeBERTa-V3 model with different number of injected fake passages N .; The EM score for DeBERTa-V3 model with different number of injected fake passages N .
Figure 4: Distribution of error sources when the model is misled by a fake passage and gives a wrong answer.; Distribution of error sources when the model is misled by a fake passage and gives a wrong answer.
Figure 5: An example of human annotation that follows all instructions of the annotation guideline.; An example of human annotation that follows all instructions of the annotation guideline.
Figure 6: The annotation interface in the Amazon Mechanical Turk.; The annotation interface in the Amazon Mechanical Turk.
Examples of original passages and their corresponding fake versions, where the information changes are highlighted. These examples represent six common types of created misinformation.
Effects of different modes of misinformation attacks on the open-domain QA performance in SQuAD.
Effects of different modes of misinformation attacks on the BM25 and ColBERT-V 2 retrievers.
QA performance under the reading comprehension settings with clean and noisy contexts.