ContraQA: Question Answering under Contradicting Contexts

1 INTRODUCTION . A typical Question Answering ( QA ) system ( Chen et al. , 2017 ; Yang et al. , 2019 ; Karpukhin et al. , 2020 ; Lewis et al. , 2020b ) starts by retrieving a set of relevant context documents from the Web , which are then examined by a machine reader to identify the correct answer . Existing work equate Wikipedia as the web corpus . Therefore , all retrieved context documents are assumed to be clean and trustable . However , real-world QA faces a much noisier environment , where the web corpus is tainted with misinformation . This includes unintentional factual mistakes made by human writers and deliberate disinformation intended to deceive . Aside from human-created misinformation , we are also facing the inevitability of AI-generated misinformation . With the continuing progress in text generation ( Radford et al. , 2019 ; Brown et al. , 2020 ; Lewis et al. , 2020a ) , realistic-looking fake web documents can be generated at scale by malicious actors ( Zellers et al. , 2019 ) . The presence of misinformation — no matter deliberately created or not , no matter human-written or machine-generated — affects the reliability of the QA system by bringing in contradicting information in the context documents . Figure 1 shows a question and five context documents , which give contradicting answers to the question . Only one context document ( in green ) is factually correct , while the rest are human-written or machine-generated fake information . Faced with such contradicting contexts , even human readers must be familiar with “ Super Bowl 50 ” or rely on Web search to invalidate these fake contexts . Although current QA models often achieve super-human performance under the idealized case of clean contexts , we argue that they may easily fail under the more realistic case of contradicting contexts , especially when they do not have the ability to identify fake information and reason over contradicting contexts . We seek to study risks of misinformation to QA models by investigating how QA models behave under contradicting contexts that are mixed with both real and fake information . Since there is no public QA dataset that marks and introduces contradicting contexts , we construct one ourselves : CONTRAQA , a large-scale dataset that specifically serves this need . Our dataset is constructed on top of SQuAD 1.1 ( Rajpurkar et al. , 2016 ) . For each context paragraph P in SQuAD , we create a fake version P ′ by modifying information in P , such that : 1 ) certain information in P ′ contradicts with the information in P , and 2 ) P ′ is fluent , consistent , and looks realistic . We include both human-written and model-generated fake contexts in our dataset to simulate a realistic environment . For the human-written part , we ask Mechanical Turkers to write fake contexts by modifying the original context . For the model-generation part , we propose a strong rewriting model , namely BART-FG , which can controllably mask and re-generate constituency spans in the original context to produce fake contexts . The original contexts , human-written and model-generated fake contexts are mixed together blindly and presented to QA model to answer a given question . A sample of CONTRAQA is shown in Figure 1 , where the model is presented with multiple contradicting contexts to predict the answer for a given question . This stimulates a real-world situation where both real and fake information are retrieved as the context documents for open-domain QA . A robust QA model should be able to deal with misinformation and properly handle contradictory information . Unfortunately , from extensive experiments , we find that existing QA models are vulnerable under contradicting contexts brought about by misinformation , regardless of whether the fake contexts are manually-written or model-generated . State-of-the-art QA models ( Devlin et al. , 2019 ; Liu et al. , 2019 ; Joshi et al. , 2020 ) all suffer from a significant drop in their exact match ( EM ) score by 20–30 when presented with contradicting contexts in CONTRAQA . Our analyses show that 1 ) our proposed context-rewriting BART-FG can create more deceiving misinformation than GPT-2 ( Radford et al. , 2019 ) , and that 2 ) though machine-generated fake contexts are deceiving , they are generally not on par with human-written ones . We find that humans are still much better at making subtle and efficient edits to create contradicting information . To defend against the potential threat of misinformation , we build a more robust QA system that integrates question answering with a misinformation discriminator in a joint fashion . Decisions made by discriminator assist QA models in identifying likely misinformation , which in turn improves the EM score by 17.2 % , assuming access to a sufficient level of training data . We plan to release CONTRAQA publicly , helping pave the way for building more robust QA systems . Although there are multiple types of misinformation in the real world , such as hoaxes , rumors , or false propaganda , we focus on the misinformation that brings conflicting information to the QA contexts . However , our work gives a general threat model for QA under misinformation , including an attack model that mixes both real and fake information in the QA context , and a defense model that combines question answering and misinformation detection . Followup research can easily build upon this threat model by studying how other types of misinformation mislead the QA systems . We summarize our contributions as follows : • To the best of our knowledge , this is the first work to investigate QA under contradicting contexts . •We construct a large-scale dataset CONTRAQA that includes contradicting contexts produced by both humans and neural models . •We propose BART-FG , a novel framework that generates fake contexts by iteratively modifying constituency spans of the original context . • To defend against the threat of misinformation , we propose a model that unifies question answering and misinformation discrimination . 2 RELATED WORK . Adversarial Attacks in QA . Adversarial attacks aim to trick QA models intentionally by perturbing their input contexts . These perturbations are expected to preserve the meaning of contexts while degrading model performance . Ribeiro et al . ( 2018 ) make grammar perturbations such as replacing What has with What ’ s . Other perturbations include adding distractor sentences ( Jia & Liang , 2017 ; Wang & Bansal , 2018 ) , modifying phrases ( Maharana & Bansal , 2020 ) , and human-in-theloop edits ( Bartolo et al. , 2020 ) . Although we also make modifications to the QA contexts , our setting differs from the task of adversarial attacks in QA . First , we aim to make the modified contents contradict with the original paragraph , while adversarial perturbations are expected to preserve the original meaning . Second , the goal of generating contradicting contexts is to study the impact of misinformation on QA models . In contrast , adversarial contexts aim to reveal the weaknesses of systems upon local changes that are imperceptible by humans . Improving Robustness for QA . Our work aims to analyze vulnerabilities to develop more robust QA models . Current QA models demonstrate brittleness in different aspects . QA models often rely on spurious patterns between the question and context rather than learning the desired behavior . They might ignore the question entirely ( Kaushik & Lipton , 2018 ) , focus primarily on the answer type ( Mudrakarta et al. , 2018 ) , or ignore the “ intended ” mode of reasoning for the task ( Jiang & Bansal , 2019 ; Niven & Kao , 2019 ) . QA models also generalize badly to out-of-domain ( OOD ) data ( Kamath et al. , 2020 ) . For example , they often make inconsistent predictions for different semantically equivalent questions ( Gan & Ng , 2019 ; Ribeiro et al. , 2019 ) . Notably , a concurrent work ( Longpre et al. , 2021 ) shows QA models are less robust to OOD data where the contextual information contradicts with the learned information . Different from that work , we propose a brandnew angle for QA robustness which studies the vulnerability of QA models under misinformation . Although our work and Longpre et al . ( 2021 ) both create contradictory contexts , we use them to study different problems . Moreover , we include both human-created and model-generated contradictory contexts , which are more flexible and diverse than their entity-based contradictions . Combating Neural-generated Misinformation . While we are excited about the recent progress in neural text generation , they can be misused to generate realistic-looking and catchy hallucinations , such as fake news ( Zellers et al. , 2019 ) and fake online reviews ( Garbacea et al. , 2019 ) . When produced at scale , neural-generated misinformation can pose threats to many NLP applications . We believe we are the first to study the risk of neural-generated misinformation to QA models and propose a misinformation-aware QA system as a countermeasure . 3 DATASET : CONTRAQA . We construct the CONTRAQA dataset as follows . We first select 10,026 unique context paragraphs from SQuAD 1.1 , including all the 2,036 unique paragraphs from the validation set and 8,000 paragraphs randomly sampled from the training set . For each selected paragraph CR , we create a set of N fake contexts ( CF1 , · · · , CFN ) by modifying some information in CR , with the requirement that each fake context look realistic while containing contradicting information with CR . We use two different ways to create fake contexts : 1 ) via human edits : we ask online workers from Amazon Mechanical Turk ( AMT ) to produce fake contexts by modifying the original context , and 2 ) via BART-FG : our novel generative model BART-FG , which iteratively masks and re-generates constituency spans from the original context to produce fake contexts . 3.1 MANUAL CREATION OF FAKE CONTEXTS . To solicit human-written deceptive fake contexts , we release 10K HITs ( human intelligence tasks ) on the AMT platform , where each HIT presents the crowd-worker with one context paragraph CR we selected . We ask workers to modify the contents of the given paragraph to create a fake version , following the below guidelines : • The worker should make at least M edits at different places , where M equals to one plus the number of sentences in the contexts CR . • The worker should make at least one long edit that rewrites at least half of a sentence . • The edits should modify key information to make it contradict with the original , such as time , location , purpose , outcome , reason , etc . 1 • The modified paragraph should be fluent and look realistic , without commonsense errors . To select qualified workers , we restrict our task to workers who are located in five native Englishspeaking countries1 , and who maintain an approval rating of at least 90 % . To ensure the annotations fulfil our guidelines , we give ample examples in our annotation interface with detailed explanations to help workers understand the requirements . The detailed annotation guideline is in Appendix B . We also hired three computer science major graduate students as human experts to validate a HIT ’ s annotation . In the end , 104 workers participated in the task . The average completion time for one HIT is 5 minutes , and payment is $ 1.0 U.S. dollars/HIT . The average acceptance rate was 93.75 % . 3.2 MODEL GENERATION OF FAKE CONTEXTS . Aside from human-written fake contexts , we also want to explore the threat of machine-generated fake contexts to QA . This source may be more of a concern than human-created misinformation , since they can easily be produced at scale . Recently introduced large-scale generative models , such as GPT2 ( Radford et al. , 2019 ) , BART ( Lewis et al. , 2020a ) , and Google T5 ( Raffel et al. , 2020 ) , can produce realistic-looking texts , but they do not lend themselves to producing controllable generation that only replaces only the key information with contradicting contents . Therefore , to evaluate the efficacy of realistic-looking neural fake contexts , we propose BART Fake Contexts Generator ( BART-FG ) , which produces both realistic and controlled generated text by iteratively modifying the original paragraph . As shown in Figure 2 , for each sentence S of the original paragraph , BARTFG produces its fake version S′ via a three-step process : 1 ) Constituency Parsing . We first apply constituency parsing to extract constituency spans from the input sentence to represent its syntactic structure . We use the off-the-shelf constituency parser from Joshi et al . ( 2018 ) in AllenNLP2 , which achieved 94.11 F1 on the Penn Treebank . 2 ) Constituency Masking . We then randomly select a constituency phrase3 ( a non-terminal in the parse tree ) and replace it with a special mask token [ MASK ] . We choose to mask constituency phrases instead of random spans as : 1 ) constituents represent complete semantic units such as “ Super Bowl 50 ” , which avoids meaningless random phrases such as “ Bowl 50 ” ; and 2 ) constituents often represent important information in the sentence — such as time , location , cause , etc . 3 ) Mask Filling . We fill in the mask by generating a phrase different with the masked phrase . The mask is filled by the BART model fine-tuned on the Wikipedia dump with a new self-supervised task called gap constituency filling , introduced later . The above pipeline is iteratively run forK times to generate sentence S′ from S. We choose to make the edits iteratively rather than in parallel to model interaction between multiple edits . For example , in Figure 2 , where the previous edit changed “ Santa Clara ” to “ Atlanta ” , the next edit may change “ California ” into “ Georgia ” to make the contents more consistent and realistic . 1Australia , Canada , Ireland , United Kingdom , USA 2https : //demo.allennlp.org/constituency-parsing 3Our considered constituency types : ADJP , ADVP , NP , PP , SBAR , SBARQ , SINV , VP , SQ , WHNP , WHPP Gap Constituency Filling ( GCF ) Pre-Training . To train the BART model to learn how to fill in a masked constituency phrase , we propose a novel pre-training task named Gap Constituency Filling ( GCF ) . For each article in the Wikipedia dump that consists of T sentences [ S1 , S2 , · · · , ST ] , where each sentence is a word sequence St = [ wt1 , · · · , wt|St| ] , we construct the following training data for t = 2 , · · · , T − 1 : Input : S1 , St−1 , wt1 : a−1 , [ MASK ] , w t b+1 : |St| , St+1 Output : w t a : b = [ w t a , · · · , wtb ] where the output represents a masked constituency span that starts with the a-th word and ends with b-th word . The input is the concatenation of the first sentence S1 , the previous sentence St−1 , the current sentence St with one constituency being masked , and the subsequent sentence St+1 . The BART model is fine-tuned to predict the output given the input on the entire Wikipedia dump . This task trains the BART model to predict the masked constituent , given both global contexts ( S1 ) and local contexts ( St−1 , St+1 ) .

Summary: The authors studied how contradictory information affects the accuracy of QA systems.  They created a new dataset of ~10k from SQuAD with added contradictory paragraphs (context). The contradictory data was generated by two different ways. The authors employed Amazon Mechanical Turks raters to rewrite the original context. They also designed a BART-FG model to automatically produce such context by replacing spans with model generated value.   The authors also proposed a way to help QA systems avoid contradictory information. A RoBERTa-based model was used to classify a context into trustworthy or not, with an accuracy around 80%. The trustworthiness score is then used to weigh the final result, together with existing scores of quality/confidence.  Evaluation was done with the new dataset. It showed that 1) adding contradictory information hurts QA performance badly, 2) the RoBERTa-based classifier can help regain some of the loss, but not all of them. The authors also measured the effectiveness of contradictory information creation, where human takes the top place by producing the least altered context with largest effect on final outcome.  The authors promised to share the dataset and the source code / weights of the proposed models. They further discussed the potential ethical impact of releasing the data, arguing that it's net beneficial.