Using contradictions improves question answering systems
Year: 2023

Abstract:
This work examines the use of contradiction in natural language inference (NLI) for question answering (QA). Typically, NLI systems help answer questions by determining if a potential answer is entailed (supported) by some background context. But is it useful to also determine if an answer contradicts the context? We test this in two settings, multiple choice and extractive QA, and find that systems that incorporate contradiction can do slightly better than entailment-only systems on certain datasets. However, the best performances come from using contradiction, entailment, and QA model confidence scores together. This has implications for the deployment of QA systems in domains such as medicine and science where safety is an issue.

heading: 1 Introduction
text: Safety in NLP systems is unresolved, particularly in biomedical and scientific contexts where hallucination, overconfidence, and other problems are major obstacles to deployment (Ji et al., 2022; Kell et al., 2021). One active area of research to solve these issues is natural language inference (NLI) (Li et al., 2022). NLI is the task of determining whether a hypothesis is true (entailed), false (contradicted), or undetermined (neutral) given some premise.
Current NLI systems typically focus only on entailment to verify hypotheses—they calculate the degree to which a hypothesis is supported by the premise. But the premise can provide another signal: contradiction. Regardless of how well a hypothesis is entailed by the context, it can also be more or less contradicted, which could affect whether it is accepted or rejected. Contradictions are an important signal indicating whether some statement might be unacceptable given a premise. In some cases where we might not know if a statement is supported, we should still ensure we are rejecting statements that are outright contradicted.
We wondered if adding this signal to a question answering (QA) system might improve performance and safety. To this end, we propose a method that reformulates answers from the QA system as hypotheses for NLI, calculates the entailment, contradiction, and neutrality of each hypothesis, and then selects the best one based on a combination of these results (Figure 1). We show that across 16 QA datasets (9 multiple choice and 7 extractive), the best approach is to use entailment, contradiction, and confidence scores together. Using only contradiction is roughly on par with, and sometimes better than, using only entailment.

heading: 1.1 Related work
text: NLI for question answering has been explored by several authors in various settings; see Paramasivam and Nirmala (2021) for an overview.
One of these settings is selective question answering for extractive QA, where selective refers to abstention when the system is not confident enough in its answer (Kamath et al., 2020). Chen et al. (2021) have found that NLI systems are able to verify the predictions made by a QA system in this setting, but their result is limited to only selecting a top k% of answers. Moreover, they
827
do not provide an approach for improving overall performance, nor do they show the effect of incorporating contradiction directly (but do so indirectly by analyzing non-entailed passages).
In the related setting of multiple choice QA and fact checking, Mishra et al. (2021) have explored the use of entailment, finding that NLI models do well at these tasks by themselves, but can perform even better when they are adapted to in-domain data and longer premises. Yet their method uses only a two-class NLI set up (entailed or not entailed), which doesn’t tell us much about directly using the contradiction signal. Pujari and Goldwasser (2019) does incorporate the contradiction signal showing the power of contradiction to improve machine comprehension but does not analyze its effects separately from entailment.
Other QA settings in which NLI has been used include open domain (Harabagiu and Hickl, 2006) and multi-hop (Trivedi et al., 2019). Thus far, approaches tend to focus on entailment. To our knowledge, our work is the first to directly assess using contradictions for QA isolated from entailment.
Outside of question answering, a domain that uses contradictions is factual consistency—the task of ensuring that a collection of utterances is faithful to a source document. Li et al. (2022) provide an overview. Typically, entailment is still the main focus, but Laban et al. (2022) propose an NLI-based method to ensure the consistency of a summary with a source document using contradiction and neutral scores in addition to entailment, beating out previous systems.
Other researchers have used contradictions to identify consistency errors across Wikipedia (Schuster et al., 2022; Hsu et al., 2021) or generate credible character dialogue (Nie et al., 2021; Song et al., 2020).

heading: 2 Methods
text: We tested the effect of contradictions in two QA settings and a total of sixteen question-answer datasets. Our approach is broadly similar to both Chen et al. (2021) and Mishra et al. (2021) in that we use most of the same datasets for evaluating NLI reranking for multiple choice QA and extractive QA. Unlike both, we incorporate contradiction directly as a signal for reranking answers.
Briefly, for each dataset, we used pretrained QA models to produce answers and confidence scores for the dataset’s questions. We refer to the confi-
dence scores below as QA. We then trained QA2D models (where QA2D stands for "question-answer to declarative") to turn the answers into the declarative hypothesis format required for NLI. For example, the question-answer pair "What is the most abundant metal in the Earth crust? Copper." might be rephrased as "The most abundant metal in the Earth crust is copper" (see Appendix D for more details).
With the question contexts as premises, we then used NLI models to classify every premisehypothesis pair into three classes, each with an associated score: entailed (E), contradicted (C), and neutral (N). After that, we trained logistic regression calibration models to find which linear combination of the four scores—QA, E, C, and N—was best able to pick the answers accurately.
When evaluating performance, we applied the selective QA approach from Kamath et al. (2020) to rank answers using combinations of the four scores, and then consider only those that the model was most confident in answering. We compared selecting the top 20% and 50%. In the multiple choice setting, it was also possible to rank all potential answers according to the four scores, unlike in the extractive QA setting where the QA model produced only one answer per question, so we evaluated performance with that approach as well (see appendix A for details).

heading: 3 Experimental setting
text: In the multiple choice setting, we tested 9 datasets. Two of them are in-domain, since the pretrained QA models we used were finetuned on them. Specifically, we used a RoBERTa large model (Liu et al., 2019) finetuned on the RACE dataset (Lai et al., 2017), as well as two DeBERTa v3 variants, base and xsmall (He et al., 2021a), finetuned on the SciQ dataset (Welbl et al., 2017).
In the extractive QA setting, we used 7 datasets: five from the MRQA 2019 task (Fisch et al., 2019), as well as SQuAD 2.0 (Rajpurkar et al., 2018) and SQuAD adversarial (Jia and Liang, 2017). The SQuAD model is the in-domain dataset: it was used to pretrain (Rajpurkar et al., 2016) the two QA models we used, DistillBERT (Sanh et al., 2020) and BERT-Large (Devlin et al., 2019). Like Chen et al. (2021), we used the Natural Questions dataset for calibration.
In both settings, all datasets contain the relevant context that can be used by the QA models to select
answers. More detail on the datasets and QA models is available in appendices B and C respectively.
See appendices D, E, and F for details on the QA2D, NLI, and calibration models. Our models follow the setups described in Kamath et al. (2020), Chen et al. (2021), and Mishra et al. (2021). The main interesting detail is that the calibration models were trained on a holdout set of 100 samples from a single domain, using logistic regression, as in Chen et al. (2021).

heading: 4 Results

heading: 4.1 Multiple choice setting
text: For most multiple choice datasets, the best accuracy—when ranking all potential answers—is attained when using a calibrated model combining QA confidence, entailment, and contradiction (QA+E+C in Table 1). Only for the in-domain case (RACE-C) does the uncalibrated RoBERTa-RACE model perform on par with that. Using QA scores combined with either entailment (QA+E) or contradiction (QA+C) achieves similar performance, with contradiction winning by a small margin: 84.33% average accuracy compared to 84.31%.
To inspect these trends further, we performed a correlation analysis of the NLI classes and QA confidence scores with the correct answer (appendix G). We found that besides QA confidence, it is the contradiction score that has the strongest correlation with the correct answer. The analysis also showed that the neutral class score (N) had almost no effect, which is why it is omitted in all results.
When using the selective QA approach and evaluating only the 20% of 50% most confident answers, the best performance is attained with the QA+C combination (Table 2). This model is the only one that beats just using the QA confidence score on average. It is stronger than QA+E+C and QA+E for both coverage percentages.
Contradiction alone, without QA confidence scores (C), also beats both entailment alone (E) and entailment with contradiction (E+C) for both coverages. These results match our intuition that the less contradicted an answer, the more likely it is correct, even in cases where there is uncertainty about its entailment.

heading: 4.2 Extractive QA setting
text: Similar results occur when evaluating the extractive QA datasets with 20% and 50% selective coverage (Table 3). The QA+C model does better than QA
alone, and C alone does better than E+C or E alone, indicating the importance of the contradiction signal here too. However, entailment seems to matter more for extractive QA, as the best F1 score overall was from QA+E in the 20% coverage case, and QA+E+C in the 50% case.

heading: 5 Discussion
text: Contradiction with background context is a useful signal that NLP systems can use to infer answers to questions. This is not necessarily a superior strategy to using entailment, but our results show that combining these two signals can improve performance beyond what QA models can achieve on their own. These results are interesting because using contradictions comes with potential benefits for the safety of NLP systems and, as a result, their deployment in domains such as medicine or science. Namely, that there are many potential cases where we are not sure if a statement is entailed directly by a background context but we may be sure that the statement is not refuted by a background context. In two-class NLI settings where we focus only on entailment, neutral and contradiction are collapsed together and we don’t have this guarantee.

heading: 6 Limitations
text: Our work comes with some limitations. It is uncertain whether our results in two specific settings, multiple choice and extractive QA, would extend to more general settings for NLI, although the use of contradictions for factual consistency by Laban et al. (2022) suggests that they could. Additionally, 3-class NLI is not sufficient to capture all the natural language relations that might be needed to verify an answer. As such more challenging datasets in other settings and more granular NLI settings should be attempted.
Another limitation involves answer ranking and the associated computational cost. The main reason we did not test answer ranking in extractive QA is that we did not generate diverse outputs, but another reason is that such a procedure grows prohibitively expensive as the domain becomes more open. In a fully open domain, ranking would require a quadratic evaluation for each context passage against each reformulated answer candidate (Schuster et al., 2022). Future work should look at comparison approaches that amortize this cost, such as NLI-based dense passage retrieval (Reimers and Gurevych, 2019).

heading: A Answer ranking procedure
text: In the multiple choice setting, we performed an answer ranking procedure to pick the answer to a given question among a set of alternative answers N , using both NLI class scores and QA confidence scores. (This is distinct from the selection procedure for the top 20% or 50% of answers we used in both settings.)
Similar to Harabagiu and Hickl (2006), answers are ranked based on the highest probability from the calibration model σ, given a linear combination of the QA or NLI scores given an answer n ∈ N answer set. When a single feature is used, such as an NLI class or the QA score, no calibration is made and σ is simply the identity of the confidence score. In the case of contradiction only, σ is the inverse of the contradiction confidence score, indicating the least contradicted answer is being selected. Formally, our approach can be described as:
argmax N σ(QAn;NLIn)
where QAn is the QA model confidence score for answer n, and NLIn represents the various NLI class scores for n.
We did not use this approach in extractive QA, because we found that asking the model for the top K = 4 answer produced almost the same four answer alternatives with slightly different spans each time.

heading: B Datasets
text: Tables 4 (multiple choice) and 5 (extractive QA) outline the datasets we used. Additional details such as train size and preprocessing steps are available in the references provided. When space doesn’t allow CosmosQA is aliased to Cosmos, MCScript to MCS, MCScript-2.0 to MCS2, and MCTest to MCT. The only preprocessing step we performed was to filter out questions where no context passage is provided. Validation splits (as opposed to test splits) are used in the CosmosQA and QASC cases, since context passages or gold standard answers are not available for these datasets.

heading: C QA models
text: Table 6 outlines the pretrained QA models that we used and the datasets they are trained on. All these models are publicly available on the Hugging Face hub under the locations listed. Where space doesn’t allow, RoBERTa-RACE is aliased as RACE.
We trained the two DeBERTa-v3 models (xsmall and base) as shown in Table 7. They were trained using the Hugging Face trainer API (Wolf et al., 2020) with an Adam optimizer at a learning rate of 5.60e-05 with weight decay of 0.01. All models and inference were performed on 1 Tesla P100 GPU. Full instructions on reproducibility as well as trained models are provided in the publicly available code, including directions to weights and biases to inspect the training runs, full parameter set, and evaluation suites.

heading: D QA2D models
text: A QA2D model reformulates a question-answer pair to a declarative statement (Demszky et al., 2018). As noted in Chen et al. (2021) and Mishra et al. (2021), the QA2D reformulation is critical to using NLI models in QA since the proposed answer needs to match the format of NLI. We trained a T5-small model (Raffel et al., 2020) on the dataset proposed by Demszky et al. (2018) for QA2D since we found almost no noticeable differences in performance in larger models. This used the same setup as the DeBERTa-v3 models xsmall and base (see Table 7).
Unlike Chen et al. (2021), we found that regardless of size, these QA2D models struggled with long questions or questions with complex syntax and would often leave the answer out of the statement. In order to solve this, constrained decoding that required the answer to be in the statement was tried. However, this often produced ungrammatical or nonsensical statements. We settled with the following heuristic to postprocess QA2D outputs: If less than 50% of the tokens in the answer were in the statement then we appended the answer to the end of the statement. 50% was used to account for rephrasing the answer or swapping pronouns. While some statements resulted in answer redundancy, this was better than having hypotheses which left out the answer.
Future work on QA2D should focus on how these models can be used outside of the domains in the dataset provided by Demszky et al. (2018). Finally it is important to note that erroneous QA2D outputs could effect the quality of the whole pipeline see Chen et al. (2021) for a more detailed analysis of this.

heading: E NLI models
text: NLI is used to classify whether the reformulated answer is contradicted, entailed, or neutral with respect to a context passage. We used the whole context, as Schuster et al. (2022) and Mishra et al. (2021) demonstrated that long premises still performed adequate though not as well as sentencelength premises. Using the whole context avoids needing to use decontextualization as is required in Chen et al. (2021).
We used two DeBERTa-based models (He et al., 2021b) trained on the MNLI dataset (Williams et al., 2018) (called mnli-base and mnli-large) and an ALBERT model (Lan et al., 2019) trained on the ANLI dataset in addition to various other NLI datasets (called albert-anli) (Nie et al., 2020). Table 6 contains the Hugging Face references to the NLI models After inference, the confidence scores are used for answer selection and performance evaluation.
E.1 Model size and approach performance analysis
Table 8 mirrors Table 1 in the main text, but shows the accuracy results for uncalibrated E, C, and E+C
in the main mnli-large model, as well as the results with the other NLI models, mnli-base and albertanli. Table 9 shows selective QA accuracy in the multiple choice setting where answer selection is done through ranking before we rank answers for selective QA. Selective QA on extractive QA using DistillBERT (table 10) shows that QA+E+C does best in all cases and contradiction only does second best at 50% coverage.

heading: F Calibration models
text: Like Kamath et al. (2020) and Chen et al. (2021) we developed a set of calibration models in order to perform answer ranking. A calibration model is trained on a set of posterior probabilities from downstream models to predict whether an answer is correct.
To compare the effect of using different combinations of NLI class confidence scores, we trained a logistic regression model on linear combinations of the following features: QA indicates that the QA model confidence score is being used, E indicates the entailment score, C indicates the contradiction score, and N indicates the neutral score. Like in Chen et al. (2021), all calibration models are trained on a holdout set of 100 samples from a single domain using logistic regression which predicts, given the confidence scores of the downstream models, whether the answer is correct. A multi-domain calibration approach like in Kamath et al. (2020) was not used since the focus was a minimum experiment to test the viability of leveraging different NLI classifications.
F.1 Regression Analysis
To illustrate the characteristics of the calibration models, we present a regression analysis for the multiple choice setting (Table 11). The results indicate that as the mnli model gets larger, the calibration model uses its NLI confidence scores more. Importantly, entailment coefficients are stronger than contradiction coefficients in all cases.

heading: G Correlation Analysis
text: Since we are using the NLI and QA model scores to construct the setups above, it is useful to know how these factors correlate with the correct answer. Table 13 shows how each NLI class correlates both by score and by actual classification (score > 50%) as compared against QA model confidence score. The multiple choice analysis shows
answers from the RoBERTa-RACE model and the extractive QA analysis shows answers from the BERT-large model trained on SQuAD. The correlation analysis presents Spearman rank correlations.
What we see is that in the multiple choice setting, the confidence score has a strong correlation with the correct answer, which makes sense given the confidence score is a softmax over the multiple choice classes. Extractive QA confidence scores have a much weaker correlation and tend to have less correlation than entailment has with the correct answer. Despite the results presented above, contradiction only has a notable correlation with the correct answer when the score is used rather than the classification. This is a point in favor of our approach of using confidence scores for NLI rather than classifications.
Interestingly, in the extractive QA case, the neutral class is more negatively correlated when selecting for contradiction when using classification. Our conjecture would be that in the extractive QA case, we don’t have much to compare against. When looking at the per dataset correlations for the multiple choice setting (Table 12), we see that in most cases, other than the QA confidence scores, the contradiction scores have the strongest correlations with the correct answer out of any NLI class and neutral, as we would expect, tends to have very weak correlations. We do not present the per dataset correlation for extractive QA as they are very weak, which we again hypothesize comes from having no answers to compare with.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: Future work on QA2D should focus on how these models can be used outside of the domains in the dataset provided by Demszky et al. (2018). Finally it is important to note that erroneous QA2D outputs could effect the quality of the whole pipeline see Chen et al. (2021) for a more detailed analysis of this