Accurate and Nuanced Open-QA Evaluation Through Textual Entailment (2024)

Abstract: Open-domain question answering (Open-QA) is a common task for evaluating large language models (LLMs). However, current Open-QA evaluations are criticized for the ambiguity in questions and the lack of semantic understanding in evaluators. Complex evaluators, powered by foundation models or LLMs and pertaining to semantic equivalence, still deviate from human judgments by a large margin. We propose to study the entailment relations of answers to identify more informative and more general system answers, offering a much closer evaluation to human judgment on both NaturalQuestions and TriviaQA while being learning-free. The entailment-based evaluation we propose allows the assignment of bonus or partial marks by quantifying the inference gap between answers, enabling a nuanced ranking of answer correctness that has higher AUC than current methods.

heading: 1 Introduction
text: Open-domain question answering (Open-QA) is a long-established task requiring systems to generate precise answers to factual questions on any topic, from information in a large corpus of text (Voorhees and Tice, 2000; Zhang et al., 2023). A more restricted form of open-domain QA where answers are short is still regarded as challenging and as a reasonable test for the capabilities of recent large language models (LLMs) (Anil et al., 2023; Touvron et al., 2023), particularly when it comes to the assessment of LLM honesty (Yang et al., 2023), calibration (Tian et al., 2023). Open-QA benchmarks (Joshi et al., 2017; Kwiatkowski et al., 2019; Lee et al., 2019; inter alia), consisting of pairs of curated questions and manually-annotated gold answers, have been under intense scrutiny because current automated evaluations have been found primitive, flawed, and insufficient to capture
Code and data of the work are available at https://github. com/U-Alberta/QA-partial-marks.
the true capabilities of Open-QA systems (Chen et al., 2019; Boyd-Graber and Börschinger, 2020; Kamalloo et al., 2023; Wang et al., 2023).
Open-QA Evaluators. Let S be the set of finite strings. Given a question q ∈ Q ⊂ S, an OpenQA system generates a free-text system answer a ∈ A ⊂ S , while reference gold answer(s) a∗ ∈ A∗ ⊂ S are provided by humans. In the typical setting, an evaluator f : Q×A×A∗ 7→ {0, 1} compares the system answer a with the gold answer(s) a∗ to provide an evaluator judgment of whether the system correctly answered the question q.
While a wide variety of evaluators would be possible, current Open-QA benchmarks resort to fairly strict and primitive evaluators, which do a poor job with under-specified questions or when the system provides an answer that is either more general or more specific than the gold standard, and are believed to have hindered the understanding of LLM’s “emergent” abilities (Schaeffer et al., 2023).
Ambiguity in Open-QA benchmarks. Questions from Open-QA benchmarks are often ambiguous and under-specified (Boyd-Graber and Börschinger, 2020), leading to multiple possible answers that are not always covered by the gold answers (Si et al., 2021). Figure 1 presents an example from the NaturalQuestions benchmark (Kwiatkowski et al., 2019) where “Oak Island” is the sole gold answer to the question “Where is the TV show The Curse of Oak Island filmed?”. However, due to the lack of specificity, a case can be made that more specific answers such as “on Oak Island, a small island off the coast of Nova Scotia, Canada”, or more general answers such as “Nova Scotia, Canada” should be accepted. The former covers the gold answer and provides more details, while the latter has a lower level of specificity than the gold answer. Exact word matching, a commonly used evaluator, fails with both answers. More advanced automated evaluations, including
2575
semantic similarity models and LLM in-context learning, are also shown to be incapable of capturing such intricacies and deviate from human judgment (Kamalloo et al., 2023; Wang et al., 2023).
Our contributions. We study semantic relations between system answers and gold answers that consider whether system answers cover the gold answer while providing more details, or vice versa. Naturally, textual entailment, the task of determining whether a piece of text entails or contradicts another, is a suitable and training-free tool for this categorization. We propose to use textual entailment for the evaluation of Open-QA systems, and show that entailment-based evaluation metrics, even when used without finetuning, are consistent with human judgments and are more effective in capturing the true capabilities of a range of Open-QA systems when evaluating system answers on both NaturalQuestions (NQ) and TriviaQA (TQ; Joshi et al., 2017). We also propose to use entailment-based evaluation metrics to assign bonus or partial marks to system answers by quantifying the inference gap between system answers and gold answers. We argue that our metric offers a more informative and fairer alternative to current binary evaluation metrics. Such a more accurate and nuanced QA evaluation scheme is valuable in solidifying the large body of concurrent studies that build on short-answer QA evaluations.

heading: 2 Related Work
text: Typical Open-QA evaluators rely on exact word match accuracy (lexical match), F1 score over word matches (formally defined in Bulian et al. (2022)), some semantic similarity model such as BERTScore (Zhang et al., 2020) or BLEURT (Sellam et al., 2020), or zero-shot or in-context learning using an LLM (Chen et al., 2023b; Kamalloo et al., 2023). These approaches have been under scrutiny, however. Bulian et al. (2022), Kamalloo et al. (2023) and Wang et al. (2023) have looked at quantitatively assessing the correctness of automated evaluators by comparing the judgments they produce against gold judgments produced by human experts. They find that unsupervised automated evaluators, including those powered by pre-trained foundation models and LLMs, are not consistent with human judgments.
Wang et al. (2023) released the EVOUNA dataset for the evaluation of automated evaluators, with 3,020 questions from NQ and 1,938 from TQ. Ques-
tions are filtered to exclude those with outdated gold answers, and system answers generated by state-of-the-art Open-QA systems, namely DPR (Karpukhin et al., 2020) + FiD (Izacard and Grave, 2021), InstructGPT and ChatGPT (Ouyang et al., 2022), GPT-4 (OpenAI, 2023), and BingChat, are annotated with gold judgments. An ideal automated evaluator should produce judgments that are consistent with human judgments, and thus, have higher F1 scores and accuracies when evaluated against human judgments. We base our work on EVOUNA and study the relation between gold answers and system answers, and subsequently derive an entailment-based evaluator f̂ that is more consistent with human judgments. We also extend the range of f̂ from a {0, 1} binary prediction to R to achieve a more informative and fairer evaluation.
To overcome traditional metrics’ lack of semantic understanding and, henceforth, underestimation of performance, recent efforts have focused on understanding the semantic equivalence of answers and developing evaluators accordingly (Si et al., 2021; Bulian et al., 2022; Kamalloo et al., 2023). We argue that the semantic equivalence is not the only relation between valid system answers and gold answers. A valid answer can range from a vague, less informative one (e.g., a range, time period, or region) to a very specific and detailed answer (e.g., a precise number, time, or location).
Bulian et al. (2022) proposed to accept all answers that contain at least all relevant content of the gold answer and no misleading content, while making no explicit distinction between semantic equivalence and entailment. We extend this idea and propose to assign a partial order to system answers with regard to how much relevant information the answer contains relative to the gold answer.

heading: 3 The Answer Hierarchy
text: Let A∗ denote the set of gold answers for an OpenQA benchmark. We define two other sets of answers: Asup is the set of superior answers that provide more information than what is in the gold standard, and Ainf is the set of inferior answers that only address the question partially. Given a system answer a and the corresponding gold answer a∗, we say that a ∈ Asup if and only if a entails a∗ within the context of the question. (This naturally extends to the case where multiple gold answers are given). Similarly, we say that a ∈ Ainf if and only if it is entailed by a∗. Finally, an answer would be
incorrect if and only if a and a∗ are not entailed by each other; in this case a ∈ S− (Asup ∪Ainf ). As a special case, it follows that answer a is equivalent to the gold answer a∗ iff. a ∈ Asup ∩Ainf .
Before textual entailment, question-answer pairs are rewritten as declarative statements (introduced by Demszky et al. (2018) as QA2D) using GPT3.5, as the question and context are important for assessing answers (Kamalloo et al., 2023). For example, the gold answer in Figure 1 is converted into the statement “The TV show The Curse of Oak Island is filmed on Oak Island”. Inspired by LLM’s strong performance in natural language inference tasks (Qin et al., 2023), we use GPT-3.5 to conduct textual entailment tests. In the finalized approach, two steps, converting a question-answer pair to a declarative statement, and performing textual en-
tailment test on declarative statements, are perform by a LLM in a few-shot manner. We validate that for both steps GPT-3.5 as the LLM achieves high statistical reliability (Appendix A.2) in terms of the agreement across different seeds, and high validity (Appendix A.3) in terms of the alignment with human labels. Implementation details and a worked example are provided in Appendix A.1.
System answers deserve partial credits and bonus credits. The above-described entailmentbased evaluator reveals that Asup ⊕Ainf (the disjoint union) represents a considerable amount of valid system answers that would otherwise be disregarded (Table 9, 10). Assuming the hierarchy holds, we would see a 10.1% and a 6.5% increase in accuracy for NQ and TQ, respectively, which account for the reported underestimation of QA
performance (Bulian et al., 2022; Wang et al., 2023). We validate the hierarchy by demonstrating that higher positions correspond to better answers judged by humans. This is supported by one-tailed Fisher’s exact tests, all yielding significant results with p < 0.01 with the exception of DPR-FiD and Asup−A⋆ in TQ. Details of the statistical tests can be found in Appendix B.2.
The answer hierarchy is a superior automated evaluator. Treating Asup ∪ Ainf as correct answers1 bring the evaluation results of various systems closer to human performance, as shown in Table 1, and confirms the observation that current evaluators2 unfairly misrepresent the capabilities of those systems. However, unlike previous studies which resorted to manual inspections of answers, our evaluator allows the same observation in an automated way. Moreover, our evaluator can be used for any benchmark. The system where entailment does not improve the performance is DPR-FiD, which is an extractive model that outputs a span of text that requires less semantic understanding to evaluate than complete sentences. Nevertheless, the entailment evaluator assessed that system very closely to the numbers reported in the literature.
Although learning-free, entailment is comparable to finetuned evaluators. Bulian et al. (2022) and Kamalloo et al. (2023) advocate for learned evaluators to close the gap between automated and human evaluation. For comparison, we partition EVOUNA-NQ by questions into 50:50 train/test splits, and finetune a Llama-2 (Touvron et al., 2023) model which only performs slightly better than our method. We show that explicitly including entailment as a feature improves the finetuned model (+NLI in Table 2). Moreover, we finetune another Llama-2 model with the same training data, but with system answers as contexts to predict gold
1Alternative choices discussed in Appendix B.1. 2Details about baselines in Appendix C.1.
answers, in order to use conditional V-information (CVI; Hewitt et al., 2021; Chen et al., 2023a) as another evaluator that builds on usable information from the system answers. The results are shown in Table 2. Entailment also yields better results than in-context learning with four examples (best prompting in Table 1). Details of the finetuning process and the learned baselines are in Appendix C.2.
Out-of-the-box entailment outperforms prompt engineering. Since our method is implemented solely with GPT-3.5, it can be seen as a prompt/flow-engineering method that outperforms the best prompt engineering technique among those Kamalloo et al. (2023) extensively explored (best prompting in Table 1), such as Chain-of-Thought (Wei et al., 2022) and in-context learning. Meanwhile, the pre-processing and entailment tests can be implemented independent of LLMs, for example using DeBERTa (He et al., 2023) as the NLI model and Llama-2-7B as the question to statement conversion model (small in Table 1), while still achieving improved results.

heading: 4 Towards Partial Marks
text: Bulian et al. (2022) demonstrated by examples that the seemingly continuous F1 score is not indicative of how close the system answer is to the gold answer. Going beyond directly using the classification probability from the NLI model (Chen et al., 2021), we hypothesize that measuring the inference gap, i.e. how many steps, assumptions, and additional pieces of information are needed to derive a system answer a from a gold answer a∗, can be used to assign partial marks in a way that reflects semantic closeness. Inspired by Chain-ofThought prompting, explainable natural language inference (Camburu et al., 2018), and LLM-based decompositions of implicit content (Hoyle et al., 2023), we propose to use LLM (GPT-3.5 in our experiments) to explain step-by-step the inference process behind how a∗ entails a, along with assumptions and additional knowledge required (“Inference”). Based on the explanation, a score of inference difficulty is directly produced by the LLM (“LLM Score”), or the number of steps is counted (“#Steps”) as partial marks. Details of the scoring schemes, worked examples, and alternatives are discussed in Appendix D.
We have examined the inter-set ranks in the answer hierarchy in §3. When it comes to the intra-set ranking of partial answers in Ainf −Asup,
Brunner-Munzel tests (Brunner and Munzel, 2000) show that both LLM Score and #Steps, as well as other baselines in Table 3, assign higher scores to human-accepted answers (p < 0.001). Quantitatively, Table 3 shows that the inference-processbased scores have higher AUROC on EVOUNANQ than F1 score or using GPT-3.5 to directly assess system answers on a 5-point scale (“LLM Score”), indicating that partial marks assigned by our method are suitable for capturing the nuanced goodness differences between system answers.

heading: 5 Conclusion
text: In theory, textual entailment is considered AIComplete (Dagan et al., 2009) - an embodiment of general AI that solves all AI tasks, Open-QA evaluation included. In practice, we showed that state-of-the-art textual entailment provides a simple yet powerful replacement for Open-QA evaluation, and it offers the prospect of soft and partial marks.

heading: Limitations
text: Our current work studies the potential of textual entailment as a fairer and finer-grained replacement for Open-QA evaluation. We only explored using the method for benchmarking QA systems. However, it remains a highly interesting topic to investigate how it can be used as a softer signal for training QA systems with the potential of improvements, given the success of smoothed labels (Hinton et al., 2015; Szegedy et al., 2016). We only studied QA benchmarks consisting of mostly factoid questions and relatively short and simple answers. For QA tasks that require more complex, and potentially multi-passage multi-facet answers, it is unclear how well the original entailment method can be directly applied. Future work is required to investigate the entailment relations and the matching between multiple units of meaning, such as in Laban et al. (2022), to extend our work
to more complex QA tasks.

heading: Acknowledgements
text: We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). This work is also supported in part by a gift from Scotiabank. Icons in Figure 1 are designed by OpenMoji under CC BY-SA 4.0 license, and by Flaticon-Freepik.

heading: A Entailment Test Implementation

heading: A.1 Detailed Settings
text: When using entailment to obtain the answer hierarchy in §3, we use gpt-3.5-turbo-1106. The gold answer-question and system answer-question pairs are converted to two declarative statements using the first prompt in Table 14. The two examples in the first prompt are chosen from EVOUNA-NQ, and as the dataset size is large enough, we do not need to use a separate dataset for prompt engineering. The two declarative statements are then used as the premise and hypothesis and vice versa in the second prompt in Table 14 to obtain the entailment classification in two directions. For all GPT-3.5 API calls, we set seed=42, temperature=0.0 to ensure reproducible results, and set max_tokens=300.
Here is a working example of the entailment test on a system answer generated by InstructGPT and a gold answer from NQ:
Question: where is fe best absorbed in the body Gold answer: in the duodenum System answer: Iron is best absorbed in the small intestine.
Gold statement: Fe is best absorbed in the body in the duodenum. System statement: Iron is best absorbed in the small intestine.
Entailment test: Gold statement entails system statement, but not the other way around. Therefore, the system answer belongs to Ainf − Asup. Meanwhile, human annotator judged the system answer as correct in EVOUNA.

heading: A.2 Assessment of Reliability
text: To assess the statistical reliability (consistency) of our method, we measure the agreement across different random seeds, and the potential impact on the overall performance. We repeat the LLM-based steps on EVOUNA-NQ and EVOUNA-TQ subsets of size 2,000, each using random seeds 0,1,2,3 for GPT3.5 calls while keeping the rest of the settings controlled.
Reliability of question-answer to statement conversion. We calculate the consistency of generated statements from the same question-answer pairs across different seeds using BLEU and exact sentence matching, as shown in Table 4. These results indicate that the generated statements are fairly consistent across different seeds with almost all statements being identical or very similar.
Reliability of textual entailment test. We measure the agreement of textual entailment predictions across different pairs of seeds for the same golden-system answer pairs using Cohen’s Kappa, as in Table 5. The results are interpreted as almost perfect agreement according to Landis and Koch’s (1977) guideline.

heading: Dataset 0 vs 1 0 vs 2 0 vs 3 1 vs 2 1 vs 3 2 vs 3
text: Reliability of hierarchy construction. The result from textual entailment is used to categorize a system answer into one of the sets in the hierarchy (Table 9 and 10). Again we measure the agreement of the categorization across different pairs of seeds using Cohen’s Kappa. The results in Table 6 show even better agreement than the textual entailment test as multiple candidate golden answers are considered in this step.
Reliability of QA evaluation. We assess whether different seeds lead to different QA system evalu-
ation results (Table 1) as reflected by the variance of F1 scores and accuracy. On the two subsets, different seeds have virtually no impact on the overall QA evaluation as seen in Table 7.

heading: A.3 Assessment of Validity
text: Converting a question-answer pair to a declarative statement (known as QA2D) is a well-established task. Demszky et al. (2018) provided a dataset where the dev set has 10,344 question-answer pairs with a human-written declarative statement (test set unavailable). We compare our 2-shot LLM generated statements with the human-written statements using BLEU and ROGUE (Table 8), and the generations are very similar to human-written statements and a fine-tuned T5 baseline3.
Our zero-shot prompt for textual entailment (Table 14) is adapted from Qin et al. (2023). They have tested the validity of this textual entailment test method on RTE and CB datasets and reported a high accuracy of 0.86 and 0.89 respectively. Zhong et al. (2023) used a slightly different prompt for the same task and reported GPT-3.5 NLI accuracy to be higher than finetuned BERT-large and RoBERTalarge on both MNLI-m and RTE.
Finally, the validity of final QA evaluation is confirmed by the high correlation with human judgment (Table 1).
3domenicrosati/QA2D-t5-base on Huggingface.
B Inter-set Order Validation

heading: B.1 Hierarchy of Answer Sets
text: The entailment test organizes system answers into a hierarchy of sets: in Table 9 and 10, the four rows corresponds to the four sets at different levels of the hierarchy: (1) Asup − Ainf , (2) Asup ∩ Ainf , (3) Ainf −Asup, and (4) S − (Asup ∪Ainf ). The size of the sets are shown in the Count column.
We propose to treat the union (denoted as ∪) of Ainf and Asup as the correct system answers. We chose to include Ainf −Asup as we follow the human annotation guideline of EVOUNA that considers the lack of specificity in questions and accepts answers of all levels of specificity. Meanwhile, alternative choices like excluding Ainf −Asup (denoted as −) have negligible impact on the discussion, as the Ainf − Asup sets have a small size for both NQ and TQ, as shown in Table 9 and 10. We report the evaluation results of excluding Ainf −Asup in Table 11.

heading: B.2 Statistical Tests
text: As summarized in §3, we conduct statistical tests to verify if the four sets do have a order. We hypothesize that the higher the rank, the more likely the system answer is correct. We use one-tailed Fisher’s exact test do a pairwise comparison the distribution of human judgements in the four sets with DPR-FiD, a method with extractive nature that makes semantic understanding excessive and lexical matching sufficient, excluded. The results are shown in Table 12.

heading: C Baseline Method Details

heading: C.1 Unsupervised Evaluators
text: Wang et al. (2023) evaluated multiple unsupervised evaluators, including lexical match, BERTScore, and GPT-3.5, on both EVOUNA-NQ and EVOUNATQ. We make the comparisons with the numbers reported in their paper and refer the readers to Wang et al. (2023) for the detailed settings of those baseline evaluators. They also explored four additional prompting methods for the GPT-3.5 evaluator: Ignoring Background Information, Giving Reasons, Chain-of-Thought, and In-Context Learning, with the exact prompts provided in their paper. For each category in Table 1, we choose the best performing method among the four for comparison as in the GPT-3.5 (best prompting). This represents the upper bound performance of their prompt engineering efforts that is only achievable if an oracle exists that knows the best prompt for each QA system.
For Entailment (small), we use the same prompt as in Table 14 row 1, but with 4-bit quantized Llama-2-7B-GPTQ4 instead of GPT-3.5 as the model for question to statement conversion. We use a finetuned DeBERTa-v3-large5 by Reimers
4TheBloke/Llama-2-7B-GPTQ on Huggingface. 5cross-encoder/nli-deberta-v3-large on Hugging-
face.
and Gurevych (2019) as the NLI model.

heading: C.2 Learned Evaluators
text: We perform a half-half partition of the EVOUNANQ dataset by question type to create a training set and a test set, where no question-answer pairs with the same question falls in the same split. A Llama-2-7b-chat-hf model is finetuned on the training set by inserting the question, gold answer, system answer, and human judgment into the templates in Table 15. During inference, the same templates are used with human judgment left empty. Finetuning is done with the Huggingface PEFT (Mangrulkar et al., 2022) and TRL (von Werra et al., 2020) libraries. For CIV, two models with and without system answers as rationales are finetuned on the training set in the same fashion using templates in Table 16.

heading: D Partial Mark Scoring
text: If the entailment test shows that the system answer is in Ainf −Asup, we use GPT-3.5 and the prompt in Table 17 row 1 to generate an explanation of what inference process is required to deduce the system answer from the gold answer (Inference). The example system answer a2 in Figure 1 is in Ainf − Asup, and the explanation generated is as follows:
1. The TV show the Curse of Oak Island is filmed on Oak Island. (Given in S1)
2. Oak Island is located in Nova Scotia, Canada. [[INFO]]
3. Therefore, the TV show the Curse of Oak Island is filmed in Nova Scotia, Canada. (Combining steps 1 and 2)
Given the inference process explanation, we manually design the following partial mark scoring heuristics:
1. CIA: -#Step*10-#INFO*3-#ASSUMPTION*5
2. C: -#Step*10
3. IA: -#INFO*3-#ASSUMPTION*5
As an alternative, we use GPT-3.5 to score the difficulty of the inference process in the 5-point scale by providing the prompt in Table 17 row 2 as an additional message after the explanation step (Inference+LLM Score). The LLM Score baseline skips the explanation step and directly use GPT-3.5 to provide a 5-point-scale score using the prompt in Table 17 row 3.
The three manually designed scoring scheme are not significantly different from each other or from the automated Inference+LLM Score as shown in Table 13.

heading: Future Work
text: Future work is required to investigate the entailment relations and the matching between multiple units of meaning, such as in Laban et al. (2022), to extend our work
to more complex QA tasks