Analysis of LLM’s “Spurious” Correct Answers Using Evidence Information of Multi-hop QA Datasets (2024)

Abstract: Recent LLMs show an impressive accuracy on one of the hallmark tasks of language understanding, namely Question Answering (QA). However, it is not clear if the correct answers provided by LLMs are actually grounded on the correct knowledge related to the question. In this paper, we use multi-hop QA datasets to evaluate the accuracy of the knowledge LLMs use to answer questions, and show that as much as 31% of the correct answers by the LLMs are in fact spurious, i.e., the knowledge LLMs used to ground the answer is wrong while the answer is correct. We present an analysis of these spurious correct answers by GPT-4 using three datasets in two languages, while suggesting future pathways to correct the grounding information using existing external knowledge bases.

heading: 1 Introduction
text: Question Answering (QA) is one of the hallmark tasks that evaluate language understanding capabilities of NLP systems. We are currently witnessing the flourishment of highly capable large language models (LLMs) that solve this complex task, requiring both knowledge and inference skills, with an impressive accuracy (Bang et al., 2023). On the other hand, it has been shown that LLMs can generate content that contradicts facts (Bang et al., 2023; Ji et al., 2023), and several verification results have been reported regarding the evaluation of LLMs’ internal knowledge and whether LLMs can provide answers based on facts (Wang et al., 2023; Manakul et al., 2023; Lin et al., 2022; Zheng et al., 2023; Pezeshkpour, 2023).
At this point, it is not clear exactly to what extent such LLMs possess the knowledge needed to solve QA problems and how accurately they perform inference to leverage that knowledge. How often do LLMs rely on “hallucinated” knowledge during inference? Can these hallucinations be remedied by
structured knowledge bases (KBs) carefully crafted by humans? Previous studies have reported that correct answers are often obtained despite errors in the reasoning path that LLMs output to solve QA (Bao et al., 2024; Sprague et al., 2024; Nguyen et al., 2024; Ishii et al., 2024). Ishii et al. (2024) shows the specific error patterns by question type in such cases and the possibility of complementing errors with KBs using JEMHopQA dataset1, which has evidence information in the form of triples, but their analysis is limited to one dataset in Japanese.
In this paper, we focus on investigating how such “spurious” correct answers by LLMs occur more deeply in other datasets and languages. We use three datasets from two languages – HotPotQA (Yang et al., 2018) with R4C (Inoue et al.,
1https://github.com/aiishii/JEMHopQA
24
2020) and 2WikiMultiHopQA (Ho et al., 2020) for English, and JEMHopQA for Japanese. These three datasets present the task of outputting the knowledge (derivation) that serves as the evidence for the answer in the form of derivation triples (as in Fig. 1), so they can be used directly to measure the spuriousness of correct answers in QA. In addition, we investigate the extent to which gold derivation triples in each dataset are covered by existing KBs, suggesting that hallucinatory knowledge can be corrected by combining LLMs with such KBs.

heading: 2 Analysis Method

heading: 2.1 Datasets
text: In this analysis, we use questions, answers, and supporting evidence from widely used HotPotQA, 2WikiMultiHopQA, and JEMHopQA, which are Wikipedia-based multi-hop QA datasets. We use R4C for derivation triples of HotPotQA, and randomly extract 100 instances from the development set as HotPot. We randomly extract 100 instances from the 2WikiMultiHopQA development set as 2Wiki and use all 120 instances from the JEMHopQA development set as JEMHop. Table 1 summarizes the details of these datasets, where the average number of derivation triples are roughly the same across them.
In these datasets, questions comprise of three different types2: (i) Comparison questions, where
2Although 2WikiMultiHopQA has an “Inference” type, we
the two derivation triples have the same relation, as in at the top of Fig. 1; (ii) Composition questions, where two derivation triples share a “bridge” entity, as in the example at the bottom of Fig. 1 where “Augustus” serves as the bridge; (iii) Bridgecomparison, which combines a composition with a comparison, where a comparison is made after finding the bridge entity, e.g., “Which film has the director who is older, Aardram or Land and Freedom?”. The distribution of these three types of questions is shown in Table 2.
As Ishii et al. (2024) reports that comparison questions (numerical comparisons in particular) are more susceptible to spurious correct answers, we created additional datasets that include the samples of such questions in our study. The number of numerical comparison questions differs considerably across our dataset (HotPot: 4%, 2Wiki: 17%, JEMHop: 28%), so we created focused datasets consisting only of numerical comparisons by taking 30 samples from the development set of the datasets, resulting in HotPot_NC, 2Wiki_NC, and JEMHop_NC. We also extracted 30 multi-hop QA instances that compare numerical values from DROP (Dua et al., 2019), a widely used QA dataset that requires mathematical operations, and use them as the analysis set DROP_NC.
The supporting evidence in each dataset is in the form of triples representing a semi-structured relationship (e.g., “date of birth”) between a subject entity (“Ivan Foster”) and an object entity (“1943”), as shown in Fig. 1. The questions are those that require multi-hop reasoning, and each questionanswer pair is accompanied by two or more derivation steps. The task of evaluating LLMs using each dataset is, given a question Q, (i) to predict the answer A, and (ii) to generate a derivation D that justifies A.

heading: 2.2 Evaluation Metrics
text: Answers For HotPot and 2Wiki, we use exact match (EM) and partial match, F1 score measuring the average overlap between gold and predicted answers. For JEMHop, we use similarity match (SM) score based on the Levenshtein distance.
Derivations To account for differences in the structure of the triples and to measure semantic matches, the authors manually evaluated derivation triplets. Even if predicted derivation has a different surface form from the gold derivation, it is consid-
consider it a subtype of composition in this paper.
ered correct if the information contained is correct, in the form of a triple, and sufficient to answer the question.
Note that each dataset provides evaluation scripts for both answers and derivation triples, but we use these scripts to evaluate answers only and rely on human evaluation for derivation triples.

heading: 2.3 Evaluation Setup Using GPT-4
text: We use gpt-4-0613 model via OpenAI API with the prompt for the Chain-of-Thought (CoT) (Wei et al., 2022) 5-shot setting as a method of eliciting the derivation triples that the model uses to infer. More specifically, the CoT 5-shot prompt consists of an instruction to provide a CoT reasoning path, along with 5 few-shot samples. To ensure that the setting of the CoT 5-shot prompt to output the inference path at the same time as the answer does not affect the accuracy of the answer, we also use zero-shot (ask a question only) and non-CoT 5-shot (include 5 random samples from the training set) prompts (see examples in Appendix B).
Based on the results of preliminary experiments, we use temperature parameters of 0.1, 0.2, and 0.0 for HotPot, 2Wiki, and JEMHop, respectively. The maximum token limit is set to 32 for the zero-shot and 5-shot prompts, and to 256 for the CoT prompt. Due to the sampling-based decoding of GPT-4 API, we run each experiment three times and report the average of all runs.

heading: 3 Results and Discussion

heading: 3.1 How well can GPT-4 answer multi-hop questions correctly?
text: In Table 3, the first three rows show the results for the answers in the zero-shot, 5-shot, and CoT 5-shot settings for each dataset. In all datasets,
3Note that this classification table does not include the formatting errors that occurred in two cases in JEMHop and one case in HotPot_NC as derivation triple errors, so the total does not add up to 100%.
the 5-shot setting performed better than the zeroshot setting, and the CoT 5-shot setting achieved the highest accuracy. These results confirm that the CoT 5-shot prompt setting, which outputs the derivation triples simultaneously with the answer, does not affect the accuracy of the answers.
The last two rows show that composition questions are significantly harder to answer correctly than comparison questions in all datasets. A major factor for this large difference is suspected to be that in comparison questions, the two subject entities are explicitly mentioned in the question and the answers tend to be binary (choosing one of the two entities), while in composition question, a bridge entity is implicit and must be identified, and the answers for these questions tend not to be binary (an entity as an answer). Bridge-comparison questions fell in the middle as this tasks for a binary answer while needing to identify a bridge entity.

heading: 3.2 When do spurious correct answers occur?
text: Table 4. shows the performance of GPT-4, where we present the results in a matrix along both answer correctness and derivation triple correctness. Cases where the derivation triples were considered correct even though they differed from the gold derivation triples in this evaluation are described in detail in Appendix C. We found that only 0- 1% cases had an error in inference (Answer is F and Derivation is T); the remaining cases had errors in derivation (i.e., hallucination). As shown in the table, spurious correct answers (Answer is T and Derivation is F) comprise 18% of all cases (which is 31% of the correctly answered cases) in 2Wiki and 15.8% (which is 25% of the correctly answered cases) in JEMHopQA, showing that they occur also quite frequently in English. More than 90% of these spurious correct answers occur in comparison questions and bridge-comparison; they occur less frequently in HotPot because there are fewer comparison questions.
The question type that generated spurious correct answers most frequently (38% on 2Wiki and 68% on JEMHop) was questions comparing numerical values or dates (see detail in Appendix A). Therefore, we also manually classified the correctness of the derived triples and answers for the numerical comparison questions, adding the evaluation of HotPot_NC, 2Wiki_NC, JEMHop_NC and DROP_NC (see in §2.1) in the CoT 5-shot setting4.
4As DROP lacks evidence information, few-shot examples
The results are in Table 5. In this table, we find that as much as 36-46% of the answers were spuriously correct in 3 of the 4 datasets – with the exception of HotPot_NC, where the rate of spurious correct answers remained lower at 16.7%. While it was not obvious to us why HotPot_NC behaved differently, we could see why spurious correctness happens often in numerical comparison: they occur when the relative order of numbers or dates are not affected even when there is an error in derivation triples. This is also observed when we analyzed the results of bridge comparison questions in 2Wiki – out of 14 correct answers of this type, 10 were in fact spurious in the same manner as the numerical comparison questions: there was an error in the identification of bridge entity (identifying a wrong person), but the relative order of the dates required for the answer was unaffected. In order for the answers to be spuriously correct in this way, the error margin for the numbers/dates in the grounding knowledge must be small enough so as not to impact the relative order. Exactly how “wrong” or “close” GPT-4 is when it comes to the numerical aspect of the grounding information deserves further investigation; we leave this for future work.

heading: 3.3 Can External KBs Remedy Spurious Correct Answers?
text: GPT-4 “hallucinated” wrong derivation triples in 50-60% in each dataset as a whole. We investigated whether this knowledge hallucination can be fixed by using external KBs.
For this, we used two existing KBs on Wikipedia,
of CoT-5shot are created using the same data as 2Wiki.
namely Wikidata (Vrandečić and Krötzsch, 2014) and Shinra5 (Sekine et al., 2019). The latter extracts attribute-value pairs from Japanese Wikipedia articles and structures them according to the ENE (Sekine, 2008) categories in Sekine et al. (2020); this is used for JEMHopQA only as it is in Japanese. Also, in 2Wiki, all hallucinated derivation triples can be found by Wikidata as the questions of 2WikiMultihopQA derive from the knowledge triples in Wikidata. Therefore, we studied the extent to which gold derivation triples in each dataset are covered by external KBs for HotPot and JEMHop only. Knowledge representation in these KBs is compatible with the derivation triples used in our task, allowing for a straightforward application.
In Table 6, the first three columns show the coverage of derivation triples of each dataset for GPT-4, Wikidata, and GPT-4 combined with Wikidata. We assume that the derivation triples generated by GPT4 in answering the questions are GPT-4’s internal knowledge and estimate GPT-4’s coverage by calculating how well GPT-4’s internal knowledge covers the gold derivation triples in each dataset. As a multi-hop question requires two or more triples to answer, a partial coverage statistic is also given. We see that GPT-4 provides complete evidence for 51% and 48% of HotPot and JEMHop questions respectively, but if combined with Wikidata, it can cover up to 59% and 63% respectively. The last three columns show the coverage of derivation triples of Shinra, GPT-4 combined Shinra and GPT-4 combined with both KBs. GPT-4 and both KBs seem to complement each other well: GPT-4 combined with both KBs achieves 81.7% of coverage, up by
5http://shinra-project.info/
31% as compared with GPT-4 alone (48.3%). This indicates that a further improvement in multi-hop QA task is possible by combining LLM with existing KBs, a fruitful direction for future research.

heading: 4 Conclusions
text: In this paper, we presented the evaluation of GPT-4 on multi-hop QA in three datasets in English and Japanese, focusing on how the answers are/are not grounded on the knowledge internal to the model. The results show that almost all of the incorrect answers are due to knowledge hallucination, and that even when the answer is correct, up to 31% of them (40% in numerical comparison questions) are in fact spurious. We also showed that the knowledge GPT-4 uses for grounding is complementary with external KBs, indicating a future direction of integrating them for solving multi-hop questions. Our analysis is based on the assumption that the derivation triples generated by the LLM are reasoning of the LLM, but we hope to clarify whether this assumption is correct in the future.

heading: Acknowledgements
text: This work was supported by JSPS KAKENHI Grant Number JP20269633 and 19K20332. The authors would like to thank the anonymous reviewers for their insightful feedback.

heading: A Detailed Types of Spurious Correct Answers
text: Table 7 shows the percentage of spurious correct answers by question type in each dataset. They mainly appeared in comparison and bridgecomparison questions, with numerical comparison being the most frequent (38% in 2Wiki comparison , 50% in 2Wiki bridge-comparison, 68% in JEMHop).
Spurious correct answers of comparison questions. Table 8 shows examples of spurious correct answers in comparison questions. In “numerical comparison”, the relative order of dates (e.g., "1212" vs "1248") or values (e.g., "1.5" vs "2.0") in GPT-4’s derivations matched the gold, despite incorrect date or values. In “shared predicate”, the answer condition (e.g., whether authors are the same in both entities) was unaffected, despite different authors ("Meka Tanaka" vs "Oreko Tachibana") in GPT-4’s and gold derivations.
Spurious correct answers of composition questions. Table 9 shows examples where the answer was correct despite incorrect bridge entities. In one case, different princes were from the same family and birthplace. In others, the bridge entity was unspecified or non-existent, suggesting the model knew the answer in advance. For example, GPT-4 correctly answered "World War II" for when a facility was established, despite using a non-existent bridge entity.
Spurious correct answers of brigecomparison questions. Table 10 shows examples in “numerical comparison” and “shared predicate” types. The answer was unaffected despite wrong bridge entities, as the relative order of dates (e.g., "1936" vs "1956") or conditions like directors’ countries remained unchanged.

heading: B Example of Prompts for GPT-4
text: The following are examples of the three types of prompts we used in our experiments:
HotPot 2Wiki JEMHop
Comparison Numerical comparison 16.7% 38.9% 68.4% Entity selection 16.7% 0.0% 10.5% Shared predicate 0.0% 0.0% 15.8%
Composition Entity or value answer 66.7% 5.6% 5.3%
Bridge-comparison Numerical comparison 0.0% 50.0% - Shared predicate 0.0% 5.6% -

heading: C Detailed Manual Evaluation of Derivations
text: In the manual evaluation of the derivations output by GPT-4, even if the derivations did not exactly
match the gold derivations, they were considered correct if they were in the form of triples and provided sufficient information to derive the answer from the question. The specific cases considered correct are as follows:
i Differences in wording (tense, synonymous verbs or nouns, presence or absence of modifiers).
ii Differences in granularity of information (geographic, temporal, etc. units).
iii Differences in type of information.
iv Differences in the amount of information contained in a triple (cases where multiple triples of information in the gold are combined into one in the pred (GPT-4 output) and vice versa).
v Differences in how triples are formed ( the subject and object of the triple are opposite, or part of the object of the gold triple is included in the relation of the pred triple, etc.).
Examples for each pattern are shown in Table 11.

heading: Future Work
text: We also showed that the knowledge GPT-4 uses for grounding is complementary with external KBs, indicating a future direction of integrating them for solving multi-hop questions. Our analysis is based on the assumption that the derivation triples generated by the LLM are reasoning of the LLM, but we hope to clarify whether this assumption is correct in the future