Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs (2024)

Abstract: Large language models (LLMs) have demonstrated strong reasoning abilities when prompted to generate chain-of-thought (CoT) explanations alongside answers. However, previous research on evaluating LLMs has solely focused on answer accuracy, neglecting the correctness of the generated CoT. In this paper, we delve deeper into the CoT reasoning capabilities of LLMs in multi-hop question answering by utilizing knowledge graphs (KGs). We propose a novel discriminative and generative CoT evaluation paradigm to assess LLMs’ knowledge of reasoning and the accuracy of the generated CoT. Through experiments conducted on 5 different families of LLMs across 2 multi-hop question-answering datasets, we find that LLMs possess sufficient knowledge to perform reasoning. However, there exists a significant disparity between answer accuracy and faithfulness of the CoT generated by LLMs, indicating that they often arrive at correct answers through incorrect reasoning. 1

heading: 1 Introduction
text: While large language models (LLMs) have shown great potential as general-purpose task solvers, they tend to be unreliable reasoners (Bang et al., 2023). Prior research suggests that LLMs demonstrate reasoning-like behaviors as the number of parameters increases (Wei et al., 2022). Notably, Chainof-Thought (CoT) prompting, where LLMs are explicitly instructed to decompose questions into a sequence of logical steps before generating answers, has achieved impressive performance in various reasoning tasks (Wei et al., 2022; Kojima et al., 2022). However, as LLMs function as black-box models, the mechanism behind their reasoning processes remains largely unknown.
∗Equal contribution 1Code and data are available at: https://github.com/
MinhVuong2000/LLMReasonCert
Previous research measures the reasoning ability of LLMs by reporting their performance, e.g. accuracy, on the downstream tasks that require reasoning (Huang and Chang, 2023). This evaluation strategy cannot provide a direct assessment of the reasoning steps. Hence, it remains unclear whether their strong performance is the result of true reasoning ability or simple heuristics. Recent studies on analyzing CoT reasoning introduce perturbations to prompts, including the injection of invalid reasoning paths, incorrect facts, or the addition of arbitrary symbols to the few-shot examples (Madaan et al., 2023; Wang et al., 2023a; Ye et al., 2023). These studies show that various aspects of prompts, such as query relevance, style patterns, and the correct ordering of reasoning steps,
2862
are more important than the validity of reasoning in the in-context demonstrations. While revealing interesting insights into the reasoning process of LLMs, prompt perturbation-based methods still cannot directly evaluate the correctness of reasoning steps. Automatically verifying CoT reasoning steps is still an open challenge due to the unstructured nature of its freeform rationales.
In this paper, we go beyond evaluating only the final answers to directly analyzing the intermediate reasoning steps generated by CoT prompting in multi-hop question-answering (QA) tasks. To tackle the unstructured nature of CoT, we introduce a novel evaluation framework that grounds LLMs’ responses in knowledge graphs (KGs) and verify whether it forms a faithful path to the given KGs. Before evaluating more open-ended generative reasoning skills, we design discriminative tests to assess whether LLMs can identify faithful reasoning paths when perturbed with factual errors, incoherent, and misguided reasoning steps. Our discriminative evaluation results reveal that LLMs possess certain knowledge of valid reasoning under sufficient knowledge conditions. Building on this observation, we further propose the generative evaluation to measure the reasoning ability of LLMs and detect fine-grained reasoning errors (see Figure 1). In the generative evaluation, we instruct LLMs to generate CoT in a structured format, enabling us to parse their responses into a structured reasoning path and validate against KGs. Our ablation study with human experts shows that our framework achieves good accuracy in reasoning path retrieval and evaluation.
We use the proposed evaluation framework to understand the CoT reasoning process of five stateof-the-art LLMs on two complex QA tasks, which require performing multi-step reasoning to answer the questions. Our study reveals that
• LLMs contain sufficient knowledge to conduct reasoning. However, they are still limited in considering the coherence of the reasoning and hallucinations during CoT generation.
• The correct final answer may not necessarily follow from faithful reasoning. We observe a significant gap between answer accuracy and reasoning faithfulness. It highlights the necessity of directly evaluating the reasoning steps rather than solely scoring the final answers.
• The performance gap between the final an-
swer and reasoning worsens as the model size increases. As the answer accuracy also increases with the model size, it suggests that the bigger models may have the knowledge of the final answer without the need to perform reasoning.
• Better prompting strategies such as selfconsistency or instructing LLMs with planning can further improve both the final answer and reasoning faithfulness.

heading: 2 Preliminaries
text: Chain-of-thought (CoT) Reasoning Chain-ofthought (CoT) (Wei et al., 2022) is a reasoning framework that prompts LLMs to generate a stepby-step reasoning process S = {s1, s2, . . . , sn} to a question q, where si is a natural language sentence describing a step in the reasoning process.
Faithful CoT A faithful CoT should satisfy the following properties (Creswell and Shanahan, 2022): (i) there are no factual errors, (ii) the reasoning process is coherent (i.e., the conclusion of previous step si−1 should be the prerequisite of the current step si), (iii) the reasoning process leads to the correct answers. Examples of violations of these properties are shown in Figure 1.
Knowledge Graphs (KGs) Knowledge graphs (KGs) are structured representations of knowledge that contain abundant facts in the form of triples G = {(eh, r, et) | eh, et ∈ E , r ∈ R}, where eh and et are head and tail entities, and r is the relation between them; E andR are the entity and relation sets respectively. A path in KGs is a sequence of triples: P = e0
r1−→ e1 r2−→ . . . rl−→ el, connecting the entity e0 to the entity el.
Reasoning Paths Given a question q and the answer a, a valid reasoning path P ∗ = eq
r1−→ e1 r2−→ . . .
rl−→ ea is a path that connects the topic entity eq of q to the answer entity ea of a in KGs. The reasoning path P ∗ expresses a valid reasoning process for answering the question according to the KG. Example 1. Given a question “Who is the brother of Justin Biber?”, we can find a valid reasoning path P ∗ in KGs as: Justin Bieber child_of−−−−−→ Jeremy Bieber father_of−−−−−−→ Jaxon Bieber. It indicates: (i) Justin Bieber is the child of Jeremy Bieber, and (ii) Jeremy Bieber is the father of Jaxon Bieber. Thus, the brother of Justin Bieber is Jaxon Bieber.
Faithful CoT Grounded by KGs We verify the faithfulness of the LLMs’ CoT reasoning by grounding it with KGs. By treating each reasoning step as a triple in KGs, we convert the CoT into a reasoning path. If a reasoning path starting from the question and ending at the answers exists in KGs, we deem the CoT of LLMs faithful. A grounded example is shown at the bottom of Figure 1.

heading: 3 Evaluating the CoT Reasoning of LLMs
text: We propose a framework to evaluate the CoT reasoning process of LLMs with the help of KGs. Specifically, we propose two evaluation modules: discriminative evaluation and generative evaluation. The discriminative evaluation investigates whether LLMs possess enough knowledge for conducting faithful reasoning and the generative evaluation further analyzes whether LLMs can provide faithful reasoning process during CoT generation. The overall framework is shown in Figure 3.

heading: 3.1 Discriminative Evaluation
text: The discriminative evaluation aims to analyze whether the LLMs possess enough knowledge to conduct faithful reasoning. i.e. whether it can recognize certain properties of faithful reasoning, including no factual error, coherence and leading to correct answers. We hypothesize that if the LLMs possess sufficient knowledge for faithful reasoning, they should be able to distinguish valid reasoning paths from invalid ones given the question and answer. Following previous studies that evaluate the factual knowledge inside LLMs (Luo et al., 2023b), we feed both the valid and invalid reasoning paths to the LLMs and ask them to predict the validity of these paths. This allows us to assess the reasoning knowledge inside LLMs by analyzing their prediction accuracy. We carefully design prompts to describe the task and instruct LLMs to provide the prediction. Figure 2 shows an example of the zero-shot prompt template.
A valid reasoning path is a sequence of triples that can be used to derive the answer of given question. The valid reasoning paths are extracted from the ground-truth reasoning paths2 P ∗ ∈ P∗. We generate three types of invalid reasoning paths P ′ by breaking specific properties of a faithful CoT:
• Factual error reasoning path: we construct 2The ground-truth reasoning paths are constructed from the SPARQL queries provided in the datasets. The detailed construction is shown in Appendix A.
the invalid paths with factual errors by randomly corrupting entities within the valid reasoning path. This would result in some factual errors in the reasoning path, which are not valid for answering the question.
• Incoherent reasoning path: we shuffle the triples of valid paths to construct an incoherent reasoning path. Even though the facts within the paths are accurate, the overall coherence of the path is compromised.
• Misguided reasoning path: we randomly sample the paths starting from other questions in KGs. These paths are factually correct and coherent, but they are not related to the questions and lead to incorrect answers.
To thoroughly assess the reasoning abilities of LLMs, in addition to the zero-shot prompt, we have also developed few-shot, zero-shot CoT, and fewshot CoT prompts. The details of these prompts are shown in Appendix F.1.
Findings The results of the discriminative assessment are shown in §5.1. From the results, we can conclude that LLMs possess enough knowledge to identify factual errors as well as reasoning path relatedness, but have limitations in considering the coherence of reasoning paths and CoT generation. Therefore, we propose the generative evaluation to further assess the faithfulness of CoT reasoning in LLMs’ generation.

heading: 3.2 Generative Evaluation
text: The generative evaluation aims to assess the faithfulness of the CoT reasoning process generated by LLMs. Our main idea is to ground LLMs’ CoT into KG and verify whether it forms a valid path. To address the challenge of evaluating unstructured CoT, we carefully design a prompting strategy to instruct LLMs to output the CoT in a structured format. This enables us to parse LLM’s response into a structured reasoning path, which can then be validated against KG. The example prompts and structured CoT output are provided in Appendix F.2.
Specifically, given a generated CoT response S of question q, we first construct a reasoning path P̂ by retrieving triples from the KGs. Then, we evaluate the validity of the reasoning path by checking whether the path coherently connects the question and answer entities in the KGs. The details of these steps are explained in the following subsections.

heading: 3.2.1 Reasoning Path Construction
text: Given a CoT response S = {s1, s2, . . . , sn}, we first retrieve a triple3 T = (eh, r, et) for each step si in the CoT response. The retrieved triple is the structural representation of each reasoning step, which can be used to construct the reasoning path for evaluation.
Previous works usually retrieve triples by iden-
3We noticed in almost all cases in our experiments, a CoT step corresponds to one KG triplet. The extension to multiple triplets per CoT step is left for future work.
tifying the entities and relations mentioned in the sentence and linking them to the KGs (Lan et al., 2021; Wang et al., 2021). However, this process is not scalable to KGs. Inspired by the recent fact retrieval method (Baek et al., 2023), we represent the reasoning step s and triples in a unified embedding space and retrieve the triple T based on their embedding similarity.
For all the triples in a KG G, we verbalize each triplet into a sentence by concatenating the entities and relation x = ”eh r et.”. Then, we use the Sentence-BERT model (Reimers and Gurevych, 2019) to obtain its embedding hT = E(x). These embeddings are constructed in advance and saved in a vector database for efficient retrieval. Similarly, the embedding of a given reasoning step s is computed as hs = E(s). Then we retrieve the topK triples from KG by calculating the embedding similarity between hs and hT as:
τi = f(hs, hTi), Ti = (eh, r, et) ∈ G, (1)
where τi denotes the similarity score of triple Ti, and f(·, ·) is a non-parametric scoring function that measures the similarity between two embeddings. We adopt cosine similarity as the scoring function.
The embedding-based retrieval method may lead to the omission of entities mentioned in the reasoning step. To solve this problem, we also take into account the presence of head and tail entities in the reasoning step in the scoring function. The final
score for each retrieved triple is calculated as,
τ̃i = τi + ϵh + ϵt
3 , (2)
where ϵh and ϵt represent the fuzzy-match ratio of head and tail entities in the reasoning step, which are range from 0 to 1, where 0 denotes no existence, 1 denotes a complete match. The overall retrieval process is presented in the Algorithm 1 in the Appendix B.
Thus, we could obtain a set of triples T = {T1, T2, . . . , Tn} for the CoT response S. Then, we construct the reasoning path P̂ by connecting the triples in T .

heading: 3.2.2 Reasoning Path Evaluation
text: By evaluating the validity of constructed reasoning paths, we can assess the faithfulness of the CoT reasoning process generated by LLMs. Specifically, we evaluate the validity of the constructed path P̂ from three aspects:
• Factual correctness: P̂ contains factual error if the similarity score τ̃i of any retrieved triples are below a factual threshold σ.
• Coherence: given a factually correct path, it is incoherent if there exists a step where its premise is not the conclusion of the previous step.
• Final answer correctness: given a factually correct and coherent path, whether the final answer is correct, i.e. matched with groundtruths.
Validity of Reasoning Path The prerequisite and conclusion at each reasoning step are considered head and tail entities, respectively. If the reasoning path P̂ can connect the question and answer entities in the KG, we can conclude that it is a valid path. The detailed algorithm is shown in the Algorithm 2 in the Appendix B.
Fine-grained Assessment In addition to the binary evaluation, we also report the minimum edit distance between the constructed reasoning path P̂ and the ground-truth path P ∗. This serves as a fine-grained assessment of CoT reasoning capability. We adopt a widely used sequence alignment algorithm - Needleman Wunsch algorithm (Needleman and Wunsch, 1970) to obtain continuous alignment scores (i.e., edit distance), which indicate how close the constructed reasoning path is to the
ground-truth reasoning paths. If multiple groundtruth paths exist, we report the score against one with the highest match rate. The detailed algorithm is shown in the Algorithm 3 in the Appendix B.

heading: 4 Experiment Settings
text: We use the proposed evaluations to understand the CoT reasoning process of the state-of-the-art LLMs on complex question-answering (QA) tasks which requires performing multi-step reasoning to answer the questions. Through analysis, we seek to answer the following research questions (RQs)
• RQ1: Do LLMs have the knowledge of faithful reasoning? We leverage the discriminative evaluation to test whether LLMs can identify valid reasoning paths. This evaluation focuses on assessing LLMs’ knowledge about the properties of faithful reasoning described in Section 2.
• RQ2: Can LLMs express such knowledge to generate faithful reasoning? Utilizing our generative evaluation framework, we assess the capacity of LLMs to produce coherent and correct reasoning. We also investigate various factors, such as model size and prompting strategies, to understand their impact on reasoning capability.
Dataset We conduct experiments on two QA datasets: Complex WebQuestions (CWQ) (Talmor and Berant, 2018) and GrailQA (Gu et al., 2021) which contain up to 4-hop questions. To evaluate multi-step reasoning capability, we filter out single-hop questions in the test set. Table 1 shows the statistics of the filtered test set. The generated reasoning paths are validated against Freebase (Bollacker et al., 2008) - an open knowledge graph containing around 88M entities, 20K relations, and 126M triples. More details can be found at Appendix C.1.
Large Language Models We evaluate the reasoning capability of several LLMs with instructionfollowing capability at different sizes, including Mistral (7B) (Jiang et al., 2023), Qwen (7B, 14B)
(Bai et al., 2023), Vicuna (33B) (Chiang et al., 2023), LLaMA2-Chat (70B) (Touvron et al., 2023) and ChatGPT (175B)4(OpenAI, 2023). The details of model versions are available in Appendix C.2. We set temperature as 0.7 and top p as 0.9 for generation in all models.
Prompting Strategies We experiment with multiple CoT prompting strategies, including
• Few-shot CoT Five examples with structured CoT followed by the answer are added to the prompt (Figure 12 in Appendix F.2).
• Few-shot CoT with planning (CoT-Plan) We also explore the ability of LLMs to plan and decompose the relations required to reach the answer before verbalizing the CoT reasoning. In particular, we add the ground-truth plan (Luo et al., 2023a) (i.e., a relation path pointing to the answers) into each example. An example prompt is given in Figure 13 in Appendix F.2.
• Few-shot CoT with self-consistency (CoT-SC) Beyond the conventional CoT prompting, we also experiment with SelfConsistency (Wang et al., 2023c), a more sophisticated method designed to mitigate the inconsistencies in CoT reasoning by aggregating the final answer through majority votes. In our evaluation, we sample four outputs, and report the maximum performance across all the outputs.
Evaluation Framework Implementation Given a question from the benchmark, in discriminative evaluation, we construct the invalid paths by randomly perturbing the ground-truth paths extracted from SPARQL (Kumar et al., 2019). The implementation detail is described in Appendix A. In generative mode, we use FAISS (Johnson et al., 2019) as the vector database, Sentence-BERT (Reimers and Gurevych, 2019) as the employed embedding model and partial ratio fuzzy matching5 as the entity scoring function. We retrieve top-10 triples and set the factual threshold σ of 0.7.
Evaluation Metrics For discriminative evaluation, we report the accuracy of detecting valid reasoning paths from invalid ones. For generative
4Previous works mentioned ChatGPT having 175B parameters (Meyer et al., 2023). However, OpenAI still doesn’t give any official news about ChatGPT’s model size.
5https://github.com/seatgeek/thefuzz
evaluation, we report CoT reasoning performance of LLMs with the following metrics: (i) final answer accuracy, (ii) faithful reasoning score, and (iii) minimum edit distance between the generated and ground truth paths. As different LLMs vary in instruction-following capabilities and guardrail implementations, we may encounter responses with unstructured format or abstained answers (Luo et al., 2023b). Therefore, we classify LLMs’ responses into four groups: abstained (A), unstructured (U), faithful reasoning (FR), and unfaithful reasoning (UR). We use the F1 score to measure the faithfulness of CoT reasoning where precision and recall are calculated as P = FRFR+UR and R = FRFR+UR+A+U . Detailed implementations are described in Appendix C.3. Results of precision and recall are presented in Appendices D.2 and D.3.

heading: 5 Main Results

heading: 5.1 Discriminate Evaluation
text: Finding 1: LLMs possess knowledge of valid reasoning The overall discriminative evaluation results are shown in Table 2. Based on the results, it is evident that all LLMs achieve a high level of accuracy in distinguishing valid reasoning paths. This indicates that LLMs, which are pre-trained on large-scale corpora, already possess certain knowledge to perform reasoning tasks effectively. However, when using few-shot prompts, there is a noticeable decrease in performance for Mistral and LLaMA2. This could be attributed to the sensitivity of these particular LLMs towards the provided few-shot examples. The detailed results of each perturbation type are illustrated in Appendix D.1, where the accuracy of incoherent paths is lower than other types. We speculate that LLMs cannot capture structural information in the context (Guo et al., 2023). Moreover, the few-shot CoT fails to improve the accuracy in identifying valid paths. We speculate that LLMs are prone to hallucination during CoT generation, resulting in
incorrect predictions. We can conclude that despite having enough reasoning knowledge, LLMs still face limitations in conducting faithful reasoning during CoT generation.

heading: 5.2 Generative Evaluation
text: Table 3 shows the performance of LLMs in generative evaluation mode. Overall, ChatGPT demonstrate superior performance in terms of both final answer accuracy and faithfulness of the reasoning. Surprisingly, Mistral 7B, despite being the smallest model, exhibits competitive performance comparable to larger models within the <50B range. Furthermore, enhancing prompting strategies with planning (CoT-Plan) and self-consistency (CoTSC) results in substantial improvements across all LLMs, especially for smaller models.
Finding 2: The correct final answer may not necessarily result from faithful reasoning As shown in Table 3, there is a notable discrepancy between the accuracy of the final answer and the reasoning process. The average gap is 15.76% for CWQ, and 16.44% for GrailQA. While advanced prompting may improve answer and reasoning accuracy, this performance gap mostly stays consis-
tent. Interestingly, Vicuna achieves reasonable answer accuracy but has the lowest reasoning performance of all the models, suggesting its reasoning ability is inferior, even when compared to small models like Mistral and Qwen-7B. This finding highlights the inadequacy of relying on final answer accuracy as a proxy to gauge reasoning ability.
Finding 3: The reasoning gap worsens as the model size increases It can be seen that the reasoning performance increases gradually with model size, proving the reasoning ability of bigger models. However, the gap between answer and reasoning performance also gradually increases with model size and the correctness of the answer. While LLaMA2-70B and ChatGPT rank first in performance, their gaps are also the highest. Meanwhile, the smallest-size models, including Mistral-7B and Qwen-7B, hold the lowest gap on CWQ and GrailQA, respectively. We speculate that larger LLMs may grasp the question context better or have more knowledge to provide the correct answer directly without performing reasoning.
Finding 4: Better prompting strategy can improve both the answer and reasoning accuracy
without worsening the reasoning gap The use of enhanced prompting strategies such as CoT-Plan and CoT-SC lead to improvement of both the answer and reasoning accuracy across most LLMs. However, the gap between them remains consistent, regardless of prompt strategy.

heading: 5.3 Analysis
text: Reasoning Errors We present a detailed breakdown of the reasoning errors in Figure 4. The results reveal that factual errors account for the majority of errors, indicating that LLMs tend to generate incorrect information during reasoning. As GrailQA is a more complex dataset, LLMs have a higher percentage of coherence errors on GrailQA than CWQ. Interestingly, even when the generated CoT paths are free from factual and coherent errors, LLMs may fail to produce correct answers, evidenced by a substantial amount of answer errors. Error case examples are shown in Appendix D.4.
Ablation Study To ensure the effectiveness of our generative evaluation framework, we randomly select 100 CoT responses generated by ChatGPT in CWQ dataset and asked two human experts to evaluate the constructed reasoning path. The detail of the human evaluation study is described in the appendix E. The results presented in Table 4 demonstrate that our method can accurately detect both faithful and unfaithful reasoning paths. This further confirms the efficacy of our approach in evaluating CoT reasoning.
Parsing Error While we carefully design prompts to instruct LLMs to generate a structured
CoT, there are still corner cases where LLMs generate unstructured and abstention responses due to their unpredicted behaviors. As reported in Appendix D.2, the unstructured and abstention rates are less than 20% in CWQ dataset and can be mitigated with CoT-Plan and CoT-SC.

heading: 6 Related Work
text: Reasoning with LLMs While LLMs have proven to offer a variety of reasoning abilities, they still tend to hallucinate facts, making them unreliable and imperfect (Qiao et al., 2022). Several studies have concentrated on improving their reasoning capacity through prompting (Wang et al., 2023c; Ye and Durrett, 2022; Wiegreffe et al., 2022). CoT (Wei et al., 2022) is a prompting approach that has demonstrated notable improvements in reasoning performance. A significant enhancement compared to CoT, self-consistency (Wang et al., 2023c), is a scheme where multiple CoTs are generated and the most consistent self-generated answer is selected. Recently, self-consistency was extended with Tree of Thoughts (ToT) (Yao et al., 2023), which models the reasoning process with a tree. ToT allows LLMs to interactively backtrack and explore alternate chains of reasoning, avoiding getting stuck on a single line of incorrect reasoning. Ye and Durrett (2022) mitigate the effect of unreliable rationales by calibrating the prediction probability based on the factuality of CoT. Wiegreffe et al. (2022) train a Seq2Seq model to filter out unacceptable rationale. Liu et al. (2021) utilize GPT-3 (Brown et al., 2020) with few-shot prompting to generate knowledge and prompts the downstream language models.
Reasoning Evaluation Evaluation of the reasoning ability of LLMs has been undertaken for two main purposes: to enhance the reasoning ability of LLMs (Lyu et al., 2023; Li et al., 2023; Tyen et al.,
2023; Chen et al., 2023) and to quantify the reasoning ability of LLMs (Wang et al., 2023b; Atanasova et al., 2023). For instance, Huang and Chang (2023) gauges the reasoning ability of LLMs by assessing their performance on reasoning benchmarks such as GSM8K and BIG-bench for downstream tasks. However, this evaluation strategy is unable to offer a direct assessment of the reasoning steps. Tyen et al. (2023) release the BIG-Bench Mistake dataset, which includes logical errors in CoT reasoning steps. Using this benchmark, Tyen et al. (2023); Chen et al. (2023) illustrate the inability of state-of-the-art LLMs to identify mistakes and reasoning errors, even in unequivocal cases.

heading: 7 Conclusion
text: We propose an evaluation framework to understand the CoT reasoning capability of LLMs beyond the sole assessment of final answer accuracy. With the help of a KG and a careful prompting strategy, we can turn the unstructured CoT into a structured format for automatic evaluation. Our framework consists of two evaluation modules: (i) a discriminative module that isolates the effects of different reasoning errors to verify LLMs’ knowledge about reasoning, and (ii) a generative module to assess the generated CoT reasoning. While LLMs showcase remarkable capabilities in generating correct answers, our study emphasizes the need for more nuanced evaluations of their reasoning processes. Addressing the gap between the final answer and reasoning accuracy remains a critical area for further exploration in enhancing the true reasoning capabilities and interpretability of LLMs.

heading: Limitation
text: The limitation of our work includes:
• We consider a CoT step corresponding to one KG triple and a single correct answer for each question. However, LLMs may generate a sentence containing more than two relations. This can be tackled by returning top-K candidates from Algorithm 1 and a dynamic program algorithm expanded from Algorithm 2.
• We assume the availability of completed knowledge graphs (KGs) for factual retrieval. We leave the incorporation of the knowledge graph completion methods to improve the comprehension of the retrieval algorithm as future works.
• This study mainly focuses multi-hop reasoning questions over knowledge graphs (KGs). Nevertheless, there exist intricate reasoning inquiries, such as those in mathematics or logical reasoning, which involve unstructured replies that are not easily resolvable through KGs. The establishment of verification frameworks for diverse forms of reasoning queries plays a significant role in enhancing the reliability and utility of responses generated by LLMs. This aspect is still an open and challenging problem, requiring extensive explorations within the research community.

heading: Acknowledgments
text: The authors are grateful to the anonymous reviewers for their helpful comments. The computational resources of this work are supported by the Multimodal Australian ScienceS Imaging and Visualisation Environment (MASSIVE)6. This material is based on research partially sponsored by the DARPA Assured Neuro Symbolic Learning and Reasoning (ANSR) program under award number FA8750-23-2-1016 and the DARPA Knowledge Management at Scale and Speed (KMASS) program under award number HR00112220047. MinhVuong Nguyen was supported by VinAI Research.

heading: A Ground-truth Reasoning Path Construction
text: To obtain graph and ground truth reasoning paths for each question, we utilize SPARQL query, topic entities, answer entities provided in datasets to construct subgraphs, then find paths from topic entities to answer entities as the ground-truth reasoning paths. In particular,
• We replace SELECT query in each SPARQL query into a CONSTRUCT query to return the corresponding graph. An example CONSTRUCT query and its returned graph from the CWQ dataset is shown in example 2.
• In the returned graph, we use NetworkX7 library to find the paths starting at one of the topic entities and ending at one of the answer entities provided in original dataset. As a result, we obtain a list of ground-truth reasoning paths used for both discriminative and generative evaluation.
Example 2. An example CONSTRUCT query and the corresponding returned graph.
CONSTRUCT query
PREFIX ns: <http://rdf.freebase.com/ns/> CONSTRUCT WHERE { FILTER (?x != ?c) FILTER (!isLiteral(?x) OR lang(?x) = ” OR langMatches(lang(?x), ’en’)) ?c ns:location.country.administrative_divisions ns:m.02g__4 . ?c ns:location.country.languages_spoken ?x . }
Corresponding returned graph
m.03gyl location.country.administrative_divisions m.02g__4 . m.03gyl location.country.languages_spoken m.02dhwl . m.03gyl location.country.languages_spoken m.064_8sq .

heading: B Generative Evaluation
text: The details of triple retrieval, reasoning path evaluation, and fine-grained path evaluation are shown in the Algorithms 1 to 3, respectively.

heading: C Experiment Settings

heading: C.1 Datasets
text: We adopt two benchmark KGQA datasets: Complex WebQuestions (CWQ)8(Talmor and Berant, 2018) and GrailQA9(Gu et al., 2021) in this work,
7https://networkx.org/ 8https://www.tau-nlp.sites.tau.ac.il/compwebq 9https://huggingface.co/datasets/grail_qa
Algorithm 1: Triple Retrieval Algorithm Input: Step s; Top-K; embedding model E;
knowledge graph G. Output: Triple T
1 hs ← E(s) 2 [Ti, τi] K i=1 ← G.search(hs, k) 3 C = [] 4 for i← 1 to K do 5 ϵh, ϵt ← Fuzzy-match(Ti, s) 6 τ̃i ← τi+ϵh+ϵt3 7 C.append((τ̃i, Ti)) 8 end 9 T = argmax
Ti∈C τ̃i
especially we use the test split of CWQ, and validation split of GrailQA. We only keep questions requiring more than 2-hop reasoning. The number of questions is shown in Table 1. Both CWQ and GrailQA can be reasoned based on Freebase KGs10(Bollacker et al., 2008). To reduce the KG size, we combine the subgraphs obtained from SPARQL queries as the final KG. The detail of subgraphs extraction is shown in Appendix A.

heading: C.2 Large Language Models
text: The LLMs used in experiments are shown in Table 5. We utilize available checkpoints from HuggingFace11.

heading: C.3 Evaluation Metrics
text: Faithful Reasoning Score. To enable robust and truthful reasoning, many LLMs adopt the guardrail techniques to abstain from providing answers when they are uncertain (Liu et al., 2023; Luo et al., 2023b). Also, LLMs exhibit different instructionfollowing capabilities, which may result in the CoT response in unstructured format. Thus, we need to consider both the abstained and unstructured responses in the evaluation. We define the precision and recall as,
Precision = #correct#correct+#incorrect , (3) Recall = #correct#correct+#incorrect+#abstained+#unstructured .
(4)
The F1 score is then the harmonic average of the precision and recall. The details of abstained
10https://github.com/microsoft/FastRDFStore 11https://huggingface.co/
Algorithm 2: Reasoning Path Evaluation Input: Reasoning path P̂ ; threshold σ; knowledge graph
G; ground answer entity A. Output: Validity v.
1 factual_error← False 2 order_error ← False 3 answer_error ← answerP̂ ! = A 4 for Ti ∈ P̂ do 5 if τi < σ then 6 factual_error← True 7 else 8 continue 9 if headTi ! = tailTi−1 then
10 order_error ← True 11 end 12 end 13 end 14 coherent_error ← ¬factual_error ∧ order_error 15 answer_error ← ¬coherent_error ∧ answer_error 16 if factual_error ∨ coherent_error ∨ answer_error
then 17 v ← False 18 else 19 v ← True 20 end 21 return v
and unstructured response detection are in Appendices C.4 and C.5.
Answer Accuracy The answers generated by LLM can have a different length than the groundtruth, so we utilize fuzzy matching to check the correctness of answer. It checks whether the generated answer appears in the ground-truth answers, and vice versa.
Edit Distance We further look into the finegrained evaluation of the reasoning path by calculating the edit distance (Needleman and Wunsch, 1970): The minimum number of edits (add/remove the reasoning steps) required to convert the CoT path to the ground-truth path.

heading: C.4 Abstained Answer Detection
text: We detect abstained answers through following specific keywords in LLMs’ responses:
Algorithm 3: Fine-grained Path Evaluation Input: Reasoning path P̂ ; Ground-truth paths
P∗ Output: Edit distance u
1 m← 0 2 u← 0 3 for P ∗ ∈ P∗ do 4 u′ ← NeedlemanWunsch(P̂ , P ∗) 5 if u′ < u then 6 u← u′ ▷ Get minimum edit distance. 7 end 8 end 9 return u
LLM Model Implementation Mistral 7B mistralai/Mistral-7B-Instruct-v0.1 QWen 7B Qwen/Qwen-7B-Chat QWen 14B Qwen/Qwen-14B-Chat Vicuna 33B lmsys/vicuna-33b-v1.3 LlaMA2 70B meta-llama/Llama-2-70b-chat ChatGPT GPT-3.5-turbo

heading: C.5 Unstructured Answer Detection
text: As the instruction shown in prompts 12 and 13, a structured CoT should follow the following format:
A structured answer
1. <step1> 2. <step2> ... (T|t)he answer( to the question)? is ? ?(.∗?)?.?
Therefore, the CoT responses that do not match the pattern are identified as unstructured.

heading: D Additional Results

heading: D.1 Detailed Results of Discriminative Evaluation
text: We illustrate the discriminative evaluation results of different types of reasoning paths in Table 6. For each error types, LLMs reach the best performance under factual errors and misguided reasoning path. However, the performance under incoherent paths is lower, which could due to the limits of LLMs in understanding structural context. Besides, the performance of valid reasoning path is slightly lower than invalid reasoning path. Because it requires both three properties (i.e., factual errors, incoherence, misguidance) satisfied at the same time.

heading: D.2 Detailed Results of Precision
text: Based on the definition in eq. (3), precision ignores the abstained and unstructured responses. The detailed precision results as well as the abstained and unstructured ratio are shown in Figures 5 to 7.
The precision of the answer gradually improves with the size of the model. However, Mistral 7B and Qwen 14B are competitive in terms of reasoning performance compared to models with ten times more parameters. While Vicuna 33B has an intermediate model size, its performance is low. It is also worth noting that Qwen variants exhibit a high rate of abstention. This leads to abstaining from answering uncertain questions and achieving higher precision, but at the cost of reduced recall for correct answers and reasoning.

heading: D.3 Detailed Results of Recall
text: The recall is calculated on all types of responses. Thus, we directly report the recall in Table 7.

heading: D.4 Case Studies
text: We present the detailed cases of different error types in Table 8.

heading: E Human Evaluation
text: This section aims to be two-fold. Firstly, it provides a sanity check of the proposed prompt in instructing LLMs to generate CoT in a structured format. Secondly, we aim to construct a small test set to evaluate the capability of the proposed framework in detecting the faithful CoT.

heading: E.1 Annotation
text: Firstly, a sample of 100 questions is chosen randomly from the CWQ dataset along with the re-
sponses generated by ChatGPT using proposed prompts. Each question in the sample is matched with reference answers, which include the final answer provided in the datasets, as well as all the ground-truth reasoning paths extracted during the preprocessing phase outlined in appendix A. This data is supplied to annotators for classification purposes.
The annotators are two PhD students who are familiar with the related works and have experience working with knowledge graphs. To prepare them for the task, we provide them the definition of faithful reasoning and error types. Given sample cases that consist of the question, answer, the ground-truth reasoning paths (the desired reasoning steps), and the model’s initial CoT response, two human experts performed a trial run to align their understanding and coding criteria for faithful CoT responses by answering two following questions:
• Q1: If LLM response is an incorrect final answer? True labeling if LLM final answer does not hit the reference answers; False otherwise.
• Q2: If LLM CoT is an incorrect reasoning? True labeling if there exists any step its relation, subject/object is in groundtruth reasoning paths but there is no link between them or the object/subject/relation is not in the ground-truth reasoning paths; False otherwise.
The human annotation dataset acquired ultimately comprises 100 samples, with each sample consisting of the question, ground truth paths, LLM answers, a column indicating the annotated incorrect answer (1 for True labeling of Q1, 0 otherwise), and another column indicating the annotated incorrect reasoning (1 for True labeling of Q2, 0 otherwise).

heading: E.2 Framework Evaluation
text: Given the human annotation data provided above, our generative evaluation framework is assessed to ensure its quality prior to its application in all experiments. Initially, the responses from the LLM are inputted into the generative evaluation mode to obtain predictions for Q1 (predicted incorrect answer) and Q2 (predicted incorrect reasoning). Consequently, these two outcomes are compared with the annotated dataset to assess the precision, recall, and f1_score for the final score. The metrics are shown in table 4.
0
20
40
60
Mi str
al 7B
Qw en
7B
Qw en
14 B
Vic un
a 33
B
Lla ma
2 7 0B
Ch atG
PT
answer precision reasoning precision abstention rate unstructured rate
CWQ
0
10
20
30
Mi str
al 7B
Qw en
7B
Qw en
14 B
Vic un
a 33
B
Lla ma
2 7 0B
Ch atG
PT
answer precision reasoning precision abstention rate unstructured rate
GrailQA
Figure 5: The precision of LLMs using few-shot CoT prompt.
0
20
40
60
Mi str
al 7B
Qw en
7B
Qw en
14 B
Vic un
a 33
B
Lla ma
2 7 0B
Ch atG
PT
answer precision reasoning precision abstention rate unstructured rate
CWQ
0
10
20
30
Mi str
al 7B
Qw en
7B
Qw en
14 B
Vic un
a 33
B
Lla ma
2 7 0B
Ch atG
PT
answer precision reasoning precision abstention rate unstructured rate
GrailQA
Figure 6: The precision of LLMs using few-shot CoT - Plan prompt.
0 20
40
60
Mi str
al 7B
Qw en
7B
Qw en
14 B
Vic un
a 33
B
Lla ma
2 7 0B
Ch atG
PT
answer precision reasoning precision abstention rate unstructured rate
CWQ
0
10
20
30
Mi str
al 7B
Qw en
7B
Qw en
14 B
Vic un
a 33
B
Lla ma
2 7 0B
Ch atG
PT
answer precision reasoning precision abstention rate unstructured rate
GrailQA
Figure 7: The precision of LLMs using few-shot CoT - SC prompt.

heading: F Prompts

heading: F.1 Discriminative Evaluation Prompts
text: We present the zero-shot, zero-shot CoT, few-shot, and few-shot CoT prompts in Figures 8 to 11.

heading: F.2 Generative Evaluation Prompts
text: We present the few-shot, few-shot plan prompts in Figures 12 and 13.

heading: Zero-shot CoT Discriminative Evaluation Prompt

heading: Zero-shot Discriminative Evaluation Prompt

heading: Few-shot Discriminative Evaluation Prompt

heading: Few-shot CoT Discriminative Evaluation Prompt

heading: Q: {Question}

heading: Q: {Question}

heading: Future Work
text: We leave the incorporation of the knowledge graph completion methods to improve the comprehension of the retrieval algorithm as future works. • This study mainly focuses multi-hop reasoning questions over knowledge graphs (KGs). Nevertheless, there exist intricate reasoning inquiries, such as those in mathematics or logical reasoning, which involve unstructured replies that are not easily resolvable through KGs. The establishment of verification frameworks for diverse forms of reasoning queries plays a significant role in enhancing the reliability and utility of responses generated by LLMs. This aspect is still an open and challenging problem, requiring extensive explorations within the research community