Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments (2024)

Abstract: Large Language Models (LLMs) have shown potential in reasoning over structured environments, e.g., knowledge graphs and tables. Such tasks typically require multi-hop reasoning, i.e., match natural language utterance with instances in the environment. Previous works adopt LLMs to incrementally build a reasoning path, where LLMs either invoke tools or pick up items by step-by-step interacting with the environment. We propose Reasoning-PathEditing (Readi), a novel framework where LLMs can efficiently and faithfully reason over structured environments. In Readi, LLMs initially generate a reasoning path given a query, and edit the path only when necessary. We instantiate the path on structured environments and provide feedback to edit the path if anything goes wrong. Experimental results on three KGQA and two TableQA datasets show the effectiveness of Readi, significantly surpassing previous LLM-based methods (by 9.1% Hit@1 on WebQSP, 12.4% on MQA-3H and 9.5% on WTQ), comparable with state-of-theart fine-tuned methods (67% on CWQ and 74.7% on WebQSP) and substantially boosting the vanilla LLMs (by 14.9% on CWQ). Our code will be available on https://aka.ms/ readi.

heading: 1 Introduction
text: Large Language Models (LLMs) have demonstrated remarkable performance in NLP fields (Ouyang et al., 2022; OpenAI, 2023; Liang et al., 2022). To further unleash their reasoning ability in complex scenarios, delicate strategies are proposed to equip LLMs with human-like thought process (e.g., Chain-of-thought Wei et al., 2022) or leverage them as autonomous agents capable of planning, reflection and execution (Yao et al., 2023; Liu et al., 2024). One compelling scenario where LLMs showcase their potential
*This work is done during the internship at Microsoft.
training
er -
college
dback
Path Instantiation
Only when necessary Standard operation
is reasoning over structured environments (SEs) (Jiang et al., 2023a; Sun et al., 2024). With dedicate schemas, SEs (e.g., knowledge graphs, tables) abstract real-world semantics for representing, storing, and querying data with relational structures. For instance, Freebase (Bollacker et al., 2008) captures 45M entities and 3B facts over 100 domains, organized in triple patterns. The crux of successful reasoning lies in bridging the gap between natural language and the mechanism of how the SEs are represented and operated (Gu et al., 2023; Li et al., 2023).
While LLMs exhibit promising capabilities, their performance often falls short when faced with multi-hop reasoning involving large-scale SEs. To faithfully reason, prior works adopt an iterative way that start from certain elements (e.g., entity, relation in KG, column in Table), instantiate on SEs and then gradually expand the reasoning path while consuming just relevant portions instead of the entire environment (Sun et al., 2024; Jiang et al., 2023a; Huang et al., 2024). This step-by-step interaction could mitigate hallucination of LLMs to some extent. However, the reasoning efficiency is sacrificed and thus hinder the practical feasibility. For a simple constraint, say “the daughter of Obama”, such methods probably require two LLMcalls to first query the relations around “Obama” and then select “father_of” from returned candidates. Moreover, at each step, LLMs make one
4275
choice based on history, making it prone to error propagation. Alternatively, fine-tuned methods inject environments into model parameters by tuning with human-labeled supervision. During inference, they recall schema patterns to build reasoning paths (Zhang et al., 2022; Saxena et al., 2020; LUO et al., 2024; Ding et al., 2024; Xiong et al., 2024) without interaction with SEs. This end-to-end paradigm is efficient. However, it is never ensured that the model output can be grounded on SEs. Study shows that 50% paths of RoG (LUO et al., 2024) failed to yield faithful results. In addition, they heavily rely on annotations, which are difficult to obtain for large-scale SEs. Further research is thus needed to achieve efficient (i.e., less LLM-calls) and faithful (i.e., grounded on SEs) reasoning over massive SEs.
Therefore, we seek to propose an interaction paradigm that leverage LLMs to support complex reasoning on large-scale SEs faithfully and efficiently. Conducting empirical study on KGQA task, we find that 46%-60% of reasoning paths initially generated by LLMs can be well instantiated, which inspires us to fully exploit LLM’s intrinsic planning ability in complex reasoning. While the idea of plan-and-refine is straightforward and applied in various real world tasks (Qiao et al., 2023; Collaboration, 2023; Ichter et al., 2022), it’s worth noting that there exists few research on application in SEs when the initial plan encounters obstacles.
In this paper, we propose Reasoning-PathEditing (Readi), a novel framework that leverages the intrinsic planning ability of LLMs (Figure 1). In Readi, LLMs initially generate a reasoning path, which is then instantiated on SEs to facilitate faithful reasoning. Path editing is triggered only if corrections are necessary. This way, we alleviate the burden of step-by-step interaction for LLMs, resulting in improved overall efficiency. To harness the information of large-scale SEs, instead of injecting the entire static SEs into the model, we collect reasoning log as immediate feedback which includes details such as the position of stuck points, associated relations, half-way done instances, etc. This dynamic guidance refines the reasoning path more targeted and further enhances faithfulness. Our experiments on Question Answering over Knowledge Graphs (KGQA) and Tables (TableQA) demonstrate that Readi surpasses existing solutions in terms of both LLM-calls and accuracy.
We summerize our contributions as follows: • We introduce Readi, a novel framework where
LLMs reason efficiently and faithfully over largescale structured environments. Notably, Readi is the first to fully harness LLMs intrinsic planning ability for reasoning in such contexts. • In comprehensive experiments across five multi-hop reasoning tasks in KGQA and TableQA, Readi outperforms other LLM-based solutions and surpasses most fine-tuned methods. Specifically, it achieves 67.0% Hit@1 on CWQ, 78.7% on WebQSP and state-of-the-art results on MQA-1H. • We give detailed analysis which highlights the performance of Readi’s reasoning path generation and editing modules. Experiments demonstrate that Readi, with an average of 1.55 calls for editing, significantly reduces the number of LLM-calls compared to the step-by-step interaction paradigm (which costs 4 to 8 calls). Furthermore, the reasoning log reveals that Readi exhibits characteristics akin to human thought process.

heading: 2 Related Works
text: Step-by-step reasoning over structured environments by LLMs. Introducing massive SEs (e.g., Freebase (Bollacker et al., 2008) captures 45M entities and 3B facts over 100 domains) into LLMs context windows is impractical. Existing works break the task down to incrementally construct a reasoning path. They either treat LLMs as an agent to invoke tools based on history states and observations (Liu et al., 2024; Qin et al., 2023), or design iterative procedures where LLMs are responsible for picking up items on SEs (Gu et al., 2023; Jiang et al., 2023a; Sun et al., 2024). These works reach faithfulness by leveraging the reasoning ability of LLMs for tool or item selection. However, their performance is concerned with three shortcomings: 1) The iterative interaction with SEs is cumbersome, requiring quite a few LLM-calls, which is especially not efficient for complex reasoning tasks. 2) The greedy step-by-step decision lacks a global view of the path, making it prone to error propagation. 3) The accumulated prompts are lengthy where LLMs may lose attention of history or candidates. To ease these problems, we propose to directly generate a reasoning path, and edit it with feedback when the instantiation gets stuck. End-to-End reasoning over structured environments by fine-tuning. Fine-tuned models memorize the environments through training over annotations. They either directly generate a path and then ground it on SEs (LUO et al., 2024; Huang et al.,
2023b; Shu et al., 2022; Hu et al., 2022), or retrieve relevant items to build a path (Zhang et al., 2022; Saxena et al., 2020; Ding et al., 2024). Such end-toend reasoning shows efficiency with no interaction with SEs. However, they have three limitations: 1) The grounding of reasoning path only depends on model outputs, without ensuring faithfulness on SEs. To remedy this issue, they rely on a wider beam at the expense of more retrieved instances. 2) They rely heavily on annotations, which are expensive for massive environments. 3) The performance drops substantially on data unseen during training (Gu et al., 2021; Huang et al., 2023a), which is common in real-world scenarios. To alleviate these problems, instead of fine-tuning, we propose to instantiate LLMs reasoning path. Then, if anything goes wrong, we call LLMs to edit the path.
Plan-and-Refine Reasoning with LLMs. For faithfulness of LLMs reasoning, previous works adopt LLMs to refine the output (Pan et al., 2023). Some require LLMs to self-correct, prompting them to identify and correct errors (Madaan et al., 2023; Pourreza and Rafiei, 2023). Such methods achieve limited improvement, since they rely only on the intrinsic knowledge of LLMs, without any access to the environment. Alternatively, other works require LLMs to refine the previous plan based on environmental feedback (e.g. code error messages) (Chen et al., 2023; Qiao et al., 2023). The feedback provides the execution results and possible errors, which are more purposeful and thus effective. However, how to collect feedback for large-scale structured environments remains an open question. In Readi, we collect immediate reasoning log through the instantiation of reasoning path, including the position of error, half-way done instances and associated relations.

heading: 3 Task Definition
text: Reasoning over structured environments (SEs), e.g., question answering over Knowledge Graphs (KGQA) and Tables (TableQA), typically requires matching a question with instances in SEs to constrain the answer. An intermediate reasoning path is a structural representation of the question, as a bridge between the question and SEs. Figure 2 exemplifies some reasoning paths and their instances on KG. Note that a KG is a set of triple patterns, i.e., {(e, r, e′)|e, e′ ∈ E , r ∈ R}, where E and R refers to the set of entity and relation, respectively.
Formally, given a question Q and n topic en-
Implementation – KB as a demo
h
Sasha
tities E, the reasoning path P is conditioned by several constraints in Q. It is worth noting that P can represent complex constraints, (i.e., P can be single-constrained or multi-constrained). A single-constrained P is from only one topic entity. For example, a sequence of relations from the entity to the answer (Example Q1) or a Chain-of-thought (Wei et al., 2022). A multiconstrained P is from multiple topic entities, consisting of multiple paths to constrain the answer (Example Q2). Correspondingly, we instantiate P on KG to obtain instances. For Example Q1, the reasoning path “father_of→university” from “Obama” is instantiated to “Obama
father_of−−−−−−→

heading: Malia
text: college−−−−→UCMC” and “Obama father_of−−−−−−→ Sasha college−−−−→UCMC”.
Following Sun et al. (2018), we model reasoning over structured environments as an retrieve-thenreason task, where we leverage LLMs to build the reasoning path given Q. Then we reason over the path instances to obtain the answer.

heading: 4 Methodology

heading: 4.1 Overview
text: For better illustration, we adopt KGQA, a challenging scenario of multi-hop reasoning over massive environments, to showcase concrete implementation (Refer to Appendix A for TableQA). A running example is in Figure 3. Given a question and topic entities, we leverage the planning ability of LLMs to generate the initial reasoning path P (Section 4.2). Then we instantiate P on KG (Section 4.3). If the entire P is successfully instantiated, we take the intersection of all instances, then generate the answer. If any path in P gets stuck, we collect error messages to guide further refinement (Section 4.4). Such process runs until P is fully instantiated or a maximum edit time is reached. Please refer to Appendix G for concrete prompts of each module.
Implementation – KB as a demo

heading: 4.2 Reasoning Path Generation
text: Inspired by Li et al. (2023), we leverage in-context learning (ICL) to generate the initial reasoning path, as shown in Figure 3(a). Given a question and n topic entities, we utilize Chain-of-Thought (CoT) to generate the initial path, consisting of n constraints starting from each topic entity. For the example in Figure 3, we have a two-constrained path (from “France” to constrain the bordering countries and from “Nijmegen” to airports serving it and then to countries containing these airports).

heading: 4.3 Reasoning Path Instantiation
text: We instantiate the reasoning path P on KG and merge the instances. The main difficulty lies in how to sequentially match the natural language (NL) relations in P with relations in KG. Our instantiation involves two steps: relation-binding and path-connecting, as shown in Figure 3(b).
For relation-binding, given a path consisting of some NL relations (e.g., for “[France] border”, we have one relation “border” expressed in NL), we bind them to relation schemas in KG. Since a relation r in NL may have similar relation schemas in KG with analogical format and semantic meanings (Li et al., 2023), we leverage BM25 and Contriever (Izacard et al., 2022) to retrieve similar relations r̂ as candidates for r. For example, we bind “border” with 3 candidate relations “adjoin, near_by, locations” in KG (the orange italicized relations in Figure 3(b). This way, we obtain candi-
date relations for all relations in P . For path-connecting, note that the starting entity of each path is given. For the example in Figure 3(b), we need to check if there exists any path instance in KG from “France” where relations in this instance sequentially match the bound candidate relations of all relations in P . Specifically, if any relations in candidates “adjoin, near_by, locations” connects to “France” (the orange bold “adjoin”), we obtain entities satisfying the constraint “[France] border”, so the NL relation “border” is instantiated to “adjoin” on KG. Then, we repeat the same process for the remaining relations in P to finally instantiate the whole path.
If any path in P is not successfully instantiated, e.g., none of the candidate relations r̂ connects to the current entity, this is the necessary time to invoke editing (Section 4.4). If the entire P is successfully instantiated or the maximum edit time is reached, we merge all KG instances by intersection to answer the question (Section 4.5). Please refer to Appendix B.1 for concrete implementation.

heading: 4.4 Reasoning Path Editing
text: The goal of editing is to help LLMs identify the error of previous reasoning path and provide some error messages, consisting of 2 steps: summarization and preparation, similar to an error traceback of coding, as shown in Figure 3(c).
For summarization, we categorize the reasons of error as follows: i. “irrelevant NL relation rerr”: none of the KG relation candidates of rerr con-
nects to current entity; ii. “empty reasoning path”; iii. “path ends with compound nodes”1. Thus, if the instantiation goes wrong, we detect error position err, current NL relation rerr (none for reason iii) and current ending entity êerr (before which the NL relations are successfully instantiated).
For preparation, we collect some useful messages: 1) the reason of error. 2) the currently halfway-done instances. 3) relations around êerr is accessible, which might be the candidates to amend the halfway-done instances. Finally, an LLM is called to edit the previous path based on Q and these error messages by ICL.
For the example in Figure 3, we have the NL relation “border” instantiated to “adjoin” and obtain a compound node at the end of instance, which triggers reason iii to invoke LLMs for editing. Therefore, we concatenate reason iii, halfway-done instance “France adjoin−→ Compound Node” and candidate relations around the “Compound Node” (e.g., “country” and “relationship”) as error messages. More details are in Appendix B.2.

heading: 4.5 QA Reasoning
text: Upon obtaining the merged KG instances Sq, i.e., the intersection of instances for each path (constraint), we build an LLM-based reasoning module to answer the question Q. We concatenate Q and Sq as input and prompt an LLM to generate the answer. Note that the form of Sq is a set of triple patterns (entity, relation, entity′). For entities, we convert the mid in KG to the corresponding friendly name. For relations and compound nodes, we keep their original forms in KG. Specifically, we ask LLMs to pick the knowledge triples used to answer Q in a CoT manner. For the example KG instances from “France” in Figure 3, we have 2 knowledge triples “(France, adjoin, compound node), (compound node, country, German)”, based on which, the LLM can reason that “the country bordering France” is “German”.

heading: 5 Experiments
text: To comprehensively evaluate the reasoning ability of Readi on large-scale structured environments, we experiment on two complex tasks, KGQA and TableQA. (Please refer to Appendix A for detailed implementation of TableQA).
1Compound value node, i.e., blank node, is typical in KG to express some complex entities, such as an event.

heading: 5.1 Datasets
text: We evaluate on three KGQA and two TableQA datasets. Detailed statistics are in Appendix C.1. WebQuestionsSP (WebQSP) (Yih et al., 2016) contains KGQA questions from Google query logs with up to 2-hop reasoning on Freebase, mostly requiring a single-constrained reasoning path. ComplexWebQuestions (CWQ) (Talmor and Berant, 2018) is a complex KGQA benchmark, challenging for up to 4-hop reasoning on Freebase, with 55% multi-constrained questions. MetaQA (Zhang et al., 2018) is a KGQA dataset from movie domain, with 3 levels of difficulty, denoted as MQA-1H, MQA-2H and MQA-3H. WikiTableQuestions (WTQ) (Pasupat and Liang, 2015) contains questions over 421 tables, challenging for complex aggregation operations, e.g., count, argmax, and sorting. WikiSQL (Zhong et al., 2017) is a large-scale complex dataset based on Wikipedia tables, requiring comparison, aggregation and arithmetic operations.

heading: 5.2 Baselines
text: KGQA Baselines. Training-based methods fine-tune pre-trained language models (PLMs): EmbedKGQA (Saxena et al., 2020), NSM (He et al., 2021), TransferNet (Shi et al., 2021), SR+NSM+E2E (Zhang et al., 2022), UniKGQA (Jiang et al., 2023c), ReasoningLM (Jiang et al., 2023b) and RoG (LUO et al., 2024). Inference-based methods call LLM-APIs: Davinci-003 (Ouyang et al., 2022), GPT3.5, GPT4 (OpenAI, 2023), AgentBench (Liu et al., 2024), StructGPT (Jiang et al., 2023a). All baselines assume golden topic entities are given. TableQA Baselines. Training-based methods findtune PLMs: TAPAS (Herzig et al., 2020), UnifiedSKG (Xie et al., 2022), TAPEX (Liu et al., 2022). Inference-based methods call LLM-APIs: Davinci003 (Ouyang et al., 2022), GPT3.5, GPT4 (OpenAI, 2023), StructGPT (Jiang et al., 2023a). Note that Inference-based methods model TableQA as an information retrieval task.
Following Tan et al. (2023), we adopt Hit@1, assessing whether the predicted entity is correct, to evaluate KGQA. We adopt denotation accuracy, assessing whether the prediction exactly matches the golden, to evaluate TableQA. We elaborate baselines in Appendix C.3. We also discuss indepth comparison with ToG (Sun et al., 2024) and DATER (Ye et al., 2023) in Appendix F.

heading: 5.3 Implementation Details
text: We adopt gpt-3.5-turbo (OpenAI, 2022) and GPT4 (OpenAI, 2023) as LLMs, denoted as ReadiGPT3.5 and Readi-GPT4. Temperature is 0.3 for all modules. For relation-binding, we deploy a Pyserini as a hybrid searcher with BM25 and Contriever (Izacard et al., 2022). For each relation generated by LLMs, we retrieve top 5 similar relations on Freebase. For instantiation, we deploy a Virtuoso server following the instructions2. More details can be found in Appendix C.

heading: 5.4 Main Results
text: Results for KGQA As shown in Table 1, overall, Readi significantly outperforms all Inferencebased methods, the vanilla LLMs and most trainingbased methods on all datasets, demonstrating the effectiveness of Readi. Compared with inferencebased methods, Readi substantially boosts the results of the vanilla LLM (by 8.6% on WebQSP and 14.9% on CWQ), demonstrating that Readi enables LLMs to practically interact with structured environments. Moreover, Readi-GPT3.5 already significantly surpasses state-of-the-art results with LLM-APIs (by 4.7% on WebQSP and 12.4% on MQA-3H), and Readi-GPT4 further enhances the performance. With fewer LLM-calls, our directlygenerated reasoning path and editing framework
2https://github.com/dki-lab/Freebase-Setup

heading: Training-based Method

heading: Inference-based Method
text: still achieves better performance. Compared with Training-based methods, without large-scale supervision and cost of beam search, Readi achieves comparable performance (e.g. 67.0 Hit@1 on CWQ) by some demonstration examples, showing the effectiveness of Readi. Additionally, Readi sets new state-of-the-art results MQA-1H, showing the effectiveness of editing which provides pertinent feedback upon instantiation errors.
Results for TableQA We experiment on TableQA scenario requiring multi-hop reasoning over tables to show the generalizability of Readi. As
shown in Table 2, overall, Readi outperforms all Inference-based and most Training-based baselines, setting state-of-the-art results on WTQ, demonstrating the effectiveness of our framework. Compared with Inference-based methods, Readi surpasses previous state-of-the-art methods by 9.5% and the vanilla LLM by 5.9% on WTQ, showing that Readi significantly improve the LLMs performance. Compared with Training-based methods, Readi, without massive annotations, is significantly superior on WTQ, while trailing behind on WikiSQL. This may be due to the i.i.d. distribution between the training and testing sets of WikiSQL, favoring the results of fully-trained methods. Interestingly, Readi-GPT3.5 are comparable with Readi-GPT4. This might because we asks LLMs to reason (with aggregation operations) directly based on retrieved table items. Further analysis shows that Readi-GPT3.5 has more chance of invoking editing than Readi-GPT4, which offers more pertinent feedback from the environments. Elaboration on effectiveness of editing is in Section 6.1, Section 6.2 and Appendix D.2.

heading: 6 Analysis
text: We further analyze Readi’s modules, reasoning path and efficiency on 1000 test samples of CWQ. For fairness, we base all evaluation on our instantiation and reasoning modules. Please refer to Appendix D for more detailed analysis of Readi.

heading: 6.1 Ablation Study
text: Effectiveness of reasoning path generation and editing modules. As shown in Table 3, we adpot the answer coverage rate (AC, rate of instances containing the answer) and QA performance (Hit@1) for illustration. We also analyze the robustness of editing module with a Corrupt path and an Empty path. First, Readi establishes a plug-and-play nature for both modules, showing their effectiveness. With only an initial path (we instantiate the path and maintain the longest path if it goes wrong),
Readi reaches comparable results (1st row). With an Empty initial path (Empty columns), the editing modules still yield competitive results close to the full Readi. Second, editing with LLMs significantly improves the performance (2nd and 3rd rows), further showing the effectiveness of the editing module. Third, generally, higher capacity of LLMs leads to better results for both modules, which meets our expectation. Lastly, Readi shows robustness for reasoning path editing, performing well even with an Empty or even a Corrupt path (Corrupt and Empty columns).

heading: 6.2 Reasoning Path Analysis
text: We compare reasoning path of Readi with representative fine-tuned methods, i.e., Subgraph Retrieval (SR) (Zhang et al., 2022) trains an encoder to retrieve relation and then builds a path, RoG (LUO et al., 2024) tunes a Llama 2 (Touvron et al., 2023) to generate a path. Quality of Readi’s reasoning path. We adopt answer coverage rate (AC) and number of retrieved knowledge (#RK) as the quality of graph. Ideally,
the higher AC and lower #RK, the better. Also, we analyze the QA performance (Hit@1), shown in Table 4. First, Readi’s initial path is comparable with fine-tuned methods, with GPT4 surpassing them by a large margin (5.0% and 6.3% Hit@1 than RoG and SR, respectively), showing the effectiveness of our reasoning path. Second, with some necessary editing, Readi obtains substantially higher AC, with a little higher #RK (GPT4) and even lower #RK (GPT3.5), and significantly higher QA performance than fine-tuned methods, showing the effectiveness of LLMs editing. Third, with wider beam size, fine-tuned methods gain higher AC, yet a drastically growing #RK, and lower Hit@1 than Readi, illustrating our superiority. Extensive features of Readi’s reasoning path. The quality of reasoning path is multi-dimensional. To further show insights of reasoning path by LLMs, we design some metrics, including the instantiation progress, error types, etc. We compare with SR, RoG and Golden (Outermost in Figure 4). More detailed analysis is in Appendix D.1.
Insights driven by Figure 4 are three-folded: 1) Readi’s initial path already achieves better in QA results. More than a half (60%) of Readi’s initial paths can be fully instantiated (LLMs called only once), showing that Readi effectively unleash the strong planning ability of LLMs. For each path, a large proportion (averagely 74%) can be instantiated, again indicating the possibility to safe LLM-calls. 2) With necessary editing, Readi gets significantly closer to golden (From green to red in Figure 4), exceeding compared methods in many metrics, especially the QA results. 3) Interestingly, the path by LLMs establish a human-like nature, for humans tend to get stuck at points they have never seen in real world. For Compound Node Ending Rate (special for KG), fine-tuned methods are close to Golden, showing that they memorize the
structured information. However, they trail behind behind Readi on QA results, demonstrating that unfaithfulness still exits for fine-tuned methods.

heading: 6.3 Efficiency Evaluation
text: How many LLM-calls do we need? Due to unexpectable nature of LLMs output, we cannot give an exact number of LLM-calls for Readi. Instead, we present the distribution of number of LLM-calls for editing in Figure 5 and some exemplar cases in Table 5. Note that there are at least 2-hop and up to 4-hop reasoning required for CWQ (theoretically 4-8 LLM-calls for iterative interaction). The instantiation success rate and average instantiate progress in Figure 4 already demonstrate that Readi can save a bunch of LLM-calls. In Figure 5, most of the time the initial reasoning path is already instantiable, no need for more LLM-calls. Averagely, the LLM is called 1.55 times to edit the path, saving more invocation than iterative interaction. Also, cases in Table 5 show that, with the necessary editing, LLMs can correct previous path based on some error messages during instantiation.

heading: 7 Conclusion
text: In this paper, we propose a novel framework Readi where LLMs can reason over structured environments efficiently and faithfully. In Readi,
LLMs initially generate a reasoning path which is then instantiated on environments to facilitate faithful reasoning. Path editing is triggered only if the instantiation gets stuck. We showcase the implementation of Readi on knowledge graph, evaluate the effectiveness on KGQA and TableQA and analyze extensive features of Readi. Our work also shed lights on practically interaction between natural language and structured environments, where LLMs plays a crucial role to bridge the gap.

heading: 8 Limitations
text: Although Readi achieves strong performance in complex reasoning task over structured environments, e.g., knowledge graph and tables, there are still some limitations of our method. First, we leverage only two LLMs as backbone to evaluate Readi. Therefore, more experiments can be done to test performance of Readi for other LLMs (including the inference-based and training-based). Second, we adopt reasoning path as a representation of natural language utterance and model the task as a information-retrieval one. An interesting direction is semantic parsing, e.g., text2SQL, which we will leave for future work. Moreover, we can also test Readi on question answering over database or other reasoning fields. Third, the instantiation in our current implementation of KGQA is intuitive but may induce many queries for SPARQL engine if encountered some “big” entities. Finally, our proposal, to fully harness intrinsic planning ability of LLMs and incorporate structural reasoning log to edit the path, performs well and saves some LLM-calls in experiments, which does not mean we should always depend on LLMs plans. And although the initial path by LLMs can be well instantiated, this just means the path by LLMs corresponds to some instances on KG, but not ensures that the path exactly matches with the constraints in the question.

heading: 9 Acknowledgements
text: The authors would like to thank Jue Zhang, Yuzhong Qu, Dapeng Li, Jinmao Li, Kaikai An, Yurong Wu and Wentao Ding for their valuable feedback. The authors would also like to thank Xiaokang Zhang for providing the reasoning path of Subgraph Retrieval.

heading: B.1 Reasoning Path Instantiation Details
text: In this part, we demonstrate more details of pathconnecting mentioned in Section 4.3. Assume that we have a reasoning path starting from an entity, with m relations R. All relations are bound to candidates R̂. For the example in Figure 3, we have a reasoning path “[Nijmegen] serve_airport → contain” and have “r1 : serve_airport” bound to “r̂1 : airport, terminal, serving port” in KG, and similarly have “r2 : contain” bound to “r̂2 : contained by, contains, place of ”. The purpose if path-connecting is to check if there exists an instance in KG where the starting point is entity “Nijmegen”, then connect to some relations in r̂1, and then connect to some relations in r̂2.
The path-connecting is essentially an Breadthfirst search (BFS) algorithm, with a queue que containing possible entities connected to current relations. Firstly, we push the starting point (e.g., “Nijmegen”) into que. Second, at each time,
we pop out current head of que and check if current candidate relations (e.g., “adjoin, terminal, serving port”) can connect to it on KG. If so, we successfully instantiate “serve_airport” on KG, and we can obtain some entities which are “serve_airport” of “Nijmegen” (e.g., “WZ air.” and “NTA.”). Then, we add these entities to que. Such BFS searching continues until all relations are instantiated or anything goes wrong.
Note that for a “big” entity, their may be a substantial number of tail entities for a relation. For example, there are hundreds of entities for “France
location−→ ”. This may end up with a longer instantiation time and shed lights on future improvements of Readi. In this case, we adopt a threshold to constrain the size of que.

heading: B.2 Reasoning Path Editing Details
text: Section 4.4 covers the basic process of reasoning path editing. Here are some implementation details for the preparation step, the purpose is to adopt pertinent structural information on KG to help LLMs identify the error position of previous reasoning path and edit it. For the half-way done instances in the Error Messages, we use “compound node” to indicate all cvt nodes in Freebase (Bollacker et al., 2008), and we just sample several (as a hyper-parameter) instances of each instantiated relation to showcase the path instances in KG of the previous path. The loss of information from sampling is minor because the messages just inform the LLMs that the relations in the reasoning path can be grounded to certain instances on KG. For candidate relations, we also adopt a threshold to constrain the size, if there are too many of them, we filter out using similarity search (same embedding as relation-binding) according to the original question.
For example, there are hundreds of compound node instances connected to “France adjoin−→ ”, and we just use “France adjoin−→ compound node” to rep-

heading: Dataset Training Dev Test

heading: KGQA Dataset

heading: TableQA Dataset
text: resent the instances. If their are hundreds of candidate relations connected to these compounds nodes, we use the question “What country bordering France contains an airport that serves Nijmegen” to filter out top 35 similar candidates.

heading: C Detailed Experimental Setups

heading: C.1 Dataset Statistics
text: We experiment Readi on test sets of all datasets. We adopt Hit@1 and denotation accuracy for KGQA and TableQA, respectively. We evaluate Readi on 3 KGQA and 2 TableQA datasets (Descriptions in Section 5.1). Statistics of datasets are shown in Table 7. Note that we model all datasets as an information retrieval task, instead of a semantic parsing one.

heading: C.2 LLM API version
text: For CWQ and WebQSP, we use gpt-3.5-turbo-16k0613 and gpt-4-32k for GPT3.5 and GPT4, respectively. For others, we adopt gpt-3.5-turbo-0613 and gpt-4 for GPT3.5 and GPT4, respectively.

heading: C.3 Baselines
text: KGQA baselines. Training-based methods:
• EmbedKGQA (Saxena et al., 2020) adopts
an encoder to retrieve relevant entities and generate the answer.
• NSM (He et al., 2021) adapts neural state machine for KGQA and rank entities in a retrieved subgraph.
• TransferNet (Shi et al., 2021) adopts a transparent framework to rank entities according to different parts of a question in a subgraph.
• SR+NSM+E2E (Zhang et al., 2022) trains an encoder to retrieve relevant relations and build a path from retrieved relations.
• UniKGQA (Jiang et al., 2023c) retrieves a sub-graph and rank the schemas in a unified way.
• ReasoningLM (Jiang et al., 2023b) designs an entity encoding and training framework to rank entities in a sub-graph.
• RoG (LUO et al., 2024) trains a LLama 2 (Touvron et al., 2023) to firstly generate a path, second ground this path to the knowledge graph, and then generate answer based on the grounded graph.

heading: Inference-based methods:
text: • Davinci-003 (Ouyang et al., 2022), GPT3.5 and GPT4 (OpenAI, 2023) are based on LLMAPIs. We adopt few-shot in-context learning to ask the model to output the answer of question in order to test the models inherent knowledge of datasets.
• AgentBench (Liu et al., 2024) is an agentbased method asking LLMs to call tools based on tool-discription, history and observations.
• StructGPT (Jiang et al., 2023a) requires LLMs to iterative pick up relations and entities based on current returned candidates.
TableQA baselines. Training-based methods:
• TAPAS (Herzig et al., 2020) predicts denotation by selecting table cells and optionally applys an aggregation operator to the selection.
• UnifiedSKG (Xie et al., 2022) sequentializes the table and tunes a T5-3B model to answer the question.
• TAPEX (Liu et al., 2022) guides language models to mimic a SQL executor.
Inference-based methods:
• Davinci-003 (Ouyang et al., 2022), GPT3.5, GPT4 (OpenAI, 2023) are based on LLMAPIs. We adopt few-shot in-context learning to ask the model to output the answer, based on the question and the entire table.
• StructGPT (Jiang et al., 2023a) iteratively filter out columns and rows of table and ask the LLMs to answer the question.
Note that all Inference-based baselines are information retrieval ones.

heading: D Detailed Analysis

heading: D.1 Elaboration on Reasoning Path Analysis
text: Note that quality of reasoning path is multidimensional. To further show some insights of reasoning path by LLMs, we meticulously design some metrics. We introduce average Length of predicted path (LPP), average length of instantiated path (LIP), average instantiation progress (AIP, average of LIP/LPP for each question), instantiation success rate (ISR) and Compound nodes ending rate (CER), where all length mentioned above refers to number of relations on path. We compare with fine-tuned SR (Zhang et al., 2022) and RoG (LUO et al., 2024) (beam=1). The Golden Path is obtained by extracting relations from golden logical forms in the dataset, which is only used for analysis.
The absolute performance of different metrics of Readi are shown in Table 8. Also, Figure 6 demonstrates some instantiate progress of Readi-GPT3.5. Figure 7 shows some relative results of different metrics for Readi-GPT3.5, compared with finetuned methods and the Golden. Additionally, we show the distribution of number of LLM-calls for editing for Readi-GPT3.5 in Figure 8.
It is shown that the initial reasoning path by Readi is already comparable with fine-tuned ones. With some necessary editing, Readi significantly gets closer to golden. In addition, without awareness of the environments, a large number of Readi’s initial path (49%) get stuck at compound nodes, while fine-tuned methods well memorize the schemas in structured environments. However, with pertinent feedback upon instantiation errors, Readi substantially alleviates this problems and reaches higher QA performance than fine-tuned methods, again demonstrating that unfaithfulness still exits even with large-scale fine-tuning.

heading: D.2 Performance of varied MAX_EDIT_TIME
text: We have reported the distribution of Editing times in Figure 5 (GPT4) and Figure 8 (GPT3.5) to show the efficiency of Readi. To some extent, the distribution also showcases the quality of the Reasoning Path and Effectiveness of Editing module. Note that we set the MAX_EDIT_TIME to eight to test the quality, which can be set smaller to save more resources. To further provide some insights, we experiment on different MAX_EDIT_TIME setting on CWQ subset in Table 9. Results show that with more chances to edit, the reasoning path gets better, further showing the effectiveness of Editing. And four times of editing already achieves an comparable outcome.

heading: D.3 Performance of Retriever
text: The performance of retrieval may affect the instantiation. We calculate the recall of retriever in Table 10, showing that the performance of retrival is not the bottleneck of Readi. Moreover, our implementation is adapted from KB-BINDER (Li et al., 2023), which has already shown effectiveness of binding relations to KG.

heading: D.4 Token Cost
text: As indicated in our title, we discusses how LLMs can efficiently and faithfully interact with structured environments. By efficiency, we focus on the way of interaction, i.e. less LLM invocations (Sec-
tion 1), to obtain pertinent information on largescale structured environments for multi-hop reasoning. As in some industrial circumstances, the times of LLM-calls are strictly constrained. However, some might concern the token cost of Readi as a measure of efficiency, though it is not our priority. Here, we report the average token cost per question in Table 12. We utilize Python library tiktoken, for GPT3.5 with six-shot for generation, 5-shot for editing and MAX_EDIT_TIME set to 4 for CWQ and 2 for WebQSP.
Note that it is non-trivial to directly compare the exact token cost with other methods, for the sake of number and format of examples, uncontrollable LLM-API traffic, and even the difficulty of the task. For example, StructGPT (Jiang et al., 2023a) claims to incorporate 32-shot examples, but the exact few-shot prompts are not provided. Readi’s token cost can also be modulated by adjusting the MAX_EDIT_TIME. Furthermore, excessive calls or overly challenging tasks may cause the LLMAPI to fail in providing the appropriate content or format, exceeding the MAX_TRY_TIMES, which also affects the token cost. That’s also why we analyze the times of LLM-call as efficiency of our interaction framework in the main body of paper.

heading: D.5 Qualitative Error Analysis
text: We analyze 200 randomly-sampled error cases of CWQ and WebQSP in Table 11. We divide the error into four categories (Section 4.4 and 6.2): i. "Instantiation Error": Even after Editing, the path is still not fully instantiated. ii. "Answer not Covered": Answer not in retrieved knowledge. iii. "QA Error": Even the answer is covered, the QA
output is still wrong (e.g., hallucination). iv. "False Negative": For example, the ground truth of “the two continents Turkey is in” is “Eurasia”, and the model output “Europe and Asia”.
Based on Table 11, first, our reasoning path can be well instantiated and utilized to answer the multihop reasoning question. Second, there is still room for improvement of the retrieved knowledge, which we would like to focus in our future work. Third, the hallucination in QA reasoning still exits, with GPT4 performing better than GPT3.5.

heading: E Generalizability
text: Note that our focus is not on comparative analysis of various LLMs. We propose Readi to enhance LLMs reasoning over structured environments. Therefore, we utilize LLMs with strong understanding and reasoning capabilities (e.g., GPT3.5) to show their performance with Readi. Here we discuss generalizability of Readi framework. Generalizability to domain-specific KG. The concept of reasoning path is a structured representation of a multi-hop reasoning process (Section 3). By the intrinsic planning ability in Section 1, we adopt LLMs strong question understanding and reasoning ability to directly generate the reasoning path. Here is an intuition of Readi: we humans can navigate multi-hop reasoning challenges across various domains by recognizing named entities and relations of the task. LLMs, trained on large-scale natural language corpus, may develop these abilities. Therefore, Readi is designed to operate without the need of domain-specific knowledge.
We have experimented on both large-scaled KG (CWQ and WebQSP) and domain-specific KG (MetaQA), where Readi shows impressive performance (Section 5). One concern is generalization to specific domain with massive relations. Here, we illustrate how Readi handles such cases with an example.
Consider a question “What’s the nationality of Lebron James”. Assume that the LLM has
few domain knowledge and initially generates “[Lebron James] nationality” as the reasoning path, whereas the ground truth path in KG is “[Lebron James] born_in - city_of”.
When instantiation (Section 4.3), we bind “nationality” to relations in the KG but find none of them connected to [Lebron James], so we invoke editing. In the error message, we include relations around [Lebron James], which is [born_in, sex, father_of, . . . ], as a hint for Editing (Section 4.4 and Appendix B.2).
With the semantic understanding that “born_in” can relate to "nationality of Lebron James", the LLM can correct previous path. The ablation of Editing (Section 6.1) and analysis of Reasoning Path after Editing (Section 6.2) further demonstrate these. Note that relations around [Lebron James] is limited, compared with those in the whole KG.
The whole idea mirrors human cognitive strategies when conducting web searching for multi-hop reasoning tasks, where we flexibly adjust our subconscious plan based on information in browsers. Genalizability to other LLMs. Here, we show Readi’s geralizability to other LLMs. Due to API availability, we test Readi with Llama-270b-chat on CWQ and WebQSP samples. The MAX_EDIT_TIME is set to 4 for CWQ and 2 for WebQSP. For fair comparison, we adopt our GPT3.5-based QA reasoning module. Table 13 shows that Readi can generalize to other opensourced LLMs.
We also discuss Readi’s generalizability to finetuned models. RoG (Section 6.2) tunes a Llama 2 to generate a reasoning path, showing the feasibility of generating a path by fine-tuning. We’ve shown that Readi performs well with Editing by LLM to improve the path. One concern is about Editing with a fine-tuned model, depending on annotations, which we would go deeper in future works.

heading: F Comparison with ToG and DATER
text: We further compare Readi with ToG (Sun et al., 2024) and DATER (Ye et al., 2023).

heading: F.1 With ToG
text: Think-on-Graph (ToG) is an LLM-based KGQA method, which also reason the answer with an LLM in an information retrieval manner. Here, we discuss the differences between our Readi and ToG. As discussed in Related Works (Section 2), ToG adopts an LLM to step-by-step filter out some Knowledge Graph (KG) instances, similar to beam search. Note that there is no reasoning path, i.e., a structural representation of the question by LLMs, in ToG’s methodology. Interaction Paradigm. Our Readi’s novelty lies in the following aspects: 1) Required Capability. In ToG, the LLM traverses on the KG, similar to beam search, to filter out entities or relations by scoring all candidates at each step, which mainly requires discrimination ability. In contrast, our Readi requires the strong understanding and reasoning ability to maintain a structural representation of the entire question. 2) Way to Introduce the Environment. To obtain structural information in large-scale KG, ToG introduces all relations around an entity, or all tail entities around a relation, as candidates for LLMs to score the distribution. Such process can be cumbersome because there are sometimes massive candidates on KG, e.g., locations in France. On the other hand, in Readi, we collect pertinent reasoning log only upon the instantiation errors. Readi fully unleash the understanding ability of LLMs and ease the burden of multi-turn scoring for long candidate lists. 3) Grounding. To obtain instances, in ToG, LLMs select items at each step, restricted by the beam size, hindering it from reasoning questions requiring logical operations (based on a set of instances for aggregation, comparison, etc). Conversely, for Readi, we introduce a novel instantiation module (Section 4.3) to obtain all instances based on constraints in the reasoning path. Generalizability. Readi can generalize to more circumstances (Appendix E): 1) Supported En-
vironments, Think-on-Graph (ToG) only focuses on KG. However, Readi is a more general interaction framework for reasoning over structured environments. We showcase the concrete implementation of both KGQA (Section 4) and TableQA (Appendix A). 2) Supported Reasoning Tasks. ToG, whose KG instances are restricted by the beam size, falls short for questions requiring logical operations, e.g., aggregation, comparison, etc. However, Readi, adopting LLMs to maintain a structural representation of the question, is not restricted by any beam size, which can cover such reasoning cases by LLMs reasoning with retrieved instances. Experimental Results. We did not cover the results of ToG in main results, for the sake of fair comparison. The evaluation metric from the official published code of ToG differs from the standard Hit@1. Conversely, Readi’s evaluation strictly follows Tan et al. (2023), which is also used by all compared LLM-based baselines. While ToG does not provide any output files and the reported results are not reproducible, here we report Readi’s results based on ToG’s metrics in Table 14, where our Readi, with less LLM-calls, still outperforms ToG, overall.

heading: F.2 With DATER
text: We have considered DATER but did not directly compare with it for the following reasons. The base LLMs. DATER adopt Codex (Chen et al., 2021) as the backbone LLM, which is different from ours. Codex performs significantly better than GPT3.5 in TableQA (9.1% higher acc on WTQ (Liu et al., 2023)). However, Codex is close-soured and does not offer API-calls any more3. Therefore, it is not fair to directly compare Readi (based on GPT3.5) with DATER. The way of reasoning over Tables. Our experiment is to show if our interaction framework, Readi, can adopt LLMs to effectively obtain useful information from structured environments and then reason the answer. Therefore, we compare with methods modeling TableQA as an Information Retrieval task (refer to Section 3 and Section 5.2). However, DATER uses a text2SQL model to obtain facts in table. This involves external tools (the SQL Interpreter) to handle the logical and numerical operations (e.g., aggregation, compare and calculation), which is not fair to compare with.
3https://platform.openai.com/docs/deprecations

heading: G Prompt List
text: Note that we do not modify prompts for baseline methods. Prompts for the vanilla LLMs is in Table 15, following (Sun et al., 2024). For TableQA, the prompts for vanilla LLMs is the same in Table 17. Detailed prompts for each module of Readi is in Table 16 (KGQA) and Table 17 (TableQA).
For number of few-shot demonstrations, on CWQ (Talmor and Berant, 2018) and WebQSP (Yih et al., 2016) we adopt 6 shots for reasoning path generation and 5 shots for other modules. For MQA(Zhang et al., 2018), the shot number for reasoning path generation follows (Li et al., 2023) and we adopt only 3 shots for editing and reasoning (we donnot design a reasoning module for MQA). For WTQ (Pasupat and Liang, 2015) and WikiSQL (Zhong et al., 2017), we use 7, 2, 7 shots for generation, editing and reasoning, respectively.

heading: Future Work
text: One concern is about Editing with a fine-tuned model, depending on annotations, which we would go deeper in future works