Tree-of-Traversals: A Zero-Shot Reasoning Algorithm for Augmenting Black-box Language Models with Knowledge Graphs (2024)

Abstract: Knowledge graphs (KGs) complement Large Language Models (LLMs) by providing reliable, structured, domain-specific, and up-to-date external knowledge. However, KGs and LLMs are often developed separately and must be integrated after training. We introduce Tree-of-Traversals, a novel zero-shot reasoning algorithm that enables augmentation of black-box LLMs with one or more KGs. The algorithm equips a LLM with actions for interfacing a KG and enables the LLM to perform tree search over possible thoughts and actions to find high confidence reasoning paths. We evaluate on two popular benchmark datasets. Our results show that Tree-ofTraversals significantly improves performance on question answering and KG question answering tasks. Code is available at https: //github.com/amazon-science/

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12302–12319 August 11-16, 2024 ©2024 Association for Computational Linguistics
Knowledge graphs (KGs) complement Large Language Models (LLMs) by providing reliable, structured, domain-specific, and up-to-date external knowledge. However, KGs and LLMs are often developed separately and must be integrated after training. We introduce Tree-of-Traversals, a novel zero-shot reasoning algorithm that enables augmentation of black-box LLMs with one or more KGs. The algorithm equips a LLM with actions for interfacing a KG and enables the LLM to perform tree search over possible thoughts and actions to find high confidence reasoning paths. We evaluate on two popular benchmark datasets. Our results show that Tree-ofTraversals significantly improves performance on question answering and KG question answering tasks. Code is available at https: //github.com/amazon-science/ tree-of-traversals

heading: 1 Introduction
text: Large language models (LLMs) are used for a range of knowledge-intensive tasks such as information retrieval (Zhu et al., 2023b), summarization (Zhang et al., 2023), and question answering (Tan et al., 2023). Trained on large amounts of textual data, these models learn a wide breadth of information. However, LLMs suffer from several limitations – they produce hallucinated information (Ji et al., 2022; Bang et al., 2023), lack deep domain-specific knowledge (Pan et al., 2023b), and have a static knowledge cutoff when training ends.
Knowledge graphs (KGs) naturally complement LLM weaknesses. KGs contain up-to-date information covering general (Vrandeić and Krötzsch, 2014; Lehmann et al., 2015) and/or domainspecific topics (Abu-Salih, 2020; Liu et al., 2019; Zhu et al., 2017; Choi and Lee, 2019; Farazi et al.,
∗Work done while interning at Amazon Corresponding author: esmarkow@usc.edu
2020) in highly structured and interpretable format. Augmenting an LLM’s ability to reason and respond in natural language with an external KG’s up-to-date knowledge presents a path toward a reliable and factual LLM.
The rise of powerful LLMs with new capabilities has renewed interest in combining LLMs with KGS. Numerous survey and position papers have recently emphasized their combined potential (Pan et al., 2023a; Zhu et al., 2023a; Yang et al., 2023; Pan et al., 2023b). Existing works augmented LLMs with KGs in multiple ways, such as integration into pretraining (Yasunaga et al., 2022), fine-tuning (Zhang et al., 2022), or later adaptation with subsequently trained components (Lin et al., 2019; Hu et al., 2022). All of these carry some limitations. In particular, training or fine-tuning a large scale LLMs is computationally expensive. In some cases, model weights are unavailable publicly. Finally, the largest KGs require their own servers and cannot be integrated in-memory with LLMs. Additionally, previous works do not consider the case of augmenting with multiple KGs.
An algorithm that allows the augmentation of a powerful black-box LLM with any number of internal or external KGs without training from scratch or fine-tuning the model is valuable. Such a zeroshot algorithm would enable several innovative use cases such as (1) customers using a black-box LLM API in conjunction with an internal domain-specific KG, (2) integration of personalized KGs into an LLM without the risks associated with training a model with such personal data, (3) integration of deep domain knowledge via an array of API accessible KGs (e.g., IMDb1, MusicBrainz2).
We introduce Tree-of-Traversals, a novel algorithm that addresses the above issues by allowing the augmentation of any powerful LLM with arbitrary number of KGs in a zero-shot fashion. It re-
1IMDb API 2MusicBrainz API
12302
quires no training, functions with black-box access to the LLM, and works with any API accessible KG. Our contributions are:
1. Tree of traversals: A novel zero-shot algorithm for augmenting any powerful LLM with arbitrary number of KGs and enabling advanced KG reasoning using tree search.
2. Evaluation of Tree-of-Traversals on two question answering tasks: 2WikiMultiHop and QALD-10 and comparison with baselines.
3. Development of a new dataset to test combined reasoning over a general and a domainspecific KG, and evaluation of Tree-ofTraversals on this dataset.
We conduct detailed experiments on three models of varied sizes hosted on Amazon Bedrock and present detailed ablation studies.

heading: 2 Tree-of-Traversals
text: The Tree-of-Traversals algorithm maintains a local KG subgraph that is repeatedly expanded until it contains all the information required by an LLM to answer the given query. At the start, a local KG subgraph is initialized to contain the entities present in the original query. It is then expanded
using a tree search algorithm to choose actions and thoughts generated by an LLM in order to obtain relevant knowledge from the KG using a KG interface. The algorithm halts when the LLM is able to answer the original query using the local KG subgraph as context. Tree-of-Traversals consists of three major components. (1) A knowledge graph interface implemented to interact with one or more required KGs. (2) An action state machine (ASM) which is a finite state machine that defines the feasible space of actions, states, and prompt templates when the LLM interacts with a KG to expand the local KG subgraph. (3) A tree search algorithm which defines the overall LLM search trajectory such as best first search, backtrack upon making a mistake, and termination condition when an answer is found.

heading: 2.1 Knowledge Graph Interface
text: The knowledge graph interface allows Tree-ofTraversals to interact with one or multiple KGs. Let K = (E ,R, T ) be a single KG. E is the set of entities in which each entity consists of an identifier, a label, and an optional description (e.g., Q35332, ‘Christopher Nolan’, ‘British-American filmmaker’). R is the set of relation types, each consisting of an identifier, a label, and an optional inverse label (P57, ‘director’, ‘is director of ’). T
is the set of edges or facts in the KG in which each edge is of the form (s, r, o) where s, o ∈ E and r ∈ R, e.g., (‘Inception’, ‘director’, ‘Christopher Nolan’). Tree-of-Traversals can be used for K as long as the following interfaces are implemented.
1. initialize(q) → E0: It takes as input a query q, extracts entities from q and returns the linked entities E0 ⊂ E where E0 are the entities from K that are referenced in q.
2. get_relations(Eselected)→ Roptions: It takes a set of entities, Eselected, and returns the relation types, Roptions ⊂ R, that Eselected have in K: {r|(s, r, o) ∈ T , s ∈ Eselected}
3. get_edges(Eselected, r) → Tadded, Eadded: It takes a set of entities and a selected relation type, and returns all edges with relation type r for source entities in Eselected: {(s, r, o) ∈ T |s ∈ Eselected, r = r, o ∈ E}. It also returns the new entities, Eadded, that are entities reached with Tadded.
This interface is implemented with SPARQL queries when available; otherwise, we use the graph API that is available for the KG. For multiple KGs, each interface is implemented separately.

heading: 2.2 Action State Machine (ASM)
text: One of the challenges with developing a zero-shot LLM algorithm that works with arbitrary KGs is that the LLM does not know what relations are available in the graph or what relations are valid for a given entity. Few-shot or in-context learning approaches can only cover a handful of the possible relation types (e.g., Wikidata has over 11,000 relation types) (Brown et al., 2020).
To overcome these issues we break the task of expanding a local KG subgraph into multiple subtasks. We use a finite state machine with the following actions: Think,
Answer, ExpandKG, Select_Entities, and Select_Relation, and states: default, selecting-entities, selecting-relation, and done as shown in Figure 2. This is named as Action State Machine (ASM) throughout the paper.
From the default state, Tree-of-Traversals can either Think, Answer, or choose to ExpandKG. After Tree-of-Traversals chooses to ExpandKG, it first is prompted to Select_Entities about which it needs more information (e.g., ‘Inception’ in Figure 1). It is then prompted to Select_Relation from a list of candidate relations provided by the KG interface’s get_relations method. After selecting a relation (e.g., ‘cast member’ in Figure 1), all edges containing one of the selected entities as the source and the selected relation as the relation are added to the local KG subgraph. Tree-of-Traversals is then able to Answer, Think, or ExpandKG again.
Prompt Templates. Each state in the ASM other than the ‘done’ state is associated with a unique prompt template. Prompt templates are filled with information from the local KG subgraph and the KG interface before presenting them to the LLM. A customized prompt for each state allows us to present precise and relevant information along with specific instructions for each state to the LLM simplifying the LLM’s task. For instance, the prompt for ‘selecting-entities’ presents the available options of entities to choose from that are in the local KG subgraph (see Figure 3). Prompt templates for all the states are in the Appendix F.1-F.3.
Local KG subgraph is represented using a a token efficient YAML format that minimizes repetition for multiple edges on the same entity.
Invoking KG Interfaces. Outside of initialization, there are two times in which the ASM needs to invoke the KG interface: (1) When constructing the selecting-relation prompt, the algorithm calls get_relations(Eselected) to gather available choices of relations. (2) When executing the transition from selecting-relation to default after having selected a relation type r, the algorithm calls get_edges(Eselected, r) so that it can add the new edges and entities to the local subgraph.

heading: 2.3 Tree Search Algorithm
text: Our approach draws inspiration from the Tree-ofThoughts approach (Yao et al., 2023) in which an LLM is given increased reasoning power by allowing generation of multiple thoughts at a time and

heading: Original Query:
text: building a search tree over the resulting reasoning chains. We extend this approach to augment LLMs with KGs through allowing the generation of actions in addition to thoughts, and building a search tree over them. Challenges arise because Tree-ofThoughts was not designed to incorporate actions and was not designed for knowledge intensive question answering tasks. As a result, we introduce some modifications: incorporation of actions via the ASM, a slightly different search procedure and stopping condition to better handle QA, and a different sampling procedure for improved diversity when doing constrained sampling.
Algorithm 1 presents the Tree-of-Traversals tree search algorithm. Given a query q it begins with initialization using initialize(q) as described in Section 2.1. After initialization, it searches by (1) choosing the unexplored node to expand based on the node’s value assigned by the value function (Best First→ Depth First as tie-breaker), (2) sampling k actions from the LLM (k is branching factor) using the prompt associated with the selected node’s state in the ASM, (3) for each of the sampled action, applying the transition function, and (4) evaluating the value of the resulting nodes with the LLM value function. The search stops when an answer is found with a node value exceeding the threshold τ . To bound the search space we add two hyper-parameters: max depth of the tree search, after which the algorithm is forced to transition to the done state (i.e., answer the query), and max expansions after which the model halts exploration and returns "Cannot find answer".
Algorithm 1 Tree-of-Traversals Input: q ← query, k ← branching factor, τ ← answer threshold, P ← LLM, F ← ASM Output: Answer to q
1: procedure TOACTIONS(q) 2: T ← ϕ ▷ Empty tree 3: Y ← ϕ ▷ Empty answer set 4: s0 ← initialize(q) 5: T .add(s0) 6: while !finished do 7: s← choose_node(T ) 8: ▷ Sample k actions from P using the action
state and associated prompt of s from F(s) 9: a← sample_actions(s, k,F , P ) 10: for a ∈ a do 11: ▷ Apply action a to state s according to F 12: s′ ← transition(s, a,F) 13: ▷ Evaluate the resulting state s′ using P 14: s′.value← evaluate(s′, P ) 15: T.add(s′) 16: if s′.action_state = done then 17: Y.add(s′) 18: if s′.value > τ then 19: ▷ When answer exceeds threshold, stop search 20: finished← True 21: return argmaxy∈Y y.value
Value Function Guidance. Tree-of-Traversals computes the value of a node to determine its utility. This value is created by the LLM using evaluation prompts (step evaluate in Algorithm 1). The value can be between 0 to 1 where 1 indicates highest utility. We use two types of evaluation prompts: one for intermediate states and one for answer states. The prompts include the original query, the local KG subgraph, the trajectory of previous actions, followed by instructions for evaluating the node (see Appendix F.4 and F.5). These values are then used to guide the exploration of the action space. Specifically, choose_node returns the unexplored node with the highest value (Best First). If there are nodes with the same value, Depth First Search is used.
Chain-of-Traversals. In some cases we can find the answer to a query with a single sequence of thoughts and actions using the ASM and KG interface. This is equivalent to Tree-of-Traversals with a branching factor of k = 1. We refer to this as Chain-of-Traversals. While useful for compari-
son, experiments show the benefit of considering multiple branches.

heading: 2.4 Tree-of-Traversals with Multiple KGs
text: Augmenting an LLM with more than one KG mainly involves building KG interfaces for each of the added KGs. There are a few other changes to the algorithm. (1) The entities extracted in initialize(q) are matched with each KG interface. (2) When presenting the options of relations during selecting-relation, get_relations(Eselected) is called on each KG interface. (3) When adding a new entity to the local KG subgraph we call an entity linking function for the other KG interfaces. We allow for the entity linking function to be a separate function in the KG interface or to fallback on initialize(o) where o is the text label of the entity that has just been added. This makes allows the use of explicit links between common KGs if available while still functioning without.

heading: 3 Experiments
text: We evaluate Tree-of-Traversals using three different models available on Amazon Bedrock3: ClaudeInstant (claude-instant-v1), Llama2 70b (llama2-70b-chat-v1), and Llama2 13b (llama2-13b-chat-v1). AWS Bedrock provides on-demand access through a single API to various Foundation Models, including both open source and black box ones. This is precisely a use case Tree-of-Traversals is designed for.

heading: 3.1 Tasks and Datasets.
text: We first evaluate the Tree-of-Traversals algorithm on two common tasks used for evaluating an LLM’s knowledge: 2WikiMultiHop and QALD-10. To allow testing on complex questions requiring knowledge from multiple KGs we create a new dataset that requires knowledge from multiple KGs.
2WikiMultiHop Dataset (Ho et al., 2020) was constructed by extracting multi-hop templates from HotPotQA (Yang et al., 2018), combining templates to get complex reasoning questions, generating candidate questions with Wikidata, and then confirming that the entity mentions for each edge appear in the Wikipedia paragraphs. Answers to these questions can be derived from both Wikipedia and Wikidata (Vrandeić and Krötzsch, 2014). We
3https://aws.amazon.com/bedrock
subsample 500 questions from the test set following the approach used in ReAct (Yao et al., 2022), including the sampling seed value of 233.
QALD-10 Dataset (Usbeck et al.) is a multilingual Knowledge Graph Question Answering (KGQA) dataset with 395 questions created by humans, translated into other languages, and then constructed as a SPARQL query over Wikidata4. These questions are more varied in terms of reasoning structure than the questions in 2WikiMultiHop (e.g. requiring multiple answers or aggregations). We used the questions in English language.
MusicBrainz-x-Wikidata Dataset One novel use-case of Tree-of-Traversals is synthesizing and reasoning over multiple knowledge graph sources. There is no existing dataset that requires synthesizing information from multiple KGs to answer individual questions. Therefore, to test the reasoning ability using multiple KGs, we create a new dataset, MusicBrainz-x-Wikidata, containing 109 questions that require reasoning with information from both MusicBrainz and Wikidata. Unlike Wikidata which contains general knowledge, MusicBrainz is a deep domain-specific database on the music industry. The vast majority of this information is unlikely to be known by large language models. We construct the MusicBrainz-x-Wikidata dataset with human annotators who were provided with instructions to find reasoning paths between these two KGs, the question types to focus on, and some example questions. The curated questions were checked for ambiguity and sensibility. By design, each question
4https://github.com/KGQA/QALD-10
requires extracting information from both knowledge graphs in order to answer it successfully. In addition to the reasoning types in 2WikiMultiHop, this dataset contains questions that involve aggregations, comparisons of aggregations, qualifications, and complex combinations of these. An example can be seen in Figure 3.1. Detail on instructions, question types, and examples are in Appendix A.

heading: 3.2 Metric.
text: We use Exact Match Included (EM-in) as the evaluation metric. EM-in is 1 if the ground truth answer appears with an exact match anywhere in the answer and 0 otherwise. This accounts for the propensity of an LLM to output answers in a sentence with varying syntax. This is a common metric but is often referred to interchangeably with Exact Match (EM) (Sun et al., 2023). When there are multiple answers, we compute average EM-in over all labels.

heading: 3.3 Comparison Baselines.
text: We experiment against three related approaches that can work with any black-box LLMs: (1) Chainof-Thought (CoT) prompting (Wei et al., 2022) (2) ReAct (Yao et al., 2022) which iterates between generating thoughts and generating actions for searching and retrieving from a text knowledge base like Wikipedia, and (3) Forward Looking Ac-
tive Retrieval (FLARe) (Jiang et al., 2023b) which iterates between generating thoughts and then retrieving from a knowledge base to correct inaccuracies.

heading: 3.4 Implementation details.
text: For all models, we use a sampling temperature of 0.0 for the LLM when multiple samples are not required and a temperature of 1.0 when diverse samples are required. We test our approach in two settings: (1) with a branching factor of k = 1 termed as Chain-of-Traversals, and (2) with a branching factor of k = 3 termed as Tree-of-Traversals. In both setups, we set maximum depth to 7 which means that upon reaching the default action state beyond depth 7, the only available action is to answer the question. For Tree-of-Traversals, we set the maximum total expansions to 20. The answer threshold τ is set to 0.8 which corresponds to high confidence answers supported by the KG according the evaluate prompt (Appendix F.5).
For 2WikiMultiHop and QALD-10, we use Wikidata as the knowledge graph (Vrandeić and Krötzsch, 2014). For MusicBrainz-x-Wikidata, we use MusicBrainz in addition to Wikidata. We implement the KG interface for Wikidata using Wikidata SPARQL queries5, and implement the KG in-
5https://query.wikidata.org/
terface for MusicBrainz KG using the MusicBrainz API6.

heading: 4 Results and Discussion
text: Table 1 presents the results of our experiments on 2WikiMultiHop and QALD-10. For all the models, Tree-of-Traversals outperforms the baselines on 2WikiMultiHop, setting state-of-the-art results for these tasks in the zero-shot setting. We hypothesize that much of this gain is due to Tree-of-Traversals’ access to the knowledge base via proposed KG interface and its thought-action procedure guided by the ASM. This is evident, as even Chain-ofTraversals, which does not perform a tree traversal (including multiple thoughts/actions, backtracking, and node value computation), significantly outperforms ReAct’s knowledge grounding: 8.7% higher accuracy than ReAct→ CoT on 2WikiMultiHop when averaged over all models. Compared to Chain-of-Traversals, Tree-of-Traversals further improves performance. It gives on average a 12.8% absolute accuracy increase for 2WikiMultiHop and 4.3% absolute improvement for QALD-10. We note that the better performing the model is, the more it stands to gain from Tree-of-Traversals as noted by the difference between Llama-70b and Llama-13b.
On MusicBrainz-x-Wikidata (Table 2), which contains challenging reasoning questions requiring access to two KGs, there is an average of 37.4% relative improvement from Tree-of-Traversals over Chain-of-Traversals as shown in Table 2. Both Chain-of-traversals and Tree-of-Traversals outper-
6https://musicbrainz.org/doc/ MusicBrainz_API
form Chain-of-Thoughts on this dataset. We only compare with Chain-of-Thoughts as other retrieval methods have low coverage over the MusicBrainz knowledge base.

heading: 4.1 Effect of the Value Function.
text: Tree-of-Traversals relies on signal from the value function to pick the final tree trajectory. If the values were arbitrary, then Tree-of-Traversals would not do any better than Chain-of-Traversals. Figure 5 shows that there is a meaningful signal from the value function for all models. There is an average performance difference of 31.0% between the accuracy for answers valued at 1.0 vs 0.0. This represents an average relative improvement of 83.2% when selecting answers valued at 1.0 over those valued at 0.0. See Appendix C for individual profiles of each model’s value function.

heading: 4.2 Effect of Backtracking.
text: To determine the effect of backtracking in Treeof-Traversals, we ask the counter-factual of how would the model perform if it could not backtrack (Figure 6). We limit the analysis to 2WikiMultiHop questions in which Tree-of-Traversals generates answers on a subtree, and the model eventually backtracks on that subtree (i.e., the cases where we have a counter-factual). As a result, these questions are generally more challenging than the overall distribution of questions. In these cases, we compare the result if the highest-valued answer was taken from the first searched subtree compared to the ultimate answer after backtracking. We find that the ability to backtrack gives Tree-of-Traversals a significant accuracy increase ranging from 4.1% to 12.3%.

heading: 4.3 Performance on MusicBrainz-x-Wikidata.
text: For MusicBrainz-x-Wikidata dataset, we only compare against Chain-of-Thought since the implementations for other baseline algorithms do not have access to similar music-specific knowledge bases. Despite this, we observed some interesting results. Chain-of-Thought does far worse on MusicBrainzx-Wikidata than on the more general datasets based exclusively on Wikipedia/Wikidata (10.1%-13.8% vs. 25.2%-47.4%). This could be because LLMs are trained on vast amount of general knowledge such as that found in Wikipedia, but they are not trained with as much data from specific domains such as music. Besides the presence of KG interface and ASM with Tree-of-Traversals, this could be an additional reason why Tree-of-Traversals does 2.2 times as well as Chain-of-Thought on MusicBrainz-x-Wikidata. This demonstrates the importance of augmenting LLMs with domainspecific KGs and/or multiple KGs which the proposed Tree-of-Traversals is capable of.

heading: 4.4 Analysis of Baselines.
text: ReAct underperforms Chain-of-Thought in most cases. This phenomenon was seen in the original ReAct paper which noted that their approach significantly reduced hallucinations despite lower accuracy (Yao et al., 2022). Therefore, falling back on Chain-of-Thought results in an accuracy improvements for for Llama2-70b and Claude-Instant. We note that claude-instant vastly underperforms Llama2-70b on ReAct. The primary reason for this is claude-instant refuses to "search" people and apologizes after performing invalid actions. Despite this, ReAct→CoT still improves over CoT on both 2WikiMultiHop and QALD-10. FLARe improves model performance on 2WikiMultiHop but not on QALD-10. This may be due to better overlap between the text knowledge base for 2WikiMultiHop.

heading: 5 Related Works
text: Knowledge Base Question Answering There is a large history of work that has looked into answering questions using a knowledge graph (Wu et al., 2019; Lan et al., 2021). The approaches can broadly be broken into those that attempt to parse the question into a logical form (Berant and Liang, 2014; Luo et al., 2018; Zhu et al., 2022), and information retrieval approaches (Bordes et al., 2015; Chen et al., 2019)
Knowledge Enhancement. Recently, researchers have looked into enhancing pretrained language models and LLMs using knowledge graphs. Many works have focused on incorporating knowledge graph data into LLMs through training, often with architectural changes to the model (Zhang et al., 2019; Wang et al., 2019; Peters et al., 2019; Yamada et al., 2020; He et al., 2021). Some of these have found success using specialized graph encoder layers (Yasunaga et al., 2021; Sun et al., 2021). Some have sought to mix this process with pretraining of the language model (Yasunaga et al., 2022). The limitations of including KGs during LLM training are: the models are incapable of incorporating KG updates without retraining, are not able to change KG source (e.g., a domain-specific one), and may have low explainability resulting from learning knowledge in model weights rather than explicit retrieval. In addition, these methods add complexity to the training process, increase cost, and do not work with LLM without access to model weights. As a result, scaling these methods has often been seen as too risky.
Other methods treat a KG as a more complete source of truth and use the LLM to generate structured queries for the KG. Tianle et al. (2023) and Choudhary and Reddy (2023) teach the LLM to generate a logical KG query which then returns matching entities from the KG. These methods share similarities with the semantic parsing based
approaches mentioned earlier. However, by returning a logical query for the KG, these methods lose the reasoning and commonsense abilities of the LLM.
Some approaches have looked at injecting KG or knowledge base data into LLM prompts. Many methods do a single round of retrieval based on the query (Lewis et al., 2020; Li et al., 2023) using a retrieval mechanism, such as dense passage retrieval (Karpukhin et al., 2020). These methods cannot answer more complex multi-hop questions as the initial retrieval is unlikely to contain the secondary or tertiary information that will ultimately be required. Later methods have made the retrieval process iterative. E.g. FLARe (Jiang et al., 2023b) uses prediction of the upcoming sentence to retrieve relevant documents and regenerates until the sentence contains high-confidence tokens. In Wang et al. (2023) multi-round QA format is used for multi-round retrieval. ReAct (Yao et al., 2022) does multiple rounds of thoughts and actions to query a text-based knowledge base. Jiang et al. (2023a) repeatedly interfaces with a KG. In other parallel works, (Sun et al., 2023; Wen et al., 2023) explore methods for building KG context for LLMs using path and neighborhood based search methods. The above methods do not incorporate tree search or forms of advanced reasoning beyond the LLM’s innate ability. They cannot explore multiple reasoning paths or solutions and cannot backtrack if making a mistake. This results in lesser capabilities. Additionally, none of the above methods explore reasoning over multiple KGs.
Multiple Knowledge Base QA. Some works studied QA on questions derived from differing domains. These works learn to pick between different domain-specific QA models, each trained on a single knowledge base (Puerto et al., 2021; Geigle et al., 2021; Puerto et al., 2023). Besides requiring training, such systems cannot answer questions requiring the synthesis of information from different domains (MusicBrainz-x-Wikidata).

heading: 6 Conclusion
text: Tree-of-Traversals is a powerful algorithm that enables LLMs to utilize KG reasoning with zero examples, no schema dependency, and no training. We demonstrate its efficacy experimentally on multiple LLMs and datasets. We hope Tree-ofTraversals continues to be developed and see value in studying integration with personalized user KGs
as well as with other domain-specific datasets.

heading: 7 Limitations
text: Tree-of-Traversals is slower than simpler retrieval alternatives. Improving the value function would reduce the number of LLM and KG API calls by avoiding incorrect paths along the tree to explore. Other engineering solutions, such as hosting graph servers, could be implemented to accelerate LLM and KG access. In terms of the token cost, the KG text representation is token efficient compared to many retrieval methods using unstructured knowledge bases as the latter tend to maximize usage of the LLM context window. However, compared to non-retrieval baselines, there is significant increase in token usage cost.
The types of KG questions that can be answered are limited by the context window, search depth, and LLM’s reasoning ability. For instance, a very large aggregation (‘How many mountains have elevation above 3500 meters?’) would require more entities than what fits in the LLM context. Future work could look at adding other actions to the ASM, such as aggregations, that could distill the local KG into more relevant information.
EM-in is also an imperfect metric. False negatives can arise from discrepancies in date and number formatting, text formatting, and aliasing. False positives can arise in cases in which the model does not definitively answer but includes the correct answer in its response.

heading: 8 Ethical Impact
text: We do not anticipate Tree-of-Traversals to introduce new areas of risk but it may have unstudied effects on existing risks of LLMs or KGs. We highlight the following areas. (i) Tree-ofTraversals gives new capabilities to LLMs after training. While positive in terms of accuracy, we have not evaluated its effect on wider safety metrics. (ii) Our evaluation is limited to only English versions of KGs and datasets. Tree-of-Traversals should be evaluated in other languages to ensure consistent and fair experience. (iii) We have not performed analysis under misleading or deceptive knowledge graphs. Using a publicly modifiable knowledge graph does come with the risk that information could be deceptively changed.
In terms of positive ethical impact, making this research public democratizes access to knowledgeaugmented LLMs as this method does not require
a large training investment or owning the language model to build and customize.

heading: A MusicBrainz x Wiki
text: Additional information on the creation of the MusicBrainz x Wiki dataset.

heading: A.1 Creation steps
text: The dataset was created by annotators in a semiautomatically supported manner. We developed an automated tool to support the following process:
• Find linking entities that are present in both MusicBrainz and Wikidata of various types (artist, label, place, event, etc.)
• Find related entities that could be used in a question to perfectly identify the entities above (e.g., The place owned by the Detroit Tigers→ Tiger Stadium)
• Find related entities that could be used in a question to ambiguously identify a linking entity, so that a qualifier could be added to disambiguate the linking entity.
• Count the number of edges in Wikidata or MusicBrainz for each relationship type an entity has.
We use these tools to create an initial set of multihop questions falling into specific reasoning categories. We then have human annotators check each question for sensibility (grammatically correct and can be definitively understood) and ambiguity (one right answer in the KGs). If possible, the question is reworded or fixed to not contain ambiguities. We are then left with a final set of 109 questions.

heading: A.2 Composition of Questions
text: The composition of questions for Musicbrainz x Wiki can be found in table 3.

heading: A.3 Annotators
text: The annotators consisted of approximately 10 research scientists and engineers, all English speaking and based in the United States.
B Implementation Details

heading: B.1 Diversity Oversampling.
text: One problem with Tree-of-Thoughts is that when there are constrained options to choose from, the LLM becomes repetitive and fails to produce diverse outputs (Yao et al., 2023). Their solution used a "propose prompt" with additional instructions for proposing multiple distinct thoughts. However, this
adds significant complexity to the prompt for purposes that are only tangential to the task being completed. A simpler approach we employ is to generate diverse actions by oversampling. Specifically, if the branching factor is k, we sample 2 ∗ k possible actions from the LLM and then extract the first k unique ones to use. We only employ this for selecting-entities and selecting-relation as the action choices are limited and diversity is required. This change does not noticeably affect the computation cost of Tree-of-Traversals as multiple actions are sampled for the same prompt and the average prompt input size (100s-1000s of tokens) is generally much larger than the average generation length for these actions (<20 tokens). Latency remains similar as multiple actions are sampled in parallel. With this, we consistently generate diverse options when selecting-entities or selecting-relation.

heading: B.2 Hyperparameters
text: Tree-of-Traversals does not have a huge number of hyperparamters. Most of the hyperparameters were chosen analytically, though some were chosen based on preliminary experiments. We share some information on choosing them. τ was chosen analytically in conjunction with the prompt for evaluate on the done state (Appendix F.5). Chosen to be 0.8 so that Tree-ofTraversals only stops when it is confident that the answer is supported by the KG.
The temperature was chosen to be 1.0 to promote diversity. If implementing with other models which can have temperature >1.0, we recommend hyperparameter tuning or using a sample prompt of each type and analytically choosing the temperature such that diverse but sensible actions are output.
The branching factor k = 3 was chosen based on small scale experimentation. Larger k makes
the model spend longer searching, taking longer time and increasing expenses. Smaller k did not generate as much diversity and options. It is worth noting that even with k = 3 and diversity sampling, sometimes the actual number of unique outputs is less than 3.
The max depth was chosen analytically to be minimal while enabling answering of most questions. Some questions in QALD-10 and MusicBrainz-x-Wikidata would require a greater depth to answer.
The maximum expansions cutoff was chosen purely to ensure the model does not hang on a single question too long. The actual value is somewhat arbitrary and has minimal impact on accuracy.

heading: B.3 KG Interface
text: We present some additional details of the KG Interface.
The initialize function is a three step process. First a LLM call and prompt is used to extract named entities. Second, for each extracted entity candidates are searched for using the KG API. Finally, another LLM call is used to match the extracted entity to the best candidate. get_relations and get_edges are implemented exclusively using the respective KG API. For Wikidata, these are each one or two SPARQL query (forward and reverse edges). For MusicBrainz, they each require multiple API calls as the endpoints are separated for different entity types. E.g. there is a separate endpoint for Artists and Recordings.
C Value Function Profiles
We profile each value model’s value function to assess their characteristics. Figure 7 shows the individual answer value vs accuracy profiles for each model on 2WikiMultiHop. All models have a positive correlation between the answer value and the answer’s accuracy. However, as is to be expected, there are differences with Llama2-13b having the lowest correlation and Claude-Instant having the highest correlation.
Figure 7 only look at the value function response for final answer states as those are the only ones with a directly associated accuracy score. While accurately evaluating answer states is more important, we also want to see some useful signal from the values for intermediary states. Figure 8 shows the average value for nodes on the search path to
the answer, separated by whether the answer scores as correct or not. We would expect actions that ultimately lead to a correct answer to have higher values than those that lead to incorrect ones. We clearly see this with the correct distribution being further right than the incorrect distribution. This is least pronounced for Llama2-13b which is to be expected. We also note that the distributions shape is different between the Llama2 models and Claude-Instant.
These results indicate that larger, stronger models, will likely see more improvement in the value function, and thus, more utility from Tree-ofTraversals .

heading: D Prompt Engineering
text: Some prompt re-engineering was required when switching between models for both our methods and ReAct. This engineering was limited primarily to formatting of responses. For example, Llama2 would start responses with "So the next action should be" while we required the action to start with "THINK", "EXPAND_KG" or "ANSWER". Instead we incorporated “So the next action should be:” into the prompt so that the output began with the chosen action. For selecting-entities we added “You have selected the following entities to expand:”. For selecting-property we added “I suggest selecting property:”. These issues could also be solved by changing the parsing of the presented prompts.

heading: E Fallback to Chain-of-Thought (→CoT)
text: We fallback to Chain-of-Thought for the ReAct baseline when either no answer is provided after depth 7 or when one of the following phrases appears in the answer: ’determine’, ’unable’, ’cannot’, ’unknown’, ’unsure’, or ’not possible’. While these certainly could appear in intentional answers, we find that they occur consistently and exclusively in non-answer style responses.

heading: F Prompts
text: The following pages contains the prompts used in Tree-of-Traversals. The dynamic components that change based on the query, KG state, action history are shown in color.
F.1 Prompt for default action state

heading: Original Query:
text: Who is Bob Dylan’s maternal grandmother?

heading: Knowledge Graph Entities:

heading: Q392: Bob Dylan - American singer-songwriter
text: Q62519478: Beatrice Stone Knowledge Graph Edges: Bob Dylan:
mother: Beatrice Stone
Previous Actions: EXPAND KG: I should search for the mother of Bob Dylan SELECT ENTITIES: Q392 SELECT RELATION: P25 - has mother
You are a superintelligent AI equipped with the ability to search a knowledge graph for definitive, up-todate answers. Your task is to interface with the knowledge graph in order to answer the above query. You will be able to expand the knowledge graph until you have found the answer. Think in detail before acting or answering.
Available actions: ’THINK’ - Generate relevant thoughts to solving the problem. This could include recalling well known facts
from memory. e.g.,
THINK: I should search for the movies directed by... THINK: I know that Biden is American, therefore... THINK: I see that John Cena is in both The Independent and Vacation Friends, therefore...
’EXPAND_KG’ - Search for edges of entities in the knowledge graph using an external API. This is a useful function for getting to the correct answer. e.g., EXPAND_KG: I should search for the country of origin of Jonathon Taylor ’ANSWER’ - Generate the final answer once the problem is solved. Just state the best answer, do not output a full sentence.
e.g., ANSWER: No ANSWER: Harry Potter ANSWER: [Harry Potter, Ron Weasley, Hermione Granger]
F.2 Prompt for selecting-entities action state

heading: Original Query:
text: Who is Bob Dylan’s maternal grandmother?

heading: Knowledge Graph Entities:

heading: Q392: Bob Dylan - American singer-songwriter
text: Q62519478: Beatrice Stone Knowledge Graph Edges: Bob Dylan:
mother: Beatrice Stone
Previous Actions: EXPAND KG: I should search for the mother of Bob Dylan SELECT ENTITIES: Q392 SELECT RELATION: P25 - has mother EXPAND KG: I should search for the mother of Beatrice Stone
Current task EXPAND_KG takes two parameters. The first is the entity or entity group to get more information about.
Select which entity or entities from the KG to expand. Provide the QIDs. Options include [Q392, Q62519478] SELECT ENTITIES:
F.3 Prompt for selecting-relation action state

heading: Original Query:
text: Who is Bob Dylan’s maternal grandmother?

heading: Knowledge Graph Entities:

heading: Q392: Bob Dylan - American singer-songwriter
text: Q62519478: Beatrice Stone Knowledge Graph Edges: Bob Dylan:
mother: Beatrice Stone
Previous Actions: EXPAND KG: I should search for the mother of Bob Dylan SELECT ENTITIES: Q392 SELECT RELATION: P25 - has mother EXPAND KG: I should search for the mother of Beatrice Stone SELECT ENTITIES: Q62519478
Your current task is to select the property (PID) to expand along for the selected entities. The selected entities are: [Q62519478] The options of properties to choose from are:

heading: P31 - has instance of

heading: P21 - has sex of gender

heading: P27 - has country of citizenship

heading: P735 - has given name ...
text: Select exactly one property (PID e.g., P10) from those listed above SELECT PROPERTY:
F.4 General Prompt for evaluate

heading: Original Query:
text: Who is Bob Dylan’s maternal grandmother?

heading: Knowledge Graph Entities:

heading: Q392: Bob Dylan - American singer-songwriter
text: Q62519478: Beatrice Stone Knowledge Graph Edges: Bob Dylan:
mother: Beatrice Stone
Previous Actions: EXPAND KG: I should search for the mother of Bob Dylan SELECT ENTITIES: Q392 SELECT RELATION: P25 - has mother EXPAND KG: I should search for the mother of Beatrice Stone SELECT ENTITIES: Q62519478
Your current task is to evaluate the above knowledge graph and action history. Based on the original query, the current knowledge graph, and the action history, give the likelihood that
the model will correctly answer the question.

heading: If the most recent action provided information towards the goal and followed the preceding thought, give a
text: high score. If the last action was unhelpful, give a low score.
The output should be a number between 0 and 1 with one decimal. Do not output anything else.
RATING [0.0-1.0]:
F.5 Prompt for evaluate on answer (done)

heading: Original Query:
text: Who is Bob Dylan’s maternal grandmother?

heading: Knowledge Graph Entities:

heading: Q392: Bob Dylan - American singer-songwriter
text: Q62519478: Beatrice Stone Q62519478: Florence Sara Stone Knowledge Graph Edges: Bob Dylan:
mother: Beatrice Stone
Beatrice Stone: mother:
Florence Sara Stone
Provided answer: The answer is Florence Sara Stone.
Your task is to score the correctness of the provided answer based on the original query, and the knowledge graph. Give a pessimistic score from 0.0 to 1.0 on how likely the answer is to be correct. 0.0 if definitely wrong 0.0 if unable to answer based on the knowledge graph 0.5 if unsure 0.7 for probably correct but not confirmed in knowledge graph 1.0 for definitely correct and confirmed in knowledge graph.
Give reasoning to get to the correct answer. Then provide a score. e.g., Reasoning... So the score for the provided answer should be...

heading: Future Work
text: Future work could look at adding other actions to the ASM, such as aggregations, that could distill the local KG into more relevant information. EM-in is also an imperfect metric. False negatives can arise from discrepancies in date and number formatting, text formatting, and aliasing. False positives can arise in cases in which the model does not definitively answer but includes the correct answer in its response