DARA: Decomposition-Alignment-Reasoning Autonomous Language Agent for Question Answering over Knowledge Graphs (2024)

Abstract: Answering Questions over Knowledge Graphs (KGQA) is key to well-functioning autonomous language agents in various real-life applications. To improve the neural-symbolic reasoning capabilities of language agents powered by Large Language Models (LLMs) in KGQA, we propose the DecompositionAlignment-Reasoning Agent (DARA) framework. DARA effectively parses questions into formal queries through a dual mechanism: highlevel iterative task decomposition and low-level task grounding. Importantly, DARA can be efficiently trained with a small number of highquality reasoning trajectories. Our experimental results demonstrate that DARA fine-tuned on LLMs (e.g. Llama-2-7B, Mistral) outperforms both in-context learning-based agents with GPT-4 and alternative fine-tuned agents, across different benchmarks in zero-shot evaluation. This makes such models more accessible for real-life applications. We also show that DARA attains performance comparable to state-of-the-art enumerating-and-rankingbased methods for KGQA1.

heading: 1 Introduction
text: Language agents (Wang et al., 2023b; Sumers et al., 2023; Xi et al., 2023) building on Large Language Models (LLMs) have emerged as a new paradigm of artificial intelligence (AI) systems to perform complex tasks in various environments. These agents primarily rely on LLMs as central controllers to perceive environments, autonomously plan, and reason to fulfill user tasks. However, operating within environments rich in structured data, such as Knowledge Graphs (KGs) (Bollacker et al., 2008; Vrandecic and Krötzsch, 2014), presents unique challenges for the agents. For instance, performing Question Answering over Knowledge Graphs (KGQA) demands various capabilities from
1Code, model are released at https://github.com/ UKPLab/acl2024-DARA
language agents. The agents need to decompose the user’s question into subtasks (planning), interact with the KG to obtain schemas (tool usage), select appropriate schema items aligned with subtasks (schema alignment), and construct executable logical forms deriving the answer (logical reasoning) (Figure 1). Previous research, e.g., tool learning with foundation models (Qin et al., 2023) and AgentBench (Liu et al., 2024), has equipped off-the-shelf LLMs with multiple functions to conduct KGQA in the In-Context Learning (ICL) setup (Brown et al., 2020). Although ICLbased agents are easy to use and finetuning-free, the performance of open-sourced ICL agents falls short significantly compared to classical enumeratingand-ranking-based methods (Shu et al., 2022; Gu et al., 2023), in which all possible reasoning paths starting from anchor entities in the question are enumerated and ranked. Despite this, LLMs-based agents continue to advance to be more powerful as LLMs are scaled up and rapidly improve every year.
While ICL-based language agents with GPT4 (Achiam et al., 2023) show strong capabilities in this task, the use of commercial LLMs raises practical concerns related to privacy, costs, and model flexibility. For example, running AgentBench with
3406
GPT-4 over only ~4,500 test examples incurs an expenditure of 1,300 U.S. dollars in stark contrast to ~$30 of the proposed model trained on opensourced LLMs (see section 5.5 for details), which raises critical concerns for a variety of applications.
To enhance the agent capabilities of LLMs and circumvent the challenges of ICL-based LLM agents, AgentTuning (Zeng et al., 2023) fine-tunes AgentLMs based on Llama-2 (Touvron et al., 2023) using the AgentBench framework for six different environments including Knowledge Graphs. However, the performance of fine-tuned AgentLMs still lags behind GPT-4 agents on structured data processing tasks including Knowledge Graphs, Databases, and Operating Systems.
In this paper, we propose the DecompositionAlignment-Reasoning Agent (DARA) framework for the KGQA task. DARA is a hierarchical framework, which comprises two connected modules: a high-level task decomposition and low-level task grounding module. The former is responsible for decomposing the given question into smaller tasks, while the latter derives executable logical forms for the decomposed task. Within the task grounding module, two integral components contribute to the overall functionality. The schema item selection component identifies appropriate schema items (relations, classes) relevant to the task, while the logical form construction component assembles a logical form based on the selected schema as well as the preceding logical form. To enhance the selection of relations, we propose a relation selection method, termed skim-then-deep-reading, where DARA scans relations of current entities and selects n promising relations to deeply read their descriptions. Task decomposition and task grounding work in tandem until the full logical form is constructed.
To generate reasoning trajectories for fine-tuning, we initially employ GPT-4 to translate linearized logical forms into natural language. Unlike previous work (Gilardi et al., 2023; Xu et al., 2024) showed the success of GPT-4 in annotation, we found the conversion quality falls short of expectations and can be subsequently enhanced through human verification. To assess the efficacy of DARA, we conduct comprehensive experiments on three popular benchmark datasets, i.e., WebQSP (Yih et al., 2016), GraphQ (Su et al., 2016), and GrailQA (Gu et al., 2021). Our findings confirm that DARA substantially outperforms both the ICL-based and the alternative fine-tuned LLM
agents. To conclude, our contributions are:
• We propose a novel language agent framework for KGQA, Decomposition-AlignmentReasoning Agent (DARA). It surpasses the framework proposed in AgentBench, by explicitly disentangling high-level task decomposition and low-level task grounding (schema items selection and logical form construction).
• Experiments show that fine-tuned DARA achieves state-of-the-art performance compared with both ICL-based and other fine-tuned agents (AgentLMs and fine-tuned AgentBench) across the three important benchmarks in zero-shot evaluation. Moreover, training with 768 reasoning trajectories, we show that DARA can achieve highly competitive performances comparable to enumerating-and-ranking-based models trained on larger data.
• Our experiments reveal the ongoing challenge of generating high-quality reasoning trajectories for language agents in KGQA with GPT-4. This is in contrast to previous studies that demonstrate the success of ChatGPT or GPT-4 in annotation (Gilardi et al., 2023; Xu et al., 2024). This observation suggests a potential avenue for future research: how to automatically generate high-quality data for language agent use cases where the most advanced LLMs (e.g. GPT-4) face their limitations.

heading: 2 Related work
text: KG-enhanced LLM Reasoning is a popular paradigm to reduce hallucination and unfaithful reasoning chains of LLMs. In this approach, retrieved triplets from KGs and parametric knowledge within LLMs (i.e., knowledge stored in their parameters) (Petroni et al., 2019; Roberts et al., 2020) work in tandem to derive the final answer to a given question. The Knowledge-Driven Chainof-Thought framework (Wang et al., 2023a) refines LLM reasoning using an external QA model based on KGs. Think-on-Graph (Sun et al., 2024a) and StructGPT (Jiang et al., 2023b) utilize off-the-shelf LLMs to traverse over graphs to find the most relevant knowledge and integrate them with parametric knowledge of LLMs to produce the final answer. Despite their potential, these methods face challenges when the parametric knowledge in LLMs is incorrect or outdated. In addition, in scenarios of
conflict between parametric knowledge and external non-parametric knowledge, Qian et al. (2023) reveals that LLMs are prone to the distraction of external knowledge, when the latter is irrelevant, leading to worse performance.
LLM-based Autonomous Agents for KGQA. Unlike KG-enhanced LLM reasoning which focuses on refining reasoning chains using KGs, LLM agents are able to conduct more complex tasks such as constructing logical forms for a given question by utilizing a set of more powerful human-like capabilities (Sumers et al., 2023) including question decomposition and logical reasoning. Tool learning with foundation models (Qin et al., 2023), AgentBench (Liu et al., 2024), and AgentTuning (Zeng et al., 2023) are three representative works. Qin et al. (2023) asked LLM agents to directly write SPARQL (Standard Protocol and RDF Query Language) program. In contrast, AgentBench adopts intermediate representation, namely s-expression (Gu et al., 2021), to represent SPARQL and construct the s-expression program step-by-step. AgentTuning fine-tunes LLMs with reasoning trajectories. However, all of them achieve poor performance due to the limitations of ICL-based agents and the framework design.
Enumerating-and-ranking-based Methods are a prevalent paradigm for KGQA (Yih et al., 2015; Lan and Jiang, 2020; Luo et al., 2018; Abujabal et al., 2017). To narrow down the large search space of KGs, it first enumerates all possible candidate logical forms from neighborhood of topic entities or retrieve similar logic forms from training examples. Subsequently, a ranker is applied to select the best one. However, this brute-force approach can lead to exponential candidates and thus suffer from scalability and coverage issues. To alleviate these issues, generation-augmented methods (Ye et al., 2022; Shu et al., 2022) generate diverse logical forms based on retrieved candidates. Besides, dynamic bottom-up semantic parsing approaches (Gu et al., 2023; Gu and Su, 2022) construct the final logical form incrementally and prune the search space on the fly. Although these methods can achieve high performance, they are time-consuming and necessitate expert-crafted rules for logical form construction. In contrast, LLM-based agents leverage LLMs for planning, grounding, and reasoning, offering enhanced explainability, efficiency, and flexibility. Moreover, LLM agents automate the entire process from task
decomposition to relation selection, to logical form construction, eliminating the need for extensive enumeration based on expert-crafted rules.

heading: 3 The Approach

heading: 3.1 Overview
text: An overview of DARA is described in Algorithm 1. Formally, given a knowledge graph G, a question Q, and a set of actions A, the objective is to construct a logical form L, e.g., s-expression (Gu et al., 2021), that yields the final answer to the question Q. To achieve this objective, DARA iteratively performs task decomposition (Section 3.2.1) and task grounding (Section 3.2.2) until L is finalized. During the iteration i, for decomposed task Ti (line 7), DARA grounds it against G (line 16). It takes actions in A to fetch schema items from G and selects the suitable one to construct the step-level logical form Lij (Section 3.2.2). This process iterates multiple times until Lij can fulfill the task Ti (line 15-24). Subsequently, Lij will be assigned to Li with the task level id (e.g. s-exp-1) (line 19). DARA incrementally construct the full logical form L via assembling task-level logical forms. The base conditions for exiting loops are autonomously determined by DARA. A concrete example illustrating this process is presented in Figure 2.

heading: 3.2 The Reasoning Framework

heading: 3.2.1 Iterative Task Decomposition
text: Iterative task decomposition serves as the highlevel planner in our framework. Decomposing questions into executable tasks is a fundamental ability in human cognition (Pelletier, 2004). By breaking down complex questions into smaller, more manageable tasks, agents can better grasp the underlying intent and devise an executable plan to find the answers. As we will demonstrate in section 5.2, generating all subtasks in the single-pass manner at the beginning (Wang et al., 2023c; Sun et al., 2024b) often leads to redundant or irrelevant ones. DARA incorporates and advocates iterative task decomposition, which generates one task at a time building on the results of the previous task. During the process, the agent will decide if more tasks are needed based on whether the current sexpression can get the answer to the given question. This adaptive approach allows DARA to dynamically refine its focus and tailor subsequent subtasks, contributing to a more effective reasoning process.

heading: 3.2.2 Task Grounding
text: Skim-then-Deep-Reading Relation Selection To ground the decomposed tasks against KGs, previous efforts (Liu et al., 2020; Yasunaga et al., 2022) pre-train models on KG-augmented data. However, such methods have limitations in scalability and coverage. They can struggle to fully capture the extensive and diverse topics, entities, and relations present in large-scale KGs (e.g., Freebase has 44 million topics and 2.4 billion facts.)
To alleviate this issue, DARA introduces a twostage relation selection method based on the recently emerging powerful natural language understanding ability of LLMs. DARA first invokes get_relations or get_relevant_relations to obtain relations and selects n most likely candidates among them (Step 1.1: Skim Relations in Figure 2). Subsequently, descriptions of these selected candidates are obtained using the get_descriptions function (Step 1.1: Deep Reading Descriptions in Figure 2). By reading these descriptions, the most suitable one is selected (Step 1.1: Schema Selection in Figure 2). Note that this approach eliminates the need for LLM agents to extensively learn the environment in advance, making it more adaptable and transferable across different scenarios.
Logical Form Construction As depicted in Figure 2, DARA automatically constructs the logical form of the current step based on the selected schema item and the preceding logical form (Steplevel S-expression in Figure 2). Unlike ICL-based agents which have to acquire it via explanation
in the prompt, the syntax of constructing logical forms is learned during fine-tuning. For the syntax and complex usage of logical forms (s-expression), please refer to Appendix A.3.

heading: 3.3 The Action Space
text: The action space A of DARA consists of a set of functions to interact with the KG. For Freebase, the space of A is as follows:
• get_relations(entities, topk): This function returns top-k 1-hop both incoming and outgoing relations for given entities.
• get_relevant_relations(task, topk): When no entities are mentioned in the question, this function retrieves top-k relevant relations according to the decomposed task.
• get_classes(entities, topk): This function returns top-k classes of given entities.
• get_relevant_classes(task, topk): This function retrieves the top-k relevant classes determined by the decomposed task.
• get_descriptions(schema): This function retrieves descriptions of input schema.
These actions are taken during task grounding to get necessary schema items from KGs. The input argument entities in above functions is either the entities appearing in the question or the intermediate results (s-expression) during the reasoning. The functions get_relevant_relations and get_relevant_classes utilize a bi-encoder retriever, specifically all-mpnet-base-v2 (Reimers
Algorithm 1: DARA Input: A knowledge graph G, a questionQ, a set of
actions A Output: Grounded logical form L
1 // Reasoning Trajectory 2 Traces← Q ; 3 // Initial task-level logical form 4 L0 ← null ; 5 i← 1 ; 6 while True do 7 Ti ← task_decomposition(Traces); 8 // No further task needed 9 if Ti is null then
10 L ← Li−1 ; 11 break ; 12 end if 13 Traces← Traces+ Ti; 14 j ← 1 ; 15 while True do 16 Lij ← task_grounding(Traces,A,G) ; 17 Traces← Traces+ Lij ; 18 if Lij completes Ti then 19 Li ← replace_id(Lij) ; 20 Traces← Traces+ Lj 21 break ; 22 end if 23 j ← j + 1 ; 24 end while 25 i← i+ 1 ; 26 end while 27 return L
and Gurevych, 2019), fine-tuned on <question, schema items> in training data to fetch relevant relations or classes for questions lacking entities. In the case of functions get_relations and get_classes, we utilize the same retriever to filter the top-k candidates after getting schema items from the KGs. The rationale behind this choice is twofold: 1) it alleviates the selection burden on DARA; 2) it preserves context length for multiple rounds of reasoning. The input s-expression of above functions are mapped to SPARQLs for querying the KG using APIs.

heading: 4 Experiment Setup

heading: 4.1 Constructing Fine-tuning Data
text: We create reasoning trajectories for DARA using GraphQ, WebQSP, and GrailQA. Pairs of <question, s-expression> in those datasets are filtered based on criteria, including relation diversity, removal of duplicates, and inclusion of complex questions with at least two subtasks. Subsequently, we linearize the s-expression and instruct GPT-4 to translate it into natural language. In total, we obtain 768 instances with reasoning trajectories. The construction details and the example of reasoning
trajectories are provided in Appendix A.

heading: 4.2 Zero-shot Evaluation
text: One of the preliminary challenges of interacting with KGs is the substantial volume of unseen relations, classes, and entities during training. To simulate the real-world unseen scenario and for a fair comparison, we conduct zero-shot evaluation as introduced by GrailQA (Gu et al., 2021), referring to schemas in the test data not appearing in training data. For GrailQA, the evaluation is performed on the development data since the golden entity linking results in the hidden test data are unavailable. The resulting statistics of the filtered test data are presented in Table 1.

heading: 4.3 Evaluation Metrics
text: We use two evaluation metrics: exact match which evaluates if the predicted and gold logical forms are semantically equivalent (Gu et al., 2021) and F1 score based on predicted and gold answers.

heading: 4.4 Baselines
text: We compare DARA with three different baselines: ICL-based Agents, Fine-tuned Agents, and Bottomup semantic parsers. As DARA is a semanticparsing LLM agent, we do not include retrievalaugmented generation methods in our baselines such as Think-on-Graph (Sun et al., 2024a), StructGPT (Jiang et al., 2023b), which only measures the top-1 accuracy of answers (Hits@1). The difference is detailed in Appendix B.
ICL-based Agents. We use ICL-based LLM agents in AgentBench as baselines, which provides seven functions to interact with KG and perform logical reasoning. GPT-4 and Llama-2-chat (70B) serve as the backbone LLMs.
Fine-tuned Agents. To evaluate the efficacy of DARA against alternative fine-tuned LLM agents, we include models in AgentTuning (Zeng et al., 2023), i.e., AgentLM-7B, AgentLM-13B as baselines. AgentLM 7B/13B fine-tuned Llama-2-7B and 13B on 324 reasoning trajectories generated by GPT-4 in the AgentBench framework. As AgentLMs trained on different examples with DARA, to conduct a direct comparison, we also
convert training data of DARA into AgentBench Framework and fine-tune the Llama-2-7B, called AgentBench-7B. The only difference between AgentBench-7B and DARA (Llama-2-7B) lies in the reasoning framework being utilized, as all other factors remain consistent. One example of the reasoning trajectory in AgentBench Framework is shown in Appendix Table 12.
Bottom-up semantic parser. Although comparing with bottom-up semantic parsers is not an apples-to-apples comparison, we still include two state-of-art models ArcaneQA (Gu and Su, 2022) and Pangu (Gu et al., 2023). This comparison allows for a comprehensive assessment of DARA’s performance in the realm of semantic parsing, shedding light on the effectiveness of language agents and potential advancements in the domain.
ArcaneQA employs a constraint decoding method to generate an executable sub-program at each step. Pangu follows the enumeration-thenrank paradigm. At each step, it enumerates all possible subprograms and applies a discriminator to select top-k candidates with the highest scores.

heading: 4.5 Implementation Details
text: We full fine-tuned DARA based on Llama 2 7B/13B (Touvron et al., 2023), CodeLlama 7B (Rozière et al., 2023), Mistral-7B (Jiang et al., 2023a) with 2-4 80GiB H100 and use one 40GiB A100 GPU for inference. The implementation details can be found in Appendix C.

heading: 5 Experiment Results

heading: 5.1 Overall Performance
text: As presented in Table 2, DARA exhibit consistent superiority over both ICL-based agents and fine-tuned counterparts across three datasets. It is crucial to highlight that alternative fine-tuned agents, such as AgentLM-7B, AgentLM-13B, and AgentBench-7B, exhibit inferior performance compared to GPT-4 agents. For instance, on GrailQA, AgentLM-13B lags behind GPT-4 by 10.88% while DARA (Llama-2-7B) and DARA (Llama-2-13B) outperform GPT-4 by 11.82% and 14.46% (F1), respectively. Although AgentBench-7B is trained using the same data as DARA, it trails DARA (Llama2-7B) by 18.57%. These results serve as strong evidence of DARA’s effectiveness and superiority in handling KGQA tasks. DARA shows flexibility with different back-end LLMs and exhibits improved performance commensurate with the capa-
bilities of the selected model–opting for stronger LLMs results in enhanced performance. Llama-213B can achieve better results than Llama-2-7B. It is noteworthy that DARA (Mistral-7B) either outperforms or achieves comparable performance with DARA (Llama-2-13B) on GraphQ and GrailQA datasets. DARA can also be utilized in the ICL fashion using prompting. When equipping GPT-4 with DARA, as demonstrated in Table 3, we show that DARA (GPT-4) achieved the best performance on 400 hardest questions with longest reasoning paths from GrailQA and GraphQA.
In comparison to bottom-up parsers, DARA showcases its prowess. DARA (Mistral-7B and Llama-2-13B) outperforms both AcraneQA and Pangu (T5-large) on GraphQ and WebQSP. Notably, DARA (Mistral-7B) outperforms AcraneQA by 21.97% on GraphQ. For GrailQA, while DARA lags behind Pangu (T5-large) by 11.41% (F1score), it is important to note that Pangu utilized over 57 times more data (44,337 examples) for training, emphasizing the substantial efficiency of DARA in learning from a limited dataset (768 examples). On WebQSP, DARA (Llama-2-7B) achieves the best performance, outperforming Pangu (T5large) by 5.95%. However, an interesting observation is that the performance of all models is much lower than that of models trained on the other two datasets. Pangu (T5-large) can achieve a 78.9% F1 score on the whole test data of WebQSP (Gu et al., 2023) while it only achieves a 36.72% F1 score on zero-shot evaluation. Shu and Yu (2024) has a similar observation on cross-dataset evaluation. The main reason is the different data construction methods and limited expressivity of current s-expression. We provide detailed analysis in Appendix E.
5.2 Deatailed Analysis on DARA Components The effectiveness of Iterative Task Decomposition (ITD). As shown in Table 4, when equipped with the single-pass pre-decomposition (PD), the performance drops more than 6% on GraphQ, GrailQA and 14.39% on WebQSP. After inspecting the errors made by PD, we found models cannot generate perfect decomposed tasks in a singlepass. The generated tasks are often redundant or incorrect. For the question: Which conference sponsor also sponsored the conference series with GridRepublic? In the pre-decomposition approach, the decomposed tasks are as follows: Task 1 - Find the conference series with GridRepublic, Task 2 - Find the sponsor of the conference series with
Model GraphQ-hard GrailQA-hard WebQSP EM F1 EM F1 EM F1 ICL-based Agents
AgentBench (GPT-4) 41.5 41.6 36.0 44.9 25.0 32.1 DARA (GPT-4) 48.0 52.0 51.5 61.6 26.8 33.6
Fine-tuned Agents
DARA (Llama-2-7B) 46.0 48.8 41.0 51.0 30.4 42.67
Table 3: The performance of DARA in the in-context learning fashion on difficult questions.
GraphQ GrailQA WebQSP
DARA 62.74 77.71 42.67
w/ PD 56.50(6.24↓) 71.22(6.49↓) 28.28(14.39↓) w/o SDR 56.79(5.95↓) 75.12(2.59↓) 40.77(1.90↓) w/o FT Rtr. 61.29(1.45↓) 75.76(1.95↓) 36.18(6.49↓)
Table 4: Ablation study of different components of DARA (Llama-2-7B) under F1 scores.
GridRepublic, and Task 3 - Find the conferences sponsored by the same sponsor, in which Task 3 is unnecessary for answering the question. The ITD method, on the other hand, would stop the iteration at Task 2 and return the answer. These results underscore the importance of ITD for DARA, as it allows for dynamic task decomposition, leading to improved performance across various datasets.
The role of the Skim-then-Deep-Reading relation selection method. The results presented in
Table 4 demonstrate the significance of the skimthen-deep-reading relation (SDR) selection strategy. When this strategy is not employed, the performance experiences a decrease, especially in GraphQ (5.95% drop). To illustrate the impact, for the question: What vocal range is Pavarotti? In the absence of the skim-then-deep-reading selection strategy, DARA selects the incorrect relation, music.artist.track. However, with the selection strategy in place, the correct relation, music.opera_singer.voice_type is selected, whose description contains information about the vocal range. These results highlight the critical role that the skim-then-deep-reading relation selection strategy plays in enhancing DARA’s performance especially when the literal meaning of relations cannot express the needed information.
The fine-tuned retriever for schema filtering. To better select the relations or classes, we leverage a fine-tuned retriever to filter the top five candidates for the DARA (Section 3.3). As shown in Table 4, the fined-tuned retriever (FT Rtr.) contributes to the improvement in model performance. Here, an off-the-shelf retriever all-mpnet-base-v22 is used. On WebQSP, without the trained retriever, the performance drops by 6.49%.
2https://www.sbert.net/docs/ pretrained_models.html

heading: 5.3 Error Analysis
text: To have a concrete understanding of the utility of DARA, we compare its reasoning trajectories with those of ICL-based agents (GPT-4, Llama-2-70B chat) and fine-tuned AgentBench-7B. The common errors among these agents were identified during the grounding phase. AgentBench-7B exhibits a deficiency in schema understanding, selecting wrong relations. In contrast, GPT-4 showcases a superior grasp of schema understanding. However, both agents tend to prematurely terminate tasks when reaching a CVT node with n-ary relations3. DARA, on the other hand, effectively identifies the CVT node through its skim-then-deep-reading relation selection component. Moreover, the unthoughtful designed action space in the AgentBench framework not only limits the agents’ capabilities for addressing questions that involve class information but also hinders their performance in handling queries lacking explicit entities. Furthermore, for ICL-based LLM agents, the challenge of instruction-following persists. For example, Llama2-chat (70B) encounters challenges in following the reasoning pipeline provided in the demonstration, leading to failure function calls. For fine-tuned AgentBench-7B, without an explicit task decomposition module like DARA, it struggles to obtain effective high-level guidance for some complex questions. Comparison examples between those agents are provided in Appendix F.

heading: 5.4 The Quality of GPT-4 Generated Data
text: The GPT series has demonstrated exceptional performance in data generation and annotation (Gilardi et al., 2023; Wang et al., 2023d; Xu et al., 2024). We investigate whether reasoning trajectories converted by GPT-4 suffice for language agents fine-tuning in KGQA. We fine-tuned DARA on GPT-4 converted trajectories on both Llama2-7B and Mistral-7B. As illustrated in Figure 3, DARA with different back-end LLMs exhibits a similar pattern. Despite data size increasing from 1k to 3k, the performance on three datasets always lags behind that of 768 human-verified examples. Upon closer inspection of the generated data, we identified the following issues. GPT-4 struggles to follow the structured output format, i.e., confusing ‘step’ and ‘task’ and misinterprets the intent of the given logical form, thus generating wrong tasks.
3CVT is the mediator in Freebase so the agent needs a further step to get the final answer.
Furthermore, the schema design of Freebase (e.g. CVT) increases the difficulty of comprehension. AgentTuning (Zeng et al., 2023) corroborates our findings, reporting a low success rate when utilizing GPT-4 to generate reasoning trajectories across six diverse environments, ranging from 0.52% on Mind2Web to 35.2% on AlfWorld. These collective observations highlight a potential future direction for automatically generating high-quality intermediate trajectories for language agents in various environments, particularly when GPT-4’s performance is constrained.

heading: 5.5 The Running Expenses of Models
text: In Table 5.5, we analyze the running expenses of various models on 4,559 test examples. Finetuned agents, such as DARA, outperform ICL-based LLM agents like GPT-4 and Llama-2-Chat (70B) in both cost and speed. GPT-4 stands out as the most expensive, with costs sixty times higher and speed four times slower than DARA (Llama-2-7B). Pricing information for the GPT-4 experiment is available on the Azure webpage4. To assess the price of GPUs used for Llama-2, we refer to Replicate.com5. While the cost may be affected by different factors (e.g. GPU service providers, optimized inference methods), the affordability and efficiency of fine-tuned agents remain evident.

heading: 6 Conclusion
text: In this paper, we present DARA, a fine-tuned LLM agent for KGQA. Experimental results highlight the superior performance of DARA with various backend LLMs, compared to ICL-based LLM agents as well as alternative fine-tuned language
4https://azure.microsoft.com/en-us/ pricing/details/cognitive-services/ openai-service/
5https://replicate.com/pricing
agents. Moreover, DARA proves to be more costeffective and time-efficient than ICL-based agents. Additionally, our findings reveal challenges faced by GPT-4 in converting structured logical forms into natural language for reasoning trajectory construction. In the future, we will center on developing methods to generate high-quality reasoning trajectories for LLM agent fine-tuning in KGs and other symbolic environments where GPT-4 faces difficulties.

heading: 7 Acknowledgements
text: We thank Kexin Wang, Hiba Arnaout, Imbesat Hassan Rizvi, Haritz Puerto, and Tilek Chubakov for their valuable discussion and feedback. This work was supported by the Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA) through the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research.
Limitations
Although DARA shows superiority over ICL-based and other fine-tuned LLM agents, there are several limitations that call for further improvement. First, DARA lacks the error-correcting ability, which means it cannot correct itself when erroneously decomposing the question or selecting incorrect relations. Humans excel in fixing errors based on the current observation and the final goal. Exploring approaches like Reflextion (Shinn et al., 2023) could enhance error-correcting capabilities, although it is important to note that such methods currently work effectively only with very powerful LLMs, such as GPT-4. Second, we performed evaluations in the zero-shot setting due to the high cost of commercial LLMs. To create fine-tuning data, we use GPT-4 to translate 768 golden logical forms to natural language, which increases the complexity of the method. Third, following
previous research, we only evaluate it on popular datasets using Freebase as the backend. For other knowledge graphs such as Wikidata, the current intermediate representation s-expression can not be directly transferred. Generalizing DARA to other knowledge graphs will be our future work.

heading: A Reasoning Trajectories Creation
text: For <question, s-expression> pairs in the original dataset, we linearized the s-expression and break down it into several tasks according to logical operations (refer to Appendix A.2). These tasks serve as golden decomposed tasks. To convert them into natural language, we prompt GPT-4 coupled with the corresponding question using the prompt in Appendix D.1. For the reasoning trajectories of the skim-then-deep-reading relation selection component, we automatically create them using the schema items in golden s-expression and functions in Section 3.3. Then the data is inspected and rectified by humans.
A.1 Dataset Quality Inspection
Fine-tuning LLMs with large but very noisy data contributes little to performance improvement (Zhou et al., 2023; Lee et al., 2023). We manually inspect selected data from the original
dataset. Several issues are identified: unnatural expressions, redundant logical forms, ambiguous questions, and inconsistencies between questions and logical forms. GrailQA is a large-scale dataset aiming at evaluating different level generalization abilities of KGQA models. Among 500 sampled data points, roughly 15% contained errors. The main issue is the wrong directionality of relations. In the case where the question is asking about operating systems that include 386bsd, the logical form is searching for operating systems belonging to 386sd (JOIN computer.operating_system.parent_os 386BSD). Another common issue is improper comparative operations. For instance, the question is asking about the comparison more than while ge (greater than or equal to) is adapted in logical forms. Besides, some errors involve redundant components within the query, which is unnecessary to answer the question. GraphQ is a medium-sized dataset where questions are generated by expert annotators according to the query graph. It has multiple paraphrased questions in the same logical form. We only retain one question for each logical form. WebQSP is another medium-sized dataset comprising questions sourced from Google query logs. This dataset includes implicit entities not mentioned in the questions and complex logical operations. Furthermore, it contains many ambiguous/open questions. For example, what did Stephen Hawking become famous for? is an open question. The answer could be his profession, book, discovery, etc. The data statistics of these three datasets are shown in Table 6.
A.2 Logical Operations
The common logical operations are as follows:
• Superlative (ARGMIN/ARGMAX): it is used to get entities with the maximum value of a given attribute.
• Intersection (AND): it is used to get an intersection between two sets of entities.
• Comparative (LT/LE/GE/GE): it is used to filter properties according to numerical constraints.
• Count (COUNT): it is used to count the number of a set of entities.
• Projection (JOIN): This operation is used to get the other side entity of a triplet via the relation.
We break down the logical form into subtasks according to the above logical operations. For projection operation, it may be used due to the KG structure rather than the question. For such cases, we do not break it down.
A.3 Logical form used in DARA We use s-expression (Gu et al., 2021) as the logical form to represent SPARQL. The definition and syntax of logical operations are as shown in Table 7.
A.4 Statistics of training reasoning trajectories
As shown in Table 8, we utilize 768 reasoning trajectories to train DARA.

heading: B Difference between RAG methods and Semantic-parsing methods
text: Retrieval-augmented generation method (RAG) such as Think-on-Graph, StructGPT as discussed in the related work (Section 2), which uses Hits@1 as the metric to measure the top-1 accuracy, which does not consider the recall. Semantic-parsing methods (e.g. DARA, Pangu, ArcaneQA, AgentTuning) derive logical forms to query the KGs, which use the F1-score to measure recall and precision of answers obtained from logical forms. Given these fundamental differences in goals and metrics, drawing direct comparisons between these methodologies may not fully capture their respective strengths and limitations. RAG methods rely on the LLM to synthesize responses based on extracted triplets and its internal knowledge, rather than directly querying a KG. For a question with multiple answers, ToG cannot guarantee getting all answers while semantic parsing methods (e.g. DARA) can get all answers from the KGs by executing the derived logical forms. To give a quantitative analysis, we compare ToG (GPT-4) with DARA (llama 2-7B, mistral-7B) on 400 GrailQA test questions with multiple answers using both hit@1 and F1-score. As shown in Table 9, DARA outperforms ToG in the tested 400 examples. We can see ToG has a quite low F1 score as we discussed before. The Hit@1 score is also lower than DARA. Evaluating ToG on the F1 score might seem unfair, given its design for high accuracy in single-answer scenarios. Conversely, using hit@1 does not fully capture the recall and precision of logical forms generated by DARA.
C Implementation Details
Training We fine-tuned Llama-2 and CodeLlama using 2-4 80GiB H100 for 10 epochs using 2e5 learning rate with a 3% warm-up ratio and 4
batch size. Deepspeed is utilized to accelerate the training process. For Mistral-7B, the learning rate is 2e-6 learning rate with a 10% warm-up ratio.
Inference To do inference on Llama 7B/13B, we use one 40GiB A100 GPU while 2 80GiB A180 GPUs are used to test Llama-2-chat-70B. We call Azure OpenAI service API for GPT-4 inference.

heading: D Prompt and reasoning trajectories
text: D.1 Prompt used to convert logical forms to natural languages
We present the prompt and demonstrations used to convert broken-down logical forms into natural languages in Table 10.
D.2 A exemplary reasoning trajectory of DARA
As shown in Table 11, we provide the complete reasoning trajectory of DARA, which includes iterative task decomposition and the two-stage skim-thendeep-reading relation selection method.
User: You are knowledgeable about Freebase. You need to describe what is the intent of logical forms in the given subtasks and steps. To find the answer to the question ’Which type of dish common in manado cuisine contains sugars? The linked entities are sugars (m.06x4c), and manado cuisine (m.0102k5v9).’, it has the following subtasks. Note: Some tasks may have multiple steps due to the schemas of the KG. Task 1: Step1:(JOIN food.dish.ingredients m.06x4c) Task 2: Step1:(JOIN (R dining.cuisine.dishes) m.0102k5v9) Task 3: Step1:(AND task1 task2) Task 4: Step1:(JOIN food.type_of_dish.dishes task3) To help you understand the relations used in the above steps, we provide the following descriptions of them. 1. the incoming relation ’food.dish.ingredients’, which describes please enter the main ingredients usually included in this dish. The type of its head entity is ’food.dish’ (A dish is a food prepared and presented in a certain way. These are foods that are ready to be served and eaten as a meal itself, or part of a larger meal). 2. the outgoing relation ’dining.cuisine.dishes’, which describes the dishes of cuisine. The type of its tail entity is ’food.dish’ (A dish is a food prepared and presented in a certain way. These are foods that are ready to be served and eaten as a meal itself, or part of a larger meal). 3. the incoming relation ’food.type_of_dish.dishes’, which describes the dishes of type of dish. The type of its head entity is ’food.type_of_dish’ (This type is used to classify dishes into broad categories, such as appetizers, soups, desserts, etc.). Please describe the intent of the above tasks and steps. Assistant: To solve this question, we can break it down into the following tasks: Task 1: Determine the dishes that contain sugars. - Step 1.1: use the incoming relation ’food.dish.ingredients’ of surgars with the operator JOIN to find dishes whose ingredients contain sugars. Task 2: Identify the dishes in Manado cuisine. - Step 2.1: we can use the outgoing relation ’dining.cuisine.dishes’ of manado cuisine to find the dishes in Manado cuisine. Task 3: Find the common dishes from Task 1 and Task 2. - Step 3.1: we can use the operator AND to get the common dishes from Task 1 and Task 2. Task 4: Determine the type of dish among the common dishes. - Step 4.1: to get the type of dish, we use the the incoming relation ’food.type_of_dish.dishes’ of those entities obtained from Task 3 to get the final answer. User: To find the answer to the question ’which academic was an dashiell that was born in japan? The linked entities are Japan (m.03_3d), dashiell (m.03m3mgq).’, it has following tasks. Each task may have multiple steps due to the schemas of the KG. Task 1: Step1:(JOIN people.person.place_of_birth m.03_3d) ... Please describe the intent of the above tasks and steps. Assistant: To solve this question, we can break it down into following tasks: Task1: find people who were born in Japan. ... User: To find the answer to the question ’{question}’, it has following tasks. {subtasks} To help you understand the relations used in above steps, we provide the following descriptions of them. {description} Please describe the intent of above tasks and steps.
Table 10: The prompt used to convert linearized logical forms to natural language.
User: You are an agent that answers questions based on the knowledge stored in a knowledge base. For a given question, you find the answer via iteratively task decomposition. For each task, you need to complete it using the provided functions. After completing the task, you need to decide if further tasks are needed baed on the current results. The provided functions are: 1. get_relations(variable: var) -> list of relations. A variable can be either an entity (represented by the mid) or a set of entities (i.e., the result of a previous query). This function helps to navigate all relations in the KB connected to the variable, so you can decide which relation is the most useful to find the answer to the question. A simple use case can be ’get_relations(g.013c2d)’, which finds all relations/edges starting from the entity g.013c2d. The argument of get_relations should always be an entity or a variable (e.g., #0) and not anything else. 2. get_neighbors(variable: var, relation: str) -> variable Given a variable, this function returns all entities connected to the variable via the given relation. Note that, get_neighbors() can only be used after get_relations() is used to find a set of viable relations. A simple use case can be ’get_neighbors(m.013c1m, people.person.profession)’, which returns the profession of m.013c1m in Freebase. 3. intersection(variable1: var, variable2: var) -> variable. Given two variables, this function returns the intersection of the two variables. The two variables MUST be of the same type! 4. get_relevant_relations(task: str) -> list of relations. Given the decomposed task, this function helps to retrieve related relations. Note that, this function can only be used when no entity is mentioned in the task. Otherwise, get_relations() should be used to get relations of the entities. 5. argmax(variable: var, attribute: str) -> variable. Given a variable, this function returns the entity with the maximum value of the given attribute. A simple use case can be ’argmax(variable, age)’, which returns the oldest entity belonging to the variable. 6. argmin(variable: var, attribute: str) -> variable. Given a variable, this function returns the entity with the minimum value of the given attribute. A simple use case can be ’argmin(variable, age)’, which returns the youngest entity belonging to the variable. 7. count(variable: var) -> int. Given a variable, this function returns the number of entities belonging to the variable. 8. get_classes(variable:var) -> list of classes. A variable is a set of entities (i.e., the result of a previous query). This function returns the classes of given entities, which can be used to filter entities of certain classes. 9. get_relevant_classes(task: str) -> list of classes. Given the deomposed task, this function returns related relations of a decomposed task when there is no entitiy in the task. Otherwise, get_classes() should be used. 10. lt(atrribute: str, value: float) -> variable. Given a attribute and a numerical value, this function returns the entity whose attribute is less than (lt) the given value. This function can only be used after get_relations() or get_relevant_relations(). Similarly, le(attribute:str, value:float) is to find entities whose attribute is less than or equal to (le) the given value. ge(relation:str, value:float) is to find entities whose attribute is greater than or equal to (get). gt(relation:str, value:float) is to find entities whose attribute is greater than (gt) the given value. 11.get_descriptions(relation_list:str) -> descriptions of given relations. This function returns descriptions of the given relation list. The input relation list should be seperated by comma. After getting relations using get_relations or get_relevant_relations, if you are unsure which relation to select, select several most possible candidates and use this function to get descriptions of them to help you make the final decision. 12. get_attributes(variable: var) -> list of attributes. This function helps to find all numerical attributes of the variable. Please only use it if the question seeks for a comparative accumulation (i.e., argmax/argmin, le/lt/ge/gt) or numerical attributes of entities. 13. get_relevant_attributes(task: str) -> list of attributes. This function finds relevant attributes of a task.
Note it can only be used when there is no entity in the question! Otherwise, get_attributes should be used. After a variable is produced along the process, you need to judge whether a variable is the final answer to the question. Each variable is represented as an id starting from 0. For example, #0 is the first variable, #1 is the second variable, and so on. Once you find the answer, respond with ’Final Answer: #id’, where id is the id of the variable that you think is the final answer. For example, if you think #3 is the final answer, you MUST respond with ’Final Answer: #3’ at the end to finish the question. You can only take ONE action at a time!! After you get the observation from its execution, you can take another action. You can take at most 15 actions to find the answer to the question. Here is a demo that you can use to get started. Question: which bi-propellant rocket engines use unsymmetrical dimethylhydrazine and have a dry mass of less than 980.0? The linked entity is Unsymmetrical dimethylhydrazine (m.017q1y). The given question can be decomposed into the following subtasks: # Task 1: Find bi-propellant rocket engines that use unsymmetrical dimethylhydrazine as fuel. ## Step 1.1:### Action 1.1.1: get_relations(m.017q1y) ### Observation 1.1.1: [spaceflight.rocket_engine_fuel.rocket_engines, chemistry.chemical_compound.average_molar_mass]. ### Thought 1.1.1: From the above relations, I use the outgoing relation ’spaceflight.rocket_engine_fuel.rocket_engines’ of Unsymmetrical dimethylhydrazine with the function get_neighbors to find bi-propellant rocket engines that use this fuel. ### Action 1.1.2: get_neighbors(m.017q1y, spaceflight.rocket_engine_fuel.rocket_engines) ### Observation 1.1.2: variable #0, which are instances of spaceflight.bipropellant_rocket_engine. # Task 2: Find bi-propellant rocket engines with a dry mass less than 980.0. ## Step 2.1:### Action 2.1.1: get_relevant_relations(Find bi-propellant rocket engines with a dry mass less than 980.0.) ### Observation 2.1.1: The relevant relations are spaceflight.rocket_engine.dry_mass, spaceflight.rocket_engine_fuel.rocket_engines, spaceflight.bipropellant_rocket_engine.wet_mass, spaceflight.rocket_engine_cycle.rocket_engines. ### Thought 2.1.1: we can use spaceflight.rocket_engine.dry_mass with the function lt to find bi-propellant rocket engines with a dry mass less than 980.0. ### Action 2.1.2: lt(spaceflight.rocket_engine.dry_mass, 980.0(ttp://www.w3.org/2001/XMLSchema#float)### Observation 2.1.2: #1, which are instances of spaceflight.bipropellant_rocket_engine. # Task 3: Find the bi-propellant rocket engines that satisfy both Task 1 and Task 2. ## Step 3.1:### Thought 3.1.1: we can use the function intersection to get the common bi-propellant rocket engines from Task 1 and Task 2. The final answer would be the set of bi-propellant rocket engines that use unsymmetrical dimethylhydrazine and have a dry mass less than 980.0. This should be the final s-expression. ### Action 3.1.1: intersection(#0, #1) ### Observation 3.1.1: #2, which are instances of spaceflight.bipropellant_rocket_engine. ### Thought 3.1.2: #2 should be the final answer. # Final answer:#2. Do you understand it? Assistant: Yes, I’ve understood your instruction and the demonstration. User: Great! The new question is {question}
Table 13: DARA for ICL Agents

heading: E Why is there a performance discrepancy between WebQSP and GrailQA, GraphQ?
text: One significant factor lies in the different sources of the questions. Questions of GrailQA and GraphQ are derived from logical forms, which are structured and explicitly aligned with underlying KGs. Conversely, WebQSP collects questions from real users’ query logs on Google search, often containing common sense or complex logical operations. Intuitively, when the SPARQL is longer but the question is short, it often signifies a requirement for implicit knowledge or complex reasoning. To explore this hypothesis, we simply calculate the ratio between the lengths of SPARQL queries and questions, visualizing it in a violin plot as shown in Figure 4. Compared with GrailQA and GraphQ, we observe that WebQSP exhibits more extreme cases in the length ratio, suggesting that it contains more short questions with long SPARQL queries. Some of them (cf. Table 14) include complex logical constraints (e.g. time comparison) that cannot be represented by s-expression. Besides, Humans can formulate concise and abstract questions based on common sense, which poses a challenge for parsing questions in the absence of such knowledge or an ability to leverage it. For example, to answer the question what year did Seattle Seahawks go to the Super Bowl? The model needs to know Superbowl serves as the final match of the American National Football League. Therefore, it needs to find the season when the Seattle Seahawks were not only the champion but also the runner-up. Those real questions from humans can be more flexible, and
obscurer than synthetic questions derived from logical forms. In light of these findings, it is evident that advancing the field of knowledge graph reasoning requires the design of more realistic datasets, the creation of more expressive intermediate representations, and the exploration of new model paradigms that can effectively harness commonsense knowledge. but sometimes such synthetic questions lack clarity and authenticity, which may impede the LLM agent’s understanding. For example, which industry is ayala land in which real estate and rental and leasing is also in? is ambiguous and unnatural.
Question what did james k polk do before he was president?
SPARQL PREFIX ns: <http://rdf.freebase.com/ns/> SELECT DISTINCT ?x WHERE { { SELECT ?pFrom WHERE { ns:m.042f1 ns:government.politician.government_positions_held ?y . ?y ns:government.government_position_held.office_position_or_title ?x ; ns:government.government_position_held.basic_title ns:m.060c4 ; # President ns:government.government_position_held.from ?pFrom . } } ns:m.042f1 ns:government.politician.government_positions_held ?y . # James K. Polk ?y ns:government.government_position_held.office_position_or_title ?x ; ns:government.government_position_held.from ?from .
FILTER(xsd:dateTime(?pFrom) - xsd:dateTime(?from) >0) }
Question what super bowl did peyton manning win?
SPARQL
SELECT DISTINCT ?z WHERE { ns:m.027jv8 ns:sports.pro_athlete.teams ?y . ?y ns:sports.sports_team_roster.team ?x . #team ?x ns:sports.sports_team.championships ?z . # super bowls ?z ns:common.topic.notable_types ns:m.01xljv1 . #super bowl ?z ns:sports.sports_championship_event.champion ?c . ?z ns:time.event.start_date ?a .
# Check the time overlap FILTER(NOT EXISTS {?y ns:sports.sports_team_roster.from ?sk0} || EXISTS {?y ns:sports.sports_team_roster.from ?sk1 . FILTER(xsd:datetime(?sk1) - xsd:datetime(?a) <= 0) })
FILTER(NOT EXISTS {?y ns:sports.sports_team_roster.to ?sk2} || EXISTS {?y ns:sports.sports_team_roster.to ?sk3 . FILTER(xsd:datetime(?sk3) - xsd:datetime(?a) >= 0 ) }) }
Question which country in north america is divided into provinces?
SPARQL PREFIX ns: <http://rdf.freebase.com/ns/> SELECT DISTINCT ?x WHERE { ns:m.059g4 ns:location.location.contains ?x . # North America ?x ns:common.topic.notable_types ns:m.01mp . # Country ?x ns:location.location.contains ?y . ?y ns:common.topic.notable_types ?t . # All the possible "province" type FILTER ((?t = ns:m.01nm) || (?t = ns:m.02_1y_9) || (?t = ns:m.02_3ny_) || (?t = ns:m.02_3phk) || (?t = ns:m.02_3r2r) || (?t = ns:m.02_3rt3) || (?t = ns:m.02_3zf4) || (?t = ns:m.02_40h1) || (?t = ns:m.02_96lm) || (?t = ns:m.02yxk5c) || (?t = ns:m.02zd6yn) || (?t = ns:m.03z96kq) || (?t = ns:m.04g7rg9) || (?t = ns:m.04js0h5) || (?t = ns:m.065rjpr) || (?t = ns:m.078_8dm) || (?t = ns:m.0hzcb3l) || (?t = ns:m.0hzcb5p) || (?t = ns:m.0hzcb69) || (?t = ns:m.0hzcb7p) || (?t = ns:m.0hzcd76) || (?t = ns:m.0hzcd7v) || Question who was president after franklin d. roosevelt?
SPARQL PREFIX ns: <http://rdf.freebase.com/ns/> SELECT DISTINCT ?x WHERE { # President of the United States ns:m.060d2 ns:government.government_office_or_title.office_holders ?y1 . # Franklin D. Roosevelt ?y1 ns:government.government_position_held.office_holder ns:m.02yy8 ; ns:government.government_position_held.to ?to . # President of the United States ns:m.060d2 ns:government.government_office_or_title.office_holders ?y2 . ?y2 ns:government.government_position_held.office_holder ?x ; ns:government.government_position_held.to ?from .
FILTER(xsd:dateTime(?from) - xsd:dateTime(?to) >0) } ORDER BY xsd:dateTime(?from) LIMIT 1
Question who is meredith gray married to in real life?
SPARQL PREFIX ns: <http://rdf.freebase.com/ns/> SELECT DISTINCT ?x WHERE { ns:m.0268x2f ns:tv.tv_character.appeared_in_tv_program ?y . ?y ns:tv.regular_tv_appearance.actor ?z . ?z ns:people.person.spouse_s ?s . ?s ns:people.marriage.spouse ?x . ?s ns:people.marriage.type_of_union ns:m.04ztj . Filter (?x != ?z) FILTER(NOT EXISTS {?s ns:people.marriage.from ?sk0} || EXISTS {?s ns:people.marriage.from ?sk1 . FILTER(xsd:datetime(?sk1) <= "2015-08-10"^^xsd:dateTime) }) FILTER(NOT EXISTS {?s ns:people.marriage.to ?sk2} || EXISTS {?s ns:people.marriage.to ?sk3 . FILTER(xsd:datetime(?sk3) >= "2015-08-10"^^xsd:dateTime) }) ?s ns:people.marriage.from ?sk4 . } ORDER BY DESC(xsd:datetime(?sk4)) LIMIT 1 Question what was franklin d roosevelt ’s job before president?
SPARQL PREFIX ns: <http://rdf.freebase.com/ns/> SELECT DISTINCT ?x WHERE { { SELECT ?pFrom WHERE { ns:m.02yy8 ns:government.politician.government_positions_held ?y . # Franklin D. Roosevelt ?y ns:government.government_position_held.office_position_or_title ?x ; ns:government.government_position_held.basic_title ns:m.060c4 ; # President ns:government.government_position_held.from ?pFrom . } } ns:m.02yy8 ns:government.politician.government_positions_held ?y . # Franklin D. Roosevelt ?y ns:government.government_position_held.office_position_or_title ?x ; ns:government.government_position_held.from ?from .
FILTER(xsd:dateTime(?pFrom) - xsd:dateTime(?from) >0
Table 14: Questions in unseen schemas test data of WebQSP with unseen schemas whose length ratio is larger than 10.

heading: F Error Analysis
text: As discussed in section 5.3, we compared DARA(Llama-2-7B) with ICL-based agents (GPT4) and fine-tuned Agentbench-7B. We show three examples in Table 15 ∼ Table 17.

heading: Future Work
text: Generalizing DARA to other knowledge graphs will be our future work