Enhancing LLMs for Knowledge Base Question Answering by Chain-of-Decomposition
Authors: Yonggang Zhang
Jianqi Gao
Jie Lu
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs; LoRA; KBQA
Primary area: learning on graphs and other geometries & topologies

Abstract: Large language models (LLMs) have demonstrated remarkable success across diverse domains through in-context learning or fine-tuning. However, adapting LLMs to Knowledge Base Question Answering (KBQA) remains challenging, as KBQA necessitates multi-step reasoning over large-scale structured knowledge bases. 
Directly prompting LLMs with entire knowledge bases incurs prohibitive computational costs, while existing methods provide limited guidance on effectively fine-tuning LLMs for such complex reasoning tasks.
In this work, we propose Chain-of-Decomposition (\texttt{CoD}), a novel framework that decomposes KBQA into three modular steps: (1) an LLM-free retrieval module to extract query-relevant subgraphs from the knowledge base, (2) a parameter-free reformulation step that transforms retrieved contexts into structured reasoning paths, and (3) a lightweight LLM-based reasoning module trained to evaluate the logical validity of each path. By isolating computation-heavy retrieval and rule-based reformulation from LLM reasoning, \texttt{CoD} reduces task complexity and enables efficient fine-tuning focused solely on the final verification step. Comprehensive experiments demonstrate that Llama-2 7B, fine-tuned with the proposed \texttt{CoD} surpasses strong baselines, including GPT-4 augmented with retrieved knowledge, achieving state-of-the-art performance on WebQSP and CWQ benchmarks. Our code is publicly available at \url{https://github.com/YonggangZhang9412/KBQA-CoD}.

Published as a conference paper at ICLR 2026
ENHANCING LLMs FOR KNOWLEDGE BASE QUESTION ANSWERING BY CHAIN-OF-DECOMPOSIIIO
Yonggang Zhang
Jianqi Gao Shanghai University
Hong Kong University of Science and Technology zhangygeust. hk
jiangi gaoßshu. eddu.cr
Jie Lu
University of Technology Sydney jie. lu@uts.......................... edu..ttseeddua...
ABSTRACT
Large language models (LLMs) have demonstrated remarkable success across diverse domains through in-context learning or fine-tuning. However, adapting LLMs to Knowledge Base Question Answering (KBQA) remains challenging, as KBQA necessitates multi-step reasoning over large-scale structured knowledge bases. Directly prompting L.LMs with entire knowledge bases incurs prohibitive computational costs, while existing methods provide limited guidance on effectively fine-tuning LLMs for such complex reasoning tasks. In this work, we propose Shain-of-Decompositiee
(CoD), novel framework that decomposes KBQA into three modular steps: (1) an LLM-free free retrieval module to extract query-relevant subgraphs from the knowledge base, (2) a parameter- frreeer--rreeeeeeee reformulation step that transforms retrieved contexts into structured reasoning paths, and (3) lightweight LLM- -baeed reasoning module trained to evaluate the logical validity of each path. By isolating computation henpyyytion-heavy retrieval and rule-based reformulation from LLM reasoning, CoD reduces task complexity and enables efficient fine-tuning focused solety the final verification step. Comprehensive experiments demonstrate that Llama-2 7B, fine-tuned with the proposed CoD surpasses strong baselines, including GPT-4 augmented with retrieved knowledge, achieving state-of-the-art performance on WebQSP and CWQ benchmarks. Our code is publicly available at KBQA-BOQQACOO CoD.
https: wwt.buttttth..gggggg wtt. wwwmm//ogggggghanngggg
INTRODUCTION
The great achievement of large language models (LLMs) have attracted widespread attention (Brown et al., 2020; Ouyang et al., 2022b; OpenAI, 2023). The striking feature of LLMs is their ability to handle complex tasks through reasoning (Wei et al., 2022; Wang et al., 2023b). Despite their im- pressive performance, I.MSs have substantial limitations when facing complex knowledge reasoning tasks (Pan al., 2024) that require multi-hop reasoning. In this regard, advanced works propose to adapt LLMs to these complex tasks by either prompting LLMs with knowledge (Sun et al., 2023; Sui al., 2024) or fine-tuning LLMs with carefully designed data and objectives (Luo al., 2023b).
However, it is challenging to adapt LLMs to the tasks of knowledge base question answering (KBQA). This results from the fact that KBQA aims to reason about answers for an input query, which requires a large-scale knowledge base and deep reasoning. Specifically, prompting LLMs with such a large- scale knowledge base in an in-context learning manner leads to high computation costs. Meanwhile, fine-tuning LLMs to promote the reasoning capability relies heavily the utilized training data and objective functions (Ouyang et al., 2022a). this regard, some works propose to scale down the size of the knowledge base by retrieving query-relevant context (Sun et al., 2023; Sui et al., 2024). Advanced works propose to fine-tune LLMs with appropriate objective functions (Luo et al., 2023a;b), aiming to encourage LLMs predict the answer directly or indirectly. However, the existing works mainly focus on improving the capability of LLMs for complex reasoning, while overlooking the strategy reduce the task complexity.

---

Published as a conference paper at ICLR 2026
1. Retrieval
Knowledge Graphs
3. Reasoning
Quexstin::
Which country SpaceX located in?
Prompt: whether the conclusion: (Tod Entryy can be deduced from the question. if yes. return "yes", ", it no, return "wo"; given |Reusoning Puih) deductive mannerr
USA
LLMs
2. Reformulating
Reasoning Path
Prediction
Logical Form
Answer
mamied to
Pathl: Eloa Musk
Talulah: Riley.
Japan
Patht:
JOIN (ARGMAX (JOIN ))
No
Path2: Elon Musk
lound
SpaceX
located
USA
Path2:
JOIN (AND UJIN m.0d _bv))
Yes
Elon Musk
CBO
Tesla
made in
China
Path3:
JOIN (AROMAX (JOIN R))
No
South
Path4:
Elon Musk
born in
Africa
Path::
AND (ARGMAX (JOIN R))
No
Fatity
Figure 1: CoD overview: Query-relevant information is first retrieved. Reasoning paths are then constructed. Finally, LLMs are fine-tuned to determine whether reasoning paths are logically valid.
In this work, we propose Chain-of-Decomme Decomposition (CoD), a novel framework that simplifies the task by decomposing KBQA into three modular steps. As shown in Fig. 1 and Eq.(5), we decompose the KBQA tasks into chain with three subtasks: retrieval, reformulating, and reasoning by factoring the distribution of the answer generation. The factorization is based on our constructed causal graph (Fig. 2), which formulates the answer-gent seweerr-enrratinn process. First, the objective of the retrieval task is to scale down the knowledge base by returning query-relevane information, which is an LLM-free retrieval module to extract qucry-relevant subgraphs from the knowledge base. This leads to simple binary task problem'. Second, the objective of the reformulating task is transform the retrieved context into reasoning paths, which is a parameter-free reformulation step. Namely, we can perform the reasoning path construction by designing some rules, leading to no learnable parameters for the task. Ultimately, the objective of the reasoning task is to determine whether a reasoning path is logically valid, which a lightweight LLM-based reasoning module. Through the decomposition, we surprisingly find that two of these tasks are LLM-free, which isolates computation- rule-based reformulation from LLM reasoning. Consequently, our decomposition provides detailed instructions on how to simplify KBQA tasks and fine-tune L.MMs for only one task, which drastically reduces task complexity and enables efficient fine-tuning of L.LMs..
retrieval and
To verify the efficacy of the proposed CoD, we conduct experiments on two benchmark KBQA datasets: WebQuestionSP (WebQSP) (Yih et al., 2016) and WenQuesionsss (CWQ) (Tal- mor & Berant, 2018 8). Comprehensive results demonstrate that fine-tuning Llama-2 7B with CoD can outperform all baseline methods by a considerable margin, achieving state-of-the (SOTA) performance. Moreover, the fine-tuned Llama-2 7B outperforms the GPT4 model with retrieved knowledge.
Our main contributions be summarized as follows.
We construct causal graph to formulate the answer-generation process in KBQA. Based on the graph, we factorize the distribution of answer prediction, which motivates a decomposi- tion strategy to reduce the complexity of the KBQA task.
We propose chain-of-deccmposiiie
(CoD), novel framework to adapt LLMs to KBQA
by decomposing the task into three modular steps: retrieval, reformulating, and reasoning. CoD promotes LLL--based KBQA by isolating retrieval and reformulation from reasoning. resulting in a simple reasoning task for LLMs.
The proposed method CoD dramatically outperform the existing methods, achieving state- of-the-art (SOTA) performance. Moreover, fine-tuning Llama-2 7B with CoD outperforms
'Our experimental results (Table 2) demonstrate that employing relatively small generative model, i.e., T5, achieve exciting performance.
2

---

Published as a conference paper at ICLR 2026
the GPT4 model with retrieved knowledge, highlighting its ability to empower lightweight LLMs for complex reasoning.
PRELIMINARY
2.1 KNOWLEDGE BASE QUESTION ANSWERING
Knowledge base question answering (KBQA) is reasoning task based on a large-scale structured knowledge base, i.c., knowledge graph (KG). The KG G is composed of factual knowledge in the form of a set of triples: G = {(e,, T, e,) ex e, E, RJ. where e, and e, are the subject entity and object entity connected by the relation r.
The goal of the KBQA task is to construct function f to predict answers A A(Q) for input query Q in the form of natural language question based on the knowledge from G. i.c., Á = f(Q.)) 9). Here, we leverage A(Q) to represent the fact that the set of answers varies with the input query Q. In this work, we follow previous work (Jiang et al., 2023) and merely consider the close-set scenarios where the topic entities TQQ) used in Q and the answer entities A(Q) used in A are involved in g, e.g. A(Q) C &. More details can be found in Appendix A.
2.2 TWo STRATRGIES FOR KBQA
There are mainly two types of approaches realize the predictor f(Q,9). The first approach is straightforward, realizing f by directly generating answers,
min MinKLLLLAAAAA))
(I)
where A and Q denote the answer and query random variables, P(AIQ, 9) represents the target distribution of answer generation, denotes the parameter of the predictor f. Given the objective function in Eq. (1), we can complete the KBQA task by converting it to either classification problem (Sun al., 2019) or a generation problem (Saxena et al., 2022). Note that the used KG is typically large. Thus, the prediction is usually based on retrieved subgraph relevant to the query.
Different from the approach of directly generating answers, semantic parsing-basd based methods parse the input query into logical form (LF),, which is then executed over G to obtain the answers. This can be formalized as follows,
min
(2)
where M indicates the LF, P(M G) is the distribution of LF generation, 9 the function designed to generate LF parameterized by 6. In this regard, the answer is generated by executing the SPARQL query converted from the predicted LF M (MMM 9) against the KG,
à Execute(Convert(MM)
where i denotes the optimized parameter of g. Execute(.) is the query execution function, and Convert() represents the conversion function transforming LF M to SPARQL query. Similar to the direct generation approach, the semantic parsing approach usually predicts with retrieved subgraph relevant to the query the used KG is large (Yu et al., 2022; Luo et al., 2023b).
Given query Q, these two strategies retrieve question-reled relevant context C, leading to of reasoning paths Z. Subsequently, the answer A can be directly (2 //C - a) or indirectly (2 /C - m - a) produced with these reasoning paths or context. Here, the reasoning path E is formatted as
series of directed relations and the corresponding entities: 2 = 6g 1i 61 r2 _,,.... re, e with the i-th right_song,
entity e, E E and the i-th relation r; E R. For instance, The Baltimore Fight Song
Baltimore Ravens championahips, Super Bowl XXXV, Super Bowl XLVII means that "Baltimore Fight Song" is the fight song of the team "Baltimore Ravens", and "Baltimore Ravens" is the champion of "Super Bowl XXXV" and "Super Bowl XLVII""
3

---

Published as a conference paper at ICLR 2026
Z
A
M
Figure 2: Causal graph of the answer generation process in KBQA. Each node represents a random observable, where 9, Q, C, Z, A, M are knowledge graph, question, context, reasoning path, answer, and logical form, respectively.
METHOD
3.1 MOTIVATION
The KBQA task is challenging for LLMs (Luo al., 2023a;b) due the fact that completing KBQA requires complex reasoning. In this regard, one promising approach is to fine-tune LLMs to promote their performance. However, fine-tuning LLMs with limited data to accomplish a complex task would be challenging. To this end, we revisit the processes widely adopted to accomplish KBQA tasks, aiming to simplify the KBQA task for L.M fine-tuning.
For direct answer-generation approach, we can directly generate the answer by training a pre- dictor Le( AAQ, 9) to approach the target distribution P(A Q, , 9) using the objective function in Eq. (1). As it is challenging to fine-tune LLMs with limited data for complex tasks, we aim to simplify the task. In this regard, the factorization approach used in causal inference provides straightforward approach (Peters et al., 2017). The answer-generation process can be formalized by a causal graph (Peters et al., 2017). as depicted in Figure 2. Based on the graph, we can factorize the probability of generating answers P(A 9) using the independence between variables. Given the causal graph formalizing the answer-generation process, factorize the joint probability of all variables as follows,
PIQ.G.C,ZA = POPPPPPICCI
2),
(4)
where the input query Q and the knowledge base G are the causes of the context C' C CG P(CIQ.9)
denotes the mechanism of retrieving question-relevant context over the knowledge base, P(Z the mechanism to generate reasoning paths Z. and P(AIQ Z) stands for the mechanism of reasoning the answer based on the question and reasoning paths. Note that we omit the logical form random variable M when considering the direct answer generation approach.
According to the factorization above and using the chain rule of probabilities, we can rewrite the probability of generating answer P(A[Q, G) as follows?.
P(AQQ,9)
P(ZIC) P(AQQ,2) P(CIQ.9)
(5)
Z
reason
retrieve
where we marginalize over the context and the reasoning path Z. The factorization is intuitive. Specifically, we can reason about the answer using the query and reasoning path, i.e., P(A[Q, 2). This is based on the fact that we can reformulate the retrieved context into a specific type of reasoning path using P( P(Z|C). Here, the premise is that can retrieve query-relevant context from the knowledge base using the input query, i.e., P(CIQ,9).
approach, we can decompose the logical form (LF)-generation process using the causal graph in Figure 2 by omitting the answer A. Based on the graph, we can factorize P( (MMQ, 9), i.e., the probability of generating I.Fs. To this end, first factorize the joint probability of all variables as follows,
For logical
M, = PIO)P/
(6)
where the mechanism of retrieving vvestion-relevant context over the knowledge base P(CIQ.9) 9)
and the mechanism to generate reasoning paths Z using the context P(Z(C)) are the same as those
Here, consider the context and question discrete variables.
4

---

Published as a conference paper at ICLR 2026
under the direct answer-generation approach, and P(MIQ, Z) stands for the mechanism of reasoning the LF based on the question and reasoning paths. Then, we have an intuitive decomposition
P(MIQ,9) - Σ P(M 2) Σ PIZIC)
R
reformolate
According to Eq. (7), we can reason about LFs using P(MIQ, 2), where we reformulate the retrieved context into reasoning paths using P(ZIC). Here, the premise that we can retrieve query-relevant context using P(CIQ.9).
Factorization provides an alternative approach decomposing complex KBQA tasks into simple subtasks. Namely, we can decompose the objectives of these two strategies into three subtasks: reasoning, reformulating, and retrieving. Moreover, the objectives of reformulating and retrieving are the same for the direct answer-generation approach and the logical-form generation approach.
3.2 OBJECTIVE OF REASONING MODEL
LLMs have achieved exciting success in various reasoning tasks. Thus, we employ an open-source LLM to realize the reasoning model following previous work (Luo et al., 2023a;b). Inspired by previ- ous work (Yu et al., 2022), we encourage LLMs to generate answers and logical forms simultaneously. This also aligns well with our factorization perspective, these approaches share the same two sub-objectives, i.e., retrieving and reformulating.
According to Eq. (5), aim to construct reasoning model approach P(A[Q, by
min (L( (A)Q..
(8)
where 0, represents the learnable parameter of the reasoning model fo. to generate answers. Note that the number of the candidate values of A is large, making the task challenging. In this regard, propose to transform the multiple-class classification task into a binary classification task by grouping answer candidates/entities into one class with the of the candidates/entitis into the other class. This transforms the original answer A E A to the label Ya Z) E {0. 1}. where (Q Z) = 1 represents that the input pair (Q, Z) can derive the correct answer and Ya (Q,, Z) means the failure of deriving the correct answer. We can rewrite the objective function in Eq. (8) as follows,
min La (0,) (A(fo.(A(Q) Z), Ya (Q,
9,
where -'E(-,) denotes the cross- entsspy loss. As LLMs are trained for generation tasks rather than classification tasks, we map the label the token space, i.e., (Q, Z) is labeled as "yes" and "no" for Y, (Q, 2) 0.
For Le-gerrrttion approach, according to Eq. (7), we aim to construct a reasoning model to estimate P(M]Q, Z) by
min (L 2), fa. (M 2)),
(10)
8,
where 0, represents the learnable parameter of the reasoning model fa, to generate LFs. To fine-tune LLMs, we realize the objective in Eq. (10) using the widely adopted objective of next-token prediction with LF as the ground-trute label,
M
min I. (0,) A log II
Jo. M. Q, 2, Mo;
(11)
where we use M to represent M (Q, Z) for brevity and M; denotes the i-th token in the LF M. Thus, can fine-tune I.LMs construct reasoning model using the following objective,
min L, (0,) A (0,))) + Lm (0,).
(12)
3.3 OBJECTIVE OF THE REFORMULATING MODEL
According Eq. (5), we aim reformulate the context into reasoning paths approach P(Z[C) by min M., 10
(13)
5

---

Published as a conference paper at ICLR 2026
where C' denotes the retrieved context or subgraph and hw is the reformulating model parameterized with w. In KBQA, reformulating context into reasoning path can be realized in a handcrafted manner. Namely, we can realize P(Z(C)) in parameter-fre free scheme by designing a rule of transformation. For instance, we can design a template to reformulate the given context into a reasoning path. Consequently, we have K (T((P(Z (), h (Z (Z(C)) O, as enable the translation of context into reasoning paths, we propose a deterministic algorithm that operates without necessitating model training.
is an optimal solution. Thus, to
For instance, we can reformulate the context as follows (more details about the reasoning path template can be found in Appendix B.). Note that not all reasoning paths can lead to the answer. In the case of dense subgraphs, the reformulation step filters structurally invalid path combinations, reducing the exponential search space to linearly structured chains. For sparse subgraphs, imposing connectivity constraints through path construction still yields significant gains over directly feeding raw triples the L.M,, as confirmed by the ablation in Table 3 (+14.33% improvement from the "Only RP" variant).
Retrieved context:
(Penn State, contained by, State College). (Penn State, colors, Royal blue and white), (Beaver Stadium, contained by, University Park).
(Penn State, contains, Beaver Studium), (State College.contained by, Pennsylvania).
Reasoning paths:
21
Penn State
cont neduiy,
State College
pontt contailidddd
Pennsylvania; University Park.
22 Penn State contains,
Beaver Stadium consained.by,
3.4
OBJECTIVE OF THE RETRIEVING MODEL
According to Eq. (5), aim construct retriever approach P(CIQ.9) by min KL(P(C1Q,9) on, (C(1,))))
(14)
where C' denotes a subgraph of the structured knowledge base and de. is the retrieving model parameterized with 0z. The computational cost of traversing all subgraphs of large-scale knowledge base is prohibitive. Thus, we follow previous works to shrink the number of candidate contexts by filtering out entities and their corresponding relations that lead to large hop count (or path length). This results in a small-scale graph G.(Q), where Q represents the set constructed for the question Q.
Although filtering out large number of entities and tuples leads small-scale graph G(Q), the number of the rest of the entities could be large. Moreover, the small-scale graph G(Q) varies with the query Q. Thus, is challenging to train a classifier. In this regard, we propose to divide relations O in (() into two classes. Namely, we merge all answer-relevant relations into a single set" and merge the remaining relations into another class. Consequently, we can retrieve answer-relevant context by determining whether a relation is answer-relevant. Note that the retrieved relation and its corresponding subject and object entities are used to construct the set of answer-relevant contexts. Thus, we can rewrite the objective of the retrieving model
min CE( (Q, (2), Y,, 0.
(15)
where Ye (Q, E {0,1) stands for the label of the relation O. Namely, Y²(Q,0) = represents that the relation O is answer-relevant, while (Q, O) 0 means that the relation will not be included in the context. The retrieving task is a traditional binary classification task. Thus, we propose to train relatively small model, e.g., T5, rather than fine-tuning LLMSs to construct the retrieving model.
3.5 DISCUSSIONS
We find that RoG (Luo et al., 2023b) also formally employs the concept of reasoning path, which is regarded as a natural baseline of our method. Mathematically, by assuming that the generation of the answer A to the question Q is independent of the question-relevant context C' (Assumption A1)
3 For entity in the query, retrieval is completed by determining whether relation should be involved.

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of CoD with different baselines on the two KBQA datasets.
Backend Models
Model
WQSP
CWQ
Hissel I(
F1(
Hitter ((%))
F1(%)
TIARA (Sho et al,, 2022)
75:20
78:90
Non LLM
UnKGQQ (Jiang et al. 20123)
HGNNet (Chin at. 2022)
77.20
76.90
76.60
72.20
51.20
68.90
68.50
49.40
Zeso-shot
54.37
52.31
34.87
28.32
Pronsttngg LLM Ouly (GPT:335)
Few shot CoT (Wel et al, 2022)
56.33
53.12
43.21
35:25
57.42
54.72
43.21
35.85
Zero shot Few shot
02.32
59.71
42.71
37.95
Promsting LLM Only (GPT44
05:85
02:71
51.52
43.70
CoT (Wei et al., 2022)
72.11
65.37
53.51
44.76
Pronsttngg LLM KG (GPT333)
FiDeLastSui et. 2024)
ToG (Sun et 2023) 2023)
75.13
79.32
76.78
72.32
57.59
63.12
56.96
61.78
Prompting 1LM (GPT)))))
LaDeLaattSi et.al, 2024)
ToG (Sun et al., 20233
$1.84
$4.39
75.97
76:32
68.51
21.47
60.20
68.32
DeCAF (Yu es 2022)
92.10
70.42
Fineuning LLM + KG
KD CuT (Wang et al., 2022aa
RoG (Lan et al., 20226)
83.15
73:70
69.31
50.20
50.50
61.39
56.17
200 (ours)
N6.54
81.24
77.42
65.70
and the generation of reasoning paths Z is independent of the knowledge base 9 (Assumption A2), we show in Appendix C that our method intrinsically subsumes RoG as special exemplar of itself'. Methodologically, contrast to RoG which solves the optimization problem of Eq. (I) under the EM framework, where rigorous uniform assumption is explicitly imposed, our method, thanks to the reformulation of P(A(Q. 9) in Eqs. (5,7), can address the optimization problem of Eq. (1) by aligning each subterm in P(A|Q, 9) and fe(AQQ.9) as shown in Eqs. (13,14,15).
3.6 OVERVIEW AND IMPLEMENTATION
Before verifying the effectiveness of our CoD, we give detailed implementations to ensure the reproducibility of our experiments. We first fine-tune a small generative model Φe_, i... T5, to retrieve question-relevant context. Then, reformulate the retrieved context using hand-crafted and achieved by the function h2. Finally, we fine-tune LLMs fo, to generate logical form and identify whether the reasoning path derive correct answers. Namely, for a given input query Q = and a structured knowledge base G, our method CoD predicts by
rule, which is parameter-fi
A,
So. reasOn
hg
$0 (q,9),
(16)
reformulate
retrieve
where Á and M denote the set of the predicted answers and LFs. This is because CoD generates a pair of answers and LFs for each reasoning path. In this regard, we follow previous work to predict answers using executable LFs with the highest confidence. If the predicted LFs M is non-executable, the predicted answer is constructed using all tail entities of reasoning paths with a = "yes".
4 RELATED WORK
Knowledge retrieval in KBQA. In multi-hop KBQA scenarios, the goal of knowledge retrieval is to extract relevant documents from the knowledge graph that are relevant the given question. The process begins with initial topic entities, from which it is necessary to select relevant neighboring triples from large-scale KGs form path leading the answer entities. Traditional lexical models such as BM25 (Robertson et al., 2009) and PageRank (Sun et al., 2018; He et al., 2021) have been extensively employed KBQA. However, these methods often overlook the semanties of the question, potentially impairing the efficiency and accuracy of retrieval. Recent advancements in the field have seen the emergence of dense semantic retrieval models, including Dense Passage Retrieval(Karpukee et al., 2020), SimCSE (Gao al., 2021), and Contriever (Izacard et al., 2021), employ bi-encoder architectures to transform sentences into oow-dimensional dense vectors.
Knowledge reasoning in KBQA. The reasoning stage in KBQA focuses accurately identifying answer entities by traversing relations starting from the topic entities. Early approaches (Miller
WWhile these independence assumptions provide theoretical connections prior work, they serve analytical tools rather than operational requirements; the effectiveness of CoD is ultimately validated empirically.
7

---

Published as a conference paper at ICLR 2026
et al., 2016; Sun al., 2018; 2019; Jiang et al., 2022) employ specialized network architectures such as Key-Vate Value Memory Networks to model the multi-hop reasoning process. Recent works have proposed parsing questions into structured query languages (c.g., logical forms (LFs)) and executing them via query engines to obtain answers (Lan et al., 2019; Das et al., 2021; ; Huang et al., 2021). These methods typically encoder-decoder architectures, such T5 (Raffel al., 2020), to generate structured queries. In the area of LLMs, DECAF (Yu et al., 2022) combines LF and LLMs reasoning jointly generate answers. ChatKBQA (Luo al., 2023a) generates logical forms through fine-tuning LLMs based solely on the input question, and obtains the final answers through query engines. The generated logical forms may be inaccurate. To effectively utilize the path information of knowledge graphs, some studies use subgraphs obtained through retrieval for reasoning. TOG (Sun et al., 2023) leverages the reasoning capabilities of open-source L.M,, iteratively exploring various possible reasoning paths on the KG through prompt tuning, until the LLM determines that it can answer the question based on the current reasoning path. ROG (Luo et al., 2023b) proposes planning retrieval reasoning framework that synergizes LLM to achieve faithful and explainable reasoning. More details can be found Appendix E.
EXPERIMENTS
5.1
SETUP
Datasets. We conduct experiments on two benchmark KBQA datasets: WebQuestionSP (WehQSP) (Yih et al., 2016) and ComplexWebuuesttine Freebase KB reasoning. The former contains 4,737 simple natural language questions with SPARQL queries, while the latter contains 34,689 more complex questions with SPARQL queries. Training and testing splits of each dataset follow the same setting as those of previous work (Yu et al., 2022).
(CWQ) (Talmor & Berant, 2018), both available for
Table Perfor-
mance compari- son of different
Method
WebQSP
CWQ
top5-acc
top10-acc
top20-acc
top5-acc
10p10-acc
top20-acc
retrieval models DPR (Karpukhin et al., 2020)
35.02
44.17
53.87
10.24
17.90
28.14
on the two KBQA BM25 (Robertson al., 2009)
15.25
20.50
22.70
2.77
6,59
11,95
Sentence-BEi CcD (ours)
30.69
43.32
55.83
9.13
17.10
32.82
datasets.
91.04
95.00
96.77
89.93
92.82
95.11
Evaluation Metrics. To obtain fair comparison, we adopt the F1 score and Hits@1 evaluate the coverage of all answers and the single top-ranked answer, respectively, following previous work (Shu et al., 2022; Yu et al., 2022). In terms of retriever comparison, top-k accuracy is utilized to demonstrate whether the correct reasoning paths searched within the top-k retrieval results.
Baselines. The basclines include: Non-LLMs- method, Prompting-LLae LLM-based method, and Finetuning- LLL---LM--baaa method. Below are the details. (1) Non-LLMs based method. TIARA (Shu et al., 2022) improves question answering by using multi-grained retrieval and constrained decoding to enhance the accuracy and robustness of KBQA. UniKGQA (Jiang et al., 2023) unifies the retrieval and reasoning in both model architecture and parameter learning. HGNet (Chen et al., 2022) proposes hierarchical query graph generation approach improve the performance of KBQA. (2) Frompting-LLM-boee -based method. ToG (Sun et al., 2023) involves prompting the LLM to iteratively explore various potential reasoning paths within KGs until concludes that the question can be answered based on the retrieved path. FiDeLis (Sui et al., 2024) proposes a sarrivallexpllratee method that incorporates knowledge graphs into I.LMs, exploiting their deductive reasoning abilities to enhance knowledge retrieval and reasoning. (3) Finetuning- LLM based method. KD-CoT (Wang et al., 2023a) extracts relevant knowledge from KGs to create accurate reasoning paths for LLMs. DeCAF (Yu et al., 2022) integrates semantic parsing with LLM reasoning to jointly produce answers, achieving notable performance in KGQA tasks. RoG (Luo et al., 2023b) incorporates structural knowledge from knowledge graphs (KGs) into neural networks during pretraining and fine-tuning We use Llama2-7B 7B as the base LLM, with batch size of 4 and a learning rate of 5e -5 More details can be found in Appendix F.
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance evaluation of the pro- posed CoD with differ- ent hyperparameters, i.e., beam sizes.
F WQSP
F1% CWQ
Beam Size
Only L.
Only RP
CoD
Only L.F
Only RP
CoD
bm=1
71.60
74.21
75.56
44.55
45.65
62.13
bmm2
74.46
76.67
78.37
47.48
50.09
65.83
bm=3
75.04
77.26
80.87
48.25
50.67
68.27
bm=5
79.60
78.62
81.24
49.40
51.37
65.70
5.2 MAIN RESULTS
Table illustrates the comparative performance of our method against various baseline methods. CoD improves @its@1 and Fl by over 2% and 2% on the WebQSP dataset and by over 5% and 1% on the CWQ dataset, respectively, compared to the previous optimal methods. The data clearly indicate that our method surpasses all baselines, even those enhanced with GPT-4-turbo, including strong contenders, like DeCAF and FiDeLis. Among all methods based prompting. GPT-4-turbo consistently delivers superior results compared to GPT-3.5-turke particularly on the CWQ dataset, suggesting its enhanced ability to comprehend and process complex queries.
Among them, the non-LMs- based methods have limited performance improvements due to their limited reasoning capabilities. The Prompting- LLLM LLM Only-based methods have poor performance due to the lack of knowledge graphs. For the methods based on Prompting-L LLMM + KG and Finetuning- LLM + KG, RoG combines LLM with KG achieve faithful and explainable reasoning. FiDeLLi uses the retrieval-explon www.eeoattooninnerrreeee
framework to enable large models to perform KBQA in deductively verified manner, making full use of the advantages of LLM in logical reasoning, and achieves state-of-the performance. However, this kind of method may introduce too many paths, resulting in limited performance improvement. Our method effectively improves the accuracy of knowledge graph retrieval by designing an enhanced retriever.
5.3 RETRIEVERS PERFORMANCE COMPARISON
To verify the effectiveness of the proposed retriever, conducted the experiments as shown in Ta- ble 2. It can be seen that our method is significantly higher than DPR, BM25, and Sentence- BERT. In other words, when per- forming knowledge reasoning, our method only needs to retrieve a small number of paths to fully include the required reasoning paths, effectively reducing the impact of unnecessary paths the performance of the model in the reasoning stage. Addition- ally, we paired CoD with differ- retrievers. The corresponding results and analysis are presented in Appendix D.
score(%)
H(%@1(%)
100
100
odl
60
60
2012
20
20
WOSP
CWQ
WQSP
CWQ
Figure 3: The performance of CoD with only generated answers (indicated as Ans. only) or LF executed answers (indicated L.F only).
5.4 ABLATION STUDIES
Beam size decides the maximum number of candidate outputs remaining after beam search. We conduct experiments to verify its impact on model performance. As shown Table 3, with the increase in beam size, the performance of the model gradually improves, and when bm=3, the performance gradually stabilizes. This suggests that, when generating answers through beam search, LLM tends to provide the correct answer within the top-generated responses. To verify the significance of the combination of LF-executed answers and generated answers CoD, we conducted ablation studies with results in Figure when bm=5, where the performance of joint utilization surpasses that of either the answer-only or LF-only approaches. The effectiveness is more distinct for the CWQ dataset, indicating robustness when encountering complicated KB questions. Additionally,

---

Published as a conference paper at ICLR 2026
experiments on the impact of different prompt templates and robustness analysis presented in Appendix D. To further evaluate generalization, we test CoD on GrailQA across ii.d., compositional, and zero-shot settings. CoD achieves 78.2% overall FI, outperforming DECAF (76.0%) by 2.2 percentage points, with consistent gains in compositional (+0.9%) and zero-shot (+1.3%) splits, demonstrating that the decomposition strategy transfers robustly across different distribution shifts.
CONCLUSION
In this paper, we present Chain-of- Decompoosiiiii (CoD), a novel framework that addresses the chal- lenges of adapting large language models (LLMs) to Knowledge Base Question Answering (KBQA). Based on our causal graph, we factorize the answer-generation process and isolate computation- heavy retrieval and rule-based reformulation from LLM reasoning. We decompose KBQA into three modular steps an LLM-frree retrieval module, a parameter-free reformulation step, and lightweight LL-M-based reasoning module- siinisicinnnillyy reduce task complexity and enable efficient fine-tuning of LLMs. Experiments on benchmarks demonstrate that Llama-2 7B fine-tuned with CoD outperforms strong baselines, including GPT-4 augmented with retrieved knowledge, achieving state-of-the-ar performance. The efficacy of the proposed CoD highlights the importance of task decomposition enhancing LLM reasoning capabilities for complex reasoning tasks.
I Limitation. While CoD achieves notable advancements, it has limitations. We solely employ Llama- 2 7B as the base LLM, overlooking more advanced models like Llama-3 (Grattafiori et al., 2024), QWen, and DeepSeek-RI (Guo et al., 2025). Exploring these advanced LLMs to further improve CoD's '' performance a promising direction for future work.
ACKNOWLEDGMENTS
We thank the reviewers and area chairs for their valuable and constructive comments, which have significantly improved the quality of this paper.
REPRODUCIBILITY STATEMENT
We outline the pipeline of our proposed method in Fig. I and provide implementation details in Appendix B. Key data and settings are also included in Appendix A for reference.
REFERENCES
K. Bollacker. Freebase A collaboratively created graph database for structuring human knowledge. Proc. SIGMOD' 08, 2008.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, fass/2055141663 2020. URL https: : aawiviiiiiv org/ abs /2005.00... 14165.
Yongrui Chen, Huiying Li, Guilin Qi, Tianxing Wu, and Tenggou Wang. Outlining and filling: hierarchical query graph generation for answering complex questions over knowledge graphs. IEEE Transactions Knowledge and Data Engineering, 3548)83333-3357 2022.
Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay- Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. Case- reasoning for natural language queries over knowledge bases. arXiv preprini urXiv:2 1XX.::2104.08722. 2021.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv: 20Xiv:2104.0882/ 2021.
10

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arX1v:240..21783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprins arXiv:2501.12945 2025.
Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In Proceedings of the 14th ACM international conference on web search and data mining. pp. 553-561, 2021.
Xin Huang, Jung Jae Kim, and Bowei Zou. Unseen entity handling in complex question answering over knowledge base via language generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 547-557, 2021.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint
18, 2021.
Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. Great truths always simple: A rather simple knowledge encoder for enhancing the commonsense reasoning capacity of pre-trained models. arXiv preprint arXiv:2205.013: 01841, 2022.
Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji-R Ji-Rong Wen. Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: wppprppeeeviieeeeeeeeeeeeeee net/ wow.mmmd==63RvyaAA...
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04400 04906, 2020.
Yunshi Lan, Shuohang Wang, and Jing Jiang. Knowledge base question answering with topic units. 2019.
Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Lidong Bing, Shafiq Joty, and Soujanya Poria. Chain of knowledge: A framework for grounding large language models with structured knowledge bases. arXiv preprint 230.::22005.3269, 3, 2023.
Haoran Luo, Zichen Tang, Shiyao Peng. Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, Wei Lin, et al. Chatkbqa: A generate-then-retriew framework for knowledge base question answering with fine-tuned large language models. arXiv preprint arXiv:2310008975. 2023a.
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310.01061, 2023b.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Ja- son Weston. Key-value memory networks for directly reading documents. arXiv preprint arXiv: 16.61666033222 126, 2016.
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. Unik-qa: Unified representations of structured and unstructured knowledge for open-domain question answering. arXiv preprint arXiv:2012.14610, 2020.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08874,, 2023. doi: 10.4550AARX1..2333.677744 URL httttt:/s:/dde .org/100000 48550/ www.v 0..
2303.
11

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022a. URL http: / http:/aaaprrrseeeeeeeeeee nips. cc cc/paper_files. files/ htppppppr/////// 2002 /hash/ 4 oat.a4411111110000001---aaaaaaaa
html.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Anmual Conference Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022b. URL http: http://aapppeeeeeeeeeeeee nips - tc/ http:/pppr__feeeeeee htppp/appp/////////epeeeeaee 2202 /hash/ blefde53be 6 4 7 010.0b66667711111000000000000000 -ttp:aaatootaatataattt-------aoa Cow.eetteeeooeeeeeeenecneee html.
Shirui Pan, Linhao Luo. Yufei Wang, Chen Chen, Jiapu Wang. and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Trans. Knowl. Data Eng., 36(7): 3580-3599, 2024. doi: 10.110997KKD TKDE . 2024 3002233......
http//ooiooogg/////////o/ org. 1109/ 1109/ https:/ //tp:::///d...//
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21((10): 1-1111667662 2020.
Stephen Robertson, Hugo Zaragoza, al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 344):333-389 2009.
Apoorv Saxena, Adrian Kochsick, and Rainer Gemulla. Sequencceto--sequence knowledge graph completion and question answering. arXiv preprint xrX1v:2203.
Yiheng Shu, Zhiwei Yu, Yuhan Li, Börje Karlsson, Tingting Ma, Yuzhong Qu, and Chin Yew Lin. Tiara: Multi julti-grained retrieval for robust question answering over large knowledge bases. arXiv preprint arxiv:2210. 12.2225....22........ 2022.
Yuan Sui, Yufei He, Nian Liu, Xiaoxin He, Kun Wang, and Bryan Hooi. Fidelis: Faithful reasoning in large language model for knowledge graph question answering. arXiv preprint 33873 2024.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W Cohen. Open domain question answering using early fusion of knowledge bases and text. arXiv preprini
2018.
Haitian Sun, Tania Bedrax- Weiss, and William W Cohen. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. arXiv preprint arXiv:1904.095. 095377 2019.
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang. Chen Lin, Yeyun Gong, Heung-Yeue Yeung Shum, and Jian Guo. Think--n-graan on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.0766
2023.
Alon Talmor and Jonathan Berant. The web as a knowledge-bert base for answering complex questions. 2018.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307. 092888 2023.
12

---

Published as a conference paper at ICLR 2026
Keheng Wang, Feiyu Duan, Sirui Wang. Peiguang Li. Yumsen Xian, Chuantao Yin, Wenge Rong. and Zhang Xiong. Knowledge-driven cot: Exploring faithful reasoning in llms for knowledge nnowledge-intessvee intensive question answering. arXiv preprint arXiv:2308. 13259, 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL https: /ttp:///ee/peeeeee....e. forum? d= PI. N.M..W.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference
Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 2022, 2022. URL http: htttp::/paper....tt nips ec cc/paper_fiileeecceeeeeeeeeeppee files / htpppaappp////////ee 2022 /haah/ Abstrrrtaatttaaacaaaaaattet Conference htmll.
Abstract
Yike Wu, Nan Hu, Guilin Qi, Sheng Bi, Jie Ren, Anhuan Xie, and Wei Song. Retrieve-rewrite- kg-to-text enhanced iims framework for knowledge graph question answering. arXiv preprint arXiv:2309. 1206, 2023.
Wen Tau Yih, Matthew Richardson, Chris Meek, Ming Wei Chang, and Jina Suh. The value of semantic parse labeling for knowledge base question answering. 2016.
Donghnn Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Wang, Zhiguo Wang, and Bing Xiang. Decaf: Joint decoding of answers and logical forms for question answering over knowledge bases. arXiv preprint 2rX1v:2210.00063, 2022.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A SETTING OF KNOWLEDGE BASE QUESTION ANSWERING
Knowledge base question answering (KBQA) is complex task that requires factual knowledge and deep reasoning to accomplish accurately. In KBQA, factual knowledge typically is the form of set of triples, i.e., subject entity, object entity, and the relation of these two entities. The deep reasoning refers to the fact that the input query typically requires complicated multi-hop reasoning. Namely, on the knowledge graph G, the answer entities are multiple hops away from the topic entities mentioned in the natural language query.
There mainly two datasets used in the literature, i.e., WebQuestionSP (WebQSP) (Yih al., 2016) and Complex WebQuestions (CWQ) (Talmor & Berant, 2018). These two datasets result from the Freebase (Bollacker, 2008) that contains around 88 million entities, 20 thousand relations, and 126 million triples. The data split strategy is the same as previous work (Sun et al., 2018; Luo et al., 2023b). Specifically, 2, 826 samples are used for training, while 1, 628 used for testing on the WebQSP dataset. For CWQ, we leverage 27, 639 samples for model training and 3, 531 for testing. Here, the sample comprises pair of the input query and the corresponding answers".
These two datasets contain up 4-hop questions, which require deep reasoning. The deep reasoning challenging even for large language models (LLMs). For instance,
Query:
Which book written by the author who married the actress starring in the movie directed by the director who won Oscar for movie starring Tom Hanks?
Multi-hop reasoning:
Tom Hanks atarcod.in
Forrest Gump
directedLay
Robert Zemeckis
narrled.to
Richard Matheson auther.os,
I Am Legend
We can see that 4-hop question is challenging even for humans.
B TEMPLATE DESIGN OF REASONING PATH
According to our factorization, we show that the reformulating model could be handcrafted rules, leading to a parameter- free algorithm. Namely, we can design a template to convert the retrieved context into the required reasoning paths. Here, provide details for the template design.
For the retrieved context, i.e., a set of triples, we group triples into four classes according to the number of hops. Then, determine a series of candidate reasoning paths based the principle of whether the head entity in the i-th hop is consistent with the tail entity in the i + 1-th hop. The overview of the reformulating model can be found in Algorithm 1.
Algorithm Extract Reasoning Path for Input Query
1: procedure EXTRACT REASON PATH(dala)
2: L dala[" (tripledss)
3: # Link the triplets
4: linkedd_ (RI.L.TIPEETS(E)
5:
# GGnerate reasoning paths
6:
reason-p GENERATE (ATH((iell)))))
7:
# Replace entity indexes with entity names
WETERRATEETTTLEEEEEEEEE 9: return new-path
8:
new -palh GENERATE. WITH_ WATTLNNMM(rreson.,
end procedure
'In these two datasets, the SPARQI query is also provided. Thus, we convert the SPARQL of each input query into logical forms.
14

---

Published as a conference paper at ICLR 2026
In experiments, the retrieval model can filter out most irrelevant relations. Namely, we have limited number of tuples in each group with the same hop. Moreover, the number of tuples that link head-to-tail entities between different groups is also usually limited. Thus, the reformulating model typically results in a limited number of reasoning paths, i.e., less than 10.
C MATHEMATICAL CONNECTION TO RoG
We construct general causal graph by rethinking the widely adopted approaches to knowledge base question answering (KBQA). Namely, the causal graph formulates the generation process of answer A or logical form M for given input query Q. Our factorization approach results from the general causal graph. Thus, our factorization general approach that could be realized using different assumptions.
We show that, under some assumptions, a recent outstanding approach, RoG (Luo al., 2023b), for KBQA is a special case of our factorization. To clarify the connection, we first list two assumptions used in RoG.
Assumption 1. When conditioning on the context Z, the generation of the answer A to the query Q is independent of the question-relevant context C.
This assumption is intuitive, as the query and reasoning path can determine the answer. Assumption The generation of reasoning paths Z is independent of the knowledge base conditioning on the input query Q.
However, this assumption could be violated in many practical scenarios. In particular, the reasoning path is based on the knowledge graph (or knowledge base) G. Namely, the reasoning path usually varies with the utilized knowledge graph G. Thus, the assumption could be relatively strong for many real-world scenarios.
Based on these two assumptions, we can show that our factorization Σn P(AQQ, 2) E. P(ZIC)P(CIQQ.)) 9) gives general formulation and the objective used in RoG Σz P(A(Q,C,2)P(Z1Q)) is special case of our factorization.
Σ P(AIQ, Z) Σ P(ZIC)P(CIQ.9)
-P(AQQ,9)
(17)
= =Σ P(A7,009))
(18)
Z C
(19)
Σ
(20)
ΣΣ
(21)
Σ
D PLAII,S..)P(CCO0G.033
(22)
Z
C
Σ
Σ P(CIQQ.9.2) P(A(I,)
(23)
(24)
where Eq. (21) holds based on Assumption 1, Eq. (22) holds based on the Assumption 2, and Eq. (24) exactly the learning objective RoG (Luo et al., 2023b). Thus, RoG a special case of our factorization under Assumptions and 2.
15

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
In this section, further present the experimental results and in-depth analysis for additional research questions.
Robustness analysis. To evaluate the robustness of the proposed CoD method, conduct ex- periments using different backbone LLMs. As shown in Table 4, we extend our evaluation to two additional base LLMs: Llama3-8B and Qwen2.5-7B. These models are fine-tuned within the CoD framework and evaluated on two knowledge- based question answering (KBQA) benchmarks, namely WebQuestionsSP (WQSP) and ComplexWeh WebQuestions (CWQ). The results indicate that CoD main- tains robust performance across different LLMs, underscoring its adaptability and generalizability.
Table 4: The performance of CoD across different backbone LLMs on the WQSP and CWQ datasets.
WQSP
CWQ
Model
F1
Hits @
F1
Hits @
Llama2 7B
81.24
86.54
65.7
77.42
Llama3 8B
81.03
86.29
79.61
79.33
Qwen2.5 7B
80.39
85.42
66.53
77.85
Table 5: The impact of various retrievers on the performance of the CoD model on the WQSP and CWQ datasets.
WQSP
CWQ
Model
F1
Hits
F1
Hits @ 1
CoD (DPR)
78.71
83.37
60.13
73.15
CoD (BM25)
77.93
81.33
59.52
71.83
CoD (SentenceBERT)
83.52
60.62
66.53
74.02
CoD (Ours)
80.39
86.54
65.70
77.42
Table 6: The impact of different prompt templates on the performance of the CoD model.
WQSP
CWQ
Model
Hits@
Hits@
CoD (Direct Generaton)
77.53
82.52
62.77
75.16
CoD (CoT)
79.36
86.15
64.94
76.39
CoD (Ours)
81.24
86.54
65.70
77.42
The impact of different retrievers. To investigate the impact of different retrievers on the perfor- mance of the CoD model, conduct experiments as shown in Table 5. The results demonstrate that CoD achieves promising performance when using our specially designed retriever. This finding underscores the importance of accurate retrieval paths for the effectiveness our approach. The com- parative analysis with other popular retrievers (DPR, BM25, and Sentence-BERRT further validates the necessity of our retrieval component design.
The impact of different prompt templates. To investigate the impact of different prompt templates on the CoD model's performance, we conduct experiments as shown in Table 6, we evaluate three answer generation templates using the retrieved reasoning paths: direct generation, Chain-of-Thowe (CoT), and our proposed deductive reasoning approach. The choice of prompt template indeed has a impact the performance of the CoD method. Both reasoning approach outperform direct generation. Furthermore, our method demonstrates slightly superior performance to CoT and exhibits better robustness and stability across different templates.
(CoT) and our deductive
16

---

Published as a conference paper at ICLR 2026
E ADDITIONAL RELATED WROK
To effectively utilize the path information of knowledge graphs, some studies use subgraphs obtained through retrieval for reasoning. PullINet (Sun et al., 2019) retrieves relevant subgraph from the KB and uses graph neural networks to predict answer entities within these subgraphs. UniKGQA (Oguz al., 2020) unifies the knowledge base retrieval and reasoning processes into single model with PLM, achieving SOTA performance. (Li et al., 2023) break down questions into subquestions and generate SPARQL queries using fine-tuned llama model (Touvron et al., 2023) to fetch knowledge from KGs. (Wu et al., 2023) make full use of the retrieved knowledge of the knowledge graph through steps such as Rewreteeeeerrtt---eee
and fine-tuning the large model to obtain the final answer.
F IMPLEMENTATION DETAILS
All experiments are conducted on single NVIDIA A40 GPU. For the retrieval module, we fine-tune T5-base model using the AdamW optimizer (8) =0,9, 32=0.999) with a learning rate of 2e-5 and batch size 4, training for 100 epochs on both WebQSP and CWQ. which takes approximately hours and 8 hours respectively. For the reasoning module, fine-tune (rank=8, a=16) with a learning rate of 5e-5, batch size of 4, and a maximum sequence length of 2048. The reasoning module trained for 100 epochs WebQSP ( 12 hours) and 5 epochs CWQ ( 24 hours).
LLaMA-2-7B using LoRA
17


Reviews:
[{"review_id": "FoOQ5XgPso", "reviewer": "Reviewer_Ebcb", "summary": "The paper introduces Chain-of-Decomposition (CoD), a novel framework that enhances large language models (LLMs) for Knowledge Base Question Answering (KBQA) by decomposing the reasoning process into three modular components: (1) an LLM-free retrieval module that extracts query-relevant subgraphs, (2) a parameter-free reformulation step that converts retrieved contexts into structured reasoning paths, and (3) a lightweight LLM-based reasoning module to validate logical consistency. Grounded in a causal factorization of the answer-generation process, CoD isolates computationally intensive retrieval and rule-based reasoning from LLM fine-tuning, substantially improving efficiency and interpretability. Experiments on WebQSP and ComplexWebQuestions show that Llama-2 7B + CoD achieves state-of-the-art performance, surpassing GPT-4 with retrieved knowledge and previous methods like RoG and FiDeLis.", "questions": "See the Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. CoD innovatively formalizes KBQA through a causal graph–based factorization, offering a principled decomposition of retrieval, reformulation, and reasoning.\n2. By decoupling heavy retrieval from LLM reasoning, CoD reduces computational cost and enables lightweight fine-tuning focused on logical verification.\n3. CoD significantly outperforms GPT-4 + retrieval and previous SOTA frameworks (e.g., RoG, FiDeLis) on WebQSP and CWQ, showing broad robustness.\n4. The authors provide clear mathematical formulation, implementation details, and reproducibility statements, enhancing credibility and clarity.", "weaknesses": "1. The independence assumptions in the causal graph (e.g., P(A|Q,Z) being independent of C) are difficult to satisfy in real-world knowledge graph reasoning, which may undermine the theoretical rigor of the framework.\n2. The retrieval, reformulation, and reasoning modules are trained separately without a unified loss function or gradient propagation, potentially leading to suboptimal global optimization.\n3. Although the paper emphasizes logical path validation, it lacks visualizations or path-level case analyses; the ablation studies mainly focus on hyperparameters rather than validating the effectiveness of the causal decomposition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gX03RyGRcR", "reviewer": "Reviewer_FkYc", "summary": "This paper proposes Chain-of-Decomposition (CoD), a framework for Knowledge Base Question Answering (KBQA) that decomposes KBQA into three modular steps: 1) Retrieval: an LLM-free module (using T5) that extracts relevant subgraphs from a knowledge base; 2) Reformulation: a rule-based transformation that converts subgraphs into reasoning paths; and 3) Reasoning: a lightweight LLM-based verifier that determines whether a reasoning path is logically valid. The authors proposes a causal graph (Figure 2) that formalizes the generative process of answers, allowing the authors to express the conditional probability $P(A | Q,G)$ as a sum over retrieval, reformulation, and reasoning components. Empiricially, CoD outperforms GPT-4 and prior fine-tuning frameworks such as ROG, FiDeLis, and DeCAF on two benchmarks: WebQSPand ComplexWebQuestions, achieving new SOTA performance.", "questions": "[Q1] I had some difficulty understanding the integration of logical form and binary \"yes/no\" prediction. How does the LLM-based reasoning module integrates logical form generation and binary \"yes/no\" prediction, are both outputs produced jointly or alternately?\n\n[Q2] What are some potential challenges and risks to transfer CoD to larger LLMs and more complex KBs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "[S1] Simple-but-effective decomposition of the workflow. I think there is a lot of value in finding simple yet competitive approaches as proposed herein as regularizers for future methodologies. The idea of isolating retrieval and rule-based steps from reasoning reduces computational burden and is shown through emprical evidences to be effective. The graph formulation and factorization (Fig 2, Eq. 5, 7) also sets a clear stage for understanding the problem map out. \n\n[S2] Potential efficiency benefits: Two of three stages are LLM-free, and the actual reasoning component is relatively small. This will reduce training as well as inferencing cost, potentially beneficial in real-world scenarios. \n\n[S3] Strong empirical results: The method is shown to induce significant improvements over finetuned baselines as well as GPT-4 + KG on two benchmarks.", "weaknesses": "[W1] Evaluation scope: Only WebQSP and CWQ are used; inclusion of more diverse KBQA datasets (e.g., GrailQA/GrailQA++, GraphQ) would strengthen generalizability. One related comment is: the authors mentioned MetaQA in their Abstract, yet MetaQA results are not reported in the main content. While this might be a typo, I was curious what would happen if the proposed method is applied on MetaQA. \n\n[W2] The presentation can be improved. E.g., The bottom-right portion of Figure 1 is underspecified, the symbols in the logic forms appear out of context, making them a bit hard to interpret; The causal interpretation may be overstated, the method is more of a probabilistic modularization than genuine causal analysis/reasoning, and the assumption 2 (independence between Z and G) seems a bit not straightforward. I would recommend revising the presentation narrative of the logical relationship between the RVs. \n\n[W3] Language and structure: Some grammatical and stylistic issues (e.g., \"dramatically outperform\" should be \"drastically outperforms\"; \"lead to a simple binary task problem1\" the footnote seems a bit unnecessary. I would recommend a round of thorough reading to eliminate such artifacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LnBcYyIROi", "reviewer": "Reviewer_HJHf", "summary": "This paper proposes a KG-RAG (called KBQA by authors) method based on reasoning paths generated through a deterministic algorithm, with answers subsequently produced by LLMs. However, the paper’s writing is quite poor and difficult to follow. For example, there is a lack of explanation of the logical form introduced in Figure 1 and Section 2.2, weak motivation in Section 3.1, and an unclear description of the graph simplification process in Section 3.4.", "questions": "1. In Equation (1), is $P(A|Q,Z)$ means the ground-truth distribution?  \n   Why use a probabilistic formulation instead of the ground-truth answer directly?\n \n2. In lines 177–214, do the authors mean the decomposition of $f_\\theta$ instead of $P$?  \n \n3. In Section 3.2, the output of $f_\\theta$ is ambiguous—it seems to represent classification in Equation (9) but generation in the equation (11). This should be clarified to avoid confusion.  \n   For example, do the authors mean $g_\\theta$ in Equation (11)? Furthermore, the $f_\\theta$ defined in Section 3.2 seems not consistent with the one in Section 2.2.\n \n4. In Section 3.3, the complexity and necessity of reasoning paths are not discussed. If the extracted triples form a dense structure, the number of possible paths could grow exponentially. Conversely, when the extracted triples are few or sparse, path construction might be unnecessary—one could simply input all triples into the language model.\n \n5. The benefits compared to other methods that also use paths, such as RoG and ToG, should be explained more clearly. The current comparison with RoG in Section 3.5 is confusing:  Why should we assume that “the generation of answer A to question Q is independent of the question-relevant context C”? This assumption completely contradicts the purpose of KG-RAG. The same concern applies to the second assumption in line 310.\n \n6. In Section 3.4, the process for simplifying the input graph is entirely unclear:  \n   - (1) The graph-shrinking procedure is not well defined or justified, and there is no guarantee for how well it preserves relevant-information to the query.  \n   - (2) The simplification step that merges relations into two classes is confusing, as the answer-related relation should not be known for the input data.\n \n7. Since the extraction process in Section 3.4 directly feeds the graph into an LLM, why not train the LLM to generate the answer directly instead of producing intermediate outputs?\n \n8. In Table 1, the last column suggests that the F1 score of **HGNet** should be the highest. Can you clarify this inconsistency?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "-Claims improved performance on two widely used benchmark datasets.", "weaknesses": "1. The overall writing quality needs substantial improvement. The meanings of the notations $f_\\theta$, $g_\\theta$, and $P$ are not clearly explained and are used inconsistently throughout the paper (see Questions 1–3).  \n   In particular, the extraction process—where the reduced graph is directly input into an LLM—is poorly explained and unconvincing (see Questions 6–7 below).\n \n2. The paper lacks discussion and comparison with recent representative works, such as ToG [1].\n\n  **Reference:**  \n   [1] *Think-on-Graph: Deep and Responsible Reasoning of Large Language Models with Knowledge Graphs.* ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tta3xh4R4G", "reviewer": "Reviewer_viyW", "summary": "This paper demonstrates the theoretical equivalence between two paradigms in knowledge base question answering (KBQA): directly generating the answer versus generating a logical form and then executing it. Based on this observation, the authors propose CoD, a \"Retrieve-Construct-Answer\" KBQA framework. By decomposing the KBQA task into distinct stages, CoD reduces the input burden of large-scale knowledge graphs on LLMs and improves reasoning and answering performance.", "questions": "1. The appendix mentions that CoD is a generalization of RoG, yet the experimental results do not reflect a substantial performance difference to support this claim. Furthermore, in Table 1, the RoG row under the same backbone LLM (LLaMA-2-7B) shows consistently lower numbers compared to those reported in the original RoG paper. Have the authors analyzed the reasons for this discrepancy?\n\n2. The model is fine-tuned using WebQuestionsSP and ComplexWebQuestions datasets. How well does this generalize to other benchmarks? Specifically, how does CoD—and particularly its comparison to RoG—perform on GrailQA, GraphQ, or other datasets (e.g., those used in KBQA-o1)?\n\n3. The experiments provided may not fully demonstrate the advantages of CoD:\n\n    (1) In Appendix D, the results indicate a clear performance gain when using LLaMA-3 8B over LLaMA-2 7B on CWQ. Would newer open-source models such as LLaMA-4 17B, Qwen3 8B, or Qwen3-30A3B offer further improvements, especially given their resolution of known issues in earlier models?\n\n    (2) One widely adopted and competitive solution for KBQA is GraphRAG. Could the authors articulate CoD's advantages over GraphRAG-style systems (e.g., HippoRAG)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The motivation of the paper is clearly presented. The authors rigorously derive the mathematical formulation of both KBQA paradigms and prove that they share the same optimization objective for the first two steps. This theoretical insight provides a solid foundation for the introduction of CoD and establishes a clear correspondence between the two approaches.", "weaknesses": "1. The overall presentation and figure design could be improved. First, Figure 1 is somewhat confusing: the answer to the question \"Which country is Elon Musk from?\" should be South Africa, corresponding to Path 4 in the figure. It is unclear why an incorrect reasoning path is shown in the framework's overview figure. Second, the captions of Tables 2 and 3 are aligned to the left side of the table content, which appears awkward.\n\n2. Although CoD involves fine-tuning a language model, the paper does not disclose details of the training process.\n\n3. The paper lacks further analysis of the experimental results. For instance, identifying bottlenecks of CoD through ablation or diagnostic studies (though this may require additional experimental conditions) would provide valuable insights.\n\n4. There are a number of typographical and grammatical issues throughout the paper:\n\n    (1) In Figure 1, the prompt contains \"it no,\" which should be corrected to \"if no,\" and the quote marks around \"return no\" are incorrect.\n\n    (2) The acronyms ToG and RoG are inconsistently capitalized as TOG and ROG in several instances.\n\n    (3) In Appendix E, line 5, \"STOA\" should be \"SOTA\".\n\n    (4) The phrase \"reason paths\" appears multiple times and should be corrected to \"reasoning paths\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)