From Parse-Execute to Parse-Execute-Refine: Improving Semantic Parser for Complex Question Answering over Knowledge Base
Authors: Wangzhen Guo
Linyin Luo
Hanjiang Lai
Jian Yin
Venue: EMNLP 2023 Main (2023)
Keywords: KBQA
Logical Form
BART

Abstract: Parsing questions into executable logical forms has showed impressive results for knowledge-base question answering (KBQA). However, complex KBQA is a more challenging task that requires to perform complex multi-step reasoning. Recently, a new semantic parser called KoPL has been proposed to explicitly model the reasoning processes, which achieved the state-of-the-art on complex KBQA. In this paper, we further explore how to unlock the reasoning ability of semantic parsers by a simple proposed parse-execute-refine paradigm. We refine and improve the KoPL parser by  demonstrating the executed intermediate reasoning steps to the KBQA model. We show that such simple strategy can significantly improve the ability of complex reasoning. Specifically, we propose three components: a parsing stage, an execution stage and a refinement stage, to enhance the ability of complex reasoning. The parser uses the KoPL to generate the transparent logical forms. Then, the execution stage aligns and  executes the logical forms over knowledge base to obtain intermediate reasoning processes. Finally, the intermediate step-by-step reasoning processes are demonstrated to the KBQA model in the refinement stage. With the explicit reasoning processes, it is much easier to answer the complex questions. Experiments on benchmark dataset shows that the proposed PER-KBQA performs significantly better than the stage-of-the-art baselines on the complex KBQA.

From Parse-Execute to Parse-Execute-Reine: Improving Semantic Parser for Complex Question Answering over Knowledge Base
Wangzhen Guo Linyin Luo
Hanjiang Lai" Jian Yin
School of Computer Science and Engineering, Sun Yat-Sen University (guowzh6, luoly36) @maill.sysu.cdu.cm [Iaihanj3, issjyin) @ @mail.sysu.dd..cn
Abstract
Many algorithms (Zhang et al., 2018; Yih al., 2016; Schlichtkrull et al., 2018; Das et al., 2021; Cao et al., 2022a) have been proposed for KBQA. A notable research line for KBQA is semantic parsing (Yih et al., 2016; Cao et al., 2022a). It encodes the questions into formal meaning repre- sentations//ogical forms (Lan et al.. 2022), then the logical forms are executed over the knowledge base give the final answers, which is called parse-execute paradigm. KaFSP (Li and Xiong, 2022) a knowledge-aware fuzzy semantic pars- ing framework for KBQA. RuG-KBQA (Ye et al., 2021) is a rank-and-generate approach for solving generalization problem in KBQA. Several bench- marks KBQA datasets have also been proposed to promote research in this field. For example, MetaQA (Zhang al., 2018), WebQSP (Yih al., 2016) and LC-QuAD (Dubey et al., 2019) are the datasets for KBQA.
Parsing questions into executable logical forms has showed impressive results for knowledge- base question answering (KBQA). However, complex KBQA is more challenging task that requires to perform complex multi-step reasoning. Recently, new semantic parser called KoPL (Cao al., 2022a) has been pro- posed to explicitly model the reasoning pro- cesses, which achieved the state-of-the-art on complex KBQA. In this paper, we further ex- plore how to unlock the reasoning ability of semantic parsers by simple proposed parse- execute-refine paradigm. We refine and im- prove the KoPI. parser by demonstrating executed intermediate reasoning steps to the KBQA model. We show that such simple strat- egy can significantly improve the ability of propose three components: a parsing stage, an execu- tion stage and refinement stage, to enhance the ability of complex reasoning. The parser uses the KoPL to generate the transparent log- ical forms. Then, the execution stage aligns and executes the logical forms over knowledge base to obtain intermediate reasoning processes. Finally, the intermediate step-by-step reason- ing processes are demonstrated to the KBQA model in the refinement stage. With the ex- plicit reasoning processes, it is much easier to answer the complex questions. Experiments benchmark dataset shows that the proposed PER- KBQA performs significantly better than the stage-of-the-ar baselines on the complex KBQA.
complex reasoning. Specifically,
Recently, due to the complexities of users' in- formation requirements, the more challenging task called complex KBQA (Lan et al., 2022) has been studied, which requires multi-step/compooittinn reasoning. For complex KBQA, it is important that the QA models can be learnt to explicitly model the reasoning processes, which can provide better inter- pretation of compositional reasoning. Hence, Cao et al. (2022a) had proposed benchmark dataset KQA Pro and knowledge-oriented programming language (KoPL). KQA Pro is large-scale dataset for complex KBQA and provides explicit reasoing process for each QA pair. KoPL (Cao et al., 2022a) is compositional and interpretable programming language for complex reasoning. The KoPL parser provides human understandable descriptions of log- ical forms. Based on the large-scale pretrained language model BART (Lewis et al., 2019), the BART-based KoPL has shown impressive results on KQA Pro dataset.
1 Introduction
Knowledge-bel question answering (KBQA) (Lan al., 2022) is task that answers natural language questions over given knowledge graph. KBQA has become hot research topic since can lead to a large number of applications, c.g., intelligent assistants.
Although KoPL can explicitly describe the rea- soning processes, the logical forms generated from
*Corresponding Author

---

semantic parser are the intermediate results but not the final answer, which may result in sub-optimized solutions. For example, the slight generation devia- tion (Das et al., 2021) may cause the logical forms to run incorrectly. Thus a natural question arises: Can we use these intermediate results to further improve the complex reasoning ability?
ical operations (Lan et al., 2022). The existing solutions for Complex KBQA can be divided into two mainstream approaches, known as semantic parsing-based (SP-based) methods and informa- tion retrieval-based (IR-based) methods.
SP-based methods (Yih et al., 2016; Das et al., 2021) follow a parse- cxeece paradigm: parse question into a logical form such as SPARQL or KoPL, and execute it against the KB to obtain the answer. IR-based methods (Miller et al., 2016; Saxena et al., 2020) follow a retrieve-and and-generate paradigm: retrieve a question-specine graph and directly generate answer with text encoder-decoder. SP-based methods rely heavily on the parser to generate logical form for each question, while they enjoy the advantage of powerful sequence-to- sequence pretrained language models. IR-based methods apply the answer generation module make accurate prediction conditioned on the ques- tion and the retrieved context. IR-based methods require the retrieved context to be helpful.
In this paper, we further explore how to un- lock KoPL. and KBQA model for improving its capability of complex reasoning. We find that the complex reasoning ability can benefit from demonstrating the KoPL's intermediate results to the KBQA model. That is, with the intermedi- ate step-by-step reasoning processes from KoPL, we can elicit the multi-step reasoning for complex KBQA and achieve better performances compared KoPL. Observed from that, we propose novel parse-execute-refin paradigm for complex KBQA, Figure 1 shows an overview of the proposed ap- proach called PER- KBQA, which includes pars- ing stage, exccution stage, and a refinement stage for complex KBQA. Given a question, the parser first utilizes sequence-to-sequenee model to generate the logical forms. Since KoPL, can explicitly describe the reasoning process, we use KoPL as the logical forms in the parsing stage. Then, the execution stage aligns and executes the generated logical forms over the knowledge base. With that, we can obtain the intermediate step-by- step reasoning processes. Finally, in the refinement stage, the intermediate reasoning results are used as execution contexts. We combine the question and the execution contexts as inputs to the KBQA model. The main idea of this paradigm is that the answer may benefit from the intermediate step-by- step reasoning processes.
Recently, the KoPL parser is proposed from KQA Pro (Cao et al., 2022a), and it is transpar- ent about the execution process and can provide luxuriant information of the intermediate process, which makes it possible for our proposed method to combine both of the mainstream methods to make full use of their strengths.
ODQA. Open domain question answering aims to answer factoid questions based on large col- lection of documents. The mainstream solution for open domain QA is that firstly use a retriever to se- lcct a small subset of passages where some of them might contain the answer to the question, and then utilize an extractive or generative reader to identify the answer. The retriever can be sparse vector space models such as TF-IFF (Chen et al., 2017) and BM25 (Robertson et al., 2009), trainable dense vector space models such as DPR (Karpukhin al., 2020) and MDR (Xiong et al., 2020). The extrac- tive reader predicts a span from the retrieved con- text to answer an open domain question in pre- vious works such as DrQA (Chen et al., 2017), HGN (Fang et al., 2019) and DPR (Karpukhin et al., 2020). Recently, the generative reader achieved further improvements in multiple open domain QA benchmarks, such FiD (Izacard and Grave, 2020), KG-FiD (Yu et al., 2021) and PATH- FiD Yavuuz et al., 2022). In our work, we can view the proposed refinement stage as the genera- tive reader, and the proposed parsing and execution
We evaluate the PER-KBQA on KQA Pro dataset and the extensive experimental results show that 1) the proposed PER-KBQA performs signifi- cantly better than the existing state-of of-the-art base- lines, and 2) PER-KBQA is a simple method that can further unlock the KBQA model for improving its complex reasoning ability. Besides, PER-KBQA provides insight into novel way to combine the intermediate results into the final answer.
2 Related Work
KBQA. Complex Knowledge Base Question An- swering aims to answer complex questions over knowledge bases, which always involves multi- hop reasoning, constrained relations and numer-

---

Question 1: What country associated with Quincy, known as Gem City, where Mary Astor was born?
Parsing
Predicted KoPL:
Find
FilterStr official name Gem City
Mary Astor
QueryRelationQualiffr
Find
Quincy
location of birth
country
Knowledge Base:
alignment
Mary Astor data birth: 1906/5/3
place
Quincy nick name: Gem City
birth
country: United States of America
Execution
Execution Context: Find <arg> Quincy <return> Quincy <func> FilterStr <func> QueryRelationQualffer <arg> location
of birth <arg> country <return> United States of America <mask>>
Refinement
Bart Encoder
Bart Decoder
Question
Execution Context
The answer is:
United States of America
Figure I:
An overview of PER- PER-KBQA approach, It consists of three stages: Parsing, Execution and Refinement. Parsing stage generated the logical form based on the question, and then execution stage aligned the knowledge base and obtained the execution context, and finally refinement stage generated the answer conditioned on the question and the execution context.
stages the retrieval in ODQA.
3.1 Parsing
The parsing stage models the logical form (i.c.. KoPL) in a structured way. A sequence-to tt sequence model is widely utilized to generate the logical form token by token conditioned on the question. Since the pretrained encoder-decoder lan- guage models (e.g., BART (Lewis al., 2019)) have achieved considerable success on the ques- tion answering task, we leverage BART to generate KoPL following KQA Pro (Cao et al.. 2022a).
3 PER-KBQA
In this section, we formulate the complex KBQA task and describe the implementation of three stages of PER-KBQA In this paper, consider the KoPL language that explicitly describes the reasoning process as our logical form. Since only the KQA Pro (Cao et al., 2022a) dataset provides the KoPL,, we follow the KQA Pro setting. That is, each natural language question is paired with executable logical forms (i.e., KoPL). The logical forms can be executed against the KB to obtain the answer.
Given the KoPL 4, we convert its functions and arguments into the structured textual sequence:
4 = P1 <func> P2... <func> Pas Ps = fa <arg> argz2),
where 4 consists of n programs and the k-th pro- gram Pa includes function fx with its arguments ar 9k1/22 The function fx can have 27 forms described in KQA Pro (Cao et al., 2022a). We in- sert special tokens <func>, <arg> as indicators to separate the functions and their corresponding arguments respectively.
Task Definition: Let q be natural language query, K. be a symbolic knowl- edge base and the training dataset be AN)].
D = {(11,4, 4, A1), (g2, 12, where 4 represents the KoPL that can be executed against the KC to obtain the ground truth answer A, corresponding to the query 9:- Given the knowl- edge base K and a question g, the pamanng based QA methods are generate the KoPL i and exccute it predict the answer A. In this paper, we explore how to use these logical forms that describes the reasoning process from KoPL to further unlock the KBQA model.
We train BART to autoregressively generate the la per token at each step conditioned on the question d-- Thus the conditional generation is to minimize cross-entropy loss as
-
log Per (15lи<s, (1)
N
i=

---

where OP is the parameters of BART. The parsing stage is basically similar the BART-based KoPL method proposed by KQA Pro (Cao et al., 2022a). Only the question used to generate the logical form and the information in the knowledge base corresponding to the question is not utilized.
larity of the words embeddings encoded by the pre- trained language model BERT (Devlin et al., 2018) or encoded by the pretrained EmbedKG model TransE (Bordes et al., 2013). However, noted that we perform the alignment operation during the execution process and the candidate pool is always large, so we use the Jaccard similarity which is more efficient.
3.2 Execution
In the execution stage, we execute the generated logical forms over knowledge base to obtain the corresponding intermediate results. However, it is not guaranteed that the generated functional ur- guments will be consistent with the relations and attributes (edges) of the entities in knowledge base K. For example, in Figure the generated text official name and location of birth mismatch the attribute nick name and relation place of birth re- spectively. Although they are semantically similar, the logical forms can not be executed successfully in this knowledge base K.
3.3 Refinement
After the parsing and execution stages, we obtain the execution contexts that consist of the logical forms and the corresponding intermediate results from the knowledge base, which describe the step- by-step reasoning process. The refinement stage utilizes the exccution contexts to improve the com- plex reasoning ability.
Specifically, our proposed refinement stage or- ganizes the predicted KoPL 4 and the intermediate results obtained from the execution stage in the form of the step-by-step reasoning path:
To alleviate this problem, propose two steps: Alignment. We first explicitly align the gener- ated functional arguments 4 with the relations and attributes in based on their overlap: 2) Execution. We then execute the aligned logical form I over the knowledge base K. which obtains list of entities as intermediate results for all programs in 4,
C = P1 <func> P2... <func> Pay
Ph. fur <arg> argki ... <return> rk: (2)
Tk cj 09 Cms
More specially, in the alignment step, for each program Pk in KoPL 4, we define candidate pool P/z that contains the entity-speciie relations or attributes in the knowledge base. For example, in Figure the candidate pool PiiterStr contains the attributes' key-value pairs from the entity Quincy. For each generated functional arguments in Ps. we utilize the Jaccard similarity to find the most similar relations or attributes that exists in knowledge base K:
where C; is the execution context of logical form i; and Pı denotes the intermediate result of the k-th function that consists of list of returned entities {e,}. And the <return> is special token in- serted as an indicator of the returned results. our experiments, m, which is the number of entities that the k-th function Pe returns, ranges from to 5. If there are more than 5 entities returned, we randomly sample 5 entities to reduce the expensive quadratic computation complexity about the input length.
aligned arg max argb
a argkl la argi
)
aCPp.
Then we concatenate the question qi and the execution context C; as following:
where arga is the generated functional argument, E Pp, is the relation or attribute in the knowl- edge base and argj aligned is the aligned functional argument. Then the generated functional argument argx is replaced by the most similar relation or attribute arg, aligned" in the knowledge base to make the logical form executable. If the generated argu- ment exists in the knowledge base, it always aligns with itself. Otherwise, it aligns with the most simi- lar edge in the knowledge base.
X; Question: qi
Contert C₁
The answer is
and obtain the transformed training set ((Xi, A:)) based on the original training set.
The goal of the refinement stage is train an- other BART model to autoregressively generate the answer A; conditioned on X,. Thus the answer generation is to minimize cross-ent cross-entropy loss over
Previous work (Das et al., 2021) have studied the alignment methods such as using the cosine simi-

---

the transformed training set {(x,, A:)3 as
to further refine the language model for complex reasoning, This is also why we choose KoPL as the semantic parser since it can provide transparent and human understandable reasoning processes, which are more similar to natural languages.
Ad logpon (Uglu<s,, (3)
[(BR)
N
where OR is the parameters of the answer refine- ment model.
Experiments
noted that the last function Pn returns the answer in (may be correct or incorrect) when exe- cuting the predicted logical form 4. We mask the returned answer r,, of the P. in the context O₁ with a certain probability p when training the answer generation model. So we can avoid learning the spurious correlation between the returned answer In and the gold answer A,, and make the model attend to the question and reasoning path to predict the answer. We further explore the mask strategy in the ablation study.
In this section, we conduct the experiments to com- pare our proposed approach with various baselines, and provide detailed analysis and ablation study on the results.
Dataset: Since KoPL is recently proposed parser that explicitly describes the reasoning pro- cess and only the KQA Pro (Cao et al., 2022a) dataset provides KoPL, we use KQA Pro, a chal- lenging dataset for complex question answering over knowledge base for evaluation.
There are some motivations to introduce the re- finement stage. 1) The logical forms are the in- termediate outputs but not the final answer. Thus the logical forms may not be executed successfully against the knowledge base K. to obtain reason- able answer because of the slight generation devia- tion (Das et al., 2021), even though the alignment operation is performed. Fortunately, the intermedi- ate reasoning processes from KoPL. are transparent and explainable, which can provide abundant in- termediate results as clue information to improve the capability of complex reasoning. 2) Recently, fusion-in-decoder (FID) (Izacard and Grave, 2020) and its extension work, e.g., KG-FiD (Yu al.. 2021) and PATHFiD (Yavuz et al., 2022), have shown impressive performances. These QA meth- ods leverage the passage retrieval with generative reader for open domain QA, and have shown that the QA models can benefit from the context from the retrieval module. Inspired by that, our pro- posed method also takes advantage of the execu- tion context for improving performances. In this paper, we use the intermediate step-by-ster reason- ing processes as the execution contexts instead of the passage retrieval. From this perspective, we can view the parsing and execution stages as retriever and consider the refinement stage as a generative reader in FID (Izacard and Grave, 2020). 3) In- spired by the thain-of-thoughhe prompting meth- ods (Wei et al., 2022; Kojima et al., 2022; Li et al., 2022), which have shown that the intermediate rea- soning steps can improve the ability of pre-training language models to complex reasoning, we thus use the intermediate reasoning steps from KoPL
In KQA Pro training set, each instance consists of the textual question, the corresponding KoPL, ten candidate choices, and the golden answer. And there are two settings: open-ended setting and multiple-choice setting. We conduct our experi- ments in the open-ended setting where the ten can- didate choices not used. Besides, it provides figh-quality knowledge base which has rich literal knowledge (attributes and relations about entities) as well as qualifier knowledge (qualifier about at- tributes and relations). We can execute KoPL over the knowledge base to obtain the answer.
Metries: We evaluate the QA models based on the accuracy of the test set in KQA Pro dataset. Fol- lowing the KQA Pro (Cao et al., 2022a), we provide in-depth analysis of models' fine-grained reason- ing abilities for Complex KBQA including Multi- hop. Qualifier, Comparison, Logical, Count, Verify, and Zero-shot, where Multi-hop denotes multi-hop questions, Qualifier and Comparison measure the ability to query the literal knowledge or qualifier knowledge of the relations attributes, Count and Verify rely the intermediate results of entities list, Zero-shot requires the QA models to answer the questions related to unseen relations.
Baselines and Implementation Details: We reproduce the strong baseline BART based KoPL parser proposed in KQA Pro (Cao et al., 2022a). Based on this bascline, implement three stages of our approach. We also report the published meth- ods in the leaderboard including KVMNet (Miller et al., 2016), SRN (Qiu et al., 2020), EmbedKGQA (Saxena et al., 2020), RGCN (Schlichtkrull et al., 2018), GraphQ IR (Nie et al., 2022). Following

---

Multi-
Compari- son
Zero-
Model
Overall
Qualifier
Logical
Count
Verify
hop
shot
KVMemNet
16.61
16.50
18.47
1.17
14.99
27.31
54.70
0.06
SRN
12.33
EmbedKGQA
28.36
26.41
25.20
11.93
23.95
32.88
61.05
0.06
RGCN
35.07
34.00
27.61
30.03
35.85
41.91
63.88
0.00
BART SPARQL
89.68
88.49
83.09
96.12
88.67
85.78
92.33
87.88
BART KoPL
90.55
89.46
84.76
95.51
89.30
86.68
93.30
89.59
GraphQ IR
91.70
90.38
84.90
97.15
92.64
89.39
94.20
94.20
(ER-KB)A((ours)
93.82
92.93
89.83
97.57
92.99
89.01
95.30
91.69
Table I: Accuracy of various baselines and method KQA Pro test set.
PE
qi
C₁
recall
the KoPL parser that provides the transparent rea- soning steps, our execution stage can make align- ment efficiently from the proposed candidate pool instead of huge comparison space. With the pro- posed execution stage, the intermediate results instead of failing execute the programs when there are slight generation devi- ations. Second, the intermediate reasoning steps make it much easier to perform complex reasoning. The refinement stage uses the intermediate results to further generate a reasonable answer. Since the execution context can provide abundant informa- tion of the reasoning process including reasoning steps and their intermediate results, they impor- tant clues the answer. Therefore, our proposed parse-cxecute-reiie paradigm is beneficial all measures including Multi-hop, Qualifier, Compari- son, Logical, Count, Verify, and Zero-shot.
93.26
95.27
Table 2: The recall of the golden answer on KQA Pro validation set. PE denotes that we utilize Parsing and Execution stages to generate the answer. And q. C, rep- resent the question and execution context respectively.
successfully obtain
BART KoPL (Cao et al., 2022a), we utilize two in- dependent bart-base models to instantiate the pars- ing and refinement stages respectively. We set the learning rate for BART parameters 3e-5, the learning rate for other parameters as le-3, and the weight decay as le-5. We used the optimizer Adam (Kingma and Ba, 2014) for all models. Please note that our proposed execution stage is simple, effi- cient and requires no learning.
Table 2 an example that shows is beneficial to demonstrate the exccuted intermediate reason- ing steps to the KBQA model. In this example, we do experiments on the KQA Pro validation set as the golden answers are available. PE denotes how much percent of the golden answers are generated from the parsing and execution stages, which is the result of the parse-execute paradigm. After the stages, we can obtain the execution context C; (see Eq. 2), which the intermediate step-by-step rea- soning process. We also report the recall of [qr, Ct), which represents what percent of the golden an- swers are contained in the questions and execution contexts. Please note that when we calculate the re- call of [g,, Ct], also include the Verify question instances since the model can generate "yes/no" answer only based on the question and context, and the "yes/no" " is not necessary to appear in the con- text. From Table 2, we can observe that the percent of the golden answers in the questions and contexts
4.1 Quantitative Results
Overall Results. In Table 1, we compare the perimental results of our proposed method with the existing approaches for KQA Pro the same open-ended setting. We observe that our method PER-KBQA outperforms the existing published work on the KQA Pro benchmark. Compared to the sota, our proposed method shows significant performance gain in terms of most evaluated mea- sures. In particular, method improves the var- ious complex reasoning skills for the QA system, e.g., multi-hop reasoning and qualifier knowledge query.
Compared to the sota method BART KoPL (Cao al., 2022a) which follows parse-execute paradigm, our proposed parse-exccute-refine paradigm performs significantly better and has sev- cral advantages. 1) First, we can explicitly align the knowledge base in the execution stage. Because of

---

Multi-
Compari- son
Zero-
Model
Overall
Qualifier
Logical
Count
Verify
hop
shot
Parsing
90.72
89.69
84.83
95.65
89.60
88.41
93.51
89.21
+Lxx
92.66
91.54
86.78
96.82
91.49
89.01
94.41
92.20
+Exe+Ref 93.82
92.93
89.83
97.57
92.99
89.01
95,30
91.69
Table 3: Ablation study of three stages on KQA Pro test set. +Exe denotes that we combine Parsing stage and the Execution stage. +Exe+Rel represents full model with three stages.
04.4
the refinement stage. Figure shows that if we do not mask the last returned answer during training, the answer generation model is likely to predict the answer based on the returned answer in the exe- cution context and hardly obtain the performance gain. This because the model will learn the spuri- ous correlation between the last returned result and the golden answer.
24.4
$4.0
11.11
3421
04.2
04.9
00.2
To choose the mask ratio, we do experiments the validation set using different values (i.e., 0.1, 0.2. 0.9). The accuracy increases the mask probability p of the returned result increases until p reaches 0.6, and performance starts to drop after that. We can also see that the mask ratio is not sensitive. For example, the best performance is 94.62% (p 0.6) and the worst performance is 94.21% % (p = 0.1). The two results are close. Intuitively, in the situation where the information of the answer is almost provided, the model is less likely attend the question and the reasoning path in execution context, and hence lacks the gen- eralization ability. Under the circumstance where the answer basically do not appear in the context, the model can still generate correct answers based on the reasoning path. Thus we select reasonable value and set the mask probability 0.6 to conduct our experiments KQA Pro test set.
00.2
mA5 protis 05y
Figure 2: Ablation study of mask strategy on KQA Pro validation set.
is higher than those generated from the parsing and execution stages. It indicates that some of the golden answers in the intermediate results are not sufficiently utilized in the parse-execute paradigm. There is room for refinement stage improve the performance only based on the questions and the proposed execution context. In summary, the re- finement stage can further improve the ability of complex reasoning by fully exploring the interme- diate step-by-step reasoning processes.
4.2 Ablation Study
Three Stages: this set of experiments, we ex- plore the performance of three stages respectively. Parsing denotes that we reproduce the BART KoPL from KQA Pro (Cao et al., 2022a) our first stage. Table shows the comparison results. It can be observed that the execution stage effective. Dur- ing this stage, we can access the knowledge base and explicitly reason over the structure of the KB, which is significant to promote the generalization ability in the Zero-shot setting, the multi-hop ca- pability and other fine-grained reasoning abilities. And we can further make improvement after we take advantage of the execution context generate answer the refinement stage.
4.3 Case Study
To further understand our proposed approach PER- KBQA, we do case study to demonstrate how the three stages work. Shown in Figure 3, the pre- dicted KoPL and execution contexts are organized in a structured text form express the reasoning processes. Our proposed method can generate cor- reet answer in the situation where the answer is contained in the execution context (the first case Figure 3), and can predict the correct answer in the situation where the reasoning path is incomplete and the answer do not appear in the question or the execution context (the second case in Figure 3). In the first case, the predicted KoPL, which is
Mask Strategy: In this set of experiments, we aim to explore why we leverage mask strategy in

---

Q: Who is shorter, David Brown (the point guard) or Stephen King (the one that was born in 86]?
Gold KoPL: Find(point guard) <func> Relate(position played
speciality <arg> backwerd] <func> (Dnvid Brown) <func> <func> Find(Stephon King) <func> FilterData(date of birth carg> 1986-03-06 <arg> =] <func> SalectBetweom(helght carg> less)
Pred KoPL: Find(point guard) <func> Relate(position played team speciality <arg> backward) <func> DDvddDavid Brown) <func> AndO <func>
Sind(Stephen King) <func> FilterYear(date of birth <arg> 86 <arg> =) <func> SelectBetween(height <arg> less)
Parse pred ans: null
Parse+ pred Stephen King
Execution Context: Find <arg> point guard <func> Relate carg? position played on team heam/spent speciality carg> backward <return> Robert Smith Jason
Kidd Chris Paul Jack Thompson| Chris Paul <func> Find <arg> David Brown <func- And <return> David Brown <func> <arg> Stephen King <func- FilterYear <arg> birth <arg> 86 <arg> <return> Stephen King SelectBetween
<arg> height <arg> less <return> Stephen King
Parse+Eue Exxe -Ref pred Stephen King
Gold ans: Stephen King
Q:
ending that small forward position Michael
nickname Airness?
Gold KoPL: Find(Michall Jordan) <func> FilterStrincknamme <arg> His Aimess) <func> Find(small forward) <func> Queryfelationuuallleeeppottooonn played team speciality <arg> and time)
Pred KoPL: Find(small forward) <func> Find(Michael Jordan) <func> FilterStríiickmame <arg> Her Aimess) <func> Queryledationvvlffeelppsssio played on team speciality <arg> end time)
Parse pred None
Parse+Exe pred ans: None
Execution context: Find <arg> small forward <func> Find <arg> Michael Jordan <func> FilterStr carg> nickname <arg> Her Airness <return>
Michael Jordan <func> QueryRdationnuuallle <arg> position played on team / speciality <arg> end time Parse+ Exe + Ref pred ans: 2003 Gold 2003
Figure 3: Case analysis about our proposed method. We mark the error generations from the first stage and their ground truths in red. And the intermediate results are marked in green.
generated by the first parsing stage only condi- tioned on the given question, deviates slightly from the golden KoPL because of the lacking the spe- cific information of the knowledge base. And the predicted KoPL can not be executed against the knowledge base to obtain the right answer. How- ever, with the execution stage, we simply align the relations or attributes in the knowledge base to make the logical forms executable and can get the right answer. The refinement stage further utilizes the execution context to generate correct answer since the right answer is contained in the execution context.
5 Conclusion
In this work, we propose a novel approach called PER-KBQA for complex question answering over the knowledge base. Our method consists of three stages: parsing, execution and refinement. The parsing stage generates logical forms conditioned on the question. The execution stage alleviates the problem of slight generation deviation via the alignment the knowledge base and obtains the intermediate results. The refinement stage com- bines the logical forms and corresponding interme- diate results as the reasoning process and regards as execution context to generate the answer. Ex- periments on KQA Pro benchmark show that our method can significantly improve the performance in terms of most evaluated measures.
In the second case, the generated KoPL from the first stage failed to give the right answer corre- sponding to the knowledge base. Even though we apply the alignment operation of the second stage, it can still not give the right answer because of the disturbance from other relations or attributes in the knowledge base. So the information of the correct answer is not contained in the execution context. However, the refinement stage can generate the cor- rect answer based on the question and the reasoning path of the execution context. It indicates that utiliz- ing the mask strategy to train the refinement model is beneficial for the model to store some specific information about the knowledge base. Thus the refinement model can generate answer correctly by attending to the question and the incomplete reasoning path of the execution context.
Limitations and Future Work
Our proposed method is skillful at answering com- plex questions over knowledge hase, but our work also has limitations. Our approach relies on the log- ical forms that can provide transparent intermediate results and only the KoPL. is transparent to the best of our knowledge. However, Cao et al. (2022b) had shown the promising results transfer the KoPL to other complex KBQA benchmarks such as WebQSP (Yhh et al., 2016), CWQ (Talmor and Berant, 2018). So in the future, we plan to adapt our method to other complex KBQA benchmarks.

---

Acknowledgment
Gautier Izacard and Edouard Grave. 2020. Leverag- ing passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.0128
This work is supported by the National Natural Sci- ence Foundation of China (U191 1203, U2001211, U22B2060), Guangdong Basic and Applied Ba- sic Research Foundation (2019B151513000 2021A1515012172) Key-Area Research and
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.0499 049006.
Development Program of Guangdong Province 0020801011000011
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint
References
arXiv:1412.6955
Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. Advances in neural information pro- cessing systems, 26.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205. 1996.
Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Complex knowledge base question answering: A survey. IEFE Transactions on Knowledge and Data Engineering.
Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie. Yutong Xiang. Lei Hou, Juanzi Li, Bin He, and Han- wang Zhang. 2022a. Kqa pro: A dataset with explicit compositional programs for complex question an- swering over knowledge base. In Proceedings of the 60th Annual Meeting of the Association for Compu- national Linguistics (Volume 1: Long Papers), pages 6101-6119.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-to-se -o--uueeecceeeeeeeeeeeeeeeee pre pre-trinnin for natural language generation, translation, and comprehension. arXiv preprint arXiv: 1910. 13461.
Shulin Cao, Jiaxin Shi, Zijun Yao, Xin Lv, Jifan Yu, Lei Hou, Juanzi Li, Zhiyuan Liu, and Jinghui Xiao. 2022b. Program transfer for answering complex questions over knowledge bases. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 8128 8140.
Junzhuo Li and Deyi Xiong. 2022, Kafsp: Knowledge- aware fuzzy semantic parsing for conversational ques- tion answering over a large-scale knowledge base. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 461-473.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. arXiv preprint orXiv:/704.000 00051.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Guang Lou, and Weizhu Chen. 2022. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02336
Jian
Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole. Ethan Perez, Jay-Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. Case-based reasoning for natural language queries over knowledge bases. arXiv preprint 1rX.v:2104.08762.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir- Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly read- ing documents. arXiv preprint arXiv: 1606,03 022.
Lunyiu Nie, Shulin Cao, Jiaxin Shi, Qi Tian, Lei Hou, Juanzi Li, and Jidong Zhai. 2022. Graphq ir: Uni- fying semantic parsing of graph query language with intermediate representation. arXiv preprint 22Xiv:2005.12078
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprini arXiv:1810.0480)
Yungi Qiu. Yuanzhuo Wang. Xiaolong Jin, and Kun Zhang. 2020. Stepwise reasoning for multi-relatte question answering over knowledge graph with weak supervision. In Proceedings of the 13th international conference on web search and data mining, pages 474-
Mohnish Dubey, Debayan Banerjee, Abdelrahman Ab- delkawi, and Jens Lehmann. 2019. Le-quad 2.0: A large dataset for complex question answering over wikidata and dbpedia. In International semantic web conference, pages 69-78. Springer.
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2019. Hierarchical graph network for multi-hop question answering. arXiy preprint arXiv: 1911:1911.03631
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends in Information Re- trieval, 3(4):333-389

---

Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-hor hop question answering over knowledge graphs using knowledge base embeddingg.. In Proceedings of the 58th ammual meeting of the as- sociation for computational linguistics, pages 4498- 4507.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolu- tional networks. In European semantic web confer- ence, pages 593-607. Springer.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. arXiv preprint arXiv: 1803.06643.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint 2rXfv:2201./9901
Wenhan Xiong, Xiang Lorraine Li. Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mchdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, al. 2020. Answering complex open-domain questions with multi-hop dense retrieval. arXiv preprint 2rXiv:2009.1277
Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, Ni- tish Shirish Keskar, and Caiming Xiong. 2022. Mod- cling multi-hop question answering as single se- quence prediction. arXiv preprint arXiv:2205.0922 092266
Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2021. Rng-kbqa: Generation augmented iterative ranking for knowledge base ques- tion answering. arXiv preprint urXiv:2109008677
Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming- Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Anmual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201-206,
Donghnn Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yim- ing Yang, and Michael Zeng. 2021. Kg-fid: In- fusing knowledge graph in fusion-in-decoder for open-domain question answering. arXiv preprint arX.v:2110.04330.
Yuyu Zhang. Hanjun Dai, Zornitsa Kozareva, Alexan- der Smola, and Le Song. 2018. Variational reason- ing for question answering with knowledge graph. In Thirty-secome second AAAI conference on artificiall intelli- gence.


Reviews:
[{"review_id": "0cu5K0yT0D", "reviewer": "Reviewer_MbeL", "summary": "", "questions": "1. Can you provide any insights or explanations as to why the zero-shot performance decreases after refinement? \n2. Has your approach been tested on the non-IID data splits of the KQA Pro benchmark to validate compositional generalization? \n3. Were the two cases presented in Section 4.3 selected randomly or based on certain criteria? \n4. What is the average length of the execution context that is fed into the refinement model? Would the performance be enhanced by providing more than five results to the refinement model? \n5. Is it possible that an LLM could outperform a fine-tuned BART as the refinement model due to its ability to incorporate both open-domain knowledge and KB for answer refinement?\n6. I'm not certain if the previous works on KQA Pro utilized KB schema or ontology for alignment. If not, I believe the method's additional dependence on KB schema information should be addressed in the discussion.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and kind suggestions. Firstly, previous works have studied the alignment between generated logical forms and KB schemas, but almost of them needed to train a complex neural model to calculate the semantic similarity for alignment. However, in our situation, we performed the alignment operation during the execution process. Since the candidate pool is always large, we used the Jaccard similarity which is more efficient (Line 273-282). Secondly, the intermediate reasoning results were used as supplementary supervision in previous works, but in our approach, we utilized the intermediate results as context for further refinement instead of supervision information. Thirdly, our method did not perform worse generalization in non-IID settings. Instead, our method can make significant improvement on compositional generalization (see To Question 2). We will cite and discuss these references.\n\nTo Question 1: This is because our method focuses on the overall metrics (e.g. Multi-hop, Qualifier, Verify), not just zero-shot. Hence, we did not design complex methods to improve the zero-shot performance. In this paper, to improve the generalization, we only mask the returned answer from Execution stage in a certain probability during training the refinement model. We can expect that the use of more complect and mature methods will improve generalization. We will explore this in our future work.\n\nTo Question 2: Thanks for your constructive suggestions. Following KQA Pro(Cao et al., 2022a ), we conduct the experiments, which focus on compositional generalization to longer sequences or to greater compositional depths than have been seen in training. Since the train/val/test set of compositional generalization are not available, we re-split the train and val sets. We take the instances with short programs as training examples (length <= 9), and those with long programs as valid examples (length > 9). The experimental results are shown as below.\n\n| |BART KoPL   |   PER-KBQA(ours)|\n|--------|--------|---------|\n|Compositional Generalization|76.62 | **93.95**|\n\nThe experimental results showed that our approach can generalize to non-IID data splits of the KQA Pro benchmark and can significantly improve the performance of compositional generalization.\n\nTo Question 3: We randomly selected cases in Section 4.3\n\nTo Question 4: The average length of the execution context is about 400. We conducted the experiments on the validation set to study whether the performance be enhanced by providing more than five results to the refinement model.\n\n|Number of intermediate results |  3 | 5 | 7|\n|---------|--------|--------|--------|\n|Val Acc | 94.46  |  **94.62**  |  94.20 |\n\nThe experiments showed that much more intermediate results cannot improve the performance. An interpretation is that much more intermediate results may induce more noise for refinement and less intermediate results may decrease the recall of the gold answers, both of which may do harm to the performance.\n\nTo Question 5: It is possible. However, directly applying the question and execution context as input for LLMs may not make improvement in the case that LLMs have not seen the knowledge base. Hence, other techniques such as in context learning and retrieval augmentation need to be further explored.\n\nTo Question 6: We utilized the similarity score function to align with the KB. It is general and is not dependent on KB schema information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The deviation problem commonly exists in today's neural semantic parsers. This paper proposes a simple yet effective approach PER-KBQA for alleviating this problem by involving the intermediate execution results to refine the errors in the generated logical forms, in order to improve the end-to-end accuracy for complex KBQA tasks. The motivation and methodology of PER is therefore quite easy to understand. Based on the experimental results, the performance gain also looks quite promising. The outcome of the ablation study & case study also sounds reasonable and can provide solid evidence to support the effectiveness of the proposed approach. ", "reasons_to_reject": "The performance improvements of PER-KBQA mainly come from two sides: the KB schema alignment at the second stage, and the answer refinement at the third stage using the intermediate results. Both ideas are not novel and have been explored in many previous KBQA works. For example, both (Herzig and Berant, 2018) and (Wang et al., 2021) suggest schema linking alongside semantic parsing to reduce the mismatch between the generated logical form structures and KB schemas. Using the intermediate reasoning results as supplementary supervision has also been proposed by previous works like (He et al., 2021) and (Cao et al., 2022). \n\nAlso, if the intermediate results are presented to the refinement model, the end-to-end accuracy may improve only because the model is memorizing more KB schema & ontology, which may lead to worse generalization in non-IID settings. As shown in the ablation study, the refinement stage actually impairs the zero-shot performance. \n\nRef:\nHerzig, Jonathan, and Jonathan Berant. \"Decoupling Structure and Lexicon for Zero-Shot Semantic Parsing.\" In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1619-1629. 2018.\nWang, Bailin, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. \"RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers.\" In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7567-7578. 2020.\nHe, Gaole, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. \"Improving multi-hop knowledge base question answering by learning intermediate supervision signals.\" In Proceedings of the 14th ACM international conference on web search and data mining, pp. 553-561. 2021.\nCao, Shulin, Jiaxin Shi, Zijun Yao, Xin Lv, Jifan Yu, Lei Hou, Juanzi Li, Zhiyuan Liu, and Jinghui Xiao. \"Program Transfer for Answering Complex Questions over Knowledge Bases.\" In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8128-8140. 2022.\n", "questions_to_authors": "1. Can you provide any insights or explanations as to why the zero-shot performance decreases after refinement? \n2. Has your approach been tested on the non-IID data splits of the KQA Pro benchmark to validate compositional generalization? \n3. Were the two cases presented in Section 4.3 selected randomly or based on certain criteria? \n4. What is the average length of the execution context that is fed into the refinement model? Would the performance be enhanced by providing more than five results to the refinement model? \n5. Is it possible that an LLM could outperform a fine-tuned BART as the refinement model due to its ability to incorporate both open-domain knowledge and KB for answer refinement?\n6. I'm not certain if the previous works on KQA Pro utilized KB schema or ontology for alignment. If not, I believe the method's additional dependence on KB schema information should be addressed in the discussion.", "paper_topic_and_main_contributions": "This paper aims to enhance the semantic parser for complex KBQA by introducing a PER (parse-execute-refine) paradigm. This approach involves three stages: parsing, execution, and refinement. The parsing stage trains a BART neural parser to generate KoPL logical forms which are claimed to be transparent and explainable. The execution stage aligns and executes the logical forms over a knowledge base to obtain intermediate results of the sub-queries. In the refinement stage, these intermediate results from the second stage are demonstrated to another BART model together with the execution result of the complete logical form generated at the first stage, to achieve better end-to-end KBQA accuracy. The key contribution of the paper is the performance improvements of its proposed PER-KBQA approach, which can be credited to (a) the explicit alignment of the KoPL arguments to the knowledge base schema (i.e., relations and attributes), and (b) the answer refinement based on both the original execution results and the additional results from the intermediate logical form reasoning process. The results of the experiment indicate a significant improvement on the benchmark KQA Pro. Additionally, the results of the ablation study and case study provide supportive evidence for the paper's motivation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See \"Reasons To Reject\"", "Typos_Grammar_Style_and_Presentation_Improvements": "a. \"KQAPro\" at line 72, 73, 082, 121, 239 should be \"KQA Pro\".\nb. be consistent about the wording of \"pre-trained\" or \"pretrained\" model\nc. line 309: \"Quesion\" should be \"Question\""}, "extra_text": {}}, {"review_id": "RVUO14bCKZ", "reviewer": "Reviewer_yo9y", "summary": "", "questions": "1. Can you introduce more details about the alignment discussed in line 257?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and kind suggestions. Firstly, LLMs needs to rely on large scale of data and large model, while we used small language model that can make us easily train the sematic parser and the answer generator. Secondly, both CoT and our method leveraged the intermediate reasoning steps, but CoT needs to be used by LLMs to work. Instead, to achieve better reason capability, our method that used small language model needs more detailed design such as alignment and refinement. Thirdly, though LLMs is making huge success, we do not think that not leveraging LLMs is outdated.\n\nTo Question 1: The detailed alignment step was described in Line 262 ~Line 172. The goal of the alignment process is to help execute the generated KoPLs successfully, which is to deal with the cases that the generated arguments do not appear in the knowledge base. Thus, we define functions to find the closest concepts, entities or relations in the knowledge base to replace those out-of-KB arguments that are generated. We use Jaccard Similarity to measure the distance for simplicity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Even though the proposed method is simple, it is sound and light-weight for this specific task (i.e., complex question answering over knowledge base)\n2. The experimental design is thorough, encompassing a broad range of scenarios to validate the efficacy of the approach.", "reasons_to_reject": "1. The impact of the proposed method is limited. The importance of the middle step for reasoning tasks is well known (e.g., CoT). This paper mainly focuses on extending the idea of using the middle reasoning step to the complex KBQA problem.\n2. The overall technical paradigm of the proposed framework is slightly outdated. It would significantly strengthen the paper if the authors could include a discussion about using LLM language models on this task. (e.g., for parsing and the final merging)", "questions_to_authors": "1. Can you introduce more details about the alignment discussed in line 257?", "paper_topic_and_main_contributions": "This paper aims to improve the performance of a semantic parser called KoPL, which has been used for complex knowledge base question answering (KBQA). Specifically, instead of only showing the final results to the final QA model, this paper argues that showing the middle steps could help the final QA model achieve overall performance.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "2Di4QGmTfZ", "reviewer": "Reviewer_poiZ", "summary": "", "questions": "A) Does the Jaccard Similarity in 3.2 refer to a character-based similarity?\n\nB) If I understood correctly in line 326: \"spurious correlation\" is not a spurious correlation as it is the best answer the initial model can come up with right? Maybe find another wording here. Same in 514\n\nC) How is the accuracy for the Parsing only setup measured? You write that the information in the knowledge base is not used but the KoPL queries only produce an actual answer upon execution, right? My guess is that the <return> token and a subsequent answer is also generated alongside the query but this is not very clear and not shown in any of your examples. Please make it clear that this stage only uses the model weights when discussing the evaluation.\n\nD) Does the prior work on KQA Pro not use any alignment? This may be answered somewhere in the paper but I can't seem to find it now.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and appreciating our work.\n\nTo Question A: Yes, the Jaccard Similarity refers to character-based similarity.\n\nTo Question B: According to your suggestion, we will remove the word and rewrite the sentences.\n\nTo Question C: The Parsing only setup means that we only used the question-logical form pairs to train the semantic parse model and during inference time we can obtain the logical form given a question and execute it against the KB to get the final answer. The information about the knowledge base is not used during training the semantic parse model. Since the second stage may produce empty answer such as None, thus, in our experiments, we did not append the <return> token and a subsequent answer when there is an empty answer from the second stage.\n\nTo Question D: To our best knowledge, we firstly apply alignment on KQA Pro.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a clear improvement over existing work using the same benchmark datasets. It makes use of a novel parse-execute-refine architecture and shows clear improvements over existing work without introducing extreme complexity. While the work is currently limited to one dataset, the authors state that it can (with the use of existing methodologies) be applied and tested on other datasets.\n\nThe paper is written in a clear style with the main argument being very easy to follow. While there are some minor typos/grammar errors they did not impact my understanding.", "reasons_to_reject": "The evaluation is (by necessity due to the focus on KoPL) limited to a single dataset and thus not comparable to a wide body of knowledge base QA work. This limitation, however, is clearly stated!", "questions_to_authors": "A) Does the Jaccard Similarity in 3.2 refer to a character-based similarity?\n\nB) If I understood correctly in line 326: \"spurious correlation\" is not a spurious correlation as it is the best answer the initial model can come up with right? Maybe find another wording here. Same in 514\n\nC) How is the accuracy for the Parsing only setup measured? You write that the information in the knowledge base is not used but the KoPL queries only produce an actual answer upon execution, right? My guess is that the <return> token and a subsequent answer is also generated alongside the query but this is not very clear and not shown in any of your examples. Please make it clear that this stage only uses the model weights when discussing the evaluation.\n\nD) Does the prior work on KQA Pro not use any alignment? This may be answered somewhere in the paper but I can't seem to find it now.", "paper_topic_and_main_contributions": "The paper presents sota results on KQA Pro Knowledge Base Question Answering dataset. It relies on existing work that presented a parse-execute architecture but improves it by adding a refinement step. Multiple architecture variations are tested in an ablation study, clearly indicating that the refinement step yields performance improvements.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"The SPARQL 051\ntransduces natural language utterances into graph 052\nquery representations.\" this sentence does not make sense to me, the query language doesn't interact with natural language utterances, right?\n\n101: \"Observed from that\" -> Based on those observations?\n\n171 and 172: add space after TF-IDF and BM25 before citation\n\n238 and 416: Bart -> BART\n\nTable 3: Count column, two numbers are the same 89.01 -> should both be bold unless this is some copy-paste error\n\n431: you claim significance, implying statistical significance which is not analyzed. Perhaps go with a more neutral word like \"clear\" instead. There are a couple more instances of this.\n\n616: Kqa -> KQA\n\n472: \"how much percent of\" -> what percentage of"}}]

Author rebuttal:

Meta-review: This paper presents a new parse-execute-refine framework that interleaves logical form generation with KB execution to let a LM take the execution context into account when generating an answer.

This method yields significant gains over comparable baselines on overall and most partial metrics, including significant gains across the board over the relevant LLM baseline. These are all evaluated on a single dataset.

While the results are promising, the paper could benefit from a more thorough comparison to related work that focused on similar tasks and datasets. Alignment and intermediate execution are not novel (they even preceed neural semantic parsing) and, while the approach here is novel, the paper would benefit from a deeper discussion of how it relates to other work.
Decision: Accept-Main