DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases
Authors: Donghan Yu
Sheng Zhang
Patrick Ng
Henghui Zhu
Alexander Hanbo Li
Jun Wang
Yiqun Hu
William Yang Wang
Zhiguo Wang
Bing Xiang
Venue: ICLR 2023 poster (2023)
Keywords: knowledge base
question answering
information retrieval
semantic parsing

Abstract: Question answering over knowledge bases (KBs) aims to answer natural language questions with factual information such as entities and relations in KBs. Previous methods either generate logical forms that can be executed over KBs to obtain final answers or predict answers directly. Empirical results show that the former often produces more accurate answers, but it suffers from  non-execution issues due to potential syntactic and semantic errors in the generated logical forms. In this work, we propose a novel framework DecAF that jointly generates both logical forms and direct answers, and then combines the merits of them to get the final answers. Moreover, different from most of the previous methods, DecAF is based on simple free-text retrieval without relying on any entity linking tools --- this simplification eases its adaptation to different datasets. DecAF achieves new state-of-the-art accuracy on WebQSP, FreebaseQA, and GrailQA benchmarks, while getting competitive results on the ComplexWebQuestions benchmark.

Published as a conference paper at ICLR 2023
DECAF: JOINT DECODING OF ANSWERS AND LOGICAL FORMS FOR QUESTION ANSWERING OVER KNOWLEDGE BASES
Donghan Y Sheng Zhang", Patrick Ng", Henghui Zhu?, Alexander Hanbo Li², Jun Wang?, Yiqun Hu², William Wang". Zhiguo Wang², Bing Xiang
CCareeii Mellon University AWS Al Labs, New York, USA
ddyu2ecs. cmu. edu; edu P(zshe, patricng, zhiguow)er amazon. com
ABSTRACT
Question answering over knowledge bases (KBs) aims to answer natural language questions with factual information such as entities and relations in KBs. Previous methods either generate logical forms that can be executed over KBs obtain final answers or predict answers directly. Empirical results show that the former often produces more accurate answers, but suffers from non-execution issues due to potential syntactic and semantic errors in the generated logical forms. In this work, we propose novel framework DECAF that jointly generates both logi- cal forms and direct answers, and then combines the merits of them to get the final answers. Moreover, different from most of the previous methods, DECAF is based simple free-text retrivval without relying on any entity linking tools this sim- plification eases its adaptation to different datasets. DECAF achieves new state- of-the-art accuracy WebQSP, FreebaseQA, and GrailQA benchmarks, while getting competitive results on the ComplexWeb WebQuestions benchmark]
INTRODUCTION
Knowledge Bases Question Answering (KBQA) aims to answer natural language questions based on knowledge from KBs such as DBpedia (Auer et al. 2007). Freebase (Bollacker et al.) 2008 or Wikidata (Vrandečic & Krötzsch 2014). Existing methods can be divided into two categories. One category is based on semantic parsing. where models first parse the input question into logical form (e.g., SPARQL [hommeaux] 2011) or S-expression (Gu et al.
202I)) IT) then execute the logical
2021 Gu al. 2021 Ye al., 2022]. The other category of methods directly outputs answers without relying on the the logical- form exccutor (Lan et al. 2019 Sun al. 2019 Saxena al. 2022 Oğuz et al. 2022). They either classify the entities in KB to decide which are the answers (Sun et al.
form against knowledge bases to obtain the final answers (Das et al.
2019) or generate the answers
using a
framework (Saxena al.
2022 Oguz et al. (022)).
Previous empirical results (Ye et al., 2022 Das et al. 2021 Gu et al. (202)) show that the semantic parsing based methods can produce more accurate answers over benchmark datasets. However, due to the syntax and semantic restrictions, the output logical forms can often be non-executable and thus would not produce any answers. On the other hand, prrret-answer-prddiett to generate output answers, albeit their answer accuracy is usually not good as semantic parsing based methods, especially over complex questions which require multi-hop reasoning (Talmor & Berant 2018). To our knowledge, none of the previous studies have leveraged the advantages of both types of methods. Moreover, since knowledge bases are usually large-scale with millions of entities, most previous methods rely entity linking to select relevant information from KB for answering questions. However, these entity linking methods are usually designed for specific datasets, which inevitably limits the generalization ability of these methods.
methods can guarantee
In this paper, we propose a novel framework DECAF to overcome these limitations: (1) Instead of relying on only either logical forms or direct answers, DECAF jointly decodes them together, and further combines the answers executed using logical forms and directly generated ones to obtain
Work done during internship at AWS AT Lahs
Our code is available at http://gtth..mmmmiaaaalddddeeeet

---

Published as a conference paper at ICLR 2023
the final answers. Thus the advantages of both methods can be leveraged in our model. Moreover, unlike previous methods using constrained decoding (Chen et al. 2021aa or post revision (Das et al. 202I) to produce more faithful logical forms, we simply treat logical forms regular text strings just like answers during generation, reducing efforts of hand-crafted engineering. (2) Different from previous methods which rely on entity linking (Yih al. 2013 Li et al. 2020) to locate entities appeared in questions and then retrieve relevant information from KB. DECAF linearizes KBs into text documents and leverages free-text retrieval methods to locate relevant sub-graphs. Based on this simplification, DECAF brings better adaptation to different datasets and potentially different KBs due to the universal characteristic of text-based retrieval. Experiments show that simple BM25 retrieval brings surprisingly good performance across multiple datasets.
We conduct experiments on four benchmark datasets including WebOSP (Yth al., 2016), Com- plex WleeueeQusssinns (Talmor & Berant, 2018), FreebaseQA (Jiang et al. 2019), and GrailQA (Gu al. 2021). Experiment results show that our model achieves new state-of-thh-an results on We- bQSP, FreebaseQA, and GrailQA benchmarks, and gets very competitive results the Complex We- bQuestions benchmark. This demonstrates the effectiveness of DECAF across different datasets and question categories.
2 RELATED WORK
Semantic parsing based methods for KBQA first parse the input question into logical form (LF) then execute it against KB to obtain the answers. ReTrack (Chen et al.) 2021a) uses a grammar- based decoder to generate LFs based on pre-defined grammar rules, and a semantic checker to discourage generating programs that are semantically inconsistent with KB. RaG-KBQA (Ye et al.] 2022) first enumerates possible LFs based the entity in the input question. Then a ranking-and-
generation framework is applied to output the final LF. ArcaneQA (Gu & Su 2022) generates the LFs dynamically based on the execution results of LFs generated at intermediate steps. TIARA (Shu retrieval method to select relevant KB context for logical form
et al. 2022) proposes a multi-graine generation. All these methods rely on an external executor such as a SPARQL server to execute LFs for final answers. If the LFs are not executable, then no answers will be produced.
methods for KBQA directly output answers without relying on the LF executor. PullNet (Sun et al.] 2019) retrieves a subgraph of KB related to the input question and applies graph neural networks to predict the answer entities in the subgraphs. KGT5 (Saxena al. 2022) uses a sequence-to-seeqeece framework to directly generate answers only based on the input question. Uni (oguz al. 2022) is also based on a sequence-to-see to-sequence framework, but first retrieves relevant triplets from KB and then generates answers based on the combination of the input question and retrieved triplets. Although answers can always be produced without the need of LF executor, this type of method usually underperforms semantic parsing based methods on public benchmarks (Talmor & Berant 2018| Gu et al. 2021 2022).
Srreet-Answer-Prediiiee
Entity Linking & Knowledge Linearization. Real-world KBs are usually very large and with millions of entities and triplets. The algorithm to ground the input question onto a relevant sub- graph of KB is important. Entity linking is the most common way for this. CBR-KBQA (Das) et al. 2021) combines an off-the-shelf model ELQ (Li al. 2020) with an NER system provided by Google Cloud API for entity linking. RnG-KBQA YYe et al. 2022) also uses ELQ for the WebQSP dataset, while it uses a BERT-based (Devlin et al. 2019) NER system and trains another BERT- based entity disambiguation model for the GrailQA dataset. Previous works usually design different methods to optimize the performance of different datasets. A recent study (Soimaan et al. (2022) also shows that entity linking models are usually domain-specific and hard transfer across domains. Different from these methods, DECAF reduces this burden by linearizing KBs to text documents and leveraging simple text-retrieval methods. Experimental results show that this is not only more general but also empirically effective. Similar to our method, UniK-QA (Oguz et al.) 2022) also linearizes the KB and conducts retrieval. However, UniK-QA still requires entity linking (Yih et al.) 2015) to reduce the retrieval range, and it only generates direct answers without studying logical forms for questions requiring complex reasoning. Besides the studies above, UnifiedSKG (Xie et al. (022) is relevant since it studies the generation of logical forms and direct answers for KBQA. However, it does not study combining the advantages of both logical forms and direct answers, and further assumes that the ground-truth question entities are provided, which dramatically eases this
2

---

Published as a conference paper at ICLR 2023
Logical Form: (AND game_engine (JOIN game__ engine, developer m. clvv)
Linearize
KB
Documents
Prefix:
Semantic Parsing: what video game...
Execution
Answer:
Input Question:
Freescape
what video game engine did incentive software develop?
Retriever
Reader
Output
Combiner
Freescape
Prefix:
Question Answering: what video game...
Answer:
Freescape
Figure 1: Model framework of DECAF. We text-based retrieval instead of entity linking to select question-related information from the KB. Then, we add different prefixes into the reader generate logical forms and direct answers respectively. The fogeccl-for--eeecutedd answers and directly-generant answers are combined to obtain the final output.
problem. Li et al. 2222)) generates either an answer or an LF for an input question based on model choice, but studies the problem of Text-Table QA, which is inherently different from KBQA.
3 METHOD
In order to (I) leverage the advantages of both logical forms and direct answers and (2) reduce the dependency on entity linking models, we propose a novel framework DECAF. As shown in Figure 1, the whole KB is first transformed into text documents. Then, for an input question, the retriever retrieves relevant passages from lincarized KB documents, which will be combined with the input question into the reader. DECAF reader is seuuence-to-equenee generative model and uses different prefixes to generate logical forms (LFs) and direct answers respectively. Finally, the executed answers by LFs and generated direct answers combined to obtain the final answers.
3.1 KNOWLEDGE BASES LINEARIZATION
Given a question, retrieving relevant information from KBs is non-trivial since KBs are large-scale and complicated with both semantic (names of entities and relations) and structural (edge between entities by relations) information. On the other hand, recent studies have shown the effectiveness of text-based retrieval for question answering (Chen et al. 2017 Karpukhin et al.) 2020 Izacard Grave 202T). By linearizing KBs to text corpus, we can easily leverage these successtul text- retrieval methods to select semantically relevant information from KBs.
We describe how to linearize the knowledge base. Considering the original KB format to be the most common Resource Description Format (RDF), which contains triplets composed of one head en- tity, relation, and tail entity. For example, (Freescape, game game_engine developer,, Incentive Software) means that Incentive Software is the developer of game engine called Freescape. To linearize this triplet, we simply concatenate them with spaces to be a sentence as Freescape game engine devel- oper Incentive Software Note that also replace punctuation marks in relation spaces. In this case, this sentence contains both semantic information (entity and relation names) and structural information (relation between two entities). Then we further group sentences with the same head entity to become a document like Freescape game engine developer Incentive Software. Freescape release date 1987... This grouping mainly to preserve the structural information of the 1-hop subgraph corresponding to the head entity. In some KBs like Freebase (Bollacker et al., 2008), there exist hyper-triplets that express complicated relationship. which is discussed Appendix [.. |.. Following (Karpukhin et al., 2020). we truncate each document into multiple non-overlap passages with a maximum 100 words. In the next section, we show how to retrieve from these passages.
3.2 RETRIEVAL
The retriever retrieves relevant passages from the linearized KB based on the input question. We consider two kinds of retrieval methods: sparse retrieval and dense retrieval. For sparse retrieval,
3

---

Published as a conference paper at ICLR 2023
use BM25 (Robertson et al. 2009 which is based TF-IDF scores of sparse word match between input guestions and KB-linearized passages. For dense retrieval, we apply the DPR (Karpukhin) et al. 2020) framework, which is based on similarity in the embedding space between input ques- tions and passages from two fine-tuned BERTs (Devlin et al. 2019). We refer readers to the original paper for details of the fine-tuning process. During inference, suppose there totally N passages in the knowledge source (PI, PP......... pN}. DPR applies the passage encoder Ep(-) to encode all the passages and store embeddings in memory. For an input question g, DPR applies the ques- tion encoder E-(-) to obtain its representation. and then the passages are retrieved based on the dot-product similarity: Lvurieee = argtop-k; (,opp,,K(Eppp (pi( Eq(4)). Then it applies FAISS[Johnson al. 2019) to conduct an efficient similarity search due to the large number of passages N. Through this step, we can retrieve //atrieee << N passages which are potentially relevant to the input question.
3.3 READING
The reader takes the retrieved passages and the original question input and generates the targeted output. Recent advanced sequence-to-seqeene
models like BART (Lewis et al., (020) and T5 (Raffel et al. 2020) can be utilized here. In order answer complicated multi-hop questions, cross-passage reasoning is important for the reader. One way is to concatenate all the passages and let the self-attention in the Transformer module capture these patterns. However, this can be inefficient when the number of retrieved passages is very large because of the quadratic computation complexity in self-attention. To achieve both cross-passage modeling and computation efficiency, we apply Fushion-ins in-Decoder (FiD) (Izacard & Grave (202)) based on T5 as the reader model. FiD encodes the concatenation of the input question and each passage separately but decodes the output tokens jointly. Specifically, we denote the question as q and the retrieved passages as {Pr, Iri e Tratrivee J. The encoding process for each passage Pr; is:
P. Encoder(concat(( Pr.)) BL,xH,
(1)
where is the hidden dimension, and Lp is the total sequence length of question concatenated with passage. T5 (5-Encoder(()) is the encoder module of T5. Then the token embeddings of all passages output from the last layer of the encoder are concatenated and sent to the decoder to generate the output tokens Toutpun:
Toupput Decoder( (Pi;; P2; Prassee E R |easise Ln
By concatenating the encoder output embeddings, the decoder can generate outputs based on joint modeling of multiple passages.
3.4 JOINT DECODING ANSWERS AND LOGICAL FORMS
Motivated by the success of adding prefixes to control the generation of large-scale language models (Raffel et al., 2020 Xie et al. 2022). we use a shared sequence-to-sequence model to generate both logical forms and direct answers, and differentiate the two processes by prompting the model with different prefixes. The encoding-decoding process in Equation ① and becomes:
P panswer
Encoder(connca((refix answer -4.Pr.), T Tanswer Decoder( P pansser
ponswer
Misiee D)
(3)
P!F Enncode((conca(prrefi*** 4, (.:):))) Tir Decoder( [p
PLF
where Tanswer and Tuy are the output answer tokens and logical form tokens respectively. prefix answer and prefix are the prefixes for answer generation and logical form generation respectively, which we set as Question Answering: and Semantic Parsing:. For example, given the question What video game engine did incentive software develop?, we'll first retrieve relevant passages using the retriever. Then we add these two prefixes to produce two different inputs: Question An- swering: what video game engine did incentive software develop? and Semantic Parsing: what video game engine did incentive software develop?" For the first input, we train the model to generate the target output answer Freescape directly While for the second input, we aim to
answer
For direct answer generation, only consider returning one single answer. We'll discuss the possibility of returning multiple answers together in Appendix A.3
4

---

Published as a conference paper at ICLR 2023
WehQSP
CWQ
Freebase
Model
Hits@
F1
Hits@1
Hits@1 1
PullNet (Sun al.
2019)
67.8
62.8
47.2
EmQL (Sun et al.
2020)
75.5
NSM
Hee
2021
74.3
53.9
FILM Verga et al.
2021
54.7
63.3
KGT5 Saxena al. 2022
56.1
36.5
CBR-SUBG (Das et al.
2022)
72.1
52.1
SR+NSM (Zhang et al. 2022
69.5
64.1
50.2
UniK-QA Oggz el al. 2022)
79.1
QGG (Lan & Jiang 2020)
74.0
44.1
HGNet (Chen et 202Tb)
70.6
70.3
65.3
ReTrack (Chen et al.
202Ta
74.7
74.6
(Das al. 202
72.8
70.4
ArcaneQA (Gu & Su 2022)
75.3
Program Transter (Cao al.
2022)
74.6
76.5
58.1
RnG-KBQA (Ye et al.
2022
75.6
RnG-KBQA (TS-large)
76.2 0.2
DECAF (BM25 FiD-large)
79.0 0.4
74.9 0.3
68.1 0.5
78.8 0.5
DECAF (DPR + FiD-large)
80.7 = 0.2
77.1 1 0.2
67.0 0.4
79.0 0.6
DECAF (BM25 FiD-3B)
70.4
DECAF (DPR + FiD-3B)
82.1
78.8
Results on the test splits of 3 benchmark datasets: WebQSP, CWQ, and Freehase The two blocks of baselines direct-answer-predictinn and semantic parsing based methods spectively, We run 5 independent experiments for FiD-large based DECAF and report mean and standard deviation. means that we replace the original reader T5-base with TS-large and rerun experiments to have fair comparison with our method.
Table I:
generate the logical form (AND evg. computer.game gam...ngineeeeee...ee (JOIN cvg. computer -ame-- engine.develonn developer m. (d- ) which is simply treated as text strings during generation with- out constrained decoding. However, instead of directly generating the entity ID like m.0d -hv in the logical form, we replace it with the corresponding entity name like Incentive Softwre and add special tokens "(" and "I" identify The revised logical form to be generated be- developer Incentive Software 11). After generation, we replace the entity names with their IDs for execution. To ensure the 1-1 mapping between entity name and ID) we preprocess the KB by name disambiguation, which introduced in Appendix A.2 During training, we fine-tune the whole reader in a multi-task learning manner, where one input question contributes to two training data pairs with one for answer generation and the other one for logical form generation. During infer- ence, we use beam search to generate multiple candidates for both logical forms and answers, with the same beam size B.
comes (AND ovg. wwg..cootteeggaaemm gea.ie.ngiineeeeeeeeeee (JOIN ovg. comput ww..gae.engiieeee
Next, show how combine them to obtain the final answer. We first execute these logical forms against the KB using an executor" Suppose the list of executed answer set is [A] ... .ALF (B' < B since some logical forms can be non-executable) and the list of directly generated answer set is [Aapsserr ... A We consider two situations: (1) If B' = means none of these logical forms are executable, the final answer is simply A Aunswer (2) Otherwise when B' > we use weighted linear combination: we first assign the score AS(k) for AL and the score (1 A)S(k) for A answer where and S(k) is a score function based on the answer rank k. If an answer set appears both in executed answer list and generated answer list with ranks and respectively. then score the sum of two scores: XS(i) (1 A)S(j)) Finally, we select the answer set with the highest score as the final output. We leave the exploration of other combination methods as future work.
À < 1 is hyper-parameter controlling the weight of each type of answers.

---

Published as a conference paper at ICLR 2023
Model
Overall
L.D.
Compositional
Zero-Shot
QGG (Lan & Jiang 2020)
36.7
40.5
33.0
36.6
Bert+Ranking (Gu et al.
2021
58.0
67.0
53.9
55.7
ReTrack (Chen al. 202
65.3
87.5
70.9
52.5
S2QL (Zan et al. 2022)
66.2
72.9
64.7
63.6
ArcaneQA (Gu Su 2022)
73.7
75.3
66.0
RiG-KBQA Ye et al.
2022
74.4 (76.9)
89.0 (88.3)
71.2 (69.2)
69.2 (75.1)
RnG-K
(Trel
(77.1)
(88.5)
(69.8)
(75.2)
UniParser (Liu et al. 2022)
74.6
71.1
69.8
DeCC (Anonymous)
77.6
75.8
72.5
TIARA (Shu et al. 2022)
78.5
76.5
73.9
DECAF (BM25 + FiD-large
76.0 (78.7)
90.5 (90.2)
79.0 (78.7)
68.0 (73.7)
DECAF (DPR FaD--arge)
(75.4)
(89.7)
(75.8)
(69.0)
DECAF (BM25 + FiD-3BB
78.7 (81.4)
89.9 (89.7)
81.8 (80.1)
72.3 (78.4)
Table 2: Fl scores on the test split of GrailQA. The numbers in the parentheses are F1 scores on the dev split. means that we replace the original reader T5-base with T5-large and rerun experiments.
4 EXPERIMENT
Experiment Settings. We use the full Freebase (Bollacker et al., 2008) data pre- pre-processed by Wang al. (2021) the KB for all benchmarks. The total number of entities, relations, and triplets are about 88 million, 20k, and 472 million respectively. The total number of passages after linearization is about 126 million. For the retrieval module of DECAF, we use BM25 implemented by Pyserini (Lin et al.) 202T) and Dense Passage Retrieval (DPR) (Karpukhin et al., 2020) with BERT Devlin al. (2019) as question and passage encoders. We train separate DPRs on each dataset following the same training process and use the same model architecture in the original paper. The number of retrieved passages 100 if not specified. For the reading module, we leverage Fusion-in- Decoder (Izacard & Grave 2021) based on the T5 (Raffel al.) (020) model: FiD-large and FiD-3B with 770 million and billion model parameters respectively. For the decoding beam size, we use 10 for FiD-large model and 15 for FiD-3B model.
We evaluate DECAF on four benchmark datasets: WebQSP YYih et al., 2016), ComplexWcbuues WebQues- tions (CWQ) (almorr & Berant 2018), FrecbaseQA (Jiang al.) 2019), and GrailQA (Gu et al. 202T] Specifically, GrailQA provides the categories of questions: i.id., compositional, and zero- shot, which be used to evaluate model performance datasets provide both ground-truth answers and logical forms except FreebaseQA, where our model only generates answers the final output without using the logical form. More details about these datasets are shown in the appendix. Following previous work (Ye et al. 2022 Das et al. 2021 1 Oguz
different levels of generalization. All
al.) 2022), we evaluate our model based on metrics Hits @ and FI, where Hits single top-ranked answer while Fl also considers coverage of all the answers.
focus on the
4.1 MAIN RESULT
We compare with both direct answer prediction and semantic parsing based methods and run independent experiments for FiD-large based DECAF to report mean and standard deviation. We don't do this for DECAF with FiD-3B due to the limitation of computation resource. We denote our model in the form of DECAF ((Retriever) (Reader)) such DECAF (BM25 FiD--lrrge.. If the retriever is not specified, it means we choose the better one for each dataset respectively.
As shown in Table 0 DECAF achieves new SOTA on WebQSP and FreebaseQA dataset, outper- forming both firect-answer-prediitioo (first block) and semantic parsing based (second block) base- lines. On WebQSP, DECAF improves the previous highest Hits @ by 3.0% and F1 by 2.3%. Note that one of the best-performnng baseline UniK- -QA uses STAGG (Yih et al. 2015) for entity linking on WebQSP, which is not publicly available and thus can not be applied to other datasets. Compared to UniK-QA, we see that DFCAF (DPR + FiD-large) improves the Hits 1 by 1.6% with the same model size, demonstrating the effectiveness of our method even without entity linking. On Free- baseQA, DECAF improves the SOTA Hits @ significantly by 15.7%. On CWQ dataset, DECAF
We locally up a Virtuoso triplestore service following the GrailQA paper.

---

Published as a conference paper at ICLR 2023
0.0
0.2
0.4
0.45
0.49
0.51
0.55
0.6
0.8
1.0
S(k) 1/k S(k) = B 1: +
54.7
54.7
56.1
56.3
56.5
78.7
78.7
78.7
78.7
78.7
54.7
55.6
56.4
56.5
56.5
78.3
78.3
78.4
78.6
78.7
Table 3: F1 scores on GrailQA (dev) using DECAF (BM25 + FiD-large) based on different values of x which is the weight of LF-executed answers in the answer combination function. S(k) is the score function of answer rank k, and B is the generation beam size.
WebQSP
CWQ
GrailQA (dev)
Model
Hits@1 1
F1
Hits
FI (O)
F1 (1)
FI (C)
Fl (Z)
DECAF (Answer only)
74.7
49.8
50.5
54.7
59.4
38.3
59.5
DECAF (LF only))
74.3
74.0
55.2
72.4
88.2
76.3
63.9
DECAF
80.7
77.1
68.1
78.7
90.2
78.7
73.7
Mon-Executable LF%
11.3
30.2
15.8
6.2
9.6
22.5
DECAFsep sep (Answer only)
74.2
49.5
47.9
54.6
57.7
38.8
59.8
DECAFsep (IFFnlly)
72.7
73.1
54.3
74.0
91.4
75.2
66.0
DECAF xep
80.6
77.1
66.5
80.3
92.9
78.9
75.3
Non-Executable LF%
12.3
32.8
15.9
5.2
11.9
22.4
Table We study the performance of our model when only using generated answers (Answer Only) or executed answers by logical forms (LF Only). O, I, C, Z means overall, i.i.d., compositional and zero-shot. DECAF sep means using a separate reader for answer generation and logical form generation respectively instead of a joint reader. We also show the percentage of questions where none of the generated LFs are executable.
achieves very competitive results, the same the current SOTA method CBR-KBQA. CBR-KBQA is based on the K-Nearest Neighbors approach, which is complementary to our method. We also that increasing reader size from large (770M) to 3B significantly improves model performance. Moreover, BM25 and DPR lead to different results: DPR performs significantly better than BM25 on WebQSP, slightly better on FreebaseQA, and worse on CWQ. The possible reason is that DPR is trained to retrieve passages containing the answers based on distant supervision (Karpukhin et al.) 2020), which do not necessarily contain the relations and entities that appeared in the logical form, especially for complex questions. Thus it may hurt the performance of logical form generation which results a final performance degeneration.
Table 2 shows results the GrailQA dataset, where we listed Fl scores of Overall, LLD.., Compo- sitional, and Zero-shot questions respectively. We see that with FiD-large reader, DECAF achieves better overall performance than the published SOTA method RnG-KBQA. Since original RnG- KBQA uses basbase, we also equip it with T5-large for a fair comparison and list the results on the dev set, where DECAF (BM25 + FiD-large) still significantly outperforms it by 1.6% in overall F1 score. With FiD-3B reader, the overall F1 of DECAF is improved by 2.7% compared to using FiD-large reader, and surpasses the current SOTA method TIARA which is anonymous submis- sion on the GrailQA leaderboard. Notably, DECAF performs the best in compositional questions with an Fl score 5.3% higher than the best-performing method.
Overall we see that DECAF achieves new SOTA results on WebQSP, FreebaseQA. and GrailQA with very competitive results on the CWQ benchmark. Even with simple BM25 retrieval, DECAF outperforms most of the baselines across all the benchmark datasets, which previous studies usually use different entity linking methods specially designed for different datasets.
4.2 ABLATION STUDY
In this section, we conduct some ablation studies to answer the following questions:
What is the best way to combine LF-executed answers and generated answers? In Section 3.4 we introduced way combining LF-executed answers and generating answers to obtain the final answer. (1) When none of the LFs are executable,
the top-1 1 generated answer as output. In Table 4 we show the percentage of questions where none of the generated LFs are executable.
We achieve the st rank the GrailQA leaderboard of 09/06/20222.

---

Published as a conference paper at ICLR 2023
Model
Combination Type
WebQSP
GrailQA (dev)
ArcaneQA
None
75.6
76.9
ArcaneQA + DECAF (FiD-large Answer Only)
LF Answer
75.8
77.4
ArcaneQA DECAF (FiD-3B, Answer Only)
LF Answer
75.8
77.5
ArcaneQA + RnG-KBQA
I.F
76.0
77.0
RaG-KBQA
None
76.2
77.1
RuG-KBQA DECAF (FiD-large, Answer Only)
LF Answer
76.9
77.1
RnG-KBQA DECAF (FiD-3B, Answer Only)
LF Answer
77.0
77.1
RnG-KBQA+ ArcaneQA
LF+ LF
77.2
77.1
DECAF (FiD-large)
LF + Answer
77.1
78.7
DECAF (FiD-3B)
LF+ Answer
78.8
81.4
Table 5: Results of the combination of baseline methods and comparison with our proposed model DECAF. X + Y means that first the answer produced by X as the output; if answer is produced, we use the answer produced by Y. F1 scores are reported for both WehQSP and GrailQA (dev) datasets.
It can be observed that directly generating LFs for hard questions, such as CWQ and Zero-shot GrailQA, shows significantly higher noe-executable rate than that for easy questions. (2) If any LF is executable, we use the answer with the highest combination scores XS + (1 S)S()), where Ả is the hyper-parameter weight for FF-executed answers, and and j are the rank in the L.- executed answer list and generated answer list respectively. We test two different score functions S(k) = 1/k and (kk = B k + 1 where B is the beam size, and k is the rank in the candidate list. From the results in Table 3 we see that the model performance is improved when À increases. Specifically (1) the F1 score increases dramatically when changes from 0.49 to 0.51, where the former usually chooses generated answer final answer while the latter selects the LF-executed one. This demonstrates that LF-executed answers are much more accurate compared to generated ones, which can also be validated in the next section. (2) X 1.0 gives the best results, which means that we can simply select the first executed answer if exists, otherwise choose the first generated answer. Results on WebQSP and CWQ show the same patterns, as listed in Appendix A.5 Thus we set x = 1.0 as the default setting of DECAF in all the experiments.
How do answer generation and LF generation perform respectively? DECAF combines gener- ated answers and executed answers to obtain the final answers. In this section, we study the impact of them separately. We introduce two new models (1) only use generated answers (Answer Only) and (2) only use executable answers by logical form (LF Only). As shown in the first group of models in Table 4 the performance of LF Only is consistently better than Answer Only. Except on WebQSP, the latter has similar Hits@ 1, but the former has significantly better F1 score, which shows that LF can produce much better answer coverage. We also see that the combination of these two strategies is significantly better than any of them separately. For example, on the dev set of GrailQA, the overall F1 of DECAF is 6.3% higher than DECAF (LF only) and 24.0% higher than DECAF (Answer Only). In Appendix A.44 provide detailed comparison between the results ob- tained from logical-form and answer generation methods. This comparison covers various question types, number of relations present in the logical form, and the total number of correct answers.
Should we use a joint reader or separate readers for answer and LF generation? We add another variation of our model called DECAF sep. which uses two separate readers generate direct answers and logical forms respectively instead of a shared reader, while other parts remain the same as DECAF. As shown the second group of models in Table 4 we see that the overall performance is similar to DECAF, showing that a shared reader with prefix is as capable as two separate readers. However, training performs better on answer generation and LF generation while on GrailQA, it performs worse on LF generation compared to two separate readers.
interesting to see that on CWQ, a shared reader with multi-task
How does the ensemble of baseline methods perform? We select two published best semantic parsing baselines, ArcaneQA (Gu & Su 2022) and RnG-KBQA (Ye et al., 2022), and ensemble them with our DECAF model. That is, if their generated logical forms are not executable, we use the directly generated answers by DECAF as the final answer. We also ensemble these two baselines: if the logical forms generated by one of them other method will be used. As shown in Table 5 where Fl scores are reported, we see that they still underperform DECAF. Upon thorough analysis, find that, for ArcaneQA and RnG-KBQA,
not executable, the logical forms generated by the
8

---

Published as a conference paper at ICLR 2023
WchQSP
CWQ
GrailQA
Freebase(A
Model
H@ 100
R@ 100
H@ 100
R@ 100
H@ 100
R@ 100
H@ 100
R@ 100
BM25
81.2
67.7
63.5
57.7
89.9
84.7
93.5
93.5
DPR
91.6
80.6
71.4
65.6
87.6
81.0
95.0
95.0
Table Retrieval results (answer name match). H@ 100 and R @@ 100 stand for the answer hits rate and recall rate of 100 passages, respectively.
90
77.5
80
75.0
Hits@1
WebQSP
72.5
CWQ
70
I.I.D.
70.0
FreebaseQA
Compositional
Zero-Shot
60
65.0
Overall
510
20
50 #Passages
100
510
20
50
100
# Passages
(a) Results on GrailQA.
(b) Results on WebQSP, CWQ, and FreebaseQA
Figure 2: DECAF (FiD-large) performance based on different number of retrieval passages.
very few questions have non-executable logical forms. For ArcaneQA, the corresponding question percentages are 0.7% and 1.4% on WebQSP and GrailQA. For RnG-KBQA, they are 1.8% and 0%. For DECAF (FiD-large), they 11.3% and 15.8%. The reason is that ArcaneQA uses constrained decoding and RnG-KBQA uses logical form enumeration to increase the execution rate of their generated logical forms. However, high execution rate does mean high accuracy. We argue that for some questions, instead of forcing the model to generate executable logical forms, switching to direct-answer generation a better choice.
Does retrieval capture useful information for answering questions? We first evaluate the re- trieval results of BM25 and DPR on 4 datasets T We observe that: (1) DPR performs better than BM25 on all the datasets except GrailQA, which contains zero-shot questions where DPR may have poor performance due to generalization issues. (2) On WebQSP, GrailQA and FreebaseQA. the retrieval method can achieve over 80% hits rate and recall rate, demonstrating the effectiveness of retrieval. (3) The performance is not as good on CWQ dataset, where most questions require multi-hop reasoning. This problem be potentially mitigated by iterative retrieval (Xiong al., 2021 Qi et al. 2021), which we leave as future work. We then analyze the effect of retrieval on the final QA performance Figure where show the results of retrieved passages. We see that on all the datasets, with the increase of passage number (from 100), the model performance is improved. Specifically, on GrailQA dataset, over different cat- egories of questions, we see that the performance over I.I.D. questions increases the least while it improves the most over zero-shot questions. This because I.I.D. questions can be handled well by memorizing training data while zero-shot questions require KB knowledge to be well answered.
datasets with different numbers
CONCLUSION
In this paper we propose a novel method DECAF to jointly generate direct answers and logical forms (LF) for knowledge base question answering. We found that combining the generated answers and LF-executed answers can produce more accurate final answers. Instead of relying on entity linking, DECAF is based
sequence-to-sqquence framework enhanced by retrieval, where we transform the knowledge base into text and use sparse or dense retrieval to select relevant information from KB guide output generation. Experimental results show that achieve the state-of-the-arr on WebQSP, FreebaseQA, and GraiQQA. Our work sheds light on the relationship between more general semantic parsing based methods and
preedctpne methods. It would be interesting to further explore this direction on other tasks involving both semantic parsing and end-to-end methods like table-based question answering or programming code generation.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
We acknowledge the importance of the ICLR Code of Ethics and agree with it. This work retrieves information from linearized KB and decodes logical forms and answers jointly, and uses combiner to obtain the final answer. One common concern for the KBQA model is that if bias or fairness issues exist in KB, our framework may propagate bias or unfairness when answering the questions based such KB. In addition, adversarial attacks may alter the behavior and performance of the proposed model an unexpected way. Therefore, the model should always be used with caution in practice. We believe this work benefit the field of KBQA, with the potential to benefit other fields involving retrieval-thens -henrdddggn modeling.
REFERENCES
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: nucleus for web open data. The semantic web, pp. 722-735. Springer, 2007.
Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a col- laboratively created graph database for structuring human knowledge. SIGMOD Conference, 2008.
Shulin Cao, Jiaxin Shi, Zijun Yao, Xin Lv, Jifan Yu, Lei Hou, Juanzi Li, Zhiyuan Liu, and Jinghui Xiao. Program transfer for answering complex questions over knowledge bases. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8128-8118111 8140, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. 10.18663//1/2022.aacl lon.:.......l- URL htt.lp/////llaattooooooooo//////
acl- long. acl-long.
559
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open- domain questions. In ACL, 2017.
Shuang Chen, Qian Liu, Zhiwei Yu, Chin- Yew Lin, Jian-Guang Lou, and Feng Jiang. Retrack: A flexible and efficient framework for knowledge base question answering. In ACL, 2021a.
Yongrui Chen, Huiying Li, Guilin Qi. Tianxing Wu, and Tenggou Wang. Outlining and filling: Hierarchical query graph generation for answering complex questions over knowledge graph. ArXiv, abs/211l 111111.00000 .00732, 2021b.
Rajarshi Das, Manzil Zaheer, Dung Ngoc Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. Case-based reasoning for natural language queries over knowledge bases. In EMNLP, 2021.
Rajarshi Das, Ameya Godbole, Ankita Rajaram Naik, Elliot Tower, Robin Jia, Manzil Zaheer, Han- naneh Hajishirzi, and Andrew McCallum. Knowledge base question answering by case-based reasoning over subgraphs. In ICML, 2022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v//N19-110 1423. URL https:
7/acl anthe hrw..org/N19. 14233
Yu Gu and Yu Su. Arcaneqa:
Dynamic program induction and contextualized encoding for knowl- edge base question answering. ArXiv, abs/2204.08100 2022.
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 202/, pp. 3477-3488, 2021.
Yu Gu, Vardaan Pahuja, Gong Cheng, and Yu Su. Knowledge base question answering: A semantic parsing perspective. arXiv preprint arXiv:2209.04990) 2022.
10

---

Published as a conference paper at ICLR 2023
Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji rong Wen. Improving multi- hop knowl- edge base question answering by learning intermediate supervision signals. Proceedings of the 14th ACM International Conference Web Search and Data Mining, 2021.
E. Prud hommeaux. Sparql query language for rdf. 2011.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. EACI., 2021.
Kelvin Jiang, Dekun Wu, and Hui Jiang. Freebaseqa: new factoid qa data matching trivia-style question-answe answer pairs with freebase. In NAACL, 2019.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):35--547, 2019.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering. ArXiv, 05s/2004.04906, 2020.
Yunshi Lan and Jing Jiang. Query graph generation for answering multi-hop complex questions from knowledge bases. In ACL, 2020.
Yunshi Lan, Shuohang Wang, and Jing Jiang. Knowledge base question answering with topic units. In Proceedings of the Twenty-Eighh International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 5046-5052. International Joint Conferences on Artificial Intelligence Organization, 7 2019. doi: 10.24963/ijccii.2119701. 701
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequeniceto--sqquence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computaational Linguistics, pp. 7871-7880, On- line, July 2020. Association for Computational Linguistics. doi: 10. (6653////002..cc-maaii URL sectps://allanthology./ org. 20wp/2020... acl mminn. 703
Alexander Hanbo Li, Patrick Ng. Peng Xu, Henghui Zhu, Zhiguo Wang, and Bing Xiang. Dual reader-parser on hybrid textual and tabular evidence for open domain question answering. In ACL, 2021.
Belinda Z. Li, Sewon Min, Srinivasan Iyer, Yashar Mehdad, and Wen-tau Yih. Efficient one- pass end-to-end entity linking for questions. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 6433-6441, Online, November URL
2020. Association for Computational Linguistics. doi: 10. 122665//1122022.mmmmmmmmimm https://aclanthollee wwwptthhhoooo..ooo///.. 2320 ottp///ooooo.ooo22222mmomooaaaaa
Jimmy J. Lin, Xueguang Ma. Sheng-Chieh Lin, Jheng- Hong Yang, Ronak Pradeep, Rodrigo Nogueira, and David R. Cheriton. Pyserini: python toolkit for reproducible information re- trieval research with sparse and dense representations. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
Ye Liu, Semih Yavuz, Rui Meng. Dragomir R. Radev, Caiming Xiong, and Yingbo Zhou. Uni- parser: Unified semantic parser for question answering knowledge base and database. ArXiv, abs/2211.00 1.05165,5 2022.
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stanislav Peshterliev, Dmytro Okhonko, M. Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. Unik-qa: Unified representations of structured and unstructured knowledge for open-domain question answering. In NAACL-HLT, 2022.
Peng Qi, Haejun Lee, Tg Sido, and Christopher Manning. Answering open-domain questions of varying reasoning steps from text. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3599-3611 Online and Punta Cana. Dominican Republic, November 2021. Association for Computational Linguistics. doi: (0.18653/1122221.mmm/. main. 292. httpssts: /acl hhtppp/s//acaaannhhllgggggg
pp-ma 292

---

Published as a conference paper at ICLR 2023
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910. 2020.
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Retrieval, 3(4):333-38) 2009.
Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. Sequence-to-sequence knowledge graph
completion and question answering. In ACL, 2022.
Yiheng Shu, Zhiwei Yu, Yuhan Li, Börje F. Karlsson, Tingting Ma, Yuzhong Qu, and Chin- Yew Lin. Tiara: Multi-grained retrieval for robust question answering over large knowledge bases. ArXiv, abs/2210.12925, 2022.
Hassan Soliman, Heike Adel, Mohamed H. Gad-Elrab, Dragan Milchevski, and Jannik Strötgen. A study on entity linking across domains":" which data is best for fine-tuning? In REPLANLP, 2022.
Haitian Sun, Tania Bedrax- Weiss, and William W. Cohen. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. ArXiv, 00t519944.09337, 2019.
Haitian Sun, Andrew O. Arnol, Tania Faithful embeddings for knowledge base queries. arXiv: Learning. 2020.
Fernando Pereira, and William W. Cohen.
Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions. In NAACL. 2018.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William Cohen. Adaptable and interpretable neural memoryover symbolic knowledge. In NAACL, 2021.
Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communi- cations of the ACM, 57(10)778-85, 2014.
Zhiguo Wang, Patrick K. L. Ng. Ramesh Nallapati, and Bing Xiang. Retrieval, re-ranking and multi-task learning for knowledge-base question answering. In EACL, 2021.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida Wang, Victor Zhong. Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.05996. 2022.
Wenhan Xiong, Xiang Lorraine Li, Srinivasan Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. Answering complex open-domain questions with multi-hop dense retrieval. International Conference on Learning Representations, 2021.
Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. Rug-kbqa: Generation augmented iterative ranking for knowledge base question answering. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
Wen-tau Yih, Ming-Wes Wei Chang, Xiaodong He, and Jianfeng Gao. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1321-1331, Beijing. China, July 2015, Association for Computational Linguistics. doi: 10.3115/v 1128. URL https://as httt:s///acclattlooooooooo//
Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming- Wei Chang, and Jina Suh. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp- 201-206, 2016.
12

---

Published as a conference paper at ICLR 2023
Daoguang Zan, Sirui Wang. Hongzhi Zhang, Yuanmeng Yan, Wei Wu, Bei Guan, and Yongji Wang- 2 ql: Retrieval augmented zero-shot question answering over knowledge graph. In Advances in Knowledge Discovery and Data Mining: 26th Pacific-Asia Conference, PAKDD 2022, Chengdu, China, May 16-19, 2022, Proceedings, Part III, pp. 223-236. Springer, 2022.
Jing Zhang, Xiaokang Zhang. Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5773-5784, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. 1 URL htops: / a cnthoolcg...rr//22222 accl wewellloo...3.. long. 396
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 KB LINEARIZATION: HYPER-TRIPLES
In Section 3.1 we introduced how we linearize vanilla triplets in KBs. Note that triplets have se- mantic constrains express complicated relations. For example, it is hard to express that Richard Nixon and Pat Nixon got married in The Mission Inn Hotel & Spa, which involves three entities. In Freebase, this is solved by introducing new node called CVT node, which serves con- necting entity for such hyper-triplets but has no meaning (name) itself. In this example, an entity with id m. m.02h98gq is introduced which involves triplets (m. (m.02h98gg. marriage.spouse. Pat Nixon), (m. 02h98gq.
curriage.spousee Richard Nixon), and (m.02h98gq. marriage.location locationnoo _of_cmoooooy, The Mission Inn Hotel & Spa). In this case, instead of concatenating CVT node id into sentence, we ignore this node while grouping other entities and relations into one passage: marriage spouse Richard Nixon. marriage spouse Par Nixon. marriage location of ceremony The Mission Inn Hotel & Spa. We illustrate our KB linearization in Figure
Original KB Triplets
Original KB Hyper-Triplets
The Mission Inn Hotel Spa
Proprietary software
Richard
marriage.locaioon
license
Freescape
Incentive Software
marriage. spouse
developer
CVT node
Pat Nixon
marriage. spouse
Converted Text
Converted Text
Freescape license Proprietary software. Freescape developer Incentive Software...
marriage spouse Richard Nixon. marriage spouse Pat Nixon. marriage location The Mission Inn Spa.
Figure 3:
Knowledge base linearization. We show examples of how we linearize triplets (two entities and one relation) and hyper-triplets (multiple entities and relations with a central CVT node).
A.2 ENTITY NAME DISAMBIGUATION
In KB Linearization (Section 3.1 and logical form generation (Section 3.4. we need to maintain mapping between an entity ID and an entity name. We can usually assume that one entity ID can be mapped to exactly entity name, while other names are treated as aliases. However, one entity name can not necessarily be mapped to exactly one entity ID, due to the ambiguity of entity names. For example, in Freebase (Bollacker al. 2008), the name Over You refers different songs, Sun can refer to the star, or an American R&B band. We employ a simple strategy to deal with this ambiguity issue: adding unique suffix vk where & is the number of current entities that has the same name. For example if there are three entities named as Sun, we'll rename them as Sun, Sun v/, Sun v2. The order of renaming does not matter long as the names are already differentiated. In this case, during knowledge base lincarization, instead of concatenating the original name of entities into sentences, concatenate their disambiguated names. During model generation, the output entity name can be easily mapped to the corresponding entity ID.
A.3 MULTI-ANSWER GENERATION
One real-world question can involve multiple correct answers. Like What are the countries with over billion population?? the correct answer includes both China and India. This is also true the benchmark datasets, as shown in Table 7 We see that only about 50% questions in WebQSP have a single answer, and about 12% contains even over 10 correct answers. It's challenging for direct- answer-generation methods to generate multiple answers accurately. Although DECAF generate a single answer for the direct answer generation. we also explore the possibility of multi-answer generation here. First tried returning top-K (K 1) generated answers for each question with
14

---

Published as a conference paper at ICLR 2023
beam search. However since the numbers of correct answers to different questions different, we find that increasing K even degrades the model performance. Then we tried to generate the concatenation of multiple answers splitted by special token "I". For example, for the question What are the countries with over / billion population?, the generated output will be "China India". After generation original (default) method as DECAF (Single Answer) while this one as DECAF (Multiple Answers). The results are shown in Table where we see that generating multiple answers can improve the Answer Only performance, but not the overall performance since it hurts the performance of logical form generation. Thus our final model, we still use single-answe answer generation. We leave the exploration of multi-answer generation as future work.
split the output by the special token obtain all the answers. We denote the
Dataset
2
5
10
WebQSP 51.2%
27.4%
8.3%
12.1%
CWQ
70.6%
19.4%
4.0% 6.0%
GrailQA 68.4%
16.2%
5.3%
9.9%
FreebaseQA
90.7%
9.0%
0.3%
0%
Table Percentages questions containing a answers in each dataset.
GrailQA (dev)
Model
F1 (O)
Fl (I)
FI (C)
Fl (Z)
DECAF (Single Answer)
78.7
90.2
78.7
73.7
Answer Only
54.7
59.4
38.3
59.5
LF only
72.4
88.2
76.3
63.9
DECAF (Multiple Answers)
78.1
93.5
76.7
71.9
Answer Only
60.9
70.7
44.4
63.6
LF Only
69.4
91.9
73.7
57.8
Table 8: Results comparison between single-answer generation and multi-answer generation in the direct-answerr-eeerraiion part of DECAF (BM25 + FiD-large).
A.4 DIRECT COMPARISON BETWEEN LF-EXECUTED AND GENERATED ANSWERS
Based on DECAF (FiD-large) over the GrailQA (dev) dataset, we directly compare the LF-executed answers and direct-generated
different question cafegories. Table shows different comparison results, where we calculate the percentages of questions that they perform equally well, LF better. Answer better and they both have the Fl score of We see that direct-answen generation is more advantageous on the zero-shot questions (15.3%) compared to I.I.D) (4.4%). and compositional (6.7%) ones. This also corresponds the second example in Table 14 where we see that, for zero- shot questions with schemas unseen during training, generating the correct logical forms containing the schemas can very difficult.
We also conduct similar analysis based on the number of relations in the ground-truth logical form of each question. Table T shows that direct-answer generation more advantages when the number of relations increases (7.3% 18.8% 29.1%), where the logical forms become more complicated and harder to generate.
Finally, we conduct analysis based on the number of ground-truth answers of each question. In Table I we see that logical-font generation has more advantages when the number of answers increases (from 14.8% to 82.7%). This is reasonable since the difficulty of logical form generation is not necessarily correlated with the number of answers, while this not the case for direct-answer generation.
A.5 ADDITIONAL ABLATION STUDIES
What is the best way to combine LF-executed answers and generated answers? Following Section 4.2, we report the results of answer combination over WebQSP and CWQ datasets. As shown in Table T2] increasing À can improve model performance. The optimal value is À = which
15

---

Published as a conference paper at ICLR 2023
Question Category
Overall
LLD.
Compositional 48.4%
Zero-Shot
LF better
33.0%
40.6%
23.4%
Answer better
10.8%
4.4%
6.7%
15.3%
Equal F1
39.4%
47.2%
28.1%
40.8%
Zero F1
16.7%
7.8%
16.8%
20.5%
Table We compare the Fl score per question of L.-exxeuueed answers and generated answer based on the category of each question.
#Relation
2 3
LF better
33.1%
35.7%
17.9%
Answer better
7.3%
18.8%
29.1%
Equal F1
46.4%
22.6%
9.1%
Zero F1
13.3%
22.8%
43.9%
Table 10:
We compare the FI score per question L.Feexeutee answers and generated answer based on the number of relations the ground-truth logical form of each question.
#Answer
2-4
5-9
LF better 14.8%
71.4%
74.4%
82.7%
Answer better
10.6%
13.0%
12.2%
8.1%
Equal F1 56.2%
1.0%
0.3%
0%
Zero FI
18.4%
14.6%
13.1%
9.2%
Table 11:
We compare the FI score per question LF-executed answers and generated answer based on the number of ground-truth answers of each question.
means that we can simply choose the top-1 LF-executed answer as the final answer if any of the logical forms is executable. If none of them is executable, we still use the directly-generatedd answer.
WebQSP/A
0.0
0.2
0.4
0.45
0.49
0.51
0.55
0.6
0.8
S(k) 1/k
49.8
49.8
50.6
51.0
51.3
76.7
76.7
76.9
77.1
77.1
S(k) B - ++1
49.8
51.0
51.5
51.8
51.9
75.2
75.2
75.2
76.0
77.1
CWQ/A
0.0
0.2
0.4
0.45
0.49
0.51
0.55
0.6
0.8
1.0
S(k) 1/k
50.5
50.5
52.7
53.5
54.4
68.5
68.5
68.6
68.6
68.6
S(k) = B k
50.5
53.1
54.3
54.6
54.6
68.3
68.3
68.4
68.6
68.6
Table 12: Fl scores on WebQSP and Hits@1 I scores on CWQ using DECAF (FiD-large) based on different values of x, which is the weight of LF-exccuted answers in the answer combination function. S(k) is the score function answer rank l and the generation beam size.
Oracle combination between FF-executed answers and generated answers. We conduct the oracle combination by selecting the better answer from the direct generation and LF-exccution based on its Fl score with the ground-truth answer. In Table 13 we report the oracle results compared to the original results. We see that the oracle combination has better performance compared to our original model, but that much (< points improvement), indicating that our current combination method, first LF execution then direct generation, is simple but very effective choice.
How does the size of training data affect the model performance? We study the influence of the size of the training data. Specifically, we want to study how it influences answer generation and LF generation respectively. We focus on the GrailQA dataset, and vary the number of training data from 500, 2000, 10000 to 44337 (all). As shown in Figure 44), the performance of DECAF improves as the increase of training data. More importantly, we see that the performance of LF generation improves much more significantly than DECAF (Answer Only). This shows that training the logical form generation requires more data than answer generation, because logical forms are usually more complicated than answers.
During inference, what is the effect of beam size? In this section, we study the effects of genera- tion beam size during inference. As shown in Figure b), when vary the beam size from 1,2,5,10
16

---

Published as a conference paper at ICLR 2023
WebQSP
CWQ
GrailQA (dev)
Model
F1
Hits@1
(O)
FI (I)
Fl (C)
(Z)
DECAF (FiD-large)
77.1
68.1
78.7
90.2
78.7
73.7
DECAF (FiD-large, Oracle)
79.6
70.1
80.7
91.4
80.5
76.1
DECAF (FiD-3B)
78.8
70.4
81.4
89.7
80.1
78.4
DECAF (FiD-3B, Oracle)
81.8
72.1
83.6
91.3
81.6
81.0
Comparison between our model with original combination method (first LF-executed answers then direct-generated answers) and our model with oracle combination method where select the answer with better Fl according to the ground-truth answers.
Table 13:
70
75
60
DecAF
DecAF (Answer Only)
50
DecAF Only)
40
DecAF
60
DecAF Only)
DecAF Only)
55
500
2,000
10,000 # #Taiining Data
44,337 (All)
1
2
5
10
20
Bearn Size
(a) Results based on different sizes training data
(h) Results hased on different beam sizes
Figure 4: Ablation study on training data size and generation beam size over GrailQA (dev) dataset.
20, the performance of DECAF improves. However, we that the performance DECAF (LF Only) is significantly improved while DECAF (Answer Only) barely changes. This shows that the overall performance improvement mainly comes from logical form generation instead of answer generation. This is because the logical form is difficult to generate, and beam size is important to long sequence generation especially when we enumerate the generated logical forms until we find the one that is executable. However, for answer generation, which is usually in short length, beam size won't have large effect. We also show the results of different beam sizes on WebQSP and CWQ datasets in Figure where we have similar observations to GrailQA.
70
75
70
60
DecAF
Hits@1
65
DecAF (Answer Only)
55
DecAF (LF Only)
60
50-
DecAF
55
DecAF (Answer Only)
DecAF (LF Only)
50
12
20
40
12
5
Beam Size
Beam Size
(a) Results WebQSP
(b) Results CWQ
Figure 5: Ablation study generation beam size using DECAF (FiD-large).
A.6 ERROR ANALYSIS
We conduct case-based error analysis on our model. As shown in Table 14 we list 3 cases where Answer Only generation is wrong, I.F Only generation is wrong, and both of them are wrong.
17

---

Published as a conference paper at ICLR 2023
In the first example, need to find the "carlies" composition. We that the logical form is not complicated while the direct answer generation needs to reason over the completion time of compositions and find the minimum of them, which is difficult. In the second example, that the reasoning not difficult since it only contains one relation about locomotive class, but the logical form relatively long which makes the generation more challenging. We that the generated logical form misses the steam part and results in an non-executable prediction. For these two cases, DECAF can still output the correct answer due answer combination. However, this is not the case for the last example, which a compositional question involving multiple relations. We see that both the generated answer and logic form are wrong. The generated logical form neglects one join operation and makes mistake the relation about release date. This means that our model can be further improved deal with such complicated questions.
Question:
Which composition was completed the earliest?
Gold Answer:
Ce fut en mai
Gold Logical Form:
(ARGMIN music. composition music composition datte date- complete eted))
Gen Answer:
Composition for Piano Orchestra by David Bowie
Gen Logical Form:
(ARGMIN music. wwwposit ion music. composition.coo completed) Ce fut mai
LF Executed Answer:
DECAF's Answer:
Ce fut en mai
Question:
British rail class 04 belongs to which locomotive class?
Gold Answer:
0-6-0
Gold Logical Form:
(AND seil.steam.locoeee wwwoomootiieiiiiee wtw..ottwwweee......iiuoooooooo.
(JOIN Locomotive.claseee m. 02rh--)) 0-6-0
Gen Answer: Gen Logical Form:
(AND rail. www.oooooiiioo.....oooo wetptttvveeeeeeeeeeem frooniggures figuration (JOIN rail. locomot wheel con fonfi configuration.
1 lccomot ive c asses m.02rh )
LF Executed Answer:
Not Executable 0-6-0
DECAF's Answer:
Question:
Which browser was most recently released by the creators of mpd?
Gold Answer:
Internet Explorer for Mac
Gold Logical Form:
(ARGMAX (AND computer, web_browser
(JCIN (R computer.
software.devellper coppra..ootarreee (JOIN computer.
file_format. format_creaton m.0210900))) computer.software.co www..aaww....rrrst.....ea..... Webkir
Gen Answer: Gen Logical Form:
(ARGMAX (AND hemputer.web.roowar.. (JCIN (R computer.
coftware.developer serveellperssottarree software) m. 0210900))
Computer.softwate relcasc.dad Not Executable
LF Executed Answer: DECAF's Answer:
WebKit
Table 14: Case-based enror analysis over GrailQA (dev) dataset, where Gen is the abbreviation for Generated, We show cases where only generated answer is wrong, only generated LF wrong, and both of them are wrong.
18


Reviews:
[{"review_id": "sKUIMGbXqz", "reviewer": "Reviewer_B5YN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes DecAF, a model for KBQA. It jointly decodes both logical forms (LF) and text answers and combines the decoded answer(s) with the answer obtained by executing the LF over a KB. The authors show that this results in better performance than other state-of-the-art models on multiple datasets. A brief overview of the method is:\n(i) a KB is first linearized into text documents so that text-retrieval methods can be used select relevant subgraphs (BM25/dense retriever is used),\n(ii) a T5-based FiD is used to encode the concatenation of the question and passages (multiple passages are retrieved, so this is performed for each (q,p_i) pair),\n(iii) this model is trained to generate both LF and text  answers by prompting it with suitable prefixes,\n(iv) a simple scoring function is used to choose the best answer from a list of candidates containing both direct and LF-executed answers.\n\nThere are quite a few advantages of this method: \n(i) no explicit entity linking is required, \n(ii) direct decoding of answers is used as a fallback when valid LFs are not generated, thereby increasing the chances of getting the right answer,\n(iii) choosing relevant subgraphs has been a difficult problem to solve, and treating it as a text-retrieval problem sidesteps this,\n(iv) method is general enough to be applied to other tasks like QA on tables, etc.\n\nOverall the results are promising, though it struggles a bit in the zero-shot setting. The analysis and ablations answer most if not all questions which naturally arise from the Method section.\nQuestion/concern regarding the linearization of the KG: LLMs (in this case T5) are pre-trained on well-formed sentences. However, the retrieved subgraph is the form of <head relation tail> which is not well-formed. How will this affect its representation? \n\nAs such I don't find any other weaknesses in the paper.", "summary_of_the_review": "-", "strength_and_weaknesses": "-", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mzdGWyDf9W", "reviewer": "Reviewer_9YKE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method named DECAF for Question answering over knowledge bases. DECAF first retrieves relevant facts in KBs and then leverages multiple prompts to generate logical forms and direct answers using a Fushion-in-Decoder architecture. The final answer entity is produced by combining these two types of outputs. Experiments on benchmarks of WebQSP, FreebaseQA, GrailQA, and ComplexWebQuestions demonstrate the effectiveness of the proposed method.\n", "summary_of_the_review": "This paper proposes a pipelined method for question answering over KBs by jointly generating logical forms and direct answers. The motivation is clear, and the technique is empirically effective. My main concern is the lack of novelty in the whole pipeline in terms of integrating direct answers and the ones generated by logical forms. Analyses of the integration effects are also somewhat weak.", "strength_and_weaknesses": "**Strengths**\n* The motivation for combining the merits of logical forms and direct answers is reasonable. \n* The method demonstrates strong performance, especially on WebQSP, FreebaseQA, and GrailQA benchmarks.\n* The method avoids the entity linking step to locate entities by linearizing knowledge bases and incorporating retrieval.\n* The paper is well-structured and easy to follow. \n\n**Weaknesses**\n* The combination mechanism of logical forms and direct answers lacks novelty.\n* The collaboration between logical forms and direct answers can be analyzed more in-depth.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "lRSmDk1vJPq", "reviewer": "Reviewer_ZmkY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Goal: The paper addresses the brittleness challenge with semantic parsing based QA systems. For some questions models don’t produce bad logical forms that are not executable, which means they get no answer. \n\nMethod: Use a direct answer generator. Instead of using a separate answer generator, you can use the same “reader” to generate both the logical form and an answer. If the logical form executes, use its answer. Otherwise, use the direct answer.\n\nEvaluation: The paper compares against models that only use direct answer generation methods or ones that use logical form execution methods, on multiple datasets.\n\nKey Contributions: \nThe combination method and an empirical evaluation that shows it works better on multiple datasets.\nAblation studies and analysis that shed some light on joint training for logical form and answer generation, sensitivity to answer combination method, impact of retrieval system, and beam search choices. \n", "summary_of_the_review": "The main idea is well-motivated. The basic idea that combining directly generated answer when logical form based execution fails is useful but is expected. The interesting contribution perhaps is in showing we can achieve this without having to ensemble outputs from two separate systems. But otherwise I am struggling to see what new useful information we learn here. \n\nUpdate after author response: The analyses and experiments from the authors provide additional information that add value to the paper.\n", "strength_and_weaknesses": "Strengths:\n1. The main idea of combining two types of answering methods is simple, well-motivated and turns out to be effective.\n\n2. Having a single reader that can do both logical form generation and direct answer generation, rather than having two separate readers is efficient. \n\nWeaknesses:\n\n1. While I am convinced that the paper makes a convincing argument for the above contributions, it doesn’t seem interesting enough. It only appears to be a step beyond ensembling, where the systems being ensembled are just two routes to answer from the same reader.  \n\nUpdate after author response: As the authors argue in their rebuttal, this characterization is a bit restrictive and there is value to empirically demonstrating that a single reader can be used to provide two routes. \n\n2. The ablation analyses, unfortunately, don't yield any interesting insights --- what works is to take the answer from the logical form if it executes otherwise use the direct answer; joint training doesn't yield added benefits over separate readers; DPR works better than BM25; more training improves all models; more passages improves; larger beam sizes improve\n\n\n3. Combining two routes of answering is useful but it doesn't quite shed light on the failure modes of the individual methods. For example, it would be nice to gain insight on the questions for which we can get answers via the direct generation method but not via logical form execution. \n\nUpdate after author response: The authors have added additional analyses that provide more details on where the combination works and to some extent on why the combination works. \n\nAdditional Comments/Questions to the authors:\n\n1. Other types of combinations: Since you are effectively combining two types of methods for generating answers, would it be useful to think about other choices of two methods for combining answers? This could be a simple “ensembling” strategy which serves as a comparison point. The ablation you did in Table 4 is one such instance. For example, you could take one of the baseline methods for direct answer and combine it with answers from one of the execution based methods. You can also combine two methods of the same type also. \n\n2. Oracle experiments: It would be useful to know what the upper-bound performance you can hope for if you allowed an oracle to choose between the top answer from the direct and execution based method? Or is it the case that the execution method if it returns an answer is always the correct one. Examples in Table 9 in Appendix are helpful but it doesn’t quite answer the question above.\n\n3. Question difficulty: It would be useful to know the performance of the direct answer generation accuracy on the set of questions for which the logical form based execution fails. Is this accuracy the same as the accuracy over the entire set of questions? Or is this subset somehow easier/harder for direct answer generation?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X2A7c27t8-V", "reviewer": "Reviewer_Azh2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a retriever+reader model for knowledge-base question answering (KBQA) where the retriever and readers are models that are usually used for open-domain textual question answering - e.g. BM25/DPR models for retriever and FiD model as the reader. The KB triples are first linearized into text. Each triple (e1, r, e2) is converted into the textual format by using concatenating the textual name of entities and using a space delimiter (e.g. (Freescape, game engine.developer, Incentive Software) → Freescape game engine developer Incentive Software). Next, triples for an entity are grouped into a single document. Following DPR, each document is split into multiple paragraphs (each 100 tokens)\n\nGiven a query, a retriever first retrieves paragraphs (which are essentially triples). This is followed by feeding the retrieved paragraphs to an encoder-decoder FiD reader model. The technical contribution of the paper is to train a model to produce both the logical forms and directly the answers. Depending on the output type, a separate prompt is prepended to the input. \n\nThe use of the textual form for entity names also eliminates the requirement of an entity linker model which links the mentions of entities in the question to entities in the KB. Instead, that is handled by the retriever which uses lexical matching to get the relevant triples.\n\nThe answers obtained by executing the generated logical form and the answers that are generated directly are further combined using a simple weighted addition. \n\nThis proposed simple method does well on multiple KBQA benchmarks - WebQSP, CWQ, GrailQA, FreebaseQA.", "summary_of_the_review": "I think this is a simple model that works well. I think the paper will improve if it addresses why training to generate both the logical forms and answers would be better. Moreover, apart from this, the techinical novelty of the paper is rather limited. Hence, I am leaning towards a weak accept score.", "strength_and_weaknesses": "**Strengths**\n\n- There is a growing body of work in QA literature which are unifying the models that work for different QA modalities - text, KB, tables. This paper is another example that shows powerful models developed for textual QA are effective for KBQA.\n- The paper presents a simple and straightforward method that is effective on multiple KBQA benchmarks. Also developing good entity linking models has been challenging needing a lot of supervised data. Aiming to solve it by textual retrieval is simple and interesting.\n- The paper was easy to follow.\n\n**Weaknesses**\n\n- A question that this paper does not address is the intuition behind why training a model to produce both logical forms and direct answers is expected to work better? Is it because for some questions it is easier to predict the logical forms and for some it is easier to directly decode the answer and hence it is best to combine both? I think the paper would benefit with an intuition behind why the model is working better\n- The technical novelty of the paper is limited. Even though the proposed method is simple (which I believe is a strength), the use of textual QA models is not novel. The training for producing both logical forms and direct answers is new, however is presented without much motivation.\n- Question: These datasets contain many questions which have multiple answers. How do you handle that during training and testing? Is the direct-decode model trained to produce all answers?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster