Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Goal: The paper addresses the brittleness challenge with semantic parsing based QA systems. For some questions models don’t produce bad logical forms that are not executable, which means they get no answer. 

Method: Use a direct answer generator. Instead of using a separate answer generator, you can use the same “reader” to generate both the logical form and an answer. If the logical form executes, use its answer. Otherwise, use the direct answer.

Evaluation: The paper compares against models that only use direct answer generation methods or ones that use logical form execution methods, on multiple datasets.

Key Contributions: 
The combination method and an empirical evaluation that shows it works better on multiple datasets.
Ablation studies and analysis that shed some light on joint training for logical form and answer generation, sensitivity to answer combination method, impact of retrieval system, and beam search choices. 

strength_and_weaknesses: Strengths:
1. The main idea of combining two types of answering methods is simple, well-motivated and turns out to be effective.

2. Having a single reader that can do both logical form generation and direct answer generation, rather than having two separate readers is efficient. 

Weaknesses:

1. While I am convinced that the paper makes a convincing argument for the above contributions, it doesn’t seem interesting enough. It only appears to be a step beyond ensembling, where the systems being ensembled are just two routes to answer from the same reader.  

Update after author response: As the authors argue in their rebuttal, this characterization is a bit restrictive and there is value to empirically demonstrating that a single reader can be used to provide two routes. 

2. The ablation analyses, unfortunately, don't yield any interesting insights --- what works is to take the answer from the logical form if it executes otherwise use the direct answer; joint training doesn't yield added benefits over separate readers; DPR works better than BM25; more training improves all models; more passages improves; larger beam sizes improve


3. Combining two routes of answering is useful but it doesn't quite shed light on the failure modes of the individual methods. For example, it would be nice to gain insight on the questions for which we can get answers via the direct generation method but not via logical form execution. 

Update after author response: The authors have added additional analyses that provide more details on where the combination works and to some extent on why the combination works. 

Additional Comments/Questions to the authors:

1. Other types of combinations: Since you are effectively combining two types of methods for generating answers, would it be useful to think about other choices of two methods for combining answers? This could be a simple “ensembling” strategy which serves as a comparison point. The ablation you did in Table 4 is one such instance. For example, you could take one of the baseline methods for direct answer and combine it with answers from one of the execution based methods. You can also combine two methods of the same type also. 

2. Oracle experiments: It would be useful to know what the upper-bound performance you can hope for if you allowed an oracle to choose between the top answer from the direct and execution based method? Or is it the case that the execution method if it returns an answer is always the correct one. Examples in Table 9 in Appendix are helpful but it doesn’t quite answer the question above.

3. Question difficulty: It would be useful to know the performance of the direct answer generation accuracy on the set of questions for which the logical form based execution fails. Is this accuracy the same as the accuracy over the entire set of questions? Or is this subset somehow easier/harder for direct answer generation?

clarity,_quality,_novelty_and_reproducibility: The paper is well-written. Methods, experimental setup, and analyses are described clearly and should be reproducible. 


summary_of_the_review: The main idea is well-motivated. The basic idea that combining directly generated answer when logical form based execution fails is useful but is expected. The interesting contribution perhaps is in showing we can achieve this without having to ensemble outputs from two separate systems. But otherwise I am struggling to see what new useful information we learn here. 

Update after author response: The analyses and experiments from the authors provide additional information that add value to the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a retriever+reader model for knowledge-base question answering (KBQA) where the retriever and readers are models that are usually used for open-domain textual question answering - e.g. BM25/DPR models for retriever and FiD model as the reader. The KB triples are first linearized into text. Each triple (e1, r, e2) is converted into the textual format by using concatenating the textual name of entities and using a space delimiter (e.g. (Freescape, game engine.developer, Incentive Software) → Freescape game engine developer Incentive Software). Next, triples for an entity are grouped into a single document. Following DPR, each document is split into multiple paragraphs (each 100 tokens)

Given a query, a retriever first retrieves paragraphs (which are essentially triples). This is followed by feeding the retrieved paragraphs to an encoder-decoder FiD reader model. The technical contribution of the paper is to train a model to produce both the logical forms and directly the answers. Depending on the output type, a separate prompt is prepended to the input. 

The use of the textual form for entity names also eliminates the requirement of an entity linker model which links the mentions of entities in the question to entities in the KB. Instead, that is handled by the retriever which uses lexical matching to get the relevant triples.

The answers obtained by executing the generated logical form and the answers that are generated directly are further combined using a simple weighted addition. 

This proposed simple method does well on multiple KBQA benchmarks - WebQSP, CWQ, GrailQA, FreebaseQA.
strength_and_weaknesses: **Strengths**

- There is a growing body of work in QA literature which are unifying the models that work for different QA modalities - text, KB, tables. This paper is another example that shows powerful models developed for textual QA are effective for KBQA.
- The paper presents a simple and straightforward method that is effective on multiple KBQA benchmarks. Also developing good entity linking models has been challenging needing a lot of supervised data. Aiming to solve it by textual retrieval is simple and interesting.
- The paper was easy to follow.

**Weaknesses**

- A question that this paper does not address is the intuition behind why training a model to produce both logical forms and direct answers is expected to work better? Is it because for some questions it is easier to predict the logical forms and for some it is easier to directly decode the answer and hence it is best to combine both? I think the paper would benefit with an intuition behind why the model is working better
- The technical novelty of the paper is limited. Even though the proposed method is simple (which I believe is a strength), the use of textual QA models is not novel. The training for producing both logical forms and direct answers is new, however is presented without much motivation.
- Question: These datasets contain many questions which have multiple answers. How do you handle that during training and testing? Is the direct-decode model trained to produce all answers?
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written and was easy to follow

Quality: The proposed method is simple and the efficacy of the model is tested on multiple KBQA benchmarks. I would say that the work is of high quality

Novelty: The technical novelty of the work is rather limited. 

Reproducibility: The authors promise that the code will be opensourced. The retriever and reader models are fairly standard and well-known and should aid in reproducibility
summary_of_the_review: I think this is a simple model that works well. I think the paper will improve if it addresses why training to generate both the logical forms and answers would be better. Moreover, apart from this, the techinical novelty of the paper is rather limited. Hence, I am leaning towards a weak accept score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method named DECAF for Question answering over knowledge bases. DECAF first retrieves relevant facts in KBs and then leverages multiple prompts to generate logical forms and direct answers using a Fushion-in-Decoder architecture. The final answer entity is produced by combining these two types of outputs. Experiments on benchmarks of WebQSP, FreebaseQA, GrailQA, and ComplexWebQuestions demonstrate the effectiveness of the proposed method.

strength_and_weaknesses: **Strengths**
* The motivation for combining the merits of logical forms and direct answers is reasonable. 
* The method demonstrates strong performance, especially on WebQSP, FreebaseQA, and GrailQA benchmarks.
* The method avoids the entity linking step to locate entities by linearizing knowledge bases and incorporating retrieval.
* The paper is well-structured and easy to follow. 

**Weaknesses**
* The combination mechanism of logical forms and direct answers lacks novelty.
* The collaboration between logical forms and direct answers can be analyzed more in-depth.
clarity,_quality,_novelty_and_reproducibility: The writing and presentation of this paper are good, and the proposed method is well-motivated. Mitigating the non-execution issue of logical form generation is a problem of significance in semantic parsing-based KBQA methods. The incorporation of a retriever and Fushion-in-Decoder is a good design. But the way to combine executing results of logical forms and answers directly generated is straightforward, lacking technical novelty to some extent. Performance improvement over logical form-based methods can be easily expected by simply using a weighted ranking score, especially when logical forms are not executable. Related to this concern, how these two types of generation targets complement each other deserves more in-depth discussion, in addition to the analyses on lambda. For example,  in addition to considering the executability of logical forms, on what kinds of questions can each method demonstrate its unique merits？ And how does the complementary effect form? 

summary_of_the_review: This paper proposes a pipelined method for question answering over KBs by jointly generating logical forms and direct answers. The motivation is clear, and the technique is empirically effective. My main concern is the lack of novelty in the whole pipeline in terms of integrating direct answers and the ones generated by logical forms. Analyses of the integration effects are also somewhat weak.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes DecAF, a model for KBQA. It jointly decodes both logical forms (LF) and text answers and combines the decoded answer(s) with the answer obtained by executing the LF over a KB. The authors show that this results in better performance than other state-of-the-art models on multiple datasets. A brief overview of the method is:
(i) a KB is first linearized into text documents so that text-retrieval methods can be used select relevant subgraphs (BM25/dense retriever is used),
(ii) a T5-based FiD is used to encode the concatenation of the question and passages (multiple passages are retrieved, so this is performed for each (q,p_i) pair),
(iii) this model is trained to generate both LF and text  answers by prompting it with suitable prefixes,
(iv) a simple scoring function is used to choose the best answer from a list of candidates containing both direct and LF-executed answers.

There are quite a few advantages of this method: 
(i) no explicit entity linking is required, 
(ii) direct decoding of answers is used as a fallback when valid LFs are not generated, thereby increasing the chances of getting the right answer,
(iii) choosing relevant subgraphs has been a difficult problem to solve, and treating it as a text-retrieval problem sidesteps this,
(iv) method is general enough to be applied to other tasks like QA on tables, etc.

Overall the results are promising, though it struggles a bit in the zero-shot setting. The analysis and ablations answer most if not all questions which naturally arise from the Method section.
Question/concern regarding the linearization of the KG: LLMs (in this case T5) are pre-trained on well-formed sentences. However, the retrieved subgraph is the form of <head relation tail> which is not well-formed. How will this affect its representation? 

As such I don't find any other weaknesses in the paper.
strength_and_weaknesses: -
clarity,_quality,_novelty_and_reproducibility: -
summary_of_the_review: -
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new method for question answering over knowledge bases (KB), by jointly generating logical forms that can be executed on the KB and by generating the answers directly. The two results are then combined to get the best of both worlds, leading to state-of-the-art results on benchmarks.

Overall, while the reviewers were a bit concerned by the lack of novelty, as the proposed method can be seen as a combination of existing works, they also believed that this method is simple and more efficient than existing work. One interesting contribution of the paper is the removal of entity linking, by instead relying on standard information retrieval over linearised KB. This makes the application of the method easier over new domains or KB, and potentially opens the integration of such systems to more general QA models. Because the method is simple, potentially leading to a more widespread adoption of such techniques, and the empirical results are strong, I recommend to accept the paper.

During the discussion with reviewers, they suggested that some ablations (comparison to ensembling, performance on different types of questions) should be moved to the main paper as they would make it stronger.
justification_for_why_not_higher_score: Limited technical novelty, application might be of limited interest of the ICLR community (in term of number of people).
justification_for_why_not_lower_score: Simple method which works well, solid experimental results, paper well written.
Decision: Accept: poster