Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to learn a multi-hop KBQA model which contains a retrieval and reasoning model that shares the same architecture. Unifying the retrieval and reasoning module let the model share more learned knowledge. Experiments show great performance on three benchmark multi-hop reasoning datasets.
strength_and_weaknesses: The propose model is simple and effective. The results are also very impressive. It can be a go-to solution for multi-hop KBQA. I have some questions about the implementation.

1. Without any of the proposed technique (w/o Pre, Trans), the model already outperforms the previous state-of-the-art. Do you know why this happen?
2. The reasoning and retrieval module share the same input structure and same model architecture, but it seems they do not share the same parameters. This sounds weird to me, and I am not sure why this will lead to improvement in model's performance. How much improvement comes from the pretraining of question-relation matching (see also Fig 3(c))? You should consider emphasizing the pretraining strategy if it leads to a big improvement.
3. What is the size of the KB you use for WebQSP and CWQ? Do you use the full Freebase?

clarity,_quality,_novelty_and_reproducibility: Overall the paper is clear to read.

The paper should include more details about key choices in the experiments, e.g. size of KB. Also, it makes me worried why the ablated results without any of the introduced technique (in Table 4) can outperform the previous state-of-the-art.
summary_of_the_review: The paper delivered good results in their experiments and the intuition of having a shared reasoning and retrieval model is promising. However, it is hard to tell whether the biggest improvement comes from the advertised techniques. Please consider emphasize on your contributions. Having good numbers itself does not warrant an acceptance of the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel solution which uses pretrained language models to fuse the retrieval and reasoning stages for knowledge graph question answering. The authors also conduct extensive experiments to verify the effectiveness of the proposed model. 
strength_and_weaknesses: Strength.
This paper proposes a novel solution which uses pretrained language models to fuse the retrieval and reasoning stages for knowledge graph question answering.

Weakness.
1.	Please present more implementation details about the baselines. 

2.	It seems that most of the baselines are GNN-based models, which do not include additional knowledge. Whereas the solution in this paper uses pretrained models to introduce additional knowledge, which makes the experiments less comparable and persuasive. 

3.	To render this paper more convincing, I suggest the authors present more baseline models which use similar pretrained models to do this task. To name a few:
a.	Xin Huang, Jung-Jae Kim, and Bowei Zou. 2021. Unseen Entity Handling in Complex Question Answering over Knowledge Base via Language Generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 547–557, Punta Cana, Dominican Republic. Association for Computational Linguistics.

b.	Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum. 2021. Case-based Reasoning for Natural Language Queries over Knowledge Bases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9594–9611, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
clarity,_quality,_novelty_and_reproducibility: This paper is well written and the solution is somewhat novelty. 
summary_of_the_review: This paper proposes a novel solution which uses pretrained language models to fuse the retrieval and reasoning stages for knowledge graph question answering. The paper is well written and the solution is somewhat novelty. However, the experiments lack comparison with some latest work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a model to do multi-hop question-answering over Knowledge Graphs. The core model responsible for relevance is trained once and shared between the initial retrieval of the subgraph and the detailed reasoning phases. A key claim of the paper is that by unifying the model for the two phases, we can get better quality in the answers than doing them separately.

The proposed UniKGQA model consists of two main parts: Pretrained Language Model (PLM)-based semantic matching and the propagation of matching information through the graphs. The PLM (RoBERTa in implementation) is fine-tuned with task specific data to capture the connection between entities and relationships. Its parameters are shared between the retrieval and reasoning phases.

Evaluation was done on three different datasets. The UniKGQA model was compared with recent baselines and showed on-par results on two of them and significantly better results on WebQSP. The paper included detailed ablation study and different fine-tuning setups to show the benefit of using a joint model for two phrases.

strength_and_weaknesses: Strength

1) Evaluation results: The evaluation results on WebQSP showed clear advantages of the proposed model. Ablation study and comparison under different setup helped support the paper's claim.

Weakness

1) Writing can be improved. Key abbreviations are not explained at first use (PLM / PPR / topic entity etc, it's always good to reduce reader's guess work). There are typos and proof-reading misses. 

2) The explanation of Abstract Subgraph is not clear. What if the tail entities in the same set branch out to different 2nd-hop entities? It's not all clear to the Reviewer. An example would help.

3) It's better to explain more on the key differences between the WebQSP/CWQ sets and MetaQA, as UniKGQA shows clear improvements on the former but not the latter. Giving some examples would help too. 

List of concrete issues that should be improved.
1) Figure 2. the order of h_m and h_q seems reversed.
2) Page 4, MIP section, it's better to explain "topic entity" when used first time.
3) Page 4 to the bottom, "are a learnable vector" -> "is a ..."; "of if the entities" -> "of the entities being"?
4) Page 5 to the bottom, "Eq 4" should be "Eq 6"?
5) P7 Section 4.2 second paragraph, it's not clear if unified model is the key reason of improvement, we need more supporting evidence.
6) P8 Table 4, Why use "Trans"?  
clarity,_quality,_novelty_and_reproducibility: Clarity can be improved. The writing could be improved. Figures are too complicated. 

Quality of the work is ok: the paper did extensive experiments on different dataset to help understand the performance deltas.

Novelty is less significant: Using the sample pre-trained model feels more incremental and the benefit is only justified by better evaluation results.

Reproducibility is not clear. The authors didn't mention open-source or other means to reproduce.
summary_of_the_review: The Reviewer suggest weak accept of the paper after fixing the writing issues.

The paper's main contribution is clear and justified by the evaluation result, that using a unified model does show advantages in overall and break-down comparison with other recent methods.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new approach for multi-hop question answering over knowledge graphs (KGQA). Most current KGQA systems have a separate retrieval stage, in which relevant parts of the knowledge graph are selected, followed by a reasoning stage aiming at obtaining the correct answer. The proposed approach unifies both stages.

Strengths: 
- Simple and effective model
- Clear main contribution
- Strong evaluation results

Weaknesses:
- Doubts about some claims made in the paper
- Some parts of the writing and explanation could be further improved
justification_for_why_not_higher_score: While the paper presents a simple and effective model, some weaknesses remain, which made the acceptance decision itself difficult. Therefore, I believe it should not be presented as a spotlight paper at ICLR.
justification_for_why_not_lower_score: The paper provides a new solution to a problem with a significant impact. While some weaknesses were pointed out, they concern mostly the writing style and explanations rather than the core contributions of the paper. Therefore, I recommend acceptance.
Decision: Accept: poster