SPGQA: A Dataset for Controllable Experiments on LLM and KG Fusion for Question Answering
Authors: Mikhail Salnikov, Andrey Sakhovskiy, Irina Nikishina, Aida Usmanova, Angelie Kraft, Cedric Möller, Debayan Banerjee, Junbo Huang, Longquan Jiang, Rana Abdullah, Xi Yan, Elena Tutubalina, Ricardo Usbeck, Alexander Panchenko


Reviewer paper summaries: This paper studies the KGQA with the subgraph information. A new dataset based on Wikidata KG and featured by the subgraph reasoning patterns are proposed. It fills the gap between the previous KGQA dataset such as CWQ and WebQSP that majorly based on the chain-based reasoning. Specifically, the question follows from Mintaka, the answers are derived by T5-largge, the subgraph is identified by the shortest paths in Wikidata KG. Results shows that the dataset is somehow hard in terms of precision, recall, and F1, but somehow easy in terms of Accuracy.
Summary of strengths: 1. The subgraph reasoning path is little discussed in KBQA community, and the dataset proposed is important.
2. the proposed is still challenging in terms of precision, recall, and F1 score.
3. The transformer encoder-only experiments demonstrate the necessasity of subgraph information.
Summary of weaknesses: 1. The answers are generated by the T5-large without no human annotation. The data quality might be a concern.
2. The benchmark in Table 2 does not include many recent methods.
Comments, suggestions and typos: In terms of subgraph information, I think there are some important works are missing.

- The research about knowldge graph complex query answering, which studies the complex logical structures, are missing in the related works. The complex logical structures are missing. Here are some typical dataset works (tree-like [1] subgraph [2]) that the authors might want to go through.

- Regarding the subgraph reasoning patterns, the following work [3] this year tried to tackled it.

1. Ren, H., & Leskovec, J. (2020). Beta embeddings for multi-hop logical reasoning in knowledge graphs. Advances in Neural Information Processing Systems, 33, 19716-19726.
1. Yin, H., Wang, Z., & Song, Y. Rethinking Complex Queries on Knowledge Graphs with Neural Link Predictors. In The Twelfth International Conference on Learning Representations.
1. Liu, L., Wang, Z., Qiu, R., Ban, Y., & Tong, H. (2024). Logic Query of Thoughts: Guiding Large Language Models to Answer Complex Logic Queries with Knowledge Graphs. arXiv preprint arXiv:2404.04264.
Overall assessment scores: 3.5