Self-VEQA Agent Self-Verification Enhanced Question Answering Agent
Authors: Zhehuan ZHAO, Menghan Wang, WenshuoFeng, Bo Xu


Reviewer paper summaries: This paper investigates the KBQA and proposes a framework with LLM based on a QA agent and a verification agent. The QA agent answers the questions with the inference chain, and the verification agent tests whether the verification chain is correct. The input of verification is the output QA agent combined with the subgraph from KG. The output of the verification agent (those verified as correct) is stored in a ``memory'' and then injected into the prompt of the QA agent. The results are conducted on some classic KBQA datasets.
Summary of strengths: - The proposed method is simple and demonstrates some potential for generalizing to more complex scenarios.
- The framework's performance is better than that of language models directly.
Summary of weaknesses: 1. Some part of this paper definitely needs revision. For example, in line 240, this sentence is redundant. Also, in Section 3.3, the paragraphs are broken.
2. Some non-agent baselines are missing, such as RoG [1], which outperform the proposed approach by a large margin.

[1] Luo, L., Li, Y. F., Haffari, G., & Pan, S. (2023). Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310.01061.

3. Another question is that in the Ablation study Table 2, the QA agent (see -mem and -ver setting, should be the QA module itself and GPT-3.5-turbo solely) performs much better than the reported ChatGPT in Table 1. One might suspect that the LLM evaluations in Table 1. is biased. The actual performance of one LLM can be better with more advanced modules such as GPT-4 or GPT-4o and using better prompts. To what extent the Self-VEQA Agents can outperform LLM baselines with more advanced but still simple tricks is unclear.

4. Lastly, the process of searching on the knowledge graph is not explained. I didn't find any details after reading the Section 3.2. Are there any algorithms?
Comments, suggestions and typos: see above.
Overall assessment scores: 1