CIR3: Comprehensive and Faithful Question-Answer Generation
Authors: Sami Saadaoui, Eduardo Alonso


Reviewer paper summaries: This paper studies generating QA paris with multi-agent LLMs from domain-specific documents.
The proposed CIR3 first identifies multiple subtopics from documents with few-shot prompting of LLMs.
Then a Writer is assigned to a subtopic and generates QA pairs based on some perspective.
The QA pair generation by writers continues until the curmudgeon judges that the outputs meet the requirements.
Summary of strengths: * It is an interesting idea to utilize LLMs in a multi-agent setting to generate QA pairs.
* The experimental results on two datasets show that the proposed method improves statistical, encoder-based, and LLM-based scores compared to the two baselines.
Summary of weaknesses: * I doubt that comprehensiveness is crucial for QAG. For some applications, comprehensive QA pairs for each document are not necessarily required. I think this paper lacks justification as to why comprehensive QA pairs for every document are crucial.
* For clarity, this paper may need to improve the descriptions of the proposed method. E.g., I could not figure out from the main text what exactly Transactive Reasoning is, and whether perspectives are identical to subtopics or not.
Comments, suggestions and typos: * L43: "the input answer" -> Indeed, answers are usually used as input to QG in the literature, but it is not common knowledge for readers who are not familiar with QAG.
* L149: "We shift our focus from the typical task of QG to the more challenging QAG, which is inherently more difficult due to the limited search space and increased risk of producing duplicate QA pairs." -> Many studies have worked on QAG so far, so just studying QAG cannot be counted as a contribution.
Overall assessment scores: 2.5