DomainRAG: A Chinese Benchmark for Evaluating Domain-specifc Retrieval-Augmented Generation
Authors: Shuting Wang, Jiongnan Liu, Shiren Song, Jiehan Cheng, Yuqi Fu, Peidong Guo, Kun Fang, Yutao Zhu, Zhicheng Dou


Reviewer paper summaries: The paper identified 6 main capabilities of a RAG-based system. The paper then developed a new synthetic dataset for testing these capabilities of the RAG. The dataset was generated from public Internet web pages and documents.
The paper reported the experiment result of the LLM evaluation from different perspectives (i.e., noisy reference, table, faithfulness)> Other perspectives are not thorough (i.e., intent, multi-doc).
Summary of strengths: - The paper addressed six main areas for the RAG-based system, which is great.
- The paper then creates a synthesized dataset to evaluate LLM regarding these areas.
- The paper reported evaluation results for some of the areas (e.g., noisy reference, table, faithfulness)
Summary of weaknesses: - The paper introduced a synthesized dataset. However, the quality control is barely presented (other than "manually checked"). I think this is a bad practice to create a dataset generated by GPT3/4 and then dump it onto the Internet without a rigorous quality control process and data distribution control. GPT3/4 even though can generate smooth QA, its questions are usually too simple and too shallow compared to the real human question. This analysis of the dataset is not provided. The paper did not provide detailed examples of the questions and answers (e.g., in the Appendix for some qualitative review)
- The structural (sub)dataset seems to be distant from the conventional understanding of structural reading comprehension (e.g., See https://arxiv.org/abs/2309.08872). The paper seems to consider the table HTML tag as the "structure" of the document. However, the structure of a document is much broader than just a table structure. I suggest the paper define a new term (maybe Table understanding) for this particular task to reduce the confusion.
- Even though the paper addresses the multi-doc issues, the evaluation of multi-doc is very poor. Retrieval performance is the main issue of multi-doc that the paper did not offer.
Comments, suggestions and typos: L460: missing period (".")
Overall assessment scores: 2.5