ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
Authors: name: Minghao Li; name: Ying Zeng; name: Zhihao Cheng; name: Cong Ma; name: Kai Jia
Published: 2025-08-14 03:33:43

Abstract: The advent of Deep Research agents has substantially reduced the time
required for conducting extensive research tasks. However, these tasks
inherently demand rigorous standards of factual accuracy and comprehensiveness,
necessitating thorough evaluation before widespread adoption. In this paper, we
propose ReportBench, a systematic benchmark designed to evaluate the content
quality of research reports generated by large language models (LLMs). Our
evaluation focuses on two critical dimensions: (1) the quality and relevance of
cited literature, and (2) the faithfulness and veracity of the statements
within the generated reports. ReportBench leverages high-quality published
survey papers available on arXiv as gold-standard references, from which we
apply reverse prompt engineering to derive domain-specific prompts and
establish a comprehensive evaluation corpus. Furthermore, we develop an
agent-based automated framework within ReportBench that systematically analyzes
generated reports by extracting citations and statements, checking the
faithfulness of cited content against original sources, and validating
non-cited claims using web-based resources. Empirical evaluations demonstrate
that commercial Deep Research agents such as those developed by OpenAI and
Google consistently generate more comprehensive and reliable reports than
standalone LLMs augmented with search or browsing tools. However, there remains
substantial room for improvement in terms of the breadth and depth of research
coverage, as well as factual consistency. The complete code and data will be
released at the following link: https://github.com/ByteDance-BandAI/ReportBench