Understanding DeepResearch via Reports
Authors: name: Tianyu Fan; name: Xinyao Niu; name: Yuxiang Zheng; name: Fengji Zhang; name: Chengen Huang; name: Bei Chen; name: Junyang Lin; name: Chao Huang
Published: 2025-10-09 07:03:43

Abstract: DeepResearch agents represent a transformative AI paradigm, conducting
expert-level research through sophisticated reasoning and multi-tool
integration. However, evaluating these systems remains critically challenging
due to open-ended research scenarios and existing benchmarks that focus on
isolated capabilities rather than holistic performance. Unlike traditional LLM
tasks, DeepResearch systems must synthesize diverse sources, generate insights,
and present coherent findings, which are capabilities that resist simple
verification. To address this gap, we introduce DeepResearch-ReportEval, a
comprehensive framework designed to assess DeepResearch systems through their
most representative outputs: research reports. Our approach systematically
measures three dimensions: quality, redundancy, and factuality, using an
innovative LLM-as-a-Judge methodology achieving strong expert concordance. We
contribute a standardized benchmark of 100 curated queries spanning 12
real-world categories, enabling systematic capability comparison. Our
evaluation of four leading commercial systems reveals distinct design
philosophies and performance trade-offs, establishing foundational insights as
DeepResearch evolves from information assistants toward intelligent research
partners. Source code and data are available at:
https://github.com/HKUDS/DeepResearch-Eval.