DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research

## Abstract

Abstract Deep research systems represent an emerging class of agentic information retrieval methods that generate comprehensive and well-supported reports to complex queries. However, most existing frameworks rely on dynamic commercial search APIs, which pose reproducibility and transparency challenges in addition to their cost. To address these limitations, we introduce DeepResearchGym, an open-source sandbox that combines a reproducible search API with a rigorous evaluation protocol for benchmarking deep research systems. The API indexes large-scale public web corpora, namely ClueWeb22 and FineWeb, using a state-of-the-art dense retriever and approximate nearest neighbor search via DiskANN. It achieves lower latency than popular commercial APIs while ensuring stable document rankings across runs, and is freely available for research use. To evaluate deep research systems’ outputs, we extend the Researchy Questions benchmark with automatic metrics through LLM-as-a-judge assessments to measure alignment with users’ information needs, retrieval faithfulness, and report quality. Experimental results show that systems integrated with DeepResearchGym achieve performance comparable to those using commercial APIs, with performance rankings remaining consistent across evaluation metrics. A human evaluation study further confirms that our automatic protocol aligns with human preferences, validating the framework’s ability to help support controlled assessment of deep research systems. Our code and API documentation are available at https://www.deepresearchgym.ai .

## 1 Introduction

Recent advances in Large Language Models (LLMs) have driven a transformation in information access paradigms, moving beyond ranked retrieval toward systems capable of synthesizing comprehensive report-style responses to complex queries. These deep research systems aim to address complex and open-ended information needs, combining iterative retrieval with multi-step reasoning and generation, autonomously navigating and evaluating diverse sources to construct well-supported reports. Prominent commercial examples include OpenAI [19] and Perplexity [24] deep research modes, which have demonstrated how these systems can significantly enhance user experience when addressing intricate questions requiring synthesis across multiple perspectives, domains, and sources. Recent industry developments further underscore this shift in information access technology, with Google moving towards AI-driven search tools [27], and Apple announcing plans to integrate services such as OpenAI and Perplexity into its Safari browser [7].

As deep research systems gain prominence, they introduce novel evaluation challenges. Being agentic by design, these systems rely on iterative search, retrieval, and reasoning over vast collections of online data, making meaningful evaluation dependent on access to environments with broad and diverse coverage that faithfully simulate real-world behavior. Yet, such infrastructures remain scarce, as efficient and stable large-scale retrieval tools are largely unavailable to the research community, forcing reliance on commercial web search APIs. While convenient, these APIs introduce critical limitations: their proprietary nature restricts transparency in the retrieval processes, hindering research on search itself, and their continuous evolution undermines reproducibility and fair benchmarking.

To address these challenges, we introduce DeepResearchGym as an open-source benchmarking framework specifically designed to enable transparent and reproducible evaluation of deep research systems. At the core of our framework is a free and open-source search API built upon public web snapshots comprising millions of documents, such as ClueWeb22 [22] and FineWeb [23]. This API exposes standardized endpoints for both document retrieval and content access, enabling integration with long-form generation pipelines.

Our search infrastructure design emphasizes transparency and reproducibility, aiming to support realistic search behavior without the variability introduced by commercial services. The retrieval pipeline consists of publicly available components, including the document collections, a state-of-the-art embedding model, and a scalable approximate nearest neighbour search index. This setup allows researchers to audit system behavior, analyze the influence of retrieved evidence, and rerun deep research experiments under reproducible search conditions, since retrieval results remain stable over time. We provide code to support local deployment of DeepResearchGym’s infrastructure, supporting full pipeline reproducibility, as well as experiments using diferent retrieval models and/or document collections. Empirical retrieval evaluations show that the system achieves strong retrieval quality with minimal loss from approximate search, while maintaining response times below those attained by commercial APIs.

Furthermore, DeepResearchGym includes a comprehensive evaluation protocol designed to assess deep research systems in realistic, challenging scenarios. We build upon the Researchy Questions dataset [28], which was initially created as a retrieval benchmark curated from commercial search logs. This dataset represents complex, high-engagement non-factoid queries, making it an interesting testbed for deep research systems. Our evaluation extension shifts the focus from assessing retrieval effectiveness to evaluating the quality of deep research systems’ responses. We employ an LLM-as-a-judge methodology [6] to automatically assess responses along critical dimensions that capture essential qualities of effective deep research reports, leveraging Researchy Questions’ ground-truth documents to provide more reliable judgments. These dimensions include alignment with user information needs, factual grounding, and overall report quality.

To empirically ground our framework, we apply DeepResearchGym’s evaluation protocol to assess a diverse set of commercial and open-source deep research systems.
Our findings highlight two key insights: first, systems maintain performance across evaluation metrics when integrated with DeepResearchGym’s search API, indicating that the our infrastructure maintains report quality on par with commercial search setups. Second, the results highlight that comprehensive coverage of user information needs is the most challenging perspective, indicating room for improvement in how current systems address complex, multi-faceted queries. Together, the results support DeepResearchGym as a promising sandbox environment for advancing research in deep information synthesis.

## 2 Related Work

Early work on Retrieval-Augmented Generation (RAG) systems focused on improving performance on knowledge-intensive question answering by retrieving supporting documents from large corpora and conditioning generation on this evidence to enhance factual accuracy [13, 37, 43]. Building on this foundation, several deep research systems have been optimized for short-form factoid-style answering. These include reinforcement learning approaches that enable search agents to autonomously navigate the web, issue iterative queries, and synthesize concise responses [10, 33, 42], as well as prompt-based methods like Search-o1 [14], which equips LLMs with the ability to trigger web searches when encountering knowledge gaps, leveraging the collected evidence to guide synthesis. While effective for short-form question answering, these approaches are not designed to support the generation of detailed reports that require broader synthesis, reasoning, and integration across multiple sources [20].

A complementary line of work has advanced towards comprehensive long-form report generation frameworks. GPTResearcher [5] orchestrates multi-agent workflows to coordinate planning, retrieval, and drafting across hybrid data sources, incorporating techniques such as report planning [38] and query decomposition [4] to enhance long-form synthesis, while maintaining coherence and completeness. Building on these paradigms, other deep research systems emphasize agentic tool use to extend reasoning capabilities beyond pure text-based retrieval. For instance, OpenDeepSearch [1] implements two agentic variants: one that follows an action-observation cycle, allowing the model to iteratively query external resources and refine its reasoning; and another that augments this by generating and executing Python scripts for more complex computational tasks. Agentic Reasoning [40] similarly combines multi-agent collaboration with code execution, contextual memory, and dynamic knowledge-graph construction via a dedicated mind-map agent, enabling structured exploration of complex problems. HuggingFace’s OpenDeepResearch initiative [9] follows similar directions in an open-source framework, integrating code agents that perform advanced retrieval, analysis, and reasoning through Python execution, while emphasizing transparency and modularity.

A common limitation across these systems is their reliance on commercial web search APIs such as Tavily [35] and SERPer [32] for document retrieval. These APIs provide limited transparency into document indexing and ranking, are subject to dynamic changes over time, and restrict researchers’ ability to fully control or replicate retrieval conditions, posing challenges for reproducibility and fair evaluation. In parallel, existing evaluation protocols have primarily focused on related but distinct tasks.
For example, widely used benchmarks such as GAIA [16], HLE [25], and FRAMES [12] were designed to assess capabilities in expert-level short-form question answering. While these benchmarks have significantly advanced evaluation methodologies, they are not tailored to the long-form synthesis and structured reporting objectives of deep research systems.

Beyond these benchmarks, efforts have been made to attain more targeted methodologies for evaluating long-form generation. FACTScore [17] and SAFE [39] decompose outputs into atomic claims and verify their factual consistency against external sources. For retrieval-augmented systems, ARES [30] and RAGChecker [29] offer modular evaluations that explicitly link generated claims to retrieved evidence, providing fine-grained diagnostics of relevance and faithfulness. Long^2RAG [26] extends this approach by introducing Key Point Recall (KPR), which evaluates how well long-form answers capture essential content from retrieved sources by measuring coverage of salient points.

## 3 DeepResearchGym

This section presents DeepResearchGym as an open-source framework designed to support reproducible research on deep research systems. To address the challenges related to the reliance on commercial web search APIs, DeepResearchGym offers a controlled sandbox environment built on large-scale web corpora. It provides a state-of-the-art retrieval API, alongside an evaluation protocol that extends the Researchy Questions dataset to measure long-form report quality.

### 3.1 Search Sandbox

This subsection introduces our search API, designed to enable reproducible retrieval for deep research systems. We begin by describing the underlying web corpora, followed by an overview of the dense retriever and the ANN indexing approach used to enable efficient search. Finally, we outline the API interface, including available endpoints, supported arguments, and response format.

#### 3.1.1 Web Corpora

DeepResearchGym currently indexes two major large-scale web datasets, namely the English subset of ClueWeb22-B [22], and the FineWeb CC-MAIN-2024-51 snapshot [23], which also contains only English documents.

ClueWeb22 was collected in 2022 and comprises approximately 10 billion web pages. It is organized into three categories, each representing different segments of the web. Category B, known as ClueWeb22-B, approximates the super head of the web, encompassing the most frequently visited pages (e.g., pages from Wikipedia, major news outlets, and other top domains). It includes around 200 million web pages, with approximately 87 million in English. These pages were sampled based on their likelihood to satisfy user information needs, as estimated by a commercial search engine’s importance scoring. Low-quality and spam pages were filtered during sampling to enhance the dataset’s overall quality. This subset is particularly valuable for systems that aim to balance broad coverage with high precision, reflecting realistic conditions encountered in commercial web search.

FineWeb is a large-scale English web corpus collected from 96 Common Crawl snapshots between 2013 and 2024. It comprises approximately 15 trillion tokens of cleaned and deduplicated web data. The dataset employs rigorous filtering, deduplication, and quality control measures, resulting in a high-quality resource for LLM training. We focus on the most recent crawl, from week 51 of 2024, which includes over 180 million documents, capturing more recent trends compared to earlier data. This makes the collection particularly valuable for queries that require up-to-date information, reflecting the evolving nature of web content and user interests.

By indexing these corpora independently, DeepResearchGym provides researchers with flexible evaluation settings: from highly curated, stable distributions (ClueWeb22) to more recent, although noisier, environments (FineWeb). This enables systematic studies of performance across different levels of document quality, topical diversity, and temporal drift.

#### 3.1.2 Search Indexes

To enable efficient state-of-the-art retrieval across our selected corpora, we built a distributed dense retrieval backend combining state-of-the-art embedding models and approximate nearest neighbor search. More specifically, we leverage the MiniCPM-Embedding-Light model [8, 21], i.e. a dense retrieval encoder trained on 260 million query-document pairs, generating 1024-dimensional document representations. The model leverages bidirectional attention mechanisms [2] and weighted mean pooling [18] to capture long-range dependencies in documents up to 8192 tokens. It achieves competitive performance on multiple benchmarks, and shows good generalization ability given a zero-shot performance of 55.27 in nDCG@10 on the BEIR benchmark [36], outperforming other popular alternatives such as bge-large-en-v1.5 [3] and jina-embeddings-v3 [11], which achieve 54.29 and 53.88 in nDCG@10, respectively.

We index these document embeddings using DiskANN [34], i.e. a graph-based approximate nearest neighbor framework optimized for billion-scale search with low latency and memory overhead. To ensure scalability, each corpus is partitioned into independent shards of up to 25 million documents, which are separately indexed for distributed deployment. During search, shards are queried in parallel, and the top-ranked results are merged to produce the final ranking, enabling efficient and flexible retrieval at web scale.

To ground the retrieval effectiveness of our search system, we evaluated it on the Researchy Queries test set, where clicked documents serve as relevance labels. Table [1](https://arxiv.org/html/2505.19253v2#S3.T1) presents the retrieval performance, considering the number of retrieved documents $K=100$, while varying $L$, i.e. a DiskANN search-time parameter that controls the size of the candidate neighbor list explored during search. Increasing $L$ typically boosts recall and ranking quality by allowing more thorough exploration of the search graph but comes at the cost of reduced query throughput. We provide metrics computed given the ground-truth clicked documents (MRR@n, nDCG@n, and R@n), as well as approximate nearest neighbor recall (ANN R@n), computed based on exact-search results. The marginal gains observed with increasing $L$ indicate that the approximation error introduced by ANN search is minimal, solidifying the retrieval quality of our infrastructure.

#### 3.1.3 Retrieval API

Figure: Table 1: Retrieval performance of the DeepResearchGym /search API as measured over the Researchy Questions test set.
Refer to caption: x1.png

DeepResearchGym provides a retrieval API designed to support deep research systems over the aforementioned corpora. The API exposes two primary endpoints: (i) the /search endpoint, which accepts a text query and returns a ranked list of documents from the selected corpus, and (ii) the /fetch endpoint, which retrieves the archived textual content of a document given its URL, as captured in the ClueWeb22 or FineWeb crawls.

The /search endpoint supports document retrieval over the previously introduced corpora, i.e. ClueWeb22 and FineWeb. By operating over these collections, it enables consistent and reproducible search results across experiments, eliminating variance caused by changing web content or live index updates. This stability is critical for benchmarking deep research systems that require dependable retrieval behavior during long-form generation. As for search-time DiskANN parameters, our API defaults to a dynamic behavior of $L=K\times 5$, since, by definition, $\min(L)=K$. Since deep research systems typically issue queries sequentially rather than in batches, we evaluate our API’s latency in this single-query setting and compare it to commercial alternatives. Figure [1](https://arxiv.org/html/2505.19253v2#S3.F1) presents the results: the left panel shows percentile-based end-to-end latency for our API across different values of $K$ (the number of retrieved documents), while the right panel compares latency against commercial APIs for $K=10$, i.e. a common setting for deep research systems. Our API consistently responds in under half a second, outperforming commercial services. Note that our reported latency values include not only ANN search, but also API-inherent network requests and query encoding, and can be further improved through batching mechanisms if necessary.

In turn, the /fetch endpoint addresses a specific challenge in deep research systems supported by static web corpora. During generation, systems retrieve documents via the /search endpoint, accessing versions captured during the crawl. Their final reports cite the original URLs associated with these documents. However, the live content of such URLs may have changed or disappeared since the original crawl. To mitigate this discrepancy, the /fetch endpoint serves archived snapshots of documents as captured during the crawl, ensuring that the original content of URLs cited in reports can be retrieved. This design enables the construction of isolated deep research pipelines that are independent of dynamic or degraded external sources. The endpoint maintains a median latency of 0.09 seconds per single request.

Unlike commercial APIs that often require paid subscriptions or enforce strict usage limits, our API is freely available for research use once users obtain access to the underlying corpora. FineWeb access is immediate, while ClueWeb22 access requires signing a lightweight data license agreement that reflects the dataset’s research-only usage terms and copyright conditions. Upon completion of this agreement, users can access the full ClueWeb22-based endpoints and optionally download the ClueWeb22-B subset for local deployment. To support this local implementation, we provide open-source code for setting up the complete retrieval pipeline. To safeguard user privacy, the API does not log query traffic unless users explicitly opt in for research participation.

### 3.2 Deep Research Evaluation Methods

To demonstrate how DeepResearchGym can support evaluation of deep research systems, we instantiate an evaluation protocol built around the Researchy Questions dataset [28]. This dataset was derived from Bing search logs, and contains complex queries associated with user engagement, making it well-suited for benchmarking systems that synthesize long-form, multi-perspective reports. While the sandbox is agnostic to the specific evaluation task and compatible with a broad range of use cases, we introduce this protocol to fill a current gap in the evaluation landscape, and to provide initial empirical observations using our API. Our protocol scores generated the reports along three axes: overall report quality, factual faithfulness to retrieved sources, and alignment with user information needs.

#### 3.2.1 Researchy Questions

Evaluating deep research systems requires queries that naturally drive extensive information exploration and synthesis. The Researchy Questions dataset [28] was curated specifically to capture such queries. Rather than featuring simple factoid questions, the dataset consists of approximately 96,000 real-world information-seeking queries that led users to engage with multiple documents during search sessions, as measured by aggregated click distributions over ClueWeb22. For reference, Appendix [A](https://arxiv.org/html/2505.19253v2#A1) shows a sample of queries together with clicked document URLs.

The heavy engagement with diverse sources reflects the essential challenges deep research systems are designed to address: synthesizing information across multiple perspectives, reconciling conflicting evidence, and constructing comprehensive responses. In contrast to short, answerable queries that focus on locating a single fact, Researchy Questions exhibit the complexity that require broader exploration and deeper understanding. While the dataset was originally introduced for evaluating retrieval performance, its properties make it a strong foundation for studying long-form generation grounded in multi-document evidence. In the next section, we describe our evaluation methodology for extending the use of Researchy Questions to benchmark deep research generation.

#### 3.2.2 Long-form Report Evaluation Metrics

Deep research systems focused on providing report-like answers face multiple challenges inherent to long-form generation evaluation [41], where outputs must be assessed not only for linguistic fluency and informativeness, but also for factual grounding and content relevance. We follow a tri-faceted evaluation framework that assesses the alignment with user information needs, factual grounding, and overall quality of generated answers.
The Appendices contains all the prompts used for LLM-based metrics (Appendix [B](https://arxiv.org/html/2505.19253v2#A2)), an example report (Appendix [C](https://arxiv.org/html/2505.19253v2#A3)), and its detailed evaluation (Appendix [D](https://arxiv.org/html/2505.19253v2#A4)).

Report Relevance: As the primary metric for assessing user satisfaction, we evaluate how well the generated reports address the user’s underlying information needs. Given that Researchy Questions are derived from real-world web search sessions, we leverage the set of documents clicked by users as a proxy for ground-truth information targets. Following the Key Point Recall (KPR) methodology [26], we extract salient points from each ground-truth document using an LLM guided by structured prompts, capturing the core content users engaged with. We then assess each generated report for semantic inclusion of these key points, computing the KPR score as $\frac{1}{M}\sum_{j=1}^{M}c_{j}$, where $M$ is the total number of key points and $c_{j}$ indicates whether key point $j$ is supported by the report, as judged by an LLM.

To complement recall, we also compute Key Point Contradiction (KPC), which measures whether the report introduces statements that conflict with any key points. This score captures potential misinformation or misleading content, defined as $\frac{1}{M}\sum_{j=1}^{M}d_{j}$, where $d_{j}$ is 1 if the report contradicts key point $j$, as judged by the same LLM used for the KPR metric. Together, these metrics provide a user-centered assessment of both coverage and factual consistency relative to real-world search intents.

Retrieval Faithfulness: Beyond relevance, we assess the factual grounding of generated reports, adapting the LLM-as-a-judge approach of the TREC-RAG evaluation process [37]. Our automatic citation evaluation pipeline follows a three-stage process. First, factual claims are extracted from the report, along with any URLs referenced as support. Second, the content of each cited source is retrieved. Third, an LLM is prompted to assess whether the cited source adequately supports the corresponding claim. This procedure captures both the presence of citations and their substantive validity.

Given a report, we compute the primary metrics established by the TREC-RAG evaluation. Citation recall measures the proportion of factual claims that include at least one citation, i.e., $\frac{N_{\text{cited}}}{N_{\text{total}}}$, where $N_{\text{cited}}$ represents the number of claims with citations and $N_{\text{total}}$ represents the total number of claims. This metric quantifies how consistently the system grounds its assertions in external evidence.

In turn, citation precision evaluates the quality of citations for claims that include references. Each claim-citation pair receives a support score $s_{i}$, where full support (score = 1) means all key aspects of the claim are fully supported by the cited source; partial support (score = 0.5) means some aspects of the claim are supported, but the support is incomplete; and no support (score = 0) means the cited source does not substantively support the claim or is irrelevant. Citation precision is then computed as the average score across all cited claims, i.e., $\frac{1}{N_{\text{cited}}}\sum_{i=1}^{N_{\text{cited}}}s_{i}$.

Report Quality: To capture aspects of writing quality and analytical depth, we employ another LLM-as-a-Judge protocol [6], prompting a strong LLM to evaluate each answer along two key dimensions: clarity, reflecting logical coherence and linguistic fluency; and insightfulness, capturing analytical nuance and the depth of reasoning presented. These dimensions are commonly used in long-form generation evaluation [15, 31] and provide complementary evidence of the overall presentation quality of the generated content.

## 4 Benchmarking Deep Research Systems

**Table 2: Comparison of deep research systems on the Researchy Questions test set using (i) each system’s original commercial search API and (ii) DeepResearchGym’s search API. Scores are judged by gpt-4.1-mini-2025-04-14. Systems marked with * are not tailored for long-report generation.**
|  | Relevance | Faithfulness | Quality |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| System | Commercial | DRGym | Commercial | DRGym | Commercial | DRGym |  |  |  |  |  |  |
| KPR | KPC | KPR | KPC | Precision | Recall | Precision | Recall | Clarity | Insight | Clarity | Insight |  |
| perplexity-sonar-deepsearch | 72.50 | 1.12 | – | – | 55.65 | 99.22 | – | – | 89.50 | 89.26 | – | – |
| gpt4-search-preview | 40.01 | 1.69 | – | – | 57.68 | 56.11 | – | – | 70.13 | 59.13 | – | – |
| GPT-Researcher | 60.61 | 1.52 | 64.67 | 1.42 | 89.11 | 94.29 | 85.36 | 90.82 | 86.37 | 81.52 | 83.70 | 78.01 |
| OpenDeepSearch | 32.92 | 0.97 | 42.81 | 0.84 | 85.86 | 97.78 | 81.32 | 94.82 | 59.20 | 47.04 | 61.48 | 49.51 |
| HuggingFace-DeepSearch | 33.00 | 0.81 | 35.22 | 1.35 | 0.35 | 0.29 | 0.10 | 0.10 | 57.52 | 47.98 | 58.34 | 52.36 |
| Search-o1^∗ | 28.92 | 0.34 | 29.93 | 0.38 | 0.00 | 0.00 | 0.00 | 0.00 | 29.38 | 36.81 | 30.31 | 37.87 |
| Search-R1^∗ | 5.52 | 0.81 | 4.95 | 0.80 | 0.00 | 0.00 | 0.00 | 0.00 | 9.48 | 11.87 | 9.07 | 11.18 |

This section reports empirical results from benchmarking a diverse set of deep research systems using our evaluation protocol. We compare performance across retrieval settings, analyze per-query consistency, and validate metric reliability through human judgments.

### 4.1 Experimental Setup

To evaluate the current landscape of deep research systems, we conducted a systematic benchmarking study, following the protocol described in Section [3.2.2](https://arxiv.org/html/2505.19253v2#S3.SS2.SSS2) with gpt-4.1-mini-2025-04-14 as the LLM judge. We use a subset of the previously introduced Researchy Questions dataset, namely the top 1,000 queries from the test set, ranked by the number of documents clicked during the original search sessions. This ranking naturally favors queries that drive extensive exploration, aligning with the goals of deep research systems.

We evaluate a diverse set of deep research systems spanning both commercial and open-source implementations. The commercial systems include gpt4-search-preview from OpenAI and sonar-deepresearch from Perplexity, which represent the strongest variants available through the respective APIs. On the open-source side, we include GPT-Researcher and HuggingFace DeepSearch. All four systems are capable of generating long-form reports. We also evaluate three academic systems. OpenDeepSearch produces similarly comprehensive outputs, while Search-o1 and Search-R1 focus on concise, short-form answers. Although not designed for deep research tasks, these last two systems serve as lower-bound references and help verify that our evaluation metrics capture meaningful differences in generative capabilities. All systems are evaluated using their default configurations.

Experiments leveraging DeepResearchGym’s search API default to the ClueWeb22-B corpus, given the higher alignment with the Researchy Questions benchmark. Recall that, as discussed in Section [3.1.3](https://arxiv.org/html/2505.19253v2#S3.SS1.SSS3), FineWeb is also available through our API endpoints, and, unlike ClueWeb22, this does not require signing the research-use license.

### 4.2 System-level Evaluation

Table [2](https://arxiv.org/html/2505.19253v2#S4.T2) presents evaluation results for each system under two distinct retrieval configurations: (1) using the system’s original commercial search API, and (2) using the standardized DeepResearchGym search API. The results reveal several important insights. First, systems generally maintain their relative performance rankings across both retrieval settings, confirming that DeepResearchGym’s search API provides sufficient retrieval quality to support effective report generation.

Second, we observe consistent patterns in the relative difficulty of different evaluation dimensions. Even top-performing systems like perplexity-sonar-deepsearch and GPT-Researcher achieve notably higher scores in report quality metrics (Clarity, Insight) compared to information coverage metrics (KPR), suggesting that linguistic fluency has outpaced comprehensive content synthesis. This pattern holds across both retrieval environments, indicating an intrinsic challenge in deep research that transcends retrieval infrastructure.

Third, significant performance gaps exist in citation practices. Systems like GPT-Researcher and OpenDeepSearch demonstrate strong citation precision and recall, effectively grounding factual claims in retrieved evidence. In contrast, systems such as HuggingFace’s deepresearch exhibit near-zero citation metrics, stemming from fundamental design choices that omit explicit citation mechanisms. This highlights a key distinction in how different approaches conceptualize evidence grounding, whether through explicit source attribution or implicit knowledge integration.

The evaluation also reveals an interesting trade-off in commercial systems, which tend to achieve excellent narrative quality but sometimes at the expense of granular citation precision. Manual inspection indicates two common patterns: (1) these systems often use citations to justify broader report sections rather than specific claims, and (2) they sometimes reference URLs that cannot be fully crawled. This suggests a potential tension between optimizing for narrative coherence versus precise evidence anchoring in current system designs.

### 4.3 Query-level Analysis

Figure: Figure 2: Query-level correlation between commercial search APIs and DeepResearchGym API evaluation metrics. The empty plot reflects a non-citation system.
Refer to caption: x3.png

To further investigate the consistency of system performance across individual queries, we conducted a fine-grained analysis comparing results obtained under each system’s original retrieval API, against those those from DeepResearchGym’s API, focusing only on the systems explicitly geared towards long-report generation. Figure [2](https://arxiv.org/html/2505.19253v2#S4.F2) presents scatter plots of per-query scores across our three evaluation axes. The Spearman correlation coefficients quantify the alignment between retrieval configurations.

The analysis reveals distinct patterns across evaluation dimensions. For relevance (KPR), stronger systems exhibit moderate to high correlation, indicating that query-level retrieval effectiveness is largely preserved when transitioning to DeepResearchGym’s corpus. However, mid-range queries show some score variability, suggesting that certain information needs are more sensitive to differences in retrieval infrastructure. In contrast, report quality metrics demonstrate lower per-query correlation, despite high absolute scores for top systems. This implies that while narrative fluency and coherence are robust to retrieval changes, they are not tightly coupled with individual query characteristics.

Retrieval faithfulness shows the lowest per-query correlation across systems, indicating that this dimension is sensitive to differences in retrieved evidence. Changes in the retrieved documents can shift not only how well claims are supported, but also the claims themselves, leading to variation in citation faithfulness scores across retrieval setups. While average scores across queries remain stable, with some individual queries yielding consistently high scores across both sources, the broader pattern lacks alignment, with most points scattered and with no clear linear trend. This variability underscores the importance of using a standard retrieval API when benchmarking deep research systems, as it helps control for retrieval effects and ensures that observed differences stem from model behavior rather than different access to evidence.

### 4.4 Human Evaluation

To validated our automatic evaluation protocol and assess its alignment with human judgments, we conduct a systematic human evaluation study. The evaluation set comprises 210 queries with their corresponding generated reports. For each query, we present annotators with two reports generated by different systems, asking them to select the better one based on overall informativeness, coherence, and factual accuracy. Seven annotators (all also co-authors of this paper) participated in the study, with each query independently evaluated by three different individuals. To mitigate potential bias, the evaluation was conducted double-blind: annotators were unaware of which systems produced each report, and system assignments were randomized across queries. Report presentation order was also randomized to prevent positional bias, and ties were disallowed to enforce binary preferences.

Figure: Table 3: LLM versus human agreement (Cohen’s $\kappa$).
[Uncaptioned image]: x4.png

To assess inter-annotator reliability, we compute pairwise Cohen’s $\kappa$ scores across all annotated comparisons. The average pairwise $\kappa$ is 0.87, indicating a high level of agreement and supporting the consistency of human judgments. Building on this, Table [4.4](https://arxiv.org/html/2505.19253v2#S4.SS4) presents Cohen’s $\kappa$ values measuring agreement between LLM-based automatic judgments and human preferences across each evaluation dimension, while Figure [3](https://arxiv.org/html/2505.19253v2#S4.F3) presents pairwise win rates, showing the percentage of times each system was preferred over another based on human annotations. The KPC metric was excluded from the analysis in Table [4.4](https://arxiv.org/html/2505.19253v2#S4.SS4) due to a high rate of LLM-judged ties, leaving insufficient samples for analysis. The results demonstrate substantial agreement and the same trend regarding relative model performance. This validates our LLM-as-a-judge evaluation protocol, strengthening confidence in the benchmark results presented in the previous sections.

## 5 Conclusion and Future Work

DeepResearchGym offers a reproducible sandbox for developing and benchmarking deep research systems, providing a stable alternative to commercial search APIs. By anchoring retrieval to high-quality web corpora and combining the search API with a multi-dimensional evaluation protocol, our framework enables controlled benchmarking of systems that generate long-form reports.

Our results demonstrate that DeepResearchGym’s search API provides a reliable research-grade substitute for commercial retrieval infrastructures. Across multiple evaluation axes, all evaluated systems maintain comparable performance when transitioning from proprietary search APIs to our transparent and reproducible retrieval environment, affirming that DeepResearchGym preserves the retrieval fidelity necessary to support complex report-style synthesis. By isolating system behavior from fluctuating retrieval conditions, DeepResearchGym offers a stable foundation for reproducible and fair benchmarking of deep research systems.

Future extensions to DeepResearchGym can perhaps expand the coverage to larger and more recent web corpora, enabling evaluation of time-sensitive queries and emerging topics. Moreover, the integration of domain-specific benchmarks may further support assessment in high-stakes contexts such as healthcare or law, where retrieval precision and factual reliability are critical. Finally, the platform opens space for testing new agent architectures that incorporate structured planning, iterative search, and reinforcement learning, advancing research on deep research systems.

Limitations: Current limitations include: the reliance of deep research systems on proprietary and evolving LLMs for generation, which restricts full output reproducibility; evaluation variance introduced by LLM-as-a-judge protocols despite prompt/output standardization; and the use of static web corpora, which limits coverage of emerging or time-sensitive content. Ethical risks also apply, such as amplification of bias, misinformation, or misuse of retrieved content, although DeepResearchGym is targeted at academic research, instead of real-world deployments.

## Acknowledgments and Disclosure of Funding

## References

- Alzubi et al. [2025]
Salaheddin Alzubi, Creston Brooks, Purva Chiniya, Edoardo Contente, Chiara von Gerlach, Lucas Irwin, Yihan Jiang, Arda Kaz, Windsor Nguyen, Sewoong Oh, Himanshu Tyagi, and Pramod Viswanath.
Open Deep Search: Democratizing Search with Open-source Reasoning Agents.
*ArXiv*, abs/2503.20201, 2025.
- BehnamGhader et al. [2024]
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy.
LLM2Vec: Large language models are secretly powerful text encoders.
In *Conference on Language Modeling, COLM*, 2024.
- Beijing Academy of Artificial Intelligence [2024]
Beijing Academy of Artificial Intelligence.
bge-large-en-v1.5, 2024.
URL [https://huggingface.co/BAAI/bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5).
Hugging Face model page.
- Bonchi et al. [2008]
Francesco Bonchi, Carlos Castillo, Debora Donato, and Aristides Gionis.
Topical query decomposition.
In *International Conference on Knowledge Discovery and Data Mining, KDD*, 2008.
- Elovic [2025]
Assaf Elovic.
GPT Researcher: LLM-based Autonomous Agent for Deep Research, 2025.
URL [https://github.com/assafelovic/gpt-researcher](https://github.com/assafelovic/gpt-researcher).
- Gu et al. [2024]
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo.
A Survey on LLM-as-a-Judge.
*ArXiv*, abs/2411.15594, 2024.
- Gurman et al. [2025]
Mark Gurman, Lydia Nylen, and Shiyin Lai.
Apple Eyes Move to AI Search, 2025.
URL [https://www.bloomberg.com/news/articles/2025-05-07/apple-working-to-move-to-ai-search-in-browser-amid-google-fallout](https://www.bloomberg.com/news/articles/2025-05-07/apple-working-to-move-to-ai-search-in-browser-amid-google-fallout).
Bloomberg News.
- Hu et al. [2024]
Shengding Hu et al.
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies.
In *Conference on Language Modeling, COLM*, 2024.
- HuggingFace [2025]
HuggingFace.
Open Deep Research, 2025.
URL [https://github.com/huggingface/smolagents/tree/main/examples/open_deep_research](https://github.com/huggingface/smolagents/tree/main/examples/open_deep_research).
- Jin et al. [2025]
Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning.
*ArXiv*, abs/2503.09516, 2025.
- JinaAI [2024]
JinaAI.
jina-embeddings-v3, 2024.
URL [https://huggingface.co/jinaai/jina-embeddings-v3](https://huggingface.co/jinaai/jina-embeddings-v3).
Hugging Face model page.
- Krishna et al. [2025]
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui.
Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation.
In *Conference of the North American Chapter of the Association for Computational Linguistics, NAACL*, 2025.
- Lewis et al. [2020]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
In *Annual Conference on Neural Information Processing Systems, NeurIPS*, 2020.
- Li et al. [2025]
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou.
Search-o1: Agentic Search-Enhanced Large Reasoning Models.
*ArXiv*, abs/2501.05366, 2025.
- Liu et al. [2023]
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.
G-eval: NLG evaluation using gpt-4 with better human alignment.
In *Conference on Empirical Methods in Natural Language Processing, EMNLP*, 2023.
- Mialon et al. [2024]
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom.
GAIA: a benchmark for general AI assistants.
In *International Conference on Learning Representations, ICLR*, 2024.
- Min et al. [2023]
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.
In *Conference on Empirical Methods in Natural Language Processing, EMNLP*, 2023.
- Muennighoff [2022]
Niklas Muennighoff.
SGPT: GPT sentence embeddings for semantic search.
*ArXiv*, abs/2202.08904, 2022.
- OpenAI [2025]
OpenAI.
Introducing Deep Research, 2025.
URL [https://openai.com/index/introducing-deep-research/](https://openai.com/index/introducing-deep-research/).
OpenAI Blog. May 11, 2025.
- OpenAI [2025]
OpenAI.
Deep Research System Card, 2025.
URL [https://cdn.openai.com/deep-research-system-card.pdf](https://cdn.openai.com/deep-research-system-card.pdf).
Accessed May 11, 2025.
- OpenBMB [2024]
OpenBMB.
MiniCPM-Embedding-Light, 2024.
URL [https://huggingface.co/openbmb/MiniCPM-Embedding-Light](https://huggingface.co/openbmb/MiniCPM-Embedding-Light).
Hugging Face model page.
- Overwijk et al. [2022]
Arnold Overwijk, Chenyan Xiong, and Jamie Callan.
ClueWeb22: 10 Billion Web Documents with Rich Information.
In *International Conference on Research and Development in Information Retrieval, SIGIR*, 2022.
- Penedo et al. [2024]
Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf.
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.
In *Annual Conference on Neural Information Processing Systems NeurIPS*, 2024.
- Perplexity AI [2025]
Perplexity AI.
Introducing Perplexity Deep Research, 2025.
URL [https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research](https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research).
Perplexity Blog. Accessed May 11, 2025.
- Phan et al. [2025]
Long Phan et al.
Humanity’s Last Exam.
*ArXiv*, abs/2501.14249, 2025.
- Qi et al. [2024]
Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, and Wei Xu.
LONG^2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall.
In *Findings of the Association for Computational Linguistics, EMNLP*, 2024.
- Reid [2024]
Elizabeth Reid.
Generative AI in Search: Let Google Do the Searching for You, 2024.
URL [https://blog.google/products/search/generative-ai-google-search-may-2024/](https://blog.google/products/search/generative-ai-google-search-may-2024/).
Accessed: 2025-05-12.
- Rosset et al. [2024]
Corby Rosset, Ho-Lam Chung, Guanghui Qin, Ethan C. Chau, Zhuo Feng, Ahmed Awadallah, Jennifer Neville, and Nikhil Rao.
Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents.
*ArXiv*, abs/2402.17896, 2024.
- Ru et al. [2024]
Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Cheng Jiayang, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Binjie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, and Zheng Zhang.
Ragchecker: A fine-grained framework for diagnosing retrieval-augmented generation.
In *Annual Conference on Neural Information Processing Systems, NeurIPS*, 2024.
- Saad-Falcon et al. [2024]
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia.
ARES: an automated evaluation framework for retrieval-augmented generation systems.
In *Conference of the North American Chapter of the Association for Computational Linguistics, NAACL*, 2024.
- Saha et al. [2024]
Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li.
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation.
In *Conference of the North American Chapter of the Association for Computational Linguistics, NAACL*, 2024.
- Serper [2025]
Serper.
Serper.dev Search API, 2025.
URL [https://serper.dev](https://serper.dev).
Accessed May 11, 2025.
- Song et al. [2025]
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen.
R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning.
*ArXiv*, abs/2503.05592, 2025.
- Subramanya et al. [2019]
Suhas Jayaram Subramanya, Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnaswamy, and Rohan Kadekodi.
DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node.
In *Annual Conference on Neural Information Processing Systems, NeurIPS*, 2019.
- Tavily [2025]
Tavily.
Tavily Search API, 2025.
URL [https://tavily.com](https://tavily.com).
Accessed May 11, 2025.
- Thakur et al. [2021]
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych.
BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.
In *Conference on Neural Information Processing System, NeurIPS*, 2021.
- Thakur et al. [2025]
Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin.
Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges.
*ArXiv*, abs/2504.15205, 2025.
- Wang et al. [2023]
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim.
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models.
In *Annual Meeting of the Association for Computational Linguistics, ACL*, 2023.
- Wei et al. [2024]
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le.
Long-form factuality in large language models.
In *Annual Conference on Neural Information Processing Systems, NeurIPS*, 2024.
- Wu et al. [2025]
Junde Wu, Jiayuan Zhu, and Yuyuan Liu.
Agentic Reasoning: Reasoning LLMs with Tools for the Deep Research.
*ArXiv*, abs/2502.04644, 2025.
- Xu et al. [2023]
Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi.
A Critical Evaluation of Evaluations for Long-form Question Answering.
In *Annual Meeting of the Association for Computational Linguistics, ACL*, 2023.
- Zheng et al. [2025]
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu.
DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments.
*ArXiv*, abs/2504.03160, 2025.
- Zhou et al. [2024]
Yujia Zhou, Yan Liu, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Zheng Liu, Chaozhuo Li, Zhicheng Dou, Tsung-Yi Ho, and Philip S. Yu.
Trustworthiness in Retrieval-Augmented Generation Systems: A Survey.
*ArXiv*, abs/2409.10102, 2024.

## Appendix A Researchy Questions

Table [4](https://arxiv.org/html/2505.19253v2#A1.T4) shows a sample of 5 queries from the Researchy Questions [28] dataset, along with hyperlinks to 10 of its user-clicked documents.

**Table 4: Sample of the Researchy Questions dataset.**
| Query | References |
| --- | --- |
| Is the COVID vaccine dangerous | [Link1](https://hopkinsmedicine.org/health/conditions-and-diseases/coronavirus/is-the-covid19-vaccine-safe),<br>[Link2](https://hopkinsmedicine.org/health/conditions-and-diseases/coronavirus/covid-19-vaccines-myth-versus-fact),<br>[Link3](https://uab.edu/news/health/item/12143-three-things-to-know-about-the-long-term-side-effects-of-covid-vaccines),<br>[Link4](https://theburningplatform.com/2021/11/04/the-covid-vaccines-are-the-most-dangerous-vaccines-in-human-history-by-a-long-shot),<br>[Link5](https://jayceescommentaries.wordpress.com/2021/06/25/the-covid-vaccines-are-toxicdeadly-and-harmful-to-your-health),<br>[Link6](https://peckford42.wordpress.com/2021/09/18/the-dangers-of-covid-19-booster-shots-and-vaccines-boosting-blood-clots-and-leaky-vesselsnew-discoveries-in-the-immunology-of-sars-cov-2-and-covid-19-vaccines),<br>[Link7](https://undercurrents723949620.wordpress.com/2021/03/25/the-hushed-long-term-risks-of-covid-19-vaccines),<br>[Link8](https://mediamatters.org/laura-ingraham/fox-guest-says-covid-vaccine-downright-dangerous-and-will-send-you-your-doom),<br>[Link9](https://mizzima.com/article/alarm-grows-researchers-warn-dangers-covid-19-shots),<br>[Link10](https://steverotter.com/side-effects-of-covid-shot) |
| Why is there a chip shortage | [Link1](https://bbc.co.uk/news/business-58230388),<br>[Link2](https://ccm.net/computing/hardware/59-why-is-there-a-chip-shortage),<br>[Link3](https://bbc.com/news/business-58230388),<br>[Link4](https://forbes.com/sites/forbestechcouncil/2021/11/17/why-the-global-chip-shortage-is-lasting-so-long-and-why-it-will-have-a-positive-end),<br>[Link5](https://techrepublic.com/article/the-global-chip-shortage-what-caused-it-how-long-will-it-last),<br>[Link6](https://marketplace.org/2021/09/17/whats-behind-chip-shortage),<br>[Link7](https://gmauthority.com/blog/2021/08/why-is-there-a-microchip-shortage-in-the-first-place),<br>[Link8](https://the-sun.com/tech/3886855/global-chip-shortage-explained),<br>[Link9](https://rocketit.com/when-will-the-chip-shortage-end),<br>[Link10](https://tomsguide.com/news/the-global-chip-shortage-may-be-coming-to-an-end-heres-why) |
| Can there be knowledge that is independent of culture? | [Link1](https://ukessays.com/essays/philosophy/the-knowledge-which-are-independent-of-our-culture-philosophy-essay.php),<br>[Link2](https://graduateway.com/is-our-knowledge-dependent-of-our-culture),<br>[Link3](https://gradepanda.com/can-there-be-a-knowledge-that-is-independent-of-culture),<br>[Link4](https://studymode.com/subjects/can-we-have-beliefs-or-knowledge-which-are-independent-of-our-culture-page1.html),<br>[Link5](https://criticalhomework.com/can-there-be-a-knowledge-that-is-independent-of-culture),<br>[Link6](https://quora.com/can-there-be-knowledge-that-is-independent-of-culture),<br>[Link7](https://usaelitewriters.com/can-there-be-a-knowledge-that-is-independent-of-culture),<br>[Link8](https://bartleby.com/essay/can-beliefs-or-knowledge-be-independent-of-p3r84ejdm6s),<br>[Link9](https://ukessays.com/essays/philosophy/beliefs-or-knowledge-that-is-independent-of-culture-philosophy-essay.php),<br>[Link10](https://assignmentden.com/can-there-be-a-knowledge-that-is-independent-of-culture) |
| Why gas prices are so high | [Link1](https://poynter.org/fact-checking/2021/why-are-gas-prices-so-high-and-who-is-to-blame),<br>[Link2](https://cbsnews.com/news/gas-prices-high-profiteers-biden),<br>[Link3](https://forbes.com/sites/davidblackmon/2021/03/25/gasoline-prices-are-high-and-going-higherheres-why),<br>[Link4](https://instituteforenergyresearch.org/gas/why-are-gas-prices-so-high),<br>[Link5](https://metro.co.uk/2021/09/20/why-are-gas-prices-so-high-in-the-uk-and-are-they-rising-15287056),<br>[Link6](https://fool.ca/2021/10/14/heres-exactly-why-gasoline-prices-are-going-crazy),<br>[Link7](https://nationalinterest.org/blog/reboot/joe-biden-reason-we-have-high-gas-prices-186656),<br>[Link8](https://spectator.org/gas-prices-rising),<br>[Link9](https://bbc.co.uk/news/business-58090533?at_medium=rss&at_campaign=karanga),<br>[Link10](https://bbc.co.uk/news/business-58090533.amp) |
| Does religion cause war | [Link1](https://compellingtruth.org/religion-cause-wars.html),<br>[Link2](https://huffpost.com/entry/is-religion-the-cause-of-_b_1400766),<br>[Link3](https://deadlineessay.com/does-religion-cause-war),<br>[Link4](https://bethinking.org/is-religion-harmful/does-religion-cause-war),<br>[Link5](https://ivoryresearch.com/samples/essay-on-does-religion-cause-war),<br>[Link6](https://bartleby.com/essay/does-religion-cause-war-f3caegpztc),<br>[Link7](https://whistlinginthewind.org/2012/05/22/religion-as-a-cause-of-war),<br>[Link8](https://huffingtonpost.co.uk/2014/11/14/religions-war-cause-responsible-evidence_n_6156878.html),<br>[Link9](https://bulletin.hds.harvard.edu/does-religion-cause-violence),<br>[Link10](https://bartleby.com/essay/religion-causes-war-pknhl43tc) |

## Appendix B LLM-as-judge Prompts

This section details all the prompts used throughout this work for LLM-as-a-judge evaluation protocols. Note that all the provided JSON output formats were enforced through structured decoding.

### B.1 Key-point Extraction Prompt

Arguments:

- •
query: search query
- •
text: text of relevant document

Prompt:

This prompt follows the one used by Long^2RAG [26].

### B.2 Key-point Merging Prompt

Arguments:

- •
key points extracted from multiple documents

Prompt:

### B.3 Key-point Verification Prompt

Arguments:

- •
key_point: a single ground-truth key point
- •
answer: a report generatd by a DeepResearch system

Prompt:

### B.4 Claim-URL Extraction Prompt

Arguments:

- •
report: a report generated by a deep research system

Prompt:

### B.5 Qualitative Judgments

Clarity

Insightfulness

## Appendix C Example Report

This section presents an example of a question and the corresponding report generated by GPTResearcher. Appendix [D](https://arxiv.org/html/2505.19253v2#A4) shows detailed evaluation results for this report.
Query:
Why Have Used Car Prices Increased?

Generated report:

## Appendix D Evaluation Example

This section details the evaluation protocol along all described evaluation metrics for the report presented in the previous section. We start by showing the ground-truth key-points extracted from the relevant ClueWeb-22 documents, along with key-point recall and contradiction. Then, we show citation faithfulness and report quality metrics.

### D.1 Ground-truth Key Points from Clicked Documents

Query: Why Have Used Car Prices Increased?

Extracted key-points:

- 1.
Car dealers were closed during Covid-19 lockdowns, leading to fewer new cars sold and a decline in used cars being part exchanged, causing low supply in the used car market.
- 2.
A global semiconductor shortage has caused a smaller supply of new cars, leading more buyers to the used car market and causing supply and demand issues, contributing to unprecedented rises in used car prices.
- 3.
Increased demand for used cars is driven by consumers treating themselves to used cars instead of holidays, swapping expensive lease cars for affordable used models, and savings-rich customers, dealers, and rental fleets pushing up prices.
- 4.
Used car dealerships have experienced a shortage of stock as trade-ins have reduced, and decreased supply from fleet sales, repossessions, off-lease cars, and rental companies not selling used cars because they cannot buy new vehicles, shrinking supply and pushing prices up.
- 5.
New car prices are rising due to short supply, which normally caps used car prices, but now both new and used car prices are increasing simultaneously.
- 6.
Used car prices are expected to keep rising in the summer due to ongoing chip shortage and demand, but may stabilize in the fall.
- 7.
Certain car sectors like the Audi Q7, sports cars, premium cars, SUVs, diesels, and sub-£20k petrol cars in small and medium market sectors are experiencing the greatest price increases and consumer interest during lockdown.
- 8.
Affordable, cheap to run cars under £6k are expected to perform well as buyers may return to public transport or car sharing later.
- 9.
Expansion of London’s Ultra Low Emission Zone (ULEZ) is causing owners of older diesel cars to sell them at lower prices in London, affecting local used car prices, while outside London demand for older diesel cars and all cars is strong, causing prices to rise.
- 10.
The rise of online dealers has changed the market and contributed to the used-car price surge.
- 11.
Since forecourts opened on 12 April, dealers have been overrun with people and supply is very low, with supply down 10.8% compared to 2019, and demand growing significantly, leading to record price growth rates and increased sticker prices as advised by Auto Trader.
- 12.
The Covid-19 pandemic shuttered factories and disrupted shipping routes globally, causing a backlog that is a chief cause behind a massive 25% climb in used car prices in 2021.
- 13.
The pandemic changed consumer demand for cars, forcing many to cancel or postpone travel plans in 2020, leading to unprecedented demand for cars in spring 2021 as vaccines and relaxed public-health rules allowed travel.

### D.2 Key-Point Recall and Contradiction

Table [5](https://arxiv.org/html/2505.19253v2#A4.T5) summarizes key-point evaluation. The report does not contradict any of the keypoints, hence KPR for this report would be computed as 6/13, and KPC as 0/13.

**Table 5: Summary of LLM evaluation labels for 13 claims.**
| Key Point | Label | Summary |
| --- | --- | --- |
| 1 | Supported | COVID-19 reduced trade-ins and part-exchanges, lowering supply. |
| 2 | Supported | Chip shortages reduced new car supply, boosting used demand. |
| 3 | Omitted | Consumer behaviors like swapping leases and treating themselves not mentioned. |
| 4 | Supported | Dealer inventory shortages from fleet and rental supply issues. |
| 5 | Supported | Tariffs raised new car prices and pushed buyers toward used cars. |
| 6 | Omitted | No mention of summer/fall trends or chip shortage timing. |
| 7 | Omitted | No reference to vehicle types like SUVs or diesel in lockdown context. |
| 8 | Omitted | Cars under £6k and expectations for public transport recovery not covered. |
| 9 | Omitted | No mention of ULEZ or regional UK pricing differences. |
| 10 | Supported | Online dealers and market changes linked to price surges. |
| 11 | Omitted | No mention of April forecourt reopening or Auto Trader commentary. |
| 12 | Supported | Pandemic factory closures and shipping delays noted as price drivers. |
| 13 | Omitted | 2020 demand surge post-vaccine and lockdown easing not included. |

### D.3 Retrieval Faithfulness

Table LABEL:tab:llmclaims presents a sample of 6 claims extracted from the document, together with supporting URLs and justifications. Shown claims were rated as being fully supported by the source URLs.

**Table 6: Sample of LLM-evaluated claims for factual accuracy**
| # | Claim | Justification | Source(s) |
| --- | --- | --- | --- |
| 1 | Used car prices for one- to five-year-old vehicles increased by approximately 1% year-over-year as of March 2025, translating to an average price bump of about $317. | The citation explicitly states that used car prices increased 1% YoY as of March 2025, translating to a $317 increase—matching the claim. | [USA Today](https://www.usatoday.com/story/money/2025/04/11/used-car-prices-are-rising-2025/83050309007/) |
| 2 | The average used car price rose roughly $170 from February to March 2025, reaching $25,180. | Fully supported by the source, which gives the exact figure and monthly change. | [KBB](https://www.kbb.com/car-news/average-used-car-price-starts-to-rise/) |
| 3 | A 25% tariff was applied to all new cars entering the U.S. in early 2025, with further tariffs on parts scheduled. | The source details the 25% tariff beginning in April 2025 and pending parts tariffs. | [KBB](https://www.kbb.com/car-news/average-used-car-price-starts-to-rise/), [Carketa](https://carketa.com/auto-tariffs-used-car-pricing-inventory/) |
| 4 | Imported vehicles could see price increases of up to $6,000 due to tariffs, with domestic vehicles also rising around $3,600. | The cited article provides these specific figures directly. | [CNBC](https://www.cnbc.com/2025/04/12/auto-tariffs-sales-costs.html) |
| 5 | Automakers responded by pausing shipments, adjusting strategies, or freezing U.S. exports, shrinking affordable vehicle supply. | Source confirms automakers are freezing exports and adjusting due to tariffs, limiting supply. | [Carscoops](https://www.carscoops.com/2025/04/used-cars-just-saw-their-first-price-bump-in-over-two-years/) |
| … (Claims 6–21 not shown) |  |  |  |
| 22 | Dealers are offering flexible financing and personalized experiences to retain customers amid price fluctuations, indirectly supporting higher price points. | The citation confirms this strategy for retaining customers during price volatility. | [Carketa](https://carketa.com/auto-tariffs-used-car-pricing-inventory/) |

### D.4 Quality

Below we show the LLM-judge output for both clarity and insightfulness dimensions:

Clarity (Score: 9/10)

The answer is exceptionally clear, well-structured, and logically organized, resembling an in-depth report with clearly marked sections and a strong logical flow. Each section addresses a distinct factor contributing to the increase in used car prices, such as tariffs, supply chain disruptions, economic conditions, consumer preferences, and dealer strategies, without substantive overlap or redundancy. The introduction and conclusion effectively frame and summarize the analysis. The use of precise data and references enhances clarity and rigor. Minor overlap exists in the economic and financing conditions section where some points slightly relate to consumer preferences, but these are sufficiently distinct in focus. The answer avoids ambiguity, redundancy, and conversational filler, maintaining a professional and analytical tone throughout. Overall, it meets the criterion of clarity at a very high level, with only negligible issues preventing a perfect score.

Insightfulness (Score: 9/10)

The answer demonstrates a high level of insightfulness by providing a comprehensive, well-structured analysis that goes beyond common knowledge. It synthesizes multiple relevant factors—tariffs, supply chain disruptions, economic conditions, consumer preferences, and dealer strategies—into a coherent explanation of why used car prices have increased. The report includes specific, concrete examples such as the 25% tariff on new cars, the impact of COVID-19 on lease returns, and detailed data from reputable sources, which grounds the analysis in practical reality. It also highlights less obvious connections, such as how tariffs on new cars indirectly inflate used car prices through demand shifts and supply constraints. The inclusion of segment-specific trends and regional variations adds depth and reframing to the topic. However, the report could improve by providing more explicit real-world examples of entities implementing strategies to mitigate these issues and the outcomes thereof, which would elevate it to a perfect score. Overall, the answer meets the criterion of insightfulness very well, with only minor room for enhancement in operational recommendations and examples.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. line 144, how do you define "low quality"?
2. line 178-179, how do you define "minimal loss"?
2. Section 4.2, why are subjective report quality metrics (Clarity, Insight) comparable to objective information coverage metrics (KPR)?
soundness: 2
strengths: 1. The paper contributes an open-source sandbox that can potentially facilitate reproducible research in deep research agents.
2. The authors deliver some insight into fine-grained performance of popular deep research agents.
summary: This work introduces DeepResearchGym, an open-source sandbox for deep research agents. The sandbox features a search API implementation that is reproducible to facilitate future research and agent development. Based on the sandbox, the authors further evaluated several state-of-the-art deep research agents and presents insight into their performance along different axes, including relevance, faithfulness, and quality. Results show that agents can achieve comparable performance when using DeepResearchGym compared to using commercial search APIs.
weaknesses: 1. line 154-155: While the authors use recent data sources to construct the sandbox, it is unclear from the paper how to keep the sandbox "up-to-date".
2. The contribution beyond the sandbox is limited, as the authors mostly follow existing work in their evaluation protocol. 
3. While I understand the importance of having a reproducible environment for benchmarking deep research agents, I fail to see the discussion on the actual benefit from this paper. What are the evaluation nuances that DeepResearchGym helps to capture which are infeasible with time-varying search APIs?
4. In section 4.3, the authors use the variability observed in query-level analysis to justify the importance of using a standard retrieval API. I think the logic is flawed. Doesn't this show that the evaluation metrics, which heavily relies on textual overlap, are sensitive the search APIs used?
5. For section 4.4, I don't see the meaning of this pairwise human evaluation when the main results in the paper are pointwise. Why not letting human evaluators follow the same protocol as LLM-as-a-judge and compute score correlations?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is the correlation between report length and KPR? Are higher-scoring systems simply more verbose?
2. What are the most common failure modes? Can you provide qualitative analysis of queries where systems perform poorly vs. well?
soundness: 3
strengths: 1. **Addresses critical reproducibility gap**: The field urgently needs standardized benchmarking infrastructure. The paper directly tackles cost, transparency, and reproducibility issues with commercial APIs.
2. **Comprehensive multi-dimensional evaluation**: The three-faceted framework (relevance, faithfulness, quality) captures different aspects of report generation quality, going beyond simple surface-form metrics.
3. **Nuanced analysis**: Query-level correlation analysis (Figure 2) and query log analysis (Appendix A) provide insights beyond system-level aggregates.
summary: This paper introduces **DeepResearchGym**, an open-source benchmarking framework for evaluating deep research systems. The framework addresses critical reproducibility and transparency issues in current evaluation practices that rely on proprietary, dynamic commercial search APIs. The main contributions are: (1) A **free search API** built on large-scale public web corpora (ClueWeb22 and FineWeb) with dense retrieval (MiniCPM-Embedding) and approximate nearest neighbor search (DiskANN), achieving lower latency than commercial alternatives while ensuring stable rankings; (2) An **evaluation protocol** extending the Researchy Questions benchmark with LLM-as-a-judge metrics measuring report relevance (Key Point Recall/Contradiction), retrieval faithfulness (citation precision/recall), and report quality (clarity/insightfulness); (3) **Empirical validation** showing systems maintain comparable performance when switching from commercial to DeepResearchGym APIs, with human evaluation confirming metric reliability ($\kappa = 0.72$-$0.89$).
weaknesses: 1. **Heavy reliance on a single judge model without robustness analysis**: All automatic evaluations use GPT-4.1-mini exclusively as the judge, with no ablation studies using alternative models such as GPT-4o, Claude Sonnet, or open-source alternatives. This creates potential concerns about judge-specific biases, particularly since some evaluated systems (like OpenAI's deep research) are based on GPT models.
2. **Insufficient examination of static corpus limitations and ground-truth quality**: While the paper demonstrates that systems maintain performance when switching to static corpora, it does not analyze when this approach is adequate versus problematic. There is no quantitative assessment of which query types are temporally sensitive and might suffer from the 2022-2024 snapshot. Additionally, the evaluation assumes that clicked documents from search logs constitute comprehensive ground truth, but this assumption is never validated. Users may click tangential documents or miss important sources, yet the paper provides no analysis of click quality or relevance distribution.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses
soundness: 3
strengths: 1. This paper proposes DeepResearchGym, an open-source benchmarking framework specifically designed to enable transparent and reproducible evaluation of deep research systems. Being free and open-source makes this work a valuable resource to the community.

2. Empirical evaluations show that the system achieves strong retrieval quality with minimal loss from approximate search, as well as maintaining response times below those attained by commercial APIs.

3. The paper is well-written and easy to follow.
summary: The paper introduces DEEPRESEARCHGYM, an open, reproducible sandbox for evaluating deep research systems that generate long-form, evidence-grounded reports. It provides a free search API over public web corpora (ClueWeb22 A/B and FineWeb), implemented with a dense retriever (MiniCPM-Embedding-Light) and DiskANN, plus endpoints to search and to fetch archived page snapshots for stable, auditable evidence. The authors pair this with a tri-part evaluation protocol on Researchy Questions that scores (i) coverage via Key-Point Recall/Contradiction, (ii) retrieval faithfulness via citation recall/precision, and (iii) report quality (clarity, insight) using LLM-as-judge prompts. Empirically, systems keep similar rankings when swapping commercial search for this API, and the API achieves sub-0.5s median latency, suggesting it is a practical, research-grade substitute for proprietary search backends.
weaknesses: 1. Although DeepResearchGym helps the reproduction of deep research systems with a higher response speed and lower cost, using static corpora may under-serve very time-sensitive queries.

2. Despite showing high correlation with human evaluation, the empirical results only on Researchy Questions using GPT-4.1 as a judge might not comprehensively and robustly reflect the system's actual performance to serve as a replacement of search engine. Additionally, more fine-grained comparison between the proposed API vs different commercial API would be much appreciated. What's the fundamental difference between different search APIs, and how would that affect the performance of deep research systems?

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: * Line 145: The authors state that low-quality and spam pages were filtered during sampling. Could you clarify if this filtering was part of the original ClueWeb22 dataset's preprocessing, or if an additional filtering step was applied? If it was an additional step, please provide more details on the process.
* The paper mentions that users can switch between corpora (Lines 200-202), yet all evaluations are conducted on either commercial search engines or the ClueWeb22 dataset. The justification for including the FineWeb corpus is unclear. To demonstrate its value and analyze the system's performance across different retrieval corpora, please consider adding experiments based on FineWeb, perhaps in the appendix.
* Figure 1 indicates high efficiency for the search API/retriever in DeepResearchGym. A discussion of the factors contributing to this efficiency would be a valuable addition. Explaining the underlying design choices could further highlight the innovative aspects of the system.
* The evaluation is limited to a small number of systems. While the benchmark aims for fairness, two of the systems evaluated do not use the provided search and fetch APIs, making direct comparison difficult. To provide a more direct and comprehensive comparison, have the authors considered evaluating other generative models within the same standardized pipeline?
* The design of DeepResearchGym is well-suited for efficiency comparisons between different systems (e.g., measuring the number of searches, fetches, or conversational turns). However, the paper currently lacks a discussion on these efficiency metrics. Including an analysis of efficiency would significantly strengthen the evaluation.
* It's always good to include human evaluation as justification for the LLM judges. However, the authors did not discuss the background of the annotators and the instructions for their annotation. Since the task requires expert-level knowledge, it is necessary to have more details regarding human evaluation.
soundness: 3
strengths: * Provide a transparent and fair environment to compare various deep research systems.
* Include extensive metrics during evaluation
summary: This paper introduces DeepResearchGym, a benchmark for evaluating deep research systems. The benchmark provides standardized search and fetch APIs to ensure a fair and transparent environment for comparing various systems. Although the benchmark utilizes an 'LLM-as-a-judge' approach, its metrics are validated by human annotation and high inter-annotator agreement. A comparative analysis of two commercial, three white-box, and two baseline deep research systems was conducted using this benchmark.
weaknesses: * Novelty: The current corpus and queries of DeepResearchGym are from existing datasets (ClueWeb22, FineWeb, and Researchy Questions). I am very willing to raise my score if there are any missing innovative details in the methodology.

* A few missing details in the benchmark construction and evaluation. Please see the details in the questions.


Decision: Reject