UR$^2$: Unify RAG and Reasoning through Reinforcement Learning

## Abstract

Abstract Large Language Models (LLMs) have shown remarkable capabilities through two complementary paradigms: Retrieval-Augmented Generation (RAG), which enhances knowledge grounding, and Reinforcement Learning from Verifiable Rewards (RLVR), which optimizes complex reasoning abilities. However, these two capabilities are often developed in isolation, and existing efforts to unify them remain narrow in scope—typically limited to open-domain QA with fixed retrieval settings and task-specific constraints. This lack of integration constrains generalization and limits the applicability of RAG-RL methods to broader domains.
To bridge this gap, we propose UR 2 ( U nified R AG and R easoning), a general framework that unifies retrieval and reasoning through reinforcement learning. UR 2 introduces two key contributions: a difficulty-aware curriculum training that selectively invokes retrieval only for challenging problems, and a hybrid knowledge access strategy combining domain-specific offline corpora with LLM-generated summaries. These components are designed to enable dynamic coordination between retrieval and reasoning, improving adaptability across a diverse range of tasks.
Experiments across open-domain QA, MMLU-Pro, medical, and mathematical reasoning tasks demonstrate that UR 2 (built on Qwen-2.5-3/7B and LLaMA-3.1-8B) significantly outperforms existing RAG and RL methods, achieving comparable performance to GPT-4o-mini and GPT-4.1-mini on several benchmarks. We have released all code, models, and data at https://github.com/Tsinghua-dhy/UR2 .

## 1 Introduction

Large Language Models (LLMs) have achieved remarkable performance across diverse tasks by incorporating external knowledge (Retrieval-Augmented Generation, RAG) (Lewis et al., 2020; Borgeaud et al., 2022; Izacard et al., 2022) and optimizing reasoning through reinforcement learning with verifiable rewards (RLVR) (Guo et al., 2025a). RAG methods enable LLMs to access external knowledge, while RLVR shows strong gains on mathematical and logical reasoning (Zeng et al., 2025; Chen et al., 2025). Motivated by these successes, recent work has begun to integrate retrieval and reasoning: for example, Search‑o1 (Li et al., 2025) embeds an agentic RAG workflow into the LLM’s chain‑of‑thought, and RAG‑Gym (Xiong et al., 2025) proposes a unified RL‑based training framework for RAG agents. Similarly, RAG-RL methods—which learn to invoke retrieval through RL—such as R1-Searcher (Song et al., 2025a) and Search-R1 (Jin et al., 2025) use RLVR to train models on *when* and *what* to retrieve during reasoning, improving performance in open-domain QA.

Despite recent progress, RAG-RL frameworks remain limited in scope. Most methods focus narrowly on open-domain QA, with retrieval tied to fixed reasoning steps or static knowledge sources like Wikipedia. However, paradigms that work well on open-domain QA often fail to transfer to broader domains. Two key limitations persist: (1) models struggle to achieve optimal reasoning-retrieval trade-offs, often over-emphasizing one component; (2) retrieved documents contain significant noise, and current prompt-only approaches lack effective noise handling mechanisms, potentially degrading to basic Chain-of-Thought reasoning. For instance, R1-Searcher and Search-R1 assume access to Wikipedia, ill-suited for tasks requiring specialized or real-time information. While methods like DeepResearcher attempt training in real web environments, they face inefficiencies due to the noisy and unstructured nature of online data (Zheng et al., 2025). Other methods like ZeroSearch (Sun et al., 2025), use LLM-generated corpora to simulate retrieval, avoiding API costs but risking hallucination and loss of real-world complexity.

To address the limitations of existing RAG-RL approaches—such as static retrieval, limited domain generalization, and poor robustness in noisy environments—we propose a general and adaptive framework, UR^2 (Unified RAG and Reasoning), which uses RL to dynamically coordinate retrieval and reasoning.
Unlike prior methods that rely solely on static corpora (e.g., Wikipedia) or simulate retrieval with synthetic content, UR^2 combines both: it leverages task-specific offline corpora for accurate grounding, augmented with LLM-generated summaries for efficiency and generalization.
To address the imbalance between retrieval and reasoning in prior methods, we design a difficulty-aware curriculum that adaptively controls when to trigger retrieval during training. Specifically, retrieval is used only for hard instances, encouraging the model to rely on internal reasoning when possible and to learn retrieval strategies only when necessary. This reduces retrieval overhead, improves query quality on challenging questions, and preserves reasoning capability across tasks.

We train UR^2 on Qwen-2.5-3B/7B-Instruct (Yang et al., 2024) and LLaMA-3.1-8B-Instruct (Dubey et al., 2024) across MMLU-Pro, Medicine, Math, and open-domain QA. During training, these models spontaneously develop key cognitive behaviors: self-verification through retrieval, intermediate reasoning validation, and hypothesis revision based on external evidence. UR^2 outperforms previous state-of-the-art (SOTA) methods by 5.8% (7B) and 19.0% (3B) on average, with peak gains of 9.5% and 29.6%. Notably, our 7B model matches GPT-4o-mini and GPT-4.1-mini(^1^11https://chat.openai.com/) , and generalizes well across domains and model architectures.

##### Our main contributions are summarized as follows:

- •
We propose the first unified retrieval-reasoning RL framework that adapts to diverse tasks beyond open-domain QA, representing an important milestone for AI systems combining parametric and external knowledge.
- •
We design a unified data representation and training scheme bridging retrieval and reasoning, with difficulty-aware curricula and LLM-summarized corpora for accurate grounding and broad generalization, implemented via a modular two-stage framework.
- •
Comprehensive experiments demonstrate that UR^2 surpasses advanced RAG and RL methods without expert demonstrations and generalizes robustly across domains.

## 2 Related Work

### 2.1 Retrieval‑Augmented Generation

RAG enhances LLMs by incorporating external information to reduce hallucinations (Gao et al., 2023). Early RAG methods concatenate retrieved documents with input prompts (Lewis et al., 2020; Izacard et al., 2022; Borgeaud et al., 2022). Subsequent approaches have evolved in multiple directions: advanced RAG methods incorporate sophisticated retrieval and re-ranking strategies (Gao et al., 2023; Peng et al., 2024); post-hoc verification methods address hallucinations by retrieving documents based on generated responses (Li et al., 2024; Sun et al., 2024); and Graph-based RAG methods integrate knowledge graphs for multihop reasoning (Edge et al., 2024; Hu et al., 2025b; Peng et al., 2024). Recent RAG-RL frameworks have explored retrieval integration during training via real-time or synthetic retrieval (Zheng et al., 2025; Sun et al., 2025). However, these approaches remain constrained by static retrieval strategies, limited domain generalization, and inability to dynamically coordinate retrieval with reasoning across diverse task types.

### 2.2 Reinforcement Learning for Retrieval‑Enhanced Reasoning

RL has emerged as a key technique for significantly improving LLM capabilities, evolving from early policy gradient methods such as REINFORCE (Williams, 1992) to more advanced algorithms like PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024).
Recent methods, including ARENA (Ren et al., 2025), Search‑R1 (Jin et al., 2025), and R1‑Searcher (Song et al., 2025a), demonstrate that RL enables LLMs to effectively learn multi-step reasoning and retrieval strategies without requiring human feedback.
These works collectively highlight a clear shift from fixed retrieval heuristics to learned, RL-driven retrieval policies, which form the foundation for our unified framework, with retrieval becoming increasingly parameterized rather than merely prompt-guided.

## 3 Method

We propose UR^2, a general framework that tightly integrates retrieval-based grounding with explicit step-by-step reasoning via RL. Unlike previous approaches restricted to open-domain QA or reliant upon static corpora, UR^2 supports a broad range of tasks, including mathematical problem solving and domain-specific QA. To achieve this versatility, UR^2 leverages a LLM-summarized retrieval corpus (Section  [3.1.1](https://arxiv.org/html/2508.06165v3#S3.SS1.SSS1)) and a difficulty-aware curriculum that adapts training based on task hardness and knowledge demands (Section  [3.1.2](https://arxiv.org/html/2508.06165v3#S3.SS1.SSS2)).

Figure: Figure 1: Overview of the UR^2 training pipeline. The top illustrates LLM-summarized retrieval corpus, difficulty-aware curriculum design and a two-stage reward design for retrieval activation and answer optimization. The bottom horizontal bars indicate: (1) Open-domain QA: ablation results under LLM-as-a-judge; and (2) MMLU-Pro: comparison with baselines using EM score.
Refer to caption: x1.png

### 3.1 Difficulty-Aware Training with Hybrid Knowledge Access

#### 3.1.1 LLM-summarized retrieval corpus

UR^2 employs a LLM-summarized retrieval corpus designed to accommodate diverse task domains, comprising:

- •
Domain-specific offline corpora (e.g., curated medical knowledge bases, full Wikipedia content, or Wikipedia abstracts);
- •
Concise summary or fallback response generated by LLMs, following the structured approach of Search-o1 (Li et al., 2025), which enables the system to reject queries requiring complex reasoning beyond the scope of retrieval (see Appendix [D.3](https://arxiv.org/html/2508.06165v3#A4.SS3)).

This hybrid corpus design enhances retrieval accuracy, reduces hallucinations, and improves generalization across a variety of reasoning scenarios.

#### 3.1.2 Difficulty-Aware Curriculum Design

We organize this part into two components: (1) training data selection based on difficulty levels; and (2) task mixing strategy to balance retrieval and reasoning exposure.

Trainning Data Selection. To promote fine-grained reasoning and retrieval behaviors, we categorize training samples by their difficulty levels. For each question, we perform 20 rollouts using Qwen-2.5-7B-Instruct and compute the average performance score ($s$). Based on the score $s$, questions are categorized into three difficulty levels: Easy ($0.8\leq s\leq 1.0$), Medium ($0.5\leq s<0.8$), and Hard ($0.2\leq s<0.5$).

Following prior studies (Yu et al., 2025; Guo et al., 2025b), instances with extremely low performance scores ($s<0.2$) are filtered, as overly difficult samples hinder effective learning. We adopt a sampling ratio of 7:2:1 for hard, medium, and easy questions, prioritizing challenging examples to enhance reasoning and retrieval capabilities.

Task Mixing Strategy. To effectively balance retrieval and reasoning capabilities, we design two strategic task mixtures:

- •
Mathematical reasoning with open-domain QA: Mathematical tasks are separated by difficulty. Hard mathematical problems use retrieval-augmented prompting (Figure [2](https://arxiv.org/html/2508.06165v3#S3.F2)), while the others rely on pure step-by-step reasoning. QA data consistently uses retrieval since these questions need external knowledge.
- •
Multiple-choice reasoning tasks: We combine MedQA training data and synthetic MMLU-style datasets. Among these, most hard-level questions are used for retrieval-augmented training with a smaller portion for direct reasoning, maintaining an overall 1:1 ratio between the two approaches.

This controlled task composition brings several benefits. First, it ensures diverse exposure to both retrieval-intensive and reasoning-intensive formats, helping the model generalize across different tasks. Second, by associating retrieval usage with question difficulty, the model learns to rely on external knowledge only when necessary, rather than overusing retrieval indiscriminately. Third, this approach saves computational resources by activating retrieval only when needed for hard problems, preserving direct reasoning for simple cases. More detailed experimental configurations are provided in the Section [4](https://arxiv.org/html/2508.06165v3#S4) and Appendix [C.1](https://arxiv.org/html/2508.06165v3#A3.SS1).

### 3.2 Two-Stage Optimization for UR 2

Given the limited tool invocation capabilities of base models, especially in reasoning-integrated scenarios, we design a two-stage optimization framework to systematically develop retrieval skills and reasoning proficiency. We train UR^2 using REINFORCE++ (Guo et al., 2025a), a streamlined variant of PPO tailored. To prevent overfitting to retrieved content, we adopt retrieval masking (Song et al., 2025a; Jin et al., 2025). Our implementation is based on the REINFORCE++-baseline provided by OpenRLHF (Hu et al., 2024).

The training objective is defined as:

$$ $J_{\text{UR}^{2}}(\theta)=\mathbb{E}_{x,\{y^{i}\}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y^{i}|}\sum_{t=1}^{|y^{i}|}y^{i}_{t}\cdot r_{i,t}\cdot\hat{A}_{i,t}\right]$ (1) $$

where the importance weight is:

$$ $r_{i,t}=\frac{\pi_{\theta}(y^{i}_{t}\mid x,y^{i}_{<t};o_{i})}{\pi_{\text{old}}(y^{i}_{t}\mid x,y^{i}_{<t};o_{i})}$ (2) $$

and the normalized advantage is:

$$ $\hat{A}_{i,t}=\textit{Norm}_{\textit{batch}}\left(\textit{Norm}_{\textit{group}}\left(R_{i}-b\right)\right)$ (3) $$

The advantage $\hat{A}_{i,t}$ is computed by subtracting the group-level reward baseline and applying normalization across the group and batch to improve learning stability. Here, $x$ denotes the input prompt, $\{y^{i}\}$ are the sampled trajectories, $o_{i}$ is the retrieved context and $b$ is the group-level baseline (mean of $R_{i}$). See Appendix [C.1](https://arxiv.org/html/2508.06165v3#A3.SS1) and Section [4.4](https://arxiv.org/html/2508.06165v3#S4.SS4) for detailed implementation.

#### 3.2.1 RAG-based Rollout

UR^2 enables the model to issue retrieval queries during reasoning rather than pre-retrieving all information upfront. As illustrated in Figure [2](https://arxiv.org/html/2508.06165v3#S3.F2), the prompting mechanism enforces key principles: queries target single facts grounded in external knowledge, retrieval occurs when needed during the reasoning process, and strict format constraints using special tokens demarcate retrieval actions.

This design allows the model to strategically leverage external knowledge by learning *when* to retrieve and *what* to query for purposeful and grounded reasoning.

Figure: Figure 2: Instruction prompt used to guide retrieval-augmented reasoning in UR^2. See Appendix [D.2](https://arxiv.org/html/2508.06165v3#A4.SS2) for details.

#### 3.2.2 Stage 1: Retrieval Capability Activation

We use UR^2 with Qwen-2.5-7B-Instruct on mathematical and open-domain QA tasks as an example. In Stage 1, the model trains on mathematical problems requiring retrieval calls in the specified format (Figure [2](https://arxiv.org/html/2508.06165v3#S3.F2)). The objective is not answer accuracy, but to enforce correct usage of the retrieval tool and promote retrieval-invoking behavior. This specialized training runs for only 10 steps. Further details on task setup and extensions to other models are provided in Appendix [C.5](https://arxiv.org/html/2508.06165v3#A3.SS5).

The total reward is:

$$ $\displaystyle R_{i,\text{stage1}}=R_{i,\text{format}}+R_{i,\text{retrieval}}-P_{i,\text{fallback}}$ (4) $$

where (1) Format Reward: $+1$ for fully compliant output; $-1$ per violation (e.g., malformed tags, overlength queries, missing retrieval, or illegal tokens); (2) Retrieval Reward: $+3$ for one valid query, $+4$ for two or more; (3) Fallback Penalty: $-0.5$ per fallback fault.

This stage equips the model with retrieval capabilities and promotes effective integration of retrieved information during generation.

#### 3.2.3 Stage 2: Answer Quality Optimization

Building on Stage 1, we incorporate correctness feedback to refine generation quality while preserving retrieval behaviors. The updated reward function is:

$$ $\displaystyle R_{\text{i, stage2}}=R_{i,\text{answer}}+R_{i,\text{format}}-P_{i,\text{fallback}}$ (5) $$

where (1) Answer Reward: $+2$ for correct answers, 0 for incorrect; (2) Format Reward: $+1$ for fully valid format; 0 otherwise; (3) Fallback Penalty: $-0.5$ per fallback fault.

By decoupling retrieval skill acquisition (Stage 1) from generation optimization (Stage 2), we ensure stable convergence and interpretable credit assignment across complex reasoning trajectories.

## 4 Experimental Settings

### 4.1 Training Datasets

We build a unified training set covering math (SimpleZoo-RL (Zeng et al., 2025)), open-domain QA (R1-Searcher (Song et al., 2025a)), and multi-choice medical QA (MedQA (Jin et al., 2021)). For MMLU-Pro(Wang et al., 2024a) domains (philosophy, history, economics), we generate synthetic questions via Qwen-3-32B(^2^22https://huggingface.co/Qwen/Qwen-3-32B). After deduplication and data selection using Qwen-2.5-7B-Instruct on 20 rollouts per question, we obtain 3K samples each for math, open-domain QA, and MedQA, and 2K samples for each MMLU-Pro domain. Details are in Appendix [C.1](https://arxiv.org/html/2508.06165v3#A3.SS1).

### 4.2 Evaluation Benchmarks

We evaluate generalization across four task families:
(1) Math Reasoning: MATH500 (in-domain) (Hendrycks et al., 2021), Minerva (OOD) (Lewkowycz et al., 2022); metric: LLM-as-a-judge.
(2) Medical QA: MedQA (5-choice, in) (Jin et al., 2021), MMLU-Pro Medical (M-Med, OOD); metric: EM.
(3) MMLU-Pro: Philosophy, History, Economics (in), Law (OOD); metric: EM.
(4) Open-Domain QA: HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020) (in); Bamboogle (Press et al., 2023), MusiQue (Trivedi et al., 2022) (OOD); metrics: F1 and LLM-as-a-judge.

### 4.3 Baselines

We compare UR^2 to:
(1) Vanilla Methods: Chain-of-Thought (Kojima et al., 2022), Standard RAG (Borgeaud et al., 2022; Izacard et al., 2022) (top-$k$=10).
(2) Advanced RAG Methods: Search-o1 (Li et al., 2025), Self-Ask (Press et al., 2023), and RAT (Wang et al., 2024b), which combine reasoning with retrieval using prompt.
(3) CoT-RL Methods: R1-like methods including Open-Reasoner-Zero (Hu et al., 2025a), SimpleRL-Zoo (Zeng et al., 2025), and General-Reasoner (Ma et al., 2025).
(4) RAG-RL Methods: R1-Searcher (Song et al., 2025a), R1-Searcher++ (Song et al., 2025b), Search-R1 (Jin et al., 2025), and ZeroSearch (Sun et al., 2025).
(5) Vanilla RL: Baseline implementation following the same training setup and datasets as UR^2, with RAG-RL applied to open-domain QA and CoT-RL to mathematical and multiple-choice tasks.

We use Qwen-2.5-3B-Instruct, Qwen-2.5-7B-Instruct, LLaMA-3.1-8B-Instruct, GPT-4o-mini, and GPT-4.1-mini as backbones (see Appendix [C.2](https://arxiv.org/html/2508.06165v3#A3.SS2) for configs).

### 4.4 Implementation Details

Retrieval uses BGE-large-en-v1.5(^3^33https://huggingface.co/BAAI/bge-large-en-v1.5) and the KILT (Petroni et al., 2021) Wikipedia corpus (100-word segments, 29M documents) following (Song et al., 2025a). Open-domain QA uses Wikipedia abstract corpus(^4^44https://nlp.stanford.edu/projects/hotpotqa/enwiki-20171001-pages-meta-current-withlinks-abstracts.tar.bz2). Unless otherwise noted, all models use GPT-4.1-mini as the summarizer during training and GPT-4.1 during evaluation with top-$k$ = 10, while mathematical tasks are summarized by Qwen-3-32B. For evaluation, we sample 500 instances from each benchmark. We use $G=16$ rollouts. 7B and 8B models use training batch size 256, rollout batch size 64; 3B doubles both. Learning rate = 2e-6. Up to 4 retrieval turns are allowed. All models are trained for up to 2 epochs on 8×A100 GPUs. See Appendix [C.1](https://arxiv.org/html/2508.06165v3#A3.SS1) and [C.2](https://arxiv.org/html/2508.06165v3#A3.SS2) for details.

## 5 Experimental Results

Our UR^2 framework achieves SoTA performance across reasoning and retrieval tasks, enabling 7B models to match or exceed the GPT model family while significantly outperforming existing RAG and RL-based methods. More comprehensive baseline results can be found in Appendix [B.1](https://arxiv.org/html/2508.06165v3#A2.SS1).

### 5.1 Main Results on Reasoning Tasks

As shown in Table [1](https://arxiv.org/html/2508.06165v3#S5.T1), UR^2 demonstrates substantial improvements across all reasoning tasks on the Qwen-2.5-7B model, achieving average scores of 53.3% on MMLU-Pro, 65.9% on Medicine, and 71.0% on Math benchmarks, representing gains of 3.7%, 5.7%, and 1.2% over the strongest CoT-RL baseline Open-Reasoner-Zero. Across model scales, UR^2 shows consistent advantages: on Qwen-2.5-3B, it achieves even larger performance gains with 9.1% improvement on MMLU-Pro and 8.6% on Medicine over Vanilla RL, demonstrating that UR^2 provides greater benefits for models with limited knowledge but strong reasoning capabilities. On LLaMA-3.1-8B, it achieves 43.4% on MMLU-Pro, outperforming all baselines. Notably, our method achieves competitive performance with the more capable closed-source GPT-4o-mini model on several tasks. As shown in Tables [1](https://arxiv.org/html/2508.06165v3#S5.T1) and [6](https://arxiv.org/html/2508.06165v3#A2.T6), advanced RAG methods degrade performance on smaller models and require unacceptable source consumption (except Search-o1).

**Table 1: Performance on reasoning and math tasks. We report EM scores (in %) on MMLU-Pro and MedQA, and LLM-as-a-judge scores (in %) on math benchmarks. ${\dagger}$ = in-domain, ${\ddagger}$ = out-of-domain. Best results are bold; second-best are underlined.**
| Method | MMLU-Pro | Medicine | Math |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Hist.^† | Phil.^† | Econ.^† | Law^‡ | Avg | MedQA^† | M-Med^‡ | Avg | Math500^† | Minerva^‡ | Avg |  |
| GPT-4o-mini |  |  |  |  |  |  |  |  |  |  |  |
| CoT | 56.7 | 53.1 | 70.4 | 38.2 | 54.5 | 71.4 | 67.0 | 69.2 | 78.0 | 65.6 | 71.8 |
| Standard RAG | 57.0 | 52.3 | 68.6 | 36.2 | 53.5 | 70.6 | 64.2 | 67.4 | 77.1 | 68.4 | 72.8 |
| Advanced RAG Methods |  |  |  |  |  |  |  |  |  |  |  |
| Self-Ask | 56.3 | 48.5 | 67.8 | 31.2 | 51.0 | 72.4 | 68.0 | 70.2 | 62.9 | 45.2 | 54.1 |
| RAT | 57.5 | 55.3 | 73.0 | 34.2 | 55.0 | 74.4 | 70.6 | 72.5 | 77.5 | 64.2 | 70.9 |
| Search-o1 | 53.5 | 55.3 | 69.8 | 35.4 | 53.5 | 75.2 | 66.6 | 70.9 | 78.6 | 68.3 | 73.5 |
| Qwen-2.5-7B |  |  |  |  |  |  |  |  |  |  |  |
| CoT | 42.3 | 45.7 | 63.4 | 26.6 | 44.5 | 57.2 | 52.0 | 54.6 | 76.6 | 59.4 | 68.0 |
| Standard RAG | 44.6 | 41.1 | 57.8 | 26.0 | 42.4 | 54.2 | 53.2 | 53.7 | 73.8 | 54.6 | 64.2 |
| Advanced RAG Methods |  |  |  |  |  |  |  |  |  |  |  |
| Self-Ask | 40.7 | 42.1 | 60.0 | 26.2 | 42.3 | 51.8 | 47.8 | 49.8 | 74.9 | 57.7 | 66.3 |
| RAT | 47.2 | 44.7 | 64.4 | 30.0 | 46.6 | 60.0 | 53.2 | 56.6 | 74.4 | 55.5 | 65.0 |
| Search-o1 | 42.8 | 45.9 | 63.2 | 29.6 | 45.4 | 58.2 | 52.8 | 55.6 | 78.2 | 60.3 | 69.3 |
| CoT-RL Methods |  |  |  |  |  |  |  |  |  |  |  |
| General Reasoner | 47.9 | 44.2 | 65.9 | 30.4 | 47.1 | 58.4 | 54.4 | 56.4 | 76.6 | 62.1 | 69.8 |
| Open-Reasoner-Zero | 50.0 | 46.6 | 67.5 | 34.2 | 49.6 | 61.6 | 58.8 | 60.2 | 80.7 | 58.8 | 69.8 |
| SimpleRL-Zoo | 35.7 | 36.9 | 55.2 | 25.4 | 38.3 | 57.2 | 51.0 | 54.1 | 77.1 | 50.7 | 63.9 |
| Our Implementations |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla RL | 52.2 | 43.5 | 64.0 | 33.8 | 48.4 | 64.2 | 57.4 | 60.8 | 78.2 | 59.4 | 68.8 |
| UR^2 (Ours) | 53.2 | 53.0 | 72.2 | 35.0 | 53.3 | 69.6 | 62.8 | 66.2 | 80.9 | 61.0 | 71.0 |
| Qwen-2.5-3B |  |  |  |  |  |  |  |  |  |  |  |
| CoT | 33.6 | 32.3 | 48.8 | 20.6 | 33.8 | 39.4 | 36.8 | 38.1 | 63.6 | 39.9 | 51.8 |
| Standard RAG | 37.8 | 36.5 | 51.4 | 23.2 | 37.1 | 45.6 | 40.0 | 42.8 | 65.3 | 40.8 | 53.1 |
| Our Implementations |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla RL | 40.7 | 34.7 | 55.0 | 24.6 | 38.7 | 51.8 | 47.6 | 49.7 | 68.0 | 43.9 | 56.0 |
| UR^2 (Ours) | 47.8 | 49.3 | 63.9 | 30.0 | 47.8 | 59.8 | 56.8 | 58.3 | 69.4 | 45.0 | 57.2 |
| LLaMA-3.1-8B |  |  |  |  |  |  |  |  |  |  |  |
| CoT | 37.8 | 40.9 | 53.4 | 29.0 | 40.3 | 59.6 | 52.6 | 56.1 | 48.4 | 34.4 | 41.4 |
| Standard RAG | 43.6 | 33.9 | 51.0 | 26.6 | 38.8 | 56.4 | 53.2 | 54.8 | 45.0 | 31.4 | 38.2 |
| Our Implementations |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla RL | 44.6 | 36.9 | 53.0 | 26.4 | 40.2 | 66.8 | 57.4 | 62.1 | 45.5 | 43.4 | 44.4 |
| UR^2 (Ours) | 48.3 | 38.6 | 58.0 | 28.8 | 43.4 | 68.6 | 58.4 | 63.5 | 54.5 | 39.0 | 46.8 |

### 5.2 Main Results on Open-domain QA

As demonstrated in Table [2](https://arxiv.org/html/2508.06165v3#S5.T2), UR^2 achieves strong performance on open-domain QA, with Qwen-2.5-7B reaching 58.5% average F1 score, outperforming the strongest RAG-RL baseline Search-R1 (56.1%) by 2.4%. UR^2 demonstrates particularly robust out-of-domain generalization, achieving 64.5% on Bamboogle and 35.8% on MusiQue, surpassing all baselines. Across model scales, UR^2 maintains consistent advantages: on Qwen-2.5-3B, it achieves 55.3% F1, improving 8.2% over Searh-R1; on LLaMA-3.1-8B, it reaches 56.3%, competitive with specialized RAG-RL methods. Notably, our 7B model surpasses GPT-4.1-mini (55.4%) by 3.1%, demonstrating UR^2’s effective dynamic coordination of retrieval and reasoning.

**Table 2: Performance on open-domain QA tasks. We report F1 and LLM-as-a-judge (LSJ) scores, both in %. ${\dagger}$ = in-domain; ${\ddagger}$ = out-of-domain.**
| Models | Types | Methods | Hotpot^† | 2Wiki^† | Bamb.^‡ | MusiQ.^‡ | Avg |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | F1 | LSJ | F1 | LSJ | F1 | LSJ | F1 | LSJ | F1 | LSJ |  |  |
| GPT-4.1-mini | Vanilla Methods | CoT | 43.7 | 59.2 | 48.6 | 60.8 | 59.2 | 76.0 | 28.3 | 35.4 | 45.0 | 57.9 |
|  | Standard RAG | 54.5 | 74.4 | 41.3 | 52.4 | 46.4 | 51.2 | 21.9 | 28.4 | 41.0 | 51.6 |  |
| Advanced RAG | Self-Ask | 65.4 | 75.0 | 52.7 | 57.4 | 71.7 | 75.2 | 31.6 | 35.0 | 55.4 | 60.7 |  |
|  | RAT | 56.9 | 64.2 | 45.7 | 49.0 | 60.3 | 62.4 | 29.0 | 31.4 | 48.0 | 51.8 |  |
|  | Search-o1 | 53.1 | 74.0 | 44.4 | 60.6 | 63.7 | 71.2 | 28.6 | 33.4 | 47.5 | 59.8 |  |
| Qwen-2.5-7B | Vanilla Methods | CoT | 24.9 | 31.0 | 25.1 | 27.6 | 41.3 | 43.2 | 14.8 | 12.2 | 26.5 | 28.5 |
|  | Standard RAG | 49.2 | 62.8 | 32.8 | 37.6 | 38.9 | 40.0 | 14.4 | 14.6 | 33.8 | 38.8 |  |
| Advanced RAG | Self-Ask | 28.8 | 61.0 | 22.2 | 45.4 | 28.9 | 42.4 | 13.6 | 19.6 | 23.4 | 42.1 |  |
|  | RAT | 37.9 | 40.6 | 23.3 | 23.6 | 31.6 | 30.4 | 14.4 | 12.4 | 26.8 | 26.8 |  |
|  | Search-o1 | 50.9 | 61.6 | 45.2 | 48.6 | 37.5 | 39.2 | 20.6 | 19.8 | 38.6 | 42.3 |  |
| RAG-RL | R1-Searcher | 71.8 | 78.0 | 57.9 | 63.6 | 56.5 | 53.6 | 33.3 | 32.6 | 54.8 | 57.0 |  |
|  | Search-R1 | 72.4 | 78.8 | 61.0 | 63.8 | 58.9 | 56.8 | 32.2 | 32.0 | 56.1 | 57.9 |  |
|  | R1-Searcher++ | 59.0 | 64.2 | 61.2 | 64.4 | 60.8 | 59.2 | 33.8 | 32.8 | 53.7 | 55.2 |  |
|  | ZeroSearch | 46.0 | 50.4 | 38.4 | 38.6 | 35.8 | 38.4 | 14.7 | 13.8 | 33.7 | 35.3 |  |
| Our Implementations | Vanilla RL | 70.9 | 78.8 | 61.2 | 62.4 | 63.3 | 63.2 | 34.4 | 34.4 | 57.5 | 59.6 |  |
|  | UR^2 (Ours) | 71.2 | 79.4 | 62.6 | 65.0 | 64.5 | 62.4 | 35.8 | 34.6 | 58.5 | 60.4 |  |
| Qwen-2.5-3B | Vanilla Methods | CoT | 26.6 | 27.2 | 22.7 | 22.6 | 31.2 | 33.6 | 11.3 | 9.6 | 23.0 | 23.3 |
|  | Standard RAG | 50.6 | 57.0 | 29.8 | 30.4 | 26.1 | 27.2 | 9.7 | 7.4 | 29.1 | 30.5 |  |
| RAG-RL | Search-R1 | 63.1 | 69.2 | 49.5 | 53.4 | 48.3 | 48.0 | 27.6 | 27.8 | 47.1 | 49.6 |  |
|  | Zero-Search | 42.7 | 45.8 | 26.1 | 27.6 | 32.4 | 31.2 | 16.9 | 17.0 | 29.5 | 30.4 |  |
| Our Implementations | Vanilla RL | 65.9 | 73.6 | 54.9 | 58.0 | 59 | 57.6 | 30.0 | 29.6 | 52.5 | 54.7 |  |
|  | UR^2 (Ours) | 67.7 | 76.0 | 55.2 | 58.6 | 57.8 | 58.4 | 30.5 | 31.6 | 55.3 | 56.2 |  |
| LLaMA-3.1-8B | Vanilla Methods | CoT | 28.6 | 31.6 | 16.4 | 17.8 | 43.0 | 42.4 | 9.8 | 10.8 | 24.5 | 25.7 |
|  | Standard RAG | 47.5 | 54.4 | 26.2 | 26.4 | 26.5 | 28.0 | 10.1 | 10.2 | 27.6 | 29.8 |  |
| RAG-RL | R1-Searcher | 70.8 | 76.8 | 59.6 | 62.2 | 64.7 | 62.4 | 31.1 | 29.4 | 56.6 | 57.7 |  |
| Our Implementations | Vanilla RL | 70.0 | 77.6 | 61.2 | 64.2 | 60.6 | 63.2 | 32.7 | 31.8 | 56.1 | 59.2 |  |
|  | UR^2 (Ours) | 70.1 | 78.8 | 60.1 | 63.2 | 60.7 | 63.2 | 34.3 | 34.0 | 56.3 | 59.8 |  |

## 6 Further Analysis

Additional experimental results are provided in the Appendix, including further ablation studies (Appendix [B.2](https://arxiv.org/html/2508.06165v3#A2.SS2)), the impact of LLM summaries and corpus on UR^2 performance (Appendix [B.3](https://arxiv.org/html/2508.06165v3#A2.SS3)), comparative analysis of retrieval integration in RL training (Appendix [B.4](https://arxiv.org/html/2508.06165v3#A2.SS4)), unsuccessful attempts on reasoning models (Appendix [B.5](https://arxiv.org/html/2508.06165v3#A2.SS5)), and illustrative case studies (Appendix [E](https://arxiv.org/html/2508.06165v3#A5)).

### 6.1 Impact of Online Search

To test scalability under online retrieval, we compare local corpus with real-time search (Table [3](https://arxiv.org/html/2508.06165v3#S6.T3.tab2)). Online search yields consistent gains on MMLU-Pro and medical tasks, and substantial improvements on 2Wiki and Bamboogle, demonstrating strong generalization to scenarios requiring up-to-date or non-Wikipedia knowledge. The only exceptions are math—where Qwen-3-32B’s parametric knowledge already covers the required formulas and axioms—and HotpotQA, where rate

**Table 3: Comparison of UR^2 Qwen-2.5-7B-Instruct using Local Corpus vs. Online Search across different tasks. ${\dagger}$ = in-domain; ${\ddagger}$ = out-of-domain.**
| Corpus | MMLU-Pro | Medicine | Math |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Hist.^† | Phil.^† | Econ.^† | Law^‡ | Avg | MedQA^† | M-Med^‡ | Avg | Math500^† | Minerva^‡ | Avg |  |
| Local Corpus | 53.2 | 53.0 | 72.2 | 35.0 | 53.3 | 69.6 | 62.8 | 65.9 | 80.9 | 61.0 | 71.0 |
| Online Search | 57.7 | 57.8 | 71.0 | 35.0 | 55.4 | 70.4 | 65.4 | 67.9 | 78.7 | 61.2 | 70.0 |

limits block access to many gold Wikipedia pages. Notably, our setting does not enforce full top-$10$ coverage, which naturally introduces noise and better reflects real-world retrieval conditions. Overall, these results confirm the robustness of our method in noisy online environments.

### 6.2 Ablation Study

To validate the effectiveness of UR^2, we conduct comprehensive ablation studies on its key components. As shown in Table [4](https://arxiv.org/html/2508.06165v3#S6.T4), all variants exhibit performance degradation. The W/o Stage-1 variant causes notable drops (5.2% in History, 4.2% in Economics), demonstrating that explicit retrieval activation is essential. The W/o $P_{\text{fallback}}$ variant shows slight improvements on Law and MedQA but generates unreasonable queries, such as “which option is right”. The W/o LLM Summary variant completely fails, as models degrade to pure CoT, highlighting the necessity of addressing retrieval noise in RAG-RL methods. The W/o Task Mixing variant shows minimal changes, indicating that our selective strategy improves efficiency while enhancing accuracy. Using alternative LLMs for corpus summarization (4omini/Qw3-8B) results in consistent 3-4% drops across tasks but still outperforms vanilla RL, demonstrating our method’s adaptability in resource-constrained settings. These results show that our two-stage training, difficulty-aware retrieval, and carefully designed reward components work synergistically to achieve superior performance.

**Table 4: Ablation study of Qwen-2.5-7B-Instruct on MMLU-Pro and medical reasoning tasks. “w/o Task Mixing” means retrieving for all samples. ${\dagger}$ = in-domain; ${\ddagger}$ = out-of-domain.**
| Method | MMLU-Pro | Medicine |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Hist.^† | Phil.^† | Econ.^† | Law^‡ | Avg | MedQA^† | M-Med^‡ | Avg |  |
| UR^2 | 53.2 | 53.1 | 72.2 | 35.0 | 53.3 | 69.6 | 62.8 | 66.2 |
| w/o Stage-1 | 48.0 | 51.1 | 68.0 | 30.9 | 49.5 | 67.6 | 63.0 | 65.3 |
| w/o $P_{\text{fallback}}$ | 52.0 | 51.3 | 68.4 | 36.6 | 52.1 | 71.4 | 62.0 | 66.7 |
| w/o Task Mixing | 52.2 | 51.9 | 68.2 | 33.2 | 51.4 | 70.0 | 63.6 | 66.8 |
| w/o LLM Summary | – | – | – | – | – | – | – | – |
| Vanilla RL | 52.2 | 43.5 | 64.0 | 33.8 | 48.4 | 64.2 | 57.4 | 60.8 |
| 4omini Summary | 49.3 | 48.8 | 67.4 | 32.4 | 49.5 | 65.0 | 59.2 | 62.1 |
| Qw3-8B Summary | 49.1 | 49.9 | 67.8 | 30.6 | 49.4 | 64.8 | 58.2 | 61.5 |

Table [5](https://arxiv.org/html/2508.06165v3#S6.T5) validates our difficulty-aware data selection strategy. Despite using significantly less data, filtered datasets achieve comparable or superior performance, particularly on out-of-domain tasks (Bamboogle improves from 58.9% to 62.7%). This confirms that RL training benefits more from high-quality, difficulty-balanced samples than large-scale unfiltered data, enabling computationally efficient training while maintaining strong performance across diverse tasks.

**Table 5: Training Data Ablation for Qwen-2.5-7B-Instruct (Vanilla RL on Open-domain QA w/o Summary and Math Tasks). We report F1 scores (%) for open-domain QA.**
| Method | HotpotQA^† | 2Wiki^† | Bamboogle^‡ | MusiQue^‡ | Math^† | Minerva^‡ |
| --- | --- | --- | --- | --- | --- | --- |
| Raw Data | 72.3 | 61.9 | 58.9 | 36.1 | 79.0 | 60.5 |
| Filtered Data | 71.0 | 62.0 | 62.7 | 33.8 | 78.2 | 59.4 |

## 7 Conclusion

In this work, we presented UR^2, a unified framework that integrates retrieval-augmented generation with reasoning through reinforcement learning. Unlike existing RAG-RL approaches limited to specific domains, UR^2 demonstrates versatility across mathematical reasoning, medical QA, and open-domain tasks. Our key innovations—difficulty-aware curriculum learning and an LLM-summarized retrieval corpus—enable dynamic retrieval-reasoning coordination by learning *when* and *what* to retrieve based on problem difficulty, while preserving native reasoning capabilities. UR^2 represents a significant step toward adaptive AI systems that flexibly combine parametric knowledge with dynamic information access.

## References

- Borgeaud et al. (2022)
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al.
Improving language models by retrieving from trillions of tokens.
In *International conference on machine learning*, pp. 2206–2240. PMLR, 2022.
- Chen et al. (2025)
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al.
An empirical study on eliciting and improving r1-like reasoning models.
*arXiv preprint arXiv:2503.04548*, 2025.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv e-prints*, pp. arXiv–2407, 2024.
- Edge et al. (2024)
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson.
From local to global: A graph rag approach to query-focused summarization.
*arXiv preprint arXiv:2404.16130*, 2024.
- Gao et al. (2023)
Tianyu Gao, Zhiyuan Liu, Xu Han, Ningyu Zhang, Jilin Tang, Maosong Sun, et al.
Retrieval-augmented generation for large language models: A survey.
*arXiv preprint arXiv:2302.05100*, 2023.
- Guo et al. (2025a)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025a.
- Guo et al. (2025b)
Yiduo Guo, Zhen Guo, Chuanwei Huang, Zi-Ang Wang, Zekai Zhang, Haofei Yu, Huishuai Zhang, and Yikang Shen.
Synthetic data rl: Task definition is all you need.
*arXiv preprint arXiv:2505.17063*, 2025b.
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset.
*arXiv preprint arXiv:2103.03874*, 2021.
- Ho et al. (2020)
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa.
Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps.
In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), *Proceedings of the 28th International Conference on Computational Linguistics*, pp. 6609–6625, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.
doi: 10.18653/v1/2020.coling-main.580.
URL [https://aclanthology.org/2020.coling-main.580/](https://aclanthology.org/2020.coling-main.580/).
- Hu et al. (2024)
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao.
Openrlhf: An easy-to-use, scalable and high-performance rlhf framework.
*arXiv preprint arXiv:2405.11143*, 2024.
- Hu et al. (2025a)
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum.
Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.
*arXiv preprint arXiv:2503.24290*, 2025a.
- Hu et al. (2025b)
Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao.
GRAG: Graph retrieval-augmented generation.
In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), *Findings of the Association for Computational Linguistics: NAACL 2025*, pp. 4145–4157, Albuquerque, New Mexico, April 2025b. Association for Computational Linguistics.
ISBN 979-8-89176-195-7.
URL [https://aclanthology.org/2025.findings-naacl.232/](https://aclanthology.org/2025.findings-naacl.232/).
- Izacard et al. (2022)
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave.
Few-shot learning with retrieval augmented language models.
*arXiv preprint arXiv:2208.03299*, 2022.
- Jin et al. (2025)
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning.
*arXiv preprint arXiv:2503.09516*, 2025.
- Jin et al. (2021)
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits.
What disease does this patient have? a large-scale open domain question answering dataset from medical exams.
*Applied Sciences*, 11(14):6421, 2021.
- Kojima et al. (2022)
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.
Large language models are zero-shot reasoners.
*Advances in neural information processing systems*, 35:22199–22213, 2022.
- Korkmaz (2024)
Ezgi Korkmaz.
Understanding and diagnosing deep reinforcement learning.
*arXiv preprint arXiv:2406.16979*, 2024.
- Lewis et al. (2020)
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
*Advances in Neural Information Processing Systems*, 33:9459–9474, 2020.
- Lewkowycz et al. (2022)
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al.
Solving quantitative reasoning problems with language models.
*Advances in neural information processing systems*, 35:3843–3857, 2022.
- Li et al. (2024)
Weitao Li, Junkai Li, Weizhi Ma, and Yang Liu.
Citation-enhanced generation for LLM-based chatbots.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 1451–1466, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.79.
URL [https://aclanthology.org/2024.acl-long.79/](https://aclanthology.org/2024.acl-long.79/).
- Li et al. (2025)
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou.
Search-o1: Agentic search-enhanced large reasoning models.
*arXiv preprint arXiv:2501.05366*, 2025.
- Ma et al. (2025)
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen.
General-reasoner: Advancing llm reasoning across all domains.
*arXiv preprint arXiv:2505.14652*, 2025.
- Nagarajan et al. (2018)
Prabhat Nagarajan, Garrett Warnell, and Peter Stone.
Deterministic implementations for reproducibility in deep reinforcement learning.
*arXiv preprint arXiv:1809.05676*, 2018.
- Peng et al. (2024)
Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang.
Graph retrieval-augmented generation: A survey.
*arXiv preprint arXiv:2408.08921*, 2024.
- Petroni et al. (2021)
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al.
Kilt: a benchmark for knowledge intensive language tasks.
In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp. 2523–2544, 2021.
- Press et al. (2023)
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis.
Measuring and narrowing the compositionality gap in language models.
In *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp. 5687–5711, 2023.
- Rajbhandari et al. (2020)
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.
Zero: Memory optimizations toward training trillion parameter models.
In *SC20: International Conference for High Performance Computing, Networking, Storage and Analysis*, pp. 1–16. IEEE, 2020.
- Ren et al. (2025)
Jingyi Ren, Yekun Xu, Xiaolong Wang, Weitao Li, Weizhi Ma, and Yang Liu.
Effective and transparent rag: Adaptive-reward reinforcement learning for decision traceability.
*arXiv preprint arXiv:2505.13258*, 2025.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Song et al. (2025a)
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen.
R1-searcher: Incentivizing the search capability in llms via reinforcement learning.
*arXiv preprint arXiv:2503.05592*, 2025a.
- Song et al. (2025b)
Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen.
R1-searcher++: Incentivizing the dynamic knowledge acquisition of llms via reinforcement learning.
*arXiv preprint arXiv:2505.17005*, 2025b.
- Sun et al. (2024)
Hao Sun, Hengyi Cai, Bo Wang, Yingyan Hou, Xiaochi Wei, Shuaiqiang Wang, Yan Zhang, and Dawei Yin.
Towards verifiable text generation with evolving memory and self-reflection.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 8211–8227, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.469.
URL [https://aclanthology.org/2024.emnlp-main.469/](https://aclanthology.org/2024.emnlp-main.469/).
- Sun et al. (2025)
Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang.
Zerosearch: Incentivize the search capability of llms without searching.
*arXiv preprint arXiv:2505.04588*, 2025.
- Trivedi et al. (2022)
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.
MuSiQue: Multihop questions via single-hop question composition.
*Transactions of the Association for Computational Linguistics*, 10:539–554, 2022.
doi: 10.1162/tacl˙a˙00475.
URL [https://aclanthology.org/2022.tacl-1.31/](https://aclanthology.org/2022.tacl-1.31/).
- Wang et al. (2024a)
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al.
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.
*Advances in Neural Information Processing Systems*, 37:95266–95290, 2024a.
- Wang et al. (2024b)
Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li, Xiaojian Ma, and Yitao Liang.
Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation.
*arXiv preprint arXiv:2403.05313*, 2024b.
- Williams (1992)
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning.
*Machine learning*, 8:229–256, 1992.
- Xiong et al. (2025)
Guangzhi Xiong, Qiao Jin, Xiao Wang, Yin Fang, Haolin Liu, Yifan Yang, Fangyuan Chen, Zhixing Song, Dengyu Wang, Minjia Zhang, et al.
Rag-gym: Optimizing reasoning and search agents with process supervision.
*arXiv preprint arXiv:2502.13957*, 2025.
- Yang et al. (2024)
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al.
Qwen2. 5 technical report.
*arXiv e-prints*, pp. arXiv–2412, 2024.
- Yang et al. (2018)
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning.
Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pp. 2369–2380, 2018.
- Yu et al. (2025)
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025.
- Zeng et al. (2025)
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He.
Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.
*arXiv preprint arXiv:2503.18892*, 2025.
- Zhang et al. (2025)
Xiaoying Zhang, Hao Sun, Yipeng Zhang, Kaituo Feng, Chao Yang, and Helen Meng.
Critique-grpo: Advancing llm reasoning with natural language and numerical feedback.
*arXiv preprint arXiv:2506.03106*, 2025.
- Zheng et al. (2025)
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu.
Deepresearcher: Scaling deep research via reinforcement learning in real-world environments.
*arXiv preprint arXiv:2504.03160*, 2025.

## Appendix A Limitations, Future Directions, and AI Usage

While UR^2 shows strong performance across diverse tasks, some limitations remain. First, we have not scaled beyond 8B parameters due to computational limits. Second, our reliance on LLM-summarized corpora may not fully reflect the complexity of raw web content. Third, the two-stage training and corpus preprocessing add extra computational cost. Despite these issues, UR^2 achieves substantial gains (up to 29.6% improvement) and generalizes well across domains.

Future work will explore updated models and frameworks, scaling UR^2 to 32B parameters, and incorporating online corpora during training to better capture real-world retrieval dynamics. We also plan to investigate more efficient training strategies to reduce costs.

##### Use of AI Tools.

In preparing this work, we used commercial LLMs (e.g., Claude 4.0) for non-creative assistance such as language polishing, formatting, and minor code edits. These tools were not involved in method design, experimental setup, or any substantive creative contribution.

## Appendix B Additional Experimental Results

### B.1 Comprehensive Supplementary Results on Open-Domain and Reasoning Tasks

Tables [6](https://arxiv.org/html/2508.06165v3#A2.T6) and [7](https://arxiv.org/html/2508.06165v3#A2.T7) provide supplementary experimental results, focusing on Advanced RAG methods across different model scales and GPT-4o-mini performance on open-domain QA tasks.

**Table 6: Extended results on GPT-4o-mini, Qwen-2.5-3B-Instruct, and LLaMA-3.1-8B-Instruct across reasoning tasks. We report EM scores (%) for MMLU-Pro and MedQA, and LLM-as-a-judge scores (%) for math benchmarks. ${\dagger}$ = in-domain; ${\ddagger}$ = out-of-domain.**
| Method | MMLU-Pro | Medicine | Math |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Hist.^† | Phil.^† | Econ.^† | Law^‡ | Avg | MedQA^† | M-Med^‡ | Avg | Math500^† | Minerva^‡ | Avg |  |
| Qwen-2.5-3B |  |  |  |  |  |  |  |  |  |  |  |
| CoT | 33.6 | 32.3 | 48.8 | 20.6 | 33.8 | 39.4 | 36.8 | 38.1 | 63.6 | 39.9 | 51.8 |
| Standard RAG | 37.8 | 36.5 | 51.4 | 23.2 | 37.1 | 45.6 | 40.0 | 42.8 | 65.3 | 40.8 | 53.1 |
| Advanced RAG Methods |  |  |  |  |  |  |  |  |  |  |  |
| Self-Ask | 33.1 | 30.5 | 44.2 | 20.2 | 32.0 | 39.6 | 36.0 | 37.8 | 58.3 | 40.3 | 49.3 |
| RAT | 37.8 | 33.5 | 50.6 | 18.4 | 35.1 | 47.4 | 42.6 | 45.0 | 67.9 | 44.9 | 56.4 |
| Search-o1 | 33.9 | 34.5 | 50.6 | 20.6 | 34.9 | 44.6 | 37.4 | 41.0 | 69.4 | 43.0 | 56.2 |
| Our Implementations |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla RL | 40.7 | 34.7 | 55.0 | 24.6 | 38.7 | 51.8 | 47.6 | 49.7 | 68.0 | 43.9 | 52.4 |
| UR^2 | 47.8 | 49.3 | 63.9 | 30.0 | 47.8 | 59.8 | 56.8 | 58.3 | 69.4 | 45.0 | 57.2 |
| LLaMA-3.1-8B |  |  |  |  |  |  |  |  |  |  |  |
| CoT | 37.8 | 40.9 | 53.4 | 29.0 | 40.3 | 59.6 | 52.6 | 56.1 | 48.4 | 34.4 | 41.4 |
| Standard RAG | 43.6 | 33.9 | 51.0 | 26.6 | 38.8 | 56.4 | 53.2 | 54.8 | 45.0 | 31.4 | 38.2 |
| Advanced RAG Methods |  |  |  |  |  |  |  |  |  |  |  |
| Self-Ask | 39.8 | 32.1 | 47.0 | 23.4 | 35.6 | 53.0 | 42.8 | 47.9 | 46.9 | 27.0 | 37.0 |
| RAT | 42.3 | 37.7 | 52.6 | 28.6 | 40.3 | 63.8 | 56.0 | 59.9 | 50.1 | 36.8 | 43.5 |
| Search-o1 | 32.6 | 32.5 | 46.0 | 28.0 | 45.9 | 56.0 | 46.0 | 56.6 | 41.5 | 27.8 | 34.7 |
| Our Implementations |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla RL | 44.6 | 36.9 | 53.0 | 26.4 | 40.2 | 66.8 | 57.4 | 62.1 | 45.5 | 43.4 | 44.4 |
| UR^2 | 48.3 | 38.6 | 58.0 | 28.8 | 43.4 | 68.6 | 58.4 | 63.5 | 54.5 | 39.0 | 46.8 |

The extended results reveal significant performance limitations of Advanced RAG methods for open-source models. On Qwen-2.5-3B, Self-Ask achieves only 32.0% on MMLU-Pro, substantially underperforming even basic CoT (33.8%). RAT shows inconsistent performance, achieving competitive results on medical tasks (45.0%) but poor performance on Law (18.4%), indicating fragility in cross-domain generalization. Search-o1 demonstrates moderate effectiveness, reaching 41.0% on medical tasks, but fails to achieve consistent improvements across reasoning domains. On LLaMA-3.1-8B, Advanced RAG methods exhibit mixed results. While RAT achieves reasonable performance on Medicine (59.9%) and Math (43.5%), Self-Ask and Search-o1 show notable degradation compared to basic CoT on several sub-domains. These results highlight the challenge of scaling sophisticated retrieval mechanisms to diverse model architectures and reasoning tasks.

GPT-4o-mini establishes strong performance on open-domain QA, with Search-o1 achieving 48.9% F1 average, significantly outperforming other Advanced RAG methods (41.3 and 41.8%). Additionally, RAT and Self-Ask incur prohibitive API costs due to their sentence-level analysis and rewriting operations, making them impractical for large-scale deployment. Notably, Standard RAG achieves competitive performance (42.1% F1) on GPT-4o-mini, suggesting that larger commercial models can effectively leverage simple retrieval without sophisticated coordination mechanisms. The performance gap between GPT-4o-mini (48.9%) and smaller models, such as Qwen-2.5-3B (27.8%) for Search-o1, highlights the substantial challenge of achieving effective retrieval-reasoning integration in resource-constrained settings and validates the necessity of our specialized framework design.

**Table 7: Extended results of GPT-4o-mini, Qwen-2.5-3B, and LLaMA-3.1-8B on open-domain QA. We report F1 and LLM-as-a-judge (LSJ) scores, both in %. ${\dagger}$ denotes in-domain datasets; ${\ddagger}$ indicates out-of-domain.**
| Models | Types | Methods | Hotpot^† | 2Wiki^† | Bamb.^‡ | MusiQ.^‡ | Avg |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | F1 | LSJ | F1 | LSJ | F1 | LSJ | F1 | LSJ | F1 | LSJ |  |  |
| GPT-4o-mini | Vanilla Methods | CoT | 46.5 | 51.2 | 35.0 | 35.4 | 55.2 | 62.4 | 24.9 | 26.8 | 40.4 | 44.0 |
|  | Standard RAG | 59.6 | 69.6 | 43.0 | 45.8 | 46.7 | 46.4 | 19.3 | 21.6 | 42.1 | 45.9 |  |
| Advanced RAG | Self-Ask | 45.0 | 50.4 | 36.9 | 40.0 | 59.3 | 57.6 | 26.1 | 27.8 | 41.8 | 44.0 |  |
|  | RAT | 53.8 | 59.2 | 34.1 | 34.8 | 53.0 | 51.2 | 24.3 | 24.8 | 41.3 | 42.5 |  |
|  | Search-o1 | 64.3 | 73.4 | 47.3 | 52.0 | 54.5 | 56.0 | 29.6 | 30.2 | 48.9 | 52.9 |  |
| Qwen-2.5-3B | Vanilla Methods | CoT | 26.6 | 27.2 | 22.7 | 22.6 | 31.2 | 33.6 | 11.3 | 9.6 | 23.0 | 23.3 |
|  | Standard RAG | 50.6 | 57.0 | 29.8 | 30.4 | 26.1 | 27.2 | 9.7 | 7.4 | 29.1 | 30.5 |  |
| Advanced RAG | Self-Ask | 33.8 | 47.2 | 21.0 | 28.8 | 30.6 | 32.0 | 14.5 | 14.8 | 25.0 | 30.7 |  |
|  | RAT | 30.1 | 32.2 | 15.1 | 15.4 | 30.6 | 28.0 | 11.0 | 8.2 | 21.7 | 21.0 |  |
|  | Search-o1 | 36.4 | 37.6 | 30.8 | 31.8 | 31.4 | 32.0 | 12.5 | 10.0 | 27.8 | 27.9 |  |
| RAG-RL | Search-R1 | 63.1 | 69.2 | 49.5 | 53.4 | 48.3 | 48.0 | 27.6 | 27.8 | 49.6 | 47.1 |  |
|  | Zero-Search | 42.7 | 45.8 | 26.1 | 27.6 | 32.4 | 31.2 | 16.9 | 17.0 | 29.5 | 30.4 |  |
| Our Implementations | Vanilla RL | 65.9 | 73.6 | 54.9 | 58.0 | 59 | 57.6 | 30.0 | 29.6 | 52.5 | 54.7 |  |
|  | UR^2 | 67.7 | 76.0 | 55.2 | 58.6 | 57.8 | 58.4 | 30.5 | 31.6 | 55.3 | 56.2 |  |
| LLaMA-3.1-8B | Vanilla Methods | CoT | 28.6 | 31.6 | 16.4 | 17.8 | 43.0 | 42.4 | 9.8 | 10.8 | 24.5 | 25.7 |
|  | Standard RAG | 47.5 | 54.4 | 26.2 | 26.4 | 26.5 | 28.0 | 10.1 | 10.2 | 27.6 | 29.8 |  |
| Advanced RAG | Self-Ask | 43.0 | 50.8 | 27.3 | 29.8 | 41.5 | 44.8 | 16.8 | 16.4 | 32.2 | 35.5 |  |
|  | RAT | 44.5 | 48.8 | 16.4 | 15.6 | 39.7 | 39.2 | 17.0 | 16.0 | 29.4 | 29.9 |  |
|  | Search-o1 | 53.0 | 59.4 | 37.5 | 38.4 | 30.0 | 30.4 | 15.9 | 16.2 | 34.1 | 36.1 |  |
| RAG-RL | R1-Searcher | 70.8 | 76.8 | 59.6 | 62.2 | 64.7 | 62.4 | 31.1 | 29.4 | 56.6 | 57.7 |  |
| Our Implementations | Vanilla RL | 70.0 | 77.6 | 61.2 | 64.2 | 60.6 | 63.2 | 32.7 | 31.8 | 56.1 | 59.2 |  |
|  | UR^2 | 70.1 | 78.8 | 60.1 | 63.2 | 60.7 | 63.2 | 34.3 | 34.0 | 56.3 | 59.8 |  |

### B.2 Additional Ablation Results

To ensure a fair comparison, we evaluate Vanilla RL MCQ (MMLU-Pro and Medicine tasks), which trains on mixed multiple-choice tasks similar to UR^2. As shown in Table [8](https://arxiv.org/html/2508.06165v3#A2.T8), Vanilla RL MCQ exhibits task-dependent performance: on Qwen-2.5-7B it improves Medicine performance (62.8% vs. 60.8%) but lowers MMLU-Pro scores (47.0% vs. 48.4%), with the reverse trend on 3B models. Despite these gains, UR^2 consistently outperforms both Vanilla RL variants across all domains and scales, achieving average improvements on MMLU-Pro of 5.9% for 7B models and 9.1% for 3B models, confirming that its advantage arises from the unified retrieval-reasoning framework rather than task mixing alone.

**Table 8: Ablation study of Vanilla RL on Qwen-2.5-7B-Instruct and Qwen-2.5-3B-Instruct across multiple-choice reasoning tasks.**
| Method | MMLU-Pro | Medicine |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Hist.^† | Phil.^† | Econ.^† | Law^‡ | Avg | MedQA^† | M-Med^‡ | Avg |  |
| Qwen-2.5-7B |  |  |  |  |  |  |  |  |
| Vanilla RL MCQ | 47.2 | 46.1 | 61.8 | 33.0 | 47.0 | 65.6 | 60.0 | 62.8 |
| Vanilla RL | 52.2 | 43.5 | 64.0 | 33.8 | 48.4 | 64.2 | 57.4 | 60.8 |
| UR^2 | 53.2 | 53.0 | 72.2 | 35.0 | 53.3 | 69.6 | 62.8 | 65.9 |
| Qwen-2.5-3B |  |  |  |  |  |  |  |  |
| Vanilla RL MCQ | 42.3 | 37.1 | 57.4 | 25.0 | 40.6 | 50.2 | 45.0 | 47.6 |
| Vanilla RL | 40.7 | 34.7 | 55.0 | 24.6 | 38.7 | 51.8 | 47.6 | 49.7 |
| UR^2 | 47.8 | 49.3 | 63.9 | 30.0 | 47.8 | 59.8 | 56.8 | 58.3 |

We further conduct ablation studies on UR^2 in open-domain QA tasks (Table [10](https://arxiv.org/html/2508.06165v3#A2.T10)).The w/o Math Data variant shows minimal impact (0.3-1.4% drops), confirming multi-task training preserves QA performance. Additionally removing LLM Summary causes larger drops on out-of-domain tasks (2.0% on MusiQue) while maintaining in-domain performance, indicating LLM-summarized corpus benefits generalization. The weaker Stage-1 variant shows the largest degradation on Bamboogle (5.5% drop), highlighting proper retrieval initialization is crucial for complex multi-hop reasoning. These results validate our design choices contribute meaningfully across diverse task types.

**Table 9: Ablation Study of Qwen-2.5-7B-Instruct on open-domain QA. We report F1 scores (in %) here. The second variant removes LLM Summary on top of the first variant without Math Data.**
| Method | Hotpot^† | 2Wiki^† | Bamb.^‡ | MusiQ.^‡ |
| --- | --- | --- | --- | --- |
| UR^2 | 71.2 | 62.6 | 64.5 | 35.8 |
| w/o Math Data | 70.9 | 61.2 | 63.3 | 34.4 |
| w/o LLM Summary | 71.0 | 62.0 | 62.7 | 33.8 |
| weaker Stage-1 | 69.5 | 62.2 | 59.0 | 34.4 |

Overall, the ablations confirm that Stage-1 initialization is crucial for complex reasoning, difficulty-aware filtering yields better performance with fewer samples, and task mixing improves efficiency without accuracy loss. Importantly, LLM-summarized retrieval highlights the necessity of addressing retrieval noise in RAG-RL methods, guiding more stable and generalizable reasoning.

### B.3 Impact of LLM Summary and Corpus on UR 2 Performance

Table [10](https://arxiv.org/html/2508.06165v3#A2.T10) examines the robustness of UR^2 across different LLM summary sources. Remarkably, our framework maintains strong performance regardless of the summarization model quality. While GPT-4.1 achieves the best results (53.4% average), even using smaller open-source models like Qwen-3-8B (52.0%) or budget-friendly APIs like GPT-4o-mini (50.8%) yields substantial improvements over Vanilla RL (48.4%). Most notably, the w/o Summary variant still achieves 50.2%–demonstrating that our two-stage training and retrieval-aware prompting mechanisms are inherently robust and not dependent on expensive summarization models. This flexibility makes UR^2 practically deployable across various computational budgets while maintaining its effectiveness, confirming the generalizability of our approach beyond specific model configurations.

Table [11](https://arxiv.org/html/2508.06165v3#A2.T11) investigates the impact of different corpus configurations on UR^2’s performance across open-domain QA tasks. The results reveal several key insights about corpus design choices. First, using Wikipedia abstracts (Abs, released with HotpotQA) versus full articles (Full) shows task-dependent effects: abstracts perform better on easy questions (HotpotQA), while full articles excel on complex reasoning tasks requiring broader context (2Wiki, Bamboogle, MusiQue). Second, the presence of LLM summarization consistently improves performance across all configurations, with average F1 scores increasing by 6.5-10.8% when summaries are applied. Notably, UR^2 maintains competitive performance even without summaries (50.6% F1 with Abs, 47.4% with Full), substantially outperforming ZeroSearch’s reliance on synthetic content. The retrieval frequency (#R) analysis shows that UR^2 strategically balances retrieval calls—using fewer retrievals than Search-R1 while achieving superior performance, demonstrating more efficient knowledge utilization.

**Table 11: Performance of UR^2 and baselines on open-domain QA datasets across different corpus configurations. Abs denotes corpora based on Wikipedia abstracts, while Full uses full articles. For each corpus, we use top-$10$ documents with summaries and top-$5$ without. #R represents the number of successful retrievals per question.**
| Corpus | Summ. | Models | Hotpot^† | 2Wiki^† | Bamb.^‡ | MusiQ.^‡ | Avg |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| F1 | LSJ | #R | F1 | LSJ | #R | F1 | LSJ | #R | F1 | LSJ | #R | F1 | LSJ | #R |  |  |  |
| Abs | ✓ | ZeroSearch | 46.0 | 50.4 | 0.66 | 38.4 | 38.6 | 0.73 | 35.8 | 38.4 | 0.54 | 14.7 | 13.8 | 0.62 | 33.7 | 35.3 | 0.64 |
| Search-R1 | 72.4 | 78.8 | 1.92 | 61.0 | 63.8 | 3.16 | 58.9 | 56.8 | 2.58 | 32.2 | 32.0 | 2.92 | 56.1 | 57.9 | 2.64 |  |  |
| R1-Searcher | 71.8 | 78.0 | 1.93 | 57.9 | 63.6 | 2.17 | 56.5 | 53.6 | 2.02 | 33.2 | 32.6 | 2.33 | 54.9 | 57.0 | 2.11 |  |  |
| UR^2 | 71.2 | 79.4 | 2.22 | 62.6 | 65.0 | 2.72 | 64.5 | 62.4 | 2.30 | 35.8 | 34.6 | 2.61 | 58.5 | 60.4 | 2.46 |  |  |
| Abs | ✗ | ZeroSearch | 44.1 | 47.0 | 0.64 | 32.9 | 31.8 | 0.66 | 32.6 | 35.2 | 0.52 | 14.3 | 11.8 | 0.61 | 31.0 | 31.5 | 0.61 |
| Search-R1 | 65.8 | 72.4 | 2.68 | 41.8 | 51.6 | 3.54 | 44.8 | 44.8 | 2.96 | 25.1 | 24.1 | 3.49 | 44.4 | 48.2 | 3.17 |  |  |
| R1-Searcher | 69.7 | 75.2 | 2.16 | 56.6 | 58 | 2.45 | 41.7 | 40.0 | 2.38 | 23.7 | 22.4 | 2.84 | 47.9 | 48.9 | 2.46 |  |  |
| UR^2 | 67.6 | 73.6 | 1.98 | 59.1 | 59.6 | 2.53 | 47.5 | 47.2 | 2.10 | 28.2 | 25.4 | 2.43 | 50.6 | 51.5 | 2.26 |  |  |
| Full | ✓ | ZeroSearch | 44.3 | 48.8 | 0.74 | 36.5 | 36.8 | 0.90 | 46.3 | 44.8 | 0.70 | 19.3 | 20.0 | 0.81 | 36.6 | 37.6 | 0.79 |
| Search-R1 | 66.0 | 67.2 | 2.01 | 60.6 | 65.6 | 3.12 | 70.0 | 71.2 | 2.06 | 37.8 | 39.0 | 2.69 | 58.6 | 60.8 | 2.47 |  |  |
| R1-Searcher | 62.9 | 68.0 | 1.97 | 62.5 | 66.8 | 2.15 | 69.0 | 65.6 | 1.86 | 36.7 | 37.8 | 2.24 | 57.8 | 59.6 | 2.06 |  |  |
| UR^2 | 62.6 | 68.0 | 2.11 | 63.3 | 67.6 | 2.73 | 73.0 | 74.0 | 2.13 | 40.4 | 42.0 | 2.55 | 59.8 | 62.9 | 2.38 |  |  |
| Full | ✗ | ZeroSearch | 39.2 | 41.6 | 0.63 | 34.0 | 33.8 | 0.67 | 34.1 | 36.0 | 0.50 | 13.4 | 11.8 | 0.58 | 30.2 | 30.8 | 0.59 |
| Search-R1 | 57.4 | 60.6 | 2.75 | 49.2 | 51.0 | 3.50 | 57.6 | 55.2 | 2.82 | 26.9 | 26.4 | 3.40 | 47.8 | 48.3 | 3.12 |  |  |
| R1-Searcher | 57.6 | 61.6 | 2.24 | 56.0 | 59.0 | 2.37 | 57.5 | 57.6 | 2.07 | 26.8 | 26.6 | 2.63 | 49.5 | 51.2 | 2.33 |  |  |
| UR^2 | 54.6 | 60.6 | 2.03 | 54.5 | 55.8 | 2.51 | 52.6 | 49.6 | 2.06 | 27.8 | 26.2 | 2.38 | 47.4 | 48.1 | 2.25 |  |  |

Table [12](https://arxiv.org/html/2508.06165v3#A2.T12) examines corpus selection for domain-specific tasks, comparing general Wikipedia against specialized MedQA textbooks for medical reasoning. The results demonstrate that domain-specific corpora provide marginal improvements when summarization is applied (70.2% vs. 69.6% on MedQA), but this advantage diminishes without summaries. More importantly, the performance gap between summarized and non-summarized variants is substantial (8.4% on MedQA with Wikipedia), highlighting that effective summarization is more critical than corpus specialization. This finding suggests that UR^2’s LLM-summarized approach can effectively bridge the gap between general and specialized knowledge sources, making it practical for deployment across diverse domains without extensive corpus curation.

**Table 12: Ablation study of UR^2 on the medical reasoning tasks. We compare different corpus (Wikipedia vs. MedQA Textbooks) and the effect of applying summarization. “w/o Summary” uses top-3 retrieved document.**
| Corpus | Summ. | Medicine^† | M-Med^‡ |
| --- | --- | --- | --- |
| Wikipedia | ✓ | 69.6 | 62.8 |
| Textbooks | ✓ | 70.2 | 63.8 |
| Wikipedia | ✗ | 61.2 | 59.2 |
| Textbooks | ✗ | 62.0 | 58.0 |

Collectively, Tables [10](https://arxiv.org/html/2508.06165v3#A2.T10),  [11](https://arxiv.org/html/2508.06165v3#A2.T11), and [12](https://arxiv.org/html/2508.06165v3#A2.T12) demonstrate UR^2’s robustness across three critical dimensions: corpus configuration, domain specialization, and summarization quality. The framework maintains strong performance whether using abstracts or full articles, general or specialized corpora, and expensive or budget-friendly summarizers. Most remarkably, even without any summarization, UR^2 achieves competitive results through its two-stage training and difficulty-aware retrieval mechanisms. This comprehensive ablation validates that UR^2’s effectiveness stems from its fundamental architecture rather than dependency on specific external resources, confirming its practical applicability across diverse computational and domain constraints.

### B.4 Comparative Analysis of Retrieval Integration in RL Training

Figure [3](https://arxiv.org/html/2508.06165v3#A2.F3) reveals key differences between UR^2 and Vanilla RL MCQ on Qwen-2.5-3B-Instruct in training. Vanilla RL saturates early at step 40 with 1.1 reward, with later gains mainly due to repeated data every 47 steps. In contrast, UR² steadily improves to 1.4 reward by step 85, matching the 15.7% relative benchmark gain. Retrieval frequency remains dynamic after Stage 1, showing selective use. UR² also generates longer outputs post-training, indicating deeper reasoning. This extended training capability demonstrates that retrieval-augmented approaches fundamentally expand model capacity limits, enabling continuous learning beyond traditional RL saturation points.

Figure: Figure 3: Comparison of Vanilla RL MCQ and UR^2 performance on Qwen-2.5-3B-Instruct across training steps. Peak test set performances are indicated.
Refer to caption: x2.png

### B.5 Unsuccessful Attempts on Reasoning Models

We also conducted experiments on the R1-like model DeepSeek-R1-Distill-Qwen-7B(^5^55https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B). However, when applying the MMLU-Pro prompting setup, we observed that the model lacked any retrieval capability. This remained true even after replacing the original searching special tags with alternative tokens <search></search> and <information></information>, which were shown in the ablation study (Table [4](https://arxiv.org/html/2508.06165v3#S6.T4)) to more effectively trigger retrieval. These results indicate a degradation of tool-usage ability after extensive chain-of-thought training. Due to computational constraints, we did not extend training to more updated models such as the Qwen-3 series. We plan to supplement this work with relevant code and experiments in future updates.

## Appendix C Training Details

### C.1 Training Setting Details

We train UR^2 using the REINFORCE++ algorithm (Guo et al., 2025a), a simplified variant of Proximal Policy Optimization (PPO) designed to encourage exploration. In particular, we discard the critic and omit both KL-divergence and clipping terms, following previous findings (Zhang et al., 2025; Song et al., 2025a; Chen et al., 2025) that excessive regularization can impede effective strategy learning in sparse-reward scenarios.

To reduce overfitting to retrieved content, we adopt a retrieval masking strategy (Sun et al., 2025; Song et al., 2025a; Jin et al., 2025), which treats retrieved external knowledge as part of the observation space rather than trainable input. This encourages the model to reason based on retrieved information without directly optimizing on it. Our implementation builds upon the REINFORCE++ baseline provided by OpenRLHF (Hu et al., 2024).

Each prompt is rolled out $G=16$ times. We use the mean reward of each rollout group as the baseline for computing the advantage of each sample. To stabilize training, we apply a two-stage normalization scheme: normalization is first performed within each rollout group, followed by global normalization across the full batch.

Training is conducted with DeepSpeed ZeRO-2 (Rajbhandari et al., 2020) for memory efficiency. We use gpt-4.1-mini-2025-04-14 as the summarization model during training. Token limits per generation turn are set to 3072 for math tasks, 1536 for multiple-choice questions (MCQ), and 512 for open-domain QA. Sampling parameters are fixed as temperature = 1.0 and top_p = 0.9.

We train for up to 2 epochs. In practice, most models achieve optimal performance within 1.5 epochs. Therefore, we report results from the checkpoint with the best test set performance within the first 1.5 epochs. We save checkpoints every 5 steps for single-task training and 3B models, and every 10 steps for larger-scale experiments. The specific training steps for each reported model are detailed in Table [13](https://arxiv.org/html/2508.06165v3#A3.T13) below. W/o Stage-1 variant in Table [4](https://arxiv.org/html/2508.06165v3#S6.T4) replaces the special tags with <search></search> and <information></information>, removing the initial retrieval capability activation stage. The Weaker Stage-1 variant in Table [10](https://arxiv.org/html/2508.06165v3#A2.T10) employs a modified training protocol based on UR^2 Qwen-2.5-7B-Instruct for MCQ tasks, where retrieval-related rewards are only provided during the initial 10 training steps. The Qw3-8B variant in Table [4](https://arxiv.org/html/2508.06165v3#S6.T4) uses Qwen-3-8B for summarization with max_tokens = 2048, temperature = 0.3, and top_p = 0.7. Specifically, the retrieval reward assigns 0.5 for single retrieval attempts and 1.0 for multiple retrievals ($\geq$2), reflecting a more conservative retrieval activation strategy than that of our proposed method.

Table [13](https://arxiv.org/html/2508.06165v3#A3.T13) summarizes the training configurations and checkpoint details across all model scales. Two key observations can be drawn:

First, for Qwen models, performance consistently improves as more training compute is introduced via our UR^2 method. The method’s design—encouraging structured retrieval behavior—ensures that increased steps and effective epochs lead to meaningful gains across tasks.

Second, while UR^2 also improves performance on LLaMA-3.1-8B, training on this model is observed to be less stable. Performance tends to saturate early (e.g., low effective epochs despite higher step counts), for both Vanilla RL and UR^2 variants. This indicates that LLaMA-3.1-8B may require different training strategies to maintain learning dynamics over time. Future work will explore alternative foundation models and optimization schedules to improve convergence and stability.

**Table 13: Training checkpoint details for UR^2 models. Checkpoints were saved every 5 steps for 3B models and single-task training, and every 10 steps for larger models. Main experiments use the full training configuration, while ablation studies vary specific components.**
| Model | Training Dataset | Dataset Size | Checkpoint Step | Training Epochs |
| --- | --- | --- | --- | --- |
| Qwen-2.5-3B - Main Experiments |  |  |  |  |
| UR^2-Math&QA | Math&QA | 6000 | 47 | 1.0 |
| UR^2-MCQ | MMLU&Medqa | 9000 | 85 | 1.2 |
| Vanilla RL-Math | Math | 3000 | 15 | 0.64 |
| Vanilla RL-QA | QA | 3000 | 40 | 1.7 |
| Vanilla RL-MMLU | MMLU | 6000 | 47 | 1.0 |
| Vanilla RL-MedQA | Medqa | 3000 | 40 | 1.7 |
| Vanilla RL-MCQ | MMLU&Medqa | 9000 | 40 | 0.57 |
| LLaMA-3.1-8B Models - Main Experiments |  |  |  |  |
| UR^2-Math&QA | Math&QA | 6000 | 30 | 0.32 |
| UR^2-MCQ | MMLU&Medqa | 9000 | 30 | 0.21 |
| Vanilla RL-Math | Math | 3000 | 30 | 0.64 |
| Vanilla RL-QA | QA | 3000 | 47 | 1.0 |
| Vanilla RL-MMLU | MMLU | 6000 | 60 | 0.64 |
| Vanilla RL-MedQA | Medqa | 3000 | 30 | 0.64 |
| Qwen-2.5-7B - Main Experiments |  |  |  |  |
| UR^2-Math&QA | Math&QA | 6000 | 40 | 0.43 |
| UR^2-MCQ | MMLU&Medqa | 9000 | 100 | 0.71 |
| Vanilla RL-Math | Math | 3000 | 40 | 0.43 |
| Vanilla RL-QA | QA | 3000 | 25 | 0.53 |
| Vanilla RL-MMLU | MMLU | 6000 | 94 | 1.0 |
| Vanilla RL-MedQA | Medqa | 3000 | 47 | 1.0 |
| Vanilla RL-MCQ | MMLU&Medqa | 9000 | 60 | 0.43 |
| 7B Models - Ablation Studies |  |  |  |  |
| Ablation-MCQ-w/o $P_{\text{fallback}}$ | MMLU&Medqa | 9000 | 110 | 0.78 |
| Ablation-MCQ-w/o Stage-1 | MMLU&Medqa | 9000 | 110 | 0.78 |
| Ablation-MCQ-w/o Task Mixing | MMLU&Medqa | 9000 | 120 | 0.85 |
| Ablation-MCQ-QW3 summary | MMLU&Medqa | 9000 | 50 | 0.36 |
| Ablation-MCQ-4omini summary | MMLU&Medqa | 9000 | 50 | 0.36 |
| Ablation-Math&QA weaker Stage-1 | Math&qa | 6000 | 80 | 0.85 |
| Ablation-QA w/o LLM summary | QA | 3000 | 60 | 1.28 |
| Ablation-QA Raw data | R1-Searcher | 8148 | 70 | 0.55 |
| Ablation-Math Raw data | SimpleRL-Zoo | 16662 | 10 | 0.056 |

### C.2 Evaluation Details

All evaluations are performed using vLLM version 0.6.5. The vLLM version of Qwen-3 used is 0.8.5.post1. In evaluation, We maintain the same max_tokens limits used during training: 3072 for math benchmarks, 1536 for MCQ, and 512 for open-domain QA per generation step. For GPT-family models, these limits are increased to 4096, 2048, and 1024, respectively. For sampling during evaluation, we use more conservative hyperparameters: temperature = 0.3 and top_p = 0.5, aiming for higher answer consistency. Summarization for math tasks is conducted using Qwen-3-32B with max_tokens = 8192, temperature = 0.3, and top_p = 0.7. Final evaluation summarization is performed using gpt-4.1-2025-04-14 with max_tokens = 2048, temperature = 0.3, and top_p = 0.5.

The RL methods mentioned in this paper all follow the settings described in their original works. Specifically, Open-Reasoner-Zero, General Reasoner, SimpleRL-Zoo, R1-Searcher, Search-R1, and ZeroSearch are implemented using the Qwen-2.5-Base models. Although an Instruct version of Search-R1 exists, its performance is significantly inferior and thus excluded from comparison. R1-Searcher with LLaMA-3.1-8B adopts the Instruct variant. Vanilla methods, including CoT and standard RAG, are applied using the Instruct versions for all open-source models.

Advanced RAG Baseline Implementations:

Search-o1 with Retrieval-Augmented Generation: We adapt the Search-o1 framework (Li et al., 2025) to operate within a controlled evaluation environment. While maintaining its core iterative reasoning mechanism and document analysis capabilities, our implementation leverages the KILT Wikipedia corpus with BGE-large-en-v1.5 embeddings for knowledge retrieval. This approach consolidates the multi-agent architecture into a unified model with structured prompting, ensuring consistent evaluation across all baselines while preserving the essential reasoning patterns.

Self-Ask with Retrieval-Augmented Generation: Our implementation follows the Self-Ask framework’s (Press et al., 2023) question decomposition strategy, employing batch retrieval from the local KILT corpus to enhance efficiency. The system maintains the characteristic “Follow up:” and “Intermediate answer:” reasoning chain format, with stopping criteria incorporating both semantic completion detection and a maximum of 10 follow-up questions. When decomposition challenges arise, the framework seamlessly transitions to standard RAG, ensuring robust performance across diverse question types.

RAT (Retrieval-Augmented Thought): We adapt RAT (Wang et al., 2024b) for unified evaluation across reasoning and QA tasks. The framework retains the core principle of knowledge-enhanced reasoning while operating at the paragraph level rather than the sentence level, with corresponding modifications to the prompting strategy. This design choice maintains consistency with our evaluation infrastructure while capturing RAT’s fundamental insight of augmenting reasoning processes with relevant external knowledge.

All advanced RAG methods operate within a standardized retrieval infrastructure: documents are retrieved from the 100-word segmented KILT Wikipedia corpus (29M documents in total). For GPT-family models, we use top-$k$=10 retrieval. Due to model limitations, LLaMA and Qwen variants use top-$k$=5. For summarization or other auxiliary operations beyond reasoning, each model performs the processing itself rather than relying on GPT-4.1, ensuring consistency with its own capabilities.

Online Corpus Retrieval Implementation:

To evaluate the generalization capability of UR^2 with real-world web content, we implement an online corpus retrieval system that dynamically fetches and processes web documents. Unlike the offline Wikipedia corpus used during training, this online retrieval mechanism provides access to up-to-date information from the internet.

The online retrieval pipeline consists of three main components:

Web Search and Content Extraction: We utilize the Bing Search API to retrieve relevant URLs based on the model’s search queries. To ensure robust retrieval quality, we implement a multi-round crawling strategy with up to three rounds of attempts. In each round, the system fetches $k\times 3$ candidate URLs and crawls them in parallel using a thread pool with 256 workers. The system implements intelligent retry logic—if the initial $k$ URLs fail to provide sufficient valid content, it automatically attempts to crawl additional URLs from the candidate pool. This approach significantly improves the success rate of obtaining high-quality content.

HTML-to-Markdown Conversion: Raw HTML content from web pages often contains noise such as navigation elements, advertisements, and scripts. We deploy a dedicated service using ReaderLM-v2-1.5B Model (^6^66https://huggingface.co/jinaai/reader-lm-1.5b) through the vLLM framework to convert HTML to clean Markdown format. The preprocessing pipeline removes script tags, style elements, base64-encoded images, and other irrelevant content using optimized regular expressions. The model then generates readable Markdown that preserves the main textual information while discarding formatting artifacts. To improve efficiency, we implement an LRU cache with a capacity of 10,000 entries, achieving significant speedup for repeated content.

Content Summarization: The summarization prompt is carefully designed to distinguish between knowledge-based queries (which can be answered with factual information) and reasoning-based queries (which require complex computation). For knowledge-based queries, the model extracts and presents relevant facts; for reasoning-based queries, it returns a fallback message indicating that direct reasoning is more appropriate. The summarizer here is GPT-4.1-2025-04-14.

The entire pipeline is orchestrated through a FastAPI service that handles concurrent requests efficiently. Rate limiting is enforced for the Bing API (95 requests per second) to comply with usage policies. The system maintains detailed logging for debugging and performance monitoring, tracking metrics such as cache hit rates, crawling success rates, and end-to-end latency.

Due to network and hardware limitations, a small portion of Wikipedia pages failed to be crawled correctly, and a subset of queries did not receive valid responses. Given constraints on time and budget, no additional remediation was applied to these cases. However, this reflects the system’s alignment with real-world deployment settings, where large-scale QA systems must be robust to occasional retrieval failures and operate under imperfect infrastructure.

This online retrieval implementation enables UR^2 to access current information beyond its training data, demonstrating its ability to integrate real-time knowledge into the reasoning process.

### C.3 Training Dataset Details

We construct a unified training set that spans multiple task domains to ensure comprehensive coverage of diverse reasoning and knowledge-based challenges. For mathematical reasoning capabilities, we incorporate data from the training split of SimpleZoo-RL, which provides a rich collection of mathematical problem-solving scenarios from (Hendrycks et al., 2021; Cobbe et al., 2021). Note that since the original SimpleZoo-RL data is relatively simple, medium- and hard-difficulty questions are largely missing, resulting in an overall easy:medium:hard ratio of 1:1:1 rather than the 7:2:1 used in Section [3.1.2](https://arxiv.org/html/2508.06165v3#S3.SS1.SSS2). Moreover, due to limitations of LLaMA-3.1-8B-Instruct, we substitute easy-difficulty questions for hard ones during training. To enhance open-domain QA performance, we include samples from the R1-Searcher dataset, which spans a broad range of questions derived from the training sets of 2Wiki and HotpotQA. For specialized domain knowledge, particularly in the medical field, we utilize multi-choice questions from MedQA, ensuring our model can handle domain-specific reasoning in healthcare contexts.

To further diversify our training data and extend coverage to humanities subjects, we generate synthetic questions in three additional domains: philosophy, history, and economics. These synthetic questions are created using Qwen-3-32B and follow the MMLU-Pro format to maintain consistency with established academic evaluation standards. Specifically, we use 5-shot prompting with MMLU-Pro development set examples to generate 10 questions with 4–10 options each. We discard format-non-compliant questions and observe the model’s tendency to generate simple questions with 4–5 options, so we request the model to produce additional options and increase the difficulty for each question. For quality control, we use GPT-4o-mini-2024-07-18 to evaluate each question’s correctness three times, discarding any question identified as incorrect in any evaluation. We then employ Qwen-2.5-7B-Instruct for difficulty assessment, finding approximately 80% of questions are easy-level. We randomly sample difficult questions as seeds for subsequent generations, using different seeds for each batch. Given that downstream test sets contain subject subdivisions (e.g., Economics encompasses microeconomics, macroeconomics, and econometrics), we utilize Qwen-3-32B to classify questions by subdomain, ensuring comprehensive coverage. We repeat this pipeline for 3–4 iterations to obtain the final training set.

Notably, our synthetic questions differ from MMLU-Pro in emphasizing multi-hop reasoning rather than specific knowledge points. This is evident of our results in Table [1](https://arxiv.org/html/2508.06165v3#S5.T1) where Vanilla RL shows limited improvement over CoT Baseline for Qwen-2.5-7B-Instruct and LLaMA-3.1-8B-Instruct (3.9% and $-0.1$% respectively), demonstrating no overfitting to the test set. Despite these characteristics, UR^2 consistently achieves improvements across models, validating our method’s effectiveness.

### C.4 About Fallback Fault in Retrieval Corpus Construction

When the policy model generates an invalid search query that triggers a fallback message from the LLM summarizer (i.e., This query requires design, computation, or complex reasoning, which exceeds the capabilities of a search engine. Please input another query or proceed with direct reasoning.), we observe that due to the use of retrieval masking, the model gradually learns to treat the content within <info>...</info> as informative for reasoning. As a result, when a fallback fault is encountered, the model tends to hallucinate. Therefore, we append the following visible message after </info> during training to mitigate this issue: It seems that this query exceeds the capabilities of the retrieval system. We may consider rephrasing it into a more fact-based and searchable question that does not require complex reasoning, or proceed with direct reasoning based on prior knowledge.

### C.5 Stage 1 Training Details

Due to the involvement of multiple models and tasks, Section [3.2.2](https://arxiv.org/html/2508.06165v3#S3.SS2.SSS2) only presents the stage-1 setup for Qwen-2.5-7B-Instruct on math and open-domain QA. Here, we elaborate on the initialization strategies for other models and tasks.

##### Math and Open-Domain QA.

We use the discarded math training samples with rollout accuracy below 0.2 as cold-start data. These harder examples naturally increase the likelihood of triggering retrieval. For Qwen-2.5-3B-Instruct, its limited capacity makes it more prone to Format violations when invoking retrieval. Since each violation incurs a $-1$ penalty, the original retrieval reward (+3 for one query, +4 for two or more) becomes insufficient to incentivize retrieval. To address this, we increase the retrieval rewards to +5 and +7, respectively. In contrast, LLaMA-3.1-8B-Instruct tends to retrieve for almost every question in early steps. To prevent over-reliance on retrieval and preserve reasoning ability, we remove the extra reward for multiple queries and assign a fixed +3 reward upon any retrieval activation.

##### MMLU-Pro and Medicine Tasks.

Unlike math tasks, MMLU-Pro and medicine tasks often require domain-specific knowledge, and retrieval is less likely to lead to fallback faults. For LLaMA-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, a weak reward signal is sufficient during early training: +0.5 for one valid retrieval and +1 for two or more. Unlike the original stage-1 design for math and open-domain QA, this version also incorporates answer rewards from the beginning, facilitating early alignment with task-specific correctness (i.e., no longer relying on cold-start data).In this variant, retrieval rewards are only applied during the first 10 training steps and then disabled.

For Qwen-2.5-7B-Instruct trained on math and open-domain QA, we adopt the stage-1 setup originally used for the MMLU-Pro and medicine tasks, corresponding to the weaker Stage-1 variant in Table [10](https://arxiv.org/html/2508.06165v3#A2.T10).

For Qwen-2.5-3B-Instruct, we extend Stage 1 to 15 steps. To encourage retrieval, outputs that do not invoke any retrieval call are penalized with a $-1$ Format Reward (non-accumulative).

### C.6 On Randomness and Reproducibility

RL training is known to exhibit inherent instability and variability across runs, often leading to divergent results even under identical settings (Nagarajan et al., 2018; Korkmaz, 2024). This randomness is attributed to factors such as stochastic policy updates, environment interactions, and non-deterministic hardware behavior. Despite these challenges, our experiments demonstrate remarkable stability. Thanks to the incorporation of Batch Normalization and Group Normalization in reward calculation, all models converge successfully in a single training run. The only exception is the UR^2 model of Qwen-2.5-7B on MCQ tasks, where the initial training unexpectedly resulted in zero retrieval activations for unknown reasons. Subsequent reruns corrected this behavior, highlighting the generally robust training process in our framework.

During evaluation and result aggregation, we employed a non-zero temperature setting to maintain controlled output diversity, thereby enhancing performance and mitigating the risk of repetitive generations. Due to the substantial API costs associated with GPT-4.1, conducting multiple evaluation runs to average results was not feasible. Nevertheless, given that the datasets contain approximately 500 samples—providing sufficient statistical power—we performed a targeted reproducibility assessment on HotpotQA using the UR^2 Qwen 7B-Instruct model. Specifically, three independent evaluation runs yielded F1 scores of 71.7, 71.9, and 71.2, respectively. These consistent results indicate that stochasticity exerts minimal influence on evaluation metrics and comparative model assessments. Furthermore, we conducted supplementary evaluations on all identified outlier cases across baseline and proposed methods to ensure the robustness of our findings.

### C.7 API Consumption

We measured the API usage cost of UR^2 Qwen-2.5-7B-Instruct on MCQ tasks and its w/o Stage-1 variant. Training 100 steps with UR^2 using GPT-4.1-mini cost approximately $320, while the w/o Stage-1 variant cost around $100. Additionally, summarization and testing on HotpotQA using GPT-4.1 for UR^2 Qwen-2.5-7B-Instruct cost about $20 per run. Since the training is a one-time expense, we consider the overall training-related consumption acceptable. Furthermore, experiments reported in Section [B.3](https://arxiv.org/html/2508.06165v3#A2.SS3) and Table [4](https://arxiv.org/html/2508.06165v3#S6.T4) show that substantial performance gains can be achieved without relying on closed-source models, suggesting that open-source models or less expensive APIs provide a viable alternative for achieving comparable improvements.

## Appendix D Prompts used in Experiments

### D.1 Prompts of LLM-as-a-Judge

### D.2 Prompts of Evaluation and Training

### D.3 Prompts for Summarizing

### D.4 Prompts for Baseline Methods

## Appendix E Case Study

### E.1 Search independently based on different difficulty levels and questions

Figure: Figure 4: UR^2 can search independently based on different difficulty levels and questions. Left:For simpler questions, UR^2 considers that external knowledge support is not necessary and does not perform a search. Right:For questions of a certain difficulty, UR^2 will actively search for answers.
Refer to caption: figs/select.jpg

### E.2 Planning

Figure: Figure 5: UR^2 can formulate plans and dynamically adjust them during the reasoning process. Left:UR^2 develops a plan and completes it in two steps. The first step is to search for the required knowledge, and the second step is to check each option individually. Right:UR^2 demonstrates clear planning behavior by decomposing the diagnostic task into sequential reasoning steps, identifying knowledge gaps, and querying external information to support its final decision.
Refer to caption: figs/plan_1.jpg

### E.3 Cross Validation

Figure: Figure 6: This case demonstrates UR^2 cross validation ability by verifying the same candidate answer ”The Ninth Gate” from multiple angles. It confirms the director and year, validates the actress’s participation, and contrasts international vs. North American commercial and critical success. The multi-step confirmation strengthens the confidence in the final answer, showcasing robust cross validation behavior.
Refer to caption: figs/cross_verification.jpg

### E.4 Reflection(Retry Retrieve/Refine Query)

Figure: Figure 7: After an initial analysis of the patient’s symptoms,UR² conducted two rounds of retrieval: first a broad query (”Which disease causes…”), then refined to a targeted question (”Which immunodeficiency causes…”) after recognizing the initial query was insufficient. This fully demonstrates the LLM’s capability to refine queries.
Refer to caption: figs/reflection_1.jpg

Figure: Figure 8: In this case, UR^2 initially retrieved general critiques related to ticking bomb scenarios but found no direct reference to Marcia Baron’s argument. Recognizing this gap, it engaged in Reflection by refining the query to focus on “specific unrealistic assumptions”, thereby exemplifying its capability to self-correct and deepen its information retrieval process.
Refer to caption: figs/reflection_2.jpg

Figure: Figure 9: UR^2 initially retrieved general information about Gran Colombia’s dissolution, but upon reflection recognized this was insufficient for precise answer selection. It then refined the query to investigate specific internal conflicts, ultimately identifying elite sectionalism as the key undermining factor, demonstrating effective use of Reflection through query refinement.
Refer to caption: figs/reflection_3.jpg

Figure: Figure 10: This case demonstrates UR^2 capacity to reflect and refine its query strategy when initial retrieval fails. By shifting from a direct search for Nathanson’s view to analyzing types of retributivism, the model successfully deduced the answer through indirect reasoning. This exemplifies its performance in reflection and retry-based retrieval refinement.
Refer to caption: figs/reflection_4.jpg

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see the weaknesses above.
soundness: 3
strengths: - The paper addresses a well-defined and significant problem: the suboptimal trade-off between retrieval and reasoning in current models . The core idea of a difficulty-aware curriculum is an intelligent and novel solution. It directly trains the model to learn when to trust its parametric knowledge versus when to seek external knowledge, a critical capability for building more efficient and robust agents.

- The two-stage optimization process is a key strength . Stage 1 provides a clever solution to the "cold start" problem often seen when training RL policies for tool use, by first rewarding the model for simply learning the tool-calling format. Decoupling this from Stage 2 (answer correctness) likely leads to a more stable training signal and better credit assignment.

- The experimental validation is exceptionally thorough and is a major strength of this work.
summary: This paper introduces $UR^{2}$, a general framework that unifies Retrieval-Augmented Generation (RAG) and reasoning through reinforcement learning to overcome the limited scope of prior RAG-RL methods. The framework's key innovations are a difficulty-aware curriculum that selectively invokes retrieval only for challenging problems and a hybrid knowledge access strategy that combines domain-specific offline corpora with LLM-generated summaries. Experiments across diverse tasks, including open-domain QA, MMLU-Pro, medical, and mathematical reasoning, show that $UR^{2}$ significantly outperforms existing RAG and RL methods on several benchmarks.
weaknesses: - As the authors acknowledge, the experiments are limited to 8B-parameter models. If scaling to larger models is too costly, I wonder whether smaller models can also benefit from the proposed method. Since using retrievers or tools with small models can help mitigate limited memorization capacity of the smaller models [1,2], it would be important to include experiments on smaller models.

- The curriculum design, while effective, introduces a significant one-time computational cost. To categorize questions by difficulty, the authors must "perform 20 rollouts using Qwen-2.5-7B-Instruct and compute the average performance score" for each training sample. This is a substantial preprocessing step that adds overhead before the main RL training can begin.

- The framework's robustness relies heavily on the "LLM-summarized retrieval corpus" to handle noise, as shown by the w/o LLM Summary ablation. This introduces a dependency on another powerful LLM (e.g., GPT-4.1) during the data preparation or inference pipeline, which adds complexity, latency, and cost that are not fully accounted for in the primary model's performance.


[1] Kang, Minki, et al. "Knowledge-augmented reasoning distillation for small language models in knowledge-intensive tasks." Advances in Neural Information Processing Systems 36 (2023): 48573-48602.


[2] Kang, Minki, Jongwon Jeong, and Jaewoong Cho. "T1: Tool-integrated self-verification for test-time compute scaling in small language models." arXiv preprint arXiv:2504.04718 (2025).

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 1
questions: - What exactly does the Vanilla RL baseline represent? Does it follow the same architecture as Search-R1 with different data, or is it a simplified UR^2 variant?
- Did you compare REINFORCE++ against GRPO or PPO under the same setup? If not, how can we be sure that the gains are not due to the RL algorithm itself?
- How much of the reported improvement remains if all methods are trained on the same corpus and datasets? In particular, does UR^2 still outperform Search-R1 when using the same Wikipedia-based retrieval corpus?
- How are the LLM-summarized corpora evaluated in terms of factual accuracy and hallucination?Any quantitative evidence that they indeed reduce retrieval noise would be valuable.
soundness: 1
strengths: - The idea of difficulty-aware curriculum and task-mixing strategy is interesting and intuitively appealing.
- The paper presents comprehensive evaluations across multiple domains (math, medicine, open QA).
summary: This paper proposes UR^2, a reinforcement learning (RL) framework that unifies retrieval-augmented generation (RAG) and reasoning. The authors point out two limitations in prior RAG-RL frameworks (e.g., *Search-R1*): (1) existing models fail to find an optimal trade-off between retrieval and reasoning, often overemphasizing one component; (2) retrieval introduces significant noise, which can degrade chain-of-thought reasoning quality.

To address these issues, the paper introduces two main components: (1) a hybrid retrieval corpus that combines domain-specific offline corpora with LLM-generated summaries, and (2) a difficulty-aware curriculum, where retrieval is invoked only for hard instances while easy ones rely on internal reasoning.

Through experiments on various benchmarks (MMLU-Pro, MedQA, MATH, and open-domain QA), the authors report that UR^2 improves performance compared to existing RAG-RL methods and achieves comparable results to GPT-4o-mini with 7B-scale models.
weaknesses: - Fairness and comparability of evaluation. The proposed method introduces an augmented knowledge base (LLM-summarized and task-specific corpora) as a key component. However, previous RAG-RL methods (e.g., Search-R1, R1-Searcher) are not restricted to specific corpora and could also utilize similar augmented data. Therefore, evaluating UR^2 solely on newly curated corpora while prior methods rely on static Wikipedia datasets makes the comparison unfair. If the augmentation of the corpus itself is a core contribution, the paper should explicitly analyze the effect of this augmentation (e.g., via retrieval corpus ablation), similar to prior work such as [1].
- Limited novelty and weak justification of the method. The overall training pipeline remains very close to the Search-R1 style frameworks, merely adding a two-stage optimization process. However, the paper does not convincingly explain how this design resolves the original issue mentioned in the introduction (the imbalance between retrieval and reasoning). The conceptual connection between the two-stage RL setup and this problem statement is unclear.
- Unclear source of performance gain. The improvement might stem from broader training dataset coverage or the use of enhanced corpora rather than from the proposed algorithm itself. Because the training data distribution and retrieval sources differ across baselines, it is hard to isolate whether gains come from the method or from data and corpus augmentation. A controlled comparison under identical conditions would strengthen the claim.
- Lack of clarity and reproducibility in RL setup. The term “Vanilla RL” baseline is ambiguous. It's unclear whether it corresponds to a GRPO-based single-stage model (as in Search-R1) or a simplified UR^2 variant. Moreover, the use of REINFORCE++ instead of PPO or GRPO (common in previous works) is not justified or compared. Without such clarification, it is difficult to determine whether improvements arise from the proposed method design or simply from a change in the RL algorithm.

[1] Lyu et al., Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: Please see Weaknesses above.
soundness: 1
strengths: * The focus on bridging knowledge grounding with RAG and reasoning is timely and meaningful.
* The approach to optimize retrieval and answer generation through reinforcement learning is reasonable. 
* The proposed framework achieves superior performance over existing RAG and RL baselines.
summary: This paper introduces the framework that unifies retrieval-augmented generation (RAG) and reasoning through reinforcement learning (RL), with the motivation that knowledge grounding and reasoning have been developed in isolation from previous works. Specifically, the proposed framework includes (i) a difficulty-aware curriculum that dynamically decides when to invoke retrieval (encouraging models to rely on internal reasoning for easy problems and use retrieval only for harder ones), and (ii) a hybrid knowledge access strategy that combines domain-specific offline corpora with LLM-generated summaries. Also, the proposed framework is trained via a two-stage RL process that separately optimizes retrieval capability and answer quality. Experiments on diverse open-domain QA and reasoning tasks show that it outperforms existing RAG and RL baselines.
weaknesses: * While one of the core focuses of this work is on reasoning, it is questionable whether the model truly performs reasoning (like DeepSeek-R1 style models), or if the improvements mainly stem from retrieval and answer optimization without reasoning. Also, for reasoning, instead of using pure instruction fine-tuned models (like LLaMA or Qwen), it is questionable whether the reasoning (that the authors aim to achieve) can be performed when simply using more advanced reasoning models (e.g., DeepSeek-R1). Overall, the reasoning aspect feels underdeveloped despite the explicit focus in this work.
* It is unclear how the LLM-generated summaries are constructed, as well as their scale. Additionally, it is also questionable whether they are of high quality and not vulnerable to hallucination. 
* The idea of invoking retrieval only for difficult queries is similar to Self-RAG and Adaptive-RAG [A, B]. The discussions and (probably) experimental comparisons with them are required. Also, it would be great if the authors could perform an efficiency analysis on it (i.e., whether the overall pipeline becomes efficient when invoking retrieval if necessary).
* Reward design, which is a manually designed combination of format, retrieval (or generation), and fallback rewards, seems somewhat ad hoc. More analysis on it would be useful.
* The difference between the proposed approach and the Vanilla RL baseline is unclear. 
* The performance gains on the Match domain are marginal, which may warrant further discussion.

---

[A] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

[B] Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: 1. UR2 employs difficulty-aware curriculum training to selectively invoke retrieval. Could this strategy negatively impact the model’s intrinsic reasoning ability in certain domains?

2. Since difficulty classification requires 20 rollouts, how scalable and efficient is this approach for larger datasets?

3. UR2 relies on LLM-summarized retrieval corpora, which may not capture the complexity of dynamic, real-world environments. How well would UR2 generalize to online or rapidly changing corpora, and are there strategies to improve its robustness in such settings?

4. The paper claims that LLM-generated summaries reduce retrieval noise and hallucination. Could you provide clearer evidence or analysis for this claim, rather than inferring it indirectly from performance drops in the w/o-summary setting?

5. UR2 uses reinforcement learning to coordinate retrieval and reasoning, which introduces additional training cost and occasional instability. Could simpler decision mechanisms (e.g., heuristics or lightweight classifiers) offer similar performance? If not, what makes RL uniquely effective in this setting?
soundness: 2
strengths: 1. This work makes a clear conceptual contribution by unifying retrieval and reasoning through reinforcement learning, bridging two previously isolated paradigms (RAG and RLVR). Unlike earlier approaches that are limited to fixed retrieval settings, UR2 introduces difficulty-aware curriculum training, enabling dynamic and efficient retrieval invocation.

2. The paper provides strong empirical evidence supporting its contributions. The proposed approach is evaluated across four distinct domains (open-domain QA, MMLU-Pro, medical QA, and mathematical reasoning) and multiple model scales (3B–8B), consistently outperforming strong RAG-RL baselines such as Search-R1 and R1-Searcher.
summary: This paper proposes UR2 (Unified RAG and Reasoning), a reinforcement learning framework that integrates retrieval and reasoning in large language models. UR2 employs difficulty-aware curriculum training to selectively invoke retrieval for challenging problems and adopts a hybrid knowledge access strategy that combines domain-specific offline corpora with LLM-generated summaries. These components enable dynamic coordination between retrieval and reasoning, improving adaptability across diverse domains. Experiments on open-domain QA, MMLU-Pro, medical, and mathematical reasoning tasks show that UR2 outperforms existing RAG-RL baselines.
weaknesses: 1. The proposed framework relies heavily on LLM-summarized corpora rather than raw or dynamically retrieved web content. While this design helps reduce noise and stabilize training, it may also limit the model’s ability to handle the complexity, variability, and unpredictability of real-world retrieval scenarios. This reliance raises concerns about generalization and robustness beyond curated or offline corpora, particularly in open-domain or rapidly evolving knowledge settings.

2. Although the framework shows strong performance with models up to 8B parameters, it has not been evaluated at larger scales. This leaves open the question of how well UR2 would scale to more powerful models (e.g., 30B+ parameters) and whether the observed performance gains would persist in such settings.

3. The reinforcement learning training shows occasional instability and required reruns to correct failed retrieval activation in some settings. This indicates potential reproducibility issues and sensitivity to initialization or hyperparameters.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. The "Task Mixing Strategy"  creates a significant train-test discrepancy. During training, the framework manually feeds the model different prompts based on external difficulty labels (e.g., pure reasoning for "easy" math, RAG for "hard" math). However, during evaluation, a single, unified prompt is used for all math problems , and the external difficulty is absent. Why was this manually-partitioned curriculum chosen? Would it not be a more robust and end-to-end solution to use the same unified prompt (from testing) during training and let the RL learn this adaptive retrieval policy on its own?
2. The paper's Stage 1 functions as a "warm-up" to teach the model retrieval syntax. Why was RL chosen for this step? In the literature, this type of tool-use is commonly achieved via SFT, which is generally considered more controllable, stable, and lightweight than RL. Could the authors justify their choice of using RL for this warm-up phase?
soundness: 2
strengths: The research problem this paper tries to addresses (unifying RAG with reasoning through RLVR) is both important and timely. The paper's primary strength then lies in its extensive and rigorous experimentation in tackling this challenge. The authors are commended for validating their framework across four diverse and challenging domains (Math, Medical, QA, and MMLU-Pro) and comparing it against a comprehensive suite of state-of-the-art RAG and RL baselines . Furthermore, the work is clear and appears highly reproducible. The authors provide commendable transparency by detailing the exact training configurations and prompts in the appendices , which is a significant contribution to the community.
summary: This paper introduces $UR^2$, a framework designed to unify Retrieval-Augmented Generation (RAG) and COT reasoning across diverse domains like math and medicine using RL. It introduces two primary contributions. On one hand, it achieves stable and efficient training by employing a two-stage optimization process and a difficulty-aware task mixing strategy. On the other hand, it explicitly reduces retrieval noise by using a separate, powerful LLM to pre-process raw documents into clean, concise summaries before they are presented to the policy model.
weaknesses: 1. The paper's methodological contribution appears limited. The primary contributions—a two-stage "warm-up" optimization , a "difficulty-aware" data mixing strategy , and the use of an external LLM to summarize retrieved information —are arguably a collection of well-established engineering techniques. While empirically effective, these components may not represent a significant methodological innovation that meets the high standard for publication at ICLR.
2. This concern about novelty is amplified by the framework's heavy dependence on the LLM summarization module. The paper's own ablation study demonstrates this component is critical. This suggests that the $UR^2$ itself is not learning to handle retrieval noise; rather, this core challenge is outsourced to a separate external model. This reliance diminishes the self-contained contribution and robustness of the proposed framework.
3. When I going through the paper, I observed several typos. 
- We train UR2 using REINFORCE++ (Guo et al., 2025a), a streamlined vari­ant of PPO tailored.
- Trainning Data Selection