HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation
Authors: Peilin Wu
Mian Zhang
Kun Wan
Wentian Zhao
Kaiyu He
Xinya Du
Zhiyu Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Retrieval Augmented Generation
LLM Agent
Agentic RAG
Question Answering
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: This work introduces HiPRAG, a RL training method that uses hierarchical process rewards to teach agentic RAG systems to search more efficiently by reducing sub-optimal behaviors like over-search and under-search.

Abstract: Agentic Retrieval-Augmented Generation (RAG) is a powerful technique for incorporating external information that Large Language Models (LLMs) lack, enabling better problem solving and question answering. However, suboptimal search behaviors exist widely, such as over-search (retrieving information already known) and under-search (failing to search when necessary), which leads to unnecessary overhead and unreliable outputs. Current training methods, which typically rely on outcome-based rewards in a Reinforcement Learning (RL) framework, lack the fine-grained control needed to address these inefficiencies. To overcome this, we introduce $\textbf{Hi}$erarchical $\textbf{P}$rocess Rewards for Efficient agentic $\textbf{RAG}$ (HiPRAG), a novel training methodology that incorporates a fine-grained, knowledge-grounded process reward into the RL training. Our approach evaluates the necessity of each search decision on-the-fly by decomposing the agent's reasoning trajectory into discrete, parsable steps. We then apply a hierarchical reward function that provides an additional bonus based on the proportion of optimal search and non-search steps, on top of commonly used outcome and format rewards. Experiments on the Qwen2.5 and Llama-3.2 models across seven diverse QA benchmarks show that our method achieves average accuracies of 65.4\% (3B) and 67.2\% (7B), outperforming strong agentic RAG baselines. This is accomplished while dramatically improving search efficiency, reducing the over-search rate from over 27\% in baselines from previous work to just 2.3\% and concurrently lowering the under-search rate. These results demonstrate the efficacy of optimizing the reasoning process itself, not just the final outcome. Further experiments and analysis demonstrate that HiPRAG shows good generalizability across a wide range of RL algorithms, model families, sizes, and types. This work demonstrates the importance and potential of fine-grained control through RL, for improving the efficiency and optimality of reasoning for search agents.

Published as a conference paper at ICLR 2026
HIPRAG: HIERARCHICAL PROCESS REWARDS FOR EFFICIENT AGENTIC RETRIEVAL AUGMENTED GEN- ERATION
Peilin Wu', Mian Zhang', Kun Wan', Wentian Zhao², Kaiyu Hc', Xinya Du', Zhiyu Chen' 'The University of Texas Dallas, 'Adobe Inc. {peilin. wu, zhiyu. datenaesst
ABSTRACT
Agentic Autrivall-Augmented Generation (RAG) is a powerful technique for in- corporating external information that Large Language Models (LLMs) lack, en- abling better problem solving and question answering. However, suboptimal search behaviors exist widely, such as over-search (retrieving information already known) and under-search (failing to search when necessary), which leads to un- necessary overhead and unreliable outputs. Current training methods, which typi- cally rely on outcome-based rewards in a Reinforcement Learning (RL) frame- work, lack the fine-grained control needed to address these inefficiencies. To overcome this, we introduce Hierarchical Process Rewards for Efficient agentic RAG (HiPRAG), a novel training methodology that incorporates a fine-grained, knowledge- showledge-roounded process reward into the RL training. Our approach evalu- ates the necessity of each search decision on-the-fly by decomposing the agent's s reasoning trajectory into discrete, parsable steps. We then apply hierarchical reward function that provides an additional bonus based on the proportion of op- timal search and non-search steps, on top commonly used outcome and format rewards. Experiments on the Qwen2.5 and Llama-3.2 models across seven diverse QA benchmarks show that our method achieves average accuracies of 65.4% (3B) and 67.2% (7B), outperforming strong agentic RAG baselines. This is accom- plished while dramatically improving search efficiency, reducing the over-search rate from over 27% in baselines from previous work to just 2.3 % and concurrently lowering the under-search rate. These results demonstrate the efficacy of optimiz- ing the reasoning process itself, not just the final outcome. Further experiments and analysis demonstrate that HiPRAG shows good generalizability across a wide range of RI algorithms, model families, sizes, and types. This work demonstrates the importance and potential of fine-grained control through RI., for improving the efficiency and optimality of reasoning for search agents.
INTRODUCTION
Large Language Models (LLMs) augmented with retrieval have rapidly evolved into agentic RAG systems that autonomously issue search queries, incorporate external knowledge, and perform multi-step reasoning (Singh et al. 2025) Li et al. 2025a; Zhang et al. 2025). In particular, recent frameworks integrate reinforcement learning (RL) to empower LLMss with the ability decide when and what to retrieve during step-by-stey reasoning (Jin et al.) 2025b Song et al., 2025a Chen et al.) 2025). However, along with this potential comes a critical drawback: today's agentic RAG agents often exhibit two suboptimal search behaviors: over-search, where the agent issues unnecessary or redundant retrievals (Wu et al. 2025 Qian et al., (0256), and under-search, where fails retrieve external knowledge when it is actually needed, which undermine their accuracy and efficiency (Wu) al., 2025 Shen al. 2024). These observations highlight that simply pairing L.LMs with a search tool is not enough; the manner in which the agent uses the search tool must be optimized.
We have released code and model wtttt::://///t www.gghtub.... cow lddal /217/Hi PRAG

---

Published as a conference paper at ICLR 2026
search Check
Step
Under zearch Check
nessonin
Sear Eh auary
Madel
ruosoning
Factual cornect? Logic dameet?
Format Reward
concustoo
cnnciusion
conclusion
Answer Reward
Samantic Equivalen??
Fermet
Assae"
Step
w/ search
w/ search
search
Rellout
Medel
Step
Step
Step
Step
Answer
Fornat Reward
Format
Araver Check
Reward
Fornat
Format X
Fornot Answer
Ansaer
Answer
/
Figure 1: A general overview of the HiPRAG training workflow. The policy model generates a multi-step reasoning trajectory, and each step is evaluated on-the-fly detect suboptimal search behaviors. A final hierarchical reward is then computed by combining a process bonus for step optimality with rewards for the final answer's correctness and proper formatting.
Recent research has turned RL signals tune the search behavior of agentic RAG system or search agents. One type of work proposed length retrieval times-based penalties to encourage shorter reasoning trajectory (Song al., 2025b; Wang et al.) (025a).. Such heuristics can reduce redundant steps but risk oversimplifying the problem: the model may learn avoid searches alto- gether even when searches are necessary (Wang et al.) 2025a), therefore exacerbating under-scarch.
Other researchers have incorporated model confidence (Wu et al. 2025) and knowledge awareness
(Huang et al. 2025b [un et al. 2025) into the reward. These advances demonstrate the promise of process-level reward shaping in RAG. Yet, important limitations remain: confidence thresholds and knowledge classifiers are imperfect proxies that can misjudge when to search, and learned process reward models may introduce bias or only weakly align with true step quality. Crucially, none of these methods gives the agent explicit, specific feedback on each retrieval decision: whether a particular search superfluous or a missing search should have been taken is not enforced in fine-grained way.
In this work, we introduce HiPRAG, a Hierarchical Process reward framework for agentic RAG, to address the above challenges. Instead of evaluating only by the final outcome or coarse prox- ies, HiPRAG explicitly parses the agent's s reasoning trajectory into structured steps and constructs reward multiple levels to optimize both correctness and efficiency. First, the agent is guided to produce a well-structured reasoning trajectory that is parsable by the rule-based method, with a correct final answer. On top of that, we design a process reward function that provides bonus ward that increases linearly as the ratio of optimal steps (not over-search or under-search) of the trajectory increases, accompanied by set of fast and robust methods for detecting over-search and under-search for each step in the reasoning trajectory on-the-fly during training, achieved by di- rectly prompting an external LLM for verification. These step-level bonus rewards are gated by the format and final answer's correctness to ensure that the agent is not over-punished for taking suboptimal steps if correct format or answer presented, which harms the reasoning ability. By hierarchically constructing the reward for format and outcome correctness as well process quality, HiPRAG provides more fine -rained and accurate training signal than prior methods while avoiding the drawbacks of them.
We conduct experiments on seven question-answering benchmarks covering knowledge-intents single and multi-hop queries. HiPRAG delivers significant gains in both accuracy and retrieval ef- ficiency across diverse evaluations. Empirically, HiPRAG achieves average accuracies of 65.4% (3B) and 67.2% (7B), computed with Cover Exact Match, significantly outperforming strong agen- tie RAG baselines. More importantly, achieves unprecedented gains efficiency, reducing the over-search rate from over 27% in baselines from previous work (Wu et al. 2025) to just 2.3%, while concurrently lowering the under-search rate to as low as 29.0%. Our method shows good generalizability across different model families (Qwen2.5 and Llama-3.2) 3.2), RL algorithms (PPO and
2

---

Published as a conference paper at ICLR 2026
GRPO), model sizes (3B and 7B), and model types (base and instruct). Our major contributions as follows:
We propose HiPRAG, novel Reinforcement Learning training methodology that uses hierarchical, knowledge-aware process reward to provide fine-grained supervision on the search behavior of agentic RAG systems.
We introduce efficient and direct method for detecting over-search and under- search behaviors on-the-fly during training, accompanied with a set of output format parsable by rule-based methods, enabling the effective application of our process-based reward.
We empirically demonstrate that our approach enhances both the accuracy and efficiency of various L.MS and RI algorithms, across multiple question-answert benchmarks, show- ing its strong generalizability.
2 RELATED WORK
Agentic RAG & Tool Use The integration of external knowledge into LLMs has evolved into more dynamic, agentic systems. Frameworks like ReAct (Yao al. 2023) demonstrated that LLMs can synergize reasoning and acting, paving the way for agents that can autonomously decide when and what to retrieve. This led to a new class of agentic RAG systems that interleave retrieval with multi- step reasoning, often structured Chain-of-BR -fttreriveeaalll (Wang et al., 2025b) and DeepRAG (Guan et al. 2025) further refined this by structuring the retrieval process itself into sequential steps handle complex queries. To improve the decision-male maaiing capabilities
chain-of-thoughh process (Trivedi et al. 2023). Variations like
these agents, recent work has increasingly turned to RL (Singh et al. 2025 Li et al. 2025a Zhang al. 2025). RL frameworks train agents to develop optimal policies for invoking tools, including search engines. For instance, [Huang et al.] (2025a) use RL and curriculum learning enhance RAG. This aligns with a broader trend of using RL improve general tool use in LLMs. Works like ToolRL (Qian et al. (025a) and ToRL (Li et al. 2025ch have shown that RL from rewards based on task success can significantly scale and improve the tool-integration capabilities of LLMs. Our work builds on this foundation, applying RL not just to achieve a correct final outcome but to optimize the efficiency of the retrieval process itself.
Efficient Agentic RAG & Tool Use While agentic RAG enhances reasoning, it often introduces inefficiency, such as redundant or unnecessary tool calls. A key research direction has been to make retrieval adaptive, triggering it only when the model's internal knowledge is insufficient. Early approaches relied on heuristics or classifiers to detect uncertainty and determine the need for retrieval (Mallen et al. 2023 Dhole 2025). More sophisticated methods learn to assess the LLM's real-time information needs or self-awareness from its internal states to make dynamic retrieval decisions (Su et al., 2024 Zubkova et al. 2025 Yao al., 2025 Huanshuu et al., 2025). Concurrently, RI. has emerged as powerful tool for optimizing the efficiency of tool use. In the context of RAG, RL has been used create search-efficient models (Sha et al. 2025) and to mitigate suboptimal search behaviors by reducing uncertainty (Wu et al. 2025). Works like R1 R1-Searcher++ (Song et al., 20256) and other synergistic reasoning agents (Huang et al., 2025b) use RL to incentivize dynamic and necessary knowledge acquisition. ReARTeR (Sun et al. (2255) introduces framework with trustworthy process reward model to score and refine each step in RAG pipeline. This mirrors efforts in the broader tool-use domain to mitigate tool overuse. For example, SMART (Qian et al.) 2025b), SMARTCAL (Shen et al., 2024), and OTC (Wang et al. 2025a) train agents to be self- aware and make optimal tool calls, often using RL. Yue et al. (2025) and Ye et al. (2025) also use verifiable stepwise rewards to promote more efficient general reasoning paths. While these methods improve efficiency, they often rely proxies like model confidence separately trained reward models. HiPRAG differs by introducing direct, on-the-fly evaluation of each search step's necessity, providing more explicit training signal for efficiency.
HIPRAG
Our approach, which we term HiPRAG, introduces fine-grained process-based reward mechanism into the reinforcement learning loop for training agentic RAG systems. This is achieved through three key aspects: (1) A redesigned, explicitly structured output format that enables rule-based
3

---

Published as a conference paper at ICLR 2026
parsing of reasoning steps described in Section 3.1 (2) An efficient method for detecting over-
search and under-search behaviors described in Section 3.2] and (3) A hierarchical reward function
that dynamically prioritizes correctness and search efficiency described in Section 3.3 An overview
of the HiPRAG workflow can be found in Figure 1
3.1 DECOMPOSING REASONING TRAJECTORY INTO PARSABLE STEPS
A primary obstacle implementing process rewards in agentic RAG is the difficulty of parsing an agent's 's reasoning trajectory, which includes one or more reasoning steps. For each step, the agentic RAG system resolves one sub-query by either using search tools or using its own parametric knowl- edge. Existing frameworks like Search-RI (Jin et al. 2025b) often generate reasoning within a series of <think> XML blocks, interleaved with other blocks about search queries and retrieved informa- tion. This format, though maintaining the fluency of reasoning. makes it hard to isolate and evaluate individual steps of reasoning for the following two reasons: (1) Ambiguous Step Boundaries: A single <think> block does not map to discrete reasoning step. It often mixes the conclusion from a previous action with the reasoning and planning for the current action, making it difficult to programmatically isolate a self-contained step. (2) Implicit Internal Reasoning: Non-search steps, where the agent relies on its parametric knowledge, are not explicitly tagged. They are embedded prose within the <think> block, making it challenging to differentiate them from the analyti- cal text that precedes a search query without extra natural language understanding. Therefore, this format makes it infeasible to isolate and evaluate individual steps of reasoning during RL training, as it would require slow and expensive calls to a powerful LLM for post-hoe interpretation, which significantly slows down the RL training process and introduces high error rate.
To overcome this, we enforce a structured, machine-parsable output format during RL training. We modify the agent's prompt and rollout logic to generate its entire reasoning trajectory within single <think> block, which in turn contains a sequence of discrete <step> blocks. Each step can either be search step or a non-search step, distinguished by the presence of a <search> block containing search queries and <context> block containing retrieved information. Formally, complete reasoning trajectory T for a given question is sequence of n steps with a final answer
T {81, (2,,82; Sn; a). Each step 8i. for E [1,n] can either be a search step, which is tuple R = (Ti, qi, C, o.) if the model chooses to search, or be a non-search step, which a tuple s NN -rr,, o.). where r; is the model's 's reasoning block containing the planning and analysis within this step. 9: is the search query, C; is the retrieved context, o, is the conclusion or summary of the knowledge gained the step. An example of a pipeline that does the inference for this format is shown in Algorithm [ the equivalent reasoning trajectory is included in Figure
An example comparing format before and after for the same question with
We ensure adherence to this schema through two approaches in parallel. First, the agent's system prompt is updated with explicit instructions and few-shot examples demonstrating the correct usage of all the XML tags. Figure shows the prompt with parsable output format. Second, as detailed in Section 3.3] our RL framework applies a positive reward for correct outputs, thus incentivizing the model to consistently produce parsable trajectories.
3.2 ON-THE-FLY DETECTION OF SUBOPTIMAL SEARCHES
With trajectories segmented into discrete steps, we can implement efficient checks for over-search and under-search during the RI. training phase.
Over-search Detection. Previous methods for detecting over-search involved complex re- generation pipelines, where the search context was removed and a fixed instruction was appended to prompt the model to rely on its internal knowledge (Wu et al. 2025). This approach is brittle, as the appended instruction can conflict with the agent's original reasoning flow and produce unnatural, low-quality outputs. It also computationally expensive to re-generate if the search appears at the end of long reasoning trajectory. We propose more direct and robust method. For each search step S² = (ri, qs, C; o,), we take its search query qi and directly prompt the policy model with standalone question. We then obtain re- generatee answer, of. An external LLM judge is used to assess the semantic equivalence of the original step's 's conclusion o, and the re- re-generated answer of- The prompt for external L.LM judge is shown in Figure 5 If they are equivalent, the search was
4

---

Published as a conference paper at ICLR 2026
redundant, and the step is flagged over-search. This method is not only faster but also provides a more reliable signal by isolating the core knowledge required by the query.
Under-search Detection. For each non-search step SN (r;, o;), we verify the factual and logical accuracy of its reasoning r; and conclusion o; by prompting external verifier model to assess the correctness of r; and o;- The prompt for external verifier model is shown in Figure 6 If the content is found be incorrect, the step is flagged as under-search, the agent failed to utilize the search tool to retrieve necessary information, leading to a hallucination or factual error. Some under-search cases may appear as suboptimal or incomplete but locally correct intermediate steps. We intentionally avoid penalizing these cases because step completeness often depends on the agent's global multi-step plan and is subjective to determine at the step level without expensive counterfactual supervision. Penalizing porrec-buttincomplet steps tends to push agents toward over-search by discouraging reliance on parametric knowledge.
In actual implementation, both of the detection methods can work concurrently to improve the de- tection speed. For conducting over-search detection over batch of data during RL rollout phase, the re-generation step can be executed separately through batch generation before using the external LLM judge to further improve the training speed.
3.3
HIERARCHICAL PROCESS REWARD CALCULATION
A naive length confidence-based reward function that penalizes search can over-suppress the agent's retrieval capabilities, leading to poor performance on knowledge-intensive tasks. Our goal incentivize optimal search behavior to improve performance and efficiency, while maintaining the basic ability of reasoning with a search tool. In order to achieve this, the rewards need to be dynamically focused on incentivizing format and final answer correctness at the early stage of the RL training. while shifting its focus towards reasoning efficiency and optimality after the basic search ability has been established, by providing a higher reward be differentiated from the original outcome and format reward. To this end, we design a hierarchical reward function that prioritizes correctness and format adherence before rewarding process optimality, while shift its focus towards process optimality after the reasoning ability has established, top of the outcome format reward version of Search-R. R. reward (Jin et al. 2025a).
Hierarchical process reward. Let A(T) E (0, indicate the final answer a of the trajectory
T's correctness (here indicate that the trajectory follows the required format (an example of the implementation of F(T) be found in Algorithm and and 3p. Let N(T) be the number of steps in the trajectory T, with the number of optimal (neither over-search nor under-search) steps Nowr (T) be
Cover Exact Match introduced in Section 4.11. and F(T) {0. 1)
Nowr (T)
s (T) Over )))+ +
E (T) N
-Under(s
where Over(-) and Under(.) are the detectors from Section 3.2 With a format weight Ny E [0.1] and process bonus coefficient Ap 2 O, we define single merged reward:
R(T) A(T)(1 21) + Xy F(T) Ap ((T) Nonr(T) N(T)
(1)
This expression is algebraically equivalent to the standard outcome format reward used in prior work when =0= and it adds gated process bonus only when both the answer and the format are correct. In particular, the reward becomes R(T) 1++ Xp
Non(T) whenever ((T)= A
N(T)
collection of all the symbols used can be found in Table 3
This hierarchical structure ensures that the agent is first incentivized to produce well-formed rea- soning trajectory with correct answers. Only once it achieves this primary goal does it receive an additional reward bonus for the efficiency and validity of its reasoning path, which avoids the pitfalls of over- sup-suupression while directly encouraging the model to develop a more nuanced understanding of its own knowledge boundaries.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENT SETUP
This section details the experimental framework used to evaluate HiPRAG. We outline the datasets, evaluation metrics, models, and training procedures to ensure reproducibility and provide clear context for our results.
4.1 DATASETS& METRIC
Our experimental data is composed of wide coverage of both single and multi-hop Question An- swering (QA) samples, modeled after the one used in Search-R1 to ensure a fair and direct com- parison with prior work. The training set is combination of the official training sets from NQ Kwiatkowski et al.) 2019) and HotpotQA (Yang et al. 2018). This creates a diverse corpus for teaching the agent both single-fact retrieval and multi-hop reasoning, which is crucial for learn- ing efficient reasoning. To assess both in-domain and out-of-domain generalization, we evaluate our models on comprehensive test set composed from the development or test set of seven QA datasets: NQ. PopQA (Mallen et al., 2023). TriviaQA (Joshi et al. 2017). 2WikiMultiHopOA (Ho al. 2020), Bamboogle (Press et al.) 2023). , HotpotQA, and MuSiQue (Trivedi et al. 2022).
The primary metric for our evaluation is Cover Exact Match (CEM) (Song et al. 2025a). This metric determines correctness by checking the ground-truth answer string is present the model's generated answer. We choose CEM over strict Exact Match because modern LLMs are optimized for generating longer, explanatory responses. Imposing a strict match would penalize valid answers embedded in more verbose text and might not accurately reflect the model's capabilities. We also assess the model's efficiency by measuring the Over-search Rate (OSR) and Under-search Rate (USR), which are defined as the ratio of over-search and under-search steps among all identifiable search steps and non-search steps, within a set of test samples. Given a set of samples D Diest with their reasoning trajectory T, the OSR and USR can be caleulated according Equation 2
ΣTEDIO 1 E TT: Over(s
USR ΣTED H E T (nder
OSR
(2)
ΣTED, HR E TTH
ΣTEDu KNNR E TTH
4.2 BASELINES
We compare our method against comprehensive set baselines that represent different paradigms in strieval-augmennee generation: (1) Direct Inference: Direct generation without any retrieval mechanism. (2) Standard RAG: conventional RAG setup where retrieval performed once
based on the initial query (Lewis al. 2220).. (3) Prompt-Based Agentic RAG: Methods that rely on sophisticated prompting to achieve multi-step reasoning and search, including IRCoT (Trivedi al. 2023) and Search-ol (Li al. 2025b). (4) RL-Based Agentic RAG: State-of-the-arr methods that use reinforcement learning to train search agents, including Search-R1 (Tin et al. (0256)) R1- Searcher (Song et al. 2025a RI-Searcher++ (Song et al. 2025b), and 8-GRPO (Wu et al. (2025). Among these, R1-Searcher++ and 3-GRPO are explicitly designed to improve search efficiency, making them strong baselines focused on efficiency.
4.3 TRAINING DETAILS
All RL-based models were trained with four NVIDIA A100 80GB GPUs. The training process was conducted for total of 400 steps and saved the checkpoint every 50 steps. For evaluation, we adopted the following checkpointing strategy: if the training process completed without instability, the final saved checkpoint is used for testing. However, if the training reward collapsed, we use the last stable checkpoint saved before the collapse to ensure fair evaluation of the model's 's best-learned state.
Our experiments leverage several models for different roles within the framework. The main exper- iments are conducted using the Qwen2.5-(3B/7B) Insttruct models (Qwen et al. 2025). To analyze performance across different model families and types, also conduct experiments with Llama- (Grattafiori et al. 2024) and Qwen2.5-3B. For the detection of suboptimal searches during training and evaluation, utilize small-sized proprietary models that provide fast inference speed while maintaining sufficient performance. Over-search detection is performed by GPT-4.1 mini (OpenAl 2025a), while under-search detection relies on GPT-5 mini (OpenAl 2025b).

---

Published as a conference paper at ICLR 2026
General QA TriviaQA
Multi-Hop QA
Method
Avg.
NQ
PopQA
HotpotQA
2Wiki
MuSiQue
Bamboogle
Direct Inference
27.0
26.8
40.1
58.7
16.0
7.9
15.9
31.8
Standard RAG
51.2
54.7
65.7
56.9
18.6
45.3
IRCoT
27.5
36.0
42.5
51.4
37.6
19.4
20.6
36.4
Search-ol
40.2
42.2
58.5
56.1
45.6
15.1
19.3
43.9
R1-Searcher
60.0
73.0
58.2
60.4
60.3
32,9
55.8
60.6
R1 R1-Searcher++
61.0
73.5
59.0
64.2
63.2
32.3
58.7
62.1
Bearch-R1
61.2
73.6
56.5
54.0
63.6
24.8
48.4
60.3
Search-RI-ste Ri-step
62.4
74.4
57.3
54.8
25.3
49.6
61.2
3-GRPO
65.0
75.0
60.0
53.0
66.0
24.0
52.0
62.5
3-GRPO-step"
62,4
73.9
61.3
52.1
66.0
22.8
54.4
62.1
HiPRAG-3B
68.7
75.5
66.3
57.4
67.4
24.1
41.6
65.4
HiPRAG-7B 71.2
76.3
63.2
62.4
71.7
34.1
52.8
67.2
Table 1: Main results of CEM (higher is better) in percentage seven QA benchmarks. Best and second-best overall averages are marked in bold and underline. Search-R1-step refers to the model trained with Search-RI v0.3's output format reward with HiPRAG's output format. 3-GRPO-step' refers to the model trained with 3-GRPO's reward with HiPRAG's output format. The HiPRAG-3B and 7B here refer to the model with highest Avg. CEM score among all trained HiPRAG models.
For our main experiments, we use Proximal Policy Optimization (PPO) (Schulman et al. 2017 as the core RL algorithm. PPO selected for its demonstrated training stability in complex LLM fine- tuning scenarios, especially on the previous works on search agent development using RL. To assess the impact the RI. algorithm choice, we also perform experiments using Group Relative Policy Optimization (GRPO) (Shao et al. 2024). For GRPO we keep the same training parameters as the PPO training with a group size
For our retrieval environment of all experiments, we follow the Search-RI R1 setup and use the 2018 Wikipedia dump Karpukhin et al. 2020) as the knowledge source, with E5-base (Wang et al. 2024) serving as the retriever. In each search step, the top-3 relevant passages are returned.
In terms the inference parameters, we set the temperature and top p to during the rollout stage of RL training to make sure high possibility of generating desired reasoning trajectory. In testing, we set the temperature and top to the models' default value. For hyperparameters the reward function, we set Ay=0.2 and 2,=0.4 for the main experiments. We also explore the results under different AØ and Xy in Section 5.3 and Section 5.2
RESULTS & ANALYSIS
This section presents a comprehensive analysis of HiPRAG. We evaluate performance against state-of-the-art baselines and conduct detailed studies the influence of model size. model family, and reinforcement learning algorithms. We also include ablation studies to validate our design choices and conclude with a qualitative study.
5.1 MAIN RESULTS
We compare HiPRAG against suite of strong baselines on seven question-answering benchmarks. As shown Table I our approach outperforms baseline methods across both 3B and 7B model sizes. The HiPRAG-7B model achieves an Avg. CEM score of 67.2%, a notable improvement over the next-best baseline, RI-Searcher++ (62.1%). This demonstrates that our fine-grained, process- based reward mechanism effectively guides the agent to develop more robust and accurate rea- soning trajectories. We also present a qualitative case study comparing the reasoning trajectory from baseline and HiPRAG-trained model in Appendix K as well as a brief analysis of efficacy of our methods Appendix F We also report the raw average number of retrieval calls per ques- tion (Avg. #Searches). Under controlled Qwen2.5-3B 5- Instruct 3---trruuttt PPO setup, HiPRAG reduces Avg. #Searches from 2.45 (Search-RI) and 2.15 (3-GRPO) down to 1.75 on average across seven benchmarks, corresponding 29% and 19% reduction respectively (Table 12

---

Published as a conference paper at ICLR 2026
Base Model
RL Algo.
Method
Avg. CEM
Avg. OSR
Avg. USR
1amna-3.2-3B-nnaee
PPO
bascline
56.4
7.3
57.6
PPO
HiPRAG
64.8
6.0
49.7
Qwen2.5-3BBInntruu
GRPO
baseline
58.5
8.4
52.1
Qwen2.5-3BBInsitn
GRPO
HiPRAG
4.1
33.2
Qwen2.5-3B
PPO
baseline
60.3
3.8
44.0
Qwen2.5-3B
PPO
HiPRAG
63.4
3.2
41.9
Owen2.5-3B-Instree
PPO
baseline
59.3
6.1
47.5
Qwen2.5-3B- -
PPO
HiPRAG
64.1
4.9
38.1
Qwen2.5-7B 3-7B-Instruet
GRPO
baseline
5.2
43.3
Qwen2.5-7B5 5-B-truuuttt
GRPO
HiPRAG
67.2
2.3
32,6
Been2..5-B--nsteeee
PPO
bascline
53.3
7.6
33,9
Qwen2.5-7B IIIstructt
PPO
HiPRAG
64.5
6.2
29.0
Owen2.5-3BBIInstree
PPO
HiPRAG (over-search only)
58.8
4.9
52.7
Qwen2.5-322 Instruct
PPO
HiPRAG (under-search only)
63.3
6.6
16.9
Owen2.5-3B-Insee
PPO
HiPRAG (Ap = 0.2)
59.6
5.5
44.5
Qwen2.5-3BBInstrul
PPO
HiPRAG ()p- 0.6)
62.5
5.2
39.0
Table 2: Summary average scores in percentage of CEM (higher is better), OSR (lower is better), and USR (lower is better) for HiPRAG models trained with different parameters. Best and second- best values for each metric are marked in bold and underline. Baseline here refers to the reward with process bonus disabled (Ap 0). A detailed report each individual dataset is in Appendix
crttic/rewrdssmmma
Owen2.5 Instruct PRO HIPRAS
GRPO HIRRAS
0.9
0.5
0.6
0.6
0.4
0.5
0.2
150
200
250
300
350
Step
Stops
200
300
-1
(a) Reward curves of Swee2255-B-IInstern model trained with PPO/GRPO + HiPRAG with respect of the training steps.
(b) Curves of the ratio of searches among rea- soning steps of Dwwen.55-BB-Insttee and Llama- 3.2-3B-Instruct.
Figure 2: Reward curves for different RL algorithm and curves of the ratio of searches among all reasoning steps for different model families.
5.2 ANALYSIS ON INDIVIDUAL PARAMETERS
Influence Model Size Larger models generally exhibit stronger reasoning capabilities with bet- ter accuracy. Our experiments confirm this trend, according to Table 2 as HiPRAG-trained 7B models consistently outperform their 3B counterparts. However, our process-basee reward approach allows smaller models to achieve remarkable performance, closing the gap larger models. For instance, our HiPRAG model trained based on Qwen2.5-3B- not only surpasses strong external 7B baselines like R1 R1-Searcher++ (62.1% 1% Avg. CEM), but also outperforms the 7B counterpart trained with our baseline reward (61.2% Avg. CEM). This indicates that training methodology is the model size with conventional rewards. Furthermore, a larger model size tends to provide more efficient search decisions. This is evident with GRPO, where the 7B model, on top of its higher accuracy, exhibits better efficiency (2.3% Avg. OSR, 32.6% Avg. USR) than the 3B model (4.1 Avg. OSR, 33.2% Avg. USR).
+ GRPO (64.4% Avg. CEM)
effective path to performance gains than simply scaling
Influence of Model Family Dwen2.5-3B-Innstrul and 3.----3.----Ittttrre models with HiPRAG for comparison. Although both models achieve comparable peak accuracy after training with HiPRAG, their underlying be-
To assess the generalizability of HiPRAG, trained both the
8

---

Published as a conference paper at ICLR 2026
havior and efficiency differ. As shown in Figure 2b the Llama-3B model shows a higher tendency to rely on its parametric knowledge with more non-search steps initially, resulting in a higher under- search rate. After training. the Qwen-3B model achieves its high Avg. CEM of 64.1% with lower suboptimal search rates (4.9% Avg. OSR, 38.1 18.1% Avg. USR) compared to the Llama-3B model's 6.0% Avg. OSR and 49.7% Avg. USR, based on Table [
This suggests that while HiPRAG is effective across different model families, the base model's 's inherent tendencies can influence final search efficiency.
Influence of RL Algorithm To explore the impact of different RL algorithms HiPRAG method, we experimented with both PPO and GRPO on Qwwn225-39B7B--Isstuuc models. PPO offers better training stability, often completing the full training run without reward collapse, whereas GRPO consistently has the potential to achieve higher final performance and converges faster. A compar- ison of reward curve for Qwen-3B models can be found in Figure 2a As seen in Table 2 GRPO yields a higher Avg. CEM for both 3B (64.4% vs. 64. 16) 64) and 7B (67 .2% vs. 64.5% 5% models and results in more efficient search behavior (e.g.. 2.3% OSR for 7B-GRPO vs. 6.2% for 7B-PPO). This aligns with findings in related literature, where GRPO's ' critic-free approach often proves to be more sample-efficient for L.LM training with the trade-off of less training stability.
Influence of Format Reward Weight We experimented with different format reward weights A E (0.2, 0.4, 0.6) while keeping =0 fixed (Qwen2.5-3B-Instte + PPO). Xy=0.2, used in our main experiments, provides the best trade-off: it achieves the highest Avg. CEM (64. 1%) and the best Avg. OSR (4.9%), while larger values over-emphasize the instrumental goal of format correctness and degrade performance (Table 14.
Influence of Instruction Tuning on Base Model To understand the impact of instruction-tuning on the base model before applying HiPRAG for RI., we compared the performance of HiPRAG on base model (Qwen2.5-3BB and its instruction-tuner counterpart (Qwen2.5-3BB) --3e....-B-nsstrcct)) Based on Table 2 the instruct-tuned model exhibited a higher initial reward, as its pre-training makes it more adept at following the structured output format required by our framework. Our hicrarchical reward, which gates the process bonus until both the answer and format are correct, favors models that quickly learn this structure. However, the base model eventually caught up, converging to a similar reward level. Interestingly, the base model, once fully trained, may achieve higher Avg. CEM score (65.4% vs. 64.1%) %) and lower Avg. OSR (3.2% vs. 4.9%). This could be because learns the reasoning and search behaviors from the RL objective more purely, without the potential biases introduced during the instruction pestuucton-tuning phase.
5.3 ABLATION STUDIES
To validate the key components of the HiPRAG methodology and systematically isolate the sources of its performance gains, we conducted series of ablation studies. These experiments are designed to deconstruct our approach by: (1) evaluating the impact of the new parsable output format indepen- dent of the process reward, (2) determining the optimal weighting of the process bonus coefficient, Ap. which governs the reward hierarchy, and (3) demonstrating the necessity of addressing both over-search and under-search behaviors concurrently, rather than in isolation.
Influence on Output Format To isolate the effect of the format and reward change, we trained model variant called Search-RI-step" This model uses the same outcome format reward as the original Search-R1 v0.3 model (Jin et al. (225a) but is enforced to use our new parsable output format. Besides, we also adapt the format change to 3-GRPO and trained /-GRPO-step'. The results from both variants in Table [ show that our structured format maintains the performance, with slight increase in some of the datasets. This confirms that the new parsable output format is robust foundation and that the significant performance gains of our full method attributable to the process-be broses reward mechanism it enables, not merely an artifact of the format change.
Influence of Process Bonus Coefficient We tested our hierarchical reward with different values for the process bonus coefficient Ay. which determines the weight of the coreecorres ratio. Based on Table a coefficient of 0.4 provided the optimal balance, yiclding the highest performance (64.1 1% Avg. CHM). A lower value of 0.2 behaved similarly to outcome-only reward, failing to

---

Published as a conference paper at ICLR 2026
sufficiently incentivize efficiency (59.6% Avg. CEM). This is reflected in its higher Avg. OSR of 5.5% and Avg. USR of 44.5%. A higher value of 0.6 over-prioritiee step purity at the expense of final answer correctness. leading to a slight performance degradation (62.5% Avg. CEM). The optimal A of 0.4 achieved the best trade-off, with a low 4.9% Avg. OSR and 38. 1% Avg. USR. Importantly, Ap controls the relative reward spread among already-correct trajectories: when Xp is too large, the gradient becomes dominated by differences in step ratios among correct outputs, encouraging short, conservative plans and reducing the model's willingness add corrective steps that could fix borderline errors.
Training with Over-search or Under-search Only We isolated the components of our process reward by training models with penalties for only over-search or only under-search. In these abla- tions, the ignored step type excluded from the computation of process bonus ratio. If trajectory contains zero steps of the evaluated type, we set the process bonus to the average bonus within the current training batch for stability. Based on Table 2 training to reduce only over-search proved insufficient, yielding low Avg. CEM of 58.8% 58.8% While this approach successfully reduced the Avg. OSR to 4.9%, it caused the model to become too hesitant to search, resulting in a very high Avg. more effective (63.3% Avg. CEM), underscoring
USR of 52.7%. Targeting only under-search that preventing hallucination is more critical than improving efficiency. This method dramatically lowered the Avg. USR just 16.9% but made the agent overly reliant its search tool, slightly in- creasing the Avg. OSR to 6.6%. However, the best performance was achieved only when penalizing both suboptimal behaviors simultaneously (64. 1% Avg. CEM), confirming that holistic approach to search optimization is necessary.
6 CONCLUSION
This work tackles the pervasive inefficiencies in agentic RAG systems, which are often trained with coarse, outcome-based rewards that fail to correct suboptimal search behaviors. We introduced HiPRAG, a novel training methodology that incorporates hierarchical, knowledge-aware process reward to instill more efficient reasoning. By decomposing agent trajectories into parsable steps and evaluating the necessity of each search action on-the- nn-thhe-fly HiPRAG provides the fine-grained super- vision necessary to curb both over-searching and under-searching. Our experiments show that this approach not only achieves state-of-the-an performance on 3B and 7B models but also dramatically improves efficiency, reducing the over-search rate and under-search rate. This research validates that the path to creating powerful and efficient LLM search agents lies in optimizing the reasoning process itself, not just the final outcome.
ETHICS STATEMENT
The authors of this paper have read and adhered to the ICLR Code of Ethics. This work, focuses on improving the efficiency and accuracy of Al agents, and we have considered the ethical implications of our methodology. While any capable AI system presents potential for misuse, our method aims to foster more reliable and grounded AI by explicitly penalizing factual errors (under-searching) and reducing computational waste sover-searching). We acknowledge that the system may inherit biases from its underlying data (Wikipedia) and foundation models. By improving agent efficiency, work also contributes more sustainable AI by reducing the energy consumption of deployed models.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our work. Section 4 provides comprehensive overview of our experimental framework, including the specific training and evaluation datasets, evaluation metrics (Cover Exact Match, Over-search Rate, and Under-search Rate), and baselines used for comparison. Section 4.3 details our training procedure, specifying the models (Qwen2.5, Llama-3.2), RL algorithms (PPO, GRPO), hardware, retrieval environment, and key hyperparame- ters. The core components of our HiPRAG methodology the structured output format, the on-the-fly detection mechanism, and the hierarchical reward function-are thoroughly described in Section To further facilitate replication, the Appendix contains the pseudocode for inference process
10

---

Published as a conference paper at ICLR 2026
(Algorithm along with the exact prompts for enforcing the parsable output format (Figure and for detecting suboptimal searches (Figures and
REFERENCES
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan, Wen Zhang, Huajun Chen, Fan Yang. Zenan Zhou, and Weipeng Chen. Research: Learning to reason with search for Ilms via reinforcement learning, 2025. URL
2503. 19470
Kaustubh D. Dhole. To retrieve or not to retrieve? uncertainty detection for dynamic retrieval augmented generation, 2025, URL
09292
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur. Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak. Chloc Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve. Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes. Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- s00 Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji. Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nic, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero. Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu
11

---

Published as a conference paper at ICLR 2026
Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog. Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- Baptiste Gaya, Jeff Marcus, Jeff Tang, Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang. Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L.. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji. Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang. Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan. Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay. Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar. Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li, Tianjun Zhang. Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiao- jian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia Ye Qi, Yenda Li. Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary De Vito, Zef Rosnbrick, Zhao- duo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL 21783
Xinyan Guan, Jiali Zeng, Fandong Meng, Chunlei Xin, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Jie Zhou. Deeprag: Thinking to retrieval step by step for large language models, 2025. httpss:t/://// /tt////awwv..rrga/////////////// 01142
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp.
Barcelona, Spain (Online), December 2020. International Com- mittce on Computational Linguistics. doi: 10.18653/v1/2022 1/003////2....inmmmmin
URL https:
7/acl anthol wechhcllgyy.ooo wgy.oogg/0200. col ng- mai n.087
12

---

Published as a conference paper at ICLR 2026
Jerry Huang, Siddarth Madala, Risham Sidhu, Cheng Niu, Hao Peng, Julia Hockenmaier, and Tong Zhang. Rag-rl: Advancing retrieval-auggmented generation via and curriculum learning, 2025a.
URL httpss:/aaxii...... /attss///aaxxv.orrr////////33..3 12759
Ziyang Huang. Xiaowei Yuan, Yiming Ju, Jun Zhao, and Kang Liu. Reinforced internal-external knowledge synergistic reasoning for efficient adaptive search agent, 2025b. URL https://
arxiv.org/abs/25555. 07596.
Liu Huanshuo, Hao Zhang, Zhijiang Guo, Jing Wang, Kuicai Dong, Xiangyang Li, Yi Quan Lee, Cong Zhang, and Yong Liu. CtrlA: Adaptive eetreval-augmmened generation via inherent con- trol. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACI. 2025, pp. 12592-12618, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 256-5. doi: 10.18653/v (0/653/v1/2025.52 wi..02...ininnsacl.... I Finoings-acl.6520
URL httpp:://aeeeeattoooooooo.o/ otg../...t....ooooo.ooo///222.
Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O. Arik, and Jiawei Han. An empirical study on reinforcement learning for reasoning-seare search interleaved IIm agents, 2025a. URL https:
//arxiv. wwwxiv.erg/aza. wtt.eergggabbbuueeeeeeeee
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. 2025b.
anttpa://arxiv..r////// 29530
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguis- fies (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for org/
Computational Linguistics. doi: 10. 10118665////11711147,,
URL
P17-11477
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 1.116653/112222..mmmmmmmmm URL
wwwplooyyooogoooooo 2020. sannp-mmain. 550/
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162/aac URL
nttps://aclaattoloss wwtp:t/ooooooooooooo//11112 10267
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nip tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829548
Jian Li, Xiaoxi Li, Yan Zheng, Yizhang Jin, Shuo Wang, Jiafu Wu. Yabiao Wang, Chengjie Wang, and Xiaotong Yuan. A survey ai search with large language models. Preprints, July 2025a. doi: 10204444peppimm222000002222 prepring wtw.iittt222222... 2024.
URL htttps://doi. orr.oi..orr///ioo (tt.::o.00r111100000..
Xiaoxi Li, Guanting Dong. Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models, 2025b. URL. 05366
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl, 2025c. URL https:
13

---

Published as a conference paper at ICLR 2026
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non coraammettiic memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings the sl Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1802-9822, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. http://w//222...ooogog...
OpenAl. Introducing gpt-4.1 in the api. OpenAl Blog, April 2025a. URL https:://penai..omm index/gpt-4-1
OpenAl. Introducing gpt-5, OpenAl Blog, August 2025b. URL httpp:://ooena..co/////eee//
int rnduciing---
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711, Singapore, December 2023, Association for Computational Linguis- tics. doi: 10. 118886633/122223 wwndiiiiinnnnemm...17777 URL Findings eanggg-enniip 378/
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- -ürr Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs, 2025a. URL httpp:://axxii../
abs/2502 13958
Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tün Gokhan Tur, and Heng Ji. Smart: Self-aware agent for tool overuse mitigation, 2025b. URL.
Petps://arxtor wwtp://wwwg/////////////
14355
Qwen,
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL https: arxiiv www.s/orgrrrro.e abs///bb///// 2412.15111111
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. http://ppp////ai/.o////aaaa/a// 06347
Zeyang Sha. Shiwen Cui, and Weiqiang Wang. Sem: Reinforcement learning for search-efficient 07903
large language models, 2025. URL
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of math- ematical reasoning in open language models, 2024. URL
03300
Yuanhao Shen, Xiaodan Zhu, and Lei Chen. SMARTCAL: An approach self-aware tool- use evaluation and calibration. In Franck Dernoncourt, Daniel Preofiuc-Pietro, and Anasta- sia Shimorina (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 774-789, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi:
URL
Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. Agentic retrieval-augmennte generation: A survey on agentic rag, 2025. URL httpp///aaa....og//aaa//////////
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao Lei Fang, and Ji-Rong Wen. RI-searcher: Incentivizing the search capability in 1lms via reinforcement learning, 2025a.
14

---

Published as a conference paper at ICLR 2026
Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-searcher++: Incentivizing the dynamic knowledge acquisition of Ilms via reinforcement learning, 2025b. URL httpp:://arxv..org abs/2505. 17005
Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. DRAGIN: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12991-13013, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/1/ 0024.acl-long.700 702. URL
wttp://///.../////o/////////.../ acl- long. 702/
Zhongxiang Sun, Qipeng Wang, Weijie Yu, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Yang Song, and Han Li. Rearter: Retrieval-augmenter reasoning with trustworthy process rewarding. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '25, pp. 1251-1261, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400715921. doi: 10.1145/3726302..3330000
URL https:
7/col.or//10. 1ttp://0.1155/32332222277737 3730102
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Mul- tihop questions via single-hop question composition. Transactions of the Association for
Computational Linguistics, 10:539-554 2022.
doi: 10.1162/aalla.0044 URL https:
//aclanthclogy. wttp:oppg/ooooooooo 1tw.-1..2...1111117
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interlcaving trieval with chain-of thouhh reasoning for Knowledge-intensive multi-step questions. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10014-10037, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.186333 acl-long.557. URL attpps://aclanttollggs. wwwp://...g/22.../... htt.aaaatttoooooooor//2/.....e
Long.5577
Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Acting less is reasoning more! teaching model to act efficiently, 2025a. URL/httss:///att //wp:vv.aaiivvv/aa wwwpiiioorr//boooo wtt///r/////a/ooa /254 .et.//2550444.070000
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by
contrastive pre-training, 2024.
htRL:hhtss:://// : //aaiii.///vv.///aa www.wwwwwwwww....aea/oereeraeaee 2tt//2222..0 03533
Liang Wang. Haonan Chen, Nan Yang. Xiaolong Huang, Zhicheng Dou, and Furu Wei. Chain-of- org.oogggaa.ooo. wwwp////.... 25042.11114
retrieval augmented generation, 2025b. URL
Peilin Wu, Mian Zhang, Xinlu Zhang, Xinya Du and Zhiyu Zoey Chen. Search wisely: Mitigating sub-optimal agentic searches by reducing uncertainty, 2025. URL htttps://arxii.oor
2505.17288 17811
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL /tttps://aaxiis org/ abs/absss/5555 09388
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and aunichii Tsujii (eds.), Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369- 2380, Brussels, Belgium, October-Novemble 2018. Association for Computational Linguistics. doi: 10. 10.18653/v/DD1--122 URL URL htt.:: //c anthol www.chhoooy..rgg/// 1-299//
15

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Zijun Yao, Weijian Qi, Liangming Pan, Shulin Cao, Linmei Hu, Liu Weichuan, Lei Hou, and Juanzi Li. SeaKR: Self-aware knowledge retrieval for adaptive retrieval augmented generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Vol- ume Long Papers), pp. 27022-27043, Vienna, Austria, July 2025, Association for Compu- tational Linguistics. ISBN 979-8-89176-251-0. doi: 10. 11865311/2025 www.eoo...llo....... 1312. URL
1312/
Chenlu Ye, Zhou Yu, Ziji Zhang, Hao Chen, Narayanan Sadagopan, Jing Huang, Tong Zhang, and Anurag Beniwal. Beyond correctness: Harmonizing process and outcome rewards through rl training, 2025. URL htt.:/ttss:/arttt.orr/////// http://////aaaaaaoo/////oasssa// ww0e 03403
Chuhuai Yue, Chengqi Dong, Yinan Gao, Hang He, Jiajun Chai, Guojun Yin, and Wei Lin. Promot- ing efficient reasoning with verifiable stepwise reward, 2025. URL https:///axxiy: org/ /abs////////
2508 10293
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, Yifan Zhou, Yang Chen, Chen Zhang. Yutao Fan, Zihu Wang, Songtao Huang, Yue Liao, Hongru Wang, Mengyue Yang, Heng Ji. Michael Littman. Jun Wang, Shuicheng Yan, Philip Torr, and Lei Bai. The landscape of agentic reinforcement learning for Ilms: A survey, 2025. /ttl:
02547
Hanna Zubkova, Ji-Hoon Park, and Scong- Whan Lee. Sugar: Leveraging contextual confidence for smarter retrieval, 2025, URL https: /tattps://rxii..../ wwtpa/wwwww..////////...././///e 04899
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A EXAMPLE OF SEARCH-RI FORMAT VS. HIPRAG FORMAT
In this section we present example of transforming from the original Search-RI output format to our HiPRAG output format shown in Figure 3 Original format interleaves multiple <think> blocks with search operations, making step boundaries ambiguous. Our format uses explicit <step> tags within a single <think> block. Each colored region represents logical reasoning step. Our format clearly delineates these with <step> tags, enabling deterministic parsing. Step (orange) demonstrates internal reasoning without search. The original format embeds this implicitly in prose; our format explicitly marks it as step with only <reasoning> and <conclusion> tags.
17

---

all necessary information. The company is AMD, the CEO is Dr. Lisa Su, and the clesing stock price provids the Published as a conference paper at ICLR 2026
Comparison of Reasoning Trajectory Formats
(a) Original
HiPRAG Parsable
Question:
Who is the current CEO of the compuny that makes graphics cards
Question: Who the current CEO of the compuny that makes the graphics cards
PlayStation, what was their stock price the close of the last trading day?
PlayStation, what was their stock price the close of the last trading day?
<think>
<think> <step>
multi-part question. need to figure out the lateat PlayStation, who makes its graphics card, who the is, and the'r recent stock price. I'll start by finding the most recent PlayStation consale.
Okay, this
<reasoningo This multi-part
question. First, I need to ident the latest PlayStation console. Then, need to find cut which company makes its graphics carda. I suapect either Sony itself, WIDIA, AMD, but I need to be certain. I will start by identifying the latest PlayStation model. </reasoning> <search> latest playstation consale madel </search>
that company
</think>
<search> latest playstation console
model </search>
<information>
PlayStation console the PlayStation 5 (PS5), which was
<context> The latest PlayStation console is the PlayStation 5 (P85).
released by Soay November 2020.
features
2-based
</context>
GPU.
<conclusion> PlayStation (PS5)
</information>
</conclusions
<think>
</step>
So, the latest consale is the PlayStation 5,
<step>
know console is the PS5, read to find the manufactuzer of its graphics processing unit <conclusion> AMD </conclusion> </step>
<reasoning> Now
and the graphics card is made by AMD.
Now that I've ident fied the company as need find current CRO.
</reasoning>
</think>
<search>
CBC of AME </search>
fied the company AND Now need to find its current CRO. </reasoning> currest CEO AMD </search> <context> The current Fresident and CEO AND Dr. Lisa Su.
<step>
<reasoning>
<information>
and chicf executive offices of Advanced Miere Devicea (AMD) She has held the CEO positicn since </information>
</context>
<think>
<conclusion> Dr. Liss Su
Lisa The final p'ece of information I need is the closing laat trading day. Since today is Sasurday, July 19, 2025, I need the price from Friday, July 18, 2025. </think>
Great, the
</conclusion>
</step>
price
<step>
<reasoning> have the company (AMD) and the CEO (Lisa Su) Now I need price. Today Saturday, July 19, 2025, so need the closing price from Friday, July 18, 2025. </reasoning>
<search> AMD stock price closing July 18 2025 </search> <information
<search> AMD stock price closing July 18 2025 </search>
On Priday, Joly 2025, the stock for Advanced Micro Devices, Inc.
<context> AMD (Advanced Micra Devices, Inc.l closed at $175. 40 on Friday, July 18, 2025. </context> <conclusion> $175.40 </conclusion>
(ticker: AMD) closed at a price of
$175.40 per
</information>
<think>
</step> </think>
<answer>
Dr. Lisa Su and $175.40. </answer>
$175.40. final answer.
</think>
<answer> Dr. Lisa Su and $175.40.
</answer>
Figure 3: Comparison of reasoning trajectory formats for the same multi-hop question. Fach logical step highlighted in a consistent color across both formats show the correspondence. The actual retrieved documents here are replaced by their summarization to improve readability.
18

---

Published as a conference paper at ICLR 2026
B SYMBOLS
Symbol
Description
T
A complete reasoning trajectory for a given question.
The final answer within trajectory
The i-th step in reasoning trajectory.
g
A search step, represented as tuple (ri, q: o.).
NR
A non-search (internal reasoning) step, represented as tuple (r,, 0.).
The reasoning block of step, containing planning and analysis.
The search query generated in a search step.
6
The retrieved context returned from a search query.
The conclusion or summary of knowledge from step.
The re-generated answer used for over-search detection.
A(T)
A function indicating if the final answer is correct (I) or not (0).
F(T)
function indicating if the trajectory follows the required format (1) not (0).
N(T)
The total number of steps in trajectory T.
coer
The number of optimal (correct) steps in trajectory T.
Over(.)
The detector function that identifies an over-search step. The detector function that identifies an under-search step. A hyperparameter representing the format weight.
Under
Xy
Xp
A hyperparameter representing the process bonus coefficient.
R(T)
The final hierarchical reward calculated for trajectory T'.
Table 3: Description symbols and notations used.
C ALGORITHMS
Algorithm Inference with Parsable Steps
Require: Question model retriever Retrieve, step budget B, top-K, preprocessed input (including in- structions and user question) y
Ensure: Transcript with single <think> container and final <answer>
1: y f <chinkt><st.p>>>>>>>>>>>>>>>>>>> creassppceeesnnnnn
i
2: while < B do
3: A GENERATE(T,, until boundary E
y A
boundary = </answer> then 6: break
b final answer already produced
7:
else if boundary </searc>> then
8:
LASTBETWEEN(J, <search>, </search>)
Retrieve(s, K); e FORMAT(D)
10:
y/ - y <cccttettt>>>>>>> || cll </ </context><conclusioo
11:
continue
remain in the same step
12:
else if souudary=</come I usi on> then
13:
V 1--- <><//tpp>>>>>
--i -i++
15:
i < B then
16:
y yll <step><rrasonnnng>
17:
end if
18: end if
19: end while
20: & y </think>>
21: if not CONTAINS(J, </answer>) then
22: y/ /+vl yl <answer> GENERATE(T, until </answer>
23: end if
24: return y
19

---

enforce strict order Published as a conference paper at ICLR 2026
Algorithm 2 Format Checker F(T)
Require: Model output y (string)
Ensure: F(I") E {0, and (optionally) (T)). the number of steps; return (0, -1) on failure
1: y NORMAI.IZF())
D. canonicalize newlines; keep text intact then return -1)
2: if COUNT((; <think>> * COUNT(Y, </think>)
3: (Repen; iclose) f BETWEEN, <think>, </think>)
indices of content
4: if TRIM(Y iapen) * then return (0,-1)
> no stray text before <think> the content inside <think>
5: 1 BETWEEN(Y: <think>, </think>)
6: post <- Visse + |</think>/
if not STARTS WITHFTSSWSWAFSWSSSPP (post, <answer>) then return (0,-1)
8: COUNT(pOst, <answer>) & or CoUNT(pOst, </answer>) + then return (0, -1)
9: ans + -EETEEEN <answer>, </answor>)
10: if TRIM(ans) = then return (0, -1)
11: TRIM(post after </answer>) # then return (0,-1)
no trailing junk
12:
> Inside <think>, allow only a sequence of well-formed <step> blocks
13: t 0: N -
14: while True do
15:
FINDFROM(t, <step>, if - then
16:
17:
TRIMM(t :)) then return (0, -1} break
> no stray text between steps
18:
19:
end if
20:
TRIRIM(tj:: :)) * then return (0, -1)
D no text before next <step>
21:
& + FINDFROM(t, </stop>, j)
22:
if -- then return (0, -0)
23:
st +-t <<seppp
step body
24:
not VALIDATESTEP(() then return (0, 10)
25: p p+ += |<stpp>;
N+1
26: end while
27: if NN- < then return (0, -1)
D must have at least one step
28: return NI
Algorithm 3 VALIDATESTEP accepts exactly two step schemas (with or without search)
Require: Step body (string)
Ensure: True iff matches one of the allowed formats below and tags are unique, ordered, and properly nested
I: 8 4- STRIP(s)
COUNT(s, <reasoning>) * or COUNT(s, < </reasoning>) & then return False
3: (Fo,Ta) + SPANBETWEEN <reasoning>, </ </reasoning>)
4: if TRIMsS]0: r.]) then return False
<reasoning> must be first then return False
5: if COUNT((, <conclusion>) or COUNT(s, < </conclusion>)
(Co,ee) + SPANBETWEEN(S,, <conclusion> </conclusion>)
7: co S re then return False
> <conclusii lusionlusonn must come after </reasoning>
8: if TRIM(s(cc + ///////clssionn :]) # then return False
>> step must end at </conclusion
9: Case A: step with search: <reasoning>
<search> <context> <conclusion>
10: CONTAINS(S, <search>) or CONTAINS(S, <context>) then
11:
if COUNTTs, <search>) * 1 or COUNT(s, </search>) # then return False
12:
if CoUNT(s, <context>) * 1 or COUNT(s, </contxxt>) * then return False
13:
(qo, go) + SPANBETWEEN(S, <search>, </scarch>)
(c) SPANBETWEEN(S, <context>, </context>)
15:
if 2 9o or qe 2 vo or x 2 co then return False
16:
iFTRIM(s|Te: qol) # or TRIM(s|ge : xol) * e oor TRIM(a[Te col) * then return False
17:
return True
18: end if
19: P. Case B: step without search: <reasoning> + cconclusion>, and no <search>/<context>
20: CONTAINS(S, <search>) or CONTAINS(S, <context>) then return False
21: if TRIM(s/re= col) # then return False
22: return True
20

---

Published as a conference paper at ICLR 2026
D PROMPTS
System Prompt for Parsable Output Format
Answer user questions by thinking step-by-step. Your entire reasoning process must be encapsulated within single <think> < //tt>>>/thn>>>>/ block, which contains one or more <step>>/sten blocks. Each step must begin with your analysis in <reasoning>, If you identify knowledge gap, you may ehearch>queryee seeerry/search> search> query search engine; search results will then be provided in <cont tag. Every step must end with a <conelusions summarizing what you learned in that step. After your thinking process is complete, provide the final, conclusive answer inside an <answer> tag placed immediately after the closing </think> tag. You can use as many steps as you need. Ensure all XML tags properly formed and nested.
**## Output Format Specification
Your output must follow this overall structure. The <think> block contains all steps, and the <answer> block follows
<think>
<step>
</step>
<step>
</step>
</think>
<answer>Your final, conclusive answer to the user's s
question.</aat </answer>
**## Step Formats (to be used inside <think>)**
Format 1: Step with a Search
<step>>
Yreasoning>Your detailed analysis..<<rrassinng <search>The precise search query...<sseare <context> (Provided by system] </coatet/
<conclusion>Your conclusion for this step. coonclusioo </step>
Format 2: Step without Search (Internal Reasoning)
<step>
<reasoning>Your detailed analysis... </reasoning>
<conclusion>You conclusion for this step. concclusion </step>
Figure 4: Input prompt for generating HiPRAG's parsable output format with the new XML tagging system.
21

---

Published as a conference paper at ICLR 2026
Prompt for Over-search Detection
You are an expert in Natural Language Understanding and Semantic Analysis. Your goal is determine if these statements semantically equivalent-that is, if they mean the same thing and convey the same core information. Provide your answers with single boolean value "True" or "False" the tag <enswer></answer> (e.g. <answer> answer> or FanswrrFalsst//nseer
Figure 5: Prompt for Over-search Detection
Prompt for Under-search Detection
You expert Fact-Checker and Logic Verifier. Your task is to evaluate single, isolated reasoning step from Al agent. This step was generated without using search tool. Your goal to determine the agent made mistake by not searching. based only on the information within this single step and your own general knowledge.
Analyze the provided step by asking questions:
Factual Accuracy: the statement the <reasoningx<rreesonningg and (conclussoon><//ooelllss
factually correct?
2. Internal Logie: Does the <conclusion><4 wwwiiiioo///ouuuiiiiiooooooaaaae wwt./osonciooooooo logically follow from the creasoiiggx<reessoiinn provided within this same step?
If both questions answered correctly, provide your answers with single boolean value "True" "False" in the tag <answer><<anawer> (e.g. <answer>\rue<//neee
<answer>Falsed <ansser>aale<</sseee>> aanswrrr))//
Figure 6: Prompt for Under-search Detection
22

---

Published as a conference paper at ICLR 2026
E DETAILED REPORT FOR CEM, OSR, AND USR
Base Model
RL Algo. Method
NQ
General QA TriviaQA
PopQA
HotpotQA
2Wiki
Multi-Hop QA MuSiQue
Bamboogle
Avg.
L,2m3-3,2-3B- 3.2-3B-IIsttuus
PPO + HIPRAG
71.6
77.2
61.0
51.7
67.9
25.7
43.2
64.8
Dowen2.5-38-Imae
GRPO baseline
59.6
69.1
57.3
52.4
61.4
20.6
24.8
58.5
Qwea2.5-3B-Imn
Qwen2.5.38
PPO + buseline
60.6
71.7
55.8
54.3
63.7
24.1
40.8
60.3
Owen2.5.30
PPO + HiPRAG
68.7
75.5
66.3
57.4
67.4
24.1
41.6
65.4
Qwen2.5.38--8 Insttnet
PPO haseline
60.9
70.1
$2.0
37.6
59.3
Quea2.5-388-nstte
PPO + HIPRAG
65.6
73,9
62.1
53.6
69.6
26.0
32,8
64.1
Owen2.5-72 Itsttructtt
GRPO + baseline
62.4
74.4
57.3
54.8
64.2
25.3
49.6
61.2
Qwee2.5-78-Insii
GRPO HiPRAG
71.2
76.3
63.2
62.4
71.7
3A.1
$2.8
67.2
Quea2,5-78-Inaiin
PPO + baseline
55.6
67.5
43.5
49.4
58.5
44.0
53.3
Qwea2.5-7B- I-sttuctt
PPO + HiPRAG
66.2
75.7
58.4
59.9
66.2
34.3
$2.0
611.5
Owea2.5.3B 1oea2.5:3B-aa Instrcctt
PTO HiPRAG (over-seurch
61.9
66.9
54,9
52.2
65.4
25.5
39.2
58.8
Qwen2.5. Instruct
(under-seurch only)
Qwen2.3 30322-5-----mme Instructt
PPO + HIPXAG (Ap 0.2)
61.9
71.2
56.8
33.7
62.2
25.4
31.2
59.6
Qwva2.5-3 -asttttttttt
PTO HIPRAG (
66.6
74.4
60.5
55.5
64.4
25.6
38.4
62.5
Table 4: Cover Exact Match in percentage of HiPRAG models trained with different parameters on seven QA benchmarks. Best and second-best results for each dataset are marked in bold and underline, Baseline here refers to the reward with process bonus disabled (Ap =)=0).
Algo. Method
General TriviaQA
PopQA
HotpetQA
Multi-Hop MuSiQue
Bamboogle
Avg.
Llama-3.22 3.2-3B I-structt
PPO + baseline
12.5
15.4
5.0
1,8
3.7
2.7
8.7
7.3
Llama-3.2-3BB. 3.ntuct
PPO + HIPRAG
11.9
13.3
4.6
1.8
3.1
5.0
6.0
Quee2.5.30
17.0
Qwen2 Quca2.5-38-Imse 5--ttuuttt
GRPO + HiPRAG
4.4
9.8
2.2
3.0
2.9
14
3.9
4,1
Owea2.5.3B
PPO baseline
9.0
299
1.5
1.7
4,3
3.8
Qwee2.5.38
Qwen2.5.30 Itn-srrutt
PPO + buseline
13.5
1.8
3,9
12.8
6.1
Quen2.5-3HHIInmmme
PPO + HiPKAG
6.0
11.0
4,5
2.5
2.8
11,
4.9
D
baselinc
Qwen2.5.7B- Instruct
GRPO HiPRAG
5,4
L8
0.3
4.8
2.3
Ques2.A-78-Instten
PPO baseline
11.6
19.8
6.1
7.4
27
8,5
19,6
7.6
Qwea2.5-77-Insstuct
Qwen2.5. Insttncttt
PPO + HiPRAG (over-search only)
11.0
25
28
11.5
4.9
Owen2.5. 3B Instruct
PPO + HiPRAG (under-search only)
8.2
15.6
5.9
5.3
277
3.1
6.4
6.6
Qwen2.5. 10em2.5-3B-liee Ins-trettttt
PPO HiPRAG
0.2)
7.4
11.4
2.3
12.1
5.5
Dowea2.5338-Imsiii
PPO + HiPRAG (A,
13.1
1.3
2.2
6.3
5.2
Table 5: Over-search Rate in percentage (lower is better) second-best values for each dataset are marked in bold and underline. Baseline here refers to the reward with process bonus disabled (Ap=0) 0).
QA benchmarks. Best and
Base Model
RL Algo. Method
NQ
TriviaQA
General QA
PopQA
HotpotQA
2Wiki
Multi-Hop QA
MuSiQue
Bamboogle
Avg.
1l2ma-3,2-3D.
PPO HiPRAG
35.3
48.4
31.7
50.5
55.3
64.3
10.3
49.7
Qwen2.5.38. Instrrcct
GRPO baseline
61.9
63.9
59.6
46.
49.1
61.9
22.3
52.1
GRPO
Qwe32.5.3B
PPO + baseline
33.3
66.7
30.8
38.5
47.5
66.7
0,0
44.0
Owen2.5.3B
PPO + HiPRAG
43.9
36.4
42,3
41.9
42.6
$6.8
16.7
41.9
Qwen2.5- 2225--9--ssttrr
PPO haseline
33.2
Qwee2.5.38. Instrtuu
HiPRAG
Qwee2.5- Sovea25-78-Innitae
GRPO + baseline
40.5
34.3
43.8
40.9
45.0
56.2
20.0
43.3
Owea2.5-7B. 7Bs-nsttt
GRPO HiPRAG
30.2
34,9
34,9
40.5
24.4
37.3
41.7
32.6
Qwee2.5.77. Instruct
13.9
Qwea2,5-78-
PPO + HiPRAG
57.1
44.9
25.5
20.0
34.6
$7.1
1.6
29.0
Qwen2.5:33 Instruct
PPO HIPRAG (ower-search
54.5
55.2
48.9
44.7
53.7
78.3
20.0
52.7
Qwee2.5. Instructt
(under-seurch only)
20.4
Qwen2.5. 3B Instructt
PPO + HiPRAG (A, 0.2)
24.0
39.5
55.8
41.3
45.5
80.1
30.1
44.5
Qwee2,5.3H4 In----tttttttt
PPO + HIPRAG
27.2
34.1
33.6
69.6
$1.2
$3.3
1.6
39.0
Table 6: Under-search Rate in percentage (lower is better) on seven QA benchmarks. Best and second-best results for each dataset marked in bold and underline. Baseline here refers to the reward with process bonus disabled (Ap 0).
23

---

Published as a conference paper at ICLR 2026
F ADDITIONAL ANALYSIS ON EFFICACY
F.1 FORMAT CORRECTNESS PERCENTAGE ANALYSIS
To verify the effectiveness of our prompting and reward strategy for enforcing a machine parsablee output, analyzed the format correctness across all test samples generated by our final HiPRAG- trained models. Our analysis shows that 96.3% of all generated trajectories successfully adhered to the required format. This high percentage confirms that the models effectively learned to produce the structured output, which is a critical prerequisite for the successful application of our on-the-fly process reward mechanism.
F.2 EFFICACY OF OVER-SEARCH & SEDERRSSAARH DETECTION
The reliability our process-based reward hinges on the accuracy of the over-search and under- search detection methods. To validate these, we manually inspected the detection results for 200 randomly selected reasoning trajectories from our test set evaluations. The manual audit revealed a 98.3% accuracy rate for over-search detection and 95.6% accuracy rate for under-search detection. These high accuracy figures confirm that our on-the-fly LLM-based judges provide a reliable and effective signal for identifying suboptimal search behaviors during RL training.
F.3 EFFICACY OF CEM METRIC
To ensure our primary evaluation metric, Cover Exact Match (CEM), accurately reflects model per- formance, we manually inspected its judgments on 100 randomly sampled question-answer pairs from our test results. Our review found that the CEM metric's assessment of correctness aligned with human judgment in 98% of cases. This confirms that CEM is robust metric for this task, ap- propriately handling valid answers embedded within the longer, more explanatory responses typical of modern LLMs, thus avoiding unfairly penalizing models for verbosity.
G ADDITIONAL STUDY ON ROBUSTNESS TO LLM JUDGE CHOICE
To study sensitivity to external LLM judges used for on-the-fly detection, we conducted an ex- periment replacing our standard proprietary LLMs (GPT-4.1 mini (OpenAl 2025a) and GPT-5 mini OpenAl 2025b)) with open-source models of varying capabilities, specifically the Qwen3- A-.-A----nnntttTTtni
(Instruct for over-search, Thinking for under-search) (Yang et al. 2025). We use greedy decoding with fixed random seed for all 4 judges to ensure stability and reproducibility. For other parameters, we maintain the exact experimental setup as our main Qwen2.5-3B Instruct PPO experiment.
and
LLM Judges
Avg. CEM
Avg. OSR
Avg. USR
Standard (GPT)
64.1
4.9
38.1
Strong Open (30B)
64.0
4.8
39.2
Weaker Open (4B)
63.4
4.9
42.4
Table 7: Summary of Performance (Avg, across 7 Benchmarks).
G.1 ANALYSIS
1. Sensitivity of Over-Search Detection (OSR): The OSR remained nearly constant across all judge configurations. This suggests that the task of detecting semantic equivalence robust and can be handled effectively even by smaller, non-reasoning "Instruct" models. The definition of redundancy does not require complex reasoning chains, making it less sensitive to model size.
2. Sensitivity of Under-Search Detection (USR): The USR showed higher sensitivity. The weaker Qwen3-4B judges resulted a USR increase from 38.1% % to 42.4%. We hypothesize that weaker judges are less rigorous in factual/logical verification, occasionally failing to flag "hallucinated"
24

---

Published as a conference paper at ICLR 2026
reasoning premature conclusions. This allows the agent to terminate the search process earlier than optimal, leading to the slight degradation in the final accuracy (CEM).
3. Performance Degradation: Despite the increase in USR with weaker judges, the overall degra- dation not catastrophic. This confirms that the hicrarchical process reward framework itself pro- vides stable training signal even with sub-optimal judges.
G.2 DETAILED RESULTS BY DATASET
Below we provide the detailed breakdown of performance across all seven QA benchmarks used in the paper.
LLM Judges
NQ
TriviaQA
PopQA
HotpotQA
2Wiki
MuSiQue
Bamboogle
Avg.
Standard (GPT)
65.6
73.9
62.1
55.6
69.6
26.0
32.8
64.1
Strong Open (30B)
65.3
73.6
62.9
55.2
69.2
26.4
32.4
64.0
Weaker Open (4B)
64.7
73.7
60.8
56.5
69.5
24.4
31.5
63.4
Table 8: Cover Exact Match (CEM) by Dataset.
LLM Judges
NQ
TriviaQA
PopQA
HotpotQA
2Wiki
MuSiQue Bamboogle
Avg.
Standard (GPT)
6.0
11.0
3.9
4.5
2.5
2.8
11.5
4.9
Strong Open (30B)
5.9
10.8
3.8
4.8
2.4
1.7
11.9
4.8
Weaker Open (4B)
5.7
9.5
4.2
4.6
3.9
2.6
11.3
4.9
Table 9: Over-Search Rate (OSR) by Dataset.
LLM Judges
NQ
TriviaQA
PopQA
HotpotQA
2Wiki
MuSiQue
Bamboogle
Avg.
Standard (GPT)
11.1
44.4
61.9
25.0
32.0
10.1
8.7
38.1
Strong Open (30B)
11.9
48.1
63.0
22.1
40.4
10.9
5.3
39.2
Weaker Open (4B)
10.7
54.0
60.3
27.2
45.5
18.5
10.3
42.4
Table 10: Under-Search Rate (USR) by Dataset.
G.3 ANALYSIS OF JUDGMENT ERRORS AND ROBUSTNESS TO JUDGE SELECTION
To address the reviewer's concern about the judgment errors, we performed a manual evaluation of judgment accuracy and analyzed the sensitivity of our method to these errors.
Following the protocol in Appendix [ sampled 200 trajectories from the test set and established human-labeled ground truth for step-level Over-Search and Under-Search decisions, for all the steps in those trajectories.
Judge Accuracy:
As shown in Table 11 all models maintain high agreement with human judgment.
We analyzed the correlation between judge accuracy and down- stream agent performance to quantify the impact of mislabeling:
Impact of Judgment Errors:
Under-search:
The Under-search judge is the primary driver of performance variance. The main risk is False Negative (failing to flag an under-search). This effectively disrupts the reasoning trajectory, causing the agent stop searching prematurely, or directly leads to incorrect final results.
Over-search: Over-search mislabels typically result in step being flagged as redundant (or not) without altering the information available to the agent. Consequently. these errors rarely change the final answer path, resulting in minimal impact overall accuracy.
25

---

Published as a conference paper at ICLR 2026
External LLMM Judges
Over-search Accuracy
Under-search Accuracy
Standard (GPT)
98.3%
95.6%
Strong Open (30B) Weaker Open (4B)
97.5%
94.5%
96.5%
92.5%
Table 11: Judge Accuracy against Human Ground Truth.
H ADDITIONAL STUDY ON AVERAGE NUMBER OF SEARCHES
In addition OSR and USR, we report the average number of search steps per question (Avg. #Searches) as raw efficiency metric:
ΣTEDEA ( E TT
Avg. #Searches
(3)
Dost
For fair comparison, this table uses the same Qwen2.5-3B-Instrnee + PPO for methods.
Method
2Wiki
Bamboogle
HotpotQA
MuSiQue
NQ
PopQA
TriviaQA
Avg.
Search-RI
3.38
2.79
2.78
3.39
1.55
1.75
1.53
2.45
3-GRPO
2.85
2.15
2.45
2.50
1,28
1.50
0.90
2.15
HiPRAG (Ours)
2.48
2.02
2.14
2.55
1.01
1.02
1.01
1.75
Table 12: Average number of searches per question (lower is better).
HiPRAG's step-wins wise process reward achieves good efficiency by evaluating the necessity of each step on-the-fly, consistently using the fewest searches while maintaining higher accuracy. We also emphasize that the Over-search Rate (OSR) and Under-search Rate (USR) are critical efficiency met- rics. Naive penalties on trajectory length or search frequency often cause models to avoid searching even when necessary (increasing under-search), which leads to performance degradation.
I COMPUTATIONAL TIME ANALYSIS DURING RL TRAINING
We profile the RL training pipeline quantify the overhead introduced by HiPRAG's on-the-fly reward assignment. The profiling is conducted on Qwen2.5-3B. Seen2.5-3B-Instreeee IB--Instuctt + PPO using 8 NVIDIA A 100 80GB GPUs (veRL with vLLM inference and FSDP training).
Component
Time (s)
Percent
Rollout generation
206.24
63.55%
PPO update
46.19
14.23%
Reference critic inference
30.09
9.27%
HiPRAG reward assignment (re-generation & judge calls)
42.03
12.95%
Total
324.55
100%
Table 13: Runtime breakdown of one RL training step (wall-clock).
The specific overhead introduced by HiPRAG (the batched re-generation and external API calls) ac- counts for only 12.95% of the total training time. This is small increase, especially when weighed against the significant gains in sample efficiency and final model performance.
To effectively reduce overhead in the extra process of reward computation, we utilize the following methods during training.
Batched Generation: The "re- ree-generation" check for over-search is performed via batch generation. We collect the queries from the rollout, aggregate them into several batches, and process the queries in each batch in parallel.
26

---

Published as a conference paper at ICLR 2026
Targeted Scope:
Re-generation is only triggered for search steps, not every step in the
trajectory.
The re-generation prompts the model for a direct answer or relevant information to the search query, which significantly shorter than the full reasoning chain generated during rollout even without actively limiting the generation length.
Generation Length:
Asynchronous API Calls: As noted in Section 3.2 the external detection is executed separately. The batch processing of API calls to the external verifier occurs in parallel with the local model's batched re-generation, therefore effectively reducing the latency. Specifically, for over-search detection, the verification calls to external LLM judges start once the first batch has finished and continue concurrently in the later batches (for example, for re-generation batch I, execute batch / verifier API calls); for under-search detection, these verification calls to external LLM judges be executed concurrently with the generation of over-search detection as they are independent processes.
Cost Reduction for LLM judges: We utilize cost-effective models for verification. On non-reasoning model for over-search detection for further cost saving,
top of that, because the nature of over-search detection is checking the equivalence of the conclusion of the search step and the g--gnneration result, which does not require complex reasoning abilities. For under-search detection, due to the need of detecting logical error in the non- search step, reasoning model is required.
J ADDITIONAL STUDY FOR FORMAT REWARD WEIGHT Xy
To address the impact of the format reward, we conducted study varying X E {0.2, 0.4, 0.6] using the -ensttst-B- PPO setup, keeping the process bonus fixed at Xp 0.4.
The results clearly demonstrate that our chosen value of ' = 0.2 provides the optimal trade- off between answer accuracy (Avg. CEM) and search efficiency (Avg. OSR/USR). As shown in Table 14 the X 0.2 setting achieves the highest accuracy and the best efficiency scores, while larger values over-emphasize the instrumental goal of format correctness and degrade performance.
X
Avg. CEM
Avg. OSR 4
Avg. USR 1
Observation
0.2
64.1
4.9
38.1
Best trade-off of accuracy and efficiency. Degraded accuracy and efficiency.
0.4
63.2
5.2
37.9
0.6
61.2
5.4
41.0
Over-emphasis format harms performance.
Table 14: Changing format weight (Ay) with Ap = 0.4 fixed. The 'y = 0.2 setting used in our paper achieves the highest accuracy and best efficiency scores.
J.1 SENSITIVITY ANALYSIS AND DISCUSSION
This trend is expected, as it aligns with our core intuition that answer correctness is significantly more important than format correctness. We justify this principle as follows:
Terminal Goal vs. Instrumental Goal:
Format correctness (F(T)) is an instrumental goal. Its primary purpose is to make the agent's reasoning trajectory parsable, which in turn enables our hierarchical process reward to be calculated. In contrast, answer correctness (A(T') is the terminal goal of the entire agentic RAG task. The user's objective is to get the right answer; well-formatted but incorrect trajectory is ultimately failure. To prioritize answer correctness, the format weight X must be small.
Preventing "Reward Hacking": If X were set too high, the agent could "hack" the re- ward function by learning to produce perfectly formatted, minimal-step trajectories (e.g., single non-search step) that are factually incorrect. This would maximize the F(T) compo- nent of the reward while ignoring A(T') and the process bonus. A low Xy (like 0.2) ensures the reward landscape is always dominated by the incentive to get the answer right. This principle is directly reflected in our hierarchical reward function.
27

---

Published as a conference paper at ICLR 2026
NQ
TriviaQA PopQA
HotpotQA 2Wiki MuSiQue Bamboogle
Avg.
CEM (f) by dataset (%) under X ablation
0.2
65.6
73.9
62.1
55.6
69.6
26.0
32.8
64.1
0.4
64.7
72.7
61.3
55.8
68.4
26.5
33.5
63.2
0.6
63.5
71.5
60.0
53.9
67.2
23.8
30.8
61.2
OSR (4) by dataset (%) under y ablation
0.2
6.0
11.0
3.9
4.5
2.5
2.8
11.5
4.9
0.4
6.3
11.4
4.0
3.7
3.0
3.9
11.2
5.2
0.6
6.1
11.0
4.4
5.1
2.9
3.2
13.0
5,4
USR ()) by dataset (%)
under ' ablation
0.2
11.1
44.4
61.9
25.0
32.0
10.1
8.7
38.1
0.4
12.0
43.2
60.8
22.1
35.1
9.6
9.1
37.9
0.6
13.4
46.8
64.3
27.6
34.8
11.3
9.7
41.0
Table 15: Detailed performance breakdown for individual datasets under Xy ablation.
K
CASE STUDY
To illustrate the practical benefits of our HiPRAG framework, examine specific case where baseline model fails due to inefficient reasoning, while our trained agent succeeds. The mod- els here trained based Qwen2.5-3B 3-BB2.B-BB-B.. Instrrcct. The question posed is: "What is the place of birth of the performer of song Slow Down (Lacy Dalton Song)?". The baseline model overlooks the crucial parenthetical information, "(Lacy Dalton Song)," and initiates broad search for the song "Slow Down." This leads to an unnecessary five-step process where it identifies three different artists who have song by that title and then searches for the birthplace Furthermore, through over-search detection, the answer of the final search step's 's query about the birthplace of Selena Gomez is answered correctly and equivalent Prairie, Texas" in reasoning trajectory. The final answer of the baseline is incorrect due to the in- terruption of unnecessary searches. This is classic example of over-searching, where the agent performs redundant and irrelevant lookups, ultimately failing to provide single, correct answer. In agent correctly parses the entire question its first, non-search step. identifying Lacy J. Dalton the specified performer. It then executes single, targeted search for her place of birth. This two-step. optimal reasoning path-one internal reasoning step followed by necessary search-avoids the inefficiencies of the baseline, leading directly to the correct answer. This case clearly demonstrates how HiPRAG's process-oriented rewards cultivate more nuanced and efficient reasoning strategy, improving both accuracy and search economy.
each one individually.
the original answer "Grand
contrast, the HiPRAG-traine
28

---

Published as a conference paper at ICLR 2026
Baseline Model (Inefficient Reasoning)
Question: What is the place of birth of the performer song Slow Down (Lacy J. Dalton Song)?
Golden answers: ("Bloomshurg, Pennsylvania', "The Only Town in Pennsylvania', 'Bloomsburg' <think>
<step>
<reasoning>1 don't have direct information about the perforner the song Slow Down' and their place of birth. birth.</reee </reasoning <search>What is the performer of the song 'slow Down' ?< <//search <context>... [Returns Doc :: Bobby Valentino, Doc 2: Douwe Bob, Doc 3: Selena Gomez] ...</contet>>> cconclusion-Based on by three different performers: Bobby Valentino, Douwe Bob, and Selena Gomez. don't have information ion about the place of birth of the performer of this s/occc//ccl
search results, 'Slow Down'
song
</step>
<step>
creasoningssince I don't t have information about the performer's s place of birth, I will first need to determine who the performer of the song 'Slow Down' is </reasnning <search>Who is the perforner <context>... [Returns Doc :: Bobby Valentino, Doc 2: Douwe Bob, Doc 3: Selena Gomez] ...</context>>
the song 'slow Down't ?</search>
<conclusion<Pased the search results, the performer of the song sslow Down can be either Hobby Valentino, </omels//conclssoon
Bob, or Selena
</step>
<step>
<reasoming>Noww that I know the performers, will need find the place birth of each of them... </reasoning> <search>What is the place of birth of Bobby Valentino?<ssearch <context>... [Returns birthplace: Chatham, Kent] </context> <conclusion>2ased the seareh results, Bobby Valentino was born Chatham, Kent. </conclusion>
</step>
<step>
<reasoningpNoww that I have the place of birth of Bobby Valentino, will need find the place of birth of Douwe Bob. Bob. </resoonnsn
<search>What is the place of birth of Douwe Bob?</search> (context>[Betuun Returns birthplace: Amsterdan, Netherlands </contex>> <conclusion/Based the search results, Douwe Bob born in Amsterdam, Netherlands. stt.erllandd.//cocclussoso </conclusion>
</step>
<step>
creasoningsFinally, I will need to find out the place of birth of Selena Gomez. Gomez.</rest </reasoning> <search>What is the place of birth of Selena Gomez? G <context> Returns birthplace: Grand Prairie, Texas </context> <conclusion>"ased Grand Prairie,
the search results, Selena Gomez was born in
</step>
</think>
<answer>Based on the information gathered, the pertormer of the song 'Soow Down' can be either Bobby Valentino, Douwe Bob, or Selena Gomez. Hobby ialentino was born in Chatham, Kent. Douwe Bob was born in Amsterdam, Netherlands. Selena Gomez was born in Grand Prairie, Texas. </answer>
Figure 7: Case study: Baseline reasoning trajectory. The model has five unnecessary search steps (over-searching) and an incorrect, ambiguous final answer.
29

---

Published as a conference paper at ICLR 2026
HiPRAG-trained Model (Efficient Reasoning)
Question: What is the place of birth of the performer of song Slow Down (Lacy J. Dalton Song)? Golden answers: ("Bloomsburg, Pennsylvania', 'The Only Town in Pennsylvania', 'Bloomsburg' <think>
<step>
<reasoningsWhat do need know to answer this question? First, need to find this performer of Slow Down. Then I need to find out the place of birth of the <conclusion>Acordinng to the question, the performer of the song Slow Down (Lacy J. Dalton Song) is Lacy Dalton... </conclusion>> </step>
<step>
<reasoning<Noww that I have the answer to the first part, need to find the place of birth of the performer. Where can find this information?
need search for st.s/reasoning>
<search=?lace of birth of Lacy J. Dalton</search>
<context>Doc (Title: "Lacy J. Dalton") Lacy J. Dalton (born Jill Lynne Byrem; October 13, 1946, Bloomsburg, Pernsylvania) is American country .oger....//cooettt.
<conclusionAAcordding the context, the place birth of Lacy </conclusion>
J. Dalton is Bloomsburg, </step>
</think>
<answer>According to the information found, the place of birth of the performer of the song Slow Down (Lacy J. Dalton Song) is Bloomsbuzg, Pennsylvania. Lacy Dalton was born on October 13, 1946, in Bloomaburg, Pennsyl vania. </answer>
Figure 8: Case study: HiPRAG-trained reasoning trajectory. The model correctly identifies the key entity in non-search step, then executes single, targeted search to find the correct answer efficiently.
30

---

Published as a conference paper at ICLR 2026
L USE OF LARGE LANGUAGE MODELS
We acknowledge the use of the GPT-5 language model provided by OpenAl in the final stages of manuscript preparation. This tool employed exclusively for identifying and correcting typo- graphical and grammatical errors, ensuring clarity and precision in the written presentation. Its use was strictly limited linguistic refinement and did not impact the study's s conceptual framework, research methodology, data analysis, or conclusions. All intellectual contributions and substantive content remain those of the authors. The usage of all other LLMs (GPT-4.1 mini, GPT-5 mini, Qwen2.5, and Llama-3.2) mentioned in this work is part of the experiments themselves.
31


Reviews:
[{"review_id": "l4bxv6muhf", "reviewer": "Reviewer_1kv8", "summary": "In this paper, a new training method, named Hierarchical Process Rewards for Efficient agentic RAG (HiPRAG), is proposed to address common inefficiencies in agentic Retrieval-Augmented Generation (RAG) systems. The authors identify two key problems: over-search (needlessly retrieving known information) and under-search (failing to retrieve necessary information, leading to errors). Current Reinforcement Learning (RL) methods rely on coarse, outcome-based rewards. The proposed method tackles this by incorporating a fine-grained, hierarchical process reward into the RL loop and achieves good results.", "questions": "Please check the \"Weaknesses\" and also provide comments on the following questions:\n\n1. The paper defines under-search as a factual or logical error in a non-search step. This may be a useful proxy for hallucination, but I wonder how the proposed method behaves with the other under-search situations, such as suboptimal or incomplete knowledge. If I understand it correctly, the current detector, which only checks for correctness, would not flag this as an under-search and would incorrectly reward it as an \"optimal\" non-search step.\n\n2. Can you provide more details on when the reward collapse occurs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is, in general, well written with good structure and is easy to follow.\n\n2. The hierarchical process reward formulation well balances correctness with efficiency through a gated bonus structure, which avoids the classic RL issues of over-penalizing necessary search.\n\n3. In this paper, the authors conducted various design choices with a lot of experiments with multiple models (Qwen, Llama), sizes (3B, 7B), types (base, instruct), and RL algorithms (PPO, GRPO). And the proposed method achieves good results compared to other baselines across several datasets.\n\n4. The proposed method can significantly reduce the over-search rate, which is very useful and impressive.", "weaknesses": "1. The proposed reward mechanism relies on \"on-the-fly\" calls to external LLM judges to detect over-search and under-search during the RL training loop, which could potentially add significant computational overhead, cost, and API call latency to every training step. \n\n2. A further question to this reliance is that the LLM models may behave differently over time. Can we still keep the consistency of the proposed method?\n\n3. The method assumes the judge is accurate. If the judge mislabels a step (false positive over-search or false negative under-search), the agent may be clipped incorrectly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3FPstKT3tO", "reviewer": "Reviewer_5auK", "summary": "This paper proposes HiPRAG, a RL framework for agentic RAG that introduces hierarchical process rewards to guide both accuracy and search efficiency. Instead of relying solely on outcome-based rewards, HiPRAG decomposes the trajectory into discrete steps and introduces step-level rewards to penalize over-searching and under-searching. The framework employs external LLMs to detect these inefficiencies on-the-fly and incorporates the results into a hierarchical reward formulation.", "questions": "1. How is the “Training with Over-search or Under-search Only” setting implemented? Are the ignored types treated as optimal steps in the reward computation?\n2. The ablation study section enumerates many results without clear structure; summarizing the main findings would improve readability and help highlight key insights.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The hierarchical reward structure provides a novel and interpretable mechanism for optimizing process-level efficiency rather than only final-answer correctness.\n2. Unlike previous works on faithful process supervision, this paper specifically targets over-searching and under-searching issues in multi-hop retrieval, and presents a well-motivated solution.\n3. The proposed method achieves consistent improvements across different model sizes (3B, 7B) and RL algorithms (PPO, GRPO), with significantly reduced over- and under-search rates.", "weaknesses": "1. The over-/under-search detection relies on commercial LLMs (GPT-4.1-mini, GPT-5-mini) as external judges, which raises concerns about stability, cost, and reproducibility. The impact of judgment errors is not analyzed.\n2. The on-the-fly verification by external LLMs adds training complexity, yet the paper provides no analysis of computational cost, runtime, or GPU-hour consumption.\n3. The role of $/lambda_p$ is unclear. Since the process reward is already gated by answer and format correctness (i.e., a hierarchical design), it should not directly influence final answer quality. Why does increasing $/lambda_p$ to 0.6 “over-prioritize step purity at the expense of answer correctness”? This suggests that λₚ may not be a meaningful parameter. Please clarify its actual effect on training dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pIV5Bacwq1", "reviewer": "Reviewer_5yKr", "summary": "This paper focuses on the over-searching or under-searching phenomena in Retrieval-Augmented Generation, which are critical for both effectiveness and efficiency. The authors propose HiPRAG, a hierarchical reward function that supervises search or non-search decisions at each step through reinforcement learning. Specifically, this reward function comprises two key components: an over-search detection module and an under-search detection module, both implemented using two external LLMs. Extensive experiments across 7 QA datasets demonstrate the effectiveness of HiPRAG.", "questions": "1. Could you provide comparisons with existing efficiency-aware RAG methods (e.g., adaptive retrieval, mechanisms)? How does HiPRAG perform relative to these approaches in terms of both effectiveness and efficiency?\n\n2. How sensitive is HiPRAG to the choice of reward LLMs?\n\n3. What is the performance degradation when using smaller/weaker LLMs for reward generation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed HiPRAG is well-motivated, effectively addressing the over-searching or under-searching issues in RAG. This reward function enhances the original outcome reward by incorporating the over-search and the under-search detection modules.\n\n\n2. This paper provides comprehensive ablation studies examining various aspects of the proposed method, including the impact of process bonus coefficients and different RL algorithm, which offer valuable insights into HiPRAG's mechanisms.", "weaknesses": "1. The experimental evaluation lacks comparisons with the efficiency-aware RAG methods. While this paper is not the first work to consider the over-searching issue, it fails to discuss or compare with relevant prior works. Consequently, all baselines in the experiments are limited to standard R1-like RAG methods.\n\n\n2. Although the paper aimes to improve efficiency in RAG, it lacks crucial efficiency metrics such as the average number of search steps or inference time per question.\n\n\n3. Both over-search and under-search detection modules rely on the LLM-as-judge method, which may not be reliable, particularly when using the smaller LLMs. Therefore, GPT-4.1-mini and GPT-5-mini are used for the reward quality. It is worth further discussion on the influence of the different reward LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IePnSj6Xl6", "reviewer": "Reviewer_DZXL", "summary": "The main contribution of this work is HiPRAG, a novel Reinforcement Learning training methodology that incorporates a fine-grained, knowledge-aware \"Hierarchical Process Reward\". This approach is achieved by first decomposing the agent's reasoning trajectory into discrete, parsable <step> blocks to distinguish between search and non-search steps ; it then evaluates the necessity of each search decision \"on-the-fly\" during training, using external LLM judges to identify \"over-search\" and \"under-search\" behaviors ; finally, it applies a hierarchical reward function that provides an additional \"gated process bonus\" based on the proportion of optimal steps, which is added only when both the final answer and the format are correct. Experiments show HiPRAG achieves 67.2% average accuracy (7B), outperforming strong baselines , while dramatically improving efficiency by reducing the over-search rate from over 27% to just 2.3%.", "questions": "1.Why did the authors not provide an ablation study for λf ? How was the value of 0.2 determined? How sensitive are the model's final performance and convergence stability to the choice of λf ?\n\n2.The knowledge source used for the experiments is the 2018 Wikipedia dump , while the judge model for 'under-search' detection is gpt-5-mini, which possesses 2025-era knowledge. If the policy model provides a non-search answer that was factually correct based on the 2018 knowledge base,but is outdated according to the 2025 judge's knowledge, will the judge incorrectly flag this step as an 'under-search'? Could this \"temporal misalignment\" between the retrieval knowledge source and the judge's internal knowledge base introduce significant noisy rewards, thereby contaminating the training process?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.The quality of the experimental methodology is high. The paper's conclusions are based on extensive experiments across seven QA benchmarks, multiple model families (Qwen, Llama), and multiple RL algorithms (PPO, GRPO). The ablation studies in Section 5.3 are exhaustive and provide strong support for HiPRAG's key design choices.\n\n2.A key finding is that the HiPRAG model trained based on Qwen2.5-3B-Instruct + GRPO not only surpasses strong external 7B baselines like R1-Searcher++, but it also outperforms the 7B counterpart trained with the authors' baseline reward . This indicates that the authors' training methodology is a more effective path to performance gains than simply scaling the model size with conventional rewards, which has significant implications for resource-constrained research and deployment.\n\n3.The paper's presentation is very clear and well-organized, making a complex methodology easy to understand. The figures in the paper are of high quality and greatly enhance understanding.", "weaknesses": "The Over-search Detection method  requires the policy model to perform a second, complete generation. This \"re-generation step\" effectively doubles the inference cost for each search step during the RL rollout phase. Although the authors mention that \"the re-generation step can be executed separately through batch generation\", this still represents significant training overhead. A discussion on the impact of this training method on the total training time would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)