ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
Authors: Mingyang Chen
Linzhuang Sun
Tianpeng Li
sunhaoze
ZhouYijie
Chenzheng Zhu
Haofen Wang
Jeff Z. Pan
Wen Zhang
Huajun Chen
Fan Yang
Zenan Zhou
Weipeng Chen
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Reinforcement Learning
Chain-of-Thought Reasoning
Retrieval-Augmented Generation
Multi-hop Question Answering
Primary area: applications
TL;DR: Training LLMs to combine reasoning with external knowledge retrieval via RL without any supervised data on reasoning steps.

Abstract: Large Language Models (LLMs) have shown remarkable capabilities in reasoning, exemplified by the success of OpenAI-o1 and DeepSeek-R1. However, integrating reasoning with external search processes remains challenging, especially for complex multi-hop questions requiring multiple retrieval steps. We propose ReSearch, a novel framework that trains LLMs to Reason with Search via reinforcement learning without using any supervised data on reasoning steps. Our approach treats search operations as integral components of the reasoning chain, where when and how to perform searches is guided by text-based thinking, and search results subsequently influence further reasoning. We train ReSearch on Qwen2.5-7B(-Instruct) and Qwen2.5-32B(-Instruct) models and conduct extensive experiments. Despite being trained on only one dataset, our models demonstrate strong generalizability across various benchmarks. Analysis reveals that ReSearch naturally elicits advanced reasoning capabilities such as reflection and self-correction during the reinforcement learning process.

ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
Mingyang Chen', Linzhuang Sun?, Tianpeng Li', Haoze Sun', Yijie Zhou', Chenzheng Zhu', Haofen Wang", Jeff Z. Pan', Wen Zhang,, Huajun Chen', Fan Yang",, Zenan Zhou', Weipeng Chen
Baichuan Inc. University of Chinese Academy of Sciences
Tongji University "The University Edinburgh 'Zhejiang University
{chenmingyang, yangfan) whaneaaaaicchiiiiai chuan- inc. com https://gi higggttthbb..... com Agent-RL/RRReearcch RL/ReSearch
Abstract
Large Language Models (LLMs) have shown remarkable capabilities in reasoning, exemplified by the success of OpenAI-ol and DeepSeek-RI. However, integrating reasoning with external search processes remains challenging, especially for com- plex multi-hop questions requiring multiple retrieval steps. We propose ReSearch, a novel framework that trains LLMs to Reason with Search via reinforcement learning without using any supervised data on reasoning steps. Our approach treats search operations integral components of the reasoning chain, where when and how to perform searches is guided by text-based thinking, and search results subse- quently influence further reasoning. We train ReSearch on (ive2.5-7B(-Instruu and Swven.5-33BB-Innttree
models and conduct extensive experiments. Despite being trained on only one dataset, our models demonstrate strong generalizabil- ity across various benchmarks. Analysis reveals that ReSearch naturally elicits advanced reasoning capabilities such reflection and self self-correction during the reinforcement learning process.
Introduction
In recent years, Large Language Models (LLMs) have demonstrated remarkable performance across wide array of tasks [1, 5, 13, 39]. Beyond leveraging internal knowledge acquired during pre- training, LLMs exhibit the capability to utilize external tools, particularly search engines, to retrieve factual and time-sensitive information, thereby mitigating instances of hallucination [4, 14, 21, 25]. This capability, often referred to as Retrieval-Augmenne Generation (RAG), has been the subject of extensive investigation in recent literature [2. 6, 38, 42]. Despite the effectiveness of RAG, designing robust multi-step RAG strategies applicable to complex real-world problems remains significant challenge. This particularly crucial, as many real-world issues are inherently complex and necessitate several steps of reasoning [23, 33, 35].
The past year has witnessed considerable advancements in LLMs' reasoning abilities, particularly through chain- reasoning before producing final outputs [37, 41]. This progress is exemplified by the success OpenAI-ol [17], and DeepSeck-RI [5]. These developments emphasize the importance of test-time scaling in reasoning, enabling LLMs to decompose intricate problems into manageable intermediate steps [16, 28). This reasoning capacity is also vital for the efficacy of RAG, especially when addressing complex questions that require multiple retrieval steps. Nonetheless, training LLMs
"Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

60
Ba- Re--RatGen
Nav Genaniiion
Nave RAG
$2.2 R2
50
40-9
$8.1
LLM-as-a-Judge a-Judgeeeeee (%)
40
26-2
93.4
30
20
31.9
50.5
20.5
0
20
1G.1
14.2
120
10
0
HolpalQA
2Wiki
MuSiQue
Bamboogle
Comparative performance of ReSearch and baseline methods on benchmark datasets. All baselines are built upon Qwen2.5-32B- -nntttttuut See Section 3 for details.
Figure 1:
conduct interactive reasoning alongside information retrieval continues to present open challenge for the research community. Most existing approaches to multi-step RAG rely on manually designed prompts or heuristics, which are not only labor-intensive but also lack scalability for more intricate problems [19, 23, 33J. Additionally, labeling reasoning steps in a multi-step RAG framework is often impractical due the associated costs and time constraints.
Reinforcement learning (RL) has emerged as a promising avenue for enhancing reasoning capabilities without the need for supervised data regarding reasoning steps [5, 24]. This approach holds potential for training LLMs to exhibit reasoning skills solely based on simple reward signals derived from final outcomes. Recent advancements in RL-based training for LLMs have demonstrated significant improvements in complex reasoning tasks, where models learn to decompose problems into man- ageable steps through trial and eтror rather than explicit instruction. Models such DeepSeek-R have shown that rule-based reward functions can effectively guide LLMs to develop sophisticated reasoning patterns autonomously. Despite these successes, current approaches primarily focus on enhancing internal reasoning capabilities, with limited exploration of how to effectively combine this reasoning process with external knowledge retrieval.
In this paper. we propose a novel framework for training LLMs to Reason with Search via re- inforcement learning, which term ReSearch. The reasoning chain in this framework is not only composed of text-based thinking (i.e., enclosed by <think </think>) as DeepSeek-R1, 1, but also search query (i.e., enclosed by <search> </search>) and retrieval results (i.e., enclosed by <result> </result>). We treat the search operation as part of the chain-like reasoning process, and the search operation will interact with text-based thinking. Specifically, when and how to perform search will be steered by previous text-based thinking and the search results will infuence subsequent text-based thinking. In the framework, we don't provide any supervised data reasoning steps for LLMs to imitate, instead, we leverage reinforcement learning (i.e., GRPO) to incentivize LLMs to perform reasoning with search.
We train ReSearch from scratch on Qwen2.5-7Br (B-sss--ttt and (ren2.5-32B(-nnttrret) and conduct extensive experiments multi-hop question answering benchmarks that need multi-step reasoning and multiple information retrieval. Our trained models show significant absolute improvements range from 8.9% to 22.4% over the baselines, shown in Figure 1. Furthermore, our training is only conducted on one specific training set, and trained models are evaluated on multiple benchmarks, showing the generalizability of our framework. Our contributions
follows:
* By emphasizing the interaction between reasoning and search, we propose novel framework ReSearch that using reinforcement learning to train L.MMs to reason with search from scratch, without any supervised data reasoning steps.

---

We train ReSearch on different scales of models, and conduct extensive experiments on multi-hop question answering benchmarks, showing the effectiveness of this framework. The trained models show significant generalizability and potential for more realistic scenarios.
By analyzing the training process, demonstrate that ReSearch effectively elicit reasoning capabilities with search progressively itself, and that reasoning abilities such as reflection and self-correction can be incentivized without relying any pre-defined heuristics.
Method
Drawing inspiration from the success of OpenAl-ol and DeepSeek-RI in learning to reason, incorporate search operation into the reasoning process and train LLMs from scratch using reinforce- ment learning (i.e., GRPO) without any labeled data on reasoning chains, making LLMs learn Reason with Search SReScar.h). In this section, we first show the details of training ReSearch, dive into the details of the GRPO and how to conduct rollout with search during reinforcement learning ($2.1). Then, we demonstrate the prompt template design directing the LLMs to generate the defined format of rollout ($2.2), and finally, we introduce the reward modeling for guiding the optimization of reinforcement learning ($2.3).
y/1
Y1
A
Reference Model
LLM Policy
1/2
1'2
A
Group Computation
Search La
Question
Reward Calculation
YG
Aq
Rollout
Reward
Adventage
(a)
cresult: Clibbek founded in 1812 es the </resul t>
Irpput Question
2
<thinko To answer this question </thinks
<search> When was Citibank. ... </search>
cresult> Cltbbrk
founded </result>
Who United the year that Crtiban was founded??
C3
[Q Search
<search- Who was </search- <resultx President </result>
<thinlo Now, need to find out </think>
LLM Pollcy
the United States
the year that Citibank was foundeed was \ lboeedjJames Madison) \ < c/answer>
cthinko To answer this
Generate: till search
question.
ny searching for the founding year of Clibbak. </ /ttinkk esearchs founded </search>
Retrieve search result
Concat then continue
Output Rollout
Figure 2: The training overview of ReSearch. (a) The GRPO pipeline. (b) The details of the rollout generation process.
2.1 Reinforcement Learning
When handling complex multi-step tasks needing retrieval (i.e., multi-step RAG), reasoning is crucial for steering multiple retrieval (i.e., search) operations, mainly on when and how to perform search. It's challenging collect labeled reasoning data with search for supervised fine-tuning LLMs imitate how to reason with search. Fortunately, reinforcement learning has shown impressive performance in training LLMs to conduct reasoning, which can elicit reasoning capabilities from LLMs without any supervised data. In general, the main idea behind reinforcement learning here to sample multiple reasoning-winn wwihiinggwwth-harrhhh chains (i.e., rollouts) and optimize the policy (i.e., LLMs) maximize the probability of generating rollouts with higher rewards, as described in Figure
Group Relative Policy Optimization Specifically, this work, we use Group Relative Policy Optimization (GRPO) the learning algorithm, which estimate the baseline from group of rollouts

---

instead of training a separate critic model in Proximal Policy Optimization (PPO). Given an existing policy Thw and an reference policy Fha base on G rollouts {yr [8 Te (+2z) for each input D, the objective of GRPO is to optimize the policy To by maximizing the following objective:
[(0) BgwD,,
No (-1x)
1
G
To (x) Now (Ualzr) (z))
To xx)
(1)
min
A,, clip
1 e, + A: -19DKI. (Tol (Tol|
G
Ta (V: [z)
where A, (rr- mean( (ry (11) /std( (r, (ry] 1) is the normalized advantage of the i-th rollout in current group, is the clipping ratio, and / the KL loss coefficient. Moreover, KL divergence penalty is added to the objective to prevent the policy from deviating too much from the original reference policy LLMs. The illustration of GRPO is shown in Figure 2(a).
Rollout with Search Compared with conventional rollout that only contains text-based thinking as reasoning, the rollout in ReSearch also contains search queries and retrieval results. We use <search> and </sear to enclose the search queries and <result> and </result> to enclose the retrieval results, and such instruction is described in the prompt templates, which will be introduced later in $2.2. The rollout process is an iterative process between text-based thinking, search queries, and retrieval results as described in Figure 2(b). Specifically, when the generation process encounters </search> tag, the query between the last <search> and current </search> tags will be used as the search query to retrieve relevant factual information, and the retrieval results will be enclosed by <result> and </result> tags. Then, existing rollout concated with the retrieval results will be used as the next input to generate following response iteratively, until the generation encounters end-of-sentence (cos) tag (i.c., <endoftext> or <im andeend>>in in Qwen-2.5 Models).
Retrieval Result Masking In original GRPO, the loss is calculated by all the generated tokens in the whole rollout. However, in ReSearch, the rollout contains retrieval results, which are not generated by the training policy, but retrieved by the search environment. Therefore, we mask the retrieval results in the loss calculation to avoid the training policy from being biased towards the retrieval results. That is, during the computation of Equation 1, we only consider the tokens in the text-based thinking and the search queries, and ignore the tokens in the retrieval results.
2.2 Training Template
Since we orchestrate the rollout process by identifying our defined special tags (c.g., stopping at </search> and transferring control to the search environment), it is crucial for policy LLMs to gener- ate output in the defined format. To guide the LLMs in understanding this rollout format cermat-specfffallly the tags indicating when the search operation is invoked we created two prompt templates: one for the base (i.e., pre-trained) model and another for the instruction-tuned model. As shown in Table 1, inspired by DeepSeek-R1. R1, these templates are designed to be simple and concise, ensuring that the model can act natural progression during the reinforcement learning process. Specifically, for the base model, this template, filled with specific user question, will be used as direct input to the LLMs. For the instruction tuned model, its prompt template serves as the system prompt, utilized in conjunction with the corresponding chat template of the instruction-tuned LLM.
2.3 Reward Modeling
During reinforcement learning of ReSearch, there is no supervised reasoning data, and we only use simple reward on rollouts to guide the optimization of LLMss. Experimentally. only rule-based reward function is enough to successfully elicit capabilities of reasoning with search for LLMs. Our reward function considers following two parts: answer reward and format reward.
Answer Reward:
We calculate the correctness of the final answer in \boxed() and the ground truth answer via Fl score.
Format Reward: We check whether the rollout correctly follows our defined format as described in the prompt templates, mainly checking the correctness of tags and existence of [boxed{) the answer.

---

Table 1: Prompt templates for training from base model and instruction tuned model. For the base model, prompt will be replaced with the actual question. For the instruction-tunee model, this template is used as the system prompt.
Prompt Template For Base Model
A conversation between User and Assistant. The user asks a question, and the assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. During thinking, the assistant can invoke the wikipedia search tool to search for fact information about specific topics if needed. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags respectively, and the search query and result are enclosed within <search> </search> and <result> </result> tags respectively. For example, <think> This is the reasoning process. </think> <search> search query here </search> <result> search result here </result> <think> This is the reasoning process. </think> <answer> The final answer is (boxed(answer (answer here] </ //answer>. the last part of the answer, the final exact answer is enclosed within \boxed[] with latex format. User: prompt. Assistant:
System Prompt Template For Instruction Tuned Model
You are helpful assistant that can solve the given question step by step with the help of the wikipedia search tool. Given a question, you need to first think about the reasoning process in the mind and then provide the answer. During thinking, you can invoke the wikipedia search tool to search for fact information about specific topics if needed. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags respectively, and the search query and result are enclosed within <search> </search> and <result> </result> tags respectively. For example, <think> This is the reasoning process. </think> <search: ch> search query here </search> <result> search result here </result> <think> This is the reasoning process. </think> <answer> The final answer is (boxed(answer here) </ answer>, In the last part of the answer, the final exact answer is enclosed within (boxed{) with latex format.
Specifically, for the final reward of a rollout:
fl pped, ag), if f1 score is not O
0.1,
score 0 and format correct score is 0 and format incorrect
(2)
0,
where Opeed is the final answer in (boxed and agt is the ground truth answer, and fl (Apred agi) is the F1 score between Gpred and agt
3 Experiments
3.1 Experiment Setup
To evaluate the effectiveness of ReSearch, we conduct extensive experiments mainly on multi-hop question answering benchmarks that need multi-step reasoning and multiple information retrieval. Our ReSearch is trained from Qwen2.5-7B, Qwen2.5-7B B--n2.557B-Inssee Instruct, owen225-322B and Qwen2.5-32B Instruct [39]. During training, we only use data from training set of MuSiQue [32], since has various types of multi-hop questions and constructed via fine-grained quality control.
Benchmarks We use four standard benchmarks on multi-hop question answering tasks, including HotpotQA [40], 2WikiMultiHopQA [7], MuSiQue [32], and Bamboogle [19]. Specifically, Hot- potQA, WikiiMultiHooQA and MuSiQue are constructed among wikipedia or wikidata [34], via different multi-hop hop mining strategies with crowd-sourcing, while Bamboogle is manually constructed dataset with 2-hop questions, where all questions are sufficiently difficult to be unanswerable by popular internet search engine. Our evaluation is conducted on the full dev set of HotpotQA, 2WikiMultiHopAA, and MuSiQue, and the test set Bamboogle, including 7405, 12576, 2417, 125 samples respectively, Note that discard the context documents from the original datasets for
5

---

HotpotQA, 2WikiMuliHoopAA and MuSiQue, and only use the question and answer pairs for evalu- ation. We use an open-ended retrieval environment based on wikipedia to retrieve the background knowledge for all the datasets, which we introduce later.
Baselines We first compare ReSearch with two naive baselines: (1) No RAG: Use corresponding instruction-tuned model to generate answer directly without any RAG, and (2) Naive RAG: A naive retrieval-based setting that concatenate the retrieval results with question and then generate answer directly. Furthermore, we also consider two approaches focusing on improving multi-step RAG: (3) Iter-RerGen [23]: method synergizes retrieval and generation an iterative manner, and (4) IRCoT [33]: An iterleaving method, which use retrieval and the thain-of-hue chain-of-thought (CoT) guide each other. Since these methods are prompt-basedd we use iustruction-tune' models in same size as our ReSearch to implement them for fair comparison.
Evaluation Metrics For evaluate the correctness of the final answer, first Exact Match (EM) where the prediction is correct if it matches the ground truth answer exactly. However, such exact match is too strict for our setting, since the retrieval environment is open-ended and the result is described by natural language. Therefore, we also consider LLM--as---judee (LJ) for automatic evaluation, where we use gpt-4o-mini with our defined judge prompt to score the correctness of the final answer. Such judge prompt is shown in Appendix A.
Implementation Details We conduct our training and evaluation Qwen2.5-7B. Qwen2.5-7B: Instruct, Qwen2.5-32B and Qwen2.5-322 3--32B-Innttret..... The reinforcement learning framework is built on verl [26]. We only use the training set (19938 samples) of MuSiQue for training, and the number of training epochs is 2. The retrieval environment is based on FlashRAG [10]. standard toolkit for RAG research. We use E5-base-v2 [36] the retriever and Wikipedia data from Dec. 2018 as the knowledge base [11]. All the corpus indexing and embedding has been preprocessed by FlashR AG. During the rollout in training and evaluation, we retrieve top-5 results for each query. For baseline methods, we use the implementation from FlashRAG. For details about model training, please refer to Appendix B.
3.2 Main Results
The main results of baselines and ReSearch are demonstrated in Table 2, and we show the methods based on LLMs with different sizes respectively. From the main results, we can draw the following observations:
Effectiveness of ReSearch Compared with all the baselines, ReSearch achieves significant im- provements on all the benchmarks, which demonstrates the effectiveness of our proposed framework. Specifically, among all the benchmarks, the average improvement of ReSearch over the best baseline 15.81 % in exact match and 17.56 in LLLM-as-a-judee for Qwen2.5 model with 7B parameters. For Qwen2.5 model with 32B parameters, the average improvement is 14.82% % in exact match and 15.46 % LLLM-as-ajjudge.
Comparison between base and instruction-tumed models We train ReSearch from both base and instruction-tunee models with 7B and 32B parameters respectively, and note that they are all trained using reinforcement learning from scratch without any supervised fine-tuning. From the results, we can observe training from the instruction-tun tuned model can further improve the performance of ReSearch. Such observation is consistent among all the benchmarks and model sizes.
Generalization Ability During reinforcement learning, ReSearch learns the ability of reasoning with search, which is independent of specific knowledge or multi-hop patterns, and such ability is generalizable. Our model ReSearch is only trained on the training set of MuSiQue dataset, but from the results, we can observe that it can generalize to other benchmarks with different question types and structures, which demonstrates the generalization ability of ReSearch.
3.3 Further Analysis
We investigate the important metrics during training ReSearch in this section. Specifically, the response length and number of search operations during training are shown in Figure 3 respectively.

---

Table 2: Exact Match (EM, %) and LLM-as-a-Judge (LJ, %) results on multi-hop question answering benchmarks. The best results are highlighted in bold, and the best results across baselines are underlined.
HotpotQA
2Wiki
MuSiQue
Bamboogle
Model
EM
EM
LJ
EM
EM
LJ
Qvenn.5-7B(-Instrucc)
Naive Generation
19.18
30.64
25.76
27.87
3.76
10.38
10.40
22.40
Naive RAG
31.90
49.59
25.78
29.52
6.21
12.78
20.80
32.00
Iter-RetGen
34.36
52.22
27.92
31.86
8.69
16.14
21.60
35.20
IRCoT
30.33
52.06
21.57
30.65
6.99
14.19
24.80
36.80
ReSearch-Qwen-78
40.57
60.26
44.67
50.06
21.68
32.19
43.20
54,40
ReSearch-Qwen-177 -i--ween---nnstrue
43.52
63.62
47.59
54,22
22.30
33.43
42.40
54.40
(wen2.5-32B(Instee
Naive Generation
24.63
38.26
27.23
29.68
6.12
14.23
18.40
29.60
Naive RAG
36.46
55.73
30.38
34.87
9.27
15.97
23.20
40.80
Iter-RetGien
39.81
58.80
33.64
38.22
12.49
20.11
29.60
44.80
IRCoT
28.44
55.44
13.53
29.50
7.82
18.20
31,20
47.20
ReSearch--wwn-322
42.77
64.27
38.52
45,59
26.40
37.57
54.40
66.40
ReSearch-Qwen-322 -tstttru
46.73
67.70
44.90
50.30
26.40
38.56
56.80
67.20
The curve of training reward and validation reward are shown Figure The validation conducted on a part of development set of MuSiQue dataset with 100 random samples, and conducted every 10 steps during training.
Research Owco
Iaannct
Nedearch Owen-78
Rediourch Cwen-125-hhsruut
ReSearch- Qaaaart---
300
1.5
320
163
40
100
Training
1%
100
12A
Training
120
Figure 3: Response length and number of search operations during training.
ReSoarch -wenn- Morur-88issnon Biistrut
NuS carch Owen-78
RuSccarh Qwenn Insruct
ReSearch Qwen- 325
6:40
0.2
Tradning Stop
Traising Stop
Validation Stop
Valilatinn Stop
Figure 4: Training and validation reward during training.
Response Length We define the response length as the total number of tokens in model's output, excluding retrieval results, which can be interpreted the test-time cost of reasoning. From the first two figures in Figure 3, it is evident that the response length generally increases throughout the training process. Instruction- tuned models exhibit longer response lengths compared base models for both the 7B and 32B parameters. Additionally, for the 32B models, the response length initially decreases during the first 20 training steps before increasing again after approximately the 60th step. This observed behavior may be attributed to the 32B models having more inherent knowledge, leading

---

them to generate longer responses initially, and do not utilize retrieval. However, after receiving some guidance from reward signals early in the training, they begin to leverage searches more frequently. thereby reducing their reliance on generated knowledge.
Number of Search Operations We also count the average number of search operations in one rollout during training, which is shown in the last two figures in Figure 3. As depicted in the figures, the number of search operations shows consistent growth throughout the training process. Such trend indicates that for complex multi-hop questions, the model gradually learns to use search to retrieve relevant information iteratively multiple times.
Training and Validation Reward We show the training and validation reward during reinforcement learning of ReSearch in Figure 4. The reward increases sharply during the first 20 training steps, and then gradually increases. The cold-start reward is higher for instruction-tun sutood models for both 7B and 32B models. For 7B models, the rewards converge to similar level for both base and instruction- -unnd models, while for 32B models, the training reward of instruction-tuned models is higher than that of base models.
3.4 Case Study
Table 3: case study of ReSearch during training.
Question:
Who won the 2018 presidential election in the country where the political party of Martín Ramírez Pech operates? Answer:
Andrés Manuel López Obrador
<think> To find the answer, I first need to determine the country in which Martín Ramírez Pech's political party operates. will search for "Martín Ramírez Pech" to find out this information. </think>
<search> Martín Ramírez Pech < </search>
<result> "Martín Ramírez", Martín Ramírez Martín Ramírez (January 30, 1895 February 17, 1963) was ... |n "Martín Ramírez", and $134,500 in New York in 2011 1. Martín Ramírez Martín Ramírez (January 30, 1895
\n "Martín Ramírez", including brown paper bags, </result>
scraps of examining-tabb table paper, and book pages
<think> appears that made mistake in my search query. Instead searching for "Martín Ramírez Pech", the results are showing information about different "Martín Ramírez". To find out the right information, need to look for Martín Ramírez Pech specifically and the political party associated with him. I will modify the search query to "Martín Ramírez Pech political party" for the correct information. </think>
<search> Martín Ramírez Pech political party </search>
<result> "Cuauhtémoc Blanco", " of Morelos. In the 2015 legislative elections, he won in a closely contested election ... ... in "Labor Party (Mexico)", for PT and 75 deputies and 16 \n "Graco Ramírez". Graco Ramírez Graco Luis Ramírez Garrido Abreu (born 26 June 1949, in Villahermosa, Tabasco) is Mexican </result>
senators for the PES. The alliance has received
<think> Now, we see that Martín Ramírez Pech is associated with the political party known as the Labor Party (PT) in Mexico. I now need to continue this line of reasoning to find out who won the presidential election in 2018 in Mexico. </think>
<search> 2018 Mexican presidential election < </search>
<result> "2018 Mexican general election". (INE) officially declared the new process underway on 8 September 2017. The presidential election was won by Andrés Manuel López Obrador of |n "2018 Mexican general election". irregularities including reports of vote-buying. spending beyond legal campaign general election General elections were held in Mexico on July 2018 </result> <think> According to the search results, the 2018 Mexican presidential election was won by Andrés Manuel López Obrador as the candidate for the National Regeneration Movement (MORENA). </think>
In "2018 Mexican general election", 2018 Mexican
<answer> The final answer (bored(Andrés Manuel Lôpez Obrador) </answer>
8

---

To give more intuitive understanding of the utility of ReSearch, we show case study in Table 3. This case from the reinforcement learning process of Qwen2.5-32B-img Insttrctt model. Where text enclosed by <think> and </think>, <search> and </searcht ch>, and <answer> and </answer> are generated by the model, and the text enclosed by <result> and </result> are retrieved from the retrieval environment. For clarity, we use
to represent the truncation of the retrieval results. From this case, we can see that the model can effectively break down the complex question and conduct reasoning within <think> and </think>. Such reasoning process is crucial for guiding when and what to search, and leading to the final answer in a multi-step manner.
Self-elicited Reflection In addition, also observe reflection phenomenon in the model's response. As depicted in the second thinking step in Table 3, the model states, "I made a mistake", recognizing that the previous search query failed to retrieve useful information. It then corrects itself in the third thinking step by generating a more effective search query to obtain the relevant information. Note that such reflection ability is not explicitly trained or designed in the prompt templates, but is naturally elicited from the model itself during reinforcement learning.
Related Work
4.1 Reinforcement Learning with LLMs
Reinforcement learning [30], which aims to maximize the expected return of an agent's policy through interactions with the environment, has emerged as a crucial technique for LLMs, from aligning with human values to enhancing reasoning capabilities. A significant development was Reinforcement Learning from Human Feedback (RLHF) [18], which uses Proximal Policy Optimization (PPO) [22] with reward models trained on human preferences. Several methods have since improved upon PPO. including Direct Preference Optimization (DPO) [20], Simulated Preference Optimization (SimPO) [15], and Group Relative Policy Optimization (GRPO) [24]. Recently, reinforcement learning has demonstrated remarkable success in enhancing reasoning capabilities, as evidenced by notable achievements such as OpenAl-ol [17], DeepSeek-R1 [5], and Kimi k1.5 [31]. However, the reasoning capabilities of LLMs under RAG settings remain largely unexplored. Meanwhile, several concurrent works have also begun to investigate reinforcement learning for enhancing LLM reasoning with tool use [9, 29].
4.2 Retrieval-Augmente Generation for LLMs
RAG techniques [8, 12] augment LLMs by retrieving external knowledge and incorporating it into the generation process. Extensive research has been conducted this area, encompassing various aspects such as retriever optimization [27, 38], query refinement [3], and self-reflection mechanisms [2]. For more complex scenarios, particularly in addressing multi-hop questions, iterative RAG models [23, 33) have been developed that alternately perform retrieval-enhanced generation and generation- enhanced retrieval. Additionally, supervised learning approaches have been explored using annotated trajectories of multi-step retrieval [21, 35]. Despite these advances, the application of reinforcement learning to enhance reasoning capabilities within RAG-style tool-augmented environments remains an open and under-explored research direction.
5 Conclusion
In this paper, we introduced ReSearch, a novel framework that trains LLMS to reason with search via reinforcement learning without requiring any supervised data reasoning steps. Our approach integrates search operations as integral components of the reasoning chain, where text-based think- ing guides when and how to perform searches, and search results subsequently influence further reasoning. Through extensive experiments on multiple multi-hop question answering benchmarks, we demonstrated that ReSearch achieves significant improvements over baseline methods. The results also indicate the framework's 's potential for more realistic scenarios. Analysis of the training process revealed that ReSearch naturally elicits advanced reasoning capabilities such as reflection and self-correction, without relying on pre-defined heuristics. This work highlights the effectiveness of integrating reasoning and search operations through reinforcement learning, offering a promising direction for developing more capable and reliable L.LM-based systems for complex multi-hop tasks.

---

References
[1] Anthropic. Claude 3.7 sonnet and claude code, 2025. URL news/claude- 3-
com/
[2] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In ICLR. OpenReview.net, 2024.
[3] Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue. Yike Guo, and Jie Fu. RQ-RAG: learning to refine queries for retrieval augmented generation. CoRR, abs/2404.00611 2024.
[4] Mingyang Chen, Haoze Sun, Tianpeng Li, Fan Yang, Hao Liang, Keer Lu, Bin Cui, Wentao Zhang, Zenan Zhou, and Weipeng Chen. Facilitating multi-ture function calling for llms via compositional instruction tuning. CoRR, abs/2410. 12952, 2024.
[5] DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjic Ji, Erhang Li, Fangyun Lin. Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo. Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability 1lms via reinforcement learning. CoRR, abs/2501. 12948, 2025.
[6] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-augmeetee generation for large language models: A survey. CoRR, abs/2312.10997,, 2023.
[7] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing A multi-hop QA dataset for comprehensive evaluation of reasoning steps. In COLING, pages 6609-6625. International Committee on Computational Linguistics, 2020.
[8] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.0322 1(2):4, 2022.
[9] Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- rl: Training Ilms to reason and leverage search engines with reinforcement learning. CoRR, hix/2503.095111 2025.
[10] Jiajie Jin, Yutao Zhu, Xinyu Yang, Chenghao Zhang, and Zhicheng Dou. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. CoRR, abs/2405. 13576, 2024.
[11] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pages 6769-6781. Association for Computational Linguistics, 2020.
[12] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Reureetal-augmented generation for knowledge-intent integgsvveeeteee nlp tasks. Advances neural information processing systems, 33:9459-9413 2020.
10

---

[13] Mingan Lin, Fan Yang, Yanjun Shen, Haoze Sun, Tianpeng Li, Tao Zhang, Chenzheng Zhu, Tao Zhang, Miao Zheng. Xu Li, Yijie Zhou, Mingyang Chen, Yanzhao Qin, Youquan Li, Hao Liang, Fei Li, Yadong Li, Mang Wang, Guosheng Dong, Kun Fang, Jianhua Xu, Bin Cui, Wentao Zhang, Zenan Zhou, and Weipeng Chen. Baichuan alignment technical report. CoRR, 0bs/2410.1194444 2024.
[14] Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn LLM agents. NeurIPS, 2024.
[15] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-12422 2024.
[16] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel J. Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. CoRR, abs/2501.19393, 2025.
[17] OpenAI. Learning to reason with LLMs, 2024. URL https://openai.com comtaai...ooiiiiiaaaaaa commindex// learning-to- reasor- wwwson-with-lmmmm
[18] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[19] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap language models. In EMNLP (Findings), pages 5687-5711. Association for Computational Linguistics, 2023.
[20] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NewrIPS, 2023.
[21] Timo Schick, Jane Dwivedi-Yos Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In NeurlPS, 2023.
[22] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707. 7074477777 2017.
[23] Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing aetrieeal-augmented large language models with iterative retrieval-generation synergy. EMNLP (Findings), pages 9248-9274 -974.99444 Association for Computational Linguistics, 2023.
[24] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.033000, 2024.
[25] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving Al tasks with chatgpt and its friends in hugging face. In NeurIPS, 2023.
[26) Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. CoRR, 1b2/24 2024.
[27] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPL.U: setreeeal-augmentedd black-box language models. In NAACL-HLT, pages 8371-8384 Association for Computational Linguistics, 2024.
[28) Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling L.LM test-time compute optimally can be more effective than scaling model parameters. CoRR, htts/2408.033/11 2024.
11

---

[29] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang. and Ji-Rong Wen. Re-searcher: Incentivizing the search capability in llms via reinforcement learning. CoRR, abs/2503.05592, 2025.
[30] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning. Journal of Cognitive Neuroscience, 11(1):126 134, 1999.
[31] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.1259) 2025.
[32] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Trans. Assoc. Comput. Linguistics, 10:539-554 2022.
[33] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensin multi-step questions. ACL (1), pages 10014-10037. Association for Computational Linguistics, 2023.
[34] Denny Vrandecic and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Com- mun. ACM, 57(10)778-85, 2014.
[35] Junjie Wang. Mingyang Chen, Binbin Hu, Dan Yang, Ziqi Liu, Yue Shen, Peng Wei, Zhiqiang Zhang, Jinjie Gu, Jun Zhou, Jeff Z. Pan, Wen Zhang, and Huajun Chen. Learning to plan for retrieval-augmentedd large language models from knowledge graphs. In EMNLP (Findings), pages 7813-7835. Association for Computational Linguistics, 2024.
[36] Liang Wang, Nan Yang, Xiaolong Huang. Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. CoRR, abs/2212.03533,, 2022.
[37] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thoure prompting elicits reasoning in large language models. In NeurlPS, 2022.
[38] Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. Corrective retrieval augmented generation. CoRR, abs/2401.15884, 2024.
[39] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, 15112 2024.
[40] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi- question answering. In EMNLP, pages 1369-2388 Association for Computational Linguistics, 2018.
[41] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. In NeurlPS, 2022.
[42] Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. Dense text retrieval based on pretrained language models: A survey. ACM Trans. Inf. Syst., 42(4))89:1-894 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim the scope of this paper in the beginning of Abstract and Instruction, and at the end of the Introduction, we show the contribution of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Appendix C.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

This work mainly focus on the empirical evaluation, and the theoretical analysis is not included.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The main algorithm and training regime are provided in the Method section, and the implementation details are also provided in Section B.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided the code and usage instructions in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. wttpcccpppppceeeeeeee.eec...ceee http//udddss/cooee
ssiiiislicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The training setup is provided in Section 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In experiments, the results conducted by 3 independent runs, with different random seeds, and we report the mean performance across these runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the compute resources in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:///eerpppeeeeeeeeee http://ublcettteitteeieeeeeeeeee
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our work in Appendix D.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit the creators of the assets, and respect the license and terms of use. For benchmark datasets, HotpotQA is distributed under the Apache-2.0 license, MuSiQue is distributed under CC-BY-4.0 license, Bamboogle is distributed under MIT license, and 2WikiMultiHopAA is distributed under Apache-2.0 license. For the code, verl is distributed under Apache-2.0 license, and FlashRAG is distributed under MIT license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi thcode. paperswithcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //tp:///ee....eeeee Wwteurips.cc/oofeeee /LLM) for what should or should be described.
19

---

A Prompt for LLM-as-a-Judge
Prompt for Extracting Scenarios
You will be given question and its ground truth answer list where each item can be a ground truth answer. Provided a pred_ answer, you need to judge if the pred_answe correctly answers the question based on the ground truth answer list. You should first give your rationale for the judgement, and then give your judgement result (i.e., correct or incorrect).
Here is the criteria for the judgement:
1. The predLanswer doesn't need to exactly the same as any of the ground truth answers, but should be semantically same for the question.
2. Each item in the ground truth answer list can be viewed as a ground truth answer for the question, and the answer should be semantically same to at least one of them.
question: (question) ground truth answers: (gt_answer) pred_answer: (pred_answer)
The output should in the following json format: "jjson
"rationale": "your rationale for the judgement, as a text", "judgement"; "your judgement result, can only be 'corrett' or Vincorrect''
Your output:
B
Implementation Details
Our training is conducted on 8 8 Nvidia H800 GPUs, with full parameter optimization and gradient checkpointing. We show some important parameter settings Table
Table 4: Implementation details ReSearch.
Parameter
Value
Learning Rate
1c-6
Train Batch Size
256
Number of Training Epochs
Number of Rollout
5
Rollout Temperature
1.0
KL Loss Coefficient
0.001
Clip Ratio
0.2
C Limitation
While our work demonstrates promising results in training L.MMs to reason with search, there are some limitations to consider. Our current framework primarily focuses on scenarios where the
20

---

answers are relatively concise and can be objectively verified through simple metrics like FI score. This approach may not generalize well tasks requiring longer, more nuanced responses, where more sophisticated reward modeling would be necessary to effectively guide the reinforcement learning process. Additionally, like many existing works in retrieval-augmentte Wikipedia the primary knowledge base for RAG operations, following the common practice due to the availability of standardized open-source knowledge bases. This limitation means we have not yet explored the framework's 's effectiveness with other types of specialized domain-specific knowledge bases that might be more appropriate for certain applications. Future work could investigate extending approach to handle more complex response types and diverse knowledge sources beyond general encyclopedic knowledge.
generation, our study utilizes
D Broader Impact
Our work on ReSearch has several potential positive societal impacts. By improving the ability of LLMs to reason with search capabilities, this framework can enhance the accuracy and relia- bility of Al systems knowledge-intensive tasks, particularly benefiting fields such as education, scientific research, and fact-checking. The framework's 's ability to break down complex questions into manageable steps while verifying information through external sources could help reduce the spread of misinformation and improve the quality of Al-assisted self-reflective capabilities that emerge during training could lead to more transparent and explainable AI systems, fostering greater trust between users and AI technologies. However, we acknowledge potential concerns that warrant consideration. The increased efficiency in information retrieval and processing could lead to higher computational resource consumption, potentially contributing to environmental impacts through increased energy usage. Additionally, while our framework improves accuracy, there remains a small possibility of reinforcing existing biases present in the search results or knowledge bases used for training. To mitigate these concerns, we recommend implementing energy-efficient training strategies and regularly auditing the search sources used in the system. Overall, we believe the benefits of more accurate, transparent, and reliable AI systems outweigh these manageable risks, partieularly the technology continues to evolve with appropriate safeguards and monitoring mechanisms place.
cerision-maaing. Furthermore, the
21


Reviews:
[{"review_id": "Gv35gPQTOZ", "reviewer": "Reviewer_NC8m", "summary": "This paper proposes using GRPO training for multi-hop QA  datasets, which require search for answering. Since outcome-level reward is utilized for RL training (based on answer accuracy and partial credit for format adherence), the proposed ReSearch framework does not require step-level labelled reasoning data, which is very expensive, and requires substantial human effort for generation. This method considers search as an integral component of generating the reasoning for multi-hop QA tasks and aims to guide the search procedure (when and where to search) by using text-level thinking steps. Empirical results show the effectiveness of Research compared to standard RAG and some existing test-time prompt based approaches.", "questions": "Please address the issues raised in the 'Weaknesses' section above. I will be happy to revise my score if those concerns are sufficiently alleviated during the rebuttal. Score updated after the rebuttal.", "limitations": "Discussed adequately in Appendices C and D.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1) ReSearch aims for the LLMs to learn when and where to search and how to use the result to generate the next thinking step via RL. This is an interesting avenue for tackling complex, multi-hop QA problems and has some novelty compared to standard prompting based and/or agentic approaches. RL training also encourages self-reflection and self-correction. Using GRPO eliminates the need of having a critic network (although that aspect is not the contribution of this work and is inherent to GRPO algorithm).\n\n2) ReSearch is trained on a single multi-hop dataset yet demonstrates strong performance on diverse, standard benchmarks; HotpotQA, 2Wiki, MuSiQue, and Bamboogle.\n\n3) The paper is clear and easy to read. Sufficient details are provided for a reader to understand the technique proposed.\n\nWeaknesses:\n\n1)  All the baselines considered in this work are prompt based. There is no comparison to any other method which require either RL training or SFT. In particular, if the comparison with an SFT approach is not presented, how can we understand the usefulness of the RL pipeline used here? One could use the same rollouts used in this paper to do an SFT after filtering the data based on the correctness of the final answer (as in the STAR algorithm, https://arxiv.org/pdf/2203.14465). Can GRPO outperform such a simple SFT strategy?\n\n2)  Despite integration of search steps, rewards are still based on final answer correctness and instruction following capability (outcome-level), not the quality of the search decisions. This may lead to suboptimal or inefficient search patterns (unjustified calls or missing retrievals) because there is no intermediate reward signal assessing search utility. Given that process-level reward modelling is better than outcome-level rewards in other domains (e.g. math and logical reasoning), is there a way to extend ReSearch to utilize process rewards? \n\n3) Some discussion (and comparison to) of recent related work in the same area (e.g. R-Search (https://www.arxiv.org/pdf/2506.04185) and Search-R1 (https://arxiv.org/pdf/2503.09516)) is required for delineating the novelty of this work. Both of these aforementioned works tackle the exact same task (multi-hop QA with search) and use RL training. A careful methodological comparison between ReSearch and these papers would strengthen this paper.\n\n4) There is no MCTS style baseline considered. Given the nature of these tasks, test-time compute using tree-structured heuristic search could be another potential avenue to improve performance. Are there any existing baseline which does that in this domain? If there is any, the RL training approaches need to be compared against such test-time methods.", "was_revised": true, "final_justification": "I have carefully read the other reviews and the authors' rebuttals. Overall, my concerns are addressed adequately, so I am raising my score to reflect that.", "rebuttal": "We sincerely appreciate the reviewer's efforts and insightful comments to improve our manuscript. Below, we address the weaknesses (W) raised.\n\n---\n\n**[W1] Comparison with simple SFT strategies**\n\n*Why RL beats plain SFT*\n\nSupervised fine-tuning (SFT) can only imitate trajectories that the current model already happens to generate; it cannot move beyond the quality of those teacher samples. Reinforcement learning (RL) instead allows on-policy exploration: the model tries new tool-calling sequences, receives scalar rewards, and updates itself accordingly. This feedback loop teaches the model not only what a good trajectory looks like but also why it is better than the alternatives—crucial for tool-augmented agents, where many useful reasoning paths are so unlikely under an untrained policy that they never appear in an SFT dataset.\n\n*Our STaR-style SFT baseline*\n\nTo give SFT its best shot, we reproduced a STaR-like pipeline: (i) use Qwen-2.5-7B-Instruct to sample five roll-outs for every MuSiQue training query (same prompt as our main experiments); (ii) keep only the trajectories whose final answer matches the ground truth; and (iii) fine-tune Qwen-2.5-7B on the resulting nearly 15k trajectories. Results show that this filtered-SFT model improves over prompt-only baselines but is consistently outperformed by our RL-trained ReSearch agent, underscoring the added value of RL.\n\n| Method     | HotpotQA | 2Wiki | MuSiQue | Bamboogle |\n|:---------|:----------:|:------:|:-------:|:---------:|\n| Qwen2.5-7B (STaR-style SFT) | 33.83 | 33.76 | 14.81 | 35.80 |\n| Qwen2.5-7B (ReSearch) | 40.57 | 44.67 | 21.68 | 43.20 |\n\nWe intentionally did not reuse rollouts generated during GRPO training, because those samples are themselves produced by an ever-improving policy—a that benefits inherent to RL.  Using a static model (here we use Qwen2.5-7B-Instruct) keeps the comparison faithful to standard SFT practice.\n\n---\n\n**[W2] Extending ReSearch with process‑level rewards**\n\nWhile our current design deliberately uses a simple, cheap reward (answer correctness + format) to avoid heavy annotation, we agree that ReSearch can be naturally extended to incorporate process signals. Concretely, the following heuristics can be injected into our framework without altering the GRPO loop: \n\n- Redundancy penalty: If a retrieval step returns passages that highly overlap with previously retrieved content, assign a negative reward to discourage wasted calls and encourage informational diversity.\n\n- Self-critique feedback: Use a lightweight verifier to flag unsupported or logically inconsistent intermediate statements, yielding sparse negative rewards.\n\nThese additions keep our RL loop intact (still GRPO) and only modify the reward function. We will add a short discussion of the above options and leave them as future work, acknowledging that richer process signals are orthogonal and complementary to our current outcome-level scheme.\n\n---\n\n**[W3] Discussion of R‑Search & Search‑R1 (contemporaneous work)**\n\nWe appreciate the pointer. Both R‑Search (June 2025) and Search‑R1 (March 2025) were released after March 1, 2025 and thus fall under NeurIPS’s “contemporaneous work” policy. Per the call for papers, such work should be cited and discussed but the absence of full empirical comparison is not grounds for rejection. \n\nMethodologically, ReSearch differs along several axes:\n\n- Reward design: We use purely outcome-level GRPO—without step labels, PRMs, or auxiliary verifiers. In contrast, R‑Search incorporates process-level rewards or external evaluators (such as cross-family reward models for evidence quality). Furthermore, Search‑R1 adopts rule-based rewards based on exact string matching, whereas our work uses F1‑score as a smoother and more informative reward signal compared to exact match.\n- Model scale. We report results on larger backbones (7B and 32B), whereas R‑Search/Search‑R1 primarily train/evaluate 3B–7B models.\n\nWe will add a concise paragraph to clearly position ReSearch relative to these contemporaneous papers.\n\n---\n\n**[W4] Comparison to MCTS-style test-time search**\n\nReasonRAG [1] integrates Monte Carlo Tree Search (MCTS) into an agentic RAG pipeline to offline-generate process‑supervised data (step scores) before training. It is thus a close, tree‑search–based counterpart in this domain. Under the same evaluation protocol (Qwen2.5‑7B‑Instruct backbone, Exact Match metric), ReSearch consistently outperforms ReasonRAG on all four benchmarks:\n\n| Method     | HotpotQA | 2Wiki | MuSiQue | Bamboogle |\n|:------------|:---------:|:------:|:------:|:--------:|\n| ReSearch   | 43.5     | 47.6 | 22.3   | 42.4     |\n| ReasonRAG  | 38.4     | 43.6 | 12.8   | 36.0     |\n\nThese results suggest that reinforcement learning with a simple outcome reward can match—and even surpass—the performance of more complex MCTS-based supervision.\n\n---\n\n*References*\n\n[1] Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning. 2025"}, {"review_id": "N1a2MeMcvO", "reviewer": "Reviewer_XkNw", "summary": "This paper introduces a framework to train LLMs to reason with search tools. To achieve this, the proposed framework applies reinforcement learning without process supervision. Experiment results show the effectiveness of the method with strong generalizability.", "questions": "see above.", "limitations": "Although the paper train model to answer open-ended question, the answers of these question are relatively short and subjective, making them easy to be evaluate. More complicated scenarios should cover longer answers and more complicated evaluation rubrics.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper study a timely topic of training reasoning models to use tools and answer open-ended question. The provided technical details are helpful for the community to learn how to build LLM applications.\n2. The results demonstrate the effectiveness of the method. In particular, extensive evaluation of various benchmarks for single training show promising generalizability.\n3. The introduced method is very simple, making it attractive for future exploration.\n\nWeaknesses:\n1. This paper lacks analysis on how different technical details affect the effect of the method. Given that the data used to train and evaluate in this study contain answers that are easy to evaluate, it is unclear potential challenges for researchers to study more complicated scenarios such as writing tech report.\n2. Although the author make comparison to baseline methods such as native RAG, it is unclear the specific cases where the reasoning models equipped show advantages. It is unclear how training such reasoning agents bring improvement in these search tasks in specific. And what might previous method failed to cover.\n3. The effeciency of the proposed framework in training and deployment in unclear.", "was_revised": true, "final_justification": "This is a timely work that provides many important details for the community and the author also supplement insightful comments in the discussion period which helps progress the field.", "rebuttal": "We sincerely appreciate the reviewer's efforts and insightful comments to improve our manuscript. Below, we address the weaknesses (W) raised.\n\n---\n\n**[W1.1] How different technical details affect the effect of the method**\n\nOur method is intentionally minimalist, and all other components follow standard settings. Consequently, there are relatively few knobs to ablate. Nevertheless, we did run the most informative comparison we could devise—pure SFT vs. our RL training—to isolate the benefit of exploration. We give a STaR-style [1] SFT baseline:\n\n(i) Use Qwen-2.5-7B-Instruct to sample five rollouts for every MuSiQue training query (same prompt as our main experiments); (ii) keep only the trajectories whose final answer matches the ground truth; and (iii) fine-tune Qwen-2.5-7B on the resulting nearly 15k trajectories. Results show that this filtered-SFT model improves over prompt-only baselines but is consistently outperformed by our RL-trained ReSearch agent, underscoring the added value of RL.\n\n| Method     | HotpotQA | 2Wiki | MuSiQue | Bamboogle |\n|------------|----------:|------:|--------:|----------:|\n| Qwen2.5-7B (STaR-style SFT) | 33.83 | 33.76 | 14.81 | 35.80 |\n| Qwen2.5-7B (ReSearch) | 40.57 | 44.67 | 21.68 | 43.20 |\n\nThe resulting filtered-SFT model does outperform prompt-only baselines, but it is consistently weaker than our RL-trained ReSearch agent on all four benchmarks. This experiment confirms that the extra exploration and credit assignment provided by RL—not merely the presence of chain-of-thought data—drives the performance gains.\n\n**[W1.2] Study more complicated scenarios**\n\nWe deliberately began with tasks whose answers admit a rule‑based F1 reward so we could cleanly test whether reinforcement‑trained reasoning + search works at all, without confounding factors from noisy evaluators. We agree that extending to open‑ended outputs (e.g., multi‑page tech reports) demands richer feedback signals. Our framework is agnostic to the reward source, and in ongoing work we are replacing the string‑match reward with (i) rubric‑guided LLM judges and (ii) learned reward models that score content quality and coherence. We will add this discussion to the revision.\n\n---\n\n**[W2] Where the reasoning agent shows advantages**\n\n*(1) Decomposition & planning for complex questions.*\n\nMany queries require breaking a goal into ordered sub‑goals and scheduling searches accordingly—something native RAG/prompt baselines do not do. Example (abridged):\n\n “Name the famous bridge in the city where the composer of La costanza trionfante degl’amori e de gl’odii was born.”\nReSearch explicitly: (i) finds the composer (Antonio Vivaldi), (ii) locates his birthplace (Venice), and (iii) retrieves the bridge (Rialto Bridge). A naïve RAG baseline, lacking this plan, often latches onto an unrelated bridge (e.g., Bridge of Sighs).\n\n*(2) Iterative correction when the first retrieval is wrong.*\n\nAs shown in our case studies (Sec. 3.4), when an early search is off-target, ReSearch can detect the mismatch, revise its plan, and issue alternative queries. Baselines typically accept the first hit, propagating early errors with no recovery mechanism.\nThese two capabilities—explicit multi-step planning and self-correction—explain why training a reasoning agent improves search tasks: it closes the loop between “think → search → verify → adjust,” which prior methods do not cover. We will make these points clearer in the revision.\n\n---\n\n**[W3] Training and Deployment Efficiency**\n\nAs specified in Appendix B, we clarify that our training experiments were conducted on 8 nodes × 8 Nvidia H800 GPUs (64 H800s in total). For deployment/inference, we use a single node with 4 H800 GPUs. We will make this information more explicit in the paper and further emphasize the practical deployment efficiency of our framework.\n\n--- \n\n*References*\n\n[1] STaR: Bootstrapping Reasoning With Reasoning, 2022."}, {"review_id": "YPleKfP3nJ", "reviewer": "Reviewer_75QP", "summary": "This paper presents ReSearch, a reinforcement learning framework that enables LLMs to reason with external search in a unified, text-driven manner. Unlike prior approaches that rely on supervised reasoning steps, ReSearch treats search as an integral part of the reasoning process—learning when and how to search through natural language thought. \n\nThe method is trained on Qwen2.5 models using only one dataset, yet demonstrates strong generalization across multiple multi-hop QA benchmarks. The authors also observe emergent behaviors such as reflection and self-correction during training.", "questions": "1. As mentioned above, I'm particularly curious about whether this pure RL training approach can achieve stable improvements on more challenging benchmarks such as BrowseComp and BrowseComp-ZH, as well as the potential challenges it might encounter.\n\n2. How do the performance comparisons look across different training strategies: pure RL, pure SFT, and SFT+RL combinations?\n\n3. Have you explored any scaling laws for this task?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper trains ReSearch on different scales of the Qwen2.5 models, using only the MusiQue dataset for training. Despite this, it achieves performance gains across multiple benchmarks, demonstrating strong generalization capabilities.\n2. The training framework is relatively straightforward and does not rely on supervised data. And the codebase is easy to understand and follow.\n\nWeaknesses:\n1. The evaluation lacks coverage of more complex benchmarks, such as BrowseComp and BrowseComp-ZH, which limits the assessment of the model's performance in more challenging scenarios.", "was_revised": true, "final_justification": "The authors provide a reasonable justification for focusing on Wikipedia-only retrieval to isolate the effects of RL. Their discussion of BrowseComp highlights meaningful challenges and outlines clear future directions. The comparison with STaR-style SFT is appropriate and supports their claims about RL benefits. Overall, the response adequately addresses the main concerns.", "rebuttal": "We sincerely appreciate the reviewer's efforts and insightful comments to improve our manuscript. Below, we address the weaknesses (W) and questions (Q) raised.\n\n---\n\n**[Q1 & W1] On more challenging benchmarks**\n\nWe share the reviewer’s interest in the performance of ReSearch on full‑web agent benchmarks such as BrowseComp. \n\nOur current study confines itself to Wikipedia retrieval so that we can cleanly measure how RL decides when to search versus when to keep thinking without additional confounders. BrowseComp, in contrast, exposes a much richer environment: the agent must issue DOM clicks, scroll, fill forms, and render JavaScript. Although a few recent systems have reported BrowseComp results by leveraging high-level “search + summary” APIs, this action space is still well beyond the \"think, search\" interface used here.\n\nThe good news is that the key skill ReSearch learns—deciding when to query and when to reason internally—is orthogonal to the concrete low-level actions. In principle, the learned policy can sit atop any web-navigation layer once one is available. Nevertheless, several obstacles must be overcome before we can expect stable gains:\n- Richer action repertoire.  BrowseComp introduces clicks, back/forward, scrolling, and form submission. A naive flat policy would face an intractably large exploration space; hierarchical or curriculum RL is required.\n- Much longer horizons with sparse rewards.  Solving a BrowseComp question often involves ten or more navigation steps and yields only a single binary reward. Without intermediate shaping signals, the advantage estimates become extremely noisy.\n- Context bloat.  Raw HTML plus extracted text quickly exceed the model’s capacity unless working-memory or on-the-fly summarization is introduced.\n\nWe have begun addressing these issues and outline three concrete extensions for future work:\n- Action layer.  We are integrating a lightweight DOM-navigation wrapper so the agent can click and scroll rather than issuing only textual queries.\n- Memory and compression.  We plan to add working-memory and retrieval-result summarization to control context length during long-horizon interactions.\n- Complex task generation.  We will create harder, multi-step question sets comparable to BrowseComp to stress-test the upgraded agent.\n\nWe'll address these issues in our future work.\n\n---\n\n**[Q2] Performance across different traning startegies**\n\nFor an SFT baseline, we reproduced a STaR-style [1] pipeline as follows: (i) use Qwen-2.5-7B-Instruct to sample five rollouts for every MuSiQue training query (same prompt as our main experiments); (ii) keep only the trajectories whose final answer matches the ground truth; and (iii) fine-tune Qwen-2.5-7B on the resulting nearly 15k trajectories. Results show that this filtered-SFT model improves over prompt-only baselines but is consistently outperformed by our RL-trained ReSearch agent, underscoring the added value of RL.\n\n| Method     | HotpotQA | 2Wiki | MuSiQue | Bamboogle |\n|:---------|:----------:|:------:|:-------:|:---------:|\n| Qwen2.5-7B (STaR-style SFT) | 33.83 | 33.76 | 14.81 | 35.80 |\n| Qwen2.5-7B (ReSearch) | 40.57 | 44.67 | 21.68 | 43.20 |\n\n---\n\n**[Q3] Scaling Laws for this task**\n\nSince our end‑to‑end RL + retrieval training is compute‑intensive, we restricted experiments to two backbone sizes (7 B, 32 B) and a fixed rollout budget, so we did not pursue formal scaling‑law analysis. We agree that understanding how performance scales with data, compute, and model size is crucial for tool‑augmented agents and plan to investigate these factors in future work; we will note this limitation in the revision.\n\n---\n\n*References*\n\n[1] STaR: Bootstrapping Reasoning With Reasoning, 2022."}, {"review_id": "zRUS9ZLrSh", "reviewer": "Reviewer_KyAb", "summary": "This work studies on LLM reasoning with search via reinforcement learning. To address the challenges, authors propose a novel framework, named ReSearch, which integrates the search operations into the reasoning process. Experiments on Qwen2.5-7B/32B models across several multiple-hop reasoning benchmarks (HotpotQA, 2Wiki, MusiQue and Bamboogle) demonstrate the effectiveness of the proposed method.", "questions": "How do you address the issue of growthly sequence length during multi-turn search in the training process? I understand that after several rounds of search, the concatenation of search results and generated reasoning tokens could become excessively long. Although the search results do not contribute to loss calculation, their cumulative expansion during iterative search still leads to substantial computational overhead. How is this challenge resolved in your approach?", "limitations": "No.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The work studies on an interesting and important topic for agentic AI. Beyond naive \"search everything\" or \"never search\" approaches, the core question of when, how, and where to integrate search into the reasoning process is fundamental for building modern LLM agents. I think this work is the one of the earliest exploration in this field.\n\n2. The proposed ReSearch method is well-motivated and novel. Building upon the naive GRPO, this work excels in the multi-turn interaction with search tools which is triggered by the special tokens, and retrieval result masking which enables policy model to focus only on the reasoning part. I appreciate the simple but effecitve design of the method.\n\n3. Significant improvements across several multi-hop question answering tasks. Authors evaluates 7B(small) and 32B(large) models on four standard multi-hop QA benchmarks with insightful analysis on response length and number of search rounds. The experiments are comprehensive and convicing.\n\n**Weaknesses**\n\n1. Comparisons with prompt-based baselines and naive RAGs appear somewhat unfair. More robust baselines, such as DeepSeek-R1, should be considered for a more comprehensive evaluation.\n\n2. The base models used are all non-reasoning models. Investigations into reasoning models would be highly valuable, including models like QwQ-32B-preview, the Qwen3-series, and DeepSeek-R1-Distilled-Qwen models.", "was_revised": true, "final_justification": "My concerns are addressed. I will maintain my positive score.", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful comments and suggestions. Below, we address each of the weaknesses (W) and questions (Q) raised.\n\n---\n\n**[W1] Comparisons with more robust baselines** \n\nWhile we did not directly run more robust baselines, such as DeepSeek-R1, we collected their evaluation results from other recent works [1] that follow the same benchmark datasets as ours. The exact match (EM) scores for DeepSeek-R1 and Qwen3-235B-A22B are taken from [1], while the results for ReSearch-Qwen2.5-7B-Instruct and ReSearch-Qwen2.5-32B-Instruct are from our own experiments. The comparison is shown below: \n\n| Model | HotpotQA | 2Wiki | MuSiQue | Bamboogle | \n|:-----------|:---------:|:-------:|:--------:|:---------:|\n| DeepSeek-R1|   48.0   | 54.0  |  27.5   |   52.0    |\n| Qwen3-235B-A22B |   44.5   | 45.3  |  27.6   |   43.8    |\n| ReSearch-Qwen2.5-7B-Instruct |   43.5   | 47.6  |  22.3   |   42.4    |\n| ReSearch-Qwen2.5-32B-Instruct|   46.7   | 44.9  |  26.4   |   56.8    | \n\nIt is evident that, although our models are considerably smaller than DeepSeek-R1 and Qwen3-235B-A22B, their performance is overall comparable across these challenging multi-hop benchmarks.\n\n---\n\n**[W2] Investigation of training on reasoning bases**\n\nThank you for highlighting the potential value of evaluating ReSearch on models that already contain some form of chain‑of‑thought reasoning. Our current study deliberately starts from non‑reasoning checkpoints for two reasons:\n\n1. Clean attribution.  Starting from a “plain” model lets us see exactly how much reasoning‑with‑search our RL pipeline adds, without mixing in skills learned elsewhere.\n\n2. Compute and openness.  Reasoning‑enhanced models (e.g., DeepSeek‑R1‑Distilled‑Qwen, >30 B params) are heavyweight, use private data, and are costly to reproduce. In contrast, vanilla Qwen‑2.5 is fully open and fits within our compute budget.\n\nThis keeps the study controlled and reproducible. We agree that plugging ReSearch into QwQ‑32B‑preview, Qwen3, or distilled DeepSeek variants is a promising next step. We will release code hooks so that the community can replicate these experiments.\n\n---\n\n**[Q1] Issue of growing sequence length**\n\nWe did not introduce any truncation or external‑memory mechanism in the current study. As described in §3.1, the retriever returns at most the top‑5 passages per query, keeping each search block to only a few hundred tokens; in practice, the full context seldom approaches the model’s 32 k‑token window. Figure 3 further shows that the reasoning tokens remain well below this limit across multiple turns, so the computation comfortably fits within our hardware budget. \n\nWe acknowledge, however, that context growth becomes a real bottleneck for longer‑horizon tasks. Recent work suggests two complementary remedies: (i) treating the thinking content as a working memory to retain important reasoning or facts on previous steps [2]; and (ii) summarizing each retrieved passage before appending it, preventing unbounded expansion [3]. These techniques are orthogonal to ReSearch, and we plan to integrate them in future work.\n\n---\n\n*References*\n\n[1] DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learning, 2025. \n\n[2] MEM1: Learning to Synergize Memory and Reasoning for Efficient Long‑Horizon Agents, 2025. \n\n[3] WebSailor: Navigating Super‑Human Reasoning for Web Agents, 2025."}]

Author rebuttal:

Meta-review: After a review round and a rebuttal where the authors and all reviewers engaged with the reviews there is a unanimous decision that this paper is technically solid and makes a good contribution to the research literature. Therefore this paper should be accepted to NeurIPS.

Key strengths include:

* Good articulation of the need for search when answering queries for multi-hop QA.

* Better performance than several benchmarks by levering good querying strategies, resulting in fewer queries and better performance.

Note that all reviews contain questions and guidance for updating and refining the paper and we strongly encourage the authors to take these into account with revising the paper.
Decision: Accept (poster)