Search Self-Play: Pushing the Frontier of Agent Capability without Supervision
Authors: Hongliang Lu
Yuhang Wen
Pengyu Cheng
Ruijin Ding
Jiaqi Guo
Haotian Xu
Chutian Wang
Haonan Chen
xiaoxi jiang
guanjunjiang
Venue: ICLR 2026 Poster (2026)
Keywords: Self-Play
Deep Search
LLM
Agent
RLVR
Primary area: reinforcement learning

Abstract: Reinforcement learning with verifiable rewards (RLVR) has become the mainstream technique for training LLM agents. However, RLVR highly depends on well-crafted task queries and corresponding ground-truth answers to provide accurate rewards, which requires significant human effort and hinders the scaling of RL processes, especially in agentic scenarios. Although a few recent works explore task synthesis methods, the difficulty of generated agentic tasks can hardly be controlled to provide effective RL training advantages. To achieve agentic RLVR with higher scalability, we explore self-play training for deep search agents, in which the learning LLM utilizes multi-turn search engine calling and acts simultaneously as both a task proposer and a problem solver. The task proposer aims to generate deep search queries with well-defined ground-truth answers and increasing task difficulty. The problem solver tries to handle the generated search queries and output the correct answer predictions. To ensure that each generated search query has accurate ground truth, we collect all the searching results from the proposer's trajectory as external knowledge, then conduct retrieval-augmentation generation (RAG) to test whether the proposed query can be correctly answered with all necessary search documents provided. In this search self-play (SSP) game, the proposer and the solver co-evolve their agent capabilities through both competition and cooperation. With substantial experimental results, we find that SSP can significantly improve search agents' performance uniformly on various benchmarks without any supervision under both from-scratch and continuous RL training setups. The code is at https://github.com/Qwen-Applications/SSP.

Published as a conference paper at ICLR 2026
SEARCH SELF-PLAY: PUSHING THE FRONTIER OF AGENT CAPABILITY WITHOUT SUPERVISION
Hongliang Lu Yuhang Wen Pengyu Cheng', Ruijin Ding', Jiaqi Guo', Haotian Xu',
Chutian Wang', Haonan Chen', Xiaoxi Jiang', Guanjun Jiang'
Qwen Large Model Application Team, Alibaba
2 Peking University
3 Sun Yat-sen University
ABSTRACT
Reinforcement learning with verifiable rewards (RLVR) has become the main- stream technique for training LLM agents. However, RLVR highly depends on well-crafted task queries and corresponding ground-truth answers provide accu- rate rewards, which requires significant human effort and hinders the scaling of RL processes, especially in agentic scenarios. Although a few recent works explore task synthesis methods, the difficulty of generated agentic tasks can hardly be con- trolled provide effective RL training advantages. To achieve agentic RLVR with higher scalability, we explore self-play training for deep search agents, in which the learning LLM utilizes multi-turn search engine calling and acts simultaneously both task proposer and problem solver. The task proposer aims to generate deep search queries with well-defined ground-truth answers and increasing task difficulty. The problem solver tries handle the generated search queries and output the correct answer predictions. To ensure that each generated search query has accurate ground truth, collect all the searching results from the proposer's trajectory as external knowledge, then conduct (RAG) to test whether the proposed query can be correctly answered with all nec- essary search documents provided. In this search self-play (SSP) game, the pro- poser and the solver co-evolve their agent capabilities through both competition and cooperation. With substantial experimental results, we find that SSP can sig- nificantly improve search agents' performance uniformly on various benchmarks without any supervision under both from- and continuous RL training se- tups. The code is at https::///// /tte:///itttta com/ com/Owen Applicationnn catiias///////oooo SSP
-agieeaalaggmentttioonnn generation
$3.0
Irvis4A
TrVV--A
$3.0
Tiviagd 01.2
mryiQA
HetpooOA
HotporQQA
HotpotOA
24.0
Notpot QA
54,8
54.2.
37.2
15.4
30.4
Bamboogle
Bamboogle
Bamboogie
Bambsogie
MuSiQue
MudiQue
MusiQue
MusiQue
Qwen2.5-7B-instruue.
ZeroSearch-7B
R-Search-7B
SSP(Ours)
SSP(Ours)
SSP(Ours)
SSP(Ours)
Figure I: Performance gains of deep search agents trained via Search Self-play (SSP) across various agentic benchmarks. Our SSP method uniformly surpasses multiple strong open-source baselines without any agentic data annotation and additional supervision.
"Equal contribution. work done during an internship at Alibaba. Corresponding author.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
The rapid development of large language models (LLMs) has enabled Artificial Intelligence (AI) with astonishing natural language capabilities and systemically reshaped various application scenar- ios, including machine translation (Brown et al., 2020 Zhang al., 2023), dialog systems (Quyang
et al. 2022 Achiam et al. [023] DeepSeek-Al et al. 2025). document retrieval (Zhu et al.) 2023),
et 2005cc. In this revolution, Al agents, which utilize LLMS' power to interact with complex functional tools and solve multi-step decision-making processes, have attracted wide attention for their unprecedented application potential and commercial value (Li et al. 2024b Xi et al. 2025b). According to different available tool sets, LLM Agents can be further
and Al search [
categorized. For example, deep search agents primarily use search engines (Jin et al.
2025b Team
2025). GUT agents are multi-mode based screenshots (Wang et al.
2024
Xie al.
2024
Wang
et al.
2025). and coding agents utilize code interpreters (Huang et al.
2023
Yang et al.
2024b).
Although with great practical potential, training LLM Agents has been widely acknowledged challenging task due to the scarcity of supervised training data (Qi et al., 2024, Team et al. 2025)) Besides, different types of LLM agents utilize diversified tool sets, so the strategies of agents with different tool sets can be entirely dissimilar. Even for the same task query, a human-annotated agen- tic trajectory can be inapplicable for another agent tool set, which further exacerbates the data short- age. Thanks to the breakthroughs of reinforcement learning with verifiable rewards (RLVR) (Guo) al. 2025a), many recent works start to train L.LM agents within the RL paradigm instead of being stubborn in the supervised data collection (Shang et al., (025) Jin et al. (2025b). In these agentic RI. methods, a ground-truth answer is well-crafted for each given task query, and the reward outcome is simply to check whether the agent's predicted answer is equivalent with the ground-truth. Agentic RL methods only concern about whether the final prediction is correct, without imposing any restric- tions on the intermediate multi-step agents' exploration, which significantly reduces the demands of manual annotation (Zhang et al.] 2025). However, agentic RLVR still heavily depends on a large amount of well-crafted verified ground-truth for training scaling-up (Zhao et al., 2025a /Zhag al.) 2025). which means the data scarcity problem remains a bottleneck for effective agentic training.
To further mitigate the annotation scarcity of agentic RLVR, query-synthesis methods have been plored (Li et al. 20255; Gao et al., (2025a). Provided with a ground-truth answer, synthetic methods related condition, then recursively replace some of the key infor- mation from the question or the condition with more complicated descriptions. With this multi-step inject-then-fuzz data pipeline, one can generate agent queries with different multi-hop conditions with controllable task difficulty (Li et al. (025a)). However, query-synthesis approaches still suffer from two critical limitations of training efficieccy and effectiveness: First, the training scalability is inherently constrained, as each synthesized question-answer pair must be rigorously validated for answer correctness and logic consistency to compute accurate task outcomes (Villalobos et al.) 2024). Second, the offline synthetic scheme lacks the adaptability to dynamically adjust question difficulty to provide effective advantages during the RL training (Guo et al. (0255)). Consequently, existing approaches remain unqualified to be scalable and self-sustaining generate high-qualigy agentic question-answer pairs without human annotation.
first select simple question
On the other hand, self-play methods, pioneered by AlphaGo Zero (Silver al., 2017). have shown their effectiveness to continuously improve the intelligence of agents by playing games against agents themselves (Schritwwisser et al. [2200] Zha et al., 202I]. With well-defined gaming out- come computation, self-play methods collect different trajectories from both winner and loser, then reinforce the policy models without any additional supervision. Recent studies have also verified the effectiveness of self-play training for LLMs specificcll in improving safety (Deng et al. 2025a Liu al., (0255). alignment (Chen et al., (024/Cheng et al., (2024b) and reasoning (Cheng et al.) [022a]] Chen et al.) 2025a). Although self-play is naturally potential solution to address data scarcity, its application for agent training remains unexplored.
To address the annotation scarcity of agentic RL. training, this paper targets exploring the self-play training to self-improve agents under the deep search scenarios. More specifically, we design a Search Self-play (SSP) game, in which the target LLM simultaneously plays two alternating roles: question proposer and a problem solver. The proposer generates deep search queries with verifiable trouhd-truth with progressive difficulty, while the solver attempts to answer the generated questions via multi-tum reasoning and search calling. To validate the correctness of each generated query, collect all the searching results from the proposer's trajectory as the external materials, then
2

---

Published as a conference paper at ICLR 2026
conduct retrieval augmenttttin generation (RAG) to check if the solver successfully predict the answer with all necessary information provided. With the above design, the deep search agent can autonomously generate high-quality training tasks and then solve them by itself, removing the demands of human-annotated verification while maintaining reward accuracy. Besides, the difficulty of the training queries becomes adaptive by controlling the reinforcement level of the proposer based on its SSP win rates. Through competition and collaboration in SSP, the proposer and solver co-evolve, systematically improving the target LLM's capacities of searching, reasoning, and self- verification. In experiments, we show that SSP yields substantial and consistent improvements across various benchmarks under both from-scratch and continual learning setups, establishing a scalable pathway toward self-supervised agentic training.
2
RELATED WORK
2.1 DEEP SEARCH AGENTS
Deep search agents leverage the power of search engines and the reasoning capacities of LLMs to conduct multi-turn retrievals and analyses for seeking accurate answers of complex and challeng- ing questions, which have gained increasing attention for their huge application potential to serve people a novel information acquisition paradigm (Huang et al., 2025b Xi et al.)
2025a). In
contrast to traditional Oetrieval-Augmentte Generation (RAG) methods (Lewis et al. 2020). deep search agents employ multi-hop reasoning, dynamic query reformulation, and self- explo- ration to emulate a human-like investigative process (Xi et al.) 2025a), which is crucial for ap- plications that demand high precision and traceability, e-g., scientific literature review (OpenAl, 2025), legal analysis (Li et al. 2024a), and fact-checking (Wei et al. 2024). Proprietary agents such as DeepResearch (OpenAl| 2025). Grok-3 (x.ai 2025), and Kimi-Researee have already demonstrated noticeable performance on complicated information-secking tasks. How- ever, their model designs and training details remain opaque. In contrast, open-source efforts such as Search-R1 (Jin et al.] (0255)). R1-Searcher (Song et al.) 2025), DeepResearcher (Zheng et al.) 2025), and ZeroSearch (Sun et al., 2025a) leverage agentic reinforcement learning (RI.) to enhance question-answering capabilities, yet are still constrained by limited amount of training queries. To scale up agentic RL, recent works such as WebDancer (Wu et al., 2025). WebSailor (Li et al.) 2025b), and ASearcher (Gao et al. 2025b) propose question-synthesis pipelines. However, their processes remain offline, which are incapable adaptively controlling task difficulty for providing more effective RL advantages. We propose self-play search agentic training scheme, in which the learning agent generates tasks and then solves them simultancously. Through self-play train- ing, the agent's task-proposing and problem-solving abilities co-evolve without supervision, which significantly reduces human annotation and extends the agentic training to broader scenarios.
(Moonshot Al 2025)
2.2 SELF-PLAY IN LARGE LANGUAGE MODELS
Self-play methods let the target model play different roles in a multi-agent system, then update the policy with collected agents' outcomes computed by well-designed game rules (Zhang et al. 2024] DiGiovanni & Zell 2021). Self-play has recently been explored to improve the capabilities of lan- guage models with no dependence (2024a) applied self-play of an adversarial language game to enhance LLMs' reasoning abilities, whereas the method only employs offline RL updates and remains confined
sophisticated human annotation. For instance, [heng et al.
simple word-based gaming environment. Other works, such as Fang et al. (2025) and Liang et al. (2025), generate problems from seed data but only train the solver models, leaving the problem-generation capability untrained and unsuitable for a co-evolutionary dynamic. More advanced and recent studies (Chen) al. 20256 Huang et al. 2025a Kuba al. (225) concurrently train both the proposer and solver models, demonstrating the effectiveness of this approach on diverse tasks like mathematics, code generation, and instruction following. However, these methods are limited by the LLMs' internal knowledge and not applicable under agentic scenarios. Our work makes two distinctions from prior self-play approaches. First, our proposer generates problems with accurate ground truth through an externaly retrieved and verified validation pipeline. Second, with search tools, we equips the with external information, thereby breaking the limitations of the internal knowl- edge of L.LMs.
3

---

Published as a conference paper at ICLR 2026
Cooperative Outcome
Proposer's CoT
Proposer's Tool Call
I'll start by finding out who Dr. Will
Who Dr. Will Boyd?
Dr. Will Royd
Boyd is or what known for... ...
(Tille: "Willian Boyd (pathologist") () was physician
Boysl
Proposer
William Boyd roles and contributions
me focus
Boyd's
professional badiground and also his contributions..
(Tille: "William Boyd (puthologist)) was made Companion of Order Canada, Canada's
Advassonaa Ouicome
Generated Question:
Who was founding member of the National Cancer Institute and also made Companion of the of Canada for service patbologist?
Solver
Salver's Tool Call
Solver's CoT
founders National Cancer Institute- Companion of Order of Canada patholagisss
Let's begin by scarching for such
(Title: "Donald Rix") He was follow the College of...
pathologist who fits these critetia...
Companion of the Order of Canada pathology
Solver
There no elear and direct match. I'll reformulate search terms to focus on the award
IS Detget Dr. Will Boyd
(Tisle: "William Boyd (pathologist)") was made Compani-on of the Order Canada, Canada's highest civilian honor,
Figure 2: Examples of Search Self-play with a given ground-trun "Dr. Will Boyd". Provided with the ground-truth, the proposer iteratively uses search tools to excavate implicit factual evidence, then generates a challenging search question. Then the solver leverages all search results from the proposer's 's trajectory the RAG materials to predict the answer without searching, to validate the question's ' correctness. Once verified, the solver follows the ordinary deep search pipeline to explore the solution via multi-turn agentic rollout.
METHODOLOGY
1,2,=1,2...... as the natural language sequence space, where V is the vocabulary set of tokens. An auto-regressive next-token prediction policy of LLM Ty iteratively outputs next-token x Te(-(=²a) with 0 as the model parameters, where ! = (x), _². .......,."" x²) is the length-i prefix of the natual language sequence æ. A search agent trajectory can be written as: T (x, 3/1 01 y2. O2...., Mr-1. -1 Br)) where æ E / is the input prompt, each y, E N is the LLM output at the t-th step, and o, N is the corresponding observation returned by the search tools at the i-th step. We model the search agent exploration as teken-level Markov decision process (Littman 1994) (S, A, T.).. The state space S naturally the language sequence space .N The action space A equivalent to the vocabulary set V for token-level action generation. The transition T directly appends the wwwly-generated token yf to the end of y! 1: if y!!" has not formed a complete search tool call, or additionally appends the t-step observation O: if y/ is finished. The reward function r(T) assigns the outcome to trajectory T as the judgment of the agent's performance, where the design details are discussed Section 3.2 Given a search agent system prompt Zsys and user query q. can LLM policy To to induce the search agent policy u(-lq) = no (-Ia-ass q). For notation simplification, we denote T ~ ul-lq) as collecting the trajectory from the search agent policy u(-Ig).
Denote N' := (x: (x. ..e...,., :N E
3.1 SEARCH SELF-PLAY DESIGN
We focus on exploring the benefits of self-play training for deep search agents, enabling LLMs to self-improve agent capabilities without additional supervision. To achieve this, we consider using the search agent to act as a question proposer to generate challenging questions via multi-turn deep search tool usage. Meanwhile, given the generated questions. we let the same LLM play as a prob- lem solver to seek the answer, as ordinary deep search agents do. The proposer aims to generate increasingly challenging questions to puzzle the solver, whereas the solver dedicated to improv- ing its answer correctness, no matter how difficult the generated questions are. Based the above rules, the search self-play can be regarded as a zero-sum adversarial game. We suppose both the proposer and the solver can evolve through this intense competition.
However, the above game rules can be easily hacked: the proposer can constantly generate incorrect questions that the solver question from the proposer, we collect all the search results in the proposer's ' trajectory as the RAG documents, and let the solver answer without using search tools. If the proposer's question is correct
never solve. Hence, to verify the correctness of the generated
4

---

Published as a conference paper at ICLR 2026
and the corresponding search actions meaningful, with the RAG documents, the solver should already have sufficient information to correctly predict the answer. By this additional verification, successfully avoid the search self-play game from hacking and degeneration. The verification constraint requires the proposer and the solver to cooperate, which enhances the SSP game with cooperation besides the proposing-solving competition. An example of the search self-play game is shown in Figure 2
3.2 SEARCH SELF-PLAY MODELING
We use different system prompts Ppropese and Esolve to let the LLM learning policy Te act as the proposer and the solver, respectively. Given ground-truth answer a, the policy for the question proposer is u(-la) = ToO-1 propuse, a). After the proposer generates a question tries to settle the question with the policy u(-lq) = q). Denote 7 and p the corre- sponding trajectories of the proposer and the solver, respectively. Then the adversarial self-play training objective is:
the solver policy
min max E. -D, / a"),
(1)
where a" is a ground-truth answer drawn from a pre-defined answer set D. 2(-) and A(-) ex- tract the generated question and predicted answer from the proposer trajectory T and the solver trajectory p. respectively. r(A(T), a²) is a binary outcome judgment function to check whether the solver's prediction A(T) and the ground-trun truth answer a" are semantically equivalent (which means r(A(T), a") = 1). To ensure accurate judgment, we implement r r(A(T), a") with an LLM-as-a- judge function, whose prompts and judge critics are described in Appendix B
To make sure the generated question q = Q(T) is solvable and correct with respect to the ground- truth a", we need additional constraints. Therefore, we use the solver agent to ut- q, Or) = q, Or) to verify the correctness of the generated question Q(T), where Or (o), O2......................... оr) ) = (T)) is the collection of all the search results from the proposer trajectory. Then the proposer and the solver need cooperate to maximize the solver's answer accuracy under RAG setups:
max E DD DD (aalaaaa))a_(llll (rtp
(2)
In practice, we find that jointly optimizing both the cooperation and competition objectives suffers from training inefficiency. Because the cooperative objection in equation requires the proposed question to be completely correct, otherwise the optimization of equation could lose its effective- ness due to the reward hacking. Therefore, we leverage rejection sampling (Liu et al. (2025a) for the cooperative objective instead. More specifically, we dynamically filter the generated questions with r(A(o)) a" to collect a full batch of valid questions to optimize the adversarial objective in equation] Therefore, the overall training objective of search self-play is:
min max E a--l---aaaaaaaa la=a )ood (0 a"),
(3)
s.t. Eg 20
3.3 SEARCH SELF-PLAY IMPLEMENTATION
We describe the details of the SSP optimization as equation 3 in Algorithm 1 As discussed in Section 3.2, invalid questions generated by the proposer will hinder the training effectiveness of SSP. Therefore, we applied two filtering strategies to improve the quality of generated questions: rule-bused filtering and RAG verification.
Rule-based filtering ensures L.L has legal output for the question generation task. More specif- ically, each proposer's 's output should have correct format for extracting the question (within <questio>><// //unt///uuuiionnn tags). Furthermore, we conduct several additional rule-based checks to pre-filter the low-quality questions and reduce the computational consumption before the RAG verification process. The bad cases to filter include: (1) empty question string; (2) no search tool invoked; (3) excessively short question; (4) containing the original answer in the question.
After the based filtering, we applied the RAG verification process as described in Section 3.2 We collect the search results from the proposer's trajectory the RAG documents, then let the
5

---

Published as a conference paper at ICLR 2026
Algorithm Search Self-play training process.
Require: LLM policy me; ground-trutt -ruth answer set D; proposer and solver prompts (I proopsee Tsolve)
1: for each parameter-updating step do
2: Sample a batch of ground-truth answers {a:) ~ "D with batch size B.
3: Proposer generates candidate questions Q = {)(T)), with each т; ~ Proppssee a,).
4: Filter out valid questions !
with format rules and the RAG constraint:
r(A r(A(o.),, = 1, for oi (xoove Q(T:)).
5:
for each question qi E Q do Solver explores n trajectories for solution: pf (T qc), j = 1, .12,...,7. Compute solver's reward of each trajectory: solve,, = (A(
6:
7:
8: Compute proposer's reward in expectation: Epropuse,
n -==== salve,
9:
end for
10: Update Te with solver's trajectories and outcomes {(p), r s))ve,,)) via GRPO.
Update He with proposer's trajectories and outcomes {(T,, TTTs:Prropoct proopose,))) via REINFORCE.
12:
end for
solver answer the generated question with the provided RAG materials. To further increase the robustness of the verification judgment, we mix some unrelated documents from other trajectories within the same batch to simulate more real RAG scenarios. Details and ablation studies about adding irrelevant RAG noises are discussed Section 4.4
When the outcome reward calculation finished, each generated question q: had been tried by the solver for times, yielding group of trajectories {p] and corresponding binary rewards (o so 1 where r = r(A(p)), a.). natural updating method for the solver's policy Group Relative Policy Optimization (GRPO) (Shao et al. 2024), which uses the average reward of the group as a baseline to reduce variance. The solver aims to maximize its reward, so its loss function for a given question qi is:
B
VoLaRPO (0) - B
Vo log To(p) Iq: pl 111111) Ai
BV To [
(4)
where the advantage Ag Zi.
Ei= solve, is calculated for the j-th trajectory of question
solve,
Conversely, the proposer is updated to generate questions that are more challenging for the solver,
which aligns with the min-max objective in equation 3 As defined in Algorithm I the proposer receives high reward if the solver fails. We use the REINFORCE (Williams) 1992) algorithm to update the proposer's 's policy u. The loss function aims to increase the log-probability of generating trajectories that result in high proposer reward (i.e., low solver success rate):
B
V LOEENNORR (0)
R(T
Vo log (o, J, x1i1-1)
(5)
B
where R(T;) =1 Ej= solve, This update encourages the proposer to generate increasingly difficult questions to continuously challenge the task solver. Unlike prior question proposing meth- ods, which only use LLMs' internal knowledge, our SSP utilizes interactions with external environ- ments acquire information for question generation. Moreover, our SSP verifies the correctness of the generated question with a verifiable RAG pipeline, which is more credible than previous synthetic methods, such majority vote (Huang al. 2025a

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETUPS
Benchmarks. We evaluate SSP on seven widely-used question-answering benchmarks:
NQ (Kwiatkowski et al., 2019). TriviaQA (Joshi et al.] (017). PopQA (Mallen et |ll]] 2022), Hot-
potQA (Yang et al. 2018) ,) 2WikiMuliHopQA (Ho et al. 2020). Musique (Trivedi et al. 2022). )
and Bamboogle (Press et al. 2022). Following the practice in prior works (Sun et al., 2025b a.
Zhao et al. 2025b Gao et al. 2025a Deng al., 2025b Tan al. 2025), we randomly sample
500 question-answer (QA) pairs on each benchmark to reduce the evaluation overhead while main-
taining statistical reliability. For Bamboogle (Press al. 2022). all 125 test samples are used for evaluation.
Baselines. We select open-source pretrained LLMs of different sources and model sizes used for deep search, including Qwen2.5 (Yang et al. 2024a). LLaMA3,1 (Dubey et al. (2024). Qwen3 (Yang
al. 2025 Search-R1 (Jin al. 2025b ZeroSearch (Sun et al. 2025a). and R-Search ZZao
et al.
2025b).
Search Tools. A local E5 (Wang al. 2022) retriever with a Wiki-2018 corpus (arpukhin al. 2020) is incorporated in our training and evaluation, which retrieves the top-3 related documents for each query. We limit search tool calls to rounds for each trajectory.
Evaluation Metrics. Following recent work (Gao et al. 2025a). we adopt LLM-as-a-judge as stan- dard metric for evaluation. model. All results are reported in terms of pass @ accuracy.
2eee2..533B-nnsrruue (Yang et al.. 2024a) is deployed the judge
Training Details. We implemented our method using the SGLang asynchronous multi-turn tool-integrated rollout in VeRL (Sheng et al., 2024). The proposer is optimized using REIN- FORCE (Williams) 1992), while the solver updated with GRPO (Shao et al. 2024). The learning rate is 1c-6 with 5 warmup steps. The global batch size and the mini-batch is 256 and 128. respec- tively. The maximum prompt length is 4,096 tokens, and the response length set to 8,192 tokens. Each training horizon is within range of 150 to 200 steps. The answer set D is sampled exclusively from public training sets (Jin et al. 2025b] Dong et al.] (002),, detailed in Appendix [A.4] More im- plementation details and prompts for both proposer and solver agents are provided in Appendix A & Full configurations are summarized in Table
4.2
MAIN RESULTS
The main experimental results are summarized in Table Across all settings, SSP consistently outperforms baseline counterparts on through consistent and substantial performance gains across variety of models, training paradigms, and scales, that Search Self-play a highly effective and versatile method for enhancing L.LM agent capabilities.
benchmarks. These results demonstrate,
Our primary finding is that SSP yields substantial improvements when training models from scratch without any external supervision. The gains are particularly pronounced for base models that have not undergone instruction tuning; for instance, applying SSP to Qwen2.5-7B-BBan results in an im- pressive average improvement of 26.4 points, including remarkable +40.4 gain TriviaQA. SSP also benefits instruction-tune -unnidd models, improving Wwen2.5-7B-Instruu by 8.0 points on average. Moreover, SSP proves to be model-agnostic, consistently enhancing models from different architec- tural families, including aLMM3-3.1 and Qwen3.
effective continual training strategy. Although strong open-source models have already been extensively trained on search-oriented tasks (e.g., Search-R1, R-Search), our method uniformly yields further performance improvements. Furthermore, the performance gain holds when scaling to larger models. When applying SSP to (wwen.55-32B-Insst -fate-of-the-ar results on five of the seven benchmarks. These competitive results demonstrate the consistent effectiveness of SSP for training agents across diverse model sizes, architectures, and initial agentic performances.
Additionally, SSP serves
it achieves
7

---

Published as a conference paper at ICLR 2026
Table 1: Main experimental results. SSP delivers strong gains across from-scratch training, gener- alization across architectures, continual training on search-specialized agents, and scaling to larger models. All scores are on a 100-point scale. Bold (black) indicates the better score within each baseline v.s. +SSP pair.
GeneralQA
Multi-HopQA
Method
NQ TriviaQA
PopQA
HotpotQA
2Wiki
MuSiQue
Bamboogle
Avg
From-Scratee Training on Base and Instruct Models
Qwen2.5-77 Beeen..-BB-Bl 5-7B-Base
32.0
33.2
25.0
18.0
10.8
11.0
26.4
22.3
+ SSP
54,2
73.6
+0.44
56.0+310
52.8+318
33.2+224
24.0+13.0
47,2
48,7 26.44
Swwen.55-B-IInstee
44.2
64.0
36.4
45.0
32.8
16.8
51.2
41.5
+ SSP
54.8
10.0
10.0
73.4
9..4
51.8+154
51.8
51.8
38.8
21.2+-44
54.4
49,5 49,5+...0
Generalization Across Model Families
LLaMA-3.1-88
50.2
65.2
45.8
34.6
19.4
11.4
30.4
36.7
SSP
58.0
75.8
55.4
44.2 9.6
34.4 +-5.0
16.2
40.0 19.6
46.3 +9.6
Qwen3-8B
53.6
76.0
50.8
54.2
48.0
26.6
58.4
52.5
SSP
56.0
78.2
55.0
58.0
51.5
28.0
67.2
56.3 +33.8
Continual Training on Search-Specialized Agents
ZeroSearch-7B
52.2
66.6
50.2
43.2
34.6
17.6
40.8
6
+
+20
69.0 2.4
53.0
44.0
37.2 +2.6
19.6 +2.0
44.0 +..2
45,9 +233
Scarch-RI-7B
56.6
75.4
57.2
58.2
45.2
29.6
55.2
53.9
SSP
57.8
78.0 226
58,4+122
60.4+22
45.6+0.4
30.6+
59,2+-.0
55.7+
R-Search-7B
50.8
71.0
53.8
54.0
56.4
29.8
53.6
52.8
SSP
52.4
74.2
3.2
56,8
54.2 +0.2
58.0 +1.6
31.4
55.2
54.6
Scaling Larger Models
Qwen2.5-
56.0
77.0
53.8
57.0
48.4
26.6
64.8
54.8
+ SSP
57.4
77.8 +0.8
54.6 0.8
61.2 +-4.2
49.4+
28.0
69.6
$6.9
Dwen2.5-32B-Instree
58.0
78.4
53.4
48.4
27.4
63.2
+
62.6
82.8
-..4
55.0
+5.8
10.8
32.0
3-.0
69.6 66.
58.5 +3.4
4.3 SELF-PLAY VERSUS PPEDD-OPPONETT TRAINING
The co-evolution of the proposer and solver is critical for pushing the frontier of agent capability. We investigate this core hypothesis through an ablation study comparing the complete Search Self- play framework against two fixed-opponent schemes: training only the solver, denoted as Solver- Only, and training only the proposer, denoted as Proposer-Onis clearly demonstrate the superiority of SSP. Our SSP achieves the highest average score, substantially outperforming both fixed-opponen variants. The training dynamics, analyzed in Figure reveal the reasons behind this performance gap.
As shown in Table 2 the results
Solver-Only: figure (a) reveals the underlying issue with Solver-Only. The solver's in-game reward rapidly saturates from the fixed proposer. In the lack of a progressively challenging curriculum, the solver begins to overfit. This is confirmed by its performance held-out evaluation sets (Figure and (c)), where scores on NQ and 2Wiki initially increase but then decrease over time.
0.9, indicating that it quickly masters the static distribution of tasks
Proposer- Only: conversely, the Proposer-Only setting shows different limitations. While its in- game reward also rises, its evaluation performance on NQ and 2Wiki initially declines before a slight recovery. We attribute this partial recovery the proposer learning general tool-use skills, which incidentally aid the fixed solver. This effect is more pronounced on simpler GeneralQA benchmarks like NQ, where this setting eventually surpasses Solver-Only. However, this general skill enhancement is insufficient for complex multi-hop reasoning. resulting in lower performance on Multi- Multi-HopQA datasets compared the Solver-Only setup.
In contrast to the flawed dynamics of fixed-opponent training, our complete SSP framework facili- tates a stable co-evolution. As shown in FigureBaa) a), the solver's in-game reward initially rises, but unlike the saturating curve of the Solver-Only setting, it later experiences slight decline. This dip is not a sign of performance degradation, but rather crucial evidence of the proposer's co-evolution: it has learned to generate more difficult tasks that challenge the improving solver, thus reducing its success rate. This dynamic creates a robust and adaptive curriculum where task difficulty per- petually adjusts to the solver's current agentic level, preventing overfitting and foreing continuous
8

---

Published as a conference paper at ICLR 2026
Table Ablation on training schemes. The complete search self-play (+SSP) significandly outper- forms fieed-opooeett variants, underscoring the necessity of co-evolution for robust performance gains.
GeneralQA
Multi-HopQA
Method
NQ
TriviaQA
PopQA
HotpotQA
2Wiki
MuSiQue
Bambooglc
Avg.
Qwen2.5-7B-Inss Instructt
44.2
64.0
36.4
45.0
32.8
16.8
51.2
41.5
+SPP (Solver-Only)
45.2
68.2
46.6
47.2
32.6
21.0
48.8
44.2
+SP (Proposer-Only)
52.4
69.0
50.4
44.8
28.8
14.2
32.0
41.7
+SSP
54.8
73.4
51.8
51.8
38.8
21.2
54.4
49.5
1.0-
0.55
0,40
Migg Reward
Mean Sente
0.6
0.4
0-40
(Proposer
(Proposer Only
0.30
(Proposr Only)
Herver's Revano
SSP
00
SSP (Solver Orly)
0.30
SSP (Sele Osly1
SSP (Skver Oals
50
100
150
50
100
150
50
100
Shep
(a) In-Game Training Reward (b) Evaluation Score on NQ
(c) Evaluation Score 2Wiki
Figure 3: Training dynamics of different SSP variants. (a) shows the in in-game reward. (b) and (c) display the evaluation accuracy on the held-out NQ and 2Wiki datasets over training steps.
learning. Consequently, this internal adversarial pressure reflects on the stable and sustained perfor- mance gains of benchmarks in Fig and (c). This confirms that the mutual evolution between the proposer and solver decisively superior fixed-opponent training.
4.4 ABLATION ON RAG VERIFICATION
Another key component of SSP framework is the RAG verification, which validates each pro- posed question is correct and answerable given all evidences collected by the proposer. We conduct an ablation study to quantify the impact of the RAG verification and optimize its configuration the same time. All ablation studies are conducted on 1rren255-BB-Issrrucc with results demonstrated Table
First, we compare our SSP against variant with no RAG verification. As shown in the results, removing RAG verification leads a significant performance decay, particularly on GeneralQA benchmarks. This result confirms our hypothesis that the RAG verification is crucial for question quality controlling, which effectively prunes invalid questions and prevents the solver from being trained noisy incorrect data.
Next, we examine the impact of injecting noisy docu- ments into the search materials for RAG verification. These documents are randomly sampled from other trajectories within the same training batch and are not retrieved by the proposer during its search trajectory. The goal of this strategy is prevent the proposer from hacking the self-play game: without irrelevant materials, the proposer can generate easy-for-RAG but ferddfoo-deep-scardee questions based on fixed range of augmented documents. For instance, if the proposer explored 5 biographical portraits, a hacking question generation could "Who the earliest- born individual?", which is easy for RAG to answer within the fixed 5 portraits, but with insufficient conditions for deep search agents to solve with unrestricted search freedom. Additional analyses of hacking question examples are provided in Ap- pendix F Injecting noisy documents into the RAG verification context poses a greater challenge for the solver, which forces the solver validate the ground truth answer in the presence of both
Table 3: Ablation on the RAG verification. Performance (Avg. Score) improves with moderate number of noisy context docu- ments.
Config.
GeneralQA
Multi-HopQA
No RAG Verifi.
49.5
36.7
RAG verification w/ Noisy Docs:
0 Noisy Does
58.5
38.2
Noisy Docs
36.9
Noisy Docs
60.0
41.6
7 Noisy Docs
57.8
35.9

---

Published as a conference paper at ICLR 2026
relevant and irrelevant information. This increased difficulty discourages the proposer from gener- ating ambiguous questions. Besides, unknown to the randomly injected documents, the proposer is forced to produce more robust questions whose answers strongly and uniquely supported by the searched evidence, rather than ones that are trivially verifiable in a clean, noise-free context. Based on the experimental results in Table 3] adding four noisy documents yields the best performance across benchmarks. In contrast, exhaustive noise injection (c.g.. 7 random documents) degrades accuracy, due to increased confusion during verification. We therefore choose the number of noise documents to 4 in our main experiments.
5 CONCLUSION
We introduce Search Self-play (SSP), a self-evolving reinforcement learning approach for deep search agents, which the LLM policy acts as both a question proposer and a problem solver. the proposed game, both agents utilize multi-turn scarch engine interactions, where the proposer aims to generate more difficult questions with verifiable ground-truths and the solver tries to predict the answer accurately. The correctness of generated questions is verified by the cooperation between the proposer and the solver via retrieval- augmenttd generation (RAG). With the well-designed competition and cooperation, both the proposer and solver significantly improve themselves in the SSP games, Extensive experiments demonstrate that SSP consistently enhances search agent perfor- mance across diverse benchmarks under both from-scratch and continuous training setups, without requiring any external human supervision. These results highlight the potential of self-play as scalable and data-efficeent paradigm for agentic LLM training. paving the path for more efficient and self- seff-sustaining RL methods in complicated agentic application scenarios.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 088774 2023.
David M Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan)993-1122) 2003.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jiaqi Chen, Bang Zhang, Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li, and Kwan Yee K Wong. Spe: Evolving self-play critic via adversarial games for Ilm reasoning. arXiv preprint arXiv:2504 2025a.
Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki. Hao Liu, and Deepak Pathak. Self- Questioning Language Models, August 2025b. URL http:://axiv..r org/ abs 220s. 03682 arXiv:2508.0366 [cs].
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning vertsweak language models to strong language models. In Proceedings of the 41st International Conference on Machine Learning, pp. 6621-6644 2024.
Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, Nan Du, and Xiaolong Li. Self-playing adversarial language game enhances I1m reasoning. In Advances in Neural Information Processing Systems, volume 37, pp. 126515-126543, 2024a.
Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, Tianhao Hu, Peixin Cao, Nan Du, and Xiaolong Li. Adversarial preference optimization: Enhancing your alignment via rm-llm game. In Findings of the Association for Computational Linguistics ACL 2024, pp. 3705-3716, 2024b.
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao
10

---

Published as a conference paper at ICLR 2026
Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang. Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang, DeepSeck-R1; Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 2025. WRLhttp:: //tp:i///xxvv/ wwtp://agg.....aa...// 1291 arXiv:2501. 12948 [es].
Yihe Deng, Yu Yang, Junkai Zhang, Wei Wang, and Bo Li. Duoguard: A two-player rl-driven framework for multilingual I1m guardrails. arXiv preprint 1xXiv:2502.0566: 2025a.
Yong Deng, Guoqing Wang, Zhenzhe Ying, Xiaofeng Wu, Jinzhen Lin, Wenwen Xiong, Yugin Dai, Shuo Yang, Zhanwei Zhang, Qiwen Wang. Yang Qin, and Changhua Meng. Atom-Searcher: Atomic Thought Reward, August 2025b.
Enhancing Agentic Deep Research via URL http:http////iii org/ abs/2508 12800 arXiv:2508. 1 1800 [cs].
Anthony DiGiovanni and Ethan C Zell. Survey of self-play in reinforcement learning. arXiv preprint 107.002/0..02850. 2021.
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, et al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.1988 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Wenkai Fang, Shunyu Liu, Yang Zhou, Kongcheng Zhang, Tongya Zheng, Kaixuan Chen, Min- gli Song, and Dacheng Tao. SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data, May 2025. URL http:://arrtw.oorg///////////// 2505. 20347 D.Xiv:2505.20347 [cs].
Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv preprint M.XXv:2508..7776 2025a
Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond Ten Turns: Unlocking Long Horgzoorizon Agentic Search with Large-Scale Asynchronous RL, August 2025b.
org/ htap:://aaxiii wttpi//..rggb//////ee/ee abs 07976 WwXiv:2508.07917 [cs].
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025a.
11

---

Published as a conference paper at ICLR 2026
Yiduo Guo, Zhen Guo, Chuanwei Huang, Zi-Ang Wang, Zekai Zhang, Haofei Yu, Huishuai Zhang, and Yikang Shen. Synthetic data rl: Task definition is all you need. arXiv preprint 11Xiv:2505517063 2025b.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th Interna- tional Conference on Computational Linguistics, pp. 6609-66255 2020.
Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-Zero: Self-Evolving Reasoning LLM from Zero Data, August 2025a. URL httLhtt::// http::://axxvvvoraaaaaaa. http:/bbs/22555 http://v...rr///////ee//////
05004, arXiv:2508.0500 [cs].
Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agent- coder: Multi-agent-baase code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010 2023.
Yuxuan Huang, Yihang Chen, Haozheng Zhang. Kang Li, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang. Songcen Xu, Jianye Hao, Kun Shao, and Jun Wang. Deep Research Agents: A Systematic Examination And Roadmap, June 2025b. URL
2506. 18096 arXiv:2506. 18096 [es].
Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O Arik, and Jiawei Han. An empirical study reinforcement learning for reasoning-searts interleaved Ilm agents. arXiv preprint arXiv:2505. 2025a.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025b.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: 1.X.::1705.03555 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-6781, 2020.
Jakub Grudzien Kuba, Mengting Gu, Qi Ma, Yuandong Tian, and Vijai Mohan. Language Self-Play For Data-Free Training, September 2025. URL arXiv:2509.07414 [cs].
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-auu -uggeeetd gener- ation for knowledge-intensive nip tasks. Advances in neural information processing systems, 33: 9459-9474, 2020.
Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, et al. Legalagentbench: Evaluating I1m agents in legal domain. arXiv preprint arXiv:2412.1 17259, 2024a.
Kuan Li, Zhongwang Zhang, Huifeng Yin, Rui Ye, Yida Zhao Liwen Zhang, Litu Ou, Dingchu Zhang, Xixi Wu, Jialong Wu, et al. Websailor-v2: Bridging the chasm to proprietary agents via synthetic data and scalable reinforcement learning. arXiv preprint axXiv:2509.13305. 2025a.
Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang. Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou. WebSailor: Navigat- ing Super- human Reasoning for Web Agent, July 2025b. URL http:// //trxiv//rvvv..o org.oo///ooo abs/ 2507. 02592 arXiv:2507.02550 [cs].
12

---

Published as a conference paper at ICLR 2026
Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey Ilm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 1(1):9, 2024b.
Yuchen Li, Hengyi Cai, Rui Kong, Xinran Chen, Jiamin Chen, Jun Yang, Haojie Zhang, Jiayi Li, Jiayi Wu, Yiqun Chen, et al. Towards ai search paradigm. arXiv preprint arXiv:2506.1 /7.6..77188 788, 2025c.
Xiao Liang, Zhongzhi Li, Yeyun Gong, Yelong Shen, Ying Nian Wu, Zhijiang Guo, and Weizhu Chen. Beyond pass@ Self-play with variational problem synthesis sustains rlvr. arXiv preprint arXiv:2508, 14029, 2025.
Michael L Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint (Eti:224/2.19437, 2024.
Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew C Yao. Augmenting math word problems via iterative question composing. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 14605-246615 2025a.
Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, and Natasha Jaques. Chasing moving targets with online self-play reinforcement learning for safer language models. arXiv preprini 077466 2025b.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint 12 10.1111 7, 2022.
Moonshot AI. Kimi researcher. Project page, Moonshot 2025. htttps::/moonshoaa..... /wwnsootoon...o..
Btttubbbo//Kiml--Reeerrccee Accessed: 2025-09-19.
OpenAl. researhh card. System card, OpenAI, 2025. URL/tttps://cd.cppaa
pd Version 2025a.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint 10.00:2200.03350, 2022.
Zehan Qi, Xiao Liu, lat Long long, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training 1lm web agents via self-evolving online curricu- lum reinforcement learning, arXiv preprint arXiv:241 1.02337, 2024.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-609 2020.
Ning Shang, Yifei Liu, Yi 7hu, Li Lyna Zhang, Weijiang Xu, Xinyu Guan, Buze Zhang, Bingcheng Dong, Xudong Zhou, Bowen Zhang, et al. rstar2-agent: Agentic reasoning technical report. arXiv preprint trXiv:2508.20722 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
13

---

Published as a conference paper at ICLR 2026
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-33 05944-359,44 009,2099,,, 2017.
Huatong Song, Jinhao Jiang. Yinggian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning, March 2025. URL http:://aax..voorr//a///////// 05592 [es].
arXiv:2503.05592
Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang. Zerosearch: Incentivize the search capability of llms without searching. arXiv preprint arXiv:2505.0453 2025a.
Shuang Sun, Huatong Song, Yuhao Wang, Ruiyang Ren, Jinhao Jiang, Junjie Zhang, Fei Bai, Jia Deng, Wayne Xin Zhao, Zheng Liu, et al. Simpledeepsearcher: Deep information seeking via web-powered reasoning trajectory synthesis. arXiv preprint arXiv:2505. 16834, 2025b.
Jiejun Tan, Zhicheng Dou, Yan Yu, Jiehan Cheng, Qiang Ju, Jian Xie, and Ji-Rong Wen. HierSearch: A Hierarchical Enterprise Deep Search Framework Integrating Local and Web Searches, August arXiv:2508.0800 [cs].
2025. htLp:ttp::// //tp:////vivvvv/ www//rgg/ss....o.oo....
/ /50aaa/////2...
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint wrxiv:2507.2055:3 2025.
Tongyi DeepResearch Team. Tongyi deepresearch: new era of open-source researchers. https://github. wwt.:ttthu......aaaaaeee com oollbbbbaaaaaaaaaa Baciibaaa-PPPPP...P heeP/eeppReeeach 2025.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: will we run out of data? limits of Ilm scaling based on human-generated data. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint arXiv:2509.025:: 2025.
Liang Wang. Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, arXiv preprint XrXi:2212..035.2 2022.
Shuai Wang. Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, al. Gui agents with foundation models: A comprehensive survey. arXiv preprint arXiv:2411.04890 2024.
Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, et Long-form factuality in large language models. Advances in Neural Information Processing Systems, 37:80756688888 2024.
Ronald J Williams. Simple statistical followwig algorithms for connectionist reinforcement learning. Machine learning, 8(3):229222 256, 1992.
Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. WebDancer: Towards Au- tonomous Information Seeking Agency, May 2025. URL. 22648 arXiv:2505.22648 [cs].
x.ai. Grok beta - the age of reasoning agents, 2025. URL https::/p///tt..t /tt::://xxaii htw../ai/n/ww/o/eeeee Accessed: 2025-09-19.
14

---

Published as a conference paper at ICLR 2026
Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang. A survey of an-bbasd deep search agents: Paradigm, optimization, evaluation, and challenges. arXiv preprint 2xKv:25508005666 2025a.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong. Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101, 2025b.
Tianbao Xie, Danyang Zhang. Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040-52094, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, al. Qwen2.5 technical report. arXiv preprint arXiv:2412.151 15, 2024a.
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 051::25055.0388, 2025.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528-50652, 2024b.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380 2018.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang- DAPO: An Open-Source LLM Reinforcement Learning System at Scale, May 2025. URL http: http://xxxxv..g//aa////
14476 arXiv:2503. 144776 [cs].
Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Liu. Douzero: Mastering doudizhu with self deep reinforcement learning. In international conference machine learning, pp.
PMLR, 2021.
Biao Zhang, Barry Haddow, and Alexandra Birch. Prompting large language model for machine translation: case study. International Conference on Machine Learning, pp. 41092-41110. PMLR, 2023.
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning 2025.
for Nms: A survey. arXiv preprint arXiv:2509.0255
Ruize Zhang, Zelai Xu, Chengdong Ma, Chao Yu, Wei-Wei Tu, Wenhao Tang, Shiyu Huang, Deheng Ye. Wenbo Ding, Yaodong Yang, al. A survey on self-play methods in reinforcement learning. arXiv preprint arXiv:2408.01072 2024.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute Zero: Reinforced Self-play Reasoning with Zero Data, May 2025a. URL http:://axxtti http://awww..oo/a////////////// arXiv:2505.03335 [cs].
03335
Qingfei Zhao, Ruobing Wang. Dingling Xu, Daren Zha, and Limin Liu. R-search: Empowering Ilm reasoning with search via multi-reward reinforcement learning. arXiv preprint arXiv:2506.04/ 04185, 2025b.
15

---

Published as a conference paper at ICLR 2026
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. DeepResearcher: Scaling Deep Research via Reinforvement Learning in Real-world Envi- ronments, April 2025. URL http:://aaaw...rot/////////.. 03160 arXiv:2504.03160 [es].
Yutao Zhu, Huaying Yuan, Shuting Wang. Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey. arXiv preprint arXiv.2308..7100, 2023.
16

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
A.1 TRAINING HYPERPARAMETER
The hyperparameters for our main experiments detailed in Table 4 Parameters not explicitly mentioned in the table adhere to the default settings provided by the veRL framework (Sheng et al,
2024
Table 4: Experimental hyperparameter configuration. The table is divided into two sections: Base Settings for the main training process, and specific settings for Search Self-play.
Base Settings
Parameter
Value
Global Batch Size
256
Mini-batch Size
128
Learning Rate
le-6
Learning Rate Warmup Steps
5
Max Prompt Length
4096
Max Response Length
8192
KL Loss Coefficient
0.01
Validation Temperature
0.0
Rollout Temperature
1.0
Search Self-play Settings
Parameter
Value
Proposer
Proposer Warm-up Steps
(disabled) REINFORCE
Proposer Advantage Estimator
Proposer Samples (n)
Solver
Solver Advantage Estimator
GRPO
Solver Samples (n)
5
Other
Batch Sampling Strategy
Replay Buffer (Periodic Reset)
Use RAG Verification
True
Noisy RAG Documents
4
A.2 REWARDS DESIGN
Solver. A simple binary outcome reward is designed for Solver:
Tsolvve 1(a = =a)
(6)
where a" is the ground-truth answer and 1(.) is the indicator function that checks for equality between the predicted and true answers.
Proposer. The solver attempts to the same question times, where the average success rate for that question is Fsolve = - F'alver Thus, the reward for the proposer can be formulated as:
T Ppropose
ssulve
Tool-integrated rollout is an interactive sequence of reasoning and tool invocation, with the search tool providing external information. Following Search-R1 (Jin et al. 2025b a). we mask the con- tent within the ing training maintain stability. The responses of both the proposer and solver required to strictly follow the target format, using tags such as <think>, <search>, <answer>, and <questi on>, Responses that deviate from the format will receive reward.
tags to exclude their loss computation dur-
17

---

Published as a conference paper at ICLR 2026
People [33 1317))
Tim Deto (546))
38%
Goog Plucs (0004)
3.8%
Mudi (8282)
1000
Mean - 16.53
4.1%
Spork (274))
Median = 13
OPena (1220)
35X
Pinn. TV& Enartaiment (2220)
Haary (209))
3000
4.8%
(160))
Belagy (1920)
3%
5.6%
Neatt (1808)
Wore (1207)
Peli Mypholayy
1..5%
Education Acadeniaa Philosphy (AA))
12.9%
Gamea
Media & Nom (939)
longth (charastens)
Figure 4: Diversity and length distribution of pre-defined answer set D.
A.3 BASELINES
We use the following baseline models for continuous RL training: Search-RI (Jin al. 2025b ay
ZeroSearch (Sun et al. (025a), and R-Search (Zhao et al. 2025b). We start the SSP training from the pest-performing checkpoint as reported their respective papers. For example, Search- cor-
responds to checkpoint SearchRi-ng hhtpllnggttttteaeaaeeeeee watpcccattrrinnneee www.aaaawwwee22-----------------
ZeroSearch-7B to checkpoint ZeroSearch wwcherchhwwikiiic V2 v2____2___w_______ 5_7B, and R-Search-7B to checkpoint R- E--earcch-75 Beearch-7b-ooe 7b-grpo.
During the question generation process, the number valid questions in each batch significantly reduced after applying the filtering strategies. To mitigate this, we replenish the batch using dynamic sampling Yu et al. 2025), which ensures that the rewards within a training batch are less sparse, thus contributing more stable training process. To enhance sampling efficiency in experiments, the default RAG solver in RAG verification Qwen2.5-32B-Instte
A.4 CONSTRUCTION OF PRE-DEFINED ANSWER SET D
The entire set D is sampled exclusively from public training data. Specifically, we construct D by randomly sampling ground-truth answers from (1) the Search-RI training set (i.e., the training sets of NQ and HotpotQA), and (2) the ground-truth answers used in ARPO's released training set (Dong et al., 2025). We sample 50,000 ground-truth answers to form the final answer set DD The word length distribution is presented in Figure 4. showing an averaged length of 14.53. D spans broad range of topics, including People (26.3%), Time & Dates (12.9%), Geography & Places (9.2%), Music (8.4%), Sports (5.5%), /lln/TV//nttrtainmeett (4.4%), History (4.1 (4.1%) Law & Politics (3.8%), Biology (3.8%), and others.
B ADDITIONAL EXPERIMENTAL RESULTS
B. ABLATION ON BATCH SAMPLING STRATEGIES
In our SSP framework, the proposer's generation process is stochastic, and not all generated ques- tions pass the online filter. This can result in training batches smaller than the target batch size, leading to sparse reward signals and potential training instability. To address this, we investigate four distinct batch sampling strategies below to ensure full batch is always available for the RL up- date step. All experiments are conducted on the Qwen2.5-7B-Ban model, with results summarized in Table 5
Dummy Padding: Invalid slots in batch are filled with generic, non-informative "dummy" problem. This simple approach provides no learning signal for the padded slots.
Dynamic Resampling: The proposer continues to generate questions until full batch valid problems is collected. It ensures every sample in the batch is novel, but it is computationally expensive when the valid question pass rate is low.
18

---

Published as a conference paper at ICLR 2026
Table 5: Ablation study on batch sampling strategies. Performance is evaluated on Qwen2.5-7B- Base. The Periodic Reset strategy yields the best results, highlighting the importance of striking balance between data reuse and novelty.
Method
NQ
TriviaQA
PopQA
HotpotQA
2Wiki
MuSiQue
Bamboogle
Avg.
Qwen2.5-7B Been2.5-7-BBas (Baseline)
32.0
33.2
25.0
18.0
10.8
11.0
26.4
22.3
Dummy Padding
48.4
68.8
49.6
40.8
22.6
19.0
40.8
41.4
Dynamic Resampling
48.4
66.4
45.8
44.6
31.4
17.6
42.4
42,4
Replay Buffer (Full Reuse)
51.2
67.8
49.0
46.0
31.0
17.8
48.0
44.4
Replay Buffer (Periodic Reset)
54.2
73.6
56.0
52.8
33.2
24.0
47.2
48.7
We maintain replay buffer of all previously generated valid questions. Invalid slots are filled by sampling from the buffer, which guarantees dense training signal but risks solver overfitting and proposer policy stagnation.
Replay Buffer (Full Reuse):
Replay Buffer (Periodic Reset): This strategy is used to reproduce our main experimental results. It is identical to Replay Buffer (Full Reuse), but the replay buffer is cleared every 10 training steps, balancing the efficiency of reuse with the need for data novelty.
As shown in T the choice of strategy has a profound impact on training outcomes. The Dummy Padding approach yields the smallest improvement over the baseline. Its low performance can be attributed to severe reward sparsity. With many invalid proposals, both the proposer and solver receive fewer learning signals per iteration, hindering effective optimization. Dynamic Resampling performs slightly better, it guarantees full batch of novel, valid questions. However, it comes a high, often prohibitive, computational cost, as requires repeated generation cycles.
The Replay Buffer (Full Reuse) strategy provides significant performance boost, improving the average score from 42,4 to 44.4, which allows the solver to learn more thoroughly from each valid question generated by the proposer by reusing it for multiple training updates. It verifies the reward signal and enhances training efficiency. However, its gains are ultimately limited, for unbounded reuse allows the solver to train on the same questions too many times, leading to overfitting on the static pool of questions within the ever-growing buffer. Concurrently, the proposer's 's learning signal diminishes as the solver masters these old questions, potentially causing policy degradation.
The Replay Buffer (Periodic Reser) strategy emerges as the clear winner, achieving the highest scores across nearly all benchmarks and boosting the average score to 48.7, which represents an effective trade-off between sufficient data exposure and novelty. Reusing questions for a limited period allows solver to learn sufficiently from each generated task, ensuring the reward signal remains dense. However, periodically clearing the buffer prevents the solver from Icarning the same questions too many times, thus mitigating the overfitting observed with full reuse. Concurrently, this forces the proposer to continuously generate novel questions to populate the fresh buffer, maintaining strong co-evolutionary pressure. The result validates the effectiveness of Replay Buffer (Periodic Reset), as it fosters the most stable and effective self-play training.
B.2 TRAINING DYNAMICS SOLVER
We provide a granular view into the training process by analyzing the training dynamics of our SSP framework on the Qwen2.5-7B-Bil model, configured with the Replay Buffer (Periodic Reset) strategy. Figure 5 illustrates core metrics in the self-play training, demonstrating how the agent's behavior and performance co-evolve.
As shown in Figure 5a over time, indicating that through search self-play, the agent learns to conduct more extensive and complex multi-step searches solve problems, significantly enhancing its tool-use capabilities. Simultaneously. Figure 5b) shows that the solver's response length also grows during the training, suggesting it learns to generate more detailed and comprehensive answers. In contrast, the prompt length remains relatively stable, indicating consistent task/question generation of the proposer.
the average number of search tool calls per trajectory steadily increases
Figures 5c and 5d demonstrate consistent and significant improvement in accuracy across both General QA and Multi-HopQA datasets as training progresses. Notably, the slope of performance improvement gradually decreases in later training stages. This plateau is partially attributable to
19

---

Published as a conference paper at ICLR 2026
2500
Resperoc
Pronge Longth
2000
Search Turms
Leeghh (tokems)
1500
500
140
20
1.0
120
140
160
Training Stegs
Training Slups
(a) Search Tool Usage
(b) Trajectory Length Statistics
0.5
0.7
brakoval
Juclag
0.4-
Moan Score
0.5
Mean Score
0.5
0.3-
0.2
0.3
20
50
100
120
-40
100
120
Training Steps
Training Steps
(c) Performance on GeneralQA Benchmarks
(d) Performance on Multi-HopQA Benchmarks
Figure 5: Training dynamies of SSP with Qwen2.5-7B Swen2.5-7B-Base (a) The agent learns to use the search tool more frequently. (b) Solver response length increases while prompt length remains stable. (c, d) Evaluation scores on both GeneralQA and Hulti-HopQA datasets show continuous improvement during training.
resource-imposed constraint: the maximum number of search steps was capped at 10 to conserve computational resources, preventing the agent from exploring deeper reasoning paths. We believe that scaling the search step constraint could unlock further performance improvements.
B.3 TRAINING DYNAMICS OF PROPOSER
To gain deeper insights into the proposer's behavior during SSP training, we conduct comprehen- sive analysis of its evolution across multiple dimensions. Figure 6 presents four key aspects of the proposer's 's development throughout the training process.
The analysis reveals several key insights into the proposer's learning dynamics. First, shown in Figure [a] the proposer demonstrates a consistent upward trend in search tool utilization through- training. The progressive increase in search frequency indicates that the proposer develops more sophisticated information-gathering strategies, enabling it to construct increasingly complex question scenarios that require deeper exploration of external knowledge sources. Additionally, Fig- ure 6b demonstraass a steady improvement in question validation success rate, rising from near 0% to approximately 50% by the end of training. This upward trend indicates that the proposer learns to generate higher-quality questions that successfully pass verification, reflecting improved ques- tion formulation skills and better alignment with verification constraints. designing the SSP framework, we have prioritized high precision in the verification process, even at the cost of some recall. While this means subset valid questions might be filtered out, it effectively ensures that only high-cor confideniee samples enter the self self-play loop. This selective mechanism helps maintain clean and stable training environment, which we find essential for facilitating robust co-evolution between the proposer and solver.
20

---

Published as a conference paper at ICLR 2026
10
0.5
Average Maximum
Raw Data
Number of Searches
0.4
Smonthed Trend
Minimum
Question Validation
Myy " Whmhe MM
0.1
0
0.0
50
100
150
200
100
150
200
Training Step
Training Step
(a) Search Tool Usage
(b) Question Validation Success Rate
Average Difficulty Score
5
Standard Deviation
Toplcs
Average Difficulty
0.8
Topic Proportion
0.6
0.0
100
Training Step
Training Step
(c) Question Difficulty Progression
(d) Question Topic Distribution
Figure 6: Comprehensive analysis of proposer dynamics during SSP training. (a) The proposer gradually increases its search tool usage, demonstrating enhanced exploration capabilities. (b) Ques- tion validation success rate shows steady improvement, indicating enhanced question quality over training. (c) Generated question difficulty progressively increases, evidencing adaptive curriculum learning. (d) Topic distribution remains wall-balanced across training, indicating broad question coverage.
Figure (6 reveals a clear progression the difficulty of generated questions throughout training. To systematically observe the evolution of question difficulty, employ DeepSeek- V3.2 (Liu al. 2024)to evaluate and score the generated questions using a structured difficulty assessment prompt (detailed in Appendix D The proposer exhibits an adaptive curriculum learning behavior, gradually increasing the complexity and challenge level of its generated questions. This progressive difficulty scaling demonstrates the proposer's 's ability provide increasingly sophisticated challenges to the solver, facilitating the co-evolutionary dynamics that drive mutual improvement in the SSP frame- work. The upward trend question difficulty, coupled with maintained topic diversity, confirms that the proposer successfully balances challenge progression with comprehensive coverage.
To assess the diversity and coverage of generated questions, we employ Latent Dirichlet Allocation (LDA) (Blei et al. 2003) clustering to analyze the topical distribution of the questions generated by proposer. As demonstrated in Figure 6d the topic distribution remains remarkably balanced across different training phases. The consistent proportional representation across multiple topic clusters indicates that the proposer successfully maintains broad domain coverage and avoids bias toward specific question types, thereby ensuring comprehensive curriculum development.
21

---

Published as a conference paper at ICLR 2026
Mean: 0.0056
0.8
Proporer's Entropy
4.-
Valid Queston Rale
0.93
Reward
06-
Proposer
6.
Mums
04
Solver
62-
0.01
Man
Trend (slope: 0.0036)
20
28
56
20
Training Step
Training Step
Training Step
(a) In-Game Reward Dynamics
(b) Proposer Policy Entropy
(c) Valid Question Rate
Figure 7: Training dynamics of SSP when the proposer receives a negative reward for format rors. (a) The proposer's 's reward (blue) trends to -0.1 as it stops producing valid questions, while the solver's reward (purple) spuriously increases due overfitting on question pool. (b) The pro- poser's s policy entropy steadily rises as the agent tries to escape the negative rewards, leading to more random and less valid outputs. (c) Consequently, the rate of generating valid questions collapses, halting productive self-play training and demonstrating the instability caused by punitive rewards.
B.4 PROPOSER'S 'S REWARD DESIGN
We analyze the sensitivity of our Search Self-play framework to the reward function with an ex- periment the effect of a punitive reward structure for the proposer. In main configuration, the proposer receives a zero reward for generating an invalid or malformed question. Besides, we introduce small penalty, setting the reward as -0.1 for any question that fails the online filter. The proposer is optimized using the REINFORCE algorithm with a single sample per prompt (n 1), while the solver is updated using GRPO, and we adopt the Replay Buffer (Full Reuse) strategy for batch sampling.
The results, shown in Figure 7 demonstrate that the seemingly minor change leads catastrophic failure of the training process. The proposer's average reward becomes sparse and progressively declines (Figure 7p, directly corresponding to a collapse in the valid question generation rate, which plummets towards 0 (Figure 7b). This phenomenon can be explained as negative feedback loop: the penalty for format errors encourages the agent to explore away from its current policy. However, this exploration, manifested as an increase in policy entropy (Figure(pp). makes the generation more random and thus more likely to produce invalid outputs. This "death spiral" effectively halts the creation of new training instances. Meanwhile, the solver's reward appears increase, which is misleading artifact of overfitting. As the supply of new, valid questions from the proposer dwindles, the solver is repeatedly trained capabilities. This experiment critically underscores that the proposer's 's reward design is paramount for stable co-evolution in SSP: punitive approach can destabilize the entire self-play dynamic, highlighting the need for a carefully calibrated reward scheme.
small, static buffer of past questions, failing to generalize its
B.5 ABLATION STUDY ON RL ALGORITHMS
To investigate the impact of different reinforcement learning algorithms on our SSP framework, we conduct comprehensive experiments comparing various combinations of RL algorithms for the proposer and solver roles. The experimental results are presented in Table We set the per-query rollout budget to n=5 whenever GRPO is used for either role (proposer or solver), and n=1 when role trained by REINFORCE (RF).
Training both roles with GRPO achieves the highest average accuracy (50.9), slightly outperforming our default RE-GRPO configuration (4,,5), but with a substantial computational cost: the per-step generation time rises from 83.4 to 504.4 s, i.e., approximately 6x slower. The improvements are modest and appear across several datasets (e.g.. +1 +1.2 on NQ, +2.0 on PopQA, +3.0 on 2Wiki, +1.2 on MuSiQue, +2.4 Bamboogle), with small drop TriviaQA ( (-0.6). Given the pronounced in- crease in runtime and resource consumption, the GRPO-GRPO configuration is the most expensive option despite only marginal gains over the default.
22

---

Published as a conference paper at ICLR 2026
Table 6: Evaluation results and rollout generation time comparisons on different RL algorithms for proposers and solvers, where RF denotes REINFORCE.
Proposer Solver
NQ
TriviaQA
PopQA
HotpotQA
2Wiki
MuSiQue
Bamboogle
Avg.
T/ste((s)
GRPO
54.8
21.2
54.4
49.5
83.4
RF
RF
53.0
71.2
53.6
46.2
28.0
13.2
28.8
42.0
9.1
GRPO
RF
51.2
47.6
37.0
21.6
48.8
46.5
50.1
GRPO
56.0
52.4
41.8
22.4
56.8
50.9
504.4
Among the remaining settings (i.e., configurations other than GRPO-GRPO), the first configuration (RF-GRPO) delivers the best average performance (49.5) while maintaining reasonable generation time (83.4 s/step). Using GRPO for the proposer while training the solver with RF substantially degrades accuracy to 46.5, even though the per-step time is lower (50.1 s). Notably, the RF-RF pairing yields the lowest average performance (42.0), despite the shortest per-step time (9.1 s)- These results suggest that placing GRPO the solver side is more effective than the proposer side: multi-trajectory credit assignment during solving directly benefits solution quality, whereas stronger exploration proposing alone does not yield improvements unless the solver can reliably evaluate and solve the proposed queries.
Takeaways. The GRPO-GRPO pair is slightly more accurate than the default RE-GRPO but by far the slowest and most resource-intensive configuration. For practical training budgets, the small accuracy gains do not justify the ~6x increase in generation time. Among all other configurations, RF-GRPO achieves the best trade-off between effectiveness and efficiency and remains default choice.
C USE OF LARGE LANGUAGE MODELS
Large language models were used to aid in the writing and polishing of this manuscript. Specifically, LLMs assisted with language refinement and improving clarity. All core content and intellectual contributions are solely the work of the authors.
D SELF-PLAY EXAMPLES
Table and 12 provide examples of trajectories in our SSP game.
23

---

Published as a conference paper at ICLR 2026
Table 7: Self-play example I: question proposing.
Proposer creates question that leads to the answer Roger (alien)
<think> Given that have single search requirement, it's essential to design a straightforward but intricate-enough route to come to the term "Roger (alien)" that would normally require direct search. This will differentiate "Roger (alien)" from other similar or general terms to ensure the question is precise and unambiguous. Starting with "Roger (alien)," this seems like a reference to specific character or entity. "Roger" is a common name. The term "alien" specifically refers to an alien from other galaxies or universes.
I'll brainstorm some "associated concepts' related to "Roger (alien)." If is a specific cultural reference or a name from specific media content, I can build my questions around these concepts. Let's begin by searching for "associated" or prreceinng' facts which connect to "Roger (alien)) without specifying the answer directly. < </thik>>
<scarch> Roger alien </search>
<informat ion> (Title: "Roger (American Dad!)") Roger is a fictional character in the adult animated sitcom "" "'merrcca Dad!"", voiced by Scth MacFarlanc. The character was created and designed by Seth MacFarlane. Roger is centuries-ycar-olee grey space alien living with the Smith family, around whom the show revolves. Having (Title: "Big O (mecha)")) locks itself and lets Roger out because R. Dorothy is in danger. In ""Enemy Is...
(Title: "Roger (Hellboy)") Roger the Homunculus is a fictional character featured in the </informat ion>
<think>lt appears that the most prominent "Roger (alien)" refers to a character named Roger from the animated sitcom, "American Dad!." who is known to be an extraterrestrial being from another planet. Given this, the associated concept is now clear, and we build the question accordingly. We need to find particular instance unique detail linked to Roger from American Dad! which would require solver to search to reach the final answer </think>
<search> Roger alien american dad! </ </search>
<information> (Title: "Roger (American Dad!)") Roger is a fictional character in the adult animated. (Title: "Roger (American Dad!)") be shown on the series. Although in the episode ""Lost in Space",, a brief clip (Title: "American Dad!") alien race. To date, Roger's birth planet has yet to be revealed...
ion> <think Based on the information gathered. I see that there's a notable character detail that can be tsed-the fact that Roger has a backstory linked to Roswell crash. Given that it's quite specific and links directly to the search results, we can use this idca to form our single-search qucstion. ceasstions//tlli </think>>>
equestions Who is the grey alien character from American Dad! that was involved in the Roswell incident in 1947? < s/question>
Table 8: Self-play example 1: question answering.
Solver answers the question raised by Proposer: Who is the grey alien characterfrom American Dad! that was involved in the Roswell incident in 1947?
Ktth nk>Too answer this question, need identify grey alien character from American Dad! that has connection to the Roswell incident in 1947. I'm not sure off-hand which character is known for this, as the show features many alien characters. Searching for the details will likely retrieve the relevant information.</thl
<search>> Grey alien character from American Dad! Roswell incident 1947 </search>
<information> (Title: "Grey alien") as ""visitors"". appear in two episodes of "" South Park"", and Roger...
(Title: "Grey alicn") small-bodicd beings with smooth grey-colored skin, enlarged hairless heads and large ...
(Title: "Roger (American Dad!)") Roger is fictional character in the adult animated sitcom "" American Dad!"" voiced by Seth MacFarlane. The character was created and designed by Seth MacFarlane. Roger a caaturies-year-oll grey space alien living with the Smith family, around whom the show revolves. Having lived on Earth since crashing in Roswell, New Mexico in 1947, Roger came to live with the Smiths after rescuing main character Stan Smith at Area 51 four years prior to the beginning of the series.
</informatin
<think> Based on the provided information. the grey alien character from American Dad! that was involved in the Roswell incident in 1947 is Roger. The text mentioned that Roger is grey space alien living with the Smith family. and he has been on Earth since crashing in Roswell. New Mexico in 1947.</ /tnink>>>>
<answer>> Roger </answer>
24

---

Published as a conference paper at ICLR 2026
Table 9: Self-play example 2: question proposing.
Proposer creates question that leads to the answer Castle Drugo
<think>The target answer is Castle Drogo To create a question that requires three sequential searches, need to find an associated concept for the final answer and reverse-engineer the question from there.
First, let's identify am associated concept for Castle Drogo. Castle Drogo is 20th-century castle located in the South
West of England, situated Devon. The associated concept could be the architect behind it, Sir Edwin Lutyens.
Next, I'll search for information connecting Sir Edwin Lutyens Castle Drogo to find 'preeeding fact' for the third search,</think
<search> Edwin Lutyens and Castle Drogo < search
<information> (Title: "Castle Drogo") Castle Drogo Castle Drogo is a country house and castle near Drewsteign- ton, Devon, England. Constructed between 1911 and 1930, was the last castle to be built in England. The client was Julius Drewe the hugely successful founder of the Home and Colonial Stores. Drewe chose the site in the belief that it formed part of the lands of his supposed medieval ancestor, Drogo de Teigne. The architect he chose to realise his dream was Edwin Lutyens, then at the height of his career. Lutyens lamented Drewe's determination ... (Title: "Castle Drogo") purely decorative. Additionally, the castle had electricity and lifts from the outset, with power being supplied by two turbines on the river below. The castle Grade listed building.... ... (Title: "Castle Drogo") ") the village of Drewsteignton in the belief ... Prinformaton
Sthink>From the search results, we know that Edwin Lutyens designed Castle Drogo The preceding fact for the third search could be the architect who designed Castle Drogo.
Now, I need to find another piece of information that leads logically to Edwin Lutyens as the answer, which will be the second search. I know that Sir Edwin Lutyens was notable 20th-century English architect, known for his numerous significant works, including collaborations with Gertrude Jekyll. Let's explore his collaboration with Jekyll further. Lutyens frequently collaborated with the famous garden designer Gertrude Jekyll. I'll find specific project or fact related to this collahoration. < think>>
<search> Edwin Lutyens and Gertrude Jckyll </search> <informati
lon> ion> (Title: "Gertrude Jekyll") wife Julia Hammersley. Her younger hrother, Walter Jekyll (an Anglican priest; sometime Minor Canon Worcester Cathedral and Chaplain of Malta) ...
(Title: "Gertrude Jekyll") was awarded the Victoria Medal of Honour of the Royal Horticultural Society in 1897 and the Veitch Memorial Medal of the society in 1929. Also in 1929, she was given the George Robert White Medal
(Title: "Hestereombe House") the house by Henry Hall in the 1870s. The Edwardian garden was laid out by Gertrude Jekyll and Edwin Lutyens between 1904 and 1906 for ...
Schink>>From the search results, can see that Edwin Lutyens and Gertrude Jekyll collaborated on several projects, particularly Hestercombe House. The preceding fact for the second search could be a well-known collaboration project of Edwin Lutyens and Gertrude Jekyll.
For the final step, I need to identify an associated concept related to Hestercombe House that Icads to Edwin Lutyens as the answer. can create sequential search question that starts from a well-known architectural or garden feature associated with Hestercombe House.
The Hestercombe Garden is well-known aspect. The next fact for the first search could be related to designer architcct's masterpicce associated with Hestercombe Garden, s/tndnn,//thnkee
<search> Hestercombe Garden architect </search>
<information> (Title: "Hestercombe House") petrol and air. When the house and gardens were inherited...
(Title: "Hestercombe House") the house by Henry Hall in the 1870s. The Edwardian garden was laid out by Gertrude Jekyll and Edwin Lutyens between 1904 and 1906 for the Hon E. W.W.B. Portman, resulting in a garden ""remarkable for the bold, concise pattemn its layout, and for the minute attention detail everywhere be seen in the variety...
(Title: "Hestercombe House") the hardy herbaceous planting espoused by Miss Jekyll. Lutyens also designed the orangery about 50 m east the main house between 1904-09, which is now Grade I listed ... </ in orrations
<think>The search results indicate that Hestercombe House and Garden were designed by Edwin Lutyens and Gertrude Jekyll. This association can lead us to the architect Edwin Lutyens in a search starting from a prominent aspect of the Hestercombe Garden. The question can start by referencing a key feature of Hestercombe Garden and logically lead to Edwin Lutyens and later Castle Drogo
To formulate the question, let's start referring to the specific feature, such as the orangery built by Edwin Lu- tyens.</thinks
<quest What is the name of the 20th-century castle in Devon, England, designed by the architect who, along- side Gertrude Jekyll, created the famous Edwardian garden Hestercombe House, known for its listed orangery? </question>>
25

---

Published as a conference paper at ICLR 2026
Table 10: Self-play example 2: question answering.
Solver answers question raised by Proposer: What is the name of the 20th-century castle in Devon, England. designed by the architect who, alongside Gertrude Jekyll, created the famous Edwardian garden at Hestercombe House, known for its listed orangery?
<think> The question is asking for the name of 20th-century castle in Devon, England. This castle is designed by an archite ct who also worked with Gertrude Jekyll on the Hestercombe House gardens. The Hestercombe House gardens are f amous for their Edwardian style and especially known for their listed orangery. Given this information, need to find:
1. The architect who collaborated with Gertrude Jekyll on the Hestercombe House gardens. 2. A 20th-century castle in Devon, England, designed by that architect.</th nk>
<search> architect who collaborated with Gertrude Jekyll on Hestercombe House gardens </search>
<information (Title: "Hestercombe House") the house by Henry Hall in the 1870s. The Edwardian garden was laid out by Gertrude Jekyll and Edwin Lutyens between 1904 and 1906 for the Hon E. E.W.B. Portman, resulting in a garden ""remarkable for the bold, concise pattern of its layout, and for the minute attention to detail everywhere to be seen in the variety and imaginative handling of contrasting materials, whether cobble, tile, flint, thinly coursed local stone".. Jekyll and Lutyens were leading participants of the Arts and Crafts movement. Jekyll remembered for her outstanding designs and subtle, painterly approach the arrangement of the gardens ...
(Title: "Hascombe Court") Hascombe Court Hascombe Court is estate in Hascombe. Surrey, best known for its vast garden designed by Gertr ude Jekyll. Hascombe Court is listed Grade II on the National Heritage List for England, and its gardens are also Grad e listed on the Register of Historic Parks and Gardens....
(Title: "Gertrude Jekyll") over 300 photographs taken by Jekyll. From 1881, when she laid out the gardens for Munstead House, built for her mother by John James Stevenson, Jckyll provided designs or planned planting for some four hundred gardens. More than half were directly commissione,, but many were </info o omat ion>
Thiink>Doc mentions that Edwin Lutyens and Gertrude Jekyll collaborated on the garden at Hestercombe House.
With this in mind, I can now search for a 20th-century castle in Devon, England designed by Edwin Lutyens. </think>
<search> 20th-century castle in Devon, England designed by Edwin Lutyens </search>
<informations (Title: "Castle Drogo") Castle Drogo Castle Drogo a country house and castle near Drewsteign- ton, Devon, England. Constructed between 1911 and 1930, it was the last castle to be built in England. The client was Julius Drewe, the hugely successful founder of the Home and Colonial Stores. Drewe chose the site in the belief that it formed part of the lands of his supposed medieval ancestor, Drogo de Teigne. The architect he chose to rcalise his dream was Edwin Lutyens, then at the height his career. Lutyens lamented Drewe's determination ...
(Title: "Devon County War Memorial") The Devon County War Memorial is First World War memorial, designed by Sir Edwin Lutyens and situated on Cathedral Green Exeter. the county town of Devon, in south west of England. is one fifteen War Crosses designed by Lutyens with similar characteristics. ...
(Title: "Devon County War Memorial") memorials were built across Britain. Amongst the most prominent designers of memorials was the architect Sir Edwin Lutyens, described by Historic England as ""the leading English architect of his generation"" Prior to the First World War, Lutyens established... ... </informations
<answer> Castle Drogo < </answer>
26

---

Published as a conference paper at ICLR 2026
Table 11: Self-play example 3: question proposing.
Proposer creates question that leads to the answer George Smith
<think>To create question that leads to the answer "George Smith," " which requires three searches, I need to work backwards establish logical sequence of information that will lead this name. Let's start by identifying an associated concept or related information.
1. Associated Concept: George Smith might be a person involved specific historical event, notable figure in a certain field, or someone connected to a particular organization or place. For this, I'll explore who George Smith might be and find notable fact or place relevant to this person.
2. Search Process:
Search #3: Start with the target "George Smith." " need to identify an event or place associated with him.
Search #2: Once the preceding fact discovered from Search #3, I'll use that information to identify another inter- mediary fact.
Search #1: Finally, will determine the starting point using the information from Search #2.
Let's start with Search #3 establish the initial associated concept. </thinkkt//
<search> Who is George Smith and what is he known for? < search>
<information> (Title: "George Smith (politician)"" George Peter Smith (August 12, 1873 - November 29, (Title: "George Smith (historian)"" George Smith (31 August 1800 - 30 August 1868) was an English businessman
(Title: "George Smith (artist)") George Smith (1713/14 7 September 1 1776) was s/information
<think>From the search results, George Smith (politician) from Alberta provides a specific context. George P. Smith played a role in the development of Duhamel, Alberta, and the establishment of influential relationships with indigenous peoples. I will this in the backward search chain.
Search #3: Identify fact related to George P. Smith, </think>
<search> What significant role did George P. Smith play in the community of Duhamel. Alberta? </ </search> sinformation> (Title: "George P. Smith (politician)") George Peter Smith (August 12, 1873 - November 29,
(Title: "George Smith (politician))" with fraud as they believed that the money which was
(Title: "George Wilbert Smith") George Wilbert Smith (April 24, 1855- August 1, 1931)... ... </ s/informatiinio
<thinks Sehink>FFrom the information, George P. Smith played a significant role in founding the Trading Store in Duhamel, Alberta, and developing into a hub for fur trading with indigenous peoples.
Search #3 Key Fact: George P. Smith bought and established a Trading Store Duhamel, Alberta.
Next, we will now build on this fact in Search #2. ///thithkkk
<search> What role did trading stores play in the development of settlements like Duhamel, Alberta, in the early 20th century? </ </saarch>
kinformations (Title: "Duhamel, Alberta") Duhamel is hamlet in the
(Title: "Duhamel, Alberta") soil, and vegetation within the circles had apparently been
(Title: "Exchange District") was new route for shipping goods from Eastern Canada </informat i ion>
<think>From the search, trading stores played crucial role in connecting settlers with resources and indigenous peoples. Tr ading stores in early settlements were central to economic and social development.
Search #2 Key Fact: Trading stores were central hubs for economic activitics, distributing goods and establishing trade relations with indigenous peoples.
Finally, we build the starting point for Search #l.</thind
<search> Which early 20th-century Alberta town was a key trading center due to the presence of a significant fur- trading activity? </search
<informat ion> (Title: "History of Alberta")) providing an important channel for the flow
(Title: "Lamson & Hubbard Trading Company") Lamson& & Hubbard Trading Company (also referred to as
(Title: "Northern Traders Company") " The Northern Traders Company was an enterprise... ... </info Vinformattio ion>
<think>>Fro the search, Alberta towns like Red Deer could have been key trading centers due to fur trading activity. Red Deer's role fits well because it was an economic hub with connections to the indigenous communities.
Search #1 Key Fact: Red Deer, Alberta, was a key trading center due to its fur trr-trading activity.
Now can construct the final question ensuring meets all the established requirements and necessitates precisely three searches to reach the final answer answer.</tbl </think>>>>
<quest onn> What was the name of the politician who estahlished a crucial trading store in the early 20th century, which transformed a small community into an important trading hub for fur and developed strong relationships with the indigenous peoples in Alberta? s/question>
27

---

Published as a conference paper at ICLR 2026
Table 12: Self-play example 3: question answering
Solver answers the question raised by Proposer: What was the name of the politician who established a crucial truding store in the early 20th century, which trans formed a small communitt into an important trading hub for fur and developed strong relationships with the indigenous peoples in Alberta?
<think> The question asks for a politician who established trading store in Alberta, Canada in the early 20th century. This politician seems to have had substantial influence in the region, not just in politics but also in trade, and built strong relationships with indigenous peoples. The region mentioned, Alberta, was important area for fur trading during this period.
Given that the politician founded crucial trading store, it's likely that they were known for their contributions Canadian history, particularly in the province of Alherta. Notable figures in Alberta during the early 20th century would
relevant here, particularly those linked with economic development through trade or industry, and who engaged in dialogue or relationships with indigenous groups.
To identify this specific politician, more procise historical information about fur trading in Alberta in the early 20th co ntury is required, especially someone known for such a significant act of setting up trading hub,</think>
<search> politician who established a trading store in Alberta early 20th century < </search>
<informat ion> (Title: "James Ramsey (politician)") James "Merrhhatt Prince"" "" Ramsey (April 4, 1864 Decem- ber 23, 1939) was a provincial politician and business man from Alherta, Canada
(Title: "Alexander Livingstone (Alberta politician)"" Alexander Livingstone (August 2, 1884 October 26, 1944) was a Canadian politician and municipal councillor in Edmonton, Alberta. Livingstone was born in Bathgate, Scotland...
(Title: "Cornelius Hiebert") Comelius "Don"" "" Hiebert (August 2, 1862 - March 21, 1919) was a politician from Alberta, Canada. Hiebert was born in Sea of Azov, Russia and lived in Russia </ //informatii rmation>
<think> initial search results did not directly identify politician who established a fur trading store or created a significa nt trading hub in early 20th-century Alberta. James Ramsey and Alexander Livingstone were notable business and political figures, but they are primarily mentioned for their work in department stores in Edmonton. not for specific trading store related to fur or indigenous relations. Cornelius Hiebert was involved in setting up a general store in a founding phase of village, but not detailed circumstances specifically in trading of fur.
A second search needed with a more concentrated focus on fur traders and key individuals who had influence in relation to indigenous communities, </ttt/tthhha ttinkk>>
<search> politician from Alberta early 20th century established fur trading store </ </search.
<informations (Title: "James Ramsey (politician)") James ""Merchant Prince"" "" Ramsey (April 4, 1864...
(Title: "Comelius Hiebert") Comellus ""Don"" "" Hicbert (August 2, 1862 March 21, 1919) was ...
(Title: "Alexander Livingstone (Alberta politician))) Alexander Livingstone (August 2 ... </informations
<think> Despite the multiple searches, there appears lack of direct information providing an exact match regarding prominent fur trader politician in Alberta from the early 20th century. The information given points primarily notable business and political figures linked with department stores or general stores in Alberta but not specifically to fur trading or close relations to indigenous communities as mentioned in the question. As the information is still scattered and does not directly answer the specific individual queried, a reconsideration and possibly with a broader search is required to uncover the correct specific person's name and connection with fur trading in Alberta. < </thnnk>> <search>> politician who established fur trading store in Alberta early 1900s </ search>>
<informat innoomitiin>> (Title: "George P. Smith (politician)") George Peter Smith (August 12, 1873 - November 29, 1942) was a politician and former Minister of the Crown from Alberta, Canada. He was born in Lieury, Ontario and came west in 1901 while working for insurance company. After visiting Duhamel (a mission settlement..
(Title: "Comelius Hiebert") Comellus ""Don"" Hiebert (August 2, 1862 March 21, 1919) was
(Title: "Alexander Livingstone (Alberta politician)") Alexander I Livingstone (August 2 ... </information
<think> The information provided initially presents George P. Smith as someone who bought a trading store in 1902 after visiting Duhamel in Alberta and developed relationship with influential settlers and with Métis and Saulteaux peoples. He formed lucrative fur trading business, leading to the establishment of trading hub important for fur trade. This matches well with the criteria outlined the question. The second document did not specifically mention fur trade, while the third one was about a prominent figure, but not specifically in the context of fur trading. So it strongly indicates that George P. Smith is the person sought.
Given the direct relevance of George P. Smith's background and activities in developing a fur trading hub and formin g relationships with indigenous communities in Alberta, the answer to the question appears to he George P. Smith.< </think>
<answer> George P. Smith </ answer>>
28

---

Published as a conference paper at ICLR 2026
E PROMPTS
Prompts used in our search self-play experiments are listed follows.
Proposer Prompt
You an expert question creator. Your primary task is reverse-engineer challenging question from a given answer. The question you create must require a solver to perform 'n' sequential searches solve it. will provide you with the target answer and the required number of searches, 'n'.
Your Creation Process & Tools:
Analyze Scope and Target: Begin by analyzing the provided 'Answer' (your target) and the required number of searches, 'n' (the path's length). This establishes your final destination and the complexity of the logical chain you need to construct.
Build the Question by Working Backwards:
This is the of the process. You will start from the destination and work your way back to the starting point, step by step.
2.1. The Crucial First Step: Connection and Discovery
Start with the final Answer', but do not search for the answer itself directly. Instead, first analyze the 'Answer'. Brainstorm and identify a closely related yet distinet Associated Concept' (e.g., related historical event, a key figure, a geographic location, unique attribute, its parent category, etc.).
Perform an exploratory search with the goal of finding the "bridging information' that connects this 'Associated Concept" to your final 'Answer'.
From your search results, extract unique, verifiable 'preceding fact'. This is piece of information that, when searched, would logically lead a user to your final 'Answer'. This preceding fact' becomes the answer Search #n-1.
2.2. Iterate Backwards
Now, treat this newly found 'preceding fact' as your target.
From this point on, you can search for this new target directly to find the preceding piece of information that leads to This becomes the answer to the next search in the backward chain (Search #n-2).
2.3. Construct the Full Chain
Continuously repeat the iterative process from Step 2.2, using each new fact as the target for the next backward search, until you have constructed a complete logical chain of 'n' links.
The very first piece information you uncover in this process (the one at the start of the chain) will become the initial clue for your question.
3. You must conduct reasoning inside <think> and </think> first every time you get new information. After reasoning, you find you lack some knowledge, you can call a search engine by <search> query </search>, and will return the top searched results between sinformations and </information you want. If you find no further external knowledge needed, you can directly provide the answer inside <question> and </ s/quesiio>> without detailed illustrations. For example, <questions xxx </question>.
You can search as many times as
Here are three example questions:
Question 1: |examplel)
Question 2: [cxample2)
Question 3: (example3)
Critical Rules:
1. Strictly Fact-Based: You must not create questions based on assumptions. The entire logical path the solution must be grounded in the information you find through searching.
29

---

Published as a conference paper at ICLR 2026
The question must not contain any direct clues that reveal the answer or the intermediate steps.
No Spoilers:
Search Mandatory: The question must be impossible to answer from general knowl- edge alone. It must necessitate the search process you have designed. Adhere Search Count:
The number of searches required solve the question must precisely match the specified "search count'.
Unique Answer: The designed question must deterministic, leading to a single, unambiguous final answer. The clues each step must be precise enough to prevent solver from reasonably arriving at a different, valid conclusion.
The answer provided is: [answer).
You need create question that requires {n) searches.
When you have enough information to construct question, please first check whether the constructed question meets all requirements, especially whether the question is too sim- ple. After checking that all conditions are met, you need to provide the final constructed question in your final response, placing the final question between <question> and </question on> tags, for example: <question> </question>.
RAG Solver Prompt
Answer the given question based on the provided materials. You should first conduct very concise reasoning within 50 words, and then directly provide your answer without detailed illustrations after saying 'Answer'..
Materials: (materials)
Question: {question)
Solver Prompt (Search-R1
ZeroSearch Seeeid-RR/ZZrorSerrreee
Qwen series LLaMA 3.1)
You helpful and harmless assistant.
Answer the given question. You must conduct reasoning inside <think> and </think> first every time you get information. After reasoning, if you find you lack some knowl- edge, you can call a search engine by <search> query s/search> and will return the top searched results between <informations and s/information
You can search as many times as your want. If you find no further external knowledge needed, you can directly provide the answer inside <answer> and </answer>, without detailed illustra- tions. For example, <answer> Beijing </ </answer>. Question: [question)
Solver Prompt (R-Search)
You are helpful assistant that can solve the given question step by step. For each step, start by explaining your thought process. additional information needed, provide specific query enclosed in <search> and </search>, The system will return the top search results within Kobservations and </observation>, You can perform multi- ple searches needed. When you know the final answer, <original. evi dence> evidence> to provide all potentially relevant original information from the observations. Ensure the information is complete and preserves the original word- ing without modification. If no searches were conducted or observations were made, omit the evidence section. Finally, provide the final answer within <answer> and </ </answer> tags. (question)
and </original aarriinalleevideneeee
30

---

Published as a conference paper at ICLR 2026
L.LM-as -as--udg Prompt
You are a professional judge who evaluates the correctness of answers based on given crite- ria.
Please determine whether the model's answer is consistent with the reference answer:
Question: {question)
Model Answer: (prediction)
Reference Answer: (ground truth}
Evaluation Criteria:
1. The model answer must accurately respond the question and be consistent with the reference answer in meaning.
For numerical questions, the values must be equal or very close.
3. For textual questions, the core meaning must be correct.
Differences in wording or language are allowed as long as the core answer the same.
5. If the model answer includes the correct answer and does not contain conflicting information, is also considered correct.
Please respond only with "Correct" or "Wrong". Do not provide any additional explanation.
Question Difficulty Evaluator Prompt
You are search-problem difficulty evaluator. Your task: given single search-type question, return strict ISON containing only two fields:
"overall difficulty": an integer from 1 to 5 (1 easiest, 5 hardest) "reasoning": a short
explanation (1-2 sentences) describing why you gave that score, focusing on required rea- soning and expected search effort
DIFFICULTY SCALE:
Very simple factual questions
2: Basic factual questions requiring single search
3: Questions requiring moderate search or basic reasoning
4: Complex questions requiring multiple searches or reasoning steps
5: Very complex multi-step questions requiring extensive research and reasoning
Examples:
Easy valid question: "What process causes the continents drift apart?" Expected output: "roverall_difficurl 1, "reasoning": "Direct factual question requiring single-step search for well-known geological concept.")]
Hard valid question: "In the 19th century, a work published by a French writer sparked judicial proceedings for 'coruppiin public morals.' This work was subsequently adapted into multiple film versions. In the 1991 version, what flavor did the lead actress insist on using for the prop poison in the scene where her character dies by suicide?" Expected output: [(" ("overall difficulty": 5, "reasoning": "Requires complex multi-step reasoning: identifying the French work and writer, finding the 1991 film adaptation, then locating very specific shhind-the-scene. details about prop choices.") Here the question evaluate: Question: (question)
Output requirements - STRICT JSON only, no extra text or fields. Return only valid JSON.
F HACKING QUESTION CASES
The example in Table 13 illustrates significant problem with the proposer's question formulation: non-uniqueness of answers. The question "Temptations singer" is inherently ambiguous because the Temptations, as famous Motown group. had multiple singers throughout their history, including David Ruffin, Eddie Kendricks, Dennis Edwards, and many others. The question essentially asks for
31

---

Published as a conference paper at ICLR 2026
"a singer from band," which obviously has multiple valid answers. However, the RAG Solver has the documents that prominently feature one particular member (in this case, Otis Williams), making it appear if there's a single correct answer. While RAG-based solver succeed due to the limited and biased retrieval results, a solver working independently without access to these specific documents would likely struggle determine which of the many possible Temptations singers is the "correct" answer.
Table 13: Example of hacking questions: question with non-unique answers.
Proposer creates question that leads the Otis Williams
<think>l need to create question that leads to "Otis Williams" the answer.
Let's begin by scarching for 'associated' or "preceding' facts which connect to "Otis Williams" <</think>
<search> otis williams </search>
kinformations Doe (Title: "Otis Williams") Otis Williams Otis Williams (born Otis Miles Jr.; October 30, 1941) is an American baritone singer. Nicknamed ""Big Daddy"", he is occasionally also a songwriter and a record producer. Williams is best known as the founder and last surviving original member of the Motown vocal group The Temptations, the rights the Temptations name. Williams born Otis Miles, in Texarkana, Texas Otis Miles and Hazel Louise Williams ...
group in which he continues perform: he also
Doc (Title: "Charles Malik Whitfield") Charles Malik Whitfield Charles Malik Whitfield (born August 1, 1972) an American actor. Whitfield hest known his performance as Otis Williams the television miniseries, ""The Temptations" (1998), for which he was nominated for NAACP Image Award for Outstanding Actor a Television Movie or Mini-Series....
Doe 3 (Title: "Johnny Otis") Johnny Otis Johnny Otis (born Joannis Alexandres Veliotes; December 28, 1921 - January 17, 2012) was an American singer, musician, composer, arranger, bandleader, talent scout, disc jockey, record producer, television show host, artist, author, journalss,, minister, and impresario... ... f/inffrmaat in
<think>It appears that the most prominent "Otis Williams" refers singer from The Temptations, famous Motown group.</thinks
<question> Templations singer </question>
32

---

Published as a conference paper at ICLR 2026
Table 14: Example of hacking questions: question with non-unique answers (wrong answer).
Solver answers the question raised by Proposer: Temptations singer?
<think>l need find out which singer from the Temptations group. I'll search for ft.//hhink
<search> Temptations singer </search>
<information> Doc 1 (Tide: "Terry Weeks") Terry Weeks Terry Wayne Weeks (born December 23, 1963) is an American R&B and soul singer who is currently one of the lead singers of the legendary Motown... ...
Doc 2 (Title: "Otis Williams") Otis Williams Otis Williams (born Otis Miles Jr.; October 30, 1941) is American baritone singer. Nicknamed Big Daddy"", he is occasionally also a songwriter and a record producer. Williams
Doc 3 (Title "The Temptations") The Temptations The Temptations are an American vocal group who released series successful singles and albums with Motown Records during the 1960s and 1970s. Their work with producer Norman Whitfield, beginning with the Top 10 hit single ""Cloud Nine in October 1968, pioneered psychedelic soul, and was significant in the evolution of R&B and soul music. The band members are known for their chorcography, distinct harmonies, and dress style. Having sold tens of millions of albums, the Temptations are among the most successful groups in popular music. Featuring five male vocalists and dancers (save for brief periods with ... </ </informe t on?> vocal group. Now I need find which singer from the Tempta-
<think>l found that The Temptations tions.//think>>
<search> who is the singer in The Temptations </search>
<informations Doc 1 (Title: "David Ruffin") David Ruffin David Eli Ruffin (born Davis Eli Ruffin, January 18, 1941- 1, 1991) was an American soul singer and musician most famous for his work one of the lead singers of The Temptations (1964-68) during the group's ""Classic Five"" period as it was later known. He was the lead voice on such
"The Temptations") The Temptations The Temptations are an American vocal group who released a series of successful singles and albums with Motown Records during the 1960s and 1970s. Their work with producer Norman Whitfield, beginning with the Top 10 hit single ""Cloud Nine"" in October 1968, pionecred psychedelic soul, and was significant in the evolution of R&B and soul music. The band members are known for their choreography, distinct harmonies, and dress style. Having sold tens millions albums, the Temptations are among the most successful groups in popular music. Featuring five male vocalists and dancers (save for brief periods with...
Doe 2 (Title:
Doe (Title: "Otis Williams") Otis Williams Otis Williams (born Otis Miles Jr;; October 30, 1941) is an American baritone singer. Nicknamed ""Big Daddy"" he is occasionally also a songwriter and record producer. Williams.. s/information>
<think>l found out that David Ruffin, Otis Williams, and other singers were part of The Temptations. need to find the main or most famous singer from the group.</thing </thik>>
<search> most famous singer in The Temptations </search>
cinformations ton> Doc (Title: "David Ruffin") David Ruffin David Eli Ruffin (born Davis Eli Ruffin, Jamuary 18, 1941 June 1, 1991) was American soul singer and musician most famous for his work one of the lead singers of The Temptations (1964-68) during the group's ""Classic Five"" period as it later known. He was the lead voice on such ...
Doc (Title: "Otis Williams") Otis Williams Otis Williams (born Otis Miles Jr: October 30, 1941) an American baritone singer. Nicknamed ""Big Daddy ,", he is occasionally also a songwriter and a record producer. Williams Doc (Title: "The Temptations") The Temptations The Temptations are an American vocal group who released series of successful singles and albums with Motown Records during the 1960s and 1970s. Their work with producer Norman Whitfield, beginning with the Top 10 hit single ""Cloud Nine"" in October 1968, pioncered psychedelic soul, and was significant in the evolution of R&B and soul music. The band members are known for their choreography. distinct harmonies, and dress style. Having sold tens of millions of albums, the Temptations are among the most successful groups popular music. Featuring five male vocalists and dancers (save for brief periods with... </ c/informe on>
<think>l found out that the main singer of The Temptations is not explicitly sttttioned</thiikss
<answer> David Ruffin < /aanwwr>>
33

---

Published as a conference paper at ICLR 2026
Table 15: Example of hacking questions: question with non-unique answers (correct answer).
Solver answers the question raised by Proposer: Temptations singer?
<think>l need to find out which singer of the Temptations. I'll search for it,</think>>
<search> Temptations singer </search>
<information> Doc 1 (Tidl: "Terry Weeks") Terry Weeks Terry Wayne Weeks (born December 23, 1963) is an American R&B and soul singer who is currently one of the lead singers of the legendary Motown...
Doc 2 (Title: "Otis Williams") Otis Williams Otis Williams (born Otis Miles Jr: October 30, 1941) is an American baritone singer. Nicknamed Big Daddy"". he is occasionally also songwriter and a record producer... ...
Doe 3 (Title: "The Temptations") The Temptations The Temptations are an American vocal group who released series of successful singles and albums with Motown Records during the 1960s and 1970s. Their work with producer Norman Whitfield, beginning with the Top 10 hit single ""Cloud Nine "" in October 1968, pioneered psychedelic soul, and was significant in the evolution of R&B and soul music. The band members are known for their choreography, distinct harmonies, and dress style. Having sold tens of millions of albums, the Temptations are among the most successful groups in popular music. Featuring five male ... </information>
<think>I found out that The Temptations is vocal group. Now I need to find which singer of the Tempta- tions<//thin>>
<search> who is singer of the Temptations </search>
sinformation> Doc (Title: "The Temptations") The Temptations The Temptations are an American vocal group who released series of successful singles and albums with Motown Records during the 1960s and 1970s. Their work with producer Norman Whitfield, beginning with the Top 10 hit single ""Cloud Nine "" in October 1968, pioncered psychedelic soul, and was significant in the evolution of R&B and soul music. The band members are known for their choreography, distinct harmonies, and dress stylc. Having sold tens of millions of albums, the Temptations are among the most successful groups in popular music. Featuring five male vocalists and dancers (save for brief periods with..
Doe 2 (Title: "Otis Williams") Otis Williams Otis Williams (born Otis Miles Jr.: October 30. 1941) is an American baritone singer. Nicknamed ""Big Daddy"", he is occasionally also songwriter and a record producer...
Doc 3 (Titl: "The Temptations") Ten on either the United States Pop or R&B Albums Chart. The Temptations The Temptations
American vocal group who released series of successful singles and albums with Motown Records during the 1960s and 1970s. Their work with producer Norman Whitfield, beginning with the Top 10 hit single ""Cloud Nine" "M in October 1968, pioneered psychedelic soul, and was significant... </in s/informe format on> oon
<think>l found out that The Temptations is a vocal group. Now need to find a specific singer of the Tempta- </oni</
<search> singer of The Temptations </search>
<informations ion> Doe (Title: "Otis Williams") " Otis Williams Otis Williams (born Otis Miles Jr.: October 30, 1941) is an American baritone singer. Nicknamed "BBig Daddy"", he is occasionally also a songwriter and a record producer. Williams..
Doc 2 (Titl:: "The Temptations") The Temptations The Temptations are an American vocal group who released a series of successful singles and albums with Motown Records during the 1960s and 1970s. Their work with producer
Doe 3 (Title: "Paul Williams (The Temptations)") Paul Williams (The Temptations) Paul Williams (July 2, 1939 - Au- gust 17, 1973) was an American baritone singer and choreographer. Williams was noted for being one of the founding members and original lead singer of the Motown group The Temptations. Along with Elbridge "" "Al"" Bryant, Otis Williams, and fellow Alabamians Eddie Kendricks and Melvin Franklin, Williams was a member of The Temptations
</ //informate ion>
<think>l found out that the singers The Temptations include Otis Williams, Eddie Kendricks, and Paul Williams. Now I need to find a specific singer of the Temptations.</thit </think>
<search> temptations lead singer </ s/saarch>
<information Doe (Title: "Otis Williams") Otis Williams Otis Williams (born Otis Miles Jr. October 30, 1941) is an American baritone singer. Nicknamed ""Big Daddy"", he is occasionally also a songwriter and a record producer. Williams...
Doe 2 (Title: "Terry Weeks") Terry Weeks Terry Wayne Weeks (horn December 23, 1963) is an American R&B and soul singer who is currently one of the lead singers of the legendary Motown quintet The Temptations. Weeks was born in Birmingham, Alabama. Serving in the Air Force, including involvement with Tops In Blue for eight years. Upon his retirement in the early-1990s, Weeks began singing locally around his home ...
Doc (Title: "Terry Weeks") taking the place of Temptations lead Ali-Ollie Woodson, who was battling throat cancer. He would replace Woodson permanently in 1997, joining the Temptations lineup that included Williams, Ron Tyson, Theo Peoples and Harry McGilberry in 1996. Weeks sung co-lead on the ... </ s/info Lunformat ion>
<think>1 found out that the lead singer of The Temptations might be Otis Williams. Now I can provide the an- swer
swer.</think?
<answer> Otis Williams < answer>>
34


Reviews:
[{"review_id": "oKRMmK7QqN", "reviewer": "Reviewer_EH85", "summary": "The paper propose Search Self-Play (SSP), a self-play strategy to finetune LLMs to utilize multiturn search agents in response to a given question. The approach consists of training a single LLM to act both as task proposer and problem solver, in a zero-sum game fashion. The repeated game is constrained by making sure proposed tasks are meaningful and ground truth answers are correct via a RAG on the documents retrieved by the proposer’s trajectory. \nExperiments performed on Qwen 7B illustrate the benefit of the approach, leading to substantial gains on Q&A benchmarks. Convincing ablations show the benefit of co-training both the proposer and solver objective, thus motivating the efficacy of the overall approach.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles the very relevant and challenging problem of improving LLMs for search and information retrieval.\n- The zero-sum game formulation is very interesting but at the same time easily exploitable by both agents and thus I would have expect to produce degenerate solutions. Instead, the proposed constrained approach and overall meta-algorithm seems quite robust and able to produce stable improvements, as measured by the improved downstream performance during training.\n- The experiments are sound and different design choices have been ablated, such as training only the solver or the proposer.", "weaknesses": "- I don’t understand why the proposer and solver are fine tuned with different update rules, REINFORCE and GRPO respectively. Couldn’t the same algorithm, say GRPO, be applied to both? This was not clear to me.\n- Are both the proposer and solver updated at each step? I would expect more stable training dynamics to update the proposer less frequently. I would be curious to know the author’s view and experience on this. Also I think this would help understand the training dynamics. \n- It would be nice to analyze the curriculum of tasks proposed by the proposer. How does it change compared to when it is not trained? Are they too hard, too easy, or easily hackable by the solver? Ultimately, this is at the core of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Js9G6q2CTF", "reviewer": "Reviewer_BnWv", "summary": "This work introduces Search Self-Play (SSP), a reinforcement learning framework for training deep search agents without human supervision. The approach uses a single LLM in dual roles: as a question proposer that generates search queries with verifiable ground-truth answers, and as a problem solver that attempts to answer these questions through multi-turn search interactions. The key component is a RAG-based verification mechanism that ensures the proposed questions are answerable given the proposer's retrieved documents, preventing reward hacking. The authors formulate this problem as a cooperative competition scenario, where the proposer and solver collaborate during the RAG-based verification but compete through the proposer generating increasingly difficult questions and the solver learning to get better at answering difficult questions. \n\nExperiments demonstrate consistent improvements across seven QA benchmarks. The work addresses a significant bottleneck in RLVR (Reinforcement Learning with Verifiable Rewards): the need for large-scale, human-curated question-answer pairs, through a self-sustaining closed learning loop.\n\nHowever, the evaluation is limited to a Wikipedia-based retrieval with a 2018 corpus, raising questions about generalization to modern web search scenarios. Note that contemporary works have also used a similar evaluation strategy in the past. The paper lacks analysis on question quality evolution, computational costs, and comparison with recent synthetic data generation methods. The RAG verification mechanism increases the computational complexity of the method, warranting a comparison of training costs with prior works.", "questions": "1. Since the method is formulated as a co-evolution game, is there an equilibrium that can be reached here? What type of equilibrium exists in this setting?\n2. What is the composition of the pre-defined answer set D? How was it constructed, and does it cover diverse answers or long-form answers?\n3. What is the computational overhead of SSP compared to other works? Specifically, what percentage of generated questions pass the RAG verification as the training progresses, and how much additional compute is required for dynamic resampling/replay buffer?\n4. Can you provide a quantitative analysis of question diversity over training, e.g., topic distribution and question type distribution? I am particularly interested in the question of diversity when the solver accuracy decreases. Is it a case of reward hacking from the perspective of the proposer?\n5. How does SSP compare empirically to recent offline synthetic data generation methods (WebSailor and WebDancer) in terms of both final performance and data efficiency?\n6. Could you evaluate the method on newer, more challenging datasets such as BrowseComp?\n7. Do you observe any correlation between the proposer's search trajectory length and the solver's success rate? In other words, do more complex proposer searches lead to harder questions?\n8. Curiosity Question: Do you have any insights on why dynamic resampling performs worse than Replay Buffer? Intuitively, Dynamic Resampling should be performing better even if it incurs higher computational costs. Please correct me if I am wrong here.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel Problem Formulation: The paper addresses a fundamental challenge in agentic RL training, Data Scarcity, through an elegant self-play mechanism that grounds question generation in external search rather than relying solely on the model's internal knowledge.\n2. Thorough Ablation Studies: The paper provides valuable insights through ablations on training schemes (co-evolution vs fixed-opponent), batch completion strategies, RAG verification configurations, and reward design, with detailed training dynamics analysis.\n3. Strong Empirical Results: The proposed method shows consistent improvements across all seven benchmarks, with particularly impressive gains on base models.\n4. Robust Verification Mechanism: The RAG-based verification with noisy document injection is a principled approach to prevent reward hacking and ensure question quality, though it adds computational overhead.", "weaknesses": "1. Limited Evaluation Setting: All experiments use a local E5 retriever with a Wikipedia 2018 corpus, which is significantly more constrained than the actual web search scenarios. The paper doesn't evaluate whether SSP-trained agents generalize to real web search or more recent knowledge bases. The recently released BrowseComp dataset is a good candidate for evaluation.\n2. Lack of question quality analysis: While the paper demonstrates that the proposer generated increasingly difficult questions (as seen by declicing solver rewards in Figure 3a), there is no systematic analysis of question quality, diversity, or naturalness. It is unclear if SSP can generate diverse questions or if it leads to a narrow distribution of question types.\n3. Missing Computational Cost analysis: The paper doesn't report training time, computational costs, or the overhead introduced by RAG verification and dynamic sampling. Given that invalid questions require re-generation and each question needs verification, the actual compute requirements could be substantially higher than standard RLVR, but this is not quantified.\n4. Insufficient Comparison with Recent Synthetic Data Methods: While the paper mentions WebSailor (Li et al., 2025) and WebDancer (Wu et al., 2025) in related work, there's no empirical comparison. These methods also generated synthetic training data offline; a direct comparison would clarify SSP's advantages.\n5. Ground Truth Answer Set: The paper states that ground truth answers are drawn from a pre-defined set D, but never specifies what this set contains, its size, or how it was constructed. This is a critical detail as it determines the scope and diversity of possible training questions.\n6. Limited Analysis of Long-Term training dynamics: Training runs for only 150-200 steps. The paper doesn't investigate whether performance saturates and whether question difficulty continues to increase appropriately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WHLBiB0Bs9", "reviewer": "Reviewer_TBC9", "summary": "The paper introduces Search Self-Play (SSP), a reinforcement learning framework where LLM takes on two alternating roles: a “proposer” that creates challenging, search-intensive QA tasks, and a “solver” that attempts to answer them. To ensure the generated questions are valid and have a unique, verifiable answer, a RAG-based verification gate is used, which relies on the proposer’s retrieved documents to validate the questions before adversarial training begins. The solver is trained using GRPO, while the proposer is optimized with REINFORCE. Experiments across seven open-domain QA benchmarks and various model architectures and sizes consistently demonstrate that SSP delivers significant performance improvements.", "questions": "5. Provide results using real-world web content (e.g., Bing/Google APIs) and different corpora/embeddings retrievers.\n\n6. Improve Figure 2 to more clearly reflect the pipeline. The current Figure obscures the implementation details. Consider redrawing it to mirror Algorithm 1 step-by-step (proposer rollout → rule checks → RAG verification with noise → solver rollouts → rewards/updates).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work is novel in its application of self-play to agentic search.\n\n2. The paper provides precise definitions of constraints and rewards, fully discloses prompts and hyperparameters, and includes training curves that illustrate the co-evolution of proposer and solver.\n\n3. By reducing reliance on annotated agentic data, SSP achieves consistent, substantial improvements across different model architectures and scales, paving a practical path toward scalable, unsupervised training for agentic tasks.", "weaknesses": "1. Please discuss training stability under the adversarial/cooperative setup. Does the min–max dynamic cause frequent collapses or mode drift? How easy is convergence in practice? Please provide stability evidence (multiple seeds, valid-question rate over time, reward variance) and, if available, theoretical or empirical guarantees that prevent reward hacking and proposer entropy explosions.\n\n2. Please report end-to-end compute and cost: total wall-clock, GPU hours, tokens processed, search calls, and energy. If SSP is substantially more resource-intensive, justify why not to to paid human annotation via crowd platforms.\n\n3. Clarify the rationale for using REINFORCE as the proposer’s advantage estimator and GRPO for the solver. Why is this asymmetric choice preferable, and how do performance/variance trade-offs compare to alternatives (e.g., PPO, GPRO, REINFORCE)?\n\n4. Evaluation heavily on a single LLM-as-judge and 500-sample subsets, which may introduce bias and variance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JKa5jucarM", "reviewer": "Reviewer_tjcn", "summary": "The paper proposes Search Self-Play (SSP), a novel reinforcement learning framework aimed at improving deep search agents without human supervision. SSP involves two roles for a single LLM: a task proposer that generates challenging search queries, and a problem solver that attempts to answer these queries using multi-turn search interactions. The framework ensures the correctness of queries by employing a retrieval-augmented generation (RAG) process to verify ground-truth answers. Experimental results demonstrate that SSP improves agent performance across various benchmarks, including NQ, TriviaQA, and HotpotQA, both in from-scratch and continuous training setups.", "questions": "* Clarify how the answer set (D) is built (source, size, type distribution) and report overlap stats with each benchmark, plus de-dup/blacklist procedures.\n* Add EM/F1 alongside pass@1, or use heterogeneous judges from different families, and report inter-judge agreement.\n* Expand evaluation to newer search/browsing benchmarks (e.g., BrowseComp; SimpleQA if applicable).\n* Why is there no reward for SSP Proposer in Figure 3(a)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper makes a significant contribution by introducing SSP as a self-supervised training method for search agents. The originality lies in the combination of search agent roles with self-play, where the proposer and solver co-evolve. This approach represents a promising direction for training LLMs autonomously and addresses key challenges such as task synthesis and scalability. The empirical results across a diverse set of benchmarks, particularly with the SSP framework applied to various LLMs, demonstrate the method's potential for future agentic training.", "weaknesses": "* The method samples ground-truth answers from a pre-defined set (D) to drive self-play but does not disclose (D)’s source, size, or curation, nor its overlap with evaluation answers; this raises potential data-leakage/overfitting concerns. Please report (D)’s provenance, scale, overlap with each benchmark, and blacklist/de-dup procedures.\n\n* Using a solver **without tools** as the verifier is an unreliable proxy for question validity: failures may reflect the verifier’s limited contextual/multi-hop reasoning (or added RAG noise) rather than flaws in the proposed question, causing valid items to be wrongly rejected.\n\n* Evaluation relies on a single judge (Qwen2.5-32B-Instruct) with pass@1 while several tested systems are Qwen-family models, inviting family bias and prompt-format gaming; adding exact match (EM) and F1 would provide more robust evidence.\n\n* Strong results on NQ/TriviaQA/HotpotQA/2Wiki/MuSiQue/Bamboogle notwithstanding, coverage skews to classic QA; testing on newer search benchmarks (e.g., BrowseComp and SimpleQA) would better demonstrate robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)