Towards Agentic Self-Learning LLMs in Search Environment

## Abstract

Abstract We study whether self-learning can scale LLM-based agents without relying on human-curated datasets or predefined rule-based rewards. Through controlled experiments in a search-agent setting, we identify two key determinants of scalable agent training: the source of reward signals and the scale of agent task data. We find that rewards from a Generative Reward Model (GRM) outperform rigid rule-based signals for open-domain learning, and that co-evolving the GRM with the policy further boosts performance. Increasing the volume of agent task data—even when synthetically generated—substantially enhances agentic capabilities. Building on these insights, we propose Agentic Self-Learning (ASL), a fully closed-loop, multi-role reinforcement learning framework that unifies task generation, policy execution, and evaluation within a shared tool environment and LLM backbone. ASL coordinates a Prompt Generator, a Policy Model, and a Generative Reward Model to form a virtuous cycle of harder task setting, sharper verification, and stronger solving. Empirically, ASL delivers steady, round-over-round gains, surpasses strong RLVR baselines (e.g., Search-R1) that plateau or degrade, and continues improving under zero-labeled-data conditions, indicating superior sample efficiency and robustness. We further show that GRM verification capacity is the main bottleneck: if frozen, it induces reward hacking and stalls progress; continual GRM training on the evolving data distribution mitigates this, and a small late-stage injection of real verification data raises the performance ceiling. This work establishes reward source and data scale as critical levers for open-domain agent learning and demonstrates the efficacy of multi-role co-evolution for scalable, self-improving agents. The data and code of this paper are released at https://github.com/forangel2014/Towards-Agentic-Self-Learning

## 1 Introduction

Autonomous agents built on Large Language Models (LLMs) have evolved rapidly, from early prompt-engineering pipelines to recent reinforcement learning–trained systems capable of self-improvement in interactive environments.
Early work such as ReAct-style reasoning–acting frameworks and reflective methods constructed agent loops through manually designed prompt templates, memory modules, and decision-making heuristics Shinn et al.; Yao et al. (2022); Sun et al. (2023).
These policy-evolution approaches refine the agent’s reasoning process via self-reflection, co-adaptation, or prompt rewriting, and can effectively integrate with other inference-time enhancing technique such as Retrieval-Augmented Generation (RAG).

A second wave of research extends the RL with Verifiers and Rewards (RLVR) paradigm, pioneered in reasoning-focused models such as OpenAI o1 Jaech et al. (2024) and DeepSeek-R1 Guo et al. (2025), to the agent setting. Works like WebAgent-R1 Wei et al., Search-R1 Jin et al. (2025) and R1-Searcher Song et al. (2025) train LLM-based agents end-to-end with reinforcement learning in interactive environments through rule-based reward signals, enabling them to continuously refine planning, tool use, and long-horizon strategies without relying solely on static instruction-tuned datasets.

In parallel, there has been a growing trend toward *self-learning Large Reasoning Models (LRMs)*, which aim to improve reasoning ability through reinforcement learning, but without relying on large quantities of annotated human data.
Within this paradigm, methods such as Absolute-Zero Zhao et al. and R-Zero Huang et al. can achieve self-evolution LRMs under zero-labeled-data conditions.

However, most existing approaches still rely heavily on human-annotated datasets or rule-based environments (e.g., code interpreters) to provide verifiable rewards within the RLVR paradigm.
Such dependence significantly constrains the potential of LLM agents to scale the self-learning paradigm to open-domain settings, where reliable, automatically verifiable feedback is scarce and task structures are less formalized.

Motivated by these developments, we seek to explore whether the self-learning paradigm can be effectively extended to scale the training of agentic systems, freeing them from reliance on both human-curated datasets and predefined rule-based reward functions.
To this end, we begin by conducting controlled experiments in the *search-agent* setting introduced by Search-R1 Jin et al. (2025), in order to investigate two key factors for LLM-based agentic reinforcement learning: the *source of reward signals* (§[3.1](https://arxiv.org/html/2510.14253v2#S3.SS1.SSS0.Px1)) and the *scale of agent task data* (§[3.1](https://arxiv.org/html/2510.14253v2#S3.SS1.SSS0.Px2)).
Our findings reveal that:

- •
Compared to rigid, rule-based rewards, signals produced by a *Generative Reward Model* (GRM) are more effective for scaling LLM agents to open-domain scenarios. Moreover, when the GRM is co-evolved with the policy model, thereby progressively acquiring stronger generative discrimination capabilities, agent performance improves further.
- •
Increasing the quantity of agent task data — even when tasks are synthetically generated — can substantially enhance LLMs’ agentic capabilities.

Figure: Figure 1: The framework of Agentic Self-Learning (ASL).
Refer to caption: x1.png

Based on these observations, we present Agentic Self-Learning (ASL) — a fully closed-loop, multi-role reinforcement learning framework that unifies task generation, policy execution, and evaluation within a shared tool environment and LLM backbone. ASL is built upon three mutually reinforcing functional roles: the *Prompt Generator*, the *Policy Model*, and the *Generative Reward Model*.
Through iterative optimization of these three roles, ASL embodies an agentic, self-improving loop: generating diverse and adaptively challenging tasks, solving them, and refining evaluation criteria in tandem. This design not only mitigates the need for external supervision but also enables scalable, open-domain learning that is robust against reward hacking.

To empirically evaluate the efficacy of PURM, we conduct a series of experiments to answer the following three questions:

Can ASL effectively train the LLM agent compared to existing methods?
ASL delivers steady, iteration-over-iteration gains and ultimately surpasses strong RLVR method such as Search-r1, which excels early but quickly plateaus and degrades. Other self-learning methods (e.g., Absolute Zero, R-zero) also show early gains followed by stagnation. In contrast, ASL continues improving even under zero-data conditions, highlighting its superior sample efficiency and robustness.

Does the three roles in ASL co-evolve during the training process?
We observe a clear closed-loop synergy: the Prompt Generator produces progressively harder questions, the Generative Reward Model improves its verification accuracy each round, and the Policy Model steadily increases task accuracy. This virtuous cycle—harder question-setting, sharper verification, and stronger solving—confirms coordinated co-evolution.

What factors might limit the ongoing evolution of ASL and how can we mitigate them?
The main bottleneck is the GRM’s verification capacity. When the GRM is not updated, the Prompt Generator learns to exploit scoring blind spots, triggering reward hacking and stalling Policy Model progress. Continual GRM training on the evolving distribution mitigates this failure mode and sustains improvement over longer horizons. Moreover, injecting a small amount of real verification data further lifts the performance ceiling, indicating that ASL’s upper bound is effectively set by GRM capability. Practically, a two-phase strategy works well: rely on self-generated data to continually calibrate the GRM, then apply a modest real-data calibration late in training to refresh the ceiling and unlock additional gains.

In summary, this paper makes the following foundational contributions:

- •
Through controlled experiments, we identify the *source of reward signals* and the *scale of agent task data* as two critical factors for scaling LLM agents in open-domain settings.
- •
We propose ASL, the first multi-role closed-loop agentic self-learning framework, enabling simultaneous co-evolution of task generation, problem solving, and evaluation.
- •
Through experiments, we demonstrated that ASL can effectively coordinate the improvement of problem generation, solving, and verification capabilities over multiple rounds of iteration; meanwhile, our validation results revealed that the capability boundary of GRM is an important factor limiting the upper bound of ASL, and we further proved that this upper bound can be raised by training GRM with either synthetic or real data.

## 2 Related Work

Policy Evolution. Currently, the development of self-evolving agents focuses primarily on two key aspects. One area emphasizes the innovation of internal learning mechanisms to drive agent evolution. This aspect is centered on optimizing the agent’s internal learning, decision-making, and adaptation processes, typically without relying on structured adjustments to the external task environment. For instance, Reflexion Shinn et al. introduces a natural language feedback mechanism that replaces traditional weight updates, enhancing decision-making capabilities through error-driven self-reflection memory. Mutual-Taught Shi et al. (b) employs the Expectation-Maximization (EM) algorithm to jointly optimize the policy model and reward model, allowing for collaborative adaptation to distribution shifts. AgentEvolver Belle et al. adopts a multi-role collaboration mechanism, which enables agents to rewrite prompts and decision codes independently without human intervention and further promotes the autonomous optimization of internal mechanisms. The essence of these methods lies in improving the agent’s abilities through enhancements in internal training algorithms, feedback mechanisms, or data generation strategies.

Task Adaptation. Another research focus is the dynamic evolution of task difficulty and structure. A meticulously designed task environment serves as a powerful driving force for evolution (Gao et al.). AlphaEvolve Novikov et al. proposes an evolutionary code generation framework that utilizes multi-round collaboration among large language models (LLMs) and difficulty cascading to optimize algorithm design while the Self-Challenging Agent (SCA) Zhou et al. introduces a "code-as-task" framework that achieves a leap in tool invocation capabilities through the generation of self-verifiable tasks. TaskCraft Shi et al. (a) is dedicated to the automatic generation of complex multi-tool combination tasks, systematically honing the reasoning and planning abilities of the agents. While these methods have demonstrated significant effectiveness in structured tasks and coding, they often fall short in addressing complex semantic understanding and judgment scenarios in open-domain question-answering texts. To bridge this gap, our work presents a self-evolving agent oriented toward question-answering texts, with the core innovation of using distribution entropy as both a measure of difficulty and an evolution reward signal, thereby constructing a closed-loop task evolution engine.

Agent-R1. Traditional large language models (LLMs) rely on human-designed workflows. In contrast, end-to-end reinforcement learning (RL) methods empower agents to autonomously make decisions and perform actions by learning optimal policies through interactions with dynamic online environments. WebAgent-R1 Wei et al. implements multi-round interactive end-to-end optimization to support scalable online interactive learning within complex network environments. NB-Agent Zhang et al. integrates a code-driven interaction paradigm with distributed reinforcement learning training, and further introduces sandbox-based acceleration techniques and memory optimization strategies to achieve efficient large-scale online training. Furthermore, Absolute Zero Zhao et al. and R-Zero Huang et al., which leverage Reinforcement Learning with Verifiable Rewards (RLVR), generate and solve tasks respectively through self-play mechanisms and two-role adversarial evolution, breaking through the dependence on human-labeled data and promoting the self-evolution. The deep integration of reinforcement learning with intelligent agents opens up new possibilities for the autonomous completion of long-term and complex tasks.

## 3 Agentic Self-Learning

We propose Agentic Self-Learning (ASL), a multi-agent self-improvement framework designed to enable large language models (LLMs) to continuously evolve their reasoning, generation, and evaluation capabilities in an autonomous closed loop.
We first introduce two key factors we have identified that influence RL training for LLM agents: the scale of agent task data and the source of reward signals (§[3.1](https://arxiv.org/html/2510.14253v2#S3.SS1)). We then present the three key components of the ASL framework—the prompt generator, the policy model, and the reward model (§[3.2](https://arxiv.org/html/2510.14253v2#S3.SS2)), and finally describe the training procedure of the ASL framework (§[3.3](https://arxiv.org/html/2510.14253v2#S3.SS3)).

Figure: Figure 2: Performance comparison between rule-based, fixed GRM, and policy-shared GRM reward functions. A co-evolving GRM yields the best reward quality and downstream policy performance.
Refer to caption: x2.png

### 3.1 Empirical Motivation for ASL Design

Before finalizing the ASL architecture, we conducted two preliminary studies to investigate key design choices in agentic RL with verifiable rewards, which directly motivate our proposed multi-role closed-loop framework. Following the settings in Search-R1 (later introduced in §[4.1](https://arxiv.org/html/2510.14253v2#S4.SS1)), we select Qwen-2.5-7B-instruct as the base model and train and test the policy models in search-based question-answering environment. We first conducted the following two controlled experiments to identify the *source of reward signals* (§[3.1](https://arxiv.org/html/2510.14253v2#S3.SS1.SSS0.Px1)) and the *scale of agent task data* (§[3.1](https://arxiv.org/html/2510.14253v2#S3.SS1.SSS0.Px2)) as two critical factors for scaling LLM agents in open-domain settings.

#### (1) The Source of Reward Signals.

In RLAIF-style settings, the choice of reward function critically influences policy optimization. We compared the following three variants:

- •
Rule-based Reward: answers are judged using substring exact match with gold answers.
- •
Fixed Generative Reward Model: the base policy model (reference model) acts as a agentic judge producing rewards.
- •
Policy as Reward Model: the generative reward model shares parameters with the policy model so the generative reward model can generalize from improvements in the policy model’s agent capabilities to achieve stronger agentic verification ability.

As shown in Figure [2](https://arxiv.org/html/2510.14253v2#S3.F2), both generative reward settings outperform purely rule-based rewards, confirming that a generative judge can provide more informative and tolerant reward signals. Furthermore, the GRM sharing parameters with the policy model achieves the highest and keeping growing accuracy, suggesting that generative reward models with stronger agentic verification ability are even better reward functions for LLM agentic RL. This empirical evidence underpins our decision to include a trainable policy-shared GRM as a core role in ASL.

#### (2) The Scaling of Agentic Data.

We next investigated whether scaling the amount of *agent-generated* training data benefits policy learning when using a GRM as the reward function. Starting from the same meta prompts (Appendix [10](https://arxiv.org/html/2510.14253v2#A1.F10)), we generate varying quantities of search-based QA tasks (1k, 10k and 46k examples) and accordingly train the LLM agent on them.

Results in Figure [3](https://arxiv.org/html/2510.14253v2#S3.F3) show a consistent improvement with increased generated data volume. This finding motivates ASL’s closed-loop Prompt Generator design — by autonomously producing high-quality, diverse tasks at scale, ASL continuously supplies the policy with valuable new training data without human annotation cost.

These two findings jointly justify the ASL multi-role design:
(1) a co-evolving GRM is superior to fixed or rule-based rewards, and
(2) enabling the agent to scale high-quality, verifiably judged tasks is a viable path to sustained performance growth.

Figure: Figure 3: Effect of scaling the amount of GRM-rewarded agentic data on downstream accuracy. Larger amounts of verifiably judged agent-generated data substantially improve policy performance.
Refer to caption: x3.png

### 3.2 Constituent Roles

#### Prompt Generator.

Conditioned on a *meta prompt*, the Prompt Generator is responsible for producing new training tasks in the form of problem–solution pairs $(x,a)$, where $x$ denotes the synthesized question and $a$ denotes its reference or ground-truth answer. In addition to producing initial tasks, the Prompt Generator evolves the task distribution over iterations, thereby adaptively increasing task complexity to match the model’s learned capabilities.

#### Policy Model.

Given a task $x$, the Policy Model generates a candidate solution $y$. This role essentially models the problem-solving capability of the system and is the primary target for performance improvement. Its outputs are subsequently evaluated by the Generative Reward Model to obtain scalar feedback signals.

#### Generative Reward Model.

Provided with a problem $x$ and a generated solution $y$, the Generative Reward Model produces a correctness score $s\in\{0,1\}$. The generative reward model itself is also trained via RL to improve its ability to faithfully and consistently assess outputs.

### 3.3 Training Process

ASL operates in an iterative cycle that progresses through RL training the three roles in sequence:
$\text{Prompt Generator}\rightarrow\text{Generative Reward Model}\rightarrow\text{Policy Model}\rightarrow\text{Prompt Generator}\dots$. At each stage, one role is actively optimized while the others provide fixed contextual behavior within the loop. Assuming we are at the $t$th iteration:

#### Phase 1: Prompt Generator Training.

Given a meta prompt $u$ from Prompt Generator training data $D_{PG}^{(t-1)}$ that is constructed in $t-1$ iteration, the Prompt Generator synthesizes a batch of $N$ candidate problem–answer pairs $\{(x_{n},a_{n})\}_{i=1}^{N}$. Then, for illustrative purposes, consider a single pair $(x,a)$ in these generated questions.

As the Prompt Generator may generate questions that beyond the verification ability of the Generative Reward Model, we first evaluate its validity using the current Generative Reward Model: the GRM performs $K$ independent rollouts to assess the correctness of $(x,a)$, producing a set of scores $\{s^{k}\}_{k=1}^{K}$. The average score $\bar{s}$ is compared against a predefined threshold $C$. Only if $\bar{s}>C$ is $(x,a)$ retained as a valid problem instance, otherwise, $(x,a)$ will be discarded.

For each valid $(x,a)$, we then store the pair into the Policy Model training set $D_{PM}^{(t)}$, ensuring that high-quality tasks contribute to downstream policy learning. Additionally, to compute the reinforcement reward signal for the Prompt Generator, we let the current Policy Model attempt to solve $x$ via $M$ rollouts, yielding a set of responses $\{y_{m}\}_{m=1}^{M}$. The GRM then scores each response, resulting in corresponding verification scores $\{s_{m}\}_{m=1}^{M}$.
These scores play a dual role:

- •
We add the triples $\{(x_{m},y_{m},s_{m})\}_{m=1}^{M}$ to the GRM’s training data $D_{GRM}^{(t)}$ of $t$th iteration.
- •
We measure the entropy of this score distribution,$r_{\mathrm{PG}}=\mathrm{Entropy}(s_{1},s_{2},\dots,s_{m})$ and use it as the reward signal to update the Prompt Generator’s parameters. Intuitively, higher entropy indicates that the task elicits diverse performance from the Policy Model, reflecting informative and discriminative task design.

#### Phase 2: Generative Reward Model Training.

In this phase, the Generative Reward Model is optimized to produce evaluation scores that are both consistent and faithful to verifiable correctness. Given a problem–solution pair $(x,y)$ the GRM performs $N$ independent rollouts, yielding a set of predicted scores $\{\hat{s}_{n}\}_{n=1}^{N}$. For each predicted score $\hat{s}$, we compare it against the reference score $s$. This comparison produces a binary correctness indicator for each rollout, reflecting whether the GRM’s judgment matches the reference evaluation.

This binary correctness signal is then served as the reward for GRM $r_{GRM}$ and used to update the GRM’s parameters via reinforcement learning, following the RLVR (Reinforcement Learning with Verifiable Rewards) paradigm. By grounding its learning signal in rule-based verification, the GRM incrementally improves its ability to assign precise and reliable scores to diverse model outputs, which in turn strengthens downstream policy optimization.

#### Phase 3: Policy Model Training.

In this phase, the Policy Model is trained to improve its problem-solving ability on tasks generated by the Prompt Generator. For a given problem $x$, the Policy Model performs $N$ independent rollouts, producing a set of candidate answers $\{y_{n}\}_{n=1}^{N}$. Each answer is then evaluated by the Generative Reward Model (GRM), yielding a corresponding set of scores $\{s_{n}\}_{n=1}^{N}$.
These scores play a dual role:

- •
Reinforcement signal for the Policy Model — the collection $\{s_{n}\}_{n=1}^{N}$ is directly used as the reward signal $r_{PM}$ in the Policy Model’s RL optimization objective, encouraging it to maximize expected performance over the task distribution.
- •
Difficulty feedback for the Prompt Generator — we compute the average score
$\bar{s}=\frac{1}{N}\sum_{n=1}^{N}s_{n}$. If $\bar{s}>0.5$, the task is considered too easy for the current Policy Model; we then assign a difficulty flag $f=\text{HARDER}$, indicating that the Prompt Generator should increase task difficulty in subsequent iterations. Conversely, if $\bar{s}\leq 0.5$, the task is considered too challenging, and we set $f=\text{EASIER}$, signaling that the Prompt Generator should generate simpler variants.
Finally, each triplet $(x,a,f)$, where $a$ is the reference answer of $x$, is stored into the Prompt Generator’s training dataset $D_{PG}^{(t)}$ for use in the next iteration.

### 3.4 Closed-loop Evolution

ASL will continuously cycle through the three training phases above for self-learning. Phase 1 will stop when the number of $D_{PM}^{(t)}$ reaches a certain threshold, and Phases 2 and 3 will stop when their respective training rewards converge.
Through the cyclic optimization of these three roles, ASL implements a self-reinforcing curriculum: the Prompt Generator gradually proposes more challenging tasks, the Policy Model adapts to solve them more effectively, and the Generative Reward Model continuously refines its evaluation criteria. Since all roles share parameters, skill improvements in one role naturally benefit the others, accelerating overall capability enhancement. Over time, ASL enables the LLM to autonomously expand both its problem-solving range and evaluation robustness without any direct human supervision.

## 4 Experiments

In this section, we first introduce the settings used in our experiments (§[4.1](https://arxiv.org/html/2510.14253v2#S4.SS1)), then we conducted a series of experiments to answer the following three research questions:

- •
RQ1: Can ASL effectively train the LLM agent compared to existing methods? (§[4.2](https://arxiv.org/html/2510.14253v2#S4.SS2))
- •
RQ2: Does the three roles in ASL co-evolve during the training process? (§[4.3](https://arxiv.org/html/2510.14253v2#S4.SS3))
- •
RQ3: What factors might limit the ongoing evolution of ASL and how can we mitigate them? (§[4.4](https://arxiv.org/html/2510.14253v2#S4.SS4))

### 4.1 Settings

Models and Framework. Following the experimental setup adopted in Search-R1 Jin et al. (2025),
we conduct experiments with Qwen-2.5-7B-Instruct. For the retrieval component, the 2018 Wikipedia dump (Karpukhin et al., 2020) is used as the knowledge corpus, and we employ the E5 retriever (Wang et al., 2022). The system prompts that used for format and tool description can be found in Appendix [A](https://arxiv.org/html/2510.14253v2#A1).
The number of retrieved passages is fixed to 3 across all methods. For the training framework, we implement the agentic reinforcement learning component based on VeRL Sheng et al. (2024). The user prompts and generation examples of three roles can be found in Appendix [A](https://arxiv.org/html/2510.14253v2#A1),[B](https://arxiv.org/html/2510.14253v2#A2).

Data. We conduct evaluation on seven widely-used benchmark datasets, covering both general and multi-hop question answering scenarios. The general QA category includes Natural Questions (NQ) Kwiatkowski et al. (2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2022). The multi-hop QA category consists of HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), and Bamboogle (Press et al., 2022).
These datasets span a diverse set of search and reasoning challenges, enabling a comprehensive assessment of model performance across both in-domain and out-of-domain settings.

Baselines. We select the following agent training and self-evolving methods for comparison.

- •
Search-R1 Jin et al. (2025) Search-R1 directly brought the RLVR paradigm used in OpenAI o1 Jaech et al. (2024) and DeepSeek-R1 Guo et al. (2025) to the training scenario of search agent.
- •
Absolute Zero Zhao et al. Absolute Zero proposed a Proposer-Solver self-play framework. It proposes to encourage Proposer to generate low (but non-zero) success rate questions and verify the correctness of Solver through Python execution. We implement the Absolute Zero training in our agentic search-based QA setting.
- •
R-Zero Huang et al. R-Zero proposed a Challenger-Solver co-evolving framework. It leverages an uncertainty reward and repetition penalty to guide the training of Challenger and trains the Solver through RLVR. We implement the R-Zero training in our agentic search-based QA setting.

### 4.2 ASL vs baselines

Figure: Figure 4: The test accuracy of LLM agents trained by ASL and baseline methods.
Refer to caption: x4.png

We first compare ASL with other agent training methods. As shown in Figure [4](https://arxiv.org/html/2510.14253v2#S4.F4), Search-r1, an RLVR approach trained on real data, achieves the best performance early in training but then quickly converges and degrades due to poorer generalization. Absolute Zero and R-zero show similar behavior: they make substantial gains in the first two iterations with Proposer/Challenger generating harder questions are Solver effectively learn from the tasks, but they then stall and enter a plateau in the third iteration and later. In contrast, ASL steadily improves the agent’s capability over multiple iterations and ultimately surpasses search-r1 even under zero-data conditions.

### 4.3 Synergistic Co-evolution of the Three Roles

We track how the three roles in ASL—Prompt Generator, Generative Reward Model, and Policy Model evolve across iterations. As shown in Figure [5](https://arxiv.org/html/2510.14253v2#S4.F5), the closed-loop training leads to the following coordinated improvements:

Question generation strength: In each iteration, we sample 2,000 generated questions and have the same fixed evaluator (Search-r1 trained for 1,024 steps) solve them. The blue bars plot this accuracy. As iterations progress, the accuracy generally declines, indicating that the generator produces increasingly challenging problems that better stress a fixed solver.

Verification strength: We build a verification test set from the original test set by representing each ground-truth pair $(q,a^{+})$ as $(q,a^{+};correct)$, and using GPT-4o-mini to synthesize a wrong answer $a^{-}$, yielding $(q,a^{-};wrong)$.
The green bars show the GRM’s accuracy on this verify set, which rises with each iteration, evidencing stronger discrimination between correct and incorrect solutions.

Solving strength: The yellow bars report the policy model’s task solving accuracy. This steadily increases across iterations, showing that the solver benefits from harder questions and sharper supervision from the verifier.

Together, these trends demonstrate that ASL drives a virtuous cycle: stronger question-setting and stricter verification improve the training signal, which in turn elevates solving performance; the improved solver and verifier then enable the generator to craft even harder questions. ASL thus advances question generation, judging, and solving in tandem.

Figure: Figure 5: Comparison across iterations. Blue: accuracy of a fixed evaluator (Search-r1, trained 1,024 steps) on 2,000 questions generated in each iteration—lower values indicate harder generated problems. Green: GRM accuracy on the verify set constructed from $(q,a^{+};correct)$ and GPT-4o-mini–derived negatives $(q,a^{-};wrong)$—higher is better judgment. Yellow: policy model solving accuracy—steadily improving. These patterns show the three roles co-evolve under ASL.
Refer to caption: x5.png

### 4.4 Identifying and Lifting the Ceiling of ASL

Identify reward hacking. An important factor limiting RL scaling is reward hacking. To investigate whether the hacking of policy model towards GRM is the bottleneck in ASL’s evolution, we ablate the GRM training phase and plot the resulting training and testing curves (Figure [6(a)](https://arxiv.org/html/2510.14253v2#S4.F6.sf1)). In the early iterations, the system exhibits a healthy co-evolutionary dynamic: the Prompt Generator (PG) progressively proposes harder problems, while the Policy Model (PM) correspondingly improves its solving ability during its training phase. Each time the training stage switches from one component to the other, we observe a sharp drop in rewards. This drop reflects the sudden increase in the counterpart’s capability—PG confronts a stronger solver, and PM faces more challenging prompts—exactly the kind of “difficulty escalates, ability catches up” rhythm one expects from a well-structured training loop.

However, by the third cycle a failure mode emerges. Because the GRM is not explictly trained and therefore has limited verification ability, the PG learns to generate problems that are excessively difficult or even unsolvable. These prompts lie far outside the GRM’s calibration range, causing its scores on such out-of-distribution inputs to become effectively random. The resulting high uncertainty inflates the entropy-based reward signal, which the PG then exploits to obtain spuriously high returns. This is a textbook instance of reward hacking: the reward becomes decoupled from true problem quality or solvability, and the PG is incentivized to game the scoring mechanism rather than improve the curriculum.

Consequently, the PM cannot extract meaningful learning signal from unsolvable prompts and fails to make further progress, leading to a plateau in test performance. Taken together, these dynamics indicate that a non-updating GRM not only becomes the bottleneck for ASL’s evolution but also opens the door to exploitative behaviors, underscoring the necessity of co-training or continual calibration of the reward model to track the evolving data distribution.

Figure: (a) The training and testing curve of ASL while GRM is NOT trained. The system exhibits a healthy co-evolutionary dynamic in early iterations but then trapped in reward hacking.
Refer to caption: x6.png

Mitigating reward hacking. When we reintroduce the GRM training phase into the ASL loop, the early reward hacking behavior is substantially reduced (Figure [6(b)](https://arxiv.org/html/2510.14253v2#S4.F6.sf2)). Co-training the GRM with the evolving distribution of prompts and solutions stabilizes its scoring on harder inputs, dampens spurious entropy rewards, and allows ASL to make steady progress for a longer training horizon. In other words, as PG proposes more challenging problems and PM adapts, a concurrently updated GRM continues to calibrate its judgments, preventing the PG from exploiting scoring blind spots and preserving a meaningful learning signal for PM.

That said, even with GRM training we observe gradual convergence after several iterations, suggesting a capacity ceiling governed by the GRM’s verification ability. To test whether strengthening the GRM can lift this ceiling, we augment the GRM training in the third iteration with a small amount of real verification data (1% of the Search-R1 training data, constructed through the same method in §[4.3](https://arxiv.org/html/2510.14253v2#S4.SS3)). This targeted calibration further improves PM’s downstream solving performance and raises the test accuracy compared to training the GRM solely on self-generated data, as reflected by the right-hand curves in Figure [6(b)](https://arxiv.org/html/2510.14253v2#S4.F6.sf2).

These findings indicate that ASL’s upper bound is effectively determined by the GRM’s capability. Self-learning can progressively push the GRM’s boundary by exposing it to the current distribution of prompts and solutions, yielding longer and more productive training. Moreover, injecting a modest amount of real verification data in later stages provides a strong anchor for the GRM, refreshes the system’s ceiling, and unlocks additional gains. In practice, this suggests a two-phase strategy: rely on self-generated data to continually calibrate the GRM during most of training, then apply a small, carefully curated real-data boost to the GRM to extend ASL’s effectiveness in the final rounds.

## 5 Conclusion

Contributions.
In this work, we present Agentic Self-Learning (ASL), a novel multi-role, closed-loop reinforcement learning framework for training LLM-based agents without human supervision or pre-defined rule-based rewards. By unifying task generation (Prompt Generator), solution synthesis (Policy Model), and adaptive evaluation (Generative Reawrd Model) within a shared backbone and tool environment, ASL enables these components to co-evolve through iterative reinforcement signals.
Our controlled studies reveal two pivotal factors for scalable agentic RL: the source of reward signals and the scale of agent-generated task data. ASL addresses both by (i) replacing brittle rule-based judges with a co-trained GRM capable of nuanced verification, and (ii) continuously expanding the training distribution through adaptive task generation informed by entropy-based difficulty control.
Empirical results across diverse open-domain QA benchmarks demonstrate that ASL not only surpasses recent self-evolving agents (Search-R1, Absolute Zero, R-Zero) in long-term improvement but also avoids common reward-hacking pitfalls through GRM co-training. Moreover, we identify that GRM can be the bottleneck of the long-time evolving of ASL framework, and injecting small amounts of high-quality real verification data into late-stage GRM training further lifts the performance ceiling, suggesting a practical and scalable hybrid strategy.

Limitations and Outlook.
Looking forward, the ASL framework provides a general recipe for autonomous agent development, supporting open-ended reasoning, adaptive curricula, and robust reward modeling. However, our current studies are limited to the scenario of text-based search agent. Future work could extend ASL to multimodal agents, multi-turn planning domains, and real-world interactive platforms, paving the way for truly self-improving, domain-agnostic AI systems.

## Ethics Statement

This paper proposes the Agentic Self-Learning framework. All experiments are conducted on publicly available datasets. Thus there is no data privacy concern. Meanwhile, this paper does not involve human annotations, and there are no related ethical concerns.

## Reproducibility statement

The data and code of this paper are released at https://github.com/forangel2014/Towards-Agentic-Self-Learning

## References

- (1)
Nikolas Belle, Dakota Barnes, Alfonso Amayuelas, Ivan Bercovich, Xin Eric Wang, and William Wang.
Agents of change: Self-evolving LLM agents for strategic planning.
URL [http://arxiv.org/abs/2506.04651](http://arxiv.org/abs/2506.04651).
- (2)
Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Qihan Ren, Cheng Qian, Zhenghailong Wang, Minda Hu, Huazheng Wang, Qingyun Wu, Heng Ji, and Mengdi Wang.
A survey of self-evolving agents: On path to artificial super intelligence.
URL [http://arxiv.org/abs/2507.21046](http://arxiv.org/abs/2507.21046).
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Ho et al. (2020)
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa.
Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps.
*arXiv preprint arXiv:2011.01060*, 2020.
- (5)
Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu.
R-zero: Self-evolving reasoning LLM from zero data.
URL [http://arxiv.org/abs/2508.05004](http://arxiv.org/abs/2508.05004).
- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*, 2024.
- Jin et al. (2025)
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning.
*arXiv preprint arXiv:2503.09516*, 2025.
- Joshi et al. (2017)
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer.
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.
*arXiv preprint arXiv:1705.03551*, 2017.
- Karpukhin et al. (2020)
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.
Dense passage retrieval for open-domain question answering.
In *EMNLP (1)*, pp. 6769–6781, 2020.
- Kwiatkowski et al. (2019)
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al.
Natural questions: a benchmark for question answering research.
*Transactions of the Association for Computational Linguistics*, 7:453–466, 2019.
- Mallen et al. (2022)
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi.
When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories.
*arXiv preprint arXiv:2212.10511*, 7, 2022.
- (12)
Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog.
AlphaEvolve: A coding agent for scientific and algorithmic discovery.
URL [http://arxiv.org/abs/2506.13131](http://arxiv.org/abs/2506.13131).
- Press et al. (2022)
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis.
Measuring and narrowing the compositionality gap in language models.
*arXiv preprint arXiv:2210.03350*, 2022.
- Sheng et al. (2024)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
*arXiv preprint arXiv: 2409.19256*, 2024.
- Shi et al. (a)
Dingfeng Shi, Jingyi Cao, Qianben Chen, Weichen Sun, Weizhen Li, Hongxuan Lu, Fangchen Dong, Tianrui Qin, King Zhu, Minghao Liu, Jian Yang, Ge Zhang, Jiaheng Liu, Changwang Zhang, Jun Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou.
TaskCraft: Automated generation of agentic tasks, a.
URL [http://arxiv.org/abs/2506.10055](http://arxiv.org/abs/2506.10055).
- Shi et al. (b)
Tianyuan Shi, Canbin Huang, Fanqi Wan, Longguang Zhong, Ziyi Yang, Weizhou Shen, Xiaojun Quan, and Ming Yan.
Mutual-taught for Co-adapting policy and reward models, b.
URL [http://arxiv.org/abs/2506.06292](http://arxiv.org/abs/2506.06292).
- (17)
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
Reflexion: Language agents with verbal reinforcement learning.
- Song et al. (2025)
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen.
R1-searcher: Incentivizing the search capability in llms via reinforcement learning.
*arXiv preprint arXiv:2503.05592*, 2025.
- Sun et al. (2023)
Wangtao Sun, Xuanqing Yu, Shizhu He, Jun Zhao, and Kang Liu.
Expnote: Black-box large language models are better task solvers with experience notebook.
*arXiv preprint arXiv:2311.07032*, 2023.
- Trivedi et al. (2022)
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.
musique: Multihop questions via single-hop question composition.
*Transactions of the Association for Computational Linguistics*, 10:539–554, 2022.
- Wang et al. (2022)
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei.
Text embeddings by weakly-supervised contrastive pre-training.
*arXiv preprint arXiv:2212.03533*, 2022.
- (22)
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, and Lihong Li.
WebAgent-R1: Training web agents via end-to-end multi-turn reinforcement learning.
URL [http://arxiv.org/abs/2505.16421](http://arxiv.org/abs/2505.16421).
- Yang et al. (2018)
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning.
Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
*arXiv preprint arXiv:1809.09600*, 2018.
- Yao et al. (2022)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
*arXiv preprint arXiv:2210.03629*, 2022.
- (25)
Junjie Zhang, Jingyi Xi, Zhuoyang Song, Junyu Lu, Yuhua Ke, Ting Sun, Yukun Yang, Jiaxing Zhang, Songxin Zhang, and Zejian Xie.
L0: Reinforcement learning to become general agents.
- (26)
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang.
Absolute zero: Reinforced self-play reasoning with zero data.
URL [http://arxiv.org/abs/2505.03335](http://arxiv.org/abs/2505.03335).
- (27)
Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, and Sainbayar Sukhbaatar.
Self-challenging language model agents.
URL [http://arxiv.org/abs/2506.01716](http://arxiv.org/abs/2506.01716).

## Appendix A Prompts

Figure: Figure 7: System prompt for the Generative Reward Model and Policy Model.

Figure: Figure 8: System prompt for the Prompt Generator.

Figure: Figure 9: User prompt for Prompt Generator tt iteration 0.

Figure: Figure 10: User prompt for Prompt Generator at iteration 1.

Figure: Figure 11: User Prompt for Prompt Generator At Later Iterations

Figure: Figure 12: User prompt for Prompt Generator at later iterations.

## Appendix B Generation Examples of Three Roles in ASL

Figure: Figure 13: Example of Prompt Generator.

Figure: Figure 14: Example of Policy Model.

Figure: Figure 15: Example of Generative Reward Model.

## Appendix C The Use of Large Language Models

We used LLMs to assist with part of the translation and language polishing during the preparation of this paper.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 4
presentation: 4
questions: 1. How well does ASL generalize beyond search-based QA? Could the same framework be applied to domains like coding?
2. Since the GRM is identified as a bottleneck, have you explored ways to scale or diversify it?
3. How do you ensure that the synthetically generated tasks remain diverse, realistic, and representative of real-world challenges?
soundness: 2
strengths: - Human-free learning: the authors demonstrate scalable self-learning without human annotations or rule-based rewards, a significant step toward autonomous LLM training/self-improvement.
- Generalizability of the method and experiments: the work offers, in principle, a scalable, domain-agnostic recipe for self-improving agents, extensible to broader open-domain or multimodal tasks.
summary: This paper presents Agentic Self-Learning (ASL). This closed-loop reinforcement learning framework enables LLM agents to self-improve without human-labeled data or rule-based rewards. Through controlled experiments in a search-based agent environment, the authors identify two key determinants of scalable agentic training: the reward source and the scale of the data. ASL unifies task generation, policy execution, and reward evaluation within a shared LLM backbone, allowing the Prompt Generator, Policy Model, and GRM to co-evolve through iterative reinforcement. Empirical results show that ASL achieves sustained performance gains, surpassing strong baselines such as Search-R1, Absolute Zero, and R-Zero, even under zero-labeled-data conditions.
weaknesses: - Limited domain scope: the experiments are confined to a text-based search QA environment, leaving it unclear how ASL would generalize to other domains (e.g., coding or multimodal tasks).
- Complex training loop: the multi-role co-evolution process increases implementation complexity and computational cost.
- Non-significant experimental results: most experimental curves do not seem to show statistical significance.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. How does continual training mitigate reward hacking? What details can you provide toward the exact training mechanism?
2. If injecting verification data seems to help, is this not contrary to the hypothesis that GRM data is better?
3. Fig 2, why the sudden drop in reward for the rule-based method at the end?
4. Is there a tradeoff between learning efficiency and final performance?
5. How did you discover the reward hacking behavior in 4.4? How can we be assured there aren't other failure modes?
6. You mention the generative judge "can provide more informative and tolerant reward signals", are you able to elaborate? Which aspect of training induces these qualities
soundness: 2
strengths: The paper tackles a challenging problem and proposes a novel framework which has some promise of self-improvement. It identifies the generative reward model is the bottleneck for self improvement, as well as identifying/mitigating an instance of reward hacking. There are interesting design decisions such as sharing of policy and reward model parameters.
summary: This paper examines whether self-learning using LLM generated rewards can outperform rule based verifiable rewards.Their agentic self-learning approach outperforms strong baselines such as Search-R1. The primary concern of reward hacking is mitigated by continued training of the Generative Reward Model and late-stage verification data injection. This framework consists of multiple components: the prompt generator, rewarded by the entropy of the score distribution of the downstream policy; the generative reward model, this is trained to mimic the verifiable reward and a policy model, which provides candidate solutions.
weaknesses: The primary concern with RLAIF is the difficulty in avoiding reward hacking, whilst interesting I am yet to be convinced the proposed approach solves this problem.
 
Prompt Generator: Whilst rewarding based on the entropy of responses seems like an intuitive solution, it isn't clear how it would avoid reward hacking for solutions with many answers, i.e. always asking to roll a 1000 sided dice just has high entropy in responses/scores but is a degenerate prompting strategy.
 
Reward Model: As I understand it, this model is trained on the binary signal of if the reward model matches the verifiable reward output. Therefore, I am unsure how it can outperform the verifiable reward signal it is approximating when evaluated on the same task (training a model of the verification signal could perhaps generalize to domains where a signal is not present).
 
Policy: Asking the prompt generator for easier or harder prompts based on the score initially seems sensible however it could induce failure modes induced by the policy being punished for doing better, as if it answers correctly the questions become harder. Whilst perhaps not breaking it warrants discussion, particularly regarding the efficiency of learning.
 
Figure text is too small, with no information regarding seeds or uncertainty.
 
4.4 Ablation is useful to see that each component is learning however, as it's a closed loop system, it's hard to tell if the model has reward hacked without witnessing model outputs. For instance, I'm struggling to infer how you would notice the reward hacking behavior you mention from this plot. Therefore we're just taking the claim that at least one reward hack exists at face value without knowledge of how you discovered it and if there are any other degeneracies.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: 1. The presented method is novel and creative, focusing on a recent topic of artificial intelligence.

2. The reading flow is good. The method is clearly presented.
summary: The work introduces a new framework, Agentic Self-Learning, that aims to train an LLM with minimal human effort. This framework comprises three components: a prompt generator that creates tasks for a policy model to solve, and a generative reward model trained against a reference score to provide a reward signal to the prompt generator, which maximizes the entropy of the return, and to the policy model, which maximizes the return. Agentic Self-Learning is evaluated against competitive baselines, and multiple ablations are presented to understand its behavior.
weaknesses: I. Many details are missing to assess the presented method properly:

   a. By learning to increase the entropy of the return over a batch of predicted tasks, the prompt generator is incentivized to make simple tasks simpler and hard tasks harder. A comment that provides intuition for why this behavior leads to tasks well-adapted to the current policy model is missing.

   b. No details are provided to explain the way the difficulty feedback is implemented. Given that the prompt generator is trained to maximize entropy, listening to the feedback would increase the loss because it would guide the model to generate hard tasks simpler and simple tasks harder. Clarifying this aspect would be helpful.

   c. Line 280, the statement "This comparison produces a binary correctness indicator for each rollout" is not specific enough. Moreover, in Line 279, "we compare it against the reference score s", no details are given to describe the reference score.

   d. In Section 3.1, the environment in which the proposed method is evaluated is not described. Adding a short description would be beneficial.

   e. Line 312, "all roles share parameters", no information on which parameters are shared is disclosed.

   f. VeRL is used as a base algorithm; however, no justification for this choice is provided. Additionally, it would be preferable to discuss how the performances would change if another algorithm were selected.

   g. The lack of a preliminary Section increases the entry barriers of the presented approach, which lowers the potential audience for this work. Adding a preliminary section would strengthen the impact of the presented work.

   h. Adding a comment on the importance of the hyperparameters such as the length of the different phases would improve the usefulness of the presented approach. Additionally, commenting on the influence of the capability of the initial policy on the performance of the presented framework would be interesting.


II. Overall, the presentation can be improved:

   a. In Figure 5, the confidence intervals can be added.

   b. Parentheses are missing around many citations. For example, Line 139, "Shi et al." should have parentheses. Moreover, many citations do not have dates.

   c. The acronym PURM, in Line 98 is not defined.

   d. In Line 188, "(1)" should be written instead of the first "(3.1)", and "(2)" should be written instead of the second "(3.1)".

   e. The font size of the all figures should be much bigger.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. In the GRM training phase, the GRM performs N independent rollouts that generate a series of predictions for the score. These predicted scores are then compared against the reference scores - where do the reference scores come from? This seems like a bottleneck that the authors attempted to avoid by using the GRM but if GRM training relies on an outside verifier (i.e. human labels), then the positives of the GRM are lost.
2. How could this method be translated into more traditional RL settings (i.e. interactions with Atari or Mujoco environments)?
soundness: 2
strengths: This area of self-learning is of significant importance for producing autonomous agents that can improve their performance for new tasks. As far as this reviewer can tell, the work is original and distinct from the previous works in the literature.
summary: This paper proposes a self-learning paradigm for improving the capabilities of LLM based agents. The authors propose a method that leverages GRMs, which they find to outperform the rule based methods. The benefit of the GRM is that the GRM can be further improved through RL training as the prompt generator produces harder and harder prompts, the GRM can be further trained to produce rewards for this task.
weaknesses: 1. Results seem to be across a single seed of training (I couldn't find information about training setup i.e. number of seeds in the manuscript) which is very limited for research in RL. Even a small handful of seeds (i.e. 5) combined with the correct statistical methods (i.e. IQM[1]) can provide valuable insights. I strongly suggest that the authors work to include additional training runs of their method

2. Combined with #1, results are shown as point estimates and therefore we cannot evaluate the statistical significance of the results. Leveraging IQM [1] would provide insights into whether the experimental results are significantly different from each other or not. 

3. The figures are hard to read as the text and labels are much too small.

4. The authors re-introduce acronyms for their methods (i.e. Generative Reward Model (GRM) line 283 and line 068) throughout the text. Please introduce the acronym once when the method is first introduced, and then use the acronym going forward. 

5. I think that a discussion of scaling in RL is needed to give the reader full context of the literature. From the way the paper is written, a reader could think that scaling isn't done in RL outside of this proposed setting which is untrue. There are many works [2-4] that study parameter scaling in RL, while a few works [5, 6] provide similar insights into the relationship between scaling and the amount of data used during training, albeit in a more traditional RL problem setting. 

Citations
1. https://arxiv.org/abs/2108.13264
2. https://arxiv.org/pdf/2506.17204
3. https://arxiv.org/abs/2405.16158
4. https://openreview.net/forum?id=kfYxyvCYQ4
5. https://arxiv.org/pdf/2506.03404?
6. https://arxiv.org/abs/2503.05126


Decision: Reject