Title: SeRL: Self-play Reinforcement Learning for Large Language Models with Limited Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Self-Instruction;Self-Rewarding

Self-Play

Reinforcement Learning
Abstract: Recent advances have demonstrated the effectiveness of Reinforcement Learning (RL) in improving the reasoning capabilities of Large Language Models (LLMs). However, existing works inevitably rely on high-quality instructions and verifiable rewards for effective training, both of which are often difficult to obtain in specialized domains. In this paper, we propose Self-play Reinforcement Learning (SeRL) to bootstrap LLM training with limited initial data. Specifically, SeRL comprises two complementary modules: self-instruction and self-rewarding. The former module generates additional instructions based on the available data at each training step, employing comprehensive online filtering strategies to ensure instruction quality, diversity, and difficulty. The latter module introduces a simple yet effective majority-voting mechanism to estimate response rewards for additional instructions, eliminating the need for external annotations. Finally, SeRL performs conventional RL based on the generated data, facilitating iterative self-play learning.
Extensive experiments on various reasoning benchmarks and across different LLM backbones demonstrate that the proposed SeRL yields results superior to its counterparts and achieves performance on par with those obtained by high-quality data with verifiable rewards. Our code is available at https://github.com/wantbook-book/SeRL.

SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data

Abstract

Recent advances have demonstrated the effectiveness of Reinforcement Learning (RL) in improving the reasoning capabilities of Large Language Models (LLMs). However, existing works inevitably rely on high-quality instructions and verifiable rewards for effective training, both of which are often difficult to obtain in specialized domains. In this paper, we propose Se lf-play R einforcement L earning (SeRL) to bootstrap LLM training with limited initial data. Specifically, SeRL comprises two complementary modules: self-instruction and self-rewarding. The former module generates additional instructions based on the available data at each training step, employing robust online filtering strategies to ensure instruction quality, diversity, and difficulty. The latter module introduces a simple yet effective majority-voting mechanism to estimate response rewards for additional instructions, eliminating the need for external annotations. Finally, SeRL performs conventional RL based on the generated data, facilitating iterative self-play learning. Extensive experiments on various reasoning benchmarks and across different LLM backbones demonstrate that the proposed SeRL yields results superior to its counterparts and achieves performance on par with those obtained by high-quality data with verifiable rewards. Our code is available at https://github.com/wantbook-book/SeRL .

1 Introduction

Recent breakthroughs of Large Language Models (LLMs) have demonstrated their potential to revolutionize a wide range of fields [40, 90, 91], such as healthcare [4, 31, 51], robotics [25, 37, 61], and web services [16, 41, 67, 72]. Notably, frontier models such as OpenAI-o3 [49] and DeepSeekR1 [6] have proven that Reinforcement Learning with Verifiable Rewards (RLVR) can endow LLMs with advanced reasoning capabilities [16, 39, 67, 71, 85, 86], enabling significant achievements in mathematics [6, 76] and programming [21, 26]. A key factor in these developments is the availability of large-scale supervised datasets [7, 14, 78, 80], which incorporate instructions and corresponding ground-truth labels for reward computation.

However, in specialized domains such as clinical diagnostics [44, 58, 60] and aerospace engineering [5, 36, 75], collecting such high-quality data and verifiable supervision remains a significant challenge. This difficulty stems from the fact that acquiring human-labeled data in these areas is particularly labor-

intensive, often requiring substantial expert knowledge and considerable time investment [55, 59]. Moreover, such human-generated annotations are inherently difficult to scale. As an alternative, data generated by LLMs offers a more scalable and cost-effective solution [12, 13, 30, 79, 82]. Nevertheless, this approach usually depends on access to highly capable expert-level LLMs, which may not always be readily available.

To remedy this issue, recent efforts have explored the self-instruction paradigm [11, 27, 63], where LLMs autonomously generate instructions and corresponding responses. While effective for supervised fine-tuning [10, 17, 88], these methods are not directly applicable to online RL settings, which pose two critical challenges. First, existing methods typically generate a fixed dataset without accounting for the evolving capabilities of the LLM during online learning, rendering the static data progressively less suitable. Second, directly treating model-generated responses as ground-truth labels often introduces noise or inconsistency, leading to unreliable reward signals for RL.

In this work, we propose Se lf-play R einforcement L earning , termed as SeRL, where LLMs bootstrap training via autonomous instruction generation and reward estimation under limited initial data. Technically, SeRL consists of two key modules: (1) The self-instruction module iteratively performs few-shot generation to obtain new instructions based on the initial data, producing instructions at each training step that align with the current capability of the model. We introduce a robust online filter that removes low-quality or redundant instructions and enforces difficulty suited to the current ability of the LLM. (2) The self-rewarding module employs majority voting over sampled responses, assigning high rewards to those aligning with the consensus. This enables effective reward estimation without relying on verifiable labels. Based on these two modules, we perform unsupervised RL training on the generated data. Our key contributions are summarized as follows:

We explore how to leverage RL to incentivize the reasoning abilities of LLMs in data-scarce scenarios, a highly practical yet underexplored challenge for the LLM research community.

We propose the SeRL framework, comprising two core modules: self-instruction, which leverages few-shot generation to obtain high-quality instructions from limited data, and self-rewarding, which estimates rewards through majority voting without relying on external supervision.

Extensive experiments on diverse benchmarks demonstrate that the proposed SeRL, even with limited initial data, outperforms other advanced self-play counterparts and achieves results comparable to the baseline trained on full high-quality data with verifiable rewards.

2 Background

Reinforcement Learning for LLMs. We denote the LLM parameterized by θ as π θ . For language generation, given an instruction x , the model outputs a sequence y = ( y 1 , . . . , y T ) , where y t ∼ π θ ( x , y <t ) and y <t = ( y 0 , . . . , y t -1 ) . RL [22, 38, 56] has recently proven effective in improving the logical reasoning abilities of LLMs. In the RL setting, we additionally require a reward signal. We denote the reward for a given input-output pair ( x , y ) as R ( x , y ) . In this work, while our framework is not tied to any specific RL algorithm, we choose Reinforce++ [20] for its robustness and stable performance across diverse tasks. The RL training objective is defined as follows:

where D is the dataset of instructions, s t = ( x , y <t ) , ϵ is a clipping hyperparameter introduced in PPO [53] to stabilize training, and π θ old denotes the model parameters before the most recent update. Different RL algorithms compute the advantage term ˆ A ( s t , y t ) in different ways. The details of the Reinforce++ algorithm are provided in Appendix C.1.

Self-Instruction. We denote the initial dataset as D seed = { x i } N i =1 , where N is small in data-scarce domains and insufficient for effective RL training. To address this, we generate a new batch of instructions with a rollout batch size n rbs at each training step t . The generated batch at step t is denoted as B t gen = { x i } n rbs i =1 . This step-wise generation better matches the evolving capabilities of

Model at step t

Model at step t+1

1) Self-Instruction

2) Self-Rewarding

Next Training Step

Train batch

Self-Play Reinforcement Learning Framework

r

r

r

GenData

t

What multiple of 15

is closest to 2009?

2010

step1: .., step2: ...

1995

step1: .., step2: ...

2010

step1: .., step2: ...

2000

2010

Answer

Count

2

1995

1

2000

1

Majority Voting

Reward

r=1

r=0

r=1

r=0

2) Self-Rewarding

step1: .., step2: ...

Answer

Chain-of-Thought

Reward: r

i

= 1 if Answer

i

== Answer

maj

else 0

Answer

maj

is 2010

r

Train batch

r

r

1) Self-Instruction

Few-Shot

{More Generated

Math Questions}

Question

Generation

Generate more high-quality math questions

based on the following problems:

2. Simplify $(x+15)+(100x+15)$.

SeedData

Too difficult

Too easy

r

mean

> 0.8

r

mean

< 0.2

Keywords

Length

Similarity

Online

Filter

GenData

Difficulty

X

+

A

the model. We define the full generated dataset for iteration n as D n gen = ⋃ t ∈T n B t gen , where T n represents the set of all generation steps within iteration n .

3 Self-play Reinforcement Learning

In this work, we introduce SeRL to tackle tasks in data-scarce scenarios. SeRL addresses the challenge by expanding the limited initial dataset and enabling unsupervised reinforcement learning without relying on verifiable labels. SeRL comprises two core components: (1) Self-Instruction. Given a small set of initial data, the model generates additional instructions using few-shot generation. To ensure high data quality, sufficient diversity, and appropriate difficulty, we employ a robust online filtering strategy. (2) Self-Rewarding. We design a self-rewarding mechanism based on majority voting, which allows the model to estimate rewards accurately without the need for verifiable labels, thus supporting fully unsupervised RL training.

3.1 Self-Instruction

In data-scarce scenarios, we aim to generate additional instructions from the limited available data to support RL training. To this end, we design the self-instruction mechanism, as illustrated in the blue region of Fig. 1, which consists of the following two steps.

Instruction Generation. The LLM is prompted in a few-shot manner to generate new instructions. Each few-shot prompt consists of randomly selected examples from the initial dataset and examples

from the already generated dataset. The reason why we use instructions from the initial dataset is that we aim to expand the dataset while maintaining the same distribution as the initial one, enabling the model to effectively learn the distribution of scarce data. The reason we do not rely solely on the initial dataset as few-shot examples is that, as training progresses, we aim to adapt the examples to the evolving capabilities of the model. Specifically, we also selectively incorporate recently generated instructions as few-shot examples for subsequent generation, enabling the model to produce instructions that better match its current level of competence. The prompt template of instruction generation is provided in Tab. C.4.

Online Instruction Filter. To ensure the quality, diversity, and moderate difficulty of the generated instructions, we implement a robust online filtering strategy. Specifically, we discard instructions that meet any of the following criteria: (1) a ROUGE-L [35] score exceeding a predefined threshold with existing instructions, to prevent generating semantically similar instructions that reduce diversity; (2) the presence of specific keywords such as 'image', 'graph', or 'picture', which refer to visual content that LLMs cannot process; (3) excessively long or short, since long instructions often contain repetitive phrasing while short ones tend to lack necessary context, both leading to invalid instructions; (4) having a majority answer ratio outside a specified range, ensuring that the retained instructions maintains suitable difficulty for training. Formally, we retain only instructions whose average reward r mean falls within two tunable bounds, γ diff ≤ r mean ≤ γ easy . r mean is calculated as the majority answer ratio from multiple responses of the instruction in our majority-voting reward method.

The motivation behind this dual-end clipped difficulty filtering strategy is to avoid scenarios where the responses are either entirely correct without any information gain or entirely incorrect with no feasibility for improvement. Such extreme cases result in zero advantage in algorithms like Reinforce++ [20] and GRPO [54], which can lead to gradient vanishing [65, 42] and consequently significant model degradation. In the ablation studies presented in Section 4.3, we demonstrate that removing the difficulty filtering leads to reward hacking.

3.2 Self-Rewarding

In our self-instruction setup, only instructions are generated, and the absence of verifiable labels presents a key challenge for reward computation. Interestingly, Yue et al. [83] suggests that RL with verifiable labels can be viewed as converting the Pass@K performance of a model into Pass@1. Inspired by this insight, we explore whether a similar benefit can be obtained in the absence of verifiable labels. To achieve this, as illustrated in the green region of Fig. 1, we introduce a majorityvoting self-rewarding that serves two purposes: (1) it improves inference efficiency by enabling the model to reach Maj@N performance with just a single response (Pass@1) after RL training, (2) and it offers a straightforward yet effective way to estimate rewards without the need for verifiable labels. For a given instruction x i , we sample n vote responses, each consisting of a chain-of-thought reasoning c k i and a final answer a k i :

Since we adopt a majority-voting reward estimation method, we define the majority answer for a given instruction x i as Maj ( { a k i } n vote k =1 ) , where { a k i } n vote k =1 denotes the set of n vote answers sampled for x i . For each a k i , we compute the corresponding reward R k i = Verify ( a k i , Maj ( { a k i } n vote k =1 )) . The Verify function is defined as:

As instances with all-correct or all-incorrect responses have already been eliminated by the online filter, a majority answer can always be identified. If there is a tie among multiple majority answers, the one with the shortest length is selected based on the principle that a more concise response is preferred to reduce verbosity. We refer to this self-rewarding strategy as majority-voting reward. It serves as a replacement for traditional reward computation methods that rely on reward models or verifiable labels, providing a practical and efficient approach to reward estimation.

3.3 Overall Framework

To enable continual improvement in data-scarce scenarios, we adopt a combination of self-instruction and self-rewarding. As shown in the yellow region of Fig. 1, the model iteratively generates new

instructions from limited data and estimates rewards for sampled responses using majority voting. This process enables sustained reinforcement learning across training iterations. To match the evolving capability of the model, we generate a batch of instructions at each training step t using self-instruction. For each instruction in B t gen , we sample n vote responses and compute the majorityvoting reward. The resulting triplets { ( x i , y i , r i ) } , consisting of the instruction, sampled response, and computed reward, are used to update the model parameters in one training step. At the next training step t +1 , we repeat the same process to generate B t +1 gen = { x i } n rbs , followed by one step of RL training. This cycle repeats across iterations, gradually refining the performance of the model.

4 Experiments

4.1 Experimental Setup

Models and Training Settings. We conducted experiments using two model series: LLaMA-3.2-3BInstruct [46] and Qwen-2.5-7B-Instruct [78]. We simulate a data-scarce scenario in the mathematics domain by limiting the initial dataset to 500 instructions, which are uniformly sampled across all difficulty levels from the MATH training set [18]. We denote this initial set of 500 instructions as D seed. For training, we adopt the OpenRLHF [19] framework and employ its Reinforce++ [20] algorithm for RL. Detailed training hyperparameters are provided in Tab. 6 and Tab. 7.

Baselines. We compare our method against two multi-round iterative training baselines and one RL baseline with ground-truth rewards (RL-GT): (1) Iterative RPO (I-RPO) [50], which runs for three iterations. In each iteration, four responses are sampled for each instruction in the MATH training set. Each response is scored using a rule-based reward, and the highest and lowest are selected as the chosen and rejected responses for DPO training. Rule-based methods [54, 50] determine correctness

by extracting the answer from the response using regular expressions and comparing it against the ground-truth answer to assign a reward. (2) Self-Rewarding DPO (SR-DPO) [81], also run for three iterations. In each iteration, a new dataset D gen = { x i } N i =1 is synthesized based on D seed using the same self-instruction approach as ours. For each generated instruction, four responses are sampled, and model-based rewards are used to construct preference pairs for DPO training. Model-based methods [81] rely on manually designed principles or scoring rubrics, which the model uses to evaluate the response and assign a reward. (3) RL-GT is run for a single iteration, using the MATH training set for RL training. The rewards are computed using a rule-based reward function.

Since the original reward prompts in SR-DPO are intended for general tasks, we modify them to better fit mathematical scenarios, ensuring a fair comparison (see Tab. C.5). To ensure fairness in training data scale, we note that RL-GT and I-RPO use 7,500 instructions from the MATH training set, and SR-DPO generates 7,500 instructions per iteration. For our method, we online-generate instructions during training, keeping the same number of training gradient steps as other baselines to ensure the total training data scale remains consistent, thereby guaranteeing a fair comparison.

Evaluation Benchmarks. We evaluate model performance using five math-specific benchmarks and one general-purpose benchmark. (1) The math benchmarks include MATH-500 [34], which focuses on medium-difficulty problems; MATH-Hard [18], which contains level-5 competition problems from AMCand AIME to assess advanced reasoning; ASDiv [47], a dataset of 2,305 diverse elementarylevel math word problems; College Math [57], which covers university-level topics such as algebra and calculus; and TabMWP [43], consisting of 38,431 problems that combine textual and tabular data to test multi-step reasoning. (2) To assess general reasoning ability, we include MMLU-Pro [64], an enhanced version of MMLU covering STEM, humanities, and social sciences. All evaluations are conducted using greedy decoding with a maximum of 1,024 new tokens.

4.2 Experimental Results

Our approach matches the performance of the method using extensive data with verifiable rewards. As shown in Tab. 1, after just the first round of unsupervised RL training on generated data, our model achieves performance comparable to RL-GT. After the second round, LLaMA-3.2-3BInstruct even outperforms RL-GT across nearly all evaluation benchmarks. LLaMA-3.2-3B-Instruct continues to improve in the third round. After three rounds of iterative training, Qwen-2.5-7B-Instruct also outperforms RL-GT on the MATH-500, ASDiv, and TabMWP benchmarks. Additional results of our method for more iterations are provided in Appendix D.2.

Our method outperforms other multi-round iterative approaches. Across all iterations, our method consistently outperforms both SR-DPO and I-RPO on LLaMA-3.2-3B-Instruct. We observe that both SR-DPO and I-RPO exhibit noticeable model degradation over multiple iterations. For SR-DPO, it performs poorly on LLaMA-3.2-3B-Instruct, while showing slightly better results on Qwen-2.5-7B-Instruct. This suggests that model-based reward mechanisms may depend on model scale, as larger models are more likely to produce accurate reward estimates. To better understand the limited effectiveness of SR-DPO, we further analyze its reward estimation accuracy in Section 4.3 (c). In contrast, I-RPO shows a decline in performance over multiple training rounds on both models. We hypothesize that this degradation results from the inclusion of a negative log-likelihood (NLL) loss term in its training objective, which may cause the model to overfit the chosen responses. As a consequence, this overoptimization reduces generalization and ultimately leads to performance drops.

Our method generalizes well to general reasoning tasks. To better evaluate generalization, we assess our method on the MMLU-Pro benchmark. As shown in Tab. 2, LLaMA-3.2-3B-Instruct achieves consistent gains over multiple iterations in certain categories, whereas Qwen-2.5-7B-Instruct

shows minimal improvement. We attribute this to Qwen-2.5 already being extensively fine-tuned on MATH data, as further rule-based RL training on the MATH training set also fails to improve its MMLU-Pro performance. LLaMA-3.2-3B-Instruct shows consistent improvement in the STEM category, with additional gains in Humanities, eventually surpassing RL-GT. Performance in the Social and Other categories remain largely unchanged. This is likely because our training data is math-focused, enhancing reasoning skills that benefit STEM tasks. The improvement in Humanities is mainly due to the Law subcategory, where many questions involve logical reasoning. As the reasoning ability of the model improves, its performance on Law tasks also increases.

4.3 Ablation Study

(a) The dual-end clipped difficulty filtering mechanism effectively prevents reward hacking. As shown in Fig. 2, when we remove the difficulty filtering, the model exhibits clear signs of reward hacking: the average reward continues to increase with training steps, but the accuracy on the MATH-500 test set drops sharply after around 100 steps.

To investigate this issue, we perform a case study (Tab. E.2) and observe that although the intermediate reasoning is often correct, the model consistently ends with the final statement: 'The final answer is: 0 .' This behavior arises because our reward function is based on majority voting. If all n vote sampled responses from the LLM produce the same (albeit incorrect) answer, such as 0 , each response receives a high reward, despite being a wrong answer. The root cause of this phenomenon lies in the difficulty of the instruction. When an instruction is too hard, the model struggles to produce consistent answers across samples, leading to unstable or misleading reward signals. To address this, we introduce a dual-end dif-

0

100

200

300

Gradient Steps

0.0

0.2

0.4

0.6

0.8

1.0

Reward

Reward w/o filter

Reward w/ filter

Acc w/o filter

Acc w/ filter

0.0

0.1

0.2

0.3

0.4

0.5

ficulty filtering mechanism to filter out instructions with insufficient answer agreement, i.e., those with r mean < γ diff. Similarly, overly easy instructions do not contribute meaningfully to learning and are also filtered by applying the upper threshold r mean > γ easy. With this dual-end clipped difficulty filtering mechanism, the reward hacking issue is mitigated in subsequent experiments.

(b) All of our proposed data filtering strategies demonstrate their effectiveness. To demonstrate the effectiveness of each filtering strategy in the online filtering process, we conducted an ablation study as shown in Tab. 3. The results show that every filtering strategy contributes to the model's performance, as removing any of them leads to a decline in overall results. In particular, omitting the difficulty filter may cause reward hacking, as illustrated in Fig. 2.

(c) Our method effectively mitigates the bias inherently introduced by self-instruction and self-rewarding. Our proposed method is specifically designed to minimize this bias as much as possible, and matches the performance of methods using extensive data with verifiable rewards, despite the limited amount of seed data used in our method.

For the self-instruction module, we introduce an online filtering strategy that maintains data quality while promoting diversity in the generated samples. We have provided a comprehensive analysis

of the generated data in terms of quality, difficulty, and diversity, as detailed in Appendix F. These results demonstrate the reliability of our self-instruction method for data generation.

For the self-rewarding module, as stated in the related works [81, 87, 92], estimation bias is inevitable in self-rewarding settings. Our majority-voting reward method assigns rewards based on the consistency among multiple sampled responses, offering greater stability than scoring individual responses. To demonstrate this, we calculate the cosine similarity between the estimated rewards and the ground-truth rewards, with higher similarity indicating greater accuracy in reward estimation. As shown in Fig. 3, on both LLaMA-3.2-3B-Instruct and Qwen-2.5-7B-Instruct, the majority-voting reward demonstrates significantly higher alignment with the rule-based reward, indicating superior accuracy in reward estimation. In contrast, the low accuracy of the model-based reward on LLaMA3.2-3B-Instruct likely accounts for the underperformance of SR-DPO on this model. Additionally, the model-based reward shows higher consistency with the rule-based reward on Qwen-2.5-7B-Instruct, which accounts for the relatively better performance of SR-DPO on Qwen-2.5-7B-Instruct. More comparisons of reward methods and alignment metric results can be found in Appendix E.1.

0

100

200

300

400

500

Question Index

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Cosine Similarity

sim(GT, Model-based)

sim(GT, Majority-voting) (Ours)

Avg(sim(GT, Model-based)) = 0.17

Avg(sim(GT, Majority-voting)) = 0.75

0

100

200

300

400

500

Question Index

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

Cosine Similarity

sim(GT, Model-based)

sim(GT, Majority-voting) (Ours)

Avg(sim(GT, Model-based)) = 0.49

Avg(sim(GT, Majority-voting)) = 0.85

(a) The result of LLaMA-3.2-3B-Instruct.

(b) The result of Qwen-2.5-7B-Instruct.

Figure 3: Cosine similarity between rule-based reward and majority-voting / model-based reward on MATH-500. Each point represents the similarity over 16 sampled responses per instruction. "sim(GT, Model-based)" refers to the cosine similarity between the rule-based reward and the model-based reward, while "sim(GT, Majority-voting)" refers to the cosine similarity between the rule-based reward and our majority-voting reward. The dashed lines in the figure indicate the average values.

To further assess the effectiveness of our majority-voting reward, we conduct an ablation study shown in Tab. 4. In this setting, we remove the self-instruction mechanism and perform one round of majority-voting reward RL using the full MATH training set, denoted as RL-MV. Results show that this variant achieves performance comparable to RL-GT across all test datasets. Notably, LLaMA3.2-3B-Instruct outperforms RL-GT on MATH-500, MATH-Hard, College Math, and TabMWP, while Qwen-2.5-7B-Instruct consistently surpasses RL-GT on nearly all benchmarks.

(d) Our method is applicable to different RL algorithms. To verify the robustness of our method across different RL algorithms, we evaluate it under various RL setups. RLOO [1] and Reinforce++ [20] use k 1 KL divergence estimation, while GRPO [54] adopts k 3 KL divergence estimation. The formulations for both k 1 and k 3 KL divergence estimations are detailed in Appendix B. Since k 1 is integrated into the per-token reward term, we set a smaller KL coefficient for training RLOO and

Reinforce++ to avoid strong KL regularization. All other hyperparameter settings remain the same. As shown in Tab. 5, all algorithms achieve comparable performance, demonstrating the generality of our approach. Among them, the Reinforce++ algorithm delivers the best overall results. The GRPO algorithm achieves relatively strong performance in the first iteration but shows continued improvement only on MATH-Hard, ASDiv, and TabMWP in subsequent rounds, while its performance declines on MATH-500 and College Math. Similarly, RLOO exhibits performance degradation on MATH-Hard and College Math after the third iteration.

Reinforce++ demonstrates greater robustness compared to GRPO and RLOO due to the following reasons: (1) GRPO normalizes the estimated advantage over the n vote responses for each question, whereas Reinforce++ estimates the advantage over all responses across all questions, resulting in a larger group and more stable estimation. In addition, GRPO uses an external k 3 KL estimation, which involves an exponential term. This may cause large spikes in the gradient, leading to instability during training. (2) RLOO applies a preprocessing step that subtracts the average reward of other responses from the reward of each individual response, thereby increasing the reward gap between different responses. However, since self-rewarding may introduce inaccurate reward estimations, the amplified reward differences in RLOO could lead to greater bias when the estimation is incorrect. As a result, its performance tends to be worse than that of Reinforce++.

5 Related Work

Self-Instruction Methods. Recent studies show that RL can significantly enhance the reasoning abilities of LLMs. However, it depends on high-quality instructions and verifiable labels, which are scarce in specialized domains such as clinical diagnostics [58, 44, 60] and aerospace engineering [36, 5, 75]. This challenge has spurred interest in synthetic data generation. Previous work has employed powerful expert models like GPT-4 to generate synthetic data [79, 13, 32, 30, 82], but these methods are costly and rely on external proprietary systems. To mitigate this dependency, self-instruction frameworks such as Wang et al. [63] bootstrap data from a small set of seed examples via fewshot generation and heuristic filtering to ensure quality and diversity. Many other studies [88, 10, 27, 11, 17] have extended the self-instruction framework, focusing on improving data efficiency, diversity, and quality. Some studies train dedicated models for instruction generation. For instance, WizardLM [74] evolves instruction data in terms of complexity, diversity, and quality, while TeaMsRL [15] extends it by defining prompt-based augmentation actions and training an Instructor LLM via RL to apply them. In contrast, we find that existing models already possess strong instructiongeneration abilities. Without training a separate model or designing additional augmentation prompts, our method leverages few-shot generation and an online filtering strategy to produce diverse, highquality instructions (see Appendix F). Unlike these offline SFT approaches with fixed data, RL enables online data generation that adapts to the model's evolving capability. However, it still requires verifiable labels and mechanisms to prevent reward hacking.

Self-Rewarding Methods. Recent successes of RL in LLMs largely rely on large-scale instructions with verifiable rewards [73, 23, 66]. However, labeled data typically requires substantial human effort and is limited in both efficiency and quality. As a more efficient alternative, self-rewarding

methods that estimate rewards without relying on verifiable labels have gained increasing attention. Model-based self-rewarding methods like Bai et al. [2] and Yuan et al. [81] use predefined rules to generate preference pairs for Direct Preference Optimization (DPO) training. Fränken et al. [9] maximize mutual information between principles and responses, while Zhang et al. [87] is based on semantic entropy clustering. RLSC [33] optimizes the model by directly maximizing its most probable response, while our majority-voting method is based on multiple responses, leading to greater stability and more consistent outputs. TTRL [92], as a concurrent work, also employs a majority-voting self-rewarding mechanism in an unsupervised reinforcement learning setting. However, our approach differs in two important aspects: (1) we generate diverse synthetic training instructions from limited data through self-instruction, unlike test-time training of TTRL on fixed test sets; (2) our method supports multiple rounds of iterative updates, enabling continual self-improvement and refinement of the policy, while TTRL only performs a single-round adaptation on a static test set.

6 Discussion

Ceiling of Self-Iteration. We begin by discussing the performance upper bound of LLMs within a single iteration. When performing RL training using only majority-voting reward, the maximum achievable Pass@1 on the training set is inherently limited by the Maj@K before training. If the LLM has strong generalization capabilities, the Pass@1 upper bound on an in-domain test set should also align with the Maj@N of the model on the test set before training. So, what is the upper bound of LLM capability across multiple iterations? If the Maj@N of the model continues to improve after each round of training, then each iteration defines a new, higher ceiling. In this case, the LLM is capable of approaching increasingly better performance bounds, suggesting the potential for unbounded, continual improvement through iterative self-training. However, as shown in Fig. 7, the Maj@16 performance of both LLaMA-3.2-3B-Instruct and Qwen-2.5-7B-Instruct on MATH-500 does not continue to improve with more iterations. This observation aligns with the analysis by Yue et al. [83], which suggests that RL with verifiable reward primarily transforms Pass@K ability into Pass@1, while the upper bound of a model is constrained by its underlying capability. Nevertheless, our method remains valuable, as it enables efficient reasoning in data-scarce scenarios through unsupervised RL training with data augmentation.

Limitation of Majority-voting Reward. In scenarios where there is no deterministic final answer such as writing tasks, or where the correctness of the process is more important such as mathematical proofs, our majority voting strategy may not be applicable. However, for most knowledge-based scenarios, even when a numeric answer is not required as in mathematical problem solving, the question can often be reformulated into a multiple-choice format, making our majority voting approach still applicable. Another limitation is that the reliability of the reward signal in self-rewarding methods can be compromised for challenging problems, as also discussed in prior work [81, 87, 92]. To mitigate this issue, we have designed a difficulty filtering strategy that removes instructions for which the model shows uncertainty in generating consistent answers. This acts as a form of curriculum learning, where the model starts training on simpler problems and gradually moves toward more complex ones. As a result, the estimation bias induced by unreliable self-rewards is reduced during training. While this strategy does not completely resolve the issue, we believe it is a promising direction and worth further exploration in future work.

7 Conclusion

In this work, we introduce SeRL, a framework designed to bootstrap LLM training from limited initial data. SeRL consists of two key components: a self-instruction module and a self-rewarding module. The self-instruction module expands the initial data by generating new instructions, with robust and effective online filtering applied to ensure quality, diversity, and appropriate difficulty. The self-rewarding module uses a majority-voting strategy to estimate rewards for generated responses, removing the need for external labels. Based on the generated data, SeRL performs standard RL training in an iterative, self-improving manner. Extensive experiments across multiple reasoning benchmarks and LLM architectures show that SeRL consistently outperforms strong baselines and matches the performance of methods trained with large-scale, high-quality labeled data.

References

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740 , 2024.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335 , 2024.

Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, et al. The future landscape of large language models in medicine. Communications medicine , 2023.

Brian J Connolly. Development of an aerospace engineering evaluation set for large language model benchmarking. In AIAA SCITECH 2025 Forum , 2025.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 , 2025.

Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863 , 2024.

Jan-Philipp Fränken, Eric Zelikman, Rafael Rafailov, Kanishk Gandhi, Tobias Gerstenberg, and Noah D. Goodman. Self-supervised alignment with mutual information: Learning to follow principles without preference labels. In NeurIPS , 2024.

Manon Galloy, Martin Balfroid, Benoit Vanderose, and Xavier Devroey. Selfbehave, generating a synthetic behaviour-driven development dataset using self-instruct. In 2025 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) , 2025.

Shen Gao, Zhengliang Shi, Minghang Zhu, Bowen Fang, Xin Xin, Pengjie Ren, Zhumin Chen, Jun Ma, and Zhaochun Ren. Confucius: iterative tool learning from introspection feedback by easy-to-difficult curriculum. In AAAI , 2024.

Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094 , 2025.

Anna Goldie, Azalia Mirhoseini, Hao Zhou, Irene Cai, and Christopher D. Manning. Synthetic data generation & multi-step rl for reasoning & tool use. arXiv preprint arXiv:2504.04736 , 2025.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Shangding Gu, Alois Knoll, and Ming Jin. Teams-rl: Teaching llms to generate better instruction datasets via reinforcement learning. arXiv preprint arXiv:2403.08694 , 2024.

Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al. Is your llm secretly a world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411.06559 , 2024.

Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xiaoying Zhang, Zhaoran Wang, and Yang Liu. Human-instruction-free llm self-alignment with limited samples. arXiv preprint arXiv:2401.06785 , 2024.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurIPS , 2021.

Jian Hu, Xibin Wu, Zilin Zhu, Weixun Wang, Dehao Zhang, Yu Cao, et al. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143 , 2024.

Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.03262 , 2025.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, et al. Qwen2.5-coder technical report. arXiv preprint arXiv:2409.12186 , 2024.

Haobo Jiang, Guangyu Li, Jin Xie, and Jian Yang. Action candidate driven clipped double q-learning for discrete and continuous action tasks. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F. Chen, Shafiq Joty, and Furu Wei. Preference optimization for reasoning with pseudo feedback. arXiv preprint arXiv:2411.16345 , 2025.

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 , 2019.

Shyam Sundar Kannan, Vishnunandan LN Venkatesh, and Byung-Cheol Min. Smart-llm: Smart multiagent robot task planning using large language models. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2024.

Ruhma Khan, Sumit Gulwani, Vu Le, Arjun Radhakrishna, Ashish Tiwari, and Gust Verbruggen. Llmguided compositional program synthesis. arXiv preprint arXiv:2503.15540 , 2025.

Jungwoo Kim, Minsang Kim, and Sungjin Lee. Sedi-instruct: Enhancing alignment of language models through self-directed instruction generation. arXiv preprint arXiv:2502.04774 , 2025.

Nikita Kitaev and Dan Klein. Constituency parsing with a self-attentive encoder. arXiv preprint arXiv:1805.01052 , 2018.

Nikita Kitaev, Steven Cao, and Dan Klein. Multilingual constituency parsing with self-attention and pre-training. In ACL , 2019.

Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402.13064 , 2024.

Junkai Li, Yunghwei Lai, Weitao Li, Jingyi Ren, Meng Zhang, Xinhui Kang, Siyu Wang, Peng Li, Ya-Qin Zhang, Weizhi Ma, and Yang Liu. Agent hospital: A simulacrum of hospital with evolvable medical agents. arXiv preprint arXiv:2405.02957 , 2025.

Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning. In Findings of the Association for Computational Linguistics ACL 2024 , 2024.

Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. Confidence is all you need: Few-shot rl fine-tuning of language models. arXiv preprint arXiv:2506.06395 , 2025.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 , 2023.

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , 2004.

Beiming Liu, Zhizhuo Cui, Siteng Hu, Xiaohua Li, Haifeng Lin, and Zhengxin Zhang. Llm evaluation based on aerospace manufacturing expertise: Automated generation and multi-model question answering. arXiv preprint arXiv:2501.17183 , 2025.

Haokun Liu, Yaonan Zhu, Kenji Kato, Atsushi Tsukahara, Izumi Kondo, Tadayoshi Aoyama, and Yasuhisa Hasegawa. Enhancing the llm-based robot manipulation through human-robot collaboration. IEEE Robotics and Automation Letters , 2024.

Shunyu Liu, Jie Song, Yihe Zhou, Na Yu, Kaixuan Chen, Zunlei Feng, and Mingli Song. Interaction pattern disentangling for multi-agent reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Shunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song, and Dacheng Tao. A survey of direct preference optimization. arXiv preprint arXiv:2503.11701 , 2025.

Shunyu Liu, Yaoru Li, Kongcheng Zhang, Zhenyu Cui, Wenkai Fang, Yuxuan Zheng, Tongya Zheng, and Mingli Song. Odyssey: Empowering minecraft agents with open-world skills. In International Joint Conference on Artificial Intelligence , 2025.

Shunyu Liu, Minghao Liu, Huichi Zhou, Zhenyu Cui, Yang Zhou, Yuhao Zhou, Wendong Fan, Ge Zhang, Jiajun Shi, Weihao Xuan, et al. Verigui: Verifiable long-chain gui dataset. arXiv preprint arXiv:2508.04026 , 2025.

Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu. Ghpo: Adaptive guidance for stable and efficient llm reinforcement learning. arXiv preprint arXiv:2507.10628 , 2025.

Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610 , 2023.

Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, et al. Towards accurate differential diagnosis with large language models. Nature , 2025.

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2020.

Meta-AI. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, 2025. URL https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ .

Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In ACL , 2020.

Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, et al. Can generalist foundation models outcompete specialpurpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452 , 2023.

OpenAI. Introducing openai o3 and o4-mini, 2025. URL https://openai.com/index/ introducing-o3-and-o4-mini/ .

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. In NeurIPS , 2024.

Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. Llm-based agentic systems in medicine and healthcare. Nature Machine Intelligence , 2024.

Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715 , 2024.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024.

David Silver and Richard S Sutton. Welcome to the era of experience. Google AI , 2025.

Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction . MIT press Cambridge, 1998.

Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403.02884 , 2024.

Ehsan Ullah, Anil Parwani, Mirza Mansoor Baig, and Rajendra Singh. Challenges and barriers of using large language models (llm) such as chatgpt for diagnostic medicine with a focus on digital pathology-a recent scoping review. Diagnostic pathology , 2024.

Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of llm scaling based on human-generated data. arXiv preprint arXiv:2211.04325 , 2024.

Guangyu Wang and Xiaohong Liu. Medical large language model for diagnostic reasoning across specialties. Nature Medicine , 2025.

Jiaqi Wang, Enze Shi, Huawen Hu, Chong Ma, Yiheng Liu, Xuhui Wang, Yincheng Yao, Xuan Liu, Bao Ge, and Shu Zhang. Large language models for robotics: Opportunities, challenges, and perspectives. Journal of Automation and Intelligence , 2024.

Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571 , 2025.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In ACL , 2023.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In NeurIPS , 2024.

Chenxing Wei, Jiarui Yu, Ying Tiffany He, Hande Dong, Yao Shu, and Fei Yu. Redit: Reward dithering for improved llm policy optimization. arXiv preprint arXiv:2506.18631 , 2025.

Muning Wen, Junwei Liao, Cheng Deng, Jun Wang, Weinan Zhang, and Ying Wen. Entropy-regularized token-level policy optimization for language agent reinforcement. arXiv preprint arXiv:2402.06700 , 2024.

Fangzhou Wu, Shutong Wu, Yulong Cao, and Chaowei Xiao. Wipi: A new web threat for llm-driven web agents. arXiv preprint arXiv:2402.16965 , 2024.

Sean Wu, Michael Koo, Lesley Blum, Andy Black, Liyo Kao, Zhe Fei, Fabien Scalzo, and Ira Kurtz. Benchmarking open-source large language models, gpt-4 and claude 2 on multiple-choice questions in nephrology. NEJM AI , 2024.

Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-metajudge. In EMNLP , 2025.

Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675 , 2024.

Haoyi Xiong, Jiang Bian, Yuchen Li, Xuhong Li, Mengnan Du, Shuaiqiang Wang, Dawei Yin, and Sumi Helal. When search engine services meet large language models: visions and challenges. IEEE Transactions on Services Computing , 2024.

Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprint arXiv:2312.11456 , 2024.

Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning. arXiv preprint arXiv:2502.19613 , 2025.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations , 2024.

Surendra Yadav. Aeroquery rag and llm for aerospace query in designs, development, standards, certifications. In 2024 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) , 2024.

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122 , 2024.

An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report, 2025.

Qwen: An Yang, Baosong Yang, Beichen Zhang, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 , 2025.

Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. In NeurIPS , 2023.

Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412.18319 , 2024.

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020 , 2025.

Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, and Xian Li. Naturalreasoning: Reasoning in the wild with 2.8m challenging questions. arXiv preprint arXiv:2502.13124 , 2025.

Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837 , 2025.

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. In NeurIPS , 2022.

Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.12937 , 2025.

Kongcheng Zhang, Qi Yao, Baisheng Lai, Jiaxing Huang, Wenkai Fang, Dacheng Tao, Mingli Song, and Shunyu Liu. Reasoning with reinforced functional token tuning. arXiv preprint arXiv:2502.13389 , 2025.

Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised llm reasoning incentivization. arXiv preprint arXiv:2504.05812 , 2025.

Chenyang Zhao, Xueying Jia, Vijay Viswanathan, Graham Neubig, and Tongshuang Wu. Self-guide: Better task-specific instruction following via self-synthetic finetuning. In COLM , 2024.

Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21493 , 2025.

Hanwei Zhu, Xiangjie Sui, Baoliang Chen, Xuelin Liu, Peilin Chen, Yuming Fang, and Shiqi Wang. 2afc prompting of large multimodal models for image quality assessment. IEEE Transactions on Circuits and Systems for Video Technology , 2024.

Hanwei Zhu, Haoning Wu, Yixuan Li, Zicheng Zhang, Baoliang Chen, Lingyu Zhu, Yuming Fang, Guangtao Zhai, Weisi Lin, and Shiqi Wang. Adaptive image quality assessment via teaching large multimodal model to compare. In Advances in Neural Information Processing Systems , pages 3261132629, 2024.

Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084 , 2025.

Appendix

Table of Contents

A Additional Related Works

Self-Iteration Methods. Multi-round iterative training enables models to reach their performance upper bound progressively, and self-iteration has been widely explored. Pang et al. [50] constructs preference pairs using rule-based rewards for DPO training. Chen et al. [3] adopts adversarial learning based on human-labeled responses, making the method reliant on label quality. Dong et al. [8] performs DPO-based iteration with a fixed reward model, while Wu et al. [70] formulates training as a zero-sum game using a learned preference model. Rosset et al. [52] proposes DNO, which uses an expert model to estimate win rates by regressing internal rewards through batch-policy iteration. In contrast, we propose a majority-voting reward mechanism that requires no external supervision. Compared to prior offline approaches [81, 84, 69], our method adopts an online RL framework. We also address data-scarce scenarios by generating instructions through self-instruction that adapt to the evolving capabilities of the model, and by using a simple yet effective majority-voting reward for accurate and efficient self-iteration.

B KL Divergence Estimators

There exist multiple estimators for KL divergence, such as k 1 , k 2 , and k 3 , defined as follows:

GRPO uses the k 3 estimator as an external loss component, while Reinforce++ and RLOO adopt the k 1 estimator, which is incorporated into the reward as a KL reward.

C Implementation Details

All experiments are run on a cluster with 8× NVIDIA RTX A6000 GPUs, a 96-core Intel Xeon Gold 5318Y CPU, and 512 GB RAM.

C.1 Details of Reinforce++ Algorithm

Assuming a training batch contains n bs questions and each question is associated with n vote sampled responses, G ij ( s t , y t ) denotes the return at the t -th token of the j -th sampled response y i,j for the i -th question x i . We compute the mean and standard deviation of the return across the batch as:

Then, the advantage is computed as:

We would like to further clarify the computation of G ij ( s t , y t ) as follows:

where R ij ( s k , y k ) denotes the token-level reward at the k -th token of the j -th response to the i -th question and KL ij ( k ) denotes the token-level KL divergence at the k -th token between the current

model and the initial model for the j -th response to the i -th question. A negative sign is applied to serve as a KL penalty. Reinforce++ adopts the k 1 estimator.

In practice, the reward is assigned at the response level, consistent with other related works [62, 89, 54]. Therefore, only the last token receives a reward, and the above equation 9 can be simplified as:

Here, R ( x i , y j ) denotes the reward assigned to the j -th response of the i -th question.

C.2 Details on Chain of Thought

During the RL sampling stage, we prompt the model with 'think step by step and give the final answer', encouraging it to first generate a chain of thought (CoT) and then produce the final answer.

C.3 Details on Filtering Strategies

The proposed online instruction filter consists of four components:

(1) The similarity filter computes the ROUGE-L score between each newly generated instruction and all existing instructions in the dataset. If the score exceeds a predefined threshold (set to 0.7 following the prior work [63]), indicating high similarity to existing instructions, the new instruction is filtered out to encourage diversity.

(2) The keywords filter removes instructions containing specific keywords such as "image", "graph", "picture", "file", "map", "draw", "plot", or "write a program", as they refer to visual content or capabilities beyond the model's scope. In addition, instructions starting with punctuation or non-English characters are also excluded.

(3) The length filter removes instructions that are either excessively long or short. Instructions exceeding 150 words often contain redundant content or even include solutions, while those with fewer than 3 words typically lack sufficient context for problem solving. This filtering step helps maintain the overall quality and clarity of the generated instructions.

(4) The difficulty filter evaluates the proportion of majority answers among the generated responses for a given instruction. Instructions with excessively high or low majority proportions are filtered out to ensure that the retained instructions maintains suitable difficulty for model learning. We set the thresholds as γ diff = 0 . 2 and γ easy = 0 . 8 , as shown in Tab. 6.

C.4 Details on Instruction Generation

We adapt the instruction generation prompt from prior self-instruction work [63] to better suit the mathematical domain. The modified prompt template is shown in Box C.1. The n shot examples used in the few-shot context are randomly sampled from both the seed data and the generated instructions. Specifically, one-fourth ( n shot / 4 ) are drawn from the seed data, while the remaining three-fourths ( 3 n shot / 4 ) from previously generated instructions.

C.5 SR-DPO Rewarding Prompt

Since the prompt used in the original SR-DPO paper [81] is designed for reward estimation on general tasks, it performs poorly on mathematical problems. To ensure a fair comparison in our experiments, we modify the original prompt to better suit the evaluation of mathematical tasks, as shown in Box C.2. Since it is also a self-rewarding method, we use the language model, which is to be trained, as the reward model to estimate the reward.

C.6 RL Hyperparameters

The experimental settings [19] for training LLaMA-3.2-3B-Instruct and Qwen-2.5-7B-Instruct using our SeRL method are shown in Tab. 6 and Tab. 7.

Box C.1: Few-shot instruction generation prompt

Box C.2: Reward estimation prompt in mathematical tasks

Review the user's math question and the corresponding response using the additive 5-point scoring system described below. Points are accumulated based on the satisfaction of each criterion:

-Add 1 point if the response understands the problem and introduces at least one relevant mathematical concept or formula, even if the derivation is incomplete or has gaps.

-Add another point if the response provides the main idea or a key result addressing a substantial part of the problem, yet still contains calculation mistakes, skipped steps, or loose rigor.

-Award a third point if core computations are broadly correct and the logic is mostly coherent; only minor slips or omissions remain, and the overall workflow can be followed and reproduced.

-Grant a fourth point if all calculations are accurate and the reasoning is rigorous, step-by-step, with each step contributing meaningfully to the final answer and no irrelevant or redundant content.

-Bestow a fifth point if, in addition to the above, the solution is expertly presented: clearly structured and well-formatted, cites necessary theorems or justifies key steps, proactively verifies the result (e.g., substitution, extrema check), and possibly offers alternative insights or elegant shortcuts-without extra fluff.

After examining the user's instruction and the response:

-Briefly justify your total score (up to 100 words).

-Conclude with the score using the format: 'Score: <total points>'.

Remember: assess strictly from a math-correctness perspective; logical soundness and computational accuracy are paramount, while clarity and expert insight elevate higher scores.

D Additional Experiments

D.1 SFT Baselines

We conducted this experiment and compared SFT using the sampled truly correct responses of the model. As shown in Tab. 8, we observe a consistent drop in performance. This may be due to the limited quality of self-sampled responses or the model's existing proficiency on those samples, resulting in minimal gains from further fine-tuning. To validate our findings, we additionally conduct SFT using data distilled from Qwen2.5-Math-7B-Instruct, and as we hypothesized, fine-tuning with higher-quality data is indeed necessary to enhance the capabilities of the model.

D.2 Results of More Iterations

We additionally report results for 4 to 6 rounds in Tab. 9, where we observe that the performance of the model gradually converges. As discussed in Section 6, we believe this convergence behavior is largely due to the limited capacity of the underlying foundation model. This observation aligns with findings from EMPO [87], which suggests that pre-trained language models already possess strong reasoning capabilities. In this context, RL post-training may primarily help activate latent reasoning patterns learned during pretraining, rather than introduce new ones.

It is worth emphasizing that our goal is not to achieve unlimited iterative improvement through self-play, which is currently an unrealistic expectation and an unsolved problem across the field. As stated in the introduction, our work focuses on combining self-instruction and self-rewarding methods to incentivize the reasoning abilities of LLMs in data-scarce settings, aiming to achieve performance comparable to training on full high-quality data with verifiable rewards.

D.3 Additional Results in the Medical Domain

To demonstrate that our method can be applied to other domains with limited data, we also conducted experiments in the medical domain. Specifically, we used 500 randomly selected instructions from the MedQA [48] training set as the seed data for our method and compared it with a supervised reinforcement learning baseline trained on the full 10.2k dataset (RL-GT). We evaluated both models on three medical benchmarks: MedQA, PubMedQA [24], and NephSAP [68]. All evaluations are conducted using greedy decoding with a maximum of 1,024 new tokens. As shown in Tab. 10, our SeRL achieves performance comparable to the RL-GT baseline, demonstrating its effectiveness.

E Additional Reward Analysis

E.1 Comparison with Alternative Rewarding Methods

Self-rewarding methods are fundamentally designed to approximate the true reward. Their effectiveness can be evaluated by measuring how closely the estimated rewards align with the ground-truth rewards, as greater agreement indicates more accurate reward estimation. To quantify this, we calculate multiple metrics including cosine similarity, mean squared error (MSE), and mean absolute error (MAE) between the estimated rewards and the ground-truth rewards, where higher similarity and lower error indicate greater accuracy in reward estimation. In our evaluation, we treat the rule-based reward as the ground-truth reward and measure the alignment of the majority-voting, model-based, entropy-based [87], and CAI reward with it. CAI reward is the reward model we train based on the work of Bai et al. [2]. As shown in Tab 11, our method achieves the highest accuracy among all compared approaches.

E.2 Reward Hacking

As evidenced by the learning curves presented in Section 4.3, we observe that the training process becomes prone to reward hacking without the dual-end clipped difficulty filtering strategy. To further investigate the nature of this reward hacking behavior, we conduct a case study, as shown in Box E.1. Notably, we find that although the model correctly derives the answer ( 3 , π 2 ) during the Chain-ofThought (CoT) reasoning, it still outputs 'The final answer is: 0 ' at the end. We also provide a detailed explanation in Section 4.3 of how the dual-end clipped difficulty filtering strategy helps prevent reward hacking by design.

To more clearly present the impact of the difficulty filtering strategy on generated instructions, we analyze the difficulty and diversity of the first-round dataset generated by LLaMA-3.2-3B-Instruct before and after the difficulty filtering strategy. We denote the generated dataset from iteration n as D n gen = ⋃ t ∈T n B t gen , where T n represents the set of all generation steps in iteration n . Accordingly, the first-round generated dataset is denoted as D 1 gen . We hypothesize that more difficult instructions require more complex CoT reasoning, which tends to result in longer responses. Qwen-3-32B [77], with its strong overall performance, can correctly answer most questions in the dataset, making it a reliable proxy for assessing instruction difficulty. Therefore, we use Qwen-3-32B to answer the instructions in D 1 gen and compare response length distributions as a proxy for difficulty. As shown in Fig. 4a and Fig. 4b, applying the difficulty filtering leads to a reduction in both extremely short and excessively long responses, aligning with our design goal of removing overly easy and overly difficult instructions. To assess data diversity, we compute the ROUGE-L similarity between each

# Instructions

Box E.1: Case study of LLaMA-3.2-3B-Instruct outputs after reward hacking.

Convert the point (0 , 3) in rectangular coordinates to polar coordinates. Enter your answer in the form ( r, θ ) , where r > 0 and 0 ≤ θ < 2 π.

## Step 1: Recall the conversion formulas from rectangular to polar coordinates.

The conversion from rectangular coordinates ( x, y ) to polar coordinates ( r, θ ) can be done using the formulas: r = √ x 2 + y 2 and θ = tan -1 ( y x ) .

## Step 2: Substitute the given coordinates. √

Given the point (0 , 3) , substitute x = 0 and y = 3 : r = 0 2 +3 2 = 3 , and θ = tan -1 ( 3 0 ) .

## Step 3: Evaluate the angle.

Since tan -1 (3 / 0) is undefined, note that the point lies on the positive y -axis, which corresponds to θ = π 2 .

## Step 4: Write the final answer.

The polar coordinates are: (3 , π 2 ) .

The final answer is:

0

800

600

400

200

0

Response Length (tokens)

0

500

1000

1500

2000

(a) Response length distribution of D 1 gen before difficulty filtering.

0.0

0.2

0.4

0.6

0.8

1.0

ROUGE-L

0

100

200

300

400

500

# Instructions

(c) Distribution of ROUGE-L scores between D 1 gen and D seed before difficulty filtering.

0

500

500

400

300

200

100

0

# Instructions

Response Length (tokens)

1000

1500

2000

(b) Response length distribution of D 1 gen after difficulty filtering.

# Instructions

1000

800

600

400

200

0

ROUGE-L

0.0

0.2

0.4

0.6

0.8

1.0

(d) Distribution of ROUGE-L scores between D 1 gen and D seed after difficulty filtering.

generated instruction x i ∈ D 1 gen and the D seed. For each x i , we define:

We visualize the distributions of D 1 gen and D seed based on Φ( x i , D seed ) . As shown in Fig. 4c and Fig. 4d, the ROUGE-L distributions before and after filtering indicate that the diversity of D 1 gen is largely preserved, suggesting that the dual-end clipped difficulty filtering strategy effectively controls for difficulty without influencing instructions diversity.

F Additional Generated Instructions Analysis

To validate the reliability of the data generated by our self-instruction method, we conduct a comparison against both the seed dataset D seed and the full MATH training set in terms of length, quality, difficulty, and diversity. The dataset D seed consists of 500 examples uniformly sampled from different difficulty levels within the MATH training set and serves as the few-shot context for all subsequent data generated through our self-instruction method. Specifically, D 1 gen , D 2 gen , and D 3 gen respectively

Box F.1: Instruction quality evaluation prompt

You are an expert evaluator of math problems. Please evaluate the overall quality of the following math problem.

Math Problem:

<INSTRUCTION>

First, analyze this problem in detail, considering all aspects including clarity, educational value, appropriateness, and whether it tests meaningful mathematical concepts.

After your analysis, provide your rating on a scale from 0 to 5, 0 = Extremely poor quality mathematical problem 5 = Excellent quality mathematical problem where:

Your response should include your detailed analysis followed by your rating. Make sure to include your final rating in this exact format at the END of your response:

FINAL RATING: [0-5]

Remember to use only whole numbers (integers) from 0 to 5 for your rating, not decimals or fractions.

denote the data collections cumulatively generated by LLaMA-3.2-3B-Instruct during the first, second, and third iterations of training.

F.1 Length and Quality

We compare the length and quality score of the datasets D 1 gen , D 2 gen , D 3 gen , D seed, and the MATH training set. The length is measured by the number of tokens obtained using the Qwen-3-32B tokenizer, where a longer instruction typically indicates higher complexity. Invalid long instructions composed of repetitive sentences have already been removed by our online filtering strategy. Among the remaining instructions, a greater average length suggests higher semantic and structural complexity. The quality score is computed by prompting Qwen-3-32B to evaluate each instruction based on criteria such as completeness of the instruction description, clarity, and overall usefulness or relevance. The evaluation prompt is provided in Box F.1.

As shown in Tab. 12, the results reveal that both the instruction lengths and quality scores of the generated instructions are comparable to those of D seed and the MATH training set. This suggests that the instructions produced by our method not only maintain sufficient complexity comparable to that of the original datasets but also achieve a similar level of semantic quality, as assessed by the evaluation model. These findings indicate that our self-instruction strategy yields high-quality instructions, despite being carried out in a data-scarce setting.

MATH train

seed

1

gen

2

gen

3

gen

0

200

400

600

800

685.4

605.9

496.7

425.1

432.1

(a) Comparison of response lengths across D 1 gen , D 2 gen , D 3 gen , D seed and the MATH training set. To ensure consistency in response generation, we set the generation temperature to 0, thereby enabling deterministic greedy decoding.

sell

have

save

plan

hold

find

produce

make

surround

charge

offer

score

take

fill

contain

represent

paint

cost

buy

give

total

loaf

type

cookie

box

shelf

sale

length

money

trip

event

project

day

gallon

liter

centimet

r

maximu

value

length

volume

area

type

box

product

widget

batch

garden

type

co

kie

garden

area

flowerbed

lawn

fee

rate

fare

price

discount

promotion

type

deal

point

average

grade

hour

minute

worke

exam

tank

pool

bot le

vat

marble

total

pencil

gal on

number

value

curve

room

wal

garden

col

r

\$6$.

ap le

rice

box

shirt

se

dog

-0.

5x

box

v

mean

(b) Analysis of verbs and nouns diversity in D 1 gen .

0.0

2.5

5.0

7.5

10.0

12.5

15.0

Dimension 1

2.5

0.0

2.5

5.0

7.5

10.0

12.5

15.0

Dimension 2

MATH train

seed

1

gen

F.2 Difficulty

To assess the difficulty level of each dataset, we employ Qwen-3-32B to generate responses for all instructions. We then compare the average response lengths across datasets, based on the assumption that more challenging instructions generally elicit longer reasoning chains from LLMs, thereby leading to longer responses. As shown in Fig. 5a, the average response lengths for the generated datasets are close to those of the MATH training set. This suggests that the generated data exhibits a comparable level of difficulty to the original dataset.

F.3 Diversity

Verb-Noun Structures. We analyze the verb-noun structures of instructions in D 1 gen . Specifically, we employ the Berkeley Neural Parser [28, 29] to extract the main verbs nearest to the root and their first direct noun objects from each instruction. Fig. 5b shows the top 20 verbs with their top 4 associated nouns, revealing diverse and syntactically rich verb-noun constructions in the generated instructions.

ROUGE-L Distribution. We compute the ROUGE-L score between each instruction in D 1 gen and D seed. The ROUGE-L distribution is shown in Fig. 4c. As observed, most generated instructions exhibit low ROUGE-L similarity with D seed. This indicates that the model does not merely mimic D seed but instead generates diverse and novel instructions.

UMAP Projection. We project the BERT features of the instructions in D 1 gen using UMAP [45], as shown in Fig. 6. The visualization shows that the generated instruction points are not tightly clustered but instead exhibit a wide distribution similar to that of the MATH training set. This phenomenon further demonstrates the diversity of the generated dataset.

G Additional Analysis of the Self-Iteration Upper Bound

As shown in Fig. 7, we observe that after multiple iterations, the Maj@16 performance of the model does not exhibit a consistent upward trend. Since Maj@16 serves as an empirical upper bound for Pass@1 under our framework, the lack of improvement in Maj@16 suggests that the performance ceiling of the model remains unchanged. This observation is consistent with the conclusion of Yue et al. [83], which states that reinforcement learning with verifiable rewards primarily converts Pass@K performance into Pass@1 without fundamentally improving the capacity of the model. Although our method does not lead to incremental improvements in Maj@K, it still achieves substantial gains in Pass@1 performance in data-scarce scenarios through unsupervised reinforcement learning. These results highlight the practical utility of our

iter0

iter1

iter2

Iteration

55

60

65

70

75

80

Qwen-2.5-7B-Instruct

LLaMA-3.2-3B-Instruct

approach, particularly in specialized, data-scarce domains where labeled data is limited.

H Broader Impact

This work explores reinforcement learning for enhancing the reasoning capabilities of LLMs in datascarce scenarios, which has the potential to significantly broaden the accessibility and applicability of advanced AI systems. By reducing reliance on large-scale, high-quality labeled datasets, our method could enable the development of performant LLMs in data-scarce domains such as clinical diagnostics [58, 44, 60] and aerospace engineering [36, 5, 75], where expert-annotated data is often expensive or impractical to obtain. However, the ability to generate and reinforce synthetic data also introduces potential risks. Without careful oversight, unsupervised self-improvement loops may propagate or even amplify biases present in the initial seed data. Additionally, models trained in data-scarce settings may exhibit spurious generalization patterns, especially in high-stakes domains like clinical decision-making or aerospace control. To mitigate these risks, we incorporate a robust online filtering mechanism to enforce baseline quality and control difficulty during data generation. Nevertheless, domain-specific filtering and safety measures will be essential in future deployments, especially when adapting the framework to sensitive or high-impact application areas.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims clearly delineate the contributions and scope of this work. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 6 provides a comprehensive discussion of the limitations of this work. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This work does not include theory assumptions or proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Appendix C provides detailed experimental information to ensure the high reproducibility of this work.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The data is publicly available with appropriate references provided in the main text, and the code is accessible via a github link with sufficient instructions.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Appendix C outlines all training and testing details necessary for understanding the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Training Large Language Models with Reinforcement Learning requires substantial computational resources and time. Therefore, we report the experimental results using a fixed random seed for reproducibility in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix C provides a detailed description of the computer resources utilized in this work.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The work adheres fully to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Appendix H provides a detailed discussion of the potential societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work does not involve risks for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The assets utilized in this paper are appropriately referenced and accessible to the public.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code is publicly available via a github link and includes detailed instructions for use.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve Crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: This work involves a Reinforcement Learning method for training Large Language Models, which is comprehensively illustrated in the main text.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Self-Play Reinforcement Learning (SeRL) which is a novel approach to bootstrap the LLM training with limited data. It contains two modules called self-instruction and self-rewarding which are used to generate the instructions and assign the response rewards for this additional data. SeRL leverages self-instruction and self-rewarding to increase the limited data during the post-training of the LLMs.\nThe empirical results show that SeRL outperforms the advanced self-play methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n**Quality.** The paper provides a self-play method to enlarge the dataset for the LLMs post-training. The self-instruction is filtered by several criteria to ensure the quality of the generated instructions. In the self-rewarding module, a vote-based reward is used. The ablation study supports the claim of the effectiveness of the self-instruction and self-rewarding modules.\n\n**Clarity.** The paper is well-structured in the presentation of the methodology and experiments.\n\n**Significance.** In practice, it is hard to collect a large amount of data for the LLMs post-training in reinforcement learning. The proposed method provides a way to enlarge the dataset for the LLMs post-training.\n\n**Originality.** The self-rewarding with the vote-based mechanism is novel and derived from the recent proposed work with the 'RL with verifiable labels can be viewed as converting a model’s Pass@K performance into Pass@1'.\n\n\n**Weaknesses:**\n\n**Quality.** The paper provides the empirical results showing that SeRL outperforms the existing methods and two ablation studies to demonstrate the effectiveness of the self-instruction and self-rewarding modules. However, the first ablation study only demonstrates the difficulty filter on LLaMA. It should contain a comprehensive analysis of all filtering criteria.\n\n**Clarity.** The Online Instruction Filter is not very clear and should be elaborated with details.\n\n**Significance.** The paper provides a novel method to enlarge the dataset for the LLMs post-training but lacks methods to mitigate the potential bias introduced by the self-instruction and self-rewarding modules, which limits the practical deployment of the method.\n\nQUESTIONS:\n- Can you elaborate more on the Online Instruction Filter? Can you provide more results on the various criteria used in the filtering?\n- Why use the cosine similarity for evaluating the self-rewarding? \n- I am interested in Table 4. Why does the best performance appear in iterations 1-3 for the three RL methods? And why the Reinforce++ show the most stable and robust performance?\n- I am curious how the difficulty filtering is adopted. The reward is assigned with 0 or 1 for the response. How do you determine the ratio of the difficulty filtering from 0.2 to 0.8?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Self-Play Reinforcement Learning (SeRL), a method for training language models with reinforcement learning without any reward labels and with only a very small initial prompt dataset. The authors position this approach as an alternative to reinforcement learning with verifiable rewards, i.e., they aim to improve language model capabilities and reasoning (as opposed to preference-based RL fine-tuning).\n\nThe method is based on synthetic data generation on two fronts: (1) The initial set of prompts is successively expanded by asking the current iteration of the LLM in-training to propose new prompts in a few-shot manner (self-instruct). Prompts are generated and then filtered with the aim to generate solvable tasks (e.g., no tasks involving visuals) that are neither too difficult or too easy, as measured by the agreement of language model responses when sampling repeatedly. (2) Responses for all prompts (both the initially provided ones and the self-generated ones) are rewarded based on majority voting, i.e., by sampling multiple responses, assuming the majority response is the ground-truth, and then applying a 0/1 reward.\n\nIn their experimental evaluation the authors show that this pipeline can improve language model performance (particularly that of LLaMA-3.2-32B-Instruct and Qwen-2.5-7B-Instruct), reaching and sometimes even exceeding performance of a language model trained on a larger superset of the data with (rule-based) ground-truth rewards. They demonstrate this with a dataset focused on math. Their evaluation also largely focuses on math, but extends to general reasoning. Their method reaches performance of the GT-reward baseline and outperforms baselines with similar iterative self-labeling schemes, but based on DPO instead of scalar-reward RL (I-RPO, SR-DPO).\n\nIn their ablation studies the authors further demonstrate the importance of the components of the proposed framework: The difficulty filtering is necessary to avoid reward hacking, the majority-based rewarding is competitive with ground-truth rewards even when tested in isolation without self-instruct, and the framework performance varies with but is reasonably robust to changes in the underlying reinforcement learning algorithm.\n\nThe main novelties seem to be the difficulty-based filtering in the self-instruct phase and the majority-vote based self-reward, as well as the iterated online setting.\n\nSTRENGTHS AND WEAKNESSES:\nIn the following, I discuss strength and weaknesses on the dimensions of quality, clarity, significance, and originality.\nWeaknesses are marked with a **W**, minor weaknesses with an **M**, notes that do not impact my score with **N**, and strengths with an **S**.\nWeaknesses are numbered for reference purposes.\n\n## Quality\n\n- **S**: The submission is technically sound, all claims are supported by experimental evidence. The methods, metrics, and benchmarks used for evaluation are appropriate. The paper is complete without obvious missing parts.\n- **W1** The empirical evaluation could be more extensive, particularly covering domains beyond math. The authors evaluate on one general reasoning task, but train exclusively on math. This is in slight conflict with the motivation, which mainly discusses hard-to-verify domains beyond math.\n- **W2** Is there any reason you could not do supervised fine-tuning on the majority response instead? If not, this would be a natural baseline to compare against.\n\n## Clarity\n\n- **S** The paper is generally well-written and well-organized.\n- **W3** How exactly does your chain-of-thought approach work? As far as I can tell that is not described in the paper.\n- **W4** It is not entirely clear what exactly is meant by rule-based and model-based rewards.\n- **M1** The explanation of REINFORCE++ is not entirely clear, would benefit from an appendix (e.g., how exactly is the advantage computed?)\n- **M2** The background on self-instruction (line 81) reads like method, not background. It would be valuable to discuss what the actual background is.\n- **M3** 104: Details on how few-shot is selected are missing.\n- **M4** 109: Details on filtering are missing. For example, which words are filtered?\n- **M5** 117 Can you elaborate on why this would result in zero advantage?\n- **M6** 119 Can you give a reference on the vanishing gradients?\n- **M7** 171 How does the number of gradient steps in your approach compare to the baselines?\n- **M8** 213 Why does RL-GT improve other but SeRL does not?\n- **M9** It would be good to have the \"with clipped\" variant in Figure 2 for comparison.\n- **M10** It would be good to see more training plots like Figure 2 in addition to the final results, to see if performance saturates.\n- **M11** 291 Details on self instruct are lacking. This paragraph should \"compare and contrast\" better.\n- **M12** The self-rewarding methods paragraph does not discuss how all the listed approaches compare to the proposed approach.\n- **N** It is a bit hard to understand what underline in the tables mean, especially as the meaning differs between the two tables.\n- **N** I think it is not entirely correct to call the observed behavior reward hacking, since the reward itself is modified. It seems like a sort of collapse.\n- **N** The explanation of the return on line 78 is confusing; similar for advantage on line 80\n- **N** You have previously introduced RL at a token level, now (line 149) you discuss it on a response level.\n\n## Significance\n\n- **S** The method is relatively simple yet significant.\n- **W5** The significance is limited by the evaluation on only a single domain and tasks on which majority voting is appropriate.\n\n\n## Originality\n\n- **S** The work is original to the best of my knowledge. I am not perfectly familiar with all related work, however.\n- **S** Particularly the difficulty filtering for self-instruct, the majority-vote based rewards, and the online setting, seem novel.\n- **N** You may want to discuss https://www.arxiv.org/pdf/2506.06395 as concurrent work.\n\nQUESTIONS:\nPlease see the numbered weaknesses, focusing on the major ones (**WX**). Most of them are actionable, i.e., can be considered questions, although many of the minor ones (**MX**) do not require an individual response. Actual questions regarding the understanding of the paper are listed as weaknesses in the clarity section. The minor points have little impact on my score.\n\nI consider W2, W3, and W5 to be of particular importance. Note that W1/W5 can be addressed by clearly acknowledging and discussing the limitation, and do not necessarily require new experiments.\n\nLIMITATIONS:\n- **W6** The limitations should be discussed in more depth. Of particular relevance are:\n\n    - The limitation to domains where majority voting is applicable.\n\n    - The evaluation on only a single domain.\n\n    - The risk of collapsing to a single response, as both policy and rewards are iteratively updated.\n\n    - Evaluation with only one seed.\n\n    - The risk to reinforce existing biases, particularly in domains that are not as clear-cut as math (briefly discussed in appendix G).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SeRL (Self-play Reinforcement Learning), a framework for training Large Language Models with limited initial data by leveraging self-instruction and self-rewarding modules. Self-instruction generates high-quality instructions based on available data, while self-rewarding estimates response rewards via majority voting, eliminating the need for external annotations. SeRL achieves performance comparable to models trained with extensive, high-quality data and outperforms other self-play methods across various reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The idea of enabling model self-play through self-instruction and self-rewarding is interesting.\n- The authors demonstrate the effectiveness of SeRL across multiple benchmarks, achieving results comparable to those trained with full high-quality data and verifiable rewards.\n\n**Weaknesses**\n\n- In Table 1, the performance improvement of the model slows down as the number of iterations increases. In Table 4, the performance even decreases during the iterations on some benchmarks and algorithms. Does this imply that the SeRL framework cannot support more rounds of self-play iterations, which is key to continual self-improvement? It is suggested that the authors supplement the results with more iterations (e.g., iter4, 5, 6, etc.).\n- In this paper, the authors mainly conduct evaluations on math-related benchmarks. However, one of the claimed contributions is to address the problem of high-quality data and verifiable supervision scarcity in specialized domains such as clinical diagnostics and aerospace engineering. It is suggested that the authors supplement evaluations in these scenarios to illustrate the generalizability of the method and support the claimed contributions.\n- The method of Self-instruction performs data augmentation based on seed data, which leads to the issue that the distribution of generated problems is strongly correlated with the seed data. Will this affect the model's generalization ability?\n- The self-rewarding mechanism relies heavily on the reasoning capabilities of the policy model itself. For challenging problems (e.g., AIME-level questions), the reliability of the reward signal may be compromised since it may not obtain a correct answer through voting. However, this is a common issue with current Self-Rewarding Methods, so it is not my core concern.\n\nQUESTIONS:\n1. In the filtering strategy introduced in Section 3.1, how is filtering based on average accuracy implemented? My main confusion is how the ground truth answer is obtained. I noticed that in Table C.1, there is no requirement to generate the gt simultaneously when generating the question. If gt is obtained through subsequent voting, what methods are there to ensure its reliability?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduces SeRL, a framework for aligning large language models (LLMs) through a fully automated, self-bootstrapping RL process. SERL eliminates the need for any human-labeled preference data or reward models. The experiment results demonstrate that models trained with SeRL show strong performance, outperforming those trained with SR-DPO, I-RPO.

STRENGTHS AND WEAKNESSES:
Strengths
1. The paper is well written and easy to follow. 

2. SERL provides a practical and robust framework that can significantly reduce the training cost.

Weaknesses

1. If I understand correctly, the self-reward strategy heavily relies on existing models, and there is limited discussion on how to choose the best self-reward models, and if the reward evaluator is miscalibrated or biased, it could result in suboptimal behaviors.

2. While comparisons are made to baselines, SeRL has conceptual overlap with Constitutional AI [1], but this is only briefly mentioned. A deeper empirical or theoretical comparison could enhance clarity.
[1] Bai, Yuntao, et al. "Constitutional ai: Harmlessness from ai feedback." arXiv preprint arXiv:2212.08073 (2022).

3. Some related works are missing, e.g., [2] and [3]

[2] Xu, Can, et al. "Wizardlm: Empowering large language models to follow complex instructions." arXiv preprint arXiv:2304.12244 (2023). 

[3] Gu, Shangding, Alois Knoll, and Ming Jin. "TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning." arXiv preprint arXiv:2403.08694 (2024).

QUESTIONS:
1. How does SeRL compare with Constitutional AI in terms of resource use and stability?

2. What is the minimum evaluator quality required for SERL to work?

LIMITATIONS:
More discussion of failure modes and self-reward bias would be helpful.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-04