ReDit: Reward Dithering for Improved LLM Policy Optimization
Authors: Chenxing Wei
Jiarui Yu
Ying Tiffany He
Hande Dong
Yao Shu
Fei Yu
Venue: NeurIPS 2025 poster ()
Keywords: Discrete Rewards
Reward Signal Perturbation
Gradient Optimization
Convergence Speed
Primary area: deep_learning
TL;DR: We presents Reward Dithering, a technique that enhances reinforcement learning in large language models by adding random perturbations to reward signals, improving training efficiency and convergence speed while maintaining performance.

Abstract: DeepSeek-R1 has successfully enhanced Large Language Model (LLM) reasoning capabilities through its rule-based reward system. While it's a ''perfect'' reward system that effectively mitigates reward hacking, such reward functions are often discrete. Our experimental observations suggest that discrete rewards can lead to gradient anomaly, unstable optimization, and slow convergence. To address this issue, we propose ReDit (Reward Dithering), a method that dithers the discrete reward signal by adding simple random noise. With this perturbed reward, exploratory gradients are continuously provided throughout the learning process, enabling smoother gradient updates and accelerating convergence. The injected noise also introduces stochasticity into flat reward regions, encouraging the model to explore novel policies and escape local optima. Experiments across diverse tasks demonstrate the effectiveness and efficiency of ReDit. On average, ReDit achieves performance comparable to vanilla GRPO with only approximately 10% the training steps, and furthermore, still exhibits a 4% performance improvement over vanilla GRPO when trained for a similar duration. Visualizations confirm significant mitigation of gradient issues with ReDit. Moreover, theoretical analyses are provided to further validate these advantages.

ReDit: Reward Dithering for Improved LLM Policy Optimization
Chenxing Weit Jiarui Yu², Ying He Hande Dong", Yao Shu'r Richard Yu! & Guangdong Lab of Al and Digital Economy (SZ), China
CCollege of Computer Science and Software Engineering, Shenzhen University, China
'Hong Kong University of Science and Technology (Guangzhou), China
Tencent, Shenzhen, China
School of Information Technology, Carleton University, Canada weichenxinng222 @email............. szu. edu.. en, waoshu@hkust-tae edu, cn
Abstract
DeepSeek-RI has successfully enhanced Large Language Models (LLMs) rea- soning capabilities through its rule-based reward system. While it's "perfect" reward system that effectively mitigates reward hacking, such reward functions are often discrete. Our experimental observations suggest that discrete rewards can lead to gradient anomaly, unstable optimization, and slow convergence. To address this issue, we propose ReDit (Reward Dithering), method that dithers the discrete reward signal by adding simple random noise. With this perturbed reward, exploratory gradients are continuously provided throughout the learning process, enabling smoother gradient updates and accelerating convergence. The injected noise also introduces stochasticity into flat reward regions, encouraging the model to explore novel policies and escape local optima. Experiments across diverse tasks and different LLMs demonstrate the effectiveness and efficiency of ReDit. On average, ReDit achieves performance comparable to vanilla GRPO with only approximately 10% the training steps, and furthermore, still exhibits 4% performance improvement over vanilla GRPO when trained for a similar dura- tion. Visualizations confirm significant mitigation of gradient issues with ReDit. Moreover, theoretical analyses are provided to further validate these advantages.
Introduction
Reinforcement learning (RL) is pivotal in Large Language Models (LLMs) development [1 2 4. Initially, RL from human feedback (RLHF) [5] was employed to align pre- me-ttrained LLMs with human preferences 77 [] This typically involves training a separate reward model (RM) on human preference data [9. which then guides the LLM policy optimization [1O]. While effective, this approach introduces considerable training overhead [TT]. Subsequently, methods like Direct developed, enabling L.LMs to learn directly from preference data and thus bypassing explicit RM training. However, these methods still require extensive collection of high-quality preference data. For reasoning tasks such as mathematics and coding, DeepSeek-RI R1 [14] with Group Relative Policy Optimization (GRPO) proposes an alternative: optimizing the LLM policy directly using rule-based reward system 16 171. thereby avoiding the need for external RMs or large preference datasets. For instance, such system might assign a reward of 1 for outputs meeting predefined criteria (e.g., correctness, format compliance) and 0 otherwise [14]. The simplicity and unbiased nature of these rule-based rewards prevent LLMs from hacking them, potentially fostering enhanced reasoning capabilities [18]
Preference Optimization (DPO) [12] [3]
corresponning author.
Work done during an internship at Tencent.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) GRPO Training Dynamic on GSM8K
(b) GRPO Training Dynamic on GSM8K with ReDit
30
5.0
5.0
25-
25-
4.5
4.5
Gradieat Norma
Gradieat Norm
20-
20
-4.0
Reward
-4.0
Reward
15-
Stepe 1000, Reward: 4,74,
Apcuracy: $5.70
-3.5
Skep: 1000, Reward: 4.84
Accurasy: 89.16
-3.5
10-
5-
3.0
5-
3.0
0--
200
400
600
800
1000
200
400
600
800
1000
Truining Step
Training Step
Gradient Norm
Raw Reward
Smoothed Reward
Vanishing Gradient Points
Exploding Gradient Points
Figure 1:
Training Dynamics of Gradient Norm and Reward for Qwen2.5-7B [3] GSM8K 32
Dataset. The left and right figures compare original gradient norm (before gradient clipping 33 and reward trends across training steps. The original GRPO method (the left figure) suffers from significant gradient instability- both vanishing (red dots, norms < 0.01) and exploding (purple asterisks, norms 5). In contrast, ReDit with Gaussian reward smoothing (the right figure) effectively stabilizes optimization throughout training.
However, such reward functions often discrete, posing significant optimization challenges 19 21) Consider an RL scenario with binary reward [22]: a policy model receives 1 for a correct answer and 0 otherwise. During early training phases, policy LLM rarely generates completely correct answers, resulting in predominantly zero rewards across mini-batches (23]. Although the model may engage in exploratory behavior on difficult examples, the corresponding gradients remain minimal due to small advantage magnitudes [24]. Thus, these hard examples and potentially beneficial explorations (24] are largely unexploited during the early stages. Conversely, the model may repeatedly reinforce easy examples (25). thus reducing incentives to explore alternative strategies for more difficult problems [26]. This phenomenon can lead training stagnation in intermediate and advanced stages. Consistent with this, as shown in Fig. (the left figure), we observe that the
policy model frequently suffers from gradient vanishing 27128 28 or explosion [30] during these
phases. This combination of insufficient exploration and gradient instability substantially impedes model convergence, representing a critical obstacle to efficient RL LLMs.
This observed phenomenon highlights that even per- feetly accurate discrete reward functions face signif- icant limitations within gradient-basee optimization frameworks. Lending theoretical support to this, recent studies [34] 35| 36 have established that a singular focus on increasing reward model accuracy does not necessarily translate to enhanced language model per- formance. particular, [36] theoretically substantiates the necessity for effective reward models integrate adequate variance and uncertainty to enable efficient op- timization. The theoretical details are given in Sec. 3.2.
Reward Cumulative Distribation Comparison
1.0
Reward
Cumulative Probability
R 0.01
0.8
Reward with ReDing - 0.05)
Reward with ReDin 0.1)
0.4
0.0
Consequently, we believe that an excellent reward sys- tem should achieve careful equilibrium between cor- rectness and sufficient variance.
-0.2
0.0
0.2
0.4
0.6 Value
0.8
1.0
1'2
Figure 2: The figure illustrates how ReDit of different variances gradually smooth the reward distribution, showing the smoothing effect of perturbations of different variances on the reward distribution.
Inspired by these observations and theoretical insights, we propose ReDit, a simple but effective technique that introduces zero-mean random perturbations to discrete reward signals during training. By adding small random noise to the reward function (Fig- 2). ReDit effectively softens the original hard reward function. Compared to hard reward function, softened one can provide greater reward variance within mini-bare mini-batches,

---

which, as indicated by previous research, can lead to enhanced model performance and accelerated convergence.
Fig. (the right figure) illustrates the impact of our proposed ReDit on LLM policy optimization for the GSM8K [32 task. The orange lines indicate that during early training phases, GRPO with ReDit achieves significantly higher average rewards compared to the baseline (GRPO without ReDit), demonstrating the efficacy of our method. We hypothesize that ReDit encourages broader exploration by assigning varied rewards to outputs that only partially meet strict evaluation criteria, thereby accelerating convergence. Towards the end of training (1000 steps), while both policy models attain high rewards, our approach with ReDit exhibits superior performance on the test set, indicating enhanced generalization. Additionally. ReDit demonstrates more robust gradient updating, As shown in Fig. (the left figure), phenomena such as gradient vanishing (red point) and explosion (purple star) emerge during training with the baseline. In contrast, GRPO with ReDit maintains stable gradients throughout the training process. These findings highlight the advantages of ReDit: more stable policy optimization, faster convergence, and improved overall performance.
Moreover, theoretical analysis indicates that a greater reward variance can enhance performance and accelerate convergence in reinforcement learning [36]. We increase reward variance within mini-batches while preserving the expected gradient through reward dithering. By carefully injecting noise into the reward function, ReDit achieves a balance between reward signal fidelity and reward variance, leading to enhanced policy optimization.
In summary, main contributions are:
We observe that policy optimization under discrete reward functions suffer from unstable gradients and slow convergenc((Section 3.1
We propose Reward Dithering (ReDit), a simple yet effective technique that introduces perturbations to discrete rewards. This method is shown to accelerate convergence speed and enhance final model performance (Algorithm [[ and Section
Extensive experiments across diverse downstream tasks, reinforcement learning algorithms, and perturbation distributions demonstrate that ReDit achieves superior performance and enhanced convergence (Section 5
Theoretical analysis proves that ReDit produces an unbiased estimate of the original gradient (Proposition [ and introduces beneficial gradient variance that mitigates vanishing and exploding gradients (Proposition 2)
Preliminaries
We frame LLM generation as sequential decision-making problem solvable via RL. The process is modeled Markov Decision Process (MDP) (37 where the state 8; = q; Oct includes the prompt q and generated tokens Oct, the action is the next token selected from the vocabulary, and the policy (o:(())) is parameterized by 0. The goal is to optimize the policy to maximize the expected sequence-level reward R(g. o) E
1= r(se, 0:) over the prompt distribution pQ:
J ( Eg-pa [Eowmat- [R(g, o)]]
(I)
Recently, GRPO 15 was proposed as PPO alternative that eliminates the need for independent RMs and value functions. GRPO typically processes sparse, discrete rewards directly, rather than continuous RM scores. For tasks like mathematical reasoning, this discrete reward R(g, o) E {0, 1) is often determined by simple function checking correctness format. GRPO estimates the advantage ÁGRPO by sampling G' responses {or Jo and normalizing their discrete rewards within the set. Its objective function, which includes a KL divergence term DKL (eer) for stability, is given by:
lo.1
(f) = Eyupo
min (rs)(0)A(ORro TRPO .lip (rr, c, + AGRRO)
A
G
(2)
-gE DKI. (-))(
where (0)
Tu(oLe|s..e). Subsequent methods such as DAPO 38 GR.O [39]. and REIN-
To (e.,))
FORCE++
40 generally adopt this discrete reward paradigm (see Appendix A for more related
3

---

GRPO Training Dynamic GSM8K
(b) GRPO Training Dynamic Math
5.0
25-
4.5
2.5
Gradieat Norm
20-
4.0
Rewura
2.0
Reward
Gradieal
15-
3.5
4
10-
-1.5
8-
3.0
2-
-1.0
0-
2.5
0-
200
400
600
800
1000
1000
2000
3000
4.000
5000
Truining
Training Step
Gradient Norm
Raw Reward
Smoothed Reward
Vanishing Gradient Points
Exploding Gradient Points
Figure 3:
Qwen2.5-7B [31] Gradient norm and reward training dynamics of standard GRPO on GSM8K and MATH datasets. During the whole optimization process, the gradient of standard GRPO unstable, and there are a lot of gradient vanishing or gradient exploding cases.
work). While simplifying the overall RL process by avoiding complex RMs, this shift to discrete, sequence-level rewards introduces significant optimization challenges. The inherent sparsity and abrupt value changes (e.g., 0 to 1) hinder policy gradient estimation and lead to training instability (Section 3.1
Motivation
This section articulates the fundamental motivations driving our research and establishes the critical challenges that our work aims to address. In Section 3.1 we examine the optimization challenges inherent discrete reward structures, followed by an exposition of the theoretical principles informing our methodological framework in Section 3.2
3.1 Difficulties in Optimization Caused by Discrete Rewards
Optimizing LLM policies using algorithms like GRPO in conjunction with discrete sequence-level rewards (e.g., binary correctness metrics) presents significant optimization challenges. Fig. 1 plots the policy gradient norm (blue line) and average reward (orange line) during standard GRPO training on the GSM8K and MATH datasets, respectively. Two main issues are immediately apparent: Gradient Vanishing. The figure illustrates instances where the gradient norm approaches zero (red dot), oc- curring when most examples in GRPO batch yield iden- tical binary rewards. Consequently, the population relative advantage estimate ÁGRPO becomes negligible across ex- amples, providing insufficient learning signals and caus- ing training stagnation. This phenomenon is evident in
Test Accuracy on
Accuracy, 49.4555
Gradicnt Vanishing Accumeyy Stagnation)
49
Text Accuracy (%6)
45
ITING
Grudicat Explacion (Aecuracy Drop)
Fig. 3 the right figure) post-step 2000.
39
Gradient Explosion. Conversely, training dynamics ex- hibit sporadic sharp spikes in gradient norm (purple as- terisks) when small policy changes cause sequences to transition from incorrect (reward 0) to correct (reward I). These transitions create disproportionately large advantage estimates for newly successful sequences, triggering sud- den, destabilizing gradient updates as shown Fig. the left figure). Such spikes induce reward fluctuations in sub- sequent steps, hindering smooth convergence and learning efficiency.
55,000
Trinnnn Sleps
Figure 4: GRPO has unstable perfor- mance on the MATH test set. The fig- ure plots the test accuracy achieved for the checkpoints saved during the train- ing run shown in Fig. the right figure).
The discrete, sparse rewards induce unstable oscillations between vanishing and exploding gradients. Fig. demonstrates that model performance fluctuates correspondingly with these oscillations. This

---

inherent instability not only compromises optimization efficiency but also serves as a key motivation for our research.
3.2 Theoretical Principles to Address the Limitations of Discrete Rewards
To overcome the critical challenges with discrete rewards outlined in Section [.1] we propose a direct approach enhance reward signal quality. Our solution derives from key theoretical insights in policy optimization, particularly Theorem | and Theorem 2 which reveal fundamental relationships between reward variance, model accuracy, and learning efficiency. See Appendix B.1 for definitions.
Theorem (Policy network optimization time lower bound). From Theorem 1 in 47). Sup- pose that we maxinrize the objective (Eq. (T), using a general autoregressive policy ne (x) IP s y For any 7 > O, prompt x E X, and reward function r, the time it takes until Ey-- (1x) [x) 2 By- yxx, + is:
Ω Exxss (xary x))]
(3)
The reward variance is: vary~ (rx, = Ex (x0) Ey x)) y']]].
Theorem I establishes that the time f. required for policy improvement is inversely proportional to reward variance. When rewards exhibit insufficient variance variance-failing to adequately differentiate between high-quality and low-quality outputs under policy na, convergence slows significantly. This finding suggests that strategically increasing reward variance can accelerate policy convergence.
Theorem 2 (Policy network optimization time upper bound). From Theorem 2 in 41). Assume wo is a policy of the form Ty (yy (x) = (oftmax(6...x)) Given a prompt x E S, let y > O and denote by t., > O the initial time of By- initial policy Te(o), a perfect RM converges to by that can be arbitrarily large, while a relatively inaccurate RM has an upper bound of (TTecO) (1 (x)
(x.))))) 2 Ey- (-2c) (///////// (r. y)) + 7. For any
Complementarily, Theorem 2 demonstrates that effective reward models must incorporate calibrated degree of uncertainty. This controlled uncertainty creates essential exploration space during early training stages, preventing premature convergence and facilitating more efficient optimization.
While perfectly accurate reward functions resist reward hacking, they paradoxically impede opti- mization by producing discrete rewards with minimal variance and insufficient randomness. This limitation severely constrains the growth rates of both training reward TRM and true reward ro during policy gradient updates. To address this fundamental tension, we introduce ReDit-a a method that injects zero-mean perturbations into discrete rewards. This approach preserves the expected reward value while introducing beneficial variance and controlled uncertainty in each update step, dramatically improving both model performance and convergence speed.
4 Reward Dithering (ReDit)
As discussed previously, the discrete nature of rewards commonly used in GRPO can lead to unstable gradient dynamics. To address this, we propose ReDit. The core idea, detailed in Algorithm is to inject calibrated, zero-mean perturbations into the discrete rewards obtained from sampled outputs before using them to compute the GRPO objective for policy updates. Importantly, our ReDit method preserves the overall optimization structure of the GRPO objective function as defined in Eq. 2 the optimization still aims maximize this objective.
The crucial modification introduced by ReDit lies in how the advantage term AGRPO within Eq. is computed. Instead of directly using the raw discrete rewards r; r(o:) obtained for each sampled output o; in the batch {o: GG (line 3 in Algorithm l). we first compute smoothed rewards 7, This is done by adding independently sampled zero-mean perturbation (e.g., from N(0, o²) or or a) to each discrete reward (line 4 in Algorithm D:
+
(4)
5

---

These smoothed rewards [ix]e= are then used as the basis for calculating the advantage. GRPO often computes advantage based on the relative performance within the batch. typically involving normalization. With ReDit, the core component of the advantage calculation, which relies on these rewards, is effectively modified as follows:
Ti mean (meam(frxl)) (frx)
f. mean (Fxle=))
GRPO x
std(fr )E
ReDit
A
Dithering
X std (
(5)
Thus, the relative standing of each output o, within the batch, which informs its advantage AGRPO used in Eq. (2) is determined by the continuous smoothed reward r; rather than the discrete r,. This substitution transforms the optimization landscape. By introducing continuous variations via Fs. the added noise provides informative, non-zero gradients even when discrete rewards r; are sparse or identical within batch, mitigating gradient vanishing. It also dampens the sharp changes in expected advantage resulting from small policy shifts affecting discrete outcomes, thus reducing the likelihood of gradient explosion. This overall smoothing effect facilitates more stable gradient flow, enabling more robust and efficient optimization of the policy Ty using the GRPO objective (line in Algorithm I
Algorithm 1 ReDit within one optimization step
1: Input: Base policy n Discrete reward function r: -> (0,1,2,3,...) Prompt q Number of samples G. Noise parameters: Gaussian std dev >> 0r Uniform radius 0.
2: Output: Updated policy Te.
3: Sample G outputs {o ~ Te q) and compute v; <- r(o:) for i -....,C G.
4: Sample & (0, o²) or 21[aa,,00 a) and compute F; r, 6 for i = 1,..., G./// Generate noise and smooth rewards.
5: Compute JGRPO using {fi) and 0 Optimize(Pals JGRPO, f:).// Optimization
6: return Updated policy πg.
Empirical Results
This section presents a thorough evaluation of our ReDit framework, assessing its effectiveness and efficiency. We begin by detailing the datasets and experimental configurations in Section Subsequently, Section 5.2 provides a comprehensive analysis of the primary findings. To isolate the contributions of key components, we also conduct ablation studies, the results of which are presented in Section 5.
5.1 Datasets and Setup
To rigorously evaluate the effective- ness of our proposed ReDit frame- work, conducted extensive exper- iments. The specific experimental set- tings are detailed below.
Table 1: Comparison of the mean and variance of accuracy
for different baselines under 9000 steps on GSM8K.
Name
DAPO
DR.GRPO
REINFORCE++
Datasets. Our dataset selection and setup largely follow the methodology of [15), primarily to assess the math- ematical reasoning capabilities of the models. This encompasses mathemat- ical problem-solving datasets such as GSM8K [32 and MATH 42 as well as the multimodal
Baseline
87.52
86.13
86.25
w/ ours(Gauss)
89.34 (+:: 0.04)
87.69 (#: 0.06)
87.96 (::: 0.03)
w/ ours(Uniformm
88.57 (+: 0.01)
87.34 (#: 0.07)
87.59 (:: 0.09)
+1.82
+1.56
+1.71
geometric reasoning dataset Geometry3K (43. Each dataset provides distinct training and test splits, which utilize accordingly for model training and subsequent evaluation. See the Appendix D.I for details of the dataset.
We designed dataset-specific reward functions. For the GSM8K dataset, which involves simpler problem structures, we implemented several reward types: accuracy-based, strict format adherence, sort format adherence, integer value correctness, and inference step adherence.
Reward Functions.

---

DAPO
D&GRPO
REINFORCE+
and Test Accurccy (%g
Crap -0 49%
Gap 0.14%
-0.0%
100
300
301
de
100
Bic
Training Steps
Training Steps
Training Steps
Figure 5: Accuracy of different GRPO variants (DAPO, DR. DR.GRPO, REINFORCE++) tested on the GSM8K dataset. The horizontal dashed line highlights the performance of using ReDit at about 1000 training steps, and even after 9000 steps, its accuracy comparable the baseline.
For the more complex MATH and Geometry3K datasets, our supervision relied solely on accuracy- based and inference-based based reward functions. Detailed implementations of these reward functions are provided in the Appendix D.2
Initial Policy. To rigorously assess the effectiveness of ReDit without confounding factors introduced by supervised fine-tuning (SFT), we initialized our experiments directly with instruct models without any additional SFT training. Previous research by 44 demonstrated that even random rewards can enhance performance for Qwen models. Therefore, we conducted comprehensive evaluations instruction-tung models, including Swen2.5--B-Instneee Owen2.5-VL-T Instruct [31] Llama-3.2 2.,2-3B-Instruue 1lama-3.1 3-1-88-Instuuc [I]. Mistral-7B-IInstruee-vv. [55, to establish the generalizability of ReDit.
across a diverse
Random Seeds. Our method incorporates random sampling, which can introduce variance to the experimental outcomes. To thoroughly assess the impact of this stochasticity and ensure the robustness of our results, we executed all main experiments using multiple distinct random seeds. Specifically, we selected five seeds: None (no seed), 42, 123, 888, 2025, and 9999. All metrics reported in our final results represent the mean and variance computed across these five independent runs.
Other Training Settings. For parameter-efficien fine-tuning, we employed Low-Rank Adaptation (LoRA) [46]. Our implementation leverages the official GRPO implementation within the TRL. library 47 Specific configurations for LoRA and GRPO parameters are detailed in the Appendix [.3 Model evaluation was conducted using the OpenCompass [48). All experiments were executed on one NVIDIA H20 GPU.
5.2
Main Results
In our main experiments, we vali- date the effectiveness of our proposed ReDit. For these experiments, we pri- marily either uniform smoothing kernel with radius 0.05 or a Gaus- sian smoothing kernel with standard deviation a/ V3. More exper- imental results can be found in the Appendix
Table 2: Test mean and variance of accuracy comparison across datasets for original Backbone, GRPO, and ReDit.
GSM8K
MATH
Geometry3K
Backbone GRPO(Baseline)
84.91
39
40.43
89.07
48.01
43.10
ours(Gauss)
90.76 (#: 0.06)
52.55
0.03)
44.67 (#: 0.03)
ours(Uniforrm)
90.46 0.07)
51.96
0.06)
44.36 0.04)
+1.69
+4.54
+1.57
Accelerated Convergence Across Datasets and LLMs. We demon-
strate that integrating our proposed method, ReDit, with GRPO substantially accelerates convergence and improves final performance across a wide range of datasets (Fig. 7 and LLMs, including Llama- 3.2-3B, Llama-3.1 --B. Ministral-8B and Mistral-7B (Fig. (6). On tested models, both Gaussian and uniform variants of ReDit enable GRPO to reach competitive performance level within merely 1000 training steps. Notably, this performance already surpasses that of the baseline GRPO trained for the full 9000 steps. Consequently, ReDit not only enhances training efficiency but also leads to
7

---

Llama-3.2-33 --Ins-sstrrcttt
Llamu-1.1-8B 1-8B-Instruct
Miaiitrrl-SS-IInsteee
Mistral-7B Bistral-7B--aae Tt-truct
Tedt Accurncy (14)
alaven
Training Steps
Training Steps
Training Stops
Training Steps
GRPO
GRPO + Gua ReDu
GRIO Unifamm ReDii
Figure 6: Accuracy of different LLMs on GSM8K. ReDit improves training efficiency and final performance in various LLMs.
GSM8K
MATH
Geometry3K
Test Accuracy (%)
Outperform 0.09%
Outperform 0.79%
Dutperfor 1.77%
300
200
Training Steps
Truining Steps
Trining Steps
GRPO
GRPO Gauss ReDit
GRPO Uniform ReDit
Figure 7: Test accuracy across datasets. The horizontal dashed line marks ReDit's performance 1000 steps, which GRPO fails to match even after 9000 steps.
superior final accuracy. The Gaussian variant. in particular, consistently yields the strongest results and promotes more stable training trajectories with lower volatility compared the baseline. We also present more experimental results in the appendix. The results on the code dataset are detailed in Appendix [.. the results on the full parameter fine-tuning method are detailed Appendix H.2 and the results on the Deepseek-Ri R1 distillation model are detailed in Appendix
Generalization Diverse Baselines. Fig. presents results from applying ReDit to additional reinforcement learning baselines (DAPO, Dr.GRPO, and REINFORCE++) on the GSMSK dataset. Across all algorithms, ReDit (both Gaussian and uniform variants) consistently enhances performance and accelerates learning. Beyond these early-stage improvements, ReDit also substantially boosts the final accuracy of these baselines, as quantitatively demonstrated in Table [ (Table 2 complement the qualitative evidence in Fig. 5, confirming that ReDit enables faster and more stable learning across diverse algorithms.
These accuracy gains
Optimal Performance with Scheduled Perturbation. We further investigate convergence behavior under various scheduled perturbation schemes: SquareRoot, Cosine, and CosineReverse perturba- tions. These schedules dynamically adjust perturbation variance throughout training, potentially benefiting model learning. Fig. in the Appendix [.4 illustrates the different perturbation schedules, while Fig. Sprresents their performance. Compared to standard GRPO, ReDit achieves both faster convergence and superior final performance, with the CovineReverse perturbation schedule yielding particularly strong results. Additional details are provided in the Appendix F.4
5.3 Ablation Studies
Perturbation variance affects performance. study the sensitivity of ReDit to the perturbation amplitude, we performed an ablation study by varying the parameter a in the Gaussian smoothing kernel with standard deviation σ a/v5 V3. This effectively changes the variance of the applied perturbation. As shown Fig. D applying reward smoothing (i.e., for any a > 0.00) consistently leads to faster convergence compared to the baseline without smoothing (a = 0.00). Moreover, in most cases, increasing the perturbation amplitude (larger a) tends to improve the final performance of the model. Notably, the configuration with 0.05 shows superior performance, achieving not only

---

Scheduled Perturbation accuracy GSM8K.
Performance Comparison of Different Variance
92
Penk Accuracy: 91.84% (CvstneRevarse
Fastest Convergence 89.37% (a-0.05)
Perk Accuricy 90.46% ta (a:0.85)
91
Test Accuracy (%)
91
90
90
89
Test Accuracy
87
GRPO
87-
Standard Gamn
SquaacRest Guzar
86
85-
Conne Cinsue
85
a00.02
a-0.03
a-0
0-0.1
0
1000
2000
3000
4000
5000
5000 7000 2000 9000
84
0
1000
2000 5000 4000 5000
6000
7000
8000
9000
Training Steps
Training Steps
Figure& 8: Cevire perturbation achieves the best performance.
Figure 9: Appropriate perturbation achieves the best performance.
the fastest convergence but also the best peak model performance, see the Fig. alnnotation. However, these results highlight key trade-off. While moderate perturbations are beneficial, excessive perturbations (c.g.. a = 0.5) may over-smooth the reward landscape. This may mask the original reward signal and lead to performance degradation. Conversely, if the perturbation variance is too small (c.g., a = 0.01), the smoothing effect is small and the improvement over the baseline is limited. This suggests that there an optimal perturbation variance. We recommend conducting preliminary experiments smaller dataset to effectively determine this optimal variance before applying it to larger-scale training scenarios. For a detailed theoretical introduction to a, please refer to Section
Isolating the Effect on Discrete Rewards. To verify that the performance gains of ReDit stem specifically from smoothing discrete rewards, we conducted crucial ablation study. In this exper- iment, we replaced the discrete reward signal with a continuous one generated by a reward model pre-trained on human preference data. This model provides continuous quality score within the range [0.1]. We then applied the ReDit perturbation mechanism directly to these continuous rewards. The results, presented in Fig. TO show that applying ReDit in this setting yields no discernible impact on either the convergence speed of model or its final performance. This outcome strongly indicates that the benefits of ReDit are nullified when the reward landscape is already smooth. We therefore conclude that the efficacy of ReDit lies specifically in addressing the optimization challenges inherent to sparse and discrete reward signals.
Performance Comparison of Different Uniform Variance
Impact of Gradient Operations on Accuracy
Test Accurcyy (%)
87-
-B Gestimn Clipping
Dysattic Sumpling
81
RaDa
XM Unifem ReDi
Unifom ReDia
1920
5200
Training, Steps
Training Steps
Figure 10: ReDit has little effect on improving the performance of GRPO based RM.
Figure 11: Appropriate perturbation achieves the best performance.
Comparison with Direct Gradient Manipulation Baselines. We benchmark ReDit against estab- lished techniques that directly address gradient instability: Gradient Clipping [49], which mitigates exploding gradients, and Dynamic Sampling (38), which alleviates vanishing gradients. The objective to compare our ReDit approach with methods that operate directly on the gradient signal. As illustrated in Figure 11] ReDit substantially outperforms both baseline methods. We attribute this performance gap to the inherent limitations of these heuristics. Gradient Clipping, for instance, crudely truncates gradient magnitudes, a non-principled operation that can introduce significant

---

estimation bias. Conversely, while Dynamic Sampling can be effective for vanishing gradients, it offers no mechanism prevent gradients from exploding. In contrast, ReDit stabilizes the training process by smoothing the reward, which provides more principled solution to prevent both gradient vanishing and explosion, thereby leading to more efficient and effective training.
Theoretical Insights
We provides theoretical analyzing how perturbing discrete reward signals with, e.g.. Gaussian noise, accelerates RL convergence, offering a principled explanation for observed empirical benefits.
Problem Setup. Our analysis uses simplified RL framework (from Eq. ①T) focusing binary rewards R(g. E {0, l} for complete outputs (e-g.. GRPO (I5), not token-level rewards. We investigate how Gaussian noise NN(0, injection improves convergence. The perturbed objective is:
((m) = Exppa (ooro((1))Rqg.0))))
where the perturbed reward is R(q. o) = R) +c. +c.
Proposition 1 (Unbiased estimate of gradient). Introducing noise will still ensure the unbiased estimate of the gradient of the original optimization target Eq. (T), that is:
E
[oo( = = [
Remark. Proposition I provides theoretical proof that introducing Gaussian noise perturbations into the discrete reward signal preserves the unbiased nature of the policy gradient estimate. This means that, under the perturbed reward, the expected direction of the policy update consistent with the original objective being optimized. Maintaining this unbiased nature ensures that the injected noise does not introduce systematic biases into the learning dynamics, thus providing a theoretical basis for the empirical observation that approach helps consistently improve performance. See Appendix B.2 for a detailed proof.
Proposition 2 (Introducing the variance of gradient estimation). Suppose we optimizing non-degenerate stralegy, that is, its gradient Vo log ma is not completely zero. Introducing noise will introduce gradient noise on the originally calculated gradient, and its variance is:
Var(Gradient Noise) = o². [ log 0
Remark. In Proposition analyze how Gaussian reward perturbations affectt the variance of policy gradient estimates. Adding Gaussian noise e ~ NN0, to the reward introduces a "gradient noise" component proportional to Vo log (o qq in the gradient estimate. The increased variance has significant optimization benefits: Mitigate vanishing gradients: Gradient noise provides consistent stochastic updates even when the original gradient terms are small vanishing. thus helping to avoid flat regions. Avoid exploding gradients: The randomness induced by the noise enables the optimization trajectory to probabilistically bypass unstable regions of high eurvature. Furthermore, the noise variance can be adjusted to control the magnitude of the gradient noise for optimal results. This mechanism enhances the robustness of policy optimization and explains the empirical improvements observed in training stability and convergence speed from reward perturbations. For detailed derivation, see Appendix B.3
7 Limitations and Conclusions
ReDit Improve the stability of reinforcement learning with zero-mean reward noise theoretically smoothing gradients, preventing gradient instability, and accelerating convergence by increasing reward variance. Benchmarks verify significant improvements in convergence speed and performance. While our approach works, requires careful tuning of the perturbation variance (although we adopt small dataset search strategy, extensive experiments still needed), and future work will aim to achieve automatic variance.
10

---

References
[1] @I@ Meta.
The llama 4 herd.
https://at www.mmea.ccmm met.a.com//log........oo.o. 2025.
1 1amma-4-mulle
Accessed: 2025.
[2] Anthropic. Claude 3.5
sonnet.
https///ww.anthroocc...co/////e
claude-3-55 3------------tt- 2024.
Accessed: 2024.
[3] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Bernadet--hhairoo Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopss, Jonathan Gordon, Morgan Grafstein. Scott Gray, Ryan Greene. Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Hcaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Eukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Knueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung. Daniel Levy, Chak Ming Li. Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Man- ning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishknn, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reicchio Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen Oeeefe Jakub pachocki, Alex Paino, Joe Palermo, Ashley Pan- tuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry. Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Kata- rina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Flizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
[4] Chenxing Wei, Yao Shu, Ying Tiffany He, and Fei Yu. Flexora: Flexible low-rank adaptation for large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Moham- mad Taher Pilehvar, editors, Proceedings of the 63d Annual Meeting of the Association for
11

---

Computational Linguisties (Volume 1: Long Papers), pages 14643-14682, Vienna, Austria, July 2025. Association for Computational Linguistics.
[5] Paul E. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 4302 4310, Red Hook, NY, USA, 2017. Curran Associates Inc.
[6] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 19X.:0:909.08500 099999 2019.
[7] Hao Lang, Fei Huang, and Yongbin Li. Fine-tuning language models with reward learning on policy. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1382-1392, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[8] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
[9] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback, 2024.
[10] Nathan Lambert. Reinforcement learning from human feedback, 2025.
[11] Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li. Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods. IEEE Trunsactions on Neural Networks and Learning Systems, page 1-21, 2024.
[12] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[13] Chenxing Wei, Hong Wang, Ying He, Fei Yu, and Yao Shu. Test-time policy adaptation for enhanced multi-turn interactions with Ilms, 2025.
[14] DeepSeek-AL. Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao. Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin. Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li. Miaojun Wang. Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X.
12

---

Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan. Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
[15] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[16] Dingwen Kong and Lin F. Yang. Provably feedback-efficien reinforcement learning via active reward learning. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc
[17] Zhiqiang Wang, Pengbin Feng, Yanbin Lin, Shuzhang Cai, Zongao Bian, Jinghua Yan, and Xingquan Zhu. Crowdvlm-rl: Expanding ability vision language model for crowd counting using fuzzy group relative policy reward, 2025.
[18] Harris Chan, Volodymyr Mnih, Feryal Behbahani, Michael Laskin, Luyu Wang, Fabio Pardo, Maxime Gazeau, Himanshu Sahni, Dan Horgan, Kate Baumli, Yannick Schroecker, Stephen Spencer, Richic Steigerwald, John Quan, Gheorghe Comanici, Sebastian Flennerhag, Alexander Neitz, Lei M Zhang, Tom Schaul, Satinder Singh, Clare Lyle, Tim Rocktäschel, Jack Parker- Holder, and Kristian Holsheimer. Vision-language models as a source of rewards. In Second Agent Learning in Open-Endedne Endedness Workshop, 2023.
[19] Desik Rengarajan, Gargi Vaidya, Akshay Sarvesh, Dileep Kalathil, and Srinivas Shakkottai. Reinforcement learning with sparse rewards using guidance from offline demonstration. In International Conference on Learning Representations, 2022.
[20] Gautham Vasan, Yan Wang, Fahim Shahriar, James Bergstra, Martin Jagersand, and A. Rupam Mahmood. Revisiting sparse rewards for goal-reaching reinforcement learning, 2024.
[21] Prasoon Goyal, Scott Niekum, and Raymond J. Mooney, Using natural language for reward shaping in reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI' 19, page 2385 2291..1911111 AAAI Press, 2019.
[22] Niladri S. Chatterji, Aldo Pacchiano, Peter L. Bartlett, and Michael I. Jordan. On the theory of reinforcement learning with once-per-episode feedback. In Proceedings of the 35th International Conference Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc.
[23] Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. Enhancing reinforcement learning with dense rewards from language model critic. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 91 19-9138, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[24] Alex James Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in reinforcement learning from human feedback. In International Conference on Machine Learning, 2024.
[25] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang. and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
[26] Lex Weaver and Nigel Tao. The optimal reward baseline for gradient-bast bbaed reinforcement learning. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAr01,, page $38-545, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc
13

---

[27] Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin. Vanishing gradients in reinforcement finetuning of language models. In The Twelfth International Conference on Learning Representations, 2024.
[28] Mohammed Sharafath Abdul Hameed, Gavneet Singh Chadha, Andreas Schwung, and Steven X. Ding. Gradient monitored reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 34(8):4106 1444006444 086664111 11, 2023.
[29] Shauharda Khadka and Kagan Tumer. Evolution-guided policy gradient in reinforcement learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 1196- 1208, Red Hook, NY, USA, 2018. Curran Associates Inc.
[30] Kaichen Zhang, Yuzhong Hong. Junwei Bao. Hongfei Jiang, Yang Song. Dinggian Hong, and Hui Xiong. Gvpo: Group variance policy optimization for large language model post-training 2025.
[31] Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
[32] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
[33] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2020.
[34] Hamish Ivison, Yizhong Wang. Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[35] Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in RLHF: When better reward models don't yield better language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2980-2989, Miami, Florida, USA, November 2024. Association for Computational Linguisties.
[36] Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? In The Thirteenth International Conference on Learning Representations, 2025.
[37] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes, 2015.
[38] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang. Hongli Yu, Weinan Dai, Yuxuan Song. Xiangpeng Wei, Hao Zhou, Jingjing Liu, WingYing Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source IIm reinforcement learning system at seale, 2025.
[39] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding -ileee--lkke training: A critical perspective, 2025.
[40] Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rihf algorithm with robustness to both prompt and reward models, 2025.
[41] Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D. Lee, and Sanjeev Arora. What makes reward model good teacher? optimization perspective, 2025.
14

---

[42] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[43] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning, 2021.
[44] Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang. Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in rlvr, 2025.
[45] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[46] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[47] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforce- ment learning.
com wtt.:/u.....oogggg......... http://ugggggggggggg////// 2020.
[48] OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github. com/ open- compass
2023.
[49] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2020.
[50] Han Zhang. Yu Lei, Lin Gui, Min Yang, Yulan He, Hui Wang, and Ruifeng Xu. CPPO: Continual learning for reinforcement learning with human feedback. In The Twelfth International Conference on Learning Representations, 2024.
[51] Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning, 2025.
[52] Andrew Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transforma- tions: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, ICML 199, page 278-288 San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc.
[53] Marek Grzeš. Reward shaping in episodic reinforcement learning. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS '17, page 565-573, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.
[54] Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham M. Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[55] Maryam Fazel, Rong Ge, Sham M. Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for linearized control problems, 2018
[56] Chenxing Wei, Yao Shu, Mingwen Ou, Ying Tiffany He, and Fei Richard Yu. Paft: Prompt- agnostic fine-tuning, 2025.
15

---

Acknowledgement
This work was supported by the Shenzhen Science and Technology Program (Grant No. ZDSYS2020552171400000 the National Natural Science Foundation of China (Grant Nos. 62271324, 62231020, and 62371309), and the Open Research Fund from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) (Grant No. GML-KF-24-32).
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation.
While "[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we
unable to find the license for the dataset used"). ). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best, judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims are put in the abstract and Section T
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
16

---

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: For further details, please refer to the "Limitations and Conclusions" section, found Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
The claims are put in the abstract and Appendix
B
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
17

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: For detailed information, please refer to the "Method" section in Section 4 Specific experimental settings are
Complete pseudocode can be found in Algorithm [
discussed in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide the instruction and code in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co /c/
/ Soteeeee////eSeedteeeoe iotomissioon ssionP olicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new soun-s benchmark).
18

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https} //nips. cc/ wttpp/ccccccccccccccccccccccc
des SCoeee/odeSSeem ss onPol icy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Specific experimental settings are discussed in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For a detailed comparison, please refer to the experimental sections in Sec- tion
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
19

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [No]
Justification: Specific experimental environments be found in See 5.1 Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have checked the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper mainly studies the impact of discrete rewards on reinforcement learning. does not have any direct positive or negative social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
20

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
We have checked this and confirmed the paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original papers or URLs of the codes, models, and data sets used this article have been cited in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assett are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, wwwpppawittheeeeo.m.pp.pp.. shocooeee com. datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the instruction supplemental material about our code and data.
Guidelines:
The answer NA means that the paper does not release new assets.
21

---

Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper only utilizes LLMs for writing and editing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-stande non-standard components.
http:/rrrriisccccoeeeeeeeeeee//e http///nneeeeeeee//e0ee /025/ Please refer to our LLM policy (nttps::/neurrl /tta::////uri..../ ca/
LLM) for what should should not be described.
22

---

A Related Work
Reinforcement Learning with Discrete Rewards. Group Relative Policy Optimization (GRPO) 151 utilizes discrete rewards generated by a rule-based reward function to guide the policy model update. This reward function, known for its simplicity and unbiasedness, effectively mitigates reward hacking and has demonstrated strong performance. However, GRPO faces challenges related to slow training speed and unstable gradients during training. To address these issues, various methods have been proposed. DAPO (38) introduced a dynamic sampling strategy to improve gradient effectiveness by dynamically filtering invalid samples, thereby increasing sample efficiency, although this reduced training speed. CPPO [50) prunes completions with low absolute advantages, significantly reducing the number of gradient calculations and updates required, which enhances training efficiency but can lead to gradient estimation errors. GPG [1 directly optimizes the original reinforcement learning objective, eliminating the need for proxy loss function and improving training efficiency. However, this simplification may result in significant divergence between the actor and policy models. DR.GR (39] improves token efficiency while maintaining inference performance. Despite these efforts, a critical challenge remains: these algorithms largely neglect the inherent difficulties introduced by discrete rewards during the optimization process. The oscillations caused by gradient vanishing and exploding are major contributors to the slow optimization speed. Our work specifically aims to overcome the challenges in gradient optimization that arise from using discrete rewards.
Addressing Reward Design Challenges in LLM Reinforcement Learning. Designing effective reward functions for identifying optimal strategies the context of Large Language Models (LLMs) [2 53 54]. However, a consensus on the optimal approach for LLM reinforcement learning has not yet been reached [55]. The RLHF framework proposed training a reward model to score L.MM outputs [5. A recurring challenge. by numerous studies, that low reward model accuracy can induce reward hacking 34 35 36)
well-established area of research outside
noted
Conversely, improving accuracy often reduces reward variance, which can slow down policy model convergence due to vanishing gradients [27]. Although the reward function presented in GRPO provides perfectly correct rewards, thereby avoiding reward hacking, it exacerbates gradient instability and hinders optimization speed [5 39). Recent theoretical findings indicate that successful reward function requires trade-off between variance and inaccuracy [41]. Motivated by this, work seeks to design a reward function that effectively addresses the problem of reward hacking while simultaneously facilitating efficient optimization.
B
Theorems and proofs
B.1
Definitions
From Definition 1, in 41 The accuracy and variance of the reward function is as follows:
Definition 1. Given a prompt x E X, the accuracy of reward model TRM : X y + [[1, 1] with respect to a distribution 'D over unordered output pairs is defined by:
[ sign(rRMr(z, v) (R,M(I,)))) = sign(ra(s, - ro(l,
AcCs,p(TRM) = Egy.'' YoD
where na the ground truth reward, 1[: an indicator function, and sign :R + { (-1, 13 is the sign function ?
x + [-1, 1). the Definition 2. Given a policy mo, prompt X, and reward model FRM xxxx
reward variance induced by TAS for no and x is defined by:
( (TREN (2, v) Ey) ( [TRM (oynm/taoo/r////t 27
Var (o vil == E Ey-mr
(8)
For set of prompts, accuracy refers to the mean accuracy the set.
23

---

B.2 Proof of Proposition|
1
The proof of Proposition expressed as follows:
Proof. By the policy gradient theorem, the gradient of the original objective expands to:
VaJ(mo) Pu- Bowno [R(g, Vo log [T(o)))].
(9)
For the noise-injected objective, its gradient becomes:
(TT( B RRqq log (o(ola) logmmooo)))))
(10)
Substituting R(q, o) Rlgg o) + e and leveraging linearity of expectation:
13
Voj o [[8[g, +VV log Ne (o|g)
(11)
o [ o) Vo log (o(44) +B, Eg,o (Vo logg)()))))
(12)
[[VoJ]
Zero-mean noise: E.[c] by definition of (0, o²). Thus, the cross-term vanishes:
V [[V +0= E[VeJ].
(13)
B.3 Proof of Proposition 2
Proof. Consider the perturbed objective function with noise- auggented reward R(g, o) + The estimated value of the gradient of the noise enhancement objective function using n samples is:
v = Σ
V log To(o:)))) (R(gs 0:) + +;)]
(14)
where E; ~ NN )²) is the Gaussian noise. The original reward gradient is:
v Vog ())l))))))))) (R(o)))).(((......)) o:)]]
(15)
Under this condition, the Eq.
14 simplifies to:
vi(0)=
V
v log (o( 4].
(16)
origin gradient
gradient
While the expectation [c] implies the noise contribution's 's mean is zero, the variance of the gradient term persists. To compute this variance, we use the definition: Var(X) = B[x²] (E[()))
Applying this to the noise-induced component e Vo log no we get:
Var (e Vo log (olg)) = [ V log no [1]
(E [e Vo log (o(o)))))
(17)
Since Eie] 0, the second vanishes. For the first term, note that:
E Var(c) (R(e))² = o² 0 ===- o².
(18)
This allows us to simplify the variance expression to:
Var(noise gradient) Var (e Vp (a) = o² E Vo log To (o|g))) >0
(19)
provided Vo log vy is not identically zero (a reasonable assumption for non-degenerate policies).
24

---

(a) GRPO Training Dynamic Math
(b) GRPO Training Dynamic Math with ReDit
20-
2.5
2.5
15
Gradieat Norm
15
2,0
2.0
Reward
Reward
Gradival
10-
-1.5
-1.
1.0
5-
5
00.5
-0.5
0-
0-
0.0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Truining
Training Step
Gradient Norm
Raw Reward
Smoothed Reward
Vanishing Gradient
Exploding Gradient Points
Figure 12: Training Dynamics of Gradient Norm and Reward on Math Dataset.
C Training Dynamic
In this section, we show more Training Dynamic information.
Figure 12 shows the training dynamics of using and not using ReDit the Math dataset, indicating that using ReDit can solve the problems of gradient oscillation and gradient vanishing, and improve training stability
(a)Training Dynamic GSM8K with Uniform ReDit
(h) Training Dynamic GSM8K with Gauss ReDit
5.0
6-
4.5
4.5
Gradient Norm
Gradient Norm
9.
g
4.0
Rewura
4.0
-3.5
335
3.0
3.0
0-
0-
200
400
600
800
1000
200
400
600
800
1000
Training Step
Training Step
Gradient Norm
Raw Reward
Smoothed Reward
Vanishing Gradient Points
Exploding Gradient Points
Figure 13: Training dynamics of gradient norm and reward on the GSM8K dataset, showing the
impact of perturbations of different distributions.
(a) Training Dynamic Math with Uniform ReDit
(b) Training Dynamic Math with Gauss ReDit
2.5
2.5
15
2.0
Gradieat Norm
2.0
Reward
Gradieal
10.
-1.5
11.5
110
1.0
5-
5-
0.5
0-
0-
0.0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Training
Training Step
Gradient Norm
Reward
Smoothed Reward
Vanishing Gradient Points
Exploding Gradient Points
Figure 14: Training dynamics of gradient norm and reward on the Math dataset, showing the impact of perturbations of different distributions.
25

---

Fig 13 and Fig 14 Training dynamics using uniform and Gaussian perturbations. For both uniform
and Gaussian perturbations, ReDit shows amazing gradient stability and training stability.
D Experimental setting
D.1 Dataset
In this section, introduce the statistics of the dataset and the additional processing performed the dataset. The statistics of the dataset are shown Table
Table 3: Number of samples in the train, validation, and test datasets for various dateset.
Number of samples
train dataset
validation dataset
test dataset
GSM8K MATH
7473
1319
7506
5003
Geometry3K
2100
300
601
In addition, We added new templates to the original dataset to ensure the model could complete the required tasks and output formats. It is important note that the added templates did not alter the original dataset, and special processing was performed for different LLMs. The specific examples are follows:
Dataset Format of GSMSK
dataset: 6SM8K
"prompt";
("role*: "system", "content"; "Respond the following Format:
<reasoning> </ressoning> <answer> ...</answer>>))
("role*: "user", "content": *What is the lergast single-digit primo numbar?*),
I"role*: "saslatant", "content"; *creasoning> is diviable by 3 and divieible by but prime. </reasonings
<answer>7</anevee ("zole"; "user", "content"; (question])
annser*; (answer)
Dataset Format of MATH
dataset: KATH
"prompt":
(*rolet: "system", "content": "Reapond folloxing format
cressonings </ressoning> <answer> </as/wewee>>")
(*role*: "user", "content"; "Iquestion)
Let's think step by step and output the final ansver within Wbcxedd).
*ansver"; (answer)
26

---

Dataset Format of 3K0metry3K
dstaset: Geometzy3K "prompt": (*role*; "user", "content": "type"; "Image", "image": (image),
"type": "text", "text"; [question] " FTRST think about the reasoning process Internal oque and then provide the final answes. The zeasoning process MUST BE enclosed within <thinks 1,l,
tags. final answer MUST put in (boxed]
aanserr*: (answer)
D.2 Reward function
We design five reward functions for the GSM8K dataset and show how to implement ReDit:
GSMSK Accuracy Reward Function
def correctness_rewardd haadd__fn_wwthhhhe______aa
wwth_hhittihtttco noi se (prompts, completions, answer
**kwargs) 1ist float]: aet extract_number str) -> str:
match = re. search (r' \d+', s)
return match (0) if match else responses [completion[0] [0] [' contentt] completions]
completion
6
= prompts [-1] [' content')
extrracted_responsee
settract_xmlaaaee (r) for in responses
original_rewards [2.0 if extract_numbes (r) extract_number
a) else 0.0 for r, a in zip (extrratte_ answer) ]
ReDit add noi soewwars
random. uni form 2.0, 2.0) for r
2
inoisy_ rewards Ir random, gauss (0, 2.0 m (3 ** 0.5)) for in original rewards]
return noisy rewards
27

---

GSMSK Int Reward Function
def rert_reward__ee func wuthwddhhhh_hhhhhhhheeeeee with noise (completions, **kwargs) -> 1ist [float
1:
responses [completion [0] contentt) for completion in completions]
extracted [extract xm1 answer (r) for in responses
ariginal_rewarce = [0.5 if r.isdigit() () else 0.0 for in cxtracted[responsee)
6
ReDit add
moisy_reward - [r randon. unt.oommnifoormm 0.5, 0.5) for r in original_rewarde
#noisy_rewards [r random. gauss (0, 0.5 (3 0.5)) for in seiginal_rewarde
9
return noisy rewards
GSM8K Strict Format Reward Function
def strict format Aoriitt reward htw.auchewcciiiii.oooaeeeooooepo
nois (comletioon,, **kwargs)
List|float/: pattern
\S] *?</answer>s" completion_contents [completion [0] ["content"] strrip for completion in completions]
matches (re.match (pattern, content, re.DOTALL |
MULTILIILI
for content in completion_conterr
original_rooards = [1.0 if match else 0.0 for match in matches]
ReDit add noi sa_ssy__wwaaa [r random. uni form (-m form 1.0, * 1.0) for r
or #noisy_rewards [r random, gauss (0, 1.0 m (3 ** 0.5) for
in original_rewann return noisy rewards
0
GSMSK Sort Format Reward Funetion
def soft format reward func_with with noise (completions, **kwargs)
1ist [floa]]] pattern r"* <reassringg http://ssas.aaaaaaaaaaaaaaaaaoaa < IN8
\S] #2</anweer>"*
completion_content = [completion [0] ("content") .strip() for completion in compietions]
matches [re.match (pattern, content, re.DOTALL MULLILILLINE) for content
completion_contentee
5
original_rewards = [1.0 if match else 0.0 for match in matches]
ReDit add
noisy_rewards inoisy_ rewards Ir + random, gauss (0, 1.0 m (3 ** 0.5)) for in original rewards]
random. uni form 1.0, 1.0) for r
9
return noisy rewards
28

---

GSMSK Reasoning Format Reward Function
def
(compleeionn.oooo...c.cco **kwargs) -> [ist
float]:
def count_xin xm1 (text) -> float:
count - 0.0
if count (* ("<reasoning>>nn -- 1:
count 0. 125
if text.count (MAns/reaaccinng/nn)
count += 0.125
1f count 0.125
("\n<answer>>\n") = 1:
#count len (text (tt..tttplltttt/a//aaoeeeeeeee
[-1]) +0.001
if toxt cout. (*\ns/anewer>"" = 1:
count += 0.125
count - (len (teet..pple ("An</answer>>) [-1]) 10.0001
return count contents [completion [0] ("content") for completion in
completions]
reiginal_rewaaee = Icount_xml. (c) for in contents]
Rebit add
9
noisy_rewards [r random. uniform (-m 0.5, m A 0.5) for x in
original_rewarde inoisy_rewards [r landdm..guusse (0, 0.5 m (3 ** 0.5)) for in return original_rewards
As shown in the above code block, ReDit does not need to be modified in complex way, only the reward function needs to be modified, and any method can be easily integrated. The reward functions of other datasets can be found in the code.
D.3 Specific experimental parameters
In this section, we present the experimental parameters, including LoRA parameters, GRPO and other baseline experimental parameters.
Table 4: LoRA Parameters
LoRA Target
LoRA Rank
LoRA Alpha
LoRA Dropout
q Proj
64
0.05
Table 5: GRPO Parameters
Learning Rate
Num Generations
Epochs
5e-6
4
10
Table 6: DAPO Parameters
Clip Ratio Low
Clip Ratio Low
Clip Ratio C
Num Generations Max
0.2
0.28
10.0
10
29

---

APPS train, APPS test (pase
APPS train, Humanfivll test (passa1)
CodnCentaats train, CodeCondasts test (pass))
0.75
16.0
T44180 3005
Teeein Steps
Toar Steps
-E3-
Figure 15: Performance comparison on three code generation benchmarks: (left) APPS test, (center) HumanEval test, and (right) CodeContests test. The pass@1 accuracy is reported across training steps. Both ReDit variants (Gauss 0.05 and Uniform 0.05) consistently and significantly outperform the GRPO baseline, confirming the general applicability of our method to the coding domain.
E
Additional experiments
E.1
Results on the Code Generation Datasets
To validate the general applicability of our method (ReDit) beyond mathematical reasoning, we conducted a comprehensive evaluation on the domain of code generation. We performed experiments on three widely-used coding benchmarks: APPS, HumanEval, and CodeContests. This evaluation was designed to test the hypothesis that the core benefit of ReDit-stabilizing the learning signal to improve optimization is a general principle that is not limited to single domain.
The results are presented in [S] The plots consistently demonstrate that both the Gaussian (Gauss 0.05) and Uniform (Uniform 0.05) variants of ReDit significantly and consistently outperform the GRPO baseline across all three coding benchmarks. On datasets, our method not only achieves higher final pass&1 accuracy but also exhibits faster convergence rate. This strongly suggests that ReDit provides a more robust optimization pathway, and its benefits generalize effectively to complex tasks such code generation.
E.2 Results on Full Parameter Fine-Tuning
To confirm that the benefits of ReDit are not limited to parameter-efficient fine-tuning (PEFT) methods like LoRA, we conducted additional experiments using full parameter fine-tuning approach. This evaluation addresses whether ReDit's s effectiveness is general property of the optimization process itself, rather than an artifact of a specific tuning method [56].
For these experiments, we utilized a setup that differs from our primary PEFT experiments; specifi- cally, we employed the VERL framework for training with 8 GPUs. We evaluated this full fine-tuning setup on our three mathematical reasoning benchmarks: GSM8K, MATH, and Geo3k.
The results are presented in 16 The plots clearly demonstrate that ReDit (both Gauss 0.05 and Uniform 0.05 variants) consistently outperforms the GRPO baseline across all three henchmarks in this demanding full-tuning setting. The performance gap particularly notable the MATH and Geo3k datasets, where the GRPO baseline shows signs of stagnation, while our method continues to improve. These findings confirm that ReDit is robust and general- pprrpoee technique, delivering consistent performance gains in both parameter-efficient and full fine-tuning paradigms.
E.3 Results on DeepSeek Distillation Models
potential concern regarding our primary results is that they are predominantly focused on the Qwen2.5 model family. To further demonstrate the robustness and architectural generalizability of ReDit, we conducted additional experiments to validate its effectiveness on specialized reasoning models.
Specifically, we evaluated our method on two expert distillation models: DeepSeek-R--Distie
Llama-8B and ReepSeek-RI--Dstill---eeee We used the GSM8K benchmark compare their
30

---

GSMEK, Full Fine-Tuning
MATH, Full Fine- Tuniln
GRo3k, Full Fine-Tuiing
66.0
800
00.8
Taaiigg Staps
Tealing St6s
Taliig Stps
Figure 16: Performance comparison on mathematical reasoning benchmarks using full parameter fine-tuning. The plots show test accuracy across training steps for (left) GSM8K, (center) MATH, and (right) Geo3k. In this full-tuning setting, both ReDit variants (Gauss 0.05 and Uniform 0.05) consistently achieve higher test accuracy than the GRPO baseline, confirming that our method's benefits generalize beyond parameter-efficiee tuning (PEFT).
DeepSeek-R1 SeepSeek-RR-Distlll RiDa--s-ll--LLLaa--B. L8, GSMBK
DeepSeek-RR -iieeeek-RR-Dssll----eeeee GSMBK
88.0
95.0
36.5
93.5
Test Accuracy
85.9
92.9
1099
2000
3000
4040
$000
eate
7000
6000
8000
1800
2000
3000
3000
7000
2009
9000
Training Stops
Training Steps
Uniform 0.05
Figure 17: Evaluating ReDit's generalizability on specialized reasoning models. The plots show and (right) These results confirm that ReDit's performance advantage over the GRPO baseline holds across different model architectures, not just the Qwen models used in the main experiments.
Test Accuracy (%) on the GSM8K benchmark for (left) Soassee-RR--issill---eeeee
mathematical reasoning performance against the GRPO baseline. The results are presented in 17 The plots clearly show that both the Gaussian (Gauss 0.05) and Uniform (Uniform 0.05) variants of ReDit consistently outperform the GRPO baseline on both DeepSeek models.
This finding is significant it confirms that ReDit's ability to stabilize the training signal and improve performance is general principle. is not limited to single model family but holds true across various model architectures, including those specifically optimized for reasoning tasks.
F
More result
In this section, we present detailed numerical results for all experiments.
F1
Main Result
In this section, we show the results in Figure 7 the performance of GRPO and GRPO+ReDit on different datasets.
Tables
9 show the comparison ReDit on different datasets. ReDit significantly improves the convergence speed of GRPO. At any same step, ReDit achieves better performance.
31

---

Table 7: Performance Comparison of Different Training Steps on the Math Dataset
Method \Step
1000
2000
3000
4000
5000
6000
7000
8000
9000
Instruct model
GRPO Uniform ReDit
47.86
49.46
47.18
47.28
47.26
47.57
47.63
47.89
48.01
50.02
50.23
50.34
50.78
50.96
51.27
51.37
51.37
51.96
Gauss ReDit
49.78
50.73
51.03
51.07
51.53
51.43
52.01
52.01
52.55
Table 8: Performance Comparison of Different Training Steps the GSM8K Dataset
Method \Step
1000
2000
3000
4000
5000
6000
7000
8000
9000
Instruct model
84.91
GRPO
85,70
86.01
86.47
86.73
87.13
87.78
88.52
88,73
89.07
Uniform ReDit
89.16
89.16
89.31
89.31
89.31
89,99
89.99
89.99
90.76
Gauss ReDit
89.02
89.37
89.61
89.54
89.54
89.54
89.61
89.61
90.46
F.2
Baseline Result
In this section, we present all numerical results in Fig. As shown in Table 10 demonstrate the effect using ReDit on GSM8K based on the GRPO improvement method. The experimental results show that ReDit can also improve the convergence speed and performance these algorithms.
F,3 Variance Result
In this section, we show more results on the performance ReDit as the perturbation changes. As shown in Figure [8] the variance of uniform perturbation is similar to the variance of Gaussian perturbation, and the appropriate variance can achieve the best performance. The specific numerical results shown in Tables 11 and 12
Performance Comparison of Different Uniform Variance
Peak Accuracy 90.76% (a-0.05)
Fastest Convergence 89.16% (a=0.05)
Test Accuracy (%)
0.00
000.5
0.01
0-
0.02
0.5
84
1000
2000
3000
4000
5000
6000
7000
s000
9000
Training Steps
Figure 18: ReDit uniform perturbation performance changes with variance.
F,4 Scheduled Perturbation Result
In this section, we show the changing trends different scheduled perturbation strategies, as shown in Figure We took the perturbation of Gauss distribution an example and conducted experiments. The experimental results are shown in Table [3 The CosineReverse strategy shows the best performance.
32

---

Table 9: Performance Comparison of Different Training Steps on the Geometry3K Dataset
Method iStep
1000
2000
3000
4000
5000
6000
7000
8000
9000
Instruct model
40.43
GRPO
40.60
42.93
38.77
39.77
38.94
39.10
40.10
41.36
43.10
Uniform ReDit
43.37
43.89
44.01
44.23
44.23
44.23
44.12
44.36
44.36
Gauss ReDit
43.67
43.98
44.03
44.25
44.25
44.25
44.25
44.67
44.67
Table 10: Performance Comparison at Different Training Steps on Different Baseline
Method SStep
1000
2000
3000
4000
5000
6000
7000
8000
9000
DAPO
84.99
86.20
86.35
86.35
86.75
87.04
87.12
87.17
87.52
Uniform ReDit
87.03
87.15
87.26
87.54
87.54
87.69
87.83
88.03
88.57
Gauss ReDit
87.76
87.96
88.01
88.01
88.10
88.37
88.67
88.96
89.34
DR.GRPO
84.69
84.23
84.53
84.91
85.67
85.67
85.67
85,90
86.13
Uniform ReDit
86.27
86.36
86.45
86.54
86.75
87.03
87.26
87.16
87.34
Gauss ReDit
86.47
86.23
87.10
87.16
87.56
87.67
87.67
87.67
87.69
REINFORCE+
84.91
84.69
85.06
85.14
85.14
85.14
86.10
86.17
86.25
Uniform ReDit
86.21
86.11
86.67
86.31
86.75
87.01
87.26
87.59
87.59
Gauss ReDit
86.17
86.27
86.47
86.83
86.83
87.06
87.63
87.76
87.96
Table 11: Performance Comparison of Different variance on the Gauss Perturbation
Variance \Step
1000
2000
3000
4000
5000
6000
7000
8000
9000
0.01
85.97
87.01
87.40
87.54
87.92
88.76
88.84
89.54
89.54
0.02
86.40
87.70
88.16
89.23
89.39
90.22
90.14
90.14
90.14
0.05
89.02
89.37
89.61
89.54
89.54
89.54
89.61
89.61
90.46
0.1
87.64
89.08
89.69
89.84
90.07
89.84
89.84
89.84
90.07
0.3
87.87
88.48
88.78
88.93
89.39
89.39
89.39
89.46
89.46
0.5
86.81
87.57
87.41
87.64
87.64
87.95
88.32
88.48
88.95
Table 12: Performance Comparison of Different variance on the Uniform Perturbation
Variance \Step
1000
2000
3000
4000
5000
6000
7000
8000
9000
0.01
85.67
86.79
87.43
87.68
87.96
88.05
88.89
89.01
89.09
0.02
85.44
86.52
87.20
88.03
88.32
88.46
88.99
89.31
89.53
0.05
89.16
89.16
89.31
89.31
89.31
89,99
89,99
89,99
90.76
0.1
88.17
88.25
89.01
89.01
89.84
89.54
89.54
89.61
89.61
0.3
87.49
88.25
88.25
88.02
88.17
87.95
88.93
88.70
88.78
0.5
86.73
87.72
87.64
87.64
87.79
88.48
87.87
88.02
87.87
Table 13: Performance Comparison of Different Scheduled Perturbation Methods
Method \Step
1000
2000
3000
4000
5000
6000
7000
8000
9000
SquareRoot
88.10
89.31
88.93
89.69
89.46
89.46
89.46
89.46
90.22
RooteeRoo
88.55
89.54
89.46
90.07
90.07
89.31
89.61
89.54
89.69
Factor
88.25
88.63
89,69
89.46
89,23
89.54
89.46
89.31
89.69
FactorReverse
88.48
88.32
89.39
88.78
88.93
89.54
89.61
89.76
89,46
MutilFactor
87.87
89.31
89.01
89.01
89.01
89.61
89.16
89.61
89,46
MutiiFactooRevvrre
88.17
88.78
88.86
89.01
88.93
88.93
89.39
89.16
89.54
Cosine
88.32
88.32
89,39
89,84
89,76
89,61
90.14
90.46
90.23
CosineReverse
89.08
87.95
89.54
89.08
89.16
90.37
90.07
90.84
91.84
33

---

Nusde
FacloeNokee
FadefRovun
400
400
Mucill NrtoeNoiee
CunelPactarfeevmeeeiii
CostneSchedtier
Costnell cherechhher
0.2
0.4
Tiae Stys
Time Stgg
8p--k
8pm-ks
Figure 19: ReDit scheduled perturbation Variance trend with training step (taking the original variance as 0.05 as an example)
34


Reviews:
[{"review_id": "nQGQyrxSvV", "reviewer": "Reviewer_TpNZ", "summary": "This paper makes an interesting discovery that by adding a noise to the discrete 0-1 reward, the performance of the RL can be improved. Specifically, the authors of this paper credit the improvement to the smoothen reward landscape, which in turn resolves the gradient vanishing and gradient explosion issues. Experiments on both LLMs and VLMs with different RL algorithms, such as DAPO, GRPO, and REINFORCE++. This paper also provides theory analysis on how the added noise can help to accelerate the RL training process.", "questions": "1. The proposed method is somewhat similar to Dropout, Data augmentation, and Stochastic Gradient Descent. All these methods introduce noise to the training process and improve the performance. Could you discuss the differences and similarities between the proposed method and these methods?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The proposed method is simple and effective. For RL training, by adding a noise to the discrete 0-1 reward is almost like a free lunch. This method is orthogonal to modalities and different RL algorithms, which means it can be easily adopted to existing RL frameworks.\n\n2. This method has solid theoretical analysis, including the motivation from the perspective of grad_norm and the convergence analysis. \n\n## Weaknesses\n\n1. The proposed method is somewhat similar to several other regularization methods, such as Dropout, Data augmentation, and Stochastic Gradient Descent. It would be better if the authors could discuss the differences and similarities between the proposed method and these methods.\n\n2. Regarding the experiments. I would suggest the authors to evaluate the model on more widely used benchmarks, such as AIME-24 for LLMs.", "was_revised": true, "final_justification": "I decide to keep my score", "rebuttal": "We thank Reviewer TpNZ for their positive assessment of our work. We appreciate their recognition of the method's key advantages, including its simplicity, effectiveness, broad applicability, and the rigorous theoretical analysis provided. We address your concerns below.\n\n---\n\n# Responses to Weaknesses 1 and Questions 1\n\n\nWe thank the reviewer for this insightful comment and the opportunity to clarify the relationship between our proposed method, ReDit, and other established techniques.\n\nWe agree that ReDit shares a conceptual similarity with methods like Dropout, Data Augmentation, and SGD, as they all leverage randomness to improve the training process. However, the key difference and innovation of ReDit lie in what is being perturbed, how, and for what purpose. While the others are general-purpose techniques, ReDit is specifically designed to address challenges in reward-based optimization. Here is a detailed comparison:\n\n## ReDit\nReDit injects noise directly into the reward signal just before gradient computation. The primary objective is to smooth discrete rewards into a continuous (as illustrated in our Figure 2). This targeted intervention is designed to stabilize the gradient, mitigate the vanishing/exploding gradient problems common in settings with sparse rewards, and accelerate convergence (as illustrated in our Figure 1).\n\n## Comparison with Other Methods\n**Dropout**: This method applies noise by randomly setting a fraction of the model's internal activations to zero during training. Its main goal is regularization—preventing complex co-adaptations between neurons to reduce overfitting. In contrast, ReDit perturbs the external learning signal (the reward), not the model's internal state, to directly improve the quality of the gradient for optimization.\n\n\n**Data Augmentation**: This technique introduces randomness by applying transformations to the input data (e.g., rotating an image). This encourages the model to learn invariant features, thereby improving its ability to generalize. While Data Augmentation focuses on expanding the input space to enhance generalization, ReDit operates on the output/reward space to improve the stability and efficiency of the optimization process itself.\n\n**Stochastic Gradient Descent (SGD)**: The randomness in SGD comes from estimating the gradient using mini-batches of data. However, this process can fail when rewards are sparse and discrete. For example, if all samples in a mini-batch yield the same reward (e.g., all zeros), the variance of the rewards is zero, causing the gradient to vanish and halting the learning process. ReDit directly addresses this failure mode. By injecting controlled noise, ReDit ensures that the reward signal within a batch has a non-zero variance, thus always generating a usable and informative learning signal for the model to update its parameters.\n\nIn summary, while all these methods utilize stochasticity, ReDit is fundamentally distinct in its target of perturbation, its mechanism of action, and its underlying objective.\n\n# Responses to Weaknesses 2\n\nWe thank the reviewers for their valuable suggestions. We trained the model using the MATH training set, tested it on AIME24, and reported the results. The results are as follows:\n\n| method/ acc  | 0    | 1000 | 2000  | 3000  | 4000  | 5000  | 6000  | 7000  | 8000  | 9000  |\n|--------------|------|------|-------|-------|-------|-------|-------|-------|-------|-------|\n| GRPO         | 3.33 | 6.67 | 6.67  | 6.67  | 6.67  | 10    | 10    | 10    | 10    | 13.33 |\n| Gauss 0.05   | 3.33 | 10   | 10    | 13.33 | 13.33 | 16.67 | 16.67 | 16.67 | 16.67 | 20    |\n| Uniform 0.05 | 3.33 | 10   | 13.33 | 13.33 | 13.33 | 13.33 | 16.67 | 16.67 | 20    | 20    |\n\nAs shown in the table, our proposed methods (`Gauss 0.05` and `Uniform 0.05`) consistently outperform the GRPO baseline throughout the training process. They achieve a significant improvement in training efficiency and performance. In addition, Redit performs better on out-of-domain datasets than on in-domain data because Redit introduces randomness in the reward, which may lead to the model generalizing to unknown domains.\n\n---\nWe would like to thank Reviewer TpNZ for your positive feedback and recognition again! We sincerely hope our response has addressed your remaining concerns and can increase your opinions of our work. We will also include this discussion in our revised paper. If you have any other questions, we would like to address.\n\nSincerely, Authors"}, {"review_id": "Wx2kHw2Yw3", "reviewer": "Reviewer_HRjb", "summary": "This paper proposes Reward Dithering (ReDit), a method that can mitigate the limitations (e.g., gradient anomaly, unstable optimization, and slow convergence) of the discrete reward of Group Relative Policy Optimization (GRPO) by introducing zero-mean random noise to the discrete reward. This paper evaluates ReDit on mathematical reasoning tasks such as GSM8K, MATH, and Geometry3K. Experiment results show that ReDit improves the accuracy, compared to GRPO. Also, this paper provides theoretical analysis which proves that ReDit produces an unbiased estimate of the original gradient, introduces beneficial gradient variance, and improves convergence speed.", "questions": "- Q1. Figure 5 and 6 shows multiple lines, but does not provide a legend. Please provide a proper legend on the figures. I assume that the orange line indicates ReDit with Gaussian noise, the blue line represents ReDit with uniform noise, and the red line denotes the baseline.\n- Q2. The authors evaluate ReDit only on mathematical reasoning tasks such as GSM8K, MATH, Geometry3K. Can does ReDit improve the accuracy on coding tasks such as HumanEval, APPS and CodeContests?", "limitations": "The authors provide some limitations in Section 5 (i.e., Limitations and Conclusions).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Some strengths of this paper can be summarized as follows:\n- S1. First of all, this paper is well written.\n- S2. The main idea of adding random noise to the discrete reward seems simple and effective. Especially, ReDit seems to improve the optimization stability and convergence speed.\n- S3. This paper provides mathematical analysis that proves the effectiveness of ReDit.\n\nSome weaknesses of this paper can be summarized as follows:\n- W1. One of main concerns of this paper is that actual performance improvements such as accuracy do not seem significant. For example, the improvement of ReDit compared to GRPO is only 1.69 (i.e., from 89.07 to 90.76) on GSM8K. On MATH, it is about 4.54 (i.e., from 48.01 to 52.55).", "was_revised": true, "final_justification": "I maintain my initial rating, 4: Borderline accept. The main idea of ReDit is to add random noise to the discrete reward. ReDit seems to improve optimization stability and convergence speed. In my initial review, I pointed out one weak point and raised two questions: W1. actual performance improvement, Q1. missing legend in some figures, and Q2. performance on coding tasks. The authors provided thoughtful responses to my comments and questions. The authors showed that ReDit works well on coding tasks by providing additional experiments. However, the performance improvements on downstream tasks do not seem significant.", "rebuttal": "We thank the reviewers for their valuable feedback and appreciate their appreciation for the well-written work, the concise and effective methods, and the mathematical analyses that support the findings. We address your concerns below.\n\n---\n\n# Responses to W1\nWe thank the reviewer for their question, as it allows us to clarify the principal objective and contribution of ReDit. Our method is primarily designed to address a fundamental optimization challenge: the gradient anomalies (e.g., vanishing/explosion) that impede training stability and efficiency in GRPO. By resolving these issues, ReDit yields significant gains in training efficiency, often achieving comparable or superior results with as little as one-tenth of the training steps required by the baseline. A key aspect of our contribution is that this robust solution is achieved with remarkable simplicity, typically requiring only a single line of code. Therefore, the final accuracy improvement, while a valuable and positive result, should be interpreted as a secondary consequence of this stabilized process, with the core innovation being the efficient resolution of the underlying training instability itself.\n\n# Responses to Q1\nWe sincerely apologize for the missing legends in Figures 5 and 6 and thank the reviewer for pointing out this oversight. The legends were inadvertently cropped during the figure generation process, and we are sorry for any confusion this may have caused. Your assumption is largely correct. To clarify, the markers in the figures correspond to the following methods:\n\n* **Red Circles**: Baseline (GRPO)\n* **Yellow/Orange Squares**: ReDit with Gaussian noise\n* **Blue Triangles**: ReDit with Uniform noise\n\nWe will ensure that both Figures 5 and 6 are updated with clear and complete legends in the final version of the paper.\n\n# Responses to Q2\n\nThank you for this excellent question regarding the applicability of ReDit to domains beyond mathematical reasoning. This is a crucial point, and we are pleased to confirm that ReDit's benefits generalize effectively to coding tasks.\n\nTo validate this, we conducted a comprehensive set of experiments on three widely-used coding benchmarks: **APPS**, **HumanEval**, and **CodeContests**. The results consistently show that ReDit improves performance over the GRPO baseline.\n\nBelow are the detailed results for each benchmark.\n\n---\n\n## **Setting: APPS train data, APPS test data (pass@1)**\n*(Note: For APPS, which has multiple difficulty levels, we report the pass@1 accuracy across all problems)*\n\n| method/acc   | 0       | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| GRPO         | 31.42   | 32.47   | 33.16   | 33.16   | 33.63   | 34.65   | 34.65   | 34.78   | 34.78   | 34.78   |\n| Gauss 0.05   | 31.42   | 34.69   | 34.71   | 35.13   | 35.63   | 36.27   | 36.27   | 37.71   | 37.71   | 37.75   |\n| Uniform 0.05 | 31.42   | 34.61   | 35.01   | 35.01   | 35.51   | 36.21   | 36.21   | 36.91   | 37.21   | 37.45   |\n\n---\n\n## **Setting: APPS train data, HumanEval test data (pass@1)**\n\n| method/acc   | 0       | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| GRPO         | 47.13   | 47.83   | 47.96   | 48.36   | 48.36   | 48.91   | 49.67   | 49.99   | 49.99   | 50.07   |\n| Gauss 0.05   | 47.13   | 50.21   | 50.45   | 50.45   | 50.69   | 51.01   | 51.45   | 51.66   | 51.66   | 51.66   |\n| Uniform 0.05 | 47.13   | 49.96   | 50.01   | 50.13   | 50.13   | 50.79   | 51.03   | 51.36   | 51.36   | 51.36   |\n\n---\n\n## **Setting: CodeContests train data, CodeContests test data (pass@1)**\n\n| method/acc   | 0       | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| GRPO         | 15.48   | 16.75   | 16.78   | 17.31   | 17.31   | 18.26   | 19.07   | 19.31   | 19.31   | 19.31   |\n| Gauss 0.05   | 15.48   | 19.25   | 19.57   | 19.57   | 20.03   | 20.14   | 20.75   | 20.75   | 20.75   | 20.75   |\n| Uniform 0.05 | 15.48   | 19.13   | 19.26   | 20.05   | 20.05   | 20.05   | 20.46   | 20.63   | 20.63   | 20.63   |\n\nAs these results demonstrate, both Gaussian and Uniform variants of ReDit consistently and significantly outperform the GRPO baseline across all three coding benchmarks. This suggests that the core benefit of ReDit—stabilizing the learning signal to improve optimization—is a general principle that is not limited to a single domain and applies effectively to code generation tasks as well.\n\n---\nWe would like to thank Reviewer HRjb for your positive feedback and recognition again! We sincerely hope our response has addressed your remaining concerns and can increase your opinions of our work. We will also include this discussion in our revised paper. If you have any other questions, we would like to address.\n\nSincerely, Authors"}, {"review_id": "cXUmTpLrzv", "reviewer": "Reviewer_puQV", "summary": "The paper addresses the challenges of optimizing large language models (LLMs) under discrete reward signals, which often lead to gradient instability, slow convergence, and inefficient exploration. The authors propose ReDit, a simple yet effective method that introduces zero-mean random noise (e.g., Gaussian or uniform perturbations) to discrete rewards, thereby smoothing the reward landscape. This approach mitigates gradient vanishing and explosion by providing continuous exploratory signals, accelerating convergence, and improving final model performance. Theoretical analysis confirms that ReDit preserves unbiased gradient estimates while introducing beneficial variance. Empirical results across tasks like GSM8K, MATH, and Geometry3K demonstrate that ReDit achieves comparable performance to vanilla GRPO with only ~10% of the training steps and a 4% improvement when trained equally. The method also generalizes to other RL algorithms (e.g., DAPO, Dr.GRPO) and exhibits robustness across perturbation schedules.", "questions": "#### Questions\n\nBesides the weaknesses above, further questions are as follows:\n\n- Since DAPO and VAPO use several tricks to improve the RL training process, do these two methods have gradient exploding cases like the results in Figure 3?\n- L222 - L224: The authors claim that they want to isolate the effects of SFT. However, the authors uses two instruction models.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "#### Strengths\n\n- The idea is interesting.\n- The paper provides theoretical analysis of the proposed method.\n- The paper is easy to follow.\n\n#### Weaknesses\n\n- The paper only evaluates ReDit using LoRA. However, it is not clear the performance of ReDit with full fine-tuning.\n- The paper does not evaluate ReDit on more challenging tasks, such as AIME24.\n- The paper does not use reasoning models for experiments, such as DeepSeek-R1-Distill-7B. Therefore, the paper does not show the effectiveness of ReDit on reasoning tasks.", "was_revised": true, "final_justification": "All my concerns are addressed.", "rebuttal": "We thank the reviewers for their valuable feedback. We are pleased that they appreciated our engaging concepts, clear and accessible writing, and solid theoretical analysis. We address your concerns below.\n\n\n## Responses to Weaknesses 1\nThank you for raising this important point. To address the question of whether ReDit's benefits extend beyond parameter-efficient tuning, we have conducted additional experiments using a full fine-tuning approach. We can confirm that ReDit's effectiveness is not limited to LoRA and that it provides consistent performance gains in the full fine-tuning setting as well.\n\nFor these experiments, we used a setup that differs from the primary one in our main paper. Specifically, we utilized the **VERL framework** for training with **8 GPUs**. The detailed results of this full fine-tuning evaluation on our three mathematical reasoning benchmarks are presented below.\n\n---\n\n### **Setting: GSM8K, Qwen2.5-7B-Instruct, Full Fine-Tuning**\n\n| method/acc   | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| GRPO         | 86.72   | 87.03   | 87.48   | 88.71   | 88.16   | 88.73   | 89.56   | 89.67   | 90.53   |\n| Gauss 0.05   | 89.74   | 89.87   | 89.87   | 90.01   | 90.02   | 90.13   | 90.57   | 90.67   | 91.01   |\n| Uniform 0.05 | 89.71   | 89.91   | 89.91   | 90.03   | 90.09   | 90.24   | 90.51   | 90.69   | 91.09   |\n\n---\n\n### **Setting: MATH, Qwen2.5-7B-Instruct, Full Fine-Tuning**\n\n| method/acc   | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| GRPO         | 48.93   | 50.63   | 51.13   | 51.26   | 51.37   | 51.52   | 51.73   | 51.73   | 51.73   |\n| Gauss 0.05   | 51.03   | 51.13   | 51.54   | 52.01   | 52.37   | 52.91   | 53.02   | 53.57   | 53.96   |\n| Uniform 0.05 | 51.10   | 51.31   | 51.31   | 52.02   | 52.35   | 52.57   | 53.00   | 53.13   | 53.87   |\n\n---\n\n### **Setting: Geo3k, Qwen2.5-VL-7B-Instruct, Full Fine-Tuning**\n\n| method/acc   | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| GRPO         | 41.63   | 42.23   | 42.23   | 42.23   | 42.81   | 43.01   | 43.07   | 43.57   | 43.57   |\n| Gauss 0.05   | 43.02   | 43.54   | 43.96   | 43.96   | 44.02   | 44.02   | 44.51   | 44.32   | 44.12   |\n| Uniform 0.05 | 42.91   | 43.02   | 43.02   | 43.02   | 43.72   | 44.01   | 44.32   | 44.32   | 44.47   |\n\nAs the results clearly demonstrate, ReDit consistently outperforms the GRPO baseline across all three benchmarks in the full fine-tuning setting. These findings confirm that ReDit is a robust and general-purpose technique, delivering benefits for both LoRA and full fine-tuning paradigms.\n\n\n## Responses to Weaknesses 2\n\nWe thank the reviewers for their valuable suggestions. We trained the model using the MATH training set, tested it on AIME24, and reported the results. The results are as follows:\n\n| method/ acc  | 0    | 1000 | 2000  | 3000  | 4000  | 5000  | 6000  | 7000  | 8000  | 9000  |\n|--------------|------|------|-------|-------|-------|-------|-------|-------|-------|-------|\n| GRPO         | 3.33 | 6.67 | 6.67  | 6.67  | 6.67  | 10    | 10    | 10    | 10    | 13.33 |\n| Gauss 0.05   | 3.33 | 10   | 10    | 13.33 | 13.33 | 16.67 | 16.67 | 16.67 | 16.67 | 20    |\n| Uniform 0.05 | 3.33 | 10   | 13.33 | 13.33 | 13.33 | 13.33 | 16.67 | 16.67 | 20    | 20    |\n\nAs shown in the table, our proposed methods (`Gauss 0.05` and `Uniform 0.05`) consistently outperform the GRPO baseline throughout the training process. They achieve a significant improvement in training efficiency and performance. In addition, Redit performs better on out-of-domain datasets than on in-domain data because Redit introduces randomness in the reward, which may lead to the model generalizing to unknown domains.\n\n\n## Responses to Weaknesses 3\n\nThank you for this suggestion regarding the choice of models for our experiments. We would like to address this in two parts.\n\nFirst, we wish to clarify that all experiments presented in our main manuscript are indeed focused on **reasoning tasks**. Although our primary model is based on Qwen2.5, our methodology guides the model to first generate a reasoning path (e.g., chain-of-thought) before outputting the final answer. Therefore, the evaluation metrics directly reflect the model's reasoning capabilities on these tasks.\n\nSecond, to further address your concern and demonstrate the effectiveness of ReDit on specialized reasoning models, we have conducted new experiments on **DeepSeek-R1-Distill-Llama-8B** and **DeepSeek-R1-Distill-Qwen-7B**. The results on the GSM8K benchmark are presented below.\n\n---\n### **Setting: DeepSeek-R1-Distill-Llama-8B, GSM8K**\n\n| method/ acc  | 0       | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| GRPO         | 82.11   | 83.13   | 84.01   | 85.16   | 85.16   | 86.27   | 86.27   | 86.57   | 86.96   | 87.03   |\n| Gauss 0.05   | 82.11   | 86.73   | 86.39   | 87.13   | 87.13   | 87.13   | 87.78   | 87.78   | 87.78   | 87.78   |\n| Uniform 0.05 | 82.11   | 86.43   | 86.57   | 87.21   | 87.54   | 87.54   | 87.54   | 87.54   | 87.54   | 87.54   |\n\n---\n### **Setting: DeepSeek-R1-Distill-Qwen-7B, GSM8K**\n\n| method/ acc  | 0       | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| GRPO         | 89.14   | 89.67   | 89.67   | 89.67   | 90.36   | 91.57   | 92.74   | 93.15   | 93.15   | 93.46   |\n| Gauss 0.05   | 89.14   | 92.17   | 92.56   | 93.49   | 93.49   | 93.57   | 93.74   | 94.01   | 94.01   | 94.01   |\n| Uniform 0.05 | 89.14   | 92.57   | 92.57   | 93.13   | 93.42   | 93.42   | 93.75   | 93.99   | 93.99   | 93.99   |\n\nThese new results clearly show that ReDit continues to outperform the GRPO baseline on DeepSeek distillation models. This confirms that ReDit's ability to stabilize training and improve performance is a general principle that holds true across various model architectures, including those specifically optimized for reasoning.\n\n## Responses to Questions 1\nThank you for this insightful question regarding the gradient behavior of other advanced RL algorithms like DAPO and VAPO. This is an important point, and we provide the following analysis based on their methodologies and our own observations.\n\n* **Regarding DAPO:** DAPO is an improvement upon GRPO that introduces Dynamic Sampling to increase the number of effective samples used during training. This technique is effective at mitigating the **gradient vanishing** problem. However, it does not have a specific mechanism to control **gradient explosion**. In our own experiments, while running DAPO, we still observed instances of both gradient vanishing and gradient explosion, similar to the phenomena shown for GRPO in Figure 3. This suggests that while DAPO improves one aspect of gradient stability, it does not fully resolve the anomaly, leaving a clear need for methods like ReDit.\n\n* **Regarding VAPO:** VAPO represents a different class of algorithm. It is a **value-model-based** method, which distinguishes it fundamentally from value-model-free methods like GRPO and DAPO. Due to this architectural difference, its gradient dynamics are likely to differ significantly from those we analyzed for GRPO. A direct comparison is further complicated by the fact that **VAPO is not yet open-source**. We are actively monitoring its status and plan to conduct a thorough analysis of its gradient behavior as soon as the code becomes publicly available.\n\nIn summary, existing methods like DAPO partially address but do not eliminate the gradient anomalies, while a direct, empirical comparison with VAPO is not currently feasible.\n\n\n## Responses to Questions 2\nThank you for pointing out this ambiguity in L222-L224. We sincerely apologize for the lack of clarity in our original wording.\n\nYou are correct that our base models are instruction-tuned. Our intention was to differentiate between two distinct stages of Supervised Fine-Tuning (SFT):\n\n1.  **General Instruction-Tuning:** The initial SFT performed on broad, instruction-following datasets to create the capable base instruction model. The models we use have undergone this stage [1].\n2.  **Task-Specific SFT:** A subsequent, optional SFT stage where a model is further fine-tuned on the training data of a specific downstream task (e.g., the GSM8K training set).\n\nWhen we claimed we wanted to \"isolate the effects of SFT,\" we were referring specifically to **Task-Specific SFT**. Our goal is to compare our RL-based approach directly against the common practice of further fine-tuning a general instruction model on a specific downstream dataset.\n\nWe understand how our original phrasing was confusing and we will revise L222-L224 in the final manuscript to make this crucial distinction explicit. Thank you again for helping us improve the clarity of the paper.\n\n---\n### Reference:\n[1] Qwen Team, Qwen2.5 Technical Report\n\n---\nWe would like to thank Reviewer puQV for your positive feedback and recognition again! We sincerely hope our response has addressed your remaining concerns and can increase your opinions of our work. We will also include this discussion in our revised paper. If you have any other questions, we would like to address.\n\nSincerely, Authors"}, {"review_id": "38QTlMnQZk", "reviewer": "Reviewer_UG9k", "summary": "Proposes a reward smoothing technique, called ReDit, for improving the optimization stability of policy gradient algorithms such as GRPO, in the context of training language models to solve tasks using a discrete reward (e.g., a reward of one if the answer is correct and zero otherwise). ReDit simply adds zero-mean noise, with a fixed standard deviation, to the rewards of outputs. The motivation behind ReDit is that sufficient reward variance has been shown in prior work to be important for efficient optimization.\n\nExperiments across datasets and policy gradient methods demonstrate that ReDit leads to improved performance and training stability. Theoretical insights are provided as to why the introduced noise accelerates optimization.", "questions": "--", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n---\n1. Reads relatively well and is easy to follow. Provides sufficient motivation for the proposed method, based both on an empirical analysis and existing theory on the connection between reward variance and policy gradient optimization.\n\n2. The proposed method is simple, yet demonstrates promising results, in which case I view the simplicity as a strength. Ablations over different noise schedules and standard deviations are also useful.\n\n\nWeaknesses\n---\n1. The empirical evaluation lacks error bars or a measure of statistical significance over random seeds (e.g., standard deviation). This makes it difficult to assess whether the improvement brought forth by ReDit is consistent and significant. I strongly suggest that the authors run additional seeds and include the results. This is particularly important with policy gradient methods, which are notorious for their instability (cf. [1]).\n\n&nbsp;\n\n2. The proof of Proposition 3 is incorrect, and I do not believe that Proposition 3 is true as currently stated.\n    - For the upper bound on $t_\\gamma$, the result in [37] specifies that there exist inaccurate reward models that lead to fast optimization, and not that for every inaccurate reward model this holds. Thus, it is not true that in general when applying noise to the reward $t_\\gamma$ will be bounded as specified in Proposition 3.\n    - For the lower bound on $t_\\gamma$, Theorem 1 in [37] considers a deterministic reward function and cannot be invoked as is. If the reward is perturbed with noise $\\epsilon$, then the expected objective becomes $J(\\pi_\\theta) = E_{q \\sim p_Q} [E_{o \\sim \\pi_\\theta (\\cdot | q)} [ E_\\epsilon [\\tilde{R} (q, o)] ]]$, where $\\tilde{R} (q, o) = R(q, o) + \\epsilon$. Following the same arguments in Theorem 1 of [37] would lead to an analogous lower bound, but that depends on $var_{o \\sim \\pi_\\theta (\\cdot | q)} ( E_{\\epsilon} [\\tilde{R} (q, o)] )$ as opposed to $var_{o \\sim \\pi_\\theta (\\cdot | q)} (R(q,o))$. Since the introduced noise $\\epsilon$ is zero-mean, we have that $E_{\\epsilon} [\\tilde{R} (q, o)] = R(q, o)$, and so $var_{o \\sim \\pi_\\theta (\\cdot | q)} ( E_{\\epsilon} [\\tilde{R} (q, o)] ) = var_{o \\sim \\pi_\\theta (\\cdot | q)} (R(q,o))$. That is, the lower bound will not depend on the noise $\\epsilon$.\n    \n    Note that the discrepancy above stems from the fact that the analysis of [37] considers optimizing the expected objective with exact gradients, as opposed to using batch-based estimates. The expected gradients are not modified by the zero-mean noise, while as Proposition 2 in this paper shows, the batch gradients are. In my view, a potential explanation for the benefit of ReDit is that, intuitively, it can help smooth out batch gradient estimates, especially when the expected gradient is small due to low reward variance, in which case the gradient estimates can be unstable.\n\n&nbsp;\n\n3. There are quite a few places where the clarity of the presented results can be improved. In particular, as detailed below in the additional comments part of the review, notation is used without proper introduction and there are details missing regarding what is plotted in the figures. Though, these issues should be amendable rather easily.\n\n&nbsp;\n\nOverall, I believe that the paper has potential due to its clarity and the promising empirical results. However, the lack of error bars or standard deviation across random seeds in the experiments (in particular, Tables 1 and 2 and Figures 5 and 6) and the incorrect Proposition 3 prevent me from recommending acceptance. If the authors are able to provide results with additional random seeds to solidify the significance of their approach and either correct or remove Proposition 3, I will gladly reassess and consider raising my rating.\n\n\n[1] Henderson, P., et al., Deep reinforcement learning that matters, AAAI 2018.\n\n\n\nAdditional comments\n---\n1. In lines 49 and 81, the citation to [32] seems incorrect as it does not discuss the connection between reward variance and optimization.\n\n2. Regarding Figure 1: \n    - Why do the rewards not start from similar values? Since it is the same dataset and base model I would expect them to start from roughly the same value.\n    - Is the reward reported for ReDit with or without the noise? For fair comparison, the results should also include the reward without noise (if this is not already the case), whether in the same plot or another.\n    - What is the scale of the rewards? Without further information, I would expect them to be 0 for incorrect and 1 for correct responses, but the rewards in the figure are between 2 and 5.\n\n3. In the red box of Theorem 2: What is a “hint”? Should that just be “prompt”?\n\n4. In the red box of Theorem 2: The text around what the result says is imprecise. From looking at the corresponding theorem in [37], it shows that there exists a perfectly accurate reward model and a relatively inaccurate model for which the bounds on $t_\\gamma$ hold.\n\n5. More broadly, regarding the red boxes of Theorems 1 and 2, the notation and which training algorithm is considered is not properly introduced. For example, what are $f_\\theta$, $S$, and $\\mathcal{X}$? It is possible to look at the original theorems from [37], but it is worth being self-contained. Alternatively, since these boxes refer to results from prior work, whose exact technical setting may not be critical for the current paper, it may be worth converting the boxes to being more informal, including the main takeaway while referring to the original theorems in [37] or an appendix for the full details.\n\n6. In line 167: $r_{RM}$ is not used elsewhere as far as I can see, so I would recommend dropping that unnecessary notation, especially given that the rewards in this paper are not given by a reward model.\n\n7. Typo in line 138: “Eplosion” -> “Explosion”.\n\n8. Typo in line 195: There is no line 8 in Algorithm 1. Perhaps this should have been line 5?\n\n9. Figure 6 is missing a legend. Currently, it is not clear which line corresponds to which method. The red line seems to correspond to GRPO, but what are the remaining two lines? Are they GRPO combined with ReDit for either Gaussian or uniform noise?\n\n10. Figures and tables do not appear in the order that they are referenced, which can be confusing. For example, Table 2 and Figure 6 are referenced before Table 1 and Figure 5.\n\n11. Typo in line 283: “We provides a theoretical analyzing how…” is not grammatically correct.\n\n12. Equation 5 is missing an expectation over the introduced noise $\\epsilon$, which does appear in the proof of Proposition 1.\n\n13. It is worth explicitly specifying what the variance of “Gradient Noise” refers to in Equation 7.\n\n14. The statements in lines 302 to 305 on why the additional gradient noise is beneficial should either be hedged or refer to evidence in the literature. Currently, they are phrased as concrete truths, while they seem to be more on the line of intuitive arguments.", "was_revised": true, "final_justification": "The initial review raised three main concerns: lack of error bars or a measure of statistical significance in the experiments, the proof of Proposition 3 was incorrect, and some clarity issues. During the discussion period the authors have addressed all three concerns by providing additional experiments, specifying how they would remove Proposition 3 and replace it with intuitive arguments based on other parts of their theoretical analysis (leaving full formalization to future work), and detailing changes to improve clarity. \n\nAs a result, I am happy to update my assessment and recommend acceptance. The proposed method is simple, yet demonstrates promising results, and so can be of potential interest to the community.", "rebuttal": "We sincerely thank the reviewers for their insightful feedback. We are grateful for their recognition of our approach's clarity and simplicity, its grounding in both empirical and theoretical motivations, and the utility of our ablation study. We address your concerns below.\n# Responses to Weaknesses 1\nThank you reviewer for this insightful and crucial suggestion. To address this concern, in addition to the initial experiments, we conducted extensive new experiments, running our approach (ReDit) and all baselines on 5 different random seeds (`42`, `123`, `888`, `2025`, `9999`), and `None` means no random seed is set. As you suggested, in the revised manuscript, all tables will be updated with the mean and variance, and the accuracy plots will be updated with error bars.  We alse show below detailed results for our `ReDit (Gauss 0.05)` approach on all seeds and baselines.\n\n## **Setting: gsm8k, gauss,0.05**\n\n| seed/acc     | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| None         | 89.02   | 89.37   | 89.61   | 89.54   | 89.54   | 89.54   | 89.61   | 89.61   | 90.46   |\n| 42           | 89.07   | 89.35   | 89.60   | 89.53   | 89.53   | 89.36   | 89.60   | 89.69   | 90.43   |\n| 123          | 89.01   | 89.36   | 89.63   | 89.51   | 89.51   | 89.47   | 89.67   | 89.64   | 90.48   |\n| 888          | 89.02   | 89.27   | 89.69   | 89.56   | 89.52   | 89.67   | 89.65   | 89.79   | 90.39   |\n| 2025         | 88.97   | 89.39   | 89.61   | 89.59   | 89.59   | 89.65   | 89.63   | 89.72   | 90.35   |\n| 9999         | 89.08   | 89.31   | 89.58   | 89.49   | 89.60   | 89.73   | 89.66   | 89.53   | 90.51   |\n| **mean** | **89.03** | **89.34** | **89.62** | **89.54** | **89.55** | **89.57** | **89.64** | **89.66** | **90.44** |\n| **std** | **0.04** | **0.04** | **0.04** | **0.03** | **0.04** | **0.13** | **0.03** | **0.09** | **0.06** |\n## **Setting: math, gauss,0.05**\n| seed/acc     | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| None         | 49.78   | 50.73   | 51.03   | 51.07   | 51.53   | 51.43   | 52.01   | 52.01   | 52.55   |\n| 42           | 49.73   | 50.71   | 51.04   | 51.09   | 51.53   | 51.42   | 52.03   | 52.03   | 52.59   |\n| 123          | 49.71   | 50.69   | 51.07   | 51.02   | 51.57   | 51.37   | 52.09   | 52.09   | 52.51   |\n| 888          | 49.76   | 50.67   | 51.13   | 51.03   | 51.51   | 51.25   | 52.00   | 52.05   | 52.53   |\n| 2025         | 49.73   | 50.74   | 50.96   | 50.99   | 51.46   | 51.39   | 52.08   | 51.95   | 52.56   |\n| 9999         | 49.70   | 50.79   | 50.93   | 51.02   | 51.37   | 51.47   | 52.06   | 52.06   | 52.58   |\n| **mean** | **49.74** | **50.72** | **51.03** | **51.04** | **51.50** | **51.39** | **52.05** | **52.03** | **52.55** |\n| **std** | **0.03** | **0.04** | **0.07** | **0.03** | **0.07** | **0.08** | **0.04** | **0.05** | **0.03** |\n## **Setting: Geo3k, gauss,0.05**\n| seed/acc     | 1000    | 2000    | 3000    | 4000    | 5000    | 6000    | 7000    | 8000    | 9000    |\n|--------------|---------|---------|---------|---------|---------|---------|---------|---------|---------|\n| None         | 43.67   | 43.98   | 44.03   | 44.25   | 44.25   | 44.25   | 44.25   | 44.67   | 44.67   |\n| 42           | 43.62   | 43.91   | 44.01   | 44.22   | 44.22   | 44.23   | 44.22   | 44.62   | 44.62   |\n| 123          | 43.68   | 43.95   | 44.05   | 44.24   | 44.24   | 44.22   | 44.27   | 44.68   | 44.68   |\n| 888          | 43.69   | 43.99   | 44.02   | 44.25   | 44.23   | 44.29   | 44.25   | 44.61   | 44.62   |\n| 2025         | 43.61   | 43.89   | 44.03   | 44.29   | 44.29   | 44.29   | 44.22   | 44.63   | 44.63   |\n| 9999         | 43.69   | 44.03   | 44.09   | 44.21   | 44.21   | 44.31   | 44.31   | 44.67   | 44.69   |\n| **mean** | **43.66** | **43.96** | **44.04** | **44.24** | **44.24** | **44.27** | **44.25** | **44.65** | **44.65** |\n| **std** | **0.04** | **0.05** | **0.03** | **0.03** | **0.03** | **0.04** | **0.03** | **0.03** | **0.03** |\n\nAs these tables show, the standard deviation across different runs is consistently very low (typically ≤ 0.1). This provides strong empirical evidence that ReDit's performance gains are stable, consistent, and not an artifact of a specific random initialization. These new results confirm that ReDit robustly outperforms the baselines.\n# Responses to Weaknesses 2\nWe sincerely thank the reviewer for this detailed and insightful critique of our theoretical analysis in Proposition 3. Your careful examination has helped us identify an error in our original proof and allows us to clarify our argument. We agree that our initial proposition had issues and will be revised. We address the upper and lower bounds separately below.\n## On the Upper Bound (We agree with the reviewer)\nRegarding the upper bound, you are entirely correct. We apologize for this incorrect derivation. We erroneously extrapolated the result from [37], which specifies that *there exist* certain inaccurate reward models that lead to faster optimization. We cannot prove that our specific noisy reward model, $r_{RM} = r + \\epsilon$, satisfies this property in the general case.\nAction: We will remove the claim regarding the upper bound from Proposition 3 in the final version of the manuscript.\n## On the Lower Bound (A clarification on the objective function)\nRegarding the lower bound, we would like to point out that we believe there is a misunderstanding in the understanding of the objective function in the theoretical framework of [37] and appreciate the opportunity to clarify.\nWe apologize for not emphasizing in the paper that in the theoretical analysis (whether Proposition 1, Proposition 2 or Proposition 3), we are consistent with the assumptions in [37] and all theoretical analysis considers using exact gradients to optimize the expected objective function. The impact of ReDit on the gradient is detailed in our Propositions 1 and 2. As shown above, ReDit does not change the expectation of the gradient (i.e., the estimator is unbiased), but it significantly affects the variance of the gradient, which is directly related to the noise variance $\\sigma^2$. Based on this theoretical setting, let us revisit the objective function being optimized. Following the approach of [37], the policy $\\pi_\\theta$ is updated to maximize the expected reward given the reward model $r_{RM}$:\n\n$\n\\tilde{J}(\\pi\\_\\theta) = \\mathbb{E}\\_{q \\sim p\\_Q} \\left[ \\mathbb{E}\\_{o \\sim \\pi\\_\\theta(\\cdot|q)}r\\_{RM}(q, o) \\right]\n$\n\nIn our approach, the reward model is the noisy reward, where $r_{RM} = r + \\epsilon$. This corresponds to $\\tilde{R}$ in Equation (5) of our paper.\nThe key lies in the application of Theorem 1 in [37]. Theorem 1 gives a lower bound on the optimization result of $\\tilde{J}(\\pi_\\theta)$ based on the properties of the reward model $r_{RM}$ and its relationship to the true reward $r$. In our application, $r_{RM}$ is the reward model used by Redit, and the deterministic 0-1 reward $r$ is its true reward. Therefore, the lower bound correctly depends on quantities derived from $r_{RM}$, such as its variance $\\text{Var}(r_{RM}) = \\text{Var}(r) + \\sigma^2$.\nTherefore, we maintain that the lower bound in Proposition 3 is valid under this interpretation, which is consistent with the framework of [37].\nFinally, thank you for your insightful observation: in practice, ReDit also smooths batch gradient estimates, thereby improving training efficiency. This is consistent with our empirical findings. \n### Reference:\n[37] Noam Razin et al.. What makes a reward model a good teacher? an optimization perspective.\n## Responses to Additional Comments\nWe sincerely thank the reviewer for their thorough and constructive feedback. We will address all points in the revised manuscript. We summarize our planned changes below:\n\n**1. Typos, Citations, and Minor Corrections (Comments 1, 5, 9, 10, 13):**\nThank you for catching these. We confirm these are errors and will make the following corrections:\n* The citation in L49/81 will be corrected to [37].\n* All typos will be fixed (e.g., L138 \"Explosion\", L195 reference to line 5, L283 grammar).\n\n**2. Presentation and Clarity Improvements (Comments 4, 7, 8, 11, 12, 15):**\nWe will improve the clarity of our presentation across the paper. This includes:\n* Updating figure captions and restoring legends (Figs 1, 5, 6) to clarify reward scales and method details.\n* Ensuring consistent notation throughout the paper (e.g., unifying $r_{RM}$ and $\\tilde{R}$).\n* Adding definitions to make all theorems self-contained (Sec 2.2.2).\n* Reordering all figures and tables to match their reference order.\n* Explicitly defining the \"Gradient Noise\" variance in Eq. 7 as the variance of the gradient's L2 norm.\n\n**3. Substantive Clarifications and Revisions (Comments 2, 3, 6, 14, 16):**\n* **Fig. 1 Rewards:** The initial rewards differ because the baseline uses the raw 0-1 signal, while ReDit uses the noisy signal ($r + \\epsilon$) for gradient updates. The plot shows this noisy reward; for comparison, a plot of the reward *without* noise will be added to the appendix.\n* **Theorem 2 Text:** We agree the description was imprecise and will revise it.\n* **Eq. 5 Expectation:**  We will update Equation 5 according to the formula provided in Section Responses to Weaknesses 2.\n* **Adding Supporting References:** We agree and will add supporting references for L302-305.\n---\nWe would like to thank Reviewer UG9k for your positive feedback and recognition again! We sincerely hope our response has addressed your remaining concerns and can increase your opinions of our work. We will also include this discussion in our revised paper. If you have any other questions, we would like to address.\n\nSincerely, Authors"}]

Author rebuttal:

Meta-review: - Scientific Claims: This paper proposes a method that adds zero-mean random noise to discrete binary rewards in LLM reinforcement learning to address gradient anomalies caused by non-smooth loss landscapes. The method theoretically maintains unbiased gradient estimates while introducing beneficial variance, and empirically achieves comparable performance.

- Strengths: The method's simplicity combined with broad applicability across algorithms, domains, and training paradigms represents significant practical value. Strong theoretical foundation with rigorous proofs of unbiasedness and beneficial variance, plus comprehensive experimental validation across mathematical reasoning, coding tasks, and specialized models with proper statistical rigor.

- Weaknesses: Absolute performance improvements are modest (1-4%), and theoretical completeness is limited after removing the contested Proposition 3 during rebuttal. The paper lacks systematic hyperparameter sensitivity analysis and could benefit from broader comparison with other reward smoothing techniques.

- Rebuttal and Discussions: Authors successfully addressed all major concerns through extensive additional experiments (>15 new settings) and collaborative theoretical problem-solving, transforming initially mixed reviews to unanimously positive (4,4,5,5). Critical issues of statistical validation and theoretical soundness were fully resolved, while experimental breadth and technical understanding were significantly enhanced through constructive reviewer engagement.

- Reasons for Accept: The work addresses a fundamental optimization problem in RLHF with a principled solution that targets root causes rather than symptoms, demonstrated through exceptional scientific rigor during the rebuttal process. The method's simplicity ensures broad adoptability while the 90% training step reduction offers immediate practical value, representing a paradigm shift from reactive optimization fixes to proactive landscape modification.
Decision: Accept (poster)