Title: ReDit: Reward Dithering for Improved LLM Policy Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Discrete Rewards

Reward Signal Perturbation

Gradient Optimization

Convergence Speed
Abstract: DeepSeek-R1 has successfully enhanced Large Language Model (LLM) reasoning capabilities through its rule-based reward system. While it's a ''perfect'' reward system that effectively mitigates reward hacking, such reward functions are often discrete. Our experimental observations suggest that discrete rewards can lead to gradient anomaly, unstable optimization, and slow convergence. To address this issue, we propose ReDit (Reward Dithering), a method that dithers the discrete reward signal by adding simple random noise. With this perturbed reward, exploratory gradients are continuously provided throughout the learning process, enabling smoother gradient updates and accelerating convergence. The injected noise also introduces stochasticity into flat reward regions, encouraging the model to explore novel policies and escape local optima. Experiments across diverse tasks demonstrate the effectiveness and efficiency of ReDit. On average, ReDit achieves performance comparable to vanilla GRPO with only approximately 10% the training steps, and furthermore, still exhibits a 4% performance improvement over vanilla GRPO when trained for a similar duration. Visualizations confirm significant mitigation of gradient issues with ReDit. Moreover, theoretical analyses are provided to further validate these advantages.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nProposes a reward smoothing technique, called ReDit, for improving the optimization stability of policy gradient algorithms such as GRPO, in the context of training language models to solve tasks using a discrete reward (e.g., a reward of one if the answer is correct and zero otherwise). ReDit simply adds zero-mean noise, with a fixed standard deviation, to the rewards of outputs. The motivation behind ReDit is that sufficient reward variance has been shown in prior work to be important for efficient optimization.\n\nExperiments across datasets and policy gradient methods demonstrate that ReDit leads to improved performance and training stability. Theoretical insights are provided as to why the introduced noise accelerates optimization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n---\n1. Reads relatively well and is easy to follow. Provides sufficient motivation for the proposed method, based both on an empirical analysis and existing theory on the connection between reward variance and policy gradient optimization.\n\n2. The proposed method is simple, yet demonstrates promising results, in which case I view the simplicity as a strength. Ablations over different noise schedules and standard deviations are also useful.\n\n\nWeaknesses\n---\n1. The empirical evaluation lacks error bars or a measure of statistical significance over random seeds (e.g., standard deviation). This makes it difficult to assess whether the improvement brought forth by ReDit is consistent and significant. I strongly suggest that the authors run additional seeds and include the results. This is particularly important with policy gradient methods, which are notorious for their instability (cf. [1]).\n\n&nbsp;\n\n2. The proof of Proposition 3 is incorrect, and I do not believe that Proposition 3 is true as currently stated.\n    - For the upper bound on $t_\\gamma$, the result in [37] specifies that there exist inaccurate reward models that lead to fast optimization, and not that for every inaccurate reward model this holds. Thus, it is not true that in general when applying noise to the reward $t_\\gamma$ will be bounded as specified in Proposition 3.\n    - For the lower bound on $t_\\gamma$, Theorem 1 in [37] considers a deterministic reward function and cannot be invoked as is. If the reward is perturbed with noise $\\epsilon$, then the expected objective becomes $J(\\pi_\\theta) = E_{q \\sim p_Q} [E_{o \\sim \\pi_\\theta (\\cdot | q)} [ E_\\epsilon [\\tilde{R} (q, o)] ]]$, where $\\tilde{R} (q, o) = R(q, o) + \\epsilon$. Following the same arguments in Theorem 1 of [37] would lead to an analogous lower bound, but that depends on $var_{o \\sim \\pi_\\theta (\\cdot | q)} ( E_{\\epsilon} [\\tilde{R} (q, o)] )$ as opposed to $var_{o \\sim \\pi_\\theta (\\cdot | q)} (R(q,o))$. Since the introduced noise $\\epsilon$ is zero-mean, we have that $E_{\\epsilon} [\\tilde{R} (q, o)] = R(q, o)$, and so $var_{o \\sim \\pi_\\theta (\\cdot | q)} ( E_{\\epsilon} [\\tilde{R} (q, o)] ) = var_{o \\sim \\pi_\\theta (\\cdot | q)} (R(q,o))$. That is, the lower bound will not depend on the noise $\\epsilon$.\n    \n    Note that the discrepancy above stems from the fact that the analysis of [37] considers optimizing the expected objective with exact gradients, as opposed to using batch-based estimates. The expected gradients are not modified by the zero-mean noise, while as Proposition 2 in this paper shows, the batch gradients are. In my view, a potential explanation for the benefit of ReDit is that, intuitively, it can help smooth out batch gradient estimates, especially when the expected gradient is small due to low reward variance, in which case the gradient estimates can be unstable.\n\n&nbsp;\n\n3. There are quite a few places where the clarity of the presented results can be improved. In particular, as detailed below in the additional comments part of the review, notation is used without proper introduction and there are details missing regarding what is plotted in the figures. Though, these issues should be amendable rather easily.\n\n&nbsp;\n\nOverall, I believe that the paper has potential due to its clarity and the promising empirical results. However, the lack of error bars or standard deviation across random seeds in the experiments (in particular, Tables 1 and 2 and Figures 5 and 6) and the incorrect Proposition 3 prevent me from recommending acceptance. If the authors are able to provide results with additional random seeds to solidify the significance of their approach and either correct or remove Proposition 3, I will gladly reassess and consider raising my rating.\n\n\n[1] Henderson, P., et al., Deep reinforcement learning that matters, AAAI 2018.\n\n\n\nAdditional comments\n---\n1. In lines 49 and 81, the citation to [32] seems incorrect as it does not discuss the connection between reward variance and optimization.\n\n2. Regarding Figure 1: \n    - Why do the rewards not start from similar values? Since it is the same dataset and base model I would expect them to start from roughly the same value.\n    - Is the reward reported for ReDit with or without the noise? For fair comparison, the results should also include the reward without noise (if this is not already the case), whether in the same plot or another.\n    - What is the scale of the rewards? Without further information, I would expect them to be 0 for incorrect and 1 for correct responses, but the rewards in the figure are between 2 and 5.\n\n3. In the red box of Theorem 2: What is a “hint”? Should that just be “prompt”?\n\n4. In the red box of Theorem 2: The text around what the result says is imprecise. From looking at the corresponding theorem in [37], it shows that there exists a perfectly accurate reward model and a relatively inaccurate model for which the bounds on $t_\\gamma$ hold.\n\n5. More broadly, regarding the red boxes of Theorems 1 and 2, the notation and which training algorithm is considered is not properly introduced. For example, what are $f_\\theta$, $S$, and $\\mathcal{X}$? It is possible to look at the original theorems from [37], but it is worth being self-contained. Alternatively, since these boxes refer to results from prior work, whose exact technical setting may not be critical for the current paper, it may be worth converting the boxes to being more informal, including the main takeaway while referring to the original theorems in [37] or an appendix for the full details.\n\n6. In line 167: $r_{RM}$ is not used elsewhere as far as I can see, so I would recommend dropping that unnecessary notation, especially given that the rewards in this paper are not given by a reward model.\n\n7. Typo in line 138: “Eplosion” -> “Explosion”.\n\n8. Typo in line 195: There is no line 8 in Algorithm 1. Perhaps this should have been line 5?\n\n9. Figure 6 is missing a legend. Currently, it is not clear which line corresponds to which method. The red line seems to correspond to GRPO, but what are the remaining two lines? Are they GRPO combined with ReDit for either Gaussian or uniform noise?\n\n10. Figures and tables do not appear in the order that they are referenced, which can be confusing. For example, Table 2 and Figure 6 are referenced before Table 1 and Figure 5.\n\n11. Typo in line 283: “We provides a theoretical analyzing how…” is not grammatically correct.\n\n12. Equation 5 is missing an expectation over the introduced noise $\\epsilon$, which does appear in the proof of Proposition 1.\n\n13. It is worth explicitly specifying what the variance of “Gradient Noise” refers to in Equation 7.\n\n14. The statements in lines 302 to 305 on why the additional gradient noise is beneficial should either be hedged or refer to evidence in the literature. Currently, they are phrased as concrete truths, while they seem to be more on the line of intuitive arguments.\n\nQUESTIONS:\n--\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenges of optimizing large language models (LLMs) under discrete reward signals, which often lead to gradient instability, slow convergence, and inefficient exploration. The authors propose ReDit, a simple yet effective method that introduces zero-mean random noise (e.g., Gaussian or uniform perturbations) to discrete rewards, thereby smoothing the reward landscape. This approach mitigates gradient vanishing and explosion by providing continuous exploratory signals, accelerating convergence, and improving final model performance. Theoretical analysis confirms that ReDit preserves unbiased gradient estimates while introducing beneficial variance. Empirical results across tasks like GSM8K, MATH, and Geometry3K demonstrate that ReDit achieves comparable performance to vanilla GRPO with only ~10% of the training steps and a 4% improvement when trained equally. The method also generalizes to other RL algorithms (e.g., DAPO, Dr.GRPO) and exhibits robustness across perturbation schedules.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths\n\n- The idea is interesting.\n- The paper provides theoretical analysis of the proposed method.\n- The paper is easy to follow.\n\n#### Weaknesses\n\n- The paper only evaluates ReDit using LoRA. However, it is not clear the performance of ReDit with full fine-tuning.\n- The paper does not evaluate ReDit on more challenging tasks, such as AIME24.\n- The paper does not use reasoning models for experiments, such as DeepSeek-R1-Distill-7B. Therefore, the paper does not show the effectiveness of ReDit on reasoning tasks.\n\nQUESTIONS:\n#### Questions\n\nBesides the weaknesses above, further questions are as follows:\n\n- Since DAPO and VAPO use several tricks to improve the RL training process, do these two methods have gradient exploding cases like the results in Figure 3?\n- L222 - L224: The authors claim that they want to isolate the effects of SFT. However, the authors uses two instruction models.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Reward Dithering (ReDit), a method that can mitigate the limitations (e.g., gradient anomaly, unstable optimization, and slow convergence) of the discrete reward of Group Relative Policy Optimization (GRPO) by introducing zero-mean random noise to the discrete reward. This paper evaluates ReDit on mathematical reasoning tasks such as GSM8K, MATH, and Geometry3K. Experiment results show that ReDit improves the accuracy, compared to GRPO. Also, this paper provides theoretical analysis which proves that ReDit produces an unbiased estimate of the original gradient, introduces beneficial gradient variance, and improves convergence speed.\n\nSTRENGTHS AND WEAKNESSES:\nSome strengths of this paper can be summarized as follows:\n- S1. First of all, this paper is well written.\n- S2. The main idea of adding random noise to the discrete reward seems simple and effective. Especially, ReDit seems to improve the optimization stability and convergence speed.\n- S3. This paper provides mathematical analysis that proves the effectiveness of ReDit.\n\nSome weaknesses of this paper can be summarized as follows:\n- W1. One of main concerns of this paper is that actual performance improvements such as accuracy do not seem significant. For example, the improvement of ReDit compared to GRPO is only 1.69 (i.e., from 89.07 to 90.76) on GSM8K. On MATH, it is about 4.54 (i.e., from 48.01 to 52.55).\n\nQUESTIONS:\n- Q1. Figure 5 and 6 shows multiple lines, but does not provide a legend. Please provide a proper legend on the figures. I assume that the orange line indicates ReDit with Gaussian noise, the blue line represents ReDit with uniform noise, and the red line denotes the baseline.\n- Q2. The authors evaluate ReDit only on mathematical reasoning tasks such as GSM8K, MATH, Geometry3K. Can does ReDit improve the accuracy on coding tasks such as HumanEval, APPS and CodeContests?\n\nLIMITATIONS:\nThe authors provide some limitations in Section 5 (i.e., Limitations and Conclusions).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper makes an interesting discovery that by adding a noise to the discrete 0-1 reward, the performance of the RL can be improved. Specifically, the authors of this paper credit the improvement to the smoothen reward landscape, which in turn resolves the gradient vanishing and gradient explosion issues. Experiments on both LLMs and VLMs with different RL algorithms, such as DAPO, GRPO, and REINFORCE++. This paper also provides theory analysis on how the added noise can help to accelerate the RL training process.

STRENGTHS AND WEAKNESSES:
## Strengths

1. The proposed method is simple and effective. For RL training, by adding a noise to the discrete 0-1 reward is almost like a free lunch. This method is orthogonal to modalities and different RL algorithms, which means it can be easily adopted to existing RL frameworks.

2. This method has solid theoretical analysis, including the motivation from the perspective of grad_norm and the convergence analysis. 

## Weaknesses

1. The proposed method is somewhat similar to several other regularization methods, such as Dropout, Data augmentation, and Stochastic Gradient Descent. It would be better if the authors could discuss the differences and similarities between the proposed method and these methods.

2. Regarding the experiments. I would suggest the authors to evaluate the model on more widely used benchmarks, such as AIME-24 for LLMs.

QUESTIONS:
1. The proposed method is somewhat similar to Dropout, Data augmentation, and Stochastic Gradient Descent. All these methods introduce noise to the training process and improve the performance. Could you discuss the differences and similarities between the proposed method and these methods?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03