DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization
Authors: Gang Li
Ming Lin
Tomer Galanti
Zhengzhong Tu
Tianbao Yang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Large Reasoning Models
Discriminative Learning
Primary area: deep_learning
TL;DR: We proposed a framework for reinforcing large reasoning models with discriminative constrained optimization , grounded in the principle that increasing the scores of positive answers while decreasing those of negative ones.

Abstract: The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning models (LRMs). In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation of question-level difficulty bias arising from its group relative advantage function. We also identify a connection between GRPO and traditional discriminative methods in supervised learning. Motivated by these insights, we introduce a new **Discriminative Constrained Optimization (DisCO)** framework for reinforcing LRMs, grounded in the principle of discriminative learning: increasing the scores of positive answers while decreasing those of negative ones. The main differences between DisCO and GRPO and its recent variants are: (1) it replaces the group relative objective with a discriminative objective defined by a scoring function; (2) it abandons clipping-based surrogates in favor of non-clipping RL surrogate objectives used as scoring functions; (3) it employs a simple yet effective constrained optimization approach to enforce the KL divergence constraint.  As a result, DisCO offers notable advantages over GRPO and its variants: (i)  it completely eliminates difficulty bias by adopting discriminative objectives; (ii) it addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach, yielding long and stable training dynamics; (iii) it allows the incorporation of advanced discriminative learning techniques to address data imbalance, where a significant number of questions have more negative than positive generated answers during training. Our experiments on enhancing the mathematical reasoning capabilities of SFT-finetuned models show that DisCO significantly outperforms GRPO and its improved variants such as DAPO, achieving average gains of 7\% over GRPO and 6\% over DAPO across six benchmark tasks for a 1.5B model.

DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization
Gang Li"
Ming Lin
Tomer Galanti!
Zhengzhong Tu
Tianbao Yang
Texas A&M University
(gang-li, 1i, galanti, tzz, tianbao-yane @tamu. edu
Linming04@gmail.c
Abstract
The recent success and openness of DeepSeek-RI have brought widespread atten- tion to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning models (LRMs). In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation question- level difficulty bias arising from its group relative advantage function. We also identify connection between GRPO and traditional discriminative methods su- Discriminative
pervised learning. Motivated by these insights, introduce Constrained Optimization (DisCO) framework for reinforcing LRMs, grounded in the principle of discriminative learning: increasing the scores of positive answers while decreasing those negative ones. The main differences between DisCO and GRPO and its recent variants are: (1) it replaces the group relative objective with discriminative objective defined by scoring function; (2) it abandons clipping-based surrogates favor of non-clipping RL surrogate objectives used scoring functions; (3) it employs simple yet effective constrained optimization approach enforce the KL divergence constraint. As result, DisCO offers notable advantages over GRPO and its variants: (i) completely eliminates difficulty bias by adopting discriminative objectives: (ii) it addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach, yielding long and stable training dynamics; (iii) it allows the incorporation of advanced discriminative learning techniques address data imbalance, where significant number of questions have more negative than positive generated answers during training. Our experiments on enhancing the mathematical reasoning capabilities of SFT-finetuned models show that DisCO significantly outperforms GRPO and its improved variants such DAPO, achieving average gains of 7% over GRPO and 6% over DAPO across six benchmark tasks for 1.5B model
1 Introduction
The recent success and openness of DeepSeek-RI have sparked a surge of interest in large reasoning models (LRMs), partieularly in the context of fine-tuning via reinforcement learning (RI.) [23]. The core approach involves iteratively generating synthetic data using the reasoning model and applying a rule-based reward mechanism to label the outputs. These rewards are then used to update the policy, i.e., the reasoning model itself. Notably, this framework, featuring novel policy optimization method called Group Relative Policy Optimization (GRPO), has enabled DeepSeck-RI to achieve performance comparable advanced proprietary LRMs that time such as OpenAl-ol many reasoning benchmarks. As result, GRPO has rapidly become a focal point for advancing LRM capabilities, particularly in domains like mathematics and scientific reasoning.
The code is available at: https://github. http:///hhhhbbccoommoplaaaaaaaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Several efforts have sought to replicate the performance of DeepSeek-RI or to further enhance while few others have tried to identify
reasoning models using GRPO 67
31
42
27
69
74
Useful tricks have been introduced to For instance, DAPO [79] employs two distinet clipping hyperparameters to mitigate entropy collapse, encouraging exploration. Dr. GRPO [40] removes the variance normalization in advantage function, aiming to mitigate the issue of difficulty bias. However, these approaches remain heuristic and ad-hoc, lacking principled foundation and falling short of fully addressing GRPO's inherent limitations. Our analysis identifies that Dr. GRPO continues to suffer from the difficulty bias issue, while our experiments show that DAPO may induce excessive entropy growth, producing highly random outputs. This motivates us to explore a central question:
its inherent limitations with potential remedies 40
79
39
improve GRPO 44
79
27 13
How can we design more effective optimization methods for reinforcing large reasoning models in a principled manner without inheriting the limitations of GRPO?
This paper addresses the above question through a complete redesign of the objective function, grounded in the principles of discriminative learning. Specifically, we first analyze the objective function of GRPO and its variants under binary reward setting, leading to two key insights: (1) the root cause of GRPO's s difficulty bias lies in its group relative advantage function, which induces disproportionately small weights to questions that are either too easy or too hard; and (2) there exists a conceptual connection to traditional discriminative approaches in AUC maximization, which aim to increase the scores of positive outputs while decreasing that of negative outputs.
Building upon these insights, we propose a principled optimization framework for reinforcing large reasoning models based on discriminative learning. Specifically, we optimize discriminative objective using a proper scoring function over input-output pairs, which increases the score of positive outputs and decreases that of negative ones. The flexibility of our framework allows to leverage simple non-clipping RL surrogate objectives as scoring functions without suffering from entropy instability, and to incorporate advanced discriminative techniques to address data imbalance in generated rollouts. To ensure training stability, we adopt a simple yet effective constrained optimization method to enforce a trust region constraint bounding the KL divergence between the updated model and the old model. Our experiments for mathematical reasoning show that DisCO significantly outperforms all baselines for fine- tnn-ttnnng DeepSeek-Ri RD-istlll Ri-Distill-Qwen and -Llama models with a maximum 8k response length for both training and inference, and also achieves a better performance than GRPO that uses a maximum 24k length for training and 32k length for inference.
Our main contributions are summarized as follows:
We present analysis of GRPO's objective function, identifying the root cause of difficulty bias and revealing its conceptual connection to classic discriminative methods for AUC maximization.
We introduce a principled discriminative constrained optimization framework for reinforcing large reasoning models, which avoids both difficulty bias and training instability. This framework gives rise to a family of methods we refer to as DisCO.
We demonstrate significant improvements of our DisCO method over GRPO and four other baselines, including DAPO, through experiments for fine-tuning LRMs on mathematical reasoning tasks, with evaluations across six benchmarks.
2 Related Work
Large Reasoning Models (LRMs). Recent advances of LRMs, such as OpenAl ol [51). DeepSeek- RI (23 and Kimi K1.5 63), have demonstrated strong reasoning capability in solving complex tasks. Departing from earlier approaches in I.LMs, such as Chain-of-thought (CoT) prompting [66
48. 81), Tree-of-Thought (78). Monte Carlo Tree Search [18 64 71]. a major breakthrough was achieved by scaling RL training using verifiable rewards to incentivize LLMs to learn through self- exploration (33 63]. Inspired by DeepSeek-Bi - core algorithm GRPO [59] the research community
has actively pursued improved techniques for large-scale RL training, focusing primarily three directions: algorithm design [79]
63 61), reward curation 82 67 [9] and sampling
strategies [79 27
83
30].
Our work falls under the category of algorithm design.
Among these, Dr. GRPO 40 identifies response-level length bias and question-level difficulty bias in GRPO algorithm, advocating the removal of length and advantage normalization to improve token efficiency. DAPO [79 highlights several limitations of GRPO, such as entropy collapse, training instability, and biased loss, and addresses them through techniques like decoupled clipping,

---

dynamic sampling, and token-level policy loss. GPG [3] introduces a simplified REINFORCE. based objective that eliminates the need for both the critic and reference models, thereby enhancing scalability for RL. training. TRPA [61 simply uses the Direct Preference Optimization (DPO) objective and KL. divergence regularization for fine-tuning LRMs. It can be recovered from our basic approach, which uses logistic function as the surrogate loss, the log of likelihood ratio with respect to a frozen reference model as the scoring function, and the KL divergence as a regularization rather than a constraint. However, it does not address the imbalanced rollouts. The uniqueness and significance of our contributions lie in the analysis of GRPO objective and its variants that reveal key limitations, and the integration of advanced discriminative learning approaches for handling imbalanced rollouts and efficient constrained optimization technique for ensuring training stability.
Reinforcement Learning (RL). RL is a learning paradigm centered on control and decision decision-making. in which an agent optimizes a target objective through trial-and-error interactions with its environ- ment [8. RL approaches are typically categorized into model-based [60 53 49 17 and model-free
Among model-free methods, the evolution from Vanilla Policy and PPO [57] has influenced the development of GRPO. In the context of fine-tuning LLMs, another line of work is RL from human feedback (RLHF). An early example of connecting RL with LLMs dates back to OpenAl's s work on integrating human preferences to improve text generation tasks, such as summarization using the PPO algorithm [85] This approach was later extended to fine-tune LLMs for instruction following and/or alignment helpfulness
methods 68 62
46
56
57
38 20
Gradient 168 62 to TRPO 156
and harmlessness [22 3 22 75). Due to the high data requirements and training costs of standard RLHF, off- off-policy methods like DPO [54] and its variants 2 16 72 44 24], have been proposed to reduce reliance on explicit reward models. Another line of on-policy algorithms for RLHF, such as RLOO 11. ReMax (36), and REINFORCE+ [29), has been introduced to reduce the computational burden by removing the critic network in PPO. While some works [4 32
29
10 70] attempt to
adapt RLHF techniques for reasoning tasks, they have not yielded significant improvements.
Discriminative Learning. Parallel to RL, discriminative learning is another classical learning paradigm. that has been studied extensively for many traditional tasks, including multi-class classifi- cation [14, 15 6 AUC maximization [76 80). and learning to rank [9] 19 77. These methods are grounded in the common principle of increasing prediction scores for positive (relevant) labels (data) while decreasing scores for negative (irrelevant) ones. Nevertheless, discriminative learning remains under-explored in the cotext of LLM training. Recently, Guo et al. (25) proposed discriminative probabilistic approaches for supervised fine-tuning of LLMs. However, unlike our approach, they did not employ an RL framework with verifiable rewards to fine-tune LRMs.
3 Preliminaries
We consider fine-tuning a generative reasoning model To parameterized by O. The old model in one step of learning is denoted by Tola- It is used to generate answers for a set of input questions. Given a question q (with prompt included), the generated output o follows the distribution (asa which includes reasoning traces and the final answer. Specifically, output o is generated token by token, i.e., 02 (oaa(-lg, ocx), for L = 1,... ,o/.- We consider a rule-based reward mechanism that returns binary value for given question q and its corresponding answer in the output which uses either exact match against extracted answer or formal verification tool [23] 33 55). Let r() E {1,0} denote the reward assigned to an output o with respect to the input q. Let p(g) Eowna((-l)) [r(o]g)] E [0, 1], which quantifies the difficulty of the question q under the model Told- We denote by ( (:14) the conditional distribution of outputs when the reward is one (i.e., positive answers) and by (-1() the conditional distribution of outputs when the reward is zero (i.e., negative answers). By the law of total expectation, for any function g(o, q) we have
Eownaa(-1o) loo,, 4) = P(q) o to (1Im) (9 4) + (1 - p(z)) B 10 9).
(1)
Group Relative Policy Optimization (GRPO). The key idea of GRPO is to generate multiple outputs for an input and define a group relative advantage function. For analysis, we consider the expectation formulation instead of empirical average of the GRPO objective for maximization:
|o]
1
TTe Te 14, o21) (ota (g, oce)
Alobe))
(0) = Bo
31 (Tel |TTet)
(2)
1=1
3

---

We ght Question
Dat. pesquestion accuracies
Sclved
001
cero
025
03
Ain
G670 -11
A22
0970.10
(b)
Figure 1: (a) Weight on questions based on correctness probability p; (b) Histogram of per-question accuracy evaluated in the GRPO learning; (c) Comparison of the ratio of questions with 100%
correctness probability; (d) Comparison of the ratio of questions with 0% correctness probability.
where [(x y) = min(ry, clip(æ; 1-e, 1++)y). A(olg) ((lolg)-Ea-aee Ba the advantage Varro'maaa ( (("l())
function that quantifies how much better the reward of o is compared to average reward, The is a frozen reference model.
40 Many of them retain the advantage function A(olg) while modifyyng other components such typer-paramete the normalization factor and the likelihood ratio. Several works employ an unnormalized advantage
Recently, several variants GRPO have been introduced 779 13
39
82
function A(olg)= = r(ol4) - Eo r(o'lg)
40 13
4 Analysis of GRPO and its variants
In the following analysis we assume p(g)= E (0, 1); otherwise we can remove them from consideration as done in practice 23 42 59].
Proposition 1 Let us consider the objective of GRPO and its variants with the following form:
lol
Jo(0) Eg Bownottt
Σs
Te(oolq. o21)
A(ooo)
(3)
Assume that f(x, y) is non-decreasing function of x such that f(x y) 1y 0)yf+ 11y 0) D) (x, 1), where both f+, are non-decreasing functions of x, then we have
S
[0) = =Ey/p(a)(1 p(q)) (1))) (1)l))))) [3 (o. Sg (o'.9)].
(4)
where 8 (o. 9) Toj I |+ (o(oolg.002)) 1) In particular, for GRPO we huve
1
and 8p (0.
1 [ol Ela
/
xx [00(011..0cc)) TTaul (.00))
f (z, 1) = min(x, c), f (xx mmx(xr, , 1 1 c).
(5)
The assumption of f(r y) indeed holds for GRPO and its variants. We will present the analysis for several variants of GRPO in Appendix B.3
Remark:
The proof of the above proposition is included in Appendix B.1 and is inspired by 47 with differences that lead to two new insights from Proposition [ regarding the two components of To. First, let us consider the component Ba-ath (oortttlll).''/ee (2).0))))) _ (o, q) 8p (o', 4)). Since both f+ and f are
non-decreasing functions of the first argument, then both s (o. q) and sø (o. are non-decreasing functions of (o.lg.. oct). Hence, maximizing To would increase the likelihood of tokens in the positive answers and decrease the likelihood of tokens in the negative answers. This makes sense as we would like the new model to have high likelihood of generating a positive (correct) answer and low likelihood of generating negative (incorrect) answer. This mechanism is closely related to traditional discriminative methods of supervised learning in the context of AUC maximization [77]. which aims to maximize the scores of positive samples o ~ ( while minimizing scores of negative samples o' (-4a(- where the acts like the classification task in the AUC maximization. Hence, in the context of discriminative learning, we refer to 8 (o, q) and 8 (o, q) as scoring functions. Therefore, E Bowe (owa of (014) (-19), [s* (o,4) s (o q)) is discriminative objective.
Second, let consider the component w w(g) = (1 - p(g)), which acts like weight scaling the discriminative objective for each individual input question. It is this component that leads to difficulty bias. As shown in Figure I [a), questions with very high p(g) values (close to 1) very

---

low p(q) values (close to 0) receive small weights for their discriminative objectives, causing the optimization focus primarily on questions of intermediate difficulty while paying little attention to hard questions (p(4) R 0) and easy questions (p(q) R 1). This mechanism may significantly hinder the learning efficiency. Intuitively, if the generated answers have only one correct solution out of 10 trials, i.e. p(g) = 0.1, we should grasp this chance to enhance the model instead of overlooking it. On the other hand, even when we encounter an easy question with a probability of p(g) = 0.9, we should keep improving the model rather than being satisfied because it still makes mistakes with respect to this question. Our hypothesis is that removing this weight could accelerate the training. To validate this hypothesis, we conducted series of empirical experiments for fine-tuning 1.5B model as described in Section 6 We start by examining whether a substantial number of questions have correctness probabilities (p(q)) near 0 or 1. As shown in Figure []), during GRPO training, the correctness probabilities across individual questions appear broadly distributed, with many near 0 or
1. Then, we compare the original GRPO with a variant that removes weight vI 10l1-Ppi. p(g)):
JGRPO_ RW
E. de of as [s (0,g) 8 (d', .)] 3Dr. (ver))
(6)
The results are shown in Figure (c) and dd). We can observe that the variant without the weighting mechanism quickly achieves higher ratio of 100% correctness and lower ratio of 0% correctness, confirming the detrimental impact of the inappropriate weighting.
We note that the difficulty bias has been pointed out in recent work Dr. GRPO 40 To mitigate this issue, Dr. GRPO uses the un-normalized advantage function A(olg). However, with similar analysis as above (cf. Appendix B.33) we can derive that Dr. GRPO still has a question-level weight w(g) = (1 - p(g)) before the discriminative objective. As shown in Figure [a), this weight mitigates but does not climinate the imbalanced weight across questions.
5 A Discriminative Constrained Optimization Framework
While the last section has suggested a tangible remedy to address the difficulty bias of GRPO and its variants by removing the weight before the discriminative objective, there are other issues of the scoring function of GRPO and its variants. Next, we propose a general discriminative learning framework for reinforcing LRMs and incorporate advanced techniques to facilitate the learning.
5.1 A basic approach
Motivated by the connection with AUC maximization, we redesign the objective directly from the principle of discriminative learning. For given question q. let so q) denote scoring function that measures how likely the model To "predicts" the output o for a given input g Then the AUC score for the "task" q is equivalent to E Fo o' ~na [T(s((o, q) > so so q))]. Using continuous surrogate function É, we form the following objective (in expectation form) for maximization:
(0) =
(ss((,,)) q) sg(o'.g)) .q).
Different surrogate functions E(-) can be used. For comparison, with GRPO, we simply use the identity function ((s) = 8. One difference from the diseriminative objective (6) is that we use a single scoring function (o q) for both positive outputs and negative outputs o. It is notable that the different scoring functions for positive and negative outputs in (6) actually arise from the clipping operations of GRPO objective. Recent works have found that the clipping could lead to entropy collapse (79.. In addition, the clipping could cause the vanishing gradient, which may also slow down the learning process. To avoid these issues, we consider non-clipping scoring functions.
Scoring functions. We consider two choices of scoring functions, i.c., log likeliiod and likelihood ratio. The log-likelihood (log-L) scoring function is defined by sa g) = -/ D1e log Ne (or (4, oc2)).
The likelihood ratio (L-ratio) scoring function is computed by se (0, - Jo| Σ Taulor q. In Appendix [B.2] we discuss the connection between the two scoring functions and the surrogate objectives of vanilla policy gradient methods 68 and TRPO (56), respectively.
selorlg,ecu),
Stabilize the training with Constrained Optimization. Training instability is long-standing issue in RL [56] 577. Different methods have been introduced to ensure stability. Recent RL-based
methods for learning reasoning models either follow the clipping operation of PPO [57] or use the
KL divergence regularization DxL(TT) (Tee) or DKL (Toial [To)
[59 61 52). However, the clipping
in the context of generative models, "predicts" is like "generates".
5

---

operation could lead to the entropy collapse 779 which we try to avoid by using the non-clipped scoring function. The KL divergence regularization Kx. (Ta Teet) while being used in traditional RL is not effective for preventing entropy collapse [27 79 Regarding the regularization with DyL. [ earlier studies [56 57 has found that it would be difficult to choose a single value of the regularization parameter that performs well across different problems or even within a single problem where the the characteristics change over the course of learning. To tackle this issue, we revisit the idea of trust region constraint of TRPO (56), i.e., restricting the updated model 0 in the trust region 1x (Total TT)) S 8. As a result, we solve the following discriminative constrained optimization problem:
max (1( = En (10) wel (so((,, 2) - ss(0'.
(
s.t. (Tala | S 8.
For sake of efficiency, we use different optimization approach from TRPO to solve the above constrained optimization. Inspired by the recent advances of non-convex ineqaulity constrained optimization algorithm [35], we adopt squared-hinge penalty function for the constraint and solve the following problem with an appropriate penalty parameter 3:
max Egg E (1))) (s0(0,.) 9) - so(o', q)) - 3 ([ | - 6]²
where |+]+ max 0}. It has been shown that under an appropriate assumption regarding the constraint function and 8. solving the above squared-hinge penalized objective (9) can return KKT solution of the original constrained problem (8) We refer the readers to 35 for more in-depth analysis of this approach.
Finally, we would like to emphasize the difference between using the squared-hinge penalty function
and the regular KI. divergence regularization ,3 ( [To). (To)al|TTO) The squared-hing penalty function
has dynamic weighting impact for the gradient, V (T))
0) 22 (Tolal [TT)
6]+ VDKL (DDlalTallll (Tatal|Ta) such that if the constraint is satisfied then the weight 2,3 [Dx (Foual [T)) 8]+-
before the gradient of the regularization term DKL(TToL (Te) becomes zero. This means the KL divergence regularization is only effective when the constraint is violated. In contrast, the regular KL
divergence regularization 3IDxL (Towll (T) always contributes a gradient , VVD (Tota (T)) no matter whether the constraint is satisfied or not, which could harm the learning.
5.2 An improved approach for tackling imbalanced rollouts
One advantage of designing the objective based on the principle of discriminative learning is the ability to leverage a wide range of advanced techniques from the literature to improve training. A key challenge in RL fine-tuning for reasoning models is the sparse rewards, which lead imbalance in generated rollouts. Specifically, for some questions where p(g) < 1. the number of negative outputs can significantly exceed the number of positive ones. This reflects classic data imbalance issue,
which has been extensively studied in the discriminative learning community 84 58 50
To address
this issue, we consider distributionally robust optimization (DRO)
84 50
Let us first discuss why the basic approach could be ineffective for combating the imbalanced rollouts. The objective function J. is motivated by maximizing AUC for each question g. i.e.,
E Powat o' [T( (s((0, q) > so(o', -4))] However, when there is much more negative data than positive data, AUC is not good measure. For example, let us consider scenario where there are I positive 04 and 100 negatives {o_...., 1000 J. the scores of these data are s(o2_ q) = 0.9, s(o+.0) g) = 0.5, s(o2. q) = s(o2. ... = s00100. = 0.001, then the AUC score is 99 = 0.99. The AUC score is high but is not informative as the model still generates the negative data o!_ more likely than the positive data 0+. In the literature, this issue has been addressed by maximizing a partial AUC score, which considers the pairwise order between all positives and the top ranked negatives. We utilize a surrogate function of partial AUC score formulated from the perspective of DRO 84
T00
Consider a question g and a positive data o. We denote by Q the set of probability measures Q on negative data given g (absolutely continuous with respect to a- (q)). Denote by [xx Ro (: q)) the KL divergence between distribution Q and the negative data distribution ( A DRO

---

Table 1: Comparison of different methods for reinforcing large reasoning models. "L-ratio" means
likelihood ratio, "log-L" means log- lielliielhood, "proper" means any proper scoring function.
Method
Difficulty Bias
Clipping
KI. Divergence
Score Function
Tackles Imbalanced Rollouts
GRPO 25
Yes
Yes
regularization. Rir
clipped L-ratio
No
Dr. GRPO 40
Yes
Yes
No
clipped L-ratio
No
DAPO 79
Yes
Yes
No
elipped L-ratio
No
GPG [R]
Yes
No
No
log-L
No
TRPA 61
No
No
regularization, Rold
log L-ratio
DisCO
No
No
constraint, Raid
proper
Yes
formulation for partial AUC maximization is given by
84 Theorem 2]:
inf QE@
TQku(Q, (-12) + Eo' (Eo'>ols((o.)) se(o',9))
sold q) sa(o. so(o.0)) ))
= =- =r log
E o' ( exp
As result, we construct the following DRO-based objective for maximization:
sp sa((o,
se so(0,
T2(0) -.g (o log
F. B exp
(10)
T
is easy to show that T2) = (0) by Jensen's inequality for the convex function - log. Hence, maximizing J2 (0) will automatically increasing (0). However, the reverse is not true. This also explains why maximizing T2( could be more effective than maximizing (0). The above risk function is also known optimized certainty equivalents (OCE) in mathematical finance (4]. We would like to point out that although OCE or DRO has been considered for RL in existing works 665 773,, they differ from our work for addressing different issues. Wang al. [65 apply the OCE to compute robust reward, replacing the standard expected reward used in traditional RL settings. Xu al. (73 adopt DRO for direct preference optimization of L.Mss, aiming mitigate the noise in human preference data by addressing the distributional shift between the empirical distribution of (q, o, o') and its true underlying distribution.
Finally, we solve the following discriminative constrained optimization problem by using the same squared-hinge penalty method:
max F2(0) - Eg own (11)) log (
Fog (E (Barnaigtto) exp (so(o', (o(0',.) -0s(s,s0)))) (
(11)
s.t. ( ( S 6.
To differentiate the approach for solving 8 and [I]. we refer to the former as DisCO-b and the
latter DisCO. In practice, all expectations will be replaced by empirical averages and the KL
divergence is also estimated at each iteration by using sampled data following [52). We present full
algorithm in Algorithm [ in Appendix. Finally, we give a comparison between DisCO and existing RL fine-tuning methods for reinforcing LRMs from different aspects in Table
6 Experiments
In this section, we empirically evaluate the effectiveness of the proposed DisCO by comparing with GRPO and other variants for reinforcing SFT-tinetuned models.
Task Setting. We validate our method on mathematical reasoning tasks. Specifically, we use the CeesScaleR-Previeww Srrviee-DDttsst 42 for training, which includes AIME problems from 1984 2023, AMC problems before 2023, and questions from the Omni-MATH [21] and Still [45] datasets, totaling approximately 40.3k unique problem-ans pairs. We evaluate models on six benchmark datasets: AIME 2024, AIME 2025, MATH 500 (8 37), AMC 2023, Minerva [34], and Olympiad Bench (O-Bench) 26) Following
we adopt the pass@) metric [11] averaged over k: = 16 responses for each question to ensure the reliability of model performances. The metric for each question is calculated both the training and evaluation of our method and the baselines (unless otherwise specified), the maximum response length is limited to 8k tokens. To verify the generalizability of our method to other datasets, we also conducted experiments on DAPO-Math- 17k [79) dataset, which included in the Appendix A.3
23 42
Σ 1= I(o; is correct), where denotes the i-th generated response. For

---

Table 2: Comparison with baseline models and baseline methods for fine-tuning 1.5B models. OpenAl-ol-prrviee is included as a reference. MRL denotes Max Response Length utilized in training/testing. The shaded models are trained by other works and the shaded numbers are reported in their original works in [42] All other results are either evaluated on existing models or on the models trained by us using different approaches. Methods in the bottom area are all for fine-tuning DeepSeek-R R
model on the same DeepScaleR dataset. DS is short for DeepSeek-
RI, DSR is short for DeepScaleR.
Model/Method
MRL(TrainTesst)
AIME 2024
AIME 2025
MATH
AMC 2023
Minerva
O-Bench
Avg.
OpenAT-ol --rrvveww
0.4
0.814
DS Distill Qwen 1.5B
32k+ 33k
0.288
0.263
0.828
0.629
0.265
0.433
0.451
DS-Distill-Qwen- -eiil--Qen-1.5B
32k+ 8k
0.181
0.215
0.758
0.515
0.237
0.353
0.376
STILL-3-1.5B-proviee
29k 32k
0.325
0.248
0.844
0.667
0.290
0.454
0.471
DSR-1.5B-Previes
24k 32k
0.431
0.304
0.878
0.736
0.302
0,500
0.525
DSR- Sereeie
24k 8k
0.358
0,258
0.860
0.679
0.297
0.473
0.488
GRPO
8k
0.277
0.242
0.838
0.647
0,276
0.462
0.457
GRPO-ER
8k/ 8k
0.298
0.242
0.839
0.649
0.279
0.452
0.460
Dr. GRPO
8k 8k
0.252
0,238
0.831
0.631
0,268
0.440
0.443
DAPO
8k 8k
0.310
0,252
0.848
0.675
0.296
0.456
0.473
TRPA
Sk. 8k.
0.354
0,235
0.835
0.653
0,283
0.458
0.470
DisCO (I.-ratin)
8k / 8k
0.381
0,306
0.878
0.746
0.319
0,512
0.524
DisCO (log-L)
8k. / 8k
0.404
0.317
0.876
0.758
0.333
0.509
0.533
Table 3: Comparison with baseline models and baseline methods for fine-tuning 7B models. Methods in the bottom area are all for fine-tuning DeepSeek-R1-DD RI-Distill-Qwen 7B model on the same Deep- ScaleR dataset.
Model/Method
MRI (Train/T//))))
AIME 2024
AIME 2025
MATH 500
AMC 2023
Minerva
O-Bench
Avg.
DS Distill Qwen 7B Seeill-Qwen--B
32k+ / 32k
0,560
0.396
0.923
0.825
0,380
0.568
0.609
0.402
0:292
0.873
0.355
0.513
GRPO-LEAD-7B
0.470
0.345
0,893
0,748
0,372
0,500
0.555
TRPA
8k Sk
0,570
0,870
0,780
0,360
0,550
GRPO
0.498
0.394
0.916
0.592
GRPO-ER
8k 8k
0,515
0.381
0.916
0.825
0,376
0,544
0,593
Dr. GRPO
0.488
0.346
0,910
0.792
0.368
0.546
0.575
DAPO
8k/ 8k
0.454
0.335
0.907
0.799
0,368
0.535
0.570
TRPA
8k/ 8k
0,510
0.367
0.898
0.779
0.379
0.534
0.578
DisCO (L-ratio)
8k/
0.583
0,421
0.923
0.852
0,399
0.585
0.627
DisCO (log- L.)
8k/ 8k
0.558
0.410
0.927
0.854
0.410
0,592
0.625
Table 4: Comparison with baseline models and baseline methods for fine-tuning 8B models. Methods in the bottom area are all for fine-tuning ScaleR dataset.
model on the same Deep-
Model/Method
MRL/Trrin/ Test)
AIME 2024
AIME 2025
MATH 500
AMC 2023
Minerva
O-Bench
Avg.
DS Distill-Llamme 8B
32k+ 32k
0.506
0.346
0.896
0.815
0.295
0.541
0.566
32k+ / 8k
0,348
0.238
0.825
0.652
0.267
0.440
0.462
GRPO
Sk/ 8k
0.410
0.873
0.759
0.516
GRPO+ER
Sk/ 8k
0.408
0.277
0.882
0.785
0.311
0,511
0.529
Dr. GRPO
Sk/ 8k
0.423
0.285
0.867
0.786
0.300
0.497
0.526
DAPO
Sk/ Sk
0.333
0.308
0.879
0.794
0.325
0.522
0.527
TRPA
8k 8k
0.454
01.779
0.864
0.756
0.289
0.518
0.527
DisCO (L-ratio)
8k/ 8k
0.506
0.356
0,900
0.831
0.326
0.553
0.579
DisCO (log-L)
8k/ 8k
0,523
0.354
0.896
0.843
0.331
0.560
0.584
Models. We conduct experiments with fine-tuning three models: SeepSeee--R--issill--Neeee model (1))) DeepSeek-RP -1 --Distill-Qwen- model (Q7B), and DeegsSee-RR--Disill-Llaaaa (L8B). All are distilled reasoning models.
Baselines. We primarily compare our methods with five most recent state-of-the-arr reinforcement
learning methods, including (1) GRPO [3]; (2) GRPO with an entropy regularization (GRPO-
ER) that adds an entropy on probabilities of output tokens as regularization to prevent entropy collapse, which is used by DeepScaleR 42): (3) Dr. GRPO 40); (4) DAPO's objective 779: (5) TRPA [6]] For comprehensive evaluation, we also include a set of reasoning models that are trained from the same base model by other studies with various techniques, such as (6) STILL-3-1.55 preview [2), which adapt GRPO by periodically replacing the reference model after a fixed number of training steps; (7) DeepScaleR(DSRR 1.50SR Preview that uses maximum response length of 24k for
training 42); (8) GRPO-LEAD-TE 7B 82 which extends GRPO by incorporating length-dependent
8

---

GRPO
GRPO GRPO
DAPO
DisCO raratl)
DisCo
04k-
044
Stop
Stop
Srup
(b)
(C)
(d)
Figure 2: Training dynamics different methods: left two are for fine-tuning DeepSeek-R -Distilll wB model. (a), (c)
Qwen-1.5B model and right two are for fine-tuning DeepSeek-Re plot the training reward (averaged over generated outputs for questions used in each step) vs the number of training steps (cf. Algorithm Ip: (b), (d) plot the generation entropy training steps.
rewards, explicit penalty terms, and difficulty-biis based advantage reweighting to encourage concise and precise reasoning.
Training Details. For all the methods, we tune the constant learning rate in [5e- 1e-6, 2] with AdamW optimizer with weight decay as 0.01. Generally, learning rate of 2e-6 works better for the Q1.5B model, 1e for the Q7B model, and 5e for the L8B model. We employ training batch size of 128, a mini-batch size of 32, and 8 responses for each question. The temperature is set to 0.6 for both training and evaluation, following the usage recommendation from [23] For GRPO, 3 is set to 0.001 as commonly used [12 42). For GRPO-ER, we use a coefficient of 0.001 for the entropy regularization 422 For DAPO, we set flow to 0.2 and thigh to 0.28 by following their paper. For our method, 8 is set to 10 based on the empirical observation that the average KL divergence is around 2 * 10-5 and B set to 10 such that the effective weight of the KL. regularization when the constraint is violated by 8 is on the order of B * 5 0.1. Since L-ratio and log-L scoring functions for L-ratio and
have different orders, we choose {0.5, 1,5, 10). For fair comparisons, we do not implement Dynamic Sampling [79] for DAPO and other methods, as introduces approximately three times the sampling cost at each training step. All methods are run for 1,400 steps on Q1.5B models and 1,000 steps on Q7B/L8B models. Evaluations are conducted every 200 steps, and the best performance for each method is reported.
10 for log-L scoring function, from
6.1 Comparison with Baselines
Performance. We evaluate all the models across six mathematics-focuser benchmark datasets to demonstrate the effectiveness of DisCO. The results are summarized in Table 2 3 and 4 From Table for Q1.5B models, can observe that proposed DisCO methods consistently outperform other baselines by a large margin. Notably, DisCO (log-L) with 8k length for both training and inference achieves an 7% average improvement over GRPO and surpasses DeepScaleR-1 1.5.le---1.---ieiiee that was trained with maximum 24k length and evaluated with 32k length. similar trend is observed for Q7B models and L8B models (Table and Table 4). where DisCO significantly outperforms all competing approaches.
Training Dynamics. We compare the training dynamics of different methods in terms of training rewards and generation entropy. From Figure 2 for fine-tuning Qwen-1.555 1.5B and Qwen-7B models, we see that all baselines suffer from premature saturation due either entropy collapse for GRPO, GRPO-ER, Dr. GRPO or excessive entropy growth of DAPO, which leads to an early deterministic highly random policy. The entropy collapse phenomenon is also observed by 779 27 42
TRPA that uses a KL divergence regularization is also observed with instability in the generation entropy in later steps (around 1100 for Q1.5B model and around 800 for Q7B model). In contrast, our methods with the two scoring functions are most stable, with training rewards kept increasing and generation entropy maintained around 0.22. We also include training dynamics for the L8B model in Appendix A.2 which follows a similar trend.
6.2 Ablation Studies
(left) compares DisCO with DisCO-b using the L-ratio scoring function for training Q7B models for 1000 steps. The comparison clearly demonstrates the significant improvements of DisCO over DisCO-b, especially on the difficult AIME datasets. We also compare DisCO with DisCO-b for other settings in Appendix A. and observe that DisCO is consistently better than DisCO-b on average in all settings. It also notable that DisCO-b with different scoring functions are also better than other basclines trained evaluated by us in Table 2 and Table 3
DisCO vs DisCO-b. Figure 3

---

DIsCO
DisCO-b
0.52
0.54
Generation Entropy
0.4
Pass@l (Avg.
0.50
Pass@1
0.7-
0.48
0.6
0.45
0.
0.44
Cloped L-raiz Ing
0.42
Clppeet
0.40
0.3
AIME AIME 23
NATH
'Sic Winend G-gendiyg..
0.38
Clipeed
Letatin
Clisped 1eainn
Leatinn
Logat
0.0
Log
Step
1000
Figure 3: Ablation studies: left for comparing DisCO vs DisCO-b; middle and right for comparing
clipping with non-clipping scoring functions.
0.70
0.56
0.65
Constrained Opt.
Pass@1 (Avg.)
0.54
0.52
Pass@1
0.50
0.50
0.48
0.55
0.46
0.45
(L-ratio)
0.44
0.50
DisCo (log-L)
0.42
0.40
0.40
DaC0 DICCD-b
log-L
log-L
(0g
00g
Blas
Rog.
Liodi
Figure 4: Ablation studies: left for comparing KL regularization constrained optimization; middle for sensitivity of DisCO w.r.t. the hyperparameter r; right for contribution of each component.
Clipping vs Non-Clipping scoring functions. We compare non-clipping scoring functions L-ratio, log-L with clipped L-ratio (5) in our DisCO-b approach for training Q1.5B models Figure (middle and right). For the clipped L-ratio, adopt two versions: with Chigh 0.2 to align with GRPO objective, and amother with Chigh = 0.28 similar to DAPO objective. We can see that clipped L-ratio with thigh 0.2 causes entropy collapse while clipped L-ratio with thigh 0.28 leads to excessively high entropy level, both yielding worse performance than non-clipping scoring functions.
KI. Regularization vs Constrained Optimization. We investigate the advantages of constrained optimization over KL regularization for DisCO. Specifically, the KL regularization weight set to the commonly used 0.001 2 42). As shown in Figure ( constrained optimization performs better than KL regularization on both QI.SB and Q7B models. Moreover, during our experiments, observed that KL regularization leads to instability in training Q7B models, similar to TRPA, which indicates that KL regularization not sufficient to stabilize training.
Sensitivity of hy pyprpaaraeeter т. We study the sensitivity of DisCO to hyperparameter T on training Q1.5B models. Similar to above experiments, we run DisCO for 1400 steps with different E (0.5, 1, 5, 10). The result shown in Figure 3 (right) indicates that DisCO is not sensitive to 7 in these ranges.
Effeet of each design choice. We analyze the individual contribution of each component in DisCO by replacing its components separately with other designs. We experiment on Q1.5B models and compare with (1) DisCO-b that removes hard negative weighting; (2) adding question-level weight bias (1 - p(q) to DisCO-b, (3) replacing the KL-divergence constraint with KL-divergence regularization DisCO-b, and (4) using a clipping scoring function with Ehigh 0.2 in DisCO-b, respectively. From Figure 4 (right), we can see that each of our proposed components is important in DisCO's improvement, where the use of non-clipping scoring function of vital importance.
Conclusion
In this work, we have proposed novel discriminative constrained optimization framework for reinforcing large reasoning models, motivated by the analysis of the GRPO objective. The proposed framework is grounded the principle discriminative learning, avoiding difficulty bias and enhanc- ing training stability with constrained trust region optimization. The experiments on mathematical reasoning demonstrated the significant superiority of our approaches, compared with GRPO and its recent variants. While this work focuses on binary rewards, future extensions could incorporate discriminative ranking objectives, like (9) to handle non-binary rewards. It would be interesting to apply the proposed approaches for fine- fine-tuning larger models other reasoning tasks.
10

---

Acknowledgements
We are grateful for the reviewers' constructive comments. G. Li and T. Yang were partially supported by NSF grant #2306572.
References
[1] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402. 14740, 2024.
[2] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455, PMLR, 2024.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
[4] Aharon Ben-Tal and Marc Teboulle. An old-new concept of convex risk measures: The optimized certainty equivalent. Mathematical Finance, 17:449-476, 02 2007.
[5] Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran EI- Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf. Oren Tropp. Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Ger- ald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Subara, Olivier Delalleau, Zijia Chen, Zhilin Wang. David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Aleksander Ficek, Mchrzad Samadi. Jocelyn Huang. Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Bran- don Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor-Yiing Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky. Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen. Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yuting Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman. Erick Galinkin, Michael Evans, Shaona Ghosh, Katherine Luna, Leon Derezynski, Nikki Pope, Eileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Chris Alexiuk, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geifman, and Eric Chung. nemotro:m Efficient reasoning models, 2025.
[6] C.M. Bishop. Puttern recognition and machine learning, volume 4. Springer New York, 2006.
[7] Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent, In Proceedings of the 22nd international conference Machine learning, pages 89-96, 2005.
[8] Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li. Survey on large language model-enhanced reinforcement learning: Concept, taxonomy, and methods. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[9] Zhe Cao, Tao Qin, Tie- Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: From pairwise approach to listwise approach. volume 227, pages 129-136, 06 2007.

---

[10] Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimization for mathematical reasoning. arXiv preprint arXiv:2406. 2024.
[11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large 2021.
language models trained on code. arXiv preprint
[12] Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al. An empirical study on eliciting and improving rl-like reasoning models. arXiv preprint 2xXiv:5503.0454, 2025.
[13] Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546. 2025.
[14] C. Cortes and V. Vapnik. Support vector networks. Machine Learning, 20:273-297, 1995.
[15] Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernel- based vector machines. Mach. Learn. Res., 2:265-292 March 2002.
[16] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
[17] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv:1803. 001011, 2018
[18] Xidong Feng. Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint 2rXiv: 17179, 2023.
[19] Yoav Freund, Raj Iyer, Robert E Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences. Journal of machine learning research, 4(Nov): 2003.
[20] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[21] Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985 2024.
[22] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXvv:2407.21783 2024.
[23] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.11244, 2025.
[24] Siqi Guo, Ilgee Hong, Vicente Balmaseda, Tuo Zhao, and Tianbao Yang. Discriminative finetuning of generative large language models without reward models and preference data. arXiv preprint arXiv:2502. 18679, 2025.
[25] Siqi Guo, Ilgee Hong, Vicente Balmaseda, Tuo Zhao, and Tianbao Yang. Discriminative finetuning of generative large language models without reward models and preference data, 2025.
[26] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, arXiv preprint 14000 2024.
12

---

[27] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan. Chaojie Wang, Peng Cheng, Xi- aoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tian- wen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner series. http:///ttte//errrrrcooooa//eeee oottiiooonni..iiiiiiiooooooooooo notion. wwtiiionnsssseeeeeeeeeeeeeeeeeee witessttttssseeee Seyworr-OOpenReeeee facrees100bbSaa222204444466666
2025. Notion Blog.
[28] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprini urXiv:2203.03887 2021.
[29] Jian Hu. Reinforee++: A simple and efficient approach for aligning large language models. arXiv preprint 2tti:::5500.02262, 2025.
[30] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner- An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24299, 2025.
[31] HuggingFace. Open rl: A fully open reproduction of deepscek-rl. htttps//huggingfaace co/bloggopenn-r1 2025. Blog.
[32] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679 2024.
[33] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training. 2025.
[34] Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
[35] Gang Li, Wendi Yu, Yao Yao, Wei Tong, Yingbin Liang, Qihang Lin, and Tianbao Yang. Model developmental safety: A safety-centric method and applications in vision-language models. arXiv preprint arXiv:2410.039955. 2024.
[36] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi- Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:2310. 10505, 2023.
[37] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[38] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Brez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1eXivv:509.000 02971, 2015.
[39] Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-baser reasoning models. arXiv preprint arXiv:2503.22344, 2025.
[40] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078. 2025.
[41] Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa. and Ion Stoica. Deepeoder: A fully open-source 14b coder at 03-mini level. hrt.:4/.........aaaaaaaaeaaeaaea 2025.
Notion Blog.
13

---

[42] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta.
Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa. and lon Stoica. Deepscaler: Surpassing ol -reeviw with a 1.5b model by scaling rl. https://preatt--radio
b75. htw.aaaaseeeeeeeaaaaaeeeeeeaaeaa -rreriii- a-aithh---11 ---------------- Model-by-
Scaling-RL- RL- (tt.-000000022166660660333333100
2025. Notion Blog.
[43] Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin. and Hang Li. Reft: Reasoning with reinforced fine-tuning, arXiv preprint arXiv:2401.0880 1.89677 3, 2024.
[44] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-12422 2024.
[45] Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report slow-thinking reasoning systems. arXiv preprint arX1v:2412.09413, 2024.
[46] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PmLR, 2016.
[47] Youssef Mrouch. Reinforcement learning with verifiable rewards: Grpo's effective loss, dynam- ics, and success amplification, 2025.
[48] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501. 2025.
[49] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automation (ICRA), pages 7559-7566. IEEE, 2018.
[50] Hongseok Namkoong and John C. Duchi. Variance-basee regularization with convex objectives. In Proceedings of the 31 International Conference Neural Information Processing Systems, NIPS' 17, page 3975-2984, Red Hook, NY, USA, 2017. Curran Associates Inc
[51] OpenAI. Learning to reason with 1lms. https://openal) www/index//earnnnggeee beason-with. 11ms/ 2024. Blog.
[52] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[53] Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. imagination--augmenntd agents for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[54] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, (6:53728-53744 2023.
[55] Z. Z. Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, Z. F. Wu, Zhibin Gou, Shirong Ma, Hongxuan Tang, Yuxuan Liu. Wenjun Gao, Daya Guo, and Chong Ruan. Deepseck-provers mathematical reasoning via reinforcement learning for subgoal decomposition, 2025.
Advaneing formal
[56] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015,
14

---

[57] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:170.00647, 2017.
[58] Shai Shalev-Shwarts and Yonatan Wexler. Minimizing the maximal loss: How and why? CoRR, abs/1602.01699, 2016.
[59] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint X2ix:222422.3300000 2024.
[60] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815. 2017.
[61] Xuerui Su, Shufang Xie, Guoqing Liu, Yingce Xia, Renqian Luo, Peiran Jin, Zhiming Ma, Yue Wang, Zun Wang. and Yuting Liu. Trust region preference approximation: A simple and stable reinforcement learning algorithm for Ilm reasoning. arXiv preprint 41v:22504004544, 2025.
[62] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[63] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.
2025.
[64] Trieu H Trinh, Yuhuai Wu, Quoe Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations, Nature, 625(7995):476 482, 2024.
[65] Kaiwen Wang, Dawen Liang, Nathan Kallus, and Wen Sun. reductions approach to risk- sensitive reinforcement learning with optimized certainty equivalents, 2025.
[66] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances neural information processing systems, 35:24824-24837, 2022.
[67] Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503, 10460, 2025.
[68] Ronald J Williams. Simple statistical ment learning. Machine learning, 8:229-256, 1992.
algorithms for connectionist reinforce-
[69] Xiaomi LLM-Core Team. Mimo: Unlocking the reasoning potential of language model from pretraining posttraining, 2025.
[70] Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arX1v2245.000551, 2024.
[71] Huajian Xin, ZZ. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang. Xuan Lu, Qiushi Du, et al. Deepseek-prover-vl.. 5: Harnessing proof assistant feed- back for reinforcement learning and monte-carlo tree search. arXiv preprint arXiv:2408. 08152, 2024.
[72] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of I1 performance in machine translation. In International Conference on Machine Learning, pages 55204 55224. PMLR, 2024.
[73] Zaiyan Xu, Sushil Vemuri, Kishan Panaganti, Dileep Kalathil, Rahul Jain, and Deepak Ra- machandran. Distributionally robust direct preference optimization, 2025.
15

---

[74] Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. Dancegrpo: Unleashing grpo on visual generation, 2025.
[75] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[76] Tianbao Yang and Yiming Ying. Auc maximization in the era of big data and ai: A survey, 2022.
[77] Tianbao Yang and Yiming Ying. Auc maximization in the era of big data and ai: A survey. ACM computing surveys, 55(8):1-37, 2022.
[78] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:1 166:11809-18888 2023.
[79] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint Mix::25033 2025.
[80] Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3040-3049, 2021.
[81] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-1548, 2022.
[82] Jixiao Zhang and Chunsheng Zuo. Grpo-lead: A difficulty-aware reinforcement learn- ing approach for concise mathematical reasoning language models. arXiv preprint arXiv:250..09696 2025.
[83] Xiaojiang Zhang. Jinghui Wang. Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang. Yinghan Cui, Chao Wang, Junyi Peng, et al. Srpo: A cross-domain imple- mentation of large-scale reinforcement learning on Ilm. arXiv preprint arXiv:2504.14220 2025.
[84] Dixian Zhu, Gang Li, Bokun Wang, Xiaodong Wu, and Tianbao Yang. When AUC meets DRO: Optimizing partial AUC for deep learning with non-conved convergence guarantee. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Reseurch, pages 27548-27517 PMLR, 17-23 Jul 2022.
[85] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 190.088999 2019.
16

---

A More Experimental Results
For all the experiments on 1.5B models, each run consumes 4 40G A100 GPUs and each training step takes approximately 6 minutes. For all the experiments on 7B models, each run consumes 1*8 80G H100 GPUs and each training step takes approximately 6.5 minutes.
A.1 Detailed comparison between DisCO and DisCO-b
In this part, compare DisCO and DisCO-b with different score functions different models. As shown in Figure DisCO consistently demonstrates better performance compared to DisCO-b across all settings, with higher average scores observed in each case. This consistent advantage highlights the effectiveness of the full DisCO framework. Additionally, it is worth emphasizing that even the DisCO-b variants, with L-ratio log-I. scoring functions, outperform all other baseline methods that are presented in Table 2 and Table These results collectively underscore the robustness and general effectiveness of the DisCO approach.
A.2 Training dynamics for fine-tuning 8B model.
In this part, we present the training dynamics of different methods for fine-tuning the DeepSeek-RR 1- Distill-Llama-8B model Figure 6 Similar to observation in Figure for fine-tuning 1.5B and 7B models, we can see that GRPO, GRPO-ER, and Dr. GRPO still suffer from entropy collapse while DAPO leads to excessive entropy growth, all accompanied by premature saturation in training reward. TRPA with a KI. divergence regularization is also observed with instability in the training, indicating the insufficiency scoring functions and the KL constraint demonstrate the greatest stability, with training rewards continuing to rise and generation entropy remaining around 0.2.
KL regularization to stabilize training. In contrast, our methods with the two
A.3 Experiments on DAPO-Math-11 -7K dataset.
In order demonstrate that the improvements achieved by DisCO fundamental, rather than relying on specific properties of the dataset, we conducted additional experiments on the DAPO- Math-17K dataset 799 using 1.5B models, training them for 1400 steps. As shown in Table DisCO
05-
D
DsCO-s
Dicoo
DxCO
Daro
Pare01
1
so SaNtE MAT
20 200 MA
SANEE SANCE HATH
(a) L-ratio 1.5B
(b) log-L on 1.5
(c) L-ratio 7B
(d) log-L on 7B
Figure 5: Comparison between DisCO-b and DisCO on different models with different score functions.
GAPO-ER
DAPO
DisCO (L-ratiol
DisCO (log-L)
Generation Entropy
0.6
1.
0.5
Reward
0.65
0.4
250
500
153
3300
0.60
0.2
200
400
1000
0.0
200
400
600
800
1000
Step (a)
Step
(b)
Figure 6: Training dynamics of different methods for fine-tuning DeepSeek-HR R ----istill-LLaaaa model. (a) plots the training reward (averaged over generated outputs for questions used in each step) vs the number of training steps; (b) plots the generation entropy vs training steps.
17

---

Table Comparison with baseline methods for fine-tuning DeepSeek-RI-De CeepSeek-R1-Distilll Qwen- 1.5B models on DAPO-Math 17K dataset.
Model/Method
MRL(Train/Iee Test)
AIME 2024
AIME 2025
500H500
AMC 2023
Minerva
O-Bench
Avg.
Dos-Ditill-Qwee-H
32k+ 32k
0.288
0.263
0.828
0.629
0.265
0.433
0.451
DS DS--iittill Des-iiil-Qwee-l...
32k+ / 88
0.181
0.215
0.758
0.515
0.237
0.353
0.376
GRPO
8K / 8k
0.342
0,256
0.842
0.672
0.267
0.458
0.473
GRPO-ER
0.290
0.260
0.287
Dr. GRPO
8k 8k
0.300
0.250
0.849
0.705
0.292
0.464
0.477
DAPO
0.275
0.229
0.653
0.256
0.444
TRPA
8k 8k
0.346
0,2799
0.836
0.683
0.281
0.450
0.479
(L-rttio)
8k
0.413
0,310
0.775
0.307
0.495
DisCO (log-L)
8k /
0.460
0.317
0.873
0.775
0.320
0.502
0.541
methods still outperform other baselines by large margin, demonstrating the generalizability proposed method to other datasets.
B More Theoretical Results
B.1 Proof of Proposition
1
Proof Since Bown (99) = p(g), Var Varo-na(to)))ll = = p(()), we have
p(p))
ifr(olg) = =1,
A(ola)
plg)
(12)
p(g)
ifr(olg) = 0
1-p(
According to 7. we have
To(0olu; 0c1)
To oce) A(olg))
By Bowssa(-11))
1
lol
mo(uely, Uc))
By p(() Fou
A(oog)
Tour(oelq
To(orl9. q, 021)
A(olg) ]
+ - -L(() (
Totalor q oee)
1=1
lol
Te(oelg.0c1)
1 1-p(q)
E.g p(q) om (t (:10)
Σ
(13)
Toda Oze)
p(g)
T0(0r Oct)
p(g)
+ (1 -i(())) on
Tota
V
p(q)
lol
To(oela, Oce)
E, vvp(()) p(a))
E BoNT
Tota(Orlg, Ocz)
To(orlq, 0ce) 1)
E D T 1)
Totd 0z0)
where the last equality is due to the assumption about f(x,y). For GPRO, we have f (xx = min(x, clip(æ, e, + e)) min(x, 1 e) and f (xx max(x, e, E))
max(x, 1-6)
18

---

Algorithm Discriminative Constrained Optimization
1: Input: Initial policy model To, reward function r, question set D, hyperparameter 6, 3,
2: Policy model Fe 7T0
3: for Step 1, ... T" do
4: Sample batch of questions B from
5: Update the old policy model Vold TTg
6: For each question q E B, sample responses {o: [0:):_1 (ola(-19) denoted by Sg and partition it into S and S, based on rewards r(o:1g) E {0. for minibatch B, E B do
7:
8:
Compute KL divergence estimator by
DKL EqEB... Zoes, 10
Σ
Σ
log (oe mo to
BE DES, Compute gradient estimator of T2(0) by
E
9:
G1
Σ
Vso(o, ( (Tlog log E exp( (0000000000
DES
o'ESy
Compute gradient estimator constraint by G2 = 2/3 KL 81+ VT KL
11:
Update To with Adam-W using the gradient estimator G = G1 G2
12: end for
13: end for
B.2
Connection between discriminative objectives and surrogate objectives in RL
The score function L-ratio is inspired by the same principle as the surrogate objective TRPO TRPO aims maximize the following objective subject to constraint:
56
lol
To(oelg, oct))
maxE, Eo wes (11)
Tata ocs) A(ox)
0
(14)
s.t. DKL (
[Talal (T)) S 8.
When we apply the advantage function
12
to the objective, we have
xo(o:19,0cc) A(olg)
Eg BoNTToal))
(or 4: O.e))
lo To(oela, 0c)) A(olg)
= =Eg p(g)) E om (111) lo|
Toul 0cr)
20(0r1,,0c1 A(olg)
+ (1 11--poa 1-p(()) onn (20)
Tota 19, oze)
To(0:l9,0aa)
1 - 1-p(g)
Eg p(g) E F (-1g)
Told( a<c)
p(g)
t=1
Te q, Oze)
p(g)
+ (1 - 11p(p)))B
DMt (1))
To 1g, ozc))
1 1-p(g)
lol mo(orla,0ce oce) Tol 021)
Exvp(g) (Ip(g))
B
Bann (11))
om Comm (111) lo
(ou (z.
(15)
This gives the exact scoring function L-ratio: so 1/ Del xo(orlg,ec))
Faulor q.0cs) After removing the improper weight (1( p())), Eqn. (15) is same Eqn. with {(s)
In the reinforcement learning literature, vanilla policy gradient methods also gain significant attention due to their simplicity and remarkable performance. The vanilla policy gradient(VPP) methods work
19

---

Table 6: Weighted discriminative objectives and their scoring functions 8 st g) and & (o, q) for different methods, where of.) is the sigmoid function.
Objective
F a- tallllo (2, (o.0) -6 (dl9))
GRPO
[(g) VPlall p(g]]. ((s)
%6 (o. g) t) Enll mini Bela keesl
55(d. 1o1 Elin maxt nololleaial
Rad 9.
Dr. GRFO
[g
P(0i1 plal. [6)
56 (o.g)) En min! mateeleee
1 c).
57(0'.)) Emmi maxi malol
DAPO
w(g)
Vplaad - -(g)). ((s)
% (b.
of min( Relon meel
caign)
% (v/.g) Fe-netiin
maxd Paatoll (1 teow
GPG
aplokl
p(a). [())
(0,9) Swepeinll log seloollgg.o
45 = Rasaauill Enll log me (ollg,
TRPA
En Tog (oloules.))
(0'.9) Ex11 mlllh
by computing an estimator of the policy gradient and plugging it into a stochastic gradient algorithm. The most commonly used surrogate objective function for gradient estimator has the form:
lol log To (oelo
JvPG = ByEoowaa (1)
(16)
[o|
Similar to the derivation above, by plugging in the advantage estimator Eqn. 12
we have:
Jvre EyVplg)(1 p(a))
B BoNn t2
log no (oel,, 0ct)
|o|
(17)
lo
E onn (1 (-19)
log (or 10 (0:1),0)))
|o
t
This directly motivate the score function log-I.: sp(o, = To
Σ oo
log Tog Rct). After remov-
ing the inappropriate weight on questions, Eqn. (17) same as Eqn. with ((s) =s. B.3 Analysis of other variants of GRPO In this part, we show that the other variants of GRPO still have difficulty bias on questions. Let's start with Dr. GRPO. In Dr. GRPO, the un-normalized advantage function is employed:
A(olg) = 1 - p(g), r( = 1,
(18)
pp(()) rl = 0
With f(z,v) min(ry, clip/ e, + c)y), we have
lol
To o2e)
Acolo) ]
JDAGRPO = =1E, Bownau
Tota( Oce)
= Rgg p(g) om (11))
To q, oze)
A(olg)
Totalor ocx)
To(oely, ace)
A(ol))
(1 p(g))
F. B
Touu
(19)
[o]
no To(oi yg, (2e) 1 p(q) 1 Tota(or oce) 021))
= Eg
p(g) 18 dd
To(ocl9, oze)
+ (1 - (1-(())) out a (11)
Tota(Orlq, oze)' p(4)
= 1E,p(g) p(q)) E (ow (o, q) - E E ( (o,
20

---

molor + 1), = E max( matad 1 - e). We where of (0. E min( g. can see that the difficult bias w()) = p(a) p(4)) on questions persists in Dr. GRPO.
Secondly, let's reformulate the DAPO objective to show the question-level bias. With f(z, y) = min(xy, clippf; 1 clow: thigh )y) and advantage estimator
(12). the expected version of DAPO
is
lol
To(or, Oce) (ata oci)
A(ola))
TDAPO Bowtret 9) Bownuu(tt
To(oelg, 0ct) A(olg)
Eg p(q) on (1 (-19) (o Bonnou(--a)l
Tota act)
To(orl9, oct)
A(olo))
+ (1 -1-110 -(()) owa Bownaalloa))
Told ocee)
To (orlo, oce)
1 1-p(g)
p(g))
Eg p(a) om Bownou((la))ll 10|
(ota oct)
1
p(g)
Te(0elg,
p(q)
)
+ - -1((()))BB con Eawnas
Tota(orlq, ozi)
V (1pp()))
B. (Bg)(()(1 p(())]
E Bount S/ (0.
EE 1m (110 (0.g)
(20)
where 8 (0, Burnan volo| Dael min( To(ai 1 c), Sg (0, lo: Σ
1
uu (v. (g,
mo(or q,0cu) <). We can see that the difficult bias w(g) = v/pl)) (1(()()-p(g) is placed on max( narto questions persists in DAPO.
Thirdly, we show the difficult bias in GPG objective. In GPG, aA(olg)) q) is employed as their advantage estimator. Thus, we have
Jo]
JGPo =B,Bor Eownaae-la))
a log No To(o, q, 0x())())
lol
Egg p(y) Bo omm Bownsaul-
a log no (c))
Jo
+ (1 (1-p(g))B p(9)) Bo (1)0)) (11 B 12)
a log (
(21)
lol
Eg P(g) E (119) Bownaa
a log (0((010 (1-p(c))
1=1
+ (1 - -r(())BBBB (01 Bownaal-
a log (ogoooo) (-p(a))
= =E - p(a))
Bown (11) (o,
(0,
E Bown (o.4)
where
80 (o,4)
=
Bonnaetionlo EPe log ne (or (a, occ) and sp (0,
Eannant |o E log To (o(0llg, O. We can that the difficult bias w(() op(a)(1 p(4)) is on questions in GPG.
Finally, summarize the question-level weights and their score functions for other variants of GRPO in Table G
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions are clearly stated in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
The limitations of the work are discussed in the last section of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification:
All assumptions are clearly stated and proofs are provided in Appendix.
22

---

Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detailed instructions reproduce the results are provided in the experimental part of the main paper. Furthermore, we provide source code in the supplementary materials for reproducing the algorithm.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification: Source code is provided in the supplementary materials for reproducing the proposed algorithm.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
aubl:c//guidee/Ddeeellnniisss/// iooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https /mips..cc/pullc//uuui guwdes/ddee/e/e Coc//ooooeeessseeeeeeo hwtpibbmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed information about experimental settings and hyperparameters is provided in the experimental part of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[No
Justification: We follow the practice in this research area. Error bars are not reported because it would be too computationally expensive for training large language models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided the experimental platform and running time of one experiment for our algorithm Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: Research conducted in the paper conform with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We haven't identified any social impacts of this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[NA]
Justification:
This paper proposes an algorithm without posing such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Datasets and methods utilized in this paper properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
27

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The LLM is used only for writing and editing purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurd/: //taaarrppp.c//o/oeeeeer//////// for what should or should not be described.
28


Reviews:
[{"review_id": "nZv1Rqv6Rk", "reviewer": "Reviewer_2CBw", "summary": "This paper proposes several modifications to GRPO:\n- Does not clip advantage\n- Removes down-weighting of questions with very high or low reward\n- More stable KL divergence term using a squared-hinge penalty function \n\nThey train and compare their method (Disco) with GRPO, Dr. GRPO, DAPO, TRPA. They also compare against Deepseek-R1-1.5B PReview and OpenAI-o1-Preview. Disco trained model outperforms these variants on math tasks.", "questions": "- I'm curious what the interplay is between the expected reward and entropy collapse in your algorithm. These concepts seem to be closely related, since entropy collapse would also lead to a bimodal expected reward distribution (e.g. you could be oversampling one particular incorrect or correct answer and get extremely low/high rewards, respectively). If you reduce entropy collapse, I imagine less datapoints would have extremely low or high rewards. Yet Disco would also precisely focus more on these outlier examples. I wonder if some effects of these two design choices are somewhat cancelling each other out? \n- If you remove the clipping for GRPO + use the same squared-hinged KL  term, does the performance improve?", "limitations": "- Based on the insights provided, are there any particular datasets you would foresee Disco to be particularly effective or suboptimal in comparison to GRPO? e.g. perhaps you can try comparing these algorithms on  datasets with a small number of very easy and hard examples.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "- [Good performance] Based on numbers reported, it looks like the algorithm works well. \n- [Organization needs some work. Contributions are weakly communicated --- multiple things happening with the algorithm, might be good to decompose the paper into separate sections] There seems to be two things the paper is trying to communicate: that GRPO is downweighting examples with extreme rewards + standard clipping/KL divergence term causes instability. However, in the experiments, it's very unclear where the gains are actually coming from. It might be good to more closely decompose the effect of each design choice to the performance in the main body + provide more extensive ablation studies that demonstrate the problem tackled. \n- [Concurrent work] The idea of upsampling examples with the highest and lowest rewards is also explored in concurrent work Not All Rollouts are Useful [https://arxiv.org/abs/2504.13818]. \n- [No hyperparameter tuning for baselines] Based on the experimental section, it looks like hyperparameters are manually tuned for their algorithm, but baseline hyperparameters are reported numbers adapted from previous papers. It's not clear whether these hyperparameters are optimal for their setup.", "was_revised": true, "final_justification": "From what I understand, clipping stabilizes training but performance degrades. Doing non-clipping but replacing the KL regularization term to stabilize training achieves better performance.\n\nI think the experiments addresses my concerns. I increase my score to 5.\nI think the organization of the paper could possibly improve to communicate why changing each component of GRPO was necessary.", "rebuttal": "We thank the reviewer for providing a comprehensive review. Below we would like to address raised concerns.\n\n**Q1:** In the experiments, it might be good to more closely decompose the effect of each design choice on the performance. \n\n**A**:  We thank the reviewer for the constructive suggestion! The proposed method incorporates three key design choices:\n(1) removing question-level weight bias,\n(2) using a non-clipping scoring function, and\n(3) applying a KL-divergence constraint.\n\nThe effectiveness of (1) has been demonstrated in the comparison between GRPO and GRPO$_\\text{rw}$, as shown in Fig. 1 (c, d). Additionally, we conducted new experiments on training a 1.5B model, comparing DisCO with it variant that includes question-level weight bias while keeping all other components unchanged. The results are presented in the table below.\n\nThe benefit of (2) is validated in Figure 3 (second from the left), which shows that using the non-clipping scoring functions yields significant improvements over its clipping-based counterpart, when all other components are the same.\n\nTo evaluate the impact of (3), we conducted another experiment on the 1.5B model, comparing DisCO with a variant that replaces the KL constraint with KL-divergence regularization. The results are also summarized in the table below.\n\nWe can see that each component contributes to the improvements, where the use of non-clipping scoring function contributes the most. We will incorporate the discussion into our revision.\n\n| Method                     | [pass@1(Avg.)](mailto:pass@1(Avg.)) | Method                     | [pass@1(Avg.)](mailto:pass@1(Avg.)) |\n| ------------------------- | -----------------------------------  | ------------------------- | ----------------------------------- |\n| DisCO (log-L)             | 0.533                                | DisCO (L-ratio)           | 0.524                               |\n| DisCO --> w/ weight bias            | 0.522 (↓ 0.011)            | DisCO --> w/ weight bias             | 0.511  (↓ 0.013)          |\n| DisCO -->Clipping scoring  func. | 0.430  (↓ 0.103)              | DisCO --> Clipping scoring func.| 0.430  (↓ 0.094)                |\n| DisCO--> KL Regularization          | 0.519    (↓ 0.014)                   | DisCO -->  KL Regularization         | 0.523 (↓ 0.001)             |\n\n\n**Q2:** The idea of upsampling examples with the highest and lowest rewards is also explored in concurrent work [r1].\n\n\n**A:** Thank you for pointing out the interesting work. While [r1] introduces max variance down-sampling to train on only an informative subset, we'd like to clarify that our method **does not** apply up-sampling or down-sampling to rollouts. Instead, we proposed a discriminative objective using all generated rollouts for more effective learning than GRPO. It would be interesting to integrate [r1]'s sampling approach into our framework. \n\n\n**Q3:**  Based on the experimental section, it looks like hyperparameters are manually tuned for their algorithm, but baseline hyperparameters are reported numbers adapted from previous papers. It's not clear whether these hyperparameters are optimal for their setup.\n\n**A:**  Thank you for the thoughtful comment! We would like to provide further clarification on our hyperparameter tuning process.\n\n**First**, we tune the learning rate for **all methods** from the set $\\{1\\text{e}{-6}, 2\\text{e}{-6}\\}$ (see lines 310–311).\n\n**Second**, for our proposed DisCO methods, we tune only one additional parameter, $\\tau$, over a small range of three values per variant. The other hyperparameters, $\\beta$ and $\\delta$, are selected based on heuristics and are not tuned (lines 318-319). Our ablation study (Figure 3, right) shows that the performance is not sensitive to the choice of $\\tau$.\n\n**Third**, for the baseline methods GRPO, GRPO-ER, TRPA, we follow the hyper-parameter settings used in prior works [r2, r3, r4] because they have used exactly the same model and dataset as our experiments. Dr. GRPO does not introduce any additional hyperparameters beyond the learning rate. For DAPO, we used $\\epsilon_{\\text{high}} = 0.28$, as in the original paper. While we acknowledge that some studies may use a different value, we tested an alternative setting $\\epsilon_{\\text{high}} = 0.24$ to address this concern. The result, shown in the table below, shows that $\\epsilon_{\\text{high}} = 0.24$ performs even worse than $\\epsilon_{\\text{high}} = 0.28$.\n\nIn summary, the improved performance of **DisCO** does not stem from extensive or unfair hyperparameter tuning.\n\n| Model           | AIME 2024 | AIME 2025 | MATH 500 | AMC 2023 | Minerva | O-Bench | Avg.  |\n| --------------- | --------- | --------- | -------- | -------- | ------- | ------- | ----- |\n| GRPO            | 0.277     | 0.242     | 0.838    | 0.647    | 0.276   | 0.462   | 0.457 |\n| DAPO($\\epsilon_{high}$ =0.24)       | 0.292     | 0.246     | 0.838    | 0.665    | 0.288   | 0.464   | 0.465 |\n| DAPO($\\epsilon_{high}$ =0.28)        | 0.310     | 0.252     | 0.848    | 0.675    | 0.296   | 0.456   | 0.473 |\n| DisCO (log-L)   | 0.404     | 0.317     | 0.876    | 0.758    | 0.333   | 0.509   | 0.533 |\n\n\n**Q4:**  On the interplay  between the expected reward and entropy collapse in your algorithm. Are there less datapoints would have extremely low or high rewards in DisCO?  Would Disco focus more on these outlier examples?\n\n**A:** We would like to clarify a misunderstanding regarding DisCO. We agree that **entropy collapse** did contribute to saturation of expected rewards for GRPO, GRPO-ER and Dr. GRPO as observed in Fig. 2. Regarding DisCO, \n- **(1)** the generation entropy remains stable while the expected reward keeps increasing (see Fig. 2); \n- **(2)** We verified the reward distribution for questions in our method averaged over the first 200 steps. As shown in the following table, it does not exhibit fewer datapoints with extremely low or high rewards. It is similar to that in GRPO (Fig. 1, second from the left). \n- **(3)** It is **not true** that DisCO focuses more on the examples with extremely low or high rewards.  Indeed, DisCO is motivated by giving the same weight to all questions regardless of their expected rewards. Our objective in (10) has an effect of weighting negative rollouts for each question, focusing more on hard negative rollouts.  \n\n\n| Question Accuracy | 0%   | (0, 25%] | (25%,50%] | (50%,75%] | (75%,100%) | 100% |\n| ----------------- | ---- | --------- | --------- | --------- | ---------- | ---- |\n| Ratio             | 0.24 | 0.13      | 0.11      | 0.13      | 0.1        | 0.29 |\n\n\n**Q5:** If you remove the clipping for GRPO + use the same squared-hinged KL term, does the performance improve?\n\n**A**: We have experimented with the suggested variant of GRPO, keeping its advantage function, but replacing its clipping-based scoring function with non-clipping based L-ratio, and using the squared-hinge KL penalty instead of its original KL regularization. This indeed is the variant of the second row (right) in the first Table of this rebuttal. We summarize the results below, from which we can see that its average performance 0.511 is better than GRPO (0.457) but still worse than DisCO (0.524). \n| Model            | [pass@1(Avg.)](mailto:pass@1(Avg.)) |\n| ---------------- | ----------------------------------- |\n| GRPO             | 0.457                               |\n|  No Clipped GRPO + squared-hinged KL | 0.511                               |\n| DisCO (L-ratio)  | 0.524                               |\n\n\n**Q6:** Based on the insights provided, are there any particular datasets you would foresee Disco to be particularly effective or suboptimal in comparison to GRPO? e.g. perhaps you can try comparing these algorithms on datasets with a small number of very easy and hard examples.\n\n**A:** The DeepScaleR-Preview-Dataset used in our experiments includes a wide range of questions, spanning from easy to challenging for the base models. We believe that the improvements achieved by DisCO are fundamental, rather than relying on specific properties of the dataset. To further support this claim, we conducted additional experiments on the DAPO-Math-17K dataset [r5] using 1.5B models, training them for 1400 steps. As shown below, DisCO still consistently outperforms GRPO and other baselines.\n\n| Model                | MRL(Train/Test) | AIME 2024 | AIME 2025 | MATH 500 | AMC 2023 | Minerva | O-Bench | Avg.  |\n| -------------------- | --------------- | --------- | --------- | -------- | -------- | ------- | ------- | ----- |\n| DS-Distill-Qwen-1.5B | 32k+   /   8k   | 0.181     | 0.215     | 0.758    | 0.515    | 0.237   | 0.353   | 0.376 |\n| GRPO                 | 8k   /    8k    | 0.342     | 0.256     | 0.842    | 0.672    | 0.267   | 0.458   | 0.473 |\n| DAPO                 | 8k   /   8k     | 0.275     | 0.229     | 0.812    | 0.653    | 0.256   | 0.441   | 0.444 |\n| TRPA                 | 8k   /   8k     | 0.346     | 0.279     | 0.836    | 0.683    | 0.281   | 0.450   | 0.479 |\n| DisCO (L-ratio)      | 8k   /   8k     | 0.413     | 0.310     | **0.874**    | **0.775**    | 0.307   | 0.495   | 0.529 |\n| DisCO (log-L)        | 8k   /   8k     | **0.460**     | **0.317**     | 0.873    | **0.775**    | **0.320**   | **0.502**   | **0.541** |\n\n\n[r1] Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning\n\n[r2] An empirical study on eliciting and improving r1-like reasoning models.\n\n[r3] Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl.\n\n[r4] Trust region preference approximation: A simple and stable reinforcement learning algorithm for llm reasoning.\n\n[r5] DAPO: An Open-Source LLM Reinforcement Learning System at Scale."}, {"review_id": "GQ1clK5YRc", "reviewer": "Reviewer_Hi9A", "summary": "The paper proposes a new Discriminative Constrained Optimization framework for reinforcing large reasoning models (LRMs). Starting from group relative policy optimization (GRPO), DisCO firstly replace the group relative objective with a discriminative objective defined by a scoring function, and employs a constrained optimization approach to enforce the KL divergence constrainet. Experimental results show that DisCO outperforms GRPO and DAPO on 1.5B and 7B model with the same training queries.", "questions": "1. The equation 9 seems to have very similar formulation as Direct Policy Optimization. Is there any essential connection and difference between the proposed DisCO and online DPO?\n2. The reward distribution of training queries are highly correlated with the initial policy and the training data. If the initial policy is strong, and the training queries are hard (for example, using the queries provided by DAPO[1] or OREAL[2], is DisCO still effective and better than previous methods?\n[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k\n[2] OREAL: Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning.  https://huggingface.co/datasets/internlm/OREAL-RL-Prompts", "limitations": "See weaknesses.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. From Table 2&3, the proposed method can outperform previous works on both 1.5B and 7B models and exhibits better stability in training than previous works.\n\nWeaknesses:\n1. There are too many equations that make the paper hard to follow, especially there are many symbols that do not have explaination and requires the readers to guess. For example, function $g(o,q)$ is a one time symbol that can be essentially replaced by $f(x, y)$, $f^{+}$ and $f^{-}$ can be omitted. There are also many abbreviations that show up without explanation, such as VPG, GRPO_RW in the equations.\n2. The claimed motivation that previous methods have difficulty bias seems to be a bit emperical and does not have theoretical support.\n3. In Table 3, when the initial policy is strong, the improvements of DisCO is small, and the baseline methods such as GRPO and DAPO even decrease the performance of initial policy, is there any explanation?", "was_revised": true, "final_justification": "The authors' response and clarification have addressed my concerns, thus I raised my rating from 4 to 5.", "rebuttal": "We thank the reviewer for providing constructive comments. Below we would like to answer the raised questions.\n\n**Q1:** On the math symbols that do not have explanation. For example, function $g(o,q)$ is a one time symbol that can be essentially replaced by $f(x,y)$, $f^+$ and $f^-$ can be omitted. There are also many abbreviations that show up without explanation, such as VPG, GRPO_RW in the equations.\n\n**A:** We apologize for the confusion!\n- $g(o,q)$ in line 134 is any function, introduced for a statement of a general fact in equation (1), which is used in the proof of Proposition 1. We will move this to the appendix in the revision. \n- $f^+$ and $f^-$ in Proposition 1 are assumed to be non-decreasing functions for decomposing $f(x, y)$. Their specific forms for GRPO are actually the form in $s_{\\theta}^+(o, q)$ and $s_{\\theta}^-(o, q)$ in Proposition 1, i.e.,  $f^+(x, 1)=\\min(x, 1+\\epsilon)$ and $f^-(x,1)=\\min(x, 1-\\epsilon)$.   For other methods, their specific forms can be found in Table 4. We will make it  clearer in the revision. \n\n- VPG stands for Vanilla Policy Gradient method, and GRPO_RW stands for a GRPO version with the question-level weight bias removed (line 183). We will revise the paper to make the term clearer.\n\n\n\n\n**Q2:** The claimed motivation that previous methods have difficulty bias seems to be a bit empirical and does not have theoretical support.\n\n**A:** We politely disagree. Our finding regarding the difficulty bias of GRPO is through a theoretical analysis of GRPO's objective by connecting it with a discriminative objective in Proposition 1. The empirical studies in Figure 1 are used for further supporting our claim. \n\n\n**Q3:** In Table 3, when the initial policy is strong, the improvements of DisCO is small, and the baseline methods such as GRPO and DAPO even decrease the performance of initial policy, is there any explanation?\n\n\n**A:** There is a misunderstanding in the interpretation of comparison with the initial policy in Table 3.   It is important to compare their performance against the **second entry** in Table 3: *DS-Distill-Qwen-7B (32k+/8k)*, which represents the initial policy evaluated with the same 8k response length used for GRPO, DAPO and DisCO. The **first entry**, *DS-Distill-Qwen-7B (32k+/32k)*, uses a much longer 32k response length at inference and is included as a reference to highlight DisCO’s efficiency under **smaller generation budgets** (8k during both training and inference).\n\n\nIn fact, both GRPO and DAPO **do improve** upon the initial policy when evaluated under the **same response length constraint**—raising performance from 0.513 to 0.592 (GRPO) and 0.570 (DAPO), respectively. Also, the improvements of DisCO are still significant, from 0.513 to 0.627 or 0.625.\n\nPlease also note that while the performance of GRPO and DAPO has already saturated, DisCO continues to exhibit an upward trend as the number of training steps increases (Figure 2c).\n\n\n**Q4:** The equation 9 seems to have very similar formulation as Direct Policy Optimization. Is there any essential connection and difference between the proposed DisCO and online DPO?\n\n**A:** Indeed DPO's objective can be regarded as a discriminative objective and can be recovered in (9) by choosing $s_{\\theta}(o,q) = \\beta\\log\\frac{\\pi_{\\theta}(o|q)}{\\pi_{ref}(o|q)}$, $\\ell(s) = \\log(1+\\exp(-s))$, and removing the squared hinge penalty function. Nevertheless, the objective in (10) for DisCO is different from DPO, as it has an effect on weighting more on hard negative rollouts. \n\n\n\n\n**Q5:**  Is DisCO still effective and better than previous methods on other datasets like those provided in DAPO or OREAL papers?\n\n**A:** We thank the reviewer for the constructive comment. We conducted additional experiments on DAPO-Math-17k dataset with 1.5B models, training them for 1400 steps. The results are summarized below. We can observe that DisCO methods achieve slightly better performance than those trained on DeepScaleR-Preview-Dataset, consistently outperforming GRPO and other baselines. \n\n| Model                | MRL(Train/Test) | AIME 2024 | AIME 2025 | MATH 500 | AMC 2023 | Minerva | O-Bench | Avg.  |\n| -------------------- | --------------- | --------- | --------- | -------- | -------- | ------- | ------- | ----- |\n| DS-Distill-Qwen-1.5B | 32k+   /   8k   | 0.181     | 0.215     | 0.758    | 0.515    | 0.237   | 0.353   | 0.376 |\n| GRPO                 | 8k   /    8k    | 0.342     | 0.256     | 0.842    | 0.672    | 0.267   | 0.458   | 0.473 |\n| DAPO                 | 8k   /   8k     | 0.275     | 0.229     | 0.812    | 0.653    | 0.256   | 0.441   | 0.444 |\n| TRPA                 | 8k   /   8k     | 0.346     | 0.279     | 0.836    | 0.683    | 0.281   | 0.450   | 0.479 |\n| DisCO (L-ratio)      | 8k   /   8k     | 0.413     | 0.310     | **0.874**    | **0.775**    | 0.307   | 0.495   | 0.529 |\n| DisCO (log-L)        | 8k   /   8k     | **0.460**     | **0.317**     | 0.873    | **0.775**    | **0.320**   | **0.502**   | **0.541** |"}, {"review_id": "fX2M005mhh", "reviewer": "Reviewer_CL8D", "summary": "This paper addresses the problem difficulty bias and training instability in existing Large Reasoning Models (LRMs) during reinforcement learning by proposing a novel framework. The study optimizes scoring functions to enhance scores for correct answers while suppressing those for incorrect answers, employing a constrained optimization approach to ensure training stability. Additionally, it introduces Distributionally Robust Optimization (DRO) techniques to mitigate data imbalance issues during training. Experiments demonstrate that the proposed architecture outperforms existing GRPO and its variants on reasoning tasks, significantly enhancing model performance.", "questions": "1.Could you quantify the individual contribution rates of each improvement in DisCO through ablation studies in detail?\n2.Will the DisCO algorithm pose computational efficiency challenges as models scale further?\n3.In hyperparameter selection, have you accounted for the impact of different hyperparameter combinations on model generalization capabilities?", "limitations": "Yes，the authors have explicitly stated several limitations of their work in the paper, noting that their method primarily \noptimizes for binary rewards and that experiments were constrained by computational resources, covering only 1.5B and 7B models.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1.The paper offers detailed theoretical analysis and experimental validation, exposes GRPO's limitations, and presents targeted solutions. The well-designed experiments cover multiple benchmark tasks and various model sizes. The results show that the new architecture in the paper is better than existing methods in improving model performance. Also, the paper discloses enough experimental details for other researchers to reproduce the experiments.\n2.This study is highly significant for enhancing the reasoning abilities of large - scale reasoning models in complex tasks. By addressing key issues in current methods, it offers a more effective and stable model optimization approach, which will advance research in related fields.\n\nWeaknesses\n1.In the experimental part, although the paper provides a large number of experimental results and analyses, the descriptions of certain experimental details may not be in-depth enough, such as the selection and tuning process of hyperparameters.\n2.When presenting related work and introducing the DisCO method, the paper emphasizes the limitations of GRPO and its variants but rarely mentions their merits and applicability in specific scenarios. This may bias readers' understanding of different methods, preventing them from objectively evaluating the relative advantages and disadvantages of each approach in various contexts.", "was_revised": true, "final_justification": "Thanks for the authors' thorough and clear responses. All my questions have been fully addressed, and I have no further comments. Therefore, I recommend acceptance.", "rebuttal": "We thank the reviewer for providing helpful comments on improving our paper. Below we would like to answer raised questions.\n\n**Q1:** In hyperparameter selection, have you accounted for the impact of different hyperparameter combinations on model generalization capabilities?\n\n\n**A:** We tuned two hyper-parameters for DisCO, i.e., the learning rate and the $\\tau$. The process of hyperparameter selection is the same as model selection, which accounts for the generalization performance. We report the performance of the best hyper-parameters in the tuning range that yields the best average test performance. We did the same model selection for all baselines. We will add more details in the revision. \n\n\n**Q2:** Mentioning the merits and applicability of GRPO in specific scenarios. \n\nA: We thank the reviewer for the constructive comment. In the opening paragraph of the Introduction, we highlighted the importance of GRPO in achieving performance comparable to advanced proprietary models across many reasoning benchmarks. Additionally, in the second paragraph of the Related Work section, we discussed key contributions of GRPO and its variants to provide readers with a foundational understanding of each approach.\n\nWe agree with the reviewer that each method has its own strengths. In Section 7, we acknowledged the limitation of our work in focusing on binary rewards, whereas GRPO and its variants can naturally handle non-binary reward signals without modification. We will make this clearer in the revision.\n\n\n\n**Q3:** Could you quantify the individual contribution rates of each improvement in DisCO through ablation studies in detail? \n\n**A:** Thank you for the helpful suggestion. The proposed method incorporates three key design choices:\n(1) removing question-level weight bias,\n(2) using a non-clipping scoring function, and\n(3) applying a KL-divergence constraint.\n\nThe effectiveness of (1) has been demonstrated in the comparison between GRPO and GRPO$_\\text{rw}$, as shown in Fig. 1 (c, d). Additionally, we conducted new experiments on training a 1.5B model, comparing DisCO with its variant  that includes question-level weight bias while keeping all other components unchanged. The results are presented in the table below.\n\nThe benefit of (2) is validated in Figure 3 (second from the left), which shows that using the non-clipping scoring functions yields significant improvements over its clipping-based counterpart, when all other components are the same as DisCO.\n\nTo evaluate the impact of (3), we conducted another experiment on the 1.5B model, comparing DisCO with a variant that replaces the KL constraint with KL-divergence regularization. The results are also summarized in the table below.\n\nWe can see that each component contributes to the improvements, where the use of non-clipping scoring function contributes the most. We will incorporate the discussion into our ablation studies.\n\n| Method                     | [pass@1(Avg.)](mailto:pass@1(Avg.)) | Method                     | [pass@1(Avg.)](mailto:pass@1(Avg.)) |\n| ------------------------- | -----------------------------------  | ------------------------- | ----------------------------------- |\n| DisCO (log-L)             | 0.533                                | DisCO (L-ratio)           | 0.524                               |\n| DisCO --> w/ weight bias            | 0.522 (↓ 0.011)            | DisCO --> w/ weight bias             | 0.511  (↓ 0.013)          |\n| DisCO -->Clipping scoring  func. | 0.430  (↓ 0.103)              | DisCO --> Clipping scoring func.| 0.430  (↓ 0.094)                |\n| DisCO--> KL Regularization          | 0.519    (↓ 0.014)                   | DisCO -->  KL Regularization         | 0.523 (↓ 0.001)             |\n\n\n**Q4:** Will the DisCO algorithm pose computational efficiency challenges as models scale further?\n\n**A:** The DisCO algorithm does **not** introduce any extra computational overhead, compared with GRPO and its variants. They have similar computational costs."}, {"review_id": "WdNazxP3iR", "reviewer": "Reviewer_ewTG", "summary": "The authors propose a new reinforcement learning (RL) optimization framework for large reasoning models based on discriminative learning. Specifically, it employs a score-based discriminative objective instead of a group-relative policy and uses a scoring function in place of clipping. Training is stabilized through a simple constrained optimization method. The proposed framework demonstrates strong potential across multiple reasoning benchmarks, achieving better performance than GRPO while producing shorter responses.", "questions": "- What happens if we increase the response length (e.g., 24k or 32k)? Does it improve DisCO's performance notably?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**[Strengths]**\n- The paper is well-written and easy to follow.\n- The proposed framework to alleviate difficulty bias is interesting and achieves meaningful results across multiple benchmarks.\n- The ablation study is detailed and comprehensive.\n\n**[Weaknesses]**\n- The framework only considers binary rewards (though the authors acknowledge this). It would be more compelling if it were shown to work with non-binary reward settings as well.\n- The paper misses an ablation study on response length. It would be helpful to understand how DisCO's performance varies with different response lengths.", "was_revised": true, "final_justification": "The authors addressed my concerns. I will increase my score (4-->5).", "rebuttal": "We thank the reviewer for dedicating the time to providing helpful feedback on our paper.\n\n**Q1:** The framework only considers binary rewards (though the authors acknowledge this). It would be more compelling if it were shown to work with non-binary reward settings as well.\n\n**A:** We agree that it will strengthen the work. However, the focus on the binary reward setting helps us to better illustrate the limitation of GRPO through theoretical analysis. We believe the improvement brought by DisCO is non-trivial and expect to extend its strength to non-binary rewards by considering other discriminative objectives, e.g., ranking losses. \n\nAdditionally, to make our work more compelling, we further validated our method on a model from another LLM-family, i.e., DeepSeek-R1-Distill-Llama-8B, training them for 1000 steps. The results are summarized below. We observed that our proposed DisCO methods still  outperform other baselines by a large margin, demonstrating the generalizability of the proposed method to other models.\n\n| Model               | MRL(Train/Test) | AIME 2024 | AIME 2025 | MATH 500 | AMC 2023 | Minerva | O-Bench | Avg.  |\n| ------------------- | --------------- | --------- | --------- | -------- | -------- | ------- | ------- | ----- |\n| DS-Distill-Llama-8B | 32k+   /   32k  | 0.506     | 0.346     | 0.896    | 0.815    | 0.295   | 0.541   | 0.566 |\n| DS-Distill-Llama-8B | 32k+  /   8k    | 0.348     | 0.238     | 0.825    | 0.652    | 0.267   | 0.440   | 0.462 |\n| GRPO                | 8k   /    8k    | 0.410     | 0.240     | 0.873    | 0.759    | 0.307   | 0.506   | 0.516 |\n| GRPO-ER             | 8k   /   8k     | 0.408     | 0.277     | 0.882    | 0.785    | 0.311   | 0.511   | 0.529 |\n| Dr. GRPO            | 8k   /   8k     | 0.423     | 0.285     | 0.867    | 0.786    | 0.300   | 0.497   | 0.526 |\n| DAPO                | 8k   /   8k     | 0.333     | 0.308     | 0.879    | 0.794    | 0.325   | 0.522   | 0.527 |\n| TRPA                | 8k   /   8k     | 0.454     | 0.279     | 0.864    | 0.756    | 0.289   | 0.518   | 0.527 |\n| DisCO (L-ratio)     | 8k   /   8k     | 0.506     | **0.356**     | **0.900**    | 0.831    | 0.326   | 0.553   | 0.579 |\n| DisCO (log-L)       | 8k   /   8k     | **0.523**     | 0.354     | 0.896    | **0.843**    | **0.331**   | **0.560**   | **0.584** |\n\n**Q2:** It would be helpful to understand how DisCO's performance varies with different response lengths. For example, What happens if we increase the response length (e.g., 24k or 32k)?\n\n**A:** We thank the reviewer for the constructive comment. We have conducted an experiment using 16k response length for training and inference on 1.5B models. The results shown below demonstrate that increasing response length brings additional improvements, compared with 8k length for training and inference. \n\n| Model                | MRL(Train/Test) | AIME 2024 | AIME 2025 | MATH 500 | AMC 2023 | Minerva | O-Bench | Avg.  |\n| -------------------- | --------------- | --------- | --------- | -------- | -------- | ------- | ------- | ----- |\n| DS-Distill-Qwen-1.5B | 32k+   /   32k  | 0.288     | 0.263     | 0.828    | 0.629    | 0.265   | 0.433   | 0.451 |\n| DSR-1.5B-Preview     | 24k   /   32k   | 0.431     | 0.304     | 0.878    | 0.736    | 0.302   | 0.500   | 0.525 |\n| DisCO (L-ratio)      | 8k   /   8k     | 0.381     | 0.306     | 0.878    | 0.746    | 0.319   | 0.512   | 0.524 |\n| DisCO (log-L)        | 8k   /   8k     | 0.404     | 0.317     | 0.876    | 0.758    | 0.333   | 0.509   | 0.533 |\n| DisCO (L-ratio)      | 16k   /   16k   | 0.410     | 0.340     | 0.885    | 0.748    | 0.324   | 0.515   | 0.537 |\n| DisCO (log-L)        | 16k   /   16k   | 0.404     | 0.333     | 0.891    | 0.761    | 0.321   | 0.531   | 0.540 |"}]

Author rebuttal:

Meta-review: This paper proposes DisCO (Discriminative Constrained Optimization), a reinforcement learning method for large reasoning models that alleviates difficulty bias and training instability in existing methods like GRPO.

During the discussion period, the authors addressed all major concerns through comprehensive experiments and clarifications:

1. Ablation studies and component contributions (Reviewers 2CBw, CL8D). The authors provided detailed ablation results showing each component's contribution: removing question-level weight bias, using non-clipping scoring functions (the largest contributor), and applying KL constraints. These experiments clearly decompose the effectiveness of each design choice.
2. Hyperparameter tuning fairness (Reviewer 2CBw). The authors clarified that they tuned the learning rate for all methods and one additional parameter without extensive grid search for their own method DisCO. Furthermore, baselines were applied under similar experimental settings by prior works, so it was fair for the authors to follow the training settings. 
3. Theoretical clarity and notation (Reviewer Hi9A). The authors clarified mathematical notation, explained the theoretical basis for difficulty bias through Proposition 1, and corrected the interpretation of Table 3 results.

All four reviewers converged on acceptance after the rebuttal. The recommendation is to accept this paper.
Decision: Accept (poster)