Title: DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Large Reasoning Models

Discriminative Learning
Abstract: The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning models (LRMs). In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation of question-level difficulty bias arising from its group relative advantage function. We also identify a connection between GRPO and traditional discriminative methods in supervised learning. Motivated by these insights, we introduce a new **Discriminative Constrained Optimization (DisCO)** framework for reinforcing LRMs, grounded in the principle of discriminative learning: increasing the scores of positive answers while decreasing those of negative ones. The main differences between DisCO and GRPO and its recent variants are: (1) it replaces the group relative objective with a discriminative objective defined by a scoring function; (2) it abandons clipping-based surrogates in favor of non-clipping RL surrogate objectives used as scoring functions; (3) it employs a simple yet effective constrained optimization approach to enforce the KL divergence constraint.  As a result, DisCO offers notable advantages over GRPO and its variants: (i)  it completely eliminates difficulty bias by adopting discriminative objectives; (ii) it addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach, yielding long and stable training dynamics; (iii) it allows the incorporation of advanced discriminative learning techniques to address data imbalance, where a significant number of questions have more negative than positive generated answers during training. Our experiments on enhancing the mathematical reasoning capabilities of SFT-finetuned models show that DisCO significantly outperforms GRPO and its improved variants such as DAPO, achieving average gains of 7\% over GRPO and 6\% over DAPO across six benchmark tasks for a 1.5B model.


Peer reviews:
comments: SUMMARY:
The authors propose a new reinforcement learning (RL) optimization framework for large reasoning models based on discriminative learning. Specifically, it employs a score-based discriminative objective instead of a group-relative policy and uses a scoring function in place of clipping. Training is stabilized through a simple constrained optimization method. The proposed framework demonstrates strong potential across multiple reasoning benchmarks, achieving better performance than GRPO while producing shorter responses.

STRENGTHS AND WEAKNESSES:
**[Strengths]**
- The paper is well-written and easy to follow.
- The proposed framework to alleviate difficulty bias is interesting and achieves meaningful results across multiple benchmarks.
- The ablation study is detailed and comprehensive.

**[Weaknesses]**
- The framework only considers binary rewards (though the authors acknowledge this). It would be more compelling if it were shown to work with non-binary reward settings as well.
- The paper misses an ablation study on response length. It would be helpful to understand how DisCO's performance varies with different response lengths.

QUESTIONS:
- What happens if we increase the response length (e.g., 24k or 32k)? Does it improve DisCO's performance notably?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem difficulty bias and training instability in existing Large Reasoning Models (LRMs) during reinforcement learning by proposing a novel framework. The study optimizes scoring functions to enhance scores for correct answers while suppressing those for incorrect answers, employing a constrained optimization approach to ensure training stability. Additionally, it introduces Distributionally Robust Optimization (DRO) techniques to mitigate data imbalance issues during training. Experiments demonstrate that the proposed architecture outperforms existing GRPO and its variants on reasoning tasks, significantly enhancing model performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.The paper offers detailed theoretical analysis and experimental validation, exposes GRPO's limitations, and presents targeted solutions. The well-designed experiments cover multiple benchmark tasks and various model sizes. The results show that the new architecture in the paper is better than existing methods in improving model performance. Also, the paper discloses enough experimental details for other researchers to reproduce the experiments.\n2.This study is highly significant for enhancing the reasoning abilities of large - scale reasoning models in complex tasks. By addressing key issues in current methods, it offers a more effective and stable model optimization approach, which will advance research in related fields.\n\nWeaknesses\n1.In the experimental part, although the paper provides a large number of experimental results and analyses, the descriptions of certain experimental details may not be in-depth enough, such as the selection and tuning process of hyperparameters.\n2.When presenting related work and introducing the DisCO method, the paper emphasizes the limitations of GRPO and its variants but rarely mentions their merits and applicability in specific scenarios. This may bias readers' understanding of different methods, preventing them from objectively evaluating the relative advantages and disadvantages of each approach in various contexts.\n\nQUESTIONS:\n1.Could you quantify the individual contribution rates of each improvement in DisCO through ablation studies in detail?\n2.Will the DisCO algorithm pose computational efficiency challenges as models scale further?\n3.In hyperparameter selection, have you accounted for the impact of different hyperparameter combinations on model generalization capabilities?\n\nLIMITATIONS:\nYes，the authors have explicitly stated several limitations of their work in the paper, noting that their method primarily \noptimizes for binary rewards and that experiments were constrained by computational resources, covering only 1.5B and 7B models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new Discriminative Constrained Optimization framework for reinforcing large reasoning models (LRMs). Starting from group relative policy optimization (GRPO), DisCO firstly replace the group relative objective with a discriminative objective defined by a scoring function, and employs a constrained optimization approach to enforce the KL divergence constrainet. Experimental results show that DisCO outperforms GRPO and DAPO on 1.5B and 7B model with the same training queries.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. From Table 2&3, the proposed method can outperform previous works on both 1.5B and 7B models and exhibits better stability in training than previous works.\n\nWeaknesses:\n1. There are too many equations that make the paper hard to follow, especially there are many symbols that do not have explaination and requires the readers to guess. For example, function $g(o,q)$ is a one time symbol that can be essentially replaced by $f(x, y)$, $f^{+}$ and $f^{-}$ can be omitted. There are also many abbreviations that show up without explanation, such as VPG, GRPO_RW in the equations.\n2. The claimed motivation that previous methods have difficulty bias seems to be a bit emperical and does not have theoretical support.\n3. In Table 3, when the initial policy is strong, the improvements of DisCO is small, and the baseline methods such as GRPO and DAPO even decrease the performance of initial policy, is there any explanation?\n\nQUESTIONS:\n1. The equation 9 seems to have very similar formulation as Direct Policy Optimization. Is there any essential connection and difference between the proposed DisCO and online DPO?\n2. The reward distribution of training queries are highly correlated with the initial policy and the training data. If the initial policy is strong, and the training queries are hard (for example, using the queries provided by DAPO[1] or OREAL[2], is DisCO still effective and better than previous methods?\n[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k\n[2] OREAL: Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning.  https://huggingface.co/datasets/internlm/OREAL-RL-Prompts\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes several modifications to GRPO:\n- Does not clip advantage\n- Removes down-weighting of questions with very high or low reward\n- More stable KL divergence term using a squared-hinge penalty function \n\nThey train and compare their method (Disco) with GRPO, Dr. GRPO, DAPO, TRPA. They also compare against Deepseek-R1-1.5B PReview and OpenAI-o1-Preview. Disco trained model outperforms these variants on math tasks.\n\nSTRENGTHS AND WEAKNESSES:\n- [Good performance] Based on numbers reported, it looks like the algorithm works well. \n- [Organization needs some work. Contributions are weakly communicated --- multiple things happening with the algorithm, might be good to decompose the paper into separate sections] There seems to be two things the paper is trying to communicate: that GRPO is downweighting examples with extreme rewards + standard clipping/KL divergence term causes instability. However, in the experiments, it's very unclear where the gains are actually coming from. It might be good to more closely decompose the effect of each design choice to the performance in the main body + provide more extensive ablation studies that demonstrate the problem tackled. \n- [Concurrent work] The idea of upsampling examples with the highest and lowest rewards is also explored in concurrent work Not All Rollouts are Useful [https://arxiv.org/abs/2504.13818]. \n- [No hyperparameter tuning for baselines] Based on the experimental section, it looks like hyperparameters are manually tuned for their algorithm, but baseline hyperparameters are reported numbers adapted from previous papers. It's not clear whether these hyperparameters are optimal for their setup.\n\nQUESTIONS:\n- I'm curious what the interplay is between the expected reward and entropy collapse in your algorithm. These concepts seem to be closely related, since entropy collapse would also lead to a bimodal expected reward distribution (e.g. you could be oversampling one particular incorrect or correct answer and get extremely low/high rewards, respectively). If you reduce entropy collapse, I imagine less datapoints would have extremely low or high rewards. Yet Disco would also precisely focus more on these outlier examples. I wonder if some effects of these two design choices are somewhat cancelling each other out? \n- If you remove the clipping for GRPO + use the same squared-hinged KL  term, does the performance improve?\n\nLIMITATIONS:\n- Based on the insights provided, are there any particular datasets you would foresee Disco to be particularly effective or suboptimal in comparison to GRPO? e.g. perhaps you can try comparing these algorithms on  datasets with a small number of very easy and hard examples.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-07"}