Hybrid Reinforcement: when reward is sparse, better to be dense
Authors: Leitian Tao
Ilia Kulikov
Swarnadeep Saha
Tianlu Wang
Jing Xu
Sharon Li
Jason E Weston
Ping Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Hybrid rewards for reinforcement learning
Primary area: foundation or frontier models, including LLMs

Abstract: Post-training for reasoning in large language models has increasingly relied on verifiable rewards: deterministic checkers that provide $0$–$1$ correctness signals. While reliable, such binary feedback is brittle—many tasks admit partially correct or alternative answers that verifiers under-credit, and the resulting all-or-nothing supervision limits learning. Reward models  offer richer, continuous feedback, which can serve as a complementary supervisory signal to verifiers. We introduce HERO (Hybrid Ensemble Reward Optimization), a reinforcement learning framework that integrates sparse verifier signals with dense reward model scores in a structured way. HERO employs stratified normalization to bound reward-model scores within verifier-defined groups, preserving correctness while refining quality distinctions, and variance-aware weighting to emphasize challenging prompts where dense signals matter most. Across diverse mathematical reasoning benchmarks, HERO consistently outperforms reward model-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks. Our results show that hybrid reward design retains the stability of verifiers while leveraging the nuance of reward models to advance reasoning.

Published as a conference paper at ICLR 2026
HYBRID REINFORCEMENT:
WHEN REWARD IS SPARSE, BETTER TO BE DENSE
Leitian Tao 1.2 Ilia Kulikov Swarnadeep Saha', Tianlu Wang', Jing Xu', Sharon Liº, Jason Weston', Ping Yu! FAIR at Meta
University of Wisconsin-Madison
ABSTRACT
Post-training for reasoning large language models has increasingly relied on verifiable rewards: deterministic checkers that provide (-1 correctness signals. While reliable, such binary feedback is brittlemmany many tasks admit partially correct alternative answers that verifiers under-credit, and the resulting all-or-nothing supervision limits learning. Reward models offer richer, continuous feedback, which can
complementary supervisory signal verifiers. We introduce HERO (Hybrid Ensemble Reward Optimization), a reinforcement learning frame- work that integrates sparse verifier signals with dense reward model scores in a structured way. HERO employs stratified normalization to bound reward-model scores within verifier-defined groups, preserving correctness while refining qual- ity distinctions, and variance-aware weighting to emphasize challenging prompts where dense signals matter most. Across diverse mathematical reasoning bench- marks, HERO consistently outperforms reward model-only and verifier-only base- lines, with strong gains on both verifiable and hard-to-verify tasks. Our results show that hybrid reward design retains the stability of verifiers while leveraging the nuance of reward models to advance reasoning.
Sungli High Quallt Angrest
X
Sumple with Law Quality
(a) Reward Model
(b) Rulc-hased Rewards
(c) Hyhrid Reinforcement (HERO)
Negative
Reward
Roweed
Positive
Negalliee
Figure 1: Comparison of reward signals from different supervision sources. Reward Models (a) provide smooth but sometimes misaligned scores, occasionally assigning high values to incorrect responses and low values to correct ones. Rule-based rewards (b) enforce strict binary (0-1) boundary: they rarely give false positives, but due to their stringent criteria, many predictions that actually correct receive reward of 0 simply because they fail to pass the rule. HERO (c) uses the rule as a gate, which significantly reduces false positives. At the same time, by integrating the reward model signal, HERO assigns higher reward scores to those cases that would have been false negatives under (b), resulting in more accurate and informative supervision.
INTRODUCTION
Reasoning lies at the heart of human intelligence, and increasingly, at the frontier of large language model (L..M) capabilities (Jaech al., 2024; Guo al., 2025; Zhang et al., 2025b). In tasks such as mathematical problems generating proofs, reliable reasoning requires models to generate logically
Work done during internship Meta.

---

Published as a conference paper at ICLR 2026
consistent multi-step solutions that culminate in a verifiably correct outcome. Verifiable rewards im- plement this idea by running a deterministic checker such as an exact numeric or string match, unit test, symbolic equivalence check-on candidate solution y for input Z. The checker either accepts or rejects the output, producing a sparse but unambiguous signal r(x: y) E {0,1), which reinforcement learning can propagate through the entire trajectory. Building ciple, reinforcement learning from verifiable rewards (RLVR) (Chen et al., 2025b) uses these bi- nary signals to train policies toward solutions that pass the checker. Recent systems-includingg DeepSeek-RI PeepSek--R1-have have advanced this paradigm at scale, leveraging verifier- srafiier-gronded feedback to im- prove reasoning (Guo al.. 2025; Zeng et al., 2025; Luo et al., 2025; Yang et al., 2024a).
this prin-
However, strict 0-1 verification is coarse and brittle: many reasoning tasks allow for partially correct solutions, equivalent answers alternative formats, or open-ended outputs that resist exact match- ing. In such cases, symbolic verifiers may under-credit valid solutions (false negatives) or fail to provide any useful signal (Ma et al., 2025; Huang et al., 2025a). Even when applicable, binary re- wards induce sparsity: if all rollouts for prompt receive the same label (all Os or 1s), group-relative methods such as GRPO (Shao et al., 2024) yield zero relative advantage and thus no useful policy gradient, stalling policy improvement. This brittleness not only reduces sample efficiency but also skews optimization toward easier, strictly verifiable cases leaving the hardest and most informa- tive prompts underutilized. Our motivating analysis in Section 3.1 further highlights this limitation: on samples where answers are hard to verify, rule-based verifiers frequently fail with correctness. Reward models, in contrast, offer dense supervision by scoring responses on a continuum (Yang et al., 2024b; Liu et al., 2024; Zhang et al., 2025c; Lyu et al., 2025; Liu et al., 2025). Rather than collapsing all incorrect answers into the same category, they capture nuanced quality differences such as partial correctness, clarity of reasoning steps, or proximity to the ground truth. This graded feedback enriches training signals, helping policies learn from partially correct reasoning paths and better allocate credit across diverse rollouts. However, naively combining these dense reward CT model signals with a binary verifier output often destabilizes training. Specifically, when the reward model's 's continuous signals are naively blended with binary correctness checks, the resulting reward can become noisy or misaligned with the expected semantics of correctness. Figure I illustrates this tradeoff: while reward models offer smooth but misaligned signals, rule-based verifiers enforce correctness but lack nuance. Thus, it remains an open question how to design an effective hybrid framework that preserves the reliability of verifiers while harnessing the richness of reward models.
To address this challenge, we propose HERO (Hybrid Ensemble Reward Optimization), a reinforce- ment learning framework that integrates verifier-anchored and dense reward-model signals to pro- vide reliable yet informative supervision. HERO tackles the instability of naive blending through two key innovations. First, it introduces a stratified normalization scheme that bounds reward-model scores within verifier-defined correctness groups This ensures that dense feedback refines learning only within the set of responses deemed correct by the verifier, preserving correctness guarantees while exploiting nuanced distinctions. Second, HERO employs a variance-aware weighting mecha- nism that adaptively adjusts the contribution of different prompts during training. Easy prompts, where most responses are uniformly correct incorrect, contribute little additional learning signal and are down-weighted. reward-moda scores provide valuable discrimination are emphasized. These components allow HERO to overcome the brittleness of purely binary rewards and the noisiness of dense signals.
contrast, harder prompts-where candidate responses vary widely and
We evaluate HERO diverse math reasoning benchmarks that training with three regimes: training with easy-to-verify samples where exact final-answer checking is possible, hard-to-verify samples with partially correct or format-sensitiv solutions, and mixed settings combining both. Across dif- ferent LLM backbones, HERO consistently outperforms both RM-only and verifier-only baselines, in all three regimes. Notably, on hard-to-verify tasks evaluation based on Qwen-4B-Bar HERO achieves 66.3, which surpasses RM-only training (54.6) by +11.7 points and verifier-only training (57.1) by +9.2 points. Ablations further confirm that anchoring dense signals to verifiable correct- ness and adaptively reweighting difficult prompts are both critical for stability and efficiency.
2 PRELIMINARIES
Dense reward via reward modeling. Reward modeling learns a scalar function r(z, y) that eval- uates the quality of a response y given a prompt ær. Based on the Bradley-Terry model (Bradley &
2

---

Published as a conference paper at ICLR 2026
Terry, 1952), the reward function is typically trained pairwise preference data by minimizing
LR -E(p:.....) (r(x,, Me) r(x, yr))],
where denotes the sigmoid function, Vr. is the response that is considered preferred in a compar- ison, and yr is the response considered less preferred. Once learned, can guide reinforcement learning to align the model with human preferences.
Sparse reward via verifier. Reinforcement learning with verifiable rewards (RLVR) leverages deterministic function r(z, y) to assess correctness, assigning sparse reward (e.g.. 1 for correct, for incorrect). All tokens in a response share the same reward, providing unambiguous supervision for tasks with objective ground truth. In mathematical problem solving, reward function is based on verifier that checks whether the model's solution matches the ground-truth reference under cquivalence transformations. Specifically, math verifier typically parses the predicted solution into structured form (e.g.. symbolic expression, final numeric answer, proof step), simplifies it, and compares it against the reference solution using symbolic algebra tools or logical equivalence checks. The reward function is based on the verifier:
(1, if ys is equivalent to Yref given I, 0 otherwise.
(z, Dis Bteer)
(2)
Group Relative Policy Optimization. GRPO (Shao al., 2024) extends RLVR by optimizing over multiple responses per prompt rather than treating them independently. Instead of relying on a single trajectory, GRPO compares groups of candidate solutions and assigns relative advantages, which stabilizes learning and improves exploration. It also incorporates clipping (as in PPO) to prevent unstable updates and adds a KI. penalty to keep the policy close to reference model. This group-based formulation helps alleviate the gradient sparsity problem of pure verifier rewards and makes optimization more sample-efficient than standard PPO (Yu et al., 2025).
3 METHODOLOGY
3.1 MOTIVATION: DELVING INTO RULE-BASED VS. RM-BASED VERIFIERS
Building on the preliminaries, we now examine how the two supervision paradigms - rule-based verifiers that provide sparse but precise correctness signals, and reward models that offer dense but potentially noisy preferences behave on tasks where correctness is difficult to verify. Since the reliability of training hinges the quality of supervision, understanding their respective strengths and weaknesses is crucial. To this end, we use the HardVerify sywferrff_MMar Math benchmark (Xu et al., 2025), which focuses on challenging verification scenarios. This benchmark consists of 250 hard-to- veriff math problems, including 1 15 manually selected Olympiad questions (He et al., 2024) and 10 MATH test set questions (Hendrycks al., 2021) that are prone false negatives due to complex answer formats, as well as 125 Big-Math questions (Albalak et al., 2025) with a Llama-3. 1-8B (Dubey et al., 2024) pass rate below 0.05 and identified difficult by human experts. To ensure diverse range of response qualities, for each problem we generate answers using three different models: Llama- 3. 1-B, Llama-3.3-70B, and Qwen3-8B. This results total of 750 responses, which we use to conduct the verifier analysis presented in Table 1.
Limitations of rule-based verifiers. To better understand the trade-offs among different verifi- cation approaches, we compare several representative verifiers. For rule-based verifiers, we con- sider math reward. py from the verl library, the math verify module from verl, and the parse and verify functions from the Math-Verify library. addition, we include a generative model-based verifier Sarr---Laaggeeeal-eeereee thought answer verification. This model has demonstrated strong performance and serves as an effective alternative to traditional rale-based methods.
(Ma et al., 2025)), which is specifically trained for chain-of-
Results in Table highlight clear precision pecill trade-offs. Function Function-based rules offer high pre- cision but low recall. For example, the math-reward. py checker is highly conservative: it almost never produces false positives (FPR=0.3%) but fails to recognize many correct answers, resulting in very low recall (10.1%). more advanced variant, math verify.py (in verl), achieves the best
3

---

Published as a conference paper at ICLR 2026
balance near-zero false positives with substantially higher recall. The math.verify library extends coverage with normalization heuristics (c.g., handling formatting differences or units) but remains brittle for mismatched orderings such as lists vs. sets, yielding only 38.6% recall.
Table 1: Performance Comparison of Rule-based Verifier, LLM-as- Veriffee,, and Reward Models.
Type
Verifier
Recall
Precision
FPR |
Acc.
math_reward (verl)
10.1
97.5
0.3
53.6
math_verify (verl)
68.4
100.0
0.0
83.7
Rule-based
math.verify (library)
38.6
96.1
1.6
67.6
Generative Model-bel Model-based
TIGB--Lab/geeerra--eeiiiiee
49.5
89.3
6.3
70.9
AceMath-7B-RM threshold
91.7
67.7
46.4
73.2
AceMath-7B-RM. w threshold
84.2
72.7
33.5
75.6
RM-based
AceMath-7B-RM. w threshold
73.8
76.6
23.9
74.9
AceMath-7B-RM w threshold 7
62.4
78.5
18.1
71.9
Reward modeling can generalize to hard-to-verify samples. We further examine how reward models behave on hard-to-verns verify samples. Since correctness is directly checkable for verifiable data, most reward models for mathematical reasoning are trained on these verifiable samples (Yang al., 2024b; Liu et al., 2024; Zhang et al., 2025c; Lyu et al., 2025; Liu et al., 2025). This raises the question: to what extent such models generalize to tasks where correctness cannot be directly verified?
Here, investigate this issue by analyzing the performance of math-focused reward model (AceMath-7B-RM) on the same hard-to-verify tasks. We assess the model using different score thresholds given the scores generated. As shown in Table 1, setting the threshold ering predictions with RM 2 as correct) yields high recall of 91.7% and broad overall coverage, substantially surpassing rule-based verifiers. However, this comes at the cost of lower precision. Increasing the threshold enhances precision but leads to decrease in recall.
(i.e., consid-
The need for hybrid reward design. Our analysis underscores key tension: neither rule-based verification nor reward models alone sufficient. Purely binary verifiable rewards can be brittle and overly conservative, especially hard-to-verify samples. This only reduces sample efficiency but also skews optimization toward easier, strictly verifiable leavin the hardest and most informative prompts underutilized. Reward models, in contrast, offer dense supervision by scoring responses on a continuum and can capture nuanced quality differences such as partial correctness or proximity the ground truth. These complementary strengths and weaknesses motivate hybrid approach: anchoring supervision in symbolic verifiers to preserve correctness, while enriching it with the dense signal of reward models drive effective policy learning. In the next subsection, we describe our proposed approach in detail.
3.2 HERO: HYBRID ENSEMBLE REWARD OPTIMIZATION
Motivated by these findings, our design principle is that rulc-based rewards should continue to guide the overall reasoning dynamics, while reward models as supplementary signals enrich train- ing. We therefore propose hybrid reward framework that (i) augments binary correctness with dense reward-model scores and (ii) scales supervision according prompt difficulty. We describe both components in detail below.
Dense signals anchored to verifiable correctness. As argued in the motivation, binary verifiers alone provide stable but overly coarse supervision, while reward models offer nuanced distinctions that are easily corrupted if left unconstrained. However, find that naive combination of rule- based verification and reward modeling signals tends to undermine the stability of training and render the hybrid approach ineffective (see Appendix A.3). Specifically, when the reward model's

---

Published as a conference paper at ICLR 2026
continuous signals are naively blended with binary correctness checks, the resulting reward can become misaligned with the expected semantics of correctness.
To address this, we propose stratified normalization, which rescales the continuous scores of the ward model (RM) to align with the range used by traditional binary aule-based methods. Specifically, let (rik r=ll
I. S (0, 1) denote the rule-based verifier outputs and {rRM) J C IR the corresponding
reward-mode scores for a group of N rollouts. We first partition the responses according to rrules and then apply min-mat normalization within each group to FRM resulting in:
FRM - min TRM
a + 2a
Trule = 0,
f(z,0)=
max TRM
min TRM
(3)
TRM min TRM
- (3 + 2/3
Trule- 1.
FRM
min FRM +
Here a, B E (0, 1] control the allowable ranges for incorrect and correct groups, with >> prevent- ing division by zero. In practice, we set e to relatively small value so that the training dynamics are primarily led by rule-based rewards, with the reward model's s contribution serving as a supplemen- tary signal. Figure (c) illustrates the effect of our hybrid reward design compared to ''M (a) and Tmlle (b) alone. This design notably differs from traditional pure verifiable rewards, especially for hard-to-verify samples and for groups where all responses are either positive or negative. In such cases, pure rule-based methods do not distinguish between different rollouts, providing no learn- ing signal. As illustrated in Figure 1(c), HERO introduces reward differences within regions where the rulc-based rewards are all or all 1, thereby enabling meaningful gradient flow even when the rule-based verifier assigns the same outcome to all rollouts.
The stratified normalization in our hybrid approach is designed to provide the best of both worlds: verifiers ensure the preservation of correctness semantics by constraining the score ranges, while reward models enhance the supervision by introducing gradations within each group. Incorrect responses are clearly distinguished from correct ones, and correct responses prioritized based on their relative quality. In this manner, dense signals are anchored to symbolic correctness, mitigating the sparsity observed in pure RLVR.
Variance-aware advantage reweighting. In the motivation, we argued that not all prompts equally informative: trivial ones provide little learning signal, while challenging prompts better reveal differences across candidate solutions. A shortcoming of the original GRPO algorithm is that it treats all prompts uniformly, ignoring this variability. The consequence is inefficient use of training capacity-eas prompts dominate optimization even though they provide little additional guidance, while difficult prompts that expose meaningful distinctions are underutilized. To realign training effort with informativeness, we introduce a variance-aware weighting scheme. For each prompt, let on denote the standard deviation of reward-model scores across candidate responses, with ō as running mean. This variance reflects uncertainty: higher values suggest greater disagreement and thus richer training signal. We define a bounded monotone weighting function:
1
Waifficulty (ow) = Wmin + (Wmax - -mmin) + exp( k(o. - -))
(4)
where Wmin and Wmax set the minimum and maximum weights, and k controls the slope of the transition. In practice, we treat these as tunable hyperparameters: unless otherwise stated, we use
Wminn 0.5, Wmax 2.0, and & = 5, ensuring that difficult prompts are up-weighted by at most 2x, while trivial prompts retain least half weight. The final shaped reward is
Triaal (2, Waiffiuitty (a.) (...).f(a.s)
-(x,
(5)
This design operationalizes our intuition: ambiguous, high-variance prompts are emphasized be- cause they reveal more about model weaknesses and reward-model sensitivity, while trivial, low- variance prompts downweighted to avoid wasting capacity. In doing so, the training process not only remains anchored to verifiable correctness through f, but also allocates learning effort to the most challenging and informative parts of the data.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Training datasets. A central question is whether reasoning skills acquired through RLVR on ver- ifiable data can generalize tasks whose correctness cannot be mechanically checked. To empiri- cally examine this, we construct three distinct training datasets based on subsets of the OPENMATHI- REASONING (Moshkov al., 2025) benchmark: (1). easy-to-verify examples, (2). hard-to -e-iifff examples, and (3). a mixture of the two. For casy-to-verify training data, we sample 2,000 problems whose final answers can deterministically validated using a rule-based math verifier (verl). For the
regime, we likewise sample 2,000 problems from OPEN SPENMATHRFASONING where the correct answers have more flexible formats that make rule-based verification challenging (see Appendix A.2.2 for how we filter well as some qualitative examples). For the mixed set, we combine 1,000 easy-to-verify and 1,000 hard-to-verify problems, allowing the policy to benefit from both robust exact-check supervision and nuanced feedback from unverifiable cases.
Models. To evaluate the generalizability of method across different backbone models, conduct experiments using the following models: we use Qwen3-4B-BB -B--ase (Yang et al., 2025) and MoooThinne--B--Hyrrid--
base (Wang et al., 2025). To stabilize RL training dynamics, we first perform supervised fine-tuning (SFT) on each base model as a cold start (see Appendix A.2.1 for details). All RL experiments are initialized from the same SFT checkpoint.
Baselines. As preliminary points of reference, we also report the performance of the base model and a cold-start SFT model. The main baselines are: (1) Reward model, which uses the AceMath- RM-7B reward model (Liu al., 2024); (2) Rule-based verifier, which relies on binary, rule-based rewards, marking sample as correct only if the normalized final answer matches the ground truth via math-verify (library) in the VERL repo (Sheng et al., 2025). Our method, HERO, is hybrid approach that combines (I) and (2) into single reward, making them the most ap- propriate baselines for comparison. We also compare HERO with a generative model-based veri- fier-TIGER-Lab BEEiIGER-Labggeeee LaiiEER-Labgggeraaa aeaeerrl-eriiiee--aae rectly prompted to act as a verifier, as detailed in the Appendix (see Table 7).
di- with large language model (Qwen2.5-7B-Instt)
Evaluation. Since our training data contains both tasy-to-verify and hard-to-verify samples, we aim to evaluate whether the model can acquire generalizable reasoning abilities. To this end, we lect six test sets: four in which all answers are easy to verify, and two in which the answers are hard vlr)fy.(1). Easy-to-verify testsets includes 4 benchmarks: MATH500 (Hendrycks et al., 2021), AMC (Li et al., 2024). Minerva (Lewkowycz et al., 2022), and Olympiad (He et al., 2024). We report pass @ averaged over 8 seeds in Table Following (Yang et al., 2024b), we use temperature 0.6 and top-p 0.95, generate N = 8 candidates per problem, and evaluate the first decoded output (pass@) 1). Reported numbers are means over seeds. Correctness is decided by math verifier (normal- ized numeric/string match with task-specific (o)t-)rocessingg).)... Hard-to-verify testsets: We use temperature 0.6 and top-p 0.95, generate N = candidate per problem. Since symbolic checkers cannot reliably provide binary labels for open-ended solutions, we adopt LLM-as-a-judge pro- tocol. Specifically, we use GPT-4o to compare model outputs against ground- truth answers. We evaluate using the HardVyrifyy-Mar Based on the results in Section 3.1, we find that HardVerify--aate is not a particularly challenging filter, using math verify yields relatively good results. Therefore, to further evaluate performance on hard-to-verify reasoning tasks, we additionally collect the TextBookReasoning dataset (Fan et al., 2025) (see Appendix A.2.3 for more details).
benchmark (Xu et al., 2025), which consists of 250 samples.
4.2 MAIN RESULTS
Performance of HERO on the Qwen-based Model Table 2 shows that HERO consistently outper- forms both the reward-model-only and rule-based verifier baselines across all three training data settings: (1) easy-to-verify data, (2) hard-to-verify data, and (3) a mixture of the two. For each casy-to-vert
training setting, evaluate four datasets where the targets
tasks, well as two datasets where the targets are hard-to-verify tasks. When trained on easy-to-verify data and
evaluated four easy-to-verify test sets, HERO achieves average score of 62.0, outperforming

---

Published as a conference paper at ICLR 2026
Table 2: Performance of HERO trained with Qwen3-4B-Base both easy-to-verity and hard- to-verify reasoning tasks. The first block reports results verifiable tasks (MATH500, AMC, Minerva, Olympiad: with average), while the second block presents results on hard-to-verify tasks (HVM, TBR). We compare our approach HERO -which combines two reward signals-with two baselines corresponding to the two signals: AceMath-7B-RRt (a reward model) and math.verify (verl), which uses 0/1 rule as the reward.
Easy-to-verify tasks
Hard-to-verify tasks
MATH500
AMC
Minerva
Olympiad
Avg.
HVM
TBR
Avg.
Qwen3-4B-Basc
67.5
44.1
29.4
32.1
43.3
45.2
40.2
42.7
SFT Cold Start Model
69.1
50.3
39.1
34.3
48.2
50.8
43.3
47.1
Training with easy-to-verify samples
AceMath-788 -B--a-h----M-------
80.2
61.6
40.6
43.3
56.4
$2.0
54.6
nath.verify (verl)
61.3
44.0
45.5
61.0
53.1
57.1
HERO (Ours)
85.4
69.4
44.5
48.9
62.0
73.2
59.3
66.3
Training with hurd-to-verify samples
AccMath-7B-Ra
79.6
58.8
39.9
42.1
55.1
48.2
53.7
nath.verify (verl)
76.2
28.7
38.2
47.4
58.4
50.0
54.2
HERO (Ours)
80.0
63.4
40.7
43.1
56.8
59.0
54.0
56.5
Training with mixed samples AceMath-7B-RM,
79.6
58.8
39.9
42.1
55.1
58.4
49.6
54.0
nath.verify (verl)
38.0
56.1
55.3
58.9
IERO (Ours)
81.6
64.4
42.1
47.0
58.8
71.4
56.7
64.1
both RM-only (56.4) and rulc-based training (58.3). We attribute this improvement to our strati- fied normalization, which allows hybrid training to exploit both positive and negative correctness groups: while verifier-only training collapses all-correct or all-incorrect batches (yielding zero rel- ative advantage), HERO preserves meaningful gradients within each group through dense intra- grrupp rewards. The advantage our approach becomes even more pronounced when training on hard- to-verify samples, where rule-based verifiers brittle and reward models tend to be noisy. Here, HERO attains 56.8, surpassing RM-only (55.1) by +1.7 points and rule-based verifiers (47.4) by sub- stantial +9.4 99.4 points. This improvement is due anchoring continuous reward-model scores within verifier correctness groups, which prevents reward drift and ensures stable learning even when binary labels saturate. By combining the precision of rule-based verifiers with the smooth discrimination of reward models, HERO is able to leverage partially correct reasoning paths that would otherwise be discarded by rule-based systems, thereby improving both stability and coverage. When trained on mixed data, which combines easy-to-verify and hard-to-verify samples, HERO again achieves the average (58.8), outperforming RM-only 155.1) and rule-based verifier (56.1) on verifiable tasks.
The advantage becomes even clearer on hard-to-verify evaluations (HVM, TBR), where rule-based verifiers fail to capture partial correctness and reward models are prone to drift. Here, HERO attains 66.3 when trained casy-to-verify data, outperforming RM-only (54.6) by +11.7 points and rule- based training (5 by +9.2 points. When trained on hard-to-verify samples, still leads with 56.5 compared to 53.7 (RM-only) and 54.2 (rule-based). Under mixed training, HERO reaches 64.1, sur- passing both baselines by large margins on hard-to-verify tasks. These results highlight that hybrid reward design generalizes robustly across both verifiable and hard-to-verify tasks, yielding stable improvements regardless of whether evaluation relies on symbolic checking or model judgment. Overall, hybrid reward learning delivers consistent improvements across all settings, demonstrating that structured reward integration is critical for reasoning tasks that go beyond strict verifiability. We note that the magnitude of gains Table 2 naturally varies across training regimes due to differ- ences in reward quality, rather than instability of HERO. With easy-to-verify training data, the verifier accurate and often positive, so HERO can fully exploit its hybrid design and achieve large im- provements; with tond-to-veriffy data, the verifier rarely fires and many prompts receive all-O labels, so the learning signal is weaker and gains are necessarily smaller. In the mixed regime, easy-to- verify samples provide strong verifier anchor while sand-to-veriff samples reduce the domain gap to difficult test sets, which explains why improvements are modest on verifiable tasks but large again on hard-to-verify evaluations.

---

Published as a conference paper at ICLR 2026
Table Performance of HERO trained with DetoTThinker--B--yyrri---- and hard-to-verify reasoning tasks. The first block shows results on four easy-to-verify tasks, which reported pass averaged over 8 seeds. The second block show results on two hard-to-verify testsets, which reported GPT4.1 judges results.
both easy-to-verify
Easy-to-veriis tasks
Hard-to-verify tasks
AMC
Minerva
Olympiad
Avg. f
HVM
TBR
Avg. 1
Berriinkc--B--Hyrid-BBBB- SPT Cold Start Model
32,0
15.3
9.1
11.0
16.9
26.0
21.1
23.6
56.0
35.9
19.7
21.6
33.3
27.6
26.4
27.0
Training with casy-to-verity samples
AccMath-7B-RM.
62.3
38.4
26.2
25.5
38.1
29.6
27.8
28.7
nath_verify (verl)
60.1
39.4
26.7
24.1
37.6
31.6
28.9
30.3
HERD (Ours)
63.0
40.6
30.1
26.7
40.1
28.4
36.7
32.6
Training with hard-to-verify samples
AceMath-7B-RR
60.7
33.8
22.4
24.9
35.4
32.0
29.8
30,9
verify (verl))
60.0
29.7
23.9
24.8
34.6
28.8
26.7
27.8
HERD (Ours)
64.9
41.6
27.9
29.6
41.0
32.4
36.7
34.6
Training with mixed samples
AceMath-7B-RM
34.4
24.0
23.8
29.3
30.1
verify (verl)
33.7
24.7
24.0
27.6
28.7
(Ours)
65.2
28.1
29.3
34.8
Performance of HERO on the OctoThinker-based Model On Qwen3-4B- Breeae (Table 2), which already shows strong performance, HERO consistently delivers clear improvements across all evalua- tion settings. On Octo -hinnee--BB-Hybrid--- of 16.9 on verifiable and 23.6 on hard-to-verify tasks, HERO achieves substantial absolute and relative gains. When trained easy-to- easy-to-verify samples, it reaches 40.1 verifiable and 32.6 hard-to- verify evaluations, surpassing both the reward-model baseline (38.1/28.7) and the rule-based verifier (37.6/30.3). Training hard-to-verify samples yields similar improvements, achieving 41 .0/34.6 compared to 35.4/30.9 (RM-only) and 34.6/27.8 (verifier-only). Training on mixed training sam- ples, it maintains the highest averages of 40.2 and 33.2, outperforming all baselines by 4-6 points. These results show that hybrid reward design generalizes robustly across model scales-preserving the verifier's stability for stronger models like Qwen3-4B-Bas while bringing large relative gains to weaker ones such as OctoTThinke Thinker-80 -BB H---Hybrid-Bae -asee.
(Table 3), which starts from much weaker baseline
Verifier-only training struggles hard-to-verify tasks. Symbolic verifiers, while precise, per- form poorly on open-ended or format-sensitive reasoning. On Qwen3-4B-BB (Table 2), verifier- only training reaches only 47.4 on hard-to-verify taske than HERO (56.5), RM-only (53.7), and even slightly below the SFT baseline (47.1). Similar degradation appears on OctoThinker-8hi Hybrid-Bi Base (Table 3), where verifier-only supervision lags far behind HERO. The core issue that weaker models often produce rollouts that are uniformly labeled () or 1, causing group-relative ob- jectives such as GRPO to yield zero gradients. In contrast, HERO adds dense intra-group variation via reward-model refinement, preserving gradient flow even when binary labels saturate and allowing the policy to separate partially correct from entirely incorrect solutions.
4.3 ADDITIONAL ABLATIONS
Dense negative ranges are more important than positive samples. We evaluate the role of dense negative and positive reward on the setting of training with easy-to-verify samples based on Qwen- 4B-Base. We found dense reward the negative group play a more criticak role stabilizing training and improving learning efficiency than dense reward in the positive group devided by HERO.. While positives signal correctness, negatives provide richer supervision by penalizing diverse rea- soning errors. Notably, dense negative rewards but maintaining sparse verifier positive rewards boosts performance on verifiable tasks from 59.4 to 61.4, and even more on hard-to- terver tasks from 62.2 to 68.4 4 (see Figure 2). This demonstrates that well-calibrated negative ranges tial: they provide broader feedback, enabling the model to detect subtle errors and generalize better complex cases.
8

---

Published as a conference paper at ICLR 2026
Negative
Range Ablation
20
Verifibble
80
Verifiable
Mixed
15.
Hand-to-wwrily
75-
72.4
18.1
52.3
66.6
92.5
85-
Accuracy
6033
61.
50.3
54/
59.3
56.4
None
Pas Only Neg Only Pos+Neg
Figure 2: (a) Impact of using positive and negative dense ranges. Dense negative rewards con- tribute more to stable learning than positive samples. (b) Effect of varying reward ranges under different training regimes. Smaller ranges perform best on verifiable tasks, while larger ranges benefit mixed settings by providing denser feedback.
Reward range selection crucial for balancing stability and performance. We conducted ab- lation studies to investigate the impact of varying reward ranges model performance by training with casy-to-verify samples based on Qwen-4B-Base as shown in Figure 2(b). For verifiable tasks, smaller reward ranges (e.g., a 0.05) yielded the best results, the rule-based verifier's preci- sion benefits from a tighter range that minimizes noise and maintains stability. Expanding the range beyond this threshold led diminishing returns and increased noise. In contrast, for mixed tasks, where many samples fail the rule-based verifier, the learned reward model plays a larger role. Here, larger reward ranges (e.g., a = 0.1 or = 0.2) provided richer signals, allowing the model to learn more effectively from harder tasks. However, expanding the range beyond certain point caused a slight performance drop due to overfitting or excessive noise. Overall, careful tuning of the re- ward range, particularly for the negative rewards, is crucial to balancing stability and performance: datasets with relatively dense informative rewards and few all all-positivveaall-ner negative groups tend to benefit from smaller ranges, whereas datasets with many all- polootiiivealllllleeoeleolee--le -eliaaivv groups are better served by slightly larger ranges that inject more intra- atrra-group variation.
Variance-aware reweighting in HERO im- proves the model's reasoning ability. We evaluated variance-aware reweight- ing based on reward-model score vari- ance, which emphasizes ambiguous, high- variance prompts while down-weighting trivial ones to reduce overfitting with the setting of training with easy-to-verif, sam- ples based on Qwen-4B-Ban Base. This dynamic adjustment yields consistent gains, particularly on veri-ty-verify tasks where dense signals most informative. As shown in Table 4, reweighting improves accuracy on both verifiable and hard-to-verify benchmarks, with larger gains in the latter (+3.8), confirming that focusing capacity on uncertain samples leads more robust and generaliz- able improvements.
Table 4:
Variance-aware reweighting improves perfor- mance on both verifiable and hard-to-verify samples.
Methods
Easy-to-verify Hard-to-verify
w/o reweighting
60.8
69.4
w reweighting
62.0
73.2
5 RELATED WORK
Reinforcement learning from verifiable rewards. Reinforcement learning from verifiable re- wards (RLVR) leverages deterministic correctness checks-sut such as passing unit tests or matching reference answers to enhance learning (Shao et al., 2024). Early program synthesis work demon- strated that agent-generated trajectories validated against ground truth outperform supervised ap- proaches (Bunel et al., 2018; Chen et al., 2021). For LLMs, rule-based verification plays a crucial role in filtering, providing training signals, and supporting benchmark evaluations (Xiong et al., 2025; Yu et al., 2025; Shao et al., 2024). Recent extensions include: outcome-driven RL (GRPO) for grounding and rubric-anchored RL which introduces structured rubrics for open-ended response evaluation (Huang et al., 2025b); verifier-free free RL strategies like VeriFree, which bypass explicit checking by directly maximizing the probability of generating the reference answer while achiev- ing performance par with verifier-hased methods (Zhou et al., 2025); and cross-domain RIVR,
9

---

Published as a conference paper at ICLR 2026
which employs LLM-derived scoring for domains lacking reference answers (Su et al., 2025). De- spite these advances, function-based rule verifiers remain high- precciseeiioon but low-recall: they often assign zero reward to semantically correct yet textually divergent outputs (Huang et al., 2025a), which has motivated the use of learned, model-based verifiers (Huang et al., 2025a; Chen et al., 2025a; Ma al., 2025; Xu et al., 2025). However, the coverage and generalization of LLMe-aseed verifiers are still limited (Li et al., 2025), and in many existing "hybrid" schemes they are invoked only relabel a subset of failures, ultimately producing binary 0/1 signals. As a result, these ap- proaches continue to suffer from sparse outcome-level rewards and the classical RLVR issue that all-positive or all all-negative rollouts yield vanishing advantages, limiting data efficiency. In contrast to previous work, we propose hybrid approach that combines rule-based verification with contin- uous, dense reward signals from learned models, allowing us to maintain the stability of verifiers while addressing their sparsity. By anchoring dense signals to symbolic correctness and introducing a variance-aware weighting mechanism, our method enables more informative, stable, and sample- efficient learning on both verifiable and hard-to-verify tasks.
Reasoning on hard-to-verify tasks. As the reasoning capabilities of LLMs have reached new heights, increasingly challenging reasoning benchmarks have been proposed (Phan et al., 2025; Zhang et al., 2025a). These problems often involve complex outputs, such as natural language representations and intricate mathematical or physical formulas. In such cases, rule-based verifi- cation methods, while effective for well-defined problems, struggle to capture the nuances of these tasks. Recent work focused on the LLMs as judges, where LLMs assess the quality of generated responses (Chen et al., 2025a; Ma et al., 2025; Huang et al., 2025a; Xu et al., 2025; Li et al., 2025), enabling more nuanced evaluations. However, despite its conceptual simplicity. LLM-as-judge may not always produce reliable assessments for comain--peccffe or long-form data. Some recent work proposes going beyond binary labels from verifiers for hard-to-verify tasks. For example, Gurung & Lapata (2025) applies reasoning traces in Next-Chapter Prediction (NCP) for long-form story gener- ation via likelihood estimation, while Tang et al. (2025) uses Jensen's s evidence lower bound to treat chain-of offin-of-ttt of-thought reasoning steps as latent variables in the generative process. They directly discard the verifier component. In contrast, our work retains the use of verifiable rewards but enhances su- pervision through the introduction of reward model. Peng et al. (2025) directly add the verifiable correctness signals and the human preferences for agentic tasks.
CONCLUSION
We introduced HERO (Hybrid Ensemble Reward Optimization), which combines rule-based verifier Trule E {0, 1} with a dense reward model via stratified normalization and variance-awar weight- ing. By anchoring soward-model scores to verifier-defined correctness groups and emphasizing informative prompts, HERO preserves the precision and stability of verifiers while supplying dense, trajectory-sensitive feedback, mitigating gradient sparsity and RM-only drift. Empirically, HERO consistently outperforms RM-only and verifier-only baselines across verifiable, hard-to-verify, and mixed regimes and across two backbones, showing that structured hybrid reward design is effective for math reasoning. HERO is a first step toward more general hybrid reward frameworks: it currently relies on reasonably accurate rule-based signals in math domains, and extends naturally to richer verifiers, process-level supervision, and adaptive weighting schemes. We hope these results and analyses provide a useful foundation for future work on combining symbolic and learned feedback for reasoning beyond strictly verifiable settings.
REFERENCES
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, et al. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502. 17387, 2025.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-34 1952.
Rudy Bunel, Matthew Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. Leveraging grammar and reinforcement learning for neural program synthesis. In ICLR, 2018.
10

---

Published as a conference paper at ICLR 2026
Ding Chen, Qingchen Yu, Pengyuan Wang, Wentao Zhang, Bo Tang, Feiyu Xiong, Xinchi Li, Minchuan Yang, and Zhiyu Li. xverify: Efficient answer verifier for reasoning model evalua- tions. arXiv preprint arXiv:2504.1048) 2025a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint wrXiv:2107.03377 2021.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning A survey of long chain-of- thought for reasoning large language models. arXiv preprint (tt.:2503.09566 2025b.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Run-Ze Fan, Zengzhi Wang, and Pengfei Liu. Megascience: Pushing the frontiers of post-training datasets for science reasoning. arXiv preprint arXiv:2507.16812, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint 2tti:::55011.9444. 2025.
Alexander Gurung and Mirella Lapata. Learning to reason for long-form story generation. arXiv preprint arXiv.2503.22822, 2025.
Chaoqun He. Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint 2XX::22002.14008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Yuzhen Huang, Weihao Zeng, Xingshan Zeng, Qi Zhu, and Junxian He. Pitfalls of rule-and model- based verifiers-a case study on mathematical reasoning. arXiv preprint arXiv:2505.22203, 2025a.
Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Xi- aomeng Hu, Yanmei Gu, Yuanyuan Wang, Zhengkai Yang, Jianguo Li, and Junbo Zhao. Rein- forcement learning with rubric anchors. arXiv preprint, 2025b. (tt.:2508.129900
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed EI-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1672)) 2024.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances neural information processing systems, 35:3843-3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Xuzhao Li, Xuchen Li, Shiyu Hu. Yongzhen Guo, and Wentao Zhang. Verifybench: A systematic benchmark for evaluating reasoning verifiers across domains. arXiv preprint arXiv:2507 09884, 2025.
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. Skywork reward-- Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352. 2025.

---

Published as a conference paper at ICLR 2026
Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. vancing frontier math reasoning with post-training and reward modeling. arXiv preprint arXiv:2412.1508. 2024.
Acemath: Ad-
Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler: Surpassing ol -preevie with a 1.5b model by scaling rl, 2025. Notion Blog.
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprini arXiv:2502.0678) 2025.
Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General-reasoner: Advancing Ilm reasoning across all domains. arXiv preprint arXiv:2505, 14652, 2025.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art math- ematical reasoning models with openmathreasoning dataset. arXiv preprint (tl::2500.689) 2025.
Hao Peng. Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou, and Juanzi Li. Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems. arXiv preprint arXiv:2502. 19328, 2025.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501. 142449, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Expanding rl with verifiable rewards across diverse domains. arXiv preprint, 2025. arXiw:2503.23829,
Yunhao Tang, Sid Wang. Lovish Madaan, and Rémi Munos. Beyond verifiable rewards: Scaling re- inforcement learning for language models unverifiable data. arXiv preprint arXiv:2503. 19618, 2025.
Zengzhi Wang. Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid- training incentivizes reinforcement learning scaling. arXiv preprint 2rXiv:5506.2512, 2025.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, al. minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.11333 2025.
Zhangchen Xu, Yuetai Li, Fengqing Jiang, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Tinyv: Reducing false negatives in verification improves for Ilm reasoning. arXiv preprint arXiv:2505.14625 2025.
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-mant technical report: Toward mathematical ex- pert model via seff-improvement, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. / 122, 2024b.
12

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 27Xi::2500.09388, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14470 2025.
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models in the wild, 2025.
Jie Zhang, Cezara Petrui, Kristina Nikolić, and Florian Tramèr. Realmath: A continuous benchmark for evaluating language models on research-level mathematics. arXiv preprint arXiv:2505. /2575, 2025a.
Kaiyan Zhang. Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, et al. A survey of reinforcement learning for large reasoning models. arXiv preprint arXhv:2509.088/ 0088277 2025b.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.03301, 2025c.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint, 2025. arXiv:2505.21492
13

---

Published as a conference paper at ICLR 2026
Appendix
A Experiments
14
A.1 Experimental setup
14
A.2 Data preparation..
16
A.3 More experiments
18
Qualitative analysis
20
B.1 Reward model qualification ability
20
B.2 Qualitative analysis of rule-based verifiers
21
C Limitations and Future Work
The Use of Large Language Models(LLM))
22
A EXPERIMENTS
A.I
EXPERIMENTAL SETUP
Category
Hyperparameter
Value
Train file
OPENMATHREASONING
Max prompt length
1024
Data
Max response length
8192
Filter overlong prompts
True
Base model
Qwen3-4B-BB
LR
1 10-6
KI. loss coefficient B Entropy loss
Actor Model
Use dynamic batch size
True
Rollout engine
vllm
GPU mem utilization
0.6
Rollout
Train rollout
8
Temperature
1.0
Reward
Rule Based
Math.Verit
Reward Model Based
AceMath-RM-7B
Mini Batch size
128
Full Batch size
512 (4 step off-policy)
Critic Warmup
0
Trainer
GPUs/node
4
Nodes
8
Total epochs
20
Clip Ratio
(0.2, 0.28)
Table 5; Key hyperparameters used for GRPO training OPEN (Moshkov et al., 2025) in the verl (Sheng et al., 2025) framework for the Qwen-4B-Ban
14

---

Published as a conference paper at ICLR 2026
Category
Hyperparameter
Value
Train file
OPENMATHIREASONING
Max prompt length
1024
Data
Max response length
4096
Filter overlong prompts
True
Base model
OctoThinker--B-tyyriio
LR
10
KL loss coefficient 3
0.001
Actor Model
Entropy loss
Use dynamic batch size
True
Rollout engine
vilm
GPU mem utilization
0.6
Rollout
Train rollout
Temperature
1.0
Reward
Rule Based
Math.Verify
Reward Model Based
AceMath-RM-78
Mini Batch size
128
Full Batch size
512
(4 step off- off-policy)
Critic Warmup
Trainer
GPUs/node
Nodes
Total epochs
20
Clip Ratio
(0.2, 0.28)
Table 6: Key hyperparameters used for GRPO training OPENMATHRFASONING (Moshkov al., 2025) in the verl (Sheng et al., 2025) framework for the OctoThinneer-B--yyyridBBBBBB
HERO paraaaaeeesss For hybrid reward training for both Qwen-4B- Base and OctoThinker-SB- Hybrid-Base, we set the range parameters a and 3 depending on the task type. For easy-to-verify tasks, we adopt a tighter setting
0.05 exploit the high precision of rule-based verifiers while minimizing noise. For mixed and hard-to-verify tasks, where the reward model contributes more substantially
supervision, relax the range to a 3 0.1 provide richer feedback. For variance-awart reweighting, we fix the weighting bounds wmin = 0.4 and Wmax 3.0, with a steepness parameter
6 in the logistic weighting function. These values ensure that trivial prompts are down-weighted, while highly uncertain prompts-where reward-model scores vary wideie-receive stronger emphasis without destabilizing training.
Training syper-parameters Tables 5 and 6 provide overview of the hyperparameter configu- rations used in our GRPO training runs with -wwen3-4BBBase and OctooTinner--a 8B--asee The tables cover settings across data preparation, actor model optimization, rollout generation, reward specifi- cation, and trainer configuration. They highlight the consistent use of OPENMATHREESION the training corpus, the integration of both rule-based and reward-model signals, and the adoption of scalable rollout and training strategies within the verl framework. Together, these summaries document the experimental setup and ensure reproducibility across different backbone models. In addition, we employ the HuggingFace math verify library to provide standardized rule- biled ver- ification responses against ground-truth answers, which guarantees consistency in supervision across all experiments.
as
Evaluation details. For casy-to-verifyy test sets, we follow Yang et al. (2024b): we use temperature 0.6 and top-p 0.95, generate N 8 candidates per problem, and report pass@ I (first decoded output) averaged over 8 seeds (Table 2). Correctness is determined by math verifier (normalized numeric/string match with task-specific post-processing). For hard-to-verify test sets, the
15

---

Published as a conference paper at ICLR 2026
same temperature and top-p but generate N sample per problem, and rely GPT-4o as a judge to compare model outputs with ground-trung answers. HardVerify-Math (Xu et al., 2025) contains 250 samples and, discussed in Section 3.1, is not particularly strict filter, since math verifier already achieves relatively good performance. To further stress-test hard-to-verify reasoning, we additionally evaluate the TextBookReasoning dataset (Fan et al., 2025); see Appendix A.2.3 for construction details.
A.2 DATA PREPARATION
A.2.1 SUPERVISED FINE-TUNING DATASET PREPARATION
We found that initiating RL training directly from the base model often resulted in instability, partic- ularly in the absence of cold start. For instance, the Qwen3-4B-BB model frequently produced mixed-language outputs and generated irrelevant content during the early stages training. Sim- ilarly, the octothinker base model demonstrated multi-turn behavior, leading to highly variable sponse lengths. To mitigate these issues and enhance the stability of RL training, we first conducted two epochs of cold-start supervised fine-tuning (SFT) before beginning RL. To avoid unintentional distillation from more capable models, we used the base model itself to generate responses. These outputs were then filtered, retaining only samples that satisfied the following criteria: the response contained the correct final answer, was entirely in English, and did not exhibit any unstop issues. For cold start training, we ultimately used only 2,000 SFT samples.
A.2.2 TRAINING DATA FILTER FROM OPENMATHREASONIE
In this paper, we focus reasoning questions that have extractable answers. To this end, exclu- sively utilize data from the Opesonthneasooning dataset, selecting only those examples where the problem type is set to has answer extracted. From the CoT split, extracted 40k examples. For each example, we generated solutions and extracted the predicted answers, which were then verified using math_ verifier (verl). We randomly sampled 2k examples that passed the verifier serve as verifiable training data, and another 2k examples that failed verification as hard-to-verify training samples. These
were combined create mixed training dataset for reinforcement learning (RL) training. We use math verifier (verl) to filter all the samples.
A.2.3 HARD-TO-VERIFY EVALUATION BENCHMARK FROM TEXTBOOKREASONEE
GPT-4o filter prompt for TextBookReasoningg
"I am looking for math questions that are suitable for evaluating math model. Please help select questions that meet the following criteria:
1. The question must be clear and unanbiguous.
2. The question must have specific, factual, and answerable solution (not open-ended subjective).
3. The question must NOT require a proof or explanation of reasoning.
The question must NOT be statement; it should be direct question.
For each question provide, please respond with: \"Conclusion: Suitable\ in the end if the question meets all the criteria above.
\'Conclusion: Not Suitabley"
If the question does not meet the criteria, briefly explain why."
Figure 3: GPT-4o filter prompt for TextBookReasoning.
16

---

Published as a conference paper at ICLR 2026
To construct more challenging and reliable benchmark for hard-to- tasks, employ the BattBookReasoiing benchmark. The following criteria were used to filter and refine the dataset for the evaluation:
1. Pass-through Math Verification Filter
The initial step in filtering was to ensure that the answers in the dataset did not pass the math. verify check, ensuring that the questions and answers involved certain level of complexity or ambiguity that would make them challenging for standard verifiers.
2. Llama 3.3 70B Instruct Model for Natural Reasoning
The dataset was further refined by using the Llama 3.3 70b instruct model to answer natural reasoning prompts. Only the prompts for which Llama could not provide answer were kept for further evaluation. This step ensured that the dataset included questions that required more advanced reasoning abilities, beyond the capabilities of standard models.
3. GPT-4 as the Final Filter
Finally, GPT-4 was used to filter out questions that still met the criteria of being complex and hard-to-verify, GPT-4's ability to handle nuanced reasoning ensured that only the most challenging prompts remained. The prompt is shown as Figure
This process ultimately resulted in refined set of approximately 750 prompts suitable for hard-to- verify task evaluation.
Prompt Template for Hard-to- Verify Tasks Evaluation The evaluation of student answers to these prompts is based on the following template, which uses GPT-4 to compare the student's 's answer against the ground truth:
Math Question Selection Criteria The following prompt was used to select math questions suit- able for evaluating a math model. The criteria for question selection are outlined below:
A.2.4 HARD-TO-VERIE PROMPT
We set the hard-to-verify evaluation prompt shown in Figure 4. This template is designed to assess whether a student's response matches the reference answer without re-solving the question. By explicitly instructing GPT-4o to perform equivalence checking rather than problem solving, the protocol minimizes leakage of additional reasoning and focuses purely on correctness judgment. The structured format, including the question, ground truth, and student answer, ensures consistency across evaluations and reduces prompt sensitivity, making it suitable for benchmarking performance on hard-to-verify tasks.
Prompt Template for hard-to- to-verify tasks evaluation via GPT-4o.
Question: {question)
### Ground Truth Answer: (ground_truth)
Student
(student_answer)
For the above question, please verify if the student's ' answer is equivalent to the ground truth answer.
Do not solve the question by yourself; just check if the student's answer is equivalent the ground truth
If the student's answer is correct, output "Final Decision: Yes". If student's answer is incorrect, output "Final Decision: No".
Assistant:
Figure 4: Prompt Template for hurd-to-verify tasks evaluation via GPT-4o.
17

---

Published as a conference paper at ICLR 2026
Table Comparison with model-based verifiers Qwen-3-4B-Base Results are pass@1 aver- aged over 8 seeds across verifiable and hard-to-verinn reasoning tasks. HERO consistently outperforms both General Reasoner and Qwen2.5-7h 1-B-I-rrr- under all regimes.
Fasy-to-verify tasks
Hard-to-verify tasks
МАTH500
AMC
Minerva
Oly mlympiad
Avg.
HVM
TBR
Avg.
Training with casy-to-verity samples
General Reasoner
82.8
62.8
43.8
45.0
58.6
62.8
54,0
58.4
Qwen2.5-7B-Instrucc
83.7
58.1
43.1
47.4
58.1
68.0
57.1
62.5
(Ours)
85.4
44.5
59.3
66.3
Training with hard-to-verify samples
General Reasoner
78.6
56.3
38.7
41.5
53.8
59.6
48.4
54.0
Qwen2.5-7B In-trrsttt
78.2
60.5
41.8
41.7
$5.6
57.2
51.7
54.5
HERO (Ours)
80.0
40.7
56.8
54.0
56.5
Training with mixed samples
General Reasoner
81.4
61.2
43.2
46.5
58.1
64.0
54.0
$9.0
Qwen2.5-7BB B-TB-tnstruct
80.4
40.5
48.0
58.0
68.8
57.7
63.3
HERO (Ours)
81.6
64.4
42.1
47.0
58.8
71.4
56.7
64.1
A.3 MORE EXPERIMENTS
Training Rewards
Validation Accuracy
0.7
WHarrn AcCuracy
0779
182
Figure 5: RL training curves on MATH500 (easy-to-verify training). Left: mean rule-based reward computed by merh.verrf.er. Right: validation accuracy MATH500.
RL training curves comparison between HERO and baseline. Figure 5 compares the RL dy- namics of the rule-based baseline and HERO when both trained only easy-to-verify samples and evaluated on MATH500. On the left, we plot the mean rule-based reward (computed by math verifier) over training steps. The rule-based baseline starts with relatively low reward and increases slowly, ending around 0.63 after 300 steps. In contrast, HERO climbs much faster and reaches substantially higher plateau (around 0.7 0.75-0.78), consistently staying above the baseline throughout training. On the right, we show the corresponding validation accuracy on MATH500, The rule-based baseline improves from 0.692 at step 0.802 step 300, following a gradual up- ward trend. HERO starts slightly higher at 0.728, quickly jumps above 0.80 by step 30, peaks around 0.852 at step 120, and then fluctuates in the 0.84-0.86 range (e.g., 0.864 at step 240 and 0.842 step 300). Overall, HERO not only achieves higher final accuracy, but also maintains persistent 4-8 point advantage over the rule-based baseline across most of training, indicating that the hybrid reward improves both convergence speed and the entire training trajectory on easy-to-verify data.
Hybrid reward surpasses model-based verifiers across all the three regimes. To further assess whether hybrid reward learning outperform existing model-based verifiers, compare HERO against two representative systems: General Reasoner, a frozen 1.5B verifier model that provides binary correctness judgments (Ma et al., 2025), and W large instruction-tuit
18

---

Published as a conference paper at ICLR 2026
verifier (Yang al., 2024b). As shown in Table 2, HERO consistently achieves higher accuracy than both model-based verifiers under all training regimes. When trained with verifiable samples, HERO attains average score of 62.0, outperforming General Reasoner (58.4) and Qwen2.5-7B-Instneee (62.5) while maintaining greater stability across datasets such as MATH500 (85.4 versus 82.8 and 83.7) and AMC (69.4 versus 62.8 and 58.1). In the hard-to-verify regime, the advantage becomes more pronounced: HERO reaches 56.5, exceeding General Reasoner (54.0) and Qween.5-7B-Instruue (54.5), demonstrating that hybrid reward learning provides more reliable supervision even when symbolic verification is unreliable and model- -ased signals are uncertain. In the mixed setting, which combines both verifiable and open-ended samples, HERO again leads with 64.1, surpassing General Reasoner (59.0) and Swren.55-BB-Instruue (63.3). These results highlight that integrating verifier-anchored and reward-model signals yields not only better accuracy but also more consistent generalization across regimes, outperforming larger model-based verifiers despite using no addi- tional model parameters or external training data. The improvement underscores that structured reward integration, rather than sheer verifier scale, is the key to effective and robust reasoning opti- mization.
The proposed method does not rely on large reward models. A natural ques- tion is whether stronger supervision re- quires scaling up the reward model itself. To isolate this factor, we replace the 7B reward model in HERO with much larger 72B reward model, keeping the verifier and all training configurations fixed. As shown in Table 8, the larger reward model yields only marginal improvement on verifiable tasks (62.8 vs. 62.0) and even slightly underper- forms on hard-to-verify tasks (71.4 vs. 73.2). This confirms that the gains HERO primarily come from its hybrid reward formulation tomuuation-through stratified normalization and variance-awar weight- ing rather than from reward model scaling. Practically, this means that HERO can achieve strong results with compact reward models, offering better efficiency and deployability without sacrificing accuracy.
Table 8: Impact of reward model size: larger RM provides in HERO no remarkable gain over the HERO with smaller RM.
Reward model
Easy-to-verify
Hard-to-verify
AceMath-RM- -MBB
62.0
73.2
CccMath-RM-775
62.8
71.4
Naively combining rule-based rewards and reward signals from reward model- ing does not perform well. A direct in- tegration of rule-based verification and re- ward signals from reward modeling, with- out proper structural alignment, often dis- rupts the stability of training. As shown Table 9, when the weight of the based reward is varied (a = 0.1, 0.5, and 0.9), the combined reward performance remains suboptimal, with scores ranging from 55.9
Table 9: a represents the weight of the rule-based re- ward.
Methods
Easy-to-verify
Hard-to-verify
Reward combine (a=0.1)
57.6
60.2
Reward combine (a=0.5)
58.7
61.4
Reward combine (a=0.9)
55.9
60.4
HERO (Ours)
62.0
73.2
to 58.7 for verifiable tasks and 60.2 to 61.4
for hard-to-verify tasks. Specifically, when the continuous signals from the reward model are naively combined with binary correctness checks, the resulting reward can become noisy or misaligned with the intended notion of correctness. Without explicitly constraining the continuous scores within the rigid framework of the verifier's correctness criteria, reward-man model outputs can be distorted by im- perfections in the model, diminishing both interpretability and precision in the feedback. Moreover, the lack of safeguard to differentiate true positives from noisy results can lead the model exploit unintended patterns, which may not align with human expectations. As result, an unrefined fusion of these two reward signals can dilute the benefits both approaches, destabilizing the learning process.
Reward models hack faster on hard-to-verify samples. Since the reward model (RM) is trained on catcome-bassed verifiable samples (Liu et al., 2024), is important to examine its behavior across datasets with varying levels verifiability. We evaluate four datasets: DAPO (Yu et al., 2025), which is easy to verify; OpenMath Verifiable, which passes the math verifier; OpenMath Non-
19

---

Published as a conference paper at ICLR 2026
Reward
MAT Accuracy
Opeahlat Nix
OocaMach Vartable
DATO
Openstads Vaniiabl
MXTIIS Accuncy
Revaa Maia
700
350
Training
Trxtning
Figure 6: Reward model qualification ability on mixed groups: (a) distribution of AUROC scores, (b) AUROC box plot, (c) cumulative distribution of AUROC, and (d) AUROC performance categories.
Verifiable, which is harder to verify; and OpenMath Mix Verifiable, which combines both. As shown in Figure the RM rapidly increases the reward mean across all datasets, with the sharpest gains OpenMath data, the reward mean climbs steeply from below 5 to over 30 within the first 100 training steps, and peaks above 40 by step 150. However, MATH500 accuracy collapses shortly after, dropping from around 0.75 at step 50 to below 0.2 by step 100, and effectively to zero by step 150. A similar trend appears on Mix Verifiable: accuracy initially rises about 0.8 step 100 but then crashes nearly zero by step 150, despite the reward mean continuing to rise steadily past 35, In contrast, OpenMath Verifiable shows slower but steadier progress: rewards grow more gradually, and accuracy improves to about 0.8 by step 120 before stabilizing without collapse. DAPO also exhibits stable optimization, rewards increase moderately. These results highlight
Veriiiiiie and OpenMath Mix Verifiable. For example,
Non- Verifiable
with accuracy consistently around clear mismatch: rapid reward gains on hard-to-verify tasks are not evidence of genuine reasoning improvement, but rather reward hacking that leads to catastrophic accuracy collapse. This illustrates the brittleness of relying solely dense reward models and motivates hybrid reward frameworks that combine cerificr-anchore reliability with the nuance of dense signals.
B QUALITATIVE ANALYSIS
B.1 REWARD MODEL QUALIFICATION ABILITY
To better understand the reliability of reward-model supervision, analyze its ability approxi- mate the verifier signal as a binary classification task. We randomly take all the rollouts from one step (the 250 for the verifiable samples training) during the training. Specifically, we treat the reward model's 's raw scores as logits and the verifier's ' outputs as ground-truth binary labels, then compute AUROC statistics measure discriminative power.
Figure shows four complementary views. The histogram (top-left) reveals a strong skew toward high AUROC values, with mean of 0.79 and median of 0.92, indicating that the reward model often ranks correct responses above incorrect ones. The box plot (top-right) highlights robustness but also exposes several low outliers where the model fails to separate classes. The cumulative dis- tribution (bottom-left) confirms that roughly 80% of groups achieve AUROC above 0.7. Finally, the performance categorization (bottom-right) shows that 56.8% of groups reach "excellent" AUROC (2 0.9), while only 13.7% fall into the "random/poor" range (0.4-0.6).
These results suggest that although the reward model is not perfect, provides reliable ranking signals in the majority of cases. Importantly, this supports the of dense reward signals to refine learning within verifier-defingt groups: while the verifier anchors correctness, the reward model adds discriminative power that helps differentiate among responses of varying quality. The presence of failure cases further justifies our hybrid framework, which uses stratified normalization to bound reward-model signals within verifier groups, ensuring stability when AUROC is low.
20

---

Published as a conference paper at ICLR 2026
Scores Groups Only)
Groupz Onlyy)
Mears
1.0
Median: 517
0.5
30
0.6
0.4
10-
0.0
Distribution
Groups Onlyy)
Performance Categories (Mixed Groups Only)
10 7-4.5
(3.50.71
0.5
27.4%
Bandorihoo
23
Com
0.1
Ss 8%
Dawellek ted 9)
0.0
0.0
Figure 7: Reward model qualification ability on mixed groups: (a) distribution of AUROC scores, (b) AUROC box plot, (c) cumulative distribution of AUROC, and (d) AUROC performance categories.
B.2 QUALITATIVE ANALYSIS OF RULE-BASED VERIFIERS
Table 10 highlights representative behaviors of rule-based and model-based verifiers. math.py is overly strict, failing on minor formatting variations such as boxing or punctuation (Rows 1-2), while math verify. improves recall through normalization. The Math-Verify library handles simple sur- face mismatches but struggles with structural differences like disjoint ranges or multiple valid tuples (Rows 4-5). contrast, o3 the most permissive: credits partially correct sets (Row 3) and para- metric families with renamed symbols (Row 6), which increases coverage but risks over-crediting. These cases illustrate the precision- recall trade-off: rule-based verifiers enforce exact symbolic cor- rectness but miss semantically equivalent or partially correct answers, whereas model judges offer flexibility at the cost reliability. This motivates our hybrid design: HERO anchors dense reward signals rule-based correctness, ensuring robustness to format variance, while leveraging model- or RM-derived scores to provide graded feedback on harder cases involving subsets, orderings, or parametric equivalence.
C LIMITATIONS AND FUTURE WORK
While HERO demonstrates clear advantages over RMM-nly and verifier-only training, several limi- tations remain. First, the method depends on the availability and reliability of rule-based verifiers: when these brittle domain-mismmtched,, the partitioning into correctness groups may be biased, weakening the benefits of stratified normalization. More broadly, our method is explicitly designed for settings where a rule-based signal and a dense reward-model signal can be combined; when such a verifier is unavailable or highly unreliable, the current HERO formulation is not directly appli- cable. Second, because the reward model is trained primarily on outcome-based verifiable data, can become miscalibrated on harder, verifaiiee formats, and although our framework constrains its scores, residual bias or spurious correlations may still be exploited. Third, HERO introduces sensitivity to hyperparameters such as (a:, 3) and the weighting slope k, and increases training over- head due to concurrent verifier and RM calls. Finally, evaluation on non- non-verifiable tasks often relies on LLM-as-judge protocols, which introduce prompt sensitivity and annotation noise. Future work will focus on improving verifier coverage with hybrid symbolic- learned approaches, incorporating process-level supervision capture reasoning quality beyond final answers, and developing adap-
21

---

Published as a conference paper at ICLR 2026
Ground
Prediction
math.p mathhveriff-py(ved)) Mathaverify library
f(x) 2z
Vaoxed (f(o) 2x)
X
(6, (6.3), (9,3), (9,
[(6,3)), Mboxed ((),3)), Vaoxed ((5,5)). \boxed ((54,5))
(0, 1, (1,00,,
(1,11,),,,
(1 : vt)
Final Answert Mooxed ((),,, Vooxed ((-1,-1,01), Moxxed
(- J/ V3 vT
((//ssrrt (3), (3), (3))), {(11/sert (3),-1/\ (3).-1/\sart (2))).
13, 1,
Answer:
1-2,-1,0,1,2) and Maoxed 13,111,,,
11,7100 103, 105), (3,5, 101, 107)
Final Answer: possible lists are Vboxed ((3,5,181,187) and (01.7, (03,7,13.1185)
[(x) az + (where b is arbilrary inleger, positive integer with mbo(a)=0)
Final Answer: \boxed (f(r)rcnid). where 18 100(100) and integer
x
arbitrary
prime
Table 10: Examples demonstrating agreement between different math verification tools.
tive range and weighting schemes that calibrate dense signals online. These directions can further strengthen the stability and generality of hybrid reward frameworks for reasoning.
Mathematical reasoning benchmarks are a natural first testbed for HERO, as they provide mature, gigh-rrcisson verifiable rewards (e.g., programmatic checks for final answers) and are widely used in recent RLVR systems such as DeepSeek-R1. 1. For this reason, we intentionally restrict our em- pirical study to math reasoning, where HERO's assumptions are well satisfied and comparisons to existing verifier-based RL methods are most meaningful. In contrast, many mon-mathematiall, open- ended, or multimodal tasks currently lack clear, high-pree eigh-precision rule-based verifiers, making them less aligned with our present problem setting. We therefore do not claim that HERO, instantiated this paper, already solves open-ended generation; instead, these tasks remain outside the scope of experiments. Nonetheless, HERO itself is agnostic the specifie form of r in principle, any structured checker (such as code-execution tests, safety or formatting constraints, or even rubric- guided LLM-as-judge signals) could play this role. Systematically extending HERO such "soft" verifiers for open-ended and multimodal reasoning, and studying the resulting trade-offs between stability, coverage, and bias, an important direction for future work.
D THE USE OF LARGE LANGUAGE MODEL)(LLM)
In our project, we use LLM for writing polishing.
22


Reviews:
[{"review_id": "zYNaYaWIZg", "reviewer": "Reviewer_vePn", "summary": "The paper proposes HERO, a reinforcement learning framework designed to enhance reasoning capabilities in large language models (LLMs) by integrating sparse, binary verifiable rewards with dense, continuous scores from reward models. The core innovation lies in addressing the brittleness of binary verifiers and the misalignment of reward models through two mechanisms: stratified normalization, which bounds reward model scores within verifier-defined correctness groups, and variance-aware weighting, which emphasizes challenging prompts with high score variance. Built on GRPO, HERO aims to provide stable, informative supervision for mathematical reasoning tasks. Empirical results on verifiable benchmarks and hard-to-verify ones demonstrate that HERO outperforms baselines like RM-only RL and verifier-only methods.", "questions": "- How would HERO perform on open-ended reasoning?\n- How sensitive is HERO to the choice of α and β?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and easy to follow.\n- The stratified normalization and variance-aware weighting are intuitive and simple to implement.\n- Comprehensive experiments across multiple reasoning benchmarks and architectures demonstrates the effectiveness of the proposed method.", "weaknesses": "- The “hybrid reward” idea that combines verifier and reward-model signals is conceptually intuitive and has already been explored in prior work. \n- Moreover, the conclusions that (1) function-based rules exhibit high precision but low recall, and (2) reward models have lower precision, are not novel and have been thoroughly demonstrated in prior studies (e.g., arXiv:2505.22203).\n- The evaluation scope remains narrow: the method’s generalization to non-mathematical, open-ended, or multimodal reasoning tasks is not shown.\n- While ablations are mentioned, the paper lacks detailed quantitative results for key hyperparameters such as α/β ranges or variance-weighting factors, limiting the interpretability and reproducibility of the method’s stability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qfnF8MIDBt", "reviewer": "Reviewer_rKUs", "summary": "This paper proposes a hybrid reward to integrate reward model and binary verifier scores. The hybrid reward is shown to be effective on multiple models across multiple tasks including verifiable and hard-to-verify ones.", "questions": "Please see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem is well motivated. \n- The proposed method is intuitive and simple to implement.\n- Extensive experiments to demonstrate the effectiveness of HERO.\n- The paper is well-written and easy to follow.", "weaknesses": "- The min-max normalized reward depends on the result of RM and r_rule. How does HERO handle reward drift/misalignment inherited from RM and r_rule?\n- How effective is HERO for non-verifiable tasks such as instruction following? Will HERO harm model performance on these tasks?\n- The choice of parameters $\\alpha$ and $\\beta$ are highly depend on tasks. It would be great to provide some guidelines on choosing these parameters for task agnostic models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Maq7Aylqs", "reviewer": "Reviewer_VZE7", "summary": "This paper proposed HERO. The core idea is to combine sparse but reliable verifier-based binary rewards with dense but noisy reward-model scores. HERO proposes Stratified normalization, which anchors RM scores within verifier-defined correctness groups to preserve semantic correctness, and Variance-aware reweighting, which emphasizes prompts with higher score variance. Experiments on multiple math reasoning benchmarks (e.g., MATH500, AMC, Olympiad, HardVerify-Math, and TextBookReasoning) show that HERO outperforms both verifier-only and RM-only baselines across verifiable, hard-to-verify, and mixed settings.", "questions": "1. Could you justify Figure 5 and provide additional training-step vs. benchmark performance tables/curves to show that HERO consistently outperforms baselines, rather than only at a specific checkpoint?\n\n2. Could you provide more explanations and justifications about why gains of HERO vary dramatically across different training regimes in Table 2? How does the ratio of verifiable/non-verifiable training samples affect HERO gains over baselines?\n\n3. Could you report results using both rule-based and LLM-based verification for OlympiadBench to confirm accurate and consistent evaluation results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clear technical insight: preserves correctness while enriching reward density.\n\n2. Comprehensive experiments on multiple model (Qwen3-4B, OctoThinker-8B) and datasets, with detailed ablations and parameter sensitivity analyses.\n\n3. Writing and presentation are clear.", "weaknesses": "1. The paper provides only one limited RL training curve (Appendix Fig. 5), where the hybrid reward benefit is unclear and performance even collapses at later steps. This is confused. The caption is also wrong for Figure 5. More RL training curves results are needed to confirm the consistent HERO superiority over baselines instead of only showing performance on some/final checkpoint.\n\n2. In Table 2, the gains of HERO vary dramatically across training regimes. When trained on verifiable samples, HERO outperforms baselines by over 10 points, yet when trained on hard-to-verify samples, the average improvement shrinks to <3 points. In the mixed regime, the improvement flips again: small on verifiable tasks (~4 points) but large on hard-to-verify tasks (~10 points).\nThese results raise questions about what drives the effectiveness of the hybrid reward. Without a clearer explanation, it is difficult to determine whether HERO’s benefit stems from the hybrid design itself.\n\n3. Potential unreliability in Olympiad benchmark evaluation. Many Olympiad problems are themselves hard to verify with math_verify, which may distort scores. Rule-based and llm-based ablations should be done on the Olympiad problems to see whether the evaluation scores are accurate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GGioH3qrWw", "reviewer": "Reviewer_Bqqo", "summary": "This paper introduces HERO (Hybrid Ensemble Reward Optimization)- a framework that integrates dense signals from reward models with binary-valued feedback from rule-based verifiers. The paper systematically reports the merits of each individual approach while highlighting its limitations; they further caution against naively combining the two approach. The proposed solution uses a `stratified' strategy to leverage expressiveness of reward models within structures imposed by the verifier. Experimental evaluations across a wide range of regimes in math reasoning tasks demonstrate that HERO performs significantly better than reward model only or verifier only baselines.", "questions": "Please see Weaknesses, above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(+) HERO’s use of stratified normalization together with a difficulty-aware weighting scheme simultaneously enables overcoming limitations of binary-valued feedback signals and misaligned scores assigned to incorrect responses. \n\n(+) The proposed mechanism has been evaluated on math reasoning benchmarks with a wide range of objectives ranging from exactly correct to partially correct solutions while additionally taking into consideration the solution format. \n\n(+) Strong results in hard-to-verify tasks compared to reward model-only and verifier-only schemes. Extensive ablations demonstrate the importance of each component of HERO in achieving this improved performance. \n\n(+) Limitations of rule-based verifiers and reward modeling are demonstrated separately across multiple verifiers and a reward model for tasks where correctness can be difficult to verify, justifying the `hybrid’ approach adopted in this paper. \n\n(+) The authors further demonstrate (experiments reported in the Appendix) that naively combining reward signals with rule-based verification might not be sufficient to result in improved performance. \n\n(+) Overall, this paper is extremely well-written, and the authors systematically lay out the merits of each component approach, highlight their limitations, and point out how the proposed hybrid approach combines the merits while overcoming shortcomings of the individual approaches. Experiment results provide strong backing to the hypotheses presented.", "weaknesses": "(-) The `ensemble' characteristic of HERO is not immediately apparent from the writing of the paper. Additional clarification on this aspect will be helpful. \n\n(-) The explanation for the design principle guiding HERO (reward signals as guides to augment rule-based methods) mentioned in Lines 192-196 of Sec. 3.2 might be helpful in understanding the shape of the hybrid reinforcement graph relative to the other two graphs. \n\n(-) The discussion on future directions in Lines 485-486 can be improved. Perhaps some of the material from Appendix C can be moved to this part of the paper. \n\n(-) Only two sets of values for $\\alpha, \\beta$ in Eqn. (3) have been considered (i.e., $\\alpha = \\beta = 0.05$, $\\alpha = \\beta = 0.1$). It might be interesting to see the specific role of each parameter on the performance of HERO, and the possible intuition for such performance. \n\n(-) Line 747 in Appendix: ables —> Tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)