Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models
Authors: Guanxu Chen
Yafu Li
Yuxian Jiang
Chen Qian
Qihan Ren
Yang JingYi
Yu Cheng
Dongrui Liu
Jing Shao
Venue: ICLR 2026 Poster (2026)
Keywords: language models
reinforcement learning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) for large language models (LLMs) has achieved remarkable progress in enhancing LLMs’ reasoning capabilities on tasks with clear correctness criteria, such as mathematical reasoning tasks. Several training metrics, such as entropy or response length, have been observed to correlate with different reasoning behaviors in reinforcement learning. Prior approaches incorporate such priors through reward or advantage shaping, which often relies on hand-crafted penalties and preferences (e.g., higher-is-better or lower-is-better). However, without careful hyper-parameter tuning, these directional priors can be overly biased and may lead to failure. To this end, we introduce ***C****onditional adv****AN****tage estimati****ON*** (***CANON***), amplifying the impact of the target metric without presuming its direction. Specifically, *CANON* regroups the sampled responses into two groups based on the higher or lower value of a target metric, measures which metric trend contributes to better performance through inter-group comparison, and identifies the better response within the same group. In summary, *CANON* based on entropy consistently outperforms prior methods across three LLMs on both math reasoning and high-complexity logic tasks. When applied to response length, *CANON* further improves token efficiency, yielding a more favorable Pareto frontier in the performance–cost trade-off.

Published as a conference paper at ICLR 2026
CONDITIONAL ADVANTAGE ESTIMATION FOR REINFORCEMENT LEARNING IN LARGE REA-
SONING MODELS
Guanxu 2h 2.1+ Yafu Li
Yuxian Jiang'
Chen Qian' Jing Shao't
Qihan Ren
JingYi Yang3. 3.1 Yu Cheng
Dongrui Liu't
1 Shanghai Artificial Intelligence Laboratory, 2 Shanghai Jiao Tong University, 3 Fudan University, Renmin University of China, The Chinese University of Hong Kong Im.cex& sjt.euu..
yafuly@ gmail.com
[liudongrui. shuojing/ppjlaborgg ppllab.org.nn
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) for large language mod- els (LLMs) has achieved remarkable progress in enhancing LLMs' reasoning capabilities on tasks with clear correctness criteria, such as mathematical reasoning tasks. Several training metrics, such as entropy response length, have been observed to correlate with different reasoning behaviors reinforcement learning. Prior approaches incorporate such priors through reward advantage shaping, which often relies on hand-crafted penalties and preferences (c.g., higher-is- better lower-is-better better) However, without careful hyper-parameter tuning, these di- rectional priors can be overly biased and may lead to failure. To this end, we introduce Conditional advANtage estimatiON (CANON), amplifying the impact of the target metric without presuming its direction. Specifically, CANON regroups the sampled responses into two groups based on the higher lower value of a target metric, measures which metric trend contributes to better performance through inter- group comparison, and identifies the better response within the same group. In summary, CANON based on entropy consistently outperforms prior methods across three L.LMs on both math reasoning and high-complexity logic tasks. When applied to response length, CANON further improves token efficiency, yielding a more favorable Pareto frontier in the performance-cost trade-off.
INTRODUCTION
Recently, Large Reasoning Models (LRMs) such as Gemini 2.5 Pro (Comanici et al. 2025), DeepSeek- R1 (Guo et al.) (025), and OpenAI-ol (Jaech et al.) (024), continue to push the boundaries of performance on reasoning tasks. A key technique driving this success Reinforcement Learning with Verifiable Rewards (RLVR), which enables models to refine answers through multi-step reflection. Algorithms designed for RLVR, most prominently GRPO (Shao et al. 2024) and its variants (c.g., DR.GRPO, Liu et al. (2025a)), have become central to achieving superior performance.
In previous works, some training metrics are observed to be closely correlated with model behavior, which can guide the training process and improve L.MMs' performance (Hassid et al. 2025 Gandhi et al. 2025 Wang et al., 2025). To incorporate such human prior, some methods integrate these metrics through reward shaping (Arora & Zanette 2025 Luo et al. 2025) and advantage shaping (Chen et al., 2025b; Cheng et al. 2025) to guide the model's 's reasoning behavior. For example, an over-length penalty is used to boost reasoning efficiency, and the entropy signal leveraged to maintain exploration for better performance.
However, these methods usually introduce human priors by adding penalty and reward terms, which hold handcrafted priors that specific metrics are either to be higher-is-better or be lower-is-better Without careful hyper-parameter selection, these priors can be overly biased and drive specific metries up or down directly, thus failing to enhance performance robustly. Simple handcrafted priors towards one specific direction are hard to work in different scenarios. For instance, higher-entropy responses tend to be exploratory and may correctly answer complex questions, whereas lower-entropy responses
"Ou code is available at C CANON
1 "ooresponning Author.

---

Published as a conference paper at ICLR 2026
exhibit higher certainty and achieve greater accuracy on most questions within their capability (Cheng iet al.) 2025 Prabhudesai et al. 2025| Wang et al. 2025). Therefore, we aim to amplify the impact of specific metric changes without presupposing preferences, naturally identifying inherent tendencies in model rollouts that can be leveraged to facilitate learning of beneficial behaviors, such as enhancing exploration or improving reasoning efficiency.
To this end, we regroup the sampled responses into two groups based on the higher or lower values of a given metric during the process of RLVR training. Specifically, we can sort the sampled responses according to the value and split them into two groups. Based this, we propose Conditional advANtage estimatiON (CANON), which computes the inter-group advantage by comparing a response with the group that it does not belong to, and gets the intra-group advantage across its own group conversely. The inter-group advantage reveals which trend of metrics leads to higher accuracy. Meanwhile, the intra-group advantage identifies better responses within the same group.
Taking the metric of entropy an example, if groups with lower entropy (i.e., higher certainty) yield higher average rewards, the inter-group advantage tends to select correct responses with low entropy, efficiently exploiting existing features to boost performance. In contrast, correct rollouts with higher entropy receive more advantages in the intra-group comparison because the average reward of their group is lower, thereby encouraging truly effective exploration. We theoretically prove that when the two groups have equal size, the inter-group advantage amplifies the impact of the grouping metric on the advantage computation. In this setting, DR. GRPO can be formulated as a uniform weighting of these two advantages, which is special case of CANON.
We consider the metrics of generation entropy and response length, evaluating the effectiveness of CANON on three open-weight LLMs across six math reasoning benchmarks and three challenging logic reasoning tasks. Empirical results show that emphasizing the inter-group advantage based on entropy yields 1.9- 1.9-point accuracy gain on math tasks. In contrast, for high-complexity reasoning problems, the intra-group advantage proves crucial, achieving 5.2-point improvement on the most challenging subset. Through scheduling of these advantages, CANON further achieves a superior and comprehensive performance across three models and two tasks. Furthermore, CANON based on response length substantially enhances reasoning efficiency, establishing a new Pareto fronticr in the eerformancc-efffciennyy trade-off. In low-token-budget scenarios for math tasks, it achieves 2.63x higher performance and reduces token consumption by 45.5% % at the same performance level.
2 RELATED WORK
Advantage Estimations in Reinforcement Learning. In PPO, the advantage estimation is provided by Generalized Advantage Estimation (GAE, Schulman et al. (2015)). avoid the computational cost of the critic model, several methods, such as ReMax (Li et al. 2023), RLOO (Ahmadian et al., 2024). GRPO Shao et al. (2024), and REINFORCE++ (Hu) (025). y, utilize alternative techniques like baseline reward and group-relative rewards for advantage estimation. ReMax compares the rewards with the baseline reward from the greedy decoding response. REINFORCE++ estimates the advantage by the normalization operation across the global batch for all queries. RIOO and GRPO estimate the advantage in group relative manner. RLOO computes the average rewards of all other solutions in the group as the baseline reward, and GRPO utilizes the normalized rewards among the sampled solutions as the advantage estimation. Compared to GRPO, our method splits sampled responses into two groups based on specific conditions and selects the appropriate condition through inter- and intra-group comparisons, thereby efficiently optimizing key patterns that boost task performance.
Reinforcement Learning with Verifiable Rewards. RLVR leverages the existing RLHF objective (Schulman et al., (017) but replaces the reward model with a verification function, which is available in domains with verifialle answers, such as mathematics reasoning tasks (Guo et al.) 2025) Lambert et al. 2024). Yu et al. (2025); Liu et al. (2025b); Chen et al. (202a)) consider the importance sampling techniques and contribute novel training paradigms and optimization objectives for better and more
stable reasoning capabilities. Due to the sparse rewards during training. past methods utilize not only
accuracy-based rewards but also explicitly integrate additional signals through reward shaping (Arora
& Zanette, 2025 Luo et al. (025) and advantage shaping (Chen et al. 2025b Cheng et al.
2025)
to guide the model's 's reasoning and reflection. Arora& & Zanette (2025) and Luo et al. (2025)
utilize
an over-length penalty to boost reasoning efficiency. CCen et al. ( (025b) and (Cheng et al.
2025)
consider the entropy as a measure of exploration and reshape the ad advantage computation. Gandhi
et al. (2025) also observes four key cognitive behaviors of initial reasoning behaviors and strengthens
2

---

Published as a conference paper at ICLR 2026
Selution
Reward
Metric
Inter-Group
Intra-Grae
R2
Fly
A
Policy Model
Verifier
Conditional Regrouping
Componsoo
00-1
Ro-1
No-1 -
A
0G
Ci
AG
Conditional Regrouping
Metrics (Lengtr/Entropyl.
Comparison
Inter- Group
Ra
Meani
A
Sorting
PG--1
/th
(-iiher
Meont
A
Regrouping
C2
C
R₃
Intra-Group
Overview of CANON. CANON regroups all the sampled responses based on the value of specific metric, and computes the advantages through inter-group and intra-group comparison.
Figure 1:
the capacity for saff-impproveent. However, these methods usually introduce human priors by adding penalty and reward terms, which hold handcrafted priors that can be overly biased and may fail to enhance performance without careful hyper-paramete selection. Our work amplifies the impact of specific metric changes without presupposing preferences, leveraging them to facilitate learning of beneficial behaviors.
PRELIMINARIES
Proximal Policy Optimization (PPO, Schulman et al. (2017) is a widely used method for policy optimization of LLMs. PPO utilizes the clip mechanism to update policy stably. PPO maximizes the following optimization objectives.
lol Σ
min (o.,())A,,, (ro..
(8) Egg DolgDDDonnan on (19)
(I)
|0| t=1
where Thas and me are used to denote the policy model before and after the update. is query sampled from the data distribution D, and the output is generated by me The clipping function with clip ratio e computed as clip (x) = max (min(x, a), b) and the importance sampling ratio at mo =0(0)1g.0.0)
time step is defined (f) Regs ( (9.19.0c)
To avoid the computational cost the critic model, GRPO (Shao et al. 2024) estimates the advantage in group relative manner. They sample G different solutions for the current query q as the group G = {olo ~ Teaa (-14)), and calculate the normalized rewards as advantages within the group G
Ro - mcan (Ro lo E G₂H)
AGRPO
std( (Ral /o G,))
Due to the success of DeepSeek-R1, several studies have proposed improvements based on GRPO. DR.GRPE (Liu al., 2025a) uses the GRPO advantages without standard deviation normalization and develops a leven-level loss without length bias.
CONDITIONAL ADVANTAGE ESTIMATION
Group-based advantage estimation methods, such as GRPO, typically use the average reward of all sampled responses within the group as a baseline reward. This may fail to provide clear feedback signal for policy optimization due to the ambiguity which performs conditional regrouping by splitting all sampled responses into groups based on the value of a specific metric. Leveraging these two groups, inter-group advantage identifies the metric trend that yields higher accuracy through cross-group comparison, while intra-group advantage
the comparison target. We propose CANON,
3

---

Published as a conference paper at ICLR 2026
selects superior responses within the same trend and prioritizes correct answers from the group with a lower average reward.
4.1 CONDITIONAL REGROUPING
To explicitly introduce a comparison target, we regroup all the sampled responses based on specific conditions. Given any condition c, denote the set of all outputs for the current query q that satisfy this condition the sampled group G, as C = folo satisfy c, o E G, The set of outputs that do not satisfy the condition can be denoted by C- = Gg \ C.. In this work, we focus on studying the relative conditions given by the training metrics, such as the entropy and length of the sampled responses. Specifically, we divide the responses into two non-overlapping groups based on the value of the metrics, as shown in Figure
4.2 ADVANTAGE ESTIMATION BASED ON REGROUPING.
Given two groups, we can compute the inter-group advantage through comparison between different groups.
R, mean ( E E
A inter 9,o,1
R. mean( ( E Gy if, G7
Meanwhile, we also compute the intra- intra-group advantage by comparing each response with the mean reward of its own group.
R. mean ( E Gf G
A
intra
q.o,3
R. mean( (R E G,
Although this may appear similar the estimation of DR GRPO within a smaller scope, due the differing average advantages between groups, the intra-group advantage prioritizes correct responses from the group with a lower average reward (1 mean( (f RRo' lo G 1 mean { Ro Jo' GT) } when mean( (Ro E G < mean( (Rorr E G7 ). We can further combine the above two advantages into unified formulation.
CAMON = 9,o,t
inter
q,o.t + (1 14) jinira
9,0,1
(5)
where y controls the balance between the inter-group and intra gera-group advantage. Figure I strates a concise case of the computation of CANON.
demon-
To ensure that the advantages introduced by conditional regrouping provide clearer contrastive signal, we theoretically analyze the situations under which inter-group advantage, compared to DR.GRPO, yields a stronger advantage signal response to reward gaps under specific conditions.
Theorem 1 (Situations with clearer advantage signal (proved Appendix E
Suppose that condition is based on numerical comparisons and can be derived through sorting of metrics. Further assume that the sampled response o to query q satisfy condition with probability p E (0, 1), and E, satiefy e RR,] + E, not satiafy el [Ro). Then, we have:
Ammer ADR.GRPO
> 1, only when |C C7 if is constant.
(6)
Based on Theorem [ we divide the responses into two equally sized groups. In this way, DR.GRPO can be expressed special case of this unified form when μ 0.5.
4,0.t R. mean( (Role d E G²))
1
Ainer
iintra 2 Aqo,tt
1
2
Moreover, rather than a direct numerical amplification, CANON amplifies only the advantage at- tributable to the metric used for grouping, without amplifying the influence of other factors.
4

---

Published as a conference paper at ICLR 2026
Table I: Overall performance based on Owen2.5-Math-77. We compare with the following baselines: (I) Twen2.5-Math-77 --sttr--nnttt (Qwen-Instruct), (2) prior advantage estimation methods. All models are evaluated under unified setting. Bold and underline indicate the best and second-best results, respectively.
Math Reasoning
High Complexity Reasoning
Model
AIME
AIME 25 Olympiad AMC MATH-500 GSM8K
Tokens
Acc
Mid
Large
XLarge
Tokens
Acc
Base
16.0
26.4
41.6
61.2
61.6
2046
35.8
0.0
0.1
3303
0.2
Instruct
10.7
9.7
39.7
49.3
82.2
94.8
1077
47.7
11.6
6.2
3,5
2647
7.1
Previous Advanlage Estimation
23.3
90.3
54.3
R++
20.3
19.7
45.8
58.3
82.6
90.0
4107
52.8
33.8
11.9
3.3
9923
16.3
RLOO
25.0
18.7
51.1
64.3
84.0
91.0
2537
55.7
33.9
14.4
5.8
10610
18.0
GRPO
22.3
47.3
60.6
83.8
90.8
3730
53.8
31.5
5.2
9406
17.2
DR GRPO (M = 0.3)
27.7
20.3
48.4
63.4
83,2
91.1
1522
55.7
39.2
24.4
15.1
4896
26.2
Entropy-related Basclines
Entropy Adv
26.7
16.7
50.8
65,3
87.6
90.8
2389
56.3
30.8
17.1
7.5
8207
18.5
Clip-Cor
26.3
49.0
56.1
Our Methods (Conditional Groups based on Lenggh))
MANON-Tntra
21.7
19.0
49.9
63.0
86.2
92,2
2176
55.3
41.8
25.6
14.7
4364
27.4
CANON-Tnter
27.3
19.3
47.6
64.2
82.6
91.0
1008
55.3
42.7
28.6
17.1
3652
29.5
Methods (Conditional Groups on Entropy)
CANON-Iatra
25.0
16.0
48.9
62.7
84.4
91.1
2959
54.7
39.1
27.8
20.3
3101
29.1
CANON-Inter 32.7
18.7
51.7
64.2
87.0
91.1
1466
57.6
36.3
25.8
14.9
4415
25.7
CANON-Dynamic
63.3
56.7
29.2
Theorem 2 (Selective amplification based on specific metrics (proved in Appendix Eb). Consider independent conditions C1 and 0z, and their corresponding sets C1 and C2 (i.e., P(o E C1 n C2lg. 8) = P(o Cilg, P(o E C2l9. f)). When we fix the condition C1 then for any value of the performance gap a24, 02- and the probability P(o E C2lg. 0) that induced by whether 02 is satisfied, we have
Alnter based on
9,0,t
is a constant.
(8)
A GRPO
which says CANON based on the condition ej will not amplify the influence of another independent condition c2.
Therefore, CANON, when grouped by a specific metric, amplifies the influence of that metric during training, yet it does not predefine preference for the magnitude of the metric. This design allows it to incorporate human priors while mitigating bias, which fully aligns with our original motivation.
4.3 ALIGNING WITH TRAINING TARGET THROUGH WEIGHTED ADVANTAGE
According to Section 4.2 the selection berween different trends of metrics only takes place in the inter-group advantage. By weighting different conditions within the inter-group advantage calculation, this enables fine-grat tne-grained control over the trend of metrics with only tiny differences compared to DR. GR.O. For instance, by slightly reducing the weight of longer responses, CANON can accomplish reasoning of high token efficiency through the RL process. Specifically, the inter-group advantage in the Eq. 5 should be replaced with Ainter q.o.t. where is the weight of specific group, and A q.otta is
defined as:
-f
{
Rg a * mean( E (tteelldGtttttttt..ttttttt Gt
A iinter
(9)
R. me E (,
For example, setting a as 0.9 can achieve substantial length reduction with little performance drop. where C considered the group with longer responses.
EXPERIMENTS
The empirical evaluation of CANON consists three parts. Firstly, we demonstrate the effect of intra-group and inter-group advantages, respectively, across six math reasoning benchmarks and one
5

---

Published as a conference paper at ICLR 2026
Performaino
Perforrancc
Rewards
Casola emr godbd))
CANoleaner (m1
Training Steps
Training Steps
Training Steps
(a)/rrinning score.
(b) Test performance of math tasks.
(c) Test performance of logic reasoning tasks,
CoRDDe-rtraa lawd ciaoo
1 rethinking
Positive Gain
Longth
Entropy
Training Steps
Training Steps
Training Steps
(d) Response length.
(e) Per-token generation entropy.
(1) Benefits of more rechinking.
Figure 2: The training dynamics and average test performance of CANON Inter, DR.GRPO, and CANON-Intra.
high-complexity logic reasoning benchmark. In the second part, we perform several scheduling tricks to get the frontier in both tasks. At last, by weighting the longer responses with 1, we achieve efficient reasoning that reaches a better Pareto frontier.
5.1 PERFORMANCE OF INTRA-GROUP AND INTER WOTER-GROEE GROUP ADVANTAGES.
Training Setup. We select the response length and the per-token generation entropy, respectively, to regroup the sampled solutions. We use subset with 45k prompts from 1attt-
(Hugging Face 2025) that filtered and constructed by Yan et al. (2025). Following DR.GRPO (Liu et al. 2025a) and DAPO Yu et al. 2025), we correct the response-level length bias and utilize the clip-higher strategy (Chigh 0.28) for all experiments. We also remove both the KI. loss and the entropy loss. We sample 16 responses per prompt and use temperature=1.0 for rollout generation. Our rollout batch size is 512, and the train batch size 32. The responses the same prompt are separated into two evenly sized groups by sorting ordinal variables. We conduct the main experiments on Qwen2.5-Math-77 (Yang al. 2024) following Zeng al. (2025); Liu et al. (2025a); Yan et al. (2025). We expand Owen2.5-Math Math-7B'ss 7B's B's context limit from 4096 to 16384 by changing the rope theta from 10000 to 4000d" We set the maximum answer length to 8192 and the learning rate is set to 1e-6. We use Math- Verify give the 0-1 score for both training reward and evaluation accuracy.
Evaluation Setup. We evaluate the math reasoning capabilities six commonly used benchmarks, such as MATH-500 (Hendrycks et al. (02I), GSMSK (Cobbe et al. 202I), AMC (Li et al. 2024). OlympiadBench (He et al.) 2024), and AIME 24/25. Due to the tiny size of AIME 24/25 and AMC, we report Avg as the test accuracy. For the other benchmarks, we compute the Pass @ as the test performance. We calculate the average performance and token cost across all benchmarks. All models are evaluated under the same setting with a temperature of 0.6. The values Table [ are the percentage accuracy of the models evaluated. We also select three high-complexity subsets of Zebralogic (Lin et al. 2025) with their solution space sizes greater than 10 (Mid), 10 (Large), and 109 (XLarge), respectively. In this experiment, we record six metrics. including training reward, generation entropy. response length, the test performance of math tasks and logic reasoning task, and marginal improvement gained from reflection.
1.0 in Eq. 9 and present the results of μ 0.0 will conducted in Section 5.2 and the adjustment in Eq. 5 A more detailed scheduling on will covered in Section (.3 We compare
Baselines. In this subsection, we fix
(CANON-IntIae and μ = 1.0
CANON with two types of baselines: (1) Mann25--Math-BB--nstee
(Instruct, Yang et al. (024)).
(2) previous advantage estimation methods, such as ReMax, REINFORCE++ (R++), RLOO.
GRPO, and DR. GRPO, and (3) entropy-related baselines, such as Entropy Adv (Cheng et al.
2025
and Clip-Cov (Cui et al. 2025)
'The original context limit leads unacceptable length clipping ratio. Please Figure in Appendix C.3

---

Published as a conference paper at ICLR 2026
Table 2: Overall performance of CANON- across three different models and two tasks. All models are evaluated under unified setting. Bold and underline indicate the best and second-best results, respectively.
Math Reasoning
High Complexity Reasoning
Model
AIME 24 AIME Olympiad AMC MATII-500 GSM8K
Tokens
Acc
Mid
Large
XLarge
Tokens
Acc
Qwen2.5 M-tth-
GRPO (i- 0.5)
21.7
20.3
48.4
63.4
83.2
91.1
1522
55.7
39.2
24.4
15.1
4896
26.2
Cosin- Firm Inter- Later-LLtrr--tte
30.0
17.7
50.7
63.3
86.6
91.8
1452
56.7
40.4
30.5
16.6
3535
29.2
Firn- Inter-Later--iiee Later-Intra
28.0
20.3
52.4
64.6
84.2
92.6
1328
57.0
41.7
26.6
16.5
3862
28.3
Qwea2.5
DR GRPO (p- 0.5)
13.3
11.0
43.9
48.8
27.0
84.3
2381
46.4
23.7
9.7
5.0
9215
12.8
Cosin-Fira--nteer Inter--ter--i- aarer-rrrr- ater-
17.3
13.7
40.6
50.0
76.0
83.9
2387
46.9
19.2
8.9
4.2
10382
10.8
Bert-Inter-Later-nnie
16.0
10.0
42.4
50.2
78.6
83.3
2479
46.8
27.0
16.3
7,9
7070
17.0
Llama3. 1-88
PR.GRPO (p-
11.3
22.0
21.1
13.8
9.7
5864
22.6
26.0
18.4
Trat-inier-Late--niir
2,0
0.0
8.7
9.9
31.8
80.1
3488
22.1
25.1
17.5
10.6
5892
17.7
Inter-group advantage achieves higher accuracy and lower length in math tasks. The experimen- tal results are shown Table MNNON-Inter based on Entropy achieves an average performance of 57.6 among six math benchmarks, which is 1.9 points higher than the DR. DR.GRPO (55.7). Specif- ically, CANON-Inter based on Entropy has the best performance on four of the six benchmarks, and is highly competitive with the top-performing models the rest. In AIME24, the model's performance is 5.0 points higher than the DR. GRPO's. Meanwhile, CANON based on Length reduces the token cost by 33.8% compared with GR.G while maintaining nearly unchanged performance (55.7 vs. 55.3).
The benefit of intra-group advantage grows as the logic reasoning task's complexity increases. Table D demonstrates that CANON- Intra based on Entropy achieves higher performance of 2.9 points and 36.6% shorter length compared with GR.G Its performance edge over GR.G increases (from -0.1 to 3.4 and then 5.2) when the complexity becomes higher. The results of CANON-Intra based on Length shows another trend, whose inter-group advantage makes the best performance in this task.
Training dynamics reflect different roles of CANON-Intra and CANON-Inter. To be specific, we record training curves under the setting of CANON based on Entropy. The training dynamic shown in Figure iindicates that both the training reward and the test performance of the math tasks increase rapidly when only -ANON-Inter is utilized (μ = 1.0). Its generation entropy stably decreases, and the response length changes smoothly. When using only CANON-Int (μ = 0.0), the responses show a greater tendency for exploration. We divide the responses into two groups by counting reflection patterns and calculate the gap in average reward between the group with more and fewer reflections (Figure 2f). Figure 2 demonstrattes that the trend of high-complexity reasoning
performance is highly consistent with the curve of reflection gains. In the later stages of training (after approximately 90 steps), the reflection gain curve of untra-group advantage increases and finally crosses the zero point. At the same time, its performance experiences rapid growth, significantly outperforming the other two advantages.
5.2 BALANCING PERFORMANCE VIA ADVANTAGE SCHEDULING
As shown in Table I and Figure 2 CANON- Inter and
outperform DR.GRPO on the math reasoning task and the complex logic reasoning task, respectively, but neither can achieve the best performance on both simultaneously. To this end, we schedule the CANON-Inter and CANON-Intrs by leveraging accuracy and the training steps to achieve a better balance between the two scenarios.
Setup. We conduct experiments across six math benchmarks and three complex logic reasoning tasks on Qwen2.5-Mathh 7B (anng et al.) 2024), Llama3. 1-BB (Dubey et al. 2024). and Qwen2.5-Man -athh the dataset introduced in Section 5.I Due to the weak capability of Llama3.1 1-8B, we collect a simpler dataset with 35k samples from
1.5B Yang et al., 2024). For the two Qwen series models,
Tour open-source datasets and follow the other training setups described in Section 5.1 Please see the
7

---

Published as a conference paper at ICLR 2026
details of this newly constructed dataset in Appendix C.5 We draw radar chart with the average performance of the two scenarios for visualization, and the results for CANON with scheduling are denoted as CANON-Dynamis
Scheduling strategies. All of the strategies are based on the coefficient μ in the Eq. 5 which balances the CANoN-Inter and CANON-Intra. We try four scheduling strategies utilizing the training accuracy and training steps, respectively: (1) Leat--nnte---e---ee 1 1- A. where A denotes the mean accuracy of current whole batch; (2) First-Intral caarr--aaee-Intee the value of yı A. (3) Cosin-First- Sersst-nttr--atr--nnee
We set the value of μ to We set We schedule the value of μu from high low using a cosine annealing function with restarts and warm-up. (4)
www----irrstt--atttteet Inter. We schedule the value of je from low to high using a cosine annealing function with restarts and warm-up. Please see Appendix C.6 for more details. The shown results of CANON-DyDe Dynamic i c are derived from one of the tried scheduling strategies that achieve strong performance in both scenarios.
consistently performs better than DR.GRPO across three models and two tasks. As shown in Table all three models demonstrate the same trend that performs better than the baseline by first applying Inter-group advantage and then using Intra-group advantage. Qwen2.5-1.5B performs particularly well under accuracy-based scheduling, possibly because its training accuracy range (0-0.6) aligns well with its learning progress. In contrast, the other two models may achieve higher final accuracies, which-under the same scheduling scheme-trige and consequently lead to suboptimal final performance. We utilize fixed min/max values of ç by applying cosine annealing based on training steps, achieving higher performance.
First-Inter-Later-
excessive exploration
Moreover, different models may have different num- bers of parameters and different levels of capability. A specifically designed strategy is acceptable for bet- ter performance in practice. In this way, we select strategy Cosin-First Seitt-FFsst--nneeee /nwtee--teaee--eeeeeeee Math-7B and 1lamm3.1
Uama
Lama
22.
Complex Logic Reasoning
Math Ressoning
for Qwen2.5- and strategy First-Inter- Later-Intra for (0wen2.5-Man 5-Mthh--hhh----- 10-h-11.133 1.5B to draw Figure out- performs DR. GRPO across all models and tasks, achieving a superior and more comprehensive perfor- mance. Although its math performance on Qwen2.5- Math-7B lags slightly behind CANON-Intez it still makes better performance than DR. GRPO. The radar chart illustrates the trade-off between two types of tasks faced by CANON-Intez and CANON-Int25 between two types of tasks, as well as the balanced but mediocre performance of DR.GRPO.
Qwen-711
Qwen-778
3 As shown in Figure 3
46.
Qwen-1.58
Qwee-1 .58
DILGITO
CANON
CANON-Irer
CANON-Dymamic
Evaluation for three LLMs across two types of reasoning tasks. We apply model- specific schedule for a given model that consis- tently yields leading results across both mathemat- ical and logical reasoning tasks
Figure 3:
5.3 WEIGHTED CONDITIONS FOR EFFICIENT REASONING.
Training Setup. In this subsection, we utilize CANON based on response length with μ 0.5 in the Eq. and tune the in the Eq. where C is considered the group with longer responses. A larger & means less compression of length. We follow the training setups described in Section 5.1 and reduce the maximum response length to 3072 for better efficiency. To be specific, we use CANON-Eff to denote the results of CANON with weighted conditions of length.
Evaluation Setup under different token budgets and varying hyperparameter settings of differ- ent methods. To systematically assess LRMs' reasoning efficiency (Qu al. (2025), we introduce two types of curves: moddet--perfrmmnn different coefficients for all compared baselines. Specifically, we set a maximum budget for each benchmark based on its difficulty and the average unconstrained output length of LRMs (Appendix curves.
curves for each LRM and cost- performance curves of
C.2). then slice the same response at various budget ratios to draw the Moreover, we tune the length-controliigg coefficients of each baseline to draw the cost-performance curves, recording their average performance and token cost to enable a comprehensive and fair comparison. Please see the subsection Pareto frontier for the specific hyperparameters. In every comparison, the closer to the upper-left corner, the better (which represents high accuracy and high efficiency at the same time).
8

---

Published as a conference paper at ICLR 2026
Table The comparison between different methods towards efficient reasoning. Bold and underline indicate the best and second-best results, respectively. The detailed performance is from the top- performing models for each method, specifically a=0.96 for CANON-Eft We include CANON-Eft with 0.88, which has comparable performance with the baseline Length Reward (*),
AIME 24
AIME 25
Olympiad
AMC
MATH-500
GSM8K
Overall
Acc
Tokens
Acc
Tokens
Ace
Tokens
Ace
Tokens
Acc
Tokens
Ace
Tokens
Acc
Tokens
DR.G
29.0
1640
19.0
1586
49.0
1172
64.6
1214
85.8
728
91.9
349
56.6
1115
Clip Length
28.0
1177
18.3
1177
47.3
915
63.1
956
84.8
612
92.9
291
55.7
855
Length Reward,
31.7
1190
18.0
1208
46.7
864
61.8
937
84.6
546
91.9
255
56.2
869
Length Reward,
27.3
1087
13.7
1027
46.4
707
61.0
779
83.0
463
92.2
198
53.9
710
CANON-EE (ax 0.88)
27.3
816
15.3
862
43.9
582
59.3
649
84.4
386
91.4
166
53.6
577
CANON-EA (a: 0.96)
29.7
1216
19.0
1136
48.4
881
62.3
936
85.8
533
92.0
233
56.2
822
Performaree
Parformarce
-2.3
CANUN ef Dang
Petfor
GRPD (g-1.0)
Langth keward (x)
Cip Length
Length
Length Reward
CANON (Ours, 2-0.96) 96)
CANON (Curs, a.0.86)
Gollggr
Token Budges
Budget
Token Cost
(a) CANON-EFf with 0.96 con- sistently outperforms baselines meth- ods.
(b) CANON-EFE with = 0.88
(c) The Pareto frontier in the trade-
achieves significantly better perfor- off between performance and to- mance low token budgets.
ken efficiency.
Figure 4: compares the reasoning efficiency of CANON Eff against baselines under various token budgets.
and Cost-Performance Curves for Efficient Reasoning. This figure
Baselines. We select three types of baseline methods towards efficient reasoning: (I) Clip Length that directly clips the maximum output length (Hou et al. 2025). (2) Length Reward (+) that adds length penalties terms in the training reward (Luo al. (025), +coeff meanG, (L) 1)), and (3) Length Reward (+) that multiplies a normalized length coefficient on the reward (Arora Zanette
(0255) *(1 coeff * sigmoid( T- (-meang, (L.) ))). All these baselines are conducted with DR.GRPO. sida, (L)
CANON achieves better performance with shorter responses compared with baselines. We present the detailed performance of the top- hop-performing models for each method across various benchmarks in Table CANON-E.ff with a = 0.96 Pareto dominates the results of Clip Length and Length Reward (+), reducing the length by 26.3% compared to DR.GRPO while only decreasing performance by 0.4 points. Figure shows that CANON -f with a 0.96 consistently outperforms the baseline methods in both low-token-budgel and tot-ttokk--buugee scenarios. Since models trained with the Length Reward (+) exhibit significantly lower length with low performance the same time, is difficult to fairly compare with other baselines. To this end, we include an additional model trained with CANON Eff with a = 0.88 that has comparable performance. 4b indicates that CANON with = 0.88 shows better token efficiency compared with Length Reward (+), achieving 2.63 times the performance DR. GRPO low-token-budke scenarios, while reducing token consumption by 45.5% at the same performance level.
CANON achieves a better Pareto frontier and stably explores the entire frontier. To draw the cost-performance curves for each method, we draw the Pareto frontier of CANON-E.f with the results of a 0.5, 0.7, 0.8, 0.88, 0.96. For Length Clipping, we respectively present the results with maximum lengths of 2048 and 1024 in the Pareto frontier. For Length Reward (+), penalty coefficients of 0.001, 0.004, 0.005, and 0.1 are used, respectively. For Length Reward (+), we utilize the coefficients of 0.05, 0.2, and 0.4. 4e shows that all the frontier from baselines are dominated by the frontier of CANON-EF Ef'ss. 's.. It is noteworthy that after the coefficient of Length Reward (+) is adjusted from 0.004 to 0.005, its performance drops from 54.8 to 22.5. In contrast, CANON-EFF remains consistently stable, exploring the Pareto frontier efficiently.

---

Published as a conference paper at ICLR 2026
6 ANALYSIS
In this section, analyze how CANON-Dynamic and CANON-EFE effectively improve the task performance and reasoning efficiency.
GRPO)
Entropy
2=1.
p-1.0 (CARON
Different advantage combinations of CANON select dif- ferent trends of the target metries. Due to the different baseline rewards being compared, CANON- Inter tends to favor correct answers from the group with a higher aver- age reward, while CANON-Int selects correct answers from the group with lower average reward. We compare the effects of CANON on their target metries across seven different settings, with ranging from 0.0 to 1.0. When entropy is considered, figure S shows that a larger μ (favoring more CANON-Inted) leads reduction in entropy, whereas a smaller (favoring more CANON- CANON-Intral Intra) promotes an increase in entropy. The results demonstrates a hierarchical trend in the metric changes, indicating the effectiveness of controlling and select- ing different trends from CANON-Inter and CANON-Intra. In this way, CANON-Dynam can boost the task performance by adjusting different combinations of the two components.
Training Steps
Figure 5: CANON shows hicrarchical trends of target metrics through differ- ent combinations of CANON-Inter and CANON-Intra.
CANON can achieve positive gains of more rethinking and high training efficiency through scheduling of two advantages. As shown Figure 6 we record the perfor- mance genuinely brought by reflections and the curve of training reward. Although CANON- Intra achieves pos- itive gains from more reflections, its training reward expe- riences a significant decline. In contrast, CANON-Ints err which shows similar trend of GRPO. has not yet achieved positive returns even by step 360, but main- tains higher training reward. CANON-Dynamic. c, the other hand, not only achieves positive gains of re- thinking but also makes training reward on par with CANON-IntS Inter's. s. This explains why CANON CANON-Dyname -Dynamic can achieve comprehensive leading performance in both math and complex logic reasoning tasks.
Goin of Rethinking
Roward
CANON-Intra
CANON-Inter
CANON Dynamic
Training Sceps
Figure 6: CANON-Dynand a with sched- uled μ has positive gains of rethinking and high training score at the same time.
CANON amplifies only the advantage attributable to the metric used for grouping, without amplifying the in- fluence of other factors. As shown in Table 4 directly scaling the advantage (A formance the way CANON does. Any minor gains likely stem from faster learning progress due to an effectively larger learning rate, but this comes at the cost of degraded on out-of-domain logical rea- soning tasks. This suggests that the key to CANON" success not simply amplifying the advantage signal, but rather selectively amplifying specific signals, and that's why we introduce a regrouping operation.
Table 4: The performance comparison be- tween the direct numerical amplification of advantage and CANON.
* 2) fails to improve per-
Methods
Math
Logic
DR.GRPO
55.7
26.2
Direct Numerical Amplification
Numerical Scaling
56.1
25.1
Entropy Adv
56.3
18.5
CANON
CANON-
54.7
29.1
CANON-Inter
57.6
25.7
7 CONCLUSION
In this paper, we introduce CANON, a novel reinforcement learning framework for large reasoning models that leverages human priors on training metrics (e.g., entropy, response length) without presuming their directional impact performance. Extensive experiments across six math reasoning benchmarks and three high-complexity logic reasoning tasks demonstrate that CANON significantly outperforms prior advantage estimation methods like DR.GRPO. CANON also supports flexible weighting of different metric trends, where CANON based on response length achieves a superior Pareto frontier in the performance-effiiien
trade-off. Our analysis further confirms that CANON promotes beneficial behaviors such as effective exploration and reflection, which are critical for solving complex reasoning problems.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by Shanghai Artificial Intelligence Laboratory.
ETHICS STATEMENT
This work aims introduce human priors about key metrics into reinforcement learning by proposing a novel advantage estimation framework named CANON, which amplifies the impact of target metrics without presuming preferences. The experiments in this paper limited reasoning tasks conducted on open-source models, datasets, and benchmarks, which will not raise ethical concerns. We hope to explore the potential of CANON enhance the security of large language models in the future. thereby promoting their reliable and trustworthy development.
REPRODUCIBILITY STATEMENT
We aim to include both the high-level and low-level details of our method in the setup paragraphs of Section and Appendix to reproduce our results. All experiments are conducted on open-source LLMs and benchmarks. We employ open-source datasets for the Qwen series L.MSs, provide detailed description the prompts used for training and evaluation, and comprchensively present the construction process of the training dataset for the Llama series LLM. Our code implementation is based on VeRL. (Sheng et al. 2024), which applied with focused modifications in the advantage computation part, enhancing the reproducibility of our work. Please access code base via the following anonymous link: CANON
BIBLIOGRAPHY
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint w7Xiv:2402.14740, 2024.
Daman Arora and Andrea Zanette. Training language models reason efficiently. arXiv preprint arXiv:2502.044463, 2025.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-ml: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506. 13585, 2025a.
Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Seed-grpo: Semantic entropy enhanced grpo for uncertainty-awart policy optimization. arXiv preprint arXiv:2505. 12346, 2025b.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. /4758, 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint (..::2100.14101 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06260 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617. 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
11

---

Published as a conference paper at ICLR 2026
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self improvvig reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01130 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprini arXiv:2501. 2025.
Michael Hassid, Gabriel Synnaeve, Yossi Adi, and Roy Schwartz. Don't overthink it. preferring shorter thinking chains for improved IIm reasoning. arXiv preprint arXiv:2505. 17813, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3828-3850, 2024.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504 1456, 2025.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinbardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103. 03874, 2021.
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long -hain-of-thoughn of lims via reinforcement learning, arXiv preprint NXXv:2504400196, 2025.
Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03222, 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025. ht..httss: /ttp:::///xivvvio httpp//g/./////.222222 2429
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URI. https S.githuu.com hg ng face agface/open--r! open-rl
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 1.Xi::2412.16720. 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 2rXiv:2411.151244 2024.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. htttp::///uggingfacee co/ datasets Nww.nnauinnnnnnnnmmmmaamaaaaaaai 2024. Hugging Face repository, 13:9.
Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effec- tive, and efficient method for aligning large language models. arXiv preprint arXi::2310.10005 2023.
Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebralogic: On the scaling limits of llms for logical reasoning. arXiv preprint arXiv:2502.011 100, 2025.
Zichen Liu, Changyu Chen, Wenjun Li. Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl -l-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025a.
12

---

Published as a conference paper at ICLR 2026
Zongkai Liu, Fanqing Meng. Lingxiao Du, Zhixiang Zhou, Chao Yu, Wenqi Shao, and Qiaosheng Zhang. Cpgd: Toward stable rule-based reinforcement learning for language models. arXiv preprint arXiv:2505.12504, 2025b.
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: Length-harmonizing fine-tuning for ol-like reasoning pruning. arXiv preprint arXiv:2501.12570, 2025.
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22660 2025.
PyTorch. R - PyTorch Documentation, 2025. URLhtttps:///oct wytooccc..p....c....oooooo. org/ / optim. 1r sww.scheeerr.. LambdalR,, Ac- cessed: November 19, 2025.
Xiaoye Qu, Yafu Li, Zhaochen Su. Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, Peng Li, Wei Jing Shao, Chaochao Lu, Yue Zhang, Xian- Sheng Hua, Bowen Zhou, and Yu Cheng. A survey efficient reasoning for large reasoning models: Language, multimodality, and beyond, 2025. URL https: http:::///wxwww..o///a/a
2503. 2563.21614
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprini arXiv: 2409. / 19256, 2024.
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 464-472 IEEE, 2017.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01939, 2025.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning reason under off-policy guidance. arXiv preprint arXiv:2504. M5X::25004.14445. 2025.
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024.
wrgg/bs//////o.oooooooo....... 2409. 12122
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint ar Xww::2055.00388. 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source I1m reinforcement learning system at scale. arXiv preprint arXiv:2503.14476 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
13

---

Published as a conference paper at ICLR 2026
A LIMITATIONS.
Based on feasibility and motivation, this work focuses on conditions that can be specified through numerical ordering, without exploring conditions that are more complex and harder verify. Due to limitations in paper length and computation resources, this work primarily conducts the CANON based on two metrics srrriccsreepoons. length and entropy-whhe while other training metrics remain unexplored. Additionally, the paper considers only one metric at time, without attempling to incorporate multiple metrics simultaneously. This demonstrates that the perspective and framework proposed in this work is flexible and hold significant potential for extension, which can be further explored in future research.
B THE USE OF LARGE LANGUAGE MODELS.
LLMs primarily assist this work in two aspects: one hand, they are used for aiding our writing, and on the other hand, they sometimes serve as a coding assistant during the programming of our code base
C EXPERIMENTS DETAILS.
C.I RETHINKING PATTERNS.
Following Gandhi al. 2025), firstly samples 10000 responses of Qwen3-32B Yang al. (2025) and utilize the modified prompts from (Gandhi et al. 2025) to collect the rethinking patterns of verification, sub-goal setting, and backtracking. Then we match these patterns in few Question- Answer instances and filter out overly frequent conjunctions, overly short words, and semantically ambiguous phrases. The number of remaining keywords and regular expressions is 334 for verification, 1036 for sub-goal setting, and 532 for backtracking.
C.2 THE MAXIMUM TOKEN BUDGET SETUPS.
We set the maximum token budget for each benchmark based its difficulty and the average token length observed from models trained with DR.GRPO, as shown in Figure 5 When plotting the performance-budee curve, normalize the maximum token budget of each benchmark to 1.0. We then evaluate the performance of all benchmarks under token budgets ranging from 0.1x to 1.3x their respective maximum budget, averaging the results across benchmarks at each budget ratio and displaying them in the figure.
Table 5: Benchmark-wise Maximum Token Budget.
Benchmark
Avg. Tokens (unlimited)
Max Token Budget
GSM8k
349
600
MATH-500
728
1500
AMC
1214
1800
OlympiadBench
1172
1800
AIME 2024
1640
2000
AIME 2025
1586
2000
C.3 REASONS FOR EXPANDING THE CONTEXT WINDOW OF MODELS FROM QWEN2.5 -MATH SERIES.
Initially, we uses the setting of Section 5.1 however, during the training process, too much length clipping (> 30%) results in nearly incomparable experimental outcomes, as shown in Figure 7 Therefore, we expand Qwen2.5-Math-77
context limit from 4096 16384 and set the maximum output length 8192, which alleviates this phenomenon.
C.4 SYSTEM PROMPT.
For the training and inference of Qwen series models, share the system prompt as follows.
14

---

Published as a conference paper at ICLR 2026
Foger
Training Score
Micheer La Caroggee
Thaining Steps
Training Steps
Figure 7: The ratio of answers truncated due reaching the maximum output length.
Figure 8: The score curves of the training set and validation set from the newly constructed dataset with 35k data and the original dataset used for the Qwen series models, respectively.
Your task is to follow systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring. reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, present your reasoning using the format: "<think> An thoughts < </think> \n". Each thought should include detailed analysis, brainstorming, verification, and refinement of ideas. After "</think> \n" in the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration in the Thought section. If applicable, include the answer in \boxed for closed-form results like multiple choices or mathematical solutions.
C.5 CONSTRUCTION OF TRAINING DATASET FOR LLAMA3.1-88
Since the pretraining of Llama3.1 10mma3.1-85 1-8B lacks data for long chain-of of of-thought and mathematical reasoning.
its average training reward based on the original dataset used for Qwen2.5- remains below 0.2. To enhance training efficiency, we employ three Llama series models (Llama3. 1-BB, Llama3. 1-8BB
Instruct, Ilaam3. 1-70B) to generate solutions for each problem across four datasets (training set of GSM8k (Cobbe et al. (021), training set of MATH (Hendrycks et al. 2021), 46k subset of IpenRR-Matth-
(Hu et al. 2025 Yan et al., 2025), and Decp 103k (He et al. (0055)) We then filter out questions whose accuracy of Pass @ O, ultimately selecting 35k samples for training the Llama3.1 1-BB model. Concurrently, due to L 1-8B's limited instruction-follwiii capability, simplify the output format requirements in its system prompt.
Your task is to follow a systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring. reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, each thought should include detailed analysis, brainstorming, verification, and refinement of ideas. In the Solution section, provide the final, logical, and accurate answer, clearly derived from the exploration in the Thought section. applicable, include the answer in \boxed for closed-form results like multiple choices mathematical solutions. Let's think step by step.
The training curves for this 35k dataset and the original 46k training dataset over 150 training steps are shown in the Figure g It demonstrates that 1.lam33.1-88 has significantly higher learning effectiveness the newly constructed dataset.
C.6 SCHEDULING STRATEGIES OF CORFFICIENT BALANCE CANON- INTER AND CANON-INTRA.
We try four different scheduling strategies and show the best of them for each model. Figure shows the dynamics of μi in the training process from the Inter-Laterr (μ = 1- A) and Sare-hitra---e---- (p A). -aeti-----i--ntte---at------ee and Cosin-First- Iinsst--nntrrr--aee---iii
15

---

Published as a conference paper at ICLR 2026
Table 6: Detailed experimental results of CANON on Llama3. and Qwen2.5-Math- All models are evaluated under unified setting. Bold and underline indicate the best and second-best results, respectively.
Math Reasoning
High Complexity Reasoning
Model
AIME Olympiad AMC MATH-500 GSM8k
Tokens
Acc
Large XLarge
Tokens
Acc
Qwen2.5 S.Math. 1,5B
DR.GRPO
13.3
11.0
43,9
48.8
77.0
84.3
2381
46.4
23.7
9.7
5.0
9215
12.8
CANON-Tntrra
75.2
11.5
CANON Inter
CANON-Iater
14.3
9.3
41.8
49.4
78.8
82.7
1876
46.1
23.1
9.6
5.8
8342
12.8
CANON- CANON-Dynamic
16.0
10.0
42.4
50.2
78.6
83.3
2479
46.8
27.0
16.3
7,9
7070
17.0
Llama3.1-88
DR.GRPO (μ U.5)
1.3
0.3
8.3
11.3
32.0
78.9
9476
22.0
21.1
13.8
9.7
6370
14.9
CANON-Intan -Ittre
1.0
0.0
7.7
10.2
27.2
78.9
23961
20.8
24.3
16.7
10.3
17753
17.1
CANON-Inter
2.7
0.0
8.0
10.0
31.6
79.8
3671
22.1
17.9
13.8
9.5
1331
13.7
CANON-Dynamic
33.8
2354
1685
18.9
schedule the value of f with cosine annealing function V with restarts and warm-up:
+ 1
Mmax
if w
w
V
Pmin + (Mmax - Mmin))
1 cos
π
S-w
if 2 w and s' = 8 mod
(10)
where e denotes the number of restart and the warm-up step. s the current step of training and S is the total step. Mmax and Mmin denote the specified maximum and minimum values of μ. We use c = 3, w = 30 and = 150 for both strategies.
In strategy
we utilize μ = V with Mmma = 1.0 and Mmnn 0.4, respectively, while in strategy Setir--iis--ntrr-aate-----
we utilize 1 - v with fmax 0.6
and Mmin = 0.0, respectively. The changes in μ under these strategies are shown the Figure 10 Ultimately. based on training performance, we selected strategy Qwen2.5-7B and Llama, and strategy First-Inter-Lone Late--Lterriiii for Qwen2.5-1.5B.
for
Mi
Warm-up
Restart
Coeffrient
Coefficient 5
Training Steps
Training Steps
Figure 9: The changes of for two schedul- ing strategies based on accuracy during training.
Figure 10: The changes of for two scheduling strate- gies based on training steps during training.
C.7 DETAILED EXPERIMENTAL RESULTS ON LLAMA3.1 QWEN2.5-MATH-
Here we show the detailed test results for Llama3.1 and Qwen2.5-Math. 1atth11550 1.5B for the compar- ison between CANON, CANON Dynamic c and its baseline. As shown Table 6 consistent with Owen2.5-Mathh7
Llama3. 1-8B achieves superior performance math tasks with CANON- Inter and leads on logic tasks with CANON-IntE ra, while CANON-Dynan across both tasks. On 0wen2.5-Math- 1.5B, CANON- enter does not achieve a lead in math perfor- mance; however, its dynamic variant CANON DANON-Dynamic still surpasses the baseline both tasks, demonstrating the effectiveness of CANON.
outperforms the baseline
16

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS.
D.1 CANON BASED ON ANOTHER METRIC.
To further verify that CANON remains effective under other grouping criteria, conduct new experiments that use the number of per-token reflection steps in each response as the grouping metric. As shown in Table 7 it exhibits a trend similar to entropy- based CANON. Although both are slightly inferior to the entropy-base based CANON, CANON-Inter still outperforms the baselines mathematical reasoning. and CANON-Intra achieves better performance than the baselines complex logic reasoning. Figure IT demonstrate that CANON-Inter favors less-reflection responses and CANON- encourages more reflections.
Table Overall performance CANON based on Per-token Reflection for Qwen2.5-Math- All models are evaluated under a unified setting. Bold and underline indicate the best and second-best results, respectively.
Math Reasoning
High Complexity Reasoning
Model
AIME AIME Olympiad AMC MATH-500 GSM8k Tokens
Acc
Mid
Large
XLarge
Tokens
Acc
DR.GRPO = 0.5)
27.7
20.3
48.4
63.4
83.2
91.1
1522
55.7
39.2
24.4
15.1
4896
26.2
Our Methods (Conditional Groups based Per-token Reflection)
CANON Intra
25.0
16.0
50.4
62.8
85.4
91.3
1912
55.1
41.0
25.5
15.5
4834
27.3
CANON Inter
26.7
18.3
51.9
65.4
85.4
92.2
1739
56.6
37.4
17.0
8.5
7835
21.0
J.000 (CANON-
(28L
1am10 (CANDN-inter)
Reflections
Reward
Training Steps
Training Steps
Figure 11: The training dynamic of CANON based on per-token reflection.
D.2 OTHER ALTERNATIVE SCHEDULING STRATEGIES OF BANON-DYNAMEE
We conduct further experiments that utilize other alternative scheduling strategies that were used to be performed in the scheduler of learning rate, including Lambda strategy (PyTorch) (025) and Syclic-triangula2 strategy (Smith 2017). Following the setting of tried scheduling strategies, both of these strategies schedule μ from 1.0 down to 0.4.
The new experimental results in mance than DR.GRE GRPO on math tasks but performs worse on logic reasoning tasks. This may be because they fail to sufficiently leverage intra-group advantages in the later stages of training. This experiment demonstrates the rationale behind CANON's tried scheduling strategies and highlights the practical flexibility of the CANON framework.
that Lambda strategy achieves slightly better perfor-
D.3 DYNAMIC SCHEDULING ON LENGTH-BASED CANON.
When we consider response length, CANON-Inter tends to produce shorter responses, whereas CANON-Ind Intra favors longer ones. However, these trends in response length do not translate into performance gains. This is precisely why we only applied dynamic scheduling to the entropy-base variant of CANON. To prove this, we try dynamic scheduling on length- based CANON under the setting of entropy-based CANON-iniiinn longer ones as training progresses. The results are shown below. Although the responses are shorter than those of DR. GRRR the math performance slightly declines, shown in Table While this
favoring shorter responses and gradually shifting toward
17

---

Published as a conference paper at ICLR 2026
Table 8: Overall performance of CANON- -ynamii based on other two scheduling strategies for Qwen2.5-Math--7 All models are evaluated under a unified setting. Bold and underline indicate the best and second-best results, respectively.
Math Reasoning
High Complexity Reasoning
Model
AIME 24 AIME Olympiad AMC MATII-500 GSMSk
Tokens
Acc
Mid
Large
XLarge
Tokens
Acc
DR.G (p- 0.3)
1522
55.7
39.2
15.1
4896
CANON Dannl--Dnamiii I based on Entroyy
Cosin Cosin-Fira: Firn Inter-Later--eeee Later-LLt---
30.0
17.7
50.7
63.3
86.6
91.8
1452
56.7
40.4
30.5
16.6
3535
29.2
Firn- Inter- Later-Intra
28.0
20.3
52.4
64.6
84.2
92.6
1328
57.0
41.7
26.6
16.5
3862
28.3
Cyelie-trinnglar2 stralegy
24.0
18.0
49.3
63.3
84.8
91.3
1647
55.1
37.4
22.1
14.5
5203
24.6
Lambda strateg
26.0
22.0
49.3
63.5
85.2
91.5
1744
56.3
37.1
21.8
13.6
5297
24.1
method shows improved performance on complex reasoning tasks, it still does surpass either CANON-Inter or CANON-Intra based on length.
Table 9: Overall performance of CANON-Dynamic e based on length response for Qwen2.5-Maan All models are evaluated under unified setting. Bold and underline indicate the best and second-best results, respectively.
Math Reasoning
High Complexity Reasoning
Model
AIME 24
AIME 25
Olympiad AMC MATH-500 GSM8k
Tokens
Acc
Mid
Large
XLarge
Tokens
Acc
DR.GRPO (y+ 0.5)
27.7
20.3
48.4
63.4
83,2
91.1
1522
55.7
39.2
24.4
15.1
4896
26.2
CANON based on Length
CANON-Int.Fo
21.7
19.0
49.9
63.0
86.2
92.2
2176
55.3
41.8
25.6
14.7
4364
27.4
CANON-Inter
27.3
19.3
47.6
64.2
82.6
91.0
1008
55.3
42.7
28.6
17.1
3652
29.5
CANON-nynamic
91.7
D.4 ANALYSIS OF S HYPERPARAMETER TUNING COMPLEXITY
Although CANON-Dynamic introduces a hyperparameter μu to balance exploitation and exploration, unlike conventional regularization coefficients, μ carries rich physical meaning and does not add significant complexity. When μ equals 0.5, DR.GRPO achieves the simplest form of balance through a static weighted average. This observation inspired us: if more dynamic balancing mechanism exists, it is natural that this method could outperform DR.GRPO this is precisely why CANON-Dyname works.
To analyze the hyperparameter tuning complexity re-introduced by μ,, we train Qwen2.5-Math with 4K context length using entropy-basee CANON, showing how model performance and entropy vary with μ. Table [0 indicates that, μ increases from 0 rn) to (CANON-Inted) in-domain mathematical performance steadily improves, out-of-domain logical reasoning perfor- mance gradually declines, and entropy consistently decreases revealing clear trend. Therefore, introducing μ does not increase the difficulty of hyperparameter tuning, rather, it extends the CANON framework and offers new insights how we can utilize CANON.
Table 10: Performance and entropy across different y values.
μ
0.0
0.2
0.4
0.5
0.6
0.8
1.0
Math
54.2
54.9
56.0
56,6
56.7
56,4
57.9
Logic
27.1
25.1
24.6
23.8
25.6
22.9
22.5
Entropy
2.40
1.28
0.46
0.39
0.26
0.19
0.15
D.5 ANALYSIS OF a's HYPERPARAMETER TUNING COMPLEXITY
Insights from CANON reveal that the choice of metric trend primarily occurs in the inter-group advantage computation. Therefore, in scenarios where inference efficiency is desired, we only need to slightly reduce the reward weight for the long-response group in CANON-Inter, prompting the model to favor shorter answers. Mcanwhile, since CANON-Intra remains unchanged, CANON-EFE fully preserves the model's exploration capability, achieving a superior cifoommanc-efffcieenee Pareto frontier. The hyperparameter a introduced in CANON-Eff not only allows flexible tuning
18

---

Published as a conference paper at ICLR 2026
toward specific application needs and thorough exploration of the Pareto frontier, but also ensures greater stability and smoother behavior compared to baselines (like the collapse of Length Reward (+)) because it minimally alters the training process (only modifying the inter-group advantage computation).
To analyze the hyperparameter tuning complexity re-introduced by a. We show the detailed per- formance and token length of these CANON-Eff models. Table ?? indicates that a gradually decreases from 0.96 to 0.5, model performance declines modestly, while the number of tokens con- sumed drops significantly- agginn aligning with understanding. Therefore, in efficient reasoning tasks, introducing a not only avoids increasing hyperparameter tuning difficulty but also enables users to apply CANON-Eff more flexibly according their specific needs.
Table 11: Performance and token cost across different a values.
Ct
0.5
0.7
0.8
0.88
0.96
Performance
44.5
49.5
52.0
53.6
56.2
Token Cost
198.9
317.9
420.6
576.6
822.4
D.6 CANON SELECTS APPROPRIATE METRICS AS THE TARGET.
We conduct a simple ablation study on the target metrics
considered by CANON. As shown in Table T2. random grouping achieves only the same performance as the base- line method while producing longer responses, thus failing to improve either performance or efficiency compared to based on the
Table 12: The accuracy and token cost of
CANON-InE enter with different metrics.
Methods
Acc
Tokens
DR GRPO
55.7
1522
the baseline. In contrast, CANON-Int response length excels in the token efficiency with 33.8% shorter responses, and the entropy-basee CANON-Inter delivers the best performance (57.6 points) among the comparisons.
Random regrouping
55.7
1557
CANON Inter
based on Length
55.3
1008
based Entrayy
57.6
1466
E DETAILED DERIVATION OF THEOREM AND
Theorem 1 (Situations with clearer ad advantage signal). Suppose that condition is based numerical comparisons and can be derived through sorting of metrics. Further assume that the sampled response
to query satisfy condition with probability p (0, 1), and E, satisfy [R.] F E not sarisfy el [R.]-
Then, we have:
A
>
1, only when |C+ = C7 is is constant.
(11)
(ĀDR GRPO
Proof of Theorem 7 Given a prompt q, the of all responses that satisfy condition e can be denoted as We use = (o E Clg,0) 0) E (0, 1) to describe the probability that a response o satisfying condition e is provided to the prompt y by LLM with parameter 0. Assuming that when condition is satisfied, the probability of the correct response is a., and when condition e is not satisfied, the probability of the correct response a... Denoting the correctness of the response query q as R., then we have:
EpEC [Ro] a. and Eoge
(12)
E. DR.GRPO
Sampling group of responses Gy the prompt q. the advantage ÅDR of response be calculated as:
DRR.RPO = Ro -mean( (Ro E Gyy))
(13)
19

---

Published as a conference paper at ICLR 2026
We A (o, c) to denote the average advantage of the responses that satisfy condition c. and utilize /DR.GRO (o, c) to describe the average advantage of the other responses that do not satisfy condition c.
A DR (0,c) c) = EbEc Aggoot ADR.CRPO]
= EbEc [Ro] - EoeG, [Ro]
a.+ - (P(o Cl Eocc [Ro] + P(o & Cla, 0) BogcclR
= a++ -pa+ (1 p)a_ = (a+ a- ){1 - -p)
(14)
DRRRRO (0,c) (a-- la+)p
(15)
E.2 INTER-GROUP ADVANTAGE (CANON-INTE3)
We sort the sampled responses based on the numerical value considered by condition c, and split them position Ao into two groups. Based on the symmetry of the inter-group advantage, can denote these k responses C..- We use À [C!
to simplify the notation, and denote the average inter- G.,
group advantage with (a p) for the responses that satisfy condition Ax c, c, is utilized to represent the average inter-group advantage of those responses that do not satisfy condition c.
Then, we can compute the average reward of each group as follows.
BEEC [Ro] = [P(o E Cla, 0,0 E C Evec Ro] + P(o & Cla. 0,000.0. E EoogE R..]
E Ro] + X A if
Eoec |
if <
Lat + >x a- a a Af
(16)
if
E B [Ro] P(oo Clq, 0, C+ Eoec [Ro] + P(o & Clq,0,0 C Bogc
Boc [ol] + E-REoge (Ro, X <p
a, 2p
(17)
5 + ,f
if
Therefore, we can calculate the average advantages:
Ax(o, c. p) = Eoecl - P(o E 1tl9.0,00 0,0 e C)E [Rol - P(o & C+ CJE, eec, [R,/
if
0.
[Ro]
A A1p-A p a+ Ja
P- if <p
a4 if
(18)
x(1-) p(1-1) (a+ af
Ax(o,c = Eogel RR - P(OE (YOEET111.0 1, 0,0 & C)E C)E Roo] - P(o C 1,,0,0 $ C)E Jec, [Ro
if
1a + 1,
=Eofc (Rol
0
ifA
p(1-) (a_ af),
p|
(19)
- if <p a
20

---

Published as a conference paper at ICLR 2026
E.3 COMPARISON
We have the ratio between inter-group advantage and DR.GRPO:
[Ax(o, c,
p if p
(20)
ADR
App
and
Ax(o,cpP) ADRRCRROO (o,c)
i
(21)
if
To accentuate the impact of specific condition on advantages, the following is required:
1- À
> >ifA2 p, and À
>
(22)
X(1 K1-P) p)
(1 - Alp
Then have
P
X < if and X 2 2-p
if < p. 1+p
(23)
If |Ctt1 is constant, is also constant. Due to 1 and
p < 1. X needs to satisfy À S
2-p
1+p
and X 2 2 at the same time, consequently restricting the value of À to 0.5. In this way, we have 0.5, and finally |Ctt1 |C,
Theorem 2 (Selective amplification based on specific metries (proved in Appendix Ed). Consider independent conditions C₁ and and their corresponding sets C₁ and C2 (i.e., P(o E C n C2lg, 0) P(o C) E C2lg, 0)). When we fix the condition et, then for any value of the performance gap 02+, Q2- and the probability P(o E C2lg. 0) that induced by whether Co is satisfied, we have
A Ago L GRPO
based on
a constant.
(24)
A
which says CANON based on the condition ej will not amplify the influence of a'oothee independent condition C2
Proof of Theorem 2 According to Eq. 20 and 21 the scaling factor depends only on the probability Pi P(o C llg.00 0) that response o satisfying condition e is provided to the prompt by an LLM with parameter A. Therefore, any irrelevant condition C2 and its associated parameters cannot affect this ratio.
21


Reviews:
[{"review_id": "uuL29rEXRE", "reviewer": "Reviewer_dLuf", "summary": "This paper introduces Conditional Advantage Estimation (CANON), a novel method for reinforcement learning in large reasoning models. The authors identify a key limitation in prior work: current methods often rely on hand-crafted penalties or preferences for training metrics (like entropy or response length), which can be overly biased and require careful tuning. CANON avoids this by not presuming a direction for the metric's impact. Instead, it regroups sampled responses based on the metric (e.g., high-entropy vs. low-entropy) and computes two novel advantage estimates. The \"inter-group\" advantage identifies which metric trend leads to higher accuracy, while the \"intra-group\" advantage identifies the best responses within a given trend. \n\nExtensive experiments show that CANON outperforms strong baselines like GRPO on math and complex logic reasoning tasks. Furthermore, when applied to response length, the method establishes a new, superior Pareto frontier for the performance-efficiency trade-off.", "questions": "See above weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper demonstrates a novel and well-motivated advantage estimation method. splitting responses into conditional groups and calculating inter- and intra-group advantages directly addresses the problem of biased, hand-crafted priors. The theoretical analysis showing DR.GRPO as a special case of CANON further solidifies the contribution.\n2. The authors provide convincing results across multiple models and challenging benchmarks. \n3. The proposed method has a practical impact on reasoning efficiency with some token reduction at the same performance level or higher performance in low-token-budget scenarios.", "weaknesses": "1. Curious why don't the authors put CANON-Dynamic's results in Table 1? Wouldn't that be more intuitive?\n2. The paper's experiments are focused exclusively on response length and entropy.\n3. The paper argues against hand-crafted tuning, but the CANON-Dynamic method, which achieves the best-balanced performance, relies on a handcrafted hyperparameter and may need to tune it for different tasks seperately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iXtWjADNyT", "reviewer": "Reviewer_Xngh", "summary": "This paper proposes Conditional advANtage estimatiON (CANON). CANON regroups responses based on specific metrics, leading to two subgroups of responses. By leveraging inter-group and intra-group statistics, CANON introduces the inter-group advantage and intra-group advantage, which provide a stronger advantage signal. The authors further introduce several strategies to enhance this approach, and experiments on reasoning tasks validate its effectiveness.", "questions": "Please see **Weaknesses** part. I am looking forward to the authors’ response, and may reconsider this paper based on that.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper is well-written.\n2.\tThe ablation studies in Section. 6 are informative. The difference caused by the ratio of inter-group and intra-group advantage seems an interesting topic to be discovered further.", "weaknesses": "1.\tAs shown in Theorem 1, CANON yields a stronger advantage signal. However, it is essentially a numerical amplification (adding the difference between the means of the two groups) of the advantage, and consequently, of the loss. What is the difference between CANON and this variant? If they are the equivalent, what is the purpose of regrouping?\n2.\tCANON splits responses into two subgroups. However, since such regrouping may help training, what would happen if we performed a finer-grained regrouping? For example, we could split responses into four, six, or eight groups and calculate the corresponding advantages.\n3.\tDR.GRPO is outdated in the context of LLM reasoning. It is suggested that the authors compare CANON with more recent baselines, especially those focusing on entropy. \n4.\tThe ablation studies focusing on different roles of inter-group and intra-group advantage (in Section 5.1 and Section 6) are valuable. However, despite the experiment observation, it is suggested that the authors provide a more detailed theoretical analysis on it, especially the reason behind Fig. 5 and Fig. 2.\n5.\tCANON-Dynamic appears effective. However, there exist several dynamic scheduling methods, It is suggested that the authors compare theirs with other alternative scheduling strategies, both theoretically and experimentally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4wfNqnUSNR", "reviewer": "Reviewer_Aak7", "summary": "This paper proposes Conditional advANtage estimatiON (CANON), sorting the sampled responses according to a metric and splitting them into two groups, instead of a hand-crafted prior with intra- and inter-group advantage. Experimental results demonstrate that CANON, based on entropy, consistently outperforms previous methods across three LLM backbones on both mathematical and complex logical reasoning benchmarks. Moreover, when applied to response length, CANON improves token efficiency and achieves a better performance–cost Pareto frontier. The authors conducted experiments around mathematics and logical reasoning on several models. However, I don't think this is enough to demonstrate the effectiveness of the method, and the biased report prevents me from giving a score of acceptance, see Weakness and Question. If the author can provide a convincing explanation, I am willing to increase my score.", "questions": "1. Why are alpha=0.96 and 0.88 selected in the analysis of Table 2? Do these parameters need to be carefully picked?\n2. Other questions, please see Weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The author provides experiments and ablation/analysis experiments to validate the proposed method.\n2. The motivation is clear.\n3. The paper is well written and easy to understand except for section 5.3.\n4. The author provides theoretical analysis and has the potential to incorporate some previous work into a more unified view.", "weaknesses": "I have no doubts about the method itself, but there are several issues with the experimental part that directly affect whether the paper should be accepted.\n1. The author mainly reports the results on Qwen-2.5. Although only AIME 2025 performed worse than other baselines, it is the only test set that is later than the training time of Qwen-2.5. Due to the risk of data leakage in Qwen-2.5, I suspect that the effectiveness improvement brought by the method is related to data leakage. Logic reasoning tasks can, to some extent, demonstrate that the method is still effective. Without this part of the experiment, I would give it a score of 4, but I still look forward to more model experiments that prove the effectiveness of the method. At least supplement all the results of the models involved in Figure 3.\n2. The results and their explanation in Figure 5.3 are very misleading. I suggest the author make revisions and truthfully report the results of various strategies, rather than picking choices that are beneficial for reporting. In my understanding, CANON-Dynamic is a combination of several selected strategies, rather than a fixed strategy. The author's report and explanation can easily be misleading, so I give 2 points in the Presentation. Due to the lack of representation of other results of the model in Figure 3, and the fact that the baseline (DR.GRPO) in the figure has its own advantages and disadvantages compared to non-combined results (CANON-Inter/-Intra), the experiments in this paper cannot demonstrate that the method performs well on other models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AxJ6q3Lrfl", "reviewer": "Reviewer_pJ7Q", "summary": "This paper introduces CANON, an advantage estimation method for RLVR. Instead of relying on hand-crafted human priors such as the assumption that lower entropy is better or that higher entropy is better, CANON dynamically identifies which metric trend leads to better performance. It works by conditionally partitioning sampled responses into two equal-sized groups based on a target metric, then computing two types of advantages: CANON-Inter (comparing against the other group) and CANON-Intra (comparing within its own group). The overall advantage is computed as $\\mu$ × CANON-Inter + (1 - $\\mu$) × CANON-Intra. The authors find that CANON-Inter works better for math tasks while CANON-Intra works better for complex logic tasks. CANON-Dynamic schedules $\\mu$ during training based on training accuracy or training steps, achieving a balance between the two approaches.", "questions": "- Is there a principled way to incorporate multiple metrics simultaneously within a single unified advantage calculation?\n- In the upper right corner of Figure 1, wouldn't it be clearer if the inter- and intra-group advantages were each visually subdivided to show their two constituent subgroups? This would help readers more readily understand that each advantage metric is calculated using two distinct subgroups.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- CANON selects its own direction by empirically determining which metric trend performs better, rather than rolling with a fixed assumption. The motivation to remove hand-crafted directional priors and let the data determine which metric trend is beneficial appears novel and well-justified.\n- The method provides a systematic way to control the emphasis on exploration (favored by CANON-Intra) and exploitation (favored by CANON-Inter). This offers a new approach to balance a metric, which is especially valuable for entropy that benefits from careful, dynamic balancing.", "weaknesses": "- While adaptive scheduling makes sense for entropy (balancing exploration and exploitation), its value is less clear for response length, where shorter responses are almost always preferable for efficiency.\n- The choice between CANON-Dynamic and cosine-based scheduling appears to depend on the specific model being trained, with no universally optimal approach, thus requiring trial and error to determine the best strategy.\n- This re-introduces hyperparameter tuning complexity, as the values of mu and alpha (group proportion parameter), along with their scheduling and adjustments, constitute additional hyperparameters that require careful calibration, which contradicts the paper's stated motivation.\n- The method only considers one metric at a time, and it remains unclear what other metrics beyond entropy and length could be meaningfully employed with this approach.\n- The evaluation metric for the AIME benchmarks, Avg@10, seems insufficient given their small sample size. To my knowledge, Avg@64 is more commonly used in practice to achieve stable and reliable results for such tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)