Self-Aligned Reward: Towards Effective and Efficient Reasoners
Authors: Peixuan Han
ADIT KRISHNAN
Gerald Friedland
Jiaxuan You
Luyang Kong
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
large language model
Efficiency
Internal Signal
Primary area: reinforcement learning
TL;DR: We propose self-aligned reward (SAR), an internal-based reward design based on perplexity drop, that can enhance reasoning accuracy and efficiency at the same time.

Abstract: Reinforcement learning with verifiable rewards has significantly advanced reasoning with large language models (LLMs) in domains such as mathematics and logic. However, verifiable signals provide only coarse-grained or binary correctness feedback. This limitation results in inefficiencies like overly verbose or repetitive reasoning. Existing length-based solutions (e.g., length penalty) compromise accuracy. To address this deficiency, we introduce **self-aligned reward (SAR)**, a generic, universally applicable self-guided signal that complements verifiable rewards to enhance both reasoning accuracy and efficiency in RL. Specifically, SAR is defined as the relative perplexity difference between an answer conditioned on the query and the standalone answer, thereby favoring responses that are concise and query-specific. Quantitative analysis reveals that SAR reliably judges answer quality: concise, correct answers score higher than redundant ones, and partially correct answers score higher than entirely incorrect ones. Evaluation on 4 different models across 7 benchmarks shows that integrating SAR with prevalent RL algorithms like PPO and GRPO reduces answer length by 30%, while improving accuracy by 4%. Our analysis also shows that SAR generalizes well to out-of-domain tasks and achieves a Pareto-optimal frontier between correctness and efficiency compared to state-of-the-art baselines. We also show that SAR shortens unnecessary elaboration while preserving advanced reasoning behaviors. These results highlight the promise of self-aligned reward as a fine-grained complement to verifiable rewards, paving the way for efficient and effective LLM training. All of our code implementations and data are publicly available at GitHub.

Published as a conference paper at ICLR 2026
SELF-ALIGNED REWARd: TOWARDS EFFECTIVE AND EFFICIENT REASONERS
Peixuan Han' Adit Krishnan Gerald Friedland Jiaxuan You' Chris Kong²t
University of Illinois Urbana-Champan Champaign AAmazon Web Services
ph16@illinois edu,, luyankon amazon. com
ABSTRACT
Reinforcement learning with verifiable rewards has significantly advanced rea- soning with large language models (LLMs) domains such as mathematics and logic. However, verifiable signals provide only coarse-grained binary correct- ness feedback. This limitation results inefficiencies like overly verbose or repet- itive reasoning. Existing magth-bbased solutions (e.g., length penalty) compromise accuracy. To address this deficiency, we introduce self-aligned reward (SAR), a generic, universally applicable self-guided signal that complements verifiable re- wards to enhance both reasoning accuracy and efficiency in RL. Specifically, SAR is defined the relative perplexity difference between an answer conditioned on the query and the standalone answer, thereby favoring responses that are concise and query-specific. Quantitative analysis reveals that SAR reliably judges answer quality: concise, correct answers score higher than redundant ones, and partially correct answers score higher than entirely incorrect ones. Evaluation on 4 differ- ent models across 7 benchmarks shows that integrating SAR with prevalent RL algorithms like PPO and GRPO reduces answer length by 30%, while improving accuracy by 4%. Our analysis also shows that SAR generalizes well out-of- domain tasks and achieves a Pareto-optimal frontier between correctness and ef- ficiency compared to state-of-the of-the-art baselines. We also show that SAR shortens unnecessary elaboration while preserving advanced reasoning behaviors. These results highlight the promise of self-aligned reward fine-grained complement to verifiable rewards, paving the way for efficient and effective LLM training. All of our code implementations and data are publicly available at Github
Acc
X
SAR
PPO X
Acc
SAR
Acc
GRPOX
SAR
GRPOX
SAR
45
GRPO
ER
BR
X
15
15
15
PPO X
ER
15
Basc
Basc
Base
BB
Eff
Base
20
-60
-40
-20
20
2.0
40
-100
-50
Qwen3-1.7B
Qwen3-4B
Phi-3.5-minl
Gemma3-18
Figure I: Training with self-aligned reward enhances both efficiency and accuracy. We present the relative gains in efficiency and accuracy compared to the respective base model in math reasoning benchmarks. Efficiency gain is measured the drop in average response length.
INTRODUCTION
Recently, reinforcement learning (RL) with verifiable rewards has attracted broad attention in LLM training, showing remarkable improvements in reasoning skills (Guo et al., 2025; Jaech al., 2024).
"Wok done during internship at Amazon Web Services.
I "oorespooniig author.
1https:/ :/g/ thub. com / ama zon- sci ence/Se f-Al Rffect i and effi officient Reasoners

---

Published as a conference paper at ICLR 2026
Table I: Comparison of different reward designs.
Reward
Continuous
Internal
Content-Aware
Correctness
Conciseness
Correctness
X
Reward Model (Ouyang et al., 2022)
/
x
/
x
Length Penalty (Chen al., 2025a)
/
/
X
Entropy (Agarwal et al., 2025)
/
/
/
X
x
Self-Aligned (Ours)
/
/
/
/
However, such verifiable signals are inherently discrete and coarse: they only judge final answer correctness, but fail to capture finer distinctions among responses. For instance, an unnecessarily long solution receives no penalty as long as the final answer correct, and an almost correct response is treated the same as completely wrong one. This limitation often induces "overthinking" where models generate unnecessary elaborations that increase latency and cost (Sui et al., 2025).
To this end, researchers have proposed heuristic regulations such as length penalties or brevity- oriented objectives (Luo et al., 2025; Aggarwal Welleck, 2025). While effective in reducing out- put verbosity. these methods often penalize both redundant and essential reasoning, thereby harming accuracy when necessary intermediate steps are suppressed. Consequently, this line of approaches struggles to balance efficiency with correctness. Using external signal sources, such as reward mod- els, is also undesirable due to their vulnerability to reward hacking. This underscores the necessity of developing internally grounded reward mechanisms that provide precise and detailed guidance, discerning necessary reasoning from redundant elaboration.
To close this gap, we introduce Self-Aligned Reward (SAR), self self-guided proxy to judge answer quality (Equation (6)) based on perplexity, an informative metric modeling uncertainty (Friedland et al., 2024; Agarwal et al., 2025). Specifically, SAR evaluates the perplexity of an answer both in isolation and when conditioned on the query, and then measures their relative difference between the two. Consequently, the reward promotes answers that are highly confident under the query context but unlikely to arise independently without the query, which typically corresponds to responses that are concise and strongly aligned with the query. Notably, SAR is the only fine-grained approach that promotes accuracy and efficiency at the same time, as shown in Table 1.
We first conduct quantitative analysis of different types of answers to demonstrate that SAR pro- vides an accurate fine-gre fine-grained reward landscape over answers of different qualities (Section 4). We then train I.LMs by combining SAR and verifiable reward in PPO and GRPO, two prevalent re- inforcement learning algorithms. We find PPO and GRPO with SAR (denoted as SA-PPO and SA-GRPO) achieve notable gains over baselines across 4 models and 7 benchmarks, improving accuracy by 4% and efficiency by 30% (Section 5.2). Moreover, SAR outperforms length- rewards with a Pareto-optimal front in the accuracy-efficiency trade-off (Section 5.3). In addition, demonstrate the advantages of SAR over confidence-bait based methods (Section 6.1) and provide an analysis of its reasoning behaviors (Section 6.2). Our findings suggest that combining verifi- able rewards with intrinsic model self-judgment offers new paradigm for RL training, enabling improvements in both reasoning capability and efficiency.
RELATED WORK
Reinforcement Learning for LLMs. Reinforcement learning (RL) has emerged a powerful paradigm for tnnettnnig large language models (LLMs) to enhance their performance in reasoning tasks (Ouyang et al., 2022; Guo et al., 2025). Proximal Policy Optimization (PPO) (Schulman et al., 2017) and Group Relative Policy Optimization (GRPO) (Shao et al., 2024) are typical algorithms that are widely adopted in diverse scenarios and domains (Kulkarni et al., 2024; Chen et al., 2025b; Han et al., 2025a; Liu et al., 2025b). Improvements to these algorithms have also been proposed, namely GRPO (Liu et al., 2025a), GSPO (Zheng et al., 2025), and Clip-Cov (Cui et al., 2025).
Efficient Reasoning. Reasoning models often suffer from overthinking (Su et al., 2025; Cuadron et al., 2025; Sui et al., 2025), leading to unnecessary computation burdens. Prompt engineering (Han et al., 2024; Ma et al., 2025a; Renze & Guven, 2024a) and instruction tuning (Yu et al., 2024; Kang et al., 2025; Xia et al., 2025; Han et al., 2024; Ma et al., 2025b) have been widely used
2

---

Published as a conference paper at ICLR 2026
to mitigate this drawback. Recently, researchers have explored reward signals in RL to penalize lengthy answers (Aggarwal & Welleck, 2025; Team et al., 2025b) or relative lengths (Luo et al., 2025; Arora & Zanette, 2025). In addition, Yeo et al. (2025) explores the relationship of the length penalty with reasoning behaviors and Chen et al. (2025a) proposes a difficult--enssitive method for token compression. However, these methods sacrifice accuracy to gain efficiency, which is a key issue this paper aims to address.
Self-judging of LLMs. The self-judgment capabilities of LLMs (Renze & Guven, 2024b) play key role in scalable training and self-evolution. For instance, self-confidence estimation is widely used reduce hallucination (Geng et al., 2023; Wen al., 2024; Ji et al., 2023). probing internal states provides a lens to analyze model behavior (Han et al., 2025b; Chen et al., 2024), and majority voting (Wang et al., 2022) critical inference-time scaling techniques (Snell et al., 2024). Self- judging has also been applied to train reasoning models, with primary focus on ontropy-based metries (Agarwal et al., 2025; Lei et al., 2025; Zhao al., 2025). Zhang et al. (2025) provides theoretical foundation to self-judged signals.
METHODOLOGY
3.1 REINFORCEMENT LEARNING FORMULATION
Proximal Policy Optimization (PPO) (Schulman et al., 2017) and Group Relative Policy Optimiza- tion (GRPO) (Shao et al., 2024) are two widely adopted RI. algorithms. Given the current policy Tg, query q. and the sampled rollout a, PPO and GRPO can be formulated as:
(To) Eg~DDawTo(a))
1
|a| min fryAj, clip(r,, 1- +) B KL (eet)
[
(a | g, 01.)-1)
where rj
[ata 4, (1..j-11)) Aj = Q(q. V a1g.1...))
(2)
[ouro(To) = E.g an... a.. Norale)
N EN
Ejed min (rigd,, clip(ri,, +)A, (L
(3)
noe (a:,, 9, (..1...-11) A, H(g.a.) mean[ a1...)]
where ri.j Tola (:.1...-1))
(4)
In the above formulas, A is the advantage, D is the training dataset, and r refers to the importance sampling ratio. N, e. Lmax and 3 are hyperparameters. Specifically, we use Dr. GRPO (Liu et al., 2025a), an unbiased variant of GRPO this paper. For verifiable tasks, the reward for PPO and GRPO can be obtained by comparing the model answer with the expected ground truth (gt):
RpPOIGRPO a, gt) (vv(gg,,gt) [1 if gt in a,
otherwise.
Although RL with verifiable reward has set new benchmarks reasoning tasks, it induces undesired characteristics such redundant output due to the coarse binary nature of the reward. To address this issue, two widely used methods introduce targeted penalities on output length: O1-pruner (Luo et al., 2025) and Efficient Reasoner (Arora & Zanette, 2025) (referred to O1 and ER). Their detailed formulations are provided in Section B.2. Despite their effectiveness, both methods inevitably trade off accuracy, which constrains their broader applicability.
3.2 SELF-ALIGNED REWARD
We propose Self-Aligned Reward (SAR) (denoted as RSA), an unsupervised holistic signal that combines generation quality, conciseness, and query-answer relevance. The self-aligned reward estimated entirely by the model policy without any external feedback. We then combine our reward with the binary verifiable reward to address the shorteomings of previous methods, aiming to improve both accuracy and generation efficiency. The reward for self self-aligned PPO and aligned GRPO (SA-PPO and SA-GRPO) can be formulated follows?:
3

---

Published as a conference paper at ICLR 2026
RSA-PPOIGRPO as as.gt) RyR + aRsA, RSA =clip (pl((.)) - ppl(a; (q) 11,1)
(6)
ppl(a.)
where ppl(a) = पिं Ej= log Pogp(ryll ppl(alg) (िं E Pogg(1,1....1....
The self-aligned reward captures the relative difference between ppl(a) and ppl(alg). or the condi- tioned perplexity drop, based on the current model policy. Perplexity reflects the likelihood that the model will produce the given response, with lower values indicating higher confidence. Thus, RSA can be interpreted as: "How much less likely does the answer become if the query is not present?"
When an answer is tightly tailored to the query, the conditioned perplexity ppl(alg g) will be signif- icantly lower than the standalone perplexity ppl(a), leading to higher RSA. In contrast, if parts of the answer are irrelevant to the query or include noisy, verbose text, the two perplexities will be similar, leading
smaller RSA- Therefore, larger value of RSA indicates the answer's stronger dependency and better alignment to the query.
Section 4 provides in-depth analysis of this reward formulation, where we demonstrate that self- aligned reward encourages concise reasoning and effective use of query information, thereby im- proving both accuracy and training efficiency, and penalizing noisy verbose answers. In addition, the computation of RsA integrates seamlessly into the RL pipeline with negligible computational overhead, as shown Sections and 6.3.
CASE ANALYSIS: WHY SELF-ALIGNED REWARD WORKS
This section analyzes the self self-aligned reward formulation, providing insights on how SAR enhances the reasoning effectiveness and efficiency.
4.1 SAR PROVIDES ACCURATE AND FINE-GRAINED SIGNAL.
An ideal reward design should be able to rate different types of answers based on their qualities. To assess different reward functions, we analyze 6 different types of answers to 200 math questions sampled from five common math benchmarks (section 5.1) in Table 2. Types (1) to (4) obtained by sampling Qwen3-1.7B rollouts at temperature = 1, and using GPT-4o to annotate the responses (see Section C for details). Types (5) and (6) are artificially synthesized to simulate memorization, where the LLM directly extracts answers from its knowledge without reasoning.
We calculate advantage values over the 6 types of answers to the same question (a 1), and report the average over 200 questions. The advantage values of SA-GRPO accurately rate answers of different qualities.
Table 2:
Answer Type
Length
AGRPO
AGRPO-al
AGRPOSE
ASA.GRPO (Ours)
(I): Correct and concise
143.5
0.5
1.04
0.81
1.15
(2): Correct but redundant
236.0
0.5
0.35
-0.11
1.00
(3): Partly correct with wrong answer
457.1
-0.5
-1.28
-0.65
-0.01
(4): Completely wrong irrelevant
405.8
-0.5
-1.19
-0.65
-1.44
(5): Correct but thought
5.0
0.5
1.04
1.25
-0.04
(6): Incorrect and no thought
5.0
-0.5
0.04
-0.65
-0.68
We present the advantage values (Equation (4)) for each type of response using the following re- wards: verifiable GRPO rewards (AGRpO), length-based rewards from the Ol and ER methods (AaRro-ol, ACRPO-ER) and our self-aligned reward ASA-GRPO. From Table 2, we can observe that SAR provides rich and accurate signals to different answers, exhibiting the following features:
SAR favors concise and correct answers; it gives a lower reward to long and redundant answers, promoting efficiency. Section 4.2 explains this phenomenon in detail.
SAR provides partial credit to partly correct answers and penalizes completely irrelevant ones, helping the model learn basic reasoning patterns in the initial stage of training.
Sinne ppl 0, always have RSA In practice, RSa lies in the (0, 1) range cases.
4

---

Published as a conference paper at ICLR 2026
SAR penalizes the synthesized "no thought" answers, even if they're correct and short. This indicates that the reasoning process plays a critical role SAR, and memorization is discouraged.
On the other hand, the verifiable reward (GRPO) fails to discern answers that are both correct and incorrect, and Ol or ER focus solely on response length, making signals biased and not suitable for maximizing accuracy. For instance, ER isn't applicable to wrong answers, and O1 favors irrelevant answers over partly correct ones, simply because they're re shorter. We also provide a qualitative example in Table 12 illustrate the different answer types.
4.2 SAR PROMOTES EXPLOITING QUERY INFORMATION EFFICIENTLY
In this experiment, we take deeper look at SAR at the token level, aiming to reveal which tokens contribute more to the overall score. Specifically, we decompose the self-aligned reward (Equa- tions (6) and (7)) to calculate the token-level score":
RSA ppl(alg) ppl(a)
Plailue11.g11 = -1-6 E/ tog Play "1...--1)
(8)
We then define v(a,) log P(a, P(a.1a.at..3-1)
Pla (c.] to measure the importance of each token d, on the whole metric RSA- token with a higher u(ay) is considered valuable, while a token with low or even negative v(a,) indicates it's less informative and independent of the query.
5
Question: Janct's ducles 16 per She three for breakfast every morning and balees muffins for her the
friends every day
at farmers' market duily for per duck egg, How much in dollars does the make every day at the farmers' mariset?
0
Answer: Janet's duckss lay 16 eggs per day. She eats 3 for breakfast every morning. So remain Janet
baies eggs in muffins every for firends, subtract these as well: 13 4 eggs. Hence, after considering all the consum ptions, Janet sells eggr to the farmers' marlect daily. She sells each egg at $2, Therefore, Janet's daily revenue is 9 $2 $10.
to
-5
v(ag)
Figure 2: An illustration of token-level importance scores (i.e. v(a,)). Red color means the token considered informative for our self-aligned reward, and blue indicates a negative v(a,). Tokens extracting new information from the query get high scores.
From Figure 2, we observe that tokens drawing on information from the question for the first time, like "Janet", "duck", "16", tend to receive high scores. This is because such information is present in the query but absent from previous answer tokens, making P(a; [q, (1...--) relatively high while P(aylai... remains low. In contrast, repeating information already generated, like mentioning "Janet" for the second time, results in low scores as both probabilities become similarly high.
Generally, tokens in the earlier part of answer typically achieve higher v(a,) values, since tracting new content from the query is easier at the beginning. Later tokens, by comparison, struggle to introduce novel information once much of the query has already been incorporated. This explains why SAR favors short, concise answers and promotes more efficient generation.
5
EXPERIMENTS
5.1
EXPERIMENTAL SETTINGS
Datasets. We utilize five math reasoning benchmarks: GSM8k (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), NuminaMath 1.5 1L5 et al., 2024), GSM-symbolic (Mirzadeh et al., 2024), and AIME 1983-20224 (Veeraboina, 2024), covering difficulty range from primary school to competition level. For training, we combine the training splits of the first three datasets, while the remaining two are not present during training to evaluate generalization.
Models. We utilize four base models: Qwen3-1.7B (Base), Qwen3-4B (Base) (Yang et al., 2025), Phi-3.5-mini (Instruct, 3.8B) (Abdin et al., 2024), and Gemma3-IB (Instruct) (Team et al., 2025a), covering different sizes and model families.
We omit the clipping for simplicity.
5

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation on math benchmarks. Self-aligned reward signifieantly reduces token usage while achieving the best reasoning accuracy. GRPO-O1 and GRPO-ER results in the table use 0.05, and SA-GRPO uses 0.2. Best results among each model are bolded.
GSMBk
GSM-sym
MATH
NuminaMath
AIME
Average len
Setting
len
acc
Icn
acc
len
len
len
AES
Qwen3-1.7B
69.22
281
46.76
365
56.89
700
24.58
1242
9.97
1539
41.48
825,4
0.000
+PPO
82.71
332
62.88
460
63.04
784
27.02
1358
9.11
1718
48.95
930,4
0.773
+SA-PPO
85.51
281
68.02
369
66.27
S87
32.05
866
10.61
1015
52.49
623.6
1.572
+GRPO
84.53
335
67.66
413
67.31
697
33,44
1091
13.93
1278
53,37
762.8
1.509
+GRPO-O1
84.15
285
65.78
363
66.51
528
33.15
775
13.61
910
52.64
572.2
1.652
+GRPO-ER
82.71
244
64.74
320
66.54
-57
34.35
980
12.22
1232
52.11
666.6
1.474
+SA-GRPO
85.51
267
67.66
346
67.96
564
36.03
13.50
54.13
602.0
1.795
Qwen3-4B
76.19
315
66,22
435
63.79
652
38.80
1142
20.15
1281
53.03
765.0
0.000
+PPO
91.36
373
85,42
485
82.15
1132
50.80
1995
26.90
2680
67.33
1333.0
0.606
+SA-PPO
92.12
266
83.52
345
78.67
652
49.09
1133
28.19
1397
66.32
758.6
1.260
+GRPO
92.62
320
86.30
414
82.51
821
53.79
1485
30.11 1
2113
69.07
1030.6
1.165
+GRPO-O1
91.89
262
86.76
350
80.41
549
51.06
866
28.62
1016
67.75
608.6
1.592
+GRPO-ER
92.80
219
85.24
297
79.48
503
50.47
854
28.94
1022
67.38
579.0
1.596
+SA-GRPO
93.40
239
87.64
323
82.63
762
57.70
1358
35.69
1788
71.41
894.0
1.564
Phi-3.5-mini
71.19
287
56.86
398
33.83
1132
11.56
1492
2.35
943
35.15
850.4
0.000
+PPO
82.49
209
64.88
322
38.56
831
11.45
1707
3.85
926
40.24
799.0
0.784
+SA-PPO
87.04
238
73.52
344
46.34
300
17.43
586
2.79
501
45.42
393.8
1.997
+GRPO
87.56
235
73.76
337
48.69
415
16.37
582
3.21
672
45.91
448.2
2,003
+GRPO-01
79.83
203
62.24
279
44.41
390
14.08
509
2.57
583
40.62
392.8
1.316
+GRPO-ER
82.03
154
63.54
240
43.61
363
12.55
511
2.47
606
40.84
374,8
1.368
+SA-GRPO
87.95
207
72.72
289
50.99
356
16.08
455
3.22
535
46.19
368.4
2.137
Gemma3-1B
42.15
325
19.58
427
36.40
1042
12.40
1414
2.25
2279
22.56
1097
0.000
+PPO
56.86
942
30.28
1187
44,17
1408
15.57
1904
1.60
2422
29.69
1572.6
1.146
+SA-PPO
55.80
683
31.00
936
42.94
1025
14.22
1523
1.50
1683
29.10
1170.0
1.383
+GRPO
59.97
1208
34.70
1552
45.40
1693
16.59
2264
2.25
2613
31.78
1866.0
1.343
+GRPO-O1
60.80
873
32.06
1157
44.76
1298
16.16
1871
2.35
2449
31.23
1529.6
1.528
+GRPO-ER
59.44
748
32.36
1063
46.23
1131
16.41
1545
2.04
1696
31.29
1236.6
1.808
+SA-GRPO
61.26
552
34.52
772
46.60
952
16.70
1302
2.14
1509
32.24
1017.4
2.218
Settings. We train reasoning models with the following methods:
PPO, GRPO:
two traditional RL algorithms, using only the verifiable reward.
GRPO-O1, GRPO-ER: algorithms with length penalties, aimed to enhance efficiency. See Sec- tion B.2 for details.
SA-PPO, SA-GRPO: algorithms using self-aligned reward (SAR). See Section 3.2 for details.
For training details and hyperparameters, refer to Section B.3.
Metries. In this work, we focus on accuracy and efficiency (measured by average answer length) of L.M reasoning. We report these two metrics as well as an Accuracy-Eiffciengg trade-off Score
(AES). For a trained policy ma and its base model Weef we define A len len( -len(ra) and
len(Rat
A acc acc(mT) - Then, AES(Ta) = A len +y acc measures the trade-off." (1a))
5.2 MAIN RESULTS
From Table 3, we observe that baseline approaches struggle to balance accuracy and efficiency. PPO and GRPO significantly improve accuracy the cost of longer and possibly redundant an- swers. While GRPO-O1 and GRPO-ER effectively reduce computation overhead, they compromise reasoning capability. In contrast, SAR delivers substantial improvements efficiency. Across four base models, SA-GRPO consistently achieves the highest reasoning accu- racy while maintaining highly efficient reasoning, with at least 4% improvement in accuracy and 30% reduction in length compared to GRPO. Notably, SA-GRPO produces answers of comparable
both accuracy and
"The choice of value reflects which aspect does the user prioritize. Since accuracy is the most important factor in most use cases, we set 7 = 5 in practice.

---

Published as a conference paper at ICLR 2026
even shorter length than GRPO-O1 and GRPO-ER, which are explicitly designed for efficient rea- soning. A similar phenomenon observed for SA-PPO, indicating that self-aligned reward applies to diverse RL algorithms. We show two examples comparing GRPO and SA-GRPO in Section F.2.
These findings highlight the superiority of the self-aligned reward mechanism. With a more fine- grained and intelligent signal, SAR makes the model preserve the necessary reasoning which are closely related to query information, and reduces unnecessary content that introduces token burdens, leading to more effective and efficient reasoning.
5.3 SELF-ALIGNED REWARD ACHIEVES A PARETO-OPTIMAL IN DYNAMIC BALANCE
Performance Cain (%)
Performance Gain (%)
2
21
Length Reduction (%)
Length
-2
..01
15
03
.3f
45
20
.1f
405
.29
.37
Reduction (%)
60
.05
.1
-2
.01
.01
-6
X
GRPO
ER
.03
.05
O1-pruner
SA-GRPO (Ours)
-6-
-10
(a) Qwen3-1.7B.
(b) Qwen3-4B.
Performance Gain (26)
Performance Gain (%)
2
Length Reduction (%))
2
Length
-2
10
30
10
Reduction 80
(%)
.01
*
.03
.05
.05
40
-10
.01
.03
.05
-6
.1.15
.1
-14
.05
-10
.15
(c) Phi-3.5-mini.
(d) Gemma3- -B.
Figure 3: Accuracy-eficiency balance of different algorithms. SA-GRPO reaches a Pareto- optimal curve and achieves notable gains on both axes. Numbers around each point indicate the a values, and & indicates using 20% of the total training data with the same total training steps.
The hyperparameter a in Equation (6) controls the weights of verifiable reward and self-aligned reward, offering flexibility to focus more on efficiency or accuracy. Therefore, we train SA-GRPO with different values and compare it with GRPO-O1 and GRPO-ER on 2-dimensional plots.
From Figure 3, we can observe that the curves for SA-GRPO are consistently on the top-right side over curves for GRPO-O1 and GRPO-ER, indicating SA-GRPO balances accuracy and efficiency better than length-based methods. Moreover, length-based methods are always under the x-axis in most cases, showing that these methods sacrifice accuracy for reduced tokens, while SA-GRPO is the only method consistently above the x-axis (illustrated as the light purple area in the figure), showing better accuracy and better efficiency compared to vanilla GRPO. The Pareto-optimal bal- ance between accuracy and efficiency of SA-GRPO further demonstrates the effectiveness of SAR.
5.4 SELF-ALIGNED REWARD GENERALIZES TO LOGICAL. REASONING
In this section, we examine the generalization ability of self-aligned reward (SAR) to different domain - logical reasoning via two benchmark datasets: LogicBench (Parmar et al., 2024) and ProntoQA (Saparov & He, 2022). Table 4, we find that SA-GRPO improves accuracy compared to GRPO on 5 out of 8 columns, and outperforms length-based methods (GRPO-O1 and ER) on all cases. Similarly, SA-PPO outperforms the PPO baseline on 6 columns. SAR maintains an efficiency gain in the logical domain, not only reducing answer length compared to GRPO and vanilla length- based methods in most cases. These results show that SAR generalizes well to the out-of- out-of-domain logical reasoning benchmarks, validating our overall reward design.

---

Published as a conference paper at ICLR 2026
Table 4: SA-PPO and SA-GRPO maintains robustness and efliciency in out-of-domain tasks.
(a) LogicBench.
(b) ProntoQA.
Qwen3
Qwen3
Phi-3.5
Gemma3 -IB
Qwen3
Qwen3
Phi-3.5
Gemma3
Model
-1.78
-4B
-mini
Model
-1.7B
-4B
-mini
-IB
acc
len
acc
len
acc
len
acc
len
acc
len
acc
len
acc
len
acc
len
Base
56.0 182
75.4
336 66.0 309 50.6
303
Base
69.0
616
89.6
988
94.8 339
56.2 712
PPO
60.2
340
80.0
455 60.8
229
48.8
264
PPO
71.8
920
97.8
561
96.4
301
56.4
904
SA-PPO
64.6
368
77.6
342
64.2
205
55.6
241
SA-PPO
65.6
735
99.0
405
98.2
304
57.6
690
GRPO
64.2
372
78.4
416
67.6
243
53.0
277
GRPO
74.0
756
100.0
546
98.2
254
57.4
963
GRPO-01
60.6
264
76.0
350
51.0
212
52.6
223
GRPO-01
73.0
505
98.8
469
87.2
227
59.0
938
GRPO-ER
61.4
206
77.6
263
49.8
50.0
250
GRPO-ER
73.6
424
99.0
309
86.6
205
58.6
919
SA-GRPO
65.0
278
80.4
208
57.4
173
55.2
222
SA-GRPO
77.8
482
99.8
301
90.8
216
60.6
733
6 ANALYSIS
6.1 ABLATION STUDY
In this section, we investigate two critical components of our algorithm that make internal signals effective during training. Firstly, the self-aligned reward (Equation (6) measures conditioned per- plexity drop, which is the relative difference between ppl(alq) and ppl(a), as a proxy for query- answer relevancy. However, existing methods focus mainly on ppl(alg) alone, which measures entropy or uncertainty(haao et al., 2025; Agarwal et al., 2025). our approach combines verifiable reward with self-aligned reward. We ablate these components and derive the following rewards to compare against SA-GRPO (R RvR aRsA):
R RsA: this reward removes the verifiable reward, using only the self-aligned reward.
R lvv + alEM RvR olog pal(a q):: this reward uses "entropy minimization", metric measuring self-confidence, as the internal reward, instead conditioned perplexity drop:
Rt = REM log (alg)) this reward is the entropy minimization reward (Agarwal et al., 2025), ablating both components.
Table 5: GRPO training results on Qwen3-4B with different internal reward signals (a = 0.2). The SA-GRPO formulation obtains optimal performance among baselines.
GSM8k
GSM-sym
MATH
NuminaMath len
AIME
Average
Setting
acc
len
acc
len
acc
acc
len
acc
len
acc
len
Base
76.19
315
66.22
435
63.79
652
38.80
1142
20.15
1281
53.03
765.0
RVR
92.62
320
86.30
414
82.51
821
53.79
1485
30.11
2113
69.07
1030.6
REM
79.15
319
67.94
414
68.89
970
39.86
1885
21.44
2556
55.46
1228.8
Rsa
39.65
87
19.54
102
24.72
74
16.48
80
4.39
79
20.96
84.4
Rve + Ress
92.25
351
87.36
414
81.83
779
54.78
1371
33.01
1767
69.85
936.4
RVR aRRa
93.40
239
87.64
323
82.63
762
57.70
1358
35.69
1788
71.41
894.0
From Table 5, we can observe that both verifiable signal and conditioned drop measurement are crit- ical for optimal performance. Specifically, REM shows limited accuracy gain but a large efficiency drop, and RSA converges to shallow reasoning with fewer tokens and poor accuracy. Failure of these methods indicates that the ground-truth signals still critical for models to develop reasoning skills and ensure training stability, similar to what's found in Zhang et al. (2025)
RVR + aREM also underperforms SA-GRPO in both accuracy and efficiency, indicating that mini- mizing entropy is less effective than our approach, which is maximizing conditioned perplexity drop. This is because conditioned perplexity drop provides a more accurate measure of answer quality, as demonstrated in Section 4. Moreover, it avoids issues such as overconfidence or entropy collapse, which can hinder exploration (Zhang et al., 2025; Cui et al., 2025). Overall, the results suggest
8

---

Published as a conference paper at ICLR 2026
"verifiable signal" and "conditioned perplexity drop" are two crucial components in SAR, without which SA-GRPO won't be able to reach optimal performance in accuracy and efficiency.
6.2 REASONING BEHAVIORS OF SA-GRPO
Previous work (Gandhi et al., 2025; Zeng et al., 2025) has shown that certain reasoning behaviors critical to effective and deep reasoning. this section, we investigate four typical behaviors: back- tracking, verification, subgoal setting, and enumeration. Specifically, we use GPT-4o to annotate the reasoning behaviors for models trained with different algorithms (see Section C for details).
Table 6: Frequency of reasoning behaviors. B, V, S. E refer to Backtracking, Verification, Subgoal setting and Enumeration, respectively. Scores are in percentages. The base model is Qwen3-1.7B. Unlike length-based methods, SA-GRPO maintains high usage of reasoning behaviors.
Datasct
MATH
NuminaMath
AIME
Average
Behavior
B
V
S
E
B
V
S
E
B
V
S
E
Base
24.2
90.8
12.2
5.4
27.0
93.4
34.8
4.4
36.2
93.2
37.4
38.0
GRPO
1.0
29.2
95.4
14.6
4.6
38.8
97.8
44.2
5.0
42.2
98.2
49.0
43.4
GRPO-O1
1.2
26.6
90.4
14.6
4.8
37.0
93.2
39.6
4.2
43.4
96.0
42.8
40.6
GRPO-ER
1.4
26.0
89.8
13.4
6.2
37.0
90.4
40.4
5.8
44.2
96.2
47.6
41.2
SA-GRPO
0.8
29.2
93.4
13.8
7.0
36.0
93.8
43.8
8.8
47.0
97.0
49.6
43.0
From Table 6, we can observe that GRPO-O1/ER exhibit fewer reasoning behaviors compared to GRPO, reasoning behaviors require additional tokens which are penalized by sength-based reward functions. However, SA-GRPO maintains almost the same frequency of reasoning behaviors with GRPO, notably with 30% fewer tokens than GRPO, This stems from the self-aligned reward's content-aware feature, showcasing its ability to accurately distinguish useful reasoning behaviors from unnecessary content, guiding model behaviors in a more accurate and unbiased manner.
6.3 TRAINING COST OF SELF-ALIGNED REWARD
We report the training cost for SA- GRPO in Table from which we can find that SA-GRPO doesn't 't in- troduce burdens "Update" phase compared to vanilla GRPO. The only additional computation in SAR is (p)(a) (ppl(alg) is already calculated in GRPO, for KL penalty and impor- tance sampling), which only requires a forward pass, making highly ef- ficient. Additionally, SA-GRPO even takes less time in the "Rollout" phase due to reduced answer lengths. This suggests that enhancing reasoning efficiency not only valuable during inference it can also accelcrate RL training and maximize the gain of reasoning capability under limited computation.
Table 7: Comparison of training time. We report GPU hours (Training time GPU count) of training Qwen3-4B for the first 200 steps.
Method
Rollout
Update
Total
GRPO
32,95
15.13
48.08
GRPO-O1
30.36
15.92
46.28
GRPO-ER
30.92
15.24
46.16
SA-GRPO
31.44
15.20
46.64
7 CONCLUSION
In this work, we propose Self-Aligned Reward (SAR), an internal perplexity-basee signal evalu- ating the answer's relevancy with the query, enabling fine-grainee supervision beyond binary cor- rectness. Through comprehensive experiments on base models and 7 benchmarks, we demon- strated that SAR enables reinforcement learning to achieve consistent gains of up to 4% in accuracy while reducing response length and computational cost by 30%. Moreover, SAR exhibits a favor- able accuracy-efficiency balance compared with length-based baselines, offering a fine-grained and content-awe aware reward signal that complements verifiable correctness. Our analysis further shows that SAR generalizes robustly out-of-domain tasks and preserves advanced reasoning behaviors, underscoring its broad applicability. Our findings highlight the significance of incorporating intrin- sic fine- graigre model self-assessmentt into the RI. framework.

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We ensure the reproducibility of the paper from the following aspects:
In Section A, we describe the datasets and their preprocessing methods. All datasets open-sourced.
Dataset:
Our contribution is self-aligned reward (SAR), simple yet effective internal sig- nal. We present the formula and explanations of SAR in Section 3.2, and show the core code for implementing SAR in Section B. In addition, formulations and implementations of the haselines also presented. Our code is based on VERL, an open-sourced and widely adopted RL framework.
Method:
Training configurations are presented in Section B.3. Readers reproduce the exact training results following these settings.
Training:
The paper's analytical experiments involve annotating with LLM. Prompts for such annotation are presented in Section C.
Prompting:
LLM USAGE STATEMENT
Large language models were not used in the writing this paper, except for the sample responses from trained LLM models for illustrative purposes. All written content and experimental code were generated solely by the authors.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone, 2024. URL https://arxiv. org/abs/2404.14219 2:6, 2024.
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effec- tiveness of entropy minimization in IIm reasoning. arXiv preprint arXiv:2505. 15134, 2025.
Pranjal Aggarwal and Sean Welleck. LI: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.0440 2025.
Daman Arora and Andrea Zanette. Training language models to reason efficiently. arXiv preprint arXiv:2502.04466 04002, 2025.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Yc. Inside: Lims' internal states retain the power of hallucination detection. arXiv preprint arXiv:2402.03744, 2024.
Weize Chen, Jiarui Yuan, Tailin Jin, Ning Ding, Huimin Chen, Zhiyuan Liu, and Maosong Sun. The overthinker's diet: Cutting token calories with difficulty-aware training. arXiv preprint 1921122005.1921, 2025a.
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang. Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-rl: Reward modeling as reasoning. arXiv preprint arXiv:2505.02387. 2025b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang. Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking: Examin- ing the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.0822: 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo. Haozhan Li, Yuchen Fan. Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint 250. 2025.
10

---

Published as a conference paper at ICLR 2026
Gerald Friedland, Xin Huang, Yueying Cui, Vishaal Kapoor, Ashish Khetan, and Sanjiv Das. Pplqa: An unsupervised iiformation-theoree quality metric for comparing generative large language models. arXiv preprint arXiv:2411.15320, 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cogni- tive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint wrXiv:2503.0137., 2025.
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. arXiv preprint arXiv:2311.08200 2023.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint vrXiv2551..1945 2025.
Peixuan Han, Zijia Liu, and Jiaxuan You. Tomap: Training opponent-aware IIm persuaders with theory of mind. arXiv preprint arXiv:2505.22961 2025a.
Peixuan Han, Cheng Qian, Xiusi Chen, Yuji Zhang. Denghui Zhang, and Heng Ji. Safeswitch: Steering unsafe Ilm behavior via internal activation signals. arXiv preprint vrXiv:2502.01042, 2025b.
Tingxu Han, Zhenting Wang. Chumrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware IIm reasoning. arXiv preprini urXiv:24/2.185477 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.038/7 2021.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint wrXiv:2412.16720 2024.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating IIm hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1827-1843, 2023.
Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter thain-of-thought without compromising effectiveness. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, pp. 24312-24320 2025.
Mandar Kulkarni, Praveen Tangarajan, Kyung Kim, and Anusua Trivedi. Reinforcement learning for optimizing rag for domain chatbots. arXiv preprint arXiv:2401.00 106800, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Shiye Lei, Zhihao Cheng, Kai Jia, and Dacheng Tao. Revisiting Ilm reasoning via information bottleneck. arXiv preprint 2XXv:2557.11839 2025.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783 2025a.
Zijia Liu, Peixuan Han, Haofei Yu, Haoru Li, and Jiaxuan You. Time-rl: Towards comprehensive temporal reasoning in llms. arXiv preprint arXiv:2505. 13508, 2025b.

---

Published as a conference paper at ICLR 2026
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-prumer: cangth-hrronizingg fine-tuning for ol-like reasoning pruning. arXiv preprint arXhv:250.112770, 2025.
Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking. arXiv preprint arXiv:2504.0999 09858, 2025a.
Xinyin Ma, Guangmian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length- compressible chain-of-thought tuning. arXiv preprint arXiv:2502.0601, 2025b.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv: 24.00522,, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. arXiv preprint arXiv:2404.15522, 2024.
Matthew Renze and Erhan Guven. The benefits of a concise chain of thought on problem-solving in large language models. In 2024 2nd International Conference on Foundation and Large Language Models (FLLM), pp. 476-483. IEEE, 2024a.
Matthew Renze and Erhan Guven. Self-rellectonn in Ilm agents: Effects on problem-solving perfor- mance. arXiv preprint wrXiv:240.006682, 2024b.
Abulhair Saparov and He He. Language models are greedy reasoners: systematic formal analysis of th-thugt- arXiv preprint arXiv:2210.0/22 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06477 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rIhf framework. In Proceedings of the Twentieth European Conference Computer Systems, pp. 1279-1297, 2025.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.
Jinyan Su, Jennifer Healey, Preslav Nakov, and Claire Cardie. Between underthinking and over- thinking: An empirical study of reasoning length and correctness in llms. arXiv preprint arXiv:2505.00127, 2025.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: survey on efficient reasoning for large language models. arXiv preprint 1641:12503.1649, 2025.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786 2025a.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llss. arXiv preprint arXiv:2501. 2025b.
12

---

Published as a conference paper at ICLR 2026
Hemish Veeraboina. Aime problem set (1983-2024),
https://www kaggle com/ datasets wttt://hhwwsseeeeeeeeeioeoe http::/ooo....oooooooooooo------
2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv: 220i::2200.117,,, 2022.
Bingbing Wen, Chenjun Xu, Robert Wolfe, Lucy Lu Wang, Bill Howe, et al. Mitigating overconfi- dence large language models: A behavioral lens on confidence estimation and calibration. In NeurlPS 2024 Workshop on Behavioral Machine Learning, 2024.
Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression lims. arXiv preprint arXiv:2502.12065 2025.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain- of-thought reasoning Ilms. arXiv preprint arXiv:2502.03373, 2025.
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023, 2024.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 2oo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Yanzhi Zhang. Zhaoxi Zhang. Haoxiang Guan, Yilin Cheng, Yitong Duan, Chen Wang, Yue Wang, Shuxin Zheng, and Jiyan He. No free lunch: Rethinking internal feedback for Ilm reasoning. arXiv preprint arXiv:2506. 1125006.772111 17219, 2025.
Xuandong Zhao, Zhewei Kang, Aosong Feng. Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505. 19590, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy op- timization, 2025. URL https: //arxiv...vvv////a////// org/ abs/2507 18071.
13

---

Published as a conference paper at ICLR 2026
A DATASETS
We evaluate our approach on five mathematical reasoning datasets with varying difficulty levels, and two logical reasoning datasets. We list the datasets and provide details below.
A.1 MATIEMATICAL REASONING BENCHMARKS
GSM8k (Cobbe et al., 2021): dataset of grade school math word problems.
2. MATH (Hendrycks et al., 2021): challenging dataset of mathematics problems covering various topics.
3. NuminaMath 1.5 (Li al., 2024): comprehensive dataset containing 860k pairs of competition math problems and solutions. We select a subset from the Open-rl project'.
4. GSM-symbolic (Mirzadeh et al., 2024): A dataset of GSM8k-style problems with differ- ent numbers. This dataset is free from data contamination, making suitable to evaluate generalization ability.
5. AIME 1983-2024 (1244VVeer (Veeraboina, 2024) collection of problems from the American Invi- tational Mathematics Examination spanning over four decades. Requires complex reason- ing.
To ensure parsing correctness, we only select questions where the answer is a single integer or fraction number, and remove questions involving geometric plots or hyperlinks. During training, we combined the training splits of GSM8k, MATH, and NuminaMath. GSM-symbolic and AIME were used exclusively for evaluation to test out-of-distriiution generalization.
A.2 LOGICAL REASONING BENCHMARKS
To evaluate the generalization capabilities of our approach beyond mathematical reasoning, we used the following logical reasoning benchmarks:
1. ProntoQA (Saparov & He, 2022): A dataset evaluating multi-step syllogistic reasoning.
2. LogicBench (Parmar et al., 2024): A comprehensive benchmark for evaluating logical reasoning capabilities of language models across various logical relationships.
These datasets were used only for evaluation purposes and were not part of the training process. Both datasets are in the form of multiple-choice questions, where ProntoQA has 2 choices and LogieBench has 4 for each question.
Table 8 shows the statistics of all datasets used.
A.3 EVALUATION DETAILS
We construct a unified prompt format for all questions:
System Prompt:
You are a reasoning expert assistant. Given a question, you will use your reasoning skills to solve the problem.
User Prompt: [Question]
Please explain your reasoning process before providing an
During evaluation, we parse the last integer or fraction number the model's output and compare it with the ground truth using math_verifs fy package. We don't introduce a strict answer format, as previous work (Zeng et al., 2025) suggest it may hinder exploration.
/0pp//pppnn--mmmeeee httpps: //www//www////.o//////////.. http///wwkggg...eee//// com/ dataseta aeaasset/eemmii shveeraboina/ na/ mmm-probl werccblem-seeee 1983-2224
htttps:/hugginggn
co/ http//aaaatt/eeeeeeeeeeeeee /0ppnk1--aaettaaaaaaamaa 220k
14

---

Published as a conference paper at ICLR 2026
Table 8: Sizes of datasets.
Dataset
GSM8k
MATH
NuminaMath
GSM-sym
AIME
ProntoQA
LogicRench
Train 7473
5654
10000
1319
3742
2742
5000
933
500
500
B IMPLEMENTATION DETAILS
B.1 IMPLEMENTATION OF SAR
We implemented all baselines, well approach building the VERL open-source frame- work Sheng et al. (2025). The VERL framework provides standard implementations PPO and GRPO. To implement approach, need to calculate the perplexity values for the rollouts given the query (ppl(a)q) as well as the perplexity standalone rollouts (ppl(a)).
In most RL algorithms, calculating log probabilities for rollouts necessary to apply KL penalty. These log-probs can be directly applied to calculate ppl(al4) without extra cost:
compute_ ppl prohs mo.k):
log_probe: Tensor (batch_ seq_ien)
mask: Boolean Tenaor [batch_aize, [en], positions of part positions of quezy and padding Lokens aze 0.
..)
sun_log cun_log_proia procas (log_probs lengths musk. (dimm=11)
sasik).suuddimmi)
avg_log_probe prros / lengths return terch.exp (xppeeexppppee (-avg_)og__probb)
The calculation of ppl(a) consists of two steps. The first step is to construct the sequence without the user prompt (we still keep the system prompt):
construct_ sttitrrutt_emtt___ueeee ion Idata, tokenizer, e_max_les)
data: Jaseproto, standard pretocol data exchange VERL. data containg "input_ids", ds", "attentiion_mak* and "position_ids" indd"" which are Tensors; batch_aise, Lemxx_lenta_axx_eee Query are left-padded and anawers are right-padded, which the
always bagin index q_max_len.
emply_g ["System: (sys prompp) wwtserr/nnssistsantt empty_g_tokens = Lokniize padding="left", max mex_tenngheg_max_..
batch_s'ze dsta dsta["inpot ("inpot_ shapee for key in "input__iss" "actention_mask" mositiionids"))
10
11
data[key] :q_max_len] empty_q_tokens [key] .repett (batch_size,
12
last_query_pos data sata("positionnide"" first_ Sirst_answer_pox datal"pozition_idd"] gap first_ siraa__nnswe__pes
q_max_ee
14
g_mex_len] the firs: answer token
15
jast_que=y_pos
this gap should ba zero
16
"caal""potioo_ooe [t, q_max_len: - unsqueeeeee (-1) Shift the answer positions backward by this gap so they are continucus
13
18
return data
The second step is to calculate log probabilities for the "new" sequences. Combining them, implement the self-aligned reward:
R_Sa, tokenizes, q_max_les policy]: ppl_ga = computo
data tata"" ""responss_mosk")
emply_g_data construct_ www.tyyyyeestooneeeeeee
(data, tokenizer, (_max_les) a @mpute_pplltepty// log log_ooog_pooooo probs lompty_g_data
empty_q lor_____ogg___bb_______ = pol policy, walloyyycalee. ca calc___oll
probe, datal" "atal["esgonne__assk
return max (ppl_ Fpl_ga) ppl_a,
In conclusion, the allf-aligned reward calculation is fully compatible with VERL's existing opti- mizations, making it highly efficient for training large models.
15

---

Published as a conference paper at ICLR 2026
B.2 IMPLEMENTATION OF GRPO-O1 AND GRPO-ER
Signals besides verifiable reward have also been widely studied, where enhancing efficiency is prevalent direction. In this paper, consider Op-uner (Luo al., 2025) and Efficient Reasoner (ER) (Arora & Zanette, 2025), two typical length-based methods as baselines. Since the original works used offline RL algorithms, made slight changes to the formula order to adapt to GRPO:
of gt) RVR aal, Rot clip( mean N len(as)
-1,1)
len(a:)
2 o
mean mean|len( [len( Acor)
len(a:)
if AVR otherwise,
RORPO-ER(G gt) = RyR + RER, RER
std|len/ Acer)
+ eps
(10)
0
where (xx) 1te== is the Sigmoid function, and A fala aI...N A R, a, refers to the set of all correct answers.
We also provide the pscudocode for calculating Rot and RER for reference.
(length: List [int], correctness: [bool]):
avg - statistics. men.iicss.eennnn.mmmm (length) (Lengg)))
ol_scores
for len in langth:
x (avg len) len
(min 1), -1)
seppend zeturn ol_scores
10
R_ER(length: [intl, ecrrectness: List (baol)))
12
If corractneas.comm (True) <=
13
zeturn [0] len
14
cor_lens [Len for len, correct in (length,,,, cosrectness) if correct]
15
avg = meriittiii.menn/coooleee.
std statietiee..sdevlcoe..e..
17
er_scores = [] for len, cor ziptlength, correctneas]:
18
19
if cor:
20
x (avg lent / s:d
(1 math.exp (xp(=x))
22
er er_scores
24
er apw.ad
25
return er ar_acares
B.3 TRAINING SETTINGS
Table 9; Training configs for PPO and GRPO.
Models trained 4 NVIDIA H100 80GB GPUs. For PPO and GRPO, we list all train- ing hyperparameters in Table 9. PPO and SA- PPO use the same config; GRPO, GRPO-O1, GRPO-ER and SA-GRPO also the same config; the only exception is a in the reward function. The default for GRPO-O1 and GRPO-ER are 0.05, and 0.2 for SA-GRPO. Us- ing a different a will make these algorithms fo- more accuracy efficiency, shown in Figure 3. We use vLLM (Kwon et al., 2023) framework for inference. Models are evalu- ated using NVIDIA G100 80GB GPU. By default, the maximum response length is 4096, and greedy decoding is used (T 0).
Hyperparameter
PPO
GRPO
Actor learning rate
le
Critic learning
2e
6
train_hatch "ain_hatch_size"
128
mini_batch.se
micro batch size
16
Training step
Max response length
4096
Num rollouts
Rollout temp (7)
1.0
KL penalty (.)
1e -
Advantage clip (c)
0.2
The three batch sizes in the table are hyperparameters the VERL framework. wwiinbatch_siie is the batch size for sampling rollouts, binn_batchhstte is the batch size to perform policy updates, and bitcoo.btch_siie the batch size for rollouts and back-propagation. Theoretically, result, set smaller larger value depending the compute resources.
won't 't affect the training
16

---

Published as a conference paper at ICLR 2026
C LLM ANNOTATION PROMPTS
Below is the prompt categorize responses to 4 types in Section 4:
System Prompt:
You are an expert reasoner and LLM judge. Given a reasoning problem and an answer, you need to category the answer into one of the following categories:
1: Correct and concise answer. Small mistakes
acceptable.
Correct answer, but a lengthy, or contains unnecessary steps. Small mistakes are acceptable.
3: Partly correct answers, which makes some mistake and fails to reach the final ground truth.
4: Completely wrong or irrelevant answers, indicating the model doesn't understand the problem.
0: The answer is correct but contains extra content after the answer, like random characters or talking about an irrelevant topic.
#*Output #ormat:*
[thought] Provide your thought process how you identify the reasoning Sitaviorss(/thoughhy [answer] One single number, indicating the type the answer. [/answer]
The one-shot example:
Query: Making a cake requires I cup of flour, 1 cup of sugar, and 2 eggs. Suppose have 2 cups of flour, 3 cups of sugar, and 6 eggs, how many cakes can you make?
Ground truth answer: 2
Answer: shall first determine how many cakes each ingredient can support. Flour allows 2 cakes (2/1=2), sugar allows (3/1=3), and eggs allow 3 (6/2=3). Therefore, can make at most 3 cakes. Wait second, the maximum number actually depends on the the low-resource ingredient. In this it's flour, which only supports 2 cakes. The final answer is 2.
[thought] The answer correctly understands the problem and provides a concise solution. It identifics the limiting ingredient (flour) and calculates the maximum number of cakes that can be made based on that. The reasoning is clear and follows logical sequence. [/thought] [answer] [/answer]
Below is the prompt to identify reasoning behaviors in Section 6.2:
System Prompt: You are an expert reasoner LLM judge. Given a reasoning problem and an an- swer, you need to identify the reasoning behaviors exhibited in the solution. There are four reasoning behaviors that requires identification:
1. *VVerification*. This behavior involves systematically checking intermediate results assump- tions to ensure they are correct.
Example: "Let's verify this result by checking the two expressions are always equal."
2. #*Backtracking*** This hehavior occurs when the model explicitly revises its approach after de-
tecting enror realizing that the current path won't 't lead to the solution.
Example: "The assumption that 0 doesn't work, we need try something else."
**Subgoal S*: This behavior involves breaking down a complex problem into smaller, more manageable steps.
Example: "The first step is to find the range of and respectively."
4. **Enumeration Solving problems by exhaustively considering multiple cases or possibilities.
Example: " After investigating days of a week, only Wedensday and Friday satisfies the conditon."
**Guidelines for Identification:
It is possible for single solution to exhibit multiple behaviors or none of them.
Your annotation should based on the reasoning process, just the final answer.
You should only count a behavior if concretely contributes the reasoning process.
**Output *ormat:**
[thought] Provide your thought process how you identify the reasoning behaviors. [/thought] [answer] Behavior(s) separated by commas, "None" if no behavior is identified, [/answer]
17

---

Published as a conference paper at ICLR 2026
D ADDITIONAL ANALYSIS AND THEORY
D.1 GRPO GRADIENT ANALYSIS
To understand how GRPO and our SA-GRPO approach optimize model parameters, we analyze the gradient computation. The gradient of the GRPO objective is:
VOJGRPO
Vo BDwD (ar) moo (ar (1
W (0) A
(11)
la.
rij Vo log To (0rs ggaies))
E DD 1
A
(12)
N
lasl
j=1
where
To(ais
9. (g,r))
T
(r.((0) Tig.
Te (ii
9 ((,)))))
This gradient formulation reveals that GRPO updates model parameters by adjusting the likelihood of generating each token in the rollout responses, weighted by the importance sampling ratio rı. and the normalized advantage A; The advantage is calculated based solely the verifiable reward or other explicit reward functions, without considering the intrinsic quality or relevance of responses.
In standard GRPO, the advantage value A; is the same for all tokens within single answer, com- puted as:
A, R(g, a.) mean| [(g, aI...n)l
(13)
For correct answers, the advantage is positive, pushing the model to increase the probability of generating such answers. For incorrect answers, the advantage is negative, steering the model away from these outputs.
However, this approach treats all correct answers equally, regardless of their efficiency, relevance, or quality of reasoning. Similarly, gives all incorrect answers the same negative feedback, missing opportunities to reinforce partially correct reasoning paths or penalize completely irrelevant outputs differently.
D.2 THEORETICAL ANALYSIS OF SAR
D.2.1 SELF-ALIGNED REWARD FUNCTION
The key innovation, self-aligned reward RSA, can be derived from the perplexity measures:
RSA max (ppl(a) ppl(ala) -1)
(14)
ppl(a)
(pp((al)) ppl(a)
min
(15)
Expanding with the definitions of perplexity:
e T-7 Σ/sl log Pog 2
RSA = min
(16)
[2] 2n log(P (aa
M) E/ P )1)) log( (og
min
(17)
Pla 14.1,
min
Ing
- ) ,2
(18)
18

---

Published as a conference paper at ICLR 2026
Defining the token-level contribution v(a,) = log Pla.la.at....))) Playlar...
we get:
(e E! m((s),
RSA min
(19)
This formulation reveals that RSA measures the geometric mean of the likelihood ratio between generating tokens conditioned on the question versus generating them without the question context. Each token O, contributes v(a,) to the overall reward.
A positive v(a;) indicates that token a, is more likely be generated when conditioned on the question, suggesting that it leverages information from the query. Conversely, a negative v(a,) sug- gests that the token is less likely when conditioned the question, indicating potential irrelevance or redundancy.
D.2.2 SA-GRPO OPTIMIZATION TRAJECTORIES
The combined reward RSA GRFolg, ai, gt) = RvR + aRRA leads to modified advantage calculation:
A.A. GRAG (RvR(g, a, gt) + aaRss(q..)) mean[(Avv[g o....N.ght
gt) ) + a aRsA (g, a1...))) (20)
During optimization, SA-GRPO's gradient updates follow three key paths:
1. Correctness Optimization: Through RvR. SA-GRPO increases the likelihood of generating outputs that contain correct answers, similar to standard GRPO.
2. Self- Delf-Alignment Optimization: Through RSA. SA-GRPO encourages: Higher probability for tokens that effectively utilize question information (positive v(a;)) Lower probability for tokens that are redundant irrelevant (negative v(a,)) Concise reasoning by penalizing unnecessary rep- etition, since repeated information yields low va,) values as it becomes predictable from previous tokens
Memorization Penalty: SAR naturally penalizes memorized answers. If an answer is mem- orized, (a)(a) will be low due to the model's strong prior the memorized sequence, while the relative drop in perplexity when conditioned on the question would be minimal, resulting in a low or negative RSSA.
These optimization paths allow SA-GRPO to simultaneously improve accuracy and efficiency with- requiring separate reward models human preference data. The hyperparameter controls the balance between correctness and self-alignment objectives, enabling flexible tuning for different accuracy-efffiienncy trade-off's As training progresses, we observe from fig.
1. Early Training Phase: During early iterations, SA-GRPO primarily optimizes for correctness,
as RVR provides the strongest gradient signal. This establishes foundation of accurate reasoning.
2. Mid Training Phase: Once the model achieves reasonable accuracy, the self- alignmenttt reward RsA becomes more influential. The optimization begins focus on improving the efficiency of correct responses by: Removing tokens with low v(az) values (those that don't 't effectively leverage question information) Preserving tokens with high o(a,) values (those that directly address the question)
Late Training Phase: the later stages, SA-GRPO fine-tunes the balance between accuracy and efficiency. The combined reward creates a Pareto frontier where further improvements in efficiency come at diminishing costs to accuracy.
Unlike ffficiency-focused methods like Ol-pruner or Efficient Reasoner, which directly reward shorter responses regardless content quality, SA-GRPO's optimization content-aware. se- lectively preserves tokens that contribute meaningful information relative the question, while removing those that don't. In contrast, length- based rewards may inadvertently remove important reasoning steps if they blindly optimize for shorter responses.
The theoretical convergence of SA-GRPO can be expressed as finding the optimal policy z that maximizes:
19

---

Published as a conference paper at ICLR 2026
Table 10: SAR results on vision-language models.
GSM8K
GSM-sym
MATH
NuminaMath
AIME
Average
Setting
acc
len
acc
len
acc
len
acc
len
acc
len
acc
len
Qwen2-2B-VI.
14.33
1573
5.98
1873
14.94
15.84
3.65
2411
0.54
2404
7.89
1969.0
+GRPO
60.96
150
27.72
209
25.25
321
9.19
512
0.86
561
24.80
350.6
+GRPO-O1
58.38
115
25.44
164
23.20
79
9.99
22
0.32
17
23.47
79.4
+GRPO-ER
58.30
122
26.52
179
23.73
212
8.32
330
0.43
383
23.46
245.2
+SA-GRPO
59.59
121
27.66
174
21.22
84
9.52
56
1.29
57
23.86
98.4
G4mma3-4B
74.75
264
57.40
373
68.33
837
35.81
1370
18.76
1843
51.01
937,4
+GRPO
89.01
409
78.68
644
76.54
1177
41.9
2012
18.01
2357
60.83
1355.8
+GRPO-O1
87.87
292
75.02
464
76.67
834
41.68
1435
18.54
1814
59.95
967.8
+GRPO-ER
88.17
338
73.36
623
75.73
979
41.94
1817
17.36
2268
59.31
1205.0
+SA-GRPO
89.16
407
79.04
654
75.55
1130
40.48
1920
18.11
2379
60.47
1298.0
(ppl(a)-ppiaa ppl(alg) -1)
T6 arg max Sa-DDanmp(ll)) RvR(q, a, gt) + max
(21)
ppl(a)
At this optimal policy, each generated token in the response contributes maximally to either obtain- ing the correct answer or efficiently utilizing information from the question, with minimal redun- dancy or irrelevance.
In practice, this theoretical optimum must balance against the KL divergence penalty that prevents the model from straying too far from the reference policy, ensuring that the learned improvements remain grounded in the model's original capabilities.
E ADDITIONAL EXPERIMENTS
E. TRAINING TRAJECTORIES
In this section, show the training trajectories of GRPO, SA-GRPO and SA-GRPO without verifi- able reward (a setting discussed in Section 1). We report the verifiable reward, self-aligned reward, and average response length at each step.
1200
0.8
Correctnesss Reward
Self-aligned Reward
0.3
000
0.6
CRPO-SA wijn GRPO
0.25
Length
GRPO-SA
0.4
400
0.15
200
0.2
Step
Step
Step
(a) Verifiable reward.
(b) Self- allg-aligned reward".
(c) Average answer length.
Figure 4: Training plots for Qwen3-4B.
E.2 SELF-ALIGNED REWARD ON VISION LANGUAGE MODELS
In this section, we extend SA-GRPO to vision language models.
While Rea isn't used in training the GRPO model, still calculate and record the values for comparison.
20

---

Published as a conference paper at ICLR 2026
Table 10 summarizes the performance of Qwen2-2B-VV and Gemma3-4B across five reasoning benchmarks under different optimization strategies. Consistent with prior findings, SA-GRPO yields strong gains for purely text-based reasoning- -mstt notably in GSM8K, MATH. and momina-whhere its self-aligned reward mechanism effectively tailors outputs to the specific query. For Gemma3- 4B, SA-GRPO achieves the highest average accuracy (72.64%) exceeding the best GRPO variant by over one percentage point, while maintaining competitive output lengths. These improvements suggest that SA-GRPO's reward shaping promotes concise, context-aware reasoning paths in textual domains.
However, the results also reveal a notable limitation: SA-GRPO's advantages do transfer as strongly to vision-language models such Qwen2-2B-V VL. While modest gains are observed in certain benchmarks, its performance is less consistent compared to GRPO, particularly in tasks like GSM-sym and AIME where visual interpretation is coupled with reasoning. We hypothesize that SA-GRPO's self-alignment mechanism, optimized for semantic relevancy in text, is less effective when the query interpretation depends heavily on multimodal fusion. In such cases, reward signals based solely on linguistic alignment may fail to capture errors introduced in the visual grounding stage, leading to weaker overall gains.
These findings suggest that while SA-GRPO well-suited for text-centric reasoning, its application to multimodal settings may require integrating visual- aware reward components. Future work could explore hybrid reward functions that jointly evaluate semantic correctness and perceptual grounding, enabling SA-GRPO to extend its benefits vision-language reasoning tasks.
E.3
EFFECT OF REMOVING THE KL COEFFICIENT
Table 11: Effect of removing the KL penalty. The base model is Qwen3-1.7B in this experiment.
GSM8k
GSM-sym
MATH
MutinaMath
AIME
Average
Setting
acc
len
acc
len
acc
len
acc
len
acc
len
acc
len
Base
69,22
281
46.76
365
56.89
700
24.58
1242
9.97
1539
41.48
825.4
GRPO
84,53
335
67.66
413
67.31
697
33.44
1091
13.93
1278
53.37
762.8
GRPO (no KL)
84.69
341
69.72
430
68.31
685
33.55
1121
13.61
1320
53.97
779.0
GRPO-01
84.15
285
65.78
363
66.51
528
33.15
775
13.61
910
52.64
572.2
GRPO-O1 (no KL)
83.93
275
66.40
356
67.13
578
32.46
875
14.15
1086
52.82
634
SA-GRPO
85.51
267
67.66
346
67.96
564
36.03
841
13.50
992
54.13
602.0
SA-GRPO (no KL)
85.75
246
67.62
322
66.73
511
34.06
788
12.43
933
53.31
560
In Equation (3), the KL. penalty term is used to control the divergence between the reference model and the actor policy, which is critical to preserving the base model's capabilities. We investigate the effect of removing the KL Coefficient (i.e. setting 3 in Equation (3)) in this section.
From Table I., observe that for all RL algorithms, the performance gap between using and not using KL is relatively small. This indicates that training remains stable and does not diverge signifi- cantly from the base model. It also suggests that the self-aligned signal is sufficiently consistent and well-aligned to guide learning even without strong distributional anchoring.
Notably, removing the KL penalty produces opposite effects for GRPO/GRPO-O1 and SA-GRPO. For GRPO and GRPO-O1, it increases accuracy but reduces efficiency, while for SA-GRPO, decreases accuracy but improves efficiency. Since the verifiable reward RvR directly linked to accuracy and the self-aligned reward RSA is not, this pattern suggests that Rvg dominates in GRPO-
Ol (reward- = RvR + a.a.), whereas the self-aligned reward RiA dominates in SA-GRPO (reward RvR + a Rts.)) Across all variants, there's a consistent trade-off between accuracy and efficiency: moving toward higher accuracy tends to reduce efficiency, and vice versa.
21

---

Published as a conference paper at ICLR 2026
F CASES
F.1 DIFFERENT ANSWER TYPES AND ADVANTAGE VALUES
Table 12: An example for different types of answers and their advantage values.
Query: Janet's ducks lay eggs per day. She three breakfast every moming and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
Answer Type and Example
Length
AGRPO
AGRPO-a1
AGRFOER
GRPO (Ours)
We break this down step by step: Janet's ducks lay eggs per day. She eats eggs for breakfast every morning, So, 16 13 eggs remain after Janet cats breakfast. She bakes 4 eggs in muffins every day for her friends. So, need to subtract these well: 4 eggs. Hence, Janet sells eggs to farmers' market daily. sells each egg at $2. Therefore Janet's daily revenue is 9 eggs $2 $18. So. Janet makes $18 per day the farmers' market.
(I) Correct and Concise:
143
0.5
0.64
0.63
0.99
(2) Correct but redundant: answer the question, we need to first calculate how many eggs Janet uses for breakfast and for baking daily. Janet eats 3 eggs for breakfast every morning. Additionally, she bakes muffins with eggs daily. Let's sum the total number eggs used for both breakfast and baking: 3 (breakfast) (baking) 7 eggs total used daily. Now that we know she uses 7 eggs daily, we can determine how many fresh eggs she has left after accounting for her usage. Since ducks lay eggs per day, we subtract her total egg usage from the daily egg production:
236
0.5
0.29
-0.29
0.84
(total eggs
produced)- 7 (total eggs used) 9 fresh eggs remaining. Finally, find out how much money Janet makes selling the fresh eggs, multiply the number of remaining fresh eggs by the price per egg. She makes $2 per fresh duck egg: 9 (fresh eggs left) x S2 (price per egg) $18 daily revenue. Therefore, Janet makes $18 day at the farmers' market, selling the leftover fresh eggs from her ducks.
Janet's ducks lay 16 eggs per day. To determine how many fresh eggs she sells each day, need subtract the number of eggs she eats for breakfast and number she uses
(3) Partly Correct, Wrong Answer:
457
-0.5
-1.5
-0.83
-0.16
muffins: Fresh eggs daily eggs per day eggs hreakfast 4 eggs for muffins = 16- 3-4 eggs per day So, Janet sells 2 fresh duck eggs daily at the farmers' market. Since she sells each egg for $2, her daily carnings
be calculated by multiplying the number fresh ducks by their price: Daily carnings Number of eggs sold x Price per egg 2 eggs x $2 per egg $4 Therefore, Janet makes $4 every day the farmers' market... (Truncated)
(4) Irrelevant or Completely Wrong: is never 30 "C 256 in winter. On the shortest day, in the Aretic Circle, the low is 117 9C. On the longest day, the low is 17 "C transform this
-0.5
-0.77
-0.83
-0.29
Thus, 10 'C is the average low. We
into simpler version. Janet's ducks lay 16 eggs per day, eat 3, use 4. Janet makes 13 eggs. She sells them for $0.05 each: 13 0.05 50.65... (Truncated)
(5) Correct, but no reasoning (likely memorized): The answer 18.
-0.5
1.17
1.17
-0.16
(6) Incorrect, and reasoning: The answer is 0.
5
-0.5
0.17
0.17
-1.20
22

---

Published as a conference paper at ICLR 2026
F.2 COMPARISON OF GRPO AND SA-GRPO
F.2.1 CASE 1: IMPROVED CONCISENESS
Question: Let 8, and be solutions to the equation
2n 3x Ax = 0.
Find y² s² + t..
7
Ground truth:
4
Qwen3-1.7B GRPO Answer:
To find y² + 8² + where r, 8, and l are the solutions to the equation 2x 3x. + 4r = 0, we
can use the relationships between the roots and the coefficients of the polynomial. Specifically, we can use the identity:
- 5
+ + t)² 2(rs + rt st)
First, we need to find the values of r + + t, rs + rt + st, and rst using the coefficients of the polynomial.
The general form of cubic polynomial is ar + bur" + car + O. For our polynomial 2x 3xx +4r - 0, have:
2, b b=-3, 4, d=
Using Vieta's formulas, we know:
r
2
e 2,
78 +
2
-1
rst
2
Now, we can substitute these values into the identity for ,2 2 8:
2 2 3 + 12 (r + + 1)2 2(rs + rl + s6)
Substituting the values we found:
3 2 2
1² + 8² + 8
2
Calculating each term:
3
2
2
4'
So,
2
2 +
2
To subtract these, we need common denominator:
16
4
Thus,
23

---

Published as a conference paper at ICLR 2026
2
2
16
16
4
4
Therefore, the answer is:
Qwen3-1.7B + SA-GRPO Answer:
To solve this problem, sums and products of its For the polynomial
use Vieta's formulas, which relate the coefficients of the polynomial to
2x
3x
+ 4x -1 0,
the roots are 8, and f. Vieta's formulas tell
The sum of the roots +
The sum of the product of roots taken two time ra + rt + at = =
The product of the roots rst
2
need find y² + s2 f2. We can use the identity:
x 2+s + (r + 1)² 2(rs + rl st).
Substituting the values from Vieta's formulas:
(3) (
2 + 12
2(2).
2
9
8
4
2
2 8
9 16
9 9- 16
3 1+3
-7
Therefore, the answer is
4
F.2.2 CASE 2: IMPROVED CONCISENESS AND ACCURACY
Question:
How many integers satisfy
+ and 1x - 3) >
Ground truth: 2
Qwen3-1.7B + GRPO Answer:
To solve the problem, we need to find the integers x that satisfy both incqualities:
|.|z| + 5 < 7
2. la
3 > 2
Let's solve each inequality step by step.
Step Solve the first incquality | <
24

---

Published as a conference paper at ICLR 2026
Subtract from both sides:
|x| < 2
This absolute value incquality can be rewritten
2 2<
So, the integers that satisfy this incquality are:
1,0,1
Step Solve the second inequality
2
This absolute value inequality can be split into two separate inequalities:
--3>2 or x - 3 3<
Solving these inequalities:
x > or << 1
So, the integers that satisfy this inequality
2, -1 2,
Step Find the intersection of the solutions from both inequalities
need to find the integers that satisfy both - x < 2 and x > Jor <
From the first inequality, the integers are:
マ= -
From the second incquality, the integers are:
-2, 3,,
The intersection of these two sets of integers is:
1,0,11
Step 4: Count the number of integers in the intersection
The integers that satisfy both incqualitics are:
-1,0, 0,
There are 3 integers in this set.
Therefore, the number of integers æ that satisfy both inequalities is:
Qwen3-1.7B + SA-GRPO Answer:
To solve the problem, we need to analyze the two inequalities separately and then find the intersection of the solutions.
Step Solve the first inequality |ar] +
Subtract from both sides:
|x| 2
This means that must be within units of
2 < x <2
Step 2: Solve the second inequality
2
25

---

Published as a conference paper at ICLR 2026
This inequality be broken into two separate inequalities:
x 3-3>2 2 or x --33 -2
Solving these, get:
x> I<1
Step 3: Find the intersection of the solutions
We need to find the values of that satisfy both inequalities:
-2
x > or x<1
The intersection of these intervals is:
-2<06 <1
Step 4: Identify the integers in the interval <
The integers in this interval are:
-1,
Step 5: Count the integers
There are 2 integers that satisfy both inequalities.
Therefore, the number of integers æ that satisfy the given conditions is:
26


Reviews:
[{"review_id": "o9v3PM1Imq", "reviewer": "Reviewer_oz4c", "summary": "The paper aims to improve the conciseness of LLM outputs without sacrificing correctness. It proposes self-aligned reward (SAR), which considers the difference between the perplexity of answers when conditioned vs. not conditioned on the query. In contrast, existing approaches penalize the length of answers without concern for relevance to the query. SAR provides a Pareto improvement over baselines, improving both accuracy and conciseness.", "questions": "How do (5) in Section 4.1 “simulate memorization”?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has a clear motivation, improving conciseness of LLM answers, and provides a simple intuitive solution, expressed as a reward function in (6).\n\nExperiments are very comprehensive, considering a variety of models, datasets, and metrics. I especially appreciated the plots in Figure 3, showing performance gain vs. length reduction. It supports the main argument of the paper well: SAR results in the biggest performance gain across all length reduction levels.\n\nBesides the gains in efficiency, the example in Section 4.2 illustrates that SAR could even be useful as an interpretability technique. It shows how SAR can be used to rank the relevancy of each output token to a given query.", "weaknesses": "In Section 4.1, evaluating different types of answers using different reward functions to see what SAR captures that alternative reward functions don’t is a great idea. The main difference seems to be that SAR is the only reward function that penalizes answers of type “Correct but no though”. However, how this type of answers are generated is not explained well in the text (“artificially synthesized to simulate memorization” is not a sufficient explanation). Without clarity on what this answer type stands for, the results in Section 4.1 are not helpful.\n\nColumns in Table 1, “Continuous”, “Internal”, “Content-Aware”, “Correctness”, and “Conciseness”, are not defined. It is ambiguous what criteria is meant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "don51I6JRb", "reviewer": "Reviewer_SNsy", "summary": "To address the overthinking or repetitive reasoning in large language models, this work proposes a self-aligned reward combined with a verifiable reward to enhance the accuracy and efficiency of the mathematical reasoning capabilities of LLMs. The self-aligned reward $R_{SA}$ is based on the perplexity drop from the answer's stand-alone perplexity to the perplexity when the answer is conditioned on the query $ppl(a) - ppl(a|q)$. In other words, if the answer is highly dependent on the query (not generic tokens), then the difference will be high, and the model can receive a larger self-aligned reward.\n\nThe proposed reward is applied with PPO and GRPO for 4 different LLMs across 5 mathematical reasoning datasets and 2 logical reasoning datasets. Detailed ablation studies are also conducted, indicating the importance of applying both the verifiable and self-aligned reward at the same time, and the effectiveness of the perplexity-based reward can reduce the output sequence length.", "questions": "* Section 6.3 (Training cost of self-aligned reward) could move to the appendix. On the other hand, some interesting contents in the Appendix, e.g. the analysis of the self-aligned reward function in Appendix D.2.1 or experiments of vLLMs in Appendix E.2, could move to the main text.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed reward function is well motivated. Measuring the difference between the stand-alone perplexity of an answer and its perplexity conditioned on the query provides a reasonable approach to capturing trivial/generic reasoning steps. Combining the verifiable (external) reward with the intrinsic reward signal effectively optimises the model for both accuracy and efficiency.\n* The proposed reward signal is integrated with PPO and GRPO to optimise four different LLMs (two Qwen variants, Phi-3.5, and Gemma3), and is evaluated across seven diverse datasets covering mathematical and logical reasoning tasks.", "weaknesses": "* The idea of combining a verifiable reward with a length penalty is well aligned with recent developments in reinforcement learning for reasoning optimisation. However, this topic has been widely studied in prior work as well. For example, T1 [1] integrates correctness with a rule-based negative reward for undesirable reasoning behaviours, ThinkPrune [2] introduces an iterative pruning strategy to encourage concise reasoning, and Self-Braking Tuning [3] employs a frequency-based masking approach to mitigate overthinking during supervised training. A more detailed comparison with these approaches, particularly T1, would substantially strengthen the paper’s contribution and clarify how the proposed method advances beyond existing studies.\n* A statistical significance test is missing from the experimental results. Some reported improvements appear marginal, for instance, in Table 3, the average accuracy of GRPO and SA-GRPO is 52.49 and 54.13, respectively. Without a significance test, it is difficult to determine whether these improvements are statistically meaningful. Including such tests would enhance the reliability of the evaluation.\n* The Entropy method [4], which is listed as one of the main related works in Table 1, is only included in the ablation study (Section 6.1 and Table 5) but not in the main comparative results (Sections 5 and 6.2). Given its conceptual relevance, excluding Entropy from the main comparison raises concerns about fairness, as the proposed method is otherwise compared primarily against weaker length-penalty baselines. A more comprehensive comparison would provide a clearer picture of the model’s relative performance.\n* The authors claim that the intrinsic reward signal is less prone to reward hacking compared to external signals (L067–068). However, this claim is not sufficiently supported by experimental evidence. In fact, the results suggest the opposite: the accuracy of the model optimised with the intrinsic reward ($R_{SA}$) drops from 69.07 to 20.96 on average (Table 5), indicating that the model may be exploiting or “hacking this reward signal. An additional experiment or analysis demonstrating the robustness of the intrinsic reward would be necessary to substantiate the authors’ argument.\n\n[1] T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (Hou et al., ICML 2025)\n\n[2] ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning (Hou et al., arXiv 2025)\n\n[3] Self-Braking Tuning: Let LRMs Break Free from Overthinking via Self-Braking Tuning (Zhao et al., NeurIPS 2025)\n\n[4] Entropy: The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning (Agarwal et al., arXiv 2025)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KfgHzwoFGN", "reviewer": "Reviewer_vsNW", "summary": "The paper introduces \"self-aligned reward\", the difference in perplexity on the generated solution with and without the query in-context, as an additional reward term to binary verifiable rewards in order to incentivise concise, query-specific reasoning. The authors motivate SAR through a case study that connects the GRPO advantages assigned to responses with various response types and that analyses the token-level contributions to the overall SAR term. The paper demonstrates that using SAR for training with PPO or GRPO leads to a Pareto improvement in token efficiency and accuracy. An ablation study on the reward terms, including a comparison to entropy regularisation, demonstrates the importance of the complete proposed reward combination. Additional analysis on qualitative reasoning behaviours and training costs are used to further illustrate the impact of SAR.", "questions": "Do you have an intuition for why SA-GRPO leads to considerably more backtracking behaviour on NuminaMATH and AIME?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper effectively uses a case study to motivate the design decisions behind SAR, which provides an intuition for the subsequent results. The method itself convincingly improves on the length minimisation baselines in striking a balance between accuracy and token efficiency. The analysis section features appropriate ablations and the reasoning behaviour frequencies provide further evidence that reasoning traces from models without SAR or length penalties are bloated with subsequences that do not correspond core reasoning behaviours. The paper is well-written and results are presented clearly.", "weaknesses": "Using GPT-4o to annotate responses without checking the quality of these annotations, such as by comparing to gold-standard human annotations on a subset, means that the corresponding analysis sections cannot be taken to be fully reliable. \n\nIt would be useful to show AES for a few different values of gamma. This would compliment the following section on accuracy-efficiency tradeoff nicely.\n\nWhilst it's intuitive that entropy regularisation in the reward can lead to overconfidence and entropy collapse, explicitly verifying that using delta perplexity (SAR) mitigates this would support this claim, which is currently based only on accuracy differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bwhcoczMTF", "reviewer": "Reviewer_F8KC", "summary": "The paper proposes Self Aligned Reward, SAR, an internal signal that measures the relative perplexity drop between an answer alone and the same answer conditioned on the query. SAR is added to a standard verifiable reward within PPO or GRPO. The goal is to reward answers that are concise and tailored to the query, reducing redundant reasoning while preserving or improving accuracy. Experiments across four small models and seven math and logic benchmarks report about four percent accuracy gains with roughly thirty percent shorter outputs, and a favorable accuracy versus efficiency frontier compared with length penalties such as O1 and ER.", "questions": "How sensitive are results to prompt templates that require explicit step by step derivations versus direct short answers? Does SAR over penalize long but necessary derivations when the verifier still rewards only final correctness? Please report token reduction broken down by reasoning behaviors. \n\nCan SAR be computed with teacher forcing on partial prefixes during rollout to provide denser token level rewards, rather than only sequence level returns. This might make the signal less noisy without a learned critic.\n\nWhat happens if the query contains spurious details? Since SAR rewards dependence on q, models might over attend to red herrings. Add a study with injected distractors.\n\nHow do you prevent echoing of the query to raise RSA? Show an explicit penalty or a post hoc filter and quantify its effect.\n\nDo results hold when the base model is much stronger? Most results are for one to four billion parameters. Please include at least a mid size base to test scaling.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- SAR is a small change to existing RL loops and uses only log probabilities already computed, so the added cost is minimal. The paper even reports slightly faster rollouts due to shorter generations.\n-  The token level decomposition v(a_j) is a nice lens that explains why early, query specific tokens get rewarded, connecting SAR to information use rather than length alone. \n-  Across Qwen3 1.7B and 4B, Phi 3.5 mini, and Gemma3 1B, SAR variants reduce tokens and match or beat accuracy. The Pareto plots make the trade off story visually clear.", "weaknesses": "- RSA rises when an answer looks unlikely without the query. This can be increased by echoing or paraphrasing query tokens or by using rare surface forms rather than better reasoning. The qualitative analysis shows discouraging memorized short answers, but an explicit adversarial check would help. Consider adding tests where the model repeats entity strings or injects rare tokens to inflate ppl(a) relative to ppl(a|q).\n- GRPO and PPO commonly use a KL term to a reference policy. SAR implicitly prefers answers that are improbable unconditioned on q, which could tug against or amplify KL. The paper should analyze this interaction and show the same KL settings are used across baselines.\n- SAR uses the current policy to score both ppl(a) and ppl(a|q). If calibration shifts during training, the reward scale can drift, and the clipping in Equation 6 may hide pathologies. A schedule or normalization across batches might be needed. Please report the distribution of RSA over training and a sensitivity study to the clip range.\n- Length penalized GRPO variants often trade accuracy for brevity, yet there are recent methods that make brevity adaptive to difficulty. The paper compares O1 and ER, but please confirm both got per model alpha sweeps as wide as those used for SAR and include best in class brevity methods with per instance stopping rules where applicable. The Pareto plots for SAR vary alpha closely. Ensure comparable sweeps for others", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)