Mirage or Method? How Model–Task Alignment Induces Divergent RL Conclusions
Authors: Haoze Wu
Cheng Wang
Wenshuo Zhao
Junxian He
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
reasoning
Primary area: reinforcement learning
TL;DR: “RL shortcuts” for LLMs (e.g., one example, noisy/no rewards, negative samples) only work under strong model-task alignment.

Abstract: Recent advances in applying reinforcement learning (RL) to large language models (LLMs) have led to substantial progress. In particular, a series of remarkable yet often counterintuitive phenomena have been reported in LLMs, exhibiting patterns not typically observed in traditional RL settings. For example, notable claims include that a single training example can match the performance achieved with an entire dataset, that the reward signal does not need to be very accurate, and that training solely with negative samples can match or even surpass sophisticated reward-based methods. However, the precise conditions under which these observations hold—and, critically, when they fail—remain unclear. In this work, we identify a key factor that differentiates RL observations: whether the pretrained model already exhibits strong *Model-Task Alignment*, as measured by pass@k accuracy on the evaluated task. Through a systematic and comprehensive examination of a series of counterintuitive claims, supported by rigorous experimental validation across different model architectures and task domains, our findings show that while standard RL training remains consistently robust across settings, many of these counterintuitive results arise only when the model and task already exhibit strong model-task alignment. In contrast, these techniques fail to drive substantial learning in more challenging regimes, where standard RL methods remain effective.

Published as a conference paper at ICLR 2026
MIRAGE OR METHOD? How MODEL-TASK ALIGN- MENT INDUCES DIVERGENT RL CONCLUSIONS
Haoze Wu'n Cheng Wang2* Wenshuo Zhao' Junxian He
*Zhejiang University National University of Singapore *HKUST
waithz@zuaa. zju. edu. on
wangchengeu.
nww.eennneuuuuueeeoeee... nus. edu junxianhêcse ust .kk
ABSTRACT
Recent advances in applying reinforcement learning (RL) large language models (LLMs) have led to substantial progress. In particular, series of remarkable yet of- ten counterintuitive phenomena have been reported LLMs, exhibiting patterns not typically observed in traditional RL settings. For example, notable claims include that a single training example can match the performance achieved with an entire dataset, that the reward signal does not need to be very accurate, and that training solely with negative samples can match or even surpass sophisticated reward- based methods. However, the precise conditions under which these observations hold-and, critically, when they fail-remain unclear. In this work, we identify a key factor that differentiates RL observations: whether the pretrained model already exhibits strong Model- Task Alignment, as measured by pass @ accuracy on the evaluated task. Through a systematic and comprehensive examination of series of counterintuitive claims, supported by rigorous experimental validation across dif- ferent model architectures and task domains, our findings show that while standard RL training remains consistently robust across settings, many of these counterintu- itive results arise only when the model and task already exhibit strong model-task alignment. In contrast, these techniques fail to drive substantial learning in more challenging regimes, where standard RL methods remain effective. Code is avail- able at https: http:::://gtttbb com www.t mww//mommeleee wwweellttssallaaaagee
Strong Model-Task Alignment
Weak Model-Tass Alignment
X Math
Puzzle
Counterfactual
Math
Operation
Cipher
Before AL
Standard Al.
Sparious reward
Tost- Tost-Tima Time AL
One-shot Rtl.
Nagative
sample training
Batore RL.
Standard
Spurious reward
Test Time-Tmme AL
One- shat Rt.
Negative-
sample training
Figure 1: Model-task alignment, which is measured by pass @k accuracy the evaluated task, drives distinct outcomes from the same series RL approaches.
"Equal Contribution. Work done during visit HKUST.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Reinforcement Learning (RL) (Sutton al. 1998) has emerged as a transformative post-training technique for Large Language Models (LLMs), enabling them to follow instructions (Ouyang et al.
2022) and align with human preferences Ziegeer et al. 2019 Rafailov et al., 2024). particularly
prominent application focuses on enhancing reasoning capabilities, as exemplified by breakthrough
models such as OpenAI-ol (Jaech et al., 2024). DeepSeek-RI (Guo et al., 2025), QwQ (Team)
2025), and Kimi-1.5 (Team et al. 2025]. J. These systems demonstrate remarkable performance across
reasoning-intensinn domains including coding (Jain et al. (024)), mathematics (Lewkowycz et al.
2022
He et al. 2024), and logical reasoning (Liu et al. 2025 Chen et al. 2025).
While RL yields significant performance improvements in LLM reasoning-mirroriig the success of RL in traditional domains such as games (Silver et al. b we also observe several remarkable yet often counterintuitive empirical phenomena. These effects appear to be unique to LLMs and would be considered unexpected in traditional RL settings. For instance, single training examples can match or rival full-dataset training performance (Wang et al. (025), ground-truth reward may be surprisingly dispensable (Shao et al. 22025), and training with negative samples alone match sophisticated aeward-based methods (Agarwal et al. (025).
Although these findings have generated considerable enthusiasm, the precise conditions under which they hold, and when they break down, remain insufficiently explored. Given that these observations may have important implications for RL practices, it is concerning that the conclusions are largely based on limited experimental settings, where Qwen models (Qwen et al. 2025) trained on mathematical tasks dominate the landscape.
To this end, we carry out a systematic empirical investigation of several notable RL claims, supported by rigorous experimental validation across diverse model architectures and task domains. Concretely, we experiment with both Qwen and non-Qwen models on math and other tasks. Our controlled experiments reveal that model-task alignment, defined as the degree to which model capabilities match task requirements, is a critical indicator for categorizing RL observations. Specifically, models benefit from noisy rewards, test-time RL (Zuo et al., 2025), minimal training, and negative- training primarily within their domains of expertise. where these techniques fail for unfamiliar tasks even though standard RL training can succeed. Interestingly, we also observe that certain meta-patterns hold consistently across different settings. For instance, one-shot RL training is generally effective for the specific task to which the training example belongs, and eegative-sample training helps stabilize model entropy, even though it does not always lead overall improvements in accuracy.
We evaluate the "alignment" between model capabilities and task requirements using pass @k accuracy, which find be reliable indicator for distinguishing these counterintuitive RL phenomena. Our hypothesis is that strong, inherent model capabilities can be readily activated through minimal training, even when guided by incorrect reward signals, whereas unfamiliar tasks demand substantially more effort-case that we argue dominate when scaling up RL compute. Concurrent work (Wu) al., 2025) investigates the mechanism behind spurious rewards and attributes their effectiveness primarily to data leakage in Qwen models on the test set. However, our results suggest otherwise: we find that spurious rewards remain effective even in the absence of contamination, provided the model already exhibits strong alignment on the evaluated task.
Our study reveals that, unlike traditional RI. training, distinet RI. mechanisms emerge in the context of LLMs, depending on whether the pretrained model is already familiar with the target tasks. On the one hand, this suggests that RL phenomena should be interpreted with extra caution, as they may only reflect one of these two mechanisms. On the other hand, it also opens up opportunities for jointly optimizing base model pretraining (or mid-trainine -triiiininnn and RL post-training. For example, one might enhance the domain-specift capabilities of the base model during mid-training, enabling effective RL with limited training data and potentially inaccurate reward signals, or alternatively, allocate most compute resources to the RL stage using carefully curated training data and precise reward signals.
2 ON UNIQUE PHENOMENA OF RL TRAINING LLM REASONING
Reinforcement Learning from Verifiable Rewards (RLVR) has achieved significant success in im- proving language model reasoning. While similar gains in accuracy from standard training have
2

---

Published as a conference paper at ICLR 2026
also been observed in traditional RL domains such as games, we have noticed several phenomena that appear unique to LLMs and would not typically be expected in conventional settings. For example, we highlight several remarkable, and at times counterintuitive, observations below: (a) Unexpected robustness to unreliable or absent rewards: [Shao et al. (2025) demonstrate that random and incorrect reward signals can improve model performance, while Agarwal et al. (2025) show that reward-freee, catropy-minimzzatinn objectives can rival reward-based approaches. Test- Time Reinforcement Learning (TTRL) proposed by Zuo al. (2025) further reinforces this trend by generating reward signals through aggregating majority-vote outcomes, thereby guiding the model to evolve itself on the test set. Together, these suggest surprising fault tolerance RL training that challenges standard assumptions about the critical role of accurate reward signals. (b) One-shot training sufficiency: Wang et al. (2025) report that training on a single carefully selected example can match or exceed performance from full dataset training, challenging assumptions about data volume requirements. (c) Negative-only signal effectiveness: Zhu et al. (2025) demonstrate that
using exclusively negative reward signals achieves comparable results to standard RL training while maintaining beneficial entropy properties.
These findings carry significant implications. If broadly confirmed, they would necessitate shifts in resource allocation- suchioon--uheee as prioritizing data selection algorithms over dataset scale, questioning the necessity of highly accurate reward modeling, and potentially introducing new research directions. Therefore, we believe it is important to assess whether these conclusions hold in general, and if not, under what conditions they succeed or fail. Clarifying these patterns would not only help us understand the limitations of the current findings but also, in the opposite direction, reveal new opportunities for modifying models so that these findings become valid, thereby making RL training substantially easier. In this work, we will investigate these observations through controlled experiments comprehensively.
2.1 CENTRAL HYPOTHESIS: MODEL- TASK ALIGNMENT DEPENDENCY
As most of the findings discussed above are based on mathematical reasoning tasks using Qwen models (Qwen et al. 2025 Yang et al. (202)), natural question arises: do these results generalize to other settings? For instance. [Shao et al. (2025) reported that spurious rewards were ineffective with mathematical tasks. However, we argue that treating Qwen+math as merely special case is an overly superficial categorization. It remains unclear what specifically makes mwen+math unique, and what the deeper, more essential factors might be. We propose guiding hypothesis for designing and categorizing experimental settings, which we call Model-Task Alignment Dependency: the effectiveness of these unique RL. findings fundamentally depends on the degree of alignment between model's 's inherent capabilities and the requirements of the task domain. In other words, they depend on the model's ss proficiency on the evaluated task. This hypothesis may or may not hold, but we will use it as a framework to categorize experimental settings in terms of whether the model-task combination is aligned or misaligned.
Llama Meta 2024) models
Quantifying Model- Alignment with pass @ k. To systematically evaluate the degree of alignment between a model's inherent capabilities and the requirements of specific task domains, we employ the pass @ metric our primary measure of model-task proficiency. Pass @k represents the probability that at least one correct solution appears among k independent samples generated by the model for given problem. This metric effectively captures how well a model's existing knowledge and reasoning patterns align with the demands of a particular task.
Formally, for problem x; from evaluation dataset DD we generate samples (n 2 k) and count the number of correct samples as C₁ then the unbiased estimator of pass@k k over the dataset is: pass@1 @k= := Ee,DD [1 ("* (2)
2.2 STRATEGIC MODEL AND TASK SELECTION
Building on our Model-Task Alignment Dependency hypothesis outlined in Section 2.1 cally design model-task combinations that test the boundaries of current claims in RL for language model reasoning. Our experimental design is motivated by the critical need to distinguish between findings that represent universal RL properties versus those that emerge from specific model-task capability alignments. We evaluate two representative language models from different families: Owen2.5-7B-Base (Qwen al. 2025) and Ilama-3.1
we strategi-
(Meta 2024), enabling system-
3

---

Published as a conference paper at ICLR 2026
atic comparison across model architectures with varying baseline capabilities while controlling for architectural differences at comparable parameter scales.
Our evaluation encompasses mathematical and logical reasoning domains. For mathematical reason- ing, we employ AIME24 (AIME, 2024), MATH500 (Hendrycks et al., (02I) and AMC23 (AMC) 2023). For logical reasoning, we utilize SynLogic (Liu et (005) (synthetic puzzles with 35 task types, we use the validation split), BBH (Suzgun et al.) 2022) (multi-step reasoning tasks),
BBEH Kazemi al. (2025) (extended-diffcully version), and KOR-Bench (Ma et al. 2024)
(knowledge-orthooona reasoning across five categories).
AIME 2024
Puzzle (KOR- Bench subset)
Operation (KOR-Bemt subset)
Qwen demonsttrates
Both models exhibit limited inlserent logical reasoning abilities
math ematical reasoning cap abilities
0.8
0.8
0.8
0.6
0.6
0.6
Both models demonstrai alc strong inherent reasennng capubilities
0.4
0.4
0.4
0.2
0.2
Qwen2.5
5-70
0.2
Qwen2 5-70 Instruct
4 Number Samples
16 32 128 256515
4 Number Samples
16 32 128256512
16 32 128256512 Number of Samples
Figure 2: Pass @k for different tasks. Different LLMs have significantly different abilities different tasks, which will affect how the RL techniques perform across model-task combinations.
To operationalize our hypothesis, we systematically measure alignment strength using pass@k metrics across all model-task combinations. As demonstrated in Figure 2 models exhibit markedly different inherent capabilities across domains. Based on comprehensive evaluation (full results in Appendix we identify cases of strong model-task alignment, such as Qwen2.5 on mathematical domains and both models on Operation and Counterfactual subsets of KOR-Bench, as well as weak model-task alignment cases, such as Ilama3.1 on mathematical domains and both models on other logical reasoning tasks. This categorization enables us to test whether counterintuitive RL phenomena are artifacts of specific model-task alignments or represent fundamental properties of reinforcement learning in language model reasoning.
2.3 THE CONTAMINATION HYPOTHESIS
Concurrent work from Wu et al. (2025) proposed an alternative hypothesis, where they specifically focus on the spurious reward pattern and suggest that stems primarily from dataset contamination during pre-training, They further confirmed the presence of data leakage in the Qwen models on several mathematical benchmarks. While we acknowledge contamination as a valid concern, our hy- pothesis diverges by emphasizing the distinction between contamination and inherent task proficiency. In particular, models may demonstrate strong task performance without direct contamination of the test data. In what follows, we categorize different experimental settings based their contamination and inherent model-task alignment status. Later, in our experiments, we will empirically show that contamination is not the underlying cause; instead, model-task alignment serves more reliable differentiator.
To verify our hypothesis, we extend contamination analysis beyond Qwen-Math combinations. Following Wu et al. (2025), we evaluate model generation given partial prompts while preserving word boundaries (more details are provided in Appendix D). We employ greedy decoding and calculate both exact match (EM) rates and ROUGE-L scores, where ROUGE-L scores of 1.0 indicate perfect reconstruction. Table
Appendix Dishow that Operation and Counterfactual subsets have no contamination, yet both models demonstrate strong inherent reasoning capabilities with high pass @ k scores (see Appendix(C) C). As we will show in our experiments, contamination is not the necessary condition for the effectiveness of these RL phenomena. Based on our contamination analysis and pass@1 @k k measurements, we categorize experimental settings into three groups: Red (Potential Contamination Strong Model-Task Aligmment): Qwen2.5 mathematical domains. Gray (No Contamination + Weak Model-Task Alignment): Llama3. on mathematical domains; both models Synlogic, BBH, BBEH, and Logic, Cipher, Puzzle subsets of
Green (No
4

---

Published as a conference paper at ICLR 2026
AMC 23
MATH500
Puzzle
Operation
Model
Portion
ROUGE
EM
ROUGE
EM
ROUGE
EM
ROUGE
EM
0.4
63.78
23,91
50.36
8.20
19.56
0.00
21.37
0.00
Qwen2.5-7B
0.6
64.42
33.73
60.98
21.20
19.62
0.00
24.25
0.00
0.8
73.23
49.39
66.42
40.20
19.24
0.00
20.18
0.00
0.4
27.18
0.00
23.09
0.60
18.27
0.00
21.83
0.00
Llama-3.1-8B
0.6
30.64
0.00
40.56
3.80
17.31
0.00
18.34
0.00
0.8
44.54
4.81
48.33
17.8
15.85
0.00
16.75
0.00
Table 1: Contamination Analysis across model-task combinations. Portion refers to the truncation
ratio of the prompt used to test whether models can complete the remaining content. Red indicates potential contamination with strong model-task alignment; Gray indicates no contamination with weak model-task alignment; Green indicates no contamination with strong model-task alignment.
Contamination + Strong Model- Tasel-Taskkaa Alignment): Both models on Operation and Counterfactual subsets of KOR-Bench.
3 EXPERIMENTAL SETUP
Training Datasets and Evaluation. Except for the experiments on Test-Time RL (Section
we use DeepScaleR (Luo al., (2225) as the training set for mathematical tasks and the training split of SynLogic-Easy (Liu et al. 2025) for logical tasks. Evaluation datasets are as described in
Section 2.2 Following Syn oogi (Liu et al. 2025), all evaluations are conducted
zero-shot setting, with @vg@8 metrics computed for AIME 2024 and SynL.giic to mitigate variance.
Our experiments default to using the DAPO algorithm with a group size of 16. Its effectiveness has been demonstrated on math (Yu et al. 2025) and logic tasks (Liu et al. 2025). We set flow = 0.2, thigh 0.28, max prompt length = 2048, max generation length 8192. We use dynamic sampling, which is crucial for improving the reward on SynLogic, and set max_ num gen batches = 2. For logical tasks, each sampled batch often contains very few samples with non-zero reward variance. We apply
Training Configuration.
strategies: (1) if neither generated batch contains samples with non-zero reward variance, the second batch is used for training; (2) if the number of available samples is smaller than the batch size, samples are duplicated. We do not length penalty. In most experiments, we set Ir = le le- batch size 128, mini batch size 64, temperature = 1.0. We fix all key hyperparameters across experiments ensure that observed differences primarily reflect mase aligmment rather than tuning effort.
4 RQ1 REWARD SIGNAL: How CRITICAL Is IT?
This section investigates the role of reward signal quality and its impact on RL performance for LLMs. Previous work in Reinforcement Learning with Human Feedback has shown that more accurate reward models do not always lead to better downstream performance (Chen et al. 2024). In the specific context of L.LM reasoning, initial studies found that models with strong inherent reasoning abilities exhibit surprising robustness to noisy reward signals, whereas weaker models show poor noise tolerance (Lv et al. 2025 Shao al. 2025). Building on these findings, we extend the analysis by considering diverse reward signals across different model-task combinations. Hyperparameters and training runs for 300 steps (see Appendix for more details).
follow Section 3.
4.1 RESULTS
We present results in Table From the results, we identify three critical findings regarding the impact of reward signal quality on model performance (Appendix [ additional discussion):
Ground Truth Rewards Consistently Outperform All Alternatives. Across both model families and all task domains, utilizing ground truth rewards consistently yields the highest performance improvements. For instance, Qwen2.5-7B achieves substantial gains AIME24 (14.2 vs. baseline 3.3) and MATH500 (71.0 vs. baseline 40.8) when trained with accurate reward signals. This establishes ground truth rewards as the gold standard for RL training in reasoning tasks.
5

---

Published as a conference paper at ICLR 2026
Math Tasks
Logic Tasks
KOR Benchmark
AIME24
MATTIS00
AMC
SynLogic
BBII
BBRH
OP
CF
Puzzle
Logle
Cipher
Qwen2.5-7B Family
Base
3,3
40.8
31,0
1.5
45.2
1.2
27.2
17.2
0.8
8.0
4.8
RLVR (Extemal Reward)
Conrect
14.2-10.9
71.0
62.44-1.44
12.64-41.1
62.7+11.55
6.843.6
8244555
79.6
16.8410.0
46.4+39.4
20.4,15.6
Random
10046.7
57.5416.7
45.7+1477
10224.77
00-1..
53.6400
30.8-156
1.240.4
6.8-1.2
3.6-1.9
Incorrect
6.7-3.4
$7.0.10.2
43.1
30.3
0.0
608
12.8-4.4
0.4...4
6.4-1.8
3.2-1.6
Formar
6.7
14.5
48.9 417.
1.5..0
44.4
2,4
37.2
21.6
0.80
6.8
4.4
Rewarded Reinforcement Learingg
10.0
482
28,.1.3
33.6 11.6
0.2
2644229.2
1633-..
0.80.0
6.8 1.2
3.2 1.0
EM
11.6.9.5
708
27.8
1.5..
37.5
0.0 67.24-80
27.2430.0
0.80
6.8
3.2-1.6
Llamma31-88 --80a33----tttttrccc Faunity
Bose
3.3
32.5
20.2
38.6
60.4
86.4
2.0
28.8
8.4
RLVR (External Reward)
Conrcct
6.7-3.4
6.
25.1444.9
21.0,20.2
49.1
4.3,0.2
76.0.15.
88.8
15.6,11..6
34.4.7..
11.6433.2
Random
3.30.0
2688-
21.3+1.1
0.0...
32.1-6.5
4.1..0
09.244.8
87.2-o.s
0.8-1.2
23.6-5.2
4.0
Incorrect
2.1 1.2
1..7-1.
0.8.0
8.4
38 0.3
70.0..0
83.2 s.2
0.8
19.2
4,4
Formar
3.1
31.5-100
18.7-1.5
0.8
36.4
4.16.0
68.8...4
85.6
2.0g
28.0
6.4.2.0
Sclf-Rowarded Reinforeement Learinng
1.3
46.2
23.0
0.77
2.7
4.3,0.2
67.2...55
83.2
8.2
2.0g.
28.0
0.8
8.8
5.1.1.8
38.3,6.8
25.0..4.a
0.80.0
3.8
73.6,13.2
2.00.0
23.6
5.2
Table 2: Comprehensive evaluation of different reward signals in RL. "Vote" denotes Majority Voting, "EM" means entropy minimization on self- eeff-generated samples only; OP: Operation CF Counterfactual. Red indicates potential contamination with strong model-task alignment; Gray indicates no contamination with weak model-task alignment; Green indicates contamination with strong model-task alignment.
Model-Task Alignment Determines Robustness to Noisy Rewards. The effectiveness of noisy reward signals depends on model-task alignment strength across our three experimental categories. In settings with strong alignment (Red and Green categories), models demonstrate surprising robustness to spurious rewards, with Qwen2.5-70 maintaining reasonable performance on mathematical tasks and both models showing improvements on Operation and Counterfactual tasks even with random rewards. Conversely, in weak alignment settings, spurious rewards consistently fail to provide meaningful improvements. as seen with Llama3. on mathematical tasks and both models on challenging logical reasoning benchmarks. This pattern confirms that alignment strength, rather than contamination alone, determines robustness to noisy rewards.
Relf-Rewarded Reinforcement Learning methods, including majority voting and entropy minimization, consistently underperform compared to external reward-based approaches. While self-rewarded methods shows some promise on math- ematical tasks for Qwen2.5-7B (majority vote achieves 69.4 on MATH500), fails to match the performance of ground truth external rewards and shows poor generalization to logical reasoning tasks across both model families.
Limited Effectiveness of rel--Re Methods.
4.2 TEST-TIME RL
Test-Time Reinforcement Learning (TTRL) Zuo al., 2025) addresses a fundamental challenge in LLM development: how to improve model performance on unlabeled test data without access to ground-truth labels for reward signals. It prompts the model to generate multiple responses to each test question and use the most frequent answer as the label for reward signals. Although the model is trained on the unlabeled test set, this approach is essentially no different from RewfrRe Reinforcement Learning when majority voting is employed. Thus, remains effective for different models and domains beyond mathematics.
also curious whether TTRL
Table shows shows the results of the Qwen and Llama models different tasks. Due the limited scale of the test dataset, we trained for 30 steps on all test datasets. It could be observed that in settings where the madel-task alignment is strong, TTRL yields substantial improvements, exemplified by

---

Published as a conference paper at ICLR 2026
Qwen math tasks and Operation subset. For tasks in which the model lacks initial prior knowledge, TTRL fails to deliver improvements or yields only marginal gains. As discussed by Zuo et al. (2025). majority voting is the foundation of TTRL. We also recorded the variation of Maj @ 16 during the training process; the results are shown in T We can observe that, in settings where TTRI. yields substantial improvements, Maj @ 16 consistently rises throughout training. Especially for Qwen Operation subset, it achieves an absolute gain of 16.4 points. This further underscores that TTRL's efficacy hinges on strong model-task alignment, rather than on contamination.
Model
MATH500
SynLogic
OP
Model
MATH500
SynLogic
OP
Qwen2.5-7B
40.8
1.5
27.2
Llama-3.1-8B-Inste +TTRI.
32.5
60.4
+TTRI.
62.1 221.3
1.8 0.3
55.6
41.2,8.7
0.8..0
83.6 23.62
Table Test Time Reinforcement Learning (TTRL) performance changes. TTRL produces signifi- cant gains only when model-task alignment is strong (red and green cells).
Step
Step
Step 10
Step 15
Step 20
Step 25
Step 30
Owen+Math500
54.2
60.6
64.3
68.2
67.1
69.3
70.5+16.3
Qwent-SynLogic
2.2
3.0
3.7
4.4
4.4
4.4
5.2++3.0
Qwen+OP
53.6
55.6
57.2
58.8
60.04+16.4
Llama-Math500
46.3
48.6
51.3
53.2
53.9
55.0
54.7+8.4
Llama+SynLogic
1.5
1.5
2.2
1.5
2.2
2.2
2.2+0.7
Llama+OP 73.6
78.0
79.6
84.0
83.6
86.8
88.4+14.8
Table 4: The variation of Maj @ 16 training progresses. In tasks where TTRL brings significant improvements (red and green), Maj @ 16 continues to improve with training.
5 RQ2 - Is ONE-SHOT ENOUGH FOR RL TO WORK?
Wang al. (025)) demonstrated that training
single carefully selected question yield performance comparable to full dataset training, challenging conventional assumptions about data volume requirements in RL. Wang et al. (2025) designs selection algorithm based the variance of training rewards, and we denote samples selected by this algorithm maelected for mathematical tasks and leelected for logical tasks. In addition that, we also randomly selected one two samples from the dataset to form (myandom: (random) and (m (aadom". random! for comparison. The specific examples we used are detailed in Appendix K. The remaining experimental settings are consistent with those described in Section 3 and we train models for 300 steps.
Math Tasks
Logic Tasks
Dataset
KOR Benchmark
AIME24
MATH500
AMC
SynLogie
BBH
BBEH
OP
CF
Puzzle
Logic
Cipher
Qwen2.5.7B
3.3
10.8
31.0
1.5
45.2
1.2
27.2
17.2
14.2+10.9
71.0+30.3
62.44:114
42.6
62.7+17.5
6845..6
82.4.5.
79.6402.4
1684,00.
46.4.35.4
20.4.15.
andom-11
10.7.7.4
58.7417.
23.1./2.1
088-..7
40.2 5.o
0..1.2
60.4
36.8
0.8.0
64-1.6
4.4-..4
andom-2 2
12.5 90.2
22.0
22.7
2..0..
43.1
1.2g.0
67.2
140.0
56.8
a6.8
20,1.2
3.2
4.8p.0
selected-1
12.3
65.2
55.2
0.8
39.9
0.0
69.2 142.0
38.4:2122
0,80.0
8.00.0
6.4
Ilama3.1 In8--nrrrett
3.3
32.5
20.2
0.8
38.6
4.1
60.4
86.4
2.0
28.8
8.4
full set
6.745.4
38.646.1
25.1
21.0.20.2
49.1
4.3...2
76.0.
88.8....4
15.6,15.6
34.4....
11,6,
3.8.0.5
30.5 2.0
20.1
0.8
s.5
38 0.a
13.6
o.a
1.2 0.8
0.8
8.8
random-2
2.7-0.8
33.1
21.
0.8g.0
36.7 1.9
4.1o.0
70.0.0.6
86.40.0
2..0
1.6
8.40.0
selected-1
3.7.
30.3
22.3.
0.8
34.4 4.9
38
69.2...
88.8...44
2.0.00
19.2
6.8
Table One-shot RL Results. OP: Operation; CF: Counterfactual. We only observe the effectiveness of one-shot reinforcement learning in settings with strong model-task alignment (red and green).
7

---

Published as a conference paper at ICLR 2026
5.1 RESULTS
We present results in Table 5 Based on the experimental results, identify two critical findings regarding the effectiveness of one-shot reinforcement learning:
One-shot RL Success Depends Model-Task Alignment. The effectiveness of one-shot re- inforcement learning is highly contingent on the alignment between model capabílities and task domain requirements. In strong alignment settings (Red and Green categories), both models demon- strate remarkable ability to generalize from single examples: Qwen2.5-7B achieves performance comparable to full dataset training on mathematical tasks (MATH500: 65.2 vs. full training 71.0), show substantial improvements on Operation
while both Qwen2.5 5-BB and Llama3. and Counterfactual tasks (e.g., Llama on Operation: 69.2 vs. baseline 60.4). However, this success does not extend to weak alignment settings, where both models show minimal improvements across challenging logical reasoning benchmarks. This suggests that one-shot RI. serves as an effective fine-tuning mechanism only when models already possess strong foundational capabilities.
Sample Selection Strategy Shows Limited Impact. Contrary to expectations, the sophisticated sample selection algorithm proposed by Wang et al. (2025). does not consistently outperform random sample selection. For Qwen2.5-7h5 on mathematical tasks, both selected and random samples achieve similar performance levels (MATH500: selected 65. 2 random 58.7 and 63.0), while for Llama3. 1.mna3..1-BB.. 1ee.1-88--Intttee Instruct, the differences are negligible across all benchmarks. This finding challenges the assumption that reward variance-based selection provides substantial advantages over simpler random sampling approaches.
Train with Isimle
Train with Iiid
Train with (sarected(
acC1 (oriliaa(rlloout
acc; shxlooolllll
acCg
acCi
0.8
Qwen Accurscy
acc
aCGood
0.6
aCC sno(rollout)
aCCμ
0.4
acCadd
0.2
acc1 shortrollaut)
8CC -neelote/lllloutt
acC -mzurobbu
acCg
Llama Accuracy
BCC.on
aCCood
0.
0.4
0.2
120 0800180220 Steps
120 18000022000000000 270000000 Steps
30
120 18001000000000000 240 Steps
Figure 3: The changes in two models' accuracy during the training. If the initial rollout accuracy is non-zero, both models rapidly fit the employed samples (Isimple, bmid) and exhibit generalization within the same subtask; however, we observe no generalization to puzzles of other types.
5.2 DISCUSSION
Wang et al. (2025) showed that training on single sample for mathematical tasks can quickly improve the accuracy of that sample and also lead to improvements
the test set. We attempt to verify this conclusion on logical tasks. Considering that the initial rollout accuracy of the model on lacleated we additionally sample two examples whose initial rollout accuracies on Qwen2.5-7B are 16 and 1/16 (on 1--ma-3.1-B---- SSS-Instruct are 3/16 and 1/16), denoted as lsimple and Imid- During training, we track three metries: the rollout accuracy of these examples acC1-shot, the accuracy of the subtask to which this example belongs (n-distribution) acc;a, and the accuracy of other subtasks in SynLogic (fut-of-diste d-sstribuuiion acCood- The results shown in Figure
8

---

Published as a conference paper at ICLR 2026
One-shot RL possesses the ability to generalize within the distribution. When the problem is relatively simple (with an initial rollout accuracy that is not zero), the model's rollout accuracy on that sample quickly increases. Although the initial rollout accuracy of midd Qwen is only one-fifth that of Lsimpte (on Llama is one-third), it still attains a high rollout accuracy within a few dozen steps. Since GRPO and DAPO compute advantages via intra-group normalization, the model is unable to derive any informative feedback from samples whose initial rollout accuracy is zero. Moreover, we observe that the test accuracy for the same subtask also continues improve, demonstrating effective within distributiornn generalization.
One-shot RL struggles to generalize other types of logic puzzles. We find that while models can improve on tasks similar to their training example, they fail to transfer learning to different puzzle types. This suggests that one-shot learning primarily exploits existing model capabilities rather than developing new reasoning skills.
RQ3 DOES RL WORK WITH ONLY NEGATIVE SAMPLES?
Recent work (Zhu al., 2025) has demonstrated that training exclusively negative samples be surprisingly effective for model reasoning. However, these findings are primarily observed in scenar- ios with strong model-task alignment. We investigate whether negative-only training generalizes to weak model-task alignment scenarios, where models lack strong foundational capabilities.
Implementation Details. In our implementation, negative Sample Reinforcement (NSR) masks out all trajectories with reward 1 (correct answers) when computing the policy gradient, leaving only negative-rewarded samples to drive updates. Conversely, Positive Sample Reinforcement (PSR) ignores trajectories with reward 0 and optimizes only on positively rewarded samples. All other hyperparameters remain identical to the DAPO baseline described Section
6.1 RESULTS
Math Tasks
Logic Tasks
KOR Benchmark
AIME24
MATH500
AMC
SynL.oglc
BBH
BBEH
OP
CF
Puzzle
Logic
Cipher
Qwen2.5-78
3.3
40.8
31,0
27.2
17.2
8.0
4.8
DAPO
14.2
71.0,30.2
62.4 62.4..1.4
42.64-41.
62.7447.5
6.855.6
82.4..5.2
79.6-.2.4
16.84
46.4..0.44
20.4
NSR
10.0
687.277.
63.5
1,6.0
4..2...0
16400.4
604 .5.4.
-6.8
2.,1122
688 1.2
4.8o.0
PSR
10.0
20.3.
63.1 432.1
24.8
223.3
57.1411.9
4.3
73.6
28.4.
9.2.
31.2.,23.2
11.2,a..
32.5
20.2
60.4
28.8
DAPO
6.748.4
38.6
25.1
21.0
19.1
4.3
76.0.
88.8
15.6 18.0
34.47.0
11,6
NSR
7.944..6
36.9
36.9444
24.7.4.5
0.0-..a
34.2
4.34
67.2
86.40.0
2.0g
28.0
-o.
5.2-3.
PSR
4.
34.7
23.6.3.4
13.0,11
4
4.1.
69.2.8.6
3.2
12.0
34.4. 6
10.8,
Table 6: Results of NSR and PSR under different settings. When Model-Task alignment is strong, both NSR and PSR yield pronounced performance gains for all models (Red and Green). Conversely, under weak alignment, NSR-trained models exhibit no noticeable improvement (Gray).
Table summarizes the performance of NSR and PSR relative to the full-signal DAPO baseline across our three experimental categories. It reveals distinct patterns based on model-task aligmment:
Strong Model-Task Alignment Enables Effective Negative-Sample Learning. In settings with strong model-task alignment (Red and Green categories), both NSR and PSR show comparable effectiveness, recovering most of the performance gains achieved by full-signal DAPO. For Qwen2.5- 7B mathematical tasks, both approaches achieve -95% of the DAPO improvement (MATH500: NSR 68.7 7 and PSR 70.3 vs. DAPO 71.0). This demonstrates that when models already possess strong domain capabilities, either positive-only negative-only signals can effectively drive learning.
Weak Model-Tase Alignment Reveals Superior Performance of Positive-Only Signals. weak aligmment settings (Gray category), PSR consistently outperforms NSR across logical reasoning tasks. For instance, on SynLogic, PSR enables meaningful improvements (Qwen2.5-7B: 1.5 vs. 24.8, Llama3. 1-8B: 0.8 vs. 13.0), while NSR shows minimal gains. Overall, while PSR and NSR

---

Published as a conference paper at ICLR 2026
Logic Tasks
Math Tasks
2.0
0.5
0.5
Eatropy
Tatroppy
0.4
0.5
DAPO
DAPO PSR
PSR
0.0
150
Step
Step
Figure 4: Entropy Dynamics of Qwen2.5-7B during Training. NSR can maintain the exploration space of RL, but larger exploration space is not always favorable, in logical tasks.
demonstrate comparable effectiveness in strong alignment settings, PSR emerges as the more robust approach in challenging domains where models lack expertise.
6.2 DISCUSSION
The relationship between positive and negative samples in reinforcement learning is fundamentally connected to the exploration-epllitattiii trade-off, with entropy serving as a key mediator. To elucidate these dynamics in our experimental context, we examine how different sample types affect the exploration-exploitaiie balance through their impact on training entropy.
Negative Signals Help Maintain Exploration. Figure plots token-level entropy throughout training. Consistent with Zhu et al. (2025). NSR slows entropy collapse, especially on mathematical tasks suggesting that penalising only erroneous trajectories can preserve output diversity. However, the flatter entropy curve on logical tasks corresponds to poorer final accuracy.
7
MORE ANALYSIS
To further strengthen our hypothesis, we conduct several additional analyses:
Our conclusion holds under broader settings. Beyond Qwen and LLaMA and the domains of mathematics and logic, we further validate our conclusions on code-generation tasks (Appendix G)
and Mistral (Appendix H.. model that performs poorly on both mathematics and logic. Across these broader settings, our coamennt-based hypothesis continues to hold.
KL change under standard RL as auxiliary model-task alignment indicator. In Appendix I we show that, addition to Pass K, the magnitude of the KL divergence change induced by standard RL (before vs. after training) can
an auxiliary quantitative indicator of model-task alignment. We further verify that adding a KI. penalty during training does not change our main conclusions.
Pass @ K strongly predicts RL gains. Figure T0 to directly visualize the relationship between our alignment proxy (Pass @ K) and the gains from standard RL. Across different K values (e.g., K-32, 128, 512) and across the RL phenomena we study, we consistently observe positive trend: performance improvements grow as @ass K increases.
CONCLUSION
This work reveals that Model-Task Alignment strength, measured by pass@k accuracy, serves as the fundamental determinant of when counterintuitive RL phenomena emerge in language model reasoning. We demonstrate that remarkable behaviors Dehaviors-inctudingg robustness to spurious rewards, one- shot training effectiveness, and negative-only signal sufficiency meaffffencymmaifesstt primarily when models already possess strong foundational capabilities in the target domain, functioning more as capability elicitation mechanisms rather than genuine learning drivers for unfamiliar tasks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This project is partially supported by Hong Kong RGC ECS Grant 26218125, Hong Kong RGC CRF Grant C6003-24Y, and NSFC Grant 62306177.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility of results, conduct all experiments using publicly available mod- els and datasets. We provide complete implementation code, detailed hyperparameter configurations, and step-by-step reproduction instructions in the supplementary material.
REFERENCES
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effec- tiveness of entropy minimization in Ilm reasoning, 2025. URL ULL.attpa://axx.voor/////
2505 15134
AIME. Art of Problem Solving -
artofproblemsolving.com
https://
aatofproblemmsovviee com wwwo///wii/iiex..../iee php/AIME / Problems p and
Solutions 2024. [Accessed 26-08-2025].
AMC. Art of Problem Solving atofproblemsoolingg.cmm artofprobl emsolving wwtom/wwkiiiii http://eeeeeerrpra/////111P1 _nd Solutions 2023. [Accessed 26-08-2025).
https://
Jiangjie Chen, Qianyu He, Siyu Yuan, Aili Chen, Zhicheng Cai, Weinan Dai, Hongli Yu, Qiying Yu, Xuefeng Li, Jiaze Chen, Hao Zhou, and Mingxuan Wang. Enigmata: Scaling logical reasoning in large language models with synthetic verifiable puzzles, 2025. URL http::t/////. /ttp://aaxvvvvvv org
abs / 20..251119999
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL
wwg.xiiv..gg..o abs /2107.2210.0m 03374
Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in rlhf: When better reward models don't yield better language models. arXiv preprint arXiv:2410.06554, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.1290 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024.
Dan Hendrycks. Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL
patts://aarxe httpsss//arxiv.orr/aasss 03874

---

Published as a conference paper at ICLR 2026
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishkyy Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXhv:2412.16720, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL
org
07974
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxiv
brg/aba/ 2310. 06825
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, San- ket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoe V. Le, and Orhan Firat. Big-bench extra hard, 2025. URL https:: hnttts::/arxiv.com org/abs 2502 19187
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer. Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances Neural Information Processing Systems, 35:3843-3857, 2022.
Jiawei Liu. GitHub ganler/code-rl: Reproducing R1 for Code with Reliable Rewards github.com. com http:////ttttob..cccoaaaaeee///s r1 2025. [Accessed 21-11-2025].
Junteng Liu, Yuanxiang Fan, Zhuo Jiang, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, Yunan Huang, Mozhi Zhang, Pengyu Zhao, Junjie Yan, and Junxian He. Synlogic: Synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond, 2025. (Rttpst:://axxttt
www. /ab//ab///// /200. 19641
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol prevview with a 1.5b model by scaling rl. Link 2025. Notion Blog.
Ang Lv, Ruobing Xie, Xingwu Sun, Zhanhui Kang, and Rui Yan. The climb carves wisdom deeper than the summit: On the noisy rewards in learning to reason. arXiv preprini arXiv:2505.22653. 2025.
Kaijing Ma. Xinrun Du, Yunran Wang, Haoran Zhang. Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, Wenhao Huang, and Ge Zhang. Kor-bench: Benchmarking language models on orowloage-orthogonal reasoning tasks, 2024. URL https://aaxix org/abs//2 2410. 06526
Meta. The llama herd of models, 2024. URL
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025, URL https://a xx xiv. org wtw./////////o 15115
12

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URL
https://arxiv org/ abs 2200 18290
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506. 2025.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. Mastering chess and shogi by self-play with general reinforcement learning algorithm, 2017a. URL http://axivvvromo org/ abs 1112.1111111 01815
David Silver, Julian Schritwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, L. Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-359, 2017b. URthttss:://a semanticscho lar.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thouge can solve them. arXiv preprint arXiv:2210.09261, 2022.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. Scaling reinforcement learning with llms. arXiv preprint 2wXix::501.12599 2025.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwen m. wtt.mm..t....... http:o///olloo//////o
Yiping Wang, Qing Yang. Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.2057,, 2025.
Minggi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao Senjie Jin, Xiaoran Fan, Yuhao Zhou, Yanwei Fu, Qin Liu, et al. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. arXiv preprint arXiv:2507.
2025.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu. Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via seff-improvement, 2024. Utttttts:://are v. org/ abs/ 240...449... 12122
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang. Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL wtttttss//aaiii...ttt org/ abs/ /abb//255.. 2505. 09388
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint 250i::2503..44700 2025.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Dangi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in Ilm reasoning, 2025. URL https: wtlllatttt/sxiv..oooo //arxiv..zg// 220/2 01347
13

---

Published as a conference paper at ICLR 2026
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909 1909.00999 2019.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuckai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprini arXiv:2504.1608 16084, 2025.
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
As non-nate non-native English speakers, we used LLMs solely to assist with grammatical correction and linguistic polishing of the manuscript. The LLM not involved in any aspect of conceptual development, experimental design, data analysis, or interpretation of results. All scientific content, including hypotheses, methodology, figures, and conclusions, was generated independently by the authors. The use of the LLM was strictly limited to improving clarity and fluency of expression in English, ensuring that language barriers do not impede the accurate communication our research contributions.
B IMPLEMENTATION DETAILS
Following the setting described in Section 3 we train with different rewards for 300 steps on mathematical and logical reasoning tasks, respectively. The format reward is different from that of Shao et al. (2025), we use the same format as SynLogii:
<think>thinking process</hinnk
answer</answer>
We 7 0.5 for the random reward. For incorrect rewards, reward rollouts that produce incorrect answers during training for logic and code tasks. Apart from these changes, the definition of the reward functions remains consistent with spurious reward (Shao et al. 2025
C MORE PASS @K RESULTS
AIME24
AMC23
MATH500
Q2em1.5-00
Qwer2.5-71
Qwer2.5-73
0.8
0.8
0.6
0.6
0.6
0.4
0.2
0.2
0.2
0.0
128 256
512
Number of Samples
Number Samples
Number of Samples
Figure @ass@k k for math tasks. Qwen demonstrates strong capabilities across all three mathematical evaluation datasets.
D CONTAMINATION EVALUATION
D.1 IMPLEMENTATION DETAILS
Our contamination analysis follows systematic prompt truncation methodology to evaluate potential data leakage across model-task combinations, Original prompts truncated varying ratios (0.4, 0.6, and 0.8) while preserving word boundaries, and models are asked to complete the remaining content using greedy decoding for deterministic outputs. We measure contamination using ROUGE-L scores between model completions and the actual remaining prompt content, where perfect score of 1.0 indicates complete reconstruction and potential contamination. The evaluation pipeline employs distributed processing to handle complex mathematical expressions and prevent evaluation timeouts, with results aggregated across multiple rollouts to ensure statistical reliability.
D.2 MORE RESULTS
E DISCUSSION ABOUT RQI
Shao et al. (2025) observed that in mathematical tasks, employing ground truth rewards decreases the frequency of code usage in
How Different Reward Signals Affect the Behavior of LLMs.
15

---

Published as a conference paper at ICLR 2026
Cipher
Counterfactual
Logic
Qwam15-71 5-10
Owen2.5-11
Qwee2--73
0.8
pasogll
0.6
0.6
0.6
0.4
0.2
0.2
0.2
0.0
8
16
32
64
128
256
512
16
32
64
128
256
512
2
4
8
16
32
64
128
256
512
Number Samples
Number Samples
Number Samples
Operation
Overall
5-B
Owi27.5 1-18
Qwae2.577
0.8
puose
0.6-
0.6
0.6
0.4
0.2
0.2
0.2
0.0
0.0
0.0
16 32
128
256
512
52 Number Samples
16
128
256
512
2
4
8
16
32
64
128
256
512
Number Samples
Number of Samples
Figure 6: Pass@k for KOR-Bench. Both models demonstrate strong inherent reasoning capabilities in Operation and Counterfactual subtasks, but exhibit limited inherent logical reasoning abilities in Cipher, Puzzle and Logic.
Portion=0.4
Portion=0.6
Portion=0.8
Task Type Benchmark
Model
ROUGE
EM
ROUGE
EM
ROUGE
EM
Qwen2.5-7B
63.78
23.91
64.42
33.73
73.23
49.39
AMC
Math Tasks
Llama-3.1-8B
27.18
0.00
30.64
0.00
44.54
4.81
Qwen2.5-7B
50.36
8.20
60.98
21.20
66.42
40.20
MATH500
Llama-3.1-8B
23.09
0.60
40.56
3.80
48.33
17.8
Qwen2.5-7B
44.64
10.00
48.69
13.33
60.08
30.00
AIME24
Llama-3.1-8B
26.08
0.00
30.80
0.00
50.50
13.33
Qwen2.5-7B
19.56
0.00
19.62
0.00
19.24
0.00
Puzzle
Llama-3.1-8B
18.27
0.00
17.31
0.00
15.85
0.00
Qwen2.5-7B
21.37
0.00
24.25
0.00
20.18
0.00
Operation
Llama-3.1-8B
21.83
0.00
18.34
0.00
16.75
0.00
Logic Tasks
Qwen2.5-7B
18.88
0.00
19.96
0.00
18.66
0.00
Counterfactual
Llama-3.1-8B
19.02
0.00
19.39
0.00
18.94
0.00
Qwen2.5-7B
22.08
0.00
27.28
0.00
28.23
0.00
Logic
Llama-3.1-8B
21.38
0.00
28.37
0.00
28.42
0.00
Qwen2.5-7B
34.61
0.00
41.03
0.00
44.77
0.00
Cipher
Llama-3.1-8B
29.59
0.00
36.95
0.00
42.93
0.00
Table Extended Contamination Analysis across model-task combinations. Red indicates potential contamination with strong baseline performance; Gray indicates contamination with weak baseline performance; Green indicates no contamination with strong baseline performance.
model responses. Their study also revealed that, in contrast to Qwen2.5-Mard (Yng al. 2024), the accuracy improvement of the Qwen2.5 Base model was primarily attributed to a shift from code-based reasoning to language-based reasoning. As shown in Table g we identify analogous trends in mathematical tasks. Specifically, for logic puzzles, the application ground truth rewards similarly reduces the incidence of code in responses. However, other types of rewards, particularly
16

---

Published as a conference paper at ICLR 2026
format and random rewards, do not demonstrate significant impact diminishing code usage frequency. We speculate that, throughout the RL training process, ground truth rewards can steer the model away from its old reasoning pattern ( (ie., producing reasoning responses with code and toward a more natural, language-based reasoning pattern.
MATH500
SynLogie
Reward Type
Before RL
After RI.
Before RI
After RI.
Correct
12.4
21.7
Random
94.2
48.2
89.1
57.3
Format
96.7
50.7
Incorrect
28.1
28.3
Table 8: Code Usage Count of Qwen2.5-7B before and after RL training with different rewards.
As shown in Table spurious rewards effective only the Operation and Counterfactual for the Llama model; consequently, we also report the frequency of code-based reasoning before and after training these two tasks. As shown in Table we observe that, both before and after RI training, Llama almost never invokes code during the reasoning proees. We attribute the sporadic use of code (0.8) the fact that some SynLogic tasks explicitly require outputs to be presented code blocks. This indicates that Llama and Qwen exhibit distinct reasoning patterns even though they both benefit from noisy reward signals in these settings.
Operation
Counterfactual
Reward Type
Before RL
After RL
Before RL
After RL
Correct
0.8
0.0
Random
0.0
0.0
0.0
0.0
Format
0.0
0.0
Incorrect
0.0
0..
Table Code Usage Count Llama-3.1 1-11----nnstuue before and after RL training on two tasks.
F MORE DISCUSSION ABOUT DIFFICULT EXAMPLE IN ONE-SHOT RL
During training with selected, apart from the rollout accuracy (reward) remaining consistently at metrics such as entropy and response length also exhibit almost changes. As shown in Figure 7 after 300 training steps, the model still maintains a large reinforcement learning exploration space.
response length
580
Respronse Length
560
Entropy
5:40
1.8
520
entropy
500
100
150
250
100
150
200
300
Stop
Step
Figure 7: Training Dynamics of Qwen2.5-7B when trained with selecteed- Entropy and response length exhibit almost no changes.
17

---

Published as a conference paper at ICLR 2026
G RESULTS ON CODE GENERATION TASKS
We additionally include code-generation tasks to further validate the generality of our findings. Specif- ically, evaluate on HumanEval (Chen et al. 2021) and LiveCodeBench 2024.8-2025.13 (Jain ct al. 2024). The live nature LiveCodeBench ensures that data contamination almost impossible. However, both models achieve relatively high Pass scores HumanEval. To assess whether these high scores may result from contamination, we conduct an analysis analogous to that in Table The results are shown in Table I0 Similar to the math tasks, we observe higher risk of data contamination in Qwen. Accordingly, we categorize Qwen/HumanEvaal as red and Llama/Human as green. And we show the Pass @ curves for LiveCodeBench in Figure Neither model exhibits sufficiently strong capability on this benchmark: as K increases, their performance does not rise as sharply as it does on tasks such as Operation (shown in Figure 6). Consequently, we categorize
Qwen/Liiveoode and Llama/LiveCoode as gray.
Model
Portion
ROUGE
EM
Model
Portion
ROUGE
EM
0.4
56.32
9.7
0.4
23.42
0.0
Qwen2.5-7B
0.6
60.89
14.3
LLama3.1-8B
0.6
32.13
0.8
66.81
36.7
0.8
47.32
8.4
Table 10: Contamination Analysis on HvaanEval.
LiveCodeBench
1.0
Qwen2.5-7D
Llama -D-tnrrrcc
0.8
Pass
0.6
0.4
0.2
0.0
2
16
32
64 128
Number of Samples
Figure 8: Pass @ K curves for LiveCodeBench.
For training, we use code-r1-12k (Liu 2025) the training data. Importantly, to keep the setup consistent with the math and logic tasks, we assign a correctness reward of 1 only when the generated code passes all test cases; otherwise, the model receives no correctness reward all. Because code outputs are difficult to aggregate via majority voting, we do not evaluate the Vote setting in this task. To reduce variance, we report @ following the same protocol as SynLogic and AIME24.
HvmanEval
LiveCodeBench
HumanEval
LiveCodeBench
Qwen2.5-7B
52.4
6.2
Llama3.1-8B
68.4
8.3
Correct
65.2
15.4
Correct
15.1
12.0
Format
58.546.1
5.2 1.0
Format
69.2
8.940.6
Random
56.1
5.0 1.2
Random
67.4_1.0
7.3
Incorrect
53.0.p.6
4.2-20
Incorrect
68.8 40.4
6.0_2.3
EM
58.1
6.2.
EM
70.4 +2.0
8.0 q.3
1-shot
51.5 0.9
6.0 0.2
1-shot
69.0 40.6
6.5
NSR
55.1
7.4.
NSR
70.4
6.8 1.5
Table 11: Code-generation results on HumanEval and LiveCodeBench. The 1-shot samples are selected based on Wang et al. 2025's s method.
18

---

Published as a conference paper at ICLR 2026
The results of the two code evaluation tasks are shown in Table II From the table, can that under the standard RL setting, the model shows clear improvements on both HumanEval and LiveCodeBench. Under non-standard settings, only the red-category case- Busee-Owe//uumalte the green-category case Jlama//umanFval-ceaa observed in the weak-alignment setting. This is consistent with our observations math and logic tasks.
and small gains, while no consistent improvement is
H RESULTS OF A MORE WEAKLY ALIGNED MODEL: MISTRAL-7B-VO.J
We additionally include Mistral-7B-v0.1 (Jiang et al. 2023) in the weak-alignment setting. Its performance math and logic tasks is worse than that of Llama-3, 3. to provide more weakly aligned settings. Results are shown in Table 12 We can see that Mistral shows stable improvements math and logic tasks only under the standard RL setting. This further reinforces the validity of our conclusions under the weak-alignmmet setting.
which allows us
Muth Tasks
Lagie Tisks
KOR
SynLogie
OP
CF
Puzzle
Logic
Cipher
Mistral-7B-vt. 1 (Weakly Aligned)
Base
0.0
6.8
3,4
0,0
27.1
11.2
22.8
2.4
28.4
2.4
RLVR (External Reward)
GroundTroth
3.3
17.6,10..
12.5...1
14.6414.6
38.9411.5
3.2
26.8..5..
43.4-20.
9.847.4
46.2.17.s
Format
0.0...00
0.4
200
0.,00.0
24.3 2.8
0.4
1464
262.
20 0.4
26.8 1.0
32,
Random
0.0....0
6.4
3.2...22
0.4.0.4
21.2
0.4.0.0
9.2
19.4
20.0..
22.4
2.0.0.4
Incorrect
0.0....0
5.6
1.0-2.4
0.,00.0
13.4
0.0
9.0
19.2
18800.0
21.0-.44
1.8-0.6
Self- Rewarded Reinforcement Leaming
0.4...4
1..
3.2
0.0.0.00
14.2 12.9
0.4,0.0
12.8.1..
5..
1/2 1.2
188
9.0
3..
EM
0.0..0.0
2.-
0.8
23.4
0.4.0.0 10.4
19.2
1.4
24.4
2.0
Fow-shot Reinforvemnt Learning
(selected)
0.4.0.4
11.00
2.8 0.6
p0.00.0
4.7
0.4
2.4 19.8
3.0
24.000
29.6.1.2
2.0 0.4
Negative Sampling Reinforcement Learning
NSR
0.8 1..8
644
3.0
0.0,0.0
24.2 2.9 0440..0 124 23.2
10.4
2..
21.0 4.4
2.0 0.4
Table 12: Results of Mistral-7B-vO.1.
H.1 DISCUSSION
In Tables 2 and 6, although Llama-Math is a weakly aligned model-task pair, both self-rewarded methods and NSR consistently lead to performance improvements. Here we provide an explanation for this phenomenon: Although we categorize Llama-Math as weakly aligned model-task pair, its alignment is still noticeably stronger than that of Lagic. We see two reasons for this:
1. As shown in Table [ although we do not observe the severe contamination found in Qwen, the Llama-Man Math combination exhibits higher risk of data contamination compared with Llama-Logic (FM = 0)
2. As shown Figure while the model's performance does not increase rapidly as grows, its curve is still sigmificantly sharper than that of tasks such as Puzzle in Figure 6
That being said, model-task alignment strength may be more continuous rather than falling into just three discrete categories. When the alignment is slightly stronger, as in the case of LLama on math tasks, some RL methods may begin to take effect. The results on Mistral further support conclusion: due to its weaker alignment, neither NSR nor the self-reward method achieves consistent performance gains on the math tasks. Identifying such an "emergence boundary" in terms of model-tast alignment strength interesting direction for future work.
19

---

Published as a conference paper at ICLR 2026
I KL DIVERGENCE BETWEEN THE INITIAL AND TRAINED POLICIES
During training, consistent with DAPO Yu et al. 2025). we removed the KL. regularization term from the loss function. This enables to fairly compare the impact of different training methods on the output distribution under varying levels of alignment.
We explore in three representative settings: Qwen MATH500 (red), Qwen on Operation (green), and Qwen on SynLogic (gray). Specifically, we use trained policies to generate trajectories via greedy decoding. Then, we feed these trajectories into the untrained reference model obtain the log-pni uig-prbbabilities for each token. We follow the approach used in DeepSeek-Ri (Guo et al. 2025 for computing KI. divergence:
Torl log To(0; Oci)
Traf q. 0zii) To(0s Ocs)
KL(TTll (ref))
and compute the average KL over all tokens. We present the results in the Figure From the results, observe that:
In the standard RL setting, weaker model-task alignment leads to larger divergence between the pre- and post-training output distributions (the gray is substantially higher than the green and red ones).
In the weak-alignment setting (gray), only standard RL and PSR drive the model away from the reference model and achieve substantial performance improvements (+41.1 I and
+23.3 SynLogic).
In the strong- alinnnetlggmment setting (green and red), neither the reward choices nor the sampling methods substantially amplify the divergence between the pre- and post-training output distributions.
MATH500
0.30
SynLogic
Operation
0.20
0.10
0.05
0.00
Standard RL
Format
Random
Incerrect
Vote
EM
s-shot
NSR
PSR
Figure 9: KL. divergence between the initial and trained policies under different settings.
Based these results, hypothesize that in the alignmmtn setting, large deviations from the initial policy are unnecessary; small updates are sufficient to improve performance. This also helps explain why spurious-reward methods still be effective, the initial policy is already close to the correct solution. contrast, in the weak-alignment setting, merely oscillating around the initial policy does not yield meaningful gains- substaaabbla improvement requires moving further away under the guidance of correct rewards to form a stable and effective reasoning pattern.
I.1 POTENTIAL CHANGES INTRODUCED BY ADDING KL REGULARIZATION
natural question whether introducing KL regularization during training would affect our con- clusions. Considering that the difference between the pre- and post-training output distributions is small the gtrong-aligmmen setting, we believe that introducing KL regularization does not affect the overall trend of performance changes, although it may influence convergence speed (Yu et al.) 2025). To assess the impact under the weak-alignmment setting, particularly for standard RL and PSR, set the KI. penalty coefficient 0.001 and retrain Qwen the logic tasks. The experimental
20

---

Published as a conference paper at ICLR 2026
results shown in Table + However, the overall trend remains the same as the setting without regularization, indicating that the presence absence of KL regularization does not affect our conclusions.
Introducing KL regularization leads to slight drop in performance.
SynLogic
BBH
BBEH
Pretrain
1.5
45.2
1.2
Standard
42.6441..1
62.7.17.5
6.8
Standard + KL
33.6432.1
52. 46.9
4.86
PSR
24.8 423.3
57.1
4.31
PSR KI.
17.3
49.1 +3.9
3.8 4..6
Table 13: Performance of different training methods with and without KL regularzztiion under the weak-alignment setting.
1.2 DISCUSSION
From Figure 9 we believe that the divergence between the pre- and post-training output distributions in a successful standard RI. run (without collapse) can serve as a complementary indicator of model-task alignment strength. This is also consistent with intuition: in the strong-all aliggmmeet setting, the model does not need to extensively explore regions far from the initial policy to achieve performance gains. In contrast, in the weak-alignment setting. successful training process must sufficiently explore regions farther away from the initial policy in order to discover an effective reasoning pattern. Moreover, the consistency between KL divergence and Pass @ K. further reinforces that Pass @ K can be viewed as a reliable proxy for model-task alignment.
J CORRELATION BETWEEN PASS @ K AND RL GAINS
we present the relationship between RL gains and Pass @K. Each data point in the figure and And each point represents task, with points in the first column averaged over different reward signals. We can find that across different K values (32, 128, 512) and across the various RL phenomena we study, we consistently observe that performance improvements grow as @ass@K K increases. This further strengthens the connection between model-tast alignment strength and performance gains.
In Figure 10
derived from the results reported in Tables 3 5
K FEW-SHOT RL EXAMPLE DETAILS
21

---

Published as a conference paper at ICLR 2026
Spurious Reward
Few-shot
Negalive-Sample
0.4
Cwan w/ Green Sotting
0.3
0.20
Qwen Gray Setting
0.2
0.1
0.00
0.0
-0.05
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
@332
0.4
0.3
Performance A
0.20
0.15
0.2
0.1
0.00
0.0
-0.05
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Pass@ 128
0.4
0.3
0.20
0.15
0.2
0.1
0.1
0.00
0.05
0.4
0.6
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Pass@ 512
Figure 10: Pass K versus performance gains across tasks and RL settings.
Details of example Maefected
How many positive divisors do 9240 and 13860 have in common?
Details of example Myandom
The angles of quadrilateral PQRS satisfy LP = 32Q 42R 6/S. what is the degree measure of /P?
22

---

Published as a conference paper at ICLR 2026
Details of example m rondom
Given a finite sequence S (a), U2 ,,, Un) of real numbers,
let A(S) be the sequence 01+02
On +00
of
real numbers. Define (S) (S)) Suppose I r> 0, and let 2 2 m 1,,, define (S) A(A
A(S) and, for each integer m,
(1,0, ;...., 100). If (S) = (2m) then what is x? AND
2,2x 2,3z + are in geometric progression, the fourth term is:
Details of example
selented
Here's s a mathematical expression: ?-?+ (695)
2. The digits the left side of the equation have been replaced with question marks. Each question mark corresponds digit between 0 and 9. You need to try replacing the question marks with the correct digits to restore the expression. put the complete expression with the filled in digits between [[ and ]] at the end of your response, 10]]
with no other content, like this: [12
Details of example 4random
Solve this cryptarithm: RRYUU + UYR + U = RYUUU (where RRYUU is 5-digit number, UYR is a 3-digit number, U is 1-digit number, and RYUUU is 5-digit number). Each letter represents unique digit. Find the digit substitution that makes the equation true.
23

---

Published as a conference paper at ICLR 2026
Details of example Y random
In this Number Wall puzzle, add walls (marked as (A') to divide the grid into islands. Each island must contain exactly one number, and its size must equal that number. Grid:
+---t
X
3
X
X
X
X
X
X X
Rules:
Each island must contain exactly one number. The total number of cells in an island (including the number cell) must equal the value of that number.
All cells within island must be connected horizontally or vertically.
Walls (marked as (A') cannot form 2x2 or larger continuous rectangles.
All islands must be separated by walls. AND
the cryptarithm: MMII MIXIMM MMXIIX, each letter stands for different digit (MMII is 4 digits, MIXIMM is digits, and MMXIIX is 6 digits). Determing what each letter represents to make the equation true.
Details of example simple
this word sorting challenge, you need rearrange words in increasing based on a modi alphabet where 1,z and are the first letters. Words to sort:
yachted, coelomic, harateen. Write your final answer inside: [boxed, like this: |boxedwordl, word2, word3.
24

---

Published as a conference paper at ICLR 2026
Details of example miid
You are an expert proficient in Dyck language, where you must complete all types of unclosed brackets (e.g., [], <>) language sequences. You need to analyze the steps of bracket pairing according to Dyck language rules. Given an initial Dyck language sequence and steps for deriving the closed bracket sequence (presented in a thinking process format), your task is to identify locations with incorrect reasoning in the Dyck language, and there may be multiple errors. This could be forgetting to close bracket, using the wrong closing bracket, or incorrectly copying a subsequence of closing brackets in the next step. Task: Check the sequence to ensure brackets are properly closed. Input: [[(())1 should process the input one by one and track the stack confiquration.
Thought
Thought Stack: Empty
Thought 3: [ Stack: Empty
Thought [ ; Stack: [[
Thought 5:
Stack: [[(
Thought 6:
; Stack: [[
Thought 7:
Stack: [[{
Thought 8: ; Stack: [[
Thought
; Stack: [
Thought 10: ] ; Stack:
Empty
Thought 11: ; stack:
Thought 12: ; Stack: Empty
Thought Now, we have reached the end. The final stack
is empty.
Question: Are there any reasoning errors this sequence?
25


Reviews:
[{"review_id": "CWSbxDZmbT", "reviewer": "Reviewer_DMgH", "summary": "This paper argues that many of the recent, slightly “too good to be true” RL-for-LLMs findings (spurious/noisy rewards still work, one-shot RL, test-time RL, negative-only signals) only work when the base model is already well aligned with the task, and that this alignment can be read off from pass@k on that task. When alignment is weak, they say, those tricks collapse, while plain RLVR/DAPO still helps.", "questions": "see weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This works has studied different claims in RL literature for LLM to show their effectiveness.", "weaknesses": "1. The experimental evidence is drawn almost entirely from Qwen2.5-7B and Llama-3.1-8B on math and KOR-style reasoning tasks (AIME24, MATH500, AMC, Operation, Counterfactual, Puzzle, Cipher). The conclusion that model–task alignment is a *fundamental* determinant therefore generalises beyond the evaluated model/task scope.\n\n2. The paper equates model–task alignment with pass@k, but no prior work is cited to justify this choice of proxy. It would strengthen the argument to motivate pass@k more formally or to report an additional alignment signal (e.g. a divergence-based measure between base and trained models) as a complementary view.\n\n3. The description in Appendix D.3 of “incorrect” and “format” rewards is underspecified: the exact criteria for each reward type and the RL setup used in that subsection should be made explicit, and the authors could briefly justify why they chose this particular RL recipe over newer GRPO-style variants (e.g., Dr.GRPO).\n\n4. The set of weakly aligned models is fairly narrow (mainly Llama-3.1-8B in low-pass@k regimes). Including another family with weaker out-of-the-box performance on these tasks (e.g. Gemma 3) would make the alignment–effectiveness pattern more convincing.\n\n5. The paper should state earlier and more prominently that DAPO is the default RL method used in the experiments, so readers do not have to infer it from later sections/appendices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u0Bl7BXSzQ", "reviewer": "Reviewer_FtyV", "summary": "The paper identifies Model-Task Alignment, measured by pass@k accuracy on the evaluated task, as a key factor that explains the counterintuitive phenomena that have been reported in LLMs but not observed in traditional RL settings. These phenomena include single training example matching the performance on the entire dataset, the reward signal not needing to be very accurate, training only with negative samples matching the performance of sophisticated reward-based methods, efficacy of test-time RL, etc. The paper demonstrates across multiple model-task combinations, spanning two language models (Llama and Qwen), and multiple mathematical and logical reasoning domains, that strong model-task alignment explains the counterintuitive phenomena observed in LLMs but not in traditional RL. In settings with weak model-task alignment, standard RL approaches work while the counterintuitive techniques fail", "questions": "1) Do you have the model-task alignment results for other LLMs (eg: mistral, deepseek, gemma) or different model sizes?\n\n1) Was hyperparameter tuning performed for each model-task combination and the counterintuitive RL phenomenon being explored?\n\n2) Why does self-rewarded RL seem to work for the Llama-math combination in Table 2, which has no contamination with weak model-task alignment? Any factors that may explain this?\n\n3) Similar to the previous question, Negative Sampling Reinforcement (NSR) also seems to work for the Llama-math combination in Table-5. What explains these observations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper critically evaluates existing claims in RL for LLMs, such as the effectiveness of spurious rewards, one-shot training, and negative-only signal sufficiency, and identifies Model-Task Alignment as a key factor determining if these claims hold true.\nAdditionally, the paper provides a comprehensive and systematic examination of these counterintuitive reinforcement learning (RL) phenomena in large language models (LLMs), supported by rigorous experimental validation across different model architectures and task domains. Furthermore, they show that dataset contamination is not the underlying cause, as was proposed in earlier works, by demarcating model-task combinations into those potentially contaminated and those not potentially contaminated in their experiments. Moreover, the paper is well-written and easy to follow, with dedicated sections for each of the reported counterintuitive phenomena.", "weaknesses": "The experiments primarily focus on Qwen and Llama models (of similar parameter scale, 8b), which may limit the generalizability of the results to other LLMs. Further experiments encompassing more LLMs and model parameter scales would help to understand if the results are more general. Additionally, while the Model-Task Alignment hypothesis is well-supported, the paper could explore other factors that might also contribute to the observed phenomena, such as the role of pretraining data, model sizes, or architectural differences.\n\nAdditionally, the paper does not explicitly mention if hyperparameter search was performed for each model-task combination and the counterintuitive RL phenomenon being explored (eg: negative samples, spurious rewards, single example training, test-time RL etc.).  Different combinations might need different hyperparameters to elicit their best possible performance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S4rmk1zWlo", "reviewer": "Reviewer_nBRb", "summary": "This paper examines various hypothesis and claims about the performance of RL methods for language models. In particular, they investigate numerous phenomena around RL -- training with noisy rewards, training on single examples etc. and show how their performance is strongly correlated with \"model-task alignment\".", "questions": "not any at this time.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* the paper is well organized and generally easy to follow. the purpose of each section is clearly defined.\n* the experiments cover a number of different hypotheses for why certain phenomena appear\n* The authors consider multiple model types.\n* The insights provided by the paper are generally useful\n* using pass@k for model-task alignemnt is an interesting idea, but digging more into the realm of RL I think this might really be an indicator of how much exploration is necessary for RL to work.", "weaknesses": "The paper claims that \"model-task alignment\" measured by pass@k is predictive of whether or not several of these RL-based phenomena occur. However, the authors only use a handful of benchmarks and bin the \"model-task alignment\" into either strong or weak. It would be interesting to see if the authors could plot the pass@k (the proposed proxy metric) versus the increase in performance to actually see the correlation. From the results pass@k does seem predictive, but it's hard to know for sure without a ton of information. Doing more tasks for all methods would be excessive, but it could be interesting to see at least one representative.   For example, one could consider a coding domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kuk11J7Fx3", "reviewer": "Reviewer_QY7Y", "summary": "This papers presents an empirical study on some reported phenomenas in RL training such as: 1) training with random erroneous rewards (random reward), 2)  self rewarding LLMs via Test time RL i,e using the majority vote answer from a rollout as a true one , 3)  one shot , i,e trainig with a single selected example 4) GRPO training with only positive or negative sampling. Some recent work Wu et al 2025 explained some of these phenomenas as due to the contamination of the training set of some of LLMs with the test set. This paper probes another hypothesis that the contamination is not the only reason behind such phenomenas but the alignment between the task and the LLM at hand. The alignment between task and LLM is measure with pass@k of the LLM on the test set of this task. Authors focus on Qwen and Llama family of models and divide task and models to three areas: strong alignment and strong contamination (red), strong alignment and weak contamination (green) , and weak alignment and no contamination (gray). Throughout extensive experimentation authors find that pass@k on a task is a predictive of the performance of the model with standard RL, but also its robustness to the phenomenas listed above. A high pass@k wether contaminated or not pretaining translates into a lift in performance for standard RL and the variants above (TTRL, one shot RL, random reward, positive , negative sampling), and low pass@k leads to a bad performance across the board.", "questions": "* what is the group size you used in DAPO? \n* do you think the conclusions will change if we add the KL regularization to pi_{ref} in the training of these RL ? \n* can you study the KL of those perturbed RL to see how the perturbation of the reward impacts the KL emprically ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is very well written and a very nice and easy read with all experiments well articulated and well presented, I personally enjoyed reading the paper. \n\nThe fact that pass@k of the reference model is a predictive of the performance of standard RL is not surprising and it has been even shown theoretically in multiple works showing that best of n is an approximation of the RL optimal policy see for example best of n through the smoothing lengths or in Asymptotics of Language Model Alignment or in  information theoretic gaurantees of alignment in LLMs. It is interesting to develop similar results to robustify these results to data and reward perturations as presented in this paper.", "weaknesses": "One interesting point the paper is not studying in the analysis is to follow the KL(pi|| pi_{ref}) when using standard RL versus the non standard RL (TTRL, one shot RL, PSR, NSR). \n\nIt would be interesting to compare the E_{test set }KL(pi|| pi_{ref}) , to see how the perturbations of the rewards/ sampling impacts the deviation from the reference models, in the green, red and gray area. Note that using DAPO we don't have the regularization to pi_{ref} so this is a fair study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)