From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones
Authors: Lifan Yuan
Weize Chen
Yuchen Zhang
Ganqu Cui
Hanbin Wang
Ziming You
Ning Ding
Zhiyuan Liu
Maosong Sun
Hao Peng
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Large Language Model
Reasoning
Exploration
Primary area: foundation or frontier models, including LLMs
TL;DR: We show that once a model has acquired the necessary atomic skills for a task, RL enables the composition of these skills into more complex capabilities when properly incentivized.

Abstract: Does reinforcement learning (RL) teach large language models (LLMs) genuinely new skills, or does it merely activate existing ones? This question lies at the core of ongoing debates about the role of RL in LLM post-training. On one side, strong empirical results can be achieved with RL alone even without preceding supervised finetuning; on the other, critics argue that RL contributes little beyond reweighting existing reasoning strategies. This work provides concrete evidence that LLMs can acquire genuinely new skills during RL by composing existing ones, mirroring one of the central mechanisms by which humans acquire new cognitive skills \citep{Anderson1982Acquisition}. To mitigate data contamination and other confounding factors and to allow precise control over task complexity, we develop a synthetic framework for our investigation. Specifically, we define a skill as the ability to infer the output of a string transformation function $f(x)$ given $x$. Once an LLM has already learned $f$ and $g$ prior to RL, our experiments reveal that RL enables it to learn unseen compositions of them $h(x)=g(f(x))$. Further, this compositional ability generalizes to more difficult problems such as compositions of $>2$ functions unseen during training. Our experiments provide surprising evidence that this compositional ability, acquired on the source task, transfers to a different target task. This transfer occurs even though the model has never trained with RL on any compositional problems in the target task, as long as it has acquired the target task's atomic skills prior to RL on the source task. Our qualitative analysis shows that RL fundamentally changes the reasoning behaviors of the models. In contrast, neither of the findings is observed in next-token prediction training with the same data. Our systematic experiments provide fresh insights into the learning behaviors of widely-used post-training approaches for LLMs. They suggest the value of building base models with the necessary basic skills, followed by RL with appropriate incentivization to acquire more advanced skills that generalize better to complex and out-of-domain problems.

Published as a conference paper at ICLR 2026
FROM fx) AND g(x) TO f(g(z)) LLMS LEARN NEW SKILLS IN RL BY COMPOSING OLD ONES
Lifan Yuan-*, Weize Chen²*, Yuchen Zhang Ganqu Cuist, Hanbin Wang",
Ziming You', Ning Ding2.3r
Zhiyuan Liu Maosong Sun?, Hao Peng'
1 University of Illinois Urbana-Champaign
Tsinghua University
3 Shanghai AI Laboratory
Peking University
lifan40 1llllnnnooii 8. edu
chenwz21 @ma i 1s ts wnw.iighuaa.uuu.u.
ABSTRACT
Does reinforcement learning (RL) teach large language models (LLMs) genuinely new skills, or does it merely activate existing ones? This question lies at the core of ongoing debates about the role RL in LLM post-training. On one side, strong empirical results can be achieved with RL even without preceding supervised fine- tuning; on the other, critics argue that RI. contributes little beyond reweighting ex- isting reasoning strategies. This work provides concrete evidence that LLMs acquire genuinely new skills during RL by composing existing ones, mirroring one of the central mechanisms by which humans acquire new cognitive skills (Ander- son 1982). To mitigate data contamination and other confounding factors, and to allow precise control over task complexity, we develop synthetic framework for our investigation. Specifically, we define a skill as the ability to infer the out- put of string transformation function f(xx) given x. When an L.LM has already learned f and y prior to RI., our experiments reveal that RI. enables it to learn unseen compositions ity generalizes to more difficult problems such compositions of > 2 functions unseen during RL training. Our experiments provide surprising evidence that this compositional ability, acquired on the source task, transfers to different target task. This transfer occurs even though the model has never trained on any com- positional problems in the target task, and the only requirement is that the model has acquired the target task's atomic skills before its RL training on the source. Our qualitative analysis shows that RL fundamentally changes the reasoning be- haviors of the models. In contrast, none of the findings is observed in next-token prediction training with the same data. Our systematic experiments provide fresh insights into the learning behaviors of widely-used post-training approaches for LLMs. They suggest the value building base models with the necessary basic skills.
them h(x) = g(f())). Further, this compositional abil-
INTRODUCTION
Reinforcement learning (RL) has achieved broad success in improving large language models (LLMMs) on a variety of tasks especially reasoning (OpenAl 2024 Deep.ind 2025), even directly building upon the base model without any preceding supervised fine-tuning (DeepSeek-Al et al.) 2025). Despite the profound success, recent work finds the exploration of RL is impeded by the entropy collapse phenomenon (Cui et al. 2025b Liu et al. 2025a Yu et al. 2025). and the perfor- mance gaps between base and RL-trained models diminish as the number of samples (k) increases in pass@k evaluations Yuue et al. 2025). In addition, some argue that the "aha moments" in RL training (OpenAT 2024, DeepSeek-Ai et al., 2025) are not emergent but merely the result of am- plifying existing cognitive behaviors present in base models (Gandhi et al. 2025 [Lu et al. 2025b Zhao et al. 2025), which casts shadow on whether L.MMs learn new skills during RI. training (Wu itt al. 2025a). Such observations diverge from established RL findings that predate LLMs, where
models were trained from scratch and learned new skills (Silver et al. 2016 2017 OpenAl et al.
"Equal Contribution. Orders are determined randomly. Corresponding author.

---

Published as a conference paper at ICLR 2026
Clean Testbed for RL Analysis: String Transformation Prediction
def Pinno (5)6)
Can you predict the output of "Euse_"e _61 ''abe''" without wwitiig code?
Decontamination:
return Join]
db
*2" for ch in
compensu, (xpoutata) (a) ded dune
Well- Defined Atomic Skill: y=fix
if no5
rutsre
Atomic Task
Controllable Task Difficulty
resals [R1E]]
tune 361 abe
Fuze 36: (offe115 1)
Can you predict the output of 'funn_ (func 16 "abc without writing code?
Level (Atomic) Level 2 (Compositional)
ceauit appendreh)
retorn Josn (34228))
Pinc15(ftme22
String Transformation
Compositional
(Compositional)
The RL Compositionality Hypothesis
(1) / RL=Compositional Data is the Key
Task Method Atomic Skill Compositional Skill
Atomic Skills
Base LLM
Comp.
Comp.
RL
NTP Training
Atomic
RL
Compositional
Skill Transfers across Domains
Pre-RL RL LLM
Task
Atomic Skills
Task
Compusiless
Generalizable
Atomic Skills
Compositional Skills
Atomic Skills
Compositional Skills
RL on Basic
Compositional Skills
(3)
/
RL does not Merely Rerank
Passilkk
Base
RL
Hanke
Baza
Compositional Skills
Sample Number k
Sample Number Level 3+
Level 1-2
Figure I: An overview of our research framework and key findings. (Top) We introduce clean string transformation testbed to scientifically analyze RL's capabilities. (Bottom-Left) Our central RI. Compositionality Hypothesis posits that training on simple composites with RL unlocks generalizable compositional skills. (Bottom-Righh
Our experiments validate this, showing that: (1) compositional data combined with RL is the key ingredient for learning this new skill; (2) the learned skill transfers across domains; and (3) RL significantly improves difficult problems where the base model fails, while only reranking on problems it solves well.
2019). The fact that LLMs are pretrained on vast data prior RL may contribute to these diver- gences and call for further investigation into the following important research questions: (1) Does RI. teach new skills LLMs? (2) If so, how to incentivize it? (3) Are the skills generalizable? Answering these questions will advance our understanding of LLM learning behaviors and inform the high-stakes trade-off in resource allocation between pretraining and post-training.
We provide concrete evidence that LLMs indeed learn new skills in RL, by composing and general- izing existing skills to solve more complex problems; For such learning to happen, there should be proper incentivization in RL. Our investigation is grounded in the cognitive skill acquisition process by humans, inspired by Anderson (1982). ). which argues that humans learn new skills by composing and then internalizing existing ones. Unlike prior works (Gandhi al., 2025 Yue al. 2025).
choose to construct a controlled synthetic framework that facilitates:
Decontaminated evaluation: We design string transformation prediction task with unique func- tions assigned meaningless identifiers (e.g., func_1 6) to prevent inference from function names. During RL, function definitions hidden. The tasks will then be unsolvable without going through our atomic skills acquisition training. This setup enables us to investigate the RQs con- trolling for confounders.
Well-defined atomic and compositional skills: nonddecomposable transformations, and compositional skills as their nested combinations. For example, given input string
We define atomic skills as single,
func_16 16 (x) represents an atomic skill, while func_15 (func_16 (x:) ) requires compositional reasoning.
Controllable difficulty: As each skill is instantiated of composition through the depth of nesting. As shown in Fig. [ the model must perform de- ductive reasoning to give the output string after given transformation, e.g., Level-1 difficulty problem func 16 (x) and a Level-2 one func 1 (func 15 (x) .. Here the difficulty level is determined by the number of atomic functions composed.
Python function, control difficulty
2

---

Published as a conference paper at ICLR 2026
With our framework and two-stage training protocol that separates atomic from compositional skill acquisition, we conduct experiments with Llama-3.1-3 15.1-88-Insttuct (Dubey et al. 2024 and answer the RQs as follows:
RL teaches new compositional skills. RL on Level-2 problems, receiving only correctness- based outcome rewards without reasoning demonstrations, substantially improves generalization on more difficult problems: performance on unseen Level-3 tasks improves from near-zero to 30%, and Level-4 to 15%. This generalization does not occur in baseline trained with rejection fine-tuning (RFT) on the same Level-2 problems. This shows that RL enables the acquisition of compositional skills.
Both RL and compositional incentives are essential for skill acquisition. In contrast to the substantial accuracy improvements from RL. on Level-2 compositional problems, RFT on the same data and RI on Level-1 1 atomic problems both yield little improvements on problems higher than Level-2 (e-g., less than 1% improvement at Level-3). This may explain why [Sun et al. (2025) conclude that RL does not promote compositional generalization, their training includes no explicit incentive for composition.
The learning achieved by RL generalizes to held-out evaluation, more difficult problems, and even a different task. All findings above are based on held-out evaluation of compositional problems consisting of atomic skills (functions) unseen in RL training. And as aforementioned, models RI-trained on Level 2 problems show non-trivial gains on problems up to Level 4. For cross-task transfer, compositional RL on the string task boosts accuracy on the unseen Level-3 Countdown problems to 35% for a model with the prerequisite Countdown atomic skills.
Our findings challenge the recent view that current RL with verifiable rewards (RLVR) (Lamber) iet al. 2025) merely utilizes reasoning patterns in base models rather than learning new reasoning abilities Yue al., 2025 Wu et al., 2025a). This view is based on the observation that the pass @ performance gap between RL-trained and base models narrows as & increases Yue et al. 2025) We conjecture that this observation arises, at least in part, from evaluating and RI. training on tasks where base models already achieve high pass@k, possibly due to pretraining on similar tasks that is beyond the control of most academic researchers; thus RL has little incentive to learn a skill that the base model already has. To confirm this conjecture, experiments show that RL substantially improves pass@ on challenging compositional problems where base model's pass @ A: is near zero (See Fig. 5). This reveals what we term the "reranking illusion," namely aggregate metrics on mixed- difficulty benchmarks can mask genuine skill acquisition by conflating capabilities of different types. Our qualitative analysis confirms that RL fundamentally changes reasoning behavior. As shown in Fig. 6 compositional errors, i.e., ignoring composition and misunderstanding function relationships, drop substantially, while failures shift primarily to atomic prediction errors (55%). This behavioral transformation indicates genuine acquisition of compositional skills.
Our findings have important implications for LLM development and highlight RL's critical role in post-training, particularly its potential for easy-to-hard generalization and cross-task transfer. They call for closer coordination between base model development and post-training strategy from skill acquisition perspective.
BACKGROUND
The Recent Pessimistic View on Whether RL Teaches New Skills to LLMs. RL in LLMs builds on model pretrained on vast data. While supervised warm-starts are a common technique in tradi- tional RL (Silver et al. 2016 Vinyals et al. 2019. De La Cruz Jr et al. 2019 Silva & Gombolay 202T). the large-scale and general general-purpose purpose nature of LLM creates a different scenario. On one hand, this strong prior enables base LLM to sample reasonable rollouts and thus perform RI. directly with- out any preceding supervised fine-tuning (DeepSeek-Al 2025); on the other hand, it becomes difficult to distinguish genuine skill acquisition from activation of existing capabilities during RL training.
al.] 2025 Pan et al. 2025 Zeng et al.)
Recent work tries to investigate this but uses loose definitions of "skill", often relying on proxies such as the continually increasing frequency of certain reasoning patterns (Gandhi et al. 2025 Zhao et al., 2025 Liu et al. 2025b) or the diminishing gaps between the pass accuracy of models
before and after RL, as shown in the bottom right chart in Fig. [ Yue et al. 2025 Liu et al., 2025a
Wu et al. 2025a He al. 2025 Wen al. 2025
Zhu et al. 2025). Although these studies show
3

---

Published as a conference paper at ICLR 2026
that RL activates behaviors already present in the base model, they did not directly prove that no new skill is learned during the process. Moreover, the pass@k k results can be misinterpreted for many reasons: (1) The causal relation between performance and each skill remains unclear, thus it is not guaranteed that everything learned can be translated into improvements in pass@ k accuracy on downstream tasks. (2) The evaluation tasks only provide an obscure overall view, lacking fine- grained analysis on problems of different difficulty levels or domains. (3) The result is confounded by the possibility that the model has limited room or incentive to learn new skills if it already performs well prior to RL, especially when RL is conducted on data that overlaps with or closely resembles the data used during next-token prediction (NTP) training. (Wu et al.] [025cc; Shao al.]
2025 Wang et al., 2025 Cui et al. [025a]] Yu et al. 2025 Liu et al.) 2025a Wu et al.) 2025b).
Together, these highlight the urgent need for deeper analysis of tasks through a clean framework,
in which the skills are clearly defined and contribute to the performance causally, and evaluated in finer granularity.
Compositional Learning as Testbed Grounded in Cognitive Skill Acquisition in Humans. Although the pessimistic conclusions about RL in LLMs from prior works are debatable, they at least indicate that the success of RL depends on strong base models. This motivates our study of skill composition, where RL learns new abilities by leveraging those already acquired by the base model. Compositional reasoning provides an ideal framework for investigating skill acquisition because it naturally separates atomic knowledge, which mirrors how humans learn cognitive skills (Anderson] 1982). Notably, it is established in cognitive science that both composed skills and the meta-ability to learn composition are non-trivial new skills (Anderson 1982 Lake et al. 2016). For clarity, we refer learning new skills as the former throughout this paper. Learning compositional skills helps the model to generalize to more challenging problems and new domains beyond training data, which we will show later. In the field of AI, compositional reasoning has been widely studied before LLMs and has been considered a necessary property of generalization. (Fodor & Pylyshyn 1988 Lake et al.] 2016; Andreas et al., 2015). More recently, Yin et al. (2025) achieved compositional improvements through in-context learning rather than RL, while SSnn et al. (2025) found that directly RL in atomic skills fails in compositional generalization. Comparing the two works, we conjecture that explicit incentive to composition is necessary.
3 RESEARCH FRAMEWORK
In this work, we define "new skills" as novel reasoning strategies that enable models to solve pre- viously unsolvable problems through systematic combination of existing capabilities. We address three critical research questions: (1) Does RL teach new skills to LLMs? (2) If so, how to incen- tivize it? (3) Are the learned skills generalizable?
Hypothesis 1 (The RL Compositionality Hypothesis). Once a model has acquired the necessary atomic, non-decomposable skills for a task through NTP training, RL with proper incentivization can teach the model to learn new skills by composing atomic skills into more complex capabilities.
3.1 TASK DESIGN: DEDUCTIVE REASONING ON STRING TRANSFORMATION PREDICTION
To test our hypothesis while avoiding confounders from data contamination and unclear skill bound- aries, we design a controlled synthetic task with the following properties: (1) Atomic skills are well defined so that models can learn the fundamental skills separately before RL. Each string trans- formation function has clear, deterministic behaviors that can be learned independently. (2) Task difficulty can be controlled by adjusting the compositional complexity of the atomic skills, allowing us test generalization across complexity levels. (3) RL and evaluation tasks do not appear in the LLM pretraining corpus, ensuring that improvements stem from learning rather than memorization.
Specifically, our task involves deductive reasoning on string transformations. Given an input string and a composition of deterministic transformation functions such as f(-) and g(-), models must predict the output string after applying the specified transformation (e.g., y (((x))) We construct 25 unique string transformation functions as atomic skill spanning var- ious computational patterns including character manipulation, reordering, filtering, and structural modifications (see Appendix SD for complete specifications). To mitigate potential contamination, we assign meaningless identifiers to string functions as shown in Fig. I so that it is impossible to infer the functionality with function names only.
Task Definition.
4

---

Published as a conference paper at ICLR 2026
Difficulty Level. We control compositional complexity through Difficulty Levels corresponding to nesting depth, with Level n involving n-ffuncion composition. For instance, Level involves single function application (e.g., fune 16 (x) shown in Fig. [T]. while Level involves two-function composition (e.g.. func 1 6 (Func 15 (x) ) The controlled difficulty provides a fine-grained inspection of model performance, rather than a vague overall number adopted in prior work (Yue)
et al. 2025 Liu et al. 2025a Wu et al. 2025a).
3.2 TRAINING AND EVALUATION PROTOCOL.
Training consists of two stages to separate atomic skill acquisition from compositional skill learning, simulating realistic post-training pipelines.
Stage Training: Atomic Skills Acquisition via RFT. Models learn "atomic skills" in this stage via rejection fine-tuning Dong al. (2023). Specifically, we collect training data by prompting the model with explicit function definitions to generate correct reasoning trajectories. However, we re- move these definitions from the prompts during the fine-tuning process. This compels the model to predict the output based solely on the function identifier, ensuring it internalizes the function's 's behavior, defined here as the atomic skill, rather than relying on in-context instructions. Crucially, the data collection phase of this stage is the only time models are exposed to the function implemen- tations. An example can be found in Fig. 7
Compositional Skill Training via Either RFT or RL. In this stage, models see only function names and compositions, such as func.2 2 ( (func_1 (x) ), with function definitions hidden. See Fig. [ for examples. This forces reliance on internalized atomic knowledge while learning systematic composition. We compare two approaches: (1) Composition via online RL provides models with binary rewards based output correctness and updates through Group Rel- ative Preference Optimization (GRPO) Shao et al. (2024), testing whether RL is necessary for the acquisition of compositional skills. (2) Composition via offline RFT trains models with NTP correct reasoning trajectories for compositional problems, serving as a baseline to examine whether exposure to compositional examples alone enables composition.
Stage 2 Training:
We use Llama-3.1 1-..--88-Insttte -8B--nsttuct, which is identified as a cleaner testbed for RL by recent work (Shao al. 2025 Agarwal al. 2025 Wu et al.) (2025)), to further minimize the effect of data contami- nation besides our string tasks. For more details, please refer to Appendix A
Held-out, Easy-to-Hard, and Cross-Task Evaluation. We assess generalization using rigorous held-out evaluation. evalutoutevlluttionn. In Stage I, models are trained on all 25 atomic functions (Appendix()) In Stage 2, the functions are partitioned into two disjoint sets: the model trains only on compositions from one set, while the other is held out for evaluation. We test model generalization across various difficulty
levels, using Countdown (Gandhi al., 2024 Pan et al. 2025) as a testbed for task transfer.
4
RL AS A PATHWAY TO GENERALIZABLE SKILL ACQUISITION
4.1 LLMS ACQUIRE NEW COMPOSITIONAL SKILLS DURING RL
Our first experiment directly test our RL Compositionalltyy Hypothesis (Hypothesis D. To do so, we start from an identical Stage base model and apply three different Stage 2 training configurations, allowing us to isolate the impact of incentivizing composition during RL: (1) RL Level 1, trained only on atomic tasks; (2) RL Level 2, trained only on two-level compositions; and (3) RL Level 1+2, trained on a uniform mix. We then evaluate their ability to generalize to held-out tasks from Level up to Level 6, testing whether they can solve problems with unseen function compositions and higher nesting levels than seen in RL training.
As shown in blue curves in Fig. 2 training Level alone leads to high accuracy on Level 1, peaking at around 90%, but fails to generalize. Its accuracy on Level task remains below 25%, and on Level 3 through 6, it is consistently near zero. This demonstrates that learning only the atomic skills through RL is insufficient for learning effective composition.
In contrast, incorporating compositional tasks into RL training yields transformative results. Both the RL Level and RL Level 1+2 models demonstrate strong performance to generalize to prob- lems with nesting depths exceeding their training data. On Level 3, their accuracy improves from
5

---

Published as a conference paper at ICLR 2026
Leve Lave 112
RL Lovel
Fi Loved 2-12
0.25
Lave L12
0.09
8:00
0.3
0.2
0.1
6:30
Accuraty
Lovel
Lovel
Loval3
Lave
0.06
Fit Leved
Loved
5,000
14 Lave
0.00
2,00
0.66
2.000
220
300
430
500
000
100
200
320
400
620
1.20
220
300
500
600
Level44
Lover5
Lenel
Training Stops
Figure 2: Test Accuracy on held-out tasks vs. RL training steps, each related to one held-out task difficulty level. The dark blue curve indicates that training atomic skills alone (RL Level 1) yields nearly no compositional ability on held-out functions. In contrast, including Level 2 data in RL unlocks strong generalization more complex problems (Levels 3-6).
5% to around 30%, and from 1% to 15% on Level 4, which are all significant improvements over the RL Level model. And this trend continues on even Level 5, indicating both models learn a generalizable principle of compositional reasoning rather than merely memorizing solutions. This validates our hypothesis that RL. can teach genuinely new skills, but only when the training objective explicitly incentivizes their use. These results provide us with evidence to answer RQ1:
TAKEAWAY 1
RL on compositional data teaches new skills that generalize to unseen compositions of known atomic skills.
4.2 RL IS THE KEY INGREDIENT TO THE NEW COMPOSITIONAL SKILLS
Our previous experiment shows that compositional data is necessary for RL teach new composi- tional skills, but can a supervised method, such as RFT, achieve the same results as RL when given the exact same compositional (Level 2) data? To address this question, we train model with iterative RFT on the same Level 2 problems and conduct head-to-head comparison against the RL Level 2 model from $4.1 with both having started from the identical Stage base model.
The results in Fig. 3 show significant difference in performance from Fig. 2 The RFT model's accuracy is significantly worse than RL across all compositional levels and has only marginal im- provement over the first iteration. For example, the RL Level model achieves 64% on Level and 27% on Level 3, significantly outperforming the RFT model. Surprisingly, the RFT model attains only 15% accuracy Level-2 problems. This indicates that RFT fails to generalize even to held-out compositional problems of the same difficulty as its training data, let alone higher difficulties. These results provide the evidence answer RQ2:
Level 3 it never surpasses 2.6%. In contrast,
TAKEAWAY 2
RFT, even with on compositional data, is suboptimal for learning compositional skills; RI., in addition to compositional training data, is another important factor in learning generalizable compositional skills.
4.3 COMPOSITIONAL SKILLS LEARNED IN RL ARE TRANSPERABLE, BUT ATOMIC SKILLS ARE PREREQUISITES
While experiments demonstrate that RL can teach generalizable compositional skills within task, collecting compositional RL data for every new domain is impractical. We therefore test the transferability of the learned compositional skill. Specifically, we conjecture that RL enables models to compose atomic skills on Task after learning composition on Task A, if the model has already acquired the necessary atomic skills for Task B.

---

Published as a conference paper at ICLR 2026
Level
Level
0.23
0.23
0.13
0-4-
6.1-
0.00
0:75
Augous
0.000
0.029
0.000
0:00
0.007
Iecaaion
Figure 3: RL vs. RFT on Compositional Tasks. RL (red dashed line) achieves substantially higher accuracy across all levels, while iterative RFT fails to learn a generalizable skill.
Table 1: Model configurations for the task transfer experiment.
Stage
Stage 2
Model Configuration
String Atomic RFT
Countdown Atomic RFT
String Atomic RL
String Comp. RL
String- Base RI. 1.142
Brring-Base-
Multi-Base
Multi-Base + RL LI
Multi-Base + RL L1+2
Experimental Setup. We test this conjecture on the Countdown task, where model must con- struct mathematical expression from given set of integers to reach target number (see $E for examples). In Countdown, a Level task requires the model to construct a mathematical expression using & given integers to reach target number. The minimum level for Countdown is Level 2. We compare four models to test our hypothesis, as detailed in Tab. These configurations allow us to compare "atomic-skill-oni
baseline (Multi-Base) against models with either transferred atomic RL (Multi-Base + RL LI) or transferred compositional RL (Multi-Base + RL L1+2), as well as control model from 1 that has the compositional skill but lacks the necessary atomic knowledge of Countdown (String- Base + RL L1+2). Note that none of the models are trained on Countdown with RL in Stage 2, and are only trained on our string task.
We evaluate these models on unseen, more challenging Countdown problems (Levels 3-5). We report the Avg the average accuracy across 32 responses sampled at temperature 1.0.
Results. The results in Fig. provide clear evi- dence supporting our hypothesis. The String-Baat Base RL Ll+2 model fails completely. The Multi- Base model achieves reasonable accuracy ap- proximately 17% at Level 3 but still struggles at higher levels. Multi-Base RL LI shows marginal improvement over Multi-Base, increasing racy to around 20% at Level 3, with the advan- tage diminishing
035
030
Avgg Accumary
0.25
0.0
0:35
0.10
complex problems. The Multi-Base + RL L1+2 model achieves surpris- ingly strong performance. achieves 35% ac- curacy at Level 3, outperforming the Multi-Base baseline by more than 18% accuracy. This advan- tage persists at higher complexities, reaching ap- proximately 6% at Level 4, where other models largely fail and achieve near-zero accuracy. The results show that the compositional skill learned from string transformation transfers to countdown, acting as a meta-skill that enhances the use of the target task's atomic knowledge. Finally, the comparison between Multi-base + RL and String Base + RL. L1+2 confirms our fundamental assumption that -ssk-ipec atomic skills are prerequisites for compositional skills to be effective.
0.05
0.00
Countiiven
Figure 4: Avg@32 Accuracy the Count- down Task. Atomic skills are a prerequi- site for task transfer, and that compositional RL (Multi-Base + RL L1+2) on the unrelated string task offers significant performance im- provement on Countdown. Note that none of the models are trained with RL on Countdown.

---

Published as a conference paper at ICLR 2026
Level
Level
1.5
6.-
Levd 212
0.-
1. Lovel
1ovel1332
Sum
04
5.
Loxel
0.2
60-
2
Lovel
B8 Dave
RTT Dase
FFT Zern
31:33
6.16
0.0
Level1.22
3.05
2.00
FLLand 2+22
082
00:00
636
0.00
606
08323
27
27
Number of Samples
Figure 5: Pass@k performance across varying difficulty levels. On easy problems (Levels 1-2), the performance gap shrinks with more samples, consistent with the reranking narrative. On hard problems (Levels 3-8), the gap widens substantially, suggesting new skill acquisition.
These results may explain recent findings on generalizable RI improvements. For example, Logic- RL (Xie al., 2025) reports performance gains on mathematical problems after training on logic puzzles, and Guru CCheng et al., 2025) shows that domains with greater pre-training exposure benefit more from cross-task generalization. We suggest that L.MSs have already acquired essential atomic skills through large-scale pre-training, particularly in mathematics and coding. Thus, incentiviz- ing compositional skills through RL one task helps combine task-specific skills more effectively across domains. In contrast, domains with less pre-training exposure may lack sufficient atomic skills, limiting compositional skill transfer to downstream tasks. With this finding, we answer RQ3:
TAKEAWAY 3
Compositional skills learned through RL are transferable to a different task where the model possesses the atomic skills.
4.4 RL EXPANDING PERFORMANCE LIMITS IS NOT A FALSE PROMISE
Our findings strongly suggest that RL can teach compositional skills that are novel to the base model. This directly challenges recent arguments that RI merely "reranks" model responses, dis- tilling @ass@k k performance of the base model into pass@1 1 (Yue al. 2025 Wu et al. 2025a). This conclusion is drawn based on shrinking pass @ k performance gap between base and RL-tuned models as & increases. However, we argue this conclusion may stem from two issues: (1) evaluating on mixed-skill benchmarks, therefore improvement in specific skill, like composition, can be masked in pass@) @ k if other required skills remain a bottleneck, and (2) using RL training that does not properly incentivize the new skill in the first place.
Our controlled framework allows us to dissect both issues. By isolating the compositional skill at varying difficulty levels, we can reliably assess skill acquisition (addressing issue 1), and by compar- ing different RL. training setups 1)..1). we can test the effect of proper incentivization (addressing issue 2). We compare pass 1000 performance
each difficulty level of our test set, selecting k 1000 as sufficiently large and practically meaningful budget. Larger budgets would become impractical, as any reasonable model could theoretically achieve pass
The results are presented in Fig. 5 Both RL Level and RL Level 1+2 models are trained from RFT base model using RL in Stage 2. The RL Level I model, which not incentivized properly to learn composition, exhibits a similar trend to the RFT base across almost all levels. On easier problems (Levels and 2) where the RFT base model already shows solving potential evidenced by high @ass@k, the performance gaps between RL Level 1+2 model and the RFT model shrink as increases, aligning with the trends observed Yue et al. (2025); Wu et al. 2025a)). However, a completely different trend is observed on more challenging compositional problems (Levels 3-6). The RL Level 1+2 model's performance substantially outperforms the RFT base with an increasing gap k grows. For example, at Level 5, the performance gap the RFT base grows from 4% at pass@ @ to approximately 25% at pass@ 1024. This divergence is clear evidence of new skill acquisition. The results suggest that the pessimistic observation of "RL. does not push performance
8

---

Published as a conference paper at ICLR 2026
limits" in prior work may be explained by the lack of incentive for RL to learn new skills, as the base model already achieves high pass @k performance.
TAKEAWAY 4
The prior conclusion that RLVR only utilizes base models' reasoning patterns without learn- ing new abilities is likely an artifact evaluating and RL training on tasks that base models already achieve high pass k; thus RL has little incentive to learn new skill.
4.5 BEHAVIORAL ANALYSIS: RL TRANSFORMS FAILURE MODES
While our results show that training with compositional data unlocks promising generalization, fundamental question remains: do models trained under different setups exhibit different behav- iors, or do they simply differ in capability while showing similar failure modes? To investigate this, we analyze the failure modes of different models on Level 3 problems of our string task.
classify responses into five categories: (1) Correct, (2) Ignores Com- position (e.g., analyzing only single function), (3) Incomplete Trace (recognizes composition but terminates early), (4) Incorrect Composition (e.g., misinterprets nesting), and (5) Atomic Er- ror (errors in atomic functions prediction without the above). Categories 2-4 indicate difficulties with handling compositional problems. And while still incorrect, category 5 represents appropriate compositional behavior, as the error is not due to a lack of compositional skill.
We use Gemini-2.5-PP
We compare four models: RFT Base (after Stage training). RFT Level 2 (after Stage 2 training on Level 2 problems with RFT), RL Level 1, and RI Level 2, all from previous sections, The latter three models are all trained from the RFT Base.
RFT Base
Level
43.8%
50.0%
Fig.6 reveals substantial similarities in the failure pat- terns of RFT Base, RFT Level 2, and RL Level mod- els. Their failures are dominated by ignoring the com- position entirely (all 550%) and misunderstanding the compositional structure (all >35%).
12.9
Cames
riseplus Trom
In contrast, the RL Level model demonstrates funda- mentally different behaviors. It completely eliminates "Ignores Composition" errors and correctly solves 28.1% of the problems. Crucially, its primary fail- ure mode becomes "Atomic Error." This shows that compositional RL not only improves accuracy but teaches models to parse and execute compositional plans, shifting failures from high-level misunderstand- ings to lower-level execution errors. See |T for exam- ples of different model responses.
15.1%
Figure 6: Distribution of failure modes on Level 3 string tasks. An "atomic error" counted only when the model success- fully resolves the compositional structure of the problem (i.e., it neither ignores the compositional operators nor misinterprets the nesting). Consequently, models that lack sufficient compositional capability do not have their atomic errors revealed or counted.
TAKEAWAY 5
Rather than merely improving accuracy, RL on compositional problems fundamentally trans- forms the model's 's behavior, enabling it to cor- rectly understand and handle compositions.
5 CONCLUSION
The debate over whether RL can teach LLMs new skills has been clouded by experiments bench- marks where LLMs already perform well, using coarse-grained metrics that obseure the learning of new capabilities. By stepping back to cleaner, more controllable experimental environment, findings provide clear and optimistic answer: RL can teach genuinely new and powerful skills when the training task properly incentivize composition.

---

Published as a conference paper at ICLR 2026
Our work focuses the paradigm where RL acts upon pre-existing atomic competencies, mirroring the standard practice of carrying out RL based on pre-trained models. While our results show that possessing atomic skills is sufficient condition for RL to unlock compositional capabilities, do not claim it is strictly necessary. However, given the inefficiency of random exploration discover- ing complex atomic behaviors from scratch, argue that the synergy between supervised learning for atomic skill acquisition and RL for compositional generalization represents the most commonly adopted and promising path. Our findings suggest that the pessimistic conclusion that RL does not learn new skills may stem from inappropriate evaluation setups rather than fundamental constraints of RL itself. Future work may investigate the open question of whether RL can scaled to acquire both atomic and compositional skills simultaneously without supervised scaffolding.
LIMITATION
One limitation of this work the reliance on synthetic tasks for evaluation. We deliberately designed our string transformation framework to enable rigorous experiments with well-defined atomic and compositional skills, thus isolating causal learning mechanisms. However, our synthetic tasks may not fully capture the complexity and nuance of real-workd reasoning scenarios where skills are less clearly delineated and compositional structures are more varied. We acknowledge that demonstrat- ing these findings in natural reasoning domains, such as mathematical problem-solving, code gen- eration, or scientific reasoning, represents an important direction for future work. Extending these findings to realistic applications remains a valuable open challenge.
ACKNOWLEDGEMENTS
This work is supported by NSF Grant No. CHE2505932, an Amazon AICE Award, gift funding from A12, and grant from Coefficient Giving.
REFERENCES
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable fectiveness of entropy minimization in Ilm reasoning. ArXiv, abs/2505.15134, 2025. URL
https://apt
782149
John R. Anderson. Acquisition of cognitive skill. Psychological Review, 89, 1982.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. 2016
IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 39 48, 2015.
Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang. Nilabjo Dey, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li. Richard Fan, Jian- shu She, Chengqian Gao, Abulhair Saparov, Haonan Li, Taylor W. Killian, Mikhail Yurochkin, Zhengzhong Liu, Eric P. Xing, and Zhiting Hu. Revisiting reinforcement learning for LLM rea- soning from A cross-domain perspective. CoRR, abs/2506. 14965, 2025. doi: 10.48550/ARXIV, 3506.14963 URL ItLhtts:: http:///ooooooo..//////o 48550 (wt.5///aXii.... 2506. 14965
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455. 2025a.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models. CoRR, abs/25555.22117 2025b. doi: 10445500ARRIV.25052220
URL
httpp:://dot www.oogg.......oo...ooooo 48550 http//aiiivv... www..i............. 22617
Gabriel V De La Cruz Jr, Yunshu Du, and Matthew E Taylor. Pre-training with non-expert human demonstration for deep reinforcement learning. The Knowledge Engineering Review, 34:e10, 2019.
10

---

Published as a conference paper at ICLR 2026
Google DeepMind. Gemini 2.5:
Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. ArXiv, abs/2507.0626.. 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jun-Mei Song, Ruoyu Zhang, Runxin Xu. Qihao Zhu, Shirong Ma, Peiyi Wang, Xiaoling Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao. Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bing-Li Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dong-I.i Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin. Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Jiong Cai. Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, M. Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang. Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan. Runji Wang, R. J. Chen, Ruiqi Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shao- Kang Wu, Tao Yun, Tian Pei. Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wen-Xia Yu, Wentao Zhang, Wangding Xiao, Wei An, Xiaodong Liu, Xi- aohan Wang, Xi aokang Chen, Xiaotao Nie, Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li. Xiangyu Jin, Xi-Cheng Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li. Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yao- hui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yi Xiong, Ying He, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu. Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yu-Jing Zou, Yujia He, Yunfan Xiong, Yu-Wei Luo, Yu mei You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yan- ping Huang, Yao Li, Yi Zheng, Yuchen Zhu, Yunxiang Ma, Ying Tang, Yukun Zha, Yuting Yan, Zehui Ren, Zehui Ren, Zhangli Sha. Zhe Fu, Zhean Xu, Zhenda Xie, Zhen guo Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zi-An Li, Ziwei Xie. Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. Nature, 645, 2025.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and T. Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. ArXiv, abs/2304.06767, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur' elien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cris tian Cantón Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits. David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Geor- gia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joc Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju-Qing Jia, Kalyan Vasuden Alwala, K. Upasani, Kate Plawiak, Kegian Li, Ken-591 neth Heafield, Kevin R. Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuen ley Chiu, Kunal Bhalla, Lauren Rantala- Yeary, Lau- rens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pa- supuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Has- san, Naman Goyal, Narjes Torabi, Niko lay Bashlykov, Nikolay Bogoychev, Niladri S. Chatterji,
11

---

Published as a conference paper at ICLR 2026
Olivier Duchenne, Onur cCelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasié, Peter Weng, Prajwaal Bhargava, Pratik Dubal, Praveen Krishman, Punit Singh Koura, Puxin Xu, Qing Hc, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ron nie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sa hana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shao- liang Nie, Sharan Narang, Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhos- ale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujiwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Vir ginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whit ney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag. Yashesh Gaur, Yasmine Babaci, Yiqian Wen, Yiwen Song. Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakipos, Aaditya K. Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury. Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Ben- jamin Leonhardi, Po- Yao (Bernie) Huang, Beth Loyd, Beto de Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichten- hofer, Damon Civin, Dana Beaty, Daniel Kreymer, Shang- Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzm an, Frank J. Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz. Gada Badcer, Geor- gia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Han Zha, Haroun Habeeb, Har- rison Rudolph, Helen Suk. Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina- Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kaixing(Kai) Wu, U KamHou. Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, A Lavender, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli. Martynas Mankus, Matan Hasson, Matthias Lennie, Matthias Reso, Maxim Gro- shev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko. Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Mun ish Bansal. Nandhini San- thanam, Natascha Parks, Natasha White, Navy ata Bawa, Nayan Singhal, Nick Fgeho, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab. Pavan Balaji, Pe dro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollár, Polina Zvyagina, Prashant Ratanchan- dani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li. Rebekkah Hogan, Robin Battey, Rocky Wang, Ro- han Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe. Soumith Chintala, Stephanic Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Kumar Gupta, Sung-Bae Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang,
12

---

Published as a conference paper at ICLR 2026
Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Mon- tanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xia Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Ilama herd of models. ArXiv, abs/2407.21783, 2024.
Jerry A. Fodor and Zenon W. Pylyshyn. Connectionism and cognitive architecture: A critical anal- ysis. Cognition, 28:3-71, 1988.
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D. Goodman. Stream of search (sos): Learning to search in language. ArXiv, abs/2404.03683, 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, nathan lile, and Noah D. Goodman. Cognitive behaviors that enable iaffovipgo reasoners, or, four habits of highly effective stars. COLM, 2025.
Andre He, Daniel Fried, and Sean Welleck. Rewarding the unlikely: Lifting grpo beyond distribution sharpening. arXiv preprint arXiv:5566023355 2025.
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. CoRR, abs/1604.002$$ 2016.
Nathan Lambert, Jacob Daniel Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouba Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Christo- pher Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hanna Ha- jishirzi. Tülu 3: Pushing frontiers in open language model post-training. COLM, 2025.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. CoRR, abs/2505 24864, 2025a. doi:
24864.. URL http::://dooooooooo
10. 08.44855/axX11 www.xiv..250... 24864
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. COLM, 2025b.
OpenAl. Openai ol system card. ArXiv, 2024.
OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's 's cube with a robot hand. CoRR, abs/1910.07113, 2019. URL http:// /ttt:i///axiv.../ www.orgg///..... abs/ 1910.111... 07113
Jiayi Pan, Junjie Zhang, Xingyao Wang, Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://gthub.ccm//aiiiiaaaau Panniiyee///in 2025. Accessed: 2025-01-24.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hanna Hajishirzi, Pang Wei Koh, and Luke S. Zettlemoyer. Spurious rewards: Rethinking training signals in rlvr. ArXiv, 10s42506.10947,0000 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song, Mingehuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, a0332400.033000, 2024.
Andrew Silva and Matthew Gombolay. Encoding human domain knowledge to warm start rein- forcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 5042-5050, 2021.
13

---

Published as a conference paper at ICLR 2026
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre. George den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lilli- crap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529(7587):484 489, 2016. doi: 10. N0.108/NATUREI600
URL
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforce- ment learning algorithm. CoRR, abs/1712.01815, 2017. URL pett://arxiie wwgp///.oorgg////oo///o// 1712. 01815
Yiyou Sun, Shawn Hu, Georgia Zhou, Ken Zheng, Hanna Hajishirzi, Nouha Dziri, and Dawn Xi- aodong Song. Omega: Can Ilms reason outside the box in math? evaluating exploratory, compo- sitional, and transformative generalization. ArXiv, abs/2506. 1880, 2025.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg. Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu Wang. Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforce- ment learning. Nat., 575(7782):350-354 2019. doi: 11001//41156-0019122222 https: http///do.og/....//// 1038/ 41586
URL
019 1724-z
Yiping Wang, Qing Yang. Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.2557, 2025.
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base 1lms. arXiv preprint arXiv:2506.1 14245, 2025.
Fang Wu, Weihao Xuan, Ximing Lu, Zaïd Harchaoui, and Yejin Choi. The invisible leash: Why RLVR may not escape its origin. CoRR, abs/2507.114433 2025a. doi: (0.48555//RXXV.2507 14843. URL httt:p:://ao..or//100 211111..4555/aa/
2507.14843
Haoze Wu, Cheng Wang, Wenshuo Zhao, and Junxian He. Mirage or method? how
model-task alignment induces divergent rl conclusions.
2025b. URL hhtpps://api.
Mingqi Wu. Zhihao Zhang, Qiaole Dong. Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, et al. Reasoning or memorization? unreliable results of reinforce- ment learning due to data contamination. arXiv preprint arv:v22507 200020057...32... 2025c.
Tian Xie, Zitian Gao. Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing LLM reasoning with rule-based reinforcement learning. CoRR, abs/a/02. 14768, 2025. doi: 10.4855/ARXIV.2500211 14768. URL https: 2502. 14768]
Fangcong Yin, Zeyu Leo Liu, Liu Leqi, Xi Ye, and Greg Durrett. Learning composable chains-of- thought. ArXiv, abs/2505.22635. 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Ving Ma, fa-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: open- source LLM reinforcement learning system at scale. CoRR, 14b.//2553.14766 2025. doi: 10. 48550/ARXIV.2500 14476. URL httpstt::: http:/otto.oo/////////
www.orgl11... 48550 rxxvv 7503.14476
14

---

Published as a conference paper at ICLR 2026
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does inforcement learning really incentivize reasoning capacity in llms beyond the base model? ArXiv, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 20o: Investigating and taming zero reinforcement learning for open base models in the wild. CoRR, http/2503.188992 2025. doi: 10.48550ARRXI.2555..... 18892. /tttsss:://ao 10. 10.48550/axx1v.2555 2wt.l/..55..1.8899.
Rosie Zhao, Alexandru Meterez, Sham M. Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RL post-training amplifies behaviors learned in pretraining. CoRR. abs/2504.07912, 2025. doi: 10.4550/ARXIV:550000999 URL https:://oll: org/10.
011500/arxiv.204..000
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in llm reasoning. arXiv preprint arXiv:2506.01347, 2025.
15

---

Published as a conference paper at ICLR 2026
A TRAINING DETAILS
A.1 STAGE
All models in our experiments, except for the Multi-Bass variants in $4.3 start from the same Stage base model. The process for creating this model is detailed below.
Data Generation. We first generate 50k Level-1 problems by randomly sampling function and string input (length 3 to 10). For each problem, collect 10 responses from the model (temperature 1.0, max length 8192). To focus training on
Llama-3. 1..ma-3.1-8--- BL-8B-Instrus problems the model finds non-trivial, we discard any problems where the base model achieves 100% accuracy. We then collect all remaining correct responses to form the SFT dataset. Before fine- tuning, remove the function definitions from the prompts, keeping only the function identifier and the input. This results in around 16k training instances.
Rejection Fine-Tuning. We fine-tune the model for 2 epochs with learning rate of 2 10-5 and global batch size of 128. Other hyperparameters follow the default settings provided by the veRI. framework
A.2
STAGE
Data Generation. The problems used in Stage 2 are created using similar process in Stage 1, with the critical differences being that function definitions are hidden and problems can involve composition. For Level-2 problems, prompts for Stage I vs. Stage 2 can be found in Appendix C For the Level only and Level 2 only dataset, we create 50k problems. For the Level 1+2 mixed configuration, we create 25k problems for each level and combine them.
randomly select and compose two functions. Example
Reinforcement Learning. For RL experiments, we use DAPO et. 2025) ) as the optimization algorithm. We enforce strictly on-policy setup by setting both the training batch size and mini- batch size to 16. For each prompt, we generate 16 rollouts using a sampling temperature of 1.0 and maximum response length of 8192. During training, we filter out any problems for which all rollouts are correct or all are incorrect. We use a learning rate of x 10 and set the coefficients for both KL divergence and entropy loss to O.
Rejection Fine-Tuning For the iterative RFT baseline, we use a learning rate of 2 x 10-5 and batch size of 128. The process iterative: the model from the previous iteration used generate a new set of rollouts. These rollouts are then filtered for correctness to construct a new SFT dataset, following the same procedure used in Stage 1. The model then trained on this new dataset to produce the model for the next iteration.
B EVALUATION DETAILS
String Transformation Prediction Task. For the string transformation prediction task, the evalu- ation set consists 256 randomly generated problems for each level from to 8, resulting a total of 2048 test problems. During evaluation, we generate responses using a sampling temperature of 1.0 and maximum response length of 8192 tokens.
Countdown Task. The evaluation set for the Countdown task consists of 128 problems for each level. The evaluation setup is the same as evaluating on the string transformation prediction task.
C EXAMPLE PROMPTS FOR STAGE AND STAGE 2
Here we present the example prompts used in our experiments, Fig. shows the prompt used for Stage data generation, where the function definition is provided to guide the model. However,
https://github. wwwp://wwwcooooooooooooonoeeeeee ne/ver
16

---

Published as a conference paper at ICLR 2026
You aze civen a code:
Euno__ 15
**"Remove adjscent deplicate characters (comprese repeate). 82
resalt
in ph reauit
return 10in
dar main_solurionn
sotaon 1610_16110)
Can you predict outgut "nein_valu valluiion iui theas")" withous wei triiing any uode? Please resxon and put your fital answez Collowing jaon Fornals (Moulpuc") Kyout oulpuul) wheae Kyour outiput> should tinal string.
Figure 7: Example prompt for Stage I response collection. Note that here the function definition is given, but during Stage 1 RFT, the function definition removed.
Example prompt for Stage and Stage 2 Level training You given
def maia_aolucion())) retarn tuno_161x)
Can you predict the output of "mainn w wishout writing any codo? Ploase reason and put your final ansvee in the rollowing 35on format: ("ouspus": eyour sutgut> where eyour output> should be the final string.
Example prompt for Stage Level training You are çiven code:
del main (an(s); sebasn Funs_ (fu),_16(s), 3)
Can you predict the outgut train solusion wowioootteieessseee without writing any code? ease reaean and put your ansvez follswing 15on format: (Mouspus") syour outgut>], eyour output> final string.
Figure 8: Example prompts for evaluation and Stage 2 training.
during the actual Stage training (and throughout Stage 2). this definition is removed, forcing the model to rely on the internalized atomic skills associated with the function identifier. Fig. g shows the prompt used for Stage and Stage 2 training.
D A COMPLETE LIST STRING TRANSFORMATION FUNCTIONS
Here we provide the full list of string functions we use. Note that we replace the function names with meaningless identifiers such func.1 in experiment.
def deterministic shuffle wtwpaaiiissttsssseeeeeeeeeeeee (s)
**"Reorder characters using a fixed multiplier permutation.
L = (sn 1f == 0:
return
multiplier
while ged (multiplier, L) !=
multiplier +-
return " (i * multiplier) & 1.] for i in range (L))
def repeat_str n): "*"Repeat the string exactly n times."ns
return
def menove_vowels. """Remove vowels from the string. unn vowels "aeiouλBIOU"
return join (ch for ch s not vowels)
def sort_chars (s)
a*ngort the characters in the string.""
17

---

Published as a conference paper at ICLR 2026
return " "ijoin (sorted (s))
def reverse_words (s)
"*"Reverse the order of words in the string. """ words s.split()
return join (reversed (words))
def add prefix pre):
**aadd fixed prefix the string."" ""*
return pre +
def add suffix suf):
"""Add fixed suffix to the string."" ""*
return + suf
def interlace_str 82)
"""Interlace two strings character by character (iterative)
result - []
Lenl, 1en2 Len (sl), Len
for i in range (max (ienl, 1en2)):
it lenl:
appult (s1 (i])
if < 1en2:
result apsult. (i])
".join (resuit)
def rotate_str (s, n):
"*"Rotate the string by positions using slicing. """
if not s: return
n n (s)
return a[n:] + s[:n]
def nirror mirror_str(s): """Append the reversed string to the original. ""o return + s[::-1]
(s)
def alternate_casol (s):
*""Alternate the case of characters (even-index lower, odd-inde indixndex upper)."" "". return " jjojn (ch. if $ else ch.upper() for i, in enumerate(s))
def shift_charass, shift): ***
Shift alphabetical characters by fixed amount (wrapping around).
Non-letters remain unchanged.
def
if fa' <= ch <= 'z';
return chr (chr((ch) - ord('a') + shift) $ 25 ord((a()) elif 'A' <= ch <= 12':
chr (chr ord('A') shift) 26 (rA())
return ch
return " ".join (shift_ char (ch) for ch in s)
18

---

Published as a conference paper at ICLR 2026
def vowel_to_nunber (8)
"""Replaco vowels with numbers: a/A->1, c/E->2, 1/I-23, o/0-24, u/u ->5."mm
mapping = ('a': '1', 'e': '2', <if: 13 fo': '4', fu': '5', <Ar: ( ', 'g': '2', "I': '3', 'o": '4', ror: (5") return join (mapping. (ch, ch) for ch in s)
def Lnsert_sepaae (sparatoos, sep): ***Insert fixed separator between every two characters. ""* return sep.join(s)
def duplicate every every_char cs)
"""Duplicate every character the string.""" """
return " (chn 2 for ch in s)
def fancy_brackets (s)
"*"Enclose each character fancy brackets. return " "oonn""ee"" + ch + "yy" for ch in 8)
def compress seappress_repeatteeeee (
"*"Remove adjacent duplicate characters (compress repeats).
if not 8: return
result [s[0]]
for ch s[1:]:
if ch != result[-1]]
result.appent (ch)
return ' jojoonn (result)
def recursive_reverse
**muecursively reverse the string."" ""*
if "";
return
return
+ s00]
def coop_concat (s, n): "*"Concatenato s with itself n times using a loop.
result = for in range(m): result return result
def rotteerotate(s. n):
***Rotate the string using while 1oop (n times) uun count
while count < and s:
= s[l]] s[C]
count += 1
return
def rocuralve_ s2):
*#"Recursively interlace two strings character by character.
if not al s2:
return sl 82
retura s1|0] + s2]0] + Secureive_interriie (s1[1:), s2[1:])
19

---

Published as a conference paper at ICLR 2026
def Loop filter, wtw.eerrnoaapaaaaaae (
"*"Remove non-alphabetic characters using an explicit loop. "** result "" for ch in 8: if ch.isalpha() (): result ch return result
def verity_even (eeeiy_the___gghhh
***
Verification operator: if the length of s is even, return sy
otherwise remove the last character.
return if (s)
else s[:-1]
def backchain (s, depth):
Backtracking operator: deterministically transform contains a digit.
Applies a fixed sequence of transformations recursively.
has_digia digit (t): def has
return any (ch. isdigit for ch in t)
transformations =
lambda t: t "1"
lambda t: "2" + t,
lambda t: r.replace "3"),
lambda ti t[::-1],
1
def helper (t, d) if hts_digit
return
if -- 0:
return None
for trans in transformations:
new trans (t)
res helper (new_t, d 1) if res is not None: return res return None
result helper depth)
return result result is not None else
def backchain_pallnddoon (s, depth): ***
Back chaining: try to transform s into a palindrome.
If is not already a palindrome and depth permits, append its reverse and try again.
if s[::-1);
return s
if depth <= 0: return
new_s 8[::-1]
return backchain www.aiirnppaiidoommmmmm (new_s, depth 1)
note: the string concatenation, i.e., 44 is also considered as function]
20

---

Published as a conference paper at ICLR 2026
Listing All functions
E EXAMPLE FOR COUNTDOWN TASK
Level 2 Example
Using numbers 42), create equaticn equaie use arithmetic operations the final
number only once, your <think> </think> tage, example
Level 3 Example
using tae 195, create equation equals And the "y
be your </thinko </answar>
Figure 9: Example prompts for Countdown task.
F MODEL RESPONSE EXAMPLES
The following sections contain the complete, unfolded responses from each model for the case studies.
F.1 FULL RESPONSES TO THE LEVEL 2 PROBLEM
RFT Model Response
To predict the output of imain_solution(" we need to understand how the function 'func_7' interlaces two input strings. [Authors' Note: It's not wrong, but misses another function f2]
Here's is step-by-step breakdown:
'fune: 7(s1, s2)' is called with "vptqj" the first string 'sl" and ""nar"' the second string 's2".
The function iterates over the maximum length between the two input strings. In this case, the maximum length is 4, which is the length of "narr".
The function appends characters from both strings at each index 'i' to the 'result" list. an index is out of range for a particular string, it simply skips appending a character from that string.
4. So, the 'result' list will contain characters from both strings, interleaved.
Here's 's the interleaved sequence:
Index 0: "v" (from "vptqq"") ') and "n"" (from ""nar"", 'result ['v', 'n']'
Index 1: "p" (from "vptqj"") and "'a" (from ""nar"", so 'result = ['v', 'n', 'n','p, 'a'l'
Index 2: "(" (from "'ppaii" and "r" (from ""nar""), so 'result = ['v', 'n', 'p', 'a,, '. "T
Index 3: "q"" (from "'ptqii") and " (from ""nar""), so 'result = ['v', 'n', 'a.,'a' ".'.. 'q"""l
Finally, we pass this 'result" list the 'join()* function to form the final interleaved string.
The final output of "main "aain_solution"naa""" will be: ""ynpartqj"". [Authors' Note: End too early, there's still one function that hasn't been analyzed.]
So, the JSON output should be: ("output": "vnpartqi")
21

---

Published as a conference paper at ICLR 2026
RL Level Model Response
To predict the output of 'main solution("nar")) need to understand how the function 'func_7' interlaces two strings character by character.
Here's s a step-by-step breakdown:
1. 'func_7' takes two strings 'sl' and 's2' as input.
It creates an empty list 'result' to store the interlaced characters.
3. It calculates the lengths of 's1" and 's2' and stores them in 'lenl" and 'len2', respectively,
It loops from '0' to the maximum of 'lenl' and 'len2' using a 'for' loop.
5. Inside the loop, for each iteration 'i', it checks if 'i' is less than 'lenl' and 'len2'. it
appends the 'i-th' character 'sl" and 's2" to the 'result' list, it exists.
6. After the loop, it returns the 'result" list string by joining all its elements using "" ".joon(resultt'.
Now, let's analyze the 'main solution' function:
'def main solutsoouuion():: return (func_7('vptgj', x) + fune func_2('xbh'))
Here, 'sl' is "vptqi" and 's2' is 'x', which is ""nar" in this case.
Now, let's interlace the two strings character by character:
' [s] 1s2
result'
0 I'v' I'n' [['v,, 'n']
1 1''p' I'a' [['v,, nn, "t',''p'aaa
2 't' ['''''''[[v'' nn,, p 'a', 't''r))
4 [[[",, 'n', ".'''' "',
3 ''l',''['',,''' t, [''qq]
5 Ilv'''n', "p', "2,,"",' ".
6
[ll'''' "a","", 1., ""',',
The loop iterates up to the maximum the lengths of 'sl' and 's2', which is this case. After the loop, 'func_7 7' returns the 'result' list as a string by joining all its elements.
The 'result' list is TV,, 'n', 'p', 'a', '. '., 'q', "TI.
The 'func_2* function is used to remove vowels from string. It takes string 's" input and returns a new string with all vowels removed.
In the 'main solution' function, 'func 2' is used to remove vowels from the string "xbh'' but the output of 'func_7' is used as the input for 'func_2' [Authors' Note: the model mistakes the relationship between the two functions]. However, in the solution function, 'fune 2' is not actually used with the interlaced string but with the string "xbh"'
The "main solution' function returns the result of ffun 7(vpptqj', 'nar') func 2('xbh')
Since 'fune 2 not used with the interlaced string, the output of 'main solution("nat will be the interlaced string 'IV", 'n', 'p', 'a', 't, 'r, 'q', 'jl'' as string, which is:
'vnpatrqj"
So, the output of 'main solution("nai is: ("output": "vnpatrqi")
22

---

Published as a conference paper at ICLR 2026
RL Level Model Response
To predict the output of 'main mainnsolutionn"na""" we need to understand what the function 'func_7' does and how it is used in main.solution'.
'func_7 7' interlaces two strings character by character. It takes two strings 'sl' and 's2' input and returns new string where characters from 'sl' and 's2' are alternately appended.
In 'main 'main solution', 'func_ 7 is called with "vptqi" as the first string and the input 'x' as the second string. The result of 'fune_7' then concatenated with the result of "unc"2('xbh')'
Now, let's consider what happens when 'mainosolution" is called with "nar" the imput.
First, we need to call 'func 7 with "vptqj" and ""nar" as the two strings.
The characters in "vptqj"' are "v', 'p', 't', 'q', "J"
The characters in ""nar" are "n', 'a'," 'r""
Now, let's interlace these characters using 'func_7'.
The maximum length of the two strings is 5. So, 'fune 7' will iterate up to 5 times.
Here's what happens during each iteration:
1. i [slf]]] 'v' and 's2[i] 'n' The result list is 'I'v', 'n']'.
1= 'p' and 's2 = 'a'". The result list II'v', 'n', a''''
". 't" and *s2[i] 'r" The result list is '['v', 'n', 'p" 'a', 't' 'r..
4. = ='q" "q""an and 's2 = .". The result list 'P'V,, 'n', 'p', 'a', ' 'r', 'q''..
and 's2[i] .. The result list is '('v,, 'n', 'p', 'a,, 't', ' 'q', J..
The final result list after iterating over all characters from both strings is LVV,, 'n', 'p', aa, T. T 'q', J..
Now, we need to find the characters in "xbh" and remove them from 'func 2', The characters "xbh" are "x', 'b', "h"
Now, let's consider what 'func_2" does. This function removes all vowels from a string.
The input string 'func_2" 2 is "xbh' The vowels in "xbh" are (there vowels in "xbh"').
However, we need to remove all vowels from "xbh".. Since there are no vowels in "xbh"', the string remains the same.
The function 'func 2' will return "xbh" is.
Now, let's consider what happens when we concatenate the result from finn_7(Cvptqj" "nar")" with (inc_2((xbh")) As explained earlier, 'func_7('vptqi", "nar")" will return "vnpatrqj"".
Now, since 'func 2('xbh') just returns "xbh'', the final output of 'main solation("nar"' will be the concatenation of "vnpatrqj" and "xbh".
23

---

Published as a conference paper at ICLR 2026
The concatenation of "vnpatrqj" and "xbh't is "vnpatrgjxbh". So, the final output of "main soaation"na"""""" is "vnpatrgjxbh". Here's the JSON output:
("output": "vnpatrqixbh")
F.2 FULL RESPONSES TO THE LEVEL 3 PROBLEM
RFT Model Response
To predict the output of "mainnsolutionn"tooek. let's analyze the code.
1. The 'main solution' function calls 'func_14 with the input string 'x'. 'func_14 takes a string "s" and returns a new string where each character's case alternated based its index.
Even-index characters (i.e., those at 0, 2, 4, etc. indices) are converted to lowercase. Odd-index characters (i.e., those at 3, etc. indices) are converted uppercase.
3. The input string 'x" to 'main-solution' is ""ttoek"..
Now, let's apply the logic of func 14" to the input string ""took"":
Index 0 is even, so 'h' is converted to lowercase: 'h'. Index is odd, so 't" converted to uppercase: 'T". Index 2 is even, so 'o" is converted to lowercase: 'o'. Index 3 is odd, so is converted to uppercase: 'E'. Index 4 is even, 'k' is converted to lowercase: 'k'.
The resulting string is "hToEk"'
Therefore, the output of 'main polution("htoek") is: [Authors' Note: Ends early, there's 's still one function that hasn't been analyzed]
("output": "hToEk")
RL Level Model Response
To predict the output of "main www.ssiinnnooooeet"" we need understand what the 'func_10" function does and how it is used in 'main_son "
'func_10" takes string 's" and returns new string where:
Even-indexed characters are converted to lowercase Odd-indexed characters are converted to uppercase
'olinosolution' calls 'func_10" with the string 'x' after applying the 'func.2 func- tion, which removes all vowels from 'x'. [Authors' Note: the model fails correctly understand and parse the nesting function]
Now, let's apply these steps:
1. 'func_2' removes vowels from ""took"", resulting in "htk"". 10 is applied to ""htk"..
24

---

Published as a conference paper at ICLR 2026
"h' is at index 0 (even), so it becomes 'h" (lowercase),
'I" is at index (odd), so becomes (uppercase).
becomes "o" (lowercase). -e" at index (odd), so it becomes 'E" (upperease).
"o" is at index (even),
'k' is at index (even), becomes 'k' (lowercase).
The resulting string ""ToEk"".
However, the question asks us to predict the output without writing any code, so we might have mistakenly included the characters from the original string "htoek"" instead of the output of 'func 2'.
To correctly predict the output, we should first apply 'func_2' to ""took"". The vow- els in "htock" are "o" and "c". Removing these vowels leaves us with "htk"
Now, applying 'func. 10' to "htkk":
-'h' is at index (even), so becomes 'h' (lowercase).
"t' is at index I (odd), becomes 'T' (uppercase).
- 'k' is at index (even), so becomes 'k' (lowercase).
The resulting string is "hTT"".
This is the output of 'func 10("htk"))' which is what 'main solution' returns. Therefore, the output of
is "hTk".
The final answer is: ("output": "hTk")
RL Level 2 Model Response
To predict the output of "mainnssluttonn(("teee)")) let's analyze the code.
1. The 'func_2' function removes all vowels from given string 's'. The 'func 14' function takes string "s" and returns new string where every character in 's" is duplicated. The 'func_10" 10 function iterates over the characters in the input string 's'. makes the characters at even indices lower case and the characters at odd indices upper case.
Now, let's see what happens when we call "main.solution("htceel
First, need find the output of ffune 2("htoek"").
The string "htoek" contains the characters "h", "t", "o", "e", "k". All characters that are vowels (i.e., "o" and "e") will be removed from the string.
After removing vowels, the string 'func.2("htoek") will contain only the characters "h", "t", "k"
Now, we need find the output of "func_14("htoee""
The characters in "htock" are "h", "t", "0","""""" "e", "k"
Applying 'func_ 14 to each character:
"h" becomes "hh"
25

---

Published as a conference paper at ICLR 2026
"I" becomes "tt" "o" becomes "oo" "e" becomes "ee" "k" becomes "kk"
The string "func_14("tooek" will be "hhttooeekk""
Now, we need to find the output of "uncc1l0(hhttoooeekkk"" We will iterate over the characters in "hhttooeekk" along with their indices.
The characters in "hhttooeekk" and their indices are:
'"h" index
- -"h" at index
-"t" at index
-"t" at index
-"o" at index
-"o" index
""e" at index 6
""e" index 7
"k" at index 8
"k" index
Now, let's apply the function 'func_10* "hhttooeekk".
The character at index ("h") will become lower case because the index is even. So, "h" becomes "h".
The character at index ("h") will become upper case because the index odd. So, "h" becomes "H".
The character at index ("t") will become lower case because the index is even. So, "t"
becomes "t".
The character at index ("t") will become upper case because the index is odd. So, "("
becomes "T"
The character at index 4 ("o"" will become lower case because the index is even. So, "o"
becomes "o".
The character at index 5 ("o") will become upper case because the index is odd. So, "o"
becomes "O".
The character at index 6 ("e"") will become lower case because the index is even. So, "e"
becomes "c".
The character at index 7 ("c") will become upper because the index is odd. So, "c"
becomes "E"
The character at index 8 ("k"") will become lower case because the index is even. So, "k"
becomes "k".
The character index ("k") will become upper case because the index odd. So, "k"
becomes "K"
The final string func 10("hhttoocckk"" will be "hHrToOeEkk".
Now, let's call "main_solutionn"hteed
Since 'main_solution(x)" calls (imne_10ffun__1((tnn203)))) need to find the out- put of ffuncc14(fune (tncc first.
We already found that 'func 2("htoek") will contain only the characters "h", "t" "k"
Now, let's apply 14 "htk".
26

---

Published as a conference paper at ICLR 2026
The characters in "htk" are "h", "t", "k".
Applying 'func. 14' each character:
"h" becomes "hh"
"t" becomes "tt"
"k" becomes "kk"
The string 'func 14("htk")) will be "hhtrkk".
Now, we need find the output of (0thh_k(""hukk"")
The characters in "hhttkk" and their indices are:
- "h" at index 0
""h" index
"'t" at index 2
-"t" at index
''k" at index 4
-"k" at index
Now, let's apply the function 'func_10" to "hhttkk".
The character at index 0 ("h") will become lower case because the index is even. So, "h" becomes "h".
The character at index ("h") will become upper case because the index is odd. So, "h" becomes "H"
The character at index 2 ("t") will become lower case because the index is even. So, "t"
becomes "t".
The character index 3 ("t") will become upper case because the index is odd. So. "("
becomes "T"
The character at index 4 ("k") will become lower case because the index is even. So, "k"
becomes "k".
The character at index 5 ("k") will become upper because the index is odd. So, "k"
becomes "K"
The string 10("h_k((h will be "hH(TkK".
Now, we need to call 'main "oainssooutioo"""to""""""
As 'main solution(x) calls 'func. 10(func 14(func 2(x))), the output of
'main ("ltokon""htoek))) will be the output of 'func 10(fune 14(func 2("htoek")))'
We already found that 'func_14("hhtttoccck" will be "HHITOOcERK".
Now, we need to find the output of 'func. "func. 10""H""HTTTTOO"""))
We already calculated this in the previous step. The output of 'func_ "0mhT1(("HHT)))E)EK))) will be "hHrToOeEkK".
However, we need to recall that 'fune 22 removes vowels from the string, So. find the output of 'func 10(func 14(func 2("htock"))) we need to first find the output of 'fune 14(func 2("htoek"))'
already calculated this earlier. The output of 2("htock"))* will be "hhutkk".
Now, need to find the output of "inc__0("hhttk""
27

---

Published as a conference paper at ICLR 2026
We already calculated this earlier. The output of 'func. 10("hhok("h will be "hHt- TkK".
The solution function calls 'fune_ 10(fnnc. 14ffe_44fff 14(func_2(x))). To find the final output of soliiissoution""""tee")) we need to find the output of 'func_ ( (4(ffncc2(("oooocet 2("htoek"))"
The string "htock" has characters "h", "t" "o", "c", "k" The vowels in this string are "o" and "e".
we remove these vowels from "htock", we left with "htk".
Now, let's apply 'func_14" "htk".
The characters in "htk" are "h", "t", "k"
Applying 'func_14 14' to each character: "h" becomes "hh"
"(" becomes "It"
"k" becomes "kk"
The string 'fune_ 14("htk""" will be "hhttkk".
Now, let's apply 'func_10' "hhttkk".
We already calculated this earlier. The output of 'func 10("hhttk"")" will be "hHt- TkK".
The solutisol function calls fune_10(fune 14(unnc 14(()))_(())) To find the final output of "main_son "htoek",""" we need to find the output of 'func. (4(uncc 2("htoek")))
The characters in "htock" are "h", "t", "o", "c", "k". The vowels in this string are "o" and "c".
If we remove these vowels from "htoek", we left with "htk".
Now, let's apply 'func_14" "htk".
We already calculated this earlier. The output of 14mh__1("hdk")) will be "hhttkk".
Now, we can apply 'func 10' to "hhutkk".
We already calculated this earlier. The output of "0"hhkk" will be "hHt- TkK".
So, the output of "mainnssluutoonn(ttee"""" will be "hHrTkK".
The final answer is:
("output": "hHtTkk")
28


Reviews:
[{"review_id": "IliaDpV1QF", "reviewer": "Reviewer_to4c", "summary": "The paper proposes an evaluation of an LLM on compositional coding tasks, mainly to understand what extra value RL actually adds to compositional task performance.\nOverall idea is interesting, but the clarity and the depth of analysis are somewhat limited.", "questions": "1. The writing quality can definitely be improved — there are several typos, some awkward phrases (“perform decently well” or “may remain limited new skills” etc.) that are confusing.\n\n2. Would be nice if the focus didn’t stay only on compositional tasks… maybe discuss generalization to unseen atomic tasks too.\n\n3. The connection to in-context or few-shot learning isn’t clear; authors do not mention how this fits into this line of work.\n\n4. Why only LLaMA 3.1? adding more models could make the work more solid and convincing.\n\n5. It might also help to check if a mix of RL and RFT training gives different outcomes — that could add an extra angle of analysis.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper identifies and designs a coding-based compositional task setup that seems to probe model behaviour in a somewhat controlled way.\n\n2. Experiments do show (at least in some parts) that RL helps improve compositional performance, esp. when moving from simpler to more complex tasks.\n\n3. Discussion of prior work and related efforts is well-placed.", "weaknesses": "1. The novelty feels rather limited — much of the evaluation and task protocol seems adapted from existing works. The only clear “new” thing is using RL for level-1 and level-2 training, which looks like a small extension.\n\n2. Only one model (LLaMA-3.1) is used, and with just 1-2 kinds of tasks (string manipulation and countdown types). That’s too narrow to make strong claims.\n\n3. The plots mostly differ only by acc., avg@16, and pass@k numbers... also why choose k = 1000 ? feels arbitrary.\n\n4. It’s also unclear why RL-based training reduces the basic atomic skill performance — this is quite puzzling and not explained. Figure 6 and its description comes across as this failure mode is \"Acceptable\".\n\n5. No theoretical or mathematical grounding provided, which makes the claims feel a bit unsupported.\n\n6. The paper doesn’t provide a limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0iJTNgxvHO", "reviewer": "Reviewer_oiQ7", "summary": "This paper argues that reinforcement learning (RL) can teach large language models genuinely new reasoning abilities by composing existing skills, not just reweighting old ones. Using a clean string-transformation testbed, the authors show that when models first master atomic skills and then receive RL on compositional tasks, they generalize to unseen and deeper compositions. The learned compositional skill even transfers to other domains like arithmetic reasoning, provided the model already has the basic atomic knowledge.", "questions": "To what extent does model scale influence the emergence of compositionality? It would be helpful to know whether the same behaviors hold for smaller or larger models, as this would indicate whether compositional skill learning is a general property of RL or dependent on scale.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper offers an original and well-grounded perspective on the role of RL in LLMs, framing it as a process of compositional skill acquisition rather than simple reward reweighting. Its experimental design is exceptionally clean and controlled, allowing for causal interpretations. The writing is clear and logically structured, with a thoughtful connection to cognitive theories of human skill formation that strengthens its conceptual depth. The results are robust and systematically analyzed, making a compelling case that RL can produce genuinely new reasoning capabilities.", "weaknesses": "While the paper is conceptually strong, several aspects could be improved to strengthen its empirical and interpretive depth. First, the experimental validation is confined to synthetic string tasks and the Countdown domain which is clean but showing similar compositional learning in more natural reasoning settings would make the claims far more compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q2BFR9qULS", "reviewer": "Reviewer_xBeE", "summary": "This paper investigates whether reinforcement learning (RL) genuinely teaches new reasoning skills to large language models (LLMs). Using a synthetic, contamination-free string transformation benchmark, the authors define “atomic” and “compositional” skills and show that RL with compositional data enables models to generalize to deeper compositions unseen during training. In contrast, rejection fine-tuning (RFT) or RL on atomic data fails to do so. The learned skill also transfers modestly to a different task (Countdown) when atomic skills are present. Behavioral analyses reveal that RL changes reasoning patterns rather than merely reranking outputs. Overall, the paper argues that RL can create new compositional reasoning abilities if properly incentivized.", "questions": "See weakness, thank you", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The question (“does RL truly teach new skills?”) is central and timely.\n\nThe synthetic setup is impressively controlled and eliminates confounders such as data contamination.\n\nClear evidence that RL + compositional data leads to new behaviors unseen in supervised baselines.\n\nStrong writing quality and interpretability of experiments (the “Takeaway” summaries are exemplary).", "weaknesses": "The paper’s analysis mostly rely on synthetic string transformations. While clean, the setting is far from real RL pipelines. There’s no guarantee that the same mechanism (composition of atomic skills) is what happens in open-ended reasoning tasks.\n\nThe paper defines a “new skill” as the ability to compose existing ones, but that might not satisfy the broader notion of novel reasoning or conceptual abstraction. In essence, this could still be pattern recombination rather than true skill learning. It is still possible that these compositionality is stored in the base model, and RL is just a better way to rediscover that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rlN1IkdlZi", "reviewer": "Reviewer_UU3i", "summary": "This paper argues that LLMs can acquire genuinely new skills during RL by composing existing ones. \nUsing a controllable synthetic framework, this paper shows that LLMs compose learned string transformation functions via RL, and such compositional capability can transfer to different tasks. The paper also identifies that the key is to do RL on compositional data which incentivizes the compositionality.", "questions": "1. I wonder how is the conclusions connect with the enropy/diversity view of RLVR?\n\n\n2. What is the specific reason of setting the coefficients for both KL divergence and entropy loss to 0?\n\n\n3. How do you ensure that RL and RFT in stage 2 are in a fair comparison? How are the hyperparameters selected, do both see a same amount of training samples? These questions seemed not to be reported.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Studying whether RL geniunely teaches new skills is a very timely topic.\n\n- The controllable framework plus decontamination procedure enhances the soundness of the conclusion.\n\n- The paper is well-written and easy to read.", "weaknesses": "1. While the the paper provides a good controllable framework to understand what's happening during RL, I wonder what the actionable point is for future works. For example, Takeaway 3 implies that, one could hand-design (or synthesize) some compositional samples manually and controllably (just like the string transformation functions proposed), and train on those samples to incentivize a better compositional reasoning in more practical domains (than the countdown task) like code or math. I hope the authors can comment on this front or add some experimental results (even preliminary) in the paper.\n\n2. The paper claims that RL on compositional data can teach compositional skills, 'if the atomic skills are already learned'. It's not clear that whether the condition is necessary. What if training RL directly on a mixture of atomic data and compositional data (thus without the stage 1)? Would RL be able to learn atomic and compositional skills together? I understand this might make experimental setup more complicated, such as leaking the function definitions, but maybe good to add a discussion on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)