RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?
Authors: Yiyou Sun
Yuhan Cao
Pohao Huang
Haoyue Bai
Hannaneh Hajishirzi
Nouha Dziri
Dawn Song
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Reinforcement Learning
Generalization
Learnability
Primary area: foundation or frontier models, including LLMs

Abstract: It remains an open question whether LLMs can acquire or generalize genuinely new reasoning strategies, beyond the sharpened skills encoded in their parameters during pre-training or post-training.
To attempt to answer this debate, we introduce DELTA — Distributional Evaluation of Learnability and Transferrability in Algorithmic Coding, a controlled benchmark of synthetic coding problem families designed to probe two fundamental aspects: learnability—can LLMs, through reinforcement learning (RL), solve problem families where pretrained models exhibit failure with large enough attempts (pass@K=0)?—and transferability—if learnability happens, can such skills transfer systematically to out-of-distribution (OOD) test sets? Unlike prior public coding datasets, DELTA isolates reasoning skills through templated problem generators and introduces fully OOD problem families that demand novel strategies rather than tool invocation or memorized patterns.
Our experiments reveal a striking grokking phase transition: after an extended period with near-zero reward, RL-trained models abruptly climb to near-perfect accuracy.
To enable learnability on previously unsolvable problem families, we explore key training ingredients such as staged warm-up with dense rewards, experience replay, curriculum training, and verification-in-the-loop.
Beyond learnability, we use DELTA to evaluate transferability or generalization along exploratory, compositional, and transformative axes, as well as cross-family transfer. Results show solid gains within families and for recomposed skills, but persistent weaknesses in transformative cases. DELTA thus offers a clean testbed for probing the limits of RL-driven reasoning and for understanding how models can move beyond existing priors to acquire new algorithmic skills.

Published as a conference paper at ICLR 2026
RL GROKKING RECIPE: How DOES RL UNLOCK AND TRANSFER NEW ALGORITHMSS IN LLMs?
Yiyou Sun', Yuhan Cao; Pohao Huang', Haoyue Bai?, Hannaneh Hajishirzi
Nouha ! Dawn Song't
Universiit California, Berkeley, UUniversity of Wisconsin, Madison, Univerrity of Washington, 4Ai2
ABSTRACT
It remains an open question whether LLMs can acquire or generalize genuinely reasoning strategies, beyond the sharpened skills encoded in their parameters during pre-training or post-training. To attempt to answer this debate, we intro- duce DELTA Distributional Evaluation of Learnability and Transferrability in Algorithmic Coding, a controlled benchmark of synthetic coding problem families designed to probe two fundamental aspects: learnability can LLMs, through rein- forcement learning (RL), solve problem families where pretrained models exhibit failure with large enough attempts (pass@ K=0)? and transferability- if learn- ability happens, can such skills transfer systematically to out-of out-of-distribution (OOD) test sets? Unlike prior public coding datasets, DELTA isolates reasoning skills through templated problem generators and introduces fully OOD problem families that demand novel strategies rather than tool invocation or memorized patterns. Our experiments reveal striking grokking phase transition: after extended period with near-zero reward, RL-trained models abruptly climb to near near-perfect accuracy. To enable learnability previously unsolvable problem families, we explore key training ingredients such as staged warm-up with dense rewards, experi- ence replay, curriculum training, and prrfficatton-in-the-loooo we use DELTA to evaluate transferability or generalization along exploratory, compositional, and transformative axes, as well as cross-famin fanily transfer. Results show solid gains within families and for recomposed skills, but persistent weak- nesses in transformative cases. DELTA thus offers a clean testbed for probing the limits of RL-driven reasoning and for understanding how models can move beyond existing priors to acquire new algorithmic skills. Code is available attps://github: com/ wwc.lllzzeeeebrrlre -orll-roo-eeee recipe
Beyond learnability,
INTRODUCTION
central question for RL on language models is whether it merely sharpens latent skills or enables genuinely new reasoning, Some argue RL only refines existing heuristics embedded in the model's parameters Yuue et al. 2025 Wu al. 2025). while others see it as way to unlock emergent problem- solving (Liu et al. 2025b a). We make this debate testable using two criteria: learnability, which asks if RL. can instill procedure the model could not previously execute; and generalization, which asks if that procedure transfers to diverse Out-of-distributio (OOD) cases rather than memorized patterns. Addressing these questions requires a dataset with tightly controlled train-test splits that systematically probe both properties.
Why controlled problem families matter? Uncontrolled open benchmarks in math/coding (e.g.. Numina-Mart (Li et al. 22244) DeepMath (He et al., (025), OpenCodeReasoningg (NVIDIA (225)) mix topics and difficulty, blurring the line between capability sharpening and genuine acquisition. Controlled synthetic families remove these confounds: can precisely vary distributions and difficulty, attribute gains to specific skills, detect phase transitions, and systematically test transfer to OOD variants.
Main dataset contributor, independent researcher. Indicates equal advising role in alphabetical order.

---

Published as a conference paper at ICLR 2026
RL Learnability Study
Manufactoria
BouncingSim
NEW
Modilc oven 43
Detaet Mandlibbe
Puzzle Rule
Toy Exanple Le
Input colored
Example
Exploration
Dovr Enderns
Putters renove one front color and route by color
Boana:
stapes hesagan,
Phase
Conver-
Grokking!
centers 1250, 250: ordentation"
Pointers add color end
Task: predict position
objectsi
Phase
Toy Exanple
snaper trzengie
Problem: ACCEPT
red
at time t.
posstion: 1254 253)
velocityi 117-32,
RL Generalization Study
START start:
# Solution Code
Screen
Gare
Problen
Explorative Gen.
Trein
Corpositional Gen.
Trein
Test
check
ROTAT BOX NOV BOX
PULLER_AS blue_checks
NULTI_8OX GRAVITY NULT1_001
[R] None
[EMPTY] end
[8] blue check
Competition Coding
Transforatt Ive Gen.
Donoin Generalization
Rcaing Reaning Deaai
Oiecador
END end
Problen Fanilies
Train
Test
Train
Test
Problen Families
SQRT DC
SEOMENT TREE
START
CDO MO_ALG
MEET INDD
Speee serreiaannn
Synthetíc. Coding Problem Scope/Families
Controlled RL Study Studd
Figure 1: Overview of DELTA with controlled RI. studies. Left: Synthetic Programming Problem fami- lies Mess-Manufactoria with custom syntax and puzzle-like rules, BounceSim with physical simulation, etc. Right: Controlled RI. experiments. Top: Learnability shows grokking, where shifts from long exploration sudden convergence, uncovering strategies beyond reference models. Bottom: Generalization extends OMEGA (Sun) et al. 2025) across four axes exss-lorrrttrry. Compositional, Transformative, and Domain-lewell-estingg adapta- tion to harder or recombined tasks.
Why programming problems? GRPO/PPO pipelines typically rely on pass/fail reward: a perfect solution earns +1, anything else earns Guo al. (2025). This sparsity can stall learning on hard families, In math, grading intermediate steps is expensive and hard to scale. Programming, however, naturally supplies fine-grate ine-grained feedback through test cases, which act as dense rewards. A practical approach is to start training with test-case-basee based rewards to encourage partial progress, then transition to binary outcome reward lock in exact solutions. This staged scheme is crucial for helping L.MSs acquire genuinely new procedural strategies, and while coding offers uniquely scalable setting, the underlying insight using intermediate signals before enforcing strict correctness may apply to other reasoning-heavy domains such as math or formal logic.
To address this need, we introduce DELTA, a controlled yet diverse benchmark for programming problems. DELTA consists of synthetic problem families drawn from different domains, each generated from templated problem generators, allowing us to study phenomena such as difficulty scaling, knowledge transfer, and Icarnability in clean and isolated setting.
RL Learnability Study. We reveal an underexplored grokking phenomenon during RL training. While recent works argue that RL cannot exceed the limits of its reference model Yue al. 2025
Wu et al. 2025), our evidence suggests otherwise. On hard problems where the base model achieves pass K Qu standard RL with binary rewards collapses due the absence of positive signals. By contrast, a staged angime-warming pass/fail-first guides exploration into region where full solutions become reachable, then sharpens these into verified completions, producing a long exploratory plateau followed by sudden grokking to near-perfect accuracy (Figure top-right).
with fine-grained proxy rewards before switching to strict
RL Generalization Study. DELTA extends OMEGA's controlled tests along three axes aligned with Boden's 's creativity typology (Boden 1998): (1) Exploratory- extend known skills within family (e.g., hexagon to octagon); (2) ciappoiiiionalccoiiie
previously separate skills (e.g.. bouncing ball with both rotating obstacles and boxes); (3) Transformative disfooomuiivedddcovveer unconventional solutions (e.g., special initial states that guarantee periodicity). Our results show that RL-trained models generalize to harder and composed variants, but performance drops with complexity, and transformative cases remain the most challenging.
Herr @ K refers to a large value of K (e.g., 128). Thus, pass K 0 indicates that the model fails to solve the task after many sampled attempts.
2

---

Published as a conference paper at ICLR 2026
Difficulty Ladder of Manufactoria -iittr-iditt Coding Puzzles)
1.0
gpt-S
04-mini
0.8
1iaude--onne44220250551
0.6
0.4
0.2
0.0
APPEND TEXACT
START
ENDS
REGEX
HAS
COMPR
PREPEND" NUTATE BIT OF
SYMM MINMAX
ADD
accept
Accept tapes
Replace
Accept pattern HARD 'R(n)B(n)R(n) (ns1)
tape
BASIC
subsequence
"GGRBD"
EASY
'RB"
MEDIUM
The Manufactoria difficulty ladder. 14 problem families are grouped into Basic, Easy, Medium, and Hard levels according to average performance across four popular L.Mss. Each test split contains 20-50 problems, and full pass rale are averaged over 4 independent runs.
Figure 2:
controlled dataset (DELTA): We design suite of synthetic programming problem families that isolate reasoning skills, enabling clean tests of learnability (can RL unlock procedures absent in the base model) and generalization (do these procedures transfer systematically to OOD cases). Unlike prior coding or math datasets, DELTA introduces fully OOD problems (Manufactoria) and richly graded rewards, avoiding tool-based shorteuts and data confounds.
Main contributions.
2) Sharpening or discovery, depending on setup: We provide clear evidence that RL is not limited sharpening existing abilities in reference models. On hard families where base models fail (pass@ K=0), staged training with dense-to-binary rewards produces grokking phase transition a sudden leap from failure to mastery - showing that RL can indeed discover strategies the base model could not execute. At the same time, in easier regimes with weaker setups, RL primarily sharpens existing skills. Which outcome emerges depends critically on the reward design, data mix, task hardness, and training recipes.
3) Three-axis generalization analysis: We evaluate how these learned strategies transfer along exploratory, compositional, and transformative axes. Results show strong generalization in exploratory and recomposed cases, but persistent failures in transformative shifts, highlighting both the promise and limits of RL-driven reasoning and the generalization challenges must work
2 DELTA: CONTROLLED PROGRAMMING PROBLEM FAMILIES
We operationalize learnability and generalization with DELTA, controlled suite of synthetic programming families.
From OMEGA to DELTA. OMEGA (Sun et al. 2025 offers 40 synthesizable math families to study exploratory, compositional, and transformative generalization in the spirit of Boden (Boden) 1998). DELTA complements this by shifting to programming, where templated generators yield automatically verifiable tasks with tunable difficulty and clean distributional controls. Compared to OMEGA, DELTA further provides unique benefits and improvements: Novel OOD problem family. Math tasks in OMEGA remain within familiar domains (e.g.. algebra, geometry), which can plausibly appear in pretraining corpora. In contrast, DELTA includes hand-crafted out-of- distribution (OOD) problem scope called Manufactorian which uses entirely novel program syntax and problem-solving strategies. b) Harder shorteut with tools. Many synthetic math items can be solved by executing Python (e.g., computing a matrix rank). In DELTA, the target is the program itself: models must synthesize a correct solution rather than delegate computation to external tools. e) Rich reward signal. Programming enables cheap, graded feedback via per-test case pass rates, which supports staged training (dense reward then binary full-pass reward).
In DELTA, we design problems from five major scopes, illustrated in Figure I problem families in detail.
We next introduce the
2.1 MANUFACTORIA OUT-OF-DISTRIBUUTT PROBLEMS FOR LEARNABILITY STUdY)
Manufactoria is classic Flash game (2010) in which players build automated factories to sort robots based on their colored tape patterns. The underlying logic resembles constructing finite-state automata or tag systems using two special node types (pul ler, pat i nter). While the original game
3

---

Published as a conference paper at ICLR 2026
ROT_OBE BTBBB (Rotating Object)
ROT_BOX (Rotating Box]
MOV_ MOV_BOX (Moving Box)
GRAVITY (rravity Included)
MULTI_BO BBX
MULTI OBJ
Models
(multiple boxes)
(multiple objects)
GPI-S- 52
11
61
47
31
25
12
64
5B
0
25
83
81
db
59
32
36
0
0
52
22
Full Pusle Sale (60
04-Miri
15
57
39
Claddd
2
18
50
32
Owwn3-2 358.
50
0
9
0
45
27
54
32
18
25
0
10
50
Difieulty
tanic
mediom
herd
extrme
Bettc:
eany
medium
Herd
extrme
bansc
eway
meddum
hard
extrame
Bassc
caty
mendum
Rerd
extreme
tassc
4nty
medium
hard
oetremn
basic
eaty
meetium
etteme
Figure 3: Full-pass rate (%) BouncingSim by model, family (ROT-OBJ, ROT_ECK MOV_BOX, GRAVITY,
MULTI BOX,, MULTI.OBJ), and difficulty tier DEASIC-EXTREDE Warmer colors denote higher accuracy; cell values are mean full-pass rates per split over 4 runs on 50 test problems each.
is implemented in 2D space, we formalale it into a custom programmatic syntax, as illustrated in Figure I Details are provided in Appendix A..
Justified OOD-ness. This task is OOD for several reasons: a) The original game solutions were stored only as images on legacy websites. Our converted program syntax is entirely novel and unavailable to any LLM during pretraining; b) We do not reuse existing game challenges. Instead, we design new problem families inspired by the mechanics but synthesized by the authors, and these are entirely unseen to LLMs; c) The puzzle strategies are qualitatively different from conventional tasks. With only two available node types with limited functionality,
programming or solving requires distinctive reasoning patterns not captured by standard coding strategies.
scalable difficulty ladder. In total, we construct 14 synthetic problem families. For example, the family tagged HAS (Figure 2 requires accepting tapes that contain subsequence such as GCRBB, which can be synthesized by using arbitrary color strings. Manufactoria is organized into BASIC + EASY - MEDIUM + HARD tiers, enabling matched studies across model scales. BASIC/EASY families (e.g., START, EXACT) suit small models (e.g., 1.5B, 4B) for learnability, while MEDIUM/HARD families require more advanced insight and appropriate for probing SOTA systems (e.g.. GPT-S-class). Because the syntax and families are novel, Manufactoria also serves as an OOD benchmark for open LLMs, enabling apples-to-apples comparisons with SOTA LLMs on truly novel tasks. Medium tasks expose larger gap: only GPT-5 achieves non-trivial success, while other models collapse near zero. Hard families remain unsolved across the board, underscoring the sharp transition in difficulty and the limits of the current model.
2.2 BOUNCINGSIM (2D SIMULATION PROGRAMMING TASKS FOR GENERALIZATION STUDY)
We include widely used community test-a 2D bouncing- ball simulation program-ofte treated as a proxy for geometry- aware reasoning in LLMs (Wiggers) 2025). The goal is to synthesize a program that simulates elastic collisions in polygonal containers and returns the exact object state at a queried timestamp: strong solutions require precise collision detection/response and numerically stable integration.
Task design. To replace informal, visually judged demos with a rigorous benchmark, we make the task: (a) verifiable-eact prompt specifies deterministic initial state (positions, velocities, container geometry); the program must output the object's location at a target time and is scored against an oracle; (b) symthesizableiinntaaeee
are generated by varying the configuration in Figure [ with ground- -ruth trajectories produced by Box21 (c) composable- ppmposable-ssigle-skeee
families (c.g..
ROT_BOX, ROT_OBJ) can be combined into multi-skill families (c.g., ROT_BOX_OBJ), and (d) difficulty-comtroolledd vary polygon vertex counts, object speeds, box motion, gravity, and the number of objects/boxes to create BASIC--EAYY-MEDDIE H - tiers. Detailed configurations are provided in Appendix A
Generalization axes. To align explicitly with the three generalization axes defined in OMEGA (Sun) al. 2025), as exemplified in Figure a) Exploratory generalization: Training problems feature standard box sizes with relatively sparse collisions, while test problems use smaller containers that induce denser and more frequent collisions. b) Compositional generalization: Training isolates distinct skills handling rotating boxes (ROTAT BOX) and rotating objects (ROTAT 03J). Testing
https://boo2d..rrg
4

---

Published as a conference paper at ICLR 2026
then evaluates the combined scenario simultaneously, requiring the model to integrate the two skills. c) Transformative generalization: Training covers common variants such as ROTAT BOX, but testing introduces qualitatively different dynamics-for example, special initial conditions that yield perfectly periodic bouncing trajecto- ries (e.g., object oscillating vertically with horizontal drift). Further examples and details of these generalization setups are provided in Appendix A Evaluation results. Figure 3 summa- rizes full-pass rates across six families-ROT _O3J, ROT BOX, MOV BBOX, GRAVITY, MULTI _BO,, JULTI-OBJ- and five difficulty tiers for four representative models. GPT-5 leads overall, but accu- racy degrades with difficulty and composition: MULTI_BOX is challenging even at BASIC (~30%) and MULTI OBJ drops sharply-from 800 at BASIC to ~ 100% by MEDIUM. Other LLMs trail substantially-tyiiena
where both the box and the object rotate
<30-40% on the pasy-to-mediun tiers and near-zero on HARD/EXTREME and most compositional settings. Overall, BouncingSSm represent valuable testbed for understanding what these models can and cannot do; whether they reinforce existing skills or discover new ones; by enabling systematic study of learnability and generalization.
2.3 COMPETITION CODING PROBLEM FAMILIES
We add three real-world domains comainns-compettiiv programming, SQL,, and LEAN, Although not strictly OOD (given their online popularity), they remain challenging (c.g.. gpt-5-high reaches only 2% on hard-tier LiveCodeBench- problems into fully controlled families that support learnability and generalization studies. A brief construction overview appears in the main text, with details in Appendix [A] Each family groups problems sharing the same core algorithm (e.g.. Mo's algorithm, CDQ divide-and-conquer)) and is named after that algorithm. For each family, we: (1) gather 5-7 seed tasks verified to use the target algorithm; (2) perturb their contexts by relying on an expert-provided solution strategy and background, then use LLM to change narrative surface while preserving the solution; and (3) filter and verify by requiring brute-force solution to pass all tests, ensuring perturbation consistency, We release 5 families (~500 items each).
(Zheng et al. 2025)). We include them in DELTA to expand seed
LEARNABILITY STUDY: CAN RL UNCOVER NEW STRATEGIES AND HOw TO ACCELERATE IT?
A central debate in recent research concerns whether reinforcement learning (RL) can endow models with reasoning abilities beyond those their base model.
The skeptical view. Yue et al. (2025) argue that although RIVR-
Pass@ Comparison Before/After RL Training
trained models outperform their base models at small & (e.g., A = 1), the base models achieve equal superior pass @k performance when & is large. Their coverage and perplexity analyses suggest that reasoning capabilities are ultimately bounded by the base model's support. Similarly, Wu et al. (2025) provide a theoretical argument that RLVR cannot extend beyond the base model's representational limits.
Pass@k over Test Set (aanufactoria-HAAS
1.0
Clear Evidence:
0.8
RL can uncover stralegies beyond
the limits
0.6
reference models.
0.4
The optimistic view. In contrast, Liu et al. (2025b) demonstrates
After RL (Step-800)
0.2
Owen3 48 Instruch-250
that ProRL can expand reasoning boundaries on tasks where the base model performs corrly-Specifically in letter-formed 2D puzzles
0.0
from Reasoning Gym (Stojanovski et al. 2025).
64
128
(Number of Attempts)
Our contribution: clean testbed and clear evidence for RL enable grokking in LLMs. Existing evidence in favor of RL's gen- eralization often comes from large, heterogeneous training corpora. This makes it difficult to isolate why and how RL might discover novel strategies. To address this, DELTA offers a controlled environ- ment: synthetic problem families that are both out-of diitt--diitiiittionn internally consistent (free data confounds). We focus on the (requiring novel strategies) and training / 100 test instances), where the reference model @wen3-4B --ne------nnttuct---- pass rate at pass @ 128. As shown in Figure R our staged RL training strategies enables the model to fully solve this family, achieving 100% full pass rate. Next, detail how this is made possible.
Figure 4: Pass@k comparison before and after RL training on the Manufactoria-
family (742 achieves 0% full
5

---

Published as a conference paper at ICLR 2026
3.1 BASIC SETUP
Unless otherwise specified, the reference model is Owenn--4B-nnstruct Training and testing datasets drawn from single combined problem families introduced in Section 2 By default, each training step consists of 48 prompts with 16 rollouts. The learning rate is set to 5 x 10 For code training, the default reward signal is full pass, a binary indicator of whether a program passes all test cases. In later experiments, we also consider per-test pass rate as the reward signal, measuring the fraction of test cases passed. A more detailed experiment setup parameter descriptions are included in Appendix B We also provide complementary experiments with alternative model families, sizes, and problem domains in Appendix C.I
3.2 HOW TO SOLVE "PASS@K=0" TASKS WITH RL?
The skeptical position that RL cannot exceed the boundaries of the base model understandable for a simple reason: GRPO (Guo et al. 2025) depends on reward differences across rollouts. If no rollout ever succeeds (as in "pass K=0" tasks), there is no gradient signal to learn from. Indeed, as Figure a) shows, naïve GRPO training stagnates. Thus, the central challenge is:
If no rollout achieves a full pass, how can RI. propagaee a meaningful learning signal?
Strategy (al: RL (GRPO) with Full-pass Rate
Strategy RL (GRPO)
Per-test Pass Rate
Taining Data Full Phrs Rate
Pass (Reward)
Rate
Por-test Pass Rate (Reward)
0.06
0.6
0.4
0.04
0.4
0.02
@
0.2
Trining Steps
400
600
0.
200
400
Training Step
600
800
1000
1200
0.0
8 1.0
After "warm-up" training with
Tra ning Data Full- Poss Rat
Full Pass Rate (Reward)
per-test
0.8
0.06
pass rate, resumed
Exploration
Phase
0.24
Full Pass Rate
0.4
with the full
pass rate
Grokking!
Convergence
0.02
zero non-zero
0.2
82.
Phase
0.00
50
Trining Step
100
150
200
250
0.0
100
200
300
Training Stegs
500
600
700
800
Strategy RL (GRPO)
Phase (Per-test Pass Continued (Full Pass Rate)
Figure 5: Comparison of strategies solving "pass@K=tf K=0" tasks. Directly optimizing for full-pass rate under GRPO fails. (b) Training with per-test pass rate provides a smoother reward but quickly saturates. (c) two-phase training warming up with per-test pass rate, then switching to full-pass reward. All training is performed on Manufactor ia-ettrrriaassss family and the reference model Qwen3-4B -
Per-test pass rate training. One solution is to exploit partial credit. Instead of the all-or-nothing full pass rate (reward 1 only if all test cases pass), we use finer-grained per-test pass rate, a continuous reward in [0, 1]. As Figure (b(b) shows, this signal provides initial learning traction. However, it quickly saturates after ~ 100 steps, and the full-pass rate remains negligible (<0.01%).
Warm-up phase. Even though it can not serve as full surrogate loss, we find that the per-test pass rate can serve as an important warm-up stage that pushes the model out of the all-zero region. As shown in Figure 5 a), this signal allows the model to move beyond the all-zero region; although the full-pass rate remains < 1%, the model begins to accumulate positive gradients.
Exploration and grokking. From this warm-up checkpoint, we switch to RL with the binary full-pass reward. Figure 5 illustrates the dynamics: For 4550 steps, the model remains in an exploration phase, with full- full-pass rate still 1%. After sudden grokking moment, the model discovers the key strategy to solve the family. Training then enters a convergence phase, where RL sharpens and consistently reinforces the successful reasoning path. At convergence, the RL-trained model achieves

---

Published as a conference paper at ICLR 2026
Curriculum Stage 2:
Stage on Target Problems:
Curriculum Stage
Problenoc Manufacttrts REOEX
Probrems: Manufactoria HAS
(Basic Level Problems):
Pull Pass Rate (Seward)
1.0
Training Data Full-Pass Rate
1.0
Verufacttori-TTAlI AFTAT////EN//EAACT
Resumed Training w/ Different
9.8
0.4
Talning Eerr Ful
0.6
0.8
.022
Problems
0.6
Curriculum Learning on Different Problems
0.00
402
005
1200
0.2
0.4
Troblems: Samurectons-IIii
Curriculum Stage 2:
0.0g
Training Steps
Full Pass Rate (Reward)
0.2
Pull Pass Rate (Reward)
0.8
Final Stage on Target Problems: Problems: Monufactoria HAS
0.6
0.0
0.4
Full Pass Rate (Reward)
Training Steps
200
400 Training Steps
600
BCB
0.0
Models first train on basic problems (START APPEND/ /EXACT) before branching into of two intermediate curricula: (i) Stage 2-REGEX, which lcads successful transfer and high pass rates on the target HAS family, or (ii) Stage 2-COMPR, which fails to transfer and platcaus at low performance.
Figure 7: Contrast of the two-stage curriculum learning for
pass@k @k compared to the reference model (Figure 4). We also observe this phenomena with other model families, sizes, and problem domains in Appendix C.I
nearly a 100% absolute improvement in pass
3.3 ATTEMPTS TO ACCELERATE RL GROKKING
natural follow-up question is how to shorten the exploration phase and enable grokking to emerge earlier. We examine the following strategies:
Experience replay. The long exploration phases mainly stem from the sparsity of positive reward signals. A nat- ural way to alleviate this is to retain successful reasoning traces and reinsert them into future rollouts a technique known as experience replay (Zhang et al. 20255), ), closely related expert iteration (Anthony al. 2017). In our experiments, we log successtul traces in each sampling round and, when the same query reappears, append up to three of the most recent successful traces to the rollout. As shown in Figure experience replay does help the model grok at an earlier stage. However, its convergence speed is still slower than the baseline GRPO algorithm, likely because the reused traces are off-policy.
Experience Replay
0.8
Experience Reploy Feesback---theeitrr
0.6
0.4
0.2
0.0
Figure Comparison of training strategies for accelerating RL grokking. "No Trick" denotes the standard training setup in Figure bj) "Experience Replay" logs and reuses successful traces, and "Experience Re- play + Feedback- -eeracck-i-tttee in-the-loop" further injects verifier's 's feedback into the inference.
in-thh-looo. Another plausible strategy is to directly include failure feedback in the generation process, encouraging the model to improve its full pass rate earlier. We achieve this by replacing the EOS token with feedback (e.g., failure test cases) and letting the model continue generating. As shown in Figure applying this beedback-in-the-loe grokking moment. However, it also reduces training stability, likely due to the off-policy injection of feedback tokens. A common failure case is that the model, even after receiving explicit feedback. persists in its original (incorrect) solution.
once can indeed expedite the
3.4 MORE INVESTIGATION INTO THE WARM-UP PIIASE
Selective curriculum learning as alternative. A natural question is whether the warm- up effect can be achieved through curriculum learning across problem families. To explore this (Figure 7}, we designed three-stage curriculum training. After training basic families (START/ APPEND/ /XXACT), models were exposed either to Stage 2-REGEX or Stage 2-COMPR before transferring the target HAS tasks. These two problem families have similar difficulty levels
7

---

Published as a conference paper at ICLR 2026
according to Figure 2 Despite similar difficulty, the outcomes diverge: the REGEX curriculum leads to successful transfer and near- pear-complete mastery of HAS at final RL stage, while the COMPR curriculum fails to progress beyond low pass rates. This difference can be traced to task compatibil- ity-bott REGEX and HAS revolve around detecting or matching subpatterns (e.g. "accept tapes with pattern (BRB) + (RR) vs. "accept tapes with subsequence GGRBB"), whereas COMPR emphasizes numerical interpretation and branching tests (c.g., "treat color B as I and R as 0, accept if the number 2 277).. These results suggest that effective curricula must not only control difficulty but also align structurally with the target family. While curriculum learning can thus be highly effective, its success depends on finding suitably related families to bridge the reasoning gap-gammpplee something that is not always feasible. In contrast, warm-up training with dense rewards remains broadly useful as it does not require additional family design or mixing.
Warm-up Helps Beyond the "pass @ k=0" Regime Even when the base model exhibits a small but non-zero success rate (pass@k= > 0), a brief per-test-reward warm-up improves stability and speed. Empirically, we observe faster and smoother convergence compared to training full-pass from scratch (see Appendix C.2).
Limitation. It is important to note that not every problem family can be "unlocked" by ua training. For instance, as shown in Figure g even when using per-test pass rate rewards, the model fails to escape the all-zero regime on the harder Ma nu iattoooa--rPPPE family. The per-test signal rises modestly but quickly saturates, while the full- pass rate remains stuck at zero throughout training. This suggests that warm-up with per-test pass rate training is not a universal recipe: its effectiveness depends on the model's s capacity and difficulty of the target family.
Werm-up Phose (e): RL (GRPO) with Full-pass Rate
Probiens: Manufactoraa PREPEND (MEDIUM)
Full Pass Rate
0.08
Per-test (Reward)
0.3
0.06-
0.04
0.1
0.02
Stays with all-O
0.00
250
500
750 1000
1250
Training Step
Figure 8: Warm-up training on the harder ManufactoriaaPRaeeM family.
GENERALIZATION STUDY
(a) Training Curve
(b) Explorative Generalization (Before RL)
(c) Compositional Gen. (d) Transformative Gen. (Before AL]
(Basic Level)
(Before RL)
1.0
1.0
Training Data Full Pass Rabe
0.5
0.8
(Atter RI.)
(Attor RL)
(Atter RL)
0.6
0.5
Pres
71
20
0.4 Grokking!
0.65
59
0.6-
0.2
0.2-
20
0.0
200 Training Steps
0.0
ROT. GBJ
ROT BOX
MOX
GRANTTY MULTI BOX MULTI.OBU 16/50 DE/S0E WOUA SONTIC
Figure 9: Generalization Study on BOUNCINGSIM. (a) Training full-pass rate on the Basic-level mixture (6 familics, 1k each) for Qwen3-4B Instruct with binary full-pass reward shows a sharp grokking jump near step 200. (b) Explorative generalization: Before RL (top) the model rarely solves any OOD cases; after RL (bottom) it transfers to Easy/Medium/Hatd variants with diminishing gains as difficulty increases (bars aggregate 6 families 4 tiers; 100 prompts per cell, averaged over 4 runs). (c) Compositional generalization: Zero-shot composition of skills. (d) Transformative generalization: Qualitatively new dynamics (e.g., special periodic trajectories) remain near zero after RI. Results are averaged over runs.
Setup. We study how far the learned programmatic skills transfer beyond the training distri- bution. Unless noted, the reference model is Qwen3-4B-Instruce We train on a Basic-level mixture of six single-skill families-ROT. _OBJ, ROT_BOX, MOV_BOX, GRAVITY, BULTI-OBJ-win 1K instances per family (6k total). Because the base model has non-zero full-pass on some basic instances, we directly optimize a binary full-pass reward (all tests pass) for 300 gradient steps; all other hyperparameters follow Section 3 Evaluation spans three axes exeeloovpraative, compositional, and transformative and reports full pass rate (fraction of prompts for which the synthesized program exactly matches the oracle on all unit tests). For explorative generalization
JULTIBBOX.
8

---

Published as a conference paper at ICLR 2026
consider four difficulty tiers (Basic=ID, Easy/Medimm/HardOO000 crossed with the six families; each bar in Figure 9 aggregates, More detailed setup is in Appendix
Training dynamics (Fig. 9 We again observe sharp grokking phase transition: after long plateau of near-zero reward, performance on the training mixture jumps around the step 200 to 0.7 full pass rate, indicating the emergence of stable simulation code that handles elastic collisions.
Generalization results (Fig. Db-d). RL-trained models transfer beyond the training distribution, but with varying success across axes. In explorative generalization, performance is strong on Basic (ID, 70-85%) and carries over to Easy (50-75%), though gains shrink on Medium (15-50%) and nearly vanish on Hard (single digits). For compositional generalization, the model demonstrates surprising skill integration: unseen combinations such as ROT_BOTT
MOV _BOX +GRAVITY and MULTI BOX+MULTI JBJ achieve 60-70% full-pass (vs. near-zero before RL), in contrast to the weak compositional transfer reported in OMEGA (Sun et al. 2025). We attribute this to coding tasks composing structurally (merging simulation modules) rather than strategically (inventing new reasoning steps). Finally, in transformative generalization, models remain near zero on qualitatively novel dynamics such as perfectly periodic or degenerate trajectories, which demand the discovery of new invariants and align with the persistent difficulty of transformative math generalization.
Takeaways. RI. discovers executable simulators that (i) transfer well to parametric shifts and (ii) compose across skills, but (iii) struggle when the test distribution demands qualitatively different solution schemas. Coding tasks appear more amenable to structural composition than symbolic math, yet transformative "schema creation" remains open challenge. Figure summarizes these trends.
RELATED WORK
Coding benchmarks and synthetic datasets. Human written or collected coding benchmarks like APPS (Hendrycks et al. (02T). CodeContests (Li et al., 2022), HumanEvvll (Chen et al. 2021), MBPP (Austin el al.) 2021 and TACO (Li et al. 2023) established functional-correctness eval evaluation with tests. Synthetic datasets like KodCode (Xu et al. 2025) introduced a large-scale synthetic coding dataset with LLM spanning simple exercises to advanced algorithmic challenges. DELTA builds on this trend on a more fine-grand ane-grained level, generating families of coding problems to isolate specific reasoning strategies and to test learnability and generalization under controlled distribution.
Study on grokking. Grokking (Power et al., 2022) is when a model memorizes small algorithmic training sets and only later suddenly generalizes after prolonged training. Explanations span train-test loss-landscape mismatch [Liu et al. (2022), double-descent via pattern-learning speeds [Davies et al. (2023), and gradient-spectrum splits between slow generalization and fast memorization Lee et al. (2024). h. Beyond traditional neural network settings, small transformers also grok on synthesized graph-based tasks (Wang et al., 2024| Abbamov et al. 2025). Yet most work targets supervised, toy datasets; whether grokking occurs in RL on difficulty reasoning tasks remains unclear. To our knowledge, DELTA is the first to show that, under suitable training, grokking can emerge during RL fine-tuning of large language models.
6 DISCUSSION AND IMPLICATIONS FOR FUTURE STUDY
call to study the hard subset. Recent benchmarks in math and code often report averages over mixed pools (Huan et al., 2025; Guha et al. 2025 Liu et al.) 2025b a), where small subset of genuinely hard instances (those with pass @ for strong pretrained models) is washed out by easier items. Our results show that this "hard frontier" exhibits distinct learning dynamics-mos notably a grokking-like phase transition under RI. that can require hundreds or even thousands of training steps per problem. In large hetcrogencous pools, the probability of repeatedly sampling and solving any single hard case is diluted, further suppressing the signal. We therefore advocate that future evaluations explicitly isolate and track this subset, so that progress on truly novel reasoning is not obscured by aggregate metrics.
Beyond coding: from math to science. Coding affords dense, verifiable feedback, which lets RL cross the learnability gap on problems that were previously unsolved. The same principle can extend math and science when fnnined signals are available: rubric-based scoring, stepwise checkers,
9

---

Published as a conference paper at ICLR 2026
theorem prover verification, and simulation- or constraint- maasee evaluators. We expect that porting the insights to these domains will enable RL to tackle currently unsolvable problems.
How we train is important what train. While scaling data matters for LLM training, our results indicate that the training procedure is equally critical. First, natural, curriculum-styy progression does not always exist for all hard problems in large mixed corpus; adding loosely related families does not reliably smooth learning and can fail to help (see Fig. 7 Second, concrete training sucices-such staged warm-up with dense rewards, experience replay, and verification/feedback -oo- great promise for improving performance on hard-tier problems. More broadly, both sharpening (refining existing priors) and discovery (acquiring new strategies) can both happen; which regime emerges depends on the setup. The right RL infrastructure, reward design, data mixture, and level of task hardness can be decisive together,, these factors let us extract substantially more performance from RI. and reach conclusions that would appear unattainable under different configurations.
ACKNOWLEDGEMENT
This work was supported in part by the National Science Foundation under the ACTION program. Computational resources were provided by the Allen Institute for Al (AI2) cluster. We thank all collaborators and reviewers for their valuable feedback and contributions.
REFERENCES
Roman Abramov, Felix Steinbauer, and Gjergji Kasneci. Grokking in the wild: Data augmentation for real-world multi-hop reasoning with transformers. arXiv preprint arXiv:2504.20752, 2025.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abbs/2108.07732, 2021. URL httpt:ttp///mm http://axxvvvr.o./.. /bs
2108. 07732
Margaret A Boden. Creativity and artificial intelligence. Artificial intelligence, 103(1-2):347-356 1998.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Xander Davies, Lauro Langosco, and David Krueger. Unifying grokking and double descent. arXiv preprint arXiv:2303.061 06. 2023.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Racof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint arXiv:2506.04177 041788 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456. 2025.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. arXiv preprint 2r.iv:2/05.0900) 2021.
10

---

Published as a conference paper at ICLR 2026
Maggie Huan, Yuetai Li, Tuney Zheng. Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general Ilm capabilities? understanding transferability of Ilm reasoning. arXiv preprint arXiv:2507.00432, 2025.
Jaerin Lee, Bong Gyun Kang, Kihoon Kim, and Kyoung Mu Lee. Grokfast: Accelerated grokking by amplifying slow gradients. arXiv preprini arXiv:2405.2023: 2024.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https://hugging
co/ MO0/A--N0//ummnaMMat----11 (attp:://qithus com http://dch.u.ccooooooorptttooaaa
aimo aemo-pprorress--tiee//bboo http:///ww.obbmm..i...a/a http:///wwwwnnnn///teeeteeoooeee numaat/ lll__dataseetee pdf)) 2024. 2024,
Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang. Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14833 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-lews code generation with alphacode. arXiv preprint
2022.
Mingjie Liu, Shizhe Diao, Jian Hu, Ximing Lu, Xin Dong, Hao Zhang, Alexander Bukharin, Shaokun Zhang, Jiaqi Zeng, Makesh Narsimhan Sreedhar, et al. Scaling up rl: Unlocking diverse reasoning in Iims via prolonged training. arXiv preprint (tti::2507.120000 2025a.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint wtX.:x2555.5286666 2025b.
Ziming Liu, Eric Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01111 2022.
NVIDIA. Openccodereasoonnng-22 Dataset on Hugging Face, 2025. URLnttts://hugggggaaa License: CC-BY-4.0.
co/ lo//attattt//vdddaa///oooeeeeeee
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- cralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02117, 2022.
Zafir Stojanovski. Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakcem Adefioye, Jean Kaddour, and Andreas Köpf. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards, 2025. URL
24760
Yiyou Sun, Shawn Hu, Georgia Zhou, Ken Zheng, Hannaneh Hajishirzi, Nouha Dziri, and Dawn Song. Omega: Can llms reason outside the box in math? evaluating exploratory, compositional, and transforratiie generalization. arviv preprint arXiv:2506. 18880, 2025.
Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey the edge of generalization. CoRR, abs/2405. 2024. URL http:// dblp.
de de/da
aaauunnallll.a wttp/ /ttp://orr//ccccoo/// aasp:///www......aaaaa.a..a...ae 200oss-2245--000 -
Kyle Wiggers. People are benchmarking ai by having it make balls bounce in rotating shapes. TechCrunch, 2025. Retrieved from http:///checmmmmcocmo//////11//2 bennherarr-beecchkiiiggg. hnt.rriiaae--bll---oo----e---ieo
Fang Wu, Weihao Xuan, Ximing Lu, Zaid Harchaoui, and Yejin Choi. The invisible leash: Why rlvr may not escape its origin. arXiv preprint MXXi:25507.1485:3 2025.
Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. arXiv preprint arXiv:2503.0m 02250000.09 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint 2r5i 13837, 2025.
11

---

Published as a conference paper at ICLR 2026
Hongzhi Zhang, Jia Fu, Jingyuan Zhang, Kai Fu, Qi Wang, Fuzheng Zhang, and Guorui Zhou. Rlep: Reinforcement learning with experience replay for Ilm reasoning. arXiv preprint arXiv:2507.07551, 2025.
Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, et al. Livecodebench pro: How do olympiad medalists judge Ilms in competitive programming? arXiv preprint arXiv:2506.11922 2025.
12

---

Published as a conference paper at ICLR 2026
A DATASET DETAILS
A.1 MANUFACTORIA
Manufactoria is classic Flash game (2010) in which players build automated factories to sort robots based on their colored tape patterns. The underlying logic resembles constructing finite-state automata or tag systems using two special node types (puller, lez, painter). While the original game is implemented in 2D space, we re-formalize it into a custom programmatic syntax, as the syntax defined as a prompt below.
Prompt Template of Manufactoria Problems
Manufactoria Solution
Domain Specific Language describing Manufactoria puzzle solutions in text format.
## Overview
Manufactoria is puzzle game where you build automated factories to sort robots based on their colored tape patterns. Robots enter your factory carrying sequences colored tape, and you must route them to the correct destinations based on the given criteria.
## Game Mechanics
### Robots and Tape
**Robots**: Each robot carries a sequence colored tapes
**Tape Colors**: Primary colors are Blue (B) and Red (R), with additional
Yellow (Y) and Green (G) for advanced puzzles
**Tape Representationn:: Sequences
represented strings
(e.q., 'RBRR', "BBR", or empty string **m")
### Operations
**Pull**: Remove tape from the front of the robot 's sequence
**Pain***: Add colored tape to the end of the robot's 's sequence
**Route**: Direct robots through the factory based on their current tape state
### Objective
Route robots to the correct destinations based on their final tape configuration and the puzzle requi **Accepted**: Robot reaches the END node
**Rejected**: Robot is routed to the NONE node, or caught in an infinite
lcop, or robot reaches the END node but fails to meet the puzzle's acceptance criteria
DSL Syntax
### Program Structure
Every solution must start with "START" directive and end with an "END" directive, wrapped in ""manufactoria ... "., ...""
mmmanfaactoria
START start:
NEXT Lndxt_node___ id>
Factory logic goes here
13

---

Published as a conference paper at ICLR 2026
END end
### Node Types
***# 1. Puller Nodes
Pullers remove specific colors from the front of the robot's 's tape sequence and route based on the current front color.
**Re//Blue *
manufactoria
PULLER RB
[R] <next_node_ide id>
Route and remove color if front tape is Red
[B] <next_node_idi
Route and remove color if front tape is Blue
[EMPTY] idd
Route if no tape or front tape is neither R nor B
YYellow/Green Puller:**
manufactoria PULLER_YG YG <node_id>:
[Y] <next_node_id
Route and remove color if front tape is Yellow Route and remove color if front tape is Green
[G] <next_node_id>
[EMPTY] <next_node_idt ids
Route if no tape or front tape is neither nor G
*NNot**: Unspeci fied branches default to "NONE", which rejects the robot.
#### 2. Painter Nodes
Painters add colored tape the end of the robot sequence and continue to the next node.
mannfaactoria PAINTER_RED <node_id>; NEXT cnext_node_idr idd
PAINTER_ BLUE <node_idr _id: NEXT <next_node_idk
PAINTER_YELLOW <node_id>: NEXT <next_node_ict
PAINTER_GREDT <node id>: NEXT <next_node_id>
## Syntax Rules
1. **Node IDs**: Must be unique identifiers (alphanumeric characters
and underscores only)
2. **Comments**: Lines starting with " are comments (single-line only)
3. w.Indentatione:: Use consisten spaces tabs for route definitions Colors must be uppercase (R, B, Y, G)
4. **Case Sensitivitys:;
5. i:Termination+::
Robots routed to "NONE" are rejected
Robots routed to the END node are accepted sccetee((bjeettive_clauss
14

---

Published as a conference paper at ICLR 2026
Final factory code should be wrapped in triple backticks with ... markers
6. **Code Blocks**:
## Example
H a simple example that accepts robots with exactly one red tape (ending tape should be empty):
""manufactoria
START start:
NEXT entry
PULLER_RE RB entry: [R] end
END end
Task
Your task is design factory with code with following functionality:
(criteria)
The End of Prompt
The criteria are defined in the Table | with different problem families.
Problem Family
Difficulty
Criteria Examples
APPEND
BASIC
Accept any input and append the sequence RBR the end of the tape.
EXACT
Accept if the tape is exactly RBB. Accept the tape starts with BR. BASIC
BASIC
START
ENDS REGEX
EASY
Accept if the tape ends with BB.
EASY
Accept if the tape matches the regex pattern (RBR) + (B) ex- actly.
HAS
EASY
Accept if the tape contains the substring RYY (must be consecu- tive).
COMPR
EASY
Treat Blue as and Red as 0. Accept the binary number is greater than or equal to 13.
PREPEND MUTATE BIT OP
MEDIUM
Put the beginning of the tape.
MEDIUM
Change all RB to BR sequentially.
MEDIUM
Treat Blue and Red 0. Apply bitwise OR with 16 to the binary number.
FDIV
HARD
Treat Blue as 1 and Red as 0. Apply floor division by 4 to the binary number. Accept strings that match the pattern ([n]B(n+1] for any
SYMM
HARD
1.
MINMAX
HARD
Treat Blue I and Red as Output the maximum of 11 and input.
ADD
HARD
Treat Blue as and Red 0. Apply add 8 to the binary number.
Table 1: Manufactoria Problem families with difficulty levels and acceptance criteria.
We synthesize each problem family by starting from a parameterized template (alphabet, tape operation(s), acceptance predicate, and any numeric thresholds) and then perturbing those parameters
15

---

Published as a conference paper at ICLR 2026
within constrained search space. Discrete knobs (e.g., start/end substrings, regex subpattems, token rewrites) are toggled or swapped with near neighbors, while numeric knobs (e.g., bitwise constants, comparison thresholds, additive offsets, division factors) are jittered by small deltas drawn from valid set (e.g., +1, +2, powers of two), with guards to keep the task well-posed and nontrivial.
A.1.1 EXAMPLE TRACES COMPARISON OF MANUFACTORIA HAS PROBLEM
Here one example response of Qwen3-4B-Instee on a problem in MANUFACTORIA-HAN "Ac- cept if the tape contains the substring BRRR (must be consecutive)".
Failure Case
To design **Manufactoria solution** that **accepts robots whose tape contains the substring 'BRRR'** (consecutive), and **rejects therwise**, we need to carefully build factory that:
**Detects** the presence of the sequence "BRER" in the robot 's tape. Only accepts robots when this exact pattern appears **somewhere** their tape.
Rejects all others (including those with "BR", "BRRR" but not full "BRER", or other combinations).
Understanding the Requirement
We are looking for **consecutive** "BRRR" that is, the letters followed by three R's in row.
We can't just look for "B" and then three "R's separate steps | since the robot has a sequence, and we must detect **this exact substring*t.
## Strategy
We will use **state-based pattern matching** using **pullers** to simulate finite state machine.
We '1l define states that track how close we are to matching "BRRR";
State
Meaning
'S0"
Start:
looking for the first 'B'
'S1"
After seeing 'B': now expecting R
"S2"
After seeing 'R': now expecting R
'S3"
After seeing RR: now expecting (final match)
"S4"
Match found: accept (go to END)
*55"
Failed to match (hit any invalid character or didn't complete) reject
We 11 use **PULLER RB** nodes (since only and are used) to process the tape from front to back.
We want to **build machine that scans** the robot s tape until it finds "BRRR"
Once the substring is found, we route the robot to the **NN
it fails, we route it to *rejection node** (via "NONE").
16

---

Published as a conference paper at ICLR 2026
## Design Steps
1. **Start** node sends to scanning state.
2. Use a series of pullers to detect:
If the front is "B", go to next state ('S1')
If the front is "R", stay in current state or fail.
If the tape is empty or neither, reject (fail).
3. After each 'R", advance state.
4. If we ever get "BRRR", go to "END".
5. If we see anything else (e.g.r 'G', Y,, non after "B"), reject.
Because the only tape colors in this puzzle are ***** and * **do not need** yellow or green.
So only need **PULLER_RBE RBx*.
Implementation
manufactoria
START start:
NEXT s0
S0:
[B] sl
[R] s5
[EMPTY] s5
81:
[R] s2
[B] s5
[EMPTY] s5
82:
[R] 83
[B] s5
[EMPTY] s5
83:
[R] end
[B] s5
[Y]
85:
NONE
END end
## Explanation
State | Action
------ [--_===]]
"s0": Start | Wait for "B" 4 go to 's1" If "R" or empty + reject ("s5)..
"sl": After 'B" Expect R go to "s2". If "B" or empty fail reject.
17

---

Published as a conference paper at ICLR 2026
" After 'RR' Expect another go to "s3". Any other fail.
"s3": After 'BRR' If next is "R', we have "BRRR", go to **END**. Otherwise fail.
"s5": Failure state All invalid wrong transitions "NONE" **rejected**
**No alternative paths**
accepted.
>
This correctly *oonly passes robots with "BRRR" in their tape**, and rejects:
"BRR" (missing last R)
>
"BBRR" (fails at 'B" after 'B')
"BRER" (breaks the sequence)
> Any tape with 'B" not followed by at least three 'R's
Empty or non-matching tapes
## Example Test Cases
Input Tape Accepted? Why
------------
"BRRR" Yes Found "BRRR" reaches "end"
"BBRRR" Yes | First "B" goes to "sl", then "R" "22", "R" 4 "s3', "R" "end"
"BRBR" No | After "B", then 'R', then "B" + fails (not 'R") reject
"BRR" No No final "R" rejected
"RRB" No No "B" at start rejected
(empty) No Empty rejects
'RER" No No "B" at start, never starts sequence reject
Works as required.
## Final Solution
""mannfactoria
START start:
NEXT s0
s0:
[B] sl
[R] s5
[EMPTY] s5
81:
[R] s2
[B] s5
[EMPTY] TY] s5
82:
[R] 83
[B] s5
[EMPTY] s5
s3:
[R] end
[B] s5
[EMPTY] s5
85:
NONE
18

---

Published as a conference paper at ICLR 2026
END end
This solution correctly accepts robots with the substring rejects all others.
** 'BRRR ** and
The End Trace
Here another example response (partial) of Qwen3-4B-Instrucc after RL training with grokking (as in Figure (c)) on test problem in MANUFACTORTA-HASS "Accept if the tape contains the substring BRRR (must be consecutive)".
Success Trace
We are tasked with designing factory the **Manufactoria DSL**
that:
*Accept*** robots whose tape contains the substring ** BBRRR ** (four censecutive characters: B, R, R, R)
jeces** all others.
This is a **string matching** problem via finite automata. Since the tape is processed from left to right (front to back), and we are only allowed to use pullers (which remove from the front), we must simulate finite state machine (FSM) that tracks progress toward detecting "BRRR".
We can't assume the input is long ordered exact substring ** "BRRRR"*** ** in the input sequence, **anywhere**, and accept if and only if it appears.
have detect the
must design **automaton** that:
Keeps track how much "BRRR" it has matched so far When it sees match, it goes to an accepting state Otherwise, continues scanning, possibly resetting or continuing partial matches
We do **oot** allow adding tape (painting), only pulling (removing from front), and routing based on the current front character.
## Step Define state transitions
will build the automaton with states:
-s00" initial state
-ss1" after matching B
-ss2" after matching BR
-s33" affer matching BRR
-s4" matched BRRR ACCEPT
19

---

Published as a conference paper at ICLR 2026
Now, what about the tape that is exactly "BRRR"? ?
s0
sl
- s1 + R 82
s2 83
s3 + R go to "end"
What about if the tape is 'R'?
s0 + R s0 + then R + s0 forever never reaches end 4 rejected + correct.
What about "
s0 + sl then if next is s2 etc.
So yes.
What about "BBR"?
- BB + sl
s2 then if no more, it stops never accepts correct, because BRRR not present.
But what about "BERRR"? yes, after seeing second B + sl then s2 + s3 R + end Accepted
So it works.
Are there any *smissing** cases?
What if the tape starts with Y?
30 + y goes to ygo + Y + s0 4 same so eventually if B comes, we can proceed.
Eventually, if "BRRR" appears later, we will catch it.
What if "BRRR" appears in the middle? Like "X BRRR Y
X + if s0 then B sl s2 R s3 + R end
Perfect.
We never require the tape to start with B just that BRRR appears anywhere.
So it's correct.
## Final Answer
'"'manffactoria
START start: NEXT s0
PULLER RB s0:
[R] s0
[B]
20

---

Published as a conference paper at ICLR 2026
[EMPTY] ygo
PULLER YG yg0:
[Y] 80
[G] 80
[EMPTY] s0
PULLER RB sl:
[R] s2
[B] sl
[EMPTY] yg1
PULLER YG ygl: [Y] s0
[G] s0
[EMPTY] 80
PULLER RB s2:
[R] s3
[B] sl
[EMPTY] yg2
PULLER_KE YG yg2: [Y] s0
[G] s0
[EMPTY] 80
PULLER RB s3:
[R] end
[B] s1
[EMPTY] yg3
PULLER YG yg3:
[Y] s0
[G] s0
[EMPTY] s0
END end
This factory accepts any robot whose tape contains the substring "BRRR"" (consecutive), and rejects otherwise.
This satisfies the requirement.
The End of Trace
21

---

Published as a conference paper at ICLR 2026
A.2 BOUNCINGSIM
We include widely used community test-a 2D bouncing-ball simulation program- treated as a proxy for geometry-aware reasoning in LLMs (Wiggers (025). The goal is to synthesize program that simulates elastic collisions in polygonal containers and returns the exact object state at queried timestamp; strong solutions require precise collision detection/response and numerically stable integration.
Prompt design. We provide prompt example of the bouncing ball coding problems in ROT BOX problem family below.
Prompt Template of BouncingSim Problems
Polygon Dynamics Prediction
In this task, you will implement a single function predict_position (t) that computes the 2D positions of all balls at an arbitrazy future time under idealized mechanics. The function parses the scene configuratio (containers, balls, and physics/metal and handles boundary collisions with finite-size treatment, and returns list where each element is the [x, y] position (rounded to 2 decimals) of ball at time t. Each evaluation of must be computed directly from initial conditions and scene mechanics with no hidden state or accumulation across calls. Rendering, animation, and explanatory text are out of scope; prefer closed-form reasoning and avoid coarse time- stepping except where narrowly required for collision resolution.
reconstructs the motions, detects
### Mechanics (General)
Kinematics: Use closed-form equations under constant acceleration:
(t)==x0+x0++++0 5xax*t (tt)=y0+yy++++055+a) 5+aay
Collisions: Perfectly elastic. Reflect velocity using vt = dott (v, "n) where n" is the inward unit normal at the contact. Finite size: Use polygon contact. Derive reqular shapes from ('sides', 'radius', 'center', 'rotation') irregular convex polygon balls use provided vertices.
Geometry: Trregular convex polygons (if present) are simple (non self- intersecting). Ball finite size must be respected in all interactions. Units: Positions in meters; time in seconds; angles in radians; velocities in m/s; accelerations in m/s"2. Cartesian Axes: +X is right, +Y is up.
### Constraints
Implement only predict_position (t); no other entry points will be called. No global variables; no variables defined outside the function. Do not import external libraries (except math); do not perform I/0; do not print; do not use randomress.
Numerical output must be round (value, 2); normalize -0.0 to 0.0.
### Verification and output contract
Return list of positions per ball for the provided [xx1,y1,, [x2,y21,...) Each call must be computed independently (no state carry-over between calls). You should assune that the ball will hit the wall and bounce back, which will be verified in test cases.
### Scene description
#### Containers
Container 1: reqular polygon with 3 sides, radius 225. 00m, center at (750, 750); initial orientation 0. 000 rad; constant angular velocity 0.170 rad/s
Objects
22

---

Published as a conference paper at ICLR 2026
Ball 1: reqular polygon (3 sides), radius 40. 0m, initial position
(750, 750), initial velocity (-220. 61, 6. 2121) m/
### Physics
effective gravity (treated zero)
#*# Dynamics
No additional time-varying mechanisms.
#*# Conventions for this scene
Containers are convex regular polygons (parameters: 'sides', 'radius', 'center'), unless otherwise specified.
Angle baseline: By default, the initial orientation is 0.000 rad, pointing to the first verrex along +X (standard Cartesian axes);
positive angles rotate CCW about the container center.
Pollygn vertices (if provided) are CCW and form a simple convex polygon.
Container 'radius' denotes the circumradius (meters). For balls: irregular convex polygons rely on provided vertices (no radius mentioned); reqular polygons may be derived from httees//ddddusceeetr/ttteeet
Containers are kinematic (infinite mass, prescribed motion); impacts not alter container motion.
### Task
Number of balls: 1
Your should think step by step and write python code. The final output should be in the following format:
[Your thinking steps here .... (optional)
.** python
[Your Python code here]
Define
(t) returning list of length n n_balss; each
element is [x_i, y_i] (rounded to 2 decimals) for Ball at time t (seconds)
### Output
Required format: function
(t: float) -> [1x1,y1),
[x2,y2],... coordinates as 2-decimal floats
The End of Prompt
We construct large-scale dataset for elastic collisions of polygonal objects in polygonal containers, designed to probe geometry-awand reasoning and numerically stable simulation code-generating models (Wiggers 2025). Each instance provides fully specified physical scene and a programmatic task: predict the exact object state at one or more queried timestamps. Below we detail our scene taxonomy, generation and validation pipeline, prompt/evaluate protocol, and the difficulty schedule.
A.2.1 SCENE TAXONOMY
We factor the space of scenes into orthogonal "axes" that control distinct physical effects or composi- tion, allowing systematic sampling and compositional generalization:
ROT-OBJ (Inner rotation):
the ball (modeled convex polygon) has nonzero angular velocity; collisions remain perfectly elastic.
BOX (Outer rotation): the container rotates; optionally, time-varying angular speed injected via sinusoidal envelope.
ROT-BOX
the container follows prescribed path (sinusoidal or Lissajous), inducing moving-bounde reflections.
MOV BOX (Outer translation):
GRAVITY; gravity be tiny/small/large, tilted, chaotic (random direction with time variation).
23

---

Published as a conference paper at ICLR 2026
MULTI-BOX Multii-ontaiiee):: multiple non-overlapping polygonal containers are placed; single ball is spawned in the first container unless otherwise specified.
MJLTT_OBB (Multi-object): multiple balls are spawned in single container with non-overlapping initial placement.
All containers and balls are convex polygons; collisions use perfectly elastic model (restitution 1.0) with finite-size handling (ball centers are constrained by the container's incircle).
A.2.2 PARAMETERIZATION AND PLACEMENT
Scenes are defined in a global, display-agnostic metric space. The workspace size fixed to 1500 m 1500 m with baseline container diameter of 300 m. Difficulty scales the geometry (e.g.. container diameter factor), polygon arity (number of sides), ball radii, speeds, and multiplicities. Objects are sampled and placed under strict feasibility constraints:
Non-overlap: initial ball-ball overlap is rejected by a circle- atrreeapprooimmation test; multi multi-container layouts must respect a minimum center-to-cente gap.
Feasible incircle: ball centers sampled inside the container's incircle minus safety margin; scenes violating this bound are rejected.
Units: positions in meters; time in seconds; angles in radians; velocities and accelerations in SI units. All randomization seeded and stored scene metadata for reproducibility.
A.2.3 GENERATION AND VALIDATION PIPELINE
The dataset produced in three stages, repeated for every requested problem family combination and difficulty level:
(1) Scene synthesis. Given target problem family set (e.g., ROT ROT.BOX) and difficulty, draw parameters from faatlemm-fmiil-sseecfiii ranges (polygon arity, speeds, rotation rates, translation amplitudes, gravity modes) and write normalized JSON scene: container(s), ball(s), physics (including time-varying profiles), and comprehensive metadata (difficulty name, seed, key timestamps, etc.). Difficulty levels scale geometry (container factor, polygon arity), ball radii, kinematics (linear and angular speeds), gravity complexity, and multiplicity (containers/balls) as shown in Table 2
(2) Numerical sanity check. Each synthesized scene is validated for step-size stability before acceptance. We simulate the scene a small set of reference timestamps under two integrators/tine steps (a validation baseline the ground-truth step) and require the maximum screen-space deviation to remain below a tight threshold (15 px). Scenes that exceed this threshold or violate geometric feasibility (overlap outside-incircle are discarded and resampled up retry budget.
(3) Dataset assembly. For every accepted scene we choose evaluation timestamps and compute ground-truth positions using the higher-fidelity integrator. We then construct a task prompt and serialize a JSONL entry containing: messages (the task), a list of test assertions (per timestamp), instance id, difficulty index, the explicit timestamp list, and error tolerance tag (default 50px) used during automated checking.
A.2.4 SPLITS AND COMPOSITION
We design three complementary splits to probe distinct generalization properties. Each split is parameterized by which axes, difficulties, and timestamp regimes are exposed during training vs. evaluation.
Design principles. (1) Factorized skills. Axes isolate orthogonal mechanics (inner vs. outer rotation, moving boundaries, gravity, multiplicity, periodicity). (2) Controlled distribution shifts. Difficulty scales geometry, multiplicity, and dynamics; OOD splits increase complexity without changing the core mechanics.
24

---

Published as a conference paper at ICLR 2026
Table 2: Problem- Problem-by-diitee by-difficulty configurations (aggregated from generalor defaults), Abbreviations: f container diameter factor (relative to 300m base); out/in outer/inner polygon sides; ball radius (m); v = linear speed range (m/s); w = angular speed (rad/s); amp translation amplitude (m): g = gravity mode; cts number of boxes; = number of balls.
Problem family Basic (0)
Easy (1)
Medium (2)
Hard (3)
Extreme (4)
30T.08J
f1.5; out 3-4; 3-4;
1114; out 3-5; in 5-6;
f1.3; 3-6; 6-7;
1122; 3-7; in 7-8;
f1.0; 3-7; in 8;
r 40; w 0.1-0.2; v
r 35; w 0.2-0.5;
r 30; w 0.5-1.0;
r 30; w 1.0-2.0 (tv):
30; w 2.0-2.5 (tv);
200-400
600-600
600-800
v 66008800
600-800
B0T_BOX
f L.S: out 3-4; in
f 1:4: out 5-6:
in 5-6; 0.2-0.5;
f 1:3: out 6-7;
in
f 1.2: out 7-8: in
f 0.8: out 8-10; in 8-10; w 2.0-3.0
3-4; w 0.1-0.2;
v
6-7: 0.5-1.0;
7-8; 1.0-1.5
(tv):
200-400
400-600
600-800
800-1000
1000-1200
ROX
115; out 3-4; amp
1.4; out 5-6; amp
out 6-7; amp
f 1.2; 7-8; amp
1.0: amp 90-120: Lis-
8-10;
0-10: sinld (0.1): v
20-40: sinld (0.5):
40-60: sinld (1.0):
60-90: Lissajous;
200-400
400-600
400-800
800-1000
sajous (chaotic); v 1000-1200
GRAVITY
1.5; out 3-4; g = tiny; v 200-400
11; out 5-6; g small; v 400-600
f 113; out 6-7; g large; 600-800
f 1.2; out 7-8; g tilted; 800-1000
1.0; out 8-10; g úilee: v 1000-1200
MOLTI_BOX
cts 2; 1.5; out 3-4;
cts 2; 1.4; out 5-6,
cts 3; 1.3; out 6-7:1
cts 4; f 1.2; out 7-8,
cts 6, 1.0; out 8-10;
40; v 200 400
35; v 400-600
30; v 600-800
25, 800-1000
r0; v 1000-1200
30T_BALL
f; out 3-6;
3: f2.5: out 3-6:
n 4-5; f25:; out 3-6;
n 5-6; 225:; 3-6;
n f.5; out 3-6;
3-6; 20; 200-400
20; v 400-600
20; 600-800
20; 800-1000
20; v 1000-1200
Explorative generalization (within- difficully shift). This split tests robustness increased geometric/dynamic complexity while keeping the same "skill". We train on single-family scenes at Basic difficulty and evaluate on the same family at higher difficulties.
Train: single-family scenes at Basic (0). We generate 1000 examples in such split.
Test (ID): single-family scenes at Basic (0). We generate 100 additional examples in such split.
Test (OOD): Easy- -este-- (1-4) at the same family; We generate 100 additional examples in each difficulty.
Rationale:
isolates the effect of tighter geometry (smaller containers, more sides), higher velocities, stronger/tilted gravity, and larger multiplicity (more containers/balls) while holding the family- specific mechanics fixed.
Compositional generalization (skill composition). This split probes whether models learned modular skills that compose. Concretely, we exemplify by composing inner and outer rotations at test time after training them in isolation.
Train: GOTAT_BOX (outer rotation only) and ROTAT _OBJ (inner rotation only), both at Basic difficulty. We generate 1000 examples each family.
Test (OOD composition):
SETAATBBXXORRRR (outer+inner rotation simultaneously) at Basic (0) level. Container angular velocity and object spin are drawn independently at the current difficulty level. We generate 100 additional examples in such a split.
Rationale: assesses whether learned collision handling in rotating frame combines with inner-spin kinematics without interference.
Transformative generalization (qualitative strategy change). Here the test-time data is qualita- tively different from anything seen in training-foo -for instance, perfectly periodic trajectories that arise from special construction.
Train: single-famill scenes at Basic (0). We generate 1000 examples in such a split.
Test (transformative OOD): periodic configurations (even-sided container, symmetry aalgeed initial velocity) using list-prompt mode with a fixed periodic grid; we evaluate cycle consistency and phase accuracy over evenly spaced timestamps. We provide example theorem below that supports such periodic case construction.
Rationale: measures whether models trained on generic dynamics can extrapolate to a qualitatively different but mathematically structured regime DDD periodic motion in polygonal symmetry).
25

---

Published as a conference paper at ICLR 2026
Periodic Construction (transformative setting). We exploit closed-form condition that yields perfectly periodic, normal "shuttle" trajectories between two concentric, co-rotating regular polygons. This result underpins the periodic test cases in ROT BOX transformative split and provides an analytical knob to dial the fundamental period via the angular velocity.
Theorem (Periodic bounce between two concentric regular n-gons), Setup. Let Po and P. be two concentric regular n- n-gons (n 2 3) with circumradii R > R > 0. Both polygons rotate rigidly with the same constant angular velocity w about their common center. At time 1, = 0 a point mass ("ball") is placed on the inward normal to a side of Po and moves with speed v > along that normal toward P.. Collisions with sides are perfectly elastic, and motion is confined to the annular region between the polygons. The initial pose has one vertex on the +x-axis.
Let (R))= R (os((T/1)) denote the apothem of a regular n-gon with circumradius R, and define the normal gap
A = oo (R))) = (Ro R.) cos (.))
Thus A is the (signed) distance between the parallel supporting lines of the corresponding side family in Po and P..
Claim (closed-form condition). The ball executes uniform periodic motion bouncing back and forth at constant speed along a fixed set of parallel sides with a repeating impact pattern- if and only if there exists an integer E Z such that
k 2TT
n Ro Rs) cos
Equivalently, with the one-way flight time
A
(Ro RR) cos(T//))
Jay
the periodicity condition is
2T n
W, tily
When this holds, the fandamental bounce period and the orientation recurrence 2(R.. R₂) cos(T//))
2T
n.
Toounce 2, tay
Torient
WW
k/o
The minimal nonzero periodic rotation corresponds to =
Proof sketch. (1) In regular nogon, opposite sides are parallel; the distance between their supporting lines is 2a 2a(R). R)). For concentric, co-oriented P.. P., the normal gap between the corresponding supports is A = =(R.)) - al (R). (2) Launching exactly along side normal produces specular reflections that preserve a straight, normal shuttle between parallel side families; the speed remains so euch one-way flight takes lay- = A/v. For the next impact to occur on a side parallel to the previous one (so that the normal shuttle and impact geometry repeat), the side orientations must recur, which in regular n-gon happens modulo 2t/ n. Hence w, tay = 0 (mod 2 n.), yielding the stated condition.
v.v. (3) During a one-way flight, the polygons rotate by w, lay-
Construction recipe for ROT_BOX (periodic). To instantiate periodic test scenes in the transforma- tive split
1. Choose n (even makes the normal families align with diameters) and set circumradii (R,, Ra) (or effective radii after finite-size shrink/expand).
2. Pick speed e >> O and launch along a side normal of P. (avoid vertex alignment by tiny phase offset).
Set the box angular velocity using [k] in the closed form, n, R.) //j) and co-rotate any inner boundary present, or equivalently use Wrel for differential rotations.
2
4. The resulting shuttle has Thounce 2(R. Rs) (os///n)// and repeats in orientation every Turient = nd/v. For evaluation, sample timestamps on a uniform grid over several bounce periods to probe phase stability.
26

---

Published as a conference paper at ICLR 2026
A.3 COMPETITION CODING
Competition Code is well-established domain where participants solve complex algorithmic prob- lems. specified problem, the solver program is required to generate the correct output for every imput in the provided test suite. We curate algorithmic families and collect several problems per family from various well-known competitive programming platforms. We propose phased perturbation pipeline create comprehensive OOD dataset.
A.3.1
SEED FAMILIES AND COVERAGE
We curate 3-5 seeds per algorithmic family. The current collection includes:
Mo's Algorithm (4): LuoguP1494, LuoguP4462, LuoguP4887, Lu0guP5047
Segment Tree Decomposition (3): CF981E, CF1140F, LuoguP5787
CDQ D&C (3): CF848C, CF1045G, LuoguP4093.
Meet-in-the-Middle: CEO12015-D2T1,
LuoguP2962,
SPOJ-ABCDEF,
USACO20/2USSpen- 3
Square Root Decomposition (5): CF710D, CF797E, CF1207F, LuoguP3396, Lu- oguP8250.
Each seed problem is tagged with public problem code in websites like CodeForces, ArCoder, and Luogu. Per seed, we target 5-10 perturbation strategies (configurable; default 10). For narrative coverage, maintain library of 20 background templates (c.g., Campus Life, Ancient Warfare, Cyber Security, Energy Grid, Xuanhuan Fantasy), and by default rewrite each perturbed seed into all backgrounds.
A.3.2 SYNTHESIS PIPELINE
Phase 1: Standardize seed problems. This phase transforms heterogeneous problem statements into a unified specification. First, the framework parses raw Markdown extract core fields such as the problem statement, input/output formats, constraints, and examples, and utilize LLMs to reduce typographic ambiguities and make semantic clarifications.
Phase 2: Produce enumeration--asse solutions for standardized seed problems. This phase generates diverse set of feasible, though not necessarily optimal, reference implementations for each standardized seed problem. Emphasis is placed on reliability rather than optimality, ensuring have correct solutions for small test cases.
Phase Produce enumeration- based test case generators for standardized seed problems. This phase synthesizes test case generators grounded in original seed problems. By curating prompts for LLMs, generators designed cover representative distributions and adversarial conditions.
Phase 4: Generate perturbation strategies. This phase generates strategies how to perturb problems systematically, Each strategy seed is curated by human expert with at least years of competitive programming experience and designed for making a perturbation while keep the main solution unchanged. These strategy seeds are standardized and extended to strategies with detailed instructions.
solutions and test case generators according strategies. Phase generates standardized perturbed problem statements, based on perturbation strategies. Similar to phase 2 and phase 3, we generate corresponding solutions and test case generators based on enumeration. When generating solutions, we provide the original problem and solution effectively improve the reliability.
Phase 5-7: Generate perturbed problems, enumeration-
Phase Produce input constraint sanity check test case generators for standardized perturbed problems. To enhance the robustness of our evaluation, this phase produces input constraint sanity check test case generators. Curated test case generators are designed for testing whether the solution code handle big test cases in reasonable small time. Test case constraints manually adapted to the Python programming setting, guaranteeing no brute-force solutions can pass and all correct Python solutions can be accepted.
27

---

Published as a conference paper at ICLR 2026
Phase 9: Produce background rewrites. Finally, this phase provides effective approach to generate OOD samples. By utilizzig 20 background settings, the standardized perturbed seed problems rewritten in different background stories, maintaining the same imput/output formats and solutions. All these rewritten problems are final and ready to be involved in training.
A.3.3 EXAMPLE 1: SEGMENT TREE DECOMPOSITION BIPARTITE OVER TIME Seed (excerpt).
"Given Each of the (m) edges is active on an interval (II, r)) over the discrete timeline (1..k). For each time (0) determine whether the active subgraph bipartite."
Perturbation strategies (from Phase 2, sample).
Two-interval activation. Replace each edge's interval (QI. r)) with exactly two disjoint subintervals (11,71), [12, r2)). The solver continues to use DSU-rollback over a segment tree covering time.
Interval-EEvett rewrite. Convert each interval to two explicit events: an add at (l), remove (r+1). Feed the event list unchanged into the segment-tree time.
Event-pair splitting, Expand each add/remove into two sub-events (e.g., preparelapply) to stress timeline density without changing the rollback design.
Before/After (Strategy-Ievel variant). Before (seed): time-varying edges with single intervals ((I, r)). After (strategy 1): "Each edge is active exactly on two disjoint intervals ([4, ril) and (ll2, r2)). For each (t) in (1..k,, is the subgraph bipartite? ?" Algorithmic essence and complexity remain the same: DSU with rollback over segment tree on the time axis, O((n + m) log k).
A.3.4 EXAMPLE 2: SQUARE RoOT DECOMPOSITION - --AHH-BUCEET GROUP SUMS
Seed (excerpt).
"Given an array value. For many queries with modulus (pjn), report the sum of numbers in bucket (x), where index (k) helongs bucket (I mod p). Updates assign value, y."
Strategy-level perturbation (background-agnostt Grouped Sequence Sum and Update Queries:
Before: group by (k: mod p). After:
"Define Hii = Σt=o Syi* mod M. Sum queries ask for the total over indices mapping to given hash value (g): updates set A; <- r.."
This preserves the bucket-sum structure and the O(.) behavior under small-(M) caching and updates, matching the seed's enumeration profile while modestly changing the grouping function.
Background rewrite (Campus Life). Before (strategy-level): abstract group sums under (H(i).
After (background): Campus Club Scores:
"Student IDs (I..N) are assigned clubs by a polynomial function (C()). Queries ask for the total score in club (g): updates change a student's score."
Narrative terms shift (students/clubb/sccores) but the formal mapping (C(i) and the I/O grammar remain intact so the variant's enumerator and the background rewrite both agree on the 100-case oracle.
A.3.5 SUMMARY
By enumeration- first solutions and enforcing strategy-level clarity before rewriting, the pipeline makes large-scale, verifiable perturbation feasible. Standardization, deterministic test generation, and back- ground consistency checks together ensure that every variant-deve faithful to the core algorithm and produces outputs consistent with the seed's brute-force oracle. This methodology yields rich, well-structured families suitable for training, evaluation, and pedagogical use.
narrative diversity-rennt remains
28

---

Published as a conference paper at ICLR 2026
B EXPERIMENT DETAILS
Models. We use Swen3-4B-Instruce as the reference instruction-tume model for all experiments in this paper.
Training Details. We fine-tune with GRPO (Guo et al. 2025) using the Open-Instruct framework Unless otherwise noted, the key arguments are:
--beta 0.0
--num --num_ haam__uunnuu__prppppsrroooooooo
48
rrun__saapleeepeerrrroroogeeeoee 16
--k -kl_estiimaoor k13
-1earminggrate 5e-7
-max_tooen__eegtth 12240
"pax_prrompittoke__eeeee 2048 \
-response__ength 10192
-pack_length 12240 12240 \
-agplyyveriffablle_eraaaa true
-moo_sscop_penalty True \
-roo_sscop_eemaltyyalllll 0.0
+temperature 1.0
-total_episodds 1000000
-deepsseed_ssage 2
-nuumminni_batches
-muun_learners_eer_oooee 8
--num_epochs
ww_____eesor_paralleeeees size 1
--clip -clip_higher 0.3
--vl --vilmm_um__nginee 8
_i__shheulle__type constant --seed 1 -grradiett_cheekopnntiin
Across all eeseriment--includinng the multi-stage schedules in the paper vary only (i) the train/eval datasets, (ii) the base/reference model, and (iii) the scoring mode (full-pass reward vs. per-test reward) to match the setting.
Datasets for learnability (Section3) Marufactoria-HA:: 742 training and 100 test examples. SaauffoctooilaaSATA//PPPPP
EXACT: 350 training examples in total across the three fam- ilies. Manufactoria 560 training examples. examples.
535 training
Datasets for generalization 4).ction44.. Unless otherwise specified, for each curated problem family and each difficulty, we sample 1,000 training problems (Appendix AA.244). In the setup of Figure 9 a), the training set contains six families at the Basic level, totaling 6,000 training samples. Evaluation comprises:
In-distribution (ID): 100 test samples from the same Basic difficulty as training.
Explorative (OOD): 100 test samples per family at each higher difficulty (Easy, Medium, Hard).
Compositional (OOD): 100 test samples per composed family at Basic difficulty.
Transformational (OOD): 100 test samples per setting.
Evaluation Protocol. Evaluation uses the same sampling configuration as training. Each score is averaged over 4 runs.
Compute Resources. Each RL run uses 16 NVIDIA H100 GPUs across two nodes and completes in ~3 days for 1,000 optimization steps.
/open- nstruct instren
29

---

Published as a conference paper at ICLR 2026
(a) Models: Qwen3-4B-Instruct/ Problem Family: Manufactoria-REGEz
1.0
Training Data Full Poss Rate
Explor-
Convergence
RL Collapse
0.8
ation
Phase
Phase
0.6
0.4
Grokking!
0.2
0.0
200
400
600
800
1000
1200
Training Steps
(b) Models: Qwen3-4B-Instruce Problem Family: BouncingSim-Al1
1.0
Training Data Full Pass Rate
Exploration Phase
Convergence
Phase
0.6
0.4
Grokking!
0.2
0.0
100
150
200
250
300
Training Steps
(c) ModelssNemotton-114 / Problem Family: Manufactoria-HAS
1.0
Training Data Pass Rate
Exploration
Convergence Phase
0.8
Phase
0.6
0.4
Grokking!
0.2
0.0
0
100
150
200
250
300
Training Steps
Figure 10: Grokking across models and tasks. (a) Qwen3-4B 4B-Instruct Manafactoria-REGES 4B-Instruct BouncingSin-AAl (same training setup as Figure 2. (c) Nemotron- 14B Manufactoria-HAA Curves plot training-date full pass rate versus training steps. A consistent pattern emerges: a long exploration phase, an abrupt grokking transition, and a convergence regime; (a) also exhibits an RL collapse when training continues past convergence.
Qwen3-
C ADDITIONAL EXPERIMENTS
C.1 GROKKING GENERALIZES ACROSS MODELS AND PROBLEM FAMILIES
Figure 10 demonstrates that the RL grokking phenomenon, extended low-signal exploration phase followed by an abrupt phase transition and rapid convergence in training-data full-pass rate, can arise across (i) model sizes and families and (ii) distinct problem scopes.
After a long plateau, performance surges and subsequently enters convergence regime. Continued training eventually triggers an RL collapse, highlighting the need for stabilization or early stopping once solutions con- solidate. Panel (b) uses the same model BouncingSim-All, wollw ball-bouncing simulation
Panel (a) shows Qwen3-4B --e-------s trained on Manufactoria-Pee
30

---

Published as conference paper at ICLR 2026
(a) Problem Family: cDQ_ac
(b)
Problem Family: KEET IN_MID
NID
(c) Problem Family: sqRT DC
1.0
1.0
1.0
Training bem Fall Pozz Rate
Exploration
Convergence
Exploration
Convergence
Exploration
Convergence Phase
Phase
Phase
Pers
0.8
Phase
Phase
Phase
0.4
04
Training
Training
0.2
82
02
0.0g
100
200
300
400
500
600
0.0g
300 Treining Steps
200
400
500
0.0
100
200
300
400
500
600
Training Stegs
Treining Steps
Figure 11: Grokking across competition code tasks. (a) CDQ-DC; (b) MEET-IN-MID, (c) SQRT-DC. Curves plot training data full pass
training steps.
coding suite for real-world coding tasks. The same exploration to phase-transition to a convergence pattern appears. Panel (c) swaps the model family and scale to Nemotron-14B on Manufacto- ria-HAS again reproducing the grokking phenomenon. We also provide additional experiments on the competition coding task problem families in Figure IT that demonstrate the same trend.
Together, these results indicate that grokking is not artifact of a particular backbone or a single synthetic family. emerges with different parameter counts, across independent model lineages, and on tasks ranging from symbolic program synthesis to physics-driven simulation code. This supports view that RL can discover new procedural strategies rather than merely sharpening pre-trained ones.
C.2 WARM-UP BENEFITS BEYOND THE "PASS @ K=0" PROBLEMS
Warm-up with per-test rewards is not only rescue mechanism for tasks where the base policy never succeeds; also helps when the initial success probability small but non-zero (pass@k In this regime the binary full-pass reward still provides a weak and high-variance signal, which can lead to slow unstable improvement. A short warm-up phase with dense, per-test rewards (here: 100 steps) (i) accelerates discovery of partially correct behaviors, (ii) better stability, and (iii) delivers a more reliable starting point for the subsequent binary-reward phase. Empirically, we observe faster and steadier convergence with warm-up. whereas training that optimizes full-pass from scratch can remain sluggish and brittle, sometimes exhibiting late-stage regressions even after partial progress.
0).
Models: Qwen3-4B-Instruc / Problem Family: Manufactoria-REGEX
With Warmup (Per-test Pass Rate)
Training Data Full Pass Rate
Without Warmup
0.8
0.6
0.4
0.0
200
400
600
800
1000
1200
Training Step
Figure 12: Warm-up helps when pass@k small but non-zero. Training curves on Mantfactoria REGEX with -W-en33-B-nsstrucc The blue curve is trained after 100-step warm-up using per-test rewards, then switched to the hinary full-pass objective; it achieves faster and steadier gains. The orange curve trains full-pass from scratch and improves slowly with occasional regressions.
31


Reviews:
[{"review_id": "gckLTBt9YE", "reviewer": "Reviewer_UbX7", "summary": "The paper studies on a family of synthetic coding problems whether RL can learn tasks that were previously unsolvable and generalize to new unseen tasks. The primary contribution lies in controlling which distributions RL training and testing happen on. The takeaway from the learnability experiment is that using dense rewards for coding problems (fraction of passed tests) during a warmup-phase followed by the standard binary reward can turn some of the previously unsolvable tasks (i.e., 0 pass@128 prior to RL) into solvable ones. The takeaway for the generalization experiments is that RL enables both exploratory and compositional generalization, but fails at the transformative ones.", "questions": "how was the code competition dataset used? generally it's not obvious to me which tasks were trained on and which were tested on in some of the experimental results.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- the paper studies highly relevant and important questions at the intersection of RL & LLMs: what are the limitations of RL for learning new skills and generalizing to new domains\n- the obtained results are interesting and put into the context of the broader literature that has previously discussed some these points", "weaknesses": "- the connection to grokking is unclear and confusing: in grokking, the model overfits the train set initially with a sudden increase in the test performance. the learnability experiments have close to 0% pass rate on the training set at the beginning so there's no train-test performance gap. my interpretation of the results is that the initial phase of close to 0% pass rate reflects a sparsity of non-zero gradients, and the training run first needs to accumulate sufficient informative/positive samples to reinforce.\n- the paper claims that dense rewards can generally enable learning on previously unlearnable problem distributions, but the empirical evidence for this is limited to synthetic toy tasks. the explanation for this seems plausible, but it's unclear wether using dense rewards is necessary or significant in more realistic use cases, where the dataset consists of a mixture of problem types and difficulty levels. in realistic cases, generalization between different problems might be sufficient to enable learning previously unsolvable tasks and adding dense rewards might not have any effect. adding more experimental evidence to support the claims would greatly benefit the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7xYhJLGkWM", "reviewer": "Reviewer_Hvky", "summary": "The paper considers the capabilities of RL-finetuned LLMS, specifically, whether they are able to learn to solve previously unseen problems, and how well do they generalize to harder versions of seen problems. To this end, it makes three main contributions:\n\nFirst, the authors present Delta-Code: A controlled benchmark of synthetic coding problems, designed to evaluate: \n1) the “learnability” of how to solve problems that have not been encountered during an LLM's pre-training. To this end, Delta-Code incorporates problems related to solving puzzles of a classic Flash game (Manufactoria).\n 2) the generalization to harder instances of the problems encountered during training. To enable this evaluation, Delta-Code contains coding tasks related to 2d bouncing object simulations.\n\nSecond, the authors utilize Delta-Code to evaluate whether fine-tuning an LLM using RL can allow said LLM to solve coding problems that it has not seen before and which require “new strategies”. They show that coding problems allow for richer supervision (by using the ratio of unit test passed by the proposed solution, instead of a binary reward) and that this supervision eventually enables the LLM to learn to solve the novel coding tasks. It is also noted that this approach can also fail for very challenging problems.\n\nThird, the authors use Delta-Code to evaluate the generalizability of an RL-finetuned LLM. The use three generalizability axes: explorative, compositional and transformative, and conclude that these models fail to generalize when the test coding problems require solutions which are significantly different than the ones encountered at training time.", "questions": "Why were competition coding problems added to the benchmark?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Delta-Code introduces a new family of coding problems that have not been encountered by current LLMs, making it a good tool for fair evaluation of their capabilities. Moreover, the family of 2d simulation coding problems could be useful for more reliable evaluation of a model’s problem-solving-generalization capabilities. Together, these can facilitate further research in this direction.\n\nAccording to the paper, there is currently a debate on whether using RL to finetune an LLM allows for novel problem solving strategies to be learned. If so, then this paper represents a good step towards answering this question.", "weaknesses": "Readability: Many of the terminology used in the paper is not well defined, but its meaning is only explained intuitively. For example, learnability; transferability; explorative, compositional and transformative axes of generalization, reasoning strategies; skill family. Perhaps these are not well-defined terms, but with only an intuitive definition, it’s hard to judge how well the proposed benchmark evaluates them. In addition, RL training of LLMs is a big part of this paper, but the setting is never explicitly stated, with some details becoming available only at the end of page 5. I think the paper would greatly benefit from a “Background section” which clearly defines the background knowledge needed to better understand it.\n\nIt is possible that I am wrong, but the experimental results in sections 3 and 4 do not seem surprising, so they appear to be limited in their significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t7bPmu4i1B", "reviewer": "Reviewer_Zh7M", "summary": "This paper introduces a benchmark and process for controllable generation of synthetic questions in those categories. The three benchmarks include augmented programming questions seeded from LiveCodeBench-Pro, 2D elastic collision simulation, and the puzzle game Manufactoria. The benchmark is designed to be able to generate questions with interpretable distribution shifts with the categories, for example smaller or larger polygons in the elastic collision simulator or difficulty in the coding questions (as measured by LLM success rate). The paper then runs various RL experiments on Manufactura and BouncingSIM. They show advantages to record-replay and switching from denser proxy rewards (number of unit tests passed) to the target reward mid training can both improve RL performance. Multiple different dynamics in the BouncingSIM are explored with separate training on subsets and then testing on different subsets or combinations of dynamics. They show very little transfer between different classes, but going from two classes of dynamics to their composition preserves much of the original accuracy.", "questions": "- Figure 4 has overlapping text on the x axis.\n- groking is a technical term which involves training accuracy saturating with little increase in test accuracy initially and (much) later an increase in test accuracy with little to no train accuracy increase. Is that the intended use here?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The paper introduces two interesting new benchmarks with can be decomposed into subsets of their mechanics to generate different types of problems\n- There are some nice demonstrations of some techniques for RL\n- There is a pretty clear difference shown between the compositional versus different classes in BouncingSIM", "weaknesses": "- I do not see a coherent story between the different tests and experiments run and I think most of the experiments do not support the goals laid out in the introduction\n- I was frequently confused about what benchmarks were being described and what experiments were run\n- I often could not tell whether a figure was representational or actually data\n- Generally I found this paper difficult to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zOh32PRO14", "reviewer": "Reviewer_eAUM", "summary": "The paper examines the learnability and generalization of the RL algorithm in the domain of large language models. The paper presents two benchmarks, Manufactoria and BouncingSim, for testing novel algorithmic skills and compositional algorithmic skills. These tasks are proven to be unsolvable for LLMs with an adequate sample size. To test the learnability of the collected benchmark, the authors design a method to assign partial credit to a single test case and then transition to full credit. With this method, the RL algorithm (GRPO) exhibits the grokking phenomenon, and its performance grows rapidly within a few steps. Additionally, the experiment suggests that with the selected curriculum, the RL algorithm can learn more effectively. Bouncing2D is designed to test the generalization. The experiment results show that RL can exhibit good exploration and compositional generalization, but still yield poor transformative results.", "questions": "1. In section 2.1, these problems are not common programming problems or Turing-machine tasks. What do the problems belong to in terms of problem complexity? Does the problem belong to push-down automata or simply a finite-state machine? Or is the problem much harder than a Turing machine?\n2. Following Q1, how much OOD-ness comes from the program syntax? If the program syntax is written in Python, can modern LLMs solve these problem families? Does the Grokking phenomenon come from the syntax? Does the language model suddenly understand the syntax and solve the problem perfectly?\n3. In section 3.2 and Figure 6, how does the result `its convergence speed is still slower than the baseline GRPO algorithm, likely because the reused traces are off-policy` come from? Figure 6 shows Experience Replay and baseline converge to a common performance.\n4. Although Bouncing2D is designed for testing generalization, what is the learnability of different difficult tiers?\n5. What kinds of generalization ability are needed to learn across different families in Manufactoria? Or what are the atomic algorithmic skills needed between families to make the generalization impossible?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Two different benchmarks to test the learnability and generalization of RL algorithms.\n2. There are debates about the Grokking phenomenon; the work provides a testbed for testing Grokking, which is quite hard for the most performing language model, GPT-5. With a simple warm-up that includes unit-test credit assignment, the model exhibits a 'grokking' moment in the experiment.", "weaknesses": "1. The justification of OOD-ness is not trivial to me, see Questions 1 and 2 below. \n2. The paper discusses learnability and generalization on two different datasets. There is no strong correlation between these two datasets. See Questions 4 and 5 below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KmmYBItvVV", "reviewer": "Reviewer_wA9H", "summary": "The authors of this paper present a controlled, fully verifiable benchmark suite for program synthesis spanning Manufactoria DSL puzzles, BouncingSim bouncing ball simulation, and competitive programming tasks. The authors position this benchmark suite as a programming-domain counterpart of OMEGA (synthesizable math families), producing unit-test verifiable instances, yielding rich per-test reward signals and making external tool-shortcutting much harder.\n\nWith this benchmark suite, the authors investigate learnability, whether RL can train a model to solve problem families that the base model fails at, even after many sampled attempts (high pass@K), and generalization, whether the learned procedures transfer to OOD problems along the exploratory, compositional, and transformative axes.\n\nThe key empirical finding is that a two-phase reward schedule, which involves a warm-up stage with dense per-test rewards and then switching to a binary full-pass reward, induces a grokking-like transition. After a long, low-reward plateau, performance abruptly jumps to high accuracy and then stabilizes, including on families where the base model had a pass@K=0. Across tasks, gains transfer well on the exploratory and compositional splits, while performance on the transformative split remains weak. The authors interpret this as evidence that RL can unlock solution strategies beyond what is reachable by sampling the base model alone. The authors also caution that the warm-up is not a universal recipe and the benefits depend on model capacity, target family difficulty, and curriculum compatibility.", "questions": "Could the authors clarify how they decided to switch from dense to binary (fixed step vs plateau/success trigger), the warm-up length (was it only illustrative?), and whether they tried a gradual weight shift from dense to binary?\n\nMinor Comments/Typos:\n- Specicaly -> Specifically\n- consruction -> construction\n- BouncingSim vs BounceSim. Figure 1 uses BounceSim while the rest of the paper uses BouncingSim. Is this on purpose?\n- Figure 1 claims four axes, including “Domain level”, but experiments report three (exploratory, compositional, and transformative). Is there a reason why there were no results from the Domain-level?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Clear Learnability Results**: On a family where the base model has pass@K=0, the two-phase (dense to binary) schedule yields a grokking-like transition from failure to full pass. The grokking phase is a strong signal that the model has found an algorithmic procedure.\n\n**Generalization Along Principled Axes**: Using OMEGA-style exploratory, compositional, and transformative splits, they show strong transfer on the first two and pinpoint limits on the third. This is a useful signal for where future work is needed.\n\n**Comprehensiveness**: Broad experimental sweep across domains, families, defining thoughtful difficulty ladders, and training recipes, including ablations on replay, feedback-in-the-loop, and curriculum-based data, supporting the central claims.\n\n**Dataset Contribution**: The authors supply an OOD problem space by introducing a bespoke Manufactoria DSL, a new syntax unlikely to appear in pretraining and pairing it with templated generators. The authors also organize families into a scalable difficulty ladder (Basic, Easy, Medium, Hard) based on aggregate model performance, so researchers can stage experiments by capability.", "weaknesses": "**Under Characterized Reward Switch Schedule:**\nThe two-staged dense -> binary schedule is central to the learnability claim, yet the paper does not ablate warm-up length or switch timing. The only concrete examples are an illustrative 100-step saturation during per-test warm-up, 450-step post-switch exploration, and a 100-step warm-up case in the appendix. The schedule hyperparameters otherwise appear fixed. This leaves open the minimal warm-up needed, whether earlier/later switches change sample efficiency or stability, and the robustness of this claim. More details strengthen the result and clarify robustness.\n\n**Evidence for Curriculum Task Compatibility:**\nThe paper’s claim that “curriculum must be task-compatible” is compelling, but with only two examples (REGEX -> HAS, COMPR -> HAS), it is hard to generalize and see if the effect holds across other families or difficulty tiers, or whether there are cases of strong negative transfer. A small source x target compatibility/staged results matrix (warm-up on source S -> RL on target T) across more families/tiers, alongside this paper’s BouncingSim slice, would strengthen the paper and enable the research community to understand and quantify this paper’s claims of how curriculum design impacts learnability dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)