RL for Reasoning by Adaptively Revealing Rationales
Authors: Mohammad Hossein Amani
Aryo Lotfi
Nicolas Baldwin
Samy Bengio
Mehrdad Farajtabar
Emmanuel Abbe
Robert West
Venue: ICLR 2026 Poster (2026)
Keywords: reasoning
reinforcement learning
curriculum learning
sequence modeling
Primary area: foundation or frontier models, including LLMs
TL;DR: using a dynamic per-sample curriculum for RL training of reasoning models on math datasets

Abstract: Learning in the combinatorially large output space of sequence generation problems is challenging as providing expert demonstrations scales poorly with sequence length, and RL struggles with sparse rewards. Between dense demonstrations in supervised training and no demonstrations in reinforcement learning lies an underexplored regime: partial supervision. We ask whether some classes of sequence learning problems become efficiently learnable by exploiting this gap. We address this by introducing adaptive backtracking (AdaBack), a per-sample curriculum learning algorithm that reveals a partial prefix of the target output. The supervision length is adjusted dynamically for each sample based on the model’s past reward signal, allowing it to incrementally learn to complete reasoning chains by conditioning on correct partial solutions. We investigate this intermediate regime between SFT and RL and argue that per-sample curriculum learning is more than a trade-off between efficiency and generality—it can succeed in tasks with long sequences of latent dependencies where SFT and RL both fail to generalize. Using a synthetic task with latent parity constraints, we show that AdaBack reliably solves problems that are otherwise intractable. On three mathematical reasoning benchmarks, DeepScaleR, MATH, and GSM8k, we find that AdaBack enables models to solve problems that RL alone cannot, acquiring new reasoning capabilities through incremental exposure to partial solutions.

Published as a conference paper at ICLR 2026
RL FOR REASONING BY ADAPTIVELY REVEALING RATIONALES
Mohammad Hossein Amani'
Aryo Lotfi
Nicolas Mario Baldwin'
Samy Bengio 1.2
Mehrdad Farajtabar
Emmanuel Abbe!
Robert West
EPFL 2 Apple
ABSTRACT
Learning in the combinatorially large output space of sequence generation prob- lems is challenging providing expert demonstrations scales poorly with sequence length, and RL struggles with sparse rewards. Between dense demonstrations supervised training and no demonstrations in reinforcement learning lies under- explored regime: partial supervision. We ask whether some classes of sequence learning problems become efficiently learnable by exploiting this gap. We address this by introducing adaptive backtracking (AdaBack), a per-sample curriculum learning algorithm that reveals a partial prefix of the target output. The supervi- sion length is adjusted dynamically for each sample based on the model's past reward signal, allowing it to incrementally learn complete reasoning chains by conditioning correct partial solutions. We investigate this intermediate regime between SFT and RL and argue that per-sample curriculum learning is more than a trade-off between efficiency and generality- it can succeed in tasks with long sequences of latent dependencies where SFT and RL both fail to generalize. Using synthetic task with latent parity constraints, show that AdaBack reliably solves problems that are otherwise intractable. On three mathematical reasoning bench- marks, DeepScaleR, MATH, and GSM8k, we find that AdaBack enables models solve problems that RL alone cannot, acquiring new reasoning capabilities through incremental exposure partial solutions.
INTRODUCTION
The reasoning capabilities of Transformers (Vaswani et al. 2017) have been extensively studied across domains such mathematics (Saxton 2019 Cobbe al. 2021 Hendrycks et al. 1021 Lewkowycz et al. 2022), algorithmic reasoning Veličković et al.) 2022). and code generation (Chen et 2021). These studies indicate that the reasoning performance of such models is significantly enhanced by training and inference mechanisms involving explicit reasoning traces or rationales,
commonly known scratchpad (Nye et al., (021) chain-of-thought (Wei et al. 2023)
However, acquiring large-scale, high-quality reasoning traces for specialized domains, such as mathematics, poses substantial challenges. Reinforcement learning (RL)-inspired methods have emerged as promising solution to this challenge. By utilizing (potentially noisy) reward functions or verifiers (e.g., checking final answers math problems), language models can generate novel reasoning traces, effectively using the model as an RL policy. For instance, the REINFORCE algorithm (Williams) 1992) has been employed by methods like STaR (Zelikman generating multiple solutions per problem and selectively fine-tuning on the correct ones. More advanced methods utilize algorithms like Proximal Policy Optimization (PPO) (Schulman al.) 2017) and, more recently, Group Relative Policy Optimization (GRPO) (Shao al.) (024)..
al. 2022).
Despite recent progress, RL-based approaches for structured reasoning tasks continue to face signifi- cant hurdles. The key challenge lies in exploration: as reasoning chains grow longer, the space of valid output sequences increases exponentially, while reward signals remain sparse and often binary. Consequently, the probability of sampling correct solution through random exploration diminishes exponentially with sequence length. As a result, standard RL tends to reinforce reasoning paths that already assigned negligiiee probability by the pretrained model. Empirical evaluations by

---

Published as a conference paper at ICLR 2026
Havrilla et al. (2024) and Yue et al. (2025) support this observation, showing that RL fine-tuning primarily amplifies existing behaviors without substantially exploring novel solution trajectories.
These limitations motivate investigation of third regime: the space between supervised fine-tuning (SFT) and RL. We study whether breaking down long reasoning chains into partial rationales-and and adaptively conditioning the model only prefix of the target solution during training- can make exploration tractable, and ultimately expand the class of problems that sequence models can learn solve. This leads to our central research question:
Can reinforcement learning, when guided by adaptive partial supervision, teach models genuinely new reasoning capabilities? Specifically, for sequences learning tasks of variable lengths, can it enable discovery of solutions that were previously exponentially (in sequence length) unlikely under the model's ' initial distribution?
To illustrate this clearly, consider reasoning task consisting of sequential steps, each of which must be performed correctly for the task to succeed (e.g., proving math theorem). For simplicity, assume the model performs each step correctly with a constant probability p. Naively attempting the entire task yields a success probability of p", meaning that positive reinforcement signals would be exponentially infrequent on average, every p-" iterations maaiinn standard RL impractical. To mitigate this, we propose adaptive guidance: initially, we reveal all but the final step of the solution from the training dataset, using RL solely for this final step. Consequently, the model receives feedback with probability p every iteration. As performance improves, we progressively conceal more steps of the solution, training the model complete increasingly larger segments, maintaining frequent positive feedback. Eventually, the entire reasoning chain is learned step-by-step. effectively transforming complex search space with success probability yd into n simpler sub-searches, each with success probability (p)) While this example includes simplifying assumptions, we empirically demonstrate this phenomenon's 's practicality on a parity task described in Section 2.2
The scenario described above is simplified and idealized. In real-world problems, reasoning steps may not be clearly distinguishable, and datasets typically contain tasks with varying difficulties and solution lengths, making uniform partial reveals inefficient. To address these practical challenges, we propose a per-sample adaptive algorithm that dynamically adjusts the revealed portion of each solution based on its inherent difficulty. Specifically, we leverage the GRPO framework (Shao et al. 2024), which naturally estimates task difficulty by generating multiple rollouts per question and averaging the rewards. We term this method adaptive backtracking (AdaBack), detailed further in Section
Backtracking strategies have precedents in RL-for for example, the method of Salimans and Chen (2018). In language modeling, R3 (Xi et al. 2024) applies a curriculum that progressively trains models complete reasoning chains from earlier positions. Building on these ideas, we ask: how can such slicing be adapted per sample, guided by model performance at each iteration, without relying on heuristically designed global curriculum?
Contributions Our contributions are as follows:
We propose AdaBack, per-sample adaptive curriculum learning algorithm that dynamically adjusts the amount of supervision during RL training based on reward feedback. AdaBack enables each sample to progress at its own rate, eliminating the need for manual curriculum staging or handcrafted schedules, transitioning from full supervision to full exploration in a data-driven way,
We show that AdaBack bridges the gap between SFT and RL, enabling learning regimes where both fail. On synthetic parity task with sparse rewards, we demonstrate separation result: AdaBack reliably solves the task, while SFT, RL, and their combination all fail.
On standard mathematical reasoning benchmarks (DeepScaleR, MATH, and, GSM8K), we show that AdaBack improves performance over standard RL and SFT+RL pipelines. We also show that AdaBack applied base models often matches the performance of SFT-initialized counterparts.
We introduce further two variants of GSM8k: Base-7, which uses an unfamiliar numerical format never by the model during pretraining, and Tensor-2, which concatenates problems to increase reasoning depth. AdaBack achieves strong performance on both, demonstrating robust generalization to symbolic shifts and longer-horizon reasoning.
2

---

Published as a conference paper at ICLR 2026
1f 10 P/ U(O,p!")
/00
x/2 [3x+1
Precz
x odd.
what ((LL))
Evalastin each value, (1)) 3-1 4 4:000) R4) 412 2:00 ADD (2) 2/2 1; tinaly, WWW.RRRRRRLOOOO ~4)
1 <:: Pl)) Utf!"
Figure 1: AdaBack Update Rule. At epoch i, we sample a supervision ratio
U Puin
(4) and
condition the model'ss generation the question and the corresponding partial answer (shown in black text, with unrevealed content grayed out).
average reward is below a threshold т. we increase supervision the next epoch by sampling from the red interval p(", U(p pmax)) )- Otherwise, if the reward exceeds т, reduce supervision and sample from (O, Pi ), the green interval, to make task harder.
Finally, identify limitation: for instruct-tuned models or models where pretraining has already exposed the model to most problem types, AdaBack together with standard RL training provide no benefits, underscoring their limitation in aiding exploration where uncertainty is low.
2 ADAPTIVE BACKTRACKING
The core idea is to expose prefix of the target sequence during training and gradually reduce this supervision based on model performance. Unlike fixed-step or handcrafted curricula, this method allows each sample progress its own pace, naturally balancing difficulty and learning progress.
For RL algorithms like GRPO (Shao et al. 2024) or RLOO (Ahmadian et al. 2024) that perform multiple rollouts for each sample, dynamically adjust the supervision level that the average sample reward across the rollouts stays close to a desired amount (for example, 50% of the answers correct for each math question). For other RL. algorithms, the criteria change the supervision level can depend on other estimation of the state's 's value.
Problem Setup
We denote random variables using capital letters. Let an input sample be repre- sented as sequence of tokens X =X,, X2,..., Xt). corresponding such as a question or problem description. The desired output is chain-of-thought (CoT) style where each Y, represents a token or step the model's 's reasoning
natural language prompt,
response Y (Y,, ....., Ym), process. We assume we have dataset of such (X, Y) pairs.
Our goal is to train model to generate a correct reasoning sequence conditioned on input X. Importantly, the correct CoT is not necessarily unique: multiple reasoning paths may Icad to valid answers. However, we assume that correctness is verifiable via reward model r R(X1:ts Yi.)) where if the generated output Ý accepted as correct. We denote by T'ormat [0, 1) the reward for parsable generation when the generation has the correct format, and assume that 0 otherwise. This setup reflects many real-world reasoning tasks. For instance, in mathematical problem solving, final answers can often be verified, and different CoT traces may yield the same final result.
2.1 METHIО
and its corresponding target output (() (x ...., y(il), reveal the first k tokens of Y. YLs. where k (ما)) mm: and ( [0, is sample-specific supervision portion. The model is trained to continue generation conditioned on the input and the revealed prefix (=(
At each training step, given an input (0) "
,..,, ((0)
k+1lmg ~ Po x()) y(i), 1ike where m - 1 is the
length of the generated continuation.
Adaptive Update Rule For each training sample i, maintain interval |P Pmax]] initialized
as [0. 1), from which we uniformly sample the supervision portion / for sample i at epoch t. After generating of predictions and receiving average reward r:
(6) (or other estimations of the state's
3

---

Published as a conference paper at ICLR 2026
value if not using GRPO), update this interval based fixed reward threshold T:
( If r < T:
Poin Pt
(8)
If r 2T
(
Pixxx
t Pi
Pmii < 0.0
The next level of supervision is then uniformly sampled from this updated range:
Upmiin P P
Pz+1
This process is visualized in Figure
Intuitively, the model receives less supervision when performs well and more when struggles, enabling an automatic per-sample curriculum that adapts to the model's learning progress. This procedure naturally drives the model toward completing longer and more challenging portions of the target sequence, only when it shows competence easier prefixes. The threshold r governs how strictly we evaluate model success: higher values require stronger reward signals ge.g., higher average rewards) for the supervision ratio to decrease. In general terms, this process performs form of stochastic binary search over the supervision ratio p. using reward feedback as a success signal. The goal is to minimize the amount of revealed rationale while ensuring the model continues to receive useful reward signals. As the model improves, the supervision ratio naturally decreases.
For samples with no reward history, we initialize p(0) from global moving averages pmin and Pmaxx which are updated over time using exponential moving averages. Additionally, to close train-test. distribution mismatch, with a small probability we randomly set the portion to zero. These training details are further discussed in Appendix
This adaptive scheme enables each training sample to follow its own trajectory from full supervision to full generation, providing flexible and data-efficient approach curriculum learning in structured sequence tasks. This per-sample scheduling ensures that each training point advances only when ready, allowing the model to incrementally acquire reasoning skills without overfitting fixed patterns.
2.2 CHAIN-OF-PARITERR A SYNTHETIC ENVIRONMENT FOR STUDYING REASONING
To better understand learning dynamics in isolation from the complexities of natural language and pretraining, we introduce synthetic sequence modeling task called the chain-of- chain-of-parities. This task could be viewed as a contextual blind cliff walk, inspired by Schaul et al. (2015), adapted to reflect the challenges of thain-of-thougee reasoning.
We aim to address a fundamental question: Are there sequence learning tasks that cannot be learned by SFT, RL, their naive combination but that can be solved by Adaptive Backtracking? We construct such a task and demonstrate that the answer is affirmative.
Given binary input sequence X {0,1)*, the goal is to generate an output sequence Y1, Z YL, Zr of length 2L, where
Y, E (0,1) is unconstrained (both values are acceptable).
Z, is the parity of X,, Y,, and Z,-1, Z, Z + Y: + X₁, where @ denotes the XOR function and 7 = 0.
This design enforces a latent, step-wise structure over the output. The Y; values act as "scratch- pad"- arbitarry values that influence Z, through an accumulating parity computation. In essence, each 7; encodes the parity of the prefix (XI,..., X, Yi...., ..)....Y, making the correctness of Z; de- (CoT) steps. This recursive dependency
pendent on the accuracy Z;-1 and prior mirrors real-world CoT tasks, where early mistakes cascade through the solution. Although there are 2 valid outputs per input due to unconstrained Y,, generating one randomly has a probability of just 2 2 to sparse-rewee reward regimes common in reasoning tasks.
We consider learning this task from a dataset of n uniformly sampled sequences (X,, Y)). This setup highlights the inherent limitations of SFT and RL:
RL fails: Rewards are sparse, and discovering a single valid output via random exploration becomes exponentially unlikely as the problem length grows.
4

---

Published as a conference paper at ICLR 2026
SFT fails: For small training sets, supervised training alone cannot learn this task. Specifically, the task involves learning n - parity functions of degree three, for which the theoretical sample complexity is well-studied (Abbe et al., 2023a Kou et al. 2024). For example, in the statistical query (SQ) model (Kearns] 1998), weakly learning a degree-k parity from I. input bits requires at
least I samples. Consequently, regular fixed-size neural networks trained via stochastic
gradient descent (SGD) cannot even weakly learn a degree-three parity from only n 0
samples (Abbe et al.) (023a))
With limited number of training samples, such = (L) discussed above, SFT does not provide weak learning. Therefore, the exploration of the model remains random, and rewards remain exponentially sparse, hindering meaningful learning through standard RL.
SFT + RL fails:
In contrast, curriculum learning with adaptive supervision succeeds in solving this task using a limited number of training samples. Suppose the reward threshold is fixed 0.5. The adaptive curriculum can initially present the input question X along with almost the entire iolution-namel Yi,..., 11-1-t the model. In this case, the model only needs generate the final step, Yz, Zr. Assuming the model has learned the task format during pretraining, the probability of generating a valid final step is approximately 0.5, in contrast 2 when generating the entire sequence.
This setup enables the model to explore and learn the final reasoning step. Specifically, for each partial sequence X1...... Xz. .i..... ZL-1, the model produces two valid continuations: Yz, Z and Y2, Zi, where x' = 4 1 denotes the complement of x. This variation allows gradient- based learners infer that Y,, directly influences Zr: flipping Yz changes Zr. while all other inputs remain fixed. Consequenlly, learning Eµ reduces to learning degree-two parity function, one input (Yt) is already learned, which has lower sample complexity than learning the degree-thres parity function. In particular, degree-two parities can be learned with ((L) samples (Glasgow| 2024 Kou et al. 2024). Once the model masters the final step, the curriculum is adjusted to reveal shorter solution prefix (up to ZL-2), requiring the model to generate YL-1, ZL. YL, ZL. Since it has already learned generate Yz. Zz, the focus shifts to learning the (L 1)-th step. This process is repeated iteratively by gradually shortening the hint until the model learns the full sequence. This progression demonstrates that there exist sample-size regimes where standard SFT RL fails, but AdaBack successfully learns the task.
We note that the argument above is simplified, as the actual learning complexity depends on the specific learning model used. We will discuss such subtleties in Appendix [B] Here, we focus on the empirical evidence supporting this phenomenon. We consider the case of L 16 with training set size of n 1024, using the Llama 3.2 IB model (Grattafiori et al.) 2024). The reward function assigns value of 1 to fully correct sequences and 0.1 to sequences that are syntactically valid (i.e., correct number of bits).
We first perform SFT on the training set to help the model learn the task format. We then compare the performance of standard RL and AdaBack on the task. As shown in Figure 2 (left), AdaBack achieves substantial reward early in training and progressively reduces the portion of the solution that is revealed, leading to seamless learning of the task. In contrast, standard RL (Figure right) fails to obtain meaningful rewards and does not learn the task.
This experiment highlights how AdaBack enables guided exploration via partially correct sequences, effectively expanding the class of learnable problems in structured sequence modeling. It also establishes clear separation between SFT + standard RL and AdaBack: while the former struggles with sparse rewards, AdaBack offers a principled middle ground, avoiding both the limitations of full supervision and the exploration challenges of standard RL..
3 EXPERIMENTS
In this section, we evaluate AdaBack on real-world reasoning tasks. In Section 2.2 that AdaBack enables models to solve problems that are intractable under standard supervised fine-tuning, reinforcement learning, or their combination. Our goal here is to investigate whether AdaBack improves generalization and reasoning ability beyond what standard RL pipelines achieve on world reasoning tasks.
we showed
We use Ô to hide the logarithmic factors.
5

---

Published as a conference paper at ICLR 2026
AdaBack
Standard RL and R3
R3
Train Reward
Text Reward
RewardPPrtion
0.6
Rew
Test Reward
0.4
Aug. Portion
Standard AL
Train Reward Rewerd
0
200
400
600
800
1000
0
2000 4000
8000
8000
10000
12000
14000 18000
lberation
Iteration
Figure Training Dynamics. Left: Training and test rewards along with supervision ratios throughout training. With AdaBack, Llama 3.2 IB successfully learns the task in under 700 iterations. Right: Training and test rewards for SFT+RL (red) plateau 0.1, indicating that only the output format-learned during supervised pretraining- has been retained. Test reward for R3 (Xi et al., 2024) is shown in purple: it reaches only 0.8 reward after more than 16,000 iterations. R3 segments training examples at all whitespace positions and applies RL uniformly over these fragments, resulting in inefficiency due to its non-adaptive strategy.
We train models using the GRPO algorithm (Shao et al. 2024) on base models of the Llama-3 family (Grattafiori al. 2024). For faimess of attribution and clarity, have followed the standard practice of works like (Shao et al. 2024: Xi et al.) 2024) by using simple greedy or sampling- based decoding for our comparisons with a prompt as simple as 'Please provide the final answer to the question in \boxed All baseline experiments are ran until convergence.
The standard RL recipe involves first performing SFT rationales, followed by RL fine-tuning (Gulcehre et al., 20033; Guan et al., (2025). However, this paradigm has recently been challenged by models trained solely with RL updates such as GRPO, without any SFT phase (DeepSeck-A) 2025). Accordingly, we compare four variants: GRPO on base model, GRPO on an SFT model, KaaBack-GRPP on base model, and ddaBack-GRPP on an SFT model. Further experimental details are provided in Appendix D
Generalization on Natural Language Reasoning Tasks We evaluate on three standard math- ematical reasoning benchmarks: DeepScaleR (Luo et al., (025). MATH (Hendrycks et al. 1 and GSM8k (Cobbe et al.) [202T] In line with Mirzadeh et al. (2024) and ci ct al. (2024). to assess variants of GSM8k:
generalization beyond pretraining exposure, we introduce
Base-7 GSM8k, all numeric quantities and computations are represented in base-7 While the problems themselves are unchanged, this symbolic shift forces the model to reason over format it has not encountered during annaiiiigg-aaloouuuuu to deploying a model in a culture with a different numerical base. Performance on this dataset requires the model to generalize beyond familiar surface forms and reasoning chains observed in pretraining.
We create Tensor-2 GSM8k inspired by Hosseini et al. (2024), by concatenating pairs of GSM8k problems and their solutions into single instances, yielding longer reasoning chains. While the resulting chains may be semantically disjoint, this construction increases the number of sequential reasoning steps required in the generation. This setup tests whether models can scale their reasoning over longer inputs and outputs, beyond what is required in the original GSM8k.
Table [ summarizes performance across these tasks. We find that AdaBack consistently outperforms both GRPO and SFT+GRPO, particularly in out of pretraining distribution settings like Tensor- GSM8k. Notably, AdaBack applied directly to the base model often matches or exceeds the performance of the standard RL applied to the base and SFT-initialized models, suggesting that adaptive partial supervision on base model may serve as a more effective prior than initializing from SFT checkpoint. Interestingly, AdaBack applied on the base model may even outperform
We dropped (fewer than 1000) samples where the question answer required division, as base- 10 divisions have periodic representations in hase-7.

---

Published as a conference paper at ICLR 2026
AdaBack applied the SFT model as in the Tensor-2 experiment. We elaborate this phenomenon further in the text.
Table 1: Final test accuracy for each method across tasks and model sizes.
Method
DeepScaleR
MATH
GSM8k
Base-7 GSM8K
Tensor-2 GSM8k
3B
IB
3B
1B
3B
1B
3B
Base+RL
6.8
6.6
6.4
15.0
7.9
63.7
4.8
4.9
0.0
0.0
SFT+RL
7.1
9.1
7.4
17.7
36.7
72.7
14.4
45.4
6.9
42.7
AdaBack
9.0
10.6
9.1
19.1
39.2
73.3
18.4
43.9
8.5
49.2
SFT+AdaBack
9.5
12.5
9.5
19.9
43.2
70.7
24.5
49.9
11.3
42.2
Per-Sample Curriculum Without Manual Staging Curriculum learning typically requires hand- crafted stages, scheduling heuristies, and careful tuning of hyperparameters- -uccessssscchhhh-- as how long to train at each difficulty level and with what parameters, and when to transition to harder ones. In contrast, AdaBack performs an automatic per-sample curriculum by adjusting supervision based on simple reward threshold. Each training point progresses at its definitions. Although our implementation uses a stochastic binary search over supervision ratios, this is not essential: linear search or other adaptive strategies could be substituted. The key principle to allow each example to progress at its own pace, without requiring global curriculum design extensive hyperparameter tuning. As seen in Figure (left) average portions naturally decrease and rewards increase with AdaBack without any boilerplate curriculum scaffolding. т is the only hyperparameter needed be set for AdaBack, and training is not sensitive it. We have elaborated the choice of r in Section
pace, without global stage
AdaBack
Standard RL vs. AdaBack
Parisa PaPortino
Test Rewerd
Avg. Portion
Adaback
Standard
Reward
Reward
2000
40000
6000
8000
10000 12000
14000 16000
2500
5000
7500
10000
12500
15000
17500
Itaration
AdaBack
Standard AdaBack
0.6
Test Reward
Avg.
Adaback Test Reward
Standard RL
Test Reward
2000
4000
6000
8000
10000
2000
6000
6000
8000
10000
Figure 3: AdaBack vs. Standard RL Across Model Initializations. Results from training Llama3- 1B GSM8k dataset. The top shows results for models initialized with SFT, while the bottom row shows base (non-SFT) models. The left column presents AdaBack training dynamics: train reward increases as supervision ratios (portions) decrease. The right column shows standard RL.
Training Dynamics and Initialization Figure 3 shows training curves comparing AdaBack and standard RI. across SFT and base model initializations. In both settings, AdaBack achieves better
7

---

Published as a conference paper at ICLR 2026
rewards both training and test compared to standard GRPO training. We provide additional training figures in Appendix
Does AdaBack Expand the Model's Solution Space? Yue et al. (2025) argue that RL fine-tuning
reweights a model'ss 's output distribution without expanding its effective reasoning capacity. To test this claim, we evaluate models using pass@1 @ metric that captures the breadth of plausible solutions. Figure shows shows that AdaBack significantly improves pass@k @k over standard RL both the base and SFT models, especially at large k. These gains are most pronounced when AdaBack is applied to the base model, again suggesting that it facilitates the discovery of new solution modes rather than just refining existing ones. If SFT-free RL with AdaBack introduces novel capabilities, we expect pass @ ko to increase even when base model coverage low, which is what we show in Figure In some cases, such as on the Tensor-2 GSM8K (see [T) AdaBack on the base model even outperforms AdaBack on the SFT-initialized model, suggesting that SFT may sometimes restrict the search space too early, limiting the benefits of exploration The latter can be observed in Figure as well.
with SFT
without SFT (base madel)
1.0
Adstlack
0.8
Base
0.6
2
Figure 4: Pass@k for Llama3-1B 1B SFT-initialized models (left) and base models (right) on GSM8k. AdaBack keeps significant gap compared to standard RL and improves performance higher k even without SFT suggesting it expands the solution distribution rather than reweighting known answers (contra Yue al. 2025
When Does AdaBack Fail to Help? On the MATH dataset with Llama 3.2 3B- 3B-Instruct, we observe no gains from AdaBack. As shown in Figure TO both train and test rewards saturate quickly, with nearly all questions solved within few hundred iterations. In Figure we illustrate the training and test reward/accuracy trajectories for Qwen2.5-1.5B (both base and instruct) (Yang et al. (024) on the GSMSK dataset, where we observed similar behavior. We observed the same phenomena in the case of GSM8k for the 3B model in Table This happens when RL reaches perfect training reward within a few iterations during training, i.e., when the dataset is too simple for the model. This suggests that these models have likely encountered much of the dataset during pretraining, leaving little room for RI. improvements or for AdaBack to assist exploration. In such cases, AdaBack provides limited highhht-i that its main strength lies in tasks where sparse reward symbolic mismatch creates a significant learning barrier.
2 reports a direct comparison between AdaBack and R3 three datasets of increasing difficulty- GSM-k < MATH DeepScaleR (Luo et al. 2025) using Llama-3 IB and Llama-3 3B base models. Across all settings, AdaBack and R3 rely on the same ground-trutt solutions and reward functions; the only difference is the curriculum mechanism. This isolates the effect of adaptive per-sample supervision from the mixture approach R3.
Comparison with R3 on GSMSk, MATH, and DeepScaleR Table
Overall, we observe a clear trend aligned with dataset difficulty. On DeepScaleR, AdaBack outper- forms R3 both 1B and 3B models. On MATH, AdaBack has an edge for the IB model, though
3 @ is the probahility that least one out of k model-generated outputs correctly solves given problem Chen et al. (202T); [Brown et (2024)
'n an extreme case, the model may just memorize the answers during the SFT phase, removing hope for any exploration.
8

---

Published as a conference paper at ICLR 2026
this edge diminishes for the larger 3B model. On GSM8K, AdaBack performs slightly below R3, though the absolute gap remains small relative to the inherent variance of RL post-training.
We believe two structural factors underlie this patterm:
Segmentation brittleness on complex reasoning traces. MATH and DeepScaleR contain long, multi-line reasoning steps, LaTeX/MathJax blocks, and irregular formatting. These properties make R3's slicing into fixed "stages" brittle and sensitive to heuristic delimiter choices, whereas AdaBack eliminates the need for any segmentation hyperparameters by using reward-driven adaptivity. On the other hand, for GSM8k, different steps of the solution can be easily separated using the newline delimiter. The latter explain he superior performance of R3 and suggests that slicing solutions based on individual reasoning steps (as in R3) may work better than slicing the solution at random points (as done by AdaBack) when clean segmentation possible. However, as discussed earlier, separating reasoning steps is not always feasible (e.g., in MATH and DeepScaleR).
Per-sample adaptivity becomes increasingly important on heterogeneous, long-horizon tasks. Harder datasets such as MATH and DeepScaleR contain many questions of widely varying structure and solution length. R3 differs from AdaBack in two aspects:
1. It slices all solutions into an equal number of pieces (e.g.. 5), which is set globally for the whole dataset. Such global slicing does not account for variations in question difficulty and cannot adapt to intra-dataset heterogeneity. AdaBack's 's per-sample curriculum, which adjusts the supervision length independently for each example, is particularly advantageous in these settings and supports more stable progression on difficult problems.
2. Instead of imposing a curriculum over the data, R3 creates mixture from all questions and sliced solutions. This contrasts with AdaBack's 's easy-to-hard curriculum. The use of a curriculum becomes increasingly important as the difficulty of the training set increases. In particular, Abbe et al. (2023b) provide theoretical results comparing curriculum learning and data mixtures: curriculum learning is beneficial when easy samples within distribution are rare; conversely, if such samples are prevalent, standard training on the mixture may be more efficient than a curriculum.
To further support our argument, refer the reader back to the chain-of ofaaiiie--arities experiment presented in Figure [ where it shown that as the number of (non-trivial) reasoning steps in a task increases, the AdaBack approach can significantly outperform R3 (making training > 20 more efficient in Figure 2). The real-world datasets used in our paper still contain only few reasoning steps. We believe AdaBack will better demonstrate its advantages on morizon reasoning tasks (e.g., math Olympiad problems). However, we point out that experimenting in such regimes presents major challenges regarding data acquisition and large-scale compute.
Table 2: Comparison of R3 and AdaBack on GSM8K, MATH, and DeepScaleR
Method
GSM8k
MATH
DeepScaleR
IB (base)
3B (base)
IB (base)
3B (base)
IB (base)
3B (base)
R3
41.5
74.2
7.8
19.2
6.6
9.6
AdaBack
39.2
73.3
9.1
19.1
9.0
10.6
RELATED WORK
We also provide a more extensive literature review in Appendix A
Limits of Reinforcement Learning for Reasoning Several studies have highlighted the difficulty of applying reinforcement learning (RL) to structured reasoning tasks. Exploration remains a major bottleneck: reward signals are sparse, and correct reasoning chains exponentially rare in the output space. Havrilla et al. (2024) empirically evaluated several RL strategies on reasoning benchmarks and found that models fail to discover solutions outside the support of the base SFT model. They tested curriculum-inspa inspired techniques such as backtracking-aae -ttrttin the model partway through a prioritized
solution and gradually shifting the start point earlier Salimans and Chen

---

Published as a conference paper at ICLR 2026
level replay (BRR)-sampling more frequently from difficult problems Jiang et al. (2020). Despite these interventions, performance gains were negligible: RL primarily amplified answers already assigned non-trivial probability by the pretrained model. These findings align with recent theoretical arguments by Yue et al. 2025). who claim that RI fine-tuning reweighs existing reasoning paths but fails to induce fundamentally
capabilities.
In contrast, we find that curriculum learning with partial supervision can induce such capabilities. Through carefully designed adaptive exposure to partial solutions, our models learn to complete problems that neither SFT nor RL could solve-even partially- the limitations of standard RL exploration and the promise of fine-grained curricula.
Curriculum Learning Curriculum learning was originally introduced to improve generalization and convergence by training models on simpler examples before harder ones (Bengio et al. 2209). In RL, backtracking (Salimans and Chen 2018) and RFCL Tao et al. (2024) adopt this principle by initializing rollouts from later points in expert demonstrations and gradually shifting the start point backward.
In language modeling, R3 Xi et al. 2024) introduces a step-wise curriculum where the model is trained to complete reasoning chains from progressively earlier points. However, despite being motivated as a dynamic curriculum, R3 effectively performs static data augmentation: demonstrations are sliced at delimiter positions (e.g., newlines in GSM8K), and each slice is treated as an independent training sample. In datasets without consistent delimiters, R3 falls back on uniformly partitioning each solution into fixed number of segments (e.g., five), regardless of problem difficulty or solution length, and then mixing all segments across the dataset. This reliance on global heuristics limits scalability and applicability to domains without clear step boundaries, such as MATH (Hendryeks) eet al. 202T) or our synthetic parity benchmark. In particular, for the parity task we show that R3 has significantly worse learning speed and generalization in Figure 2 This consistent with the theoretical results of Cornacchia and Mossel (2023); Abbe et al. (2023b) where it is proven that applying curriculum over distributions can significantly improve the speed of convergence over simply mixing the distributions,
Our approach differs in both granularity and adaptability. We introduce an adaptive per-sample curriculum that dynamically controls the fraction of the output revealed to the model during training. Unlike R3, we do not rely on external step segmentation or -ppasfc-specific structure. This fine- grained supervision allows the model to incrementally learn reasoning behaviors directly from reward signals, without explicit hinting or hard-coding intermediate targets.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
Curriculum learning is often seen as a training heuristic to improve convergence. In this work, we show that per-sample curriculum learning via adaptive partial supervision enables fundamentally new reasoning capabilities that are inaccessible to standard supervised fine-tuning or reinforcement learning alone. Our proposed method, AdaBack, succeeds RL and SFT fail. offering a constructive separation result. On real-world datasets such as MATH (Hendrycks et al. (2021), GSM8k (Cobbe et al., 202I). ). and its variants, AdaBack outperforms baseline RL and SFT+RL pipelines, often matching or exceeding standard RL applied on SFF-initialized models even when applied directly to a base model.
synthetic parity task where both
These results suggest that AdaBack not only improves sample efficiency, but also facilitates explo- ration of new solution modes, as evidenced by consistent gains in pass@k. However, we also observe that AdaBack provides limited benefit when the base model has already memorized much of the task, as appears to be the case with modern reasoning-apt models and math datasets.
A key limitation of our current implementation arises in the large dataset regime. When the number of unique training samples is high, most examples are seen infrequently and therefore rely heavily on global moving averages for supervision scheduling. This may reduce the effectiveness of per-sample adaptation. One promising direction for future work is to define supervision schedules not per sample, but per region in embedding space-e.. e.g., using the average supervision level of the k nearest neighbors of each sample. This would preserve the benefits of adaptation while remaining scalable in data-rich settings.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
E. Abbe and C. Sandon. On the universality of deep learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20061 Curran Associates, Inc., 2020. /tL/httpp://pprrceedddi neuri s. /at.://pr..000111133332222222130 Paper pdf
E. Abbe and C. Sandon. Polynomial-time universalityy and limitations of deep learning. Communica- lions on Pure and Applied Mathemattix, 7atta/ttcee..7711111114443333333 2023, doi:
a.22121. 121
E. Abbe, P. Kamath, E. Malach, C. Sandon, and N. Srebro. On the power of differentiable learning versus PAC and SQ learning. In Advances in Neural Information Processing Systems, volume 34, 2021.
E. Abbe, H. B. Adsera, and T. Misiakiewicz. Sgd learning neural networks: leap complexity and saddle-to-saddle dynamics. In The Thirty Sixth Annual Conference on Learning Theory, pages 2552-2623. PMLR, 2023a.
E. Abbe, E. Connacchia, and A. Lotfi. Provable advantage of curriculum learning on parity targets with mixed inputs. Advances in Neural Information Processing Systems, 36:24291-24321, 2023b.
E. Abbe, S. Bengio, A. Lotfi, C. Sandon, and O. Saremi. How far can transformers reason? the globality barrier and inductive scratchpad. Advances in Neural Information Processing Systems, 37:27850-27895, 2024.
A. Ahmadian, C. Cremer, M. Gallé, M. Fadace, J. Kreutzer, O. Pictquin, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024. URL https: /ttp:v/aa.ivv.v/.i org/ abs 1444022..444
M. H. Amani, N. M. Baldwin, A. Mansouri, M. Josifoski, M. Peyrard, and R. West. Symbolic autoencoding for self-supervised sequence learning. CoRR, abs/2402. 2024. doi: 10.48550 /ARXIV.2402. 10575. URL. https::/d///tt
/ogg/10 48550 2402. 10575
S. Bae, J. Hong, M. Y. Lee, H. Kim, J. Nam, and D. Kwak. Online difficulty filtering for reasoning oriented reinforcement learning. CoRR, abs/2504.03380, 2025. doi: 14445550AAXXI..55400.555 ww.448555/aiix. www.aarwwv........0 2504.0333333
URL //ttp::::/oi..rogo/o/oo
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, and et. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL htttps://arxiiv.
2204. 05862]
B. Barak, B. L. Edelman, S. Goel, S. Kakade, E. Malach, and C. Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. arXiv preprint arXiv:2207.08799, 2022.
Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pages 41-48, 2009.
Y. Bengio, N. Léonard, and A. C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. ArXiv, abs// 343.2228.34.... 2013.
B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL tttttt:s/////xxv... g/ 21787
M. Chen, J. Tworek. H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, Mīshkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert- Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji. S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code, 2021. URL https: : arxiv v. org abs/
21 2107. 03374
11

---

Published as a conference paper at ICLR 2026
X. Chen, J. Lu, M. Kim, D. Zhang, J. Tang, A. Piché, N. Gontier, Y. Bengio, and E. Kamalloo. Self-evolving curriculum for LLM reasoning. CoRR, 149700 2025. doi: 10.48550/ARX IV IV.2505.14970.
petttp::/dbi..oree
K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint arXiv: 011.//922.1..8 2019.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv 10.
preprint 1e.::2210..44466 2021.
E. Comacccia and E. Mossel. mathematical model for curriculum learning for parities. In International Conference Machine Learning, pages 6402-6423, PMLR, 2023.
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning,
2025. URtps::://aaxx wwwp:/wwwwww..rraa///ba//// /ttpo///////////aaaaaooo///////a 12204 12948
B. Edelman, Goel, Kakade, E. Malach, and C. Zhang. Pareto frontiers in deep feature learning: Data, compute, width, and luck. Advances in Neural Information Processing Systems, 36, 2024.
G. Feng, B. Zhang, Y. Gu, H. Ye, D. He, and L. Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36, 2024.
C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel. Reverse curriculum generation for reinforcement learning. In Conference robot learning, pages 482-495. PMLR, 2017.
S. Gao, A. Bosselut, S. Bengio, and E. Abbe. Abstral: Augmetting Ilms' reasoning by reinforeing abstract thinking, 2025. URL
07751
M. Glasgow. SGD finds then tunes features in two-layer neural networks with near-optimal sample complexity: A case study in the XOR problem. In The Twelfth International Conference on Learning Representations, 2024. /eKL/attsss/ppeeeeveeeee.es http://wwww.eeetooooddddddddde
ZB16
S. Goyal, Z. Ji, A. S. Rawat, A. K. Menon, S. Kumar, and V. Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations, 2024. UtLhttts::// hwwp//veeeeeeeeeeeeeeeeeeeeeeeee /
A. Grattafiori, A. Dubey, and Others. The llama herd of models, 2024. URL httth:t///xxxiiii org/ abc//bb//aa 2407 21783
X. Guan, L. Zhang, Y. Liu, N. Shang. Y. Sun, Y. Zhu, F. Yang, and M. Yang. rathr-math: Small 1ims can master math reasoming with self-evolved deep thinking, 2025. URL https 04519
//arxiv.org/abs/
C. Gulcehre, T. Paine, S. Srinivasan, K. Konyushkova, I. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, W. Macherey, A. Doucet, O. Firat, and N. de Freitas Reinforced self-training (rest) for language modeling, 2023.
http///aivvv////////v///a///aaav org/ abs abs/2308 08998
A. Havrilla, Y. Du, S. C. Raparthy, C. Nalmpantis, J. Dwivedi- Yu, M. Zhuravinskyi, E. Hambro, S. Sukhbaatar, and R. Raileanu. Teaching large language models to reason with reinforcement learning, 2024. http::ttt:://ttt wrg..ooor.gg..o.. abs /200//240..3 04642
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, Song, and Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. https: //aa////axvvv
org/abs/2103. 03874
A. Hosseini, A. Sordoni, D. Toyama, A. Courville, and R. Agarwal. Not all Ilm reasoners are created equal. arXiv preprint arXiv:2410.01748, 2024.
M. Jiang, E. Grefenstette, and T. Rocktäschel. Prioritized level replay. International Conference Machine Learning, 2020.
12

---

Published as a conference paper at ICLR 2026
L. Kaiser and S. Bengio. Discrete autoencoders for sequence models. ArXiv, 1b200
M. Kearns. Efficient noise-tolerant learning from statistical queries. J. ACM, 45(6),98333)) Nov. 1998. ISSN 0004-5411. doi: 10.1145/9333772293351 URL 145/293347.29332
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
Y. Kou, Z. Chen, Q. Gu, and S. M. Kakade. Matching the statistical query lower bound for SkS-sparse parity problems with sign stochastic gradient descent. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL http:: /ttppreve//peeevieew .nt forum? d EbSSBvw(Wwi
J. Lanchantin, S. Toshniwal, J. Weston, S. Sukhbaatar, et al. Learning to reason and memorize with self-notes. Advances Neural Information Processing Systems, 36, 2024.
A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, Schlag, T. Gutman-Solo al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 14858, 2022.
Q Li, L. Cui, X. Zhao, L. Kong, and W. Bi. GSM-plus: A comprehensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers. In L.-W, Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 3961-2984 Bangkok, Thailand, Aug. 2024. Association for URL httptttttss/aaaiiii
Computational Linguistics. doi: 10. 10.18655/w1/22224.al-ooog
logy. www./org//20 20ogg//20//440000 2024.024..20 11022..cc-1110 acl.llnng..... 163/
B. Liu, S. Bubeck, R. Eldan, Kulkami, Li. A. Nguyen, R. Ward, and Y. Zhang. Tinygsm: htlgg//b//// abs abs/2312 2312.19221111111
achieving -SSUURLER URL http:: //aaiivvaaiiiaa//aa org/
M. Luo, S. Tan, J. Wong, X. Shi, W. Y. Tang, M. Roongta, C. Cai, J. Luo, I. E. Li, R. A. Popa. and I. Stoica. Deepscaler: Surpassing ol -peeview with a 1.5b model by scaling rl. hhtpps:/ // wwt.onooiion.om pretty-radio-Be www.-raioo--a not.ooonnnnnnnnnooooomoo wwtootionnnitteeeeee htt.://c.......eeeeeeaeaaaaaaaaa
ith-a-l- 1-ta-l1-5 5B MBB-MMdel1-- - by wtt.aoow.........0000000000000e.
2025. Notion Blog.
W. Merrill and A. Sabharwal. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations, 2024. bttppt:::///ppnntt view. net/ forum NjNG1Ph8Wh
I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models, 2024. URL
ditts://aarxia www//ogg//...o 2410. 10..0000000 05229
S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone. Curriculum learning for reinforcement learning domains: A framework and survey. The Journal of Machine Learning Research, 21(1):3322-7411, 2020.
M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, C. Sutton, and A. Odena. Show your work: Scratchpads for intermediate computation with language models, 2021.
T. Olausson, A. Gu, B. Lipkin, C. Zhang, A. Solar-Lezama, J. Tenenbaum, and R. Levy. LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5153-5176, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10. 10.188 /1886//////2233.. 1ttp:////2....mmmmmmmmm..33.. https://aclantion
URL
wwtg///002202 e0tt2:3.mmmmpmmaaaa. 313/
R. Y. Pang, W. Yuan, K. Cho, H. He, S. Sukhbaatar, and Weston. Iterative reasoning preference optimization, 2024a. htll:htppss//axxi.orrr/aab///
19733
13

---

Published as a conference paper at ICLR 2026
R. Y. Pang, W. Yuan, K. Cho, H. He. S. Sukhbaatar, and J. E. Weston. Iterative reasoning preference optimization. ArXiv, abs/2404.19733, 2024b. Dtt.ttsss//pb...eeatt....tooatrr or g/ (CopppsDD:2694555
N. Razin, H. Zhou, O. Saremi, V. Thilak, A. Bradley, P. Nakkiran, J. Susskind, and E. Littwin. Vanishing gradients in reinforcement finetuning of language models, 2024. URL httpp:
abs /200 .20003
C. Resnick, R. Raileanu, S. Kapoor, A. Peysakhovich, K. Cho, and J. Bruna. Backplay: "man muss immer umkehren",
T. Salimans and R. J. Chen. Learning montezuma's ' revenge from a single demonstration. ArXiv, abs/1812.03333 2018.
R. J. Sánchez, L. Conrads, P. Welke, K. Cvejoski, and C. O. Marin. Hidden schema networks. In A. Rogers, J. L. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pages 4764-4798. Association for Computational Linguistics, 2023.
D. Saxton, E. Grefenstette, F. Hill, and P. Kohli. Analysing mathematical reasoning abilities neural models. arXiv preprint 1wtxXivv:9000000 005501157, 2019.
T. Schaul, J. Quan, I. Antonoglou, and Silver. Prioritized experiecce replay. CoRR, 15.5550.0 13022595
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017. htt.ttts::/// httpthttts/:/aaiiiiiii http:////wwwww.w.oo/aaa///////// wrtp
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
URL htt.::tt///// //arxi/rvvvv org/ abs 2402. 03300
T. Shi, Y. Wu, L. Song, T. Zhou, and J. Zhao. Efficient reinforcement finetuning via adaptive curriculum learning. CoRR, abs/2504.05520, 2025. doi: 10.48550/ARXIV 48550/ 05520
URL
https:://oi.org///
S. Sukhbaatar, E. Denton, A. Szlam, and R. Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning, 2018a.
0811/10011100 09083
S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and R. Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play, 2018b. URL peatps://arxiv.oot
03. 05407
S. Tao, A. Shukla, T.-k. Chan, and H. Su. Reverse forward curriculum learning for extreme sample and demonstration efficiency in rl, 2024.
J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and Higgins. Solving math word problems with process- and outcome-based feedback, 2022. URL https:://axxie org/ abs 221.21221..11111111
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Veličković, A. P. Badia, D. Budden, R. Pascanu, A. Banino, M. Dashevskiy, R. Hadsell, and C. Blundell. The clrs algorithmic reasoning benchmark. arXiv preprini 22X.:2205...555 2022.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of- thought prompting elicits reasoning in large language models, 2023.
R. J. Williams. Simple statistical
algorithms for connectionist reinforcement
learning. Machine learning, 8:229-2566 256, 1992.
Z. Xi, W. Chen, B. Hong. S. Jin, R. Zheng, W. He, Y. Ding, S. Liu, X. Guo, J. Wang. H. Guo, Shen, X. Fan, Y. Zhou, S. Dou, X. Wang, X. Zhang, P. Sun, T. Gui, Q. Zhang, and X. Huang. Training large language models for reasoning through reverse curriculum reinforcement learning, 2024.
14

---

Published as a conference paper at ICLR 2026
S. Xue, Z. Huang, J.-Y. Liu, X. Lin, Y. Ning, B. Jin, X. Li, and Q. Liu. Decompose, analyze and rethink: Solving intricate problems with human- like reasoning cycle. In Neural Information Processing Systems, 2024. https: wtw.aaaicccooooo.rrrr.ooooooo.
276117098
A. Yang. B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
K. Yang, A. Swope, Gu, R. Chalamala, P. Song, S. Yu, Godil, Prenger, and Anandkumar. Leandojo: Theorem proving with retrieval-augmented language models. Advances in Neural Information Processing Systems, 36:21573-21612, 2023.
Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang. Y. Yue, S. Song, and G. Huang. Does reinforcement learning really incentivize reasoning capacity in iims beyond the base model?, 2025. URL https://arxiv> arx.vv org/ atsp//////200..2
E. Zelikman, Y. Wu, J. Mu, and N. Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
E. Zelikman, G. Harik, Y. Shao, Jayasiri, N. Haber, and N. D. Goodman. Quiet-star: Language models can teach themselves to think before speaking, 2024. URL CRLhttps: ptttps://arxiv..oo org/a) /bb s/2403. 09629
15

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
Seratchpads and Chain-of- Chain-of-Thought Strategies It is widely believed that success on challenging problems requires model to know how use intermediary computations in the context, to reason and deduct the final answers. Nye et al. (202T) proposed supervised training of Transformers to use scratchpads in addition to final answer, showing improvements on tasks like executing Python code and evaluating polynomials. Similarly, WWi et al. (203) proposed chain-of-thought prompting, showing that large language models can generate scratchpads via in-context demos and without explicit training. Moreover, Kojimaa et al. (2022) studied zero-shot chain-of-thouge generation for language models. Lanchantin et al. (2024) introduced the concept of self-notes, showing benefits of interleaving the intermediate reasoning steps within the question/context. /ooya et al. (2024) introduced pause tokens which act as place-holder tokens providing models with more computation time before output generations.
Several works have shown that allowing transformers to produce a chain-of-thought would increase their expressivity (Feng al. 2024 Merrill and Sabharwal 2024). Further, AAbee et al. (2024) put forward the notion of globality degree of a task as a hardness measure and show that scratchpads can make learning more efficient by breaking the globality of task. They also proposed inductive scratch- pads which impose Markovian structure over reasoning steps, improving length generalization. Gao et al. (2255) proposed AbstRaL,, showing improved robustness by training language models on mathematical abstraction of reasoning traces, instead of natural language chain-of oftthoughts... Outside of reasoning and natural language scratch pads, unsupervised learning of intermediary symbolic sequences using straight-through gradient estimators (Bengio et al. 2013) has been studied in (Amani
cct al. 2024 Sánchez et al.) 2023 Kaiser and Bengio 2018).
RL and Reasoning Benchmarks Reinforcement learning has emerged as a dominant paradig gm for post-training language models tasks involving sparse, verifiable reasoning signals Havrilla ct al. (2024). Our reasoning tasks and settings are closest to STaR (Zelikman et al. 2022). Quiet- STaR (Zelikman et al. 2024), and -STaR-MATH (Guan et al., 2025). which apply RL improve mathematical and symbolic reasoning through selective training on verified rationales.
In parallel. alignment with human preferences has also leveraged RL techniques- most prominently Proximal Policy Optimization (PPO (Schulman et al. (017 (Bai et al., 2022, Pang et al. (024a). Recent efforts have explored variants of REINFORCE style updates and reward model bootstrapping. often using iterative fine-tuning pipelines. Interestingly, multiple works have reported that iterative filtering and fine-tuning on correct completions can match or exceed PPO performance in some
domains (Gulcehre et al., 2023: Ahmadian et al. 2024).
Two standard benchmarks for mathematical reasoning used in most of the just-mentioned work in RL for reasoning are GSM8K (Cobbe et al. 2021) and MATH (Hendrycks et al. 202 datasets. GSM8K consists of grade-school arithmetic problems requiring multi-step solutions, often accompanied by natural language justifications. MATH contains nigher-difficuly competition-stye math problems with structured step-by-step solutions. Nevertheless, RL approaches can be applied to any task for which verifiers other forms of reward functions are available, including coding (Chen al. 2021
and formal math (Yang et al., 2023).
Alternatively to RL, some natural language questions can be translated into formal logic and answered using automated theorem provers-for for instance, Olausson et al. (2023) maps questions into first-order logic using an LLM, then delegates inference to symbolic prover. AbstRaL also solves GSM8k-style problems by first converting them into abstract symbolic equations and then using a solver (Gao et al. 2025).
Process vs Outcome Supervision [eeaao et al. (2022) compared process-based feedback (reward each correct intermediate step) with outcome-based feedback (reward only at the final answer) on GSM8k. They found final answer accuracy can be similar with outcome-only training, but the quality of the reasoning steps was much higher with process supervision-the process supervision reducing reasoning errors dramatically. This could be another motivation for backtracking methods as the model learns by reinforcing small steps conditioned correct chain-of-thoughts.
16

---

Published as a conference paper at ICLR 2026
Curriculum Learning in Reinforcement Learning Several other works in reinforcement learning have explored curriculum strategies to overcome sparse reward challenges (Narvekar et al. 2020) Florensa et al. (2017) uses reverse curriculum for training, where starting states become increasingly difficult during training. However, compared to |Salimans and Chen (2018), these starting states do not come from demonstrations. Backplay (Resnick al.j [022) demonstrated strong gains in environments like Pommerman and Atari by starting episodes near the goal state using a single demonstration and gradually moving the starting point backward, enabling the agent to outperform
the suboptimal demonstrator, Prioritized Level Replay (PLR) (Jiag et al. 2020) focuses training on the hardest levels in procedural environments like Proegen levels where the agent performs poorly. (Sukhbaatar et al. (201 8b) proposed an automatic curriculum where a teacher agent proposes increasingly difficult tasks for learner agent, leading to emergent complex curriculum strategies without any hand-designed task progression. This was extended in Sukhbaatar et al. (2018a) to learn goal embeddings and reusable low-level policies through self-play
et al. 2019) by adaptively replaying
Another subset of works in curriculum learning (Chen et al. 2025 Shi et al. 2025 Bae al. 2025) primarily adjust dataset/batch-levar curricula, i.e., selecting which questions to include in each RL batch. By contrast, AdaBack introduces per-sample curriculum, adaptively revealing partial rationales while sampling questions uniformly. Thus these approaches are complementary: AdaBack can be combined with batch-level curricula but unlike these works which do not benefit from chain-of-thought (CoT) demonstrations, AdaBack uses COT supervised data to further improve efficiency.
Comparison with State-of-the-art Mathematical Reasoning Models Although recent leader- board results show strong performance, they are not directly comparable to our work because they typically rely on much larger models, additional math-heavy pretraining datasets, or complex prompting and inference- timee searching strategies Xue et al. (2024); Pang et al. (2024b); [iu et al.) (2023). In contrast, our focus is on isolating the contribution of AdaBack itself by comparing it fairly against standard RI. baselines under controlled settings. Importantly, AdaBack is not an alternative to these methods but complementary training enhancement. It can be combined with larger models, post-training strategies, or prompting techniques like chain-of-thought prompting and majority voting Xue et al. (2024) without overhead. This makes AdaBack a foundational component that strengthens models from the ground up and can integrate seamlessly into more complex state-of-the- pipelines Pang et al. (2024b).
B DISCUSSION ON LEARNING PARITIES
The problem of learning parity functions has been extensively studied in the theory of machine learning (Abbe and Sandon 2020 2023 Barak et al., 2022; Abbe et al., 2023a; Edelman al. 2024 Glasgow 2024). Typically. the task is defined as follows each bit in the input is sampled independently and uniformly from the Rademacher distribution (ad(1 (i.e.. -1 or with equal probability). The target function is a parity over subset of bits, i.e., I z; for some S S [d]. The size of S, denoted |SS), is called the degree of the parity. When |S| = Oa(1) is constant (with respect to the input dimension), the problem referred as learning a sparse parity.
Due to symmetry, we often discuss learning a parity of degree l without specifying the particular subset S. Most learning formulations assume the degree is known while the identity of S must be learned, yielding a hypothesis class of size (f). It is well established that the difficulty of learning sparse parities increases with the degree k. In the statistical query (SQ) model (Kearns) 1998), it has been shown that learning a parity of degree & requires ((d5) queries, which translates to (2(d"-1)
samples (Kou et al. 2024).
Similar lower bounds have been conjectured/shown for neural networks. In particular, fully connected networks with bounded width and depth and rotationally invariant weight initializations are conjec- tured to require (2(dmax((- I,1)) online gradient steps to learn degree-k parity (Abbe et al. 2023a). This has been proven under specific assumptions, such as for noisy population gradient descent (Abbe)
and Sandon, 2023 Abbe et al. 2021). Recently, Kou et al. (2024) showed that this lower bound is
'We follow standard theoretical conventions and use 1 values for the bits. Equivalent results hold for 0, 1 bits. Moreover, since Transformers embed inputs into continuous vectors, empirical results are invariant to the bit representation.
17

---

Published as a conference paper at ICLR 2026
tight: variant of SGD (namely, sign-SGD) can learn the task in Ollog d) iterations using batches of size ((d--1). The problem becomes more subtle in the offline setting where samples can be reused. In this case, Abbe et al. (023a) conjecture the optimal sample complexity to be @(dmax(</2.1)) We note that one can often reduce the sample complexity at the cost of using larger models, where the width//epth of the model scales with (Edelman al. 2024).
In our proposed chain-of-parities task, an input sequence of length I. contains T- parity targets of degree 3 and one initial parity of degree 2. Learning all of these parity targets jointly with single model may be harder (due to interference) or easier (due to shared information, such as the influence of previous bit), depending on the model cichiletture-particllle in Transformers. A simplifying assumption is to treat the learning of each parity target as independent. Under this assumption, the problem reduces to learning parities of degree 3, for which, based on the discussions above, one would not expect learnability using only (L)) samples, In contrast, the
positional embeddings
d 2 parity should be learnable with @()) samples (Glasgow 2024 Kou et al. 2024).
This explains why supervised fine-tuning with ((L)) samples fails to weakly learn the degree-3 parity targets Z, for i >> 1. Consequently, SFT followed by standard reinforcement learning also fails, as the RL phase remains unguided which makes the probability of stumbling upon a valid completion (and receiving reward) exponentially small.
We now explain why AdaBack can succeed where SET and RL fail. Consider a scenario in which the first L-1 steps of the solution, i.e.. Y1, Zi..... .......,,,.1.... YL-1, must complete Yr. Zt. In this setup, there are two valid completions: (Y,, 2₁) and its com- plement (Yi, Z1,), where x' denotes the bitwise complement of x. The fact that the same prefix admits two such completions indicates that if Z is parity of the previ-
ZL-1. provided, and the model
Xz, X,,-.., Y Y1,..., 7--1 ous coordinates, then Yr lies the support of this parity. This effectively reduces the problem of learning a degree-3 parity function for Zz, to that of learning a degree-2 parity function (since one bit is already revealed)-4 a task that is learnable with (()) samples. This introduces a setting where AdaBack is able learn the task, while the combination of SFT and standard RL fails.
We do not explore the exact training dynamics here, as they depend on the model architecture and optimization details. Instead, we empirically validate this separation result in Sec. 2.2
C METHOD DETAILS
Improving Convergence via p 0 Injection We observed that always sampling supervision ratios from p.a² U(p P(lx)) led to slower convergence and a distribution mismatch between training and validation. Specifically, when large portion of samples remain difficult (i.e., (()) for many i), the model would be significantly less exposed to fully unsupervised completions during training, while validation is always conducted at p = O.
To address this, we introduced a small amount of supervision-free training into the curriculum: with 10% probability, we sample o,") 0 directly, while using the uniform interval sampling with the remaining 90% probability. This stochastic exposure to 0-portion training helped close the train-test gap and accelerated convergence in the early phases of AdaBack training.
Bootstrapping via Global Averages For samples with no reward history, initialize n from global moving averages Punin and Pmax, which are updated over time using exponential moving averages:
Pmin +- oxpmnn (1 a)) -min
(3)
Pmax f- (p(() max + (1 - a)) Pmax
Here [0, 1] is momentum parameter controlling the smoothing rate.
Choice of the reward threshold 5 Curriculum learning often requires careful hyperparameter tuning-such deciding how long to train at each stage and with what optimizer parameters or
18

---

Published as a conference paper at ICLR 2026
learning rate. AdaBack simplifies this process by collapsing these decisions into single parameter: the reward threshold т. Intuitively, т controls the average reward the model receives; the binary search mechanism ensures that the supervision is adapted to maintain this target. We experimented with several values of r and found the final results to be largely robust, except near extreme values (i.e., 0 1). We chose
0.5 in main experiments, this maximizes reward variance, following the insights Razin et al. (0024) on avoiding vanishing gradients for PPO updates.
Sensitivity to the Reward Threshold We evaluate AdaBack with reward threshold values r E (0.1,0.2...., 0.9) for RL training of the finetuned Llama3.2-1B GSM8k. Figures 5 and 6 report training reward, test accuracy, and the evolution of the average revealed portion over the course of approximately 12k RL iterations.
It can be observed that the learning dynamics is generally similar for all the considered thresholds. only for high thresholds such as = 0.8 and 0.9 that a different behavior is observed the beginning of training. These higher thresholds result in a higher supervision portion and train reward and slightly worse test reward at the beginning of training Nevertheless, as training continues, can be observed that the final performance, train/test rewards, and portions converge to similar values, becoming essentially indistinguishable. This indicates that AdaBack is highly robust to the choice of Given this general robustness, take the value 0.5 in main experiments which aims to maximize the reward variance, consistent with the theoretical studies of Razin et al. (2024).
0.40
0.5
0.35
Than Denwerr
6.30
0.5
0.5
0.30
0.4
0.3
0.
6:5
Daralion
Baration
Figure 5: Training reward (left) and test accuracy (right) across different AdaBack reward thresholds. Although there is some difference at the begining of training for high thresholds (r = 0.8 and 0.9), the learning curves and performance become indistinguishable training goes on.
D EXPERIMENT DETAILS
Computation and Budget We conducted experiments using small language models ranging from IB to 3B parameters. Input sequence lengths varied across tasks, with a maximum input length of 2048 tokens and generation lengths up 2048 tokens. Depending on model size and sequence length, we used nodes equipped with either 4 or NVIDIA A100 GPUs. Including all diagnostic and development runs, our total compute usage amounted approximately 80,000 A100 GPU hours.
Reinforcement Learning Setup We used the GRPO algorithm for all RL experiments, with rollouts (generations) per input sample. Unless otherwise noted, we trained all models for at least 10,000 iterations ensure convergence long-term behavior. To further ensure the fairness of comparisons, we continued each standard GRPO training at least as long as the corresponding AdaBack training. Exploring different learning rates, we found Ir 10- to give stable and rather fast training. We report final test accuracy or reward as the average over the last 5 checkpoints. In rare cases where performance deteriorated at the end of training, we averaged over the last. iterations with increasing reward.
Our RL batch size was 256. We did not use a KL penalty or entropy regularization, neither in the loss nor in the reward.
"Note that these thresholds reveal a larger part of the answer at the beginning, hence, is natural that the test reward is comparatively lower the heginning.
19

---

Published as a conference paper at ICLR 2026
T- 0:1
0.5
0.3
== 0:4
0.4
0.5
T- 0.6
t- 0.7
Portion
0.8
t- 0,9
0.1
0.0
Iteration
Figure 6: Dynamics of the average revealed portion for different reward thresholds. High thresholds such as r = 0.8 and r = 0.9 result a larger average revealed portion in the initial part of training. Nonetheless, as training continues, all thresholds converge stable supervision levels. Combined with Figure this suggests AdaBack's curriculum adapts similarly across a wide range of thresholds.
For supervised-fine tuning of the base models, we used a validation set to adjust the hyperparameters and also the number of fine tuning iterations.
For datasets like Tensor-2 GSM8k and the chain-of of- ofrritiites task, where the output format is nontrivial, we applied a format reward of Fforma 0.1 encourage structured outputs. This was unnecessary for models initialized from SFT, which already generate syntactically valid sequences and thus begin training with an initial reward near 0.1. However, for consistency, we kept this term in all settings.
Chain-of-Parities Task Setup We use 1024 labeled examples with sequence length L 16. In this setting, the probability of generating a valid solution by chance is 2 making reward signals extremely sparse. As result, standard RL fails make progress, even when initialized from an SFT checkpoint. The curriculum introduced by AdaBack mitigates this by incrementally revealing intermediate reasoning steps, allowing learning proceed from the final step backward.
E
LLM USAGE
We have used ChatGPT and Gemini models to improve text quality at phrase and paragraph level.
F ADDITIONAL FIGURES
We show how portions and the train and rewards evolve for GSM8K, MATH, Base-7 GSM8k. and Tensor-2 GSM8k in Figures and respectively.
20

---

Published as a conference paper at ICLR 2026
AdaBack
Standard RL vs. AdaBack
0.7
0.6
Reward
Train Reward
Adaback
0.4
Avg.
0.3
Test Reward
Test Reward
Standard RL
Train Reward
Test Reward
0.0
2000
4000
8000
12000
2000
12000
Iteration
Reration
Figure 7: AdaBack on MATH. Results from training Llama3-3B base model (non-SFT) on MATH dataset. The left column presents AdaBack training dynamics and the right column shows standard RL for comparison.
Standard vs.
0.8
RewardPortion
0.6
Test Reward
Train Roward
0.4
Avg. Portion
Test Reward
Aww Seperte
Standard RL
Reward
0.0
5000
10000
15000
20000
5000
10000
15000
20000
Iteration
lteration
Figure 8: AdaBack B GSM8k. Results from training Llama3-1B IB fine-tuned model (SFT) on Base-7 GSM8k dataset. The left column presents AdaBack training dynamics and the right column shows standard RL for comparison.
AdaBack
Standard RL AdaBack
0.8
Reward/Fartion
0.6
Train Reward
Adaback
0.4
Avg. Portion
Test Roward
0.2
Train Reward
0.0
2000
6000
34000 16000
4000 8000 10000 12000 14000 16000
lteration
Iteration
Figure 9: AdaBack Tensor-2 GSM8k. Results from training Llama3-IB fine-tuned model (SFT) on Tensor-2 GSM8k dataset. The left column presents AdaBack training dynamics and the right column shows standard RL for comparison. Note that for this task, outputting an answer with the correct format has 0.1 reward. So the observed rewards are higher than the actual accuracies.
21

---

Published as a conference paper at ICLR 2026
1.0
0.9
0.8
Reward
0.7
0.6
0.5
0.4
Adaback Train Reward
Standard RL
Train Reward Test Reward
0.3
Test Reward
500
1000
1500
2000
2500
3000
3500
Iteration
Figure 10: Lack of Learning Signal on MATH. On the MATH dataset, Llama 3.2 3B-Instruct shows minimal learning dynamics, with both train and test rewards saturating early. Nearly all questions are solved within few hundred iterations, leaving room for AdaBack provide further benefit. This may be due to the model having been heavily exposed to examples similar to this dataset during pretraining.
0.8
Reward
0.6
0.4
0.2
Qwen2.5-1.55
Qwen2.5-1.55 5-8---ruutt
Train Reward
Train Reward Test Reward
Test Reward
0.0
1000
2000
3000
4000
Iteration
Learning dynamics of Qwen2.5-1.5B models on GSM8k. Training reward and test accuracy for both the base and instruct variants. Both models rapidly saturate the test reward after only a few iterations, cxhibiting limited learning signal, mirroring the saturation effect previously reported for Llama-3.2-3B InIInstruct (see Figure IO). This suggests that the dataset effectively solved by these models during pretraining and provides limited learning signal for RL fine-tuning.
Figure 11:
22


Reviews:
[{"review_id": "qaXUXV74yJ", "reviewer": "Reviewer_gh9Y", "summary": "This paper introduces AdaBack, an adaptive curriculum method for reinforcement learning that dynamically reveals partial solution prefixes (rationales) based on performance feedback. For each example, AdaBack maintains a supervision ratio $\\rho$ that determines the proportion of the solution prefix shown to the model. The model is then trained using standard RL with a verifiable reward on these partial solutions. The supervision ratio is updated via a stochastic “binary search” around a reward threshold. AdaBack aims to bridge the gap between supervised fine-tuning (dense feedback) and reinforcement learning (sparse feedback) by maintaining a non-vanishing reward rate. Experiments on MATH, GSM8k, and the symbolic manipulated variants of GSM8k demonstrate that AdaBack outperforms GRPO, especially on tasks with sparse rewards.", "questions": "- How many random seeds were used for the accuracy results in Table 1?\n- What value of $\\alpha$ was used for the global average initialization, and how sensitive is final performance to this choice?\n- What is R3’s performance on the MATH and GSM8k datasets under the same setup?\n- Have you tested AdaBack on other model families, such as Qwen, to evaluate its generalization? I understand that Qwen may have saturated the MATH dataset, but there are other, more challenging datasets where a dense reward could still be beneficial.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed method improves model performance on mathematical datasets. In particular, it demonstrates greater benefits on the Base-7 and Tensor-2 GSM8k datasets—variants of GSM8k created by symbolic manipulations and effectively reduce the density of reward signals during training. These results align with the goal of addressing the sparse reward problem.\n\n- The paper includes and analyzes negative results (e.g., saturation on MATH for Llama‑3.2‑3B‑Instruct and Qwen‑2.5), which increases the paper’s value for future work. \n\n- The proposed method is well-motivated and clear-explained.", "weaknesses": "- The paper does not specify how many random seeds were used to compute test accuracy. Given the high randomness in LM sampling, this omission raises questions about statistical robustness, especially for Table 1.\n- The method learns a per-problem parameter $\\rho$ and initializes it using a global EMA with hyperparameter $\\alpha$, but the choice and sensitivity of $\\alpha$ are not discussed. \n- Although the approach is inspired by R3, no baseline comparison with R3 is included on the mathematical datasets, leaving open how much of the improvement derives from the adaptive curriculum itself.\n- The experiments are limited to Llama models. Prior studies indicate that model families like Qwen behave differently in RL fine-tuning. Thus, the generality of AdaBack’s gains remains uncertain.\n- The method requires on ground-truth rationales, which constrains scalability and comparability to pure RL approaches that rely only on verifiers.\n- (Minor issue) Some recent work on curriculum learning in RL fine-tuning, such as AdaRFT (Shi et al., 2025), SEC (Chen et al., 2025), and RORL (Bae et al., 2025), is not discussed in the related work section.\n\nShi, Taiwei, et al. \"Efficient reinforcement finetuning via adaptive curriculum learning.\" arXiv preprint arXiv:2504.05520 (2025).\n\nChen, Xiaoyin, et al. \"Self-Evolving Curriculum for LLM Reasoning.\" arXiv preprint arXiv:2505.14970 (2025).\n\nBae, Sanghwan, et al. \"Online difficulty filtering for reasoning oriented reinforcement learning.\" arXiv preprint arXiv:2504.03380 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9qeKnN8IyY", "reviewer": "Reviewer_HHwF", "summary": "This paper proposes AdaBack, an adaptive backtracking algorithm that dynamically adjusts the level of supervision during reinforcement learning training for reasoning tasks. By revealing partial prefixes of target solutions and adapting the supervision ratio based on reward feedback, AdaBack aims to bridge the gap between supervised fine-tuning and RL, enabling models to learn complex reasoning chains more effectively. The authors demonstrate its efficacy on synthetic tasks (e.g., chain-of-parity) and real-world benchmarks (MATH, GSM8k), showing improvements over standard RL and SFT+RL baselines, particularly in out-of-distribution settings.", "questions": "None", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Evaluations span synthetic tasks and real-world reasoning benchmarks, demonstrating broad applicability.\nThe method is well-motivated and clearly explained, with intuitive visualizations of the training process.", "weaknesses": "The approach closely mirrors R3’s reverse curriculum framework, which also uses partial demonstrations to create a curriculum. The distinction—adaptive per-sample scheduling versus fixed stages—is valuable but not thoroughly differentiated in terms of impact.\nWhile outperforming vanilla RL, comparisons to R3 or other curriculum-based RL methods are absent, leaving the reader uncertain about the method’s advantages over existing alternatives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sEv7CApWuQ", "reviewer": "Reviewer_Rx4E", "summary": "Authors identify shortcomings in SFT and RL for sequential reasoning tasks:\n- SFT: data collection is expensive and theoretically insufficient for certain symbolic tasks given insufficient training data\n- RL: outcome rewards are sparse and the policy really gets meaningful signal when a successful trajectory is found which may be exponentially unlikely (in solution length)\n\nAuthors propose a middle ground between SFT and RL that uses RL training with problem contexts augmented with prefixes of SFT train data. Moreover, the proportion of the SFT solution revealed to the model in training is dynamically adjusted based on model performance.\n\nTheir method produces encouraging results on GSM8K and MATH.", "questions": "- How does compute compare across these methods?\n  - SFT has the benefit of not needing to sample new sequences, so having a cost controlled comparison might be instructive.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear conceptual contribution:\n- The paper identifies a meaningful regime between SFT and RL and proposes a principled mechanism (adaptive partial supervision) rather than a heuristic curriculum schedule. This framing is conceptually clean and well-motivated.\n\nPer-sample adaptive curriculum is novel and well-justified:\n- Unlike prior curriculum or backtracking approaches that rely on global schedules or coarse segmentation heuristics (e.g., R3), AdaBack adapts per-instance based on reward signals. This allows difficulty to scale automatically in heterogeneous datasets.\n\nEmpirical Analysis:\n- The training dynamics figures, pass@k exploration analysis, and comparisons across both base and SFT-initialized models provide a clear and multi-angle understanding of how the method behaves. The results are presented in a way that makes the mechanism’s effects interpretable.\n\nPromising results on reasoning benchmarks: On GSM8k (including the proposed Base-7 and Tensor-2 variants) and MATH, AdaBack produces consistent improvements over standard RL pipelines", "weaknesses": "- limited scale: the models explored in this work are limited to 3B parameters\n  - The deepseek R1 report asserts that smaller models have lower capacity from benefiting from RL to discover novel reasoning patterns and are better suited towards SFT based distillation from stronger models (which somewhat addresses the stated drawbacks of SFT, if a teacher model is available to produce substantial distillation data)\n  - this point is somewhat understandable if it's resource constrained, but the scaling behaviour of this approach still should be further studied\n\n- mitigated impact on instruction-tuned models: the comparison between standard RL and AdaBack on SFT initialized models shows minor differences with slow learning overall. SFT mid/post training pipelines have become fairly standardized in most major model releases, so is the proposition to replace these SFT steps with AdaBack instead?\n\n- reliance on ground truth rationales inherits SFT's data collection drawback", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CivClYyc4A", "reviewer": "Reviewer_hbbC", "summary": "The paper tackles the problem of RL for long-horizon, CoT reasoning, where exploration is hard because correct sequences are exponentially rare. It proposes AdaBack, a per-sample curriculum that, for each training example, reveals a variable-length prefix of the gold rationale and uses the reward signal to shrink or expand that revealed prefix over time, effectively doing a stochastic binary search on “how much to reveal.” The method is instantiated on top of GRPO and aims to occupy the underexplored middle ground between fully SFT on rationales and fully unsupervised RL. The authors show (i) on a synthetic chain-of-parities task that SFT, RL, and SFT+RL all fail, but AdaBack succeeds, and (ii) on MATH, GSM8k plus two harder GSM8k variants (Base-7 and Tensor-2) that AdaBack improves over RL and SFT+RL, sometimes even when starting from a base model rather than an SFT one (Table 1). They also identify a regime where AdaBack does not help -- when the base/instruct model has essentially memorized the dataset—arguing that AdaBack’s main value is in exploration-limited settings.", "questions": "1. Could you conduct experiments on a more difficult benchmark such as AIME24 or AIME25?\n2. You mention that AdaBack can be used with non-GRPO RL (via other value estimates). Did you try plain REINFORCE/RLOO, and if so, were the gains comparable?\n3. Could you report variance / multiple seeds for the chain-of-parities experiment in Fig. 2?\n4. How many rollouts per question were used in GRPO, and does AdaBack still help if we reduce that number (i.e., with weaker reward estimation)?\n5. Could you give a systematic sensitivity analysis about the hyperparameter $\\tau$?\n6. In Table 1, SFT+AdaBack on GSM8k 3B is actually worse than AdaBack on the base model (70.7 vs 73.3). Can you clarify this?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Originality: The paper identifies a concrete, underexplored regime “between SFT and RL” and operationalizes it as adaptive partial supervision that is per-sample, not global.\n2. Experiments span standard math (MATH, GSM8k) and two carefully designed variants (Base-7 and Tensor-2) that stress symbolic shift and depth, and AdaBack improves or matches baselines in most cells of Table 1.\n3. Fig. 4 suggest AdaBack expands the solution distribution, not just reweights existing solutions.", "weaknesses": "1. The benchmarks are too easy, especially GSM8k.\n2. The key hyperparameter $\\tau$ effectively sets how aggressively the supervision interval is shrunk/expanded, but there is no systematic sensitivity analysis (e.g., $\\tau$=0.1, 0.3, 0.5, 0.7) to show robustness across tasks.\n3. It seens like AdaBack is not that general. The authors focus on tasks that have a narrow scope.\n4. The method keeps per-sample supervision intervals and falls back to global moving averages when samples are rarely seen; this is acknowledged as a limitation (Sec. 5), but the paper does not quantify how quickly performance degrades as dataset size grows. A synthetic experiment varying number of unique problems would clarify scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)