LILO: Learning to Reason at the Frontier of Learnability
Authors: Thomas Foster
Anya Sims
Johannes Forkel
Jakob Nicolaus Foerster
Venue: NeurIPS 2025 poster ()
Keywords: llm reasoning
reasoning
reinforcement learning
curriculum learning
unsupervised environment design
Primary area: deep_learning
TL;DR: Prove with theory and empirical results that prioritising training on questions with "medium" level of difficulty is beneficial for training reasoning models with RL

Abstract: Reinforcement learning is widely adopted in post-training large language models, especially for reasoning-style tasks such as maths questions. However, as we show, most existing methods will provably fail to learn from questions that are too hard, where the model always fails, or too easy, where the model always succeeds. Much human effort is therefore spent continually producing datasets of questions of a suitable difficulty for state-of-the-art models. Given this, we consider how to algorithmically identify questions that allow for maximally efficient training. We introduce a method, LILO (Learnability Improves LLMs Optimally), that prioritises training on questions with high variance of success, known as learnability, and we provide theory proving LILO maximises the expected improvement of the model. We run a wide range of experiments over multiple base models, algorithms and reasoning datasets to demonstrate that LILO consistently improves final test accuracy and can yield a 3x reduction in the number of training steps required to reach it. We explore how questions with high learnability can be efficiently identified, and discuss how learnability can be scaled to produce LLM agents that autonomously and open-endedly expand the frontier of human knowledge.

LILO: Learning to Reason at the Frontier of Learnability
Thomas Foster
Anya Sims?
Mattie Fellows!
Johannes Forkelt
Jakob Foerster
Abstract
Reinforcement learning is a widely adopted component large language model post-training, especially for reasoning-styl -tyle tasks such as maths questions. How- ever, as we show, most existing methods will provably fail learn from questions that are too hard, where the model always fails, or too easy, where the model always succeeds. Much human effort is therefore spent producing datasets of questions of suitable difficulty for state-of-the-art models. Given this, consider how al algorithmically identify questions that allow for maximally efficient training. We introduce a method, LILO (Learnability Improves L.LSs Optimally), that priori- tises training on questions with high variance of success, known as learnability, and provide theory which shows that LILO enables the expected improvement of the model to be large. We run a wide range of experiments over multiple base models, algorithms and reasoning datasets demonstrate that LILO consistently reaches higher final test accuracy, and can do so in 3x fewer training steps. We explore how questions with high learnability can be efficiently identified, and discuss how learnability can be scaled to produce L.LM agents that autonomously and open- open-endedly expand the frontier human knowledge.
Introduction
Reinforcement learning (RL) has become crucial post-training step of many state-of-the-ae lessoing--focssee large language models (LLMss, notably DeepSeek-RI [1. Tulu (2). and OpenAl's O1 (3. Policy-gradient algorithms suchh as PPO, VinePPO, and GRPO have become the de facto standard, using advantage estimation to reinforce promising sections of answers and penalise errors. However, we show, many existing methods will provably fail learn from questions that hard, where the model always fails, or too easy, where the model always succeeds. Given that RL with LLMs is extremely compute intensive, training on such questions is huge waste of resources. Much human effort and millions of pounds are therefore spent continually producing new datasets of suitable difficulty for state-of-the-an models train on.
In this work, we ask a fundamental question: Can we algorithmically identify questions that are optimal to learn from? To answer this, revisit the formal framing of teaching LLMs to reason with RL, connecting it to the existing literature of automatic curriculum learning (ACL), unsupervised environment design (UED) and learnabiliy- deffied as the variance of success of model over multiple attempts at a given question. We fill a vital gap in the literature, with a proof that the expected improvement of the model constrained by the the learnability of the questions attempted during training.
This motivates to develop novel method for prioritising training questions with high learnability, We name it LILO- "Learnability Improves LLMS Optimally". LILO continuously adapts to the model's ability during training, identifying questions that the model answer correctly, but
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

not consistently, and which, as we will show, are theoretically most useful to learn from. The implementation LILO is extremely simple, and we demonstrate can be added seamlessly to existing training libraries (OatL.M [ and VinePPO [5] in fewer than 20 lines of code. Running L.I.0 requires minimal extra compute cost, exploiting the current trend of generating many attempts question when teaching LLMS to reason with RL.
We evaluate LILO across three RL algorithms (GRPO 66 PPO and VinePPO [ three training datasets of varying size and difficulty (the often standard GSM8K S dataset, the more challenging
MATH 9 dataset, and the larger and more diverse ORZS7K TO and two base models (Rho- IB [[]] and
In doing so, we are the first to present evidence that existing training methods waste significant compute with training on questions with zero learnability. In contrast, LILO prioritises training on high learnability questions which increases final fest accuracy by several percentage points, and can do so in fewer training steps. Furthermore, we evaluate LILO on a variety of unseen test datasets, such as CollegeMath [13] and OlympiadBench [14], and find that adding L.I.O improves reasoning on out-of-distribution questions.
Our theory and results show that Learnability provides a principled approach to selecting training data for reasoning models, and we conclude with a discussion of future work required to develop this line of research into fully open-ended curricula for language model agents.
To summarise contributions:
Theory:
which provides mathematical argument that expected policy improvement is upper bounded by a quantity which increase linearly with learnability.
Section 3 proves Theorem 3.1
Method: Section
presents a simple and efficient algorithm for prioritising training on questions with high learnability.
Results: Section presents the first results that prioritising learnability during the RL training of LLMS impovves training speed by 3x whilst boosting final performance.
Insights: Section contributes numerous valuable insights for training LLMs with RI, uncovering 1) how questions selected by LILO correlate with human-interpretaale factors, 2) that learnable questions become progressively harder to find and 3) how even training highly learnable questions doesn't 't always boost performance, due in part reasoning models' large train-test generalisation gap
2 Background
2.1 Training LLMs to reason with RL
The goal of training LLMs with RL is to maximise the expected reward J(0) = EquP, a>to (r a)), where q is question sampled from distribution P, and is an answer to the question generated by LLM Ny. To find the parameters 8 E θ which maximise J (0). answer generation is framed a Markov Decision Process (MDP), where each state s, in the MDP represents a sequence of tokens, and a policy autoregressively chooses the next token 0 ~ To(a (5)) to be added to the current sequence. For reasoning problems, only once the entire answer has been generated is reward received, i. for a generation of length T, for = O...., T we have r(s:) = O, and only r(sr) can be non-zero. This MDP permits several other equivalent formulations that closely connect it to existing work on ACL, UED and learnability. We discuss this further in Section g and Appendix G
Policy gradient algorithms, such as RLOO [15]. GRPO 6 PPO 7 and VinePOO S are the de-facto standard for training LLMs to reason with RL, optimising (() via the policy gradient theorem
(O) E ΣT Σt= Ve (a,lst) A. The advantages A, O,..,, T characterise how much better given choice of token is compared to the model's current behaviour. The exact way the advantages are computed, and how J()) optimised for, varies for different policy gradient algorithms (see Appendix for more details). In Section we analyse how these choices affect the relationship between expected policy improvement and learnability.
2.2 Learnability
Learnability is a simple way to asses the difficulty of a question with a binary outcome. It is defined in [16 [ [8 as Learnability(aa) = Varx, [r(sT)] Exo
E (r((T) -Ex, [=

---

Pe(1 po) where ps Er, [r(s))] Given K attempts at a question, we estimate learnability by computing the empirical success rate ý = #successes/ K, with learnability then being po1 p), or K R p) with Bessel correction.
For many algorithms, it is simple to show the outcome of training on questions with zero learnability, For example, consider RLOO and GRPO, that compute advantage as A, = r(sT) - Er, [r(*T] or
- [r(sr)] where the success probability Exg [r(sT] and success variance Varz. [r(sr) ]
A, Var+g (+T))
are estimated from a batch of attempts sampled from the model. When learnability is zero, it holds that A, for all 0, ..., 1, and this question has no contribution the model update.
There is minimal prior work examining the impact of training on questions with learnability > 0. Whilst, the authors of [16] show the expected improvement of model trained by the classic algorithm REINFORCE [19] is maximised by maximising learnability, they require a very simplified learning setting with strong assumptions.
3 Relationship between Expected Policy Improvement and Learnability
In this section we introduce Theorem 3.1 which provides a mathematical argument that the policy improvement resulting from one gradient update upper bounded by a quantity which increases linearly with learnability, when using RLOO, GRPO, VinePPO, PPO or any other advantage based policy gradient algorithm. A full proof of Theorem i.T provided in Appendix
We first see that when setting 0x-1 02 + eV) for a learning rate 3 > 0, and assuming that |10x-1 Alll &. is small, we can use the first-order Taylor expansion of ((0++1)
[O, + VV, ((b.) z ((0k) + 31 Vo (0.) This yields that
((x+11) J (0,) ~ 3 V J(0+)1?
i.e. the policy improvement is proportional to the gradient magnitude. Theorem 3.1 then connects the gradient magnitude to learnability:
Theorem 3.1
Consider an MDP in which reward is obtained only in the terminal state 8r, and a parametrised policy Ty. Assume that (V, log no (o (x)))) Vo log (a(ay (se:) and Ag are uncorrelated for all t = 0, ..., T 1. Then the following statements hold:
For A, r(sT), i.e. REINFORCE:
[VJ[(0) < Er,
Vo log (a((aas))))) Eay r(sT)2
For A, = r(ST) - (sT)r( i.c. RLOO:
(1 Exp (r(sr) Emo [r(
[Vo](0)]1 S E.T
Σ
Ve log To(a)(s))
For A, V (8(41)
Th (st). i.c. PPO, VinePPO:
T-1
[Vo](0)] <
Ems
V. log (arlss))))
Ema AP
1=U
E
Vo log To avl solss))]?
Ezo (r(st) Ema [r(
where the second equality holds under the additional assumption that E.. [[[ log Te (ae assumption, it still holds that ET= E.o [A]] = Eno (r(sT) E.o [r{st)])²]]
O,... T are all equal. Even without this additional
3

---

Theorem 3.1,
combined with Equation 1 provides a mathematical argument that the policy improvement obtained from one gradient update upper bounded by the variance of the final reward, times an additional factor. When using an algorithm which follows the gradient of clipped objective, one can expect that a slightly modified version this argument still applies, since all the learning of such algorithms occurs when in the unclipped portion of the objective.
In the case of binary rewards i.e. r(sT) E {0, 1}, such as in reasoning questions, it holds that Em. (r(ST) En, [r(sr)])*] = ps(1 pe). for pa Ex [r(sr]]- Thus for RLOO, PPO and VinePPO, assuming approximately correct estimates of V through multiple rollouts (VinePPO) or a learned value network (PPO), Theorem 3.1 suggests that the magnitude of the policy gradient, and thus by Equation T the expected policy improvement, is upper bounded by learnability times an additional factor.
GRPO uses normalised advantage function, i.e. A, (r(sT) - E., (r(((r)s))))))) //oor)))/////rr/ [r(s))]
Thus see that under the assumptions of Theorem 3.1 it holds that E.,, [O)])
Em. To el²) In other words, the normalisation of the advantage function removes the dependency of the expected gradient magnitude on learnability. However, multiple attempts to replicate Deepseek R1-Zero (1) have found much better performance without this normalisation, i.e.
when using A₁ r(T)) EE (r(T))] as in RLOO, with which the expected policy improvement again upper bounded by quantity which increases linearly with learnability.
Significance tests: We performed numerous statistical tests to validate our assumptions and test Theorem 3.1 end end. These detailed in Appendix C
4 Method
In Section 3, we argued that the expected policy improvement is small when training on questions with smalllearability. Thus, in Algorithm [ we present method that, at every training step I) produces a batch of questions with high learnability and 2) trains on this batch. In Algorithm 2 we introduce simple method produce batch of questions with high learnability, based rejection sampling. The idea to first estimate the learnability a pool of DI candidate questions using a small number Meamability of attempts per question. A larger number Nmmin of attempts per question is then rolled out during training on the top-1 B3] learnable questions.
Algorithm Training with LILO
Input: Initial model parameters: 0, Number train steps: T, Training batch size: [B]
1: for = to T do
2: B + get learnable questions(O, [B) using Algorithm
3:
end for vrain_on_batchl/
Algorithm 2 Get Learnable Questions by Rejection Sampling
Input: Model parameters: 0, Size of question batch return: [B]. Size of candidate pool: ID],
Number attempts per question to calculate learnability: Nicamaiilyy
1: Sample candidate pool of D] questions from dataset
2:
Rollout Nicumability attempts per candidatc question
3: Compute success rate per question ò
Compute learnability per question = p(1 P)
5:
Return top- B questions by learnability
Many large-scale implementations of RLOO, GRPO, PPO and VinePPO for teaching LLMs to reason with RL use large values of Nuain egg VinePPO 5 uses Niain & 500. In this case, taking Nieaaaiility = 8 only requires 4% more samples, which is negligible increase in wall-clock time. The smaller scale implementations of PPO and GRPO that we use in some experiments take Nirain 8, in which case simply reuse the Nicamability samples from the selected top- B questions generated

---

during line 2 of Algorithm 11 and require no further samples during line 3. In this case, the sampling overhead is approximately 4x, which we do not consider an issue for the following reasons:
1. For larger scale implementations with larger values of Nimin rejection sampling adds minimal overhead.
2. Many of our results show that learnability prioritised training plateaus at higher level, and so even allowing baselines infinitely more samples would not give the same level final performance.
3. In this work, we are interested in studying whether training on maximally learnable batches can improve performance in LLMs, and we therefore leave developing algorithms compute learnability more optimally to future work. In Section 9 we provide a discussion of potential directions towards more sample efficient learnability-basel algorithms, including an algorithm that finds highly learnable questions with no extra sampling overhead in Appendix D.1 for use in situations where sampling speed a bottleneck.
4. As we summarise in Table D using learnability to generate samples and then discarding some is a more efficient way to scale than training on all 4x samples, as measured by wall-clock time. This is because generation engines on independent distributed nodes. Increasing training throughput is much harder, requiring the communication of weights and gradients between nodes. For this reason, leveraging additional samples during training LLMs with RL is an increasingly popular trend to compute value functions by VinePPO [5]. and rapidly speed up training with AsyncRLHE (20] and OpenInstruct [2T]. Using reward models with synthetic data also fits this trend, requiring thousands of samples fit the reward model before RL training commences.
be done with highly specialised inference
Paradigm
Algorithm
Sampling time(s)
Training time(s)
Total(s)
Nimin R N Menmability
PPO
20
200
220
Le. Small Nirii PPO, GRPO
PPO + LILO
80
200
280
Nusin >>
Mesmability
VinePOO
1500
500
2000
ic. Large Nuai PPO. GRPO, VinePPO
VinePPO LILO
1550
500
2050
Table 1: Prioritising learnability with Algorithm 2 has minimal impact on the total time for a single training iteration. Numbers taken from PPO and VinePPO runs shown in Section [6] Appendix D. discusses how high learnability batches can be computed without additional samples, for scenarios where sampling speed is bottleneck.
5
Experimental setup
RL algorithms: We add LILO to two existing open-source libraries for training LLMs to reason with
RL. The VinePPO library (5) provides implementations of PPO and VinePPO. The OAT library [4] implements GRPO tuned closely replicate the Deepseek RI results. All of the hyperparameters for PPO, VinePPO and GRPO we leave unchanged from their implementations in (5) and [4] We choose these algorithms to cover two major axes of variation for policy gradient algorithms: 1) the coarseness of credit assignment (GRPO assigns credit at the sequence level, PPO/VinePPO and token level), 2) the use of value functions (VinePPO does empirical rollouts, PPO uses value networks). All our algorithms follow the clipped objective, which has become standard.
Base models: For our VinePPO and OAT experiments, we follow their baseline setup, using Rho- Math-IB and Qwen-2.5-1.55 [2] respectively. In Section 7 we additionally explore whether LILO can be used to squeeze further performance out of models that have already undergone heavy RL training to reach -tate-of-the-art, and so we experiment with Oat-Zero-Ie [22] as the base model.
Hyperparameters: The main hyperparameters for rejection sampling in Algorithm is the size of candidate pool [D], and the value of Nicamability. the number of responses sampled per question. Ideally, these would be tuned to as small as possible whilst achieving batches with high learnability. In practice, choosing DI = 4 x [B| and Nuamabillt = 8 works well with minimal overhead. The only exception is for VinePPO on GSM8K, where the model nearing 95% train accuracy requires |D| 8 x B to produce high learnability batches.
5

---

Algorithm
Train Dataset
Speed-up (Steps)
Final test accuracy (%)
PPO
MATH
2.5x
19.1 21.8
GSM8K
1.9x
51.1 + 53.2
VinePPO
MATH
3.2x
22.8 24.9
GSM8K
3.3x
53.2 55.9
GRPO
ORZS7K
1.5x
35.5 + 37.1
Table 2: Summary of results. We compute speed-up by comparing the training step at which the highest test accuracy of the baseline reached by LIL.O. Final test accuracy compares the performance of the baseline algorithm to the final test accuracy achieved by the algorithm+LII.O.
MATH
GSMBK
Test Accuracy (9%
Test Accuracy
0.20
0.50
0.18
0.45
0.16
0.40
200
400
600
800
1000
100
200
300
400
500
600
Training step (Num model updates)
Training stop (Num model updates)
PPO
PPO
LILO
Adding LILO to PPO increases both the model improvement rate and final test accuracy when training on MATH and GSM8K.
Figure 1:
Datasets: For VinePPO and PPO experiments, we train mathematical reasoning datasets MATH [9) ( 12,000 competition problems), and GSM8K (8), (8,000 simpler grade school problems). These are chosen to demonstrate the effect of Icarnability prioritised training in situations where initial model performance high (GSM8K) and where it is initially low (MATH). We fur- ther evaluate downstream performance of the MATH-trained models CollegeMATH [3] (2,818 college-level questions) and OlympiadBench [4] (8,000 Olympiad level maths and physics competi- tions). For the GRPO experiments, 57,000 questions amalgamated from AIME 233, Numina-Ne Math 233 Tulu3 MATH 2
the OAT AiTrrrbrary and train on the ORZ57K dataset [IÖ] of
and others.
For evaluation, we follow their setup and test 5 datasets MATH Mínerva
24
Olympiad
Bench 14. AMC 23 and AIME 233
We evaluate model performance on the test sets of each dataset, using accuracy (Pass I) as the primary metric. For PPO/VincPPO this is computed using extract string matching as per the VinePPO library. For GRPO, the OAT library is more flexible, attempting to match the answer to many other equivalent mathematical forms.
Metrics:
6 Results
Adding LILO to PPO: We train on MATH and GSM8k using the VinePPO codebase. For MATH, we follow the baseline and run for 1000 timesteps (approximately 8 epochs of the data). The improvement rate is almost doubled by adding LILO, increasing by 4.8% compared to 2.1% over the course of training. PPO+LIILO reaches the best performance achieved by PPO in 2.5x fewer training steps. When training on GSM8K the runs plateau after 650 steps, with PPO+LILO achieving a slightly higher final test accuracy. It achieves this final test accuracy in 1.9x fewer steps than PPO without Learnability.
Adding LILO to VinePPO We again compare on MATH and GSMSK and find that VinePPO also benefits significantly from prioritising learnability. On MATH, it achieves 2.1% higher best test accuracy, and achieves the best test accuracy of VinePPO without learnability in 3.2x fewer steps. On GSM8K we find that finding learning questions via rejection sampling (Algorithm (2) with [D] = 4 x [B] cannot consistently learnable questions during training, as train performance closing in on 95%. We increase [D] = 8 x [B], which is enough to find learnable questions consistently enough during training to boost performance. VinePPO without learnability improves performance on GSM8K from 40.5% to 53.2%, gain of 12.7%. Prioritising lcarnability improves on this, taking

---

MATH
GSMBK
0.24
Test Accuracyy (59
0.35
Teat Accurocy
0.50
0.20
0.18
0.40
200
400
600
800
1000
100
200
300
400
500
600
Training step (Num model updates)
Training stop (Num model updates)
VinePPO
VinePPO LIOO
VinePPO LIO ([D]=ax]B|)
Figure 2: Adding LILO to VinePPO increases both the model improvement rate and final test accuracy when training on MATH and GSM8K. During training on GSM8K, train accuracy reached nearly 95% (see Figure D and LII.O began to struggle to find high learnability questions (as shown in Figure 6 Doubling the size of the candidate pool [D] in Algorithm fixed this and further improved performance (mint green line).
test accuracy from 40.5% 55.9%, gain of 15.4%. VinePPO with learnability reaches the best accuracy achieved without learnability in 3.3x fewer training steps.
We evaluate the four combinations of PPO/VinePPO training MATH/GSM8K the holdout datasets CollegeMath and Olympiad Bench. On all four combinations, learnability improves perfor- mance, for full results see Table 6 in Appendix F
Average accuracy over benchmarks
Adding LILO to GRPO Following the OAT library, we train Qwen-2.5-1.5 on ORZ57K using GRPO. Learnability again significantly improves the training dynam- ics, reaching a higher final test accuracy and the best test accuracy of GRPO without L.IOO in 1.5x fewer steps. Test accuracy is averaged over the 5 datasets: MATH, AIME, AMC, Minerva and Olympiad Bench. The individual training curves for each benchmark are shown in |Figure 4| where learnability significantly im- proves performance in all bar AIME, where accuracy is similar to the baseline.
0.37
GRPO
GRPO
0.36
Test Acc (96
0.35
0.34
0.32
Step
Figure 3: Adding LILO to GRPO training on ORZ57K nearly doubles the model improvement rate.
Amc
Olympiad Bench
Minerva
Aime
2.10
0.12
0.19
2.17
6.10
1.40
8.17
5.14
Skpp
Skp
CEPO
C3FO
Figure 4: Adding LILO to GRPO increases the model improvement rate on 4/5 unseen test datasets.
7 Further analysis
LILO introduces an interpretable training curriculum In Figure 5) we plot the average number of reasoning steps in the gold solution to each question trained at given iteration. Note that these gold solutions are not used to calculate learnability, but provide human- interprtattrr way to see what type of questions questions (fewer steps) but gradually include harder ones. On GSMSK, the model quickly masters the average question, prompting us focus more difficult ones.
being prioritised during training. For MATH, initially select simpler
Learnability doesn't solve RL's generalisation gap The results in the previous section show that RL training can increase the test accuracy of LLMs, and learnability helps this happen faster and to

---

greater extent. However. the improvements, even with learnability, are modest, with no algorithm allowing Rho-Math-IDe -B or Qwen-2.5-1B to perfect MATH. GSM8K or other datasets. However, looking at training accuracy (full training curves in Appendix Fj. we see that training accuracy is often nearly 2x higher. the GSM8K experiments, for example, the training accuracy nears % whilst test accuracy stays below 60%. While LILO causes train accuracy to increase faster, the ratio between train and test accuracy is unchanged. We plot this in Figure [0 Appendix F
Finding learnable questions gets harder through- out training F that during PPO training on MATH, we remain able to select high learnability batches. For GSM8K, rejection sampling starts be unable to find high learnability batches. This could be a factor in why PPO closes in on PPO Learnability later in training on GSM8K, whereas for MATH the two methods continue to diverge. It motivates future work dynamically choosing N at each iteration based on train accuracy, and generally spending more compute for finding learnable questions later in training.
MATH
Prosxem aentem-diiaan dificilty (anauning slens)
10
100
400
Training step updeless)
GSMK
5.2
Probiem difficllty
5.0
4.8
Learnability isn't a silver bullet We took Oat-Zero-
4.8
1.5B [22] a state-of-the-an finetune of Qwen-2.5- 1.5B [12]. and tried to further improve it on its ex- isting dataset by prioritising questions that still had high learnability. Despite being able to find batches of highly learnable questions, performance only min- imally improves. This could be due to many reasons lack capacity in the model, entering the overfit- ting/no generalisation regime, learning new ques- tions whilst forgetting old ones. We are excited to
4.4
200
400
600
800
1000
Training (rum model updates)
Questions selected by LILO cor- relate with the number of reasoning steps in the gold standard solution, despite LII.O not having access this information.
Figure 5:
investigate this further.
Tabela batch
GSMBK
40.0
0.24
02-
0 37.5
0.23
$ 35.0
0.22
0.1
0 32.5
(Oat-2ero-1133)
GIPO LILO
0.21
2000
40CU
2500 5000 3COCU
30.0
50
100
0.20
150
Tain step
updates)
fnum updatest
Training stop
Training slep
PRO
PPO LEG
Figure 7: Despite the prevalence of highly learn- ability training questions, GRPO+LILO only minimally improves Oat-Zero- This may be due to the model already having undergone sig- nificant post-training.
Figure 6: LILO struggles to find high learn- ability questions towards the end training. This happens to lesser degree MATH than on GSMSK.
8 Related work
Curricula with LLMs for RL: Rho- lb [1] shows the effectiveness of curricula during supervised training of LLMs, but prior to the initial release of this paper, there was minimal work applying curricula to RL on LLMs. While [25) try PLR [26) and backtracking [27] they see no improvement over PPO. DPOP (88 applics various hcuristics to improve the diversity of question attempts for use with DPO, but the heuristics not linked to model performance and do not change during training.
Since the initial release of this paper in February 2025, several works have been released that validate some of our findings. DAPO 29 filters out training on questions that produce all successful or all failing attempts. Kimi- questions. "Not all rollouts are useful" [31 trains on subset of question attempts with the highest variance. They provide no theory to explain why this useful and present only a single experiment with GRPO GSM8K. Llama 4 322 uses curriculum of increasingly difficult questions and filters out questions with zero advantage. They provide no theory, training code experiments to validate this choice.
30 samples questions proportional
- p. i.e., upweighting difficult
8

---

Unsupervised Environment Design: A UED problem features a family of MDPs parametrised by set of values known as "level" 33 26 The goal is to vary the 0 used during training such that performance remains high on unseen or adversarially chosen 0 at test time. The majority of previous work on learnability and UED has been on embodied-agent style robotics tasks, such as Minigrid [44, XLannd-Mnigrid [35]. JaxNav [36] and variants OpenAT's bidpeal walker [37]. However, one can view each question a reasoning dataset as a "level" of the answer-generation MDP. Viewing RL on LLMs as UED problem opens up a whole world of literature that could be adapted for LLMs to further improve capabilities. Notably, SFL [T8], which inspired early work on this paper, trains on both high learnability and randomly sampled levels throughout training on the robotics tasks above. Having proved Theorem 3.1 we train only on high learnability questions. Many elements of SFL are unsuitable for RL LLMs: Notably, maintaining buffer of high learnability questions that is only periodically refreshed leads to massive overfitting in LLMs (see Figure T0 in Appendix F). Sampling is slower for LLMS than in their JAX (38) environments, so 1) rollout for fixed number of episodes, rather than timesteps, which allows us to use fast inference engines and 2) reuse samples from learnability estimation in training. For further related work on UED and its connection LLMs and RL, see Appendix G
Limitations and future work
There are many exciting ways to further reduce LILO's sampling overhead. Appendix D.I details one such way and presents some initial results. As suggested in Section 7 it is likely more efficient to dynamically allocate more compute to finding learnable questions as it gets more difficult later in training. LII.O currently discards useful data if more than [ high learnability questions are identified. Making LILO fully asynchronous, similar to Async (20). could alleviate this. LILO also stateless no information from previous steps is used to estimate learnability in the current step. Future work could also explore using the LLM itself to estimate learnability without any actual question attempts. This is similar to some of the work on LLM confidence estimation and calibration [39].
LILO uses learnability to prioritise existing training data. Future work could look at using learnability difficulty. This could build on the UED
as metric to produce new training data at the correct level of algorithm ACCEL [40] for mutating questions, and existing work for automatically producing RLVR tasks 41.
Learnabllity does not account for inherent aleatoric uncertainty in question, e.g. "I just flipeed coin. Is it head or tails?". Such questions may not be actually learnable by the model, despite having high variance of success, and thus high learnability score. There is also no quantification of whether a question is "worth learning", e.g its relevance to the test set.
Learnabliity in prior work was only considered for binary rewards. Our Theorem suggests that more generally, including for non-binary rewards, that maximising learnability. defined as the variance of the final reward, maximises expected model improvement. However, more experiments are needed to empirically verify this. There is a growing body of work looking at unsupervised RL, in which no rewards, even tasks, are given to the model. In this situation, the agent simply explores to better understand the world around them. remains to be seen how this concept, previously implemented for
or efffusion-based world models, can be translated into the LLM domain.
10 Conclusion
In this work, revisited the formal framing of training LLMs reason using RI., connecting it to existing theories of learnability, active curriculum learning (ACL), and unsupervised environment design (UED). We contribute new theoretical results showing that prioritising learnability during training maximises expected policy improvement. Building on this, we propose LILO, simple, practical method for prioritising learnable questions during LLM training, and demonstrate its ease of integration by adding it to two widely used training libraries. We evaluate our approach using three algorithms (GRPO, PPO, and VinePPO) across three reasoning benchmarks (MATH, GSM8K, and ORZ57K). Our results show that LILO not only improves final performance by several percentage points but also requires 3x fewer training steps to exceed the same performance of training without LILO.. We conclude by discussing the limitations of current learnability- methods and outlining future directions toward developing L.MM agents capable of autonomously and continually expanding the boundaries of human knowledge.

---

References
[1] DeepSeek-AI, D. Guo, D. Yang. H. Zhang. J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang. X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng. C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu,
H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan,
J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang.
K. Yu, L. Wang. L. Zhang, L. Zhao, L. Wang, L. Zhang. L. Xu. L. Xia. M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang. Q. Chen. Q. Du, R. Ge. R. Zhang, R. Pan. R. Wang. R. J. Chen, R. L. Jin, R. Chen, S. Lu. S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu. S. Zhou, S. Pan, S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei. T. Sun, T. Wang, W. Zeng. W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang. X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang. X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang,
Y. Yu, Y. Zhang, Y. Shi, Y. Xiong. Y. He, Y. Piao, Y. Wang. Y. Tan, Y. Ma, Y. Liu, Y. Guo,
Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X.
Zhu, Y. Xu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao. Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan. Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang, "Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning," " 2025. [Online]. Available: https://axii..rr/ab///55....1. 12948
[2] N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang. J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, Soldaini, N. A. Smith, Y. Wang. Dasigi, and H. Hajishirzi, "Tulu 3: Pushing frontiers in open language model post-training." 2025. [Online]. Available:
https//arxiv.ogg/bb//4111.1
[3] OpenAl, "Learning to reason with Ilms," 2024. [Online]. Available: https//opeenai.comminde/
[4] Z. Liu, C. Chen, W. Li,, P. Qi T. Pang, C. Du, W. S. Lee, and M. Lin, rl--eeer training: A critical perspective," " 2025. [Online]. Available: https://axivvorr//b2225552..7.33
[5] A. Kazemnejad, M. Aghajohari, E. Portelance, A. Sordoni, Reddy, A. Courville, and N. Roux, "Vineppo: Unlocking rl potential for IIm reasoning through refined credit assignment," 2024. [Online]. Available: https://arxiv.or//b////////
[6] S. S. Ramesh, Y. Hu, I. Chaimalas, V. Mchta, P. G. Sessa, H. B. Ammar, and I. Bogunovic, "Group robust preference optimization in reward-free rlhf." 2024. [Online]. Available: https//arkw.orr//bb//4455..22
[7] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," 2017. [Online]. Available: hutps:/aaxi..orr/ab///0777
[8] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, I.. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, "Training verifiers to solve math word problems," 2021. [Online]. Available: https://arkiv.or//ab//111 h0tp://wwww..oraaa//11111111101. 14116
[9] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, F. Tang, D. Song, and J. Steinhardr, "Measuring mathematical problem solving with the math dataset," 2021. [Online]. Available: https:/arriv.ogg//b///00.....00
[10] J. Hu, Y. Zhang. Q. Han, D. Jiang, X. Zhang, and H. Y. Shum, "Open-reasonerr-ero An open source approach to scaling up reinforcement learning on the base model," 2025. [Online]. Available: https://arxiv.orr/ab//25533 250.//2aab/233222900
[11] Z. Lin, Z. Gou, Y. Gong, X. Liu, Y. Shen, R. Xu, C. Lin, Y. Yang, J. Jiao, N. Duan, and W. Chen, "Rho-I: Not all tokens are what you need," 2025. [Online]. Available:
https//axxivorr////s/2444...499
10

---

[12] Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Ci, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, Tu, J. Zhang, J. Yang. J. Yang, Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang. I. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu, "Qwen2.5 technical report," 2025.
[13] Z. Tang. X. Zhang, B. Wang, and F. Wei, "Mathscale: Scaling instruction tuning for mathematical reasoning," " 2024. [Online]. Available: https//ariiv.org/ahs2200......
[14] C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, Liu, L. Qi, Z. Liu, and M. Sun, "Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems," 2024. [Online]. Available: https///arxiv.or/abbs//// 2000
[15] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker, "Back to basics: Revisiting reinforce style optimization for learning from human feedback in Ilms," 2024. [Online]. Available: https:/aarxv.oor/////442.111
[16] G. Tzannetos, B. G. Ribeiro, P. Kamalaruban, and A. Singla, "Proximal curriculum for reinforcement learning agents," 2023. [Online]. Available: hutpss//arxiv.orr/bb///33 12877
[17] C. Florensa, D. Held, X. Geng, and P. Abbeel, "Automatic goal generation for reinforcement learning agents," in Proceedings of the 35th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, J. Dy and
A. Krause, Eds., vol. 80. PMLR, 10-15 Jul 2018, pp. 1515-1528. [Online]. Available: https//proceedins..mrr..p wrw.smll.rress///ttrereeeoo (aaeenaaaaa8mhmmmm ht.l
[18] A. Rutherford, M. Beukman, T. Willi, B. Lacerda, N. Hawes, and J. Foerster, "No regrets: Investigating and improving regret approximations for eurriculum discovery," 2024. [Online]. Available: https://ardw..rr//bs/24... 15599
[19] R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning," Machine Learning, vol. 8, no. 3. pp. 229-256, May 1992. [Online]. Available:
htps://doi.or// 1007/B100392696
[20] M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville, "Asynchronous rlhf: Faster and more efficient off-policy for language models," 2024. [Online].
Available:
[21] J. Hu, X. Wu, Z. Zhu, Xianyu, W. Wang, D. Zhang, and Y. Cao, "Openrlhf: An casy-to-use, scalable and high-performance rihf framework," 2024. [Online]. Available: https:/arrw..org/bb//44...111
[22] Z. Liu, C. Chen, W. Li, T. Pang, C. Du, and M. Lin, "There may not be aha moment in rl-zero-like training - a pilot study," http:///aalll...oioossss///////e
2025, notion Blog.
[23] L. B. S. S. H. Y. Z. Z. B. D. L. Z. Y.F. L. Jia LI, Edward Beech- wtwp:////ppoctccummaaaaaaaaaaaaa numina/aimo -ootinooo-rrssss-eeo prize (htips: ing and S. Polu, "Numinamath," wwwpress-prize/blloeeeee ppizecblob/mmn/reppo//uuu
httpa//poocectnmmnnaaaaaa /github.com/project- numina/aimo Hammina/ammo-pogreesss
ddta 2024.
[24] A. Lewkowyzz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Sole Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra, "Solving quantitative reasoning problems with language models," t 2022. [Online]. Available: https://arxiv.org/bbs22222 14888
[25] A. Havrilla, Y. Du, S. C. Raparthy, C. Nalmpantis, J. Dwivedi- Yu, M. Zhuravinskyi, E. Hambro, Sukhbaatar, and R. Raileanu, "Teaching large language models to reason with reinforcement learning," 2024. [Online). Available: hutps://arrv.oog//b2244.....6.62
[26] M. Jiang, E. Grefenstette, and T. Rocktäschel, "Prioritized level replay," 2021. [Online]. Available: httpss/aariivrr/abb20000...333
[27] Salimans and R. Chen, "Leammin montezuma's revenge from a single demonstration," " 2018. [Online]. Available: https:/aarii.oog/a/////12...3311

---

[28] A. Pal, D. Karkhanis, S. Dooley, M. Roberts, S. Naidu, and C. White, "Smaug: Fixing failure modes of preference optimisation with dpo-positive." 2024. [Online]. Available: https///aiiv.orr///////222222222
[29] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo. Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W.-Y Ma, Y.-Q. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang. "Dapo: An open-source Il reinforcement learning system
scale," 2025. [Online]. Available: hhtps///axii..rg//a/255....114
[30] K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, C. Tang, C. Wang, D. Zhang, E. Yuan, E. Lu, F. Tang, F. Sung. G. Wei, G. Lai, H. Guo, H. Zhu, H. Ding, H. Hu, H. Yang. H. Zhang, H. Yao. H. Zhao, H. Lu, H. Li, H. Yu, H. Gao, H. Zheng, H. Yuan, J. Chen, J. Guo. J. Su, J. Wang. J. Zhao, J. Zhang, J. Liu, J. Yan, J. Wu, L. Shi, L. Ye, L. Yu,
M. Dong, N. Zhang, N. Ma, Q. Pan. Q. Gong, S. Liu, S. Ma, S. Wei, S. Cao, S. Huang, T. Jiang, W. Gao, W. Xiong, W. He, W. Huang, W. Wu. W. He, X. Wei, X. Jia, X. Wu, X. Xu, X. Zu,
X. Zhou, X. Pan, Y. Charles, Y. Li, Y. Hu, Y. Liu, Y. Chen, Y. Wang. Y. Liu, Y. Qin, Y. Liu,
Y. Yang. Y. Bao, Y. Du, Y. Wu, Y. Wang, 7. Zhou, Z. Wang. 7. Li, Z. Zhu, 7. Zhang, Z. Wang. Z. Yang, Z. Huang, Z. Huang, Z. Xu, and Z. Yang, "Kimi k1.5: Scaling reinforcement learning with Ilss," 2025. [Online]. Available: https//arkivoorg/aa2255511..555
[31] Y. E. Xu, Y. Savani, F. Fang, and Z. Kolter, "Not all rollouts are useful: Down-sampling rollouts in Ilm reinforcement learning." 2025. [Online]. Available: hutpss//arriv.oog/bb///5044 13818
[32] Meta AI, "The llama 4 herd: The beginning of a new of natively multimodal ai innovation," " April 2025, accessed: 2025-05-16. [Online]. Available: https://a.meeaacom/bblg////
[33] M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine, "Emergent complexity and zero-shot transfer via unsupervised environment design," " 2021. [Online]. Available:
[34] M. Chevalier-Boisvers B. Dai, M. Towers, R. Perez-Vicente L. Willems, S. Lahlou, S. Pal, P. S. Castro. and J. Terry, "Minigrid &amp; miniworld: Modular &amp; customizable reinforcement learning environments for goal-oriented tasks," in Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds.. vol. 36. Curran Associates, Inc.. 2023, pp. www.eeeeiiinnnnnnnss...eeee wweurips.cc/pappe
33333-73 394. [Online]. Available: httpss//pproceedings c8916198466e8e122 (asaa2115449114/ (a1199911999--PPP Papee-
and Benchmarks.pu tilex/paper/2222/iil/
[35] A. Nikulin, V. Kurenkov, I. Zisman, A. Agarkov, V. Sinii, and S. Kolesnikov, "Xland-minigrid: Scalable meta-reinfforeement learning environments in jax," 2024. [Online]. Available: https:/aariv.og//bb331111222
[36] A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. Ingvarsson, T. Willi, R. Hammond, A. Khan, C. S. de Witt, A. Souly, S. Bandyopadhyay, M. Samvelyan, M. Jiang, R. T. Lange, S. Whiteson,
Lacerda, N. Hawes, T. Rocktaschel, C. Lu, and J. N. Foerster, "Jaxmarl: Multi-agent rl environments and algorithms in jax," 2024. [Online]. Available:
hutps//aaxiv..og/ab/3311111 10090
[37] G. Brockman, V. Cheung, I.. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba, "Openai gym," 2016. [Online]. Available:
[38] Bradbury, R. Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Miine and Q. Zhang. "JAX: composable transformations of Python+NumPy programs," 2018. [Online]. Available: http://gtttubbcom/jaa-//j/
[39] J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus, "Measuring short-form factuality in large language models," 2024. [Online]. Available:
https:/aaxi..oog/b//211....1
[40] J. Parker-Holder, M. Jiang, M. Dennis, M. Samvelyan, J. Foerster, E. Grefenstette, and Rocktäschel, "Evolving curricula with regret-based environment design," " 2023. [Online].
Available: http:://axiivorr/aa//220......00
12

---

[41] A. Zhao, Y. Wu. Y. Yue, T. Wu, Q. Xu, Y. Yue, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang, "Absolute zero: Reinforced self-play reasoning with zero data," 2025. [Online]. Available: https//arriv.ogg/a////555..333.
[42] R. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. The MIT Press, 2018. [Online]. Available: http:///ccmmpeeeeeeeeeeaeebbbbbe htw.o/////heeooooo.....mm
[43] Y. Bengio, J. Louradour, R. Collobert, and Weston, "Curriculum learning," vol. 60, 06 2009,
6.
[44] Elman, "Learning and development in neural networks: The importance of starting small." Cognition, vol. 48, no. 1, pp. 71-99, 1993.
[45] Loshchilov and F. Hutter, "Online batch selection for faster training of neural networks," 2016. [Online]. Available: https//aaaii.oorgaab//511.1.
[46] K. Kawaguchi and H. Lu, "Ordered sgd: new stochastic optimization framework for empirical risk minimization," 2020. [Online]. Available: https://arxiv.or//bbb/////
[47] A. H. Jiang, D. L. K. Wong, G. Zhou, D. G. Andersen, Dean, G. R. Ganger, G. Joshi, M. Kaminksy, M. Kozuch, Z. C. Lipton, and P. Pillai, "Accelerating deep learning by focusing on the biggest losers," " 2019. [Online]. Available: hutps://arxxv.org//bbb// httpaasss//9999 9910007622
[48] D. Weinshall, G. Cohen, and D. Amir, "Curriculum learning by transfer learning: Theory and experiments with deep networks," 2018. [Online]. Available: https://aaxiv.rr//bb///000 03020000000000007077706
[49] B. Settles, "Active learning literature survey," University of Wisconsin-Madison Department of Computer Sciences, Technical Report TR1648, 2009. [Online]. Available:
http://digitall. hit.iiiiiii..ibbrr..... wiwrary. 1793/60660
[50] M. Li and I. Sethi, "Confidence-baat based active learning." " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 8, pp. 1251-1266 2006.
[51] C. Coleman, C. Yeh, S. Mussmann, B. Mirzasoleiman, P. Bailis, P. Liang, J. Leskovec, and M. Zaharia, "Selection via proxy: Efficient data selection for deep learning," 2020. [Online]. Available: httpss//arxivoogg/abb///0......
[52] S. Mindermann, J. Brauner, M. Razzak, M. Sharma, A. Kirsch, W. Xu, B. Höltgen, A. N. Gomez, A. Morisot, S. Farquhar, and Y. Gal, "Prioritized training on points that are learnable. worth learning, and not yet learnt," 2022. [Online]. Available: https://ari..or/aa//2066..1111
[53] H. Trittenbach, A. Englhardt, and K. Böhm, "An overview and a benchmark of active learning for outlier detection with one-class classifiers," " 2019. [Online]. Available: https://arxiv.org/bbb/// 000.b////188870077999
[54] T. B. Johnson and C. Guestrin, "Training deep models faster with robust, approximate importance sampling," in Advances in Neural Information Processing Systems. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018. [Online]. Available: httpp://ppoeeeeddnnneeusspsspppp files/ files/ http://00///he/9999ddeeeebea//aa
[55] A. Katharopoulos and F. Fleuret, "Not all samples are created equal: Deep learning with importance sampling," in Proceedings of the 35th International Conference on Machine Learning. ser. Proceedings of Machine Learning Research, Dy and A. Krause, Eds., vol. 80. PMLR, 10-15 Jul 2018, pp. 2525-2534 [Online]. Available: 1aa.ooulloosstmmmiml 8t..
[56] M. Kumar, B. Packer, and D. Koller, "Self pacef learning for latent variable models," in Advances Neural Information Processing Systems, J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, Eds., vol. 23. Curran Associates, Inc., 2010. [Online]. Available:
13

---

[57] A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu, "Automated curriculum learning for neural networks." " 2017. [Online]. Available: httts://axi.orr//bb/////.0.00000
[58] Matüise, A. Oliver, T. Cohen, and J. Schulman, "Teacher-student curriculum learning," " 2017. [Online]. Available: https://arrv..oo///////70..0000
[59] N. Vithayathil Varghese and Q. H. Mahmoud, "A survey of multi-task deep reinforcement learning." " Electronics, vol. 9, no. 9, 2020. [Online]. Available: https//www..mpp.com
2079-9292/9/91136
[60] Y. Zhang, Abbeel, and L. Pinto, "Automatic curriculum learning through value disagreement," 2020. [Online]. Available:
1
[61] R. L. Gutierrez and M. Leonetti, -hooori task selection for meta-reinforeeemet learning," " 2021. [Online]. Available: hutps///axiv.oo//as///111..0
[62] Pitis, H. Chan, S. Zhao, B. Stadie, and J. Ba, "Maximum entropy gain exploration for long horizon multi-goal reinforcement learning," 2020. [Online]. Available: https:/aaniv.org//bb//00......2
[63] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine, "Skew-fit: State-covering self- supervised reinforcement learning." " 2020. [Online]. Available: https://anxiv..rr/ab///003. 03698
[64] Fournier, O. Sigaud, C. Colas, and M. Chetouani, "Clic: Curriculum learning and imitation for object control in non-rewarding environments," 2019. [Online]. Available:
http://axiivoor//bb///00...2222
[65] Dendorfer, A. OSep, and Leal-Taixé, "Goal-gan: Multimodal trajectory prediction based on goal position estimation," 2020. [Online]. Available: http://arrw.orraah/2010011111111
[66] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, "Prioritized experience replay," 2016. [Online]. Available: http::/arrav.aggab////1.1.0999
14

---

Supplementary Material
Table of Contents
Policy gradient algorithms
16
Proof of 3..orem
17
Significance tests
18
D Additional method
19
D.1 An algorithm for selecting high learnability questions with no additional sampling cost
19
Additional experimental setup details
20
E.1 Alterations to VInePPO training setup
20
Compute resources
20
Additional results
20
Further related work
24
G.I
ACL in Supervised Learning and Bandit Problems
24
G.2 ACI. in Multi-Task Reinforcement Learning
24
ACL in Goal-Conditioned RL
24
G.4
Unsupervised Environment Design
24
Examples of high and low learnability questions
26
15

---

A Policy gradient algorithms
The standard RL framework [42] considers an environment formulated as a Markov Decision Process (MDP) M = (S, A, T, R. 7).7), where S and A denote the state and action spaces, T(s++1]s;, az) denotes the transition dynamics, R(st, ae) denotes the reward function, Ao the initial state distribution and 7 E (0, 1) is the discount factor. Given an episode of length T, (s, ag: S1, 01 R ... -T: ST- 1
Rr-i] with actions sampled by the policy To [s)] the goal is to find the policy that maximizes the expected discounted return:
E poo St T
Σ t=0
Y'RR Rs
(2)
This objective can be maximized using gradient ascent with the following gradient estimator:
VoJ( =
Σ Ve log my mp (x) Re
D.D. a~Tx(-1as)
For lower variance gradient estimator, the reward Rg can be replaced with the 'advantage' A,, which quantifies how much better taking action aj in state s; is compared to the average action according to the current policy Ty. Given learning rate B, the policy parameters are updated iteratively with:
0x+1
02 + 3V (0x)
We now consider applying this RL framework question-answeriing in L.Mss, denoting the sequence of tokens that make up the question and LLM response as x = Io:n and y = your respectively. There multiple different treatments of the RL framework for LLMs. The simplest view treats response generation as "contextual bandits and to consider the question to be the initial state so = To.n. the generation of an entire response be single action do yo.7, and thus the horizon to be T 1. An alternative is to view response generation as a token-level MDP, so the state is the sequence of previous question and response tokens s = [x. 10: the action is the next token in the response a, = y/e. and the transition function is deterministic and simply involves concatenating the new token yt onto the previous state to get s = [x. Y0:z]. In RL applied LLMs single reward R is typically only received after the whole response has been generated. Therefore with on-policy data the T contextual bandits treatment and the token-level treatment with y = are equivalent.
In deep RL the advantage is typically estimated as A(a,, 82) = R(a, st) - V (s²). where Vi (sx), an additional learned value baseline, is subtracted from the reward. In LLMs. however, fitting value functions has proven tricky and instead sample-based approach involving sampling multiple responses for each prompt has become popular. There are several different popular expressions for advantage. RLOO [IS] samples G responses Je= per question and uses A = R(O)
T Dg jtz RO), Similarly, GRPO makes use of G sampled responses per prompt, instead using A (R mean, / R
This is generally applied token-wise, resulting in the following gradient estimator:
G T
(o) = E x-D, ( (-1x)
ΣΣ 1=0
V log no (vl" x, Borl-1) (())
(5)
While RLOO and GRPO assign the same advantage A for all response tokens, VinePPO [3] aims for fine-grained credit assignment. Like RLOO and GRPO, they again use a sample- based approach instead of relying on a learned value network, and estimate y [x, yo:+] by sampling multiple completions 1:7 onwards using the current policy mp and averaging their final rewards. The advantage then calculated as A, ( for V This approach avoids the difficulties of training separate value network for LLMs while still enabling token-level credit assignment.
(st), where sample-hased estimate used
PPO style clipping is often applied to increase sample efficiency by enabling use of off-policy data.
'The GRPO advantage has been found to perform better without the division by ( ( 4
16

---

B Proof of Theorem 3.1
Proof. We see that
[ B
log (4
<Bn
Vo log To (arls))A,
ErD Ve log (a (s)) Vo log (o(a)a
Ve (arlss)) Vo log (e((arls))) Ero [ArAe],
6,4=0
where in the second inequality we have used Jensen's inequality, and the last equality we have used the assumption that (Vo log my (ay )s)) Ve log No no( aa (yy) and Ar are uncorrelated for all t' ...,, 1. This proves the statements for REINFORCE, where A r(sT). and RLOO, where A, = r(sr) Ex [r(sr)]. For PPO and VinePPO, where A = V ($(+1) - V (8t). we now show that Ex, [A,Ar] = for t A t', and Ens [A]] Exe [V (st+1)²- * (s.)
Together with the fact that r(ST) V (su) and Ex, ir(sT] V V (sp). this will finish the proof of the theorem.
We that
Ex [1=00$$+)) - (s:))
=Bro
[VVT (s441) - (8
V (s:)) + (=(())))))
-Ero
[(V (8(41)) 2 ($:41) (s:)) + V (s,)
-Ero
VTT (T0(()+))) -2 (1)
Σ Ta(ar|s)) Σ (e (st)²
=Eso
[(**($++4) =2 (st)² (s) + V (st)
=Exg (VT V= (s))
(6)
where ((5++1 18t, at) is the transition function of the underlying MDP. We turn the off-diagonal terms, where without loss of generality we let > t':
[V (8++1) V (V=))((((((r)))))
V (sx
EE.,
Ery [(V** (81+1) - V (s=)) (Se42) - V (so st])
[TFT (8++1) (se])))se]]]] (st)) ( T (st:))
Σ (ael(s)))
(8) (Vao (s0
Set1
-E,,
(V
(s)( (SV41) vT (sv)))
-0.
n
17

---

C
Significance tests
GSM8K
MATH
Corr
Sig
Corr
Sig
REINFORCE: No correlation between r(sT)* 2 and
15 9t
gel
-0,0460
Yes
-0.0315
Yes
RLOO: No correlation between (r(*T) E E (sT)
and EE= Dell
-0.0169
Yes
-0.0211
Yes
GRPO: No correlation between Learnability and
t=0 gell 1|
-0.0327
Yes
-0.0319
Yes
Table Since the underlying MDP in Theorem is in principle possible that the quantities E.. V log [
3.1 depends on the question on which we train, O.... T 1. depend on the learnability of that question. In particular, they could decrease with learnability, such that the upper bound for the gradient magnitude as a whole could decrease with learnability. However, while the formal MDP changes, the weights of the underlying LLM do not depend on the question that is chosen to train. Thus we assume that in practice, those quantities do not depend significantly on the learnability the current question. This table tests this assumption empirically, and shows no statistically significant correlation between learnability and E., [[[ log (a [| [[
for any
I 0, 0..."" T
Assumption
GSM8K
MATH
Corr
Sig
Corr
Sig
No correlation between Sr ge and [V(s++1) - ((s)) V(()]
0.0357
No
0.0132
No
No correlation between 19 2 and ((*+11 - (s+)
-0.0025
No
-0.0030
No
No correlation between 11 2 and
-0.0072
No
-0.0046
No
No correlation between || 2 and learnability
-0.0057
No
0.0105
No
Table 4: This table checks the assumptions required by the proof of Theorem 3.1
GSM8K
MATH
Learning algorithm
Corr
Sig
Corr
Sig
REINFORCE
0.1629
Yes
0.3145
Yes
RLOO
0.1485
Yes
0.4262
Yes
GRPO (normalised)
0.1293
No
0.3842
Yes
GRPO (unnormalised)
0.1690
Yes
0.4178
Yes
PPO
0.3786
Yes
0.7140
Yes
VinePPO
0.3786
Yes
0.7140
Yes
Table 5: This table tests Theorem 3.1 end end, by showing the correlation between learnability and
18

---

D Additional method
D.1 An algorithm for selecting high learnability questions with no additional sampling cost
Algorithm 2 uses rejection sampling to get batches of high learnability questions. uses [D] x (hammaiity--olloutt to find [B] high learn- ability questions and generate batch of B x Nicamability question attempts to learn from. This in contrast uniform sampling, which uses
Comparison of sampling strategies
0.25
0.20
only [B] Mearmability samples.
Learnablitty
0.15
As discussed in Section 4 in many situations
this is practical and results in a negligible in- crease in wall clock time. However, for situa- tions where this is not the case, ie sampling is extremely slow and cannot be parallelised, present Algorithm which is smarter in how it samples.
0.10
0.05
0.00
Algorithm starts in the same way as Algorithm 2, sampling candidate pool of DI questions from the dataset. However, unlike Algorithm - only samples 2 attempts for each question in this pool. Algorithm 3 then proceeds to itera- tively estimate the learnability of each question in the pool, sample a promising high lcarnability question, and draw 2 further samples from it. At each iteration, the new samples further refine the learnability estimates. This smarter way to allocate given sampling budget, N.
Uniform
(eeoreenn 2)
Smarter (Algorithm 3)
Figure The smarter sampling used by Algorithm 3 produces high learnability batches comparable rejection sampling, but using 4x fewer samples. It uses the same number of samples as uniform sampling, LILO + Smarter has no additional sampling overhead.
We show the learnability of batches generated by Algorithm Bjin Figure We see that uniform sampling consistently produces low learnability batches, and, as could be expected, the rejection sampling approach consistently produces high learnability batches. However, despite using 4x fewer samples than rejection sampling. the smarter sampling used by Algorithm 3 produces batches with nearly as high learnability. Algorithm produces much high learnability batches than uniform sampling despite using the same sampling budget.
Algorithm 3 Get Learnable Questions Using Smarter Sampling
Input:
Model parameters, of question batch return, [B]
Hyperparameters: Size of candidate pool, [D]
Total number of question rollouts, N
1: Sample candidate pool of [) questions from dataset
Rollout attempts per question
3: i 2 x [D)
4: while i N do
5:
Compute success rates for each question
Compute learnability for each question = p(1 D)
7:
Select question from candidate pool with probability proportional its learnabiliity Rollout 2 more attempts for that question
8:
9:
i=i+2
10:
end while
Return N generated attempts
19

---

E
Additional experimental setup details
E.1 Alterations VinePPO training setup
We increased the number of gradient accumulation steps and added Deepspeed ZeRO stage 2 to allow for training on single 48GB Nvidia L40s GPU and thus run multiple experiments in parallel on an 8 48GB node. This did not change the overall effective batch size of 512 episodes (64 levels * rollouts per level) and thus our training dynamies are identical VinePPO.
E.2
Compute resources
The PPO and VinePPO experiments took week on 4xl GPUs for each algorithm, dataset combination. The GRPO experiments took day on 8xH200 GPUs for each of GRPO with LILO and GRPO without LILO.
F
Additional results
Table 6: Test accuracy @ for 4 different training runs on the MATH dataset. CollegeMath and OlympidBench were not seen during training. LILO consistently improves generalisation.
COLLEGE MATH (%)
OLYMPIAD
BENCH (%)
ORIGINAL SFT PPO PPO + LILO
20.3
25.4
3.5
26.4
3.8
VINEPPO
26.9
4.1
VINEPPO + LILO
28.8
4.5
GSMIK acorrry
COMIN trem accuuwcy
0.4
0.4
6.7
NIRTH
0.6
5000
5020
Traihing Olum updsted)
vinssoo
Ppo
UIO
VinePPO on MATH
b: PPO on MATH
VinePPO GSM8K d: PPO on GSM8K
Figure 9: Train accuracy when training with and without LILO. For a full discussion of experimental setup see section
In all scenarios train accuracy increases faster with LII.O.
20

---

MATH
Test accuracy
18%
25%
28%
30%
32%
35%
38%
40%
43%
45%
Train accuracy (%)
GSM8K
62%
Test accuracy (%)
60%
58%
56%
54%
52%
50%
48%
60%
70%
75%
80%
90%
95%
Train accuracy (%)
vineppo ULO
ppo
vineppo
ppo
vineppo ULO
Figure 10: Train accuracy vs test accuracy shows overfitting and generalisation of different runs. Despite LILO's higher train accuracy, all runs fall roughly on the same line, indicating the same level generalisation and degree of overfitting.
21

---

Buffer resampled every 50 iterations, 50% of batch from buffer
Train accuracy (%0
Map
8.0
60
50
40
Buffer resampled every 10 iterations, 50% of batch from buffer
Train accuracy (5g
80
70
50
40
Buffer resampled every iterations, 50% of batch from buffer
Train accuracy (96)
8.0
70-
60
40
Buffer resampled every iteration, 100% of batch from buffer
Train accuracy (36)
80
70
m AMa v /n
60
Awdmaaa m
40
150
200
250
Levels in buffer
Levels sampled iid from train dataset
Figure 11: Exploring finding learnable questions less frequently, as suggested by 18] 50% of the
batch is randomly sampled each iteration, and the rest of the batch is sampled from a buffer of high learnability questions found by LII.O. The model overfits to the high learnability questions when the buffer is refreshed every 50, 10, 5 steps. Only when updated every step did we find overfitting stopped and generalisation started to occur.
22

---

85%
80%
75%
70%
65%
60%-
55%
50%
45%
5000
10000
15000
50
100
150
200
250
5000
10000
15000
Training step (Num model updates)
Training iteration
Number of backwards passes
training levels
4x training rollouts
4x smaller leaming rate
4x training rollouts
training levels
4x smaller learming rate
4x more gradient accumulation steps
4x training rollouts
4x more gradient accumulation steps
ppo+ ULO
4x training levels
Figure 12: Training all generated samples training the top-K highest learnability samples. We compare generating the additional samples from 4x more levels, or 4x more rollouts per level. We use the additional samples 1) naively, by increasing the number of batches. 2) increasing the number of batches but decreasing the learning rate 4x and 3) increasing the effective batch size by doing 4x the gradient accumulation steps. Whilst training 4x the samples challenge the performance of PPO+LIILO, it is significantly slower due to the outsized cost of doing more model forward/backwards passes during training.
23

---

G Further related work
We now cover various framings of the language generation MDP discussed in Section 2 and their related work on Automated Curriculum Learning (ACL).
G.1 ACL in Supervised Learning and Bandit Problems
Choosing a question to train on during each step can be seen as a bandit problem. The seminal work 4 provides overview of ACL in this setting. The central idea define utility measure U()) for each question or training sample and then design a curriculum that sequences the data accordingly. Early methods 44 used human-designed heuristics for U, progressing training from the easiest to the hardest examples. More advanced ACL methods in this setting can be loosely categorized by how they define U, such as:
1.
Methods: Many methods loss prioritize training hard data points +5 46, 47]. In [48] the authors use the loss from pre-trained model to estimate the difficulty of new samples for a freshly initialized network learning new task.
2. Uncertainty or Entropy: Several papers use the entropy of the answer distribution to prioritize training on data points the model is "unsure" about [49] 50 51]. LILO can be seen as using return variance-or learnability-as an estimator of entropy or uncertainty. For Bernoulli random variables, as in the reasoning setting, maximum entropy corresponds maximum variance.
3. Information Gain or Learning Progress: In some settings, it is possible estimate or empirically compute the effect of training on a data point. This allows prioritizing samples that maximize the change in loss-i.e., the model's learning progress [52
53
49].
One can also aim to maximize the change in entropy information gain 49 53 These approaches are challenging to apply at the scale of modern LLMs.
4. Gradient-Based Approaches: One can select data points that minimize the variance the gradient estimator computed by SGD
55].
Self-paced learning [56 is early approach that allows the model to determine the pace which incorporates harder examples with higher values U. [57] 58) introduced the concept a teacher-student setup, where the teacher is trained to optimize the student's 's learning.
G.2 ACL in Multi-Task Reinforcement Learning
Each question in a training dataset can be thought of as defining its own MDP, with the same state and action spaces but unique reward function encoding the question's 's answer. This framing is equivalent to multi-task RL (MTRL) [59]. where each question constitutes a separate task.
ACL work for MTRL roughly follows the categories above. [60] uses the entropy of the value function to prioritize training on tasks with the highest uncertainty. [6 selects training tasks based on their information gain relative test tasks. Similarly, [62] prioritizes tasks that maximize entropy gain.
G.3 ACL in Goal-Conditioned RI.
Since the MDP defined by each question shares the same state and action spaces, question can be viewed as "goal" for the agent. (63) samples goals using a learned goal distribution that maximizes entropy. (64) learns which goals are teachable based on an agent's competence. It is intriguing to consider whether ideas from 165 which a GAN is trained to generate goals of appropriate difficult-couldd be applied reasoning with LLMs.
G.4 Unsupervised Environment Design
Formally, the language generation MDP is extended to use the parameter 0. Each 0 E θ corresponds to question from training dataset, and the UED adversary will determine the distribution over which 0 is sampled from O, Given a choice of 0, the starting distribution for each level pa is just

---

the question and the reward function Rg computes accuracy of the generation The transition function is unchanged.
There are some key practical differences between the L.M reasoning setting and the setting UED was originally designed for. SFL was previously employed simple, fast, vectorised environments such as Minigrid, XLand-Minigiid and JaxNav. These environments have millions of unique levels. GSM8K, on the contrary, just 8,000 questions. Models the traditional UED setting are small, perhaps IM parameters. Whilst in this work we looked at a IB parameter model, Rho-Math- (Л]. LLMs with far more parameters have been trained with RL, such as the 67 IB parameter Deepseek- RI [ SFL has therefore previously been employed in settings where each training batch contains experience from 1000s of levels and 100s of episodes per level, even on single GPU. Mention how many rollouts per question is done in Deepseek, Tulu, RLHF etc In contrast in VinePPO, they train on a batch of just rollouts for each of 64 levels.
Whilst the increase in popularity of JAX means that some popular RI. enviromments are now vectorised, the sampling of trajectories remains expensive and major bottleneck for many RL tasks. In many LLM training setups however, sampling new trajectories is far more scalable than model updates, as trajectory collection can be distributed across independent processes that require no inter-process communication. This sampling computation can be fully pipelined and leverages highly optimized specialized inference engines like vLLM. Furthermore, since trajectory sampling only involves forward passes, it eliminates the need to store activations or optimizer states, making it significantly more memory-efficiint than training updates.
Prioritised level replay (PLR) 26 generates random levels, samples trajectories, and adds high scoring levels to buffer. TD-error is typically used as the score function. ACCEL [40] extends this with a mechanism to mutate previously high-scoring levels, to generate new levels that train the agent the frontier of its capabilities. PAIRED [33) co-trains level-sclecting adversary and two agents, a protagonist and an antagonist. It aims to maximise regret by maximising the difference in performance between the protoganist and antogonist. SFL [8] discards the notion of regret, instead using learnabilityy to select which levels to replay. SFL inspired early work on this paper, but significant changes we required to transfer it to LLMs. Namely we had to compute learnability at every step, using a fixed attempt budget instead of timestep budget (more suitable for inference engines like vL.LM) and reuse trajectories generated during learnability estimation. This results in an overall simpler, cleaner algorithm to that used in SFL. Our method is also similar to Prioritised Experience Replay (PER) (66], which uses TD-error prioritisation instead of learnability prioritisation.
25

---

H
Examples of high and low learnability questions
Dataset: GSM8K
Question:
Alton owns a business. He is currently renting a space that costs $20 per week. If Alton earns $8 per day, how much is his total profit every week?
Gold Solution:
* Calculate weekly earnings: $8 7 = $56
* Subtract weekly rent: $56 $20 = $36
* Answer: 36
Number of Reasoning Steps:
Success Rate (p) at 0: 1.0
Learnability (p(1 p)) at 0:
Question: Mrs. Smith wanted to buy items worth $500. She went to boutique with the $500 but realized she would need two-fifths more money than she had. If the shop owner gave her discount of 15%, how much more money will she still need?
Gold Solution:
Calculate additional money needed: 500 200
* Total cost before discount: $500 $200 $700
Calculate discount: 0.15 700 105
Subtract discount from total cost: $700 $105 $595
Additional money needed: $595 $500 $95
Answer: 95
Number of Reasoning Steps:
Success Rate (p) at 0: 0.5
Learnability (p(1 p)) at t = 0: 0.25
Question: In a fruit salad, there are raspberries, green grapes, and red grapes. There are seven more than 3 times the number of red grapes green grapes. There are 5 fewer raspberries than green grapes. If there are 102 pieces of fruit in the salad, how many red grapes are in the salad?
Gold Solution:
Let G represent the number of green grapes.
Red grapes: 3G +
Raspberries: G
Total fruit equation: G (3G + (G (G--) 102
Simplify: 5G
102
Solve for G: 5G 100 = G 20
* Calculate red grapes: x 20 + 7 67
* Answer: 67
Number of Reasoning Steps: 10
Success Rate (p) at 0
Learnability (p(1 p)) at 0: 0
Dataset: MATH
Question: Suppose p(z) is a monic cubic polynomial with real coefficients such that p(3 2i) and p(0) 522 Determine p(z) (in expanded form).
Gold Solution:
* Since p(x) has real coefficients and 2i is root, complex conjugate 3 2 is also a root.
26

---

* The quadratic factor with roots 3 2i and 3 + 23 is:
(x - (3 - -3-33-2)) (3 + 21) = -3-3+ 21 - 3 21)
= (I (x-3)-((2i)
x²² -6
x² 6z + 13
Since p(z) is cubic, can be expressed as p(z) (x² 6x 13) r).
Given p(0) 552:
p(0) (02 6- 13) r)
52 = 13 (3)
Therefore, p(z) is:
p(x) (x² 6r 13)(s -
3 4r 6z + 24r + 13r 52
2 10x² 37x 52
* Answer: x3 - 10. + 37x 52
Number of Reasoning Steps: 29
Success Rate (p) at 0: 0
Learnability (p(1 p)) at 0: 0
Question: If the sum of the squares of nonnegative real numbers b, and is 13, and ab + be + ca = 6, then what is the sum of a, b, and c?
Gold Solution:
* Start with the identity: (a + b + c)² a² +8 2(ab bc + ca)
* Substitute the given values:
(a b c)² 13 2 13 12 25
Take the square root of both sides:
a a+b+c=iv22 #v +5
Since b, and nonnegative, + must be nonnegative.
* Answer: 5
Number of Reasoning Steps:
Success Rate (p) at 0: 0.5
Learnability (p(1 - p)) at 0: 0.25
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, each claim in the abstract in the introduction is summarised succinctly in the bullet points page 2. Each bullet point contains claim and hyperlink to the section of the paper that proves the claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section 9 contains limitations of the work, including LILO's sampling over- head, discarding of useful data and restriction to verifiable binary reward tasks. Section 7 also contains some limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
28

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The assumptions for Theorem 3.1 are present in its definition in Section B the full proof is in Appendix and is linked from Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Our method is fully described in Algorithm and Algorithm The results in
were produced using open-source codebases 5 4 and models, with some small additions of code by us. We will open source these additions, as well as our final model checkpoints upon acceptance. All the experimental methodology is provided Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
29

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: As in the previous checlist item, our the results in 6 were produced using existing open-source codebases, models and data [ This described Section We will open-source the small modifications we made to these repos upon acceptance. We do not produce any new datasets, we will open source our final model checkpoints upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
www.ssssiiiciiiiio. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the hyperparameters specific to method in Section All the other hyperparameters for training are replicated directly from the VinePPO and Oat [4] libraries, and the user is directed to these in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
30

---

Answer: [No]
Justification: Since LLM training is computationally expensive we were unable to run the additional experiments needed plot error bars. We have, however, provided training curves to aid the reader in interpreting the significance of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are discussed in Appendix E.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///wwrrrr.cccoooo//aaaaaaaa
Answer: [Yes]
Justification:
We have read and fully conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
31

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work does not have societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the assets used in this paper are properly credited. The two libraries we used for training (VinePPO and Oat) are both fully open-source.
32

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswippessie thooe com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Will open source our code upon acceptance. It very simple, and could be implemented from this paper alone. However, instructions and documentation for our released code will provided the model.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cood/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human objects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
33

---

Answer:
[NA]
Justification:
This paper does not involve crowdsoureing nor research with human objects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM usage was only used in a standard way for editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
34


Reviews:
[{"review_id": "kw5pDz9dF6", "reviewer": "Reviewer_GyS9", "summary": "Learning from questions may be inefficient as the levels of difficulty of the questions do not align with the model being trained. This paper proposes to prioritize training on questions with high variance of success referred to as learnability. A theory is provided to prove that the proposed method, termed LILO, maximizes the expected improvement of the model. Experiments validate the effectiveness of LILO and show that LILO can greatly reduce the number of training steps.", "questions": "-\tIn Theorem 3.1, why $g_{t,\\theta}$ is assumed to be independent of $A_{t’}$?\n-\tWhy does the argument that expected policy improvement increases with the variance of the final reward hold, given that the expected value of the sum of $g_{t,\\theta}$ could be decreased?\n-\tHave your explored other sample selection methods?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- The proposed method is very simple and can be added seamlessly to existing training libraries with minimal changes.\n- The paper is generally well-written and easy to follow.\n- Experimental results are promising. The proposed sample selection method greatly speeds up the training as well as improves model performance.\n\n**Weaknesses:**\n\n*Originality*: \n\n-\tThe proposed method in a simple variation of hard sampling mining in the RL setting. Based on the definition of learnability, which is the variance of binary outcomes of a question, it is maximized when the success rate is 0.5. In this case, the model is completely uncertain about the question with equal rates of success and failure, and the selected question is most challenging to the model.\n\n*Quality*:\n\n-\tSome important assumptions are not well justified in the paper. Specifically, in Theorem 3.1, $g_{t,\\theta}$ is assumed to be independent of $A_{t’}$. This needs more justification.\n-\tTheorem 3.1 only states that the expected increment in reward depends on the variance of final reward. Maximizing the variance does not necessarily indicate that the expected reward will also be increased. Moreover, based on the independence assumption, the expected value of the sum of $g_{t,\\theta}$ may be decreased. Therefore, the claim that learnability optimally improves LLMs is overstated.\n\n*Typos*:\n\n-\tLine 87: asses -> assess\n-\tLine 102: J(\\theta_k ->  J(\\theta_k)\n-\tLine 121: empty citation", "was_revised": true, "final_justification": "The authors’ response has addressed my main concerns regarding the originality of the paper and the validity of Theorem 3.1. I have increased my score accordingly.", "rebuttal": "We thank the reviewer for their time and valuable feedback that has helped us improve this work. We are encouraged by the positive comments around the both simplicity and effectiveness of our approach at boosting the performance of RLVR algorithms.  We address the weaknesses and questions one-by-one below and hope this resolves any of the outstanding concerns and that the reviewer will consider revising their rating. \n\n**Originality:** \n- You are completely right to note the long history of studying hard sample mining in the traditional RL setting. We were inspired by many of these works especially Prioritised Level Replay [26] to try to produce a simple and effective method for the LLM RLVR domain. (For the avoidance of confusion, we note that our samples are not the hardest samples, they are the ones at the correct level of difficulty, thus providing the largest gradient update)\n- Beyond hard sample mining, there is a rich history of many other types of curricula for traditional RL settings, and as you note, Learnability has been proposed before, such as in [16]. \n- We emphasise that the relevance of [16] on reasoning with LLMs is extremely small - there is no clear method for computing and prioritising learnability, there is no suggestion that this should work beyond the REINFORCE algorithm, and is it considered in traditional RL tasks.\n-  Beyond [16], it is non-trivial to decide which curricula methods from traditional RL are suited for the LLM reasoning setting:for example Havrilla et al tried some and these failed. This is true for other areas of RL too, for example value networks, which are popular in deep RL, have been shown to not transfer to the LLM setting.\n- We identified that Learnability based curricula are particularly well suited for the LLM setting because it is non-stochastic, and uses sparse, binary rewards. Even once we identified this, there was still work to do to adapt it to the LLM setting. For example, in Appendix  E Figure 11 we detail how the frequency with which you recalculate learnable questions, which is typically done every 10-50 timesteps in the traditional setting, is catastrophic in the LLM setting.\n- Crucially, this results in LILO being the first to show that a curriculum method can boost overall performance on LLM reasoning tasks. Given the popularity and resource consumption of this setting this is highly impactful and useful to the community.\n\n**Quality (The validity of Theorem 3.1):**\n\n- We greatly appreciate the valuable remarks regarding Theorem 3.1. Prompted by the reviewer’s comments, we have revisited the proof of Theorem 3.1, and realized that the assumptions needed for the proof can be relaxed. Previously we assumed independence between $g_{t, \\theta}$ and $A_{t’}$, for any $t$, $t’$. The new, less restrictive, assumptions we need, are as follows:\n    - For REINFORCE and RLOO we require that $|| \\sum_{t=0}^{T-1} g_{t, \\theta} ||^2$ is uncorrelated with $A_t = r(s_T)^2$ in the case of REINFORCE, and uncorrelated with $A_t = ( (r(s_T) - \\mathbb{E}{\\pi_\\theta} [ r(s_T) ])^2$ in the case of RLOO. This is a less restrictive assumption than the independence between $A_t$ and $g_{t’, \\theta}$, for any $t$, $t’$, and lack of correlation is much easier to test for than independence.\n    - For PPO and VinePPO we now only require that $A_t^2 = (V^{\\pi_\\theta}(s_{t+1}) - V^{\\pi_\\theta}(s_t))^2$ and $|| g_{t, \\theta’} ||^2$ are uncorrelated for any $t = 0, …, T-1$. This is a less restrictive assumption due to again replacing independence with lack of correlation, and since we no longer need any assumption on the dependence of $A_t$ and $g_{t’, \\theta}$, at different times $t \\neq t’$. \n- We in particular appreciate the excellent comment that $|| g_{t, \\theta} ||^2$ or $ || \\sum_{t = 0}^{T-1} g_{t, \\theta} ||^2$ could depend on learnability. We completely agree that those terms could depend on the underlying question of the MDP, and thus on the learnability of that question. We have thus also tested the correlation of those terms with learnability, in addition to all the revised assumptions made in the theorem. Below are all the details and results regarding those statistical tests:\n- For REINFORCE and RLOO: \n    - We randomly sampled questions, answers and gradient locations from the training process, drawing ~10,000 datapoints for each of the MATH and GSM8K experiments.\n    - We found a very small correlation between a) $|| g_{t, \\theta} ||^2$ and $A_t$ b) $||g_{t, \\theta}||^2$ and $t$, and c) $ || \\sum_{t=0}^{T-1} g_{t, \\theta} ||^2 $ and learnability. Testing end to end, we found that these small negative correlations are dwarfed by the increase of learnability itself. As explained by our theorem, $ || \\nabla_\\theta J(\\theta) ||^2$ has a strong statistically significant positive correlation with learnability. These are shown in the tables below.\n- For PPO, VinePPO: As mentioned above, we have relaxed the assumptions needed for our proof, and now only require that:\n    - $ ||g_{t, \\theta} ||^2$ and $(V^{\\pi_\\theta}(s_{t+1}) - V^{\\pi_\\theta}(s_t))^2$ are uncorrelated, for any $t = 0., …, T-1$. We found no statistically significant correlation between those terms, when testing this.\n    - $ \\mathbb{E} \\[ ||g_{t, \\theta} ||^2 \\]$ does not depend on $t$. We found no statistically significant correlation between $|| g_{t, \\theta} ||^2$ and $t$ when testing this.\n    - $ ||g_{t, \\theta} ||^2 $ does not depend on learnability. We found no statistically significant correlation between $ ||g_{t, \\theta} ||^2 $ and learnability when testing this.\n    - Testing end to end, we found that as explained by the final result of our theorem, $|| \\nabla_\\theta J( \\theta) ||^2$ has a strong statistically significant correlation with learnability.\n- To summarize, we have thoroughly tested empirically the validity of the assumptions in Theorem 3.1, and conclude that those are reasonable assumptions to make. We thus believe that Theorem 3.1 provides a well justified quantitative explanation of how LILO works under the hood.\n\n|                                                                                            | **GSM8K** |         | **MATH** |         |\n|--------------------------------------------------------------------------------------------|-----------|---------|----------|---------|\n| **Assumption**                                                                             | **Corr**  | **Sig** | **Corr** | **Sig** |\n| **REINFORCE/RLOO/GRPO (Appendix B, Equation 6)**                                           |           |         |          |         |\n| No correlation between $r(s_T)^2$ and $\\|\\|\\sum_{t=0}^{T-1}g_t\\|\\|^2$                      | -0.0460   | Yes     | -0.0315  | Yes     |\n| No correlation between $(r(s_T)-\\mathbb{E}[r(s_T)])^2$ and $\\|\\|\\sum_{t=0}^{T-1}g_t\\|\\|^2$ | -0.0169   | Yes     | -0.0211  | Yes     |\n| No correlation between Learnability and $\\|\\|\\sum_{t=0}^{T-1}g_t\\|\\|^2$                    | -0.0327   | Yes     | -0.0319  | Yes     |\n|                                                                                            |           |         |          |         |\n| **VinePPO / PPO (Tightened proof, above)**                                                 |           |         |          |         |\n| No correlation between $\\|\\|g_t\\|\\|^2$ and $V(s_{t+1}) - V(s_t)$                           | -0.0025   | No      | -0.0030  | No      |\n| No correlation between $\\|\\|g_t\\|\\|^2$ and $t$                                             | -0.0072   | No      | -0.0046  | No      |\n| No correlation between $\\|\\|g_t\\|\\|^2$ and learnability                                    | -0.0.0057 | No      | 0.0105   | No      |\n\n\n|                     | **GSM8K** |         | **MATH** |         |\n|---------------------|-----------|---------|----------|---------|\n| Learning algorithm  | **Corr**  | **Sig** | **Corr** | **Sig** |\n| REINFORCE           | 0.1629    | Yes     | 0.3145   | Yes     |\n| RLOO                | 0.1485    | Yes     | 0.4262   | Yes     |\n| GRPO (normalised)   | 0.1293    | No      | 0.3842   | Yes     |\n| GRPO (unnormalised) | 0.1690    | Yes     | 0.4178   | Yes     |\n| PPO                 | 0.3786    | Yes     | 0.7140   | Yes     |\n| VinePPO             | 0.3786    | Yes     | 0.7140   | Yes     |\n\nThis table shows the correlation between learnability and $|| \\nabla_\\theta J(\\theta) ||^2$. \n\nWe invite the reviewer to consider our supplementary explanations and assess if they resolve the raised concerns, possibly influencing their appraisal of our work. Should there be any other aspects or feedback you find relevant, please do not hesitate to share."}, {"review_id": "swEBc1W631", "reviewer": "Reviewer_xAAo", "summary": "The paper discusses a key inefficiency in RL methods for post-training in LLMs - especially when the tasks are about some form of reasoning. The central observation is that problems are either consistently solved or are tooo hard such that the LLMs might consistently fail – perhaps because the LLMs are not expressive enough. Training/exploring over these problems provides near zero learning signal for policy gradient based RL algorithms – this is the central claim. The proposal is an algorithm that exploits a concept of “learnability” and then the paper presents a method based on the concept of learnability, an algorithm for sampling “learning-*optimal*” samples.", "questions": "With regards to the proof – don’t we need fairly strong assumptions that log policy gradients are independent of the advantages? would that hold in practice – if so how likely? Wouldn’t there be fairly important implications when this assumption is violated?\n\nI put it in the weakness bit – I am not sure I understood why GRPO works even if it loses its dependability on learnability? \n\nAlso, what happens if there are potential issues in samples – GSM8K for e.g., has incorrect Q, A pairs. In some cases spurious correlations can help in answering (matching the gold answer … which are inherently incorrect) such Qs (or samples). \n\n\nThere is some dangling issue — line 121 – multiple attempts to replicate Deepseek R1-Zero [] …. ? \n\nThere is a claim that’s repeatedly made … – 3x reduction in training steps — shouldn’t that be “upto” … in select settings? Also, not sure I understand the non-congruency in experiments - PPO vs VinePPO are evaluated on similar setups, but GRPO on a completely different one?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "There are a few things that I am concerned about and mostly these are to do with claims that are a tad too generous. For instance, Learnability is a tad too strong as a term, the presented approach is oriented at making the model efficiently learn to maximise its *score* on problems that provide a variable reward signal. It does not—and cannot—guarantee that the model is learning the correct reasoning process (can it?). A problem with high \"learnability\" (e.g., a 50% success rate ) is simply a problem where the model's current strategy is unstable. It's at the \"frontier\" of what the model can solve with its existing, potentially flawed, heuristics. My worry is focusing on these problems accelerates the process of either solidifying a flawed-but-successful strategy or discovering a new one. It optimizes for finding some way to get the right final answer. The way I see it is that the method is about the efficiency of the RL training loop, not the semantic correctness of the knowledge being acquired. Indeed, a more accurate, albeit less flashy name is something along the lines of “training signal variance” or perhaps even “algorithmic utility” – is this sample at a sweet spot of difficulty where it can effectively drive *parameter updates* – instead of “learnability”. From a cog sci/pedagogical perspective, I am not sure learnability really captures the core aspects. When we talk about learning, we intuitively mean acquiring the correct, generalizable, step-by-step process to solve a problem. This implies understanding, not just successful pattern matching. Surely, Theorem 3.1 cannot capture the generality of this. \n\nAnother, thing that was a tad overstated was a “Gap in the literature” – for instance, the draft itself recognizes previous work, such as Tzannetos et al. 2024, Havrilla et al., 2024, or Smaug – pal et al., 2024 to name a few have all touched upon, with observations and conjectures. Granted that this work takes a fairly known theoretical result & empirical observations from a simplified context and generalizes it. While this generalization is a valid contribution, framing it as closing a vital gap and providing a novel proof that maximises the expected improvement, to me, comes across as an overstatement. \n\nI see that for GRPO, which normalizes the advantage, the expected gradient magnitude loses its dependency on learnability. Oddly the experiments indicate that LILO still works for GRPO … How can we reconcile this? \n\n\nOn the generalization gap between training and test accuracy – training accuracy can approach 95% while test accuracy stalls below 60%. This reinforces my earlier bit that the model isn't *learning* robust reasoning, it's just getting better at solving the specific problems it's trained on. \n\n\nFurther, I would have loved to see error analysis – what changed? Are there types of samples that have changed? Could you have solved it with say rejection sampling? Finding a causal link at inference time would have made it stronger. I can see the curriculum bit, that’s interesting, but perhaps understanding what happened at test time would be useful. \n\nThe paper as I see it seems to be a nice engineering heuristic, I do like the heuristic. The theorem, as it stands in the current draft, feels like reverse engineering to justify the methodology.", "was_revised": true, "final_justification": "I happy with their continued engagement and effort that they are putting for the review.", "rebuttal": "We thank the reviewer for their time and valuable feedback that has helped us improve this work. We are encouraged by the positive comments around the effectiveness of our method for boosting performance of algorithms that train LLMs to reason with RL. We address the weaknesses and questions one-by-one below and hope this resolves any of the outstanding concerns and that the reviewer will consider revising their rating. \n\n**The naming of “learnability”:**\n- You’re completely correct that training signal variance, algorithmic utility (or even just reward variance) are great names for this quantity. However, there is a body of previous work [16,17,18] that terms this ‘learnability’ and as such, it makes sense to use this term. \n\n**\"The model isn’t learning robust reasoning - it is just getting better at solving specific problems:\"**\n- This is indeed very much a concern with LLM reasoning, RL or even just machine learning in general. It doesn’t feel like a particularly fair critique of this paper specifically, which does not claim to make improvements in this area. \n- Methods for improving the generalisation of LLM reasoning are definitely needed (we also suggest this in Section 7), and we thank the author for acknowledging this.\n- This is a method to speed up training, and boost the overall test accuracy achieved. Given the popularity and resource consumption of LLM reasoning, this is an equally worthwhile and impactful direction orthogonal to improving generalisation.\n- As with the term “learnability”, one can take issue with established phrases such as “learning to reason”. Given the prevalence of “learning to reason” RLVR papers, it is reasonable to use this term to avoid needlessly adding new language to this topic, but we also recognise its limitations.\n\n**What happens if GSM8K has incorrect Q,A pairs?**\n- As with any most machine learning methods, if the data is wrong or low quality, the model training would likely suffer.\n- There is an argument that LILO could help to reduce the negative effects of mislabelled data. For example, if the answer is wrong and the model consistently answers correctly but is deemed incorrect, the learnability of this sample would be 0, and we would ignore training on it. This is in contrast with not using LILO which would continually prioritise training on this question.\n\n**Theorem 3.1’s closing of a “vital gap”:** \n- There is a rich history of curricula for traditional RL settings, and as you note, Learnability has been proposed before, such as in [16]. \n- We emphasise that the relevance of [16] on reasoning with LLMs is extremely small - there is no clear method for computing and prioritising learnability, there is no suggestion that this should work beyond the REINFORCE algorithm, and is it considered in traditional RL tasks.\n-  Beyond [16], it is non-trivial to decide which curricula methods from traditional RL are suited for the LLM reasoning setting:for example Havrilla et al tried some and these failed. This is true for other areas of RL too, for example value networks, which are popular in deep RL, have been shown to not transfer to the LLM setting.\n- This paper therefore provides the theory to explain why learnability based curricula are particularly well suited for the LLM setting. Previous work does not provide any theory as to why learnability would be useful for modern RL algorithms such as RLOO, GRPO, VinePPO and PPO.\n- Looking at the update magnitudes produced during training, we see that selecting the highest learnability produces 1.46x larger updates than not using learnability. Theorem 3.1 explains this, but it also explains why prioritising learnability produces 1.44x larger updates than simply rejecting any questions that are always answered correctly or incorrectly, and 1.40x larger updates than prioritising the most difficult questions. Existing theory cannot explain this.\n- Crucially, this results in LILO being the first to show that a curriculum method can boost overall performance on LLM reasoning tasks. Given the popularity and resource consumption of this setting this is highly impactful and useful to the community.\n\n**Assumptions within Theorem 3.1:**\n- We refer the reviewer to our rebuttal to reviewer GyS9, where we comprehensively address this with statistical testing of our assumptions and the theory.\n\n**LILO + GRPO:**\n- Thank you for pointing this out! We definitely should have made this clearer. We use the unnormalised version of GRPO that has shown to outperform normalised GRPO for reasoning tasks [cite DrGRPO and Replicating R1]. As we point out in Section 3, using the unnormalised GRPO does benefit from prioritising learnability.\n- We wanted to ensure that the experimental setups we were using already had optimised baselines, and therefore that LILO’s improvement could not be brought about e.g by tuning hyperparameters. We therefore used the VinePPO library which provides optimised baselines of PPO and VinePPO for GSM8K and MATH, and the Oat library which provides optimised baselines of GRPO on ORZ57K.\n\n**Error analysis and rejection sampling:**\n- In Figure 5 we visualise how the number of reasoning steps chosen by LILO changes throughout training. This is strong evidence that rejection sampling would not have been sufficient - questions with a higher number of reasoning steps only become learnable later on during training. \n- We agree that further visualisation here of the exact questions that progressed from hard to easy throughout training would be useful to the reader, and we commit to adding these in the camera-ready version. \n\n**Errata:**\n- Thank you for your diligence in picking up on the mistake on line 121, we have fixed this.\n- Thank you for the suggestion of rewording to use “up to 3x reduction”. This is indeed a fairer reflection of Table 2.\n\nWe respectfully ask the reviewer to evaluate our added details: we believe we comprehensively address any concerns, and that as such it would be reasonable to modify the assessment of our contribution. We are open to any additional feedback or relevant points you might wish to discuss."}, {"review_id": "sn1xSCyrRP", "reviewer": "Reviewer_Qkcn", "summary": "The paper proposes a sample selection mechanism that chooses the examples that are most learnable at the moment (i.e. produces the most variance in the success outcome) as the training batch. The paper finds theoretical relations between learnability and the expected gain of performance. Empirically, the paper finds that the sample selection method improves LLM reasoning on MATH, GSM8K, and ORZ57K datasets by about 5-10% compared to baselines.", "questions": "- I requested the wall clock time training curves. But I can't promise if there is a score increase given my other concerns.", "limitations": "The limitation section (Section 9) seems sufficient.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper shows consistent positive improvement (5-10%) over baselines by introducing their curriculum learning component, over 3 different datasets.\n- The authors made an effort showing theoretical insights on how learnability equates to policy improvement.\n\nWeaknesses:\n- Limited novelty: “Learnability” as a curriculum learning criterion has been proposed in [16], and this paper is just an application of learnability to LLM reasoning. Does learnability as a sample selection criterion benefit particularly to LLM reasoning? This is unclear. Therefore, the novelty is limited.\n- Definition of learnability: Although learnability has been proposed before, I have objection to the naming. Higher “learnability” simply means that the model responses are variable, but it doesn’t necessarily mean that they are more learnable. It is possible that although the model uniformly makes the same mistake on the current iteration, the sample is more learnable and after a few updates the model will fix the mistake.\n- Theoretical depth: Section 3 discusses how optimizing learnability can lead to policy improvement. However, the insight is limited, and the combination of a theoretical perspective on general RL and an application focus on LLM reasoning is strange.\n- Wall clock time comparison: Since the proposed method needs to sample a batch of rollouts for many examples and only choose to update the ones with the top learnability, it can be the case that it takes longer to train. While the paper has compared the timing, it would be good to show the training curve with wall clock time as the horizontal axis, and see if baselines can benefit from prolonged training in the main table.\n- Sampling from model policy: Taking the variance over model policy can lead to learning stagnation if the model has no good insight on how to solve the problem with their samples. Furthermore, it can also lead to ignoring certain trajectories in the training environment.", "was_revised": true, "final_justification": "The work builds on an interesting relation between \"learnability\" and LLM reasoning, and applies a sample selection mechanism. While the angle is interesting, the gain in the current experiments is modest. Therefore I recommend a weak accept.", "rebuttal": "We thank the reviewer for their time and valuable feedback that has helped us improve this work. We are encouraged by the positive comments around the improvement that prioritising learnability brings to training LLMs to reason with reinforcement learning. We address the weaknesses and questions one-by-one below and hope this resolves any of the outstanding concerns and that the reviewer will consider revising their rating. \n\n**Originality:**\n- There is a rich history of curricula for traditional RL settings, and as you note, Learnability has been proposed before, such as in [16]. \n- We emphasise that the relevance of [16] on reasoning with LLMs is extremely small - there is no clear method for computing and prioritising learnability, there is no suggestion that this should work beyond the REINFORCE algorithm, and is it considered in traditional RL tasks.\n-  Beyond [16], it is non-trivial to decide which curricula methods from traditional RL are suited for the LLM reasoning setting:for example Havrilla tried some and these failed. This is true for other areas of RL too, for example value networks, which are popular in deep RL, have been shown to not transfer to the LLM setting.\n- We identified that Learnability based curricula are particularly well suited for the LLM setting because it is non-stochastic, and uses sparse, binary rewards. Even once we identified this, there was still work to do to adapt it to the LLM setting. For example, in Appendix  E Figure 11 we detail how the frequency with which you recalculate learnable questions, which is typically done every 10-50 timesteps in the traditional setting, is catastrophic in the LLM setting.\n- Crucially, this results in LILO being the first to show that a curriculum method can boost overall performance on LLM reasoning tasks. Given the popularity and resource consumption of this setting this is highly impactful and useful to the community.\n\n**Naming of learnability:**\n- We completely agree that “learnability” is a loaded phrase, and ideally a different term, such as \"reward variance\" would be used. However there is a large body of previous work (including [16,17,18]) that terms this ‘learnability’ and as such, we must use this term. There are proper mechanisms to attempt to change this, such as community meetups, blog posts and position papers, but reviews of individual papers are not a suitable place.\n- We emphasise that LILO does NOT prioritise variability in responses or mistakes, but rather variability in rewards. It does not matter whether the model makes the same mistake every time, or a different mistake every time. If all samples are incorrect, it is a mathematical fact that existing policy gradient algorithms will not learn, no matter how many updates.\n- If there are some correct and some incorrect examples, and thus some gradient signal, it is an interesting hypothesis to consider that it might be easier to learn to fix 1 mistake than 10 different ones, and we thank the reviewer for this suggestion for future work.\n\n\n**Impact of theorem 3.1:**\n- We collected over ~10,000 gradient samples from the checkpoints used during training on GSM8K and MATH and found a statistically significant correlation between learnability and $ || \\nabla_\\theta J(\\theta)||^2$, as show in the table below. See the rebuttal to reviewer GyS9 for more information on this. \n\n|                     | **GSM8K** |         | **MATH** |         |\n|---------------------|-----------|---------|----------|---------|\n| Learning algorithm  | **Corr**  | **Sig** | **Corr** | **Sig** |\n| REINFORCE           | 0.1629    | Yes     | 0.3145   | Yes     |\n| RLOO                | 0.1485    | Yes     | 0.4262   | Yes     |\n| GRPO (normalised)   | 0.1293    | No      | 0.3842   | Yes     |\n| GRPO (unnormalised) | 0.1690    | Yes     | 0.4178   | Yes     |\n| PPO                 | 0.3786    | Yes     | 0.7140   | Yes     |\n| VinePPO             | 0.3786    | Yes     | 0.7140   | Yes     |\n\nThis table shows the correlation between learnability and $|| \\nabla_\\theta J(\\theta) ||^2 $.\n\n- The theorem is the first to explain this phenomena and directly led to our method, which in turn led to significant improvement and acceleration of LLM training methods. Given the popularity of LLM reasoning, this constitutes significant downstream impact of Theorem 3.1.\n- Looking at the update magnitudes produced during training, we see that selecting the highest learnability produces 1.46x larger updates than not using learnability. Theorem 3.1 explains this, but it also explains why prioritising learnability produces 1.44x larger updates than simply rejecting any questions that are always answered correctly or incorrectly, and 1.40x larger updates than prioritising the most difficult questions. Existing theory cannot explain this.\n\n**“Combining theoretical RL and LLM reasoning is strange”:**\n- The reviewer is correct to note that it has become commonplace in LLM papers to present methods or heuristics that work, without any attempt to explain or motivate them with theory.\n- It is our position that ideally new methods are presented with insightful theory that motivates and explains them. The required theory to explain our method did not previously exist, and thus we presented it alongside the method. We are not opposed to publishing a more general theory paper in future.\n\n**Wall clock time:**\nWhilst we did provide wall-time impact in Table 1, we agree that it would be more useful to the reader if speed-ups were reported in wall-clock time alongside speed-ups in training steps. We have therefore amended Table 2 to be the following:\n\n| Algorithm | Train Dataset | Speed-up (Steps) | Speed-up (Wall Time) | Final test accuracy (%) |\n|-----------|---------------|------------------|----------------------|-------------------------|\n| PPO       | MATH          | 2.5×             | 2.0x                 | 19.1 → 21.8             |\n| PPO       | GSM8K         | 1.9×             | 1.5x                 | 51.1 → 53.2             |\n| VinePPO   | MATH          | 3.2×             | 3.1x                 | 22.8 → 24.9             |\n| VinePPO   | GSM8K         | 3.3×             | 3.2x                 | 53.2 → 55.9             |\n| GRPO      | ORZ57K        | 1.6×             | 1.3x                 | 35.5 → 37.1             |\n\nWe cannot upload graphs during this rebuttal, but commit to changing the x-axis on training curves, Figure 1, 2, 3 and 4 for future versions.\n\n**Learning stagnation:**\n- LILO does prioritise some trajectories over others, which is precisely why it speeds up training.\n- Crucially however, it does not ignore them forever: If the model has no insight into a certain problem, and is mostly wrong when answering it, it will have low learnability, and training on it would have lower impact than training on a higher learnability question. If no questions with higher learnability exist, LILO will still train on this question. Methods that don't use LILO gain no benefit from training on these questions earlier on in the process. The same is true for questions that model consistently gets right.\n\nWe kindly request that the reviewer examines our further clarifications and assesses whether they satisfactorily address the concerns previously raised, potentially altering the evaluation of our work. We welcome any additional insights or comments that may be deemed pertinent."}, {"review_id": "NQUZgaC9Vd", "reviewer": "Reviewer_C2Lj", "summary": "The work studies the learnability of examples in RL and found that learning examples that are not too hard or too easy are the most important.  The work theoretically shows that the expected policy improvement increases with the variance of the final reward.  Motivated by this finding, the proposed LILO framework selects the examples based on learnability and can reach the comparable performance with fewer training steps when applied to varied RL training methods.", "questions": "How is N learnability decided?\nWhat is the parameter setting (e.g., temperature, top p) for decoding during sampling?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength:  \nThe paper is very well-structured, and the writing is smooth.\n\nThe finding reveals examples that are not too hard or too easy are more important to RL, which is very interesting.  The finding is theoretically obtained and evidenced by strong empirical results. The insight is clear and compelling, with promising implications for improving data curation in training LLMs more efficiently.\n\nThe proposed method is well-motivated and simple but effective.\n\nWeakness:  \nOne weakness is N_learnability in practice may need tuning to find the most suitable one. Different models may have different N_learnability. It could also depend on the setting during sampling, e.g., top_p, temperature.  Ablation study on N_learnability will be beneficial to show that LILO is robust to different choices of N_learnability.  \n\nBut the work is generally solid. The core theoretical finding is well-supported by experiments.", "was_revised": true, "final_justification": "Thanks for the authors' response. I maintain my score.", "rebuttal": "We thank the reviewer for their time and valuable feedback that has helped us improve this work. We are encouraged by the positive comments and score. We address the suggestion of an additional ablation below and hope this resolves any of the outstanding concerns.\n\n**Tuning N_learnability:** \n\nThis is a great point, and different models and hyperparameters may have different optimal N_learnability. Sadly, we don’t have the compute resources to complete a full training run for a variety of choices of N_learnabilty. However, a simple approach to finding N_learnability at the start of training could be to sample a batch of questions, and plot how the choice of N_learnability affects the overall learnability of the selected batches. We do this for GSM8K and MATH below:\n\n|       | N_Learnability | Learnability |\n|-------|----------------|--------------|\n| GSM8K | 1              | 0.000        |\n|    | 2              | 0.145        |\n|       | 4              | 0.195        |\n|       | 8              | 0.221        |\n|       | 16             | 0.231        |\n|       | 32             | 0.238        |\n|       | 64             | 0.241        |\n|       | 100            | 0.243        |\n|       |                |              |\n| MATH  |   1            | 0.000        |\n|       |   2            | 0.107        |\n|       |   4            | 0.134        |\n|       |   8            | 0.169        |\n|       |  16            | 0.175        |\n|       |  32            | 0.182        |\n|       |  64            | 0.183        |\n|       | 100            | 0.186        |\n\nN_learnability=4 or 8 seem to be a nice balance between high learnability batches and low sampling costs.\n\n\nWe hope this adequately addresses the issue highlighted and invite any more feedback or points you believe are significant."}]

Author rebuttal:

Meta-review: The reviewers converged on the view that the paper makes a clear and well-substantiated contribution: LILO provides a simple, mathematically motivated sampling strategy that accelerates RL training for reasoning tasks with LLMs and yields consistent accuracy improvements across datasets. Multiple reviewers highlighted the clarity and writing quality, the strong empirical evidence, and the usefulness of the theoretical connection between reward variance and policy improvement. While some initially questioned the originality and assumptions of Theorem 3.1, the authors’ rebuttal and additional experiments addressed these concerns convincingly, leading reviewers to raise their scores.

The main disagreements centered on scope and framing. Reviewer xAAo argued that the term “learnability” (and by extension “learning to reason”) risks overstating the contribution, since LILO improves training efficiency rather than robust reasoning generalization, but in my view "learnability" is a well established term for reward variance. Others questioned novelty relative to earlier RL curricula work and suggested that claims about closing a “vital gap” were somewhat overstated, which I agree with. Nonetheless, even these critical reviewers accepted the value of the heuristic, acknowledged the theoretical generalization to modern RL algorithms as a meaningful contribution, and increased their evaluations after discussion. With four reviewers aligned that the central claim is supported by both theory and experiments, the overall consensus favors acceptance.
Decision: Accept (poster)