RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems
Authors: Yuxiao Qu
Anikait Singh
Yoonho Lee
Amrith Setlur
Ruslan Salakhutdinov
Chelsea Finn
Aviral Kumar
Venue: ICLR 2026 Poster (2026)
Keywords: Reasoning abstractions; LLM; RL; Structured exploration; Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: A two-agent training framework for generating and applying reasoning abstractions to solve complex problems.

Abstract: Reasoning requires going beyond pattern matching or memorization of solutions to identify and implement algorithmic procedures that can be used to deduce answers to hard problems. Doing so requires reusing primitives, intermediate results, or procedures across multiple problems. While RL post-training on long chains of thought ultimately aims to uncover this kind of algorithmic behavior, the depth-first and brute-force nature of reasoning traces learned by these models suggests that this is far from a fulfilled promise. To address more effective reasoning, we introduce reasoning abstractions: concise natural language descriptions of procedural and factual knowledge that guide the model toward learning successful reasoning. We train models to be capable of proposing several useful abstractions given a problem, followed by RL training that incentivizes building a solution while using the information provided by these abstractions. This results in a two-player RL training paradigm, abbreviated as RLAD, that jointly trains an abstraction generator and an abstraction-conditioned solution generator. This setup effectively enables structured exploration, decouples learning signals of abstraction proposal and solution generation, and improves generalization to harder problems. We also show that spending more test-time compute into generating abstractions is more beneficial for performance than generating more solutions at large inference-time budgets, illustrating the role of abstractions in guiding global exploration.

Published as a conference paper at ICLR 2026
RLAD: TRAINING LLMs TO DISCOVER ABSTRACTIONS FOR SOLVING REASONING PROBLEMS
Yuxiao Qu' Anikait Singh?*, Yoonho Lee Amrith Setlur'
Ruslan Salakhutdinov', Chelsea Finn² Aviral Kumar-
Caamegie Mellon University Stanford University "Equal Contribution
ABSTRACT
Reasoning requires going beyond pattern matching or memorization of solutions identify and implement "algorithmic procedures" that can be used deduce answers to hard problems. Doing so requires reusing primitives, intermediate results, or procedures across multiple problems. While RL post-training on long chains of thought ultimately aims to uncover this kind of algorithmic behavior, the depth-first and "brute-force" nature of reasoning traces learned by these models suggests that this is far from fulfilled promise. To address more effective reason- ing, we introduce reasoning abstractions: concise natural language descriptions procedural and factual knowledge that guide the model toward learning successful reasoning. We train models to be capable of proposing several useful abstractions given problem. followed by RL training that incentivizes building solution while using the information provided by these abstractions. This results in a two-player RL training paradigm, abbreviated RLAD, that jointly trains abstraction gen- erator and an Abstraction-conditien enables structured exploration, decouples learning signals of abstraction proposal and solution generation, and improves generalization to harder problems. We also show that spending more test-time compute into generating abstractions is more beneficial for performance than generating more solutions at large inference-time budgets, illustrating the role of abstractions in guiding global exploration.
solution generator. This setup effectively
INTRODUCTION
Modern machinery for training large language models (LLMs) to reason relies on incentivizing longer chains of thought via reinforcement learning (RL). This training approach largely incentivizes "depth": subsequent training iterations increase response length by incorporating new operations that usually verify or build on top of the line of reasoning being already pursued by the model (Setlur et al., 2025). This often results very long chains of thought that appear to explore the solution search space, but in sequential, brute-force manner. In many problems, it is instead more desirable to optimize for "breadth": explore a diverse array of solution strategies, rather than committing to a seemingly good set of reasoning strategies right away (Yu al., 2025; Yue et al., 2025). Even though models trained this way succeed on some problems, they fail on problems with similar difficulty, revealing poor generalization (Ma et al., 2024; Mirzadeh al., 2024; Petrov al., 2025).
How can we help models explore a breadth of reasoning strategies for given problem? Abstractly, the most natural approach is to train models to hypothesize new strategies attack difficult problems and then attempt to utilize these strategies in the solution. We can do this by making models capable of discovering reasoning abstractions: compressed representations of shared procedures that underlic multiple candidate solutions to a problem. For example, in math reasoning, such abstractions might correspond to useful intermediate lemmas or even some intermediate steps that do not succeed but illustrate what not to do. When presented in context, these abstractions function akin to "hints" on an exam, enabling LLMs to solve harder problems by building on the insights appearing in the abstraction. That is, when conditioned the LLM to implement useful meta strategies that utilize and compose the procedural information in the abstraction as best possible solve the problem, rather than attempting to search over the procedural information itself. This naturally boosts the diversity of solution strategies and behaviors that model learns to utilize when encountering unseen problem, in contrast to committing to
abstractions, training via RL should train

---

Published as a conference paper at ICLR 2026
Standard Reasoning
Generate Abstractions by Summarizing the Future
Propose and Utilize Abstractions
Problem
Use the quadtadte fomuta moduser arthreets: compute the discrimiam D bi dac, then X (-b a vBlybw (mad mi...
beteriine the
prrme p whlth
sodisies the
congr-ennc
143.
Check esfstence mutiplicaive inverse before using k- in congruencco.. number preciselly when gootx..
Intermdiitte Stop
Correct Answerr
Iecameect Answer
Madei Generated Abstraation
Figure 1: Reasoning abstractions illustrated in the solution-space graph for a problem. We depict the solution space as graph intermediate steps leading to correct or incorrect answers. (1) Standard reasoning explores this space along one sequential chain. (2) We generate textual abstractions by summarizing which intermediate steps led to which outcomes. (3) Such abstractions can be reused to guide reasoning more efficiently.
narrow set of approaches. In RL terminology, abstractions serve as high-level subgoals, skills, or pnior--any of them depending upon context-guiding the low-level solution solution-generating policy.
In this work, we imbue LLMs with the capability of proposing and utilizing abstractions for solving problems. Concretely, we build reasoning models that, first, given an input problem, propose one or more reasoning abstractions, expressed in natural language. Subsequently, they generate a solution that utilizes the information and principles prescribed by these abstractions. To achieve this, we jointly train two LLMs via RL post-training: (1) an abstraction generator, and (2) an abstraction- conditioned solution generator. The abstraction generator is rewarded for the improvement in the accuracy of the solution generator, stemming from conditioning on the abstractions it proposes. The solution generator is rewarded to maximize accuracy in solving a problem when using the abstraction. To obtain good initialization for RL. training, we warmstart both models by running supervised fine-tuning (SFT) on data from stronger models. For the abstraction generator, we collect multiple candidate solutions and prompt stronger LLM to generate diverse abstractions. For the solution generator, we generate solutions conditioned on an abstraction.
The main contribution of this paper is the notion of reasoning abstractions, how they can be obtained, training procedures to amplify them via RL training, and an illustration of how they can be used to improve reasoning performance and exploration of the search space. Concretely, we build an approach to imbue LLMs with the capability of proposing abstractions, and evaluate the model on a variety of math-reasoning benchmarks, AIME 2025 (Mathematical Association of America, 2025), DeepScaleR Hard (Setlur et al., 2025), and AMC 2023. We find an average 44% improvement over state-of-the- long chain-of-thought RL approaches (i.e., DAPO (Yu al., 2025)) on AIME 2025, and show an effective benefit from generating diverse abstractions over brute-force solution sampling.
2 RELATED WORK
Scaling test-time compute and exploration. Recent work highlights the promise of scaling test-time compute in different ways. One approach involves parallel sampling: sampling multiple reasoning rollouts and then selecting a winner via a scoring rule (Uesato et al., 2022: Wang et al., 2023; Charniak & Johnson, 2005; Feng et al., 2024; Snell et al., 2024; Yao et al., 2023a; Hao et al., 2023; Snell al., 2024). A complementary line of work iteratively edits single trace, attempting to implement some sort of a sequential search within a single solution trace (Madaan et al., 2023; Qu al., 2024; Qu et al., 2024; Kumar et al., 2024). As such, the sequential approach performs bit worse on harder problems (Snell et al., 2024; Qu et al., 2025), where it often gets trapped in strategies that seem optimal but aren't actually (Pan et al., 2025). Yet it still performs better than parallel search on easier and medium difficulty problems (Snell et al., 2024). Our approach of proposing and leveraging abstractions enables a kind of hybrid between sequential sampling and parallel sampling, guided by the proposed abstractions. Some concurrent work (Pan et al., 2025) studies directly interleaving parallel and sequential samples, and while it is similar in theory to us, it only distills this interleaved structure into the model and does not run RL training to optimize parallel and sequential sampling procedures employed here. Prior work has also utilized mend-deiigned scaffolds to integrate multi-step evaluations of intermediate hypotheses into reasoning (Yao et al., 2023b; Ho et al., 2023; Hao et al., 2023; Li et al., 2023). In contrast, we do not rely on pre-defined interfaces but learn to automatically propose useful abstractions.
Using prior knowledge for LLM reasoning. Several threads of work converge on the idea that textual artifacts such as examples, plans, prompts, can serve as reusable knowledge that steers
2

---

Published as a conference paper at ICLR 2026
LLM behavior. Existing retrieval-augmentt generation (RAG) pipelines assume static corpus, typically of human-writte written text, and focus on improving retrieval heuristics (Lewis et al., 2020; Borgeaud et al., 2022: Trivedi et al., 2022; Verma et al., 2024; Anonymous, 2025; Li et al., 2025). Many works use L.LMs to learn or refine prompts, either in an input-agnostic fashion (Zhou et al., 2022; Yang al., 2023; Pryzant al., 2023; Fernando et al., 2023) through input-specific edits based on feedback (Shinn et al., 2023; Madaan et al., 2023; Gou et al., 2023; Yuksekgonul et al., 2025; Lin et al., 2025). Other related work explores the use of synthetic demonstrations (Zelikman et al., 2022b), scratchpads (Nye et al., 2021), and nemory-augmennt encode prior cubblem-solvingg knowledge. Two recent works demonstrate that LLMs can accumulate and reuse their own experience across tasks (Zhao et al., 2024; Suzgun et al., 2025). While one can view our abstractions as form of prior procedural and factual knowledge produced before the model's solution attempt, this knowledge is (a) input-dependent and (c) is not acquired from an external source at deployment, but rather is "proposed" by the model itself. Imbuing models with this capability requires two-player RL training process. To our knowledge, such procedures have not been used for generating textual artifacts of any type, let alone the abstractions we consider.
agents (Schäfer et al., 2020) to
PRELIMINARIES AND NOTATION
We study reasoning with LLMs, where the LLM is provided access to a problem x, and generates stream of tokens y that ends in an estimate of the answer. We assume access to a rule-based ground-truth 0/1 reward Acox(y,," y") E {0, } that measures correctness of the produced answer y, against the ground-truth solution y" for question x. For training, we are given a dataset Duain = {xx, y: ? of problems x; and solutions y that end with the correct answer. Our goal is to train the LLM K (-|x) such that it achieves high rewards on a test distribution of problems Prest.
We evaluate models via average accuracy under Peett We also measure the pass@k k metric, where for problem x, we sample k solutions y1..... .v (-Ix), and consider the problem to be solved any of these k traces is correct. This metric couples accuracy with diversity, i.c., it attains the largest value when the model effectively finds diverse, good responses. To reduce variance estimating pass@k we sample n 2 k samples per problem and use the unbiased estimator introduced in OpenAl Codex (Chen et al., 2021): - (T79/() where e S n is the number of correct samples.
4 REASONING ABSTRACTIONS AND WHY THEY ARE USEFUL.
Solving reasoning problems often requires composing both procedural knowledge (e.g., how to apply a root-finding algorithm) and factual knowledge (c.g.. relevant lemmas or intermediate results). Current approaches train models to reason via reinforcement learning (RL) on long chains of thought. However, this is often ineffective as RL often tends to optimize for "depth", producing longer traces where each subsequent segment builds on the last segment (e.g., verifying prior calculations), rather than "breadth", i.e., exploring diverse solution strategies or utilizing seemingly irrelevant procedures when needed. We now introduce the concept of reasoning abstractions, concise insights that explicitly encode a range of useful procedural and factual concepts for problem. We describe our approach for generating abstractions and demonstrate mechanisms that make them work.
4.1 PROPOSING GOOD REASONING ABSTRACTIONS BY SUMMARIZING SOLUTION ATTEMPTS
To imbue LLMs with the capability of proposing abstractions, we warmstart the LLM using a dataset consisting of problems paired with a small set of high-quality reasoning abstractions, created synthetically. Perhaps the most natural way to obtain an initial set of reasoning abstractions is to collect a diverse set of traces attempting to solve a problem and then summarize useful concepts appearing in these traces (see Figure 1 for an illustration). More formally, consider the space of possible reasoning traces for a given problem as graph where the nodes of the graph represent intermediate states encountered when solving a question (see Figure 1). Good abstractions would identify useful substructures within this larger reasoning graph. For example, an abstraction can capture whether set of strategies lead to similar outcome or another set of tactics leads to an error being consistently made.
Generating abstractions. We now quantitatively evaluate whether we can generate useful abstrac- tions by summarizing key insights in reasoning and solution traces that solve a problem. To do so, we prompt model to generate solution traces and prompt stronger model to deduce useful patterns from the responses of the first model. Concretely, we utilize the Qwen3 (Qwen Team, 2025) series of models to produce solutions and stronger reasoning model, 04-mini, to generate abstractions.
3

---

Published as a conference paper at ICLR 2026
Abstraction Provider Camparison (4B Model)
GSMBk Accuracy
GSMPlus Accuracy
CSMBe GSMPtus
imprevementt 1951
N
Long Abstt action
Short Abstraction
No Abstroctionn
Dover 68
min (3hor)
oe-mon dumg)
0.68
1.78
4B
0.50
1.78
4B
Solution Model Size
Solution Model Size
Figure 2: Benefits from abstractions hinge on solver scale, abstraction length, and solution model. Most configurations fail to yield gains; only o4-mini with long and detailed abstractions shows consistent improvements across the GSM8k and GSMPlus datasets (left). Solver capability also matters: even strong abstractions help only if the solution model is sufficiently capable (middle, right).
While this approach is not perfect, it enables us to validate the feasibility of the concept of reasoning abstractions and generate warmstart data for training LLMs to propose abstractions. To ensure that the abstractions do not "leak" content of the solution, we verify post-hoc that prompting model with only the abstraction and no problem yields zero accuracy when sampling 16 times from the base model. This makes these abstractions suitable for our study as they only provide useful information while not allowing the model to shortcut to the answer.
Evaluating abstractions. To validate the efficacy of using abstractions, we adopt simple test based on performance after conditional generation. Concretely, let us denote the LLM policy that produces a solution conditioned on the problem x as (-)x). good abstraction z a sequence of tokens that provides some useful procedural and factual information to improve model performance:
E Sg [Acc(E, (oee"y"")] By [Ace(ÿ y "]]
(1)
4.2 RESULTS AND OBSERVATIONS
Evaluation on math reasoning. After generating abstractions, we measure their quality by evaluating Equation 1, i..., by checking if conditioning the problem solver a set of abstractions improves its accuracy. Results in Figure 2 show that conditioning a problem solver on abstractions improves accuracy when two conditions hold simultaneously: (i) the abstraction is not too short (e.g., not just few words that are not informative; example in Appendix D.2) and is generated by a strong generator (o4-mini) and (ii) the solution generator has sufficient instrnction-folloiing capability (Qwen3-1.7B or Qwen3-4BB of interpreting and utilizing the generated abstraction. These results confirm that good abstractions (satisfying equation 1) exist for math problems, but neither the ability to generate them nor the ability to leverage them in solutions arises naturally. In Section 5, we will describe our method for explicitly training models to propose and use such abstractions effectively.
Evaluation on ARC-AGI. We also evaluate abstractions on the ARC-AGI benchmark. We present some details of our setup in Appendix B.3, We evaluate on 90 ARC puzzles evenly derived from the test sets of ARC-AGI 1, ARC-AGI 2, and BARC (Li et al., 2024). In Table 1, we present the pass @k and coverage performance (% of unit tests successfully passed) of the base Qwen3-4B model when conditioned on a proposed abstraction vs not using any abstraction. We see positive improvement in both metrics on this domain over multiple samples, indicating an improvement from utilizing reasoning abstractions.
pass@k k
coverage
k
w/o abs
w/ abs
w/o abs
w/ abs
14.0%
18.0%
19.5%
22.5%
17.5%
22.5%
24.0%
28.5%
4
20.0%
26.5%
28.5%
34.0%
8
22.8%
30.2%
31.8%
38.8%
16.
24.7%
33.2%
35.0%
42.9%
Table 1: pass@k accuracy and max@M @a cov-
erage on ARC-AGI. Abstractions yield con- sistent gains in both metrics.
Interpreting the generated abstractions. We also show some examples of the discovered abstrac- tions in Appendix D.3. We now attempt to interpret these abstractions by classifying them into several categories. We observe that the discovered ubstractions often correspond to useful techniques (e.g., "launchpoint" in Appendix D.3), useful lemma or heuristic principle (e.g., balind-follww in Appendix D.3), and cautionary examples that demonstrate common pitfalls encountered when solving a problem (e.g., "caution alert" in Appendix D.3). These abstractions distill complex reasoning patterns and potential approaches into useful nuggets, allowing models to generalize across struc- turally similar problems. Finally, we want to remind the reader and emphasize that these qualitative results from interpreting the discovered abstractions representative of the process being used to discover them. Our approach for generating abstractions is neither hand-engineered for interpretability or uses any such heuristics beyond summarization.
specific to individual problem, and not
4

---

Published as a conference paper at ICLR 2026
Abstraction Condtiiinn- Solution Generator
91
7(6.99)
π sol
π bbs
r
Abstraction
Reasoning Abstraction
9m
((..2.8))
Completion Reward
Completion
Query
Abstraction Generator
Abstraction-Conddttonedd Solution Generation
Figure 3: RLAD training paradigm. train an abstraction generator, R& that proposes some reasoning abstractions conditioned on the question x. denoted as z. Then, the solution generator, л. is trained to produce response. y. conditioned on the generated abstraction The reward used for training me average success rate of solution generator conditioned on proposed abstraction.
abe corresponds to the
Good abstractions exist in many domains. We also find that the summarization procedure can be used identify an initial of useful reasoning abstractions many problem domains, including healthcare, human behavior, legal reasoning, and web security. Of course, the proportion of an abstraction devoted to procedural knowledge and factual knowledge is different these domains compared to math. That said, we find that using reasoning abstractions improves performance by 30% on average over 37 tasks from RAFT (Alex et al., 2021), CLUES (Menon et al., 2022), and LegalBench (Guha et al., 2023). We show some examples in Figure and full results in Appendix B.3.
Takeaways: Reasoning abstractions summarize insights useful for guiding solution traces
Reasoning abstractions summarize procedural and factual knowledge that useful for learn- ing to solve problems via diverse strategies. Proposing abstractions generated by summarizing solution traces already improves performance of base generators by 30% for math reasoning.
5 RLAD: LEARNING TO PROPOSE REASONING ABSTRACTIONS
Having defined the notion of reasoning abstractions and shown that they improve performance when adhered to for reasoning, we will now develop an approach to train LLMs to be capable of both proposing and utilizing abstractions. Doing so requires training abstraction generator: LLM, z ~ reabe (. (x) that proposes candidate abstractions z given problem x, and an abstraction-coodition solution generator, y ~ TT (-|x, z), that produces solution y given x and abstraction z. Note that is parameterized as variable-length sequence of tokens and might consist of one more facts procedures. While our approach applies to the case when zabs produces more than one abstraction, we abuse notation and subsume multiple abstraction into one avoid clutter. We now describe RL with abstraction discovery (RLAD), our method for training these models via RL.
5.1 TRAINING rgge AND no VIA RI
The core principle behind our approach is that an abstraction z is successful at a given problem x if it can maximally help π (-Ix, 2) find correct responses to the question x, without actually leaking the answer itself. To convert this into an RL objective, we design a reward function that rewards an abstraction z with the expected success of solutions generated by =50l conditioned on z:
T Tagol (x, z) == Ryyagi (-(x.2) [Accx (y.y")),
where y* is the ground-truth answer and Accx(+s)) denotes the 0/1 accuracy on problem x. To train σæ one can then adopt the fairly straightforward approach of maximizing 0/1 binary outcome reward, now conditioned on given abstraction z sampled previously from T aaee akin to recent results RL (DeepSeek-Al al., 2025). Formally, we set the reward for a solution as: r(x, z. := Accx ")) With these reward functions in place, perhaps the most natural approach then would be to train abs maximize ragal for fixed n on a dataset of prompts D Dapht, while also iteratively training
sol
modified prompts generated by concatenating a set of sampled abstraction z on a dataset of problems, Dages. This maximization could be done via on-policy RL (e.g., GRPO (Shao et al., 2024)) or offline RL methods (e.g., DPO (Rafailov et al., 2023), STaR (Zelikman et al., 2022a)). This represents co-operative two-player game.
sol < to maximize the reward function
Challenges with naïve reward design. While the approach so far is extremely simple, it presents some challenges. In particular, the reward functions defined above result in undesirable solutions
5

---

Published as a conference paper at ICLR 2026
Question
Solution
Let be the set of positive integer divisors of 2025, Let 8 be randomly selected subset of A. The probabiiity that is a nonemply set with the property that the leest common multiple Its element 2025 mvn, prime positive integers. men.
But how do we compute the number of subsets with LCM equal to 2025?
From the cheatsheet, therd's mention of using prime factorization and considering exponent vactors. Lot me recall that.
relathelly
represented 3 5*b, where The idea that each each element A determined exponents (a, b). a and
2025
Reasoning Abstraction
<descriptions Model subsets divisors their vectors. be the set exponent choices condition subset divisors correspones to subset "LCM equals N' becomes "for esch prime i the maximum exponent chosen in the subset equals "'dddescciptions
The LCM subset of these elements 2025, which is 3rd 5*2 Therefore, for the LCM of the subsat to be exactly 2025, each prime the subset must have at leest the exponent of 2025 for that prime.
So prime 3, in the subset, there must (since 2025 has exponent for Similnry prime least one element with exponent 2 (since has exponent
element with exponent be
sexample>With primes (2,5] and exponent choices S S-((0,0), (2,1,22,),1333)) the condition "L LC-2-3355 5-2 means we must include least choice with
and expenent least one 21,,,.1,2))
Any valid subset exampies
Therefore, the problem reduces to counting the number of subsets of (including empty set) where:
Figure 4: A typical example of a reasoning abstraction proposed by our abstraction generator. In the solution, we see (in blue) references to the abstraction ("cheatsheet") and keywords from the abstraction being used meaningfully in the reasoning trace of the solution generator model.
in a rather nuanced manner: (1) if m learns to solve problem x in its entirety, it will still be rewarded highly by Tagul but is not a desirable abstraction; (2) if лह is too weak or too strong, such that is either able to always solve the problem x or never solves it, then Tagal will not provide a meaningful signal to update no mhhe; and (3) similar to the above failure modes, training sgal via on-policy RL may result in it ignoring the abstraction z altogether no matter how useful it is. Abstractly, all of these challenges stem from asymmetry in the strength of ngbe and x where one may drown out the learning signal for the other. We therefore build modified reward system for training.
Modifying reward design. We make small but consequential change the training reward system. In particular, we train Te on mixture of prompts x augmented by abstractions z and prompts x without any abstractions at all. In this process, while we utilize Acex as discussed above on a given response, we simply zero out rewards for any trace generated on x without abstractions. When utilizing KL-constrained RL, e.g., GRPO (Shao et al., 2024), =5ol is now trained to closely mimic the distribution of responses as the reference LLM on questions x but must attempt to find ways to optimize reward on the same question x when augmented with an abstraction. This can be accomplished only when ngo learns to utilize the provided abstraction carefully, hence addressing one of the challenges above. Formally, the updated versions of these reward functions are shown as:
(xx,2.7) = [0, Accx y")
if = 0 otherwise
(3)
Txgol (x, 2) = Ey Eg xx) [Accx(xx (f. y").
(4)
5.2 WARMSTARTING x3 AND rg Tabs FROM GOOD INITIALIZATIONS
While the above approach prescribes a recipe for RL training of T& and T& n500, any such recipe critically relies on the ability of the initialized model to generate somewhat meaningful abstractions and meaningful solutions conditioned on the abstraction input, respectively. right from the beginning of RL training. Inspired from the approach of running model the basic structure of long thain-of-une of-thought before RL (DeepSeck-A) Al et al., 2025; Qu et al., 2025), we run an initial phase of SFT to imbue into T ribss and n the basic capabilities of producing abstractions and attempting to follow abstractions respectively, even if the resulting models are not very good. For this initial warmstart phase, we follow the protocol from Section 4 and construct a corpus {x;; 2, M by prompting strong models. For each training problem-solution pair (x,y"), , in our training set, we first generate an abstraction z using an instruction discarding any that leak y". We then sample solution trace y conditioned on (x, z).
initial phase of SFT to imbue into the
model,
Practical approach and algorithm details. For warmstarting the abstraction generator, we utilize abstractions generated by 04-mini. We then use a weaker model (GPT-4.1-min) to evaluate each abstraction by comparing solution success with and without it, retaining only those that improve performance to form our seed set. Then, we SFT of Owen3-1.77 7B for 5 epochs this seed dataset to obtain our initial abstraction generator (Qwen Team, 2025). For the solution generator, we use the same Qwen3-1.7B model, ensuring that both components models of identical capacity.
After SFT, we employ RLAD to fine-tune the abstraction generator and solution generator via RL. For the abstraction generator, opt to "batched" offline RI. via
connctiionno

---

Published as a conference paper at ICLR 2026
RFT (Yuan et al., 2023) and RPO (Pang et al., 2024), since reward computation by rolling out the solution generator the on the fly and running online RL was infeasible using our RL infrastructure and within compute we had access to. To train the solution generator. we utilize the DAPO approach (Yu et al., 2025), and include token-level policy loss normalization and asymmetric clipping, and prompt difficulty/length eurriculum. Building upon implementation of concurrent work (Setlur et al., 2025), we employ two stage curriculum where we partition the DeepScaleR (Luo et al., 2025) mixture by success rate of the base model into three sets: (I) easy, (2) medium, and (3) hard, where we fine-tune first on easy problems with an 8K token budget and then on medium problems. We utilize the hard split as a held out, evaluation subset, which we denote as "DeepScaleR [Hard]". We outline hyperparameters and details in Appendix A. I and provide pseudocode in Algorithm 1.
Summary: RLAD method design
RLAD jointly optimizes the abstraction generator mi and solution generator notl with RL, using reward functions in Equation 3. These reward functions incentivize mool to utilize
abstractions and incentivize Tq to propose useful abstractions per problem.
EXPERIMENTAL EVALUATION
The goal of experiments is to evaluate the efficacy of RLAD in improving the reasoning capabiliities of LLMs through abstraction eestraction-uuieed solution generation. Specifically, we aim to answer the following research questions: (1) Does RLAD improve pass @ I accuracy across several reasoning benchmarks compared to direct solution generation?,
(2) How does RLAD scale as more abstractions and solutions generated?, and (3) What makes the generated abstractions useful, how faithfully they followed, and how do they guide and improve solution generation? We compare RLAD with strong models on three math reasoning benchmarks: AMC 2023, AIME 2025, and DeepScaleR Hard (Luo et al., 2025), which itself is a subset of hard problems from the OmniMATH mixture on which DeepSeek- -1 distilled Qwen-32B model attains an accuracy of I 10%. We also fine-tune tune an abstraction generator for the ARC-AGI program synthesis tasks, and conduct a similar comparison on 90 ARC puzzles evenly derived from the test sets of ARC-AGI 1, ARC-AGI 2, and BARC (Li et al., 2024). We also perform several ablations to better understand abstractions produced by RLAD.
6.1 MAIN PERFORMANCE RESULTS ON MATH REASONING BENCHMARKS
We evaluate RLAD under three settings: (1) w/o abs, without abstractions; (2) w/ abs (avg), average performance over generations conditioned on 4 proposed abstractions per problem; and (3) w/ abs (best): using the best- performing abstraction (in set of 4 proposed abstractions per problem). We observe that RLAD outperforms the base model and variant fine-tuned with RL on the same prompts via DAPO (Yu et al., 2025) without any abstractions, across all settings and benchmarks (Table 2). This highlights that RLAD can propose and leverage abstractions to improve reasoning performance. Interestingly, also note that these performance gains are not limited to instraction-condttiona. inference: even in the w/o abs setting, where no abstraction is provided during inference, RLAD improves over the prior methods, when trained with abstractions via RLAD. This suggests that exposure to diverse abstractions during training enhances the model's general reasoning ability. We observe similar trends additional benchmarks, including AIME 2024 and HMMT 2025 (see Appendix B.)), where RLAD improves in the w/o abs setting. Finally, in Appendix D, we also measure the performance of RLAD when different token budgets are allowed for reasoning while Table 2 measures performance at a budget of 32K tokens, we also measure performance at 8K and 16K budgets and find RLAD be more effective than other approaches.
AIME 2025
DeepScaleR [Hard]
AMC 2023
Approach
w/o abs (avg)
w/ abs (avg)
w/ abs (best)
w/o abs (avg)
w/ abs (avg)
w/ abs (best)
w/o abs (avg)
w/ abs (avg)
w/ abs (best)
Qwen-3-1.7B
33.75
36,25
40.00
20.21
22.14
32,50
86.41
78.01
84,53
DAPO
37.92
34,90
39.79
21.67
21.88
33.54
86.41
81.99
88.44
RLAD (Ours)
38.04
42.45
48.33
23.54
24.84
35.54
87.25
88.35
91.72
Table 2: Accuracy on math reasoning benchmarks. RLAD achieves consistent gains in abstraction-comdiioneed and w/o abstractions. Here, we measure performance on domains (AIME 2025, DeepScaleR Hard, and AMC 2023) with the base Qwen 3-1.7B model, DAPO, and RLAD. We measure performance without abstractions, with abstractions (pass@1 1 with 16 samples) and the best abstraction (pass@ 16), for each method type.
7

---

Published as a conference paper at ICLR 2026
Abstraction and Solution Generation Tradeoff Test-Time Scaling
20
3-
Figure 5: Tradeoff of abstraction and solution generation on AIME 2025. As the total inference compule budget increases (color scheme on the right). we find better performance efficiency when allocating our budget to abstraction generation rather than solution generation. for all values of normalization offset ka given to us.
6.2 UNDERSTANDING PROPERTIES OF RLAD
In this section, we conduct number of additional experiments understand the behavior of RLAD in regards to the usefulness of the proposed abstractions and algorithm performance when provided with large budget on the total inference compute.
1) "Weak-to-strong" generalization of the abstraction generator. We next evaluate the weak-to- strong generalization of our method by pairing our trained abstraction generator with 04-mini the solution generator. We use a fixed 24k generation token budget and 4 samples per question. Without abstractions, 04-mini achieves 80.38% pass@) 1, 82.26% pass@2, and 84.77% pass@4, Conditioning on both the problem and the proposed abstractions improves performance to 85.83% pass@ 1, 88.33% pass @2, 2, and 90.00% pass@4 @4 accuracy. Thus, conditioning on abstractions consistently yields higher pass @k k accuracy compared to question-only conditioning, even for this strong reasoning model. These gains demonstrate that abstractions, though produced by a comparatively weaker model, can transfer effectively to a stronger solution generator, providing evidence that abstractions generalize in a weak-to-strong setting and enables downstream improvements without additional supervision or modifications to the strong model.
2) Compute tradeoff's b/w abstraction and solution generation. Next, we study how to allocate compute be- tween generating more ab- stractions and sampling so- lutions to attain maximal performance within a given inference budget. This corresponds to a "compute-optimal strategy" (Snell et al., 2024) for partition- ing compute between abstraction and solution generation. If the model typically fails by making small local errors in its computation, then additional concise abstractions may not help it as much as simply trying again to generate locally similar solution (optimizing "depth""). In contrast, if the model tends to pursue seemingly plausible but incorrect approach and is unable easily recover or switch approaches, then conditioning on diverse abstractions can help by offering alternative high-level approaches toward the correct answer. In other words, when the model has tendency explore "depth"" over "breadth" of solution strategies, abstractions can help improve performance. With this intuition, we hypothesize that when the compute budget permits only limited number of samples, allocating more compute to sampling multiple solutions but retaining only few abstractions will improve success more. However, once pass @k for a single abstraction begins to saturate, performance gains are more likely to come from scaling the diversity of abstractions, which enables the model to explore qualitatively different regions of the solution space.
4
16
64
256
DAPO, w/o abs (pass @n?) RLAD, w/ abs (pass n x n)
0.37
0.51
0.65
0.77
0.82
0.41
0.59
0.71
0.80
0.87
Table 3: Pass@k comparison under equal compute. For each n, compare n² solution samples (no abstractions) with n abstractions solutions per abstraction. Abstraction conditioning yields consistent improvements.
To validate this hypothesis, we plot iso iso-compute scaling curves under a fixed compute budget C. we distribute between abstractions and solution generation. Specifically, we denote the number of abstractions m and the number of solutions sampled per abstraction as k. such that m x k = C. To better isolate the utility of abstractions, we make several more projections of the iso-compute curve, each with different "normalization offset" ko. This normalization offset Ko discounts performance gains that stem from trying the problem again and making local modifications (e.g., small edits that do not require new strategy changes), but measures the performance gains that stem from major changes in content of a response. Concretely, for a non non-zero kn, we plot iso-compute frontiers when m x (I- ko) C. This formulation captures the amount of compute that is spent on "meaningful"
8

---

Published as a conference paper at ICLR 2026
Distribution of Semantic Similarity (4)
Abstraction- Adherence Rates
43.33%
Costre Sinilarity
0:86
0.85
Adherence Rate (1)
0.82
30
0.8
20.42%
0.7
0.6
Type
Abstraction Type
Figure 6: instraction-coniitinn solution generation analysis. RLAD produces solutions with (left) greater semantic diversity different abstractions and (right) higher abstraction adherence than baselines.
samples that go beyond the model's local neighborhood. Figure shows these iso-compule frontiers for different values of the compute budget C. The x-axis plots the ratio between abstractions and adjusted solutions, m/ (k: ko). Each curve corresponds
fixed total compute budget.
In Figure 5, across kn E {0, 2, 4, 6, 8), shifting compute toward abstractions consistently yields greater improvements than allocating the same additional compute solution refinements, especially as the total compute budget increases. This supports the conclusion that once local errors in the chain-of-the offthhugghtt have been addressed, more effective to increase the diversity of strategies used via abstraction conditioning rather than to continue to scale long CoT sampling alone.
To obtain additional evidence support this result, we also evaluate pass@k of DAPO and RLAD for a fixed compute budget for both abstraction and non-abstraction generation on AIME 2025. Here, for given value n, we compare two approaches: (1)sampling n solutions without abstractions, and (2) sampling abstractions and then solutions per abstraction. In Table 3, abstraction conditioning consistently outperforms pure solution sampling. For example, at n = 16, abstraction conditioning achieves pass @k of 0.71 versus 0.65 without abstractions. and versus 0.82. This corroborates the efficacy of abstractions, even under matched compute budgets.
256 the gap widens to 0.87
3) Understanding the behavior of the abstraction-condiiinne property of the solution generator is the ability to follow the proposed abstractions. To study this, prompt (o4-mini) to classify whether a particular solution trace produced by trained solution generator closely adheres to a given abstraction. We ask for a binary decision on each pair of abstraction and solution, and measure the adherence rate across 200 such pairs. In Figure (right), report the adherence rates under four conditions: (1) Abstract raction, where we measure adherence rates between an abstraction and solution generated by conditioning on this abstraction itself; (2) No abstract iaaciion. where we measure the adherence rates between an abstraction and solution generated by conditioning on the problem without any abstraction; (3) Ret eval, where we measure adherence rates between an abstraction and a semantically similar prior solution to the problem; and (4) Unrelated an abstraction and solution generated via different abstraction. We find that the Abstract i in condition achieves the highest adherence rate. Intuitively, this means that the trained solution generator detected to be more likely follow the strategy guidance of a given abstraction. Additionally, measure the semantic similarity of solutions generated without abstraction conditioning, conditioned the abstraction, and across abstractions and make similar conclusion.
solution generator. A desirable
abstracte ion, where we measure the adherence rates between
Takeaways: Experimental Results
RLAD outperforms RL fine-tuning approaches that do not propose or leverage abstractions on math reasoning. Jointly scaling the number of abstractions and solution samples enables continued performance gains even when scaling solutions alone begins to saturate.
7 DISCUSSION AND PERSPECTIVES ON FUTURE WORK
We introduce reasoning abstractions: concise natural language representations of procedural and factual knowledge, as a way to expand LLM reasoning strategies. Our method, RLAD, jointly trains solution generator, achieving consistent gains
an abstraction generator and abstraction-condition on mathematical reasoning benchmarks. We show that allocating compute to diverse abstractions yields larger improvements than simply increasing solution sampling, offering complementary axis for scaling test-time compute. While our study focuses on math tasks, extending abstractions to broader reasoning and unifying abstraction and solution generation remain open directions.

---

Published as a conference paper at ICLR 2026
8 ETHICS STATEMENT
Our empirical evaluation focuses primarily on mathematical reasoning (e.g., AIME, AMC, Deep- ScaleR) and abstract pattern recognition (ARC-AGI). These domains are generally considered low-risk, as they utilize publicly available benchmarks that do not contain private or sensitive per- sonal data. The risk of generating toxic or harmful content in these structured reasoning contexts is minimal compared to open-ended generation tasks. However, we acknowledge the broader ethical considerations associated with advancing LLM capabilities. While enhanced reasoning abilities offer significant potential for scientific and educational benefits, they also carry dual-use risks if applied to malicious activities, such as generating sophisticated disinformation. We encourage the continued development of robust safety and alignment protocols alongside improvements in reasoning capabilities, which are jointly correlated as evidenced in prior work (Kim et al., 2025). Finally, we utilized LLMs such GPT5/Gemini for minor rewritings throughout the paper for better readability.
REPRODUCIBILITY STATEMENT
We committed to ensuring the transparency and reproducibility of our research. To facilitate the replication our results, we have provided comprehensive details regarding our methodology and experimental setup in the Appendix and main paper. In particular, the RLAD framework, including the two-player RL training paradigm and the specific reward design (Equations 1, and 3), is detailed in Section 5. The procedure for warmstarting the models via Supervised Fine-Tuning (SFT), including the generation of seed abstractions using stronger models (o4-mini and GPT 4.1imini) and the filtering process, is described in Section 5. The implementation details of the curriculum learning strategy are also provided in Section 5 and analyzed in Appendix B. Additionally, detailed implementation information provided in the Appendix. Here, the pseudocode for the joint RL training process is available in Appendix A. (Algorithm 1). All key training hyperparameters used for the RL methods (DAPO, RFT, RPO) are listed Appendix A. (Table 4). Our experiments utilize publicly available base models from the Qwen3 series (Qwen Team, 2025). The evaluations are conducted on standard reasoning benchmarks: AIME 2025 (Mathematical Association of America, 2025), DeepScaleR (Luo et al., 2025), AMC 2023 (Zeng et al., 2025), and ARC-AGI (Li et al., 2024), described in Section 6, with additional results in Appendix B. Qualitative examples and the prompt used for abstraction classification are included in Appendix D.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C Jess Riedel, Emmic Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, et al. Raft: A real-world few-shot text classification benchmark. arXiv preprint arXiv:2109.1400 2021.
Anonymous. Optimizing inference- reasoning in LLMs via retrieval- augmented reflection, 2025.
URL httptts//pppeeeeiiiieeee net forum? i 1d=E1 YRG3pJev.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-2240. PMLR, 2022.
Eugene Charniak and Mark Johnson. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting on Association for Computational Lin- guistics, ACL '05, pp. 173-180, USA, 2005, Association for Computational Linguistics. doi: 10.3115/1219840.1 1219862. URL https: /oii http:////o..o/o//////oo////o/o
www/oo.g//./. 311.0.3.111//1 1111111//111111111111111 1219862.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol. Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL https: /atpss//arrke /ttxiviivvvv/////////.../////ava org/ abs//ba//// 2217 03374.
Dee,Seek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li. Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni. Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I. Jin, Ruyi Chen. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu. Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei. Yang Zhang, Yanhong Xu. Yao Li. Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepscek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https: //tr///raa /arxivi v. org/ abs/ 12948.
11

---

Published as a conference paper at ICLR 2026
Xidong Feng, Ziyu Wan, Muning Wen, Stephen Mareus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like sree-search can guide large language model decoding and training, 2024.
URL https: http://aaxvvvvoo/oo org/ abs
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rock- täschel. Promptbreeder: Self-referential improvmmprveement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738. 2023.
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36:44123 44279, 2023.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model, 2023. URL https: : hitas://aaxxiii /arxiv. org/abs 2305 14992.
Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers, 2023.
URL htwp::tpp://om //apiiii///a/v...////a/ org/ abs/2212. 10071.
Tacyoun Kim, Fahim Tajwar, Aditi Raghunathan, and Aviral Kumar. Reasoning as an adaptive defense for safety, 2025. URL httph:tts/s/eett /ttp:::///axvv..oo org. abbs//// matp//557.0000
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Alcksandra Faust. Training language models to self-correct via reinforcement learning, 2024. URL https: //arxiv. wttg abs/ 2409. 12917.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. eetrieval-augment genera- tion for knowledge-intensit nlp tasks, Advances neural information processing systems, 33: 9459-9474, 2020.
Guohao Li, Hasan Abed AI Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model socicty, 2023. URL
https: : //arxivv...///////////a////aaavr www//bgg//......ooooo.. 2303 17760.
Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M. Dunn, Hao Tang, Michelangelo Naim, Dat Nguyen, Wei-Longg Zheng, Zenna Tavares, Yewen Pu, and Kevin Ellis. Combining induction and transduction for abstract reasoning, 2024. URL https: //arxxv.
2411. 02272.
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and large reasoning models, 2025. URL https:
Zhicheng Dou. Search-ol: Agentic // wwaaxiiiorrg....o...aao.o....ooa abs hta///baa225000 /200..500000 05366.
Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, and Joseph E Gonza- lez. Sleep-time compute: Beyond inference scaling at test-time. arXiv preprint arXiv:2504. 13/71, 2025.
Michael Luo, Sijun Tan, Justin Wong. Xiaoxiang Shi, William Tang. Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol- preview with a 1.5b model by scaling rl. https: http://///ettttttiaiee DeepSca 1 2025. Notion Blog.
not ion. tit
-ww.siing------eiee-eeeeee wwweriewwwtth-eeee Mode 1oyy----------- ing- RI- 9681 902c1 468005t
Jingyuan Ma. Damai Dai, Lei Sha, and Zhifang Sui. Large language models are unconscious of unreasonability in math problems. arXiv preprint arXiv:2403. 19346, 2024.
12

---

Published as a conference paper at ICLR 2026
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uni Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Relf-Refine: Iterative Refinement with Self-Feedback. arXiv e-prints, art. ar arXiv:2303. 17511, March 2023. doi: 10.48550/a2Xx..2333.7766
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534 2023.
Mathematical Association of America. 2025 American Invitational Mathematics Examination (AIME)
I: Problems and Solutions, February 2025. URL wiki/ falki/index..
//rt://prtttotoooo emsol lissoviingg ving. com/ AIME I. Art of Problem Solving Wiki entry.
index.iix...... beoddx..hp/ae/ p2p5
Rakesh R. Menon, Sayan Ghosh, and Shashank Srivastava. Clues benchmark for learning classifiers using natural language explanations. 2022.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprini arXiv:2410.05229, 2024.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021.
URL https: arw.aaiivv...o. wrwxiv.org//ba.. hbtp://b///22 2112. 00114.
Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. Learning adaptive parallel reasoning with language models. arXiv preprint 2xXiv:2504.15460 2025.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.197333 2024.
Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislay Balunović, Nikola Jovanovié, and Martin Vechev. Proof or bluff? evaluating Ilms on 2025 usa math olympiad. arXiv preprint 2rXiv:2503.21994 2025.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint 3xXi::2305003493 2023.
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive Introspection: Teaching Language Model Agents How doi: 1044850W/nXiv.2447.1822
Self- Self- Improve. arXiv e-prints, art, arXiv:2407.82119, July 2024.
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint arXiv:2407.18219 2024.
Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine- tuning. arXiv preprint arXiv:2503.07577 2025.
Qwen Team. Qwen3 technical report.
Technical report, Qwen, May 2025. URL Available at: https://hugggngfacccco///// https//ggtuubbcomm/eeeea///////
https: httpss////ggggggggggeeeee co/Owen. http:///wwsssoooooooooo/aa//////
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53748 2023.
Marlin B. Schäfer, Frank Ohne, and Alexander H. Nitz. Detection of gravitational wave signals from binary neutron star mergers using machine learning. Physical Review D, 102(6), September 2020. ISSN 2470-0029, doi: 10. 103/physrevd. 102.063015. URL http: /xx doi.org/10 11 11/00//PPsee// 02.063015.
13

---

Published as a conference paper at ICLR 2026
Amrith Setlur, Matthew Y. R. Yang, Charlic Snell, Jeremy Greer, lan Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for llms. 2025. URL https: http:://aaxiv.oo/ org. abs 2ttp//2266000002
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634 8652, 2023.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM Test-Time Compute Opti- mally can be More Effective than Scaling Model Parameters. arXiv e-prints, art. arXiv2448.03314 August 2024. doi: 10.485500arXiv.2400.. 033144
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03311, 2024.
Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory. arXiv preprint arXiv:2504.07952, 2025.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with thouugh reasoning for mnowledge-intensivy multi-step questions. arXiv preprint 12Xiv:2212.10500 2022.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022. URL
org/abs/ abwpaassl.ll.///////// 2212.112222
Prakhar Verma, Sukruta Prakash Midigeshi, Gaurav Sinha, Arno Solin, Nagarajan Natarajan, and Amit Sharma. PlanxRAG: coanning-guidees Retrieval Augmented Generation. arXiv e-prints, art. arXiv:2410.20753, October 2024. doi: 00-48550/aXXi:224022775
Xuezhi Wang, Jason Wei, Dale Schaurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URL https: wtw.iv.vvvvv./////i///a.//...ava org/abs /2taaabb///2...a /20032223202 11171.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint (xX.iv3339003400 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimban. Tree of thoughts: Deliberate problem solving with large language models, 2023a. URL https::t///.......
/wrxivaiivv/v////////ea////aaavv org/ abss2305 2305. 10601.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023b. URL https://arxiv org/abs/ 2210 03629.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source I1m reinforcement learning system at scale. arXiv preprint arXiv:2503. /4476, 2025.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.0188 01825, 2023.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13877 2025.
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 63968055):609-616 2025.
14

---

Published as a conference paper at ICLR 2026
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 35:15476- 2022a.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022b. URL https:ttt/// htctps:/aaxivv.rgg/ 2abb//2033. 14465.
Changyu Zeng, Yifan Wang, Zimu Wang, Wei Wang. Zhengni Yang, Muyi Bao, Jiming Xiao, Ahn Nguyen, and Yutao Yue. Numina: A natural understanding benchmark for multi-dimensional intelligence and numerical reasoning abilities, 2025, URL https: http/aaivvvvv/////ee//aaae .rg. abs / 2509. 16656.
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19632-196422 2024.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are numan-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2022.
15

---

Published as a conference paper at ICLR 2026
Appendices
A
EXPERIMENTAL DETAILS
A.1 PSEUDOCODE FOR RLAD
Algorithm Joint RL Training of
mg abs and 501
(z Require: Policies m6 (z x), To (y x, Datasets Daghu D Digat rates a aggha, Ongal' batch sizes N, M;
epochs E
Initialize mp whb ri
2: for e to E do
D Update abstraction policy
D rg do 4: 2i (-(x,()
3: for fx, hi
5:
Pi + a0 (x,, 2.)
6:
a Tge V sh (
7:
end for
Update solution policy
8:
for
D do {x, ]=1
6
9:
3j T6 bba (-1x;);
y, ~ no (+12),
10:
Tj rxx, 2g
11:
Rg t sol
a rgl
V noa v (,, 25, Bi,r))
for
13: end for
A.2 HYPERPARAMETERS
Hyperparameter
Value
algorithm
DaPO (Yu et al., 2025)
training steps
100
epochs
10
train batch size
128
max prompt length
3072
max response length
16384
max extrapolation length
32768
learning rate
1e-6
clip ratio (low high)
0.2 / 0.5
entropy coefficient
0.001
KL loss coefficient
0.001
KL loss type
low var kl
sampling temperature (train / val) samples per prompt (train val) max batched tokens
0.6 / 0.6
16 / 8 32768
Table 4: Key training hyperparameters used in RLAD.
B ADDITIONAL EXPERIMENTAL RESULTS
B. RLAD'S W/ ABS PERFORMANCE ON AIME 2024 AND HMMT 2025
In this section, evaluate the performance of the base model (Qwen-3-1.78) GRPO-enhanced model, and our proposed method RLAD on two math reasoning benchmarks: AIME 2024 and HMMT 2025. As shown in Table our method achieves the best performance across both datasets.
is important to note that RLAD is trained using access to abstractions, yet it also generalizes better when evaluated without abstraction. This suggests that RL.D does not merely overfit to the
16

---

Published as a conference paper at ICLR 2026
abstraction format but instead learns to effectively leverage high-level procedural guidance, leading to better generalization on challenging reasoning benchmarks.
Approach
AIME 2024
HMMT 2025
Qwen-3-1.7B
48.54
22.50
DaPO
44.17
23.13
RLAD
51.46
23.75
Table 5: RLAD's w/ abs performance AIME 2024 and HMMT 2025. RLAD outperforms both DaPO and the base Qwen model the AIME 2024 and HMMT 2025 benchmarks.
B.2 DESIGN CHOICE ABLATIONS
In this section, we conduct ablation studies to isolate the contributions of key components in RLAD. We investigate three primary design choices, with results summarized in Table (a) utilizing curriculum training strategy, (b) including prompts not annotated with an abstraction, and (c) applying reward masking to those non-annotated prompts.
Curriculum training refers to staged process where the model first learns from simpler problems and gradually transitions to harder ones. We use the protocol from Setlur al. (2025) as inspiration, who demonstrated its effectiveness for direct math problem-solving. In our setting, which incorporates abstractions, curriculum training also proves beneficial, improving both average and best-case performance from 0.38 and 0.43 to 0.41 and 0.48, respectively, compared to non-curriculmm training.
Next, we analyze the practice of including prompts without abstractions and applying reward masking. Including small fraction of these "no-abstraction' prompts is intended to better condition the solution- solution-generator on the abstractions when they are present. However, this risks the model learning shortcut by simply ignoring the abstractions. To mitigate this, apply reward masking: for completions on no-abstraction prompts, we nullify the policy reward by zeroing out the advantage, while retaining the KL. penalty for regularization. This prevents the model from over-optimizing examples that lack abstractions, a behavior that would otherwise hinder generalization.
Our findings confirm the efficacy of this combined approach. As shown Table 6, including no- abstraction prompts with reward masking is critical for performance. Ultimately, the combination of all three design choices curriculum training. the inclusion of no-abstraction prompts, and reward masking significantly outperforms alternative configurations.
Design Choice
AIME 2025 w/ abs (best)
Approach
curriculum truining including no-abstraction prompt reward masking
w/ abs (uvg)
variant 1
X
36.51
42.29
variant
37.08
42.50
variant 3
37.50
43.33
RLAD
42.45
48.33
Table 6: Design Choices in RLAD. We isolate the effects of curriculum training, no-abstraction inclusion, and reward masking. The full method achieves the strongest performance under dnstraction-condiiionn evaluation.
B.3 FULL RESULTS FOR ABSTRACTIONS NON-MATH DOMAINS
As seen in Table 7, conditioning on abstractions helps for 37 domains, where the average and best abstractions outperform standard prompting by 18.0% and 30.0% on average, respectively.
For ARC-AGI, warmstart the abstraction generator model with synthetically augmented human annotations from BARC dataset (Li et al., 2024). Table 1, we report results using an abstraction generator and solution generator instantiated Qwen3-4B.
C ABSTRACTIONS IN OTHER DOMAINS
D QUALITATIVE EXAMPLES OF MATH REASONING ABSTRACTIONS
17

---

Published as a conference paper at ICLR 2026
Breast Cancer Dotection
Twoet Hate Speech Detection
Corporate Lobbying Relevance
Bank Note Autheniication
Classlyy breast
1.lf 4-4, predict
meligaant benign RADS, shape, and margin crtaari.. ..ff BI-RADS 5, then mal malignant. ..ff BI-RADS 4 AND shape a iregular AND margin defined, then malgnaant..
derogatory [e beees cane geees, fine, (aaasesaa)
labwingg
rembursement,
2. Else verisnce predict Originl.
R&D funding, licansing the company's core products services, label "Yes."
Abstraction
classity as hate speech. ..ff the text degrades or dehumanizes protected group (nationality, race, religion..
3. Eise skewness 6, predict Fake.
... Else if the bill ahars tanes, creaits, bonds, intrastructur...
4. Elke entropy > and
skewness predict Fake.
GPT-4o-mini
60%
88%
GPT GPT-4o-mini Abs
90%
90%
94%
100%
Figure 7: Examples of good reasoning abstractions in non-math domains. Adding the abstraction to the prompt of GPT--4omini consistently improves performance unseen instances.
Types
200
Interpreting discovered abstractions. As dis- cussed in Appendix D, we classify each model- generated abstraction into four categories for case of interpretability: (I) caution alert that warns the solution generator avoid specific approach; (2) productive launchpoint that sug- gests strategic framings or problem reformula- tions that open high-potential solution paths; (3) blind-follow trajectory that prescribes repeat- able, step-by-step procedures executable with- out Aurthe insight; and (4) structural short- cut that leverages abstract insights invariants to collapse multiple reasoning steps into a single leap. In Figure 8, show that after training via RLAD, the distribution over these categories shifts, with notable increase in blind-follow abstractions, which stronger reasoning model classifies as an effective reasoning path to a successful solution as seen in Appendix D.
5.5%
Coultir Alitt
Lo huaer
Poliee
Figure Abstraction Categorization RLAD produces a diverse characterization of abstractions, which we char- acterize by prompting 04 -mi ni.
D.1 PROMPT FOR ABSTRACTION CLASSIFICATION
We prompt GPT-4o-mini with the following prompt template to classify each abstraction into one of four categories.
Post-hoc abstraction classifier prompt
You are a abstraction classifier. You will be given problem-solving heuristis abstraction used for mathematical reasoning. Your task following mutually exclusive categories, based on the primary cognitive function the heuristic serves.
classify into exactly one of the
(A) Caution alert: any abstraction that warns the reader to
double-check specific aspect of their solution to not take speci fic approach the problem.
(B) Productive launchpoint: an early move or framing that opens up high-potential trajectories. Examples Include clever reformilations or symmetries.
repeatable, sequential path that can be reliably followed to solve the problem. Examples include plug-and-play formulas that can be followed blindly, without insight. Do not choose this is further reasening is required to solve the problem.
(C) Blind-follox trajectory:
description
(D) Structural shortcut: graph paths into a single jump via insight or abstraction. This can include introducing invariants.
conceptual move that collapses multiple
(E) Other: a abstraction that does not fit into the above categories.
Give a 1-2 sentence explanation for your classification, and end your answer with exactly one of: (A), (B), (C), (D), or (E).
18

---

Published as a conference paper at ICLR 2026
abstraction
labstraction]
D.2
EXAMPLE FOR SHORT ABSTRACTION
Examples Short Abstractions
Question:
banker's s gain of a certain sum due 3 years hence at 108 per annum is Rs. 36. what is the present worth? Hint:
Consider the proportional relationship between the true discount and the present worth, and how the banker's a gain relates to these terms.
Think of the banker's gain as a component of the true di discount and how it relates the present worth through the interest rate and time period.
Question: train 125 m long passes a man running at km/hr in the same direction in which the train is going, in 10 seconds. What is the speed of the train? Hint:
Think about the speed difference between the train and the man, and how that difference relates to the time it takes for the entire train to pass by.
Consider the difference in speeds as the key factor in determining the train's speed relative to the man, and use the time taken pass the man find this difference.
D.3 EXAMPLE FOR EACI ABSTRACTION CATEGORY
Below, we show examples of abstractions classified into the four categories above.
Examples of (A) Caution alert
<description>Always record forbidden values from denominators before and after manipulation. After solving the polynomial, discard any roots that make a denominator zero or that do not satisfy the original equation, to avoid rxteess <example>In the equation (x+2) (2x-1) - x-3, 2x-1 cannot be zero (so
any that makes any denominator zero, reject it. Then verify the accepted rects the original equation.<eexample </example>
k). solving yields
<description-Keep units consistent when moving between area and length summing lengths. After extracting length from area (via square root), ensure subsequent arithmetic stays in the same unit to avoid scaling errors. </description>
<example>IE a square's s area is 10000 cm', its side is sqrt (10000) = 100 cm. To express in meters, convert 100 en to distances computed with that side length must be in meters remain consistent. crasssstntt./oxaalllll..
All later
Examples of (B) Productive launchpoint
<descriptlon--Tasslltte comparative statements into algebraic equations using the chosen variables. Phrases like "twice as many" or "one less than" correspond to miltiplication or
19

---

Published as a conference paper at ICLR 2026
cudition/subtractiion expressions. This step captures the core relationship in solvable fom.</description>
<example>IE the problen states "Group A has twice as many as Group B, write the equation x 2y. For "Group B has three fewer than Group C," you would write = 3.</cxamplo>
variable to or keeping it symbolic) terms of it. This reduces the number of independent symbels and streamlines substitutions s
<description<Select
parameter (often setting it express all other variables in
<example>Given p/q = and z/q = 2, choose as the base variable. Write 3q and 2g, all expressions involving p and be handled through alone. </example>
Examples of (C) Blind-follow trajectory
<descriptiion-lggrittmm offer a streamlined way to compute
floor-based digit counts: for y>0, the number of integer digits is floor (1og10 y) juggling large powers <example>1o count digits of y x², compute d floor (7 * 1og10
Use this to handle arbitrary exponents without
x) 1. If x-2.5, then d floor (7 * log10 (2 2+1 -
number of elements. of this to move between sums and averages when counts or totals are known. It works because "average" is defined as that ratio. s/descri//iciiinnnn <example>Suppose
<description<The
equals its total divided by its
Then its total subset has k items with mean and the count k, sum is k.m. Conversely, if you know the the mean is m S/k. For instance, if 5 items average to 10, their total is 5x10 = 50, and if you later learn the total is 60 for 6 items, the new mean becomes 60/6 = 10. </example>
Examples of (D) Structural shortcut
<description>When the distance appears in multiple geometric roles (e.g., as radius to vertex and to tangen t point), express it in different algebraic forms and equate them. Sclving the resulting equation produces the unknown variable, which then gives the desired length. s/decriptnts
<example>Ie r is both the distance from to a vertex (r sqrt (x (1/2))11) and the distance from 0 to the tangent point (r fx))), set sqrt [xa + (1./2) f(x). Solving this equation for and back-substitutin determines explicitlly closing the gecmetric problem with algebraic
<descriptien>Use the perimeter constraint atb+c=P to eliminate one variable, e.g. set c=P-a-b, reducing the problem to two degrees of freedom. This simplification turns the three-variable Heron expression into function of and b alone, facilitating analysis soummeaaion../desrriieee. <example>For target perimeter P=10, one writes c=10-a-b. Substituting into Heron's formula yields blarb)= * (5-a) * (5-b) (a+b-5)], which is now a two-variable function to study instead of three.</este <hreel</example>>>
20

---

Published as a conference paper at ICLR 2026
Dataset
Zero-shot
Best
Average
Abstraction
Abstraction
UCT Dry Bean
0.00
0.65
0.51
Wikipedia Proteinogenic Acid
0.22
0.78
0.58
UCI Student Performance
0.25
0.45
0,28
UCI Website Phishing
0.25
0.25
0.22
UCI Teaching Assistant Evaluation
0.25
0.45
0.33
UCI Contraceptive Method Choice
0.30
0.60
0.43
UCI Vertebral Column
0.30
0.75
0.64
UCI Shill Bidding
0.30
1.00
0.95
Kaggle Job Change
0.30
0.85
0.83
UCI Caesarian Section
0.38
0.75
0,64
Wikipedia Coin Face Value
0.40
1.00
0.88
UCI Wine
0.40
0.95
0.85
UCI Tic-Tac-Toe Endgame
0.40
0.80
0.42
Kaggle Campus Placement
0.40
0.85
0.72
Wikipedia Driving Championship Points
0.40
1.00
0.74
UCI Mammographic Mass
0.45
0.90
0.82
UCI Banknote Authentication
0.45
1.00
0.78
Kaggle Engineering Placement
0.50
0.85
0.79
RAFT One Stop English
0.50
0.40
0.36
LegalBench Function of Decision Section
0.54
0.72
0.61
Kaggle Entrepreneur Competency
0.55
0.65
0.58
UCI Indian Liver Patient
0.55
0.80
0.68
LegalBench International Citizenship Questions
0.56
0.74
0.63
LegalBench Abercrombie
0,56
0.80
0.67
Wikipedia Color Luminance
0.60
1.00
1.00
RAFT Twitter Hate Speech
0.60
0.90
0.76
Wikipedia Award Nomination Result
0.64
1.00
0.76
UCI Car Evaluation
0.65
0.75
0.64
Kaggle Water Potability
0.65
0.50
0.38
Kaggle Travel Insurance
0.65
0.70
0.59
UCI Internet Firewall
0.70
1.00
0.97
RAFT ADE Corpus
0.70
1.00
0.89
UCI Somerville Happiness Survey
0.70
0.80
0.68
UCI Mushroom
0.75
1.00
0.95
UCI Occupancy Detection
0.80
1.00
0,92
Kaggle Stroke Prediction
0.85
0.90
0.90
LegalBench Corporate Lobbying
0.88
0.94
0.88
Average
0.50
0.80
0.68
Table 7: Evaluation of abstractions diverse collection of 37 domains. We sampled 10 abstractions by prompting 04-mini, and measure test set accuracy while prompting GPT-40-mit mommmnn with each abstraction. We report both the average performance of the 10 abstractions and the best abstraction. We find that the average and best abstractions outperform standard prompting by 18.0 % and 30.0% on average, respectively.
21

---

Published as a conference paper at ICLR 2026
D.4
THE USE OF LARGE LANGUAGE MODELS (LLMs)
Large Language Models are used to assist with proofreading and minor wording improvements. All research ideas, experiments, and conclusions were conceived and validated by the authors. Additionally, tools such as Cursor were utilized as coding assistants during the development of the coding infrastructure for the project.
22


Reviews:
[{"review_id": "YNA8jDyxuO", "reviewer": "Reviewer_JioX", "summary": "This paper introduces RLAD, an approach to RL post-training of language models. The approach trains both a reasoning abstraction generator and an abstraction-conditioned solution generator in a cooperative two-player game set-up. Experiments demonstrate the success of the approach, and analyses demonstrate the utility of reasoning abstractions.", "questions": "1. For ARC, I’m curious how the results would compare with another baseline: best-of-N. It is known that in domains where external verification is available (such as ARC), best-of-N is a powerful strategy. So I’m wondering, is adding abstraction generation more cost-effective or best-of-N more cost-effective (Table 1)?\n\n2. Lines 70-71 says “The abstraction generator is rewarded for the improvement in the\naccuracy of the solution generator, stemming from conditioning on the abstractions it proposes.”\nBut Equation 2 is the expected reward of the solver conditioned on z, whereas from that sentence I would’ve expected it to be difference w.r.t. the reward of the solver without z. So does the actual reward subtract or not subtract the solver’s reward without conditioning?\n\n3. Did training the DAPO baseline include SFT warm-starting similar to that used in RLAD? (Weakness A1 and Question 4 currently assume the answer is “no”, but I’m not completely sure because I didn’t find any details regarding the DAPO baseline in the paper.)\n\n4. What is the performance of RLAD without warm-starting? Alternatively, how does RLAD compare with a version of DAPO that includes a similar warm-starting procedure? (Also see Weakness A1 regarding fair comparison.)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. *Originality:* Although the idea of reasoning abstractions has existed for a long time, this is likely the first work that explicitly uses RL to train the generation of reasoning abstractions as part of RL post-training.\n2. *Quality:* Comprehensive experiments and analyses are used to demonstrate the utility of reasoning abstractions for solution generation and training to generate them.", "weaknesses": "### A. Evaluation\n\n1. My main concern is with the rigor of evaluation, where there’s a reasonable doubt of unfair comparison. The model was warm-started with SFT data in the beginning, similar to DeepSeekR1, but was compared with DAPO, which doesn’t have warm-starting*.\n\n*I’m not completely sure about this since I couldn’t find details on the DaPO baseline in the paper—also see my Question 3 below.\n\n2. Furthermore, the SFT data for the abstraction generator was generated with models (o4-mini) significantly stronger than the models being post-trained (Qwen3-1.7B), which raises the question of how well the approach scales/how RLAD would be applied to models that are already among the strongest.\n\n3. The ablation studies (Appendix B2) do not study how much the performance gains are from the SFT data generated with stronger models vs. other parts of the training pipeline. (Also see Question 4.)\n\n4. The definition of iso-compute scaling curves is not well-motivated.\n  - It is unclear what $k_0$ represents in the formula $m \\times (k - k_0) = \\mathcal C$, and why this formula is supposed to capture the notion of “iso-compute”. Also, although there is a high-level explanation about how the offset discounts samples that are too similar, subtracting an offset appears to be an arbitrary choice relative to many other possible options such as dividing by a constant.\n  - Even in the formula $m \\times k = \\mathcal C$, this does not capture the extra compute used to generate the abstractions. \n\n### B. Related Work section\n\nThere is, in my opinion, an important line of past work that has been omitted in the Related Work section, namely, past work on learning and using reasoning abstractions. There’s a significant body of literature on learning and using abstractions (also often called concept learning or library learning) – see, e.g., [1], [2], [3], [4].\n\nDreamCoder, LaSR, LINC, LEMMA,\n\n[1] Ellis, Kevin, et al. \"Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning.\" *PLDI*, 2021.\n\n[2] Li, Zhening, et al. \"LEMMA: Bootstrapping High-Level Mathematical Reasoning with Learned Symbolic Abstractions.\" *MATH-AI Workshop at NeurIPS’22*, 2022. \n\n[3] Grand, Gabriel, et al. \"LILO: Learning Interpretable Libraries by Compressing and Documenting Code.\" *ICRL*, 2024.\n\n[4] Grayeli, Arya, et al. \"Symbolic regression with a learned concept library.\" *NeurIPS*, 2024.\n\n### C. Presentation\n\n1. The main number claimed in the introduction is “44% improvement over… DAPO on AIME 2025”, which should be clarified to mean relative (percentage) improvement instead of absolute improvement. Furthermore, 44% appears inconsistent with the reported numbers in the tables. In Table 2, the RLAD w/ abs (avg) result on AIME 2025 42.45%, and the DaPO w/o abs (avg) result is 37.92%, and 42.45/37.92 = 1.119, which is far from the claimed 1.44. (I did manage to find a ratio that’s close to 1.44, aka the ratio of RLAD w/ abs (best) 48.33% to no RL post-training w/o abs (avg) 33.75% is 1.432, but a) these are not the right numbers to take the ratio of, and b) 1.432 should round to 1.43 and not to 1.44.)\n\n2. The way the paper is presented suggests that “reasoning abstractions” is a novel concept introduced by the paper, which is far from reality (see Weakness B on Related Work). The novel aspect of the paper is, in my opinion, not the idea of “reasoning abstractions” themselves, but the use of RL to post-train a model to generate reasoning abstractions in tandem with an abstraction-conditioned solution generator (last three lines of the Related Work section). Clarification in the paper would be appreciated.\n\n3. High-level presentation would benefit from more explicit summaries at the beginning of each (sub)section. For example, the summary paragraph at the beginning of Section 4 would benefit from a sentence like “In this section, we describe our approach to generating reasoning abstractions via summarization of reasoning traces, and demonstrate how they improve reasoning performance by acting as hints.”\n\n4. The “Modifying reward design” paragraph is not super intuitive and would benefit from further elaboration on how it “address[es] one of the challenges” and which challenge it addresses. Currently, the reward design appears to be mainly supported by the empirical fact that it improved performance (Appendix B2), but motivation from first principles would be nice to see.\n\n5. Typos:\n- Incorrect citations, e.g., DeepScaleR Hard (line 81-82) mistakenly cited e3.\n- Line 116: “(a) … (c) …” -> “(a) … (b) …”\n- Lines 260-261: “recent results RL”\n- Inconsistent capitalization of DAPO as DAPO in the main text and DaPO in the appendices.\n\n6. Appendix C is empty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ACp799hFpp", "reviewer": "Reviewer_w65f", "summary": "The authors propose a two-player RL paradigm where they train a model that proposes \"reasoning abstractions\" based on the question and another model that actually learns to solve the problem with the proposed abstraction. They show that this leads to meaningful improvements against a baseline RL policy trained to simply solve the questions.", "questions": "- How well does the abstraction generator generalize across different datasets? For example, can they be trained on AIME and then transferred to other math datasets? Or does it overfit to that distribution of math problems.\n- Also, how much delta/performance gain are we getting from training both the solver and generator? If the solver does not go through the RL training and the only abstraction generator does, does the performance gain vary significantly?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is written clearly and their contribution in terms of the proposed algorithm and its effectiveness over a the DAPO-based RL baseline is both intuitive and clear.\n- Show some interesting analyses on how to allocate compute between the abstraction generator and the solver and points out that allocating more compute to the generator is often times more helpful than the solver.", "weaknesses": "- It is unclear that current models are not already good enough at generating abstractions based on their solutions. For example, [1] and [2] show that one can prompt LLMs to generate their own abstractions and create a \"cheatsheet\" of abstractions (whether it's cumulative or using a retriever), which also boosts their performance. So, a comparison to a baseline where one can simply prompt the model to generate its own abstraction would be compelling.\n- Some details on the training are unclear: the generator and solver are different models? How many samples were used to run the warm-start of the generator? (how many abstractions per question were generated per problem for the warm start)\n\n[1] Suzgun, Mirac, et al. \"Dynamic cheatsheet: Test-time learning with adaptive memory.\" arXiv preprint arXiv:2504.07952 (2025).\n[2] Didolkar, Aniket, et al. \"Metacognitive reuse: Turning recurring LLM reasoning into concise behaviors.\" arXiv preprint arXiv:2509.13237 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eRQsRu34Z2", "reviewer": "Reviewer_zyoJ", "summary": "This paper proposes RLAD, a way to train LLMs to actually discover and use reasoning abstractions instead of just writing longer chains of thought. The idea is to have one model generate short natural language “abstractions” (like useful hints or lemmas) and another model solve the problem using them. Both are trained together with RL so the abstraction model gets rewarded when its hints help the solver. It basically teaches the model to explore breadth instead of just depth in reasoning. Across math benchmarks like AIME and DeepScaleR, it beats long-CoT RL methods by a large margin and shows that generating abstractions is a better use of compute than brute-force sampling.", "questions": "How sensitive is RLAD to the relative strength of the abstraction generator versus the solution generator? It seems like training stability could depend heavily on their initialization or model size. could the authors clarify if balancing their capabilities was necessary for convergence or performance?\n\nCan the authors provide a deeper analysis of what kinds of abstractions emerge during training, for example, are they procedural (methods), factual (lemmas), or meta-level strategies and whether these abstractions transfer effectively to tasks outside math and ARC-AGI?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel and well motivated idea which is using reinforcement learning to discover and leverage reasoning abstractions, rather than merely extending reasoning depth which is what typical RL does. The approach is technically sound, with clear definitions, careful reward design, and strong empirical validation. The 2 player design is original and The writing is clear and well-structured, making a fairly complex setup easy to follow. Overall, it stands out as an original and solid contribution that meaningfully advances how we think about structured reasoning in large language models.", "weaknesses": "The reward design, while intuitive, could be better analyzed to show why the two-player setup works beyond simple prompt diversity effects. The paper would also benefit from more interpretability analysis of the discovered abstractions how stable they are, whether they transfer across domains, and what kind of procedural knowledge they actually encode. Finally, the training setup remains computationally expensive, which may limit broader adoption without lighter or more scalable variants.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zi9TOUqq3J", "reviewer": "Reviewer_QvJ9", "summary": "A heuristic framework that improves benchmark performance by augmenting LLM prompts with an abstraction prompt composed by another more capable LLM.", "questions": "How would you factor out the performance gains from the abstraction prompts and the stronger model?   \n\nIs there any improvement in performance when both the abstraction and solution models are the same?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "Paper is very well written and clear.  The experimental results of Table 2 look promising.", "weaknesses": "By design, the solution model is weaker than the abstraction model.   \nLines 158-161: \"Concretely, we utilize the Qwen3 series of models to produce solutions and a stronger reasoning model, o4-mini, to generate abstractions. \" \n\nIt is my understanding that for an apples-to-apples comparison, the solution model and the abstraction model should have the same capability. Otherwise, it is impossible to tell how much of the improvement was obtained from the abstractions rather than the  use of stronger model for the abstractions.  Perhaps this is already visible in Figure 2, leftmost panel, where using the weaker model for abstraction provides no benefit. The strength of the proposed approach depends on the results of a fair comparison.    \n\nIf my understanding is incorrect, am supportive of acceptance.  If my understanding is correct, support rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)