RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems

Title: RLAD : Training LLMs to Discover Abstractions for Solving Reasoning Problems
ArXiv: 2510.02263v1
Version: v1
Authors: Yuxiao Qu, Anikait Singh, Yoonho Lee, Amrith Setlur, Carnegie Mellon University, Ruslan Salakhutdinov, Chelsea Finn, Stanford University, Aviral Kumar
Sections: 28
Estimated tokens: 19.4k

## Contents
- 1 Introduction
- 2 Related Work
- 3 Preliminaries and Notation
- 4 Reasoning Abstractions and Why They Are Useful
  - 4.1 Proposing Good Reasoning Abstractions by Summarizing Solution Attempts
  - 4.2 Results and Observations
- 5 RLAD : Learning to Propose Reasoning Abstractions
  - 5.1 Training π θ abs \pi^{\mathrm{abs}}_{\theta} and π θ sol \pi^{\mathrm{sol}}_{\theta} via RL
  - 5.2 Warmstarting π θ abs \pi^{\mathrm{abs}}_{\theta} from a Good Initialization
- 6 Experimental Evaluation
  - 6.1 Main Performance Results on Math Reasoning Benchmarks
  - 6.2 Understanding Properties of RLAD
- 7 Discussion and Perspectives on Future Work
- 8 Acknowledgements
- References
- Appendices
  - Appendix A Experimental Details
    - A.1 Pseudocode for RLAD
    - A.2 Hyperparameters
  - Appendix B Additional Experimental Results
    - B.1 RLAD ’s w/ abs performance on AIME 2024 and HMMT 2025
    - B.2 Design Choice Ablations
    - B.3 Full Results for Abstractions in non-math domains
  - Appendix C Abstractions in Other Domains
  - Appendix D Qualitative Examples of Math Reasoning Abstractions
    - D.1 Prompt for Abstraction Classification
    - D.2 Example for Short Abstraction
    - D.3 Example for Each Abstraction Category

### 1 Introduction

Modern machinery for training large language models (LLMs) to reason relies on incentivizing longer chains of thought via reinforcement learning (RL). This training approach largely incentivizes “depth”: subsequent training iterations increase response length by incorporating new operations that usually attempt to verify or build on top of the line of reasoning being already pursued by the model [33]. This often results in very long chains of thought that appear to explore the solution search space, but degenerate into frequent logic switches and degenerate exploration (also referred to as “underthinking” [43]). One way to avoid this issue altogether is to directly optimize for “breadth”: train language models to explore a diverse array of solution strategies, rather than committing to a seemingly good strategy right away and does not budge from it as more test-time compute is spent [47, 49].

*How can we make models explore a breadth of reasoning strategies for a given problem?* Abstractly, the most natural approach to do so is to train models to hypothesize new strategies to attack difficult problems and then attempt to utilize these strategies in the solution. We can do this by making models capable of discovering *reasoning abstractions*: compressed representations of shared procedures that underlie multiple candidate solutions to a problem. For example, in math reasoning, such abstractions might correspond to useful intermediate lemmas or even some intermediate steps that do not succeed but illustrate what not to do. When presented in context, these abstractions function akin to “hints” on an exam, enabling LLMs to solve harder problems by building on the insights appearing in the abstraction. That is, when conditioned on abstractions, training via RL should train the LLM to implement useful meta strategies that utilize and compose the procedural information in the abstraction as best as possible to solve the problem, rather than attempting to search over the procedural information itself. This naturally boosts the diversity of solution strategies and behaviors that a model learns to utilize when encountering an unseen problem, in contrast to committing to a narrow set of approaches. In RL terminology, abstractions serve as high-level subgoals, skills, or priors—any of them depending upon context—guiding the low-level solution-generating policy.

In this work, we imbue LLMs with the capability of proposing and utilizing abstractions for solving problems. Concretely, we build reasoning models that, first, given an input problem, propose one or more reasoning abstractions. Subsequently, they generate a solution that utilizes the information and principles prescribed by these abstractions. To achieve this, we jointly train two LLMs via RL post-training: (1) an abstraction generator, and (2) an abstraction-conditioned solution generator. The abstraction generator is rewarded for the improvement in the accuracy of the solution generator, stemming from conditioning on the abstractions it proposes. The solution generator is rewarded to maximize accuracy in solving a problem when using the abstraction. To obtain a good initialization for RL training, we warmstart the abstraction generator by running supervised fine-tuning (SFT) on paired problem-abstraction data obtained with the help of stronger models. Specifically, we generate abstractions by summarizing multiple candidate solutions to a problem and prompt an LLM to generate a couple diverse abstractions. Once trained, the abstraction generator does not utilize any guidance from a larger model.

Our main contribution is the notion of reasoning abstractions, how they can be obtained, training procedures to amplify them via RL training, and an illustration of why and how they can be used to improve reasoning performance and exploration of the search space. Concretely, we build an approach to imbue LLMs with the capability of proposing abstractions, and evaluate the model on a variety of math-reasoning benchmarks, AIME 2025 [22], DeepScaleR Hard [33], and AMC 2023. We find an average 44% improvement over state-of-the-art long chain-of-thought RL approaches (i.e., DAPO [47]) on AIME 2025, and show a benefit from generating diverse abstractions over solution sampling.

### 2 Related Work

Scaling test-time compute and exploration. Recent work highlights the promise of scaling test-time compute in different ways. One approach involves parallel sampling: sampling multiple reasoning rollouts and then selecting a winner via a scoring rule [40, 42, 4, 7, 36, 45, 11, 37]. A complementary line of work iteratively edits a single trace, attempting to implement some sort of a sequential search within a single solution trace [20, 27, 28, 13]. As such, the sequential approach performs a bit worse on harder problems [37, 29], where it often gets trapped in strategies that seem optimal but aren’t actually [25]. Yet it still performs better than parallel search on easier and medium difficulty problems [37]. Our approach of proposing and leveraging abstractions enables a kind of a hybrid between sequential sampling and parallel sampling, guided by the proposed abstractions. Some concurrent work [25] studies directly interleaving parallel and sequential samples, and while it is similar in motivation to us, it only distills this interleaved structure into the model and does not run RL training to optimize the parallel and sequential sampling procedures employed here. Prior work has also utilized hand-designed scaffolds to integrate multi-step evaluations of intermediate hypotheses into reasoning [46, 12, 11, 15]. In contrast, we do not rely on pre-defined interfaces but learn to *automatically* propose useful abstractions.

Using prior knowledge for LLM reasoning.
Several threads of work converge on the idea that textual artifacts such as examples, plans, or prompts, can serve as reusable knowledge that steers LLM behavior. Existing retrieval-augmented generation (RAG) pipelines assume a static corpus, typically of human-written text, and focus on improving retrieval heuristics [14, 3, 39, 41, 2, 17]. Many works use LLMs to learn or refine prompts, either in an input-agnostic fashion [54, 44, 26, 8] or through input-specific edits based on feedback [35, 21, 9, 50, 18]. Other related work explores the use of synthetic demonstrations [52], scratchpads [24], and memory-augmented agents [32] to encode prior problem-solving knowledge. Two recent works demonstrate that LLMs can accumulate and reuse their own experience across tasks [53, 38].
While one can view our abstractions as a form of prior procedural and factual knowledge produced before the model’s attempt, this knowledge is (a) input-dependent and (c) is not acquired from an external source at deployment, but rather is “proposed” by the model itself. Imbuing models with this capability requires a cooperative RL training process. To our knowledge, prior work on textual artifacts do not quite train models to be capable of generating these artifacts on their own.

### 3 Preliminaries and Notation

We assume an LLM is provided access to a problem $\mathbf{x}$, and generates a stream of tokens $\mathbf{y}$ that ends in an estimate of the answer. We assume access to a rule-based ground-truth 0/1 reward $\mathrm{Acc}_{\mathbf{x}}(\mathbf{y},\mathbf{y}^{\star})$ $\in$ $\{0,1\}$ that measures correctness of the produced answer $\mathbf{y}$, against the ground-truth solution $\mathbf{y}^{\star}$ for a question $\mathbf{x}$. For training, we are given a dataset $\mathcal{D}_{\textrm{train}}=\{(\mathbf{x}_{i},\mathbf{y}_{i}^{\star})\}_{i=1}^{N}$ of problems $\mathbf{x}_{i}$ and solutions $\mathbf{y}_{i}^{\star}$ that end with the correct answer. Our goal is to train the LLM $\pi(\cdot|\mathbf{x})$ such that it
achieves high rewards on a test distribution of problems $\mathcal{P}_{\mathrm{test}}$. We evaluate models by measuring the average accuracy under $\mathcal{P}_{\mathrm{test}}$. We also measure the pass@k metric, where for problem $\mathbf{x}$, we sample $k$ solutions $\mathbf{y}_{1},\ldots,\mathbf{y}_{k}\sim\pi(\cdot|\mathbf{x})$, and consider the problem to be solved if any of these $k$ traces is correct. This metric couples accuracy with diversity, i.e., it attains the largest value when the model effectively finds diverse, good responses. To reduce variance in estimating pass@k, we sample $n\geq k$ samples per problem and use the unbiased estimator introduced in Codex [5]: $1-\binom{n-c}{k}/\binom{n}{k}$, where $c\leq n$ is the number of correct samples.

### 4 Reasoning Abstractions and Why They Are Useful

Solving reasoning problems often requires composing both *procedural* knowledge (e.g., how to apply a root-finding algorithm) and *factual* knowledge (e.g., relevant lemmas or intermediate results). Current approaches train models to reason via reinforcement learning (RL) on long chains of thought. However, this is often ineffective as RL often tends to optimize for “depth”, producing longer traces where each subsequent segment builds on the last segment (e.g., verifying prior calculations), rather than “breadth”, i.e., exploring diverse solution strategies or utilizing seemingly irrelevant procedures when needed. We now introduce the concept of *reasoning abstractions*, concise insights that explicitly encode a range of useful procedural and factual concepts for a problem.

#### 4.1 Proposing Good Reasoning Abstractions by Summarizing Solution Attempts

To imbue LLMs with the capability of proposing abstractions, we warmstart the LLM using a dataset consisting of problems paired with a small set of high-quality reasoning abstractions, created synthetically. Perhaps the most natural way to obtain an initial set of reasoning abstractions is to collect a diverse set of traces attempting to solve a problem and then summarize useful concepts appearing in these traces (see Figure [1](https://arxiv.org/html/2510.02263v1#S0.F1) for an illustration). More formally, consider the space of possible reasoning traces for a given problem as a graph where the nodes of the graph represent intermediate states encountered when solving a question (see [Figure˜1](https://arxiv.org/html/2510.02263v1#S0.F1)). Good abstractions would identify useful substructures within this larger reasoning graph. For example, an abstraction can capture whether a set of strategies lead to a similar outcome or another set of tactics leads to an error being consistently made.

Generating abstractions. We now quantitatively evaluate whether we can generate useful abstractions by summarizing key insights in reasoning and solution traces that solve a problem. To do so, we prompt a model to generate solution traces and prompt a stronger model to deduce useful patterns from the responses of the first model. Concretely, we utilize the Qwen3 [30] series of models to produce solutions and a stronger reasoning model, o4-mini, to generate abstractions. While this approach is not perfect, it enables us to validate the feasibility of the concept of reasoning abstractions and generate warmstart data for *training* LLMs to propose abstractions. To ensure that the abstractions do not “leak” content of the solution, we verify post-hoc that prompting a model with only the abstraction and no problem yields zero accuracy when sampling 16 times from the base model. This makes these abstractions suitable for our study as they only provide useful information while not allowing the model to shortcut to the answer.

Evaluating abstractions. To validate the efficacy of using abstractions, we adopt a simple test based on performance after conditional generation. Concretely, let us denote the LLM policy that produces a solution conditioned on the problem $\mathbf{x}$ as $\pi_{\theta}^{\mathrm{sol}}(\cdot|\mathbf{x})$. A good abstraction $\mathbf{z}$ is a sequence of tokens that provides some useful procedural and factual information to improve model performance:

$$ $\displaystyle\addcontentsline{lla}{section}{\numberline{\string\crtrefnumber{eq:abstraction_success}}{e}q:abstraction_{s}uccess}\mathbb{E}_{\widetilde{\mathbf{y}}\sim\pi^{\mathrm{sol}}_{\theta}(\cdot|\mathbf{x},\mathbf{z})}\left[\mathrm{Acc}(\widetilde{\mathbf{y}},\mathbf{y}^{*})\right]>\mathbb{E}_{\widetilde{\mathbf{y}}\sim\pi^{\mathrm{sol}}_{\theta}(\cdot|\mathbf{x})}\left[\mathrm{Acc}(\widetilde{\mathbf{y}},\mathbf{y}^{*})\right].$ (1) $$

#### 4.2 Results and Observations

**Table 1: *pass@k accuracy and max@k coverage on the ARC-AGI benchmark.* Abstractions yield consistent gains in both metrics.**
| k | pass@k | coverage (max@k) |  |  |
| --- | --- | --- | --- | --- |
| w/o abs | w/ abs | w/o abs | w/ abs |  |
| 1 | 14.0% | 18.0% | 19.5% | 22.5% |
| 2 | 17.5% | 22.5% | 24.0% | 28.5% |
| 4 | 20.0% | 26.5% | 28.5% | 34.0% |
| 8 | 22.8% | 30.2% | 31.8% | 38.8% |
| 16 | 24.7% | 33.2% | 35.0% | 42.9% |

Evaluation on math reasoning. After generating abstractions, we measure their quality by evaluating Equation [1](https://arxiv.org/html/2510.02263v1#S4.E1), i.e., by checking if conditioning the solution generator on a set of abstractions improves its accuracy. Results in [Figure˜2](https://arxiv.org/html/2510.02263v1#S4.F2) show that conditioning a problem solver on abstractions improves accuracy when two conditions hold simultaneously: (i) the abstraction is not too short (e.g., not just a few words that are not informative; example in Appendix [D.2](https://arxiv.org/html/2510.02263v1#A4.SS2)) and is generated by a strong generator (o4-mini) and (ii) the solution generator has sufficient instruction-following capability (Qwen3-1.7B or Qwen3-4B) of interpreting and utilizing the generated abstraction. These results confirm that good abstractions (satisfying Eq. [1](https://arxiv.org/html/2510.02263v1#S4.E1)) can be obtained for math problems, but neither the ability to generate them nor the ability to leverage them in solutions arises naturally. In [Section˜5](https://arxiv.org/html/2510.02263v1#S5), we will describe our method for explicitly training models to propose and use such abstractions effectively.

Evaluation on ARC-AGI. We also evaluate abstractions on the ARC-AGI benchmark. We present some details of our setup in Appendix [B.3](https://arxiv.org/html/2510.02263v1#A2.SS3). We evaluate on 90 ARC puzzles evenly derived from the test sets of ARC-AGI 1, ARC-AGI 2, and BARC [16]. In Table [1](https://arxiv.org/html/2510.02263v1#S4.T1), we present the pass@k and coverage performance (% of unit tests successfully passed) of the base Qwen3-4B model when conditioned on a proposed abstraction vs not using any abstraction. We see a positive improvement in both metrics on this domain over multiple samples, indicating an improvement from utilizing reasoning abstractions.

Figure: Figure 2: Benefits from abstractions rely crucially on strength of the solver, abstraction length, and solution model. Most configurations fail to yield gains; only o4-mini with long and detailed abstractions shows consistent improvements across the GSM8k and GSMPlus datasets (left). The capability of the problem solver conditioned on abstractions also matters: even strong abstractions help only if the solution model is sufficiently capable (middle, right).
Refer to caption: x2.png

Interpreting the generated abstractions. We also show some examples of the discovered abstractions in Appendix [D.3](https://arxiv.org/html/2510.02263v1#A4.SS3). We now attempt to interpret these abstractions by classifying them into several categories. We observe that the discovered abstractions often correspond to useful techniques (e.g., “launchpoint” in Appendix [D.3](https://arxiv.org/html/2510.02263v1#A4.SS3)), a useful lemma or heuristic principle (e.g., “blind-follow” in Appendix [D.3](https://arxiv.org/html/2510.02263v1#A4.SS3)), and cautionary examples that demonstrate common pitfalls encountered when solving a problem (e.g., “caution alert” in Appendix [D.3](https://arxiv.org/html/2510.02263v1#A4.SS3)). These abstractions distill complex reasoning patterns and potential approaches into useful nuggets, allowing models to generalize across structurally similar problems.
Finally, we would like to remind the reader and emphasize that the above qualitative results from interpreting the discovered abstractions are specific to an individual problem, and not representative of the process being used to discover them. Our approach for generating abstractions is neither hand-engineered for interpretability nor does it use any such heuristics in addition to the summarization procedure discussed above.

Good abstractions exist in many domains.
We also find that the summarization approach can be used to identify a set of useful reasoning abstractions on many domains, including healthcare, human behavior, legal reasoning, and web security. Of course, the proportion of an abstraction devoted to procedural knowledge and factual knowledge is different in these domains compared to math. That said, we find that using reasoning abstractions improves performance by 30% on average over 37 tasks from RAFT [1], CLUES [23], and LegalBench [10]. We show some examples in [Figure˜7](https://arxiv.org/html/2510.02263v1#A3.F7) and full results in Appendix [B.3](https://arxiv.org/html/2510.02263v1#A2.SS3).

### 5 RLAD : Learning to Propose Reasoning Abstractions

Having defined the notion of reasoning abstractions and shown that they can improve performance when adhered to for reasoning, we will now develop an approach to train LLMs to be capable of both proposing and utilizing abstractions. Doing so requires training an *abstraction generator*: an LLM, $\mathbf{z}\sim\pi^{\mathrm{abs}}_{\theta}(\cdot|\mathbf{x})$ that proposes candidate abstractions $\mathbf{z}$ given problem $\mathbf{x}$, and an abstraction-conditioned solution generator, $\mathbf{y}\sim\pi^{\mathrm{sol}}_{\theta}(\cdot|\mathbf{x},\mathbf{z})$, that produces a solution $\mathbf{y}$ given $\mathbf{x}$ and abstraction $\mathbf{z}$. Note that $\mathbf{z}$ is parameterized as a variable-length sequence of tokens and might consist of one or more facts or procedures. While our approach applies to the case when $\pi^{\mathrm{abs}}_{\theta}$ produces more than one abstraction, we abuse notation and subsume multiple abstractions into one to avoid clutter. We now describe RL with abstraction discovery (RLAD), our method for training these models via RL.

#### 5.1 Training π θ abs \pi^{\mathrm{abs}}_{\theta} and π θ sol \pi^{\mathrm{sol}}_{\theta} via RL

Figure: Figure 3: RLAD training paradigm. We train an abstraction generator, $\pi^{\mathrm{abs}}_{\theta}$, that proposes some reasoning abstractions conditioned on the question $\mathbf{x}$, denoted as $\mathbf{z}$. Then, the solution generator, $\pi^{\mathrm{sol}}_{\theta}$, is trained to produce a response, $\tilde{\mathbf{y}}$, conditioned on the generated abstraction $\mathbf{z}$. The reward used for training $\pi^{\mathrm{abs}}_{\theta}$ corresponds to the average success rate of the solution generator conditioned on the proposed abstraction.
Refer to caption: figures/workflow.png

The core principle behind our approach is that an abstraction $\mathbf{z}$ is successful at a given problem $\mathbf{x}$ if it can maximally help $\pi^{\mathrm{sol}}_{\theta}(\cdot|\mathbf{x},\mathbf{z})$ find correct responses to the question $\mathbf{x}$, without actually leaking the answer itself. To ensure this, we explicitly prompt the model not to reveal solution information, and we further use an LLM-based judge to verify that the generated guidance does not contain the answer.
To convert this into an RL objective, we design a reward function that rewards an abstraction $\mathbf{z}$ with the expected success of solutions generated by $\pi^{\mathrm{sol}}_{\theta}$ conditioned on $\mathbf{z}$:

$$ $\displaystyle\addcontentsline{lla}{section}{\numberline{\string\crtrefnumber{eq:reward_absgen_naive}}{e}q:reward_{a}bsgen_{n}aive}r_{\pi^{\mathrm{sol}}_{\theta}}(\mathbf{x},\mathbf{z}):=\mathbb{E}_{\widetilde{\mathbf{y}}\sim\pi^{\mathrm{sol}}_{\theta}(\cdot|\mathbf{x},\mathbf{z})}\left[\mathrm{Acc}_{\mathbf{x}}(\widetilde{\mathbf{y}},\mathbf{y}^{*})\right],$ (2) $$

where $\mathbf{y}^{*}$ is the ground-truth answer and $\mathrm{Acc}_{\mathbf{x}}(\cdot,\cdot)$ denotes the 0/1 accuracy on problem $\mathbf{x}$. To train $\pi^{\mathrm{sol}}_{\theta}$, one can then adopt the fairly straightforward approach of maximizing 0/1 binary outcome reward, now conditioned on a given abstraction $\mathbf{z}$ sampled previously from $\pi^{\mathrm{abs}}_{\theta}$, akin to recent results RL [6].
Formally, we set the reward for a solution as: $r(\mathbf{x},\mathbf{z},\widetilde{\mathbf{y}}):=\mathrm{Acc}_{\mathbf{x}}(\widetilde{\mathbf{y}},\mathbf{y}^{*})$. With these reward functions in place, perhaps the most natural approach then would be to train $\pi^{\mathrm{abs}}_{\theta}$ to maximize $r_{\pi^{\mathrm{sol}}_{\theta}}$ for a fixed $\pi^{\mathrm{sol}}_{\theta}$ on a dataset of prompts $\mathcal{D}_{\pi^{\mathrm{abs}}_{\theta}}$, while also iteratively training $\pi^{\mathrm{sol}}_{\theta}$ to maximize the reward function $r$ on modified prompts generated by concatenating a set of sampled abstraction $\mathbf{z}$ on a dataset of problems, $\mathcal{D}_{\pi^{\mathrm{sol}}_{\theta}}$. This maximization could be done via on-policy RL (e.g., GRPO [34]) or offline RL methods (e.g., DPO [31], STaR [51]). This represents a cooperative two-player game.

Challenges with naïve reward design. While the approach so far is extremely simple, it presents some challenges. In particular, the reward functions defined above can result in undesirable solutions in a rather nuanced manner: (1) if $\pi^{\mathrm{abs}}_{\theta}$ learns to solve problem $\mathbf{x}$ in itself, it will still be rewarded highly by $r_{\pi^{\mathrm{sol}}_{\theta}}$ but is not a desirable abstraction; (2) if $\pi^{\mathrm{sol}}_{\theta}$ is too weak or too strong, such that it is either able to always solve the problem $\mathbf{x}$ or never solves it, then $r_{\pi^{\mathrm{sol}}_{\theta}}$ will not provide a meaningful signal to update $\pi^{\mathrm{abs}}_{\theta}$; and (3) similar to the above failure modes, training $\pi^{\mathrm{sol}}_{\theta}$ via on-policy RL may result in it ignoring the abstraction $\mathbf{z}$ altogether no matter how useful it is. Abstractly, all of these challenges stem from an asymmetry in the strength of $\pi^{\mathrm{abs}}_{\theta}$ and $\pi^{\mathrm{sol}}_{\theta}$, where one may drown out the learning signal for the other. We therefore build a modified reward system for training.

Modifying reward design. To address these potential failure modes, we make a small but consequential change to the training reward system inspired by Kumar et al. [13]. In particular, we train $\pi^{\mathrm{sol}}_{\theta}$ on a mixture of prompts $\mathbf{x}$ augmented by abstractions $\mathbf{z}$ and prompts $\mathbf{x}$ without any abstractions at all. In this process, while we utilize $\mathrm{Acc}_{\mathbf{x}}$ as discussed above on a given response, we simply zero out rewards for any trace generated on $\mathbf{x}$ without abstractions. When utilizing KL-constrained RL fine-tuning, $\pi^{\mathrm{sol}}_{\theta}$ is now trained to closely mimic the distribution of responses as the reference LLM on questions $\mathbf{x}$ but must attempt to find ways to optimize reward on the same question $\mathbf{x}$ when augmented with an abstraction. This can be accomplished only when $\pi^{\mathrm{sol}}_{\theta}$ learns to utilize the provided abstraction carefully, hence addressing one of the challenges above.
Formally, the updated versions of these reward functions are:

$$ $\displaystyle\addcontentsline{lla}{section}{\numberline{\string\crtrefnumber{eq:reward_func_real}}{e}q:reward_{f}unc_{r}eal}r(\mathbf{x},\mathbf{z},\widetilde{\mathbf{y}})$ $\displaystyle:=\begin{cases}0,&\text{if }\mathbf{z}=\emptyset\\ \mathrm{Acc}_{\mathbf{x}}(\widetilde{\mathbf{y}},\mathbf{y}^{*}),&\text{otherwise}\\ \end{cases}$ (3) $\displaystyle r_{\pi^{\mathrm{sol}}_{\theta}}(\mathbf{x},\mathbf{z})$ $\displaystyle:=\mathbb{E}_{\widetilde{\mathbf{y}}\sim\pi^{\mathrm{sol}}_{\theta}(\cdot|\mathbf{x},\mathbf{z})}[\mathrm{Acc}_{\mathbf{x}}(\widetilde{\mathbf{y}},\mathbf{y}^{*})].\vskip-5.69046pt$ (4) $$

#### 5.2 Warmstarting π θ abs \pi^{\mathrm{abs}}_{\theta} from a Good Initialization

While the above approach prescribes a recipe for RL training of $\pi^{\mathrm{abs}}_{\theta}$ and $\pi^{\mathrm{sol}}_{\theta}$, any such recipe critically relies on the ability of the initialized model to generate somewhat meaningful abstractions and meaningful solutions conditioned on the abstraction input, respectively, right from the beginning of RL training. Inspired by the approach of running an initial phase of SFT to imbue into the model the basic structure of a long chain-of-thought before RL [6, 29], we run an initial phase of SFT to imbue into $\pi^{\mathrm{abs}}_{\theta}$ the basic capabilities of producing abstractions. We did not perform warmstart training for $\pi^{\mathrm{sol}}_{\theta}$ as we find that the base Qwen3 models we were utilizing were capable of utilizing abstractions well. For this initial warmstart phase, we follow the protocol from Section [4](https://arxiv.org/html/2510.02263v1#S4) and construct a corpus $\{(\mathbf{x}_{i},\mathbf{z}_{i},\mathbf{y}_{i})\}_{i=1}^{M}$ by prompting strong models. For each training problem-solution pair $(\mathbf{x},\mathbf{y}^{*})$ in our training set, we first generate an abstraction $\mathbf{z}$ via summarization, discarding any that leak $\mathbf{y}^{*}$.

Figure: Figure 4: A typical example of a reasoning abstraction proposed by our abstraction generator. In the solution, we see (in blue) references to the abstraction (“cheatsheet”) and keywords from the abstraction being used meaningfully in the reasoning trace of the solution generator model.
Refer to caption: x3.png

Practical approach and algorithm details. For warmstarting the abstraction generator, we utilize abstractions generated by o4-mini. We then use a weaker model (GPT 4.1-mini) to check the efficacy of each abstraction when conditioned on by comparing the success rate of the solution generator with and without an abstraction. We filter abstractions that don’t result in an increase in solution generation performance to form our seed set of abstractions. Then, we run supervised fine-tuning (SFT) for $5$ epochs on the seed dataset to obtain an initial abstraction generator. For solution generation, we directly start from Qwen3-1.7B [30], a 1.7B reasoning model distilled from Qwen3-32B.

After SFT, we employ RLAD to fine-tune the abstraction generator and abstraction-conditioned solution generator via RL. For the abstraction generator, we opt to use “batched” offline RL via RFT [48], since reward computation by rolling out the solution generator the on the fly and running online RL was infeasible using our RL infrastructure and within the compute we had access to. To train the solution generator, we build on the RL recipe from DAPO [47], and include token-level policy loss normalization and asymmetric clipping. Building upon implementation of Setlur et al. [33], we employ a two stage curriculum where we partition the DeepScaleR [19] mixture by success rate of the base model into three sets: (1) easy, (2) medium, and (3) hard,
where we fine-tune first on easy problems with an 8K token budget and then on medium problems. We utilize the hard split as a held-out, evaluation subset, which we denote as “DeepScaleR [Hard]”. Note that these splits are different from the e3 [33] work, while the principle is inspired by that work. We outline hyperparameters and details in Appendix [A.1](https://arxiv.org/html/2510.02263v1#A1.SS1) and provide a pseudocode in Algorithm [1](https://arxiv.org/html/2510.02263v1#alg1).

### 6 Experimental Evaluation

The goal of our experiments is to evaluate the efficacy of RLAD in improving the reasoning capabilities of LLMs through abstraction-guided solution generation. Specifically, we aim to answer the following questions: (1) Does RLAD improve pass@1 accuracy across several reasoning benchmarks compared to direct solution generation?, (2) How does RLAD scale as more abstractions and solutions are generated?, and (3) What makes the generated abstractions useful, how faithfully are they followed, and how do they guide and improve solution generation? We compare RLAD with strong models on three math reasoning benchmarks: AMC 2023, AIME 2025, and DeepScaleR Hard [19], which itself is a subset of hard problems from the OmniMATH mixture on which DeepSeek-R1 distilled Qwen-32B model attains an accuracy of $\leq 10\%$. We also fine-tune an abstraction generator for the ARC-AGI program synthesis tasks, and conduct a similar comparison on 90 ARC puzzles evenly derived from the test sets of ARC-AGI 1, ARC-AGI 2, and BARC [16]. Finally, we also perform several ablations to better understand abstractions produced by RLAD and discuss them later.

#### 6.1 Main Performance Results on Math Reasoning Benchmarks

We evaluate RLAD under three settings: (1) w/o abs, without abstractions; (2) w/ abs (avg), average performance over generations conditioned on 4 abstractions per problem; and (3) w/ abs (best): using the best-performing abstraction (in a set of 4 proposed abstractions per problem).
We observe that RLAD outperforms the base model and variant fine-tuned with RL on the same prompts via DAPO [47] without abstractions, across all settings and benchmarks ([Table˜2](https://arxiv.org/html/2510.02263v1#S6.T2)). This highlights that RLAD can propose and leverage abstractions to improve reasoning performance. Interestingly, these performance gains are not limited to abstraction-conditioned inference: even in the w/o abs setting, where no abstraction is provided during inference, RLAD improves over the prior methods, when trained with abstractions via RLAD. This suggests that exposure to diverse abstractions during training enhances the model’s general capability on these training problems. We observe similar trends on additional benchmarks, including AIME 2024 and HMMT 2025 (see Appendix [B.1](https://arxiv.org/html/2510.02263v1#A2.SS1)), where RLAD improves in the w/o abs setting.
Finally, in Appendix [D](https://arxiv.org/html/2510.02263v1#A4), we also measure the performance of RLAD when different token budgets are allowed for reasoning – while Table [2](https://arxiv.org/html/2510.02263v1#S6.T2) measures performance at a budget of 32K tokens, we also measure performance at 8K and 16K budgets and find RLAD to be more effective than other approaches.

**Table 2: Accuracy on math reasoning benchmarks. We measure test performance on three benchmarks (AIME 2025, DeepScaleR Hard, and AMC 2023) with the base Qwen 3-1.7B model, DAPO, and RLAD. Performance is measured under three settings: (1) w/o abs, without abstractions; (2) w/ abs (avg), average accuracy over solutions generated when conditioned on four proposed abstractions per problem; and (3) w/ abs (best), using the best-performing abstraction among the four generated in (2). RLAD achieves consistent gains across both abstraction-conditioned and no-abstraction settings.**
| Approach | AIME 2025 | DeepScaleR [Hard] | AMC 2023 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| w/o abs (avg) | w/ abs (avg) | w/ abs (best) | w/o abs (avg) | w/ abs (avg) | w/ abs (best) | w/o abs (avg) | w/ abs (avg) | w/ abs (best) |  |
| Qwen-3-1.7B | 33.75 | 36.25 | 40.00 | 20.21 | 22.14 | 32.50 | 86.41 | 78.01 | 84.53 |
| + DAPO | 37.92 | 34.90 | 39.79 | 21.67 | 21.88 | 33.54 | 86.41 | 81.99 | 88.44 |
| + RLAD (Ours) | 38.04 | 42.45 | 48.33 | 23.54 | 24.84 | 35.54 | 87.25 | 88.35 | 91.72 |

#### 6.2 Understanding Properties of RLAD

In this section, we conduct a number of additional experiments to understand the behavior of RLAD in regards to the usefulness of the proposed abstractions and the performance of the RLAD algorithm when provided with a large budget on the total inference compute.

1) “Weak-to-strong” generalization of the abstraction generator.
We next evaluate the weak-to-strong generalization of our method by pairing our trained abstraction generator with o4-mini as the solution generator. We use a fixed 24k generation token budget and 4 samples per question. Without abstractions, o4-mini achieves 80.38% pass@1, 82.26% pass@2, and 84.77% pass@4. Conditioning on both the problem and the proposed abstractions improves performance to 85.83% pass@1, 88.33% pass@2, and 90.00% pass@4 accuracy. Thus, conditioning on abstractions consistently yields higher pass@k accuracy compared to question-only conditioning, even for this strong reasoning model. These gains demonstrate that abstractions, though produced by a comparatively weaker model, can transfer effectively to a stronger solution generator, providing evidence that abstractions generalize in a weak-to-strong setting and enables downstream improvements without additional supervision or modifications to the strong model.

2) Compute tradeoffs b/w abstraction and solution generation.
Next, we study how to allocate compute between generating more abstractions and sampling solutions to attain maximal performance within a given inference budget. This corresponds to a “compute-optimal strategy” [37] for partitioning compute between abstraction and solution generation. If the model typically fails by making small local errors in its computation, then additional concise abstractions may not help it as much as simply trying again to generate a locally similar solution (optimizing “depth“). In contrast, if the model tends to pursue a seemingly plausible but incorrect approach and is unable to easily recover or switch approaches, then conditioning on diverse abstractions can help by offering alternative high-level approaches toward the correct answer. In other words, when the model has a tendency to explore “depth” over “breadth” of solution strategies, abstractions can help improve performance. With this intuition, we hypothesize that when the compute budget permits only a limited number of samples, allocating more compute to sampling multiple solutions but retaining only a few abstractions will improve success more. However, once pass@k for a single abstraction begins to saturate, performance gains are more likely to come from scaling the diversity of abstractions, which enables the model to explore different strategies.

Figure: Figure 5: *Tradeoff of abstraction and solution generation on AIME 2025.* As the total inference compute budget increases (color scheme on the right), we find better performance efficiency when allocating our budget to abstraction generation rather than solution generation, for all values of normalization offset $k_{0}$ given to us.
Refer to caption: figures/abstraction_scaling_joint.png

To validate this hypothesis, we plot *iso-compute* scaling curves under a fixed compute budget $\mathcal{C}$, we distribute between abstractions and solution generation. Specifically, we denote the number of abstractions as $m$ and the number of solutions sampled per abstraction as $k$, such that $m\times k=\mathcal{C}$.
To better isolate the utility of abstractions, we make several more projections of the iso-compute curve, each with a different “normalization offset” $k_{0}$. This normalization offset $k_{0}$ *discounts* performance gains that stem from trying the problem again and making local modifications (e.g., small edits that do not require new strategy changes), but measures the performance gains that stem from major changes in content of a response. Concretely, for a non-zero $k_{0}$, we plot *iso-compute* frontiers when $m\times(k-k_{0})=\mathcal{C}$. This formulation captures the amount of compute that is spent on “meaningful” samples that go beyond the model’s local neighborhood. Figure [5](https://arxiv.org/html/2510.02263v1#S6.F5) shows these *iso-compute* frontiers for different values of the compute budget $\mathcal{C}$. The x-axis plots the ratio between abstractions and adjusted solutions, $m/(k-k_{0})$. Each curve corresponds to a fixed total compute budget.
In Figure [5](https://arxiv.org/html/2510.02263v1#S6.F5), across $k_{0}\in\{0,2,4,6,8\}$, shifting compute toward abstractions consistently yields greater improvements than allocating the same additional compute to solution refinements, especially as the total compute budget increases. *This supports the conclusion that once local errors in the chain-of-thought have been addressed, it is more effective to increase the diversity of strategies used via abstraction conditioning rather than to continuing to scale up CoTs alone.*

**Table 3: Pass@k comparison under equal compute. For each $n$, we compare $n^{2}$ solution samples (no abstractions) with $n$ abstractions $\times$ $n$ solutions per abstraction. Abstraction conditioning yields consistent improvements.**
|  | 1 | 4 | 16 | 64 | 256 |
| --- | --- | --- | --- | --- | --- |
| DAPO, w/o abs (pass@$n^{2}$) | 0.37 | 0.51 | 0.65 | 0.77 | 0.82 |
| RLAD, w/ abs (pass@$n\times n$) | 0.41 | 0.59 | 0.71 | 0.80 | 0.87 |

To obtain more evidence to support this result, we also evaluate pass@k of DAPO and RLAD for a fixed compute budget for both abstraction and non-abstraction generation on AIME 2025. Here, for a given value $n$, we compare two approaches: (1) sampling $n^{2}$ solutions without abstractions, and (2) sampling $n$ abstractions and then $n$ solutions per abstraction. In Table [3](https://arxiv.org/html/2510.02263v1#S6.T3), abstraction conditioning consistently outperforms pure solution sampling. For example, at $n=16$, abstraction conditioning achieves a pass@k of $0.71$ versus $0.65$ without abstractions, and at $n=256$ the gap widens to $0.87$ versus $0.82$. This corroborates the efficacy of abstractions, even under matched compute budgets.

Figure: Figure 6: Abstraction-conditioned solution generation analysis. RLAD produces solutions with (left) greater semantic diversity across different abstractions and (right) higher abstraction adherence than baselines.
Refer to caption: figures/combined_figure_unified_colors.png

*3) Understanding the behavior of the abstraction-conditioned solution generator.* A desirable property of the solution generator is the ability to follow the proposed abstractions. To study this, we prompt (o4-mini) to classify whether a particular solution trace produced by a trained solution generator closely adheres to a given abstraction. We ask for a binary decision on each pair of abstraction and solution, and measure the adherence rate across 200 such pairs. In Figure [6](https://arxiv.org/html/2510.02263v1#S6.F6) (right), we report the adherence rates under four conditions: (1) Abstraction, where we measure adherence rates between an abstraction and a solution generated by conditioning on this abstraction itself; (2) No abstraction, where we measure the adherence rates between an abstraction and a solution generated by conditioning on the problem without any abstraction; (3) Retrieval, where we measure adherence rates between an abstraction and a semantically similar prior solution to the problem; and (4) Unrelated abstraction, where we measure the adherence rates between an abstraction and a solution generated via a different abstraction.
We find that the Abstraction condition achieves the highest adherence rate. Intuitively, this means that the trained solution generator is detected to be more likely to follow the strategy or guidance of the given abstraction it is conditioned on.

Additionally, we measure the semantic similarity of two solutions generated without abstraction conditioning, two solutions generated by conditioning on the same abstraction, and two solutions generated by conditioning on different abstractions in Figure [6](https://arxiv.org/html/2510.02263v1#S6.F6) (left). We compute semantic similarity by encoding solutions into vector representations using Qwen3-4B and measuring the cosine similarity between normalized embeddings. We find that the semantic similarity between two solutions generated by conditioning on the same abstraction is highest, which is perhaps expected. Two solutions generated without conditioning on any abstraction are also quite similar semantically, whereas two solutions generated by conditioning on different abstractions are visibly less similar than the other two cases, implying that conditioning on and adhering to abstractions improves diversity of reasoning traces generated.

### 7 Discussion and Perspectives on Future Work

We introduce reasoning abstractions: concise representations of procedural and factual knowledge that are expressed in natural language, as a means to broaden the reasoning strategies used by LLMs. Our method, RLAD, instantiates a two-player training framework that trains an abstraction generator and an abstraction-conditioned solution generator. RLAD yields consistent improvements across several mathematical reasoning benchmarks, outperforming existing methods for training LLMs to reason. Moreover, we show that allocating compute toward generating diverse abstractions, rather than increasing solution sampling alone, leads to greater performance gains. This highlights abstractions as a complementary axis for scaling test-time compute: while longer length chains-of-thoughts and parallel solution sampling provide some existing ways to scale compute, using abstractions provides us with an orthogonal axis to improve performance. This is particularly relevant as we start to see limited gains from standard scaling of length of chains of thought in reasoning. While we showed abstractions can be helpful, we limited our evaluation to mathematical reasoning tasks, leaving open‐ended reasoning unexplored.

Future work. It is important to study RLAD can be extended to train a single model that can both propose and utilize abstractions. Perhaps this can be done by large-scale mid-training that showcases the approach of proposing high-level strategies followed by RL training akin to DeepSeek-R1 [6]. We tried training a single model to do both abstraction generation and solution generation, after a lightweight SFT on traces showing questions paired with abstractions and corresponding solutions, but we found this approach to very quickly lose the ability of proposing abstractions over the course of RL training. We believe that this can be addressed via targeted mid-training or active interventions during RL, but this is an open direction that future work should study.
We also think that attempting to understand the phenomenon of how training with abstractions improves the performance without abstractions (Table [2](https://arxiv.org/html/2510.02263v1#S6.T2)) is also intriguing. Likely this is a result of some form of generalization, but understanding the mechanisms behind this phenomena and how to amplify them is also interesting for future work.

### 8 Acknowledgements

We would like to thank Ian Wu, Matthew Yang, Rafael Rafailov, Yuejiang Liu, Violet Xiang, Joy He-Yueya, and others in the CMU AIRe lab and Stanford IRIS Lab for discussions and feedback. We thank Schmidt Sciences and the DSAI cluster, the FLAME cluster at CMU, and Delta for providing compute support for some of the critical experiments in this paper. We also thank Foundry for providing compute. AS gratefully acknowledges the support of the NSF Graduate Research Fellowship Program and Toyota Research Institute (TRI) for funding, compute, and API credits. CF was supported by Schmidt Sciences. This work was supported by the Schmidt Sciences AI2050 program, an Amazon gift, and the Office of Naval Research under N00014-24-12206.

### References

- Alex et al. [2021]
Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, et al.
Raft: A real-world few-shot text classification benchmark.
*arXiv preprint arXiv:2109.14076*, 2021.
- Anonymous [2025]
Anonymous.
Optimizing inference-time reasoning in LLMs via retrieval-augmented reflection, 2025.
URL [https://openreview.net/forum?id=ElYRG3pJcv](https://openreview.net/forum?id=ElYRG3pJcv).
- Borgeaud et al. [2022]
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al.
Improving language models by retrieving from trillions of tokens.
In *International conference on machine learning*, pages 2206–2240. PMLR, 2022.
- Charniak and Johnson [2005]
Eugene Charniak and Mark Johnson.
Coarse-to-fine n-best parsing and maxent discriminative reranking.
In *Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics*, ACL ’05, page 173–180, USA, 2005. Association for Computational Linguistics.
[10.3115/1219840.1219862](https:/doi.org/10.3115/1219840.1219862).
URL [https://doi.org/10.3115/1219840.1219862](https://doi.org/10.3115/1219840.1219862).
- Chen et al. [2021]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.
Evaluating large language models trained on code, 2021.
URL [https://arxiv.org/abs/2107.03374](https://arxiv.org/abs/2107.03374).
- DeepSeek-AI et al. [2025]
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong
Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu,
Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Feng et al. [2024]
Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang.
Alphazero-like tree-search can guide large language model decoding and training, 2024.
URL [https://arxiv.org/abs/2309.17179](https://arxiv.org/abs/2309.17179).
- Fernando et al. [2023]
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel.
Promptbreeder: Self-referential self-improvement via prompt evolution.
*arXiv preprint arXiv:2309.16797*, 2023.
- Gou et al. [2023]
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen.
Critic: Large language models can self-correct with tool-interactive critiquing.
*arXiv preprint arXiv:2305.11738*, 2023.
- Guha et al. [2023]
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al.
Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.
*Advances in Neural Information Processing Systems*, 36:44123–44279, 2023.
- Hao et al. [2023]
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu.
Reasoning with language model is planning with world model, 2023.
URL [https://arxiv.org/abs/2305.14992](https://arxiv.org/abs/2305.14992).
- Ho et al. [2023]
Namgyu Ho, Laura Schmid, and Se-Young Yun.
Large language models are reasoning teachers, 2023.
URL [https://arxiv.org/abs/2212.10071](https://arxiv.org/abs/2212.10071).
- Kumar et al. [2024]
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust.
Training language models to self-correct via reinforcement learning, 2024.
URL [https://arxiv.org/abs/2409.12917](https://arxiv.org/abs/2409.12917).
- Lewis et al. [2020]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
*Advances in neural information processing systems*, 33:9459–9474, 2020.
- Li et al. [2023]
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.
Camel: Communicative agents for "mind" exploration of large language model society, 2023.
URL [https://arxiv.org/abs/2303.17760](https://arxiv.org/abs/2303.17760).
- Li et al. [2024]
Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M. Dunn, Hao Tang, Michelangelo Naim, Dat Nguyen, Wei-Long Zheng, Zenna Tavares, Yewen Pu, and Kevin Ellis.
Combining induction and transduction for abstract reasoning, 2024.
URL [https://arxiv.org/abs/2411.02272](https://arxiv.org/abs/2411.02272).
- Li et al. [2025]
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou.
Search-o1: Agentic search-enhanced large reasoning models, 2025.
URL [https://arxiv.org/abs/2501.05366](https://arxiv.org/abs/2501.05366).
- Lin et al. [2025]
Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, and Joseph E Gonzalez.
Sleep-time compute: Beyond inference scaling at test-time.
*arXiv preprint arXiv:2504.13171*, 2025.
- Luo et al. [2025]
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica.
Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025.
Notion Blog.
- Madaan et al. [2023]
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.
Self-Refine: Iterative Refinement with Self-Feedback.
*arXiv e-prints*, art. arXiv:2303.17651, March 2023.
[10.48550/arXiv.2303.17651](https:/doi.org/10.48550/arXiv.2303.17651).
- Madaan et al. [2023]
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.
Self-refine: Iterative refinement with self-feedback.
*Advances in Neural Information Processing Systems*, 36:46534–46594, 2023.
- Mathematical Association of America [2025]
Mathematical Association of America.
2025 American Invitational Mathematics Examination (AIME) I: Problems and Solutions, February 2025.
URL [https://artofproblemsolving.com/wiki/index.php/2025_AIME_I](https://artofproblemsolving.com/wiki/index.php/2025_AIME_I).
Art of Problem Solving Wiki entry.
- Menon et al. [2022]
Rakesh R. Menon, Sayan Ghosh, and Shashank Srivastava.
Clues a benchmark for learning classifiers using natural language explanations.
2022.
- Nye et al. [2021]
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena.
Show your work: Scratchpads for intermediate computation with language models, 2021.
URL [https://arxiv.org/abs/2112.00114](https://arxiv.org/abs/2112.00114).
- Pan et al. [2025]
Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr.
Learning adaptive parallel reasoning with language models.
*arXiv preprint arXiv:2504.15466*, 2025.
- Pryzant et al. [2023]
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng.
Automatic prompt optimization with" gradient descent" and beam search.
*arXiv preprint arXiv:2305.03495*, 2023.
- Qu et al. [2024]
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar.
Recursive Introspection: Teaching Language Model Agents How to Self-Improve.
*arXiv e-prints*, art. arXiv:2407.18219, July 2024.
[10.48550/arXiv.2407.18219](https:/doi.org/10.48550/arXiv.2407.18219).
- Qu et al. [2024]
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar.
Recursive introspection: Teaching language model agents how to self-improve.
*arXiv preprint arXiv:2407.18219*, 2024.
- Qu et al. [2025]
Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar.
Optimizing test-time compute via meta reinforcement fine-tuning.
*arXiv preprint arXiv:2503.07572*, 2025.
- Qwen Team [2025]
Qwen Team.
Qwen3 technical report.
Technical report, Qwen, May 2025.
URL [https://huggingface.co/Qwen](https://huggingface.co/Qwen).
Available at: https://huggingface.co/Qwen, https://modelscope.cn/organization/qwen, https://github.com/QwenLM/Qwen3.
- Rafailov et al. [2023]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in Neural Information Processing Systems*, 36:53728–53741, 2023.
- Schäfer et al. [2020]
Marlin B. Schäfer, Frank Ohme, and Alexander H. Nitz.
Detection of gravitational-wave signals from binary neutron star mergers using machine learning.
*Physical Review D*, 102(6), September 2020.
ISSN 2470-0029.
[10.1103/physrevd.102.063015](https:/doi.org/10.1103/physrevd.102.063015).
URL [http://dx.doi.org/10.1103/PhysRevD.102.063015](http://dx.doi.org/10.1103/PhysRevD.102.063015).
- Setlur et al. [2025]
Amrith Setlur, Matthew Y. R. Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar.
e3: Learning to explore enables extrapolation of test-time compute for llms.
2025.
URL [https://arxiv.org/abs/2506.09026](https://arxiv.org/abs/2506.09026).
- Shao et al. [2024]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Shinn et al. [2023]
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
Reflexion: Language agents with verbal reinforcement learning.
*Advances in Neural Information Processing Systems*, 36:8634–8652, 2023.
- Snell et al. [2024]
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.
*arXiv e-prints*, art. arXiv:2408.03314, August 2024.
[10.48550/arXiv.2408.03314](https:/doi.org/10.48550/arXiv.2408.03314).
- Snell et al. [2024]
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
Scaling llm test-time compute optimally can be more effective than scaling model parameters.
*arXiv preprint arXiv:2408.03314*, 2024.
- Suzgun et al. [2025]
Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou.
Dynamic cheatsheet: Test-time learning with adaptive memory.
*arXiv preprint arXiv:2504.07952*, 2025.
- Trivedi et al. [2022]
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal.
Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.
*arXiv preprint arXiv:2212.10509*, 2022.
- Uesato et al. [2022]
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.
Solving math word problems with process- and outcome-based feedback, 2022.
URL [https://arxiv.org/abs/2211.14275](https://arxiv.org/abs/2211.14275).
- Verma et al. [2024]
Prakhar Verma, Sukruta Prakash Midigeshi, Gaurav Sinha, Arno Solin, Nagarajan Natarajan, and Amit Sharma.
Plan$\times$RAG: Planning-guided Retrieval Augmented Generation.
*arXiv e-prints*, art. arXiv:2410.20753, October 2024.
[10.48550/arXiv.2410.20753](https:/doi.org/10.48550/arXiv.2410.20753).
- Wang et al. [2023]
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.
Self-consistency improves chain of thought reasoning in language models, 2023.
URL [https://arxiv.org/abs/2203.11171](https://arxiv.org/abs/2203.11171).
- Wang et al. [2025]
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al.
Thoughts are all over the place: On the underthinking of o1-like llms.
*arXiv preprint arXiv:2501.18585*, 2025.
- Yang et al. [2023]
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen.
Large language models as optimizers.
*arXiv preprint arXiv:2309.03409*, 2023.
- Yao et al. [2023a]
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.
Tree of thoughts: Deliberate problem solving with large language models, 2023a.
URL [https://arxiv.org/abs/2305.10601](https://arxiv.org/abs/2305.10601).
- Yao et al. [2023b]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models, 2023b.
URL [https://arxiv.org/abs/2210.03629](https://arxiv.org/abs/2210.03629).
- Yu et al. [2025]
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al.
Dapo: An open-source llm reinforcement learning system at scale.
*arXiv preprint arXiv:2503.14476*, 2025.
- Yuan et al. [2023]
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou.
Scaling relationship on learning mathematical reasoning with large language models.
*arXiv preprint arXiv:2308.01825*, 2023.
- Yue et al. [2025]
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang.
Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?
*arXiv preprint arXiv:2504.13837*, 2025.
- Yuksekgonul et al. [2025]
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou.
Optimizing generative ai by backpropagating language model feedback.
*Nature*, 639(8055):609–616, 2025.
- Zelikman et al. [2022a]
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman.
Star: Bootstrapping reasoning with reasoning.
*Advances in Neural Information Processing Systems*, 35:15476–15488, 2022a.
- Zelikman et al. [2022b]
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman.
Star: Bootstrapping reasoning with reasoning, 2022b.
URL [https://arxiv.org/abs/2203.14465](https://arxiv.org/abs/2203.14465).
- Zhao et al. [2024]
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang.
Expel: Llm agents are experiential learners.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pages 19632–19642, 2024.
- Zhou et al. [2022]
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.
Large language models are human-level prompt engineers.
In *The Eleventh International Conference on Learning Representations*, 2022.

## Appendices

### Appendix A Experimental Details

#### A.1 Pseudocode for RLAD

Figure: Algorithm 1 Joint RL Training of $\pi^{\mathrm{abs}}_{\theta}$ and $\pi^{\mathrm{sol}}_{\theta}$

#### A.2 Hyperparameters

**Table 4: Key training hyperparameters used in RLAD.**
| Hyperparameter | Value |
| --- | --- |
| algorithm | DaPO [47] |
| training steps | 100 |
| epochs | 10 |
| train batch size | 128 |
| max prompt length | 3072 |
| max response length | 16384 |
| max extrapolation length | 32768 |
| learning rate | 1e-6 |
| clip ratio (low / high) | 0.2 / 0.5 |
| entropy coefficient | 0.001 |
| KL loss coefficient | 0.001 |
| KL loss type | low variance kl |
| sampling temperature (train / val) | 0.6 / 0.6 |
| samples per prompt (train / val) | 16 / 8 |
| max batched tokens | 32768 |

### Appendix B Additional Experimental Results

#### B.1 RLAD ’s w/ abs performance on AIME 2024 and HMMT 2025

In this section, we evaluate the performance of the base model (Qwen-3-1.7B), GRPO-enhanced model, and our proposed method RLAD on two math reasoning benchmarks: AIME 2024 and HMMT 2025. As shown in Table [5](https://arxiv.org/html/2510.02263v1#A2.T5), our method achieves the best performance across both datasets.

It is important to note that RLAD is trained using access to abstractions, yet it also generalizes better even when evaluated without abstraction. This suggests that RLAD does not merely overfit to the abstraction format but instead learns to effectively leverage high-level procedural guidance, leading to better generalization on challenging reasoning benchmarks.

**Table 5: RLAD’s w/ abs performance on AIME 2024 and HMMT 2025. RLAD outperforms both DaPO and the base Qwen model on the AIME 2024 and HMMT 2025 benchmarks.**
| Approach | AIME 2024 | HMMT 2025 |
| --- | --- | --- |
| Qwen-3-1.7B | 48.54 | 22.50 |
| + DaPO | 44.17 | 23.13 |
| + RLAD | 51.46 | 23.75 |

#### B.2 Design Choice Ablations

In this section, we conduct ablation studies to isolate the contributions of key components in RLAD. We investigate three primary design choices, with results summarized in Table [6](https://arxiv.org/html/2510.02263v1#A2.T6): (a) utilizing a curriculum training strategy, (b) including prompts not annotated with an abstraction, and (c) applying reward masking to those non-annotated prompts.

Curriculum training refers to a staged process where the model first learns from simpler problems and gradually transitions to harder ones. We use the protocol from Setlur et al. [33] as inspiration, who demonstrated its effectiveness for direct math problem-solving. In our setting, which incorporates abstractions, curriculum training also proves beneficial, improving both average and best-case performance from 0.38 and 0.43 to 0.41 and 0.48, respectively, compared to non-curriculum training.

Next, we analyze the practice of including prompts without abstractions and applying reward masking. Including a small fraction of these "no-abstraction" prompts is intended to better condition the solution-generator on the abstractions when they are present. However, this risks the model learning a shortcut by simply ignoring the abstractions. To mitigate this, we apply reward masking: for completions on no-abstraction prompts, we nullify the policy reward by zeroing out the advantage, while retaining the KL penalty for regularization. This prevents the model from over-optimizing on examples that lack abstractions, a behavior that would otherwise hinder generalization.

Our findings confirm the efficacy of this combined approach. As shown in Table [6](https://arxiv.org/html/2510.02263v1#A2.T6), including no-abstraction prompts with reward masking is critical for performance. Ultimately, the combination of all three design choices—curriculum training, the inclusion of no-abstraction prompts, and reward masking—significantly outperforms alternative configurations.

**Table 6: Design Choices in RLAD. We isolate the effects of curriculum training, no-abstraction inclusion, and reward masking. The full method achieves the strongest performance under abstraction-conditioned evaluation.**
| Approach | Design Choice | AIME 2025 |  |  |  |
| --- | --- | --- | --- | --- | --- |
| curriculum training | including no-abstraction prompt | reward masking | w/ abs (avg) | w/ abs (best) |  |
| variant 1 | ✗ | ✓ | ✗ | 36.51 | 42.29 |
| variant 2 | ✗ | ✗ | - | 37.08 | 42.50 |
| variant 3 | ✗ | ✓ | ✓ | 37.50 | 43.33 |
| RLAD | ✓ | ✓ | ✓ | 42.45 | 48.33 |

#### B.3 Full Results for Abstractions in non-math domains

As seen in Table [7](https://arxiv.org/html/2510.02263v1#A4.T7), conditioning on abstractions helps for 37 domains, where the average and best abstractions outperform standard prompting by 18.0% and 30.0% on average, respectively.

For ARC-AGI, we warmstart the abstraction generator model with synthetically augmented human annotations from the BARC dataset [16]. We present experiments with a fine-tuned abstraction generator and a frozen solution generator in Table 1.

### Appendix C Abstractions in Other Domains

Figure: Figure 7: *Examples of good reasoning abstractions in non-math domains*. Adding the abstraction to the prompt of GPT-4o-mini consistently improves performance on unseen instances.
Refer to caption: x4.png

### Appendix D Qualitative Examples of Math Reasoning Abstractions

Figure: Figure 8: Abstraction Categorization RLAD produces a diverse characterization of abstractions, which we characterize by prompting o4-mini.
Refer to caption: figures/abstraction_classification.png

*Interpreting discovered abstractions.* As discussed in Appendix [D](https://arxiv.org/html/2510.02263v1#A4),
we classify each model‐generated abstraction into four categories for ease of interpretability: (1) caution alert that warns the solution generator to avoid a specific approach; (2) productive launchpoint that suggests strategic framings or problem reformulations that open high-potential solution paths; (3) blind-follow trajectory that prescribes repeatable, step-by-step procedures executable without further insight; and (4) structural shortcut that leverages abstract insights or invariants to collapse multiple reasoning steps into a single leap. In Figure [8](https://arxiv.org/html/2510.02263v1#A4.F8), we show that after training via RLAD, the distribution over these categories shifts, with a notable increase in blind-follow abstractions, which a stronger reasoning model classifies as an effective reasoning path to a successful solution as seen in Appendix [D](https://arxiv.org/html/2510.02263v1#A4).

#### D.1 Prompt for Abstraction Classification

We prompt GPT-4o-mini with the following prompt template to classify each abstraction into one of four categories.

#### D.2 Example for Short Abstraction

#### D.3 Example for Each Abstraction Category

Below, we show examples of abstractions classified into the four categories above.

**Table 7: Evaluation of abstractions on diverse collection of 37 domains. We sampled 10 abstractions by prompting o4-mini, and measure test set accuracy while prompting GPT-4o-mini with each abstraction. We report both the average performance of the 10 abstractions and the best abstraction. We find that the average and best abstractions outperform standard prompting by 18.0% and 30.0% on average, respectively.**
| Dataset | Zero-shot | Best | Average |
| --- | --- | --- | --- |
|  |  | Abstraction | Abstraction |
| UCI Dry Bean | 0.00 | 0.65 | 0.51 |
| Wikipedia Proteinogenic Acid | 0.22 | 0.78 | 0.58 |
| UCI Student Performance | 0.25 | 0.45 | 0.28 |
| UCI Website Phishing | 0.25 | 0.25 | 0.22 |
| UCI Teaching Assistant Evaluation | 0.25 | 0.45 | 0.33 |
| UCI Contraceptive Method Choice | 0.30 | 0.60 | 0.43 |
| UCI Vertebral Column | 0.30 | 0.75 | 0.64 |
| UCI Shill Bidding | 0.30 | 1.00 | 0.95 |
| Kaggle Job Change | 0.30 | 0.85 | 0.83 |
| UCI Caesarian Section | 0.38 | 0.75 | 0.64 |
| Wikipedia Coin Face Value | 0.40 | 1.00 | 0.88 |
| UCI Wine | 0.40 | 0.95 | 0.85 |
| UCI Tic-Tac-Toe Endgame | 0.40 | 0.80 | 0.42 |
| Kaggle Campus Placement | 0.40 | 0.85 | 0.72 |
| Wikipedia Driving Championship Points | 0.40 | 1.00 | 0.74 |
| UCI Mammographic Mass | 0.45 | 0.90 | 0.82 |
| UCI Banknote Authentication | 0.45 | 1.00 | 0.78 |
| Kaggle Engineering Placement | 0.50 | 0.85 | 0.79 |
| RAFT One Stop English | 0.50 | 0.40 | 0.36 |
| LegalBench Function of Decision Section | 0.54 | 0.72 | 0.61 |
| Kaggle Entrepreneur Competency | 0.55 | 0.65 | 0.58 |
| UCI Indian Liver Patient | 0.55 | 0.80 | 0.68 |
| LegalBench International Citizenship Questions | 0.56 | 0.74 | 0.63 |
| LegalBench Abercrombie | 0.56 | 0.80 | 0.67 |
| Wikipedia Color Luminance | 0.60 | 1.00 | 1.00 |
| RAFT Twitter Hate Speech | 0.60 | 0.90 | 0.76 |
| Wikipedia Award Nomination Result | 0.64 | 1.00 | 0.76 |
| UCI Car Evaluation | 0.65 | 0.75 | 0.64 |
| Kaggle Water Potability | 0.65 | 0.50 | 0.38 |
| Kaggle Travel Insurance | 0.65 | 0.70 | 0.59 |
| UCI Internet Firewall | 0.70 | 1.00 | 0.97 |
| RAFT ADE Corpus | 0.70 | 1.00 | 0.89 |
| UCI Somerville Happiness Survey | 0.70 | 0.80 | 0.68 |
| UCI Mushroom | 0.75 | 1.00 | 0.95 |
| UCI Occupancy Detection | 0.80 | 1.00 | 0.92 |
| Kaggle Stroke Prediction | 0.85 | 0.90 | 0.90 |
| LegalBench Corporate Lobbying | 0.88 | 0.94 | 0.88 |
| Average | 0.50 | 0.80 | 0.68 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. For ARC, I’m curious how the results would compare with another baseline: best-of-N. It is known that in domains where external verification is available (such as ARC), best-of-N is a powerful strategy. So I’m wondering, is adding abstraction generation more cost-effective or best-of-N more cost-effective (Table 1)?

2. Lines 70-71 says “The abstraction generator is rewarded for the improvement in the
accuracy of the solution generator, stemming from conditioning on the abstractions it proposes.”
But Equation 2 is the expected reward of the solver conditioned on z, whereas from that sentence I would’ve expected it to be difference w.r.t. the reward of the solver without z. So does the actual reward subtract or not subtract the solver’s reward without conditioning?

3. Did training the DAPO baseline include SFT warm-starting similar to that used in RLAD? (Weakness A1 and Question 4 currently assume the answer is “no”, but I’m not completely sure because I didn’t find any details regarding the DAPO baseline in the paper.)

4. What is the performance of RLAD without warm-starting? Alternatively, how does RLAD compare with a version of DAPO that includes a similar warm-starting procedure? (Also see Weakness A1 regarding fair comparison.)
soundness: 2
strengths: 1. *Originality:* Although the idea of reasoning abstractions has existed for a long time, this is likely the first work that explicitly uses RL to train the generation of reasoning abstractions as part of RL post-training.
2. *Quality:* Comprehensive experiments and analyses are used to demonstrate the utility of reasoning abstractions for solution generation and training to generate them.
summary: This paper introduces RLAD, an approach to RL post-training of language models. The approach trains both a reasoning abstraction generator and an abstraction-conditioned solution generator in a cooperative two-player game set-up. Experiments demonstrate the success of the approach, and analyses demonstrate the utility of reasoning abstractions.
weaknesses: ### A. Evaluation

1. My main concern is with the rigor of evaluation, where there’s a reasonable doubt of unfair comparison. The model was warm-started with SFT data in the beginning, similar to DeepSeekR1, but was compared with DAPO, which doesn’t have warm-starting*.

*I’m not completely sure about this since I couldn’t find details on the DaPO baseline in the paper—also see my Question 3 below.

2. Furthermore, the SFT data for the abstraction generator was generated with models (o4-mini) significantly stronger than the models being post-trained (Qwen3-1.7B), which raises the question of how well the approach scales/how RLAD would be applied to models that are already among the strongest.

3. The ablation studies (Appendix B2) do not study how much the performance gains are from the SFT data generated with stronger models vs. other parts of the training pipeline. (Also see Question 4.)

4. The definition of iso-compute scaling curves is not well-motivated.
  - It is unclear what $k_0$ represents in the formula $m \times (k - k_0) = \mathcal C$, and why this formula is supposed to capture the notion of “iso-compute”. Also, although there is a high-level explanation about how the offset discounts samples that are too similar, subtracting an offset appears to be an arbitrary choice relative to many other possible options such as dividing by a constant.
  - Even in the formula $m \times k = \mathcal C$, this does not capture the extra compute used to generate the abstractions. 

### B. Related Work section

There is, in my opinion, an important line of past work that has been omitted in the Related Work section, namely, past work on learning and using reasoning abstractions. There’s a significant body of literature on learning and using abstractions (also often called concept learning or library learning) – see, e.g., [1], [2], [3], [4].

DreamCoder, LaSR, LINC, LEMMA,

[1] Ellis, Kevin, et al. "Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning." *PLDI*, 2021.

[2] Li, Zhening, et al. "LEMMA: Bootstrapping High-Level Mathematical Reasoning with Learned Symbolic Abstractions." *MATH-AI Workshop at NeurIPS’22*, 2022. 

[3] Grand, Gabriel, et al. "LILO: Learning Interpretable Libraries by Compressing and Documenting Code." *ICRL*, 2024.

[4] Grayeli, Arya, et al. "Symbolic regression with a learned concept library." *NeurIPS*, 2024.

### C. Presentation

1. The main number claimed in the introduction is “44% improvement over… DAPO on AIME 2025”, which should be clarified to mean relative (percentage) improvement instead of absolute improvement. Furthermore, 44% appears inconsistent with the reported numbers in the tables. In Table 2, the RLAD w/ abs (avg) result on AIME 2025 42.45%, and the DaPO w/o abs (avg) result is 37.92%, and 42.45/37.92 = 1.119, which is far from the claimed 1.44. (I did manage to find a ratio that’s close to 1.44, aka the ratio of RLAD w/ abs (best) 48.33% to no RL post-training w/o abs (avg) 33.75% is 1.432, but a) these are not the right numbers to take the ratio of, and b) 1.432 should round to 1.43 and not to 1.44.)

2. The way the paper is presented suggests that “reasoning abstractions” is a novel concept introduced by the paper, which is far from reality (see Weakness B on Related Work). The novel aspect of the paper is, in my opinion, not the idea of “reasoning abstractions” themselves, but the use of RL to post-train a model to generate reasoning abstractions in tandem with an abstraction-conditioned solution generator (last three lines of the Related Work section). Clarification in the paper would be appreciated.

3. High-level presentation would benefit from more explicit summaries at the beginning of each (sub)section. For example, the summary paragraph at the beginning of Section 4 would benefit from a sentence like “In this section, we describe our approach to generating reasoning abstractions via summarization of reasoning traces, and demonstrate how they improve reasoning performance by acting as hints.”

4. The “Modifying reward design” paragraph is not super intuitive and would benefit from further elaboration on how it “address[es] one of the challenges” and which challenge it addresses. Currently, the reward design appears to be mainly supported by the empirical fact that it improved performance (Appendix B2), but motivation from first principles would be nice to see.

5. Typos:
- Incorrect citations, e.g., DeepScaleR Hard (line 81-82) mistakenly cited e3.
- Line 116: “(a) … (c) …” -> “(a) … (b) …”
- Lines 260-261: “recent results RL”
- Inconsistent capitalization of DAPO as DAPO in the main text and DaPO in the appendices.

6. Appendix C is empty.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - How well does the abstraction generator generalize across different datasets? For example, can they be trained on AIME and then transferred to other math datasets? Or does it overfit to that distribution of math problems.
- Also, how much delta/performance gain are we getting from training both the solver and generator? If the solver does not go through the RL training and the only abstraction generator does, does the performance gain vary significantly?
soundness: 3
strengths: - The paper is written clearly and their contribution in terms of the proposed algorithm and its effectiveness over a the DAPO-based RL baseline is both intuitive and clear.
- Show some interesting analyses on how to allocate compute between the abstraction generator and the solver and points out that allocating more compute to the generator is often times more helpful than the solver.
summary: The authors propose a two-player RL paradigm where they train a model that proposes "reasoning abstractions" based on the question and another model that actually learns to solve the problem with the proposed abstraction. They show that this leads to meaningful improvements against a baseline RL policy trained to simply solve the questions.
weaknesses: - It is unclear that current models are not already good enough at generating abstractions based on their solutions. For example, [1] and [2] show that one can prompt LLMs to generate their own abstractions and create a "cheatsheet" of abstractions (whether it's cumulative or using a retriever), which also boosts their performance. So, a comparison to a baseline where one can simply prompt the model to generate its own abstraction would be compelling.
- Some details on the training are unclear: the generator and solver are different models? How many samples were used to run the warm-start of the generator? (how many abstractions per question were generated per problem for the warm start)

[1] Suzgun, Mirac, et al. "Dynamic cheatsheet: Test-time learning with adaptive memory." arXiv preprint arXiv:2504.07952 (2025).
[2] Didolkar, Aniket, et al. "Metacognitive reuse: Turning recurring LLM reasoning into concise behaviors." arXiv preprint arXiv:2509.13237 (2025).

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: How sensitive is RLAD to the relative strength of the abstraction generator versus the solution generator? It seems like training stability could depend heavily on their initialization or model size. could the authors clarify if balancing their capabilities was necessary for convergence or performance?

Can the authors provide a deeper analysis of what kinds of abstractions emerge during training, for example, are they procedural (methods), factual (lemmas), or meta-level strategies and whether these abstractions transfer effectively to tasks outside math and ARC-AGI?
soundness: 3
strengths: The paper presents a novel and well motivated idea which is using reinforcement learning to discover and leverage reasoning abstractions, rather than merely extending reasoning depth which is what typical RL does. The approach is technically sound, with clear definitions, careful reward design, and strong empirical validation. The 2 player design is original and The writing is clear and well-structured, making a fairly complex setup easy to follow. Overall, it stands out as an original and solid contribution that meaningfully advances how we think about structured reasoning in large language models.
summary: This paper proposes RLAD, a way to train LLMs to actually discover and use reasoning abstractions instead of just writing longer chains of thought. The idea is to have one model generate short natural language “abstractions” (like useful hints or lemmas) and another model solve the problem using them. Both are trained together with RL so the abstraction model gets rewarded when its hints help the solver. It basically teaches the model to explore breadth instead of just depth in reasoning. Across math benchmarks like AIME and DeepScaleR, it beats long-CoT RL methods by a large margin and shows that generating abstractions is a better use of compute than brute-force sampling.
weaknesses: The reward design, while intuitive, could be better analyzed to show why the two-player setup works beyond simple prompt diversity effects. The paper would also benefit from more interpretability analysis of the discovered abstractions how stable they are, whether they transfer across domains, and what kind of procedural knowledge they actually encode. Finally, the training setup remains computationally expensive, which may limit broader adoption without lighter or more scalable variants.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 4
questions: How would you factor out the performance gains from the abstraction prompts and the stronger model?   

Is there any improvement in performance when both the abstraction and solution models are the same?
soundness: 2
strengths: Paper is very well written and clear.  The experimental results of Table 2 look promising.
summary: A heuristic framework that improves benchmark performance by augmenting LLM prompts with an abstraction prompt composed by another more capable LLM.
weaknesses: By design, the solution model is weaker than the abstraction model.   
Lines 158-161: "Concretely, we utilize the Qwen3 series of models to produce solutions and a stronger reasoning model, o4-mini, to generate abstractions. " 

It is my understanding that for an apples-to-apples comparison, the solution model and the abstraction model should have the same capability. Otherwise, it is impossible to tell how much of the improvement was obtained from the abstractions rather than the  use of stronger model for the abstractions.  Perhaps this is already visible in Figure 2, leftmost panel, where using the weaker model for abstraction provides no benefit. The strength of the proposed approach depends on the results of a fair comparison.    

If my understanding is incorrect, am supportive of acceptance.  If my understanding is correct, support rejection.


Decision: Accept (Poster)