Learning to Reason as Action Abstractions with Scalable Mid-Training RL
Authors: Shenao Zhang
Donghan Yu
Yihao Feng
Bowen Jin
Zhaoran Wang
John Peebles
Zirui Wang
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
large language model
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models excel with reinforcement learning (RL), but fully unlocking this potential requires a mid-training stage. Intuitively, an effective mid-training stage should both learn a strong policy prior and enable fast learning through online interactions. We formalize this intuition by presenting the first theoretical result on how mid-training shapes post-training: it acquires strong policy priors by efficiently pruning the action space and accelerates RL convergence by shortening the effective planning horizon. Moreover, we prove that temporal abstractions simultaneously compress the size of the action set and reduce the decision horizon, thereby improving regret minimization after training. Building on these insights, we introduce Reasoning as Action Abstractions (RA3), a scalable mid-training algorithm. Specifically, we derive a temporal variational bound and optimize it by iteratively discovering temporally-consistent latent structures via RL, then fine-tuning on the bootstrapped data. Experiments on code generation tasks demonstrate the effectiveness of our approach. Across multiple base models, RA3 improves the average performance on HumanEval and MBPP by 8 and 4 points over the base model and the next-token prediction baseline. Furthermore, RA3 achieves faster convergence and higher asymptotic performance in RLVR on HumanEval+, MBPP+, LiveCodeBench, and Codeforces.

Published as a conference paper at ICLR 2026
LEARNING TO REASON AS ACTION ABSTRACTIONS WITH SCALABLE
RL
Shenao Zhang Donghan Yu², Yihao Feng", Bowen Jin Zhaoran Wang', John Peebles" Zirui Wang
Northweesern University, AApple, 'UICC
ABSTRACT
Large language models excel with reinforcement learning (RL), but fully unlock- ing this potential requires a mid-training stage. Intuitively, an effective mid- training stage should both learn strong policy prior and enable fast learning through online interactions. We formalize this intuition by presenting the first theoretical result on how mid-training shapes post-training: it acquires strong pol- icy priors by efficiently pruning the action space and accelerates RL convergence by shortening the effective planning horizon. Moreover, we prove that temporal abstractions simultaneously compress the size of the action set and reduce the de- cision horizon, thereby improving regret minimization after training. Building on these insights, we introduce Reasoning as Action Abstractions (RA3), scalable mid-training algorithm. Specifically, we derive a temporal variational bound and optimize it by iteratively discovering temporally-consisten latent structures via RL, then fine-tuning on the bootstrapped data. Experiments on code generation tasks demonstrate the effectiveness of approach. Across multiple base mod- els, RA3 improves the average performance on HumanEval and MBPP by and 4 points over the base model and the next-token prediction baseline. Furthermore, RA3 achieves faster convergence and higher asymptotic performance in RLVR on HumanEval+, MBPP+, LiveCodeBench, and Codeforces.
INTRODUCTION
The potential of reinforcement learning (RL.) as universal policy-improvement operator has been demonstrated with remarkable success in training large language models (LLMs), spanning appli- cations preference optimization (Ouyang et al., 2022), mathematics (Guo al., 2025; Zeng al., 2025b), code generation (Yang et al., 2025; Zeng et al., 2025a), and agentic tasks (Team et al., 2025; Zhou al., 2025b). key factor behind these successes is the strengthened policy prior, typically obtained through mid-training (Wang et al., 2025c; Su et al., 2025), which is continued pre-training on expert data sampled from the optimal policy. Despite its widespread use, the precise role of mid-training in shaping post-training RI. remains poorly understood. Without such understanding. it is difficult to design principled and effective mid-training algorithms. Hcuristic metrics, such as the performance entropy of the initial RL policy, provide only indirect signals and do not by themselves guarantee improved downstream performance.
In this paper, we propose the first theoretical analysis of how mid- mid-training shapes post-training RL. We identify two key factors of mid-training algorithms that determine their effectiveness during RL: the efficiency of pruning the decision space and the effective planning horizon. The first factor governs the initial policy prior of RI., while the second decides the policy's potential to be improved through online interactions. To minimize regret in post-training RL, an ideal aiid-irainnn algorithm should extract from finite expert demonstrations the complete set of action subsets sufficient for all tasks, and enable fast selection among them during RL. Our results show that pruning efficiency is inversely related to the cardinality of the smallest near-optimal action subset, and that post-training RL converges faster when actions are temporally extended. These findings suggest that mid mid-training should operate in the space of action abstractions rather than primitive actions. Intuitively, learning
Work done Apple. 'Equal advising.

---

which in- Published as a conference paper at ICLR 2026
high-level "skills" that are transferable across tasks yields compact decision space and shortens the planning horizon, which makes pruning more efficient and RL more tractable.
To uncover this action hierarchy, derive temporal variational lower bound for the next-token prediction (NTP) objective. It can be optimized by iterative aspectation-maxiiizzinen volves self-supervised RL step that uses the log- oggllikelhood of expert data reward discover the hidden latent sequence, and a supervised fine-tuning step on the bootstrapped data. With an appropriate latent prior, the KL divergence enforces temporal consistency, ensuring that the latents function as coherent action abstractions. These design choices yield scalable mid-training algo- rithm, Reasoning as Action Abstractions (RA3), where the KL penalty naturally determines when rational rollouts are necessary. thereby controlling computational cost.
We evaluate RA3 on code generation tasks using Qwen and Llama base models ranging in size from 1B to 8B. The mid-training dataset consists of 3.5M code snippets totaling 1B tokens. Our re- sults show that fine-tuning on data bootstrapped with action abstractions substantially reduces cross- entropy loss and improves performance across multiple benchmarks, including HumanEval, MBPP, and their extended variants. On average, RA3 achieves a 4 point gain over NTP and an 8 point gain over the base models. Furthermore, RA3 accelerates RLVR convergence and attains higher asymp- totic performance on HumanEval+, MBPP+, LiveCodeBench, and CodeForces. Together, these findings highlight the scalability and advantages of learning action abstractions in mid mid-training.
2 BACKGROUND
Imitation Learning. A task M = AS, is an MDP defined by the state space S, action space A, reward R, and the discount factor y < 1. In the language space, states contexts that include all the previous tokens, and each action is a single token. The state transition is either deterministic by appending the new action tokens the previous context or governed by the external environment. An expert policy is the policy that maximizes the expected state-action value:
Tg E argmax nm R(s;, a.) + (VVM(()))) EaaN Σ " R( ai).
It is worth noting that when the task is inherently solvable within one step, such as math problems where r rS0. agt) = 1, the expert policy should deterministically output the ground-trute answer ag 1, to maximize the return. Since most mid-training math data has explicit human reasoning before agt. we instead focus on the multi-step decision-making domains, such as code generation and agentic tasks, where the expert trajectories
at so, i... Tg gg (ag])s))
sequence of actions.
Next-token prediction (NTP) during mid-training can be viewed as an imitation learning process on an offline expert dataset Dr, collected by rolling me on the sampled tasks M p(M). Its objective is to maximize the conditional log-likelihood:
JNTP )- = B(so:r log ग (a0.r
(O.T)) = ED
Σ log क (az 1x))
(2.1)
where s E S, at E A. the total number of tokens in one expert demonstration, π is the training policy, and so is the beginning of the sentence (BOS) token. The formula in (2.1) applies directly to actions at coarser granularity than tokens, such as sentence-level actions.
NTP is adopted in different stages of LLM training, including pre- pre-training, continued pre-training (or mid-training, if the goal to acquire reasoning foundations before RLVR), and supervised fine- tuning, We primarily focus on the mid-training stage during the three-stage training procedure: pre-training, mid-training, and RLVR post-training.
RL with Verifiable Reward. The goal of post-training RL is to maximize the expected return. A common setup for RLVR is to use a binary outcome-based reward in a single-step MDP, defined as r(s, o) verifier(s, o) to measure if the model response is identical to the ground-truth answer corresponding to the prompt question s. We adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024; Guo et al., 2025) our default RLVR algorithm in experiments. Its objective is
(T) 1 G
(0: s)
<(; 8) Tola(o; (8) AA)) -BDKa mee))
min
A,, clip
1 +
(o 1 (8)
i=1
2

---

Published as a conference paper at ICLR 2026
where O₂ (-))) B are hyperparameters, and the advantage is calculated within the group G:
A (r(s,o;) mean((r(s, (sslss))))))(((d(rrrrr)))).).)e
(2.2)
3 HOW MID-TRAINING SHAPES POST-TRAINING RL
The goal of post-training RL is minimize the regret
min E M- (VR (s))l
(3.1)
i... to learn that is near-optimal across tasks M.
An effective mid-tr mid-training stage should extract from finite expert demonstrations the complete set of action subsets sufficient to solve all tasks, and facilitate efficient selection among them during RI. for fast convergence. This perspective highlights that the efficiency of mid-training lies in pruning away useless actions and structuring the decision space so that online RL can improve policies effectively.
To formalize this, we define M2 (S, Z', R, 7) in way similar to M, except with action space restricted to z'. We say that Z' is near-optimal for M if near-optimal policies can be constructed using only the actions in 2'. The goal of mid-training is thus to prune away with high probability all "bad" action subsets Z' that are sub-optimal. Formally, we define:
Definition 3.1 (Near-Optimal Task Action Subset). action subset Z' C called e-optimal for task M if the optimal values in M and M2 satisfy A(M, 2') = (MM(s)) - VMg. (50) S <.
This definition allows us to connect mid-training with post-training RL through the following de- composition:
Lemma 3.2 (Regret Decomposition). For any 2' 2, the post-training RL regret in (3.1) satisfies
EM (5)) - V (so)] = E EM ( (M,, Z')] + EM--pts)VM (5o) - VM2g (so))
action-sel pruning error
post-training RL error
Lemma 3.2 indicates that mid-training should identify an action subspace 2 that minimizes two
sources of error: (1) the approximation error from pruning the full action space 2 down to 2, and (2) the post-training RL error incurred when planning within 2.
Corresponding these two error terms, we demystify the key factors that determine the effectiveness of mid mid-training: (I) the efficiency with which the algorithm prunes the decision space from expert demonstrations, and (2) its impact on the convergence speed of post-training RL. The first factor governs the learned prior of the initial RL policy, while the second determines the extent to which the policy can be further improved through online interactions.
3.1 MID-TRAINING ACQUIRES STRONG POLICY PRIORS VIA EFFICIENT ACTION PRUNING
A central metric of mid-training pruning is how efficiently the algorithm eliminate useless tions, since it directly determines the quality of the resulting action space given a fixed number of expert demonstrations. To formalize this, we introduce the following notion.
Definition 3.3 (Minimal Size of Near-Optimal Action Subset). Z. C Z is e-optimal action subset if it is e-optimal for all tasks. Let Ze| denote the minimal size of such subset.
According to Definition 3.1, AA itself is an optimal action set since A(M, A) = 0. Thus, Z, always exists and [z.| is finite. Now we are ready to give our first result on the pruning efficiency during mid-training.
Theorem 3.4 (Pruning Efficiency). Denote [Dgl as the rollout number in the mid- mid-training data. If
DDel = o E. log( lo)((2/)))/o)
then with probability at least 1-8. the pruning error Lemma 3.2 satisfies EM-MMM)AA(M [A(M, 2) s <1 - σ) + a 7))
3

---

Published as a conference paper at ICLR 2026
We defer all proofs to Appendix A. Theorem 3.4 shows that the number of expert samples required for action-space pruning during mid- trdiiinin decreases as both [2,] and |2| shrink. With higher pruming efficiency, i.c., a smaller [Z,]] and (Z], the probability that suboptimal actions survive mid- training decreases, leading to a smaller pruning error. This result highlights the importance of compact decision space and motivates defining 2 as a space of action abstractions instead of primi- tive actions A.
Action abstractions are defined analogously to Markov options (Puterman, 1994; Sutton et al., 1999; Precup, 2000), representing the abstraction of temporally-extended primitive actions. Each 2 corresponds to a high-level intention that executes a sequence a6..... as+r. with duration T ~ p(. 8, 2). Notably, the above result for Z also applies A special instantiation of 2, with T and aj = 2j.
Compared to the space of primitive actions A, the space of action abstraction has substantially smaller size of Z and [Zel. since each action abstraction corresponds to transferable skill span- ning multiple tasks. Consequently, learning action abstractions enables mid-training to more effi- ciently approximate a near-optimal action set and provide a stronger policy prior, thereby reducing the burden on post-training RI..
3.2 MID-TRAINING ACCELERATES RL CONVERGENCE WITH SHORTER DECISION HORIZON
The above analysis mainly considers the influence of mid- mid-training the pruming enorr and fixed the post-training RL error. Next, we analyze how mid- traininiing shapes post-training RL. Our result below is based value iteration due to its simplicity.
Theorem 3,5 (RL Convergence Rate). To achieve an e-optimality that satisfies ||V - V* S the required number of iterations N is lower-bounded by N 2 I-
1
log E(1->))
Rim, where 7 =
sup,,0 Eh s, 2] S and Rmax max,,a R(s, a).
The above result reveals that the reasoning structures acquired during mid-training influence the convergence through the duration r of the temporally-extended actions. For actions that last longer, is smaller and RL converges faster to optimality than mid-training with NTP, where 7 1 and 7 = 7. This makes intuitive sense as each Bellman backup jumps across т steps in one shot, which shortens the effective planning horizon and shrinks the eтгоr faster per iteration. Similar 1/(1 7) dependency also appears in the bound for broader RL algorithms such as policy gradient (Agarwal al., 2021; Zhang et al., 2023), which we omit due to the requirements of additional assumptions.
def Zongest www.r_/nnnnnnnnnoooo pathigggggttitiitta
ag
20
2y base case
du
gead(0):
1
81
2g inke ne table for bast path Length from each cell
dy Ian (grid) lan (geid(01) dare (11,)))-1,)).00 (9,-113 (lgcide))
Q
#2
[EX) a for raage (013 ansat
Rg apply Do transition from Iover-oxlsed naighhors
Qg
#3
21
(4g
Us
foe de, de in dars:
dg
sz, ne = 1 de. dp(or] [nol detei
Q4
geidinel
dpIel dptne] [oo]
20
dg
14 dptel ana: and dpiz]te]
2g finalire global optinus
Figure 1: (Left): The probabilistic graphical model of the action hierarchy. (Middle & Right); Examples of primitive actions in expert demonstrations (blue) and the hidden high-level temporal abstractions (green), in web agent and code generation domains, respectively.
FROM PRIMITIVE ACTIONS TO TEMPORAL ABSTRACTIONS
In Section 3, analyze the benefits of leveraging temporal action abstractions during mid-training from two perspectives: its efficiency in pruning the action space and its ability to accelerate subse- quent RL. Intuitively, there are high-level "skills" that shared across tasks, utilizing which helps
4

---

Published as a conference paper at ICLR 2026
generalization and makes planning easier, since the action space shrinks and the decision horizon reduces. We illustrate this with two examples in Figure In what follows, we introduce a principled way to extract the temporal abstractions from the primitive actions in the mid-tre mid-training data.
4.1 TEMPORAL VARIATIONAL BOUND
We begin by seeking alternative way to maximize the likelihood beyond predicting the next tokens. Specifically, we give a sequential Evidence Lower Bound (ELBO) of the NTP objective:
Theorem 4.1 (Temporal ELBO). The next-token prediction objective (2.1) lower bounded by
p( [T) 2 J [T, q) = Poor... log [a - Dxi, (9(2)|01, z0a-1) p(2) (0..)))
where p( (8,, 2-11 is the prior distribution of 2
The ELBO introduces a sequence of latents 2o:r to model the observed primitive actions Gu:r- The intuition behind it is that there are hidden thoughts or intentions, i.e., #0.7, behind the expert decisions G0:7 ~ Dr, which are not present in the mid-training data. The amortized variational inference leverages the parameterized g(2g| 81, 20:t-1) as an approximation of the true posterior of Optimizing the ELBO is equivalent to modeling the distribution of the latents 20:T and using the inferred latents to better model the likelihood p(ao:r) s0:T).
We maximize the ELBO by alternatively optimizing q and æ, in an Expectaton-Maximizatii (EM)
manner. In each EM iteration i, the step fixes Tł and updates q to maximize the ELBO, i.e.,
q: = argmax I [TT, ④) = argmax E DE Pa0
Σ log mi 8t; 21) DKL (a l1
(4.1)
RL reward at step
This corresponds to a -horizon RL procedure with the per-step reward defined the log- log-likelihood of the observed expert actions, with KL penalty that we will discuss later. Intuitively, (4.1) encour- ages the sequence of latents 2o:r sampled from % "explain" the expert decisions ag:2-
The M -tsp then fixes the updated q, and optimizes x:
T++1 argmax J(T. argmax ED [log T ([a ss, 21,2
(4.2)
"
which is simply imitating the expert trajectories that are bootstrapped with the inferred latents z1.
4.2 CONSSSSLL- LATENTS AS ACTION ABSTRACTIONS
We have transformed the maximization of likelihood on primitive actions 0o:T to an ELBO objective that learns from the bootstrapped sequence with latent trajectories sampled from the variational pos- terior g(2/|sg St, 20:t-1). Recall that our analyses in Section reveal the benefits of learning compact sets of high-level action abstractions. We will show in the following how to fulfill this goal with properly defined latent prior.
Specifically, the latent 2 in the ELBO is defined per-step for every & E [0, 7]. To let represent an abstraction of eexeerd actions that spans across T p(-1st, 24) timesteps, it is equiva- lent to let 2t 2t+1
244- This can be achieved by setting the prior p(a++ | 2e) in Theorem 4. to have large probability mass on zt, and uniformly distributed at all other positions:
p( St, 11 a6(zz (1- (=))
(4.3)
where a E [0, 1] is a hyperparameter, 8(-) is the Dirac delta function, and U(-) the uniform distribution over Z. The delta function helps preserve cemporally-consiste latent as high-level action abstraction, and the uniform distribution encourages learning a diverse reasoning foundation.
Thhe notation here deviates slightly from Section 2, where all identical latents written single z.
5

---

Published as a conference paper at ICLR 2026
5 RA3: A SCALABLE MID-TNAINING ALGORITHM
In the context of LLMs, the action abstraction 2 serves a similar role as the rational or intention, and the primitive action is the actual answer or operation in the environment. Taking code generation as an example, can be the reasoning before writing the next code block dp+rr-
In addition the theoretical benefits discussed in Section 3, the temporal consistency of the latents also makes it possible to scale up the optimization on mid-training- erating rationals 20:T corresponds to T rollouts, whose size proportional to the total number of tokens in the data. This prohibits us from making full use of the high-qualit, data in the mid mid-training corpus, which typically contains billions of tokens. Fortunately, the temporal consistency of latents rescues us from sampling rollout 2, at each timestep i, as it is kept unchanged for steps.
data. Specifically, gen-
To avoid redundant sampling, we define two types of latents: 2 <act>> and that begins with <think>>. For latents sampling new rationals since fall under the same high-level intention as d;-1- 1- For this reason, we use 24- <act>> to indicate 2g 21-1- Whenever <act>> is sampled, the rollout stops immediately. By doing so, full rollouts are only sampled when <think> is the first token, significantly reducing the RL inference cost. Besides, may rewrite the KL term in the ELBO follows.
-t that are remporally-consiteer
dr is directly generated without
Proposition 5.1. With the prior defined in (4.3), the KL term in Theorem 4.1 satisfies
DRI. (1(2)181 20.-1) |1 p( z0.- DRi. (Bem(gaa) || Ber((a)) -(1 qua g g(2aloe, ((20110,200111) 201-1))
where que= = g(2 = <act>> St, 10: Bern(-) is the Berouuli distribution, and DKL (Bern(gai)
||
Bern(a)) Ezsa [(a = <act>> log
9ust
+ 1(z of <act>> log 1
The KL is decomposed into two terms: a KL. term between Bernoulli distributions and an entropy term that encourages diversity. By setting a > Oact. the KL discourages unnecessary thinking since assigns a larger penalty to * <act> than 2₁ <act>> The penalty difference defines a threshold, guiding the model generate new rationals only when they improve the log-likelihood reward by more than this threshold. In implementation, instead of tuning a, we apply reward shaping and set fixed penalty only 2t * <act>> The proposition also indicates that the additional training cost compared to NTP can be adjusted by a. In the extreme case where a = 1, our algorithm degenerates to NTP, since DKI. (Bern(qaa)) Bern(1)) is infinite for all Jast < i.e., the policy receives an infinite penalty for generating any rationals. We provide the pseudocode in Algorithm 1.
Algorithm 1 Reasoning as Action Abstractions (RA3) for Mid- Mid-Training
1: Input: Base LLM mo. mid-training dataset Dg, penalty hyperparameter c.
2: for EM iteration in 2, do
3: Optimize qi argmax, DD wrgg Et= log m (ac (sc 21) c1( * <act>)] via RL.
4: Fine-tune T = argmax, ED , [log π (az $t, 2(] via NTP.
We enforce the two types of actions discussed above by incorporating simple format reward in the RL step that assigns zero rewards for wrong formats, which we omit in the pseudocode for clarity. We optimize the T-horizon RL using policy gradient, with advantages calculated in similar way to (2.2) in GRPO; after sampling G length-T rollouts, we set the baselines as b(sr) De E r / that independent the actions, and combine it with the state- action value to calculate the advantage at each step.
6 RELATED WORK
LLM Mid-Training RL has long been utilized for training language models (Nguyen et al., 2017; Paulus et al., 2017; Jaques et al., 2020; Ramamurthy et al., 2022; Ouyang et al., 2022). However, its potential as universal policy-improvement operator has not been fully unlocked until recently, when reasoning models learn to cast intermediate thoughts as actions and optimize them via RL (Guo et al., 2025; Zeng et al., 2025b; Liu et al., 2025; Team et al., 2025). This paradigm also guides

---

Published as a conference paper at ICLR 2026
the design of the policy prior. To obtain strong initial policies in terms of both performance and exploration diversity, mid-training (Xu et al., 2025; Wang et al., 2025c; Su et al., 2025) plays an im- portant role, which performs reasoning or agentic continued pre-training on high-quality expert data. Previous work that leverages next-token prediction during mid-training is an imitation learning pro- cess, where the data comes from rollouts of optimal expert policies, such as humans' demonstrations of device-controlling and code-writing (Rawles et al., 2023; Huang et al., 2024; Bai et al., 2024). Anchoring our findings, it is observed that learning the action hierarchy with abstraction-basee rea- soning performs better than training on primitive actions alone (Xu et al., 2024; Chen et al., 2024; Wang et al., 2025b; Xue et al., 2025). For this reason, some mid mid-training datasets augment the ex- pert data with synthetic reasoning distilled from frontier LLMs (Wang et al., 2025b; LI et al., 2024). However, the distributional shift makes it unclear how well the student LLM can benefit from these action abstractions, compared to RA3, which learns its own reasoning via RI. Besides, RA3 is more preferred considering the cost of augmenting trajectories with reasoning distillation on a large scale. In fact, for the code generation domain that we interested in, most mid- mid-training-size sized datasets mainly contain human code from the internet, without costly relabeling.
Self-Supervised RL. Optimizing the temporal variational bound involves self self-super RL step with bog-prob of the expert action as reward (Zhong et al., 2025; Ruan et al., 2025; Zhou et al., 2025a; Dong et al., 2025; Wang al., 2025a). Compared BRiTE (Zhong et al., 2025) that is for- mulated as a post-training method with a standard BLBO for single-step problems, we are motivated to learn action abstractions with the ELBO derived for temporal sequences, guided by our theoretical analysis. When the decision horizon is one, RA3 reduces to BRİTE. However, for multi-step prob- lems such as code generation, naïvely applying BRiTE corresponds treating the log-probability of the entire code sequence as single RL reward, which leads to high variance and unstable training in our setup. Moreover, Dong et al. (2025) hand-crafts an entropy-basee ing positions and trains only 4k samples with instructed reasoning models. In contrast, RA3 is theoretically grounded, scales to mid-training setups, and enables the model to autonomously learn when to skip unnecessary reasoning by generating temporally consistent latents.
rule to determine reason-
Markov Options. RL based on options (Sutton et al., 1999; Precup, 2000; Bacon et al., 2017; Zhang et al., 2021) enables agents to represent courses of actions at extended time scales and learn in the MDP with them. It greatly helps long-horizon complex tasks by naturally introducing hierarchical structures (Jong et al., 2008). Our analysis of the decision space size is partly inspired by (Brunskill & Li, 2013; 2014). Different from their study on option transfer in lifelong and multi-task RL, we mainly interested in mid-training algorithm design and its impact on post-training RL in LLMs.
7 EXPERIMENTS
Experiment Sctups. We focus on Python code generation in our experiments. The granularity of primitive actions a is a single line of code. For the two types of latents z, to avoid additional fine- tuning special tokens, we remove the newline character !n at the end of and set <act>=)n, <think>>=\ That is, after line a,, the model either only outputs \n before a++1 or generates a comment line as high-level abstraction guide the code writing. The format reward is non-zero for the think action if it begins and ends with \ and the first non-space token is #. This design ensures that the reasoning bootstrapped data has the correct syntax. The RL step of RA3 is implemented in a similar way to multi-turn RL: the code line at from the training corpus and the reasoning comment 2, alternate as turns. We adopt asynchronous rollout with the SGLang engine (Zheng et al., 2024), as batched inference often incurs idle time between turns. That is, batched rollouts must wait for all 2, to be generated before proceeding to the next turn, which is especially undesirable in our setting, where temporal consistency in is intended to reduce unnecessary reasoning.
We implement RA3 on three pre-trained models: Qwen-2.5-1.55h (Hui et al., 2024), -la2m-3.2-118 and L.ama-3.1 1-BB (Dubey et al., 2024). For the mid-training data, we select the Python data within the continued pre-training corpus of Huang et al. (2024), which consists of a large portion of high- quality internet data (2.36M code snippets, 834M tokens) and small portion of code-only synthetic data (129K snippets, 120M tokens). This leads to a total mid-training size of 3.5M code snippets with 1B tokens. Each EM iteration involves 400 gradient updates, with the first 40 being the RL policy gradient. The hyperparameters of our E step is the same as NTP: batch size of 512 with a learning rate of 2e 5. The RL step of RA3 sets the maximum length of 16, with a sampling
7

---

Published as a conference paper at ICLR 2026
temperature of 1.0 and group size of G = 3. The entropy coefficient is set to 0.001 and we do not regularize the reference KL. The RL batch size is set to 1024 with a learning rate of 2e 6. We set the penalty 0.05. AdamW optimizer is used (Loshchilov & Hutter, 2017) in experiments.
RA3 RL Reward
Reward
-1.5
Liama 2.2 2015
Owen 2.5 50
-2.01
Liama 3.1 30
20
40
Figure 2: (Left)): Curve of RL training reward. (Right): Examples of the data from mid-training and after reasoning bootstrapping, where skills are abstracted, such dummy head creation and BFS.
Mid-Training Results. We begin by analyzing the results during the EM steps of RA3. The RL training reward in the first step given in Figure 2. We find that the model can quickly learn to maximize the reward, so that most of the compute can be allocated to reasoning bootstrapping and fine-tuning in the M step. We also provide two examples of data in Figure
For the M step, an important metric is the cross-entropy (CE) loss, i.c., the negative lig-likelihood of the next token. can be observed from Figure that fine-tuning on reasoning- bootsstpebbot data significantly accelerates learning speed. This supports the scope of mid-training we defined in Section 2: the expert policy rollouts are sequences of primitive actions, such as the raw coding lines. There are hidden reasoning trajectories that the expert follows to make decisions, but are unavailable in the data. The learning is much easier when such reasoning traces are extracted via RL.
Liama 3.2-1B
Qwen-2.5-1..
3.22-8
0.8
0.7
C6 Loos
Less
0.5
0.5
0.5
0.4
0.4
Stop
Stop
Figure 3: Data bootstrapped with reasoning learned in the E step reduces the CE loss in the M step.
To better assess the algorithms, we evaluate the resulting models on the HumanEval (Chen al., 2021). MBPP (Austin et al., 2021), HumanEval+, and MBPP+ (Liu et al.. 2023) benchmarks using Ben Allal et al. (2022). We test the improvement of each EM iteration of RA3 by the average scores on these benchmarks, and compare with the NTP checkpoints that are trained on the same data. The results are shown in Figure 4. can be observed that learning to reason as action abstractions not only gives lower CE loss, but also achieves higher evaluation performance with fewer data.
Llama-3.244 3-118
Qwen-2.5-1.5B
Llama-3.1-88
NTP
NTP
NTP
Mean Accuracy
Mean Accuracy
RA3
Mean Accuracy
68.
RA3
42
26-
24
42-
38
Iteration
Iteration
Iteration
Figure 4: Evaluation results during mid-training, with accuracies averaged across four benchmarks.
We also report the pass@1 Øk results of the best checkpoint for NTP and RA3 in Table 1. The result reveals the generalization benefits of learning with temporal action abstractions.
8

---

Published as a conference paper at ICLR 2026
Table 1: Mid-training performance: pass@k HumanEval, MBPP, HumanEval+, and MBPP+.
HumanEval
MBPP
HumanEval+
MBPP+
Avg.
Model
Method
p@1
p@5
p@1 1
p@5
p@1
p@5
p@1
p@5
p@
p@5
Base
18.9
30.4
25.8
37.0
17.1
25.7
31.5
47.9
23.3
35.3
Llama 3.2 IB
NTP
21.3
34.1
27.8
45.8
17.7
29.5
34,4
51.8
25.3
40.3
RA3
25.0
38.1
32.8
46.1
22.0
33.5
39.4
54.6
29.8
43.1
Base
37.2
53.7
38.6
56.5
32,3
48.1
43.4
61.0
37.9
54.8
Qwen 2.5 1,5B
NTP
41.5
57.3
43.4
59.2
35.4
53.3
46.6
64.8
41.7
58.7
RA3
48.2
63.6
45.8
61.3
42.7
59.0
49.7
64.9
46.6
62.2
Base
36.6
57.5
45.2
59.1
30.5
54.7
51.6
65.7
41.0
59.3
Llama 3.1 8B
NTP
48.2
62.0
48.6
62.9
42.7
60.1
51.1
67.4
47.7
63.1
RA3
50.0
66.2
48.0
63.0
44.5
61.3
53.2
67.8
48.9
64.6
Post- Training RLVR Results. We then study the impact of mid-training post-training RL. We use GRPO as the default RLVR algorithm. and leverage the off- fff-the-shelf DeepCoder codebase (Luo et al., 2025). We use the contains 7.7K data, filtered from TACO (Li et al., 2023), Mattern et al. (2025), and LiveCodeBench (Jain et al., 2024). We run independent RLVR training with different random seeds (3 for small models and 2 for the 8B model), and the evaluation results are reported in Figure 5.
dataset (Fu et al., 2025) for training, which
HumanEvale
MBPP4
LiveCodeBench
Codeforces
2:11
33
100
152
300
250
50
100
150
200
250
52
100
150
200
250
120
164
231
280
200
Owen2.51.5 5.
100
122
26B
RL Step
RL. S280
13 Srep
Step
Base
NTP
RA3
Figure 5: RLVR training results (mean and standard error) from different mid-training models. We observe that mid-traine mid-training significantly improves the RLVR performance compared the base models. Besides, RA3 is able to learn more effectively during RLVR, in terms of both convergence speed and asymptotic performance, supporting theoretical results in Section 2.
Figure 6: We study the effect of penalty e on (a) the behavior of the RL step, (b) the mean accuracy of HumanEval and MBPP, (c) the average length of and (d) the ratio of full rollout samples.
Ablation Study. We first investigate the effect of the KL penalty e when training the Qwen model. Recall that regulates the temporal consistency of the latent action by setting threshold for when

---

Published as a conference paper at ICLR 2026
additional reasoning is necessary. As shown in Figure 6, small causes the q-policy to generate redundant & values at most timesteps. This behavior is expected: for almost every step, there exists some 2, o <act>> that explains d+++1 more effectively (i.e., yields higher log- leglikklihhod RL reward) than the null action 2 = <act>> However, such behavior offers no advantage over primitive-action NTP, since neither the effective planning horizon nor the decision space is reduced. Moreover, the computational overhead of RA3 can be controlled through c: a large penalty c = 0.2 requires that <think>> improves the expert log likeliiel by at least 0.2 be rewarded, an unlikely event. In this regime, the policy learns to output only <act>> and RA3 has nearly identical computational cost as NTP. In our default setup with average length of less than 6, adding only modest overhead compared to NTP while still providing substantial performance gains. This offers a practical knob to balance performance and compute when scaling RA3.
0.05, RA3 is efficient by generating less than 40% full rollouts
Besides, we construct an additional baseline that performs NTP on syn- thetic rationals. We first fine-tune a separate LLM to generate high- quality reasoning trajectories in the form of single-line comments. It is trained using 366K synthetic Python code snippets from Huang al.
(2024), which are collected by prompting the expert LLM to generate self-contained code along with reasoning comments. Then for the Qwen- 2.5-1.5B model, we perform NTP on training data bootstrapped by this reasoning-synthes model. It can be observed that NTP on synthetic ra- tionals underperforms our method. We attribute this to a key difference in how the reasoning traces are obtained: in RA3, the latents are learned by the model itself via an RL-regularized ELBO objective and are proven to form lower bound on the conditional log- likelihood. As result, the learned abstractions are more amenable further optimization and also have better learnability than external traces.
Figure 7: Comparison with synthetic reasoning NTP baselines.
8 CONCLUSION
An effective mid-training stage should extract from expert demonstrations the action subsets suffi- cient for all tasks and structure the decision space to enable fast, efficient reinforcement learning (RL) convergence. In this paper, we present the first formal analysis of how mid-training design choices influence post-training RI. Our findings highlight two critical roles mid-training should play: (i) mid-training should perform efficient action space pruning, which determines the learned policy prior, and (ii) mid- brid-tranning should accelcrate subsequent RL convergence with shorter effee- tive planning horizons, which reflects the policy's 's potential for improvement through online inter- action. We show that both perspectives favor learning in the space of temporally-exeee exteendd actions over the large space of primitive actions used in next-token prediction. Building on these insights, we propose a novel mid-training algorithm based on the temporal variational bound. This method iteratively uncovers temporally consistent reasoning trajectories via RL and fine-tunes on the result- ing bootstrapped data. Experiments on code generation tasks validate the advantages of reasoning as action abstraction, improving both the generalizability of the initial policy and the effectiveness of subsequent RLVR, in terms of convergence speed and asymptotic performance.
REFERENCES
Alckh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98))1-76. 2021.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint 100.::200..0772. 2021.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in- wild device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37:12461-12444 2024.
10

---

Published as a conference paper at ICLR 2026
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models. https: //ggthit........ com/ ceccoodeppojjec/biigeeeeee htwp://eoeeeeeaaaaaoo-------eeee harness, 2022.
Emma Brunskill and Lihong Li. Sample complexity of multi-tass reinforcement learning. arXiv preprint arXiv: arXiv:1309.66. 609.66999...,,. 2013.
Emma Brunskill and Lihong Li. Pac-inspired option discovery in lifelong reinforcement learning. In International conference machine learning, pp. 316-324. PMLR, 2014.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. Comments as natural logic pivots: Improve code generation via comment perspective. arXiv preprint arXiv:240..0754, 2024.
Qingxiu Dong, Li Dong, Yao Tang, Tianzhu Ye, Yutao Sun, Zhifang Sui, and Furu Wei. Reinforce- ment pre-training. arXiv preprint arXiv:2506.08007 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505..4228, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12946 2025.
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXhv:2411.04405, 2024.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.00 07994, 2024.
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centric dialog training via offline rein- forcement learning. arXiv preprint arXiv:2010.05848, 2020.
Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in reinforcement learning. Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems- Volume pp. 299-306 2008
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu, Numinamath. wowpppoojettt-mmiiaa. co
(https: /ttphbbttb/btt///////////tttthtt cow aimo petoo-prorrss--ppe zeeiiee/bbee wwooooominni report/ numi namiiaaddae . pdf), 2024.
2024.
Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint 23X2:2311.11852, 2023.
11

---

Published as a conference paper at ICLR 2026
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:2155582215555 2023.
Mingjie Liu, Shizhe Diao, Jian Hu, Ximing Lu, Xin Dong, Hao Zhang, Alexander Bukharin, Shaokun Zhang, Jiaqi Zeng, Makesh Narsimhan Sreedhar, et al. Scaling up rl: Unlocking di- verse reasoning in Ilms via prolonged training. arXiv preprint arXi:2500.122577 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0510/ 2017.
Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at o3-mini level, 2025. Notion Blog.
Justus Mattern, Sami Jaghouar, Manveer Basra, Jannik Straube, Matthew Di Ferrante, Felix Gabriel, Jack Min Ong, Vincent Weisser, and Johannes Hagemann. Synthetic-I: Two mil- lion collaboratively generated reasoning traces from deepseek-rl, 2025. URL https: //www. primeint inte lect.ee a / / bl log/ ssyntheticc hetteettttiteeee -icc--reeeess.
Khanh Nguyen, Hal Daumé III, and Jordan BradeGraber. Reinforcement learning for bandit neural machine translation with simulated human feedback. arXiv preprint arXiv:1707.074003 2017.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744 2022.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization, arXiv preprint 1rXiv:1705.04304, 2017.
Doina Precup. Temporal abstraction in reinforcement learning. University of Massachusetts Amherst, 2000.
Martin I. Puterman. Markov decision processes: Discrete stochastic dynamic programming, 1994.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Chris- tian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01241,, 2022.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36:59708-59721 2023.
Yangjun Ruan, Neil Band, Chris Maddison, and Tatsunori Hashimoto. Reasoning learn from latent thoughts. arXiv preprint arXiv:2503. nate:2503./8866 18866, 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Rumxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Liangcai Su, Zhen Zhang, Guangyu Li, Zhuo Chen, Chenxi Wang, Maojia Song, Xinyu Wang, Kuan Li, Jialong Wu, Xuanzhong Chen, et al. Scaling agents via continual pre-training. arXiv preprint arXiv:2509.13334 2025.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211. 1999.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534. 2025.
12

---

Published as a conference paper at ICLR 2026
Haozhe Wang. Haoran Que, Qixin Xu, Minghao Liu, Wangchunshu Zhou, Jiazhan Feng, Wanjun Zhong, Wei Ye, Tong Yang, Wenhao Huang, et al. Reverse- engeieeee reasoning for open-ended generation. arXiv preprint arXi::2509..6111 25090066,,, 2025a.
Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, al. Opencua: Open foundations for computer-use agents. arXiv preprint arXiv:2508.09122 09123, 2025b.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512, 2025c.
Haoran Xu, Baolin Peng, Hany Awadalla, Dongdong Chen, Yen-Chun Chen, Mei Gao, Young Jin Kim, Yunsheng Li, Liliang Ren, Yelong Shen, et al. Phi-4-mini-reasooiing: Exploring the limits of small reasoning language models in math. arXiv preprint arXiv:2504.21233, 2025.
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04455 2024.
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Sim- pletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning. arXiv preprint arXiv:2509.02479 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.093388 2025.
Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. arXiv preprint arXiv:2502.01718, 2025a.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar Arxvv:2503. 02.5503..8899 18892, 2025b.
Shenao Zhang. Li Shen, and Lei Han. Learning meta representations for agents in multi-agent reinforcement learning. arXiv preprint arXiv:2108, 12988, 2021.
Shenao Zhang, Boyi Liu, Zhaoran Wang, and Tuo Zhao. Model-based reparameterizzaion policy gradient methods: Theory and practical algorithms. Advances in Neural Information Processing Systems, 36:68391-68419, 2023.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs, Advances in neural information processing systems, 37: 62557-62583, 2024.
Han Zhong, Yutong Yin, Shenao Zhang, Xiaojun Xu, Yuanxin Liu, Yifei Zuo, Zhihan Liu, Boyi Liu, Sirui Zheng, Hongyi Guo, et Brite: Bootstrapping reinforced thinking process to enhance language model reasoning. arXiv preprint arXiv:2501. 18858, 2025.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.2144 2025a.
Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn agents on collaborative reasoning tasks. arXiv preprint arXiv:2503.1547. 2025b.
13

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1 PROOF OF LEMMA 3.2
Proof. Since the policy æ selects actions from 2, it an admissible policy in both M and M M Since the dynamics, rewards, and discount factors are identical for M and Mg, we have VM Mg
The results then follow from basic algebra.
A.2 PROOF OF THEOREM 3.4
Proof. Before the proof, we first define the subset of suboptimal actions.
Definition A.1 (Suboptimal Action Subset). Z' is (e, -euubosull if Ems-pso) (AMM,, (A(M,))))) 21) >
<)] 2 a.
For an (e,d)- action subset Z', according Definition A.I, we know that (M, Z') 2 c
Pr
M-n(M)
Therefore, for a randomly drawn task M p(M), the probability that Z' is not pruned away is no more than 1 a.
With |Dgl independent expert demonstrations mid mid-training, we reject Z' if VM (5o) Mig (so) + e for any i E [1. [Pell The probability that Z' is not pruned away during mid mid-training
thus no more than (1 1 o) Pel, We denote it as
p(survive (2)) S (1 - o) Pell S e ADRR
where the second inequality holds since + x S ex for all real x.
Let B = { c 2 [ } be the set of all |2|-tegtth subsets. We care about the event that there exist some bad subsets of actions that not pruned away, i.e., survive:
p U survive(Z)) (2))
p(survive 22)) Z'eB
Σ
| 2
-P = Pol
Z'EB
where the inequalities hold by applying union bound over all candidate subsets.
Plugging in |Del= = O( log( /) gives us
U Z
survive(Z') < 8,
ie., all the (c, action subsets Z' are pruned away from 2 with probability least 1 5.
In other words, any action subset 2 after pruning is not (uubossuul with high probability,
ie., it satisfies E MM~PLM) (MM, 2) > c)] < a, or equivalently (MMMMM))I(AMMMM, 2) $ c) 2
1- a, with probability at least 8.
Since the rewards are in [0, 1], we know that (s0) I En 1/(1- Thus, 2), 1/(1 7)).
Therefore, with probability at least 1 - 5, the pruning error [Mmptu)|A(M.2) s <11 o) +
a/(1- >1
A.3 PROOF OF THEOREM 3.5
Proof. We first define the value iteration operator T in the temporally extended action space as
(s)
= max Ey,s R + V('s) s 112
2E2
14

---

Published as a conference paper at ICLR 2026
where p(s'|s, 2) E, Appss' = j),, p(s', j|s, is the joint probability of transitioning to s' in steps after taking action z at s, and RG = Eim 75th Rs.
For any functions Vi, V/2 and any state have
(TV)) (T) max E R V'()(s) 2E2
mmx
Br,s' Rr + 7 V2(5') |,,2 =E2
S max #E2
V8') V2(s') s,2],
where the inequality holds since max f max g S max(f g) for any f.g.
Let alo be the sup norm on functions V S + R. Then
(TV))(s) (TV))(s) S max SE
E [[>7]
[[V Velloo] S7 [V V2ll
Since the above inequality holds for all E S, taking the supremum over 8 gives us |[TVi - oV2 S 7 | V2lloo Therefore, is 7-contraction on (1-.11-11.o)
By Banach's 's fixed-point theorem, T has unique fixed point V and value iteration Vn+1 = TV2 converges geometrically:
VV Vo S 11Vo V* 5 Rmax
where the last inequality holds since Vo = and the maximum value satisfies Vinax = E. Rmax
Rmux/ (1 7).
In order to get ||VN Voo I c, we obtain
Rmax log
N
1
c(1 - -))
A.4 PROOF OF THEOREM 4.1
Proof. For the maximum likelihood objective, after introducing the sequence 20:T of latents, have
log p(aor (0.T) = log Σ
plao:T, 20:T
(0:T)
#0:7
log Σ
plao.T, 70.r
(S.T))
q (2p.r
Sp:7)
q (=0.T
80:T)
#0:2
p( 20:r S0:T)
logg E ((007))))
q(20:T S0:T)
p(an.T, 20.T 80.T)
E 0o
log
(A.1)
( (0:T)
where the last inequality follows from Jensen's inequality. From the probabilistic graphical model, we obtain that
T
((zor1 so:r) = 9(20 So:T)
II
g(2s
s0:T, 203-11
t=1
=((0 (o) II
st, 20:-11)
(A.2)
[=1
15

---

Published as a conference paper at ICLR 2026
Besides, we have from the Bayes rule that
p(en:r, 2orr 80:T) = p(ao; 20 80:T) II p(z,, ae P0:t-; 00:t-1)
= =paa, 20 (0:T) II p(zz 00 )pp(a)) 80:7; Poct, (0:t-1)
T
= =p(ao: zo (0) II p(zz S2, 20.0 ppp(ae I se; zx)
p(zo so) II
p(ze
8t, 20.-1) II
plas I st, ze).
(A.3)
1=1
t=
Plugging (A.2) and (A.3) into (A.1) gives us
p(z
80) II p(ze 8t;, 20:
I TT-nppaa p(ax | 8c, zt)
log p(aor
80:T) 2 (
log
<(z0 80) 11 ((zr
8t, 20:e-1) 1)
7(2 8t, 20:1-1)
E (s0
logp(a:
/1 21) - llog
p(ec s2, 21)
T
ser Re-1))
log p(a, 4)) - DEL (g(ze Se, 20:111
||
p(z:
where p(2 18,, 2-) the prior distribution of the latent 2z.
B STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
We use LLMs only to polish the paper, such as improving clarity and grammar, without altering its substance original ideas.
16


Reviews:
[{"review_id": "5XX5T0iBm8", "reviewer": "Reviewer_vdRz", "summary": "This paper explores the concept of mid-training for large language models (LLMs) in reinforcement learning (RL), proposing that an effective mid-training stage can both learn a strong policy prior and enable faster adaptation through online interactions. The authors provide theoretical insights into how mid-training shapes post-training by pruning the action space and shortening the planning horizon, which accelerates RL convergence. They also investigate the role of temporal abstractions in compressing the action set and reducing decision horizons, improving regret minimization. Based on these insights, the paper introduces Reasoning as Action Abstractions (RA3), a scalable mid-training algorithm that iteratively discovers temporally-consistent latent structures via RL and fine-tunes on the bootstrapped data. Experiments on code generation tasks demonstrate that RA3 improves performance across multiple benchmarks and base models, showing faster convergence and higher asymptotic performance.", "questions": "1. Could you elaborate on the potential challenges or limitations of applying RA3 to larger-scale models? For example, does the method scale efficiently with model size, or are there bottlenecks to address?\n2. Some of the theoretical results (e.g., regarding temporal abstractions and regret minimization) are compelling but not fully intuitive. Could you provide additional clarification or examples to make these results more accessible?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The idea of mid-training is novel, and the theoretical insights about pruning the action space and accelerating RL convergence are convincing.\n2. Introducing temporal abstractions into RL training for LLMs is a creative and well-motivated approach.\n3. The RA3 algorithm is well-designed, conceptually simple, and appears easy to reproduce, making it practically useful for further research or application.\n4. The paper presents extensive experiments and ablation studies, which provide strong empirical evidence supporting the effectiveness of RA3.", "weaknesses": "1. The paper lacks experiments on larger-scale models, which could provide more insights into the scalability of the proposed approach.\n2. There is insufficient discussion on the training efficiency of RA3, particularly in terms of computational cost or resource requirements compared to other methods.\n3. The presentation of the paper could be improved. Some sections, especially the theoretical parts, are difficult to follow due to unclear explanations or insufficient detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8NqfV1eypb", "reviewer": "Reviewer_6Sot", "summary": "The paper studies mid-training for LLMs and argues that effective mid-training should (i) prune the decision space to learn a strong policy prior and (ii) shorten the effective planning horizon so post-training RL converges faster. The authors formalize both effects: they show sample-complexity bounds for pruning “bad” action subsets and prove that temporally-extended actions reduce the RL horizon, improving convergence rates. Building on this, they propose RA3 (Reasoning as Action Abstractions)—a scalable mid-training algorithm that introduces a temporal ELBO for next-token prediction, optimized via EM: an E-step uses self-supervised RL to discover a sequence of temporally consistent latent actions (the “reasoning”/intent), and an M-step fine-tunes on the bootstrapped data.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The paper gives a regret decomposition and connects mid-training to post-training RL via pruning error and RL error, with a sample-complexity result (bad-subset pruning) and a convergence-rate bound favoring temporally extended actions—a useful, general framing beyond the specific algorithm.\n\n2. The latent persistence reduces rollout frequency; the penalty turns RA3 into NTP in the limit, giving a knob for cost control. The paper also provides a concrete training recipe (group baselines, asynchronous rollouts).\n\n3. With GRPO RLVR, RA3 starts from a better prior and learns faster, achieving higher asymptotic accuracy on HumanEval+, MBPP+, LiveCodeBench, and Codeforces; ablations on the penalty c illustrate the compute/quality trade-off.", "weaknesses": "1. All experiments are in Python code generation; claims about “reasoning” and “agents” are theoretically motivated but empirically untested outside code. Extension to math, tool-use, or GUI agents would strengthen generality.\n\n2. RA3 introduces several knobs (penalty c, latent prior α/format reward, rollout group size, max latent length). The ablation on c is helpful, but broader robustness/stability sweeps (and wall-clock cost vs. NTP/RLVR) are limited.\n\n3. Baselines emphasize NTP and post-training RLVR; fewer head-to-head comparisons against other mid-training methods that distill reasoning (e.g., synthetic CoT mid-training) leave some ambiguity about when RA3’s RL-discovered latents outperform distilled traces at similar cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pRSMEIUMAa", "reviewer": "Reviewer_T6MS", "summary": "This paper presents RA3 (Reasoning as Action Abstractions), a mid-training algorithm for large language models that learns temporally consistent latent actions. The authors provide the theoretical analysis explaining how mid-training shapes post-training reinforcement learning (RL), identifying two key mechanisms: (1) efficient pruning of the decision space and (2) shortened effective planning horizons. Empirically, RA3 improves code generation accuracy across multiple models and datasets.", "questions": "My main concern is the theoretical assumption and experimental comparison. Please refer to the Weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work provides the formal framework linking mid-training design to post-training RL performance, supported by clear theorems and lemmas.\n2. The proposed method achieves consistent improvements on HumanEval, MBPP, LiveCodeBench, and Codeforces benchmarks, using both Qwen and Llama models.\n3. The whole paper is clear with good experimental presentation.", "weaknesses": "1. The theoretical assumptions (finite action subsets, independent expert demonstrations) could be idealized and may not hold in practical large-scale LLM training.\n2. Experimental comparisons are limited mainly to NTP, more baselines like BRiTE are recommended to be included.\n3. The authors are recommended to include code or dataset for reproduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)