TROLL: Trust Regions Improve Reinforcement Learning for Large Language Models
Authors: Philipp Becker
Niklas Freymuth
Serge Thilges
Fabian Otto
Gerhard Neumann
Venue: ICLR 2026 Oral (2026)
Keywords: RL from verifiable rewards
Finetuning LLMs
Trust Regions
Primary area: foundation or frontier models, including LLMs
TL;DR: Replacing PPO's clipping objective with more principled trust regions improves RL from verifiable rewards.

Abstract: Reinforcement Learning (RL) with PPO-like clip objectives has become the standard choice for reward-based fine-tuning of large language models (LLMs). 
Although recent work has explored improved estimators of advantages and normalization, the clipping mechanism itself has remained untouched.
Originally introduced as a proxy for principled KL-based trust regions, clipping is a crude approximation that often causes unstable updates and suboptimal performance. 
We replace the clip objective with a novel discrete differentiable trust region projection, which provides principled token-level KL constraints.
The projection operates on a sparse subset of the model’s most important token logits to balance computational cost and projection effectiveness. 
Our approach, Trust Region Optimization for Large Language Models (TROLL), serves as a direct replacement for PPO-like clipping during training and does not alter the model’s inference behavior.
Across mathematical reasoning and code generation tasks, model families, as well as advantage-estimation methods, TROLL consistently outperforms PPO-like clipping in terms of training speed, stability, and final success rates.

Published as a conference paper at ICLR 2026
TROLL: TRUST REGIONS IMPROVE REINFORCEMENT LEARNING FOR LARGE LANGUAGE MODELS
Philipp Becker'; Niklas Freymuth ' Serge Thilges', Fabian Otto Gerhard Neumann'
Karlsruhe Institute of Technology, Microsofft Research
ABSTRACT
Reinforcement Learning (RL) with PPO-like clip objectives has become the stan- dard choice for reward- based fine- tuning of large language models (LLMs). Al- though recent work has explored improved estimators of advantages and normal- ization, the clipping mechanism itself has remained untouched. Originally intro- duced proxy for principled KL.-based trust regions, clipping is crude ap- proximation that often causes unstable updates and suboptimal performance. We replace the clip objective with a novel discrete differentiable trust region projec- tion, which provides principled token-level KL constraints. The projection op- erates on sparse subset of the model's most important token logits to balance computational cost and projection effectiveness. Our approach, Trust Region Op- timization for Large Language models (TROLL), serves as a direct replacement for PPO-like clipping during training and does not alter the model's inference behavior. Across mathematical reasoning and code generation tasks, model fam- ilies, well aedvantage-estimatie methods, TROLL consistently outperforms PPO-like clipping in terms of training speed, stability, and final success rates.
INTRODUCTION
Reinforcement Learning (RI) has become the standard approach for fine-tuning and aligning Large Language Models (LLMs) with preferences verifiable rewards. For such post-training, the algo- rithms of choice are predominantly Proximal Policy Optimization (PPO)-style policy gradient ap- proaches (Schulman et al., 2017). They first estimate advantage function and then update the pol- icy using an importance eepoorancee-eiigheed objective, clipped to prevent the ratio between new and old poli- cies from deviating too much. Recent approaches such GRPO (Shao al., 2024), Dr.GRPO (Liu et al., 2025), GSPO (Zheng et al., 2025), and REINFORCE+ (Hu et al., 2025) improve the esti- mation of advantages and normalization, resulting in significant advances in RL for LLMss Yet, all these approaches rely on PPO's clipping- basedd policy update mechanism.
Clipping is originally motivated via trust region methods (Schulman al., 2015a; 2017), which provide a principled way to stabilize policy updates by constraining the KL divergence (Kullback & Leibler, 1951) between successive policies during training (Kakade & Langford, 2002; Peters al., 2010). While theoretically sound, implementing such trust regions is often costly, in particular with modern LLMs whose vocabularies and thus output distributions can exceed 100 000 entries (Yang et al., 2025a;b). PPO instead clips the importance ratio to sidesteps this challenge. While empiri- cally successful, it is a crude approximation of the underlying trust region (Wang et al., 2019; 2020). Crucially, it may lead to unstable optimization, poorly calibrated updates, well sensitivity to hyperparameters and implementation details, which often culminate in suboptimal performance (En- gstrom al., 2020; Andrychowicz et al., 2021; Otto al., 2021; Huang et al., 2022).
As a remedy. we introduce Trust Region Optimization for Large Language models (TROLL) m, a differentiable trust region projection approach that directly enforces token-level KL constraints be- tween discrete distributions TROLI formulates a convex optimization problem that acts as a direct replacement to PPO-like clipping objectives. For each token, TROLI projects the output distribution of the new, updated policy onto KL-trust region around the old policy that was used to sample the sequence. This process ensures that the new and old policies only differ by given bound, preventing the policy update from diverging or collapsing. The left of Figure I shows a 3 3-dimensional example
"Equal contribution. Author order was decided by a fair coin flip. Project page, code, and datasets available wttttt:t////nn /tttpa://nikki fremmuth thub. io/trol1/

---

Published as a conference paper at ICLR 2026
Told (ot q. occe)
MATH Eval
Eurus Code Eval
(o q, oct))
Mean Success Rate
0.6
0.6
(o q, 023)
0.5
0.55
0.4
TROLL
0.5
Clip
0.3
0
40
80
100
Wallclock Time (h)
Wallclock Time (h)
Trust Region Optimization for Large Language models (TROLL) overview. (Left) Ex- ample of a 3-token distribution (cat, troll, hamster). The old policy favors the troll, while the new policy shifts toward the hamster. The projection ensures that the updated policy stays within the trust region (circle). (Right) TROLL yields clear performance gains over PPO-like clipping (CLIP) on mathematical reasoning and code generation tasks, as shown for Qwen3- 14B trained with GRPO.
Figure 1:
where the old policy prefers the "troll" token, the new policy leans towards the "hamster" token, and the trust region constrains the update to keep the new policy close to the old one. The direction of the projection can be computed in closed form, while its step size is the solution convex Lagrangian dual problem. The projection leaves the new distribution unchanged if it already falls within the trust region, and can be efficiently solved parallel in practice. TROLL enables differentiation through the solution of the projection problem using the OptNet framework (Amos & Kolter, 2017), which introduces only negligible computational overhead. This process maintains gradient information even for updates that are constrained by the trust region, in contrast to PPO-like clipping, which cuts gradients for tokens whose ratios exceed the clipping threshold. Further, the trust region is only effective during training and provides zero additional overhead during model inference. To incentivize the model to stay within the trust region for successive update steps, we additionally add simple regression term between projected and unprojected tokens.
dieeeniinionll
Applying TROLL directly to LLMs is computationally infeasible, since the large token vocabulary causes prohibitively expensive projections and memory overhead. However, natural language con- tinuations and similarly LLM predictions are generally characterized by very few high-probability tokens (Zipf, 1949; Piantadosi, 2014; Kunstner et al., 2024; Duan et al., 2024; Ren & Sutherland, 2024). This property enables sparsification scheme that discards unlikely tokens, retaining only the most relevant ones. On average, as few as 5-10 10 tokens generally preserve more than 99.99 99.999% of the distribution's 's probability mass. We thus modify TROLI to handle sparse distributions, allowing to scale to modern L.MSs and act as a drop-in replacement for PPO-style clipping.
We evaluate TROLL on Reinforcement Learning from Verifiable Rewards (RLVR) problems, fo- cusing on mathematical reasoning. Using TROLL for GRPO (Shao et al., 2024) with models from the Qwen3 (Yang et al., 2025a) and Qwen2.5 (Yang et al., 2025b) families on DAPO-Math (Yu et al., 2025) yields substantial improvements in terms of training stability and success rate for eval- uations based on the number of updates as well wall clock time. Concretely, TROLL improves roughly 3-10 percentage points, or 5-15% relative over clipping for math-based reasoning. To assess robustness across algorithmic variants, Dr.GRPO (Liu et al., 2025), GSPO (Zheng et al., 2025), and REINFORCE++ (Hu et al., 2025). Across methods, TROLL consistently enables faster learning and improves success rates, indicat- ing that its benefits are independent of the underlying advantage estimation method. We demon- strate similar improvements across additional math datasets, namely GSM8K (Cobbe al., 2021a) and Burus-2-RL-Mann (Cui et al., 2025a), as well as models from the LLaMA 3 (Grattafiori et al., 2024). SmoILM3 (Bakouch et al., 2025), and Apertus (Hernández-Cane enables stable training for some models where while clipping fails entirely. Finally, we con- sider code generation using the 2-RL----RI--orl (Cui et al., 2025a) dataset, showing improvements of 7-8 percentage points, or 18-30% relative over clipping across Qwen3 model sizes.
experiment with PPO (Schulman al., 2017),
al., 2025) families. Here,
To summarize, we i) derive TROLL, a fully differentiable, principled trust region projection for dis- crete distributions that enforces per-token KL constraints, ii) introduce sparsification scheme that lets the projection scale to LLM-scale vocabularies and implement as a drop-in replacement for PPO-style heuristic clipping across RL algorithms, iii) demonstrate through experiments spanning different dddvanagg--sstiaatin methods, models and datasets that TROLL consistently improves suc- cess rates and training stability compared to clipping.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Trust Regions in Reinforcement Learning, Information thhorreic trust regions based on the KL di- vergence (Kullback & Leibler. 1951) are known to stabilize RL in classical (Kakade, 2001; Kakade & Langford, 2002; Peters et al., 2010; Abdolmaleki et al., 2015; Akrour et al., 2018) as well as modern deep learning settings (Schulman et al., 2015a; 2017). Trust Region Policy Optimization (TRPO) (Schulman et al., 2015a) limits the KL-divergence by solving constrained optimization problem. PPO (Schulman et al., 2017) simplifies this approach to a first-order method, using clipped surrogate objective to make policy optimization in RL scalable (Akkaya et al., 2019; Berner et al., 2019; Baker al., 2020). However, PPO's trust region is less principled and very sensitive to practical implementation details (Engstrom et al., 2020; Andrychowicz et al., 2021: Huang et al., 2022). Several methods dynamically adapt PPO's clipping bounds (Wang et al., 2019; Xi et al., 2025), but still require clipping. Alternative methods (Abdolmaleki et al., 2018: Song et al., 2020) rely on regularizing the expected KL. between subsequent policies. In contrast, TROLL enforces act, token-wise trust regions through a differentiable KL projection, combined with adaptive strate- gies that control the trust region size.
Trust Region Projections. A different line of work (Otto et al., 2021; ; Akrour et al., 2019) enforces trust regions using projection- based methods. These approaches first compute the policy as usual, and then project it back into a feasible set defined by a trust region constraint, In particular, Otto et al. (2021) compute exact trust region projections for each state when using Gaussian policies, leading to improvements in high-dimensional action spaces (Celik et al., 2024; Li et al., 2024a; Hoang et al., 2025; Otto et al., 2025). Similar to us, they rely on Lagrangian optimization (Boyd & Vandenberghe, 2004) and implicit differentiation (Amos& & Kolter, 2017), but focus Gaussian distributions and continuous control tasks (Brockman et al., 2016). TROLL instead proposes differentiable projections for categorical distributions and provides an efficient implementation via a sparsification scheme. This combination lets TROLL scale to modern-day LLMs while preserving the stability of classical trust region methods.
Reinforcement Learning for LLMs. Recently, RL has become a key tool in the post-training stage of LLMs. Popular frameworks include RL from human feedback (RLHF) (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022) for LLM alignment and RLVR (Lu- ong al., 2024; Lambert et al., 2024) for reasoning tasks such as mathematical problem solving or code generation. In RLHF, preference-
methods often optimize a reward while remaining close reference policy via an added expected KL penalty. This reference policy is typically the supervised fine-tuning model (Stiennon et al., 2020; Ouyang et al., 2022). For example, Di- rect Preference Optimization (DPO) (Rafailov et al., 2023) optimizes this objective in closed form on preference data, thereby avoiding policy rollouts. TROLL does not constrain the updates to a fixed reference, instead enforcing proximity to the previous step's policy to stabilize on-policy op- timization. While this work focuses on RLVR for LLMs, TROLL can also be applied to other LLM post-training settings, and more generally, discrete RL tasks.
Several recent methods avoid PPO's importance sampling and thus clipping entirely. These meth- ods combine group rewards with a standard policy gradient objective (Chu et al., 2025), utilize an additional value network (Richemond et al., 2024), use Q-learning (Clavier al., 2025), employ a contrastive loss to enable off-policy learning (Flet-Berliac et al., 2024; Cohere et al., 2025). How- ever, while these approaches are promising, current research on RLVR still often relies on advantage estimation. While not the focus of this work, we briefly show that one such non-clipping-basse ap- proach also benefits from TROLL, since its trust regions directly act on the LLM's policy.
Advantage Estimation for LLM -rainTrainng. PPO (Schulman et al., 2017) requires an expen- sive explicit value model. Especially for RLVR, where evaluating multiple rollouts per input is comparatively cheap, sample-based advantage estimation methods like Group-Relative Policy Op- timization (GRPO) (Shao et al., 2024) have become popular alternatives. Variants include GRPO Done Right (Dr.RRPO) (Liu et al., 2025), which addresses GRPO's optimization biases that favor longer responses, and REINFORCE+ (Hu et al., 2025), which uses Global Advantage Normaliza- tion (Andrychowicz et al., 2021) to improve training stability. Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025) extends importance ratios from token to sequence level to improve update stability. However, all these methods still depend on PPO-style clipping stabilize policy updates. We introduce TROLL as a more principled drop-in replacement, applicable regardless of how advantages are computed and compatible with all the approaches above.
3

---

Published as a conference paper at ICLR 2026
Sparsify
Trust Region Projection
Re
mrg KL (IT || #e)
(To STold)
Rollout Buffer
Current Token (Sparse) Probs
Context Sequence
oct
Figure 2: TROLL replaces PPO-like clipping with differentiable trust region projection approach. Given the current output distribution of an LLM (o q. Ozt). and the distribution that was used collect the sequence (ot q, Oct) for the replay buffer, TROLL enforces a per-token trust region by solving Equation 3 for each token. The resulting distribution is then used in the RL objective (Equation 5) to update the LLM parameters. To scale this approach the vocabulary size of modern LLMs, TROLL uses sparsification approach, which allows working small subset of logits while retaining most of the distribution's mass.
TRUST REGION OPTIMIZATION FOR LARGE LANGUAGE MODELS
Reinforcement Learning (RL) for Large Language Models (LLMs) finetunes the LLM's parameters, 0, using policy ratio objectives (Schulman et al., 2015a) of the form
o mo(0r q 0c)) A(or, q. o<)) (ota q, oct)
Tratio (0) Bo (olg
Σ
where (o; q, oz)) is the probability of the sampled token under the current LLM policy. Here, (ot q, Oxx)) is the token's probability under the LLM policy that was used for data collection in the previous iteration. The context sequence consists of the prompt q and prior response tokens Oct. The advantage estimate A, = A(o:, q, O<t) measures if a token is better or worse than the average behavior. Thus, maximizing (8) increases the probability of good responses while decreasing the probability of bad ones. In practice, we estimate A, using an explicit value model as in PPO (Schulman et al., 2017) or purely sample-based
in GRPO and variants (Shao al., 2024: Liu et al., 2025; Zheng et al., 2025). For such policy ratio objectives, stable and effective optimization requires keeping (o; q, Oct) and (o q. Oc)) close, so that the importance ratio To, He (o, /. R/ulo q.0).) remains close to one (Schulman al., 2015a; 2017). PPO attempts to maintain this proximity by clipping the ratio,
o min (r,4,; clip (r,, "ppo: 1 <pp) A,)
(0) = (F
(2)
However, this clipping is crude approximation of the underlying trust region principle. While prevents large updates, this approach is purely heuristic and suppresses gradients when the ratio falls outside the clipping range, resulting in unstable and inefficient learning. In contrast, token-wiss KL-based constraints offer a principled approach to limit the change between successive policies. Our method, Trust Region Optimization for Large Language models (TROLL)) implements these constraints using differentiable trust region projections (Otto et al., 2021) as a drop-in replacement for the PPO-like clipping.
3.1 DISCRETE DIFFERENTIABLE TRUST REGION PROJECTIONS
Trust Region Projection. Our projection formally solves the convex optimization problem KL (fte(o qq. 0..0 ) (o q, ozz))
no (o q, 0<1)
arg min
(3)
*o q.0cz)
KL (ft q. oze) || (or q, 0x)) Sc
4

---

Published as a conference paper at ICLR 2026
for every output token or². Intuitively, the projection finds the policy distribution closest to the cur- rent LLM policy (o | q. 2c1) while remaining within an e-bound of the policy used to collect the data for the current iteration (or (.00ll.0cc)) o.t). The solution to this optimization problem is derived in Appendix A. and given as
11" log Tota q, Oct) + log (o q, Ort))
mofor q: Oct)) x exp
(4)
17" +
which is a geometric interpolation between the logits of # | q. Oct) and Tota(o q, Qct). Here, n" acts as a step size controlling how far the projection moves the new policy to the old one. For each token, we can compute the optimal 7f" which enforces the trust region constraint by solving the convex dual of Equation 3. This dual is scalar optimization problem, which we derive and state in Appendix A.2, and can be solved with sufficient accuracy using a few iterations of ternary, or more generally, n-ary, bracketing. Furthermore, projecting only necessary if the trust region bound is violated, which is only the case for very few, but highly relevant tokens. Thus, we can avoid it for the vast majority of tokens by filtering them beforehand.
Policy Updates with Trust Region Projections. After projection, the policy To(or q. 0ct) satisfies the trust region constraint and can be used to optimize the LLM parameters 0, via an update similar to Equation 1. However, the LLM output a Q. occ)) may still deviate arbitrarily from the old policy, complicating inference and successive updates. To avoid this, we follow Otto al. (2021)
and address this by regressing the LLM output (o q, occ)) toward its projection Te(o; q. oz1), resulting in an objective
Fron (0)
(5)
[o]
E (aaaloll)))
To(0: q. 0c)) 0 (o q, occ))
1
A.) aKL (Fo((or) / (1,0z) II [To(Or g, (x)))))))
where denotes the stop gradient operator and is a usee--peed regression weight. Crucially, the projected policy (o; q, Oct) is used to compute the ratios and a regression target for the LLM output (o; q, Oct)). For the regression, we stop the gradients through To(o; q. 0ctr) so that the LLM policy (0 q, oct)) is pulled towards the output of the projection mo (0(0 q. Oct), not the other way around. The regression term only affects projected tokens and still allows policy updates up to the KL bound, making the approach robust to the choice of a. We thus set to a in all experiments for simplicity. Notably, our objective in Equation 5 makes assumption on the advantages A,. Thus, TROLL can be directly applied to a variety of existing advantage estimation methods, including PPO, GRPO, Dr.GRPO, and GSPO. Algorithm I provides pseudocode.
Making Trust Region Projections Differentiable. To propagate gradients through our projection, can rely on autograd tools such PyTorch (Paszke et al., 2019), except for the numerical op- timization of the dual. Formally, the optimal 17" a function of the LLM policy #o q. o2))- To obtain fully differentiable projection, we need the gradient any an /oRa(o. q.0 which describes how the LLM output influences the optimal step size. We follow the OptNet framework (Amos & Kolter, 2017) and differentiate the KKT conditions (Karush, 1939; Boyd & Vandenberghe, 2004) of the dual solution via implicit differentiation (Dontchev & Rockafellar, 2009) and differential calcu- lus (Magnus& Neudecker, 1989). This approach lets us compute closed-form gradient instead of differentiating through the numerical optimization. Appendix A.3 provides derivations. Appendix B provides code snippets and a schematic of our projection's 's compute graph, including gradients.
3.2 SPARSE AND EFFICIENT REPRESENTATION OF TOKEN DISTRIBUTIONS
Naively implementing TROLL requires storing and projecting the full vocabulary distribution for each token. Using Qwen3's 's tokenizer (Yang et al., 2025b) as an example, this results in an overhead of 151 936 logits per token, which is prohibitively expensive. To address this issue, we sparsify both the distributions and the implementation of the projection. We greedily select the K tokens with the largest probability mass, sort them by their mass, and then only retain those needed to cover cumulative mass of 1-14 1-5. 5. We additionally always keep the token actually selected by the LLM policy
(or q. Occe) must also remain valid distribution, i.c., Σ,, To(o: q, aoe) 1 and (ot q, oc3) 2 0 for all oc- We omit these constraints for brevity and elaborate Appendix A.
5

---

Published as a conference paper at ICLR 2026
Algorithm Optimizing LLMs with TROLL
1: LLM policy (or oct), Training data D, KL Bound
2: for step s=1... do
Sample batch questions qrDD
4: Sample responses (o ~ (or q, Oct)) using the LLM policy.
5:
Set reference policy Tol= Sparsify and save corresponding logits (ou (0: ocs)) for minibatch b do
6:
7:
Compute sparsify current logits ie (o. q, oct))
9:
Estimate advantages A, using any advantage estimation method for response tokens in parallel do
10:
11:
if KL (#o Occ)) (o: q. oct)) S then Sct (0 q, Oct) #o 1.llg.g Ocec
12:
projection needed
13:
else
14:
Compute 'i" by numerically optimizing Equation 13.
15:
Compute To (or g, Oee)) using Equation 4
project to Tau(o; qg ocee)
16:
end if end for
18:
Update LLM parameters using Equation
19:
end for
20:
end for
to ensure gradient information for this token. The top-K K filtering both upper bounds the number of kept logits, acting as a fail-safe to prevent excessive memory usage for high-entropy predictions, and allows for efficient sorting of relevant tokens. Since pre-trained LLMs generally have very low perplexity (Kaplan et al., 2020; Hoffmann et al., 2022; Ruan et al., 2024). this thresholding allows maintain almost all of the probability mass of the logit distribution with very few average kept logits. Empirically, using K-64 and 8=10-5 usually allows us to keep 99.999% of probability mass with only 10 average tokens for most of the tested model and task combinations. Finally, for the discarded tokens, we cannot assume probability of truly 0 but have to use a small default mass Pa> maintain well-behaved distributions. After sparsification, we re-normalize the kept tokens with Equation 21, taking into account the number of non-kept tokens and default mass. We perform the sparsification chunks of the full generated sequences prevent memory spikes.
While greedily keeping the Theorem A. in Appendix A.4 that it yields best possible KL approximation under mild assumptions. Additionally, under moderate assumptions, the error introduced by sparsification is bound by
tokens is intuitively useful, we additionally show in
KI. (p 4) S -KK (p' || d') + ö glog
(6)
Qmin
where and are arbitrary categorical distributions, p', the corresponding sparsified distributions, Imin ((x;)) denotes reference lower bound and >=1 the renormalization constant. Theorem A.2 provides the proof and demonstrates that, for the hyperparameters used in Qwen3, the error incurred by enforcing the trust region on the sparsified distributions rather than on the full distributions is approximately two orders of magnitude smaller than the bound itself. The sparsification reduces memory and computation cost the point where TROLL only incurs minimal overhead on mod- ern LLMs, making it a practically viable alternative to PPO-like clipping. Further, this overhead constant model size, causing its relative cost to diminish for larger models. Figure shows a schematic overview of TROLL's training, and Section 6 provides additional analysis of the sparsifi- cation and projection behavior in practice.
EXPERIMENTS
Datasets. We evaluate TROLL by finetuning LLMs for mathematical reasoning and code generation using an Reinforcement Learning from Verifiable Rewards (RLVR) setup. DAPO-Math (Yu et al., 2025) consists of 17 thousand math questions and answers that are obtained from web scraping and manual annotation. Appendix F provides example question. We randomly split off 1024 samples to provide an in-distribution evaluation dataset, and use the remaining samples for training. We refer to those sets as DAPO-Eval and DAPO-Train, respectively. Additionally, we follow the evaluation

---

Published as a conference paper at ICLR 2026
DAPO Train
DAPO Eval
MATH Eval
0.8
0.8
0.6
Success Rate
0.6
0.6
0.4
0.4
0.4
0.3
0.2
0.2
0
100
200
300
100
200
300
100
200
300
Train Step
Train Step
Train Step
Eurus-Code Train
Eurus-Code Eval
Clip
TROLL
Success Rate
0.6
0.6
600M
1.7B
0.4
0.4
14B
0.2
0.2
0
100
200
300
0
100
200
300
Train Step
Train Step
Figure 3: Comparison of TROLL (full lines) and Clip (dashed lines) across GRPO-trained Qwen3 models with 600M 14B parameters for DAPO (top) and Eurus-Code (bottom). Full-opacity lines mark smoothed results, while the background shows original values. TROLL consistently boosts training efficiency and final success rates on math-related questions and code generation tasks. These gains translate to different evaluation datasets.
setup of Cui et al. (2025b) and of MATH500 (Hendrycks et al., 2021), AMC, AIME2024 (Li et al., 2024b), AIME 2025, OMNI- MATH (Gao al., 2025), OlympiadBench (He al., 2024), and Minerva (Lewkowyez al., 2022). As in previous work (Cui et al., 2025b), we report the mean of 32 rollouts for the comparatively small AIME2024, AIME2025, and AMC datasets reduce evaluation variance. GSM8K (Cobbe et al., 2021b) contains grade school math problems with final integer answers, consisting of 8.5k training and 1.3k test problems. Finally, we consider the Burus-2-RL data (Cui al., 2025a), which contains mathematical reasoning and code generation tasks. We consider those separately, resulting in Surus--Matl and Eurus-Coue For both datasets, we use the train and validation sets is.
suite of test datasets, which call Math-Eval, comprised
The three mathematical reasoning tasks range from comparatively simple grade school problems to complex math olympiad tasks, and the code generation demands understanding of techniques such as dynamic programming, data structures and amortized analysis. Appendix D.3 provides further details on all datasets and the reward calculations.
Models. We experiment with Qwen3-10.6B, 1.7B, 4B, 8B, 14B) (Yang et al., 2025a), which we use -Instruct (Yang al., 2025b). Furthermore, include both the instruct and non-instruct versions of Llama-3. -88B, Llama-3.2-3B (Grattafiori et al., 2024), and Apertus-SB (Hernández-Can) et al. (2025) and version of Llama fine-tuned on FineMath (HuggingFaceTTB 2025). These mod- els range from 500M to 14B parameters and cover different vocabulary sizes, tokenizers, model architectures, pre-traiie pre-training paradigms, and datasets, as well as initial math capabilities.
in thinking mode, and 0wem225--(..5B,..55..3.3
al., 2025). Finally, we include Smol-LMM-3B Bakouch
Methods. We focus on GRPO (Shao al., 2024) due to its popularity and empirical success. We also include PPO (Schulman et al., 2017), which uses an explicit value model for General- ized Advantage Estimation (Schulman al., 2015b), and three additional GRPO variants, namely Dr.GRPO (Liu et al., 2025), GSPO (Zheng et al., 2025), and REINFORCE+ (RF++) (Hu et al., 2025). All methods differ in their advantage estimation and their normalization of Equation 1, they all rely PPO-like clipping, which makes them amenable to using TROLL. We com- pare each method's 's original clipping- based version to using TROLL projections, denoted as (Clip) and (TROLL), respectively. Finally, method, and GPG (Chu et al., 2025) as a clipping-fir free baseline. For the latter, we compare to the vanilla variant (GPG) and also add the TROLL projection to its policy update (GPG (TROLLL)).
compare BAPO (Xi et al., 2025)
adaptive clipping

---

Published as a conference paper at ICLR 2026
Qwen3-8B
Qwen2.5-7BB-nstrun
GRPO
Dr.RROO
PPO
GSPO
RF++
GRPO
Dr.GRPO
PPO
GSPO
RF++
DAPO Train
Clip
0.667
0.678
0.640
0.000
0.648
0.443
0.467
0.444
0.159
0.429
TROLL
0.721
0.704
0.744
0.736
0.742
0.495
0.513
0.431
0.481
0.486
DAPO
Clip
0.640
0.633
0.602
0.000
0.626
0.323
0.331
0.324
0.093
0.323
Eval.
TROLL
0.691
0.674
0.715
0.706
0.728
0.389
0.389
0.353
0.390
0.380
MATH
Clip
0.541
0.549
0.508
0.000
0.520
0.313
0.317
0.319
0.127
0.311
Eval.
TROLL
0.551
0.546
0.591
0,580
0.578
0.350
0.359
0.349
0.333
0.344
Table I: Final train and evaluation success rates on DAPO for Qwen3-8B and Owwe2.5--BBInntteee methods for different advantage estimation methods for TROLL and Clip. The better approach is marked in blue. TROLL significantly improves over Clip in most cases, and is able successfully train GSPO, where Clip causes divergence and little to no success rates on both models.
Experiment Setup. We base our experiments on the verl repository'. using default parameters and training recipes where applicable. We set the group size for the advantage normalization of all methods to 8. We use a token-level loss aggregation (Yu et al., 2025) for PPO and GRPO, and opt for method-specific loss aggregations for Dr.GRPO and GSPO. We evaluate the test datasets every 10 steps. To improve visibility, we use sliding windows of size 7 and 21 for the train and test evaluations, respectively. while showing the unsmoothed values in the background. Appendix provides additional details, including hyperparameters in Table 3. Appendix D shows all results.
5 RESULTS
Qwen Experiments DAPO-Math. We evaluate models from the Qwen3 and Owen2.5-IInstrue families (Yang et al., 2025b;a) on DAPO (Yu et al., 2025). The top of Figure 3 compares TROLL and the Clip objective for different Qwen3 model sizes optimized with GRPO (Shao al., 2024). TROLL consistently improves training performance, causing more sample-efficient training and improved success rates at convergence for all models. These results directly translate to both evaluation sets, DAPO-Eval and MATH-Eval. Interestingly, the 4B TROLL model almost matches the performance of the 14B Clip one. Figure 7 in Appendix D. shows similar performance trends across Qwen2.5- Instruct model sizes. The right of Figure 1 further compares the runtime of both variants on Qwen3- 14B, showing that TROLL's projections do not incur a significant computational overhead. Finally, Appendix F provides example sequences generated by Qwen3-14B
MA problem.
Table I compares TROLL and Clip results for Qwen3-8B and Qwen2.5-7B-Instee DR.RROO, PPO, GSPO, and RF++ TROLL generally improves success rates by 3-10 percentage points across methods and evaluated datasets. In these experiments, choosing TROLL over Clip is usually more beneficial than selecting any of the considered advantage estimation methods. Table provides results for the individual MATH datasets, while Figure 8 and Figure 9 show full training curves for Qwen3-8B and Owwen2.5-7B-Instruce respectively. Both figures show that GSPO (Clip) eventually diverges during training, while GSPO (TROLL) remains stable and achieves similar suc- cess rates to the other advantage estimation methods.
for GRPO,
Qwen Experiments on Eurus Code. The bottom of Figure 3 shows that TROLL is directly applicable code generation tasks, yielding substantial advantages over Clip for all model sizes. For instance, our Qwen3-1.78 (TROLL) improves over Qwen3-4B (Clip). and significantly outperforms the model of its own size. Specifically, we see improvements of 7-18 percentage points, which translate to a 18 -0% relative gain, when using TROLI instead of Clip.
Additional Model Families. Figure shows various models of different families and sizes on GSM8K, again indicating a clear benefit for TROLL over the Clip objective. Here, models of the Llama family often need significant number of training steps before Clip shows a positive training signal, while TROLL causes the models to start learning much faster. TROLL further enables stable and fast training for Apertus, while Clip fails in some setups. Appendix D.2 provides additional re- sults more models and the GSMBK dataset. We omit evaluations for DAPO with models other than as none matched the performance of Qwen3-1.78 in preliminary Clip experiments.
https: / thub. wwm/vollooooooooooo cww./ ne n
8

---

Published as a conference paper at ICLR 2026
Apertus-8B Instr.
Llama3.2-3B
SmoILM-3B
Clip
TROLL
GSM8K
Model
Dataset
Clip
TROLL
Success Rate
0.80
Qwen3-0.6B
GSM8K
0.828
0.833
0.60
Qwen3-8B
Kurus-Math
0,561
0.579
0.40
0.20
SmoILM3-3B
GSM8K
0.915
0.925
0.00
SmoILM3-3B
DAPO
0,580
0.606
0
100
200
300
400
500
Llama3,2-28
GSM6K
0,589
0.668
DAPO Eval
Llama3.2-38 Instr.
GSM8K
0.836
0.850
Success Rate
FineMath-Llama-BB
GSM8K
0.750
0.746
0.60
Llama3. 1-BB
GSM8K
0.000
0.759
0.55
Llama3.1-88 Instr.
GSM8K
0.855
0.872
0.50
0.45
Apertus-SB
GSM8K 0.156
0.697
Apertus-8B Instr.
GSM8K
0.688
0.824
50
100
150
200
250
Training Step
Figure 4: (Left) Final evaluations for TROLL and Clip for different combinations of models and datasets trained with GRPO. The better approach between TROLL and Clip is marked in blue. (Right) Comparison of TROLL (full lines) and the Clip objective (dashed lines) for different models trained with GRPO. TROLL generally improves over Clip, and performs well across all considered datasets. In particular, TROLL leads to significantly faster learning for different Llama models, where Clip often takes significantly more iterations obtain a positive training signal. TROLL also showcases more stable performance compared to Clip throughout training.
Additional Math Datasets. Considering other math datasets, the top rows on the left of Figure 4 shows that TROLL is also beneficial on other datasets, as evaluated on Eurus-Math for Qwen3- 8B and the simpler GSM8K for Qwen3-0.6B Appendix D.3 provides detailed success rates for Eurus-Math in Figure 13 and additional results on GSM8K for larger Qwen3 models in Figure 12.
Additional Training Algorithms. Figure 14 in Appendix D.4 shows that adaptive clipping via BAPO (Xi et al., 2025) slightly improves over regular Clip on evaluation data, but still clearly underperforms TROLL. It also shows preliminary results how TROLL can benefit non-clipping- based policy gradient methods such as GPG (Chu et al., 2025). While vanilla GPG suffers from stability issues in our experiments, these issues are resolved by adding our differentiable trust region projections, resulting in performance comparable to GRPO (TROLL).
ANALYSIS AND ABLATIONS
KL Bounds and Sparsity Threshold Experiments. We explore different values for the KL bound e and the maximum number of kept tokens K in the sparsification process for Qwen3-8B trained with GRPO the DAPO dataset. The left of Figure 5 finds that lower KI slows down training but does not affect convergence, while higher KL leads to worse success rates, likely due to too large policy updates. small K=16 causes poor updates, presumably due to poor estimates of the underlying dense distributions, while a larger K increases cost but does not improve over our default K-64. Figure 15 in Appendix E.1 provides additional detail. These results suggest that TROLL requires an accurate KL projection, while showing that there is a wide range of suitable hyperparameters for both the KL bound and the sparsification. Finally the top row of Figure 18 shows that 5-10 tokens are usually sufficient to capture 99. 99.999% of logit probabilities.
Appendix F.2 compares the training behavior of TROLL. and Clip for different batch sizes. We find that TROLL can easily deal with larger batch sizes and thus less recent data in its optimization, while Clip clearly degrades when increasing the batch size.
Batch Size. Figure
Output Diversity and Entropy. Recent work has shown that the PPO-like Clip objective tends to exploit the LLM's existing knowledge by reducing each token distribution's entropy to increase the reward (Cui et al., 2025b). In contrast, the bottom right of Figure 5 shows that TROLL preserves entropy. Additionally, Figure 17 in Appendix E.3 shows clear correlation between TROLL's ability preserve entropy and improve performance Code..

---

Published as a conference paper at ICLR 2026
TROLL
0.01
0.25
Qwen3-1B
Clip
Troll
K = 256
K 16
VRAM
34.574 GiB
36.157 GiB
Runtime
85.133
92.906
Qwen3-1.77 1.7B- MATH Eval
Mean Success Rate
0.5
Qwen3-14B
0.45
0.4
TROLL
Entropy
0.3
Clip
0.40
0.2
0.1
10
20
30
40
0
20
40
60
80
100
Wallclock Time (hours)
Wallclock Time (hours)
Figure 5: (Left) Qwen3-1.7B trained with GRPO using the TROLL projection compared to different hyperparameter choices. TROLL works well for conservative KL bounds and top-K logit selec- tions, but is slower for too conservative values and degrades slightly for too aggressive updates or token pruning. (Top Right) Memory and runtime comparison between TROLL and Clip in a con- trolled environment. TROLL imposes modest overhead compared to the cost of training the LLM parameters. (Bottom Right) TROLL generally maintains more entropy during training while show- ing higher success rates when compared to Clip, as shown for Qwen3-14B.
Projection Fraction. Comparing the fraction of clipped tokens for Clip with the fraction of pro- jected tokens for TROLL shows that both trust region approaches roughly affect the same number of tokens. The observed stability improvements are thus not merely caused by more restrained tokens. We compare both ratios for larger Qwen3 models in the middle row of Figure 18.
Response Length. The lower row of Figure 18 shows that TROLL adapts response length more quickly to ranges suitable for solving the tasks. This faster adaption reflects the faster performance improvements achieved by TROLL.
Computational Overhead. Appendix E.4 provides a controlled experiment setup for measuring TROLL's computational overhead. We find the top right table of Figure 5 that the memory overhead of maintaining sparse distributions negligible compared to storing and backpropagating through the LLM, explained in Equation 23. Further, both memory and computation time for TROLL scale only with the vocabulary size, which constant for most model families. We thus find that TROLL's overhead diminishes as model size increases. Table 5 provides detailed evaluations.
7 CONCLUSION
We introduce TROLL, trust-region based policy gradient objective that acts drop-in replace- ment for the popular PPO-clip. TROLL is based on a novel, principled, and fully differentiable trust-region projection for discrete distributions. This projection compares two distributions, in case, the token logit distributions of an old policy used to collect sequences, and a new policy that performs policy gradient updates on these sequences. Since these distributions are prohibitively large for modern vocabulary sizes, we extend the projection to sparse distributions. Here, we only keep a small subset of logits that represent the most likely token predictions, allowing us realize both data collection and the projection objective using fully sparse operations. We experimentally validate TROLL across various model families, model sizes, advantage estimation methods, and math and code generation datasets. TROLL significantly and consistently outperforms the PPO-clip objective in terms of sample efficiency and final reward across setups, while only requiring a small overhead that does not scale with model size.
Limitations and Future Work. We currently evaluate our method on dense models with up to 14B parameters. In future work, we want to scale TROLL to larger models and Mixture-of-Expeeee architectures. Similarly, it would interesting to extend TROLL to other modalities and tasks, using, for example, vision-language models, where the logit distributions and their projections may behave differently from pure language.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Abbas Abdolmaleki, Rudolf Lioutikov, Jan R Peters, Nuno Lau, Luis Pualo Reis, and Gerhard Neumann. Model-based relative entropy stochastic search. Advances Neural Information Pro- cessing Systems, 28, 2015.
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learn- ing Representations, 2018.
Ilge Akkaya, Marcin Andrychowicz. Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint 1111v:1910.07113.. 2019.
Riad Akrour, Abbas Abdolmalcki, Hany Abdulsamad, Jan Peters, and Gerhard Neumann. Model- free trajectory-basee based policy optimization with monotonic improvement. Journal of machine learning research, 19(14):1-25 2018.
Riad Akrour, Joni Pajarinen, Jan Peters, and Gerhard Neumann. Projections for approximate policy iteration algorithms. In Proceedings of Machine Learning Research, pp. 181-190, 2019.
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization a layer in neural networks. In International conference on machine learning, pp. 135-145. PMLR, 2017.
Marein Andrychowicz, Anton Raichuk, Piotr Stañczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? large-scale study. In International Conference on Learning Representations, 2021. URL https: opwnreview.e...e net pet/forunt wwweendddndnisnnii.e.
Bowen Baker, Ingmar Kanitscheider; Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. International Confer- ence on Learning Representations, 2020. URL https: //tt:::///peerevieww net htt/fformm?io SkxpxJBKwS.
Elic Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, et al. Smollm3: smol, multilingual, long-context reasoner, 2025. URL https: https///hggggggaceeeeee ct.bloggccooo/////eooooo smollm3.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung. Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pond'e e de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06688 2019.
Stephen P Boyd and Lieven Vandenberghe. Convex optinization. Cambridge university press, 2004.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1606.066 2016.
Onur Celik, Aleksandar Taranovic, and Gerhard Neumann. Acquiring diverse skills using curricu- lum reinforcement learning with mixture of experts. In Forty-first International Conference Machine Learning. 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- fems, 30, 2017.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.025.4 02546, 2025.
11

---

Published as a conference paper at ICLR 2026
Pierre Clavier, Nathan Grinsztajn, Raphael Avalos, Yannis Flet-Berliac, Irem Ergun, Omar D Domingues, Eugene Tarassov, Olivier Pietquin, Pierre H Richemond, Florian Strub, et al. Shiq: Bringing back bellman to llms. arXiv preprint (iXxi:2555.110800 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021a. URL https: //axxiv. axg/abs/2110- 14168.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprini a 111.110.1166 14168, 2021b.
Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, et al. Command a: An enterprise-ready large language model. arXiv preprint arXlv:2504.00698, 2025.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, et al. Process reinforcement through implicit rewards, 2025a. URL https: / aatiiv.iiiiv.///.i//...a/.aaaava www./os.//...ooooo.../....e...ee ht///bbs/22/ /502 01456.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo. Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint 2rX1v 2025b.
Asen I. Dontchev and Tyrrell Rockafellar. Implicit functions and solution mappings, volume 543. Springer, 2009.
Jinhao Duan, Hao Cheng. Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5050-5063, 2024.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. In International Conference on Learning Representations, 2020. URL http: http://arxvv... org/ abs///b///// 2205 12729.
Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi, Bill Wu, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, et al. Contrastive policy gradient: Aligning Ilms on sequence-level scores in supervised-friendly fashion. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 11353-21370, 2024.
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-man math: A universal olympiad level mathematic benchmark for large language models. In The Thirteenth International Conference on Learning Representations, 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, et al. The llama herd of models, 2024. URL https: /tttpp:://rxxiv... att/ii.iiiiv////.////eea//aaaavv wwwwwwwwwwwwwwaa..a..ararraaaeae org/abs 2 407 21783.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp- 3828-3888 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
12

---

Published as a conference paper at ICLR 2026
Alejandro Hernández- Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergihert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Durech, Ido Hakimi, Juan Gareía Giraldo, Mete Ismayilzada, et al. Apertus: Democratizing open and compliant Ilms for global language environments, 2025. URL https: /atxivrivvvvvv////i///aa///aaiia
2509. 14233.
Tai Hoang, Huy Le, Philipp Becker, Vien Anh Ngo, and Gerhard Neumann. Geometry-aware rl for manipulation of varying shapes and deformable objects. arXiv preprint arXiv:2502. 2025.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 30016-30030, 2022.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: Stabilizing critic-free policy optimization with global normalization. arXiv preprint arXiv:2501.0326. 2025.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022.
HuggingFaceTB. Sheemath-llama-33 https: /ttgs////hggngcceece htt.:/aaaaaaaaaeeeeeeeeeoooooo Hww./Hggggggaaeeeeace FineMath-Llas Lachhh-lamme 3B, 2025. Hugging Face model card; licensed under Apache-2.0.
Sham Kakade. A natural policy gradient. In Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic, NIPS'01, pp. 1531-1538, Cambridge, MA, USA, 2001. MIT Press.
Sham M. Kakade and John C. Langford. Approximately Optimal Approximate Reinforcement Learning. In Proceedings of the Nineteenth International Conference Machine Learning, pp- 267-274, 2002. URL https: /dl. acm. wwtp///gg/ooooo//oo/o///oo/ooo/o
/o///ol111./1 10. 5555/ 645531. 656005.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
William Karush. Minima of functions of several variables with inequalities side constraints. M. Sc. Dissertation. Dept. of Mathematics, Univ. of Chicago, 1939.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statisties, 22(1):79-86, 1951.
Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why adam outperforms gradient descent language models. Advances in Neural Information Processing Systems, 37:30106-30144 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Ma- lik, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124 2024.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Ge Li, Hongyi Zhou, Dominik Roth, Serge Thilges, Fabian Otto, Rudolf Lioutikov, and Gerhard Neumann. Open the black box: Step- based policy updates for cumporally-coreelat episodic reinforcement learning. arXiv preprint arXiv:2401. 1437, 2024a.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9 2024b.
13

---

Published as a conference paper at ICLR 2026
Rongao Li, Jie Fu, Bo- wen Zhang. Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXi::2312.14852 2023.
Yujia Li, David Choi, Junyoung Chung. Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-leve code generation with alphacode. Science, 378(6624) 2022. doi: 10.1126/science.abb 158. URL https:://///sp /t/////www /www. science. org/ /wwp//ssi/aaoooo/eeooa/seeaaeeat hwwp///ee/eeeeeeee science abq1158.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint M.XY::2503.2088., 2025.
Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Rea- soning with reinforced fine-tuning. arXiv preprint arXiv:2401.0880 009967, 2024.
Jan Magnus and Heinz Neudecker. Matrix differential calculus. Econom. Theor, 5:161-165, 1989.
Fabian Otto, Philipp Becker, Vien Anh Ngo, Hanna Carolin Maria Ziesche, and Gerhard Neu- mann. Differentiable trust region layers for deep reinforcement learning, 2021. URL https:
//openzeview....////////e//e//m/ wwt/ //oopeneerieee..ee/foornn.eeeeep wmdiimg AO1Vn.
Fabian Otto, Philipp Becker, Ngo Anh Vien, and Gerhard Neumann. Efficient off-policy learning for high-dimensional action spaces. International Conference on Learning Representations, 2025.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, et al. Train- ing language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 1607-1612, 2010.
Steven T Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112-11 2014.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53744 2023.
Yi Ren and Danica Sutherland. Learning dynamics Ilm finetuning. The Thirteenth Interna- tional Conference on Learning Representations, 2024.
Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, et al. Offline regularised reinforcement learning for large language models alignment. arXiv /9107, preprint ar 2005 140..9917, 2024.
Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of langauge model performance. Advances in Neural Information Processing Systems, 77711841-15892 2024.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1897. PMLR, 2015a.
14

---

Published as a conference paper at ICLR 2026
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation. In International Conference on Learning Representations, 2015b. URL http://arxiv.orrg absg/50bb/555.. 02438.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprini arXiv: 1707.00777 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models. arXiv preprint 2rXiv:2402.033300, 2024.
H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg. Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin Ried- miller, and Matthew M. Botvinick. V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control. In International Conference Learning Representations, 2020. URL https: http://ppppeeeeeeeeeeeeeeeeeeeee net for worommmiiiee Syy
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
Yuhui Wang, Hao He, Xiaoyang Tan, and Yaozhong Gan. Trust region-guided proximal policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of The 35th Uncertainty Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp. 113-122. PMLR, 22-25 Jul 2020. URL https: http:////ooeeeeeeeee mlr www.mmrrr.s.......m..mm.mmmmm..e
prw.sprssspeee watp//v115////11 wwtp/////1//wagga/////a html.
Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, et al. Bapo: Stabilizing off-policy reinforcement learning for 1lms via balanced policy optimization with adaptive clipping. arXiv preprint arXiv:2510.189) 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, et al. Qwen3 technical report, 2025a. URL https: a/wwiv org/ abs/ 250.//555..o 09388.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, et al. Qwen2.5 technical report, 2025b. URL https::///// /tappps://axxv..../// orw. abtp/ab2..22222..
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, al. Dapo: An open-source 1lm reinforce- ment learning system at scale, 2025. URL https:://///....... pestps://arxieee wwwaaaivorggaaaaeaeeeeee www.or///bbb/..../// /2003. 14476.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy op- timization. arXiv preprint arXiv:2507.1 18071, 2025.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint Ttt.:19999085999 2019.
George Kingsley Zipf. Human behavior and the principle of least effort: An introduction human ecology. Ravenio books, 1949.
15

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
TROLL improves the efficiency of LLM finetuning by enabling scalable trust-region optimization. While experiments focus mathematical reasoning, the method is broadly applicable other domains. As with any advance LLM training, this carries both potential benefits and risks, de- pending on the context of deployment. We believe that managing and shaping the societal impacts of increasingly powerful LLMs should not be left to individual researchers, organizations, or compa- nies alone, but they must be carefully governed and regulated by sovereign governments and strong democratic institutions.
REPRODUCIBILITY STATEMENT
All experiments in this paper rely on publicly available pretrained checkpoints. We exclusively use publicly available datasets. While some were modified, we describe these modifications and will release the processed versions upon the deanonymization of the paper. Further information, together with additional hyperparameters and training details, are provided in Appendix C. Our implementa- tion builds open-source repositories and will be made available after deanonymization.
ON LLM USAGE
We used LLMs to assist with revising grammar, style, and text flow in this manuscript. In addition, we employed LLMs to support aspects of the implementation and generate visualizations for this manuscript.
ACKNOWLEDGEMENTS
The authors acknowledge support by the state of Baden-Württemberr through bwHPC, well as the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden- Württemberg and by the German Federal Ministry of Education and Research.
A DERIVATIONS
For each output token o, the trust region projection layer solves
min KL (Ta(o: q, ocs) || (o q, q,0))))
Telo, (q,02s)
s.t. KL (Te(O: q. (z))) (ola(oe q, o<t)) < and Σ [o(o q. 0ct)] 1.
Here, the first constraint enforces the trust region the previous distribution (ota(O; q, Oct) and the second constraint ensures the resulting distribution is properly normalized. We solve the constrained optimization problem using the method of Lagrangian multipliers and start with the primal solution.
A. PRIMAL SOLUTION
To compute the primal solution this optimization problem, we first set up the Lagrangian function by introducing Lagrangian multipliers 7) and X for the first and second constraint, respectively.
16

---

Published as a conference paper at ICLR 2026
The corresponding Lagrangian is given as ((T 9,0,),,)) =KL (To q, Oct) (o q, (,t)) s/( KL (To(Or (,,0t) || Mota q, 0ct))
Σ (or q. a<1z))
log To(or q. 0ct)
Tp(0r q, Oct)
(ne + 2) + Σ To(or q, 0c))
+ + log
(o 0ct) + x
*0(0: 9:0cs)
(me + A)+
(8)
Elro(or q, oz))) ((1 + log Tolo: - lo to(or q.0c)) - 17 To 1 0.t) + A)]
We can now obtain the optimal primal solution to Equation 7 by taking the derivative the La- grangian w.r.t no(o; q, Oct), setting to and solving for To(o) q. Oct). The derivative is given by
(L( q. Q.t)).n)
θTo(0r 10000/00000 g.t))
= =Σ ((7 +) (17 1) log (o q, 0ct) (log (o: (0:1 0zx) log (ota q, Oct)) + A].
Clearly @L(ma(o (1.. (q.00)) 0 if all the individual terms of the sum 0. Thus, the problem simplifies to
0 = (n + 1) + (17 + 1) log (or q. 02t) (logã q, o.t) + nog Totalor q, oct)) +
which yields
log (o q. q.0 - n log (o q.oc.)) -n (1+1+
log (0 0g00))
(9)
n
and thus
Tog: q, 0z)) + n log (o: 0ct)
+
<0(0: q, 021) exp
exp
17 + 1
17 n 1
Tog q, cx) + n log (ot 4, 0ct))
exp
(10)
17 + 1
Crucially, this primal solution allows computing a properly normalized distribution no (o, I q. ocr) without explicitly computing À by replacing the exp in Equation 10 with softmax.
A.2 DUAL SOLUTION
The second step of the Lagrangian multiplier method is solve the dual problem which finds the optimal dual parameters given the primal solution. To that end, we insert the primal solution from Equation 9 into the Lagrangian (Equation 8). Here most terms cancel out, leading to dual of the form
D(1, X) = -JE - X 1 = - (1+++
(11)
second step towards a practically usable dual, we remove the dependency on À by exploiting the constraint it enforces, i.e.. Es, (or q, occ))] 1. Going to log space and again using Equation 9, this property yields
0 log Σ
(o, q, 0x0)
log fo q. 0zr) + nlg (ou q.0z1)
17 _ -
]
log Σ
exp
exp
+
7 1
+ log Σ
log (o q. aze) + n log Tora q, (21))))
exp
17 +
n +
17

---

Published as a conference paper at ICLR 2026
which we can rewrite as
log (o 0cr) + 1og (oua 0.r)) )}
17 -+1+\= (1) log
exp
(12)
Now, inserting Equation 12 into Equation 11 removes the dependency on À leading
log To(o; q, g,0z)) + mlog (ota 9, Oct)
D(n) = -IE + 1) log
Σ
exp
"
Using this dual, can find the optimal 17" by solving
7/ = arg max ((n) s.t. 1j 2 0.
(13)
"
We can efficiently optimize this scalar optimization problem using n-ary bracketing method described in Listing 3.
A.3 GRADIENTS
This trust region projection is trivially differentiable using standard autograd tools, except for the numerical optimization of the dual to find the optimal step size n7" Towards differentiating through this optimization problem in closed form, let us first change perspective and no longer consider the distributions Te(o: q, Ocz),, (o: q, Ozz), and ão(oz ocx)) directly but vectors g, Gold and
(log)
g(log) Here & corresponds to the probabilities of To (o q. Oct)) while (log) and i q(lea) denote to the normalized logits of Told q, Oct) and (o( g, Ozt). We further assume all 3 vectors are normalized, i.e.,
Eg=1, Σ exp aold =1 and Σ Eexp =
While this notation may seem slightly unintuitive at first, it simplifies the following derivations. As we assume the Tota(o, qg ozt) and consequently lold are constant, the only gradient we are On" interested is J7 i.e., how the original LLM's output influences the optimal step size 17". Since do not have analytical form for the optimal step size 17" but only the result of the numerical optimization, we need to introduce further analytical properties. Using the implicit differ- entiation (Dontchev & Rockafellar, 2009) and differentiable matrix caleulus (Magnus & Neudecker, 1989) techniques introduced to deep learning by OptNet (Amos & Kolter, 2017), we start by writing out the Karush-Kulha optimum at ny" Denoting the Lagrangian multiplier corresponding to the 2 0 constraint by μ and (log), those
(log)
(KKT) conditions (Karush, 1939) of the dual in Equation 13 for the
realizing that D() = KI. (no |q, Oct) || Tota(o: | 1q 0zx) are given by
q" (log g
Gold
VD("))~\((11>) = (logq
(log), Lold
and
(--n")
Stationurity
Complementary Slackness
As there is no equality constraint in Equation 13, primal feasibility is given by default. We can now take the total differentials around these conditions, which are given by
0 d (c- q² (log gela - 14) -d ( (log -
(log) (uue)) du
(14)
0= d(p(-1)") = du( (u(--))) + ((dd))")
(15)
where de vanishes as it is constant. Before proceeding, we need to rewrite the KL term
(q" (logg Gala ((laa)) in terms of ( and simplify. First, we have
d [1"olog u (loe)) = (1 logg - Lold
(log) T
(16)
and need to continue with the differential dg. Again using the primal solution Equation 10, we get
(
9old (19((00)) (log)
+ ((0g)
q softmax
(17)
17 +
18

---

Published as a conference paper at ICLR 2026
Assuming the old logits are constant, write the corresponding differential as
dq
Ōq
dq (q(leg)
dg (log)
+
drj-.
Inserting this term into Equation 16 and the plugging the result into Equation 14 yields
(1 + log 9 - fold
T dq drj" du (1 + logg - Lold
(log)
Ōq
dg (og)
(18)
On"
d7(log)
-udn" 1j dj O,
(19)
which we can use to compute the desired gradient 0q(log) On"
To this end, we consider two separate cases. First, the original KL trust region is not violated, then rf" 0 and μ >> 0. In this case, On is zero. Second, the Equation 19 directly yields that dry" = O and thus the entire gradient ar((og)
original KL trust region constraint is active and thus 1f¹ and μ 0. In this case Equation 19 gives du = 0 which simplifies Equation 18. Reordering the remaining terms gives the required gradient
Ön Og(log)
(1 + log (log) T Oq de(logy
-(1 + log q
Gold
(log)yr
(log)
yr
oq
Oxy"
The required partial derivatives can be obtained from Equation 17
0q
dq
17' +1 (D(x) q9"" and
(1 + (D(g) - (gold" (log)
(D(g)
- -((h)))
0G(Wug
where D(g) denotes a diagonal matrix with the entries of on the diagonal.
Crucially, for practical purposes, never need to explicitly materialize the matrices in the partial derivatives. The resulting backward introduces negligible computational and memory overhead and. Listing 2 shows that, in the non-sparsified case, the backward can be written in less than ten lines of python code.
A.4 SPARSIFICATION
Theorem A.1. For any pair of logits of and o(2), with a(o q, 0ct) #(0,²) q. 0z)) w.l.o.g.
(1)
the logit-wise terms that sum to the KL are equally ordered
=(0, (1)
q, 0ct)
a(o (2) q, 021)
# ( q. Oce) log
2
#(0 q. Oct) log
(20)
Tola 9, Oct)
Tota(1; q. Oce)
(2)
iff ex 2 7, where = # [q00cs)
is the current probability ratio of the pair and in Tailo"" (q,0cs) 7/s gives the multiplier of the old ratio.
7o (00c))
nala, (q,0z))
Proof. Rewrite 7(o,") (1) q. 02r) 2 #(o,²) q, o.0) as p(x1) = K. p(x2) for K 2 1 using p(a:) = (o (4) 9, O.) for clarity and similarly replace q(x.)) (o q. 0cr)) Then compare the con- tributions of ii and 22 to the KL divergence
xp(#2) log xp(x2) ((x1) 2 p(x2) log p(x2)
(112)
g(21)
klog k > log
q(x2)
and substitute g(a1) p(21) 7K
((32) p(22)
k 2 p(
x(22)
x 2 7.
19

---

Published as a conference paper at ICLR 2026
Here, the assumption that the relative likelihood i of (1) and of2) was not exponentially larger before usually holds in practice, as the token distributions are pushed farther from uniform during training (Cui et al., 2025b).
Definition A.1. For any subset S of the possible tokens, we define ps. or just p' when the mask is clear, as the sparsed distribution. For tokens not in S, it has default probability pa and the same probability as p for all others up to the renormalization constant.
(p(x)) for x E S
1 1- 1-( - (S]) Pa
(-V1-18)
p2( = p'x))
pd,
else
7
EzeSP(z)
(21)
The renormalization factor accounts for the previous total mass Exes p(x) of the kept tokens and new mass (IVI - |S]) Pa of the dropped tokens.
In the case of equal sparsification masks for distributions p. q, we can prove a practically tight upper bound for the true divergence (p q) in terms of the sparse divergence KL (p' q').
Theorem A.2. Let p. be categorical distributions over the vocabulary | with identical top-k logits, tppk(p) = topk(q) and equal total probability Exetopk(p) p(x) = -xetoph(a) ((x) = 1-5. Then the sparsed distributions p', of with density
(pp(T), for x E topk(p) [pa:
d'(x,)== = (g(()),
for E lopk(a)
px(x)=
else
Pad
else
and normalization constant (6, k; [V]. Pa) = 1 follow the inequality
KL (p g) S KL (p' || q') + 5 log
Umin
where Jmin = arg min ((x).
Proof. Rename the logits in descending order of probability under p. such that p(æo) 2 p(z1) > p())) Assume there is k < |V such that the largest logits of both p and q have exactly the total probability mass Σx= p(x;) = E ((z,) = 1 5 and the subset of largest logits is as the total mass could otherwise not be 1 1-6 8. So split the sum over logits in the KL divergence and apply both inequalities
identical. Every nondegenerate distribution has Amin S g(æ;) and all p(x:)
for
p(as)
p(x:) log p(x:)
KL (p 4)
Σ p(ax) log
Σ
((x;)
g(xra)
p(xx)) log p(x;))
8
p(x:) log
((1;))
(
p(xx) log p(zz)
+
Σ
p( p(x;))
log
5
q(z:)
quna
k
log M-11
Σ p(xx) log p(zi)
p(x))
q(x,)
Jmin kmm
Σ p(xx) log p(x;)
k-1
log
((x;))
quii
Replace p. and analogously 9, with their spansed version as defined in Definition A.1,
(rp(a.), for i << k for i 2 k
p'(x,)
(22)
(pa.
20

---

Published as a conference paper at ICLR 2026
deg_ref def TRoiLPrejection (Logggaagg prab, log ref prob, bound)
2
kl_áiv (log_target__robbceeer () * (log_target_prob - log_ref_prob)).
sum (din=-1)
needs_projeccton ki >= bound 8 only projects where necessary
... masking of needed tokens
5
solve dual problem, i.e. find #
6
opt_eta = DualSolver (logttrgetpttt
log_ref_pxode bound) primal_unnormalliess = (opt_eta pog_ref_rroi + log_target_prob) /
opt_eta 1)
primal inner inner.Log_see wog__oootaaaaaoo (dim=-1)
9
combine masked unprojected and primal Logits into one return projected_logits
10
Listing I: TROLL's differentiable projection only calls differentiable dual solver and otherwise uses standard autodiff operations.
where 7 11((11-)))1 Pa renormalizes the (1 8) mass of the selected tokens to account for the default mass (IV) - k) pa the sparsified tokens. Multiplying with ones and adding a zero to the KL bound yields the relation to the sparse KL,
VI-1
p(x;) log mp(x:) (g(2z) + ôlog
Pa
KL (p
+
y
p((x))
log
Jmin
Pd
0=0
km
=0
= 7 Σ p' (x:) log (x44) g' (xx)
k
8
IV-1
((,))
+ 5 olog
+ r
Σ
p (x;
log
amiu
(x;
1=0
1=
5
= 77 KL (p' 4 ) + log
Umin
Assuming that q's probabilities can be represented by normal single precision IEFE-754 numbers, Amin > 1.17549 10 338
10-3s, and k < [ c.g. k 256 of vocab size | 151936 while using
threshold 8 = 10 5 and default mass Pa = 10 10-12, 12, the sparse KL approximation,
(1 - 8) KL (pf || (') + 5 5log
KL (1 5
1 (VV - k) Pd
Juin
0.99999
10-5
1 151680 10-12 KL (p/ | - 10 log
1.17549 10-38
0.99999015168 KL (p' q') + 0.00075823623,
accurate enough for limiting the true divergence values the order of 0.05 as
KL (p q) < 0.99999015168 KL (p' || q') 0.00075823623
< 0.99999015168 0.05 + 0.00075823623
KL (p || g) < 0.050757743814.
B IMPLEMENTATION
While the theoretical derivation of the differentiable trust region projection can look complex, the final implementation is fairly straightforward. We give PyTorch-adjacent pseudocode for the dense variant of the primal (Listing 1) and dual (Listing 2) in the following, Note that the sparse implemen- tation mostly differs in the usage of custom sparse tensor class that maintains a default probability for the implicit entries. While this requires additional care in terms of indexing and allows for opti- mizations of, e.g., the KL computation, the general logic remains unchanged. Listing 3, shows our n-ary bracketing method optimize the dual.
21

---

Published as a conference paper at ICLR 2026
KL Term
Equation 13
Equation
Treul (0)
(o q: 0ct))
1f
To(or q Oct)
Equation 5
Ratio Term
(o: q, 0ci)
Numerical Optimization
Analytical Computation
No Gradient
Figure 6: Compute Graph from the LLM output (o q, Oci) the RL objective (0). First, the optimal step size 1i is computed for each token. Here, we use the fact that the step size for tokens that do violate the trust region trivially O. For the tokens that do violate the trust region, we need to optimize 1-D convex optimization problem to compute 17". Next, the projection computes the optimal distribution within the trust region, To(o; q, Oct), which is then used in the objective. This objective combines the standard policy ratio term from PPO with KL term to regress (o: q, Occe) towards To (Or q, Oct).
def Mual$olven forward [log_target_prob, _os_ref_prob, baund) define objective terms of log eta (such that eta 0J
opt ppt_log_eta potimizeldd
lambda log eta: dual (log_eta, ...), bounds and termination config
... save for backward return opt_log_eta.cxpl
8
def dual (log_eta, bound, log_target_prob, log_ref_prokt
eta log_sta.exp()
eta log_ref_prok / (eta + 1) inner_lse logsumexp (inner, axis=-1) negative of objective, since we minimize return eta bound (eta
inner (log_target__pg
13
inner 1se
17 def DualSolver barlsoller.bcckwrrllgga... waw.warddgggddd_oupuu.... recompute primal
18
as in rROLiProjection one plus pllppplllppppllleee louratto = pwwmal. log_re ref prob compute one plus logratio. dprimal_ dlog output implicitly dlog_output numerator primal (one_plus_ pog_rtttttootioooooo vecdot (prinal,
cac_plus_logratio (nnouucatii..usuuccce(11111 (-1 (opt_eta + 1) compute dprimal
implicitly
diff = leg_ree_prro_l log_target_prob
dprimal_dopte
primal (diff vecdot (primal, diff) unsqueeze
(-1) (opt_eta 1)*+2
return put (numerator / veedot
Listing 2: Custom forward and backward code for TROLL's dual solver.
22

---

Published as a conference paper at ICLR 2026
class Opt pptimian zerlD:
def batched_linspace (lower, upper, num_points):
Batched linspace: lower and upper are size, 2), returns (batch_size, num points)
7
steps linspace (0, num_points) return lower + (upper lower) steps
8
def (Func,
lower, upper): batch_size, num num_points x.shap batched evaluation of all points y = func
12
13
seiect min index for each batch element
min idx argmin (y, dim=1)
take left and right point L_dx min_idx -
18
idx min_idx +
1_mmp x(arange (batch_sizze
clamp (1 ixxx num_poinnss 1]]
tmpp x (arange (batch_siaaee
clamp (u (u_ idx, 0, num_ 1)]
22
new_lower where (1_idx < 0, lower), t_mp))
new_upper where (u_ixx >= num_points, upper, u_tmp)
return new_lower, new_upper
def pptimize(funa, lower, upper, num_points, max_steps, x x_threshoid) d)
batched, parallel, gradient-free, optimization of 10 function
lower, upper
refine lower and upper until convergence
for step in range (max steps):
x x(:, 1:-1]
(1, nam_points + 2)
1, (egiinieerD..ooootststt(fuu x, 1, u)
36
if ((1 < x_threshhll)) abs .all() (): break
40
x (1 u) /2
return
Listing 3: N-ary Bracketing Search.
23

---

Published as a conference paper at ICLR 2026
Model
Link
Qwen3-0.6B
attp:://huggiigiscccc////
Qwen3-1.7B
attp:://uugaaigeeeeec//eeeee 7B
Qwen3-4B
Qwen3-8B
Qwen3-14B
http:s://thaog..nngccc.ee.e/eeep 14B
Owen2.5-00.BB-Instte
Inst
Swen2.5-3B-Instrue
hetpe://hugginggacee ct//wen/ Qwen2. 3-32mm2..533-110 -0rr222--3--Lnttrreet Instruct
Qwen2.5-7B-Inst InIItrucct
hetpa://huggiinggaceee. wwt.c//////ee.....eeemm /wwe//.e22... In-truttt
Llama-3.1.8P
http:://buugiingkaccc.oo/// 11ame/ LLaama 3.1-88
Llama-3.14 1.0........BItnttree
autp:://uuugiingcccccc/o//tttt 1lama/Llame 3.1 Instruct
Llama-3.2-3B
11ame/Laam
LLama-3.22 1iiiaa..2--B-Isttruue
SioMathh-Llamn 3B
Apertus-88
Apertus- Apertus-8B-iie In-Instruc
https: /ttp
SmoILM3-3B
Table 2: Model checkpoints used as starting points for finetuning throughout this work.
C EXPERIMENTAL SETUP
C.I MODELS
Table 2 lists all model checkpoints used in this work. They are publicly available and can be down- loaded under the provided links.
We used the thinking mode for the models from the Qwen3-Family. For the non-instruct versions of Llama-3.1, Llama-3.2, and Apertus, we used the chat templates from the respective instruct versions.
C.2 DATASETS
For all math datasets, sequence-level binary rewards are computed by parsing the LLM output through a regular expression, matching against ground truth answer. For code generation, we evaluate each answer using the provided test cases. The reward is then computed as the fraction of successful tests. If the execution test case exceeds a timeout of ten seconds, the evaluation is terminated and zero reward given.
DAPO-Math. We build DAPO Train and DAPO Eval on the version of the DAPO-Math dataset provided by Cui al. (2025b)* From their training set, set aside 1024 samples as in-domain validation set (DAPO Eval), leaving 16,893 samples for DAPO Train. For broader outti--dist evaluation, we again follow Cui et al. (2025b) and use benchmark suite, we refer as Math- Eval, consisting of MATH500 (Hendrycks et al., 2021), AMC, AIME2024 (Li et al., 2024b), AIME 2025, OMNI-MATH (Gao et al., 2025), OlympiadBench (He al., 2024), and Minerva (Lewkowyez et al., 2022). We again build the data provided by Cui et al. (2025b) and also follow their protocol by computing the mean over 32 responses for small but hard AMC, AIME2024, and AIME2025 datasets while only considering single response for the other sets.
Finally, we ensure all datasets have the same system preprompt, which we provide Listing 4, and include correct and identical instructions for answer formatting.
'Their original datasets he downloaded under pettp::/githhe co.. rotyooyyee sm- RI.
24

---

Published as a conference paper at ICLR 2026
Your task to follow a systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, present your reascning using the format: "<think> (thoughts) </think>".
Listing 4: System Prompt for DAPO-Train, DAPO-Eval, and Math-Eval
Hyperparameter
Variable
Value
Trust Region Size
e
0.05
KL Regression Factor
1.0
Sparsity Remaining Mass
1 1-6 K
0.99999
Max. Sparse Tokens
64
Chunk Size
1024
Clip Value
Epno
0.2
Learning Rate
10-6 6
Gradient Max Norm
1.0
Weight Decay
0.0
Learning Rate-Schedule
constant
Learning Rate Critic (PPO only)
10-5
Weight Decay Critic (PPO only)
0.01
Sampler Per Query
8
Batch Size
32
Batches Per Step
8
Table 3: Hyperparameters. We these parameters for all experiments unless mentioned otherwise.
GSM8K. We use the publicly available train and validation sets of the GSM8K Dataset (Hendrycks et al., 2021)² without further modifications.
Eurus-Math. We the publicly available train and validation Dataset (Cui et al., 2025a) which is (which math questions, resulting in 455 261 train and 024 evaluation questions, and refer to the resulting dataset as -Maru.
of the Eurus-2RRL subset of NuminaMath-Coo CoT (Li et al., 2024b). We filter for
(Cui al., 2025a) for code generation by filtering for code questions, resulting in 25 276 train and 024 evaluation questions. We call this subset Eurus-Code. Each includes multiple test cases of inputs and expected outputs after running the parsed python code within sandbox. We use the PRIME reward manager of verl? which evaluates up to the first ten test cases the SandboxFus us on Sandbox
Eurus-Code. We the same Eurus-2-RL-DD-
The dataset consist of tasks from APPS (Hendrycks et al., 2021), CodeContests (Li et al., 2022), TACO (Li et al., 2023) and Codeforces", For the evaluation, we compute the pass@ scores for each of the four benchmarks and average the results. The validation split has 142, 377, 382 and 123 predefined questions for APPS, CodeContests, TACO, and Codeforces, respectively. The train data is split 13.7%, 38.1% 9. 37.9% and 10.3%. respectively, such that the evaluation overweights APPS and Codeforces. Empirically, the models seem to perform better on Codeforces but worse on TACO, such that the evaluation success rates slightly higher.
25

---

Published as a conference paper at ICLR 2026
500M
1.5B
3B
7B
Clip
TROLL
DAPO Train
Success Rate
0.40
0.20
0.00
100
200
300
400
500
600
700
800
DAPO Eval
Train Step
Eval
0.40
Mean Suevess Rate
Success Rate
0.30
0.30
0.20
0.20
0.10
0.10
200
400
200
400
600
800
Train Step
Train Step
Figure 7: Performance TROLL and the Clip objective across Qwen2.5-Inntruu models with 500M to 7B parameters trained with GRPO on DAPO. As in Figure 3, TROLL yields more sample-efficient training and higher rewards convergence. These improvements extend both to evaluation on in- distribution questions and to generalization out-of-distribution test datasets. Smoothed values shown in full opacity, with original curves in the background.
C.3 TRAINING SETUP
We provide hyperparameters for training setup in Table 3. eters across all experiments, except for Appendix E.I, where we always vary exactly one parameter.
We maintain consistent hyperparam-
C.4 HARDWARE
We train on clusters with Nvidia A100, H100, and H200 nodes, each equipped with 4 GPUs. For H200s.
the Dwen3-14B, Qwen3-8B and Dove2.55-7BIInstrun experiments in Section 5, For all other experiments, we use either H100 or A100 nodes, depending on model size. We train most experiments for up to 2 days, and extend algorithm convergence. We always train Clip and TROLI. on identical hardware to ensure a fair comparison.
experiments DAPO up to 4 days show
D ADDITIONAL RESULTS
D.1 QWEN ON DAPO
Figure extends the setup of the math evaluation of Figure 3 to Qwen2.5-Instruct models. Similarly to the Qwen3 results, TROLL consistently improves over the Clip objective for each model size. We further find that, generally, most Qwen2.5 models slightly overfit the training data, although this effect is less pronounced for TROLL.
Submi ons
26

---

Published as a conference paper at ICLR 2026
GRPO
Dr. GR.O
GSPO
PPO
RF++
Clip
TROLL
DAPO Train
0.80
Suocess Rate
0.60
0.40
0.20
0.00
10
20
30
40
50
60
70
80
Wallelock Time (hours)
DAPO Eval
Eval
0.60
Mean Succoss Rate
Sucuess Rate
0.60
0.40
0.40
0.20
0.20
0.00
0.00
80
60
80
Wallclock Time (hours)
Wallclock Time (hours)
Figure 8: TROLL and Clip success rates for Qwen3-8B trained with GRPO, Dr.RRPO, GSPO, PPO and RF++ training data (top), in-domain evaluation (bottom left) and out-of-domain evaluation (bottom right). Smoothed values are shown in full opacity, with original curves in the background. TROLL improves over the Clip objective for all methods. For GSPO, Clip eventually diverges, leading to 0.00% success rate on all metrics, while TROLL's optimization stays stable.
Figure and Figure 9 show complete training and evaluation curves for Table 1. We find that TROLL improves training success rates Clip for both models and across methods, to the point where Qwen3 GRPO and Dr.RPPO start to slightly overfit on the oftt--d-istriutiion MATH evaluation. Interestingly, while Clip leads to unstable performance and eventual divergence for GSPO for both Qwen2.5 and Qwen3, TROLL's token-level trust region optimization remains stable.
D.2 ADDITIONAL MODELS
Figure 10 and Figure 11 show success rates for different 3B and SB models, respectively. We find that TROLL causes some models, such as Finemath-3B, Llama3.2-33 and Llama3.1 to receive training signal in significantly fewer steps. Other models, such as Apertus-8B show more stable performance when trained with TROLL. Finally, for models that work well with the Clip objective, using TROLL generally yields some performance benefit even though the success rates GSMBK are almost saturated.
D.3 QWEN3 ON EURUS- MA TH AND GSMBK
We additionally evaluate different Qwen3 model sizes GSMBK in Figure 12, finding that most models quickly saturate on this comparatively easy task. Nevertheless, using TROLL instead of Clip generally provides small boost in performance across model sizes. Similarly, Figure 13 shows that TROLL leads improvements for Qwen3-8B trained with GRPO on Eurus-Math.
D.4 ADDITIONAL TRAINING ALGORITHMS
Finally, we evaluate Qwen3-8B using BAPO (Xi et al., 2025) as an adaptive clipping heuristic that acts alternative regular clipping, and GPG (Chu et al., 2025), which is clipping-fres policy
27

---

Published as a conference paper at ICLR 2026
Method
AIME24
AIME25
AMC
MATH
Omni-Math
Olympiad
Minerva
Qwen2.5-7B Instruct
----truutttt
GRPO
Clip TROLL
0.066
0.075
0.535
0.683
0.239
0.286
0.304
0.168
0.129
0.587
0.712
0.254
0.317
0.284
Dr.GRPO
Clip
0.103
0.067
0.560
0.662
0.242
0.288
0.295
TROLL
0.168
0.135
0.605
0.706
0.259
0.320
0.317
PPO
Clip
0.092
0.064
0.503
0.706
0.251
0.316
0.299
TROLL
0.162
0.093
0.547
0.734
0.258
0,320
0.332
GSPO
Clip
0.026
0.002
0.188
0.344
0.106
0.102
0.120
TROLL
0.159
0.076
0.531
0.699
0.257
0.297
0,310
RF++
Clip
0.096
0.033
0.556
0.666
0.239
0.291
0.299
TROLL
0.166
0.102
0.552
0.687
0.263
0.320
0.319
Qwen3-8B
GRPO
Clip 0.439
0.293
0.720
0.889
0.465
0.547
0.431
TROLL
0.547
0.353
0.790
0.812
0.465
0.497
0.391
Dr.GRPO
Clip 0,458
0.305
0.743
0.891
0.477
0.547
0.425
TROLL
0.447
0.337
0.769
0.880
0.466
0.522
0.403
PPO
Clip 0.380
0.234
0.694
0.874
0.439
0.531
0.405
TROLL
0.524
0.408
0.780
0.910
0.521
0.567
0.425
GSPO
Clip 0.000
0.000
0.000
0.000
0,000
0.000
0.000
TROLL
0.474
0.407
0.813
0.897
0.514
0.547
0.408
RF+
Clip
0.356
0.251
0.728
0.897
0.459
0.538
0.415
TROLL
0.490
0.355
0.803
0.891
0.528
0.563
0.415
Table 4: Success rates for individual MATH test datasets for Dwwe2.5-7B-Innsteee and Qwen3-8B models trained on DAPO with different advantage estimation methods. TROLL provides consistent benefits across methods and evaluation tasks, showing well-balanced improvements in performance. also successfully trains GSPO without divergence, wheres Clip eventually causes unstable updates, as shown in Figure 8 and Figure 9.
28

---

Published as a conference paper at ICLR 2026
GRPO
Dr. GR.O
GSPO
PPO
RF++
Clip
TROLL
DAPO Train
0.60
Success Rate
0.40
0.20
0.00
10
15
20
25
30
35
40
45
DAPO Eval
Wallelock Time (hours)
MATH Eval
0.40
Mean Success Rate
Success Rate
0.30
0.30
0.20
0.10
0.10
0
10 20
30
40
10
20
30
40
Wallclock Time (hours)
Wallclock Time (hours)
Figure 9: TROLL and Clip success rates across Qwen2.5-7B Instructt models trained with GRPO, Dr.GRPO, GSPO, PPO and RF++ on training data (top), in-domain evaluation (bottom left) and out-of-domain evaluation (bottom right). Smoothed values shown in full opacity, with original curves in the background. TROLL improves over the Clip objective for all methods. For GSPO, Clip eventually diverges, while TROLL's optimization stays stable.
Smol-LM3 3B
Llama3.2 3B
Llama3.2 3B Instruct
FineMath 3B
Clip
TROLL
GSM8K Train
GSMSK Eval
1.00
0.80
Success Rate
Success Rate
0.60
0.50
0.40
0.20
0.00
0.00
0
100
200
300
400
500
100
200
300
400
500
Train Step
Train Step
Figure 10: TROLL and Clip success rates for different 3B models trained with GRPO on the GSMeK training data (left) and evaluated on the GSM8K test set (right). Smoothed values are shown in full opacity, with original curves in the background. TROLL generally causes models to pick up training signal more quickly, and exhibits more stable training behavior.
gradient method. Figure 14 finds that BAPO's adaptive clipping does not significantly improve over regular clipping in our case, and still performs worse than TROL....
29

---

Published as a conference paper at ICLR 2026
Apertus 8B
Apertus 8B-Instruct
LLama3.1 8B-Instruct
LLama3.1 8B
Clip
TROLL
GSM8K Train
GSM8K Eval
1.00
0.80
Success Rate
Success Rate
0.60
0.50
0.40
0.20
0.00
0.00
100
200
300
400
500
100
200
300
400
500
Train Step
Train Step
Figure 11: TROLL and Clip success rates for different 8B models trained with GRPO the GSM8K training data (left) and evaluated on the CSM8K test set (right). Smoothed values are shown in full opacity, with original curves in the background. TROLL generally causes models to pick up training signal more quickly, and exhibits more stable training behavior.
600M
1.7B
4B
8B
Clip
TROLL
GSM8K Train
GSM8K Eval
1.00
Success Rate
Success Rate
0.80
0.50
0.60
0.40
0.00
100
300
400
100
200
300
400
Train Step
Train Step
Figure 12: TROLL and Clip success rates for Qwen3 models with 600M parameters trained with GRPO on the GSM8K training data (left) and evaluated on the GSM8K test set (right). Smoothed values are shown in full opacity, with original curves the background. Both TROLL and Clip quickly converge in all cases, although TROLI achieves slightly higher performance for most model sizes.
Although clipping-free GPG initially learns well, eventually diverges during training, likely due to unstable policy updates. The figure illustrates how TROLL can benefit such non-clipping-batt policy gradient methods. Unlike PPO-style clipping, which relies on a ratio formulation, TROLL limits updates by enforcing trust regions directly the LLM output. Thus. TROLL can in principle be applicd to any LLM post-training algorithm, including GPG. In our experiments, this approach effectively resolves GPG's stability issues, demonstrated by the GPG(TROLLL) variant.
30

---

Published as a conference paper at ICLR 2026
Qwen3 8B
Clip
TROLL
Eurus Train
Eurus Eval
Mean Success Rate
Success Rate
0.60
0.55
0.50
0.50
0.45
0.40
200
100
600
800
200
400
600
800
Train Step
Train Step
Figure 13: TROLL and Clip success rates for Qwen3-8B trained with GRPO the Burus-Math test set (right). Smoothed values are shown
training data (left) and evaluated on the full opacity, with original curves in the background. TROLL performs slightly but consistently better during training, and generalizes well to the test set.
GRPO (TROLL)
GRPO (Clip)
GRPO (BAPO)
GPG (TROLL)
GPG
0.80
Success Rate
0.60
0.40
0.20
0.00
180
240
260
280
300
Train Step
DAPO Eval
MATH Eval
0.60
Moan Success Rate
Success Rate
0.60
0.40
0.40
0.20
0.20
0.00
0.00
100
200
300
100
200
300
Train Step
Train Step
Figure 14: Success rates for Qwen3-8B on DAPO for training data (top), in-domain evaluation (bottom left) and out-of-domain evaluation (bottom right). Smoothed values shown in full opacity, with original curves in the background. TROLL performs well when combined GRPO the clipping-free GPG baseline, while vanilla GPG eventually diverges. BAPO's adap- tive clipping slightly improves over regular Clip on the evaluation datasets, but both underperform TROLL by fair margin.
with either
31

---

Published as a conference paper at ICLR 2026
TROLL
0.01
0.25
K
256
K 16
DAPO Train
0.60
Success Rate
0.50
0.40
0.30
0.20
0
5
10
15
20
25
30
35
40
45
Wallelock Time (hours)
DAPO Eval
Eval
Sucvess Rate
0.50
Mean Sucoess Rate
0.45
0.45
0.40
0.40
0.35
10
40
10
Wallclock Time (hours)
Wallclock Time (hours)
Figure 15: Qwen3-1.7B trained with GRPO using the TROLL projection compared to different hyperparameter choices DAPO training data (top), in-domain evaluation (bottom left) and out-of- domain evaluation (bottom right). Smoothed values are shown in full opacity, with original curves in the background. TROLL works well for reasonable KL bounds and top-K logit selections, but slightly degrades for too-large bounds and too few kept logits.
E FURTHER ANALYSIS
E. PROJECTION PARAMETER SENSITIVITY
experiment with different KL bounds, testing e=0.01 and c=0.25 instead the default e-0.05. Additionally, try different levels of sparsification. We switch the maximum number of kept tokens from K464 to a lower K=16 and higher K==56,, adjusting the distribution mass threshold 5 from 1e-5 to 1e-4 and 1e-6 accordingly. Figure 15 shows that lower KI. bound for the projection leads to slower learning, but eventually reaches comparable performance. In contrast, a higher KI. bound leads to worse performance, presumably because the policy moves quickly during update steps. Reducing the number of kept tokens leads to worse overall performance, which likely caused by incorrect KL estimates and thus sub-optimal projections. higher amount of kept tokens does not yield any additional benefit, however, suggesting that K-64 and c=le-5 maintain sufficiently close approximation of the real policy logit distributions.
E.2 BATCH SIZE
TROLL's trust region projection promises more stable policy updates by constraining the difference between the new policy and the policy used generate the data, in our case the responses. To evaluate this stability, we experiment with different batch sizes for Qwen3-1.7B trained with GRPO on DAPO, leaving all other parameters untouched. Here, doubling the batch size doubles the num- ber of gradient steps between data collection, thus increasing the difference between the old policy Tata(o, q, 02)) and the current policy tarpol over the course of the training step. Figure 16 shows that TROLL remains stable when increasing the batch size from
default of 256 to 2 048, indi- comparison, Clip
cating that TROLL can successfully train on older data due to its projection. shows gradual degradation in performance for each doubling of the batch size, likely because the
32

---

Published as a conference paper at ICLR 2026
256
512
1024
2048
Clip
TROLL
DAPO Train
0.60
Successs Rate
0.50
0.40
0.30
0.20
0
10
15
20
25
30
35
40
45
Wallelock Time (hours)
Figure 16: TROLL and Clip training success rates for Qwen3-1.7B trained with GRPO on DAPO for different batch sizes. TROLL remains stable for larger batch sizes, while Clip gradually and consistently degrades when increasing the batch size.
clipping does not address a potential divergence of the current policy to the policy that was used to generate the data.
E.3 GENERATED SEQUENCE ANALYSIS
We analyze the behavior of different Qwen3 and Qwen2.5 model sizes trained on DAPO using GRPO in Figure 18. There a general trend that larger models require fewer selected tokens to satisfy the sparsity mass threshold of d=1e-5, which is consistent with established LLM scaling laws (Kaplan et al., 2020). Here, as little as 5-10 tokens are sufficient to capture most of the mass for the larger models. For the larger Qwen3 models, this trend appears less pronounced, likely because these models are to some extent saturating the DAPO benchmark. We also observe clear differences in response length dynamics Clip. TROLL reduces the response length for Qwen3, while increasing it for Qwen2.55-nnstmu difference originates in the different behavior of the pretrained models used initialize learning. as the Qwen3 models tend to generate much longer responses, presumably due to their built-in thinking mode. After the RL fine-tuning with TROLL, the response lengths of both model families are more similar. In contrast, models trained with Clip show much slower shifts in response length. This quicker adjustment under TROLL aligns with the faster performance gains observed in both model families. Finally, both approaches clip or project slightly more than 0.1% of tokens for most of the training, but TROLL's projection exhibits a lot more variance and tends to increase later training stages, potentially suggesting more active involvement in the learning process. In some cases TROLL projects a lot more aggressively, although this increase in projections does not cause a degradation in model performance.
training. TROLL generally adapts the token length much faster than
This
E.4 TROLL OVERHEAD
From a practical perspective, the number of selected tokens by the sparsification and the compu- tational and memory overhead of TROLL relevant. Fair comparison of the computation and memory overhead are tricky, as both the response length and number of kept tokens varies dur- ing training and models. Initially, the average number of logits needed achieve the desired total mass changes quickly, and small models tend to require significantly more logits due to their higher perplexity.
We therefore evaluate at the initial state by preventing updates with tight KI. and clip ratio bounds. Note that setting a low, i.c. zero, learning rate is not representative since the LLM outputs must change to activate the trust region projection. Table 5 compares the memory and runtime overhead of using TROLL and GRPO with small Qwen3 models on GSM8K trained on 4x Nvidia A A100-40GB GPUs. Chunked refers to another simple memory reduction trick, where normalize and sparsify only a chunk, in this case size 1024, at once and avoid the dense single precision upcast of the entire mini batch. To compare the different model sizes with different response length characteristies fairly,
33

---

Published as a conference paper at ICLR 2026
600M
1.7B
4B
8B
Clip TROLL
Burus-Code Entropy
Eurus-Coae Train
0.8
0.6
0.6
Success Rate
Entropy
0.4
0.4
0.2
0.2
0.0
0
100
200
300
100
200
300
Train Step
Train Step
Figure 17: TROLL and Clip entropy (left) and success rate (right) for different Qwen3 models trained with GRPO on the Eurus-Code training data. Smoothed values are shown in full opacity, with original curves in the background. TROLL generally causes less decrease in token entropy while Clip shows strong negative correlation between success rate and entropy. The quick improvement of Qwen3-8B Clip around step 40 coincides with a rapid drop in entropy.
Metric (Method)
Qwen3-0.6B
Qwen3-1.7B
Qwen3-4B
VRAM (Clip)
25.415 GiB
28.418 GiB
34.574 GiB
VRAM (TROLL)
27.868 GiB
30.663 GiB
36.837 GiB
VRAM (TROLL Chunked)
27.227 GiB
29, 29.994 GiB
36.157 GiB
VRAM Delta (Chunked)
+1.812 GiB (+7.1%) +1.576 GiB (+5.5% +1.583 GiB (+4.6%)
Runtime (Clip)
30.874
43.372
85.133
Runtime (TROLL)
46.715 s
49.053
90.570
Runtime (TROLL Chunked)
47,600
50.629
92.906
Runtime Delta (Chunked)
+16.726 (+54.2%)
+7.257 (+16.7%)
+7.773 (+9.1%)
Table 5: Max allocated VRAM and runtime of one iteration. The smallest 0.6B models does not fully saturate the GPU, so the Delta results differ from the larger models. The projection overhead is independent of the model size and already below ten percent for the small 4B model and slower A100 GPU. The advantage the chunked sparsification depends on the micro batch size, so the benefit is larger for bigger GPUs.
clip all answers just 256 tokens. Note that this length is just short enough that most answers are clipped, while some prompts are still solved. Then all models have almost the maximal 256 response tokens average (about 255.4), yet still have update gradient to reach the trust region boundary.
The responses for single prompt with GRPO group size of 8 and this significantly simplified sequence length of 256, float-32 representations for the probabilities, and Qwen3's tokenizer with a vocabulary of 151 936 tokens yields memory overhead of
256 8 151936 4B ~ 1.16GiB
(23)
for the dense implementation. Sparsification instead requires an average of 5-10 logits per token (Figure 18 top), reducing the memory less than 1MiB. For each iteration, all methods need to store rollout buffer of answers, in our case of size 256. In addition, the current mini-batch for the policy update needs be stored. While this overhead can be reduced to a single answer with gradient accumulation, the rollout buffer still needs to store all outputs of the old policy. The total memory overhead reported in Table with 256 prompts is only 226.3 GiB across the four GPUs while just the dense distribution storage would already require =226 GiB of memory, showing the necessity of our sparsification.
34

---

Published as a conference paper at ICLR 2026
Qwen3
600M
1.7B
4B
8B
14B
Cli
5wen2.5-Instruct
500M
1.5B
3B
7B
TROLL
Mean Selected Tokens
40
30
30
20
10
10
50
100
200
250
400
Step
Train Step
Clip/Pro Peo/Projjectiin Ratio
10
10
10-3
10-3
10
10-4
50
100
150
200
250
300
0
200
400
600
800
Step
Train Step
Mean Response LengUh
4,000
4,000
3,000
3,000
2,000
1,000
1,000
50
100
200
250
300
200
400
600
800
Step
Step
Figure 18: Training dynamics of Qwen3 models on DAPO using GRPO. Smoothed values are shown in full opacity, with original curves in the background. Larger models need fewer tokens meet the sparsity threshold of 8=10-5 While both approaches affect ~0.1% of tokens most of the time, TROLL tends increase projection later during training without harming performance. TROLL quickly adjusts response lengths while achieving higher success rates, whereas Clip is slower to alter the response length over time.
35

---

Published as a conference paper at ICLR 2026
F EXAMPLE GENERATIONS
Prompt
system
2 Your task to follow a systematic, thorough reasoning process before providing the final solution. This involves analyzing, summarizing, exploring, reassessing, and refining your thought process through multiple iterations. Structure your response into two sections: Thought and Solution. In the Thought section, present your reasoning using the format: "<think>
3 (thoughts) </think>
Each thought should include detailed analysis, brainsterming, verificatien, and refinement of ideas. After "</think> the Solution section, provide the final, logical, and accurate answer, clearly derived from the expLoration in the Thought section. user
7 list positive integers has the fowing properties:
8 &'bullets The sum of the items in the list is $305.
9 $ s'bullets The unique mode of the list is $96.
$\bulletS The median of the list is a positive integer that does not appear in the list itself.
Find sum the squares all the items in the list. 12 Present the answer in LaTeX format: \boxod[Your answor). assistant
TROLL 14B Response
<think> </think>
Thought: Okay, let's try figure this out. The list has positive integers. Sum is 30. Unique mode is 9, so 9 appears more times than any other number, and no other number appears the same number of times as The median is positive integer in the list. Let me think about the median.
Median the middle value when sorted. If the number of elements is odd, median is the middle element. If even, it's the average of two middle numbers. But the median is positive integer not the list. So if the number elements is odd, the median is an element of the list, which contradicts. Therefore, the number of elements must be even. Let me denote the number elements, even. So n is even, say n=2k. Then median is average of k-th and (k+1 1) elements. Since median is integer, so they are both integers and their sum is even. So they are both even or both odd.
the average of those two elements is integer,
Also, mode is 9, so appears at least twice. And other number appears that many times. Let me try to find possible lists.
Let me try with n=6 elements. Then median is average of 3rd and 4th elements. Suppose elements sorted: a, b, c. d, f. Median is (c d)/2, which is integer, and not in the list. So d is even, and (c + d)/2 not equal any element. Sum a b + d+ 30.
Mode is 9, so appears more than any other number. Let's try with two If two 9s, then other numbers can appear most once. Let's try.
two 9s. Let me try to have two 9s. Let me assume elements are [x, y, Z, W, v. uJ. sorted. Suppose two 9s. Let try positions. If two 9s, they could be in positions and 2, but then median is (c + d)/2. If two 9s, maybe c and are something else.
Wait, if there are Suppose list is [1. 2, 3, 4, 9, 20J. Sum is 1+2+3+449420-39 too big.
9s, and others are unique. Let try an example.
Need sum 30. Maybe If two 9s, then other numbers are distinct. So total numbers: 9,9,a,b,c.d. All a.b,c,d distinct and different from 9. Sum: 9+9+a+b+ctd=18
numbers. Maybe more elements. Let me try n=6.
+ + + 30 =+b+++d==1
36

---

Published as a conference paper at ICLR 2026
Also, sorted list: a, b, d, 9, 92 No, depends on values. Wait, sorted list would be [a, b, c, d, 9, 9] if < b S d <9. But median is (c + dy/2. Which is integer, and not in the list. If a, c, are distinct and different from Let me try numbers.
Possible a, b, d: positive integers, distinct, not 9, sum to 12. Possible set: 1, 2, 3, (sum 12). Check they are distinct and not Yes. Then sorted list: [1,2,3,6,9,9] Median is (3 + 6)/2 4.5, not integer. Not good.
Another set: 1, 2, 4, 5 (sum 12). Sorted list: [1,2,4,5,9,9]. Median is (4 +5)/2-4.5. Still not integer.
Another set: 1, 4, 4. But duplicates. Not allowed since mode is unique 9. So can't have two 4s.
Another set: 2, 6. Again duplicates. Not allowed.
Another set: 5, Duplicates again.
How about three 9s? If three 9s, then other numbers can appear at most two times. Let me try n=7 elements. Wait, but median would be 4th element, which is in the list. But median is not in list. So must be even. So n=6, 8, etc.
Try n=8 elements. Then median average of 4th and 5th elements. Let me try with three 9s. So numbers: b, Sum is 27 + + b + c d + 30 + a + + + 3. Since positive integers, they are 1,1, 1,... but 0 not allowed. So impossible. n=6 elements, three 9s: sum would be 27 + others. If n=6, three 9s, others sum to 3. Only possible if other numbers are 1,1,1. But then mode is 9 and 1, which contradicts unique mode. So invalid.
So back to n=6, two 9s. Maybe the median is integer. For example, if and are both even or both odd. For example, if c=3, d=5: median 4, which is not in list. If list is [1. 2, 3, 5, 9, 10). Sum is 1+2+3+5+9+10=30, Mode is 1,2,3,5,9,10 all appear once. No mode. Not good. Wait, need mode 9. So need more 9s. Maybe four 9s. n=7 elements: median is 4th element. Which would be 9, but median is not in list. Contradiction. n=8 elements: median is average of 4th and 5th elements. Let me try n=8. Four 9s. Others can appear at most three times. Sum: 4*9=36 >30. Too big.
n=6 elements, two 9s. Maybe other numbers have duplicates but less than two. Wait, no, two 9s, others can appear once. So previous attempt.
Wait, what there are two 9s, and two of another number? But mode unique, so no. So all others appear once.
Wait, maybe n=5 elements. Median 3rd element, which list. But problem says median not in list. So must be even.
Maybe n=4 elements. Median is average of 2nd and 3rd elements. Let's try n=4. Two 9s. Sum is 9+9+a+b=18 or something. If sorted list: [a, b, 9, 9]. Median is (b +9)/2. Must be integer, so +9 even =b odd. Also median not in list. So +9)/2 o a, b, 9. Let me try b=5, then median is (5+9)/2=7. 7 not in list. a=12 -5=7. So list is [7,5,999]. But sorted: [5.7,99]]. Median is (7+9)/2=8, which is not in list. Okay. Mode is 9. Sum is 5+7+9+9=30. Perfect! Wait, sum is 5+7+949=30. Yes. Median is 8, which is not list. Mode 9, unique. All conditions satisfied.
+b=30 =a +b=12. and are positive integers, sorted:
$9
So the list [5,7,999]. Sum of squares: 25 + 49 81 81 25 49 74, + 81 = 155, 155 +81= 236
Wait, but wait: the median is (7 +9)/2-8,, which is integer and not in list. Mode is 9, which appears twice, others once. Perfect.
So answer is 236
Wait, but let me check if there are other possible lists. For example, a=3, b=9, but then b=9, a=3. List sorted [3.9.99]. Mode is 9, median is (9+9)/2-9, which is in list. Invalid. Another case: a=5, b=7 as above. Another possibility: a=1, b=11. But b=11 >9, but sorted list [1.11.9.9] is invalid, need to sort: [1,9,9.11]. Median is (9 +9)/2=9, which is in list. Invalid.
a=3, h=9; sorted [3.9.99], median 9 invalid.
a=5, h=7: valid.
a=7, b=5: same above.
a=2, b=10: sorted 12,9,9100]. median (9+9)/2=9 invalid.
37

---

Published as a conference paper at ICLR 2026
a=4, b=8: sorted [4,8,9.9] median (8+9)/2=8.5 not integer.
a=6, b=6: sorted [6,6,9.9]. mode is 6 and 9, invalid.
Thus only possible list is [5,7,9,]], sum of squares 236 </think>
Thought: After checking all possibilities, the only valid list is [5.7,9,9], which satisfies all conditions.
Solution:
The sum of the squares of all the items in the list is 236
Clip 14B Response
<think> </think>
Thought: Okay, let's see. need to find a list of positive integers where the sum is 30, the unique mode is 9, and the median a positive integer that doesn't appear in the list. Then, have find the sum of the squares of all the items.
First, let's recall what each term means. The mode is the number that appears most fre- quently. Since it's the unique mode, 9 must appear more times than any other number. So, there has to be at least two 9s, and no other number can appear that many times.
The median is the middle number when the list ordered. the list has an odd number of elements, the median is the middle one. If even, it's the average of the two middle numbers. positive integer that doesn't appear in the list. So, the list has even number of elements, the average of the two middle numbers must be an integer not in the list. odd, the middle number must not be in the list. But since the median positive integer, and the list contains positive integers, that's possible.
But the problem says the median
Let me try to figure out the possible lengths of the list. Let's denote the length of the list as
If n is odd, say n = 2k + 1, then the median is the (k+1 1)th element. This median must not be in the list. But if the median is not in the list, then the (k+1)th element is some number present in the list. But all elements are positive integers, so the median must be an integer. But how can the median not be in the list? For example, if the list is [1, 2, 3, 4, 5], the median is 3, which is in the list. But if the list is [1, 2, 4. 5], the median is (2+4)/2 = 3, which not the list. So, for even n, the median can an integer not the list. So, maybe the list has an even number of elements. Lett' assume that. Let's try n= 4. Then the median is the average of the second and third elements. Let's say the list is [a. b, c, d], sorted. Then median is (b + c)/2. This must be integer not in the list.
But also, the sum 30. And the mode is 9. So, 9 must appear at least twice. Let's try to construct such list.
If mn=4, 4, and mode 9, then at least two 9s. Let's say two 9s. Then the other two numbers must be different from each other and from or of them could be same another but not more than once. But since mode is unique, no other number can appear twice.
So, for example, [x. 9, 9, y]. sorted. Then the median is (9 9)/2 9. But is in the list, which violates the condition that the median is not in the list. So that's no good.
So, if we have two 9s, and the median not 9, then the two middle numbers must not both be 9. So, maybe the list is [a, 9, 9, b], but sorted. If a and b are such that the median is not But a <= <= b, then the median is 9)/2 = 9. So that's not possible. So, maybe we need more than two 9s?
third element. the list is [a, b. c. d], sorted. Then the median is 9. But is in the list, which violates the condition. So that's no good.
we have three 9s. Let's try n = 5. Then median
If 6. Then the median is the average of the third and fourth elements. Let's say the list is [a, b, x, y, c, d], sorted. Then median is (x + y/2. This must be an integer not in the list. Also, mode is So, 9 must appear more times than any other number. Let's say 9 appears three times. Then other numbers can appear at most two times.
Let me try construct such list. Let's ' say the list has three 9s. Let's arrange them in the list. For example, [a, b, 9, 9, 9, d]. sorted. Then the third and fourth elements are 9 and 9. So the median (9 + 9)/2 which is in the list. Not allowed.
38

---

Published as a conference paper at ICLR 2026
So, maybe the three 9s not in the middle. Let's say the list is [a, 9, 9, b, cl, sorted. Then the third and fourth elements are and 9. Again, median is 9. Not allowed.
So, maybe the three are spread out. For example. [a. 9, b, c, 9]. sorted. Then the third and fourth elements are b and 9. So the median is (b 9)/2. This must be an integer not in the list.
Let me pick b such that (b + 9) is even. So, b must be odd. Let's say b= 7. Then median is (7 + 9)/2 8. So the median, and it's not the list. That works.
So, the list is [a, 9. 7, 9, 9], sorted. Wait, but sorted order would be [a. 7, 9, c. 9]. But that's not sorted unless a <= 7 and c >=
Let me make it sorted. Let's say <= <= <= <= So, <= 7, and >= 9. But since there are three 9s, maybe с is 9. So the list [a, 7, 9, 9, 9, 9]. But then the third and fourth elements are 9 and 9. Median is which is in the list. Not allowed.
Hmm. So, maybe the three 9s are not consecutive. Let me try [a. 7, 9, 9, c, d], sorted. Then the third and fourth elements are 9 and 9, Median is 9. Not allowed.
So, maybe we need four 9s. Let's try that. If there are four 9s, then the mode is 9, and no other number appears more than three times.
Let's try n 7. Then the median is the fourth element. If the list is [a. b, 9, d, e, f], sorted. Then the median is 9, which is in the list. Not allowed.
n 8. Then the median is the average of the fourth and fifth elements. Let's say the list is [a, b, c, x, y, Z, W, vI. sorted. Then median (x + y)/2. This must be an integer not in the list.
If we have four 9s. Let's say the list is [a. b, c, 9, 9.9.99 9, 9, d], sorted. Then the fourth and fifth elements are 9 and 9. Median is 9, which is in the list. Not allowed.
So, maybe the four 9s are spread out. For example, [a. b, 9, c, d, 9, e, 9I, sorted. Then the fourth and fifth elements this is an integer not in the list.
с and d. So, median is (c + d)/2. Let's pick and d such that
Then median (7 + 9)/2 Not in the list, So that works. So the list is [a, b, 9, 7, 9, 9, 9I, sorted. Wait, but sorted order would be [a, b, 7, 9, 9, 9,
Let me try
and d
9]. So, and must be <= 7, and e >= 9.
Let me pick and b as small as possible. Let's say = 1, b= 2. Then the list is [1, 2, 9, 9, 9, e, 9]. Sorted, it would be [1, 2. 7, 9, 9, 9, e, 9]. But e must be >= 9. Let's say e == Then the list is [1, 2, 7, 9, 9. 9, 9]. But then the fourth and fifth elements are 9 and 9. Median is the list. Not allowed.
9, which
So, must be greater than 9. Let's say 10. Then the list is [1, 2. 7, 9, 9, 9, 10, 9]. Sorted: [1, 2, 7, 9, 9, 9, 9, 10]. Now, the fourth and fifth elements are and 9. Median 9, which is in the list. Not allowed.
So, maybe с and d are not 7 and 9. Let's try c= and d = 10. Then median is (7 + 10)/2 = 8.5, not integer. Not allowed.
6 and d == 10. Median is (6 + 10)/2 = 8. Good. So, the list is [a, b, 6, 9, 10, 9, e. 9]. sorted. Wait, sorted would be [a, b, 6. 9, 9, 9, e, 10]. So, a and b <= e >= 9.
Let me pick = 1, b= 2. Then the list is [1, 2, 6, 9, 9, e, 10]. Sorted: [1, 2, 6, 9, 9, 9, e, 10]. The fourth and fifth elements are 9 and 9. Median is 9, which in the list. Not allowed. So, maybe the fourth and fifth elements are not both 9. Let me try to have the four 9s spread out more. For example, [a, b, 9, c, d, 9, e, 9], sorted. Then the fourth and fifth elements are and d. Let mc pick = and d 8. Then median is (7 + 8)/2 = 7.5, not integer.
c c= 7 and = 9. Median is 8. So. the list is [a. b. 9, 7, 9, 9, e, 9], sorted: [a. b, 7, 9, e, 91. Again, fourth and fifth are 9 and 9. Median is 9. This seems tricky. Maybe I need to have more than four 9s? Let's try five 9s. If n 9. Then the median is the fifth element. If the list is [a, b, c. d, 9, e, f, g. h), sorted. Then the median is 9, which is in the list. Not allowed.
If 10. Then the median is the average of the fifth and sixth elements. Let's say the list has five 9s. So, [a, b, c, d. 9, 9, 9, 9, 9, el, sorted. Then the fifth and sixth elements are 9 and Median is which in the list. Not allowed.
So, maybe the five 9s are spread out. For example, [a, b, c, 9, d, e, f, 9, g, 9], sorted. Then the fifth and sixth elements are and e. Let me pick = 7 and e 8. Then median is (7
8)/2 = 7.5, not integer.
d 7 and e 9. Median is (7 + 9)/2 = 8. Not in the list. So that works.
39

---

Published as a conference paper at ICLR 2026
So the list is [a, 7,9,,, g, 9], sorted. Wait, sorted would be b, c,
9]. So, a, b. c <<= b,c
b. c Then the list [1. 2, 7, 9, 9, 9, g. 9]. Sorted: [1, 2, 7,9, 9, f, 9, 91. The fifth and sixth elements are 9 and 9. Median is 9, which is in the list. Not allowed.
Let me pick
So, maybe the fifth and sixth elements are not both 9. Let me try to have the five 9s spread out more. For example, [a, b, 9, c, d, e, f, 9, g, 9], sorted. Then the fifth and sixth elements are d and e. Let me pick d 7 and = 8. Then median is (7 8)/2 7.5, not integer. d= and c= 9. Median is 8. So, the list is [a, b, 9, c. 7, 9, f. 9, g. 9]. sorted: [a. b, 9, 9, 9, f,9, g, 9]. The fifth and sixth elements are and 9. Median is 9.
This is really challenging. Maybe need to go back to n 6 and try again. Let try
6, with three 9s. The median is (third fourth)/2. Let have the list [a, b, 7, 9, C, d], sorted. Then the third and fourth elements are 7 and 9. Median is (7 9)/ = 8. Not in the list. That works!
So, the list is [a. b, 7, 9, c, d], sorted. With three 9s. So, one of a, b, c, is Let's say e and are 9. So, the list [a, b, 7. 9, 9, 9I, sorted. Then and <=
Let me pick a b, 2. Then the list is 2, 7, 9, 9, 9]. Sum is +
Too much.
We need sum to be 30. So, let's adjust. Let me try 1, = 1. Then the list is [1, 1, 7, 9,
9]. Sum is I + + 7 9 + 9 36. Still too much.
a b 2, and change 7 to something smaller. But 7 is the third element. If I make it 6, then the list [1, 2, 6, 9, 9, 9]. Sum + + + + = 36. Still too much.
Maybe a= b, 1, and third element 5. Then list is [1, 1, 5, 9, 9, 9]. Sum is +
9+9+9 34. Still too much.
a= 1, 1, third 4. Sum is 1+1++++9+4+9+33
a= 1, b = 1, third 3. Sum
1, third
Sum 1=1++++++++
== 1. b= = I, third = 1. Sum 1 ++++++++++++ 30. Perfect!
So the list is [1, I, 1,9,, 9]. Sorted. Let's the conditions:
Sum is
+++ 9 Check. Mode is 9, which appears three times. Other
numbers appear once or twice. So unique mode Check. Median is (third + fourth)/2
(1 9)/2 = 10/2 5. 5 is not in the list. Check.
So this list satisfies conditions!
Now, the sum of the squares is 1 1 12 92 92 2 - =1+1+++++++++ =1
243 = 246.
So the answer 246
But wait, let me check there are other possible lists. For example, could there be a list with more elements?
But given the constraints, this seems to be the only possible list. So think
40


Reviews:
[{"review_id": "VWUzNzmc0J", "reviewer": "Reviewer_uGtd", "summary": "This paper introduces a novel method that replaces the standard PPO-like clip objective used in RL-based LLM fine-tuning. The authors posit that PPO's clipping is a crude approximation of a more principled KL-based trust region. TROLL substitutes this heuristic with a discrete differentiable trust region projection. For each token, the method solves a convex optimization problem that projects the new policy's output distribution onto a KL-ball centered around the old (sampling) policy's distribution. To make this practically feasible for large vocabularies, the authors additionally sparsify the predictive distribution and operate only on a small subset of the most probable logits, capturing the majority of the probability mass. Using implicit differentiation, this projection is fully differentiable, allowing gradients to propagate, unlike the gradient-cutting nature of PPO's clip. Experiments on mathematical reasoning benchmarks (DAPO-Math, GSM8K) show that TROLL consistently outperforms clipping in training stability, speed, and final success rates across various models (e.g., Qwen3, LLaMA 3).", "questions": "I am surprised about the low 0.1% clipping/projection rate displayed in Figure 14. Have you run ablations with different batch-sizes and model-staleness to better understand the effects of being on- vs. slightly off-policy?\n\nHave you investigated how the policy entropy evolves over the course of learning compared to baseline methods?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Novel and innovative approach to trust region based RL under the practical constraints of modern LLM finetuning. \nThe approach is theoretically well motivated and the paper shows convincing empirical results.\nThe paper is very well written; the appendix with detailed deviations is easy to follow. \nExperiments and empirical evidence is collected using a suite of differently open-weight model sizes and families; generally with very consistent and impressive results.", "weaknesses": "The experimental section focuses on consistently outperforming established benchmarks using a wide range of models and multiple datasets. This unfortunately leaves little space for more detailed analysis and ablations. The appendix D.6 however has at least some interesting additional detailed analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TW66hArReL", "reviewer": "Reviewer_cNjy", "summary": "Existing reinforcement learning (RL) algorithms for large language models (LLMs) including PPO and GRPO rely on clipping on policy ratios. While theoretically motivated by trust region optimization, clipping is only a crude approximation of the trust region. To this end, the authors propose TROLL, a differentiable trust region projection approach that directly enforces token-level KL constraints between discrete distributions. TROLL can directly replace the clipping objectives in PPO-like algorithms. Notably, TROLL projects the output distribution of the new, updated policy onto a KL-trust region around the old policy, and the projection can be computed in closed form.  To implement TROLL in practice, the authors introduce a sparsification scheme that discards the vast majority of effectively irrelevant, low-probability tokens. The authors verify the effectiveness of TROLL in RLVR settings on multiple LLM families across math datasets.", "questions": "- Why the authors claim that TROLL is for LLMs only? TROLL is an attempt to improve PPO, which is a general RL algorithm. Hence, TROLL should also be effective for standard (non-LLM) RL tasks. \n- If I understand correctly, it seems that the dual step size $\\eta^\\star$ needs to be calculated for every token?\n- Why the gain for GSPO much larger than GRPO and DrGRPO?\n- Could the authors please explain in more details why TROLL is much more complex than", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This work addresses an important challenge for LLM post-training.\n- The proposed method TROLL is novel. \n- The authors compare multiple LLM families with various sizes in their experiments.", "weaknesses": "## Presentation \nI personally find Section 3 (the most important algorithm section) to be poorly written and organized. In particular, I recommend the authors to include a pseudocode for the complete algorithm, from computation of $\\Tilde{\\pi}$, to its projection, to the calculation of final loss. Then write a separate subsection about the how the gradients are propagated and move the key results to the main text. As of now, it is difficult to understand both the forward and backward pass of the TROLL algorithm. This is the most critical concern to me. And I will update the evaluation if the presentation problem can be sufficiently addressed.\n\n## Related Work\nAs PPO is one of the most classical algorithms for RL, there have been many attempts to enhance its clipping objective, e.g., \n- with adaptive clipping [1],\n- with KL-regularized surrogates [3].\n\nIn particular, I am wondering why the following works of PPO (although not designed for LLMs) does not address the clipping problem of PPO. In other words, what are the unique challenges about LLMs that TROLL addresses and these prior RL works cannot. \n\nIn addition, I think [2] addresses the same challenge and the authors should consider comparing it. \n\n[1] Trust Region-Guided Proximal Policy Optimization.\n\n[2] BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping.\n\n[3] V-MPO: ON-POLICY MAXIMUM A POSTERIORI POLICY OPTIMIZATION FOR DISCRETE AND CONTINUOUS CONTROL\n\n\n## Experiments \n**Marginal Performance Gain.** Despite of the complex algorithm, the performance increase on the most popular combination for the math benchmark (Qwen + GRPO) is marginal. \n\n**Only Math Benchmark.** It would be beneficial if the authors can include more diverse RLVR benchmarks other than the math benchmarks. While it is appreciated that the authors conduct experiments on extensive benchmarks, all of them are math benchmarks and this can lead to questions about TROLL's scalability to other types of post-training tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qXlE6sMrIF", "reviewer": "Reviewer_YCtx", "summary": "The paper presents a way of fine-tuning LLM with RL that uses a more principled approach to ensure the network updates are within a trust reagion by using a token-wise kl constraint objective. This is different from the standard way of fine-tuning LLM's with algorithms like MPO that use clipping. The trust reagion is also differentuable which is important for updates. The paper also presents a way of sparcificaition for scalability where only a subset of tokes is used to compute the trust region.", "questions": "How would this scale to bigger models?\ncan you provide evaluations on other tasks than mathematical reasoning to show that TROLL is generally outperforming the baselines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper and the algorithm are presented well and intuitively. TROLL does outperform the baseline even if only by a small margin, nevertheless I think there is a big value for a more principled approach than clipping.", "weaknesses": "The gain in performance is not super big, also the scale of evaluation is a bit limited as only smaller models have been tested so it is unclear how well this will scale. Additionally was seems a bit narrow is the choice of task domains that are heavily focused on mathematical reasoning tasks. TROLL should be useful in a variety of tasks and not only for one specific domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GbGoi9Vl2l", "reviewer": "Reviewer_RHmH", "summary": "In this paper, authors considers PPO-like approaches, more specifically approaches that make use of a clipped importance sampling ratio. Instead of clipping this ratio, they propose to use a projection that keeps the KL within a certain range. In practice, this is an extension of Otto et al (2021) from Gaussian distributions to categorical distributions, with an additional spare and efficient representation of token distributions to cope with the usually huge vocabulary / action space of LLMs. The proposed approach is quite general (in replacing clipping by this projection for any methods making use of this clipping). A quite thorough experimental study is provided, showing the advantage of the approach compared to clipping, and providing interesting ablations, including illustrating the small computational overhead.", "questions": "### Clarity\n\n* Maybe nitpicking, but the paper talks about on-policy approaches all along. If it was truly on-policy, there would be no importance sampling and then no clipping.\n* It is unclear what is the overall optimization problem. It seems that problem (1) (RL objective) is solved with problem (3) (trust region projection), but how do they interplay? The term $\\tilde{\\pi}$ is the optimization variable of Eq (1) (as a side note, it is much clearer to write $J_{ratio}(\\tilde{\\pi})$ rather than just $J_{ratio}$, it is a function, making the argument explicit helps). Let call $P(\\tilde{\\pi})$ the result of Eq (3) (the argmin). Do we solve for $J_{ratio}(P(\\tilde{\\pi}))$ or something else? How does it relate to the classic KL constrained RL problem (eg as presented in TRPO)?\n* In Eq (5), it is not clear what gradient clipping means in this context, how it can be applied (and why) only partly, or what is the overall justification for doing Eq (5), and its implications. It is not even clear what policy is optimized there. Overall, this should be really clarified. \n* What is also unclear is towards what we regularize (what is $\\pi_{old}$). We can regularize towards two policies, the initial one, and the previous one (in a policy iteration context, what is fundamentally PPO or TRPO). Often, regularization is made towards both, through different term (eg, GRPO consider regularization towards both, the previous one through importance ration and clipping and the inital policy through the regularization term). Please clarify what $\\pi_{old}$ is in your case.\n\n### Related works and baselines \n\nGiven that the paper addresses the limitations of clipping, it misses a few approaches from the literature, in the context of LLMs, with the same or a very similar objective, either proposing an alternative to clipping the importance sampling ratio, or even removing the importance sampling ratio. Notably, the following papers could be discussed, and possibly considered as baselines:\n* [A] generalizes clipping by clipping the importance ratio within the gradient rather than within the objective, asymmetrically \n* [B] is a policy-gradient approach (in the sense of optimizing for a policy), relying on an additional value network but without any importance sampling, while being off-policy\n* [C,D] is an off-policy policy gradient without importance sampling\n* [E] is a method relying rather on Q-functions, but as so being off-policy without importance sampling (see also references therein for older approaches)\n\n[A] Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs, Le Roux et al, 2025.    \n[B] Offline Regularised Reinforcement Learning for Large Language Models Alignment, Richmond et al, 2024.   \n[C] Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion, Flet-Berliac et al, 2024.   \n[D] Command A: An Enterprise-Ready Large Language Model, Cohere, 2025.   \n[E] ShiQ: Bringing back Bellman to LLMs, Clavier et al, 2025.   \n\n\n### Misc\n* Eq (4) there’s probably a $\\log$ missing around $\\tilde{\\pi}$", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is overall well written and structured\n* Clipping is a commonly used heuristic, and any approach improving it can have a quite wide impact. This approach in particular is well motivated (clipping is indeed a heuristic approx of trust region) and seems to be quite efficient, with specific care to specificities of LLMs (like huge action space)\n* The experimental study is quite thorough, provide interesting ablations, and showcase the advantage of the proposed approach in many cases.", "weaknesses": "* There are clarity issues about the proposed approach, notably what optimization problem is really solved, how the trust-region projection interplays with the more general RL problem, or what justifies the heuristic of Eq (5). See questions below for more details.\n* Given that the paper studies an alternative to clipping, that’s a bit a pity that the authors do not consider as related works or even possibly baselines other alternatives to clipping, including approaches that even avoid relying on importance sampling. See questions below for more details and specific references.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)