Jackpot: Align Actor-Policy Distribution for scalable and stable RL for LLM
Authors: Zhuoming Chen
Hongyi Liu
Yang Zhou
Haizhong Zheng
Beidi Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
Reinforcement Learning
Primary area: reinforcement learning

Abstract: Reinforcement learning (RL) has become an increasingly important paradigm for improving large language models (LLMs) on alignment, reasoning, and coding tasks, yet it remains extremely costly. The majority of training time is spent on rollouts. Allowing actor and policy distributions to differ could unlock substantial scalability and efficiency benefits, such as supporting large-batch or asynchronous training, and even enabling a lightweight rollout model. However, existing importance sampling–based corrections for distribution mismatch suffer from an inherent trade-off between stability and training performance. To tackle this problem, we propose Jackpot, which leverages Optimal Budget Rejection Sampling to directly reduce the gap between actor and policy distributions. For efficiency and stability in practical training, We introduce an efficient probability estimation strategy based on Top-$K$ logits with batch bias correction, and designs a stabilized Jackpot-PPO loss that jointly accounts for both the importance sampling ratio and the trust-region constraint in PPO. Empirically, our method achieves stable improvements in large-batch and asynchronous training, and in extreme off-policy training it substantially delays the onset of collapse and delivers competitive performance. Specifically, we achieve 20\% improvement on AMC benchmarks and ~8\% AIME benchmarks over the off-policy baseline under 128$\times$ actor-policy update ratio for Qwen3-4B-Base and 64$\times$ for Qwen3-8B-Base, while achieving greater stability and better performance than prior off-policy RL methods under extreme settings.

Published as a conference paper at ICLR 2026
JACKPOT: ALIGN ACTOR-POLICY DISTRIBUTION FOR SCALABLE AND STABLE RL FOR LLM
Hongyi Liu' Carnegie Mellon University {zhuominc, honggl! 12, yangzhof, zaozz, be bwwdbbiddddedddcccc cmu edu
Zhuoming Chen"
Yang Zhou'
Haizhong Zheng
Beidi Chen
ABSTRACT
Reinforcement learning (RL) has become an increasingly important paradigm for improving large language models (L..Ms) alignment, reasoning, and coding tasks, yet it remains extremely costly. The majority of training time is spent on rollouts. Allowing actor and policy distributions differ could unlock substantial scalability and efficiency benefits, such as supporting large-batch or asynchronous training, and even enabling a lightweight rollout model. However, existing importance sampling-based corrections for distribution mismatch suffer from an inherent trade-off between stability and training performance. To tackle this problem, we propose Jackpot, which leverages Optimal Budget Rejection Sampling to directly reduce the gap between actor and policy distributions. For efficiency and stability in practical training, We introduce an efficient probability estimation strategy based on Top-K logits with batch bias correction, and designs stabilized Jackpot-PPO loss that jointly accounts for both the importance sampling ratio and the trust-region constraint in PPO. Empirically, our method achieves stable improvements in large-batch and asynchronous training, and in extreme off- off-policy training substantially delays the onset of collapse and delivers competitive performance. Specifically, we achieve 20% improvement on AMC benchmarks and 8% AIME benchmarks over the off-policy baseline under 128x actor-policy update ratio for Qwen3-4B-Ban and 64x for Qwen3-8B- 8B-Base, while achieving greater stability and better performance than prior off off-policy RL methods under extreme settings. More details of the project are available at ptsttss///iffi ai lab github. 1o/ jpt websitte ite/
INTRODUCTION
Reinforcement leamning (RL) has demonstrated substantial effectiveness the post-training large language
models (LLMs), yielding significant improvements in domains such mathematics (Guo et al.) 2025
Azerbayev et al 2023), coding (Jimenez et al. 2023 Ouyang et al.) (025), and agentic tasks (Liu et al. 2023)
Despite these successes, RL remains computationalyy ly expensive (Sheng et al. 2025| Fu et al. 2025| Zheng
et al. 2025b). with the majority of the training cost, often exceeding 70%, attributed to rollouts, wherein
LLMS generate solution trajectories for tasks order compute rewards. If actors and policies were allowed to follow different distributions, the scalability and efficiency of RL could be elevated to an entirely new level. For instance, such flexibility would make it possible support large-batch or asynchronous training, thereby improving the utilization of serving systems (Zheng et al. 2025a). Moreover, quantized or sparse models, and even distilled smaller models, could be deployed as actors to further enhance efficiency. In practice, however, the mismatch between actor and policy distributions often leads to instability and severe degradation performance (Liu al. 2025). posing a fundamental barrier to the reliable adoption of these techniques.
When the distribution gap between the actor and the policy becomes large, existing importance sampling (IS)-based correction methods (Liu et al. 2025 Wu et al. 2025b; Fu et al. 2025) perform suboptimally compared with the baseline PPO. In practice, truncated importance sampling methods (TIS) either underperform or exhibit substantial convergence gap to the on- on-policy baseline when the truncation threshold is low conservative, or TIS crashes before policy platcaus from RL training when the truncation threshold Pre(a)
set to a higher or aggressive value. The importance weight used by TIS Puga(a) Once the actor drifts too far, many tokens that the actor samples with high probability have very low probability under
"Equal contribution, alphahetically ordering based lastnames

---

Published as a conference paper at ICLR 2026
AIME 2025 avg@116
Actor-Policy KL Divergence
2.00
N: Algrenantt Cr-Palicy
Ny 4iginst
172
jeosot (Duss)
T5
15.0
150
1.25
10.0
1.00
310016
0.75
0.59
2.5
2000
4000
0000
9000 10030
0.00
2030
4000 6203 2002
10000
Total Training Exemples
Step
Figure 1: RL training requires actor-policy maintaining strong probability distribution alignment. When actor and policy aren't aligned, they will result training collapse. Here we show training setting use Owen3-1.77-BBB Base model training rollout to train a Qwen3-8B BB B-Bass model policy. Without any alignment procedures, training collapses (pink). Prior method TIS (green) also show significant gap towards Qwen3-8B-Base on-policy baseline (purple), while collapsing, using TIS sees KI. divergence also violently increasing. Our proposed method, Jackpot (yellow) maintains small KL divergence between actor and policy model probability distribution, while showing stable and competitive training convergence on-policy setting.
the policy, since Panl Punges These actor trajectories effectively treated as low-likelihook samples by the policy, causing TIS to train on tokens the policy would never select at inference and creating a widening wain-inference mismatch. This naturally raises the following question: Can we directly modify the actor's sampling distribution and sampled trajectories to reduce its distributional gap to the policy?
Rejection sampling, which can simulate target distribution from accessible proposal distribution, has a long history and has been widely applied in fields such as biology (Carrella et al. 20224), the social sciences, machine learning (Naesseth al., 2016), and statistics (Martino Miguez 2011 Gilks et al.) 1995). However, a direct application of rejection sampling is prohibitively expensive in the context of RL. Specifically, token i must be accepted with probability Xg. which typically possess vocabularies exceeding 100,000 tokens, this constant C can become extremely large, since the majority of token probabilities are exceedingly close to zero. As result, nearly all tokens proposed by the actor are rejected, rendering naive rejection sampling impractical for large-scale RI. and leading to prohibitively low sample efficiency. Fortunately, Optimal Budget Rejection Sampling (OBRS) (Verine al., classical rejection sampling and, although it does not enforce the actor
pt where \= A=max;", Pi For contemporary LLMs,
2024) relaxes the requirement distribution to be identical to the policy distribution, it provably reduces their distance and guarantees that for any rejection ratio the adjusted actor distribution is strictly closer to the policy distribution than the unadjusted one. This provides us with an opportunity fundamentally different from standard rejection sampling.
However, applying OBRS directly in RL systems introduces several technical challenges. First, PPO relies on the existence of trust region stabilize the training process, which means that modifying the actor probabil- ities through OBRS may compromise training stability. Second, in order to compute the true probabilities of the remaining tokens after the rejection process, OBRS requires access to the probabilities of all tokens in the vocabulary, which imposes significant memory overhead for modern LLMS with extremely large vocabularies.
In this paper, we propose JACKPOT, which consists of three key components. First, an OBRS-based masking mechanism ensures that the adjusted actor distribution remains strictly closer to the policy distribution. Second, an efficient probability estimation strategy introduced, which leverages Top-K logits together with batch-wise bias correction to approximate the hall-vocabulary probabilities while mitigating memory overhead. Third, we design stabilized JACKPOT-PPE loss that jointly accounts for both the importance sampling ratio and the trust region constraint in PPO, thereby preserving training stability.
To validate the effectiveness of our method, we consider two representative scenarios. (1) Large-batch training, In this setting, the LLM generates up to 128 mini-batches in single rollout step, which enables more efficient utilization of serving system hardware resources. Empirically, we observe more than a 2x improvement in mnd-to-end RL throughput compared on-policy training. However, this comes the cost substantial policy drift during training, resulting in significant divergence between the rollout actor and the updated policy. (2) Extreme off-policy training. In this setting, we employ fixed model for rollouts
2

---

Published as a conference paper at ICLR 2026
that is different from the one being optimized. This configuration introduces severe distributional mismatch, under which standard approaches typically fail and training collapses rapidly.
We organize the remainder of this paper as follows.
In Section 2 we formally introduce the distribution mismatch between actors and policy, discuss sources under different training scenarios, and review related work.
In Section 3 we describe our application of OBRS to RL and validate its effectiveness through numerical experiments and empirical observations.
S we provide detailed description of JACKPOT-PPO, including three key components: (i) OBRS masking, (ii efficient probability estimation with Top-K logits and batch bias conrection, and (iii) a stabilized LACKPOT-PPO loss that jointly considers importance sampling ratios and PPO's ' trust-region constraint.
In Section 5 we present experiments on Qwen models and mathematical reasoning tasks to validate JACKPOT. First, large-batch training, our method maintains stable learning, outperforming offline and TIS baselines and approaching the performance of the online setting. Second, in extreme off off-policy training, the proposed method substantially delays the onset of training collapse and achieves competitive performance.
Using JACKPOT, we achieve 20% improvement on AMC benchmarks and 8% AIME benehmarks over the off-policy baseline under 128x actor setor-policy update ratio for Swen3-4B-Baze and 64x for Qwen3-8B-Bar 8B--Bse, while achieving greater stability and better performance than prior off-policy RL methods under extreme settings. Overall, JACKPOT is simple to plug in, theoretically well-grounded, and holds the potential to enable more aggressive forms of off-policy RL.
2
BACKGROUND
In this section, first formalize the distribution mismatch problem that arises in RL for LLMs. We then review several strands of related work of JACKPOT.
2.1
PROBLEM SETTING: PPO OBJECTIVE AND ACTOR-POLICY DISTRIBUTION MISMATCH
We begin with the clipped objective in PPO (Schulman al. 2017). whose expectation can be written as _ (0)= min (ra(( A(cr))ccpprrrrccc)1_-1----
(I)
where (o(x)==ea.l Px)r is the likelihood ratio between the updated policy pe., and the reference policy Pief. and A(x) denotes the estimated advantage at decision x. Pinf is the inference distribution used to generate rollouts, is the reference policy distribution assumeed in the objective, and pe... is the updated policy distribution. In the standard process, it assumed that Pint = Pret, but in practice this assumption is often violated, leading to actor-polise distribution mismatch.
Distribution mismatch is common and arises for several reasons, such as minor discrepancies between the inference engine and the reference policy by FSDP engines, the use of stale or asynchronous data, or rollouts generated by approximated models (e.g. quantized, sparsified, or distilled). Such mismatches can destabilize training and therefore require additional mechanisms to correct or mitigate their impact..
2.2 RELATED WORK
RL for LLM. Reinforcement learning has been widely applied to L.Mss to improve human alignment, reasoning, coding, and other complex tasks. Beyond PPO, memory efficient methods have been proposed, including ReMax (Li et al. 2023). RLOO (Ahmadian et al. (2024), and GRPO (Shao et al., 2024). In addition, methods such SimPO (Meng et al. 2024) and DPO (Rafailov et al. 2023), which are based on offline
RL, have also been employed for human alignment. RL training systems for LLMs, such as Verl (Sheng al. 2025), AReal (Fu al. 2025). TRL (yon Werra et al. 2020), and OpenRLHF (Hu et al. 2024), have been developed to improve training throughput and scalability.
Distribution Mismatch Correction in RL, Actor-policy mismatch is common problem that has long been studied, e.g. Impala Espeholt et al. (2018). To alleviate the actor-policy distribution gap, the method introduces truncated importance sampling (TIS) to approximate the true PPO objective.
PPPO (PP0((0)= (0)= -Er-
min
Par(a)
min
(2)
(a)

---

Published as a conference paper at ICLR 2026
The truncation threshold C'is is for maintaining the stability of the range of the importance ratio. Recently, several methods apply the truncated importance sampling method to RL LLMs. Methods such as FlashRL (Liu et al. 2025). AReal (Fu et al. 2025), and LlamaRL (Wu et al. 2025b) address distribution mismatch by introducing (truncated) importance sampling ratios, typically of the form Pret/ Pints Pret/Pnt. to correct the impact of mismatch on advantage estimation. From system perspective, FP32 LM heads (Liu et al. (2025) and deterministic L.M
Inference (He & Lab 2025) are implemented to mitigate the numerical issue of serving systems when rollout.
In this paper, we proposed JACKPOT. Our method is orthogonal the above prior works. We directly modify Pior through rejection sampling and reweighting of the output probabilities so that the divergence between Pinf and the target distributions Piet is provably reduced. Moreover, techniques such as TIS can be applied on top of this improved distribution to further correct the remaining mismatch in a complementary way. JACKPOT offers a mechanism that shown to effective in stabilizing RI. training under severe mismatches.
3 CORRECTING DISTRIBUTION MISMATCH WITH BUDGETED REJECTION SAMPLING
One of the most fundamental challenges of modern RL framework for LLMs is the distributional mismatch between samples generated by our inference model (actor), or Piut- and the true reference policy distribution, or Po-. One way is through Importance Sampling, or adding an importance ratio term Phe However, as the Pua trajectories are sampled and Pio((z) can be small, the importance ratio sometimes blows up in numeric value. In practice, min( Pat Pee CC) is used Espeholt et al. (2018) 8) to cap out the dangerously large values, leading to huge bias in correcting the distribution misaligmmnnt Instead of solely relying on Importance Sampling, can we modify Piat and the sampled trajectories directly so that it is closer in probability distribution to pre??
One direct idea is Rejection Sampling (RS), or stochastically rejecting tokens in the tractories sampled with Pmut based on the difference between the two distributions. Once a token is rejected, it contributes nothing to the loss and gradient calculation. While canonical rejection sampling could resolve this, its application here-using Piot as the proposal and pe... as the target-is impractical, as rejection sampling aims for identical probability distribution after correction. The potentially large divergence between these distributions would lead to a prohibitively low acceptance rate, essentially leading to most tokens being rejected. The data efficiency of RL training will be severely degraded, failing to meet practical requirements.
To overcome this, we adopt the principled approach of Optimal Budgeted Rejection Sampling (OBRS) (Verine et al. 2024 This technique reframes the problem: instead of demanding perfect adherence to the target distribution at the cost of sample efficiency, it seeks the optimal rejection rule that, for a given target acceptance rate (a "budget"), produces distribution close possible the target. This is precisely the trade-off our problem requires.
The method employs a scaled acceptance probability, where scaling factor À is chosen to meet the desired sample throughput. A token a sampled from the proposal Pint is accepted with probability (a defined
as (a(a)= min L.
Parpi(a)
X-paela)
3.1 NUMERICAL SIMULATION
Crucially, this calibration is highly efficient; the acceptance rate remains high even when there is a large initial KI. divergence. The impact on distributional alignment is dramatic: significant reduction in KI. divergence is observed with high acceptance rates. By systematically damping the most extreme probability ratios, OBRS produces distribution that is not only provably closer to the on-policy target but also primed to yield more stable and effective PPO/GRPO policy updates.
3.2 THRORETICAL GUARANTERS
OBRS possesses proven optimality. It has been established that for any desired average acceptance rate ā E IJ. there exists a corresponding scaling factor C' that achieves it. Crucially, among all possible rejection rules that satisfy this budget, this scaled acceptance rule is the unique one that minimizes the Kullback-Leibler (KL) divergence to the target distribution pe.. - A formal statement and proof of this theorem are provided in Appendix $A.2 This guarantee ensures we are using the provably best method for trading sample efficiency for distributional accuracy.
4

---

Published as a conference paper at ICLR 2026
2/ d-way al grme moves toward Scatter
Accestance rate original How riginnt
twith referencet
y- (perfect matcho
orignnl Ac p.ar
ofter Sa quaol
24
3.000
bagge pobitaty (5)
Taed 1111
2.000
034
3.8
Beravlor probatiity A 0,540
Orggral ML Kijp ir inf)
Origina KL: Kup trlll int
Figure 2: OBRS calibration results across three views: (a) per-token probability-ratio clipping pulls the model distribution toward the target, (b) acceptance remains high (+s 95%) even at large initial KL, and (c) overall KL is reduced by roughly an order of magnitude.
The scaling factor C' acts as an explicit control knob for this trade-off. A larger C' pushes the post-rejection distribution closer the true target Pr.... at the expense of lower acceptance rate, while smaller C boosts throughput at the cost of higher divergence. In our experiments, we find C'= 1 to be a robust default.
This formulation also guarantees that applying this sampling technique always better than using the original inference distribution Piut directly. The post-rejection distribution Pkepe.C' is strictly closer to the target pe... in KL divergence than the original Piat for any choice of C'> O. We provide a summary this proof tailored to our notation in Appendix SA Our algorithm for implementing this procedure is also detailed in Appendix {
4 JACKPOT: DESIGN AND METHODOLOOGY
this section, we present details on the design considerations of JACKPOT. We show the token rejection criteria and reweighting procedures in Section, applying our rejection sampling to the PPO setup in Sections, and efficiency analysis in Section.
4.1 REJECTION AND REWEIGHTING
To bridge the inference probability distribution Piat and the target distribution Purgets we use the following critieria similar to Leviathan et al. (203). For token sampled by Pints x, we accept token with probability
min(1, Phargat (2)
>xpit(a))
Note that the above equation is P(x accepted x sampled by P)nr(r)). Once the token is rejected, it will be masked out and no longer participate in the loss and gradient propagation. After rejection, the distribution has expression:
min miin(Pin(xz) Pronar(ar)
POBRS-
min(pinr(a"))) (I') Pransa Prnosa(a")) (xx)
4.2 INTEGRATION WITH STANDARD PPO OBJECTIVE
Following Section 4.1 we have the following PPO objective and applied Truncated Importance Sampling.
P Er-pur
PPO
[(xt]
En
(5)
PPO -FEr Pae (P))(00)
min
Pnnlar) Par
C).(())
(6)
5

---

Published as a conference paper at ICLR 2026
Realistically, Plns Prey occurs and leads to training crashes. X
Ideal RL Training
Here is Jackpot.
Actor Pins" Pres
Rollout
Actor Pinf
Rejection Sampling min(1, Pre( Pinf
Probability Reweighting min(Pre ref Piny)
Policy Pref
x~Prof
Z
Policy Pres
-Piiff
The training is stable!
Rollout
The training after Jackpot stable!
Figure Illustration JACKPOT Pipeline focusing on Optimal Budgeted Rejection Sampling (OBRS) and Reweighting Procedures
On top of TIS, we can further corrects the Pint of interest by using our rejection sampling critieria and reweighting by.
min( (2) Pin((x)
Presla) .C) mmnn( min(maala (xpaeee
C)mmin max(A, Prer
).C)
Pint(a)
where Z End (an()ia((a)).. purte))). Therefore, instead of (6), we use the following PPO objective formulation,
(0) Eew Pat min Z (Z-maxA, Pee(()) oours
CPPO
Dint (x) ),0) (00))(())
(8)
4.3 WHICH POLICY TO APPROXIMATE?
Conventionally, we assume Puat = but empirically, we discover that for settings where RL training suffers from severe staleness, e.g. using large batch size or using asynchronized rollout/update cycles, approximating Pmnt + Pnew the latest updated policy edges out in performance. The rationale is that the reference policy too stale and too distant to the latest updated policy offer reliable trust region. that case, we adjust the conventional PPO objective in (2) to the following from importance sampling.
Eg [f(x)) = -Ez~Pan
Prel
f(x)]
99)
Pnew
Then, we can approximate Pnew using Par through our rejection sampling and reweighting. For the high staleness settings, we use the following approximating formulation,
Eneppe
Pief
f()) + Eewppe [min("
Pnew
-i)-min(
Peef
f22
(10)
Pnew
Pint
Pnew
where Pirt is the corrected distribution through rejection sampling and reweighting.
We then have the JACKPOT objective,
ourrs (ZinxZ- Prew (xc
),C2) min( Prel
.C2) .C2)
[(x)
(11)
Pnew 1. We offer the user to either use (8) to align to Pref Pnew
Piat
Throughout our experiments, we use À 1. depending on their target policy desired under their use cases.
4.4 STABILIZATION AND FEASIBILITY CHALLENGES
Implementing JACKPOT directly faces huge challenge of computational feasibility. Note that the weight's normalization constant, Z, requires sum over the entire vocabulary ([VI> 100,000), creating a crippling memory bottleneck from storing full logit vectors obatch_size X seq-len X severely restricts batch sizes, directly undermining the efficiency OBRS is intended to provide. Therefore, transforming this principled approach into production-ready algorithm requires non-trivial engineering: we must introduce mechanisms to both bound the importance weights for stability and develop computationally efficient, low-biat estimator for the normalization constant. To overcome the computational bottleneck of calculating Z, we employ top-k approximation, which we then de-bias empirically.
This

---

Published as a conference paper at ICLR 2026
4.4.1 TOP-K APPROXIMATION
The probability mass of language models is typically concentrated in a small subset of the vocabulary. We leverage this property by upproximating the over V with contains the most likely tokens from both the inference and current policies. Specifically, let top-k(p) be the set of tokens with the highest probability under distribution p. We define our approximation
much smaller set, V/. which
the union:
Vi= ((ppk((mmr) Utop-k(pe....)
The union is crucial because a token might be highly probable under one distribution but not the other, and the min function makes these overlapping regions important. The approximate normalization constant, Zaprox
is then: Zuppeox = Ev, min Pinr Pana(a))
4.4.2 BIAS CORRECTION
While efficient, this top-k approximation introduces systematic bias. Since the terms in the sum non-negative, omitting tokens from the full vocabulary V can only decrease the total sum. Therefore, our approximation is consistent underestimation of the true value:
For k=20, This bias could systematically alter the scale of the gradients during training. Fortunately, there is an elegant way to correct this. A key property of the framework is that the true normalization constant Z exactly equal the expected acceptance rate, a:
ā Pinr(a)) min
Paarge((a) (a) ' ">-Paar(a)
Σ
min Pinr(a), Pesrget (a)
-Z.
aEV
During the data collection phase (Algorithm 1 Phase 1), we compute unbiased empirical estimate of ã from the observed samples:
Number of accepted samples
Total number of proposed samples
This gives us two estimators for Z: the low-variance but biased Zuppoxx and the unbiased but higher-variance & We can combine them to create a de-biased, low-variance estimator. We compute a batch-wide calibration factor, K, by dividing the empirical acceptance rate by the batch-averaged Zapprox:
B 2i Zaponox
where B is the number of samples in the batch. We then apply this scalar correction each per-token Zapprxx value used in the loss calculation. This procedure scales our efficient top-k estimate to match the true expected value observed in practice, effectively removing the bias while retaining the computational benefits and lower variance of the top-k approach.
4.5 IMPLEMENTATION OVERHEAD ANALYSIS
JACKPOT is lightweight. First, JACKPOT requires no additional trajectories sampled, as the experiments we conducted in the extensive empirical studies section are using the same rollout width as the on-policy baseline. A critical distinction to Leviathan et al. (2023) is that JACKPOT doesn't reject all tokens from the first place in the trajectory where a token first rejects and resample from where. Instead, JACKPOT only mask out tokens using our rejection critieria, and no additional trajectories sampling needed. Second, no additional log, pucppprobb computation needed, since both Piet and Pnew will be computed by the standard objective (5), and JACKPOT only needs to reuse these probability distributions already computed. Third, no modification required on v...M. Since JACKPOT doesn't 't required special operator or numeric precisions, we directly based our implementation standard vLLM for rollout, without relying on custom kernels in vLLM.
JACKPOT indeed add minor additional overhead to standard PPO objective computation. The added computation comes from forcing vLLM to return top-K logprobs. Fortunately, since only use k=20 for our runs, the added extra compute only added less than 3% to the total compute. In contrast, JACKPOT helps models be trained using 64x or higher batch sizes by drastically alleviating the lack of convergence from the staleness of the actor model. Thus, comparing with the small batch and on-policy performance, achieve more than times speedup.
7

---

Published as a conference paper at ICLR 2026
Algorithm The Jackpot Algorithm
Require: Policies: current Prews reference Pref. inference Pint-
Require: Hyperparameters: OBRS threshold A, PPO clip e, Jackpot clips ei ,0, top-k count.
1: Convention: SG(-) denotes the stop-gradient operation.
2: Implementation note: Jackpot only reweights quantities from the standard rollout and PPOIGRPO
forward passes; it does perform extra model forward passes trajectory recomputation.
3: Phase 1: Efficient Rollout (Standard Generation)
4: Initialize experience buffer D D 0.
5: for each trajectory sampling step do
6: Single forward pass of Pin((-1s)), sample at ~Pin(: (+t))
7: From the same forward, compute and store top-k log-probabilities of part Top (8)).
8: Store (St.ae.uur(aa (e (t) (s))) (plus rewards, values, etc.) buffer D.
9: end for
Compute advantages A, using collected trajectories.
11: Phase 2: PPO Update with Jackpot Reweighting
12: for each mini-bated sampled from D do
13: // 1. Standard PPO Computation (reused by Jackpot)
14:
Forward pass Pnew and Per on the mini-bbtch to get logits, Pnew (at (t). Pret(as (t). and aop
Compute policy ratio: re(0)= Puew st)
(rer(aels))
16:
Compute vanilla PPO objective: Lrro = min(rr (0) (0))),,cciprr())111111))))))
// 2. Efficient Z-Approximation and Bias Correction (no extra forward passes)
18:
Construct approximation set V4:-- = (TTTP (s:). Compute
Pnew (I (t)
Zappoxx
min
Pinr(a st),
X
rEV.
Estimate correction factor using the OBRS-bases bias-correction procedure described in Sec. 4.4.2 (e.g., from batch-level OBRS statistics). Set corrected normalizer Z; + K: Zupprox.
21:
22:
// 3. Jackpot Weight Calculation
23:
OBRS weight: WOBRS = Z, max
X Pten(as Pei(
24:
Pjackpet = www.miinnwwBBs.... min
Peos (8)) .22
Parla (
// 4/4. Apply Weight to Loss
26:
Limal = =SG(ppuckpa)) Lyro-
27:
Update policy parameters new using gradient of -Cinal-
28:
end for
5 EMPIRICAL VALIDATION
In this section, we comprehensively test our method on two different and challenging misalignment settings. First, we evaluate our method on an extremely large inference batch size, while keeping the training mini-batch size the same. At the end of each caining-inference cycle, the staleness of the model weights the inference server will significantly amplified. Secondly, we evaluate our method using two separate models for training and rollout, an extreme setting where the output distribution gap is more severe than usual staleness in off off-policy RI. settings. We show that our technique enables the training model better benefit from tokens from completely distinct model. Because of the limit in space, we list detailed ablation studies on threshold selection and top-K K analysis in the Appendix
5.1 INFERENCE WITH LARGE BATCH SIZE AND TRAINING WITH MUCH SMALLER BATCH SIZE
RL training using much larger batch size for generation and smaller batch size for training is common in practice. It is usually due to either system limitation, where the generation is highly parallelizable and relatively memory-light compared training (saving optimizer states, etc.), is for better stability as
8

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation scores across benchmarks. TIS Adjustment is explained in Section B.1
GSMSK
MATH-500
AMC22 23
AMC12 2024
AIME24
AIME25
Models / Methods
Mean@4
Mcan@4
Mcan @4
M
Mcan @
Pass 16
Mcan@ 16
Pass@ 16
Qwen3-4B-Base on DeepScaleR-Preeee Preview Dataset (rollout butch size
2048; truin
batch size
32; 64x)
Policy
92.19
81.55
58.43
51.11
23.12
33.13
22.91
30.95
Off Policy
88.04
71.15
39.15
29.44
13.96
23.03
11.04
18.61
TIS
89.67
72.00
42.77
31.67
11.88
17.56
11.25
17.28
TIS Adjustment
92.76
79,50
$7.22
43,33
18.75
26.03
17.71
24.61
Jackpot (Ours)
92.24
80.05
53.92
50.00
20.63
29.48
18.13
23,63
Qwen3-4B- on DeepScaleR- Previiew Dulaset (rollout buitch size
4096; train batch size 32; 128x)
Off Policy
79.70
33.00
24.44
5.00
11.00
TIS Adjustment
20.70
19,10
7.80
5.00
1,00
4.00
1.00
2.00
Jackpor (Ours)
92.00
80.00
51,20
47,22
19.16
24.58
18.52
25.08
Qwen3-8B-Band -BB-ase on SaapsaalllR--Peiveeee Dataset (rollout batch size = 2048; train batch size = 32; 64x)
On Policy
94.24
93,99
28.95
54.44
28.95
37,89
22.50
28.54
Off Policy
91.05
77.15
50.60
40.00
18.54
28.67
14.16
21.98
TIS Adjustment
93.85
82.55
60.54
48.33
24.58
35.06
20.00
22.90
Jackpox (Ours)
94,01
83,05
63.55
54.44
26.87
36,23
20.41
26.57
Table 2: AMC22&33 results of two model training using various methods across the training steps. Mean @k/Passk
AMC22 &
Rollout Midel
Train Model
D
20
40
60
Vanilla GRPO
12.5.I.BBIT 0wen2.5:1.5BB1 Q2.5-15B-IT
Qwen2.5.3P 3-3e-Base
18...
3/58/59.0
32.2/61.5
33.3/59.0
30Q49.4
22.4/42.3
5.7/169
TIS
Q2.5.BB Base
18.1448.2
28.9/57.8
2/42/
32.5168.7
26.8/6.6
(/0
01
Jackpot (ours)
Owen2.5-3B-o
18.1488.2
29.4/59.0
31.2/57.8
33.9/60.2
31,962.77
28.8/44.2
13/4/13
AMC22 &
Rollout Model
Train Model
20
40
60
Vanilla GRPO
MATTU--T
Qwen2.5. 3B Base
18.1488.
23.3600.
22.6/57.8
24.7/53.0
17.434
39/14.
0
TIS
02.5-MATH--.5BBII
Owen2 (an22.533--Baa
18.14
25.8/59
25.0/59.0
21.7/55.4
0
0
Jackpot (ours)
-25-MATH-11BBB1
Owen2B-3B-Base
18.1448.2
27.3/57.8
29.4663.9
26.7/59.0
28.6600.
22,4/44.
29916
advocated by (Schulman et al. (2017). However, the delay in updates results in staleness, which we magnify for evaluating our method.
We use the best results under 30k examples for Qwen3-4B- Base as our metrics, and the best under 50k for Qwen3-8B-Base | The results of our run are summarized in table | On-policy RL generally converges much faster than off-policy runs. We observed that for less aggressive off-policy settings, our method's margin is quite small. However, for extreme settings, such as 4096-32 for Qwen3-4B-Ban -eesee and 2048-32 for Qwen3-8B-BB Base. We observed consistently that adS-adjusted crashes occurred way earlier than our settings, often resulting in inferior results. For example, Qwen3-4B-Ban TIS-adjusted fails to converge from the beginning. Also, as shown in Figure fig. (a). TiS-adjusted also crashes way earlier than ours under the 2048-32 setting. Across all settings tested, our method greatly improves the speed of convergence of the off-policy RI, while also resulting in performance numbers comparable to the on-policy runs.
Besides, much less aggressive setting, KV FP8 quantization result in performance crash shown in Figure 4b). Here, we apply only our rejection sampling algorithm, without clipping/tuncationn or any similar tricks. We show our performance recovers from crashes.
5.2 EXTREMR OFFPOLICY SETTINGS
In this section, we demonstrate our method's effectiveness on the extreme setting where the model under inference and training are fundamentally two separate models. Specifically, we hypothesize that our sampling algorithm can always filter out useful tokens for the model training, even given a completely exotic output response. We use Qwen2.5-3B-Ban as the trainer model, and we adopt different inference models from Qwen2.5-1.3 1.5B In-srutrttt to Owen2.5-Math 5n-ntrutct.. trained on MATH-8K [endryykk et al. 2222 dataset. We find out that our method can still improve itself under such extreme settings.
'Qwe334B-BB on-policy run crashes at 30k examples.

---

Published as a conference paper at ICLR 2026
Training Rewards Owere-BU-B
Exemples Cwen2.5-1 10..2255-1.55 MATH RolicubCwenn Base Tain
0.35
0.30
MATH--50 50000
0.25
0.20
0.15
On policy
0.10
TIS-adjusted
Oftpolicy
0.05
Only
TIS
jackpot (ours)
0.00 KV FPO
Jackpot (ours)
20000 30000 Total Examples
7500 Total Examples
Taining Staps (Batchsize 128)
(b)
Figure Overview of our empirical experiment results. (a) shows our proposed method's strength in correcting stale inference model distribution, and approaches the on-policy trend while maintaining training stability; (b) shows our proposed method to align output distribution can correct RL training instability even if applied alone; (c) The strength in our model's ability to align it in the most extremely misalignment setting where the rollout and the training model are separate different architecture, method shows early glimpse of hope and beats the baseline.
different distributions allows put
Shown in Figure 4 (c), we see that our run green curve shows clear improvement over our baseline GRPO and TIS settings. even increases by 12% MATH-500 under such an extreme misaligned setting. More results are summarized in Table showing that the improvement also exists on AMC problems. We believe our method offers a new possibility unlock much more scalability RL efficiency and performance.
CONCLUSION
We propose Jackpot, which leverages Optimal Budget Rejection Sampling to directly reduce the gap between actor and policy distribution. Empirically, our method achieves stable improvements in large-batch and asynchronous training and also demonstrates stability under extreme misalignment settings.
ACKNOWLEDGEMENT
We would like to thank computing resources of NVIDIA. This work was partially supported by Google Research Award, Google ML & System Junior Faculty Award, Amazon Research Award, Fireworks AI, Intel, Li Auto, Moffett AI, and CMU CyLab Seed funding. This material also based upon work supported by the National Science Foundation under Grant No. 2504353 and IARPA. Any opinions, findings, and conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of the National Science Foundation.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our study is centered on the development of reinforcement learning techniques for large language models. The research does not involve the use of human subjects, personal data, other sensitive information. All datasets employed are openly accessible and commonly utilized within the research community. We recognize that advancements LLMs may lead to societal risks, such as the potential misuse for generating harmful or deceptive ouiputs. To address such concern,, this work is conducted strictly within controlled academic environments, with the primary objective of enhancing the robustness and efficiency of training methodologies.
REPRODUCIBILITY STATEMENT
We have provided comprehensive account of implementation details, hyperparameters, and experimental configurations in both the main text and the appendix. This documentation is intended to ensure that other researchers are able independently reproduce our findings without ambiguity.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in lims. arXiv preprint arXiv:2402. 14740, 2024.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. arXiv preprint 1rXXv:2310.1063// 2023.
Ernesto Carrella, Joseph Powers, Steven Saul, Richard M Bailey, Nicolas Payette, Katyana A Vert-pre, Aarthi Ananthanarayanan, Michael Drexler, Chris Dorsett, and Jens Koed Madsen. Rejection sampling and agent-based models for data limited fisheries. Frontiers in Marine Science, 11:1243954, 2024.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih. Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg. and Koray Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, 2018. URL
wttp:/ii..rrgab//11822.... 01561
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505. 242988 2025.
Wally R Gilks, Nicky G Best, and Keith KC Tan. Adaptive rejection metropolis sampling within gibbs sampling. Journal of the Royal Statistical Society Series C: Applied Statistics, 44(4):455-414 472, 1995.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.129: 2025.
Horace He and Thinking Machines Lab. Defeating nondeterminism in Ilm inference. Thinking Machines Lab: Connectionism, 2025. doi: 10.64454/mm1.22550911 https//tiikingncciissiiibllgeeed mondetemiinisn--ni-------eeeenn
Dan Hendrycks, Collin Bums, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL
https: ptttps://arxiv.. wtwpiiiv.rr/////////////
03874
Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang. Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performacc rlhf framework. arXiv preprint arXiv:2405. 11143, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig. Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv.2110.0677.. 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.

---

Published as a conference paper at ICLR 2026
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXfv:2310.10505 2023.
Liyuan Liu, Feng Yao, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Flashrl: 8bit rollouts, full power rl, August 2025, htttp:://eengeeeeee notion wiw.siiteeeeeeee flash-aahhhhee rl
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang. Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jíe Tang. Agentbench: Evaluating Iims as agents. arXiv preprint arXiv: 2308.03688, 2023.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li. Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol-preview with 1.5b model by scaling r1, 2025. Notion Blog.
Luca Martino and Joaquín Míguez. A generalization of the adaptive rejection sampling algorithm. Staristics and Computing, 21 21(4):633-647, 2011.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems,
2024.
Christian A Naesseth, Francisco JR Ruiz, Scott W Linderman, and David M Blei. Rejection sampling variational inference. stat, 1050:18, 2016.
Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. Kernelbench: Can Ilms write efficient gpu kernels? arXiv preprint arXiv:2502. 10517, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 31::3328-5333855 537418
2023. 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL httttt:tt/aaaatttieee http://aaxxvvvo..
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning open language models. arXiv preprint arXiv:2402.033000 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rIhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Alexandre Verine, Muni Sreenivas Pydi, Benjamin Negrevergne, and Yann Chevaleyre. Optimal budgeted rejection sampling for generative models. In International Conference on Artificial Intelligence and Statistics, pp. 3367-3375, PMLR, 2024.
Leandro Werra, Younss Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. 2020.
Bo Wu, Sid Wang. Yunhao Tang, Jia Ding. Eryk Helenowski, Liang Tan, Tengyu Xu, Tushar Gowda, Zhengxing Chen, Chen Zhu, Xiaocheng Tang, Yundi Qian, Beibei Zhu, and Rui Hou. Llamarl: A distributed asynchronous reinforcement learning framework for efficient large-scale Ilm training, 2025a.
URL httphhtss///xiiieee als////////////////// a555 2505. 01555.24034
Bo Wu, Sid Wang. Yunhao Tang, Jia Ding, Eryk Helenowski, Liang Tan, Tengyu Xu, Tushar Gowda, Zhengxing Chen, Chen Zhu, et al. Llamarl: A distributed asynchronous reinforcement learning framework for efficient large-scale Ilm trinin. arXiv preprint arXiv:2505.24044 2025b.
12

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: orq/ a5ss 09388
Chujie Zheng, Shixuan Liu, Mingze Li. Xiong Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang. et al. Group sequence policy optimization. arXiv preprint arXiv:2507. 18071, 2025a.
Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for IIm reasoning via selective rollouts. arXiv preprint arXiv:2506.02177, 2025b.
A ANALYSIS OF OBRS
This appendix provides the theoretical foundation for OBRS. We first formally define the post-rejection distribution that results from our method. We then prove two key results:
1. Optimality: For any desired sample efficiency (i.e., acceptance rate), OBRS is the unique optimal rejection mechanism that produces a post-rejection distribution closest to the target po... terms of KL divergence (Theorem DT.
2. Monotonic Improvement:
The post-rejection distribution monotonically approaches the target distribution as the scaling factor C' increases. This guarantees that for any C' 0, OBRS reduces the KL divergence compared to using the proposal pirr directly. (Proposition DI
For notational clarity, we consider the distributions over tokens for a single, fixed prompt and omit the explicit conditioning. Let Pt Pe... denote the target distribution and Pp = Punt denote the proposal distribution.
Algorithm Implementation of Optimal Budgeted Rejection Sampling
Require: Proposal distribution Pint. Target distribution Pr... Require: Scaling factor C>> O, Number of samples accept N
Ensure: Set of accepted samples Skept
I: Initialize Suept ()
2: while |Skept| < N do
3: Sample token pir))
4: Calculate acceptance probability a min (1, Peze (a)
(
5: if (0,1)) <a then
6: Add a to Skept
7: end if
8: end while
9:
10:
return Shep
A.1
THE POST-RE/ECTION DISTRIBUTION
Recall from Definition that OBRS accepts token pp((a with probability ac(a)= ([(o))mmn) (1, ( )
The unnormalized probability of sampling and keeping token a is Pp(a) ac(a), which simplifies to pain
The overall probability of accepting any token, which denote as the acceptance rate Ze, is the sum over all possible tokens:
Z = Σ min (a). Pc(a)
aEAA
13

---

Published as a conference paper at ICLR 2026
The distribution of the tokens that are kept, which we call the post-rejection distribution Cropt, is therefore:
Puep, (a) min(pp(a).p((a)//
Ze
Special cases clarify the role C: as -,0 Zc + and Pssp,.C + Py (all tokens are kept). As C + oc, Zc + O and Prepi. + Pt. (perfect alignment with vanishing throughput). Standard rejection sampling is the special case where C2 (p)))))))))pp)))))..
A.2 OPTIMALITY FOR FIXED ACCEPTANCE BUDGET
We first establish that OBRS is not merely a heuristic but is the provably optimal strategy for minimizing distributional error given fixed efficiency budget.
Theorem 1 (Budgeted Optimal Acceptance). Fix target acceptance rate (budget) a E (0. 1). Among all possible token-wise acceptance rules a A + [0.1] that satisfy the budget constraint Rawpp [a(a)]= -=)PP(()((())) = 2, the rule that generates a post-rejection distribution Proge (0) x Pp(a)a(a) minimizing the Bullback-Leibee (KL) divergence KL(pel (Pheps) is uniquely given by the OBRS rule:
(a)
a² (²(a)= (())=nnn] ppppp(a)
for some constant X >>0 0 (equivalent to C) whose value is determined by the budget 2.
Proof. The post-rejection distribution is Phopi = Pp(())((a/2= The KL divergence is:
(o) (i(a)
(I- (Peqp))
Σ
Phepl (a)
Pr((a)log pr(a)
Pp(a)a(a)/z
P((a)
Pa((a)o Pp(a)
loge
Σ (a))looga(
Minimizing (L(PelPPwpp) is therefore equivalent to maximizing (a)looaag) subject to the constraints:
E>pp(a)a(a)== = (budget constraint)
2. 0Sa(a) 1 for all a a (valid probability constraint)
This a convex optimization problem. The Lagrangian is:
L Σ Pr (a))ooooa(a)) + + [pp(a)a(a)=>)
"ua(a(()-1) (
1 -va((a)
where Pa the KKT multipliers. From the stationarity condition ac =0, get P((a) V/-
(a)
ao(a)
a(a)
Na -v, 0. The complementary slackness conditions imply that if0 < 1, then Moo = =0, which
gives a(a)= (a) If a(aa= 1, then =0, which requires App(a) If (<aa= =0, this form is not well-defined, but the logic holds. Combining these cases, the optimal rule is cap the acceptance ratio at 1:
p=(a)
a (a) min 1, (a(a)
AAp((a)
The Lagrange multiplier À is chosen meet the budget constraint EaPp(a)a (a) =2. Uniqueness follows from the strict concavity of the log objective function.
A.3 GUARANTEED KL DIVERGENCE REDUCTION
Next, we show that our method provides a guaranteed improvement over the proposal distribution pp and that this improvement is monotonic in the control parameter C.
Proposition 1 (Monotonic KL Contraction). The function G(C)= KKL(pellPegnnc)) Pkept.)) is non-increasing for Ce (0, It is strictly decreasing wherever the set of tokens {a pr(a) < C. C-pp((a) has non-zero probability under Pt-
14

---

Published as a conference paper at ICLR 2026
Proof. Let p(a) = pp(a)/pp((). We partition the vocabulary A into two sets: Ac fa p(a)p())>))))))))))()))) C) and Bc = {a p(a) E C). On any open interval of C where this partition is constant, we can write the KL divergence G(C)== = -apt (a)looPPem,.cl as:
G(C)= Σ Pr(a)log Po(a)
Pi(la)log Pr(a)/C Po
Zc Differentiating with respect to C (and noting that only Zc depends on C) we get dc d -an((()-1oZc)) 2C 2
Ze
acAc
aEBc
dee
The acceptance rate is Zo = Pnla)
Zo dc
Its derivative is:
dZc
Σ po(a).
dC
C
aERe
Therefore, G'(C)= 1 (- C2 EuEB. (a)) 20. since all terms are non non-negative. The derivative is strictly Ze negative if Be > O. As G(C) is continuous and piecewise differentiable with non-positive derivative, is non-increasing everywhere.
Corollary 1 (Strict Improvement over Proposal). For any C OBRS produces distribution Clept, that is strictly closer to the target distribution Pt than the original proposal distribution Py Le.,
KL(pell KL(pellPeem.c c) < KL(pi lpp), wnless Pp pr or C' mminn (p),, in which case the KL divergences are both zero
Proof. From Proposition I know that KL(P: || Plap,)) is non-increasing in C. In the limit as - 0, the acceptance probability ac(a) - 1 for all a, meaning Prer,C: + Pp. Therefore, KI.c-oKl(pp [Ipppppcc KL- Pp). For any C 0, long as Pp + pt and C min( 21 there must exist some tokens for which (a(a) < (a) or p. (a) > Ca),(a), ensuring the condition for strictly decreasing KL divergence is met some interval (0,C). Thus, ((C))GG for some small e>> 0, which implies KL-(pe (Ptell < <KL(pe Ipp)
Pe
A.4 PRACTICAL GUIDANCE
Setting C. C=1 is a robust default: it contracts the per-prone per-prompt KL (strictly unless Pmut=Pt)) keeps acceptance high (Z1), and is O(1) per token. Larger C pushes qc closer Per but reduces throughput (Zc$1/C), use only if variance or bias considerations demand stronger alignment.
Compatibility, The rule uses only per-token log-probabilities already computed by PPO/GRPO, so it introduces estimators and preserves gradient flow exactly as described in Algorithm 1
B DETAILS OF THE EXPERIMENTS
B. TIS ADJUSTMENT EXPLANATION
However, the delay in updates results in staleness, which we magnify for evaluating our method.
In this section, push the above scenario to its limit by asking the inference batch size to be 64x and 128x the training batch size. Concretely, we choose to use training batch size of 32. We train models on the DeepScalerR Preview dataset Luo et al. (2025). which contains 40k challenging competition math problems. We select Qwen3-4B Base and Qwen3-88 3BB-3-BBB- Base models Yang et al. (2025) to run RL on. An important baseline to method is the Truncated Importance Sampling (TIS) in Wu al. (2025a). However, in the original technique is proposed only for the approximate models the inference server. Thus, in the following loss form.
T(Qrrt)
T ((0new) A)]
Ea-z((a)) T(foaair) Vaclip( T(
The weights update frequency is close to on-policy settings, but the gap between the approximate and efficient inference model and the training weights is the primary goal to solve. However, the original formula cannot easily be adapted for our extremely large batch setting, as there is no term regularizing the difference between Te.... and Tew. We found that very simple trick results in very strong baseline on top the TIS method, that is we write it this way.
T (Boowddetchee) V Veclip( T (((gew)) A)]
Ea-a(
T7 T(ossain))
( (Orowwdettccee)
15

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation scores across benchmarks (GSM8K, MATH-500, AMC22 AMC23).
GSM8K
MATH-500
AMC22 & AMC23
Models/ Methods
Mean @4
@
Mcan
Pass@4
Mean @4
@ass@4
Qwen3-4B-Base on DeepScaler (rollout batch size 2048; train batch size 32; 64x)
On Policy
92.19
95.988
81.55
88.00
58.43
71.76
Off Policy
88.04
95.03
71.15
82.24
39.15
55.57
TIS
89.67
95.53
72.00
80,96
42.77
56.71
TIS with Adjustment
92.76
96.09
79.50
85.81
57.22
66.61
Jackpot (Ours)
92.24
95.891
80.05
85.89
53.916
65.034
Owen3-4B-Base DeepScaler (rollout batch size = 4096;
train batch size = 32; 128x)
Off Policy
79.70
92.96
60.20
76.60
33,00
48.446
TIS with Adjustment
20.697
43.00
19.10
37.751
7.80
17.83
Jackpot (Ours)
92.00
95.00
80.00
85.50
51.20
60.35
Qwen3-8B-Base on DeepScaler (rollout batch size 2048; train batch size 32; 64x)
On Policy
94.238
96.78
93.99
96.65
28.95
37.89
Off Policy
91.05
95.62
77.15
84.90
50.60
65.20
TS with Adjustment
93.85
96.58
82.55
88.38
60.54
72.79
Jackpot (Ours)
94.01
96.63
83.05
88.76
63.55
74.12
where use the detached most recent model output distribution as the term in the importance sampling. However, the consequence is also very clear, the internal clip around ratio is now 'short-circuited or no longer useful. Nevertheless, the setting produces very strong convergence is correction over the off- policy baseline. We call it adjuusee.
To fairly compare against the baseline, we also modify our training loss as follows, effectively also "short-circuiting' the internal ratio clip. The only difference between our setting and theirs is that we use our proposed sampling method to regularize the Thaas
T (Oneww (Onewddetaceed) V Vaclip(
π (Onew
Eane((al (fooldint) "new
A)]
x ( ((newndettcceed
B.2 FULL DETAILS OF EXTREME SIZE BATCH SIZE EXPERIMENTS
C ABLATION STUDIES ON COMPONENTS OF JACKPOT, THRESHOLD, AND TOP-K
C.1 COMPONENTS' CONTRIBUTION TO JACKPOT
Rejection bridges the gap between the rollout and the current policy (see FP8 on-policy training without explicit jackpot reweighting; this is essentially the vanilla PPO loss with the probability fixed, and the OBRS distribution in this case matches the reference policy exactly). Because the rollout training distribution gap is now removed, stability is significantly better than the vanilla baseline: even without TIS, training does not crash.
However, without correct importance sampling (see the huge-staleness training regime, where the importance distribution no longer matches Pur but instead matches the current policy), training will eventually collapse.
On the other hand, jackpot reweighting does not solve the rollout torioutt gap (because under the FP8 setting it falls back vanilla on-policy training, which again leads crash). But the nage-staleness regime, where the performs correct importance sampling, tracks the proper target distribution, and keeps the overall training procedure stable and efficient.
gap is not the main issue, jackpot reweighting combined with OBRS is effective:
C.2 THRESHOLDS, CI, AND C2
Our method involves three hyperparameters: C, Cz. and the rejection threshold A. All of them are straightforward to set, and the technique is robust across wide range of choices.
16

---

Published as a conference paper at ICLR 2026
Table 4: Evaluation scores across benchmarks (AMC12 2024, AIME24, AIME25).
AMC12 2024
AIME24
AIME25
Models/ Methods
Mcan @
Pass@4
Mean @ 16
Pass@
Mean @ 16
Pass@ 16
Qwen3-4B-Base on DeepScaler (rollout batch size 2048; train batch size 32; 64x)
On Policy
51.11
65.45
23.12
33.13
22.91
30.95
Off Policy
29.44
41.802
13.958
23.03
11.042
18.607
TIS
31.667
50.844
11,875
17.561
11.25
17.278
TIS with Adjustment
43.33
60.67
18.75
26.03
17.708
24.607
Jackpot (Ours)
50.00
63.00
20.625
29.484
18.125
23.627
Qwen3-4B-Base DeepScaler (rollout batch size = 4096; train batch size 32; 128x)
Off Policy
24.44
38.41
8.00
15.73
5.00
11.00
TIS with Adjustment
5.00
11.00
1.00
4.00
1.00
2.00
Jackpot (Ours)
47.22
57.99
19.16
24.58
18.52
25.078
Qwen3-8B-Base on DeepScaler (rollout batch size 2048; train batch size 32; 64x)
On Policy
54.44
68.47
28.95
37.89
22.50
28,542
Off Policy
40.00
54.75
18.54
28.67
14.16
21.98
TS with Adjustment
48.33
59.78
24.58
35.06
20.00
22.90
Jackpot (Ours)
54.44
66.09
26.87
36.23
20.41
26.57
Table 5: FP8 on-policy training (no staleness).
Best scores before crash for the vanilla baseline.
Experiments
AIME24
AMC
GSM8K MATH500
Vanilla (best before crash)
23,958
57.530
80.900
92.665
Masking-only
25.625
62.048
83.700
92.835
Masking & reweighting
26.667
62.651
82.450
92.305
Table 6: BF16 training with rollout staleness (64/2048).
Best scores before crash for masking-onjy
Experiments
AIME24
AMC
MATH500
GSMSK
Masking-only (best before crash)
19.167
49.699
78.750
91.793
Masking & reweighting
25.625
63.855
83.800
92.400
O₁. We follow standard truncated importance sampling (TIS) choices. Empirically, selecting Ci (2,10) consistently works well, and the method is not sensitive within this interval.
C2 (upper bound for pe..// /Po.. This parameter has no practical effect on performance. We set C2 slightly larger than Ehigh (c.g., 1.28 for DAPO). where any ratio clipped by C2 would already be clipped by the PPO trust region. Thus, C2 mainly serves conceptual safeguard for ratio stability.
Rejection threshold X. Our method performs well across all experiments with a default setting of x= 1.0. and we recommend choosing À close to this value. Increasing À makes the kept-tole token distribution closer to the target policy but increases the rejection rate. If >>>>> 1, it begins rejecting tokens even when the policy and inference distributions are perfectly aligned, causing overly conservative updates. The default value 1.0 guarantees full acceptance in the matched-distributioo case and already reduces KL. substantially while keeping a high acceptance rate.
Summary. Jackpot is stable and easy to configure: C₁ is robust within the typical TIS range 10, C2 has no practical impact once chosen above 1 1 Ehigh, and c= 1.0 serves reliable default.
17

---

Published as a conference paper at ICLR 2026
Table 7: Effect of C₁ on benchmark performance. Experiment Setup: Model: Qwen3-4B-Base, C2
3.0, threshold c= 1.0, response limit: 8k, mini-batchhrrain-bathh 64/2048, PPO clip: 0.4/0.7, 100k examples. Numbers are pass@1 accuracy.
Hyperparameters
AIME24
AMC
MATH500
GSMSK
C ==
26.875
63.855
82.800
92.267
C = 3
25.625
63.855
83,800
92.703
C = 4
26.042
65.060
83.500
92.684
C1 8
26.875
63.253
83.100
92.437
Table 8: Effect of rejection threshold c benchmark performance. Experiment Sctup: Model:
Qwen3-4B-Ban (target) / Owwn3-11.7--BB B-Base (rollout); Generation length limit: 8K; Training examples: 9K. Numbers are pass @ accuracy.
Threshold
AIME24
AMC
MATH500
GSM8K
14.7
49.4
74.5
92.0
0.9
12.5
47.0
74.6
91.8
1.0
14.7
48.5
74.6
92.2
1.1
12.3
47.4
74.1
92.0
1.2
13.5
45.8
74.1
91.9
C.3 CHOICE TOP-K FOR Z APPROXIMATION
In Section 4.4
approximate the OBRS normalization constant Z by summing over the union of the top-k tokens under Pint and pa.... yiekding Zmpprox S. Z. Increasing & strictly improves this approximation but also increases the number of logits that must be materialized and stored. In the main experiments we therefore fix 20, and here we justify this choice empirically.
We first study the direct effect of k the quality of the Z estimator. For the extreme off-policy configuration, we log (i) the fraction the true normalization captured by the top-k estimator, Zuppron/ /Z, and (ii) the calibra- tion factor defined in Section 4.4.2 using k (10,20,40) As expected, larger l: improves both quantities, but with rapidly diminishing returns. Even for the smallest value k= we already capture at least 87% of Z the very beginning training, and more than 99.98% once the policy has warmed up within few steps; using == 40 increases the captured mass only slightly (from roughly 91% initially to about 100.0% in the steady state), Since the union of top-20 tokens is always superset of the union of top-10 10 tokens, these diagnosties imply that k = 20 already yields an almost exact estimator of Z while keeping the additional overhead modest.
We next evaluate how affects downstream performance. Table reports pass I accuracy on our four math benchmarks for Jackpot with & (10,20,40) 40), keeping all other hyperparameters fixed. The differences acToss choices are small and non-monotonie: k == performs slightly better on AMC and MATH500, while AIME24 and GSM8K show consistent trend. Overall, the variation is comparable to run-to-run noise, and there is no evidence that pushing beyond 20 systematically improves task performance. Taken together,
Table Effect of top-k on benchmark performance. Numbers are pass@1 accuracy.
Hyperparameters
AIME24
AMC
GSMSK MATH500
k= 10
28.958
61. 61.446
82.650
92.608
k= 20
25.625
63.855
83.800
92.703
x= 40
27.083
61.446
83.200
92.418
these results show that larger & does improve the Z estimator, but the gains become marginal k reaches 20. Since the computational overhead of our method scales roughly linearly with k we adopt i: = 20 as a practical default: it provides an accurate, well-calibrated estimate Z with negligible additional cost (less than 3% overhead in our setup), and larger values of & do not yield measurable benefits in our experiments.
18


Reviews:
[{"review_id": "k2oDpMHNTC", "reviewer": "Reviewer_ZoQc", "summary": "This paper addresses the computational cost and instability arising from actor-policy distribution mismatch in reinforcement learning for large language models. While allowing these distributions to differ could enable significant efficiency gains—such as large-batch training, asynchronous updates, or using smaller rollout models—existing importance sampling-based corrections face a fundamental trade-off between stability and performance. The authors propose Jackpot, which leverages Optimal Budget Rejection Sampling (OBRS) to directly reduce the distributional gap between actor and policy networks. The method introduces three key contributions: an OBRS-based masking mechanism that maintains closer alignment between distributions, an efficient probability estimation strategy using Top-K logits with batch-wise bias correction to handle memory constraints, and a stabilized PPO loss jointly accounting for importance sampling ratios and trust-region constraints. Evaluated on large-batch training (128 mini-batches per rollout) and extreme off-policy scenarios, Jackpot demonstrates 20% improvement on AMC benchmarks and 8% on AIME benchmarks over baseline methods while maintaining stable training dynamics under severe distributional mismatch conditions.", "questions": "- The LaTeX format of this paper does not appear to follow the standard ICLR submission template.\n\n- Does the Phase 1 Data Collection with OBRS require modifications to the sampling code in vLLM or other inference frameworks?\n\n- How should the hyperparameters $c_1$ and $c_2$ be chosen? Is there any guidance provided for their selection. Furthermore, is the method robust to different choices of these hyperparameters?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper tackles an important and practically relevant problem in reinforcement learning for LLMs: the gradient estimation bias and training instability caused by distribution mismatch between the actor and policy networks. \n\n- The proposed OBRS-based approach seems to be a principled solution to directly reduce this distributional gap.\n\n-  The empirical evaluation demonstrates promising results, particularly in maintaining training stability.", "weaknesses": "- The paper lacks a detailed discussion of the differences between importance sampling ratio-based corrections and the rejection sampling approach, despite both relying on importance sampling ratio calculations. This makes it unclear why the optimal rejection sampling method provides advantages over existing correction techniques like TIS.\n\n- The paper builds upon Optimal Budgeted Rejection Sampling (OBRS) but does not provide sufficient introduction or justification for this choice. Without adequate background and motivation, it is difficult for readers to accept that OBRS is the appropriate approach for addressing the distribution mismatch problem.\n\n- The empirical evaluation lacks comprehensive ablation studies to disentangle the contributions of individual components. It remains unclear whether the OBRS-based masking mechanism, the Jackpot re-weighting strategy, or both components are essential for achieving the reported improvements in stability and performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VlQCcr2qqV", "reviewer": "Reviewer_BCWN", "summary": "The paper aims to reduce the high rollout cost from large trajectories in reinforcement learning based post-training of LLMs while improving stability and scalability. The key idea is to sample trajectories from a lightweight actor policy different from the main training policy to make RL more efficient. To handle the resulting mismatch, the paper introduces a rejection-sampling–based approach that directly reduces the distribution gap between actor and policy. It further proposes a Top-K approximation with bias correction and a stabilized PPO loss to maintain trust-region stability. Experimental ablations shows promising gain over baselines.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea of using simple light-weight policy for sampling trajectories is interesting and effective especially for larger policies\n- bounding the KL and using rejection sampling helps in mitigating the distribution shift\n- efficient Top-K probability estimation keeps it usable for large vocabularies.\n- empirical performance are promising especially on AMC and AIME benchmarks compared to off-policy baselines.", "weaknesses": "- Although interesting, the idea is extremely similar to several parallel streams including speculative decoding, on policy distillation, weak-strong etc and the key novelty of the approach is not clear\n- The paper ensures the KL divergence between the actor and policy distribution, however how it ensures closeness to the reference policy based on which standard RLHF policies are trained? How do you ensure closeness to that? Can you show a KL plot with the reference?\n- It requires sampling multiple trajectories and computing the ratio of the probabilities, how much time it incurs additionally during training?\n- Also, what kind of light-weight models can help to mitigate this shift in an efficient way - ie does compression or smaller finetuned models or distilled models, how are different models behaviour \n- How sensitive is the approach with the threshold and how is the threshold determined? That will also affect the number of trajectories to generate? It will be helpful to provide the details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eTV32rdSCa", "reviewer": "Reviewer_43dn", "summary": "The paper introduces a new reinforcement learning framework aimed at improving the stability and efficiency of RL-based fine-tuning for large language models. The central problem addressed is the distribution mismatch between the actor (rollout) model and the policy being optimized, which often arises in large-batch, asynchronous, or off-policy training scenarios. Existing importance sampling corrections for this mismatch tend to suffer from instability or poor performance. To tackle this, the authors propose JACKPOT, a method that leverages Optimal Budget Rejection Sampling (OBRS) to directly reduce the KL divergence between the actor and policy distributions.\n\nEmpirical results on large-scale language models (Qwen3-4B and Qwen3-8B) demonstrate 20% improvement on AMC and 8% improvement on AIME benchmarks under extreme off-policy settings, as well as significant gains in large-batch asynchronous RL training. The method substantially delays training collapse and outperforms prior off-policy RL methods such as Truncated Importance Sampling.", "questions": "1. In Algorithm 1, the acceptance probability $\\alpha$ requires access to $p_{\\theta_{new}}$ at sampling time. How is this computed in practice when rollouts are generated by a separate inference actor? Are logits from $p_{\\theta_{new}}$ queried synchronously (which could double inference cost), or are they approximated, cached, or delayed? Please clarify how this fits into a large distributed RLHF or asynchronous setup, and quantify the extra compute or communication cost.\n2. The stabilized Jackpot loss uses two clipped ratios and a stop-gradient. Could you formalize or empirically show how these choices affect gradient bias and variance? Specifically, how do c1,c2 and the stop-gradient operation trade off between bias and stability? It would help to include plots of gradient variance before and after stabilization.\n3. The method is validated primarily on mathematical benchmarks (AMC, AIME, GSM8K, MATH-500). Evaluating on alignment or preference-based tasks (e.g., Anthropic-HH) would help test generality and robustness beyond numeric reasoning.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper targets a concrete and important practical bottleneck: distribution mismatch between rollout models and the policy used for updates in RL for LLMs (large-batch, asynchronous, quantized/distilled rollouts, etc.). The motivation is clearly stated and practically relevant.\n2. The paper proposes using Optimal Budgeted Rejection Sampling to directly reduce actor–policy KL, which is a principled move. The paper provides theoretical guarantees: OBRS is the unique budget-optimal token-wise accept/reject rule minimizing KL for a given acceptance budget, and the post-rejection distribution monotonically approaches the target as the scale parameter varies. These formal results strengthen the core idea.", "weaknesses": "1. The OBRS acceptance probability for a sampled token requires evaluating the ratio $p_{\\theta_{new}} (a) / p_{inf} (a)$. In practice, this implies scoring candidate tokens under the *current policy* during data collection or otherwise obtaining these probabilities. However, the paper does not fully quantify the extra compute this incurs in real serving setups. This is important because the whole motivation is improving throughput; any extra compute could reduce or eliminate the gains.\n2. The stabilized weight uses two clipped ratios and a stop-gradient on the alignment factor, trading bias for variance. The paper argues this is necessary for stability, but there is limited theoretical quantification of the bias introduced by clipping and the stop-gradient, nor diagnostics showing how much bias affects final performance in less extreme settings. More ablation (e.g., varying c1,c2, removing stop-grad) would help understand the tradeoffs.\n3. The final stabilized Jackpot objective (Section 4.4) is a pragmatic heuristic. The paper admits that it \"trades a small amount of bias (from clipping and approximation) for a massive reduction in variance\". This factorization and clipping move the method away from the clean theoretical guarantees of OBRS. The paper does not provide any theoretical analysis of how this specific, biased re-weighting scheme affects the optimization landscape, convergence, or the final policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PVxGvx1rB2", "reviewer": "Reviewer_k9JV", "summary": "JACKPOT targets, actor–policy mismatch, the PPO pain point in LLM RL. It fixes the problem at sampling time with Optimal-Budget Rejection Sampling. The naïve OBRS recipe is unusable at scale because it needs full-vocab normalizers, yields unbounded correction weights, and can kill throughput when alignment is too strong. The paper’s value is a practical pipeline: estimate OBRS on a top-k union of actor/policy tokens and calibrate the missing mass with the simple identity that the normalizer equals the expected acceptance rate; then train with a stabilized PPO objective that factorizes the correction and clips each ratio so gradients stay bounded and trust-region friendly. Empirically it demonstrates it shines at where vanilla PPO struggles, large rollout batches at fixed minibatch, and extreme off-policy collection, delivering steadier learning and better scores without changing the base model or reward loop.", "questions": "1. Clipping bias: compare unclipped vs clipped vs two-sided clipped weights on identical runs; report policy-improvement proxies, gradient norms, collapse rate.\n2. Visitation skew: show state-coverage heatmaps before/after OBRS, effective sample size per update, entropy of the kept policy, and advantage-weighted coverage of rare regions.\n3. Top-k curve: plot performance, gradient norms, and normalizer error versus k; provide a simple k rule or a dynamic k policy tied to a target acceptance band.\n4. Potential KL \"double counting\"?: With OBRS plus the clipped reference ratio acting like a trust region, an explicit KL(pθ‖pref) may over-constrain updates. add curves with and without the explicit KL term to the reference under OBRS; clarify whether the second clipped ratio already suffices or if a reduced KL is still helpful.\n5. Curious to see block-level OBRS.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Targets PPO’s real pain points. It directly attacks actor–policy mismatch in the regimes that break PPO such as huge rollout batches with the same small minibatch and extreme off-policy actors, and shows steadier learning and better scores there.\n2 Not a naïve OBRS port. The paper surfaces why a straight plug-in fails (full-vocab normalizers, unbounded weights, acceptance-throughput collapse) and then fixes each with a practical trio: top-k union for feasibility, acceptance-rate calibration for unbiased scale, and a factorized, clipped correction that keeps updates stable.\n3. Clean narrative. The writing crisply motivates the problem, diagnoses naïve failures, and walks the reader through the recipe and its effects; the algorithm is easy to implement from the description.", "weaknesses": "1. Loss clipping bias. The stabilized objective clips two likelihood ratios to keep gradients tame. That’s an intentional bias; it can underweight rare-but-informative tokens and shrink effective step size. The paper argues stability, but it does not quantify when the bias alters policy improvement or exploration.\n2. Actor–policy visitation skew. OBRS keeps tokens the current policy already likes, which shifts the behavior distribution. Alignment lowers variance, but it can also prune rare, hard states that matter for learning. Without coverage/diversity diagnostics, it is unclear whether OBRS quietly narrows what the policy ever sees.\n3. Sequence or block-level OBRS. Tokenwise rejection is simple, but it fragments credit on long sequences where decisions cohere over spans. Span- or block-level acceptance could improve temporal coherence at the same keep-rate (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)