The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
Authors: Long Li
Zhijian Zhou
JIARAN HAO
Jason Klein Liu
Yanting Miao
Wei Pang
Xiaoyu Tan
Wei Chu
Zhe Wang
Shirui Pan
Chao Qu
Yuan Qi
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Reinforcement Learning with Verifiable Reward
f divergence
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose the DPH-RL framework, which uses f-divergence as a proactive 'rehearsal mechanism' to solve the solution diversity collapse problem that arises from fine-tuning LLMs with reinforcement learning.

Abstract: A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance (Pass@k) despite improvements in single-attempt accuracy (Pass@1). This is often accompanied by catastrophic forgetting, where models lose previously acquired skills. Despite numerous proposed methods, the community's focus on the standard reverse KL-divergence has led to a surprising oversight: the potential of alternative f-divergences as a proactive solution has been largely unexamined. We argue that standard RLVR objectives—both those using the mode-seeking reverse KL-divergence  and those forgoing a divergence term entirely—lack a crucial mechanism for knowledge retention. The reverse-KL actively accelerates this decay by narrowing the policy, while its absence provides no safeguard against the model drifting from its diverse knowledge base. We propose a fundamental shift in perspective: using the divergence term itself as the solution. Our framework, Diversity-Preserving Hybrid RL (DPH-RL), leverages mass-covering f-divergences (like forward-KL and JS-divergence) to function as a 'rehearsal mechanism'. By continuously referencing the initial policy, this approach forces the model to maintain broad solution coverage. Math and SQL generation experiments show that DPH-RL both improves in-domain Pass@1 and Pass@k scores and effectively prevents catastrophic forgetting on out-of-domain tasks. Additionally, DPH-RL is more training-efficient because it computes f-divergence using generator functions, requiring only sampling from the initial policy and no online reference model. Our work highlights a crucial, overlooked axis for improving RLVR, demonstrating that the proper selection of a divergence measure is a powerful tool for building more general and diverse reasoning models.

Published as a conference paper at ICLR 2026
THE CHOICE OF DIVERGENCE: A NEGLECTED KEY TO MITIGATING DIVERSITY COLLAPSE IN REIN- FORCEMENT LEARNING WITH VERIFIABLE REWARD
Long Li Zhijian Zhou Jiaran Hao", Jason Klein Liu", Yanting Miao", Wei Pang', Xiaoyu Tan?, Wei Chu Zhe Wang Shirui Pan Chao Qut.6.; Yuan Qi
Fudan University
INFLY TECH Griffith University
University of Waterloo
*Shanghai Innovation Institute
Shanghai Academy of Artificial Intelligence for Science
seamokellle 1e com
ABSTRACT
A central paradox in fine-tuning Large Language Models (LLMs) with Rein- forcement Learning with Verifiable Reward (RLVR) is the frequent degradation multi-attempt performance (Pass despite improvements in single-attempt accuracy (Pass@) 1). This is often accompanied by catastrophic forgetting, where models lose previously acquired skills. Despite numerous proposed methods, the community's 's focus on the standard reverse-KL. divergence has led to a surpris- ing oversight: the potential of alternative f-divergences as proactive solution has been largely unexamined. We argue that standard RLVR objectives botth those using the mode-seeking reverse-KL. divergence and those forgoing a divergence term entirely- lack a crucial mechanism KL actively accelerates this decay by narrowing the policy, while its absence pro- vides no safeguard against the model drifting from its diverse knowledge base. We propose fundamental shift in perspective: using the divergence term itself the solution. Our framework, Eiversit-Prreserviie Hybrid RL (DPH-RL), lever- ages mass-covering f-divergences (like forward-KL. and JS-divergence) fune- tion as a "rehearsal mechanism'. By continuously referencing the initial policy, this approach forces the model maintain broad solution coverage. Math and SQL generation experiments show that DPH-RL surpasses the GRPO baseline by improving both in-domain Pass@1 I and Pass@k scores, and effectively prevents catastrophic forgetting on out-of-domain tasks. Additionally, DPH-RL is more training-efficiee because computes f-divergence using generator functions. re- quiring only sampling from the initial policy and no online reference model. Our work highlights crucial, overlooked axis for improving RLVR, demonstrating that the proper selection of divergence measure is powerful tool for build- ing more general and diverse reasoning models. Code and data are available at: https::/gittub..cm http:///....cooeeeeeee//et/ttttt
knowledge retention. The reverse-
INTRODUCTION
Reinforcement Learning with Verifiable Rewards (RLVR) has recently shown significant success in enhancing the mathematical and coding capabilities of Large Language Models (LLMs) (OpenAl,
2023 Team 2024 Yang al. 2025 Meta Al 2024 Guo al., 2025). Despite this progress, a critical paradox has emerged: while RLVR-tuned models consistently improve the probability of generating correct solution single attempt (Pass I), their performance when multiple attempts are permitted (Pass@k) often stagnates or even degrades compared to their base models (Yue et al., 2025). This discrepancy suggests that RLVR, rather than teaching novel reasoning, may instead overfit the model to known solution paths, thereby sacrificing diversity.
To counteract the narrowing of the model's 's output distribution (Wang al. 2025 Yue al. 2025). which reduces solution diversity, researchers have explored several parallel strategies. One major
" "Corresponding author. Equal contribution.

---

Published as a conference paper at ICLR 2026
GRPO who DARG weo KL. Reverte DFT DPHI-45
Comperison of Divergences
Roveee
Have Model's Poo-iiit
100%
Perwar
TL..niieed
80%
70%
In: Do-Damaln
Cross- Domain
Out-ot- -oom--
Domain Genernlization Degree
Figure 1: The left panel evaluates the performance gap in Pass between the RL-trained model and the Base Model across test sets with varying degrees of divergence from the training data. The right panel visualizes the distributions reverse-KL. and forward-KL.
line of work attributes the phenomenon entropy collapse and encourages exploration by control- ling entropy (Cui et al. 2025; Cheng et al. 2025; Liang et al. 2025). Another distinct approach focuses directly optimizing the Pass @k metric (Mahdavi et al. Walder & Karkhanis 2025
which serves as a more direct proxy for solution diversity than entropy. A third strategy focuses on training setups, such as using extra data (Yan et al.) Dong al.) 2025a), fine-tuning hyperpa- rameters (He et al.] 2025 Yu et al. (025), or employing a hybrid RI RIsupeeresee fine-tuned (SFT) training paradigm (Liu et al. 2025b). Alongside these methods, a fourth, more fundamental com- ponent of the RL objective- the Kullback-Leinn (KL) divergence term used to constrain policy
updates-als also plays a crucial role.
However, while the first three avenues have been extensively explored, the influence of the KL diver- gence term has remained largely under-examined. The community has almost universally adopted the standard reverse-KL divergence
No (Kazemnejad et al. 2024 (Tol|bTT = Ex, log Wre! Schulman al. 2017 Shao et al.) 2024), whose well-established mode-seeking nature (Bishop & Nasrabadi 2006) theoretically forces the policy to converge on a single high-probability solution. Our experiments confirm this: reverse-KL objective yields single solution style, while forward- KL objective generates multiple styles (Figure 4. Additionally, we observe that models trained with a reverse-KL. KL term or no KL term at all suffer from a decline in @ass@k k and catastrophic forgetting. As shown in the left panel of Figure [T both GRPO without KI. (Shao et al. (024), DAPO (Yu et al.) 2025), and Reverse KI. show a significant decline in Pass@k performance. In oft-of-domain sce- worse. These models only correctly answer about 85% of the problems they were previously able to solve (Figure 3). This phenomenon is a known challenge in sequential learning paradigms with neural networks, not a fundamental flaw of RL, which in principle does not suffer from this in tabular cases (Hamadanian et al. (025).
narios, the Reverse KL approach performs
Despite these significant issues, prior RLVR methods have almost exclusively relied the stan- dard reverse-KL divergence, while the exploration of other f-divergences has been largely con- fined to offline RI or alignment tasks ("f-PO" and "f-DPO") (Wang al. Han et al. 2025). This trend severely underestimates their potential to solve the diversity collapse problem. To bridge this Framework (DPH-RL), an approach that em-
critical gap, we introduce the Diversity ploys different f-divergences to preserve model diversity within the online RLVR framework. A straightforward example is the forward-KL. divergence (DKL (xxe(Freflllz)
E.,.., log Tre! ). whose theoretical mass-covering property penalizes the policy for failing cover all solutions the refer- ence distribution. From practitioner's 's standpoint, the forward-Ki -LL objective effectively creates an "anchor dataset", forcing the model continuously rehearse its original knowledge base-a mech- anism that mirrors human learning and prevents the catastrophic forgetting seen with reverse-Ke KL. We generalize this concept to the broader family of f-divergences, including Jensen-Shannol (JS) divergence and a-divergence (see Table 5) Our experiments demonstrate that these mass-covering divergences, specifically forward- KL and JS- IS-divergence, lead to significant improvements in both @ass I and Pass @ks, maintain strong out-of-domain performance, and achieve these gains without external models. Notably, this approach is orthogonal to existing methods that focus on entropy control or reward shaping.
2

---

Published as a conference paper at ICLR 2026
We make three main contributions:
1. Systematic Analysis of Diversity Collapse: Focusing on the KL divergence term, we provide a systematic analysis of the solution diversity collapse in RLVR. identifying the standard reverse-Kee KL divergence as a primary cause. We show that its mode-seeking nature not only suppresses Pass @k performance- oftnnnnneeeeeennn to levels below the base model but also exacerbates catastrophic forgetting and leads to poor out-of-domain generalization.
A Novel DPH-RL Framework: We reframe the role of the KL divergence, proposing its use not as mere policy constraint but as an active diversity-preserving mechanism. Based on this principle, we introduce DPH-RL, novel framework that employs mass-covering f-divergences (e.g.. Forward- KL and JS- JS-divergence) to serve as a rehearsal mechanism. It effectively outperforms GRPO in both @ass@1 I and Pass@k, while mitigating the Pass@ k degradation relative to the base model. 3. Extensive Empirical Validation: Through extensive experiments on a range of models (Llama and Qwen, 7B to 32B) and complex reasoning tasks (mathematics and SQL), we demonstrate the robustness and superiority of DPH-RL. Our method consistently outperforms prior work on both in-domain and out-of-domain benchmarks, successfully mitigating the trade-off between greedy performance and solution diversity.
2 RELATED WORK
A more recent development is RLVR (uue al. 2025 reasoning, especially in areas such as mathematics, coding, and analysis (Shao et al.] 2024 Guo
promising strategy for boosting LLM
al. 2025
Team 2024 Lambert al.
2025 Li et al. 2024c Shen & /Zhang [025] O OpenAl 2024
Team et al. 2025 Yang et al.) 2025). To address diversity collapse in RLVR, one major line of
research focuses on either directly controlling model entropy (Cui et al., 2025 Prabhudesai al.
2025| Agarwal et al.) 2025 Li et al. 2025 Cheng et al.] 2025 Liang et al. (2225 or strategically
capturing high-entropy tokens (Wang et al. 2025; Chen et al.) [025 MiniMax al. 2025). How-
ever, some research suggests that the entropy of specific tokens is more strongly correlated with
model diversity than sentence-level entropy (Wang et al., 2025 Cheng et al., 2025 Deng et al.
2025 Chen et al. 2025); consequently, other methods maintain training diversity by controlling the model's s Pass k metric [Dong et al.) [0255] Zhang et al. 2025b). From a training design perspec-
tive, methods such dynamically adjusting rollout-related hyperparameters (An et al. 2025 He
2025) or using hybrid SFT+RL (Liu et al., 2025b a training paradigm also employed to prevent diversity collapse. It worth noting that theoretically, DPH-F can be viewed as a divergence-based Experience Replay, sharing a similar philosophy with some existing work Ouyang al. (2022); Zhhan al. (2025a), while DPH-JS extends this into a Regularized Rehearsal mechanism via dual-sided sampling. Please refer to Appendix D for details.
eet al. 2025 Yu et al.
Our work most closely related to recent studies that explore f-divergences for policy optimization. Notably. Wang et al.] (2024): Han et al. (2025) replace the reverse-KL divergence in the offline DPO objective with generalized f-divergence, deriving a new family of "f-PO" algorithms. However, our work is distinct: unlike f-PO's s focus on offline preference alignment, we utilize online RLVR with verifiable rewards. Our objective is to address Pass @K k diversity collapse through mass- covering f-divergences rather than human preference alignment.
3
PRELIMINARIES
3.1 S-DIVERGENCE
In information theory, an f-divergence is function (p(p)) that measures the difference between two probability distributions and q. Given convex function f R + R such that f(1) = 0, the f-divergence defined as:
Dy(pl1a) = f g(x)
p(x) ((x)
dx
This general form allows for the unification of many common divergence measures under a single theoretical framework. The condition f(1) 0 ensures that (r(pl)) 0 if and only if p g. Many common divergences are special cases of f-divergences, corresponding to a specific choice for the generator function f. For instance, KL divergence and JS divergence are all instances of f- divergences. The generator function for JS divergence (u log u u- (u fu++1 of some prominent f-divergences and their corresponding generator functions is provided in
log 1/2 A summary
3

---

Published as a conference paper at ICLR 2026
3.2 MARKOV DECISION PROCESS
We consider a discounted Markov Decision Process (MDP) defined by the tuple (S, P.r. pu. 7), where S is the state space, A is the action space, P(s'ls, (s,a) is the transition probability, r(s) the reward function, Po is the initial state distribution, and E [0, 1) is the discount factor. A stochastic policy ={a]s) defines probability distribution actions for each state. The goal is maximize the performance objective, i.c., the expected cumulative discounted reward:
J(x) Eror
/r(ss)
where so Pu(su), a₂ Taalls)), 81+1 P(sl411s(,a))
(1)
A seminal result from Kakade & Langford (2002) expresses the performance of policy
in terms of an older policy Told using the advantage function A (s))))=(T)) = J(TTM)) +
By 1>== 7º AAaa az)l. This can be rewritten using the discounted state visitation distribu-
tion, (s((s) _ Ei p s|m), which gives the probability distribution over states encountered
under policy
J(x) (m PE)(s) Σ
ATol (8.
SES
BEA
Directly optimizing this expression is difficult because the state distribution Pr depends on the new policy I. Therefore, algorithms like TRPO and PPO optimize surrogate objective L by approximating Pr with the distribution from the old policy, Prai"
L (T) = [(Tua) + Σ Parna (s)
> [((aas)Ang Amals)) (s.
(3)
#ES
aEA
This approximation is reliable when л is close to Told, forming the basis of modern policy gradient methods.
4 METHOD
In this section, we detail our implementation of reinforcement learning (RL) with an f-divergence regularizer. Our objective is to find the optimal policy for the following optimization problem:
max Eg (Ea-mo(11)) (olpaa/alll////a/aaat////////// 17g
(4)
In this formulation, /re/ serves as the initial reference policy, such as a base or SFT model. The term r(a ()) represents the reward for taking action in response to query q. and the hyperparameter 7) controls the penalty for deviating from Tref. While standard algorithms like PPO can optimize the reward term, discussion focuses estimating the f-divergence term. A uniform application of this approach across all queries in dataset D can be suboptimal. For queries where Fref performs well, aggressive reward maximization is unnecessary and risks degrading performance. Conversely, for challenging queries where Tref struggles, the f-divergence term can overly constrain the policy To, limiting its ability explore better solutions. To address this, we propose targeted strategy that partitions the dataset D into two subsets. The methodology is divided into two phases: a pre- sampling stage and online training stage.
4.1 PRE-SAMPLING STAGE
We consider a specific instance of f-divergence, Forward-KK. defined as:
D (T (rees) A DRI (Treer]])() Ennw (og(TTor log())(a))))] To facilitate computing this expectation, we adopt pre-sampling strategy. Before training wo- we partition the dataset D. For each query Q, we generate and evaluate l independent samples. Based on a correctness threshold, each query classified as either "near-perfect" or "exploration," splitting D into Ppet and Deep- This partition allows our agent to focus challenging examples in Deep while using a KL divergence constraint to maintain performance on mastered examples in Dpet- To mitigate sampling bias, we further refine Pett- For each query in this subset, draw one final sample. We only retain the query in peet this new sample is correct; otherwise, we discard the query move it Desp-
4

---

Published as a conference paper at ICLR 2026
4.2 ONLINE TRAINING STAGE
In the online training stage, we simultaneously train the model using two distinct loss functions tailored for the Desp and Prer datasets. For samples from Deep. want the model have maxi- mum freedom for exploration. Conversely, for samples from pet, we want the model to retain its capabilities. To achieve this, employ two key f-divergences: Forward-KI. and JS divergence. Our method is thus divided into two approaches: DPH-F for forward-Ke KL and DPH-JS for JS diver- gence. Forward-Ki divergence, assigns high probability to an action, but the new policy assigns a near-zero probability. This property encourages the new policy T to maintain coverage of all modes of the reference policy Tref, thereby preserving its original diversity. JS divergence provides a symmetric and more stable alternative to KL divergence. encourages the new policy my to maintain high similarity with the reference policy Øref while achieving high performance, effectively preventing policy collapse.
we define it, penalizes instances where the reference policy Tref
4.2.1 GENERATOR- BASED IMPLEMENTATION
Our implementation relies on pre-sampling from the reference policy. This approach allows us to compute the divergence term using a static dataset, eliminating the need to run inference with the reference model during the online training loop.
Loss for Deep For challenging samples in Desp. we remove the KL divergence penalty from the loss function entirely. This allows the model to perform pure policy optimization based solely on the reward signal, enabling more aggressive exploration. Specifically, the loss function for these samples is the standard PPO-clip objective:
10. min (prr(0)Ar..
CDPH-cxp(() -E DmDaap os
G
( (-14)
i=
1=1
(5)
clip (p),c(0),111 +))++++44 A..))
Please refer to Appendix E for the definitions of the symbols in the formula.
Loss for Dpet For all f-divergences, the general formula for calculating the loss on Pper is:
(0) = Eq [Dy
(6)
mo(alo) ). where a represents a sin- gle response sampled once from the reference policy Fref (-lg). For DPH-JS, the loss term is: Eyg E. (rlf(alg)
For DPH-F, the loss term is: Eg Ea (res(alg) log (rer(alg))
aalog log (ut1)) where u = ro(alg)/ares(clt
4.2.2 OVERALL LOSS FUNCTION
The total loss for a given batch is a combination of these two objectives. For a mixed batch, we determine whether the data comes from Dyer or DDep and then calculate the corresponding loss:
CDPHRRI = Lexp (0) - ())r,())),
4.3 ENHANCED MONOTONIC IMPROVEMENT GUARANTEE
In this section, we derive an enhanced monotonic improvement guarantee for TRPO-style a algorithms that leverage our proposed method. Our framework is built upon two distinct datasets: Ppef. which stores near-perfect reasoning trajectories, and Deep. an exploration set for tasks the model has not yet mastered. Central to our analysis is a conditional reference policy. Tepl, whose definition depends on the data source. For states associated with the near-perfect dataset (D (Dpet), Tee is the policy induced by the stored correct trajectories. The f-divergence regularization term D y (I (pe) pulls the learned policy towards these expert solutions. For states associated with the exploration dataset (Desp),
5

---

Published as a conference paper at ICLR 2026
Tet is defined as the current policy This makes the regularization term (x(T] equal to zero, thereby disabling it and permitting unrestricted exploration. To analyze the effect this conditional regularization, our derivation hinges the following mild assumption.
Assumption 1. For any policy a encountered during training, there exists a constant 8 2 such that for any state s, the expected advantage of actions from the reference policy Tpef evaluated with respect to n, is lower-bounded by ò: Bopgr)pp(-1s)) [A A (s, apet) 2
This assumption is well-founded. In regions of the state space well-represented by the near-perfect dataset pef. the actions from pet are expected be superior, yielding a positive advantage (Ó 0). Conversely, for states associated with Dexp. where our method effectively applies no regularization, the bound trivially holds with O. Leveraging this assumption, we present the following theorem, which establishes stronger lower bound on policy improvement than the one in the original TRPO analysis (Schulman al. 2015).
Theorem (Enhanced Monotonic Improvement). Let an = max, DKL(TT ( [T and 02 max, (=(()-)) (Tpp((()))))) where the divergence Dj can be the forward-KL. -L, o-divergence, or Prnsenn-Shanno divergence. If A'Assumppion Anssumption[hoolee Tholds, then the following bound on policy improve- ment is guaranteed:
2ga1En
[(x) - To
( (=) (1 7)2 +eg
where c, = 1 Caraartaa Ex = max s. A (8, a)l and Cpef = max,,a (s, Cy (1-7)=-
posilive constant depending on the choice of the f-divergence.
Comparing with the monotonic improvement theorem in Schulman et al. (2015), where the right hand side of the inequality is 2yaies [1-?]² we have an additional term ey. On the distribution associated with expert dataset Ppct. where 5 >> O and effective regularization keeps a2 small, ef becomes a positive bonus. This results in a strictly better lower bound on policy improvement than in the original TRPO analysis, rigorously justifying that method accelerates convergence by leveraging known expert behavior. On the exploration set Dexp, the bound gracefully reduces to the standard TRPO guarantee. The proof of Theorem [ follows a similar line of reasoning the analyses in Schulman et al.] (2015) and Kang et al. 2018). The detailed derivation is deferred to the Appendix for the completeness.
5 EXPERIMENTS
In this section, we conduct extensive experiments to demonstrate the effectiveness and generalization of DPH-RL. The experimental setups were applied to two types of tasks: SQL (where the LLM generates SQL code and executes it using tool) and mathematical reasoning, using both Llama 3.1 Meta Al (024) and Qwen2.5 Team (2024) series models. In our experiments, we partition the dataset into Deup and Ppet based the capabilities of each model. For the Llama model, we define the Dper using a success rate of 6 out of 8 attempts, while for the Qwen model, this threshold is raised to 7 out of 8 attempts. Since our method is orthogonal to existing GRPO variants, use GRPO, DAPO, and reverse-KL. (RKL) as baselines. Please refer to Appendix G for training details.
5.1 SQL
As shown in Table in the Bird dataset, the Pass@8 scores for both GRPO and DAPO are lower than the base model, while our DPH-F and DPH-JS methods surpass the base model. This indicates that strategy possesses more robust ability to maintain model diversity. Specifically, DPH- IS shows P scores that are 4.3% and 3.3% higher than GRPO and DAPO, respectively. For the Spider dataset, which is a cross-domain SQL task, the Pass @k metrics for all models gener- ally suffer from performance degradation. However, DPH-F and DPH-JS can maintain accuracy levels closer to the base model. While DAPO performs better than GRPO on the Bird dataset, its performance is more unstable on the cross-domain data. Regarding cross-domain performance preservation, DPH-F demonstrates more powerful capability, with its Pass @ 16 scores being 9.0% higher than DAPO. This suggests that for simple tasks that the model can already handle correctly, preservation of its original capabilities is more crucial than pure exploration. We conducted experi- ments OmniSQL-322 with the results presented in Table 8 Its Greedy performance on the Bird dataset surpasses all current open-source models.

---

Published as a conference paper at ICLR 2026
Table 1: Results of Llama-3.1-8B-Insteee on SQL Tasks, The highest scores for RL Models are highlighted. Figure shows the error bars for each method over three training runs.
Bird
Spider
Model
Greedy
Pass@8
Pass 16
Greedy
Pass@8
Pass 16
Base Model
42.4
68.8
75.0
90.9
93.2
RL Models
GRPO
66.2
67.7
73.0
79.5
80.6
DAPO
60.0
67.2
69.0
71.1
75.3
76.7
RKL
60.0
69.8
71.8
71.0
79.0
80.6
DPH-F
60.4
70.1
71.6
77.4
84.5
85.7
Divergence
-0=022
60.1
67.0
69.1
75.4
78.8
80.1
-=0.5
68.0
69.8
80.4
82.5
-00
60.5
69.2
70.4
75.8
81.7
83.6
DPHIJS (Generator)
-000.01
59.8
66.7
68.1
76.4
79.1
80.8
0005
61.3
69.5
70.9
75.3
82.7
83.8
022
62.8
70.5
72.4
76.0
82.7
84.1
DPHI-J (Divergence Definition)
-0011
59.6
66.6
68.2
76.7
80.5
81.5
0.055
61.4
69.6
71.3
75.9
83.5
84.7
0,2
62.4
70.1
71.9
78.6
85.2
86.7
Table Out-of- Out-of-Domain Performance. Evaluating SQL-trained models on mathematical tasks. 0.2 indicates the use of a-divergence with a 0.2.
Pass @ 64
Pass 16
Pass @8
Model
Avg
AIME24
AMC23
Math500
Olympiad
Minerva
College Math
(SQL-trained)
Base Model
40.0
95.0
81.2
46.4
54.0
45.5
60.35
RL Models
GRPO
33.3
77.5
72.0
41.8
51.2
38.4
52.37
-DAPO
30.0
77.5
72.8
44.4
52.3
38.8
52.63
-RKL
23.3
72.5
70.8
33.8
49.2
41.1
48.45
-0=0.2
40.0
90.0
79.8
44.0
51.8
41.8
57.90
-00055
36.7
85.0
80.2
43.7
50.7
43.4
56.62
-00
43.3
87.5
80.8
43.1
54.0
44.2
58.82
DPH-F
46.7
95.0
80.8
46.1
54.3
43.0
60.98
-DPHJJS
40.0
92.5
81.2
48.2
53.8
45.7
60.23
Ablation Study Our method introduces plug-and-play f-divergence loss, weighted by hy- perparameter which we analyze in an ablation study. As shown in Table [T when the value of 7 is minimal, actual setup and performance approximate that of DAPO trained exclusively on Dpet. Increasing the value of r leads to steady growth in the Pass @ 16 score, demonstrating the effectiveness of our method. Additionally, we compared implementations of f-divergence: the "Generator" form and the "Divergence Definition" form (detailed in Appendix F). Although both yield similar performance, the "Divergence Definition" form is computationally expensive as it quires resampling from va and additional reference model to calculate Tref (alq). Consequently, "Generator" form is significantly more time-efficient.
Performance OOD Tasks As shown in Table we evaluated the diversity of our RL-trained models on five out-of-domain (OOD) mathematical datasets using a Llama model. (We excluded the AIME25 dataset due to its high fluctuation, ensuring fair comparison.) Without incorporating

---

Published as a conference paper at ICLR 2026
additional caneral-purposs data, methods like GRPO and DAPO cause models to overfit the SQL training domain, leading to a significant drop in performance on other OOD tasks. In contrast, our DPH-F and DPH-JS methods maintain higher average performance, surpassing DAPO by 8.35% and 7.6%, respectively.
0.9
60
Pass@k
0.7
Greedy
58
57
0.6
56
Uamaa-3.1
1.rra-31--18--imes 3.1-818-nnstrrc
0.5
DAPO
55
0.4
Reverse
DPH-F
54
DAPO Baseli ne
DRH-jS
0.005
0.01
0.02
0.05
0.2
16
32
64
96
128
160
192
224
256
Number of Samples
the value n
(a)
(b)
Results Llama models on the BIRD dataset. (a) Performance of different methods different pass @k levels. (b) The impact of different KL term parameters on RKL.
Figure 2:
Performance with Massive Sampling We evaluated the performance of various methods as the number of samples increases. As illustrated in Figure 2a the Pass@k k scores for all methods reach convergence by 256. Notably, standard GRPO and DAPO saturate early, plateauing at approx- imately 75%. Although DAPO outperforms GRPO at small l values, its mechanism fails to scale effectively to larger k. The DPH family of methods significantly outperforms the GRPO baseline. Regarding Reverse KL, while it yields higher performance than GRPO, it remains significantly in- ferior to the DPH series. Overall, DPH-JS emerges as the superior method across both Pass and Pass 3c metrics.
Comparisons Under Different f-divergences In Tables and 2 we analyzed the impact of dif- ferent f-divergences on the Llama model. First, DAPO GRPO, which lack any form of f- divergence constraint, experienced severe Pass @k collapse tasks. Conversely, while RKL could maintain a high @ass@k k on the training task's s test set, its per- formance collapse tasks with different distributions was any KI. penalty. This highlights the limitations of reverse-KL. it causes the model to over-focus on the training data's distribution, thereby completely sacrificing generalization. To validate the reli- ability of our experimental baseline, we show the performance of RKL with different values in Figure 2b) When 1) 0.02, the model's learning performance fails surpass that of DAPO. This in- dicates that the chosen value is too large, and therefore our selection of r = 0.01 is well-justified. The divergence sits between the forward-Ki KL and reverse-KL. effectively preventing the model from veering toward extremes. Across all datasets, it shows a clear trend: as the a value increases, it theoretically approaches the capabilities of forward-KI. KI. while experimentally maintaining higher Pass@k. k. In contrast, DPH-F and DPH-JS methods demonstrated strong generalization across differ- ent tasks. The DPH-JS method, in particular, not only maintained higher greedy performance on training tasks but also preserved a high Pass @k value out-of-domain tasks, which fully demon- strates its significant advantage in preserving a multi-peak distribution.
both in-domain and out-of-domain
severe than methods without
5.2 MATHEMATICAL REASONING
For the mathematical tasks, we present two types of results, shown Table and Table D respec- tively. Table 3 shows the Pass @ scores various test sets, while Table shows the Mean @k scores, which is the average accuracy calculated by sampling the entire dataset times. Regard- ing mathematical ability, different model families have completely different capabilities. For the Llama model, the improvement through RL is very limited; the average mean value when using GRPO only increased by 0.93, while Pass@k decreased by 3.26. By contrast, the Qwen model is
8

---

Published as a conference paper at ICLR 2026
Table 3: The Pass metric for models trained on math datasets. In the Llama experiments, the DAPO settings (0.2g)=0.288 caused training to crash. To conduct effective experiments, we aligned the settings for all Llama experiments on the Math dataset with GRPO (Chigh-..2).
Pass@64
Pass@16 16
Pass @
Model
Avg
AIME24
AIME25
AMC23
Math500
Olympiad
Minerva
College Math
Llama-3. 3.1-88--Inttrut
Base Model
40.0
23.3
95.0
81.2
46.4
54.0
45.5
55.06
RL Models
-GRPO
33.3
26.7
80.0
79.6
43.3
56.6
43.1
51.80
36.7
16.7
75.0
80.0
39.1
43.4
49.64
-DPH-F
36.7
26.7
90.0
80.6
44.3
57.3
45.5
54.44
-DPH-JS
40.0
26.7
82.5
81.4
45.8
58.1
45.1
54.23
@wen2.5-Math-7H
Base Model
63.3
56.7
87.5
88.8
61.9
56.6
42.9
65.39
RL Models
-GRPO
56.6
50.0
97.5
93.0
62.8
64.0
50.1
67.71
-DAPO
63.3
46.7
97.5
92.2
63.1
64.3
48.7
67.97
-RKI.
66.7
40.0
97.5
92.0
64.6
51.3
68.01
-DPH-F
73.33
50.0
97.5
92.4
63.8
64.8
50.9
70.39
-DPH-JS
66.7
53.3
100.0
92.8
65.2
66.2
51.0
70.74
a more capable model, which allows it to achieve a significant boost in both Mean@k (approxi- mately 20%) and Pass@k. We selected these two models to explore the performance of the DPH method under different model strengths. For Llama, DPH-JS maintained the original Pass@k value and significantly improved mean @k AIME, which demonstrates that DPH-JS is powerful and highly versatile method with strong capabilities in both exploration and preservation. Furthermore, balanced improvement than GRPO, increasing both Pass @ass@k and mean @k
DPH-JS provided averages, showing it does not sacrifice one metric for the other. For the Qwen model, the overall performance showed dual trend: a slight decrease in Pass@k an increase on other, simpler datasets. In both scenarios, the DPH-JS method achieved the best performance. AIME, it maintained diversity closer to that of the base model compared both GRPO and DAPO. On the other datasets, it achieved higher Pass @ k. This trend highlights key challenge for large models RL: the need to both solidify existing knowledge while also exploring new boundaries. The DPH-JS method successfully balances these two conflicting goals.
the difficult AIME test set, but
6 ANALYSIS
6.1 ANALYSIS OF KEEP AND EXPLORATION
Bird Additional Exploration
Spider
Bird-Keep
Spider-Keep
Additional Exploration
0RP0
DAPO
DPN-
DRW-15
10
GRP0
DAPO
DPH-7
DPW-S
78
64P0
DAPD
DRHF
DPW-5
GRPo
D6R0
DRN.
Figure 3: Evaluating RL-Tuned Llama models vs. base model: knowledge retention (keep) vs. new
solution discovery (additional exploration).
To gain more granular understanding of why method achieves higher Pass @k scores, tracted correct and incorrect samples from the base model's Pass results. These were designated data subsets D correct and D Dorrong respectively. We then compared them with the model after

---

Published as a conference paper at ICLR 2026
reinforcement learning. The proportion of samples it got correct Deagrect is its keep rate, while the proportion got correct on the incorrect samples from Duub is its additional exploration rate. As shown in Figure the keep rate of GRPO and DAPO both decreased the two datasets, while the KL. method mainly maintained a higher keep rate to maintain a higher Pass @k value. DAPO had higher exploration within the domain than GRPO, but the keep outside the domain dropped significantly. RKL could only have a high keep rate on the same Bird test set as the training set, while dropped to level similar to GRPO on Spider. DPH-JS had higher exploration on the Bird test set, while DPH-F was higher on the Spider dataset. Furthermore, Appendix H.2 shows that DPH the most stable and effective method during training. while other mainstream approaches either have poor generalization (e.g., DAPO) or diversity collapse late in training (e.g.. GRPO and RKL).
6.2 IMPACT OF SPLITTING TEX AND DPeR
Table 4: '8 of 8' indicates threshold where samples correctly solved in 8 out of 8 attempts are assigned to per in DPH-JS. 'Ful Data' implies using the entire dataset for both per and Dexp-
Bird
Spider
Model
Size of Desp
Size of Dper
Greedy
Pass@8
Pass@ 16
Greedy
Pass@ 8
Pass @ 16
GRPO
8958
0
58.5
66.2
67.7
73.0
79.5
80.6
DAPO
8958
0
60.0
67.2
69.0
71.1
75.3
76.7
8 of
8266
692
61.4
71.6
73.5
75.4
86.3
87.2
7 8
7429
1529
61.3
71.2
73.6
75.1
85.8
87.2
of 8 5 of 8
6710
2248
62.8
70.5
72.4
76.0
82.7
84.1
5783
3175
60.2
70.6
74.0
74.3
86.2
88.9
Full Data 8958
8958
57.4
73.1
76.6
71.8
88.4
90.3
In the main experiments, we established an accuracy threshold to partition the data into Dexp and Pret This section provides a detailed analysis of the impact of this threshold. As illustrated in T the performance difference between the '7 of 8' and '8 of 8 settings is negligible. At '6 of 8' the model sacrifices some diversity to achieve higher greedy performance; by removing simpler samples from Dexp. the model focuses exploration on more challenging problems, while the KL term ensures continued learning on well-mastered samples. Conversely, at '5 of 8', the significant reduction in Deep data leads to decline in greedy performance, despite yielding improved Pass @k scores. the 'Full Data' setting, greedy performance is poor-evon inferior to the GRPO base- line-yet it achieves the highest Pass@ @ k. Although the large volume of D per preserves sufficient diversity, 'Full Data" suffers from two critical issues: (I) Our goal is to preserve the diversity of correct solutions, not the noise of incorrect ones. Forcing the selection of single correct response from samples solved correctly only once out of eight attempts introduces statistical bias. (2) Regard- ing the training objective, the model can trivially mimic the sole correct sample for each query to maximize reward and minimize the KI. term, thereby optimizing the loss without genuine learning- In contrast, '8 of 8' represents an unbiased statistical approach requiring no artificial sampling. To compensate for the reduced data scale in this setting, we retain all 8 sampled trajectories for each instance during training. In our experiments, although '8 of 8' allocates only 8% of the training data to Dpet, it comprehensively outperforms DAPO and GRPO. This demonstrates the robustness of DPH-RL even under extreme conditions, suggesting that directly adopting the '8 of 8' threshold is the most straightforward strategy for its application.
CONCLUSION
We propose DPH-RL to overcome the limitations of reverse KL divergence in reinforcement learn- ing for LLMs. By leveraging f-divergence, our method mitigates both diversity collapse and catas- trophic forgetting. We demonstrate that two implementations- "Generator" form that requires no reference model during training and a "Divergence Definition" form-yiel similar performance. This makes our method highly adaptable, providing enhanced capabilities with computational costs comparable to GRPO. We explicitly recommend the DPH-JS (Generator) variant with the '8 of 8' threshold as the optimal configuration for future researchers. However, a key limitation, shared with GRPO, is the dependence on the base model's 's capabilities, which become scarce more challeng- ing datasets. Future work will therefore focus on resolving this trade-off, aiming to develop methods that can both learn effectively from imperfect data and mitigate the resulting estimation bias.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
Our datasets are all based on open-source datasets. The experimental methodology, data proportions, and hyperparameter settings are detailed in Appendix G
REFERENCES
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effec- tiveness of entropy minimization in Ilm reasoning, 2025. URL
wwwpii.orrr/aa.. abs/
2505. 15134
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scal- ing reinforcement learning advanced reasoning models, 2025. URL github. 1o/blogg/225/Pooozz/
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Seed-grpo: Semantic entropy en- hanced grpo for uncertainty-aware policy optimization, 2025. URL htt.htts///axii.orrrg
abs abs/25055 0t5//2555.1233
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective on reinforcement learning for Ilms, 2025. URL /ttps::ss////x....oooo http:///.......0.//////5....... 14758
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan. Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models, 2025. htt.ttts///aaiii.otott
2505. 22617
Jia Deng, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, and Ji-Rong Wen. Decomposing the entropy- performance exchange: The missing keys to unlocking effective reinforcement learning. 2025. URL
abs/abos555 02260
Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Ying- wei Ma, Jue Chen, Binhua Li, al. RI-plus: Countering capability boundary collapse of llms in reinforcement learning with hybrid-police optimization. arXiv preprint arXiv:2508.00 00222, 2025a.
Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Ying- wei Ma, Jue Chen, Binhua Li, et al. RI-plus: Countering capability boundary collapse of 1lms in reinforcement learning with hybrid-policy optimization. arXiv preprini arXiv:2508.0022 2025b.
Alexei A Fedotov, Peter Harremoës, and Flemming Topsoc. Refinements of pinsker's inequality. IEEE Transactions on Information Theory, 1966):441--1499 2003.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Pouya Hamadanian, Arash Nast-Esfahany, Malte Schwarzkopf, Siddartha Sen, and Mohammad Al- izadeh. Online reinforcement learning in non-stationary context-driven environments. In Inter- national Conference on Learning Representations, 2025.
Jiaqi Han, Mingjian Jiang, Yuxuan Song, Stefano Ermon, and Minkai Xu. f-po: Generalizing pref- erence optimization with f-divergence minimization. International Conference on Artificial Intelligence and Statistics, pp. 144-1152. PMLR, 2025.

---

Published as a conference paper at ICLR 2026
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In ACI (1), pp. 3828-3850. Association for Computational Lin- guisties, 2024.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner technical report, 2025. URL https:
//arxiv. abs/ 25tp:/...rrr/aaa25555223333 2531.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurlPS Dalasets and Benchmarks, 2021.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference machine learning, pp. 267-274, 2002.
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In International conference on machine learning, pp. 2469-2478. PMLR, 2018.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Ma- lik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha- jishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2025. URL https:
/ //axxiv.org/aat abbs//bb///// htt.aa///./..1... 2411. 15124
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models. In NeurlPS, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q. Jiang. Ziju Shen, al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. hhtps: //huggi www.giiggace.... htt.faacecoo/aaaaaeeee wwt.oo/daaaataaeeeeeee Numi namath 2024a. Hugging Face repository, 13:9.
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang. Bowen Qin, Ruiying Geng, Nan Huo, et al. Can Ilm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024b.
Long Li, Xuzheng He, Haozhe Wang, Linlin Wang, and Liang He. How do humans write code? large models do it the same way too. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 4638 4649, Miami, Florida, USA, November 2024c. Association for Computational Linguisties. doi: 10. 10.18653/v1/202 emnlp-main main.267. URL https::/aclaathhoottt/// oorg org/ 2024. wwwee-nnnnpmme.eee ma0--mainnnn 267/
Qingbin Li, Rongkun Xue, Jie Wang, Ming Zhou, Zhi Li, Xiaofeng Ji, Yongqi Wang, Miao Liu, Zheming Yang, Minghui Qiu, and Jing Yang. Cure: Critical-token-guiieed re-concatenation for cntropy-collapse prevention, 2025. URL http:/tps///aii....oooo
11016
Xiao Liang, Zhongzhi Li. Yeyun Gong, Yelong Shen. Ying Nian Wu, Zhijiang Guo, and Weizhu Chen. Beyond pass@1: I: Self-play with variational problem synthesis sustains rlvr, 2025. URL
htttps://arat httpaaawwwvvvorraa/a//aaaaaa/a/e 14029
12

---

Published as a conference paper at ICLR 2026
Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. Uft: Unifying supervised and reinforcement fine-tuning, 2025a. URL htthhttps://// httpppp///a..ll.oo///aaaa///s//s ab//abbs///5// 16984
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and
Wei Ping. Acereason-nemotron 1.1: Advancing math and code reasoning through sft and
synergy, 2025b. Utthttss::/ //aa:::///xvvi/ www.rxvv..vggaaaa. org/absg////o.oooooo 2506 12255666382 13284
Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu. Ghpo: Adaptive guidance for stable and efficient Ilm reinforce- ment learning, 2025c. URL http::://arii...r///aaaaaaa///aa
10628
Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Renjie Liao, and Christos Thrampoulidis. Beyond accu- racy: A policy gradient reweighting approach for pass @ k maximization in llms. In 2nd Al for Math Workshop@ ICML 2025.
Meta AL. Introducing meta Llama 3: The most capable openly available LLM to date, April 2024. meta. com/ www.o//nntaallaaaaaaaaaaaaaaaaaa Accessed: 2024-04-18.
URL
MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haigang Zhou, Haimo Zhang, Han Ding, Haohai Sun, Haoyu Feng. Huaiguang Cai, Haichao Zhu, Jian Sun, Jiaqi Zhuang, Jiaren Cai, Jiayuan Song, Jin Zhu, Jingyang Li. Jinhao Tian, Jinli Liu, Junhao Xu, Junjie Yan, Junteng Liu, Junxian He, Kaiyi Feng. Ke Yang, Kecheng Xiao, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Li, Lin Zheng, Linge Du, Lingyu Yang. Lunbin Zeng. Minghui Yu, Mingliang Tao, Mingyuan Chi, Mozhi Zhang, Mujie Lin, Nan Hu, Nongyu Di, Peng Gao, Pengfei Li, Pengyu Zhao, Qibing Ren, Qidi Xu, Qile Li, Qin Wang, Rong Tian, Ruitao Leng. Shaoxiang Chen, Shaoyu Chen, Shengmin Shi, Shitong Weng. Shuchang Guan, Shuqi Yu, Sichen Li, Songquan Zhu, Tengfei Li, Tianchi Cai, Tianrun Liang, Weiyu Cheng, Weize Kong, Wenkai Li, Xiancai Chen, Xiangjun Song, Xiao Luo, Xiao Su, Xiaobo Li, Xiaodong Han, Xinzhu Hou, Xuan Lu, Xun Zou, Xuyang Shen, Yan Gong, Yan Ma. Yang Wang. Yiqi Shi, Yiran Zhong, Yonghong Duan, Yongxiang Fu, Yongyi Hu, Yu Gao, Yuanxiang Fan, Yufeng Yang. Yuhao Li, Yulin Hu, Yunan Huang, Yunji Li, Yunzhi Xu, Yuxin Mao, Yuxuan Shi, Yuze Wenren, Zehan Li, Zelin Li, Zhanxu Tian, Zhengmao Zhu, Zhenhua Fan, Zhenzhen Wu, Zhichao Xu, Zhihang Yu, Zhiheng Lyu. Zhuo Jiang, Zibo Gao, Zijia Wu, Zijian Song, and Zijun Sun. Minimax-ml: Scaling test-time compute efficiently with lightning attention, 2025. URL https: //axxiv.rg//ab// 250. 13585
OpenAl. GPT4 technical report. arXiv preprint 2:Xiv:2303.0877,,200 2023.
OpenAl. Learning to reason with 1lms, 2024. URL. https://openat wwtennaa.ccoomammm /tom/index/ ndex
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning, 2025. URL htt.tttss///exiii.oortt 2505. (2665.22660
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMIR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0632 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YKLi. Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 240.::2442223300 2024.
13

---

Published as a conference paper at ICLR 2026
Qiannan Shen and Jing Zhang. Ai-enhanced disaster risk prediction with explainable shap anal-
ysis: A multi-class classification approach using xgboost, December 2025. URL https:
//www. ze wesearchssuare. com http/aatcceereee//1eeeeeee December 31, 2025.
Preprint, Version I, posted
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: scaling instruction tuning for mathematical reasoning. In Proceedings of the 41st International Conference on Ma- chine Learning, ICML'22. JMLR.org. 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding. Hao Hu, Hao Yang. Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng. Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang. Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma, Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Weixin Xu, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang. Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang. Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang, Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang, Zhilin Yang. Zhiqi Huang, Zihao Huang, Ziyao Xu, Zonghan Yang, and Zongyu Lin. Kimi k1.5: Scaling reinforcement learning with llms, 2025. URL URLpttpss: arxiv. wttp.agg/gg.....aa../////
Qwen Team. Qwen2.5: A party of foundation models, September 2024, URL htLhhtts::/// //we/////wwmmmmmmm wtt.bb.../bll///////oo/ qwwlll///ww/e.//////m/
Christian Walder and Deep Karkhanis. Pass@ k policy optimization: Solving harder reinforcement learning problems. arXiv preprint 2.XXv:2505.15200 2025.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: General- izing direct preference optimization with diverse divergence constraints. In The Twelfth Interna- tional Conference on Learning Representations.
Chaoqi Wang, Yibo Jiang, Chenghao Yang. Han Liu, and Yuxin Chen. Beyond reverse KL: General- izing direct preference optimization with diverse divergence constraints. In The Twelfth Interna- tional Conference Learning Representations, 2024. URL
2=rum2id=2cRemmx>>
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. 2025. URL htLLtttts////xvv.coo org/ abs/
2556600990 939
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off- oof-policy guidance, 2025. URL https://arxiv. org/abs/2504.1499 14945.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhennu Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: /ttp:///aaivv.veaa org/ abs/ / 2505 09388
Qiying Yu, Zheng Zhang. Ruofei Zhu. Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guang- ming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu,
14

---

Published as a conference paper at ICLR 2026
Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma. Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingx- uan Wang. Dapo: An soune Ilm reinforcement learning system at scale, 2025. URL
https:
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, al. Spider: A large-scale human- labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 3911-3921, 2018.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In 2nd Al for Math Workshop ICML 2025, 2025. URL https: //openrevien ew. net. for d=upeht.Vgglb
Hongzhi Zhang, Jia Fu, Jingyuan Zhang, Kai Fu, Qi Wang, Fuzheng Zhang, and Guorui Zhou. Rlep: Reinforcement learning with experience replay for Ilm reasoning. arXiv preprint vrXiv:2507.07451, 2025a.
Xuan Zhang, Ruixiao Li, Zhijian Zhou, Long Li, Yulei Qin, Ke Li, Xing Sun, Xiaoyu Tan, Chao Qu, and Yuan Qi. Count counts: Motivating exploration in IIm reasoning with count-based intrinsic rewards. arXiv preprint drXiv:2510.1661, 2025b.
15

---

Published as a conference paper at ICLR 2026
GRPO (Rervese KL)
Base Model
6%
xalve
"Rapenting
treilt doarii
Oriess
Two Styles
94%
Distribution of Counts for Different Styles
One Styine
Two Styles
DPH-F (Forward KL)
impputi other styles.
Four Bty es
Five Shytes Stylis
20%
30%
One Styles Three Stylea Styles
81N
the left, we construct a base model that outputs multiple solution styles for SQL problems. This model is then used for reinforcement learning training. We calculated the probability of the number of times the model outputted different styles across 32 samples.
Figure 4:
A THE USE OF LARGE LANGUAGE MODELS
In our work, we exclusively use LLMs for writing refinement, which means we first write a piece of text ourselves, and then use the LLM to correct grammar, formatting, and other issues. For our experiments, we also use LLMs to help us fix code bugs and generate Python code for plotting.
B MULTIPLE STYLE CAPABILITY EXPERIMENT
We construct base model that outputs five different solution styles for SQL problems. This model then used for reinforcement learning training. Next, we sample each question 32 times. Based on the output prefix, count the number of times each distinct style appears within these 32 samples. For prefixes that are not among our original five styles (e.g., the model combines several different prefixes), we categorize them "Others" and treat them an additional style.
The result shows in Figure 4 On the base model, we find that it can fully output all styles. How- ever, after training with reverse KL-constrained GRPO, most outputs degenerate into single style. In contrast, forward KL significantly mitigates the degradation of the model's ability to produce different styles.
C PRELIMINARIES ON O /DIVEERGENCE
In Table we summarize several typical f-divergences and their corresponding generator functions fu)) with the definitions of forward and reverse KL divergence adapted according to the latest research. All listed divergences adhere to the standard properties of the generator function: strict convexity, lower semi-come continutt,, and the normalization condition f(1) = 0. Notably, this table clar- ifies that both forward and reverse KL divergence under specific parameters. This provides a unified mathematical framework for measuring and opti- mizing distribution shifts using कe in subsequent model training.
be unified as special cases of the a-divergence
16

---

Published as a conference paper at ICLR 2026
Table 5: Summary of some typical I-divergences Dy(pllg) together with their generator functions (u). In this paper, we let Tref be and the formula for f-divergence references an improved version (Wang al. 2024), the definitions for forward-KL. LL and reverse- KL are swapped compared to Wikipedia. For all divergences, the generator function f R+ + R U [+oo] is strictly convex. lower-semicontinuous and satisfies [(1) = 0. Forward-KL. and reverse- KL are the special case of alpha divergence.
Name
Divergence Definition (p(pl)g)
Generator f(u)
Reverse- (L
J p(x) log p(z) dr
log
q(8)
Forward-KLL Soward-KL(a-ao = 1)
f((() log v(m) p(z)
dx
loog
a-divergence (o 1))
a(a- ((x))
p(x)
(u²-a (1-a)u -a)u-a)((ala- 1))
9(x))
p(x) 9(x)
1 dæ
Jensen-Shannon
1 [ p(x) log 2p(x))
log u
(x++1) log u=1
((=))+((()))
+ +g(z) log p(a)+o(=)
2g(x))
de
D THEORETICAL CONNECTION TO EXPERIENCE REPLAY AND REGULARIZED REHEARSAL
In this section, we elaborate on the theoretical interpretation of DPH-RL as hybrid method rooted in replay mechanisms. We demonstrate that while our method incorporates replay buffer component, significantly extends standard Experience Replay (ER) through the introduction of symmetric divergences.
D.1 DPH-F AS DIVERGENCE-BASE EXPERIENCE REPLAY
Minimizing the Forward-Ko KI. divergence, defined as Re- [- log To(x)). is mathematically equiv- alent to maximizing the liglliillhood of trajectories sampled from the reference policy Wref- In our framework, Dpef serves as curated buffer containing "perfect" trajectories. Consequently, the DPH-F objective acts effectively as an Experience Replay mechanism. By enforcing probability mass coverage on these high-quality solutions, it prevents the model from catastrophically forgetting previously mastered skills while optimizing rewards on the exploration Deup--
D.2 DPH-JS AS REGULARIZED REHEARSAL VIA DUAL-SIDED SAMPLING
While DPH-F acts as direct rehearsal, DPH-JS (Jensen-Shannon) introduces a more nuanced mechanism. Since the JS divergence symmetric and defined based on mixture distribution M = (Tref + /2) 2, its optimization necessitates dual-sided sampling.
Unlike standard replay which only samples from the historical buffer, minimizing DPH-JS involves comparing samples from two sources:
The Reference Buffer (Tref): Anchoring the model the valid distribution prevent collapse.
The Current Policy (TT): Sampling eeff-generated trajectories to regulate the model's deviation from the mixture mean.
This dual dependency makes DPH-JS a form of "Regularized Rehearsal." anchors the model to the reference distribution (similar to ER) to ensure diversity, but simultancously regulates the model's 's own exploration. This prevents the model from drifting too far into unstable regions without forcing strictly "memorize" the buffer rigidly Forward- KI. or SFT might.
17

---

Published as a conference paper at ICLR 2026
E RLVR ALGORIHTMS
E. GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
GRPO presents an innovative approach to policy learning that distinguishes itself from methods like Proximal Policy Optimization (PPO) by eliminating the need for an explicit value function. Instead, GRPO computes the advantage in group-relativ manner, offering streamlined yet effective opti- mization strategy.
For specific question-answer pair (g,a), GRPO's underlying behavior policy, Taar generates group of G' individual responses, denoted as {o:) The advantage for the i-th response within this ensemble then precisely calculated by normalizing the rewards specific to that group, {R+)
A T3 mean (R.),)
{
(R,) (R:)
Similar to the clipped surrogate objective found PPO, GRPO also incorporates a clipping mech- anism constrain policy updates. This helps maintain training stability and improve sample ef- ficiency by ensuring that new policies don't 't deviate too drastically from previous ones. Beyond this, GRPO further enhances regularization by directly adding Kullback-Leible (KL) divergence penalty term to its objective function. This penalty helps prevent the policy from drifting too far from reference policy, promoting stable and controlled learning.
The comprehensive objective function for GRPO articulated as:
(GRPO(!) -E
(g.0)D or ai (11))
0i
min
(0). Are, clip (p). -0,1--),+)+)) A...)
(10)
7 ( (T) ]
where 77 coefficient for the KL penalty, DKL DRL (To) quantifies the KL. divergence between the current policy Te and specified reference policy Tref. The term Po, represents the importance sampling ratio for the i-th response at time f, which is defined as:
To(or q, 0(,x))
Ps.
(11)
Te (oi,1 q, 0i.et)
A critical design choice in GRPO is its sample-level objective computation. Specifically, the loss is first averaged within each generated sequence, and subsequently, these sequence-level losses are averaged across various samples. This distinct computational approach, particularly when compared to teken-level optimizations, can significantly influence the algorithm's empirical performance.
Traditional GRPO applies single, unified loss function all training samples. This loss includes a KL divergence penalty, typically in the form of reverse KL divergence, which aims to keep the fine-tuned policy close to the base policy. This penalty is added directly the policy gradient loss. When applied to our experimental setup, the KI. penalty for samples from Deep would be:
LKI-GGROO Eg |-8 DxI. Tonse]]
(12)
where the reverse KL divergence is defined as:
Te (x)
DKL( (base)
Σ (x) log it
(13)
Toase (z)
Here, the term ( Thase) represents the Reverse KL Divergence. It measures the information lost when the base policy Thase approximates the new policy mo. This choice heavily penalizes the
18

---

Published as a conference paper at ICLR 2026
new policy Te for exploring actions that the base policy Tisse considers to have low probability. In other words, the new policy Te assigns a high probability to an action where the base policy Thase assigns near-zero probability, this divergence will be very large. The effect is strongly encourage the new policy No to stick to the modes (high-probability regions) of the original base policy Thsse which restricts the exploration space.
E.2 DYNAMIC SAMPLING POLICY OPTIMIZATION (DAPO)
DAPO is an enhancement of the GRPO algorithm, incorporating several key improvements. DAPO eliminates the KL penalty and refines the clipping mechanism, changing the upper bound from (1 + E) to fixed value of (1 +1++ upper)), where Eupper is set to 0.28. A core innovation of DAPO is its dynamic sampling mechanism, which moves beyond the nall-or-nothing approach sampling- Additionally, the algorithm applies token-level policy gradient loss and uses an overlong reward shaping technique.
F METHOD FOR DIVERGENCE DEFINITION
For f-divergences like forward KL, the Divergence Definition and Generator implementations are equivalent. However, for divergences such JS divergence, which require sampling from both the reference policy Tref and the new policy Wy!
( (T)) = DKL
Tbase
||
Thuse + To
DKL
Te
To
(14)
2
2
2
Since the JS divergence composed of two parts, we use data from the Pper loss to compute the first part DKL (Trasel =e For the training on the Dexp loss, we introduce a reference model to calculate the value of the second part DKL. (Tel DoanthTT
The loss for DPH-JS the Deep be written as:
G
lo:1
CDPH-cxp (0) = -E (g.a)~Depp orrite (-19)
min (pra(0)Ais) clip (p,, - E) A..))
00:
i=1
1=1
Thase + T
DKL
TTl
2
(15)
By adjusting the value of 31, make the values of the JS components within the same batch as close as possible, thereby achieving balance during training. f1 not a hyperparameter; only depends on the dataset size and 17. Its formula is:
D Deepl
B1 17*
(16)
The loss for DPH-JS on the Dper can be expressed as:
Tbase + Tg
LDPHEOT Eq
DKL
(17)
2
G TRAINING DETAILS
G. TASK SETTINGS
For the SQL task, we used the 3lama-3.1-81) 1-8B-Instruch and OmniSQL32B models. We performed RL training exclusively on the BIRD dataset et al. (2024b) and then evaluated the models on both the BIRD and Spider (Cross-Domain Domain) Yu et al. (2018) datasets. To validate the generalization of our method, we also tested it on mathematical reasoning task, treating as an out-of-domaal (OOD) evaluation.
19

---

Published as a conference paper at ICLR 2026
For the mathematical reasoning task, used the Llama-3.1-8B-Immn
and (wwe2.55-Math-77
models. We conducted RL training on a DAPO-17K subset, having first filtered out problems that
initially had @ score of O. For testing, used seven different datasets: AIME24 Li et al.
(2024)). AIME25, AMC23 Li et al. 2024a), Math500 Hendrycks et al. (021), Olympiad He et al.
2024
Minerva Lewkowywzewwwwcc..coooooo et al. 2022). and College Math Tang et al. (2224)
For the SQL tasks, the model will generate an SQL statement, and then we call the SQL executor to get the result. This result is typically a tuple, such as [(string, int), (string, int)]. We compute the Cartesian product of the prediction and the ground truth. For the math tasks, official Qwen2.5 evaluation tool for a detailed assessment, as this tool provides tailored evaluations for each dataset.
Evaluation
the
G.2 METHOD SETTINGS
Table 6: Configuration Comparison of RL Methods
Method
Online Data
Offline Data
Offiin Loss
Fhigh
Dynamic Sampling
GRPO
Desp.Dee Desp. Deet Dpet Deup
0.2
x
DAPO
0.28
RKL
0.01
0.28
a-divergence
Dior
a-divergence
0.01
0.28
DPHIF
Desp
Door
Forward KL
0.01
0.28
DPHI-JS
Deep
Doer
IS divergence
0.2
0.28
The implementation of RKL is consistent with the standard GRPO implementation. Our default implementation for DPH-RL uses the Generator method. The settings for our different methods are detailed in T For all RL algorithms, we consistently use token-level loss. In most of our experiments, the Ehigh for our DPH methods was set to 0.28. However, due to training instability in the Llama math experiments, we set the Ehigh to 0.2 for all methods.
G.3 HYPERPARAMETERS
Table 7: Hyperparameters for RL Training
Hyperpurameter
Llama-SQL
OmmiSQL-32B-SOL1
Llumu-Math
Owen2.5-Math
Batch Size
128
128
128
128
Learning Rate
le-6
le-6
2e-7
2e-7
Rollout Temperature
1.0
Rollout Top-p
0.95
0.95
0.95
0.95
Validation Temperature
1.0
Validation Top-p
0.95
0.95
0.95
0.95
PPO Epochs
1
Size Deep
6710
5169
4599
7009
Size Pep
2248
3789
1248
3199
Max Response Length
4096
Number of Rollouis
16
16
16
16
Training Fpachs
All experiments were conducted on 32 NVIDIA A800-80G GPUs using the VeRL framework for our RL algorithm implementations. For all experiments, we set the number of rollouts to 16. In our setup, the training batch size was 128, and the batch size for the Dper data was 256. This resulted in using 2048 samples from Deep and 256 samples from Dper at each learning step. yielding an effective ratio 8:1. A comprehensive list of the specific hyperparameters used provided in Table 7
In the offline phase, used the current model to perform 8 rollouts on the training data. For the SQL tasks, we separated the perfectly correct examples from the rollouts and used the remaining ones to construct Dexp- For the DAPO-17 17k dataset used in the math tasks, discarded all data where the model's ''s responses were entirely incorrect.
compuu/.com///en// wwweenn.//@u22.55555 M-Mthh
20

---

Published as a conference paper at ICLR 2026
H ADDITIONAL EXPERIMENTS
H.1 32B EXPERIMENTS FOR SQL
Table Results on Omnisql-322 for SQL tasks. * Indicates that the results are from the Bird Bench.
Bird
Spider
Model
Greedy
Pass@8
Pass 16
Greedy
P
Pass 16
OmniSQL-32B
Other Models
Infly-RL.S5 SQL--33-
70.1
XTYanQQL-Qwant SueenColees32
67.0
68.5
Command A
63.5
RL Models
GRPO
69.4
76.3
78.8
84.7
91.6
92.4
DAPO
69.9
76.8
79.2
83.8
DPH-F
70.4
78.6
80.8
86.1
DPH-JS
70.5
79.2
81.9
84.9
91.9
93.1
We also tested the effectiveness of DPH-F and DPH-JS on larger 32B model and found them to be equally effective. As shown in Table 8 please note that greedy score of 70.5 on the Bird dataset is already approaching the performance limit of open-source Single-Model:
H.2 ANALYSIS OF TRAINING PROGRESS
Liama Pasagd Progreas: Algortthma vs. Training Stups on Bird
Uaaa Passge Progress: Alporithmm vs. Training Steps on Spider
(a) Llama Bird
(b) Llama Spider
Figure 5: Llama P progress: Algorithms vs. Training Steps
In Figure we visualize the evolution of the Llama model's Pass@8 metric during training. On the Bird dataset, GRPO shows clear, gradual collapse, while DAPO's performance continuously cillates. Over the long term, DPH-F and DPH-JS both consistently maintain a higher Pass@8 value than the initial level. On the Spider dataset, DAPO performs extremely poorly, even falling below GRPO. Our experiments show that DAPO sacrifices out-of oft-of-domain generalization for improved per- formance in-domain datasets. While RKL performs well on the Bird dataset, its performance also shows a gradual collapse on the Spider dataset as training progresses.
H.3
ERROR BAR
H.4
MEAN @ K METRIC FOR MATHEMATICAN REASONING TASK
To verify that our reinforcement learning training was effective on the math tasks, we evaluated the performance using the mean @k metric Table
httpps:/bbi wwwbbrch.hhh...m .i thub. io/
21

---

Published as a conference paper at ICLR 2026
NETTRANC OF EoCeREmeNE
Figure 6: Error bar in Llama Sql. For each method in our Llama SQL experiments, we conducted three separate reinforcement learning training runs. We then selected the model that achieved the highest pass @ 16 score on the Bird.
Table 9: The Mean @k metric for models trained math datasets. To accelerate training, filtered the DAPO-17k dataset for data with a Pass @8 score of O. In the end, we trained Llama on 5.8k samples and Qwen 10.2k samples.
Mean 67 64
Mean 16
Mean
Model
Avg
ATME24
ATME25
AMC23
Math500
Olympiad
Minerva
College Math
Llama- A1-8B-Insttrul
Base Model
5.5
0.5
23.0
46.5
15.2
22.1
27.3
20.01
RT. Models
-GRPO
47.7
29.1
22.3
IPILFF
6.2
0.5
23.8
48.1
15.5
25.4
29.7
21.31
-DPH-JS
24.2
48.4
27.4
29.9
22.06
Qwen2.5-Math-7B
Base Model
12.7
32.2
48.9
22.8
13.3
19.3
22.26
Models
GRPO
25.8
10.9
60.6
75.5
37.8
38.8
40.6
41.43
-DAPO
25.9
11.1
61.3
76.3
37.6
40.7
41.70
-RKL
25.8
10.3
61.5
75.3
36.7
34.6
39.9
40.59
-DPH-F
-DPH-JS
26.6
11.2
62.7
76.5
38.5
38.9
41.9
42.33
T CASE STUDY
We have listed the cases and prompts for the tasks in detail facilitate future reproduction.
22

---

Published as a conference paper at ICLR 2026
1.1
SQL TASK
Case Study-SQL
System Prompt
You Qwen, created by Alibaba Cloud. You helpful assistant.
User Prompt
Task Overview:
You data science expert. Below, you provided with database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query answer the question.
Database Engine:
SQLite
Database Schema:
CREATE TABLE frpm (
CDSCode text, example: ['01100170109835', '011001011220071
'Academic Year' text, example: ['2014-2015']
"County Code' text, example: ['01', '02"]
'Free Meal Count (K-12)' real, - exxamle: [565.0, 186.0]
'eercent (%) Eligible Free (K-12)* real, example:
[0.5197792088316447,
0.470866059993367]
'2013-14 CALPADS Fall Certification Status' integer, example: [1]
PRIMARY KEY (CDSCode),
CONSTRAINT fk frpp_edscodde (CDSCode)
edscode FOREIGN KEY (CDSCode) REFERENCES schools
This schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints.
CREATE TABLE
Question:
Eligible free rate for K-12 'Free Meal Count (K-12)' 'Enrollment (K-12)'. What the highest eligible free rate for K-12 students in the schools in Alameda County?
Instructions:
Make sure you only output the information that is asked in the question. If the question asks for specific column, make sure to only include that column the SELECT clause, nothing more.
The generated query should return all of the information asked in the question without any missing extra information.
Before generating the final SQL query, please think through the steps of how to write the query.
Output Format: In your answer, please enclose the generated SQL query in code block:
'''gll
Your SQL query
...
Take deep breath and think step by step find the correct SQL query.
Response
To find the codes of the schools with a total enrollment of over 500, we need follow these steps:
1. Identify the table that contains the enrollment data: The 'frpm' table contains the 'Enroll- ment (K-12)* and 'Enrollment (Ages 5-17)' columns, which represent the enrollment data.
2. Identify the relevant columns: We need to sum the 'Enrollment (K-12)' and 'Enrollment (Ages 5-17)* columns get the total enrollment. 3. Identify the filter condition: We need to filter the schools with a total enrollment of over 500. 4. Write the SQL query: We can use 'SELECT' statement retrieve the 'CDSCode' column from the 'frpm' table, filtering the
23

---

Published as a conference paper at ICLR 2026
schools where the sum of 'Enrollment (K-12)' and 'Enrollment (Ages 5-17)' is greater than 500.
Based on these steps, the SQL query find the codes of the schools with total enrollment of over 500 is:
'ssql
SELECT CremocDsCode
FROM frpm
WHERE frpm. 'Enroliment (K-12) frpm.
'Enrollment (Ages 5-17) > 500;
This query the EErollment (K-12)' and 'Enrollment (Ages 5-17)' columns and filters the schools with a total enrollment of over 500, returning the "CDSCode' column.
Execution results
[[' [1'01101170109351
('01100170112607)) ), ['01100170121122']]
1.2
MATH TASK
Case Study-Math
System Prompt
Please reason step by step, and put your final answer within \ \booee(())
User Prompt
Every morning Aya goes for 9-kilometer-long walk and stops at coffee shop afterwards. When she walks at a constant speed kilometers per hour, the walk takes her hours, including minutes spent in the coffee shop. When she walks + 2 kilometers per hour, the walk takes her hours and 24 minutes, including t minutes spent in the coffce shop. Suppose Aya walks s + kilometers per hour. Find the number of minutes the walk takes her, including the minutes spent in the coffee shop.
Response
To solve this problem, we need to first determine the value of and t. We know the follow-
ing:
1. When Aya walks at km/h, the total time including the coffee shop visit is 4 hours. 2. When Aya walks at s + km/h, the total time including the coffee shop visit is 2 hours and 24 minutes (which is 2.4 hours).
First, we can set up the equations for the walking times:
1. For the first scenario:
= 4
60
2. For the second scenario:
9
1
+
2.4
+ 2
60
We can solve these equations to find & and f.
Let's subtract the second equation from the first to eliminate
1.6
8 s 2
This simplifies to:
9(s 2) 9s 1.6
s(s + 2)
18
1.6
s(s + 2)
18 1.6s(s + 2)
24

---

Published as a conference paper at ICLR 2026
18 = 1.6s² + 3.2s
1.6s² 3.2s 18 =8
We can solve this quadratic equation using the quadratic formula -bt. -bt where 1.6, b 3.2, and c 18:
8 -322 3 4. 1.6 (-18)
1.6
-3.2 v10.24 + 115.2
8
3.2
-3.2 v125.44
3.2
3.2 = 11.2
3.2
This gives us two possible solutions:
= 2.5 (positive root)
3.2
14.4
4.5
(negative root, not physically meaningful)
3.2
So, = 2.5 km/h. Now we can substitute back into one of the original equations to find t:
t
4
2.5
60
3.6 +
4
60
t 0.4
60
t = 24
So, 24 minutes. Now we need to find out how long the walk takes when Aya walks at + 1 2.5 + 0.5 km/h. The total time including the coffee shop visit will be:
9
24 + 0.4 3.44 hours
Converting 3.4 hours minutes, we get:
3.4 x 60 204 minutes
So the final answer is 204
25

---

Published as a conference paper at ICLR 2026
J PROOF OF THEOREM 1
The proof of Theorem I relies on bounding the total variation distance by various f-divergences. The following lemma, which direct consequence of the well-known Pinsker's inequality and its generalizations, establishes these relationships.
Lemma 1. For any two probability distributions P and Q defined on the same measurable space, the squared total variation distance, Dty (P, (),Q), is bounded by the Kullback-Leiin (KL), ax, and Jensen-Shannon (JS) divergences as follows:
S
where Dy is one of the specified divergences and C is positive constant that depends on the choice of f-
Proof. The proof consists of recalling the standard Pinsker-type inequalities (Fedotov et al. 2003 for each specified f-divergence and arranging them into the desired form. We define the total varia- tion distance as Drv Q) = Σ P(x)) Qx))
Kullback-Leibler (KL) Divergence: The classical Pinsker's inequality states that (kI(()))) 2 2D Q),
Rearranging this directly yields the desired bound:
Div (P.Q) DKL DRL(PI)Q)
Here, C = 1/2.
a- a-Divergence: The relationship depends on the value of a.
For 1, the a-divergence is lower-bounded by the KL divergence. Consequently, the same constant applies:
(P(()) 2 Dx1. 2 vP, = D2v((.Q) -
- For (0, 1), a similar quadratic bound holds, of the form (P(l)) > C. D2 (P, Q) for some constant C. For instance, the Hellinger distance, which corresponds to a 1/2 (up to a factor), satisfies (P, Q) 2 (P,
In all cases, suitable constant C exists.
Jensen-Shannon (JS) Divergence: For the IS divergence, there are two well-known lower bounds this form.
A commonly used quadratic inequality is:
Dss(PI[Q) V(P.Q)
The tightest possible bound overall is non-quadratic:
1 V(P.Q)
Dis (Pl()) > In(2) H₂
where Hz(p) = -p]n(p) (1 - p) In(1 - p). This provides the sharpest possible function for (VV).
Thus, for each of the considered f-divergences, we have established the existence of constant C that satisfies the inequality. This concludes the proof.
Throughout the following proof, we will repeatedly apply the lemma established in Schulman et al. 2015).
26

---

Published as a conference paper at ICLR 2026
Lemma 2 (Schulman et al. (2015).. Given two policies and n, have
J(a) J(m) E
Σ
of Ax (st: ar)]
1=
where the expectation is over the trajectories T == (so, ao, 81, a1,... and the notion E indicates that
T are sampled from f to generate
Given that T, T are a-coupled policies, define (s) = Ein An (s, we have following result.
Lemma (Lemma Kang al. (018)). ler (T,, #) be an a coupled policy pair. Then
E A R (st)
S 2a(1 - (1 a)") max [A, (s.a))
Now are ready to prove our theorem
Proof. Let's assume that the policy pairs (π, ft) and (Tet i) are a-coupled and 8-coupled, respec- tively. This can be established if their total variation distances are bounded, i.e., Drv(T, ( ft I a and DTV (Tpef, #) S 3. Our goal is to find lower bound for the performance improvement (( - L (T)) We proceed as follows:
J(x) T. (+t)
E Br
r AAAl
1=0
(Bain lr² A (st)) - Eecrappe [7' A (T ) + Rai h ATurllT Bann (7* 1 (8:)))
(n( (())) - (n( - m(T)) + BEerTT b" A A ([s)) Eg /7 (s:))
(T)
7 pm Ex h² Toe a (ss))] Ex (7' A A
(Ba, A + Es M h A (st)] Exewn / (**((()))))
1=0
2
Σ
(- 27'38 3(1 - (1- -3))
27 max A A (s,aa) + - - (1 - a))² (s.a)
max AA As(s,a)) a)) (s.a)
1=0
2
2a 2a2
5
+
(1 - 71 - 7(1 (3)
(11-)-1(11100 1-7
(7 cp + a²ex)
2
21 + 1-7
(18)
where equality b, holds from Lemma inequality d applies Lemma
3
The final result is obtained by using the fact that (1 - 7 x)) > (1 () for x (0, 1), which simplifies the denominator.
The theorem's 's full statement is established by using Lemma I to connect the KL-divergence bounds to the total variation distances (Drv), which in turn provides the conditions for the policy pairs to be a-coupled and S-coupled.
27


Reviews:
[{"review_id": "g01OiWWWCK", "reviewer": "Reviewer_jdRw", "summary": "This paper investigates the diversity collapse problem in RLVR for LLM post-training. The authors identify that the commonly used reverse-KL divergence is a key contributor to this problem, as its mode-seeking nature forces policies to converge on single high-probability solutions, leading to degraded Pass@k performance and catastrophic forgetting. To address this, the paper proposes DPH-RL, which leverages mass-covering f-divergences as a \"rehearsal mechanism\" to maintain broad solution coverage. Experiments on SQL generation and mathematical reasoning tasks with Llama and Qwen models demonstrate that DPH-RL improves both Pass@1 and Pass@k metrics while mitigating catastrophic forgetting on out-of-domain tasks.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper targets an important limitation of current RLVR methods: the degradation of Pass@K performance despite improvements in Pass@1.\n\n- The paper provides a comprehensive analysis of diversity collapse from the perspective of the KL divergence term, with both theoretical and empirical evidence.\n\n- The method is orthogonal to existing approaches, allowing for potential combinations and broader applicability.", "weaknesses": "- The improvements are modest, and no single DPH variant consistently outperforms the baselines, which limits practical usage, i.e., different variants achieve best results on different datasets, making it unclear which configuration to use in practice.\n\n- Although the paper's motivation is to address Pass@K degradation in RLVR, the proposed method does not convincingly alleviate this problem. Besides, the evaluation is limited to Pass@64, and larger K values (e.g., Pass@256, Pass@1024 done in [1]) are not reported. Such evaluations would better demonstrate whether the method truly preserves diversity at scale.\n\n- The data partitioning setting lacks justification and ablation studies. The correctness threshold differs across models (6/8 for Llama, 7/8 for Qwen) without a clear explanation or ablation studies. It is unclear how robust the method is to different threshold choices.\n\n[1] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L58BuruGto", "reviewer": "Reviewer_3S5Q", "summary": "This paper proposes a divergence-based policy optimization framework that replaces the conventional reverse KL term with forward KL or Jensen–Shannon (JS) divergence. The authors argue that forward KL can mitigate mode-seeking behavior and stabilize updates. Experiments on two representative models demonstrate improved training stability and comparable or better performance.", "questions": "* How sensitive are the results to the choice of temperature or scaling in the KL term?\n* Did you tune the KL coefficients separately for reverse and forward cases to ensure fairness?\n* Can the method be applied to off-policy or batch RLHF settings, where distribution mismatch is more severe?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The proposed method is conceptually simple and easy to implement.\n* Theoretical motivation is clear: reinterpreting divergence choice (reverse vs. forward KL) in policy optimization.\n* Experimental results on two models provide empirical support for the claimed benefits.", "weaknesses": "* The main idea—switching from reverse KL to forward KL or JS divergence—has been extensively discussed in prior work [1]. The paper should (a) more directly situate DPH-RL relative to that prior theory, (b) clarify what is new beyond applying these known divergence properties to RLVR for LLMs, and (c) compare empirically to algorithms motivated by those analyses.\n\n[1] Chan A, Silva H, Lim S, et al. Greedification operators for policy optimization: Investigating forward and reverse kl divergences[J]. Journal of Machine Learning Research, 2022, 23(253): 1-79.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6d1E6F5yN", "reviewer": "Reviewer_8NDo", "summary": "This paper re-exames the KL penalty term in RLVR with LLMs. Most approaches adopt the reverse-KL, which may degrade LLMs’ Pass@K performance. To mitigate this problem, this paper proposes to use forward-KL and consider a more general divergence family, f-divergences.", "questions": "See the above weakness.\n\nAdditionally:\n1. What does “greedy” mean in the experiment section? Pass@1 or greedy decoding?\n2. What is $\\pi_{\\text{per}}$ in section 4.3?\n3. What does the alpha in Table 2 mean? alpha-divergence?\n4. If our goal is to improve pass@K performance and we know that the base model has the highest pass@K performance, then what is the point of RL post-training?\n5. Do authors have any idea why RL training hurts pass@K performance?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies an important and interesting problem in RLVR that how the KL divergence affects RL training.", "weaknesses": "1. The motivation for forward-KL is not strong enough.\n    \n    The failure of reverse-KL does not point to forward-KL or JS divergence, especially when authors did not analyze the failure reason for reverse-KL.\n    \n2. The lack of discussion with replay buffer.\n    \n    The implementation of forward-KL requires collecting trajectories from the base model, which can be consider as a special rehearsal mechanism. Replay buffers and their application are widely studied in both of continual learning and RL. Some discussions of replay buffer may clarify this paper’s position.\n    \n3. The necessity of splitting dataset into “near-perfect” or “exploration”.\n    \n    > For queries where π_ref performs well, aggressive reward maximization is unnecessary and risks degrading performance.\n\n    If this claim is true, then any RL post-training should not include “easy” questions. However, LLMs tend to forget prior learned knowledge. Adding easy questions can help to remind LLMs, aligning author’s intention of rehearsal-stylish forward-KL.\n    \n4. The lack of in-depth analysis of forward-KL.\n    \n    Though forward-KL seems to achieve empirical success, its mechanism is still under unexplored. Although authors provide the Monotonic Improvement of the method, it is hard to connect it with the improvement. Moreover, the discussion in section 6 only shows a ratio between training, which does not contain too much information.\n\n5. Authors should set a pointer to explicitly link the definition of alpha-divergence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3upEP5DN3e", "reviewer": "Reviewer_NVcc", "summary": "This work studies the effect of divergence function choice on catastrophic forgetting, **pass@k** degradation, and diversity collapse in RL with verifiable rewards. The authors study the effects of reverse-KL on knowledge retention (described as performance preservation on tasks already solved before RLVR). Doing away with the mode-seeking reverse-KL, the authors propose a diversity-preserving framework that maintains task coverage and leads to both **pass@1** and **pass@k** improvements. The method has two stages: (1) a pre-sampling stage where the dataset is separated according to how solvable the problems are for the base model, (2) online training with two distinct loss functions—one without KL on the exploratory partition and another using mass-covering divergence functions on the mastered partition from (1). This is validated across Llama and Qwen models, showing great promise in mitigating the specialized performance vs. sampling diversity tradeoff.", "questions": "1. What do the authors mean by in-domain and out-of-domain benchmarks at the end of the introduction? \n\n2. Why is assumption 1 well-founded? \n\n3. What happens if the partitions are not well-balanced? How possible is it and what are the contingencies? \n\n4. In section 4.2, why is JS more stable? Is it due to it being more symmetric? Please clarify in either case.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "-  **Line 86 to 88** : very good distinction of RL vs sequential generation with NNs. Important to disambiguate the problems between the two fields\n\n- Clear introduction and preliminaries, easy to follow and well-motivated\n\n- Addresses a core issue in the field\n\n- No need to run inference on the reference model in the online training loop", "weaknesses": "- Relies on a good base model and lacks analysis of how balanced the partitions in the pre-sampling phase stage should be \n\n- Figures axis are sometimes tough to read. Please use a standard plotting sofware with clear labels \n\n- Unclear use of the term \"stability\"\n\n- Trains only on two Qwen models which is insufficient given the claims of the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xNBmPoLOYo", "reviewer": "Reviewer_HqbP", "summary": "This paper addresses a critical paradox observed when fine-tuning large language models (LLMs) with Reinforcement Learning under Verifiable Rewards (RLVR): models often improve on single-attempt success (Pass@1) but see degraded multi-attempt performance (Pass@k) and suffer catastrophic forgetting of diverse skills. The authors identify that a key culprit is the choice of divergence in the RL objective. Prior approaches almost universally use a mode-seeking reverse KL divergence or even omit the divergence term, which provides no mechanism to retain broad knowledge. As a result, policies narrow toward a few high-probability solutions, sacrificing diversity.\nTo remedy this, the paper proposes Diversity-Preserving Hybrid RL (DPH-RL), a novel framework that uses mass-covering f-divergences (specifically forward-KL and Jensen–Shannon (JS) divergence) as a “rehearsal mechanism” to continually reference the initial policy. In effect, the divergence term itself becomes the solution: by penalizing divergence from the base model’s distribution, the agent is forced to maintain broad solution coverage and not forget previously learned behaviors. The approach is hybrid in that it selectively applies this divergence-based regularization, allowing unrestricted exploration on parts of the task where the base model wasn’t already proficient. Experiments on math problem solving and SQL query generation tasks demonstrate that DPH-RL not only improves in-domain Pass@1 accuracy, but crucially also boosts Pass@k (multiple attempts) performance above the base model’s level, while effectively preventing catastrophic forgetting on out-of-domain tasks. The method is also efficient: it computes the f-divergence via a generator function by sampling from the fixed initial policy, avoiding the need for an expensive separate reference policy network during training. Overall, this work highlights an overlooked axis in RL fine-tuning – the choice of divergence measure – as a powerful tool for building more general and diverse LLM-based problem solvers.", "questions": "Adaptive vs. Fixed Data Partition: In your method, you partition the training data into D<sub>pef</sub> (where the base policy is nearly perfect) and D<sub>exp</sub> (exploration needed) using a fixed success-rate threshold. How sensitive are the results to this threshold choice, and did you consider a more dynamic criterion? For example, could the algorithm online-adjust the weight of the divergence term based on the model’s current performance on a state or task (rather than a hard split)? Such an adaptive approach might remove the need for manual threshold tuning; we’re curious if you experimented with or have thoughts on this extension.\nChoice of Divergence Measure: You demonstrated significant gains with forward-KL and JS divergence. Did you try other f-divergences (e.g., intermediate alpha-divergences or Csiszár divergences) in practice, and if so, how did they compare? The theory suggests a family of options – is forward-KL essentially the strongest “mass-covering” extreme, or might a milder divergence also strike a good balance? Clarification on why forward-KL and JS were chosen (and whether one consistently outperforms the other in different conditions) would be helpful for practitioners looking to apply your framework.\nBehavior with Minimal Base Knowledge: In scenarios where the base model has very low capability (e.g., near 0% success on D<sub>exp</sub> tasks), does DPH-RL effectively revert to a standard RL algorithm? Your analysis indicates that if the reference policy has no advantage (delta=0), the improvement bound reduces to the normal TRPO guarantee. Empirically, have you observed any downside to applying DPH-RL in such cases (perhaps the forward-KL term just stays inactive)? It would be reassuring to know that one at least loses nothing by using DPH-RL when the base model provides no guidance – confirming the method is robust in worst-case scenarios.\nGenerality to Other Settings: Can you comment on applying DPH-RL beyond the RLVR paradigm? For instance, in a standard RLHF setup (with a learned reward model and possibly no programmatic correctness signal), do you foresee any challenges in using a forward-KL/JS regularizer to preserve the base model distribution? Intuitively it should still help prevent policy collapse, but are there issues if the reward signal is less directly tied to correctness? Any insight or preliminary experiments in non-code, non-math tasks (e.g., dialogue) would be interesting to hear about.\nBalancing Diversity and Quality: One might wonder if there are cases where preserving diversity could conflict with optimizing reward (e.g., if some of the base model’s diverse outputs are actually incorrect or suboptimal). Did you observe any instances where the forward-KL term kept the model generating suboptimal solutions for the sake of diversity? It seems your method focuses the preservation on high-performing reference behaviors (since D<sub>pef</sub> is constructed from successful attempts), which mitigates this. Still, any anecdotal observations on how DPH-RL balances keeping “good” diverse solutions versus discarding genuinely bad ones would provide intuition on the method’s nuances.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Important Problem & Insight: The paper tackles the important issue of diversity collapse in RL-based fine-tuning of LLMs, identifying an overlooked cause – the choice of KL divergence – and proposing it as a solution. This fresh insight (using forward-KL/JS to preserve knowledge) is highly original and addresses a real need.\nTheoretical Rigor: The work is grounded in rigorous theory. The Enhanced Monotonic Improvement Theorem 1 generalizes TRPO’s guarantee to f-divergence regularization, showing a strictly better improvement bound with an added positive term from the reference policy. The assumptions and proof techniques are sound and build on well-established analysis, lending strong credibility to the approach.\nStrong Empirical Results: The experiments are thorough and demonstrate clear benefits. DPH-RL consistently outperforms baseline methods (standard RL fine-tuning with reverse-KL, and variants without KL) by achieving higher Pass@k without sacrificing Pass@1. Notably, on an out-of-domain test (e.g. Spider SQL dataset), DPH-RL models maintain accuracy much closer to the base model than others, indicating greatly reduced forgetting. These results validate that the method fulfills its intended purpose of retaining diversity while improving performance.\nBroad Applicability: The approach is relatively general. It’s compatible with different divergence choices (the authors demonstrate forward-KL and JS, and discuss extensions to a family of f-divergences) and can be plugged into existing RL algorithms (they test with GRPO, DAPO, etc.). This generality means the contribution could impact various subfields, from RLHF for LLMs to other generative modeling scenarios where mode-covering behavior is desired.\nClarity and Execution: The paper is well-written with clear motivation and analysis. Figures and tables strongly back the claims (for example, qualitative differences in solution diversity under different divergences are illustrated in figures). The method is described in sufficient detail, and the authors even address practical considerations (like how to compute the forward/JS divergence term efficiently by sampling the base model, which avoids needing an extra model and thus is training-efficient). This makes the work not only theoretically strong but also feasible and useful in practice.", "weaknesses": "While the paper presents a compelling and well-substantiated approach, a few limitations are worth noting:\n\nReliance on Base Policy Quality: The effectiveness of DPH-RL depends on the strength of the base model. If the base policy performs poorly on the target domain, there is little knowledge to rehearse, reducing the benefit of the divergence term.\n\nManual Partitioning: The hybrid framework splits training data into exploration and rehearsal sets using fixed thresholds on success rates. This manual setup introduces sensitivity to hyperparameters and may not adapt well across tasks without tuning.\n\nLimited Divergence Exploration: Although the authors motivate the use of forward-KL and JS divergence, they do not evaluate a broader set of f-divergences or analyze tradeoffs across this family. It's unclear if the performance gains are tied to these specific choices or generalize more broadly.\n\nScope of Evaluation: The experiments are confined to two domains (math and SQL) and modest-sized models. Broader tasks and larger-scale settings would strengthen claims of generality.\n\nPreservation of Undesirable Behavior: The method encourages retention of base model behavior, which may include suboptimal or biased responses. While not observed in current tasks, this risk should be acknowledged in future applications.\n\nThese issues do not undermine the core contributions but indicate valuable directions for further investigation and refinement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)