General Exploratory Bonus for Optimistic Exploration in RLHF
Authors: Wendi Li
Changdae Oh
Sharon Li
Venue: ICLR 2026 Poster (2026)
Keywords: RLHF
optimistic exploration
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Optimistic exploration is central to improving sample efficiency in reinforcement learning with human feedback, yet existing exploratory bonus methods often fail to realize true optimism. We provide a theoretical analysis showing that current formulations, under KL or $\alpha$-divergence regularization, unintentionally bias exploration toward high-probability regions of the reference model, thereby reinforcing conservative behavior instead of promoting discovery of uncertain regions. To address this pitfall, we introduce the General Exploratory Bonus (GEB), a novel theoretical framework that provably satisfies the optimism principle. GEB counteracts divergence-induced bias via reference-dependent reward regulation and unifies prior heuristic bonuses as special cases, while extending naturally across the full $\alpha$-divergence family. Empirically, GEB consistently outperforms baselines on alignment tasks across multiple divergence settings and large language model backbones. These results demonstrate that GEB offers both a principled and practical solution for optimistic exploration in RLHF.

Published as a conference paper at ICLR 2026
GENERAL EXPLORATORY BONUS FOR OPTIMISTIC EXPLORATION IN RLHF
Wendi Li, Changdae Oh, Sharon Li
Department of Computer Sciences
University of Wisconsin-Mudiion
{wli679, changdae, sharonli 1cs. www..
ABSTRACT
Optimistic exploration is central to improving sample efficiency in reinforcement learning with human feedhack, yet existing exploratory bonus methods incen- tivize exploration often fail to realize optimism. We provide theoretical analy- sis showing that current formulations, under KL or a-divergence regularization, unintentionally bias exploration toward high-probability regions of the reference model, thereby reinforcing conservative behavior instead of promoting discov- ery of uncertain regions. To address this pitfall, introduce the General Ex- ploratory Bonus (GEB), a novel theoretical framework that provably satisfies the optimism principle. GEB counteracts divergence-induced bias via reference- dependent reward regulation and unifies prior heuristic bonuses as special cases, while extending naturally across the full a-divergence family. Empirically, GEB consistently outperforms baselines on alignment tasks across multiple divergence settings and large language model backbones. These results demonstrate that GEB offers both principled and practical solution for optimistic exploration in RLHE. Code available here.
INTRODUCTION
Despite the acknowledged significance online exploration for reinforcement learning with human feedback (RLHF) (Xu et al., 2024; Tang et al., 2024; Tajwar et al., 2024), there remains a paucity of theoretical frameworks governing how explore. algorithms (Guo et al., 2024; Pang et al., 2024; Ahmadian et al., 2024) generally rely passive exploration, Le., the stochasticity of the policy itself to generate responses, with no mechanism to incentivize novelty When the optimal behavior resides in low- www-pbbbability regions, passive exploration is unlikely to discover it, leading to policies that remain trapped around local optima.
shown in Fig. (1, top), standard online RI RLHFF
diversity. As result, this approach can be notoriously sample-inefficient.
To address this, some works (Das et al., 2024; Ji et al., 2024; Mehta et al., 2023; Qi et al., 2025; Liu al., 2024) have attempted devise sample-efficient algorithms, inspired by the principle optimism in the face of uncertainty. As illustrated Fig. (2, top), the principle aims to generate responses for regions of high epistemic uncertainty, thus encouraging data collection unexplored areas for further training. To operationalize this, recent attempts (Zhang et al., 2024a; Xie et al., 2024; Cen al., 2025) encourage exploration by adding exploratory bonuses reward modeling, which is practically optimizeable for large language models. These methods intend to artificially inflate re- wards in underexplored regions, nudging the policy toward more informative data collection.
Unfortunately, our theoretical analysis in Section 3 reveals fundamental pitfall: under the common Kl-regularized RLHF, the existing theoretical framework of exploratory bonuses fails to satisfy op- timism. In particular, we prove that existing bonus formulations can undesirably drive the policy æ toward the reference policy मंत due to the divergence regulation the exploratory bonus, and the induced bonus actually biases exploration toward high-probability regions of the reference model. As illustrated in Fig. (II, bottom), the bonus disproportionately amplifies rewards for regions al- ready well-covered by Tett thereby reinforcing conservative behavior rather than driving exploration into uncertain regions. This failure is not confined to KL-divergence; we further extend our analysis the general a-divergence family and prove that the same collapse persists across wide

---

Published as a conference paper at ICLR 2026
preferred, low Tee
dispreforred, low Jef
proferzed, high Te/
diepreferred, high Xnd
policy distribution over trajectory T
(I) passive exploration
optimisiic exploration
T
encourage exploration uncertain aresn
Exploration bonus effect for optimistic reward modeling
more prominent incrrase on certain uncertaig
area
vanilla RM
(II) prior framework
(III) GEB
The upper part compares passive exploration and optimistic exploration. Optimistic exploration stimulates the trajectories 7 of small Tee (seldom visited/uncertain), while passive ex- ploration sticks to the high-TTer region, failing to approach global optima. The dashed line separates regions of high vs. low likelihood under the learning policy me. The lower part contrasts effect of the exploration bonus term in optimistic reward modeling between prior works and our GEB. Prior works often emphasize rewards in frequently visited regions, which constrains exploration within certain areas. In contrast, GEB amplifies rewards in seldom-visio regions, thereby couraging further sampling uncertain areas and successfully achieving optimistic exploration.
Figure 1:
range of divergence-regullrizee objectives. Thus, while existing approaches appear encourage exploration, they in fact undermine the very principle of optimism they aim to realize.
Motivated by these failures, we propose a new framework, General Exploratory Bonus (GEB), which theoretically unifies existing approaches while provably satisfying optimism (Section 4). GEB corrects the failure modes of prior approaches by directly introducing reference-dependent regulation into the reward. This adjustment offsets the undesired conservatism induced by diver- gence regularization, allowing the exploratory bonus to satisfy optimism increases the proba- bility of responses rarely sampled to pursue potentially more preferred answers, as shown in Fig. (III, bottom). Importantly, GEB provides a unified formulation: prior heuristic exploratory bonuses can be reinterpreted as special cases, and the framework naturally extends to the full class of a-divergences. Beyond correcting the theoretical shortcomings, GEB remains practically imple- mentable- it can be seamlessly integrated into the standard iterative RLHF loop without additional sampling cost.
We validate GEB large-scale alignment task across different divergences and model back- bones. Empirically, GEB consistently yields stronger alignment compared to its counterpart of passive exploration. For example, the three GEB variants that we consider generally outperform the iterative f-DPO (Xiong et al., 2024a) across different divergence regulations, while the most perfor- mant variant surpasses several existing optimistic exploration methods that incorporate exploratory bonuses (Zhang et al., 2024a; Xie et al., 2024; Cen et al., 2025). By analyzing the distribution of sampled responses, we validate that GEB can successfully encourage sampling the region small Tiet thereby effectively achieving optimistic exploration. We summarize our main contributions:
We formally prove that the existing theoretical framework of exploratory bonuses under KI. and a-divergence regularization fails achieve optimistic exploration.
2. We introduce General Exploratory Bonus (GEB), novel theoretical framework of opti- mistic exploration for RLHF that provably satisfies the optimism principle and unifies prior heuristic bonuses.
3. We empirically validate GEB on LLM aligmment tasks, showing improved performance and broad applicability across multiple divergence families.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES
Iterative online RLHE Let z be a prompt sampled from a distribution p and y be a response given which is sampled from policy (-|1x) modeled by language model. We denote by r(,, a real-valued reward model. An iterative online RLHF proceeds for rounds T. where each round t= 1, .... T has the following three steps: (i) The reward model r, is trained on the human preference dataset D, = {(x, y², y')}, where y² , denote the preferred and dispreferred response to æ; (ii) The policy H, is updated to maximize the reward (1 y) for responses y Te(-(ar) conditioned on prompt z; and (iii) using the updated policy, we sample ă ~ and generate multiple response pairs (U1, (2) Te(-(²). Human evaluators then annotate these pairs to produce preference-labeled data {(T, y", 3 }- The dataset for the next round is formed by D.+1 = Dr U {(,, 17)). For reward modeling step (i), we typically adopt the Bradley-Terry objective (Bradley & Terry, 1952):
Tt = arg min Ler(Dr, = arg min ((a."""")>> (rg]a y² r(x,,)))
(1)
where a denotes the sigmoid function. Next, in each step (ii), given the learned reward function fr. the policy सe is updated to maximize the expected reward, often with a KL-regularization as follows
T3 = arg max J3.KL(T, r2) = arg max (o) (e y) KL (
(2)
"
"
where 3 > 0 is hyperparameter and ref is the reference model. The effectiveness of iterative online RLHF (Xiong et al., 2024b; Dong et al., 2024) has been validated various real-world systems such as Claude (Bai et al., 2022) and LLaMA-series (Touvron et al., 2023; Grattafiori et al., 2024), but there is still much room for improvement in terms of sample-efficien exploration.
Sample inefficiency of iterative online RLHE In online RLHF, standard online sampling is usu- ally performed passively, relying solely on the LLM policy's s inherent randomness. However, the policy assigns small probability to the optimal action, passive exploration may never explore it. Some recent theoretical analyses (Muldrew et al., 2024; Dong et al., 2024) and empirical ev- idence (Chen et al., 2025; Cen et al., 2025) present that the passive approach fails sufficiently explore the prompt-response space. Particularly, Xie et al. (2024) demonstrates that the sample complexity can be exponential in 1/3 for passive exploration, which is unacceptable in the small-B regime. After then, follow-up studies propose to implement the principle "optimism towards uncer- tainty" into RLHF algorithms, i.e., encourage exploration of uncertain trajectories. Several works on this try to estimate uncertainty by leveraging some uncertainty quantification techniques, such as elliptical potential (Bai et al., 2022), Bayesian modeling (Qi et al., 2025), and epistemic neural net- work training (Liu et al., 2024). However, these methods are generally computationally prohibitive in LLM-scale settings. Therefore, recent works (Xie et al., 2024; Cen et al., 2025; Zhang et al., 2024a) propose exploratory bonuses for optimistic exploration, which can be computationally more tractable for L.L.-based optimization.
3 EXPLORATORY BONUS AND HOW IT CAN FAIL
In this section, we first provide the iterative online RLHF formulation with an exploratory bonus (Section 3.1). We then theoretically prove that the existing formulation can fail to achieve opti- mistic exploration under both KL-constrained RLHF (Section 3.2) and more general a-divergence- regularized RLHF (Section 3.3), motivating our proposed method in Section 4.
3.1 EXPLORATORY BONUS
To improve the sample efficiency of iterative online RLHE, recent works (Zhang et al., 2024a; Cen et al., 2025) introduce exploratory bonuses, which aim to encourage the policy model to explore the under-visited space given an optimistic reward estimation. These approaches modify the standard RLHF loop by adding an exploratory bonus term Loonus in the reward modeling phase. Specifically, in the t-th iteration, the reward model ra and policy Ti are optimized by
It arg min (CBr(D(,)) rLooomus (r)).
(3)
TT. arg J
arg maggmaee E Ex-ppy (1)) (2, 8) 3DDu (m])
(4)
where k. >> 0 is a hyperparameter. By Eq. 3, the reward model r, should not only fit the observed data in D,, but also learn maximize the bonus term Lionus (r).
3

---

Published as a conference paper at ICLR 2026
To boost exploration, the bonus term is designed amplify the probability mass of policy more in underexplored areas rather than incentivizing it solely towards high empirical reward areas. As mentioned in $ 2, early works RLHF optimistic exploration are computationally prohibitive in the L.M fine-tuning regime. Thus, it is necessary to set a new approach that not only aligns with the principle of optimism in the face of uncertainty but is also cost-effective. For this, derive new condition for the exploration bonus to achieve optimism, avoiding direct uncertainty quantification:
Definition 3.1 (Optimism condition for exploration bonus) Given an input prompt a and a re- sponse y. when a reward model and policy π are computed with Eq. 3 and Eq. 4, respectively, the exploratory bonus Lionus achieves optimism, if
e (r v))
<0,
(5)
(y)aa)
iT (T ())
where n, is a typical sampling policy, a joint policy on all iterations up to the current iteration.
Specifically, at the t-th iteration, the typical sampling policy т. = Ii OT o..... oo,; Ty is a joint distribu- tion of all previous policies up to the current iteration. This distribution is not directly computable; rather, it serves as a theoretical construct describing how responses in Dr are generated. In Eq. 5, rather than characterizing Lhons in its original function space, we define it by a condition on its partial derivatives with respect to two policies: current policy and typical policy. This new optimism condition not only enables us to flexibly define Looaus but also serves as a core tool for theoretically analyzing existing methods. Although Loonus appears unrelated to the current policy π it is de- fined in terms of the reward model r(æ; y) the policy-reparameteriie reward T (z. y) allows us to express r(x, y) directly in terms of follows (Rafailov et al., 2024):
(v)œ)) + B log ((x), where Z(x) = Bynnve exp(r(x, v)3 (er (r)
r(x, v) = rr (z, y) = 3 log
This is derived from the closed-form solution of the proximal preference optimization (Eq. 2) as x (y/lr) ex)(r(z.))/3) Thanks to this alternative representation of the reward, we can express and ((x)
interpret the bonus term with respect to our current policy т, which yields the following implication.
Implication. Our new optimism condition requires the derivative of the bonus term with respect to the current policy, HCoenus (Fx y)) (y/=r), to be negatively correlated with the typical policy n. In other words, as a response y is likely rare sample under n, (i.e., un- certain or underexplored responses), it should receive a larger ascending force in the policy distribution ग, i.e., higher aL (rx (x. y))/0r((()) (v(rr) This new definition of the opti- mism principle, which is specified through the lens of partial derivative alignment, ensures the exploratory bonus nudges the exploration towards an uncertain response region without explicit uncertainty quantification. In practice, π. can be substituted by the reference model policy ret or intermediate checkpoints of across iterations. We adopt the commonly used Tref as n. in our following demonstration.
3.2 FAILURE UNDER KL-CONSTRAINED RLHF
Previous works, including Zhang et al. (2024a) and Cen et al. (2025), formulate the ex- ploratory bonus with Lronus (r) = max, I (T, r). Under this formulation, optimizing the ex- ploratory bonus in Eq. 3 yields min-max bilevel objective: min, -& [Ea= (.r (x, y) (T [e]]]. Intuitively, this objective encourages not only to fit the observed data via LBT but also to assign high reward to unobserved regions by maximizing max, Bexp.yorrrc, y) in Lioonss (r). Here, theoretically show that such formulations can suffer from optimism failures under Kl-rregularized RLHF.
Lemma 3.1 (Optimism failure under KL-divergence.) Let rı arg arg min, (D. be reward model trained with the vanilla BT loss, and let T2 arg min, [L (D, r) max, (a r)] be reward model trained with an additional exploratory bonus. If the policy is optimized via Eq. 4, then Yn and r2 yield the same set of policies.
See the proof in Appendix B.I. The lemma shows that incorporating the exploratory bonus Lbonmser (r) = J8,KI (T,YT) r) into the reward training objective fails to induce the policy to sample
4

---

Published as a conference paper at ICLR 2026
from (yw- regions, i.e,, unexplored responses. That is, Lrons is ineffective for optimism. We next extend the result beyond KL divergence to a more general class a-divergence families.
3.3 GENERALIZATION TO WWWVERGENCCECOONTRAAAE
RLHF
In this subsection, we theoretically show that the failure of optimism can broadly be extended Many common divergences, such as reverse KL-divergence, Hellinger distance, and for- ward KI Kl-ddivegeenc, are special cases of a-divergence.
Table 1: Realized exploratory bonus under dif- ferent divergence classes when Lbous max, (T.
the a-divergence class.
f
exploratory bonus
reverse KL forward KL
constant
E Exnppy log
Tarlaa) <(yla)
Definition 3.2 (o-divergence class) An divergence is certain type of function D(pla) f [( de dd that measures the difference between two probability distributions p and q.
Hellinger distance
E Ex-p.y--ar
where
f(x) x 0x - (1
aal-a)
and o is hyperparameter typically with 0 a 1.
Lemma 3.2 (Optimism failure under a-divergence.) Consider objective Jo.((T,T)
Busspsrrr(sl u) + -Bu-pprrnaa <(y|z) [ where f belongs to a-divergence class. If reward is trained with = arg minm (C,,((,rrr hchoonue and policy is updated by arg maxx Jp.5(T (T, f) with Lromus = max; (z, the gradient of the bonus satisfies 8 (r) 2 O, which means Loonus encourage trajectories with large Tef more strongly, in contra- diction to the optimism principle (Definition 3.
Proof For RL objective Jz.f (T,, r), the relation between the optimal policy m and the reward can be formulated as follows,
Fj [)(1²) ((x) That (r())))))))) (x -3 n (vl
Teet( Z(x)),
(6)
where 7(x)) is normalization term and (f")-1 1 is the inverse function of f' The bi-level ob- jective can be similarly transformed to single level one by canceling the inner maximization max, by Eq. 6. The single-level objective can be written тτ arg min, CBT((,
K.EEPPPTT Z(z) (f'') )r((,) 3f( (f(z)t -f'"-1(1(()))) Since the policy is computed by arg max+ (B, r), the reward can be reparameterized by the policy with Eq. 6, which fortunately cancels (zz) Then, the optimistic reward-modeling modeling objective can be reparameterized as
(ylax) f'( T(yla) ) 35( arg min Ldpo (D, π) - -,, K.S..rppyy Trer (vle)))))) (ver)))
T(y)a))
)]
(7)
TTer(y|
Since for a-divergence, f() -aa- au- the partial derivative of Eq. 7 is (==)) which induces positively correlated gradients w.r.t. and net when 0 a = 1, hence contradictory to the optimism defined in Definition 3.1.
1, and is a constant when
According Lemma 3.2, summarize several forms of exploratory bonus induced by different a-divergences in Table 1. For clarity, these expressions are presented after removing constant coef- ficients and additive biases. In every case, the resulting bonus encourages the policy to place more probability mass on responses that the reference model already samples frequently, rather than on underexplored responses. Now, we further prove that it actually drives collapse toward neef and that the failure extends beyond a-divergence to other f-divergences.
Theorem 3.3 (Optimism failure beyond a-divergence.) When f belongs to f-divergence, and the reward function is obtained by f arg min, [(r(,rr max, Jp.r(x,r) (π.r)) and the policy is updated by arg max, Ja.r(m (T, f), the bonus term -K max, Ja. ( induces the policy model π coincide with and when x ld ((x)) is monotone function.
5

---

Published as a conference paper at ICLR 2026
The monotonic increase of (z) can be satisfied by broader divergence class beyond a- divergence, including JS- divergence and Pearson x². Please see the detailed proofs in Appendix B.2.
Intuitive understanding. The optimization of the exploratory bonus in Eq. 3 is min-man max bi-level objective, min, -x [ mization max,, the divergence constraint implicitly makes close to Tre to reduce the KL divergence. Meanwhile, outer minimization min, forces r to provide high rewards in the region of high π to maximize the expected reward. Their combination implicitly makes r focus more on the region of high Tet. As responses in high Ter area are easily sampled from scratch, prior exploratory bonuses just concentrate sampling on regions that are already frequently visited, contradictory to the optimism principle, which requires encouraging ex- ploration for responses y rarely sampled by the reference model.
y) xI [eer]]- Due to inner maxi-
GENERAL EXPLORATORY BONUS WITH OPTIMISM PRINCIPLE
Motivated by the failure of the existing optimistic exploration works, we now propose a novel frame- work, General Exploratory Bonus (GEB), and prove that it achieves optimism. We further show that prior heuristic bonuses and their broader iitiants-emere as special cases of our formulation.
Formulation of a novel exploratory bonus. As shown in the previous section, existing bonus schemes fail because the divergence constraints in max, Jp./ (TT, r) force the optimal policy π to remain close to Tee thereby biasing exploration toward regions where Teef is large. Achieving optimistic exploration requires the optimal π to counteract this effect and deviate from Te.. Our key idea is therefore to incorporate an additional deg-dependent term into the reward that offsets the influence of the divergence regularization. The resulting exploratory bonus takes the form
Lbonus max J... (T, R),
(8)
where our new reward formulation, R, now depends not only on the original reward model r(x,, y) but also on (rr(. Now, the optimal policy of max, J R(x, u)) can be obtained by replac- ing r(x, y) with R(x,y) y) in Eq 6. This yields Tr" (ulæ) = Zn(z) (f) R(a.))) ), where ZR(r) is a normalization term.
Following Lemma 3.2, we substitute <(ylz) in max, J R(x, v)) with its optimal form x" (ylaz) and then apply the reward reparameterization trick for a-divergences (Wang et al., 2024), i.e. r(I, u) = f'(T* x) Te((y//)) (z), where f specifies the divergence. Given this policy- reparameterized reward, we specify the exploratory bonus term in Eq 8 as follows:
u(I, v)
f'(u(z, v)) - f( u(2 v) (x(x)
Lrorns (. (+1x)
(9)
Zn(#t)
In Eq. we introduced u(x, y) an atomic function employed to construct the actual loss, and is given by u(z,v) = (f') [Rx v)/B) in this setup. Note that, after reward reparameterization, u(x,y) y) can be expressed in terms of ( (r) and k Moreover, as the functional form of R(x, y) is not restricted to specific class, u(x,, v) can be instantiated in many ways using these two distributions, while must satisfy u(x, y) > for all x, y to ensure that the argument of f'(-) lies within its domain. See Table 2 for the example entries we are considering in this work.
Equivalence a practical objective. In our proposed exploratory bonus, the normalization term Zx(z) in Eq. 9 cannot be eliminated. Fortunately, Lemma 4.1 (proved in Appendix B.4) shows that the training objectives with and without nx( are equivalent. This equivalence allows us to convert the objective into a more concise form, facilitating both analysis and practical implementation.
Lemma 4.1 Denote two objectives as h(u(z, y)) Eerp,. uIz y) (u(2. v)) f(((x,))) u(2.)) and (((((,)))) Ex-p.yeTH Zx(p)) f'( -f f(2(())) where u(I,)) y) is function with (y)(=) and Te)(y)a). the ratio
f'(uxx, v)) + u(z y) 8) (u(x, v)) 5 (' Zz(Z))
= A(x)
(10)
ZR(z)ule x v) (u(z,
6

---

Published as a conference paper at ICLR 2026
Table 2: GEB under different divergence classes and design of Note that (1) now the bonus term can be computed without the reference probability mass Trefi (2) all of these u instantiations meet the condition u > a when
1. The presented bonuses are simplified by removing constants.
Crans
1
1/a
arctanh(1 x) +
reverse KI. forward KI. Hellinger Distance
RE- - =((/(x))
Big- k(y)))
Pgr-agnnn arctanh(1 =(ylz))
E. Re-poyee log(1 - my)(r)))
Paupatttet log at (y)ar)
Be-p.yee log angaacttan((1 ((/=2))
Bemparee v1.5 x(y x)
Ep-p.yre /alulz)
Exmparee /rretanh(1 - -x(g/2) + 0.5
is independent of y and Axx) > O, then minimizing the two objectives, min, -h(u(x,)) v)) and min, -h(u(x, v)). yields the same class of optimal policies.
Note that the a-divergence (0 < a S 1; see Def. 3.2) naturally satisfies the condition in Eq. 10 whenever y) a. As we show in the next paragraph, enforcing u(x; v) > is straightfor- ward in practice, which grants our framework substantial flexibility and extensibility, Leveraging
Lemma 4.1, we can thus rewrite our objective in Eq. 9 into concise and analytically convenient form without the normalization term:
Doouss -5 p. [(zz, v) (u(z, v)) f(((x,))) where u(x, v) is flexibly formulated by π =(yl>) and (er(y) satisfying u(x, > a.
(11)
GEB successfully achieves optimism. Building on Lemma 4.1, we now show that our proposed framework achieves the optimism condition in Definition 3.1 (See Appendix B.5 for the proof).
Theorem 4.2 Consider an a-divergence with 0 S 1, and the exploratory bonus Loonus (Eenp.ywa u(x,, f) v)) - ((u(z,))){ where u(x, v) is function dependent on (tyaæ) and Te)(y)z). For any (x. v). if Du
82 Bx + Ted Buos, + (a-1)at du au optimism condition in Definition 3.1 is satisfied; that is,
< and u(x. y) > a, the
a
8 Lama S 0. Ow0
In our formulation, u(x, y) can be flexibly defined in terms of T (r) and Ter(y)z) as long as it sat- isfies the derivative condition in Theorem 4.2 and u(x, y) > a. In particular, when u(x, y) depends solely on (z][z) and is independent of Tee((y(æ), any function that is strictly decreasing in with u(z, y) > a constitutes a valid choice. This design flexibility underscores the extensibility of our framework. In Table 2, we list several such choices of along with their corresponding reparam- eterized exploratory bonus terms under three different a-divergences. From a practical standpoint, since Lromus is computed as expectation over e ), it does not require additional sampling and can be seamlessly integrated into iterative online RLHF Meanwhile, to avoid unintended decreases in the likelihood of preferred responses, we follow Chen et al. (2025) and restrict the computation of the bonus on rejected responses to ensure that the probability of preferred responses continues to increase.
Prior exploratory bonuses are encompassed within GEB. Although we have shown that exist- ing theoretical formulations of Coons fail to guarantee optimism, many practical implementations have nevertheless been effective through various approximations and adaptations. These approx- imations and adaptations are generally inextensible beyond the reverse KL divergence (detailed in Appendix B.3). In this paragraph, we show that these practical implementations can be nat- urally subsumed into our GEB framework, and even broader objectives can be reinterpreted as instances of optimistic exploration. For example, Zhang et al. (2024a) and Xie et al. (2024) fi- nally implement their exploratory bonus as KE log क (y]zr), which belongs to GEB when
log nT + 1 and is KL-divergence. Similarly, Cen et al. (2025) implement the ex- where Teal is a fixed calibration distribution. This also ploratory bonus as kiEz=p.gr~<ut(()) log Feet falls under GEB by setting u = Ral log * Rel log Teet + 1 and f is KL-divergence. Inter- estingly, even objectives not explicitly designed for exploration can be reinterpreted through our GEB framework. For instance, Chen al. (2025) augment the DPO loss with an additional term
a.
7

---

Published as a conference paper at ICLR 2026
ky 0- log < a) sarld/)) ). which was originally introduced to control sample complexity. In 1.
our framework, this corresponds to optimistic exploration with u = - =(v|=)
Tarlyl))
Takeaways. In summary, we introduce a general formulation of the exploratory bonus term, GEB, which showcases the following unique strengths: (1) In contrast the prior bonus terms, GEB meets the optimism condition theoretically, thus provably boosts explo- ration on relatively untapped region; (2) GEB spans broad class of instantiations as shown in Table 2, specified by the combination of the a-divergence class and the functional form of u, which provides plenty options for practitioners to choose on demand; (3) GEB is prac- tical as it does not incur additional sampling costs and can be seamlessly integrated into the existing iterative online RL.FF framework (Appendix C.I): )); and (4) GEB offers an unified understanding of existing methods by generalizing them as special cases in single flexible formulation.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Following prior works (Zhang et al., 2024a; Xie et al., 2024; Chen et al., 2025), we adopt the same iterative online algorithm as in Algorithm with three iterations, aiming to isolate the effeets of different exploration bonuses. We adopt two LLM backbones: -lama-3-8B-SFF (Dong et al., 2024) following prior works, and Mistral-Instruct--0033 (Jiang et al., 2023). The training prompt set is A.HaFewwUltrraFeedback (Dong al., 2024) as in previous works. URM-LLaMa-3.1-2 (Lou et al., 2024) serves as the preference oracle. We evaluate the outcome policies on both in- domain and out-of-domain test sets. Specifically, for the in-domain test, we use a held-out test set from UltraFeedback (Cui et al., 2024), and sample 64 times per prompt with the outcome policy to compare the average reward and win-rates against the base model. We use length- contrllcn Al AlpacaFvva2 benchmark (Dubois et al., 2024) with GPT-4 judge for out-of-domain alignment test, and MATH-500 (Lightman al., 2023) to evaluate out-of-domain reasoning ability.
Baselines. We adopt f-DPO (Wang et al., 2024), which extends DPO to the f-divergence class, as the primary baseline. We further compare GEB with three optimistic-explonan corporate exploratory bonuses- SELM (Zhang et al., 2024a), XPO (Xie al.,, 2024), and VPO (Cen et al., 2025). Since the approximations or adaptations in their implementations do not extend beyond the KL divergence, we report their results only under KL. In contrast, we introduce a new baseline, Failed Exploratory Bonus (FEB), which removes these approximations or adaptations, i.e., Eq. 13.
methods that in-
5.2 RESULTS & ANALYSES
GEB delivers robust improvements across different loss designs, divergence classes, and lan- guage model backbones. The experimental results are shown in Table 3. Across both backbones, GEB generally outperforms f-DPO and FEB. Under the KL-divergence, GEB displays better or least on- ne-par performance compared to prior exploratory-bonus methods. Notably, the win-rate in- creases over 1.82% and 0.94% under the KL-divergence, over 2.36% and 1.29% under the Hellinger Distance, compared with their f-DPO counterpart. GPT-4 evaluation on the Alpaca benchmark also shows consistent performance gains on the off-o-ddomain alignment task. While GEB maintains on par, or usually better results in MATH, showing less performance degradation beyond alignment, known as alignment tax (Noukhovitch et al., 2023; Lin et al., 2024).
GEB effectively encourages exploration in small Tee region, yielding more diverse sampling. In Figure 2, we visualize the distribution of log गरe for sampled responses in the last iteration un- der the KL divergence. When trained with the GEB, the policy model consistently samples more trajectories with smaller wref compared to the policy trained by f-DPO loss. This validates our motivation that GEB can encourage sampling trajectories of small गet for optimistic exploration. In Table 5, further calculate the distinct-n (n 1, 2, 2, 4) for the sampled responses in the last iterations under the KL divergence, which measures the diversity of a corpus. GEB generally has higher diversity scores, validating that GEB incentivizes qualitatively more diverse samples.
8

---

Published as a conference paper at ICLR 2026
lass
ADPO
ADPO
GEB-1/t
GfBearcentie
Dent
Densty
-9000
-8000
-7000
-5000
-5000
-8000
-7000
-6000
-5000
-9000
-5000
-7000
-6000
-5000
lgg sampled responses
109 mppleded responses
19g sampledd responses
Figure 2: Comparison of log Tee of sampled response in the last iteration between the general ex- ploratory bonuses and vanilla iterative DPO. GEB-T, GEB-1/T, and GEB-arctanh 1) corre- sponds + a x, 1/π, nr, and arctanh(1 T) in Table
Table 3: In-domain evaluation on different exploration bonuses. Boldface and underline indicate the best and the second-best results, respectively. GEB-T, GEB-1/M, and GEB-a HEB-arctanh(T corresponds to 1 a n, 1/T, and arctanh(1 - π + a as Table 2.
KL (a=l)
Hel. (a=0.5) WR
[-KL (a=0) WR
Avg.
WR
AvgR
AvgR
AvgR
WR
AvgR
Mistral Instruct-vO..
f-DPO
78.42
0.7480
72.69
0.6536
51.11
0.5918
67.40
0.6645
SELM
77.56
0.7530
XPO
79.71
0.7492
VPO
78.57
0.7426
FEB
78.42
0.7480
71.54
0.6525
47.53
0.5928
65.83
0.6644
GEB-T
81.00
0.7542
75.48
0.6641
51.68
0.5976
69.39
0.6720
GEB-1/T
80.00
0.7554
73.97
0.6541
52.26
0.6051
68.74
0.6715
GEB-arctanh 1)
79.71
0.7559
75.69
0.6614
52.76
0.5989
69.39
0.6721
LLaMA-3-8B-SAP
f-DPO
73.11
0.8050
71.11
0.7859
67.38
0.7579
70.53
0.7829
SELM
74.19
0,8126
XPO
72.40
0.8119
VPO
71.61
0.7971
FEB
73.11
0.8050
68.17
0.7591
67.95
0.7611
69.74
0.7751
GEB-T
74.34
0.8156
71.68
0,7840
67.67
0.7681
71.23
0.7892
GEB-1/T
74.76
0.8102
72.25
0.7859
68.17
0.7591
71.73
0.7851
GEB-arctan((T-11
74.98
0.8080
73.26
0.7877
68.89
0.7569
72.38
0.7842
Table Out-of-domain evaluation on different exploration bonuses with LLaMA-3-8B-SFT. Bold- face and underline indicate the best and the second-best results, respectively. GEB-R, GEB-1/T,
and GEB-arctanh(T 1) corresponds + a m, 1/π, and arctanh(1 T) + a as in Table 2.
KL(a=1)
(al.(a=0.5)
f-KL(a=0)
Avg.
Alpaca
Math
Alpaca
Math
Alpaca
Math
Alpaca
Math
f-DPO FEB
25.72
67.6
24.73
69.0
17.80
69.2
22.75
68.6
25.72
67.6
23.75
68.6
19.62
68.6
23.03
68.3
GEB-a
28.27
69.2
25.87
69.6
20.05
71.6
24.73
70.1
GEB-1/ 1/T
16
68.4
25.28
70.2
19.80
69.2
23.73
69.3
GEB-arctanh(TT
24.90
71.0
25.96
67.6
19.62
69.2
23.49
69.3
The choice of Since the formulation of u in Eq. 9 is flexible, the scale of the GEB term can differ substantially across designs, hence the absolute value of the bonus is less informative. Instead, we examine the relative ratio of the bonus term to vanilla RL loss /sCtoonel/\RRii which provides more consistent basis for comparison and offers better practical guidance for tuning across diverse settings. As shown in Fig. 3, performance remains stable when the ratio lies within a suitable range (1e-2 to 1e-6 in case). However, if the ratio is too large, it impedes optimization of the RL objective and degrades performance; if too small, the exploration incentive in uncertain regions diminishes, and performance reverts the vanilla baseline.

---

Published as a conference paper at ICLR 2026
76. 096
72.0%
win-rade (2663
win-caln (EXC))
68.0%
12.0%
60.0%
CCB
10.0%
GEB
GEB GEB lie
00.09
QCB
CBB liee
erctannis
Cull-erctanhi--1
02.0%
06. 2%
66.0%
60.0%
Figure 3: Experiments with different The three graphs under KL divergence, Hellinger Dis- tance, and forward KL divergence from left to right, respectively. The p. f. tanh in the legends correspond to
æ, 1/m, arctanh(1 æ) in Table 2 respectively.
Table 5: Dist-n of the sampled corpus in the last iteration under the KI divergence.
dist-1
dist-2
dist-3
dist-4
f-DPO
0.0189
0.2700
0.6349
0.8418
GEB-r
0.0192
0.2694
0.6323
0,8420
GEB-1/4
0.0191
0.2738
0.6401
0,8448
GEB-arctanh(T 1)
0.0192
0.2730
0.6391
0.8447
Table 6: The performance (Pass @ 16) of DPO and three GEB variants math reasoning tasks.
MATH500
OlympiadBench AIME 2025
Avg.
DPO
89,80
57.78
23.23
56,94
GEB-I
92,80
64.59
28.23
61,87
GEB-1/T
93,00
65.78
29.48
62.75
GFBB-arctanhor
92.80
64.59
29.38
62.26
GEB improves exploration in challenging problem instances. To further illustrate when GEB offers the greatest benefit, we conduct an additional experiment targeting prompts whose preferred (i.e., correct) answers lie in regions rarely explored by the reference model during sampling. Starting from the DAPO-MATH-I77 dataset (Yu et al., 2025), we filter out prompts for which the reference model, Qwen2.5-7B (Qwen et al., 2025), can generate a correct answer within two samples tem- perature 0.6. This removes 5K prompts whose correct answers fall in frequently visited regions, open-source
leaving 12K prompts whose correct answers are less likely to be sampled. We verifier, rather than reward models, to determine correctness. Each preference pair consists one correct response and wrong response. We set the learning rate 5e-7 with warm-up ratio of 0.1. For the reported runs under KL divergence, the k values for the three GEB variants are 50, 5e-2, and 5c-6, respectively.
As shown in Table 6, all three GEB variants consistently surpass the DPO baseline with a clear and stable performance margin. Notably, the performance gain of GEB more than 6% on Olympiad- Bench and AIME 2025. It indicates that our GEB can significantly enhance the exploration under scenarios where the correct answers lie region with small Jet i.e. underexplored region. An additional experiment is provided in Appendix D.
CONCLUSION
While recent work proposes exploratory bonuses to operationalize the "optimism in the face of un- certainty" principle, our work shows that the existing theoretical frameworks exploratory bonuses fail under KL and a-divergence regularization. To address prior theoretical pitfalls, we introduce General Exploratory Bonus (GEB). a novel theoretical framework for sample-efficient RLHF. Our approach provably satisfies the optimism principle and unifies prior heuristic bonuses. We em- pirically validate GEB on LLM alignment tasks with diverse bonus designs and LLM backbones, showing improved performance and broad applicability across multiple divergence families.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
We thank Froilan Choi and Shawn Im for their valuable suggestions on the paper, and thank all the ICLR 2026 PCs, SACs, ACs, and anonymous reviewers. This work is supported part by the AFOSR Young Investigator Program under award number 199550-23-101888- National Sci- ence Foundation under awards IIS IIS-2237037 and IIS-2331 1699, Office of Naval Research under grant number N00014-23-1-2666 Schmidt Sciences Foundation, Open Philanthropy, Alfred P. Sloan Fel- lowship, and gifts from Google and Amazon.
REPRODUCIBILITY STATEMENT
We have included all implementation details, hyperparameters, and training procedures in the paper and appendix. Our code and scripts for reproducing the experiments are publicly available through GitHub.
ETHICS STATEMENT
This work studies reinforcement learning from human feedback (RLHF) using only publicly avail- able or synthetic data, without new human subject collection. Here, by providing rigorous theoret- ical framework with strong empirical evidence, we pursue high standard of scientific excellence. We also take into account inclusiveness to make all our visualizations accessible to the unprivileged group of people by producing figures distinguished by light, shade, and marker. While RLHF has the potential to amplify biases or harmful behaviors if misused, our work is intended solely to advance safe and responsible research, and we encourage its application in alignment with ethical standards.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learn- ing from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL, pp. 12248-12267, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statisties, pp. 4447 4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang. Dale Schu- urmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline RLHF 2025.
Mingyu Chen, Yiding Chen, Wen Sun, and Xuezhou Zhang. Avoiding exp(rmaa) scaling in rlhf through preference- exploration. arXiv preprint arXiv:2502.00666, 2025.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. CoRR, abs/2506.14758, 2025.
Ganqu Cui, Lifan Yuan. Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie. Zhiyuan Liu, and Maosong Sun. Ultrafeedback:: Boosting language models with high-quality feedback. In The Thirteenth International Conference on Learning Representations, ICL.R 2024.
11

---

Published as a conference paper at ICLR 2026
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024.
Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang. Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHE. CoRR, abs/2405.07888 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. CoRR, abs/2404.04475. 2024.
Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy. Efficient exploration for llms. In Forty-first International Conference on Machine Learning, ICML, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, and et al. Anirudh Goyal. The Ilama 3 herd of models, 2024.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexan- dre Ramé, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online AI feedback. CoRR, abs/2402.04792, 2024.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in MI. safety. CoRR, htss/2199.13116. 2021.
Kaixuan Ji, Jiafan He, and Quanquan Gu. Reinforcement learning from human feedback with active queries. CoRR, abs/2402.09401, 2024.
Albert Q. Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Am Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06882 2023.
Aobo Kong, Wentao Ma, Shiwan Zhao, Yongbin Li, Yuchuan Wu, Ke Wang. Xiaoqian Liu, Qicheng Li, Yong Qin, and Fei Huang. SDPO: segment-level direct preference optimization for social agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational tics (Volume 1: Long Papers), ACL, pp. 12409-12423, 2025.
Lingull Linguis-
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. CoRR, abs/1811.078711 2018.
Wendi Li and Yixuan Li. Process reward model with v-value rankings. In The Thirteenth Interna- tional Conference on Learning Representations, ICLR, 2025.
Wendi Li, Wei Wei, Kaihe Xu, Wenfeng Xie, Dangyang Chen, and Yu Cheng. Reinforcement learning with token-level feedback for controllable text generation. In Findings of the Association for Computational Linguistics: NAACI, pp. 1704-1719, 2024.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint wwXiv:2305.200500 2023.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations. ICLR, 2024.
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jiammeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. Mitigating the alignment tax of RLHF. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, EMNLPP, pp. 580-606, 2024.
12

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. Sample-efficient alignment for Ilms. arXiv preprint arXiv:2411.01499, 2024.
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. Uncertainty- aware reward model: Teaching reward models to know what unknown. arXiv preprint arXiv:2410.00847, 2024.
Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff G. Schneider, and Willie Neiswanger. Sample efficient reinforcement learning from human feedback via active exploration. CoRR, abs/2312.002677. 2023.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. In Forty-first International Conference on Machine Learning, ICML, 2024.
Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron C. Courville. Language model alignment with elastic reset. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. CORR, abs/2404. 1000044.99333 19733, 2024.
Han Qi, Haochen Yang. Qiaosheng Zhang, and Zhuoran Yang. Sample-efficient reinforcement learn- ing from human feedback via inoomatioo-directee sampling. arXiv preprint arXiv:2502.04444, 2025.
Qwen,
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2024.
Nisan Stiennon, Long Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021 2020.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Ste- fano Ernon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of 1lms should leverage suboptimal, on-policy data. In Forty-first International Conference on Machine Learning, ICMI., 2024.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding the performance gap between online and offline alignment algorithms. CoRR, abs/2405.08444. 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2:Xiv:2307.09288 2023.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: gener- alizing direct preference optimization with diverse divergence constraints. In The Twelfth Inter- national Conference Learning Representations, ICLR, 2024.
13

---

Published as a conference paper at ICLR 2026
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang. Xionghui Chen, Jianxin Yang. Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropp minority tokens drive effective reinforcement learning for LLM reasoning. CoRR, abs/2506.01933/ 2025.
Fang Wu, Weihao Xuan, Ximing Lu, Zaïd Harchaoui, and Yejin Choi. The invisible leash: Why RLVR may not escape its origin. CoRR, abs/2507.14843. 2025.
Tengyang Xie, Dylan Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit qf-approximation for sample-efficient RLHE, CoRR, abs/2405.2 1044,, 2024.
Wei Xiong, Hanze Dong. Chenlu Ye, Ziqi Wang. Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under kl-constraint. In Forty-first International Conference on Machine Learning, ICML, 2024a.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under kl-constraint. In Forty-first International Conference on Machine Learning, ICML, 2024b.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? A comprehensive study. In Forty-first International Conference Machine Learning, ICML, 2024.
Min-Hsuan Yeh, Jeffrey Wang, Xuefeng Du, Seongheon Park, Leitian Tao, Shawn Im, and Yixuan Li. Position: Challenges and future directions of data-centric ai alignment. In Forty-second International Conference on Machine Learning Position Paper Track, 2025.
Yueqin Yin, Shentao Yang, Yujia Xie, Ziyi Yang. Yuting Sun, Hany Hassan Awadalla, Weizhu Chen, and Mingyuan Zhou. Segmenting text and learning their rewards for improved RLHF in language model. CoRR, 022220000.299, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arxiv:2503. 14476, 2025.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kai Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. CoRR, abs/2412.01981, 2024.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. In Forty-first International Conference Machine Learning, ICML, 2024.
Shenao Zhang, Donghan Yu, Hiteshi Sharma, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhao- ran Wang. Self-exploring language models: Active preference elicitation for online alignment. CoRR, abs/2405.19933 2024a.
Xuanming Zhang, Yuxuan Chen, Yiming Zheng, Zhexin Zhang, Yuan Yuan, and Minlie Huang. Seeker: Towards exception safety code generation with intermediate language agents framework. arXiv preprint arXiv:2412.11711 2024b.
Xuanming Zhang, Yuxuan Chen, Min-Hsuan Yeh, and Yixuan Li. Metamind: Modeling human social thoughts with metacognitive multi-agent systems. CoRR, abs/2505. 18944, 2025.
Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhou- futu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, and Zejun Ma. First return, entropy-eliciting explore. CoRR, abs/2507.07017. 2025.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Alignment & RLHEE Alignment (Hendrycks al., 2021; Leike al., 2018; Zhang et al., 2024b; 2025; Yeh et al., 2025) aims to ensure AI systems act in accordance with human values, preferences, and goals; and it has become critical field in Al research. To steer language models to match human preferences, Reinforcement Learning from Human Feedback (RLHF) (Stiennon et al., 2020; Ouyang et al., 2022) achieves great success and has become the standard alignment pipeline. How- ever, its computational complexity has motivated family of Direct Preference Optimization (DPO) (Rafailov et al., 2024; Azar et al., 2024; Ethayarajh et al., 2024) that forgoes explicit reward mod- eling. Despite their efficiency, recent researchers (Tajwar et al., 2024; Xu et al., 2024; Xiong et al., 2024b) reemphasize the significance of online sampling.
Optimistic exploration of RLHE.
To address the computational overheads of passive exploration in RLHE, which samples trajectories just based on randomness, some existing attempts have been devoted to sample-efficient RL algorithms. Most of the works (Das al., 2024; Ji al., 2024; Dwaracherla et al., 2024; Mehta et al., 2023; Muldrew et al., 2024) adhere to the principle of op- timism, proposing specialized prompt or response selection strategies emphasize uncertain sam- ples. While some research (Liu et al., 2024; Lou et al., 2024) propose uncertainty-awaat reward models with epistemic neural networks or bootstrap ensembles, these methods introduce additional cost. Some research also addresses the sample efficiency with different theoretical foundations, such as information theory (Qi et al., 2025), preference-incentive exploration (Chen et al., 2025). Notably, several works (Zhang et al., 2024a; Xie et al., 2024; Cen et al., 2025) introduce different exploratory bonuses, which can implement optimism toward uncertainty without additional compu- tations. However, they only focus on KL-divereence and their theoretical framework cannot result in real optimism as shown in Section 3.2.
Efficient RL for LLM. Beyond optimistic exploration, some research proposes fine-grained sig- nals for RL learning. For instance, several studies propose segment-level (Kong et al., 2025; Yin al., 2025) or token-level (Zeng et al., 2024; Li et al., 2024) reward functions for alignment or text control. Notably, for reasoning tasks, the process reward model (Lightman et al., 2024; Li & Li, 2025; Yuan al., 2024), which provides step-wise feedback for solutions, has shown promising effectiveness. On the other hand, recent research (Cheng et al., 2025; Zheng et al., 2025; Wu et al., 2025) on LLM reasoning reveals that high-entropy tokens guide the model toward diverse reasoning paths. Training with only high-entropy tokens is more beneficial for reasoning performance (Wang et al., 2025). While our approach is highly extensible, we believe the orthogonal methods can be further incorporated with our general exploratory bonus.
B OPTIMISM FAILURE OF PREVIOUS WORKS
B. OPTIMISM FAILURE UNDER KL-DIVERGENCE
We start by proving how the existing exploratory bonus term under KL-divergence instantiation fails to achieve optimistic exploration.
Lemma 3.1
Letri = arg min, L (D,r) be a reward model trained with the vanilla BT loss, and let r2 arg min, (Dn, exploratory bonus. If the policy is optimized via Eq. 4, then 1 and T2 yield the same set of policies.
max, Ja,k1 (z)r)) be reward model trained with an additional
Proof First, the inner maximization of the bonus term admits closed-form solution, π (yllz) rix.z) rimaa) /z)): where 7(x) E Ey==se((-aae 3 is normalization term. Substituting this
solution for the li-le objective of r2 reduces it to a single-level form:
72 = arg min [Cer(D,r) KEE log Eysan e
(12)
As shown in Rafailov et al. (2024), the log-ratio 3 log no (3/2x) - 3 log ) represents the same class of the original reward function through Eq. 4, thus all in the reward modeling objectives
15

---

Published as a conference paper at ICLR 2026
can be reparameterized by the log-ratio. Plugging this into Eq. 12 yields
x (
arg min (D, σ) xBx=pp log E (1)
(13)
[
Since the second term equals O, the reparameterized Eq. 12 exactly the vanilla DPO loss, that is, the reparameterized training objective of r1. Thus, the exploratory bonus in the reward training objective does not affect the final policy set.
B.2 EXTENSION BEYOND a-DIVERGENCE
The following theorem formally proves that the exploratory bonus max, (8, cannot en- courage optimism for a more general divergence class.
When f belongs I-divergence, the reward function is obtained via &f arg min, (Cr maxx (T,r, and the policy is updated by arg max, [ (T, f), the bonus term K max; Jo. induces the policy model π to coincide with Tee when x fd (x) is monotone function.
Theorem 3.3
Proof By Lemma 3.2, we can reparameterize the bonus term used for optimistic reward modeling into Eq. 6. Denote h(u) = (u) f(u). For a fixed prompt z, the training step can then be written as the following constrained optimization problem:
π (y]rr (rer(y(ar))
arg max E (1 /(
s.t.
Σm(yla) = and Vy, > O.
(14)
Then we can apply the Lagrange multiplier as
Tylax
C E
/(
(yll)) 14 (E
n T(y|>) - - (v)))
(15)
y/
where μ., the dual variables. Then utilize the Karush-Kull Kuhn-Tucke (KKT) conditions for the given optimization problem. The complementary slackness gives that Vy, 7())<((())) = 0. The stationary condition requires
OC e
N'
μr 7
(16)
()))(
Let S, = {य]τ(v|>) > 0}. For all E Sy. complementary slackness implies 778) = 0. Since N'(w) (u) is monotone function, we can obtain Vy Sy ma([)la)
s(vlz) is a constant. Then
applying the normalization constraint, Eyysna(-la)) <a(()>)) interior optimum is T"" Ty(zz) T))())
a(vlæ) 1. Hence, we conclude that the unique
The theorem implies that the reparameterized exploratory bonus attains maximum only when m and The coincide. The condition that (x) is monotone function is satisfied by a-divergence and beyond, e.g. Pearson x². Hence, the exploratory bonus -k max, (T, r) in reward training objective generally contradicts the optimism, since it cannot encourage trajectories with small initialized possibility.
B.3 PRIOR ADAPTIONS & APPROXIMATIONS CANNOT GENERALIZE
Although the theoretical justification for the previously proposed exploratory bonus does not hold, its empirically implemented loss remains effective due to various adaptations and approximations. In this subsection, however, we show that these adaptations and approximations cannot be extended beyond the KL-divergence class.
Zhang al. (2024a) modify the formulation of (o, in the optimistic exploratory bonus - max+ (T, as
(T,)) = o Ir r(z, // v/) 3D (T meet)
(17)
which adds a bias in the reward expectation term. Under KL-divergence, the original (T, r) will be zero after ee-parameterization shown in Lemma 3.1, thus the sole reparameterized bias term
16

---

Published as a conference paper at ICLR 2026
will remain as -ma log (y [z). Since (T, r) cannot be reparameterized to zero except KL-divergence, this adaptation cannot generalize then.
In the derivations of Cen et al. (2025), idcalized calibration distribution Teal is assumed to satisfy Ey- TT, y) = O. Since Teat is not directly accessible in practice, the method substitutes rejected responses to approximate expectations under E,,... These rejected samples, however, do not satisfy the defining property of Teal, making the approximation theoretically inconsistent.
The theoretical framework of Xie et al. (2024) is based Implicit Q-Approximation (refer to Lemma C.3 in the original paper) as follows,
log (z(a) rlz, v) (*)
8
(18)
log [Ter(
where V(z)* is the KL-regularized value function. Because this relation depends critically on logarithmic structure of the KL divergence, the framework cannot be extended to more general divergence families either.
In contrast, general exploratory bonus integrates scamlessly with iterative online RLHF algo- rithms and extends naturally to the entire a-divergence family. As discussed $4, all these heuristic bonus terms can be encompassed by our unified theoretical framework.
B.4 EQUIVALENCE BETWEEN SOPHISTICATED OBJECTIVE AND SIMPLE ONE.
Lemma 4.1 Denote two objectives as h(((,, y)) Be- u(x, v) (u(r, v)) f(u(x, v)) u(z,))
and M(((,, 3)) E 2(2) ('( -[( [(=((a) where u(x, 9) a function with <(yl)) and V)). If the ratio
('(u(,, v)) + u(x, v) (" (u(x, u(a.)
Z(a)
A(x)
(19)
Z(z)u(z, v) (u(x,)))
independent of y and (xx) > O, then minimizing the two objectives, min, h(((x, (u(x,))))))))))))))() y)) and min, -h(u(x; y)), yields the same class of optimal policies.
Proof Similar to Lemma 4.1, for fixed x, we can write similar formulation of the Lagrange multipliers as follows,
L = Ey h Mul(,, v)) 11 Σ n <(y - 1) Σ (,(()))
(20)
where μt, are the dual variables. Then, we utilize the KKT conditions formulation for the given optimization problem. Similar to Lemma 4.1, when T(z, v) > O, obtain
Ōh (y(z) u(x,))) (u((x)))) <))()>>) θ Tπ(()
Ou(x, v)
p1-
(21)
Similarly, we can obtain KKT conditions for M(((,, v)) as follows,
ah @π(zzz = 2xx)
(f'(((z, v)) + u(x, v) (u(x, v)) - f' u(xx,)) ((x,,))) Te(())))
Øu(x. y)
Z(x)
O
42
(22)
where (12 is another dual variable. With the condition of Eq. 20, these two partial derivatives in Eq. 21 and Eq. 22 are equivalent when μ12 = A(x). Hence, every policy that satisfies the stationary 2%
condition for also satisfies it for h. Since A(x) > 0, the second-order derivative (=*()(=) and
8% have the same sign, which indicates they share the same local minima. Hence, minimizing [
the two objectives min, -h( and min, -h(u) induces the same class of policies.
B.5 GEB ENABLES OPTIMISTIC EXPLORATION
In this subsection, prove how GEB meets the optimism principle specified by Definition 3.1.
17

---

Published as a conference paper at ICLR 2026
Theorem 4.2 Consider a-divergence f with 0 I 1, and the exploratory bonus Lbomus BEz u(z, u) (u(a, v)) f(u(x,)) v)) where w(x, u) is function dependent on (8) and Tey(y)a)) x). For any (x,y), 1), if ūn Fed Srbaay optimism condition in Definition 3.1 is satisfied; that is,
(a-1)mur
< and u(x, y) > a. the
Lamas S O. Ow
First, substituting the optimal solution of the inner maximization and utilizing reward repa- rameterization, we obtain the training objective as in Eq. 9. By Lemma 4.1, this can be equivalently expressed as
Proof
Lioous p. [u(a,, ()((((( v)) - [(((z,)))
(23)
For a-divergences, the conditions in Lemma 4.1 are satisfied. Since we have ('()) + uf) f(()) u (z- Lemma 4.1 is independent of Since u > a. Z = Eyymat(1))) > 0, thus A(x) > 0 is also
a:) and ff
aa the fraction A(x) (Z 1 a aa) (1)
satisfied. Finally, the mixed second-order derivative of Eq. 11 computed
8 Livous imiāee
Zu ŌT
82u (ax 1) 1) du du <0, 0, Ōx OTyel
= =BREgrpp
Σ
TTll Önd
(24)
a
which achieves the optimism defined in Definition 3.1.
C EXPERIMENTS
C.I IMPLEMENTATION DETAILS
Algorithm. Following prior work (Zhang al., 2024a; Xie al., 2024; Chen et al., 2025), adopt the same algorithmic backbone for empirical validation in order to isolate and compare the ef- fects of different exploratory bonuses in the loss function. This backbone bypasses reward modeling at each iteration through reward reparameterization, a procedure commonly referred to as iterative DPO (Dong al., 2024). Previous methods of exploratory bonuses further reparameterize their bonus terms to make them compatible with this framework. To extend the original framework to a-divergence, we extend the iterative DPO with f-DPO (Wang et al., 2024) loss, and likewise repa- rameterize our generalized exploratory bonus accordingly. The full procedure is summarized in Algorithm 1. Since the only difference the loss function Line Algorithm 1, our GEB does not induce any additional compute costs.
Algorithm Iterative Online Algorithm with Exploratory Bonus
Input: Reference model Tet. iteration number T,, prompt set for each interation Dı,..., Dr re- ward function r; Output: Trained model nT:
1: for iteration t 1, 2, 2,
do
2: for x E D, do
3: V1. ~ T-e((11>) and obtain the rewards r(y1), r(y));
4: Rank the reward and denote y+ as the preferred and dispreferred response between
5: V1. and update D. {a. y
arg min, LDPO K &hhoous (T) 6: update ाe with n, (optional)
7: end for
8: end for
Hyperparameter settings and environments. All experiments are conducted on two NVIDIA H200 GPUs. When training and sampling, the max length is set to 2048. For training, the batch size per device is set to 2; we enable gradient checkpointing, and the gradient accumulation step is set to 64; the learning rate is 5e-7 with cosine scheduler, and the warm-up ratio is 0.03. In the main experiments, we use the best performance with x with a suitable ratio range to f-dpo loss across 1, 2. 1e 4, 1C 6, le 8. For sampling, the temperature is set to 1. For in-domain evaluation and MATH evaluation, we set temperature to 0.6 and top-p to 0.9; we use the default setting of alpaca-eval.
18

---

Published as a conference paper at ICLR 2026
480
0.2
0.2
200
400 C2R 00c
1000
Figure 4: Comparison on the bandit policy distributions trained with DPO (left) and GEB (right). The DPO policy collapses to local optimum, while the GEB policy continues to explore and ulti- mately chooses the globally preferred action.
D TOY EXPERIMENTS
0.230
To illustrate a setting in which GEB yields sub- stantial improvement, we construct a toy exam- ple in which the most preferred action lies in rarely visited region.
80000
0122
0.000
D.1 EXPERIMENTAL SETTING
Steawkctien Spice
Figure 5: Initial reference bandit distribution ("ref") and the reward distribution. Because the most preferred action lies in a low-probability re- gion, it is rarely visited under purely passive ex- ploration.
We consider 1000-arm bandit with 1000 pa- rameters, each parameter corresponding to distinct arm. As shown Fig. 5, the most pre- ferred action lies in rarely visited region, mak- ing it unlikely to be sampled under pure passive exploration, as in f-DPO. Each experiment is run for 5000 iterations. At each iteration, the bandit policy generates 64 rollouts to form a batch 32 preference pairs. The learning rate is set to le-2 with no warm-up phase.
D.2 RESULTS AND ANALYSES
As shown in Fig. 4, the bandit policy trained with DPO becomes trapped in local optimum: its probability mass collapses onto a suboptimal action because the policy never encounters the truly preferred action during training. In contrast, all three GEB variants successfully recover the desired distribution (right panel of Fig. 4), concentrating probability on the most preferred action. This demonstrates that GEB effectively promotes exploration into low www-pobaablltty regions, enabling the policy to discover and select the optimal action despite its small initial likelihood.
E SUPPLEMENT RESULTS OF MAIN EXPERIMENTS
E. REPEATED RUN OF THE MAIN EXPERIMENT
To assess the statistical significance of GEB's performance gains, we repeat each experiment five times and report the mean and standard deviation. As shown in Table 7, GEB achieves consistently higher performance than baseline methods, with statistically significant improvements. Moreover, conduct independent p-test between GEB variants and f-DPO. The p-value is shown in Table 8. Most p-values are less then 0.05, which means the improvement of GEB is statistically significant.
E.2 SEMANTIC COHERENCE OF THE SAMPLED RESPONSES
We use GPT-4 to evaluate whether a given sentence is coherent, nonsensical, or contains meaningless content. The scoring scale ranges from 0 (fully coherent) to 3 (complete nonsense with coherent meaning). We apply this evaluation to the responses generated in the final training iteration of DPO and the three GEB variants. The resulting scores are follows.
19

---

Published as a conference paper at ICLR 2026
Table 7: Repeated in-domain evaluation on different exploration bonuses. Boldface and underline indicate the best and the second-bes best results, respectively. GEB-T, GEB-1/T, and GEB-arctanh(? 1) corresponds + a 2, 1 IT, and arctanh(1 π) + a as in Table KL (aml)
(o=0.5)
F-KL (a=0)
WR
AvgR
WR
AvgR
WR
AvgR
Mistral Instruct v0.3
F-DPO
77.49 +1.32
0,7426 =0.0201
72.70 +:0.6
0.648820.0114
50.83+0.60
0.5902 0.030
GER-E
80.60-+0096
0.7531=0.0162
74.41-40.72
0.6723+0.0211
51.75440.51
0.5985+-0.0134
GEB-1/T
79,69 1.
0.7614=0.000
73.87+0.45
0.634240.0033
51.86:00.00
0.6088=00.002)
GEB-arctanh(r
1)
79.60+1.01
0.7544=0.0212
74.57:20.5
0.6642#0.0231
51.69:1.03
0.5962::000153
ILLMA 3-8B SFT
F-DPO
+:0.6
0.7926=00
71.19:00.68
0.7915+0.0101
22.6240..00 96
0.7566++0.00%
73.90++1110
0.8088
72.08-40.74
67.79+11.14
0.7698+
GEB-1/T
0.8084=0.006
0043
72.28+0078
0.7133=0.0143
67.99 +0.75
0.7618++0..00
GEB-arctanh(t
1)
74.55+11.02
0,8032=0.0019
72.57+1003
0.7902+0.0035
68.34:0.78 76
0.76124:001300
Table 8: p-value of independent p-test between win-rates of GEB variants and f-DPO
Dnstral-Instruct-vbe
LIaMA-3-8B-SFT
f-DPO vs.
KL.
Hel.
f-KL.
KL.
Hel.
f-KL.
GEB-i
0.003
0.014
0.034
0.145
0.084
0.117
GEB-1/T
0.021
0.051
0.027
0.031
0.047
0.037
LLB-arctanh(T 1)
0.023
0.015
0.155
0.024
0.042
0.014
Table Semantic coherence scores of responses produced by policy models trained using DPO and
the GEB variants.
DPO
GEB-T
GEB-1/T
GEB-arctanhr,
KL.
1.24
1.08
1.32
1.19
Hel.
1.33
1.42
1.12
1.23
IKL.
1.32
1.38
1.32
1.30
Table 10: Ablation study on the sample responses used for the bonus term calculation. We compare the results UltraFeedback (win rate) across three divergences with all responses (all) and with the rejected responses only (rejected-only).
Method
KL all
KL rejected-only
Hel, all
Hel. rejected-only
fKL. all
fKL rejected-only
GEB-I
79.78
81.00
73.40
75.48
51.32
51.68
GEB-I
79.56
80.00
72.25
73.97
51.61
52,26
GEB-arctaus
79.64
79.71
73.11
75.69
51.25
52.76
As shown in Table the responses produced by the GEB variants exhibit semantic coherence com- parable to those generated by DPO. This indicates that, in practice, GEB promotes exploration into moderately underrepresented yet still semantically meaningful regions of the output space.
E.3 THE CHOICE OF
The three variants of the exploration bonuses in Table 2 represent different instantiations of the GEB framework. Each satisfies the optimism condition Definition 3.1 and exhibits consistent performance improvements on the alignment task.
Nonetheless, the curvature of u with respect to æ meaningfully affects the optimization dynamies of a, the function is lincar, thus the gradient
the exploratory bonus. For instance, when of u to π is constant. Therefore, the per-trajectory incentive to decrease л constant. In contrast, when u is convex-sut as u 1/TT-th the gradient magnitude diminishes results in a more conservative reduction of π compared to the linear case.
becomes larger. This
Takeaway: The curvature of with respect to governs the behavior of the exploration bonus. Greater convexity leads to a more conservative shift of probability mass toward underexplored re- gions.
20

---

Published as a conference paper at ICLR 2026
Restricting the exploratory bonus to rejected responses Restricting the bonus term to rejected responses is a common practice in prior works on exploratory bonuses (Zhang et al., 2024a; Chen et al., 2025; Cen et al., 2025). Importantly, this restriction does not constitute theoretical departure from our framework. The optimism guarantee Theorem 4.2 hinges on increasing the probability of low Wer regions, i.e., underexplored regions. Because rejected samples lie precisely in these low- probability areas, applying the bonus only to rejected responses preserves the intended optimism direction. While preserving the theoretical guarantee on the optimism, it also shows practical advantage shown in Table 10.
21


Reviews:
[{"review_id": "zd1xaqcbjD", "reviewer": "Reviewer_GVVD", "summary": "The paper addresses optimistic exploration in reinforcement learning from human feedback (RLHF),  arguing that some existing exploratory bonus methods under KL or $\\alpha$-divergence  regularization end up sampling actions with high probability under the reference policy regions  instead of sampling low probability actions. The authors prove theoretically that current  formulations fail to satisfy the optimism principle and introduce the General Exploratory Bonus  (GEB), a framework that introduces reference-dependent reward regulation to counteract  divergence-induced bias. The paper argues that GEB achieves optimism and extends naturally  across the $\\alpha$-divergence family. Experiments on large language model alignment tasks  across different divergence settings (KL, Hellinger, forward KL) and model backbones (Llama-3-8B SFT, Mistral-Instruct-v0.3) demonstrate that GEB outperforms baselines.", "questions": "1. How should practitioners choose among the three GEB variants ($u = 1 + \\alpha - \\pi, 1/\\pi, arctanh(1 - \\pi) + \\alpha$)? Can the authors provide guidance based on task characteristics or on the choice of divergence metric?\n\n2. Have the authors evaluated the performance of GEB beyond language model alignment, such as robotic control or game playing, especially in sparse-reward tasks where optimism-based exploration is generally helpful?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality \n- The theoretical analysis is insightful, formally proving through Lemmas 3.1-3.2 and Theorem 3.3  that some existing exploratory bonus formulations fail to achieve optimism under KL, $\\alpha$- divergence, and general $f$-divergence regularization, with the bonus encouraging sampling from  high probability regions under the reference policy.  \n- The framework unifies prior heuristic methods (SELM, XPO, VPO) as special cases while extending  beyond the KL-divergence to the $\\alpha$-divergence family. \n\nQuality \n- The experimental section covers multiple divergence classes, two LLM backbones, both in-domain  and out-of-domain evaluations (AlpacaEval2, MATH-500), with improvements demonstrated  through win-rates, average rewards, and diversity metrics (dist-n scores). \n\nClarity \n- The paper is well-written with clear motivation and effective visualizations (Figure 1).\n\nSignificance\n- Figure 2 provides evidence that GEB encourages sampling in low-probability reference regions,  validating the theoretical claims.  \n- GEB is shown to be integrable into iterative online RLHF without additional sampling cost.", "weaknesses": "1. Theorem 4.2's optimism condition requires Equation 12 to hold, and $u > \\alpha$, but the paper  provides no empirical verification that these conditions hold during training for the proposed $u$ designs.  \n2. All experiments use only three online iterations following prior work, which may not reveal long term dynamics or potential failure modes.  \n3. The paper does not report computational overhead relative to baselines - how much wall-clock  time does the additional bonus computation and gradient calculations require? \n4. The distinction between the three GEB variants ($u = 1 + \\alpha - \\pi, 1/\\pi, arctanh(1 - \\pi) +  \\alpha)$ is insufficiently explored - why do they perform differently, and how should practitioners  choose among them? \n5. The analysis of $\\mathcal{k}$ selection (Figure 3) reveals performance is sensitive to this hyperparameter, with suitable ranges varying across tasks and divergences. Yet, no principled  automatic selection method is provided beyond empirical search. \n6. The evaluation focuses exclusively on language model alignment; applicability to other RLHF  domains (robotic control, game playing) remains unexplored.\n7. Some of the gains reported in Tables 3 and 4 are marginal. This is possibly a task/training issue. Choosing a domain where the reference model/policy converges on a local suboptimum and GEB performing exploration to converge on the optimal solution might be a better demonstration of GEB empirical success.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bw83yBVrqZ", "reviewer": "Reviewer_E5mE", "summary": "This paper identifies a fundamental theoretical failure in existing exploratory bonus methods for reinforcement learning with human feedback (RLHF) and proposes General Exploratory Bonus (GEB) as a principled solution. The authors prove that current formulations under KL and α-divergence regularization paradoxically bias exploration toward high-probability regions of the reference model rather than uncertain regions, contradicting the \"optimism in the face of uncertainty\" principle. To address this, they introduce GEB with a reference-dependent reward regulation that provably satisfies optimism, unifies prior heuristic bonuses as special cases, and extends naturally across the α-divergence family. Empirical validation on LLM alignment tasks demonstrates consistent improvements over baselines.", "questions": "1. The practical restriction of bonus computation to rejected responses (mentioned in Section 4) lacks theoretical justification. Does this modification affect the optimism guarantee? If so, shouldn't this be analyzed formally?\n\n2. Does GEB improve the constants in the regret bound (Theorem C.1) compared to Cen et al. (2025)? The T-dependence appears identical—is there an improvement in problem-dependent terms?\n\n3. How do the three u designs in Table 2 compare empirically? This would clarify the practical importance of theoretical flexibility.\n\n4. What happens if you remove the restriction to rejected responses and compute the bonus on all responses? This would isolate the theoretical contribution from practical engineering.\n\n5. Are there problem instances or regimes where GEB provides substantially larger improvements than standard methods? Characterizing this would strengthen motivation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The failure modes of existing bonuses are rigorously proven, not merely observed. Lemmas 3.1-3.2 and Theorem 3.3 are solid contributions that clarify fundamental issues.\n\n2. The paper provides a general framework (GEB) that unifies prior work and extends beyond KL divergence.\n\n3. Figure 1 provides intuitive understanding, while mathematical sections deliver rigor. Both audiences (practitioners and theorists) can engage.\n\n4. Testing across two LLM backbones, three divergence settings, and both in-domain and out-of-domain tasks demonstrates breadth.", "weaknesses": "1. The practical implementation restricts bonus computation to rejected responses for stability, but this isn't theoretically justified. Does this modification preserve the optimism guarantee? If not, the theory's practical applicability is weakened.\n\n2. Win-rate gains are ~1-2% on UltraFeedback and marginal on out-of-domain tasks (Table 4). While consistent, these are not compelling. Concern: Is the gap between theoretical promise and empirical gains expected? Could better tuning of κ yield larger improvements?\n\n3. Missing Ablations: What is the effect of restricting the bonus to rejected responses? Ablating this would isolate theoretical vs. practical design choices.\n\n4. The regret bound in Theorem C.1 has the same dependence on T as prior work. Does GEB offer better constants? This isn't discussed, weakening the theoretical justification for improved sample efficiency.\n\n5. The paper doesn't characterize problem instances where GEB would yield larger improvements. When is exploration in low-pi_{ref}\nπref​ regions most critical?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DqJZEY46y0", "reviewer": "Reviewer_3ha8", "summary": "This paper aims to address limitations in existing exploratory bonus formulations for reinforcement learning from human feedback (RLHF). The authors claim that prior approaches under KL or α-divergence regularization fail to achieve true “optimism in the face of uncertainty.” They propose the General Exploratory Bonus (GEB) framework, which theoretically corrects these failures and empirically improves performance on several RLHF benchmarks.\n\nWhile the paper tackles an important question, it suffers from serious clarity, organization, and presentation issues. The writing quality is poor, making the technical content extremely difficult to follow. Furthermore, the claimed theoretical and empirical contributions are not clearly or convincingly demonstrated.", "questions": "Your optimism condition (Definition 3.1 and Theorem 4.2) is central to the paper, but its mathematical and conceptual justification remains unclear. Could you clarify why it is an appropriate formalization of the “optimism in the face of uncertainty” principle, and how it connects to standard exploration metrics (e.g., uncertainty, entropy, or epistemic variance)?\n\nMost reported improvements (e.g., in Tables 3 and 4) appear to be within 1–2 % over baselines. Could you provide more evidence that these differences are statistically significant and not due to noise or hyperparameter sensitivity?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "The topic (exploration in RLHF) is timely and relevant. The attempt to provide a unified theoretical treatment across divergence families could be valuable if clearly presented.", "weaknesses": "The paper’s language is riddled with grammatical errors and awkward phrasing. Many sentences are either ungrammatical or semantically unclear, which severely hinders readability. Several paragraphs are almost unreadable due to convoluted sentence structures and a lack of coherent logical flow.\nWhile Figure 1 conveys an important idea, its design is confusing: it is unclear why a line connects the four sections instead of simply displaying their values. This creates the impression of a continuous transition between them, which undermines the intended clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w9BkcEIMNX", "reviewer": "Reviewer_e552", "summary": "The paper addresses the issue of exploration in LLM finetuning with RLHF. The authors show that a previously proposed novelty bonus does not induce more exploration and extend this insight from the proposed KL-divergence to the class of $\\alpha$-divergences. Finally the authors define a condition they claim to be optimistic and develop a general exploratory bonus (GEB) based on it. Evaluation against other exploratory bonuses appears promising.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The addressed problem of insufficient exploration during LLM fine tuning is relevant and the analysis of the existing exploratory bonus is insightful.", "weaknesses": "I need to clarify that I am not an LLM expert, and am not familiar with any of the cited literature. As such I had huge problems following the paper's story and math. While I would reject this paper due to my lack of comprehension, I am willing to follow the recommendation of reviewers more experienced in this field. I admit that some of my problems might be trivial for experts in the field, but the fact that an outsider cannot follow some of the notations and definitions should not be ignored either. \n\nIn more detail:\n- While the intuition in Figure 1 is very clear, I was wondering whether optimistic exploration (as used in RL literature [1]) is even desirable in LLM fine tuning. Most of the possible trajectory space is not sampled by the reference model and exploring those parts seems counter-productive: most of those trajectories will not follow proper grammar and are out-of-distribution for the learned RLHF reward function (which might therefore prefer them although they are garbage). Both are reasons to stay away from trajectories that have very small probabilities under the reference policy. So which trajectories do the authors *actually* propose to explore?\t\n- Even after repeated attempts, I was unable to make sense of Definition 3.1. In RL literature there is a clear definition of optimism for exploration [1], which requires to estimate an epistemic upper bound on the true return/reward of an action. How is Def.3.1 related to that? What is the intuition behind an \"ideal sampling distribution\" $\\pi_s$? Later the authors mention they \"adopt the commonly use $\\pi_{ref}$ as $\\pi_s$\", but I fail to see what the 2nd order derivative of the exploration bonus w.r.t. the current policy $\\pi$ (that's what it is, right?) and the reference policy $\\pi_{ref}$ has to with optimism. \n- The introduced formalism has many undefined symbols or at least does not explain some default symbols. For example, I do not understand what the \"policy-reparameterized reward model\" in line 167 is. The left-hand side is a function of a policy, but the right hand side contains queries $x$ and completions $y$. Where do these come from? In line 163 it is defined as $r(x,y)=r(\\pi)$, so are $x$ and $y$ just omitted in the function signature? What is the idea behind this reward and what does it have to do with Def.3.1? Another example is the appearance of $f'$ and $f''$ in the proof of Lemma 3.2: is this a partial derivative? If so which one? Why did the authors not use the clearer $\\frac{\\partial f}{\\partial x}$ formalism they relied on earlier? \n- Theorem 4.2 appears grammatically malformed. The first sentence contains an if, but no then part. Later there is a $\\forall (x,y)$ quantifier, but no $x$ or $y$ appear in the theorem. The meaning of these things might be obvious to someone familiar with the literature, but a mathematical statement must be self-contained.\n- The experiments do not contain standard deviation and I could not find out whether they have been repeated at all. This might be common in LLM literature, where repeating experiments can be prohibitively expensive, but I was unable to gauge whether improving from 79.71 to 81.00 is a worthwhile improvement or just stochastic noise.\n\n \n[1] Lattimore T, Szepesvari C. Bandit Algorithms. Cambridge University Press; 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)