Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game
Authors: Barna Pásztor
Thomas Kleine Buening
Andreas Krause
Venue: ICLR 2026 Poster (2026)
Keywords: alignment
rhlf
preference optimization
game theory
human feedback
test-time improvement
Primary area: reinforcement learning
TL;DR: We propose a novel game-theoretic approach to RLHF by framing preference optimization as two-player sequential-move game.

Abstract: We introduce Stackelberg Learning from Human Feedback (SLHF), a new framework for preference optimization. SLHF frames the alignment problem as a sequential-move game between two policies: a Leader, which commits to an action, and a Follower, which responds conditionally on the Leader's action. This approach decomposes preference optimization into a refinement problem for the Follower and an optimization problem against an adversary for the Leader. Unlike Reinforcement Learning from Human Feedback (RLHF), which assigns scalar rewards to actions, or Nash Learning from Human Feedback (NLHF), which seeks a simultaneous-move equilibrium, SLHF leverages the asymmetry of sequential play to capture richer preference structures. The sequential design of SLHF naturally enables inference-time refinement, as the Follower learns to improve the Leader’s actions, and these refinements can be leveraged through iterative sampling. We compare the solution concepts of SLHF, RLHF, and NLHF, and lay out key advantages in consistency, data sensitivity, and robustness to intransitive preferences. Experiments on large language models demonstrate that SLHF achieves strong alignment across diverse preference datasets, scales from 0.5B to 8B parameters, and yields inference-time refinements that transfer across model families without further fine-tuning.

Published as a conference paper at ICLR 2026
STACKELBERG LEARNING FROM HUMAN FEEDBACK: PREFERENCE OPTIMIZATION AS A SEQUENTIAL GAME
Barna Pásztor* ETH Zurich
Thomas Kleine Buening ETH Zurich
Andreas Krause ETH Zurich
ABSTRACT
We introduce Stackelberg Learning from Human Feedback (SLHF), a new frame- work for preference optimization. SLHF frames the alignment problem a sequential-move game between two policies: a Leader, which commits to an action, and a Follower, which responds conditionally on the Leader's s action. This approach decomposes preference optimization into a refinement problem for the Follower and optimization problem against adversary for the Leader. Unlike Reinforcement Learning from Human Feedback (RLHF), which assigns scalar rewards to actions, Nash Learning from Human Feedback (NLHF), which seeks simultancous- move equilibrium, SL.FF leverages the asymmetry of sequential play to capture richer preference structures. The sequential design of SL.FF naturally enables inference-ting refinement, as the Follower learns to improve the Leader's actions, and these refinements can be leveraged through iterative sampling. We compare the solution concepts of SLHF, RLHF, and NLHF, and lay out key advantages in con- sistency, data sensitivity, and robustness to intransitive preferences. Experiments on large language models demonstrate that SLHF achieves strong alignment across diverse preference datasets, scales from 0.5B to SB parameters, and yields inference- time refinements that transfer across model families without further fine-tuning.
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) has emerged as the dominant paradigm for aligning Large Language Models (LLMs) with human preferences (Casper al., 2023; Kaufmann et al., 2023). The standard pipeline involves two stages: first, reward model is trained on a dataset of pairwise human comparisons, and second, a policy optimized via reinforcement learning to maximize this reward (Christiano et al., 2017; Ouyang et al., 2022). Despite its empirical success, RLHF relies on a critical assumption that diverse human preferences can be faithfully represented by single real-valued reward function. In practice, this assumption often fails as scalar reward models cannot capture intransitive preference structures. Even when preferences are transitive, widely used formulations such as the Bradley-Terry model (Bradley and Terry, 1952) yield learned rewards that diverge from the underlying preferences (Bertrand et al., 2023).
common alternative reward models and the Bradley-Tere assumption preference models which directly model pairwise preferences (Jiang et al., 2023). However, when preferences exhibit cycles, optimality becomes ill-defined because no single policy can dominate all others. Nash Learning from Human Feedback (NLHF) proposes the Nash Equilibrium (NE) as solution to this problem by framing preference optimization as a two-player simultaneous-move game, where the Nash equilibrium (NE) corresponds to typically stochastic policy whose actions are preferred to any other policy's actions on average (Munos et al., 2024).
We expand on this game-theoretic perspective and introduce Stackelberg Learning from Human Feedback (SIHF), which models alignment as a sequential-mon game between Leader and a Follower inspired by Stackelberg dynamics (Stackelberg, 1952). In SL.F,, a Leader first commits to an action, and a Follower then responds conditional on the Leader's choice. This asymmetry yields two key advantages. First, the Follower solves refinement problem rather than optimizing directly against a non- -tatsoati opponent and unobserved actions. This leads to more stable learning and
"Correspondence ba pasztorêai. .e

---

Published as a conference paper at ICLR 2026
quicker adaptation to the changes in the Leader's policy. Consequently, this faster rate of learning yields a more stationary feedback to the Leader that can anticipate the Follower's ' refinement more accurately and choose actions that are robust to subsequent improvements.
Perhaps more importantly, SLHF provides a principled method for inference- refinement: the ability improve model outputs at inference-time via repeated sampling. This is particularly valuable when the target preferences change between training and inference-timee Most commonly, models are trained on preferences aggregated across diverse annotators that might induce intransitive preference cycles (Section 4). However, at inference-time, outputs ultimately have to align with an individual's taste. SLHF realizes this refinement through its two components: the Leader policy produces an initial response, and the Follower policy generates refined responses conditional on the previous output. Unlike sampling from a static distribution, this produces a sequence of outputs that can efficiently explore the preference space. Crucially, this allows for performance gains through inference-tine computation alone, without any need for additional training or external feedback.
In summary, our contributions are as follows:
We introduce Stackelberg Learning from Human Feedback (SLHF), a preference optimization framework that models alignment as two-player sequential game. We formalize this game over a learned pairwise preference model and show that SL.FF admits a unique Stackelberg equilibrium under standard regularity assumptions (Section 4).
an algorithm that approximates the Stackelberg equilibrium via two-timescale gradient descent ascent. Our algorithm benefits from online RL optimization with- out the need of an explicit reward model or expensive inference with mixture policy (Section 5).
We propose
Our experimental results show that the Follower, conditioned on the Leader's output, consistently outperforms both RLHF and NLHF baselines, whereas the Leader performs similarly to the approximated Nash policy. Furthermore, we show that the Follower generalizes across models, improving outputs from independently trained policies without additional fine-tuning (Section 6).
2 RELATED WORK
Reinforcement Learning from Human Feedback (RLHF). RLHF optimizes policies using human preferences expressed through pairwise comparisons or rankings rather than explicit numeric rewards (Wirth et al., 2017; Kaufmann et al., 2023). The standard pipeline, introduced by Christiano et al. (2017), trains a reward model from human comparisons and then treats this model as proxy reward for policy optimization, typically using PPO (Schulman et al., 2017). This framework has driven progress in text summarization (Stiennon et al., 2020), question answering (Nakano et al., 2021; Menick et al., 2022), and large language model fine-tuning (Ziegler et al., 2019; Bai et al., 2022; Glaese et al., 2022; Ouyang et al., 2022). Recent work integrates reward and policy updates into a bilevel optimization loop (Shen et al., 2024; Thoma et al., 2024; Makar- Makar-Limane Limanov et al., 2024), but the reliance on a real-valued reward model remains. In particular, SGPO (Chu et al., 2025) considers Stackelberg formulation between policy and an adversarial preference distribution. In contrast, our work frames preference optimization as sequential-mov game between two policies and does not assume transitive preferences.
Limitations of Reward Modeling. Most RLHF implementations reduce preference learning to scalar reward estimation, typically based on the Moddley-Terry model (Bradley and Terry, 1952). While adequate for transitive, single-objective preferences, such models cannot represent intransitive structures and potentially misrank even transitive ones under model misspecification (Bertrand et al., 2023). Consequently, RLHF policies can be sensitive to the distribution of training compar- isons (Munos et al., 2024) and prone to mode collapse under continued optimization (Xiao et al., 2024). Intransitive preference cycles have been observed not only in human feedback (Duan et al., 2017: Alós-Ferrer et al., 2022; Casper et al., 2023) but also in L.M--genrrteed annotations (Dubois et al., 2024; Xu et al., 2025). Our approach sidesteps these issues by optimizing directly over pairwise preferences without imposing a scalar reward model.
Preference Optimization. To address the limitations of reward modeling in RLHF, IPO (Azar et al., 2023) extends Direct Preference Optimization (DPO) (Rafailov et al., 2023) by optimizing for the win rate against a reference policy. Nash Learning from Human Feedback (NLHF) casts the learning
2

---

Published as a conference paper at ICLR 2026
problem as a two-player simultaneous- move game and introduces NASH-MD-PG and NASH-EMA- PG to approximate the Nash Equilibrium (NE) of a learned preference model via mirror descent (Munos et al., 2024). Subsequent work has extended this perspective, proposing various algorithms to optimize for (approximate) NE, including ONLINE- IPO (Calandriello et al., 2024), SPPO (Wu et al,, 2024), SPO (Swamy at.,,, 2024), INPO (Zhang et al., 2025), DNO (Rosset et al., 2024), RSPO (Tang et al., 2025), NASH-RS (Liu et al., 2025) and MPO (Wang et al., 2025); sometimes with strong last-iterate convergence guarantees, e.g., EGPO (Zhou al., 2025). Because simultaneous games are symmetric, these methods typically converge to mixed strategies unless one option is overwhelmingly preferred (Liu et al., 2025). In contrast, SLHF models alignment as a sequential Stackelberg game in which Leader commits first and a Follower responds conditionally. This asymmetry yields different solution concept that can admit deterministic equilibria in the non-regularized limit.
Inference- Time Preference Improvement. Improving the capabilities of LLMs through additional computation at inference time has recently received significant attention, especially in verifiable domains such as coding or mathematics (Welleck al., 2024). Closest to our work are self-correction algorithms that aim to improve their responses without external feedback at test-time. A natural approach to self-correction is to provide instructions only without further training, which, however, can lead to performance degradation (Huang et al., 2024; Zheng et al., 2024; Tyen et al., 2024; Qu et al., 2024). Other work on training models for self-correction either assumes human or AI revisions (Saunders et al., 2022; Qu et al., 2024) or a reward function scoring responses (Welleck et al., 2023; Akyurek et al., 2023; Zhang et al., 2024; Kumar et al., 2025). Similarly to SLHF, Kumar et al. (2025) also propose to train an LLM in sequential manner, however, assume reward model and train in two-stages instead of a single loop. SLHF provides a unified alternative: its Leader-Follower structure naturally supports inference-time refinement through iterative sampling, enabling self-improvement on arbitrary preference signals without auxiliary reward models or multi-stage procedures.
3 PROBLEM STATEMENT
We consider a preference optimization problem over a finite set of contexts X' and actions y. The contexts are drawn from fixed and known distribution P E Ax, where Ax is the probability sim- plex over X. A policy π. : X - Ay maps each context x E X to a discrete probability distribution T( I)) Ay. where Ay is the probability simplex over y. We let I1 = {x: X' + Ay} denote the set of all policies. In the language modeling setting, X' typically models the set of prompts. y the candi- date responses, and is the LLM that defines conditional distribution over responses given prompts.
Let the preference function p(y > y' x) define the probability that y is preferred over y/ given x. We adopt the convention of writing >~ y when p(y > y x) 1/2. Slightly overloading notation, the preference between two policies π and n' given context x is defined as
(1) There are two common approaches to implementing the preference function p in practice. Let D {(r,, yf" b be a preference dataset, where y/"" and y; denote the chosen and rejected actions in a pairwise comparison, respectively. One approach is to frame this as binary classification problem on D and train a parametrized model to estimate (Jiang et al., 2023). Alternatively, in the language mod- cling setting, one could directly employ trained models to provide feedback by following instructions without additional training. This method is often referred to as LLM-as-a-judge (Gu et al., 2024).
p( > T'| x) (p-=((()))))- [p(y > 5 |=)].
The core objective of preference optimization is to identify a policy that consistently generates optimal or highly-preferred preferred responses. The notion of an "optimal" policy is straightforward when preferences are transitive. In such a case, for a given context z E X,, there exists an action y E Y' such that yo >~ Condorcel winner for x and represents the top element of the
for all y E y. This action is known induced total order. every context x E admits a Condorcet winner, the optimal policy is simply T (z) yr. However, real preference data often contains cycles other intransitivities, so Con- dorcet winner may not exist and policy optimality becomes ill-defined. To cope with such ambiguity, prior work adopts different solution concepts, the two most common of which we briefly review below.
3.1 BACKGROUND ON EXISTING SOLUTION CONCEPTS AND APPROACHES
Reinforcement Learning from Human Feedback (RLHF). RLHF as proposed by Christiano et al. (2017) and adapted to language modeling by Ziegler al. (2019) splits preference optimization into
3

---

Published as a conference paper at ICLR 2026
two steps. First, it assumes that the preference function p follows the Bradley-Tery model (Bradley and Terry, 1952) so that
ply > y/ =) or( -
(2)
where or (x) Ttexp(-m the sigmoid function and r X' R real-valued reward function.
The reward function is unknown so that an estimator f is used that maximizes the log-likelln log-likelihood of the dataset 2), In a second step, the policy лt" is chosen to maximize the expected reward with respect
to regularized by the Kullback-eeibe (KL) divergence against fixed reference policy o II:
[ (Eynat [f(r,, w)] (K mT)))
' E arg max Eaxp nell
Here, r 2 O and (T, || xdff is computed between a(. I)) and x. x)). Under the Bradley-Ters assumption, Equation (3) admits a unique closed-form solution (Rafailov al., 2023). However, additive score models like Bradley-Terry are provably limited in expressing cyclic intransitive preference structures, which have been empirically observed in both strategic games (Bertrand et al., 2023) and human preference data (Duan et al., 2017; Consequently, the optimal policy x² depends critically on the data distribution the training set DD especially its sampling biases (Munos al., 2024), which we elaborate more on in Section 4.1.
al., 2022: Casper et al., 2023).
Nash Learning from Human Feedback (NLHF). NLHF avoids explicit reward modeling by framing preference optimization as a two-player simultaneous game between two policies π, x' E IT (Munos et al., 2024). The optimization problem is given by:
max min Eusp p( > x 1x) - ( || ( x r'ell
TKL (TT' m
(4)
The solution to Equation (4) is a Nash equilibrium (x* 1=",,² T'"), where neither side can be improved unilaterally. The existence and uniqueness of this equilibrium follows from the concave-convex nature of the objective (Munos et al., 2024). NLHF can incorporate online feedback and makes no structural assumptions on preferences, but when no action majority- preferred the equilibrium necessarily involves mixed strategies (Liu et al., 2025), even in the absence of KL. regularization when r 0. This inherent stochasticity can be undesirable in applications where consistency and reliability are critical.
4 STACKELBERG LEARNING FROM HUMAN FEEDBACK (SLHF)
We now present Stackelberg Learning from Human Feedback (SLHF), a novel perspective on the preference optimization problem. Inspired by Stackelberg games (Stackelberg, 1952). cast preference optimization as a sequential-nig move game between two players: the Leader and the Follower. Given context æ, the Leader first chooses its action y x(- ar) The Follower then observes both the context & and the Leader's 's realized action y and responds with y' ~ w(( x. y). The Follower's policy chosen from the set {w X' x y + Ay} which allows conditioning on both the context and the Leader's action. Formally, given reference policies N- E II and we e Ω, the optimization problem is defined follows:
max min Enwn (xy (B) v/ +]]++*]K KL KL - (TT m (5) TEl WEst
where Th, FF 2 0 are player-specific regularization coefficients. We let f(T, ((T denote the objective of Equation (5), which, in the absence of regularization, defines a sequential-move constant-sum game.
SLHF decomposes the preference optimization into two complementary roles, setting it apart from single-policy methods like RLHF and NLHF., The Follower leverages its informational advantage of observing the Leader's committed action. This simplifies its task to learning specialized refinement policy that finds the best response known output, rather than optimizing against non-stationary opponent. The Leader, anticipating this refinement, learns to produce initial actions that remain strong even after the Follower's refinement. In Section 4.1, illustrate that when preferences form a cycle and no Condorcet winner exists, the Leader selects the least exploitable action, while the Follower traverses the preference cycle, covering all plausibly optimal actions with minimal samples.
The formulation in (5) differs from standard Stackelberg settings (Conitzer and Sandholm, 2006), where the Follower gets condition on the Leader's policy æ only, not the realized action y.
4

---

Published as a conference paper at ICLR 2026
Table 1:
Transitive individual annotator prefer-
Table The preference function p induced by the population in Table 1.
ences over three options {A, B, C).
Type
Preference Relationship
Proportion
A
B
C
a1
> B C B C A
a1
A
0.5
1
01
a2
Q2
B
02
0.5
1 03
a3 C
03
C
1-
-1
03
0.5
Allowing the Follower to observe y provides strictly more information whenever v is stochastic, yielding simpler and stationary best response problem. In this setting, the Leader gains no advantage from randomizing, i.c., playing a stochastic policy.
In line with previous results that the RLHF problem (3) admits a closed-form solution, show that there exists a unique solution to the SLHF problem (5). The proof is deferred to the Section A.I.
Proposition 1. Let , >> O and suppose that 7t x) > 0 for all (x, y) E X x y. For any preference function ply > yf' x) there exists unique solution (x*, ,w") to the preference optimization problem in Equation (5).
The solution (T*,w** is called a Stackelberg equilibrium. It is folklore in the algorithmic game theory literature that there exists deterministic Stackelberg equilibrium when the Leader's realized action is observed by a best responding Follower, as there always exists deterministic best response for the Follower. Thus, there is no point in randomizing for the Leader. This stands contrast to the NE, which is in general stochastic. For completeness, we provide proof in Appendix A.2.
Remark 2. For any preference function ply > v/ x), the SLHF optimization problem (5) has deterministic solution (w")"") whenever T = = 0. Note that this solution may not necessarily be unique due the lack of regularization.
4.1 COMPARISON SOLUTION CONCEPTS
Before describing how to approximate the Stackelberg equilibrium, we first contrast RLHE, NLHF, and SLHF in the Condorcet paradox (de Caritat Mis et al., 1785) described below. Consider setting with single context | and three candidate actions y = (A, B, C) Let the preference function be given by the aggregate over the population of annotators, A = {a1. 02 as}, defined in Table 1. Each type of annotator has strict preference ranking over y and we aggregate their preferences as
ply > (/>>)==> Σ ailyy > v}.
(6)
where 1 1/y >ay y'} is if y is preferred over y' by the annotator type a, and O otherwise. Table shows the aggregated preferences of the whole population. For example, p(A > B) is the probability that randomly chosen annotator prefers A B. This is true for annotator types ai and as (from Ta- ble 1), who make up a proportion on + of the population. Since (1 +02 03 = 1, this is equivalent a22 shown in Table 2. common example a cyclic relationship between three actions where A > B > C' but C > A. Hence, there exists no Con- dorcet winner in this case. More generally, the interesting case is given by a1, Q2 03 < 1/2, and this example is often referred to as the Condorcet paradox, because the annotators individually have tran- sitive preferences (Table 1), but their aggregated preferences form cycle (Table 2). For ease of pre- sentation, we consider a non-regularized problem in the rest of this section so that 7 = 7 ,f 0.
choose a a2 a3 1/3, which leads to
+ I depends on the sampling distribution of the dataset D used to estimate Suppose D contains only comparisons {A, B) and C), but not {A, C). Because A B and B > C' for all annotators, maximum-likelihood estimation, which fits a single underlying transitive reward function, yields 1. However, different sampling patterns (e.g., omitting (A, B}) can instead favor B or C. This illustrates key limitation of RLHE, as its solutions are sensitive to the specific comparisons present in D.
RLHF Solution. Our first observation is that the estimated reward function F ;: X'
F(A) > f(B) f(C), the optimal policy is n (AA
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 STACKELBERGGDA
procedure STACKELBERGGDACK, Y: n""
2: Initialize the Leader and Follower policies and w1
3: for i 1,, do
4:
Update Leader's policy: M++1 + V (T....)
5:
Update Follower's policy: Wi41 wi - 77 V (T),W)
6: Project π++1 and w1+1 to their respective probability simplices
7: end for
8:
end procedure
Nash Equilibrium. (B) = -1- 201, (C) =1 20y. In the special case of a1 = over V, i.e., it has the highest possible entropy. Unlike RLHF, this solution is inneeettindeeeedentt but it produces fully stochastic policy that may be undesirable in applications requiring decisive outputs.
The NE of the matrix game defined in Table 2 is given by T" (A) 2003.
1/3, the NE is uniform
Stackelberg Equilibrium. In SLHF, the players' sequential roles resolve the cycle. The Follower's optimal strategy is straightforward as for any action presented by the Leader, it plays the best response y' that beats it (i.e., C if y A, etc.). The Leader, anticipating this deterministic best response, chooses an initial robust action. This leads to the following equilibrium policies:
C y A w.p.
A if max a3) w.p. 1
w-(-1v)
A y = w.p.
x²))==
B
if 02 max 03) w.p.
B if y = C w.p.
if max(a 02) w.p.
When 01 02 0g = 1/3. the Leader is indifferent, and any distribution over A, B, C' (including the uniform NE) valid Stackelberg equilibrium. Unlike RLHF, this solution requires no offline dataset, and unlike NLHF, it admits a deterministic Leader and Follower policy when one type dominates.
Inference-Tinee Refinement. Motivated by applications such as text summarization, open-ended generation, and audio-visual content creation, where users can reject outputs and request new samples, we introduce the notion of inference-time refinement for preference optimization. At inference-time, single user interacts with the model and may resample actions until receiving one that matches their preference, analogous to the pass@k metric in verifiable domains. This is non-trivial because models are usually trained to reflect population preferences, yet deployment requires adaptation to an individual user.
Consider the symmetric case 01 022 = 0g = 1/3, and without loss of generality, let the user be of type di with ranking A > B > C. RLHF may return A, but depending D it could also output B or C, and repeated sampling offers no recourse. The NLHF solution uniform over A, B, C, so probability of sampling A in single draw is 1/3. By sampling N times, the probability of observing at least one similarly: the first action is sampled from the Leader's possibly uniform policy but subsequent actions are drawn from the Follower's policy, i.e., ~ w²( (. x. 1i-1) for 2 2. Following this structure, the entire
(2/3)** i.e.. 56% for = 2 and 70% for N = 3. The SLHF solution starts
the probability of sampling A within N = steps increases 67%. and for N preference cycle is traversed regardless of the Leader's s initial choice. Note that the SLHF solution supports this refinement procedure without the need of additional training required inference-time.
User: <user possppprompt>>>
Assistant: <leader <leader_responser response>
User: <user_prompt>>
User: Improve the previous answer!
Assistant:
Assistant:
(a) Prompt received as the Leader agent
(b) Prompt received as the Follower agent
Figure 1: Prompt templates used to train single-model for both Leader and Follower completions.

---

Published as a conference paper at ICLR 2026
Table Pairwise preference comparisons between the responses QWEN2.5-0.5B. RLOO, NASH-
MD-PG, and STACKELBERG algorithms. Each cell represents the preference model's average score for the row algorithm over the column algorithm.
QWEN2.5-0.5B
RLOO
NASH-MD-PPG
STACKELBERGGDA
LEADER
FOLLOWER
QWEN2.5-0,58
0,000
0,407
0.279
0,266
0,200
RLOO
0.593
0.000
0.393
0.387
0.344
NASH-MD-PG
0.721
0.607
0,000
0,497
0.406
STACKELBERGGDA-LEANE
0.734
0.613
0,503
0,000
0.395
STACKELBERGGGI FOLLOOWRRR
0.800
0.656
0.594
0.605
0.000
STACKELBERG GRADIENT DESCENT ASCENT (STACKELBERGGDA)
We now introduce STACKELBERGGDA, two-timescale Gradient Descent-Ascent (GDA) algorithm designed for the sequential- movee preference optimization problem in Section 4. performs simultaneously gradient ascent and descent update steps on the Leader and Follower policies, and w, with step size 17' and y², respectively, to find the max min solution to f w) defined in Equation (5). It two-timescale algorithm as we choose of >> rf resulting in w adapting faster than æ. We denote the two-timescale coefficient as = " / After each update, both policies are projected back onto their respective probability simplices to ensure feasibility.
The function f(T, w) is concave in and convex in w. While standard gradient descent-ascent with equal learning rates has ergodic convergence guarantees in this setting (Korpelevich, 1976; Chen and Rockafellar, 1997; Nemirovski, 2004; Auslender and Teboulle, 2009; Nedić and Ozdaglar, 2009), we instead adopt a two-timescale variant. This choice is motivated by its stronger convergence guarantees in more general nonconvex-comeave regimes (Lin et al., 2025), well as its empirical success in both Actor-Critic methods (Prasad et al., 2015) and the training of Generative Adversarial Networks (Heusel et al., 2017). This becomes especially valuable for the practical implementation of STACKELBERGGDA for large state and action spaces and parameterized policies below.
Scalable Implementation of STACKELBERGGDA for LLM Fine-Tuning Direct optimization the full policy spaces II and Ω is infeasible when X' and y are large, in LLM fine-tuning, To address this challenge, we parametrize π and w and estimate gradients from batches. Crucially for LLM fine-tuning, the Leader and the Follower can share the same parametrization by using the prompt template shown Figure/ 1, which allows us to reduce the memory requirements. Additionally, framing the Leader and the Follower policies as multi-turn dialogues enables us use any policy trained for multi-turn conversations as both -t and we All implementation details and pseudocode provided in Section B.
EXPERIMENTS
We conduct a series of experiments to validate the Stackelberg formulation and the efficacy of STACKEL Our evaluation is designed to answer three primary questions:
1) How does STACKELBERGGDA compare against established RLHF and NLHF baselines in controlled preference optimization task?
2) Can the Leader-Follower structure SLHF enable effective inference-time refinement, and does this capability generalize to improving outputs from other models?
3) Does the approach scale effectively to the large-scale, general-purper purposs fine-tuning of LLMss?
Section 6.1 addresses the first two questions by aligning models on a dataset with diverse human preference signals. In the appendix, also provide further results iterative improvements with increased inference-time computation (Section D.2), ablations of the hyperparameter x (Section D.3),
'This follows from Munos al. (2024). For completeness, we provide formal proof in Section A.3, and discuss the convergence hehavior and limitations of BTACKELBERGGDA Section A.4.

---

Published as a conference paper at ICLR 2026
Table 4: Test-time improvement using different models for the initial response (Leader) and the improvement (Follower). Each cell represents the preference model's average score for the Follower's responses over the Leader's responses.
Leader
QWEN2.5-00 5.
RLOO
NASH-MD
STACKELBERGGDA
Follower
QWEN2.5-0.5B
0,549
0,443
0,363
0.362
RLOO
0.534
0.403
0,369
0.360
NASH-MD--PG
0.708
0.600
0.493
0.476
STACKELBERGGDA
0.803
0.665
0.600
0.606
and additional scaling results (Section D.4). Section 6.2 then tackles the third question by applying STACKELBERGGDD within large-scale, open-source post-training pipeline.
6.1 EMPIRICAL COMPARISON OF SOLUTION CONCEPTS
Dataset. We use the HELPSTEER2 dataset (Wang et al., 2024), which contains 11,826 human- annotated single-turn dialogues, to estimate the preference function p and its prompts during the training loops. We choose this dataset due to its high-quality human annotations along five attributes (helpfulness, correctness, coherence, complexity, and verbosity) that allows us estimate diverse preference profile. Further details on the preference model specification and the resulting intransitivity provided in Section D.I.
Compared Methods. We compare STACKELBERGGDA with RLOO (Ahmadian et al., 2024) and NASH-MDD-G (Munos et al., 2024) which represent the RLHF and NLHF frameworks, respec- tively. We use these baselines because they are well-established and come with robust, well-tested open-source implementations. This ensures that comparison reflects differences between the frameworks rather than implementation details. All models are fine-tuned from the QWEN2.5- 0.5B² model and run for 1,000 gradient steps with batch size of B 32. We sweep learning rates E f1c-6, 5c-6. 6, le 1c-5} and KL penalties т E (0.001, 0.01, 0.1] for all algorithms. For NASH-MD-PG. additionally vary the mixture parameter 3 0.25, 0.5, 0.75, 1), and for STACKELBERGGDA the two-timescale coefficient E {1, 5, 10). Models are selected by aver- age preference over QWEN2.5-0.51 yielding best setting y) le- and 0.001, with 3 = 0.75 for NASH-MD-PE and = 5 for STACKELBERGGDE
All implementations use the Transformers (Wolf et al., 2020) and TRL (von Werra al., 2020) libraries, with the AdamW optimizer (Loshchilov and Hutter, 2019).
6.1.1 ROUND-ROBIN TOURNAMENT
Table reports pairwise preference scores between the initial QWEN2.5-0.5B and the three fine-tuned models. The first responses of
and NASH-MD-PC, achieve roughly 73% preference over QWRN2.5 500.BB and 61% over RLOO., while tying at 50% when compared to each other. This outcome aligns with settings where multiple high-quality responses exist and the Stackelberg and Nash equilibria coincide (Section 4.1).
Crucially, applying the FOLLOWER of STACKELBERGGDA yields a marked performance gain. achieves 80% preference over QWEN2.5-00 -B.5BB 66% over RLOO, 60% over NASH-MD MD-PG, and even outperforms the responses it was conditioned in 60.5% of comparisons. Thus, two-turn inference procedure provides substantial gains at the cost of single additional generation.
improve its own initial responses
6.1.2 INFERENCE- TIME REFINEMENT
We further evaluate each model's ability to act as a Follower, refining outputs from other mod- els. Although only STACKELBERGGDA is explicitly trained for this task (and only best
Phtpps: hhttps://hugiinggfacee com Qwen2. 5-0-58- rust
8

---

Published as a conference paper at ICLR 2026
Table 5: AlpacaEval 2.0 results comparing trained models GPT-4 TURBO. Length-controlled (LC)
winrate alleviate the length bias of the GPT-4 judge.
Model
LC Winrate
Winrate
GPT-40-2002-05-13
37.46
51.33
1AAM--3.1-TULU---- BPO
33.37
40.15
STACKELDERRGDDA-FOLLLWWEE
28.89
61.58
STACKELBERGGDAA-LEADDEE
27.09
49.24
METAA-LLAMA-331-88--NN
20.85
21.84
LLAMA-3.1-TULU-33-8881
8.83
14.26
respond itself), apply the same refinement procedure all models test their abil- ity to generalize. Specifically. for every pair of Leader and Follower models selected from QWEN2.5-0.5B. RLOO, NASH-MD-PPG SAA we first generate response with the selected Leader and then apply the Follower prompting template (Figure I(b)) to produce a potentially improved response. We refer these as the Leader and Follower outputs, respectively. Exhaustively evaluating all Leader-Follower inference-time refinement under diverse initial conditions. Table 4 reports the resulting preference scores, which indicate how often the Follower output is preferred over the Leader's generation.
pairs allows to measure each model's capacity for
STACKELBERGGDA consistently improves across all Leader models; most notably over QWEN2.55 0.5B and RLOO, while achieving gains of up to 60% even when refining outputs from NASH- MD-PG or itself. In contrast, QWEN2.5-0. OWEN2.5-0.5B and often degrade the quality of outputs from other Leaders. NASH-MD-PG enhance responses from 1.E.2.5-0.5B and RLOO, but its 70% preference score over QWEN2.5- 0.5B still falls short of its 73% self-improvement rate reported in Table 3. These findings extend prior work on verifiable domains (Huang et al., 2024; Zheng et al., 2024; Tyen et al., 2024; Qu et al., 2024) by showing that explicitly training improve given outputs is crucial and mere instruction prompting is insufficient to reliably enhance responses with respect to human preferences.
and RLOO only improve upon responses from
6.2 GENERAL PURPOSE FINE-TUNING
To evaluate STACKELBERGGDA for large-scale LLM fine-tuning in general chat applications, we adopted the Tulu 3 prsi-ngai pipeline (Lambert et al., 2024). Using prompts from its preference dataset", we trained the 8B- parameter Supervised Fine-Tuned (SFT) checkpoint', denoted LLAMA- 11-TULUU---BB-STTT
Responses from the resulting Leader and Follower policies were evaluated by the META-LLLAM-3.. 1...M----31------ 1--70-IINTRRCG model (Weerawardhena et al., 2025), used
with
automatic preference judge. Final models benchmark shown to approximate human judgments (Dubois et al., 2024). We report both the standard win rate and the Length- (LC) win rate, the latter designed mitigate length bias in LLM judge evaluations.
evaluated on AlpacaEval 2.0,
As shown in Table 5, both STACKELBERGGDA-LEADEE and STACKELBERGGDA- FOLLOWWR substantially improve the win rates of the initial model and outperform MLETA-LLAM--3 8B-INSTRUCT, which shares the same base model. While LLAMA-3. 1-TuT.U-3-8B-DDOO and GPT-40-2024-05--1 achieve higher LC win rates, STACKEL attains superior standard win rates surpassing even GPT-40-2022-05-13, the top-performing model on the public leaderboard. Notably. the same model used for AlpacaEval 2.0's evaluation, which may inflate its LC win rates and its true performance is likely closer to StaCkELBERGGD) 's. We attribute the gap between LC and standard win rates for STACKELBERGGDA to META-LLAMA- 3.10--3.17000 77B--INSTUCC'S length bias and consider refining the feedback source promising avenue for future work.
was trained on outputs generated by GPT-4,

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
We introduced Stackelberg Learning from Human Feedback (SLHF), a two-player sequential-move framework that directly optimizes pairwise preference signals without requiring real-valued reward models. We proposed librium and scale challenging tasks such as aligning LLMs with human preferences. Empirically, STackelberGGDA 's Leader policy matches or exceeds standard baselines while the Follower policy consistently improves outputs at inference time, even when paired with models it was not trained with.
to efficiently approximate the unique Stackelberg equi-
Limitations. Similarly to NLHF, key limitation of our approach is its reliance on well-specified and representative pairwise preference function, which can be challenging to obtain in open-ended or under-specified domains. Moreover, although the sequential formulation enables inference-time refinement through conditional generation, it currently operates without real-time user interaction. Future work could integrate active preference elicitation and personalized refinement, allowing SL.HF to adapt dynamically to individual user preferences at test time. Finally, BTACKELBERGGDD currently has ergodic but not last-iterate guarantees; developing SLHF algorithms with last-iterate convergence (c.g., via extragradicnnoopimisse or mirror-prox is an open direction.
ACKNOWLEDGEMENTS
This work was primarily supported by the ETH AI Center through an ETH AI Center Doctoral Fellowship Barna Pásztor and an ETH AI Center Postdoctoral Fellowship Thomas Kleine Buening. Thomas Kleine Buening was additionally supported by the EPSRC Prosperity Partnership FAIR (grant number EP/V056883/1. We thank Marian Schneider for his support in the experiments technical setup.
REFERENCES
A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In L.-W. Ku, A. Martins, and V. Srikumar, editors. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2024.
A. F. Akyurek, E. Akyurek, A. Kalyan, P. Clark, D. T. Wijaya, and N. Tandon. RL4F: Generating natural language feedback with reinforcement learning for repairing model outputs. In A. Rogers, Boyd-Graber, and N. Okazaki, editors, Proceedings of the 6l Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), 2023.
C. Alós-Ferrer, E. Fehr, and M. Garagnani. Identifying nontransitive preferences. Technical report, Working Paper, 2022.
A. Auslender and M. Teboulle. Projected subgradient methods with non-euclidean distances for non-differentiable convex minimization and variational inequalities. Mathematical Programming, 120:27-48, 2009.
M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. A General Theoretical Paradigm to Understand Learning from Human Preferences. Proceedings of Machine Learning Research, 238:4447-444 4455, 10 2023.
Y. Bai, Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 20iv:2204.05800 2022.
A. Beirami, A. Agarwal, J. Berant, J. Eisenstein, C. Nagpal, A. Theertha Suresh, G. Research, and G. DeepMind. Theorctical guarantecs on the best-of-n alignment policy. arXiv preprint 2rX1v:2401.01879 2024.
Q. Bertrand, W. M. Czarnecki, and G. Gidel. On the limitations of the elo, real-world games are transitive, not additive. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, 2023.
10

---

Published as a conference paper at ICLR 2026
R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324 345,,,,, 1952.
D. Calandriello, D. Guo, R. Munos, M. Rowland, Y. Tang, B. A. Pires, P. H. Richemond, C. Le Lan, M. Valko, T. Liu, R. Joshi, Z. Zheng, and B. Piot. Human Alignment of Large Language Models through Online Preference Optimisation. In Proceedings of the 41st International Conference on Machine Learning, 2024.
S. Casper, X. Davies, C. Shi, T. K. Gilbert, Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. Wang, S. Marks, C.-R. Segerie, M. Carroll, A. Peng, P. Christoffersen, M. Damani, S. Slocum, Anwar, A. Siththaranjan, M. Nadeau, E. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. BIyık, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield- Menell. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. arXiv preprint 2.0.v::2337./52117.. 7 2023.
G. H. Chen and R. T. Rockafellar. Convergence rates in forward-backwart splitting. SIAM Journal on Optimization, 7(2):221 1997.
F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep Reinforcement Learning from Human Preferences. In Proceedings of the 31st Conference on Neural Information Processing Systems, 2017.
X. Chu, Z. Zhang, T. Jia, and Jin. Stackelberg game preference optimization for data-efficient alignment of language models, 2025.
V. Conitzer and T. Sandholm. Computing the optimal strategy commit to. Proceedings of the ACM Conference on Electronic Commerce, 2006:82-90, 2006.
J. A. N. de Caritat Mis et al. Essai sur I' l'application de l'analyse à la probabilité des décisions rendues la pluralité des voix. Imprimerie royale, 1785.
Duan, J. Li, Y. Baba, and H. Kashima. A generalized model for multidimensional intransitivity. In J. Kim, K. Shim, L. Cao, J.-G. Lee, X. Lin, and Y.-S. Moon, editors, Advances in Knowledge Discovery and Data Mining, pages 840-80 852, Cham, 2017. Springer International Publishing. ISBN 978-3-319-57529...
Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. 2024.
J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D' Amour, D. Dvijotham, A. Fisch, K. Heller, S. Pfohl, D. Ramachandran, P. Shaw, and J. Berant. Helping herding? reward model ensembles mitigate but do not eliminate reward hacking, 2024.
M. Geist, B. Scherrer, and O. Pietquin. A Theory of Regularized Markov Decision Processes. In Proceedings of the 36th International Conference on Machine Learning, 2019.
A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint (XXX::220991137555 2022.
Gu, X. Jiang, Shi, H. Tan, Zhai, C. Xu, W. Li, Y. Shen, Ma, H. Liu, Y. Wang, and Guo. A Survey on LLM-as-a-Judge. 11 2024.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Proceedings of the Thirty-first International Conference Neural Information Processing Systems, 2017.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Yi, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
11

---

Published as a conference paper at ICLR 2026
J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou. Large language models cannot self-correct reasoning yet. In Proceedings of the Tweifth International Conference on Learning Representations, 2024.
D. Jiang, X. Ren, and B. Y. Lin. LLM-BLENDER. Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, volume 1, 2023.
T. Kaufmann, Weng, V. Bengs, and E. Hüllermeier. Survey of Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2312. 14W.232..14425, 2023.
G. M. Korpelevich. The extragradient method for finding saddle points and other problems. Matecon, 12:747-756, 1976.
A. Kumar, V. Zhuang. R. Agarwal, Y. Su, D. Co-Reyes, A. Singh, K. Baumli, S. Iqbal. Bishop, R. Roelofs, I.. M. Zhang, K. McKinney, D. Shrivastava, C. Paduraru, G. Tucker, D. Precup, F. Behbahani, and A. Faust. Training language models to self-correct via reinforcement learning. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025.
N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang. J. Yang. R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi. Tulu 3: Pushing Frontiers in Open
Language Model Post- Dast-Training. 11 2024.
B. Y. Lin, A. Ravichander, X. Lu, N. Dziri, M. Sclar, K. Chandu, C. Bhagavatula, and Y. Choi. The unlocking spell base LLMs: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations, 2024.
T. Lin, C. Jin, and M. I. Jordan. Two-timescale gradient descent ascent algorithms for nonconvex minimax optimization. Journal of Machine Learning Research, 26(11):1-44) 2025.
K. Liu, Q. Long, Shi, W. Su, and J. Xiao. Statistical impossibility and possibility of aligning Iims with human preferences: From condorcet paradox to nash equilibrium. arXiv preprint arXiv:2503. 10990. 2025.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proceedings of the Interna- tional Conference on Learning Representations, 2019.
J. Makar-Limanov, A. Prakash, D. Goktas, N. Ayanian, and A. Greenwald. Sta-rlhf: Stackelberg aligned reinforcement learning with human feedback. Coordination and Cooperation for Ageti- Reinforcement Learning Methods Workshop, 2024.
J. Menick, M. Trebacz, V. Mikulik, J. Aslanides, F. Song, M. Chadwick, M. Glaese, S. Young, L. Campbell-GGllinghann G. Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.1 /144 2022.
R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, D. Guo, Y. Tang, M. Geist, T. Mesnard, A. Michi, M. Selvi, S. Girgin, N. Momchey, O. Bachem, D. J. Mankowitz, D. Precup, B. Piot, and G. Deepmind. Nash Learning from Human Feedback. In Proceedings of the 41st International Conference on Machine Learning, 2024.
R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: browser-assisted arXiv:21 2021.
question-answerte with human feedback. arXiv preprint
A. Nedić and A. Ozdaglar. Subgradient methods for saddle-point problems. Journal of optimization theory and applications, 142:205-228, 2009.
A. Nemirovski. Prox-method with rate of convergence o (I/t) for variational incqualities with lipschitz continuous monotone operators and smooth convex-coneave saddle point problems. SIAM Journal on Optimization, 15(1):229-25 221,, 2004.
L. G. Openai. J. Schulman, O. Jacob, and H. Openai. Scaling Laws for Reward Model Overoptimiza- tion. In International Conference on Machine Learning, 2023. ISBN 10760v...
12

---

Published as a conference paper at ICLR 2026
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang Sandhini Agarwal Katarina Slama Alex Ray John Schulman Jacob Hilton Fraser Kelton Luke Miller Maddie Simens Amanda Askell, P. Welinder Paul Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th Conference on Neural Information Processing Systems, 2022.
H. Prasad, P. LA, and S. Bhatnagar. Two-timescale algorithms for learning nash equilibria in general- sum stochastic games. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, 2015.
Y. Qu, T. Zhang, N. Garg, and A. Kumar. Recursive introspection: Teaching language model agents how to self- improve. Advances in Neural Information Processing Systems, 2024.
Qwen. :, A. Yang, B. Yang. B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, Yang, J. Tu, J. Zhang, Yang. Yang. J. Zhou, J. Lin, Dang, Lu, K. Bao, K. Yang. L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Su, Y. Zhang. Y. Wan, Y. Liu, Cui, Z. Zhang, and Z. Qiu. Qwen2.5 Technical Report. arXiv preprint arXiv:2412. 24... 15v2, 12 2024.
R. Rafailov, A. Sharma, F. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct Preference Opti- mization: Your Language Model is Secretly Reward Model. In Advances in Neural Information Processing Systems, 2023. ISBN 2305.182900
C. Rosset, C.-A. Cheng, A. Mitra, M. Santacroce, A. Awadallah, and T. Xie. Direct Nash Optimiza- tion: Teaching Language Models to Self-Improve with General Preferences. In arXiv preprint arXiv:2404.03715, 2024.
W. Saunders, C. Yeh, J. Wu, S. Bills, L. Ouyang. J. Ward, and J. Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022.
Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170.006777 2017.
G. Sessa, R. Dadashi, L. Hussenot, J. Ferret, N. Vieillard, A. Ramé, B. Shariari, S. Perrin, A. Friesen, G. Cideron, S. Girgin, P. Stanczyk, A. Michi, D. Sinopalnikov, S. Ramos, A. Héliou, A. Severyn, M. Hoffman, N. Momchev, O. Bachem, and G. Deepmind. BOND: Aligning LLMs with Best-of-N Distillation. arXiv preprint ArXiv 14622, 2024.
Z. Shao, P. Wang. Q. Zhu, R. Xu. J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, arXiv preprint arXiv:2402.0333
2024.
H. Shen, Z. Yang. and T. Chen. Principled penalty-based methods for bilevel reinforcement learning and rlhf. In Proceedings of the 41st International Conference on Machine Learning, 2024.
H. v. Stackelberg. Theory of the market economy. Oxford University Press, 1952.
N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, Lowe, C. Voss, and A. Radford Dario Amodei Paul Christiano. Learning to summarize from human feedback. In Proceedings of the 34th Conference on Neural Information Processing Systems, 2020.
G. Swamy, C. Dann, R. Kidambi, Z. S. Wu, and A. Agarwal. A Minimaximalist Approach to Reinforcement Learning from Human Feedback RLHF/ PbRL. In Proceedings of the 41 International Conference Machine Learning, 2024.
X. Tang, S. Yoon, S. Son, H. Yuan, Q. Gu, and I. Bogunovic. thee-theo regularized self self-play alignment of large language models, arXiv preprint arXiv:2503. 000000 2025.
V. Thoma, B. Pásztor. A. Krause, G. Ramponi, and Y. Hu. Contextual bilevel reinforeement learning for incentive alignment. In Proceedings of the Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
13

---

Published as a conference paper at ICLR 2026
G. Tyen, H. Mansoor, V. Cärbune, Y. P. Chen, and T. Mak. Lims cannot find reasoning errors, but can correct them given the error location. In Findings of the Association for Computational Linguistics ACL 2024.
L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, Thrush, N. Lambert, S. Huang, Rasul, and
Q Gallouédec. Trl: Transformer reinforcement learning, 2020.
M. Wang, C. Ma, Q. Chen, L.. Meng, Y. Han, J. Xiao, Z. Zhang, J. Huo, W. Su, and Y. Yang. Mag- netic preference optimization: Achieving last-iterate convergence for language model alignment. In The Thirteenth International Conference on Learning Representations, 2025.
Z. Wang, Y. Dong, O. Delalleau, J. Zeng, G. Shen, D. Egent, J. Zhang, M. N. Sreedhar, and O. Kuchaiev. Helpsteer 2: Open-source dataset for training top-performing reward models. In Proceedings of the eight Conference on Neural Information Processing Systems, 2024.
S. Weerawardhena, P. Kassianik, B. Nelson, B. Saglam, A. Vellore, A. Priyanshu, S. Vijay, M. Aufiero, A. Goldblatt, F. Burch, Li, He, D. Kedia, K. Oshiba, Z. Yang. Y. Singer, and A. Karbasi. Llama-3. 3. coanndationai-securitye 1alllmm----sntuctt technical report, 2025.
S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi. Generating Sequences by Learning to celf-Correct. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.
S. Welleck, A. Bertsch, M. Finlayson, H. Schoelkopf, A. Xie, G. Neubig, Kulikov, and Z. Har- chaoui. From decoding to meta-generation: Inference-time algorithms for large language models. Transactions on Muchine Learning Research, 2024. ISSN 2835-8856.
R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229 256, 1992.
C. Wirth, R. Akrour, G. Neumann, and J. Fürnkranz. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-44 2017.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and M. Rush. Transformers: State-of-the-art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language System Demonstrations, 2020.
Processing:
Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang. and Q Gu. Self-Play Preference Optimization for Language Model Alignment. arXiv preprint arXiv:2405.0mm 00675, 2024.
J. Xiao, Z. Li, X. Xie, E. Getzen, C. Fang, Q. Long, and W. J. Su. On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization. arXiv preprint arXiv:2405.1 16455, 2024.
Y. Xu, L. Ruis, T. Rocktäschel, and R. Kirk. Investigating non-transitivity in llm-as-a-judge, arXiv preprint arXiv:2502. 200.22202..40044. 2025.
M. Zamani, H. Abbaszadehpeivasti, and E. de Klerk. Convergence rate analysis of the gradient descent-ascent method for convex-concave saddle-point problems. Optimization Methods and Software, 39(5):967-989, 2024.
J. Zhang, M. Hong, and S. Zhang. On lower iteration complexity bounds for the convex concave saddle point problems. Mathematical Programming, 194(1):901-935, 2022.
Y. Zhang, M. Khalifa, L. Logeswaran, J. Kim, M. Lee, H. Lee, and L. Wang. Small language models need strong verifiers to self-correct reasoning. In L.-W. Ku, A. Martins, and Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, 2024.
Y. Zhang, D. Yu, B. Peng, L. Song, Y. Tian, M. Huo, N. Jiang, H. Mi, and D. Yu. Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No- No-Regret Learning. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025.
14

---

Published as a conference paper at ICLR 2026
H. S. Zheng, S. Mishra, H. Zhang, X. Chen, M. Chen, A. Nova, L. Hou, H.-T. Cheng, Q. V. Le, E. H. Chi, et al. Natural plan: Benchmarking llms on natural language planning. arXiv preprint arXiv:2406.04552, 2024.
R. Zhou, M. Fazel, and S. S. Du. Extragradient preference optimization (egpo): Beyond last-iterate convergence for nash learning from human feedback. arXiv preprint 1rXiv:2503.09944 2025.
D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine- Muning Language Models from Human Preferences. arXiv preprint arXiv:/909.08593, 2019.
15

---

Published as a conference paper at ICLR 2026
CONTENTS OF APPENDIX
A Proofs
16
A.1 Proof of Theorem
16
A.2 Proof of Theorem 2
17
A.3 Concave-Convex Property of f
17
A.4 Comments on the Convergence of STACKELBERGGDD (Algorithm 1)
17
Scalable Implementation of STACKELBERGGDA
17
C Implementation Details
19
Additional Experimental Results
19
D.I Preference Model
19
D.2 Iterative Improvements Test-time
................................
20
D.3 Ablation on the two-timescale coefficient
22
D.4 Model Scaling
A PROOFS
A.1 PROOF OF THEOREM ]
Proof. First, assume that the Leader's policy is fixed and consider the Follower's optimization problem
min Eep. [ (e Ip(y > s' x)] + 7 KLr,, (w [
The optimization problem in (7) is equivalent to Equation (3) for the reward function r(ā, v/) ply > s/ 2) with contexts (x. y) and context distribution ~ p & æ. As result, Equation (7) has unique closed-form solution (Geist al., 2019; Rafailov et al., 2023; Azar et al., 2023) given by
w x.y) Z(x, v)
(o v) exp P/ v -11)) x))
where Z(x, Ey ( x. v) exp (+b plu' xx) is partition factor that depends only
on (z, y) and Th Hence, w can be expressed as a function of (z, y) and wel without explicit dependence on
Now, define the following reward function for the Leader's optimization problem
r(x,v) v) = Ey' (12,))) > y/ [=]]-
(8)
Note that w is unique so that r(x, y) is scalar. We can now restate Equation (5) for the Leader's optimization problem
max E Exep B (r v)] - "KL, [
which is again a KI regueaalari optimization problem that admits a closed-form solution
n² T" xx)
Z(x) π ,< (y) x)) exp (+tr(z, v))
16

---

Published as a conference paper at ICLR 2026
A.2 PROOF OF THEOREM 2
Proof. This lemma is folklore in the algorithmic game theory community and can be quickly verified.
Let x: E X. Given any action y E y. there exists not necessarily unique y y minimizing ply > y/ I Hence, irrespective of the Leader's policy T(- I:), there always exists a Follower's deterministic best response policy Wbr(: r, y) with Wor (y/ x, = 1 for some y'. other words, the Follower always has a deterministic best response policy,
Similarly, the optimization problem for the Leader given some context x reduces to finding y that maximizes By (yy/wwrr--/kg)) (p(y > y/ =)] so that the SLHF optimization problem admits determinsitic solution.
A.3 CONCAVE-CONVRS PROPERTY OF
We show here that the objective function in Equation (5) of the Stackelberg optimization problem concave-convex. Similar results were established in the context of NLHF by Munos et al. (2024).
Throughout this section, assume X 1 and omit from the notation for clarity. All results extend directly to the general case with a finite context space X.
Then, the objective function of Equation (5) is given by
()) = By - [p(y > s/) - r'KLLK ( x +
kLy (w (1 (()
(9)
The first term is bilinear in and w, shown by expanding the expectation:
By [plu v') = Σ ((V) Erlu v')w(u' lv).
y'EY'
The KL terms are convex in their respective arguments. Hence, f is bilinear when 74 = 7F
A.4 COMMENTS ON CONVERGENCE OF STACKELBERGGDA (ALGORITIIM 1)
We here want to briefly comment the ergodic convergence guarantee of STACKELBERGGDA that is a consequence of well-known results in the literature. We also briefly discuss limitations and future work achieve better theoretical convergence guarantees by adapting existing ideas from the optimization and NLHF literature to the SLHF problem.
As previously shown, the SLHF objective f(T, w) is concave-convex. It is well-known that two- timescale GDA then converges in the ergodic sense, i.e., the averaged iterates converge to the equilibrium with gradient complexity O (Nedić and Ozdaglar, 2009). In the strongly-concave- strongly-convex setting, standard results also tell us that two-timescale GDA converges in the last-iterate with complexity O(x² log {) (Zhang et al., 2022: Zamani et al., 2024; Lin al., 2024). Unfortunately, the KI. divergence is not strongly convex w.r.t. the & norm so that these results do not directly apply, and deriving linear last-iterate guarantees for STACKELBBRGGI BERGGDA is challenging. In future work, it will be interesting analyze whether, e.g., extragradient (Zhou et al., 2025) mirror descent (Munos et al., 2024) approaches that have been successfully applied to NLHF, can be adapted to the SLHF framework to guarantee fast last-iterate convergence.
B SCALABLE IMPLEMENTATION OF STACKELBERGGDA
When fine-tuning large language models, the context and action spaces X and Y are far too large to op- timize over and Ω directly. To address this, we introduce a practical variant STACKELBERGGDA in Algorithm 2.
Policy Parameterization. We replace the tabular policies æ and w with neural parameterizations Ty and Wy (e.g., transformer networks). This renders the policy spaces tractable via their parameter vectors 8 and Φ. however, the concave-convex property does not necessarily carry over to the parameters 0 and Φb
17

---

Published as a conference paper at ICLR 2026
Batched, Variance-reduced Gradient Estimates. Exact evaluation of the expectations in f is infeasible due to the expectation over the context and action spaces. Instead, at each iteration we sample a batch of size B:
{Zi Vis Vi p:) p:) 23 ~ P. Vi ~ no(. mz); sf ~ wo(- xi Bi), Pr = plue > 3i xz).
We then form unbiased estimates as
B
B
pi - T 1i Ve To(s xi), Vos B
Σ
pi - - V log (Vg aa vi),
with likelihood ratios A.f
na (m.
(walong) and ke"
(y. P1:34) The derivation of these gradients
follow the policy gradient method described in Williams (1992). The gradient estimators are naturally compatible with additional variance reduction techniques such as subtracting a constant baseline.
Single-Model Instantiation. Simultaneously training two sillion-parameter transformer models is memory-prohiee prohibitive.. Similarly to SCORE (Kumar et al., 2025), we collapse both Leader and Follower into one model Ty by using distinct chat templates (Figure 1). When the model only given the context x, use the template in Figure ((a) that only includes model is given both the context æ and an action y. we use the template in Figure 1(b) that includes both the context and the action y, as well as predefined instruction improve the action y.
the prompt. When the
Then, letting x द.. denote the two-timescale weight coefficient, we optimize the model to minimize the following loss function
B Σ i=1
B Σ (p: - ==(!") log No (vl xs vi). (10)
(0))
Px. -M,) log ( ax))
Gradient steps on L(O) realize the two-time-seut gradient descent-ascent updates via a single network, thereby substantially reducing memory usage.
Algorithm 2 STACKELLBERGDD (Practical)
1: procedure STACKELBBEGGGDACC P,,,))
2: Initialize the parameter @ for the shared model
3: for i = 1, do
4: for 1= L..... do
5:
Sample prompt 26 ~
6:
Sample Leader response using the prompt Figure I(a): y no(- (x))
7:
Sample Follower response using the prompt in Figure 1(b): vi ~ To 26, 26,36)
8:
Observe preference feedback ps, = plys > v/ (Tb)
end for 9:
10: Update the weights 0 according to the loss in Equation (10): 0 + -0-17 v]((0
11: end for
12: end procedure
Computational Comparison. At training time, the computational cost of STACKELBERGGDA is comparable to standard online RLHF/NLHF algorithms, with only marginal overhead from the Follower's 's prompt. Specifically, STACKELBBERGGIN
requires two samples per prompt, matching most NLHF methods and remaining more efficient than algorithms like NASH-MD-PG (Munos et al., 2024) or ONLINEIPO (Calandriello et al., 2024) that rely on expensive mixture policies. While SPO (Swamy et al.,, 2024) uses only one sample, its efficacy on LLM-scale tasks is not yet explored. Finally, compared to RLHF, STACKELBERGGD
avoids the high memory cost of PPO (Schulman al., 2017) which is due to the value function estimation. Recent memory-efficient RLHF methods, such as RLOO (Ahmadian et al., 2024) and GRPO (Shao et al., 2024), only resolve this memory issue by introducing sampling costs comparable to, higher than, STACKEL BERGGDA.
18

---

Published as a conference paper at ICLR 2026
C IMPLEMENTATION DETAILS
Implementation Codebase. We trained RLOO and NASH-MD-PPG using their implementations in the TRL Python package (von Werra et al., 2020). For all training runs, including reward modeling, we used Low-Rank Adaptation (LoRA) (Hu et al., 2022) with rank r = 32, scaling factor a 64, and dropout rate set 0.1. The codebase with instructions is included in the supplementary material for reproduction.
LLM--as---Jddge Implementation. In Section 6.2, we used the prompt depicted in Figure as an input for training. We calculate the preference probability for the first completion as the softmax probability for the two tokens corresponding to the model identifier strings.
1...AAA--.1-0----.. 1-1------NSRRUUE to provide a feedback to STACKELBERGGDA during
Compute Resources. Experiments in Section 6. were conducted on single node with 8 Nvidia GeForce RTX 4090 GPUs. The total compute time, including hyperparameter sweeps, approx- imately 4,000 GPU-hours. The training run in Section 6.2 was conducted on a single node with Nvidía GH200 GPUs using approximately 1,300 GPU-hourS.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 PREFERENCE MODEL
We estimate the preference model used to fine-tune the LLMs by treating the five attributes in the HELPSTEER2 datasets (Wang al., 2024) distinct annotators, denoted by the set A = (helppulnees correciness, coherence, complexily, verbosily), and define v as a uniform distribution over A. For each attribute a E A, we estimate a reward function fa using the Bradley- Terry model and maximize the log- -ogliikel on the training dataset D = {I y 3
mii
((((x;,);)) v(),.)))) M(r(I ("") + r(z,,,)))
We here decided which response the winning and losing one in the dataset by comparing the attribute scores provided by the annotators. The additional regularization ensures that the rewards are centralized around zero (Eisenstein al., 2024). For the attributes correctness, helpfulness, and coherence, we consider higher scores to be better while for verbosity and complexity lower values are more preferable. This is in accordance with the scoring criteria described in Wang et al. (2024). Each reward function is trained independently. initialized from the QWEN2.5-1.35 B model with single linear head. We train each model for 5 epochs on the training prompts and completions with batch size 32, learning rate 1c-4, and regularization coefficient À 0.01. The final accuracies of the models on the validation dataset are 78%, 65%, 61%, 60%, and 59% for verbosity, complexity, correctness, helpfulness, and coherence, respectively.
The overall preference function is then defined as
p(u x 3 (x)
Σ aEV
v( f (x)
(11)
We evaluate the non-transitivity of the preference model p defined in Equation (11) on the validation prompts and five responses from each of the four models used for comparison in Section 6. For each prompt, we construct a complete directed graph between the 20 completions as nodes and edges directed from the non- preferrfr completion towards the preferred one. Figure 3 illustrates this directed graph on the first prompt of the validation dataset. 57% of the directed graphs include cycles, which illustrate the intransitivity of the preference function p on the completion space y Furthermore, there are almost 19 million cycles in the dataset across all prompts. From these cycles, 3.79% is length 9 or shorter, 41 11.64% is between length 10 and 12, 42.72% is of length 13 or 14, and
'https: : httpsss///gggiig///eeee ctt.taacccccccc/coooooooeooeeeee wow/oocs///eeeecooocccc htt///////t////ae///////ooooo/// 7https: htttts:://huggingfacee 'https://hughe col unsl oth www.thhgggggnttttt rnst
httpo///r///ann/////aaaa/ss trainer
19

---

Published as a conference paper at ICLR 2026
User:
require leaderboard for various large language models. I'11 provide you with promots given to these models and their corresponding outputs. Your task is assess these responses and select the model that produces the best output from a human perspective.
Instruction
"instruction": "*" (prompo) "**
## Model Outputs
Here are the uncrdered outputs from the models. Each output is associated with specific model, identified by unique model identifier.
poodel_identifine "A" "output": ""* (r"eepponseO)
"model iddeediienttii "B", "output": """ (respponsel)" ""*
## Task
Evaluate the models on the basis of the quality and relevance of their results, and select the model that generated the best result. Reply with the identifier of the best model. Our evaluation will only take into account the first character of your answer, so make sure it contains only one of the identifiers and nothing else (no quotation marks, spaces, no lines, ...).
Assistant:
Figure 2: L.MM-as-a-judge prompt for META-LLAMA-3. 1-O0R-TSTUUUC
9.85% is longer. Also, 33.31% of the prompts have at least but more than 9 cycles, 17.12% has between 10 and 99, 3.63% has between 100 and 999, and 2.54% has at least 1000.
D.2 ITERATIVE IMPROVEMENTS AT TEST-TIME
We extend the experimental results from Section 6 by analyzing iterative improvements and per- formance scaling with increased test-time computation. Our results suggests that with increasing test-time computation the benefit of fine-tuning using both RLOO or NASH-MD-PG is negligible compared to using the base model OWEN2.5-0.57 In contrast, STACKELBERGGDA yields strict improvements. Building the example in Section 4.1, assume that at test-tine assttimme single annotator a ~ // and a context I ~ P are sampled.
For the base model QWEN2.5-0.577 and the models with fine-tuned with RLOO and MASH-MD-PO we independently sample N responses yi...., VN- For STACKELBERGGDA which inherently supports iterative refinement, generate the first sample from the Leader policy y1 x² x))
20

---

Published as a conference paper at ICLR 2026
Algorithms
Qwen2.5-0.5B
RLOO
Nash-MD-PG
StackelbergGDA
Condorcet Winner
15
10
Figure Directed graph based with completions generated by the fine-tuned models and edge directions representing the preference between them.
and subsequent responses from the Follower policy ~ wn( x, Bi-1) for i 2 2. We define V1:N (y)), (yI)...,y))..
In line with prior work on Best-of-N sampling (Openai et al., 2023; Beirami et al., 2024; Dubois al., 2024; Sessa et al., 2024), evaluate the quality of the N samples by computing the maximum reward obtained for each attribute under the sampled annotator's reward model, that is,
/ (x, V1.N)
(z. V1....9N
(12)
Analogous to the preference function defined Section 3, in this section, we compare two models and n' w.r.t. the preference functions derived from the annotator-specific reward functions under Best-of-N sampling:
x x)) = E yn (m) s1x- (1x))
(F, 2 (I,
(13)
Notational Note. We here adopt slight abuse of notation. Specifically, we write V1.N .T( x)) denote the sampling N responses from model म, even though this notation does not faithfully represent the sampling procedure used by STACKELBERGGDE NASH-MD-PG, the samples !/1..... UN are drawn i.i.d. from a single model T(: .)). In contrast, for STACKELBERGGDA. the sampling process is inherently autoregressive:
For QWEN2.5-0.2 2......-..B. RLOO, and
first draw y1 a=( x) from the Leader policy, and then generate y4 ~ (- x, 91-1) for > using the Follower policy. Despite this difference,
overload the notation V1-N T(. x) to unify the presentation in Equations (1 (12) and (13). the case of STACKELBERGGDA shorthand for the autoregressive sampling process described above.
this notation should interpreted as
Previous work has shown that Best-of-N sampling can rival the performance of RLHF-based fine- tuning (Dubois et al., 2023; Sessa al., 2024; Beirami et al., 2024). Motivated by this, compare the preference scores defined in Equation (13) of RLOO, NASH-MD- MD--M PG, and STACKELBERGGDA with respect to the base model Q.E52.5-0.5B. Throughout this section, consider the maximum number of samples to be N = 5.
Results. Table reports the preference scores for RLOO. While the model initially (i.e. N = 1) performs competitively complexity and verbosity attributes, iterative sampling reveals collapse
21

---

Published as a conference paper at ICLR 2026
Table 6: Preference scores for RLOO versus QWEN2.5-0.5B across all attributes function of the number of test-time samples N.
Attribute
Number of Samples N
2
3
4
5
Coherence
0.521
0.338
0.262
0.207
0.164
Complexity
0.892
0.842
0.801
0.771
0.754
Correctness
0.388
0.212
0.139
0.098
0.070
Helpfulness
0,322
0.150
0.087
0,057
0,036
Verbosity 0.846
0.777
0.728
0.695
0.669
Average
0.594
0.464
0.403
0.366
0.338
into a single preference mode. In particular, we observed deterministic outputs for RLOO the generic response: "I apologize, but I'm unable to engage in conversations about political topics. you have any other questions or need further assistance with a different subject, feel free to ask." As result, the model's 's diversity and coverage deteriorate, and overall preference scores (relative to QWEN2.5-0.5B. decline sharply as N increases.
In contrast, NASH-MD-PG demonstrates some benefit from additional sampling. as shown in Table 7. Its preference score for verbosity remains stable and it shows moderate improvement in coherence. However, for remaining attributes (correctness, helpfulness, and complexity) its gains are slower than those achieved by QWEN2.5-0.35 with Best-of-N sampling. Consequently, the overall preference score of NASH-MD-PG declines with increasing N, suggesting that the model fine-tuned with NASH-MD-PP does not improve notably (compared to the base model) when the number of samples drawn at test-time increases.
On the other hand, STACKELBERGGDA exhibits more favorable behavior. As shown in Table 8, while the preference score on verbosity and complexity taper off with more samples, STACKELBERGGDA achieves notably faster gains on coherence, correctness, and helpfulness. For these attributes, the preference rate improves by 10 percentage points or more, making STACKELBERGGR method among the three to demonstrate consistent improvement over QWEN2.5-4 5. as N increases. This means that the performance of STACKELBERGGUE effectively scales with test-time compute.
the only
The strong emphasis on complexity and verbosity by RLOO is expected, as it optimizes the average reward across all five attributes, and these two dimensions yield the highest values. However, for NASH-MD-PP MD-PG. this outcome less expected. We hypothesize that it stems from its training objective, which pits the policy against a mixture of the reference policy QWEN2.5-0 -.55. and the most recent iteration. Once NASH-MD-PG outperforms QWEN2.5-0.517 on all attributes, it begins focusing on attributes where further improvement over itself is possible, namely, complexity and verbosity. Nevertheless, this skewed emphasis is suboptimal: annotators prefer models that perform well on all attributes. In fact, policy that focuses on coherence, correctness, and helpfulness is preferred by 60% the annotators. STACKELBERGGIAE Follower policy separately (though potentially unified in single model) helps mitigate this imbalance across attributes. This leads to a more balanced policy that is preferred by a wider range of annotators.
's asymmetric formulation that trains Leader and a
D.3 ABLATION ON THE TWO-TIMESCALE COEFFICIENT
Table 9 and Table 10 present ablations the follower weight parameter x in STACKELBERGGDAA 's loss function (10) when fine-tuning the OWEN2.55-0.55 and 1..5 models, respectively. Each row reports the average preference scores over the corresponding initial policy, for both the Leader and Follower policies, on the training and validation splits. These results highlight the importance of balancing the two components of BEACKELBERGGIN In general, moderate values of и can help the Follower improve without compromising the Leader too severely, but excessively large weights may impair both players.
asymmetric training objective.
In Table 9, we observe that increasing k leads to a gradual decline in the Leader's performance. While the Follower benefits from increasing from 5, performance worsens 10 for both the
22

---

Published as a conference paper at ICLR 2026
Table 7: Preference scores for NASH-MD-PG versus QWEN2.5-0.5B across all attributes function of the number of test-time samples N.
Attribute
Number of Samples N
2
3
5
Coherence
0.731
0.743
0.757
0.763
0.760
Complexity
0.761
0,743
0.732
0.726
0.727
Correctness
0.633
0.615
0.620
0.617
0.615
Helpfulness
0,645
0,633
0.640
0.641
0,638
Verbosity 0.858
0.855
0.850
0.849
0.852
Average
0.726
0.718
0.720
0.719
0.718
Table 8: Preference scores for STACKELBERGGDA versus QWEN2.5-0.5B across all attributes function of the number of test-time samples N.
Attribute
Number of Samples N
2
3
5
Coherence
0.778
0.850
0.866
0.875
0.873
Complexity
0.714
0.666
0.628
0.600
0.592
Correctness
0.670
0.762
0.791
0.795
0.803
Helpfulness
0.692
0.767
0.783
0.786
0.791
Verbosity 0.833
0.820
0.798
0.777
0.765
Average
0.738
0.773
0.773
0.767
0.765
Leader and Follower, indicating overemphasis the Follower's loss can destabilize the overall optimization.
Table 10 shows similar trend for the larger QWEN2.5-1.5B model. Due the decrease of performance above = 5 in Table 9, we carry out the ablation on a finer grid k E {1, 3, 4, 5}. Moreover, we evaluate each model after 2000 training steps as larger base model requires more gradient updates to converge. While the performance of K = -1 stands out in Table 10, we observe that is overfitting verbosity and complexity by responding every prompt with short, non non-informative answers asking for further information such as "Certuinly! If you need detailed insights technical topics like that, feel free to ask-I'm here to assist with informatively aligned queries!". On the contrary to the collapse observed for RLOO in Section D.2, the model remains stochastic with the responses having similar information content. This outcome demonstrates the effectiveness of STACKELBERGGDA in optimizing its objective despite the qualitatively undesirable responses.
D.4 MODEL. SCALING
We extend our round-robin comparison from Section 6.1.1 larger models within the Qwen2.5 family, specifically, OWEN2.5-1.5p and QWEN2.5-3B (Qwen et al., 2024). These evaluations demonstrate that S STACKELBERGGDA continues be on par outperform baselines even as model size increases. Since larger models require more training updates to reach convergence in our setup, we train NASH-MD-PG for 1. 1,500 steps and STACKELBERGGDA for 2,000 steps. The RLOO method converges earlier and requires only 1,000 steps even for these larger models. We fix the for both scales, based on our ablation results Section D.3.
follower-weight parameter at
Table 11 summarizes results for models fine-tuned from QWEN2.5 51..B. Both NASH-MD-PG and STACKELBERGGDA clearly outperform the base model and the RLOO baseline. While the Leader policy of BTACKELBERGGDA underperforms compared to NASHI-MD-PG, the Follower performance,
policy conditioned on the Leader's 's responses matches or exceeds NASH-MD mirroring the improvements observed when starting from the QWEN2.5-0.55 in Table 3. As noted in Section D.3, this performance gap between the Leader and NASH-MD-RR PG could likely be reduced by tuning 6, albeit at the potential cost of Follower quality.
23

---

Published as a conference paper at ICLR 2026
Table Ablation on the follower weight parameter in STACKELBERGGDA'S loss function (10)
fine-tuning the QWEN2.5-0.5B model. Scores show the average preference over the base model.
Train
Validation
Follower Weight i
Leader
Follower
Leader
Follower
0.768
0.804
0.761
0.784
0,743
0.814
0.723
0.806
0.725
0.800
0.710
0.783
Table 10: Ablation on follower weight parameter k: in STACKELBERGGDA'S 's loss function (10)
fine-tuning the QwEN2.5-1.5D model. Scores show the average preference over the base model.
Train
Validation
Follower Weight
Leader
Follower
Leader
Follower
0.848
0.852
0.850
0.851
0.718
0.737
0.719
0.730
0.767
0.806
0.771
0.803
0.733
0.811
0.736
0.807
0.720
0.819
0.720
0.818
Table 12 shows analogous comparisons for models initialized from QWEN2.5-3B Here, STACKEL- BERGGDA again performs strongly, with its Follower policy matching or surpassing NASH-MD-PG across most pairwise matchups, and both algorithms outperforming the base model. NASH-MD-PPG and STACKELBERGGDE are closely matched when compared directly. Due to compute limitations, we capped training 2,000 steps for these larger models. Nonetheless, the Leader policy continued to improve near the end of training, suggesting further gains in preference score may be possible with additional updates.
24

---

Published as a conference paper at ICLR 2026
Table 11: Pairwise preference comparisons between the responses of QWEN2.5-0.55 QWEN2.5- 1.5B, RLOO, NASH-MD--PG. and BtaCkELBERGGIDA algorithms. Fine-tuned models trained from the QWEN2.5-1.55 Each cell shows the average preference score of the row model over the column model.
QWEN2.5-0.5B
QWEN2.5-1.5B
RLOO NASH-MD-PG
STACKELBERGGDA
LEADER
FOLLOWER
QWEN2.5-0,5B
0.000
0.479
0.379
0.188
0.271
0.166
QWEN2.5-1.5B
0.521
0.000
0.401
0.209
0.293
0.187
RLOO
0.621
0.599
0.000
0.197
0.310
0.175
NASH-MD-PG
0.812
0.791
0.803
0.000
0.623
0.489
STACKELBERGGDA
0.729
0.707
0.690
0.377
0.000
0.313
LEADER
STACKELBERGGDA
0.834
0.813
0.825
0.511
0.687
0.000
FOLLOWER
Table 12: Pairwise preference comparisons between the responses of QWEN2.5-0.5B. QWEN2.5-3B, RLOO, NASH-MD-PG; and STACKELBERGGDA algorithmss Fine-tuned models trained from the DWEN2.5-3B Each cell shows the average preference score of the row model over the column model.
QWEN2.5-0.5B
QWEN2.5-3B
RLOO
NASH-MD-PG
STACKELBERGGDA
LEADER
FOLLOWER
QWEN2.5-0.58
0.000
0.504
0.399
0.187
0.304
0.187
QWEN2.5-3B
0.496
0.000
0.412
0.199
0.319
0.179
RLOO
0.601
0.588
0.000
0.173
0.338
0,201
NASH--M-PGG
0.813
0.801
0.827
0.000
0.638
0.507
STACKELBERGGDA
0.696
0.681
0.662
0.362
0.000
0.312
LEADER
BEARGEL
0.813
0.821
0.799
0.493
0.688
0.000
FOLLOWER
25


Reviews:
[{"review_id": "5SoTeSM28W", "reviewer": "Reviewer_oDxv", "summary": "This paper proposes a stackleberg game formulation of the RLHF problem, in contrast to prior works which either use a bradley-terry model or search for a nash equilibrium.  The authors demonstrate that their stackleberg formulation is able to resolve common problems with BT-based models, i.e. cyclical preferences, while also allowing for further test-time adaptation using the follower model.", "questions": "* How is the reference for the follower defined? Does this reference model even make sense if the prior model hasn't been trained as a follower?\n* For a lot of LLM methods, compute matters. Could the authors comment on any difference in compute requirement vs nash vs BT model + PPO? What do results look like at compute parity?\n* Baselines: could the authors comment on why a method like SPO was not relevant / considered as abaseline? It is also based on nash equilibrium?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is generally easy to follow and can be understood. The overview of differetn approaches is nice too!\n* The section demonstrating the different types of preference relationships that different models can address is very nice and useful! \n* The approach is well justified theoretically. \n* The experimental evaluation considers both preference dataset evaluation and general finetuning\n* to my knowledge, using a stackleberg game for learning from feedback is novel.", "weaknesses": "* The experiment section lacks any ablations on the choices made. For example, how does the two-timescale schedule affect performance?\n* The method seems like it will be computationally more expensive. \n* I am not sure why the stackleberg formulation makes sense. I can see how the nash formulation can resolve ambiguities in preferences vs BT, but realistically when would I want to have a leader and follower? Using the follower will double inference costs. \n* the gains of the leader vs the nash models seem marginal at best. This seems to indicate that a lot of the performance gains might be coming from just using more compute / tokens for a response i.e. adding context.\n* The length bias seems really strong in the Alpaca results.\n\n\nNit:\n* In eq 5, the order might be more intuitive if the leader is the inner optimization and the then follower moves after? the notation for eq is also not super well defined -- and it would be nicer if the symbols for the leader and follower were more clearly introduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MGU1b0ojDo", "reviewer": "Reviewer_U2fY", "summary": "This paper introduces Stackelberg Learning from Human Feedback (SLHF), a new framework for aligning LLMs that models the problem as a sequential-move game between two policies: a Leader that commits to an initial response and a Follower that refines it. This sequential approach avoids the need for a single scalar reward model, unlike traditional RLHF, allowing it to better handle complex or intransitive preferences. The paper also proposes an algorithm, STACKELBERGGDA, to find the game's solution. A key advantage of this framework is its natural ability to perform inference-time refinement, where the Follower can be used to iteratively improve the Leader's output. Experiments demonstrate that the SLHF Follower policy not only improves upon its own Leader's outputs but also consistently refines and enhances the responses from other, independently trained models without any additional fine-tuning.", "questions": "- In the practical implementation, the Leader and Follower share parameters. Could this limit the follower's ability?\n- In Appendix D.3, this paper mentions that \"increasing κ leads to a gradual decline in the Leader’s performance. While\nthe Follower benefits from increasing κ from 1 to 5\". How to balance the performance of the leader and the follower? Which is more important in practice? \n- Regarding \"refining outputs from other models\", Does it imply the Follower learns a universal refinement rather than just a policy specific to its own Leader?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unlike standard RLHF, SLHF optimizes directly over pairwise preferences without collapsing them into a single scalar reward, allowing it to handle complex and intransitive preference cycles.\n- The Leader-Follower structure naturally supports improving model outputs at inference time, as the Follower is explicitly trained to refine a given response, allowing for iterative improvement with more computation.\n- By decomposing the problem, the Follower solves a simpler refinement task against a fixed action rather than a non-stationary opponent, leading to more stable learning.", "weaknesses": "- The method's success heavily relies on having a \"well-specified and representative pairwise preference function, which can be unavailable. \n- The experiments suggest the method can be sensitive to biases in the preference judge (in this case, an \"LLM-as-a-judge\"). The authors attribute the gap between standard and length-controlled win rates to the judge model's \"length bias,\" which the SLHF model may have learned to exploit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cwwbOtucn2", "reviewer": "Reviewer_qE41", "summary": "The paper introduces Stackelberg Learning from Human Feedback (SLHF), a new framework that models alignment as a sequential two-player game between a Leader (the action proposer) and a Follower (the conditional refiner). Unlike RLHF (scalar rewards) or NLHF (simultaneous equilibria), SLHF exploits sequential asymmetry to capture richer preference structures and support inference-time refinement. The authors propose STACKELBERG-GDA to efficiently approximate equilibria and scale training to large LLMs (0.5B–8B). Empirical results show SLHF achieves strong alignment across diverse datasets, with Follower policies improving outputs even when transferred to unseen models.", "questions": "Despite the discussion of model non-transitivity in Appendix D1, can you elaborate further on the merits of intransitivity coverage, compared to real-valued reward models?\n\nIn the majority of RLHF literature, people rely on 'transitivity' assumptions for its simplicity, while in real-world datasets, binary reward models, e.g., the Bradley-Terry (BT) model, when explicitly used, are known to be subject to 'intransitivity' because they rely on scalar variables that assume all preferences are transitive.\n\nFor your information, the literature below studied representative preference datasets in the real world, where the 'transitive' relationship between preference annotations may not always hold. \n- https://arxiv.org/abs/2409.19325 (Duan et al., 2017)", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper introduces an innovative game-theoretic Stackelberg structure for preference learning. \n\nThe proposal is rooted in the existence of intransitivity in pairwise preferences. It proposes a rational computational solution that replicates the logic with additional transparency into the learning and inference process.\n\nExperiments showed that it outperforms or matches RLHF/NLHF baselines across multiple datasets.\n\nSome theoretical foundations are discussed, i.e., qualitative connections to RLHF and NLHF, constructive conditions for numerical approximation, standard regularity assumptions, an equilibrium analysis, and an optimization algorithm (Stackelberg-GDA).\n\nSource code and assets are open.", "weaknesses": "The two-policy framework, i.e., Leader policy and Follower policy, increases computational and training costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bXLcbgtV8G", "reviewer": "Reviewer_Uhos", "summary": "This paper introduces Stackelberg Learning from Human Feedback (SLHF), a framework for preference optimization that models alignment as a sequential-move game between a Leader policy and a Follower policy. The authors propose StackelbergGDA, a two-timescale gradient descent-ascent algorithm to approximate the Stackelberg equilibrium. The framework naturally enables inference-time refinement through iterative sampling. Experiments on language models ranging from 0.5B to 8B parameters demonstrate that SLHF achieves strong alignment performance, with the Follower policy consistently outperforming RLHF and NLHF baselines.", "questions": "Please address my concerns in the Weakness section. There is no other questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- **Strong motivation and intuition:** The paper provides excellent motivating examples, particularly the Condorcet paradox analysis in Section 4.1, which clearly illustrates how SLHF, RLHF, and NLHF differ in handling intransitive preferences.\n\n- **Practical value of inference-time refinement:** The Leader-Follower structure naturally supports inference-time improvement without additional training, which is valuable for LLM applications. The empirical results in Table 4 demonstrate impressive cross-model generalization.\n\n- **Comprehensive experiments:** The paper includes thorough empirical evaluation across multiple model scales (0.5B to 8B parameters) and provides detailed ablations and additional results in the appendix.", "weaknesses": "### Major Weaknesses\n1. **Mischaracterization and Missing Literature on NLHF** \n\n**Issue:** The claim in the introduction that \"simultaneous play forces both players to optimize against a moving opponent which can hinder convergence\" is not accurate. Recent NLHF works have established polynomial or even linear convergence rates to Nash equilibrium: https://arxiv.org/abs/2312.00886, https://arxiv.org/abs/2401.04056, https://arxiv.org/abs/2410.16714, https://arxiv.org/abs/2503.08942. The paper should acknowledge these theoretical convergence guarantees rather than suggesting NLHF inherently struggles with convergence.\n\n**Critical omission:** A closely related work (https://arxiv.org/abs/2502.18099v2) that also studies Stackelberg games for LLM alignment was published over half a year ago and cannot be considered concurrent work. This significantly undermines the novelty claim. The authors must thoroughly discuss this work and clarify their contributions relative to it.\n\n2. **Problematic Characterization of Mixed Strategies**\n\n**Issue 1:** At the end of Section 3, the authors state: \"when no action is majority-preferred the equilibrium necessarily involves mixed strategies. This inherent stochasticity can be undesirable in applications where consistency and reliability are critical.\"\nThis characterization is misleading for several reasons: (1) In RLHF/NLHF/SLHF with KL regularization ($\\tau > 0$), the optimal policy is always a distribution over responses, not deterministic. (2) A \"deterministic policy\" in LLM is not well-defined, unless we sample with temperature 0, but the model still outputs a probability distribution.\n\n**Issue 2:** The claim before Section 4.1 that \"there exists a deterministic Stackelberg equilibrium\" suffers from the same conceptual problem. With regularization (which the paper uses throughout), policies must be stochastic. The best response is essentially the RLHF solution when viewing win-rate as reward.\n\n3. **Lack of Theoretical Analysis:** The paper provides no convergence guarantees for either Algorithm 1 or Algorithm 2. Key questions remain unanswered: Does StackelbergGDA converge to a Stackelberg equilibrium? What is the convergence rate? Under what conditions does convergence occur? Given that RLHF/NLHF methods now have established convergence theory, the lack of any theoretical analysis for SLHF is a significant weakness. The paper should at minimum discuss the challenges in proving convergence or provide experimental evidence of convergence behavior.\n\n4. **Algorithm Presentation Issues:** Inconsistency: Algorithms 1 and 2 are essentially different algorithms (analogous to OGDA vs. OMWU, https://arxiv.org/abs/2006.09517). For consistency and clarity, I suggest replacing Algorithm 1 with a theoretical version of Algorithm 2.\n\n### Minor Weaknesses and Questions\n\n5. **Baseline choice:** The paper uses Nash-MD-PG as the primary NLHF baseline, which has been shown to converge extremely slowly (see Section 5 of https://arxiv.org/abs/2503.08942). How would SLHF compare against faster NLHF algorithms like those mentioned above?\n\n6. The intransitivity analysis (57% of graphs contain cycles) is interesting but could be expanded—what is the typical cycle length? How does this compare to other datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)