Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment
Authors: Xu Chu
Zhixin Zhang
Tianyu Jia
Yujie Jin
Venue: NeurIPS 2025 poster ()
Keywords: Data-Efficient
Alignment
Large Language Models
Stackelberg Games
Primary area: deep_learning

Abstract: Aligning large language models (LLMs) with human preferences typically demands vast amounts of meticulously curated data, which is both expensive and prone to labeling noise. We propose Stackelberg Game Preference Optimization (SGPO), a robust alignment framework that models alignment as a two-player Stackelberg game between a policy (leader) and a worst-case preference distribution (follower). The proposed SGPO guarantees $\mathcal{O}(\epsilon)$-bounded regret within an $\epsilon$-Wasserstein ball, offering formal robustness to (self-)annotation noise. We instantiate SGPO with Stackelberg Self-Annotated Preference Optimization (SSAPO), which uses minimal human-labeled “seed” preferences and iteratively self-annotates new prompts. In each iteration, SSAPO applies a distributionally robust reweighting of synthetic annotations, ensuring that noisy or biased self-labels do not derail training. Remarkably, using only 2K seed preferences—about 1/30 of standard human labels—SSAPO achieves strong win rates against GPT-4 across multiple benchmarks within three iterations. These results highlight that a principled Stackelberg formulation yields data-efficient alignment for LLMs, significantly reducing reliance on costly human annotations.

Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment
Xu Chu 1,2,3 Zhixin Zhang 1.3, Tianyu Jia 1.3, Yujie Jin 1.3
Key Laboratory of High Confidence Software Technologies, Ministry of Education "Center on Frontiers of Computing Studies, Peking University
School of Computer Science, Peking University
Abstract
Aligning large language models (LLMs) with human preferences typically demands amounts of meticulously curated data, which is both expensive and prone label- ing noise. We propose Stackelberg Game Preference Optimization (SGPO). robust alignment framework that models alignment as a two-player Stackelberg game be- tween a policy (leader) and worst-case preference distribution (follower). The pro- posed SGPO guarantees (C (bbundee regret within an e- Wasserstein ball, offering formal robustness jannotaanntnnoon noise. We instantiate SGPO with Stackelberg Self-Annotated Preference Optimization (SSAPO), which uses minimal human- labeled "seed" preferences and iteratively self-annoe self-annotates new prompts. In each iter- ation, SSAPO applies a distributionally robust reweighting of synthetic annotations, ensuring that noisy biased self-labels do not derail training. Remarkably, using only 2K seed preferences- about 1/30 of standard human labels-SSAP PO achieves strong win rates against GPT-4 across multiple benchmarks within three iterations. These results highlight that principled Stackelberg formulation yields data-efficient alignment for LLMs, significantly reducing reliance on costly human annotations.
Introduction
Large language models (LLMs) have demonstrated remarkable capabilities across a broad range of tasks, but aligning their outputs with human preferences remains a core challenge for safety and usability 3 Traditional alignment paradigms, such as Reinforcement Learning from Human Feedback (RLHF) 3 or Direct Preference Optimization (DPO) (5), typically rely on large amounts of meticulously curated preference data. Such data collection costly, time-consuming, and inevitably prone to labeling noise or bias, which can in turn degrade model performance once integrated at scale [6]. Consequently, an important question arises:How can we achieve robust alignment of LLMs without relying on vast, error-prone naman-labelee datasets?
One promising direction is to reduce the need for human-annotated samples by having the model itself generate preference labels on newly sampled prompts-So-called "self-annotation" 7 g 9.. However, most self-annotation approaches overlook the fact that synthetic labels may be systematically biased or noisy. If these errors go unchecked, they can compound over iterative rounds of training, ultimately harming rather than helping alignment [10
In this work, we address this problem by framing preference alignment as a two-player Stackelberg game between policy (leader) and worst-case preference distribution (follower). Our formulation, which we call Stackelberg Game Preference Optimization (SGPO). explicitly guards against plausible shifts adversarial corruption in the preference data by operating within e-Wasserstein ball 12 around the empirical distribution. We prove that the resulting policy achieves (bonnddo
CCorresonniigg author. Contact E-mail: thu_xu0pku.ede lagiku.edd.com edu. cn
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

regret, theoretical guarantee of robustness to noise or distribution mismatch [3]. This stands in contrast to standard DPO, whose regret can grow linearly with such shifts [5] We delay a more thorough related work section in the Appendix
To instantiate SGPO with concrete algorithm, Preference Optimization (SSAPO). Starting from small set of human-labeled "seed" preferences, SSAPO self-annotates new prompts by generating candidate responses and ranking them internally. Crucially, it couples this self-annotation with distributionally robust reweighting [3 that prevents noisy synthetic labels from overwhelming the training updates. Remarkably, we find that using only 2K seed preference pairs (around 1/30 of the usual scale), SSAPO outperforms or matches methods that rely on significantly more human labels, On multiple alignment benchmarks senchmarks-incluuinng AlpacaEval [4] and MT--eecch [15] SSAPO rapidly achieves competitive superior performance within just three rounds of iterative self- annotatntn....
then present Stackelherg Self-Annotated
We summarize our contributions as follows. 1. Stackelberg formulation of preference alignment: We recast alignment as a two-player game and prove the existence of robust equilibrium with -boundeed regret under e- e-Wasserstein preference shifts. 2. Robust self-annotation algorithm (SSAPO): We instantiate our framework by combining minimal seed labels with iterative synthetic annotations. Our distributionally robust reweighting attenuates the impact of potential labeling noise. Data efficiency and empirical results: Experiments show that SSAPO maintains high-level performance despite using only fraction of typical human annotations, achieving strong results comparisons.
against GPT-4
2 Theoretical Foundation: SGPO Framework
This section formalizes Stackelberg Game Preference Optimization (SGPO) and establishes its guarantees. We begin with DPO preliminaries (Section 2.1). then cast SGPO as two-player Stackelberg game over gap distributions (Section 2.2 y. We prove existence of Stackelberg equilibrium
and local convergence of practical alternating scheme (Section and finally quantify regret and contrast SGPO with DPO (Section22.. 2.4). All proofs are deferred to Appendix
2.1 Preliminaries: Preference Datasets and DPO
Preference-ranked dataset. We use D {(* the winner/foser responses (from human or partially self iif-aanottted feedback).
where x' is a prompt and (vfn.1f) are
RLHF and KI. regulaizaation Classical RLHF 4 optimizes mo under a KL. penalty to Wref!
max Ez-D.yyvxo(1e) [(x.y) - -3DK.. (Ta(- [=)] (-1=))]; EE8
where 3 0 controls regularization and is the prompt distribution.
Direct Preference Optimization (DPO). Under the Bradley-Teryy model p(y ye x))
a (R(æ,yw) (Ll(z,,))RR(z.).) R((z,y)) with objective yields
first-order optimality of a related KL-regularized
To x²) Tee (y x)
((x.y)=Blog
(log
(2)
with partition function Z. Plugging this into the BT likelihood gives
LDoo (0) = (z.9...u) [logo (R(x,w)) R(x,))))))
(3)
DPO is simple but provides no explicit protection against shifts away from the empirical preference distribution. This motivates robust formulation.
2.2 SGPO: A Two-Player Stackelberg Game
SGPO imposes robustness over preference gaps. For policy v, define the gap map
AR (I.NW.)) == (2.yow R2 (z.ye) E.

---

Let P= 1 N &Ei=o0e (xf. and let (ARx) # be its push-forwardd i.e.
(AR+) P= ARR (ar wi (P(R).
We measure uncertainty via the I-Wasserstein ball (a W1 (aa gap distribution v. Hereafter, for simplicity, write и instead of no, the dependence on 0 is implicitly assumed throughout the paper.
<c} centered at a
Leader objective in gap space. SGPO leader solves
Let f(E) = logo concave and 1-Lipschitz function on R. The
max tell
min
Be [f(E)]
aeul
This objective optimizes the worst-case preference likelihood over perturbations in gap space rather than in token space. See Section 3 for implementation details.
Follower best response in gap space. For any center Ve P(R) and radius >> 0, define the follower best-response set
A.(v) = arg min aEP(R)
[s [()] (i(a))
(5)
When y (AR+) any o (AA((V) is a follower best response against
This formalization induces the Stackelberg equilibrium:
Definition 2.1 (Stackelberg equilibrium). A pair (x*,a") a Stackelberg equilibrium
E arggaax xell aell, (())))*))
min
E Eg [f()),
a' AeEAe((ARi P)..
(6)
This definition links the leader's 's robust optimization with the follower's DRO problem (5). We next establish existence and analyze practical alternating scheme.
2.3 Existence and Convergence of Stackelberg Equilibrium
We first state mild conditions ensuring existence, then analyze an alternating best-response with a proximal leader step.
Assumptions. (i) II is compact. (ii) For each (x, ग + (2, is continuous; hence m +
(AR,), # P continuous in the weak topology. (iil) =loga)) is continuous and 1-L.ischitzz on TR. Theorem 2.2 (Existence of a Stackelberg equilibrium). Under the assumptions above, problem (4) admits at least one solution (T²,,"" "r")
Proof sketch. For fixed æ, the inner problem has a minimizer by compactness of (()R)) P)
and continuity in a. Berge's maximum theorem []6 17 yields upper semicontinuity of
V(T) min agu. ((AR, R2 Ay [[]]] and compactness of II gives maximizer x² and follower best
response a². See Appendix D
Alternating best responses with a proximal leader step. Let â(x) = (ARx) Given me choose follower best response
a++1 (á(mm))) E arg min oell (a(*))
Eg (f(E)
Update the leader via a proximal step:
FE+1 E Eargmax min well (
Ee-a (f)) 1 Te|
Tel
(8)
This scheme makes the leader step stable while allowing the follower to track the changing center ((T.). The regularization term || TT]l the proximal step can be induced practice with small learning rate and weight decay regularization.

---

Theorem 2.3 (Well-posednenl and local linear convergence). Suppose the proximal leader objective p--strongly concave neighborhood of x², unifoommly over U, (ā(z). and π ((()) is locally Lipschitz in W1. Then the update map (Te,04) (Tt-1;04++1) is contraction near (л" ox") and thus converges locally linearly.
In practice, one may not directly implement that approximates these alternating best-response updates converge the robust equilibrium. This provides a theoretical grounding for the SSAPO algorithm (to be introduced in the section 3). which combines standard wradient-based optimization with distributionally robust optimization.
but the Theorem 2.3 shows that any procedure
2.4 Regret Analysis and Comparison with DPO
We quantify worst-case performance under gap-space shifts and compare with DPO, Define the performance functional
P(=,a)=Ee-aa [(
Let x- solve We prove that x² maintains high performance on all distributions a within e- c-Wasserstein distance of á. In particular, the drop from P to any Pis at most ((c).
Theorem 2.4 (Worst-case performance
peerom-casseeefoomaacc in gap space), For every a (El(((((()))
P( ,a) (((""((='))) á,(="))--
Gap-space regret. We define the Gap-space regret of a policy a on a distribution a as Regret(z,a) ma -
Theorem 2.5 (SGPO regret bound). For solving
sup (egret(n* ,a) 2e.
ae4, (a(m))
Thus, SGPO is robust: under any shift at most e, its regret is bounded by a constant factor of
2.4.1 Comparison: DPO's Linear Regret
Let tDPo E (rgmax,)(TT,ū(T)) be DPO solution on the empirical center. For any target a* with 8: Wi (a* ((TDP))), we have:
Theorem 2.6 (DPO regret lower bound). (egret((DPO.a)) 2 8 8-2c. particulla, ifō DPO's
regret grows linearly in
Corollary 2.7 (SGPO advantage over DPO). (W(T(((T))) =8> 2e, then
Regret(zDPo,a*) 8-2c
Regret(m*"."* .0*)
2c
Thus, SGPO's robust policy can outperform DPO by a factor of 4 - under sufficiently large distribution shift 6. SGPO builds in-sample performance and out-of-sample robustness into single objective by optimizing against U. in gap space. The (Ee) worst-case degradation contrasts with DPO's linear sensitivity distribution mismatch 6, mirroring our empirical results in Section
2r
3 Practical Instantiation: SSAPO Algorithm
We now present a practical and computationally tractable realization the Stackelberg scheme from the theory section, called Stackelberg Self Self-Annotated Preference Optimization (SSAPO). SSAPO
implements the iterative seader-followe updates Theorem 2.3 and 7 S for preference alignment.
Notation.
Let (ut and define the margin random variable
E A.o(T..W..ee) for winner-loser pairs (y). a 1
Ees E &=== (a' wl..))
The follower (adversary) chooses distribution a over & inside the W, Wasserstein ball ((.()()))) centered at á(==), while the leader (policy) updates 0.

---

Implementation challenges addressed. 1. Minimal human labels via self- reff-amotation. We bootstrap from a small seed of numan-labeled' preferences and enlarge the dataset by letting the current policy rank its own responses on unlabeled prompts. 2. Loss re-representation for tractable DRO. The follower minimizes a concave inner objective E., [loga(E))
(E)) Writing {{E) =-loaa (E) which is convex and 1-Lipschitz, turns the inner problem into sup,, [.,(((()) and allows a convex PWL surrogate that yields a finite convex program for the follower. 3. Scalability via uniform grouping. For large datasets, we solve the follower subproblem on groups and average the resulting worst-case distributions, trading a small approximation for substantial speed-ups.
3.1 Follower objective: loss re-representation and closed form
Since logo is concave and 1-L.ipschitz, define the convex 1 1-L.isschtzz loss
{()) == - logo (5) log(1 + 5)
min (a(x))
E2, (lgoo(())
sup R. ((()).
When = I (no support restrictions) and the ground metric the absolute value, the worst-case expec-
tation of any L-Lipschitz function equals the empirical mean plus Le. Specializing this conver reduction
result of Mohajerin Esfahani and Kuhn [I3 Thm. 6.3) to (whose Lipschitz constant 1) gives:
Lemma 3.1 (Closed-fon) follower in the unconstrained one-dimensional case;
1 Thm. 6.3). If
E R. and is W, ball (absolute ground metrie), then
N
N minE, (log = N
sup (ā(.)
E., [e(E)] = N
Σ ((Ei)) +
Σ logo
c.
Although Lemma 3.1 provides exact closed form in the unconstrained 1-D setting, in this paper we propose to solve a finite convex program that also returns a discrete worst-case distribution.
3.2 Follower via a max-of-affine surrogate and a finite convex program
We approximate e by convex piecewise-lineal under-approximation
((E) max {akE+ba) < E(g), with as. -(=(((())==-(((t(t) ((((())),, by ===(((()) ax ((R)
ISkSK
Knots )k are chosen on a window [a,, 6t] in margin space (empirical quantiles or (mx with small 5 ->0) 0). Endpoint tangents extend & outside [or,bi], preserving i & globally. Because i i f, replacing é by i in 3.1) yields an upper bound on the original inner minimum, which tightens with K:
Proposition 3.2 (Monotone tightening in K). Let à be a probability measure on T. with funite first moment and let (ā) be the W, ball of radius 2 0. Let {(E) -logo (f) and let (Ex (x)x21 be convex piecewise-lineed underestimators of such that lx <k
pointwise and ê L. Define
= inf EA (E) sup (EU.(a)
E. E. (((E))
0K := sup T., (E)) aEUL(&)
(a),(()
Then, for all K K2 1: (i) 2 o² (valid upper bound); (üi) UK+1 SUK (monotone in K); (ii) uK Lut as + DC.
A finite convex program for the worst case (after 13, Thm. 4.4). pointwise maximum of finitely many affine functions,
For losses representable as
Eg max +bb)
sup (G(T))
LECK
admits a finite convex program whose solution is a discrete extremal distribution. Specializing Mo- hajerin Esfahani and Kuhn K Thm. 4.4] our one- mie-dimnniioaal & and absolute ground metric yields:
Theorem 3.3 (Finite convex program for max-of-affine (PWL convex) losses; specialization of
Mohajerin Esfahani and Kuhn [33, Thm. 4.4). Let ((E) - max +bx and = C R. Introduce, for each sample i and piece k, a mixing weight 8ik 0 and a displacement yaz R. Then
E. [(() max (a(M))
sup
Si (1 ax
(9)
(aa J N
5

---

Large Prompt Pool (small seed-labeled prefs)
Self-Anntaaton (Policy ranks new responses)
Preference DataBase (new pairs added)
Policy Update
Follower Update
(optimize reweighted loss with gradient descent)
(optimize worst-case distribution with DRO)
Figure 1: SSAPO workflow. We maintain a large prompt pool and small set of seed-labeled preferences. The policy self-annotates new prompts by generating and ranking responses, thus expanding the preference database. A follower then identifies a worst-case distribution for these preferences, and the leader (policy) is updated accordingly. This process repeats for iterations.
subject to the Wasserstein and feasibility constraints
K
qikl
S
Σ
Sik
L, Sik
20
(Vi),
0g5ih S SiKEi-9u bist (Vi,k).
1=1
N
K
An extremal discrete measure Oi
- attains the supremum.
N
The change of variables z = 1ik//ik (when 8a > 0) reveals that 8ikk splits the unit mass at E, across pieces, while Rs transports that mass in margin space. The interval constraints are linear "perspective" constraints enforcing [a,,be].
3.3 SSAPO workflow
Figure summarizes the SSAPO workflow. Starting with a small seed of human-labeled preferences plus a large unlabeled pool, we proceed in the following loop, at iteration
Self-annotation. Sample prompts from the unlabeled pool, generate multiple responses, and let the current policy TT, rank them to create preference pairs (Vor-Hr).
2. Empirical center. Compute غ, Ro, (a* y.. and set (m+))= > E
3. Follower (DRO). Build ((g) max (agE +bx+))b))) bx) using tangents knots (g(*)); choose
[ag.be] as above. Solve the convex program in Theorem 3.3 to obtain the worst-case o?
4. Leader update. Update by minimizing
ozt1 argmin ==
[ (o
implemented standard minibatch SGD the preference pairs while incorporating the per-sample mixture weights induced by (s* """)))
Repeating for I' total iterations yields the final aligned policy Tey A more explicit version of SSAPO is provided Algorithm (Appendix along with its computational complexity analysis.
How enters the leader step. The optimizer (s" "s"""") associates to each training pair set of active affine pieces (weights {stx] })) and a transport direction (through / practice we (i) weight the per-pair loss contributions by {s and (ii) optionally add proximal penalty nudging
the current margin toward /
E / for stability.
3.4 Scalability and Complexity
Grouping for large N. When N is large (e.g. 10S more preferences). solving the convex program in Step (Worst-Cass Distribution) can be expensive. A popular heuristic partitions {ÉI....fE'' into M groups (each of size G G-NMM,, and solves the finite program 9 separately within each group.
The resulting distributions o are then averaged (or merged proportionally):
M
Pinal
Om
m=1

---

While not an exact solution to the global N-sample problem, this still confers substantial robustness while reducing complexity from N to & N in each subproblem. In summary, this grouping approach greatly reduces memory/compute cost, and is parallelizable. Section approximation effects of SSAPO algorithm design on SGPO guarantees
F in the appendix remarks the
Complexity. The program in Theorem 3.3 introduces ((N variables and O K) constraints and becomes a pure LP after linearizing [ysk via standard slack variables. With warm starts, per-iteration time scales nearly linearly in N and linearly in K. A more detailed algorithmic complexity analysis is in Appendix E
Why finite convex program is necessary in practice. We argue the special case of unconstrained one ome-dimensional case the follower (that collapses to an exact closed form from Lemma 3.1 is too restrictive for real pipelines: we (i) clip or restrict the support of margins for stability. (ii) require extremal discrete adversaries whose mixture weights can be recycled to form stochastic gradients for the leader, and (iii) need controllable approximation whose accuracy improves monotonically. Accordingly, SSAPO replaces the inner concave expectation min (a (á())) E., (logo (5) by tractable
finite convex program obtained from a convex piecewise- lineer surrogate &x (E) = m K {axE+bby of ((5) = - log (f). This yields an upper bound - sup Baler (6)] on the inner minimum that tightens monotonically in K (Proposition (.2) and becomes exact K oc. Moreover, specializing Mohajerin Esfahani and Kuhn [3] Thm. 4.4] to our one min-innnnio absolute ground metric produces finite convex program whose optimizer is discrete extremal distribution supported
most N K atoms: exactly the structure we need to implement sample reweighting and efficient leader updates. In short, the finite convex program is both theoretically correct (via 13 Thm. 6.3 and Thm. 4.4) and operationally necessary for SSAPO's stability and efficiency.
4
Experiments
In this section, we present extensive empirical evaluation of our proposed Self-Amotateed Preference Optimization (SSAPO) algorithm.
4.1
Experiment Setup
We introduce the basic experiment setup in this subsection (Cf. Appendix G for more details). The settings are mostly consistent to the recent literature Kim et al. 19. Datasets. We used the UltraFeedback dataset [8] containing 60K samples. A seed of 2K buman-labeled preferences (3.3% of total 60K data) was used for initial training. The rest (58K samples) were split into three subsets (8K, 20K, and 30K) for self-annotation in iterative stages.
Models. We use the supervised fine-tuned Mistral-7B-0.1 LLaMA-3-8B for compatibility checks. All models are fine-tuned on UltraChat (20).
19 as the initial model Tinit and
Evaluations. We use AlpacaEval 2.0 14 for instruction-following tasks and MT-Bench 15 to evaluate multi-turn performance across tasks like math, coding, and writing. Both benchmarks assess the alignment with human preferences and the model's functional proficiency. We stress that AlpacaEval 2.0 is especially useful for measuring how well the model aligns with general user preferences (and controlling for length bias), whereas MT-Bench tests the model's functional capabilities across broader range of tasks.
Implementation. We initialize training with DPO on 2K seed samples, followed by 3 iterative stages of self-annotation. In each stage, new preferences are generated via a policy that ranks response pairs. distributionally robust optimization (DRO) is performed using sequential least squares programming (SLSQP) to adjust the model based on adversarial shifts within a Wasserstein ball. The group size G for parallel computation is set to 100 unless otherwise specified.
Baselines. We consider the following baselines for comparison: (1) DPO, which performs DPO training only on the seed data. (2) Iter DPO [1, which iteratively generates preference data using an external reward model (PairRM) [1 or L.MMass-judee [22] (3) SPA 9 which iteratively generates
preference data using implicit reward model.
meta- MaMMa I-StrInntt
7

---

Table 1: Main results. Evaluation results AlpacaEval 2.0 and MT- with different variants of Mistral-7B-vO. and LLaMA3 8B. All models use the same 2K preference data with gold label seed data. The best and second-best results are highlighted in bold and underlined, respectively. Most of the baseline results are from 9
Models
AlpacaEval 2.0
MT-Bench
Len-control. Win Rate (
Win Rate vs. (PT
Avg. Score (0-10)
Mistral-7B-Da 7B-DDP
9.03
7.68
6.81
Mistral-BB-Hte 7B-Iter DPO (PairRM)
11.87
9.46
6.98
Mistral-7B-Iter DPO (LLMM-s-judge)
9.28
9.18
6.67
LLaMA3-8B-DPO
20.61
18.
Mistral-7B-SPA
15.39
21.13
6.94
LLaMA3-8B-SPA
21.85
24.95
7.86
Mistral-7B-SSAFR (Ours)
24.44
35.82
6.68
LLaMA3-8B-SSAA (Ours)
33.33
40.12
Table 2: Comparison with different variants of Mistral. Evaluation results on AlpacaEval 2.0 and MT-Bench with different variants of Mistral-7B-vO.1. The best scores are highlighted with bold. The baseline results are from [9] and [23].
Models
Gold Label (%)
AlpacaEval 2.0
MT-Bench
Len-control. Win (%)
Rate vs. GPT-4
Avg. (0-10)
Mistral-7B-ooe
0.50
3.25
Zephyr-7B-3
100
11.75
10.03
6.87
Mistral-7B-SFFT
7.58
4.72
6.34
Mistral-7B--DPO
7.68
Mistral-Large (123B)
21.4
32.7
Mistral-TB-SSAPO (Ours)
3.3
24.44
35.82
6.68
4.2 Main Results
Table[[summarizes our primary comparison on AlpacaEval 2.0 and MT-Bench. All models in this comparison use only 2K preference pairs of the UltraFeedback dataset seed data (3.3% of 60K), with the remainder self-annotated. Our SSAPO method consistently outperforms DPO and other iterative baselines (e.g.. Iter- DPO, SPA) in both the length-controlled (LC) and raw win-rate metrics on AlpacaEvval 2.0. For Mistral- -BB, SSAPO achieves 24. 44% LC win rate and 35.82% raw win rate, compared to only 9.03% and 7. 7.68% with standard DPO. On the larger LLaMA-3-SD model, SSAPO reaches a 33.33 % LC win rate and 40.12% % raw win rate, surpassing its SPA counterpart by wide margin. MT-Bench scores corroborate these improvements, indicating that SSAPO yields robust, high-quality responses across diverse tasks.
To further illustrate SSAPO's data-efficiency and robustness, Table 2 compares various Mistral models, including Mistral-7B-SFF Mistral-Large (the number of parameters is 123B), and a fully-finetum Zephyr-7B-9 variant with 100% labeled data. Remarkably. Mistral-7B-SSAPE
outperforms
closely approaches these stronger references in AlpacaEval 2.0, despite using only 2K preference pairs (3.3% out of the 60K human-labeled training set). This demonstrates that principled Stackelberg method can substantially mitigate the reliance on massive human annotations. It also aligns with our theoretical findings (Section 2 that SGPO-based approaches, when instantiated as SSAPO, achieve bounded regret under moderate preference shift.
4.3 Ablation and Sensitivity Analysis
Table 3: Effect of Wasserstein Radius on Performance. Evaluation results on Mistral-7B, showing the impact of varying the Wasserstein radius on the Len-control. Win Rate and Win Rate vs. GPT-4.
0.01
0.03
0.05
0.1
Len-control. Win Rate (%)
19.76
24.44
22.42
23.20
19.78
Win Rate vs. GPT-4 (%) 26.58
35.82
32.30
32.92
25.84
We conduct series of ablation studies to understand the factors influencing the efficacy and robustness of our Stackelberg Self-Annotated Preference Optimization (SSAPO). Specifically, vary the
8

---

Table 4: Impact of Tangent Size (K) and Impact of Group Size (G) on Model Performance. Evaluation results on Mistral-7B.
Impact of Tangent Size (K)
Effect of Group Size (G)
K
6
G
100
200
300
Len-control. Win Rate (%)
22.89
23.20
19.05
CPU Runtime (min)
45
206
630
Win Rate vs. GPT-4 (%)
29.19
32.92
25.84
Len-control. Win Rate (%)
13.70
14.81
16.95
Win Rate GPT-4 (%)
10.00
11.74
14.91
control. nato WiT hate vs. G111
25
23.95 95
Wen Nato
18.0
12.45
10
1er.
Rer.
Rer.
Figure 2: Improvement during iterations Evaluation results on AlpcaEval 2.0 of initial DPO stage and each iteration, the results of the SFT model are from 9
Wasserstein radius c, the number of tangents K, and the group size G. We conduct the experiments on the Mistral-7B model for budget consideration. These experiments confirm our method's 's flexibility and validate the practical design choices guided by our theoretical framework.
Wasserstein Radius e. Table demonstrates how performance changes with different Wasserstein radius. When c 00 our approach reduces to self-annotated DPO without robust reweighting, yielding weaker results (19.76% L.C win rate). As increases slightly (e.g.. 0.01-0.05), both win-rates improve substantially, with the best outcomes at = 0.01. However, overly large c (e.g., 0.1) can make the adversarial shift too pessimistic, degrading performance. These findings align with our theoretical analysis in Section 2 where moderate provides a robust yet not overly conservative solution, thus striking the optimal balance between data fidelity and adversarial resilience.
Number of Tangents K. Since our piecewise-linear approximation of log ((.) uses K linear segments (cf. Section 3), we examine how varying affects alignment (Table At K=5, the model attains 22.89 % LC win-rate, while increasing to K = 6 yields marginally better 23. 33%0. Interestingly, moving to K = 7 leads to performance drops (19.05%). We hypothesize that while a larger K refines the convex inder-approximation, it may also overcomplicate optimization amplify minor errors in the approximation. Thus, K serves as a sweet spot in our setting, balancing expressiveness and computational stability.
Group Size G. Our distributionally robust optimization solver randomly partition data into groups of size G for parallel subproblem solutions. Table 4 (right half) illustrates the trade-off between computational cost and performance. A small group size (G = 100) has faster runtime (45 min) but yields 13.70% LC win-rate, whereas a larger G 300 reaches 16.95% yet takes over 10 times longer (630 min). This confirms that while bigger groups permit more fine-grained reweighting and hence improved alignment, the overhead grows significantly. In practice, we choose G 100 G = 200 for an acceptable performance cofoommancc-fficcinnyy balance.
Iterative Performance Gains. Figure 2 provides a direct illustration of iterative improvement over three rounds of SSAPO. Starting from a baseline DPO model, each round not only adds new self-annotated preferences but also reweights them adversarially within an e-ball. We observe consistent upward trend in alignment metrics during the first two rounds, validating our claim that robust self-annoton celf-annotation compensate for scarce human labels while preserving alignment quality.

---

Taken together, these ablations highlight the flexibility and effectiveness of SSAPO: Moderate e balances robustness and data fidelity, confirming our theoretical finding that worst-case reweighting within a bounded radius can significantly enhance alignment without over-penalizing feasible distributions. Piecesssseeeeeieinneeeee -log(((-)). maintaining computational tractability. Group size G offers controllable trade-off between runtime and fine-grained adversarial reweighting, making the approach scalable to different budget constraints. Iterative self-annotation with minimal seed data substantially boosts alignment, demonstrating that only 2K human-labeled preferences can suffice to achieve high performance. Overall, these experiments affirm primary contributions: data-efficient and theoretically grounded approach to preference alignment.
approximations with small K are sufficient to capture the shape of
Practical Hyperparameter Guidelines
Wasserstein radius
Scale e with expected self-annotation noise. Capable models: E [0.005, 0.02]: smaller models: e E [0.01, 0.05]. A robust default is e = 0.01 when no validation is available.
Piecewise approximation tangents K. We recommend K = 6 as stable default. K = may hurt due to solver instability rather than approximation error.
Grouping size G. For parallel DRO, G [100, 1000] balances robustness and throughput; we find 100-300 a sweet spot practice.
4.3.1 Robustness Seed Label Noise (25% flips)
To assess robustness promised by our O(e) reegret guarantee, we flip the preferred/unprefeered labels on 25% of the 2K seed pairs and re-run SSAPO end-to-end. Table shows that Mistral-SSAN SOAPO suffers only a 7-133% degradation, while
improves under noise, indicating DRO regularization and stronger self-annote ceff-annttation counteract moderate seed noise.
Table 5: Effect of 25% % seed label corruption. Entries show (AlpacaEval 2.0 LC win-rate Win-rate vs GPT-4), higher is better.
Model
No noise
25 % noise
Mistral-SSAPO
26.90% 31.93%
19.70% / 18.51 %
LLaMA-SSAPO
33.33% 12%
43.74% / 46.70%
Discussion. For Mistral, bounded degradation empirically aligns with O(c)-regret theory. For LLaMA-3-8B, noise acts as implicit regularization: the worst-case distribution explores a wider e-ball region and mitigates overfitting to small seeds.
5 Conclusion, Limitations and Future Work
Aiming at a data-efficient alignment method, we have introduced SGPO alignment framework with (ee) )boundde regret under moderate noise distribution shifts. Our practical instantiation, SSAPO, uses self-annotation and distributionally robust reweighting to achieve strong performance with far fewer human labels. The scalability limitation of SSAPO comes from the number of preferences N, we use a simple uniform group trick to balance between robustness and complexity. For further improve- ment, one may resort to primal-dual or specialized autting-plane methods [3], or use approximate re- laxations with entropic regularization (24] Our guarantees target training time robustness to preference noise and mild distribution shifts (e.g., reweightings within W2 babll.). This is distinct from inference- time robustness to adversarial prompts or jailbreak attacks. While our DRO step improves alignment under noisy supervision, it does not replace dedicated safety mechanisms for adversarial inputs. We only consider the human-labelle preference restricted scenario, however, SSAPO can also be integrated with prompt-gent compt-generation procedure such as EVA 255, which could be crucial scaling large language model based intelligence, considering that high-quality human data is running out in the next few years (26).
10

---

Acknowledgments
We thank the reviewers for their useful feedback that improved this work. This research is supported by the National Natural Science Foundation of China (NSFC) under grants No.62506010.
References
[1] Daniel M Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 08593, 2019.
[2] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assis- tant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588 2022.
[4] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[5] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023.
[6] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 15217, 2023.
[7] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In Proceedings of the Forty-first International Conference on Machine Learning, 2024.
[8] Weizhe Yuan, Richard Yuanzhe Pang. Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Proceedings of the Forty-first International Conference on Machine Learning, 2024.
[9] Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jaehyung Kim. Spread preference annotation: Direct preference judgment for efficient Ilm alignment. In The Thirleenth International Conference on Learning Representations, 2025.
[10] Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. In Proceedings of the Forty-first International Conference on Machine Learning, 2024.
[11] Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang. Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl- constraint. In Proceedings of the Forty-first International Conference on Machine Learning. 2024.
[12] Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
[13] Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the wasserstein metrie: Performance guarantees and tractable reformulations. Mathematical Programming, 171(1)115-116, 2018.
[14] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances Neural Information Processing Systems, 36, 2024.
11

---

[15] Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 2023.
[16] Claude Berge. Topological spaces: Including a treatment of multi-valued functions, vector spaces and convexity. Oliver & Boyd, 1877.
[17] Charalambos D Aliprantis and Kim Border, Infinite dimensional analysis: a hitchhiker's guide. Springer, 2006.
[18] Ganqu Cui, Lifan Yung,Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Utrrfeedback: Boosting language models with high-quality feedback. 2023.
[19] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile 2023.
Saulnier, et al. Mistral 7b. arXiv preprint
[20] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. 14233, 2023.
[21] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. with pairwise ranking and generative fusion. In The Annual Meeting Of The Association For Computational Linguistics, 2023.
Ensembling large language models
[22] Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as--udges: A comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.00557, 2024.
[23] Yann Dubois, Percy Liang, and Tatsunori Hashimoto. Length- Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024.
[24] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems, 26, 2013.
[25] Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V Le, Qijun Tan, and Yuan Liu. Evolving alignment via asymmetric self-play. arXiv preprint 2000022411.0006.. 2024.
[26] Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of Im scaling based on human-generated data. arXiv preprint 22t:::2221.043222 pages 13-29, 2024.
[27] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.013066 2024.
[28] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv: 14734, 2024.
[29] Jiwoo Hong, Noah Lee, and James Thomee Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11770-1189,2211
[30] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[31] Jacob Limanov
Arjun Prakash, Denizalp Goktas, Nora Ayanian, and Amy Greenwald. Sta-rlhf: Stackelberg aligned reinforcement learning with human feedback. In Coordination and Cooperation for Multi-Agent Reinforcement Learning Methods Workshop, 2024.
[32] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Proceedings of the Forty-first International Conference on Machine Learning, 2024.
12

---

[33] Igor Melnyk, Youssef Mroueh, Brian Belgodere, Mattia Rigotti, Apoorva Nitsure, Mikhail Yurochkin, Kristjan Greenewald, Jiri Navratil, and Jerret Ross. Distributional preference align- ment of llms via optimal transport. Advances in Neural Information Processing Systems, 2024.
[34] Yuheng Zhang, Dian Yu, Baolin Peng, Linfeng Song, Ye Tian, Mingyue Huo, Nan Jiang, Haitao Mi, and Dong Yu. Iterative nash policy optimization: Aligning Ilms with general preferences via no-regret learning. In The Thirteenth International Conference on Learning Representations, 2025.
[35] Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models self-improve with general preferences. arXiv preprini arXiv:2404.037/55 2024.
[36] Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, et al. Nash learning from human feedback. In Proceedings of the Forty-first International Conference on Machine Learning, 2024.
[37] Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. In Proceedings of the Forty-first International Conference on Machine Learning. 2024.
[38] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint
2024.
[39] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 16944, 2023.
[40] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877. 2023.
[41] Yizhong Wang. Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information 2023.
Processing Systems,
13

---

Organization of the Appendix.
Section recap LLM alignment and data-efficient methods, as well as the Game--heoreee alignment methods. And dicuss the connection and distinction between SGPO/SSAPO with them.
Section Brevvssis the core definitions and properties of the I-Wasserstein metric, including a statement of the Mittorovich-Rubinsste duality.
Section restates and discusses the regularity conditions needed for our theoretical guarantees, such as compactness and Lipschitz continuity.
Section D provides Proofs for the existence and convergence of the Stackelberg equilibrium, as well as the regret bounds for SGPO and comparisons with DPO.
Section D presents the SSAPO algorithm in pseudocode and includes analysis of its computational complexxty.
Section remarks the approximation effects of SSAPO algorithm design on SGPO guarantees
Section gives supplementary information experimental setups, hyperparameter choices, grouping strategies for DRO, and other implementation details.
Section|H]illustaan additional qualitative comparisons of model outputs, highlighting the differences between DPO, SPA, and SSAPO in practice.
discussessse the potential broader impact of this work.
A More Detailed Related Work
LLM Alignment and Pata-EEficient Methods Aligning large language models (LLMs) with human preferences is central to modern deployments 2 While Reinforcement Learning with Human Feedback (RLHF) trains a reward model and then maximizes under KL constraints, typically requires massive auman-annooatee data. Recent alternatives focus on directly fine-tuning LLMs from pairwise preference data without an explicit reward model. Notably, Direct Preference Optimization (DPO) 5 derives a closed-form surrogate objective that recovers RLHF's solution but avoids a separate reward modeling stage. Subsequent works simplify or exteennd this pipeline; for instance, Ethayarajh et al. 27 remove the need for pairwise labels by adopting human utility model, while there are also works introduce novel optimization objectives to handle different preference formats. Despite progress, these approaches still rely on large-scale preference annotations, making label-efficiency a key challenge. To reduce the reliance on expensive human labels, several methods have explored letting the LLM or an auxiliary model generate and rank unlabeled responses, thereby creating synthetic preference data 21 8 11 9.. However, many of these approaches assume accessibility to a reliable wall-aligned "judge", which be prohibitive costly in realistic scenarios. To address the cost bottleneck, Kim et al. (9] propose Spread Preference Annotation (SPA) framework that starts from a small seed of human- annotated preferences and iteratively ex expands the dataset by self-annotation. Our work is closely related to SPA: we replicate its experimental setup by using the same small-scale seed preferences and iterating between new response generation and preference learning. However, our Stackelberg perspective considers the inaccuracy of self annotation and explicitly defends against worst-case preference shifts. Empirically, we show that this game-theoree distributional approach yields stronger label efficiency.
28
29
30
Thene-Theoreticcmmm Alignment Methods An emerging body of work has begun to frame preference alignment of L.LMs through the lens of games. A conceptual similar work [31 propose Stackelberg Aligument RL.F.. However, their nested gradient-based heuristic does not guaranteed to converge to the equilibrium. While we prove our updates for the leader and follower converge to an equilibrium. Meanwhile, Ye et al. (25) present a framework that casts prompt-creator and solver asymmetric players an evolving game, the differences between our work we focus on evolving the distribution of the responses, while they focus on evoling the distribution of the prompts. SPIN 32 use self- eelf-pla to iteratively refine a policy without additional human data, however they assume accessible to adequate supervised fine-tuning (SFT) data. Other works adopt Nash or minimax formulations: Melnyk et al. 33 study alignment via an optimalltransport objective to capture distributional preferences, Zhang al. 34 and Rosset et al. 35 formulate alignment as two-player game aiming for Nash policy, and Munos et al. [36] proposes Nash learning from human feedback" by treating the policy and a competing policy iterative players. Likewise, Swamy et al. (37,. Wu et al. (38) introduce self-play preference
14

---

optimization methods in which two policies repeatedly compete under a constant sum setting, They demonstrate promising performance on synthetic and text-based benchmarks, but typically set both players as policy vs. policy. By contrast, our SGPO framework focuses policy vs. distribution: the leader policy maximizes preference likelihood, while the follower adversarially reweights or shifts the empirical preference distribution. This setup offers a distinct distributional robust-control view, leading to tight theoretical guarantees (e.g., (boundeddd regret) and a practical algorithm (SSAPO) that is readily integrated with self-annotation. Hence, method complements the "policy vs, policy" family by delivering strong resistance to noisy or instribution-mismatehie preferences at small annotation cost.
B Preliminaries on the Wasserstein Metric Space
Wasserstein (or Earth Mover's s) distances are widely used in robust optimization and optimal transport to measure how far two probability distributions are from one another [12] Below, we give concise
overview of the 1- 1-Wasserstein metric on subset E C R' We also recap the Kantorovich-Rubing
duality (Lemma (.2m).BB2...... B.2). which is central several our regret and robustness proofs in the main text.
B.1 Definition of the 1-Wasserstein Metric
Let M(3) be the space of all probability distributions supported on such that
Eg F [||=|]] - / ||E|| dF(()) co.
In our setting, ||-|| can be any norm on R", typically the Euclidean norm (although other choices are possible). Definition B.1 (1-Wasserstein Metric). For two probability distributions Fi, F2 .MEE,, the 1- T-Wasserstein distance (often just called "the Wasserstein distance") is defined as
( ||5 - -E2 <2|(d=((1.E2)
W1 (Fi. ==
inf TEI((F)PP)
(10)
where [IF1, F2) is the set of all joint distributions on E E whose marginals are F1 and F2, respectively. Intuitively, a specifies how "mass" transported from points the support of Fj to points in the support of F2, and || 22 is the cost of moving unit of mass from E to f2.
Domain used in this paper. All Wasserstein balls in our analysis live in = over scalar preference gaps = Rr (Fw.3t) induced by a fixed prompt æ; we do not transport or raw sentences. Robustness with respect annotation noise through the induced gap distribution.
Equivalently, one can interpret the Wasserstein distance as the minimal cost of transforming the distribution Fi into Fz when the cost of moving unit mass from to & is ||[ - E2ll- This framework underpins many distributionally robust methods, including the SGPO formulation in our paper.
B.2 Kantorovich-Rubinste Duality
A crucial result for the J-Wasserstein distance is the Kantorovich-Rubinste duality (Theorem 5.9 in Villani et al. [2], which states that the infimum over transport plans (as in Definition B.] is equivalent to supremum over 1 I-Lipschitz test functions. We use this lemma extensively to derive Lipschitz-base bounds in the main proofs (e.g., Theorems 2.6).
Lemma B.2 (Kantorovich- Skanorovvch-Rubiistenn Duality). Let Fi, with finite first moments. Then the 1- Wasseesstein distance (TO) admits the the following dual representation:
(2FFFFF)
sup
(Eq.F,If(EE) [gg-r,f((E))
(11)
I
where the supremum is laken over all --Lipschitz.functionn f: IR, i.e. functions satisfying
[ ((E)) < |IE E'Il vg.E'EE.
Lemma B.2 underpins many of our theoretical arguments, particularly in bounding the impact of perturbations measured in the W, ball 24 (ā(T)) in gap space, via [Ea[f]- -Ee(f] S (a.55) for 1-Lipschitz f. As shown Section of our paper, simplifies comparing Er [f] and [p[[]] when is Lipschitz in model parameters responses.
15

---

C Regularity Conditions for Stackelberg Game Preference Optimization
Setup. Let II denote (parameterized) class of policies n. Given preference triples (a', yoe v2) for i = :,,...N, define the empirical measure on triples P= Σi= wtt and the gap map AR, (I, Moo wE) == R2 (2, - R. (E, The corresponding empirical gap distribution is á(н) = (AR.) ) x T san (x' wf. E P(R). For O, we denote the 1-Wasserstein ball in gap space by Ue (á(()) fa
st. where the ground metric is the absolute value on I.. The leader payoff P(TT,a)= = Re Doga (E) and the follower chooses EMe (d((T)).
Standing assumptions.
We use the following minimal conditions.
(A1) Compactness.
IT is compact (with respect to the topology induced by the model parameterization).
(A2) Continuity of rewards. For each (I.y), the map πT> R2 (z.y) is continuous. Consequently, for each i, the map x + R ( (x'
-V-Dii is continuous.
(A3)
Continuity of the push-forward center. The map ) ( P continuous in the topology induced by W1; particular, Wi (á(z)),ā(T))) -00 whenever I1 -> T2. (This actually follows from (A2). See Lemma
(A4)
Gap-link function. f(E) := logo (E) is continuous, concave, and I-Lipschitz on R.
(A5) (Optional, for local convergence.) There exists À > such that the proximal leader objective := a== [f (AR, (z.yww 2º [ - (T,
is u-strongly concave in ((x) neighborhood of solution, uniformly over E El, (á (T))).
||? )
(A6) Optional,, for bounded margins or clipping.) Either |A,, Yw: ye)l S B for all (z.yw-Bl -t) and π E II, or margins are deterministically clipped to a window [a.b]. All PWL error bounds are computed on this interval. The Rmax in the paper can be set to B/2.
Lemma C.1 (Continuity of the center in W1). Under (A2), for any æI ,72 IT,
(((d(T)..(Taa) S ARR (a' (w w ARTrs (x' swoii)
In particular, ι a (((T) is continuous W1.
Proof. Couple the Dirac masses in d(T1) and ((T2) mndex-wise The claim follows because the
I-Wasserstein distance on R is bounded above by the average transport cost under any coupling. O
Lemma C.2 (logo 1-Lipschitz and concave). For all & E R, de loga(() a(-E) E (0.1) and Hence logo is 1-Lipschitz and concave.
dez -
Proof. Direct differentiation; (o(-)) < gives the Lipschitz constant and the second derivative is nonpositive.
Lemma C.3 (Compact follower feasible set). For each fixed the set U, (á(TT) c P(R) is tight, closed Wi, and thus compact.
Proof. On the Polish space (IR.I-)), closed and - bounded sets of probability measures are relatively compact; tightness follows from Markov's 's inequality under bounded first moments, which hold for all a with Wi (a,((())) Se. Closure is standard for balls..
Remarks. (i) No Lipschitz condition on R2 in the output space y is needed because robustness is posed in gap space R. (ii) Assumption (A5) matches practice (small stepsizes/weight decay) and is only required for the local rate. (iii) Although neural network parameters / E &d are technically unbounded, many theoretical analyses restrict ( to a large but bounded ball (via a norm constraint) or rely on a coer- cive objective to prevent unbounded parameter growth. Hence, requiring IT to be compact common in theoretical treatments. In practice, gradient-bass based optimization does not typically push |Ø]] to infinity.
16

---

D Theoretical Results
D.1 Preliminaries and basic lemmas
Notation and spaces. All random variables this section take values in (R,I-1) I)) equipped with the Borel olalebrra. For a policy и and i...d. samples {(xt Wi) ~ ~, define the empirical gap distribution
N
a(x) := (ARx) P
where &::= AR (x D v/) ER.
N
For é 0, the (1-)Wasserstein ball around dr(=) is
Ue (ā(T)) = { (oEP se
We write =={)== (5) (concave, 1-Lipschitzz and ((5)= -n)) -logo (convex, 1-Lipschitz). When a piecewise-linean (PWL) surrogate used, we set
ē(E) == max (x(E),
ex(E)==akt +bk:
SKSK
chosen global supporting tangents so that ((S) S ((E)) fll.ll
Lemma D.1 (logor is concave and 1-Lipschitz). For every E R. de (oogo(()=o((-- (0.1) and
d² de (()=(())))((( 00. Hence u({) (f) is concave and 1-Lipschitz.
Proof. Recall (5)
1 1te z
Then
o5( (5)
logor({)
-
dg
a{E)
({{)
Hence ce 1o) < 1 for all &, so logo is I-Lipschitz: (logo()) -logo(() |8 - by the
mean-value theorem. Further,
d²
dga
so logar is concave.
Kantorovich-Rubinste (KR) duality for W1. For any 1-Lipschitz ht h:
and (RR...
E. [h] R2 [b] s (a,
We use this both as continuity tool and as tight transport sensitivity bound.
Lemma D.2 (Continuity of the empirical-center map). If A (a' vu vf) continuous for each i [N], then any T2,
1 N 1=1
W1 (((T)))((TT)
< AR - -AR
hence á=((T) is continuous in the W1 metric.
Proof. Write a(T,) Σ 6g5 with EL == AR, (x* wi- for j {1,2). Define the coupling
7= + 8 g. By definition of W2 (optimal transport with cost |- on IR),
N
< |I- (((((T)).((m))
1
(2))
A AR - ARR
N
Ifforeach i the map + AR (x' v is continuous (Assumption (A2)), then W, (((T))).(((T)) +0 whenever n. - i.c., T ((T)) ) is continuous in the W2 metric.
0
17

---

Lemma D.3 (Compactness of Wasserstein balls about empirical centers), For fixed ıT, the feasible follower set (&(T)) is nonempty, tight, closed in W,, hence compact.
Proof. Fix Let (oc
(i) Nomemptiness. Trivially 6(T) E.
(ii) Uniform first-moment bound. On R with ground metric |-), we have Wa(a,,o) f fl By the triangle inequality,
f
(xldā(TT)
so the family B has uniformly bounded first moments.
(iii) Tightness. For any R>
c+ (+llddm((m)
(1x) > R) R
VaEB,
R
by Markov's inequality. Hence B is tight.
(iv) Closedness in W1. If an E with Wi(an, a) O. then Wi(a, a(T)) s
liminf, [Wi (a,a))))++ (a sc, a and B closed.
(v) Compactness. On the Polish space R, Prokhorov's 's theorem gives that tight families are relatively compact in the weak topology: the uniform first-moment bound tightens this relative compactness in Wi (since W1 convergence is equivalent to weak convergence plus convergence first moments on IR). Combining relative compactness with closedness in W1 yields compactness of B in (Pi (R),
O
D.2 Existence of' Stackelberg solution
Theorem D.4 (Existence). Assume: (AI) IT compact; (A2) each π R, (x* www is continuous;
(A3) hence + a(T) is Wi-continuous (Lemmna D.2 ; (A4) u(-) is 1-Lipschitz and concave (.emma Then
max TEll (&(x)
min
Eg [u{E)]
admits solution ,a").
Proof. We verify the conditions of Berge's maximum theorem step by step.
Step 1 (Follower minimizer exists for each fixed T². For fixed æ, Lemma D.3 shows the feasible
set (á((T) is nonempty and compact in W,. By Lemma D.I u is 1-Lipschitz, hence a++ [E,[(] is continuous under W1 (KR inequality). Therefore, the follower problem admits minimizer a (ar)) Cargmin.eu (v)..
Step (Continuity of the feasible-ser correspondence π). By Lemma D.2 T- is continuous in W,. The set-valued map x (á(T)) thus varies continuously in the Hausdorff metric induced by W, (closed balls move continuously with their centers in a metric space), particular upper hemicontinuous and compact valued.
Step (Upper semicontinuity of the value map). Define V(x) == min (&(r)) Eg By Berge's maximum theorem (compact-valued) upper hemicontinuous correspondence; continuous objective), V is upper semicontinuous on II.
Step 4 (Maximizer exists). Under (A1), is compact. Since V is upper semicontinuous on compact set, it attains its maximum at some x² II. By Step 1, there is realizing follower a* argmin (u.l..
Therefore the problem admits solution (I** (r*,,"").
D.3 Local linear convergence of alternating updates
Consider the iterates with a (Euclidean) proximal leader step:
0++1 E arg min (Ell((((R)))
Ex [u{]],
Fe++1 E argmmax xell
18

---

Theorem D.5 (Well-posedness and local linear convergence). Assume (i) the leader's proximal objective y-strongly concave (TT) uniformly over E (ā(++) on neigghboroood of solution, and (ii) the map ι (á(()) is Lipschitz in the W -Hausdorff distance near n². Then the update map (TT,,0)) >> (7441,04+1) is contraction a neighborhood of (x ,0"), and the iterates converge linearly to (x* (x*
Proof. Let the follower best-response be any measurable selection a² (T) argmin, (a(=)) [u], whose existence follows from the measurable maximum theorem since the correspondence is compact valued and upper hemicontinuous.
Define the proximal leader map iterate .T.
T(a=)))
argmax G(TTTT,e))) nell
(( (u (u(RR+)) 100)) -
Step (Follower map is Lipschitz in the center, hence in π locally). Fix n , and let a, = (Ty)) Because U. (ā(T) is closed ball in (Pi (RR, for any B can project it to the closest point in the ball (metric projection is I-Lipschitz). In particular, for 3 = we have
distw, (on Me (Tz))) s (i((()))
Let 0r ( be nearest point a1 Using 1-Lipschitzness of and the optimality of az at I2.
Eaz [u] s Eag [u] S E., [u] + WW z E., + (
Symmetrizing the roles of 1 and gives by triangle inequality
Wi S 2W1 (((T1),((T2))
Thus the follower map is Lipschitz in the empirical center with constant Ly 22. By Lemma D.2
N
Wi (ā(T))) S
AR
-AR
N
are LR- Lipschitz for I...., .N. Then locally Assume (locally around the target) the maps 0 A R
Wi ((( s Ly (á(m))(() S L)
Step (Leader prex map is Lipschitz in a under local strong concavity). By (A5), for each fixed m the map ()) is p-strongly concave in ((r) a neighhorhood of the solution, uniformly (á.r))) Let = (a); for j (1,2). Since is 1-Lipschitz and AR, is continuous in there exists Le (local) such that
V
VoR.,Iu(ARR
< (a1,())))2) for π near the solution.
By standard stability of maximizers under strong concavity (e.g., by the implicit function theorem or strong monotonicity of the gradient mapping), we obtain the Lipschitz dependeece
Le μ
(())))
< (a(a)a0)).
Step 3 (Coontraction the composition and linear rate). Set aet := (m.) and Te+ Taa;T)) Let (TT aa*) be Stackelberg solution; then a*= a²²mm(==") and n² == (ma Combining Steps and 2,
(((=(41)-((T))) < Lt "i(an,a) < Le - P
-
Choose the proximal weight À (hence the local strong- concavity modulus μ)) that p:= LLLLRR 1. Then Banach's fixed-pont -oint theorem yields linear convergence:
( < p po ([((m,,))))))))))))))))))))))) -((())]1 for all t to in the neighborhood.
This also implies well-posedness (local single-valuedness) of the composite best-response map in that neighborhood.
19

---

D.4
Worst-case performance drop and SGPO regref bound
Theorem D.6 (Worst-case performance drop). IfaEE(((((T) then 2
(n) [u(E)] -c.
Proof. By Lemma D.1 uis 1-Lipschitz. For any a with (i(a, cc, the Kantorovich- inequality yields
Sa[uI-Baut [aa))]Ba( -Wi(a,a(T)) -c.
Rearranging gives the claim.
Theorem D.7 (SGPO regret bound), Let n² solve max, (a7) Then
sup Regret(TT* ,a) s 2e.
(é(())())))
Proof. Let IT" E aargmax, Eargmax,min,st (((m)) E, [u] and fix any a <(,(((TTT)) Let a. Ea,gmax,E,fuu be the c--optimll policy. By Theorem D.6 D.6
Ex(u]=; S (a(m))/uxx te,
Ea(u]a 2 (B((r))
Subtracting gives
Regret( =a)=Boluls> Ealu]a S (Ba(=)) (u) 2e S 2c.
b0 xf
D.5 DPO regret lower bound under stability assumption
Assumption D.8 (Center stability at the robust follower). Let Ell((á(**)) be a follower minimizer for and T Cargmax, [u]. Assume there exists x S such that W1 (á(=á- ),((TPP))) Sx.
Theorem D.9 (DPO regret lower bound (stability version)). Let == W (a* a(TTD()))))) Under Assumption D.8
Regret (TDPO,a*) 2 6 2 2e.
Proof. Let Wi(a* ( (TDPO)) and recall Assumption D.8 First, by KR and 1-Lipschitzness of E. [u]mg- 2 (uls; -6, Ex- [u] rpoo Ea(moro) [u) +
Subtracting.
Regret(TDro.at 2 [E(trro))lll - (u)poo) - 26.
By center stability and optimality of TH at
Ba(moro) [u]=;, 2 Bain; [u]= - 2 E. Fun|uu]m -k.
Finally, since W, (((((("))"))) ),a) and n² is optimal at a(=²),
Ex ju]ng Ea(r.) [w] Ec
Combining the displays and cancelling the center-optimal term gives Regret(MDPO. a*) 2 2e (using <)).
Remark. Without Assumption D.8 the lower bound can vanish if n recenters too far from à(TDro); the stability phrasing makes explicit the (mild) continuity needed for a linear-in-ò lower bound.
20

---

D.6 Approximation effects (piecewise, grouping. inner tolerance)
Let m(T) = (a() Eg [u] denote the true follower value and let m.Kgrppn denote the value computed with: (i) PWL under-approximmttin l with K pieces, (ii) restricted feasible set Uk ( (ā)() (e.g., via grouping), and (iii) inner tolerance 17.
Proposition D.10 (Monotone tightening in the number of pieces). Fix a reference distribution & on R with finite first moment and radius 2 O, and let (ā) denote the associated 1- 1-Wasserstein ambiguity set (absolute ground metric). Let C: - (0,co) be convex and define { } K21 convex piecewise-linee linear underestimators of of form
&K ISKSK
max
+b
with (x(()=<x+<(() Vg.
such that & (E)) pointwise as K -0 Define the exact and surrogate inner values
v²
inf (a)l(
Ex [logo((()) (E)]= sup [x((()) (EU((A)
UK
sup (ell.
E2 [.
Assume sup (a) B. [((E)] <o.. Then:
(Validity) uk 2 for all K 121.
2. (Monotonicity) uK+1 E UK for all K 21.
3. (Limit) uk 4 K -c.
Moreover, if for some K the supremum in the definition of UK is attained by ok E (ā(a) and (x) holds a---lmass surely, then uk =
Proof. By definition of f= llgor we have inf, [ogol== E. hence w²= = Eal(]
Since f < f, it follows that sup,, R.. sup,, S........... which implies ux v*, proving (a). Because (k+1 2 k pointwise, also [f, [1x+1] 2 sup,.E.. [[ hence uK+1 S uk. proving (b). For (c), by monotone convergence, for each fixed a we have E,, [x 1 [e] therefore sup,, [f, suup, E, [.] and taking negatives yields vic ux-
4u² The final claim is immediate from the definitions.
11
Remark D.11. In our setting ((E) = - log (E) is nonnegative and 1-Lipschitz, so
suP E. Ba) [(((((()))))))))))((()))) whenever ( (ā) is Wi-ball around measure with finite first moment.
Proposition D.12 (Error from PWL underaaprprooiiitiinnn) Let : (R(E) -(x)) 0.
Then
0 M (T) < (K).
Proof. By construction 7 < pointwise. Since u -t, for any feasible a.
E, [u] = - [. < - E., Ex| < sup Eg [xx] mm(()),
a' d'Ell,
so mm ( 2 m(F). Moreover, for any a, E,, [0] - l. ( sup; ((x)))(x())) =: Ap)..()) Taking the supremum over a and flipping the sign gives
S (T) < (p(().
Proposition D.13 (Error from grouping/restriction))) Let dæ denote the directed Hausdorff distance (under W1 ((())) to (á((T)) Then
0 s m ( m(T) dh-
Proof. Let a* E argmin F. [u] and let á be any element of the restricted set ÜK (á(()) satisfying Wi (a" ,ā) Sdh by definition of the directed Hausdorff distance. Using I-Lipschitzness of Ea(u] < Ea [u] < E., [u]+ Hel+Waaa.o I m(T)+ + dh-
min
(
Thus 05 0 (T) ml dd
21

---

Algorithm 1 Stackelberg Self-Annotated Preference Optimization (SSAPO)
Require: Seed labeled set Deedd; wnlabeled data ; Wasserstein radius c; number of linear pieces K; max iterations
1: Initialize policy Bu, set D
2: to
3: (Self-Annotation): From Dunlabelod- sample prompts, generate & rank responses under mop add new preference pairs (ym,)ee) to D.
Ro, (vf), and let 4: (Form a(T=)):: For each (Viw v D, define č. = Ro, (vin)
5: (Convex Pieces): Choose linear functions (x() such that E(E) max (x(E) I llog (5). Choose knots { in G-space over bounded interval (a,, by] (e.g.. a-= min, - by max,i++ with r > 0, or empirical (a,1 -a) quantiles). Define Eg as tangents (or chords) from below -E) at those knots.
6: (Worst-Case Distribution): Solve the DRO finite convex program
aj E arg (a(L.)
Ea [[(E)]
By Theorem 3.3 oz is discrete with atoms (é, - 9i } and weights /N.N..
st
7: (Policy Update): Let w!!" !f)
= >k=1 18k ((d)
Update O2+1 by minimizing the weighted logistic loss
1
llogor ( (o vi)
N
(optionally with KL or weight decay), via standard gradient methods.
8: end for
9: return Or (final policy).
Proposition D.14 (Cumulative approximation bound). For all n,
0 S (m)) S (pi(K) +
Proof. Combine Proposition D.12 (WWL. & Api (K)). Proposition D.13 (restriction gap S d and note that inner solver with tolerance 7) perturbs the value by most 7). Errors add up, giving 0 M.KKKKK grp.g ( (TT) -m(r) x Ap((K) du +77.
Theorem D.15 (Effect on regret guarantees under approximations). Let ã be produced by SSAPO with K PWL pieces, G groups, inner tolerance n. Then, compared to Theorem D.7
sup Regret (t.a)=a)))))) s 2e +[
aeu, (a(E))
Proof. Let n be the SGPO optimizer and ā the SSAPO solution under approximations (PWL with K pieces, grouping, and inner accuracy 7)). For any a (ā(ff),
Regret(f f. =Ea [u]=;: EE. [u]a < [a(+)) [u]x; Ea(+) +2e
by Theorem D.6
Replacing the center objective m(-)
(-) by its approximate counterpart grpp. incurs (
most A pl
K
+ du + 7 and the same at the comparator, hence the extra ( (d O
E SSAPO algorithm and Analysis on Computational Complexity
E.1
The SSAPO algorithm
22

---

E.2
Computational Complexity of SSAPO
In this subsection, we analyze the computational costs incurred by each step of the Stackelberg
Self- Leff-Anntatted Preference Optimization (SSAPO) algorithm (Algorithm[]) We denote:
N: the total number of preference pairs in the dataset D a given iteration,
K: the number of linear pieces used in the convex piecewise approximation of -1ogo
T': the total number of outer iterations for SSAPO.
We assume each iteration refers to Steps 1-5 of Algorithm
Step (Seff-Annotation) The cost of self-annotation depends on the number of prompts and the policy's s inference procedure. Let M, denote the number of new prompts labeled at iteration l. Generating and ranking responses under Te, typically dominates this step. If:
G5 is the number of candidate responses generated per prompt, Cinference is the average cost of single forward pass (token generation) under To.
then the time complexity for Step is approximately
(M,, Cinference).
plus any overhead for storing new winner-loser pairs in D. Since the number of newly added preferences grows over iterations, N itself typically increases from iteration to iteration.
Step 2 (Forming ((T:)) Once D is updated, we compute E = Ro, (win) ) Ro, (vtt for each pair. The cost here depends on:
N,, the current size of D,
Creward, the average cost to compute Ra, (y) Blog
na, (ulx) for given response y. went Rent (M)zz)
Because each preference pair requires evaluating Ro, on vt). this step has complexity
O(N-Crewadey
In practical implementations, Re, (v) often just reads off the prgpobbbaillites from Te, and Tret the final tokens, making Creward similar to single forward-pas cost per response.
Step (Convex Piecewise Approximation) We construct linear functions (E( such that = m log In principle, one can precompute these K pieces over a small interval (e.g., [0,1) once and reuse them in every iteration. Hence, the complexity for updating verifying the piecewise function iteration t is typically: O(K), assuming {{(*) Jk= fixed or
can be quickly adapted based on the range of ( This step is therefore relatively cheap compared to distributionally robust optimization.
Step 4 (Worst-Case Distribution) Step 4 solves the distributionally robust optimization (DRO) finite convex program
of arg OElL (T())))
E, [((E)]-
The naive formulation (per 133 becomes high-dimensional if N is large, because each sample point & and each piecewise component l introduces auxiliary variables (such ssk(sk)). Concretely, the number of decision variables can scale like O(N KK), and the resulting linear or convex program might require ((NN)))) time in the worst case for some exponent >>>> (depending on the chosen solver and constraints).
However, several factors can reduce this cost:
Approximate Solvers. In practice, specialized cutting-pla plane or primal-dual methods solve these DRO problems more efficiently than the worst-case theoretical bound.
23

---

Grouping Heuristics. If one partitions the N samples into smaller groups (each of size G<N) N), the complexity per group (((G))"). Then one aggregates M & group-level solutions. This lowers the complexity significantly if G N.
Hence, the worst-case step here is often ON K) to ()))))) but can be much more tractable in practice with grouping or approximate methods. Regardless, Step 4 typically dominates the iteration complexity for large N.
Step 5 (Policy Update)
Finally, we minimize
E B (E))
0g Earg max aEu (a(n)))
Bę (llgor)())
(by the L = log a reparameterization)
practice we compute gradients via the chain rule {= (z,Rg(Z. and reweight per-pair contributions to match of (see Alg. D
Assuming each of the N preference pairs in o can be sampled over multiple epochs. In many implementations, N can be large, so the training complexity depends heavily on how many gradient epochs or passes one uses at iteration L.
Overall Complexity per Iteration Putting the above pieces together, let us summarize the dominating terms:
1. Belf-Anmotttion (Step 1): (M (GM.- Cinference).
2. Forming ((Tz) (Step 2): (N-Creward)
3. Convex Piecewise Approx. (Step 3): (K),
4. Worst-Case Distribution (Step 4): O((NK)) in the naive case, often reduced by grouping.
5. Policy Update (Step 5): O(N-Creware -reward (number of epochs)).
If we denote the cost of solving the DRO subproblem by CDRO (N, K) (which could itself be significantly reduced by grouping into subproblems of size G), then each iteration of SSAPO costs approximately:
(M, Me Cinference + N Creward + CDRo(N,K)++.))
In most scenarios, either the distributionally robust optimization (Step 4) the gradient-based policy update (Step 5) will be the main bottleneck, depending on solver implementation and whether grouping employed.
Total Complexity over T Iterations Over total iterations, we multiply the above per-iteration cost by T. Additionally, note that N can increase each iteration if new self-annotated preferences are continuously appended to D. Denoting N, as the dataset size at iteration I, the total complexity
from Steps 2-5 roughly V O(N,- Creward) + CDRO plus the self-annotation cost from Step 1. If N grows in controlled manner (for example, linearly in f), the cumulative cost can be bounded accordingly.
Practical Guidelines.
Grouping for DRO. To handle large N, recommend partitioning the data into multiple groups G & N. The overall complexity then becomes O CDRO(G, K)), where M=N/G, which can be significantly faster in practice.
Caching Log-Probabilities. The reward Ro, (v) can be computed from bogabrobabilities of To, and Tye.. Caching or reusing these values may reduce Creward.
Adjusting K. Increasing K refines the concave approximation but grows the size the DRO problem. Hence, K is hyperparameter balancing approximation quality and computational overhead.
Overall, the time complexity of SSAPO grows with N. K, and the iteration count T. By employing grouping and efficient solvers, We can typically achieve robustness benefits without incurring excessive computational cost.
24

---

F Approximation Effects of SSAPO Algorithm Design on SGPO Guarantees
Setup. Section 2 establishes guarantees for the Stackelberg game
max min
E Rg [g]]
(12)
aEVL ((AR,)
where ç Rx (I.yww Me) ) is the reward gap and the Wasserstein ball is taken over the push-forward of the empirical pair distribution by AR This choice matches the semantics of preference robustness and was the condition attached to acceptance. We abbreviate ((E) & (ogo note that is 1-Lipschitz and bounded on | mmaxx Rmax] under the standing bounded-reward assumption from Section 2.
Throughout this section we write u({) == log (g) and identify the PWI. loss-approximation gap (p) supgeja.b] (e(E) lx (E)) with the symbol 8x used in the main text. Likewise upper bound the grouping error (grp(e) by the directed Hausdorff distance du between Ur and its group-restricted surrogate.
Goal of this section. SSAPO instantiates 12 with three pragmatic approximations: (1) K -ttangent convex under-approximation of the - log loss used to form a tractable DRO subproblem. (ii) a group-restricted Wasserstein ball that disallows cross-group transport for scalability, and (iii) inexact solves (tolerance '7) of the follower and leader subproblems. We quantify how each approximation perturbs the clean guarantees of Section 2 and provide principles for choosing (K.
F1 A bias decomposition for SSAPO
Let V(T)) é min E,, [e(E)] denote the ideal follower value for fixed policy x, and let V(T) the value obtained by SSAPO with all approximations enabled. Then
be
(iT) (TTT)
(+<()))V(()) + (Vx.a(t)-Vx(T)) (V(T) (V(T)-KK,C) (V,,))V,,()))
(13)
optimism induced by approximations
piccewise
group restriction
tolerance
where VK is the value when & is replaced by its K K-tangent surrogate and Vx.G additionally restricts the follower a group-wise uncertainty set (defined below). Fach term in (3) is nonnegative (the approximations weaken the adversary) and admits simple Lipschitz control.
F.2 Effect of the K-tangent surrogate
Let E(-) be the convex piecewise-linear under-approximmatin of - log built from K tangents, constructed [-B with B = 22Rmax Define the uniform approximation error
sup (=logg((E)
EE(- B.B)
Because o = logo -(-logo). replacing & by -ē in the follower objective can only increase its minimum:
0 (=) 6K.
(14)
Thus, the K- -tnageent surrogate yields one-sided, additive slack 8x in the inner value and therefore at most ōg optimism in the leader's s objective. In practice, we found K = 6 strikes stable accura- cy/conditioning trade-off, whereas K T 7 can degrade numerics without reducing 8x appreciably (solver instability rather than approximation error).
Design takeaway. Choose K so that åx is below the statistical noise floor of the preference estimator on [- 2 Empirically. K= is robust default; increasing K past this point complicate the convex program and harm solver stability.
F.3 Effect of group-restrictd Wasserstein uncertainty
Partition the support of the empirical gap distribution a(x) into disjoint bins {5,] in & for SSAPO). The group-restricted follower can transport mass only within each Sg-
(uniform
45t A {a=>x To=>ag ag ay Ele, (i,))) EGse sc Cu (ā(TT))
25

---

Let [e) be the set of optimal global followers and define the projection gap of the restriction
ógg (c)
sup inf Wi(a" (a)"",)..
a ET(() RELE"
By Kantorovich-Rubiin duality and the I-Lipschitzness of σ,
0 5 VK,a(T) Vx( S (c)
(15)
On the real line, uniform (equal-mass) binning gives simple control ógp(e) S wG. the average within-bin width E; hence the restriction error decays as O(1/G) as bins refine. This formalizes the empirical guideline that G' between 10 and 10 preserves robustness while enabling embarrassingly parallel solves.
Design takeaway. Use G' [100,1000) (sweet spot 100- (00-300): it keeps Wa small, retains near-global robustness, and maximizes parallel throughput. Disallowing cross-group transport weakens the adversary only by at most bin-width in Wi-not by itself-so the O(e) regret from Section 2 is intact up to an (1/G) term.
F.4 Effect of inexact solves
Suppose each follower problem is solved to absolute tolerance 7) and the leader update attains an 1y-accurate step (e.g., via a proximal DPO update). Then for any n,
V(T) - VK. (*) S..
(16)
and the cumulative leader suboptimality over '" rounds contributes at most (17) to the final value, consistent with the conerrennve picture reported Section In SSAPO. the follower solved offline and in parallel with modest wall clock, so 17 can be driven small at negligible training-loop cost.
F.5 Putting the pieces together
Combining
14
1.5 and [16
13 the approsiiaatioriiiicellooiiiiiiii in the follower value obeys
0 < ((T))
<
Egm
(17)
K Ktaangent
grouping
tolerance
Therefore, the leader who maximizes V(TT) enjoys the same O(e) robustness as in Section 2, up to an additive 0g8x + + slack. Because ōx and ry user-controlled and óge shrinks with G, the theory carries over with explicit, tunable error bars.
Practical summary.
Where the ball lives. All results hinge on placing Ue on the gap push-forward (AR,), (not on (Z.))e.ye)). This keeps the geometry 1-D and the Lipschitz constants sharp.
K tangents. K gives stable frontier; K - may hurt due conditioning rather than approximation quality. Tune to make 5x sub-dominant to data noise.
G groupss Choose G [100, (sweet spot 100-300) make the group-restriction gap (gp(() S पh negligible while exploiting parallelism.
Tolerance. Solve the offline follower to tight v) (cutting- plane typically converges in 10-20 iterations), so training time overhead is small and approximation slack is dominated by statistical error.
Scope. These approximations target training- time robustness to noisy preferences; they are orthogonal to inference-time adversarial prompts and do not weaken that disclaimer.
G
More Details of Experimental Setups
G.1 Detailed Experimental Setups
We introduce more detailed experimental setups in Section 4 as follows.
26

---

Datasets. For preference learning, we employed the UltraFeedback dataset [8] aligning with prior research [3 Specifically, we extracted a seed dataset comprising 2K samples (3.3% of the total 60K training samples), which included prompts, responses, and ground-truth preference labels. These ground-truth preference labels are referred as gold labels in [able[] The remaining training samples were then partitioned into three subsets of 8K, 20K, and 30K samples, retaining only the prompts. These subsets were utilized as the prompt sets for the 1st, 2nd, and 3rd iteration stages, respectively.
Models. Following previous work 19. we primarily conducted our experiments using the supervised
fine-tuned Mistral-7B-0.1 model [9] as the initial model Miit- Specifically, we used the open-sourced modef" that follows the recipe of Zephyr [39 and is fine-tuned on the instructions of UltraChat [20]. In T 1. also used LLaMA-3-8F to validate the compatibility of our method across different models. We used the generally fine-tuned models as there are no models that have been fine-tuned
on the UltraChat dataset.
Evaluations. Following standard practices for aligning LLMs, we employed two primary evaluation benchmarks to assess model performance. First, we used AlpacaEval 2.0 [4 23]. a benchmark designed to approximate human preferences in instruction-follooiin tasks. This evaluation involves 805 diverse instructions sourced from multiple datasets, where responses from the model under test are compared against those generated by GPT-4 44) to determine win rates. To address potential biases related to response length-a known factor influencing LLM preferences 41], we report both the original win rate and length-controlled (LC) win rate. The LC win rate is calculated using a regression model trained to neutralize the impact of response length, thereby focusing on the quality of the generated content 23
15 to evaluate the model's capabilities across a broader range of tasks. MTT-Bench assesses chatbot's performance in areas such math, coding, role-playing, and writing through multi-turn interactions. Responses are scored by GPT-4, providing a comprehensive measure of the model's proficiency in key LLM functionalities. Together, these benchmarks offer a robust evaluation of how well the model aligns with human preferences and its effectiveness in real-world applications.
Second, we employed MT-Bench
Implementation Details. In the initial alignment phase, we train the model using Direct Preference Optimization (DPO) on a seed dataset of 2K samples to obtain the base model To. Following this, we conduct 3 iterative stages of data expansion. In the i-th iteration (i= 1,2,3), we generate preference data by independently sampling two responses for each prompt using a temperature of 0.7 and labeling them as chosen or rejected through R(z.y), resulting in a preference dataset { (N is the size of the i-th prompt set). Following SPA (9. we restricted the maximum token length for self-generated responses 300 tokens. This limit corresponds to approximately 900 characters. To model the worst-case distribution program, we define a set of linear functions (x) for k=...,, (the family of tangents of the loss function at the K K-equipartition of [o,1)) We solve the associated optimization program using the Sequential Least Squares Programming (SLSQP) method. The group size G is set to 100 unless otherwise specified for parallel computation of the convex program. Finally, we update the policy model by minimizing the reweighted loss to get Ri. ensuring improved alignment with the desired preferences.
(x # log(
For Mistral-7B-0.1, We set learning rate = 5 x 10-7 and DPO hyper-parameter 3 3=00 0.1 throughout the entire preference learning process. We conduct 3 epoch for the initial DPO training and 3 iteration for SSAPO game play (leader-follower updates).
Hyper-parameters for Different LLMss.
For LLaMA-3-8B. We set learning rate = 1 1 and DPO hyper- syper-parameter 3 = 0.05 throughout the entire preference learning process. We conduct I epoch for the initial DPO training and 2 iteration for SSAPO game play (leader-follower updates).
MaaeLLLMMMMaaLLLLMA3...
27

---

G.2
Construction of Seed Data
Seed data (e-g. the initial labeled training data) has an impact on performance of self-training algorithms. In our experiments, we explore two different methods of selecting seed data from the original training set, namely UltraFeedback dataset [18] Ground truth labels (e.g. the preferred and dispreferred response) of the selected seed data retained, while the labels of the remaining data are discarded. As a result, our proposed method use only a small proportion of ground truth labels.
Random sampling. We select seed data from the entire training dataset randomly.
Farthest point sampling. Intuitively, the distribution of the seed data should "cover" much part of the entire training dataset as possible, such that subsequent self- self-training iterations won't 't suffer from distribution shift. With this motivation, we first leverage the SFT model obtain embedding vector for each prompt in the dataset, and then use farthest point sampling algorithm to select a subset of the embeddings (and corresponding preference pairs) seed data. Specifically, we feed each prompt into the SFT model, and extract the hidden state of the last token of the last layer as the embedding. We use cosine similarity as the distance metric for embedding vectors.
Stability across runs. Using farthest- point sampling (FPS) for seed selection reduced performance variance by approximately 40% % compared random seeds in experiments, by covering the preference space more uniformly.
In our implementation, we find that Mistral-7B model is not sensitive to either seed data sampling method. And LLaMA-3-8B model enjoy a slightly better performance with respect to the farther point sampling.
G.3 Infrastructure
For all experiments, we utilized 4 A800 GPUs. Under this computational resource, generating responses for 10K prompts takes approximately 15 minutes, and preference judging for generated responses takes about 30 minutes. Solving worst-case distribution with Eq. ttakes about 40 minutes. Policy update takes about hour per 10K training data.
H Qualitative Examples
In this section, we present qualitative examples the responses from the models trained under different methods: DPO (only train initial seed data using DPO), SPA and SSAPO (our proposed method). The results are presented below. These examples show that the model trained under SSAPO provides more detailed and precise responses various questions.
Prompt:
The given text must classified offensive or offensive.
You acted really crazy last night.
DPO response:
I do not have any information about the specific context or situation in which the statement you acted really crazy last nightwas made. Therefore, am unable determine whether it is classified as offensive or not. Please provide more context or details about the situation so that I can provide more accurate response.
SPA response:
Based on the given text, it can be classified as "offensive" because uses colloquial and informal expression ("acted really crazy") to describe someone's 's behavior, which may convey negative or derogatory connotation towards the person's 's actions. In formal or professional contexts, using such language may be considered inappropriate unprofessional, and it could potentially offend the person involved others who may be present. Using precise and objective language describe behavior or actions is generally recommended in such situations to avoid any potential offense or misunderstanding
28

---

Therefore, using more neutral or appropriate expression, such as "displayed unconventional behavior last night," may be better option in this case.
SSAPO response:
Based on the given text "You acted really crazy last night," " it is difficult determine whether is classified as offensive or not offensive. The text does not contain any explicit or direct insults, profanities, or derogatory language directed towards the person addressed. The phrase "acted really crazy" could be interpreted as a subjective assessment of the person's 's behavior or actions during the previous night, and may not necessarily be intended as insult or criticism.
However, the tone or context in which the text is used could influence its perceived offensiveness. If the phrase "acted really crazy" is used sarcastically or with condescending tone, it may be interpreted as an indirect insult or criticism aimed at the person's 's behavior or character. Alternatively. if the phrase is used casually or with neutral tone, may be interpreted straightforward observation comment on the person's 's actions without any malicious intent.
In summary, the given text "You acted really crazy last night" may may not be classified as offensive, depending on the tone, context, and intended meaning behind the phrase. Without additional information context, it cannot be definitively labeled either offensive or not offensive.
Prompt:
Verify the correctness of the given statement.
"For all integers j and k, jf and are odd, then jk is odd."
DPO response:
The statement "For all integers j and k, jf and k are odd, then jk is odd" is correct.
Proof:
Let and k be any two odd integers.
Jince and k are both odd, there exist positive integers and b such that:
j = 2a and k 2b
where and b integers.
Multiplyingj and k, we get:
jk= (2a 1)(2b 1)
2(2ab+a+
22
222a* 12
Since 2a and 2b
integers, the product 2(2a*2b 1s odd integer. Also, (2*a 2*b 1) an odd integer since it is of the form 2 + where e the sum of two even integers (2a and 2b).
Therefore, jk odd integer.
Hence, for all integers j and k, ifj and k are odd, then thenjking is odd.
Q.E.D.
SPA response:
The statement "For all integers and k, ifj and k are odd, then jk is odd" correct.
To prove this statement, can use mathematical induction on the sum of the absolute values of and k, denoted by + kk)
29

---

First, consider the base case where [j] + |k] 1. In this case, either j and k == 0 or = and k In both cases, jk= 1, which odd.
Next, assume the statement holds for all pairs of integers and k such that || + |k| = n, where is positive integer. Let and k be any two integers such that [i] + |
and k 1 or vice versa. Without loss of generality, let li = = and [k] = 1. T + and k = I or
2. Then, either []
j= 1) and =11. In both cases, jk = (n 1) = which is odd by the induction hypothesis.
By mathematical induction, the statement holds for all integers j and if and are odd, and jk is odd. Thus, the statement correct.
SSAPO response:
The given statement "For all integers j and k, jf and k are odd, then jk is odd" is correct.
To prove this statement, we need show that for all integers] and both and k odd, then their product jk is also odd.
Let and k be two odd integers, i.e., == 2n and k 2m 1, where and any integers.
Then, their product jk can be written as:
jk= (2n + 1) (2m + 1) 4nm + 2n + 2m + 2(2nm + m))
Since m are integers, the product 2nm + + m is also an integer. Therefore, the above expression jk is equal to the sum of an even number (2(2nm + n + m)) and an odd number (I).
As the sum of two odd numbers is always even, the sum 2(2nm + m) + is an odd number.
Hence, the product jk= (2n 1)* (2m + 1) 2(2m22222mmmmm m) is odd integer, which proves the given statement.
Therefore, the given statement "For all integers j and k, if and k are odd, then jk is odd" is correct.
Prompt:
Find the answer that best describes the underlined SAT word. Select the correct option and explain the meaning of the underlined word.
Despite the _cccpphhoy,,.... the student tried study.
A. Loud sounds
B. Difficult subject
C. Late hour
D. Low lighting
DPO response:
The correct option is A. Loud sounds. The word cacophony means loud and confusing mixture of sounds." In this sentence, it describes the noise level in the environment where the student is trying to study. Therefore, the underlined word cacophony refers to the loud sounds that are making it difficult for the student to concentrate.
SPA response:
A) Loud sounds (cacophony)
Explanation:
The word "cacophony" refers loud and unpleasant combination of sounds, typically characterized a discordant or confusing mixture of noise. In the context of the sentence, "despite the _cacophony" sug-
30

---

gests that the student is trying to study despite the overwhelming and distracting noise around them. The underlined word "cacophony" modifies the phrase "loud sounds" and provides descriptive adjective.
Option B) Difficult subject, C) Late hour, and D) Low lighting are not relevant to the given context as they do not refer to loud unpleasant sounds. Option A) "loud sounds" is synonym for "cacophony" and accurately describes the situation mentioned in the sentence.
Therefore, the correct option for the underlined word is A) Loud sounds (cacophony).
SSAPO response:
Option A: "Despite the cacophony, the student tried to study"
The underlined word "cacophony" means loud chaotic combination of sounds. It describes situation where there is an overwhelming or unpleasant mixture of noise or sounds.
Cacophony is derived from the Greek words "kakos" meaning "bad" or "ugly" and "phone" meaning "sound". is commonly used to describe a variety loud or unpleasant sounds, such as traffic noise, barking dogs, screaming children, discordant music. In the given sentence, the student is trying to study despite the distracting and disruptive noise or sounds, which creates a cacophony. The word "despite" implies that the student is facing obstacle hindrance, which is the excessive noise or chaos, but they still persist in their efforts to concentrate and study.
Therefore, the meaning of the underlined word "cacophony" is loud or chaotic combination of sounds, which creates an unpleasant or distracting environment for the student's studying. Option the correct answer. Option B refers to a difficult subject, Option C refers a late hour, and Option D refers to low lighting, none of which are relevant to the given context.
I
Broader Impact
Our work aims to improve the data efficiency and robustness of language model alignment by for- mulating preference optimization Stackelberg game and introducing self-annotation mechanism. By reducing reliance on large-scale human-labeled data, our framework could democratize alignment research and make it more accessible smaller organizations, labs, and communities (those lack substantial annotation budgets). Morcover, robust optimization against noisy or adversarial preference distributions may help mitigate unintentional bias if the seed data deviate from the true user preference distribution.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the introduction section, we delineate the problems addressed by this work and outline our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the conclusion section (Section S Conclusion, Limitation and Future work), we highlight the limitations of the current work and suggest directions for future research.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech -totteexttttttttt system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimeess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer: [Yes]
Justification:
We provide the complete theoretical proof' in Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Answer: [Yss]
Justification: We provide detailed experiment results in Appendix G Besides, code is anonymously available //tp:::////nmmmuu 4open wwp.....piiene.................. Cccience/x/SSAPPP 6888
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. case of closed-source models, it may be that access to the model is limited in some way (e-g., to registered users), but it should possible for other researchers have some path to reproducing verifying the results.
the
5. Open access to data and code
33

---

Question:
Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is anonymously available at r/ #/SSAPO-660
4open. science/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https
for more details.
While we encourage the release of code and data, we understand that this might not be pos- sible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines antops://nips.cc/ppute
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity. the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide experiment settings in Section fadd Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Due to the computational constraints associated with training and evaluating large-scale models, we could not perform multiple repetitions of each experiment. Thus, traditional error bars based on multiple runs are not included. However, we have taken other appropriate measures to demonstrate robustness and significance, including clearly reporting the experimental settings, model hyperparameters, and performing ablation studies or comparisons to baselines wherever possible. Additionally, we've ve detailed any relevant sources of variability or uncertainty in the experimental setup and results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma enror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: I have read the NeurIPS Code of Ethics and confirm our research in the paper conforms with Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the potential broader impacts in Appendix ||
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
35

---

If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deptoyment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The framework proposed in our paper does not extend to application domains requiring safeguards. Additionally, the datasets used are widely-used node classification datasets, thus climinating the need for specific safeguards.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all referenced works.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode, com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: https://anonymont 4open. wtt..aiceence////P--BBeB-.-. phase, release our httptp/ss//gtthbb..mmmmmm/mll
We release
code anonymously during reviewing phase at During the camera ready
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used an important, original, non-standard component of the core methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy hutt:://heeuusss.cco///oorrrrr/p for what should or should not be described.
38


Reviews:
[{"review_id": "7oJciNYls4", "reviewer": "Reviewer_pf3d", "summary": "This paper presents Stackelberg Game Preference Optimization (SGPO), a game-theoretic framework for aligning LLMs with minimal human supervision. It formulates alignment as a two-player Stackelberg game between a policy and an adversarial preference distribution constrained by a Wasserstein distance. The proposed SSAPO algorithm starts with a small set of human-labeled preferences and expands them through iterative self-annotation, while using robust reweighting to counteract noisy labels. Theoretical guarantees are provided, and empirical results show that SSAPO achieves strong performance using 2K human-labeled preferences for initial training, outperforming several existing alignment methods.", "questions": "1. In Table 1, why Mistral-7B-DPO, Mistral-7B-Iter DPO (PairRM), Mistral-7B-SPA have higher average score on MT-Benchmark compared to Mistral-7B-SSAPO? Also, in Table 2, why the Mistral-7B-SSAPO has lower average score on MT-Benchmark compared to Zephyr-7B- β and Mistral-7B-DPO? Did the authors verify the baseline results presented in the tables?\n\t2. Can the authors comment on how does SSAPO compare to other alignment frameworks (such as minimax-based) in both theory and practice?\n\t3. Can the authors comment on the robustness of the final policies, i.e., are the final policies robust to adversarial or misaligned prompts (e.g., jailbreaking, harmful completions)?\n\t4. Can the authors comment on how stable SSAPO is across multiple runs and how scalable is the DRO stage if group sizes increase beyond 1 million samples?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Game-theoretic formulation with empirical validation: SGPO ensures convergence and regret, adding rigor to alignment under preference uncertainty. SSAPO performs better than more data-intensive techniques such as SPA, DPO, and some models that have been fully annotated by humans (Zephyr-7B).\n\n2. Data-efficient alignment: Uses only 2K human preference pairings to achieve high victory rates against GPT-4. Also, the framework supports scalability through uniform group clipping.\n\n3. Theoretical insights and empirical validation: The paper presents theorems on worst-case external distributions, along with Stackelberg Equilibrium and its convergence guarantees while showcasing empirical validation of their proposed SGPO based game theoretic approach.\n\nWeaknesses:\n1.\tAbstraction of Stackelberg formulation: Although formal, the Wasserstein DRO formulation is overly detached from the usual training process for LLMs. Only a weak connection exists between the reward/logit terms and the real model behavior.\n2.\tLimited theoretical comparison: Although similar game-theoretic work (such as SPIN and Nash-RLHF) is mentioned, the study doesn't go into detail on why Stackelberg is better in alignment circumstances.\n3.\tEmpirical Diversity: The evaluation is limited to the UltraFeedback dataset along with AlpacaEval 2.0 and MT-Bench. While these benchmarks are well-established, they lack diversity in linguistic styles, user intents, and task modalities. Incorporating a broader range of evaluation settings would strengthen the paper’s claims of generalizability.", "was_revised": true, "final_justification": "Thanks for the responses. I believe this is a good contribution (and worthy of being accepted) and I will maintain my positive rating for this paper.", "rebuttal": "We sincerely thank the reviewer for recognizing the formal rigor of our Stackelberg formulation. The clarifications below demonstrate how our theoretical framework directly addresses practical challenges in LLM alignment, achieving state-of-the-art results with 30x fewer labels while providing formal robustness guarantees that existing methods lack.\n\n---\n\n## W1:  Abstraction of Stackelberg formulation.\n\nWe thank the reviewer for highlighting this important connection. Our Stackelberg formulation directly extends the standard DPO framework by introducing distributional robustness.\nStandard DPO optimizes: $$\\max_{\\pi}E_{(y_w,y_l) \\sim \\hat{P}}[\\log\\sigma(R_\\pi(y_w) - R_\\pi(y_l))] $$\nOur approach optimizes: $$\\max_{\\pi}\\min_{P \\in \\mathcal{U_\\epsilon}(\\hat{P})}E_{(y_w,y_l) \\sim P}[\\log\\sigma(R_\\pi(y_w) - R_\\pi(y_l))] $$\nThis modification addresses a fundamental weakness in existing methods: vulnerability to noisy preference annotations. Our practical algorithm **SSAPO** implements this through three concrete steps that directly map to standard LLM training:\n\n1. Collect preference data (standard practice)\n2. Compute worst-case distribution within $\\epsilon$-ball (our novel contribution, Section 3.1)\n3. Update policy using standard DPO loss on reweighted data\n\nThis is a natural extension of existing iterative methods (e.g., SPA) that adds robustness without fundamentally changing the training paradigm. We will clarify this connection more explicitly in Section 3.2.\n\n---\n\n## W2:  Comparison against game-theoretic approaches.\n\nWe appreciate this suggestion. Our Stackelberg formulation offers three key theoretical advantages over existing game-theoretic approaches:\n1. Robust: Unlike policy-vs-policy games (SPIN, Nash-RLHF), our policy-vs-distribution formulation directly addresses preference noise—the primary challenge in LLM alignment. This leads to our O(ε)-bounded regret guarantee (Theorem 2.5), which neither SPIN nor Nash-RLHF provides.\n2. Tractable: Our alternating optimization converges linearly to equilibrium (Theorem 2.3) with well-understood DRO subproblems. In contrast, Nash equilibria require solving coupled fixed-point problems that are computationally intractable at LLM scale.\n3. Practicical: Our framework seamlessly integrates with self-annotation, requiring only 3.3% labeled data while achieving superior performance. This addresses real-world constraints that pure game-theoretic methods ignore.\n\n---\n\n## W3:  Evaluation protocol.\n\nWe acknowledge that broader evaluation would strengthen our empirical claims. Our experimental design follows the established **SPA[1] protocol (ICLR'25 Oral, 10/8/8 ratings)** to ensure fair comparison with state-of-the-art baselines.\nAlpacaEval 2.0 and MT-Bench represent the field's consensus benchmarks, evaluating diverse capabilities including instruction following, multi-turn reasoning, coding, and creative writing across 800+ prompts. Our significant improvements over SPA (24.44% vs 15.39% on AlpacaEval) demonstrate the method's effectiveness within this standard evaluation framework.\nWe appreciate and agree with your valuable suggestion that testing on additional domains would be valuable future work.\n\n---\n\n## Q1:  MT-Bench results.\n\nThe MT-Bench results follow expected patterns when examined carefully. Across Tables 1-2, Mistral models show remarkably low variance on MT-Bench (6.34-6.98 range) regardless of method, suggesting the Mistral model architecture may have limited sensitivity to this benchmark. This contrasts sharply with the high variance on AlpacaEval (9.03-35.82%), where different methods clearly separate.\nThis pattern appears in the original SPA paper as well: SPA underperforms Iterative DPO on MT-Bench despite dominating on AlpacaEval. The consistency of this pattern across multiple studies suggests AlpacaEval provides better discrimination for Mistral-based models. Notably, our method achieves the highest performance on the more discriminative benchmark (35.82% vs 21.13% for SPA).\n\n---\n\n## Q2:  Comparison to other frameworks.\n\nAs described in Appendix A, some concurrent works adopt Nash or minimax formulations for LLM alignment. While these words typically set both players as policy vs. policy, our SGPO framework focuses on policy vs. Distribution, leading to tight theoretical guarantees and a practical algorithm **SSAPO** that is readily integrated with self-annotation. Moreover, the target problems these works aim to address are fundamentally different from ours. Please refer to our response to **W2** for more details.\n\nBeyond the theoretical advantages, our approach offers practical benefits: 30x reduction in human labels versus standard methods. Requires only adding DRO reweighting to standard DPO. Minimax formulations typically require adversarial training that is unstable at LLM scale. Our Stackelberg approach decomposes into stable, well-understood subproblems.\n\n---\n\n## Q3:  Robustness of final policies.\n\nOur method provides training-time robustness against noisy preference data, which is fundamentally different from inference-time robustness against adversarial prompts.\nSpecifically, our theoretical guarantee (Theorem 2.5) ensures that if test preferences fall within the $\\epsilon$-Wasserstein ball of training preferences, performance degradation is bounded by $O(\\epsilon)$. This protects against annotation noise and minor distribution shifts.\nHowever, the worst-case distribution P* optimizes for mathematical objectives, not semantic categories like jailbreaking. Therefore, while our method improves alignment robustness, it does not replace dedicated safety measures for adversarial prompts. We will clarify this distinction in Section 2.4.\n\n---\n\n## Q4:  Stability and scalability.\n\nRegarding stability: Our farthest-point sampling (FPS) strategy for seed selection significantly improves consistency across runs by ensuring better coverage of the preference space compared to random sampling. In our experiments, FPS reduced performance variance by approximately 40%.\n\nRegarding scalability: Our grouping strategy (Section 3.2.1) enables practical deployment. With group size $G=100\\sim 1000$, we achieve near-optimal robustness while maintaining $O(G·K)$ complexity per group with parallel processing. For $N=1M$ preferences, using $G=1000$ yields $1000$ parallel jobs of manageable size.\n\nPractical recommendation: We suggest $G\\in [100, 1000]$ as the sweet spot balancing computational efficiency and solution quality. Larger $G$ provides diminishing returns while significantly increasing computational cost. We will add these concrete guidelines to Section 3.2.1.\n\n*[1]* SPA: Direct Preference Judgment for Efficient LLM Alignment, ICLR'25"}, {"review_id": "htGJKw2PAr", "reviewer": "Reviewer_a9ir", "summary": "The paper addresses the challenge of aligning large language models (LLMs) with human preferences, which traditionally demands vast amounts of meticulously curated data that is both expensive and prone to labeling noise. The paper proposes Stackelberg Game Preference Optimization (SGPO), a robust alignment framework that models alignment as a two-player Stackelberg game. This formulation provides formal robustness to (self-)annotation noise and guarantees $O(\\epsilon)$-bounded regret, meaning the performance drop under distributional shifts is at most proportional to $\\epsilon$. As a practical instantiation of SGPO, the paper introduces SSAPO which begins with a minimal set of human-labeled “seed” preferences and iteratively self-annotates new prompts. Experiments demonstrate that SSAPO maintains high-level performance despite using only a fraction of typical human annotations, achieving strong win rates against GPT-4 across multiple benchmarks within three iterations.", "questions": "1. Can the authors elaborate on the types of \"labeling noise\" or \"bias\" that traditional alignment methods are prone to, and how SSAPO's approach specifically addresses these issues beyond just reducing data volume?\n2. Can the authors provide more discussions on the Assumption C.1, especially the compactness assumption. How to impose compactness to guarantee an acceptable Stackelberg solution in practice?\n3. Can the authors elaborate on the arguments used to establish the convergence of policy updates in the proof of Theorem 2.3? It is unclear how to do this with a argmax-min.\n4. How is the quality of seed dataset inferred in a real world application? How to know if the self-training won't suffer from distributional shift?\n\nMinor:\nPlease use consistent numbering of theoretical results in the main paper and the appendix, and include a reference to the proof in the main paper.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper recasts alignment as a two-player game, proving the existence of a robust equilibrium with $O(\\epsilon)$-bounded regret under $\\epsilon$-Wasserstein preference shifts.\n2. The distributionally robust re-weighting attenuates the impact of potential labeling noise.\n3. SSAPO achieves strong performance using only a small number of human-labeled seed preferences, with the empirical results showing SSAPO outperforming or matching methods that use significantly more human labels.\n\nWeaknesses:\n1. SSAPO relies on two key approximations that deviate from the ideal SGPO solution, potentially weakening the theoretical guarantees in practice.\n2. Worst-case distribution computation can be computationally expensive, raising scalability concerns.", "was_revised": true, "final_justification": "The authors have clarified all my concerns which has prompted me to keep my score.", "rebuttal": "We sincerely appreciate your valuable feedback—we hope our comprehensive responses below fully address your concerns and demonstrate the contribution of **SSAPO**.\n\n---\n\n## W1: Approximations and Theoretical Guarantees.\n\nWe appreciate the reviewer's thoughtful consideration of our approximations. We emphasize that these approximations are carefully designed to preserve the core theoretical guarantees while ensuring computational tractability.  As described in Appendix F,  **these two key approximations do NOT change the Stackelberg essence of SSAPO**: the policy is trained against worst-case reweightings within an ϵ-Wasserstein distance. As a result, it retains the key benefit of  bounded regret for moderate shifts (Theorem 2.5), while remaining computationally tractable.  Empirically (Section 4), these approximations still yield substantial label-efficiency and robustness gains over standard DPO. \n\n---\n\n## W2: Computational Scalability.\n\nWe acknowledge the computational considerations and have conducted extensive analysis to ensure practical scalability:\nWhile the theoretical worst-case complexity is $O((NK)^\\gamma)$, our implementation achieves near-linear scaling through:\n1. **Efficient DRO solvers**: We employ specialized cutting-plane methods that typically converge in 10-20 iterations, far below theoretical bounds\n2. **Intelligent grouping**: Our uniform grouping strategy (G=100-200 samples) reduces complexity to $O(M·K)$ where M=N/G, achieving 50-100x speedup with minimal performance loss\n3. **Parallelization**: Group-level computations are embarrassingly parallel\n\nIn practice, **SSAPO** adds only 20-30% computational overhead compared to iterative DPO while delivering substantial robustness gains. This modest increase is justified by the significant improvements in label efficiency and performance stability.\n\n---\n\n## Q1: Addressing Labeling Noise and Bias.\n\nThank you for this important question. Our work specifically targets the critical challenge of preference annotation noise in self-training scenarios:\n\nTraditional methods like DPO assume clean preference labels, but real-world data contains systematic biases: preference flips, annotation inconsistencies, and model-induced biases during self-labeling. Recent work [1] shows even human-labeled datasets contain 15-20% noise rates.\n\n**SSAPO** addresses these challenges through principled robustness: **Worst-case optimization**（optimize against adversarial reweightings within an ε-Wasserstein ball), **formal guarantees** ( $O(\\epsilon)$-regret bound ensures bounded performance degradation even under adversarial noise), and **Empirical validation**: We observe that after the initial DPO training using seed data, pseudo-labeling introduces >20% noise. As self-training progresses, the noise rate in pseudo-labels continues to increase. After the first round of self-training, SSAPO achieves approximately **4% higher self-annotation accuracy compared to SPA[2]**, and consistently maintains superior self-annotation accuracy relative to SPA throughout all training rounds. \n\nThe robustness of **SSAPO** is particularly crucial for self-training paradigms where the model generates its own training data, creating a feedback loop that can amplify initial biases. To further validate this robustness, we conduct an additional experiment by flipping the preference labels of 25% instances of the 2K seed data.  **Our additional experiment with 25% label corruption provides strong empirical validation of SSAPO's theoretical robustness guarantees**.\n\n#Additional experiment for SSAPO robustness proof\n|  Base Model       | no noise |25% noise | \n|-----------|-------|--------  |\n| Mistral-SSAPO | 26.90%/31.93% | 19.70%/18.51%     | \n|  Llama-SSAPO | 33.33%/40.12% | 43.74%/46.7%     | \n\nFor Mistral model, performance represents only a ~7-13% degradation despite 25% label corruption. This bounded degradation empirically validates our O(ε)-regret guarantee from Theorem 2.5. The degradation likely stems from suboptimal ε selection for Mistral's specific characteristics—our ablation study (Table 3) shows that ε=0.01 was optimal for Mistral-family models, but noisy settings may require different ε values to balance robustness and data fidelity.\n\nFor Llama model, we observe **counterintuitive improvement with 25% label corruption, demonstrating that **SSAPO**'s distributionally robust optimization can leverage noise as implicit regularization**. The 25% corrupted labels force the worst-case distribution P* to explore a wider region of the ε-Wasserstein ball, preventing overfitting to the small 2K seed set. Additionally, Llama's superior self-annotation capabilities enable it to implicitly correct noisy preferences during iterative updates.\n\nThese contrasting behaviors show that Mistral is less sensitive to preference optimization, making it a conservative test case. The Llama results showcase SSAPO's full potential when paired with models having strong self-annotation capabilities, where the Stackelberg game dynamics can actively exploit the adversarial reweighting to improve alignment even under noisy supervision.\n\nThis experiment strengthens **SSAPO**, not only defending against preference noise (as shown by Mistral's bounded degradation) but can potentially benefit from it through principled distributionally robust optimization (as shown by Llama's improvement).\n\n---\n\n## Q2: Compactness Assumption in Practice.\n\nThank you for highlighting this practical consideration. The compactness assumption is naturally satisfied through the problem structure without requiring explicit constraints.\n\n**Automatic satisfaction through DPO structure:**\n- The log-sigmoid objective inherently bounds outputs: $\\log \\sigma(\\cdot) \\in [0, \\log 2]$\n- KL regularization $D_{KL}(π_\\theta || π_{ref})$ creates an implicit trust region around $\\theta_{ref}$\n- Gradient norms remain bounded: $||\\nabla_\\theta J|| \\le \\beta\\cdot\\log 2$\n\n**Practical implementation:**\n- Standard optimization with learning rate η and T steps naturally constrains parameters: $||\\theta_T - \\theta_0|| \\le \\eta \\cdot T·\\max_t ||\\nabla_\\theta J|| = O(\\eta·T)$\n\nThis explains why DPO/RLHF methods succeed without explicit parameter bounds. The combination of bounded objectives and KL regularization creates a self-contained optimization landscape where compactness emerges naturally from the problem structure.\n\n---\n\n## Q3: Convergence Analysis for Policy Updates.\n\nWe appreciate the opportunity to clarify our convergence analysis. The argmax-min structure requires careful treatment through variational inequality theory:\n\n**Key insight**: The policy update can be characterized as solving a variational inequality (VI) problem. Define\n $$F_t(\\pi) := -\\nabla_\\pi \\min_{P \\in \\mathcal{U_\\epsilon}} J(\\pi,P).$$ \nThen  $\\pi_{t+1}$ satisfies: \n$$\\langle F_t(\\pi_{t+1}), \\pi - \\pi_{t+1} \\rangle \\geq 0, \\quad \\forall \\pi \\in \\Pi.$$ \n( By Danskin's theorem and first-order optimality conditions for the constrained maximization problem)\n\n**Convergence guarantee**: Under Assumption 3.1, the mapping $\\mathcal{T}: \\pi_t \\mapsto \\pi_{t+1}$ is a contraction with rate $\\gamma_\\pi < 1$. \n\nStep 1: Show $F_t(\\pi)$ is strongly monotone with parameter $\\mu > 0$: $\\langle F_t(\\pi_1) - F_t(\\pi_2), \\pi_1 - \\pi_2 \\rangle \\geq \\mu \\|\\pi_1 - \\pi_2\\|^2$. This follows from the concavity of $J(\\pi,P)$ in $\\pi$ and the log-sigmoid structure, which induces strong concavity when restricted to the compact set $\\Pi$. \n\nStep 2: Show $F_t(\\pi)$ is Lipschitz continuous with constant $L_F$: $\\|F_t(\\pi_1) - F_t(\\pi_2)\\| \\leq L_F \\|\\pi_1 - \\pi_2\\|$. \nBy Danskin's theorem, $F_t(\\pi) = -\\nabla_{\\pi} J(\\pi, P'(\\pi))$, where $P'(\\pi) = arg \\min_{P \\in \\mathcal{U^t_\\epsilon}} J(\\pi,P)$. \n(The Lipschitz property follows from $R_{\\pi}$ in $\\pi$ with constant $L_R$, yielding $L_F = L_R(1 + \\epsilon L_R)$. )\n\nStep 3: Apply standard variational inequality theory. For strongly monotone, Lipschitz operators, the solution mapping is a contraction with rate: $\\gamma_\\pi = \\sqrt{1 - \\left(\\frac{2\\mu}{L_F}\\right)^2} < 1$ Therefore: $\\|\\pi_{t+1} - \\pi_{t+1}'\\| \\leq \\gamma_\\pi \\|\\pi_t - \\pi_t'\\|$ for any two sequences starting from different initial points, implying convergence to a unique fixed point. \n\nThe monotonicity parameter $\\mu$ can be explicitly computed from the problem data: for the log-sigmoid objective with bounded rewards $|R_\\pi(y)| \\leq R_{\\max}$, we have $\\mu \\geq \\beta e^{-2\\beta R_{\\max}}/(4(1+e^{-2\\beta R_{\\max}})^2)$. \n\n We will add a more detailed proof in the final version.This analysis extends beyond standard gradient ascent convergence by accounting for the coupled min-max structure, providing stronger guarantees for our Stackelberg formulation.\n\n---\n\n## Q4: Seed Data Quality and Distributional Shift.\n\nOur framework is designed with practical deployment considerations.  While we assume clean seed data in experiments, this is practically achievable—obtaining a small set of high-quality human annotations is feasible and cost-effective. Moreover, **SSAPO** demonstrates resilience even with noisy seed data, as addressed in **Q1**.\n\nWe appreciate your concern about potential distributional shift. However, we argue that self-training will not suffer from such shift. As detailed in Section 3.2, the prompts in our experiments are sampled from real-world data. Using real-world prompts ensures $p(x) $remains consistent.  Responses sampled from $\\pi_{\\theta_t}$ evolve smoothly with the policy. Our worst-case optimization specifically addresses the primary source of shift—labeling noise in self-annotation. The key insight is that **SSAPO** transforms the distributional shift problem into a robustness problem, which our framework is explicitly designed to handle through the Wasserstein uncertainty set formulation.\n\n*[1]* Impact of Preference Noise on the Alignment Performance of Generative Language Models, COLM'24\n\n*[2]* SPA: Direct Preference Judgment for Efficient LLM Alignment, ICLR'25"}, {"review_id": "ppdGeC4AOp", "reviewer": "Reviewer_eQUq", "summary": "This paper addresses the challenge of aligning large language models (LLMs) with human preferences without relying on massive amounts of expensive, manually labeled data. The authors propose a novel game-theoretic framework called Stackelberg Game Preference Optimization (SGPO). In this setup, the LLM plays the role of a leader in a two-player game, where it tries to optimize its behavior while being robust to the worst-case preference distribution caused by annotation noise or distribution shifts. They then introduce a practical algorithm called SSAPO, which begins with a small set of human-labeled data. From there, it gradually expands the dataset through self-annotation, while using a robust reweighting strategy to reduce the impact of noisy or unreliable synthetic labels. Theoretical analysis shows that SGPO offers bounded regret and robust convergence, and the empirical results are strong. Using only 2K labeled pairs (3.3% of the dataset), SSAPO significantly outperforms baselines like DPO and Iter-DPO across benchmarks such as AlpacaEval 2.0 and MT-Bench. The paper also includes thoughtful ablations, validating design choices like the Wasserstein radius and grouping strategy for efficient optimization. Overall, it presents a data-efficient and theoretically grounded method for robust LLM alignment.", "questions": "1. Since the SSAPO pipeline depends on a DPO-trained model initialized with 2K seed labels, how would performance be affected if the seed set contained annotation noise or was reduced further, say to 1K or 500 samples? Have you conducted any sensitivity analysis in that direction?\n2. The paper shows that the best win rate occurs when the Wasserstein radius ϵ is set to 0.01. Do the authors have any guidance or heuristics for selecting this value in a new domain where validation labels may not be available?\n3. In the construction of the piecewise-linear approximation for the −log σ loss, performance drops when K is increased to 7. Could the authors provide more intuition or empirical observations behind this degradation? Is it due to overfitting, instability, or numerical issues?\n4. SSAPO achieves strong win rates against GPT-4 in both AlpacaEval 2.0 and MT-Bench. Could the authors provide example failure cases or qualitative comparisons where SSAPO still lags behind GPT-4, to better understand current limitations?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The authors tackle a highly relevant issue in LLM alignment on how to reduce dependence on large-scale human-labeled preference data, which is both expensive and increasingly limited. What makes their approach interesting is that, instead of just building on existing methods, they introduce a new perspective by framing the problem as a Stackelberg game. In this setup, the model learns while being challenged by a worst-case preference distribution, which helps it stay robust even when the data is noisy or shifted.\n2. The paper presents a practical and scalable implementation of the Stackelberg Game Preference Optimization framework through the SSAPO algorithm. They address the intractability of optimizing the follower’s DRO problem by introducing a piecewise-concave under-approximation of the non-convex objective (−log σ loss), and reformulating it as a tractable convex program. The grouping strategy for large-scale DRO optimization further improves computational efficiency without compromising the theoretical guarantees, making the method applicable to real-world LLM alignment tasks.\n3. The paper covers two competitive LLM backbones (Mistral-7B and LLaMA3-8B) and standard alignment benchmarks (AlpacaEval 2.0 and MT-Bench). SSAPO achieves consistent improvements over strong baselines like DPO, Iter-DPO, and SPA using only 2K human-labeled examples. Ablation studies support the design choices, such as the selection of the Wasserstein radius (ϵ), the number of tangents (K) in the piecewise loss approximation, and the group size (G) for parallelized optimization.\n\nWeaknesses:\n1. The empirical performance of SSAPO appears sensitive to hyperparameter choices. For instance, Table 3 shows that the optimal performance on Mistral-7B occurs at ϵ = 0.01, while both smaller (ϵ = 0) and larger values (ϵ = 0.1) significantly degrade the win rate. Similarly, Table 4 shows that changing the number of tangents from K = 6 to K = 7 reduces win rate from 23.20 to 19.05. We would like to see whether the authors have any heuristics or adaptive strategies for setting these parameters, especially for new model or dataset configurations.\n2. The entire training loop is initialized from a DPO-trained model using 2K seed preference pairs. However, we wonder how robust SSAPO is when the initial seed is either smaller in size or of lower quality. Since self-annotation propagates from this starting point, any noise or bias in the seed may get amplified over iterations. We would like to see an analysis where the seed set is perturbed or reduced to test SSAPO’s stability under less favorable initialization.\n3. Although the paper includes strong comparisons against DPO, Iter-DPO with LLM-as-judge, and SPA baselines, we would like to see more insight into the qualitative behavior of SSAPO against GPT-4, especially since it achieves high win rates in AlpacaEval and MT-Bench. For instance, what kinds of prompts or failure cases does SSAPO consistently outperform or underperform on compared to GPT-4? We wonder if deeper behavioral or error analysis could provide a clearer picture of how SSAPO aligns with human preference in nuanced tasks.", "was_revised": true, "final_justification": "After carefully reviewing the rebuttal, I have updated my score from 4 to 5. The authors provided clear and well-supported responses to all major concerns. For hyperparameter sensitivity, they offered useful practical guidelines grounded in their theoretical framework, which makes the method easier to apply in real-world settings. Their explanation of the performance drop with higher K values helped clarify that the issue stems from optimization instability rather than poor approximation. To address seed data robustness, they conducted an additional experiment with 25% label corruption, showing minimal degradation in performance and confirming the model's theoretical guarantees. This significantly improved my confidence in the method's stability when dealing with noisy annotations. Finally, the behavioral comparison with GPT-4 was appreciated. The analysis showed that SSAPO performs strongly on structured reasoning tasks but has room for improvement in more creative or stylistic generations. This balance between strength and limitation was communicated transparently and with thoughtful insight.\nOverall, the rebuttal strengthened the contribution and demonstrated both robustness and practical value.", "rebuttal": "We sincerely thank the reviewer for the thorough evaluation and constructive feedback. Our responses below address the identified concerns while providing practical deployment guidelines and additional theoretical insights that strengthen the overall contribution. We believe this analysis demonstrates that **SSAPO** offers a principled solution to data-efficient LLM alignment with significant value to the alignment community.\n\n---\n\n## W1: Hyperparameter Sensitivity and Practical Guidelines.\n\nWe appreciate this important concern about hyperparameter selection. Building on our theoretical analysis in Section 2, we provide concrete practical guidelines:\n\nFor Wasserstein radius $\\epsilon$ selection: Our theoretical framework suggests that ε should scale with the expected noise level in self-annotations. We propose the following adaptive strategy: For larger, more capable models: Use $\\epsilon \\in [0.005, 0.02]$ as self-annotation quality is typically higher. For smaller models: Use $\\epsilon \\in [0.01, 0.05]$  to account for higher self-annotation noise.\n\nFor number of tangents $K$: Our analysis reveals that $K=6$ consistently provides the optimal balance across different model architectures and datasets. The performance degradation at $K=7$ occurs **due to optimization instability rather than approximation quality** - additional tangent points introduce unnecessary complexity that hampers convergence of the distributionally robust optimization solver. We recommend $K=6$ as a robust default that requires minimal tuning.\n\nThese recommendations align with our regret bounds in Theorem 2.5, where moderate $\\epsilon$ values provide $O(\\epsilon)$-bounded regret while avoiding overly conservative solutions that degrade performance.\n\n---\n\n## W2: Quantity and Quality of Seed Data.\n\nThank you for raising an excellent point about seed data robustness. While our current experiments follow the established **SPA [1] protocol (ICLR’25 Oral, rated 10/8/8)** with clean seed data, we acknowledge that real-world scenarios may involve smaller or noisier seed sets. \n\n**Necessity of smaller seed sets**: Our experiments follow the SPA protocol, which is designed for data-scarce settings. The small number of seed data（2K of 60K in total, only 3.3%）reflects this context, and our results are directly comparable to the SPA baselines, where we demonstrate significant improvements in performance. Though ablation study on even less seed data may further demonstrate the generalizability **SSAPO**, the current setup is already sufficient to validate the efficacy and data efficiency of our method.\n\n**Theoretical robustness**: **SSAPO**'s distributionally robust formulation inherently provides some protection against seed data noise. The Stackelberg game framework optimizes against worst-case preference distributions within a Wasserstein ball, which naturally accounts for perturbations in the initial distribution.\n \nTo further validate **SSAPO**'s robustness against label noise in seed data, we conduct an additional experiment by flipping the preference labels of 25% instances of the 2K seed data.  **Our additional experiment with 25% label corruption provides strong empirical validation of **SSAPO**'s theoretical robustness guarantees**.\n\n#Additional experiment for SSAPO robustness proof\n|  Base Model       | no noise |25% noise | \n|-----------|-------|--------  |\n| Mistral-SSAPO | 26.90%/31.93% | 19.70%/18.51%     | \n|  Llama-SSAPO | 33.33%/40.12% | 43.74%/46.7%     | \n\nFor Mistral model, performance represents only a ~7-13% degradation despite 25% label corruption. This bounded degradation empirically validates our $O(\\epsilon)$-regret guarantee from Theorem 2.5. The degradation likely stems from suboptimal ε selection for Mistral's specific characteristics—our ablation study (Table 3) shows that $\\epsilon=0.01$ was optimal for Mistral-family models, but noisy settings may require different $\\epsilon$ values to balance robustness and data fidelity.\n\nFor Llama model, we observe **counterintuitive improvement with 25% label corruption, demonstrating that **SSAPO**'s distributionally robust optimization can leverage noise as implicit regularization**. The 25% corrupted labels force the worst-case distribution P* to explore a wider region of the $\\epsilon$-Wasserstein ball, preventing overfitting to the small 2K seed set. Additionally, Llama's superior self-annotation capabilities enable it to implicitly correct noisy preferences during iterative updates.\n\nThis experiment strengthens **SSAPO**, not only defending against preference noise (as shown by Mistral's bounded degradation) but can potentially benefit from it through principled distributionally robust optimization (as shown by Llama's improvement).\n\n---\n\n## W3: Failure Cases and Behavioral Analysis.\n\nWe recognize the value of deeper qualitative analysis and have conducted additional error analysis comparing SSAPO to GPT-4.\n\nBy analyzing cases from the Alpaca Eval Benchmark, we observe that **SSAPO demonstrates advantages over GPT-4  on structured reasoning tasks** (e.g. mathematical proofs or logical deduction). For example:\n\ninstruction: \n```\nHow many words are spoken daily on Earth? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step.\n```\nGPT4 response: \n```\n... Let's break it down step-by-step:\n\n1. World Population: ... the world population was approaching 8 billion people ...\n2. Speaking Population: ... let's assume that 75% of the population speaks daily ...\n3. Words per Day: ... an individual might speak around 7,000 words per day ...\n4. Calculation: ... Total words per day = Number of speaking people * Average words spoken per person per day ... = 42 trillion words/day\n ...\n```\nSSAPO response:\n```\n... Step-by-Step Explanation: \n\n1. Estimation of Global Population: ... According to the United Nations, the world's population as of 2021 is approximately 7.9 billion people......\n2. Average Daily Communication: ... Assuming that each individual spends an average of 4 hours daily engaging in verbal and written communication, this translates to roughly 31.6 billion hours of communication per day worldwide ...\n3. Word Count per Hour of Communication: ...  the average spoken word rate for English speakers ranges from 120 to 180 words per minute, with an average of 150 words per minute ...\n4. Calculation of Total Word Count: Multiplying the estimated global population by the average daily communication hours and the average word count per hour of communication yields a total estimated word count of 500 billion words spoken daily on Earth.\n ...\n\n```\n\nOn the other hand, we  find that **SSAPO falls short of GPT-4 in creative writing tasks requiring high linguistic sophistication**. For Example:\n\ninstruction: \n```\nIf you were a Shakespearean character, how would you declare your love for someone in a soliloquy?\n```\nGPT4 response: \n```\n... Here's an example:\n\n(Imagine a dimly lit stage, a single spotlight shining down upon the forlorn lover, who steps forward to share the depths of their heart.)\n\nO, speak, thou lustrous beacon of the night,\nWith stars enkindled in thine eyes so bright.\nThou art the daybreak to my endless gloom,\nThe rose that in my barren heart doth bloom.\nForsooth, this love, a tempest full of grace,\nHath swept upon my soul without a trace.\n...\n```\nSSAPO response:\n```\nOh, my dearest, my heart doth overflow\nWith a love so pure, so deep, and true,\nMy soul doth long to declare my plight,\nTo thee, my fair one, my eternal muse.\n ...\n```\n\nWe hypothesize that the aforementioned strengths and weaknesses of SSAPO may stem from specific preference biases in the training data. We use **UltraFeedback [2]** dataset for training, which primarily annotates preferences based on four criteria: instruction-following, truthfulness, honesty, and helpfulness. As a result, alignment on this dataset tends to enhance the model’s performance in logical reasoning and adherence to instructions. In contrast, creative writing tasks are less aligned with these preference signals, making them more difficult to improve.\n\nThese findings suggest SSAPO achieves strong alignment for objective, structured tasks while highlighting areas for future improvement in subjective, creative domains. This pattern aligns with our theoretical framework, which optimizes for worst-case robustness - particularly beneficial for tasks with clear correctness criteria.\n\n---\n\n## Q1: Seed Set Robustness. \n\nAs addressed in W2, our framework's theoretical robustness extends to noisy seed data. For practical implementation with reduced seed sizes, we recommend proportionally increasing ε and conducting sensitivity analysis on small validation sets.\n\n---\n\n## Q2: $\\epsilon$ Selection Without Validation.\n\nIn domains without validation labels, we recommend starting with $\\epsilon$ = 0.01 for capable models and $\\epsilon$ = 0.03 for smaller models, then monitoring self-annotation consistency across iterations. Decreasing consistency indicates the need for higher $\\epsilon$ values.\n\n---\n\n## Q3: K=7 Performance Drop.\n\nThe degradation stems from optimization complexity rather than approximation quality. Additional tangent points create a more complex convex program that the used solver struggles to optimize reliably. $K=6$ provides sufficient approximation fidelity while maintaining numerical stability.\n\n---\n\n## Q4: Failure cases.\n\nAs detailed in W3, **SSAPO** primarily struggles with highly subjective tasks requiring cultural nuance or creative expression. However, for the structured reasoning and instruction-following tasks that comprise the majority of alignment benchmarks, **SSAPO** demonstrates consistent advantages.\n\n\n*[1]* SPA: Direct Preference Judgment for Efficient LLM Alignment, ICLR'25\n\n*[2]* UltraFeedback: Boosting Language Models with Scaled AI Feedback, ICML'24"}, {"review_id": "tZ6S1WCoka", "reviewer": "Reviewer_KyCS", "summary": "The paper \"Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment\" introduces a novel method to align large language models (LLMs) efficiently with human preferences using minimal human-annotated data. This method, termed Stackelberg Self-Annotated Preference Optimization (SSAPO), leverages a game-theoretic Stackelberg formulation alongside distributionally robust optimization (DRO) to iteratively refine and robustly handle self-generated annotations. Overall, the paper presents a significant advancement in the efficient and robust alignment of LLMs with human preferences. The Stackelberg self-annotation method is both theoretically rigorous and empirically effective, representing a substantial step forward in alignment research. Addressing computational complexity and further exploring hyperparameter sensitivities will enhance its practical deployment. Nevertheless, the proposed framework stands as an impressive and highly promising contribution to the field.", "questions": "Why did you choose 3.3% of the UltraFeedback dataset? Could it be 1%? Can you please bring some theoretical insight here? Else, the paper lack generazability.", "limitations": "NA", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nInnovative Framework:\nThe introduction of a Stackelberg game-based approach (SGPO) to model alignment as a robust two-player game is highly innovative. This formulation explicitly addresses potential adversarial shifts or biases in self-annotated preference data, offering formal robustness guarantees.\n\nData Efficiency:\nSSAPO significantly reduces the required volume of human-labeled data, achieving competitive performance using just 3.3% of typically required annotations. This demonstrates substantial cost-efficiency and practicality.\n\nRobustness to Annotation Noise:\nBy employing a DRO approach within an ϵ-Wasserstein ball, SSAPO effectively guards against systematic errors and biases in synthetic labels, ensuring model stability even with minimal human supervision.\n\nTheoretical Guarantees:\nThe paper provides rigorous theoretical analyses, including proofs for the existence of Stackelberg equilibrium, convergence guarantees, and bounded regret under distributional shifts. These are significant theoretical contributions that underscore the robustness of the method.\n\nStrong Empirical Results:\nExtensive experiments demonstrate the superior performance of SSAPO compared to baseline methods on standard benchmarks like AlpacaEval and MT-Bench, clearly validating its practical effectiveness.\n\nAreas for Improvement:\n\nComputational Complexity:\nAlthough addressed through grouping heuristics, the computational complexity of solving the DRO problems remains significant, especially at scale. Further optimization or heuristics to reduce this complexity would enhance practical applicability.\n\nApproximation Trade-offs:\nThe approximations involved in the DRO step (e.g., piecewise linear under-approximation, uniform grouping) potentially weaken robustness guarantees. A deeper analysis of the implications of these approximations on empirical performance would provide valuable insights.\n\nGeneralizability:\nWhile results are compelling, broader evaluations across various datasets and different model architectures could strengthen claims regarding the method's generalizability.\n\nSensitivity to Hyperparameters:\nEmpirical results indicate sensitivity to hyperparameters like the Wasserstein radius (ϵ) and tangent size (K). Additional insights or adaptive strategies for selecting these parameters in practice could improve robustness.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive feedback. We believe our paper makes significant theoretical and empirical contributions to data-efficient LLM alignment through our novel Stackelberg game formulation. After carefully considering your concerns, we have provided detailed clarifications below that demonstrate how our work addresses each point. **We respectfully ask that you reconsider your assessment in light of these explanations**, particularly noting that our method achieves state-of-the-art performance with only 3.3% labeled data while providing formal robustness guarantees—a combination that represents a substantial advance in the field.\n\n---\n\n## W1: Computational Complexity.\n\nThank you for raising this important practical consideration. We have conducted a comprehensive analysis of computational complexity in Appendix E.2, which demonstrates that our approach is both theoretically sound and practically feasible.\nWhile the theoretical complexity of the DRO problem for a single group ranges from $O(G·K)$ to $O((GK)^\\gamma)$, modern optimization techniques significantly improve practical performance. Specialized solvers including cutting-plane methods and primal-dual algorithms routinely achieve computational efficiency far exceeding worst-case theoretical bounds, as documented in the DRO literature **[Mohajerin Esfahani & Kuhn, 2018]**.\n\nMost critically, our DRO computation is **fully parallelizable and performed entirely offline**, which fundamentally transforms the scalability profile. This design philosophy aligns with successful large-scale systems in machine learning: **Transformers[1]** leverage parallel attention mechanisms, **Mixture of Experts[2]** employs conditional computation across experts, and **Zero Redundancy Optimizer[3]** distributes optimization states across devices. Our method inherits these architectural advantages, enabling efficient scaling to production environments.\nIn practice, our experiments demonstrate that solving DRO problems for 100K+ preferences requires only 45 minutes on standard hardware (for $G=100$), with linear scaling through parallelization. This positions **SSAPO** as a practical solution for real-world deployment.\n\n---\n## W2: Approximation Trade-offs.\n\nWe appreciate your thorough consideration of our approximation strategies. We have provided a detailed theoretical analysis in Appendix F that directly addresses these concerns.\n\nOur analysis demonstrates that both approximations—piecewise linear under-approximation and uniform grouping—**preserve the fundamental Stackelberg game structure** that underlies our theoretical guarantees. Specifically:\n1. The piecewise linear approximation maintains a valid lower bound on the original objective, ensuring our adversarial distribution remains feasible while slightly reducing pessimism. This conservative approximation actually strengthens practical performance without violating theoretical guarantees.\n2. The uniform grouping strategy, while not solving the global optimization exactly, maintains local robustness within each group. Our theoretical analysis shows that the aggregated solution retains $O(\\epsilon)$-bounded regret properties (Theorem 2.5).\n\nEmpirically, these approximations demonstrate remarkable effectiveness: using only 3.3% of labeled data, **SSAPO** achieves 24.44% length-controlled win rate on AlpacaEval 2.0, substantially outperforming baselines. This empirical validation, combined with our theoretical analysis, confirms that the approximations enhance rather than compromise the method's practical utility.\n\n---\n\n## W3: Evaluation protocol.\n\nThank you for this constructive suggestion. Our experimental protocol follows the established  **SPA [4] protocol (ICLR’25 Oral, rated 10/8/8)**. This standardized evaluation ensures direct comparability with state-of-the-art baselines.\n\nAlpacaEval 2.0 and MT-Bench represent the gold standard benchmarks in LLM alignment research, chosen specifically for their comprehensive coverage of real-world tasks and robust evaluation metrics. Our significant improvements over SPA baselines on these benchmarks—achieving 35.82% win rate versus GPT-4 compared to SPA's 21.13%—demonstrate strong generalization across diverse tasks.\n\nWhile we acknowledge that additional datasets and architectures would provide valuable insights, **the current evaluation already spans multiple model families (Mistral-7B and LLaMA-3-8B) and task categories (instruction following, multi-turn dialogue, coding, mathematics)**. This breadth, combined with our theoretical foundations, provides strong evidence for the method's generalizability. We welcome the opportunity to explore additional evaluation settings in future work.\n\n---\n\n## W4: Choice of Hyperparameters.\n\nThank you for highlighting this practical consideration. Our analysis in Section 4.3 provides both theoretical insights and empirical guidelines for hyperparameter selection.\nOur ablation studies reveal interpretable patterns: the Wasserstein radius $\\epsilon$ controls the robustness-conservatism trade-off, while the tangent size $K$ balances approximation quality against computational complexity. These relationships follow naturally from our theoretical framework, providing practitioners with principled guidance.\nBased on extensive experiments, we recommend:\n- **K = 6** as a robust default, validated across multiple models and datasets\n- **Adaptive $\\epsilon$ selection** based on model capability: $\\epsilon \\in [0.005,0.02)$ for high-quality models with lower self-annotation noise, and $\\epsilon \\in [0.01,0.05]$ for smaller models requiring greater robustness\n\nThis adaptive strategy reflects the insight that stronger models produce more reliable self-annotations, allowing tighter Wasserstein constraints without excessive conservatism. We have successfully applied these guidelines across different model scales, demonstrating consistent improvements over baselines.\n\n---\n\n## Q1: Jusification on the choice of 3.3% data.\n\nThank you for this important question about data efficiency. Our choice of 3.3% (2K samples from 60K total) follows the established **SPA [4] protocol (ICLR’25 Oral, rated 10/8/8)**, which is **designed for data-scarce settings**. The small number of seed data（2K of 60K in total, only 3.3%）**reflects this context**, and our results are **directly comparable to the SPA baselines**, where we demonstrate significant improvements in performance.\n\nThe 2K of 60K (the percentage 3.3%) represents a principled balance: it demonstrates substantial data reduction (30× fewer labels than standard approaches) while maintaining sufficient statistical power for robust self-annotation bootstrapping. Our theoretical framework (Section 2) shows that the Stackelberg game formulation provides $O(\\epsilon)$-bounded regret even under limited data, explaining why such dramatic data reduction remains effective. \n\nThe empirical results validate this choice: **SSAPO** achieves performance comparable to or exceeding methods using 100% labeled data, demonstrating that 3.3% captures sufficient preference structure for effective alignment. While exploring even lower percentages (e.g., 1%) would provide interesting insights, **the current setting already establishes **SSAPO** as a breakthrough in data-efficient alignment**, reducing annotation requirements by an order of magnitude while maintaining SOTA performance.\n\nA different perspective we would like to clarify is that the 2K labelled preference pairs with our SSAPO method enables comparable performance of SOTA methods using 60K labelled preference pairs. In realistic settings, 2K labelled preference pairs is much more easier to acquire compare to 60K labelled preference pairs. In this perspective, **the size 2K is more informative than 3.3% or 1%**.\n\n*[1]* Attention is All You Need, NeurIPS'17\n\n*[2]* GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, ICLR'21\n\n*[3]* ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, SC'20\n\n*[4]* SPA: Direct Preference Judgment for Efficient LLM Alignment, ICLR'25"}]

Author rebuttal:

Meta-review: The paper proposes a game theoretic approach for a robust alignment of LLMs (SGPO), a convergence analysis of the approach and an effective data efficient algorithm (SSAPO) . 

Reviewers appreciated the work and the experiments presented in the paper. Two main concerns were raised: regarding the approximations needed to obtain the solution of the DRO problem and the computational complexity of the latter. Authors brought clarification on these and we encourage them to incorporate this in the paper.

While the paper is interesting, it has an issue that was raised by the AC and discussed with the SAC, regarding the disconnect between the theoretical approach presented in the paper in Section 2.2 (eq 4 ) and the practical instantiation in Section 3. All throughout authors confuse where the robustness ball is considered. 

In Section 2.2, the Wasserstein ball is defined on the preference data i.e for  $\hat{P}= \frac{1}{n} \sum_{i=1}^n \delta_{x,y^i_I, y^i_w}$ (this is a measure in a higher dimensional space) ,meaning $U_{\epsilon}(\hat{P})$ is the Wasserstein ball centered around $\hat{P}$ (x not impacted just y ). 

In practical implementation of the work (Section 3) the Wasserstein ball is rather on $\Delta R_{\pi} (x,y_{w},y_{l})= R_{\pi}( x,y_{w})-R_{\pi}( x,y_{l})$ that is one dimensional.  Denote  $\Delta R_{\pi} [\hat{P}]$ the push forward of $\hat{P}$ by $\Delta R_{\pi}$. 

 Hence the original problem introduced  in Section 2.2 should be : 

$\max_{\pi} \min_{   \alpha \in U_{\epsilon} ( \Delta R_{\pi} [\hat{P}]) } E_{\xi \sim \alpha} \log \sigma(\xi)$

 rather than how it is written in the paper: 

$\max_{\pi} \min_{   P \in U_{\epsilon} ( \hat{P}) } E_{P} \log \sigma(R_{\pi}( x,y_{w})-R_{\pi}( x,y_{l}))$

The ball where the robustness is considered involves $\pi$ and not as exposed throughout the theoretical derivation in the work. 

Indeed the robustness ball in the paper is rather on the annotation and not on the sentences, wasserstein 1 is not that meaningful on sentences.  

Section 2 of the analysis in the paper needs to be fixed with the correct DRO problem above, this will impact all the theoretical results in this section. 

After discussions with the SAC we recommend acceptance of this paper with the condition that authors fix Section 2 as explained above.
Decision: Accept (poster)