AMPO: Active Multi Preference Optimization for Self-play Preference Selection
Authors: Taneesh Gupta
Rahul Madhavan
Xuchao Zhang
Chetan Bansal
Saravan Rajmohan
Venue: ICML 2025 poster (2025)
Keywords: Preference Optimization
Active Learning
Multi-Preference Optimization
RLHF
Primary area: deep_learning->large_language_models
TL;DR: Active Selection of on-policy generation for enhanced group-based preference optimization

Abstract: Multi-preference optimization enriches language-model alignment beyond pairwise preferences by contrasting entire sets of helpful and undesired responses, enabling richer training signals for large language models. During self-play alignment, these models often produce numerous candidate answers per query, making it computationally infeasible to include all of them in the training objective. We propose Active Multi-Preference Optimization (AMPO), which combines on-policy generation, a multi-preference group-contrastive loss, and active subset selection. Specifically, we score and embed large candidate pools of responses, then pick a small but informative subset—covering reward extremes and distinct semantic clusters—for preference optimization.The resulting contrastive-training scheme identifies not only the best and worst answers but also subtle, underexplored modes crucial for robust alignment. Theoretically, we provide guarantees of expected reward maximization using our active selection method. Empirically, AMPO achieves state-of-the-art results on AlpacaEval with Llama 8B and Mistral 7B. We release our datasets [here](https://huggingface.co/Multi-preference-Optimization).

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Taneesh Gupta"
Rahul Madhavan 22
Xuchao Zhang'
Chetan Bansal
Saravan Rajmohan
Abstract
Generation
Filtering
Multi-preference optimization enriches language- model alignment beyond pairwise preferences by contrasting entire sets of helpful and undesired responses, enabling richer training signals for large language models. During self-play align- ment, these models often produce numerous can- didate answers per query, making it computa- tionally infeasible to include all of them in the training objective. We propose Active Multi- Preference Optimization (AMPO), which com- bines on-policy generation, a multi-preferee group-contrastive loss, and active subset selection. Specifically, we score and embed large candidate pools of responses, then pick small but informa- tive subset-covering reward extremes and dis- tinct semantic clusters- for preference optimiza- tion. The resulting contrastive training scheme identifies not only the best and worst answers but also subtle, underexplored modes crucial for robust alignment. Theoretically, we provide guar- antees of expected reward maximization using our active selection method. Empirically, AMPO achieves state-of-the-ar results on AlpacaFial with Llama 8B and Mistral 7B. We release our datasets here.
Answer
Batinga Selection. AMPO Naive
L
Answer
Qiuery
Clustering Based Selection: AMPO Coreeee
Answer
Model learns ueing a suboet Ik out of responses.
Auswwer 21
Kiapcack Based, Selectioc AMPO OptSwlect
good of opernitation
leads
Fast Fast-rraining Optimizarioo
Filtered, subest e rosconses
Optimization
Figure 1. Overview of the Active Multi-Preference Optimization framework. Given query, the LLM generates diverse responses, which are evaluated by a rater model. Selected responses with different ratings and semantics are then used to train and align the LLM through preference optimization. Active selection of the preferenees to optimize over improves training dynamics.
when providing feedback (Cui al., 2023; Chen al., 2024a; Gupta et al., 2024b). By learning from multiple "good" and "bad" outputs simultaneously, these approaches deliver richer alignment signals. At the same time, an im- portant trend in alignment is the shift to on-policy data generation, where the policy learns directly from its own distribution of outputs at each iteration (Chen et al., 2024b; Kumar et al., 2024; Wu al., 2023; 2024). This feedback loop can accelerate convergence ensuring that the training data stays relevant to the model's 's behavior.
1. Introduction
Preference Optimization (PO) has become standard ap- proach for aligning large language models (LLMs) with human preferences (Christiano et al., 2017; Ouyang et al., 2022; Bai et al., 2022). Traditional aligmment pipelines typ- ically rely on pairwise or binary preference comparisons, which may not fully capture the subtleties of human judg- ment (Rafailov et al., 2024; Liu et al., 2024a; Korbak al., 2023). As a remedy, there is increasing interest in multi- preference methods, which consider entire sets of responses
However, multi-prefereee neck: modern LLMs can easily generate dozens of candidate responses per query, and incorporating all of these into a single training objective sible (Askell et al., 2021). Many of these sampled responses end up being highly similar or near-duplicates, providing limited additional information for gradient updates (Long et al., 2024). Consequently, naive attempts to process all generated responses cause both memory blow-ups and di- minishing returns in training (Dubey al., 2024). Given these constraints, identifying small yer highly informa- tive subset of candidate responses is critical for effective multi-preferentt learning.
alignment faces a serious bottle-
become computationally infea-
Rahul Equal contribution 'Microsoft *IISc, Bangalore. Correspon-
dence to: Tancesh Gupta tataanegpta@mmmrosooott Madhavan <mrahul @ iis..com>..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To understand the challenge of selecting informative re- sponses, consider the query's s answer space as a response

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
landscape (See Figure 2). Each point in this landscape represents a possible response, characterized by its seman- tic properties (its location in the embedding space) and its quality (determined by reward model). Furthermore, the LLM's current policy defines a probability density over this landscape. A naive approach of randomly sampling re- sponses and treating them equally might overemphasize fre- quently generated areas, even if they contain only mediocre or slightly problematic answers. This risks overlooking crit- ical feedback from less common, yet highly informative, re- gions-sucd such as subtle failure points in underexplored seman- tic terrains, or exceptionally good responses that are rarely generated. Therefore, an ideal selection strategy must ac- tively explore this landscape, identifying responses that are not just "good" or "bad" but also semantically distinct, cov- ering reward extremes, and exposing underexplored modes that are crucial for robust alignment (Yu et al., 2024). In this paper, we show that this targeted selection can be tied to an optimal way of suppressing undesired modes under a mild Lipschitz assumption (see Section 7).
Generations are cheap
AMPO helps select an informative subset of data for optimization
Optimization is costly
Selection is hard
Figure 2. A learner can easily generate responses to given query, but selection of a much smaller subset & < n to train on is hard problem. This paper addresses this problem through techniques from clustering as well as knapsack related problems.
negative-selection strategy, and (iii) empirical evaluations showing that AMPO achieves state-of-the-art results com- pared to strong alignment baselines like SIMPO. Altogether, our approach enables models to learn more reliably from diverse sets of model behaviors.
1.1. Our Contributions
Algorithmic Novelty: We propose Active Multi- Preference Optimization (AMPO), an on-policy frame- work that blends group-based preference alignment with active subset selection without exhaustively training on all generated responses. This opens out avenues for re- search on how to select for synthetic data, as we outline in Sections 5 and 9.
At its core, the problem of efficiently selecting the most impactful responses for feedback aligns with the princi- ples of active learning (Cohn et al., 1996; Ceravolo et al., 2024; Xiao et al., 2023). By selecting a small yet semanti- cally diverse subset of responses, the model effectively cre- ates a curriculum for itself. Rather than passively training on random or exhaustively sampled data, an active learner queries the examples that yield the greatest improvement when labeled. In our context, we actively pick handful of responses that best illustrate extreme or underexplored behaviors- whether very good, very bad, or semantically distinct (Wu et al., 2023). This helps the model quickly eliminate problematic modes while reinforcing the most desirable responses. Crucially, we remain on-policy: af- ter each update, the newly refined policy generates a fresh batch of responses, prompting another round of active subset selection (Liu et al., 2021).
Theoretical Insights: Under mild Lipschitz assumptions, we show that coverage-based negative selection can sys- tematically suppress low-reward modes and maximizes expected reward. This analysis (in Sections 6 and 7) con- nects our method to the weighted K-medoids problem, yielding performance guarantees for alignment.
State-of-the-Art Results: Empirically, AMPO sets a new benchmark on AlpacaEval with Llama 8B, surpassing strong baselines like SIMPO by focusing on a small but strategically chosen set of responses each iteration (see Section 8).
Optimization (AMPO), a framework that unifies (a) on-policy data gen- eration, (b) group-based preference learning, and (c) ac- tive subset selection. Specifically, we adopt reference- free group-contrastive objective known REFA (Gupta et al., 2024a), which jointly leverages multiple "positive" and "negative" responses in single loss term. On top of this, we explore various active selection schemes-ranging from simplest bottom-K K ranking (Meng et al., 2024) to coreset-basee basee clustering (Cohen-Addad et al., 2021: 2022; Huang et al., 2019) and more theoretically grounded "Opt- Select" method that ties coverage to maximizing expected reward. Our contributions are: (i) a unifying algorithmic pipeline for multi-preeerence alignment with active selec- tion, (ii) theoretical results demonstrating that coverage of distinct clusters à la k-medoids, can serve as an optimal
We propose Active
Dataset Releases: We publicly release our AMMO-coreset-seeettoo 2MPO-Opt-Selecti.on datasets on Hugging Face. These contain curated response subsets for each prompt, facilitating research on multi- preference alignment.
and
2. Related Work
Recent advances in preference optimization (Rafailov et al., 2024; Azar et al., 2023; Hong et al., 2024a) have moved beyond simple pairwise comparisons to include multiple responses per query. This shift is largely driven by datasets like UltraFeedback (Cui et al., 2023), which provide scalar
2

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
rewards for diverse candidate outputs. Within this multi- preference paradigm, methods such as InfoNCA (Chen et al., 2024a) utilize noise-contrastive objectives to align mod- els with scalar rewards. AMPO builds upon these multi- preference approaches by employing REFA (Gupta et al., 2024a), a group-contrastive objective that contrasts sets of selected and rejected responses to emphasize multiple highly informative (positive or negative) examples.
Response Evaluation and Embedding. Each response Yj (for j = ,..... NN is assigned scalar reward rj R(z, Uy) E [0,
(I)
where R a fixed reward function. We also embed each response via ej E(1)) Rdd where & is encoder capturing semantic properties. The distance between any two responses 1/j and yr in this embedding space is denoted d(ey, et) (e.g., Euclidean or L2 distance).
A erucial development in LLM alignment is the adoption of on-policy or "self-play" data generation (Chen et al., 2024b; Wu et al., 2024). While ensuring training data relevance and accelerating convergence, this process can generate a vast number of candidate responses per query. Incorporating all these responses into the training objective becomes compu- tationally infeasible and leads diminishing returns due to high similarity and redundancy (Askell et al., 2021; Long et al., 2024).
Budgeted Subset Selection and Coverage. Given the N generated responses, our objective is to select subset S C (y1,..., yN of size K < N, where K is a pre-specified budger of responses to be used for training. The selection aims to maximize a utility function 24 that considers factors such as response quality (rewards), probability of generation, and embedding space coverage. Formally,
arg max S' s'C u(s', frytujes, (oyteees)). (2)
S
To address this computational bottleneck, AMPO integrates principles from active learning (Cohn et al., 1996; Settles, 2009). Our active subset selection strategies draw from combinatorial optimization and clustering techniques, such as weighted k-medoids and coreset construction (Har-Peled & Mazumdar, 2004; Cohen-Addad et al., 2022). These methods enable AMPO to efficiently identify a small, high- impact subset of responses that effectively cover the diverse landscape of generated outputs, encompassing both reward extremes and distinct semantic regions, thereby facilitating robust and efficient alignment.
A key aspect of a "good" subset S* is its coverage of the original N responses. High coverage implies that for any response y/ from the original N candidates, its minimum distance to any response yl E S* is small. More formally, we can define coverage cost for a chosen subset S with respect to the initial N responses is defined as:
N
coverage cost(S) = Σ
min d(e,, e1).
91ES
3. Notations and Preliminaries
A subset S provides high coverage if this sum is minimized, ensuring that the selected K responses representative of the diverse characteristics present in the initial pool of N. The active selection strategies discussed later (Section 5) aim to find such high-coverage subsets.
On-policy alignment of LLMSs with learnt preference scores often involves generating multiple candidate responses (say N responses) for given prompt. Utilizing all these N candidates for training can be computationally prohibitive and may offer diminishing returns due to response simi- larity. Our framework, Active Multi-PPeeerence Optimiza- tion (AMPO), addresses this by focusing on the active se- lection of a small, yet highly informative, subset of these responses within pre-specified budget (say budget is K with K < N). This section establishes the notation and foundational concepts for generating responses, evaluating them, defining selection criteria like coverage, and choosing subsets for efficient alignment using a group-contrastive objective.
Group-Contrastive Alignment with REFA. Once the subset S* (of size K) is selected, it is partitioned into set of accepted responses St and set of rejected responses S-, such that S = S USS and S n - = 2. The specific criteria for this partitioning can vary (e.g., based on reward thresholds, or a one-vs-many split as in Algo- rithm 1). For a query x, we train using the reference-free conp-conteattivee objective REFA (Gupta et al., 2024a) by contrasting these two sets:
Queries, Policy, and Response Generation. Let D (21,.2...... .2....,... .MM} } be a dataset of M queries (or prompts), each from a larger space X. We have a policy model Po (v I)), parameterized by 6, which produces distribution over possible responses y E y For each query Ii, we gener- ate a pool of N candidate responses by sampling from (a 2;;) at a fixed temperature (e.g., Temp. = 0.8) For notational simplicity, we consider a single query x and its N sampled responses (V,....,NN]
Σ
exp se so
(31
1z)]
3, S-
LREFA (0) = ---log log
(3)
exp (o (Us 1z)
9s E(S+US-)
1i.2. }
where the score for a response yj. se (yy Iz) incorporates its log-probability under the current policy Pa(y; xx) and its associated reward ry. This score is given by:
so (yg x) = log Po(y; xx) + arlr - Fl.
3

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Algorithm 1 AMPO: One-Positive vs. K-Active Negatives
Algorithm 2 AMPO-CORESET via k-means
1: Input: (1) A set of responses }Br} sampled from Poly æ); (2) Their rewards {ri). embeddings {e: and probabilities {m:}: ; (3) Number of negatives K, initial Pe. and hyperparameter a
Input:
(1) N responses, each with embedding e; Rd and
rating ri
3: (2) Desired number of negatives K
2: Output: (i) Positive y4; (ii) Negatives {ys (iii)
4:
Updated parameters O via REFA
5: Step I: Run K-means on embeddings
3: 1. Select One Positive (Highest Reward)
6: Initialize (c1,...,, C. I2 (c.g., via K K-means++)
4: i4 + arg max; N...NN Tú, Y+ f-
repeat
5: 2. Choose K Negatives via Active Selection
(1i) arg min / -c,l.
i 1=1,..,,
6: Q < fl....,NY N} \ {i+}
Cj Zarto=s
Eanto= 6
j j= j= K
7: S ACTIVESELECTIONS
(ri), (esh, ()
10: until convergence
8: 3. Form One-vs.- K REFA Objective
9: F +- "4 "4+E, +-, r
11:
1+K
12: Step 2: In each cluster, pick the bottom-rated response
10: For each y4:
13: For each j E [l...., K}, define Cj = { T(1) }}
11: (a(1)) = log Polys x)) + arr; - -rl
14: Then 1j = arg minkec, Ts j = 3=1....
expp :1)
15:
12: LREFA
log
exp (b++)) + E exp (w)))
16: Step Return negatives
4. Update Model Parameters: 0 -n VALREFA (0)
17: S = { [ 12 :..... .xk
14: return The chosen positive Y/+, the negative set {ui}ees,,an the updated parameters 0
18: return S as the set of K negatives
1. A reward function r; = R(z, y1) E [0,
Here, a is a hyperparameter scaling the influence of the reward, log Poly, x) is the generation probability of re-
2. An embedding e; = =\(Y:).
3. A model probability क. Po (yt Ir).
sponse Vj given query æ, and F meany, es (2, vy))- REFA encourages the model to increase the collective pref- erence score of responses in S+ relative to those in S This procedure extends to any dataset D by summing LREFA across all queries. Subsequent sections detail strategies for selecting S and partitioning it maximize training efficiency and alignment quality.
Selection algorithms may be baasg selection (to iden- tify truly poor excellent answers) with coverage-based se- lection (to explore distinct regions in the embedding space), we expose the model to both common and outlier responses. This ensures that the REFA loss provides strong gradient signals across the spectrum of answers the model is prone to generating. In Algorithm 1, ACTIVESBLEETIONN is generic subroutine that selects set of K "high-impact" negatives. We will detail concrete implementations (e.g. bottom- K by rating, clustering- based. etc.) in later sections.
4. Algorithm and Methodology
Our methodology employs a one-vs-K selection scheme: one best response chosen as positive, and an active sub- routine selects K negative responses from the remaining N 1 candidates. This active selection must balance three key objectives:
5. Active Subset Selection Strategies
This section details two effective strategies for actively selecting negative responses within AMPO: AMPO- BottomK, which selects the lowest-rated responses, and AMPO-Coreset, a clustering- method ensuring broad semantic coverage by selecting one negative per cluster. We connect AMPO-Coress to coreset construction literature (Section E).
Probability:
High- hruh--probbbility responses under Po(y | x)) can dominate even suboptimal by reward.
Rewards:
Simply selecting extremes by reward misses problematic "mediocre" outputs.
Semantics: Diverse but undesired responses in distant em- bedding regions must be penalized.
5.1. AMPO-Bottomi
is the most direct approach that we use for comparison: given sampled responses and their scalar rat- ings (r:) we simply pick the K lowest-rated responses as negatives. This can be expressed as:
While positives reinforce single high-reward candidate, active negative selection balances probability, reward and diversity to systematically suppress problematic regions of the response space.
S argtopk,(-1)) K),
(4)
Algorithm. Formally, let {yi..... yN be the sampled responses for a single prompt x. Suppose we have:
which identifies the K indices with smallest ri. Although

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Algorithm 3 AMPO-OPTSELECT via Solving MIP
Algorithm 4 AMPO-OPTSELECT via Coordinate Descent
1: Input: Candidates {ye Ji=1 with Ti, e; integer K
1: Input: Set / = (I,...,,N, N), integer K, distances Ar.- rewards {ra}
2: Compute 4op = arg max, T4
3: Let W/ = cxp( F - r;)) with 7 as mean reward
Find stop = arg max, Pi
4: Solve Problem equation to get {a5h, feigh
3: Compute w; exp( T - rz) and di.j Ais
5: Let Sneg = { }} (size K)
4: Initialize random subset S [ I (hrpp} of size K
6: refurn [to J U neg for REFA training
5: while improving do
6: Swap jout E S with Jin $ S if it decreases
Eier wi minjes
conceptually simple, this method can be quite effective when the reward function reliably indicates "bad" behavior. Fur- thermore to break-ties, we use minimal cosine similarity with the currently selected set.
7: end while
return Sneg = 5 (negatives) and hoo (positive)
Lipschitz constraint on the policy. Concretely, if the distance is d = ||e, eyl1. and the model's Lipschitz constant is L, then the probability of y, cannot remain above L d if Vj is forced to probability zero
5.2. AMPO-Coreset (Clustering-Basse -ased Selection)
IOPO-BOTTOMK may overlook problematic modes that are slightly better than the bottom-K, but fairly important to learn on. A diversity-driven approach, which we refer to as AMPO-CORESET, explicitly seeks coverage in the embedding space by partitioning the N candidate responses into K clusters and then selecting the lowest-rated response within each cluster. Formally:
From an expected reward perspective, assigning zero prob- ability to low-reward responses (and their neighborhoods) improves overall alignment. To capture this rigorously, ob- serve that the penalty from retaining below-average answer Yi be weighted by:
3j -ar min iEC;
ri ri.j-I...,K,5-- { ( }
(5) where ir is (for instance) the mean reward of (ri). Intu- itively, wi is larger for lower-reward ye, indicating is more harmful to let y and its neighborhood remain high proba- bility.
wi = exp (r - -r),,
where Cj is the set of responses assigned to cluster j by K K-means algorithm (Har-Peled & Mazumdar 2004; Cohen- Addad et al. 2022; see also Section E). The pseudo-code is provided in Algorithm 2.
Next, define a distance matrix
This approach enforces that each cluster-a potential "mode" in the response space- contribtuss at least negative example. Hence, AMPO-CORESET can be interpreted as selecting representative negatives from diverse semantic regions, ensuring that the model is penalized for a wide variety of undesired responses.
Ar = ||: - -5ll2- 1 SSijsN.
(6)
Selecting a subset S C {I,...,,N N of "negatives" to pe- nalize suppresses the probability of each in proportion to minjes Arj- Consequently, natural cost function mea- sures how much "weighted distance" yi has to its closest chosen negative:
6. Opt-Select: Active Subset Selection by Optimizing Expected Reward
N
Cost(S) =
Σ
wi min Aij- jES
We propose Opt-Select, a strategy for choosing K negative responses and one positive to maximize expected reward un- der a Lipschitz assumption. Opt-Select models the local in- fluence of penalizing negatives, formulating an optimization problem suppress low-reward regions while preserving high-reward modes. We present solutions via mixed-integer programming (MIP) and local search.
Minimizing equation 7 yields subset S of size K that "covers" or "suppresses" (large w;) as possible. We then add one positive index keop with the highest ry to amplify a top-quality answer. This combination of one positive plus K negatives provides a strong signal the training loss.
many low-reward responses
Interpretation and Connection to Weighted k-medoids. If each negative j "covers" responses within some radius (or cost) A.j. then equation 7 is analogous to a weighted K K-medoid objective, where we choose K items (negatives) to minimize a total weighted distance. Formally, this can be cast as a mixed -ittegr program (MIP) (Problem below). For large N, local search offers an efficient approximation.
6.1. Lipschitz-Driven Objective
Let {y) |i= be candidate responses sampled on-policy, each with reward r; E 1] and embedding e; E RRd Suppose that if we completely suppress a response yj (i.e. set its probability to zero), all answers within distance ||e, eyll must also decrease in probability proportionally, due to a
5

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
6.2. Mixed-Integer Programming Formulation
where T'ax is the rating of the best-rated response. Dividing by a normalization factor (such exp(rmma r) I.), one arrives at cost akin to w; di,; with w; = exp(f- rz).
Define binary indicators zj = 1 if we choose yj as a neg- ative, and žú,j 1 if i is assigned to j (i.e. minjes A is realized by j). We write:
Remark 6.1. This aligns with classical min-knapsact of minimizing some costs subject to some constraints, and has close alignment with the weighted K K-medoid notions of "covering" important items at minimum cost.
N W/ Vi
Problem P:
min [ye(0,1), (0,11 9:20 i=1
(8)
7. Theoretical Results: Key Results
S.L.
Σ 3ラ = K, Zi.j S rzz
Zi.j = 1,
This section presents core theoretical statements underpin- ning AMPO's active selection. Full proofs are in Appen- dices C-E. We assume a budget of K responses to be se- lected from N candidates.
j=1
5=1
y1 S Arj + M (1 - أزjي,
y 2 Ar, M (1 - 2,,), Vi
(9)
7.1. Setup and Assumptions
where M- m A.j. In essence, each is forced to assign to exactly one chosen negative j. making y/ A. Le. the distance between the answer embeddings for answer {i Minimizing E, Wi Vi (i.e. equation 7) then ensures that low-reward points (w, large) lie close to at least one penalized center.
(A1) L-Lipschitz Constraint. When response yj is pe- nalized (probability Pj 0), any other response yi within embedding distance A., must satisfy p. S A...
(A2) Single Positive Enforcement. We allow one highest- reward response Whop to be unconstrained, i.c. Phop is not pulled down by the negatives.
Algorithmic Overview. Solving P gives the K negatives Sweg. while the highest-reward index 'top is chosen as a positive. The final subset } U Sneg is then passed to the REFA loss (see Section 4). Algorithm 3 outlines the procedure succinctly.
(A3) Finite Support. We focus finite set of N candi- date responses .. yn } and their scalar rewards {r,), each embedded in Rd with distance A.,j = lle: ey ||
7.2. Optimal Negatives via Coverage
6.3. Local Search Approximation For large N, an exact MIP can be expensive. A simpler local search approach initializes random subset S of size K and iteratively swaps elements in and out if it lowers the cost equation 7. In practice, this provides an efficient approximation, especially when N or K grows.
Theorem 7.1 (Optimality of OPT-SELECEE Under assump- tions (AI HA3), let S* be the set of K "negative" responses that minimizes the coverage cost
N
Cost(S) = Σ exp(( -T)
ri) min Aij jES
(10)
1=1
Intuition. If yi is far from all penalized points j E S, then it remains relatively "safe" from suppression, which is undesirable if r; is low (i.e. w, large). By systematically choosing S to reduce Σ, w/ minjes di,j, we concentrate penalization on high-impact, low-reward regions. The local search repeatedly swaps elements until no single exchange can further reduce the cost.
where F is a reference reward (e.g. average of frit). Then S also maximizes the expected reward among all Lipschitz- compliant policies of size K (with single positive). Conse- quently, selecting St and allowing Proop Re 1 is optimal.
Sketch of Proof. (See Appendix C for details.) We show one-to-one correspondence between minimizing coverage cost Σ, w, minjes As,, and maximizing the feasible ex- pected reward E, rep; under the Lipschitz constraint. Low- reward responses with large w; must lie close to at least one negative E S; else, they are not sufficiently suppressed. A mixed-integer program encodes this cost explicitly, and solving it yields the unique S* that maximizes reward.
6.4. Why "Opt-Select"? A Lipschitz Argument for Expected Reward
We name the procedure "Opt-Select" because solving equa- tion& 8 (or its local search variant) directly approximates optimal subset for improving the policy's ' expected reward. Specifically, under Lipschitz constraint with constant L, assigning zero probability to each chosen negative yj im- plies neighboring answers y! at distance dij cannot exceed probability L d Consequently, their contribution to the "bad behavior" portion of expected reward is bounded by
7.3. Local Search for Weighted K-Medoids
(A4) Weighted K-Medoids Setup. We have N points (1....,N) NN in metric space with distance d(.,,) :) 2 0, each with weight w; 2 0. Our goal is to find a subset S of size K to minimize Cost Ei= w; minjes j(i,
exp(rinxx - r;) ( T. (....)):
6

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Mistral-Instrucl (7B)
Llama 3-Instruct (8B)
Method
AlpacaEval 2
Arena-Hard
MT--Benhh
AlpacaEval?
Arena-Hare
MT-Bench
LC (%)
WR (%)
WR (%)
GPT-4
LC (%)
WR (%)
WR (%)
GPT-4
Base
17.1
12.6
28.4
28.4
26.9
7.93
RRHF
25.3
24.8
18.1
7.6
31.3
28.4
26.5
7.9
SLiC-HFI
24.1
24.6
18.9
26.9
27.5
26.2
8.1
DPO
26.8
24.9
16.3
7.6
40.3
37.9
32.6
8.0
IPO
20.3
20.3
16.2
7.8
35.6
35.6
30.5
8.3
CPO
23.8
28.8
22.6
32.2
28.8
KTO'
24.5
23.6
17.9
7.7
33.1
31.8
26.4
ORPO
24.5
24.9
20.8
7.7
28.5
27.4
25.8
8.0
R-DPO!
27.3
24.5
16.1
7.5
41.1
37.8
33.1
8.0
SIMPO
30.1
32.3
21.1
7.6
47.6
44.7
34,9
7.5
IMPO-BottomK
32.1
37.0
22.1
7.7
50.8
50.5
45.2
8.1
AMPO-Coreset
37.3
22.6
47.8
AMPo-Opt-Select 33.1
37.8
22.8
7.7
51.6
51.2
46.4
8.0
Table Comparison of various preference optimization baselines on AlpacaEval, Arena- Hard, and MT-Bench benchmarks for Llama- 3 Instruct (8B). LC-WR represents conth-controlled win rate, and WR represents win rate. Best results bold, second bestnd-best are underlined. Our method (AMPO) achieves SOTA performance across all metrics, with different variants achieving either best or second-bes, results consistently.
Theorem 7.2 (Local Search Approximation). Suppose we apply a 1-swap local search algorithu select K medoids. Let S be the resulting local optimum and let S* be the globally optimal subset. Then
8. Experiments
8.1. Experimental Setup
Model and Training Settings: For our experiments, we utilize a pretrained instruction-tuned model (meta- Baama//etallanaa3------n--mt
Cost($ 5 Cost(S*). The running time is polynomial in N and K.
as the SFT model. These models have undergone extensive instruction tuning, mak- ing them more capable and robust compared the SFT models used in the Base setup. However, their reinforce- ment learning with human feedback (RLHF) procedures remain undisclosed, making them less transparent.
Sketch of Proof. (See Appendix D for complete proof.) Assume by contradiction that Cost(S) > 5 CSst(S"). We then show there exists profitable swap (removing some j E S and adding j E S**) that strictly decreases cost, contradicting the local optimality S.
reduce distribution shift between the SFT models and the preference optimization process, we follow the approach in (Tran et al., 2023) and generate the preference dataset using the same SFT models. This ensures that setup is more aligned with an on-policy setting. Specifically, we utilize prompts from the UltraFeddbcck dataset (Cui et al., 2023) and regenerate the resonses using the SFT models. For each prompt x. we produce 32 responses by sampling from the SFT model with a sampling temperature of 0.8. We then use the reward model (Skywork/SkyworkRRews Llama-3.1-8B-ve
7.4. Coreset Guarantee for AMPO-Coreset
(A5) Bounded-Diametee Clusters: For AMPO-CORESET, we assume the N non-positive candidate responses can be grouped into semantic clusters, each with embedding space diameter at most dmaax-
Intuition: AMPO-CORESSE selects one lowest-rated nega- tive from each of the K semantic clusters. Under the Lips- chitz constraint (A1), penalizing this single representative from a counded--iimeteer cluster (A5) effectively suppresses all other semantically similar (i.e., same-clusteri responses. This ensures broad coverage
(Liu et al., 2024b) score all the 32 responses. Then the response selected based the Active Subset selection strategies a.) AMPO-Bottomk b.) AMPO-Coreset c.) AMPO-Opt-Select
the response landscape.
Formal Result: (Theorem E. Appendix E). The induced policy's 's maximum expected reward is at least
In our experiments, we observed that tuning hyperparame- ters critical for optimizing the performance Carefully selecting hyperparameter values significantly impacts the effectiveness of these methods across various datasets. We found that setting the 3 (inverse temperature) parameter in the range of 5.0 to 10.0 consistently yields strong perfor-
Fmax
L dmax>
(11)
ie. within additive Ldmax the unconstrained optimum given assumptions on cluster diameter (dmax) and the pol-
icy's smoothness (T.).

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
30-
30
30
20
20
20-
16
10
10
0
MNNT
10
10
20
20
20
Seleded Resperoz
Sdiciid Responic
Selatei Resuzse Kaperess
30
30
1420
Hemponass
30
Other Recosmo
Odher Resporecs
Rapasuss
-50 -20
-10
20
-10
-20
-10
10
20
50
-30
-20 -10
10
20
30
LKE Dim
Figure 3. t-SNE visualization of projected digh-dimensional response embeddings into a 2D space, illustrating the separation of actively selected responses. (a) IMPO-BottomK (baseline). (b) AMPO-Coresee (ours). (c) Opt-Select (ours). We see that the traditional baselines select many responses close to each other. based on their rating. This provides insufficient feedback to LLM during preference optimization. In contrast, our methods simultaneously optimize for objectives including coverage, generation probability as well as preference rating.
mance, while tuning the 7 parameter within the range of 2 to 4 further improved performance. These observations high- light the importance of systematic hyperparameter tuning to achieve reliable outcomes across diverse datasets.
Further balances reward extremity, and embed- ding coverage, yielding well-separated response clusters and more effective supervision for preference alignment.
Opt-Select:
Figure 3 demonstrates that our selection strategies significantly improve response diversity com- pared to traditional baselines. By actively optimizing for coverage-aware selection, our methods miligate redun- dancy in selected responses, leading to better preference modeling and enhanced LLM alignment.
Key Takeaway:
Evaluation Benchmarks We evaluate our models using three widely recognized open-ended instruction-following benchmarks: MTT-Bench (Zheng et al., 2023), AlpacaEval2 (Dubois et al., 2024), and Arena-Hard v..1. These bench- marks are commonly used in the community to assess the conversational versatility of models across diverse range of queries.
Impact of Temperature Sampling for Different Active Se- lection Approaches To analyze the impact of temperature- controlled response sampling on different active selection approaches, conduct an ablation study by varying the sampling temperature from O to 1.0 in increments of 0.25 on AlpacaEval2 benchmark as demonstrated in Figure 4. We evaluate our active selection strategies observe a general trend of declining performance with increasing temperature.
AlpacaEval 2 comprises 805 questions sourced from five datasets, while MT-Bench spans eight categories with total of 80 questions. The recently introduced Arena- Hard builds upon MT--Bench, featuring 500 well-defined technical problem-solving queries designed to test more advanced capabilities.
We adhere the evaluation protocols specific each bench- mark when reporting results. For AlpacaEval 2, we provide both the raw win rate (WR) and the rate (LC), with the latter being designed to mitigate the in- fluence of model verbosity. For Arena-Hard Hard, we report the win rate (WR) against a baseline model. For MT-Bench, we present the scores evaluated by 116- which serve as the judge model.
Key Takeaway: AMPo-Coreset and AMPO-Opt-Sele demonstrate robustness to temperature variations, whereas LC-WR of SimPO and bottom-k selection are more sensitive.
win
Effect of gamma for Active Selection Approaches To investigate the sensitivity of core-set selection to different peper-parameter settings, we conduct an ablation study on the impact of varying the gamma as shown in Figure 5, As gamma increases from 1 to 3, we observe consistent improvement in both LC-WR and WR scores.
8.2. Experimental Result
Impact of Selection Strategies on Diversity. Figure 3 shows t-SNE projection of response embeddings, high- lighting how each selection method samples the answer space:
Key Takeaway: This highlights the importance of tuning gamma appropriately to maximize the effectiveness of active-selection approaches.
AMPO-BottomK: Tends to pick tight cluster of low-rated responses, limiting coverage and redundancy in feedback. AMPO-Coreset: Uses coreset-based selection to cover more diverse regions, providing coverage of examples.
Robustness to Reward Model Choice To assess the ro- bustness of AMPO to the choice of reward model, we evalu- ate performance using two distinct reward models: Skywork- Reward-LM and CRMM-Reward Table 2 presents results
8

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Best Worst
AMPO Bottonk
AMPD Coreset
AMPO-DySSelx
AMPO-Batan
AMPO-Cunset
AMPo-0pSSdell
53
52-
1.C.0
49
WR (%)
WR
44
1.C
50
50
0.25
Temperature
0.5
0.75
1.0
0.25
Temperature
0.5
0.75
1.0
48
Gamma
Gamma
2
Figure 4. Effect of Sampling Temperature on different baselines for on the AlpacaEval 2 Benchmark: (a) Length-Controlled Win Rate (LC) and (b) Overall Win Rate (WR).
Figure 5. Effect of Gamma on AlpacaEval2 for Active Subset Se- lection Strategies.
AlpacaEval 2
Arena-Hard
MTT-Bench
across three AMPO selection strategies Bottom-k, Core- set, and Opt-Select-on AlpacaEval 2, Arena-Hard, and MT-Bench.
Method
LCC (%)
WR (%)
WR (%)
GPT-4
AMPo-Opt-Seleen (1vs3)
49.6
48.5
46.1
8.03
AMPO- Saee--pp--eeee Select (1vs5)
50.3
49.9
43.9
7.84
AMPo-Opt-Select (Ivs7)
$1.6
51.2
46.4
8.11
AlpacaEval
Method
Reward Model
Table 3. Effect of increasing the negative set size (K) in AMPO- Opt-Select on AlpacaFval2, Arema-Hard, and MT-Bench.
I.C (%)
WR
AMPO-Bottone
Skywork-Reward-
50.8
50.5
AMPO--Corsse
Skywork-Reward-LMM
52.4
52.1
AMPO-Opt-Ser Opt-eecctt
Skywork-Reward-Lle
51.6
51.2
AlpacaEval
Arena-Hard
MTT-eecch
Method
AMPO-Bottomk
HMMM-Reaard-LM.
51.5
49.3
(C(()
WR (%)
WR (%)
GPT4
AMPO-Coreset
L.MM-Reward-M
52.5
49.7
AMPo-Opt-Select (N= 16)
50.6
50.1
45.5
7.76
AMPO-Opt-Select
GRM-Reward-LLM
52.9
51.7
AMPo-OpttSSlect (N (N=220
51.1
50,5
45.7
7,88
AMPO-Ope Sapo-Opt-Sellee (N=
51.6
51.2
46.4
8.11
Table 2. Comparison of AMPO-baseline on AlpacaEval 2 using LLaMA-3- (8B) across different reward models
Table 4. Effect increasing number of responses (N) for selection
using AMPO-Opt-Seller (1 vs 7) setting on AlpacaEval2, Arena- Hard, and MT-Bench.
We observe that the relative ranking of methods remains largely consistent across reward models, with Opt-Select and Coreset outperforming Bottom-k across metrics.
in AMPO-Opt-Select under a fixed 1-vs-7 contrastive set- ting. Specifically, we compare performance when N e 16, 24, 32, as shown in Table 4.
Key Takeaway: AMPO exhibits robust generalization across distinct reward models, indicating that its effec- tiveness not tied to specific reward functions.
Our findings reveal that while increasing N leads to consis- tent improvements across all evaluation benchmarks, the per- formance gains are marginal. Notably, even with N = 16, the results remain competitive, suggesting that AMPO-Opt- Select effectively identifies high-quality contrastive sets with limited candidate pools. Nonetheless, a larger N in- troduces greater response diversity, which can enhance the coverage of the preference space and lead to modest per- formance gains culminaun at N 32 with the highest scores across AlpacaEval 2, Arena-Hard, and MT-Bench.
Effect of Negative Set Size (K) in AMPO To examine how the number of negative comparisons affects perfor- mance, we evaluate AMPo-Opt-Select with increasing val- ues of K in the 1-vs-K selection saattey--eecificallly K E 3, 7. The results are presented in Table 3 across AlpacaEval 2, harda-Hard, and MT-Bench.
We observe that even with a small number of negatives (c.g..
K 3), AMPO maintains strong performance, indicating that we identify the high-utility contrastive examples. As K increases, performance improves slightly, peaking at 1-vs-7, yet the marginal gains diminish.
Key Takeaway: Increasing the pool size of generated responses for AMPO improves performance, but the method remains strong even at lower N, demonstrat- ing its efficiency and robustness in low-sample settings.
Key Takeaway: AMPO is highly effective even with a small number of negative samples, and further increases in K yield diminishing returns. This shows that our method may work in resource-constranee alignment settings where generating large negutive sets is costly.
9. Discussion & Future Work
Iteration via Active Synthetic Data Generation. The on-policy, roverage-focused active selection in AMPO natu- rally surfaces candidates for synthetic data generation. This opens avenues for future work in co-adapting the policy and reward model through this actively generated data via a robust pewicy-reward feedback loop.
Effect of Total Number of Responses (N) in AMPO- Opt-Select We investigate the impact of varying the total number of generated responses N available for selection

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Impact Statement
semi-supervised learners. Advances in neural information processing systems, 13:22243-222555 2020.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Chen, Z.,, Deng, Y., Yuan, H., Ji, K., and Gu, Q- Self-play fine-tuning converts weak language models to strong lan- guage models. arXiv preprint arXiv:2401.0133
2024b.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
References
Arya, V., Garg, N., Khandekar, R., Meyerson, A., Munagala, K. and Pandit, V. Local search heuristic for k-median and facility location problems. In Proceedings of the thirty- third annual ACM symposium on Theory of computing, pp. 21 20229,12220000 2001.
Cohen-Addad, V., Saulpic, D., and Schwiegelshohn, C. A new coreset framework for clustering. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pp. 169-182, 2021.
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint www.::2122.0086000
Cohen-Addad. V., Green Larsen, K., Saulpie, D., Schwiegelshohn, C., and Sheikh- Omar, O. A. Improved coresets for euclidean k-means. Advances in Neural In- formation Processing Systems, 35:2679-2694, 2022.
2021.
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm to understand learning from human preferences. ArXiv, 10..22000..0. 2023.
Cohn, A., Ghahramani, Z., and Jordan, M. Active learning with statistical models. Journal of artificial intelligence research, 4:129-145, 1996.
Azar, M. G., Guo, 7. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455, PMLR, 2024.
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultraeedbbck: Boosting lan- guage models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
Bachem, O., Lucic, M., and Krause, A. Practical core- set constructions for machine learning. arXiv preprint arXiv: 1703.06476, 2017.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767. 2023.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., ct al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05880 05862, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178: 2024.
Cacchiani, V., Iori, M., Locatelli, A., and Martello, S. Knap- sack problems-an an overview of recent advances. part ii: Multiple, multidimensional, and quadratic knapsack prob- lems. Computers Operations Research, 143:105693, 2022.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 24Xhv:2444.04475 2024.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model aligmment prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Ceravolo, P., Mohammadi, F., and Tamborini, M. A. Ac- tive learning methodology in Ilms fine-tuning. In 2024 IEEE International Conference Cyber Security and Resilience (CSR), pp. 743-749. IEEE, 2024.
Feldman, D. Core-sets: Updated survey, Sampling tech- niques for supervised or unsupervised tasks, pp. 23-44, 2020.
Chen, H., He, G., Yuan, L., Cui, G., Su, H., and Zhu, Noise contrastive alignment of language models with ex- plicit rewards. arXiv preprint 200.00940,. 2024a.
Feldman, D., Schmidt, M., and Sohler, C. Turning big data into tiny data: Constant-size coresets for k-means, pea, and projective clustering. SIAM Journal on Computing, 49(3):601 2020.
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. Big self-supervised models are strong
10

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Gupta, A. and Tangwongsan, K. Simpler analyses of local search algorithms for facility location. arXiv preprint arXiv:0809.255% 2008.
R., et al. Training language models self-correct via reinforcement learning. arXiv preprint arXiv:2409./2917 2024.
Gupta, T., Madhavan, R., Zhang, X., Bansal, C., and Rajmohan, S. Refa: Reference free alignment for multi-preference optimization. arXiv preprint arXiv:2412.16377 2024a.
Liu, A., Bai, H., Lu, Z., Sun, Y.. Kong, X., Wang, S., Shan, J., Jose, A. M., Liu, X., Wen, L., et al. Tis- dpo: Token-level importance sampling for direct prefer- ence optimization with estimated weights. arXiv preprint arXiv:2410.04350, 2024a.
Gupta, T., Madhavan, R., Zhang, X., Bansal, C., and Rajmo- han, S. Swepo: Simultaneous weighted preference opti- mization for group contrastive alignment. arXiv preprint arXiv:2412.04628. 2024b.
Liu, C. Y., Zeng, I.., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint 2024b.
arXiv:2410.
Har-Peled, S. and Mazumdar, S. On coresets for k-means and k-median clustering. In Proceedings of the thirty- sixth annual ACM symposium on Theory of computing, pp. 291-300, 2004.
Liu, J., Zhou, Z., Liu, J., Bu, X., Yang. C. Zhong, H.-S., and Ouyang, W. Iterative length-regularized direct preference optimization: A case study on improving 7b language models to gpt-4 level. arXiv preprint arXiv:2406. 1 2024c.
Hartigan, J. A. and Wong, M. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statis- tical society. series (applied statistics), 28(1):100-108, 1979.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sam- pling improves preference optimization. arXiv preprint arXiv:2309 2.Xx:2339006667,, 2023.
Hong, J., Lee, N., and Thore,, J. ORPO: Monolithic pref- erence optimization without reference model. ArXiv, abs/2403.07691, 2024a.
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., and Tang, Self-supervised learning: Generative or contrastive. IEEE transactions on knowledge and data engineering. 35(1):857-876, 2021.
Hong, J., Lee, N., and Thorne, J. Orpo: Monolithic prefer- ence optimization without reference model. In Proceed- ings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 11 11170-11189) 2024b.
Long, D. X., Ngoc, H. N., Sim, T., Dao, H., Joty, S.. Kawaguchi, K., Chen, N. F., and Kan, M.-Y. Lims biased towards output formats! systematically evaluating and mitigating output format bias of lims. arXiv preprint arXiv:
Huang, L., Jiang. S., and Vishnoi, N. Corescts for clustering with fairness constraints. Advances in neural information processing systems, 32, 2019.
2024.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint WrXi:24405.14734 2024.
Kellerer, H., Pferschy. U. Pisinger, D., Kellerer, H., Pferschy, U., and Pisinger, D. Introduction to np- completeness of knapsack problems. Knapsack problems, pp. 493-493, 2004a.
Oh Song, H., Jegelka, S., Rathod, V., and Murphy, K. Deep metric learning via facility location. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5382-5390, 2017.
Kellerer, H., Pferschy, U., Pisinger, D., Kellerer, H., Pfer- schy. U., and Pisinger, D. Multidimensional knapsack problems. Springer, 2004b.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Kim, D., Kim, Y., Song, W., Kim, H., Kim, Y., Kim, S., and Park, C. sdpo: Don't use your data all at once. arXiv preprint arXiv:2403. 0222400,.977..,. 19270, 2024.
Korbak, T. Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. Pretrain- ing language models with human preferences. In Inter- national Conference on Machine Learning, pp. 17506- 17533. PMLR, 2023.
Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization.
arXiv preprint arXiv:2404.19773. 2024.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. arXiv preprint arXiv:2403. 19159, 2024.
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs,

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Qi. B., Li, P., Li, F., Gao, J., Zhang. K., and Zhou, B. Online dpo: Online direct preference optimization with fast-slow chasing, arXiv preprint arXiv:2406.00554, 2024.
Xiao, R., Dong, Y., Zhao, J., Wu, R., Lin, M., Chen, G., and Wang, H. Freeal: Towards human-free active learn- ing in the era of large language models. arXiv preprint arXiv:2311.1566 2023.
Rafailov, R., Sharma, A., Mitchell, E.,, Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Durme, B. V., Murray, K., and Kim, Y.J. Contrastive preference optimization: Pushing the boundaries of LLM perfor- mance in machine translation. ArXiv, abs/2401.08411 2024.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
Yu, Y., Zhuang, Y., Zhang, J., Meng, Y., Ratner, A. J., Kr- ishna, R., Shen, J., and Zhang, C. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06444 2017.
Yuan, W., Kulikov, I., Yu, P., Cho, K., Sukhbaatar, S., We- ston, J., and Xu, J. Following length constraints in in- structions. arXiv preprint arXiv:2406. 177444 2024.
Sener, O. and Savarese, S. Active learning for convolutional neural networks: core-set approach. arXiv preprint arXiv: VrXiv:/708.00m 1708.00089, 00489, 2017.
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language mod- els with human feedback without tears. arXiv preprint arXiv:2304 05302, 2023.
Settles, B. Active learning literature survey, 2009.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I.,
Zhang, Y., Feng, S.,. and Tan, C. Active example selection for in-context learning. arXiv preprint arXiv:2211.04486, 2022.
Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. SLiC-HF: Sequence likelihood calibration with human feedback. ArXiv, abs/2305. 10425, 2023.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z.,, Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46622, 2023.
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial In- lelligence, volume 38, pp. 18990-18998, 2024.
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. Á., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749. 2024.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprin! 1rXiv:190.085550 2019.
Tran, H., Glaze, C., and Hancock, B. Iterative dpo alignment. Technical report, Technical report, Snorkel AI, 2023.
Wu, Y., Sun, Z. Yuan, H., Ji, K., Yang. Y., and Gu, Q Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024.
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P.. Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training. Advances in Neural Information Process- ing Systems, 009990000333,,,,-111119113333331 2023.
12

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
SUPPLEMENTARY MATERIALS
These supplementary materials provide additional details, derivations, and experimental results for our paper. The appendix organized as follows:
Section A provides a more comprehensive overview of the related literature.
Section B provides additional experiments supplement the experiments provided in the main part of the paper.
Section C provides theoretical analysis of the equivalence of the optimal selection integer program and the reward maximization objective.
Section D shows a constant factor approximation for the coordinate descent algorithm in polynomial time.
Section E provides theoretical guarantees for our k-means style coreset selection algorithm.
Section F provides the code for computation of the optimal selection algorithm.
Section G provides t-sne plots for the various queries highlighting the performance of our algorithms.
A. Related Work
We start this survey with a high-level overview of the broader Reinforcement Learning from Human Feedback (RLHF) literature, then deep dive into preference optimization and multi-preference optimization, and finally discuss active learning and subset selection techniques relevant to our work.
Reinforcement Learning from Human Feedback (RLHF) has emerged as robust alignment paradigm for language models. Early methods, such as Trust Region Policy Optimization (TRPO) (Schulman et al., 2015) and Proximal Policy Optimization (PPO) (Schulman et al., 2017), extend direct RL methods by constraining policy updates for stability. PPO, in particular, has been successfully applied to RLHF, allowing LLMs to produce outputs aligned with human preferences (Ziegler et al., 2019; Ouyang et al., 2022). However, the complexity of training separate reward models and the potential instability direct RL prompted simpler approaches.
Preference Optimization in RLHE
Direct Preference Optimization (DPO) (Rafailov et al., 2024) simplifies LLM alignment by optimizing a contrastive loss directly over paired preference data, bypassing the intermediate reward modeling step. This makes DPO computationally efficient and suitable for limited preference datasets. A wide array of DPO extensions and alternative preference optimization methods have since emerged. These include variants like Identity Preference Optimization (IPO) (Azar et al., 2024), self-play preference optimization (Wu et al., 2024), preference ranking optimization (Song et al., 2024), rejection sampling optimization (Liu et al., 2023), and generalized preference optimization (Tang et al., 2024). Many of these methods also address common DPO limitations, such as need for fixed reference model, which adds complexity. Works like RRHF (Yuan et al., 2023) and SLiC-HF (Zhao al., 2023) propose rank-based loss techniques. KTO (Ethayarajh et al., 2024) framework inspired by prospect theory that directly learns desirability, while RAFT (Dong al., 2023) introduces list-wise finetuning approach. Other notable methods include SPIN (Chen al., 2024b), which treats the model part of an adversarial game, CPO (Xu et al., 2024), which reworks the DPO objective, and ORPO (Hong et al., 2024b), which unifies SFT and preference training. SimPO (Meng et al., 2024) removes the reference model and incorporates length normalization to mitigate verbosity issues. Further variants like R-DPO (Park et al., 2024), LD-DPO (Liu et al., 2024c), sDPO (Kim et al., 2024), IRPO (Pang et al., 2024), OFS-DPO DPO (Qi et al., 2024), and LIFT-DPO (Yuan et al., 2024) address specific challenges like length bias, reasoning chains, and training stability.
13

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Multi-Preeerence Optimization. Traditional preference optimization methods primarily rely on pairwise comparisons. However, the advent of richer datasets, such as UltraFeebbcck (Cui et 2023), which provide multiple graded responses per query, highlights the necessity of multt-preference optimization. These methods move beyond simple binary preferences by leveraging all available positive and negative responses simultaneously, leading to more nuanced feedback signals (Rafailov et al., 2024; Cui et al., 2023; Chen et al., 2024a). Multi-preference objectives can reduce alignment bias and better approximate the true preference distribution by incorporating the diversity of acceptable and suboptimal responses. Examples include InfoNCA (Chen et al., 2024a), which utilizes a noise-contrastive objective based on scalar rewards. MPO (Gupta et al., 2024b), builds upon this by introducing deviattio-based weighting, giving stronger influence to responses that deviate significantly (positively or negatively) from the average quality. Refa (Gupta al., 2024a) fixes some of the common issues with MPO relating to multi-preference optimization including length bias as well as fixed reference. We build upon this framework to work on the problem of response selection in multi multi-preference optimization. Here, we emphasize highly informative examples while mitigating the overemphasis on less informative negative samples, common challenge in these contrastive methods.
On-Policy Self-Play. A key advancement in reinforcement learning that directly impacts LLM alignment is self-play or on- on-policy data generation. In this paradigm, the model continuously updates its policy and re-generates data from its evolving distribution (Silver al., 2016; 2017). This ensures that the training set remains aligned with the model's 's current behavior (Christiano al., 2017; Wu et al., 2023; 2024), accelerating convergence and maintaining data relevance. However, this dynamic generation process can significantly inflate the number of candidate responses per query, thereby motivating the need for selective down-sampling of training examples to manage computational load.
Active Learning for Policy Optimization. The notion of selectively querying the most informative examples is central to active learning (Cohn et al., 1996; Settles, 2009), which aims to reduce labeling effort by focusing on high-utility samples. Several works incorporate active learning ideas into reinforcement learning, e.g., uncertainty sampling or diversity- selection (Sener & Savarese, 2017; Zhang et al., 2022). In the RLHF setting, Christiano et al. (2017) highlight how strategic feedback can accelerate policy improvements, while others apply active subroutines to refine reward models (Wu et al., 2023). By picking small yet diverse set of responses, we avoid both computational blow-ups and redundant training signals.
Links with Classical Problems. Our work draws heavily from classic problems in machine learning and combinatorial optimization related to selecting representative subsets. Clustering techniques such as K-means and K-medoids (Hartigan & Wong, 1979) are used to group points and ensure coverage over semantically distinct modes in the embedding space (Har-Peled & Mazumdar, 2004; Cohen-Addad et al., 2022). These methods connect to the facility location problem (Oh Song et al., 2017), which seeks to minimize the cost of "covering" all points with a fixed number of centers, often addressed via coreset construction (Feldman, 2020). Furthermore, when selecting a subset of size K to cover or suppress "bad" outputs, the objective can be framed as a min-knapsack or combinatorial optimization problem (Kellerer et al., 2004a). Such formulations often involve integer programs (Chen et al., 2020), for which approximate solutions can achieve strong empirical results in high- nigh-dimensional scenarios (Cohen-Addad et al., 2022; Har-Peled & Mazumdar, 2004). Our method frames the selection of negative samples in a Lipschitz coverage sense, thereby enabling both theoretical guarantees and practical efficiency in multi multi-prefeeence alignment.
Collectively, our work stands at the intersection of multi-preference alignment (Gupta et al., 2024a; Cui et al., 2023), on-policy data generation (Silver al., 2017; Ouyang et al., 2022), and active learning (Cohn al., 1996; Settles, 2009). We leverage ideas from clustering (k-means, k-medoids) and combinatorial optimization (facility location, -naasakkps)kk (Kellerer et al., 2004b; Cacchiani et al., 2022) to construct small yet powerful training subsets that capture both reward extremes and semantic diversity. The result is an efficient pipeline for aligning LLMs via multi preff-ereeerence signals without exhaustively processing all generated responses.
14

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
B. Additional Experiments
AlpacaEval 2
Arena-Hard
MT-Bench
Method
Reward Model
LC (%)
WR (%)
WR (%)
GPT-4
AMPo-Opt-Select-fy (Ivsk)
Skywork-Rewand Rewardd--
51.6
51.2
46.4
8.11
AMPO-Opt-Select-ff (Ivsk)
Skywork-Rewarde
52.16
51.58
45.4
8.07
Table Comparison of AMPO- AMPO-Opt-Seed Opt-Select variants with and without é based selection on AlpacaEval, Arena-Hard, and MT-Bencch The & variant improves LC and WR on AlpacaEval, while slightly underperforning on Arena-Hare and MT-Bench. Best results are in bold.
Reward Models as Classifiers vs. Regressors To further analyze how reward scores influence alignment, we compare two approaches forming preference pairs and computing loss in SaaPo-Opt-Seleee to Ollassfiicttion-stlle and & (magnitude-aware).
For both settings, we generate a fixed number of responses per prompt. The response with the highest reward is placed in the positive set y while contrastive negative subset is selected via Opt-Select.
to (Uniform Preference Weighting): Each preference pair contributes equally to the loss, regardless of the magnitude of reward difference. This reflects a pure classifier-style view of the reward model: only the relative ordering matters, not the exact values.
( (Reward Sei-WWegt Preference): Each preference pair is weighted by the absolute deviation of the rejected response's s reward from the mean reward value, i.e., = |reward; - reward]. This encourages the model to prioritize learning from examples with larger reward separation, treating them as more informative for optimization.
Table 5 presents the results. While & improves AlpacaEval metrics, to performs better on Arena-Hard and MT-Bench, which are known to be noisier and more ambiguous in reward calibration. These findings reinforce the hypothesis that reward magnitude may not always reflect true quality, especially when misaligned with task-specific evaluation criteria.
Key Finding: While weighting by reward magnitude can improve alignment on clean datasets, uniform weighting with classifier-style preferences (Ey) offers better robustness across varied and noisy evaluation settings supporting recent trends advocating for classification-basee use of reward models.
Why 1-vs-k Preference Selection is Superior k-vs-k We ablate between 1-vs-k and k-vs-k preference construction strategies in AMPO, where the number of total responses is held fixed. In the 1-vs-k setting, we select the single highest- scoring response as the positive and sample h diverse negatives using AMPO strategies. In contrast, the k-vs-k setup selects multiple top-scoring responses and treats them equally as positives, paired against k negatives.
AlpacaEval
Arena-Hard
MT-Bench
Method
Reward Model
LC (%)
WR
WR (%)
GPT-4
AMPO-BAL -PPO-Botomk (1vs7)
Skywork Skywork-Rewand-IAN M Lewaadk-Reward-LM
50.8
50.5
45,2
8.11
AMPO-Bottomk (4vs4)
45.44
51.25
42.2
7.77
AMPO-CCresee (1vs7)
Skywork-Rewand
52.4
47.8
8.12
AMPO-Coreset (4vs4)
Skywork-Reward Reward-LM
46.61
51.4
46.3
7.67
AMPO-Opi-Sae SpPO-ett-Sellcttt (1vs7)
Skywork-Rewand-
51.6
$1.2
46.4
8.11
AMPO-Opt-Select (4vs4)
47.16
52.5
44.9
7.72
Table 6. Dynamic AMPO-Based Top/Bottom Response Selection Across Evaluation Benchmarks for Llama-3-Instruct (8B)
Theoretical Motivation If the goal is to maximize expected reward, the optimal strategy-wher only the sampling probabilities over responses can be controlled-is to assign the highest probability to the response with the maximum reward score. This ensures reward- weight- sampling favors the best response. Including multiple responses in the positive set can dilute this probability mass and introduce ambiguity, especially when the difference between top-ranked responses is small or noisy.
15

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
This analysis is formalized in Section B.1 of the paper, where show that concentrating probability the single best response, while distributing mass away from contrastive negatives, is provably optimal in expectation under fixed response budget.
AlpacaEval
Arena-Hard
MT-Bench
Method
Reward Model
LC (%)
WR (%)
WR (%)
GPT-
AMPO-Opt-Select (1vs3)
49.6
48.5
46.1
8.03
AMPO--Opt-Selctt (2vs2)
Skywork-Reward-LLN
48.64
47.85
42.1
7.87
Table Dynamic AMPO-Based Top/Bottom Response Selection with Ekywork-Reward-H Across Evaluation Benchmarks for Llama-3- Instruct (8B).
Empirical Evidence We validate this hypothesis with empirical results presented in Table 6 and Table 7. Across Bottom-k, Coreset, and Opt-Select variants, 1-vs-7 configuration consistently outperforms 4-vs-4, particularly in terms of LC win rate (AlpacaEval2), Arena-Hard, and MT-Bench. Similarly, 1-vs-3 outperforms 2-vs-2.
This suggests that: Selecting single clear positive introduces less ambiguity. Including multiple positives inject noise if some "positive" responses marginal or inconsistent. A broader negative set (k larger) allows for better contrast and generalization.
The 1-vs-k preference setup is theoretically optimal for maximizing expected reward and empirically leads to better performance. This supports our design choice of using single, high-confidence positive response when constructing preference data for alignment.
Key Finding:
C. Extended Theoretical Analysis of OPT-SELECT
In this appendix, we present more detailed theoretical treatment of AMPO-OPTSELECT. We restate the core problem setup and assumptions, then provide rigorous proofs of our main results. Our exposition here augments the concise version from the main text.
C.1. Problem Setup
Consider single prompt (query) æ for which we have sampled N candidate responses { {11, V2, .... BN Each response y: has:
A scalar reward E [0, 1].
An embedding e, e Rdd
We define the distance between two responses y/ and Yj by
Aij = |e; -esl-
(12)
Throughout we rescale the embedding so that maxi.j Aig = the Lipschitz constant L E (0, 1] then compares quantities of the same scale.
We wish leam policy (p:), where p: and N Pi 1. The policy's expected reward is
ER(p)
Σ riP
(13)
Positive and Negative Responses. We designate exactly one response, denoted Visop as positive (the highest-reward
candidate). All other responses are potential "negatives." Concretely:
We fix one index sop with dtop arg max Ti- JE(L....N)
We choose a subset S (L...., \ (doop} size K, whose elements are forced to have pj = 0. (These are the "negatives.")
16

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Tre-bnea,,ng
If several responses attain the maximal reward we keep the with the smallest index; thus hop is unique.
C.1.1. LIPSCHITZ SUPPRESSION CONSTRAINT
We assume mild Lipschitz-like rule:
(A1) L-Lipschitz Constraint. pj for some j S, then for every response yi, must have
pi LA L l|ee ejll.
(14)
The effect that whenever we force a particular negative to have p; 0, any response near j in embedding space also gets pushed down, since p; I L A.,. By selecting a of K negatives covering many "bad" low-reward regions, we curb the policy's s probability of generating undesirable responses.
Goal. Define the feasible set of distributions:
F(S) {pst: PJ = =00- E S min Ai, di (oop) US}.
(15)
JES
Feasibility condition.
For a given S the constraint set F(S) is non-empty iff
min
Ar,j S 1/L. JES
S
Hence assume K and L are chosen that the above inequality holds for at least subset of size K.
We then have two-level problem:
N
max 8 SC NW (top) |S|=
PiPi.
(EF))
E.P:=
subject
Piup unconstrained (no Lipschitz bound).
(16)
We seek S that maximizes the best possible Lipschitz-compliant expected reward.
C.2. Coverage View and the MIP Formulation
Coverage Cost. To highlight the crucial role of "covering" low-reward responses, define weight
W; == Tmax Ti
(17)
where 7max = max, Tj. Then a natural coverage cost is
N Σ
Cost(S)
min Ai.j JES
(18)
small minjes Ai,j means response is "close" to at least one negative center j. If is low, then w; is large, so we put higher penalty on leaving uncovered. Minimizing Cost(S) ensures that important (low-reward) responses are forced near penalized centers, thus suppressing them in the policy distribution.
MIP for Coverage Minimization. We can write mixed-integer program:
17

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Problem P: min
W. Vi+
(0.1)
(0,1)
==
9:20
N Σ j=1
2j K,
subject to
Zi,j S Izj. Zj=1 Zig =1 Vi,
(19)
1/ A.j + M (1 - -i,s);
Vi 2 A.j - M (1 - -iz)); Vi.h.
where M max;, Aig- Intuitively, each 2 indicates if is chosen negative; each 2 indicates whether is "assigned" to j. At optimality, Y: minyes Ai,j, the objective Σ, W/ Y: is precisely Cost(S). Hence solving P yields S that minimizes coverage cost equation 18.
Lemma C.1 (Coverage cost controls negative reward). Under (A1)-(33), suppose of size K satisfies the feasibility condition, ie. there exists (p:) E F(S) with E, 1. Then for every normalized {p,} (L.e. W(p,} E F(S)), huve:
N
N
mmax ri) Pi S
(mma - r2) min Ar, = I. C.Cos(S): JES
Consequently,
max (N: (FF()
Σ
TiPi = Tmax
min (.i)CF(()
Σ
(Fimx - rs)pp
is maximised exactly when Cost(S) is minimised. Furthermore, this bound is tight: one can set
Ps = min Aiii (i (ipp)
Piuo =1 Σ
min Aigs jES
JES
ipftop
which is feasible by assumption, and gives (r'mx - r:)p: L Cost((()), so max E, ripsi = Tmax L Cost(S).
Proof. By (A1), any & S U {oppp satisfies p; < L minjes Aij. hence (Tmax r:)p; S L (Tmax r.)) minjes A.g. Summing over yields the claimed bound, and the equivalence between minimising Cost(S) and maximising Σ, rip, follows by writing
Σ rape = Tmax Σ Pi Σ (rmax - rs)p; = -max Σ (Tmax ri)ps.
and observing the inequality becomes equality for the choice above.
C.3. Main Theorem: Optimality of P for Lipschitz Alignment
Theorem C.2 (Optimal Negative Set via P). Let S* be the solution the MIP P in equation 19, L.e. it minimizes Cost(S). Then S* also maximizes the objective equation 16. Consequently, picking S* and allowing any distribution Atop arg max; r; yields the optimal Lipschitz-complinng policy.
Proof. By construction, solving P returns S* with Cost(S")
min Cost(S). By Lemma C.I. minimising Cost(S)
is|=K
indeed maximises the feasible expected reward, so such an S* simultancously maximizes the best possible feasible expected reward. Hence S" is precisely the negative set that achieves the maximum of equation 16.
18

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Interpretation. Under a mild Lipschitz assumption in embedding space, penalizing (assigning zero probability to) small set S and forcing all items near S to have small probability is equivalent coverage problem. Solving (or approximating) P selects negatives that push down low-reward modes effectively as possible.
C.4. Discussion and Practical Implementation
OPT-SELECT thus emerges from optimizing coverage:
1. Solve approximate the MIP P find the best subset S S {I....,,N) NN \ {dop))
2. Force Pj 0 for each S; retain soo with full probability (Piop 2 1), subject to normalizing the distribution.
In practice, local search or approximate clustering based approaches (e.g. Weighted K-Medoids) can find good solutions without exhaustively solving P. The method ensures that near any chosen negative j. all semantically similar responses i have bounded probability ps L A.j. Consequently, OPT-SELECT simultaneously covers and suppresses undesired modes while preserving at least one high-reward response unpenalized.
Additional Remarks.
The single-positive assumption reflects practical design where one high-reward response is explicitly promoted. This be extended to multiple positives, e.g. top K* responses each unconstrained.
For large N, the exact MIP solution may be expensive; local search (see Appendix D) still achieves constant-facte approximation.
The embedding- Lipschitz constant rarely known exactly; however, the coverage perspective remains valid for "sufficiently smooth" reward behaviors the embedding space.
Overall, these results solidify OPT-SELECT as a principled framework for negative selection under Lipschitz- based alignment objectives.
19

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
D. Local Search Guarantees for Weighted K-Medoids and Lipschitz-Reward Approximation
In this appendix, we show Theorem D.I that a standard local search algorithm for Weighted K-Medoids achieves a constant-factor approximation in polynomial time.
D.I. Weighted K-Medoids Setup
We are given:
A set of N points, each indexed by [I....,NN.
A distance function d(i,j) 2 0, which forms metric: d(i,j) +(k.)) 0. d(i.)) = d(j.t).
A nonnegative weight wy for each point
A budget K, 1 S K KSN.
We wish pick subset S S [1....,,N] N) of medoids (centers) with size |S| K that minimizes the objective
Cost(S) Σ
wi min d(i,)).
(20)
JES
We call this the Weighted K-Medoids problem. Note that medoids must come from among the data points, as opposed to K K-median or K K-means where centers can be arbitrary points in the metric or vector space. Our Algorithm reduces to exactly this problem.
D.2. Coordinate Descent Algorithm via Local Search
Our approach to the AP-hardness of Algorithm was to recast it as a simpler coordinate descent algorithm in Algorithm 4, wherein we do a local search at every point towards achieving the optimal solution. Let COST(S) be as in equation 20,
1. Initialize: pick any subset S C {1,.... N) of size K (e.g. random greedy).
2. Repeat: Try all possible single swaps of the form
S' = (5\( (í}.
where j E S and & S.
3. If any swap improves cost: i.e. CSs < Cost(S), then set S <- and continue.
4. Else terminate: no single swap can further reduce cost.
When the algorithm stops, we say S is local optimum under 1-swaps.
D.3. Constant-Factor Approximation in Polynomial Time
We now present and prove result: such local search yields a constant-factor approximation. Below, we prove a version with a factor 5 guarantee for Weighted K-Medoids. Tighter analyses can improve constants, but 5 is commonly cited bound for this simple variant.
Theorem D.1 (Local Search for Weighted K- K-Medoids). Let S be an optimal subset of medoids of size K. Let S be any local optimum obtained by the above 1-swap local search. Then
Cost (S) 5 Cost(S*).
(21)
Moreover, the procedure runs in polynomial time (at most ((x)) "worse-case" swaps in principle, but in practice each improving swap decreases cost by a non-negligible amount, thus bounding the iteration count).
20

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Remark D.2. We follow the result from Arya et al. (2001) who define the locality gap of the single-swap local-search procedure the worst-case ratio between the cost of any local optimum and the global optimum. They prove that for the metric K-median problem, this gap is exactly 5. More precisely, permitting only one swap per step guarantees
CS - 5 C
(22)
for every local optimum S and global optimum S*
Sketch of Arya et al. (2001)'s Analysis. They partition the data according to the Voronoi cells of the global optimum, then show via "coupling" argument (together with repeated improve the solution, the total service cost from each cell is bounded by five times its optimal cost.
bounds) that whenever local swap cannot
Proof. Notation.
Let S be the final local optimum of size K.
Let S* be an optimal set of size K.
For each point i, define
rj = di, = min d(t,
rf = min j). jES
jES
Thus Cost(S)= = E, W4 and Cost(S")) = Es W; r.-
Let c(S) E; wi S) as shorthand for Cost(S).
Step Construct "Combined" Set. Consider
st = S S..
S
We have [St] $ 2K. Let c(t) = E, w. d(i,St).
Observe that
diis!) min( di, dii,))) min(rs, r
Hence
((S) = Σ ws min ri, r.).
We will relate c(ST) to c(S) and c(S*).
Step 2: Partition Points According to St. For each j E S" define the cluster
c(j*) {i j arg min d(i,j")). j'ES-
Hence (C(j*) f E S*} a partition of (1,...,N.. We now group the cost contributions by these clusters.
Goal: Existence of' a Good Swap. We will assume c(S) > 3c((*) and derive a contradiction by producing a profitable swap that local search should have found.
Specifically, show that there must be center S* whose cluster C(j*) is "costly enough" under S, so that swapping out some center j S for j* significantly reduces cost. But since & was local optimum, no such profitable swap could exist. This contradiction implies c(S) < (c(S)).
Step 3: Detailed Bounding.
We have
N r
N <
c(S') Σ
minfra
Σ im
w; r = c(S*).
21

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Similarly,
((S) - ==1
Wi r; = ds).
Hence c(St) I min((((S), c(S*)}. Now define
N D
N Σ
Σ
wi [r
min ra r7]]
ws (r; --)
where (x)+ max(æ, 0}. By rearranging,
W6 ri
wy min Th r.] D.
Thus
(5) - c(s)) = D 2 c()) - c(S))
So
D 2 (8) (8*).
Under the assumption c(S) 5c(8*), we get
D > 4c(S)).
Step 4: Find a Center j' with Large D Contribution. We now "distribute" D over clusters C(j*). ). Let
Dj- Σ (r: - -r7)
(ECU*)
Then = Σ e*
Dj.. Since D 4c(S*), least one j S* satisfies
Dj. 4 c(S*) c(S*)
4
(23)
8-1 K
because |8*| K. Denote this center Hange and its cluster C* C (Jürge):
Step 5: Swapping,"" j into S. Consider the swap
Sawap =
S f jout
U
Junnee
where jout is whichever center in S we choose to remove. We must show that for an appropriate choice of jout. the cost ((Sawap) is at least (r; rz)+ smaller on average for the points in C*,, forcing a net cost reduction large enough to offset any potential cost increase for points outside C*
In detail, partition S into K clusters under Voronoi assignment:
26) = {i:j= arg min d(i,x)), j JES.
#5$
1
Since |S] = K, there must exist at least one jout E S whose cluster ((oun) has weight
Σ (E0(
wi
K
We
i=1
remove that jout and add Farge
22

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
Step 6: Net Cost Change Analysis. After the swap, the change in cost is A Ain + Dout. The "in-gain" for points
i C C(iarge) is bounded by:
in Σ IEC
wi (d(ii, (smap) - d(i,)) S we (r: rf)+ -Djtarge
Σ
(24)
IEC
The "out-loss," Dout represents the potential cost increase for points not in C*, primarily those that were served by the removed center Jout. Bounding this term is the most complex part of the proof.
Remark D.3 (Bounding Sout). The analysis in Arya et (2001) uses a series of clever applications of the triangle inequality show that the cost increase, Dutt bounded relative the cost of the optimal solution. A simplified (though non-trivial) result of this bounding shows that for appropriately chosen Jout, the increase can be bounded such that:
c(S*) K
A
(25)
This bound is sufficient to complete the proof. We defer the detailed derivation of this specific bound to the original literature and proceed with this result.
Step 7: Arriving at a contradiction. Combining our bounds, the total change in cost is:
o(S))
c(SSwap) - -c()) = -in + Dout I Diiar K
c(8") From Step (Eq. 23), we know Harge > 4 K
SSbbsituting this in gives:
((8")
c(S*)
c(S*)
cl - <
< 0.
K
K
K
This shows strict decrease in cost, which contradicts the local optimality of Š. Therefore, our initial assumption must be false, and conclude that c(5) ((S)).
Time Complexity. At each iteration we try all O(K N) possible 1-swaps. By maintaining for each point i its distance to the nearest center in S, we can update the total cost in ((N) time per swap check; hence each pass costs O(K N Moreover, letting
N
Wiot =
Σ
Wis
D Dmax = max d(i,)).
[.]
1=1
have
0 < c(S) S
Wiot Dinx-
Since all weights and distances from the finite input, there is minimum positive gap 5 > 0 between any two distinct cost values. Therefore each improving swap decreases c(S) by at least 6, so there can be at most
W tot Dmax
5
such swaps. Altogether the algorithm performs
O(NN) x of Win Dmax
poly(input size)
total work, i.e. it runs in polynomial time.
Remark D.4 (Improved Constants). A more intricate analysis can tighten the factor 5 in Theorem D.1 to or 4. See, e.g., (Gupta& & Tangwongsan, 2008; Arya et al., 2001) for classical refinements. The simpler argument here suffices to establish the main principles.
23

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
E. Theoretical Guarantee for AMPO-Coreset
This appendix provides the theoretical motivation for AMPO-CORESET selection strategy. We first introduce concept of coreset and then present formal theorem showing that, under certain clustering assumptions, this strategy yields policy with a guaranteed additive bound on its expected reward.
Coresets for Representative Selection. The term coresel originates in computational geometry and machine learning, referring to small, weighted subset of data that approximates the entire dataset with respect to particular objective loss function (Bachem et al., 2017; Feldman et al., 2020). In the context of AMPO-CORESET the K-means clustering subroutine identifies representative embedding-spate regions. By choosing single worst-rated example from each region, we mimic a coreset-based selection principle: our selected negatives approximate the distributional diversity of the entire batch responses. This ensures the model receives penalizing signals for all major modes of undesired behavior, mitigating the risk of ignoring infrequent but problematic minority clusters.
E.1. Additive Guarantee under Counded-Diameen Clustering
Recall from Appendix C that we use normalized weights
W
mmax ry),
W0;
Tmax Ti W
so that Σ W3 = =1 1.
This allows Lemma C.I (Coverage-cond controls negative reward) to give the tight bound max Errpp rip: Tmax L (S) We now show that under the clustering assumption of AMPO-Coreset, the cost term is bounded by dmax- Theorem E.1 (Additive Ldmax Guarantee for Coreset Selection). Suppose the N candidate responses can be partitioned into K clusters (C1,.... Ck} in embedding space, each of diameter at most dmaax
l|e -erl S dmax (j=1.... K).
ECEEC
Let the negative set be formed by picking one arbitrary index i, C.y from each cluster Cj. Then, the maximum expected reward achievable by Lipschitz-complion policy using this negative set S is bounded by:
max Py=0 (YEES) P.S. minnes |e--a|l
Σ TiPi 2
Tmax
L dmax:
where Tmax = max; r,, This guarantees the expected reward is within an additive error of Ldmax of the highest possible reward.
Proof. We use the result from Lemma C.I, which states that the maximum expected reward is Timxx L Cost where cost function uses normalized weights wi = (Tmax rs)/W. We need to bound Cost(S) for our chosen S.
N
Cost(S) =
Σ
w; min le, - -erl--
LES
1=1
For any point ys it belongs to cluster Cj. By construction, the set of negatives S contains the point i7 from that same cluster. Therefore, the distance from y, its closest negative in S is most its distance to V:7 which is bounded by the cluster diameter:
min IES
|c: cil < ||c,
o S dmaxx ViE
Since this holds for all points i, we can bound the cost:
N
Cost(S) =
Σ
wi
min
le - eill <
Σ wi dmax = dmax
Wi
d
IES
Sdmax
24

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
where the final step uses the fact that the normalized weights the maximum expected reward gives:
Substituting this bound back into the expression for
max
Ti Pi
Tmax
L Cost(S)
2
Tmax
L dmax
which completes the proof.
Remark Seeeributioo-Depeendee Guarantee). The above theorem provides a deterministic guarantee for a fixed set of N points. In practice, we learn from finite sample of responses drawn from an unknown underlying distribution D. If we learn K clusters from sufficiently large i.i.d. sample responses, standard uniform convergence arguments (see, e.g., (Bachem et al., 2017)) show that these empirical clusters will, with high probability, also cover new responses drawn from D. Consequently, for high fraction of new queries, the policy derived from the coreset selection strategy is expected to achieve near-optimal reward, with an additive error similar to the Ldmax bound.
F. Optimal Selection Code
In this section we provide actual code used to compute the optimal selection.
import numpy as np
from scipy- pppptyyssaaaaa aiiia.ddistttt distance import cdist
def solve_ Leraa__searhhh_ee min nosmmlierdll vectors: np. adardar,,y, rating: np.ndarray, k; int,
max_iter: int 100, random_seed:
42
):
Normalize ratings
rating_min np. min (rating) rating_max np. (rating)
rating_nermal ized (rating rating_min) (rating_max rating_nin if rating_max
zating_min else no.zeros__ikke (rating) 0.5
Identify top-rated point excluded_top_inder int (np. awwpparggaxiioooo (rating_noomm izd))
Reduce dataset
sew_to_cld |idx for idx in range (len (ennnnrainng) (em(taating_mmmmlled)) if idx != excludes__topiindex
vectors_reduced np. delete (vectors, excluded_top_indess axis-0)
rating_reduced np. delete (ratiin_normmaizee, cxcluded__op_inddo
Compute L2 distances and normalize 0:
if len (erating_reduuseeee return excluded ptcpuueed_oop_ieeeee___e_ index, None, [I, [], []
distance matrix = edist recctors_reduced, cectors_reduced_ metric= fucii=ceeelldeaar distanceematrile /= cestance__atrix..oe
if distance_mat matrix. max > 1e-12 else
Compute weights
mean_ratinggreduie = mean (rating_reduced)
w = np. ep.exp (mean (mean_rating reduced ratingg reduced)
Local search setup def compute_objectt jocttteeejeeeeeeeeeeeeoeee (chosen_set))
return sum(w) sum(w[i] min (anddaaannc_mmariiil. il for j in chosen_sct) for 1 in range len (w)
rng np. ra.dem dewault_rngnggggoooee (randor_seed) indices np.arange (len (rading_reduced]
25

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
current_set set ( choice (all_indices, size-k, replace-False)) if len reting_reduced) else set (all_indices) current_cost compute_objectivee (current_sst
Local search loop improved = True while improved:
improved = False
best_swap (None, None, 0)
for oou (strreett___s for i_in in all_indices: if i_in not in current_set:
candidate_set (current_set (i_out]) (i_in)
improvement current_com
compute compute_objectivee (canddiiee set)
(candidate
if improvement best swap_sswpl221
best_swap swap (i_uut, i in, improvement)
if best_swap [2] > 1e-12:
current_set, remove (best_swap(0))
current_set.cem (best_swap
current_cost best_swap [2] improved = True
inreen_indices_rriiiiee Burrent_set]_
[new_to_old[i] for in (curreent (new_to_oldij] for j sseted (all_indices)
return excluded_top_indee chosen_indices__rrggheee [0], rejected_ ind cndiccssri [:k],
chosen_indiccs__rriggiii rejectes_indicessorrignnn
26

---

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
G. Visualization of t-SNE embeddings for Diverse Responses Across Queries
In this section, we showcase the performance of our method through plots of TSNE across various examples. These illustrative figures show how our baseline Bottom-k Algorithm (Section 5.1) chooses similar responses that are often close each other. Hence the model misses out on feedback relating to other parts of the answer space that it often explores. Contrastingly, often notice diversity of response selection for both the IMPOEOPPSSLECT and AMPO-CORESET algorithms.
150
150
Heapome
Reqporse Rejected Resporsos
Rejacted Repprrses
Rejecee Resporsx
100-
Otha Resparses
Oiki Repuzses
Othei Reipozis
50
50
SSNB Din
1.
-50
-50
-50
2
100
-150
-150
-150
-100
-50
100
-100
-50
100
-100
-50
50
(a) 1.
Resposs 42
30
Reeponse
Resporne
Rejental Rerрpeees
Rejected Ilupprece
Rejucted Requernxx
20
Othi: Resporses
20
Ochec Resperses
20
Oiher Reqparses
1SND 1SNB Dim
10
-10
-20
-20
-20
-10
-20
10
-10
-20
20
10
-10
-20
20
40
SNC Dim
(b) 2.
SSN Dim
10
10
2014
s
-10
-10
-10-
-20
20
20
-30
Rejected Reppstoci
Other Repperec
-30
Reiotied Ropasiss
Other Requerss
-30
Recctiad Reipesio
Other Recpassa
-20
-20
-20
6SNB Dim
(c) 3.
Figure 6. t-SNE visualization of projected high-dimensional response embeddings into 2D space, illustrating the separation of actively selected responses. (a) AMPO-BottomK (baseline). (b) AMPO-Coreset (ours). (c) Opt-Select (ours). Traditional baselines select many responses close each other based their rating, providing insufficient feedback the LLM during preference optimization. In contrast, our methods optimize for objectives including coverage, generation probability, and preference rating.
27


Reviews:
[{"review_id": "bTCdiuo0nV", "reviewer": "Reviewer_kqJE", "summary": "This paper studies the multi-preference optimization problem in which two sets of helpful and undesired responses are contrasted during self-play alignment. The authors propose Active Multi-Preference Optimization (AMPO), a framework comprising on-policy generation, a multi-preference group-contrastive loss, and active subset selection. AMPO scores and embeds large pools of candidate responses, then selects a small but informative subset covering reward extremes and distinct semantic clusters for preference optimization. The authors provide theoretical guarantees and empirical results showing AMPO achieves good performance on benchmarks (MT-Bench, AlpacaEval2, and Arena-Hard v0.1) with Llama 8B while  GPT-4-Preview-1106 as a judge model to give scores for responses.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and helpful review, and for acknowledging the strengths of our work. We appreciate the opportunity to address your concerns point-by-point. We have prepared a detailed response with additional experiments in the full rebuttal document (provided separately) and summarize the key clarifications below. We hope these responses are satisfactory and would be grateful if you would consider increasing your score.\n\n**Response to Weakness 1: Group Preference vs. Scores & Reward Model Bias**\n\nThis is an insightful point. Our investigation, detailed in the full rebuttal document, shows:\n\n1.  **Score Usefulness:** Using exact reward scores ($\\ell_1$-weighting) provides only a marginal gain over using scores simply to classify responses ($\\ell_0$-weighting), primarily on AlpacaEval 2 (see Table 1 in the detailed document). This suggests the main value comes from identifying the preferred group, aligning with literature viewing RMs as effective classifiers rather than precise regressors [Farebrother et al., 2024].\n2.  **Reward Model Bias:** We tested AMPO with two different reward models (8B Skywork and 2B GRM). The results demonstrate robustness, with AMPO performing well with both, indicating low sensitivity to the specific RM choice or its potential biases (see Table 2 in the detailed document).\n\n**Response to Weakness 2: Selection of Single Positive Response**\n\nOur choice to select the single best positive response against multiple negatives is motivated by:\n\n1.  **Theoretical Optimality:** Our analysis (Section B.1 in the paper) shows that maximizing expected reward involves concentrating probability mass on the single best-rated response while suppressing negative ones.\n2.  **Empirical Validation:** Ablations comparing 1-vs-k settings against k-vs-k settings (e.g., 1v7 vs 4v4, 1v3 vs 2v2) consistently show that the single positive approach yields better performance across benchmarks (see Tables 3 & 4 in the detailed document).\n\nBoth theory and experiments support focusing on the single highest-reward response for optimization.\n\n**Response to Weakness 3: Misleading Title (Human vs. LLM Feedback)**\n\nWe acknowledge the historical context. However, the current LLM alignment field widely uses LLM-generated preferences or reward models trained on initial (often human) pairwise data as a standard, scalable practice (e.g., SimPO [Meng et al., 2024], RPO [Adler et al., 2024], SPPO [Wu et al., 2024]). Furthermore, strong correlation between LLM judges and human preferences is well-documented [Zheng et al., 2023; Kim et al., 2023; Lambert et al., 2024]. Given this contemporary context, we believe the title accurately reflects our contribution within the standard paradigm where LLM-derived feedback is common and validated.\n\n**Response to Weakness 4: Typo in Line 135 / Eq. (5)**\n\nThank you for catching this. We acknowledge the typo and will correct it in the final version to match Equation 5.\n\n**Response to Weakness 5: Limited Experiments (Model & Hyperparameters)**\n\nWe have addressed this by:\n\n1.  **Adding Mistral-7B Experiments:** We now include results for Mistral-7B-Instruct-v0.2, demonstrating AMPO's effectiveness on another widely used model (see Table 5 in the detailed document).\n2.  **Analyzing Hyperparameter k:** We provide experiments varying the number of negative samples ($k$) for AMPO-Opt-Select (1vs3, 1vs5, 1vs7). Results show consistent improvement with $k$, but diminishing returns after $k=3$, supporting the efficiency of our subset selection (see Table 6 in the detailed document).\n\n**Response to Weakness 6: Practicality of Assumptions**\n\nWe appreciate the concern regarding our theoretical assumptions. As detailed in the full rebuttal document:\n*   Assumptions like (A1) Lipschitz, (A3) Finite Support, and (A5) Bounded-Diameter Clusters are standard in related theoretical analyses (e.g., metric bandits, clustering, policy learning) to enable formal guarantees.\n*   We acknowledge the gap between these idealized assumptions and real-world LLM behavior. Our method is empirically robust even when these assumptions hold only approximately.\n*   We will explicitly discuss these limitations and the practical implications in the revised manuscript to clarify the applicability.\n\n**Response to Minor Comments:**\n\n*   We will ensure consistent notation (N vs k) in the final version.\n*   We will review the main paper to ensure key related work is adequately summarized.\n\n---\n\nWe hope these clarifications and the additional experiments detailed in the full rebuttal document effectively address your concerns. We believe AMPO offers a novel and empirically strong approach to multi-preference optimization. We respectfully request you consider these points in your final assessment, and increase your score should you find that we have addressed your concerns.\n\nPlease find attached a document where we detail our response: <https://drive.google.com/file/d/1VQvnqa7dsX8RunB2u71tsOPzye7JkWBe/view?usp=sharing>", "claims_and_evidence": "The claims made in the paper are supported by clear and convincing evidence, including both theoretical and empirical results.", "methods_and_evaluation_criteria": "I do not find any issue with the method and evaluation criteria, except that the authors should have tested their methods across different models instead of only Llama 8B.", "theoretical_claims": "I have made a bird's-eye pass of the proofs of the theoretical results in the Appendix, and they appear correct to me. However, I have not read the proofs in detail.", "experimental_designs_or_analyses": "I do not find any issue with the method and evaluation criteria.", "relation_to_broader_scientific_literature": "To the best of my knowledge, the paper covers the relevant related scientific literature.", "essential_references_not_discussed": "The key related works are discussed.", "other_strengths_and_weaknesses": "#### **Strengths of paper:**\n1. This paper considers the multi-preference optimization problem, whose efficient solutions will be useful for LLM alignment. \n\n2. The authors propose an AMPO framework that combines on-policy generation, multi-preference optimization, and active learning for efficient multi-preference optimization. They overall propose four AMPO algorithms: AMPO-BottomK, AMPO-Coreset, AMPO-OptSelect (MIP), and AMPO-OptSelect (Coordinate Descent) ) in which the last two algorithms optimize expected reward for active subset selection. The authors also provide theoretical guarantees under some assumptions of optimality for the proposed algorithms\n\n3. The authors also verify the performance of the proposed algorithms via experiments using different benchmarks and release their datasets on Hugging Face.\n\n\n#### **Weaknesses of paper:**\n1. The motivating examples of why one should consider group-based preference optimization for robust alignment when scores for each response are available as scores have more information and one can directly use them than only considering group-based preference. Further, the scores may not be accurate or even have an inherent bias from the rating model used in the AMPO framework.\n\n2. It is unclear why only one positive response and $k$ negative responses are selected in each of the algorithms; why not use the range of scores (very high, high, low, and very low) and choose them across the response space? There should be some theoretical reason or ablations to justify the choice made in the paper.\n\n3. The paper's title is a bit misleading as it gives the impression that the paper considers the problems when only human preference feedback is available (i.e., binary feedback telling which one is better between two responses).\n\n4. There is a mismatch in the definition of $S^{-}$ in Line 135 and Eq. (5). \n\n5. **Limited experiments:** Only Llama 8B is used for the experiment. Additional experiments should be added to show how hyperparameters of the proposed algorithms (N and k) influence overall performance. \n\n6. **About assumptions:** It is unclear how well the assumptions taken in the paper hold in practice.", "other_comments_or_suggestions": "Check the weaknesses raised in **Other Strengths And Weaknesses**.  \n\n**Minor comment or suggestions:**\n1. For completeness, the authors should briefly add the most relevant related work in the main paper.\n2. The first two algorithms use $N$ to denote the number of responses, whereas the last two use $n$.", "questions_for_authors": "Please address the weaknesses raised in **Other Strengths And Weaknesses**.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "NZqEnV5123", "reviewer": "Reviewer_gkZq", "summary": "The paper introduces the AMPO (Active Multi-Preference Optimization) method, which aims to improve the alignment performance of large language models (LLMs) through active negative sample selection in multi-preference optimization. The main contributions are: proposing several active selection strategies to choose representative response subsets for more efficient multi-preference alignment training; demonstrating that coverage-based negative selection can systematically suppress low-reward modes and maximize expected reward. The paper achieves advanced results on multiple benchmarks and releases relevant datasets for future research.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and helpful review. We have conducted several additional experiments, detailed in the full rebuttal document (provided separately), and summarize the key findings here. We hope these responses adequately address your points and would be grateful if you would consider increasing your score.\n\n**Response to Weakness 1: Limited Number of Experiments**\n\nTo broaden the validation of AMPO, we have significantly expanded our experiments as detailed in the full rebuttal document:\n\n1.  **New Model Setting:** We added results for **Mistral-7B-Instruct-v0.2**, demonstrating AMPO's effectiveness beyond Llama-3 (see Table 1 in the detailed document).\n2.  **Reward Model Ablations:** We tested AMPO with a **2B GRM-Reward-LM** [Yang et al., 2024] alongside the original 8B Skywork RM, showing robustness to the RM choice (see Table 2 in the detailed document).\n3.  **Single vs. Multiple Positive Responses:** Experiments confirm that contrasting **1 positive vs. k negatives** consistently outperforms k-vs-k settings, validating our primary approach (see Tables 3 & 4 in the detailed document).\n4.  **Variation with k (Number of Negatives):** We analyzed performance as $k$ increases in the 1-vs-k setup. Results show diminishing returns after $k=3$, supporting our claim that selecting a subset (like 1 vs 7) is efficient (see Table 5 in the detailed document).\n5.  **Variation with/without Rating Weight:** Ablations comparing $\\ell_1$ (using scores) vs. $\\ell_0$ (classification only) weighting show marginal differences, indicating the main benefit comes from the select/reject preference signal (see Table 6 in the detailed document).\n\nAs a sample of the results obtained, we provide our results in the new model setting as below:\n\n| Method              | Reward Model      | AlpacaEval 2 LC (%) | AlpacaEval 2 WR (%) | Arena-Hard WR (%) | MT-Bench GPT-4 |\n| :------------------ | :---------------- | :-----------------: | :-----------------: | :---------------: | :-------------: |\n| Base                | -                 | 17.1                | 14.7                | 12.6              | 7.5             |\n| Best vs Worst (SimPO)| Skywork-Reward-LM | 30.1                | 32.3                | 21.1              | 7.56            |\n| AMPO-Bottomk        | Skywork-Reward-LM | 32.1                | 37.0                | 22.1              | 7.7             |\n| AMPO-Coreset        | Skywork-Reward-LM | 32.8                | 37.3                | 22.6              | **7.75**        |\n| AMPO-Opt-Select     | Skywork-Reward-LM | **33.1**            | **37.8**            | **22.8**          | 7.72            |\n\n*Table 1: Performance on Mistral-7B-Instruct. Best results in **bold**.*\n\n\n\nWe believe these comprehensive experiments substantially strengthen the paper's empirical validation across different scenarios.\n\n**Response to Weakness 2: Explanation of Training Data Size**\n\nWe apologize for the lack of clarity. We use the standard **UltraFeedback dataset**.\n*   The initial dataset has 61.1k training examples. Due to filtering for AMPO strategies requiring $k=7$ rejected responses, our final dataset consists of **59.2k training** and **1.92k validation** examples.\n*   Using UltraFeedback is standard practice (e.g., SimPO [Meng et al., 2024], RPO [Adler et al., 2024]).\n*   The specific filtered datasets are released (links provided in the detailed document).\n\n**Response to Weakness 3 & Question 2: Advantages of Opt-Select over Coreset**\n\nWhile AMPO-Coreset is simpler and strong at T=0, AMPO-Opt-Select offers key advantages:\n\n1.  **Stability at Higher Temperatures:** Opt-Select consistently outperforms Coreset at typical sampling temperatures (T=0.25-0.75), indicating greater robustness in practical use cases (see Figure 1 in the detailed document).\n2.  **Robustness Across Reward Models:** Opt-Select showed significantly better performance relative to Coreset when using the smaller GRM-Reward-LM, suggesting it's less sensitive to the specific RM (see Table 2 in the detailed document).\n\nTherefore, Opt-Select provides more robust alignment, particularly under varied generation settings and reward models. We will clarify this trade-off in the revision.\n\n**Response to Question 1: Other Multi-Preference Baselines**\n\nOur experiments include **Best-vs-Worst (SimPO)** adapted to multi-response data and **AMPO-Bottomk**. While other methods exist, many recent ones like **RPO-bwd** are equivalent to **InfoNCA**, which uses a different loss structure (multi-class CE) than AMPO's group contrastive approach.\n\n---\n\nWe hope these clarifications and the additional experiments detailed in the full rebuttal document address your concerns. We believe AMPO offers a valuable contribution to multi-preference alignment and respectfully request you consider these points in your final assessment.\n\nWe provide a document outlining our experimental results here: <https://drive.google.com/file/d/1XV9v5dZAQfO9mggCuW85mu_0wgoB6gdF/view?usp=sharing>", "claims_and_evidence": "AMPO's active selection strategies (such as AMPO-Coreset and AMPO-OptSelect) can cover a wider semantic space and avoid selecting overly similar responses, thus providing richer training signals. This is intuitively demonstrated through t-SNE visualization.", "methods_and_evaluation_criteria": "The paper selected three benchmarks (AlpacaEval, Arena-Hard, and MT-Bench) to evaluate model performance, but it may lack assessment on datasets targeting mathematics, code, and reasoning.", "theoretical_claims": "The paper proves that the AMPO-OptSelect strategy can maximize the model's expected reward. For large-scale problems, it proposes a local search algorithm that can approximate the optimal solution with polynomial time complexity, a conclusion supported by theory.", "experimental_designs_or_analyses": "The paper validates AMPO's effectiveness through benchmarks and compares it with existing methods (SIMPO). However, it only trains and tests on one benchmark model, and the baseline for comparison includes only one method, making the comparison less convincing.", "relation_to_broader_scientific_literature": "AMPO continues the research direction of multi-preference optimization, providing richer training signals by considering multiple \"good\" and \"bad\" responses simultaneously, which resonates with recent related work (such as SWEPO).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: The paper proposes multiple different selection strategies for negative samples in preference optimization to improve training efficiency. It proves that coverage of subsets in semantic space is an important consideration for negative sample selection.\n\nWeaknesses:\n1. The number of experiments is small, lacking persuasiveness in a wider range of scenarios.\n2. There is no explanation of the size of the training data.\n3. The selection strategy based on optimization problems (Opt-Select) is not better than the strategy based on clustering results (AMPO-Coreset), which is clearly more understandable in theory and simpler in computation. The paper fails to explain the advantages of the Opt-Select method over the AMPO-Coreset method.", "other_comments_or_suggestions": "Please refer to Other Strengths And Weaknesses", "questions_for_authors": "1. Are there other multi-preference optimization related works that can serve as experimental baselines?\n2. What are the advantages of Opt-Select compared to AMPO-Coreset?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WsQKwTw5o5", "reviewer": "Reviewer_P2gm", "summary": "The paper introduces AMPO, a method for aligning LLM using multi-response preference selection instead of traditional pairwise comparisons. It combines on-policy data generation, a group-contrastive loss, and active subset selection to train models. Various selection strategies, called bottom-k, coreset clustering, and Opt-Select, was proposed to improve preference learning by focusing on diverse and extreme responses. Experiments on Llama 8B show improved performance on AlpacaEval and other benchmarks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer P2gm for their insightful and helpful feedback. We appreciate the opportunity to address the concerns raised. We have prepared a detailed point-by-point response with supporting experiments in the full rebuttal document (provided separately) and offer a summary here. We hope these clarifications adequately address the reviewer's points and would be grateful if the reviewer would consider increasing their score should they find the responses satisfactory.\n\n**Response to Weakness 1: Method Limitation to Reward Model Scenarios vs. Pairwise Preference**\n\nWe understand the concern regarding the reliance on a reward model versus traditional pairwise human feedback. Our detailed response (Part 1 of the rebuttal document) clarifies this in three ways:\n\n1.  **Alignment with Modern Practices:** We highlight that contemporary LLM alignment commonly uses an initial phase of pairwise preference collection (human or LLM) to train a reward model (RM). This trained RM is then used to generate scalar scores for many candidate responses scalably and cost-effectively. This approach, employed by methods like SimPO [Meng et al., 2024], RPO [Adler et al., 2024], SPPO [Wu et al., 2024], and RSPO [Tang et al., 2025], *extends* rather than contradicts pairwise methods. Our method, AMPO, builds upon this standard practice.\n2.  **Validity of LLM Feedback:** We reference literature (e.g., \"LLM as Judge\" [Zheng et al., 2023], Prometheus [Kim et al., 2023], RewardBench [Lambert et al., 2024]) demonstrating that scores from well-trained RMs or strong LLM judges correlate highly with human preferences, serving as effective proxies.\n3.  **Empirical Robustness (Detailed in Document):**\n    *   **Changing the Reward Model (Part 2 & Table 1):** We show empirically that AMPO maintains strong performance even when switching from an 8B Skywork RM to a 2B GRM-based RM [Yang et al., 2024], indicating robustness to the specific RM used.\n    *   **Removing Rating Weight (Part 3 & Table 2):** We demonstrate through ablation (using L0 weighting) that using reward scores *only* to classify responses into \"select\" and \"reject\" pools achieves performance nearly as strong as using the precise scores (L1 weighting). This suggests the primary value comes from the *preference signal* (select vs. reject), aligning with findings that LLMs excel as classifiers [Farebrother et al., 2024], rather than relying solely on perfectly calibrated scores.\n\nIn summary, AMPO leverages the standard and scalable practice of using reward models (often derived from initial pairwise data) and is robust to both the specific reward model and the precise granularity of the scores.\n\n**Response to Weakness 2: Broken Dataset Link**\n\nThank you for pointing this out. We sincerely apologize for the typo in the abstract's link. The links in the contributions section were correct. The correct URLs are:\n\n*   **OPT Select:** https://huggingface.co/datasets/Multi-preference-Optimization/AMPO-OPT-Selection\n*   **Coreset:** https://huggingface.co/datasets/Multi-preference-Optimization/AMPO-Coreset-selection\n\nWe will correct this in the revised manuscript.\n\n**Response to Weakness 3: Isolating Method Contribution vs. Reward Model Choice**\n\nWe appreciate this insightful question about isolating AMPO's contribution. As summarized above and detailed in our full rebuttal (Part 2 & Table 1):\n\n1.  **Cross-RM Experiments:** We conducted experiments using two distinct reward models: the 8B Skywork RM and a 2B GRM-based RM [Yang et al., 2024]. AMPO demonstrated strong and comparable performance across both, particularly on AlpacaEval 2. This suggests the observed improvements stem significantly from the AMPO selection and optimization framework itself, not just the quality of one specific reward model.\n2.  **AMPO's Use of RM:** Furthermore, AMPO uses the reward model primarily to establish *relative rankings* – identifying the best response and selecting a diverse set of lower-quality responses for contrast. The optimization then maximizes the likelihood of the best while minimizing the joint likelihood of the negative set. This reliance on relative ranking, rather than direct supervision by noisy scalar values, enhances stability and generalizability.\n\nThe full experimental results comparing both reward models are included in the rebuttal document and will be in the updated manuscript.\n\n---\n\nWe hope these summaries, elaborated upon in the full rebuttal document, effectively address the reviewer's concerns. We believe AMPO offers a valuable contribution by efficiently leveraging multi-preference data within modern alignment paradigms. We respectfully request the reviewer consider these responses in their final evaluation.\n\nWe provide a detailed document outlining our response here: <https://drive.google.com/file/d/1NzYto9Nl7XyIDIMJKtk3uD2H_yHG2hun/view?usp=sharing>.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No due to time limit.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The key contribution is a unified pipeline for self-play alignment considering multi-preference. It could inspire more discussion in the raising topic of on-policy self-play alignment.", "essential_references_not_discussed": "Since the paper focuses on On-Policy Self-Play, I believe more related works in this area should be discussed. Currently, the related work section includes only four references, none of which are recent. This makes it less clear how the paper contributes to the field.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper is well-written and easy to understand.\n- The proposed method achieves state-of-the-art results on AlpacaEval.\n\nWeaknesses:\n\n- Perhaps the biggest concern is that the method is limited to scenarios where a reward model is available to evaluate and score all answers. This contradicts the paper's claim that the proposed solution is superior to pairwise preference. Pairwise preference addresses challenges in collecting real human feedback for RLHF.", "other_comments_or_suggestions": "The dataset link in the abstract leads to 404.", "questions_for_authors": "How does the choice of the reward model affect performance? Could the observed improvement be primarily due to a well-designed reward model rather than the proposed method itself? To better isolate the contribution of the method, it would be beneficial to design an experiment that compares its performance across different reward models.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary
The paper introduces AMPO, a framework that aligns large language models with a set of responses rather than pairwise preference data. For each prompt, it selects the single best response and contrasts it against an actively chosen subset of poorer answers drawn from on‑policy generations. Several active‑selection strategies are analysed, and variants of the framework are evaluated on multiple public benchmarks. While the empirical results are promising, the study relies on a limited set of baselines.

Recommendation
Considering (i) one clear weak‑accept review, (ii) two weak‑reject reviews whose main concerns appear to be addressed in the rebuttal (though the reviewers have not yet updated their scores), and (iii) additional results that support the performance claims, I recommend weak accept pending minor revisions. I highly recommend including more baselines.
Decision: Accept (poster)