Clone-Robust AI Alignment
Authors: Ariel D. Procaccia
Benjamin Schiffer
Shirley Zhang
Venue: ICML 2025 poster (2025)
Keywords: RLHF
AI alignment
social choice
Primary area: social_aspects->alignment

Abstract: A key challenge in training Large Language Models (LLMs) is properly aligning them with human preferences. Reinforcement Learning with Human Feedback (RLHF) uses pairwise comparisons from human annotators to train reward functions and has emerged as a popular alignment method. However, input datasets in RLHF can be unbalanced due to adversarial manipulation or inadvertent repetition. Therefore, we want RLHF algorithms to perform well even when the set of alternatives is not uniformly distributed. Drawing on insights from social choice theory, we introduce robustness to approximate clones, a desirable property of RLHF algorithms which requires that adding near-duplicate alternatives does not significantly change the learned reward function. We first demonstrate that the standard RLHF algorithm based on regularized maximum likelihood estimation (MLE) fails to satisfy this property. We then propose the weighted MLE, a new RLHF algorithm that modifies the standard regularized MLE by weighting alternatives based on their similarity to other alternatives. This new algorithm guarantees robustness to approximate clones while preserving desirable theoretical properties.

AI Alignment
Ariel D. Procaccia
Benjamin S 2 Shirley Zhang'
Abstract
and results in a large dataset of pairwise comparisons. An RLHF algorithm then takes the pairwise comparison dataset as input and outputs reward function that assigns values to answers. One reason why RLHF is an appealing technique is because of the ease of data elicitation, as it is simpler for humans to pick a favorite between two answers than it is to rank many answers or provide good 'role model' answers for the LLM.
A key challenge in training Large Language Mod- els (LLMs) properly aligning them with hu- man preferences. Reinforcement Learning with Human Feedback (RLHF) uses pairwise compar- isons from human annotators to train reward func- tions and has emerged as popular alignment method. However, input datasets in RLHF can unbalanced due to adversarial manipulation or inadvertent repetition. Therefore, we want RLHF algorithms to perform well even when the set of alternatives is not uniformly distributed. Drawing on insights from social choice theory, we introduce robustness to approximate clones, a desirable property of RLHF algorithms which requires that adding near-duplicate alternatives does not significantly change the learned reward function. We first demonstrate that the standard RLHF algorithm based on regularized maximum likelihood estimation (MLE) fails to satisfy this property. We then propose the weighted MLE, a new RLHF algorithm that modifies the standard regularized MLE by weighting alternatives based on their similarity to other alternatives. This new algorithm guarantees robustness to approximate clones while preserving desirable theoretical prop- erties.
Fundamentally, the mandate of RLHF algorithms is to solve a preference aggregation problem, where the goal is find reward functionn that best aligns with the values of the general population, given pairwise comparison data. This goal is complicated by the fact that humans often have diverse preferences, and may not agree on the best answer to a question. Luckily, the study of how to aggregate diverse preferences is not a new area in computer science, but one that has been long explored by researchers in social choice theory. Classic social choice considers settings with sets of voters and sets of alternatives, where each voter provides ranking over alternatives. These rankings are then provided as input to voting rule, which outputs a summary of the voters' preferences (such as a single winner or an overall ranking). Social choice studies the design and analysis of such voting rules.
In the RLHF setting, the 'voters' are the human annotators, who provide pairwise preferences over the 'alternatives', There are several reasonable choices for how to define an 'alternative" in RLHF, Perhaps the simplest definition is that alternative is just single answer. However, many RLHF datasets in practice contain answers to multiple questions. Therefore, an alternative could also be a question- question-answer answer pair. Finally, answers (and questions) are often generated by various LLMs, so an alternative could also be viewed as the LLM model which generated the answer. Whichever the case, an RLHF algorithm would then be the 'voting rule' which takes as input the preference data and outputs summary of the preferences in this case, single reward function. The close relationship between RLHF and voting theory means that we can take inspiration from past work in social choice to both anticipate potential pitfalls in RLHF and design better RLHF algorithms.
1. Introduction
As the reasoning capabilities of Large Language Models (LLMs) improve and as LLMS begin to play a larger role in society, it is increasingly important for LLMs to be aligned with human values. One common method used for Al align- ment is Reinforcement Learning with Human Feedback (RLHF). In RLHF, a human annotator is typically shown two answers to prompt, and asked to report which answer they prefer. This process is repeated across many annota- tors and potentially many types of questions and answers,
'Department of Computer Science. Harvard University 'Department of Statisties, Harvard University. Correspondence to: Benjamin Schiffer <bschifferl @ @g. har.haarddddd... Shirley Zhang <szhang2 @g hdd>..
One especially relevant potential pitfall is that input datasets in RLHF are not necessarily balanced in the types of ques- tions and answers that are included, whether due to random-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Clone-Robust AT Alignment
ness adversarial influence. For example, at Anthropic, questions are generated by crowdworkers, who have the flex- ibility to communicate with LLMs on any topic of interest (Bai et al., 2022). In ChatGPT, answers are typically gener- ated by LLMS, and depending on how the LLM was trained, some answers may look more similar than others (OpenAI, 2022). Because the generation processes for these datasets are not well-regulated, it would be ideal for an RLHF algo- rithm to not be sensitive to near duplicates and to perform well even on unbalanced datasets. This is important for at least two reasons. First, RLHF algorithms which are not robust to near duplicates will require more careful design of the input dataset, which may increase cost and restrict the types of questions and answers that can be included in the dataset. Furthermore, it may be more difficult add to the dataset over time, as it will be necessary to check for near duplicates in the existing dataset. Second, RLHF algorithms which are robust to near duplicates will also be more robust to both adversarial manipulation and accidental duplication.
response the second goal, we propose a new algorithm for RLHF that we call the weighted MILE. Intuitively, the weighted MLE adjusts the objective function of the MLE by down-weighting alternatives that are similar to other al- ternatives (and therefore provide less new information) and up-weighting alternatives that are different than other alter- natives (and therefore provide more new information). Our main result that the weighted MLE is robust to approxi- mate clones; we also demonstrate that it retains many of the clean interpretations of the regularized MLE.
In addition to our main result about independence of clones, we also prove an impossibility result for RLHF in the pres- ence of diverse preferences. We show that for any RLHF algorithm, there exists a population such that even with constant scaling, the distance between the RLHF algorithm output and the mean rewards of the population is arbitrar- ily large. We show this for a population that consists of a mixture of only two Bradley-Terry- thereby highlighting the inherent difficulty of aggregating preferences of populations with even simple diversity in preferences.
(BTL) models,
In social choice, if a voting rule is robust to adding dupli- cates of alternatives, it is said to satisfy independence of clones. Informally, a voting rule is independent of clones if after adding an alternative a' which equivalent another alternative a, the output of the voting rule does not change. In the RLHF setting, we can think of approximate clones as two alternatives which are very close by given distance metric and for which all annotators have very similar val- ues, where the distance metric depends on the nature of the alternatives. For example, the alternatives are textual responses, then a reasonable distance metric might be the Euclidean distance between their vector embeddings. An approximate clone of an textual response might look like the original response with an adjective replaced by syn- onym. We can then extend the concept of independence of clones to robusiness lo approximate clones. Informally, an RLHF algorithm is robust to approximate clones if adding a new alternative to the data set that is similar to an existing alternative does not significantly change the output reward function. This is intuitively a desirable property because adding a new alternative that is very similar to existing alternative does not provide much new information about annotator preferences.
We also extend the ideas of Siththaranjan et al. (2023) that relate the output of the regularized MLE to the average win rates of the alternatives. We show that the output reward function of the regularized MLE is the solution of equations, where the left hand side is the average win rates for the rewards output by the MIE and the right hand side is the empirical average win rates. We similarly show that the output of the weighted MLE has the same relationship to the empirical weighted average win rates.
system
We conclude with case study using LLM generated an- swers to single prompt. We use LLMs as annotators to generate two preference datasets, where one dataset has ad- ditional cloned alternatives. We then approximate both the standard MIFE algorithm and the weighted MLE algorithm using neural networks. In this experiment, we show that the output of the standard MLE is significantly more affected by the presence of approximate clones than the output of the weighted MLE, which supports our theoretical results.
1.2. Related Work
Building on these insights, our main research goals are to
RLHF has recently gained traction as a popular method of aligning LLMs with human preferences (Bai et al., 2022; Ouyang et al., 2022; Ziegler et al., 2019). The potential benefits of applying social choice theory to the RLHF setting have not gone unnoticed. Recent work has mapped classic social choice concepts to RLHF (Dai & Fleisig, 2024) and extended social choice axioms for RLHF (Ge et al., 2024), considered personalization as a way to address diversity (Poddar et al., 2024; Park et al., 2024), and studied other methods of aggregating diverse preferences (Zhong et al., 2024; Swamy et al., 2024).
1. evaluate the robustness of current RLHF algorithms in the presence of approximate clones, and
2. develop RLHF algorithms which are robust to approxi- mate clones.
1.1. Our Results
To address the first goal, we show that the standard RLHF algorithm, which uses the regularized maximum likelihood estimator (MLE), not robust to approximate clones. In
2

---

Clone-Robust AI Alignment
Our work particularly focuses on the social choice concept of independence of clones, which first introduced by Tideman (1987). Follow-up work has studied manipulation by cloning and clone structures (Elkind et al., 2010; 2012), and recent work has studied an even stronger notion, ob- vious independence of clones (Berker et al., 2025). In a position paper, Conitzer et al. (2024) propose various high- level directions for applying social choice to RLHF, and in particular identify independence of clones as a desirable property for RLHF algorithms, because chatbot responses may be very similar to each other. They point out that Borda count, a voting rule which is implicitly used in cur- rent approaches to RLHF (Siththaranjan et al., 2023), is not independent of clones. In our work, we elaborate on this insight by considering approximate clones and providing specific instances for which standard RLHF algorithms are not robust to approximate clones.
an annotator i states a preference for alternative 21 over alternative 22 with probability
e(i(z1)
pa > x2) = (t7(xa) + c"T(22)'
The BTL model takes into account the fact that annotator preferences may be noisy or inconsistent across queries, especially when the reward gap of two alternatives is small. When annotators are drawn randomly, we then denote the expected probability of seeing 21 preferred 22 as p" (x1 > w2) (p > =22]]-
We assume that annotator reward functions are Lipschitz continuous in the Euclidean distance between the context of two alternatives, as stated below.
Assumption 2.1. For all players i E N. the reward function rf is I Lipschitz continuous with parameter K >> 0. Formally, for any i E N and any I1, E S, (r] - r7 (x2)) S K |m - x22l22-
We highlight several papers that are especially related to ours, and include more details in Appendix A. Like us, Xu et al. (2023) are concerned about duplicates in answers shown annotators, but unlike us, their results are primarily for dichotomy models and three-way comparisons. Also like us, both Siththaranjan et al. (2023) and Chakraborty et al. (2024) give different forms of impossibility results for RLHF algorithms in the presence of diverse preferences.
Let query be a pairwise comparison q= {=1, 22), where 21 E M, and let a set of queries be denoted Q. For every 21, 22 E M,, we assume that {mi, x2) is included in Q at least once. For æ1, 22 E . and i E N, define the random function fITL (fr,((21,22):: that Pr(JETI. ((2),,))))))),,;)))) = =1) = pi(zi 222). In other words, fBTL (g. i) is one sample of annotator i's preference between 1 and 22 according to annotator i's true rewards for I1 and x2 in the BTL model. Further let fBTL(G) = SBTI. i') when i' is drawn uniformly at random from A. A preference dataset ((2) is generated from Q by choos- ing an annotator uniformly at random for each query and sampling that annotator's 's preference over the alternatives in that query, i.e. D(2)= = ((g, (V))(4)) : q Q).
{21 such
An independent work published shortly after this paper also studied the problem of clones and used different weighting scheme to address the clone problem (Berriaud & Watten- hofer, 2025).
Further afield, recent papers have considered other forms of robustness in RLHF, such as robustness to incorrect or corrupted data (Bukharin et al., 2024; Mandal et al., 2024). In our work, we specifically focus on robustness to approxi- mate clones, and expect inconsistent data due to diversity in the annotator population.
For a given preference dataset DD define pD (21 > x22) as the proportion of time that x is preferred to 22 in D. We say that a dataset D is representative if pp p
x2)
p" (I1 > x22) for all 21, E M. An RLHF algorithm ALG takes as input a preference dataset D and returns a reward function r(-) where r . + R. Note that ALG does not know any information about the annotators (such as N, Pt, or p*). The goal of ALG is to find a "good" reward function r based on D.
2. Model
Suppose we have a set of annotators N = [n] and an infinite set of all possible alternatives S. We define|S) as the volume of S and assume that [S] is finite. Each alternative s E S has a context c(s) E Rd which represents important features of the alternative. For notational convenience, we will often refer to the context c(s) simply by 8. We only observe finite subset of alternatives M = [m] C S. Each annotator has a reward function rf .M- + R. where r² (x) represents the reward of annotator i for alternative x.
For intuition, it may be helpful to keep the following prefer- ence dataset generation example in mind:
Example 2.2. Suppose that we want find a reward func- tion which evaluates responses to a specific question Z. Then S would be the set of all possible responses to Z, and M would be finite subset of responses to Z. We can gener- ate each query q E Q by randomly sampling two responses I 22 from M. We can then generate a preference datum for this query by randomly sampling an annotator i from A and asking annotator i for their preference between Ti and
Given two alternatives, an annotator expresses a preference over the two alternatives based on their reward function. As is common in RLHF, we assume that the expressed pref- erences of annotators follow a model (Bradley& & Terry, 1952; Bai al., 2022), in that
(BTL)
3

---

Clone-Robust AT Alignment
T2. The set of all preference datum then forms our dataset D, which we give input an RLHF algorithm AL.G
also be included guarantee that the optimization problem has a solution. Formally, for À > 0, define
2.1. MLE with Diverse Preferences
pP := arg min
Σ #1
pD(01 + x2) log (r((r1) + e"(m22)
We first consider the setting with only one annotator (n = 1). When n = 1, the query data is generated from single BTL model. A natural solution is to estimate the unknown reward function r" as the reward function that best matches the data in D. Using the distance metric, the reward function that best approximates the data in D (minimizes the KL divergence) when n is
r(x)²
(1)
REM
Bellback-Leibler divergence (L(() as the
Using this regularized MLE is a standard method for RLHF due its interpretability (Siththaranjan et al., 2023). Note that single BTL model is typically used to approximate DD both for the simplicity of the model and because is unknown to the algorithm. Another benefit of the above formulation is that the objective in Equation (1) is strictly convex and has a unique global minimum (Siththaranjan et al., 2023). In practice, the optimization problem can be solved approxi- mately using a sufficiently large function class for r such as neural network. In the following sections, we will analyze this standard MLB-based RLHF algorithm and propose a new algorithm with additional theoretical guarantees.
(~ ((r))
min
arg min
Σ 2MM
pD(21 > x2) -log (r(on) + er(z))
When the number of samples for every pair of alternatives is the same, fl is exactly the MLE solution for RLHF In RLHF, maximum likelihood estimation refers to finding the rewards for the single BTL model that has the highest probability of generating the observed data.
Furthermore, because the true underlying distribution is 1, standard MLE theory implies that f' will converge to r* := [.[rā] = ri as the number of comparisons for each pair of alternatives goes to infinity (Zhu et al., 2023).
single BTL model when
2.2. Average Win Rate and Borda Count
In addition to the interpretation discussed above, Siththa- ranjan et al. (2023) showed that the order in which the alternatives are ranked by the regularized MLE is the same as the order in which the alternatives are ranked by the av- erage win rate. In Theorem 2.5, we show even stronger relationship between the regularized MLE and the average win rate, which is that the regularized MLE is the unique solution to a system of equations involving the empirical average win rates. This gives additional interpretability to the regularized MLE, as the regularized MLE is therefore similar an M-estimator where the m moments correspond to the win rates of the m alternatives.
Having jP converge to E,[r;] is a natural goal in RLHF, as this means that the reward function converges to the mean rewards for the underlying population. Unfortunately, when 1, no RLHF algorithm can accurately recover [r]]]]] for every possible population even if the algorithm is given infinite query data. This negative result holds even for n = 2 and m = 2 and is formally proven in Theorem 2.3 in terms of Euclidean distance. Note that there an additive constant in this result because the BTL model is invariant to additive constants. Furthermore, Theorem 2.3 does not contradict the positive results of Zhang et al. (2022), as their results require sufficiently many alternatives in order to make the problem identifiable.
Definition 2.4. For dataset ' over alternatives M, the average win rate of alternative x E M is
1 AWRD(I) == m VEM
Σ pa > 1).
Theorem 2.3. Let n = and suppose 'D is a representative preference dataset over alternatives in M. Then for any algorithm ALG and any C > O, there exist rū and rž such
that pD AD) satisfies
Theorem 2.5. Let fi: be the regularized MLE as defined in Equation (1) and define
min OeR
Σ 3EM
(r(()) + (1)
P(x) > C.
(a) Σ (x) + (g))
2
AWR(=)
1 m
VEM
The proof of Theorem 2.3 can be found in Appendix B.
Then i: is the solution to the system of equations
Even though accurately estimating the mean reward func- tions of the population is not always possible for n > 1, the case
AWRp(x) Xi (x) + AWR(x) Vo E M.
same arguments that motivate using fa in the n can also be applied to the n > 1 case. More specifically, we can still find the single BTL model that best approxi- mates D. As is the case when n = 1, the single BTL model that minimizes the KL-divergence is also the MIE solution. When n may be greater than 1, a regularization term must
The proof of Theorem 2.5 can be found in Appendix F. Importantly, the average win rate as defined above is con- ceptually similar to the Borda count score in social choice
4

---

Clone-Robust AT Alignment
theory. Therefore, Theorem 2.5 implies a close relation- ship between the regularized MLE and the Borda count voting rule, relationship first observed by Siththaranjan et al. (2023). The close relationship between the regularized MLE and Borda count is a key aspect of the proofs in the following sections.
that of existing alternative does not significantly change the reward function. This is desirable because if the players' values are Lipschitz continuous, new alternative whose context is very similar to that of an existing alternative provides little new information. Note that this can intuitively be viewed as requiring that the output reward function is continuous in the set of alternatives.
3. Robustness to Approximate Clones
Robustness to approximate clones is also reminiscent of core ideas in differential privacy (Dwork, 2006), where the goal is to design algorithms that are robust to removing any data point. Any RLHF algorithm which satisfies robustness to approximate clones also automatically satisfies exact in- dependence of clones, which informally says that adding new alternatives which are exact clones does not change the output reward function at all. We formally define and discuss exact independence of clones in Appendix C.3.
In this section, we adapt the concept of independence of clones from social choice to the RLHF setting. In traditional social choice, independence of clones is desirable char- acteristic of voting rules which intuitively states that the winner of an election remains the same when duplicates of candidates are added to the candidate pool. The Borda count voting rule, which is closely related to the MLE in RLHF (see Section 2.2), does not satisfy independence of clones. See Appendix C.I for formal definition of independence of clones in traditional social choice.
Importantly, the regularized MLE does not satisfy Definition 3. as stated formally in Theorem 3.2. See Appendix D for the proof of Theorem 3.2.
We adapt the traditional independence of clones definition for RLHE Informally, we say that an RLHF algorithm is robust to approximate clones if adding new alternatives that are clones of existing alternatives does not significantly change the reward function that is output by the RLHF algo- rithm. Note that robustness to approximate clones in RLHF guarantees stability of the reward function instead of merely the winner, and is therefore a stronger notion. As an RLHF algorithm only has access to noisy observations regarding human preferences, we will also only require reward func- tion stability when we have representative datasets, or in other words, in cases when the empirical pairwise win rates are the same as the true pairwise win rates. If the dataset is not representative, it is not necessarily desirable that the reward function is unchanged when a clone is added because there may be value in generating a larger dataset. When the dataset contains sufficiently many queries, the empir- ical pairwise win rates will approximately equal the true pairwise win rates by the law of large numbers. Additional justification of our definition of robustness to approximate clones in RLHF can be found in Appendix C.2.
Theorem 3.2. Let io be the regularized MLE as defined in Equation (1). The algorithm (LG( fP is not robust to approximate clones.
4. Weighted MLE
In this section, we propose a modified version of the regular- ized MLE which satisfies Definition 3.1 while maintaining the interpretability inherent to the original MLE.
The main idea of the proposed algorithm is to modify the objective function by weighting each alternative by how unique that alternative is compared to the other alternatives in M. Therefore, an alternative with context very similar to the context of other alternatives will have smaller weight, while an alternative with a context very different than the context of other alternatives will have a larger weight. In- formally, the weight of an alternative is the fraction of alternatives in S that are closer to y than to any other alterna- tive in M (with ties split evenly among all tied alternatives). We define the weights formally in Definition 4.1. Definition 4.1. For any set of alternatives M S and any E S, define proj (z) = arg milyeM ||x - -vll2 S M. For y E M,, define
We are now ready to formally present our definition of robustness to approximate clones for RLHF
Definition 3.1 (Robust to Approximate Clones). An algo- rithm ALG is robust to approximate clones if for every M C S and 8 > 0 there exists an E > 0 such that the following holds. Suppose M M U {a'), where x' E S and Fa E .M such that || - 1'|| S e. Let D be a represen- tative dataset of queries over the alternatives M and let D' be a representative dataset of queries over the alternatives in M'. Let = AD) and let /' = ALG(D'). Then (x(x) f''') S and for all x E M, (f(x) - (()) s 8.
(yprroi (x) dr. wM(() [8] JEES pprojm(x)
1
Note that by this construction, ΣyEM WM(y) = 1. Us- ing these weights, we define the weighted MLE as 18 arg min, (r), where
( (MMa))emxxs)ooll > x2)
Informally, a RLHF algorithm satisfies Definition 3.1 if adding a new alternative whose context is "very close" to
fo(r) = Σ
5

---

Clone-Robust AI Alignment
(x(o))
x log ex(m1) er(ez)
0.8
Σ "M(x)r(a) JEM
0.6
0.4
Intuitively, fr(r) down-weights terms involving alterna- tives that provide less new information because they are very similar to other alternatives. Consequently, two alter- natives that are approximate clones of each other will both have smaller weights.
0.2
0
0
0.2
0.4
0.6
0.8
Figure 2. Diagram for .M {(0, 0), (1,)),
We are now ready to state our main result, which is that the weighted MLE robust approximate clones Theorem 4.2. Under Assumption 2.1, the algorithm ALGG(D) ii is robust to approximate clones.
the introduction of the clone does not significantly change the weighted MLE objective function. In the proof of The- orem 4.2, we formally show this, and conclude that the weighted MLE reward function also does not significantly change.
While we defer the formal proof of Theorem 4.2 to Ap- pendix G, we will next provide some intuition for why this result holds. Consider the Voronoi diagram of the alter- natives in M, which consists of partitioning of S into regions, where each region corresponds to all of the points in S that are closest some alternative in M. For example, if S [0, 1] [0, 1] and M {(0, 0), (1,0), (1, 1)), then we can draw the Voronoi diagram in Figure 1.
The weighted MLE is not only robust approximate clones, but also preserves many of the same interpretations as the standard MLE. For example, the weighted MLE and the standard MLE are equivalent whenever . is uniformly distributed across S, i.e. when (g) = .M In this case, Equation (2) a scaled version of Equation (1), and therefore fD F Therefore, the weighted MLE only differs from the standard MLE when the alternatives in M are not evenly distributed across S.
1 for all E M.
0.8
0.6
To further understand this new algorithm, we discuss two additional perspectives on the weighted MLE and its rela- tionship to the standard MLE.
0.4
0.2
Relationship to Weighted Average Win Rate
0
In Theorem 2.5 we established a strong connection between the MLE and the empirical average win-rate. Similarly, the weighted MLE has a close relationship with the weighted av- erage win rate defined Definition 4.3. Specifically, Theo- rem 4.4 shows that the weighted MLE is also an M-estimator solving system of equations relating the weighted average win rate of to the empirical weighted average win rate in the data set 'D.
0.2
0.4
0.6
0.8
Figure I. Voronoi diagram for .M ((),0), (L,
The weight (V) exactly corresponds to the area of the region in the Voronoi diagram that corresponds to y. So for the alternatives in Figure 1, have that (10 0)) 0.375, w.M( (M,((1,) 0)) 0.25, and u. (M,((1 1)) = 0.375.
Definition 4.3. For any dataset D over alternatives M, the weighted average win rate alternative æ M is wAWRp(x)) ΣyEM (M) > >y).
Now suppose M' = U (0.9, 1)}, i.e. M' contains clone of the alternative (1, 1). The Voronoi diagram of M'
Theorem 4.4. For any dataset over alternatives M, the reward function 12 from Equation (2) satisfies the system of equations
shown in Figure 2.
We now have that w.v ((0, 0.239875, and (M, = 0.025, wxre ((0.9, 0.395125. Therefore, the introduction of the approximate clone x' (0.9, 1) caused the weight of be split between and z', and the weights of the other al- ternatives only changed by small amount. Furthermore, because annotator preferences are continuous, we also have that pD(a', y) PD(r, y) for any alternative y. Therefore,
0.34, us w. ((1,))
wAWRD(I) = XF wAWR(x) VI M,,,,
(1, to
2
where w wAWR(E) = = WM(
3 20
The proof of Theorem 4.4 can be found Appendix Similar to the regularized MIE, major consequence is that
6

---

Clone-Robust AT Alignment
the order in which the alternatives are ranked in the weighted MLE is the same as the order in which the alternatives are ranked by weighted average win rate.
where v/1 and v/2 to be the unique elements of (U)) and proju respectively. In other words, in this case [(y1, 1/2) is simply approximating [(v1, 3/2) using the clos- est alternatives in M. Therefore, Theorem 4.6 shows that the weighted ML.E also has a natural interpretation as an approximate MLE solution that only depends on M through the projection function projM-
Corollary 4.5. For any dataset 'D over alternatives M
and any I, E M. (x) 2 (o()) if and only if wAWRD(()) 2 wAWRD(1).
Interpretation as an MLE Approximation
One interpretation of the weighted MLE is that the function (r(r) approximates what the regularized MLE objective would be if the dataset M contained every alternative in the entire alternative space S. Approximating the regularized MLE objective for the entire alternative space S is natural goal when the algorithm only is given information about subset of alternatives M C S. The MLE objective for the entire alternative space S can be written as
5. Case Study
Although contributions are primarily theoretical, we supplement our results with synthetic case study that high- lights an instance where the weighted MLE is more robust than the standard regularized MLE under diverse prefer- ences. This case study moves our theory closer to practice in a few different ways. First, L.LMs typically generate the responses seen by human annotators, and so our case study considers textual responses generated by the gpt-4o-mini model. Note that this means each alternative has only one data point, unlike our theoretical results where we assume sufficiently many comparisons for every pair of alternatives. This better represents what happens in practice when each pair of responses is newly generated by an L.LM the point when an annotator is asked to report a preference (Bai et al., 2022). LLMs have also been shown be effective as im- plicit computational models of humans (Horton, 2023), and we therefore use an LLM as stand-in in for human annotators with diverse preferences rather than assuming humans make decisions using a BTL model.
(5()) = [S] /
/
((y1. v2) 2 yES
rly)
V1.225
(3)
where the log likelihood term for the comparisons between y) and y/2 is defined as
(u(
C(y1,32) 122) = -PD(y: > 3/2) log
(r(m)) + er(va)
Theorem 4.6 (proven in Appendix H) shows how the weighted MLE objective can be written with the same struc- ture as Equation (3) using using the projm function to approximate the unknown quantities.
Theorem 4.6. The weighted MLE objective function can be writlen in terms of S and the projection function projM as:
In the case study, our goal is to train a reward function which evaluates answers to a single question: 'Describe Paris'. We use OpenAl's gpt-4o-mini model both generate textual descriptions of Paris and to simulate human annotators with diverse preferences. We consider population with three types of annotators, each of which attach a different amount of importance to seeing the topics of food, art, and romance mentioned in a description of Paris. We then construct two preference datasets for this population, one which includes additional approximate clones ("Clones") and one which does not ("Original'). More details on the annotator popu- lation and the dataset generation process can be found in Appendix I.
-1
/
A
/ 7 (y)dy.
fD(r)
C(y1,32)dd) 92)dd:dyy +
Sp
2|5]
yES
where we define the estimated log likelihood of two alterna- tives yı 1/2 ESan as
1
D(y,, (roj)((9))1 (12)
e r(=1)
x Σ
po( > 12)log
(a())) + (=(a))
21 EprojM (11) P2EprojM (12)
and we define the estimated reward squared of an alterna- tive y S as
We approximate both the standard MLE algorithm and the weighted MLE algorithm using neural networks. Each neu- ral network takes as input a context vector and outputs a reward value. To generate the context vectors, we use Ope- nAI's (ext-embeddinng---oee 3-small model extract embedding vectors from the textual descriptions of Paris. More details on the neural network training process can be found in Ap- pendix G. We run each algorithm on both datasets described in the previous paragraph and observe how the reward func- tion output by the algorithm changes across datasets. To visualize the change reward function, we evaluate each
P()) = (v)) xxprojxx())
Σ
r(a)3.
For further intuition about Theorem 4.6, note that when y1 and y/2 both have a unique closest alternative in M (i.c. (11)) = (p2)))((2)) 1), then
C(yi, = po(us > 1/2 log
(r(o() erii)
7

---

Clone-Robust AT Alignment
reward function on all of the alternatives, and then plot the mean reward for three types of answers (food, art, and romance), with enror bars corresponding to the sample stan- dard deviations.
Weighted MLE
Clones Original
0.04
Reward
Figure 3 shows how the standard MLE algorithm performs on these two datasets. Training dataset 'Original' leads to romance being the topic with the highest reward. However, training on dataset 'Clones' leads to art being the topie with the highest reward. The fact that the topic with the highest reward changes with the addition of clones highlights the lack of robustness of the MLE. Furthermore, we note that both datasets contain a relatively large amount of data, and therefore this noticeable change in the reward function can- not be attributed only to variance in the data generation or training processes.
0.00
0.02
-0.04
art
romance
food
Topics
Figure 4. Results for the weighted MLE: The yellow points show the average value of the weighted MLE reward function for dif- ferent topics when trained on dataset 'Original', The blue points show the same but when trained on dataset 'Clones'. The rewards the three topics do not change significantly, demonstrating the robustness of the weighted MI.E.
MLE
Clones Original
0.04
where clones cause a problem for the standard RLHF meth- ods, but does not imply any conclusions regarding how frequent pervasive clones may be in practice. For ex- ample, may not expect LLM-generated answers to the vanilla prompt 'describe Paris' to have such stark deviation into categories; rather, answers may be more balanced. The realized impact of approximate clones will also of course de- pend on the preferences of the annotator population. Further research could evaluate how often clones appear in practice and characterize the types of annotator populations which cause clones to be problem.
0.02
Reward
0.00
-0.0
0.04
food
Topics
Figure 3. Results for the MLE:
The yellow points show the average value the MLE reward function for different topies when trained on dataset 'Original'. The blue points show the same but when trained on dataset 'Clones'. In the presence of clones, the rewards for both art and romance change significantly, showing that the MLE is not robust to clones.
In theoretical results, assume that have sufficient comparisons between each pair of alternatives. This as- sumption may be unrealistic if the alternatives are answers generated by LLMs as in our case study, as then each re- sponse would only be involved in one comparison. The case study suggests a potential solution to this problem, which is to first cluster the original alternatives based on common features (or context) form meta-alternatives. There could still exist approximate clones among these meta-alternatives: however, each meta-alternative would likely have larger number of comparisons. One potential question for future work is to explore how different clustering schemes affect the robustness of both the MLE algorithm and the weighted MLE algorithm.
Figure 4 shows the same results for the weighted MLE algorithm. Recall that the weighted MLE algorithm requires a choice of S, which is the set of all possible alternatives. In the experiment shown in Figure 4, we chose S to be the unit cube in the high-ding nigh-dimensional context space. We also show that similar results hold for other choices of S in Appendix As shown in Figure 4, the average value for the weighted MLE of each of the different categories is roughly the same for both the dataset without clones and the dataset with clones. This shows that the weighted MLE is robust to the presence of clones, which aligns with the theoretical results of Section 4.
Even if each alternative is involved in multiple comparisons, it could be interesting to relax the assumption that we have sufficient comparisons between each pair of alternatives. As mentioned in Section 1, one choice for the alternatives in RLHF is the question/answer pairs. In this case, we would expect the dataset to only include comparisons between al- ternatives (question/ansse hansworr///wwerrrre pairs) where the question is the same, Although this would not exactly match the assump- tions in our theory, we expect that similar theoretical results
6. Discussion
In this section we discuss some limitations of our findings and outline potential directions for future research.
There is ample opportunity for further empirical research. Our case study is meant to highlight a specific instance
8

---

Clone-Robust AT Alignment
would hold with regard to robustness approximate clones.
ing from corrupted human feedback. arXiv preprint arXiv:2406 2024.
Finally, we used simple weighting scheme in Definition 4.1 to balance the objective function when the observed alternatives are not evenly distributed over the entire alter- native space. However, this is not the unique weighting scheme that can achieve this desired result. One direction for future work is to experiment with different weighting schemes see which perform the best in practice.
Chakraborty, S., Qiu, J., Yuan, H., Koppel, A., Huang, F, Manocha, D., Bedi, A. S., and Wang, M. Maxmin- rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint 24xi::2402.0822, 2024.
Conitzer, V., Freedman, R., Heitzig, J., Holliday, W. H., Ja- cobs, B. M., Lambert, N., Mossé, M., Pacuit, E., Russell, S., Schoelkopf, H., et al. Social choice should guide ai alignment dealing with diverse human feedback. arXiv preprint arXiv:2404. 2404: 2024.
Impact Statement
The focus of the paper is Al alignment, field whose ultimate goal is to make Al more beneficial. We acknowl- edge that, as with any work on Al alignment, there could be unforeseen negative consequences; further study is needed before our methods can be deployed.
Dai, J. and Fleisig, E. Mapping social choice theory to rlhf.
arXiv preprint arXiv:2404.1300 2024.
Dwork, C. Differential privacy. In International colloquium automata, languages, and programming, pp. 1-12. Springer, 2006.
Acknowledgements
This work was partially supported by the National Science Foundation under grants I[S-2147187 and IIS-2229881; by the Office of Naval Research under grants N00014-24- 2704 and N00014-25-1-2153. and by grant from the Coop- erative AI Foundation. Zhang and Schiffer were supported by an NSF Graduate Research Fellowship.
Elkind, E.,, Faliszewski, P., and Slinko, Cloning in elec- tions. In Proceedings the AAAI Conference on Artificial Intelligence, volume 24, pp. 768 768-773, 2010.
Elkind, E., Faliszewski, P., and Slinko, A. Clone structures in voters' preferences. In Proceedings of the 13th ACM conference on electronic commerce, pp. 496-513, 2012.
References
Anwar, U., Saparov, A., Rando, J., Paleka, D., Turpin, M., Hase. P., Lubana, E. S., Jenner, E., Casper, S., Sour- but, O., et al. Foundational challenges in assuring align- ment and safety of large language models. arXiv preprint arXiv:2404.00m 09932, 2024.
Ge, L.,, Halpern, D., Micha, E., Procaccia, A. D., Shapira, I. Vorobeychik, Y., and Wu, J. Axioms for ai alignment from human feedback. arXiv preprint arXiv:2405. 14754, 2024.
Horton, J. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research, 2023.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Mandal, D.,, Nika, A., Kamalaruban, P., Singla, A., and Radanović, G. Corruption robust offline reinforce- ment learning with human feedback. arXiv preprint arXiv:2402.06734 2024.
Berker, R. E., Casacuberta, S., Robinson, I., Ong, C., Conitzer, V., and Elkind, From independence of clones to composition consistency: A hierarchy of barriers to strategic nomination. arXiv preprint arXiv:2502. 16973, 2025.
OpenAL. Chatgpt. httpp:://ottt //tp///pp....//////////// cocamai..com/aa com/ index/idx////// chatgpt/, November 2022. Accessed: 2024-12-04.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
Berriaud, D. and Wattenhofer, R. Clone-resistant weights in metrie spaces: framework for handling redundancy bias. arXiv preprint arXiv:2502.03576. 2025.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(34)):3243345, 1952.
Park, C., Liu, M., Kong, D., Zhang, K., and Ozdaglar, A. E. Rlhf from heterogeneous feedback via personalization and preference aggregation. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024.
Bukharin, A., Hong, I., Jiang, H., Li, Z., Zhang, Q., Zhang, Z., and Zhao, T. Robust reinforcement learn-

---

Clone-Robust AI Alignment
Poddar, S., Wan, Y., Ivison, H., Gupta, A., and Jaques, N. Personalizing reinforcement learning from human feed- back with variational preference learning, arXiv preprint arXiv:2408. 10075, 2024.
Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. Distributional preference learning: Understanding and accounting for hidden context in rlhf. arXiv preprint 17X0::2312.08358, 2023.
Sorensen, T., Moore, J., Fisher, J., Gordon, M., Mireshghal- lah, N., Rytting, C. M., Ye, A., Jiang, L., Lu, X., Dziri, N., et al. A roadmap pluralistic alignment. arXiv preprint 240.:24000.000700 2024.
Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.0405. 2024.
Tideman, T. N. Independence of clones criterion for voting rules. Social Choice and Welfare, 40)):15-2066, 1987.
Xu, W., Dong, S., Lu, X., Lam, G., Wen, Z., and Van Roy, B. Rlhf and iia: Perverse incentives. arXiv preprint arXiv:2312.01057, 2023.
Zhang, X., Zhang, X., Loh, P.-..,, and Liang, Y. On the iden- tifiability of mixtures ranking models. arXiv preprint a3X1v:2011.13132,, 2022.
Zhong, H., Deng, Z., Su, W. J., Wu, Z. S., and Zhang, L. Provable multi-party reinforcement learning with diverse human feodback. arXiv preprint arXiv:2403.05006. 2024.
Zhu, B., Jordan, M., and Jiao, J. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pp.
PMLR, 2023.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: /9090083,, 2019.
10

---

Clone-Robust AI Alignment
A. Additional Related Work Details
In this section, we provide a more comprehensive comparison to several works that are especially related to ours. Like us, Xu et al. (2023) are concerned about the performance of current RLHF algorithms in the presence of duplicates. Like us, they show that there are simple models for human preferences under which the standard RLHF algorithms perform badly, Unlike us, their results are for specific class of models they call dichotomy models. In such models, there are two types of messages and two types of individuals, and each type of individual has reward 1 for one type of message and reward 0 for the other. Their main results also focus on whree-way comparisons, while our results deal with pairwise comparisons (which are standard in current RLHF algorithms).
We were inspired by Siththaranjan et al. (2023), who show that standard RLHF implicitly aggregates over hidden context according to Borda count. We build off their work to show that standard RLHF algorithms are not robust to approximate clones. Like Siththaranjan et al. (2023), we assume that humans have diverse preferences, but unlike them, we do not summarize these preferences by hidden context. Rather, we directly model populations with different reward functions. Note that when refer to 'context' in our paper, are referring to the underlying context of alternatives, not of annotators as in Siththaranjan et al. (2023). Like us, Siththaranjan et al. (2023) give an impossibility result for RLHF algorithms in the presence of diverse preferences. They show that every RLHF algorithm fails to exactly recover the mean reward function for some population, while our result shows that every RLHF algorithm does arbitrarily badly at finding the mean reward function for some populations.
Chakraborty et al. (2024) also evaluate the efficacy of standard RLHF algorithms when there are diverse human preferences and give an impossibility result for when standard RLHF outputs a single reward function. However, their impossibility result is of different form they bound the gap between the optimal policy overall and the optimal policy for subpopulation by the sum of total variation distances between preference distributions of subpopulations. By contrast, our impossibility result states that for any RLHF algorithm, there exists population such that the distance between the RLHF algorithm output and the mean rewards of the population is arbitrarily large.
Finally, we also note that because we study RLHF with diverse populations, this work is inherently related to the study of pluralistic alignment of AI systems, see e.g. the work of Sorensen et al. (2024) and Anwar et al. (2024).
B. Proof of Theorem 2.3
We will prove the desired result by contradiction.
We will show the desired result for any C' 2 12))(2) which will imply the desired result for all C > 0. For any C 2 log²(()) (2), define k := e 12VC, Consider the following two populations, each of which consists of two types of annotators which are equally prevalent and two alternatives a and b. The reward of each type of voter for each alternative in each population are shown in the two tables below.
Population 1
Population 2
Type 1 (50%)
Type 2 (50%)
Type (50%)
Type 2 (50%)
r(a)
0
0
r(a)
0
0
r(b)
log(2)
log(2) (2)
r(b)
log(k)
log(n + 4) - log (2k - 1)
Table I. Rewards for each annotator type in each population.
For population 1,
1
e
1
e
1
Pr(a > b)
+
2 0
(2)
2 e0 + Jog(2)
3"
For population 2,
1
e
e
Pr(a b)
2
+ clog(x)
e + 1og
1
1
1
2
I
1+ k
2 1
k+4
2k-1
1
2x
2
14 K
2
3x + 3

---

Clone-Robust AI Alignment
2x-1
2
+
2x+2
6
1
3
Therefore, for both populations the probability that alternative is preferred alternative is the same (1/3). This implies that based query data, impossible to distinguish between these two populations. Now consider arbitrary algorithm ALG that when given data such that is preferred to with probability 1/3, ALG outputs reward function y We want to show that
(2) + (z)
2
min Σ OER
(x))
a) sC
(4)
2
ZEM
cannot hold for both populations. If Equation (4) holds for population 1, then
((r((a) + a)) + (log(2) - -r (b) -a)²) s C,
min aER
which implies that there exists some a E R such that
(a) SVC
and
12gg() - (b)) - a
In order for the two equations above to both hold, must have that
(a) - (b) < log(2) + 2VC.
(5)
Similarly, if Equation (4) holds for population 2, then
(PP(a)+ aa)² (log(s) log(n + +4) log 12 12)
min aEk
sc
2
which implies that
(a) SVC vC
and
log(k) + log(x: x + 4) log(2x - (b) -0 S VVC.
2
In order for the two above equations to hold, we must have that
log(k) + log(x i + 4) log((: (2x - 1) + (p) <
2
In order for this equation to hold, we must have that
Pa) - p(()) log(k) + log(s (N + 4) - -logg22 - 1) - 2
log(k) + log( + 44 - -log(s - 1/2) - log(2) 2
2
log(k) - loo(2)
-2C
2
12V0 log(2)
2/C
2 4VC log(2)
(6)
2
However, for 2 log?(2), Equations (5) and (6) cannot both hold, and therefore we have contradiction.
12

---

Clone-Robust AT Alignment
C. Adapting Independence of Clones
In this section, we provide further details and justification for our definition of independence of clones in the RLHF setting.
C.1. Independence of Clones in Traditional Social Choice
In traditional social choice, independence of clones (Tideman, 1987) is desirable characteristic of voting rules which intuitively states that the winner of an election remains the same when duplicates of candidates are added to the candidate pool. More specifically, classic voting theory considers settings with a set of n voters (denoted N) and m candidates (denoted M). Each voter then provides a full ranking over the M candidates. A voting rule takes as input the set of rankings and outputs a single candidate. A subset K E M of candidates is a set of clones if no voter ranks any candidate in M \ K between any two candidates in K. Finally, a voting rule is Independent of Clones if and only if the following two properties hold. First, a candidate in M K is output by the voting rule if and only if that same candidate is output by the voting rule after climinating any candidate in K. Second, candidate in K is output by the voting rule if and only if some other member of K' is also output by the voting rule after eliminating any candidate in K.
C.2. Additional Justification for RLHF Definition
This section provides further justification for how we adapt the traditional independence of clones definition for RLHE, Here, we focus on how develop a reasonable definition for exact independence of clones. In the next section, we will then explain why we further consider robustness to approximate clones.
Informally, we say that an RLHF algorithm satisfies exact independent of clones if adding new alternatives that are clones of existing alternatives does not change the reward function that is output by the RLHF algorithm. There are a few major differences between the definition of independence of clones in the traditional setting and in RLHF. First, while traditional independence of clones guarantees that the winning alternative does not change, the RLHF version of independence of clones instead guarantees that the reward funetion does not change, which is stronger notion. This because traditional voting theory the focus is on the winning alternative, while in RLHF we care about the reward function over all of the alternatives. Second, the imput to an RLHF algorithm consists of query results over pairs of alternatives, rather than full rankings from every voter. Therefore, the definition of clone from traditional social choice does not carry over. Instead, we will define a clone in RLHF as a new alternative with the same context as an existing alternative, which implies that every voter has the same reward for the new alternative as for the existing one. Finally, it is generally assumed that an RLHF algorithm has access to noisy observations regarding human preferences, rather than the true rewards of each voter for each alternative. Due to randomness, it may be the case that two alternatives for which all voters have the exact same value may still look different in the dataset of query results given as input to the RLHF algorithm.
Therefore, we will say that an RLHF algorithm is independent of clones if when the empirical pairwise win rates are the same as the true pairwise win rates, then the reward function output by the algorithm is unchanged when clone is added. Note that when the dataset contains sufficiently many queries, the empirical pairwise win rates will approximately equal the true pairwise win rates by the law of large numbers. Further note that it is not necessarily desirable that the reward function is unchanged when a clone is added if the empirical pairwise win rates are not close to the true win rates. This is because in this case, there is value in generating a larger dataset, and therefore it is no longer true that adding a clone adds no new information.
C.3. Exact Independence of Clones
In this section, we formally present our definition of exact independence of clones and explain why our work primarily focuses on robustness to approximate clones. The formal definition of exact independence of clones is below. Definition C.1 (Exact Independence of Clones). An RLHF algorithm ALG satisfies independence of clones if the following holds. Consider set of alternatives [m + 1] such that the context of alternative m is the same as the context of alternative m + 1. Let D, and DD2 be representative datasets over the alternative sets [m]] and [m fm++1 1] respectively. Let ri ALG(P)). and r2 = AD2( Then 72(m + 1)= r2(m) and for all E [m.) (1) ri))).
We note that independence of clones as defined in Definition C.1 is very weak guarantee, as two alternatives are only clones if their contexts are exactly equal and reward functions only need to remain unchanged when there is sufficient data. Even so, as a result of the equivalence between the regularized MLE for RLHF and Borda count as discussed in Section 2.2, the regularized MLE algorithm does not satisfy Definition C. We formally prove this result in Appendix D. However, because we have the context of the alternatives (as defined in Section 2), we can casily adapt the regularized MLE to satisfy independence of clones by a simple pre- proppecessiig step. Recall that two alternatives have the same context if and only if
13

---

Clone-Robust AT Alignment
they clones. Therefore, we can combine the data of any alternatives with the same context in order to remove clones. Note that the regularized MLE cannot be made satisfy robustness of approximate clones with preprocessing. because Definition 3.1 must hold for any > 0.
As further motivation for moving beyond exact independence of clones, RLHF queries often ask annotators to compare textual responses generated by L.Ms,, where is unlikely that an exact response will be duplicated. Therefore, it is more realistic in RLHF to consider robustness to approximate clones. For example, an approximate clone of a textual response may substitute an adjective for its synonym, or use a slightly different grammar structure.
D. Proof of Theorem 3.2
Proof. Let M = {a, b, e) and M' = ba b, d.). where ||e- d' O. Note that M and M' differ only by c', and el is an exact clone of an alternative e E M. Suppose that D is generated by querying population which consists of three types of individuals, where each type is represented by BTL model.
Alternative r(a)
Type (40%)
Type 2 (30%)
Type 3 (30%)
In(100)
In(10)
In(1)
r(b)
In(10)
In(1)
In(100)
r(c)
In(1)
In(100)
In(10)
Table Representation of the population that generated D. The proportions of cach type are indicated in the header row, while the rewards associated with each type for every alternative are presented in the matrix.
Suppose further that D' is generated from the same population after cloning alternative c, and can be represented by the following:
Alternative
Type (40%)
Type (30%)
Type (30%)
r(a)
In(100)
In(10)
In(1)
r(b)
In(10)
In(1)
In(100)
r(c)
In(1)
In (1010)
In(10)
r(c')
In(1)
In(100)
In(10)
Define the Borda Count of an alternative given the total set of alternatives M as Borda Count winner of M to be the E M with the highest Borda Count. Further let f2D and GD be the regularized MLE estimators for these two datasets. By Theorem 2.5, fr (zx > pD (y) iff BC(x) > BC(y), and similarly for fP'.
- y), and the
To prove the theorem, therefore suffices to show that the Borda Count winner of MM not the same as the Borda Count winner in M. To see why, observe that if the Borda Count winner of M' is xr and the Borda Count winner of M is y. then it must be that & (x) GP' (x) or RD (y) pP (y) (or both). If ft (x) < f.P (I), then can choose 8 such that FP' (z) (x) 8 > O. Then because ||c c' O, there is
for which (P(()) FD' (x)l ô, which implies
that the regularized MLE is not robust to approximate clones.
remains to be shown that the Borda Count winner of M' is not the same as the Borda Count winner in M. Let represent the proportion the population that is type and let vtt,, æ) be the value of type i for alternative æ. For any two alternatives x, y M, the win percentage over y is
((te.a))
p² y)
(l(a))) + co(t))))
The following table gives BC(x, for every E M. Note that in this table, alternative is the Borda Count winner.
Similarly, the following table gives BC(x, M') for every E M'. In this table, alternative b is the Borda Count winner. We have therefore shown that the Borda Count winner of M' is not the same as the Borda Count winner in M, which proves the theorem.
14

---

Clone-Robus AI Alignment
p' (a > a)
p²(r >- 6)
(x > c)
BC(a,
a
0.50
0.64
0.45
1.59
b 0.36
0.50
0.64
1.50
c
0.55
0.36
0.50
1.41
Table Each row table represents an alternative z. The first three columns compute p (ar > y) for each alternative y. The column gives the Borda Count of alternative which is the sum of the first three columns.
p (z > a)
p" (æz b)
p" (x c)
(x c')
BC(x, M)
0.50
0.64
0.45
0.45
2.04
b
0.36
0.50
0.64
0.64
2.14
c
0.55
0.36
0.50
0.50
1.91
d'
0.55
0.36
0.50
0.50
1.91
E. Proof of Theorem 4.4
We begin with the following lemma, which we will use multiple times throughout the proof.
Lemma E.1. For an arbitrary set T E R., ler w + R+ such that EyET w(y) = 1. Ler E R+ and for any T1,32 let plan > 122 & [0, 1]. Define f == arg min, f(r), where
(x((1)
f(r) = E
((2)))(()))p() > x2) log er(zi) + cr(ors)
+ Σ w(yr(())
FTET
NET
Then strongly convex with parameter X Xmn w(x). Therefore, I has a unique global minimum r", and for any v,
f(r) f(r)) 2 m ||r r- 121
is strictly convex in r(x)) and r(x2) as shown in (Siththaranjan et al., 2023). Furthermore, any 0, because w(x) > æ E T, we have that 2/ Σzer wx))r(x)² is strongly convex in r(x) for all x T. Finally, adding a strongly convex function and a strictly convex function results in strongly convex function.
Proof. Note that the function log
)
=erl))
Proof of Theorem 4.4. Recall that w arg min, fD(r), where
((()))
[o(r)= Σ wM( 12) (t(ws) + er(o2) IIE
Σ wm(x)r(x))
+
By Lemma E.I, Fo will be the solution the equation setting the gradient of fD(r) to O. Define (a)) Item The gradient of fr(r) is the following for x E M.
@fr(r)
Ör(x)
ca(a) (w)) Σ wm(())m(() (pols= v) er(o) + terla) + pplyy > x) erly) + er(z))
e'(r)
Σ wM(()um(()) (pole> v) (1 (1-- (x())
(x)
(v(v) + (1-po( > b)) (*(x) + (
>wm(())(zx)
= -x) + Σ (w( 12 - - v).
(8)
Also note that
(x) -2) wM(v)po(>>a) wM(y)
(M 0.
(9)
2 2
15

---

Clone-Robust AT Alignment
Dividing Equation (8) by (x) and equating to 0 gives that
1 0 (r)
(2) 0 w (x)
(z) + Σ wMly)o - (2())) - uM(U)po(>> v)
yofr
(x) + Σ
uM(y)o(P (()) wM(v)pp( y)
Equation (9)
WEM
(x) + Σ (M(U)o(( (v)) Σ
wM(y)pp(E>0 y)
yEM
PEM
A (z) + Σ
( - - wAWRo(z)
yEM
Rearranging the sides, we have that
=AWRo(z)=AP((A) Σ
yEM
as desired.
F. Proof of Theorem 2.5
Proof. This result follows exactly the same steps as in the proof of Theorem 4.4, except substituting (M(()) = 1 for all x e M.
G. Proof of Theorem 4.2
We begin with the following lemma.
Lemma G.I. For arbitrary set T E Rd. let T - R4 such that ΣyET w(y) 1. Let X E R+ and for any
12 T let pl11 > x22) E 10, 1). Define i == arg min, f(r), where
(x12)
f(r): = Σ
((2)))(12)pm > log er(()) + er(oa)]
+ Σ w(y)r(y)2
yET
Then
0 0< < 11
2
and all y E T
2 2+1
IF(1) - V xw((y)
Proof. If ryy) 1 for all y then f(r) < 11++, 2. This implies that f(f) S 1 1+ 2. Furthermore, note that no finite values of can make (r) = 0 due to the log terms, and therefore ʃ(î) > 0.
Finally, using that f(F) < 1 + 2/ it must be the case that 3 (y))(yy) < 1 + }/. Rearranging terms gives the desired bound on [+()).
Now we are ready to prove Theorem 4.2.
Proof of Theorem 4.2. For any of alternatives M and any 8 > 0, will define c:= 62, where is constant relative to that depends M which we will defer the definition of until after Equations (10) and (11). Now consider any x' such that ||æ' ==|2 < for some & E M. Define M = MUM2' (x'), and suppose D and D' are representative datasets on M and M' respectively. Our goal is to bound the difference between Fh and pD', which are defined follows.
16

---

Clone-Robust AT Alignment
F arg min, (r) where
()()))
fo(r)= =-
E ww(an)wmess)ool > x2) et(z)) + er(ez)
+ Σ
wM(z)r(1)?
ZEM
pD' = arg min, I (r),
where
[r(()) --
wMr (x)) x x2) log (x(z)) + er(wz) 21.72E.M
E
ex(x1s
A
Σ (xr(z))(x) MEM'
2
The objective functions fD(r) and (r) differ in three ways. First, they have different weights (w versus w.r).. Second, they use different sets of alternatives (M versus M'). Third, they have different comparison probability functions (pd versus por). In order to compare i to r2 we will first change the weight functions to match, then change the sets of alternatives match, and then finally change the comparison probability functions to match. Therefore, we will consider two intermediate reward functions r1 and 12 that the optimal reward functions for objectives fi and f2 respectively, Informally, fi can be viewed being the same fo except that it uses different weights that correspond to the weights from fD. Similarly, f2 can be viewed as being the same as SD except it uses different comparison probability function that corresponds to the comparison probability function from fD. Finally, fi and f2 can be viewed as being the same except that they use the sets M and M' respectively.
Next we formally define r1, fi and r2, f2. Define the function wi .M- - IR as wh (y) wxr( for all y E M \ {œ) and wi (x) w (x) (x'))x). In other words, wi is the same as wave except allocates all of the probability of both x and r' to alternative sc Then can define rı
== arg min, fi(r), where
er(z1)
[(()) -
Σ 11 MM
wi (12)) x2) log
+ Σ wi (z)r(z)² SEM
(t(Z1) + er(z2)
Define p2(21 (x1 > = po > y) for all y E M. other words, P2 is the same as pD except that every comparison involving x' has the same probability as the corresponding comparison involving æ. Then
x2) for all T1, 32 E M, and define p2 > = = pDly > 2) and p2 p2(2' > y) = po( >
define 72 as
r2 == arg min, fe(r), where
f2(1)
Σ EI
x2)
(a(a1)
A 2
Σ
w. (2)(2))
(t((r) + erlon)
Lemma G.2 bounds the difference between Po and rı, Lemma G.3 bounds the difference between ra and r2. and Lemma G.4 bounds the difference between r2 and :D'.. Using the triangle inequality, we can combine these three lemmas to get that for any y E M,
(e) - wh (w)1 = (v) rı(1) + (2())) r2(v) + r2(u) - # (v) < 11 (v) -r((y) + (n((y) - r2(x)] + (rz(y) - 1W (z)l
< O(ve)
[Lemmas G.2, G.3, G.4]
(10)
and that
12 (z) - 12 (x) - 12 (x')l
< 1 (z) (())-r2(>)) (r2(z') 50' (x')l
< 2 (x) (2)))) + +12(20)) - - (x')
[Lemma G.3]
((ve).
[Lemma G.4]
(1))
17

---

Clone-Robust AI Alignment
If define e in the definition of e such that the O(ve) from the above equations is bounded by 6, then this exactly shows the desired result of Theorem 4.2.
The rest of the proof will focus on stating and proving Lemmas G.2, G.3, and G.4.
Lemma G.2. For all E M.
(2) - rr) - O(ve).
(12)
Proof. First, we note that we can choose e be sufficiently small and assume WLOG that is the closest alternative M to x'. This implies that
wm(x) S wi (x) wM(() O(e),
and that for every E EM) \ {x}
wM( - O S wi (v) = wm(w).
The previous equations together imply that for any y M,
(w)(w)) - w1 (v)) S O(e).
(13)
Note that we also have that for any a, b I.
ea
log
2 -2 |a| + 18))
(14)
c" + e
Therefore, for any satisfying |r(y)] s C for all y .M for some constant C, we have that
(r) - fo(r)
Σ
(o) 22) log
en(er)
A Σ (wi(y) (x())r(z))
(r(s)) er(oal
2
VEM
rt(z))
d
Σ (-O(e)un(<)))
0()))((22))))((mm (-0()aa(ann-0))mm(aaaa)))
x2) 12)log log (x(2)) cr(oz)
Σ (()r(x))
2
yEM
Σ (wn)(un (11) + wi (1a)))po( x2 (2+ +rai))+ + Σ O()r(ss)
DEM
-Oe)).
(15)
Similarly, for any satisfying |r(y)] - C for all y E M and some constant C, we have that
(i(r) - (r(r) < O(e).
(16)
By Lemma G.I, we have for all E M that (r S mw and (16), we have that for constants C1, C2.
2+1 and Ini |r)())
2+A Xwy (3)
Therefore, by Equations (15)
fi(ri)-f(rr)) fo(ri) -Cie
(17)
and
[o(12)) - -1((22) 2 -Ce.
(18)
Now we will show that
fo(r)) fo(t2) (C1 O2)c.
(19)
Suppose Equation (19) does not hold. Then
- -(n))-((FPP) = (Ji(r)) - fo(r))) + (f((ri))(((tt)) + (
-CE (C1 + O2)2 - Oze
[Eqs (18) and (17))
(20)
18

---

Clone-Robust AT Alignment
This is contradiction with the definition of ri, and therefore Equation (19) must hold.
We will now the contrapositive of Equation (19) to prove the desired result. Lemma E.1 implies that for any such that there exists y .M where Ir(y) - (v)) V Amin.cm (z)(=))
(C++C2le
must have
fo(r) fD((w) > (C1 + C2)c
This combined with Equation (19) implies that for all v. Ir(y) -ww(w) S desired result.
(C++C2)e
O(VE), which is the
AmiD.ca 1MM(F)
Lemma G.3. For all E M. ri (w) = r2(u). Furthermore, r2(x') r2(2))
Proof. Define (a) =
6a Ttc"
Theorem 4.4 implies that ri is the solution to the set of equations
(N)pp((< > v) = (2) + Σ (2())o(r1<))- (r)))) A2 M.
(21)
NEM
yEM
which by definition of p2 equivalent to being the solution to this set of equations:
Σ
wi()))2(2>>>> > v) Arr(z) + Σ
wi wi(v)o(1(<)) - rv))
E.
(22)
BEM
BEM
Similarly, 12 the solution the set of equations
Σ BEM
(y) v) Ar2(z)+ + Σ ( -2())) E'
(23)
BEMP
Because p2(x' > y) p2(x x > y), we see that the LHS of Equation (23) is the same for x and for x'. Therefore, r2(x) and r2(x") satisfy the same equation (which only has one solution), and therefore
(2 = (2(')
Now we will show that this implies that r1) r2( for all y E M. By definition of wi, for all we have that
Σ
wi (u)p2(2 > y) w1(x)p2(=>> x) +
Σ
wi(v)p2(>>> > v)
BEM
DEMiA
(w + uM (2'))222 x)
Σ
(.M(v)p222>>> v)
yEMia
Σ (u)ppp) 2).
(24)
NEA
Because r2(x) r2(2"), we also have that for z,
Xr2(z)+ Σ (wr(w)o(rs())
VEM
=rr((=) + (x) (r)))))))))))))))))))))))))))))) (am()(l)))) r2(x)))
Σ (Mr(w)o(ra(s) - r2(y))
- r2())) w + -2((x))
Σ (Mr(y)o(r2(3) r2(2))
vEM'(u.'')
= Arz(2>) + (w we (4x))))rrz() rz)x)) + Σ
(w - r2(v))
NEM'HZ.,''
-rr(()) + (z) r2(x))
Σ
(r())d(2(()) -ra)2( r2(1))
19

---

Clone-Robust AT Alignment
= Arz(2)) + Σ wi (1)(((22(2)) rr2())-
(25)
8EM
Combining Equations (23), (24), and (25) gives that satisfies the equation
Σ
(1)pp22 > y) Xr2(z)
Σ w (1))((2((2) - r2(1)) 42 E M.
M.
9EM
9EM
This means that 72 and rı are solutions to the same set of equations, and therefore r2(v) = ri(v) for all E M.
Finally, we can relate r2 to Ew
Lemma G.4. For all E M', Fi (v) r2(1) S O(ve).
Proof. Because D and D' are both representative datasets, by definition of p2, for all æ1, 22 M we have that P2 > (2) - pa x2) PD (11 > x22). Therefore, only way in which f2(r) and (D)(r) differ is that P2(y > x') - PD ( .)) # (y x²) for all y E M (and same for pa(a' y)). Because DD and D' are both representative data sets, the preferences are Lipschitz continuous, and the BTL model is Lipschitz continuous, we have for some constant C₁ > 0 that
|po( v) PD (x' > 8)1 pp (x v) PD (x' > m)l Cic.
and
pDD -) PD (y > z')l lp (y > z) PD. (y > x')l Cie.
Next, recall that by Lemma G.I, r and r2 both are bounded by V have that for some constant C2 > 0,
2+A Am
For any satisfying |r(v]] s
2+A
we also
(r) f2(r)
Σ (x') ((p > v) p2(2's ())a(((()) r())) (porly x3) p2(y D) (r(y) r(x')))
DEM
Σ (x'))re (v) ((pDr > v) - po(x ())))(())) r(v)) - z))o (r(v) - r(x')))
DEM
S -Ci Σ (00 r(x')))
NEM
< Cie
Σ
wae (x') (2 In(v)) +
[Equation (14)
WEM
SCe
Σ
wav (a' (w)
44
2+
Awsme
WEM
< C2e.
(26)
By the same logic we also have that
f2(r) - (r) & Cze. Note that Equations (26) and (27) hold for 2' and r = r2 by Lemma G.I, Next we will show that
(27)
fe( - f2(12) 2C2e.
(28)
Assume Equation (28) does not hold. Then have that
(r - (12) = (r - fe(r )) + (fe(1 - fe(r2)) + (fe(rz) - (rz)
> Oze 2C2e Cze
0.
20

---

Clone-Robust AT Alignment
However, this is contradiction because by definition of w we must have that /D (r (ID((1)) S. Therefore, have shown that Equation (28) holds.
Lemma E.1 implies that for any r such that there exists E M' such that (r(v) - r-2())) > / have that
2C25 Aminca Wxr(z)) we must
(2 f2(r2) 2C2c.
The contrapositive of the previous statement combined with Equation (28) implies that for all y, must have
2C2e
In (v) -r2(x) s AminzeM uM((z) = O(v)).
H. Proof of Theorem 4.6 Proof.
r(())
))
So(r)
E F1.F2EM
(M(())ee > w2) log (r(()) + ex(xz))
+
Σ wm(x)r(x)
xEM
Σ
L2 anroolal dyi
(
Iag dyz 32E5 (
po(er t x2) x log (rlas) + ex(12)
(((r))
S
VIES [p1))
S
#1.F2EM
ImeproiM(u) (U) dy |proju (3))
r(x)²
[
REM
yES
Σ
(eleprossrivv)) [proju((1))]
( (Lazeproian(22) pa(zi x x2) x log
(Tproju((>)) (12)
(r(as) + erlors)
))) dypzyz
S
Vi VII
1
Σ
r(x)² dy
2 2
[proiu(y]]
yES
MEE
((rroii(()) (x1) |proi (v2) pr(zi > x2) log (ा(ف)) + (r(za)
ea(an)
Σ
dyzdya
S
Ui U
#iEproj (v1) 22 (v2))
[S] 2
1
(proi
r(ar2
dy
yES
xEprol (w)
{ ("())dy.
fo(r)=
I(y2, v2)dy1dy2
1812
J1.9268
2|S
WES
I. Additional Case Study Details
L.I. Annotator Types
We use OpenAl's gpt-4o-mini model to simulate human annotators with diverse preferences. There are three populations of annotators, each of which attach different amount of importance to seeing 'food', 'art', and 'romance' mentioned in a description of Paris. Each the annotator types and their prevalence the overall population are shown in the table below.
I.2. Case Study Preference Dataset Generation
In the dataset 'Original', each preference datum is generated in the following manner. First, we randomly select (with replacement) two categories out of ['ooo'', 'art', 'romance'] and ask OpenAl's gpt-4o-min model to generate one response each the prompt 'briefly describe Paris with focus [category]'. Next, we randomly select one annotator type with
21

---

Clone-Robust AI Alignment
Alternative
Type (40%)
Type 2 (30%)
Type (30%)
Romance
'really like"
"mildly interested in" "very uninterested 'really like'
'very uninterested in" 'really like' "mildly interested in'
Art Food
"mildly interested in' "very uninterested in'
Table Representation of annotator population. The prevalence of each type of annotator is indicated in the header row, while the importance that each type of annotator attaches to cach category is presented in the matrix.
probability equal their prevalence the overall population. We then instruct the model to act as the selected annotator type and report a preference between the two responses. The dataset with cloned alternative generated in the same manner, except randomly select (with replacement) of simply the first three. Note that the annotator types remain the same. Each dataset contains 1000 data points. Full prompts and sample responses for all interactions with gpt-4o-mini are given below.
categories out of ['food', 'art', 'romance', "restaurants" ] instead
Prompt
You are helpful assistant. Briefly describe Paris with focus food, and highlight activity.
Response
Paris, often referred to as the "City of Light," is renowned for its rich culinary heritage and vibrant food scene. The city is paradise for food lovers, offering a delightful blend of traditional French cuisine and innovative gastronomy. Iconic dishes such as coq au vin, bouillabaisse. and ratatouille can be found in charming bistros and upscale restaurants alike. Paris is also famous for its patisseries, where you can indulge in exquisite pastries like macarons, éclairs, and the decadent croissant.
Prompt
You are helpful assistant.
Briefly describe Paris with focus on art, and highlight one activity.
Response
Paris, often regarded as the capital of art and culture, a city steeped in history and creativity. It boasts renowned museums like the Louvre, which houses masterpieces such as the Mona Lisa and the Venus de Milo, as well as the Musée d' OrOay, famous for its collection Impressionist works. The city's streets filled with iconic artistic sites, from the Montmartre district, once home artists like Picasso and Van Gogh, contemporary galleries the Marais.
22

---

Clone-Robust AT Alignment
Prompt
You are visiting Paris. You really like food, are mildly interested romance, and are very uninterested in art. You are given two answers to the question, 'Describe Paris'. Which do you prefer? Respond with only (1) or (2).
(I) Paris, often referred to as the "City Light," renowned for its rich culinary heritage and vibrant food scene. The city is paradise for food lovers, offering a delightful blend of traditional French cuisine and innovative gastronomy. Iconic dishes such coq au vin, bouillabaisse, and ratatouille can be found in charming bistros and upscale restaurants alike. Paris is also famous for its patisseries, where you can indulge in exquisite pastries like macarons, éclairs, and the decadent croissant.
(2) Paris, often regarded as the capital of art and culture, is city steeped in history and creativity. It boasts renowned museums like the Louvre, which houses masterpieces such as the Mona Lisa and the Venus de Milo, well as the Musée d' Orsay, famous for its collection of Impressionist works. The city's streets are filled with iconic artistic sites, from the Montmartre district, once home to artists like Picasso and Van Gogh, to contemporary galleries the Marais.
Response
I.3. Extracting Context and Neural Network Training
We approximate both the standard MLE algorithm and the weighted MLE algorithm using neural networks. Each neural network takes as input context vector and outputs a reward value. To generate the context vectors, we first use OpenAl's cexteemeedding---sine model to extract embedding vectors from the textual descriptions of Paris in each dataset. For each dataset, we then conduct principal component analysis (PCA) on the associated embedding vectors using the PCA class from sklearn.ded derraamddcooooitiionnn. setting the number of components equal to 1500. The PCA-transformed data is then given as input to the neural networks. The neural network we used had layers, and each hidden layer had size 32 (and output size 1). For training used the Adam optimizer with learning rate of 10-4 and batch size of 512 and implemented the training using PyTorc. We trained the neural networks for 500 steps and then averaged the results over 20 runs to form the graphs included in this paper.
L.4. Weight Estimation
To estimate the weights for the weighted MLE, sampled 100000 random vectors within the chosen space S and computed the weight for each of the alternatives in the dataset. In Figure we chose S to be the unit cube in the context vector space. Another natural choice of S is any vector such that every coordinate of that vector is within a factor of 2 of the corresponding coordinate in one of the vectors in the observed set of alternatives. By restricting to vectors with coordinates close to those in the observed set of alternatives, this is potentially a more suitable choice for S than choosing S to the entire unit eube. This is because unit cube may include many alternatives that are not reasonable answers to the question. While there are arguments for many different choices of S, also analyze this choice of S to demonstrate the robustness of the weighted MLE. The results for this choice of S are shown below in Figure 5, and the weighted MLE remains robust to the addition of approximate clones.
23

---

Clone-Robust AI Alignment
Weighted MLE
Clores Original
0.04
Reward
0.02
0.00
-0.02
-0.04
art
romance
food
Topics
Figure Results the weighted MLE when S chosen as any vector such that every coordinate within factor of one of the observed coordinates. The yellow points show the average value of the weighted MLE reward function for different topics when trained on dataset 'Original'. The blue points show same when trained on dataset 'Clones'. both cases, reward function has highest value for romance, demonstrating the robustness of the weighted MLE.
24


Reviews:
[{"review_id": "g70y7FN7ZD", "reviewer": "Reviewer_MLTh", "summary": "This paper mainly focus on the problem of unbalanced input datasets in RLHF, which is caused by adversarial manipulation or inadvertent repetition. The key motivation is to make RLHF robust towards non uniformly distributed datasets. Inspired by social choice theory, they introduced robustness to approximate clones, a desirable property of RLHF algorithms which requires that adding near-duplicate alternatives does not significantly change the learned reward function.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and feedback! Below we address your specific questions:\n\n> As an extension, if n>2 in Theorem 2.3, will the proof still stand?\n\nYes, the results do extend for $n > 2$. Learning a reward function can only become information theoretically harder for $n > 2$ because there are more variables that need to be estimated. Therefore, the same impossibility result holds for larger $n$ as well.\n\n> \"Fundamentally, the mandate of RLHF algorithms is to solve a preference aggregation problem\". Could you please share your insight on why it is related to \"aggregation\"? In my opinion, it is just to generalize preference even given conflict preferences existed.\n\nIn the field of social choice, preference aggregation refers to the problem of taking voter preferences that are potentially conflicting, and *aggregating* them into a single output that captures the voters' preferences. Mapping this onto RLHF, the annotators may have conflicting preferences, and the goal is to find a single reward function that captures the annotators' preferences. Therefore, \"aggregation\" in RLHF refers to the aggregation of all of the individual annotator preferences into one single reward function that can be used for LLM tuning.\n\n\n\n> One question towards \"near duplicate\": Sometimes these near duplicates are necessary in practice, because some topic do have a heavy weight among the entire topic distribution.\n\nThis is a great point, and we thought about this a lot while writing the paper. In practice, we definitely expect that some topics would be more heavily weighted in the overall topic distribution. In fact, having near duplicates in the dataset can be good, because more comparisons involving two common response topics gives a better estimate of the annotators' preferences between these two topics. However, we believe it is undesirable for an RLHF algorithm to reward or punish a topic based on its weight in the topic distribution. Instead, we want the final reward value for an answer topic to only depend on the annotators' preferences for that topic (and not on the topic distribution in the data set). In summary, we don't think that near duplicates are inherently bad -- we just want the final reward function to be stable regardless of the topic distribution for the observed preference data set.", "claims_and_evidence": "They show that the standard RLHF algorithm based on regularized maximum likelihood estimation (MLE) fails to satisfy this property. In contrast, a weighted MLE can alleviate this problem.", "methods_and_evaluation_criteria": "The voting rule robust towards adding duplicates of alternatives is important. The authors alims this as \"satisfy independence of clones\". Informally, a voting rule is independent of clones if after adding an alternative a', which is equivalent to another alternative a, the output of the voting rule does not change. In RLHF, there do exist \"approximate clones\", namely two alternatives which are very close by a given distance metric and for which all annotators have very similar values, where the distance metric depends on the nature of the alternatives. \n\nThe proposed new training objective is simple, via, down-weighting alternatives that are similar to other alternatives (and therefore provide less new information) and up-weighting alternatives that are different than other alternatives (and therefore provide more new information)", "theoretical_claims": "As an extension, if n>2 in Theorem 2.3, will the proof still stand?", "experimental_designs_or_analyses": "The experiment is a bit toy setting.", "relation_to_broader_scientific_literature": "This work can give some inspiration to the practical reward model training pipeline optimization, especially handling the data unbalance.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The presentation is great.\nThe motivation towards studying unbalanced dataset is clearly explained with examples provided.\nThere is one question towards the necessity of removing near duplicates.", "other_comments_or_suggestions": "NA", "questions_for_authors": "\"Fundamentally, the mandate of RLHF algorithms is to solve a preference aggregation problem\". Could you please share your insight on why it is related to \"aggregation\"? In my opinion, it is just to generalize preference even given conflict preferences existed.\n\nOne question towards \"near duplicate\": Sometimes these near duplicates are necessary in practice, because some topic do have a heavy weight among the entire topic distribution.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gi8APCxjFX", "reviewer": "Reviewer_R2Xe", "summary": "The paper addresses a key challenge in LLM alignment, that of making sure that the RLHF model is unbiased. Specifically, authors show that the distribution of data used to train the model can have a significant impact on how the RLHF model behaves, and as such it is prone to intentional or unintentional biases. This usually happens when the dataset is biased (authors talk about duplicate or near duplicate pieces of data).\n\nThe authors propose the concept of \"clone-robustness\" meaning the the model training is immune to data clones present in the dataset. This is done by using a weighted MLE algorithm that assigns lower weights to alternative data points that are similar to other data points.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! Below we address your specific questions:\n\n> Have you tested Weighted MLE on real-world RLHF datasets? Please share results if so. How does Weighted MLE perform across different types of RLHF tasks (e.g., factual questions, multi-turn dialogue)? Could you provide real-world examples where approximate clones have distorted RLHF in deployed AI systems?\n\nThis paper is primarily a theoretical contribution to the field of RLHF, as we propose a theoretical property and prove results about that property for current and new algorithms. While we included the case study as a proof-of-concept for the proposed weighted MLE, more intensive experiments using the weighted MLE in different applications is beyond the scope of our paper. We do however think this is a very important topic for future work, and we hope to provide sufficient motivation and information for practical applications of the weighted MLE in the future. \n\n\n> Key gaps are\n> - lack of discussion on weighing schemes\n> - lack of comparison with alternative solutions for clone robustness\n\nBecause robustness to clones in the context of RLHF was introduced in this paper, there is no previous work that has alternative solutions for this problem. There are voting rules from traditional social choice that are independent of clones, and these could potentially be adapted to the RLHF setting. However, such solutions would be very different than the current MLE estimation and may be less practical. We briefly mention other weighting schemes at the end of the discussion, specifically that the $w(\\cdot)$ function can potentially be replaced with other functions that upweight more unique alternatives. We are happy to include more discussion on both of these points in the final paper.", "claims_and_evidence": "Yes, most claims are well supported, for example\n\n\"standard RLHF is not robust to clones\"\n1. Theoretical proof is provided via theorem 3.2\n1. The case study in section 5 also supports the claim above\n\n\"weighted MLE ensures robustness to approximate clones\"\n1. Theoretical proof is provided via theorem 4.2\n1. Also backed up by the case study in section 5\n\nCouple of unproven claims:\n\nGeneralizability of Weighted MLE\n1. Tested only in a narrow set of scenarios (describe Paris), which makes the real world effectiveness unclear\n2. A specific weighing scheme is discussed, but its unclear whether alternative schemes might perform better or worse", "methods_and_evaluation_criteria": "The methods used (theoretical proof + experimental validation) sound correct for the problem at hand. However, the scope of the study is quite narrow, which means that it is unclear how the weighted MLE would perform on a diverse set of scenarios.\n\nThe study simulates human preferences using an LLM as an annotator. An LLM might not accurately capture human's method of annotations. Similarly, the diversity of annotator preferences is modeled using fixed categories.\n\nOne key missing piece is the lack of validation of the weighted methods against some standard RLHF datasets from industry or academia.", "theoretical_claims": "No, I did not look at the theoretical proofs in detail.", "experimental_designs_or_analyses": "### Strengths\n\n1. Directly Tests Clone Robustness: The controlled introduction of cloned responses effectively isolates the impact of near-duplicates on RLHF training.\n1. Uses Embedding-Based Similarity Measures: The use of OpenAI’s text-embedding-3-small model to represent response similarities is a reasonable approximation of how RLHF embeddings work in real-world AI training.\n1. Quantitative Analysis with Win Rate Comparisons: The study evaluates how reward scores shift across different topic categories (food, art, romance), with error bars for variance.\n\n### Weaknesses\n\n1. Use of LLMs as Annotators Instead of Human Feedback: The study simulates human preference data using an LLM (GPT-4o-mini) instead of actual human annotators.\n1. Narrow Scope of Dataset (Single Prompt: “Describe Paris”): The experiment only tests one question, meaning results may not generalize across different types of RLHF tasks (e.g., safety-critical alignment, long-form reasoning)\n1. No Evaluation on Real-World RLHF Datasets: The datasets used are synthetic, and the paper does not benchmark performance on real RLHF datasets.\n1. Limited Statistical Analysis: The paper visually presents reward differences (e.g., Figure 3 & 4) but does not conduct rigorous statistical significance tests.\n1. No Robustness Testing for Weighted MLE with Alternative Weighting Schemes: The experiment only evaluates one version of Weighted MLE with a fixed weighting function.", "relation_to_broader_scientific_literature": "Tideman (1987):  This paper builds upon Tideman's introduction of \"independence of clones\" by adapting it from voting theory to RLHF, proposing a new algorithm (weighted MLE) that ensures robustness to approximate clones.\n\nElkind et al. (2010, 2012): The authors extend Elkind et al.'s studies on manipulation through cloning by applying similar ideas to RLHF, highlighting vulnerabilities in standard RLHF algorithms and motivating their new robust solution.\n\nConitzer et al. (2024): This paper elaborates on Conitzer et al.'s suggestion that independence of clones is important for RLHF, providing concrete examples and a new algorithm addressing this issue.\n\nXu et al. (2023): The authors share Xu et al.'s concern about duplicates in RLHF datasets but extend their results beyond dichotomy models and three-way comparisons to standard pairwise comparisons.\n\nSiththaranjan et al. (2023): This paper extends Siththaranjan et al.'s insights about regularized MLE and average win rates by proving a stronger theoretical relationship, and builds upon their impossibility result for diverse preferences by showing an even stronger impossibility result.", "essential_references_not_discussed": "Christiano et al. (2017) : \"Deep reinforcement learning from human preferences\" (NeurIPS 2017): This foundational paper introduced RLHF, demonstrating that LLMs can learn from pairwise human preferences. The authors critique RLHF's vulnerability to dataset biases, but do not cite Christiano et al. (2017), where these concerns first emerged.", "other_strengths_and_weaknesses": "The key contributions / strengths are\n1. formal definition of robustness to approximate clones\n1. proving standard MLE is not clone robust\n1. Proposing and proving that weighted MLE works\n\nKey gaps are\n1. lack of validation / benchmarking against real world RLHF\n2. lack of discussion on weighing schemes\n3. lack of comparison with alternative solutions for clone robustness", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Have you tested Weighted MLE on real-world RLHF datasets? Please share results if so.\n1. How does Weighted MLE perform across different types of RLHF tasks (e.g., factual questions, multi-turn dialogue)?\n1. Could you provide real-world examples where approximate clones have distorted RLHF in deployed AI systems?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "6o59lml3pu", "reviewer": "Reviewer_CwdH", "summary": "The paper considers axiomatic AI alignment. More precisely, the paper is about Reinforcement Learning with human feedback (RLHF). As motivated by Conitzer et al. (2024), consistency with respect to clones is an interesting property for RLHF algorithms. In this paper, each alternative is identified with its context, i.e., some $d$-dimensional real vector that lies in some infinite set of finite volume $S$. Roughly speaking, the goal is to aggregate the utility functions of individual voters (annotators) into a collective utility function. The catch is that we are only given a finite set $M\\subseteq S$, query samples of the voter preferences over $M$, and want this aggregated utility function to be clone-proof.\nEach annotator has a utility function $r$ over the alternatives in $M$, and whenever asked to compare two alternatives, says that they prefer alternative $a$ over $b$ with probability $ e ^ ( r(a) ) /  [ e ^ ( r(a) ) + e ^ ( r(b) ) ] $, known as the Bradley-Terry (BTL) model. \nQueries are (two)-alternative subsets of the form \\{a,b\\}, and we denote by $Q$ a (multi)-set of queries where each possible query is contained at least once. For each query $q \\in Q$, we choose an annotator uniformly at random and obtain a sample of the annotator’s preference for that query. In total, we obtain a random dataset $D$ consisting of all queries in $Q$ and the respective responses. Given the query set $Q$ and the resulting dataset $D$, the goal is to find a social utility function $r$ that best models the collective preference of the annotators.\n\nThe first result [Theorem 2.3] states that no algorithm can always output a collective preference $r$ that is equal to the mean reward function, i.e., E_i[r_i]. This impossibility already holds true for two voters. Next [Theorem 2.5], the authors establish a relation between the average win rate and the regularized MLE, which is defined as the utility function $r^D = r$ minimizing \n\n0.5 \\lambda  \\sum_{x\\in M} r(x)^2  - \\sum_{x_1,x_2\\in M} p_D(x1>x_2) log( e^(r(x_1) / [ e^(r (x_1) + e^( r (x_2) ) ] ).\n\nThe authors then introduce their core axiom, robustness to approximate clones, which says that for each $\\delta>0$ there should be an epsilon>0 such that adding an alternative that is of distance at most epsilon to an existing alternative changes the reward functions by all alternatives by at most delta, and the utility for the almost-clone is also similar to the original utility of the alternative that it almost clones. Clearly, by being Borda-like, MLE violates robustness to approximate clones [Theorem 3.2], as it even fails robustness to precise clones.\n\nTo counteract this phenomenon, the authors introduce a Voronoi-approach to define a distribution $w_D$ over $D$: each alternative in $S$ is projected to its closest candidate(s) in $D$ and then, for the weight of $y\\in \\mathcal M$, $w_D(y)$ is calculated using the volume of the set of all $x\\in S$ that are projected onto $y$ [Definition 4.1]. \n\nThe main result of the authors is that the reweighted MLE satisfies robustness to approximate clones [Theorem 4.2].\nAnalogously to Thm 2.5,  the authors present an identity involving the weighted average win rate and the weighted  MLE estimator [Theorem 4.4], implying that the ordering induced by the weighted MLE estimator is equal to the weighted average win rate of the alternatives [Corollary 4.5]. Then, the authors argue that the weighted MLE approach is an approximation of the MLE over S [Theorem 4.6] (This essentially boils down to swapping some sums and integrals).\nThe authors then discuss a synthetic case study that illustrates the susceptibility of MLE to clones, while the weighted MLE unsurprisingly performs better.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments and feedback! Below we address your specific questions:\n\n> It is unclear to me why it is desirable that the average win rate is the sum of the empirical win rate and the reward function itself. Is there some motivation for this?\n\nThe original motivation for this result (Theorem 2.5) was to give better intuition for why the ranking induced by the MLE estimator is the same as the Borda Count ranking (which follows directly from the relationship between the estimated win rate and the empirical win rate). A secondary benefit of Theorem 2.5 is that it gives an additional motivation for the MLE solution. A natural alternative to using MLE estimation is to find the reward function that best matches the empirical win rates; Theorem 2.5 says that the BTL MLE solution is (almost) equivalent to matching empirical win rates.\n\n\n\n>In social choice, there is a strengthening of independence of clones called composition consistency. Roughly speaking, if one replaces an alternative with a component, then the probability of the alternative gets distributed to the alternatives in the component is precisely proportional to how the probabilities would have been if the component was viewed in isolation. Have you thought about whether this notion could make sense in this context?\n\n\nMapping composition consistency to RLHF is difficult for a few reasons. In social choice, composition consistency is defined by running a social choice function twice on a set of rankings: once with the clone sets grouped together and once on the winning clone set. This is possible because social choice functions map rankings (over alternatives) to alternatives, so they can be applied iteratively. In RLHF, however, we are mapping pairwise comparisons to a reward function, so we would have a type error if we tried to use an analogous definition.\n\nComposition consistency also may not be a desirable property for RLHF. In contrast to traditional social choice, RLHF assigns a reward to every alternative, and therefore the order of alternatives does not matter as much as the actual rewards. Specifically, for a set of approximate clones, we do not care about the order of the clones (like in composition consistency), but instead we want that the clones all have similar reward values. In fact, for any output reward function that is continuous, any set of approximate clones will also have approximately the same reward as desired. Therefore, composition consistency does not seem desirable/necessary in RLHF as long as the output reward function is continuous.\n\n> One criticism is that currently, the precise properties of $S$ are not defined in the preliminaries, there is no mention of being Borel-measurable or of finite(!) volume, which confused me for quite a bit and seems to be crucial for the paper if I am not mistaken. In several parts, the paper could also be more non-expert friendly, e.g., KL divergence is mentioned but not defined on page 4.\n\n\nThank you for pointing this out. We certainly agree that $S$ must be finite and measurable, and we will add more specific properties of $S$ in the model section for the final version of the paper. We will also make sure to define any more niche technical terms used throughout.\n\n> ... there is a paper by Berriaud and Wattenhofer that considers this version of the problem...\n\nThank you for bringing this to our attention! We will be sure to discuss this work in the final version of our paper.", "claims_and_evidence": "Claims are of a mathematical nature and supported by proofs", "methods_and_evaluation_criteria": "The proposed weighted MLE is natural and makes sense for the problem at hand.", "theoretical_claims": "The proofs of Theorem 2.3 and Theorem 4.6 are sound.", "experimental_designs_or_analyses": "I did not.", "relation_to_broader_scientific_literature": "The study of consistency w.r.t. to clones is indeed an important task for AI alignment, see e.g., Conitzer et al. \nThe proposed approach using Voronoi diagrams seems sensible and suits this task", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The findings of the paper would be a good addition to the conference. I like the question asked by the authors and think it is both quite natural and interesting. Further, I am a big fan, of recent works combining social choice theory and AI alignment and think that this paper provides a very interesting twist on it. For the most part, the paper is also well written, and nice to read, even for a reader who is not necessarily well versed in the literature on LLMs. \n\nOne criticism is that currently, the precise properties of $S$ are not defined in the preliminaries, there is no mention of $S$ being Borel-measurable or of finite(!) volume, which confused me for quite a bit and seems to be crucial for the paper if I am not mistaken. In several parts, the paper could also be more non-expert friendly, e.g., KL divergence is mentioned but not defined on page 4.", "other_comments_or_suggestions": "The authors consider weighting alternatives in \\mathbb R^d within a set $S$ of bounded volume. When moving from $S$ to the unbounded $\\mathbb R^d$, there is a paper by Berriaud and Wattenhofer that considers this version of the problem. Most notably, I see some similarities between their axiom 6 (alpha-locality under the addition of clones) and the here-considered Definition 3.1 (robustness to approximate clones). Further, both papers use the idea of utilizing projections for integrals and their formula for the $g$ function in Section $4$ seems to resemble the approach taken in this paper for the weighted MLE. \nAs the paper came out after the ICML deadline this is of course concurrent work and not relevant to the judgment of the paper, however, in case of acceptance, the paper should still compare to it (to keep the academic record complete).\n\n## update after rebuttal\n\nThank you for the nice rebuttal!", "questions_for_authors": "1. It is unclear to me why it is desirable that the average win rate is the sum of the empirical win rate and the reward function itself. Is there some motivation for this?\n2. In social choice, there is a strengthening of independence of clones called composition consistency. Roughly speaking, if one replaces an alternative with a component, then the probability of the alternative gets distributed to the alternatives in the component is precisely proportional to how the probabilities would have been if the component was viewed in isolation. Have you thought about whether this notion could make sense in this context?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "frSwrv4AWC", "reviewer": "Reviewer_b1RX", "summary": "The paper evaluate the robustness of current RLHF algorithms in the presence of approximate clones and develop RLHF algorithms to enhance the robustness regarding this.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! Below we address your specific questions:\n\n> why the focus of the paper is RLHF rather than reward model itself?\n\nOur paper focuses on the step of the RLHF pipeline that takes as input a  preference dataset and outputs a reward model. In RLHF, this reward model is then used to fine-tune LLMs. However, the focus of our paper (robustness to approximate clones) is a desirable property of the process of going from preference data to a reward model. We highlight implications for RLHF because RLHF is currently a very common application of reward modeling from preference data, but our results could be used in any application of reward modeling.\n\n> I am not quiet familiar with the area. Could you please explain briefly how the theoretical framework can guide the practical implementation of training real LLMs?\n\nTo use our framework to train LLMs in practice, one would execute the following two steps. First, learn a reward function by computing the weighted MLE on the given pairwise comparisons; this is the part addressed (and potentially improved) by our paper. The second step is a standard policy optimization step, where (roughly speaking) a given base LLM is fine-tuned to maximize the expected reward from step 1, for example, through proximal policy optimization (PPO). Since the second step simply takes as input a base LLM and a reward model, the output of our weighted MLE can be directly plugged into this heavily studied pipeline.", "claims_and_evidence": "Yes, I think most of the claims made in the submission clear and convincing. However, the empirical experiments (case study) are not sufficient enough for me.", "methods_and_evaluation_criteria": "See Claims And Evidence part.", "theoretical_claims": "I didn't check all the proofs in detail but the theorems provides seem to be reasonable and sounding.", "experimental_designs_or_analyses": "The paper is mostly about theory. See Claims And Evidence part.", "relation_to_broader_scientific_literature": "The paper provides a good vision for the current understanding of RM/RLHF.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See Questions For Authors", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1.  why the focus of the paper is RLHF rather than reward model itself?\n2.  I am not quiet familiar with the area. Could you please explain briefly how the theoretical framework can guide the practical implementation of training real LLMs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses a key limitation in RLHF systems—their vulnerability to "clone attacks," where repeated or near-duplicate preferences can bias the learned policy. It introduces a theoretically grounded and practically implementable solution using a weighted MLE objective that remains interpretable while being robust to such clones. The paper combines tools from RLHF, social choice theory (notably clone independence), and theoretical learning guarantees. Reviewers agree that the motivation is important, the formulation is elegant, and the results are compelling both in theory and experiment.

The paper addresses a novel and important failure mode in RLHF—vulnerability to clone preferences—through a principled, interpretable weighted MLE framework. Theoretical guarantees and an illustrative case study support its effectiveness. The formulation is clear, well-motivated, and applicable to real-world alignment problems. Despite some limitations in scope and experimental scale, the work makes a clear and meaningful contribution to the field of RLHF and alignment theory, and meets the standards for inclusion at ICML. Please incorporate all the reviewer's feedback in the final version.
Decision: Accept (poster)