Mechanism Design for LLM Fine-tuning with Multiple Reward Models
Authors: Haoran Sun
Yurong Chen
Siwei Wang
Xu Chu
Wei Chen
Xiaotie Deng
Venue: NeurIPS 2025 poster ()
Keywords: Mechanism Design
RLHF
LLMs
Incentive Compatibility
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Fine-tuning large language models (LLMs) to aggregate multiple preferences has attracted considerable research attention. With aggregation algorithms advancing, a potential economic scenario arises where fine-tuning services are provided to agents with different preferences. In this context, agents may benefit from strategically misreporting their preferences, but this could harm the aggregation performance. This paper addresses such incentive issues by framing it as a mechanism design problem: an LLM provider determines the fine-tuning objective (training rule) and the pricing scheme (payment rule) for agents. We primarily focus on training rules that maximize social welfare subject to certain regularizations, referred to as SW-Max rules. First, we show that under most circumstances, truthful reporting is sub-optimal with simply a SW-Max rule, thereby highlighting the necessity of payments. Second, we extend the VCG payment to implement SW-Max rules in dominant-strategy incentive compatibility (DSIC). We characterize sufficient conditions for payment equivalence and derive the necessary conditions for a payment rule to implement a SW-Max rule in DSIC and other principles. Third, we demonstrate that our mechanism is approximately DSIC with perturbed input, showcasing its robustness against the inevitable errors in real-world applications. Experiments on real LLM training results further confirm the practical implications of our results.

Mechanism Design for LLM Fine-tuning with Multiple Reward Models
Haoran Sun', Yurong Chenr, Wang Xu Chu', Wei Chen', Xiaotie Deng!*
CFCS, School of Computer Science, Peking University
2 Inria, École Normale Supérieure, PSL Research University
3 Microsoft Research Asia sunhaoram0301@:: tku..uu..uuuuu......mm.mmm edu. en, yurong chen@inria.ff: {chu_xu, (taotie)Opku edg.uu.dducco edu. cn, {siweiwang, weic)@microsoot.. com,
Abstract
Fine-tuning large language models (LLMs) to aggregate multiple preferences has attracted considerable research attention. With aggregation algorithms advancing, a potential economic scenario arises where fine-tuning services are provided to agents with different preferences. In this context, agents may benefit from strategically misreporting their preferences, but this could harm the aggregation performance. This paper addresses such incentive issues by framing it as a mechanism design problem: an LLM provider determines the fine-tuning objective (training rule) and the pricing scheme (payment rule) for agents. We primarily focus on training rules that maximize social welfare subject to certain regularizations, referred to as SW-Max rules. First, we show that under most circumstances, truthful reporting is sub-optimal with simply SW-Max rule, thereby highlighting necessity payments. Second, we extend the VCG payment to implement SW-Max rules in dominant-strategy incentive compatibility (DSIC). We characterize sufficient conditions for payment equivalence and derive the necessary conditions for payment rule to implement a SW-Max rule in DSIC and other principles. Third, demonstrate that mechanism approximately DSIC with perturbed input, showcasing its robustness against the inevitable emrors in real-world applications. Experiments real LLM training results further confirm the practical implications our results.
Introduction
As large language models (LLMs) [61 74] become increasingly widespread, seeking models that not only possess general capabilities but also align with their individual values. Reinforcement Learning from Human Feedback (RLHF) [14 (57 has emerged as mainstream approach achieve this alignment, where a reward model guides the reinforcement learning process using feedback signals that reflect human preferences.
However, standard RLHF becomes resource-intensive when catering to diverse preferences. Training separate LLMs for every individual or group within community, each with unique preferences, often impractical due to prohibitive computational costs and potential data privacy concerns. A more feasible alternative is train unified model that reflects collective values while still accom- modating distinct needs. Multiple-Ohjective RLHF (MORLHF) [5] 78], which aims to efficiently integrate multiple preferences into single model, offers a promising avenue for this. Further studies aim to improve MORLHF algorithms from various perspectives, including efficiency [4] 62 70)
accuracy 8 26 63 and fairness
"Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

As these techniques advance, we explore a practical economic scenario: a platform offering fine-tuning service to aggregate diverse preferences from various groups into a single LLM. These "groups"-suc as different departments within company or hospitals in the same city with various specializations-
the same core values but have slightly different focuses. Given these shared values and the high cost of fine-tuning, developing separate L.MSs for each entity is often inefficient. Nevertheless, each group must provide its specific preferences to account for these differing focuses. Finally, the training cost is shared among the groups and can be non-uniform due to their differentiated preferences.
A critical issue in this process is that groups may strategically misreport their preferences to ma- nipulate the aggregate objective for a more favorable outcome. As illustrated in a simplified RLHF framework (see Figure D. a group's true preference (rm,) could be misreported as a polarized one (rm1) to steer the model toward a more desirable outcome. However, this behavior distorts the training objective, resulting in suboptimal model for the overall community. Given the potential profitability of such strategies and the growing economic importance of LLMs, ensuring truthful preference reporting is as critical as the training algorithm itself. We therefore formalize this scenario to study its incentives. Our findings indicate that many commonly used training objectives lead to profitable misreporting strategies. However, we also demonstrate that simple incentive-coppatie cost allocation scheme can incentivize truthful reporting, and under certain conditions, this scheme is uniquely determined.
Specifically, we model this as a mulli-parameter mechanism design problem involving a fine-tuning service provider and multiple groups of agents. The mechanism consists of training rule, which aggregates the reported sizes Wi (representing a group's scale) and preferences from different groups, and a payment rule to determine their respective charges. The fine-tuning process is implemented through RLHF, with reward models representing the groups' preferences. Our focus is on training objectives aimed at maximizing social welfare with regularization constraint, referred to as SW-Max training rules. Our technical contributions, which extend beyond standard mechanism design due to the unique complexities of LLM fine-tuning objectives, are summarized as follows:
1. We show that mechanisms using only SW-Max training rules are vulnerable to profitable pref- erence misreporting (Theorem 4.2 and Theorem 4.3). This finding highlights the need for payment rule to resolve incentive issues.
2. We extend the VCG payment to ensure truthfulness for SW-Max training rules (Proposition 4.4 and further establish the uniqueness of this payment under certain conditions (Theorem 4nn and Corollary 4.. Based on that, we derive necessary conditions for payment rules to implement a SW-Max training rule in more principles (Theorem 4.11)
3. We demonstrate that our mechanism is approximately DSIC in the presence of inpul pertur- bations (Theorem 42).). This finding highlights the robustness of our mechanism against the inevitable measurement errors in real-world applications.
4. Experiments on practical LLM setups empirically validate the existence of profitable misreporting strategies and demonstrate the efficacy of our mechanism in incentivizing truthful reporting (Section
Related Work. Several recent studies have also examined incentive issues in RLHF and LLMs. Duetting et al. (25 proposed a preference aggregation mechanism that satisfies monotonicity with respect to bids; however, their work does not address strategic misreporting of preferences, which is the central challenge we tackle. Other works that consider strategic preference reporting have different focuses, such as implementing truthful rules with KI.-divergence for ad auctions [71], analyzing the implementability of various training rules [59], or modifying the RLHF objective to achieve approximate truthfulness while preserving convergence 10). In contrast, work adopts theoretical perspective to analyze representative training rules, providing a comprehensive understanding of incentive issues in RLHF Specifically, our analysis of payment equivalence helps characterize all possible payment rules that implement a training rule in DSIC.
Our research also connects to classic literature on auction design 52 54] and facility location problems 21 [58) Compared to the classic auction model, we have to consider the necessary regularization term, which makes the training rule (or the allocation rule in the auction) more complicated and prevents vanilla VCG from being applied. facility locations, agents can benefit by misreporting more polarized preference. The idea of such strategy is similar to our model.

---

Training Rule
Report
s(m ASW (AR
RLHF
"o 1:30 ADPIO
Alices' Less Preferred
Alics
Alices Strategic Report
RLHF
Bobs
Intial Model
Alices
Bobs
Alloss Mono Proferodd
Figure I:
An illustration of the incentive issue in LLM preference aggregation. When using basic training rule db in RLHF for two groups (Alices and Bobs), fixing Bobs' report rm2, Alices can gain higher utility by strategically reporting rm * rm, than truthfully reporting rm, = rm On the other hand, we have ASW (0; ri, wi, Oinit) > ASW m,, W. 0mn), which means that such strategic behavior also harms the training objective.
However, due to the complexity of the training rules that aim to catch the LLM fine-tuning scenarios and the normalization constraints of the reward models, the reporting strategies can be more complex. Further, combined with the discretized input spaces of the agents, most of our results cannot be directly derived from existing literature.
Paper Organization. The remainder of the paper is organized follows, Section 2 introduces the necessary preliminaries, and Section 3 formulates the RLHF Game. We then analyze the properties of mechanisms composed of SW-Max training rules and payment rules in Section 4 followed by presentation of our experimental results in Section 5 Finally, Jection[ooffée concluding remarks and discusses potential future research directions.
2 Preliminaries
Large Language Models. In this paper, LLMs are abstracted as stochastic mappings from prompt set, denoted by X, to a probability distribution over sequences of length up to K the output space (25]. Let T represent the set of all tokens, and define T = 0 UU 2 U... .. U 7 as the set of sequences with lengths up K. An L.L parameterized by is function I.MM, : X - A(T"). The space of L.LM parameters is denoted by θ, and it is assumed that the LLM can express any function within this space. Our theoretical model operates on each prompt independently, so we focus on a fixed prompt scenario and omit its notation for simplicity. We denote LLM,(æ) the probability of a sequence æ generated by the model LLMg.
Reward Modeling. In RLHF, reward model is a function rm X' T - R, which maps prompt-response pair to a real number, indicating humans' satisfaction with the response hased on the prompt. Similar to the L.MM case, we focus on fixed prompt scenario, the scalar feedback for a response 2 T Following prior empirical work for RLHF (57 78), we mainly consider two types of normalization constraints for the reward model: (1) The summation of the rewards over 7" is normalized to i.c. Eeer- rm = 1. (2) The maximum of the rewards Furthenmore.
rm(x: represents
also assume that the output rewards are all non-negative, i.e., (m:)) 2 0 for all æ E 7 The set of all reward model functions satisfying these conditions is denoted by R. Unless otherwise specified, the results this paper hold under both normalization schemes.
over T normalized
i.e. max rm(x)
3 Formulation of the RLHF Game
In this section, we present the formal description of the RLHF Game. The game involves one L.M provider and groups of agents, denoted by [n] {1, .,, 21.,,...,7 n). The provider has initial model

---

L with positive probability for all sequences, i.e., L.. (z) > 0 for all x E T" Each group has w, agents who share the same preference represented by a reward model rm, Let R and W S N denote the domains for each group's 's reward model and group size, respectively. The group size w should be an integer, and we assume an upper bound w for W, which is public information. The exact reward model r,, and the size w, are group i's private information. For an agent in group i, the valuation when it receives a model LLM, is denoted by vs v0( rm;), defined as follows.
Definition 3.1. An agent's valuation of model LLMM, its expected reward on the sequences generated by it: v00; rm) (mx = -xgr. (z)
In practice this can be obtained by averaging the reward of the sequences sampled from an LLM. We also discuss the influence of possible errors in this process in Section 4.3
Remark the group size W. We introduce the concept of group size to ensure that our model encompasses a broader range of scenarios. As the scales of different groups may vary, our training objective has to account for this factor to ensure fairesss. Groups are also allowed to over-report their sizes to attain a higher status in fine-tuning. The case u = 1 represents a special scenario where each group consists of exactly one agent and is included in our general model. In certain results, we note that the general model is technically more difficult than the = 1 case.
The provider first announces the mechanism, including a training rule 1o and a payment rule
R x W x θ - O
p R. x M x + R..
Both rules take n reported reward models, n reported sizes, and an initial model as input and output the objective fine-tuned model and each group's payment, respectively. The provider can choose not to charge the users by setting always equal to 0. In this case, the model coincides with most previous work on designing empirical algorithms, where agents' incentives are not considered [ 26 78 82). Specifically, the training rule seeks the model that maximizes a certain objective function OBJ. That is, rh (rm,, wi 0mn) E arg maxage (O; rm, wē 0mn)) with ties broken based on further ordering of v;(0;; (0; rm:)s.
41 63 76
After observing the announced mechanism (v), p). each group reports a reward model, rm; and its group size Wi Based on the reported information, the provider fine-tunes the model and gets the final model with parameter 0mmal = (rm, w, Omit) Each member in the group has access to the fine-tuned model, so the valuation for group i is (,yu(Onnat, rm:). The provider then charges
each group a one-time payment according to the payment rule, pa (rm, w. 0mit). All groups have quasi-linear utilities, i.c., group i's utility is the valuation it attains minus the payment:
Ug (m,, w; tb, P. rm; w/) = 10,0e(Ommaa rm.) - ps (mm, w. Oma)
The groups may strategically report, thus and / do not necessarily equal the true rm and w. The LLM provider's s goal is to achieve its training objective based on the group's s true preferences, taking into account that misreporting may distort the training outcome. To this end, it is crucial to incentivize all groups to report their information truthfully so that the provider has access to the groups' private information. These desiderata for the mechanism are formally defined as follows.
Detinition 3.2. A mechanism (1, p) satisfies dominant-strateey incentive compatibility (DSIC) if Vi, rm; Wi, rm/, w/ mii., W_i, Onii, we have
ui( (m,, it )), (w,, w=:): 1b p, rm,, w.) 2 u. ((m() 1). (wh, v0 -1)) -), rm, D., wt). (DSIC)
Definition 3.3. mechanism (4) p) satisfies individually rationality (IR) if Vi, rm,, Wi, rm W-i. Oui, we have
(m,, m) )), (wi, (i 1/, rm;, ws) 2 0.
(IR)
DSIC means that truthfully reporting the reward model and the group size yields the highest utility for any group, regardless of other groups' reports. IR means that truthfulness always yields non-negative utilities. When a mechanism (v,,)) satisfies DSIC, IR, or both DSIC and IR, we say that the payment rule p implements vp DSIC, IR, or both DSIC and IR. When we say the implementability of training rule, we refer to the property of DSIC.

---

Incentives in the RLHF Game
This section explores incentive design within the RLHF Game framework. Our focus is mainly a set of training rules that aims at maximizing social welfare with regularization, which balances efficiency and fairness and commonly used in practice aggregate various preferences [8] (56..
Denote D := Eg By(a)) f[((#)/ ((())) the divergence between probability distributions and measured by function the formal definition follows.
Definition 4.1 (SW-Max Training Rules). A Social Welfare-Maximizing training rule fine-tunes the model maximize the summation of the groups' valuations subject a regularization measured by f-divergence [3 19 70). Formally, the training objective is
OBJ(O; rú, wi min)) = (O:(0; W7, Oinit) =
Σ w. w, D (LLMM) L
where f is convex on R+ and f(1) O. We use (SW mi, w7 mit) to denote the affine social welfare.
This defines a set of training rules, and the function f includes the most commonly used regularization terms in training model. For example, f (x) Ax log refers to KL-divergence,, f(x) = ^(x 1)² refers to x2 divergence, fxx A/1 refers to total variation. We denote vp E s that /)
belongs to this set.
In the following subsections, we will first establish the necessity of payment rule for SW-Max training rules. Then, we construct DSIC mechanisms for these training rules using affine maximizer payments and demonstrate payment equivalence properties for certain distance measures f. Next, we address the influence of noise input on the DSIC property. Finally, we discuss the efficient implementations of the mechanisms in practice.
4.1 Necessity of Payment Rule
We start by showing that without payment rules, groups have incentives to misreport their preferences under most cireumstances. Our discussion focuses strategies other than simply inflating the group size wt, We assume that for Vrm, प, 0mtt the fine-tuned model 0 = (mm, wi, mai)) satisfies that LLM, (x) > 0 for Vx E T This mainly excludes extreme cases where the outcomes remain largely unchanged regardless of input, which may make the analysis meaningless. Based on this, we comprehensively analyze the relationship between optimal strategy and truthful reporting. We start with two cases with strong intuition.
Theorem 4.2. In the RLHF Game with mechanism (1),p) that 1p E SWS and p = for group i, define 8; == Hrrr = rm,()), E ET*) and rm; := minger- rm((x)):
1. If 8: = 1, truthfully reporting is the optimal strategy regardless of other groups' reports.
2. If 8; 2 2 and rm; > 0, there is a strategy that yields strictly higher utility than truthfully reporting regardless of other groups' reports.
8 an unusual case which group i has the same preference values for all x, resulting in the same valuation for any model 0. In such a case, all strategies bring the same utility and hence are optimal. However, when 8; 2 2 and rm, > 0, group i can report rm, that assigns a lower value to 2 = arg milgeTT- rm,(x)) (and a larger value to æ2 = arg max (m, in summation normalization). By doing so, group pretends to prefer æ less, thereby increasing the likelihood that the resulting fine-tuned model generates the outcomes it prefers more. The condition rm, > ensures that group is not completely uninterested in any x, which is more realistic in practice.
Further, we consider the case that s; 2 2 and rm; O. Since the minimum value is already 0, the strategy above cannot be applied. We need to analyze in more detail how the training results change when one group adjusts its reported preferences. Under certain smoothness conditions of the function f. we derive function ((x) to estimate the gradient of the valuation for group i over the reported
value rm, (az). Based on this function, we show that if ((x) o for some æ, it is always possible to find suitable direction and magnitude to report rm o rm, (ar). allowing group to achieve higher utility. The result is summarized in the following theorem. Due to the complicated form of the function t, we provide detailed version in the Theorem B.2
5

---

Theorem 4.3 (Simplified version of Theorem B.2). In the RLHF Game with mechanism (v. that E Wsw and p = when f is strongly convex and C²- -moohhhh, there exists a function when (n, rm, 1. Omic) # 0 for some it E '," truthfully reporting is not the optimal strategy.
The properties of f stated in Theorem 4.3 are also considered in optimization theory and encompass wide range of divergence measures. Combining Theorem 4.2 and Theorem 4.3 we provide a comprehensive analysis that covers the entire space of 8; and rm,- While the second theorem offers only a sufficient condition for the suboptimality of truthful reporting, we demonstrate in the proof that this condition is highly likely to occur, illustrating the impossibility mechanism that aims to maximize social welfare to incentivize truthfulness without payments.
4.2
Affine Maximizer Payment
After establishing the necessity of payment rules in this scenario, we mainly address two questions in this part:
Given a training rule ih, can we find payment rule p such that the mechanism (v) p) satisfies DSIC? This is the so-called implementability of training rule th.
2. For an implementable training rule vh, can we identify the relationship between the payment rules ps among all DSIC mechanisms (v.p). p).
For the first question, since there is an additional regularization term, we can not directly apply the vanilla VCG payment [I5 34 755 to the SW-Max training rules. To address this problem, we define ASW (0; rm, W7 Oinit). the affine social welfare function that excludes the contribution of group i from the social welfare:
ASW As; Wi, Omni ASW (0; mm, w. 0mi) wyys (0; rm:).
Then, the vanilla VCG payment can be generalized to the following form, which is also known as the affine maximizer payment rule [64] p pAr::
AFF (rm, W7 Omni) = ASW (1b irm -.i, Oinin; rm, W7, Oinit) - ASW ((b(r,, w7 Omitt; rm, W, Oinic). Pa
Following the proof of the classic VCG mechanism, we show that AFF implements SW-Max training rules in both DSIC and IR, implying that truthfully reporting both reward models and group sizes constitutes a dominant Nash Equilibrium under this mechanism.
Proposition 4.4. For any & E sw mechanism (16, 1..p" AFF) satisfies DSIC and IR, and the payment is non-negative.
The availability of the affine maximizer payment derives from the additive property of SW-Max training rules. However, this method does not apply to training rules where the objective function cannot be decomposed into additive components, such as Nash Social Welfare and the fairness- oriented objective defined in MaxMin-RLHH [1I. The implementability of an arbitrary training rule characterized by the concept of cycle monotonicity, which is discussed Section E but not the focus of this paper.
The second question is more general, so we consider the concept of payment equivalence as bridge, which is defined as:
Definition 4.5 (Payment Equivalence). An implementable training rule & satisfies payment equiva- Ience if for any two mechanisms (v,,) and (v'p') satisfying DSIC, there exists a function g; such that for Vrm, E R, wi E W
p,(mi,, ui, Omit) = p:(rm, wi, Omit + ge (rm - 0mn)
Or equivalently, when fixing mis W=, and fuits there is a constant such that p (rm,, wx) p5(mm, wx) + e for all rm, R, ws E W.
Payment equivalence indicates that the only way to modify a mechanism (v,p) to (v', p') while maintaining the property of DSIC is to add term that is independent of i's report to group i's payment function ps. Thus, the payment equivalence of the payment rule that implements it in DSIC. This notion is particularly useful in the case that
sometimes interpreted as the uniqueness

---

we can figure out certain DSIC mechanism (1), p) for và because any other payment rules p' that also implement it in DSIC can be divided into p and an independent part.
In the context of the RLHF Game, the domain of the reward models and group sizes affects payment equivalence. When vi 1, groups only report reward models, with the domain R. containing all normalized reward models m. Since this forms a connected set in Euclidean space, we can apply the result from Nisan et al. (55) show:
Proposition 4.6. When w = 1 is public information, and the agents only report the reward models, all implementable training rules satisfy payment equivalence.
However, when the group size Tē is also a part of the private information for all groups, the domain of the whole private information becomes R x W that is no longer a connected set because W S N To get more meticulous characterization of the property, we define the continuity of training rule.
Definition 4.7 (Continuous Training Rule). A training rule tbb is continuous if for any there exists 8 > 0 such that for any Oinit rm, m'', w and w', maX 12 (w,rm;(at) (x))) S 6, then max [LLMa - LLM (x)) S e, where 0 == (mm, Wi, Omni) and == (mmm', wi 0mma).
0.
The continuity requests that the training outcome be similar if the reported values are similar. This definition is natural, and we identify several continuous SW-Max training rules.
Proposition 4.8. SW-Max training rules with regularizations KL-divergence, fKL(2) = Ar log and x2 divergence, f2(z) 1(x 1)2 (A 0 is constant) are continuous.
Based on the continuity, we show a sufficient condition of payment equivalence for general training rules.
Theorem 4.9. An implementable training rule vp satisfies payment equivalence it is continuous and for Vi, rth rm--,, W. Omis there exists rm; and 8 such that ( rm- (Ws, w--s), 0mi)) for all Wi E W. In the maximum normalization case, rm; must be II.
We provide some intuitions of the theorem. Here, when fixing rm-i, w-i, and mit we can find rm; such that when group reports rm; then the reported w; will not affect the training result, rm; actually serves to connect different wy E W. For SW-Max training rules, we observe that the reward model rm that assigns the same value for all æs, i.e., Yz, rm(æ) 1 for maximum normalization, and rm(2) (x) 1/|T"" for summation normalization, serves the role of rm.. With the continuity of the training rule, this makes the domain of R x W connected in another sense that can also induce payment equivalence. Based on this, we derive the payment equivalence property:
Corollary 4.10. Each continuous training rule vb SW satisfies payment equivalence.
As continuous SW-Max training rule always satisfies payment equivalence. we can establish the relationship between AFF and any other payment rule that implements DSIC. Combined with the inherent property of pAFF, we derive the necessary conditions for payment rule to satisfy more conditions, such as non-negativity and IR.
Theorem 4.11. Given continuous training rule 1é SWS and payment rule p implements it in DSIC: If p is always non-negative, it holds that for all i, rMnl, W. and m
(r(rm, W. 0mi) 2 Pt
AFF FFF(,m W. 0ma).
If p implements 1) in IR, then for any > O and i, there exists rm is w.i. and 0 such that for all rm; and w
P. (m,, w. 0mir) S p. APF (rm, w. 0mn) + te.
This result implies that if we want design payment p to satisfy all these properties, AFF is "lower bound" for and p should be sufficiently close to AFF in some inputs.
4.3 Approximate Valuation
In this part, we study the influence of enrors generated in practice on the incentive property in the RLHF Game. We abstract as an approximate valuation problem [3] Formally, when group reports its reward model rm, the mechanism may not use rm, but rather noisy reward model rm,
7

---

Harmless vs. Humor (3,
Harmless vs. Humer (5, 5)
Harmless vs. Humar (?,
Farthul vs. Summary O,
10
10
100
101
100
10
10
30t
15a
10-3
10.
10
10
12
10
10
10:9
19.9
3.3
Weight
Weight
Weight
Waghh
10.
100
10-1
12.0
10a4
101
15a
10
10
12
0.5
0.5
0.5
0.0
3.0
Weighn 8
Weight F
We ght
Weight P
Valuation
Payment
Utiliy
Figure 2: The simulation result for the mechanism (4), AFF)) on real LLM setup. We the group number = 2, and the group size (W1, W2) for each column is in the title. We report the valuation, the payment, and the utility for group when adopting different reporting parameters and / (defined in Section 5). Truthfully reporting (a and 3 = 1) brings the highest utility for all cases.
as the input. We assume that the noise independently generated and there an underlying joint distribution F- (m)) for the r. This abstraction captures various errors that may occur in practical training. One example is that the calculation of valuation defined in Definition 3.1 requires sampling sequences from I.M,, which may result in a deviation from the true valuation.
When the groups are rational, they could be aware of the noise and consider the influence of that on their utility. For group i with reward model rm, and group size wi, it will computes expected utility U, for reporting (rm/, w!) given by
U,((mm m. :), (w/, wii)); m., w:) = E
Fm (: (mmm)....))) m) u. (rm, (w/, iI w=s); th p. rm, w:).
We consider the case that the noisy input reward models rm, and the reported reward models r,; are close. In that case, we show that when using training rule & E SSW the distance between the true optimal point and the training outcome with noisy input is bounded. Based on that, we calculate the utility of group under the mechanism (3), (.p'' AFF) and derive the approximate incentive compatibility of the mechanism.
Theorem 4.12. Assume that for any noisy input rm generated from F F-))m)), and i [n], there is (ar) -rr (x:) <.
max
#ET
Then, with training rule v E &S (bb.AAT) ensures that each group i can benefit at most 2w;c from misreporting the reward model.
This theoretical result guarantees considerable utility for truthful reporting. Since the maximum gain of misreporting group less than 2wne regardless of the others' reports, groups will tend to truthfully report in cases where finding the optimal strategy and modifying its reward model is costlier than 2w.e.
4.4 Efficient Implementation the Mechanism
At the end of the whole section, we discuss how p AFF can be implemented in practice, as Proposi- tion 4n4 Theorem 4. show that it is "unique" to implement SW-Max training rules DSIC. As defined in Equation I1. we have compute (0(mm_ W Omis) for each i aside from the final model
0" = (rm, Wi, Omn From the definition v - 0m.) == max OBJ(O: (0: em w_ +, 0mi)) finding maximum whole space O requires a whole training process. This results in n additional

---

trainings when we have n groups. To address this problem, we propose two heuristic methods that approximate the payment computation; the core of both take the maximum on constraint space O' instead of the whole space O
Heuristic I: Intermediate Models
During the training to obtain the model (rm, wi Omit)) we usually save intermediate models at different training steps. We can set O' to be these intermediate models. This requires no additional training and maintains payment non-negativity since 0" is also in O', but DSIC not strictly guaranteed as @' depends on group i's report. However, the complex dependency makes strategic manipulation practically difficult.
Heuristic 2: Early-Stopped Training
We can perform early-stopped training to compute the (mm-,,, 0m)) This means that we less powerful @' that is only dependent on rm tē mi-- Since the independence of group i, this preserves DSIC theoretically. However, the payment may be negative as v/(mm ui, 0init) may outperform the early-stopped vi(mm W-i; 0miit in terms of ASW
These heuristics provide practical trade-off: Heuristic 1 offers maximum computational efficiency with relaxed thcoretical guarantees, while Heuristic 2 preserves DSIC with moderate additional cost. From a theory perspective,
derive the following result based on Theorem 4.11
Corollary 4.13, Given a continuous training rule va E S. if the payment rule implements it in DSIC, IR and is always non-negative, then for any > 0, there exists i, rm- W-s and mit such that for all rm; and wi, denote rm- = (rm,, rm-s) and w = (wi, w-i), we have
AFF (rm, mma) S (rmm, 0ma) S PFF (rm, 0 +
This indicates that any payment rule p that satisfies all these properties must closely approximate PAFF in certain inputs. This somewhat showcases a tradeoff between theoretical guarantees and computational efficiency. A more rigorous analysis of the efficiency loss caused by these heuristics or an "impossibility theorem" regarding efficient implementation is left for future work.
5 Empirical Study
In this section, we present an empirical evaluation of the proposed mechanism. Our objectives are twofold: first, to demonstrate that in practical LLM settings. agents can benefit from misreporting their preferences and distorting the learning outcomes; and second, to intuitively show how our mechanism incentivizes truthful reporting
Models and Datasets. Our experimental setup follows the literature on Multiple-Objective RLHF [22 70 78). We consider two tasks: the Helpful Assistants task [5] and the Reddit Sum- mary task 722, using Llama-2 7b (74 as the base model for both. For the Helpful Assistants task, the initial model LLM is obtained by supervised fine-tuning a Llama-2 7b model on the Anthropic-HH dataset 5 We then apply two reward models during the RLHF process to measure harmlessness and humor, respectively. For the Reddit Summary task, the model is fine-tuned on the Commarrzeefrom-Feeddhe dataset 722. with two reward models assessing the summary's 's quality and faithfulness.
We formulate these tasks two mechanism design scenarios: the "Harmless vs. Humor" game for the Helpful Assistants task, and the "Faithful vs. Summary" game for the Reddit Summary task. In each game, we assume that there are two groups whose joint preferences are captured by a reward model. For example, in "Harmless vs. Humor, group prioritizes harmlessness, while group 2 values humor. The corresponding reward models for these preferences are denoted as rm, (harmlessness) and rma (humor), with synthetic group size vectors (W1, w2) selected from {(3, 7), (5, 5), (7,3) varying across different settings.
Implementation Details. We implement the basic training rule from Definition using the KL--dierggence as the distance measure f. To balance model optimality with training cost, we simplify the problem by replacing the entire parameter space O with a representative finite set O'.
"The code for the simulation is available at GitHub

---

Models are first trained using single reward models and then combined via the Rewarded Soups technique [62] to produce a set of hybrid models, {0), 0z..... ........pp. 0x): J. which constitute O'. Optimality is then defined over this finite set. As shown in Rame et al. [62], this approach reduces training costs while maintaining performance comparable to full multi-ohjective fine-tuning.
Given the large space of potential misreporting strategies, we focus on two simple ones:
Strategy (1): r,, = rm; and ii; ane; Naive overstatement: Exaggerating group size to gain more influence, requiring no knowledge of other groups.
Strategy (2): rm; = 3rm; + (1 3)mm: and úi wy Strategic manipulation: Leveraging other groups' preferences to downplay opposing outcomes, requiring some information about conflicts.
Intuitively, a and 3 = represent truthful reporting. Increasing a or B allows group to gain more influence in the training process. Our experiments confirm that both strategies can be profitable. However, the DSIC of our mechanism ensures that truthful reporting yields higher utility than any misreporting strategy.
Result Analysis. Since the outputs of different reward models have varying scales, we normalize all reward values to [0, 1], where the maximum and minimum values are and O, respectively. We then report the normalized valuations, payments, and utilities of group i for different reporting strategies Each column represents specific RLHF Game with given group size (wi,w2).
in Figure 2
As shown in the figure, increasing or 3 leads higher valuation for the group, confirming that groups can benefit from simple misreporting in the absence of payments. However, when the payment pAr is applied, increases with or B, offsetting the gains in valuation. This ensures that truthful reporting (a = 8 = 1) maximizes utility in all cases. Additional simulation settings are provided in Appendix
6 Conclusion and Future Work
This paper studies incentive issues in potential economic scenario where platform offers LLM fine-tuning services to aggregate preferences and agents strategically report get preferred outcome. We focus on aggregation objectives that maximize social welfare subject to regularization constraints, referred to SW-Max rules. Through comprehensive analysis of strategic reporting, we demonstrate the critical role of payment schemes in incentivizing truthful reporting under SW-Max rules. We derive sufficient conditions for payment equivalence and identify necessary conditions for implementing SW- Max rules within additional constraints. Moreover, we analyze how perturbed input will influence the mechanism to account for practical errors that inevitably arise and show that the mechanism satisfies approximate DSIC. Finally, we conduct experiments within real-world LLM setups, showcasing how the proposed mechanism effectively incentivizes truthful reporting.
Building on our proposed scenario and formulated model, we identify several promising directions for future research from both theoretical and empirical perspectives. First, exploring and modeling more general training rules could enhance our understanding of the framework. As noted in Appendix E, cycle monotonicity is necessary and sufficient condition for implementability. but its validation is complex. Identifying a simpler condition to ensure implementability and investigating properties like payment equivalence for these rules are critical next steps. Second, studying preference aggregation across multiple models, particularly with diversity considerations, is a valuable direction. Third, as discussed in Section 4.4 and incentive compatibiliyy in the RIHF Game could improve its real-world applicability. Finally, applying mechanism design theory to other large language model contexts, such as API pricing, retrieval-augmented generation (RAG), and prompt engineering, offers significant opportunities for further exploration.
4.4 developing mechanisms criteria that balance computational efficiency
Acknowledgments
This work supported by the National Natural Science Foundation of China (Grant No. 62572010 and No. 62506010). We thank all anonymous reviewers for their helpful feedback.
10

---

References
[1] Saaket Agashe, Yue Fan, and Xin Eric Wang. Evaluating multi-agent coordination abilities in large language models. arXiv preprint arXiv:2310.03903, 2023.
[2] Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. arXiv preprint arXiv:2305. 2023.
[3] Syed Mumtaz Ali and Samuel D Silvey. A general class coefficients of divergence one distribution from another. Journal of the Royal Statistical Sociery: Series B (Methodological), 28(1):131-142, 1966.
[4] Itai Ashlagi, Mark Braverman, Avinatan Hassidim, and Dov Monderer. Monotonicity and implementability. Econometrica, (8(5):1749-1772 2010.
[5] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
[6] Dirk Bergemann and Juuso Välimäki. The dynamic pivot mechanism. Econometrica, 78(2): 771-789, 2010.
[7] Sushil Bikhchandani, Shurojit Chatterji, Ron Lavi, Ahuva Mu' alem, Noam Nisan, and Arumava Sen. Weak monotonicity characterizes deterministic dominant-strategy implementation. Econo- metrica, 74(4): ([(4):109-1113) 109-1132, 2006.
[8] Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
(9] Patrick Briest, Shuchi Chawla, Robert Kleinberg, and S Matthew Weinberg. Pricing randomized allocations. In Proceedings of the twenty-firs annual ACM-SIAM symposium on Discrete Algorithms,, pages 585-597. SIAM, 2010.
[10] Thomas Kleine Buening, Jiarui Gan, Debmalya Mandal, and Marta Kwiatkowska. Strategyproof reinforcement learning from human feedback. arXiv preprint arXiv:2503.0990 2025.
[11] Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925. 2024.
[12] Yiting Chen, Tracy Xiao Liu, You Shan, and Songfa Zhong. The emergence of economic rationality of gpt. Proceedings of the National Academy of Sciences, 120(51):e23162051200 2023.
[13] Alessandro Chiesa, Silvio Micali, and Zeyuan Allen Zhu. Mechanism design with approximate valuations. In Proceedings of the 3rd Innovations in Theoretical Computer Science conference, pages 34.38,22 2012.
[14] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[15] Edward Clarke. Multipart pricing of public goods. Public choice, pages 17-33, 1971.
[16] Vincent Conitzer and Tuomas Sandholm. Self-interested automated mechanism design and implications for optimal combinatorial auctions. In Proceedings of the 5th ACM Conference on Electronic Commerce, pages 132-141, 2004.
[17] Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley Holliday, Bob M Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, et al. Social choice for alignment: Dealing with diverse human feedback. arXiv preprint arXiv:2404. 10271, 2024.
[18] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743 2023.

---

[19] Imre Csiszár. On information-type measure of difference of probability distributions and indirect observations. Studia Sei. Math. Hungar. 2:299-318, 1967.
[20] Michael Curry, Tuomas Sandholm, and John Dickerson. Differentiable economics for random- ized affine maximizer auctions. arXiv preprint arXiv:2202.02887. 2022.
[21] Zvi Drezner and Horst W Hamacher. Facility location: applications and theory. Springer Science & Business Media, 2004.
[22] Zhijian Duan, Haoran Sun, Yurong Chen, and Xiaotie Deng. A scalable neural network for dsic affine maximizer auction design. Advances Neural Information Processing Systems, 36, 2024.
[23] Zhijian Duan, Haoran Sun, Yichong Xia, Siqiang Wang, Zhilin Zhang, Chuan Yu, Jian Xu, Bo Zheng, and Xiaotie Deng. Scalable virtual valuations combinatorial auction design by combining zeroth-order and first-order optimization method. arXiv preprint 240::2400..1994,, 2024.
[24] Kumar Avinava Dubey, Zhe Feng, Rahul Kidambi, Aranyak Mehta, and Di Wang. Auctions with Ilm summaries. arXiv preprint 20ti::240400822200 2024.
[25] Paul Duetting, Vahab Mirrokni, Renato Paes Leme. Haifeng Xu, and Song Zuo. Mechanism design for large language models. arXiv preprint arXiv:2310.10822 2023.
[26] Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvi- jotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.092. 222.24444 2023.
[27] Meta Fundamental Al Research Diplomacy Team (FAIR),, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624): 1 1677-077, 2022.
[28] Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao He. Can large language models serve as rational players in game theory? a systematic analysis. arXiv preprint arXiv:2312.054888, 2023.
[29] Soheil Feizi, MohammadTaghi Hajiaghayi, Keivan Rezaei, and Suho Shin. Online advertise- ments with Imm:: Opportunities and challenges. arXiv preprint arXiv:23/1.07601, 2023.
[30] Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang. Kun Shao, David Mguni, Yali Du, and Jun Wang. Chessgpt: Bridging policy learning and language modeling. Advances in Neural Information Processing Systems, 36, 2024.
[31] Roberto Gallotta, Graham Todd, Marvin Zammit, Sam Earle, Antonios Liapis, Julian Togelius, and Georgios N Yannakakis. Large language models and games: A survey and roadmap. arXiv preprint arX1v:2402.18655, 2024.
[32] Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. Strategic reasoning with language models, arXiv preprint Xrv:v:3305. 19165, 2023.
[33] Ian Gemp, Yoram Bachrach, Mare Lanctot, Roma Patel, Vibhavari Dasagi, Luke Marris, Georgios Piliouras, and Karl Tuyls. States as strings as strategies: Steering language models with same--hherrttic solvers. arXiv preprint arXiv:2402.01100 2024.
[34] Theodore Groves. Incentives in teams. Econometrica: Journal of the Econometric Society, pages 617-631, 1973.
[35] Shangmin Guo, Haochuan Wang. Haoran Bu, Yi Ren, Dianbo Sui, Yu-Ming Shang, and Siting Lu. Large language models as rational players in competitive economics games. arXiv preprint 2308. 10032, 2023.
[36] Shangmin Guo, Haoran Bu, Haochuan Wang, Yi Ren, Dianbo Sui, Yuming Shang, and Siting Lu. Economics arena for large language models. arXiv preprint 201.02240
12

---

[37] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey progress and challenges. arXiv preprini arXiv:2402.01680, 2024.
[38] Birgit Heydenreich, Rudolf Müller, Mare Uetz, and Rakesh V Vohra. Characterization of revenue equivalence. Econometrica, 77(1):307-316, 2009.
[39] Radosveta Ivanova-Stenzel and Timothy C Salmon. Revenue equivalence revisited. Games and Economic Behavior, 64(1)1171-1 2008
[40] Athul Paul Jacob, Yikang Shen, Gabriele Farina, and Jacob Andreas. The consensus game: Language model generation via equilibrium search. arXiv preprini arXiv:2310.09138 2023.
[41] Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang. Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Per- sonalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310. 1564, 2023.
[42] Philippe Jehiel, Moritz Meyer- Ter- Vehn, and Benny Moldovanu. Mixed bundling auctions. Journal of Economic Theory, 134(1):494-51) 2007.
[43] Benjamin Laufer, Jon Kleinberg, and Hoda Heidari. Fine-tuning games: Bargaining and adaptation for general-purpose models. arXiv preprint vrXiv:2308.04399 2023.
[44] Anton Likhodedov and Tuomas Sandholm. Methods for boosting revenue in combinatorial auctions. AAAI, pages 232-237, 2004.
[45] Nunzio Lorè and Babak Heydari. Strategic behavior of large language models: Game structure vs. contextual framing. arXiv preprint arX.v:2309.05888, 2023.
[46] David G Luenberger, Yinyu Ye, et al. Linear and nonlinear programming, volume 2. Springer, 1984.
[47] Weiyu Ma, Qirui Mi, Xue Yan, Yuqiao Wu, Runji Lin, Haifeng Zhang, and Jun Wang. Large language models play stareraft ii: Benchmarks and chain of summarization approach. arXiv preprint 12X.v:2312.11865 2023.
[48] James Melbourne. Strongly convex divergences. Entropy, 22(11): 2020.
[49] Mitsunobu Miyake. On the incentive properties multi-item auctions. International Journal of Game Theory, 27:1-19, 1998.
[50] Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton. Welfare diplomacy: Benchmarking language model cooperation. arXiv preprint arXiv:2310.08901, 2023.
[51] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang. Matthieu Geist, Thomas Mesnard, Andrea Michi, al. Nash learning from human feedback. arXiv preprint arXiv:23 2023.
[52] Roger B Myerson. Incentive compatibility and the bargaining problem. Econometrica: journal of the Econometric Society, pages 61-73, 1979.
[53] Roger B Myerson. Optimal auction design. Mathematics of operations research, 6(1):58-73. 1981.
[54] Noam Nisan and Amir Ronen. Algorithmic mechanism design. In Proceedings of the thirty-first anmual ACM symposium on Theory of computing, pages 129-140, 1999.
[55] Noam Nisan et al. Introduction to mechanism design (for computer scientists). Algorithmic game theory, 9:209-242, 2007.
[56] Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
13

---

[57] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[58] Susan Hesse Owen and Mark S Daskin. Strategic facility location: A review. European jourraa of operational research, 111(3):423 447, 1998.
[59] Chanwoo Park, Mingyang Liu, Kaiqing Zhang, and Asuman Ozdaglar. Principled rihf from heterogeneous feedback via personalization and preference aggregation. arXiv preprint arXiv:2405.00254, 2024.
[60] Alessandro Pavan, Ilya Segal, and Juuso Toikka. Dynamic mechanism design: A myersonian approach. Econometrica, 82(2):601-633 2014.
[61] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAl, 2018.
[62] Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthiuu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
[63] Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187, 2024.
[64] Kevin Roberts. The characterization implementable choice rules. Aggregation and revelation of preferences, 12(2):321-34) 1979.
[65] Jean-Charles Rochet. A necessary and sufficient condition for rationalizability in quasi linear context. Journal of mathematical Economics, 16(2):191-200, 1987.
[66] Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models self-improve with general preferences. arXiv preprint arXiv:2404.03715. 2024.
[67] Michael Saks and Lan Yu. Weak monotonicity suffices for truthfulness convex domains. In Proceedings of the 6th ACM conference on Electronic commerce, pages 286-293, 2005.
[68] Tuomas Sandholm and Anton Likhodedov. Automated design of maximizzx combina- torial auctions. Operations Research, 63(5):1000-1025, 2015.
[69] Xiao Shao, Weifu Jiang, Fei Zuo, and Mengqing Liu. Swarmbrain: Embodied agent for real- time strategy game starcraft ii via large language models. arXiv preprint 4rXiv:2401.17749, 2024.
[70] Ruizhe Shi, Yifang Chen, Yushi Hu, ALisa Liu, Noah Smith, Hannaneh Hajishirzi, and Si- mon Du. Decoding-ting time language model alignment with multiple objectives. arXiv preprint arXiv:2406. 18853, 2024.
[71] Ermis Soumalias, Michael J Curry, and Sven Seuken. Truthful aggregation of Ilms with an application to online advertising, arXiv preprint AXX::2200500900 2024.
[72] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems,
2020.
[73] Pingzhong Tang and Tuomas Sandholm. Mixed-bundling auctions with reserve prices. In AAMAS, pages 729-736, 2012.
[74] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
14

---

[75] William Vickrey. Counterspeculation, auctions, and competitive sealed tenders. The Journal of finance, 16(1):8-37, 1961.
[76] Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part it: Reward modeling. arXiv preprint arXiv:2401.06006 2024.
[77] Shenzhi Wang. Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang. Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. Avalon's game of thoughts: Battle against deception through recursive contemplation. arXiv preprint 2rXiv:2310.013
2023.
[78] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36, 2024.
[79] Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study werewolf. arXiv preprint arXiv:2309.0466 2023.
[80] Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.1880 18940, 2023.
[81] Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv: 2024.
[82] Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Im- proving reinforcement learning from human feedback with efficient reward model ensemble. arXiv preprint arXiv:2401. 16635, 2024.
[83] Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian Wynter, Yan Xia, Wenshan Wu, Ting Song. Man Lan, and Furu Wei. Lim mastermind: A survey of strategic reasoning with large language models. arXiv preprint 21Xiv:2404.01230, 2024.
15

---

Limitation
The main limitation of this paper is that we mainly consider the SW-Max training rules and their theoretical properties. Further study could consider more training rules and extend our model to the DPO scenario, in which each group only provides pairs of data rather than reward model.
A Further Related Work
In this section, review relevant research across various domains that are related to our paper, including works on RLHF with multiple reward models, multi-parameter auctions, and the intersection of game theory and LLMs.
A.1 RLHF with Multiple Reward Models
Research involving multiple reward models primarily focuses developing algorithms enhance practical performance. Some studies design methods simultaneously satisfying multiple prefer- They develop more efficient algorithms extend the Pareto frontier 8T and balance issues from various perspectives [ 59 63
ences 41 62
63
70
78 81.
among different objectives
Additionally, there is body of work that trains multiple models for single preference and then ensembles them to improve the robustness of RLHF [8, 82], mitigate the influence incorrect and ambiguous preferences in the dataset (76), and reduce reward hacking [26). Unlike these approaches, our work considers how to collect misaligned preferences truthfully from different agents. As we have mentioned, these works are often assumed to be accessible to humans' actual preferences, neglecting the incentive issue for motivating rational agents to report truthfully.
A.2 Multi-parameter Auctions
Several studies have explored the properties relevant to our paper in various multi-paramens parameter auction
scenarios, such implementability [4 7 16 49 65 67 and payment equivalence 66 38 39 60) Another central topic in auction theory is to design mechanisms that satisfy DSIC and IR while maximizing the expected revenue for the auctioneer. Although the single-parameter scenario has been resolved by Myerson [53], the optimal auction design for multi-paramene parameter settings remains an open question. Therefore, there is stream of research focusing on a specific subset, affine maximizer
68 73). and proposing optimizations Compared to these works, we are the first to discuss the property of payment equivalence and the revenue-maximizing solution for SW-Max training rules in the scenario of fine-tuning LLMs.
auctions, which inherently satisfy DSIC and IR 9 42 64
enhance empirical performance 20 22 23
A.3 Game Theory and LLMss
In addition the work we review in the primary related work, others have explored the intersection of game theory and large language models from different perspectives. A line of work studies other L.M--reatted scenarios from the algorithmic game theory perspective. Laufer
(43) abstracted the
fine-tuning process as bargaining game and characterized the perfect sub-gama equilibria. Dubey et al. [24] proposed an auction where bidders compete to place their content within a summary generated by an LLM. Conitzer al. [17] considered incorporating social choice theory in LLM alignment. Feizi et al. [29] explored the potential for leveraging LLMs in online advertising systems.
More broadly, some research has proposed algorithms for training LLMs inspired by concepts in game theory, such as Nash learning from human feedback (51). consensus game [40), direct Nash optimization [66], and Gemp et al. [33]. And various studies assess LLMs from a game-
theoretical perspective, examining aspects such as rationality [12 28), behavior in matrix games [2
32
45 and performance in strategic games like auctions 33 36 Werewolf 77 80 Avalon 777]
Diplomacy 22 50) card game
(30 and electronic game
There are also comprehensive
surveys 31 37 83
16

---

B Omitted Proofs in Section 4.1
Theorem 4.2. In the RLHF Game with mechanism (1),)) that vb E &/sw and p for group i, define 8; == 1rrr = rm; x E ET' and rm; := minger- rm;(x))
1. If 84 = truthfilty reporting is the optimal strategy regardless of other groups' reports.
2. If 8; 2 2 and rm; > O, there is a strategy that yields strictly higher utility than truthfully reporting regardless of other groups' reports.
Proof. If 8; = 1, the group gets the same utility from all training outcomes. Therefore, any strategy optimal. We then analyze the 8; 2 2 and rm; in the following. First, the optimization of s can be written as an equivalent constraint programming problem on the LLMg:
LLMa(x)
arg max L.
Σ
1 (0;
Σ ØET-
LLM (
L.LM (x)
Σ LLMg(x)) =1
S.L.
#ET
LLM,(x) 2 O Vo T
Because of the assumption that the optimal policy satisfies (LM)(x)) 0 for all T we can infer that the condition LLMg(x)) 2 0, Va E T is not active for the optimal solution. Since the convexity of the function f. by KKT condition, the necessary condition for the optimal 0 is that there exists J E R [46]. such that
Oy Wi (LLMg
LLMa(()) LLM (z) )==
= Vo E T*
Σ mET
LLMa(x) = 1.
Under Definition 3.1
Be OLLMo(æ)
= (m)) so we have
a
(w) - 00' [LLM((z) (((LMMM((()) [LLMo... (2:) = Va E T
Σ
(OPT)
We mainly discuss the strategies other than simply over-reporting the group size W. We omit the notation ui for simplicity.
Next, our main technique is to construct report reward model rm,; * rm; for group i such that (e())((mm) e mm))), :), 0ma); rm.) > w; (v)((,, rm), mm)) rm,) holds for all rm and Omn-
The Summation Normalization Case. We first discuss the case of the reward model being normalized by summation. We take the Φ1 E arg maX (x), ( 22 E arg min (x)). Since minæer. rm,(æ) 0, we have rm,( < 1 and mm()))) 0. Then we take small e < min 11 (2;) rm; (x2)) and define rm/ as:
rm; (x) + x= Z1. (x) = rm; (x: - C,
x2
rm; (2),
x + x1 x # 22
Intuitively, by reporting m/, group pretends to value more for the most preferred æ and less for the least preferred æ. Let 0 40( ((mm, mm_1), mmn) and 0' 1'( m), mmn)) denote the variable in the necessary condition for LLMg and I.LMg,, and we can derive the following results.
and μ' to
(a) L (z,) > LLMg (x:) and LLMg < (x22) We prove the former by contradiction: if LLMg (x)) LLMa (x1), then by the convexity of f, we have
LLMa (x
LLMa(x1))
s'
S
LLM (ar)
I.LM
With (x1) > (x)) we can infer that jı' > μ.. However, since for all of æ1, we have m((x) & (x). to satisfy the optimal condition in (OPT), there must be for all x a Ø1,
LLMg (x)
LLMa
f
<
LLM (x)
LLM (x:
17

---

Which is equivalent to LLMg (a) < (x)) and hence results in Ever (L) < EnEr- (z) = 1. The latter, LL (22) < (LM) can be proved by totally same method.
(b) The order of LLMa (a) and LLMgr (x) for all æ & {æ). Ø2) is consistent. Without loss of generality, assume there is 23 & { Ø2) such that LLMØ (xs) LLMo( Then we have
LLMo (33) f'
(LLM
2
LLM (z)
LLM (x)
Then, infer that p $ μ. For all & {21, to satisfy Equation (OPT). there must be
LLMo (x)
F (L
f
LLM (x)
LLM (x)
which is equivalent to LLMg (x) 2 LLMp(x). Similarly, if there is as & {21, such that (LMg & LLMg (233, then for all æ & {01. there LLMg (x) S LLMg(x)).
Finally, with the results in (a) and (b), when (x:) I (LMo(()) for all a & (01, the change the utility of group can be calculated by
E (LLMa( LLM2(x))) rm;(2))
#ET
Σ
(LLM (x) - LLM, rm, (x) + (LLMa -LMM
(z1)) (m
EET*
Σ
(LLMo(x)) (z)) + (LLMg (LMa(x))) (21)
Σ
(LLM)( - LL (2:) (x) + (LLMgr (LLM((F)) rm;
D2 =
(L LLMa(
Σ
(LLMg(æ) LLM)
=mm;(x1)
(LLMo- LLMo(z)) =0
*
When I. (2r) 2 LLMa(x)) for all x * æı, 222 the change in the utility of group i can be calculated by
Σ
(LLM (x: (z) (x)
Σ
(LLM (z) (z)) (m) (x) + (LLM (x.) LLMg(xx)) (xx2) mx,)
Σ
(LLMø (x) (LM - (LLM2 LLM (x²)) (x2)
TET
Σ
(LLMg (LM2(x))) (m)) (x2) (LLM)(x22) LLMar (mz,()))
m TT
(x2)
(LLMe: -LLMg(x)) (LLMa(x2) - L
#
=xm2(x2)
(LLMMr(() - LLMo(x:) = O.
mET-
Note that both (2) and (3) are because of rm; 2 rm;())2). And unless rm;(x;) rm;(222)
which is excluded by 2 2, the "> " are hold.
18

---

The Maximum Normalization Case. The case of the reward model being normalized by the maximum similar. We take the æ arg men nm, mm((11) > 0. Then we take small rm,( and define rm/ as:
(x). Since min (x) 0, we have
(a) (m) a 7 21-
[x)
21,
With the same technique, we first show that ML (a1) < [LLMg(#g) and LLMMg L.LMg ( for all æ # I1. After that, it is easy to derive that when 2 2, the change in the utility of group satisfies
Σ BET
(LLMa (LMa(x)) rm, (x) 0.
Lemma B.1. When the training rule v/ E SW and the divergence function f is a-strongly conver and C2 -mmooohh, then satisfies Definition 4.7
Proof. As is shown in the proof of Theorem 4.2 we have two Lagrangian variables y and p for (rm, wi) and (rm, Wi), respectively. We have the following equations:
(LMa(x)
w.rm;
LLM (x)
M,
Va E TT.
LLMg (x)
Σ 1=1
-
= =M,, Va E TT"
LLM (x))
Firstly, we have 1p - pl S max E (arm;(x)) - Ei= Ei=1 (x)) Otherwise, without loss
of generality, assume that μ¹ maXrET" 12 w.rm; (x) - >i (m), then we can derive that Vo T"
8' LLMa(x)
LL (x)
<
LLMo,a (x)
LLMo (x)
This means that LLM (2r) < LLMM (a:) for all æ. which leads the contradiction. Therefore, we have for all x E T
rr
(L) LLM (x)
8 (M
LLM (x)) )
Σ warm; (x))
w/rm + + -1
S 2
w. worm; (x)
Σ w w/rm((
By C2-smoothness of and the a-strongly convexity, we have for all E T
[LLMo(x:) LLM LLM (x)
LLMg(x)
Lz
S
a
LLM (x))
LLM (x)
2LLM (x)
12
Σ w.rm:(a)) Σ
a
Therefore, for any > 0. if E warm (x: En (x)
then
[LLMa(x)) - LLLM (x) c.
Theorem B.2 (Detailed version of Theorem 4.3g In the RLHF Game with mechanism (xb, p) that SW and p = 0, when f is a-strongly convex and C2 C2-smooth, suppose group has preference
rm; and group size wi, other groups report (rm- WJ w))) and the initial model O we define
(rm;(2) (2) Brn
t(x) = Σ
f"
LLMa(w)
*ET
LLMo
in which 0 = =b(rmh Wi 0m))). When > 2 and m,; 0:
19

---

1. For the maxinum normalization case, if there exist æ1 E T,, t(x1) # 0 and 0 < mm;(xa)) <
1, truthful reporting not the optimal strategy.
2. For the summation normalization case, if there exist æ1 E T", t(x11 < and <
(m; < 1, truthful reporting is not the optimal strategy.
3. For the summattin normalization case, if there exist Φ1 E T", t(x1) > 0 and we can also find D22 E TT" such that 1 rm;(x1) 2 rm;(222 > O and LLMe (m1) S"
(
LLM (m))
ILMo (1n) fd
[LMo( I.LMMna (aa) truthful reporting is not the optimal strategy.
Proof. As shown in the proof of Theorem 4.2 the necessary condition for the solution that there exists j E R such that
LLM (x))
(.) - ff
Vo E T", Σ LL) =
LLM (a)
øET-
And by Lemma B.1 we can also use the condition Definition 4.7
The Maximum Normalization Case (1). Without loss of generality, assume that there exists 11 such that t(æ1) > 0, we take 0 < E < (m1 to construct report rm:
(x)
(m);(ar) + e, (m),
21,
x + Z
Suppose that μ1 is the Lagrangian variable for the optimal solution O' when reporting rm/, we can derive that
LLM (x)) (
LLMg (xr)
LLMa(x)) LLM (x) ))
1 44 H--=w;ck W-cll== -
-
Va E T"
With a similar analyze in the proof of Theorem 4.2 we can induce that μ¹ > and LLM (x) < (L) for all æ A 11. By the C²-smoothness of f, for each 2τ, there exits LLMg (x) S < LLM)(x)) such that
LLMg (x) - LLMa(x)
LLM (x)
LLM (x)
For convenience, we let LLM (x) refer to the corresponding for note that LLMg is not necessarily a distribution. We then compute the change the group i's utility:
(z (w: LLMs(x))
=rm, (()())((LL)M)) (LLMg (x1) (Z)) +
Σ I. (x) (x)) (z))
at
-r1;(x1)
(LLMo(x)) LLMg (x))
(LL)((()) LLLMa (z))
x Df
(μ' (m,;( rm, LLM Mom
(x)
LLM (2c) (d(
[.LM (x)
Then, we show that when the e we choose is sufficiently small, the above term is positive. We define the lower bound:
1"( LLMa
= min mET
LLMo (x)
Since function f is a-strongly convex, 81 2 a > 0. By the -2-smoothness of the f. there exists an
82 > O, such that for each satisfying max [LLMg(x) LLMa < 62, have
LLM (x) ) - LLM (x)
LLMg(z)
81 oft(21).
max #ET*
("(
min
LLM (x)
2 4|7
20

---

Besides, because of the Definition ing maX
4.7 there exists 83, such that for each (tū, rm) and (2o". mim) satisfy-
Di= w.rm, (z)
i= w/rm/ (x)] I 63, we have max, [LLMg LLLMg (x))
d2-
Combining these, we
then it is suffice show that we
63
(rm, (2)1) rm (x) (L)))LM (x)
(rm:( rm, (a),(a)))ln LLLM (x)
f"( I.MMa (m)
(LM
f" M.T.Mam (at)
(
(rm,(21) - rm; (x)) (f" LLMo(x)
f" LLMM (2)
LLM
LLM (x)
LLLM, (c0)
f"(
LLMg (2)
( LLM,, (m)
2
LLLM (x))
LL (2)
Ixm:(x)) rm;(x))
("(
[LMa(a)
8" LLM (ar)
(LM,
LLM (x)
LLMo (2)
f" LLM,, (2x)
LLM (wr)
LLM (2)
oft(21)
2
t(x1)
<|7
4
81.01
This means that
(m,,(x))
rmm,(1)))))) (2)) (x)
(rm, -m, (x)))LMM (x)
t(x1)
Σ
("(
LLMg [c)
f"( I.T.M,ua (x)
[L
2
ML (z)
t(x11)
t(xr1)
((x1)
> 0.
2
2
Combined with μ¹ > μ. the proof concludes.
The Summation Normalization Case (2). Assume that there exists æ1 such that t(z1) < 0,
select Ø2 := arg maX rm; and take 0 < < min(rm, r2,(22)) to construct report rm!
(x) E, T1: rm, (x) + E,
(x)
T2:
rm, (x),
x & (I1, 22)
Still, we use μí to denote the Lagrangian variable for the optimal solution 6' when reporting rm/. Then, there are two possibilities for the relationship between and μt. If μ S p., by the optimal condition OPT for all o æ2, we have LLM 2 LLLM (ær). Since æ2 has the highest reward value, such a change in the training outcome must be more preferred by the group we only have to consider the case that > μ'. Similarly, in this case, for all x * 21, we have (L) LLMa((x). (x). By the C3-smoothness of f. for each of æτ. there exits LLMa(x) s LLMg (x) such that
Therefore,
LLMg (x) LLMa(ar)
n' μ w.ella=a, f" LLM (x)
LLMA (x)
Let LLM (x) refer to the corresponding z for x. we then compute the change in the group i's utility:
(LLMg (z))(x)))
*ET
=rm;(w)) (m) (LLM (L)) +
Σ (LLM - LL))(x))
2
=xm((x1) Σ (LLM²(x) - LLM (x))
rm (m) (LLMg))))) LLMg (x))
(n' 1) (rm, rm, (x) (x)
(rm,(x;) Irm; (x (z:2) LLM (x2)
L.M (mn)
Wie
f LLM
LLM (ac)
LLMo (a2)
(n' (Tm Im,
LLM (x:
( L.M (m) [LMa
2
21

---

With the same technique we used in the maximum normalized case (1), we can show that with (m,, (z))) ) ( (a)
sufficient small > O, the above term Eaym
f(m)))
0. Combined
f
11M mm
2
with p' < μr, the proof concludes.
The Summation Normalization Case (3), Assume that there exists æ1 such that f(x1) > 0,and æ2. mm 2 m2,) > O, take 0 < (min(m,((z))) rm,(x1)) to construct report rm)
(2; + 6,
21.
(x)
(m; -, D2.
(m)
& T2)
Still, use yí to denote the Lagrangian variable for the optimal solution 6' when reporting rm,-
Since we know for sure that LLM2 (21) < LLMg (#1) and LLM)( > LLM (x2), by the
Ce-smoothness of f. (LMMg((aa) LLMa I LLM2( and (L)) LLMg S LLMg (z)) such that
LLMg (111
LLMgx( - LLMa((11)
n1 11 Wif (
L.Mo mm (1
LLM (211
(2)
1l μr -wie f LLMg (222
L. - (.LM((a2)
LLM (I2)
LLM (x22
Let 81 := min LL (x). by the -mmsthhthness the f. there exists an 62 > 0, such that for each 0.0' satisfying max, (a) (x)) < 82, we have
51 LLMe, (m2)
[LMa( LL (w2)
4 LLMo (m1)
(L
(x) LLMo... (x) LLM (z)
LLMg (zz) f"
LLMo,,, (m1)
max =ET*
3
(3)
We take < 62 and first prove that when taking such 6, there is μ S jr', By contradiction, if
wy
u' μ. then by condition Equation (OPT), for all & {x1, æ2). }. there is LLM (x) > LLMg(x).
Therefore, LLM (x1) - LLMa LLMg < (LM)( LLM (x²2) (x22) However, by Equation
Enn (m1 .x2) (LLMg x LLM (x)) + LLM,(xx))
if pl < μc, we get
f" (LM LLMs (x)) (x)) l" L.MMa.. (x:2)
LLLM (x22)
LLMa( LLMo (x22)
LLMoaaa (21)
L.L.o... (x1)
LLM (x22)
By Equation 3/. we can derive that
LLM (2E1) f"
LLM (2²)
8"
LL (211)
LLMo (x1)
LLM (22)
LLM (x2)
and thus, we get
LLM (x LLM > LLMa(#x2)
LL2)
which brings the contradiction.
After proving that μ S μL, we know that for all æ & (x1, x2 (LM, 2 LLM (x). Then, by
the C² -m-sohhtnssss of f. for each # x1, there exits LLMg < LLMg(z) (x) such that
L.M (x) (LM
H - = -weell==ee - ( LLM (x)
LLM (x)
22

---

Let LLM (z) refer to the corresponding x for 2, we then compute the change in the group i's utility:
(LLM (L))
-nm (z) (LLMg (x LLMg(x1)) +
Σ
rm, (z) - LLMa(x))
2/--21
=a1((x1)
(LLMg(>) (LM
rm, (2) (LLM -LMØ (x))
20
(u' (rm; ( (xt
(rm:( - (2m2((22) LLM (222
+ W36
( LLLMa (x)
1.M
f
LLM (mmg)
2/01
LL (m2)
(u' ( (Tm; rm, LLMM (x)
(a) f M.LMM.. (m)
2
With the same technique we used in the maximum normalized case (1), we can show that with sufficient small
(mm, (00))mm)) (as) LLM)) (æ) f(m)))
0, the above term Exter
0. Combined
f M
(1M
with p < the proof concludes.
C Omitted Proofs in Section 4.2
Proposition 4.4. For any th E Sw mechanism (t,pArr, satisfies DSIC and IR, and the payment non-negative.
Proof. We assume that for group i, the true reward model is rm,, and the agent number is wi. The reports of other groups (rm-i, (-:)) and the initial model is O.n-
(I) (th, p pFF) satisfies DSIC.
We compare the utility between reporting (rm,, wy) and any other (rm/, wf). For convenience, we first simplify the notations by letting
0 (((m,, ,) (w,, (-))), Omit))
0' * (((m)) m..), (w/, (-))), Omit))
The valuation of group is the valuation for each agent multiplied by the real agent number:
0 = w..; (0; rm:), 5 = w.v; (6': rm.).
According to the payment rule AFF the payment for (rm;, ws) and p/ for (rm(, w!) is
A:SW W_,, 0 -m w_i, Omnnt) ASW - im _. Omit)
p = ASW --w-Ws(Wifff ( _.,, mni)) mن w_s, Ounnt) ASW ASW (80' (8'; ; w_ Omit)
Therefore, we can calculate the change the utility:
u/ u; = (v) p (w: - pt)
= (w) rm;) + ASW (6'" :m w- Qni)
(0; + ASW ((0; rm- 17 - 0ma))
=ASW((O'; (rm,, .), (w, w) Omnn) ASW((O; (rm,, mm mm.)). (w,, w-))) O
<0.
The last inequality holds by the definition of 0 = ((((m, m :), (wi, Wt ,)), Oritt) = arg max ASW((O; (rm;, m)) (wn, 27 ()), Omni)
dee
23

---

Therefore, we can conclude that, for all mi, Wü, rm,, wi we have
u4 ( ú d. p AFF rm,, wy) 2 u:((rm, m +), (wh, it .): w, u, AFF .rm,, mz)
(2) (19pp p AFF) satisfies
We reuse the notations above and denote 0- to be the optimal parameter for groups except for i, i.e.
0 0_ v-rm_ Wi (iit).. When group truthfully report its reward model rm; and agent number wi, the utility can be written as:
U U; Pi
1o (0; rm;) ASW (0- th 0ma) ASW rm- W1 mmii)
(0;0(0;m,)) + ASW -0: rm- 11 -i 0ama) ASW (8 (8- rm W. W1 mmni)
AS rm, ni, mmi) ASW (0_ 100mmmm---- --m W- Omis)
ASW (0- rm, ú7, 0mmis ASW - mm W- Omin)
w, + ASW (0- (m;00-mmm,,,0.m0 07, Bmi)) ASW -: rm w-in mmii)
w. (0,, 2
Therefore, we can conclude that, for all m, W. we have
us (1m, w)); vb AFF .m, w.) 2 0.
Proposition 4.6. When ih = 1 is public information, and the agents only report the reward models, all implementable training rules satisfy payment equivalence.
Proof. We follow the result Theorem 1.37 in Nisan et al. 55
Lemma C.1 (Theorem 1.37 in Nisan et (55). Let R, be group i's preference domain. Assume that the R1, ....... Rr are connected sets in the Euclidean space, then all implementable training rules vb satisfy payment equivalence.
In our paper, we assume that for all E [n). R, is the set all non-negative and normalized [T* Idim vectors. Either in the summation normalization case or the maximum normalization case, this is connected set in the Euclidean space. Hence, the theorem holds.
Proposition 4.8. SW-Max training rules with regularizations KL-divergence, fkL(2) Ar log x, and x2 divergence, fz(x) = A(x 1) (A is a constant) are continuous.
Proof. (1) For (x) Az log (KL-divergence), since 7 is finite set, rewrite the training rule & as an optimization problem as follows:
(.), ALLMa( log (LMg(x))
arg max LLMs
Σ
LLMg x
Σ
LLM (x)
MET*
Σ (L =1
*ET-
LLM, (w) 0 Yo TT"
Since for KL divergence, the optimal model LLM, must satisfy that LLM, (x) > 0, for all T The necessary condition for optimal 0 is that there exists μ R. such that
LLMg (x) A = P Vo T Σ LLM)(x)
Σ w.rm((x) Alog
LLMa... (x)
Similarly, for the input (rm', w²)) there exists μl E such that the optimal H' satisfies
I. (xx) x 1L Vx B T
(x) - A log
Σ
LLM =
LLM (z)
TET.
24

---

For convenience, we define A(x) = Ei=
w (x) - Ei= (.).(()). Then the relationship
between (L) and L (z) given by
LLMg (x) LLM(((()))
Note that we also have the condition
(z) Σ
[LM =1
MET
Since Engr. LLMo ( ( et(u-n') EOET. (L A(t), we can infer that
1 = cx(x-u)
LLMo(z)) (())) S ex(u-u))
max ex A(x)
aET
#ET*
1 = )
(LMg et(n-n") #ET*
min e e
.ET
This is equivalent to
min ( p' -p max A(x).
*ET*
Thus, the difference for LLMg(x)) and LLM (x) can be bounded by
LLMM (x) - LLM,(x)) (x) = (z
LLM
4 e (a(z)+
2A(m)
2A(a)
S max max c TET
1, max 1
2ET'
For any 8 > O, when we set maX | S min log 1-51
1 } log(1 [ + +8)} we have
24(w)
-1, max 1 - 24(w) ØET
LLMMg (x) LLMa(x)) < max max øET
5.
(2) For f2(x) = x(x as an optimization problem follows:
- 1)² (x² divergence), since T is a finite set, we can rewrite the training rule v
(.),()) A (LLM)(z) - LLM (z))
Σ
(Z) Σ
LLMa TE
LLM (x)
s.t.
(LM =1
xET
(LM,
0 Vx TT.
Since we have assumed relatively large À so that the optimal model LLMa satisfies that LLMa(x) >
for all r E T The necessary condition for an optimal is that there exists p E R, such that
warm( 2A (z) LLM (x)
P Vo ETET"
Σ LLMa(x)) = 1.
LLM (x)
BET*
Similarly, for the input (rm', 17), there exists μ E RR such that the optimal 0' satisfies
(x) 2X LLM (x) - LLM (x) 1i Vo T
Σ
LL) = 1.
LLM (I)
2ET
For convenience, we define A(x) Σn_ wxrm((x) - En (z) Then the relationship between
LLMg(x) and LLMg (x) is given by
LLMg (x) LLMa(x)+ + LLM (x)
(A(x) + + 1- p²).
21
25

---

Note that we also have the condition
LLM (x)
Σ
LLM (x) = Σ
(L)
(A(x) + ()--p) -²) =
2X
ZET
ZET-
Since Exer- (z) = L, we can infer that
LL (x)
Σ
2X
PET
This is equivalent to
1 - p Σ
ZET
Thus, the difference for LLMg (I) and LLM (2r) can be bounded by
LL (z)
LLMg (x) (LMM(()]]
+1x)+
max [4(x)
2A
ZET"
For any 8 >> 0, when we max
[xx1) S Ao, we have
1
[LMM - (LM)(
max [(xx) 8
Ā aET*
Theorem 4.9. An implementable training rule dp satisfies payment equivalence if is continuous and for Vi, rm mi W- mii there exists rm,; and 0 such that v ((m,, (rmm,,mmm)) (w;, w-s). mmi)) = O for all w, E W. In the maximum normalization case, rm, must be 1.
Proof. We prove the equivalent version of payment equivalence: For any group i, when fixing other groups reports (rm_;, w and mits any two payment rules p. p' that implement & in DSIC must satisfy that there exists a constant such that (im, w:) - P/ (rm;, w;) = c for any rm, and wi, Therefore, in the rest of the proof, we suppose fixed (rm W-.) and Omn and will omit these nolutions.
Firstly, we introduce new notation t; to represent the combination (rm,, w.). whose domain is R x W. Without specially claim, t; is used to represented for the rm; and with the same superscript and subscript, for example, t (rmf', w Then, we define the functions ((---) and V(..-) as follows. 2(t',, f;) is the change in valuation from misreporting type t to reporting type f; truthfully. formal,
t(t)+t) == (1 mm. - wiv, w.
(b(tt) m..)
And tt) refers to the smallest values of I on a finite and distinct path from E to
V(,,
inf A finite and distinct sequence =f ..... dit :=t,) ==
We prove the following lemma, which is a special case in Heydenreich et al.
38
Lemma C.2 (Heydenreich al. (38). In the RLHF Game, an implemented training rale spp satisfies payment equivalence for any agent i, and any types t,, ty we have
Vit,, () -V (t,,tt).
Proof. Assume there is a mechanism ( p) that satisfies DSIC. For any two types 4; & and a finite and distinct sequence [t,, f..... t t.]. let to = t² and th+1 t, we have that
w! es (w(tt++) ") (.mm(+) (tt > ui +1 (x(((t))) ((((t))),m(++) mmmm))))) p((t)) 4O j < k.
This can be rewritten as w/th i+
e. r, - (o)((t 2 pt(((+t) - pi(t))
4A < < k.
26

---

Sum over j, we get the following inequality
k
*
Σ -=0
10
Σ
- -u:t rmf+1)
j=0
&
Σ j=0
-x((t) p(t+) - p(t)).
Since this holds for arbitrary finite and distinct sequences, we can infer that (i p(t,) - p((t)) Similarly, there is V (tt, (2) 2 p((() p(t:). Combining these results with V (4a, -V(, ti). there is
((t,, (.) -V (t,, p(t)) - p(t.) S t(t,,
which means that p(t)) p(t)) (ts,, t.). Note that this holds for arbitrary t, and t. Therefore, when for some t., the payment p(t,) is determined, then the payment for all other f,s is determined. For example, if there are any two payment rules and p' both implement db in DSIC, and we set the payment when reports preference rm defined in Equation (5) and w; = 1 as p" and p'* respectively, then Vt,
pi((:) - p(()))
(p:(1:) p²) - (v1(4)) - -0))+++ + p"
=V((m, 1),ta) V((m, 1), t.) +p" pd" -p"
Note that p" and p'* are not influenced by i's report, but they may vary for different mi Wi and mn,, which means that we can consider the term p" - p'" as a function f on (rm mmi))
Then, we show that the training rule satisfying the conditions in Theorem 4.9 is sufficient for the condition stated Lemma C.2 Firstly, we show that for any i;, E. we have V (t,, ()) + (4,,4)) Z O. By definition of the function V (;, (t,,, ft) and (tt,, f2) correspond to the shortest path from fę to t. and from E to t, respectively, which means that [tt,, f + ((4,, t.) is the shortest weight for eycle that goes through and t/. Since the SW-Max training rule is implementable, we know that the weight for any cycle is non-negate non-negative by cycle monotonicity (65). Therefore, ((t,, E)) + V t.) 2 must be satisfied.
Then we show that for any 4;; 12 and > O, V(Le, (4) + V(G, 4.) S finite and distinct sequence [t,, t!....,t th, [l] such that
We prove this by constructing a
Σ ( + Eiee (((t).))+ 1=0
This suffices for proving V(t,, () + (( t.) S since (t,,, t) and V (tt, fr) are the lower bound
for " 11 and " +1 ()) respectively.
Initially, we rewrite the LHS of Equation
by using the definition of the function I(:;))
101 1
uet ())
1=0
w ++ (2( ) mmm (v(())
wdo(((((")))m
1=0
mn i+l (LL.Ma++1 - LLMaa) mm i+
wp (LLMas - LLM ) rm]
3=0
(LLMas+1 LLM,)) (w (w5*+ - rm?)
==0
(LLMa++(()) - (z)) rm +m++ (x) (z))
+ET
27

---

In the above equations, 0² +x(()) for 0 I I k refers to the tine-tuned model when group reports t/-
By the condition, when m--.. w_; and Oinit are fixed, there exits 8 > 0 such that max,er" (x) ( I then max [LLMo(x) LLM (z)l S
_
(in max-
4
imum normalization case, we take amfr-]), where A = (b(m,, mm_,),))....... (w,, ;)=;); mmi)) and
0' ((rmm), (wi, wi:);; Oinit).
We construct the sequence P as follows: we set l = 2n, n 2 E + 1 and let t? = t = E,y For
5
each 0 S j S n1,
w wi, m =m
mm +
rm m
And for each n 11+11 j S 2n + 1.
w = w/y
mm = mm; + +j -
1)( mm
Note that the rm; is the one given by the condition in Theorem 4.9 In this construction, any rm? is either weighted average of and rmž rm' and rm.. This ensures that all reward models in the sequence are valid (normalized by summation or maximum and non-negative). We can then divide the above equation into three parts, making the w, the same in the first and the last parts.
Σ
(LLMas+1 (1) LLM (x)) +1 "rm?* (x) (x)m((x))
j=0
Σ
Σ
w. (LLMMr++1 LLM2: (x)) +xx) rm/ (x))
(a)
T
(LLM (x)) LLM (z) (x) warm (x))
(b)
*ET"
2n
( - LLMMs (z)) +1 (x) rm (x))
(c)
j=++1zEn
We first claim that (b) equals O. This is because of the property of rm, = rm." mm+1_ rmf, which can induces LLM0a LLM
Then we turn to (a). By the construction, for any x E T and 0 S S n 1. (z)
wi mm ++ (z) < I 6, so that LLM (z) LLM (x) im holds for all Then we can derive that:
WLLLMMa: (xx) LLM (m))
i+1 (x)
wi (LLM)))))) - LLM (x)) rmf - rm, (x)
1= xET"
n-1
|m (a) rm,()))
w: 4w
n
0 RE
rm; (z) mx(r))
=ET
(rm) (I) mm)) 2 2
ZET
The case is similar to (c). By the construction, for any x E T and n + 1 < j < 2n, lm/ w/m= i+1
(x)
< @ < 6, so that LLMgs (z) LLMg (x) < am holds for all Then we can
28

---

derive that:
2n
Σ
wi (LLMa (x) - LLM (m) +1 (x)
m (a))
j=++ ZET
2n
wi (LLMas++ (x)) L.LMss (x)) mm) mm; (a)
n
j=++1 xET
2n
(x)
rm; (x)
Wi
4w
jont ZET-
4 [m) rmm (x)
=ET*
S Σ 4 (x) (x)) S
#ET*
Combining the results from (a), (b), and (c), we have that under this construction,
Σ
11 10
1=0
By the arbitrariness of this is suffice demonstrate that V(t,, (2) + V(t, E 5 Therefore, proven that
V(t, ()) +
=0.
which means that (t,, f = - VV,,, By Lemma C.2 this is sufficient condition for the payment equivalence of t.
Corollary 4.10. Each continuous training rule 1o SW satisfies payment equivalence.
Proof. We construct the reward model as follows and show that this satisfies the condition in Corollary 4.10 for when the mechanism uses SW-Max training rules.
Summation Normalization Case,
rm (x)
T
(5)
Maximum Normalization Case.
We prove this by contradiction. Assuming that there exist i, rm wh4s Oints Whs un such that
= ((m) m) (w;, wi.)) mm)) * ((m)) i)) (w w-.), Omit) =: 6'
We denote the further tie- brebbrea rule as > Then, considering the optimality of 6 we have one of the following satisfied.
ASW (rm F .), (wi, a7 w) mmit) ASW(0'; (rm), .), (wi, U8 w_,), Omit),
or
ASW(O; (rm,, m) s), (wi, W i),, 0mi) ASW(O'; (rm,, m).)), (wi, at i..,, Omit)) and LLM, > LLM
Note that (0: (m;rmm) v: (f(;; rm;), and ASW(6: (mm), , m)) (w), .) 0mi) (wl w. (o,) + (SW (rm; :), (wi, w) (. we have
AS (rm;, rm..). :), (w/, 3),, Oma) > ASW(6': (rm, mm (we,, at Omm)
or
ASW(e: (rm;,m,) ;), (w/, w w) 0ma) ASW(O'; (rm,, m) (w/, 17 i) 6mn) and LLMg >m LLM
Both cases contradicted the optimality of 0'.
29

---

Theorem 4.11. Given a continuous training rule vb E MSS and a payment rule p implements it in DSIC: p always non-negative, holds that for all r9i, W. and min
pi(ri, W. 0mii) 2 Pi
AFF (rit, W. 0mn))
If p implements &e IR, then for any > O and there exists rth-i, W-is and miis. such that for all mm; and wh
Ps (rih, W, mmit) S pPPP (ri,, W, m +..
Proof. For a continuous SW-Max training rule v, we know that it satisfies payment equivalence. By the definition payment equivalence, for any other payment rule p that also implements DSIC, there exists function yi such that
rm, w7 Oinit) = Pt AFF (rm,, W, Oinic) + gi iiniiiit).
Non-negative Payment. To ensure that p. (IT, wi, mmit) 2 0 always satisfied, we have the equivalent condition:
9: (TT) W.:. mii) 2 inf ps AFF (rm,, w, mma).
m.
However, for any mm i Oiiit when we set rm, to the uniform reward model Equation (5). have shown in the previous proof that this will not change the training outcome regardless of the value of w/ and hence does not impact the ASW This means that the payment defined by the affine maximizer is exactly O, and the RHS of the above equation will always be non-negative. Therefore, there must be 9i 2 0 for all inputs, which means that for all i, rm, W. and Oiniss we have ps (rm, Vi, Omi) 2 D F (rm, Wē Omi))
Individually Rationality. To ensure the utility of any group is not negative, have to constrain the function gi as follows:
9. (r W. W. Oun))
inf un m,
AFF (AFF(mm Wi Ounit)
where we denote us AFF the utility of group i under the mechanism. We construct an extreme case such that the RHS can be sufficiently small. Without loss of generality, we assume that 7** {111, 22). The initial model LLM (x1) ) = c, LLM (z2) = 1 e. Group has preference mm; = and (xx) O, and other groups have opposite preference: rmy (x1) 0 and my(x22) = 1 for j * i. The group size is set to W'y = 1 for all l E [n].
< < 1= Wgrmm we can directly derived from the optimal condition Equation (OPT) that the final model satisfies that (.LMg(11) S L.LM, (x1)). Since AFF always non-negative, the utility of group is most rm, (x)))) LLM (x) e. To ensure that implements v in IR, we have to set gs (rm W. O S for this case. This is equivalent to Ps (mm, W. fmit) S p. AFF (rm, W. 0mi)).
In this case, we have Et ===1
F w...re
D Omitted Proofs in Section
Lemma D.1. For any rm, rm', if maxxeT. [rm(#) rm' (xx) = e, then for any model 0, we have
[o(0; - v(0; rm') s
Proof. We can derive that
rm) v(O (0m)) =1 Σ -LM)())(rm())mmmmmll <
Σ
(LM((z))l((a)) rm'(x)
#ET-
WET"
Σ
LLM,
*ET
Lemma D.2. Assume that for any noisy input rm generated from F(-Im)) and E [n], there (x)
max |Fm
rm; rm:(x)) S
30

---

Then for any vo wew and rm generated from F(-(rT), the distance between the training outcome and the optimal is bounded by:
ASW AS (rm 17 Omni i U. 0inir) 2
ASW()(Tit ú. mma)) pm, iimis) 2e
Σ wi.
Proof. Let 0 = (im, U7, miit and = (rm, v7, mmn)) 0 is the optimal parameter for biased input, and is the optimal parameter for the true input.
ASW(O; W7, 0mi)) = (0;r(0;rm,) (
Σ we (0) -c) (LLM, LLM 1=1
16
ASW (0: m., w., mma) Σ W/e
ASW(0: r0, wú Omi)
WyE
w.ey(O,mm.) (LLMM) |LLM|
Σ
Wie
(3)
n
w; (0; -)) (LLMM [LLM
Σ
WiE
i=1
n
ASW(O; m,, ni, Omi)
wye
(1) and (3) can be directly induced by Lemma D.1 and (2) holds by the definition of é.
(5mm Wi, Omis) = arg max ASW(O; m,, U7, Omni).
dE0
Theorem 4.12. Assume that for any noisy input rm generated from F(. and i E [m], there is (mm;(x)) - rm: (x)) <
max
WET
Then, with training rule s SW (1b, (FFF ensures that each group can benefit at most 2w0e from misreporting the reward model.
Proof. Recall that the calculation of payment in p AFF is
AFF (rm, w. Omi) ASW (SSW-((h -:(1/ (mm w.i, Omi)) rm, W. Omn) ASW (.WW:(bb(rhi)) W7 Omit); rm, W. Omni) Ps
Let i7 = (w,, w)),, the utility function can be written as:
ui u,((m/) m) w: th p. rm,, wx) (. rm.)) pi AFF ((mi), m. uT, Ornis
(0; rm.)) ASW - rm, úi, Omin ASW (00; rn, w. 0mn)
ASW (0: en, W. Pmm) ASW ( rm, WT Omit)
where we define 0 ((mm,, rm :): प,, 0mm)) and 0_, (ir -- Omnn). Note that the term ASW_,(0 1i, w7, ( is not influenced by the change of rm, or w..
31

---

Therefore, we can derive that for any rm-s. Wi, let 8_; = si(mm 28-1. Omitt)
E (-mmm))
[u:((rm; i) ūt th P. rma; w:) + ASW-SWW-- NSSW-((-:: (0-x;; rm, w7, 0mai
Em ~Fi (-mmm))))
ASW rm, प,, Bmit)
= -Bmnn~ (F(()(mm)))
1aa (0: rm.) + Σ
(, - (
(1) Exitg Fi( (mmm
w.v; rm:) +
Σ
(6; ( (LLMMM,
Wic
jzz
(2) Brins Fiftlam
w.i; (0,;(0 (0:rm, rm.)
+
Σ
(0; - (LLLLM|||LMMM
Wye
(3) F.m. (-(mmm)))))
W (0; rm:) +
Σ
(0, ( DDLLLMM/IILL.M...
200
jtL
Bri F((mmm))
w (0: rm.)
+
Σ
(LLMMM) 2w0e
itti
Emm, (-((m(())
w.v; (0: rm:))
2wec
its
-Em,,~A ASW(O: rm, W, 0mmi)]
2wie
-Fm (u.((mm) m). Wh v.P, rm,, wi) + ASW (0-.-tt_, w. 0mi))] -2i5e
All the in the above inequalities refers to the optimal parameter for input (rm, r) Wi, Ouit- ie. Ô = ((M. mm.)), W. Oiitt). Specifically, (I) and (3) come from the bounded distance be- tween rm; and rm; (Lemma D.1 (2) and (5) hold by the definitions: 0 (b((r,,, (mfm,,,,m))))) rm- ui, Omit) arg me ASW(0': (rm;, rm-;), W. Omn) and 0 = (((m,, rm-;). uł, Omit) = arg max ASW(0'; (mm, mm_4), W. (n And (4) holds since the inner term is irrelevant rm,-
Therefore, we get
U.mmm m. :), wi v. rm;, wi)
-Ba - m- (11,m, :))
Uu (rm, ii; th r,; w:)
= Em Bm.
Es mm (m))) H: ( rmm :),, Wi; 1h p. rm,, w:))
> >m (((m()
Ea mh
rf ()) nn (fm,, rm_ Wi; D. rm;, wi) 2wie
nh ( पh u.p. nm, wi) 200E
-U ((mL), ;) W: 1h p. r,; w:) 2wie..
E Further Discussion on General Training Rules
In practice, some other training principles do not belong to SW-Max training rules, including those that maximize the Nash Social Welfare and focus more faimess issues, like MaxMin-RI.HH [I]. As an initial study on the incentive property of the RLHF Game, we primarily consider the mainstream training rules, SW-Max training rules, that aim maximize social welfare under certain regularization.
32

---

Therefore, analyzing the properties of general forms of training rules is out of the scope of this paper. However. we also make a preliminary step for analyzing the two questions proposed Section 4.2 The second question is partly included in Theorem 4.9 and for the implementability of training rule, we utilize the notion of cycle monotonicity proposed by Rochet [65]. which is generalized version of monotonicity defined in a notation & to represent the combination of (rm,, wi) with the same superscript and subscript. We define the function 1(44 44; [,, Omi) := wayy (b), [-e), Omiit; rm.) uoqus(( [-i, Omna);; rm;) to measure group i's valuation gains from misreporting (t,) to truthfully reporting (tç) under f_, and 0m. The cycle monotonicity is defined based on this function:
scenario (53. In the RLHF Game, we use the
Definition E.1 (Cycle Monotonicity). The training rule 1p satisfies cycle monotonicity if for any group i, t4, E2 E R x W. any finite, distinct sequence of reward models [tz, e. f................ t {.] (k 2 0), and any f-i,, mit defining t! = t x+42 := t, and gh+ := t/ we have
K+1
Σ
10t 2 1- 0ma) 2 O.
For general training rules, cycle monotonicity is a sufficient and necessary condition for imple- mentability.
Proposition E.2 (Rochet 651 A training rule sb is implementable if and only if it satisfies cycle monotonicity.
Proof. We fix the other groups' report rm-,, W-.,, Omni and also omit their notations for simplicity.
We first prove the necessity: if o is implementable, it satisfies cycle monotonicity. Since &b is implementalle, there exists such that (v.p) satisfies DSIC. We use notation & to represent the combination of (rm², w For any types ti E R x W, any finite and distinct sequence of types and 15+1 l.. By the property of DSIC,
[4, (, G, 1/ (.), k 2 let &f d+t2 have
with (v () 3m + - px こ u. + a (w (+))) - (t()) vo +1
By definition of the function I, this is equivalent to (tt, ž p.(t! - p(())) v0
Sum over all j, we get
k+1
k+1 (,((((-)-((())) O.
Σ j=0
==0
By the arbitrariness of the sequence [1,, 1/ .2,..., ity.
...,.,, 1 (4.. this means that & satisfies cycle monotonic-
Then, we prove the sufficiency: By cycle monotonicity, we have that for any finite and distinet sequence [ti, f2 i,,...,, th,,,,/,,, 4/.
k+1 >
Σ 1=
10 ( 0.
By the arbitrariness of the sequence, we can infer that
V(G, + +1t,) (.) 2 0.
Since i(t/, tx) is bounded, V(t,, t is also finite and V(t, +) 2 -I(t), t(). Then, we can establish a payment rule p such that for any agent i,
pr((,) = (((,t)).
where t" E R x W is certain type.
33

---

Then, for any t, = (rm;, wi). we have
(.))))(): pa(())
= (..)()(()):))) v(t",t)
m. (w))() +1(1, 4) - -V((,,)
w, rm.) ((1,))
- -. rm.) (())
Note that (2) comes from the definition of V that:
V(t* te)
inf finite and distinct sequeence
Σ (t//. (22)
(c) gond smmss
j=0
k Σ
inf S finite and distinct sequence
u10, +1(44)
-o a to th a =a jm0
Vt",, 1 + t(t).
This means that mechanism (b)p) satisfies DSIC, and suffices to show that a is implementable.
Validating whether a training rule satisfies cycle monotonicity is a complex task. Thus, finding more concise condition that induce the implementability for general training rule subset of training rules is a valuable further direction.
F
Additional Experimental Results
Synthetic RLHF Game We construct synthetic RLHF Game: We set the group number to be 5 and assume the size of the outcome space to be 10. Each group's preference first sampled from a uniform distribution Ulo, 1 and then normalized. The group sizes are uniformly sampled from (1,2....,10) 10
We consider the misreporting strategy that used to prove Theorem 4.2
Specifically, given group's preference rm. We first find the most preferred and the least preferred outcome # arg max rm rm)).. 22 arg min mx))). Then we set the reported reward model to be rm rm(x1) e. r x22 rm (22) - e, and mm()) = rm(z) for other æs.
Table 1: Average changes in valuation and utility when adopting the misreporting strategy from Theorem 4.2 holding other groups' reports fixed. The parameter controls the extent of deviation from truthful reporting. As shown in the table, such misreporting strategy brings valuation gain but decreases the utility.
Reporting Parameter
Type
0.001
0.002
0.005
0.01
0.02
0.05
0.1
Mean
0.1073
0.2096
0.4881
0.8667
1.3674
1.7978
1.8154
Valuation (#1e2)
Std
< 0.0001
<.00001
0.0003
0.0004
0.0013
0.0026
0.0032
Mean
-0.1064
-0.4135
-2.3696
-8.1557
-23.7334
-53.1552
-55.8977
AUtility (+1e4)
Std
< 0.0001
0.0001
0.0011
0.0046
0.0196
0.0573
0.1415
We let group use this strategy and maintain the other group truthfully reporting. The payment is set according to the mechanism introduced in Section 4.2 We take 100, 000 samples and the average change in valuation and utility for group is reported Table The result shows that such a strategy can indeed improve the valuation and is, hence, beneficial when there is payment. However, with the introduced payment, no strategy will bring higher utility than truthfully reporting.
More Complex Preferences. The experiment setup of this part follows the Section two scenarios with more complex, multiple preferences.
We consider
34

---

1. We simulated scenarios from data reported in [70] (Table 6), involving three groups, each valuing helpfulness, harmlessness, and humor, respectively. Normalization and other settings follow our paper. The true group sizes and the numerical results are shown in the tables below.
2. We examined scenario where the group's preference is linear combination of two reward models. Specifically, group values 0.2 x Helpfulness + 0.8 x Harmlessness, group values 0.8 x Helpfulness + 0.2 Harmlessness, and group 3 values Humor.
All of the above results show that truthfully reporting is among the optimal strategies under the mechanism.
Table 2: Valuation, utility, and social welfare outcomes when varying reporting parameters for Group 1, with other groups' reports held fixed (a = 1 means truthful reporting). Group sizes are set as (w, W2, W2) = (3, 1). The three groups value Helpfulness, Harmlessness, and Humor, respectively, The highest value in each row is highlighted in bold.
Reporting Parameter a
0.2
0.5
1
1.5
2
Valuation
0.00
0.79
2.66
3.00
3.00
3.00
Utility (= Valuation-Paynan Payment)
0.00
0.44
0.57
0.50
0.50
0.50
Social Welfare
2.51
2.94
3.08
3.00
3.00
3.00
Table Valuation, utility, and social welfare outcomes when varying reporting parameters for Group 1, with other groups' reports held fixed (a = 1 means truthful reporting). Group sizes are set as (wi, w2, w3) = (4, 5, 3). The three groups value Helpfulness, Harmlessness, and Humor, respectively. The highest value in each row is highlighted in bold.
Reporting Parameter a
0.2
0.5
1.5
3
Valuation
0.00
0.00
1.05
1.05
3.54
4.00
Utility (= Valuation Valuation-Payment) Social Welfare
0.00
0.00
0.43
0.43
-1.83
-2.51
6.51
6.51
6.94
6.94
4.68
4.00
Table 4: Valuation, utility, and social welfare outcomes when varying reporting parameters for Group 1. with other groups' reports held fixed (3 = 1 means truthful reporting). Group sizes are set as (wi, w2, wa) = (5, 2). The three groups value Helpfulness, Harmlessness, and Humor, respectively. The highest value in each row is highlighted in bold.
Reporting Parameter 3
0.5
0.8
2
Valuation
0.00
0.33
1.31
4.67
5.00
5.00
Utility (= Social Welfare
Caluation-Paymenn
0.00
0.09
0.20
-0.72
-1.01
-1.01
6.01
6.10
6.20
5.29
5.00
5.00
35

---

Table 5: Valuation, utility, and social welfare outcomes when varying reporting parameters for Group 1. with other groups' reports held fixed (3 1 means truthful reporting). Group sizes are set as (w), W2, w23 = (3, 1, 4). The three groups value Helpfulness, Harmlessness, and Humor, respectively. The highest value in each row is highlighted in bold.
Reporting Parameter 3
0.5
0.8
1.5
2
3
Valuation
0.79
0.79
0.79
0.79
2.66
3.00
Utility (= Valuation- Payment)
0.79
0.79
0.79
0.79
-1.04
-1.58
Social Welfare
5.37
5.37
5.37
5.37
3.54
3.00
Table 6: Valuation, utility, and social welfare outcomes when varying reporting parameters for Group 1, with other groups' reports held fixed (a means truthful reporting). Group sizes are set as (Wi, W2, wa) = (2, 3, 1). The three groups value 0.8 x Helpfulness + 0.2 x Harmlessness, 0.2 Helpfulness + 0.8 x Harmlessness, and Humor, respectively. The highest value in each row highlighted in bold.
Reporting Parameter a
0.2
0.5
1.5
2
Valuation
0.53
0.53
1.03
1.51
1.60
1.60
Utility (= Dalluation-Paymenn
0.52
0.52
0.61
0.39
0.31
0.31
Social Welfare
2.92
2.92
3.01
2.79
2.71
2.71
36

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We are sure that the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
37

---

Justification:
The discussion is put in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
All assumptions and rigorous proofs are provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The full information combined with the code is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
38

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use open-source datasets, and some data is simulated from certain distribu- tions, which are described in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cc/publcc/eee.c
www.ded/eeooeeuummmmccoo ssi ssionyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
39

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All details are provided.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: For the deterministic numerical simulation, there are error bars. For others, we have provided clarification on the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All information is provided in the README file of the code.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
40

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwwpp..ccooo.......ccccaa
Answer: [Yes]
Justification: We have checked.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
41

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
42

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy /ettps://neurr cc/ //onfffrencees22255555 for what should or should be described.
43


Reviews:
[{"review_id": "BMZcAMbAk4", "reviewer": "Reviewer_gqdi", "summary": "This paper explores the setting where an LLM provider offers the service to fine-tune a base LLM in accordance with the preferences of a number, $n$, of groups each comprising some (perhaps different) number of agents. In this setting, each group provides a reward model, $\\mathrm{rm}_i$ which takes as input a prompt response pair and returns a number quantifying the group's satisfaction with the response according to their preferences. The agent's valuation of the given model is thus its expected reward across the prompt-response pairs. The provider thus wishes to fine-tune the model in accordance with the preferences of the groups.\n\nIn this context, the provider first announces the training rule and the payment rule. Then, the groups report their desired reward model and group size. The provider then fine-tunes the model and charges each group according to the payment rule. It is assumed that each group assesses the utility of the model as the difference between the valuation (times the number of individuals) minus the payment. The paper studies the possibility for groups to misreport their reward model or number of individuals in order to increase their benefit at the behest of the aggregate performance. \n\nTowards this end, it is illustrated that, in the absence of a payment rule, there exists a misreporting strategy which yields a strictly higher utility regardless of the other groups' choices in most cases of interest. Next, it is shown that the so-called affine maximizer payment rule satisfies the propert that  truthfully reporting one's reward model will yield the highest utility regardless of the other group's choices, a property called dominant-strategy incentive compatibility (DSIC).  Next, for continuous training rules, a\nconnection between all payment rules satisfying the DSIC property is derived. Finally, the effect of noise on the reported reward model is studied and a bound on the benefit under the affine maximizer payment rule is derived.\n\nThe paper concludes with some implementation details and an empirical study which serves to validate the theoretical results. Notably, it is shown that, without a payment structure misreporting can lead to a higher utility and that, under the affine maximizer payment rule, misreporting is not beneficial.", "questions": "1. It is assumed throughout that the groups have quasi-linear utilities which does not appear to be a realistic assumption in general. Indeed, if a group has a much larger budget they may be willing to trade off a lower utility for a model that better matches their preferences by increasing their payment..       \n\n2, Related to the above point, I wonder if the notion of DSIC is a realistic aim to pursue. Indeed, it appears that, under DSIC, the underlying model cannot be modified too much in response to a group's preferences perhaps even if there is a consensus between multiple groups. Is this impression correct?  \n\n3. There are a few small issues with the writing in the current version. For instance, the statement of Theorem 4.3 does not appear to be complete. Similarly, the last line of the paper is not complete. I highly recommend that the author's give the paper a few more passes to improve clarity.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, I find the paper to be generally well written. The submission addresses an important question when aggregating user's preferences and provides some principled theoretical results. While the experimental validation of these results is somewhat limited in scope, I see the main results of this paper to be at the theoretical level.\n\nMy main questions about this paper revolve around some of the assumptions which I believe to be limiting, see the questions below.", "was_revised": true, "final_justification": "The authors have adequately answered the questions I had for this submission and alleviated my concern regarding the DSIC property. I believe that these points should have been addressed more clearly in the original submission. I maintain the initial score.", "rebuttal": "We thank the reviewer for the insightful feedback and constructive suggestions. We address the specific points below.\n\n--- \n**Weakness 1**: About the assumption on the quasi-linear utility. \n> It is assumed throughout that the groups have quasi-linear utilities, which does not appear to be a realistic assumption in general. Indeed, if a group has a much larger budget, they may be willing to trade off a lower utility for a model that better matches their preferences by increasing their payment.\n\n**Response**: We understand the reviewer's concern regarding the assumption of quasi-linear utility, and we appreciate the opportunity to clarify our modeling choices. There has, in fact, been a long debate in the economic literature on which utility function to use. We adopted quasi-linear utility because it is **more common in mechanism design theory, and we believe it is suitable for this initial work studying this scenario.**\n\nWe acknowledge that in certain scenarios, groups might focus solely on the final outcome regardless of payment. **This case also induces strategic misreporting, as it is exactly the same as a mechanism without payment.** Our Section 4.1 discusses strategic behavior under such mechanisms, and the results (Theorems 4.2 and 4.3) show that commonly used objectives, like SW-Max training rules, cannot guarantee truthfulness. Further exploration in this direction could consider the design of training objectives for RLHF, which is an interesting and valuable avenue for future work. Overall, we believe the results presented in this paper serve as a crucial first step in this area, offering valuable insights that can inform these subsequent extensions.\n\n--- \n**Weakness 2**: About the DSIC property. \n> It appears that, under DSIC, the underlying model cannot be modified too much in response to a group's preferences, perhaps even if there is a consensus between multiple groups.\n\n**Response**: We appreciate the reviewer's comment regarding the DSIC property and its perceived implications for model modification. We would like to clarify that the DSIC property **does not restrict how much the LLM policy can be modified** in response to group preferences.\n\nThe extent to which the LLM policy shifts is **determined solely by the training rule**, not by the payment rule. Our theoretical analysis focuses on characterizing the payment rules that can implement DSIC for fixed training rules. Consequently, our results don't involve altering the training objective or the training process itself. The DSIC property simply ensures that groups are incentivized to report their preferences truthfully.\n\nFor instance, consider the **SW-Max training rule** defined in our paper. This includes a **regularization penalty** to constrain deviations from the initial model, a common practice in RLHF. In such a case, even if multiple groups reach a consensus, the regularization, as part of the training rule, might limit how much the LLM is modified. Crucially, within our framework, the choice of payment rule (and whether the mechanism is DSIC) does not affect the learned model.\n\n--- \n**Weakness 3**: About Theorem 4.3. and the Conclusion. \n> There are a few small issues with the writing in the current version. For instance, the statement of Theorem 4.3 does not appear to be complete. Similarly, the last line of the paper is not complete. I highly recommend that the authors give the paper a few more passes to improve clarity.\n\n**Response**: We thank the reviewer for highlighting these important issues.\n\nRegarding Theorem 4.3, we intentionally presented a simplified version in the main text to enhance readability and convey the core conceptual insight more effectively. This simplified statement aims to communicate the intuition that the SW-Max training rule is susceptible to strategic misreporting. **The full technical version of this theorem, including the concrete form of the function $t(\\cdot)$, is provided as Theorem B.2 in the appendix.** \n\nTheorem 4.3 is stated as follows. \n>In the RLHF Game with mechanism $(\\theta^\\ast, p)$ that $\\theta^\\ast \\in \\Psi^{SW}$ and $p \\equiv 0$, when $f$ is strongly convex and $C^2$-smooth, there exists a function $t$, when $t({x}, \\overrightarrow{\\mathrm{rm}}, \\vec w, \\theta_{\\text{init}}) \\neq 0$ for some ${x} \\in T^{\\ast}$, truthfully reporting is not the optimal strategy. \n\nTheorem B.2 is stated as follows. \n>In the RLHF Game with mechanism $(\\theta^\\ast, p)$ that $\\theta^\\ast \\in \\Psi^{SW}$ and $p \\equiv 0$, when $f$ is $\\alpha$-strongly convex and $C^2$-smooth, suppose group $i$ has preference $\\mathrm{rm}\\_i$ and group size $w_i$, other groups report $(\\overrightarrow{\\mathrm{rm}}\\_{-i}, \\vec w\\_{-i})$ and the initial model $\\theta\\_{\\text{init}}$, we define \n$$\nt({z}) := \\sum_{{x} \\in T^*} \\frac{(\\mathrm{rm}\\_i({z}) - \\mathrm{rm}\\_i({x})) \\mathrm{llm}\\_{\\text{init}}({x})}{f''\\left(\\frac{\\mathrm{llm}({x})}{\\mathrm{llm}\\_{\\text{init}}({x})} \\right)}, \n$$\nin which $\\theta = \\psi(\\overrightarrow{\\mathrm{rm}}, \\vec w, \\theta_{\\text{init}})$. When $s_i \\geq 2$ and $\\underline{\\mathrm{rm}_i} = 0$: \n> - For the maximum normalization case, if there exist ${x}_1 \\in T^*$, $t({x}_1) \\neq 0$ and $0 < \\mathrm{rm}\\_i({x}_1) < 1$, truthful reporting is not the optimal strategy. \n> - For the summation normalization case, if there exist ${x}_1 \\in T^*$, $t({x}_1) < 0$ and $0 < \\mathrm{rm}\\_i({x}_1) < 1$, truthful reporting is not the optimal strategy. \n> - For the summation normalization case, if there exist $x_1 \\in T^\\ast$, $t(x_1) > 0$ and we can also find ${x}_2 \\in T^*$, such that $1 > \\mathrm{rm}\\_i({x}_1) \\geq \\mathrm{rm}\\_i({x}_2) > 0$ and $\\frac{1}{\\mathrm{llm}\\_{\\text{init}}({x}_1)} f''\\left(\\frac{\\mathrm{llm}({x}_1)}{\\mathrm{llm}\\_{\\text{init}}({x}_1)}\\right) < \\frac{1}{\\mathrm{llm}\\_{\\text{init}}({x}_2)} f''\\left(\\frac{\\mathrm{llm}({x}_2)}{\\mathrm{llm}\\_{\\text{init}}({x}_2)}\\right)$, truthful reporting is not the optimal strategy. \n\nTo further improve clarity and prevent any misunderstanding, we will modify the title of Theorem 4.3 in the main text to explicitly state its relationship to Theorem B.2 in the appendix.\n\nConcerning the conclusion, we acknowledge that an error occurred during the manuscript upload process, resulting in an incomplete final sentence. We assure the reviewer that this will be corrected in the final version."}, {"review_id": "RPsMtrUnWz", "reviewer": "Reviewer_AgWA", "summary": "This paper investigates the incentive issues that arise when fine-tuning large language models using multiple reward models contributed by different agents. In such a scenario, agents may strategically misreport their preferences to manipulate the aggregated training objective. The authors model this as a mechanism design problem, where the LLM provider defines both a training rule and a payment rule. They show that naive fine-tuning without a payment rule is vulnerable to strategic misreporting. To address this, they propose extending the VCG mechanism to LLM fine-tuning by introducing an affine maximizer payment rule that ensures dominant-strategy incentive compatibility (DSIC) and individual rationality (IR). The authors further prove payment equivalence under certain conditions and demonstrate robustness to valuation noise.", "questions": "Q1: How would the proposed mechanism extend to DPO-style preference reporting, where groups only provide pairwise comparisons rather than scalar reward functions?\n\nQ2: How sensitive is the payment outcome (and DSIC property) to approximation errors in the hybrid model set Θ′ used in simulations? Would more expressive Θ′ increase DSIC robustness?", "limitations": "The paper assumes access to normalized, scalar-valued reward functions. In practical RLHF scenarios, preferences are often noisy, multi-modal, or provided in pairwise form (as in DPO).", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength.\n\n- **Novel formulation.**  The paper is among the first to formalize the incentive misalignment in multi-preference LLM fine-tuning as a multi-parameter mechanism design problem.\n- **Theoretical rigor.**  The authors provide comprehensive theoretical analysis, including necessary and sufficient conditions for DSIC, IR, and payment equivalence, extending classical results to the context of RLHF with regularization.\n- **Practical insights.**  The affine payment mechanism is empirically validated on real-world LLM fine-tuning setups (e.g., LLAMA2-7B).\n\n---\n\n## Weaknesses.\n\n- **W1: No consideration for model performance degradation**\n\nWhile the paper proves DSIC under the affine payment rule, it does not analyze whether this mechanism leads to degradation in final model quality compared to the no-payment baseline. There may be a trade-off between alignment fidelity and incentive robustness.\n\n- **W2: Scalability concern for payment calculation**\n\nComputing the affine payment requires training n+1 models (or approximating them), which can be costly when the number of agents is large. Although heuristics are proposed (Section 4.4), their theoretical and empirical impacts on DSIC guarantees are not deeply analyzed.\n\n- **W3: Limited experiments**\n\nThe empirical experiments only simulate two-agent settings. It is unclear whether the proposed mechanism remains robust in more complex settings involving many diverse agents with conflicting preferences. And I think the current experiments are not enough to support the author's claims in total.\n\n- **W4: Limited human-in-the-loop validation**\n\nThe framework is motivated by human feedback, but its evaluation is still largely synthetic or based on pre-collected datasets. A live human evaluation or annotation study would greatly strengthen the argument for practical effectiveness.\n\n- **W5: Presentation issue**\n\nThe conclusion section is incomplete and fails to summarize key findings, discuss limitations, or outline directions for future work. This weakens the overall narrative of the paper and leaves the reader without a clear takeaway or understanding of the work’s boundaries.", "was_revised": true, "final_justification": "I believe the authors' rebuttal has addressed most of my concerns. After carefully reviewing the interactions between the authors and other reviewers, I think a score of 4 is fair, so I will maintain my initial score.", "rebuttal": "We thank the reviewer for the insightful feedback and constructive suggestions. We address the specific points below.\n\n---\n**Weakness 1**: About the trade-off between alignment fidelity and incentive robustness. \n\n**Response**: We would like to clarify that **incentive robustness does not negatively impact alignment fidelity in our framework.** Alignment fidelity is determined by the alignment objective (i.e., the training rule) and the learning algorithm used to optimize it. **Our work focuses on characterizing the landscape of payment rules that implement a fixed training rule under DSIC.** These payment rules ensure that groups truthfully report their private information, and this information directly contributes to correct alignment. Since we impose no additional restrictions on either the training objective or algorithm, the resulting alignment fidelity remains unaffected.\n\n--- \n**Weakness 2**: The impacts on DSIC guarantees of the approximated payments.\n> Computing the affine payment requires training $n+1$ models (or approximating them), which can be costly when the number of agents is large. Although heuristics are proposed (Section 4.4), their theoretical and empirical impacts on DSIC guarantees are not deeply analyzed.\n\n**Response**: We appreciate this insightful question regarding the computational cost of affine payments and the theoretical implications of our proposed heuristics.\n\nFirst, let's clarify the affine payment structure. For group $i$, the payment $p_i$ is defined as:\n$$p_i = \\max_{\\theta' \\in \\Theta_1} \\sum_{j \\neq i} v_j(\\theta') - \\sum_{j \\neq i} v_j(\\theta)$$\nwhere $\\theta = \\arg\\max_{\\theta' \\in \\Theta} \\sum_j v_j(\\theta')$ is the model obtained from the main fine-tuning process.\n\nIn our original theoretical formulation, $\\Theta_1$ is set to the entire model space, i.e., $\\Theta_1 = \\Theta$. This full optimization guarantees DSIC and ensures payment non-negativity. However, finding an exact optimal point in the entire model space necessitates an additional training run for each group $i$, leading to $N$ additional training processes for $N$ groups. This computational burden is precisely the concern the reviewer raises.\n\nTo address this, we introduced two heuristics in Section 4.4:\n\n* **Heuristic 1: Restricting $\\Theta_1$ to Intermediate Models.**\n In this heuristic, we restrict $\\Theta_1 = \\Theta' \\subset \\Theta$, where $\\Theta'$ comprises intermediate models saved during the main training process that yields $\\theta$. Crucially, $\\theta \\in \\Theta_1$ is maintained, which preserves payment non-negativity. Since the models in $\\Theta'$ are themselves dependent on group $i$'s report (as they are part of the main training process), DSIC is no longer strictly theoretically guaranteed. **However, we emphasize that the nature of this dependency, from group $i$'s report to the specific models stored in $\\Theta'$, is highly complex and non-trivial. This complexity makes it exceptionally difficult for a group to strategically compute a profitable misreporting strategy.** While strict DSIC is not guaranteed, the practical difficulty of manipulation suggests that groups are still likely to report truthfully. This heuristic offers the significant advantage of incurring no additional training costs.\n\n* **Heuristic 2: Early-Stopped Training for $\\max_{\\theta' \\in \\Theta_1}$ Term.**\n Here, we perform an early-stopped training to compute the maximum in the first term, defining $\\Theta_1 = \\Theta'' \\subset \\Theta$. Similar to Heuristic 1, we ensure $\\theta \\in \\Theta''$ to maintain non-negativity. The key distinction is that this early-stopped training is specifically constructed to be **independent of group $i$'s report**. Therefore, **under this method, DSIC remains theoretically guaranteed.** This heuristic introduces a slight additional training cost, but it fully preserves the DSIC guarantee.\n\nIn conclusion, Heuristic 1 offers a computationally efficient solution with no additional training, accepting a relaxation of strict DSIC due to the practical difficulty of profitable manipulation. Heuristic 2, while incurring a minor additional training overhead, fully upholds the DSIC guarantee. We believe these two heuristics offer a practical trade-off spectrum between computational efficiency and theoretical guarantees, allowing for flexibility based on application requirements. \n\n--- \n**Weakness 3**: Limited experiments. \n\n**Response**: Due to space constraints, we included only the two-group experiments in the main paper. However, **additional experiments are provided in Appendix F**:\n\n1. We conduct a synthetic RLHF Game scenario to validate the misreporting strategy constructed in the proof of Theorem 4.2. The results show that such a strategy can improve utility when payment is absent. \n2. Based on the dataset used in Section 5, we conduct two scenarios with more complex, multiple preferences. In that setup, there are three groups, and each group's preferences are a linear combination of different reward models. \n\nWhile our experimental results support the theory, the main focus of this work is **theoretical analysis**. The empirical studies are intended to illustrate and validate theoretical claims.\n\n--- \n**Weakness 4**: Limited human-in-the-loop validation. \n\n**Response**: We agree that live human evaluation would provide stronger validation. However, such evaluations are costly and extend beyond the scope of this work, which primarily emphasizes theoretical foundations. Therefore, we mainly conduct simple experiments based on the previous empirical papers on multi-objective RLHF. \n\n--- \n**Weakness 5**: About the presentation issue. \n\n**Response**: We thank the reviewer for pointing this out. The issue in the conclusion section was caused by a formatting error during upload. This will be corrected in the final version.\n\n\n--- \n**Question 1 and Limitation**: The transformation to DPO-style preference reporting.\n> How would the proposed mechanism extend to DPO-style preference reporting, where groups only provide pairwise comparisons rather than scalar reward functions?\n\n**Response**: Adapting our proposed mechanism to DPO-style preference reporting, where groups provide pairwise comparisons instead of scalar reward functions, presents interesting challenges and opportunities. Currently, we envision two primary formulations for this extension:  \n\n1. We assume that each group possesses an underlying, latent reward function, but the groups' reports are expressed as pairwise comparisons. These comparisons are then used to train a reward model for each group, and these learned reward models subsequently guide the RLHF process. This approach aligns more closely with the existing framework of our paper. Therefore, we believe our mechanism could be applicable with slight modifications, primarily by defining the payment rule with respect to these inferred reward models. The key challenge here lies in establishing an accurate and robust mapping from the pairwise comparisons to a reward model that faithfully reflects the group's true preferences, as the inferred reward model may not perfectly capture the underlying reward function.\n\n2. In this formulation, the pairwise comparisons are used directly within the DPO training framework, without an explicit intermediate reward model. Since DPO fundamentally alters the objective landscape, our current mechanism is not directly applicable. For our mechanism to extend here, the critical step would be to reformulate how these direct pairwise comparisons translate into influence on the LLM policy in a way that allows for \"charging\" or incentivizing each individual comparison provided by the group. This would require a novel design for incentive compatibility in a purely comparative setting.\n\nWe acknowledge that a full exploration of DPO-style formulations is beyond the scope of our current paper. However, we believe that our results can inspire and inform solutions for these extended settings, particularly by considering the two formulations outlined above. The discussion part will also be added to this paper. \n\n--- \n**Question 2**: About the impact of using an approximated $\\Theta'$. \n> How sensitive is the payment outcome (and DSIC property) to approximation errors in the hybrid model set $\\Theta'$ used in simulations? Would a more expressive $\\Theta'$ increase DSIC robustness?\n\n**Response**: In our experiments, we primarily consider the case where $\\Theta'$ is fixed and independent of groups' reports. This ensures that DSIC holds regardless of the expressiveness of $\\Theta'$.\n\nThe **expressiveness of $\\Theta'$ mainly affects the training outcome**. Intuitively, a more expressive $\\Theta'$ better approximates the original space $\\Theta$, leading to a training outcome that's closer to the universally optimal one. **The choice of $\\Theta'$ also affects the payment, and the direction of this impact is ambiguous.** Increasing the expressiveness of $\\Theta'$ could either raise or lower payments.\n\nWe also note a potential fairness concern if a platform deliberately adjusts $\\Theta'$ to influence groups' payments. This highlights an important area for future investigation."}, {"review_id": "ubFboA0QaM", "reviewer": "Reviewer_N7K8", "summary": "This work sought to address an important issue in LLM finetuning using human feedback data where agents may intentionally misreport their preferences to manipulate the finetuned model to their advantage. It uses a mathematical framework for investigating this issue and a mitigation strategy that adds a payment model. The theoretical results show that truthful reporting is likely sub-optimal for individual groups when only a SW-Max rule is used. It then extends the VCG payment to implement SW-Max rules in dominant-strategy incentive compatibility (DSIC). Numerical experiments are conducted to assess the proposed approach.", "questions": "1) In Section 4.2, it seems that the affine maximizer payment rule $p^{AFF}$ depends on true $\\overrightarrow{rm}\\_{i}$ and $\\overrightarrow{w}\\_{i}$. It is unclear how to obtain $p^{AFF}$ in real world settings when true $\\overrightarrow{rm}\\_{i}$ and $\\overrightarrow{w}\\_{i}$ are unknown.\n\n2) What happens when agents (from all $n$ groups) misreport in terms of theoretically results and/or empirical results?\n\n3) I have some difficulty to see the connections between the mathematical framework in this paper and what happens in real-world LLMs fine-tuning. For example, more information is needed about what types of mis-reporting in real world can be represented by the mathematical definition of misreporting in this work. Does it consider the cases a) where agents may have knowledge about the utility function (and/or other agents’ preferences) so that they can design a reporting strategy to maximize their value or b) where agents without any such information simply over-represent their preference?", "limitations": "yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "$\\textbf{Strengths:}$\nThis work sought to address an important issue in LLM finetuning using human feedback data where agents may intentionally misreport their preferences to manipulate the finetuned model to their advantage. It uses a mathematical framework for investigating this issue and a mitigation strategy that adds a payment model, which seems quite elegant. The theoretical results show that truthful reporting is likely sub-optimal for individual groups when only a SW-Max rule is used. It then extends the VCG payment to implement SW-Max rules in dominant-strategy incentive compatibility (DSIC). Numerical results reported in this paper show the advantage of including a payment model.\n\n\n\n$\\textbf{Weakness:}$\n\n$\\underline{Section 4}$: The theoretical results in Section 4.1 show that in the absence of additional ‘intervention’/refinement, groups have incentives to misreport their preferences under most circumstances when the SW-Max Training rule is used for fine-tuning. Are there any other options to address this issue besides adding a payment rule? If yes, it’s unclear why the authors declared that the results in Section 4.1 demonstrate necessity of payment rule. I’d suggest a) presenting more accurate discussions about the implications of the results in Section 4.1 and b) discussing in Section1 other alternative approaches vs the approach of adding a payment rule used in this work, their respective pros and cons.\n\nIn Section 4.2, it seems that the affine maximizer payment rule $p^{AFF}$ depends on true $\\overrightarrow{rm}\\_{i}$ and $\\overrightarrow{w}\\_{i}$. It is unclear how to obtain $p^{AFF}$ in real world settings when true $\\overrightarrow{rm}\\_{i}$ and $\\overrightarrow{w}\\_{i}$ are unknown. Actually, if true $\\overrightarrow{rm}\\_{i}$ and $\\overrightarrow{w}\\_{i}$ are known, then it seems that there is no need to use the proposed approach. \n\n$\\underline{Section 5}$: In the numerical experiments, how was $p^{AFF}$ estimated? In addition, I have some difficulty understanding how the misreporting strategy (2) with $\\beta$ greater than 1 translates to a real-world misreporting strategy. Plus, would this strategy require the knowledge of true $\\overrightarrow{rm}\\_{i}$ and $\\overrightarrow{w}\\_{-i}$? It would be helpful if some concrete examples for misreporting can be given. For instance, one example of $\\tilde{w}\\_{i}=\\alpha w\\_{i}$ could be one agent reports its preference multiple times.\n\n$\\underline{Section 6}$ is incomplete. A discussion about limitations is included in the Appendix.\n\nOverall, while the mathematical framework is elegant, I have some difficulty to see the connections between the mathematical framework in this paper and what happens in real-world LLMs fine-tuning. For example, more information is needed about what types of mis-reporting in real world can be represented by the mathematical definition of misreporting in this work. Does it consider the cases a) where agents may have knowledge about the utility function (and/or other agents’ preferences) so that they can design a reporting strategy to maximize their value or b) where agents without any such information simply over-represent their preference? Also, is the proposed payment strategy actually implementable without prior knowledge of the true preference/awards of agents? This disconnect could diminish the potential impact of this work.", "was_revised": true, "final_justification": "The authors have addressed my major comments, so I raised my score accordingly.", "rebuttal": "We thank the reviewer for the insightful feedback and constructive suggestions. We address the specific points below.\n\n---\n**Question 1**: About the payment rule.\n> In Section 4.2, it seems that the affine maximizer payment rule depends on true $\\vec{rm}$ and $\\vec w$. It is unclear how to obtain in real-world settings when true $\\vec{rm}$ and $\\vec w$ are unknown.\n\n**Response**: We would like to respectfully clarify that in our framework, **the payment rule takes the reported information as input, not the true values.** The model's timeline is as follows:\n\n1.  The platform publicly announces the mechanism, which includes a training rule and a payment rule.\n2.  After observing the mechanism, the groups simultaneously report their information.\n3.  The LLM is fine-tuned, and payment is calculated using the **reported information** based on both the training rule and the payment rule.\n\nTo keep our notation concise, we opted not to introduce explicit notation (e.g., tildes such as $\\widetilde{\\overrightarrow{rm}}$) for reported values. We will revise the paper to make this point clearer and prevent potential confusion.\n\nFurthermore, as the reviewer highlights, the unobservability of true preferences is precisely what motivates the study of incentive compatibility in the RLHF context. We believe our characterization of DSIC from a mechanism design perspective provides valuable insights and establishes a foundation for future extensions.\n\n--- \n**Question 2**: What happens when agents (from all groups) misreport in terms of theoretical results and/or empirical results?\n\n**Response**: **The main consequence of misreporting is a reduction in the fine-tuned model's performance.**. As illustrated in the simplified example in Figure 1, when a group misreports to bias the model toward its own preferences, this results in a decline in regularized social welfare and reduced utility for other groups.\n\nTheoretically, because training is conducted on reported information while evaluation (or true welfare) is with respect to true preferences, only truthful reporting ensures optimality with respect to the social welfare objective. \n\nEmpirically, we provide detailed evidence in Tables 2–6 of Appendix F (page 33), showing that misreporting leads to significant drops in the regularized social welfare, which is the objective function for fine-tuning LLM.\n\n--- \n**Question 3**: The connection between the mathematical framework and real scenarios. \n> I have some difficulty seeing the connections between the mathematical framework in this paper and what happens in real-world LLMs fine-tuning. For example, more information is needed about what types of misreporting in the real world can be represented by the mathematical definition of misreporting in this work. Does it consider the cases (a) where agents may know the utility function (and/or other agents’ preferences) so that they can design a reporting strategy to maximize their value, or (b) where agents without any such information simply over-represent their preferences?\n\n**Response**: We are happy to elaborate on this. Our framework captures both (a) and (b) styles of misreporting, and we simulate both in our experiments. \n\n* Type (a) – *Strategic Misreporting Using Other Groups' Preferences (Line \\~369)*:\n In settings with known or predictable conflicts between groups, a group may benefit from **downplaying outcomes favored by others**. For example, if two groups have preferences $r_1 = (0.7, 0.3)$ and $r_2 = (0.3, 0.7)$, the first group might report $r = \\beta \\times r_1 + (1 - \\beta) \\times r_2$ for some $\\beta > 1$. For instance, with $\\beta = 1.2$, the reported preference becomes $r = (0.78, 0.22)$. This type of reporting aligns with more strategic manipulation and requires some information about the opposing group. Results from our paper (Figure 1) have shown that this strategy is profitable in certain cases. \n \n* Type (b) – *Naïve Overstatement (Line \\~203)*:\n  **A group may exaggerate its top preference to pull the model closer to its most favored outcome.** For instance, if true preferences are $r = (0.6, 0.3, 0.1)$, the group might report $r' = (0.6+\\epsilon, 0.3, 0.1-\\epsilon)$ for an $\\epsilon > 0$. This strategy requires **no knowledge of other groups' preferences**. Our experimental results (Table 1, Appendix F) show that this approach can be beneficial in the absence of payments.\n\nFinally, we emphasize that DSIC ensures truthful reporting yields higher utility than any misreporting strategy, including both types (a) and (b). Given the ease of implementing these misreporting strategies, particularly type (b), our study addresses a critical challenge in the RLHF context.\n\n--- \n**Weakness in Section 4** About the declaration of \"necessity of a payment rule\". \n> Are there any other options to address this issue besides adding a payment rule? If yes, it’s unclear why the authors declared that the results in Section 4.1 demonstrate the necessity of the payment rule. I'd suggest (a) presenting more accurate discussions about the implications of the results in Section 4.1 and (b) discussing in Section 1 other alternative approaches vs the approach of adding a payment rule used in this work, their respective pros and cons.\n\n**Response**: We appreciate the reviewer's constructive suggestion. In our paper, **the \"necessity of a payment rule\" is asserted within our specific modeling framework**, as detailed in our response to **Question 1**. Within this framework, Section 4.1 demonstrates that ensuring **Dominant Strategy Incentive Compatibility (DSIC)** is nearly impossible for commonly used training objectives without the inclusion of payment rules. Thus, a payment rule becomes necessary under these assumptions.\n\nWe agree that alternative paradigms are both possible and valuable. For instance, one could **design training objectives to incentivize truthful reporting instead of introducing explicit payments.** We'll expand the discussion in our related work section to include a broader perspective. For example, a recent paper by Buening et al. [1] highlights that no RLHF objective function can simultaneously guarantee strict truthfulness and social welfare optimality. They propose a pessimistic median algorithm that offers approximate truthfulness and converges to the optimal policy. While we also identify incentive issues in RLHF, our study approaches this from a mechanism design perspective, focusing on payment design to address the problem. The advantage of introducing payment is that it allows us to ensure the fine-tuned LLM remains optimal, as we don't alter the training objective. The drawback, however, is that monetary transfers may be prohibited in certain scenarios.\n\n[1] Buening, T. K., Gan, J., Mandal, D., & Kwiatkowska, M. (2025). Strategyproof reinforcement learning from human feedback. arXiv preprint arXiv:2503.09561.\n\n--- \n**Weakness in Section 5**: About the experiments. \n> In the numerical experiments, how $p^{AFF}$ was estimated?\n\n**Response**: In our empirical study, we simplify the optimization by predefining a finite model set $\\Theta'$, which is composed of LLM checkpoints. All `argmax` operations in the payment formula are then computed over $\\Theta'$, rather than the entire parameter space $\\Theta$. This approximation is computationally feasible due to the small size of $\\Theta'$. Importantly, since $\\Theta'$ is predefined and independent of group reports, this simplification preserves DSIC. Additional discussion on how $\\Theta'$'s expressiveness impacts DSIC and payment outcomes is provided in our response to **Question 2 from Reviewer AgWA**.\n\n--- \n**Weakness in Section 6**: About the conclusion and the future work.\n\n**Response**: We appreciate that the reviewer raises this issue. There is an error that occurs during file submission, which we will correct in the final version. Additionally, **we will include a detailed discussion on future work in the appendix. A preliminary version is outlined below:**\n\nBuilding on our proposed scenario and formulated model, we identify several promising directions for future research from both theoretical and empirical perspectives. First, exploring and modeling more general training rules could enhance our understanding of the framework. As noted in Appendix E, cycle monotonicity is a necessary and sufficient condition for implementability, but its validation is complex. Identifying a simpler condition to ensure implementability and investigating properties like payment equivalence for these rules are critical next steps. Second, studying preference aggregation across multiple models, particularly with diversity considerations, is a valuable direction. Third, as discussed in Section 4.4, developing mechanisms or criteria that balance computational efficiency and incentive compatibility in the RLHF Game could improve its real-world applicability. Finally, applying mechanism design theory to other large language model contexts, such as API pricing, retrieval-augmented generation (RAG), and prompt engineering, offers significant opportunities for further exploration."}, {"review_id": "bhxMQhd2hO", "reviewer": "Reviewer_a1Sa", "summary": "This paper proposes integrating payment schemes into SW-Max rules to address incentive issues where agents might strategically misreport their preferences to influence outcomes during LLM fine-tuning via RLHF across diverse groups. The authors provide theoretical analyses and proofs demonstrating the necessity, uniqueness, and robustness of the proposed payment rule. Additionally, empirical experiments conducted on two datasets support the claims and validate the effectiveness of the proposed method.", "questions": "No futher questions.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is generally well-written, clearly presenting most methodology details.\n\n2. The theoritical analysis and proofs seem solid and comprehensive, effectively convering the intuition, correctness and key characteristics of the proposed method.\n\n3. Empirical experiments on two datasets are provided to substantiate the necessity and effectiveness of the proposed method.\n\n\nWeaknesses:\n\n1. One of the central concepts, the payment rule/scheme, is clearly defined and illustrated only midway through the paper, leading to confusion when reading the introduction and related work sections.\n\n2. Some critical experimental details, such as the temperature, K, and P parameters of the LLMs, should be explicitly stated. Additionally, although the paper checklist states \"For others, we have provided clarification on the error bars,\" these clarifications seem not clearly identifiable. It is recommended to further clarify the different experiments conducted under deterministic and statistical settings, and explicitly report the variances from repeated trials in the statistical experiments.\n\n3. Despite the paper checklist stating, \"The full information combined with the code is provided,\" code access was not readily found. Providing publicly accessible code would greatly enhance reproducibility and encourage further research.", "was_revised": true, "final_justification": "The reviewer's rebuttal by incorporating these experimental details and illustrating key concepts earlier would significantly enhance the readability and quality of the paper. At this point, the rebuttal has satisfactorily addressed my concerns. And I'll keep my score for accept recommendation.", "rebuttal": "We thank the reviewer for the insightful feedback and constructive suggestions. We address the specific points below.\n\n--- \n**Weakness 1**: About the presentation. \n> One of the central concepts, the payment rule/scheme, is clearly defined and illustrated only midway through the paper, leading to confusion when reading the introduction and related work sections.\n\n**Response**: We thank the reviewer for this helpful suggestion. We agree that adding a concise description about the **payment rule and training rule** (currently defined below Line 139) in the earlier Introduction will enhance readability. We will revise the Introduction to explicitly and concisely outline the core elements of the model, including both the training and payment mechanisms. We believe this reorganization will significantly improve the logical flow and overall readability of the paper.\n\n--- \n**Weakness 2**: About the experimental details. \n> Some critical experimental details, such as the temperature, K, and parameters of the LLMs, should be explicitly stated. \n\n**Response**: We appreciate the reviewer's emphasis on transparency in the experimental setup. The hyperparameters we use are:\n\n``generation_kwargs = {\n \"max_new_tokens\": 128 if exp_type == 'assistant' else 48, \n \"min_length\": -1,\n \"do_sample\": False,\n \"num_beams\": 1\n}``\n\nFor other parameters, we follow the default settings of `AutoModelForCausalLM.generate()`. Specifically, **the temperature is set to $1.0$, and top-K is $50$. The base model used throughout our experiments is llama-2-7b-hf.** These details will be included in the Appendix in the final version for full reproducibility.\n\n--- \n> Additionally, although the paper checklist states, \"For others, we have provided clarification on the error bars,\" these clarifications do not seem clearly identifiable. It is recommended to further clarify the different experiments conducted under deterministic and statistical settings, and explicitly report the variances from repeated trials in the statistical experiments.\n\n**Response**: **The main experiments presented in the paper are deterministic simulations**, meaning that no randomness is introduced in the evaluation process. The results are based on model performances without training or sampling, and hence, error bars are not applicable in that context.\n\nHowever, **for the additional statistical experiments in Appendix F** (Table 1), we have now included standard error values. The updated table is shown below and will be included in the final version:\n\n| Reporting Parameter $\\epsilon$                  |      | 0.001   | 0.002   | 0.005   | 0.01    | 0.02     | 0.05     | 0.1       |\n|-------------------------------------------------|------|---------|---------|---------|---------|----------|----------|-----------|\n| $\\Delta$ Valuation (*1e2)                       | Mean | 0.1073  | 0.2096  | 0.4881  | 0.8667  | 1.3674   | 1.7978   | 1.8154    |\n| $\\Delta$ Valuation (*1e2)                       | Std  | <0.0001 | <0.0001 | 0.0003  | 0.0004  | 0.0013   | 0.0026   | 0.0032    |\n| $\\Delta$ Utility (= Valuation - Payment) (*1e4) | Mean | -0.1064 | -0.4135 | -2.3696 | -8.1557 | -23.7334 | -53.1552 | -55.8977  |\n| $\\Delta$ Utility (= Valuation - Payment) (*1e4) | Std  | <0.0001 | 0.0001  | 0.0011  | 0.0046  | 0.0196   | 0.0573   | 0.1415    |\n\n--- \n**Weakness 3**: About the code access. \n> Despite the paper checklist stating, \"The full information combined with the code is provided,\" code access was not readily found. Providing publicly accessible code would greatly enhance reproducibility and encourage further research.\n\n**Response**: We appreciate the reviewer's interest in reproducibility. Currently, **the code has been provided in the Supplementary Material**, along with a detailed README that explains how to run the simulations and reproduce the results. The implementation builds upon the framework introduced by Shi et al. [2], particularly the MOD mechanism.\n\nDue to the double-blind review policy, we are currently unable to release a public GitHub repository. However, we will make the code publicly available upon acceptance of the paper to ensure transparency and foster further research in this area.\n\n--- \n**The README file, which is enclosed in the Supplementary Material, is as follows:**\n\nThis code implements the empirical study section of the paper *\"Mechanism Design for LLM Fine-tuning with Multiple Reward Models.\"* The implementation builds upon *Multiple Objective RLHF* [1] and *Rewarded Soup* [2].\n\nAs outlined in the paper, we model two RLHF games: the \"Harmless vs. Humor\" game for the Helpful Assistants task, and the \"Faithful vs. Summary\" game for the Reddit Summary task. This code uses the \"Faithful vs. Summary\" game as a case study to demonstrate the training process.\n\n#### Training Procedure\n\n1. **Supervised Fine-Tuning**  \n First, we perform supervised fine-tuning on the base model using the corresponding dataset. To do so, run the following command:\n ```\n cd ./sft\n sh sft.sh\n ```\n Ensure that the base model (e.g., *llama-2-7b-hf*) is placed in the `./ppo` folder.\n\n2. **RLHF Process**  \n Next, we apply RLHF by training two models with their respective reward models. To start this process, run:\n ```\n cd ./ppo\n sh ppo.sh\n ```\n After completion, you will find the two RLHF fine-tuned models in the `./ppo` folder.\n\n3. **Rewarded Soup**  \n We then use the *Rewarded Soup* method to generate a set of hybrid models. Execute the following code:\n ```\n eval_rs_summary.sh\n ```\n\n4. **Simulation and Visualization**  \n The mechanism is simulated and visualised in the file `./ppo/stat.ipynb`. The figures and training logs are stored in the `./ppo` folder.\n\n#### References\n[1] Rame A, Couairon G, Dancette C, et al. Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. *Advances in Neural Information Processing Systems*, 2023, 36.\n\n[2] Shi R, Chen Y, Hu Y, et al. Decoding-time language model alignment with multiple objectives. *arXiv preprint* arXiv:2406.18853, 2024."}]

Author rebuttal:

Meta-review: This paper studies a very natural problem, that should be relevant to the NeurIPS community. It studies the following mechanism design problem: There are n groups with sizes w_i, and preferences rm_i mapping (prompt, reply)-pairs of an LLM to a value. The goal is to fine-tune a central LLM to these groups and preferences, with particular focus on SW-Max Training Rules which maximize a weighted sum of the preferences minus a regularization term (based on the divergence of the initial LLM and the fine-tuned LLM). The paper argues that quite generically payment-free models are susceptible to manipulation, it proposes an approach based on affine-maximization to determine payments that render truthful reporting a dominant (and IR) strategy. It then continues to examine conditions under which payment equivalence is obtained, and shows that under small noise the mechanism's incentive properties are preserved.

Overall, I think this is a worthy candidate for being accepted to NeurIPS (all reviewers have a score of 4), and I think this is a fair evaluation. The paper isn't rocket science from a mechanism design perspective, but I see clear value in formulating the problem in the right language accessible to the broader ML and NeurIPS audience. The paper also nicely pinpoints ways to manipulate system w/o payment (which should already be a useful contribution in and by itself). Wearing my AGT head, I also appreciate some some of the results that go beyond mere application of what's known in the mechanism design literature. I think this might become an influential Mechanism Design for LLMs paper.

----
Comment: There seems to be a rather closely related paper by Buening et al. "Strategyproof Reinforcement Learning
from Human Feedback" (https://arxiv.org/pdf/2503.09561). The authors websites don't mention whether this paper has already been published, it may have been submitted to NeurIPS as well. The current paper is not yet citing it, but the authors brought it up in the discussions and may wish to discuss it in the final version.
Decision: Accept (poster)