Towards Federated RLHF with Aggregated Client Preference for LLMs
Authors: Feijie Wu
Xiaoze Liu
Haoyu Wang
Xingchen Wang
Lu Su
Jing Gao
Venue: ICLR 2025 Poster (2025)
Keywords: Federated learning
RLHF
LLM
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Reinforcement learning with human feedback (RLHF) fine-tunes a pretrained large language model (LLM) using user preference data, enabling it to generate content aligned with human preferences. However, due to privacy concerns, users may be reluctant to share sensitive preference data. To address this, we propose utilizing Federated Learning (FL) techniques, allowing large-scale preference collection from diverse real-world users without requiring them to transmit data to a central server. Our federated RLHF methods (i.e., FedBis and FedBiscuit) encode each client’s preferences into binary selectors and aggregate them to capture common preferences. In particular, FedBiscuit overcomes key challenges, such as preference heterogeneity and reward hacking, through innovative solutions like grouping clients with similar preferences to reduce heterogeneity and using multiple binary selectors to enhance LLM output quality. To evaluate the performance of the proposed methods, we establish the first federated RLHF benchmark with a heterogeneous human preference dataset. Experimental results show that by integrating the LLM with aggregated client preferences, FedBis and FedBiscuit significantly enhance the professionalism and readability of the generated content.

Published as a conference paper at ICLR 2025
TOWARDS FEDERATED RLHF WITH AGGREGATED CLIENT PREFERENCE FOR LLMS
Feijie Wu', Xiaoze Liu', Haoyu Wang", Xingchen Wang', Lu Su', Jing Gao Purdue University "Sttt University of New York at Albany {wu1977, xiaoze, wang2930, lusu, jinggao) Janggao)@purdue. &purdue. edu bwang286albany.edd
ABSTRACT
Reinforcement learning with human feedback (RLHF) fine-tunes pretrained large language model (LLM) using user preference data, enabling it generate content aligned with human preferences. However, due to privacy concerns, users may be reluctant to share sensitive preference data. To address this, propose utilizing Federated Learning (FL) techniques, allowing large-scale preference collection from diverse real-world users without requiring them transmit data to a central server. Our federated RLHF methods (i.e., FedBis and FedBiscuit) encode each client's preferences into binary selectors and aggregate them capture common preferences. In particular, FedBiscuit overcomes key challenges, such as preference heterogeneity and reward hacking, through innovative solutions like grouping clients with similar preferences to reduce heterogeneity and using multiple binary selectors to enhance LLM output quality. To evaluate the performance of the proposed methods, we establish the first federated RLHF benchmark with a heterogeneous human preference dataset. Experimental results show that by integrating the LLM with aggregated client preferences, FedBis and FedBiscuit sigmificantly enhance the professionalism and readability of the generated content. https://github.coom/ Hww./////e/aaiaee//e/iieeiiiiiii
INTRODUCTION
Large language models (LLMs) exhibit broad knowledge coverage they are pretrained on
large corpus (Zhao et al. 2023a Singhal et al. 2023 Wang et al. 2023a Liu et al. 2024a). To elevate the quality of their generated content to match the professionalism and readability of hu- man writing, a common approach is to fine-tune these models using reinforcement learning with human feedback (RLHF) (Ziegler et al.) 2019; Christiano et al.] 2017 Ouyang et (022). This process relies on preference datasets, which are constructed through two main methods: human evaluation (Bai al. [002)] Ganguli et al.] [0022] [Stiennon et al., (020) and ChatGPT- rank- ing (Dubois et al. (024b). In the numan--ffor approach, set of instructions (a.k.a. prompts or user queries) paired with multiple model completions ka. generated responses), and team of labelers ranks completions of each instruction from best to worst. In the ChatGPT. based method, the dataset is developed by feeding ChatGPT with multiple inputs encompassing an instruction and a pair of completions, where it selects the superior completion for each input. As LLMs are deployed to serve diverse users, there could be a gap between the preferences of real-world users and those of Gabl.,, hindering the LLM's ability to generate responses that align with the users' preferences. Therefore, there is a need for a preference dataset that accurately reflects real world users' preferences in order to enhance the ability of LLM in content generation.
Preference Optimization m
Server
Preference
BBC
Ophmization
Private
Client's
Aligned LLM
Aligned
Preference
Privacy Leakage!
Privacy Preserved
Apgregsted Selector
Printe Quechton..
Trains 661
One straightforward approach to meeting this need to directly collect preference data from a large number of real- world users and build a comprehensive preference dataset, which can be used fine-tune an LLM central server. This strategy has been implemented in recent projects like OASST (Köpf et al. 2024). However, this approach is
880
Client's Private
Preference Selector
User
Preference
Client
Standard
Federsted RLHF
Figure 1: Comparison between standard
and federated RLHF

---

Published as a conference paper at ICLR 2025
often impractical because it involves collecting LLM users' inputs and preferences both highly private and sensitive (shown in the left part of Figure D. Most users are unwilling to share their data, and such practices may also violate regulations like the GDPR (016) and (C23) which prohibit the use of users' private data for model training without explicit consent.
To address privacy concerns, we propose utiliziig Federated Learning (FL) techniques to enable large-scale preference collection from diverse real-world users without requiring them transmit their preference data to a central server. In our design, we adopt FedAvg, a well-established FL algorithm (Konečn'y y al., 2016, McMahan et al. 2017). to learn user preferences without directly collecting their data. In line with the reward model training used in RLHF (Stiennon et al. [200)] Ouyang et al. 2022 Rafailov al. 2023), each LLM user works as a FL client and locally trains a reward model that can evaluate the quality of model completions. The server then aggregates these client-trained reward models into a global model. While this approach appears effective, we observe the following two limitations during training:
Typically, the reward model is regression model designed to output a scalar value that represents the quality of a model completion (Stiennon et al. 2020). Its optimization is based comparing reward differences between preferred and dispreferred completions, where preferred completions should receive higher rewards. However, during opti- mization, each data sample requires the retention of two computation graphs (one for the preferred and one for the dispreferred completion) throughout the forward and backward passes. This results in significant computational overhead and heavy GPU demands.
Excessive Computation Overhead:
Preference Heterogeneity and Reward Hacking: Client preferences are heterogeneous, as both instructions and preferences vary across clients. As a result, each client tends train their reward model toward local minimum, which deviates from the global optimum, leading to longer convergence times compared to centralized training. Moreover, fine-tuning the pretrained model using the reward model can lead to overfitting, where the proxy reward (the reward model's 's output) improves while the actual performance worsens. This phenomenon, known as reward hacking or reward overfitting, has been discovered in several studies (Askell et al., 2021 Michaud et al. 2020 Tien et al. 2022. Skalse et al. 2022).
In this paper, we propose to address these two limitations and propose effective and computationally efficient methods for preference collection and subsequent fine-tuning. We start with a solution that addresses computation costs. The key idea is to train a binary selector that identifies the superior response between two model completions. Compared with traditional reward model training, the binary selector requires significantly less computation. Casting binary selector training into a federated learning setting, we develop a federated binary selector training (FedBis) framework, as depicted in the right part of [igure The aggregated binary selector captures the common preferences of a large group of users, and thus can simulate a comprehensive preference dataset, facilitating RLHF approaches (c.g., DPO (Ralailov et al., )023)) to fine-tune the LLM without the need for the real client preference data.
To further address the performance deterioration due to preference heterogeneity and reward hacking, we propose method named FedBis with cluster-wise aggregation (FedBiscuit). This approach ensembles multiple binary selectors, each trained by clients with similar preferences. Since privacy concerns prevent the explicit sharing of client data, the server intermittently collects the training loss of all binary selectors on the clients. Using this data, clients are grouped into disjoint clusters, and when comparing two completions, the one favored by the majority of selectors is deemed superior. This method has two main advantages: (1) Clients with similar preferences collaboratively train a binary selector, reducing data heterogeneity and improving performance stability, and (2) Reward hacking is mitigated by employing multiple binary selectors, as it becomes difficult for the LLM to generate content that satisfies all binary selectors without making genuine improvements.
Contributions. In this paper, our contributions are highlighted as follows:
To the best of our knowledge, this is the first work to employ federated learning technique to enable large-scale user preference collection for RLHF without jeopardizing user privacy. Our proposed federated RLHF model (i.e., FedBis) encodes each client's preference information into a binary selector and aggregates all clients' binary selectors to capture their common preferences. By aligning the LLM with the aggregated client preference, we can improve the professionalism and readability of LLM''s generated content.
2

---

Published as a conference paper at ICLR 2025
We identify the inherent challenges of federated RLHF, such preference heterogeneity and reward hacking, and extend FedBis into FedBiscuit with innovative solutions to address these challenges, including grouping the binary selectors of clients with similar preferences to reduce data heterogeneity and employing multiple binary selectors to force the L.LM to improve the quality of its generated content.
We conduct extensive experiments to evaluate the performance of the proposed FedBis and FedBiscuit. Since no prior work has addressed RLHF in a federated learning setting, we es- tablish the first FL benchmark by creating a heterogeneous human preference dataset. As expected, both FedBis and FedBiscuit show significant performance improvements over the base models, Gemma and LLaMA.
2 RELATED WORK
Federated Tune-T for LLM. Recent studies have increasingly focused on fine- tuning large
language models (LLMs) using federated datasets (Sun et al. 2024 Zhang et al. 2024 2023
Yi et al. Bai et al. 2024). However, these approaches often suffer from high computation and communication costs due to the necessity of training and synchronizing the model
2023, Qin et al. 2023 2024
with clients. To mitigate these issues, lightweight methods such as black-box fine-tuning (Sun et al.,
2023) and offsite-tuning (Wu et al.] 2024a Kuang al. 2023) have emerged. Despite 2023 Lin et al.
their advancements, these methods primarily focus on fine-tuning LLMs for specific downstream tasks, neglecting user preferences the generated responses. A recent benchmark. OpenFedLLM (Ye et al. 2024). introduces FedDPO, which allows federated clients to optimize their local LLMs using DPO loss. While this approach can potentially align LLMs with human preferences, faces three key challenges: excessive computational overhead, preference heterogeneity, and the risk of reward hacking. To address these limitations, our work aims to enable LLMs alignment with a feasible and sustainable training framework in FL.
Reinforcement Learning with Human Feedback (RLHF). RLHF typically involves supervised fine-tuning, reward modeling. and reward optimization. initially proposed by Christiano et al. (2017). Proximal Policy Optimization (PPO) (Schulman et al. 2017) is a common RLHF algorithm, yet struggles with instability, inefficiency, and high resource demands (Choshen et al. 2019: Engstrom et al. 2020). These challenges have led to the development of alternative methods, such as Direct Preference Optimization (DPO) (Rafailov et al.. (023) and others (Dong al. 2023 Zhao et al. 2023b Azar el al.) 2024 Ethayarajh et al. [022]] Gulcehre et al. (023). which offer more stable and efficient solutions. However, these methods typically operate within centralized training framework, where the LLM owner retains control over the preference data. In contrast, our work aims to expand
data sources and integrate real user preferences without directly collecting their personal data.
PRELIMINARY
3.1 FEDERATED LEARNING (FL)
FL is a distributed training paradigm where a server coordinates various clients toward the same goal, i.e., training generalized model á E R (Konečn') 'y et al., 2016 McMahan al. 2017). Consider
an FL system with M clients. Denote the weight of client m as Pm such that 2 Pm 1, and we aim to optimize the following objectives:
min BER
F(O)
Σ
Pm Pm Fm (d).
mE[M]
where (d) is the expected loss on client m given the model σ. As classical FL algorithm, FedAvg can solve the optimization problem by multiple communications between the server and the clients,
followed by number of recent works (Wang et al. 2021 2022, 2023c 2024b cl Zhang et al. 2021). In each communication round r E [R] with the global model Ør, the following steps are conducted:
Model broadcast:
The server uniformly samples A clients without replacement, denoted by A. broadcasts the global model oe to the sampled clients.
Local training client E A: The client initializes the local model o with the received Φ,, In the next K iterations, the client updates its local model via Tthh $7 - V. (,")), E [K], where v is learning rate, and V Fm (e) is the local gradient on 6 and estimated by a mini-batch. We denote this K K-iteration local training by o OPTIM(M, Φr. K), which optimizes the model o with the data of client m.
3

---

Published as a conference paper at ICLR 2025
Client
Product Team
Response
Compare
Trains
Unlabeled Prompts
Server
Client1 Prompts
Pretrained LM
Client Preferance dataset
Client1 Preference dataset
Binary Selector
Pretrained LM
Reader friendly
Response
Aggregated Binary Selector
In Response
Compare
Trains
Response 1 Resgonse
DPO
Client
Pretrained LM
ClientM Preference dataset
Client Preference dataset
Binary Selector
Solector Preference dataset
Prompts
Response
Client
Selector
Step 1: Clients' Preference Data Creation
Step 2: Federated Preference Selector Training
Step LLM Fine-tuning with RLHF
Figure 2: An outline of the proposed FedBis, an RLHF method in federated learning.
Global aggregation: The server collects the local model 6,, from clients m E A and updates the
global model via weighted average aggregation, i.e., Φr+1 A EmEA Pm tm
M.
3.2
REINFORCEMENT LEARNING WITH HUMAN FEEDBACK (RLHF)
The objective of RLHF is to align a pretrained language model with human preferences so that the model can generate text that as professional and readable as human writing. DPO (Rafailov et al.) 2023) is one of the most effective ways to achieve the goal. There is a human-annotared dataset D comprising multiple samples (z, Yo V/1 i). where Yo and 1/1 are two completions under given instruction and i {0, 1} indicates that y is the preferred completion out of the pair of yo and y1 Motivated by the Bradley-Terry model (Bradley& & Terry 1952) on the formulation of human preference distrition p" (y: /)-r(z), DPO aims optimize the model O starting from Og via
min E (I) [DDP (0|z, yo, y1, i) A log (3log
me (y:lar)
[u [1x)) log Teg (on (1)) (11 ] (2)
Tep (3u
4 FEDBIS: A VANILLA AND FEASIBLE FRAMEWORK FOR ACHIEVING FEDERATED RLHF
We aim fine-tune LLM using clients' preference data, enabling it to generate reader-friendly responses. Since preference data contain sensitive personal information, some clients may be hesitant to share this information due privacy concerns. Recently, companies have developed on-device pretrained language models (e.g.. Phi-3 (Abdin et al. (2024) and Qwen (Bai et al. 2023)), )) with the latest iPhone release integrating this technology (Gunter al. 2024). This on-device feature allows clients to ask private and sensitive questions directly on their smartphones, ensuring that even the server (i.e., L.M owner) cannot access the input prompts (Wu al. 2024a).
The proposed FedBis provides a simple yet effective solution through three-step process to enable model fine-tuning with clients' preference data while preserving clients' privacy, as depicted in Figure 2 In the first step, each client builds their own preference dalaset, which is originated from the daily queries (a.k.a. prompts or instructions) to a pretrained language model (LM), the model generated a pair of responses to each query, and the client chooses the preferred one. After the construction of the preference dataset, each client independently trains a binary selector, and the server aggregates these selectors into global one and broadcasts it to the clients. This communication process repeats for a total of R times, and the clients keep the preference dataset unchanged during the training. In contrast traditional RLHF methods that train a regression-based reward model, this step employs a binary selector to alleviate computational constraints on edge devices (Wu et al. 2023b 2024b Wang al. 20236 2024a| Li et al. 2024). Additionally, the selector enables comprehensive pairwise comparisons between completions, thereby facilitating more efficient training (Liu et al. 2024b; Dubois et al. 2024b). Afterward, we utilize the well-trained binary selector to enhance the performance of L.M. Specifically, we assume the server holds set of instructions, together with pairwise responses generated by an LLM. Then, we build a preference dataset with the help of the binary selector and boost the LLM by means of DPO (Rafailov et al. 2023). This section highlights the key steps of the proposed FedBis, while Appendix offers a detailed description.
Local Training in FL. Suppose client
[M] holds set of pairwise data with the size of Am, i.e., D,m = { Vi Vi.1) (eje)n) where x; the prompt, Vi.w the preferred completion out of the pair of yiu and ye.. We reorganize these data and build a preference dataset D,, to be (IT; os Vo,1, 0), (xi, Vilt wi 121 Bi,we (i.)) Din)) for training, in which each contains
4

---

Published as a conference paper at ICLR 2025
the prompt, pair of completions and preference selection. Apparently, this dataset eliminates the position effects, and we can train the selector as a classification task. Therefore, we utilize eross-entropy (CE) loss EC to optimize the selector and formulate the expected loss
Fm(() = (o (f x, yo. 2n)]
Generating synthetic preference data for DPO. Suppose the server holds a set of instructions D. With the LLM Oo. we can generate multiple completions for an instruction æ E DD resulting in a set of n completions (yo..... ............... 1n--1) ~ Teo (y)ar) For each instruction, of completions. We then use the binary selector OR to choose the optimal completion for each pair (リy yt) where 0 S
can form a total of (?2) pairs
the first logit output is greater than the second, i.e., TT (0 lr Vý. D1 ) > Tøn (2, Mj. 1t), or i = 1 otherwise. This process builds the preference dataset 'D Dgen-
n 1. The pair labeled with i
Limitations. While the proposed FedBis effectively achieves RLHF in FL with low computational costs, there are two key limitations that remain unsolved. The first limitation is preference heterogene- ity. The proposed work effectively aggregates client preferences via clients' preference models, but it does not address the preference gap across clients when there is a huge difference in client preferences and prompts. As result, the clients optimize their local models for their own data instead of a global objective. This leads to global aggregation diverging from the global optimum, which lengthens the training time to obtain the desired performance (Karimireddy et al. 2020 Wu et al. 2023a). Another limitation is reward hacking. The proposed FedBis relies on training a single selector and using it to fine-tune the L.M. This procedure introduces an adversarial dynamic where the model "cheats" the selector into favoring certain responses without genuinely improving. Eventually, as the LLM is trained across more iterations, its performance degrades significantly, making the approach inefficient and unsustainable. In the coming section, we propose new algorithm that is able to address both limitations while maintaining low computational costs.
5 FEDBISCUIT: FEDBIS WITH CLUSTER-WISSH AGGREGATION
In this section, we aim to address the issues of preference heterogeneity and reward hacking in FedBis. To mitigate reward hacking, Eisenstein et al. (2023) and Coste et al. (2024) propose promising method that trains multiple reward models simultancously. Aggregating outputs from several models can lead more robust reward estimate. Additionally, recognizing that some clients may share similar preferences, we utilize clustered FL. (Sattler et al.] 2020 Ghosh et al. 2020 Ma et al. (023) to group clients with common preferences for joint selector training. These strategies complement each other, motivating us to combine them into a novel algorithm, FedBiscuit, which addresses both reward hacking and preference heterogeneity.
However, integrating these approaches is non-trivial, particularly when using existing clustered FL algorithms. Current algorithms predetermine the number of models and partition clients into groups, with each group training its own model. This one-to-one mapping assumes a fixed number of groups, but in practice, predefining the number of groups is challenging, and some models may end up untrained if no clients are assigned to them. If these untrained selectors are excluded, it could reduce the complexity of reward hacking, making it easier for the LLM "cheat" the selector by favoring certain responses without genuine improvement. On the other hand, including untrained selectors could misalign the LLM, leading to incorrect alignments. Therefore, there a need for a sustainable algorithm that ensures every selector contributes meaningfully to resisting reward hacking.
Problem Formulation. In this work, we consider training multiple binary selectors dIU) which independently decide better completion out of a pair. It noted that U should be an odd number because this guarantees one completion is preferred by more selectors. Moreover, to ensure that all selectors are trained without bias towards a small specific group, we mandate that these selectors be trained using evenly disjoint clusters of clients. Additionally, client's 's preference should align more closely with those within the same cluster than with those different clusters. To this end, we can formulate the following objective:
min RRUE
F(OU))
Σ Pm min Fm Fm((du)) (uE(t)
mE[M]
s.t.
max(|AMl Juejol min muin(|Mall Mul Juejoy S 1,
(5)
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 FedBiscuit
local learning rate nR, global learning rate 7e. local updates K, warm-up rounds ''' for each binary selector, total communication rounds R. client regrouping interval r, pretrained LLM Φ. Require: OPTIM(M, Φ. K) fine-tunes model & with the data of client m E [M] for K iterations and returns an optimized model.
Input:
Require: (G(()) assigns each client m E [M] to train one of the models oto) and returns a list {Um mmE[a]] indicating that client m should train the model dU... b Warm-up
D Clustered FL Training
for each u [U] do
14: Initialize 0u.0 Φy for each E [V]
2: Initialize the binary selector 0u.0 =
15:
for 00, 1, R - 1 do
3: for f 0, 1, ..... T do
16:
if r = then
4: Sample clients A C [M]
17:
{mm)me((N) CG()V))))
5: Send ou,t to clients m A
18:
end if
6:
for m E A in parallel do PutkK OPTIM(M, Da,t K)
19:
Sample clients A S [M]]
7:
20:
Send @U to clients m E A
Send Pa,tk to the server end for 10: 0u,t+1 = A Emea Puik
8:
21:
for E A in parallel do
9:
22:
UUm.r.K OPTIM(M, UU K)
23: Send Am
U,.,r,K to the server 24: end for
end for
12: the UU.T
25:
Calculate each 0 +1 via Equation
13:
end for
26:
end for
where the function Fm follows the same definition of Equation (3) ON indicates the u-th binary
selector, and M,, means set of clients using the u-th selector. By definition, Uuelu] Ma = [M]. and
Mu constraint of Equation (5).
Ma = 0. Next we explore how the proposed FedBiscuit optimizes Equation under the
5.1 ALGORITHM DESIGN
Section 4 mentions that a client m [M] holds a preference dataset Dy- Before the model training. client splits her dataset into two disjoint sets, namely, training set D m,,train and validation set Dm,vat, where [tm,train] >> [Dm.vat):
The proposed FedBiseuit consists of two phases: 1) We train each selector for couple of rounds so that all U selectors have basic capacities in selecting the preferred completion, and 2) we divide the clients into disjoint clusters of size U and train each binary selector with specific cluster. full pseudocode implementation is provided in Algorithm [
Phase 1: Warm-up. In the beginning, we initialize each binary selector sa(u E [U]) with an identical pretrained LLM 6. Subsequently, starting from = 0, we train each selector Φn for T consecutive communication rounds following the steps of FedBis: In each round, the server samples subset of clients A and broadcasts the selector du to them. Each client m E AA then locally trains the selector for K iterations using the dataset D Pm,trainn At the end of the communication round, the server aggregates and updates the selector σh. After completing the training of s.., the server initiates the training of a new selector Φu+1 by repeating the above steps until all selectors are trained.
The selectors trained with different data distributions because the clients participating in each training round are randomly selected. Consequently, all the selectors Φ10j have distinct model parameters, leading to varied performance in terms of final logit output when given an instruction and pair of completions.
Phase 2: Clustered FL Training. After the first phase, we obtain U different selectors, denoted by Ó1U1,0- Unlike FedBis, this phase includes additional step called client grouping, which partitions the clients into multiple disjoint clusters based on their preferences. In each communication round r E [R], FedBiscuit optimizes all the selectors Φ'.1,r using the following four steps:
Step 2.1: Client Grouping (G( This step is executed every 7 communication rounds, i.c.. when can be divided by т, T/r. During this step, the server broadcasts all selectors 61v),- to all clients [M]. Then, client calculates the averaged loss for each selector Φu,r using local
validation set via [Dmpaal Σ (z d)) D) (C (i 2, yo. 81)]. The server thereby collects

---

Published as a conference paper at ICLR 2025
all these losses and adopts greedy clustering approach (Sattler et al. 2020 Ma et al. 2023 assign each client to the selector where they achieve the minimum loss. However, an obvious deficiency is an imbalance where some selectors are chosen by many clients and others by few. It is noted that the selectors trained with more clients achieve remarkable performance, while some may be overfitted to specific group of clients. Therefore, the greedy clustering approach negatively impacts the overall performance when building a global preference dataset. To tackle the limitation, we propose to balance the clusters using the following steps repeatedly until the clients are evenly distributed: (i) Choose the cluster selected by the most clients, and (ii) If the cluster can accommodate clients, cap the cluster at n clients and reassign the rest to other clusters where they achieve suboptimal loss. Finally, we obtain balanced and disjoint clusters. Let client m train the Um- selector Φu... for the next rounds. After this client grouping step, FedBiscuit proceeds to the following three steps.
Step 2.2: Model Broadcast. Similar to FedBis, the server samples A clients from all clients [M] denoted by A. For each selected client m E A. the server transmits the selector oU This process can be characterized by defining Au as the group of clients chosen to train the selector Φ.. This ensures that Uu = A and Au =
Step 2.3: Local Training. The client E A receives binary selector ØUn.r from the server and trains the selector for K iterations via @7 -Um.r,k-11 UU.... v/ Fm & U...r.))
[K]. Finally,
let the updated local selector be 6Um.r.K. and the client pushes it to the server.
Step 2.4:
Global Aggregation. The server collects updated selectors from all participants A. Since there are several binary selectors, the server updates each selector with a designated group of clients. For instance, the aggregation rule for the selector E [U/ follows
0urr+1 (1-
1
Σ mEA.
Pm
Qu.
Σ mEA.
Pm Pm.ou...K
(6)
is noted that performance degradation occurs when a model is trained by clients with time-varying sizes in FedAvg (Gu et al., 202T]] Wang & Ji 2023). In other words, the weighted average aggregation strategy is no longer suitable for multi-selector aggregation due to the fluctuation in the number of clients training a specific selector in each communication round. Therefore, FedBiscuit adopts new aggregation rule as formulated in Equation (6).
FedBiscuit finally produces set of well-trained selectors dLll. x and the subsequent objective is to enhance LLM performance with the help of these selectors, as explored below.
Reinforeement-eermiing Fine-tuning with Multiple Selectors. We can leverage the methodology described in Section A.2 and one of the key steps involves constructing preference dataset incor- porating multiple selectors. For this, we employ a strategy of majority voting. Given an instruction & E and pair of generated completions (yo, y1), we assume a selector u E [U] prefers y/.,, where in E (0, 1). Therefore, the pair is assigned label i arg max n [e[[U) meaning that the completion yı is favored by most of the clients.
5.2 DISCUSSION: INTEGRATION WITH LORA
As all binary selectors are LLM, training them may consume significant communication and compu- tation overheads. Besides, multiple LLMs lead to considerable storage burdens shouldered by the server. To reduce the costs, we adopt a parameter-efficient fine-tuning approach LoRA (Hu et al. 2021). where all binary selectors share the same basc model while using different adapters.
In comparison with FedBis, FedBiscuit requires extra costs, i.e., O(MU [R/T] C"), where C is the communication cost of a selector. This is because FedBiscuit involves client grouping periodically, unilaterally transferring all selectors from the server to the clients. Despite the extra costs, extensive experiments demonstrate non- noivtrivial improvement by comparing FedBiscuit with FedBis.
FEDERATED HUMAN PREFERENCE BENCHMARK
In this section, we describe the preparation of federated human preference datasets, while the next section presents the experimental setup and quantitative analysis. We explore two open-ended text generation tasks, i.c., summarization and dnvestionansweree based on publicly available datasets. Each task comprises two components: client preference data and unlabeled prompts. Below,
7

---

Published as a conference paper at ICLR 2025
outline the process of constructing a federated preferecce dataset. Detailed information the datasets is provided in Table 3 in Appendix B
Summarization. Stiennon et al. (2220) introduces a summarization dataset that consists of Reddit posts with human-written TL,DR Völske et al. 2017). This dataset consists of two parts, one is a pretrained dataset, while the other is dataset with human preference. As suggested by Ouyang et al. (2022), we ensure a post does not appear in both datasets. We assume the pretrained dataset is stored on the server side, and 60% of data are reserved for supervised fine-tuning (SFT). The remaining 40% are used for the RLHF process to improve LLM performance and generate human-pre human-prefeered preferred
content. Since the human- human-preferemn preference dataset contains the worker ID, we partition the dataset based on the worker ID so that the dataset can be partitioned into 53 workers.
Question-Answering (QA). We reconstruct the public dataset SHP, which comprises numerous questions from Reddit posts and their corresponding user answers. The preference indicator is based on the number of likes an answer receives. Following the training of StreamSHP (Ethayarajh et al. 2022). we utilize the data with no more than 512 tokens. Given that the dataset spans 18 domains, we partition the dataset using a Dirichlet distribution with a parameter of 0.3, ensuring that no questions overlap between clients. In our experiment, which is a common setting when evaluating the performance of an FL algorithm (Jhunjhunwala et al., 2023). Figure 3 visualizes the data distribution the selected clients. For the RLHF process, incorporate 2.6K Reddit questions and 44.6K SafeRI. prompts (Dai et al. 2023).
consider training the binary selector with 200 clients,
7 EXPERIMENTS
7.1 EXPERIMENTAL SETUP
Model and computation environment. We initialize the binary selector(s) using three pretrained base models, i.e. Owen-2-0.5B (Bai et al., 2023), Gemma-2B (Team et al., 2024). and LLaMA-2- 7B (Touvron al. 2023). configuring the final layer to produce binary outputs "A" and "B" only. We adopt Gemma- 2B and LLaMA-2-7B models for the summarization and QA tasks, where both models are fine-tuned
the Alpaca dataset (Taori al. (023). Our implementation is built upon FederatedScope (Xie et al. 2023 Kuang et al. (2023). The experiments are conducted on machines with one Nvidia A100 GPU card, Intel Xeon Platinum 8369B CPUs, and 256GB RAM.
Baselines. Since no prior work systematically enables RLHF in FL,, we propose the following baselines, which extend previous studies to fit the FL and RLHF objectives. In each case, we directly optimize the pretrained model.
FedAvg: Given that preference data on clients is pairwise, each client trains its local model to improve completions based specific instructions. To minimize training costs, we employ LoRA for training and aggregation, following the approach (Sun et al. (2024).
FedDPO: Clients train their local models using the DPO loss (as defined in Equation 2) and the
server aggregates these local models into a global model using a weighted average. This method, incorporated in the LpM benchmark (Ye et al. (024), requires substantial computational resources and results in long local training wall-clock times.
Evaluation. We evaluate summarization and QA tasks using different datasets and methodologies:
Summarization task: We use a test dataset consisting of 6,553 samples, all sourced from the TL;DR dataset and excluded from the training data. The model is tasked with generating summaries for each sample. Since human-labeled summaries are available, we measure the win rate of the model-generated summaries against the human ones using the Auto-J model (Li et al. 2023a). QA task: For question answering, we use AlpacaEval 2.0 (Li et al., 2023b Dubois et al. 2024a b) assess model performance on 805 instructions. The model's 's responses are compared with those of GPT-4 and the win rate is calculated based on evaluations conducted by
Implementation. In our experiments, we train the binary selector for 500 communication rounds. In each round, we sample 5 clients for the summarization task and 10 for the QA task, and the selected clients fine-tune the binary selector locally for 30 iterations. As for FedBiscuit, the warm-up phase takes 50 communication rounds for each adapter, which is counted part of 500 communication rounds. After the training of binary selectors, we fine-tune the LLM for three epochs, and we store the checkpoint when finishing epoch of training. By default, the evaluation result reports the best-saved checkpoints. Due to the limited space, the details related to the hyperparameters are deferred to Appendix B
8

---

Published as a conference paper at ICLR 2025
Gemma-28 2B Win Rate
LLaMA-2-7B Win Rate (# Wins # Ties)
Binary Selector
Methods
(# Wins / # Ties)
Raw Model
67.27% (4408 28)
76.79% (5032/ 31)
NA
FedAvg
28.66% (1878 10)
28.23% (1850/ 10)
FedDPO
49.03% (3213 / 23)
77.02% (5047 30)
FedBis
86.69 (5681 26)
89.63% (5874 38)
Qwen-2 (0.5B)
FedBiscuit (U = 3)
78.45% (5141 / 29)
83.29% (5458) 48)
FedBiscuit (U 5)
73.97% (4847 22)
82.04% (5376 / 37)
FedBis
77.34% (5068 / 39)
86.22% (5650/ 39)
Gemma (2B)
FedBiscuit (U 3)
83.81% (5492 / 43)
91.32% (5984 28)
FedBiscuit (U = 5)
83.12% (5447 / /33)
89.93% (5893 31)
FedBis
82.56% (5410/ 28)
91.87 (6020/ 40)
LLaMA-2 (7B)
FedBiscuit (U 3)
81.90% (5367 44)
90. 80.84% (5951 37)
FedBiscuit (U 5)
79.31% (5197 /41)
90.54% (5933 / 47)
Table 1: Performance under summarization task. Bold means the best result under the pretrained model; Underline means the best result under binary selector.
7.2
QUANTITATIVE EVALUATION SUMMARIZATION TASK
a comparative analysis of win rate indicates the likelihood that generated summary surpasses its human counterpart, evaluated using the Auto-J metric. It is evident that both our proposed baseline and algorithm significantly outperform the raw model and other baselines. For both base LLMs Gemma-2B and LLaMA-2- 7B - all our methods demonstrate performance improvement of at Icast 6% over the raw model, underscoring the effectiveness of our approach irrespective of the binary selector used. Notably, all baseline approaches exhibit substantial decline in performance compared to the raw model, with decrease of at least 20% the win rate measurement.
and model-generated summaries, where the
We offer a plausible explanation for the performance drop of FedDPO, thereby highlighting the effectiveness of FedBis and FedBiscuit. Our proposed methods also incorporate DPO, differing significantly from FedDPO in terms data distribution. While FedDPO relies directly on client preference datasets, methods create set of preference data comprising three components: unlabeled prompts, pair of responses generated by the base model for each prompt, and preference selections simulated by the aggregated binary selector. Consequently, more closely with model outputs, facilitating easier guidance for the model to refine its responses into user-acceptable expressions. In contrast, FedDPO may confuse the model since it lacks outputs similar to those in the client preference datasets, leaving the LLM unsure of how to enhance the generated responses. An implicit assumption of DPO is that preference data should closely resemble model outputs; however, our proposed method may not be bound by this assumption.
generated responses align
Performance analysis on various binary selectors. Table presents the importance that a selector trained with a proper method can significantly enhance the performance of base models (i.e., Gemma- 2B and LLaMA-2-7B), A powerful selector does not mean that can significantly boost the base model performance after alignment. For example, the smallest binary selector, Qwen-2, achieves win rate of 86.69% under the Gemma-2B model, performing much better than the other two types. Different training methods different binary selectors may lead different effects. For instance, training Qwen-2 and LLaMA-2 with FedBis always better than that of FedBiscuit, while training the Gemma selector with FedBiscuit (U = 3) would achieve the best performance.
7.3 QUANTITATIVE EVALUATION ON QA TASK
Table shows that our methods, FedBis and FedBiscuit, consistently outperform the baseline ap- proaches in terms of length-controlled win rate for both the Gemma-based model and the LLaMA-2 model. The baseline FedAvg performs worse than the Raw Model, with FedAvg showing win rate of only 1.88% for the Gemma-based model and 3.93% for the LLaMA-2 model, compared to the Raw Model's 2.40% and 4.80%, respectively. FedDPO offers a slight improvement over the Raw Model, achieving win rate of 3.28% for the hemma-basel model and 4.98% for the LLaMA-2 model, but it remains inferior compared to our methods. Specifically, FedBis achieves win rates of up to 4.58% for the Gemma-based model and 5.2 5.25% for the I.LaMA-2 model, while FedBiscuit (U 3) reaches

---

Published as a conference paper at ICLR 2025
Unlabeled Prompts
Methods
Gemma-22 2B LC Win-rate (%)
LLaMA-2-7B LC Win-rate (%)
Raw Model
2.40 + 0.16
4,80 + 0.41
NA
FedAvg
1888 + 0.12
3.93 1 0.23
FedDPO
3.28 -: 0.23
4.98 1 0.30
FedBis
3.85 0.25
5.06 1 0.30
Reddit Posts
FedBiscuit (U 3)
3.70 +: 0.24
5.04 + 0.31
(2 completions)
FedBiscuit (U = 5)
3,98 - 0.24
5.42 + 0.32
FedBis
3.66 0.25
5.08 1 0.31
Reddit Posts (4 completions)
FedBiscuit (U =
4.14 + 0.27
5.34 + 0.32
FedBiscuit 5)
3.80 0.27
4.81 + 0.28
FedBis
4.58 0.30
5.25 + 0.32
LH
FedBiscuit (U = 3)
4.03 - 0.28
5.63 + 0.34
FedBiscuit 5)
3.85 + 0.27
5.42 +: 0.35
Table 2: Performance under QA task using AlpacaEval 2.0. "LC" in the table means "length-control: FedBis and FedBiscuit adopt the reward model of Gemma-2B fine-tuned on SHP dataset. Bold highlights the best result under each column, while Underline visualizes the best result under different sources of unlabeled prompts.
up to 4. 14% and 5.63%, respectively. These results confirm that our methods provide significant improvements over the baselines, enhancing the length-controlled win rates for both models.
Performance analysis various unlabeled prompts. Table 2 also shows that the of different unlabeled prompts, such as Reddit posts and SafeRLHF, significantly affects the performance of our methods. When using Reddit prompts, generating
completions leads variations in win rates. For example, with two completions, FedBiscuit (U = 5) achieves 3.98% for the Gemma-based model and 5.42% for the LLaMA-2 model. However, increasing to four completions slightly shifts the best performance for the Gemma model to FedBiscuit (U 3) with 4. 4.14%, while the win rate for the LLaMA-2 model slightly drops 5.34%. On the other hand, SafeRLHF prompts consistently yield the best results overall, with FedBis achieving 4.58% for the Gemma model and FedBiscuit (U = reaching 5.63% for the LLaMA-22 model. These findings demonstrate that SafeRLHF the most effective prompt source, outperforming Reddit-based prompts.
7.4 ABLATION STUDY
Considering both tasks, the results reveal that the performance of FedBis and FedBiscuit varies across datasets. In the summarization task (Table []. FedBis outperforms FedBiscuit across most binary selectors, achieving the highest win rates for both the Gemma-22 2B and LLaMA-2-7B models. For instance, under the Qwen-2 selector, FedBis reaches 86.73% for Gemma-2B and 89.67% for LLaMA-2-7B, while FedBiscuit (U and U = 5) are lower than those values. This is because preference heterogeneity is not critical in the summarization task. As described in Stiennon al. (2020), the dataset is collected from group of labelers who have a meeting from time to time to ensure they reach consensus. However, in the QA task (Table 22. FedBiscuit proves be the better method, particularly for the LLaMA-2-basee highest win rate 5.63% under SafeRLHF prompts, outperforming FedBis and other configurations. Although FedBis shows strength in some QA scenarios, such as the Gemma model with 4.58% win rate, FedBiscuit demonstrates superior overall performance in the QA task. Therefore, FedBis more effective in the summarization task, while FedBiscuit excels in the QA task.
model, where FedBiscuit (U = 3) achieves the
CONCLUSION
In this work, we explore a feasible framework to employ federated learning technique to enable large-scale user preference collection for RLHF without jeopardizing user privacy. Specifically, we train a binary selector across different clients using their local preference datasets, and then use the well-trained selector to align an LLM with human preferences. We propose two approaches to enable selector training: FedBis and FedBiscuit. FedBis provides framework to train a single selector, while FedBiscuit ensembles multiple selectors to address preference heterogeneity and reward hacking. We conduct empirical studies with the proposed federated human preference datasets to validate our statements and demonstrate the superiority of FedBis and FedBiscuit when aligning Gemma-2B and I.LaMA-2-1 with human preference.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENTS
This paper investigates clients' preferences using publicly available dataset, ensuring that all data sources are appropriately cited to maintain academic integrity and transparency. By leveraging this public dataset, we avoid using private or sensitive client data, thus upholding ethical standards in data usage and research practices. Furthermore, this work prioritizes the protection of clients' privacy and strictly avoids any disclosure of local data. When clients utilize their own data to fine-tune the model, robust privacy measures are in place to ensure that no other clients can access or infer any information related to their data. This approach not only safeguards individual privacy but also fosters trust and security in the application of the model.
ACKNOW
The authors would like to thank the anonymous reviewers for their constructive comments. This work supported in part by the US National Science Foundation under grants NSF CNS-2154059, NSF IIS-2141037, and NSF I1S-2226108. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint urXiv:2404.12219 2024.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. A general language assistant as a laboratory for alignment. arXiv preprint a 12.00861, 2021.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Proc. of the International Conference on Artificial Intelligence and Statisties (AISTATS') 24), pp. 4447-4455 2024.
Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. arXiv preprint arXiv:2402.11500 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.055: 2022.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning user-held data. arXiv preprint arXiv: (tfi::6611.0482.. 2016.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In Proc. of the ACM SIGSAC Conference Computer and Commu- nications Security (SP' 17), pp. 1175-1191, 2017.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3224-35, 1952.
CCPA. California consumer privacy act (ccpa), 2023. URI ccpa
Wei-Ning Chen, Christopher A Choquette Choo, Peter Kairouz, and Ananda Theertha Suresh. The fundamental price of secure aggregation in differentially private federated learning. In Prof. of the International Conference Machine Learning (ICMI. pp. 3056-3089, 2022.

---

Published as a conference paper at ICLR 2025
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. arXiv preprint
2019.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS 2017.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In Proc. of the International Conference on Learning Representations (ICLR 24), 2024.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint mrXi::2310.12773 2023.
Hanze Dong, Wei Xiong, Decpanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang. Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06777 2023.
Yann Dubois, Bal' azs Galambosi, Percy Liang and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.0447. 2024a.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In Proc. of the Advances in Neural Information 2024b. 2024b.
Processing Systems (NeurIPS
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244, 2023.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: case study on ppo and trpo. arXiv preprint 20l::2205..222.. 2020.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Proc. of the International Conference on Machine Learning (ICML 22), pp. 5988-6008, 2022.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Wenzhi Fang, Dong-Jun Han, Evan Chen, Shiqiang Wang, and Christopher G Brinton. Hierarchical federated learning with multi-timescale gradient correction. arXiv preprint 24Xiv:2409.188 18448, 2024.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint 2rXiv:2209.07858 2022.
GDPR. Regulation (EU) 2016/679 of the European Parliament and of the Council, 2016. URL https://data.com europa wera..uueee et//eli//ee//216//77////
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20) volume 33, pp. 19586-19597, 2020.
Xinran Gu, Kaixuan Huang, Jingzhao Zhang, and Longbo Huang. Fast federated learning in the presence of arbitrary device unavailability. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS 2/), volume 34, pp. 12052-12064, 2021.
12

---

Published as a conference paper at ICLR 2025
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern. Miaosen Wang, Chenjie Gu. et al. Reinforced self olffttranning (rest) for language modeling. arXiv preprint 2xX1:2338.00899 2023.
Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang. Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Shung-Cheny Chiu, David Qiu, et al. Apple intelligence foundation language models. arXiv preprint 24Xiv:2407.22 1075, 2024.
Shiqi He, Qifan Yan, Feijie Wu, Lanjun Wang. Mathias L' d'ecuyyr, and Ivan Beschastnikh. Gluefl: Reconciling client sampling and model masking for bandwidth efficient federated learning. In Proc. of Machine Learning and Systems (MLSys 23), 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.099 09685, 2021.
Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. arXiv preprint arXiv:2301.0900 2023.
Peter Kairouz, Ziyu Liu, and Thomas Steinke. The distributed discrete gaussian mechanism for federated learning with secure aggregation. In Proc. of the International Conference on Machine Learning (ICML'21) pp. 5201-5212, 2021.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Proc. of the International Conference Learning Representations (ICLR' 200), pp. 5132-5143, 2020.
Jakub Konečn'y, y, H Brendan McMahan, Felix X Yu, Peter Richt' arik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency, arXiv preprint arXiv:1610.05492 2016.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Rich ard Nagyfi, et al. Openassistant couversationn-demmcraiiinngg large language model alignment. Proc. of the Advances in Neural Information Processing Systems (NeurIPS 24)), 2024.
Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscooe--lm:: A comprehensive package for fine-tuning large language models in federated learning. arXiv preprint arXiv:2309.00336 2023.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470, 2023a.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint / 0007/09770.299 2019.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https:: // thub. www./ttoottttoocc httpaaa/aappaaapapaaaaaa lwaca_acaa_a__ _evall 2023b.
Yichen Li, Haozhao Wang, Wenchao Xu, Tianzhe Xiao, Hong Liu, Minzhu Tu, Yuying Wang, Xin Yang, Rui Zhang, Shui Yu, al. Unleashing the power of continual learning non-centralized devices: A survey, arXiv preprint arXiv:2412.13844 2024.
Zihao Lin, Yan Sun, Yifan Shi, Xueqian Wang, Lifu Huang, Li Shen, and Dacheng Tao. Efficient federated prompt tuning for black-box box large pre-trained models. arXiv preprint arXiv:2310.03123, 2023.
Xiaoze Liu, Ting Sun, Tianyang Xu, Feijie Wu, Cunxiang Wang, Xiaoqian Wang, and Jing Gao. Shield: Evaluation and defense strategies for copyright compliance in Ilm text generation. arXiv preprint arXiv:2406. (22.::9.....975.. 2024a.
13

---

Published as a conference paper at ICLR 2025
Xiaoze Liu, Feijie Wu, Tianyang Xu, Zhuo Chen, Yichi Zhang, Xiaoqian Wang, and Jing Gao. Evaluating the factuality large language models using large-scale knowledge graphs. arXiv preprint 1rXiv:2444009%: 2024b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017.
Jie Ma, Tianyi Zhou, Guodong Long. Jing Jiang, and Chengqi Zhang. Structured federated learning through clustered additive modeling. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficceent learning of deep networks from decentralized data. In Proc. of the Artificial Intelligence and Statistics (AISTATS 17), pp. 1273-1282, 2017.
Eric Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions. arXiv preprint arXiv:2012.05862. 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS 22), volume 35, pp. 27730-27744, 2022.
Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, and Shuiguang Deng. Federated full- parammeeerttr tuning of billion-sized language models with communication cost under 18 kilobytes. arXiv preprint 2rXiv:23/2.063:. 2023.
Zhen Qin, Zhaomin Wu, Bingsheng Hc, and Shuiguang Deng. Federated data-efficient instruction tuning for large language models. arXiv preprint arXiv:2410.10926 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), ), Proc. the Advances in Neural Information Processing Systems (NeurIPS 233), pp. 53728-53741. Curran Associates, Inc., 2023.
Mayank Rathee, Conghao Shen, Sameer Wagh, and Raluca Ada Popa. Elsa: Secure aggregation for federated learning with malicious actors. In Proc. ofthe the IEEE Symposium Security and Privacy (SP'23), pp. 1961-1979, 2023.
Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 32(8):3710-3722 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1777v:1007.06347, 2017.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 122-7972 2023.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS pp. 9460-9471, 2022.
Jinhyun So, Chaoyang He, Chien-Sheng Yang. Songze Li, Qian Yu, Ramy E Ali, Basak Guler, and Salman Avestimehr. Lightsecagg: lightweight and versatile design for secure aggregation in federated learning. Proc. of Machine Learning and Systems (MLSSy'22), 4:694-720, 2022.
Jinhyun So. Ramy E Ali, Başak Güler, Jiantao Jiao, and A Salman Avestimehr. Securing secure aggregation: Mitigating multi-round privacy leakage in federated learning. In Proc. of the AAA! Conference Artificial Intelligence (AAAI'23), volume 37, pp. 9864 9873, 2023.
14

---

Published as a conference paper at ICLR 2025
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. Proc. of the Advances in Neural Information Processing Systems (NeurIPS'20) 20), 2020.
Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang, Daguang Xu, Yiran Chen, and Holger R Roth. Fedbpt: Efficient federated black-box prompt tuning for large language models. arXiv preprint arXiv:2310.01467, 2023.
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in learning. arXiv preprint arXiv:2403.12313 2024.
federated
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. Stanford alpaca: An enstruction-folloiigg llama model. https: /t
com/ tat salab/stann stanfordddddooo..o..ooooooooooo alpacal 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre,, Morgane Rivi 'er. Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0829) 2024.
Jeremy Tien, Jerry Zhi- Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. arXiv preprint arXiv:2204.06601, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092888 2023.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TI; dr: Mining reddit to learn automatic summarization. In Proc. of the Workshop on New Frontiers in Summarization, pp. 59-63, 2017.
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. Survey on factuality in large language models: Knowledge, retrieval and domain comaai-speciffcctty arXiv preprint arXiv:2310.07521, 2023a.
Haoyu Wang, Handong Zhao, Yaqing Wang. Tong Yu, Jiuxiang Gu, and Jing Gao. Fedke: Federated knowledge composition for multilingual natural language understanding. In Proc. of the ACM Web Conference (WWW'22), 22), pp. 1839-1850. 2022.
Haozhao Wang, Zhihao Qu Song Guo, Ningqi Wang. Ruixuan Li, and Weihua Zhuang. Losp: Overlap synchronization parallel with local compensation for fast distributed training. IEEE Journal on Selected Areas in Communications, 39(8):2541-257) 2021.
Haozhao Wang, Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. Dafkd: Domain-aware federated knowledge distillation. In Proc. of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR'23) 23), pp. 00412-20420 2023b.
Haozhao Wang, Haoran Xu, Yichen Li, Yuan Xu, Ruixuan Li, and Tianwei Zhang. Fededa: Federated learning with cross-rounds divergence-aware aggregation. In Proc. of the International Conference on Learning Representations (ICLR") 233,, 2023c.
Haozhao Wang, Yabo Jia, Meng Zhang, Qinghao Hu, Hao Ren, Peng Sun, Yonggang Wen, and Tianwei Zhang. Feddse: Distribution- aware sub-model extraction for federated learning over resource-constrained devices. In Proc. of the ACM Web Conference (WWW'244 24), pp. 2902-2913, 2024a.
Haozhao Wang, Peirong Zheng, Xingshuo Han, Wenchao Xu, Ruixuan Li, and Tianwei Zhang. Fednlr: Federated learning with neuron-wise learning rates. In Proc. of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD pp. 3069-3080, 2024b.
Shiqiang Wang and Mingyue Ji. A lightweight method for tackling unknown participation probabili- ties in federated averaging. arXiv preprint 2xXxx:2306603401 2023.
15

---

Published as a conference paper at ICLR 2025
Xingchen Wang, Haoyu Wang. Feijie Wu, Tianci Liu, Qiming Cao, and Lu Su. Towards efficient heterogeneous multi-modal federated learning with hierarchical knowledge disentanglement. In Proc. of the ACM Conference Embedded Networked Sensor Systems (SenSys '4), pp. 592-605, 2024c.
Feijie Wu, Song Guo, Zhihao Qu, Shiqi He, Ziming Liu, and Jing Gao. Anchor sampling for federated learning with partial client participation. In Proc. of the International Conference on Machine Learning (ICML'23) (33,, pp. 37379-37416, 2023a.
Feijie Wu, Song Guo, Haozhao Wang, Haobo Zhang, Zhihao Qu, Jie Zhang, and Ziming Liu. From deterioration to acceleration: A calibration approach to rehabilitating step asynchronism in federated optimization. IEEE Transactions Parallel and Distributed Systems, 34(5):1548-1555 2023b.
Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. Fedbiot: Llm local fine-tuning in federated learning without full model. In Proc. of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 24), pp. 3345-3355, 2024a.
Feijie Wu, Xingchen Wang, Yaqing Wang, Tianci Liu, Lu Su, and Jing Gao. Fiarse: Model- heterogeneous federated learning via importance-aware submodel extraction. In Proc. of the Advances in Neural Information Processing Systems (NeurIPS volume 37, 2024b.
Yuexiang Xie, Zhen Wang, Dawei Gao, Daoyuan Chen, Liuyi Yao, Weirui Kuang, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope: A flexible federated learning platform for heterogeneity. Proceedings of the VLDB Endowment, 16(5):1059-1072, 2023.
Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. In Proc. of the International Conference on Learning Representations (ICLR 20), 2020.
Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models decentralized private data via federated learning. In Proc. of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD'244 24), pp. 1137-6147, 2024.
Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. Fedlora: Model-heterogeneouu personalized federated learning with lora tuning. arXiv preprint arXiv:2310. 132833 2023.
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP pp. 6915-6919, 2024.
Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, and Feijie Wu. Parameterized knowledge transfer for personalized federated learning. In Proc. of the Advances in Neural Information Processing Systems (NeurlPS'21) (/)), pp. 10092-10104, 2021.
Zhuo Zhang, Yuanhang Yang, Yong Dai. Qifan Wang, Yue Yu. Lizhen Qu, and Zenglin Xu. Fed- petuning: When federated learning meets the grrameter-efficieee tuning methods of pre-trained language models. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACI.233) pp. 9963-9977. 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 1(2), 2023a.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023b.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. In Proc. of the Advances in Neural Information Processing Systems (Neu 2019.
16

---

Published as a conference paper at ICLR 2025
A A DETAILED IMPLEMENTATION OF FEDBIS
The objective of RLHF is to align pretrained language model with human preferences. RLHF comprises two phases: (i) preference modeling and (ii) reinforeement-lamming fine-tuning. The first phase aims to develop model that simulates human preferences to select the superior options from numerous pairwise completions. Subsequently, the second phase enhances the language model's performance by creating preference dataset, enabling the model to generate responses preferred by humans. The following describes the proposed FedBis that achieves RLHF in FL.
A.1 PREFERENCE MODELING
We consider a practical and efficient FL scenario where not all clients but only sampled subsets of clients participate in each communication round (Yang et al. 2020) Wu et al., [202a]] He al.) 2023; Fang et al., 2024). Before the commencement of FL training. we initialize the binary selector with pretrained LLM such as LLaMA-2 (Touvron al. 2023), and set the hyperparameters. An FL algorithm requires multiple communication rounds and consists of three phases in each round, i.e., model broadcast, local training, and global aggregation. Following this paradigm, we design FedBis and optimize the selector é, i.c., in the communication round [R]. discussed as follows.
Step 1: Model Broadcast. The server uniformly samples A clients without replacement, denoted by A. Let the selector be o, in the r-th communication round, and the server broadcasts it to the sampled clients.
Step 2: Local Training. this step, client E A optimizes the selector based on local preference data. First, the client initializes the local selector 6 with the global selector ø, received from the server. Subsequently, the client trains the selector for K iterations, where the update rule between consecutive iterations follows:
6 F
0 Pm E [K]
where the gradient VIn (o approximated using a data batch sampled from the local preference dataset and can incorporate optimizers such AdamW (Loshchilov & Hutter 2017). Finally, the client m transmits the updated local selector ",,, back to the server.
Step 3:
Global Aggregation. After receiving the local selectors from the sampled clients A, the server updates the global selector:
M Σ MEA
@rt1
Pm P....K
(8)
This aggregation method, based on Li et al. (2019) where the clients are uniformly sampled to train global model, ensures consisteney with Problem (1) in mathematical expectation. To prevent the server from inferring a client's preferences from their local model, a practical solution is secure aggregation, which ensures the server only accesses the aggregated results (Bonawitz al. 2016
2017 Kairouz et al. 2021: Chen et al. 2022 So et al. 2022 2023 Rathee et al. 2023).
After R communication rounds of training, FedBis outputs a binary selector ΦR that reflects the overall preferences of all clients. The selector then be used to enhance the performance of the LLM, as discussed in the next section.
A.2 REINFORCEMENT-LEARNIIN FINE-TUNING
The reinforeementtleariing fine-tuning takes place on the server and includes two phases: 1) preference dataset is created with a pretrained LLM 6g and well-trained selector Φn from FedBis. 2) I.LM is optimized according to the objective defined in Equation ( with the generated dataset.
Step 1: Preference Dataset Generation. Suppose the server holds a set of instructions D. With the LLM Og. we can generate multiple completions for an instruction x D, resulting set of completions (Vo, .......> ....h-- 3n-1)) na, (v) For each instruction, we can form total of (?) pairs of completions. We then use the binary selector dR to choose the optimal completion for each pair (リz, 31) where 0 و < than the second, i.c., Tan pr, Uj 81 > म. (1|2,, Fj yt), or = 1 otherwise. This process builds the preference dataset D gen-
1. The pair is labeled with i 0 if the first logit output is greater
17

---

Published as a conference paper at ICLR 2025
LLM Fine-tuning. With the constructed preference dataset D gen. evolve the LLM to align with clients' preferences. Specifically, in the t-th training round, where t E {0, 10...... we sample data batch (æ, Vo. y1, t) from D gen. and update the LLM using the following rule:
Step 2:
Oz+1 - 02- v7 LDPo (Os Jz, Yo, (),
where 7 is the learning rate. The gradient computation V VLDPO is given by Rafailov al. (0223
In nutshell, we distill the binary selector's preferences into the LLM, allowing it to function as binary selector itself implicitly.
B IMPLEMENTATION DETAILS AND HYPERPARAMETERS
In this section, we include various settings, such the prompt and the hyperparameters.
B. HYPERPARAMETER SETTINGS
In our work, we fine-tune all models using LoRA, which is consistently set to rank 8, = 16, and the dropout rate 0.05. For the generation, we apply these parameters:
If it is required to generate multiple completions, then we set the temperature to 0.7. We set the maximum new tokens for 80 under the summarization task and 300 for QA tasks.
If it is required to generate single completion, then we adopt greedy search by setting the temperature to 0.0.
In the following part, we show the hyperparameter setting for different tasks:
Selector Training
RLFT
Selector Training
RLFT
Participation Rate
5/53
Participation Rate
10/200
Local Iterations
Local Iterations
30
Batch Size
32 epuchs R.MSprop
Batch Size
32
Rounds
500
Rounds
500
epochs
Optimizer
Adam W
Optimizer
AdamW
RMSprop
Hyperparameters
(0.9, 0.95)
Hyperparameters
(0.9, 0.95)
Learning rate
le
1e 6
Learning rate
le 5
1c
Table 3: Hyperparameter Settings for the Sum- marization Task
Table Hyperparameter Settings for the QA Task
B.2 DATASET DETAILS
In Section
discuss how to partition the dataset for tasks, namely, summarization and QA. 5 comprehensively presents the dataset details of both tasks, while Figure 3 visualizes the data distribution of the selected clients.
6
Table
Task
Preference
Dataset
preference
samples
# clients
Partition Rules
Max.
Min.
Std.
Unlabelled
Dataset
#unlabelled prompts
TLDDR
Open-Al
Summarization
comparison
92858
53
Worker ID
12985
2284.33
TL,DR
42782
Dirichler(O.3) 3) on categories
SHP Test
4293
Question Answering
260814
200
832.39
SafeRLHF
44578
Table 5: Dataset details for federated human preference benchmark
Special Setting for FedBist For the above two tasks, we ensemble three binary selectors (i.e., LoRAs). the warmup round, we train the selector for 50 rounds under FL framework. FedBiscuit performs regrouping every 50 rounds in the summarization task, while regrouping every 100 rounds the QA task.
B.3 MORE EXPERIMENTS
Comparison between two different model sizes. Table 6 compares the performance between two Qwen-2 models as the selectors with different sizes (i.e., 0.5B and 1.5B) when using them to fine-tune Gemma-2B model for the summarization task. These results show that while larger binary selectors (e.g.. Qwen-2-1.5BB sometimes provide slight performance improvements, smaller selectors like Qwen-2-0.5B remain competitive, particularly when applying FedBis. These findings suggest that both model type and size influence the selector's 's effectiveness, and they need to be carefully balanced based task requirements and resource constraints.
18

---

Published as a conference paper at ICLR 2025
19
39
59
79
Client ID
99
119
139
159
179
199
askacademia
askanthropology
askbaking
askcarguys
askculinary
askdocs
askengineers
askhistorians
askphilosophy
askhr
askphysics
askscience
asksclencefiction
askvet asksocialscience
changemyview
explainlikeimmfie legaladvice
Figure 3: Data distribution across different question domains on the selected clients.
Win Rate Qwen-2 (0.58)
Win Rate Qwen-2 (I.5B) (# Wins Ties)
Methods
(# Wins Tics)
FedBis
86.69 % (56817 / 26)
82.45% (5401 34)
FedBiscuit (U 3)
78.45% (5141 / 29)
80.18% (5254/ 29) 76.22% (4995 / 43)
FedBiscuit (U = 73.97% (4847 / 22)
Table 6: Performance using Gemma-2B to summarize Reddit post (i.e., summarization task), Bold means the best result under the selector.
B.4 INSTRUCTION TUNING PROMPT
The proposed work follows previous study (Zhang al., 2024) to fine-tune the model following given prompt template. This is also known instruction tuning. The prompt template is different between tasks and between the selector and base model. As result, we provide the prompts in Figure
4 for detailed study.
C SAMPLE DIALOGS
We provide sample dialogs for methods, FedBiscuit and FedBis, compared against the baselines FedDPO, FedAvg, and the raw model, highlighting their performance in summarization and question- answering tasks. Our methods consistently deliver superior results, demonstrated by the provided samples.
FedBiseuit generates precise and well-organized summaries, making it particularly effective in distilling complex information into clear and actionable insights. FedBis, while maintaining a similar level of clarity, offers more nuanced and in-depth responses, making versatile for wider range of queries. In contrast, the baseline models FedDPO, FedAvg, and the raw model tend to produce less coherent responses. FedDPO sometimes lacks structure, FedAvg oversimplifies the content, and the raw model can miss critical details, leading to less accurate results.
These samples provide clear evidence that FedBiscuit and FedBis offer better summarization and question-an performance by consistently striking the right balance between conciseness, accuracy, and detail, proving be more reliable and contextually appropriate than the baselines.
19

---

Published as a conference paper at ICLR 2025
Prompts template of LLM generation for summarization task.
Below is an instruction that describes task, paired with an input that provides further context. Write response that appropriately completes the request.
#*# Instruction:
Summarize the following Reddit post in paragraph of 50 words less.
### Input:
SUBREDDIT: (/(subreddit)
TITLE: [title]
POST: (post)
### Response:
Prompt template when training a binary selector for summarization task.
Below a forum post followed by two summaries. Pick more precise concise one that summarizes the most important points in the given forum post, without including unimportant or irrelevant details. State your choice with single capital letter, ie., "A" SUMMARY A is better, "B" if SUMMARY B is better.
#### SUBREDDIT: r/ r/(sureddiit)
TITLE: (title}
#*# POST: {post)
SUMMARY A: [output A)
### SUMMARY B: [output B)
YOUR CHOICE:
Prompts template L.LM generation QA task with an additional input.
Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
(instruction)
### Input: (imput)
Response:
Prompts template of L.M generation for QA task without additional input.
Below instruction that describes task. Write response that appropriately completes the request.
### Instruction:
(instruction)
### Response:
Prompt template when training binary selector for QA task with SHP dataset.
Below is query followed by two responses. Pick helpful response that is precise, concise, and casual. State your choice with single capital letter, i.., "A" if RESPONSE A is better, "B" RESPONSE B is better.
#*# QUERY: (instruction) }
#*# RESPONSE A: (output_A)
### RESPONSE B: (output_B) ###
YOUR CHOICE:
Figure 4: Prompt Templates.
20

---

hidden Published as a conference paper at ICLR 2025
User Input
Give me hrief scenario of a persona that would search this information and find this content helpful: (NSW Residential Building Contract for Small Works over $20,000 Designed for residential building work which is not complex over $20,000. Suitable for smaller alteration and renovation projects. Small Works Contracts (Form 10A) Pricing structure Fixed price Value range Under $7,500 Use for Minor residential alterations additions work Includes Succinet, place a quotation]
conditions suitable very small, low-risk projects
FedBiscuit
Imagine you are homeowner who wants to make some changes to your home, but you don't 't want to spend a lot of money on a major renovation. You come across a small works contract that is designed for residential building work that is not complex and has value range of under $7,500. This contract is suitable for smaller alteration and renovation projects. It is fixed price contract, meaning the price is and there costs or additional fees.
hidden
The contract is written plain language, making it easy to understand and use. It includes succinct conditions that are suitable for very small, low-risk projects. This means that if something goes wrong, you can casily resolve the issue and the contract will protect you.
Overall, this small works contract for residential building work is great option for homeowners who want make some changes to their home without spending a of money. It simple, straightforward solution that provides peace of mind and protection for your project.
FedBis
Imagine you are homeowner who wants to make some changes to your home, but you don't 't want to spend a lot of money on a major renovation. You come across a small works contract that is designed for residential building work that is not complex and has value range of under $7,500. This contract is suitable for smaller alteration and renovation projects. It is fixed price contract, meaning the price is set and there costs or additional fees.
The contract is written in plain language, making it easy to understand and use. It includes succinct conditions, which are suitable for very small, low-risk projects. The contract is intended for residential work, mcaning it is not suitable for larger, more complex projects.
Overall, this small works contract for residential building work is cost-effective and straightforward option for making small changes to your home without breaking the bank.
FedDPO
persona that would find this information helpful is homeowner who is planning to make some minor changes to their home, such as adding a new room or making some renovations. They are looking for a simple and straightforward contract that outlines the terms and conditions of the work, with fixed price and a value range of under $7,500. They want ensure that the contract is suitable for their small, low-risk project and that it is concise and easy understand.
FedAvg
I'm not sure if this is helpful, but I'm a lawyer and I've heen looking for a similar contract for a while now. I've found few, but they't re not quite what I'm looking for.
Raw Model
A persona that would search this information and find it helpful is small residential building contractor who is looking to complete small works project on their home. They are looking for fixed price contract that suitable for their project, which is not complex and will not exceed $20,000. They want contract that is simple and casy to understand, with concise conditions that outline the scope of work and the expected timeline. They are looking for a contract that is suitable for small, low-risk projects, and they want to avoid the need for quotation.
Figure 5:
Gemma QA Sample.
21

---

Published as a conference paper at ICLR 2025
User Input
Hi, have some falafel, but no tahini to put on them. Can you give me a recipe for making tahini?
FedBiscuit
Sure, here is a recipe for making tahini at home:
Ingredients:
cup of sesame seeds
tablespoons water
tablespoons honey
2 tablespoons of lemon juice
Instructions:
Place the sesame seeds food processor and process until they hecome a paste.
2. Add the water, honey, and lemon juice and process until the mixture is smooth.
3. Taste and adjust the seasoning if necessary.
4. Transfer the tahini airtight container and store in the refriggrraor for up to month.
FedBis
Sure, here is a recipe for making tahini at home:
Ingredients:
cup of sesame seeds
tablespoons water
tablespoons of honey
2 tablespoons of lemon juice
teaspoon of garlic
1/2 teaspoon salt
Instructions:
1. Place the sesame seeds in food processor and process until they become paste.
Add the water, honey, lemon juice, garlic, and salt and process until the mixture smooth.
Taste and adjust seasoning necessary.
4. Transfer the tahini to an airtight container and store in the refrigerator for up to month.
FedDPO
You make tahini home by soaking few tablespoons of sesame seeds in small amount of water for 30 minutes, then grinding them food processor. Add a tablespoon the resulting paste to your falafel to give them a delicious, nutty flavor.
FedAvg
I've been making tahini with a blender.
I put cup of sesame seeds in blender, add cup of water, and blend until smooth.
I then strain the mixture through cheesecloth-linee strainer.
I've been using the tahini for few weeks now and it's been great.
Raw Model
You can make tahini at home by soaking a few tablespoons of sesame seeds in a small amount of water for 30 minutes, then grinding them in food processor.
Add tablespoon of the resulting paste your falafel give them nutty flavor.
Figure 6: LLaMA QA Sample.
22

---

Published as a conference paper at ICLR 2025
Reddit Post: What is your most memorable act of kindness and why?
Random otherwise.
was walking to school one morning when was about when heard a voice call out "Hello? Excuse me? Young man?".
I turned and saw a frail old man who must have been in his 90s standing in the doorway of his house.
I walked over and he asked if I could help him with something. I said I would help, so followed him step-ladder in his kitchen.
He said that he needed to change the bulb but he couldn't get up the ladder. After changed the bulb, he said that he' d give me something for helping out. He reached for small tray of change on his kitchen table and picked out the only 50p coin in his tray of Ip and 2p coins.
told him he didn't have to give me anything said that was only right to reward helping out an man.
FedBiscuit
A Reddit user recounts a heartwarming act of kindness where they helped an elderly man change lightbulb his kitchen.
The stranger gave them a 50p coin as a reward their help. The act of kindness was memorable because brought a smile to the elderly man's 's face and showed that even in their old age. people can still do good deeds for others.
FedBis
A Reddit user recounts a heartwarming act of kindness where they helped an elderly man change lightbulb in his kitchen. The stranger gave them 50p coin reward for their help. The act of kindness was memorable because brought smile the elderly man's face and showed that even in their old age, people can still good deeds others.
FedDPO
Reddit user recounts a heartwarming story of 90-year-old man who needed help changing lightbulb his kitchen.
The user, who was 155 at the time, heard the man's plea for help and went to his aid. The man gave the user
FedAvg
Old man in his 90s needed help changing bulb. helped him and he gave 50p coin.
Raw Model
A Reddit user recounts heartwarming story of stranger who helped elderly man change lightbulb in his kitchen.
The stranger, who was walking to school at 15, heard a voice call out for help and followed the old man to his house.
He changed the lightbulb and the old man gave him 50p coin as reward.
The stranger didn't it have
Figure Gemma Summarization Sample.
23

---

Published as a conference paper at ICLR 2025
Reddit Post: TIFU by locking myself out of my room.
It all started with a cultural tour. Being active duty military and stationed overscas in Japan we were voluntold partake in tour of the local area. Fun and dandy right?
I was ecstatic about the free tour of the local Luckily the meeting area was a Starbucks just outside my room.
Except when woke up five minutes before the tour starts.
Que the rushing, the frantic panic to dress up, and get down in time. I left my ID which is also convenienly my room key. Two more mistakes. (Straying away from the story. the military is all about being on time and integrity. Hence "excellence in all we do".)
Realizing fucked up bad, told the host that had left my REQUIRED ID my room, which is my room key locking myself out.
I couldn't 1 go on the tour and now I'm stuck in the Starbucks waiting until one of my supervisors picks me up and lets me into my room. I'm expecting more than just little scolding, paperwork even.
FedBiscuit
The poster was locked out of their room due to leaving their ID, which was also their room key, in their rush make it on time for a cultural tour.
They stuck in Starbucks waiting for someone pick them up and let them into their room. They expect to receive a scolding and possibly even paperwork for the mistake.
FedBis
While cultural tour Japan, the poster locked themselves out of their room, leading them being stuck a Starbucks and expecting a scolding from their supervisors.
FedDPO
While on cultural tour in Japan, the poster was locked out their room leaving their ID room key in the room. They are now stuck a Starbucks waiting for a supervisor to pick them and let them into their room. They expect to receive a scolding and possibly paperwork for the mistake.
FedAvg
locked myself out my room and now I'm stuck Starbucks waiting supervisor to pick me up and let me into my room.
Raw Model
While on cultural tour in Japan, the poster was locked out their room due to leaving their ID room key in the room. They were stuck in Starbucks until one of their supervisors picked them up and let them into their room. They expect receive scolding and possibly paperwork for the mistake.
Figure 8: LLaMA Summarization Sample.
24


Reviews:
[{"review_id": "VkiB6gdQOT", "reviewer": "Reviewer_t8Tn", "summary": "Reinforcement learning with human feedback (RLHF) is a good way to fine-tune a pretrained large language model (LLM) to generate content aligned with human preferences. This paper proposes to use federated learning technique to aggregate different model information from users to enhance RLHF ability of LLM. Concretely, the proposed federated RLHF methods encode each client’s preferences into binary selectors and aggregate them to capture common preferences. This method overcomes key challenges like preference heterogeneity and reward hacking to enhance LLM output quality. Experiments on two LLM benchmarks demonstrate the effectiveness of the proposed federated RLHF algorithm.", "questions": "It may not be easy to add more experimental results in the rebuttal period. But can the proposed method also enhance RLHF ability of LLM in other NLP tasks like few-shot learning, synthetic, code completion, multi-needle retrieval?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "(1)\tThis paper provides a novel federated learning-based reinforcement learning method to enhance the output ability of LLM. To the best of my knowledge, this is the first time to employ federated learning technique to enable diverse user collection for RLHF.\n\n(2)\tThe organization of this paper is clear and easy to follow. In particular, Figure 2 clearly depicts the outline of the proposed FedBis model.\n\n(3)\tThe algorithm design in Section 5.1 is practical and new to me. Besides, the key component of the algorithm (i.e. training a binary selector and aggregating different selectors to the LLM server) effectively incorporates federated learning idea into the design of RLHF model.\n\n(4)\tThe experimental results on summarization and question-answering dataset validate the advantages of the proposed algorithm over existing methods.", "weaknesses": "(1)\tThe experimental results in Section 7 are not extensive enough. This paper conducts experiments on two NLP tasks, summarization and question-answering, and more experiments on other tasks should be complemented, like few-shot learning, synthetic, code completion, multi-needle retrieval.\n\n(2)\tI admire the proposed FL-based reinforcement learning method, but it will be more readable and more concise to summarize the contents in Section 5.1 Algorithm Design into a pseudocode (i.e. shown in an Algorithm).\n\n(3)\tLines 70-77 in Introduction is about the limitation of preference heterogeneity and reward hacking, and such limitation appears (in detail) once again in lines 224-237 of methodology. It suffices to mention such limitation once in one paper, or at least, not to emphasize it twice (e.g. not use bold to highlight it).\n\n(4)\tSection 6 is about the details of the experimental datasets, and will be better to defer it to Section 7 Experiments.\n\n(5)\tIt will be better to use \\begin{small} or other writing skill, to make Eq.(2) and Eq.(4) shown in one line.\n\n(6)\tLines 237,294, 471, isolated word\n\n(7)\tLine 268, fix “. $\\phi _{u}$”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5v6SXZYPYV", "reviewer": "Reviewer_9Rrf", "summary": "The paper deals with privacy preservation through applying federated learning to employing user preferneces to fine-tune a common LLM. The synergy of reinforcement learning, large language model, and federated learning appears apealing to me. The paper presents nicely with an intuitive comparison between standard and fedrated RLHF, making the problem easy to follow.", "questions": "What are the unique challenges posed by employing federated learning to RLHF?\nHow RLHF impact the challenges and solution design presented in the paper?\nWhat is the motivation behind the binary selector? what are the alternatives and why binary selectior, in particular, is chosen to be part of the solution?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper deals with an important research problem.\n- The paper is generally well orgnaised and presente, and easy to follow.\n- The authors proposed a sound solution to the research problem, showing advantages over baselines.", "weaknesses": "- The challenged identifiied on Page 2 seems univeral and apply to many federated learning scenarios, not specified to this paper's research context. Therefore, although the overall problem setting seems to be new, the key research problem to tackle remains conventional.\n- To deal with excessive comptuation overhad and preference heterogeneity seem to be a common issue that has been addressed by various previous efforts. The clustering approach to addressing the key issue does not appear novel to me.\n- The cons of appplying the federated learning approach might be discussed for readers to better understand the trade-off.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yNSKbd2jAd", "reviewer": "Reviewer_EwYt", "summary": "This paper leverages federated learning into reinforcement learning with human feedback (RLHF) and proposes two federated-based RLHP methdos (FedBis and FedBiscui) to fine-tune large language models using user preference while preserving data privacy. The methods employ binary selectors to aggregate client preferences without requiring direct data sharing. FedBiscuit extends the functionality of FedBis by addressing preference heterogeneity and mitigating reward hacking. Experiments demonstrate that FedBis and FedBiscui perform better on Summarization task and QA task.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1、Generally, this paper is well-written with clear explanations of the methodologies and results. \n\n2、The proposed FedBis and FedBiscui demonstrate good performance on several benchmarks, suggesting good-quality research and implementation.\n\n3、This paper addresses the heterogeneity issue, an important issue in all fields of federated learning related research.", "weaknesses": "1、It would be better to include experiments that explore the sensitivity of the hyperparameters, particularly the number of clusters $U$, which is a crucial or even central parameter in the FedBiscuit method. However, the current results only present cases for $U=3$ and $U=5$, providing limited insights from these two configurations.\n\n2、The authors should further explain why FedBiscuit with $U=3$ performs better in some cases while $U=5$ yields superior results in others. Since FedBiscuit addresses the heterogeneity issue, and as noted in existing federated learning literature, particularly in cluster-based federated learning approaches, the number of clusters is a key factor. Understanding the impact of different cluster sizes on the method’s effectiveness would be valuable.\n\n3、The additional complexity introduced by clustering and the use of multiple selectors in FedBiscuit may still pose scalability challenges, particularly in real-world applications with thousands of clients. It would be helpful to provide more analysis on the scalability of the approach, especially under varying numbers of clients and communication costs. This would shed light on the method’s feasibility for large-scale deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bz434NMbTn", "reviewer": "Reviewer_yFna", "summary": "This paper proposes a federated RLHF method specially designed for large-scale preference collection, which encodes each client’s preferences into binary selectors and aggregates them to capture common preferences. An enhanced method is also introduced, which overcomes the challenges of preference heterogeneity and reward hacking by grouping similar clients and leveraging multiple binary selectors to improve LLM output quality. The authors also established a novel federated RLHF benchmark for evaluation. Experimental results show the superiority of the proposed model.", "questions": "-Is there any data available on the training time or memory footprint of the entire training process?\n\n-Given that LoRA configuration also has an impact on performance, can the authors provide the detailed information about the LoRA configuration used in their experiments?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-The workload is impressive, every part is introduced in detail.\n\n-The idea of training binary selectors to encode client’s preferences is novel.\n\n-The two proposed methods are reasonable and both have many details in addressing the issues faced by traditional method.", "weaknesses": "-Computation overhead. This method involves additional training of binary selectors, which are pre-trained LLMs. Although the authors apply LoRA to reduce the computation cost, the time and computation overhead of fine-tuning LLM selectors are still not negligible, which limits its application scenario.\n\n-The experiments need supplements: In the section of performance analysis on different binary selectors, the experiments applied three base model with both different model structure and parameter size. To make the results more convincing, the effect of model type and model size should be explored separately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)