Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Authors: Yao Shu
Wenyang Hu
See-Kiong Ng
Bryan Kian Hsiang Low
Fei Yu
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Federated Full-Parameter Tuning
Scalability
Theoretical Guarantees
Primary area: deep_learning->large_language_models
TL;DR: We propose Ferret, the first first-order FL method with shared randomness to enable scalable full-parameter tuning of LLMs across decentralized data sources while maintaining competitive model accuracy.

Abstract: Large Language Models (LLMs) have become indispensable in numerous real-world applications. However, fine-tuning these models at scale, especially in federated settings where data privacy and communication efficiency are critical, presents significant challenges. Existing approaches often resort to parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but this typically comes at the cost of model accuracy. To this end, we propose *federated full-parameter tuning at scale for LLMs* (Ferret), **the first first-order method with shared randomness** to enable scalable full-parameter tuning of LLMs across decentralized data sources while maintaining competitive model accuracy. Ferret accomplishes this through three aspects: **(i)** it employs widely used first-order methods for efficient local updates; **(ii)** it projects these updates into a low-dimensional space to considerably reduce communication overhead; and **(iii)** it reconstructs local updates from this low-dimensional space with shared randomness to facilitate effective full-parameter global aggregation, ensuring fast convergence and competitive final performance. Our rigorous theoretical analyses and insights along with extensive experiments, show that Ferret significantly enhances the scalability of existing federated full-parameter tuning approaches by achieving high computational efficiency, reduced communication overhead, and fast convergence, all while maintaining competitive model accuracy. Our implementation is available at [https://github.com/allen4747/Ferret](https://github.com/allen4747/Ferret).

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Yao Shu Wenyang Hu" 23 See-Kiong Ng"
Bryan Kian Hsiang Low
Fei Yu"
Abstract
Computational Cost
Communication Cost
Parums (log-scale)
Large Language Models (LLMs) have become in- dispensable in numerous real-world applications. However, fine-tuning these models at scale, espe- cially in federated settings where data privacy and communication efficiency are critical, presents significant challenges. Existing approaches often resort to parameter-efficient fine-tuning (PEFT) to mitigate communication overhead, but this typi- cally comes at the cost of model accuracy. To this end, we propose federated hall-parameter tuning at scale for LLMs (Ferret), the first first-order method with shared randomness to enable scal- able full-parameter tuning of L.LMs across de- centralized data sources while maintaining com- petitive model accuracy. Ferret accomplishes this through three aspects: (I) it employs widely used first-order methods for efficient local up- dates; (II) projects these updates into a low- dimensional space considerably reduce com- munication overhead; and (III) it reconstructs local updates from this low-dimensional space with shared randomness to facilitate effective full full-parameter global aggregation, ensuring fast convergence and competitive final performance. Our rigorous theoretical analyses and insights along with extensive experiments, show that Fer- ret significantly enhances the scalability of exist- ing federated full- nall-parmeeter tuning approaches by achieving high computational efficiency, reduced communication overhead, and fast convergence, all while maintaining competitive model accu- racy. Our implementation is available at https: //gi terithuub.... www.co/oaao//oo alle4411en44 4777 /eeret.
200
180.70
10
9.78
Seconds
100
5
3.91
3,89
30,30
2.10
Convergence Rounds
Test Score
40
40.00
32.11
40
29.77
30.03
Rouge-L (%)
# Rounds
20
20
12.00
12.00
FedAvg FedKSeed
7 Ferret (Ours)
Figure I: Performance comparison of various federated full- parameter tuning algorithms on Natural Instructions dataset with LLaMA-3B. Our Ferret shows significantly improved scalability, with 6.0x reduction in computational cost and 3.3x fewer convergence rounds than FedKSeed, alongside 10 x reduction in communication overhead than FedAvg, while achieving comparable test score.
1. Introduction
Recently, Large Language Models (LLMs) have become in- dispensable tools across a wide range of real-world applica- tions, from natural language processing tasks like translation (Xu al., 2024) and summarization (Van Veen al., 2024) to complex tasks such code generation (Liu et al., 2024) and decision-making systems (Shao et al., 2023). The immense scale and versatility of LLMs make them highly valuable in practice, but they also introduce significant chal- lenges, particularly when they are fine-tuned in federated settings. Federated Learning (FL. offers a decentralized approach fine-tuning LLMs while retaining data on local clients to ensure privacy. However, while this approach effectively addresses privacy concerns, it also results in prohibitive communication overhead when the model pa- rameters of LLMs scale to billions.
'qqual contribution 'Hong Kong University of Science and Technology (Guangzhou) 'SAP 'National University Singa- pore "Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ). Correspondence to: Yao Shu <yaoshu @ hkust- gz.edu.cn>>.
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
One of the straightforward strategies to mitigate the pro- hibitive communication costs in the federated tuning of

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
LLMs is parameter-effiiient fine-tuning (PEFT). PEFT
competitive model accuracy.
methods (Hu et al., 2022; Wei et al., 2024) focus on fine- tuning only a subset of model parameters, which is able to significantly reduce the communication overhead between clients and a central server (Che et al., 2023; Zhang et al., 2023; Kuang et al., 2024; Zhang et al., 2024b). Despite the effectiveness in reducing bandwidth usage, this type of approach often compromises model accuracy (Pu et al., 2023), as fine-tuning subset of model parameters may fail to fully capture the nuances of local data distributions. Thus, recent efforts, e.g., FedKSeed (Qin et al., 2024), have been devoted to utilizing zeroth-order optimization (ZOO) (Nes- terov & Spokoiny, 2017; Berahas et al., 2022) in federated full-parameter tuning of LLMs, aiming to maintain com- petitive model accuracy while reducing the communication overhead by transmitting only thousands of scalar gradients per round between clients and central server. Unfortu- nately, this approach often suffers from its poor scalability, including increased computational cost per round and a larger number of communication rounds required for convergence, compared to FL methods that use first-order optimization (FOO), e.g., FedAvg (McMahan et al., 2017).
We present rigorous theoretical analyses and insights to support the effectiveness of our Ferret, demonstrating its theoretical advantages over other baselines and guiding its best practices.
Through extensive experiments, we demonstrate that Ferret consistently improves over existing methods in practice, offering both superior scalability and competi- tive model accuracy.
2. Problem Setup
In this paper, consider the federated full full-parameter tun- ing of an LLM using decentralized data {D,} N local clients while preserving data privacy, i.e., without sharing raw data. Specifically, given a loss function ((ss),, we aim to minimize a global objective [(w) defined as the average loss across (D,)==1 over the model parameters w E Rd of an LLM. That is,
min L(w)
A
1 N
Σ (w)(W)
Therefore, we develop federated full-paramgter Luning at scale for LLMs (Ferret), the first first-order FL approach with shared randomness to enable scalable federated full- parameter tuning of LLMs with compelling computational efficiency, reduced communication overhead, and fast con- vergence speed, while maintaining competitive model accu- racy, as shown in Fig. 1. Ferret achieves this through three aspects: First, it uses widely applied first-order methods to perform efficient local updates on each client, which usually requires fewer iterations to achieve the same local update process compared to existing Z00-based FL.. Next, Ferret projects these updates into a low-dimensional space, result- ing in a significantly reduced communication cost compared to existing FOO-based FL.. Finally, Ferret reconstructs lo- cal updates from the low-dimensional space with shared randomness for effective full-parameter global aggregation, ensuring fast convergence and competitive model accuracy compared to existing Z0O-based FL. We further comple- ment Ferret with rigorous theoretical analyses and insights, showing the theoretical advantages of Ferret over other baselines and guiding the best practices for its implementa- tion. Finally, through extensive experiments, we verify that Ferret significantly outperforms existing methods with su- perior scalability and competitive model accuracy, making it a desirable solution for deploying LLMs in large-scale federated environments.
(EIN]
(I)
where C (0))
ED, [e(w:x())]
W
Following the practice in federated learning (FL), (1) can be solved through multiple rounds of local training and global aggregation. In each communication round, each client i independently updates its local model parameters by mini- mizing its local objective (00 (w) based on its local data Dj. After local training, the clients transmit their updated local model parameters to a central server, where they are aggre- gated to form an updated global model. This updated global model is then redistributed to all clients, and the process is repeated over rounds.
The main challenge in LLM federated full-parameter tuning is to ensure the computational efficiency and the conver- gence speed of the global model while reducing the com- munication overheads, particularly given that the param- eter size d of LLMs often reaches billions. While existing first-order FL (McMahan et al., 2017: Li et al., 2020; Karim- ireddy et al., 2020) can ensure compelling computational efficiency and convergence speed by applying first-order up- dates, they typically incur (dd) communication overheads due to the need to transmit the entire set of model parameters between clients and the central server. This type of methods hence is impractical for LLM federated full-parameter tun- ing due to the enormous size of LLMs. In contrast, although zeroth-ordes FL (Qin et al., 2024) can reduce these commu- nication costs by transmitting only several scalar gradients from their finite difference- based gradient estimation with shared randomness, they often incur more computational cost to achieve the same local update progress and larger
To summarize, our contributions in this work include:
We novelly propose Ferret, to the best of our knowledge, the first first-order FL approach with shared random- ness, which significantly enhances the scalability of fed- crated full- palamereetter tuning of LLMs while maintaining
2

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
number of communication rounds to converge compared with first-order FL.. These naturally raise the question:
1 N
Σ iE|N
A
with A
(
W W--+tW W
kE[K]
(2)
Step 2: Local Updates (Line 7-9 in Algo. 1). After Step ①. each client j will perform T-iteration first-order opti- mization on its local loss function by using the randomly sampled data for its local updates. Formally, if stochastic gradient descent with local learning rate v) is used, the
Can we combine the strengths of these different types of methods to achieve scalable federated full-parameter tuning of LLMS with high computational efficiency, re- duced communication overhead, and fast convergence?
update rule for client j E [N] at iteration & E [T] of round r E [R] can then be represented as below with W,,0 + W,:
3. The Ferret Algorithm
W 7, 7- Ve
()) W -
(3)
To answer this question, we introduce Ferret. federated full- parameter tuning at scale for LLMs, in Algo. 1. We present an overview of Ferret algorithm in Sec.3.1, followed by a detailed explanation of its key techniques in Sec. 3.2.
Different from the zeroth-order update in (Qin al., 2024) that requires many local update iterations, the first-order update in (3) enables each client to efficiently and effectively adapt the global model w, to its specific data using a small T, thereby enhancing both the computational efficiency of this local update. Here, (3) can be implemented using any gradient method variant, c.g., Adam (Kingma & Ba, 2014).
3.1. Overview of Ferret
To achieve scalable LLM federated full-parameter tuning, our Ferret algorithm combines the strengths of both first- order FL, which offers efficient computation and fast conver- gence, and zeroth-order FL, which reduces communication overhead. Specifically, Ferret (a) follows first-order FL to apply first-order optimization methods for local updates clients, ensuring both computational efficiency and fast con- vergence, and (b) draws inspiration from zeroth-order FL by projecting updates into a oiw-dimeniional space using random bases that can be regenerated using shared random- ness among clients for the reconstruction of these updates, thereby reducing communication overhead.
Step 3: Projected Updates (Line 10-12 in Algo. 1). After completing the local updates above, each client j randomly chooses a single new seed s0t) to generate K new random bases {v
( H==1 and employ these K new random bases to project the local update ()) into a K K-ddimensional coordi- K
nates ( based on the techniques in See. 3.2. Seed s()) and projected coordinates { Jk=1 are then shared with other clients to facilitate the next round of global ag- gregation. By sharing only one single random seed and K projected coordinates among N clients where random bases { can be regenerated for global aggregation as shown in Step ① above, the communication overhead in LLM aall-parmeeter tuning is therefore considerably re- duced compared with first-order methods (e.g.. FedAvg) especially when T < d. The communication of seed s can be mitigated if the same seed is used across all rounds E [R]. which can further reduce communication overhead.
jk=1
Our Ferret algorithm operates by repeating the following three sequential steps over many communication rounds, denoted by r E [IR]. where R is the total number of rounds. For simplicity, we omit the subscript r from the seeds, ran- dom bases, and projected coordinates in our notation.
Step D: Global Aggregation (Line 3-6 in Algo. 1). At the beginning of the first round (r = 1), each client initializes its local model parameters using the pre-trained model pa- rameters Wo- i.e., W- For subsequent rounds (r > 1), each client j E [N] receives the random seeds s and the corresponding K projected coordinates (1%²)) =k=1 of every client E [M] from the previous round. These random seeds (i.e., shared randomness) are then used to generate K =k=1 for each client i. These random bases, along with the corresponding projected coordinates (x 1, are applied reconstruct local up- dates as İ (1) in every client i. The global model is then updated by aggregating these local contributions as follows:
3.2. Update Projection and Reconstruction
As mentioned before, we aim to project the local updates into K-dimensional coordinates (K & d) to substantially reduce the communication overhead in LLM full-parameter tuning. To accomplish this, let A E Rd denote any local up- date, and let V = [v1 V2 ... Vxl E dx represent the K random bases generated by any random seed s, we solve the following convex minimization problem to determine the K- "<k]
(3) (>
d-dimensional random bases {v(. |
dimensional projected coordinates 7 [71 72
arg min Vy - - y
(4)
As V is singular with K < d, the close-form of y and its corresponding reconstruction À will be
As in (Qin et al., 2024), we can get K random seeds from single seed 8 (9) and them to generate K random bases inde- pendently, So, one seed is sufficient for each client.
7 = >V A= V-V A. (5)
3

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
random bases [v when computing {7 sequen- tially. Consequently, approximately using A below
Algorithm 1 Ferret
reconstruct the true update A
Input: Wo, N, R. T. K. for each round r E [ do
for each client
[M] in parallel do
A 3=(K))'VV'' 'V A
(7)
if r > 1 then // Step Q Global Aggregation li=1 Receive ) and (7k
whose efficacy will be theoretically justified in Sec. 4.1. Finally, our (6) and (7) simplify the update projection and reconstruction in (5) into straightforward matrix multiplica- tions.
N.K
Ji=1,k=1
Generate bases {vR') R==1-=1 using )
N.K
W, ",-1tw,-2 DiE(N EK Th vk
(0)
Wr0 + W
Block-Wise Reconstruction. The computational complex- ity of O(Kd) and storage complexity of (max K, (,d)) for our reconstruction in (7) is still prohibitively costly, particu- larly for LLMs with billions of parameters. To address this, we propose a block-wise reconstruction technique to reduce both computational and storage complexities. Specifically, suppose the full dimension d is divided into L blocks, each with dimension d, such that Σie(ry di = d. Let A, be the update for block and K₁ (with ΣiEJE K₁ = K) be the number random bases allocated to this block. We propose to compute 7/t and reconstruct A, using random bases Vr of dimension dj x K₁ as follows:
for 1 E [T] do // Step 2: Local Updates
r,t w Vi(w (3) - x
// Step ①: Projected Updates
Randomly set s (5) and generate bases (vk)
(
10
fk=1 A (5) (1) compute (
11
===1 with (6)
Dij
Send and (Tk) Jk=1 to the central server 12
Choice Random Bases V. Particularly, is a rect- angular matrix with ones on its main diagonal, meaning that each V& is standard basis vector, (5) simplifies to V A, which then corresponds to block-wise di- mension selection for local update projection and recon- struction. However, this approach significantly reduces the number of parameters updated per round as K < d, poten- tially hindering the overall tuning performance. We thus propose to sample each element Vy (A: E [K]) indepen- dently from normal distribution with bounded 2-norm, i.c., l|vlll $ 1, aiming realize and stabilize full- palammrrrr tuning of LLMS for competitive overall performance. To achieve this, we can sample from truncated normal dis- tribution: v ~ (O, 1) with v E [-1/Vd, 1/ l/Vd] instead. The efficacy of this bounded norm will be demonstrated in Sec. 4.1 shortly.
> = k): V D
A = (p( -M/K)) "V,V A.
(8)
Here, P: _ 20(1/ /01///////
This trick reduces the stor-
2011 20(1//4)-1
age complexity to O (mmxx( ( d that is straightfor- ward to verify, and lowers the computational complexity to O Kíd)). Of note, (8) also significantly reduces the computational complexity of global aggregation compared to existing methods (Qin et al., 2024) (verified in Sec. 5). This block-wise reconstruction thus further enhances the scalability of our Ferret in the federated full-parameter tun- ing of LLMss. Interestingly, this Mlock-wise strategy has also been widely applied in other fields due to it efficacy (Frantar & Alistarh, 2023; Luo et al., 2024).
Reconstruction w/o Inversion. Unfortunately, (5) incurs computational complexity of O(K² d K") and storage complexity of ((Kd) owing to the inversion of V V in (5), which is prohibitively costly, especially when K is large and d reaches billions. Since V V is scaled empirical covariance for the aforementioned distribution of an identity covariance matrix (Vershynin, 2012). we propose to ap- proximate T V with I (i.c., x K -dmmenssional identity matrix) and (5) as
4. Theoretical Analyses and Insights
We now provide theoretical analyses substantiate the ef- fectiveness of Ferret: (a) reconstruction analysis in Sec. 4.1; (b) convergence analysis in Sec. 4.2; and (c) scalability and beyond in Sec. 4.3.
Y (pK) - A.
(6)
4.1. Reconstruction Analysis
20(1//d)-1' where vi( 7aaa)) and $ (+aa) is the probability density function (PDF) and cumulative distribution function (CDF) of the standard normal distribu- tion evaluated at 1/Vd, respectively. This approximation leads to improved computational complexity of O(Kd) and storage complexity of O (max K, d)). where the storage complexity reduced due the in-place operations on
Here, / A 20(1/Va//V
Theorem 1 (Unbiased Reconstruction). Given the re- construction in (7), we have
E
A
To begin with, we demonstrate in Thm. I that our reconstruc- tion (7) is unbiased, with the proof provided in Appx. B.1.

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Of note, Thm. 1 shows that (a) the scalar 1/(pK) is crucial for (7) to achieve an unbiased reconstruction of the ground- truth update A, and (b) our (7) avoids the bias commonly found in zeroth-order FL methods (Berahas et al., 2022), including FedZO (Fang et al., 2022) and FedKSeed (Qin et al., 2024). As a result, (7) is expected to provide a more accurate update reconstruction as shown below.
when A is replaced by V x )),, implying that zeroth- order method in fact aims to approximate our reconstruction (7). (b) In practice, > O. So, zeroth-order method leads to a biased reconstruction with an additional error term of Be/2 compared to our (7), and this error will accumulate over T local iterations, implying that our (7) can indeed be more accurate as we have demonstrated above. (c) In addition, zeroth-order method typically coupled with a single gradient (i.e., Ve(w; x )),, whereas our (7) can be applied to any vector, making it more general. Overall, these results further verify the advantages of our (7) over the zeroth-order method used in (Fang et al., 2022; Qin et al., 2024), which we will also support empirically in Appx. C.5.
Theorem 2 (Reconstruction Error). Given the recon-
struction in (7), we have
E Ι - I|]
2ln(2d) p
2In(2d) pK
S max 2
|A||
Proposition 1 (Block- Reconstruction Speedup). For block-wise reconstruction (8) of size L,
We then demonstrate the efficacy of our reconstruction in (7) by theoretically bounding the difference between the reconstructed update Á and the ground truth A in Thm. 2. The proof is in Appx. B.2. Of note, 1/p typically has an asymptotic rate of (d), which we will verify empirically in Appx. C.5. Thm. 2 offers three critical insights of our Ferret: (a) Our reconstruction in (7) incurs a reconstruction error at rate of (d for T local update iterations when vd= 2 K, which generally aligns with the results in (Vershynin, 2010). This indicates that the reconstruction error of our (7) can be linearly reduced by increasing K. (b) Ferret avoids additional constant error items (Berahas et al., 2022) that are caused by the biased estimation in these zeroth-order FL methods, implying that our (7) can be more accurate. We will justify this further in our Thm. 3 below. (c) Thanks to the independence from the iterations (i.e.. T) of local updates in Thm. 2, Ferret prevents the error accumulation over the local update iterations T, which is a common issue in zeroth-order FL methods (Fang et al., 2022; Qin et al., 2024).
)
Σ diki LE[L]
Σ de VELLI
K₁ = d.k. LEILI
Σ
We next highlight the computational advantage of our block- wise reconstruction (8) in Prop. 1. The proof is in Appx. B.4. Prop. I indicates that by dividing the reconstruction of d- dimensional updates into smaller blocks {d we get a reduction in overall computational complexity that is strictly less than that of the full dimension d in (7). E.g.. when d1 = = dz and K₃ = Σiery] Kid, Kd/L, L, showing that our block-wise recon- struction (8) reduces the computational complexity of (7) by factor of I/L. This implies that increasing the number of blocks L can further enhance the computational efficiency of our block-wise reconstruction (8).
KL, we have
Proposition 2 Bllock-iise Reconstruction Error). For block-wise reconstruction (8) size L, when vdd z K₁ for any I E [L],
Theorem 3 (Connection with Zeroth-Order Method), Define 9k A ) ()
in which each el- ement v in Va is sampled from v ~ NO 1) with A [91 9k] and V A [V1 V22 vxl Rd assume E(::-) is B-smooth w.r.t its first argument, the zeroth-order reconstruction Vg/ K used in (Fang et al., 2022; Qin et al., 2024) then incurs:
EE
||
11]
<Ō VE(T
Σ
الها prki
V VET 1-1/Vd,1/VB
which is minimized by choosing K₁ x v A /pi.
We conclude by analyzing the error induced by our block- wise reconstruction (8) and the corresponding optimal ran- dom bases allocation in Prop. 2. The proof is provided in Appx. B.5. Prop. 2 demonstrates that reconstruction er- ror can be minimized by adaptively allocating the number of random bases according to the gradient norm of each block. This is intuitively reasonable because larger gra- dient norm typically indicates a need for more immediate model updates in practice. Hence, this insight not only provides a thcoretical foundation for optimizing Ferret but also offers practical guidance. That is, by aligning the num-
1 Vg- K K
1 1 VV VVVTV((W::'" S Se.
We then show in Thm. the connection between our update projection (6) and zeroth-order method used in (Fang et al., 2022; Qin et al., 2024). The proof is provided in Appx. B.3. Thm. 3 delivers three essential insights: (a) When + 0, the reconstruction Vg/ K in zeroth-order method is equivalent to VV (()) K and shares a similar form of (7)
5

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
ber of random bases with gradient norms, practitioners can enhance reconstruction accuracy and overall model perfor- mance. This adaptive approach ensures efficient use of com- putational resources, making Ferret versatile and effective across different datasets and federated learning scenarios.
4.3. Scalability and Beyond
With the theoretical results above, we summarize the scal- ability of Ferret and compare it to existing methods like zeroth-order FL (c.g., FedZO and FedKSeed) and first-order FL (e.g., FedAvg) in Tab. 1.
4.2. Convergence Analysis
Computation Per Round. Of note, Ferret enjoys a com- putational complexity of O(T,T) for any client i E [N] per round, where 7i is the per-iteration complexity of the first-order update (including forward and backward passes) in (3), and T is the number of local iterations. This is com- parable to the well-established FedAvg. In contrast, both FedZO and FedKSeed incur a complexity of (T, with To being the per-iteration complexity of the zeroth-order update (i.e., forward pass) and K representing the number of forward passes. As first-order updates use more accurate gradients, T will be smaller than K (i.e., T < K) to attain the same local update progress. Although T1 can be at most twice To. our Ferret is still more computationally efficient than FedZO and FedKSeed (see Sec. 5).
In this subsection, we present the convergence of Ferret in our Thm. 4 below when using stochastic gradient descent (SGD) for the local updates in (3). To simplify the analysis, we primarily focus on deriving theoretical results for a ho- mogeneous setting, where ( (w) [(w) in (I). Results in the heterogeneous setting can be derived by following the same proof idea.
Theorem 4 (Convergence). Define D - ((wo) minw (. Assume that [(w) is 8-smooth and non- convex, and E|ll (w) - V((w: x) S o² for any x, w, when choosing n S 203T in Algo. 1, the fol- lowing holds for federaled full-paramett tuning with c(0( (w) = -(ww,
Communication Per Round. As only one seed and K (0) ) Jk=1 from client E [AV] need to be transmitted per round in Algo. I with K < d. Ferret incurs a communication overhead of O(K). ). which is similar to that of FedKSeed. This is significantly more efficient than FedAvg and FedZO, which have a communication complex- ity of (d) due to their need to transmit the entire model (or gradients). This significantly reduced communication cost therefore makes Ferret especially suitable for federated full-parameter tuning of LLMs with billions of parameters.
projected coordinates (7k
min E [IVZ(w,)1)] s so 5 ( TTTO² rE(R)
D + +
where (Rt is the half-open interval [o, R). Especially, by choosing 7 = 202TVR in Algo. 1, the number of communication rounds are required to be R = O(1/e) to achieve an convergence error.
Rounds to Converge. As revealed in Sec our Ferret benefits from unbiased update reconstruction in (7) (vali- dated in Thm. 1), enabling fast convergence with a small number of communication rounds to achieve e convergence error (see Thm. 4). This is significantly more efficient than zeroth-order FL methods like FedZO and FedKSeed, which require many more communication rounds to converge due to poor gradient estimation (Fang et al., 2022). FedAvg, applying the ground truth local update for its global aggre- gation, surely converges with the fewest rounds. Overall, Ferret remains a strong choice for federated full pallmmarreeeer tuning of LLMs, even in terms of rounds converge.
Its proof is in Appx. B.6. Particularly, when T' = 1, Thm. 4 recovers the result of standard SGD (Ghadimi & Lan, 2013). Thm. 4 provides three essential insights: (a) Thanks to our improved update reconstruction (7) as justified above, Fer- ret avoids the additional constant terms accumulated over T local iterations, which are typically caused by the biased gra- dient estimation in zeroth-order FL. methods (e-g., FedZO and FedKSeed) (Fang et al., 2022), thereby highlighting the superior advantage of Ferret over these zeroth-order FL methods in convergence speed. (b) Given a proper 1), Ferret shares the same communication round complexity as SGD, at a rate of O(1/e²), showing that the communication round complexity of Ferret is asymptotically comparable to that of standard SGD. (c) This communication rounds com- plexity is improved over that of zeroth-order FL methods (Fang et al., 2022) due to its independence from d and other constant factors required by these gareer-order FL methods, further highlighting the advantage of Ferret in communica- tion round complexity and its improved efficacy in federated full- call-marretter tuning over these methods.
Beyond Scalability. Our Ferret also offers benefits in adaptability, generalization, and privacy. Unlike FedKSeed, which is limited to SGD, Ferret is highly adaptable, because both global aggregation (2) and local update (3) in Ferret can be implemented with any gradient method variant, e.g., the widely used AdamW (Loshchilov & Hutter, 2019) in LLM training. This adaptability thus makes it much easier to integrate Ferret into existing centralized tuning workflows for LLMs, facilitating a seamless transition to federated tun- ing. Besides, since Ferret enables federated tuning with full
6

---

Ferret: Federated Full- Full-Parameter Tuning at Scale for Large Language Models
Table 1: Comparison of scalability (computation and communication per round, and #rounds to converge) and other factors (adaptability, generalization, and privacy). Here, > K > T. Symbols: o (fewer is better), 2 and (more is better).
Scalability
Others
Method
Type
Comp.
Comm.
#Rounds
Adapt.
Gen.
Privacy
FedZO
ZOO
() ((d)
000
FedKSeed
ZOO
((T ((K)
000
x
0
FedAvg
FOO
((r)T)
O(d)
Ferret (ours)
FOO
((T)))
O(K)
0.0
parameters, it is expected to deliver strong generalization performance as other federated full-parameter tuning meth- ods like FedAvg, as supported in Sec. 5. Finally, by transmit- ting only seeds and low-dimensional projected coordinates among clients, rather than the entire model (or gradients) in FedZO and FedAvg, Ferret ensures improved privacy for federated full-pan -araametrr tuning of LLMs.
Table 2: Comparison of Rouge-L (%) among various algo- rithms. Each cell reports the mean + std of Rouge-L scores from the final round of four runs, each with different ran- dom seed. The highest and second-highest scores are shown in bold and underline, respectively.
Natural Instructions
Dolly-15K
Algorithu
DataJuicer11.3
LLaMA-3B
Dataluicer
LLaMA-3B
Overall, Ferret strikes an optimal balance between compu- tational efficiency, communication overhead, convergence speed, and other critical factors such as adaptability, gener- alization, and privacy. This makes it a scalable and desirable solution for federated full-pat aalammrrrettr tuning of LLMs.
FedPTuning
19.61 a: 2.71
25.41 1: 1,14
23.98 a: 3.23
30.30 :: 1.16
FedPrompr
6.04 + 0.12
8,95
2.47
32.73 + 0.87
24.50
4.78
FedTT-SGD
0.85
27.23 0.68
29.28
FedIT
22,30 +: 0.42
0.50 28.13
30.80
0.98
31.23 + 1.51
FedZO
21.74 1.91
29.46 0.38
26.99 0.17
31.67 = 0.35
FediKSed
22.33 + 1.72
29,77 0.75
30.91 0.29
34.56 0.28
FodAvg
23.95 +: 2,76
32.11 + 0.70
29.67 = 1.26
30,98 = 1.66
Ferret (ouns)
24.99 0.99
20.03 0.99
30.63
0.84
34.57 0.57
5. Experiments
Table 3: More comparison of Rouge-L (%) among various algorithms. Each cell reports the mean + std of Rouge-L. scores from the final round of four runs, each with a different random seed. The highest and hiihhst scores are shown in bold and underline, respectively.
In this section, we evaluate the efficacy of Ferret, following the practice in FedKSeed (Qin al., 2024). We primarily compare Ferret with other federated full-parameter tuning baselines, including both zeroth-order methods (e.g., FedZO (Fang et al., 2022) and FedKSeed (Qin et al., 2024)) and first-orde methods (e.g., FedAvg (McMahan et al., 2017)). Our evaluations use DataJuicer- 1.33 (Chen et al., 2023) and LLaMA-3B (Touvron et al., 2023a) on the Natural Instruc- tions (Wang et al., 2022) and Dolly-15K (Conover et al., 2023) datasets, as well as larger models (i.e., LLaMA2-7B and ILaMA2-13B (Touvron et al., 2023b)) the CodeAl- paca (Chaudhary, 2023) and GSM8K (Cobbe et al., 2021).
CodeAlpaca
GSMSK
Algorithm
LLaMA2-7B
LLaMA2-13B
LLaMA2-7B
LLaMA2-13B
FodIT
4.66 0.18
6.10
30.31 =
13.46 0.34
FedZO
4.58 0.26
6.19 0.32
30.41 1 0.31
13.63 1 0.34
FedKSeed
8.33 = 0.98
10.70 +: 0.47
28.26 +: 3.60
33.67 = 1.15
FodAvg
15,41 + 0.43
14.68 + 0.26
38.30 = 0.40
39.82 = 0.17
Ferret (ours)
12.10 : 0.47
11.84 + 0.91
36.10 1
34.50 1.42
FL Settings. In each round of federated learning, 5% of clients were randomly selected to participate. Following the same practice in FedKSeed (Qin et al., 2024), we set the total number of communication rounds to 40 for the NI dataset and 60 for Dolly-15K for all baselines. Due to the compelling efficiency of our method, we set the total number of communication rounds to 12 for the NI dataset and 20 for D5K for Ferret. However, for more com- plex tasks such as CodeAlpaca and GSM8K, we run all algorithms, including our Ferret, for 20 rounds to ensure a fair comparison. First-order baselines trained locally for one epoch, and FedKSeed trained for 200 steps, while our Ferret algorithm trained for 10 iterations (i.e., T = 10 in Algo. 1). The K value was set to 4096 for FedKSeed. All
approaches perform local update with a batchsize of to re- duce memory consumption. For each local update iteration in Ferret, we accumulate the gradients from 4 samples.
More experimental details and ablation studies are provided in Appx. C.I and Appx. C.5, C.6 respectively.
5.1. Comparison on Accuracy
We present the model accuracy of different federated tuning methods in Tab. 2 and 3. The results in Tab. 2 demonstrate that federated full hall-parameter tuning methods (including Fe- dAvg, FedZO, FedKSeed, and Ferret) generally achieve better model accuracy compared PEFT-based federated
7

---

Ferret: Federated Full- Full-Parameter Tuning at Scale for Large Language Models
Table 4: Comparison of per-round computational cost and communication overhead on LLaMA-3B, including (a) the computational costs of local updates, global aggregation, and overall cost per round; and (b) the per-round commu- nication cost. The improvement achieved by our Ferret is reported in brackets using blue (compared with FedKS and orange (compared with FedAvg).
FedKS FedAvg Ferret
FedKSood
FadAvg
Text Loxs
Loss
4
Ferrot
2
2 4
10
12
0
2
10
12
Number of Rounds
Number of Rounds
Computational Cost (Sec.)
Communicution Cost
(a) DataJuicer-
(b) LLaMA-3B
Algorithm
Local Updaze
Global Aggr.
Overall
(f parums.)
Figure 2: Comparison of communication rounds required by Ferret, FedKSeed, and FedAvg for convergence on Natural Instructions with (a) DataJuicer-1.33 and (b) LLaMA-3B.
FedZO)
32.6
32.9
6.0x
FedKSeed
56.9
123.8
180.7
8,2x10'
FedAvg
1.8
0.3
2.1
6.0x 10"
Ferret (ours)
5.6 (10.24)
24,7 (5.0x)
30,3 (6.0x)
7,8=10 (10"x)
Table 6: Peak GPU memory footprint of different methods.
Table 5: Comparison of per-round computational cost and communication overhead on ILaMM2-7B, focusing on (a) the computational costs from local updates, global aggrega- tion, and the overall cost per round; and (b) the per-round communication cost. The improvement achieved by our Ferret is reported in brackets using blue (compared with FedKSeed) and orange (compared with FedAvg).
Method
DataJuicer- 1.3B
LLaMA-3B
FedAvg
9.9 GB
19.1 GB
FedKSeed
3.5 GB
7.8 GB
Ferret (ours)
9.9 GB
19.1 GB
GSM8K using LLaMA2-7B (see Tab. 5), where the caleu- lation of computational cost and communication overhead is in Appx. C.2 and more comparison on LLaMA2-13B is in Appx. C.4. The results in Tab. 4 and Tab. 5 demonstrate that compared with FedKSeed, Ferret achieves substantial reductions in computational costs: a 10.2 improvement for
Computationul Cost (Sec.)
Communicution Cost
Algorithm
Local Update
Global Aggr:
Overall
params.)
FedZO
54.1
0.7
$4.8
10
FodKSoed
117.0
$10.0
627.0
8.2x
FedAwg
Ferret (ours)
8.9 ())
88.3 (5.8x)
97.2 (x.5
6.4x10' (10"x)
local updates on LLaMA-3B and 13.1x on LLaMA2-7B, 5.0x improvement in global aggregation on LLaMA-3B and 5.8x on I.LMMA27B,, as well 6.5x improvement for overall computational cost per round on LLaMA-3B and 6,8 on LLaMA2-7B. These advancements stem from several key innovations: our first-order local updates, which reduce the number of required iterations; block-wise recon- struction, which optimizes global aggregation; and precise reconstruction, which significantly decreases communica- tion round complexity. Furthermore, compared to FedAvg that does not leverage any shared randomness, Ferret ex- hibits an enormous reduction in overall communication costs, i.e., 10 x on LLaMA-3B and 10 x on LLaMA2-7B. This emphasizes the ability of Ferret in scaling federated full aall-parameter tuning.
tniing methods (such as FedPTuning, FedPrompt, FedIT- SGD, and FedIT). This underscores the importance of full- parameter tuming for Large Language Models (LLMs). Im- portantly, the results in both tables show that our proposed method consistently delivers strong or competitive perfor- mance across four different scenarios. Specifically, on the Natural Instructions dataset, our method outperforms all others for different model sizes, with up 2.66% im- provement over the next best method, FedKSeed. On the 15K dataset, our method maintains competitive per- formance. Moreover, on both the Code.Alpaca and GSM8K datasets, our method achieves noticeably improved accuracy over FedIT and other zeroth-order baselines (i.e., FedZO and FedKseed). Of note, Ferret slightly underperform FedAvg, likely due to reconstruction errors caused by our method for these complex tasks. Overall, these results have well demon- strated the ability of our method to sustain strong model accuracy in practice across various datasets and model sizes.
In Fig. 2, we also compare the convergence speeds of our Ferret with other baselines (e.g., FedKSeed and FedAvg) Natural Instructions (with DataJuicer- 1.3B and LLaMA- 3B). The findings show that, Ferret converges remarkably fast, requiring only two communication rounds in line with FedAvg compared to the 40 rounds needed by FedK This results in a 20x reduction in communication round complexity for both DataJuicer-1.33 1.3B and LLaMA-3B.
5.2. Comparison on Scalability
Since we focus on federated full-parameter tuning of LLMss, we primarily provide a detailed scalability comparison of this type of methods, including FedZO, SeddSeed, FedAvg, and Ferret. We evaluate their scalability performance on Natural Instructions using I.L.MA-3B (see Tab. 4) and
5.3. Comparison on GPU Memory Consumption
We further provide the comparison on GPU memory con- sumption for different methods in Tab. 6. The results demon-
8

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
-P- fi
I= 181
284
22
225
Tost Loss
Rouged (%)
30-
Text Lass
Boudoe Toogee (5))
2.00
(Sec.)
1.-
1.5
10-
1.3
1225
K1
2.5g
4.5
4 Number of Rounds
52
2 62
2 122
2 154
Number of Rounds
(b) Generalization Figure 3: Convergence and generalization of our Ferret under varying K on Natural Instructions with DataJuicer- 1.3B where 2K, corresponds to the communication cost of 7.8x 10 per round in Tab. 4.
(a) Convergence
(a) Convergence
(b) Generalization & Time Cost
Figure 5: Convergence, generalization, and projection time cost per round of Ferret under varying L on Natural Instruc- tions with DataJuicer-1 1.3B where L = 194 is used in Tab. 2.
ing Allocation of K. In Fig. 4, we present the conver- gence and generalization of Ferret under different alloca- tion schemes for K in our block-wise reconstruction, using the same experimental setup described in Appx. C.I, where K2 x ||,,| corresponds to our results in See. 5. Notably, Fig. 4 shows that Ferret achieves both faster convergence and improved generalization performance by following the best practices guided by Prop. 2. These findings therefore validate the significance and correctness of our Prop. 2.
1.2
Ri L--
Less
Ri |Ail
Rouge-1 (%)
1.0
8
28
0.8
10-
8
12
16
20
Number of Rounds
(a) Convergence
(b) Generalization
Convergence and Generalization of Ferret under Vary- ing In Fig. 5, we present the convergence, generalization, and projection time cost of Ferret under varying block sizes L on the Natural Instructions dataset with DataJuicer-1.3B. using the same experimental setup described in Appx. 20. Notably, Fig. 5 shows that increasing the number of blocks (i.c., a larger L) leads to improved convergence and reduced time cost for projection and reconstruction, while the gener- alization performance remains comparable. This improved convergence is likely due to the logarithmic term in the re- construction error of our (7), reduces the dimensionality of each block, thereby minimiz- ing reconstruction error. In addition, the reduced time cost aligns with our analysis in Sec. 4.1 and the empirical results shown in Fig. 8(c), further highlighting the efficacy of our bise reconstruction method (8).
Figure 4: Convergence and generalization of Ferret under varying allocation scheme of K for our block-wise recon- struction, in which K, x || || corresponds to our results in the previous sections.
strate that our proposed method Ferret maintains the same peak GPU memory footprint as FedAvg, requiring 9.9 GB and 19.1 GB for eataJuicer-1.33 and LLaMA-3B models, respectively, on the Natural Instructions dataset. Although FedKSeed, as a zeroth-order method, exhibits lower mem- ory requirements, our Ferret delivers superior converged performance (refer to See. 5.1) and improved scalability (refer to Sec. 5.2) without introducing additional memory overhead compared to the baseline FedAvg method.
larger number of blocks
5.4. Ablation Studies
Convergence and Generalization of Ferret under Vary- ing K. In Fig. 3, we present the convergence and generaliza- tion of Ferret under varying K on the Natural Instructions dataset with DataJuicer-1.33 1.3rerrr-1BB,,... using the same experimental setup as deseribed in Appx. C. Notably, Fig. 3 shows that: (a) a larger number of random bases (i.e., a larger Ko) generally leads to improved convergence, while the general- ization performance remains comparable: (b) 2K, already provides compelling convergence and generalization per- formance, and further increasing K yields only marginal improvements in convergence; and (c) a slight decrease in generalization performance as K increases is likely due to the reduced regularization effect from noisy gradients.
6. Conclusion
In this paper, we introduce Ferret, which offers highly de- sirable solution for scalable federated full- parameter tuning of LLMs. By achieving high computational efficiency, fast convergence, and reduced communication overhead, Ferret overcomes the limitations of existing methods, striking an improved balance among these critical factors. Moreover, our rigorous theoretical analyses and extensive experiments validate Ferret ploying LLMs in large-scale federated settings. While our work focuses on the performance and efficiency of Ferret, a critical area for future research is the in-depth privacy analysis of our method and exploring mechanisms to ensure data security.
robust and reliable approach for de-
Convergence and Generalization of Ferret under Vary-

---

Ferret: Federated Full- Pull-Parameter Tuning at Scale for Large Language Models
Acknowledgements
Fang, W., Yu, Z., Jiang, Y., Shi, Y., Jones, C. N., and Zhou, Y. Communication--ffiiinn stochastic zeroth-order op- timization for federated learning, IEEE Trans. Signal Process. 70:5058-550 5073, 2022.
This research/project is supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMLP- 2024-001). Any opinions, findings and conclusions rec- ommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore. This research/project is supported by SAP and Singapore's 's Economic Development Board under the Industrial Postgraduate Programme.
Feng, H., Pang. T., Du, C., Chen, W., Yan, S., and Lin, M. Does federated learning really need backpropagation? arXiv:2301.12195 2023.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Proc. ICML, 2023.
Impact Statement
Ghadimi, S. and Lan, G. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2368, 2013.
This paper presents work whose goal is to advance the field of federated learning for large language models. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y.. Wang, S., Wang, L.., and Chen, W. Lora: Low-rank adaptation large language models. In Proc. ICLR, 2022.
References
Berahas, A. S., Cao, L., Choromanski, K., and Scheinberg, K. theoretical and empirical comparison of gradient approximations in derivative-free optimization. Found. Comput. Math., 22(2):507-560, 2022.
Hu, W., Shu, Y., Yu, Z., Wu, Z., Lin, X., Dai, Z., Ng, S.- K., and Low, B. K. H. Localized zeroth-order prompt optimization. Proc. NeurlPS, 2024.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S. J., Stich, S. U., and Suresh, A. T. SCAFFOLD: Stochastic con- trolled averaging for federated learning. In Proc. ICML, 2020.
Bernstein, J., Wang, Y., Azizzadenesheli, K., and Anand- kumar, A. SIGNSGD: compressed optimisation for non- convex problems. In Proc. ICML, 2018.
Chaudhary, S.
Code alpaca: An instruction- following Ilama model for code generation. http://github.comm// httpa/hcooo/ashi88144444 14ccoeaapaaa, 2023.
Kingma, D. and Ba, Adam: A method for stochastic optimization. Proc. ICML, 2014.
Kuang, W., Qian, B., Li, Z., Chen, D., Gao, D., Pan, X., Xie, Y., Li, Y., Ding, B., and Zhou, J. Federatedscope Ilm: A comprehensive package for fine-tuning large language models in federated learning. arXiv:2309.00363 2023.
Che, T., Liu, J., Zhou, Y., Ren, J., Zhou, J., Sheng, V. S., Dai, H., and Dou, D. Federated learning of large lan- guage models with parameter-efficient prompt tuning and adaptive optimization. In Proc. EMNLP, 2023.
Kuang, W., Qian, B., Li, Z.. Chen, D., Gao, D., Pan, X.. Xie, Y., Li, Y., Ding, B., and Zhou, J. Federatedscope- A comprehensive package for fine-tuning large language models in federated learning. In Proc. KDD, 2024.
Chen, D., Huang, Y., Ma. Z. Chen, H. Pan, X., Ge, C., Gao, D., Xie, Y., Liu, Z.,, Gao, J., al. Data-juicer: one- stop data processing system for large language models. 2oXiv:2309.02033, 2023.
Lau, G. K. R., Hu, W., Diwen, L., Jizhuo, C., Ng. S.-K., and Low, B. K. H. Dipper: Diversity in prompts for producing large language model ensembles in reasoning tasks. In NeurIPS 2024 Workshop on Foundation Model Interventions, 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction- tuned Ilm, 2023.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. Proc. EMNLP, 2021.
Dorfman, R., Vargaftik, S., Ben-Itzhak, Y., and Levy, K. Y. Docofl: Downlink compression for cross-device feder- ated learning. In Proc. ICML, 2023.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogencous networks. In Proc. ICML., 2020.
10

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81. Association for Computational Linguistics, 2004.
Shao, Y., Li, L.,, Dai, J., and Qiu, X. Character-Ilm: A trainable agent for role-playing. In Proc. EMNLP, 2023.
Shu, Y., Lin, X., Dai, Z., and Low, B. K. H. Het- erogeneous federated zeroth-order optimization us- ing gradient surrogates. In ICML 2024 Workshop on Differentiable Almost Everything: Differentiable Relaxations, Algorithms. Operators, and Simulators. 2024.
Lin, X., Wu, Z., Dai, Z., Hu, W., Shu, Y., Ng, S.-K., Jaillet, P., and Low, B. K. H. Use your instinct: Instruction opti- mization using neural bandits coupled with transformers. In Proc. ICML, 2024.
Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, B. Deep gra- dient compression: Reducing the communication band- width for distributed training. In Proc. ICLR, 2018.
Taori, R., Gulrajani. I., Zhang, T., Dubois. Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stan- ford alpaca: an instruction-following Ilama model. https://gtthub. com/tatsu lat-ttsss--aa///tafooddd alpaca, 2023.
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Proc. NeurIPS, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv:2302. 13971, 2023a.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In Proc. ICLR, 2019.
Luo, Q. Yu, H., and Li, X. Badam: A memory efficient full parameter optimization method for large language models. In Proc. NeurlPS, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv:2307.09288. 2023b.
Malladi, S., Gao, T., Nichani, E., Damian, A., Lee, J. D., Chen, D., and Arora, S. Fine-tuning language models with just forward passes. In Proc. NeurlPS, 2023.
Tropp, J. A. al. An introduction to matrix concentra- tion inequalities. Foundations and Trends@ in Machine Learning. 8(1-2):1-230, 2015.
Maritan, A., Dey, S., and Schenato, L. Fedzen: Towards su- perlinear zeroth-order federated learning via incremental hessian estimation. arXiv:2309.1 17174, 2023.
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., al. Adapted large lan- guage models can outperform medical experts in clinical text summarization. Nature medicine, 30(4): 114-11114222 2024.
McMahan, B., Moore, E., Ramage, D.,. Hampson, S., and y Arcas, B. A. Communication-effiieen learning of deep networks from decentralized data. In Proc. AISTATS, 2017.
Nesterov, Y. E. and Spokoiny, V. G. Random gradient- free minimization of convex functions. Found. Comput. Math,, 1722)5277-566 2017.
Vershynin, R. Introduction to the non-asymptotic analysis of random matrices. arXiv: 11.0111.0000 1.3027, 2010.
Vershynin, R. How close is the sample covariance matrix the actual covariance matrix? Journal of Theoretical Probability, 25(3):655-686 2012.
Pu, G., Jain, A., Yin, J., and Kaplan, R. Empirical analysis of the strengths and weaknesses of peft techniques for 1lms. S.KV:2330..14999 2023.
Wang. Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran, A. S., Naik, A., Stap, D., et al. Suppr-naturalinstructinsss Generalization via declarative instructions on 1600+ nip tasks. In Proc. EMNLP, 2022.
Qin, Z., Chen, D., Qian, B., Ding, B., Li, Y., and Deng, S. Federated full parl-merreeter tuming of billion-sized language models with communication cost under 18 kilobytes. In Proc. ICML, 2024.
Rahimi, M. M., Bhatti, H. I., Park, Y., Kousar, H., Kim, D.-Y., and Moon, J. Evofed: leveraging evolutionary strategies for communication-effiiiee. federated learning. In Proc. NeurlPS, 2024.
Wei, C., Shu, Y., He. Y. T., and Yu, F. R. Flexora: Flexible low rank adaptation for large language models. utXiv:2408.1077 2024.
Rothchild, D., Panda, A., Ullah, E., Ivkin, N., Stoica, I., Braverman, V., Gonzalez, J., and Arora, R. Fetchsgd: Communication-effine efforcienffi federated learning with sketch- ing. In Proc. ICML., 2020.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L.. Van Durme, B., Murray, K., and Kim, J. Contrastive preference optimization: Pushing the boundaries of 1lm performance in machine translation. In Proc. ICML., 2024.

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Xu, M., Wu, Y., Cai, D., Li, X., and Wang, Federated fine- tuning of billion-sized language models across mobile devices, arXiv:2308.1388 13894, 2023.
Zelikman, E., Huang, Q., Liang, P., Haber, N., and Good- man, N. D. Just one byte (per gradient): A note low-bandwidth decentralized language model finetuning using shared randomness. arXiv:2306.1000 10015, 2023.
Zhang, J., Vahidian, S., Kuo, M., Li, C.,, Zhang, R., Yu, T., Wang, G., and Chen, Y. Towards building the federat- edgpt: Federated instruction tuning. In Proc. ICASSP. 2024a.
Zhang, J., Vahidian, S., Kuo, M., Li, C.,, Zhang, R., Yu, T., Wang, G., and Chen, Y. Towards building the federat- edgpt: Federated instruction tuning. In Proc. ICASSP. 2024b.
Zhang, Z. Yang, Y., Dai, Y., Wang, Q. Yu, Y., Qu, L., and Xu, Z. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained lan- guage models. In Proc. ACL, 2023.
12

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
A. Related Work
Federated Parameter- Efficieet Fine-Tuning (PEFT) for LLMs. Federated learning (FL) has become a crucial approach for training large language models (LLMs) while preserving data privacy. significant body of work in this area has focused on parameter-effiiient fine-tuning (PEFT) techniques (Hu al., 2022; Wei et al., 2024; Lester al., 2021; Lin et al., 2024; Hu et al., 2024; Lau et al., 2024) to address the substantial communication costs associated with training massive LLMs in distributed settings. These methods, such as those explored in (Zhang et al., 2023; Kuang et al., 2024; Zhang et al., 2024a; Kuang et al., 2023), aim to reduce the number of trainable parameters, thereby decreasing communication overhead. While PEFT approaches offer viable solution, they often compromise on model accuracy compared to full-parameter fine-tuning, particularly in challenging non-lID iooniindependent and identically distributed) data scenarios common in FL (Pu et al., 2023). In contrast, our work tackles the challenge of federated full-parameter tuning of L.Mss, aiming to achieve accuracy levels comparable to centralized training without the prohibitive communication costs. This gap achieving optimal accuracy with existing federated PEFT methods underscores the need for our approach.
Federated Learning with Gradient Compression. Another line of research explores gradient compression techniques to reduce communication overhead in federated learning. Methods like top-k sparsification (Lin et al., 2018), signSGD (Bernstein et al., 2018), and sketching-based compression (Rothchild et al., 2020) aim to reduce the size of transmitted gradients. Yet, these methods often achieve limited compression ratios and may not be effective for the high- high-dimensional parameter spaces of LLMs, particularly during full- palameaereterr tuning. The millions of parameters involved in full parameeeee LLM tuning far exceed the compression capabilities of these approaches. Furthermore, these techniques tailored for the unique characteristics of LLMs, potentially limiting their effectiveness. In contrast, our proposed method leverages shared randomness within first-order FL to achieve significantly higher compression rates, scaling down the communication cost by orders of magnitude. This specialization for LLMs provides a critical advantage.
specifically
Federated Learning with Shared Randomness. The use of shared randomness has emerged as a promising avenue for reducing communication costs in FL. Methods like (Qin et al., 2024; Xu et al., 2023; Maritan et al., 2023; Feng et al., 2023; Dorfman al., 2023; Zelikman al., 2023; Rahimi et al., 2024) demonstrate that transmitting only random seeds and scalar gradients can drastically reduce communication overhead. However, these methods typically rely on zeroth-order optimization (ZOO) for local updates each client. ZOO methods, while communication--fficieee are computationally expensive, requiring more rounds to converge than first-order methods like FedAvg (McMahan et al., 2017) and FedProx (Li et al., 2020). This computational inefficiency hinders scalability, particularly in large-scale federated environments. In contrast, our work introduces the use of shared randomness within first-order FL., combining the communication efficiency of shared randomness with the computational efficiency first-order optimization. To the best of our knowledge, this is the first time that shared randomness has been successfully integrated with first-order FL for LLM fine-tuning, addressing a critical gap in the field.
B. Proofs
B.1. Proof of Thm.
Suppose is randomly and independently sampled from a truncated normal distribution, i.c., v ~ NNO0, 1) with v e [-1/Vd, we have
E [v] = 0,
(9)
and also
E 2
(+)2 VAR(v)
VAR(v)
- 2 ((1/vd) -
(10)
(01/V)--0(-1))
(1(1/////V
20(1/v -1
where ( ( and Φ ( is the probability density function (PDF) and cumulative distribution function (CDF) of the
standard normal distribution evaluated at 1/Vd, respectively.
13

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
According to Sec. 3.2, each element in V is randomly and independently sampled from the truncated normal distribution above. We therefore have the following conclude our proof:
E
E
VV
E
V&V &
= A.
(11)
pk
pk
B.2. Proof Thm.
To begin with, we introduce the lemma below to case our proof.
Lemma (Matrix Bernstein Inequality, Thm. 1.6.2 in (Tropp et al., 2015)), Let X1 ... Xg be independent, zero mean, and symmetry mutrices of size d d, if & s for any [K], we then have
K Σ
1 + Cln(2d)
X 1=1
S v 2v In(d)
(12)
3
where v & E [ |.
Define Xx (vevī - pla) / K, We have
(0)
X
V& - -Pal
(6)
V T
+ +
<
K
(13)
(c)
( Vx
+p)
K
(d)
where (b) comes from triangle inequality and (c) is due to the fact that outer product V and inner product vy V shares the same operator norm. Finally, (d) results from < 1 and Va Vx S 1.
Besides, we also have
E
X
E
VAVV 20v + I
VAVA
K
E
VAVi vp + =paLa]
K2
(14)
(c)
P - p²) La
K2
(4)
K²
where (b) comes from the fact that v v S and (c) due to the fact that E vvvv T
= plg.
As result, by introducing the results above with a triangle inequality, we have
K E
x
R2
14
<
K
(15)
14

---

Ferret: Federated Full- Full-Parameter Tuning at Scale for Large Language Models
By introducing the results above into Lemma.
A - A
||
=E
1 A
VV
pk
1
S
VV
|
pk
K
(16)
E8'
X
|A
2 In(2d)
1n(2d)
pk
pk
which finally concludes our proof.
Remark 1. Sampling from truncated normal distribution (rather than standard normal distribution) ensures bounded norm, which is crucial for achieving bounded reconstruction error by our method in Sec. 3.2.
B.3. Proof of Thm.
Since the loss function ((:-)) is assumed to be 8-smooth w.r.t its first argument, we then have
f(w + <V ) f( w; ( S (Ve(w: VR +
1 1 ge l|vxll 2| se (Ve(w:x!))) Va + 2 5Be²
(17)
By dividing e on both sides of the inequality above, we have
Js vv Ve(w: (0) S Bc.
1
(18)
2
We therefore can conclude our proof using the results below:
1 K
Vg K VV VVV Ve(ws(())))
(0)
VkSl VVVV Ve(w:
K
K
Vk-9k - VVVV V.vVVee(w: ve(w; x (1)
(19)
K
vI
(( livell
9k
K
(d)
1
3E
where (b) comes from triangle inequality and (d) results from (18) and ||v S 1.
Remark 2. When + 0, (18) indicates that S vī Ve(w; (0), implying that this scalar gradient in zeroth-order method, e.g., FedKSeed (Qin et al., 2024), is an approximation of directional derivative, i.e., our projected update in (6) when A is replaced with Vl(w; (((w; (9)).
B.4. Proof of Prop.
Due to the fact that d Eie(r] dr K ElE(r K₁, and K, > 0 for any / E [L]. we have
dk-
Σ [ELL]
di
Σ Ki
Σ di
Σ Ki
Σ
diKi
(E(L)
(E(Z)
[E[L]
[E[L]
which therefore concludes our proof.
15

---

Ferret: Federated Full- Full-Parameter Tuning at Scale for Large Language Models
B.5. Proof of Prop.2 2
Based on our bise reconstruction in (8) and Thm. have
E
A
@
Σ
A
-
[E(1)
(6)
1)
Σ [E)
(20)
E
A م
[E
22m(2d))
(2de))
+
||
PiKi
PiKi
[E(L)
where (a) is based the definition of Â and A and (b) is from the fact that
A
A
0.
Given that Viz > K2 and we can then use Ô to hide the logarithm term in the result above, the following then holds:
AIl
E
A
(21)
[E(L]
To minimize the upper bound above w.r.t (Ki) with EiEWy| K₁ K, resort KKT conditions. Specifically, define
h [K,,... ,r,'' and the following Lagrangian function based on A > 0:
المه PrKi
F(k, A) 1
+
Σ
(22)
[E[T]]
[E(L)
To minimize (21), for any 1 E [Z]. K2 and À then needs to satisfy the following condition:
OF(k,
A|l /pl
0.
(23)
0k,
K?
That is.
A | K?
A K2
(24)
This finally leads K2 x V /PL. which consequently concludes our proof.
Remark 3. Prop. 2 provides looser bound than Thm. 2, primarily owing to the inequality (6) in (20). Based on this looser bound, one might expect that block-wise reconstruction would incur larger emor compared to the vanilla reconstruction in (7). However, empirical results Appx. C.5 and Appx. C.6 show that block-wise reconstruction yields comparable performance to the vanilla approach.
B.6. Proof Thm.
Of note, we follow the general idea in (Shu et al., 2024) to prove the convergence of Ferret. To begin with, we introduce the following lemmas borrowed from (Shu et al., 2024):
16

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Lemma 2. Let {u1y..., 3 be any vectors in R.. Then the following holds for any 0:
u ll
u _ 2
a
| 11 + 2a
||,||
(25)
u; + uzl
||
S (1 + a) ||:11 + [1 2) ||//|l
(26)
2
Σ
4
S
aull"
(27)
Lemma 3. For any B-smooth function f. inputs x. y in the domain of f. the following holds for any constant 17 >> 0:
| /V y s(y))l < (1 + m/3>|1a0 yl1
Let 17 S 1/(T3), we can bound the discrepancy between w and w, for any client as below
E
w rst
wr
(a)
w r 1 Vel( r.P- 1i - -w.l) well"
(6)
E
(6)
7/ + 7/[[( - -w, + (L 7,1-1) VC (w)()) (1(0--1)
w x
(V
w ----1 Ve(W ( - VL(W,) )) )11
+m
(28)
T
(1)
k/W. -w.// w.ll?
T
E
r,
+ T
|
( (--1) ve( -0( (1-1
+ +[V[(()))]
(8) T(1 + m))2
w - w, 1/ + TT + 2 [V((w,))
E
T 1
(0) 2 24 2T 0 + TT /W((,))]
where (a) is from the local update of w on each client i, and (c) is based on (26) in Lemma with = 11((T- - 1) and L(w (6) ((), (( (w!?) (i) 1) Besides, (d) results from Lemma 3 and the assumption that E [||[V[ (w) Ve(w; s o².
Finally, (e) comes from the summation of geometric series and the fact that n/ < 1/T as well as
(++1) TIT-1)
( 1)3
T(T -1]
(V++)///((T-1I)) -
-
T(T-1
3T 3T 1 +1
T
1+
T(T -1) 1]
-1)
(29)
37 +1
T(-
31 +
-1)
exp
37 +1
T
T
1
exp
3
2
12T
17

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Besides I8 V, V. = plg, one can also verify that E V.V. V.V. = p Ld we therefore have
||r+1 ww,1l
(W,+11 w,)" (w)
2
N
T
N
T
pKN
7
Ve (6)
r.t -
V.
Vl
1,1-11
1=1 t=1
2
E
Σ
N
T
V
V.V.VVVVV. ΣΣ
N
T
V
w 1-
(30)
pKN
2
N
T
Vi(
N
2
N
T
1
E
VE
where the last inequality comes from the (27) in Lemma 2. Here, we omit the subscript from the random bases V in our notation for simplicity.
Since
w Wr
(8)
10] 17 LI
Ei Ve(w
1i x 11] by replacing 7 with T, we have
0
E $1 w.l1
22nm212' +2An2 /L((,)]l
(31)
Besides, since E [Wr+1 w,l = # Em 1= (w/(( we have
E VL(w,) (WW,)) (W,+1- w
VL(W,) TVC (w (22211)
VL(W,) (W (TCC,,,((VVC)/)))
- VC(w ( + VC(W.-1) VV((w,)) + v.((W,))
N
(32)
N
nST |Z(
B E +
W
-w, w.l())
w.
1/7' 1/ [V
4nT
[VL(,,)]] 6,, 32
where (c) comes from Cauchy-Schwerr inequality and the fact that [(w (()) In addition, (d) results from (31).
(2)
Finally, based on the assumption that L is 3-smooth, we naturally have
E (C(w,+1) - -asps)))-_ E VL((w,) (Wr+1 w.)]
3 R [ || - wr11]]
2
(33)
(1912T23 - TE [IV
+ 181 2 30
18

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
By rearranging and letting y S 207 have
B [IV[(w)))
20 E [C(w,) - [(wr+1)] + +3300)T3o",
(34)
T/T
Finally, by summarizing both sides over Rt rounds and scaling them with 1/R, we have the following results to conclude our proof:
min E [VC(w))l' )11 20 (C(w)) minw f(w)) 360m3To"
(35)
rE R3)
nTR
Remark 4. Note that the large constant in (35) arises from our bound in (28) for sufficiently large T, This bound can be improved in practice by considering a smaller instead.
19

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
C. Experiments
C.1. Experimental Setup
Baselines. In line with the comparison in (Qin et al., 2024), we selected four practical methods for federated LLM tuning as our baselines: (I) FedPTuning (Kuang et al., 2024), (2) FedPrompt (Kuang et al., 2023), (3) FedIT (Zhang et al., 2024a), and (4) FedIT-SGD, a variant of FedIT that replaces Adam with SGD. In addition, we included four full-parameter tuning methods for comparison: (1) FedAvg (McMahan al., 2017), (2) FedZO (Fang et al., 2022), (3) FedMeZO, hybrid of FedAvg and MeZO (Malladi et al., 2023), and (4) FedKSeed (Qin et al., 2024).
C.I.1. SETUP ON THE NATURAL INSTRUCTION AND DOLLY-15K DATASETS
Datasets. We conducted our experiments using the Natural Instructions (NI) (Wang et al., 2022) and Dolly-15K (Conover et al., 2023) datasets, following a setup similar to (Qin et al., 2024). For the NI dataset, we allocated 738 training tasks to individual clients for local updates and reserved 119 test tasks for global evaluation, reflecting a non-IID distribution. Meanwhile, for the Dolly-15K dataset, the final task was utilized for global evaluation, while the remaining tasks were distributed among 200 clients with varying levels of label distribution skew. Rouge-L (Lin, 2004) was chosen as the evaluation metric. Given resource constraints, we selected DataJuicer-1.33 1.3B (Chen et al., 2023) and LLaMA-3B (Touvron et al., 2023a) as the base models for our study. The corresponding HuggingFace model paths are "datajuicer/LLLMMa 1B-dj- refine-150B" and Sopenln-resarrch/ppn.lllaaa..3.
Slpere-aarametes. For Ferret, the local update learning rate for each client is set to 1 x 10 10" where the selected - 10-
x 10 10-%) The global aggregation learning rates on Natural Instruction and D5Kly-15K are set to 10.0 and 3.0, respectively, which is search from [10.0,5.0 1.0]. For other baselines in Tab. 1 of our main paper, we reported their accuracy performances using the results from FedKSeed (Qin al., 2024).
learning rate is searched from [2 x 10 10-4, x 10
Prompt Template. In our experiments, the raw input data is pre-processed to follow a structured format, where we warp the input text to the Alpaca prompt template (Taori et al., 2023). The corresponding templates for the NI and Dolly- 15K dataset are shown in Tab. 7 and 8.
Table 7: Prompt template for Natural Instructions.
Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
### Instruction:
{Definition)
### Input: (input}
### Response:
1.1.2. SETUP ON THE CODEALPACA AND GSM8K DATASETS
Datasets. To further demonstrate that Ferret can also improve the capability of larger LLMs for code generation and mathematical reasoning, we conducted more experiments using the CodeAlpaca (Chaudhary, 2023) and GSMSK (Cobbe al., 2021) datasets, following similar federated setup. The CodeAlpaca dataset (of around 8.0k samples) is code dataset that consists of ten programming languages, including C, C#, C++, Go, Java, PHP, Pascal, Python, Scale, and X86-64 Assemble. We exclude the X86-64 Assembly data due to limited samples in the dataset. We uniformly randomly sampled 10% instances from the original data as the hold-out test set for evaluation, and we split the remaining 10% samples into nine subsets based on the programming language category and assign each subset to one client as its local training data. For GSMSK, its official train set is split into three subsets, where each client's 's dataset consists of grade school math questions randomly partitioned from the original dataset, forming a IID distribution. We use the official GSM8K test split as the evaluation dataset. Rouge-L (Lin, 2004) was chosen as the evaluation metric. To demonstrate the scalability of Ferret, we
20

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Table 8: Prompt template for Dolly-15K. If some data instances do not have the context attribute, we will discard the line
*## Input: the template.
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Input: }context)
### Response:
extended the experiments to larger models: LLaMA2-7B and LLaMA2-13B (Touvron et al., 2023a) the base models for our study. The corresponding HuggingFace model paths are Lallaammm/aaaa--775500 and llama/Llama 2-13h-hhh
tlyperparameter. For FedZO and FedKSeed, the local update learning rate is set to 3 x 10 for all models. For FedAvg on both LLaMA2-7B and LLaMA2-15 13B, the local update learning rate for each client is set to 10 4 and the global aggregation learning rate is set to 1.0. For Ferret on L.LMMA-7B, the local update learning rate v is set to 3 x 10 and the global aggregation learning rate is set to 5.0. For Ferret on LLaMA2-13B, the local update learning rate n is set to 5 x 10 and the global aggregation learning rate set 10.0. The selected learning rate is searched from [5 x 10 4 3 10 .1x 10 *] and the selected global aggregation learning rates is searched from [10.0, 5.0, 1.0).
C.2. Calculation of Computational Cost and Communication Overhead
In this subsection, provide the details of how the computational cost and communication cost are calculated for all methods listed in Tab. 4.
Calculation of Computational Cost. For FedZO, we follow the same hyper-parameters (b, 200, b2 = 1) for FedZO from FedKSeed paper (Qin et al., 2024), which employs 200 local update steps and perturbation for each local update step. For calculating the computational cost of FedAvg and Ferret, we apply 10 local update steps for each client. Same as our experimental setting in Tab. 1, the batch size is set to for all methods. The time cost incurred at gradient projection also included the Local Update.
In the global aggregation process for both FedZO and FedAvg, raw gradients from all clients averaged and then used to update the global model. In contrast, for FedKSeed and Ferret, the projected gradients are first aggregated through averaging, then reconstructed, and finally used to update the global model.
For the overall computation cost per round, we follow the calculation below:
Overall Local Update + Global Aggr.
Calculation of Communication Overhead. The per-round communication cost refers to the total number of parameters exchanged between client and the central server during single round. This includes both the raw projected gradients that the client sends to the server and the aggregated gradients that the client receives from the server. Each parameter (or projected gradient) is encoded as 16-bit floating point numbers. In accordance with the practice in FedKSeed, we set the number of rounds It to 40 for both FedZO and FedKSeed. Given the notable convergence rate of Ferret, we set R to 12 for both Ferret and FedAvg. Although (Qin et al., 2024) employs R = 40 for FedAvg, we use 12 to provide a strong basis for comparison and highlight the computational efficiency of Ferret.
C.3. More Comparison on LLaMA3-SB and Qwen2.5-7B Models
We conducted additional experiments on CodeAlpaca and GSM8K using LLaMA3-88 8B and Qwen2.5-7B in Tab. 9. The results demonstrate the consistent effectiveness of our Ferret, achieving near FedAvg performance with significantly reduced
21

---

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Table 9: More comparison of Rouge-L (%) among various algorithms on LLaMA3-8B and Qwen2.5-7B models. Each cell reports the mean = std of Rouge-L scores from the final round of four runs, each with a different random seed. The highest and hechne- scores are shown in bold and underline, respectively.
CodeAlpaca
GSM8K
Algorithm
LLaMA3-8B
Qwen2.5-7B
LLaMA3-8B
Qwen2.5-7B
FedZO
16,66 0.50
9.14 + 0.32
7.79 + 1.36
23.84 = 1.19
FedKSeed
5.73 + 1.26
9.14 = 0.32
7.79 + 1.36
23.84 = 1.19
FedAvg
19.88 0.67
17.47 + 0.49
45.48 + 0.51
43.86 + 0.36
Ferret (ours)
19.59 + 0.66
14.64 + 0.74
45.07 + 0.78
38.28 = 1.70
Table 10: Comparison of computational cost and communication overhead on LLaMA2-13B, focusing on (a) the com- putational costs from local updates, global aggregation, and the overall tuning process; and (b) the per-round and overall communication costs. The improvement achieved by our Ferret is reported in brackets using blue (compared with FedKSeed) and orange (compared with FedAvg).
Computational Cost (Sec.)
Communication Cost
Algorithm
Local Update
Global Aggr.
Overall
(# params.)
FedZO
114.1
25.7
139.8
2.6x 1020
FedKSeed
188.4
666.2
854.6
8.2x 10
FedAvg
24.9
25.7
50.6
2.6 10 10
Ferret (ours)
19.2 (9.8x)
169.4 (3.9x)
188.6 (4.5x)
7.6x (10%x) x)
communication overhead across models and tasks.
C.4. More Comparison of Computational Cost and Communication Overhead
Tab. 10 compares the computational cost and communication overhead of LLaMA2-13B using the GSM8K dataset. Because of GPU memory constraints, FedZO and FedAvg have slightly higher computational costs, as gradients need to be stored on the CPU. The results show that even for large models like LLaMA2-13B, Ferret still demonstrates superior scalability. Compared to FedKSeed, Ferret reduces computational costs significantly: 9.8x for local updates, 3.9x for global aggregation, and 4.5x for overall cost per bound. Additionally, compared to FedAvg, which does not utilize shared randomness, Ferret achieves a dramatic 10" x reduction in communication costs. These results, along with the evidence in Sec. 5, further highlight the scalability of Ferret in federated full-parameter tuning.
C.5. Ablation Studies on Reconstruction
Rate of 1/p w.r. Dimension d. In Fig. 6, we present the rate of 1/p where is defined in Sec. 3.2 to verify our claim following Thm. 2. The results in Fig. 6 confirm that 1/p indeed follows a rate of (d).
Comparison of Reconstruction Accuracy between Ferret and ZO Method, In Fig. 7. we present the reconstruction accuracy (measured by cosine similarity) for the d = 10 -dimensional gradient of the function F(x) = Σi xf at a randomly sampled input æ with varying K by using our method in (7) and zeroth-order method with different values of The goal is to compare the reconstruction accuracy of our (7) with that of the ZO method under varying K and The results in Fig. 7 indicate that: (a) our method (7) achieves improved reconstruction accuracy compared to the ZO method, particularly the one with an optimal e = 0, which indeed aligns with the insights from our Thm. 3; (b) both our method (7) and the ZO method exhibit the same increasing rate in reconstruction accuracy as K increases, highlighting the connection between these two methods as implied by our Thm. 3; and (c) this increasing rate is generally linear, which consistent with Thm. 2. These results therefore further verify the insights in Thm. 2 and Thm. 3, and support the advantages of our method (7) over the 70 method.
22

---

Ferret: Federated Full- Full-Parameter Tuning at Scale for Large Language Models
le6
3.0
0.07
- Feree
-B Z0 (t 1.000)
2.5
0.06
-0 Z0 0.300)
201 0.100)
Cosine Similarity
(c 0.030)
2.0
0.05
-X- 20 (r 0.010)
Z0 (c 0.001)
0.057
1.5
0.04
0.03
0.056
340
350
360
0.5
0.02
0.0
0.01
0.0
0.2
0.4
0.6
0.8
1.0
100
200
300
400
500
K
le6
Figure 6: Rate of 1/p w.r.t dimension d.
Figure 7: Reconstruction accuracy of our (8) vs. zeroth-order
method under varying K and c.
0.06
Cosine Similarity
0.10
Cosine Similarity
Ferret
Cosine Similarity
0.075
Time Cost (Sec.)
X
ZO
0.04
0.050
0.10
0.09
0.02
0.025
Ferret
* *
X-
Zo
0.05
0.000
0.00
5
10
50000
100000 150000 200000
0
5
T
d
log L)
(a) Varying T
(b) Varying d
(c) Varying L
Figure 8: Reconstruction Accuracy (measured by cosine similarity between reconstruction and ground truth) of our (8) vs.
zeroth-order method under varying T', and
Reconstruction Accuracy of Ferret under Varying T. In Fig. 8 (a), we present the reconstruction accuracy (measured by cosine similarity) of -iteraaion gradient descent update for the function F(x) Σi= sin² (x;) with learning rate of 0.1, d 5 x 10². = 1, and K 500, using our method in (8) and the zeroth-order (ZO) method described in Thm. with 0.1. The goal is compare the accumulated error from our (8) with that of the ZO method. Interestingly, Fig, (a) shows that method maintains consistent reconstruction accuracy as the number T of gradient descent iterations increases, whereas the ZO method experiences noticeable decline in accuracy. This result implics that our (8) effectively avoids the accumulated error typical in zeroth-order methods, aligning with the theoretical justification provided in Sec. 4.1.
Reconstruction Accuracy of Ferret under Varying In Fig. (b), we show the reconstruction accuracy (measured by cosine similarity) of d-dimensional gradient of F(x) = Σi= s (xr)) ) at randomly sampled input æ; with 1 and = 500, using our method in (8) and the zeroth-order (ZO) method described in Thm. with e 0.1. The goal is to compare the reconstruction accuracy rate with respect to the dimension between our (8) method and the ZO method. Interestingly, Fig. 8 (b) shows that both methods achieve the same reconstruction accuracy rate with respect d. More importantly, when d becomes large, the accuracy rate is approximately linear, which aligns with the theoretical insights provided in Thm. 2.
Reconstruction Accuracy Ferret under Varying L. In Fig. (c), we present the reconstruction accuracy (measured by cosine similarity) and computational complexity (measured by time cost) for the 5.12 x 1 -dimensional gradient of
23

---

Ferret: Federated Full- Pull-Parameter Tuning at Scale for Large Language Models
1.2
50
Adam
SGD
40
Test Loss
Rouge-L (%)
1.0
30
20
0.8
0
12 16
20
Adam
SGD
Number of Rounds
(a) Convergence
(b) Generalization
Figure 9: Convergence and generalization of Ferret under varying optimizers for local updates.
function F(x) = X sin (x:;) at randomly sampled input æ; under varying T. of the same number of dimensions and K 512, using our method in (8). The goal is to study the impact of block size L on our (8). Notably, Fig. 8 (c) shows that our block-wise reconstruction (8) significantly reduces computational complexity (in line with Prop. 1), while maintaining consistent reconstruction accuracy increases. These results further verify the efficacy of block-wiss reconstruction
(8).
C.6. More Ablation Studies on Convergence and Generalization
Convergence and Generalization of Ferret under Varying Optimizers. In Fig. 9, we present the convergence and generalization of Ferret under different optimizers for its local updates, using the experimental setup described in Appx. C.I. Notably, Fig. 9 demonstrates that Ferret achieves faster convergence with improved optimizer (e.g., Adam vs. SGD) while maintaining comparable generalization performance. These findings further support the adaptability of Ferret, as discussed in Sec. 4.3.
24


Reviews:
[{"review_id": "ncjiBkEoWw", "reviewer": "Reviewer_oeGQ", "summary": "This paper introduces Ferret, a method for full-parameter tuning of large language models (LLMs) in federated learning. It primarily addresses the challenge of communication overhead by combining the strengths of first-order optimization (efficient computation and fast convergence) and zeroth-order optimization (reduced communication overhead). The method utilizes shared randomness to project updates into a low-dimensional space, effectively reducing communication costs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer oeGQ for the positive evaluation, particularly recognizing our theoretical analyses, experimental results, and the novelty of our approach. We address the reviewer's concerns below:\n\n> W1: Reconstruction Error\n\nWe clarify that the slight performance difference compared to FedAvg is an expected consequence of the gradient compression inherent in Ferret, which enables massive communication savings. We highlight three key points:\n\n1. Our reconstruction error is theoretically bounded (Thm 2) and can be reduced by increasing $K$. This allows practitioners to balance performance and communication overhead based on their resource constraints (analysis in Appx. C.5).\n2. While the baseline FedKSeed also suffers from reconstruction error due to gradient projection, Ferret achieves more accurate reconstruction, leading to superior empirical performance, as validated in our experiments.\n3. The minimal performance difference (Ferret vs. FedAvg) on CodeAlpaca and GSM8K (Table 3) is vastly outweighed by the $10^6\\times$ reduction in communication cost (Table 4&5). This reflects our intentional design choice to **prioritize communication efficiency**, which we believe is crucial for the scalable and practical deployment of FL systems.\n\n\nWe will explicitly discuss this trade-off and include these discussions in the revised manuscript to further strengthen our paper.\n\n> W2: Computational Resources\n\nWe appreciate the reviewer's suggestion regarding computational optimization. We acknowledge that while Ferret significantly reduces communication overhead, it requires additional computational costs, especially for large models like LLaMA2-13B. We believe this is an area for future work, and we would like to clarify that:\n\n- Compared with the relevant communication-efficient baseline, FedKSeed, Ferret already achieves a significant ($\\sim6\\times$) reduction in computational cost.\n- We have already incorporated optimizations like **Reconstruction w/o Inversion** and **Block-Wise Reconstruction** (Sec. 3.2) to mitigate this computational overhead during projection and reconstruction steps. \n- We recognize that further improvements are possible. We plan to explore techniques such as quantization on gradients or adaptive projection based on gradient sparsity in future research to further reduce the computational burden while preserving communication efficiency.\n\n---\n\nThank you again for the constructive feedback. We will enhance the discussion on these trade-offs in the final manuscript. We hope this response clarifies our contributions and addresses the reviewer's concerns effectively.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have conducted a rough review of the entire proof but may have overlooked some details.", "experimental_designs_or_analyses": "In general, the experimental results are sufficient and convincing.", "relation_to_broader_scientific_literature": "See the Strengths And Weaknesses below for details.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper provides rigorous theoretical analyses to support Ferret's effectiveness, showcasing its advantages over existing methods in terms of computational efficiency, communication overhead, and convergence speed.\n2. The experimental results are sufficient and convincing.\n3. The proposed method and the addressed challenge are both meaningful and novel.\n\nWeaknesses:\n\n1. Reconstruction Error: While Ferret reduces communication overhead, the reconstruction of updates from low-dimensional projections may introduce some error, particularly for complex tasks. This is evident in the slightly lower performance of Ferret compared to FedAvg on the CodeAlpaca and GSM8K datasets. \n2. Despite reducing communication costs, Ferret still requires substantial computational resources, especially for larger models such as LLaMA2-13B. The authors are encouraged to explore ways to optimize computational efficiency in future research.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CZz0qoY8jp", "reviewer": "Reviewer_C3NM", "summary": "In this work, the authors proposed Ferret, a federated learning method for efficient full-parameter fine-tuning of LLMs, combining first-order optimization with random projection-based dimensionality reduction. It uses shared randomness to reconstruct local updates at the server, significantly reducing communication overhead. The authors provide theoretical guarantees on unbiased reconstruction and convergence, alongside experiments demonstrating reduced communication and computational costs compared to existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough evaluation and constructive feedback. We address the points raised below:\n\n> Claims And Evidence\n\n- We appreciate the reviewer's feedback on the comparison to FetchSGD. While both methods use dimensionality reduction, Ferret's technical approach and goal are fundamentally different, establishing its novelty, especially for full-parameter LLM tuning. FetchSGD uses Count Sketch (based on *hashing coordinate indices*) primarily to enable server-side state management (momentum, error accumulation) for a subsequent *biased Top-K sparse update*. Ferret, conversely, uses shared random vectors **v** to directly project the *entire first-order local update vector* **Δ** via dot products (Eq. 6), determined through convex optimization (Eq. 4). Crucially, Ferret aims to *reconstruct an approximation of the full, dense update* **Δ̃** (Eq. 7) for aggregation, vital for maintaining accuracy in full-parameter LLM tuning. This contrasts sharply with FetchSGD's goal of facilitating a *sparse* update. Furthermore, Ferret's design targets an *unbiased reconstruction* (Thm. 1), avoiding the explicit error accumulation required by FetchSGD's biased sparsification step. Ferret's novelty lies in being the first approach to uniquely combine efficient *first-order local updates* with *shared randomness for reconstructing dense updates*, specifically optimized (e.g., block-wise reconstruction) for the unique scale and demands of federated full-parameter LLM tuning.\n- Thanks for the SparseGPT suggestion (citation will be added). SparseGPT uses block-wise processing on *model weights* for *pruning*. Ferret applies it to *federated updates* (Δ) solely to improve computational efficiency during the *dense update reconstruction* step (Eq. 7 -> Eq. 8), making it scalable for LLMs. Ferret's novelty lies in using this block-wise strategy specifically for scaling *update reconstruction* in our federated tuning context, distinct from model pruning.\n\nWe acknowledge the conceptual similarities, but we emphasize that our method Ferret is technically distinct and novel based on our comparison above. In our revision, we promise to cite the SparseGPT paper, add a detailed discussion regarding the block-wise reconstruction, and also highlight the difference between FetchSGD and Ferret.\n\n> Methods And Evaluation Criteria:\n\n- We clarify our FL setting follows the previous literature (FedKSeed), utilizing 738 clients on Natural-Instruction and 200 clients on Dolly-15K datasets in the FL system.\n\n- Yes, some clients may not be leveraged for training the FL model, as we have fewer communication rounds. In each round, we independently and randomly sample 5% of clients for both NI dataset and Dolly-15K dataset to participate. As shown in Figure 2, Ferret converges rapidly (similar to FedAvg), reaching a point where additional training with more clients yield diminishing returns. We think this might be due to the nature of this data source used.\n\n> Experimental Designs Or Analyses\n\n- **Memory Footprint**: We clarify that our primary focus is communication efficiency in a standard distributed data setting where clients can perform backpropagation. While Ferret currently uses a standard SGD optimizer (implying typical client-side memory requirements for backpropagation), we acknowledge that our memory usage could be potentially reduced by integrating memory-efficient optimizers like those in [1] for future works.\n[1] Full Parameter Fine-tuning for Large Language Models with Limited Resources.\n\n- **Additional Experiments**: Follwoing the reviewer's constructive suggestion, we conducted **additional experiments** on CodeAlpaca and GSM8K using **Llama3-8B** and **Qwen2.5-7B**. The results below demonstrate Ferret's consistant effectiveness, achieving near FedAvg performance with significantly reduced communication overhead across models and tasks.\n\n**Table-R3: Performance comparison on Llama3-8B and Qwen2.5-7B models**\n| Algorithm | Alpaca | | GSM8K | |\n|-|-|-|-|-|\n| | Llama3-8B | Qwen2.5-7B | Llama3-8B | Qwen2.5-7B |\n| FedKSeed | $5.73 \\pm 1.26$ | $9.14 \\pm 0.32$ |$7.79 \\pm 1.36$ | $23.84 \\pm 1.19$ |\n| FedZO | $16.66 \\pm 0.50$ | $11.76 \\pm 0.38$ | $37.44 \\pm 0.11$ | $28.04 \\pm 0.13$ |\n| FedAvg | $\\mathbf{19.88} \\pm 0.67$ | $\\mathbf{17.47} \\pm 0.49$ | $\\mathbf{45.48} \\pm 0.51$ | $\\mathbf{43.86} \\pm 0.36$ |\n| Ferret | $\\underline{19.59} \\pm 0.66$ | $\\underline{14.64} \\pm 0.74$ | $\\underline{45.07} \\pm 0.78$ | $\\underline{38.28} \\pm 1.70$ |\n||\n\n> Response to W1 & W2.\n\nPlease see our response under `Claims And Evidence` above.\n\n> Response to W3 & W4.\n\nPlease see our response under `Experimental Designs Or Analyses` above.\n\n\n---\n\nWe sincerely thank Reviewer C3NM for the valuable feedback and are encouraged by the positive remarks on our research idea, theories, and results. We hope our clarification and additional results effectively address your concerns and increase your opinions of our work. We welcome any further questions.", "claims_and_evidence": "One of my major concerns of this work is about the technical novelty. I like this paper as a whole, especially given the interesting topic and extensive theories, however, I feel the technical novelty is somewhat overclaimed.\nAlthough the authors emphasize the unique challenges of federated full-model fine-tuning of large language models, the proposed random projection method resembles existing techniques (e.g., FetchSGD). Furthermore, the block-wise reconstruction shares conceptual similarity with SparseGPT (Frantar et al.), though SparseGPT is not cited. Specifically,\n\n- The method's projection-based approach is intuitive and theoretically grounded. However, similar methods, particularly FetchSGD (ICML 2020), already utilize random projections for federated optimization.\n\n- The block-wise reconstruction technique introduced, although useful, closely mirrors existing weight reconstruction ideas (e.g., SparseGPT), yet SparseGPT is neither cited nor compared.", "methods_and_evaluation_criteria": "I am a bit confused about the federated setup of experiments in this work. How many clients in total are used per round for the proposed method and baselines on each dataset? \n\n- According to the paper, *\"In each round of federated learning, 5% of clients were randomly selected to participate.\"* How many clients in total did you use? \n\n- In addition, you mentioned *\"Due to the compelling efficiency of our method, we set the total number of communication rounds to 12 for the NI dataset and 20 for Dolly-15K for Ferret.\"* So the proposed method may not leverage some clients on NI dataset? (12 $\\times$ 5% = 60%) And for Dolly-15K it is highly likely to happen as well? (otherwise each sampling of clients has to be perfectly non-overlapping)", "theoretical_claims": "The provided theoretical analyses on unbiasedness and convergence are rigorous and sound, representing a significant strength of the paper.", "experimental_designs_or_analyses": "- Although memory footprint analysis is provided in the appendix, the proposed method (Ferret) incurs significantly higher GPU memory costs compared to the zeroth-order optimization method (FedKSeed). Given federated learning's typical deployment constraints on resource-limited devices, this large discrepancy raises concerns about the practicality and optimality of the proposed trade-off between memory usage and communication efficiency.\n\n- LlaMA / LlaMA-2 families are too dated in 2025 and I strongly suggest that the authors consider evaluation on more SOTA LLMs such as Qwen-2.5 and LlaMA-3 families.", "relation_to_broader_scientific_literature": "The absence of a discussion with SparseGPT, despite methodological resemblance in block-wise reconstruction approaches, weakens the credibility of the claimed novelty.", "essential_references_not_discussed": "Frantar et al., \"SparseGPT: Massive Language Models Can be Accurately Pruned in One-shot,\" ICML, 2023.", "other_strengths_and_weaknesses": "S1. I found the research topic of full-model LLM finetuning under FL interesting and prompt.\n\nS2. Effective empirical demonstration of communication efficiency and fast convergence.\n\nS3. Solid theoretical analyses on unbiasedness and error bounds.\n\nS4. The paper is well-written and easy to follow.\n\nW1. Insufficient technical novelty compared to existing projection-based FL methods.\n\nW2. The block-wise reconstruction is conceptually similar to that of SparseGPT while it is not cited and discussed.\n\nW3. Practical concerns (memory and computational complexity) are inadequately addressed.\n\nW4. LLMs utilized in the experiments are somehow too outdated.", "other_comments_or_suggestions": "i think this paper has many merits such as interesting research idea, extensive theories, and nice results. My major concerns are about the technical novelty, and some evaluations of the proposed method. Hence, I will be fully open to the rebuttal, i.e., other reviewers' comments and discussion, and will adjust my score accordingly.", "questions_for_authors": "Please refer to my comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UPsGk2t9uU", "reviewer": "Reviewer_CfcA", "summary": "This work proposes a large-scale federated full parameter tuning framework for LLM, Ferret, which mainly exploits the first-order method of shared randomness. The proposed method mainly consists of three steps: using first-order methods for efficient local updates, projecting these updates into a low-dimensional space, and reconstructing local updates from this low-dimensional space. The method achieves reduced communication overhead and efficient full parameter updates. The article provides comprehensive theoretical and experimental evidence for this method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer CfcA for recognizing the comprehensive theoretical analysis, solid related work, and clarity of our paper. We would like to address your concerns below:\n\n> W1: Performance Comparison with LoRA methods.\n\n- **Clarification on Scope**: Our work focuses on **full-parameter fine-tuning** of LLMs. This choice is motivated by the goal of achieving better performance, as PEFT methods like LoRA may not consistently reach the same performance ceiling [2,3]. We primarily compare Ferret against other full-parameter federated methods (FedZO, FedKSeed, FedAvg).\n- **Comparison with FedIT (LoRA baseline)**: We appreciate the suggestion to compare Ferret with PEFT methods regarding performance, communication, and computation costs. We have provided **additional results** on FedIT (LoRA-tuning, rank=8, alpha=16) and present the results below (Table-R2 and R3).\n    - The results show that while FedIT offers lower computational costs, it incurs significantly higher communication costs compared to Ferret. More importantly, Ferret maintains a strong performance close to FedAvg and outperforms FedIT by a large margin.\n    - Due to the limited rebuttal period, we have prioritized the FedIT (LoRA-tuning) baseline. We commit to including results for other PEFT methods in our final manuscript.\n\n**Table-R2: Comparison of Computational and Communication Costs against FedIT**\n| Model | Computational Cost (Overall Sec.) | |Communication Cost (# params.)| |\n|-|-|-|-|-|\n| | LLaMA-3B | LLaMA2-7B | LLaMA-3B | LLaMA2-7B |\n| FedIT | $3.9$ |$4.5$| $4.2\\times 10^6$ |  $6.6\\times 10^6$  |\n| Ferret | $30.3$ | $97.2$| $7.8\\times10^3$ |  $6.4\\times10^3$  |\n||\n\n\n**Table-R3: Performance comparison against FedIT**\n| Algorithm | CodeAlpaca | | GSM8K| |\n|-|-|-|-|-|\n| | LLaMA2-7B | LLaMA2-13B | LLaMA2-7B | LLaMA2-13B |\n| FedIT | $4.66 \\pm 0.18$ | $6.10 \\pm 0.18$| $30.31 \\pm 0.29$ | $13.46 \\pm 0.34$|\n| FedZO  | $4.58 \\pm 0.26$ | $6.19 \\pm 0.32$ | $30.41 \\pm 0.31$        | $13.63 \\pm 0.34$ |\n| FedKSeed | $8.33 \\pm 0.98$ | $10.70 \\pm 0.47$| $28.26 \\pm 3.60$                 | $33.67 \\pm 1.15$ |\n| FedAvg       | $\\mathbf{15.41} \\pm 0.43$        | $\\mathbf{14.68} \\pm 0.26$ | $\\mathbf{38.30} \\pm 0.40$        | $\\mathbf{39.82} \\pm 0.17$ | $\\mathbf{38.30} \\pm 0.40$             | $\\mathbf{39.82} \\pm 0.17$       |\n| Ferret (ours)| $\\underline{12.10} \\pm 0.47$     | $\\underline{11.84} \\pm 0.91$ | $\\underline{36.10} \\pm 1.18$ | $\\underline{34.50} \\pm 1.42$ |\n||\n\n\n[2] Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes\n\n[3] Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs\n\n> W2: Computational cost of Ferret compared to FedAvg\n\n- This is correct and represents an **intentional design trade-off**. The increased computational cost in Ferret stems directly from the gradient projection technique employed to significantly reduce communication costs. \n- This trade-off (higher local computation for lower communication) is common in communication-efficient FL algorithms, including the baseline FedKSeed. However, Ferret significantly optimizes this trade-off: compared to FedKSeed, Ferret achieves a $\\sim6\\times$ reduction in computational overhead while providing substantial communication savings over FedAvg.\n\n\n> W3: The presentation regarding the FL setting\n\nThank you for this constructive suggestion. We will move the FL setup and key ablations into the main paper in the final version.\n\n> W4: Privacy analysis.\n\nWe agree that formal privacy analysis would be valuable. As noted in Sec. 6, we are encouraged to do so for future research.\n\n> Q1: distinguished feature of first-order methods compared to LoRA\n\nIf we understand correctly, we believe you are referring to the comparison between **full-parameter fine-tuning** (like FedAvg or our Ferret using first-order optimization) and **low-rank fine-tuning** (like FedIT, also typically using first-order optimization). The distinguished features are: \n- Full-parameter fine-tuning updates all model parameters, but FedIT only updates a small set of *low-rank* adapter parameters.\n- Standard full-parameter methods (like FedAvg) communicate all parameter updates, leading to high cost. LoRA communicates only the adapter updates (still large, as shown in Table-R2). Our method, Ferret, applies projection technique after ful-parameter gradient computation to significantly reduce communication.\n- As supported by existing literature [2,3] and our results (Table-R3), full-parameter fine-tuning generally achieves higher performance ceilings compared to PEFT methods, especially on complex tasks.\n\n> Q2\n\nWe understand your interest but are unable to address this question due to the anonymity policy during the review process. Thank you for your understanding.\n\n---\n\nWe hope these responses and the additional experimental results effectively address your concerns and clarify the contributions and positioning of Ferret. We welcome any further questions.", "claims_and_evidence": "The proposed method is clearly explained and extensive experiments are performed.", "methods_and_evaluation_criteria": "The proposed method is reasonable. The dataset and model used are reasonable.", "theoretical_claims": "I have checked the theoretical analysis, no major problems are found.", "experimental_designs_or_analyses": "The design of experiments are generally comprehensive. But, in most cases, the authors only compare with FedZO, FedKSeed and FedAvg, the other baselines are not fully compared.", "relation_to_broader_scientific_literature": "This work considers a first-order fine-tuning for LLM in FL on the basis of zero-order methods.", "essential_references_not_discussed": "The related work is solid enough, this work is most related to FedZO and FedKSeed.", "other_strengths_and_weaknesses": "S1: The theoretical analysis of this paper is comprehensive.\n\nS2: Overall, this work is well-written and easy to follow.\n\nW1: Some prompt-tuning and LoRA-tuning baselines are only compared in Table  2. I think larger model fine-tuning in Table 3, communication and computation cost in the following tables are worth to compared with these baselines.\n\nW2: In Table 4 and 5, the cost of Ferret is much greater than FedAvg. Why does this phenomenon happen? \n\nW3: FL setting is significant, should moved to the main paper and give more detailed explanations. As well as ablation study, it is crucial for the effectiveness of the proposed methods.\n\nW4: The privacy of exchanging updated weights could provide a more detailed analysis.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Q1: I am more interested in fine-tuning LLM by LoRA, what is the distinguished feature of first-order methods rather than low-rank fine-tuning?\n\nQ2: I find that this work has a preliminary published workshop version. It could be better to provide the improvement and difference to the original version.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Paper studies federated full finetuning of LLM which has significant computational and communication-overhead challenges. This paper combines techniques form federated first-order optimization and zeroth-order optimization to reduce the communication cost of finetuning LLMs in this federated setting. Paper is well written, and the experimental results are convincing in terms of reduction in communication cost. Reviewers also appreciated the additional experimental results provided during the rebuttal. However, reviewers noted that there is limited technical novelty in the proposed methods, further even though their method reduces communication cost, it may increase the memory cost at the clients when compared to some of the baseline.
Decision: Accept (poster)