Layer-wise Update Aggregation with Recycling for Communication-Efficient Federated Learning
Authors: Jisoo Kim
Sungmin Kang
Sunwoo Lee
Venue: NeurIPS 2025 poster ()
Keywords: federated learning
update recycling
communication-efficient
Primary area: deep_learning
TL;DR: A model update recycling method for communication-efficient Federated Learning.

Abstract: Expensive communication cost is a common performance bottleneck in Federated Learning (FL), which makes it less appealing in real-world applications. Many communication-efficient FL methods focus on discarding a part of model updates mostly based on gradient magnitude. In this study, we find that recycling previous updates, rather than simply dropping them, more effectively reduces the communication cost while maintaining FL performance. We propose *FedLUAR*, a Layer-wise Update Aggregation with Recycling  scheme for communication-efficient FL. We first define a useful metric that quantifies the extent to which the aggregated gradients influences the model parameter values in each layer. *FedLUAR* selects a few layers based on the metric and recycles their previous updates on the server side. Our extensive empirical study demonstrates that the update recycling scheme significantly reduces the communication cost while maintaining model accuracy. For example, our method achieves nearly the same AG News accuracy as FedAvg, while reducing the communication cost to just 17%.

Layer-wise Update Aggregation with Recycling for Communication-llfiiennt Federated Learning
Jisoo Kim', Sungmin Kang", Sunwoo Lee
IInaa University
Incheon, Republic Korea
University of Southern California
Los Angeles, CA, USA
starprin3@inha. edu, kangsung@usc. sunwool@inha, ac.kr
Abstract
Expensive communication cost common performance bottleneck in Federated Learning (FL), which makes it less appealing in real-world applications. Many communication-effciiee FI. methods focus on discarding part of model updates mostly based on gradient magnitude. In this study, we find that recycling previous updates, rather than simply dropping them, more effectively reduces the communi- cation cost while maintaining FL performance. We propose FedLUAR, Layer-wise Update Aggregation with Recycling scheme for communication-effinn first define useful metric that quantifies the extent to which the aggregated gra- dients influence the model parameter values in each layer. FedLUAR selects a few layers based the metric and recycles their previous updates the server side. Our extensive empirical study demonstrates that the update recycling scheme significantly reduces the communication cost while maintaining model accuracy. For example, our method achieves nearly the same AG News accuracy as FedAvg, while reducing the communication cost just 17%.
FL. We
Introduction
While Federated Learning has become a distributed learning method of choice recently, there still exists huge gap between practical efficacy and theoretical performance. Especially, the communication cost of model aggregation one of the most challenging issues in realistic FL environments. It is well known that larger models exhibit stronger learning capabilities. The larger the model, the higher the communication cost. Thus, addressing the communication cost issue is crucial for realizing scalable and practical FL applications.
Many communication-efficiii FL methods focus on partially 'dropping' model parameters and thus their updates. Quantization- FL methods reduce communication costs by lowering the numerical precision of transmitted model parameters, representing each parameter with lower bit-width. Truning-based FI. methods directly remove portion of model parameters to avoid the associated gradient computations and communication overhead. Model reparameeeriizatio--aan -asse FL methods adjust the model architecture using matrix decomposition techniques, reducing the total number of parameters. While all these approaches reduce the communication cost, they commonly compromise learning capability by either reducing the number of parameters degrading the data representation quality.
In this paper, we propose FedLUAR, a Layer-wise Update Aggregation with Recycling method for communication-efieee and accurate FL. Instead of dropping the updates for a part of model
CCoresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

parameters, we consider 'reusing' the old updates multiple times in layer-wise manner. Our study first defines a useful metric that quantifies the extent which the aggregated gradients influence the model parameter values in each layer. Based on the metric, a small number of layers are selected to recycle their previous updates on the server side, Clients can omit these updates when sending their locally accumulated updates to the server. This layer-wis, update recycling method allows only subset of less important layers to lose their update quality while maintaining high-quality updates for all the other layers. Our study shows that, by carefully choosing the update recycling layers, the model aggregation cost can be dramatically reduced while maintaining the model accuracy.
Our study provides critical insights into achieving a practical trade-off off between communication cost reduction and the level of noise introduced by any types of by introducing noise into layers where the update magnitude is small relative to the model parameter magnitude, the adverse impact of the noise can minimized, thus preserving FL. performance. Second, our study empirically demonstrates that the update recycling approach achieves faster loss convergence compared to simply dropping updates for the same layers. Our theoretical analysis also shows that FedLUAR converges to neighborhood of a stationary point when updates are recycled in a sufficiently small number of layers. We designed our FL method to leverage these findings and can be readily applied to various FL applications to improve scalability.
wwuiiiiaaiio-ffiiiennn FL methods. First,
We evaluate the performance of FedLUAR" using representative benchmark datasets: CIFAR-10 [19], CIFAR- 100, FEMNIST [4], and AG News [42] We first compare FedLUAR to several sfate-of-the-arr commumication-effiee
FL methods: Look-back Gradient Multiplier [2], FedPAQ [31], FedPara [12], PruneFL. [13], Deoouoout [8] and FedBAT [23]. We also compare the performance between using and not using the recycling method for advanced FL. optimizers. Finally, we provide extensive ablation study results that further validate the efficacy of our proposed method, including performance comparisons based on the number of layers with recycled updates. These experimental results and our analysis show that FedLUAR provides novel and efficient approach to reducing the communication cost while maintaining the model accuracy in FL environments.
Related Work
Structured Model Compression Several low-rank decomposition-basse FL methods have been proposed, which re- re-parameterize the model weights to reduce either computational or communication costs [12, 28, 35]. These methods modify the model architecture in structured way using various tensor approximation techniques [18]. The re-parameterization methods often increase the number of network layers, resulting in higher implementation complexity and higher computational costs. Moreover, they struggle to maintain model performance when the rank is significantly reduced. Quantization- based FL methods have been actively studied to reduce the number of bits used per parameter [5, 9, 31, 36]. Model pruning methods, such as PruneFL [13], FedMP [14], FedPruning [22], GossipFL [33], and SpaFL. [17], remove a portion of model parameters to reduce both computational and communication costs. These methods are categorized under a sketching approach. Although quantization methods reduce communication overhead, they uniformly degrade the data representation quality of all parameters, overlooking their varying contributions to the training process. The pruning methods potentially harm the model's learning capability since they directly reduce the number of parameters.
Sketched Model Compression
FL Methods - FedLAMA [21] adaptively adjusts model aggre- gation frequency in a layer wise manner. Dynamic model aggregation method proposed in [15] aggregates local models in a decentralized manner. FedKD [37] reduces communication cost by employing knowledge distillation in place of model aggregation. These methods address the high communication cost issue in FL. However, they do not consider the possibility of 'reusing' previous gradients. In this work, we focus on recycling previously computed gradients reduce the com- munication cost. Bandwidth-aware Compression Ratio Scheduling (BCRS) [34] adjusts the top-k compression ratio in network bandwith-aware manner. YOGA [26] adopts a layer-wise aggregation strategy based on layer priority, which shares conceptual similarities with our proposed method. However, assumes a pper-to-pees decentralized FL environment without a central server, which is not applicable to server-based FL scenarios.
Other Communication-Etticii
Gradient- Weight Ratio in Deep Learning A few of the recent works focus on utilizing gradient- weight ratio. Some researchers adjust learning rate based on the ratio to improve the model per-

---

formance [27, 39]. These previous works theoretically demonstrate that the gradient-weigh ratio delivers useful insights that can be utilized adjust the inherent noise scale stochastic gradients. In this study, we propose and employ similar metric in FI. environments: the ratio of accumulated updates the initial model parameters each communication round.
Gradient Dynamics- Depending on the geometry of the parameter space, the gradient may remain consistent over several training iterations [2]. has also been shown that the loss landscape becomes smoother the batch size increases, and thus the stochastic gradients can remain similar for more 'recycling' such stable gradients multiple times. By recycling previous updates, clients avoid update aggregation in some network layers. In the following section, we will discuss how to safely recycle updates in a layer-wise manner.
iterations [16, 20, 25J. We explore the possibility
3 Method
In this section, first introduce layer prioritization metric that can be efficiently calculated during training. Then, we present a communication-efficii FL method that recycles updates for layers with low priority. Finally, provide theoretical guarantee of convergence for the proposed FL method.
3.1 Motivation
Many existing communication- efficient FL methods focus on how
Gradient Weight
Weight Norm
60
0:15
to reduce the communication cost
while keeping the gradient magnitude
40
0.10
close as possible to the original.
30
For instance, update sparsification methods select subset of parameters with small gradients and omit their updates [1, 3]. Layer-wise model aggregation methods selectively aggregate the local updates subset of layers with small gradient norms [2. 21]. These methods com- monly assume that larger gradients indicate greater importance.
0.05
0.00
Layer
Layer
Layer 2
Layer 3
Layer
Layer 1
Layer Layer
(a) FEMNIST (CNN)
0.30
0.20
0.15
0.05
Figure/ shows layer-wise interme- diate data collected from FEMNIST and CIFAR-I0 training. The left-side chart compares the gradient norm and the weight norm while the right-side chart shows the ratio of the gradient norm to the weight norm. The IDs of few layers with small gradient mag- nitudes (left) and ratios (right) are shown at the bottom of the charts. The detailed settings can be found in Appendix.
0.00
Layer 14567
Layer 0,3,45
Layer 20
(b) CIFAR-10 (ResNer20)
Figure 1: The layer-wise gradient and weight ison (left) and the ratio the gradient norm to weight norm (right). It is clearly shown that the layers with the smallest gradients do not always least significantly affect the model parameter values.
compar-
The key message from this empirical study is that layers with small gradients do not necessarily show a small ratio of gradient norm weight norm. The small ratio can be interpreted as less significant impact of the update on changes in parameter value. Even when the gradient is large, if the corresponding parameter is also large, effect on the layer's output will remain minimal. From the perspective of each layer, therefore, the ratio may serve as more critical metric than the magnitude of the gradients alone. This observation motivates network layers by focusing on the ratio of gradient magnitude to weight magnitude rather than solely monitoring gradient magnitude.
explore novel approach to prioritizing
3.2 Layer-wise Update Recycling
Gradient- Gradient-Weight Ratio Analysis We prioritize network layers using the ratio of gradient magnitude to weight magnitude. Given L layers of neural network, the prioritization metric st is defined as

---

Algorithm Layer-wise Update Aggregation with Recycling (LUAR)
1: Input: Af:
the latest local updates, Á, the updates used at the previous round, Ry: the set of recycling layers, 8: the number of recyeling layers
2: Clients send out the local updates: u = [A i. 4A
R₁
3: Server aggregates the updates: = 2i= u
4: r; = [,], Vl R,
5: A = [r, ur]
6: Update the recycling scores: St,l
Eq. (I) > Eq. (2)
7: p' + Calculate pi, vl E [L]
8: R141 Random Choice([L], 6, p¹)
9: Output: Å., R(+1
follows.
A xx,i
$1.1
VI {o,...
(0, - -3
(I)
where A., is the accumulated local updates averaged across all the clients at round t for layer 4 and x is the initial model parameters of layer at round Intuitively, this metric quantifies the relative gradient magnitude based parameter magnitude. If $t,1 is measured large, expect the layer's 's parameters to move fast in the parameter space making sensitive to the update correctness. In contrast, if st, is small, the layer's parameters will not be dramatically changed after each update. We assign low priority to layer its &t is small, and high priority if it is large. In this way, all the T. layers can be prioritized based on how actively the parameters are changed after each round.
This metric can be efficiently measured on the server-side. The Xx,1 is already stored on the server before every communication round. All FL methods aggregate the local updates after every round, and thus A,, is also already ready to be used on the server. Therefore, 8t,7 can be easily measured without any extra communications. This is critical advantage considering the limited network bandwidth in typical FL environments. Layer-wise Stochastic Update Recyeling Method We design novel FL method that recycles the previous updates for a subset of layers. The first step is to calculate a probability distribution of network layers based the prioritization metric shown in (1). The probability of layer to be chosen is computed as follows:
1/80.
Pr. E1- 1/st.1
Vl E {0,
L 1-1.
Each layer has a weight factor so that it less likely chosen if its priority is low. Dividing it by
El 1/st.1 ensures the sum of all weight factors equals 1, allowing p values to be directly used as weight factor of random sampling. Second, our method randomly samples 8 layers using the probability distribution shown in (2). We define those sampled layers at round as R, Finally, the sampled layers are updated using the previous round's updates instead of the latest updates. That is, the clients do not send to the server the local updates for those i layers.
is worth noting that the weighted random sampling-based layer selection prevents the updates for low-priority layers from being recycled excessively. When low-priority layers are not sampled, their updates will be normally aggregated on the server-side and thus their 8t values be updated. We will analyze the impact of this stochastic layer selection scheme on the overall performance of the update recycling method in Section 4.
We formally define the update recycling method as follows.
|Ar.) v7 Rs
(3)
r: | Vi R.
(4)
A = [rc.u]]
(5)
where is the updates for layers not included in R., r, is the recycled updates for ò layers in R. and Å, is the global update composed of u, and rt. Algorithm 1 shows the Layer-wise Update

---

Algorithm 2 Federated Learning with Layer-wise Update Aggregation with Recycling (FedLUAR)
1: Input: a: the number of active clients per round, T: the total number of rounds
2: Ro f- empty set.
3: for t E {0..... , 1 do
4: A Random Choice([V]..a).
5: Server sends out X,, R₁ to the clients vi E [A].
6: Client receives the model: xi,0 N₁.
7:
for j E {1.... 7 do
8:
x = Local (id___ppdeett.... xi --1)
9:
end for
10: Clients calculate the update Ai = xt,r x1,0-
11:
Â R++1 LUAR(A), (A), Å.- R,)
Alg.1
12:
X++1 = X₁ Á,
13:
end for
14:
Output: XT
Aggregation with Recycling (LUAR) method. Note that the number of layers whose update will be recycled, 6, is a user-tunable hyper-parameter. We will further discuss how 5 affects the model accuracy as well as the communication cost in Appendix A.4.
Federated Learning Framework Algorithm 2 shows FedLUAR, a FL framework built upon LUAR (Alg. 1). Before the active clients down- load the initial model parameters X,, the server informs them of the set of layers to be recycled, denoted by R, (line 5). Subsequently, each client performs local training for iterations. The resulting local updates are then aggregated using LUAR (line 1). Finally, the global model
| |
St.1
low
high
update recycle
update
Server
updated using Á. Figure shows a schematic illustration of FedLUAR. Each client transmits updates only for the layers with large $t,1 values. For the remaining layers, the server recycles the previous updates. When 8 0, Å, becomes the same A, effectively reducing the method to vanilla FedAvg. In this paper, we use Fe- dAvg as the base federated optimization algo- rithm for simplicity. However, extending to more advanced FL optimizers straightforward, as LUAR agnostic to the choice optimizer.
Aggregate updetes for layers high su only.
Edge Device
Figure 2: Schematic illustration of FedLUAR.
Potential Limitations FedLUAR requires the server to notify active clients about which layer updates should be omitted when uploading their local updates. This introduces an additional communication cost compared to other FL methods. However, this extra overhead is likely negligible, as the list of recycled layer IDs (6 integers) can be transmitted along with the initial model parameters the beginning of each communication round.
3.3 Theoretical Analysis
We consider non-convex and smooth optimization problems:
m
min F(x): XER3
m Σ F.(x).
where Fx(x) EE,,,, [f(x {.) is the local loss function associated with the local data distribution [f D, of client and m the number of clients. Our analysis based on the following assumptions. Assumption 1. (Lipschitz continuity) There exists constant L > O, such that VF((y) <
C||x- yll, Vx,y dd and E [m].
5

---

Assumption 2. (Unbiased local gradients) The local gradient estimator is unbiased such that Eg.-D. f E.)l (x), Vi [m].
Assumption 3. (Bounded local and global variance) There exist two constants 0t. >> 0 and ag > O. such that the local gradient variance bounded by E f (xx E.) (x)]() - o. Vi [m]. and the global variability bounded by E [I)VF((x) (F()) S o. vi
[m].
Noise Definition We define as a stochastic gradient vector that corresponds to the å layers whose updates will be recycled. Likewise, the corresponding full-batch gradient is defined as V F(().. We quantify the ratio of || V [>xx,)] to FF(,)) which is S 1, as k. By the definition of V Fx)), goes to zero none of the layers recycle their updates. To analyze the impact of the update recycling in Algorithm 1, we also define the quantity of noise The as follows.
T4 P A
(Øl-kk- Ør.5)
(6)
The k in (6) represents the degree update staleness, which increases as the update is recycled in consecutive communication rounds. As shown in Algorithm the upper bound of k and adaptively selects the recycling layers based only on St, values. Therefore, analyze the convergence rate of Algorithm 2 without any assumptions on the le value.
proposed method does not specify
Herein, we analyze the convergence rate of Algorithm (See Appendix for proofs).
Lemma 1. (noise) Under assumption 1 ~ 3, if the learning rate S Zr the accumulated noise is bounded as follows.
|
S 4TT 2 L
8T
T-1
+ SKT
E [
t=0
LTL²
||x Xx 111]
E
m
(=0
where m is the number of clients.
Theorem 2. Under assumption 1 ~ 3, if the learning rate 1 S
1-16x and is < 16 we have
6v30CT
4
E
V |V
(F(xo) - F(xx))
(1
16F)
1=
4T
L1)
4 + 9C2 of.
16x
m
1080T 2
DG-
-16
Remark 1. Lemma I shows that the update recyeling method ensures the noise magnitude bounded regardless of how many times the updates are recycled and how many layers recycle their updates if n. is sufficiently small. This result can serve as foundation that allows users to safely recycle updates in layer-wise manner, thereby reducing communication costs.
Remark 2. Algorithmm 2 converges to a neighborhood of stationary point, as the second and third terms on the right-hand side of (7) do not vanish as T + co. Furthermore, the term of (4 + ) independent of the learning rate 1) and remains non-zero even as 1 - O. Although does not ensure convergence to an exact minimum, this rough guarantee is considered useful in real-world applications [7, 41].
In general, as the degree of non-IIDness increases, the global variance tends to grow due to greater discrepancies among local datasets. As shown in the final term the right-hand hand side of (7), recycling updates in more layers increases the coefficient 10807 "²,,"²² which in turn amplifies the final term. Consequently, the model is expected to converge more slowly. This suggests using smaller learning rate the degree of non- mon-IIDness increases to maintain the convergence rate.
1- 16x

---

3,4 Memory Usage Analysis
In FedAvg, server should receive local mod- els from all active clients. Consequently, the maximum memory footprint a d, where is the number of active clients and d is the model size. By contrast, FedLUAR receives lo- cal models from active clients except 5 layers. Thus, the memory footprint is a (d- k), where k is the size of å layers. Instead, the previ- ous global update should be kept in the mem- ory space for the 5 layers, consuming k space only. Therefore, FedLUAR's memory footprint is (d (d--k) k) + <<<0 d.
Dataset (Model)
Algorithm
5
Fooipriat (MB)
Memory
FodAvg
33.49
CIFAR-I0 (ResNe120)
FedLUAR
10
15.23
FedAvg FedLUAR
4,462.80 8D
CIFAR-I00 (WRN28- 10)
14
2,604.88
FodAvg
806. 11
FEMNIST (CNN)
FedLUAR
2
204.73
FedAvg Fedi.AA
8,294, 18
(DistillBERT)
30
1,825.42.
Table I: Comparison of memory usage observed during training.
To support our analysis, we actually measured the memory footprint of FedAvg and FedLUAR during training. First, the total number of clients is 128 and only randomly selected 32 clients are activated at each communication round. We use MPI to run FL on GPUs. Thus, each process locally train 16 models and then all the locally trained models are aggregated using each process, observed during training under this setting. We can clearly see that FedLUAR uses less memory space than FedAvg. This advantage is directly related to the reduced communication cost, which will be discussed in Section 4.3.
(). Table I shows the memory footprint of
Experiments
Experimental Settings All experiments are conducted on GPU cluster which has 2 NVIDIA A6000 GPUs per machine. We use TensorFlow 2.15.0 for training and MPI for model aggregations. reported.
All experiments were performed at least 3 times, and the average accuracies
Datasets- We evaluate the performance of our proposed method on representative benchmarks: CIFAR-10 (ResNet20 [10]), CIFAR-100 (Wide-ResNee ResNe288 [40]), FEMNIST (CNN), and AG News (DistillBert [32]). When tuning hyper-parameters, we conduct a grid search with a sufficiently small unit size (e.g., 0.1 for learning rate). To generate non-IID datasets, we use label-based Dirichlet distributions with a = 0.1, which indicates highly non-lID conditions.
Data Heterogeneity- For IID datasets, we simulate non-IID settings using Dirichlet distributions. The concentration coefficient a is set to 0.1 for CIFAR-10/100 and 0.5 for AG News.
4.1 Comparative Study
We first present accuracy comparison among SOTA communication-effice. FL methods below.
LBGM (Low-rank Approximation) [2]
PruneFL (Pruning) [13]
FedPAQ (Quantization) [31]
FedDropoutAve (Dropping) [8]
FedPara (Reparameteriiatioon) [12]
FedBAT (Binarization) [23]
Table shows the performance comparison (See Appendix for the detailed settings). The total number
of clients is 128 and randomly chosen 32 clients participate in every communication round. Note
that the FL methods cannot have exactly the same communication cost due differences in their mechanisms. To ensure fair comparisons, we find algorithm-specinf settings that achieve accuracy reasonably close to the bascline (FedAvg) while minimizing communication costs, and then compare the validation accuracy across algorithms.
Overall, FedLUAR achieves accuracy comparable the baseline while significantly reducing commu- nication costs across all four benchmarks. Notably, for FEMNIST and AG News, it matches FedAvg's accuracy with less than 20% of the communication cost. Our method also outperforms all other SOTA methods. While FedPAQ and FedBAT reduce communication cost, they suffer from noticeable accuracy drops. Regardless of the dataset, FedLUAR consistently delivers the highest accuracy among communication-effcieett FL methods. These results demonstrate that LUAR effectively finds less critical layers and recycles their updates, minimizing the communication cost without sacrificing performance.
7

---

CIFAR-ID (ResNer20)
CIFAR-100 100
FEMNNIST
AG News (DisillBERT)
Method
(WRN-28)
(CNN)
Accuracy
Comm
Accuracy
Comm
Accuracy
Comm
Accuracy
Comm
FodAvg
61.27
0.7%
59 0.8%
71.01 + 0.4%
82.66 6
+ 0.2%
1.00
LBGM
54.87 + 0.5%
0.65
57 13
+ 0.2%
0.87
69.83
+ 1.0%
0.71
77.96 1 0.1%
0.23
FedPAQ
36.15
71.54
82.72 1
FedPura
55.16 0.1%
0.51
46.14 + 0.1%
0.61
67.69 - 0.1%
0.22
75.22
+ 0.1%
0.69
PrumeFL.
56.76 1: 0.1%
0.51
59
0.1%
0.69
69.42
1: 0.4%
0.19
77.25
1 0.1%
0.22
FDA
56.54
0.3%%
0.50
51.25
0.1%
0.60
70.61
0.1%
0.25
64.94
1: 0.1%
0.50
FedBAT
39.56 0.1%
0.03
47.24 + 0.1%
0.03
68. = 0.1%
0.03
76.38
+ 0.1%
0.57
0.6%
Table 2: Classification performance comparison. Comm denotes the communication cost relative to FedAvg.
FedAvg
FedLUAR
200
CIFAR-10 200 rounds
FEMNIST 200 rounds
Number of
Agg Aggrigatons
1.50
150
1000
100
10111211441111171119022210
CIFAR-100 300 rounds
AG News 100 rounds
300
100
Aggnegatonss
Nurber
200
50
100
Layer ID
Layer
Figure 3: Number of model aggregations per layer. FedLUAR significantly reduces aggregation frequency across all benchmarks. The gap from FodAvg indicates how often updates were recycled (i.c., communications skipped).
4.2 Harmonization with Other FL Methods
The proposed FL method does not have any dependencies the local training algorithm. To demon- strate this, we apply LUAR to several advanced FL methods, including FedProx [24], MOON [29], FedOpt [30], FedMut [1]], FedACG [6], and PruneFL. [13], and analyze its effect on model accuracy. Table 3 shows CIFAR-I0 and FEMNIST accuracy comparisons (See Appendix for details). LUAR maintains the validation accuracy while significantly reducing the communication cost across all three benchmarks. Comm denotes communication cost relative to full model averaging, For instance, FedPAQ reduces communication 50% of FedAvg, while LUAR further reduces 22% FedPAQ. just 11% of FedAvg, without compromising accuracy. These results show that LUAR can effectively complement advanced FL algorithms and readily applicable real- real-world scenarios.
4.3 Communication Cost Analysis
FedLUAR enables clients to skip uploading updates for less important layers, thereby reducing com- munication costs. Figure shows the number of communications for each layer. The communication count charts indicate that FedLUAR requires significantly fewer communications than vanilla FedAvg. while achieving comparable model accuracy. An interesting observation is that, in FEMNIST and AG News, the layer with the largest number of parameters tends to be recycled most frequently, resulting substantial reduction total communication cost. However, this trend is not observed in CIFAR-10 10 and CIFAR-10 100 benchmarks. Thus, we conclude that the proposed method is independent of layer size and specific model architectures, as adaptively identifies the least significant layers regardless of the model design.
4.4 Ablation Study
To further validate the effectiveness of the proposed metric shown in (1), we conduct an ablation study as follows. Fixing the number of layers to recycle updates, ö, we measure model accuracy using different layer selection metrics. By comparing these accuracies, we can determine which metric most effective in identifying the least critical layers in terms of their contribution to the global

---

Periodic Averaging
LUAR (Proposed)
Comm 6
Periodic Averaging
LUAR (Proposed)
Comm 8
FedProx
61.74 1 0.1%
61.20
0.1%
0.54
FedProx
71.94 1 0.1%
73.45 1 0.1%
0.09
FedPMQ
57.42
0.2%
57.40
0.2%
FedPAQ
71.54
0.1%
71.15 115:::0.10 0.1%
FedOpe
62.42 0.1%
62.28
0.2%
0.50
FedOpt
72.34
0.1%
71.91
0.22
MOON
62.33 1. 1.2%
61.65
0.1%
0.51
10
MOON
71.55
0.1%
71.63 0063311000 0.1%
0.24
2
FedMut
61.27 1: 0.1%
60.42
0.1%
0.56
FedMut
71.91 +: 0.1%
72.31 +: 0.1%
0.26
FedACG
65.02 +: 1.1%
64.28
0. 1%
0.55
FedACG
72.16 + 0.1%
71.94 + 0.1%
0.21
PruneFL
56.76 1: 0.1%
55.43
0.1%
0.49
PrumeFl.
69.42 1: 0.1%
69.11
0.1%
0.11
(a) CIFAR-10 (ResNer20)
(h) FEMNIST (CNN)
Table 3: CIFAR-10 and FEMNIST performance comparison between before and after applying LUAR. LUAR is applied to half of the model layers for both datasets, using ResNer20 for CIFAR-I0 and CNN for FEMNIST. The Comm column shows the ratio of LUAR's 's cost to the FodAvg's 's cost.
CIFAR-10
FEMNIST Acc. (%) Comm.
AG News
Layer Selection Scheme
Acc. (%)
Comm.
Acc. (5) Comm.
Random
53.94%
0.48
71.10%
0.51
80.27%
0.23
Top (ingut-side)
56.03%
0.73
N/A
79.71%
0.21
Bottom (output-side)
45.02%
0,24
69.54%
0.13
81.14%
0.45
Gradient norm
55.88%
0,55
70.91%
0.70
75.06%
0.22
Deterministic recycling
48.47%
69.08%
0.02
80.22%
LUAR(Proposed))
60.15%
0.47
73.17%
0.18
82.80%
0.17
Table 4: Performance comparison with different layer selection schemes. For CIFAR-10 and FEMNIST, half the layers were reused; for AG News, 30 layers. Comm. denotes communication cost normalized to FedAAg. Selecting top layers in FEMNIST leads to early-stage divergence.
model training. In particular, we compare the classification performance between the most popular gradient-based layer selection and our proposed LUAR. Table 4 shows the performance comparisons.
This ablation study provides several key insights. First, LUAR outperforms uniform random sampling, demonstrating that our proposed metric (1) effectively identifies less critical layers. Second, even with the same metric, deterministic selection strategy yields lower accuracy. Persistently recycling updates for layers with low 8t,1 values can cause them be too much outdated, introducing excessive noise that degrades model performance. Third, LUAR consistently outperforms the gradient norm- based method, supporting our earlier observation (Fig. 1) that gradient magnitude alone insufficient to assess update importance. We thus conclude that the gradient-to- to-wegght ratio best captures update quality, achieving the highest accuracy while significantly reducing communication costs.
Additionally, we compare the classification performance of update dropping and recy- cling schemes. Many existing communication- efficient FL methods merely drop a subset of up- dates. Table 5 presents the performance compar- isons. Here, Dropping refers to the case where the 5 least critical layers are selected using LUAR and their updates are dropped instead of being recycled. As expected, Dropping achieves the same communication cost reduction as Recycling, but its accuracy is significantly lower than that of Recycling. This ablation study clearly demonstrates the superior performance of our proposed update recycling scheme.
Dataset
Dropping
Recycling
Comm, Cust
5
CIFAR-10
46.89
0.1%
50.07 1.6%
0.30
16
FEMNIST
64.69
0.2%
73.17 1.1%
0.18
AG News
77.05 + 0.1%
82.80 + 0.1%
0.17
30
Table Benchmark performance comparison between update dropping and update recycling schemes.
How much does it accelerate? Figure 4 shows the learning curves for CIFAR-10 and AG News. The x-axis represents the communication cost relative to FedAvg. To highlight the difference clearly. we selectively present comparisons among four methods only. The comparison clearly shows that FedLUAR achieves similar accuracy to FedAvg much faster than other SOTA communication-efficine FL methods. Since our method incurs little to no additional computational cost, the same performance gain can be expected in terms of the end-to-end training time in realistic FL environments. In our empirical study, we observed the same performance gains across many different FL benchmarks. See Appendix for more curve charts and the detailed experimental settings.

---

Fedlue
FedPAQ
FedPAQ
Fedlug
Fedlvgg LBG
FedPAQ
FedLUAR LBGM
FedAvg
Training Loss (Cross Ertrapy
FedLUAR
LDGN
Entrpy)
FedPAQ
Valttetoon Accurncyy 10-31
Accunayy
06
0.8
0.9
0.9
Loos
0.6
C-4
0.6
0.6
0.3
0.5 Communication Cost (Relative to FedAug)
0.5
1.0
Communication Cost
Communication Cost [Relative to FosAug)
Communication Cost (Relative to Fedkvg)
(Reiative to Fodkug!
CIFAR-10
(b) AG News
Figure 4: Learning curves for CIFAR-I0 (ResNet20) and AG News (DistillBERT), with communication cost (x-axis) normalized to FedAvg. Four representative methods are shown for clarity.
Conclusion
In this paper, we demonstrated that selectively recycling updates in specific layers can reduce communication costs in FL while preserving model accuracy. In particular, proved that the wradient-to-weigg least significant layers. This layer-wise partial model aggregation scheme is expected to facilitate the development of efficient FL applications and promote the partial model training paradigm across various deep learning fields. We consider developing Model fine-tuning method based on the update recycling scheme as promising direction for future work. A discussion of the broader impact of this work is provided in Appendix A.1.
study empirically magnitude ratio can serve as a practical metric for identifying
communication-effiieee
Large Language
Acknowledgments
This work was partly supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS-2022-001555) 15, Artificial Intelligence Convergence Innovation Human Resources Development (Inha University)) and National Research Foundation of Korea(NRI) grant funded by the Korea governmen RS-2024- 00452914).
References
[1] Dan Alistarh, Torsten Hoefler, Mikael Johansson, Sarit Khirirat, Nikola Konstantinov, and Cédric Renggli. The convergence of sparsified gradient methods. In Conference on neural information processing systems. NeurIPS, 2018.
[2] Sheikh Shams Azam, Seyyedali Hosseinalipour, Qiang Qiu, and Christopher Brinton. Recycling model updates in federated learning: Are gradient subspaces low-rank? In International Conference Learning Representations. ICLR, 2022.
[3] Sameer Bibikar, Haris Vikalo, Zhangyang Wang, and Xiaohan Chen. Federated dynamic sparse training: Computing less, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[4] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint 1812.0001 2018.
[5] Huancheng Chen and Haris Vikalo. Mixed-precistie quantization for federated learning resource-cnnstrained heterogeneous devices. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6138-6148, 2024.
[6] Bohyung Han Geeho Kim, Jinkyu Kim. Communication-efficii federated learning with accelerated client gradient. In Computer Vision and Pattern Recognition. CVPR, 2024.
[ Gabriel Goh. Why momentum really works. Distill, 2(4):e6, 2017.
10

---

[8] Gozde N Gunesli, Mohsin Bilal, Shan E Ahmed Raza, and Nasir M Rajpoot. Feddropoutavg: Generalizable federated learning for histopathology image classification. arXiv preprint arXiv:2111. 13..3300,, 2021.
[9] Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference on Artificial Intelligence and Statistics, pages 3358-2358. PMLR, 2021.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[11] Ming Hu, Yue Cao, Anran Li, Zhiming Li, Chengwei Liu, Tianlin Li, Mingsong Chen, and Yang Liu. Fedmut: Generalized federated learning via stochastic mutation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
[12) Nam Hyeon-Won Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficin federated learning. International Conference on Learning Representations. ICLR, 2022.
[13] Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 34(12): 10374-10088, 2022.
[14] Zhida Jiang, Yang Xu, Hongli Xu, Zhiyuan Wang, Jianchun Liu, Qian Chen, and Chunming Qiao. Computation and communication efficient federated learning with adaptive model pruning. IEEE Transactions on Mobile Computing, 23(3) 2303)22003-002,, 2023.
[15] Michael Kamp, Linara Adilova, Joachim Sicking, Fabian Hüger, Peter Schlicht, Tim Wirtz, and Stefan Wrobel. Efficient decentralized deep learning by dynamic model averaging. In Joint European conference on machine learning and knowledge discovery databases, pages 393-409, Springer, 2018
[16] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representutions. ICLR, 2017.
[17] Minsu Kim, Walid Saad, Merouane Debbah, and Choong S Hong. Spafl: Communication- efficient federated learning with sparse models and low computational overhead. Advances in Neural Information Processing Systems, 37:86500-86522 2024.
[18] Jakub Konečný, H Brendan McMahan, Felix Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv: 1610.05492, 2016.
[19] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[20] Sunwoo Lee, Chaoyang He, and Salman Avestimehr. Achieving small-batch accuracy with large-batch scalability via hessian-aware learning rate adjustment, Neural Networks, 158:1-14, 2023.
[21] Sunwoo Lee, Tuo Zhang, and A Salman Avestimehr. Layer-wise adaptive model aggregation for scalable federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 8491-8499, 2023.
[22] Andy Li, Milan Markovie, Peter Edwards, and Georgios Leontidis. Model pruning enables localized and efficient federated learning for yield forecasting and data sharing. Expert Systems with Applications, 242: 122847, 2024.
[23] Shiwei Li, Wenchao Xu, Haozhao Wang, Xing Tang, Yining Qi, Shijie Xu, Weihong Luo, Yuhua Li, Xiuqiang He, and Ruixuan Li. Fedbat: communication-effciene federated learning via learnable binarization. ICML'24. JMLR.org, 2024.
11

---

[24] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. In Annual Conferences on Machine Learning and System. MI.Sys, 2020.
[25] Tao Lin, Lingjing Kong, Sebastian Stich, and Martin Jaggi. Extrapolation for large-batch training in deep learning. In International Conference on Machine Learning, pages 6094-6104. PMLR, 2020.
[26] Jun Liu, Jianchun Liu, Hongli Xu, Yunming Liao, Zhiyuan Wang, and Qianpiao Ma. Yoga: Adaptive layer-wise model aggregation for decentralized federated learning. IEEE/ACM Trans- actions on Networking,
2023.
[27] Christian HX Sehmeti--Göpe and Michael Wand. On the weight dynamics of deep normalized networks. In International Conference on Machine Learning. ICML, 2024.
[28] Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavský, Valeriy Glukhov, Ivan Oseledets. and Andrzej Cichocki. Stable low-rank tensor decomposition for compression of convolutional neural network. In European Conference on Computer Vision, pages 522-539. Springer, 2020.
[29] Dawn Song Qinbin Li, Bingsheng He. Model-contrastive federated learning. In Computer Vision and Pattern Recognition. CVPR, 2021.
[30] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations. ICLR, 2021.
[31] Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communicationn-eficienn federated learning method with periodic averaging and quantization. In International conference on artificial intelligence and statistics, pages 2021- 2031. PMLR, 2020.
[32] Victor Sanh, L Debut, J Chaumond, and T Wolf. Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arxiv 2019. arXiv preprint arXiv: 108, 2019.
[33] Zhenheng Tang, Shaohuai Shi, Bo Li, and Xiaowen Chu. Gossipfl: A decentralized federated learning framework with sparsified and adaptive communication. IEEE Transactions on Parallel and Distributed Systems, 34(3):909-92 2022.
[34] Zichen Tang, Junlin Huang, Rudan Yan, Yuxin Wang. Zhenheng Tang. Shaohuai Shi, Amelie Chi Zhou, and Xiaowen Chu. Bandwidth-aware and weighte- compression for communication--ffiiiees federated learning. In Proceedings of the 53rd International Conference on Parallel Processing, pages 866-875, 2024.
[35] Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Practical low-rank communication compression in decentralized deep learning. In Conference on neural information processing systems. NeurIPS, 2020.
[36] Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Terngrad: Ternary gradients to reduce communication in distributed deep learning. Advances in neural information processing systems, 30, 2017.
[37] Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication- efficient federated learning via knowledge distillation. Nature communications, 13(1):2032, 2022.
[38] Haibo Yang. Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker partici- pation in non-iid federated learning. In International Conference on Learning Representations. ICLR, 2021.
[39] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep trainingg bert 76 minutes. In International Conference on Learning Representations. ICLR, 2022.
12

---

[40] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
[41] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
[42] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly describe our main contributions, including the proposed layer-wise update recycling method and its communication efficiency benefits, which consistent with the results shown in Section 4 and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We give the limitations in Section and Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: The theoretical results are fully supported with all assumptions explicitly stated in Section 3.3. Complete and correct proofs are provided in Appendix A.2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We report all settings and details necessary to reproduce the main experimental results in Section 4 and Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
com/ wwwsswwblteeteee FedLUAR, and the repository link is also provided at the bottom of page of the paper for reference.
Justification:
The code is publicly available at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
Bagllii/guides//odSSuubbii sooolliisssooooorcccococoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nipscc... wwtpccppcpgguupiieeecceeecooceee gww./// CCoeee/mmddSSbaaaaa
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We report all settings and details of experiments in Section 4 and Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do report error bars, as each experiment was repeated times and the mean result is presented.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4 and Appendix A.3 provide sufficient information on the computer resources required to reproduce the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http://eeurppp.eee http://errpp...ccppp/e//ttttttte hctpppcbllcccchhheeeteeeeeeeeeee
Answer: [Yes]
Justification:
The research fully adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We report broader impacts in Appendix A.I.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not involve the release of any pretrained models, data, or systems that carry high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We will make the code public upon acceptance of the paper, and the models and datasets used are publicly available, so no licenses are required.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pwperswithcee thcode. com/ conddataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not introduce new assets.
Guidelines
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any human subjects and therefore does not require IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
This paper does not use LLMs in any important, original, or non-standard way part of the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: pettps:/neeteee //neruurpp.... ec//oonfeeeccee/225////////////e for what should or should not be described.
20

---

A
Appendix
The appendix is structured as follows:
Section A.1 briefly announce broader impacts of this study.
Section A.2 provides problem definition, assumptions, and proofs of our theoretical analysis.
Section A.3 presents detailed experimental settings.
Section A.4 presents additional experimental results and analyses.
A.1
Broader Impacts
We do not anticipate any negative societal impact from our research. The proposed method accelerates neural network training in the context of Federated Learning. Faster training implies that target model accuracy can be achieved with fewer training iterations. As a result, it contributes to lower power consumption and a reduced carbon footprint.
A.2 Theoretical Analysis
We consider non-convex and smooth optimization problems follows.
min F(z) =
Fi(2),
(8)
EE
1=1
where F1(x) = EE [f(x,, (:)] is the local loss function associated with the local data distribution D; of client i and m the number of clients.
Our analysis based on the following assumptions.
Assumption 1. (Lipschitz continuity) There exists constant C > 0, such that (Z) <
C| - y/I, Vz, Rd andi E [m].
Assumption 2. (Unbiased local gradients) The local gradient estimator is unbiased such that [Vf & (F(zz) [m]]
Assumption 3. (Bounded local and global variance) There exist two constants d >> 0 and aG > O. such that the local gradient variance bounded by [ fxx {,) (x) < o Vi [m], and
the global variability bounded by E (1)) VF
Vi [m].
Herein, we analyze the convergence properties of FedAvg as follows. First, the following Lemma is slightly refined version of Lemma 3 in [30]. This Lemma is also used as Lemma 2 in [38].
Lemma (model discrepancy) For any step-size satisfying n S 2//1="' we have the following result:
m
x=1131 < 5n + =00m + 30rr"
21

---

Proof. For any client i E [m], t E [T - 11, and A E [r], we have
Bllxx xx|1²] B[I
X,- -
=Bi|xi,,-1 - x: - 17 (9i,
xx + VV VV(x)) VF(x)) [12]
= =Blll (BL,0-1 - (x 119
+ Kn (o),k-1 V V Fx -)) - -VP() + Vx(x)))
+ |E||xi, - X ---- 17 ( (VFi (xi, k-
+ VF(x) FF [|]
= Ellly (0i - (P(((((A-1) [|2]
(9)
+ +B[l X, - 17
+ VF(x))) [1]]
k-1
s Snp0î + |[||x| - Xx- - 17 (VF((x), VF(x)) VF(x)) VF(xx) [P]
s znooî + 11 27 B[|x - x=|1
(10)
+ 2 (/llVF((x
(xx) + VF((x))- - V
+ VF(x)
S snpoi + 1 + 27
B[|| A- xx |]
+ + (B[ll V.; [Fi(x(
VX )|3] + (IIV))(()) -
s + 1 + 27 -1
1
E[l xx|
+ +6r + 6Tr/² T,
1,211122,--11 - + GT7 6r)PB111V((x))1
= of + or +
1 1 2r - + 66T12 122)
E Bi|x - xx|1 + 6
where (9) is because Eggi--1l = vFi(x ). The (10) is based on the fact that
la + bl V (1 |all + (1 +1++0))) aa]bb
for any a > 0.
Next, if 17 5 2/3L7'
1
the above bound can be simplified as follows.
E 0||x|/1 xx11
S 00002 677 +
1+
1
+ 677/2 L3) 18 10]1x|,--1- (r B[IV/
27 21
0/ 6 +
1+ T -+--11 E -
1
Then, by unrolling the recursion until k- - 1 goes to O, we have
E||xi, - x.112 < Σ 1+ (++__n)
)
1
(v²00" + 6TIi0O 6rT 6-)
3=0
S (r 1)((1 +
--
1 1
)(- 1) (170) + 6-1200 + 6
< (T - 1)((1 +
1) 1)(11 -
1
+ 6r1200 + G
< 5 07 + 307 00 + 30.32 {
(11)
where (11) is because that the maximum value of (r- 1)((1 + + -1) is 19 because the right-hand side of (11) is independent of m, we have
when r = 3. Finally,
4
m
Σ
Bllxi
xx x1 < of + 30-3', 3 + +07
m
22

---

Based on the proposed update recycling method, the noise is defined as follows.
Gii -
n4
m
where 9 indicates the gradient vector that has non-zero gradients only at the layers where their updates will be recycled.
Lemma 4. (noise) Under assumption ~ if the learning rate n S Dr the accumulated noise is bounded as follows.
Σ
|
12]
< <4T>207 + 8T7200 + SKT
2
E
VF
1=0
8TL2
T-1
Σ
E
|
-
x111
(12)
m
==0
where K. is the ratio of (x)](x))] to [V
Proof.
m T-1
| 2
E
m
1=0
m
m
< 2E
2E
gl.
m
27
m
2T
m
EE
| 11
+
E
|102,111
171
m
j=0
27
(P(((()))))) 11)
E
2T
|1ots - vR((()) + v(i((())))))
E
27
m
| V ( 1|| E -
E
(13)
2T
E
[1loss
(
> 11| L [1
(x ||'
22220
E
+ +27207
23

---

where (13) is based on the fact that [[||x||] = Bi|x= - B|x]]| + ||?
Then, the right-hand side
can be further bounded as follows.
E
[mml]]]
S <4000%
2T ΣΣ
E
||]
2T ΣΣΣ
E
(
m
2T
= =4720% +
E
1=1 f=0
+ m ==
27 m
ΣΣ
E
-V(())
I <4
4T
E
[((((--))) - v -
4T
E
V [ [1]]
m
[ - vR(X)) -
4T
m
|
|||
E
E
m
==
<4T000 +
4TL2 m 7-1
ΣΣ 1=
E
47
ΣΣ E
((( 17]
m
m
4
ΣΣ 1=1
[1*t)~~
47
ΣΣ
||
E
m
==
= -f + 4rL²
ΣΣ
m m-
-
[1]]
+
4TL2
ΣΣ
(1xt
B
||ا|_ - xxol"]
m
m
1=1 1=
1=1
m
|| [(x (((x) v v m
E
[| - vf(x)) + +V
E
jm0
<4700 +
4r
ΣΣ
T-1
|| - -x--1 1
+ 2r m
ΣΣ
E
||x
- x
m
m
i= コ=
j=0
Σ
E
4 (xx viXx
47
11
m
0=
+ 47
|| - (F(x
+ 47
||<0
E
E
1=
|F ||²
V ||]]
= =f + +87 +
+ +
(14)
+ 4r m r-1
4 m
T-1 E [|xi
ΣΣ
E
-
- x
-x,xxx11
x,, [|]
m
m
==
24

---

where (14) follows |VF(-)? & V.(.)). By summing up B|[|n |2] across T rounds, we have
m |2] I of + 8T72080 + 47
T-1
Σ E
V (x x))
|
2
+ 2
B
V (xx)) 11
T-1 m --1 4r
4T
T
T-1
m
ΣΣΣ m t= im1 j=0
E
| k-j - xx-
+
| - xx11
m
0
T-1
L
T - m
I <4Tr'0), + 8T 00 + 87
E
E |||2 xxel
m
1=
I <4TT²07 0z. + 8 o + SKT
E
8TL² m
ΣΣΣ
E
[| xx1("]
where x is the ratio of the recycled update norm to the full update norm. Because all the gradients at the layers not recycled are zeroed out, the ratio lies between 0 and I; 0 < K < 1.
Lemma 5. (framework) Under assumption 1 ~ 3. if the learning rate 1) S
1 ZF. we have
2T
Lu
167
[x)(( s (1 - 16
(F(x)) - Fxx))) + 1
+ 4
o +
o
16x
m
1 16x
+ 18C²
T-1
[|x - x xx.1%]
E
16x)MT
where is the ratio of the norm of the recycling layers' gradients: |/V (xx))1 that of the full model gradients: [x,)2
Proof. We first define the following notations for convenience.
T-1
A gi. =
/=0
1=0
A = m
A + ne.
where {) is random sample drawn from the local dataset at the local step j and n4 is a noise caused by the update recycling.
Based on Assumption 1, taking expectation of F ((x+11) we have:
E[F(xe+1] ]) < F(x:) + (VF(()) ((x,x,,,)((()H1 x.l) + C
- B[||x++1 - xx
2
= F(x)) + (VF E[- + 1|
- VF + C 1111A,,11
= = + E +,)
2
= - (FF (E + - 1111
2
To
(15)
Now, let bound Ti and T3 separately follows.
25

---

Bounding Ti.
Ti (VF(x)), E A + 97 VTFFF(())))
1X,),,
1
Ihg + + + +
== コ=
(x,),,
"1
771
(x.)),
" Σ
ΣΣ (x((x)) - myme m
m
==1
V (x,),,
" Σ
- - mm
m
1=
m mVT
VITVF(x))
(VF(()))) (V VV((X))
VF T4
1TT [VF(()))
+
E
(V.())))))) vF((K)) + VT 1
2
2
mVT 1=
1 E
( + v
(16)
2
my/
VT
==
<
TTT 2
|VF(x))
2
+E
(VF(( -VE(S>))
7 +
n2
mVT 1=
m --1
" 2T
E
(x(())))) + T2g
1=0
T-1
$ 1/T
||
2
+
" ΣΣ
/) - VF((
+ "E ||mel
2
m
7
1=1 0=0
E
V F (i(()))) + n
(17)
2r
< 1IT
|VF(())] 2
E [|x2,, Xx 12]
+ " [lneel]
2
m
i=1
(F(())))) + m |
m
7)
Σ
(18)
2m²7
i=1
1=
where (16) holds because (x,1) ||| + || || 2 for x Vxx.)((x) and
MyF En (V (x.)).)). Also, (17) is based on the convexity of &g norm and Jensen's 's inequality.
26

---

Bounding T2.
1
T- = |n 11 E
Ie. + n4
i=1 1=0
1
B/B
Fig + n1 + VF((
1= 1=0
1
$ 211'E
( VF,(x)
+ 2
(F((2
+ =
m
1=0
1=
27
2n/ m
m
Σ E
Σ 1=0
9h.
+ 2
VF((xt,))+ - n
(19)
m 1=1
1=
2T L
2n m²
m
1 + mm)
+
E
(F((X)))
(20)
m
where (19) is due to the fact that E||[x] + x2 + ...
x,1121 E[||x, 1|+ +|>1/|1 + + ||x,,| 13 if
X,S are independent of each other with zero mean and Eigh, = (
Now, by plugging in (18) and (20) into (15), we have
Lap²
nc²
m
B[F(x++1) & F(x:)
17T E
[IF
oi
E [| xell
2
m
m
j=
( (npp
m
+
m²
2m2T
7
(F(((()))
+ T
n2
+ L [1mell?]] 12]
j=0
< F(x))
m E
|VF
+
LAPT
+
" E
||nel??
2
m
22
Σ
E
[|xx
xx x11
(21)
1=1
1=
where (21) holds if S J.
Summing up (21) across T communication rounds, we have
Σ -
T
Σ 0
E [IV
LTZTT 2
E [neel]]]
of.
2
m
t=
+ C
T Σ
ΣΣΣBI E
m
1 12]
m
1=
i=1
1=
27

---

Based on Lemma 1, the right-han hand side can be re-written as follows.
T-1
T-1
+ CTATT
Σ 1=0
(F(x)+1) - -F(x)) - (8TT7)
1
|VF())1
02 + 4nTTo] + S
2
t=0
8nL²
n22
B [|x2 x x011
+
m
m
t=
i=
T-1
(-M m
(8TT
|
+
+
+4 4 or &
2
t=0
CnC² T-1
| -x
xe11
E
m
1=0
After rearranging the telescoping sum, we finally have
T-1
Σ E [IV
$ (1 16K)
(F(xo) F(x)))
+
2T
Li
+
o7. +
1 16k o
16T
1
16k
m
18C²
T-1 m ΣΣΣ ==
||
xx 11
E
(1 - 16
Now, we can derive the following Theorem based on Lemma 3, 5, and as follows.
Theorem 6. Under assumption 1~3, if the learning rate 7) S 1-16ks and K < 1
16. we have
Bv30CT
Σ E [ < (1 16
4 (F(xo) F(XT)) + 1 16m
4T
Cn
+ 4 + 9C2 +4
o.. +
1080
m
1 1-16k
0=
Proof. Based on Lemma 3 and 5, we have
E [VF(Xx))|]
S (1- 16x
2
(F(xo) F(xx))
2T
Ln
+4 4 0
2
16T
1 16x
m
16k
18C2
Σ
T-1
ΣΣΣE 1=0 i=1 j=0
|| - xx [1]]
(1 16
E [VF(X))]
2
(F(xo) [2(x)) + 1- 16
2T
Li
+ 4
16T
(1 16x))IT
m
16x
18C²
T-17-1
ΣΣ
T5n + +30m2 + 30mp 2 |F(x))]1
(1 16H)T
1=0 1=0
2
(F(xo) F(XT))
2T
Cn
+ 4 + 2 +4+922
07
540 2
00
(1 16x5)ITT
16x
m
16x
5/,
T-1
||FF(x))1*
1-16k
0=
2
(F(xo) F(xx)) + 1- 16x
2T
( Ln
+ + 9C2 +4+92²
07. +
540T
(1
16k)
m
1 1- 16x
T
E
(22)
2 t=
4
4T
Ln
10
(F(xo) - F(XT)) +
+ 4 4 7L3
of +
11 16k))T
1 16k
m
1 -66 .
28

---

where (22) holds 1f S
1-16x 6/30CT
A.3 Experimental Settings in details
All our experiments are conducted on a GPU cluster that contains NVIDIA A6000 GPUs per machine. We use TensorFlow 2.15.0 for training and MPI for model aggregations. All individual experiments are performed at least three times, and the average accuracies reported. The total number of clients is 128 and randomly chosen 32 clients participate in every communication round. We use mini-batch SGD with momentum (0.9) as the local optimizer. Table 6 shows the hypae-parameter settings for all experiments, used not only for our method but also for other SOTA methods.
Implementation Details
Hyperparameters
CIFAR- (ResNet20)
CIFAR-100 (WRN-28)
FEMNIST (CNN)
(DistilBERI)
(local steps) batch size
20
20
20
20
32
32
128
min learning rate
0.2
0.1
0.01
1e-5
learning
0.2
1e-5
total epoch
200
300
200
100
weight docay
1e-1
1e-5
le-1
1e-4
decay epoch
100. 150
150, 200
100, 150
60, 80
Table 6: Hyperparameter Settings for all experiments
naturally non-III,, we generate artificial data distributions using Dirichlet's distributions. To evaluate the performance our proposed method under realistic FL environments, the concentration coefficient a is configured as 0.1 for CIFAR-10 10, CIFAR- 100, and FEMNIST, and as 0.5 for AG News. Note that these small concentration coefficient values represent highly heterogeneous distributions of local samples across clients as well as imbalance in the number of samples across labels.
Artificial Data Heterogeneity- For benchmark datasets that
Algorithm-Speeific Hyperparameter Selection - Here, we summarize the hyper-pane arper-parameter settings used to reproduce other SOTA methods, primarily following the configurations outlined in the original papers. We find algorithm-specine accuracy reasonably close to the baseline algorithm (FedAvg) while minimizing communication costs, and then measure the validation accuracy shown in Section 4.1. Table 7 and Table 8 show the hhper-yaramann parameter settings for SOTA methods and experiments shown in Table and 3, respectively, When running FedPara, both convolution layers and fully connected (FC) layers re-parameterized using their proposed method. All hyperparameters shown Table 7 and 8 are defined in the original papers.
hyper-parameters using a grid search that achieve
Algocichm
Hyperparamnters
(ResNer20)
(WRN:28) (CNN)
(DisillBERT)
FedPAQ
(quantization level)
16
FedPamn
parmmeters
0.5
0.6
LRGM
6 (threshod))
0,95
D.98
0.96
0.6
PruncFL
recmiiiguraionn
FedDropoutArg
for (federated dropoct rate)
0.5
0.4
0.75
0.5
FedBAT
p. (cocllicient, warm-up ralio)
0.5
0.5
6,
6,
Table 7: pyrraarameterr Settings for Comparative Study 4.1 of communication--fficieen FL methods
Algorithm
Hyperparameters
CIFAR- 10 (ResNer20)
FEMNIST (CNN)
FedProx
a (proximal term coefficient)
0.001
0.001
FodPAQ
(quantization level)
FedOpt
9 (server learning rate)
MOON
a (control the weight of model-contrasivee loss), (lemperature parameter)
1,5.5
FedMut
(distance scaling factor), @ (dynamic mulation lactor)
0.5.
0.5.
FedAOG
A Iglohal momentum scaling factor), 3 (penalty coefficient)
0.7. 0.01
0.7, 0.01
Table 8: Hyperparameter Settings for Harmonization with Other methods 4.2
29

---

A.4 Extra Results
Sensitivity on 8- We performed a grid search for each dataset to find the best setting which yields reasonable accuracy together with the maximum communication cost reduction. Table 9-12 show the four benchmarks' accuracy and communication costs corresponding to various d settings.
How much could be recycled safely? We also investigate the impact of 8 on model accuracy and communication costs. We divide the number of model aggregations at each layer by the total number of communication rounds to calculate the layer-wise communication cost. Then, we sum up the calculated layer-wise costs to get the total communication cost. Intuitively, the larger the the lower the communication cost. However, the model accuracy is expected to drop more layers have their updates recycled.
Table 9 and 10 show CIFAR-10 and CIFAR-100 experimental results for various ó settings. One key observation that the accuracy almost not degraded when LUAR is applied with 6 I 12 for both datasets. This means that many network layers have quite stable gradient dynamics, and thus their updates can be safely recycled. In addition, the accuracy is hardly reduced until the communication cost is reduced by almost 50%. This is significant benefit especially in FL. environments where the network bandwidth is extremely limited.
Results under varying degrees of data heterogeneity The degree of non-IIDness strongly affects the training efficiency of Federated Learning methods. We conducted additional experiments to demonstrate that FedLUAR robust to various degrees of non-IIDness. Table 13 and Table 14 show CIFAR-10 10 and AG News experimental results for various Dirchlet concentration factor a settings. In both benchmarks, FedLUAR achieves comparable accuracy to FedAvg regardless of while considerably reducing the communication cost. Therefore, we conclude that FedLUAR is robust to the degree of non- -IDnesssss.
Performance under different numbers of active clients - We conducted additional ablation study using different numbers of active clients. Table 15 and Table 16 show that, regardless of the total number of clients, FedLUAR achieves accuracy comparable to FedAvg while significantly reducing the communication cost. This ablation study demonstrates the superior scalability of FedLUAR. Learning Curves Figure 5 and 6 show the learning curve comparisons for CIFAR-100 and FEMNIST benchmarks, respectively. To highlight the difference clearly, we chose only representative methods and compare their curves to those of FedLUAR. It is clearly shown that FedLUAR achieves virtually the same accuracy as FedAvg while having significantly reduced communication cost. These results well prove the efficacy of the proposed update recycling method.
Validation Accuracy (%)
Communication
0
61.27 1: 0.7%
1.00
61.25 + 0.1%
0.84
60.92 + 1.7%
0.68
0.7%
0.47
50.07 + 1.6%
0.30
Table 9: The CIFAR-10 (ResNe120) classification performance with varying 8 settings.
8
Validation Accuracy (%)
Communication Cost
59.88 + 0.8%
1.00
4
59.85 1 0.1%
0.88
8
59.93 1 0.1%
0.76
12
59.73 4 0.6%
0.61
14
56.49 1 0.1%
0.54
1 0.7%
0.2%
Table 10: The CIFAR-100 NWrreResNer28) classification performance with varying settings.
30

---

Validation Accuracy (S)
Communication Cost
0
71.01
0.4%
1.00
71.46 1: 0.1%
0.50
73.17 1.1%
0.18
60.35
Table 11: The FEMNIST (CNN) classification performance with varying 6 settings.
Validation Accuracy (%)
Communication Cost
0.1%
1.00
10
82.82
0.1%
0.56
20
82.24 1 0.1%
0.36
30
82.80 + 0.1%
0.17
35
79.00 + 0.1%
0.08
Table 12: The AG news (DistilIBERT) classification performance with varying settings.
Method
0.1
0.5
1.0
Comm
Acc
Acc
Acc
FodAvg FedLUAR
61.27% 60.15%
76.51%
73%
0.47
76.01%
79.50%
Table 13: CIFAR-10 with various Dirchlet concentration factor a settings. The number of recycled layers. 8 10 out of 20 layers in ResNet20.
Method
0.1
0.5
1.0
FedAvg
1.00
81.53%
82, 66%
83, 22%
FedLUAR
0.17
81.88%
82.80%
82.755 75%
Table 14: AG News with various Dirchlet concentration factor settings. The number of recycled layers, ō 30 out of layers DittilBERT.
Method
(0.5)
(0.25)
(0.125)
Comm
Aoc
Acc
Acc
FedAvg
54.24%
61.27%
57.81%
FedLUAR
0.48
53.34%
60.15%
57.81%
Table 15: The 5 10 out of 20 layers in ResNet20 for FedLUAR. 64, 128, and 256 indicate the total number of clients, and the numbers in parentheses (0.5, 0.25, 0.125) represent the client activation ratio CIFAR-10.
Method
64 (U.5)
128 (0.25)
256 (0.125)
FedAvg
1.00
66.31 %
71.01%
75.97%
FedLUAR
0.14
68. 15%
73.
13%
76.72%
Table 16: The 8 2 of layers in CNN for FedLUAR 64, 128, and 256 indicate the total number of clients, and the numbers parentheses (0.5, 0.25, 0.125) represent client activation FEMNIST.
31

---

FedAvg
FedPAQ
FedLUAR
FedAvg
LBGM
FedLUAR
FedPAQ
LBGM
Training Loss (Cross Entropy)
ValiditionnAaa Accuracy (0-1)
2.5
0.6
2.0
1.5
0.3
1.0
0.5
1.0
0.5
1.0
Communication Cost (Relative to FedAvg)
Communication Cost
(Relative FedAvg)
Figure The learning curve comparisons for CIFAR-IO0 100 (Wide- ResNec28-10). x-axis represents communication cost relative to FedAvg. FedPAQ has the least amount of communication cost for 300 epochs, however loses the accuracy too much. FedLUAR nearly does not drop the accuracy while significantly reducing the communication cost.
FedAvg LBGM
FedPAQ FedLUAR
FedPAQ
FedAvg
FedLUAR
LBGM
Training Loss (Cross Entropy)
1.2
Validation Accuracy (0-1)
0.8
0.9
0.6
0.6
0.4
0.3
0.2
0.5
Communication Cost
Communication Cost (Relative to FedAvg)
(R elative to FedAvg)
Figure 6: The learning curve comparisons for FEMNIST (CNN). The x-axis represents the communication cost relative to FedAvg. FedLUAR significantly reduces the communication cost while maintaining the model accuracy.
32


Reviews:
[{"review_id": "NtjBUbUwbI", "reviewer": "Reviewer_21T7", "summary": "This paper introduces FedLUAR, a novel communication-efficient Federated Learning (FL) method designed to mitigate the high communication costs associated with training large models. The central idea is to reduce the volume of data transmitted from clients to the server by selectively \"recycling\" historical updates for certain model layers. The method proposes a layer prioritization metric, $s_{t,l}$, which is the ratio of a layer's aggregated gradient magnitude to its weight magnitude. This metric identifies layers where an update is likely to have a less substantial impact. In each communication round, FedLUAR uses a weighted random sampling scheme to select a subset of these low-priority layers. For the selected layers, clients do not transmit their newly computed updates; instead, the server re-applies the aggregated update from the previous communication round. For all other layers, training proceeds as in standard FedAvg. The authors provide a theoretical convergence analysis and conduct extensive experiments on CIFAR-10/100, FEMNIST, and AG News, demonstrating that FedLUAR can achieve comparable accuracy to FedAvg while reducing communication costs by over 80%. Furthermore, they show that the recycling mechanism is compatible with other advanced FL optimizers like FedProx and MOON.", "questions": "1.  Adaptive $\\delta$: The number of recycled layers, $\\delta$, is a crucial hyperparameter. Have you considered an adaptive schedule for $\\delta$ that might, for example, decrease the number of recycled layers as training progresses and the model fine-tunes? A dynamic approach could potentially yield a better and more robust trade-off between communication and accuracy.\n\n2.  Impact of Data Heterogeneity: The paper notes that convergence is guaranteed under non-IID conditions. Could you provide a deeper analysis or intuition on the interplay between the degree of data heterogeneity and the effectiveness of FedLUAR? Specifically, as non-IIDness increases, client updates diverge more. Does this make the *previous global update* a poorer proxy for the *current required update*, potentially slowing convergence more than in the IID case?\n\n3.  Scalability to Large-Scale Models: Regarding the server-side memory required to store historical updates, could you provide a more formal analysis of its scaling behaviour? For a model with $L$ layers and parameters $\\Theta$, the overhead appears to be $O(\\sum_{l \\in \\text{recycled}} |\\theta_l|)$. How would this fare with modern foundational models?\n\n4.  Orthogonality with Update Compression: FedLUAR is shown to be compatible with different FL optimizers. Is the mechanism also orthogonal to other communication efficiency techniques like quantization or sparsification? For instance, could one apply 8-bit quantization to the non-recycled layer updates while recycling the others? An analysis of whether these benefits are additive would significantly strengthen the paper's positioning as a general-purpose tool.", "limitations": "The authors acknowledge some technical limitations related to the method's overhead, their treatment of broader limitations, particularly concerning privacy and fairness, is absent. Their discussion of negative societal impacts is also notably shallow, simply stating that none are anticipated and focusing exclusively on positive environmental benefits, which falls short of the comprehensive self-assessment expected for responsible AI research.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- Originality and Novelty: The core concept of recycling historical updates is a significant and novel contribution. While the field has extensively explored compression techniques like quantization, sparsification, and structured updates, the idea of reusing a previous update as a proxy for the current one is a fresh perspective. The ablation study, which demonstrates that recycling is substantially better than simply skipping the updates for low-priority layers, provides strong evidence for the value of this approach. This clearly distinguishes the work from prior art.\n\n- Quality of the Proposed Metric: The paper proposes a nuanced layer-prioritization metric ($s_{t,l}$) that goes beyond simple gradient norms. By considering the update's magnitude relative to the layer's weights, it offers a more stable and potentially more accurate way to gauge an update's impact. This is a valuable insight, as it challenges the common assumption that larger gradients always correspond to more important updates. The empirical results appear to validate the effectiveness of this metric.\n\n- Technical Soundness and Empirical Rigor: The claims are well-supported by both theoretical analysis and comprehensive experiments. The convergence analysis, which shows that the noise introduced by recycling is bounded under standard assumptions, provides a solid theoretical foundation. Empirically, the method is tested across multiple datasets, models, and degrees of data heterogeneity, consistently demonstrating significant communication savings with minimal to no loss in final accuracy.\n\n- Significance and Practicality: The method's \"plug-and-play\" nature is a major strength. By demonstrating compatibility with established FL optimizers like FedProx and MOON, the authors position FedLUAR not as a competing algorithm but as a general-purpose enhancement. This significantly increases its potential for adoption by researchers and practitioners looking to make existing FL pipelines more efficient. The reported communication savings are substantial and practically meaningful.\n\nWeaknesses:\n\n- Unaddressed Server-Side Overheads: The paper claims the method requires \"slightly more memory\" on the server to store historical updates. This claim lacks quantification. For very large models (e.g., LLMs with billions of parameters), storing even a single round of global updates could be a significant memory burden, potentially becoming a new bottleneck. This aspect of practical feasibility needs a more thorough analysis.\n\n- Hyperparameter Sensitivity: The performance of FedLUAR hinges on the hyperparameter $\\delta$ (the number of recycled layers), which must be manually tuned. The paper does not provide clear guidance or an adaptive strategy for setting $\\delta$, which could make it difficult to apply in practice. An effective trade-off between communication and accuracy depends on finding a near-optimal $\\delta$, which may require a costly hyperparameter search for each new task.\n\nLimited Evaluation in Dynamic Scenarios: The theoretical analysis assumes a standard FL setting. However, the core assumption that a stale update is a good proxy might break down in highly non-stationary environments where data distributions shift rapidly across rounds. In such cases, recycling could introduce significant error and slow down convergence. The practical impact in these more challenging (and realistic) scenarios is not sufficiently explored.\n\n- Insufficient Comparison with Contemporary Layer-wise Methods: While the comparison to FedAvg is essential, the paper's positioning would be much clearer with a direct comparison to other recent layer-wise FL techniques. For instance, methods like FedLAMA [3, 4] also use layer-wise strategies (adaptive aggregation frequency) to improve efficiency. Understanding how FedLUAR's recycling mechanism compares to or complements such approaches is critical for assessing its state-of-the-art standing.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns, and I will maintain my original score.", "rebuttal": "We appreciate the reviewers’ thorough evaluations and positive feedback on our work. Below, we provide detailed responses to the reviewers’ comments and questions.\n\n**Q1**: Have you considered an adaptive schedule for \\delta?\n\n**A1**: Thank you for the valuable suggestion. Yes, we agree that automating the choice of $\\delta$ would make *FedLUAR* significantly more practical. We do consider deriving a robust way of finding the ‘safe’ number of recycled layers as a critical future work. In this work, we upper-bounded the noise introduced by update recycling in non-IID settings while preserving convergence properties. As a next step, we aim to develop a model-agnostic method for determining the number of recycled layers without violating the derived bound. Our research group is currently studying this topic to develop an efficient and automated layer-wise update recycling method.\n\n--\n\n**Q2**: Could you provide a deeper analysis or intuition on the interplay between the degree of data heterogeneity and the effectiveness of FedLUAR?\n\n**A2**: As pointed out by the reviewer, as the degree of non-IIDness increases, the convergence of *FedLUAR* becomes slower like any other FL methods. Specifically, in our convergence rate shown in (7), the final term with $\\sigma_G$ on the right-hand side likely increases as the non-IIDness becomes stronger. Since the term has a complexity of $O(\\frac{1}{1 - 16 \\kappa})$, as $\\kappa \\in [0, \\frac{1}{16})$ increases, the final term increases, making it converge more slowly. However, our analysis considers the maximum bound of $n_t$, the noise introduced by update recycling. Hence, the theoretical result is not affected by the degree of non-IIDness.\n\nWe conducted additional experiments to demonstrate that *FedLUAR* is robust to the degree of non-IIDness in practice. Thank you for this sharp question. We will add discussion to Appendix.\n\n- CIFAR-10 (ResNet20) with various Dirchlet concentration factor $\\alpha$ settings. The number of recycled layers, $\\delta = 10$ out of $20$ layers in ResNet20.\n    |   Method   |  | $\\alpha=0.1$ | $\\alpha=0.5$ | $\\alpha=1.0$ |\n    |:----------:|:-----:|:------------:|:------------:|:------------:|\n    |            |  Comm   |     Acc      |     Acc      |     Acc      |\n    |  FedAvg    | 1.00  |   61.27%     |   76.54%     |   79.73%     |\n    | *FedLUAR*  | 0.47  |   60.15%     |   76.04%     |   79.50%     |\n\n- AG News (DistillBERT) with various Dirichlet concentration factor $\\alpha$ settings. The number of recycled layers, $\\delta = 30$ out of $40$ layers in DistillBERT.\n    |   Method   |  | $\\alpha=0.2$ | $\\alpha=0.5$ | $\\alpha=0.8$ |\n    |:----------:|:-----:|:------------:|:------------:|:------------:|\n    |            |  Comm   |     Acc      |     Acc      |     Acc      |\n    |  FedAvg    | 1.00  |   81.53%     |  82.66%      |   83.22%     |\n    | *FedLUAR*  | 0.17  |   81.88%     |   82.80%     |   82.75%     |\n\nIn both benchmarks, *FedLUAR* achieves comparable accuracy to FedAvg regardless of $\\alpha$, while considerably reducing the communication cost. Therefore, we conclude that ***FedLUAR* is robust to the degree of non-IIDness**.\n\n--\n\n**Q3**: What is the extra memory footprint at the server-side?\n\n**A3**: We would like to clarify that ***FedLUAR*’s memory footprint is actually smaller than that of FedAvg** (We realized this misanalysis after submitting the paper). Since all methods require the same amount of memory to store the global model, we focus only on the memory consumption of the local models.\n\n - In FedAvg, server should receive local models from all active clients. Thus, the memory footprint is $a \\cdot d$, where $a$ is the number of active clients and $d$ is the model size.\n - *FedLUAR* receives local models from active clients except $\\delta$ layers. Thus, the memory footprint is $a \\cdot (d - k)$, where $k$ is the size of $\\delta$ layers. Instead, the previous global update should be kept in the memory space for the $\\delta$ layers, consuming $k$ space only. Therefore, *FedLUAR*’s memory footprint is $a \\cdot (d - k) + k < a \\cdot d$.\n\nTo support our analysis, we actually measured the memory footprint of FedAvg and *FedLUAR* during training. First, the total number of clients is 128 and only randomly selected 32 clients are activated at each communication round. We use MPI to run FL on 2 GPUs. Thus, each process locally train 16 models and then all the locally trained models are aggregated using *MPI_Allreduce()*. Under this setting, we measured the memory footprint of each process, observed during training.\n\n| CIFAR-10 (ResNet20) | $\\delta$ | Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg| - | 33.49 |\n| *FedLUAR* | 10 | 15.23 | \n\n| CIFAR-100 (WRN28-10) | $\\delta$ |  Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg | - | 4462.80 |\n|*FedLUAR* | 14 | 2604.88|\n\n| FEMNIST (CNN) | $\\delta$ |  Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg | - | 806.11 |\n|*FedLUAR* | 2 | 204.73 |\n\n| AG News (DistillBERT) | $\\delta$ |  Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg | - | 8294.18 |\n|*FedLUAR* | 30 | 1825.42|\n\nWe clearly see that *FedLUAR* actually consumes less memory space than FedAvg. Thank you again for pointing out this. We will add a brief discussion on this memory consumption in Section 3.2 and Appendix.\n\n--\n\n**Q4**: Is *FedLUAR* orthogonal to parameter quantization or sparsification?\n\n**A4**: Yes, our empirical study already shows that the proposed update recycling method is complementary to quantization. In Table 2, we compared the performance between with and without applying *LUAR* to FedPAQ which is a representative quantization-based federated learning method. E.g., FedPAQ’s communication cost is further reduced from $50$% to $33$% by recycling 10 layers’ updates without compromising CIFAR-10 (ResNet20) accuracy. This empirical result demonstrates that our method is well harmonized with quantization even in non-IID settings.\n\nIn addition, we performed extra experiments to show how *FedLUAR* works with sparsification methods. We compare the benchmark performance between with and without applying *LUAR*.\n - CIFAR-10 (ResNet20) with PruneFL, a sparsification-based FL method. The number of recycled layers, $\\delta = 10$ out of $20$ layers in ResNet20.\n| Method | Accuracy | Comm |\n| :--- | :---: | :---: |\n| PruneFL | 56.76% | 0.69|\n| PruneFL + *LUAR* | 55.43% | 0.49 |\n\n - FEMNIST (CNN) with PruneFL. The number of recycled layers, $\\delta = 2$ out of $4$ layers in a CNN.\n| Method | Accuracy | Comm |\n| :--- | :---: | :---: |\n| PruneFL | 69.42% | 0.19|\n| PruneFL + *LUAR* | 69.11% | 0.11 |\n\nIn both benchmarks, we see that our proposed update recycling method maintains the accuracy while further reducing the communication cost of PruneFL. So, we conclude that *FedLUAR* is complementary to sparsification methods."}, {"review_id": "jiRqAy5p8c", "reviewer": "Reviewer_sW79", "summary": "The paper introduces a communication-efficient approach to distributed training by selectively reusing updates for certain layers, avoiding aggregation in every round. Layer selection is guided by the gradient-to-weight ratio, in contrast to prior work that considers only gradient magnitude or weight norms. Experiments reveal that layers with large gradients may also have large weights, making their updates less impactful. Theoretical analysis bounds the noise introduced by partial aggregation, while empirical results demonstrate improvements in model performance, communication efficiency, and layer criticality metrics.", "questions": "1 -    On which side is the computation of layer importance performed? When only a subset of parameters is shared, does the selection happen on the client side before sending, or is it determined by the server?\n\n2 -    What does \\kappa refer to in Remark 1 (line 222)? It appears without a clear definition.\n\n3 -    Remark 2 (line 226) states that all terms on the right-hand side of inequality (7) vanish with a diminishing learning rate, except part of the second term. However, the first term seems to contain the learning rate in its discriminant - could you clarify?\n\n4 -    The evaluation and the last paragraph of Section 3.3 discuss data heterogeneity, although the method itself is not specifically designed for it. Why is heterogeneity relevant in this context?", "limitations": "N\\A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The observation on balancing weight norms with gradient norms is insightful and, according to empirical results, contributes to improved distributed training.\n\nHowever, the algorithm description lacks clarity regarding where computations occur: criticality scores are computed on the server, yet clients require them to determine which parameters to share (line 181).\n\nWhile the theoretical analysis provides bounds on the noise under a diminishing learning rate, it does not offer justification for the critical layer selection strategy.\n\nEmpirical evaluation is not taking as benchmark method in [1].\n\n[1] Kamp M, Adilova L, Sicking J, Hüger F, Schlicht P, Wirtz T, Wrobel S. Efficient decentralized deep learning by dynamic model averaging. InMachine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10–14, 2018, Proceedings, Part I 18 2019 (pp. 393-409). Springer International Publishing.", "was_revised": true, "final_justification": "The rebuttal clarified my concerns and did not change my score decision.", "rebuttal": "We appreciate the reviewer’s careful and thorough comments. Please find our responses to each of the reviewer's questions.\n\n\n**Q1**: On which side is the computation of layer importance performed?\n\n**A1**: As shown in Algorithm 1 (*LUAR*), once the local models are aggregated at the server, the **server computes the layer-wise importance** (line 6). It then sends minimal information to the clients, indicating which layers will be recycled. Based on that information, clients skip uploading those layers to the server at the end of the communication round. Therefore, *FedLUAR* does not impose any additional computational overhead on the clients for calculating layer importance.\n\n--\n\n**Q2**: What does $\\kappa$ refer to in Remark 1 (line 222)?\n\n**A2**: Thank you for pointing out this presentation quality issue. Actuatlly, $\\kappa$ is defined at line 208. This is the discrepancy between the magnitude of FedAvg’s update and that of FedLUAR’s update. In Remark 1, “small $\\kappa$” indicates that the discrepancy is small (updates are recycled in fewer layers).\n\n--\n\n**Q3**: How does the first term on the right-hand side of inequality (7) vanish?\n\n**A3**: We acknowledge that Remark 2 is written in an unclear way. *FedLUAR* guarantees convergence as $T \\rightarrow \\infty$. For simplicity, we omitted $1/T$ on both sides in (7). If we multiply $1/T$ on both sides and $T \\rightarrow \\infty$, the first term on the right-hand side vanishes (finite horizon result). We will accordingly revise the manuscript to clarify this.\n\n--\n\n**Q4**: Why is heterogeneity relevant in this context (the last paragraph of Section 3.3)?\n\n**A4**: We appreciate this critical question. We found out that our explanation was not clear enough. Please find the revised paragraph below. We hope this clarification addresses the reviewer’s concern.\n\n*In general, as the degree of non-IIDness increases, the global variance $\\sigma_G$ will likely grow due to greater discrepancies among local datasets. As shown in the final term on the right-hand side of (7), recycling updates in more layers increases the coefficient, which in turn amplifies the final term. Consequently, the model is expected to converge more slowly.*\n\n**Extra response**: Additionally, we added a brief discussion on the comparison between *FedLUAR* and the recommended related work [1]. Thank you again for bringing this work to our attention. Although we could not implement and evaluate the dynamic model aggregation method due to the limited rebuttal period, we plan to include a comparative study incorporating it in our future extension of this work."}, {"review_id": "Wqn9ryXv6e", "reviewer": "Reviewer_ocSH", "summary": "This paper proposes FedLUAR, a Layer-wise Update Aggregation with Recycling scheme for communication-efficient federated learning. FedLUAR selects a subset of layers based on a proposed metric that quantifies how aggregated gradients influence model parameter values in each layer, and recycles previous updates on the server side rather than discarding them.", "questions": "As stated in weakness.\nFurther experiments suggestions:\n- Experiments examining how varying degrees of data heterogeneity affect the proposed method's performance would strengthen the evaluation.\n- Evaluation with different numbers of participating users to understand scalability limitations and performance trade-offs.\n- Investigation of the method's benefits in few-shot learning scenarios, if applicable to the problem domain.\n- Experiments on larger datasets and modern foundation models to demonstrate practical applicability.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n- The concept of recycling previous updates rather than simply dropping them presents an interesting approach to improving communication efficiency in federated learning.\n\nWeakness:\n- The paper lacks clear theoretical justification for the Gradient-Weight Ratio Analysis. Is there an optimal proof or empirical motivation for this metric, or does it serve primarily as a heuristic function? The theoretical grounding needs strengthening.\n- As acknowledged in the limitations section, FedLUAR requires storing weights from all users in previous communication rounds, leading to significant memory overhead. This limitation may prevent the method from scaling to scenarios with large numbers of users training large models.\n- The main contributions of this work should be more explicitly stated and clearly distinguished from existing methods. \n- Beyond YOGA, the paper should include comparisons with other layer-wise model aggregation methods for federated learning with centralized servers. A comprehensive comparison would better position this work within the existing literature.", "was_revised": true, "final_justification": "This rebuttal addresses most of my concerns. I'll keep my score.", "rebuttal": "We thank the reviewer for the valuable comments. Please find our responses to each of the reviewer's questions.\n\n**Q1**: Experiments examining how varying degrees of data heterogeneity affect the proposed method's performance would strengthen the evaluation.\n\n**A1**: As pointed out by the reviewer, yes, the degree of non-IIDness strongly affects the training efficiency of FL methods. We conducted additional experiments to demonstrate that *FedLUAR* is robust to the degree of non-IIDness as follows.\n\n- CIFAR-10 (ResNet20) with various Dirchlet concentration factor $\\alpha$ settings. The number of recycled layers, $\\delta = 10$ out of $20$ layers in ResNet20.\n    |   Method   |  | $\\alpha=0.1$ | $\\alpha=0.5$ | $\\alpha=1.0$ |\n    |:----------:|:-----:|:------------:|:------------:|:------------:|\n    |            |  Comm   |     Acc      |     Acc      |     Acc      |\n    |  FedAvg    | 1.00  |   61.27%     |   76.54%     |   79.73%     |\n    | *FedLUAR*  | 0.47  |   60.15%     |   76.04%     |   79.50%     |\n\n- AG News (DistillBERT) with various Dirichlet concentration factor $\\alpha$ settings. The number of recycled layers, $\\delta = 30$ out of $40$ layers in DistillBERT.\n    |   Method   |  | $\\alpha=0.2$ | $\\alpha=0.5$ | $\\alpha=0.8$ |\n    |:----------:|:-----:|:------------:|:------------:|:------------:|\n    |            |  Comm   |     Acc      |     Acc      |     Acc      |\n    |  FedAvg    | 1.00  |   81.53%     |  82.66%      |   83.22%     |\n    | *FedLUAR*  | 0.17  |   81.88%     |   82.80%     |   82.75%     |\n\n\nIn both benchmarks, *FedLUAR* achieves comparable accuracy to FedAvg regardless of $\\alpha$, while considerably reducing the communication cost. Therefore, we conclude that ***FedLUAR* is robust to the degree of non-IIDness**.\n\n--\n\n**Q2**: Evaluation with different numbers of participating users to understand scalability limitations and performance trade-offs.\n\n**A2**: Thank you for this sharp comment. We conducted additional ablation study with various numbers of active clients as follows.\n\n - CIFAR-10 (ResNet20) experiments. The $\\delta=10$ for *FedLUAR*. In the following table, 64, 128, and 256 indicate the total number of clients, and the numbers in parentheses (0.5, 0.25, 0.125) represent the client activation ratio.\n| | | 64 (0.5) | 128 (0.25) | 256 (0.125) |\n|---------: |----: |--------: |----------: |-----------: |\n|  Method   | Comm |  Acc |    Acc |     Acc |\n| FedAvg    | 1.00 |   54.24% |     61.27% |      58.07% |\n| *FedLUAR* | 0.48 |   53.34% |     60.15% |      57.81% |\n\n - FEMNIST (CNN) experiments. The $\\delta=2$ for *FedLUAR*. In the following table, 64, 128, and 256 indicate the total number of clients, and the numbers in parentheses (0.5, 0.25, 0.125) represent the client activation ratio.\n| | | 64 (0.5) | 128 (0.25) | 256 (0.125) |\n|---------: |----: |--------: |----------: |-----------: |\n|  Method   | Comm |  Acc |    Acc |     Acc |\n| FedAvg    | 1.00 |   66.31% |     71.01% |      75.97% |\n| *FedLUAR* | 0.14 |   68.15% |     73.17% |      76.72% |\n\nWe clearly see that, regardless of the total number of clients, *FedLUAR* matches the accuracy of FedAvg while remarkably reducing the communication cost. This ablation study demonstrates the superior scalability of *FedLUAR*. We will add this discussion to Appendix.\n\n--\n\n**Q3**: Investigation of the method's benefits in few-shot learning scenarios, if applicable to the problem domain.\n\n**A3**:  Thank you for suggesting an interesting application of our method: update recycling for efficient federated few-shot learning. We consider this a promising direction for future work. Although we are unable to implement and evaluate it within the limited rebuttal period, we will certainly explore it in future research.\n\n--\n\n**Q4**: Experiments on larger datasets and modern foundation models to demonstrate practical applicability.\n\n**A4**: We recognize that the machine learning community is currently highly interested in efficient training of large foundation models. Our experimental results in Tables 1, 3, and 4 include the AG News benchmark conducted using DistilBERT, a representative modern foundation model. We believe that these results and our analysis offer useful insights into communication-efficient large-scale federated learning for the broader ML community.\n\n--\n\nFinally, we would like to clarify that ***FedLUAR*’s memory footprint is actually smaller than that of FedAvg** (We realized the misanalysis after submitting the paper). Since the global model should be held in the memory space regardless of which algorithm is used, we only focus on the local models.\n- In FedAvg, server should receive local models from all active clients. Thus, the memory footprint is $a \\cdot d$, where a is the number of active clients and d is the model size.\n- *FedLUAR* receives local models from active clients except \\delta layers. Thus, the memory footprint is $a \\cdot (d - k)$, where $k$ is the size of $\\delta$ layers. Instead, the previous global update should be kept in the memory space for the $\\delta$ layers, consuming $k$ space only. Therefore, *FedLUAR*’s memory footprint is $a \\cdot (d - k) + k < a \\cdot d$.\n\nTo support our analysis, we actually measured the memory footprint of FedAvg and *FedLUAR* during training. First, the total number of clients is 128 and only randomly selected 32 clients are activated at each communication round. We use MPI to run FL on 2 GPUs. Thus, each process locally train 16 models and then all the locally trained models are aggregated using *MPI_Allreduce()*. Under this setting, we measured the memory footprint of each process, observed during training.\n\n| CIFAR-10 (ResNet20) | $\\delta$ | Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg| - | 33.49 |\n| *FedLUAR* | 10 | 15.23 | \n\n| CIFAR-100 (WRN28-10) | $\\delta$ |  Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg | - | 4462.80 |\n|*FedLUAR* | 14 | 2604.88|\n\n| FEMNIST (CNN) | $\\delta$ |  Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg | - | 806.11 |\n|*FedLUAR* | 2 | 204.73 |\n\n| AG News (DistillBERT) | $\\delta$ |  Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg | - | 8294.18 |\n|*FedLUAR* | 30 | 1825.42|\n\nWe clearly see that *FedLUAR* actually consumes less memory space than FedAvg. Thank you again for pointing out this. We will add a brief discussion on this memory consumption in Section 3.2 and Appendix.\n\nFinally, we revised Introduction section to clearly summarize our main contributions from this research work."}, {"review_id": "AClSmOk28m", "reviewer": "Reviewer_eMSU", "summary": "This paper proposes FedLUAR, a novel federated learning method that recycles previous model updates on selected layers based on a gradient-to-weight ratio metric. The approach is motivated by the need to reduce communication costs in federated learning without sacrificing accuracy, and it challenges conventional methods that drop updates. Extensive theoretical analysis and empirical evaluations on benchmark datasets demonstrate that the layer-wise update recycling mechanism can significantly reduce communication overhead while maintaining or even accelerating model convergence.", "questions": "See weaknesses.", "limitations": "Limitations are mentioned in the text as the extra memory overhead.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces an innovative update recycling strategy that selectively reuses past layer updates instead of dropping them, offering a fresh perspective on reducing communication costs in federated learning. The method leverages a newly defined metric based on the gradient-to-weight ratio to prioritize layers effectively. \n\n2. The proposed FedLUAR framework achieves significant communication efficiency while preserving model accuracy as demonstrated by experiments on multiple datasets, including image and text classification tasks. The empirical results show that the method reduces communication cost. The experiments are comprehensive and include comparisons with several state-of-the-art communication-efficient federated learning methods.\n\n3. The paper provides rigorous theoretical analysis including convergence proofs and noise bounding under realistic assumptions. The theoretical section clearly lays out the assumptions, lemmas, and theorems necessary for understanding how the update recycling mechanism maintains the stability of model updates. The proofs are detailed enough to give confidence in the method’s practical applicability in non-convex settings.\n\n4. Extensive ablation studies are performed to validate the effectiveness of the proposed layer prioritization metric and the recycling scheme. The ablation experiments compare different layer selection strategies and reinforce the idea that the gradient-to-weight ratio is a strong indicator for identifying less critical layers. The experiments also systematically explore the impact of key hyperparameters on both communication cost and model performance.\n\n\nWeaknesses:\n\n1. The method introduces additional memory overhead on the server as previous updates for certain layers must be stored, which may become a bottleneck for models with very deep or large architectures. The paper does mention the extra memory requirement but does not quantify its impact extensively. The discussion leaves open questions on scalability when server resources are very limited.\n\n2. The approach relies on the tuning of a key hyperparameter (δ) that determines the number of layers to recycle, and optimal settings appear to be dataset dependent. The ablation results indicate that performance can drop if recycling is applied excessively, suggesting sensitivity to this parameter. There is limited guidance on how to best select δ in various federated learning scenarios in real-world deployments.\n\n3. Although the experiments are extensive, the evaluation is limited to a handful of benchmark datasets and standard models, leaving some uncertainty about how the approach will perform in more diverse or larger-scale federated learning environments. The paper does not include scenarios with extremely heterogeneous data distributions that might reveal additional challenges. Information on performance in more varied real-world applications is thus lacking.\n\n4. Some highly related works about compression in FL are not discussed. [1,2] proposed compressing the communicated parameters or the model delta. [2,3] proposed using SVD to compress model parameters. [5] can be regarded as a layer-wise model compression methods.\n\n\n[1] Z. Tang, S. Shi, B. Li and X. Chu, \"GossipFL: A Decentralized Federated Learning Framework With Sparsified and Adaptive Communication,\" In IEEE Transactions on Parallel and Distributed Systems. 2022\n\n[2] Z. Tang, J. Huang, R. Yan, Y. Wang, Z. Tang, S. Shi, A.C. Zhou, X. Chu. Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning. In Proceedings of the 53rd International Conference on Parallel Processing, 2024.\n\n[3] Chai, Di, et al. \"Practical lossless federated singular vector decomposition over billion-scale data.\" Proceedings of the 28th ACM SIGKDD, 2022.\n\n[4] Haolin Wang, Xuefeng Liu, Jianwei Niu, Shaojie Tang. SVDFed: Enabling Communication-Efficient Federated Learning via Singular-Value-Decomposition. In IEEE INFOCOM 2023.\n\n[5] H Wang, M Yurochkin, Y Sun, D Papailiopoulos, Y Khazaeni. Federated learning with matched averaging. In ICLR 2020.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort. Below are our responses and clarifications for each of the reviewer's questions.\n\n**Q1**: The paper does mention the extra memory requirement but does not quantify its impact extensively.\n\n**A1**: We would like to clarify that ***FedLUAR*’s memory footprint is actually smaller than that of FedAvg** (We realized the misanalysis after submitting the paper). Since the global model should be held in the memory space regardless of which algorithm is used, we only focus on the local models.\n- In FedAvg, server should receive local models from all active clients. Thus, the memory footprint is $a \\cdot d$, where a is the number of active clients and d is the model size.\n- *FedLUAR* receives local models from active clients except \\delta layers. Thus, the memory footprint is $a \\cdot (d - k)$, where $k$ is the size of $\\delta$ layers. Instead, the previous global update should be kept in the memory space for the $\\delta$ layers, consuming $k$ space only. Therefore, *FedLUAR*’s memory footprint is $a \\cdot (d - k) + k < a \\cdot d$.\n\nTo support our analysis, we actually measured the memory footprint of FedAvg and *FedLUAR* during training. First, the total number of clients is 128 and only randomly selected 32 clients are activated at each communication round. We use MPI to run FL on 2 GPUs. Thus, each process locally train 16 models and then all the locally trained models are aggregated using *MPI_Allreduce()*. Under this setting, we measured the memory footprint of each process, observed during training.\n\n| CIFAR-10 (ResNet20) | $\\delta$ | Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg| - | 33.49 |\n| *FedLUAR* | 10 | 15.23 | \n\n| CIFAR-100 (WRN28-10) | $\\delta$ |  Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg | - | 4462.80 |\n|*FedLUAR* | 14 | 2604.88|\n\n| FEMNIST (CNN) | $\\delta$ |  Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg | - | 806.11 |\n|*FedLUAR* | 2 | 204.73 |\n\n| AG News (DistillBERT) | $\\delta$ |  Memory Footprint (MB) |\n| :--- | :---:|:---:|\n|FedAvg | - | 8294.18 |\n|*FedLUAR* | 30 | 1825.42|\n\nWe clearly see that *FedLUAR* actually consumes less memory space than FedAvg. Thank you again for pointing out this. We will add a brief discussion on this memory consumption in Section 3.2 and Appendix.\n\n--\n\n**Q2**: There is limited guidance on how to best select $\\delta$ in various federated learning scenarios in real-world deployments.\n\n**A2**: Thank you for this sharp feedback. We acknowledge that our paper provides only limited guidance on how to find good $\\delta$ values. As pointed out by reviewer, the best setting of $\\delta$ varies depending on many factors including training data distribution, model size, and non-IIDness. In our extensive empirical study, however, we find that the method generally performs well when $\\delta$ is not larger than $\\sim 63$% of the total number of layers (averaged across all the benchmarks). A more exhaustive ablation study would help readers easily identify good hyperparameter settings. We plan to extend this work to address system heterogeneity and will include such an ablation study in future work.\n\n--\n\n**Q3**: The paper does not include scenarios with extremely heterogeneous data distributions that might reveal additional challenges.\n\n**A3**: We used Dirichlet concentration parameter of $0.1$ for artificial non-IID datasets. We also used unbalanced distributions with respect to classes. For instance, each local dataset of CIFAR-10 contains up to 3 classes only. This setting represents highly non-IID environments. Considering that many previous works use larger concentration parameter values such as $\\alpha=0.5$, $0.8$, or $1.0$, we believe our experimental settings are sufficiently challenging/extreme.\n\nNevertheless, we conducted additional experiments to further provide empirical evidence demonstrating the effectiveness of *FedLUAR*, as summarized below.\n\n- CIFAR-10 (ResNet20) with various Dirchlet concentration factor $\\alpha$ settings. The number of recycled layers, $\\delta = 10$ out of $20$ layers in ResNet20.\n    |   Method   |  | $\\alpha=0.1$ | $\\alpha=0.5$ | $\\alpha=1.0$ |\n    |:----------:|:-----:|:------------:|:------------:|:------------:|\n    |            |  Comm   |     Acc      |     Acc      |     Acc      |\n    |  FedAvg    | 1.00  |   61.27%     |   76.54%     |   79.73%     |\n    | *FedLUAR*  | 0.47  |   60.15%     |   76.04%     |   79.50%     |\n\n- AG News (DistillBERT) with various Dirichlet concentration factor $\\alpha$ settings. The number of recycled layers, $\\delta = 30$ out of $40$ layers in DistillBERT.\n    |   Method   |  | $\\alpha=0.2$ | $\\alpha=0.5$ | $\\alpha=0.8$ |\n    |:----------:|:-----:|:------------:|:------------:|:------------:|\n    |            |  Comm   |     Acc      |     Acc      |     Acc      |\n    |  FedAvg    | 1.00  |   81.53%     |  82.66%      |   83.22%     |\n    | *FedLUAR*  | 0.17  |   81.88%     |   82.80%     |   82.75%     |\n\nIn both benchmarks, *FedLUAR* achieves comparable accuracy to FedAvg regardless of $\\alpha$, while considerably reducing the communication cost. Therefore, we conclude that ***FedLUAR* is robust to the degree of non-IIDness**.\n\n--\n\n**Q4**: Some highly related works about compression in FL are not discussed.\n\n**A4**: Thank you for bringing these related works to our attention. We will include a discussion of these prior works in Section 2, highlighting their respective strengths and limitations. We expect our update recycling method to be compatible with those compression-based FL methods, as it has already been empirically shown to work with both structured and sketch-based compression approaches such as FedPAQ and PruneFL.\n\nTo further support our argument, we performed extra experiments to compare *FedLUAR* and PruneFL, a sparsification-based FL method.\n - CIFAR-10 (ResNet20) with PruneFL. The number of recycled layers, $\\delta = 10$ out of $20$ layers in ResNet20.\n| Method | Accuracy | Comm |\n| :--- | :---: | :---: |\n| PruneFL | 56.76% | 0.69|\n| PruneFL + *LUAR* | 55.43% | 0.49 |\n\n - FEMNIST (CNN) with PruneFL. The number of recycled layers, $\\delta = 2$ out of $4$ layers in a CNN.\n| Method | Accuracy | Comm |\n| :--- | :---: | :---: |\n| PruneFL | 69.42% | 0.19|\n| PruneFL + *LUAR* | 69.11% | 0.11 |\n\nIn both benchmarks, we see that our proposed update recycling method maintains the accuracy while further reducing the communication cost of PruneFL. So, we conclude that *FedLUAR* is complementary to sparsification methods."}]

Author rebuttal:

Meta-review: This paper shows a novel approach to communication-efficient federated learning that recycles historical updates for low-priority layers based on a gradient-to-weight ratio metric. The idea of reusing past updates rather than dropping them is original. The paper has a theoretical convergence analysis and empirical validation across multiple standard datasets (although focusing on CNNs), showing significant improvements, and demonstrating compatibility with other FL optimizers. Reviewers noted the dependence on a manually tuned hyperparameter related to recycled layers, the absence of large-scale experiments, however overall the consensus is that the strengths outweigh the negatives. Overall, I recommend acceptance, as the paper presents a new and well-supported method.
Decision: Accept (poster)