Layer-wise Adaptive Model Aggregation for Scalable Federated Learning
Authors: Sunwoo Lee
Tuo Zhang
Chaoyang He
Salman Avestimehr
Venue: ICLR 2022 Submitted (2022)
Keywords: federated learning
model aggregation
neural network

Abstract: In Federated Learning, a common approach for aggregating local models across clients is periodic averaging of the full model parameters. It is, however, known that different layers of neural networks can have a different degree of model discrepancy across the clients. The conventional full aggregation scheme does not consider such a difference and synchronizes the whole model parameters at once, resulting in inefficient network bandwidth consumption. Aggregating the parameters that are similar across the clients does not make meaningful training progress while increasing the communication cost. We propose FedLAMA, a layer-wise model aggregation scheme for scalable Federated Learning. FedLAMA adaptively adjusts the aggregation interval in a layer-wise manner, jointly considering the model discrepancy and the communication cost. The layer-wise aggregation method enables to finely control the aggregation interval to relax the aggregation frequency without a significant impact on the model accuracy. Our empirical study shows that FedLAMA reduces the communication cost by up to $60\%$ for IID data and $70\%$ for non-IID data while achieving a comparable accuracy to FedAvg. 

Under review as a conference paper at ICLR 2022
LAYER-WISE ADAPTIVE MODEL AGGREGATION FOR SCALABLE FEDERATED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In Federated Learning, common approach for aggregating local models across clients periodic averaging of the full model parameters. It is, however, known that different layers of neural networks can have different degree of model dis- crepancy across the clients. The conventional full aggregation scheme does not consider such difference and synchronizes the whole model parameters at once, resulting in inefficient network bandwidth consumption. Aggregating the parame- ters that are similar across the clients does not make meaningful training progress while increasing the communication cost. We propose FedLAMA, a layer-wise wise model aggregation scheme for scalable Federated Learning. FedLAMA adap- tively adjusts the aggregation interval in layer-wise manner, jointly considering the model discrepancy and the communication cost. The layer-wise aggregation method enables to finely control the aggregation interval to relax the aggregation frequency without a significant impact on the model accuracy. Our empirical study shows that FedLAMA reduces the communication cost by up to 60% for IID) data and 70% for non-IID data while achieving a comparable accuracy to FedAvg.
INTRODUCTION
In Federated Learning, periodic full model aggregation is the most common approach for aggregat- ing local models across clients. Many Federated Learning algorithms,, such as FedAvg (McMahan al. (2017)) FedProx (Li et al. (2018)). FedNova (Wang al. (2020), and SCAFFOLD (Karim- ireddy et al. 2020)), assume the underlying periodic full aggregation scheme. However, it has been observed that the magnitude of gradients can significantly different across the layers of neural net- works Youu et al. (2019). That is, all the layers can have a different degree of model discrepancy. The periodic full aggregation scheme does not consider such a difference and synchronizes the en- tire model parameters at once. Aggregating the parameters that are similar across all the clients does not make meaningful training progress while increasing the communication cost. Considering the limited network bandwidth in usual Federated Learning environments, such inefficient network bandwidth consumption can significantly harm the scalability of Federated Learning applications.
Many researchers have put much effort into addressing the expensive communication issue. Adap- tive model aggregation methods adjust the aggregation interval to reduce the total communication cost (Wang & Joshi (2018); Haddadpour
al. (2019)). Gradient (model) compression (Alistarh cet al. (2018): Albasyoni et al. (2020)) sparsification (Wangni et al. (2017): Wang et al. (2018)) low-rank approximation (Vogels al. (2020); Wang et al. (021)), and quantization (Alistarh et al. (2017); Wen et al. (2017); Reisizaeeh et al. (2020) techniques directly reduce the local data size. Employing heterogeneous model architectures across clients is also a caamunication-effiiinn proach (Diao et al. (2020). While all these works effectively tackle the expensive communication issue from different angles, they commonly assume the underlying periodic full model aggregation.
ap-
To break such convention of periodic full model aggregation, we propose FedLAMA, a novel layer- wise adaptive model aggregation scheme for scalable and accurate Federated Learning. FedLAMA first prioritizes all the layers based on their contributions to the total model discrepancy. We present a metric for estimating the layer-wise degree of model discrepancy at run-time. The aggregation intervals are adjusted based on the layer-wise model discrepancy such that the layers with smaller degree of model discrepancy is assigned with a longer aggregation interval than the other layers. The above steps are repeatedly performed once the entire model is synchronized once.

---

Under review as a conference paper at ICLR 2022
Our focus is on how to relax the model aggregation frequency each layer, jointly considering the communication efficiency and the impact on the convergence properties of federated optimization. By adjusting the aggregation interval based the layer-wise model diserepancy, the local mod- els can be effectively synchronized while reducing the number of communications at each layer. The model accuracy is marginally affected since the intervals are increased only at the layers that have the smallest contribution to the total model discrepancy. Our empirical study demonstrates that FedLAMA automatically finds the interval settings that make a practical trade-off between the communication cost and the model accuracy. We also provide a theoretical convergence analysis of FedLAMA for smooth and non-convex problems under non- -ID data settings.
We evaluate the performance of FedLAMA across three representative image classification bench- mark datasets: CIFAR-10 (Krizhevsky et al. 20099). CIFAR-100, and Federated Extended MNIST (Cohen et al. (2017)) Our experimental results deliver novel insights on how to aggregate the local models efficiently consuming the network bandwidth. Given a fixed number of training iterations, as the aggregation interval increases, FedLAMA reduces the communication cost by up to 60% under IID data settings and 70% under non-IID data settings, while having only a marginal accuracy drop.
RELATED WORKS
The communication-efiinn global model update methods can be catego- rized into two groups: structured update and sketched update (Konečný et al. (2016)). The structured update indicates the methods that enforce a pre-defined fixed structure the local updates, such as low-rank approximation and random mask methods. The sketched update indicates the methods that post-process the local updates via compression, sparsification, or quantization. Both directions well studied and have shown successful results (Alistarh et al. (2018); Albasyoni et al. (2020); Wangni et al.] (2017); Wang et al. (2018); Vogels et al. (2020); Wang et al. (2021); |Alistarh et al. al. (2017): Reisizadeh al. (2020)). The common principle behind these methods is that the local updates can be replaced with a different data representation with a smaller size.
Compression Methods
(2017); Wen Wen
These compression methods can be independently applied to layer-wise aggregation scheme such that the each layer's local update is compressed before being aggregated. Since our focus is on adjusting the aggregation frequency rather than changing the data representation, we do not directly compare the performance between these two approaches. We leave harmonizing the layer-wise aggregation scheme and a variety of compression methods as a promising future work.
Similarity Scores - Canonical Correlation Analysis (CCA) methods are proposed to estimate the representational similarity across different models (Raghu et al. [2017)]Morcos Morcos et al. (2018). Cen- tered Kernel Aligmment (CKA) is an improved extension of CCA (Kornblith et al. (2019). While these methods effectively quantify the degree of similarity, they commonly require expensive com- putations. For example, SVCCA performs singular vector decomposition of the model and CKA computes Hilbert-Schmiidt Independence Criterion multiple times (Gretton et al. (2005)). In addi- tion, the representational similarity does not deliver any information regarding the gradient differ- ence that strongly related to the convergence property. We will propose a practical metric for estimating the layer-wise model discrepancy, which is cheap enough to be used at run-time.
Layer-wise Model Freezing- Layer freezing (dropping) is the representative layer-wise wise technique for neural network training (Brock et al. (2017) : Kumar al. (2019): Zhang & He 2020); Goutam et al. (2020). All these methods commonly stop updating the parameters of the layers in a bottom- up direction. These empirical techniques are supported by the analysis presented in (Raghu et al. (2017)). Since the layers converge from the input-side sequentially, the layer-wise freezing can re- duce the training time without strongly affecting the accuracy. These previous works clearly demon- strate the advantages of processing individual layers separately.
3 BACKGROUND
Federated Optimization We consider federated optimization problems as follows.
m
min XER!
F(x) : Σ Pr F (x)
(1)
1=1
2

---

Under review as a conference paper at ICLR 2022
Algorithm 1: FedLAMA: Federated Layer-wise Adaptive Model Aggregation.
Input: r': base aggregation interval, Φ: interval increasing factor, ps, E ( m...
, {1.... L:];
for k = to do
SGD step: x x (1(((((
for to do if h mod is 0 then Synchronize layer 1: (1,))) + Eiz pix
i=1 (pillua - x(,,) I1²) (T((di(uu(.)))
l mod ΦT' then L Adjust aggregation interval at all L layers (Algorithm 2
10
Output: uk:
where P; = ng/n is the ratio of local data to the total dataset, and F2(x) 1 EEED J((x, g) is the local objective function of client i. is the global dataset size and n, is the local dataset size.
ni
FedAvg is a basic algorithm that solves the above minimization problem. As the degree of data het- erogeneity increases, FedAAg converges more slowly. Several variants of FedAvg, such FedProx, FedNova, and SCAFFOLD, tackle the data heterogeneity issue. All these algorithms commonly aggregate the local solutions using the periodic full aggregation scheme.
Model Discrepancy All local SGD-based algorithms allow the clients to independently train their local models within each communication round. The variance stochastic gradients and heteroge- neous data distribution can lead the local models to different directions parameter space during the local update steps. We formally define such a discrepancy among the models as follows.
Σ Ppllu- - -
where m is the number of clients, u is the synchronized model, and x" is client i's local model. This quantity bounds the difference between the local gradients and the global gradients under smoothness assumption objective functions.
4 LAYER-WISH ADAPTIVE MODEL AGGREGATION
Layer Prioritization - In theoretical analysis, it is common to assume the smoothness of objective functions such that the difference between local gradients and global gradients is bounded by scaled difference of the corresponding sets of parameters. Motivated by this convention, we define 'layer-wise unit model discrepancy' a useful metric for prioritizing the layers as follows.
pullu - 77 (dim(u:)
di
(L,
where is the number of layers, is the layer index, u is the global parameters, x is the client i's local parameters, 7 is the aggregation interval, and dim(-) (-) is the number of parameters.
This metric quantifies how much each parameter contributes to the model discrepancy at each iter- ation. The communication cost is proportional to the number of parameters. Thus. Σm Pallu xi|l /di shows how much model discrepancy be eliminated by synchronizing the layer at unit communication cost. This metric allows prioritizing the layers such that the layers with smaller d value has lower priority than the others.
Adaptive Model Aggregation Algorithm We propose FedLAMA, layer-wise adaptive model aggregation scheme. Algorithm [ shows FedLAMA algorithm. There two input parameters: r' the base aggregation interval and ó is the interval increase factor. First, the parameters at layer l synchronized across the clients after every T iterations (line 6). Then, the proposed metric
3

---

Under review as a conference paper at ICLR 2022
Algorithm 2:
Layer-wise Adaptive Interval Adjustment.
Input: the observed model discrepancy at all L layers, r': the base aggregation interval, Φ the interval increasing factor.
Sorted model discrepancy: d sort (d);
Sorted index of the layers: i +- argsort (d);
Total model size: E dim(u:);
Total model discrepancy: + Σi= di * dim(u:;;
for | to do
+ (-==dd de dim(u.))/68
A (E dim(u;))/A;
Find the layer index: +- us:
if < X then
L Ti + ØT':
10
11
else
12
+
13
Output: r: the adjusted aggregation intervals at all L layers.;
di is calculated using the synchronized parameters ur (line 7). At the end of every dr iterations, FedLAMA adjusts the model aggregation interval at all the layers. (line 9).
Algorithm 2 finds the layers that can be less frequently aggregated making minimal impact on the total model discrepancy. First, the layer-wise degree of model discrepancy is estimated as follows.
Ei= d, dim(u,)
51 Σ d; dim(u:)
where d, is the 7th smallest element in the sorted list of the proposed metric d. Given layers with the smallest di values, di quantifies their contribution to the total model discrepancy. Second, the communication cost impact is estimated as follows.
El dim(u;)
A Et dim(u;)
(4)
A is the ratio the parameters at the layers with the smallest dj values. Thus, A estimates the number of parameters that will be more frequently synchronized than the others. As increases, 5 increases while X decreases monotonically. Algorithm 2 loops over the L layers finding the I value that makes and A similar. In this way, it finds the aggregation interval setting that slightly sacrifices the model discrepancy while remarkably reducing the communication cost.
Figure shows the & and A curves collected from a) CIFAR-10 (ResNet20) training and b) CIFAR- 100
10) training. The x-axis is the number of layers to increase the aggre- gation interval and the y-axis is the ój and 1 A values. The cross point of the two curves is much lower than 0.5 on y-axis in both charts. For instance, in Figure Ta), the two curves are crossed when value is 9, and the corresponding y value is near 0.2. That is, when the aggregation interval is increased at those 9 layers, 20% of the total model discrepancy will increase by a factor of á while 80% the total communication cost will decrease by the same factor. Note that the cross points are below 0.5 since the & and 1 A are calculated using the de values sorted in an increasing order.
is worth noting that Fe can be easily extended to improve the convergence rate at the cost of having minor extra communications. In this work, do not consider finding such interval set- tings because can increase the latency cost, which is not desired in Federated Learning. However, in the environments where the latency cost be ignored, such as high-performance computing platforms, FedLAMA can accelerate the convergence by adjusting the intervals based on the cross point of 1 & and ' calculated using the list of d, values sorted in decreasing order.
Impact of Aggregation Interval Increasing Factor ó - In Federated Learning, the communication latency cost is usually not negligible, and the total number of communications strongly affects the
4

---

Under review as a conference paper at ICLR 2022
1.0
detta lambda
1.0
delta lambda)
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
of layers with increased interval
of layers increased interval
a) CIFAR-10 (ResNer20)
b) CIFAR-100 (WRN28-10)
Figure 1: The comparison between the model discrepancy increase factor and the communication cost decrease factor - > for a) CIFAR-10 and b) CIFAR-100 training.
scalability, When increasing the aggregation interval, Algorithm 2 multiplies a pre-defined small constant á the fixed base interval r' (line 10). This approach ensures that the communication latency cost is not increased while the network bandwidth consumption is reduced by a factor of ø.
FedAvg can be considered as special case of FedLAMA where & is set to 1. When LAMA less frequently synchronize a subset of layers, and it results in reducing their communication costs. When increasing the aggregation interval, FedLAMA multiplies á to the base interval 7'. So, is guaranteed that the whole model parameters are fully synchronized after ØT' iterations. Because of the layers with the base aggregation interval T', the total model discrepancy of FedLAMA after Ør' iterations is always smaller than that of FedAvg with an aggregation interval of Gr'.
1, Fed-
5
CONVERGENCE ANALYSIS
5.1 PRELIMINARIES
Notations All vectors in this paper are column vectors. E Rd denotes the parameters of one local model and m is the number of clients. The stochastic gradient computed from a single training use g(x) instead. The full batch gradient is denoted by V F Fx).. We use ||-|| and Ilo to denote 12 norm and matrix operator norm, respectively.
data point & is denoted by g(x, E). For convenience,
Assumptions We analyze the convergence rate of FedLAMA under the following assumptions.
1. (Smoothness). Each local objective function is L-smooth, that is, F - VF((y) s L||x - y/1, Vi {1, m}.
2. (Unbiased Gradient). The stochastic gradient at each client is an unbiased estimator of the local full-batch gradient: (x, {)] (x)...
3. (Bounded Variance). The stochastic gradient at each client has bounded variance:
IEg (x9, 5 FF,(())''' S oz) {1,.... mm, a2 0.
4. (Bounded Dissimilarity). For any sets of weights {p; 2 m Ei Pa 1, there exist
constants 32 > 1 and x² > 0 such that Dm p:|| Fx)]]2 < f2 i== P. V (x)
If local objective functions are identical to each other, 3 and
x 0.
5.2 ANALYSIS
We begin with showing two key lemmas. All the proofs can be found in Appendix. Lemma 5.1. (Framework) Under Assumption 1 3, the learning rate satisfies v S 21. FedLAMA ensures
V (u)) |1
2 < n
E [F(ui) F(u.)] 2n
(pi)²
(5)
I K
x1.
+
E
K
k=
5

---

Under review as a conference paper at ICLR 2022
Lemma 5.2. (Model Discrepancy) Under Assumption 1 4. if the learning rate satisfies n
2(Tmox- 1 FedLAMA ensures
x xt[1p]
Tmaa - 1)o²
Ax²
K
1-A A
L A)
(6)
A
Σ
E
KLA(1 A)
where A 14:/=4/( 41f((maa 1)² and Tmaz is the largest averaging interval across all the layers.
Based on Lemma 5.1 and 5.2 we analyze the convergence rate of FedLAMA as follows.
Theorem 5.3. Suppose all m local models are initialized to the same point U1. Under As- if FedLAMA runs for K iterations and the learning rate satisfies tl S
sumption
min
FedIAMA ensures
2(Tman IIL 2 (Tmar- 11(289+1)
K
4 S nk (F)) -F(,)) + 4n Σ Lo?
E
VF(u)))1
(7)
K
1=1
+ 37 (Tmax - 1)1.2² + 617² TTmar(Taa 1)T?2?
where indicates local minimum and Tmaz is the largest averaging interval across all the layers.
Remark 1. (Linear Speedup) With sufficiently small diminishing learning rate and a large number of training iterations, FedLAMA achieves linear speedup. If the learning is 1 vK
ym and
Pi Vi (1, ,m), we have
K
E
E
IVF(#))11
So
+ +
m (
(8)
K
Vmk
E=1
If K m² the first term on the right-hant hand side becomes dominant and it achieves linear speedup.
Remark 2. (Impact of Interval Increase Factor Φ) The worst-case model discrepancy depends on the largest averaging interval across all the layers, Tmaz = Φr'. The larger the interval increase factor é, the larger the model discrepancy terms in (7). In the meantime, as Φ increases, the communication frequency selected layers proportionally reduced. So, á should be appropriately tuned to effectively reduce the communication cost while not much increasing the model discrepancy.
EXPERIMENTS
Experimental Settings We evaluate FedLAMA using three representative benchmark datasets: CIFAR-10 (ResNer20 (He et al. (2016)). CIFAR-100 NWideResNer28- 10 (Zagoruyko & Komodakis ))16)), and Federated Extended MNIST (CNN (Caldas et al. (2018)). We use TensorFlow 2.4.3 for local training and MPI for model aggregation. All our experiments are conducted 4 compute nodes each of which has NVIDIA v100 GPUs.
Due to the limited compute resources, we simulate Federated Learning such that each process quentially trains multiple models and then the models are aggregated across all the processes at once. While it provides the same classification results the actual Federated Learning, the train- ing time is serialized within each process. Thus, instead of wall-clock time, we consider the total communication cost calculated follows.
C
C
Σ
dim(u:) K1,
(9)
1=1
1=1
where is the total number of communications layer I during the training.

---

Under review as a conference paper at ICLR 2022
Table 1: (II data) CIFAR-10 classification results. The number of workers is 128 and the local
batch size is 32 in all the experiments, The epoch budget is 300.
LR
Base aggregation interval: 7 Interval increase lactor:
Validation
Comun
(FodAvg)
88.37 + 0.02%
100%
0.8
(FedAvg)
84.74 4: 0.05%
50%
(FedLAMA)
88.41 =0.01%
62.33%
0.6
24
1 (FodAvg)
80.34 = 0.3%
25%
0.6
(FedLAMA)
86.21 +0.1%%
42.17%
Table 2: (IID data) CIFAR-100 classification results. The number of workers is 128 and the local batch size is 32 in all the experiments. The epoch budget is 250.
IR
Base aggregation interval: r'
Interval increase factor: d
Validation ncc.
Comm. cost
(FedAvg)
76.50 E 0.02%
100%
0.6
1 (FedAvg)
66.97 +: 0.9%
50%
(Fe)LAMA)
76.02 10.01%
66.01%
0.6
24
1 (FodAvg)
45.01 # 1.1%
25%
(MA)
76.17 1.002%
39.91%
Slyper-Parameter Settings We use 128 clients in our experiments. The local batch size is set to 32 and the learning rate is tuned based on a grid search. For CIFAR-I0 and CIFAR-I00, we artificially generate heterogeneous data distributions using Dirichlet's distribution. When using Non-IID data, we also consider partial device participation such that randomly chosen 25% of the clients participate training at every or iterations. We report the average accuracy across at least three separate runs.
6.1 CLASSIFICATION PERFORMANCE ANALYSIS
To evaluate the proposed model aggregation scheme, we keep all the other factors the same, such as optimizer, the number clients, the degree heterogeneity, and compare the performance across different model aggregation schemes. We compare the performance across three different model aggregation settings as follows.
Periodic full aggregation with an interval of r'
Periodic full aggregation with an interval of Ør'
Layer-wise adaptive aggregation with intervals o' and
The first setting provides the baseline communication cost, and we compare it to the other settings' communication costs. The third setting is FedLAMA with the base aggregation interval 7' and the interval increase factor Due to the limited space, we present a part of experimental results that deliver the key insights. More results can be found in Appendix.
Experimental Results with IID Data We first present CIFAR-10 10 and CIFAR-100 classification results under IID data settings. Table [ and [ show the CIFAR-10 10 and CIFAR- 100 results, respec- tively. Note that the learning rate is individually tuned for each setting using grid search, and we report the best settings. In both tables, the first row shows the performance of FedAvg with a short interval 7' 6. As the interval increases, FedAvg significantly loses the accuracy while the communication cost is proportionally reduced. FedLAMA achieves a comparable accuracy to FedAvg with 7' while its communication cost is similar that of FedAvg with Ør'. These results demonstrate that Algorithm effectively finds the layer-wise interval settings that maximize the communication cost reduction while minimizing the model discrepancy increase.
Experimental Results with Non-IID Data We now evaluate the performance of FedLAMA using non-IllD data. FEMNIST is inherently heterogeneous such that it contains the hand-written digit pictures collected from 3, 350 different writers. We use random 10% of the writers' training samples in our experiments. Table 3 shows the FEMNIST classification results. The base interval T' is set to 10. FedAvg (Ø = 1) significantly loses the accuracy the aggregation interval increases. For example, when the interval increases from 10 to 40, the accuracy is dropped by 2.1% ~ 2.7%. In contrast, FedLAMA maintains the accuracy when increases, while the communication cost is remarkably reduced. This result demonstrates that FedLAMA effectively finds the best interval setting that reduces the communication cost while maintaining the accuracy.

---

Under review as a conference paper at ICLR 2022
Table 3: (Non-IID data) FEMNIST classification results. The number of workers is 128 and the
local batch size is 32 in all the experiments. The number of training iterations is 2, 000.
LR
Base aggregation interval:
Interval increase Factor:
active ralio
Validation
Comm. cost
(FodAvg)
86.04
0.01%
20
(FodAvg)
85.38 = 0.02%
50%
0.04
10
2 (FedLAMM)
25%
86.01 +100
52.83%
40
(FedAvg)
83.97
0.02%
25%
1Faal AMA)
83.61 1.000%
29.97%
10
(HodAvg)
86.59 + 0.01%
100%
20
(FedAvg)
85.50 1: 0.02%
50%
0.04
10
(Pail AMA)
50%
86.07 1.0.02%
53.2%
40
(odAAg)
83.92 = 0.002%
20%
(FALLAMA)
85.77 1:0.02%
29.88%
10
(FedAvg)
85.74 1: 0.03%
100%
20
(FddAvg
85.08
0.01%
50%
0.04
10
2 AMA)
100%
83.40 1.0002%
51.6%
(eddAvg)
83.62 0.02%
25%
10
4 (FedLAMA)
84.67 +0.02%
29.98%
Table (Non-II) -ID data) CIFAR-10 classification results. The number of workers is 128 and the
local batch size is 32 in all the experiments. The number of training iterations is 6, 000.
LR
Base aggregation interval:
Interval increuse factor. @
active ralio
Dirichlet's is cvelf.
Validation
Coman.
(FodAvg)
84.02 + 0.1%
0.4
24
(FedAvg)
25%
0.1
76.27 0.08%
25%
(FAALAAAA
83.06 10.1%
39.52%
(FedAvg)
87.39 + 0.2%
100%
0.8
24
(FFdAVy))
25%
0.5
83.36 = 0.41
25%
4 AMAA
86.37 10.02%
42.40%
(FodAvg)
89.52 0.00%
0.8
24
(FGLAvg)
100%
0.1
3/1.82 0.00%
25%
AMA)
87.47 2:0
42.49%
6
(FedAvg)
90.53 4: 0.08%
100%
0.8
24
(TedAvg)
100%
0.5
85.68 1. 0.1%
25%
(MA))
87.45 10.05%
42.73%
Table 5: (Non-IID data) CIFAR-100 classification results. The number of workers is 128 and the
local batch size is 32 in all the experiments. The number of training iterations is 6, 000.
LR
Base aggregation interval: 7
Interval increase factor: d
active ralio
Dirichle''s 's cpelf.
Validation acc.
Coman, cosl
6
(FodAvg)
79.15 0.027
100%
0.4
12
(FedAvg)
25%
0.1
76.16 0.05%
50%
([eALAMMA)
78.63 10.03%
63.14%
6
(FodAvg)
78.81 = 0.1%
0.4
12
(FedAvg)
25%
0.5
76.11 0.05%
50%
6
((FAILAMA)
77.86 10.04%
63.20%
6
(FedAvg)
79.77 = 0.04%
100%
0.4
12
(FedAvg)
100%
0.1
77.71 0.08%
50%
(FAALAMA)
79.07 10.1%
60.48%
(FodAvg)
80. 0.00%
0.4
12
(FedAvg)
100%
0.5
70 = 0.06%
50%
6
(FeULAMA)
78.88 10.05%
61.73%
the non-lID CIFAR-I0 and CIFAR-100 experimental results. We use Dirichlet's distribution generate heterogeneous data across all the clients. The detailed settings regarding Dirichlet's distribution can be found in Appendix. The base aggregation interval 7' is set to 6. The interval increase factor é set 2 for FedLAMA. Likely to the IID data experiments, we observe that the periodic full averaging significantly loses the accuracy as the model aggregation interval increases, while it has proportionally reduced communication cost. For both datasets, MM achieves a comparable accuracy to the periodic full averaging with the interval of r' while having the communication cost that is close to the periodic full averaging with the increased interval of Ør'. Especially, FedLAMA works effectively even when the Dirichlet's coefficient is set to 0.1. The coefficient of 0.1 represents an extremely high degree of data heterogeneity in terms of not only the number of samples per client but also the balance of the classes assigned to each client. These results imply that FedLAMA is a practical algorithm for Federated Learning applications with highly heterogeneous data distributions.
8

---

Under review as a conference paper at ICLR 2022
F FedAvg
FedLAMA
1000
1500
Sol comnrnncciie
1000
100
500
layer
layer
a] CIFAR-10 (ResNet20)
b) CIFAR-100 (WRN28-10)
c) FEMNIST (CNN)
Figure 2: The number of communications the individual layers. The communications counted during the whole training (non-lID data).
FedAvg
FedLAMA
4000
6.00E-0001
3.00E 1005
3.002 4005
2.007 4007
1.002 1003
5.00E +008
0.00E
1000 +000
0.00E
layer
layer
ID
(ResNel20)
CIFAR-100
Figure 3:
The total data size (communication cost) that correspond Figure 2 comparison clearly shows where the performance gain of FedLAMA comes from.
The data size
6.2 COMMUNICATION EFFICIENCY ANALYSIS
We analyze the total number of communications and the accumulated data size evaluate the com- munication efficiency of FedLAMA. Figure 2 shows the total number of communications at the individual layers. The 7' increases the aggregation interval mostly at the output-side large layers. This means the di value shown in Equation 2 the these layers are smaller than the others. Since these large layers take up most of the total model parameters, the communication cost is remarkably reduced when their aggregation intervals are increased. Figure TigureBsshoe shows the layer-wise local data size shown in Equation 9 FedLAMA shows the significantly smaller total data size than FedAvg. The extra computational cost of FedLAMA almost negligible since it calculates dj after each communication round only. Therefore, given the virtually same computational cost, FedLAMA aggregates the local models at cheaper communication cost, and thus improves the scalablity of Federated Learning.
and ø is for FedLAMA. The key insight is that FedLAMA
We found that the amount of the reduced communication cost not strongly affected the degree of data heterogeneity. As shown in Table [5) the reduced communication cost similar across different Dirichlet's coefficients and device participation ratios. That is, Fedl. .MMA can be considered as an effective model aggregation scheme regardless of the degree of data heterogeneity.
7 CONCLUSION
We proposed a layer-wise model aggregation scheme that adaptively adjusts the model aggregation interval at run-time. Breaking the convention of aggregating the whole model parameters once, this novel model aggregation scheme introduces flexible communication strategy for scalable Fed- erated Learning. Furthermore, provide solid convergence guarantee of FedLAMA under the assumptions on the non-convex objective functions and the non-IID data distribution. Our empirical study also demonstrates the efficacy of
for scalable and accurate Federated Learning.

---

Under review as a conference paper at ICLR 2022
Harmonizing FedLAMA with other advanced optimizers, gradient compression, and low-rank ap- proximation methods is a promising future work.
10

---

Under review as a conference paper at ICLR 2022
8 CODE OF ETHICS
Our work does not deliver potentially harmful insights or conflicts of interests. We also do not find any potential inappropriate application privacy/security issues. The datasets used in study are all public benchmark datasets, and our source code will be opened once the paper is accepted.
REPRODUCIBILITY STATEMENT
The software versions, implementation details, hyper-parameter settings can found in the first two paragraphs of Section 6 The entire source code used in experiments will be published as an open source once the paper accepted. We believe one can exactly reproduce our experimental results following the provided descriptions.
REFERENCES
Alyazeed Albasyoni, Mher Safaryan, Laurent Condat, and Peter Richtárik. Optimal gradient com- pression for distributed and federated learning. arXiv preprint arXiv:2010.03226,, 2020.
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovie. Qsgd: Communication-eficientt sgd via gradient quantization and encoding. Advances in Neural In- formation Processing Systems, 30:1709-1722 2017.
Dan Alistarh, Torsten Hoefler, Mikael Johansson, Sarit Khirirat, Nikola Konstantinov, and Cédric Renggli. The convergence of sparsified gradient methods. arXiv preprint arXiv:1809.100. 2018.
Andrew Brock, Theodore Lim, James Ritchic, and Nick Weston. Freezcout: Accelerate training by progressively freezing layers. arXiv preprint
2017.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: benchmark for federated settings. arXiv preprint ar arXiv:1812.00099 2018
Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pp- 2921-2926. IEEE, 2017.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264, 2020.
Kelam Goutam, S Balasubramanian, Darshan Gera, and R Raghunatha Sarma. Layerout: Freezing
layers in deep neural networks. SN Computer Science, 165):1-9, 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint (eXiv:1706002671 2017.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical de- pendence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005.
Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, and Viveck R Cadambe. Lo- cal sgd with periodic averaging: Tighter analysis and adaptive synchronization. arXiv preprint arXiv: 1910. 13598, 2019.
Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Pra- neeth Vepakomma, Abhishek Singh, Hang Qiu, et al. Fedml: research library and benchmark for federated machine learning. arXiv preprint 2007 13518.. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Under review as a conference paper at ICLR 2022
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143, PMLR, 2020.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and
Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.054922 2016.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Adarsh Kumar, Arjun Balasubramanian, Shivaram Venkataraman, and Aditya Akella. Accelerat- ing deep learning inference via freezing. In 1Ith {USENIX} Workshop on Hot Topics in Cloud Compuling (HotCloud 19), 2019.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint arXiv: (rXxv:/8/2.06127 2018.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Communication-effce
Ari S Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. arXiv preprint arXiv: 2018
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector
canonical correlation analysis for deep learning dynamics and interpretability. arXiv preprint 1rXii::706.05806, 2017.
Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-cfficiii federated learning method with periodic averaging and quan- tization. In International Conference Artificial Intelligence and Statistics, pp. 2021-2031
PMLR, 2020.
Thijs Vogels, Sai Pranceth Karimireddy, and Martin Jaggi. Practical low-rank communication com-
pression in decentralized deep learning. In NeurIPS, 2020.
Hongyi Wang, Scott Sievert, Zachary Charles, Shengchao Liu, Stephen Wright, and Dimitris Pa- pailiopoulos. Atomo: Communication-effiienn learning via atomic sparsification. arXiv preprint
arXiv:1806.04090 04090. 2018.
Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Communicaiion-effcceentt models at no extra cost. arXiv preprint arXiv:2103.03:: 033903 2021.
Jianyu Wang and Gauri Joshi. Adaptive communication strategies to achieve the best error-runtime trade-off in local-update sgd. arXiv preprint arXiv: 180008313, 2018.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective in- consistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.07481, 2020.
Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communication- efficient distributed optimization. arXiv preprint 1rXiv:1710.09854 2017.
Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang. Yiran Chen, and Hai Li. Tern- grad: Ternary gradients to reduce communication in distributed deep learning. arXiv preprint 170500770 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: 1904.0096, 2019.
12

---

Under review as a conference paper at ICLR 2022
Sergey Zagoruyko and Nikos Komodakis. 1rXiv:1605.07146. 2016.
Wide
residual networks.
arXiv preprint
Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping. arXiv preprint arXiv:2010. 13369, 2020.
13

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1
CONVERGENCE ANALYSIS
Herein, we provide the proofs of the lemmas and theorem shown in Section 5
A.1.1 PRELIMINARIES
FedLAMA periodically chooses a few layers that will be less frequently synchronized. We call these layers Least Critical Layers (LCL) for short.
Notations All vectors in this paper are column vectors. E R denotes the parameters of one local model and is the number of workers. The stochastic gradient computed from single training data point & is denoted by g(x. E). For convenience, we use g(x) instead. The full batch gradient is denoted by (x()). We use || and || llap to denote 12 norm and matrix operator norm, respectively.
Objective Function In this paper, we consider federated optimization problems as follows.
m
min XERd
F(x) = Σ PxF
(10)
ne EEED fi(x, {) the local objective function of client i. is the global dataset size and n, the local dataset size, Note that, by definition, E Pi = 1.
where p; = nn/n h the ratio local data to the total dataset, and F((x) 1
Averaging Matrix- We define time-varying averaging matrix W E IR as follows.
P
if mod Tran is
W
J, & mod Tima is 0 otherwise
(11)
I.
I is identity matrix, P is also vime-narying averaging matrix, and J is full averaging matrix. diagonal matrix that has for the diagonal elements that correspond to the LCL
First, P is parameters and p; for all the other diagonal elements. Likewise, P is another d x diagonal matrix that has O for the diagonal elements that correspond to the LCL parameters and p; for all the other diagonal elements. Then, P is defined follows.
PPl for m diagonal blocks po, for all the other blocks
P
(12)
The jth block column of P consists of P! and Po¹ following the above definition.
Here present example of P where 2 and d 2. In this example, Po 1/3 and pl 2/3. Saying the LCL is the second parameter, P is defined as follows.
0 0
0
P
003
P
p
0
(13)
1
0
P
P P
0
P
D
0
0
(14)
0
The full-averaging matrix is defined follows. First, J, a x d diagonal matrix that has p: for the diagonal elements. Then, J consists of m blocks of J, such that each column block is of J, blocks. Here we present an example of J where m and as follows.
Jo
J =
(15)
14

---

Under review as a conference paper at ICLR 2022
0
Jo J1
(16)
Jo
J1
Sar
The averaging matrix P and have the following properties:
1. Plam = ddm, J1dm ddm-
2. The product any two averaging matrices consists only of diagonal block matrices because all the blocks in P and J are diagonal.
3. PJ JP J regardless of which layers are chosen the LCL.
4. PP P regardless of which layers are chosen as the LCL.
Vectorization - We define a vectorized form of m local model parameters X E Rdm, its stochastic gradients gi: Rdm and the full gradients fū E R as follows
X vec x x
xx
g: vec {6r (x (2 (x²), ... 9m (x(xx)))
(17)
f = vec (x VF2(x ... (Fm((
The full model aggregation can written using the vectorized form of local models x and the averaging matrix J as follows.
(1.1)
(2(1.1 + 2x ( 1/3
0
(1.2)
(1 (1 + 2x (.2))
Jx
0
(2.1)
=
(1.1) + 2x 2.1)
(18)
2.2)
(2 + 2x (.22) 2
(1.2)
Fk
(45) is the j model parameter of local model i at iteration k. where Ti
We also define the following additional vectorized forms of the weighted model parameters and gradients for convenience.
R vec (Vpix), VP2x2, P
g vec {VPigh (x (xt) (x))) PPmmm (xx% (xx)
(19)
f, = vec V V (2 Pm V Fm (xx }
Assumptions analyze the convergence rate of FedLAMA under the following assumptions.
1. (Smoothness). Each local objective function is L-smooth, that is, (x) VF(())) s
L|[x- - y/I, Vi E {1, (1,...,,,)) m...
2. (Unbiased Gradient). The stochastic gradient at each client is an unbiased estimator of the local full-batch gradient: Ex]9, {)] (x)
3. (Bounded Variance). The stochastic gradient at each client has bounded variance:
[lg, (x. 5) (x)) < oi), {1.-... mm, o² 2 0.
4. (Bounded Dissimilarity). For any sets of weights {p. 2 m Em
Pt there exist
If local objective functions are identical to each other, 8 and Σ p; V Fx
constants 3 > and n² > such that > [x)] < 8
(x)
R²
0.
15

---

Under review as a conference paper at ICLR 2022
A.1.2 PROOFS
Theorem 5.1. Suppose all local models are initialized to the same point ui- Under As- sumption 1 ~ 4, if FedLAMA runs for K iterations and the learning rate satisfies S
min
FedLAMA ensures
2(mmaz I)L: 2Tmome (Tmux 11(28841)
K Σ A=I
E
IVF((u))
S nk [E]) - -(()))) + 4n Σ Lo
K
+ +n/ (Tmex 1)220² + 6r/² Taan Tmar(Tmas 1)L???
(20)
where u, indicates local minimum.
Proof. Based on Lemma 5.1 and 5.2 we have
k
E
[VF(3))]
I n
(E [[((u))) - F(u)))) + +2
Σ
p Lo²
i=1
1p (Tmax - 1)03
A,2
L A) A))
A
I
Σ E
|VF
+
1
KL2(1
A)
After re-writing the left-hand side and a minor rearrangement, we have
[IVF())))
2
(F(u))
F(u.))) 21
pi T.?
nk
K
A,2
Σ
E
|
K
1- A
km
172 [2(Tmaz
1)0²
Ax L A)
L
1 A
By moving the third term on the right-hand side to the left-hand side, we have
1
A
E [[IIF(())11
s nk
2
(E (F(,)) F(u.))) + 2n Σ Lo
m
1 A
1/ (Tmaz 1)o²
Ax
+ 1
(21)
1 A
I - A)
If A <
then As I-d
1. Therefore,
21
can be simplified as follows.
28
V
F u |?
<
4 nk
(E (F(,)) - -F(,)))) + 4n
p To
(22)
K
1=1
17 (Tmax 1)02
An?
212
+ 2
1 1-A
1 A'
The learning rate condition
A 28*41 also ensures that
5 + I-A
Based on Assumption 4,
232
< 3.
and thus
I < 5. Therefore, we have
K
m
E
) < n
(E[F(ur)= - F(u.))) + 4n)
Σ
p Lo²
1=1
+ 31/² (Tmaz 1)L20² + Tmaz (Tmax - 1)L2
We complete the proof.
16

---

Under review as a conference paper at ICLR 2022
Learning Rate Constraints In Theorem 5.3 and the other from (51) as follows.
have two learning rate constraints, one from 22
from 22
2 +1
A A<1 1 from 51
After minor rearrangement, we have a unified learning rate constraint as follows.
7 S
min
2(Tmax - 1)LL LV 2Tmaz (Tmaz 1)(232 +1)
Lemma 5.1. (Framework) Under Assumption if the learning rate satisfies rj S Z2 FedLAMA ensures
K
E
VF(uu))1
< nK
E (F()
F(u.]] 2n Σ (pi)²
K
(23)
L
-| x=11
Pc
K
Proof. Based on Assumption 1, we have
278
12²L
E [F(())+1) - F(u))] -EE
(VF(us)
Ped (xx))
Pig (xx)
(24)
Ti
T2
First, Ti can be rewritten as follows.
m
m +E (VF(un).
E
F(u)).
Pe (x)( - v(((x))))
m (VF(uu), Σ pa V((())
E
1 2
[VF(u))]
+
Σ p. P.VE(()))
- VF(uu) - Σ
ps V (x(()))
1=1
(25)
where the last equality holds based basic equality: 2a T Then, T2 can be bounded as follows.
||a|12
||b|
|a b11
T2 = E
Σ pa (xx(x)) - -E [xx(x)) + Σ
Pr IE [e (xf)]
1=1
m
E
Pi (xx) v(((E)) + D
pe V (x++)
m
< 2
Σ
Ps (xx( VF((+))
+2E
p:
172
2
p [1:g(xt) vx((x))
2E
Σ p v (x+t)
[7]
< 2
P +2E
2
Σ Pr (x
(26)
17

---

Under review as a conference paper at ICLR 2022
where the last equality holds because g: (x})) and the last inequality follows Assumption
V (xt)) has 0 mean and is independent across
By plugging in (25) and 26 into
24 we have the following.
E [[F(uR+1) - F(un)] S -2
VF [
P4 V F
(x
+ !
E
VF(u
Σ
Pr V
+ + Lo
Σ p
2
1=1
+ 1/ L
Σ
ps V
m
7 VF(ux))) )|?
111 (11-2 E
Σ
ps V F(
2
m
m
7 2
E
VF(u:) > ((F(((
+ 1 Lo
If n S 2L- it follows
E [F(u++1) F(u))
|VF(uu)) + nLo Σn p
2
VF(ux) - - (x
+
E
2
2=1
1
m [VF(u)) 1| + noo 2
$
p
(27)
2
E VF.(u:)) V.F
1
La
s (FF((u))))
2| + 2
Σ
2
+
Σ
ps
E
Iu
x
2
2
1=1
where 27 holds based on the convexity of /2 norm and Jensen's inequality. By taking expectation and averaging across K iterations, have.
(F(u++1) F(uE))
K Σ
m
|[FF(u))] + nLo >
p
7
2K
K
E
Uu - -xxx x².1
2K
18

---

Under review as a conference paper at ICLR 2022
After minor rearrangement, we have telescoping follows.
K
2 nn
E
VFF()))
E [[F(u)) F(ux+1)] +2n Σ
K
1 K m
Ius - xt(1"] P.
K k=1
i=1
2 71K
E [F((u)) F(u))] 2n Σ p3
L K m
P | -
+
K
where u, indicates the local minimum. Here, we complete the proof.
Lemma 5.2. (Model Discrepancy) Under Assumption 4, if the learning rate satisfies nj
TmTma 1)L FedLAMA ensures
K K
21 - 1)02
An²
x x=(1"]
E
1- A
L - A)
k=
(28)
A
K
E
KT2(1
A)
k=1
where A (T) 122 and Tmaz is the largest averaging interval across all the layers.
Proof. We begin with rewriting the weighted average of the squared distance using the vectorized form of the local models as follows.
m
m
x
ll VI (u: xi)11
Pt
u
i=1
= -| - 1
(29)
[]( T) 2|
where 29 holds by the commutative property of multiplication.
Then, according the parameter update rule, we have
(J 1) (J [J I)W W
Wa-1)1g:-1
(30)
where (30 holds because JW J based the averaging matrix property 3, and IW W.
Then, expanding the expression X;-1, we have
1J (J 1) B I (W 7))=2)) (J W W
= (J - I)W W - (J W Wx-2) - (J W W
Repeating the same procedure for x 8x-3, ..., X2 we have
k
J - 1)&
& Σ
Wi)g.
=
k-1
4-1
= =-n
J
W.
(31)
[=
where 31 holds because xf is the same across all the workers and thus (J - 0.
19

---

Under review as a conference paper at ICLR 2022
Based on
31
we have
xx,1 x
Pi
K
[[ - 1))8s112)
E
K
K
K-
1
27
K
Σ
1/2 E
Σ
W 8.
t=
R-
R-1
k-1
k-1
1)
T
Σ
Wi) - f.)
Σ
J
K
[=s
2112 K
k-1 1 Σ
k-1
k-1
Σ
( P.))
E
Σ
JJ
WI).
T3
74
(32)
where (32 holds based on the convexity of f2 norm and Jensen's inequality. Now, focus
bounding T3 and T4, separately.
Bounding T3
w Wi) - f.)) [t)
Σ
I=s
1-1 (J II W.) - Pa))
ΣΣ E
(33)
k=
1=
(8. f.) 2
ΣΣ
E
(J II 1
W 1)
(34)
k=
where 33
holds because g. has 0 mean and independent across s, and 34 holds based on
Lemma A.
20

---

Under review as a conference paper at ICLR 2022
Without loss of generality, replace with 0Fmar + b. where a is the communication round index and is the iteration index within each communication round. Then, we have
K/ Tmaa
Tman b-1
(B. f.) 1
Σ
E
(J
W
a=0
b=1
s=1
[=s
K/Tmar
+0-1
b.)l"
(Bo
Wi)
a=0
1=1
op
K/Tmax" -1 OTmur +6-1
aTmax +b-1
Σ
E
(8.
ft)l
W
a=0
b=1 s=aTmurt
I=8
op
K/ K/Tmaz Tmaz uTmust +b
aTmax +b-1
f.) ||
E
(g.
W
(35)
a=0
K/ Tmaz=
Tiao
(8. f.)) (//1
(36)
a
K/Tman Tinn aTmax
Σ P (gx
-
K/ A/mma
Toarr aTman
Σ
2io²
(37)
a=0
TTmes
K/ Tmas
Trarr
K/Tmas
Tmaa (Tmax - 1) 0²
1)0²
Σ
2
K (Tmax 1)
(38)
Remember FedLAMA synchronizes the whole parameters at least once after every Tmar iterations.
Thus, 35 holds because J= m7ma W, is J when @Tmars and thus J aama W
becomes O. (36) holds based on Lemma A.2
37 holds based on Assumption 3.
Bounding Tı
21

---

Under review as a conference paper at ICLR 2022
K-Tmi Tmaz
E
E
EJ
w
Wijff
l=s
K/ K/Tmue
Time
aT ar+ Σ
+b-1
27
E
Wi)f,
am0
b=1
K/Tmas
Timax
+ b--1 Σ
aTmour +b-1
E
(J
P.)
(39)
a=0
b=
s=amman
Ims
K/ ///moo
aTmax +b-1
u-marttb-
(b 1b-1)
E
P.)f.
(40)
4=0
b=l
armma +1
K/Tmaz
Tmar
Wamar +b-1 Σ
2
at aTmax + b-1
(b-1) 1)
E
(J
P1)
(41)
a=0
s=dTmost
op
K/Tman- Tmar
a7mar
27
(b-1)
(42)
b=
K/ K/Tmar-
TTmmx Tmaa
Tman Tmaxx
Σ
2
a=0
K
Tmaa (Tmax
E
fe
2
k=1
Tiaa ( (Tmar
Pe
VF (x
(43)
2
where (39h holds because J
[ +0-1p
P, becomes O) when S aT. 40
holds based on the holds based on
convexity of f2 norm and Jensen's s inequality.
holds based Lemma 442
Lemma A.2
Final Result
By plugging in 38 and
43
into
32
we have
K
E |lue - xt [7]
Pi
K
k=
2n² K
K (Tmax - 1) 0²²+
Tmma (Tmax 1)
ps E (
2
k=
1=1
17² (Tman
1)o² 182 (Tmaz 1)
m
Pal |[E
(44)
K
The local gradient term the right- side in 44 can be rewritten using the following inequality.
[|(R(()))
EE (F v F( +
| (u || +2 [IVE(U)))]
< 2
(45)
< 21 B
|lue
- -x+
+ 2
(46)
22

---

Under review as a conference paper at ICLR 2022
where 45
holds based on the convexity of E2 norm and Jensen's inequality.
Plugging in 46
into
wc have
K
un -xxxl x
E
E
K
k=
==1
(Tmaz - 1)0 + 21 21fTmax (Tmaz - 1)1,
m
uus xx(1
Pi B
K
f=1
2n1² Tmau (Tmax
1)
K m
Pi
E
VFF(u))]l
(47)
K
After a minor rearranging, we have
K
a (Tmax - 1)o2
E un - x 11 S
K
2 (Tmaz 1)L2
211² Tmax (Tmax
m Pi E
[ue [²]
K(1 20/² Taaa (Tmax 1)
k=1i=1
(48)
Let us define A Tmax (Tima 1)1.2 Then 488 is simplified as follows.
xx|19
E
ukk
K
1=1
n² (Tmax 1)o²
A
E
V
(ux))3
KT2
A)
k=1
Based Assumption have
K m
us xt11"]
P:E E
K
k=1 i=1
1 (Tmar 1)02
A
K Σ EE
27
Ax²
+
(49)
1 1 A
KL - A)
L A)
172 (Tmaa 1)o²
A,32
K
AR?
E
|VF(
(50)
1-A A
KL2(1 - A)
L 1011-A11 A)
k=1
where 50 holds based on the definition of the objective function 10
Note that true only when rate constraint as follows.
49
A >>0 0. Thus, after minor rearrangement, have learning
< 2( Tmaa 1)L
(51)
Here, we complete the proof.
A.1.3 PROOF OF OTHER LEMMAS
Lemma A.1. Consider real matrix A E Bmddy xmd; and real vector b E Bmdd, Ifb + Omdy, we have
||b||| |||lbp|lb||
(52)
23

---

Under review as a conference paper at ICLR 2022
Proof.
||b|| | |
|Ab] 2
< || op |
(53)
where 53 holds based on the definition of operator norm.
L
Lemma A.2. Suppose an md x md averaging matrix P and the full-averaging matrix J, then
J Plpp = 1.
(54)
regardless of which layers are chosen as the LCL.
Proof. First, by the definition of averaging matrix P. all the columns that do not correspond to the LCL are zeroed out in P. Then, based on the averaging matrix property and the remaining columns in P has at all different rows. By the definition of J, all the non-zero elements in zth column are the same p;, i E {1, -1, m). Consequently, the remaining columns in J P are always orthogonal regardless of which layers are chosen as the LCL, and thus the eigenvalues of
P are either or 1. Finally, by the definition of the matrix operator norm, || - P
mar [|A[] P}} = I, where X(-) indicates the eigenvalues of the input matrix.
24

---

Under review as a conference paper at ICLR 2022
255
Teding X 3 PP)
2.5
1.0
2.2
2.0
Tad AM 4)
00
2.3
23
00
2.
1.
Fedhig
Feshee FOBLARA
Fodkig 112
Eyed
Earch
Epach
Epach
a) CIFAR-10 HDD setting learning curves
2.5
Fodlig XX Fedligg 1D1
2.-
14
Feding XI
L3-
Fedlig IS
MA 16
Fediag 112
2.5
Themppre
3.5
2.5
2003
4033
5000
2008
4003
5003
02
2006
4000
6906
2000
4000
6000
fondan
CIFAR-10 non-lID setting learning
Figure 4: The learning curves of CIFAR-I0 (ResNet20) training (128 clients). a): The curves for IID data distribution. b): The curves for non-llD data distribution (a 0.1). FedAvg (x) indicates FedAvg with the interval of x. FedLAMA (x, y/ indicates FedLAMA with the base interval of and the interval increase factor of As the aggregation interval increases, FedAvg rapidly loses the convergence speed, and results in achieving a lower validation accuracy within the fixed iteration budget. In contrast, FMAL effectively increases the aggregation interval while maintaining the convergence speed.
Fad (E
28
Fedlin (17)
3.3
Foll AMA 3.
Feskig
16.4
3.0
Fellog
22
3.2
Tw (74)
103
383
163
300
300
306
100
702
Excoh
Esich
Cocch
al CIFAR-100 setting learning curves
Feskyp
Tediig
ARA
Fadlig
A
Teskig 1241
FoBAAAA 13,
Fosling (4)
FocLAtta 4)
2033
2003
2000
60:0
03
3200
4300
32
1000
CIFAR-100 non-IID setting learning curves
Figure 5: The learning curves of 100 (WideResNer28-10) training (128 clients). a): The curves for IID data distribution. b): The curves for non-IID data distribution (a 0.1). FedAvg (x:) indicates FedAvg with the interval of æ. FedLAMA (x, y) indicates FedLAMA with the base interval of and the interval increase factor of y. While FedAvg significantly loses the convergence speed as the aggregation interval increases, FedLAMA has marginl impact on it which results in higher validation accuracy.
A.2 ADDITIONAL EXPERIMENTAL RESULTS
In this section, provide extra experimental results with extensive hyper-parameter settings. We commonly use 128 clients and local batch size of 32 in all the experiments. The gradual learning rate warmup (Goyal et al. (2017) 7)) is also applied the first 10 epochs in all the experiments. Overall, the learning curve charts and the validation accuracy tables deliver the key insight that FedLAMA achieves comparable convergence speed the periodic full aggregation with the base interval
25

---

Under review as a conference paper at ICLR 2022
Fookig Foctlg
FaLLMA BS-2
EMA (6.
28-
C.
24-
Fosheg 37
C4
Fodkug 16i (33
32
Tedhip (2))
Bustian
Figure 6: The learning curves of FEMNIST (CNN) training. FedAvg (x) indicates FedAvg with the interval of x. FedLAMA (r, y) indicates FedLAMA with the base interval of and the interval increase factor of y. FedLAMA curves are not strongly affected by the increased aggregation interval while FedAvg significantly loses the convergence speed well as the validation accuracy.
(r') while having the communication cost that is similar to the periodic full aggregation with the increased interval (ér').
Artificial Data Heterogeneity - For CIFAR-I0 and CIFAR- 100, we artificially generate the het- erogeneous data distribution using Dirichlet's distribution. The concentration coefficient is set to 0.1, 0.5, and 1.0 to evaluate the performance FedLAMA across a variety degree of data hetero- geneity. Note that the small concentration coefficient represents the highly heterogeneous numbers of local samples across clients as well as the balance of the samples across the labels. We used the data distribution source code provided by FedML. (He et al. (020
CIFAR-10 Figure 4 shows the full learning curves for IID and non-III CIFAR-10 datasets. The hyper-parameter settings correspond to Table 3 and [ First, the aggregation interval increases from to 24, FedAvg suffers from the slower convergence, and it results in achieving lower val- idation accuracy, regardless of the data distribution. In contrast, FedLAMA learning curves are marginally affected by the increased aggregation interval. Table and 7 show the CIFAR-10 classi- fication performance of FedLAMA across different & settings. As expected, the accuracy is reduced as Φ increases. The IID and non-llI data settings show the common trend. Depending the system network bandwidth, @ can tuned to an appropriate value. When = 2, the accuracy is almost the same can increase o and slightly increase the epoch budget to achieve a good accuracy. Table showss the CIFAR- 10 accuracy across different 7' settings. We see that the accuracy is significantly dropped as T' increases.
slightly higher than FedAvg accuracy. If the network bandwidth is limited, one
CIFAR-100 Figure 5 shows the learning curves for IID and non-IID CIFAR- datasets. Likely to CIFAR-10 results, FedAvg learning curves are strongly affected as the aggregation interval in- creases from 6 to 24 while FMdLAMA learning curves not strongly affected. Table 9 and 10 show the CIFAR-100 classification performance of FedLAMA across different & settings. Fed- LAMA achieves a comparable accuracy FedAvg with a short aggregation interval, even when the degree of data heterogeneity is extreamly high (25% device sampling and Direchlet's coefficient of 0.1). Table 11 shows the FedAvg accuracy with different r' settings. Under the strongly het- erogeneous data distributions, FedAvg with large aggregation interval (r 2 12) do not achieve reasonable accuracy.
Figure 6 shows the learning curves of CNN training. Likely to the previous two datasets, the periodic full aggregation suffers from the slower convergence as the aggregation terval increases. FedI.AMA learning curves are not much affected by the increased aggregation interval, and it results in achieving a higher validation accuracy after the same number iterations. Table 12 shows the FEMNIST classification performance of FedLAMA across different á settings. FedLAMA achieves a similar accuracy to the baseline (FedAvg with r' = 10) even when using a large interval increase factor & 4. These results demonstrate the effectiveness of the proposed layer-wise adaptive model aggregation method on the problems with heterogeneous data distribu- tions.
FEMNIST
26

---

Under review as a conference paper at ICLR 2022
Table 6:
(IID data) CIFAR-10 classification results of FedLAMA with different e settings.
clienes
Averaging
factor:
Validation
0.8
(FedAvg)
88.37 = 0.1%
0.04%
128
32
0.5
86.33 3: 0.27
85.08 = 0.04%
Table 7: (Non-IID data) CIFAR-10 10 classification results of FedLAMA with different settings.
clients
Local batch size
T.
Active
Dirichler coeff.
Validation
(FedAvgg)
90.79 = 0.1%
100%
89.01 1 0.04%
87 97.84-2 + 0.01%
(FedAvg)
90.53 1.18%
100%
0.5
2
89.21 1: 0.2%
80.68 12%
(FodAvg)
89. = 11%
100%
0.1
2
89.00 1 0.1%
84.82 0.08%
(FedAvg)
90.34 = 121%
50%
0.13%
87.48 + 0.21%
(FodAvg)
89. 86 13%
128
30%
0.5
88.44 + O. 15%
87.29 0.18%%
(FedAvg))
87.83 1: 0.27%
50%
0.1
2
87 - 177%
85.92 0.21%
(FodAvg)
88.97 = 0.03%
25%
87.89 = 0.22%
86.61 =: 0.1%
(FedAAgg)
87.59 1 0.05%
25%
87. 0.08%
80.37 = 0.02%
(FedAvg)
0.04%
0.3
25%
0.1
83.55 = 0.02%
83.06 0.03%
Table 8: (Non-IID data) CIFAR-10 classification results of FedAvg with different 7' settings.
clients
Local batch size
LR
Active ratio
Dirichlet coeff.
Validation acc.
6
(FedAvg)
89.52 0 0.11%
(FedAvg)
87.20
0.05%
24
(FodAvg)
84.82 = 0.1%
6
(FedAvg)
84.02 4: 0.1
0.3
12
20%
(FedAvg)
82.48 + 0.27
24
(FodAvg)
76.72 = 0.1%
Table (IID data) CIFAR- 100 classification results FedLAMA with different ò settings.
chients
Local butch size
LR
Averaging interval:
Interval incresse factor: &
Validation acc.
(FedAvg)
76.50 1: 0.02%
75.99 + 0.03%
128
32
0.6
76.17 0.2%
76.15 1 0.2%
27

---

Under review as a conference paper at ICLR 2022
Table 10: (Non-IID data) CIFAR- 100 classification results of FedLAMA with different & settings.
of clients
Local batch size
LR
Active
Dirihlet coeli.
Validation
((FodAvg)
80.34 0.01%
100%
2
78.92 + 0.01%
77.16 0.05%
0.4
(FedAvg))
80.19 0.02%
100%
0.5
2
78.88 = 0.1%
78
0.08%
(FedAvg)
79.78 1 0.02%
0.2
100%
0.1
79.07 0.02%
79.32 + 0.01%
(l'edAvg)
79.94 1. 0.1%
50%
78.98 0.0T%
4
77.50 = 0.02%
0.4
(FedAvg)
0.05%
128
32
6
30%
0.5
78.37 1 0.05%
1: 0.1%
(FedAvgg)
79.62 1: 0.06%
0.2
50%
0.1
2
78.76 1. 0.002%
77.44 0.02%
(FodAvg)
78.78 + 0.02%
0.4
25%
0.02%
0.2
4
76.84 1. 0.03%
(FedAvgg
78.81 0.01%
25%
77.86 0,04%
0.4
77.01 + 0.1%
(FodAvg)
79.06 0.03%
25%
0.1
2
78 63 + 0.02%
0.00%
Table 11: (Non- data) CIFAR-100 classification results of FedAvg with different 7' settings.
# of chients
Local hatch size
LR
Active ralio
Dirichlet cnelf.
Validation acc.
6
(FedAvg)
79.78 0.02%
32
0.4
12
100%
0.1
(FodAvg)
77.71 # 0.1%
24
(TedAvy))
69.63 1 0.1%
6
(FodAvg)
79.06 4 0.03%
(TeddAgg)
0.05%
24
(FedAvg)
67.43 0.1%%
Table 12: FEMNIST classification results of FedLAMA with different & settings.
# of clients
Local batch size
LR
Averaging interval: 7
Active ratio
Interval increase factor:
Validation acc.
(FedAvg)
0.21%
86.40 1 0.13%
100%
54.67 0.1%
84. + 0.18%
(FedAAg))
56.59 1 0.2%
56.17 0.1%
128
32
0.04
12
50%
85.77 = 0.15%
0.03%
(PedAvg)
86.00 = 0.27%
86.01 1%
25%
85.62 + 0.08%
85.23 1 0.1%
28


Reviews:
[{"review_id": "VvxJuVMZBDb", "reviewer": "Reviewer_7ks6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. The problem studied is important and the idea is novel. \n2. This paper provides the convergence rate.\n3. This paper conducts extensive experiments.\n\nCons:\n1. There are some errors in theoretical analysis. Eq.(15) is not correct. Eq.(25) is also wrong. \n\n2. For Lemma 5.2, there should be a constraint for $\\eta$.\n\n3. The convergence upper bound is worse than standard FedAvg. In Remark 1, when $K=O(m^3)$, FedLAMA can achieve linear speedup. However, in [1], $K=O(m)$. Thus, this bound is much worse than [1]\n\n\n[1] On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization", "summary_of_the_paper": "This paper developed an adaptive aggregation method for federated learning. The theoretical analysis shows how the interval affects the convergence rate. The experiments show that it can reduce the communication cost. ", "summary_of_the_review": "This paper proposed a new idea for reducing communication costs. But the theoretical analysis has some flaws. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A1m3SXr27Gu", "reviewer": "Reviewer_9DWJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "They provide a communication-efficient federated optimization and relax the aggregation frequency by quantifying the model discrepancy between local models and global models and adaptively adjusting the aggregation interval in a layer-wise manner.\n\nSome questions are the following.\n\n1）The baseline is FedAvg, too few. Why not compare with other the periodic full aggregation scheme (e.g., FedProx, FedNova, SCAFFOLD, etc.), even some synchronized schemes? the synchronized schemes refer to,\n\n[1] Chen, Yang, Xiaoyan Sun, and Yaochu Jin. \"Communication-efficient federated deep learning with\nlayerwise asynchronous model update and temporally weighted aggregation.\" IEEE transactions on neural\nnetworks and learning systems 31.10 (2019): 4229-4238.\n\n[2] Xie, C., Koyejo, S., & Gupta, I. (2019). Asynchronous federated optimization. arXiv preprint\narXiv:1903.03934. \n\n2）FedLAMA seems to lead to some accuracy drop as the experiments show. They do not show the total time in terms of communication cost, just show the reduced ratio for communication. It is not clear about absolute earnings with sacrificing the accuracy. In some cases, especially in industry, the model's accuracy matters more than training time (e.g., the total training time is within several hours.)  ", "summary_of_the_paper": "In this paper, the authors propose a layer-wise model aggregation scheme in federated learning cases to reduce the communication cost. Specifically, they quantified the model discrepancy between local models and global models and adaptively adjusted the aggregation interval in a layer-wise manner. By increasing the aggregation intervals and relaxing the aggregation frequency, the method can reduce the communication cost in federated learning cases. The experimental results show it can reduce the communication cost for IID data and non-IID data compared to FedAvg.", "summary_of_the_review": "Thanks to the authors for their efforts for their theoretical analysis and experimentally proof of the proposed federated optimization scheme. The experimental results are a little weak to show the effectiveness and the practicality. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2uIz0sAQ3Dh", "reviewer": "Reviewer_YGZf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. It is an interesting idea to adjust the aggregation interval for layer-wise aggregation.\n2. Detailed theoretical analysis and experiments are conducted.\n3. The paper is well-written.\n\n\nWeaknesses:\n\n1. It is uncertain that the proposed method can really reduce communication costs. Due to the reduced frequency of aggregation and adopt layer-wise partial model aggregation strategy in each round, it is very likely that the proposed method will decrease the convergence speed of the global model. Therefore, the proposed method needs more communication rounds to achieve a convergence state, thus the total communication costs are still very high. Moreover, at each communication round, the client still needs to download the full model from the server. Therefore, the downloading communication costs haven't been improved. \n2. In the convergence analysis, Assumption 4 is a little tricky, and helps to avoid much detailed and further analysis. The convergence rate is O(1/m^½ K^½), which is loose compared to some convergence analysis of FedAvg, such as ON THE CONVERGENCE OF FEDAVG ON NON-IID DATA (https://arxiv.org/abs/1907.02189).\n3. The experiment is insufficient to support their claims. For example, the only baseline is FedAvg, and it is necessary to choose some compression methods as a baseline. Moreover, the communication cost should consider the number of communication rounds to achieve a convergence state.\n\n\nQuestions and suggestions:\n\n1. Algorithm 1 should explicitly state which steps are executed on the server or client. For example, line 3 should be on the client, and lines from 4 to 10 are on the server. Moreover, it should explicitly highlight when the communication occurs.\n2. It is necessary to draw the convergence curves of the proposed methods and baselines.\n3. It is unclear how to construct non-IID scenarios in the experiment setting.\n4. There are some federated learning methods that only update parts of its model. This paper should also consider these baseline methods. \n\nhttp://proceedings.mlr.press/v139/collins21a/collins21a.pdf\n\nhttps://arxiv.org/abs/2102.07623", "summary_of_the_paper": "This paper proposes an adaptive interval schema for layer-wise model aggregation in federated settings. The aim of the proposed method is to reduce communication costs by adaptively decreasing the frequency of layer-wise aggregation with consideration of model discrepancy. ", "summary_of_the_review": "This paper proposes an interesting layer-wise model aggregation method to reduce communication cost while trying to keep the same performance with FedAvg. However, the proposed method’s design, theoretical analysis and experiments cannot support its claim.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "17smUPuVjbw", "reviewer": "Reviewer_graH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Effectively reducing the communication overhead is an important problem for federated learning, with the recent advance of layer-wise model freezing training, this paper manages to extend this idea for network traffic reduction in federated learning. Concretely, I list the main strengths and minor weakness of this paper as below:\n\nAdvantages:\n\n+ To leverage layer-wise model freezing for federated learning is a simple but novel idea. According to the empirical study, this method seems to be effective and efficient. \n+ The paper also includes tight convergence analysis for the proposed method. \n\nDisadvantages:\n+ The communication cost calculated in Section 6 is a little sloppy, in practices, there are plenty of system advances managing to hide the communication overhead within the computation slots, e.g. PyTorch-DDP (https://arxiv.org/abs/2006.15704). In this fashion, the actual runtime may not be simply linear to the amount of network traffic. Perhaps, a more sophisticated cost model would consider both the computation power of the edge device and network condition during training, then model the cost including the mechanism of overlapping communication and computation to estimate the actual runtime. \n\n", "summary_of_the_paper": "This paper proposes a novel adaptive model aggregating schema under a federated learning setting, where it effective reduces the communication overhead without significantly sacrificing the generalization performance. Both empirical study and theoretical analysis are included to justify the effectiveness and efficiency of the proposed method.", "summary_of_the_review": "The idea of utilizing layer-wise model freezing for federated learning is an interesting idea, the paper includes reasonable empirical study and convergence analysis to illustrate the effectiveness of this proposed approach. \nAlthough some part of the cost model in the simulation is a little sloppy, I tend to advocate the acceptance of this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a layer-wise adaptive aggregation method for federated learning that seeks to reduce the communication cost. The frequency of aggregation is adjusted separately for each layer of the model that is being trained. The number of iterations $\tau$ after which each layer's parameters are averaged across clients is multiplied by a factor $\phi$ depending on the magnitude of changes to the parameters at each layer. The paper gives a convergence analysis of the proposed method and provides experimental results to demonstrate its effectiveness in reducing communication without compromising accuracy.  

Reviewer 7ks6 found some errors in the convergence analysis that were fixed by the authors in the discussion period. Reviewer YGZf increased their score to 5 after the discussion with the authors. The reviewers also gave the following suggestions to improve the paper:
1) Showing convergence curves to demonstrate that the communication reduction does not come at the cost of a slowdown in convergence.
2) The results presented in the paper shows only a small communication reduction for many of the layers. Perhaps the strategy can be improved in order to boost the communication reduction.
2) Use a more realistic model to calculate the communication cost so as to account for network delays and other costs rather than just considering the number of parameters communicated.

The scores are split on this paper. While one of the reviewers recommends acceptance, three others say that the paper is below the acceptance threshold. So I recommend a rejection while noting that the paper is close to the borderline.
Decision: Reject