AQUILA: Communication Efficient Federated Learning with Adaptive Quantization of Lazily-Aggregated Gradients
Authors: Zihao Zhao
Yuzhu Mao
Muhammad Zeeshan
Yang Liu
Tian Lan
Wenbo Ding
Venue: ICLR 2022 Submitted (2022)
Keywords: Federated Learning
communication efficiency
adaptive quantization

Abstract: The development and deployment of federated learning (FL) have been bottlenecked by the heavy communication overheads of high-dimensional models between the distributed client nodes and the central server. To achieve better error-communication tradeoffs, recent efforts have been made to either adaptively reduce the communication frequency by skipping unimportant updates, a.k.a. lazily-aggregated quantization (LAQ), or adjust the quantization bits for each communication. In this paper, we propose a unifying communication efficient framework for FL based on adaptive quantization of lazily-aggregated gradients (AQUILA), which adaptively adjusts two mutually-dependent factors, the communication frequency and the quantization level, in a synergistic way. Specifically, we start from a careful investigation on the classical LAQ scheme and formulate AQUILA as an optimization problem where the optimal quantization level per communication is selected by minimizing the gradient loss caused by updates skipping. Meanwhile, we adjust the LAQ strategy to better fit the novel quantization criterion and thus keep the communication frequency at an appropriate level. The effectiveness and convergence of the proposed AQUILA framework are theoretically verified. The experimental results demonstrate that AQUILA can reduce around 50% of overall transmitted bits compared to existing methods while achieving the same level of model accuracy in a number of non-homogeneous FL scenarios, including Non-IID data distribution and heterogeneous model architecture. The proposed AQUILA is highly adaptive and compatible to existing FL settings.

Under review as a conference paper at ICLR 2022
AQUILA: COMMUNICATION EFFICIENT FEDER- ATED LEARNING WITH ADAPTIVE QUANTIZATION OF LAZILY-AGGREGATER GRADIENTS
Anonymous authors Paper under double-blee blind review
ABSTRACT
The development and deployment of federated learning (FL) have been bottle- necked by the heavy communication overheads high-dimensional models be- tween the distributed client nodes and the central server. To achieve better error- communication tradeoffs, recent efforts have been made to either adaptively re- duce the communication frequency by skipping unimportant updates, a.k.a. lazily- aggregated quantization (LAQ),, or adjust the quantization bits for each communi- cation. In this paper, we propose unifying communication efficient framework for FL based on adaptive quantization of lazily-aggregated gradients (AQUILA), which adaptively adjusts two mutually-dependerr factors, the communication fre- quency and the quantization level, in synergistic way. Specifically, we start from careful investigation on the classical LAQ scheme and formulate AQUILA as an optimization problem where the optimal quantization level per communication is selected by minimizing the gradient loss caused by updates skipping. Mean- while, we adjust the LAQ strategy to better fit the novel quantization criterion and thus keep the communication frequency at an appropriate level. The effee- tiveness and convergence of the proposed AQUILA framework are theoretically verified. The experimental results demonstrate that AQUILA can reduce around 50% of overall transmitted bits compared to existing methods while achieving the same level of model accuracy in number of non-homogeneous FL scenarios, including Non-IID data distribution and heterogeneous model architecture. The proposed AQUILA is highly adaptive and compatible to existing FL settings.
INTRODUCTION
With the deployment of ubiquitous sensing and computing devices, the Internet of things (IoT) as well as many other distributed systems have gradually grown from concept to reality, bringing dra- matic convenience to people's 's daily life etu al. 2020 Liu et al. 2020a| Hard et al. 2018). To fully utilize such distributed computing resources, distributed learning provides a promising framework that achieve comparable performance with the traditional centralized learning scheme. How- ever, the privacy and security of sensitive data during the updating and transmitting processes dis- tributed learning have been a growing concern. In this context, fentext,eederated learning (FL) (McMahan et al. 2017: Yang et al.) 2019)) has been developed, enabling distributed devices to collaboratively learn global model without privacy leakage by keeping private data sets isolated and masking transmitted information with secure approaches like differential privacy (Abadi et al. 2016). secret sharing techniques (Bonawitz et al., (017)) and homomorphic encryption (Liu al. 20206). Due to its privacy and security preserving property and great potentials in some distributed but privacy- sensitive fields like finance and health, FL. has attracted tremendous attentions from both academia and industry in recent years.
Unfortunately, in many FL applications like image classification and objective recognition, the mod- els to be trained tend to be high-dimensional, which lead to heavy communication overheads, for example, a Resnet-152 network has over 58 million parameters (He et al. 2016). Hence, com- munication efficiency has become one of the key bottlenecks of FL. To this end, recent researches have tried to reduce the communication frequency. for example, Sun et al.] (2020) proposes the lgglly-aggee quantization (LAQ) method to reduce communication rounds by skipping some

---

Under review as a conference paper at ICLR 2022
unnecessary parameter uploads. To further reduce transmitted bits per communication, LAQ can be used jointly with gradient compression techniques, e.g. quantization and sparsification (Strom 2013 Wangni et al., 2018: Lin et al., 2018 Han et al. 2020). Moreover, MMo et al. (021)) devel- ops the Adaptive Quantized Gradient (AQG) for LAQ to adjust the quantization bit among multiple given levels during training. However, the AQG is not sufficiently adaptive, for example, in the two-level AQG with 4 bit and 2 bit, the situation of 3 bit and 1 bit is not covered at all. In a sepa- rate line of work, Jhunjhunwala for FL which can search in a given range for an optimal quantization level and achieve a better error- croorcommmuication tradeoff.
al. (202T) develops an adaptive quantization rule (AdaQuantFL)
Previous work has investigated how to optimize communication frequency or adjust quantization levels in a highly adaptive fashion, but not both. Intuitively, we ask the question, can we adaptively adjust the quantization level in LAQ to further reduce communication rouds and transmitted bits simultaneously? A straightforward approach is train LAQ jointly with state-of-the-ar adaptive quantization methods like AdaQuantFL. However, quantization level and communication frequency are mutually dependent in establishing the model convergence and must be adjusted together. For example, lazy aggregation leads to the skip of some gradients that no longer require quantization, while the choice of quantization levels directly affects the quality of a gradient update and thus whether it is selected for transmission and aggregation. Therefore, the key question of our work is how to jointly leverage these two complementary yet mutually-dependent degrees of freedom for further optimizing communication efficiency in FL.
The key idea of this paper is to select the optimal quantization bits for each communication round in LAQ by optimizing the gradient loss caused by skipping quantized updates, which gives novel quantization criterion that can cooperate with LAQ strategy to further reduce overall transmitted bits while maintaining the desired convergence properties of LAQ. The contributions of this paper are trifold and summarized as follows. 1) We propose FL framework with adaptive quantization of
gradients termed AQUILA, which simultaneously adjusts the communication frequency and quantization level in a synergistic fashion. 2) We formulate AQUILA LA as an optimiza- tion problem and develop an upperbound for the gradient loss caused by communication skipping, which gives a novel adaptive quantization criterion which is theoretically proven to be more efficient compared to AdaQuantFL. while maintaining the same convergence properties. 3) We experimen- tally evaluate the performance of AQUILA in number of non-hooogeneous FL settings, including Non-IID data distribution and various heterogeneous model architecture. The experimental results show that AQUILA can significantly mitigate the communication overhead over number of base- lines including fixed-bit LAQ and the naive combination of LAQ and AdaQuantFL. Our approach is highly adaptive and compatible to existing FL settings.
BACKGROUND AND RELATED WORK
Consider a FL system with one central server and a clients set M of M distributed clients to collab- oratively train global model parameterized by 0". At iteration k each client E MI first trains the global model 0³ on its local data D and sends the local gradient g V fm (D i 0²) to the central server. Then the server aggregates the parameters and updates the global parameter by:
g++1 0
Σ ghm MEM
M
To reduce communication overheads with gradient quantization, the stochastic uniform quantizer Alistarh et al. 2017) is usually adopted. For any local gradient g E Rd, the quantized value of its t-th dimension with quantization level b is defined as:
Qo (g:) ||g||2 sign (9:) (g, b),
where bi(g,b) is a random variable defined as follows. Let E 6-11--11 1} be an integer satisfying gil //lg ll E [1/b, (1 + 1)/b). then:
(e+ 1)/b e/b
with probability (b. |1 Hgx1)//lll) -1 otherwise.
Elg. b)
(3)
2

---

Under review as a conference paper at ICLR 2022
It is clear that with quantization level b, the number of bits for transmitting quantized gradient from a client to the central server is C. = [log (b + 1)] + d 32, with 32 bits for ||g 2, 1 bit for each sign (9:), and log2 (b + 1) bits for each (. b).
For communication rounds reduction, the LAQ proposes to let the client m E M upload its newly- quantized local gradient Qr(gm)) at iteration &s only when the change local gradient is sufficiently large, i.e.,
Ed= Ed laht1-a 0K
12
Q - 1
3(1Ea 1) (=o(@()))))) (4)
a²M²
where Qu(Om ( is the last quantized upload from client m, (=("" and Eb (m denote quantiza- tion errors, and {Ea)dm) d= are some predetermined constant weights. Notice here fixed quantization level b is used. In LAQ, if the difference between client m's newly-quantized local gradient (g and the last upload is smaller than threshold involving quantization errors and global model's in- novation, client m will skip the upload of Qo(gm)) at iteration and the central server will reuses (u (--1) for such lazy aggregation:
k++ 0²
a
Σ MEM
Qolam) = 0
Σ MEMMM
Qi(g
a M
Σ Qo(a MEM
(5)
M
M
where Mi denotes the subset of clients that skip the new gradient update and reuse the old quantized gradient at iteration Besides, 9m represents the actual gradient for aggregation from client m, which is 9m for m E MI \\ My. while g'm. for m E Mt..
Recently, AdaQuantFL is proposed to achieve a better error-communication tradeoff by adaptively adjusting the quantization levels during FL training (Jhunjhunwala et al., (021). Specifically, AdaQuantFL computes iteration k's optimal quantization level bj based on the following criterion involving training loss and initial quantization level bp:
b
V (0") (8t) bo.
(6)
where s(0") and s(0") are the global training loss at iteration 0 and k. respectively,
However, AdaQuantFI. transmits quantized gradients at every iteration. In order skip unneces- sary communication rounds and adaptively adjust quantization level for each communication jointl,, an naive approach is to quantize lazily aggregated gradients with AdaQuantFL. Nevertheless, it fails to achieve efficient communication due to a number of reasons. Firstly, given the descending trend of training loss, AdaQuantFL's criterion 6 may lead to high quantization bit number even exceeding 32 late in the training process, which is too much for cases where the global convergence is already approaching. Secondly, higher quantization level results in smaller quantization error, which will lead to lower communication threshold in LAQ's criterion (4) and thus higher frequency of trans- mission. Therefore it is desirable to develop more efficient adaptive quantization method in the aggrly-aggee setting to systematically improve communication efficiency in FL.
METHOD
3.1
ADAPTIVE QUANTIZATION LAZILY GRADIENTS (AQUILA)
Given the above limitations of the naive joint use of existing adaptive quantization criterion and lazy aggregation strategy, this paper aims to design a unifying framework for communication efficiency optimization where the quantization level and communication frequency are adjusted in synergistic and interactive way. Based on a careful investigation on LAQ, we design a novel quantization criterion where the optimal quantization level is selected by minimizing the expected gradient loss caused by skipping quantized updates. The rationale behind such strategy is, by formulating the adaptive quantization problem of lazily-aggregated gradients optimizing the expected gradient loss with respect to the number of quantization bits, we can get an adaptive quantization criterion based local gradient updates while maintaining
improving the convergence property of
3

---

Under review as a conference paper at ICLR 2022
LAQ. To get the optimization target, first derive upperbound for the expected gradient loss in terms of quantization bits as elaborated in Section 3.2 which gives novel adaptive quantization criterion ( that selects the quantization level for client iteration & based on initial quantization level bo and the change between client m's newly computed gradient 9th and the last uploaded gradient gm!
(6
bu Ilgh. - 9 gt. - 9m kh-1
The superiority of 7 comes from the following two aspects. Firstly, the gradient updates tend to fluctuate along with the training process instead of keeping descending like the loss value, and thus prevent the quantization level from increasing tremendously compared with the initial level. Secondly, with lazy aggregation criterion based on gradient updates like (4), the transmitted bits
in AQUILA is further controlled, since the gradient update for actual transmission in 77) is lower bounded by the lazy aggregation criterion, and therefore high-bit transmission for small update is more likely to be skipped.
To better fit the larger quantization errors induced by fewer quantization bits in (7). we modify the communication criterion follows to avoid the potential expansion of clients group to be skipped:
e (9th)) -Qi-- Ea-rSa
ohtila 0³
+2 Ebt. -
(8)
a2M
where all the notations are the same as in except the heterogeneous quantization level 6* and 8 1e-1 for each client. For detailed development of please refer to the Appendix. m
Server
(ulsf) suppress trassiisiion with high 65,
AQUILA
AdaQuantFL LAQ
Quastization selection
at
Lary aggrepation strategs
AdaQuanFI. & (rieh/100
LAQ
minimizz gradient loss caused by skipping
Clack
Claar:2
The schematic illustration of the communication efficient FL with AQUILA in comparison with the naive combination of AdaQuantFL. and L.Q.. The blue lines indicating the transmission of quantized gradients in AQUILA are drawn in different thicknesses to represent different quantization levels selected by various clients.
Figure 1:
The cooperation of the novel adaptive quantization criterion and the modified lazy aggrega- tion strategy (8) is illustrated Fig. la. Compared to the naive combination AdaQuantFL and LAQ where the mutual influence between adaptive quantization and lazy aggregation has not been considered as shown in Fig. 1b, our AQUILA framework adaptively optimizes the allocation of quantization bits throughout training to promote the convergence of lazy aggregation, and at the same time utilizes the lazy aggregation strategy to improve the efficiency of adaptive quantization by skipping high-bit transmission. The proposed AQUILA's effect of suppressing the transmission of high quantization bits has been verified in our experiments, as shown in Fig, 20 in the Appendix. Besides, with the adjusted lazy aggregation strategy (8, AQUILA well addresses the problem of high communication frequency in the late training process of naive combination of AdaQuantFI. and LAQ, as indicated by Fig.
The proposed AQUILA is summarized as follows in Algorithm 1. At iteration & = 0, all clients are forced to transmit local gradients quantized with the initial level by. At iteration k E {1,2,... K), the server first broadcasts the global model g² to all clients. Each client m computes g with local training data, and then uses it to select optimal quantization level by 7). Then, each client
4

---

Under review as a conference paper at ICLR 2022
computes its gradient update after quantization and determines whether to upload the update not based the communication criterion (8). Finally, the server updates the new global model 0²+1 with up-to-date quantized gradients Qba. (gm) for those clients who send the uploads at iteration k while reusing the old quantized gradients Qbs. (0th for those who skip the uploads.
(a)
Homo-MNIST-ID
(b)
Homo CIFARTO Non
AQUILA AdaQuantFL+LAQ
AQU AdaQuantFL+LAG
Number of Unskipped Clients
12
Number of Unskipped Clients
12
10:
10
VWV
MMM
MMM
20
40
60
80 100
20
40
60
80
100
Steps
Steps
Figure 2: Comparison of AQUILA and AdaQuantFL+LAQ on the number of unskipped clients per step in three experiment settings with homogeneous model architecture. AdaQuantFL. clashes with the threshold condition LAQ and results in high communication frequency due to the increasing quantization level late in the training process, but AQUILA keeps the communication frequency an appropriate level throughout the training.
Algorithm 1 Communication Efficient FL with AQUILA
Input: the number of communication rounds K, the learning rate the maximum communication level bmaz
Initialize: the initial global model parameter 8 and the initial quantization level by-
1: Server broadcasts 6" to all clients.
2: for each client m E MI in parallel do
3: Calculates local gradient 9 and sends the quantized gradient Qb (g
4: Set 9h Sh and 6%
both sides.
5: end for
6: for k 2. do
7: Server broadcasts 0 all clients.
8: for each client m M. in parallel do
9: Calculates local gradient 9m
10:
Computes the optimal local quantization level bit by
11:
if m 2 bmaz then
12:
b .maz.
end if
14:
if (8)
holds for client m then
Client mn computes and sends the quantized gradient Qoe (g
16:
Set gt. = gt and 6 byt on both sides.
17:
else
18:
Client sends nothing.
Set gth gh-1 and j% = 8k-1 on both sides.
20:
end if
end for
22: Server updates +++ by ot - a Σ m=
Qia. (g
end for
5

---

Under review as a conference paper at ICLR 2022
3.2 THEORETICAL DERIVATION AND ANALYSIS OF AQUILA
As mentioned before, in this work we bound the expected gradient loss caused by skipping updates with respect to quantization bits. Specifically, if the communication criterion (8) holds for client m at iteration k, it does not contribute to iteration k's gradient loss, otherwise, the loss caused by client m will be minimized with the optimal quantization criterion (7). In this sec- tion, the theoretical derivation of the target upper bound is based on following standard assumptions:
Assumption 1. Loss function f(0) = = MEM (0) is L-smooth, and (0) is Lm smooth.
Assumption 2. The quantization operation is unbiased with B[Qo(w)lu = and its variance satisfy: Vu E _d (((lQ())(( wll2 2 |w] S qb |w where 96 is positive constant.
Here adopt the definition of B in AdaQuan,FL, but modified it as follows:
Definition 1 (Number Bits sent from client m to the server, Bm). The total number of bits that has been sení from client to the central server unfil given time instant is denoted by Bm-
With the definition of Bm, the upper bound of the loss induced by gradient skipping is indicated by the following theorem:
Theorem 1. Under Assumption 1 and 2, the expected gradient loss caused by lazy aggregation has the following upper bound:
g
3a
dmd 1
Im
-mm
k
M
Bm
MEM
3a²
do²
do
3a
dm (d
32)
+
gm
9m
M
1)2
(b%
M
Bm
MEM
MEM
where o² denotes the maximum |! and dm is positive constant determined by Lm See the Appendix for proof details.
For each client m, the optimal by, is selected by setting the first derivative of the upper bound (9) to zero:
(0 - 20 (2 (2o2 (2)) (dm g
k-1
2
(10)
Therefore, an adaptive quantization criterion which does not rely on unknown parameters such as and dm can be achieved through dividing bt by bi, which gives the adaptive selection criterion in 7 if 6/, for each client m is chosen as the initial quantization level bo.
Comparison with LdaQuantFL. In AdaQuantFL, the optimal quantization level is selected by minimizing the upper bound of the expected squared norm of the gradient f(0)) of a non-convex objective function, and thus the convergence of AdaQuantFL is guaranteed. However, the proposed AQUILA proven be more efficient with sufficiently small learning rates as shown in Theorem Besides, is theoretically guaranteed that the squared norm of the global gradient V f( is also able to converge under AQUILA' adaptive quantization criterion developed for lazy aggregation.
Theorem 2. Under Assumption and 2, if the learning rate satisfies S (1/ML) / the following inequality holds:
g
Q2 10 001 25[ - )[((())-((('))
- f(0)) f(0")
(bth) bo
bu
- bu f(O)) - f(0))
(11)
gt.
m
x-1
where & and P are both positive constants. See the Appendix for proof details.
Theorem indicates that with appropriate learning rate, the proposed AQUILA uses less number of bits per communication compared to AdaQuantFL..

---

Under review as a conference paper at ICLR 2022
Convergence analysis. The Lyapunov function of AQUILA is defined in the same way LAQ and AQG (Mao et al. 2021)::
D @ktlld -0h-ld
((0))=1(00)) -(00))(00-1(0) +
(12)
where 0" is the optimal solution of f())
Convergence guarantee of federated learning with lazy aggregation has been well discussed in Sun
et al. (2020). More specifically, both the objective residual f(Ø") f(0") and the parameter dif- ferences term in Lyapunov function are guaranteed to descend along with the training process. Therefore, the squared norm of the gradient V f((*) is also guaranteed to converge based on the L-smoothness assumption which results in: V > = 2L(f(0") - f())))
4 EXPERIMENTS AND DISCUSSION
4.1 EXPERIMENT SETUP
Dataset. In this paper, evaluate method with MNIST and CIFARIO dataset, considering both IID and Non-IID Lon-IID data distribution. To simulate Non-IID situation, each client is assigned with two classes of data at most and the amount of data for each class is balanced.
Parameters. We set total client number M = 10, and follow the settings in LAQ, where D = 10 and E E
&D 0.8/D. D. In terms of initial quantization level, low level by = is chosen for simpler tasks with MNIST, and a larger level bo = 6 is selected for more complex tasks with CIFARIO. For both AdaQuantFL and AQUILA, we set an upperbound for quantization level as 6maz = 16 in case that the level grows too high.
Training. We train a CNN with MNIST and Resnet18 network with CIFARIO. The hyperparam- eters of our experiments are shown in Table iin Appendix.
We first evaluate our proposed AQUILA with homogeneous settings where all the local models share the same architecture as the global model. The performance of AQUILA is compared with several state-of-thh-aa methods including FedAvg (McMahan et al.) (017)) QSGD (Alistarh et al.) (017). AdaQuantFL, fixed-bit LAQ and the naive combination of AdaQuantFL with LAQ. For the choice of quantization level for fixed-bit LAQ, since AQUILA's initial level bo is 2 for MNIST and 6 for CIFARIO 10, we compare AQUILA with LAQ-2 and LAQ-6 for MNIST and CIFAR 10 respectively. Besides, the performance of LAQ with the upperbound bmaz = 16 is also evaluated.
We also evaluate our proposed AQUILA with HeteroFL (Diao et al. 2020), where the local models trained at clients' side are heterogeneous. Assume the global model at iteration A: is 0g and its size dg hg. then the local model of each client m can be selected by Ø* 8% [: dm, : hm], where gg
dm dy and hm hy respectively. In this paper, we choose three various model complexity levels {a, b, c) = [1, 0.5, 0.25].
Fig. shows the training loss lotal transmitted bits curve of experiments with IID MNIST / CI- FARTO for homogeneous model architecture, IID MNIST/ CIFARIO for 100%-50% heterogeneous model architecture, and Non-IID MNIST CIFAR1O for 100%-255; heterogeneous model architec- ture. The corresponding transmitted bits vs steps curves of the above experiment settings are shown in Fig. 4 which represents how many bits transmitted in each step. Methods without adaptive quantization and lazy aggregation like FedAvg and QSGD are not included in Fig, 4 for simplicity. All the other experiment results are provided in Appendix, with Fig. to Fig. 9 for homogeneous models, and Fig. 10 to Fig. 19 for heterogeneous models.
4.2 PERFORMANCE ANALYSIS
In this part, we analyze the performance of AQUILA with various experiment settings including Non-III IID data distribution and heterogeneous model architecture. From figures in this paper, we can observe that:
7

---

Under review as a conference paper at ICLR 2022
AQUILA achieves significant transmission reduction on both MNIST and CIFARIO as compared to AdaQuantFL and the naive combination of LAQ and AdaQuamtFL.. For in- stance, AQUILA saves 65.98% of transmitted bits compared with AdaQuantFL and 62.05% compared with the naive combination of LAQ and Ada in the IID scenario with CI- FAR10 dataset as shown in Fig. 3d, and other figures all show an obvious reduction in terms of the total transmitted bits required for convergence. Fig. 4
verifies that the quantization level selected by AQUILA will not continuously increase during training like AdaQuantFL and LAQ with AdaQuantFL.
Similarly, AQUILA outperforms all fixed-level LAQ in terms of overall transmitted bits for both MNIST and CIFARIO, as shown in Fig. 3 For example, Fig. 3a indicates that AQUILA reduces 42.6% of total transmitted bits compared with LAQ-2 and 92.32% com- pared with LAQ-16 in MNIST. The transmission reduction is 38. 76% for LAQ-6 and 78.09% for LAQ-16 in CIFARIO shown in Fig. 3e. Besides, Fig. 3 and Fig. indicate that although LAQ with fixed but low quantization level like LAQ-2 and LAQ-4 sometimes transmit smaller amount of bits per step compared with AQUILA, they suffer from lower accuracy and slower convergence. It further verifies the necessity and effectiveness of our well-designed adaptive quantization criterion which achieves fast convergence with similar low-bit transmission but without degradation of the model performance.
AQUILA also works well with heterogeneous local models. With the two various ways of distributing the heterogeneous local models in this paper, our proposed AQUILA still outperforms other methods by significantly reducing overall transmitted bits while main- taining the same convergence property and model accuracy. Please refer to Fig. [0] to Fig. 19 in the Appendix for more detailed information. These experimental results in non- homogeneous FL settings prove that our proposed AQUILA can be used in a more general and complicated federated learning scenarios.
Home- Mlme-MNIST-IID Parameters
(b)
100%-50% MN ST-IID Parameters
(c)
100% 100%-25%% MN NT- Parameters
Rodling
Fodbug
G5GD
05GD
25
OG5D
LAO-2
LAQ-2
LAD-2
Training Losa
LA0-15
FL
Training Lass
#d6QuantFL
LAG-15
Treining Loss
LAD-36
AssQuertFL
3.0
20
25
250
175
66
Total Tannsittted BitsiGR)
Total Transmitted Bits(GB)
Rotal Transmitted Birs(GB)
(8)
Homo-CIFAR10-ID Parameters
(4)
1005-500%-CIAARI Parameters
(
1005-258-CIAARI--NOA Parameters
Fodlag
Fedlwg
Foddwp
OSGD
aSGD
QSRD
LAQ-4
LAQ-A
LAEU-
Tainnin Lass
2.4
LAG 36
LAQ-6
Training Loss
-5
LAp-
Ads0-antfL
LAG-16
maning Laos
AcaQusntft
LAS
LAG 16
AcaQuuntti +1A0
AcuQuartft 1L00
AQULA
AQU LA
1.0
225
3.215
675
200 Total Transmitted BitsEGB)
Total Transmitted BitsiC:
Total Transmitted Bits/GB)
Figure 3: Training Loss vs Total Transmitted Bits. In this figure, LAQ-2 and LAQ-16 represent
LAQ with fixed quantization level of 2 and 16 respectively. AdaQuantFL+LAQ represents the naive combination of AdaQuantFL and LAQ. For heterogeneous model architecture, 100%-50% implies that half of clients have 100% of global model, whereas the other half of clients just share 50% 50% of the global model. Similarly, 100%-25% means the other half of clients just share 25% * 25% of the global model. Particularly, we zoom in the end of the curves to better compare AQUILA with other methods.
8

---

Under review as a conference paper at ICLR 2022
5 CONCLUSIONS AND FUTURE WORK
This paper proposes a communication efficient FL framework to simultancously adjust two mutually- degrees of freedom: communication frequency and quantization level. With the close cooperation of the novel adaptive quantization and adjusted lazy aggregation strategy devel- oped in this paper, the proposed AQUILA has been proven be capable of reducing the transmitted bits while maintaining the same convergence property and model performance compared against ex- isting methods both theoretically and experimentally. The evaluation with Non-IID data distribution and various heterogeneous model architectures demonstrates that AQUILA is compatible to existing FL settings. Future works include further improvements and thcoretical guarantee for FL systems with heterogeneity.
(a)
Homo- MIID--MIIST-ID Parameters
(b)
Homo- (oomo-CIFAR1O-I1 Parameters
30
LATY-2
LAQ-16
Transmitted (its(MB)
25
AdaQuantFL
Transmitted (its(MB)
AdaQuantFL-LAO
LAQ-15
AQUILA
150
Asa0uantLL
aana)aamt++ +LAD
AQULL
15-
100
10-
3.00
150
200
250
1000
1500
2000
2500
3000
Steps
Steps
(c)
100%-50%-MNIST-II0 Parameters
(d)
100%-50YV-IFARRO-IID Parameters
175
LAQ-4
Transmitted (MB)
15.0
Transmitted (its(MB)
3.00
AdaQuantFL
LAQ 2
AQUILA
AdaQuantFL
AdaQuanFL-LLA
75-
AQUILA
5.0
40
23
100
150
200
250
300
500
1000
1500
2000
2500
3000
Steps
Steps
(c)
Parameters
(f)
100%-259%-CIFAR1---00 Parameters
LAQ-2
Transmitted (MB)
AdaQuantFL AdaQuatFL-LAQ
Transmitted (MB)
100
LAQ
LAQ-4
LAQ 16
AQUILA
80
20
Steps
Steps
Figure 4: Transmitted Bits Steps. For better illustration, the results have been down samplea
The solid lines represent values after down-sampling and the shadows around them represent the true values.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Iyy Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308 -18--18, 2016.
Dan Alistarh, Demjan Grubic, Jenry Li, Ryota Tomioka, and Milan Vojnovie. QSGD: Communication-effiient sgd via gradient quantization and encoding. In Proceedings of Advances in Neural Information Processing Systems 30, pp- 1709-1720, 2017.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175-1191, 2017.
Enmao Diao, Jie Ding, and Vahid Tarokh. HeteroFL: Computation and communication efficient
federated learning for heterogeneous clients. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Zhaoyang Du, Celimuge Wu, Tsutomu Yoshinaga, Kok-Lim Alvin Yau, Yusheng Ji, and Jie Li. Federated learning for vehicular Internet of things: Recent advances and open issues. IEEE Computer Graphics and Applications, pp. 45-61, 2020.
Pengchao Han, Shiqiang Wang, and Kin K Leung. Adaptive gradient sparsification for efficient federated learning: An online learning approach. Proceedings of the 40th IEEE International Conference on Distributed Computing Systems, pp. 300-310, 2020.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Fichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint 18X.v:181..03660 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Divyansh Jhunjhunwala, Advait Gadhikar, Gauri Joshi, and Yonina C Eldar. Adaptive quantization of model updates for communication-effiieen federated learning. In Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3110-3114, 2021.
Yujun Lin, Song Han, Huizi Mao, Yu Wang. and Bill Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. In Proceedings of the 6th International Conference on Learning Representations, 2018
Yang Liu, Anbu Huang, Yun Luo, He Huang, Youzhi Liu, Yuanyuan Chen, Lican Feng, Tianjian Chen, Han Yu, and Qiang Yang. Fedvision: online visual object detection platform powered by federated learning. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pp- 13172-13179, 2020a.
Yang Liu, Yan Kang, Chaoping Xing, Tianjian Chen, and Qiang Yang. secure federated transfer learning framework. IEEE Intelligent Systems, pp. 70-82, 2020b.
Yuzhu Mao, Zihao Zhao, Guangfeng Yan, Yang Liu, Tian Lan, Linqi Song, and Wenbo Ding. Communication efficient federated learning with adaptive quantization. arXiv preprint arXiv:2104.06023. 2021.
Brendan McMahan, Eider Moore, Daniel Ramage. Seth Hampson, and Blaise Aguera y Arcas. Communication-efficin learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1282, 2017.
Nikko Strom. Scalable distributed DNN training using commodity GPU cloud computing. In Pro- ceedings of 16th Annual Conference of the International Speech Communication Association, pp- 1488-1492, 2015.
10

---

Under review as a conference paper at ICLR 2022
Jun Sun, Tianyi Chen, Georgios B Giannakis, Qinmin Yang, and Zaiyue Yang. Lazily aggregated quantized gradient innovation for communication--ffiiine federated learning. IEEE Transactions on Pattern Analysis & Machine Intelligence, pp. 1-15, 2020.
Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communication- efficient distributed optimization. In Proceedings of Advances in Neural Information Processing Systems 31. pp. 2299-1309, 2018.
Qiang Yang. Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, pp. 1-19, 2019.

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1
SUPPLEMENTARY EXPERIMENTAL RESULTS
The appendix includes supplementary experimental results, mathematical proof of Theorem 1 and Theorem 2, and detailed derivation of the novel adaptive quantization criterion and lazy aggregation strategy. Compared Fig. 3 and Fig. 4 the main text, result figures in appendix show a more comprehensive evaluation with AQUILA, which contain more detailed information including but limited accuracy vs steps and training loss vs steps curves.
Table 1: Hyperparameters in training process
Dataset
MNIST
CIFARIO
Model
CNN
ResNet18
Hidden Size
[64, 128, 256, 512]
[64, 128, 256,
512]
Data Distribution
IID
Non-IID
IID
Non-IID
Global Epoch E
300
100
3000
100
Local Batch Size B
200
100
10
Optimizer
SGD
SGD
SGD
SGD
Momentum
0.9
0.9
0.9
0.9
Weight Decay
5.00E-04
5.00E-04
5.00E-04
5.00E-04
Learning Rate 7)
0.01
0.01
0.1
0.1
Homo-MNiST-II Parameters
Homo-MNIST-ID Parameters
80
80-
Accuracy
50-
FedAvg
Accuracy(%)
600
FedAvg
QSGD
QSGD
LAQ-2
LAQ-2
LAQ-16
LAQ 16
AdaQuantFL
AdaQuantFL
AdeQuantFL=LAQ
AdaQuantFL+ LAQ
AQUILA
2.5
5.0
7.5
12.5
15.0
17.5
150
200
Total Transmitted Bits(GB)
Steps
(a) Accuracy vs Total Transmitted Bits
(b) Accuracy vs Steps
MDIS-MNISTTIID Parameters
Home MNIST-IID Parameters
FedAvg
QSGD
QSGD
2.0
LAQ-2
2.0
LAQ-2
LAQ-16
Training Loss
LAQ 16
AdaQuantFL
1.5
AdaQuantFL
Training
AdaQuantFL+LAD
Adaptive LAQ
AQUILA
AQUILA
20-
1.0
8.5-
0.5-
7.5
0
50
1.00
150
200
250
Total Transmitted Bits(GB)
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss vs Steps
Figure 5: -IDO-MNIST-IID
12

---

Under review as a conference paper at ICLR 2022
MoIST-M Parameters
Homo-MNISTRoonIDD Parameters
100
100
Accuracy(%)
Accuracy(%)
FadAvg
FedAvg
QSGD
QSGD
40
LAQ 2
40-
LAQ 2
LAQ-16
LAQ-16
20-
AdaQuantFL
26-
AdaQuantFL
AdaQuantFL+LAA
AdaQuantfi
AQUILA
AQUILA
40
Total Transmitted Bits(GB)
Steps
(a) Accuracy Total Transmitted Bits
(b) Accuracy Steps
Homo-MNISTTNon-IID Parameters
Homo-MNIST-Non-IDD Parameters
FedAvg
FedAvg
2.5
QSGD
QSGD
LAQ-16
Training Loss
LAQ-16 16
AdaQuantFL
AdaQuantFL
Training
AdaQuantFL+LAg
Adaptive LAQ
AQUILA
1.0
1.0
0.5-
0.5-
40
Total Transmitted Bits(GB)
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss Steps
Figure 6: Homo-MNIST-Nome
Homo-CIFARIO-ID Parameters
Home-CIFAR1O-ID Parameters
80
80
Accuracy(
FedAvg
(ccuracy(5))
FedAvg
t0.
QSGD
ncc
QSGD
LAQ-4
LAQ-4
40
LAQ-6
40
LAQ-6
LAQ
LAQ 16
AdaQuantFL
AdaQuantFL
AdaQuantFL+LAQ
20
AdaQuantfL+Las
AQUILA
AQUILA
200
400
800
1000
1200
500
1000
1500 Steps
2000
250c
3000
Transmitted Bits(GB)
(a) Accuracy Total Transmitted Bits
(b) Accuracy Steps
Homo-CIFAR10.IID Parameters
Homo- -000--CIFAR0--IDD Parameters
FedAvg
FedAvg
QSGD
QSGD
LAQ-1
LAQ-1
LAQ-6
Training Loss
LAQ-6
LAQ-16
1.5
LAQ-16
Training
AdaQuantFL
AdaQuantFL
AdaQuantFL+LLQ
Adaptive+
2.0-
1.0
AQUILA
AQUILA
0.5
200
400
600
800
1000
1200
S20
1000
1500
2000
2500
3000
Total Transmitted Gits(GB)
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss vs Steps
Figure COmo-CIFAR-IID
13

---

Under review as a conference paper at ICLR 2022
Homo-CIFARIO-Non-III Parameters
Home- Nomo-CIFAR1O-Non-- Parameters
FedAug
Accuracy(%%)
FodAvg
Accuracyt
QSGD
QSGD
LAQ
LAQ
LAQ
LAQ 6
LAQ-16
LAQ-16
AdaQuantFL
AdaQuantFL
AdaQuantFL
AdaQuantfL+
AQUILA
AQUILA
10
20
40
40
60
80
100
Total Transmitted Bits(GB)
Steps
(a) Accuracy Total Transmitted Bits
(h) Accuracy Steps
Homo-CIFARID-Non-II Parameters
Homo-CIFAR1O-Non--I0 Parameters
2.25
FedAvg
2.25
FedAvg
QSGD
QSGD
2.00
LAQ-4
LAQ-4
LAQ-6
Training Loss
LAQ-6
LAQ-16
LAQ-16
150
AdaQuantFL
1.50
AdaQuantFL
AdaQuantFL+LAD
Adaptive
125
AQUILA
AQUILA
1.0
1.00
0.75
10
20
40
20
40
60
100
Total Transmitted Bit)
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss Steps
Figure 8: Momo-CIFARRNon-III
Homo MNIST-IID Parameters
Homo Carroo-MNISTAAA KNIST-Non-llB IID Parameters
Transmitted Bits((MB)
Transmitted Bitts(MB)
13-
15
AddCuantFL
Addo-antFL-LAg
AQUELA
AQUALA
k--
25
105
Steps
Steps
(a) Transmitted Bits Steps
(b) Transmitted Bits Steps
Mamoo-IFAAR1--10 Parameters
Maamo-CIFAR1O-Non--l Parameters
200
200
Transmitedd BIssMBB)
Transmitted Bits(MB)
LAG-1A
250
AssQuartf
1.50
AdaQuant
AssO
1n0-
1.0
1200
2000
Steps
Steps
(c) Transmitted Bits Steps
(d) Transmitted Bits Steps
Figure Homo- Transmitted Bits vs Steps
14

---

Under review as a conference paper at ICLR 2022
100%-50% MNIST-IID Parameters
100S-50S- Parameters
Accuracy
Accuracy(3%
FadAvg
FedAvg
QSGD
QSGD
LAQ 2
LAQ 2
LAQ-16
-0-
LAQ-16
AdaQuantFL
AdaQuantFL
20
AdaQuantFL+LAS
20
AdaQuantfi
AQUILA
AQUILA
200
Total Transmitted Bits(GB)
(a) Accuracy Total Transmitted Bits
(b) Accuracy Steps
100%-50%-MNIST:ID Parameters
100%-50%-MNIST-ID Parameters
FedAvg
FedAvg
QSGD
QSGD
LAQ-2
2.0
LAQ-16
Training Loss
LAQ-16 16
3.5
AdaQuantFL
1.5
AdaQuantFL
Training
AdaQuantFL+LAg
Adaptive LAQ
AQUILA
1.0
1.0
0.5
0.5
150
200
Total Transmitted Bits(GB)
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss Steps
Figure 10: 100%-50%-MMISS-IIDD
100%-50V-MNIST-NO--ID Parameters
100%-50%-MNIST-NOn-ID Parameters
100
100
8C-
(ccuracy(%)
(ccuracy(%)
FedAug
FedAvg
QSGD
QSGD
LAQ-2
LAQ 2
LAQ 16
LAQ 16
20
AdaQuantFL
20
AdaQuantFL
AdsQuantFL+LAD
AdsQuantfL+
AQUILA
AQUILA
0.5
1.5
2.
2.5
3/0
3.5
40
60
Total Transmitted Bits|GB)
Steps
(a) Accuracy Total Transmitted Bits
(b) Accuracy Steps
100%-50%--NISST-NNN Parameters
100%%50%%-MNIST-Nom-il Parameters
FedAvg
FedAvg
2.5
2.5
QSGD
QSGD
LAQ-16
Training Loss
2.0
LAQ-16
AdaQuantFL
AdaQuantFL
Training
MaQuuntFLL-LAO
1.5
Adaptive+ LAQ
AQUILA
1.--
1.0-
0.0
0.5
1.5
2.5
2.5
3.0
3.5
20
40
60
100
Total Transmitted Bits[GB)
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss vs Steps
Figure 11: 100%-00%--MNIS-NNnnn-l
15

---

Under review as a conference paper at ICLR 2022
10FR-IDD Parameters
100%-508-CIFARRO-IID Parameters
80
Accuracy
FedAug
Accuracy(%
FodAvg
QSGD
QSGD
LAQ
LAQ
LAQ
LAQ 6
40-
LAQ-16
-0-
LAQ-16
AdaQuantFL
AdaQuantFL
20-
AdaQuantFL+LA0
20-
AdaQuantfi
AQUILA
AQUILA
300 400
500
500
1000
1500
2000
2500
3000
Total Transmitted Bits(GB)
(a) Accuracy Total Transmitted Bits
(b) Accuracy Steps
100%-50H-CIFARRO-ID Parameters
100%-509--CIARR0--1D Parameters
FedAvg
FedAvg
QSGD
QSGD
LAQ-4
LAQ-4
LAQ-6
Training Loss
LAQ-6
LAQ-16
LAQ-16
Training
AdaQuantFL
AdaQuantFL
AdaQuantFL+LAD
Adaptive+
1.0
AQUILA
AQUILA
0.5
1.00
200
100
400
500
700
800
500
1000
1500
2000
2500
3000
Total Transmitted Bits(GB)
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss Steps
Figure 12: 100%-50%-CIFAR-IID
1008-500-CIARR1--NOn-I00 Parameters
100%-509-CFAAR1--NOo-IDD Parameters
60-
60
Accuracy
25
QSGD
FedAvg
Accuracy
55
FedAvg
QSGD
50-
LAQ-4
50
LAQ-4
LAQ-6
45
LAQ-6
LAQ
LAQ 16
AdaQuantFL
AdaQuantFL
AdsQuantFL+LAD
AdaQuantfL+
AQUILA
AQUILA
10
25
40
60
Transmitted Bits(GB)
Steps
(a) Accuracy Total Transmitted Bits
(b) Accuracy Steps
Parameters
100%550N.CIFARRD--oool-ll Parameters
FedAvg
FedAvg
QSGD
QSGD
2.00
LAQ-1
2.00
LAQ-1
LAQ-6
Training Loss
LAQ-6
LAQ-16
LAQ-16
1.50
AdaQuantFL
3.50
AdaQuantFL
AdaQuantFL+LAQ
Adaptive
AQUILA
AQUILA
1.00
1.00
100
25
20
40
100
Total Transmitted Bits|GB)
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss vs Steps
Figure 13: 100%-50%
16

---

Under review as a conference paper at ICLR 2022
100%-50%% MNISSTII Parameters
100K-50%-MNNST-Non-IIP0 Parameters
275
175
Transittted Bis(MB)
15.0
Transmitted (it(MB))
11.0
12.5
125
160
ASaQuandFL
13.0
AddQuantfL
AdeQuend L-LAQ
AQULA
7.5
AQUILA
50-
5.-
20
200
230
202
25
102
Steps
Steps
(a) Transmitted Bits Steps
(b) Transmitted Bits Steps
100%-500-CIFARIO-ID Parameters
100K--50--CIARR0-NNrr- Parameters
120
1.00
Transmitted Bits:MB)
Transmitted Bits(MB)
100
LAO
La
AdsQuantii
AQULA
1200
1502
2003
2010
2000
120
Steps
Steps
(c) Transmitted Bits Steps
(d) Transmitted Bits Steps
Figure 14: Transmiiittt Bits vs Steps
100%-255-MNISTHDD
100%-25% -NIST-IID Parameters
100%-25%--NNSTTID Parameters
90-
90
30
80
Accuracy
Accuracy
FedAug
60
FedAvg
50-
QSGD
50
QSGD
LAQ-2
LAQ 2
40-
LAQ 16
40
LAQ 16
AdaQuantFL
AdaQuantFL
20
AdsQ
20
AdaQuantfL+
AQUILA
AQUILA
10
10
200
Total Transmitted Bits(GB)
Steps
(a) Accuracy Total Transmitted Bits
(b) Accuracy Steps
100%-25%% --NIST-II Parameters
100%-25%-MMISTTII Parameters
FedAvg
FedAvg
QSGD
QSGD
2.00
LAQ-2
1.75
LAQ-16
Training Lose
175
LAQ-16
Training
150
AdaQuantFL
MdaQuantFL+LAO
1.00
AdaQuantFL
Adaptive+ LAQ
AQUILA
0.7
0.75
050
0.5
50
1.00
150
200
250
300
Total Transmitted Bits(GB)
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss vs Steps
Figure 15: 100%-25%-MNNST-III
17

---

Under review as a conference paper at ICLR 2022
100%--55%-MNIS--OOO Parameters
100%-25% Parameters
20
80
Accuracy
Accuracy(%
60
FadAvg
FedAvg
QSGD
QSGD
LAQ 2
40-
LAQ 2
LAQ-16
LAQ-16
20-
AdaQuantFL
20-
AdaQuantFL
AdaQuantFL +LAQ
AdaQuantfi
AQUILA
AQUILA
0.5
1.5
2.0
255
40
Total Transmitted Bits(GB)
Steps
(a) Accuracy Total Transmitted Bits
(b) Accuracy Steps
100%6-55--MNSST-oon-- Parameters
100%-25%--MISSTNoo--11 Parameters
FedAvg
FedAvg
QSGD
QSGD
2.0-
LAQ-16
Training Loss
2.0-
LAQ-16 16
AdaQuantFL
AdaQuantFL
Training
3.5.
AdeQuantFL+LAN
1.5
Adaptive LAQ
AQUILA
0.5
0.5
0.5
1.5
2.5
40
60
Total Transmitted ltteddeeddeeee Bits(GB)
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss Steps
Figure 16: 100%-25%%-MIIST
100%-25W-CIFARRO-II Parameters
100%-233-CCIAARO--ID Parameters
Accuracy
FedAvg
Accuracy(5)
FedAvg
60.
QSGD
6cc
QSGD
LAQ-4
LAQ-4
LAQ-6
LAQ-6
LAQ
LAQ 16
AdaQuantFL
AdaQuantFL
AdaQuantFL+LAg
20
AdaQuantfL+
AQUILA
AQUILA
400
500
600
500
1000
1500 Steps
2000
250c
3000
Bits(GB)
(a) Accuracy Total Transmitted Bits
(b) Accuracy Steps
Parameters
1.0%%-25%-CIFAR10.IDD Parameters
FedAvg
FedAvg
QSGD
QSGD
LAQ-1
LAQ-1
LAQ-6
Training Loss
LAQ-6
3.5
LAQ-16
1.5
LAQ-16
Training
AdaQuantFL
AdaQuantFL
AdaQuantFL+LAQ
Adaptive=
1.0-
AQUILA
1.0.
AQUILA
0.5-
0.5
1.00
300
400
$00
600
0
S20
1000
1500
2000
2500
5000
Total Transmitted Bits(GB)
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss vs Steps
Figure 17: 100%-0000 25%-CCIRRIDD
18

---

Under review as a conference paper at ICLR 2022
Parameters
100%-25%.CHARRI--Moo.. Parameters
70-
60
Accuracy
FedAug
(%curacc(%)
FodAvg
QSGD
QSGD
LAQ
LAQ
LAQ
40-
LAQ 6
LAQ-16
LAQ-16
30-
AdaQuantFL
30-
AdaQuantFL
AdaQuantFL
AdaQuantfL+
AQUILA
AQUILA
10
15
20
20
40
60
80
100
Total Transmitted Bit)(GB)
Steps
(a) Accuracy Total Transmitted Bits
(b) Accuracy Steps
1008--5%--CIARRO-NNO----l Parameters
100%-229%-CHAAR1-NOO--ID Parameters
FedAvg
FedAvg
2.25
QSGD
QSGD
2.00
LAQ-4
LAQ-4
LAQ-6
Training Loss
LAQ-6
1.75
LAQ-16
LAQ-16
AdaQuantFL
AdaQuantFL
AdaQuantFL+LAD
Adaptive
125
AQUILA
1.25
AQUILA
0.75
0.75
10
15
20
20
40
60
100
Total Transmitted (B))
Steps
(c) Training Loss vs Total Transmitted Bits
(d) Training Loss Steps
Figure 18: 100%%-25%-CIFAA SEA---CIAR--N---III
100% 25% MN Parameters
100% 25% MNIST-Non -ID Parameters
Transmiited Bits((MB)
Transmitted Bits((B)
12-
3-
ASSQUanFL AQUIL
a.
AddQuantFL
AQULLA
25
105
Steps
Steps
(a) Transmitted Bits Steps
(b) Transmitted Bits Steps
100%-25%- -IFFRRIOOID Parameters
1004-25%..IFAR1D--NoH Parameters
Transmtted Bit(MBB)
Transmitted Bits(MB)
LAGIS to
80
AspOuartf
80
1200
3000
Steps
Steps
(c) Transmitted Bits Steps
(d) Transmitted Bits Steps
Figure 19: 100%-25%-11
(00%-255%-Transmittee Transmitted Bits vs Steps
19

---

Under review as a conference paper at ICLR 2022
Craps
(a) 1st client system
(h) 2nd client in system
(c) 3rd client system
Figure 20: The illustration about AQUILA's ability to suppress high-bit transmission. Each figure shows the relationship between local quantization level and communication skipping of one local client during training with heterogeneous models in the III scenario with CIFAR10 dataset. These figures imply a trend that communications with relatively high quantization level (e.g. higher than the initial level) chosen by 7 are mostly skipped by (8)
A.2
MATIEMATICAL PROOF
A.2.1
PROOF OF THEOREM
Without lazy aggregation, the aggregated model at iteration should be:
g++1
0²
a M
Σ Qby. (gth).
(13)
MEM
With lazy aggregation, the actual aggregated model at iteration & is:
k+1
Qst. (g
Σ
Qb
(0
(14)
M
M
MEMAM
Th EEM
With
13
and
14
the gradient loss caused by skipping can be written
0
b+1
h+1
a
Qua (gt)) Q8n 19 k-1
2
M
mEM
a M
Qas (gt)))
Qie ( 1|
M
a²
Σ
Quth (oth) - ghil - -1Qpt- (g (g (g 2) + (g.h. - -
M
MCM
3a 2
9t k-
+
3a
Σ Ele (g )11 + ( 1
(15)
M
M
MEM
MEM
where the two inequalities follow from:
Σ
02
| 2
Take expectation of both sides of 15
E
k+1
gh+1
<
3a 2
M
E
9m
m
k-1
3a
M
(EIl (1 +Bs++=+Beee 11//085-
1113)
mEM
MEM
3a
12
3a
do²
do2
M
9m
9
+
M
(8/- 1)2
+
(b+
)
mEM
mEM
3a2 2
dm
3x2
d
d
M
K
9m
9m m
-
M
( 132
(b,r )2
(16)
MEM
MEM
20

---

Under review as a conference paper at ICLR 2022
where the second inequality follows from Assumption 2 with O² denoting the maximum | and
Y6 d/b², and the last inequality is resulted from the conclusion in LAQ (Sun al. 2020), which indicates that the client m communicates with the server at most dm rounds in total K" iterations,
where dm is constant related Lmm
With the definition of Bm and Ch = d [log2 + +]] + - 32, the expected gradient loss can be written as:
3a
dm C
3a
do
do² )²
0
9m
9 m
k- 10-10
+
Σ
M
Bm
M
(bm
1/2
(b
MCM
3a
dmd [loge (1/, + +]]
3a
do²
do²
k-1
M
Bm
Im
mm
M
(bm
)2
(b.
)2
MEM
MEM
3a²
dm (d 32)
|
k--
1|
+
Im
M
Bm
mEM
3a
dmd [log2 (4
3a
do²
do²
|
k-11
M
Bm
M
(b/m
1b
(b& y²
mEM
3a2
2
dm (d + 32)
+
gm
gm
(17)
M
Bm
MEM
This completes the proof for Theorem
A.2.2 DERIVATION DETAILS OF ADAPTIVE QUANTIZATION CRITERION 7
Let H be the upper bound to be minimized:
H
3a²
dmd [logz m))
k-1
2
3a
do
+
2o²
M
Bm
M
(bk-1)2
(be,
)2
MEM
MEM
3a
dm (d + 32)
|
gh- 1
(18)
Bm
MEM
The first derivative is:
2
0H
3a 3add,d
9th
g
[log e]
2
(19)
06k
Bm Mbh
M (bbt
0H
Let
0, there is:
20 Imlogg
(bth)*
(20)
di
ghh
k
Dividing (I"h)" by (bia):
|
9 1
(bbh))
by
(21)
19th
9m
For iteration k 1, we set 6), for each client m as the intial quantization level by. and thus we get the adaptive quantization criterion 7 for iteration k 1, 2,2,... K:
| - 1|
(bth)) bo
(22)
gt. gh=
21

---

Under review as a conference paper at ICLR 2022
A.2.3 PROOF OF THEOREM 2
With the adjusted communication criterion (8), all transmitted gradient updates are larger than threshold after quantization. Therefore, all transmitted gradient updates satisfy:
g 9m
k - Qbt. (gm) + Qi fm )
9m + Qbs. (gm)
-iat-
Im
k-11
12
1
Qut.
(mm)
Q
9
[ 0 2
2d Ed
2 +
[|502,
(1 1|1 ( 1
(23)
a2M2
then,
g
0 2
(b bo
Im
9 x-1
gmm
|| g 1mm12
S bo
1
E Ed
+ ||Ean (9th (gth) 1112= +1|03--- (0 1/2
a N8²
|
9m
60
MM Ed= Ea(f
2
Ea(f(Ohtl-d)
f(0 -d) 3
||sTa, (gth 11: + |5 (9 -1/2
gth - 1|
b0
26 a lf( f(() + 3 ||Eng (oth 11 + Fet. -@ 11
|19h: - 9m
b01
26 a M
(f( f(0))
(Lm)² )² |2| 80 2
bo
(24)
2 lf(0e) f(0))
f(0") f(0")
where the second inequality comes from Assumption 1. Let P
Therefore, if
L 0l 0" |
< (1/ML) L holds, then in the experiment setting where & = E2 ED E and Lm for all E MI, there is:
(Lm) ) 1 0011
f(0") f(0))
(b < b01
< bog
(25)
28,
[f( f(0")]
f(0²) f(00)
22

---

Under review as a conference paper at ICLR 2022
A.2.4 DEVELOPMENT LAZY AGGREGATION CRITERION
8
Qba, (gm)
Im
R-11
Qos (9
9
#
+ 9m
k-1
+
Im
9m
k-1
3 2
9m
A-1
+ 2
Eat (g
-bh ( |
< 2
0h 0
k-d'
+ 2
Ebe (g
6 (g k-
2
d'
2L
0
h+
0h-d
+ cbp (gm)
Ebg (g k-1-1 |12
d=1
< 2L2 d'
gk+t @h-d 1
EBt ( ) Ebt. (g
(26)
where the second inequality comes from:
k-1 Vm (0* d).
(27)
9 vm (0*)
(28)
V Fm (0*) Vm 112 S L 0r dd-d'
(29)
Following LAQ's definition, redefine dmn E Ml as:
dm == max {d < Ea/ D2D), 1,,, D).
(30)
d
With the definition of dm and Eı 2 & 2 2 En, there is:
Ex 20²
1
for all d' satisfyingl S d' S dm.
(31)
With 26 and 31
we have:
Qath. (gth) - -Qpt
(9th ")1
12
Σ
g gl-dl² +2 E0g (gt)) 886-1 (g
(0th 1
Ed'
22 M2
d=1
D
x- 1/a
<
Σ Ed
gh++1-d
gh-
+ Eb (gth) - -Est- (0
(32)
a. a²M2 M
d=1
23


Reviews:
[{"review_id": "Z7DnQiUbik6", "reviewer": "Reviewer_9gtf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) The paper considers a relevant problem.\n2) It is toward unifying the prior work.\n3) Changes in the gradients seem a reasonable signal for the adaptation decisions.\n\nWeaknesses:\n1) On page 4, the paper mentions that AQUILA has a better method for setting the quantization level as, unlike AdaQuantFL, equation (7) doesn't increase the quantization level over the training process. However, doesn't the size of the gradients (i.e., $||g^k_m||$) decrease during the training process? If so, wouldn't it be better to compare the normalized gradients in eq. (7) rather than the gradients?        \n\n2) Have the performance improvements shrunk on the CIFAR10 dataset which is a more complex dataset than MNIST? For example, look at Figure 4 subplots (d) and (f). If so, why does this happen and what is AQUILA's benefit for more complex data distributions in the real world? ", "summary_of_the_paper": "This paper points out that model update frequency and update quantization level are not disjoint decisions in federated learning systems. Furthermore, it explains how to modify and combine prior work on each of the two decisions to develop a new adaptive aggregation method. \n\n", "summary_of_the_review": "I think this paper raises a good question about the interaction between the frequency and quality (i.e., quantization) of model updates in FL. Still, neither the intuitions nor the experiment results convince the reader much about the proposed approach. I'm marginally recommending accepting the paper mainly based on the assumptions that the problem statement is novel and the theoretical methodology is sound to my current understanding of the paper.   ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "NR2xa1QNCz", "reviewer": "Reviewer_E4L8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of this paper is to develop the framework fro FL that allows to ruduce communication cost.\nAuthors uses the following ideas as a core: 1)  the frequency of communication rounds depends on the local gradient change (Local Gradient Aggregation); 2) adaptive quantization correlates with the logic of the Lazy Gradient Aggregation - the quantization level is smaller if the local gradient difference between the current gradient and last sent one is bigger.\nThe combination of these two ideas allows to reduce the communication round amount in the late iterations of algorithm comparing to AdaQuantFL.\n\nMy main concern is the following: since the server updates the model with some new quantized gradients and with old ones for the machines that have decided to skip their update we save some amount of communication rounds TO the server. However, the update of the global model is a combination of quantized updates it can be quite ``dense'' and hard to send, but on every round of the algorithm server broadcast the new model to all machines that seems to be costly. May be the broadcast should be made in the same manner as the lazy aggregation procedure?\n\nAlso I have a minor comment to the notation $\\theta^{\\star}$ that seems to denote $f(\\theta^\\star}$.\n\nAccording to the experimental section, I failed to follow the plots: they are quite small on A4 paper; they are not colorblind friendly and I am colorblind. However, I still have a question to the Figures representing \"transmitted bits\" do they assume the communication TO server  only or together with the broadcasting.\n\n\n\n", "summary_of_the_paper": "In this paper, authors propose the framework for federated learning that adaptively adjusts the frequency of the communication rounds and the quantization efficiency in a synergistic way.", "summary_of_the_review": "In conclusion, I think that authors of this work made a good job. The idea of adaptation of quantization and frequency of communications is very nice; however the broadcasting procedure seems to be very time-consuming.\nFor me the Experimental section was impossible to follow, but I think that minor modifications of the Figures can resolve this problem.\n\nAll in all, I think that after some changes this paper can be accepted and the presented algorithm can be used in real-world FL applications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "48Q-ymnbQv5", "reviewer": "Reviewer_Dvx6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**A Strengths:**\n1) Experiments validate the superiority of the proposed method over each of its individual components (LAQ, AdaQuantFL,), as well as their naive combination, and other well-known FL methods.\n \n2) At a high level, the idea itself of combining two orthogonal directions in Federated learning, transmitting infrequently and compressing the updates, is very interesting. The fact that the trivial combination of these two approaches isn't as effective and one needs to intricately design the overall scheme is surprising, and something to keep in mind for future research in this area.\n\n**B Weaknesses:**\n1)  It was disappointing not to see an explicit convergence rate of the proposed method. \n\n2) Building on the previous point, a lack of comparison with local methods in convergence rate and experiments was disappointing. I think these comparisons are needed for a fair assessment of the paper.\n\n3) Since this paper is mainly about combining existing ideas, there is not a lot of algorithmic novelty in the paper.\n\n**C Presentation:**\n1) The paper is well-written and easy to follow.\n\n2) Is there any specific reason that authors choose to minimize the upper bound of the quantity on the LHS of Theorem 1. Choosing different metrics would lead to different bit allocation strategies. Why was this particular metric chosen then?", "summary_of_the_paper": "The paper proposes combining two orthogonal algorithms -- the lazily aggregated gradient(LAQ)method and adaptive quantization (AdaQuantFL)-- to reduce communication complexity in federated learning. In particular, while LAQ focuses on the frequency of gradient updates in each iteration, AdaQuantFL focuses on the adaptive allocation of bits across iterations. This paper proposes to combine both of these approaches for better communication complexity in federated learning. \n\nAt the heart of the combining strategy is Eq. 7, which allocates the bits used in each iteration. This equation is intern based on minimizing the upper bound on the expected deviation in the path of the iterate due to gradient skipping. The authors theoretically show that such a bit allocation strategy leads to reduced communication compared to the bit allocation strategy of AdaQuantFl.\n\n", "summary_of_the_review": "Overall, primarily based on the experiments, I recommend the paper be accepted. However, I cannot be as confident of the paper's contribution as I would like to be because of a lack of comparison with a crucial part of the literature (B.1, B.2).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two of the initial reviews of the paper were mildly positive (2 scores of 6), and one was very positive (score of 8). However, these reviews failed to notice some severe issues with the paper, which were detailed by the Area Chair in an Extra Review which was provided late. The severe issues include: clarity of exposition (undefined notation in many places) and theory (vacuous or meaningless theorems and assumptions). I apologize to the authors for not having had the chance to defend against this late review. However, the issues are indeed severe.
Decision: Reject