DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning

1 INTRODUCTION . Edge devices such as smartphones , remote sensors and smart home appliances generate massive amounts of data ( Wang et al. , 2018b ; Cao et al. , 2017 ; Shi & Dustdar , 2016 ) . In recent years , Federated Learning ( FL ) has emerged as a technique to train models on this data while preserving privacy ( McMahan et al. , 2017 ; Li et al. , 2018 ) . In FL , we have a single server that is connected to many clients . Each client stores a local dataset that it does not want to share with the server because of privacy concerns or law enforcement ( Voigt & Von dem Bussche , 2017 ) . The server wants to train a model on all local datasets . To this end , it initializes the model and sends it to a random subset of clients . Each client trains the model on its local dataset and sends the trained model back to the server . The server accumulates all trained models into an updated model for the next iteration and repeats the process for several rounds until some termination criterion is met . This procedure enables the server to train a model without accessing any local datasets . Today ’ s neural network models often have millions or even billions ( Brown et al. , 2020 ) of parameters , which makes high communication costs a concern in FL . In fact , Qiu et al . ( 2020 ) suggest that communication between clients and server may account for over 70 % of energy consumption in FL . Reducing communication in FL is an attractive area of research because it lowers bandwidth requirements , energy consumption and training time . Communication in FL occurs in two phases : Sending parameters from the server to clients ( downlink ) and sending updated parameters from clients to the server ( uplink ) . Uplink bandwidth usually imposes a tighter bottleneck than downlink bandwidth . This has several reasons . For one , the average global mobile upload bandwidth is currently less than one fourth of the download bandwidth ( Speedtest ) . For another , FL downlink communication sends the same parameters to each client . Broadcasting parameters is usually more efficient than the accumulation of parameters from differ- ent clients that is required for uplink communication ( Amiri et al. , 2020 ; Reisizadeh et al. , 2019 ) . For these reasons , we seek to compress uplink communication . A large class of compression algorithms for FL apply some lossy quantizer Q , optionally followed by a lossless compression stage . Q usually provides a “ quantization level ” hyperparameter q to control the coarseness of quantization ( e.g . the number of bins for fixed-point quantization ) . When q is kept constant during training , we speak of static quantization . When q changes , we speak of adaptive quantization . Adaptive quantization can exploit asymmetries in the FL framework to minimize communication . One such asymmetry lies in FL ’ s training time , where Jhunjhunwala et al . ( 2021 ) observed that early training rounds can use a lower q without affecting convergence . Figure 2 illustrates how time-adaptive quantization leverages this phenomenon to minimize communication . Another asymmetry lies in FL ’ s client space , because most FL algorithms weight client contributions to the global model proportional to their local dataset sizes . Figure 1 illustrates how client-adaptive quantization can minimize the quantization error . Intuitively , FL clients with greater weighting should have a greater commu- nication budget and our proposed client-adaptive quantization achieves this in a principled way . To this end , we introduce the expected variance of an accumulation of quantized parameters , E [ Var ( ∑ Q ( p ) ) ] , as a measure of the quantization error . Our client-adaptive quantization algorithm then assigns clients minimal quantization levels , subject to a fixed E [ Var ( ∑ Q ( p ) ) ] . This lowers the amount of data communicated from clients to the server , without increasing the quantization error . DAdaQuant ( Doubly Adaptive Quantization ) combines time- and client-adaptive quantization with an adaptation of the QSGD fixed-point quantization algorithm to achieve state-of-the-art FL uplink compression . In this paper , we make the following contributions : • We introduce the concept of client-adaptive quantization and develop algorithms for time- and client-adaptive quantization that are computationally efficient , empirically superior to existing algorithms , and compatible with arbitrary FL quantizers . Our client-adaptive quantization is provably optimal for stochastic fixed-point quantizers . • We create Federated QSGD as an adaptation of the stochastic fixed-point quantizer QSGD that works with FL . Federated QSGD outperforms all other quantizers , establishing a strong baseline for FL compression with static quantization . • We combine time- and client-adaptive quantization into DAdaQuant . We demonstrate DAdaQuant ’ s state-of-the-art compression by empirically comparing it against several competitive FL compression algorithms . 2 RELATED WORK . FL research has explored several approaches to reduce communication . We identify three general directions . First , there is a growing interest of investigating FL algorithms that can converge in fewer rounds . FedAvg ( McMahan et al. , 2017 ) achieves this with prolonged local training , while FOLB ( Nguyen et al. , 2020 ) speeds up convergence through a more principled client sampling . Since communication is proportional to the number of training rounds , these algorithms effectively reduce communication . Secondly , communication can be reduced by reducing the model size because the model size is proportional to the amount of training communication . PruneFL ( Jiang et al. , 2019 ) progressively prunes the model over the course of training , while AFD ( Bouacida et al. , 2021 ) only trains submodels on clients . Thirdly , it is possible to directly compress FL training communication . FL compression algorithms typically apply techniques like top-k sparsification ( Malekijoo et al. , 2021 ; Rothchild et al. , 2020 ) or quantization ( Reisizadeh et al. , 2019 ; Shlezinger et al. , 2020 ) to parameter updates , optionally followed by lossless compression . Our work applies to quantization-based compression algorithms . It is partially based on QSGD ( Alistarh et al. , 2017 ) , which combines lossy fixed-point quantization with a lossless compression algorithm to compress gradients communicated in distributed training . DAdaQuant adapts QSGD into Federated QSGD , which works with Federated Learning . DAdaQuant also draws inspiration from FedPAQ ( Reisizadeh et al. , 2019 ) , the first FL framework to use lossy compression based on model parameter update quantization . However , FedPAQ does not explore the advantages of additional lossless compression or adaptive quantization . UVeQFed ( Shlezinger et al. , 2020 ) is an FL compression algorithm that generalizes scalar quantization to vector quantization and subsequently employs lossless compression with arithmetic coding . Like FedPAQ , UVeQFed also limits itself to a single static quantization level . Faster convergence , model size reduction and communication compression are orthogonal techniques , so they can be combined for further communication savings . For this paper , we limit the scope of empirical comparisons to quantization-based FL compression algorithms . For quantization-based compression for model training , prior works have demonstrated that DNNs can be successfully trained in low-precision ( Banner et al. , 2018 ; Gupta et al. , 2015 ; Sun et al. , 2019 ) . There are also several adaptive quantization algorithms for training neural networks in a non-distributed setting . Shen et al . ( 2020 ) use different quantization levels for different parameters of a neural network . FracTrain ( Fu et al. , 2020 ) introduced multi-dimensional adaptive quantization by developing time-adaptive quantization and combining it with parameter-adaptive quantization . However , FracTrain uses the current loss to decide on the quantization level . FL generally can only compute local client losses that are too noisy to be practical for FracTrain . AdaQuantFL introduces time-adaptive quantization to FL , but requires the global loss ( Jhunjhunwala et al. , 2021 ) . To compute the global loss , AdaQuantFL has to communicate with every client each round . We show in Section 4.2 that this quickly becomes impractical as the number of clients grows . DAdaQuant ’ s time-adaptive quantization overcomes this issue without compromising on the underlying FL communication . In addition , to the best of our knowledge , DAdaQuant is the first algorithm to use client-adaptive quantization . 3 THE DADAQUANT METHOD . 3.1 FEDERATED LEARNING . Federated Learning assumes a client-server topology with a set C = { ci|i ∈ { 1 , 2 ... N } } of N clients that are connected to a single server . Each client ck has a local dataset Dk from the local data distribution Dk . Given a model M with parameters p , a loss function fp ( d ∈ Dk ) and the local loss Fk ( p ) = 1 |Dk| ∑ d∈Dk fp ( d ) , FL seeks to minimize the global loss G ( p ) = ∑N k=1 |Dk|∑ l |Dl| Fk ( p ) . 3.2 FEDERATED AVERAGING ( FEDAVG ) . DAdaQuant makes only minimal assumptions about the FL algorithm . Crucially , DAdaquant can complement FedAvg ( McMahan et al. , 2017 ) , which is representative of a large class of FL algorithms . FedAvg trains the model M over several rounds . In each round t , FedAvg sends the model parameters pt to a random subset St of K clients who then optimize their local objectives Fk ( pt ) and send the updated model parameters pkt+1 back to the server . The server accumulates all parameters into the new global model pt+1 = ∑ k∈St |Dk|∑ j |Dj | pkt+1 and starts the next round . Algorithm 1 lists FedAvg in detail . For our experiments , we use the FedProx ( Li et al. , 2018 ) adaptation of FedAvg . FedProx improves the convergence of FedAvg by adding the proximal term µ2 ‖p k t+1 − pt‖2 to the local objective Fk ( pkt+1 ) in Line 20 of Algorithm 1 . 3.3 QUANTIZATION WITH FEDERATED QSGD . While DAdaQuant can be applied to any quantizer with a configurable quantization level , it is optimized for fixed-point quantization . We introduce Federated QSGD as a competitive stochastic fixed-point quantizer on top of which DAdaQuant is applied . In general , stochastic fixed-point quantization uses a quantizer Qq with quantization level q that splits R≥0 and R≤0 into q intervals each . Qq ( p ) then returns the sign of p and |p| stochastically rounded to one of the endpoints of its encompassing interval . Qq ( p ) quantizes the vector p elementwise . We design DAdaQuant ’ s quantization stage based on QSGD , an efficient fixed-point quantizer for state-of-the-art gradient compression . QSGD quantizes a vector p in three steps : 1 . Quantize p as Qq ( p ||p||2 ) into q bins in [ 0 , 1 ] , storing signs and ||p||2 separately . ( lossy ) 2 . Encode the resulting integers with 0 run-length encoding . ( lossless ) 3 . Encode the resulting integers with Elias ω coding . ( lossless ) QSGD has been designed specifically for quantizing gradients . This makes it not directly applicable to parameter compression . To overcome this limitation , we apply difference coding to uplink compression , first introduced to FL by FedPAQ . Each client ck applies Qq to the parameter updates pkt+1 − pt ( cf . Line 21 of Algorithm 1 ) and sends them to the server . The server keeps track of the previous parameters pt and accumulates the quantized parameter updates into the new parameters as pt+1 = pt + ∑ k∈St |Dk|∑ l |Dl| Qq ( p k t+1 − pt ) ( cf . Line 11 of Algorithm 1 ) . We find that QSGD works well with parameter updates , which can be regarded as an accumulation of gradients over several training steps . We call this adaptation of QSGD Federated QSGD .

Summary: This paper studies how to compress the local model changes in federated learning to save uplink communication costs. In particular, the authors found (1) adaptively increasing the number of quantization levels (2) adaptively assigning different quantization levels on different clients, can effectively outperform previous static quantization schemes. They evaluated the proposed algorithm on multiple federated learning datasets.