DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning

1 INTRODUCTION . Edge devices such as smartphones , remote sensors and smart home appliances generate massive amounts of data ( Wang et al. , 2018b ; Cao et al. , 2017 ; Shi & Dustdar , 2016 ) . In recent years , Federated Learning ( FL ) has emerged as a technique to train models on this data while preserving privacy ( McMahan et al. , 2017 ; Li et al. , 2018 ) . In FL , we have a single server that is connected to many clients . Each client stores a local dataset that it does not want to share with the server because of privacy concerns or law enforcement ( Voigt & Von dem Bussche , 2017 ) . The server wants to train a model on all local datasets . To this end , it initializes the model and sends it to a random subset of clients . Each client trains the model on its local dataset and sends the trained model back to the server . The server accumulates all trained models into an updated model for the next iteration and repeats the process for several rounds until some termination criterion is met . This procedure enables the server to train a model without accessing any local datasets . Today ’ s neural network models often have millions or even billions ( Brown et al. , 2020 ) of parameters , which makes high communication costs a concern in FL . In fact , Qiu et al . ( 2020 ) suggest that communication between clients and server may account for over 70 % of energy consumption in FL . Reducing communication in FL is an attractive area of research because it lowers bandwidth requirements , energy consumption and training time . Communication in FL occurs in two phases : Sending parameters from the server to clients ( downlink ) and sending updated parameters from clients to the server ( uplink ) . Uplink bandwidth usually imposes a tighter bottleneck than downlink bandwidth . This has several reasons . For one , the average global mobile upload bandwidth is currently less than one fourth of the download bandwidth ( Speedtest ) . For another , FL downlink communication sends the same parameters to each client . Broadcasting parameters is usually more efficient than the accumulation of parameters from differ- ent clients that is required for uplink communication ( Amiri et al. , 2020 ; Reisizadeh et al. , 2019 ) . For these reasons , we seek to compress uplink communication . A large class of compression algorithms for FL apply some lossy quantizer Q , optionally followed by a lossless compression stage . Q usually provides a “ quantization level ” hyperparameter q to control the coarseness of quantization ( e.g . the number of bins for fixed-point quantization ) . When q is kept constant during training , we speak of static quantization . When q changes , we speak of adaptive quantization . Adaptive quantization can exploit asymmetries in the FL framework to minimize communication . One such asymmetry lies in FL ’ s training time , where Jhunjhunwala et al . ( 2021 ) observed that early training rounds can use a lower q without affecting convergence . Figure 2 illustrates how time-adaptive quantization leverages this phenomenon to minimize communication . Another asymmetry lies in FL ’ s client space , because most FL algorithms weight client contributions to the global model proportional to their local dataset sizes . Figure 1 illustrates how client-adaptive quantization can minimize the quantization error . Intuitively , FL clients with greater weighting should have a greater commu- nication budget and our proposed client-adaptive quantization achieves this in a principled way . To this end , we introduce the expected variance of an accumulation of quantized parameters , E [ Var ( ∑ Q ( p ) ) ] , as a measure of the quantization error . Our client-adaptive quantization algorithm then assigns clients minimal quantization levels , subject to a fixed E [ Var ( ∑ Q ( p ) ) ] . This lowers the amount of data communicated from clients to the server , without increasing the quantization error . DAdaQuant ( Doubly Adaptive Quantization ) combines time- and client-adaptive quantization with an adaptation of the QSGD fixed-point quantization algorithm to achieve state-of-the-art FL uplink compression . In this paper , we make the following contributions : • We introduce the concept of client-adaptive quantization and develop algorithms for time- and client-adaptive quantization that are computationally efficient , empirically superior to existing algorithms , and compatible with arbitrary FL quantizers . Our client-adaptive quantization is provably optimal for stochastic fixed-point quantizers . • We create Federated QSGD as an adaptation of the stochastic fixed-point quantizer QSGD that works with FL . Federated QSGD outperforms all other quantizers , establishing a strong baseline for FL compression with static quantization . • We combine time- and client-adaptive quantization into DAdaQuant . We demonstrate DAdaQuant ’ s state-of-the-art compression by empirically comparing it against several competitive FL compression algorithms . 2 RELATED WORK . FL research has explored several approaches to reduce communication . We identify three general directions . First , there is a growing interest of investigating FL algorithms that can converge in fewer rounds . FedAvg ( McMahan et al. , 2017 ) achieves this with prolonged local training , while FOLB ( Nguyen et al. , 2020 ) speeds up convergence through a more principled client sampling . Since communication is proportional to the number of training rounds , these algorithms effectively reduce communication . Secondly , communication can be reduced by reducing the model size because the model size is proportional to the amount of training communication . PruneFL ( Jiang et al. , 2019 ) progressively prunes the model over the course of training , while AFD ( Bouacida et al. , 2021 ) only trains submodels on clients . Thirdly , it is possible to directly compress FL training communication . FL compression algorithms typically apply techniques like top-k sparsification ( Malekijoo et al. , 2021 ; Rothchild et al. , 2020 ) or quantization ( Reisizadeh et al. , 2019 ; Shlezinger et al. , 2020 ) to parameter updates , optionally followed by lossless compression . Our work applies to quantization-based compression algorithms . It is partially based on QSGD ( Alistarh et al. , 2017 ) , which combines lossy fixed-point quantization with a lossless compression algorithm to compress gradients communicated in distributed training . DAdaQuant adapts QSGD into Federated QSGD , which works with Federated Learning . DAdaQuant also draws inspiration from FedPAQ ( Reisizadeh et al. , 2019 ) , the first FL framework to use lossy compression based on model parameter update quantization . However , FedPAQ does not explore the advantages of additional lossless compression or adaptive quantization . UVeQFed ( Shlezinger et al. , 2020 ) is an FL compression algorithm that generalizes scalar quantization to vector quantization and subsequently employs lossless compression with arithmetic coding . Like FedPAQ , UVeQFed also limits itself to a single static quantization level . Faster convergence , model size reduction and communication compression are orthogonal techniques , so they can be combined for further communication savings . For this paper , we limit the scope of empirical comparisons to quantization-based FL compression algorithms . For quantization-based compression for model training , prior works have demonstrated that DNNs can be successfully trained in low-precision ( Banner et al. , 2018 ; Gupta et al. , 2015 ; Sun et al. , 2019 ) . There are also several adaptive quantization algorithms for training neural networks in a non-distributed setting . Shen et al . ( 2020 ) use different quantization levels for different parameters of a neural network . FracTrain ( Fu et al. , 2020 ) introduced multi-dimensional adaptive quantization by developing time-adaptive quantization and combining it with parameter-adaptive quantization . However , FracTrain uses the current loss to decide on the quantization level . FL generally can only compute local client losses that are too noisy to be practical for FracTrain . AdaQuantFL introduces time-adaptive quantization to FL , but requires the global loss ( Jhunjhunwala et al. , 2021 ) . To compute the global loss , AdaQuantFL has to communicate with every client each round . We show in Section 4.2 that this quickly becomes impractical as the number of clients grows . DAdaQuant ’ s time-adaptive quantization overcomes this issue without compromising on the underlying FL communication . In addition , to the best of our knowledge , DAdaQuant is the first algorithm to use client-adaptive quantization . 3 THE DADAQUANT METHOD . 3.1 FEDERATED LEARNING . Federated Learning assumes a client-server topology with a set C = { ci|i ∈ { 1 , 2 ... N } } of N clients that are connected to a single server . Each client ck has a local dataset Dk from the local data distribution Dk . Given a model M with parameters p , a loss function fp ( d ∈ Dk ) and the local loss Fk ( p ) = 1 |Dk| ∑ d∈Dk fp ( d ) , FL seeks to minimize the global loss G ( p ) = ∑N k=1 |Dk|∑ l |Dl| Fk ( p ) . 3.2 FEDERATED AVERAGING ( FEDAVG ) . DAdaQuant makes only minimal assumptions about the FL algorithm . Crucially , DAdaquant can complement FedAvg ( McMahan et al. , 2017 ) , which is representative of a large class of FL algorithms . FedAvg trains the model M over several rounds . In each round t , FedAvg sends the model parameters pt to a random subset St of K clients who then optimize their local objectives Fk ( pt ) and send the updated model parameters pkt+1 back to the server . The server accumulates all parameters into the new global model pt+1 = ∑ k∈St |Dk|∑ j |Dj | pkt+1 and starts the next round . Algorithm 1 lists FedAvg in detail . For our experiments , we use the FedProx ( Li et al. , 2018 ) adaptation of FedAvg . FedProx improves the convergence of FedAvg by adding the proximal term µ2 ‖p k t+1 − pt‖2 to the local objective Fk ( pkt+1 ) in Line 20 of Algorithm 1 . 3.3 QUANTIZATION WITH FEDERATED QSGD . While DAdaQuant can be applied to any quantizer with a configurable quantization level , it is optimized for fixed-point quantization . We introduce Federated QSGD as a competitive stochastic fixed-point quantizer on top of which DAdaQuant is applied . In general , stochastic fixed-point quantization uses a quantizer Qq with quantization level q that splits R≥0 and R≤0 into q intervals each . Qq ( p ) then returns the sign of p and |p| stochastically rounded to one of the endpoints of its encompassing interval . Qq ( p ) quantizes the vector p elementwise . We design DAdaQuant ’ s quantization stage based on QSGD , an efficient fixed-point quantizer for state-of-the-art gradient compression . QSGD quantizes a vector p in three steps : 1 . Quantize p as Qq ( p ||p||2 ) into q bins in [ 0 , 1 ] , storing signs and ||p||2 separately . ( lossy ) 2 . Encode the resulting integers with 0 run-length encoding . ( lossless ) 3 . Encode the resulting integers with Elias ω coding . ( lossless ) QSGD has been designed specifically for quantizing gradients . This makes it not directly applicable to parameter compression . To overcome this limitation , we apply difference coding to uplink compression , first introduced to FL by FedPAQ . Each client ck applies Qq to the parameter updates pkt+1 − pt ( cf . Line 21 of Algorithm 1 ) and sends them to the server . The server keeps track of the previous parameters pt and accumulates the quantized parameter updates into the new parameters as pt+1 = pt + ∑ k∈St |Dk|∑ l |Dl| Qq ( p k t+1 − pt ) ( cf . Line 11 of Algorithm 1 ) . We find that QSGD works well with parameter updates , which can be regarded as an accumulation of gradients over several training steps . We call this adaptation of QSGD Federated QSGD .

Summary: The paper tries to reduce the communication cost of the Federated Learning (FL) algorithms.  They introduce the doubly-adaptive quantization algorithm (DAdaQuant), which adopts two quantization techniques: 1, time-adaptive quantization; 2, client-adaptive quantization.  The empirical studies also show that DAdaQuant can improve client sever compression.