Expanding the Reach of Federated Learning by Reducing Client Resource Requirements

1 INTRODUCTION . Federated Learning ( FL ) allows users to reap the benefits of models trained from rich yet sensitive data captured by their mobile devices , without the need to centrally store such data ( McMahan et al. , 2017 ; Konečný et al. , 2016a ; Smith et al. , 2017 ) . Under the FL paradigm , each device performs training on samples available locally and only communicates intermediate model updates . Network speed and number of nodes are two of the core systems aspects that differentiate FL from traditional distributed learning in data centers , with network bandwidth being potentially orders of magnitude slower and the number of worker nodes orders of magnitude larger . Together , these issues exacerbate the communication bottlenecks usually associated with distributed learning , increasing both the number of stragglers and the probability of devices dropping out altogether . The problem is further aggravated when working with high capacity models with large numbers of parameters . Insisting on training these large models using existing federated optimization methods can lead to the systematic exclusion of clients with restricted bandwidth or limited network access from the training stage , and thus to a degraded user experience once these models are served . One naive solution involves training low capacity models with smaller communication footprints , at the expense of model accuracy . As a middle ground , we could develop strategies to reduce the communication footprint of larger , high-capacity models . Recent work ( Konečný et al. , 2016b ) has in fact taken this approach , but only in the context of client-to-server FL communication . Their success with lossy compression strategies is perhaps not surprising , as the clients ’ lossy , yet unbiased , updates are eventually averaged over many users . However , server-to-client exchanges do not benefit from such averaging . As such , they remain a main bottleneck in our goal of expanding FL ’ s reach . In this work , we propose two novel strategies to mitigate the server-to-client communication footprint , and empirically demonstrate their efficacy and seamless integration with existing client-toserver strategies . The specific contributions of this paper are as follows : 1 . We study lossily compressing the models downloaded by the clients , thus addressing the open question as to whether these approaches are amenable in the context of server-to-client exchanges . We also introduce the use of the theoretically motivated Kashin ’ s representation to reduce the error associated with the lossy compression ( Lyubarskii & Vershynin , 2010 ; Kashin , 1977 ) . 2 . We introduce Federated Dropout , a technique that builds upon the popular idea of dropout ( Srivastava et al. , 2014 ) , yet is primarily motivated by systems-related concerns . Our approach enables each device to locally operate on a smaller sub-model ( i.e . with smaller weight matrices ) while still providing updates that can be applied to the larger global model on the server . It thus reduces communication costs by allowing for these smaller sub-models to be exchanged between server and clients , while also reducing the computational cost of local training . 3 . We empirically show that not only are these approaches compatible with one another , but with existing client-to-server compression . Combining these approaches during FL training ( see Figure 1 ) reduces the size of the downloaded models up to 14× , the size of the corresponding updates up to 28× , and the required local computations by up to 1.7× , all without degrading the model ’ s accuracy and only at the expense of a slightly slower convergence rate ( in terms of number of communication rounds ) . 2 RELATED WORK . We review the relevant related work given our objective of reducing the communication footprint in server-to-client exchanges in Federated Learning ( FL ) . Federated Learning Federated Learning ( FL ) is a technique that aims to learn a global model over data distributed across multiple edge devices ( usually mobile phones ) without the data ever leaving the device on which it was generated ( McMahan et al. , 2017 ) . It brings along a set of statistical ( nonIID , unbalanced data ) and systems ( stragglers , communication bottlenecks , etc . ) challenges which differentiate it from traditional distributed learning in the data center , and which have been tackled by several works . For instance , McMahan et al . ( 2017 ) propose Federated Averaging ( FedAvg ) , which in its canonical form works by ( 1 ) sending the global model to a subset of the available devices , ( 2 ) training the model on each device using the available local data , and ( 3 ) averaging the local updates to thus end a round of training . In contrast Smith et al . ( 2017 ) present a multi-task variant that also models the relationship between clients in order to learn personalized yet related models for each device . Nonetheless , all approaches we are aware of ( including the two aforementioned ones ) require continued exchanges between a central server and its clients across a potentially slow network . Communication-efficient distributed learning Distributed learning is known to suffer from communication overheads associated with the frequent gradient updates exchanged among nodes ( Wang et al. , 2018 ; Dean et al. , 2012 ; Smith et al. , 2018 ; Reddi et al. , 2016 ) . To reduce these bottlenecks , recent studies focus on communicating a sparsified , quantized or randomly subsampled version of the updates . Although these operations introduce noise , they have been shown both empirically and theoretically to maintain the quality of the trained models . We refer the reader to the introduction of Wang et al . ( 2018 ) for more details and references . In the context of FL , Konečný et al . ( 2016b ) successfully perform lossy compression on the clientto-server exchanges ( i.e . the model updates ) . Of particular interest is their use of the randomized Hadamard transform to reduce the error incurred by the subsequent quantization . This is due to the fact that the Hadamard transform , in expectation , spreads a vector ’ s information more evenly across its components ( Suresh et al. , 2017 ; Konečný & Richtárik , 2016 ) . We note , however , that neither the work on traditional distributed learning nor the work of Konečný et al . ( 2016b ) considers compressing the server-to-client exchanges . Nevertheless , in FL , downloading a large model can still be a considerable burden for users , particularly for those in regions with network constraints . Furthermore , as FL is expected to deal with a large number of devices , communicating the global model may even become a bottleneck for the server ( as it would , ideally , send the model to the clients in parallel ) . Model compression Deep models tend to demand significant computational resources both for training and inference . Using them on edge devices is therefore not a straightforward task . Because of this , several recent works have proposed compressing the models before deploying them on-device ( Ravi , 2018 ) . Popular alternatives include pruning the least useful connections in a network ( Han et al. , 2016 ; 2015 ) , weight quantization ( Hubara et al. , 2016 ; Lin et al. , 2017 ; De Sa et al. , 2018 ) , and model distillation ( Hinton et al. , 2015 ) . Many of these approaches , however , are not applicable for the problems addressed in this work , as they are either ingrained in the training procedure ( and our server holds no data and performs no actual training ) or are mostly optimized for inference . In the context of FL , we need something computationally light that can be efficiently applied in every round and that also allows for subsequent local training . We do note , however , that some of the previously mentioned approaches could potentially be leveraged at inference time in the federated setting , and exploring these directions would be an interesting avenue for further research . 3 METHODS . In this section , we present our proposed strategies for reducing Federated Learning ’ s ( FL ) serverto-client communication costs , namely lossy compression techniques ( Section 3.1 ) and Federated Dropout ( Section 3.2 ) . We introduce the strategies separately , but they are fully compatible with one another ( as we show in Section 4.4 ) . 3.1 LOSSY COMPRESSION . Our first approach at reducing bandwidth usage consists of using lightweight lossy compression techniques that can be applied to an already trained model and that , when reversed ( i.e . after decompression ) , maintain the model ’ s quality . The particular set of techniques we propose are inspired by those successfully used by Konečný et al . ( 2016b ) to compress the client-to-server updates . We apply them , however , to the server-to-client exchanges , meaning we do not get the benefit of averaging the noisy decompressions over many updates . Our method works as follows : we reshape each to-be-compressed weight matrix in our model into a vector w and ( 1 ) apply a basis transform to it . We then ( 2 ) subsample and ( 3 ) quantize the resulting vector and finally send it through the network . Once received , we simply execute the respective inverse transformations to finally obtain a noisy version of w. Basis transform Previous work ( Lyubarskii & Vershynin , 2010 ; Konečný et al. , 2016b ) has explored the idea of using a basis transform to reduce the error that will later be incurred by perturbations such as quantization . In particular , Konečný et al . ( 2016b ) use the random Hadamard transform to more evenly spread out a vector ’ s information among its dimensions . We go even further and also apply the classical results of Kashin ( 1977 ) to spread a vector ’ s information as much as possible in every dimension ( Lyubarskii & Vershynin , 2010 ) . Thus , Kashin ’ s representation mitigates the error incurred by subsequent quantization compared to using the random Hadamard transform . For a more detailed discussion , we refer the reader to Section A.3 in the Appendix . Subsampling For s ∈ [ 0 , 1 ) , we zero out a 1 − s fraction of the elements in each weight matrix , appropriately re-scaling the remaining values . The elements to zero out are picked uniformly at random . Thus , we only communicate the non-zero values and a random seed which allows recovery of the corresponding indices . Probabilistic quantization For a vector w = ( w1 , . . . , wn ) , let us denote wmin = minj { wj } nj=1 and wmax = maxj { wj } nj=1 . Uniform probabilistic 1-bit quantization replaces every element wi by wmin with probability wmax−wiwmax−wmin , and by wmax otherwise . It is straightforward to verify this yields an unbiased estimate of w. Now , for q-bit uniform quantization , we first equally divide [ wmin , wmax ] into 2q intervals . If wi falls in the interval bounded by w′ and w′′ , the quantization operates by replacing wmin and wmax in step two of the above algorithm by w′ and w′′ , respectively . 3.2 FEDERATED DROPOUT . To further reduce communication costs , we propose an algorithm in which each client , instead of locally training an update to the whole global model , trains an update to a smaller sub-model . These sub-models are subsets of the global model and , as such , the computed local updates have a natural interpretation as updates to the larger global model . We call this technique Federated Dropout as it is inspired by the well known idea of dropout ( Srivastava et al. , 2014 ) , albeit motivated primarily by systems-level concerns rather than as a strategy for regularization . In traditional dropout , hidden units are multiplied by a random binary mask in order to drop an expected fraction of neurons during each training pass through the network . Because the mask changes in each pass , each pass is effectively computing a gradient with respect to a different submodel . These sub-models can have different sizes ( architectures ) depending on how many neurons are dropped in each layer . Now , even though some units are dropped , in all implementations we are aware of , activations are still multiplied with the original weight matrices , they just have some useless rows and columns . To extend this idea to FL and realize communication and computation savings , we instead zero out a fixed number of activations at each fully-connected layer , so all possible sub-models have the same reduced architecture ; see Figure 2 . The server can map the necessary values into this reduced architecture , meaning only the necessary coefficients are transmitted to the client , re-packed as smaller dense matrices . The client ( which may be fully unaware of the original model ’ s architecture ) trains its sub-model and sends its update , which the server then maps back to the global model1 . For convolutional layers , zeroing out activations would not realize any space savings , so we instead drop out a fixed percentage of filters . This technique brings two additional benefits beyond savings in server-to-client communication . First , the size of the client-to-server updates is also reduced . Second , the local training procedure now requires a smaller number of FLOPS per gradient evaluation , either because all matrixmultiplies are now of smaller dimensions ( for fully-connected layers ) or because less filters have to be applied ( for convolutional ones ) . Thus , we reduce local computational costs .

Summary: The paper presents some new approaches for communication efficient Federated Learning (FL) that allows for training of large models on heterogeneous edge devices. In FL, heterogeneous edge devices have access to potentially non-iid samples of data points and try to jointly learn a model by averaging their local models at a parameter server (the cloud). As the bandwidth of the up/downlink-link may be limited communication overheads may become the bottleneck during FL. Moreover, due to the heterogeneity of the hardware, large models may be hard to train on small devices. Due to that, there are several recent approaches that aim to minimize communication via methods of quantization, which also aim to allow for smaller models via methods of compression and model quantization.