On Federated Learning of Deep Networks from Non-IID Data: Parameter Divergence and the Effects of Hyperparametric Methods

Federated learning , where a global model is trained by iterative parameter averaging of locally-computed updates , is a promising approach for distributed training of deep networks ; it provides high communication-efficiency and privacypreservability , which allows to fit well into decentralized data environments , e.g. , mobile-cloud ecosystems . However , despite the advantages , the federated learning-based methods still have a challenge in dealing with non-IID training data of local devices ( i.e. , learners ) . In this regard , we study the effects of a variety of hyperparametric conditions under the non-IID environments , to answer important concerns in practical implementations : ( i ) We first investigate parameter divergence of local updates to explain performance degradation from non-IID data . The origin of the parameter divergence is also found both empirically and theoretically . ( ii ) We then revisit the effects of optimizers , network depth/width , and regularization techniques ; our observations show that the well-known advantages of the hyperparameter optimization strategies could rather yield diminishing returns with non-IID data . ( iii ) We finally provide the reasons of the failure cases in a categorized way , mainly based on metrics of the parameter divergence . 1 INTRODUCTION . Over the recent years , federated learning ( McMahan et al. , 2017 ) has been a huge success to reduce the communication overhead in distributed training of deep networks . Guaranteeing competitive performance , the federated learning permits each learner to compute their local updates of each round for relatively many iterations ( e.g. , 1 epoch , 10 epochs , etc . ) , which provides much higher communication-efficiency compared to the conventional data parallelism approaches ( for intra-datacenter environments , e.g. , Dean et al . ( 2012 ) ; Chen et al . ( 2016 ) ) that generally require very frequent gradient aggregation . Furthermore , the federated learning can also significantly reduce data privacy and security risks by enabling to conceal on-device data of each learner from the server or other learners ; thus the approach can be applied well to environments with highly private data ( e.g. , personal medical data ) , it is now emerging as a promising methodology for privacypreserving distributed learning along with differential privacy-based methods ( Hard et al. , 2018 ; Yang et al. , 2018 ; Bonawitz et al. , 2019 ; Chen et al. , 2019 ) . On this wise , the federated learning takes a simple approach that performs iterative parameter averaging of local updates computed from each learners ’ own dataset , which suggests an efficient way to learn a shared model without centralizing training data from multiple sources ; but hereby , since the local data of each device is created based on their usage pattern , the heterogeneity of training data distributions across the learners might be naturally assumed in real-world cases . Hence , each local dataset would not follow the population distribution , and handling the decentralized non-IID data still remains a statistical challenge in the field of federated learning ( Smith et al. , 2017 ) . For instance , Zhao et al . ( 2018 ) observed severe performance degradation in multi-class classification accuracy under highly skewed non-IID data ; it was reported that more diminishing returns could be yielded as the probabilistic distance of learners ’ local data from the population distribution increases . Algorithm 1 Federated learning . wtk is the model parameters updated by learner k at communication round t , wt is the global model parameters at round t , K = { 1 , 2 , · · · , K } is the universal set of learners , Pk is the local training dataset of learner k , B is the local minibatch size , E is the number of the local epochs per round , η is the learning rate , and ` ( · ) is the loss function . Server executes : initialize w0 for each round t = 1 , 2 , · · · , T do for each learner k ∈ K in parallel do wtk ← LearnerUpdate ( k , wt−1 ) end for wt ← ∑ k∈K |Pk|∑ j∈K|Pj | wtk end for LearnerUpdate ( k , w ) : // Run on learner k B ← ( split Pk into batches of size B ) for each local epoch ε from 1 to E do for each batch b ∈ B do w← w − η∇ ` ( w ; b ) end for end for return w to server Contributions . To address the non-IID issue under federated learning , there have been a variety of recent works1 ; nevertheless , in this paper we explore more fundamental factors , the effects of various hyperparameters . The optimization for the number of local iterations per round or learning rates has been handled in several literatures ( e.g. , Huang et al . ( 2018 ) ; Li et al . ( 2019c ) ; Wang et al . ( 2019 ) ) ; by extension we discuss , for the first time to the best of our knowledge , the effects of optimizers , network depth/width , and regularization techniques . Our contributions are summarized as follows : First , as a root cause of performance degradation from non-IID data , we investigate parameter divergence of local updates at each round . The parameter divergence can be regarded as a direct response to learners ’ local data being non-IID sampled from the population distribution , of which the excessive magnitude could disturb the performance of the consequent parameter averaging . We also investigate the origin of the parameter divergence in both empirical and theoretical ways . Second , we observe the effects of well-known hyperparameter optimization methods2 under the non-IID data environments ; interestingly , some of our findings show highly conflicted aspects with their positive outcomes under “ vanilla ” training3 or the IID data setting . Third , we analyze the internal reasons of our observations in a unified way , mainly using the parameter divergence metrics ; it is identified that the rationale of the failures under non-IID data lies in some or all of ( i ) inordinate magnitude of parameter divergence , ( ii ) its steep fall phenomenon ( described in Section 4.2 ) , and ( iii ) excessively high training loss of local updates . 2 PRELIMINARIES . 2.1 ALGORITHM . In this study , Algorithm 1 is considered as a federated learning method , and it is written based on FedAvg ( McMahan et al. , 2017 ) .4 We note that this kind of parameter averaging-based approach has been widely discussed in the literature , under various names , e.g. , parallel ( restarted ) SGD ( Zhang et al. , 2016 ; Yu et al. , 2019 ) and local SGD ( Lin et al. , 2018 ; Stich , 2019 ) . 2.2 EXPERIMENTAL SETUP . In our experiments with Tensorflow ( Abadi et al. , 2016 ) ,5 we consider the multi-class classification tasks on CIFAR-10 ( Krizhevsky & Hinton , 2009 ) and SVHN ( Netzer et al. , 2011 ) datasets . 1For instance , Smith et al . ( 2017 ) ; Caldas et al . ( 2018 ) ; Huang et al . ( 2018 ) ; Jeong et al . ( 2018 ) ; Zhao et al . ( 2018 ) ; Zhu & Jin ( 2018 ) ; Corinzia & Buhmann ( 2019 ) ; Duan ( 2019 ) ; Li et al . ( 2019a ; b ; c ) ; Liu et al . ( 2019 ) ; Mohri et al . ( 2019 ) ; Sattler et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Yonetani et al . ( 2018 ) ; Yoshida et al . ( 2019 ) . 2we use the term hyperparameter optimization methods and hyperparametric methods interchangeably . 3This term refers to the non-distributed training with a single machine , using the whole training examples . 4Regarding the significance of the algorithm , we additionally note that Google is currently employing it on their mobile keyboard application ( Gboard ) ( Hard et al. , 2018 ; Yang et al. , 2018 ; Bonawitz et al. , 2019 ; Chen et al. , 2019 ) . In this study we deal with image classification , which is also considered as the main applications of federated learning along with the language models ( McMahan et al. , 2017 ) . 5Our source code is available at https : //github.com/fl-noniid/fl-noniid Baseline network model : For the baseline deep network , we consider a CNN model that has three 3 × 3 convolutional layers with 64 , 128 , 256 output channels , respectively ; and then three fully-connected layers with 512 , 256 , 10 output sizes , respectively ( see Appendix A.1 for more detailed description ) . We use the term NetA-Baseline to denote this baseline model throughout this paper . Regularization configuration : For weight decay , we apply the method of decoupled weight decay regularization ( Loshchilov & Hutter , 2019 ) based on the fact that weight decay is equivalent to L2 regularization only for pure SGD ( Loshchilov & Hutter , 2019 ; Zhang et al. , 2019 ) . The baseline value of the weight decay factor is set to 0.00005 . As our regularization baseline , we consider not to apply any other regularization techniques additionally . We importantly note that if without any particular comments , the results described in the following sections are ones obtained using the above baseline configurations of the network model and regularization . Environmental configuration : We consider 10 learners to have each 5000 nonoverlapping training examples ; Table 1 summarizes our configuration of data settings ; Non-IID ( N ) denotes a data setting that lets each learner to have training examples only for N class ( es ) . The data settings in the Table 1 deal with data balanced cases where learners have the same amount of local data , and they are mainly considered in the follow- ing sections ; we additionally note that one can refer to Appendix C.8 for the experiments with data unbalanced cases . For the IID and the non-IID data settings , T = 200 and 300 are used respectively , while E = 1 and minibatch size of 50 are considered commonly for the both.6 One can find the remaining configurations for the experiments in Appendix A . 3 PARAMETER DIVERGENCE . Parameter divergence is recently being regarded as a strong cause of diminishing returns from decentralized non-IID data in federated learning ( Zhao et al. , 2018 ) ( it is sometimes expressed in another way , gradient/loss divergence ( Li et al. , 2019b ; c ; Liu et al. , 2019 ; Wang et al. , 2019 ) ) . For the divergence metrics , many of the literatures usually handle the difference of each learner ’ s local model parameters from one computed with the population distribution ; it eventually also causes parameter diversity between the local updates as the data distributions become heterogeneous across learners . A pleasant level of parameter divergence could rather imply exploiting rich decentralized data ( IID cases ) ; however , if the local datasets are far from the population distribution , the consequent parameter averaging of the highly diverged local updates could lead to bad solutions away from the global optimum ( non-IID cases ) . 6Here , one epoch is used as one of the standard local iterations per round in McMahan et al . ( 2017 ) and its large number of descendants ; furthermore , more extreme cases such as E = 5 , 20 are also considered . In addition , in view of the number of local iterations , we use 100 for it during each round ; note that 100 iterations are even considered ( e.g. , up to 200 iterations is used on CIFAR-10 in McMahan et al . ( 2017 ) ) . The origin of parameter divergence . In relation , it has been theoretically proven that the parameter divergence ( between the global model parameters under FedAvg and those computed by vanilla SGD training ) is directly related to the probabilistic distance of local datasets from the population distribution ( see Proposition 3.1 in Zhao et al . ( 2018 ) ) . In addition to it , for multi-class classification tasks , we here identify in lower level , that if data distributions in each local dataset are highly skewed and heterogeneous over classes , subsets of neurons , which have especially big magnitudes of the gradients in back propagation , become significantly different across learners ; this leads to inordinate parameter divergence between them . As illustrated in Figure 1 , under the IID data setting , the weight values in the output layer are evenly distributed relatively evenly across classes if the neurons of the model are initialized uniformly . However , we can observe under the non-IID data settings that the magnitudes of the gradients are distributed depending on each learner ’ s data distribution . We also provide the corresponding theoretical analysis in Appendix B. Metrics . To capture parameter divergence under federated learning , we define the following two metrics using the notations in Algorithm 1 . Since in our analysis we compare different network architectures or training settings together in a set , the number of neurons in the probed layers can become different , and values of model parameters can highly depend on the experimental manipulations ; thus instead of Euclidean distance , in the two divergence metrics we use cosine distance that enables normalized ( qualitative ) measures . We also note that PD-VL is defined assuming the balancedness of data amount between learners , i.e. , the same numbers of local iterations per round . The reason of probing parameter divergence being important is that the federated learning are performed based on iterative parameter averaging . That is , investigating how local updates are diverged can give a clue whether the subsequent parameter averaging yields positive returns ; the proposed divergence metrics provide two ways for it . Definition 1 . For zt1 , zt2 , · · · , ztK ∈ Rd where ztk is a subset ( or the universal set ) of wtk , we define parameter divergence between local updates as PD-Ls : ζ ( zt1 , z t 2 , · · · , ztK ) = ( K 2 ) −1 ∑ i , j∈K ; i < j ( 1− zti · ztj ‖zti‖ ∥∥ztj∥∥ ) .

Summary: The paper presents an empirical study of causes of parameter divergence in federated learning. Federated learning is the setting where parameter updates (e.g. gradients) are computed separately on possibly non-IID subsamples of the data and then aggregated by averaging. The paper examines the effects of choice of optimizer, network width and depth, batch normalization, weight decay, and data augmentation on the amount of parameter divergence. Divergence is defined as the average cosine distance between pairs of locally-updated weights, or between locally updated weights and weights trained with IID data. The paper generally concludes that regularization methods like BN and weight decay have an adverse effect in the federated setting, the deepening the network has an adverse effect while widening it might be beneficial, and that adaptive optimizers like Adam can perform poorly if their internal statistics are not aggregated.