On Federated Learning of Deep Networks from Non-IID Data: Parameter Divergence and the Effects of Hyperparametric Methods
Authors: Heejae Kim
Taewoo Kim
Chan-Hyun Youn
Venue:  (2020)
Keywords: Federated learning
Iterative parameter averaging
Deep networks
Decentralized non-IID data
Hyperparameter optimization methods

Abstract: Federated learning, where a global model is trained by iterative parameter averaging of locally-computed updates, is a promising approach for distributed training of deep networks; it provides high communication-efficiency and privacy-preservability, which allows to fit well into decentralized data environments, e.g., mobile-cloud ecosystems. However, despite the advantages, the federated learning-based methods still have a challenge in dealing with non-IID training data of local devices (i.e., learners). In this regard, we study the effects of a variety of hyperparametric conditions under the non-IID environments, to answer important concerns in practical implementations: (i) We first investigate parameter divergence of local updates to explain performance degradation from non-IID data. The origin of the parameter divergence is also found both empirically and theoretically. (ii) We then revisit the effects of optimizers, network depth/width, and regularization techniques; our observations show that the well-known advantages of the hyperparameter optimization strategies could rather yield diminishing returns with non-IID data. (iii) We finally provide the reasons of the failure cases in a categorized way, mainly based on metrics of the parameter divergence.

Under review as a conference paper at ICLR 2020
ON FEDERATED LEARNING OF DEEP NETWORKS FROM NON-IID DATA: PARAMETER DIVERGENCE AND THE EFFECTS OF HYPERPARAMETRIC METHODS
Anonymous authors Paper under double-bling brind review
ABSTRACT
Federated learning, where a global model trained by iterative parameter aver- aging of locally-computed updates, is promising approach for distributed train- ing of deep networks; it provides high communication--eficienng and privacy- preservability, which allows to fit well into decentralized data environments, e.g., mobile-cloud ecosystems. However, despite the advantages, the federated learning-basc bassed methods still have challenge in dealing with non-lID training data of local devices (i.e., learners). In this regard, we study the effects of a va- answer riety of hyperparametric conditions under the non-IID environments, important concerns in practical implementations: (i) We first investigate parame- ter divergence of local updates to explain performance degradation from non-IID data. The origin of the parameter divergence is also found both empirically and theoretically. (ii) We then revisit the effects of optimizers, network depth/width, and regularization techniques; our observations show that the well-known advan- tages of the hyperparameter optimization strategies could rather yield diminishing returns with non-lID data. (iii) We finally provide the reasons of the failure cases in a categorized way, mainly based on metrics of the parameter divergence.
INTRODUCTION
Over the recent years, federated learning (McMahan et al. 2017) has been huge success to re- duce the communication overhead in distributed training of deep networks. Guaranteeing compet- itive performance, the federated learning permits each learner to compute their local updates of each round for relatively many iterations (e.g., epoch, 10 epochs, etc.), which provides much higher communication-effiiciii intra-datacenter environments, c.g., Dean et al. (2012): [Chnn et al. (2016)) that generally require very frequent gradient aggregation. Furthermore, the federated learning can also significantly duce data privacy and security risks by enabling to conceal on-device data of each learner from the server other learners; thus the approach can be applied well environments with highly private data (e,g., personal medical data), it is now emerging as a promising methodology for privacy- preserving distributed learning along with differential privacy-based methods (Hard al. 2018
compared to the conventional data parallelism approaches (for
Yang et al. 2018 Bonawitz et al. 2019 Chen et al. 2019).
On this wise, the federated learning takes a simple approach that performs iterative parameter aver- aging of local updates computed from each learners' own dataset, which suggests an efficient way to learn a shared model without centralizing training data from multiple sources; but hereby, since the local data of each device is created based on their usage pattern, the heterogeneity of training data distributions across the learners might be naturally assumed in real-world cases. Hence, each local dataset would not follow the population distribution, and handling the decentralized non-llI data still remains a statistical challenge in the field of federated learning (Smith et al.) (017). For instance, Zhao et al. (2018) observed severe performance degradation in multi-class classification accuracy under highly skewed non-IID data; it was reported that more diminishing returns could be yielded as the probabilistic distance of learners' local data from the population distribution increases.

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Federated learning. w { is the model parameters updated by learner k communication
round i, wf is the global model parameters round t, K. {1, 2, .... K} is the universal set of learners, Pk is the local training dataset of learner k. B is the local minibatch size, E' is the number of the local epochs per round, 1) is the learning rate, and ((-) the loss function.
Server exccutes:
I LearneUUpdate(k; w): // Run on learner B <split Pe: into batches of size B) for each local epoch E from I to E do for each batch b E B do
initialize w
for each round t = , do for each learner k E K. in parallel do w <- LearnerUpdattek; w end for w
w f- w r)Ve(w;)) end for end for return w to server
ΣKER
[Pal
w
E5ex
end for
Contributions, To address the non-llD issue under federated learning, there have been variety of recent work(] nevertheless, in this paper we explore more fundamental factors, the effects of various hyperparameters. The optimization for the number of local iterations per round learning rates has been handled in several literatures (c.g.. Huang et al. 22188); Li et al. (2019c); Wang et al. (2019); by extension we discuss, for the first time the best of our knowledge, the effects of optimizers, network depth/width, and regularization techniques.
Our contributions are summarized as follows: First, as a root cause of performance degradation from non-IID data, we investigate parameter divergence of local updates at each round. The parameter divergence can be regarded as a direct response to learners' local data being non-IID sampled from the population distribution, of which the excessive magnitude could disturb the performance of the consequent parameter averaging. We also investigate the origin of the parameter divergence in both empirical and theoretical ways. Second, we observe the effects of well-known hyperparameter optimization methods" under the non-IID data environments; interestingly, some of our findings show highly conflicted aspects with their positive outcomes under "vanilla" training or the IID data setting. Third, we analyze the internal reasons of our observations in a unified way, mainly using the parameter divergence metrics; it is identified that the rationale of the failures under non-lID data lies in some or all of (i) inordinate magnitude of parameter divergence, (ii) its steep fall phenomenon (described in Section 4.2). and (iii) excessively high training loss of local updates.
PRELIMINARIES
2.
ALGORITHM
In this study, Algorithm is considered as federated learning method, and it is written based on FedAvg (McMahan al. 2017) We note that this kind of parameter averaging-based approach has been widely discussed in the literature, under various names, e.g.. parallel (restarted) SGD (Zhang et al. 2016 Yu et al. 2019) and local SGD (Lin et al. 2018 Stich 2019).
2.2 EXPERIMENTAL SETUP
In experiments with Tensorflow (Abadi et al.
2016
consider the multi-class classifica-
tion tasks on CIFAR-10 (Krizhevsky & Hinton 2009) and SVHN (Netzer et al. 2011) datasets.
FFr instance, Smith (0088 Zhu Jin 2018) Corinzia & Buhmann 2019g Duan (2019) y et al. (2019) 9a blc :
2017): Caldas al. 2018 Huang al. 20118 Jeong al. 2018): Zhao et al.
(2019);
Mohri et al. (2019) Sattler et (2019)) Wang et al. ( (2019) Yonetani et al. (2018); Yoshida et al. (2019).
*we use the term hyperparameter oprimization methods and hyperparametric methods interchangeably.
3This term refers to the non-distributed training with a single machine, using the whole training examples.
'Regarding the significance of the algorithm, additionally note that Google currently employing it on
their mobile keyhoard application (Gboard) (Hard al. 2018 Yang al. 2018 Honawitz al. 2019 Chen
ct al. 2019). In this study we deal with image classification, which is also considered as the main applications
of federated lcarning along with the language models (McMahan et al. (017).
"Our source code is available https:: thub.
nonni
2

---

Under review as a conference paper at ICLR 2020
IID
Non-IID(2)
Non-IID(1)
Figure 1: Magnitude of neurons in the last fully-connected layer of the NetA-Baseline model, at the first round of Algorithm [ on CIFAR-10. Brighter ones illustrates having greater magnitude.
Baseline network model: For the baseline deep network, we consider CNN model that has three 3 x 3 convolutional layers with 64, 128, 256 output channels, respectively; and then three fully-connected layers with 512, 256, 10 output sizes, respectively (see Appendix [Λ.]] for more detailed description). We use the term NetA-Baseline to denote this baseline model throughout this paper. Regularization configuration: For weight decay, we apply the method of decou- pled weight decay regularization (Loshchilov & Hutter 2019) based on the fact that weight decay is equivalent to L2 regularization only for pure SGD (Loshchilov & Hutter 2019 Zhang et al.) 2019). The baseline value of the weight decay factor is set 0.00005. As line, we consider not to apply any other regularization techniques additionally. We importantly note that if without any particular comments, the results described in the following sections ones obtained using the above baseline configurations of the network model and regularization. Environmental configuration: We con- sider 10 learners to have each 5000 non- overlapping training examples; Table [ sum- marizes our configuration of data settings; Non-IID(N) denotes data setting that lets each learner to have training examples only for class(es). The data settings in the Ta- ble M deal with data balanced cases where learners have the same amount of local data, and they are mainly considered in the follow-
regularization base-
Table 1: Configuration of balanced data settings. #Cls/L: the number of classes in each learner's local training dataset; #Exs/CI/L: the number of training examples per class in each local dataset.
Data Setting
#Learners
#Cls/L
#Exs/CI/L
IID Non-IID(N)
10
10
500
10
N
5000/N
ing sections; we additionally note that one can refer to Appendix C.8 for the experiments with data unbalanced cases. For the IID and the non-IIID data settings, T 200 and 300 are used respectively,
while E = 1 and minibatch size of 50 are considered commonly for the both 5 One can find the remaining configurations for the experiments Appendix
PARAMETER DIVERGENCE
Parameter divergence is recently being regarded strong cause of diminishing returns from de- centralized non-lID data in federated learning (Zhao et al. 2018) (it is sometimes expressed in another way, gradient/loss divergence (Li et al. 2019b c Liu et 2019 Wang al., 2019)) ). For the divergence metrics, many of the literatures usually handle the difference of each leamer's 's local model parameters from one computed with the population distribution; eventually also causes pa- rameter diversity between the local updates as the data distributions become heterogeneous across learners. A pleasant level of parameter divergence could rather imply exploiting rich decentralized data (IID cases); however, if the local datasets are far from the population distribution, the conse- quent parameter averaging of the highly diverged local updates could lead to bad solutions away from the global optimum (non-IID cases).
'Here, one epoch is used as one of the standard local iterations per round in McMahan et al. (017)) and
its large number of descendants; furthermore, more extreme cases such as E = 5, 20 are also considered. In addition, in view of the number of local iterations, we use 100 for it during each round; note that > 100 iterations considered (e.g., up to 200 iterations is used CIFAR-10 10 McMahan al. (2017).
3

---

Under review as a conference paper at ICLR 2020
Food Assuraty Gloie Mocill
Weghtt of the Lust FulyCormeedd Layar
Veagltt of the Last Cull-Crrrede Laywr
PDVR
0
100
200
Communication Rounds
Communictionn Rounds
Communication Rounds
Figure 2: Comparison of test accuracy and parameter divergence with respect to the degree of data non-IIDness in the training of NetA- Baseline on CIFAR- 10. Note that EMD values of Non-IID(5) and Non- from the population distribution 1.0 and 1.6, respectively.
The origin of parameter divergence. In relation, it has been theoretically proven that the parame- ter divergence (between the global model parameters under FedAvg and those computed by vanilla SGD training) is directly related to the probabilistic distance of local datasets from the population distribution (see Proposition 3.1 in Zhao et al. (2018)). In addition to it, for multi-class classifica- tion tasks, we here identify lower level, that if data distributions in each local dataset are highly skewed and heterogeneous over classes, subsets of neurons, which have especially big magnitudes of the gradients back propagation, become significantly different across learners; this leads to inordinate parameter divergence between them. As illustrated in Figure [I] under the IID data set- ting, the weight values in the output layer are evenly distributed relatively evenly across classes the neurons of the model are initialized uniformly. However, we can observe under the non-IlI) data settings that the magnitudes of the gradients are distributed depending on each learner's 's data distribution. We also provide the corresponding theoretical analysis Appendix B
Metries. To capture parameter divergence under federated learning, we define the following two metries using the notations Algorithm [] Since our analysis we compare different network architectures or training settings together in set, the number of neurons in the probed layers can become different, and values of model parameters can highly depend on the experimental manip- ulations; thus instead of Euclidean distance, in the two divergence metrics that enables normalized (qualitative) measures. We also note that PD-VL is defined assuming the balancedness of data amount between learners, i.e., the same numbers of local iterations per round.
cosine distance
The reason of probing parameter divergence being important is that the federated learning are per- formed based on iterative parameter averaging. That is, investigating how local updates are diverged can give a clue whether the subsequent parameter averaging yields positive returns; the proposed divergence metries provide ways for it.
For zi, zi....,, .Zk E Ed where z is a subset (or the universal set) of wh we define parameter divergence between local updates as
Definition 1.
z z
PD-Ls: C (2), ... 2xxx
2b,
1
|
23
KJEKAKK
In addition, assume that |P&] is identical Vk K, and let w' be the vanilla-updated parame- ters, that is, the model parameters updated on the global parameters (i.e., w using IID train- ing data during the same number of iterations with the actual learners (i.e., |Px|/B). Then, for zi,, zi....,, .,,,, and z'_ E Bd where z', is a subset (or the universal set) of w' we define parameter divergence between the vanilla update and local updates as
PD-VL: : (z'ni
z(x)
1 K
Σ
2
Z
1
Relationship among probabilistic distance, parameter divergence, and learning performance. We consider Non-IID(5) and Non-IID(2) for non-lID data settings. Here we use earth mover's distance (EMD), also known Wasserstein distance, to measure probabilistic distance of each data
4

---

Under review as a conference paper at ICLR 2020
settings from the population distribution; the value becomes 1.0 and 1.6 for Non-IID( and Non- IID(2). respectively. From the middle and right panels of Figure 2 it is seen that greater EMDs lead to bigger parameter divergence (refer to also Figure 9 in the appendix). Also, together with the left panel, we can observe the positive correlation between parameter divergence and learning performance. Therefore, we believe the parameter divergence metries can help to reveal the missing link between data non-lIDness and the consequent learning performance. Note that one can also refer the similar analysis with more various EMD in Zhao al. (2018).
THE EFFECTS OF HYPERPARAMETREE METHODS
4.1 SUMMARY OUR OBSERVATIONS
From now on we describe our findings for the effects of various hyperparameter optimization meth- ods with non-IID data on the federated learning algorithm. The considered hyperparametric methods have been a huge success to improve performance in deep learning; however, here we newly iden- tify that under non-IID data settings, they could give negative/diminishing effects on performance of the federated learning algorithm. The following is the summary of our findings; we provide the complete experimental results and further discussion in the next subsection and the appendix.
Effects of optimizers. Unlike non-adaptive optimizers such as pure SGD and momentum SGD (Polyak] 1964; Nesterov 1983). Adam (Kingma & Ba 2013) could give poor performance from non-lID data if the parameter averaging is performed only for weights and biases, compared to all the model variables (including the first and second moment) being averaged.
Here we importantly note that both momentum SGD and Adam require the additional variables related to momentum as well as weights and biases; throughout the rest of the paper, the terms (optimizer name)-A and (optimizer name)-We WB are used to refer to the parameter averaging being performed for all the variables and only for weights & biases, respectively.
Effects of network depth/width. It is also known that deepening "plain" networks (which simply stacks layers, without techniques such as information highways (Srivastava et al. 2015) and shortcut connection (He et al.) 2016)) yields performance degradation at a certain depth, even under vanilla training; however this phenomenon gets much worse under non-IID data environments. On the contrary, widening networks could help to achieve better outcomes; in that sense, the global average pooling (Lin al., (2014) could fail in this case since significantly reduces the channel dimension of the (last) fulle-connected layer, compared to using the max pooling.
Effeets of Batch Normalization. The well-known strength of Batch Normalization (loffe & Szegedy 2015), the dependence of hidden activations in the minibatch (Toffe, 2017). could be- come a severe drawback in non-lID data environments. Batch Renormalization loffe 2017 helps to mitigate this, but it also does not resolve the problem completely.
Effects of regularization techniques. With non-lID data, regularizations techniques such as weight decay and data augmentation could give excessively high training loss of local updates even in modest level, which offsets the generalization gain.
4.2 DISCUSSION
We now explain the internal reasons of the observations the previous subsection. Through the experimental results, we were able to classify the causes of the failures under non-IID data into
"To the best of our knowledge, far there have been no studies about Adam to synchronize all the three of variables (i.e., weights& biases, the first moment, and the second moment) under federated learning. However, in the momentum SGD case, there have the synchronization methods with local momentunn, global momentum, and hybrid momentun. Here our simple averaging strategy has the similar philosophy with the local momentum method; one can see from Table 4 in Lin et (2018) that the simple averaging strategy can yield still competitive results compared to global momentum hyhrid momentum method.
some literatures, for instance, [Lin et al. (2018) presented
5

---

Under review as a conference paper at ICLR 2020
Weights the lus Futy Gonnected
Weights Comeerionna
Werghts Puby Connected
10
PD-VL
10
10 Rontiiaasiio
Nomts--t chrppe
10
MD Fedar 3.
Facer: 0001
10
Duadinn
10'
10'
Communication Rounds
Communicttion Rourdds
Communication Rounds
Figure 3: Inordinate magnitude of parameter divergence in the CIFAR-10 training with NetA- Deepest (left), weight decay factor of 0.0005 (middle), and Dropout (right), respectively, under the non-IID data setting. Dotted and solid lines indicate the results under IID and Non-IID(2) set- ting. respectively, WD: weight decay; DO: dropout.
Test accuracy (%) of the trials in Figure 3 Values outside/inside brackets denote one measured after the whole training and the highest one during the rounds, respectively. NMom: Nesterov momentum SGD; BS: minibatch size. Note that weight decay factor of 0.00005 is the baseline configuration.
Table 2:
Network
Method
Vanilla Training
Federated Learning
BS: 50
BS: 500
IID
Non-IID(2)
NetA-Baseline
NMom-A
83.89 (83.90)
79.39 (79.49)
81.49 (81.22)
74.11 (74,23)
NetA-
NetA-Deeper
NMom-A
85.94 (86.00)
81.23 (81.29)
83.40 (83.63)
73.67 (73.89)
NetA- -eeeeeeps
NMom-A
86.20 (86.33)
79.48 (79.53)
83.12 (83.58)
68.98 (69.64)
SttA-Baseline
NMom-A + WD: 0.0
81.13 (81.15)
79.09 (79.25)
81.68 (81.90)
73.95 (64.27)
NetA- Mas-laselene
NMom-A + WD: 0.0001
84.29 (84.46)
79.13 (79.74)
82.22 (82.25)
72.65 (72.79)
NMom-A + WD: 0.0005
83.66 (84.60)
80.48 (80.79)
82.69 (83. 14)
54.11 (54.15)
NetA-Baseline
NMom-A + DO
86.55 (86.63)
84.29 (84.32)
84.34 (84.53)
75.80 (75.89)
three categories; the following discussions are described based on this Note that our discussion in this subsection is mostly made from the results under Nesterov momentum SGD and on CIFAR- 10: the complete results including other optimizers (e.g.. pure SGD, Polyak momentum SGD, and Adam) and datasets (e.g., SVHN) are given in Appendix
Inordinate magnitude of parameter divergence. As mentioned before, bigger parameter diver- gence is the root cause of diminishing returns under federated learning methods with non-IID data. By extension, here observe that even under the same non-IID data setting, some of the considered hyperparametric methods yield greater parameter divergence than when they are not applied.
For example, from the left plot of Figure we that under the Non-IID(2) setting, the parameter divergence values (in the last fully-connected layer) become greater as the network depth increases (note that NetA-Baseline, NetA-Deeper, and NetA-Deepest have 3, 6, and 9 convolutional lay- ers, respectively; see also Appendix I for their detailed architecture). The corresponding final test accuracy was found be 74.11% %. 73.67%, and 68.98% respectively, in order of the degree of shallowness; this fits well into the parameter divergence results. Since the NetA-Deeper and NetA-Deepest have twice and three times as many model parameters as NetA-Baseline expected enough that the deeper models yield bigger parameter divergence in the whole model; but our results also show its qualitative increase in a layer level. In relation, we also provide the results using the modern network architecture (e.g., ResNet (He et al. 2016)) in Table 8 of the appendix.
can be
'From the figures of the experimental results in Appendix we can identify that in most cases the parame- ter divergence values of the first convolutional layer and the last fully-connected layer would be more dominant than those the other layers, judging from their difference of magnitude between under the IID and the non- IID data setting (please also note that log scale is used for the y-axis). We additionally remark that the results of other related studies also show the dominance of the first convolutional layer and the last colnecco layer (e.g., see Figure 2 in Zhao et al. (2018)) Therefore, our discussion here was primarily descrihed based on the results of the first convolutional layer and the last fully-connected layer.

---

Under review as a conference paper at ICLR 2020
Weighte the Last Fut Germeeted
Weights the Fuly Connected
Weights the Conneledd
10
PDVI
1110
10
10
Alar Mi
10
Net Made
A
10"
10
10'
10'
Communication Rounds
Communkcation Rounds
Communiation Rounds
Figure 4: Steep fall phenomenon in the CIFAR-I0 training with Adam-WB (left), NetC-Baseline (middle), and Batch Normalization (right), respectively, under the non-llI data setting. Dotted and solid lines indicate the results under IID and Non-IID(2) setting. respectively. BN: Batch Normal- ization; BRN: Batch Renormalization.
Table 3: Test accuracy (%) of the trials in Figure
Network
Method
Vanilla Training
Federated Learning
BS: 50
BS: 500
IID
Non-IID(2)
NetA-Baseline
Adam-WB
81.97 (82.35)
80.73 (80.76)
80.99 (81.00)
67.51 (67.82)
Adam-A
81.97 (82.35)
80.73 (80.76)
81.18 (81.45)
75.32 (75.45)
NetC-Baseline
NMom-A
84,25 (84.50)
76.70 (76.82)
80.60 (80.75)
64.06 (64.55)
NetC-Wider
NMom-A
81.48 (81 (81.53)
77.00 (77.14)
79.59 (79.85)
72.61 (72.97)
NetC-
NMom-A
83.16 (83.36)
78.39 (78.56)
80.90 (81.14)
73.64 (73.91)
NetA-Baseline
NMom-A
83.89 (83.90)
79.39 (79.49)
81.49 (81.82)
74.11 (74.23)
BetA-Baseline
NMom-A BN
85.22 (85.25)
79.58 (80.14)
83.54 (83.80)
50.46 (59.31)
NelA-Baseline
NMom-A BRN
86.36 (86.43)
82.03 (82.46)
84.24 (84.70)
70.32 (70.38)
From the middle plot of the figure, we can also observe bigger parameter divergence a high level of weight decay under the Non-IID(2) setting. Under the non-III data setting, the test accuracy of about 72 ~ 74% was achieved in the low levels (S 0.0001), but weight decay factor of 0.0005 yielded only that of 54.11%. Hence, this suggests that with non-lID data we should apply much smaller weight decay to federated learning-bat based methods. Here we note that if a single iteration is considered for each learner's local update per round, the corresponding parameter divergence will be of course the same without regard to degree of weight decay. However, in our experiments, the great number of local iterations per round (i.e., 100) made a big difference of the divergence values under the non-lID data setting; this eventually yielded the accuracy gap. We additionally observe for the non-IID cases that even with weight decay factor of 0.0005, the parameter divergence values are similar to those with the smaller factors at very early rounds in which the norms of the weights are relatively very small.
In addition, it is observed from the right plot of the figure that Dropout (Hinton et al. 2012 Sri-
vatava al., (014)) also yields bigger parameter divergence under the non- data setting. The cor- responding test accuracy was seen to be a diminishing return with Nesterov momentum SGD (i.e., using Dropout we can achieve +2.85% under IID, but only + 11.69% is obtained under non- compared to when it is not applied; see Table however, it was observed that the generalization effect of the Dropout is still valid in test accuracy for the pure SGD and the Adam (refer to also Table 13 in the appendix).
Steep fall phenomenon. As we see previously, inordinate magnitude of parameter divergence is one of the notable characteristics for failure cases under federated learning with non-IlI data. However, under the non-IID data setting, some of the failure cases have been observed where the test accuracy is still low but the parameter divergence values of the last fully-connected layer decrease (rapidly) over rounds; as the round goes, even the values were sometimes seen to be lower than those of the comparison targets. We refer to this phenomenon that these (unexpected abnormal) sudden drops of parameter divergence values indicate going into poor local minima (or saddles); this can be supported by the behaviors that test accuracy increases
steep fall phenomenon. It is inferred
7

---

Under review as a conference paper at ICLR 2020
Adam-A at 25th Round
Adam WB at 25th Round
w/o BN at 25th Round
BN 25th Round
501
50
123
Acamm-A 300th Round
Adam-WB at 300th Round
w/o BN at 300th Round
w/ BN at 300th Round
Figure 5: Loss surface of the global model parameters under Non-IID(2) setting. The red dashed boxes indicate the same level of loss value.
plausibly at very early rounds, but the growth rate quickly stagnates and eventually becomes much lower than the comparison targets.
The left plot of Through the experiments, we identified that under non-IID data environments, the performance of Adam very sensitive to the range of model variables to be averaged, unlike the non-adaptive optimizers (e.g., momentum SGD); its moment variables should be also considered in the param- eter averaging together with weights and biases (see also Table 33. The poor performance of the Adam- Adam-WB under the Non-IID(2) setting would be from twice many momentum variables as the momentum SGD, which indicates the increased number of them affected by the non-IIDness; thus, originally we had thought that extreme parameter divergence could appear if the momentum vari- ables are not averaged together with weights and biases. However, it was seen that the parameter divergence values under the Adam-WB was seen to be similar or even smaller than under Adam-A (see also Figure [1 in the appendix). Instead, from the left panel we can observe that the parame- ter divergence of Adam-We WB in the last connetco layer is bigger than that of Adam-A the very early rounds (as we expected), but soon it is abnormally sharply reduced over rounds; this is considered the steep fall phenomenon.
shows the effect of the Adam optimizer with respect to its implementations.
The middle and the right plots of the figure also show the steep fall phenomenon in the last fully- connected layer, with respect to network width and whether to use Batch Normalization, respec- tively. In the case of the NetC models, NetC-Baseline NetC-Wider, and NetC-Widest use the global average pooling, the max pooling with stride 4, and the max pooling with stride 2, respec- tively, after the last convolutional layer; the number of neurons in the output layer becomes 2560, 10240, and 40960, respectively (see also Appendix [ for their detailed architecture). Under the Non-IID(2) setting, the corresponding test accuracy respectively, in order of the degree of wideness. In addition, we can see that under Non-IID(2), Batch Normalization yyillds yields not only big parameter divergence (especially before the first learning rate drop) but also the steep fall phenomenon; the corresponding test accuracy was seen to be very low (see Table The failure of the Batch Normalization stems from that the dependence of batch- normalized hidden activations makes each learner's 's update too overfitted to the distribution of their local training data. Batch Renormalization, by relaxing the dependence, yields a better outcome; however, it still fails to exceed the performance of the baseline due to the significant parameter divergence.
found to be 64.06%, 72.61% %, and 73.64% 4,,
To explain the impact of the steep fall phenomenon in test accuracy. we provide Figure 5 which in- dicates that the loss landscapes for the failure cases (e.g., Adam- WB and with Batch Normalization) commonly show sharper minima that leads to poorer generalization (Hochreiter & Schmidhuber
For its implementations into the considered federated learning algorithm, we let the server get the proper moving variance by R Eigk (B [B(0?]), & E3 [)] at each round, by allowing each learner & collect (E (3
well as the existing moving statistics of Batch Normalization (ø denotes activations). This can he regarded federated version of distributed Batch Normazliation methods (e.g., Qin al. (0188: //ang al. (208)).
8

---

Under review as a conference paper at ICLR 2020
Meen Elach Leamer's Training loss
Mean of Eachh earners Training Loss
Mean Eacc Leamer's Traiiing
OO Facer Do
0.
0.-
Wel-WWod
WD Facts audde
Trarengton Lone
0.7
0.6
0.0
2043
0
0.3
0.2
0.2
0.1
300
Communicaiion Rounds
Communication Rouncs
Communicatio Rounds
Figure 6: Excessively high training loss in the CIFAR-I0 training with NetB BeB--Baseline (left), the weight decay factor of 0.0005 (middle), and data augmentation (right), respectively, under the non- IID data setting. Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respec- tively. WD: weight decay; DA: data augmentation. Note that training loss values of each learner were calculated on their local data before each synchronization.
Table 4: Test accuracy (%) of the trials in Figure 6 baseline configuration.
Note that weight decay factor of 0.00005 the
Network
Method
Vanilla Training
Federated Learning
BS: 50
BS: 500
IID
Non-IID(2)
NetB-Baseline
NMom-A
85.05 (85. 13)
79.74 (79.79)
82.57 (82.65)
62.82 (63.08)
NetB-Wider
NMom-A
82.74 (82.90)
77.29 (77.47)
80.03 (80.43)
72.24 (72.32)
NetB-Widest
NMom-A
83.59 (83.60)
79.31 (79.77)
81.47 (81.73)
74.35 (74.52)
KetA-Baseline
NMom-A
83.89 (83.90)
79.39 (79.49)
81.49 (81882)
74.11 (74.23)
NMom-A + WD: 0.0
81.13 (81.15)
79.09 (79.25)
81.68 (81.90)
73.95 (74.27)
NelA-Baseline
NMom-A + WD: 0.0001
84.29 (84.46)
79.13 (79.74)
82.22 (82.25)
72.65 (72.79)
Mas-BBaelinee
NMom-A + WD: 0.0005
83.66 (84.60)
80.48 (80.79)
82.69 (8)
54.11 (54.15)
NetA-Baseling
NMom-A + DA
87.16 (87.41)
83.07 (83.35)
84.85 (84.89)
73.95 (74.46)
Keskar et al. 2017), and the minimal value in the bowl is relatively greater Here it is also observed that going into sharp minima starts even in early rounds such as 25th.
1997
10
Excessively high training loss of local updates. The final cause that we consider for the failure cases is excessively high training loss of local updates. For instance, from the left plot of Figure 6 we see that under the Non-IID(2) setting, NetB-Baseline gives much higher training loss than the other models. Here we note that for the NetB-Baseling model, the global average pooling is applied after the last convolutional layer, and the number of neurons in the first fully-connected layer thus becomes 256- 256; on the other hand, stride 4 and 2, which make the number of neurons in that layer become 1024 256 and 4096 256, respectively (see also Appendix A. for their details). The experimental results were shown that Base-la has notably lower test accuracy (see 4). We additionally remark that for NetB- Baseline, very high losses are observed under the IID setting, and their values even are greater than in the non-lID case; however, note that one have to be aware that local updates are extremely easy to be overfitted to each training dataset under non-IID data environments, thus the converged training losses being high is more critical than the IID cases.
Wider and NetB-Widest use the max pooling with
The middle and the right plot of the figure show the excessive training loss under the non-IID setting when applying the weight decay factor of 0.0005 and the data augmentation, respectively. In the cases of the high level of weight decay, the severe performance degradation appears compared to when the levels are low (i.e., < 0.0001) as already discussed. In addition, we observed that with Nesterov momentum SGD, the data augmentation yields a diminishing return in test accuracy (i.e., with the data augmentation we can achieve +3.36% under IID but -0.16% is obtained under non- IID(2). compared to when it is not applied); with Adam the degree of the diminishment becomes higher (refer to Table 12 in the appendix). In the data augmentation cases, judging from that the
Based on Li et al. 2018). the visualization of loss surface was conducted by I(a, E(0" +0 + (87) where 6" is center point of the model parameters, and 8 and yy is the orthogonal direction vectors.

---

Under review as a conference paper at ICLR 2020
Table 5: Summary of the failure cases under the non-IID data setting. PD: parameter divergence.
Failure Cases
High-level Reasons
Observed Internal Causes
Magnitude of PD
Steep Fall Phenomenon
High Local
Training Loss
Adam requires much larger number of model variables than the other optimizers, which indicates the increased number of them stimulated by the data non-IlDness
Adam-WB
The increased number of model parameters could lead to bigger parameter divergence
Depth++
Width-
v
It makes the model parameters in previous iter- ations to he reflected to the current ones; thus the previous parameter divergence also affects the present
WD++
It makes local updates of learners biased to each data distribution; could aggravate the parameter divergence
w/ BN
w/ DA
The dropped nodes (or neurons), randomly lected, are different across learners; its impact on the parameter divergence would be much stronger than under the IID setting
w/ DO
parameter divergence values are not so different between with and without it, we can identify that the performance degradation stems from the high training loss (see Figures 30 and 31 in the appendix). Here we additionally note that unlike on the CIFAR-10, 10, in the experiments on S SVHN it was seen that the generalization effect of the data augmentation still valid in test accuracy (see Table 12).
5 CONCLUSION
In this paper, explored the effects of various hyperparameter optimization strategies for optimiz- ers, network depth/width, and regularization on federated learning of deep networks. Our primary concern in this study was lied non-IID data, in which we found that under non-llI data settings many of the probed factors show somewhat different behaviors compared to under the IID setting and vanilla training. To explain this, a concept of the parameter divergence was utilized, and its origin was identified both empirically and theoretically. We also provided the internal reasons of our observations with a number of the experimental cases.
In the meantime, the federated learning has been vigorously studied for decentralized data environ- ments due to its inherent strength, i.e., high communication--eficiinss and arivacy-preservablee However, so far most of the existing works mainly dealt with only IID data, and the research to address non-III data has just entered the beginning stage very recently despite its high real-world possibility. Our study, one of the openings, handles the essential factors in the federated training under the non-lID data environments, and we expect that it will provide refreshing perspectives for upcoming works.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sheny Moore, Derek G. Murrey, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Margin Wicke. Yuan Yu, and Xiaoqiang Zheng. TensorFlow: A system for large- scale machine learning. In OSDI, 2016.
10

---

Under review as a conference paper at ICLR 2020
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In SysMI, 2019.
Sebastian Caldas, Virginia Smith, and Ameet Talwalker. Federated kernelized multi-task learning. In SysMI, 2018.
Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Revisiting dis- tributed synchronous SGD. In ICLR Workshop, 2016.
Mingqing Chen, Rajiv Mathews, Tom Ouyang, and Françoise Beaufays, Federated learning of out- of-vocabulary words. arXiv preprint arXiv: 1903. 2019.
Luca Corinzia and Joachim M. Buhmann. Variational federated multi-task learning. arXiv preprint arXiv: 1906,06268, 2019.
Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Mare' Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed deep networks. In NeurlPS, 2012.
Moming Duan. Astraea: Self- Self-balancing federated learning for improving classification accuracy of mobile deep learning applications. arXiv preprint arXiv: 0926,0 2019.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv: 1811.03604, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance of ImageNet classification. In ICCV, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Geoffrey E. Hinton, Nitish Srivatava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. Improving neural networks preventing co-adaptation of feature detectors. arXiv preprint arXiv: 1207.0580, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1), 1997.
Li Huang, Yifeng Yin, Zeng Fu, Shifa Zhang, Hao Deng, and Dianbo Liu. Loadaboost: Loss-based AdaBoost federated machine learning on medical data. arXiv preprint arXiv: 1811.12629, 2018.
Sergey loffe. Batch Renormalization: Towards reducing minibatch dependence in hoeh-nnormalized models. In NeurIPS, 2017.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating deep network training by reducing internal covariance shift. In ICMI., 2015.
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Geong-Lyun Kim. Communication-efficientt on-device machine learning: Federated distillation and augmentation under non-III private data. In NeurlPS Workshop, 2018.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, 2017.
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Chengjie Li, Ruixuan Li, Haozhao Wang, Yuhua Li, Pan Zhou, Song Guo, and Keqin Li. Gradient scheduling with global momentum for non-IID data distributed asynchronous training. arXiv preprint arXiv: 1902. 0988007848. 2019a.
11

---

Under review as a conference paper at ICLR 2020
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. NeurIPS, 2018.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization for heterogeneous networks. In ICML Workshop, 2019b.
Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of FedAvg on non-lID data. arXiv preprint arXiv: 1907.02189, 2019c.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR, 2014.
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local SGD. arXiv preprint arXiv: 1808.072 07810.777, 2018.
Lumin Liu, Jun Zhang, H. Song, and Khaled B. Lataief. Edge-assisted hierarchical federated learning with non-IlD data. arXiv preprint arXiv: 1905.06641, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight deacy regularization. In ICLR, 2019.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communictioon-fficien learning of deep networks from decentralized data. In AISTATS, 2017.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In ICML, 2019.
Yurii Nesterov. method of solving a convex programming problem with convergence rate o(L/k²). Soviet Mathematics Doklady, 27(2), 1983.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Boris Polyak. Some methods of speeding up the convergence of iteration methods. USSR Com- putational Mathematics and Mathematical Physics, 4(5), 1964.
Lianke Qin, Yifan Gong, Tianqi Tang, Yutian Wang, and Jiangming Jin. Training deep nets with progressive Batch Normalization on multi-GPUs. International Journal of Parallel Programming, 2018.
Felix Sattle, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Robust and communication-efficin federated learning from non-lID data. arXiv preprint arXiv: 1903.02891, 2019.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalker. Federated multi-task learning. In NewrIPS, 2017.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. In ICML, 2015.
Nitish Srivatava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1), 2014.
Sebastian U. Stich. Local SGD converges fast and communicates little. In ICLR, 2019.
Shigiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K. Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning resource constrained edge computing systems. IEEE Journal of Selected Areas in Communications, 37(6), 2019.
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ra- mage, and Françoise Beaufays. Applied federated learning: Improving Google keyboard query suggestions. arXiv preprint arXiv: 1812.02903, 2018
Ryo Yonetani, Tomohiro Takahashi, Atsushi Hashimoto, and Yoshitaka Ushiku. Decentralized learning of generative adversarial networks from multi-clies non-IID data. arXiv preprint arXiv: 1905.09684, 2018.
12

---

Under review as a conference paper at ICLR 2020
Naoya Yoshida, Takayuki Nishio, Masahiro Morikura, Koji Yamomoto, and Ryo Yonetani. Hybrid- FL: cooperative learning mechanism using non-llD data in wireless networks. arXiv preprint arXiv: 1905.07210, 2019.
Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restart SGD with faster convergence and less com- munication: Demystifying why model averaging works for deep learning. In AAAI, 2019.
Guodong Zhang. Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In ICLR, 2019.
Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In CVPR, 2018.
Jian Zhang, Christopher De Sa, Joannis Mitliagkas, and Christopher Ré. Parallel sgd: When does averaging help? In ICML Workshop, 2016.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-IID data. arXiv preprint arXiv: 1806.00582, 2018.
Hangyu Zhu and Yaochu Jin, Multi-objective evolutionary federated learning. arXiv preprint arXiv: 1812.07478, 2018.
13

---

Under review as a conference paper at ICLR 2020
A EXPERIMENTAL DETAILS
A.1 NETWORK CONFIGURATIONS
In the experiments, consider CNN architectures, as illustrated in Figure In the network con- figurations, three groups of 3 x 3 convolutional layers are included that have 16 m, 128, and 256 output channels, respectively; denotes the number of the layers in each convolutional group. The first two groups are followed by 3 x 3 max pooling with stride 2; the last convolutional layer is followed by either the 3 x 3 max pooling with stride or the global average pooling. the case of fully-connected layers, we use two types of the stacks: (i) three layers, of which the output sizes are 256 u, 256, and 10, respectively; and (ii) a single layer, of which the output size 10. In addition, the ReLU and the softmax activation for the hidden weight layers and the output layer, respectively. Table summarizes the network models used in the experiments.
3x3 cony, 16 m +
3x3 conv, 128
3x3 conv, 128
3x3 conv, 256
3x3 conv, 256
16
max pool, /2
max pool. /s or avg pool
Input
max maxpool, pool,
fc 256
fc
3x3
v 3)
Figure 7: The network configurations.
Table The network models used in the experiments.
n
m
Pooling after the Last Convolutional Layer
BeAA-Baseiine
1
max pool, /2
3
2
NetA-Deeper
max pool, /2
NetA-Deepest
3
max pool, /2
2
NetA-Narrowest
1
pool, /2
2
NetA-Narrower
max pool, /2
2
NetA-Baseline
max pool, /2
2
NetB-Baseline
avg pool
NetB-Wider
max pool, /4
3
NetB-Widest
max pool, /2
NetC-Baseline
avg pool pool, /4 max pool, /2
NetC-Wider
NetC-Widest
A.2 THE REMAINING CONHIGURATIONS
In the experiments, we initialize the network models mostly follow the truncated normal distribu- tion with a mean of based on He et al. (001)), however we fix the standard deviation to 0.05 for the first convolutional group and the last fully-connected layer. For training, minibatch stochastic opti- mization with cross entropy loss is considered. Specifically, we use pure SGD, Nesterov momentum SGD (Polyak 1964, Nesterov 1983). and Adam (Kingma Ba 2015) as optimization methods; initial learning rates are set to 0.05, 0.01, and 0.001, respectively for each optimizer. We drop the learning rate by 0.1 at 50% and 75% the total training iterations, respectively. Regarding the environmental configurations, we predetermine each of learners' local training dataset in a random seed; the training examples are allocated so that they do not overlap between the learners. To report the experimental results, basically considered run the trials once, but for unstable ones in the preliminary tests, we chose the middle results of several runs. In every result plot, the values are plotted each round.
14

---

Under review as a conference paper at ICLR 2020
B THEORETICAL ANALYSIS ON THE ORIGIN OF PARAMETER DIVERGENCE
In relation to the federated learning under non-llD data, so far there have been several works for providing theoretical bounds explain how does the degree of the non- non-IIDness of decentralized data affect the performance, with respect to its degree (e.g.. /Zhao et al. (0188) Li et al. (2019b
Liu et al. (2019); Stich| (2019); Wang et al. (2019): Yu et al. (2019). Inspired by them, here we further study how does the non- -on-ls make the model parameters of each learner diverged.
In this analysis, we consider training deep networks for multi-class classification. Based on the notations in Algorithm !
the SGD update of learner le at round t 1 is given as
R-1
Σ T=0
P (l===)))xxPxxre V log log (
7
where fg(x: w) is the posterior probability for class q (Q is the label space), obtained from model parameters w with data examples y). and pt (1 = () is the probability that the label of data example in Pk is q. In this equation, We 1+ a is the model parameters after the T-th local iterations in the round & + 1 (Rt is the number of local iterations of each learner per round). Herein we note that ++0 @ w") is the global model parameters received from the server at the round + 1; we use the term distinguish from the term w (which indicates the local update that has sent back to the server at round t).
Then, by the linearity of the gradient, we obtain
R-1
- (d
Σ Pk(U = Va, ExEP ww=y [log fx(x; w *)
n T=0
where d, denote the neurons, in the (dense) output layer of the model w, that are connected to the output node for class q.
Parameter divergence between (d2) with the fixed k. At suppose that for learne
k.
(ay))
++
#
s G, Vy Q,0 < R. and ((ai))
c+ -a,))
++ S H. Vi, 0.0 < R,
where
(an)**
+ स
a BxEP log (x: w [)) Then, we can get
(d:)) 1- - (d,)
(d.)* -(d,)* Σ
puly = (a))) ( pxlu (a,)) +
R-1
s <I((,)) (d,)%1 +
pa = =aa)) - -plyy = ja,)) +
R
= -1Ia)* (3,1 +
Σ
[p = (a.)) (th -p = i) 1 i
+ + i)(ag) 1+ - pp(y = j) ++#
< (d.)) (2,)11 mpx (y i)
Σ T=0
(a))+++ 1+ - -
R-1
+ + () pk(y = =i)l Σ (a, 1
< <I) --, + RRHP i) + RG Il (y = i) - pk (y = 3)
(1)
From this, we can identify that the parameter difference, (d.,) (d to [y i) P3 1) it corresponds the results in Figure 1
(d.) ) r-11 is bounded according
15

---

Under review as a conference paper at ICLR 2020
Table 7: Test accuracy (%) comparison of NetA-Baseline with respect to optimizers. Values out- side/inside brackets denote one measured after the whole training and the highest one during the rounds, respectively. PMom: Polyak momentum SGD; NMom: Nesterov momentum SGD; BS: minibatch size.
Dataset
Method
Vanilla Training
Federated Learning
BS: 50
BS: 500
IID
Non-IID(2)
CIFAR-10
Pure SGD
82.47 (82.72)
78.97 (79.06)
81.12 (81.36)
68.92 (69.57)
CIFAR-10
PMom-W
81.56 (81.82
74.30 (74.57)
CIFAR-10 10
PMom-A
81.78 (81.83)
74.82 (75.06)
CIFAR-10 10
NMom-WF
83.89 (83.90)
79.39 (79.49)
81.56 (81.99)
74.21 (74.31)
CIFAR-10
NMom-A
83.89 (83.90)
79.39 (79.49)
81.49 (81.82)
74.11 (74.23)
CIFAR-10
Adam-WB
81.97 (82.35)
80.73 (80.76)
80,99 (81.00)
67.51 (67.82)
CIFAR- 10
Adam-A
81.97 (82.35)
80.73 (80.76)
81.18 (81.45)
75.32 (75.45)
SVHN
Adam-WB
92.82 (93.12)
88.70 (88.91)
SVHN
Adam-A
92.89 (93.28)
90.72 (90.95)
Parameter divergence between (d,) 1++1 with the fixed q- At |++ 1, suppose that for class g,
(ag)k ++ S G", Ah: E K. 5 < R, and (aa) 1
(aa)) + < H' vi, E =
Then, similar with Equation I we can have
(dd,) - (d,))**|l s nRHH' pily =Q) nRG IIpi q) p,(y 4)ll
C THE COMPLETE EXPERIMENTAL RESULTS
In this section we provide our complete experimental results. Before the main statement, we first note that in the following figures, C₁j denotes the j-th convolutional layer in the i-th group, and F3 denotes the j-th fully-connected layer (in relation, refer to Appendix [A.I). In addition, we remind that in this paper "vanilla" training refers to non-distributed training with single machine, using the whole data examples; for the vanilla training, we trained the networks for 100 epochs.
C.1 THE EFFECT OF OPTIMIZERS
Here we investigate the effect of optimizers. We importantly note that both momentum SGD and Adam require the additional variables related momentum as well weights and biases; the terms (optimizer name)-A and (optimizer name)-Wi WB are used to refer to the parameter averaging being performed for all the variables and only for weights & biases, respectively.
The experimental results are provided in Table 7 and Figures 10 and 11 From the table, interest- ingly we can notice that under the non-IID data setting, there exists a huge performance gap between Adam-A A and Adam-WP (= 7%). unlike the momentum SGD trials. At the initial steps of this study, we had thought that the poor performance of Adam-Wi would be from the following: Since Adam requires twice many momentum variables as momentum SGD, extreme parameter divergence could appear if they are not averaged together with weights and biases. However, unlike our expec- tations, the parameter divergence values under the Adam-WB than under Adam-A. Nevertheless, we can observe the followings for the non-lID cases: First, the parameter divergence of Adam-Wi BB in Fs is bigger than that of Adam-A the very early rounds (as we expected), but soon is abnormally sharply reduced over rounds; this can be considered the steep fall phenomenon. Second, Adam-Wo WB leads to higher training loss of each learner. We guess that these two caused the severe degradation of Adam-W in test accuracy.
seen to be similar even smaller
C.2 THE EFFECT OF NETWORK DEPTH
Here we investigate the effect of network depth. Since deepening networks also indicates that there becomes having parameters be averaged in the considered federated learning algorithm,
16

---

Under review as a conference paper at ICLR 2020
Table 8: Test accuracy (%) comparison with respect network depth. Values outside/inside brack- ets denote one measured after the whole training and the highest one during the rounds, respectively. Note that ResNet-14 and ResNet-20 have 14 layers (13 convolutional layers I fully-connected layer) and 20 layers (19 convolutional layers + 1 fully-connected layer), respectively; they use the For ref-
global average pooling after the last convolutional layer, in common with erence, in our trials with Batch Normalization and data augmentation, the ResNet-20 could achieve test accuracy of 89.18 and 89.07 under Nesterov Momentum SGD and Adam, respectively.
Dataset
Network
Method
Vanilla Training
Federated Learning
BS: 50
BS: 500
IID
Non-IID(2)
CIFAR-10
NetA- Baseline
PMom-A
81.78 (81.83)
74.82 (75.06)
CIFAR-10
MetA-Deeped
PMom-A
82,93 (83.26)
74.73 (75.16)
CIFAR-10
metA-Deepest
PMom-A
81.99 (82.75)
69.77 (70.42)
CIFAR-10
BatA-Baseline
NMom-A
83.89 (83.90)
79.39 (79.49)
81.49 (81.82)
74.11 (74.23)
CIFAR-10 10
Deeeee
NMom-A
85.94 (86.00)
81.23 (81.29)
83.40 (83.65 63.63)
73.67 (73.89)
CIFAR-10
hetA-Deepest
NMom-A
86.20 (86.33)
79.48 (79.53)
83.12 (83,58)
68.98 (69.64)
SVHN
NetA-Based
NMom-A
93.02 (93.26)
89.28 (89.70)
SVHN
NetA-Deepes Deepest
NMom-A
94.42 (94.71)
89.12 (89.84)
CIFAR- 10
BalC-Baseline
NMom-A
84.25 (84.50)
76.70 (76.82)
80.60 (80.75)
64.06 (64.55)
CIFAR-10
ResNNet-14
NMom-A
81.24 (81.64)
62.14 (62.47)
CIFAR- 10
ResNet-20
NMom-A
82.30 (82.77)
59.98 (59.98)
CIFAR-10 10
HetC-Baseline
Adam-A
(8.54()3.42)
79.71 (80.89)
81.72 (81,80)
68.41 (68.75)
CIFAR-10
ResNet-14
Adam-A
79.32 (79.99)
64.46 (64.92)
CIFAR-10
ResNet-20
Adam-A
78.40 (79.28)
63.50 (64.06)
had predicted especially under non-llI data settings that depending on their depth, it would yield bigger parameter divergence in the whole model and the consequent diminishing returns compared to under the vanilla training and the IID data setting; the test accuracy results show as expected (see Table I1 Moreover, it is also seen from Figure 12 that parameter divergence increases also qualitatively ly (i... in a layer level) under the non-IID data setting, as the number of convolutional layers increases. Note that for C21 and C31. the divergence pattern is resulted as opposed to that of Cn and Fy; however, the values of C1 and Fs would be more impactful as mentioned in Footnote [
We additionally remark from the figure that the sharp reduction of parameter divergence (in the convolutional layers) at the very early rounds when using NetA-Deepest indicates the parameter averaging algorithm did not work properly. Correspondingly, the test accuracy values in the early period were seen to be not much different from the initial one.
C.3 THE EFFECT OF NETWORK WIDTH
Following the previous subsection, from now on we investigate the effect of network width. Contrary to the results in the Section C.2 it is seen from Table 9 that widening networks provides positive effects for the considered federated learning algorithm under the non-IID data setting. Especially, one can see that compared to the max pooling trials, while the global average pooling yields higher test accuracy in the vanilla training (with the minibatch size of 50), its performance gets signifi- cantly worse under the non-lll) data setting (remind that NetB-Baseline and the global average pooling after the last convolutional layer). Focusing on the NetC models, we here make the following observations for the non-IID data setting from Figures 15 18| and [2] First, the considered federated learning algorithm provides bigger parameter divergence in F; as its width decreases (note that each input size of F is 256, 1024, and 4096 for NetC-Wider, and NetC-Widest, respectively), especially during the beginning rounds (e.g., for the NMom-A case, until about 50 rounds). Unlike in Section C.2, here we can identify that even though the parameter size of is the smallest under the global averaging pooling, it rather yields the biggest qualitative parameter divergence. Second, the steep fall phenomenon appears in F2 for the NetC-
use
Note that for the SVHN cases, the test accuracy of NetA-Deepest is higher than that of NetA-Baseline under the non-Ill data setting, but the increase is much smaller than under the IID data setting.
17

---

Under review as a conference paper at ICLR 2020
Table 9: Test accuracy (%) comparison with respect to network width. Values outside/inside brackets denote one measured after the whole training and the highest one during the rounds, respectively.
Dataset
Network
Method
Vanilla Training
Federated Learning
BS: 50
BS: 500
IID
Non-IID(2)
CIFAR-10
SstA-Narrowest
pure SGD
81.64 (81.76)
76.68 (77.06)
78.78 (78.93)
67.66 (67.9))
CIFAR-10
NetA-Narrower
pure SGD
82.46 (82.68)
78.32 (78.38)
79.58 (79.72)
68.93 (69.28)
CIFAR-10
NetA-Baseline
pure SGD
82.47 (82.72)
78.97 (79.06)
81.12 (81.36)
68.92 (69.57)
CIPAR-10
NetB-Baseline
pure SGD
84.18 (84.61)
69.22 (69.22)
74.58 (74.59)
54.39 (54.84)
CIFAR-10
NetB-Wider
pure SGD
81.41 (81.59)
76.69 (77.57)
79.49 (79.81)
65.06 (65.20 29)
CIFAR-10 10
NetB-Wider
pure SGD
82.60 (82.65)
79.10 (79.27)
80.31 (80.37)
68.77 (69.20)
CIFAR-10
BetC-Baseline
pure SGD
83.64 (83.69)
68.46 (68.47)
72.64 (72.70)
58.13 (58.51)
CIFAR-10
NetC-
pure SGD
79.88 (79.96)
76.69 (76.87)
78.08 (78.24)
68.30 (68.65)
CIFAR-10
NetC-Widest
pure SGD
82.11 (82.21)
78.28 (78.34)
79.94 (79.97)
70.33 (70.51)
CIFAR-10 10
Bas--Baseline
PMom-A
79.69 (79.84)
64,42 (64.54)
CIFAR-15 10
NetC-Widest
PMom-A
80.12 (80.36)
74.68 (75.18)
CIFAR-10
NetA-Narrowest
NMom-A
82.03 (82.06)
77.75 (77.92)
80.75 (81.02)
72.57 (72.64)
CIFAR-10
NetA-Narrower
NMom-A
83.50 (83,50)
78.11 (78.59)
81.29 (81.44)
73.44 (73.60)
CIFAR-10
NetA-Baseline
NMom-A
83.89 (83.90)
79.39 (79.49)
81.49 (81.82)
74.11 (74.23)
CIFAR-10 10
NMom-A
85.05 (85.13)
79.74 (79.79)
82.57 (82.65)
62.82 (63.08)
CIFAR-10
NetB-Widke
NMom-A
82.74 (82.90)
77.29 (77.47)
80.03 (80.43)
72.24 (72.32)
CIFAR-10
NetB-Widest
NMom-A
83.59 (83.60)
79.31 (79.77)
81.47 (81.73)
74.35 (74.52)
CIFAR-10
BetC-Baseline
NMom-A
84.25 (84.50)
76.70 (76.82)
80.60 (80.75)
64,06 (64.55)
CIFAR-10
NetC-Wider
NMom-A
81.48 (81.53)
77.00 (77.14)
79.59 (79.85)
72.61 (72.97)
CIFAR-10
NetC-Widest
NMom-A
83.16 (83.36)
78.39 (78.56)
80,90 (81.14)
73,64 (73,91)
SVHN
NetC-Baseline
NMom-A
92.64 (92.85)
82.69 (83.35)
SVHN
NetC-Widest
NMom-A
92.19 (92.50)
89.55 (90.62)
CIFAR-10
NetA-Narrowest
Adam-A
80.62 (81.02)
79.72 (79.72) 72)
79.82 (79.88)
75.06 (75.18)
CIFAR-10
NetA- BetA-Narrower
Adam-A
82.23 (82.51)
80.80 (80.80)
81.04 (81.09)
74,88 (74.88)
CIFAR-10 10
NetA-Baseline
Adam-A
81.97 (82.35)
80.73 (80.76)
81.18 (81.45)
75.32 (75.45)
CIFAR-10 10
NetB-Baseline
Adam-A
83.52 (83.62)
80.98 (81.61)
81.68 (82.29)
71.40 (71.45)
CIFAR-10
NetB-Wider
Adam-A
81.20 (81.39)
79.49 (79.62)
80.22 (80.39)
74.42 (74.67)
CIFAR-10
NetB-Widest
Adam-A
82.08 (82.29)
80.87 (80.87)
81.02 (81.20)
74.93 (75.41)
CIFAR-10
Adam-A
82.59 (83.42)
79.71 (80.89)
81.72 (81880)
68.41 (68.75)
CIFAR-10
NetC-Wider
Adam-A
80.77 (80.94)
77.31 (77.48)
78.81 (79.09)
75.09 (75.22)
CIFAR-10
NetC-Widest
Adam-A
81.53 (81.59)
78,80 (78.92)
79.91 (80.08)
76.57 (76.57)
Baseline case. Third, the global average pooling gives too high training loss of each learner. All the three observations fit well into the failure of the global average pooling.
We additionally note that when using NetC-Baseline, the results under the IID data setting shows very high loss values; this leads to diminishing returns for the pure SGD and the NMom-A cases, compared the vanilla training results with the minibatch size of 50. However, the corresponding degradation rate is seen to be much higher under the non-lID data setting. This is because the local updates are extremely easy be overfitted to the training data under the non-IID data setting; thus the converged training losses being high becomes much more critical.
C.4 THE EFFECT OF WEIGHT DECAY
Here we investigate the effect of weight decay. From Table is seen that under the non-IID data setting we should apply much smaller weight decay for the considered federated learning al algorithm than under the vanilla training or the IID data setting. For its internal reason, Figures 22 23 and 24 show that under the non-lID data setting, the considered federated learning algorithm not only converges to too high training loss (of each learner) but also causes excessive parameter divergence when the weight decay factor is set to 0.0005. Here we note that if single iteration is considered for each learner's ' local update per round, the corresponding parameter divergence will be of course the same without regard to degree of weight decay. However, in our experiments, the great number
18

---

Under review as a conference paper at ICLR 2020
Table 10: Test accuracy (%) comparison of NetA-Baseline with respect to weight decay levels.
Values outside/inside brackets denote one measured after the whole training and the highest one during the rounds, respectively. WD: weight decay.
Dataset
Method
WD Factor
Vanilla Training
Federated Learning
BS: 50
BS: 500
|ID
(on-
CIFAR-10
pure SGD
0.0
81.33 (81.33)
78.00 (78.16)
79.88 (80.09)
68.85 (69.79)
CIFAR-10
pure SGD
0.00005
82.47 (82.72)
78.97 (79.06)
81.12 (81.36)
68.92 (69.57)
CIFAR-10
pure SGD
0.0001
82.00 (82.19)
79.58 (79.74)
81.28 (81.40)
68.95 (69.03)
CIFAR-10 10
pure SGD
0.0005
82.39 (82.54)
78.92 (78.95)
80.28 (80.28)
47.43 (47.66)
CIFAR-10
PMom-A
0.00005
81.78 (81.83)
74,82 (75.06)
CIFAR- 10
PMom-A
0.0005
82.67 (83.21)
55.57 (56.25)
CIFAR- 10
NMom-A A
0.0
81.13 (81.15)
79.09 (79.25)
81.68 (81.90)
73,95 (64.27)
CIFAR-IE 10
NMom-A
0.00005
83.89 (83.90)
79.39 (79.49)
81.49 (81.82)
74.11 (74.23)
CIFAR-10
NMom-A
0.0001
84.29 (84.46)
79.13 (79.74)
82.22 (82.25)
72,65 (72.79)
CIFAR-10
NMom-A
0.0005
83.66 (84.60)
80.48 (80.79)
82.69 (83.14)
54.11 (54.15)
SVHN
NMom-A
0.00005
93,02 (93.26)
89.28 (89.70)
SVHN
NMom-A
0.0005
93.23 (93.72)
76,31 (76.82)
CIFAR-10
Adam-A
0.0
80.70 (80.73)
80.01 (80.01)
80,11 (80.19)
74.77 (74.93)
CIFAR-10
Adam-A
0.00005
81.97 (82.35)
80.73 (80.76)
81.18 (81.45)
75.32 (75.45)
CIFAR-10
Adam-A
0.0001
82.79 (83.19)
81.27 (81.30)
81.74 (81.84)
75.95 (76.02)
CIFAR-10
Adam-A
0.0005
82.93 (84.44)
81.34 (81.54)
82.33 (83.06)
69.75 (69.75)
Tead Acsurnry Globni Modil
Weic of the and Cldl--orreeeed
Weaghte of the Lart CullyCCnnreeeee
PP-IL
2
10
1000
200
500
10
10¹
10
'0'
Commurication Rounds
Communccaiion Founds
Communication Rounds
Figure 8: Comparison of test accuracy and parameter divergence under Non-IID(2) setting with respect to FedProx factor in the training of NetA-Baseline on CIFAR-10 (Optimizer: NMom- A).
of local iterations per round (i.e., 100) made big difference of the divergence values under the non-IID data setting; this eventually yielded the accuracy gap. In addition, we further observe under the non-llD data setting that even with the weight decay factor of 0.0005, the test accuracy increases similarly with its smaller values at very early rounds, in which the norm values of the weights are relatively much smaller.
Moreover we also conducted additional experiments for the related regularization techniques, FedProx (Li et al. 2019b). Under the FedProx, in order to make local updates do not de- viate excessively from the current global model parameters, at each round each learner uses the following surrogate loss function that adds a proximal term to the original objective function: ((w)+ " plementation FedP rox did not provide dramatic improvement in final accuracy, but that it could yield not only lower parameter divergence but also faster convergence speed (especially before the first learning rate drop). One can find the corresponding complete results in Figure 25
2 w - w w'-113 Figure &hoows the experimental results; as seen from the figure, in our im-
observe
C.5 THE EFFECT OF BATCH NORMALIZATION
Here we investigate the effect of Batch Normalization. For its implementations into the con- sidered federated learning algorithm. we let the server get the proper moving variance by each round, by allowing each learner k collect (R[02]), & well R Ekex (R[o²]), E [[
19

---

Under review as a conference paper at ICLR 2020
Table 11: Test accuracy (%) comparison of NetA-Baseline with/without Batch Normaliza- Hoon/eeormalliztion. Values outside/inside brackets denote one measured after the whole train- ing and the highest one during the rounds, respectively. BN: Batch Normalization; BRN: Batch Renormalization.
Dataset
Method
Vanilla Training
Federated Learning
BS: 50
BS: 500
IID
Non-IID(2)
CIFAR-10
pure SGD
82.47 (82.72)
78.97 (79.06)
81.12 (81.36)
68.92 (69.57)
CIFAR-10
pure SGD BN
85.06 (85.17)
78.33 (78.70)
82.91 (83.02)
57.56 (58.68)
CIFAR-10
pure SGD + BRN
85.67 (85.72)
81.49 (81.65)
84.00 (84.19)
66.19 (66.32)
CIFAR-10
PMom-A
81.78 (81.83)
74.82 (75.06)
CIFAR-10
PMom-A BN
83.74 (83.84)
52.76 (59.69)
CIFAR-10
NMom-A
83.89 (83,90)
79.39 (79.49)
81.49 (81.82)
74.11 (74.23)
CIFAR-10
NMom-A+ BN
85.22 (85.25)
79.58 (80.14)
83.54 (83.80)
50.46 (59.31)
CIFAR-10
NMom-A+ BRN
86.36 (86.43)
82.03 (82.46)
84.24 (84.70)
70.32 (70.38)
SVHN
NMom-A
93.02 (93.26)
89.28 (89.70)
SVHN
NMom-A BN
92.96 (93.34)
74.50 (79.73)
CIFAR-10
Adam-A
81.97 (82.35)
80.73 (80.76)
81.18 (81.45)
75.32 (75.45)
CIFAR-10
Adam-A + BN
85.76 (85.93)
83.84 (83.93)
84.88 (85.17)
39.31 (47.67)
CIFAR-10
Adam-A + BRN
85.86 (86.11)
83.18 (83.36)
84.68 (84.91)
72.49 (72.71)
as the existing moving statistics of Batch Normalization (ó denotes activations). This can be re- garded federated version of distributed Batch Normazliation methods (e.g.. et al. (2018): Zhang et al. (2018)) It is natural to take this strategy especially under the non-IID data setting: otherwise, huge problem would arise due to bad approximation of the moving statistics. Also, is additionally remarked that for Batch Renormalization we simply used a 0.01, Tmax = 2. and dmaz= in the experiments (see loffe (2017) for the description of the three hyperparameters).
is seen from Table II that under the non-IID data setting, the performance significantly gets worse if Batch Normalization is employed to the baseline; this would be rooted in that the dependence of natch-normalized hidden activations makes each learner's update too overfitted to the distribution of their local training data. The consequent bigger parameter divergence is observed in Figures [6 27 and 28 On the contrary, Batch Renormalization, by relaxing the dependence, yields a better outcome; although its parameter divergence is seen greater in some layers than under Batch Nor- malization, it does not lead to the steep fall phenomenon while the Batch Normalization does in F3- Nevertheless, the Batch Renormalization was still not able to exceed the performance of the baseline due to the significant parameter divergence
C.6 THE EFFECT OF DATA AUGMENTATION
In the implementation of data augmentation, we used random horizontal flipping, brightness & contrast adjustment, and 24 x 24 cropping & resizing in the pipeline. From Table 12 identify that under the non-III data setting, the data augmentation yields diminishing returns for the PMom-A, NMom-A, and Adam-A cases on CIFAR-10, compared to under the IID data setting; under Adam- A, especially it gives even a worse outcome. However, it is seen that the corresponding parameter divergence almost similar between with and without the data augmentation (refer to Figures 30 and B1). Instead, we are able to notice that the diminishing outcomes from the data augmentation had been eventually rooted local updates' high training losses. Here we note that in the pure SGD case, very high training loss values are found as well under the IID data setting when the data augmentation was applied (see Figure 29); ; this leads to lower test accuracy compared to the baseline,
Duurin the experiments, we observed that Batch Renormalization yields relatively heavy fluctuation of test accuracy and training loss until the first learning rate drop. This is thought to have the effect of setting Pmma and dmaz to the fixed value, as discussed in loffe (2017): however we guess that the effect of the data non-IIDness might be greater, judging from the degree of the fluctuation under the IID and the non-IID data setting.
20

---

Under review as a conference paper at ICLR 2020
Table 12: Test accuracy (%) comparison of MetA-Baseling with/without data augmentation. Values outside/inside brackets denote one measured after the whole training and the highest one during the rounds, respectively. DA: data augmentation.
Dataset
Method
Vanilla Training
Federated Learning
BS: 50
BS: 500
IID
Non-IID(2)
CIFAR-10
pure SGD pure SGD + DA
82.47 (82.72)
78.97 (79.06)
81.12 (81.36)
68.92 (69.57)
CIFAR-10
86.79 (86.97)
77.66 (78.03)
80.39 (80.48)
68.33 (68.69)
CIFAR-10
PMom-A
81.78 (81.83)
74.82 (75.06)
CIFAR-10
PMom-A DA
83.87 (83.93)
72.40 (74.01)
CIFAR-10
NMom-A
83.89 (83.90)
79.39 (79.49)
81.49 (81.82)
74.11 (74.23)
CIFAR-10
NMom-A + DA
87.16 (87.41)
83.07 (83.35)
84.85 (84.89)
73.95 (74.46)
SVHN
NMom-A
93.02 (93.26)
89. (89.70)
SVHN
NMom-A + DA
94.51 (94.66)
90.61 (91.59)
CIFAR-10
Adam-A
81.97 (82.35) 83.27 (83.54)
80.73 (80.76)
81.18 (81.45)
75.32 (75.45)
CIFAR-10
Adam-A DA
84.84 (84.96)
82.63 (82.99)
72.79 (73.67)
Table 13: Test accuracy (%) comparison of NetA-Baseline with/without Dropout. Values out- side/inside brackets denote measured after the whole training and the highest during the rounds, respectively. DO: Dropout.
Dataset
Method
Vanilla Training
Federated Learning
BS: 50
BS: 500
IID
Non-IID(2)
CIFAR-10
pure SGD pure SGD + DO
82.47 (82.72) 85.76 (85.79)
78.97 (79.06)
81.12 (81.36)
68.92 (69.57)
CIFAR-10
82.48 (82.53)
83.40 (83.52)
73.02 (73.04)
CIFAR-10
PMom-A
81.78 (81.83)
74.82 (75.06)
CIFAR-10
PMom-A + DO
84.29 (84.38)
76.83 (76.88)
CIFAR-10
NMom-A
83.89 (83,90)
79.39 (79.49)
81.49 (81.82)
74.11 (74.23)
CIFAR-10
NMom-A + DO
86.55 (86.63)
84.29 (84.32)
84.34 (84.53)
75.80 (75.89)
SVHN
NMom-A
93.02 (93.26)
89.28 (89.70)
SVHN
NMom-A + DO
94.14 (94.36)
90.61 (91.45)
CIFAR-10
Adam-A
81.97 (82.35)
80.73 (80.76)
81.18 (81.45)
75.32 (75.45)
CIFAR-10
Adam-A + DO
84.27 (84.46)
83.09 (83.20)
82.01 (82.11)
76.63 (76.68)
similar to under the non-lID cases. Also, it is additionally noted that unlike on the CIFAR-10, in the experiments on SVHN it was observed that the generalization effect the data augmentation still valid in test accuracy.
C.7 THE EFFECT OF DROPOUT
In the experiments, we employed Dropout with the rates 0.2 and 0.5 for convolutional layers and fully-connected layers, respectively. The results show that under the non-lID data setting, the Dropout provides greater parameter divergence compared to the baselines, especially in F3 (see Figures 32 33 and 34); this leads to diminishing returns for the PMom-A and NMom-A cases CIFAR- I0, compared to under the IID data setting. However, we can observe from Table 1 that the effect of the Dropout is still maintained positive for the rest of the cases.
21

---

Under review as a conference paper at ICLR 2020
C.8 THE EXPERIMENT RESULTS UNDER UNBALANCED DATA SETTINGS
As remarked in (McMahan et al. 2017), since the federated learning do not require centralizing lo- cal data, data unbalancedness (i.e., each learner has various numbers of local data examples) would be also naturally assumed in the federated learning along with non-IlDness. In relation, we also conducted the experiments under the unbalanced cases. Table 14 summarizes the considered unbal- anced data settings; they were constructed similarly to (Li ct al. 2019b) so that the number of data examples per learner follows a power law.
The experimental results under the unbalanced settings are summarized in Table 15 From the table, it is observed that our findings Section 4.1 are still valid under the unbalanced data settings. In addition, we can also see that for the unbalanced cases, the performance under Non-IID(2) setting is worse mostly than that of balanced cases while they show similar values under the IID data setting; this indicates that the negative impact of data unbalancedness is not great as that of the non- IIDness, but it becomes much bigger when the two are combined.
Table 14:
Configuration of unbalanced data settings. #Cls/L: the number of classes each learner's local training dataset; #Exs/L: the number of training examples in each local dataset.
Data Setting
#Learners
#Cls/L
#Exs/L
Mean
Std.
IID
10
10
5000
2905
Non-IID(2)
2
5000
2905
Table 15: Test accuracy (%) comparison on CIFAR-10 10 under the balanced and the unbalanced data settings. Values outside/inside brackets denote one measured after the whole training and the highest one during the rounds, respectively.
Network
Method
Balanced Data Setting
Unbalanced Data Setting
IID
Non-IID(2)
IID
Non-IID(2)
NetA-Baseline
Adam-WB
80.99 (81,00)
67.51 (67.82)
80.03 (80.27)
56.52 (59.62)
NetA-Baseling Baseline
Adam-A
81.18 (81.45)
75.32 (75.45)
81.02 (81.14)
73.12 (73.19)
NetA-Baseline
NMom-A
81.49 (81.82)
74.11 (74.23)
81.31 (81.49)
72.59 (72.86)
NetA-Deep DeeeDeepest
NMom-A
83.12 (83.58)
68.98 (69.64)
83.67 (83.68)
69.10 (69.23)
NerC-Baseling
NMom-A
80.60 (80.75)
64.06 (64.55)
81,00 (81.10)
62.23 (62.30)
NetC-Widest
NMom-A
80.90 (81.14)
73.64 (73.91)
80.67 (80.76)
71884 (72.34)
NctA-Baseline
NMom-A
81.49 (81.82)
74.11 (74.23)
81.31 (81.49)
72.59 (72.86)
NetA-Baseline
NMom-A + WD: 0.0005
82,69 (83.14)
54.11 (54.15)
82.76 (83.13)
47.85 (47.85)
DetA-Baseline
NMom-A + BN
83.54 (83.80)
50.46 (59.31)
83.38 (83.54)
45.92 (50.95)
NeLA-Baseline
NMom-A + DA
84.85 (84.89)
73.95 (74.46)
84.16 (84.19)
73.60 (74.06)
ArA-Baseline
NMom-A
84.34 (84.53)
75.80 (75.89)
84.25 (84.38)
68.83 (70.33)
22

---

Under review as a conference paper at ICLR 2020
Teat Accuracy of Global Model
Mean of Each Learner's Training Loss
0.5
HD Nendlt355 Nan-liDal
0.4
M MAN
Test Acqunac)
Traking Loss
0.3
0.7
0.2
O ID ND1
0.1
Non ID
0.5
0
100
Communication Rounds
Communtcation Flounds
Weights of C1
Weights C1
PDLs
PD-VL
10
10
Non-1in
ID21
10P
10
10a
10
10
10a
Communication Rounds
Communisation Founds
Weights of C21
Weights Cal
10
10
PD-Ls
PD-VL
10
10x
10
16
10P
10! Communication Rounds
10P
102
Communication Foundo
Weights of C31
Weights C3g
RDALS
PD-VL
10
-nn Non In-!
10P
10
10a
1.P
10
16a
Communication Rounds
Communisation Founds
Weights of F
Weights of F2
10
10
PD45
PO-WL
10
4
Non ID:5
16
Non IDIS
Non ID:2
10
10! Communication Rounds
10
10 Communication Pounds
Figure 9: Behavior comparison with respect to the degree of data non-IIDness in the training of on CIFAR-10 (Optimizer: NMom-A).
23

---

Under review as a conference paper at ICLR 2020
Test Acsuracy Global Model
Mean of Each Learne''s Training Loss
0.8
0.5
--lcmm-WW wa
0.4
Tost Agauracy
Troinng Less
0.3
67
0.2
0.1
3 MMMMoom WB
NMomA
%
100 Communication Rounds
200
300
100
200
300
Communication Rounds
Weights C3.
Weights C13
10
1
PD-L5
PD-VL
10
10
10
10
MMATT-WW Nitam-A
10
10
10
10¹ Communication Rounds
10a
Communication Rounds
Weights of C21
Weights C21
22
10
PD
PD-VL
10
10
1
10P
10
10a
10
10' Communication Rounds
102
Communication Rounds
Weights of Cg
Weights C.)
1
20
PD-L5
PD-NL
10
10
10
NMA
10! Communication Rounds
10P
Communtcation Fiounds
Weights
Weights of
RD-Ls
PO-VL
10
9
90
10"
10!
10a
10
10¹
10a
Communication Rounds
Communisation Founds
Behavior comparison between NMom-WB and Nmom-A in the training of NetA-Baseline on CIFAR-10. Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
Figure 10:
24

---

Under review as a conference paper at ICLR 2020
Teat Acsuracy Global Model
of Learner's Training Loss
0.0
0.5
Adur VeB
AdumnA
0.0
0.4
Test Acouneyy
Trining Loss
0.3
Campy
0.2
0.1
A-AlammM WB
100
200
300
0
Communication Rounds
Communication Rounds
Weights of C.
Weights of C1.
10
1
PD
PD-YL
10
10
Adem-WB AdammA
10
O A-Aam--WB WB
V Adammm
10
10 Communication Rounds
10
10' Communication Rounds
10
Weights
Weights
2
POWL
10"
10
10 49
10
Addmms
Adami
100
10a
104
10
10'
10²
Communication Rounds
Communisation Rounds
Weights of Ca.
Weights C
10
1
RD-L5
10
10
10
AdemmAmi Adam.A
Addaa WB 7 A-daaam A
10
10'
10"
Communication Rounds
Communication Rounds
Weights
Weights
2
PD-Ls
PO-VL
10
10
9
104
A-AammA
10
10!
10a
10
10¹
16a
Communication Rounds
Communisation Founds
Figure 11: Behavior comparison between Adam-WPp and Adam-A in the training of NetA-Baseling on CIFAR-10. Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respec- tively.
25

---

Under review as a conference paper at ICLR 2020
Teat Acsuracy Global Model
Mean of Learnr'ss Training Loss
0.0
0.5
Natk
0.0
0.4
Netk-Dhepast
Test Acounney
0.7
Troinnn Less
0.3
CeNAA Basckro NotA Deeper
0.1
Doopest
%
200 Communication Rounds
100
300
100
200
300
Communication Flounds
Weights of CT
Weights of C33
10
10
PD-VL
PDLLs
10
10
1
Hest--assline Nest-Dneper
Nask Bassline
Nesk--nepm
Nest-Despast
Nack-Despeat
10
10¹ Communication Rounds
10
Communication Rounds
Weights of C2
Weights
2
PD-VL
2012
10"
10 9
Catt--oasniie Nett-Dnepaet
10
10
10'
10
10P
101
10a
Communication Rounds
Communication Rounds
Weights of Cg.
Weights C.
1
2
PD-VL
10
10
10
Nett-Daepar
10
Nosk--oooniin Nank-/heeper
Neth Dospeal
Nesk Despest
10"
10'
10
Communication Rounds
Communication Rounds
Weights
Weights
2
PD-VL
PD-Ls
10
10
10
Hask-Clesaiee Neth-oopast!
16
Nosk Baseline
10
16'
10²
10
10¹
10
Communication Rounds
Communisation Founds
Figure 12: Behavior comparison of the NetA models with respect to network depth on CIFAR-10 (Optimizer: NMom-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
26

---

Under review as a conference paper at ICLR 2020
Tast Accuracy Global Model
Mean of Each Learner's Training Loss
0.9
0.5
Narowesd
0.4
Nete Wiseeie
Teat Agquracy
Treinng Less
0.3
0.7
0.2
Nest Narrowss
0.1
Nest Namower
Nost Bassline
200 Communication Hounds
100
300
100
200
300
Communication Rounds
Weights of Cy
Weights C1
10
10
PD-L5
PD-VL
10
10
10
Nett Narrowed Nettk Narroer
10
Parsk Harrowatt
Nett--arallie
Narck Dura-lin
10"
10
10¹
10a
Communication Hounds
Communication Rounds
Weights of 21
Weights C21
10
PD
10"
10
10
Noth Homowess NetA Baseline
Nost Narrooos Nesk Bosoline
10
10.
10a
10
10
10
Communication Rounds
Communisation Founds
Weights of Cg
Weights C.
10
20
P016
PD-NL
10
10
Narrowss NutA Narrwer
10
Nesk Narrowssi
No1A Busctine
10"
10P
Communication Hounds
Communtcation Flounds
Weights of
Weights of F
10
L.
PD-VL
10
10
NatA- Ntarowant NetA- Bo---Bazeiie
10
Nost NarTOOSS Nosk Sassllnn
10
10"
10a
10
10¹
10
Communication Rounds
Communisation Founds
Figure 13: Behavior comparison of the NetA models with respect to network width on CIFAR-10 (Optimizer: pure SGD). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
27

---

Under review as a conference paper at ICLR 2020
Test Acsuracy Global Model
Mean of Each Learne''s Training Loss
0.8
H Eesclin
0.9
0.0
0.8
Not Wiiess
Test Accuneyy
0.7
C -
0.6g
0-
0.55
@ 0.4
0.3
NoB Basciin Net Waseer
0.2
0.1
100
200
300
100
200
300
Communication Rounds
Communication Flunds
Weights C37
Weights C33
10
10
PD-L5
PD-VL
10
10
10
NerB--assliie NetD-Wader
5
Nee Baseline
NerB-WWider
NeVE--WWee
10 Communication Rounds
10
10 Communtzation Founds
Weights of 22
Weights C21
22
10
10
10
10
NoBB Basoero
-NBB Basolino NoB
Net
10
10
10a
10
10
10
Communication Rounds
Communisation Rounds
Weights of Cg
Weights C.)
10
20
PDLL
PD-NL
10
10
NuB Gusctino
10
NuB Wader
NoB Wadest
10 Communication Rounds
10P
Communtcation Flounds
Weights
Weights of
PO-VL
10
10
Natt Siasolin NorE Wideet
10
No Eassline
AirE-WWides
10
10a
10aa
10
10¹
16a
Communication Rounds
Communisation Founds
Figure 14: Behavior comparison of the NetB models with respect to network width on CIFAR-10 (Optimizer: pure SGD). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
28

---

Under review as a conference paper at ICLR 2020
Test Acturacy of Global Model
Mean of Each Learne''s Training Loss
0.9
Besckine
0.0
0.8
0.8
Widel
Test Aosuring
0.7
0.7
0.6
0.5
0.5
0.3
0.4
NenD Basckin
0.9
NenD Wider
0.1
NenD Wileet
200 Communication Rounds
100
300
100
200
300
Communication Fluuds
Weights of C..
Weights of C.,
10
10
80455
PD-VL
10
10
NenC Basekin NNDD WAder
NevD Baseiin
10
10¹ Communication Rounds
10
10a
Communication Rounds
Weights
Waights
PD-L5
PO-YL
10
10
10
10
Nert-Whheer
NosC
Nest
10
10"
10a
10
10¹
10a
Communication Rounds
Communisation Founds
Weights of Cg.
Weights Cg)
2
PDL5
PD-L
10
10
NoC Buseline
10
Busciine
NND Water
NoC Wider
NotC Wiceot
Nec Wilsst
10
10' Communication Rounds
10"
10
10' Communication Rounds
10"
Weights F2
Weights of
10
10
L5
PD-VL
10
10
10
Narc- dlasstru Neri- WWlieed
10
Markc dasstia
10
10a
10a
10
10¹
16a
Communication Rounds
Communisation Founds
Figure 15: Behavior comparison of the NetC models with respect to network width on CIFAR-10 (Optimizer: pure SGD). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
29

---

Under review as a conference paper at ICLR 2020
Test Acsuracy Global Model
Mean of Each Learne''s Training Loss
0.8
0.0
Narooesd
0.4
Nutt Basclre
Test Agcuracy
Traning Locs
0.3
0.1
200 Communication Rounds
100
300
100
200
300
Communication Floudss
Weights of C..
Weights of C.,
10
10
80455
PD-VL
10
10
Hask-Harowwst
Nett-Rassllie
Nartt-Fasnllie
10
10' Communication Rounds
10
10
Communication Rounds
Weights
Weights
PD-Ls
PD-VL
10
10
10
Nett--Aaroeent Nost Bosoline
1
Mack--arroaee Nartk-Slameline
10
10'
10
10P
101
10a
Communication Rounds
Communication Founds
Weights of C2.
Weights C.l
2
1
PDL5
PD-L
10
10
Nesk Narrowesl
Nusk Narrowor
Nesk Narrowo
NosA Bascline
Nest Baseline
10
10'
10"
10
10' Communication Founds
Communication Rounds
Weights
Weights of
PD-VL
10
10
10
Natt--artower Nert- Boss-Bossittee
10
Narsk Narroaat
Nark-Narooo
Nask
10
10
10a
10
10a
10a
Communication Rounds
Communisation Founds
Figure 16: Behavior comparison of the NetA models with respect to network width on CIFAR-10 (Optimizer: NMom-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
30

---

Under review as a conference paper at ICLR 2020
Tast Accuracy Global Model
Mean of Each Learne''s Training Loss
0.9
Eesclins
0.0
0.0
0.8
NeB--Wkks Wridst
Test Agouracy
0.7
0.7
0.6
0.6
00.50
0.3
0.2
Nowtl-WW der
0.1
100
200
300
100
200
300
Communication Hounds
Communication Floudds
Weights of C1.
Weights C31
10
10
P0115
PD-VL
10
Net Basekre
NeID-WAder
5
Ner Eosoline
NefS-WWider
Nett-Wideed
Nart- WWideet
10
10"
10
10a
Communication Rounds
Communication Rounds
Weights
Weights C..
2
20
PD
PD-V
10"
10
1
MartS--ooeeiaa NeES-WWider!
100
10
102
10
10¹
10²
Communication Rounds
Communication Pounds
Weights of gr
Weights C3n
1
10
PD-L5
10
10
10
10
NoB Wiier
Net Buseiin
NulB Witeot
NoB Wales!
10"
10"
10"
10' Communication Rounds
Communication Rounds
Weights
Weights of
20
PD-L
10
9
NetB Basekin Noth Widded
Narts Sie----eeelee NATS (Videst
10
10"
10a
10
16"
10
Communication Rounds
Communication Founds
Figure 17: Behavior comparison of the NetB models with respect to network width on CIFAR-10 (Optimizer: NMom-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
31

---

Under review as a conference paper at ICLR 2020
Test Acsuracy Global Model
Mean of Each Learne''s Training Loss
0.8
Busckine
0.0
0.0
0.8
Net Wilest
Test Acounecy
0.7
0.7
0.6
0.0
0.4
NenD Bascero Nent Water
0.9
0.1
200 Communication Rounds
100
300
100
200
300
Communication Flounds
Weights of C3
Weights of C..
10
10
PD
PD-VL
10
10
10
NenC Basekine NHC WAlde
NevD Basenin
NeC--Wleal
10
10¹ Communication Rounds
10
Communication Rounds
Weights of C22
Weights
22
PDLs
PD-YL
10
10 9
10
10
10'
10²
10P
10¹
10a
Communication Rounds
Communication Founds
Weights of Cg.
Weights C.1
1
20
BD-L5
PD-VL
10
10
10
NenC-Bbaraiin
10
NoC
NetC Wilios
10"
10'
10"
Communication Rounds
Communication Rounds
Weights
Weights of
2
POLs
PO-VL
10
10
10
Nart dlasndra
10
NorD Sasainn
Nett-Whouul
10
10'
10a
10
10¹
16a
Communication Rounds
Communisation Founds
Figure 18: Behavior comparison of the NetC models with respect to network width on CIFAR-10 (Optimizer: NMom-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
32

---

Under review as a conference paper at ICLR 2020
Tast Accuracy Global Model
Moan of Each Learer's Training Loss
0.9
0.5
Narrowsst
Netk Rask-la
0.8
Test Acouneyy
0.3
Treinng
0.2
0.
Ness Marrowest
Nost Narrowrr
Basoline
0.5
0
200 Communication Rounds
100
300
100
200
300
Communication Rounds
Weights of 11
Weights of C1,
10
10"
10
10
10
NotA Narroess
Nett- Narrwwrr
10
Neth Namowee
Neth Nerk--a
Gaseline
10
10"
10a
Communication Rounds
Communicatiio Rounds
Weights of 22
Weights Co.
10
P0
PDVV
10
Narroest NetA--taeniii
10
10
10
100
10'
10a
Communcction Rounds
Communication Rounds
Weights of 21
Weights of C..
10
10
10
10
1
Nott MAMTOAA
10
Busdline
10
10"
10a
10"
Communication Rounds
Communication Rounds
Weights of
Weights ofF,
10
10
10
10
9
Nett- NarT-Wad NetA- Baseline
16 Neth Norronns NetA Sardlin
160
10
10a
10
10'
16a
Communccation Rounds
Communication Rounds
Figure 19: Behavior comparison of the NetA models with respect to network width on CIFAR-10 (Optimizer: Adam-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
33

---

Under review as a conference paper at ICLR 2020
Test Acsuracy Global Model
Moan of Each Learer's Training Loss
0.0
0.8
Baselins
0.0
0.8
Nert doxt
Test Agquracy
0.7
0.0
0.6
0.05
0.4
HorB Basclin
0.2
New Wilest
Net wildr
0.1
200 Communication Rounds
100
300
100
200
300
Communication Rounds
Weights C3.
Weights of C₁,
10
10
PD-L5
PD.LL
10
10
10
NerB-Gasoline NetD-Waderr
10
NetB Barelre
Nertt vAtte
NeVE--WWdee
Neett WWliee
10
10 Communication Rounds
Communication Hounds
Weights C21
Weights Cg1
PD
10
10
10
NeBB Busobro Net
10
NotB Baseline NetB Weleot
10
10
10a
10
10'
10a
Communication Rounds
Communication Rounds
Weights of Cg
Weights of C..
10
10
PDLL
10
10
1019
NUB Water
NuB Bosctino
10
NoE Wadest
10! Communication Rounds
Communication Hounds
Weights
Weights of
L5
10
10
Narts Ciesolin North Widdest
10 NetB Baseline
NetB Whleott
10
10'
10a
10
10"
16a
Communication Rounds
Communication Rounds
Figure 20: Behavior comparison of the NetB models with respect to network width on CIFAR-10 (Optimizer: Adam-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
34

---

Under review as a conference paper at ICLR 2020
Test Acouracy of Global Model
Moan of Each Learer's Training Loss
0.9
0.8
Exselins
0.8
0.8
NetC Wriest
Test Agsurayy
0.7
0.0
0.0
0.4g
NenD Basckin
NenD Wider
0.2
0.1
NenD Wileet
200 Communication Rounds
100
300
100
200
300
Communication Rounds
Weights of C1
Weights of C.,
10
10a
PDLs
PD.L
10
10
10
Next
10
NetC
Nett Whser
NeHC-Wide:
Nutc Weooot
NeC--Wodest
10
10' Communization Rounds
10"
10aa
Communication Rounds
Weights of C21
Weights
1
PD
10
10
10
10
-all--- Nell--WWder
WirttWWidrr
10
10"
10
10
10'
10a
Communication Rounds
Communication Rounds
Weights of C2.
Weights of
2
10a
PDL5
10
10
10
NNO Water
Noz Buseline
10
NeiC Reoveins
Nait--Wide
NotC Wicest
NotC Widess
10
10' Communicatio Rounds
10"
10"
Communication Rounds
Weights F,
Weights ofF,
10
10
10
10
Narc daasstre Neri- Wilioet
16 NetC Baseline
NHC Wiiees
10
10a
10a
10
10'
16a
Communication Rounds
Communication Rounds
Figure 21: Behavior comparison of the NetC models with respect to network width on CIFAR-10 (Optimizer: Adam-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
35

---

Under review as a conference paper at ICLR 2020
Test Acturacy Global Model
Meen of Each Learner's Training Loss
0.9
WDD Facker: 0.0
0.9
0.8
0.8
WD Factor: 0,0025
WD Factor: 0031
Test Accurany
0.7
0.7
0.6
0.00
50
0.4
0.5
0.3
0.4
AD 00005
0.2
WD Factor 0 a001
0.1
WD Facter 0 0,006
0.3
200 Communication Rounds
100
300
100
200
300
Communication Rounds
Weights of C
Weights C₁,
10
10
PDLs
PD-VL
10
10"
Pacter:
0.0
10
WD Factor: 0 0001
10
WD Fadtor 0.0001
Factor 0.00000
Fantor 0 0005
Fautor 0.0005
10'
10"
10a
Communication Rounds
Communication Founds
Weights C1
Weights C2r
PD
10
WD Facter: 0.0
MD Foctor 0.0
10
ND Factor 0 00005
Factor
10
WD Fextor 0.00008
Foxtor 0.0006
10
10"
10"
10
10'
102
Communication Rounds
Communisation Rounds
Weights of G
Weights C.
10
10"
PD15
PD-VL
10
WD Factor:
Foctor 0.
10
WD Facter: 0.0051
10
WT Fadtee 0.0001
0,00005
IND Facto: 0 0.0055
Fadter 0.000
10
Communication Rounds
Communtcation Founds
Weights g
Weights of
10
L5
10
WD Packen:
WD Fedtor 0.0
10
IDD Facke: D 0oc0s
10
Factor 0.0000
Factor: 0005
Factor 0.0000
10
10
10"
10
10'
10a
Communication Rounds
Communisation Rounds
Figure 22: Behavior comparison with respect to weight decay levels in the training of NetA- Baseline on CIFAR-10 (Optimizer: pure SGD). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
36

---

Under review as a conference paper at ICLR 2020
Accuracy Glabal Model
Mean Each Training Loss
0.9
WD
0.9
0.8
0.8
WD Factor: 0 00001
WD Fackor 0,005
Test Test Accuraey
0.7
0.6
0.0
0.6
0.4
0.5
0.3
044
WD 0,00005
WD Fecoer 0.0005
WD Fecor 0.0001
0.1
100
200
300
100
200
300
Communication Rounds
Communication Floudss
Weights C,.
Weights C33
1
10
PD-Ls
PD-WL
10
10"1
0.0
10
WD Factar 0:0001
Factor 0,00006
10
WD Fattor: D 0051
Fastor: 00006
Foctor 0.0005
Fartor: 8.055
10!
10
10
Communicction Rounds
Communicatio Rounds
Weights 21
Weights C21
22
10
PD
PDWV
10
10"
WD Focto: 0.0
WD Factor 0.0
10
WD Focto: 0,00005
10
WD Factor: 000006
0.0005
0005
10
10'
10a
10
10
10
Communication Rounds
Communication Rounds
Weights of Cg
Weights Cg
10
PD-L5
PD-WL
10
10
Facten 0.9
10
WD Farter 0.0001
0:00005
104
WD Paraon D 0.001
0,00205
Twetor
Factor
10!
10
Communication Rounds
Communication Floudd
Weights of
Weights of
PD-Ls
PDWVL
10
1011
WD Poctor 00
WD Fartor 0.0
10
WD Factor 0.00000
109
WD Fartor: 00006
WD Foctor 0005
0005
10
10"
10a
160
10¹
10a
Communication Rounds
Communication Rounds
Figure 23: Behavior comparison with respect to weight decay levels in the training of NetA- Baseline on CIFAR-10 (Optimizer: NMom-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
37

---

Under review as a conference paper at ICLR 2020
Test Ansuracy Global Model
Mean of Each
Training Loss
0.0
HND
0.9
0.0
0.8
WD Fackor 0.0025
Facker: 0 00301
Test Agguracy
0.7
0.7
0.6
6.6
0.5
0.4
0.5
0.4
ADD Fackor: 0
00006
0.9
WD Factort 0 0001
ND Fackor 0 0036
0.1
200 Communication Rounds
100
300
100
200
300
Communication Founds
Weights C
Weights C33
10
10
RD-L5
PD-VL
10
WD Factor: 0001
Factor: 00000
5
WD Factor 0.0001
Factoo 3,00006
Fartor: 0.0005
Factor 0.0055
10
10
10
Communication Rounds
Communtcation Founds
Weights of 22
Weights C21
2
10
P01s
10
10
WD Factor 0.0
HD Factor 3.0
10
ID Focto: 0.00005
WD Facto 0.00006
0.005
Factbo 0.0005
10
10
10a
10P
10
10
Communication Rounds
Communisation Rounds
Weights of Cg
Weights C
10
2
8D-15
PD-NL
10
10
0.0
10
WD Famter:
00001
10
WD Fartar 0.0001
0.00025
Tarter
Parttr
10!
10P
Communication Rounds
Communtcation Flounds
Weights
Weights of
RD-Ls
PO-VL
WD Foctor: 80
WD Pactor 3.0
9
WD Factor 0 0000006
10.4
Factor 0.0000
0005
Factor
10
10
10a
10
10¹
10
Communication Rounds
Communisation Pounds
Figure 24: Behavior comparison with respect to weight decay levels in the training of NetA- Baseline on CIFAR-10 (Optimizer: Adam-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
38

---

Under review as a conference paper at ICLR 2020
Teat Accuracy of Global Model
Mean of Each Learner's s Training Loss
0.5
0.4
Test AcouRi))
Trahing Loss
0.3
0.7
0.2
0.6
0.1
0.5
0
300
Communication Rounds
Communication Flounds
Weights 51
Weights C1
2
PD-LS
PD-VL
10
10
U-000 Mn
10
Qui Aumb.1
10
10
10
10
10' Communication Rounds
10
Communication Rounds
Weights of 22
Weights
10
PD-L5
PO-YL
10
10
-0....11
10
p.0.1
p-0.1
10
10' Communication Rounds
10
10' Communication Rounds
10"
Weights Cal
Weights C.s
2
P0Ls
PD-VL
10
Q
un0. und.1
10
unoi 7umb.
10
10'
10"
10
10'
10
Commuriasion Rounds
Communication Rounds
Weights of 3
Weights of F2
1
10
PD-Ls
PD-VL
101
10"4
1
P0 Sem00.0 of
10
p-0.1
10
10! Communication Rounds
10a
10'
10a
Communication Rounds
Figure 25: Behavior comparison under Non-IID(2) setting with respect to FedP rox factor μ in the training of NetA-Baseline on CIFAR-10 (Optimizer: NMom-A).
39

---

Under review as a conference paper at ICLR 2020
Teat Ansuracy Global Model
Mean Learne''s Training Loss
0.8
0.0
Buzeline
0.0
0.4
BPN
Test Acpunne
Treinnn Less
0.3
0.2
Besefine
0.1
wn BN
0
100
200
300
100
200
300
Communication Rounds
Communication Rounds
Weights C3
Weights C33
10
2
PD
PD-VL
10
10
1019
Blase we BN
BRN
10
10a Communication Rounds
10a
10
10¹ Communication Rounds
10a
Weights C21
Weights C21
10
10
PD
PD-L
Boso Ind BRN
10
g
Dioonine BRN
10P
10
10a
10
10"
10"
Communication Rounds
Communication Rounds
Weights of Ca
Weights Ca
10
10
PD-L5
PD-NL
10
10
4
No RN
MRN
10! Communication Flunds
10P
Communtcation Flounds
Weights
Weights of
2
PD-Ls
POWL
10
Base
10
-ossiine BRN
BRN
10
10'
10a
10
10a Communisation Pounds
10a
Communication Rounds
Figure 26: Behavior comparison with/without Batch Kormalizatiom/Rnnrmalizzz in the training of NetA-Baseline on CIFAR-IO (Optimizer: pure SGD). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
40

---

Under review as a conference paper at ICLR 2020
Teat Acsuracy Global Model
Mean of Learner's Training Loss
0.8
0.5
Buzetine BPNN
0.0
0.4
Test Acouneyy
Traning Locs
0.3
0.2
Besefine
0.1
An BN
100
200
300
100
200
300
Communication Rounds
Communication Founds
Weights C2
Weights C33
10
PD
PD-VL
10
10
10
Blase we BN
--oooiin wi BV
BRN
all
10
10a Communication Rounds
10a
10
10 Communtzation Founds
10a
Weights 22
Weights C21
1
1
PDLs
PO-VL
10
10
Bosoinn BRN
vll BRN
10
10
10²
10P
10
10a
Communication Rounds
Communisation Rounds
Weights of Cg
Weights Ca
PD-L5
PD-NL
10
10
4
No RN MRN
10! Communication Rlunds
10P
Communtcation Flounds
Weights
Weights of
PD-Ls
POWL
10
Base BRN
10
Bossline BRN
10
10'
10a
10
10 Communisation Pounds
10?
Communication Rounds
Figure 27: Behavior comparison with/without Batch of on CIFAR-IO (Optimizer: NMom-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
in the training
41

---

Under review as a conference paper at ICLR 2020
Test Acouracy Global Model
Mean of Learner's Training Loss
0.9
0.5
Buzetine BPNN
0.8
0.4
Tea Acourayy
0.7
Traning Locs
0.3
0.5
0.4
0.1
EN
wi DRNN
0.3
300
100
200
300
Communication Rounds
Communication Fluudds
Weights of
Weights of C..
PD-LS
PD-YL
10"
10
10
50
--oooiin wi BN ael BRN
wel BRNN
10
10 Communication Rounds
10
10a
Communication Rounds
Weights 21
Waights
1
PD
PD-VL
10-4
10
Unenn
Dioo BRRN
10
10"
10
10P
101
10a
Communication Rounds
Communication Rounds
Weights of C,
Weights C
PDLLs
PDVV
10
10
10
BN wo BRN
B% adl BRN
10
101
10"
Communicatio Rounds
Communication Rounds
Weights
Weights
1
2
L5
10
10
Bosoline BRN
BRN
10
10!
10a
10
10¹
16a
Communication Rounds
Communisation Pounds
Figure 28: Behavior comparison with/without Batch Normaliizatoo/Reeomaaaaza in the training of NetA-Baseline on CIFAR-10 (Optimizer: Adam-A), Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
42

---

Under review as a conference paper at ICLR 2020
Test Acsuracy Global Model
Mean of Each Learne''s Training Loss
0.8
O Buacinn
0.9
0.8
Teat Agauracy
0.7
0.66
0.7
80.5
0.4
0.3
0.2
O Baseline 575 wi 04
0.1
100 Communication Rounds
200
300
100
200
300
Communication Flunds
Weights C37
Weights C33
10
10
PD-L5
PD-VL
10
10
10
Rooo knn
-Rooniee aal DIA
10 Communication Rounds
10
10 Communtzation Founds
Weights of C21
Weights C21
22
10
PO-VL
10
10
10
16
Bosolino
10
10
10a
10P
10
102
Communication Rounds
Communisation Rounds
Weights of Cg
Weights Cg)
10
2
PD-L5
PD-NL
10
1
Rovon km0
10
10! Communication Rounds
10P
Communtcation Flounds
Weights
Weights of F
2
RD-LS
PO-VL
10
9
10
DA
10
10a
10a
10
10a
16a
Communication Rounds
Communisation Founds
Figure 29: Behavior comparison with/without data augmentation in the training of on CIFAR-10 10 (Optimizer: pure SGD). Dotted and solid lines indicate the results under IID and Non- setting, respectively.
43

---

Under review as a conference paper at ICLR 2020
Test Acsuracy Global Model
Mean of Each Learne''s Training Loss
0.8
O Buaciin
0.0
0.8
Test Agcuracy
0.7
0.6
67
0005
@ 0.4
0.3
O Basoline 525 wi 0A
0.1
100 Communication Rounds
200
300
100
200
300
Communication Flunds
Weights C3.
Weights C33
10
10
PD-L5
PD-VL
10
10
10
Rooo inn
-Roonine aal DI
10
10
10
10 Communtzation Founds
Communication Rounds
Weights of 22
Weights C21
22
10
PD
10
10
10
16
C
DA
10
10
10a
10P
10 Communisation Founds
102
Communication Rounds
Weights of Ca
Weights C.)
10
10
PD-L5
PD-NL
10
Rovon 1m
10
10!
10P
Communication Rounds
Communtcation Fiounds
Weights
Weights of
PD-Ls
PD-VL
10
9
10
DA
10"
10a
10a
10
10a
10a
Communication Rounds
Communisation Founds
Figure 30: Behavior comparison with/without data augmentation in the training of NetA-Baseline on CIFAR-I0 (Optimizer: NMom-A). Dotted and solid lines indicate the results under IID and Non- IID(2) setting, respectively.
44

---

Under review as a conference paper at ICLR 2020
Test Acsuracy Global Model
Mean of Each Learne''s Training Loss
0.8
O Buadiin
0.9
0.8
Test Agcuracy
0.7
0.6
0.0
5 0.4
0.3
0.9
O Basoline
0.1
57 wi 0A
100 Communication Rounds
200
300
100
200
300
Communication Founds
Weights C3.
Weights C31
10
10
PD-L5
PD-VL
10
10
Rooo knn
-noonin aal DIA
10
10
10
10 Communtzation Founds
Communication Rounds
Weights of 22
Weights C21
2
10
P0
10
10
10
16
DA
10
10
10a
10P
10 Communisation Rounds
10
Communication Rounds
Weights of Cg
Weights C.
100
1
RD-L5
PD-WL
10
10
10
10! Communication Rounds
10P
Communtcation Flounds
Weights
Weights of
RD-Ls
PDNL
9
10
DA
10"
10a
10a
10
10a
16a
Communication Rounds
Communisation Founds
Figure 31: Behavior comparison with/without data augmentation in the training of NetA-Baseline on CIFAR-10 (Optimizer: Adam-A), Dotted and solid lines indicate the results under IID and Non- IID(2) setting, respectively.
45

---

Under review as a conference paper at ICLR 2020
Test Ansuracy Global Model
Mean of Learner's Training Loss
0.8
0.0
O Buaciin
0.4
Teat Agcuracy
Traning Locs
0.3
0.7
0.2
0.1
O Basoline
52tw00
100 Communication Rounds
200
300
100
200
300
Communication Founds
Weights C3
Weights C11
10
10
PD-L5
PD-VL
10
10
10
Roso knn
-nooniiee all
10 Communication Rounds
10
10 Communtzation Founds
Weights of C21
Weights C21
22
10
10
10
10
16
C Boselino
10
10
10a
10P
10
102
Communication Rounds
Communisation Rounds
Weights of Cg
Weights Cg)
10
2
PD-L5
10
10
Tlvon 1m
10
10! Communication Rounds
10P
Communtcation Flounds
Weights
Weights of
2
RD-LS
PO-VL
10
9
10
CO
10"
16a
10a
10
10a
16a
Communication Rounds
Communisation Founds
Figure 32: Behavior comparison with/without Dropout in the training of NetA-Baseline on CIFAR- 10 (Optimizer: pure SGD). Dotted and solid lines indicate the results under IID and Non-IID( setting, respectively.
46

---

Under review as a conference paper at ICLR 2020
Test Acsuracy Global Model
Mean of Each Learner's Training Loss
0.8
0.0
O Buacirnl
0.4
Test Agauracy
Traning Locs
0.3
67
W 100
0.2
0.1
+ Basoline
52tw00
100
200
300
100
200
300
Communicatio Rounds
Communication Floudds
Weights C3
Weights C33
10
10
PD-L5
PD-VL
10
10
10
Roso lnn
-nooiii all DO
10
10
10
10 Communtzation Founds
Communication Rounds
Weights of 22
Weights C21
22
10
PD
10
10
10
16
C Boserino
10
10
10a
10P
10
102
Communication Rounds
Communisation Founds
Weights of Cg.
Weights C.)
20
PD-L5
PD-NL
10
10
Tovon Fm
10
10! Communication Rounds
10P
Communtcation Flounds
Weights
Weights of
PD-LS
PO-VL
10
9
10
CO
10"
10a
10a
10
10a
16a
Communication Rounds
Communisation Founds
Figure 33: Behavior comparison with/without Dropout in the training of NetA-Baseline on CIFAR- 10 (Optimizer: NMom-A). Dotted and solid lines indicate the results under IID and Non-IID( setting, respectively.
47

---

Under review as a conference paper at ICLR 2020
Test Acsuracy Global Model
Mean of Each Learne''s Training Loss
0.8
0.0
O Buacirnl
0.4
Test Agauracy
Traning Locs
0.3
0.2
0.1
C Basoline
525w000
100
200
300
100
200
300
Communication Rounds
Communication Flunds
Weights C3
Weights C33
10
10
PD-L5
PO-W1
10
10
Roso lnn
-nooiie all DO
10
10
10
10 Communtzation Founds
Communication Rounds
Weights of 22
Weights C21
2
10
P0
10
10
10
16
C Boserino
10
10
10a
10P
10 Communisation Rounds
10
Communication Rounds
Weights of Cg
Weights C.)
100
2
9D-L5
10
10
10
10! Communication Rounds
10P
Communtcation Flounds
Weights
Weights of F
RD-LS
PDNL
9
10
CO
10"
10a
10a
10
10a
16a
Communication Rounds
Communisation Founds
Figure 34: Behavior comparison with/without Dropout in the training of NetA-Baseline on CIFAR- 10 (Optimizer: Adam-A). Dotted and solid lines indicate the results under IID and Non-IID(2) setting, respectively.
48

---

Under review as a conference paper at ICLR 2020
Adam-A at 25th Round
Adam-WB at 25th Round
0.05
0.00
2.04
0.24
070
0.02
0.02
1,00
00.06
Adam-A at 50th Round
Adam-WB at 50th Round
3.04
-0.06
0.004
-0.05
Adam-A at 100th Round
Adam-WB at 100th Round
0.02
0.00
0.00
0.02
00.02
1.80
-2.04
-0.04
-0.00
-0.00
-0.06
0.24
-0.02
3.00
0.02
0.04
0.66
-0.06
-0.05
-0.02
0.00
0.02
0.04
0.06
Adam-A at 300th Round
Adam-WB at 300th Round
3.04
0.24
0.02
0.02
0.00
0.00
180
-2.02
-0.22
-0.04
-0.04
0.06
00.09
-0.6
-0.05
-0.02
0.00
0.02
0.04
0.06
-0.55
-0.6
-0.02
0.00
0.02
0.04
0.06
Figure 35: Loss surface of global model parameters with Adam-WB and with Adam-A under Non-IID(2) setting.
49

---

Under review as a conference paper at ICLR 2020
w/o BN at 25th Round
w/ BN at 25th Round
0.20
0.5
0.02
0.81
0.00
00.2
-0.55
-0.66
-5.04
-0.03
-0.02
-0.03
0.00
0.01
0.52
0.01
0.01
w/o BN at 50th Round
w/ BN at 50th Round
0.64
0.04
5.02
0.01
0,11
5.00
-0.11
9.82
-0.04
-0.66
-5.53
-0.02
-0.02
0.02
0.03
0.02
0.50
5.04
w/o BN at 100th Round
w/ BN at 100th Round
5.04
0.00
6.02
0.02
5.01
-001
-0.01
-3.03
-0.83
-0.04
-0.04
-0.03
-0.02
-0.01
0.00
0.03
0.02
0.03
0.04
2..04
w/o BN 300th Round
w/ BN at 300th Round
6.04
0.00
0.03
0.03
0.02
0.02
5.00
0.00
-5.02
-0.02
-0.04
-0.03
-0.02
-0.01
0.00
0.01
0.02
0.53
0.04
-0.04
-0.03
-0.02
-0.0
0.00
0.01
0,52
5.00
0.04
Figure 36: Loss surface the global model parameters with/without Batch Normalization under Non-IID(2) setting.
50


Reviews:
[{"review_id": "r1lHGdF0FH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper presents an empirical study of causes of parameter divergence in federated learning. Federated learning is the setting where parameter updates (e.g. gradients) are computed separately on possibly non-IID subsamples of the data and then aggregated by averaging. The paper examines the effects of choice of optimizer, network width and depth, batch normalization, weight decay, and data augmentation on the amount of parameter divergence. Divergence is defined as the average cosine distance between pairs of locally-updated weights, or between locally updated weights and weights trained with IID data. The paper generally concludes that regularization methods like BN and weight decay have an adverse effect in the federated setting, the deepening the network has an adverse effect while widening it might be beneficial, and that adaptive optimizers like Adam can perform poorly if their internal statistics are not aggregated.\n\nI recommend that the paper be rejected. The main shortcoming of the paper is the lack of rigororous statistical analysis to support its conclusions. The paper contains a lot of raw data, but the discussion mainly highlights trends that the authors seem to have observed in the results, without quantifying the relative sizes of effects, how consistent they are across experimental conditions, etc. The writing is also quite unclear, to the point that I often didn't understand exactly what argument was being made.\n\nDetails / Questions:\nThe main problem is the lack of quantitative analysis of the trends the paper identifies. For example, regarding \"Effects of Batch Normalization\", there seem to be two claims made:\n1. Batch normalization makes things worse (somehow) in the federated setting\n2. Batch re-normalization still makes things worse, but not as much\nHow are these effects quantified? How large are they? Do they hold across all datasets, architectures, and optimizers considered? Ideally there would be a table summarizing each experimental manipulation, its effect on performance, whether that effect is significant, etc. Of course this requires some care because the paper is doing an exploratory analysis and there are many hypotheses to test; a good reference is [1].\n\nThe paper also relies heavily on parameter divergence as a measure of performance in federated learning, but I see no evidence presented that parameter divergence is predictive of test accuracy (which is presumably what we actually care about). Intuitively I can see how it might be related, but since divergence is basically being used as a proxy for accuracy, it is vital to show convincingly that the two are related. What do we gain by analyzing parameter divergence rather than simply comparing test accuracy?\n\nRegarding the \"steep fall phenomenon\": The paper seems to present this as an indicator that a manipulation performs poorly in the federated setting. But, isn't it a good thing if parameter divergence goes down? Why does specifically a sudden, sharp decrease in divergence indicate a problem?\n\nFinally, some improvements might be made to the experiment setup. For one, the case of completely-disjoint label sets in different local learners seems extreme to me. Wouldn't at least partial overlap be more common in practice? (This is not my area so I don't know). Experimenting with different degrees of overlap would be useful. As for network architectures, it would be valuable to look at a greater variety of standard architecture styles (e.g. ResNet, Inception, etc). I realize there are some experiments with ResNet, but the focus is mainly on the single-path VGG-like architecture. I do realize this is a lot of experiments to do.\n\nMinor points:\n* In the setting described as \"IID\" in Table 1 is not, the subsampled for each learner are not IID subsamples of the full dataset because they are class-balanced (if I'm understanding correctly)\n\nReferences:\n[1] Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan), 1-30.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJebxqCTtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper experimentally studies the reasons for the slow convergence of the Federated Averaging algorithm when the data are non-iid distributed between workers in the multiclass-classification case. Paper performs extensive experimental study and observes that the main reasons for failure are connected to (i) the parameter divergence during the local steps, (ii) steep-fall phenomena when parameters on different nodes are getting close fast, and to the (iii) high training loss. \n\nMy score is weak reject. The paper provides extensive but unclear experimental results. Improving presentation would significantly improve the paper. For example, why in experimental and theoretical study different parameter divergence metrics were used, etc (see below), why different networks use different optimizers. \nMoreover, provided experimental comparison might be unfair. The learning rate is constant throughout all of the experiments, depending only on the optimizer, but not on the neural network architecture. This can affect the final results. \n\nConcerns and questions that should be addressed:\n1. The initial learning rates were not tuned properly. It is set to be the same for different neural network topologies, which might significantly affect the results. What did the choice of initial learning rates is based on? \n\n2. Why the parameter divergence metric in Definition 1 is not the same as in the theoretical study (Appendix B)? What is the intuition behind Definition 1?\n\n3. Why the divergence of parameters is considered only at the last layer? It seems to hide many important interactions in the other layers. \n\n4. Some important experimental details --- should be added:\n   - At which moment the parameter divergence is computed in the plots? Is it computed at the end of the local iterations right before synchronization? \n   - How the training loss was computed in the plots? before or after synchronization? on the local only or the global data?\n   - Which batch size was used? \n   - Improve the figure caption to detail the experimental setup. (e.g. in fig 3. the network architecture was mentioned only for one of the figures, include which optimized was used, etc)\n\n5. In experiments on Fig. 2. and Fig.3 (middle) what is the accuracy for IID baseline? Is the observed phenomena connected to the poor network architecture or to the non-iid data? \n\n6. In table 5 of the appendix, why experiments use Adam optimizer, but not Momentum SGD as in the main paper to compare the performance of ResNet14 and ResNet20?\n\n7. Better re-prase the definition of the steep fall phenomena, now it is not very clear: in the IID setting parameter divergence values are also sometimes reducing sharply; in the network width study parameters divergence doesn’t experience sudden drop. Also, how does this phenomena (and parameter divergence too) connects to the training loss? \n\n8. Why for different experiments different baseline models are used? (NetA, NetB, NetC)\n\n\nOther minor comments: \n- Appendix B, first equation on page 13. (d_q)^t -> (d_q)^t_k; The size of gradient \\nabla_w [E ...] is different from the size of (d_q)_k. They cannot be added together.\n- page 7, last sentence of the first paragraph: what is the accuracy achieved with Batch Renormalization? Why the reason for accuracy gap is “significant parameter divergence”? on fig. 3 “parameter divergence” is smaller than for the baseline.\n- Why the name of the section on page 7 is “excessively high training loss of local updates” if later it is stated that it is actually smaller than for the IID case? \n- Defenition 1, line 4: “the then” -> “the”\n- section 3: “A pleasant level of parameter divergence can help to improve generalization” -> where was it shown?\n- section 4.2, paragraph 2: what is meant by “hyperparametric methods”?\n- section 4.2, paragraph 3: “quantitative increase in a layer level” -> not clear what does it mean.\n- page 4, effect of optimizers: what do you refer to as “all model parameters”? \n- page 5, last paragraph: Hinton et al... -> (Hinton et al…). Use \\citet(\\citep) instead of \\cite. \n- why Dropout yields bigger parameter divergence if on Fig 2, right it actually helps? \n- Last line of the page 5. Where was this observed? \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByguveDTFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors empirically investigate parameter divergence of local updates in federated learning with non-IID data. The authors study the effects of optimizers, network depth/width, and regularization techniques, and provide some observations. In overall, I think this paper study an important problem in federated learning.\n\nHowever, there are some weakness in this paper:\n\n1. The paper is nearly pure empirical. There is no theoretical analysis supporting the observations proposed in Section 4.1, which weaken the contribution of this paper.\n\n2. This paper only raises some issues in federated learning with non-IID data, and discusses the potential causes. No suggestions or potential solutions is proposed in this paper, which weaken the contribution of this paper.\n\n3. Since this is nearly a pure empirical paper, I hope the authors can make the experiments thorough. However, there are some experiments I expect to see but not yet included in this paper:\n\n    3.1. The authors only studies Nesterov momentum in this paper. However, in practice, it is more common to use Polyak momentum. I hope the authors can also study FL SGD with Polyak momentum in this paper.\n\n    3.2. In this paper, the authors assume that different workers has the same number of local data samples (in Definition 1). However, due to the heterogeneous setting, it is very likely that different workers have different numbers of local data samples, which could be another source of divergence. Furthermore, different numbers of local data samples also results in different numbers of local steps, which may also cause divergence.\n\n    3.3. [1] proposes a regularization mechanism (FedProx) to deal with the heterogeneity. Instead of studying weight decay, it is more reasonable to study the regularization technique proposed by [1].\n\n\n4. There are some missing details (maybe they are already in the paper but I didn't find them):\n\n    4.1. What is the definition of Adam-A and Adam-WB? And, what are the differences between Adam-A, Adam-WB, and vanilla  Adam? (and also, what is the \"A\" in NMom-A?)\n\n    4.2. When using Adam in federated learning, how are the variables synchronized? Note that for Adam, there are 3 sets of variables: model parameters, 1st moment, and 2nd moment. Due to the local updates, all the 3 sets of variables are not synchronized. When the authors use Adam in FL, did they only synchronize/average the model parameter and ignore the 1st and 2nd moments, or did they synchronize all the 3 sets of variables?\n\n\n----------------\nReference\n\n[1]  Li, Tian et al. “Federated Optimization for Heterogeneous Networks.” (2018).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of federated learning for non-i.i.d. data, and looks at the hyperparameter optimization in this setting. As the reviewers have noted, this is a purely empirical paper. There are certain aspects of the experiments that need further discussion, especially the learning rate selection for different architectures. That said, the submission may not be ready for publication at its current stage.
Decision: Reject