FedMix: Approximation of Mixup under Mean Augmented Federated Learning
Authors: Tehrim Yoon
Sumin Shin
Sung Ju Hwang
Eunho Yang
Venue: ICLR 2021 Poster (2021)
Keywords: federated learning
mixup

Abstract: Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device, thus preserving privacy and eliminating the need to store data globally. While there are promising results under the assumption of independent and identically distributed (iid) local data, current state-of-the-art algorithms suffer a performance degradation as the heterogeneity of local data across clients increases. To resolve this issue, we propose a simple framework, \emph{Mean Augmented Federated Learning (MAFL)}, where clients send and receive \emph{averaged} local data, subject to the privacy requirements of target applications. Under our framework, we propose a new augmentation algorithm, named \emph{FedMix}, which is inspired by a phenomenal yet simple data augmentation method, Mixup, but does not require local raw data to be directly shared among devices. Our method shows greatly improved performance in the standard benchmark datasets of FL, under highly non-iid federated settings, compared to conventional algorithms.

Published as a conference paper at ICLR 2021
FEDMIX: APPROXIMATION OF MIXUP UNDER MEAN AUGMENTED FEDERATED LEARNING
Tehrim Yoon & Sumin Shin & Sung Ju Hwang & Eunho Yang Korea Advanced Institute of Science and Technology (KAIST) Daejeon, South Korea
(tryoon93. 93, sym807 07, jhwang82, sunhoy]ekaist... www.kaastta.............. ac.
ABSTRACT
Federated learning (FL) allows edge devices to collectively learn a model with- out directly sharing data within each device, thus preserving privacy and elim- inating the need to store data globally. While there are promising results under the assumption of independent and identically distributed (iid) local data, cur- rent state-of-the erogeneity of local data across clients increases. To resolve this issue, we propose a simple framework, Mean Augmented Federated Learning (MAFI.,, where clients send and receive averaged local data, subject to the privacy requirements target applications. Under our framework, we propose new augmentation algorithm, named FedMix, which is inspired by a phenomenal yet simple data augmenta- tion method, Mixup, but does not require local raw data to be directly shared among devices. Our method shows greatly improved performance in the standard benchmark datasets of FL., under highly non-iid federated settings, compared conventional algorithms.
algorithms suffer from performance degradation as the het-
INTRODUCTION
As we enter the era of edge computing, more data is being collected directly from edge devices such as mobile phones, vehicles, facilities, and so on. By decoupling the ability to learn from the delicate process of merging sensitive personal data, Federated learning (FI.) proposes a paradigm that allows a global neural network to learn to he trained collaboratively from individual clients without directly accessing the local data of other clients, thus preserving the privacy of each client (Konečný et al., 2016; McMahan et al., 2017). Federated learning lets clients do most of the computation using its local data, with the global server only aggregating and updating the model parameters based on those sent by clients.
One of the standard and most widely used algorithm for federated learning is FedAvg (McMahan al., 2017), which simply averages model parameters trained by each client in element-wise manner, weighted proportionately by the size of data used by clients. FedProx (Li et al., 2020b) variant of FedAvg that adds proximal term the objective function of clients, improving statistical stability of the training process, While several other methods have been proposed until recently (Mohri et al., 2019; Yurochkin et al., 2019; Wang et al., 2020), they all build on the idea that updated model parameters from clients are averaged in certain manners.
Although conceptually provides an ideal learning environment for edge devices, the federated learning still has some practical challenges that prevent the widespread application of it (Li et al., 2020a; Kairouz al., 2019). Among such challenges, the one that we are interested in this paper the heterogeneity of the data, as data is distributed non-iid across clients in many real-world settings; in other words, each local client data is not fairly drawn from identical underlying distribution. Since each client will learn from different data distributions, it becomes harder for the model to be trained efficiently, as reported in (McMahan et al., 2017). While theoretical evidence on the convergence of FedAvg with non-iid case has recently been shown in (Li et al., 2020c), efficient algorithms suitable for this setting have not yet been developed or systematically examined despite some efforts (Zhao et al., 2018; Hsieh et al., 2020).

---

Published as a conference paper at ICLR 2021
Global
LocalMix
Nalivohilx
FedMix
Glebal Server
Global Server
Servar
Global Server
(xp
(K,,)))
image
Averagedi image
Averaeed Imsaee
Client
Client
Client
Client
(K, Pikatt data
méuup
(K y)
N
VxL f
(1.9)
xr
(h,y)
GlobalMixup
LocalMix
R NatveMixx
FedMix
(a) Global Mixup
(b) Local Mixup
(c) NaiveMix
(d) FedMix
Brief comparisons Mixup strategies FL MAFL. (a) Global Mixup: Raw data exchanged and directly used for Mixup between local and received data, which violates privacy. (b) Local Mixup: Mixup is only applied within client's local data. (c) NaiveMix: Under MAFL, Mixup is performed between local data and received averaged data. (d) FedMix: Under MAFL, our novel algorithm approximates Global Mixup using input derivatives and averaged data.
Figure I:
In addition to non-iid problem, another important issue is that updating model parameters indi- vidually trained by each client is very costly and becomes even heavier as the model complexity increases. Some existing works Smith et al. (2016); Sattler et al. (2019) target this issue to decrease the amount of communications while maintaining the performance of FedAvg. A more practical approach to reduce communication cost is to selectively update individual models at each round, rather than having all clients participate in parameter updates. This partial participation of clients per round hardly affects test performance in ideal iid settings but it can exacerbate the heterogeneity of weight updates across clients and as a result, the issue non-iid (McMahan et al., 2017).
In order to mitigate the heterogeneity across clients while protecting privacy, provide novel yet simple framework, mean augmented federated learning (MAFL), in which each client exchanges the updated model parameters as well its mashed (or averaged) data. MAFL framework allows the trade-off between the amount of meaningful information exchanged and the privacy across clients, depending several factors such as the number of data instances used in computing the average. We first introduce a naive approach in our framework that simply applies Mixup (Zhang et al., 2018) between local data and averaged external data from other clients reduce myopic bias.
Here, we go further in our framework and ask the following seemingly impossible question: can only averaged data in our framework that has lost most of the discriminative information, bring the similar effect as global Mixup in which clients directly access others' private data without considering privacy issues? Toward this, we introduce our second and more important approach in our framework, termed Federated Mixup (FedMix), that simply approximates the loss function of global Mixup via Taylor expansion (it turns out that such approximation only involves the averaged data from other clients!). Figure briefly describes the concept of our methods.
We validate our method on standard benchmark datasets for federated learning, and show its effec- tiveness against the standard federated learning methods especially for non-iid -id settings. In particular, we claim that FedMix shows better performance and smaller drop in accuracy with more hetero- geneity or fewer clients update per communication round, further increasing difficulty of federated learning.
Our contribution is threefold:
We propose a simple framework for federated learning that averages and exchanges each local data. Even naive approach in this framework performing Mixup with other clients' mashed data shows performance improvement over existing baselines on several settings.
We further develop a novel approximation for insecure global Mixup accessing other clients' local data, and find out that Taylor expansion of global Mixup only involves the averaged data from other clients. Based on this observation, we propose FedMix in our framework approximating global Mixup without accessing others' raw data.
We validate FedMix several FL benchmark datasets especially focusing non-iid data settings where our method significantly outperforms existing baselines while still preserv- ing privacy with minimal increases in communication cost.
2

---

Published as a conference paper at ICLR 2021
2 RELATED WORK
Federated learning Federated learning was first proposed in Konečný et al. (2016) where the prevalent asynchronous SGD (Dean al., 2012) is used update a global model in a distributed fashion. A pioneering work in this field proposed the currently most widely used algorithm, Fe- dAvg (McMahan al., 2017), which is also the first synchronous algorithm dedicated to federated setting. Shortly after, Li et al. (2020b) proposed a variant of FedAvg, named FedProx, where the authors claimed overcome statistical heterogeneity and increase stability in federated learning- Recent studies attempt to expand federated learning with the aim of providing learning in more di- verse and practical environments such as multi-task learning (Smith et al., 2017), generative models (Augenstein et al., 2020), continual learning (Yoon et al., 2020), semi-supervised learning (Jeong et al., 2020), and data with noisy labels (Tuor al., 2020). Our paper focuses on general federated settings, but it could be considered in such various situations.
However, these algorithms may obtain suboptimal performance when clients participating in FL have non-iid (Zhao et al., 2018; Hsieh et al., 2020) distributions. While the convergence of FedAvg on such settings was initially shown by experiments in McMahan et al. (2017) and later proved in good as it would have been for iid setting.
Li et al. (2020c), it does not guarantee performance Existing algorithms that pointed out this issue have major limitations, such as privacy violation by partial global sharing of local data (Zhao et al., 2018) no indication of improvement over baseline algorithms such as FedAvg (Hsieh et al., 2020). Our method aims to improve performance particularly on these non-iid situations, without compromising privacy.
Mixup Mixup (Zhang et al., 2018) is popular data augmentation technique that generates addi- tional data by linear interpolation between actual data instances. Mixup has been usually applied to image classification tasks and shown to improve test accuracy on various datasets such as CIFARIO and ImageNet- 2012 (Russakovsky et al., 2015), and, on popular architectures such as ResNet (He et al., 2016) and ResNeXt (Xie et al., 2017), for various model complexity. It is also reported in Zhang et al. (2018) that Mixup helps with stability, adversarial robustness (Zhang et al., 2018), calibration, and predictive certainty (Thulasidasan et al., 2019). Mixup is expanding from various angles due to its simplicity and popularity. First, beyond image classification tasks, its effectiveness has been proven in various domains such image segmentation SEaton-Rosen et al., 2020), speech recognition (Warden, 2018), and natural language processing (Guo et al., 2019). Also, several ex- tensions such Manifold Mixup (Verma et al., 2018), which performs Mixup in latent space, or CutMix (Yun et al., 2019), which replaces specific regions with others patches, have been proposed.
In most of the previous studies on federated learning, Mixup was partially (or locally) used gen- eral data augmentation technique. Some recent studies (Oh et al., 2020; Shin et al., 2020) proposed to send blended data to server using Mixup, but they require sending locally- and linearly-mixe (mostly from two instances) data to server at every round, therefore being susceptible to privacy issues with huge communication costs. Our work properly modifies Mixup under the restrictions of federated learning and mitigates the major challenges of federated learning such as non-iid clients.
3 MEAN AUGMENTED FEDERATED LEARNING (MAFL) AND FEDMIX
We now provide our framework exchanging averaged data for federated learning and main method approximating insecure global Mixup under our framework, after briefly introducing the setup.
3.1 SETUP AND BACKGROUND
Federated learning and FedAvg Federated Averaging (FedAvg) (McMahan et al., 2017) has been the most popular algorithmic framework for federated learning. For every communication round t = O...., T 1, client & E 1..... .. N selected for local training sends back its model wf (or only difference to reduce communication cost) to a global server. For every round, K number of clients are selected to locally update and send model parameters. The server simply averages parameters received, so that the global model wi after rounds of communications be- comes w, = Σ Pf where P& is the importance of client & based on the relative number of data in l among all selected clients at t. The updated global model is sent back to clients for the next round, which undergoes the following F. local updates via stochastic gradient descent (SGD))
3

---

Published as a conference paper at ICLR 2021
w 11+1, <- w 7 (=ff w1+1.), vt) for i 0,1,.... learning rate 7). Here, e is the loss function for learning and f(æ; we) is the model output for input a given model weight wt-
..... E 1, batch size B, and local
Mixup Mixup (Zhang et al., 2018) is a simple data augmentation technique using linear inter- polation between two input-label pairs (ar yt) and (æz, Myy augment xæ; (1 A)æ, and y = Az; + (1 X)yr The variable E [0, 1] is a hyperparameter that is chosen from the beta distribution for each training step.
3.2 MAFL: MEAN AUGMENTED FEDERATED LEARNING
The most obvious and powerful way for local models to receive information about data from other clients is simply receive raw individual data. However, under typical federated setting, each client does not have direct access to individual external data due to privacy constraints, leading to overall performance degradation. We propose federated learning framework that relaxes the limitation of accessing others' raw data and allows a more granular level of privacy depending on applications. In our new framework, termed mean augmented federated learning (MAFL), clients not only exchange model parameters but also its mashed (or averaged) data.
In MAFL, only the part that exchanges averaged data of each client has been added the standard FL paradigm (Algorithm 1). Here, the number of data instances used in computing the average, MA. controls the key features MAFL such as privacy and communication costs. Lower M, value results in more relevant information passed over, but only in cost of less secure privacy and larger communication cost. In one extreme of M₁ = 1, raw data thoroughly exchanged and privacy is protected all, which is clearly inappropriate for FL.. But, in the other extreme, all data of each client is averaged to ensure a considerable degree of privacy. In addition, it also has an advantage on communication cost; each client sends set of na na/Ma averaged data where ng is local data size of client k. The remaining question is whether it is possible to improve performance even when exchanging information that is averaged from all local data and loses discriminative characteristics.
The most naive way we can consider in our MAFL framework is to directly use the mashed data from other clients, just like regular local data. However, since mashed data has lot less usable
Algorithm 1: Mean Augmented Federated Learning (MAFL)
Algorithm 2: FedMix
LocalUpdate wr; Xg, Yg) under MAFL (Algorithm 1):
Input: D = {xx, Yx for k = Mc: number of data instances used for computing average T, y
w < wt for = E-
Split D3 into batches of size for batch( X, Y) do Select an entry Bgg By from
Initialize wo for global server
for / O...., do
for client k with updated local data do
X Y,
Split local data into My sized batches
Compute F. for each batch
(1 (x(f((1-)) w) Y)
Send all Φ. ÿ to server end
la xe(f((1- A)x; w). Ho)
fg > 04, Hir Bg
S, + K'llientss selected at random Send w; clients & S, if updated then
(derivative calculated at
x = (1 - -A)a; and y = yr for in X. Y)
each of
Aggregate all . yt to Xg, Y, Send X,, Y, to clients St end for
l + l2 E3
w - M++1 !
end
l E S do wh +- LocalUUpdate(k, wi; Xg, Yg)
end
return w
end
w 1 Ekes, pa,, pa
end
4

---

Published as a conference paper at ICLR 2021
information than local data, we can think of method of mixing it with local data:
(yaiveMix = (1 - 2) ())( ( ( >)2; + Aa)). 31) + dE (f((1- -)2; + Aæz)) (.)))..)) F3
(1) where (a); 31) is an entry from local data and (z f/y) corresponds means of Sinputs,labelss from other client j. Note that Eq. (1) can be understood as the generalization of the loss of directly using the mashed data mentioned above in the sense that such loss can be achieved if X in Eq. (1) is set deterministically to 0 and 1.
In the experimental section, we confirm the effectiveness of MAFL using ExaiveNix. However, in the next subsection, we will show how to achieve better performance by approximating the global Mixup in a more systematical way in our MAFL framework.
3.3 FEDMIX: APPROXIMATING GLOBAL MIXUP VIA INPUT DERIVATIVE
We now provide our main approach in the MAFL framework that aims to approximate the effect of global Mixup only using averaged data from other clients. Consider some client with its local data (#;; J.). It is not allowed in federated learning, but let us assume that client i has access to client j's local data (zz, B2)- Then, client would leverage (æ², Mj) to improve the performance of its local model especially in non-iid settings by augmenting additional data via Mixup:
(1 \)z; + Axy and ÿ = (1 - A)y/ + Azy-
If Mixup rate À is 1, (a my)) from client j is again directly used like a regular local data, and it would be much more efficient than indirect update of local models through the server.
The essence of our method is to approximate the loss function ( 5) for the augmented data from Eq. (2), with Taylor expansion for the first argument æ. Specifically, we derive the following proposition:
Proposition 1 Consider the loss function of the global Mixup modulo the privacy issues,
(clovalMixup (f)) {(5(1 - A)x, + >xx)), (1 X)ys + Xw;)
(3)
for cross-entropy loss fl. Suppose that Eq. (3) is approximated by applying Taylor series around the place where X & 1. Then, if we ignore the second order term (i.e., ((2²)), we obluin the following approximated loss:
(1- - X)e ()e( (()))(1)))))).)) m1 + se xe(s(11 10((1--xjeee -))as) Ms3 +
Oe
23
(4)
Zx.
where the derivative at is evaluated at x = (1 - A)æ; and Vi.
dn
While Eq. (4) still involves æ, and yı. invading the privacy of client j. the core value of Proposition gets clearer when mixing up multiple data instances from other clients. Note that the vanilla Mixup not mixing one specific instance with other data, but performing augmentations among several random selected data. In a non-iid FL environment, we can also expect that the effect will be greater as we create Mixup data by accessing as much private data as possible from other clients. From this point of view, let us assume that client has received a set of M private instances, J, from client j.
Then, the global Mixup loss in Eq. (3)
|J| Σ e (f(1 A)a, + Xa;)) A)ya + +X0s).
jEJ
and the approximated FedMix loss in Proposition becomes
EvedNix = Σ( - A)e( (
1 [J]
A)xx)) 91) + +xe(F((1- 2)x:). + de dx 2j
jEJ
Ol
= = (11-))((((1-)))
+ Ae (f((11 [[((11))x)))..)))))))())))))(()) 81
+
3
(5)
Ox
where we utilize the linearity of Equation 4 in terms of æ; and y,, and a, and 7, correspond mean of M inputs and labels in J, respectively. The algorithmic details are provided in the appendix due to the space constraint (see Algorithm 2 in Appendix A).
'Throughout the paper, we implicitly assume the classification tasks. For regression tasks, can consider the squared loss function, and the proposition still holds.
5

---

Published as a conference paper at ICLR 2021
3.4 PRIVACY ISSUES AND ADDITIONAL COSTS OF MAFL
Privacy issues of MAFL. MAFL requires exchanging averaged data by construction. Even though MAFI, exchanges only the limited information allowed by the application, it may causes new types of privacy issues. The potential privacy risk of FL or MAFL is beyond the main scope of our study, but in this section, we briefly discuss some basic privacy issues of MAFL and potential solutions.
There possibility that local data distribution can be inferred relatively easily from aver- aged data. This issue simply arises Ma is not large enough, so that individual data could be inferred from the averaged data easily. On the other hand, if n2 is not big enough, each entry in g. Yg could reveal too much about the whole local distribution of the client it came from.
It could be easy to infer ownership of each entry g. y,, if it contains client-id specific information. If clients could identify what other client each entry came from, information about local data of that client could be inferred.
Additional concerns involve identification of data by detecting change in exchanged eraged data, in case of continual learning, which involves local data change across time. This issue is exacerbated as there is update of averaged data for every minute change on local data, which makes the client receiving X, Y, easier to infer the changed portion. One simple suggestion to alleviate this issue would be to only update X,, Y, when there is enough change in local data across enough number of clients, so that such changes are not easily exploitable.
As way strengthen privacy protection under MAFL (and possibly to help with issues mentioned above), we in the server can average within entries of X g. Y, If this additional average is done across every random m entries at the server, it would effectively provide averaged data across all local data of m clients, but would result in an m-fold decrease in the number of averaged data. This variant considered Appendix J.
In case where the global server is credential, the averaged data itself should ensure privacy as it is sent to the server. A most obvious concern comes from when Mr is not large enough,
that each entry of M₁ reveals more of information of each individual input. Simply using a sufficiently large value of M2 can alleviate this issue, although this might result worse performance.
However, for clients whose n, is quite small, there is a limit for Mr to be large enough. One way to alleviate this issue is to introduce a cut-off threshold for allowing clients to send averaged data server. We report the results in Appendix H.
Communication cost Since MAFL requires sending averaged input data between server and clients, additional communication costs are incurred. However, it turns out that this additional cost is very small compared to communication cost required for exchanging model parameters, This is mainly due to the fact that input dimension is typically much smaller than number of model param- eters. Specifically, for input dimension d;, exchange of averaged data among N clients incurs 2Nd; cost (factor of for server receiving and sending the values). Meanwhile, the cost for exchange of model parameters is 2N Pm where pm is number of model parameters. Under typical circumstances, averaged data is only exchanged at the beginning of the first communication round, while model parameters have to be exchanged every round. Thus the ratio between the two costs after T commu- nication rounds is (TTP)) Since dj < Pm in general, we consider extra communication burden to be negligible (even in the worst case where we update averaged data every round, the ratio is still (pm))
FedMix also requires calculation of input derivative term in its loss function, potentially extra memory is required. We further provide additional computation costs of MAFL in Appendix G.
4 EXPERIMENTS
We test our result on various benchmark datasets with NaiveMix (direct mixup between local data and averaged data) and FedMix, then compare the results with FedAvg (McMahan et al., 2017) and FedProx (Li et al., 2020b), as well as other baseline Mixup scenarios. We create a highly non-iid environment to show our methods excel in such situations.

---

Published as a conference paper at ICLR 2021
Test Acc. (%)
08
Test Adn. (16
20-
Test Acc. (%)
75-
70-
04
65
10
Locathil
02
50
100
150
203
100
200
300
400
500
100
200
300
400
620
Comm. Round
Comm. Round
Comm. Round
(a) FEMNIST
(b) CIFAR10
(c) CIFAR100
Figure 2: Learning curves for various algorithms on benchmark datasets. Learning curves correspond results in Table (For simplicity, only show key algorithms compare.)
Table 1: Test accuracy after (target rounds) and number of rounds to reach (target test aceu- racy) various datasets. Algorithms in conjunction with FedProx are compared separately (bot- tom). MAFL-based algorithms are marked in bold.
Algorithm
FEMNIST
CIFARIO
CIFARI00
Test acc. (200)
rounds (80%)
test acc. (500)
rounds (70%)
test acc. (500)
rounds (40%)
Global Mixup
88.2
8
88.2
85
61.4
54
FedAvg "3.33"
26
73.8
283
50.4
101
LocalMix
82.8
28
73.0
267
54.8
91
NalveMfix
85,9
'2""
777.4"
198
553.8"
85
FedMix
86.5
18
81.2
162
56.7
34
FedProx
84.6
77.3
266
51.2
79
FedProx + LocalMix
84.1
39
74.1
314
54.0
90
FedProx MaiveMix
85.7
37
76.7
230
53.1
74
FedProx FedMix
86.0
32
78.9
223
54.5
63
4.1 EXPERIMENTAL SETUP
Dataset We implement the typical federated setting where clients have their own local data and one centralized server that receives and sends information from/to the clients. We utilize a large number clients and only utilize partial set of clients chosen each round to locally update. We used three popular image classification benchmark datasets: FEMNIST (Caldas et al., 2019), CIFARIO, and CIFARI00, as well as a popular natural language processing benchmark dataset, Shakespeare. See Appendix B for more details about dataset, models, and hyperparameters used. We introduce data size heterogeneity for FEMNIST dataset: each client has different size of local data, each from introduce label distribution heterogeneity for CIFAR datasets, with
unique writer. Mcanwhile, clients having data with only a limited number classes.
Algorithms We study the performance of FedMix and NaiveMix and compare with FedAvg and FedProx. We also compare method against FedAvg with Mixup within local data (labeled Lo- calMix; see Figure 1(b)), to show whether Mixup within local data sufficient to allow the model to perform well on external data. To show the effectiveness of FedMix, also compare method to the case where we perform direct Mixup with external data (and thus violating privacy, labeled Global Mixup: Figure ((a)..
4.2 PERFORMANCE OF FEDMIX AND NAIVEMIX ON NON-IID FEDERATED SETTINGS
We compare the learning curves of each method under the same federated settings, in terms of num- ber of communication rounds conducted. Comparing MAFL-based algorithms, NaiveMix shows slight performance increases than FedAvg, FedProx, and Localmix, while FedMix outperforms and shows faster convergence than all of FedAvg, FedProx, Localmix and NaiveMix for all datasets tested as in Figure 2.
While NaiveMix and FedMix is already superior to FedProx, they are parallel to FedProx modifi- cation and can be applied in conjunction with FedProx. We compare performances across FedProx

---

Published as a conference paper at ICLR 2021
Table 2:
Test accuracy after 50 rounds on Shakespeare dataset.
Algorithm
Global Mixup
FedAvg
FedProx
LocalMix
NaiveMix
FedMix
Test Acc. (%)
$4.4
54.7
54.4
53.7
$6,9
$6.9
Table 3: Test accuracy CIFARIO, under varying number of clients (N). Number of samples per client kept constant.
Table 4: Test accuracy on CIFARIO, under varying number of local data per client. Number of clients (N) kept constant. Number of data is indicated percentage of the case where all 50,000 data are used.
of Clients(NN)
20
40
60
Local data (%)
20
50
100
Global Mixup
86.3
89.2
88.2
Global Mixup
71.4
86.1
88.2
FodAvg
65.8
73.4
73.8
FedAvg
61.8
74.7
73.8
LocalMix
46.9
71.4
73.0
LocalMix
43.7
60.3
73.0
NaiveMix
62.2
75.1
77.4
NaiveMix
51.5
69.6
77.4
FedMix
68.5
76.4
81.2
FedMix
65.2
77.8
81.2
variants of various Mixup algorithms in Table 1. FedMix outperforms vanilla FedProx for various datasets, although they do fall short of default version of FedMix used for the main experiment.
To confirm whether received information is properly incorporated, compare FedMix with possi- ble Mixup scenarios under MAFL.. We show the results in Appendix D.
While Mixup is usually performed for image classification tasks, it could be applied for language models. For language datasets, since Mixup cannot be performed on input, we perform Mixup on embeddings (for detailed explanation of Mixup between hidden states, see Appendix E). When tested on Shakespeare dataset, FedMix and NaiveMix both show better performance than baseline algorithms (Table 2). Note that for this task, LocalMix has the lowest performance, and global Mixup does not result in the superior performance above federated algorithms as expected. We think Mixup does not provide performance boost for this specific task, but claim that MAFL algorithms still result in better performance compared to FedAvg.
We also claim that FedMix is superior compared to other methods under various settings, in terms of varying number of clients (N) and varying number of local data per clients. We observe superior performance FedMix compared to other algorithms for settings (see Tables and 4). We also vary the number of local epochs (E) between global updates, and still observe that FedMix outperforms other methods (see Appendix F).
FedMix compared to global Mixup with fixed mixup ratio Since FedMix approximates loss function of global Mixup for fixed value of X & 1, we can evaluate the efficiency of approximation by comparing between FedMix and a global Mixup scenario with fixed value. Table shows varying performance between global Mixup and FedMix under various values of X. As À increases, Mixup data reflects more of the features of external data, resulting in better performance of global Mixup. However, this also results in our approximation being much less accurate, and we indeed observe performance FedMix decreasing instead. The result shows that the hyperparameter should be chosen to balance between better Mixup and better approximation. However, it seems that high À results in significant decrease in both methods, probably due to external data (which is of dfstrribbtit for local distribution) being overrepresented during local update.
case
Table 5: Test accuracy on CIFARIO, under varying mixup ratio
A
0.05
0.1
0.2
0.5
Global Mixup FedMix
79.4
80.4
81.1
63.6
81.2
80.5
77.7
67.1
8

---

Published as a conference paper at ICLR 2021
Mg
My
Me
10
20
50
All
E
5
FEMNIST
MaiveMix
FedMix
86.0
85.7
85.7
86.4
86.2
86.2
86.
86.5
CIFARIO
NaiveMix
FedMix
CHAROO
Figure 3: Performance of MAFL-based algorithms for various M₁ values (left), and samples of averaged images from EMNIST/CIFARIO for various Mg values (right).
Table 6: Test accuracy after 500 rounds on CIFARIO, under varying number of classes per client.
Table 7: Test accuracy after 500 rounds CI- FARIO, under varying number of clients trained per communication round.
class/client 5
K/N- 0.25
Algorithm
10 (iid)
Algoritiu
0.1
0.15
0.5
1.0
Global Mixup
88.2
90.7
90.9
91.4
Global Mixup
89.3
89.7
88.2
91.2
90.7
FedAvg
73.8
84.2
86.8
FodAvg
63.3
73.2
73.8
76.3
83.1
Localmix
86.4
Localmix
64.7
64.5
73
77.9
79.8
NaiveMix
77.4
84.5
87.7
89.4
NaiveMix
73.6
74.7
77.4
81.4
83.5
FedMix
81.2
85.1
87.9
89.1
FedMix
74.7
76.9
80.5
82.1
84.3
Effeet of Mj compute mean In algorithm, we chose calculate Xg, Y, with all local data for each client. To observe potential effect of Mr. we varied Mj used to compute the averaged data that is sent from other clients. Inevitably, reducing Mg will result in Xg, Y, having much more rows, imposing additional computation burden and less preservation of privacy. In general, for both FEMNIST and CIFAR1O, 10, there is only small performance decline as privacy is enhanced, as can be seen in Figure 3. We show that using all local data to calculate each mean is sufficient to both preserve privacy and still have good performance.
Mixup between hidden states Manifold Mixup (Verma al., 2018) was proposed to show im- provements over input Mixup (Zhang et al., 2018) in various image classification tasks such as CIFARIO, CIFAR100, and SVHN. We discuss the possibilities and implications of applying Mixup between hidden states in Appendix E. In summary, we show that variants of using hidden states do not show meaningful advances over FedMix using input Mixup, suggesting that in general, relatively inefficient since imposes additional communication burden.
Effeet of non-iid-ness and client participation We claim that our method efficient when faced with non-iid federated settings. For example, our setting of CIFAR 10 having only data from 2 classes per client is very non-iid, as in average a pair of clients share only roughly 20% data distribution. We test settings for CIFAR10 where clients have data from greater number of classes, and while there is little difference for iid (10 class/client) setting, we observe that FedMix outperform other methods and suffer less from increased heterogeneity from highly non-iid settings (Table 6). In addition, we also observe less decline and better performance for MAFL-based algorithms, FedMix in particular, as we train less number of clients per round, reducing communication burden in cost of performance (Table 7).
5 CONCLUSION
We proposed MAFL, novel framework, that exchanges averaged local data, to gain relevant infor- mation while still ensuring privacy. Under the new framework, we first suggested NaiveMix, which is naive implementation of Mixup between local and received data. More interestingly, we pro- posed FedMix, which provides approximation of global Mixup only using averaged data. MAFL, and FedMix in particular, showed improved performance over existing algorithms in various bench- marks, particularly in non-iid environments where each client has data distributed heterogeneously. While our method is very effective and still preserving privacy, future work needs to be done to deal with various con-iid environments, desirably with better privacy and beyond image classification tasks.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea (NRF) grants 12200RRAAAAI0S90
10.2019RICIC1009192 and Institute of Information & Communica- tions Technology Planning & Evaluation (IITP) grants (N0.2017--- XAI, No.2019-0-01371, Development of brain-inspired AI with human-like intelligence, and No.2019.0-00075, Artificial Intelligence Graduate School Program(KAISTT) funded by the Korea government (MSIT).
REFERENCES
Sean Augenstein, H. Brendan McMahan, Daniel Ramage, Swaroop Ramaswamy, Peter Kairouz, Mingqing Chen, Rajiv Mathews, and Blaise Aguera y Arcas. Generative models for effective ml on private, decentralized datasets. International Conference on Learning Representations (ICLR), 2020.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. Inter- national Conference on Machine Learning (ICML) Workshop on Federated Learning for Data Privacy and Confidentialiry, 2019.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. Conference on Neural Information Processing Systems (NIPS), 2000.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. Emnist: an extension of mnist to handwritten letters. International Joint Conference on Neural Networks (ICCNN,, 2017.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoe V. Le, and Andrew Y. Ng. Large scale distributed deep networks. Conference on Neural Information Processing Systems (NIPS), 2012.
Z. Saton-Rosen, Felix J. S. Bragman, Sébastien Ourselin, and M. Cardoso. Improving data aug- mentation for medical image segmentation. Conference on Medical Imaging with Deep Learning (MIDL), 2020.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, 2015.
Hongyu Guo, Yongyi Mao, and Richong Zhang. Augmenting data with mixup for sentence classifi- cation: An empirical study. CoRR, abs/ htss/1955.08941, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Conference on Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kevin Hsieh, Amar Phanishayee. Onur Mutlu, and Phillip B. Gibbons. The non-iid data quagmire of decentralized machine learning. International Conference on Machine Learning (ICML), 2020.
Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency. International Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with ICML 2020 (FL-ICML'20) 2020.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings. Rafael G. L. D'Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song. Sebas- tian U. Stich, Ziteng Sun. Ananda Theertha Suresh, Florian Tramèr, Pranceth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. ArXiv, abs/1912.0499 2019.
10

---

Published as a conference paper at ICLR 2021
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. Conference Neural Information Processing Systems (NIPS) Workshop Private Multi-Party Machine Learning, 2016.
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37:50-60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems (MLSys), 2020b.
Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. International Conference on Learning Representations (ICLR), 2020c.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-effiiiee learning of deep networks from decentralized data. International Con- ference on Artificial Intelligence and Statistics (AISTATS), 2017.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. International Conference on Learning Representations (ICLR), 2018.
Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. Interna- tional Conference on Machine Learning (ICML), 2019.
Seungeun Oh, Jihong Park, Eunjeong Jeong, Hyesung Kim, Mehdi Bennis, and Seong- Kim. Mix2fld: Downlink federated learning after uplink federated distillation with two-way mixup. IEEE Communication Letters, 2020.
Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla. Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (ICV), 115 (3):211-252, 2015.
Felix Sattler, Simon Wiedemann, Klaus-Rohert Müller, and Wojeiech Samek. Robust and Communication-efficeeen federated learning from non-iid data. IEEE Transactions Neural Net- works and Learning Systems, 2019.
MyungJac Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mchdi Bennis, and Scong- Kim. Xor mixup:
data augmentation for one-shot federated learning. Interna- tional Workshop on Federated Learning for User Privacy and Data Confidentiality in Conjunction with ICML 2020 (FL-ICML'200 20), 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
Virginia Smith, Simone Forte, Chenxin Ma, Martin Takac, Michael I. Jordan, and Martin Jaggi. Cocoa: A general framework for communication-eefiiie.. distributed optimization. Journal of Machine Learning Research (JMLR), 2016.
Virginia Smith, Chao-Kai Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learn- ing. Conference on Neural Information Processing Systems (NIPS), 2017.
Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Conference on Neural Information Processing Systems (NIPS). 2019.
Tiffany Tuor, Shiqiang Wang, Bong Jun Ko, Changehang Liu, and Kin K. Leung. Overcoming noisy and irrelevant data federated learning. International Conference on Pattern Recognition (ICPR), 2020.
11

---

Published as a conference paper at ICLR 2021
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. International Conference Machine Learning (ICML),
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos. and Yasaman Khazaeni. Federated learning with matched averaging. International Conference on Learning Representa- tions (ICLR), 2020.
Pete Warden. Speech commands: dataset for limited-vocabiee vocabua-v speech recognition. ArXiv, 0tss/804403209,, 2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Jachong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Federated con- tinual learning with adaptive parameter communication. ArXiv, abs/2003.03 0222000.03166 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. International Conference Computer Vision (ICCV), 2019.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. International Conference Machine Learning (ICML.), 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Pazz mixup: Beyond empiri- cal risk minimization. International Conference on Learning Representations (ICLR). 2018.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, arXiv preprint arXiv:1806.0055 2018
12

---

Published as a conference paper at ICLR 2021
A ALGORITHMS
We present brief depiction of FedAvg in Algorithm
Algorithm 3: FedAvg
Input: N,T, T, K, E, B. Pk, Da (x,, YıJ,k- LocalUpdatee (k; Wt):
1,.... NN, Mt, 0,....?. Initialize Wo for global server do
w 5- Wt
for ...... E 1 do
for t = 0.....
Split D2 into batches of size for batch X, Y) do
S, < Kelients selected at random Send wy to clients & E St
w t w 7+1 V(( Y)
for 1 E E, do
end
w +- LocalUppdate(;, w.)
end
end
return w
K Ekes, pk naw4+1
Wi+1
end
B EXPERIMENTAL DETAILS
FEMNIST FEMNIST is EMNIST (Cohen et al., 2017), handwritten MNIST dataset, organized into federated setting, as in Caldas et al. (2019). EMNIST very similar to MNIST, but has several differences. It includes all 26 capital and small letters of alphabet as classes along with numbers, making it 62 classes in total to classify. Also, each image contains information of the writer of the letter. In a realistic non-iid setting, each client has local data consists of only one writer, which is about 200 to 300 samples per client in average, with differing number of samples. We use N 100 clients and trained only K 10 clients per communication round.
We used L (Lecun et al., 1998) architecture for client training. LeNet is consisted of conv layers followed by 2x2 maxpool layer then 3 fc layers. We used 5x5 conv layers with 6 and 16 channels. Following fe layers have exactly the same hidden dimension of original LeNet-5 model.
CIFARIO and CIFAR100 CIFARIO and CIFAR 100 very popular and simple image classifi- cation datasets for federated setting. Both contain 50,000 training data and 10,000 test data. We split the data into each client, N 60 in case of CIFAR 10 and N 100 in case of CIFAR100. To create an artificial non-iid environment, we allocate data such that each client only has data from 2 (20 for CIFAR100 100) randomly chosen classes. We train only K 15 clients per round for CIFAR10 and K = 10 for CIFAR100. No validation data was split and we used all training data for local training.
We used modified version of VGG architecture (Simonyan Zisserman, 2015). Modified VGGnet is consisted of 6 convolutional layers with 3 max pooling layers. 3x3 conv layers are stacked and 2x2 maxpool layer is stacked after every conv layers. Conv layers have channel sizes of 32, 64, 128, 128, 256, 256. Then 3 fc layers are stacked with hidden dimension 512. We use Dropout layer three times with probability 0.1 after the second, third maxpool layers and before the last fe layer. We re- move all batch normalization layers since it is reported that they hurt federated learning performance (Hsieh et al., 2020).
Shakespeare We use dataset from The Complete Works of William Shakespeare, which is popu- lar dataset for next-character prediction task. We partition the dataset so that each client has conver- sations of one speaking role, as in Caldas et al. (2019), which naturally results in a heterogeneous setting, as in FEMNIST. We use N = K = 60, each with different number of data (minimum is 200). Since input-level Mixup cannot be performed for discrete character labels, we performed Mixup on the embedding layer. Additional concerns for this variation is considered at Appendix E.
We used 2-layer LSTM, both with hidden dimension of 256. The recurret network is followed by an embedding layer. The output of LSTM is passed to a fully-connected layer, with softmax output of one node per character. There are 84 characters used in the dataset.
Local clients are trained by SGD optimizer with learning rate 0.01 and learning decay rate per round 0.999. We set local batch size as 10 for training. Specific hyperparameter setting for each dataset is
13

---

Published as a conference paper at ICLR 2021
Table 8: Hyperparameter settings for each dataset.
dataset
FEMNIST
CIFAR10
CIFARI00
Shakespeare
local epochs (E)
10
2
10
2
local batch size
10
10
10
class per clients
2
20
fraction of Clients (K/N)
0.1
0.25
0.1
total dataset classes
62
10
100
84
À (for NaiveMix)
0.2
0.1
0.1
0.1
À (for FedMix)
0.2
0.05
0.1
0.1
la (for FedProx)
0.1
0.1
0.01
0.001
explained in following Table 8. Throughout the experiment, M2 is fixed to local client's dataset size. Changes these parameters are indicated, if made, are stated for all experiments. Note that we use a fixed small value of À for MAFL-based algorithms to show superior performance.
C PROOF OF PROPOSITION
We demonstrate mathematical proof of Proposition for FedMix.
Starting from Eq. (3), since the loss function linear in y for cross-entropy loss l we have
()()).)) (1 x)e(s(( - A)æ; + Xæ;), (B:) + xe(S((1 A)a; Xax)) M3)
(6)
Unlike the original paper, we assume < 1, we can treat this loss as objective loss function for vicinal risk minimization (VRM). Under this assumption, each term in Eq. (6) can be approximated by independent Taylor expansion on the first and second argument of e, so that we have
ae
(1 (1- (-)))(((1-A))z)) A)xx), y/ + (1 (1-A)=
(Ax,)
dx (1 (1-A)ar..
aE
(x(F((1-))a)) vs) + +
(Ax,)).
dx (1-A)a..13;
Since À & 1, we can ignore the last term in the second row of Eq. (7), which is O(22). We simplify this equation and switch the second term in the first row and the first term in the second row, to finally obtain
(f()))0) =1 (an-))e(F((() ()((F((1))))) u1) xe xe(5((1- A)x.). +
0e dx .z.
The derivative changed to A from A(1 A) since we are ignoring ((22) terms.
0F is calculated at æ (1- A)a; and y The coefficient the last term is Jm
D COMPARISON OF FEDMIX WITH BASELINE MIXUP SCENARIOS
Since our MAFL-based algorithms could be considered as VRM, it could considered as a data augmentation (Chapelle et al., 2000). Thus, it important confirm that increased performance from MAFL not only comes from data augmentation but also from relevant information received from other clients. To check whether this true, we compare FedMix with algorithms where use either randomly generated noises as averaged data for Mixup (labeled Mixup w/ random noise) and where we use only locally averaged data for Mixup (labeled Mixup w/ local means).
In Table 9, we observe that if averaged data for MAFL is substituted for randomly generated noise or locally generated images, it does not show the level of performance FedMix is able show. Thus, we claim that FedMix properly incorporates relevant information received.
We implicitly assume the classification tasks. For regression tasks, consider the squared loss func- tion and the equivalent loss that linear
14

---

Published as a conference paper at ICLR 2021
Figure 4: Results for variants of Mixup algorithms for Mixup between hidden states. (a) Learning
curves for various algorithms with hidden representation Mixup after l = layers. (b) Learning curves for FedMix when Mixup applied after different numbers of layers.
85
FedAvg
Input
LocalMix
k=2
NaiveMix
k=4
Test Acc. (%)
FedMix
Test Acc. (%)
80
k=8
75
75
70
70
65
65
0
100
200
300
400
500
100
200
300
400
500
Comm. Round
Comm. Round
(a)
(b)
E VARIANT OF FEDMIX AND NAIVEMIX WITH MIXUP BETWEEN HIDDEN STATES
While input Mixup methods promise significant enhancements, one can expect similar performance from hidden state Mixup, originally proposed by Verma al. (2018). The authors of this work suggest that Manifold Mixup demonstrates similar, if not greater, advantage in terms of performance and adversarial robustness. We can think of variants of FedMix and NaiveMix that implement hidden state Mixup, and test if this variant outperforms vanilla methods based on input Mixup.
Although the original paper proposed randomizing the layer le just before hidden states that undergo Mixup for each batch, we propose setting this layer & constant. This is to reduce communication cost significantly, since selecting randomized layer for Mixup will require other clients having to send multiple hidden states (which usually have large dimensions). further imposing communication burden.
Another change is that while original Manifold Mixup (Verma et al., 2018) suggests backpropagat- ing the entire computational graph through the whole network, including the encoder (part of model projecting input to designated hidden representation) and the decoder (rest of the model projecting hidden representation to output). Such thing is impossible to do in typical federated setting, since the computational graph calculate hidden representation of local data and the graph calculate hidden representation other clients' data separated, and they cannot be updated simultaneously through local update (doing so requires communicating encoder weights across clients every local update, which is highly inefficient in terms of communication). Thus, during Mixup between hid- den representations, only the decoder weights selected local client will desynchronize encoder weight values for calculating hidden states of local data from those for calculating hidden states of other clients' data every local update, so that the model does not learn properly.
be updated, since only updating encoder of the
Table 9: Test accuracy after (target rounds) and number of rounds to reach (target test accuracy) on various datasets. We compare FedMix with baseline Mixup algorithms.
Algorithm
FEMNIST
CIFAR10
CIFAR100
test acc, (200)
rounds (80%)
test acc. (500) rounds (70%)
test acc. (500) rounds (40%)
NaiveMix
85,9
23
77.4
198
53,8
85
Mixup w/ random noise
$6.1
23
77.9
201
51.2
105
Mixup w/ local means
85.5
21
73.5
233
51.0
87
FedMix
86.5
18
81.2
162
56.7
34
15

---

Published as a conference paper at ICLR 2021
Table 10:
Test accuracy after 500 rounds on CI-
FAR10, under varying local epochs (E).
of Local Epochs (B)
2
5
10
FedAvg
74.4
73.8
80.7
78.9
LocalMix
63.7
73.0
74.7
80.0
NaiveMix
72.0
77.4
81.0
82.6
FedMix
75.8
81.2
83.0
82.5
To compensate for this downside, we propose performing vanilla SGD updates without Mixup after Manifold Mixup SGD (which updates weights only in decoder). The vanilla updates will have both local encoder and decoder weights to be updated, thus driving the model to have better hidden representations for Mixup. However, this difference not only imposes additional computation cost, but also does not guarantee that it will show better performance compared to input Mixup methods.
While utilizing hidden representations from other clients sound like safe idea, it does not ensure data privacy, primarily because the updating client has knowledge of the exact encoder weight val- used to calculate hidden states received, and based on our modification, the received hidden states are treated as constants during Mixup. Model inversion attacks (Fredrikson et al., 2015) 5) have been suggested recover input images from hidden states or outputs, with access weight val- ues. Thus, direct Mixup between hidden states does not guarantee data privacy. Variant of FedMix and NaiveMix can be applied during decoder training phase, so that privacy is ensured while we successfully approximate Mixup.
The performance proposed algorithms shown in Figure 4 on CIFARIO (same settings with main experiment for dataset, model, and training is used; see Appendix B). Comparison between methods Figure 4(a) shows that while variants of FedMix and NaiveMix show improved performance over existing methods, they still do not outperform dotted line). Mcanwhile, comparison between using different layers for Mixup is shown in Figure 4(b). It is shown that & 4 has fastest learning curve but converges similarly case of l 2, both being slightly outperformed by case of input Mixup.
method based on input Mixup (compare with
Considering additional computation burden required to communicate hidden states (which often have larger dimensions than raw input) and necessity to communicate the hidden states every com- munication round (since hidden representations change with encoder weights), we propose that Fed- Mix using input Mixup is superior, and use this method for our main analyses.
EFFECT OF LOCAL EPOCHS
Previous works (McMahan et al., 2017; Caldas et al., 2019) show that number of local epochs, E, affects federated learning performance. We tested the effect of E CIFARIO. In general, showed that test performance increases as E increases. In addition, we observed that under various values of E, FedMix shows the best performance compared other algorithms (see Table 10), being a close second after NaiveMix for E = 10. MAFL-based algorithms outperform existing algorithms for all values of E tested.
G ADDITIONAL COMPUTATION COST INCURRED BY MAFL
For FedMix, additional computation and memory are required edge devices during model training for each communication round, since &eaasi requires additional terms, including gradient by input, Ba compared vanilla FedAvg. We claim that FedMix does not result in additional computation burden. Specifically, we trained FedMix on CIFAR10 with the same settings as the main experiment to 70% accuracy in 1.94 hours; FedAvg takes 1.95 hours. FedMix spends a comparable amount of time to reach similar level of performance of FedAvg. While in the memory aspect, FedMix requires about twice more GPU memory allocation compared to FedAvg, this phenomenon is also observed LocalMix and NaiveMix. The extra memory burden comes from Mixup by enlarging the input dimension twice. For instance, FedAvg requires 46.00MB to allocate, LocalMix requires 84.00MB and 98. 38MB for FedMix. Calculating gradient of the input derivative gives only negligi-
16

---

Published as a conference paper at ICLR 2021
ble 2-3MB additional memory usage, which is reasonable concerning the substantial performance increase from LocalMix to FedMix.
H INTRODUCTION OF CUT-OFF THRESHOLD IN MAFL
To better ensure privacy, a cut-off threshold that prevents clients with fewer data to send averaged data could be introduced. We performed this in FEMNIST, since for such procedure be effective, heterogeneous size of local client data is necessary. We test with N 300 clients, and introduce different threshold levels to test its efficiency. addition, we also test with multiple values, to see whether threshold level affects optimal value of À for FedMix.
We present the results in Table 11. While threshold does not hugely affect performance, we observe that a moderately small threshold level of 100 results in the best performance. We suggest that as the threshold level is heightened, there less overfitting to clients with small size local data, but also results in decrease in the number of averaged data received by each client. We indeed find an appropriate value of threshold that maximizes performance.
different number of data per client, the sensitivity of À could also be different compared to when all clients have the same number of data. Results in Table 13 show that there is little change in performance by change in X, especially compared Table 5. In addition, inspection of the performance of global model on individual test data of clients does not reveal any noticeable pattern by the size of local data (see Table 13).
In case where there
Table I::
Test accuracy on FedMix with introduc- tion of cut-off threshold, tested number of threshold level. Optimal value of x is also shown.
Table 12: Test accuracy on FEMNIST,
300 under various Mixup ratio X.
threshold
A
0.05
0.1
0.2
100
150
200
Test Acc. (%)
83.0
83.3
83.1
82.9
Test Acc(%)
82.8
82.8
83.0
Aoptimal
0.2
0.05
0.1
0.2
Table 13: Mean and standard variation of local test accuracy of FedMix on FEM- NIST, tested on clients with varying local data size, under varying X.
7 mcan
<
100 std
100 S
7 > 199
? >
199
X
mean
std
mean
std
0.05
85.8
15.1
77.6
14.8
85.4
9.3
0.1
81.1
17.4
76.4
14.1
86.2
9.5
0.2
83.9
16.2
77.6
14.8
85.8
10.0
MAFL IN CONJUNCTION WITH GAUSSIAN NOISE
With results in Figure 3, we expressed concern with small values of Mj causing privacy issues with only a small performance boost, if at all. A common practice of introducing additional privacy is adding Gaussian noise. This is a popular method associated with differential privacy (McMahan et al., 2018), but adding noise alone does not guarantee differential privacy, since the noise level should be explicitly linked to differential privacy levels, e and Addition of artificial pixel-wise noise will enhance privacy but will result in quality drop of averaged data. While privacy added by noise and privacy from averaging data cannot be directly compared, we can select a noise level which in conjunction with small Mg. visually provides data privacy similar to that of maximum Mg-
Results show that the introduction of Gaussian noise does result in a decline in performance (Table 0.3, random noise
although the decline is very small. Interestingly noise gets larger
17

---

Published as a conference paper at ICLR 2021
Table 14: Performance of FedMix with Gaussian noise. refers to standard deviation of Gaussian noise.
Table 15: Test accuracy on CIFARIO varying
m, number of entries in X go Y, to be further averaged.
0.05
0.075
0.15
m
4
10
Mr 81.4
80.1
81.1
78.7
81.5
Test (%
81.2
81.6
78.4
Ma 10
80.8
79.1
79.4
81.7
M2
80.5
79.7
80.7
81.0
provides effect as data augmentation and results in performance increase compared to 0. This experiment is in line with Appendix D. We conclude that introduction of noise in averaged data could provide us with reasonable alternative to FedMix with large M₁. While method does not align directly with differential privacy, we leave as future work how FedMix could be smoothly combined with DP-related methods and how its privacy could be quantified in terms of differential privacy.
J ADDITIONAL EXPERIMENTS: VARIATIONS OF FEDMIX
Averaging within X g Further averaging between entries of X, Y, practically provides an extension of the range of viable Mg such that it exceeds Tgg in the sense that each averaged data is from multiple clients' data. Such process would also result in fewer data included in X,, Y,. so we tested effect of this procedure on model performance. Table 15 shows that for m-fold extra averaging, we even observe increase in performance, but it quickly declines m gets too large. This method provides an improvement in privacy while even possibly resulting better performance.
Effect of same-class split for averaging We perform random split of local data for averaging, but an unbalanced split, such as only averaging data with the same class labels, could result in better performance. We compared between random split and same-class split while keeping Mr 0.5mg be equal for both methods. Same-class split resulted a significant decline in performance, and we conclude that there is no advantage of such split over random split that we are using for our main results.
Table 16: Test accuracy CIFARIO, with Table 17: Test accuracy of NaiveMix CI- class/client 2, under different split meth- FARI0, under varying Mixup ratio X. ods. M₁ = 0.5ng for both splits.
random split
class split
X
0.05
0.1
0.2
0.5
FedMix
81.2
78.8
NaiveMix
79.5
79.9
80.6
29.8
NaiveMix with varying Mixup ratio X We varied Mixup ratio for NaiveMix as well. Results in Table 17 shows that NaiveMix also has intermediate optimal value of X. The drop in performance for X = 0.5 is much more dramatic than for FedMix (see Table 5 for comparison with Global Mixup and FedMix). We think that NaiveMix loss also suffers as it gives more weight the averaged data, especially for large Mg-
Heterogeneity from skewed label distribution Recent papers (Yurochkin al., 2019; Wang al., 2020) suggested an alternative heterogeneous environment, which does not limit the num- ber of classes per client but skews label distribution in local data. We used Dirichlet distribution of 0.2, 0.5 as described by Yurochkin et al. (2019) and Wang et al. (2020). Results show that FedMix still outperforms all other algorithms. We think that such label skewing introduces less het- erogeneity compared to practice of limiting the number of classes per client, but nevertheless. FedMix still the most powerful method in terms of performance.
18

---

Published as a conference paper at ICLR 2021
Table 18: Test accuracy on CIFARIO under label-skewed heterogeneous envi- ronment. We used Dirichlet distribution for uneven label distribution.
FedAvg
GlobalMix
LocalMix
NaiveMix
FedMix
0.2
83.9
91.1 91.1
84.0 88.0
85.0 88.2
86.4
0.5
87.6
88.4
19


Reviews:
[{"review_id": "FgnVP1V4EMF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors aim to approach the non-iid data issue in FL by allowing for mean of the local client data to be transmitted in addition to the model parameters. I find this work very interesting and the paper well executed. \nFirst, the authors present the logic for MAFL, which encompasses the sending and receiving of other clients' averaged data, followed by FedMix, a method for augmenting the local data-set with the averaged data from other clients. \n\nThroughout the method section and their experiments, the authors show the benefits of MAFL+FedMix by ablation to other MixUp inspired approaches. \n\nMy issues with this paper are along some different aspects:\nPrivacy:\nSending statistics of local data is inherently less private than sending model parameters alone. The authors mention this explicitly, but do not go into more detail. I understand that the notion of privacy in FL is a research topic in itself, but I would wish for a more nuanced discussion of the trade-offs here. Throughout the experiment section, the largest 'federation' of devices is N=100 for Cifar100 and Femnist. Taking cifar100 as example, each client has 50k/100 = 500 data-points, the average of which I can agree intuitively to be not very informative (at least visually) and the 'discriminative information' that the authors mention, is presumably not very high. However, 500 data-points can still be considered a large amount of data-points for the federated scenario. As the number of data-points per client $n_k$ decreases, the more information about individual data-points is contained in their average. The problem is increased as $M_k>1$ . Further, 'discriminative information' is not the only privacy-worthy information in FL. Differential Privacy, for example, is trying to quantify if an individual data-point is present in a local data-set. Since a client receives a concatenation $(X_g,Y_g) = ({\\bar{x}_1,\\bar{x}_2,...,\\bar{x}_N},{\\bar{y}_1,\\bar{y}_2,...,\\bar{y}_N})$ of all clients' averaged data-sets, an individual client's participation in the training can also not be hidden from other clients. \nFurthermore, the formulation in Algorithm 1 implicitly assumes a continual learning setup where clients might be collecting more data as training progresses. In its current formulation, the authors do not mention if the batches are re-computed randomly, opening up the possibility for attacks on the differences between batches across time. \n\nComputational Burden:\nFedMix requires computing gradients through the Taylor expansion (EQ 4), which increases computation and memory requirements. Especially in a federated setting, computation and memory are constrained resources, so I would expect the authors to provide some estimates over the additional requirements for computing gradients $\\nabla_w l_{FedMix}$\n\nExperimental Evaluation:\nI am missing some details on the setup for the FEMNIST dataset. At the moments, the authors mention selecting 100 clients, however I wonder if they used the writer-id or re-shuffled to create a controlled label-skew. If they used the writer-id, how did they select the subset of 100 clients? \n\nSome details:\nI believe in Figure 1 b), the indices above 'Local data' should be $i$, not $j$. \nDirectly below Figure 1, the sentence should begin with: \"A more practical approach to...\"\nAlgorithm 2 could be improved, I believe. I see no space constraint that would prevent including some more detailed information analogous to Algorithm 3 in the Appendix. I am assuming the gradient is calculated mini-batch wise. Ideally, the $LocalUpdate$ would receive the same arguments as those that it is being called with on the server side for example. \nTop of page 5: 'meshed' -> 'mashed'.\nJust above Eq (2): '... client i has access to ...' (remove 'an').\n\n\nThe experiments that would make this evaluation great in my opinion:\nTrain on the full FEMNIST set of 3600clients including all those clients with very small number of data-points. Then introduce a cut-off-threshold for the minimum number of data-points that each client has to have in order to send its averaged data to the server. Alternatively, add random noise to these averages in relation to how much data is present. There is probably a differential privacy formulation that would make the required noise-level explicit. This noise level or cut-off-point should give more insight on several dimensions of the proposed work:\n\n- How sensitive is FedMix to different minimum required data-points as a trade-off with privacy.\n- How sensitive is $\\lambda$ to different number of data-points per client (or the consequence of fixed $\\lambda$ generally as number of data-points per client differs). Since no other experiment has different number of data-points per client, I believe this to be relevant.\n\nAdditionally, to further increase privacy, the authors might consider (randomly) averaging some of the elements in $(X_g,Y_g)$ before sending the data to clients and study those effects.\n\nSummarizing, I want to thank the authors for this very interesting read and interesting insights. If the authors provide a more nuanced/detailed discussion of the privacy aspects of their work and extend their experimental section with the more holistic FEMNIST experiment I described above, I will raise my score! I see no violation of the CoE in this work.\n\nFinally, I cannot believe that the authors let the opportunity slide to name their algorithm 'FedUp' ;) \n \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9qWbfatB0VE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n \nThe paper proposed MAFL, a novel approach to conduct Mixup under the federated learning setting whiling preserving data privacy. The proposed FedMix scheme is inspired by Taylor’s expansion of the global Mixup formulation. The effectiveness of MAFL is justified via empirical studies over a simulated federated learning environment, which indicates that Mixup achieves better test accuracies on various machine learning tasks.\n\n##########################################################################\n\nReasons for score: \n \nMy overall evaluation score on the current manuscript is borderline reject. The research direction on studying the effectiveness of data augmentation under the federated learning setting is promising. The formulation and motivation of the proposed MAFL scheme are sound. The main justification on FedMix is from the experimental study, which can be further improved e.g. the communication cost and privacy of FedMix can be more explicitly studied. If the proposed MAFL scheme can be supported by some theoretical analysis, the current manuscript can be much stronger. I will be happy to increase my overall evaluation score if my major concerns are addressed.\n \n##########################################################################\n\nPros: \n \n1. The paper studies the effectiveness and practicality of conducting data augmentation under the federated learning scenario, which is quite promising and can potentially gain impact.\n \n2. The proposed FedMix method is motivated via using Taylor expansion to approximate the global Mixup data augmentation objective, which makes sense in general.\n \n3. Extensive experimental results are provided under the image classification and the next-word prediction tasks under the simulated non-iid environment, which indicates that FedMix enjoys high effectiveness on improving the model test accuracy under the data heterogeneity.\n \n##########################################################################\n\nCons: \n \n1. The main concern on the proposed FedMix method is communication and computation efficiency. From the proposed Algorithm 1, for each FL round, MAFL requires all available clients to upload their locally averaged data batches. It is easy to imagine in a real federated learning environment (with up to $10^{10}$ available clients), it can lead to a significant communication overhead [1]. Thus, it would be useful to explicitly study the communication cost of MAFL, e.g. report Test Accuracy vs the amount of communication for Figure 2 can help to understand the communication efficiency of MAFL better. \n2. It’s not clear how MAFL splits the local datasets. Does it a conduct random split? Would it be possible for each local client to put “similar” (e.g. data points within the same class) into the same batch? Such an approach intuitively preserves more data property in $\\bar x$ and $\\bar y$. \n3. Although a value of small $M_k$ in MAFL leads to worse data privacy, it’s easy to imagine the proposed MAFL can be combined with other differential private (DP) method e.g. [2]. It will be useful to consider a DP version of MAFL. \n4. The authors are encouraged to add the baseline of “Global Mixup” to Table 2, 3, 4, 6, 7 to understand the gap between the proposed FedMix method and the “ideal” baseline. \n5. The FedProx result on FEMNIST is a bit confusing, what accuracy will FedProx reach for running 32 FL rounds? \n6. For the image classification tasks, it seems FedMix outperforms other baselines. However, for the language task e.g. Table 2, it only matches the accuracy of NaiveMix. Does it mean FedMix can be improved for augmenting language examples? \n7. The approach to simulate data heterogeneity in the current paper can be generalized by the method proposed in [3-4]. It would be useful to consider the Dirichlet distribution based data partition strategy. \n\n[1] https://arxiv.org/pdf/1912.04977.pdf\n\n[2] https://arxiv.org/pdf/1710.06963.pdf\n\n[3] https://arxiv.org/pdf/1905.12022.pdf\n\n[4] https://arxiv.org/pdf/2002.06440.pdf\n\n#########################################################################\n\nMinor Comments: \n1. It seems the CIFAR-10 and CIFAR-100 curves in Figure 2 are not reaching to full convergence. Thus, it would be helpful to run the experiments for more FL rounds. \n2. Missing references: [1-2]. And some references are sort of outdated e.g. “Federated optimization in heterogeneous networks” (T Li et al) was accepted to MLSys 2020. \n\n\n[1] https://arxiv.org/abs/1912.04977\n\n[2] https://arxiv.org/abs/1908.07873\n\n\n###################### Post Rebuttal #############################\nMost of my concerns on the current manuscript are addressed. I tend to increase my overall evaluation score to 6.\n#############################################################", "extra_scores": {}, "extra_text": {}}, {"review_id": "-OHzmTJoo-h", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies an interesting idea that applies Mixup to Federated Learning (FL) for addressing some challenges such as non-iid data. Basically, this is an empirical paper, and the overall organization is good, easy to read. However, I have several questions.\n\n1. The authors claimed that \"FedMix approximates Global Mixup\". In that case, why do not use Global Mixup directly? By comparing (4) with (3), both Global Mixup and FedMix use private image $(X_j, y_j)$, which can not violate privacy.\n\n2. The mathematics is poorly written. \n(a) What is $\\ell$ in $\\frac{\\partial \\ell}{\\partial x}$? It could be $\\ell(x)$, $\\ell(f((1+\\lambda)x_i),y_i)$, and $\\ell(f((1+\\lambda)x_i+\\lambda x_j),y_j)$ and so on. Please write it explicitly. \n(b) The last equation on Page 5 missed a $\\frac{1}{|J|}$ on the left hand side since $\\bar x$ or $\\bar y$ means averages x_j or y_j. \n(c) In proposition 1, it says \"we ignore the second order term (i.e., $O(\\lambda^2)$)\", but why there is a $\\lambda^2$ in (4)? Please check $\\lambda(1-\\lambda)=\\lambda - \\lambda^2$.\n(d) In Algorithm 1 and Algorithm 2, what is k in \"LocalUpdate$(k,w_t, X_g,Y_g)$\" since there is no $k$ in Algorithm 2? In Algorithm 2, it seems the input is $k,w_t, X_g,Y_g$ based on \"LocalUpdate$(k,w_t, X_g,Y_g)$\", but what is $X, Y$ in $\\ell_1$ and $\\ell_2$? In addition, what is $x$ in $\\ell_3$. I gauss $w$ in Algorithm 2 should be $w_t$.\n\n3. Based on Figure 1 and Algorithms 1 and 2, $\\ell_{FedMix}$ is an approximation of $\\ell_{NaiveMix}$ rather than $\\ell_{GlobalMixup}$, which can be easily verified by using Taylor expansion. In that case, NaiveMix and FedMix is very close when $\\lambda$ is very small. However, the experimental results shows that FedMix is more closer to GlobalMixup than NaiveMix, which is not reasonable. Any explanations? Did you use the approximation of GlobalMixup as FedMix? If so, I think the presented results are not so interesting.\n\n4. The experiments only conducted on three small data sets, but in FL, we are usually interested in big data. It would be better if the authors can provides results on big data such as ImageNet. \n\n5. In table 8, why NaiveMix and FedMix use different $\\lambda$ for CIFAR10? On the other hand, why don't you try different $\\lambda$ for NaiveMix?\n\n------------After Rebuttal------------\nThe authors have addressed my main concerns, and I have updated my score to 6.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to apply Mixup to Federated Learning (FL) for addressing the challenge of non-iid data. The idea is very simple, but seems to work well in empirical evaluation. Some concerns were raised regarding the communication costs and privacy. The authors rebuttal and revised draft provide reasonable answers to these concerns. 

For the final version, it is suggested that the authors can address the following issues:

1) Improve the writing - especially the formulation of the proposed method

2) Provide more discussions and experiments on the communication costs. 
Decision: Accept (Poster)