Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies an interesting idea that applies Mixup to Federated Learning (FL) for addressing some challenges such as non-iid data. Basically, this is an empirical paper, and the overall organization is good, easy to read. However, I have several questions.

1. The authors claimed that "FedMix approximates Global Mixup". In that case, why do not use Global Mixup directly? By comparing (4) with (3), both Global Mixup and FedMix use private image $(X_j, y_j)$, which can not violate privacy.

2. The mathematics is poorly written. 
(a) What is $\ell$ in $\frac{\partial \ell}{\partial x}$? It could be $\ell(x)$, $\ell(f((1+\lambda)x_i),y_i)$, and $\ell(f((1+\lambda)x_i+\lambda x_j),y_j)$ and so on. Please write it explicitly. 
(b) The last equation on Page 5 missed a $\frac{1}{|J|}$ on the left hand side since $\bar x$ or $\bar y$ means averages x_j or y_j. 
(c) In proposition 1, it says "we ignore the second order term (i.e., $O(\lambda^2)$)", but why there is a $\lambda^2$ in (4)? Please check $\lambda(1-\lambda)=\lambda - \lambda^2$.
(d) In Algorithm 1 and Algorithm 2, what is k in "LocalUpdate$(k,w_t, X_g,Y_g)$" since there is no $k$ in Algorithm 2? In Algorithm 2, it seems the input is $k,w_t, X_g,Y_g$ based on "LocalUpdate$(k,w_t, X_g,Y_g)$", but what is $X, Y$ in $\ell_1$ and $\ell_2$? In addition, what is $x$ in $\ell_3$. I gauss $w$ in Algorithm 2 should be $w_t$.

3. Based on Figure 1 and Algorithms 1 and 2, $\ell_{FedMix}$ is an approximation of $\ell_{NaiveMix}$ rather than $\ell_{GlobalMixup}$, which can be easily verified by using Taylor expansion. In that case, NaiveMix and FedMix is very close when $\lambda$ is very small. However, the experimental results shows that FedMix is more closer to GlobalMixup than NaiveMix, which is not reasonable. Any explanations? Did you use the approximation of GlobalMixup as FedMix? If so, I think the presented results are not so interesting.

4. The experiments only conducted on three small data sets, but in FL, we are usually interested in big data. It would be better if the authors can provides results on big data such as ImageNet. 

5. In table 8, why NaiveMix and FedMix use different $\lambda$ for CIFAR10? On the other hand, why don't you try different $\lambda$ for NaiveMix?

------------After Rebuttal------------
The authors have addressed my main concerns, and I have updated my score to 6.
review_title: An interesting idea but with some unclear statements and experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:
 
The paper proposed MAFL, a novel approach to conduct Mixup under the federated learning setting whiling preserving data privacy. The proposed FedMix scheme is inspired by Taylor’s expansion of the global Mixup formulation. The effectiveness of MAFL is justified via empirical studies over a simulated federated learning environment, which indicates that Mixup achieves better test accuracies on various machine learning tasks.

##########################################################################

Reasons for score: 
 
My overall evaluation score on the current manuscript is borderline reject. The research direction on studying the effectiveness of data augmentation under the federated learning setting is promising. The formulation and motivation of the proposed MAFL scheme are sound. The main justification on FedMix is from the experimental study, which can be further improved e.g. the communication cost and privacy of FedMix can be more explicitly studied. If the proposed MAFL scheme can be supported by some theoretical analysis, the current manuscript can be much stronger. I will be happy to increase my overall evaluation score if my major concerns are addressed.
 
##########################################################################

Pros: 
 
1. The paper studies the effectiveness and practicality of conducting data augmentation under the federated learning scenario, which is quite promising and can potentially gain impact.
 
2. The proposed FedMix method is motivated via using Taylor expansion to approximate the global Mixup data augmentation objective, which makes sense in general.
 
3. Extensive experimental results are provided under the image classification and the next-word prediction tasks under the simulated non-iid environment, which indicates that FedMix enjoys high effectiveness on improving the model test accuracy under the data heterogeneity.
 
##########################################################################

Cons: 
 
1. The main concern on the proposed FedMix method is communication and computation efficiency. From the proposed Algorithm 1, for each FL round, MAFL requires all available clients to upload their locally averaged data batches. It is easy to imagine in a real federated learning environment (with up to $10^{10}$ available clients), it can lead to a significant communication overhead [1]. Thus, it would be useful to explicitly study the communication cost of MAFL, e.g. report Test Accuracy vs the amount of communication for Figure 2 can help to understand the communication efficiency of MAFL better. 
2. It’s not clear how MAFL splits the local datasets. Does it a conduct random split? Would it be possible for each local client to put “similar” (e.g. data points within the same class) into the same batch? Such an approach intuitively preserves more data property in $\bar x$ and $\bar y$. 
3. Although a value of small $M_k$ in MAFL leads to worse data privacy, it’s easy to imagine the proposed MAFL can be combined with other differential private (DP) method e.g. [2]. It will be useful to consider a DP version of MAFL. 
4. The authors are encouraged to add the baseline of “Global Mixup” to Table 2, 3, 4, 6, 7 to understand the gap between the proposed FedMix method and the “ideal” baseline. 
5. The FedProx result on FEMNIST is a bit confusing, what accuracy will FedProx reach for running 32 FL rounds? 
6. For the image classification tasks, it seems FedMix outperforms other baselines. However, for the language task e.g. Table 2, it only matches the accuracy of NaiveMix. Does it mean FedMix can be improved for augmenting language examples? 
7. The approach to simulate data heterogeneity in the current paper can be generalized by the method proposed in [3-4]. It would be useful to consider the Dirichlet distribution based data partition strategy. 

[1] https://arxiv.org/pdf/1912.04977.pdf

[2] https://arxiv.org/pdf/1710.06963.pdf

[3] https://arxiv.org/pdf/1905.12022.pdf

[4] https://arxiv.org/pdf/2002.06440.pdf

#########################################################################

Minor Comments: 
1. It seems the CIFAR-10 and CIFAR-100 curves in Figure 2 are not reaching to full convergence. Thus, it would be helpful to run the experiments for more FL rounds. 
2. Missing references: [1-2]. And some references are sort of outdated e.g. “Federated optimization in heterogeneous networks” (T Li et al) was accepted to MLSys 2020. 


[1] https://arxiv.org/abs/1912.04977

[2] https://arxiv.org/abs/1908.07873


###################### Post Rebuttal #############################
Most of my concerns on the current manuscript are addressed. I tend to increase my overall evaluation score to 6.
#############################################################
review_title: A novel method, but the experimental justification can be improved

final_score: rating: 7: Good paper, accept
review_content: In this work, the authors aim to approach the non-iid data issue in FL by allowing for mean of the local client data to be transmitted in addition to the model parameters. I find this work very interesting and the paper well executed. 
First, the authors present the logic for MAFL, which encompasses the sending and receiving of other clients' averaged data, followed by FedMix, a method for augmenting the local data-set with the averaged data from other clients. 

Throughout the method section and their experiments, the authors show the benefits of MAFL+FedMix by ablation to other MixUp inspired approaches. 

My issues with this paper are along some different aspects:
Privacy:
Sending statistics of local data is inherently less private than sending model parameters alone. The authors mention this explicitly, but do not go into more detail. I understand that the notion of privacy in FL is a research topic in itself, but I would wish for a more nuanced discussion of the trade-offs here. Throughout the experiment section, the largest 'federation' of devices is N=100 for Cifar100 and Femnist. Taking cifar100 as example, each client has 50k/100 = 500 data-points, the average of which I can agree intuitively to be not very informative (at least visually) and the 'discriminative information' that the authors mention, is presumably not very high. However, 500 data-points can still be considered a large amount of data-points for the federated scenario. As the number of data-points per client $n_k$ decreases, the more information about individual data-points is contained in their average. The problem is increased as $M_k>1$ . Further, 'discriminative information' is not the only privacy-worthy information in FL. Differential Privacy, for example, is trying to quantify if an individual data-point is present in a local data-set. Since a client receives a concatenation $(X_g,Y_g) = ({\bar{x}_1,\bar{x}_2,...,\bar{x}_N},{\bar{y}_1,\bar{y}_2,...,\bar{y}_N})$ of all clients' averaged data-sets, an individual client's participation in the training can also not be hidden from other clients. 
Furthermore, the formulation in Algorithm 1 implicitly assumes a continual learning setup where clients might be collecting more data as training progresses. In its current formulation, the authors do not mention if the batches are re-computed randomly, opening up the possibility for attacks on the differences between batches across time. 

Computational Burden:
FedMix requires computing gradients through the Taylor expansion (EQ 4), which increases computation and memory requirements. Especially in a federated setting, computation and memory are constrained resources, so I would expect the authors to provide some estimates over the additional requirements for computing gradients $\nabla_w l_{FedMix}$

Experimental Evaluation:
I am missing some details on the setup for the FEMNIST dataset. At the moments, the authors mention selecting 100 clients, however I wonder if they used the writer-id or re-shuffled to create a controlled label-skew. If they used the writer-id, how did they select the subset of 100 clients? 

Some details:
I believe in Figure 1 b), the indices above 'Local data' should be $i$, not $j$. 
Directly below Figure 1, the sentence should begin with: "A more practical approach to..."
Algorithm 2 could be improved, I believe. I see no space constraint that would prevent including some more detailed information analogous to Algorithm 3 in the Appendix. I am assuming the gradient is calculated mini-batch wise. Ideally, the $LocalUpdate$ would receive the same arguments as those that it is being called with on the server side for example. 
Top of page 5: 'meshed' -> 'mashed'.
Just above Eq (2): '... client i has access to ...' (remove 'an').


The experiments that would make this evaluation great in my opinion:
Train on the full FEMNIST set of 3600clients including all those clients with very small number of data-points. Then introduce a cut-off-threshold for the minimum number of data-points that each client has to have in order to send its averaged data to the server. Alternatively, add random noise to these averages in relation to how much data is present. There is probably a differential privacy formulation that would make the required noise-level explicit. This noise level or cut-off-point should give more insight on several dimensions of the proposed work:

- How sensitive is FedMix to different minimum required data-points as a trade-off with privacy.
- How sensitive is $\lambda$ to different number of data-points per client (or the consequence of fixed $\lambda$ generally as number of data-points per client differs). Since no other experiment has different number of data-points per client, I believe this to be relevant.

Additionally, to further increase privacy, the authors might consider (randomly) averaging some of the elements in $(X_g,Y_g)$ before sending the data to clients and study those effects.

Summarizing, I want to thank the authors for this very interesting read and interesting insights. If the authors provide a more nuanced/detailed discussion of the privacy aspects of their work and extend their experimental section with the more holistic FEMNIST experiment I described above, I will raise my score! I see no violation of the CoE in this work.

Finally, I cannot believe that the authors let the opportunity slide to name their algorithm 'FedUp' ;) 
 


review_title: Interesting work, good ideas and evaluation. Some aspects need to be improved.

Meta-review: The paper proposes to apply Mixup to Federated Learning (FL) for addressing the challenge of non-iid data. The idea is very simple, but seems to work well in empirical evaluation. Some concerns were raised regarding the communication costs and privacy. The authors rebuttal and revised draft provide reasonable answers to these concerns. 

For the final version, it is suggested that the authors can address the following issues:

1) Improve the writing - especially the formulation of the proposed method

2) Provide more discussions and experiments on the communication costs. 
Decision: Accept (Poster)