Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Federated learning suffers from non-IIDness across clients. A line of previous works address the non-IID problem by data sharing which violates the privacy requirement. In this paper, the author propose SDA-FL. Compare to the most basic FedAvg, the additional components includes

- *Image synthesis with DP-GAN*. Before training classifier, each client trains a differentially private GAN to generate synthetic data, and upload these data to PS for future data-sharing. 
- *Synthetic image labeling*. Similar to self-training, PS use local models to assign pseudo-labels to unlabeled data. These labels are updated during training, i.e., interplay between model training and synthetic dataset updating. 
- *Mixup*. For each clients, private data and shared synthetic data are mixed to alleviate non-IIDness. 
- *ServerUpdate*. Data sharing also makes it possible for the server to conduct gradient descent. 

This paper also empirically evaluate their framework under supervised and semi-supervised learning settings. Moreover, this paper study the sensitivity to privacy budget and effect of synthetic data. 
main_review: Strengths: 

1. This paper addresses important non-IID and privacy problems in federated learning. It is a straightforward idea to share synthetic data instead of real data to avoid privacy leak. DP-GAN also has theoretically guarantee of privacy preserving. The framework seems to be promising. 
2. The performance of proposed framework is good. 
3. The effect of GAN-synthetic data is evaluated by a comparison with closely related algorithms. There is an improvement comapred to NaiveMix and FedMix. 
4. The sensitivity to privacy budget is evaluated. 
5. Each additional component seems not so novel, but it is a reasonable and potentially useful combination. 
6. Clear delivering of a complicated system. 

Weaknesses: 

1. Equation (4) seems to be wrong. If the algorithm use vanilla mixup, the input of classifier $$f(\cdot; \boldsymbol{w}_i)$$ should be the mixed data $$\bar{\boldsymbol{X}}_i$$, rather than $$(1 - \lambda_1) \boldsymbol{X}_i$$ or $$\lambda_1 \hat{\boldsymbol{X}}_i$$.  $$\lambda_1$$ should be $$1 - \lambda_1$$ to align with equation (3), and vise versa. Same mistake in Algorithm 2. The author should correct them or explain. 

2. Inadequate evaluation of labelling process. It is not verified whether the interplaying labeling process is better than some naive method, e.g., 

   - Each client trains a local classifier to assign pseudo label to its synthesis data, once and for all. 
   - Each client uses conditional GAN or similar techniques to generate images directly given label. 

   The author should carefully design ablation study to evaluate this. 

3. Potential advantages introduced by ServerUpdate. An additional ServerUpdate step is introduced to the framework. Therefore, SDA-FL may take more gradient descent steps than baselines (FedAvg, FedProx, FedMix, etc.) in a communication round. People may be skeptical whether it is the additional gradient steps make the model converge faster. The author should make sure that same number of gradient steps in taken in each round. 

4. Lack of details and unclear hyperparameters in experiments. Number of clients, number of class per client for Cifar-10, $$\alpha, \lambda_2, E, B$$, hyperparameters for baselines... are not provided. More experiments details should be provided in main text, supplementary materals, or released code. 

5. Error bar is not provided. People may ask whether figure results are from only one run. It will be much better to provided result with standard deviation or confidence interval, together with the number of independent runs. 

Other concerns and questions: 

1. In SDA-FL, each participant train a GAN before training the classifier, introducing the following requirements: 

   - Each client must have sufficient data to train GAN. 
   - Each client must have adequate computing resources. (GAN often requires more resources. )
   - In each communication round, clients are selected from *ones that uploaded their GANs*. This may implicitly requires a sustainable or high participation rate of client. 

   These requirements may limit the scope of SDA-FL's application. In cross-device settings, it's hard for personal devices (like mobile phones) to meet the above requirements. For me it seems like SDA-FL can only be applied to cross-silo settings. Please comment. 

2. DP-GAN guarantee privacy preserving during data synthesis. The training process of classifier is not protected and might leak personal information. This is an orthogonal problem but the author may mention this in the text. 
summary_of_the_review: This paper addresses important non-IID and privacy problems in federated learning, and proposes a frameworks with some components are evaluated. However, components of interplaying labeling and ServerUpdate are not well-evaluated in this paper. Moreover, the details of experiments are inadequate.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new framework for federated learning to resolve the non-IID issue by sharing differentially private synthetic data. Each client pretrains a local GAN to generate synthetic data and upload the data to the parameter server. To effectively use the synthetic data, the server performs pseudo labeling and shares this information with the clients along with the global model parameters, to let the local data, including both the real and synthetic data, approach an IID distribution. The interplay between model training and synthetic data updating improves the convergence of the local models and resolves the non-IID issue. 
main_review: Strength-

* The proposed idea is clear and well-presented: The idea to use partially-private synthetic data to resolve the non-IID issue in federated learning is interesting and clearly presented. 
*Experimental result shows the improvement in performance: The authors compare their method with other federated learning methods and demonstrate the improvement in performance from the sharing of synthetic data. The trade-offs between privacy level and the quality of synthetic data are also presented. 

Weakness-

* The overhead of training local GANs in each client can be huge: The proposed scheme achieves the performance gain from sharing of synthetic data, which models the private local data with some privacy protection. The performance gain highly depends on the quality of synthetic data, but in real scenarios, it could be unrealistic to train GANs in every client using their local data to generate high-quality synthetic data.
* More empirical support would be necessary: the proposed scheme highly depends on the quality of pseudo-labels as well as the synthetic data. There are many performance measures to quantify the quality of synthetic data and pseudo labels, i.e., FID and Inception Score for synthetic data and label accuracy for the pseudo labels, which can be measured by training a classifier with the synthetic data. Quantifying such quality and examining trade-offs between privacy level, computational cost, quality of synthetic data and the overall performance will provide better insights on this work and make the presentation of empirical results more comprehensive.  

summary_of_the_review: The proposed idea of using synthetic data, generated by local GANs of each client, and pseudo labelling at PS seems to be effective to resolve the non-IID issue of federated learning. However, the overhead of training local GANs in each client can be huge, which makes this approach not very practical. The overhead should be clearly addressed to compare this work with other baselines. The current experimental results only measure the overall test accuracy, but since the trade-offs between quality of synthetic data, pseudo-labelling, and overall performance are main factors in the proposed scheme, presenting such trade-offs in more detailed manner would be helpful to support the idea.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper focused on a classical federated learning setting where data was non-iid partitioned in different local servers, and came up with a method named SDA-FL, which combined GAN-generated data, differential privacy to both address the non-iid problem and keep local data privacy. The main contributions are:
1. Utilized differential private GAN-generated data to solve the non-iid and local data privacy problems;
2. Designed a label updating mechanism to increase the model performance;
3. Tested the algorithm on CIFAR-10, MNIST, fashion-MNIST to confirm the performance of the algorithm.
main_review: The main strengths are:
1. Utilized the generative power of GAN to simulate the distribution of local data, without directly sharing it with peer and central server. Also used differential privacy, the two methods both maintained the local data privacy;
2. Utilized pseudo labeling to further maintain the generated dataset.

The main weaknesses are:
1. The privacy issue itself. Consider the extreme case, where each local client only keeps a single class of data. In this case, the first round locally pre-trained GAN-generated data uploaded to the central server will mostly contain the data which has the same label, which immediately break the privacy of the local data;
2. Lack of citations. The differential private GAN is not an original idea. This idea has been broadly explored in the GAN community recently, thus proper citations are needed. Also, more citations of federated learning algorithms focusing on non-iid setting are also missing, for example, [1][2][3];
3. The algorithms are not described clearly in the algorithm boxes. ServerUpdate is described as ''is the same as the
LocalUpdate execpt the data'', which needs more clarification;
4. Lack of details of experiments and code. Most of the details of the experiments, for example, training hyperparameters, network details are missing. Since no code is provided, it's kind of hard for one to repeat the experiments and confirm the effectiveness of the algorithm;
5. Lack of experiments. More experiments should be conducted, for example for the first task "Federated supervised learning,"  different ways of partitioning the data should be considered, rather than the extreme case alone;
6. Lack of theoretical analysis;
7. Quality of GAN-generated images. This can be a minor issue, but the quality of the GAN-generated images provided in the paper is kind of low, not sure whether it's because of the noisy gradients;
8. Another minor issue is why different choices of differential budget are only provided in the CIFAR-10 task? 


[1] Tackling the objective inconsistency problem in heterogeneous federated optimization. J Wang, Q Liu, H Liang, G Joshi, HV Poor - arXiv preprint arXiv:2007.07481, 2020

[2] H. Wang, Z. Kaplan, D. Niu and B. Li, "Optimizing Federated Learning on Non-IID Data with Reinforcement Learning," IEEE INFOCOM 2020 - IEEE Conference on Computer Communications, 2020, pp. 1698-1707, doi: 10.1109/INFOCOM41043.2020.9155494. 

[3] On the convergence of fedavg on non-iid data. X Li, K Huang, W Yang, S Wang, Z Zhang - arXiv preprint arXiv:1907.02189, 2019
summary_of_the_review: This paper focused on a challenging problem -- non-iid in federated learning and tried to combine methods from different areas to solve it, which is interesting itself, however, this paper mainly suffers from the following aspects:
1. Technical correctness. The privacy is not well maintained;
1. Lack of novelty. The proposed algorithm is more like a rough combination of several existing methods, with little innovative thoughts or theoretical explanations;
2. Design of experiments. More experiments should be carefully designed and conducted;
3. Missing proper citations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new federated learning algorithm called SDA-FL, which utilizes GANs to generate synthetic data for federated training on non-IID data. Specifically, each client pretrains a GAN to generate synthetic data and send them to the server. In each round, the server sends the global model and the synthetic data to the clients. Then, the clients update the label of the synthetic data and use both the local data and synthetic data to update the local model. The local models are sent to the server, which further averages the models and uses the averaged model to label the synthetic data. Experiments show that SDA-FL significantly outperforms the other federated learning approaches on non-IID data. The paper also studies the influence of different privacy budgets on the performance when applying differentially private GANs.
main_review: Strengths:
1. The studied problem is important. Non-IID data is a key challenge in federated learning.
2. From the experiments, the improvement of the proposed approach is significant.

Weaknesses:
1. Some important details about the algorithm are missing. What is the size of the local synthetic data per client? Is it related to the size of the local data? 
2. The GAN seems to be an important part of the algorithm. The paper directly adopts WGAN without convincing explanation. The authors can add experiments to show the effect of the GAN architectures.
3. The communication overhead may be significantly large than FedAvg since the synthetic data need to be communicated each round. The authors should theoretically or empirically analyze the communication cost.
4. In Algorithm 1, why the client needs to update the synthesis dataset? The dataset has already been updated by the server.
5. The authors claim that the local data including both the real and the synthetic data approach an IID distribution. However, there is no convincing explanation to support it. 
6. The experiments are weak. MNIST and Fashion-MNIST are two simple tasks, which are not challenging in federated learning. For the results on CIFAR-10, the differentially private version does not show superior performance compared with FedAvg.
7. There is no experimental study on the hyperparameters including $\lambda$ and the threshold. The authors should add them to understand the proposed algorithm more clearly.
8. What is the non-IID data setting in the experiments of different privacy budget?
9. Typo: the second paragraph of “Effect of privacy budget”: Table 5 -> Table 6.

summary_of_the_review: The paper proposes a new federated learning algorithm on non-IID data. Although the proposed algorithm is interesting, many details in the algorithm and experiments are missing. I believe the paper needs to be significantly improved from both the theoretical and empirical perspectives.
review_title: null

Meta-review: This paper proposes an idea to address the non-IID issue that is well-known in federated learning. After the discussion with the authors, there are still some concerns remained about the proposed approach. First and foremost, the training of local GAN at each client can be demanding computationally and statistically, which limits the practicality of their approach. Secondly, there has been other work that aims to study the non-IID issues in federated learning, as suggested by the reviewers. The authors should consider citing some of the work in this literature and compare the prior approaches with their GAN-based approach. Thirdly, there is a lack of a formal statement about the privacy guarantee in this paper. In particular, it seems that the privacy guarantee would only make sense in the cross-silo setting, in which each client has many users' data. If each client corresponds to a single user, it does not make sense to train a local GAN. The authors should consider elaborating on the privacy guarantee in the next revision.
Decision: Reject