Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper develops a robust federated learning algorithm to address feature shift in clients’ samples, which can be caused by various factors, e.g., acquisition differences in medical imaging. The paper proposes FEDFA to tackle federated learning from a distinct perspective of federated feature augmentation. The paper models each feature statistic probabilistically via a Gaussian distribution, with the mean corresponding to the original statistic and the variance quantifying the augmentation scope, from which novel feature statistics can be drawn to fulfill augmentation.
strength_and_weaknesses: Strength:
(1) The paper propose a federated learning algorithm from a novel perspective of feature augmentation.
(2) The proposed approach is conceptually simple but seems to be effective.


Weakness:
(1) The most related baseline seems to be FEDMIX. However, the evaluated datasets and the adopted model architecture in this paper are different from the datasets in the original paper of FEDMIX. There is limited discussion of the rationale of how the datasets are selected and why they are different from the original paper of FEDMIX.
(2) A key assumption of this paper is that the each feature statistic follows a multi-variate Gaussian distribution. I am not sure whether the assumption is valid, for the neural embeddings in general. The paper may discuss more details of how/when this assumption is satisfied in tasks relying on deep neural networks.
(3) Previous works (e.g., FEDMIX) provide theoretical discussions. This paper may be further improved by providing theoretical discussions.
clarity,_quality,_novelty_and_reproducibility: The paper is well motivated, and the proposed solution is conceptually simple and has good intuitions. The evaluations may be further improved by using the same datasets and model architecture in the original paper of FEDMIX.

The paper is clearly written, and it is very easy to read the paper.

The paper proposes a federated learning algorithm from a novel perspective of feature augmentation. Conceptually, it can bring additional advantages, compared to the SOTA work in this area (FEDMIX). However, the evaluations may be further improved by using the same datasets and model architecture in the original paper of FEDMIX.
summary_of_the_review: The paper proposes a federated learning algorithm from a novel perspective of feature augmentation. Conceptually, it can bring additional advantages, compared to the SOTA work in this area (FEDMIX). However, the evaluations may be further improved by using the same datasets and model architecture in the original paper of FEDMIX. A key assumption of this paper is that the each feature statistic follows a multi-variate Gaussian distribution. The paper may discuss more details of how/when this assumption is satisfied in tasks relying on deep neural networks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes federated feature augmentation to solve the data heterogeneity in federated learning. Each local model models its feature with Gaussian distribution, where the variance is the key to fulfill augmentation.
The experiments on image classification and medical image segmentation show effectiveness of the proposed work.  
strength_and_weaknesses: Strength:
1. The proposed method targets at the non-i.i.d. problem of FL, and the idea of feature level augmentation is interesting. 
2. The FedFA method is simple but effective. Empirical comparisons with classical FL methods demonstrate the effectiveness of the proposed feature augmentation method.

Weakness:
1. FedRobust and FedBN also targets at feature shift problem in FL, it is not clear why "these algorithms may still suffer significant local dataset bias". Could the authors provide in-depth theoretical analysis how the proposed method is different/superior compared with these two counterparts? Besides, the experiments should show comparisons with FedRobust and FedBN.
2. I wonder if the method utilizes the statistics of features from all layers or specific layers. The paper lacks discussions and ablations regarding this. 
3. The experiments only focus on splitting local data with different data origin. I would expect results with other kinds of local data heterogeneity, such as large variances among local data size or different modalities among locals. 
clarity,_quality,_novelty_and_reproducibility: The writing is clear. The experiments design is generally good. 
The proposed method is simple yet novel.
summary_of_the_review: The paper proposes a novel feature augmentation method for non-i.i.d. federated leaning. 
My main concern is the method only shows effectiveness with local data split by data origin. There should be more experiments for other kinds of local data heterogeneity (e.g. variant local data size). A typical FL setting is to split data with Dirichlet distribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tries to address feature shift by modelling feature statistics among clients.  Specifically, the authors model feature statistic via a Gaussian distribution.  The mean of the Gaussian distribution denote the original statistic while the variance denote the augmentation scope. The authors designed a solution to determine an appropriate variance for the Gaussian distribution. In the solution, the Gaussian variance is estimated by not only the local data but also the feature statistics from all participating clients.  In the variance estimation, the authors designed an adaptive variance fusion strategy to assign different weights to different channels in the feature statistic space.
After establishing the Gaussian distribution, the novel features can be synthesised by sampling from the Gaussian distribution.
strength_and_weaknesses: Strength:
-- modeling the feature statistic via a Gaussian distribution and generating new features by sampling from the Gaussian distribution seems interesting.
-- the client sharing statistic variances and adaptive variance fusion strategy are technically solid.

Weakness:
What concerns me is the writing of this work. I feel it is not clear enough and I feel a little hard to fully understand the technical details.
-- I feel difficult to fully understand the technical details. For example, in Eq 6, there is j  while in Eq 7 there isn't. How to use the result calculated in Eq 6 for Eq 7? 
-- How to take advantage of the shared feature statistics are not clear to me. Specifically,  the relation between Eq. 5/6/7 is not clear to me.
-- the formulation terms in this draft need refined. For example, in page 4, \mu_m^k \sim \mathcal{N} (\mu_m^k, ##).  
clarity,_quality,_novelty_and_reproducibility: This work is of fair quality. I feel the writing is not clear enough and its technical details need to be clarified. The originality of the work is good.
summary_of_the_review: This work propose to model feature statistic via a Gaussian distribution. The parameters of the Gaussian distribution is estimated from the data from active clients. For the parameter estimation, the authors designed client sharing statistic variances and adaptive variance fusion strategies.  Once the parameters are estimated, new features are generated by sampling from the Gaussian distribution. Although the idea seems interesting, the writing of this work needs significant improvement. I feel hard to fully understand the technical details.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Well known problem by now is the non-iidness of the client data causes degradations in FL trained models. The paper proposes FedFA, federated feature augmentation to address feature shift in clients data. FedFA exploits instance based statistics to augment local features to curb the introduction of novel domains because of augmentations. The proposed approach sounds promising to advance FL.
strength_and_weaknesses: # Pros:

- The paper is well-written, easy to understand and follow along.

- The idea of treating multiple channels as multiple variates and measuring the statistical variances is really interesting since they will always remain normally distributed as they are measured over mean and variances already.

- A very simple technique with no or negligible communication overheads.

- Literature is covered adequately.

# Cons:

- The FL based hyper params are missing in the paper, such as the number of rounds, the number of clients participating in training, client/device selection if any, etc.

- The scale of the benchmarks is small compared to CIFAR-10 or EMNIST, can you show the method works on one/both of these relatively large dataset.

- Also, the number of samples per client seems to be uniform and the proposed statistical measures seem to work well for something like that. Can you experiment with an unequal number of samples per client? 

- The main advantage that I see with FEDFA are the communication overheads compared to mixup kind of augmentation approaches (of course privacy is there, this on top), why not asymptotically quantify the costs?The cost analysis that is currently presented is talking about some raw numbers, they will go away if it is \BigO notation and the costs might be almost same as FedAvg.

- It looks like there is an ablation on \alpha and \p done separately, but what is the best combination, can we call it (\alpha, p) = (1, 0.5) or (1.,0, 1.0) or (0.5,0.5)?


### Minor comments: 

- “Since data coming different users … ” should be corrected

clarity,_quality,_novelty_and_reproducibility: The paper clearly explains the concepts and the presentation is easy to follow

The work has quality but it can be further improved by increasing the scale of the benchmarks.

The proposed approach is simple and novel in the realm of federated optimization algorithms which definitely contributes to the body of knowledge in the domain/literature.

The method is easy to reproduce and the results can also be tested, having access to the open sourced code can help further.
summary_of_the_review: Please refer to the strengths and weaknesses
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an augmentation technique that can create robust features to the shift of client samples in a non-iid FL environment. It is a simple method that models the statistics of local features with a Gaussian distribution and augments by considering the statistics of neighboring clients in a FL environment.

Three reviewers agreed that the submission deals with important issues in the real FL environment and the proposed method is simple yet effective, and suggested accepting this paper. One reviewer originally suggested rejecting, but the authors sufficiently addressed the concerns raised by the reviewer through the rebuttal period, and this reviewer also is convinced. 

I hope that the final version will solve all the issues raised by the reviewers, such as writing issues. In addition, the results of comparison with fedmix in datasets such as CIFAR, which were performed during the discussion period, should be further supplemented and included in the final version.
justification_for_why_not_higher_score: It's a simple idea, and it's well-deserved acceptance by showing good experimental results, but it didn't show any more surprising results than that.
justification_for_why_not_lower_score: simple yet effective methods
Decision: Accept: poster