Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper solves the bottleneck of slow and unstable convergence issues in heterogeneous federated learning especially when the client participation ratio is low. The paper proposes a novel framework that can improve consistency to alleviate the heterogeneous issue and then accelerate the convergence of federated learning.
strength_and_weaknesses: Strengths:
1. The proposed method seems effective. The introduced momentum term of the global client makes the local model close to the global one. This method does not involve additional communication costs thus it is communication efficient.
2. The paper shows a complete experimental setup covering several cases that effectively supports the proposed FedACG.
 
Weaknesses:
1. From the Algorithm and Theorem 1, it seems the client computes full gradients. Hence the convergence analysis lacks consideration under stochastic gradient variance.
2. There lacks of nonconvex convergence analysis which is more practical in the real world. 
 

clarity,_quality,_novelty_and_reproducibility: The paper proposes novel federated learning framework, and the supplementary contains demo code and the detailed hyperparameter setting, thus is good at reproducibility.
 
Several comments about the clarity and quality:
1. In Table 2, in the list of the accuracy of CIFAR-10, MOON has a 64.55% accuracy while FedACG has 63.70%, thus the best accuracy is obtained by MOON.
2. In what form of accelerated client gradient is in Table 4? It is not very clear and it is better to have more detailed illustrations
3. Some typos: In the last paragraph of the related work, “FedCAM”—>”FedCAMS”.
After Equation 4, there may be one redundant letter “s” after $\{l_i(\cdot)\}_{I=1}^N$.

summary_of_the_review: 1 . The theoretical result seems a bit weak. It only contains a convex convergence result in the full batch setting which is not practical.  Also it is not clear whether the obtained convergence rate is better than other methods.
2. While the main story is about improving FL when the participation ratio is low, in the theoretical result, I didn’t find the term corresponding to the ratio. Did the author only consider the full participation setting in the theorem?  
3. Why do we have the constraint that $0.5 < \lambda < 1$? Any intuitions?
 
This paper provides novel methods for improving slow and unstable convergence in federated learning, and solid experimental results back up the method. The convergence analysis can be further improved with detailed discussions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a communication-efficient federated optimization algorithm that deals with client heterogeneity under large-scale and low-participation FL settings. The proposed algorithm, FedACG, leverages global momentum and local regularization. This paper demonstrates outstanding performance in terms of communication efficiency and robustness to client heterogeneity.
strength_and_weaknesses: Strength:
1. Propose practical algorithm that considers more realistic and strict FL settings
2. Outstanding performance in terms of communication efficiency and model performance over many baselines
3. The paper is well-written and easy to understand.

Weakness
1. The proposed algorithm FedACG is very similar to FedProx (Li et al., 2020) in the sense that FedACG has the additional hyperparameter \lambda. For example, when \lambda = 1, FedACG becomes FedProx. It would be great if this paper could elaborate on the relationship between FedACG and FedProx, and why introducing this additional hyperparameter \lambda is beneficial. 
2.  Following on from the first point, introducing this additional hyperparameter \lambda results in better performance but it’s not without any cost. Typically, we want to keep the number of hyperparameters as minimal as possible, because hyperparameter tuning is very costly. Actually, according to Table 6(a), the performance of FedACG varies a lot depending on the value of \lambda. 
3. For the convergence analysis (i.e., Theorem 1), I am not sure why we need to assume 0.5 < \lambda < 1, which seems to be a very strict assumption for \lambda because it’s practically possible for 0 < \lambda <= 1 (or even \lambda > 1 if we want more emphasis on the momentum m). Is it possible to derive a similar convergence rate when 0< \lambda < 0.5? In addition, when \lambda = 1,  FedACG becomes FedProx. It would be great if the convergence analysis can recover the analysis of FedProx when  \lambda = 1.
4. Lastly, [1] (see reference list below) has also considered using momentum for FL, and may need to be discussed in this paper.

Reference.   
[1] Das, Rudrajit, et al. "Faster non-convex federated learning via global and local momentum." Uncertainty in Artificial Intelligence. PMLR, 2022.

clarity,_quality,_novelty_and_reproducibility: Look good to me.
summary_of_the_review: The paper proposed an interesting method for FL. But I am afraid that it has several major weaknesses to be addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper (FedACG) proposes the following additive regularization term to the learning objective function for federated learning clients:

$$
\quad\quad \frac{\beta}{2}||\theta - (\theta^{t-1} + \lambda m^{t-1}) ||^2 
$$

where $\beta$ and $\lambda$ are tuning parameters. $\theta$ is the model parameters. The $m$ momentum term is calculated at the server, which is the difference between the sum of weighted parameter values received  from clients at round $t$ and round $t{-}1$. This is similar to $\ell_2$-regularization, but with a gradient acceleration (or momentum term).

The paper uses ResNet-18 and evaluates on datasets: CIFAR-10, CIFAR-100, Tiny-ImageNet, FEMNIST, and CelebA. IID data is generated via random sampling without replacement. Non-IID data is generated via sampling label ratios from a Dirichlet distribution. All clients hold the same number of training data.

Communication efficiency is captured by (reduced) number of communication rounds to achieve a particular level of test accuracy.

strength_and_weaknesses: The proposed technique is simple and the idea is intuitive. However, my main concern is with some of the empirical evaluation. Specifically:

1. How many trials were used in running the experiments?
2. Are the results in Table 1 (a) and (b) under the non-IID setting since the description in the table says "The Dirichlet parameter is commonly set to 0.3"? 
3. The way the Dirichlet distribution sampling is used in the paper seems different from [Hsu et al. 2019], where the data is drawn from Dirchlet($\alpha,\boldsymbol{p}$), where $\boldsymbol{p}$ characterizes a prior class distribution over $N$ classes, and $\alpha{>}0$ is a concentration parameter controlling the identicalness among classes.
4. Do all the accuracy numbers use the exponential moving average with 0.9 coefficient (meaning older observations are discounted faster)?
5. In Table 1 (a): for MOON (CIFAR-100), 500R shows Acc. of 83.32%, but 81% is achieved at 543 rounds? In Table 2: for FedDC (CIFAR-10), 500R shows Acc. of 60.56%, but 64% is achieved at 509 rounds. This means over 3% was gained in 9 rounds?
6. Table 6 shows that accuracy is sensitive to the shown different settings to $\lambda$, but stable to the show different settings to $\beta$. This is surprising, and somewhat counter intuitive from the perspective of $\ell_2$-regularization.
7. Table 6 shows the more realistic FL setting with feature skew and data imbalance. The results for FedACG compared to FedDC, FedAvg, FedAvgM are close enough such that it is not clear on the statistical significance of the result. I would be curious to see incorporating data imbalance in the main experimental results.
8. The empirical evaluations are limited to image datasets and one neural network model.


clarity,_quality,_novelty_and_reproducibility: Clarity and quality can be improved as it pertains to the empirical evaluation concerns mentioned. The proposed technique FedACG is related to FedAvgM [Hsu et al. 2019] as it pertains to using a momentum term, which I do not see mentioned and discussed in Section 2 Related Work. My understanding is that FedACG and FedAvgM computes the momentum at the sever-side. The difference being FedAvgM incorporates the term to the global parameters, whereas FedACG utilizes it as a regularization term on the client-side. However, it is used and compared to in the empirical evaluation. The results are strikingly different, and some analysis would be helpful.
summary_of_the_review: The proposed technique is simple and appears effective given the empirical results. However, the empirical evaluation can be improved.

Update: I thank the authors for answering my questions. I will maintain my review. However, having read through the other reviews (and replies) I think that the comparison with (at least) FedAvgM needs to be further clarified and highlighted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a communication-efficient federated learning algorithm, FedACG to accelerate the convergence of training. Specifically, by adding a momentum to the aggregated weight and using this term as a regularizer, the authors show that the model can converge faster with higher accuracy theoretically and empirically.
strength_and_weaknesses: Strengths:
- The proposed method is simple yet effective. Figure 1 gives a clear picture of the idea and makes the paper easy to follow.
- The experimental part is comprehensive. In detail, evaluation was conducted on three representative datasets under different data distribution. And it can be observed that FedACG performed consistently better among these settings. Ablation study to investigate individual components of the method is sufficient.
- A convergence analysis was provided to show that FedACG can guarantee the convergence of the model.

Weaknesses:

- The biggest concern of this paper is the difference between FedACG and FedAvgM [1]. It seems that FedACG merely adds a regularization term to reduce variance in local clients on top of FedAvgM, in which momentum was also leveraged to update the model in the server side. Then it is surprising to see that a regularizer can contribute to the performance gain significantly. More analysis and discussion of FedAvgM is necessary.
- To show FedACG achieves a better convergence rate theoretically, there should be a comparison among all other methods, i.e., the asymptotic convergence rate of methods such as FedAvg.
- 
- It lacks a discussion about differential privacy of FedACG. It is important to a FL algorithm whether it can preserve differential privacy.

[1] Hsu, Tzu-Ming Harry, Hang Qi, and Matthew Brown. "Measuring the effects of non-identical data distribution for federated visual classification." arXiv preprint arXiv:1909.06335 (2019).
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. However, it is lacking in novelty as FedACG only adds a regularization term on top of FedAvgM, and how this term works was not carefully discussed in the paper. Detailed hyperparameter setting together with the source code is provided and I believe it is easy to reproduce results in the paper.
summary_of_the_review: Overall, the paper presented FedACG, a communication-efficient federated learning algorithm and demonstrated its advantages through extensive experiments. However, it was similar to a previous paper FedAvgM and no discussion about difference between two methods. Thus, it was marginally below the acceptance threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper propose an approach to accelerate federated learning (FL) called FedACG. The approach is closely related to some well-known previous work (most notably FedProx and FedAvgM). The paper provides a theoretical convergence guarantee in the case of smooth convex losses, and it also provides experiments illustrating the promise of the proposed approach on the MNIST, CIFAR-10, and Tiny-ImageNet datasets.

The experimental results are promising.

At the same time, this is an area that has received a lot of attention over the past few years, and the proposed approach is very similar to others in the literature. The reviewers had a difficult time understanding the key differences from previous work that allow this approach to perform so much better. After much discussion among the reviewers and authors, this was somewhat clarified. However, the reviewers and AC were still not convinced that we understood why the subtle differences result in such marked performance improvements. This was the main weakness leading towards the recommendation that this paper be rejected. There was also an extensive discussion about these points involving the AC and all of the reviewers.

There are two key ways in which the paper could be strengthened to have a better chance of being accepted next time:
1. The writing could be improved to better put in context the contributions of this work in relation to previous work, especially clarifying the differences from previous methods like FedAvgM.
2. Additional experiments or analysis should be added to convince the reader why the specific changes introduced in this paper (i.e., differences from FedAvgM, FedProx, and other nearby methods) lead to the substantial improvements observed in the experiments.
justification_for_why_not_higher_score: Insufficient evidence to support all of the claims, especially given that the proposed method is very similar to prior work in the literature.
Decision: Reject