Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper developed an adaptive aggregation method for federated learning. The theoretical analysis shows how the interval affects the convergence rate. The experiments show that it can reduce the communication cost. 
main_review: Pros:

1. The problem studied is important and the idea is novel. 
2. This paper provides the convergence rate.
3. This paper conducts extensive experiments.

Cons:
1. There are some errors in theoretical analysis. Eq.(15) is not correct. Eq.(25) is also wrong. 

2. For Lemma 5.2, there should be a constraint for $\eta$.

3. The convergence upper bound is worse than standard FedAvg. In Remark 1, when $K=O(m^3)$, FedLAMA can achieve linear speedup. However, in [1], $K=O(m)$. Thus, this bound is much worse than [1]


[1] On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
summary_of_the_review: This paper proposed a new idea for reducing communication costs. But the theoretical analysis has some flaws.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an adaptive interval schema for layer-wise model aggregation in federated settings. The aim of the proposed method is to reduce communication costs by adaptively decreasing the frequency of layer-wise aggregation with consideration of model discrepancy. 
main_review: Strengths:

1. It is an interesting idea to adjust the aggregation interval for layer-wise aggregation.
2. Detailed theoretical analysis and experiments are conducted.
3. The paper is well-written.


Weaknesses:

1. It is uncertain that the proposed method can really reduce communication costs. Due to the reduced frequency of aggregation and adopt layer-wise partial model aggregation strategy in each round, it is very likely that the proposed method will decrease the convergence speed of the global model. Therefore, the proposed method needs more communication rounds to achieve a convergence state, thus the total communication costs are still very high. Moreover, at each communication round, the client still needs to download the full model from the server. Therefore, the downloading communication costs haven't been improved. 
2. In the convergence analysis, Assumption 4 is a little tricky, and helps to avoid much detailed and further analysis. The convergence rate is O(1/m^½ K^½), which is loose compared to some convergence analysis of FedAvg, such as ON THE CONVERGENCE OF FEDAVG ON NON-IID DATA (https://arxiv.org/abs/1907.02189).
3. The experiment is insufficient to support their claims. For example, the only baseline is FedAvg, and it is necessary to choose some compression methods as a baseline. Moreover, the communication cost should consider the number of communication rounds to achieve a convergence state.


Questions and suggestions:

1. Algorithm 1 should explicitly state which steps are executed on the server or client. For example, line 3 should be on the client, and lines from 4 to 10 are on the server. Moreover, it should explicitly highlight when the communication occurs.
2. It is necessary to draw the convergence curves of the proposed methods and baselines.
3. It is unclear how to construct non-IID scenarios in the experiment setting.
4. There are some federated learning methods that only update parts of its model. This paper should also consider these baseline methods. 

http://proceedings.mlr.press/v139/collins21a/collins21a.pdf

https://arxiv.org/abs/2102.07623
summary_of_the_review: This paper proposes an interesting layer-wise model aggregation method to reduce communication cost while trying to keep the same performance with FedAvg. However, the proposed method’s design, theoretical analysis and experiments cannot support its claim.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a layer-wise model aggregation scheme in federated learning cases to reduce the communication cost. Specifically, they quantified the model discrepancy between local models and global models and adaptively adjusted the aggregation interval in a layer-wise manner. By increasing the aggregation intervals and relaxing the aggregation frequency, the method can reduce the communication cost in federated learning cases. The experimental results show it can reduce the communication cost for IID data and non-IID data compared to FedAvg.
main_review: They provide a communication-efficient federated optimization and relax the aggregation frequency by quantifying the model discrepancy between local models and global models and adaptively adjusting the aggregation interval in a layer-wise manner.

Some questions are the following.

1）The baseline is FedAvg, too few. Why not compare with other the periodic full aggregation scheme (e.g., FedProx, FedNova, SCAFFOLD, etc.), even some synchronized schemes? the synchronized schemes refer to,

[1] Chen, Yang, Xiaoyan Sun, and Yaochu Jin. "Communication-efficient federated deep learning with
layerwise asynchronous model update and temporally weighted aggregation." IEEE transactions on neural
networks and learning systems 31.10 (2019): 4229-4238.

[2] Xie, C., Koyejo, S., & Gupta, I. (2019). Asynchronous federated optimization. arXiv preprint
arXiv:1903.03934. 

2）FedLAMA seems to lead to some accuracy drop as the experiments show. They do not show the total time in terms of communication cost, just show the reduced ratio for communication. It is not clear about absolute earnings with sacrificing the accuracy. In some cases, especially in industry, the model's accuracy matters more than training time (e.g., the total training time is within several hours.)  
summary_of_the_review: Thanks to the authors for their efforts for their theoretical analysis and experimentally proof of the proposed federated optimization scheme. The experimental results are a little weak to show the effectiveness and the practicality.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel adaptive model aggregating schema under a federated learning setting, where it effective reduces the communication overhead without significantly sacrificing the generalization performance. Both empirical study and theoretical analysis are included to justify the effectiveness and efficiency of the proposed method.
main_review: Effectively reducing the communication overhead is an important problem for federated learning, with the recent advance of layer-wise model freezing training, this paper manages to extend this idea for network traffic reduction in federated learning. Concretely, I list the main strengths and minor weakness of this paper as below:

Advantages:

+ To leverage layer-wise model freezing for federated learning is a simple but novel idea. According to the empirical study, this method seems to be effective and efficient. 
+ The paper also includes tight convergence analysis for the proposed method. 

Disadvantages:
+ The communication cost calculated in Section 6 is a little sloppy, in practices, there are plenty of system advances managing to hide the communication overhead within the computation slots, e.g. PyTorch-DDP (https://arxiv.org/abs/2006.15704). In this fashion, the actual runtime may not be simply linear to the amount of network traffic. Perhaps, a more sophisticated cost model would consider both the computation power of the edge device and network condition during training, then model the cost including the mechanism of overlapping communication and computation to estimate the actual runtime. 


summary_of_the_review: The idea of utilizing layer-wise model freezing for federated learning is an interesting idea, the paper includes reasonable empirical study and convergence analysis to illustrate the effectiveness of this proposed approach. 
Although some part of the cost model in the simulation is a little sloppy, I tend to advocate the acceptance of this paper.
review_title: null

Meta-review: This paper proposes a layer-wise adaptive aggregation method for federated learning that seeks to reduce the communication cost. The frequency of aggregation is adjusted separately for each layer of the model that is being trained. The number of iterations $\tau$ after which each layer's parameters are averaged across clients is multiplied by a factor $\phi$ depending on the magnitude of changes to the parameters at each layer. The paper gives a convergence analysis of the proposed method and provides experimental results to demonstrate its effectiveness in reducing communication without compromising accuracy.  

Reviewer 7ks6 found some errors in the convergence analysis that were fixed by the authors in the discussion period. Reviewer YGZf increased their score to 5 after the discussion with the authors. The reviewers also gave the following suggestions to improve the paper:
1) Showing convergence curves to demonstrate that the communication reduction does not come at the cost of a slowdown in convergence.
2) The results presented in the paper shows only a small communication reduction for many of the layers. Perhaps the strategy can be improved in order to boost the communication reduction.
2) Use a more realistic model to calculate the communication cost so as to account for network delays and other costs rather than just considering the number of parameters communicated.

The scores are split on this paper. While one of the reviewers recommends acceptance, three others say that the paper is below the acceptance threshold. So I recommend a rejection while noting that the paper is close to the borderline.
Decision: Reject