Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a CL periods-aware federated learning framework to adaptively select the amount of clients.  

strength_and_weaknesses: Strength:
S1: The idea of augmenting federated learning from the perspective of CL periods is easy-to-implement but effective.
S2: The proposed FedCL framework is also simple and effective. It complements with state-of-the-art FL frameworks such as the popular FedAvg. Experimental results show FedCL improves the test accuracy with the same level of communication rounds.

Weaknesses:
W1: The novelty contribution of this paper could be limited. From the perspective of CL periods, the setting of FL has no intrinsic difference from the distributed machine learning, which has been studied in the cited references. Thus, although the idea of this paper is quite effective, it could be just a basic and straightforward solution.
W2: The proposed FGN metric is somewhat intuitive. While the result shown in Figure 16 shows that FGN is more lightweight than existing FedFIM metric, the test accuracy improvement of these two metrics is not reported. 

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written and easy-to-follow.
Quality: The technical quality is good. The method proposed in this paper is empirically evaluated. 
Novelty: The idea of this paper is somewhat plain. 
Reproducibility: Model and experiment settings have been clarified to contribute to the reproducibility, but it would be better if the source code is publicly available.
summary_of_the_review: The paper proposes a simple but effective framework to adaptively control the clients number in FL. The consideration of CL periods in FL is significant and the effectiveness of the proposed method is empirically evaluated. However, the idea of applying CL periods detection to distributed ML is a little plain, the proposed FGN metric is also intuitive. Consequently, I recommend this paper a borderline score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper leverages the concept of critical learning period for client selection in federated learning, with the purpose of improving communication efficiency while maintaining or even improving the learning effectiveness. Built based on a previous work that identified the existence of critical learning period in federated learning, this work proposes a light-weighted metric to detect the critical learning period in each round of federated model update, and adaptively change the number of selected clients for gradient update. Extensive experiment results on two benchmark datasets, several base models and federated learning algorithms demonstrated the promising performance gain of the proposed solution. 
strength_and_weaknesses: Strength:
+ The proposed new metric is very straightforward and easy to implement, which ensures its applicability in practice. 
+ The authors provided very extensive experimentation about the proposed solution under various settings, e.g., different degree of heterogeneity across clients, different federated learning algorithms, and various settings of its hyper-parameters. This provides a comprehensive collection of evidence about the effectiveness of the proposed solution.
+ The manuscript is well-written and easy to follow.

Weakness:
- As the cause of critical learning period is unclear, it is hard to tell if the proposed solution can fundamentally solve the problem. Throughout the paper, including the related work section, there is no discussion about why the critical learning period is happening. Is it due to non-convex optimization? And why simply increasing the number of training instances will solve the problem? If it is due to non-convexity, the starting point also matters. The performed experiments seem to suggest heterogeneity is the main cause of critical learning period. But it is never explicitly articulated. Without any analysis about the mechanism of critical learning period, it is hard to justify why the proposed solution could solve the problem.
- The paper argues that permanent damage in model training can be made when the critical learning period is not well handled, no matter how many more training instances could be provided later on. But as the proposed solution could not address the issue either: if it detects the current round is in the critical learning period, it only increases the number of clients for the next round; but the damage has already been made in this round. In addition, client sampling can also cause variance in FGN calculation, which can misclassify the critical learning period. A false negative detection would introduce the damage. And hence, it is still unclear why the proposed solution could be helpful.  

Questions:
- Even under the proposed solution, the clients are still randomly sampled. Why do not we select the clients who can best help us circumvent the critical learning period? For example, find a set of clients whose local FGN is smallest?   
- Although FedFIM is expensive to compute, it is important to compare how effective its identified critical learning period is for improving federated learning comparing to FGN’s. 
- Since the communication cost is defined by the total number of communication rounds, instead of communicated bits, why do not we start with all clients? And how would the hyper-parameter m affect the solution’s performance?
- How would the solution perform if the environment is homogeneous? For example, when we uniformly distribute the dataset onto clients, what’s the performance gain against FedAvg?   

clarity,_quality,_novelty_and_reproducibility: The paper is mostly well-written with sufficient discussions about its settings and implementation details. However, I do have several questions need authors’ clarification. 

- It is unclear how the sampling based on Dirichlet distribution was performed to create heterogeneity among clients. It would be great if the authors could provide more details on this setup in the discussion phase.
- I am a bit confused about the $\Delta l_k(\cdot)$ used in Eq (1): is it the actual change of training loss or the approximated one using the gradient? 

The novelty of this work is relatively speaking limited, it is largely based on a previous work’s finding about the existence of critical learning period in federated learning and simplify the previous metric using the change of loss function. All the provided results are empirical, and it is hard to justify if the proposed solution can completely solve the problem. 

summary_of_the_review: The provided solution is simple and intuitive, and its effectiveness is proved via an extensive set of experiments. However, this paper lacks necessary investigation of why the critical learning period happens (e.g., what’s the root cause), and therefore it is hard to justify whether proposed solution can completely address the problem.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel approach to efficiently identify the critical learning (CL) periods and designs a CL period-aware federated learning (FL) algorithm. The algorithm adjusts the number of selected clients by increasing the number of clients per round during CL periods and returning gradually to normal when outside those periods. The authors claim that the algorithm augments state-of-the-art FL algorithms by a considerable margin and shows extensive empirical results to justify their approach.
strength_and_weaknesses: ## Strengths
- The idea to leverage gradient norms to detect critical learning periods is good.
- Results for three different models show significant performance improvements.
- The algorithm is very simple to implement and can be used on top of most state-of-the-art FL algorithms
## Weaknesses
- The datasets used `CIFAR-10` and `Fashion MNIST` is fairly simple. It would have been more convincing to see the results on more complex data like `CIFAR-100`. Additionally, the models used are too simple. It would be more convincing to show results on `Resnet-18` and `Mobile Net` if possible [1].
- The baseline accuracy results using Federated Fisher Information are not given. 
- It is unclear how the federated gradient norm predicts critical learning periods.
- It would be better to show the value of the federated gradient norm and the number of clients being sampled alongside the learning curve to better understand when critical learning periods occur. 
- It is not clear when the algorithm predicts a critical learning period and if it is effective. From my understanding, it should have some dependency on the random seed used to select clients randomly. Experiments should be run on multiple seeds to check if the algorithm finds critical learning periods at different points in training.
- The authors claim FedAvg is state-of-the-art which may not be true. Results on new algorithms like FedOPT[2] should be shown as well.
- A major question is when the critical learning period occurs. Does it always occur in the early optimization stage? If so, is detecting CL necessary? I think then a simple learning rate scheduling algorithm could work.

[1] Wang, Jianyu, et al. "A field guide to federated optimization." arXiv preprint arXiv:2107.06917 (2021).

[2] Reddi, Sashank, et al. "Adaptive federated optimization." arXiv preprint arXiv:2003.00295 (2020).


clarity,_quality,_novelty_and_reproducibility: The paper is clear in its motivation and objectives. The idea is simple and original.  The authors provide code for reproducibility.
summary_of_the_review: Although the paper proposes a simple but original idea, its evaluation is made difficult as the experiments and ablation studies are not convincing enough. The experiment section needs to be improved if not revamped and more theoretical foundations for critical learning periods and their detection should be added if possible.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a solution to adaptively set client number on critical learning periods in federated learning. Specifically, the authors introduce a simple yet effective metric to locate the critical learning periods. After locating the critical learning periods, the client number can be adaptively adjusted to keep a good balance between the final performance and communication efficiency.  The introduced metric and the whole framework are proved effective in the extensive experiments.

Contributions:
1. the authors introduced a simple yet effective metric to locate the critical learning periods in federated learning.
2. the authors integrated the introduced metric into different FL methods to construct FedCL framework, which can adaptively determine the number of clients to participate in each training around.
strength_and_weaknesses: Strength:
1. the introduced metric to locate the critical learning periods is simple yet effective, which has been proved in the extensive experimental results.
2. the designed FedCL framework and the adaptive client determination strategy can keep a good balance between final performance and communication efficiency. 
3. the authors conduct extensive experiments in this work to prove the efficacy and efficiency;

Weakness:
1. the introduced metric that is used to locate the critical learning, is based on local loss changes. However, using local loss values to select active clients in FL is not new. Please refer to the following works.
- Client selection in federated learning: Convergence analysis and power-of- choice selection strategies.arxiv, 2020
- Active federated learning. arrive, 2019
- FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learning, CVPR 2022

2. The strategy of adaptive client selection has been proved in prior works. However, in prior client selection works such as FedCor,  they select the clients based on their data characteristics, rather than simply increasing/decreasing the client numbers. Compared to those prior works, the client selection strategy in this work seems too simple and straightforward.

3. I can not see an experiment using random increasing/decreasing strategy. Did I miss that in some place?
clarity,_quality,_novelty_and_reproducibility: This work is clear and of fair quality. The introduce metric and designed framework are technically solid. What concerns me the most is the novelty and contribution. 
summary_of_the_review: This work proposes a solution to adaptively set client numbers on critical learning periods in federated learning. Specifically, the authors introduce a simple yet effective metric to locate the critical learning periods. After locating the critical learning periods, the client number can be adaptively adjusted to keep a good balance between the final performance and communication efficiency.  The introduced metric and the whole framework are proved effective in the extensive experiments.

What concerns me the most is the novelty and contribution. The introduced metric that is used to locate the critical learning, is based on local loss changes. However, using local loss values to select active clients in FL is not new. I have listed some references in the comments. Please make a comparison with them and list the difference. Other than that, the strategy of adaptive client selection proposed in this work, seems too simple (i.e., increasing/decreasing the active client numbers), compared to the prior works. 

I hope the authors can provide more discussions and comparisons in the rebuttal. I am willing to adjust my score if I receive convincing feedback.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper attempts to improve distributed training of deep networks. Towards this goal the authors leverage the known phenomenon of critical learning period in federated setup to achieve higher test accuracies across multiple datasets. In particular, a new metric called FGN is proposed to identify critical learning period and then adaptively select the amount of clients participating  based on it. We appreciate the extensive empirical studies carried out by the authors, but reviewers identified issues with the intricacies. We thank the authors and reviewers for actively engaging in discussion towards improving the paper, but still many of reviewer questions remain unanswered. It would be beneficial to understand more about the FGN metric and why FGN is a good metric for identifying critical learning period. Also highlighting key challenges about critical learning in a federated setup compared to centralized setting would make the paper stronger. Finally, it is not clear where the gains are coming from in the empirical studies. Is it due to the increased communication? The communication cost measured only in terms of rounds doesn't account for increased client participation in the proposed method. Carrying out the communication cost ablation as requested by the reviewers is crucial to demonstrate effectiveness of the proposed method.
justification_for_why_not_higher_score: - Not much novelty, other than leveraging of known phenomenon of critical learning period in a federated setup.
- No explanation/motivation on why proposed FGN detects critical learning phase.
- Some discrepancy how communication cost is compared across different methods.
Decision: Reject