FedCL: Critical Learning Periods-aware Adaptive Client Selection in Federated Learning
Authors: Gang Yan
Hao Wang
Xu Yuan
Jian Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: Critical Learning Periods
Federated Learning
Client Selection

Abstract: Federated learning (FL) is a distributed optimization paradigm that learns from data samples distributed across a number of clients.  Adaptive client selection that is cognizant of the training progress of clients has become a major trend to improve FL efficiency but not yet well-understood.   Most existing FL methods such as FedAvg and its state-of-the-art variants implicitly assume that all learning phases during the FL training process are equally important.  Unfortunately, this assumption has been revealed to be invalid due to recent findings on critical learning (CL) periods, in which small gradient errors may lead to an irrecoverable deficiency on final test accuracy.  In this paper, we develop FedCL, a CL periods-aware FL framework to reveal that adaptively augmenting exiting FL methods with CL periods, the resultant performance is significantly improved when the client selection is guided by the discovered CL periods.  Experiments based on various machine learning models and datasets validate that the proposed FedCL framework consistently achieves an improved model accuracy while maintains comparable or even better communication efficiency as compared to state-of-the-art methods, demonstrating a promising and easily adopted method for tackling the heterogeneity of FL training.  


Under review as a conference paper at ICLR 2023
FEDCL: CRITICAL LEARNING PERIODS-AWARE ADAP- TIVE CLIENT SELECTION IN FEDERATED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning (FL) is distributed optimization paradigm that learns from data samples distributed across number of clients. Adaptive client selection that is cognizant of the training progress of clients has become major trend to improve FL efficiency but not yet well-understood. Most existing Fl. methods such as FedAvg and its state-of-the-ee the-art variants implicitly assume that all learning phases during the FL training process are equally important. Unfortunately, this assumption has been revealed be invalid due recent findings critical learning (CL) periods, in which small gradient errors may lead to an irrecoverable deficiency on final test accuracy. In this paper, we develop FedCL, CL periods-awar FL framework to reveal that adaptively augmenting exiting FL methods with CL periods, the resultant performance is significantly improved when the client selection is guided by the discovered CL periods. Experiments based on various machine learning models and datasets validate that the proposed FedCL framework consistently achieves an improved model accuracy while maintains comparable or even better communication efficiency as compared to state-of of-e--o-----rrr methods, demonstrating a promising and easily adopted method for tackling the heterogeneity of FL training.
INTRODUCTION
Federated learning (FL) (McMahan et al. 2017) has emerged as an attractive distributed learning paradigm that leverages large number of clients to collaboratively learn a joint model with decentral- ized training data under the coordination of a centralized server. In contrast with centralized learning, the FL architecture allows for preserving clients' privacy and reducing the communication burden caused by transmitting data to the server. While there is rich literature in distributed optimization in the context of machine learning. FL distinguishes itself from traditional distributed optimization in two key challenges: high degrees of system and statistical heterogeneity (Kairouz et al. 2019).
In an attempt to address the heterogeneity and improve the efficiency of FL, various optimization methods have been developed for FL. In particular, the federated averaging algorithm (FedAvg) (McMahan et al. 2017) is the current statc-of-ther off-the-art method for FL. In each communication round, FedAvg leverages local computation each client and employs centralized server aggregate and update the global model parameter. While FedAvg has demonstrated empirical success in heterogeneous settings, it fails to fully address the underlying challenges associated with heterogeneity, For example, FedAvg randomly selects subset of clients in each iteration regardless of their statistical heterogeneity, which has been shown diverge empirically in settings where data samples of each client follow non-identical and independent distribution (non-IID).
recent trend of improving FL efficiency focuses on adaptive client selection during the FL training
process, such as (Ruan et al. 202T Karimireddy et al., 20200 Li et al.) [2020a) Wang et al.) 2020c b
Cho et al. 2020, Wang et al. 2020a Rothchild al. [220 Lai et al.) 2021). However, these studies
implicitly assume that all learning phases during the FL training process are equally important.
Unfortunately, this assumption has recently been revealed to be invalid due to the existence of critical
learning (CL) periods, i.e., the final quality of deep neural network (DNN) model is determined by the first few training epochs, in which deficits such as low quality or quantity of training data will cause irreversible model degradation. Notably, this phenomenon was revealed in the latest series of works (Achille et al.] [0119] Jastrzebski et al. 2019; Golatkar et al. 2019; Jastrzebski et al.) (021) for centralized learning, and in (Yan et al. 2022 for FL. settings. Despite their insightful findings, there

---

Under review as a conference paper at ICLR 2023
remains to be a major gap between the observation of CL periods in FL and the goal of more efficient training and improved model accuracy. since existing client selection methods in state-of ofate-of-the-art FL algorithms wnaware of the existence of CL periods in FL, which were only identified using computationally expensive metric that emerges after the full training process.
In this paper, we close this gap by demonstrating the importance of CL periods awareness for client selection in state-of-the-ar FL algorithms. Through a range of carefully designed experiments on different machine learning models and datasets, we observe a consistently improved model accuracy without sacrificing communication efficiency by augmenting state-of-the-ar FL algorithms with CL periods. We build upon recent work by (Yan et al., 2022). who showed that if the training dataset for each client is not recovered to the entire training dataset early enough in the training process, the test accuracy of FI. is permanently impaired. We extend this notation to client selection in FI. and show that a larger number of clients are only required during these CL periods. As a result, an adaptive and efficient client selection scheme is akin to finding CL. periods in the FL training process. These CL periods can be detected in an online manner using new metric called Federated Gradient Norm (FGN). To the best of our knowledge, this is the first step taken towards exploiting CL periods for adaptive client selection in FL to mitigate heterogeneity.
Our main contributions in this paper are summarized as follows:
1. We propose practical, easy-to-compute Federated Gradient Norm (FGN) metric identify CL periods in an online manner, fixing a major paradox for connecting CL periods with client selection for the efficient FL training goal.
2. We propose simple but powerful CL periods-aware FL. framework, dubbed as FedCL., that is generic across and orthogonal to different FL methods. In particular, we use FedAvg as our building block since it is the first and the most widely used one. FedCL inspects the changes in FGN to detect CL periods in FL training process, and adaptively determines the number of clients to participate in each FL training round. With extensive empirical evaluation different machine learning models with different datasets, show that FedCL consistently achieves up to 11% accuracy improvement while maintaining comparable or even better communication efficiency compared to FedAvg.
3. We show that the CL periods awareness can be easily combined with state-of-the-art FL methods, such as FedProx (Li et al. (2020)). VRL-SGD (Liang et al. (201)) and FedNova (Wang et al. 2020c). When augmented by FedCL via manipulating the client selection, existing methods achieve up to 11%, 13% and 10% accuracy improvement, respectively, compared to training without the awareness of CL periods.
2 RELATED WORK
Critical Learning (CL) Periods. The presence of CL periods in centralized neural network training
was first highlighted in (Achille et al. 2019; Jastrzebski et al., 2019). Some other works (Golatkar al. 2019 Jastrzebski et al. 2021 Frankle et al. |020 1ec00Jastzzza Jastrzebski et al. 2020) have also highlighted the importance of early training phase in centralized learning. The existence of CL. periods in FL was recently discovered in (Yan et al., 2022). However, studying CL phenomena hinged on costly information metric (e.g., eigenvalues of the Hessian) that emerges after the full training, limiting their practical benefits. We differ from existing works by developing an easy-to-compute metric to identify CL periods during the training process in an online manner.
Federated Learning and Client Selection. The state-of-the-art method for FL is FedAvg, which was first proposed in (McMahan et al.) (017) and has sparked many follow-ups (Stich 2018 Wang Joshi 2021: Yu et al., 2019) with full client participation. In practice, only a small fraction of clients participate each training round, which exacerbates the effect of data heterogeneity. As a result, solutions with partial client participation and effect of data heterogeneity have been developed and
analyzed (Katharopoulos & Fleuret 2018 Ruan et al. 2021 Karimireddy et al. 2020 Li et al. 2020a
Wang et al.) 2020c) Li et al. 2020bb Wang et al. 2020b Ribero & Vikalo| 2020 Cho et al. 2020
Wang et al. 2020a
Yang et |021 et al., 2022 Reddi et al. |002 Haddadpour & Mahdavi
2019
Khaled al.
2020 Stich & Karimireddy] [020 Woodworth et al. |020; Horváth & Richtarik
2021
Nishio & Yonetani
2a1inMMainovskki et (2020) Pathak Pathak & Wainwright 20200 Goetz et al.
2019
Tang al. 2022)
For comprehensive introduction to FL. and other algorithmic variants
2

---

Under review as a conference paper at ICLR 2023
Algorithm 1 FedAvg
Input: M, E, 0
1: for 0, 1,... I do
2: Server selects a subset MM of .M clients at random
3: Server sends O(t) all selected clients
4: Client A E MM( updates O(4) via E iterations of SGD on Dr with stepsize to obtain (1+1))
5: Each selected client E M sends 0 ((++1) back to the server
6: Server aggregates the 0's as (((-1) EkEMO Ezemen N.
N2 0 ((+1)
7:
end for
FL., we refer interested readers to (Kairouz et al., 2019). Unlike these works that are agnostic to the existence of CL periods, we design novel CL periods-aware FL framework. Importantly, we remark that our CL periods-aware FL framework, FedCL is orthogonal to these methods, since FedCL. merely augments state-of-the-art FL method to adaptively determine the number of clients that participate in each FL. training round, rather than changing the way how the FL. method selects clients.
3 BACKGROUND
The Federated Optimization Setting, Consider the federated architecture where M clients jointly solve the optimization problem: minggedd F(0)= = Ene (O)),(O), where Pk N represents the relative sample size, and (0) N₂ EEED, (0; ( the local objective function at the k-th client. Here Ea denotes the loss function defined by the learning model, & represents data sample from local dataset DR, and M denotes the set of clients.
The Federated Averaging Algorithm. Federated Averaging (FedAvg) (McMahan al. 2017) is the first and most common algorithm used to solve the above optimization problem through aggregating the locally trained models at the central server at the end of each communication round. At the initial step, the central server randomly initializes global model (()), At each round, a fixed number of randomly selected clients run E iterations of local solver, e.g., the stochastic gradient descent (SGD) (Yu et al. 2019 Wang & Joshi 2019 202T). and then the resulting model updates are averaged. The [M S M, VI.
details of FedAvg are summarized in Algorithm where (1) C M and
Although the performance of FedAvg has been improved in both theory and practice by recent literature such as FedProx (Li et aalaa22.2000 2020a), FedNova (Wang et al. (2020c). SCAFFOLD (Karimireddy
et al. 2020), VRL-SGD (Liang et al. 2011), FedBoost (Hamer et al. 2020), FedMA (Wang et al. 2020b and FetchSGD (Rothchild al. (020), FedAvg is the first and the most widely used one. a result, we see FedAvg as our basic block. Our proposed CL periods- aware FL framewwwrkk FedCL is orthogonal and can be easily combined with these methods (see Section Moreover, FedCL also compatible with and complementary to other techniques such as gradient compression/qunnizztion (Basu et al. 2019 Haddadpour al. (02T).
CL Periods in FL. The latest series of studies have identified CL periods or the initial training phase that are important for training a high-quality model in both centralized training (Achille et al. 2019) Jastrzebski et al. 2019) and decentralized training (Yan et al. 2022). We build upon recent work by (Yan et al. 2022), who showed that if the training dataset for each client not recovered to the entire training dataset early enough in the training process, the testing accuracy of FL is permanently impaired. We extend these ideas to aid in the design of adaptive client selection in Fl. and show that more clients are only required during these CL. periods.
4 FEDCL: A CL
CLIENT SELECTION FRAMEWORK
This section describes our proposed approach efficiently detect CL periods in federated settings that lay out the rationale behind our method. The rest of this section focuses on our proposed framework FedCI. that augments client selection in state-of-the-art methods with CL periods.
3

---

Under review as a conference paper at ICLR 2023
FadFiM
00 20
15
25
41 00
40 66
46
Roard
Roand
001 0000000 0.1
[3] d- 0.2
1: - 0.3
100
0.1
0022
10 0.3
(a) CIFAR-10
(b) Fashion-MNiSS
Figure 1: Comparison of detecting CL periods in federated settings using FGN with 8 0.01 and FedFIM, where the shade and Nouble-arrows indicate identified CL periods. The results are conducted using Alex (a) CIFAR-I0 and (b) Fashion-MNIST datasets, which are non-IlI) partitioned across 128 clients using Dirichlet distributions 2ir (0.1). 2 (0.2), and 22r (0.3), respectively.
4.1 ADAPTIVE CLIENT SELECTION VIA CL PERIODS AWARENESS
Yan al. 2022) showed that the final test accuracy of FL is dramatically affected by early training phases. They setup experiments where only partial datasets are available for the first few commu- nication rounds and then continue training the model with entire training datasets for the rest of communication rounds. Surprisingly, the FL model trained in this way showed permanent impaired test accuracy performance no matter how many additional training rounds are performed after CL periods. We extend these ideas to aid in the design of adaptive client selection in FL since is major trend of improving FL efficiency and handling heterogeneity. As motivated by aforementioned works, it is clear that finding adaptive client selection scheme is akin to finding CL periods in FL training process. To this end, we begin with how to efficiently detect CL periods in FL. training process.
Detecting Critical Learning Periods. Prior works use the changes in eigenvalues of the Hessian or approximating the Hessian using (federated) Fisher information (Achille et al.) 2019 Dastrzebski] al. 2019 Yan al. 2022) as an indicator to detect CL periods. We deviate from these works and develop an approach based on federated gradient norm (FGN), which can be efficiently computed.
Considering the difference in training loss for an individual data sample e. let {) 2/ (0:{E denote the gradient of the loss function evaluated on ç. After performing a step SGD on this sample, the training loss Al- = f(O ng(0 {) {(:: {) can be approximated by its gradient norm using Taylor expansion, i.e., Al ~ g @ 1l2 As a result, the overall training loss at the t-th round, which we define as the FGN, can be approximated using the weighted average of training loss across all selected clients, i.e.,
(GN
Σ kEMi
N
AC K
ΣxEMO N
We compare the CL periods identified by our FGN approach with the federated Fisher information (FedFIM) approach in (Yan et al., 2022). From Figure [[ we observe that these two approaches yield similar results, but our FGN approach is much more computationally efficient (being orders of magnitude faster to compute) and can be easily leveraged for client selection during the training process in an online manner. For example, the FedFIM approach takes up to 9x more computation time and consumes 40x more memory than our FGN approach (See Appendix A.2. for details).
4.2 THE DESIGN OF FEDCL FRAMEWORK
We now describe FedCL, our proposed framework that adaptively determines the number of selected clients for FL. training by leveraging identified CL periods. Again, we use FedAvg as the building block, and our framework can be easily combined with other existing methods, which will illustrate in Section 5) Our FedCL builds on the identified CL periods, which can be efficiently detected by FGN as discussed previously (see Figure []. To this end, we develop simple threshold-based rule to detect the CL periods based on FGN as tollows: if
FGN(t)- FGN(t 1) 2 8,
(2)
FGN(1. - 1)
then the current round t is in CL periods, where is the threshold used to declare CL periods. We set 5 0.01 as the default value in our experiments and will investigate its impact in Section 5
4

---

Under review as a conference paper at ICLR 2023
Algorithm 2 FedCL: A CL Periods-Aware Client Selection Framework
Input: M, E, T, 6, m selected clients with initial global model (
1: for = 0, 1.... .T - do
2: Server selects a subset (() of .M clients at random
3: if (GNNt)-PG((t-I)) 2 5 then
[G
4: Clients in M update the local model and server aggregates local models via FedAvg |M t min M) Hor///DDuule //ooubling the clients in CL periods
5:
6: else
7:
[M + max m 1 fm) //Halving the clients after CL periods
8: end if
9:
end for
Per our discussions on CL periods, the final model accuracy will be permanently impaired if not enough clients are involved in CL periods no matter how much additional training is performed after the period (Yan et al., 2022). Therefore, our FedCI. framework increases the number of selected clients of FedAvg from no to 2no. implying that more clients now participate in improving the global model in the next round during the CL periods. Using the model learned from the previous round on, the initial model, the 2ng selected clients employ the FedAvg and continue the learning procedure to reach a global model 0.n..- The procedure of geometrically increasing the number of selected clients continues till the set of selected clients contains all the available M clients when the communication rounds are still in CL periods. Since the final accuracy of using partial datasets is similar to that of using all dataset after CL periods (Yan et al.) 2022), FedCL starts to gradually decrease the number of selected clients after CL periods for the sake of communication efficiency. Algorithm 2 summarizes the CL periods-aware FedAvg algorithm, dubbed as FedCL.
From high-level perspective, FedCL exploits more clients in the initial phase of the learning procedure than a fixed number of clients for FedAvg in each round, to promptly reach a global model with higher accuracy since the initial learning phase plays critical role in FL performance. By doing so, we hypothesize that the SGD is navigating to the steeper parts of the loss surface of the global model during CL periods since larger amount of data samples have contributed to the global model. However, the communication overhead of such an approach is relatively large since more clients involved in FL training in each communication round. By gradually decreasing the number of selected clients after CL periods, the communication overhead of FedCL improves without hurting the final model accuracy. The key point is that more clients join the training process in the initial learning phase, and only a smaller number of clients are needed after CL periods. As a result, FedCL consistently improves the model accuracy while maintains comparable or even better communication efficiency than FedAvg.
As the proposed FedCL in Aggorithmpprooeee identified CL periods in federated settings, one needs to specify the inner optimization subroutine (c.g., line in Algorithm 2 to quantify the improvement of the proposed approach. In particular, we set the subroutine to be FedAvg in Algorithm ssnce it is the most common algorithm and the building block of many variants in federated settings. This subroutine could be any federated learning algorithms (with possible variants), such as FedProx (Li et al. (2020a), VRL-SGD (Liang et al. 2019)
general framework augment client selection with
and FedNova (Wang et al. 2020c), which we will numerically illustrate Section 5
5 EXPERIMENTS
We first present an empirical study of FedCL in Section 5.2 and then illustrate the generalization of our proposed framework by combining CL periods with other methods such as FedProx, VRL- SGD and FedNova in Section 5.3 We study classification problems using two representative DNN models: AlexNet (Krizhevsky et al. 2012) and VGG-11 (Simonyan & Zisserman 2015) on non-III partitioned CIFAR-10 and CIFAR- 100 (Krizhevsky al. 2009), and Fashion-MNIST (Xiao et al. 2017) datasets. We further investigate the task of next-character prediction on the dataset of The Complete Works of William Shakespeare (Shakespeare) (McMahan et al. 2017). We relegate details of datasets and models, and additional experimental results, particularly on CIFAR- 100 and Shakespeare, to Appendix A
5

---

Under review as a conference paper at ICLR 2023
Rounds
Roarde
Roaads
M1
2010
010
10
(34 01
Figure 2: Test accuracy of FedAvg and FedCL using (top) AlexNet and (bottom) VGG-11 on non-lID CIFAR-10.
Figure 3: Test accuracy of FedAvg and FedCL using (top) AlexNet and (bottom) VGG-11 on non-lID Fashion-MNIST.
Fashig
FedCL
60
Hi
Recuured Rounds
8 B is d
70
Requieed Rounde
50
30
20
10
(ta-Meeshen
VGG-11
AndAee
VII
Figure 4: Communication efficiency FedAvg and FedCL on non-IlD CIFAR-10.
Figure 5: Communication efficiency of FedAvg and FedCL non-llD Fashion-MNIST..
5.1 EXPERIMENT SETUP
We implement FedCL and considered baselines in PyTorch (Paszke et al. 2017) on Python 3 with three NVIDIA RTX A6000 GPUs. For CIFAR-10, CIFAR- and Fashion-MNIST datasets, we simulate the non-IID FL scenario by considering heterogeneous partition for which the number of data points and class proportions are unbalanced. In particular, we simulate a heterogeneous partition into M clients by sampling Pk: (i) where is the parameter of the Dirichlet distribution. We choose a = 0.1, 0.2, 0.3 our experiments as done in (Wang et al.) 2020b]c). The level of heterogeneity among local datasets across different clients consider the total number of clients to be 128. The local learning rate is initialized as 0.01 and decayed by constant factor after each communication round. We set the weight decay be 10-5 The detection threshold is 8 = 0.01 and the number of local training epochs is E' 2. An ablation study conducted in Section to4 investigate the impact of these hyperparameters. We run each experiment with three independent trials and report the mean results.
be reduced when a increases. We
5.2 IMPORTANCE OF CL. PERIODS AWARENESS: FEDCL EsS. FEDAVG
In this experiment, we study the performance of FedCL. with accuracy and communication efficiency. Our goal compare FedCL to FedAvg in terms of the final test accuracy and the number of communication rounds needed for the global model to achieve good performance on the test data.
Test Accuracy. The test accuracy comparisons non-IID partitioned CIFAR-10 10 and Fashion- MNIST with FedAvg selecting 16 clients in each round are shown in Figures 2 and 3 respectively. Obviously, FedCL consistently outperforms FedAvg all scenarios with an improved final test accuracy to 11%. Its advantage is especially pronounced when the dataset is partitioned across clients using Dirichlet distribution with parameter 0. i.e., the datasets across clients are highly non-IID, Not surprisingly, we observe the importance of CL periods awareness in training efficiency, which is fully reflected via the test accuracy. For example in Figures a) and 3ja), FedCL exhibits a dramatic accuracy increase in the early phase of the FL training process. This coincides with the fact that FedCL selects a larger number of clients in each round in the early phase due to the awareness of CL periods (lines 3-6 in Algorithm() 2). Though the accuracy slightly decreases in short period due to the decreased number of selected clients (lines 7-9 in Algorithm (2). the final test accuracy is significantly improved. Our findings the importance of CL. periods awareness in the FL training

---

Under review as a conference paper at ICLR 2023
Tom Aoooaae
2bunds
Founce
Rounds
Rounds
Rhande
Raands
00
00
10-00.3
Figure 6: Test accuracy of FedProx, VRL- SGD, FedNova and FedProx-CL, VRL-SGD- CL, FedNova-CL using (top) AlexNet and (bottom) VGG-11 on non-IID CIFAR-10.
Figure 7: Test accuracy of Fedxrox, VRL-SGD, FedNova and FedProx-CL, VRL-SGD-CL, FedNova-CL. using (top) AlexNet and (bot- tom) VGG-11 on non-llI Fashion-MNIST.
FeePrix
FesPhceOL
FodProx
FedProx-CL
70
Requree Rounds
00
Reqiiee Rounds
50
8
30
20
70
10
Figure 8: Communication of FedProx and FedProx- on non-IID CIFAR-10.
Figure 9:
Communication of FedProx and FedProx-CL. on non-IID Fashion-MNIST.
process, e.g.. for client selection, seem be consistent with recently reported observations that the initial learning phase plays key role in determining the outcome of the training process.
Communication Efficiency. The benefit of CL periods awareness for FL training is further reflected via communication efficiency. In Figures
4 and we report the communication rounds required for FedCL. and FedAvg to achieve a targeted accuracy, which chosen be the final test accuracy of FedAvg as reported in Figures 2 [nd 3 respectively. Comparisons
other targeted test accuracy can be found in Appendix A.2 It is clear that FedCL requires fewer rounds to achieve the same test accuracy. Again this advantage is pronounced on highly non-llI data partitions. We further compute the average number of clients involved in each round for FedCL and FedAvg to achieve its final test accuracy. We observe that FedCL consistently improves the model accuracy (as shown Figures 3 with its average number of clients involved each round being 0.95x to 1.1 x that of FedAvg. This due the fact that more clients only needed during CL periods and smaller number of clients are need afterwards in our FedCL framework.
5.3 GENERALIZATION
As mentioned earlier, our proposed CL periods-aware FL framework, FedCL is orthogonal existing state-of-the-art methods, and hence can be easily combined with these methods by simply replacing the inner optimization subroutine (FedAvg) in Algorithm 2 with the corresponding methods. To this end, we study the generalization of FedCL and consider three state of the arts, i.e., FedProx (Li et al. 2020a). VRL-SGD (Liang al. 2019) and FedNova (Wang al. 2020c). We call the corresponding CL. perio methods as Ped VRL-SGD-CI. and FedNova-CI,, respectively.
We notice that the performance of FedProx depends on the hyperparameter μ., i.e., the coefficient associated with the proximal term of each local objective. We tune this parameter using grid search and report the best value of μ = 0.01 for AlexNet experiments and μ = 0.001 for VGG-II experiments. We present results on non-lID datasets across 128 clients with FedProx, VRL-SGD and FedNova selecting 16 clients in each iteration. Again, the CL periods awareness significantly improves the test performance of baseline methods, i.e., FedProx-CLe VRL-SGD-CDC and FedNova- CL outperforms FedProx, VRL-SGD and FedNova, respectively, in all scenarios with an improved final test accuracy up %, 13% and 10%, respectively, shown in Figures 6 andd]] 7 Its advantage is
7

---

Under review as a conference paper at ICLR 2023
Padlag
Fodhhox
FodMovi Fofflone
2.02
02
0.3
Potz
FodPooo
VRL-900-0
46-
Accatescy (%)
To-
Final Accutsoy
60
25
30
50-
20
25
4007
6
CPAAROO
(8) Fastion MAAT
(4) CIFA-0
(5) Frehion MNIST
Figure 10: Sensitivity of detection threshold.
Figure 11: Impact of non-lID degree.
Factrrx
VRL-GGD VRL BL..-DD-LL
Tachowa Fechooa-O.
Foltig
FedPiix
VRL800
Fothovo
PodCL
Cl CL
FocC
FodFige CL.
VRL 0R0000
35
56-
Final Aacutacy (15)
Final Accantaryy
30
50-
35
40
35
30
1805
182 1645
Epochs
Epochs
100 CIFAR-10
(b) Fashion MNST
(6) CIFAR-10
(b) Fashron NNST
Figure 12: Effect of local training epochs.
Figure 13: Effect of weight decays.
especially pronounced on highly non-IID dataset across clients in Figures ([a) and 7 [a). Likewise. the CL periods augmented method, e.g.. FedProx-CL. requires fewer rounds to achieve the final test accuracy of the corresponding baseline FedProx, as shown in Figures and while maintaining a comparable average number of clients involved in each round. Similar observations can be made for VRL-SGD-CL vs. VRL-SGD and FedNova-CL vs. FedNova, which can be found in Appendix A.2
5.4 ABLATION STUDY
Detection Thresholds. As discussed previously in Figure I our experiments reveal that CL periods can be efficiently identified using the easy-to-compute FGN via a simple threshold-type rule in Equation (2). We now evaluate the sensitivity of the threshold value 5 used to declare CL periods. The candidate threshold values we consider are {0, 0.01, 0.03, 0.05, 0.2, 0.35, 0.5), and the corresponding final test accuracy of FedCL. using AlexNet on non-Ill) partitioned CIFAR-10 and Fashion-MNIST illustrated in Figure TO When data partitions are highly non-IID (i.e., a 0.1), the CI. periods declaration determined by 8 has observable effect on the final accuracy. This is because as 8 becomes larger, fewer rounds the initial phase are declared as CL periods by Equation 2 As result, the effect of CL periods awareness the final test accuracy is shallowed since FedCL only uses a larger number of clients in fewer rounds compared to FedAvg according to Algorithm 2 On the other hand, our CL periods-aware framework FedCL is robust to the detection process, ie.. tolerant to detection errors with different threshold values when data partitions are not highly non-IID. Similar observations can be made for FedProx-CI. VLRLL----..LL-IIA----
and FedNova-CL. and hence relegated to Appendix A.2 To this end, we set 8 = 0.01 in all of our experiments.
A.2
Non-IID Degree. We simulate a heterogeneous data partition into M clients using the Dirichlet distribution with parameter a. From Figure II we observe that the CL periods awareness consistently improves the final test accuracy of saae-or-the-art method across all values of in consideration. For example, FedCL always outperforms FedAvg, and FedProx-CC always outperforms FedProx The benefits of CL periods awareness are especially pronounced when the datasets across clients are highly non-III (i.e., smaller value of a).. Hence, we choose a = 0.1, 0.2, 0.3 for illustrations in above experiments. For ease of readability, we set a 0.1 in the rest of ablation studies and relegate results on 0.2, 0.3 to Appendix(A.2 A.2 In addition, the non-llI degree decreases (as increases), the final test accuracy of FedCL, FedProx-CL, VRL-SGD-
and FedNova-CL increases.
This is consistent with recently reported observations, e.g., in (Lin al. 2020 Achituve et al. 2021 Gong et al. 2021) that non-III degree degrades the model final test accuracy.
Local Training Epochs. We note that the number of local training epochs (denoted E) is common
parameter shared by considered baselines, which reportedly has an impact on the performance of
FedAvg (McMahan et al. 2017 Wang et al. 2020b). To this end, evaluate the effect of E using
8

---

Under review as a conference paper at ICLR 2023
Pezling
Pechfowa FulNoww-C.
Feeiigg
PesProx
VIL-ECO
Fechove
VRL-80D-G.
FeoO
FosProo-O
VRL-500-CL
Fochowo-OL
Final Accutacy
50
30
40
30
160
23
40 15
Total Clents
Total Cliants Fedhion NNNSS
Partcipants (56 CIFA--10
Partisipants (56) (b) Fledhin MMUST
(n) GPAB-10
Figure 14: Impact of number of clients.
Figure 15: Impact of client participation rate.
AlexNet on non-lII partitioned CIFAR-10 and Fashion- MNIST with a = 0.1. The candidate local epochs we consider are E' E {1,2, 3, 5, as done in (Wang et al. 2020ch. From Figure [2] we observe that increasing the number of local epochs improves the final test accuracy general, and the CL periods awareness consistently improves the final test accuracy of state-of-the-an methods across all values of E. Since the gains test accuracy exhibit the "diminishing return effect" as the number of local epochs increases, we set E' in all of our experiments.
Weight Decay. Though the CL periods in FL are robust to the values weight decays as reported in Yan et al. 2022), the final test accuracy using AlexNet non-IlD partitioned CIFAR-10 10 and Fashion-MNIST with a = 0.1 is still affected with the values of weight decays as shown in Figure I3 Again, we consistently observe the benefits of CL periods awareness across all values of weight decays. Since the advantage decreases as the weight decay increases, we set the weight decay to be 10-5 in experiments.
Number of Clients. In all of our above experiments, we consider a FL setting with 128 clients in total. We now consider the same experimental settings above besides varying the total number of clients in the system using AlexNet on non-IID partitioned CIFAR-10 10 and Fashion-MNIST with 0.1. As shown Figure [4, the advantage of CL periods awareness exhibits across all settings, i..., FedCL (resp. FedProx-CIl outperforms FedAvg (resp. FedProx) regardless of the total number of clients. Without loss of generality, we choose M 128 in all of our experiments.
Client Participation Rate. In all of our experiments, FedAvg, FedProx, VRL-SGD and FedNova select 16 out of 128 clients to participate in each training round, i.e., the participation rate is 12.5%. We now investigate the impact of client participation rates on the model accuracy and the awareness of CL. periods using AlexNet on non-III partitioned CIFAR- 10 and Pashion-MNTS" with a = 0.1. Again, when tfate-of-the-art method is augmented with the CL. periods, the final test accuracy is consistently improved across all client participation rates. The advantage is particularly pronounced with low participation rate. This is quite intuitive since in our CL periods aware framework, FedCL selects more clients during the CL periods than FedAAg (see line in Algorithm 2) and hence the benefits are more obvious when FedAvg has low client participation rate. We select 16 clients, i.e., a 12.5% participation rate for all state-of-the-ar methods via considering the tradeoff' between final test accuracy and benefits of CL periods awareness.
CONCLUSION
In this paper, we presented FedCL,, a simple but powerful CL periods-aware FL framework for adaptive client selection. FedCL. worked by adaptively choosing more clients in CL periods during the FL training process and fewer clients elsewhere. We proposed a practical and casy-to-compute federated gradient norm (FGN) metric to identify such CL periods during the training process in an online manner. We showed that FedCL significantly improved the final test accuracy by up to 11% compared to its counterpart FedAvg using different models and datasets, while maintaining comparable or even better communication efficiency. Finally, we illustrated the generalization of our proposed CL periods aware framework via manipulating the client selection of state-of-the-art methods augmented by FedCL. In the future work, we want to extend FedCL to improve FL of different machine learning models other popular techniques such gradient compression/quantizaii personalization, and adversarial attacks. We also believe that it is important to study the performance of FedCL. on other models and datasets.
fair aggregation,

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical Learning Periods in Deep Networks. In Proc. of ICLR, 2019.
Idan Achituve, Aviv Shamsian, Aviv Navon, Gal Chechik, and Ethan Fetaya. Personalized federated learning with gaussian processes. Proc. of NeurIPS, 2021.
Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi, Qsparse-local-SGD. Distributed SGD with Quantization, Sparsification and Local Computations. In Proc. of NewrIPS, 2019.
Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client Selection in Federated Learning: Convergence Analysis and Tower-of-Choice Selection Strategies. arXiv preprint arXiv:20/0.01243, 2020.
Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Towards understanding biased client selection in federated learning. In Proc. of AISTATS, 2022.
Jonathan Frankle, David J Schwab, and Ari Morcos. The Early Phase of Neural Network Training.
In Proc. of ICLR, 2020.
Jack Goetz, Kshitiz Malik, Duc Bui, Seungwhan Moon, Honglei Liu, and Anuj Kumar. Active federated learning. arXiv preprint vrXiv:1909) 1909, 2019.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence. Proc. of NeurlPS, 2019.
Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje. Ensemble attention distillation for privacy-preserving federated learning. In Proc. of IEEE/CVF ICCV, 2021.
Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXiv: /910. 14425, 2019.
Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated Learning with Compression: Unified Analysis and Sharp Guarantees. In Proc. of AISTATS, 2021.
Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. FedBoost: A Communication-Efficiie Algorithm for Federated Learning. In Proc. of ICML, 2020.
Samuel Horváth and Peter Richtarik. A better alternative to eиror feedback for communication- efficient distributed learning. In Proc. ICLR, 2021.
Stanislaw Jastrzebski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos J Storkey. On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length. In Proc. of ICLR, 2019.
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The Break-Even Point on Optimization Trajectories of Deep Neural Networks. In Proc. of ICLR, 2020.
Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand, Giancarlo B Kerg, Huan Wang, Caiming Xiong, Richard Socher, Kyunghyun Cho, and Krzysztof J Geras. Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization, Proc. of ICML, 2021.
Peter Kairouz, Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and Open Problems Federated Learning. arXiv preprint arXiv: 192.04977 2019.
Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. In Proc. of ICML, 2020.
Angelos Katharopoulos and François Fleuret. Not All Samples Are Created Equal: Deep Learning with Importance Sampling. In Proc. of ICMI., 2018.
10

---

Under review as a conference paper at ICLR 2023
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. In Proc. of AISTATS, 2020.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware neural language models. In Proc. of AAAI, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning Multiple Layers of Features from Tiny Images. 2009.
Alex Krizhevsky, Lya Sutskever, and Geoffrey E Hinton. Imagenet Classification with Deep Convo- lutional Neural Networks. Proc. of NIPS, 2012.
Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. Oort: Efficient Federated Learning via Guided Participant Selection. In Proc. of USENIX OSDI, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated Optimization in Heterogeneous Networks. In Proc. of MLSys, 2020a.
Xiang Li, Kaixuan Huang. Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the Convergence of FedAvg on Non-IID Data. In Proc. of ICLR, 2020b.
Xianfeng Liang, Shuheng Shen, Jingchang Liu, Zhen Pan, Enhong Chen, and Yifei Cheng. Variance Reduced Local SGD with Lower Communication Complexity. arXiv preprint 19X1v:1912.12844, 2019.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Proc. of NeurlPS, 2020.
Grigory Malinovskiy, Dmitry Kovalev, Elnur Gasanov, Laurent Condat, and Peter Richtarik. From local sgd to local fixed-point methods for federated learning. In Proc. of ICML, 2020.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-Effiicee Learning of Deep Networks from Decentralized Data. In Proc. of AISTATS, 2017.
Takayuki Nishio and Ryo Yonetani. Client selection for federated learning with heterogeneous resources in mobile edge. In Proc. of IEEE ICC, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, W, 2017.
Reese Pathak and Martin J Wainwright. Fedsplit: An algorithmic framework for fast federated optimization. Proc. of NeurlPS, 2020.
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In Proc. of ICLR, 2021.
Monica Ribero and Haris Vikalo. Conamumnicaaion-Effciieeee. Federated Learning via Optimal Client Sampling, arXiv preprint 1rXiv:2007.15197, 2020.
Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. FetchSGD: Communication-EFfan Efficieet Federated Learning with Sketching. In Proc. of ICML, 2020.
Yichen Ruan, Xiaoxi Zhang, Shu-Che Liang, and Carlee Joe-Wong, Towards Flexible Device Participation in Federated Learning. In Proc. of AISTATS, 2021.
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-scale Image Recognition. In Proc. of ICLR, 2015.
Sebastian U Stich. Local SGD Converges Fast and Communicates Little. arXiv preprint 2018
11

---

Under review as a conference paper at ICLR 2023
Sebastian U Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for sgd with delayed gradients and compressed updates. Journal of Machine Learning Research, 21:1-36, 2020.
Minxue Tang, Xuefei Ning, Yitu Wang, Jingwei Sun, Yu Wang, Hai Li, and Yiran Chen. Fedcor: Correlation-based active client selection strategy for heterogeneous federated learning. In Proc. of IEEE/CVF CVPR, 2022.
Hao Wang, Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing Federated Learning on Non-IID Data With Reinforcement Learning. In Proc. of IEEE INFOCOM, 2020a.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated Learning with Matched Averaging. In Proc. of ICLR, 2020b.
Jianyu Wang and Gauri Joshi. Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-update SGD. In Proc. of SysML, 2019.
Jianyu Wang and Gauri Joshi. Cooperative SGD: A Unified Framework for the Design and Analysis of Local-Update SGD Algorithms. Journal of Machine Learning Research, 22(213):1-50, 2021.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. Proc. of NeurlPS, 2020c.
Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? In Proc. of ICML, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A Novel Image Dataset for Bench- marking Machine Learning Algorithms. arXiv preprint 170.0777, 2017.
Gang Yan, Hao Wang, and Jian Li. Seizing Critical Learning Periods in Federated Learning. In Proc. of AAAI, 2022.
Haibo Yang, Minghong Fang, and Jia Liu. Achieving Linear Speedup with Partial Worker Participa- tion in Non-IID Federated Learning. In Proc. of ICLR, 2021.
Hao Yu, Sen Yang, and Shenghuo Zhu, Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning. In Proc. of
AAAI, 2019.
12

---

Under review as a conference paper at ICLR 2023
A EXTENSIVE REVIEW AND RESULTS OF EXPERIMENTS
We provide details of datasets and models Appendix A. Appendix A.2
and additional experimental results in
Parameter
Shape
Layer hyper-paramann parameter stride:1; padding: 1
layerI.com I. weight layer wow.ll.oollleee b.as
3 x 64 X 3 x 3
64
N/A
pooling.max
N/A
kernel size:2; stride: 2
2
64 x 128 x x 3
stride: padding: 1
layer2.conv2.biar
128
N/A
pooling.max
N/A
kernel size:2: stride: 2
layer3.conv3,. weight
128 x 256 X 3
stride:1; padding: 1
Soyer3.conv3..m
256
N/A stride: padding:
layer4.conv4. weight weight
256 x 256 x 3
10yer4.cov4..0
256
N/A
pooling.mo max
N/A
kernel size:2; stride: 2
256 x 512 3 X 3
stride:1; padding: 1
layer5.com c.n.o..ovvaa
512
N/A
fayer6.conv6.. weigt.
512 x 512 3 x
stride:1; padding: 1
layer6. www.ro..covv.... bias
512
N/A
pooling.man max
N/A
kernel size:2; stride: 2
Jayer?.conv7.weile
512 x 512 x 3 x 3
stride: 1; padding: 1
layer7.com sotee77.oov7.bbaa
512
N/A
layer8. conv.. w.eiiht
512 x 512 X 3 X 3
stride: 1; padding: 1
layer8, convo. bias pooling.man max dropout
512
N/A
N/A
kernel size:2; stride: 2
N/A
p=20%
layer9.fel fc9, .teeit
4096 x 512
N/A
layer9.fcl fc9.bias
512
N/A
512 x 512
N/A
1ayer10.fe bias dropout layerl w.eeghht layer1 1.fe11.bias
512
N/A
N/A
p=20%
512 x 10
N/A
10
N/A
Table 1: Detailed information of the VGG-11 architecture used in our experiments. All non-
linear activation function in this architecture is ReLU. The shapes for convolution layers follow (Cins C,
A.1 SUMMARY OF DATASETS AND MODEL ARCHITECTURES
We implement FedCL. and considered baselines in Py Torch (Paske et al. 2017) on Python 3 with three NVIDIA RTX A6000 GPUs, 48GB with 128GB RAM. We conduct experiments the popular datasets CIFAR-I0 and CIFAR-100 (Krizhevsky et al. 2009), Fashion-MNIST (Xiao et al.) 2017) and Shakespeare (McMahan al. 2017). The CIFAR-I and CIFARI-00 dataset consists of 60,000 32 color images in 10 and 100 classes, respectively. where 50,000 samples are for training and the other 10,000 samples for testing. The Meshion-MNISS datasets contain handwritten digits with 60,000 samples for training and 10,000 samples for testing, where each sample is an 28 28 grayscale images over classes. The Shakespeare dataset consists of 74 characters with 734,057 training data and 70,657 testing data. We simulate heterogeneous partition into M clients by sampling ps: ~ Dir (a), where a is the parameter of the Dirichlet distribution (Wang et al. 2020ble Specifically, for each class of samples, set the class probability in each client by sampling from a Dirichlet distribution with the same a parameter. For instance, when a = 0.5, sampling Pk Dir(0.5) and allocating Pk.j proportion of the training instances of class l: to local client j.
We summarize the details of VGG-11 (Simonyan & Zisserman 2015) and AlexNet KKizhevsky et al.
2012) architectures used in our experiments for classification tasks in Tables T and respectively.
13

---

Under review as a conference paper at ICLR 2023
For the language task, train stacked character-level LSTM language model as in (Kim et al. 2016
McMahan et al. 2017). which is summarized in Table
Parameter
Shape
Layer aaper-parameter
layerI.com weiggh
3 x 64 x 3
stride:2; padding: 1
layerl.com .convl.bial 1.ais
32
N/A
pooling.max
N/A
kernel size:2; stride: 2 stride: padding:
layer2.conv2.ww weight
64 x 192 x x
Jayer2.conv2.bia.
64
N/A
pooling.max
N/A
kernel size:2; stride: 2
layer3.conv3. weight
192 x 384 x
stride:1; padding: 1
Sover3.conv3...
128
N/A stride:1; padding: 1
aoyer4.conv4. weight
384
256 X
1oyer4.con44...
128
N/A
layer5. conv5. weight
256 x 256 x 3
stride:1; 1; padding:
layer5, wowvo.ooviiaa bias
256
N/A
pooling.m. pox max dropout
N/A
kernel size:2; stride: 2
N/A
p=5%
layer6.fc6. weight
1024 x 4096
N/A
layer6.fc6. fcor fc6.bias
512
N/A
dropout
N/A
P=5%
layer7.fc7.weete
4096 4096
N/A
layer7.fc7 fiia.iiff..
512
N/A
layer8.fc8. weight
feeeeh layer8.fol fe8 bias
4096 x 10
N/A
10
N/A
Table 2: Detailed information of the architettueee used in our experiments. All non-linear acti-
vation function in this architecture is ReLU. The shapes for convolution layers follow (Cins Couts C, c).
Parameter layerl .ebbeding
Shape
Layer hyper-parameter
80 x 256
N/A
layer2.lstm
256 512
num layerss2,
dropout
N/A
p=5%
layer3.fc. fc. weight
512 80
N/A
layer3.fc.biar
80
N/A
Table 3: Detailed information of the LSTM architecture used our experiments.
A.2 ADDITIONAL EXPERIMENTAL RESULTS
Results complementary Section on non-IlD partitioned CIFAR-10 and datasets given in Appendix A.2.2 Experimental results on non-lID partitioned CIFAR-100 and Shake- speare datasets are presented Appendix A.2.3
2000
Menary (MB)
Time (ma).
10 100
0.0
Dalasel Model Time
Dalasel Model GPU
Figure 16: Computation time and memory sumption of FGN and FedFIM approaches to detect CL periods.
06 200003
K0a D3
Figure 17: Test accuracy of FedCL using FGN
and FedFIM approaches to detect CL periods.
14

---

Under review as a conference paper at ICLR 2023
A.2.1 COMPARISON BETWEEN FGN AND FEDFIM BASED APPROACH TO DETECT CL PERIODS
As discussed in Section we propose lightweight FGN based approach detect CL periods that can be easily leveraged for client selection during the training process in an online manner. Though the detection performance is similar between FGN and FedFIM approach as shown in Figure [ our FGN approach is much more computationally efficient. For example, consider the settings i Section 4 the computation time and memory consumption of FGN and FedFIM under the settings in Appendix A presented in Figure 16 where C+A, C+V, M+F, and F+AA represent Alex on CIFAR- 10, VGG-11 on CIFAR-10. 10, FC on MNIST, and AlexNet on Fashion-MNIITT respectively. Since the identified CL periods by using FGN are almost the same as those identified by using FedFIM (see Figure T. the test accuracy of FedCL is expected to be the same. For instance, the test accuracy of FedCL when leveraging the CL periods identified by FGN and FedFIM, which denote as FedCL (FGN) and FedCL (FedFIM), respectively, using AlexNet on CIFAR-10 is reported
Figure 17
Sead1
2ead
Sead
S
Seads
Saed
Seed2
Se632
Sead
SeadG
Roand
and
Reand
Roard
Faund
Roard
(4
231
00
[N
141
CIFAR-10
(b) Fashion-MNIST
Figure 18: The impact of random seed when using FGN to detect CL periods. The results conducted using AlexNet on (a) CIFAR-I0 and (b) Fashion-MNIS, datasets, which are non-IID partitioned across 128 clients using Dirichlet distributions Dir (0.1), Dir (0.2), and Dir (0.3). respectively.
Fuoliggg 0.11
Fushhggg - FodCii 0.25
Fudlivgte FueDlie- 0.3)
We further evaluate the impact of random seed when using FGN to detect CL periods. In particular, we randomly generate five seeds, and report the identi- fied CL periods in Figure 18] We observe that our FGN can consistently identify the CL periods across different random seeds and the identified CI. periods almost the same. This performance is further pronounced when we compare the test accuracy as shown Figure 19 These results further support the robustness of our proposed FGN metric.
Foot 0.1
Fina Acausny (%)
02707 J002 2000
Fordome (0) CIFARI1
[M Fredtion
Figure 19: The test accuracy of FedAvg and
FedCL using different random seeds.
A.2.2 ADDITIONAL RESULTS ON CIFAR- 10 AND FASHION -NNIST DATASETS
resiuggs
FacRogis
radkugi
Fadkigle 0.2)
- 0.3)
FasCLie
FedCUie 0.2)
FesCLin
FedCUpp 0.3)
FosCLin
FecClip
Roquied Rounds
80
10
Requieed Rounels
40
50
10
Ascursccy (8))
Ascurary (N) (b) vaG--f
Acturtty (%) Addeet
Aczuraty 196) (h1 vivvG 11
Figure 20:
Communication of FedAvg and FedCL on non-lID CIFAR-10.
Figure 21:
Communication of FedAvg and FedCL on non-lID MaShion-MNIST.
Communication Efficiency. Similar to Figures and we report the communication rounds required for FedCL and FedAvg to achieve some common targeted accuracies in Figures 20 and 21 Again, we observe that FedCL requires fewer rounds to achieve the same test accuracy as FedAvg The communication comparison for FedProx, VRL-SGD and FedNova that is similar to Figures and 9 are presented in Figures and 23 for VRL-SGD vs. VRL-SGD-CL. and in Figures and for FedNova FedNova-C1.
15

---

Under review as a conference paper at ICLR 2023
-000-1
Requred Roundds fo
60
RequteddRouu
30
30
10
10
10
Aleahe
VG2-11
(1) VGG-11
Figure 22: Communication of VRL-SGD and VRL-SGD-CL on non-llD CIFAR-10.
Figure 23: Communication of VRL-SGD and
VRL-SGD-CL on non-IlD Fashion-MNIIT.
FodMowa
Fodhove
FedNovo-OL
Requres Rounes
a.a 8.
70
Requird Rounds
HI
80
8
50
8.5
30
20
(n)-Meaket
VBG-11
(n)Aedee
VI/66-11
Figure 24: Communication of FedNova and FedNova-CL. on non-IID CIFAR-10.
Figure 25: Communication of FedNova and FedNova-C2 on non-IID Fashion-MNIST.
Detection Thresholds. Complementary to the results shown in Figure I0 the detection thresholds also have the same impact on the final test accuracy of FedProx-CC CL,, VRL-SGD-CI. and FedNova- CL using AlexNet on non-IID partitioned CIFAR-I0 10 and Fashion-MNIST illustrated in Figures 26 27 and
Figure 26: Sensitivity of detec- tion threshold: FedProx-CL.
Figure 27: Sensitivity of detec- tion threshold: VRL-SGD-CL.
Figure 28: Sensitivity of detec- tion threshold: FedNova-CL.
A.2.3 RESULTS ON CIFAR- 100 AND SHAKESPEARE DATASETS
Test Accuracy. We present the test accuracy on non IID partitioned CIFAR- 100 and Shakespeare datasets in Figures 29 and 30 and Figures 31 and 32 respectively. Again we observe that the CL periods awareness consistently improves the final test accuracy.
Communication Efficiency. The comparisons of communication rounds required for FedCL, FedProx-CI, VRL-SGD-CL and FedNova-CL. achieve the final test accuracy of FedAvg, FedProx, VRL-SGD and FedNova, respectively, are presented in Figures 33 34 35 and 36) respectively, using the non-IID partitioned CIFAR-101 dataset; and Figures a7d 38 respectively, using the non-IID partitioned Shakespeare dataset. Similar conclusions can be made and hence we omit the discussions here.
16

---

Under review as a conference paper at ICLR 2023
Fadiig FeaCk
Fedticx
Viu560
VRL-20-CL
Fedion
Fedioacd
1o www.hoa
Tecnhooraic 14)
1001 Se:4
Racidd
Rruice
Rounnds
POARIE
POUREE
Figure 29: Test accuracy: (top) AlexNet and (bottom) VGG-11 on non-lID CIFAR-100.
Figure 30: Test accuracy: (top) AlexNet and (bottom) VGG-11 on non-IID CIFAR-100.
Techhox
VRL-5GD
STRL-500-0.
Fachlon
Rounds
Founds
Fournds 1oyo 0033
Rourdss
-11000. 0.1
0
0C 00000.1
OS 0.3
Figure 31: Test accuracy: LSTM on non-IID
Figure 32: Test accuracy: LSTM on non-IID Shakespeare.
Shakespeare.
Tackig
FedCL
TedProx
TwPProOCL
200
Required Rounds
178
1
175
150
1
Requined Rounds
175
100
175
129
125
125
100
100
100
1:
50
20
50
050
Figure 33: Communication of FedAvg and FedCL on non-IID C-FAR-100.
Figure 34: Communication of FedProx and FedProx-CL. on non-IID CIFAR-100.
VRL 550
VRJ VRI.500-
Fedkiwa
Fwrhina--C
205
175
178
175
Requieddosse dounde
150
Requred RequreeRoune
160
125
125
125
100
100
68
0
0.3.3
0-001
2 0.2
2-0.3
0-003
0=10
0=0
(4)
Figure 35: Communication of VRL-SGD and VRL-SGD-CL on non-lID CIFAR-100.
Figure 36: Communication of FedNova and FedNova-CL. on non-IID CIFAR-100.
hethooLL
VHL-SOD
VRL-500-CL
Po/las
Fuchona
200
178
175
175
175
Requree Rounes
150
Rounds
150
125
125
100
100
100
50
30
50
0-02
2-0.3
0-0.1
2-0-2
6-0.3
0-
0-
0-0.3
0 0.1
0-0
G-
Fedtvg
Figure 37: Communication of FedAvg FedCL and FedProx vs. FedProx-CI on non- IID Shakespeare.
Figure 38: Communication of VRL-SGD
VRL-SGD-CL and FedNova vs. FedNova- CL on non-IID Shakespeare.
17

---

Under review as a conference paper at ICLR 2023
Roundy
POunEEa 6.
Roanb
Figure 39: Test accuracy of FedAdagrad, Yogi, FedAdam and FedAdagrad-CL, Fed CI, FedAdam-CI. using (top) AlexNet and (bottom) VGG-11 on non-lII
Figure 40: Test accuracy of FedAdagrad, Fed Yogi, FedAdam and FedAdagrad-CL, Fed Fed Yogi-CI,, FedAdam-CL. using (top) AlexNet and (bottom) VGG-11 on non-IID FaSHion-MNIST.
CIFAR-10.
A.3 ADDITIONAL RESULTS ON THE GENERALIZATION OF FEDCL
In Section 5.3 we discuss the generalization of our proposed FedCL. by considering three state the arts, i.e., FedProx (Li et al. (2020a). VRL-SGD (Liang al.] [219)) and FedNova (Wang et al. (022cc). We now provide additional experimental results on the generalization of FedCL by considering FedOPT (Reddi et al. 2021) with three other ofae-of-theeart methods, i.e., FedAdagrad, Fed and FedAdam. We call corresponding CL periods-awar methods as FedAdagrad-CIL Fed Yogi-O
and FedAdam-CL, respectively. We consider the same setting that in Section 5.3 From Figures 39 and 40 we again observe that CL periods awareness significantly improves the test performance of baseline methods, i.e., Fe,,Aagrad-CL, Fed and FedAdam-CL. outperform FedAdagrad, FedYogi, and FedAdam, respectively, all scenarios. Likewise, the CL periods augmented method, e.g., F-eeAdegrad-dCI requires fewer rounds achieve the final test accuracy of the corresponding baseline FedAdagrad, as shown in Figures 41 and 42 Similar observations can be made for Feg
in Figures 43 and and FedAdam in Figures and 46
FodMdagnd
Fedidagas5-L
Roculted Rounds
Requiee Rounee
50
80
10
(nl VGG-6
VGG-11
Figure 42: Communication of FedAdagrad
Figure 41: Communication of FedAdagrad and FedAdagrad-CL on non-IID CIFAR-10.
and dag on non-IID Fashion- MNIST.
Fostog
Fodhiip
Fedifigi
70
Required Rounes
L
Recquii Rounde
8 1
50
40
20
10
0.1
Alesher
(b) VGG--1
(dl VGG-11
Figure 43: Communication of Fed Yogi and
Figure 44:
Communication of Fed and Fed on non-III Fashion-MNIST..
FedYogi-CL. non-IID CIFAR-10.
A.4 RANDOM INCREASE AND DECREASE THE NUMBER OF PARTICIPATED CLIENTS
Besides deterministically increasing or decreasing the number of participated clients as in FedCL, randomly increase decrease the number. Specifically, consider two settings. On the one
18

---

Under review as a conference paper at ICLR 2023
Fadkdne
Faskdte--i
Teskdom
Feckdemm
50
Requied Roundss
70
70
Requied Rounds
80
50
1
20
20
Aleahe
VGG-11
(11 VGD-11
Figure 45: Communication of FedAdam and FedAdam-CL on non-III CIFAR-10.
Figure 46: Communication of FedAdam and FedAdam-CL on non-llI Fashion-MNIST.
hand, we fix the probability of decreasing the participated clients from m to m/2 to be 30% in each round, and investigate the impact of the probability of increasing the participated clients from to 2m in each round. On the other hand, we fix the probability of increasing the participated clients from m 2m be 30% each round, and investigate the impact of the probability of decreasing the participated clients from m to m/2 in each round. We report the final test accuracy of the model, and compare it with the FedCI. (see Algorithm ! and the FedAvg. As shown Figures (47 and 48 random increase or decrease may not necessarily improve the performance of FedAvg. This is due to the fact that the random strategy may not necessarily align with the findings critical learning periods (e.g., Achille et al. 2019 and Yan et al. 2022) that more data/clients need to be involved in early training phases.
D2
0-3
0-101
02
0.3
Final FinalAccunng Accutacy (96)
Firet Accutsacy (%)
Teckograr
Teckgg--
55-
50
Probabiity
30%
Incresse Probabilty
27%
27% Probabiity
Docrease Probabiity
CIFAR-10
(h) Fastiio MIIST
(N) CIFAR-10
(0) Fastion MMST
Figure 47: Test accuracy of FedCL when randomly increase the number of participated clients.
Figure 48: Test accuracy of FedCL when ran- domly decrease the number of participated clients.
A.5 THE NUMBER OF PARTICIPATED CLIENTS AND THE FGN
We report the number of participated clients and the FGN curve in Figure 49 We observe that more clients are involved in the early training phases where the CL periods occur.
Nu nOUr Clamns
14
2.0
0.0
0.6
Round 11FAR100
Round
Foeriin MN
Figure 49: Relationship between FGN and the number of participants.
19

---

Under review as a conference paper at ICLR 2023
redlag
32
iited Roundee
68
25
Rounds
Rourde
Roands 01
M1
I01o U2
Figure 50: Test accuracy of FedAvg and CedCL using ResNet-18 non-IID (top) CIFAR-10 and (bottom) Fashion-MNIST.
CFFA410
Fiehhon MNET
Figure 51: Communication efficiency of Fe- dAvg and FedCL using ResNet-18.
A.6 RESULTS ON RESNET- MODEI.
FedCL FedAvg: Complementary Section 5.2 We further evaluate the performance of FedCL. using the ResNet-18 model on non-IlD partitioned CIFAR- 10 and Pashion-MNIS" datasets. The test accuracy comparison non-IlD partitioned CIFAR-10 10 and Fashion-MNIST with FedAvg selecting 16 clients in each round is shown in Figure Again, we observe that FedCL consistently outperforms FedAvg in all scenarios. Similarly, the benefit of CL periods awareness for FL training is further reflected via communication efficiency as shown in
Importance of CL Periods Awareness:
Figure 51
Complementary to Section 5.3 and Appendix A.3 we study the generalization of FedCL and consider six state of the arts, i.c., FedProx, VRL-SGD, FedNova, FedAdagrad. Fegio and FedAdam, respectively, using the ResNet-18 18 model. The test accuracy comparisons are presented Figures (52 and and [33 with the communication efficiency comparisons presented Figures 59 respectively. Again, we observe that the CL periods awareness improve the performance of the state the arts.
Generalization:
SrofPric SagRira
VR 5011 URI 9A0
Fedhirah
Rellkiar
Fadie Padicapiat
Falldarn
Fedidien
Pertied
100 10
Roandh
Fbunda
0.3 Figure 52: Test accuracy of FedProx,
05 0-001
00
1000
Figure 53: Test accuracy of FedAdagrad, Fed Yogi, FedAdam and FedAdagrad-CL, Fed Fed Yogi-CL, FedAdam-CL using using ResNet-18 non-lID (top) CIFAR-10 and (bottom) Fashion-MNIST..
VRL-SGD, FedNova and FedProx-CI, VRL- SGD-CL, FedNova-CL. using ResNet-18 non- IID (top) CIFAR-10 and (bottom) Fashion- MNIST.
PecPrcs
PedPhos-CL
VRL-SGD VIL-50D-OL
Requres Rounds
B B
16
5
1IFAR-1
Fashion
0i MRIST
Figure 54: Communication of FedProx and FedProx-CL. using ResNet-18.
Figure 55: Communication of VRL-SGD and VRI-SGD-CL using ResNet-18.
20

---

Under review as a conference paper at ICLR 2023
FLshova
Fadkiagnd
Faokdegas-Ci
Requies Rounds
Requied Rourdls
20
10
1IFAR-1
(8) Fastion MNET
(0) CFFA.10
(b) onshion MIST
Figure 56:
Communication of FedNova and
Figure 57: Communication of FedAdagrad and FedAdagrad-CL using ResNet-18.
FedNova- CL using ResNet-18.
Fuakdom
20
35
Requred Rounds
Required Rounds
an eo
18
10
10
Js) 10
(8) Fashio MNST
00) 10
(b) Fashonn MNST
Figure 58: Communication of FedAdam and
Figure 59: Communication of Fed and FedYogi-CI using ResNet-18.
FedAdam- CL using ResNet-18.
21


Reviews:
[{"review_id": "niHnMuGuGX9", "reviewer": "Reviewer_dEoL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a solution to adaptively set client number on critical learning periods in federated learning. Specifically, the authors introduce a simple yet effective metric to locate the critical learning periods. After locating the critical learning periods, the client number can be adaptively adjusted to keep a good balance between the final performance and communication efficiency.  The introduced metric and the whole framework are proved effective in the extensive experiments.\n\nContributions:\n1. the authors introduced a simple yet effective metric to locate the critical learning periods in federated learning.\n2. the authors integrated the introduced metric into different FL methods to construct FedCL framework, which can adaptively determine the number of clients to participate in each training around.", "summary_of_the_review": "This work proposes a solution to adaptively set client numbers on critical learning periods in federated learning. Specifically, the authors introduce a simple yet effective metric to locate the critical learning periods. After locating the critical learning periods, the client number can be adaptively adjusted to keep a good balance between the final performance and communication efficiency.  The introduced metric and the whole framework are proved effective in the extensive experiments.\n\nWhat concerns me the most is the novelty and contribution. The introduced metric that is used to locate the critical learning, is based on local loss changes. However, using local loss values to select active clients in FL is not new. I have listed some references in the comments. Please make a comparison with them and list the difference. Other than that, the strategy of adaptive client selection proposed in this work, seems too simple (i.e., increasing/decreasing the active client numbers), compared to the prior works. \n\nI hope the authors can provide more discussions and comparisons in the rebuttal. I am willing to adjust my score if I receive convincing feedback.", "strength_and_weaknesses": "Strength:\n1. the introduced metric to locate the critical learning periods is simple yet effective, which has been proved in the extensive experimental results.\n2. the designed FedCL framework and the adaptive client determination strategy can keep a good balance between final performance and communication efficiency. \n3. the authors conduct extensive experiments in this work to prove the efficacy and efficiency;\n\nWeakness:\n1. the introduced metric that is used to locate the critical learning, is based on local loss changes. However, using local loss values to select active clients in FL is not new. Please refer to the following works.\n- Client selection in federated learning: Convergence analysis and power-of- choice selection strategies.arxiv, 2020\n- Active federated learning. arrive, 2019\n- FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learning, CVPR 2022\n\n2. The strategy of adaptive client selection has been proved in prior works. However, in prior client selection works such as FedCor,  they select the clients based on their data characteristics, rather than simply increasing/decreasing the client numbers. Compared to those prior works, the client selection strategy in this work seems too simple and straightforward.\n\n3. I can not see an experiment using random increasing/decreasing strategy. Did I miss that in some place?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "n3DLs1ct0j", "reviewer": "Reviewer_phLC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel approach to efficiently identify the critical learning (CL) periods and designs a CL period-aware federated learning (FL) algorithm. The algorithm adjusts the number of selected clients by increasing the number of clients per round during CL periods and returning gradually to normal when outside those periods. The authors claim that the algorithm augments state-of-the-art FL algorithms by a considerable margin and shows extensive empirical results to justify their approach.", "summary_of_the_review": "Although the paper proposes a simple but original idea, its evaluation is made difficult as the experiments and ablation studies are not convincing enough. The experiment section needs to be improved if not revamped and more theoretical foundations for critical learning periods and their detection should be added if possible.", "strength_and_weaknesses": "## Strengths\n- The idea to leverage gradient norms to detect critical learning periods is good.\n- Results for three different models show significant performance improvements.\n- The algorithm is very simple to implement and can be used on top of most state-of-the-art FL algorithms\n## Weaknesses\n- The datasets used `CIFAR-10` and `Fashion MNIST` is fairly simple. It would have been more convincing to see the results on more complex data like `CIFAR-100`. Additionally, the models used are too simple. It would be more convincing to show results on `Resnet-18` and `Mobile Net` if possible [1].\n- The baseline accuracy results using Federated Fisher Information are not given. \n- It is unclear how the federated gradient norm predicts critical learning periods.\n- It would be better to show the value of the federated gradient norm and the number of clients being sampled alongside the learning curve to better understand when critical learning periods occur. \n- It is not clear when the algorithm predicts a critical learning period and if it is effective. From my understanding, it should have some dependency on the random seed used to select clients randomly. Experiments should be run on multiple seeds to check if the algorithm finds critical learning periods at different points in training.\n- The authors claim FedAvg is state-of-the-art which may not be true. Results on new algorithms like FedOPT[2] should be shown as well.\n- A major question is when the critical learning period occurs. Does it always occur in the early optimization stage? If so, is detecting CL necessary? I think then a simple learning rate scheduling algorithm could work.\n\n[1] Wang, Jianyu, et al. \"A field guide to federated optimization.\" arXiv preprint arXiv:2107.06917 (2021).\n\n[2] Reddi, Sashank, et al. \"Adaptive federated optimization.\" arXiv preprint arXiv:2003.00295 (2020).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jGQOvQi7vTT", "reviewer": "Reviewer_8m8J", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper leverages the concept of critical learning period for client selection in federated learning, with the purpose of improving communication efficiency while maintaining or even improving the learning effectiveness. Built based on a previous work that identified the existence of critical learning period in federated learning, this work proposes a light-weighted metric to detect the critical learning period in each round of federated model update, and adaptively change the number of selected clients for gradient update. Extensive experiment results on two benchmark datasets, several base models and federated learning algorithms demonstrated the promising performance gain of the proposed solution. ", "summary_of_the_review": "The provided solution is simple and intuitive, and its effectiveness is proved via an extensive set of experiments. However, this paper lacks necessary investigation of why the critical learning period happens (e.g., what’s the root cause), and therefore it is hard to justify whether proposed solution can completely address the problem. ", "strength_and_weaknesses": "Strength:\n+ The proposed new metric is very straightforward and easy to implement, which ensures its applicability in practice. \n+ The authors provided very extensive experimentation about the proposed solution under various settings, e.g., different degree of heterogeneity across clients, different federated learning algorithms, and various settings of its hyper-parameters. This provides a comprehensive collection of evidence about the effectiveness of the proposed solution.\n+ The manuscript is well-written and easy to follow.\n\nWeakness:\n- As the cause of critical learning period is unclear, it is hard to tell if the proposed solution can fundamentally solve the problem. Throughout the paper, including the related work section, there is no discussion about why the critical learning period is happening. Is it due to non-convex optimization? And why simply increasing the number of training instances will solve the problem? If it is due to non-convexity, the starting point also matters. The performed experiments seem to suggest heterogeneity is the main cause of critical learning period. But it is never explicitly articulated. Without any analysis about the mechanism of critical learning period, it is hard to justify why the proposed solution could solve the problem.\n- The paper argues that permanent damage in model training can be made when the critical learning period is not well handled, no matter how many more training instances could be provided later on. But as the proposed solution could not address the issue either: if it detects the current round is in the critical learning period, it only increases the number of clients for the next round; but the damage has already been made in this round. In addition, client sampling can also cause variance in FGN calculation, which can misclassify the critical learning period. A false negative detection would introduce the damage. And hence, it is still unclear why the proposed solution could be helpful.  \n\nQuestions:\n- Even under the proposed solution, the clients are still randomly sampled. Why do not we select the clients who can best help us circumvent the critical learning period? For example, find a set of clients whose local FGN is smallest?   \n- Although FedFIM is expensive to compute, it is important to compare how effective its identified critical learning period is for improving federated learning comparing to FGN’s. \n- Since the communication cost is defined by the total number of communication rounds, instead of communicated bits, why do not we start with all clients? And how would the hyper-parameter m affect the solution’s performance?\n- How would the solution perform if the environment is homogeneous? For example, when we uniformly distribute the dataset onto clients, what’s the performance gain against FedAvg?   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "HVSp813sT14", "reviewer": "Reviewer_29oH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a CL periods-aware federated learning framework to adaptively select the amount of clients.  \n", "summary_of_the_review": "The paper proposes a simple but effective framework to adaptively control the clients number in FL. The consideration of CL periods in FL is significant and the effectiveness of the proposed method is empirically evaluated. However, the idea of applying CL periods detection to distributed ML is a little plain, the proposed FGN metric is also intuitive. Consequently, I recommend this paper a borderline score. ", "strength_and_weaknesses": "Strength:\nS1: The idea of augmenting federated learning from the perspective of CL periods is easy-to-implement but effective.\nS2: The proposed FedCL framework is also simple and effective. It complements with state-of-the-art FL frameworks such as the popular FedAvg. Experimental results show FedCL improves the test accuracy with the same level of communication rounds.\n\nWeaknesses:\nW1: The novelty contribution of this paper could be limited. From the perspective of CL periods, the setting of FL has no intrinsic difference from the distributed machine learning, which has been studied in the cited references. Thus, although the idea of this paper is quite effective, it could be just a basic and straightforward solution.\nW2: The proposed FGN metric is somewhat intuitive. While the result shown in Figure 16 shows that FGN is more lightweight than existing FedFIM metric, the test accuracy improvement of these two metrics is not reported. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject