Title: Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization
Venue: NeurIPS 2023
Accepted: yes
Keywords: federated learning

local consistency

personalized initialization

excess risk
Abstract: Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the heterogeneous dataset.
  Previous works have implicitly studied that FL suffers from the "client-drift" problem, which is caused by the inconsistent optimum across local clients. However, till now it still lacks solid theoretical analysis to explain the impact of this local inconsistency. 
  To alleviate the negative impact of the "client drift" and explore its substance in FL, in this paper, we first design an efficient FL algorithm FedInit, which allows employing the personalized relaxed initialization state at the beginning of each local training stage. Specifically, FedInit initializes the local state by moving away from the current global state towards the reverse direction of the latest local state. This relaxed initialization helps to revise the local divergence and enhance the local consistency level.
  Moreover, to further understand how inconsistency disrupts performance in FL, we introduce the excess risk analysis and study the divergence term to investigate the test error of the proposed FedInit method. Our studies show that on the non-convex objectives, optimization error is not sensitive to this local inconsistency, while it mainly affects the generalization error bound in FedInit. 
  Extensive experiments are conducted to validate this conclusion. Our proposed FedInit could achieve state-of-the-art (SOTA) results compared to several advanced benchmarks without any additional costs. Meanwhile, stage-wise relaxed initialization could also be incorporated into the current advanced algorithms to achieve higher performance in the FL paradigm.

Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization

Abstract

Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the heterogeneous dataset. Previous works have implicitly studied that FL suffers from the 'client-drift' problem, which is caused by the inconsistent optimum across local clients. However, till now it still lacks solid theoretical analysis to explain the impact of this local inconsistency. To alleviate the negative impact of the 'client drift' and explore its substance in FL, in this paper, we first design an efficient FL algorithm FedInit , which allows employing the personalized relaxed initialization state at the beginning of each local training stage. Specifically, FedInit initializes the local state by moving away from the current global state towards the reverse direction of the latest local state. This relaxed initialization helps to revise the local divergence and enhance the local consistency level. Moreover, to further understand how inconsistency disrupts performance in FL, we introduce the excess risk analysis and study the divergence term to investigate the test error of the proposed FedInit method. Our studies show that optimization error is not sensitive to this local inconsistency, while it mainly affects the generalization error bound in FedInit . Extensive experiments are conducted to validate this conclusion. Our proposed FedInit could achieve state-of-the-art (SOTA) results compared to several advanced benchmarks without any additional costs. Meanwhile, stage-wise relaxed initialization could also be incorporated into the current advanced algorithms to achieve higher performance in the FL paradigm.

1 Introduction

Since McMahan et al. [26] developed federated learning, it becomes a promising paradigm to effectively make full use of the computational ability of massive edge devices. Kairouz et al. [17] further classify the modes based on the specific tasks and different environmental setups. Different from centralized training, FL utilizes a central server to coordinate the clients to perform several local training stages and aggregate local models as one global model. However, due to the heterogeneous dataset, it still suffers from significant performance degradation in practical scenarios.

Several previous studies explore the essence of performance limitations in FL and summarize it as the 'client-drift' problem [1, 19, 22, 38, 44, 46, 48]. From the perspective of the global target, Karimireddy et al. [19] claim that the aggregated local optimum is far away from the global optimum due to the heterogeneity of the local dataset, which introduces the 'client-drift' in FL. However, under limited local training steps, local clients can not genuinely approach the local optimum. To describe this negative impact more accurately, Acar et al. [1] and Wang et al. [44] point out that each locally optimized objective should be regularized to be aligned with the global objective. Moreover, beyond the guarantees of local consistent objective, Xu et al. [46] indicate that the performance

degradation could be further eliminated in FL if it guarantees the local consistent updates at each communication round, which is more similar to the centralized scenarios. These arguments intuitively provide forward-looking guidance for improving the performance in FL. However, in the existing analysis, there is still no solid theoretical support to understand the impact of the consistency term, which also severely hinders the further development of the FL paradigm.

To alleviate the negative impact of the 'client-drift' problem and strengthen consistency in the FL paradigm, in this paper, we take into account adopting the personalized relaxed initialization at the beginning of each communication round, dubbed FedInit method. Specifically, FedInit initializes the selected local state by moving away from the current global state towards the reverse direction of the current latest local state. Personalized relaxed initialization helps each local model to revise its divergence and gather together with each other during the local training process. This flexible approach is surprisingly effective in FL and only adopts a constant coefficient to control the divergence level of the initialization. It could also be easily incorporated as a plug-in into other advanced benchmarks to further improve their performance.

Moreover, to explicitly understand how local inconsistency disrupts performance, we introduce the excess risk analysis to investigate the test error of FedInit under the smooth non-convex objective, which includes an optimization error bound and a generalization error bound. Our theoretical studies indicate that the optimization error is insensitive to local inconsistency, while it mainly affects the generalization performance. Under PŁ -condition, consistency performs as the dominant term in the excess risk. Extensive empirical studies are conducted to validate the efficiency of the FedInit method. On the CIFAR-10 / 100 dataset, it could achieve SOTA results compared to several advanced benchmarks without additional costs. It also helps to enhance the consistency level in FL.

In summary, the main contributions of this work are stated as follows:

We propose an efficient and novel FL method, dubbed FedInit , which adopts the personalized relaxed initialization state on the selected local clients at each communication round. Relaxed initialization is dedicated to enhancing local consistency during training, and it is also a practical plug-in that could easily to incorporated into other methods.

One important contribution is that we introduce the excess risk analysis in the proposed FedInit method to understand the intrinsic impact of local consistency. Our theoretical studies prove that the optimization error is insensitive to consistency, while it mainly affects the test error and generalization error bound.

Extensive numerical studies are conducted on the real-world dataset to validate the efficiency of the FedInit method, which outperforms several SOTA benchmarks without additional training costs. Meanwhile, as an efficient plug-in, relaxed initialization ( FedInit ) could also help the other benchmarks in our paper to achieve higher performance with effortlessness.

2 Related Work

Consistency in FL. FL employs an enormous number of edge devices to jointly train a single model among the isolated heterogeneous dataset [17, 26]. As a standard benchmark, FedAvg [2, 26, 48] allows the local stochastic gradient descent (local SGD) [10, 23, 45] based updates and uniformly selected partial clients' participation to alleviate the communication bottleneck. The stage-wise local training processes lead to significant divergence for each client [5, 25, 43, 44]. To improve the efficiency of the FL paradigm, a series of methods are proposed. Karimireddy et al. [19] indicate that inconsistent local optimums cause the severe 'client drift' problem and propose the SCAFFOLD method which adopts the variance reduction [6, 16] technique to mitigate it. Li et al. [22] penalize the prox-term on the local objective to force the local update towards both the local optimum and the last global state. Zhang et al. [49] utilize the primal-dual method to improve consistency via solving local objectives under the equality constraint. Specifically, a series of works further adopt the alternating direction method of multipliers (ADMM) to optimize the global objective [1, 9, 41, 52], which could also enhance the consistency term. Beyond these, a series of momentum-based methods are proposed to strengthen local consistency. Wang et al. [42] study a global momentum update method to stabilize the global model. Further, Gao et al. [8] use a local drift correction via a momentum-based term to revise the local gradient, efficiently reducing inconsistency. Ozfatura et al. [28], Xu et al. [46], Sun et al. [39] propose a similar client-level momentum to force the local update towards the last global direction. A variant of client-level momentum that adopts the inertial momentum to further improve

the local consistency level [24, 40]. At present, improving the consistency in FL remains a very important and promising research direction. Though these studies involve the heuristic discussion on consistency, in this paper we focus on the personalized relaxed initialization.

Generalization in FL. A lot of works have studied the properties of generalization in FL. Based on the margin loss [3, 7, 27], Reisizadeh et al. [31] develop a robust FL paradigm to alleviate the distribution shifts across the heterogeneous clients. Shi et al. [32] study the efficient and stable model technique of model ensembling. Yagli et al. [47] prove the information-theoretic bounds on the generalization error and privacy leakage in the general FL paradigm. Qu et al. [29] propose to adopt the sharpness aware minimization (SAM) optimizer on the local client to improve the flatness of the loss landscape. Caldarola et al. [4], Sun et al. [37, 38], Shi et al. [33, 34] propose several variants based on SAM that could achieve higher performance. However, these works only focus on the generalization efficiency in FL, while in this paper we prove that its generalization error bound is dominated by consistency.

3 Methodology

3.1 Preliminaries

Under the cross-device FL setups, there are a very large number of local clients to collaboratively train a global model. Due to privacy protection and unreliable network bandwidth, only a fraction of devices are open-accessed at any one time [17, 29]. Therefore, we define each client stores a private dataset S i = { z j } where z j is drawn from an unknown unique distribution D i . The whole local clients constitute a set C = { i } where i is the index of each local client and |C| = C . Actually, in the training process, we expect to approach the optimum of the population risk F :

where F i ( w ) = E z j ∼D i F i ( w,z j ) is the local population risk. While in practice, we usually consider the empirical risk minimization of the non-convex finite-sum problem in FL as:

where f i ( w ) = 1 S i ∑ z j ∈S i f i ( w ; z j ) is the local empirical risk. In Section 4.1, we will analyze the difference between these two results. Furthermore, we introduce the excess risk analysis to upper bound the test error and further understand how consistency works in the FL paradigm.

3.2 Personalized Relaxed Initialization

In this part, we introduce the relaxed initialization in FedInit method. FedAvg proposes the localSGD-based implementation in the FL paradigm with a partial participation selection. It allows uniformly selecting a subset of clients N to participate in the current training. In each round, it initializes the local model as the last global model. Therefore, after each round, the local models are always far away from each other. The local offset w t -1 i,K -w t is the main culprit leading to inconsistency. Moreover, for different clients, their impacts vary with local heterogeneity. To alleviate this divergence, we propose the FedInit method which adopts the personalized relaxed initialization at the beginning of each round. Concretely, on the selected active clients, it begins the local training from a new personalized state, which moves away from the last global model towards the reverse direction from the latest local state (Line.6 in

10

11

12

13

14

Input:

model w , local model w i , T , K , β .

Output:

model

w

T

.

Algorithm 1: FedInit Algorithm

Input:

model w , local model w i , T , K , β .

Output:

model

w

T

.

1

Initialize states

: initialize

w

-

1

=

w

-

1

i,

0

=

w

0

.

2

for

t

= 0

,

1

, ..., T

-

1

do

3

randomly select active clients set

N

from

C

4

for

i

∈ N

in parallel

do

5

send the

w

t

to the active clients

6

set the

w

t

+

β

(

w

t

-

w

t

-

1

i,K

)

as

w

t

i,

0

7

for

k

= 0

,

1

, ..., K

-

1

do

8

compute gradient

g

t

i,k

at

w

t

i,k

9

w

t

i,k

+1

=

w

t

i,k

-

ηg

t

i,k

end

send the

w

t

i

=

w

t

i,K

to the server

end

w

t

+1

=

1

N

∑

i

∈N

w

t

i

end

Algorithm 1). A coefficient β is adopted to control the level of personality. This offset β ( w t -w t -1 i,K ) in the relaxed initialization (RI) provides a correction that could help local models gather together after the local training process. Furthermore, this relaxed initialization is irrelevant to the local optimizer, which means, it could be easily incorporated into other methods. Additionally, FedInit does not require extra auxiliary information to communicate. It is a practical technique in FL.

4 Theoretical Analysis

In this section, we first introduce the excess risk in FL which could provide a comprehensive analysis on the joint performance of both optimization and generalization. In the second part, we introduce the main assumptions adopted in our proofs and discuss them in different situations. Then we state the main theorems on the analysis of the excess risk of our proposed FedInit method.

4.1 Excess Risk Error in FL

Since Karimireddy et al. [19] pointed out that client-drift problem may seriously damage the performance in the FL paradigm, many previous works [15, 18, 19, 30, 36, 44, 46, 48] have learned its inefficiency in the FL paradigm. However, most of the analyses focus on the studies from the onefold perspective of optimization convergence but ignore investigating its impact on generality. To further provide a comprehensive understanding of how client-drift affects the performance in FL, we adopt the well-known excess risk in the analysis of our proposed FedInit method.

We denote w T as the model generated by FedInit method after T communication rounds. Compared with f ( w T ) , we mainly focus on the efficiency of F ( w T ) which corresponds to its generalization performance. Therefore, we analyze the E [ F ( w T )] from the excess risk E E as:

Generally, the E [ f ( w ∗ )] is expected to be very small and even to zero if the model could well-fit the dataset. Thus E E could be considered as the joint efficiency of the generated model w T . Thereinto, E G means the different performance of w T between the training dataset and the test dataset, and E O means the similarity between w T and optimization optimum w ⋆ on the training dataset.

4.2 Assumptions

In this part, we introduce some assumptions adopted in our analysis. We will discuss their properties and distinguish the proofs they are used in.

Assumption 1 For ∀ w 1 , w 2 ∈ R d , the non-convex local function f i satisfies L -smooth if:

Assumption 2 For ∀ w ∈ R d , the stochastic gradient is bounded by its expectation and variance as:

Assumption 3 For ∀ w ∈ R d , the heterogeneous similarity is bounded on the gradient norm as:

Assumption 4 For ∀ w 1 , w 2 ∈ R d , the global function f satisfies L G -Lipschitz if:

Assumption 5 For ∀ w ∈ R d , let w ⋆ ∈ arg min w f ( w ) , the function f satisfies PŁ-condition if:

Discussions. Assumptions 1 ∼ 3 are three general assumptions to analyze the non-convex objective in FL, which is widely used in the previous works [15, 18, 19, 30, 36, 44, 46, 48]. Assumption 4 is used to bound the uniform stability for the non-convex objective, which is used in [11, 51]. Different from the analysis in the margin-based generalization bound [27, 29, 31, 38] that focus on understanding how the designed objective affects the final generalization performance, our work focuses on understanding

how the generalization performance changes in the training process. We consider the entire training process and adopt uniform stability to measure the global generality in FL. For the general non-convex objective, one often uses the gradient norm E ∥∇ f ( w ) ∥ 2 instead of bounding the loss difference E [ f ( w T ) -f ( w ⋆ ) ] to measure the optimization convergence. To construct and analyze the excess risk, and further understand how the consistency affects the FL paradigm, we follow [51] to use Assumption 5 to bound the loss distance. Through this, we can establish a theoretical framework to jointly analyze the trade-off on the optimization and generalization in the FL paradigm.

4.3 Main Theorems

4.3.1 Optimization Error E O

Theorem 1 Under Assumptions 1 ∼ 3, let participation ratio is N/C where 1 < N < C , let the learning rate satisfy η ≤ min { N 2 CKL , 1 NKL } where K ≥ 2 , let the relaxation coefficient β ≤ √ 2 12 , and after training T rounds, the global model w t generated by FedInit satisfies:

where λ ∈ (0 , 1) , κ 1 = 1300 β 2 1 -72 β 2 + 17 , and κ 2 = 1020 β 2 1 -72 β 2 + 13 are three constants. Further, by selecting the proper learning rate η = O ( √ N KT ) and let D = f ( w 0 ) -f ( w ⋆ ) as the initialization bias, the global model w t satisfies:

Theorem 1 provides the convergence rate of the FedInit method without the PŁ-condition , which could achieve the O (1 / √ NKT ) with the linear speedup of N × . The dominant term of the training convergence rate is the heterogeneous bias G , which is K × larger than the initialization bias D and stochastic bias σ l . According to the formulation (10), by ignoring the initialization bias, the best local interval K = O ( σ 2 l /G 2 ) . This selection also implies that when G increases, which means the local heterogeneity increases, the local interval K is required to decrease appropriately to maintain the same efficiency. More importantly, though FedInit adopts a weighted bias on the initialization state at the beginning of each communication round, the divergence term E ∥ w t -1 i,K -w t ∥ 2 does not affect the convergence bound whether β is 0 or not. This indicates that the FL paradigm allows a divergence of local clients from the optimization perspective. Proof details are stated in Appendix A.2.3.

Theorem 2 Under Assumptions 1 ∼ 3 and 5, let participation ratio is N/C where 1 < N < C , let the learning rate satisfy η ≤ min { N 2 CKL , 1 NKL , 1 λµK } where K ≥ 2 , let the relaxation coefficient β ≤ √ 2 12 , and after training T rounds, the global model w t generated by FedInit satisfies:

where λ, κ 1 , κ 2 is defined in Theorem 1. Further, by selecting the proper learning rate η = O ( log( λµNKT ) λµKT ) and let D = f ( w 0 ) -f ( w ⋆ ) as the initialization bias, the global model w t satisfies:

To bound the E O term, we adopt Assumption 5 and prove that FedInit method could achieve the O (1 /NKT ) rate where we omit the O (log( NKT )) term. It maintains the properties stated in the Theorem 1. Detailed proofs of the convergence bound are stated in Appendix A.2.4.

4.3.2 Generalization Error E G

Uniform Stability. One powerful analysis of the generalization error is the uniform stability [11, 21, 50]. It says, for a general proposed method, its generalization error is always lower than the bound

of uniform stability. We assume that there is a new set ˜ C where C and ˜ C differ in at most one data sample on the i ⋆ -th client. Then we denote the w T and ˜ w T as the generated model after training T rounds on these two sets, respectively. Thus, we have the following lemma:

Lemma 1 (Uniform Stability. [11] ) For the two models w T and ˜ w T generated as introduced above, a general method satisfies ϵ -uniformly stability if:

Moreover, if a general method satisfies ϵ -uniformly stability, then its generalization error satisfies E G ≤ sup z j ∼{D i } E [ f ( w T ; z j ) -f ( ˜ w T ; z j )] ≤ ϵ [50] .

Theorem 3 Under Assumptions 1, 2, 4, and 5, let all conditions above be satisfied, let learning rate η = O ( 1 KT ) = c T where c = µ 0 K is a constant, and let |S i | = S as the number of the data samples, by randomly selecting the sample z , we can bound the uniform stability of our proposed FedInit as:

where U is a constant and ∆ t = 1 C ∑ i ∈C E ∥ w t -1 i,K -w t ∥ 2 is the divergence term at round t .

For the generalization error, Theorem 3 indicates that E G term contains two main parts. The first part comes from the stochastic gradients as the vanilla centralized training process [11], which is of the order O (( TK ) cL 1+ cL /S ) . The constant c is of the order O (1 /K ) as c = µ 0 K , thus we have cL 1+ cL = µ 0 L K + µ 0 L . If we assume the µ 0 L is generally small, we always expect to adopt a larger K in the FL paradigm to reduce generalization error. For instance, if we select K → ∞ , then O (( TK ) cL 1+ cL /S ) →O ( T cL 1+ cL /S ) which is a very strong upper bound of the generalization error. However, the selection of local interval K must be restricted from the optimization conditions and we will discuss the details in Section 4.3.4. In addition, the second part in Theorem 3 comes from the divergence term, which is a unique factor in the FL paradigm. As we mentioned above, the divergence term measures the authentic client-drift in the training process. The divergence term is not affected by the number of samples S and it is only related to the proposed method and the local heterogeneity of the dataset. Proof details are stated in Appendix A.3.

4.3.3 Divergence Term

In the former two parts, we provide the complete theorem to measure optimization error E O and generalization error E G . And we notice that, in the FL paradigm, the divergence term mainly affects the generalization ability of the model instead of the optimization convergence. In this part, we focus on the analysis of the divergence term of our proposed FedInit method. Due to the relaxed initialization at the beginning of each communication round, according to the Algorithm 1, we have w t i,K = w t + β ( w t -w t -1 i,K ) -η ∑ K -1 k =0 g t i,k . Thus, we have the following recursive relationship:

According to the formulation (15), we can bound the divergence ∆ t via the following two theorems. Theorem 4 Under Assumptions 1 ∼ 3, we can bound the divergence term as follows. Let the learning rate satisfy η ≤ min { N 2 CKL , 1 NKL , √ N √ CKL } where K ≥ 2 , and after training T rounds, let 0 < β < √ 6 24 , the divergence term { ∆ t } generated by FedInit satisfies:

Theorem 4 points out the convergence order of the divergence ∆ t generated by FedInit during the training process. This bound matches the conclusion in Theorem 1 with the same learning rate. The

dominant term achieves the O ( NK/T ) rate on the heterogeneity bias G . It could be seen that the number of selected clients N will inhibit its convergence and the local consistency linearly increases with N . Different from the selection in Theorem 1, local interval K is expected as small enough to maintain the high consistency. Also, the initialization bias D is no longer dominant in consistency. We omit the constant weight 1 1 -96 β 2 in this upper bound. Proof details are stated in Appendix A.2.5.

Theorem 5 Under Assumptions 1 ∼ 3 and 5, we can bound the divergence term as follows. Let the learning rate satisfy η ≤ min { N 2 CKL , 1 NKL , 1 λµK } where K ≥ 2 , and after training T rounds, let 0 < β < √ 6 , the divergence term ∆ T generated by FedInit satisfies:

Theorem 5 indicates the convergence of the divergence ∆ under the PŁ-condition which matches the conclusion in Theorem 2 with the same learning rate selection. Assumption 5 establishes a relationship between the gradient norm and the loss difference on the non-convex function f . Different from the Theorem 4, the initialization bias D and the heterogeneous bias G are the dominant terms. Under Assumption 5, the FedInit supports a larger local interval K in the training process. This conclusion also matches the selection of K in Theorem 2. When the model converges, FedInit guarantees the local models towards the global optimum under at least O (1 /T 2 ) rate. Similarly, we omit the constant weight 1 1 -96 β 2 and we will discuss the β in Section 4.3.4. Proof details are stated in Appendix A.2.6.

4.3.4 Excess Risk

In this part, we analyze the excess risk E E of FedInit method. According to the theorems above,

Theorem 6 Under Assumption 1 ∼ 5, let the participation ratio is N/C where 1 < N < C , let the learning rate satisfies η ≤ min { N 2 CKL , 1 NKL , 1 λµK , } where K ≥ 2 , let the relaxed coefficient 0 ≤ β < √ 6 24 , and let |S i | = S . By selecting the learning rate η = O ( log( λµNKT ) λµKT ) ≤ c t , after training T communication rounds, the excess risk of the FedInit method achieves:

According to the Theorems 2, 3, and 5, we combine their dominant terms to upper bound the excess risk of FedInit method. The first term comes from the optimization error, the second term comes from the stability bias, and the third term comes from the divergence bias. From the perspective of excess risk, the main restriction in the FL paradigm is the divergence term with the bound of ˜ O ( 1 T 1 1+ cL ) .

The second term of excess risk matches the conclusion in SGD [11, 51] which relies on the number S . Our analysis of the excess risk reveals two important corollaries in FL:

From the perspective of optimization, the FL paradigm is insensitive to local consistency in the training process (Theorems 1&2).

From the perspective of generalization, the local consistency level significantly affects the performance in the FL paradigm (Theorem 6).

Then we discuss the best selection of the local interval K and relaxed coefficient β .

Selection of K. In the first term, to minimize the optimization error, the local interval K is required to be large enough. In the second term, since cL 1+ cL ≤ 1 , the upper bound expects a small local interval K . In the third term, since 1 1+ cL = K K + µ 0 L < 1 , it expects a large K to guarantee the order of T to approach O (1 /T ) , where the divergence bias could maintain a high-level consistency. Therefore, there is a specific optimal constant selection for K > 1 to minimize the excess risk.

Selection of β . As the dominant term, the coefficient of the divergence bias also plays a key role in the error bound. In Theorem 5, the constant weight we omit for the divergence term ∆ T is 1 1 -96 β 2 . Thus the coefficient of √ ∆ T is 1 √ 1 -96 β 2 . Combined with Theorem 3, we have the coefficient for the

24

divergence term in formulation (18) is (1+ β ) 1 βcL √ 1 -96 β 2 . Therefore, to minimize this term, there is a specific optimal constant selection for 0 < β < √ 6 24 . We validate their selections in Section 5.2.

5 Experiments

In this part, we introduce our empirical studies. Due to the page limitations, the details of the dataset, hyperparameters selection, implementation, and some extra ablation studies are stated in Appendix B.

Benchmarks. Our selected benchmarks in this paper are stated as follows. FedAvg [26] proposes the general FL paradigm. FedAdam [30] studies the efficiency of adaptive optimizer in FL. SCAFFOLD [19], FedDyn [1], and FedCM [46] learn the 'client-drift" problem and adopt the variance reduction technique, ADMM, and client-level momentum respectively in FL to alleviate its negative impact. FedSAM [29] uses the local SAM objective instead of the vanilla empirical risk objective to search for a smooth loss landscape, which focuses on the generalization performance.

Setups. Here we briefly introduce the setups in our experiments. We test our proposed FedInit on the CIFAR-10 / 100 dataset [20]. To generate local heterogeneity, we follow Hsu et al. [14] to split the local clients through the Dirichlet sampling via a coefficient D r to control the heterogeneous level and follow Sun et al. [38] to adopt the sampling with replacement to enhance the heterogeneity level. We test on the ResNet-18-GN [12, 13] and VGG-11 [35] to validate its efficiency. Actually, when the heterogeneity is strong, the performance of personalized initialization will be better. To better demonstrate the performance of our proposed method, we add additional noises to the dataset. Specifically, we first introduce the client-based biases . Among clients, we assume that the data samples are obtained differently. Because the local dataset is private and its construction is unknown, i.e., they are collected from different machines or cameras. Therefore, we change the strength of the RGB channels with a random Gaussian noise for different clients. The second noise is the category-based biases . We assume that samples for each category also contain heterogeneity. In our experiments, we add different brightness perturbations to the samples in each category by a random Gaussian noise. Based on these two noises, the heterogeneity of the local dataset is significantly enlarged. In this more realistic dataset, we can clearly observe the performance of each algorithm.

For each benchmark in our experiments, we adopt two coefficients D r = 0 . 1 and 0 . 6 for each dataset to generate different heterogeneity. We generally select the local learning rate η = 0 . 1 and global learning rate η = 1 on all setups except for FedAdam we use 0 . 1 . The learning rate decay is set as multiplying 0 . 998 per round except for FedDyn we use 0 . 999 . We train 500 rounds on CIFAR-10 and 800 rounds on CIFAR-100 to achieve stable test accuracy. The participation ratios are selected as 10% and 5% respectively of total 100 and 200 clients. More details are stated in Appendix B.1.

5.1 Experiment results

In this part, we mainly introduce the experiment results compared with the other benchmarks.

In Table 1, our proposed FedInit method performs well than the other benchmarks with good stability across different experimental setups. On the results of ResNet-18-GN on CIFAR10, it achieves about 3.42 % improvement than the vanilla FedAvg on the high heterogeneous splitting with D r = 0 . 1 . When the participation ratio decreases to 5% , the accuracy drops only about 0 . 1% while FedAvg drops almost 1 . 88% . Similar results on CIFAR-100, when the ratio de-

creases, FedInit still achieves 43 . 77% while the second best method SCAFFOLD drops about 3 . 21% . This indicates the proposed FedInit holds good stability on the varies of the participation. In addition, in Table 2, we incorporate the relaxed initialization (RI) into the other benchmarks to test its benefit. '-" means the vanilla benchmarks, and '+RI" means adopting the relaxed initialization. It shows that the relaxed initialization holds the promising potential to further enhance the performance. Actually, FedInit could be considered as (RI + FedAvg ), whose improvement achieves about over 3% on each setup. Table 1 shows the poor performance of the vanilla FedAvg . Nevertheless, when adopting the RI, FedInit remains above most benchmarks on several setups. When the RI is incorporated into other benchmarks, it helps them to achieve higher performance without additional communication costs.

5.2 Ablation

In this part, we mainly introduce the ablation results of different hyperparameters.

Hyperparameters Sensitivity. The excess risk and test error of FedInit indicate there exists best selections for local interval K and relaxed coefficient β , respectively. In this part, we test a series of selections to validate our conclusions. To be aligned with previous studies, we denote K as training epochs. In Figure 1 (a), we can see that the selection range of the beta is very small while it has great potential to improve performance. When it is larger than the threshold, the training process will diverge quickly. As local interval K increases, test accuracy rises first

1

2

3

5

10

20

local interval

K

72

74

76

78

80

82

test accuracy

Performance of different

K

Dirichlet-0.1

Dirichlet-0.6

(a) Different

K

.

-0.2

-0.1

0

0.01

0.02

0.05

0.1

0.15

relaxed coefficient

45

50

55

60

65

70

75

80

85

test accuracy

Performance of different

Dirichlet-0.1

Dirichlet-0.6

(b) Different

β

.

and then decreases. Our analysis provides a clear explanation of the phenomenon. The optimization error decreases as K increases when it is small. When K exceeds the threshold, the divergence term in generalization cannot be ignored. Therefore, the test accuracy will be significantly affected.

Consistency. In this part, we test the relationship between the test accuracy and divergence term ∆ T under different β selections. As introduced in Algorithm 1 Line.6, negative β means to adopt the relaxed initialization which is close to the latest local model. FedInit degrades to FedAvg when β = 0 . Table 3 validates that RI is required to

be far away from the local model (a positive β ). When β is small, the correction is limited. The local divergence term is difficult to be diminished efficiently. While it becomes too large, the local training begins from a bad initialization, which can not receive enough guidance of global information from the global models. Furthermore, as shown in Table 3, if the initialization is too far from the local model, the quality of the initialization state will not be effectively guaranteed.

5.3 Discussions of Relaxed Initialization

In this part, we mainly discuss the improvements of the proposed relaxed initialization.

In vanilla classical FedAvg and the most advanced methods, at the beginning of each communication round, we are always caught in a misunderstanding of the high consistency. Because the target of FL is a globally consistent solution, it is always an involuntary aggregation in the algorithm to ensure consistency. We prove that this does contribute to the efficiency of the optimization process, but it is not the best selection for generalization. To better improve the generalization, we prove that a relaxed initialization state will contribute more. We compare their difference in Figure 5.3.

global model w(t)

global model w(t+1)

local optimum 1

local optimum 2

global optimum

local relaxed Init. 1

local relaxed Init. 2

FedAvg

FedInit

As shown in the above figure, we can clearly see why FedInit contributes more to the consistency. When we move a little in the opposite direction of the last local optimization state, we will move further away from local optimal solutions in the current communication round. The working mode of RI is similar to the idea of "lookahead". Differently, (1) "lookahead" only works at the end of each stage; (2) "lookahead" only works for the global models on the global server. However, RI helps each local client to backtrack a small distance at the beginning of each stage. Therefore, after the local training in the next stage, the trained local models will get closer to each other than before.

6 Conclusion

In this work, we propose an efficient and novel FL method, dubbed FedInit , which adopts the stagewise personalized relaxed initialization to enhance the local consistency level. Furthermore, to clearly understand the essential impact of consistency in FL, we introduce the excess risk analysis in FL and study the divergence term. Our proofs indicate that consistency dominates the test error and generalization error bound while optimization error is insensitive to it. Extensive experiments are conducted to validate the efficiency of relaxed initialization. As a practical and light plug-in, it could also be easily incorporated into other FL paradigms to improve their performance.

Limitations & Broader Impact. In this work, we analyze the excess risk for the FedInit method to understand how consistency works in FL. Actually, the relaxed initialization may also work for the personalized FL (pFL) paradigm. It is a future study to explore its properties in the pFL and decentralized FL, which may inspire us to design novel efficient algorithms in the FL community.

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263 , 2021.

Muhammad Asad, Ahmed Moustafa, and Takayuki Ito. Fedopt: Towards communication efficiency and privacy preservation in federated learning. Applied Sciences , 10(8):2864, 2020.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by seeking flat minima. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII , pages 654-672. Springer, 2022.

Zachary Charles and Jakub Koneˇ cn` y. Convergence and accuracy trade-offs in federated learning and meta-learning. In International Conference on Artificial Intelligence and Statistics , pages 2575-2583. PMLR, 2021.

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems , 27, 2014.

Farzan Farnia, Jesse M Zhang, and David Tse. Generalizable adversarial training via spectral normalization. arXiv preprint arXiv:1811.07457 , 2018.

Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Feddc: Federated learning with non-iid data via local drift decoupling and correction. arXiv preprint arXiv:2203.11751 , 2022.

Yonghai Gong, Yichuan Li, and Nikolaos M Freris. Fedadmm: A robust federated deep learning framework with adaptivity to system heterogeneity. In 2022 IEEE 38th International Conference on Data Engineering (ICDE) , pages 2575-2587. IEEE, 2022.

Eduard Gorbunov, Filip Hanzely, and Peter Richtárik. Local sgd: Unified theory and new efficient methods. In International Conference on Artificial Intelligence and Statistics , pages 3556-3564. PMLR, 2021.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225-1234. PMLR, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning , pages 4387-4398. PMLR, 2020.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

Tiansheng Huang, Li Shen, Yan Sun, Weiwei Lin, and Dacheng Tao. Fusion of global and local knowledge for personalized federated learning. arXiv preprint arXiv:2302.11051 , 2023.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems , 26, 2013.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Belhal Karimi, Ping Li, and Xiaoyun Li. Layer-wise and dimension-wise locally adaptive federated learning. arXiv preprint arXiv:2110.00532 , 2021.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR, 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Ilja Kuzborskij and Christoph Lampert. Data-dependent stability of stochastic gradient descent. In International Conference on Machine Learning , pages 2815-2824. PMLR, 2018.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local sgd. arXiv preprint arXiv:1808.07217 , 2018.

Yixing Liu, Yan Sun, Zhengtao Ding, Li Shen, Bo Liu, and Dacheng Tao. Enhance local consistency in federated learning: A multi-step inertial momentum approach. arXiv preprint arXiv:2302.05726 , 2023.

Grigory Malinovskiy, Dmitry Kovalev, Elnur Gasanov, Laurent Condat, and Peter Richtarik. From local sgd to local fixed-point methods for federated learning. In International Conference on Machine Learning , pages 6692-6701. PMLR, 2020.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrallynormalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564 , 2017.

Emre Ozfatura, Kerem Ozfatura, and Deniz Gündüz. Fedadc: Accelerated federated learning with drift control. In 2021 IEEE International Symposium on Information Theory (ISIT) , pages 467-472. IEEE, 2021.

Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning , pages 18250-18280. PMLR, 2022.

Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cn` y, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295 , 2020.

Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, and Ali Jadbabaie. Robust federated learning: The case of affine distribution shifts. Advances in Neural Information Processing Systems , 33:21554-21565, 2020.

Naichen Shi, Fan Lai, Raed Al Kontar, and Mosharaf Chowdhury. Fed-ensemble: Improving generalization through model ensembling in federated learning. arXiv preprint arXiv:2107.10663 , 2021.

Yifan Shi, Yingqi Liu, Kang Wei, Li Shen, Xueqian Wang, and Dacheng Tao. Make landscape flatter in differentially private federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24552-24562, 2023.

Yifan Shi, Li Shen, Kang Wei, Yan Sun, Bo Yuan, Xueqian Wang, and Dacheng Tao. Improving the model consistency of decentralized federated learning. arXiv preprint arXiv:2302.04083 , 2023.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Hao Sun, Li Shen, Qihuang Zhong, Liang Ding, Shixiang Chen, Jingwei Sun, Jing Li, Guangzhong Sun, and Dacheng Tao. Adasam: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks. arXiv preprint arXiv:2303.00565 , 2023.

Yan Sun, Li Shen, Shixiang Chen, Liang Ding, and Dacheng Tao. Dynamic regularized sharpness aware minimization in federated learning: Approaching global consistency and smooth landscape. arXiv preprint arXiv:2305.11584 , 2023.

Yan Sun, Li Shen, Tiansheng Huang, Liang Ding, and Dacheng Tao. Fedspeed: Larger local interval, less communication round, and higher generalization accuracy. arXiv preprint arXiv:2302.10429 , 2023.

Yan Sun, Li Shen, Hao Sun, Liang Ding, and Dacheng Tao. Efficient federated learning via local adaptive amended optimizer with linear speedup. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In AAAI Conference on Artificial Intelligence , volume 1, 2022.

Han Wang, Siddartha Marella, and James Anderson. Fedadmm: A federated primal-dual algorithm allowing partial participation. In 2022 IEEE 61st Conference on Decision and Control (CDC) , pages 287-294. IEEE, 2022.

Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving communicationefficient distributed sgd with slow momentum. arXiv preprint arXiv:1910.00643 , 2019.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems , 33:7611-7623, 2020.

Jianyu Wang, Zheng Xu, Zachary Garrett, Zachary Charles, Luyang Liu, and Gauri Joshi. Local adaptivity in federated learning: Convergence and consistency. arXiv preprint arXiv:2106.02305 , 2021.

Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems , 33:6281-6292, 2020.

Jing Xu, Sen Wang, Liwei Wang, and Andrew Chi-Chih Yao. Fedcm: Federated learning with client-level momentum. arXiv preprint arXiv:2106.10874 , 2021.

Semih Yagli, Alex Dytso, and H Vincent Poor. Information-theoretic bounds on the generalization error and privacy leakage in federated learning. In 2020 IEEE 21st International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) , pages 1-5. IEEE, 2020.

Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. arXiv preprint arXiv:2101.11203 , 2021.

Xinwei Zhang, Mingyi Hong, Sairaj Dhople, Wotao Yin, and Yang Liu. Fedpd: A federated learning framework with adaptivity to non-iid data. IEEE Transactions on Signal Processing , 69:6055-6070, 2021.

Yikai Zhang, Wenjia Zhang, Sammy Bald, Vamsi Pingali, Chao Chen, and Mayank Goswami. Stability of sgd: Tightness analysis and improved bounds. In Uncertainty in Artificial Intelligence , pages 2364-2373. PMLR, 2022.

Pan Zhou, Hanshu Yan, Xiaotong Yuan, Jiashi Feng, and Shuicheng Yan. Towards understanding why lookahead generalizes better than sgd and beyond. Advances in Neural Information Processing Systems , 34:27290-27304, 2021.

Shenglong Zhou and Geoffrey Ye Li. Federated learning via inexact admm. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

A Proofs

In this section, we introduce our proofs of the main theorems in the main context. In the first part, we introduce some assumptions used in our proofs and point out their functions used for which part. In the second part, we prove the convergence rate and optimization error under the general assumptions. In the third part, we prove the uniform stability to measure the generalization error and analyze how each term affects the accuracy.

We suppose there are C clients participating in the training process and each has a local heterogeneous dataset. In each round t , we randomly select N clients to send the global model and they will train K iterations to get N local models. The local models will be aggregated on the global server as the next global model. After T rounds, our method generates a global model as the final state. We denote the total client set as C and the selected client set as N .

A.1 Assumptions

In this part, we state assumptions in our proofs and discuss them. We will introduce each assumption and develop their corollaries.

Assumption 6 For ∀ w 1 , w 2 ∈ R d , the non-convex local function f i satisfies L -smooth if:

where L is a universal constant.

Assumption 7 For ∀ w ∈ R d , the stochastic gradient is bounded by its expectation and variance as:

where σ l > 0 is a universal constant.

Assumption 8 For ∀ w ∈ R d , the heterogeneous similarity is bounded on the gradient norm as:

where G ≥ 0 and B ≥ 1 are two universal constants.

Assumption 9 For ∀ w 1 , w 2 ∈ R d , the global function f satisfies L G -Lipschitz if:

where L G is a universal constant.

Assumption 10 For ∀ w ∈ R d , let w ⋆ ∈ arg min w f ( w ) , the global function satisfies PŁ-condition if:

where µ is a universal positive constant.

Discussion. Assumption 6 ∼ 8 are three general assumptions to analyze the non-convex objective in FL, which is widely used in the previous works [15, 18, 19, 30, 36, 44, 46, 48]. Assumption 9 is used to bound the uniform stability for the non-convex objective, which is used in [11, 51]. Different from the analysis in the margin-based generalization bound [27, 29, 31, 38] that focus on understanding how the designed objective affects the final generalization performance, our work focuses on understanding how the generalization performance changes in the training process. We consider the entire training process and adopt uniform stability to measure the global generality in FL and theoretically study the importance of consistency to FL. For the general non-convex objective, one often uses the gradient norm E ∥∇ f ( w ) ∥ 2 instead of the loss difference E [ f ( w ⋆ ) -f ( w )] to measure the training error. To construct and analyze the excess risk to further understand how the consistency affects the FL paradigm, we follow [51] to use Assumption 10 to bound the loss distance. Through this, we can establish a theoretical framework to jointly analyze the trade-off on the optimization and generalization in the FL paradigm.

A.2 Proofs for the Optimization Error

In this part, we prove the training error for our proposed method. We assume the objective function f ( w ) = 1 C ∑ i ∈C f i ( w ) is L -smooth w.r.t w . Then we could upper bound the training error in the FL. Some useful notations in the proof are introduced in the Table 4.

Then we introduce some important lemmas used in the proof.

A.2.1 Important Lemmas

Lemma 2 (Bounded local updates) We first bound the local training updates in the local training. Under the Assumptions stated, the averaged norm of the local updates of total C clients could be bounded as:

Proof V 1 measures the norm of the local offset during the local training stage. It could be bounded by two major steps. Firstly, we bound the separated term on the single client i at iteration k as:

where the learning rate is required η ≤ √ 2 4( K -1) L for K ≥ 2 .

Computing the average of the separated term on client i , we have:

Unrolling the aggregated term on iteration k ≤ K . When local interval K ≥ 2 , ( 1 + 1 K -1 ) k ≤

Summing the iteration on k = 0 , 1 , · · · , K -1 ,

This completes the proof.

Lemma 3 (Bounded global updates) The norm of the global update could be bounded by uniformly sampling. Under assumptions stated above, let η ≤ 1 KL , the norm of the global update of selected N clients could be bounded as:

Proof V 2 measures the variance of the global offset after each communication round. We define an indicator function I event = 1 if the event happens. Then, to bound it, we firstly split the expectation term:

To bound the second term, we can adopt the following equation. For the vector x i ∈ R d , we have:

̸

̸

We firstly bound the first term in the above equation. Taking x i = ∑ K -1 k =0 η ∇ f i ( w t i,k ) + β ( w t -w t -1 i,K ) into E ∥ ∑ i ∈C x i ∥ 2 , we have:

̸

Then we bound the second term in above equation. Taking x i = ∑ K -1 k =0 η ∇ f i ( w t i,k ) + β ( w t -w t -1 i,K ) into ∑ i ∈C E ∥ x i ∥ 2 , we have:

1

3

Cη

KL

V

+3

Cβ

∆

+3

Cη

K

G

+3

Cη

K

B

E

∥∇

f

(

w

)

∥

.

We bound all the components in V t 2 term. Let 1 ≤ N < C , to generate the final bound, summarizing the inequalities all above and adopting the bounded V t 1 in Lemma 2, then we have:

To minimize the coefficients of each term, we can select a constant order for the term η 2 K 2 L 2 . For convenience, we directly select the η 2 K 2 L 2 ≤ 1 which requires the learning rate η ≤ 1 KL . This completes the proof.

Lemma 4 (Bounded divergence term) The divergence term ∆ t could be upper bounded by the local update rules. According to the relaxed initialization in our method, under assumptions stated above, let the learning rate satisfy η ≤ 1 KL and the relaxed coefficient satisfy β ≤ √ 2 12 , the divergence term ∆ t could be bounded as the recursion of:

Proof The divergence term measures the inconsistency level in the FL framework. According to the local updates, we have the following recursive formula:

By taking the squared norm and expectation on both sides, we have:

The second term in the above inequality is V 2 we have bounded in lemma 3. Then we bound the stochastic gradients term. We have:

Taking the average on client i , we have:

Recalling the condition of η ≤ 1 KL and combining this and the squared norm inequality, we have:

Then we multiply the 1 1 -72 β 2 on both sides, which completes the proof.

A.2.2 Expanding the Smoothness Inequality for the Non-convex Objective

For the non-convex and L -smooth function f , we firstly expand the smoothness inequality at round t as:

[ f ( w t +1 ) -f ( w t )]

According to Lemma 2 and lemma 3 to bound the V t 1 and V t 2 , we can get the following recursive formula:

t

+1

t

Here we make a comprehensive discussion on the selection of η to simplify the above formula. In fact, in lemma 2, there is a constraint on the learning rate as η ≤ √ 2 4( K -1) L for K ≥ 2 . In lemma 3 and lemma 4, there is a constraint on the learning rate as η ≤ 1 KL . To further minimize the coefficient, we select the NηKL to be constant order. For convenience, we directly select the η ≤ 1 NKL . Thus, we have:

Firstly, to remove the gradient term, we follow the [19, 48] and let 30 β 2 η 2 L CN 2 (1 -72 β 2 ) + η 2 L 2 CN -η 2 C 2 K ≤ 0 , then learning rate η ≤ N 2 CKL . Then, according to the [48], there is a positive constant λ ∈ (0 , 1) to satisfy 1 -39 ηKLB 2 N -3900 β 2 ηKLB 2 (1 -72 β 2 ) N -12 η 2 K 2 L 2 B 2 > λ > 0 . Wedenote κ 1 = 1300 β 2 1 -72 β 2 +17 and κ 2 = 1020 β 2 1 -72 β 2 +13 as two constants in the formula. Therefore, we have:

A.2.3 Proof of Theorem 1

Theorem 7 Under Assumption 6 ∼ 8, let participation ratio is N/C where 1 < N < C , let the learning rate satisfy η ≤ min { N 2 CKL , 1 NKL } where K ≥ 2 , let the relaxation coefficient β ≤ √ 2 12 , and after training T rounds, the global model w t generated by FedInit satisfies:

Further, by selecting the proper learning rate η = O ( √ N KT ) and let D = f ( w 0 ) -f ( w ⋆ ) as the initialization bias, the global model w t satisfies:

Proof According to the expansion of the smoothness inequality, we have:

Taking the accumulation from 0 to T -1 , we have:

We select the learning rate η = O ( √ N KT ) and let D = f ( w 0 ) -f ( w ⋆ ) as the initialization bias, then we have:

This completes the proof.

A.2.4 Proof of Theorem 2

Theorem 8 Under Assumption 6 ∼ 8 and 10, let participation ratio is N/C where 1 < N < C , let the learning rate satisfy η ≤ min { N 2 CKL , 1 NKL , 1 λµK } where K ≥ 2 , let the relaxation coefficient β ≤ √ 2 12 , and after training T rounds, the global model w t generated by FedInit satisfies:

Further, by selecting the proper learning rate η = O ( log( λµNKT ) λµKT ) and let D = f ( w 0 ) -f ( w ⋆ ) as the initialization bias, the global model w t satisfies:

Proof According to the expansion of the smoothness inequality, we have:

According to Assumption 10, we have 2 µ ( f ( w ) -f ( w ⋆ )) ≤ ∥∇ f ( w ) ∥ 2 , we have:

Combining the terms aligned with w t and w t +1 , we have:

Taking the recursion from t = 0 to T -1 and let learning rate η ≤ 1 λµK , we have:

We select the learning rate η = O ( log( λµNKT ) λµKT ) and let D = f ( w 0 ) -f ( w ⋆ ) as the initialization bias, then we have:

This completes the proof.

A.2.5 Proof of Theorem 4

Theorem 9 Under Assumption 6 ∼ 8, we can bound the divergence term as follows. Let the learning rate satisfy η ≤ min { N 2 CKL , 1 NKL , √ N √ CKL } where K ≥ 2 , and after training T rounds, let 0 < β < √ 6 24 , the divergence

term ∆ t generated by FedInit satisfies:

Proof According to Lemma 4, we have:

Here we further bound the gradient term, we have:

Combining this into the recursive formulation, and let the learning rate satisfy η ≤ √ N √ CKL , we have:

Let 96 β 2 < 1 as the decayed coefficient where β < √ 6 24 , similar as Lemma 4, we have:

by taking the accumulation from t = 0 to T -1 ,

The same, the learning rate is selected as η = O ( √ N KT ) and let D = f ( w 0 ) -f ( w ⋆ ) as the initialization bias and let 96 β 2 < 1 , thus we have:

This completes this proof.

A.2.6 Proof of Theorem 5

Theorem 10 Under Assumption 6 ∼ 8 and 10, we can bound the divergence term as follows. Let the learning rate satisfy η ≤ min { N 2 CKL , 1 NKL , 1 λµK } where K ≥ 2 , and after training T rounds, let 0 < β < √ 6 24 , the divergence term ∆ T generated by FedInit satisfies:

Proof According to the Theorem 8, we have:

Taking the recursive formulation from t = 0 to T -1 , we have:

According to the Theorem 11, we have:

Let 96 β 2 ≤ e -λµηK , thus we have:

Thus selecting the same learning rate η = O ( log( λµNKT ) λµKT ) and let D = f ( w 0 ) -f ( w ⋆ ) as the initialization bias, we have:

This completes the proof.

A.3 Proofs for the Generalization Error

In this part, we prove the generalization error for our proposed method. We assume the objective function f is L -smooth and L G -Lipschitz as defined in [11, 51]. We follow the uniform stability to upper bound the generalization error in the FL.

We suppose there are C clients participating in the training process as a set C = { i } C i =1 . Each client has a local dataset S i = { z j } S j =1 with total S data sampled from a specific unknown distribution D i . Now we define a re-sampled dataset ˜ S i which only differs from the dataset S i on the j ⋆ -th data. We replace the S i ⋆ with ˜ S i ⋆ and keep other C -1 local dataset, which composes a new set ˜ C . From the perspective of total data, C only differs from the ˜ C at j ⋆ -th data on the i ⋆ -th client. Then, based on these two sets, our method could generate two output models, w t and ˜ w t respectively, after t training rounds. We first introduce some notations used in the proof of the generalization error.

Then we introduce some important lemmas in our proofs.

A.3.1 Important Lemmas

Lemma 5 (Lemma 3.11 in [11]) We follow the definition in [11, 51] to upper bound the uniform stability term after each communication round in FL paradigm. Different from their vanilla calculations, FL considers the finite-sum function on heterogeneous clients. Let non-negative objective f is L -smooth and L G -Lipschitz. After training T rounds on C and ˜ C , our method generates two models w T +1 and ˜ w T +1 respectively. For each data z and every t 0 ∈ { 1 , 2 , 3 , · · · , S } , we have:

Proof Let ξ = 1 denote the event ∥ w t 0 -˜ w t 0 ∥ = 0 and U = sup w,z f ( w ; z ) , we have:

Before the j ⋆ -th data on i ⋆ -th client is sampled, the iterative states are identical on both C and ˜ C . Let ˜ j is the index of the first different sampling, if ˜ j > t 0 , then ξ = 1 hold for t 0 . Therefore, we have:

where ˜ j is uniformly selected. This completes the proof.

Lemma 6 (Lemma 1.1 in [51]) Different from their calculations, we prove the similar inequalities on f in the stochastic optimization. Let non-negative objective f is L -smooth w.r.t w . The local updates satisfy w t i,k +1 = w t i,k -ηg t i,k on C and ˜ w t i,k +1 = ˜ w t i,k -η ˜ g t i,k on ˜ C . If at k -th iteration on each round, we sample the same data in C and ˜ C , then we have:

Proof In each round t , by the triangle inequality and omitting the same data z , we have:

The final inequality adopts assumptions of E ∥ g t i,k - ∇ f i ( w t i,k ) ∥ ≤ √ E ∥ g t i,k -∇ f i ( w t i,k ) ∥ 2 ≤ σ l . This completes the proof.

Lemma 7 (Lemma 1.2 in [51]) Different from their calculations, we prove the similar inequalities on f in the stochastic optimization. Let non-negative objective f is L -smooth and L G -Lipschitz w.r.t w . The local updates satisfy w t i,k +1 = w t i,k -ηg t i,k on C and ˜ w t i,k +1 = ˜ w t i,k -η ˜ g t i,k on ˜ C . If at k -th iteration on each round, we sample the different data in C and ˜ C , then we have:

Proof In each round t , let by the triangle inequality and denoting the different data as z and ˜ z , we have:

The final inequality adopts the L G -Lipschitz. This completes the proof.

A.3.2 Bounded Uniform Stability

According to Lemma 5, we firstly bound the recursive stability on k in one round. If the sampled data is the same, we can adopt Lemma 6. Otherwise, we adopt Lemma 7. Thus we can bound the second term in Lemma 5 as:

At the beginning of each round t , FL paradigm will aggregate the last state of each client w t -1 i,K , according to our method, w t i, 0 = w t + β ( w t -w t -1 i,K ) , thus the relationship between them is:

It could be seen that if we consider the w t i, 0 -w t -1 i,K as a general update step, it is independent to the dataset. Hence, we assume a virtual update between w t -1 i,K and w t i, 0 which could be bounded by the divergence term ∆ t . Then we bound the recursive term on t .

We know that before t ⋆ K + k ⋆ = t 0 , no different data is sampled, which is, w t i,k +1 = ˜ w t i,k +1 for ∀ tK + k ≤ t ⋆ K + k ⋆ . After t 0 +1 , they become different. Thus, when t ⋆ K + k ⋆ > t 0 , let learning rate η t to be a constant

within each round t and η = c t , then we have:

We adopt the same learning rate η = c t where c = µ 0 K is a positive constant, then

A.3.3 Proof of Theorem 3

Theorem 11 Under the Assumptions 6, 7, 9, and 10, let all conditions above satisfied, we can bound the uniform stability of our proposed FedInit as:

Proof According to Lemma 5, we have:

The second term is bounded by uniform stability term as:

This completes this proof.

B Experiments

In this section, we mainly provide the detailed experimental setups in our paper, including the introduction of the benchmarks, dataset, hyperparameters selections, and adding some more experiments.

B.1 Setups

Dataset. We follow the previous works and select the CIFAR-10 / 100 [20] dataset in our experiments. In the CIFAR-10 dataset, there is a total of 50,000 training images and 10,000 test images which contain 10 categories. Each data sample is a color image with a size of 32 × 32. In the CIFAR-100 dataset, there is also a total of 50,000 training images and 10,000 test images. It contains 100 categories of the same size as CIFAR-10. For their limited resolutions, we only use general data augmentations. On each local heterogeneous dataset, we use general normalization on the images with specific mean and variance. For the training process, we randomly crop a 32 × 32 patch from the vanilla images with a zero padding of 4 . For the test process, we use the raw images.

Heterogeneity. We follow Hsu et al. [14] to introduce the label imbalance as the heterogeneous dataset. According to the Dirichlet distribution, we first generate a specific vector with respect to a constant D r to control its variance level. Usually, heterogeneity becomes stronger when D r decreases. Then according to the vector, we sample the images from the training dataset. Here we enable the sampling with replacement to generate the local dataset, which means the local clients may have the same data sample if they are assigned to the same category. This is more related to the real scenario. At the same time, it also will lose some data samples, we assume this case is due to the offline devices. This is a common case because the FL has an unreliable network connection across the devices.

Benchmarks. In this paper, we use FedAvg [26], FedAdam [30], FedSAM [29], SCAFFOLD [19], FedDyn [1], and FedCM [46] as the benchmarks. FedAvg propose the general FL paradigm based on the local SGD method. It allows partial participation training via uniformly selecting a subset of local clients. A series of developments followed it to improve its performance. FedAdam studies the efficient adaptive optimizer on the global server update, which extends the scope of the FL paradigm. SCAFFOLD indicates that FL suffers from the client-drift problem which is due to the inconsistency of local optimum. Beyond this, it uses the variance reduction technique to further reduce the divergence across the local clients. To further alleviate, FedDyn studies the primal-dual method via adopting the ADMM to solve the problem. The consistency condition works as a constraint during the optimization. It proves that when the global model converges, the local objectives will be aligned with the global one. FedCM proposes an efficient momentum-based method, dubbed client-level momentum. It communicates the global update as a correction to correct each local update to force the local client updates in a similar direction. It maintains very high consistency via a biased correction. Therefore, it relies on an accurate global direction estimation. FedSAM considers the generalization performance. Generally, we adopt empirical risk minimization (ERM) to perform the optimization process. While the sharpness-aware-minmization (SAM) studies that it could search for a flat loss landscape. Flatness guarantees a higher generalization performance. Though our focus is not the generalization, we theoretically prove that even in the FedAvg method divergence term affects the generalization error bound more than the optimization error bound. From this perspective, generalization-efficiency methods may also be connected with consistency guarantees. These are all the SOTA benchmarks in the FL community that concern more on enhancing consistency.

Hyperparameters selection. Here we detail our hyperparameter selection in our experiments. For each splitting, we fix the total communication rounds T , local interval K , and mini-batchsize for all the benchmarks and our proposed FedInit . The other selections are stated as follows.

⋆ means different selections according to the specific setups.

We fix the most hyperparameters of testing the whole benchmarks for a fair comparison. The other algorithmspecific hyperparameters are subjected to specific circumstances. The ResNet-18-GN and VGG-11 adopt the same set of selections. Then we show algorithm-specific hyperparameters:

Special hyperparameter selections . In the FedAdam method, we test that it is very sensitive to the global learning rate. Though we report the best selection is 0 . 1 , it still requires some finetuning based on the dataset and experimental setups. In the FedSAM method, we test it is very sensitive to the perturbation learning rate. Usually, it should be selected as 0 . 1 in most cases. However, in some poor-sampling cases, i.e. low participation ratio, it should be selected as 0 . 01 . In the FedDyn , we test it is very sensitive to the regularization coefficient.

Generally, it adopts the regularization coefficient to be 0 . 1 on CIFAR-10 and 0 . 01 / 0 . 001 on CIFAR-100. In FedCM , we select the client-level coefficient as 0 . 1 which is followed by Xu et al. [46] in most cases. However, on the VGG-11 model, it fails to converge with a small client-level coefficient.

B.2 Experiments

B.2.1 Curves

In this section, we show the curves of our results.

2.5

2.0

1.5

1.0

0.5

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

500

100

0

200

300

400

communication Rounds

T

%

(a) Dir-0.6 10

-100.

2.5

2.0

1.5

1.0

0.5

loss test accuracy

test accuracy

0.8

0.7

0.6

0.5

0.4

0.3

0

100

200

300

400

communication Rounds

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

500

T

%

(a) Dir-0.6 10

-100.

0.5

0.4

0.3

0.2

0.1

0.0

0

100

200

300

400

500

600

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

700

communication Rounds

800

T

%

(a) Dir-0.6 10

-100.

loss test accuracy

test accuracy

0.7

0.6

0.5

0.4

0.3

0.2

0.4

0.3

0.2

0.1

0

100

200

300

400

communication Rounds

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

500

T

%

(b) Dir-0.1 10

0

100

200

300

400

communication Rounds

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

500

T

-100.

(c) Dir-0.6 5

%

-200.

0

100

200

300

400

500

600

700

800

communication Rounds

T

0

1

2

3

4

loss

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

(a) Dir-0.6 10

%

-100.

0

100

200

300

400

500

600

700

800

communication Rounds

T

0

1

2

3

4

loss

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

(b) Dir-0.1 10

%

-100.

0

100

200

300

400

500

600

700

800

communication Rounds

T

0

1

2

3

4

loss

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

(c) Dir-0.6 5

%

-200.

0

100

200

300

400

500

600

700

800

communication Rounds

T

0

1

2

3

4

5

loss

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

(d) Dir-0.1 5

%

-200.

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0

100

200

300

400

communication Rounds

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

500

T

%

(b) Dir-0.1 10

-100.

0

100

200

300

400

communication Rounds

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

500

T

%

(c) Dir-0.6 5

-200.

0.5

0.4

0.3

0.2

0.1

0

100

200

300

400

500

600

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

700

communication Rounds

800

T

%

(b) Dir-0.1 10

-100.

0

100

200

300

400

500

600

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

700

communication Rounds

800

T

%

(c) Dir-0.6 5

-200.

To show the stable accuracy curves, we use the third-party tsmoothie.smoother to smooth the raw curve via the function ConvolutionSmoother(window_len=100, window_type='hanning') . On most setups, our proposed FedInit achieves the SOTA results. It effectively avoids negative impacts from local overfitting.

loss test accuracy

test accuracy

2.5

2.0

1.5

1.0

0.5

loss test accuracy

test accuracy

3.0

2.5

2.0

1.5

1.0

0.5

0.7

0.6

0.5

0.4

0.3

0.2

0

100

200

300

400

communication Rounds

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

500

T

%

(d) Dir-0.1 5

-200.

0

100

200

300

400

communication Rounds

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

500

T

%

(d) Dir-0.1 5

-200.

0.45

0.40

0.35

0.30

0.25

0.20

0.15

0.10

0

100

200

300

400

500

600

communication Rounds

FedAvg

FedAdam

FedSAM

scaffold

FedDyn

FedCM

FedInit

700

800

T

%

(d) Dir-0.1 5

-200.

B.2.2 Consistency of Different Initialization

In this part, we mainly test the consistency level of different β . The coefficient β controls the divergence level of the local initialization states. We select the FedAvg and SCAFFOLD to show the efficiency of the proposed relaxed initialization.

0

100

200

300

400

500

Communication Rounds

0.3

0.4

0.5

0.6

0.7

0.8

Test T op-1 Acc.

beta=-0.2

beta=-0.1

beta=0

beta=0.1

0

100

200

300

400

500

Communication Rounds

1.4

1.6

1.8

2.0

2.2

Divergence term

beta=-0.2

beta=-0.1

beta=0

beta=0.1

0

100

200

300

400

500

Communication Rounds

0.2

0.3

0.4

0.5

0.6

0.7

Test T op-1 Acc.

beta=-0.2

beta=-0.1

beta=0

beta=0.025

beta=0.05

0

100

200

300

400

500

Communication Rounds

0.8

0.9

1.0

1.1

1.2

Divergence term

beta=-0.2

beta=-0.1

beta=0

beta=0.025

beta=0.05

(a) Dir 0.6 10%-100 Accuracy and Consistency.

(b) Dir 0.1 5%-200 Accuracy and Consistency.

0

100

200

300

400

500

Communication Rounds

0.3

0.4

0.5

0.6

0.7

0.8

Test T op-1 Acc.

beta=-0.2

beta=-0.1

beta=0

beta=0.1

0

100

200

300

400

500

Communication Rounds

1.6

1.8

2.0

2.2

2.4

2.6

2.8

Divergence term

beta=-0.2

beta=-0.1

beta=0

beta=0.1

(a) Dir 0.6 10%-100 Accuracy and Consistency.

0

100

200

300

400

500

Communication Rounds

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Test T op-1 Acc.

beta=-0.2

beta=-0.1

beta=0

beta=0.025

beta=0.05

0

100

200

300

400

500

Communication Rounds

0.9

1.0

1.1

1.2

1.3

1.4

1.5

Divergence term

beta=-0.2

beta=-0.1

beta=0

beta=0.025

beta=0.05

(b) Dir 0.1 5%-200 Accuracy and Consistency.

These experiments show that the relaxed initialization (RI) effectively reduces the consistency and improves the test accuracy. In all tests, when β = 0 (green curve), it represents the vanilla method without RI. After incorporating the RI, the test accuracy achieves at least 2% improvement on each setup.

B.2.3 Communication, Calculation and Storage Costs

In this part, we mainly compare the communication, calculation, and storage costs theoretically and experimentally. By assuming the total model maintain d dimensions, we summarize the costs of benchmarks and our proposed FedInit as follows:

where N is the number of participating clients, C is the total number of clients, and K is the local training interval.

Limitations of the benchmarks. From this table, we can see that SCAFFOLD and FedCM both require double communication costs than the vanilla FedAvg . They adopt the correction term (variance reduction and client-level momentum) to revise each local iteration. Though this achieves good performance, we must indicate that under the millions of edge devices in the FL paradigm, this may introduce a very heavy communication bottleneck. In addition, the FedSAM method considers adopting the local SAM optimizer instead of ERM to approach the flat minimal. However, it requires double gradient calculations per iteration. For the very large model, it brings a large calculation cost that can not be neglected. SCAFFOLD and FedDyn are required to store 3 × vectors on each local devices. This is also a limitation for the light device, i.e. mobiles.

We also test the practical wall-clock time on real devices. Our experiment environments are stated as follows:

In the following table, we test the wall-clock time cost of each method:

From this table, due to the different communication costs and calculation costs, the practical wall-clock time is different for each method. Generally, FedAvg adopts the local-SGD updates without any additional calculations. FedAdam adopts similar local-SGD updates and an adaptive optimizer on the global server. FedSAM calculation double gradients, which is the main reason for being slowest among the benchmarks. SCAFFOLD , FedDyn , and FedCM are required to calculate some additional vectors to correct the local updates. Therefore they need some additional time costs. Our proposed FedInit only adopts an additional initialization calculation, which requires the same costs as FedAvg .

B.2.4 Training Efficiency: Communication Rounds and Time Costs

In this part, we mainly show the results of the training efficiency. We set the target accuracy and compare their required communication rounds and training time respectively. We test on the ResNet-18-GN model with the 10%-100 Dir-0.1 splitting.

The setups of the test environment are stated in Table 9. According to this table, we clearly see that some advanced methods, i.e. SCAFFOLD and FedDyn , are efficient on the communication round T . However, due to the additional costs of each training iteration, they must spend more time on the total training. FedInit is a very light and practical method, which only adopts a relaxed initialization on the FedAvg method, which makes it to be better and even achieves SOTA results.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes a novel federated learning method to alleviate the negative impact of the \"client drift\" problem and enhance consistency in the FL paradigm. The manuscript also analyzes the intrinsic impact of local consistency on optimization error, test error, and generalization error. Several experiments are conducted to validate the efficiency of the proposed method.\n\nSTRENGTHS:\n1. The proposed method outperforms baseline methods without increasing communication costs and can be easily incorporated into other methods.\n2. The manuscript provides ample theoretical analysis to demonstrate the theoretical bounds of the proposed method.\n\nWEAKNESSES:\n1. The hyperparameters were not tuned for different methods. Since the experiments are highly sensitive to hyperparameter settings, the superiority of the proposed method is likely due to the selection of hyperparameters. Given that the personalized process proposed in the manuscript bears resemblance to the momentum of parameter updates, I suspect that when tuning the learning rate for the baseline method, the baseline method may outperform the proposed method.\n2.  The results shown in Figure 1 suggest that the proposed method is highly sensitive to the choice of beta.\n\nQUESTIONS:\n1. The manuscript thinks that inconsistent local optima cause \"client drift,\" which results in the inferiority of federated learning (FL) algorithms. The proposed method aims to alleviate this divergence by employing personalized relaxed initialization at the beginning of each round. I have a question: based on my understanding, personalized initialization in each round would lead to greater differences across different clients. So, how does this strategy help alleviate the divergence?\n2. After a brief review of the theoretical analysis, I am uncertain if I correctly understood the proof. In formulas (9) and (14), the upper bounds achieve their minimum when beta=0. If that's the case, why does the personalized relaxation contribute to improving the performance of FL algorithms?\n\nLIMITATIONS:\nThe experiments in the manuscript are not rigorous enough as the hyperparameters for different methods were not tuned. This could potentially lead to an overestimation of the performance of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper proposes to initialize the local state by moving away from the current global state toward the reverse direction of the latest local state. They demonstrate theoretically and empirically that this revision can help consistency for better performance. The method is also a practical plug-in that could easily to incorporated into other methods.

STRENGTHS:
- This paper goes deeper into how consistency work in FL systems, and provides a simple and effective initialization-based solution.
- This paper gives a comprehensive theoretical analysis of the problem, which I think is a good contribution.
- The paper is well-written and easy to follow.
- Experiment shows good performance.

WEAKNESSES:
- I understand there might not be enough space for experiments and the focus looks to be theoretical analysis, but it would be better if the authors can give more experimental results for different FL settings, e.g., more clients/datasets, which may help evaluate the method better. For example,  100 clients/10% participation rate and 200 clients/ 5% participation rate can be relatively limited. Maybe can we have a curve for participation rate with the same amount of clients and perhaps a curve for number of clients to see how these FL settings influence the performance?

QUESTIONS:
- Does the method also work well for a large number of clients, e.g., 1000?
- How does participation rate influence performance?

LIMITATIONS:
The authors have stated the limitation and future work as pFL.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to solve the “client-drift” problem in Federated Learning, which is caused by the NonIID data. Specifically, this paper proposes initializing the local model of each client with its personalized model to alleviate the problem. Further, the paper theoretically analyzes the impact of inconsistency on the convergence of FL. Besides, extensive experiments also demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1.\tThe idea of initializing the local model for solving the NonIID problem is interesting.\n2.\tThe theoretical analysis for the proposed method is solid. \n3.\tThe experimental results demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n1.\tAlthough the theoretical results are sufficient to verify the convergence of the proposed method, they cannot present the effectiveness of the initialization strategy in principle. More specifically, how does the proposed method reduce the divergence term compared to the vanilla FedAvg?\n\n2.\tThe workflow of the proposed method seems not correct. Are the locations of line 10 and line 12 placed correctly?\n\n3.\tHow to obtain the value of $w_{i,k}^{t-1}$ is not clear. The client may not participate in the previous round $t-1$ under the setting of random client selection. If the local model $w_{i,k}^{t’-1}$ is obtained in many previous rounds, the motivation of bias correction using $ w_{i,k}^{t’-1} - w^{t-1}$ is not reasonable due to the significant gap between $t’$ and $t$.\n\n4.\tThe key hyperparameter K is set inappropriately. In typical FL literature, the number of local epochs is usually set to range from $5$, $10$, and up to $20$, which may contain many local iterations (mini-batches). However, the experiment of this paper only adopts a small number of local iterations instead of local epochs, e.g., 5 local iterations in Table 1, which seems manually adjusted.\n\n5.\tIt would be better if there is a figure to illustrate the principles of the proposed method.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an efficient stage-wise initialization for the federated learning paradigm, named FedInit, which could be extended as a plug-in to several existing methods. It provides the theoretical analysis on both the convergence and generalization to illustrate how consistency term affects the FL. Experiments also show its efficiency in practical scenarios.\n\nSTRENGTHS:\n1. FedInit is a light and nimble technique that does not introduce extra communication costs in the FL framework. Compared with other algorithms, FedInit can almost achieve the same performance as SOTA algorithms while maintaining the same communication cost as the vanilla FedAvg. And, Relaxed initialization (RI) could be easily extended into other advanced algorithms as a plug-in and efficiently improve their performance.\n\n2. This paper explores the impacts of global consistency constraints in FL, which is an essential and interesting problem in federated learning. It also indicates the relationships between generalization and optimization.\n\n3. This paper provides a theoretical analysis of the excess risk to comprehensively explain how the dominant factors affect the final performance (test accuracy) throughout the training process. Ablation studies are conducted to validate the conclusion of the theoretical analysis on the hyperparameters, i.e. local interval $K$, coefficient of RI $\\beta$.\n\nWEAKNESSES:\n1. The relaxed initialization (RI) looks like an extrapolation algorithm between the global model and the local models as $w+\\beta(w-w_i)$. Several methods including the benchmarks of this paper, i.e. FedProx, and FedCM, are accustomed to applying interpolation methods to adjust local training to make the entire heterogeneous training smoother. So what is their essential difference? I think the author should add a paragraph to discuss it.\n\n2. This paper involves a lot of analytical proofs, so I suggest that the author write a simple proof sketch for each sub-sections to facilitate reading.\n\n3. In Table.2, RI helps to largely improve the performance of FedSAM and SCAFFOLD but shows limited help for FedCM and FedDyn. What is the main reason for this phenomenon? Does it imply that RI is subject to some existing algorithms? I think the author should add a paragraph to discuss it.\n\n4. Some typos (for example):\n(1) Line.150 \"... ... its generality\"\n\n(2) Line.79 \"client drift\" and Line.151 \"client-drift\"\n\n(3) Line.192 the sentence is difficult to read\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nPlease see the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}