Enhance Local Consistency for Free: A Multi-Step Inertial Momentum Approach
Authors: Yixing Liu
Yan Sun
Li Shen
Baoyuan Wu
Zhengtao Ding
Dacheng Tao
Venue: Submitted to ICLR 2023 (2023)
Keywords: federated learning
optimization

Abstract: Federated learning (FL), as a collaborative distributed training paradigm with several edge computing devices under the coordination of a centralized server, is plagued by inconsistent local stationary points due to the heterogeneity of the local partial participation clients, which precipitates the local client-drifts problems and sparks off the unstable and slow convergence, especially on the aggravated heterogeneous dataset. To address these issues, we propose a novel federated learning algorithm, named FedMIM, which adopts the multi-step inertial momentum on the edge devices and enhances the local consistency for free during the training to improve the robustness of the heterogeneity. Specifically, we incorporate the weighted global gradient estimations as the inertial correction terms to guide both the local iterates and stochastic gradient estimation, which can reckon the global objective optimization on the edges' heterogeneous dataset naturally and maintain the demanding consistent iteration locally.   Theoretically, we show that FedMIM achieves the $\mathcal{O}\big({1}/{\sqrt{SKT}}\big)$ convergence rate with a linear speedup property with respect to the number of selected clients $S$ and proper local interval $K$ in each communication round under the nonconvex setting. Empirically, we conduct comprehensive experiments on various real-world datasets and demonstrate the efficacy of the proposed FedMIM against several state-of-the-art baselines.

Under review as a conference paper at ICLR 2023
ENHANCE LOCAL CONSISTENCY FOR FREE: A MULTI-STEP INERTIAL MOMENTUM APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning (FL), collaborative distributed training paradigm with sev- eral edge computing devices under the coordination of a centralized server, is plagued by inconsistent local stationary points due to the heterogeneity of the local partial participation clients, which precipitates the local client-drifts problems and sparks off the unstable and slow convergence, especially on the aggravated het- erogeneous dataset. To address these issues, we propose a novel federated learn- ing algorithm, named FedMIM, which adopts the multi-step inertial momentum on the edge devices and enhances the local consistency for free during the train- ing to improve the robustness of the heterogeneity. Specifically, we incorporate the weighted global gradient estimations as the inertial correction terms to guide both the local iterates and stochastic gradient estimation, which can reckon the global objective optimization the edges' heterogeneous dataset naturally and maintain the demanding consistent iteration locally. Theoretically, we show that FedMIM achieves the O VSKT) convergence rate with a linear speedup property with respect to the number of selected clients and proper local interval K in each communication round under the nonconvex setting. Empirically, we conduct comprehensive experiments on various real-world datasets and demonstrate the efficacy of the proposed FedMIM against several state-of-the-art baselines.
INTRODUCTION
Federated Learning (FL) is an increasingly important distributed learning framework where the dis- tributed data utilized over large number of clients, such as mobile phones, wearable devices or network sensors (Kairouz et al., (021). In the contrast to traditional machine learning paradigms, FL places a centralized server to coordinate the participating clients train a model, without collecting the client data, thereby achieving basic level of data privacy and security (Li et al.) 2020a). The common pipelines to achieve this goal includes three steps (Bonawitz et al. 2019): The server broadcasts the current model to clients at the beginning of each communication iteration; ii) The clients synchronize the local models and update the local model based on their own data; iii) The server averages the latest local models and repeats these procedures until convergence.
Despite the empirical success of the past work, there are still some key challenges for FL: expensive communication, privacy concern and statistical diversity. The first two problems are well fixed in past work(Konedee et al. 2016 Sattler et al. 2019 Hamer et al. 2020 Truex et al. 2019: Xu et al., 2019) although the last one is still the main challenge that need be deal with. Due to statistical diversity among clients within FL system, client drift (Karimireddy et al.) 2020a) leads to slow and unstable convergence within model training. In the case of heterogeneous data, each client's opti- mum is not well aligned with the global optimum. The conventional FL algorithm does not consider this data heterogeneity problem and simply applies the stochastic gradient descent algorithm to the local update. As consequence, the final converged solution of clients may differ from the stationary point of the global objective function since the average of client updates move towards the average of clients" optimums rather than the true optimum. As the distribution drift exists over the client's 's dataset, the model may overfit the local training data by applying empirical risk minimization and it has been reported that the generalization performance clients have different distributions between training and testing dataset (Liang et al.) (020). In order to overcome these problems, several solutions have been put forward in recent years. Generally,
clients' local data may exacerbate when

---

Under review as a conference paper at ICLR 2023
there are three types of methods: variance reduction based (Karimiredyy et al. tion based (Li et al., 2020b; Acar et al. 2021) and momentum based (Xu et al. 2020). Although these past works present some effective methods to reduce the client drift and im- prove the generalization performance, the problem of local inconsistency is not fully considered. In the real experiment setting, the local interval K is finite and the local update could not reach the local optimum. With the iteration running, the final points for local iteration will remain relatively stable and become dynamic equilibrium. The stability of these points determines the effectiveness of algorithms and their position will alter when different algorithms are applied. The variance among these points brings the local inconsistency problem Waan et al. (2021a). However, the analysis of these past works are not comprehensive and experimental verification of the reduced local inconsis- tency is lacking. In particular, when data heterogeneity among clients raises, the local update may repudiate mutually, that is, the direction of the local gradient could not remain compatible. Thus, the weighted average of local gradient at the aggregation stage is extraordinarily small and the mov- ing global iteration point may stagnate, which leads to low generalization performance. To settle this problem, a federated learning algorithm is required to incorporate historical information of full gradient into client local updates for scaling down the variance between local dynamic equilibrium points. Furthermore, the usage of historical full gradient information to navigate the local update ought to be considered wisely instead of simply applied in the weight of models.
2020b) regulariza- 2021] Reddi et al.
In this paper, we develop a new FL. al- gorithm enhance local consistency for free, Federated Multi-step Inertial Momentum Algorithm (FedMIM), that mitigates client drift and reduces local inconsistency. From a high-level al- gorithmic perspective, we bring multi- step inertial momentum to the local up- date, that is, multi-step momentum is placed in both weight (orange arrow shown in Figure D and gradient (yel- low arrow shown in Figure I to modify the local update. Rather than calculating the momentum updates at the server's side and transmitting them through the down-link, all the clients compute the momentum term before the local iteration, while the historical momentum is kept in the client's ' stor- age. FedMIM has two major benefits undertaking aforementioned deficiencies. Firstly, FedMIM does not acquire the server to broadcast the momentum between rounds, which curtails the com- munication burden. Secondly, in contrast to previous work that focuses on server side momentum Karimireddy et al. | (2020b) or client side momentum Xu et al. (2021). FedMIM delivers inertial mo- mentum term to introduce global information avoiding the gradient exclusion in local update when there exists large data heterogeneity among the participating clients.
FedAug Update
FedMIM Update
Figure 1: Local steps of FedAvg and FedMIM with clients.
Theoretically, we provide detailed convergence analysis for FedMIM. By setting proper local learning rate, FedMIM could achieve O (SKT) convergence rate with a linear speedup property for general non-convex setting with the number of selected clients S. local interval K and commu- nication round T', As for non-convex function under PL condition, convergence rate achieves ( +) with proper setting of local learning rate. We test FedMIM algorithm on three datasets (CIFAR- 10, CIFAR-100 and Tinylmagenet) with i.i.d, and different Dirichlet distributions in the empirical studies. The results display that our proposed FedMIM shows the best performance among the sfate-of-the-art baselines. When the heterogeneity increases extremely, the performance of the fed- erated algorithms drops rapidly due to the negative impact of enlarging the local interval, while our proposed FedMIM can efficiently maintain stability under the same experimental setups.
Contribution. We summarize the main contributions of this work as three-fold:
FedMIM algorithm delivers a multi-step inertial momentum to guide the gradient updates. We show that FedMIM successfully solves the problems the heterogeneous datasets, which benefits the cross-device implantation in practical applications.
We display the convergence analysis of FedMIM for general non non-convex function and non- convex function under PL conditions. The theoretical analysis highlights the advantage of innovating multi-step inertial momentum and presents hyperparameter conditions.
2

---

Under review as a conference paper at ICLR 2023
We demonstrate the practical efficacy of the proposed algorithm over competitive baselines through extensive experiments on CIFAR1O, CIFARI00 and TinyImagnet datasets, under the various non-ii.d. dataset splitting, increased local training interval, and different par- tial participation ratios setups. The results illustrate that FedMIM consistently surpasses several vigorous baselines and significantly handles the local data heterogeneity.
2 RELATED WORK
We focus on and categorize the existing approaches to overcoming client drift issues as three main aspects: variance reduction, regularization, and momentum-based algorithms. These methods per- form as a global correction to guide the local training, which significantly improves efficiency.
Variance reduction. By adopting the variance reduction techniques in the stochastic optimization, a series of methods are proposed to mitigate the heterogeneous inconsistency, which efficiently mitigates the client-drift problems in FL.. Karimireddy al. (2020b) employs control variants to correct for the "client-drift' in its local updates. (aarmirrddy et al. (2020a) provides a combination of control-variates and server-level optimizer state at every client-update step to ensure that each local update mimics that of the centralized method. MMtra et al. (2021) includes a gradient correction term in the local update rule that exploits memory. |Murata & Suzuki (202T) utilizes small second- order heterogeneity of local objectives and suggests randomly picking up one of the local models instead of taking the average of them when clients are synchronized. Zhao al.] (2021b) applies the vanilla minibatch SGD update or the previous gradient with a small adjustment with pre-defined probability. Zhao al.) (20 addresses communication compression and high heterogeneity by proposing compressed and client-variance reduced methods COFIG and FRECON.
Regularization. Another efficient approach is to adopt the regularization terms on the local training process to correct the local objective to approach the global optimal. Li et al. (2020b) firstly employs the prox-term into FL framework and propose the FedProx. Tran Dinh et al. (2021) proposes the Rachford splitting in the training. Zhang al. (2020) adopts the primal-dual method to the FL. Acar et al. (202T) improve the FedPD and propose a partial merged parameters method with the full merged dual variables in the global server, named FedDyn. Wann et al. (2022); Gong et al. (2022) adopt the alternating direction method of multipliers the FL. to extend the federated primal-dual methods. Fallah et al. (2020) puts forward personalized federated framework with the regularization to achieve a better generalization. T Dinh et al. (2020) incorporates the Moreau -eneelope in the local training with a stage- wised prox- term. Huang et al. (2021) proposes an adaptive weight for the regularization term to encourage the clients to aggregate more with similar neighbours. The efficient regularization methods are important to the FL. field.
FedDR with Douglas-Rachford
Global / Local momentum. Inspired by the success of the global correction technique, the exponen- tial moving average term is introduced to federated learning framework to correct the local training- Liu et al.] (2200) adopts the Momentum-SGD to the local clients to improve the generalization per- formance with a convergence analysis. Wang et al. 2019) proposes a global momentum method to further improve the stability in the server side. Xu et al. (202T) incorporate the global offset to the local client as a client-level momentum to correct the heterogeneous drifts. OOzatura et al. (2021) combine the global and local momentum update and propose the FedADC algorithm to avoid the local over-fitting, Reddi et al. (2020) sets a global ADAM optimizer with the momentum update and propose the adaptive federated optimizer in the FL. Wang et al. (2021b) corrects the pre-conditioner in the global server. Though momentum terms are the biased estimation of global information, they still contribute lot the federated frameworks in practical empirical experiments.
3 FEDMIM: FEDERATED MULTI-STER MOMENTUM ALGORITHM
In this section, we describe how FedMIM works while reducing client drift and improving conver- gence. To begin with, we provide some preliminary for FL and notations adopted in this paper in Section 3.1 We introduce the diagram of our proposed FedMIM method, and the insights of its improvement on the performance and the resistance to the local heterogeneity in Section 3.2
3

---

Under review as a conference paper at ICLR 2023
3.1 PROBLEM SETUP
Considering an FL framework with N local clients and a centralized server to handle the training process. The client i for [N] has the local private dataset D, without sharing, and the data sample & is randomly drawn from the local dataset Dj The minimization problem could be formulated as:
min f(x) == XEX
Σ fx( N
1
(1)
where fi(x) := Egg [f:( 5.) is the local loss function corresponding to the client i with the data distribution p;. Note that p: may differ among the local clients, which introduces heterogeneity.
Notations. In this paper, we consider K local iteration steps and total ''' communication rounds in the training. ||I| denotes the Euclidean l2 norm if not otherwise specified. In f-th communication round, set of active clients S, with size S is adopted. The symbol (-) represents the vectors at k-th local step on the i-th client after the / communication rounds. For simplicity, [N] represents the set {1, 2, .. ,,,,,,, ]N.. x. and x stand for global optimum and local optimum for client respectively.
3.2 FEDMIM ALGORITHM
FedAvg proposes partial scenario of the local SGD method, with the global averaged aggregation after local K up- dates, disturbed by the client drift prob- lems due the local heterogeneous case. To mitigate the aforementioned problem, we propose the novel federated learn- ing algorithm with multi-step inertial momentum-based technique, dubbed Fed- MIM, shown in the right part. Firstly we introduce the basic process of the pro- posed FedMIM. At the beginning of each communication round, the server broad- casts the global model x₃ to the activated
Algorithm 1 FedMIM
Input: model parameters X constant weight {or Jee con-
stant weight (3) {8)},er= local learning rate y!.
Output: model parameters X..
I: for t 0, 1, . 1 do
2: communicate xx local client and set xf.0
randomly select active clients-sct St at round
4:
for client E Se parallel do
5:
Local Update:
6:
6, = (x: Xx- 1/K
7:
for 0, 1, K
1 do Σjer a,ōr Byor
yi,,, x
E EjEr 05 < 1)
9:
y xi,
Ejet 33 = p)
10:
randomly sample local data Ci
clients for local training. The local clients
11:
compute stochastic gradient gi, of Vf (,22)
update their states xf,, with total K itera-
12:
xi,++1 = -Y,,,, (1 (1-E 02 mmgi,
tions and then transmit their updated mod-
13:
end for
els x to the server, while the server ag- gregates the received local models the updated global model. In the local clients,
14:
communicate x!, to the server end for
15:
16:
XI+1 Exes, x
they firstly calculate the global model in- crement & by the last global model X
17:
end for
in their local own storage. And then, the clients compute the momentum updated v',x,1 and vi...2 with a multi-step momentum. It should be noted that a and 3 are adopted for averaging weights in the multi-step momentum term. Each client calculates an unbiased stochastic gradient g and updates its state. When the local update stops, x is transmitted to the server. The iterate scheme details of FedMIM is summarized in Algorithm
Intuitive Justification To build intuition into our method, we first highlight multi-step inertial part. Lemma in appendix illustrate that &, is the exponential moving average of past client gradient. The momentum term de represent as an approximation the offset of the global loss function fx;,), that is, de V (x:). Thus, we have local update:
x x
x (1 A)md, Am fi(ii - Th ESer Bror-s)
x (1 A) - VSi (xt, (V)(xx))
(2)
m [ - pV
+ (1- - vf(x)))
For simplicity, we set the constant A Σjer aj and أ mp. This equation illustrates that FedMIM interprets the correction term the local gradient direction. This correction term matches
4

---

Under review as a conference paper at ICLR 2023
(xx)) in Eq. the difference between global and local gradient. The second term Vfi(xi, behaves like Nesterov gradient part, which means that there is global momentum /V fxx) placed on local iteration point x when client computes the local gradient in k-th local update and t-th communication round. The added global momentum pushes the local gradient calculation point to move in the same direction compared with f(( This benefits the circumstance where the data distribution among the clients differs intensively since the client's update would radiate in high data heterogeneity environment. In the meanwhile, the added global momentum dwindles gradually as the full gradient (x) reduces, and the influence of global momentum scales down with the training process. Therefore, each participating client could reach their dynamic equilibrium at the end of the training. The final correction term is controlled by the parameter a. It is notable that the local gradient part is V f pV f(x))) rather than f ( as the direction of local update is V [x(( /V I (x:)) in the front term and the correction term ought to be consistent with it.
Discussion. FedMIM saves the communication bandwidth, which is a crucial problem in FL study. During the broadcasting stage, the server only needs send the current global state x, the clients rather than the aggregation gradient information in FedCM. The storage of the client is efficiently utilized since it only needs to store J steps of historical information where is usually set to be very small in practical scenarios and the client's s storage requirement does not increase violently. Next, FedMIM simply calculates the gradient once, while FedSAM computes the gradient twice in one local iteration. Thus, the local calculation process could be condensed and total training time is much reduced. The historical global state is stored clients' storage. FedMIM brings multi-step inertial momentum, which is robust to high client heterogeneity. Since global gradient information is applied to avert the average of client update direction to be minuscule and force global iteration point to move. The introduced multi-step inertial momentum makes the gradient changes more smooth during the local training, although there are some atrocious clients who hold discordant data. The long-step looking makes the approximation exact and smooth for local training, which promotes communication efficiency and enhances the robustness to the heterogeneity in the FedMIM.
4 CONVERGENCE ANALYSIS
In this section, we provide the theoretical analysis for FedMIM focusing on the general non-convex setting. Before proposing our convergence analysis, We first state the several assumptions as follows.
Assumption For all x,y Rd, the non-convex fi is L-smooth function for all E [N], i.e.
(f ff(yv) < L||x - -yll
Assumption Let f. = f(x.) and x, is a minimizer of f. for all x Rd, the function f satisfies PL inequality if there exists the constant μ >> such that the function f satisfies the following:
1 |Vx(2)] s -
(4)
2
Assumption 3 For all E Rd, the stochastic gradient V (), computed by the sampled data on the local client i, is an unbiased estimator of V f((x)) with bounded variance of, i.e,,
Eg f((x VVf(()) E V
V oo
(5)
Assumption 4 For all E R2d, the local functions Si holds (G, B)-localll dissimilarity with f. i.e.
[V < 0² B²I
6
N
These assumptions are commonly used in federated optimization (Li et al., 2020b; Reddi et al., 2020 Karimireddy et al. 2020a b). Assumption tells the smoothness of local loss function fü that is, the gradient function of fi is Lipschitz continuous with Lipschitz constant L. Assumption shows the global function satisfies the PL conditions. The PL inequality does not require to be convex but suggests that every stationary point is a minimum. The p-PL property is implied by j-strong convexity, but it allows for multiple minima and does not require convexity of any kind.
5

---

Under review as a conference paper at ICLR 2023
Assumption bounds the variance of stochastic gradient and Assumption 4 provides the bound of the different levels the local private heterogeneity.
We now state convergence results of FedMIM. The detailed proof is stated in Appendix.
Theorem 4.1 Let Assumptions 7 hold. Assume the partial participation ratio being |S,\/N where St is uniformly sampled subset from the N clients and satisfies |St = S and let U,
K Ejer Σxz agōr-j. With the constant local learning rate satisfying ny min 4LKVA 16KE
3
and À E (0, 2 in Algorithm 1
the sequence {ur} satisfies the following upper bound:
fu fm MAKT + v
min Vf((u)) 1E[2]
where v -
(mLoi (n +
An KLG + A 72n AK?L + 3 V, C are S
two constants defined in the proof the convergence analysis (details are stated in the Appendix).
Remark 4.1 If the number of total clients N large enough, the initial state point will affect the convergence upper bound to great exlent, which requires larger local learning rate y to diminish the negative impact. Specifically, when we fix N as a constant and select a proper local interval K, let Qn = O v5 (KT), the convergence rate achieves at least O /SKT)) which indicate the linear speedup of the FedMIM and the stochastic variance dominates the upper bound of the convergence.
Theorem 4.2 Let Assumption 2 and hoold and all the conditions being similar as required by pAKT the output o chosen randomly from the sequence {ue} satisfies:
Theorem 4.1 Given qi
V (u
1| S 4p(Jo fa)e PMAKT + V
where V = 1 Loi
8m KL
18 K KA + 144 144 AKP +IN
Remark 4.2 The term introduced by initial point is exponential diminished by the communication round T. Let Tp O logg (T)
pAKT. the convergence rate achieves at least O( PST
PAKT
Remark 4.3 The B in Assumption 4 weakly inftuences the convergence bound both in Theorem 4.1 and 4.2 our proof. which indicates that the major negative impact from the heterogeneity the constant upper bound G. IfG maintains the stability without large fluctuations during the training, let = x, we have N Eim f(x.)l < G2, where G measures the local inconsistency of total clients. Enhancing the local consistency will further improve the performance in the FL frameorkk.
EXPERIMENTS
In this section, we demonstrate the efficacy of the proposed FedMIM. We test the generalization performance under different levels of the heterogeneity on the real-world dataset. To ensure fair comparison, we fix all the common hyper- super-parameters and finetune the specific parameters unique to each algorithm to search for their best performance. We provide brief introduction to the ex- perimental setups in 5.1 We compare the proposed FedMIM with the baselines and report their performance in 5.2 Some ablation studies and hyper-parametts parameters sensitivity studies are stated in 5.3
5.1 SETUPS
Dataset. We conduct the extensive experiments on CIFAR-10, CIFAR-100 (Krizhevsky et al. 2009) and Tinylmagenet (Le & Yang 2015). Both CIFAR-10 10 and CIFAR- 100 contain 50K training sam- ples and 10K test samples of images with the size of 32 x 32. Tiny Imagenet contains 200 categories of 100K training samples and 10K test samples of images with the size 64 64 selected from the Imagenet (Deng al. 2009). We divide the training dataset into N parts and deploy them to local clients without sharing access. At the beginning of each communication round in the training, we randomly crop, horizontally flip, and normalize the local dataset as the common data augmentation.

---

Under review as a conference paper at ICLR 2023
Heterogeneity, For the IID setting, the local dataset is randomly sampled. For the non-IID setting, we follow by Hsu et al. (2019) to introduce different levels of the heterogeneity by sampling the label ratios from different Dirichlet distributions. which is common federated setup in previous
works (Reddi et al. 2020 Karimireddy et al.] 2020b, Acar et al. 2021: Xu et al. 2021 Qu et al,
2022 [im et al. 2022). In addition, we superimpose color perturbation (Arjovsky et al., 2019)
which is strongly correlated to the local clients to further induce the heterogeneity. Specifically, we adopt different brightness and saturation coefficients to the different local training data samples.
Baselines. We compare the performance of several SOTA baselines, including FedAvg (McMahan ct al. 2017), FedAdam (Reddi et al. al.] 2020)) SCAFFOLD (Karimireddy et al. (020b), FedDyn (Acar et al. 2021), ) FedCM (Xu et (021)). and FedSAM (Qu et al.) (022), on the backbonn of standard
ResNet-18 network implemented in the Pytorch Model 700 (7 7 filter in the 1st conv) (He et al.)
2016) with the group normalization (Wu & He| 2018 Hsieh et al. 2020). We summarize and discuss these methods in Section 3.2 to illustrate the respective improvements and practical performance.
parametassssser selections. To ensure a fair comparison, we fix the common hyper- parameerrretr setups. We set the local learning rate as 0.1 and decay as 0.998x per round. The global learning rate is set as 1.0 to aggregate each local parameters without decaying, except for FedAdam which adopts 0.1. The local mini-batch is selected in {00, The weight decay is set as 1e-3, The local training epoch is selected in {1, 5, 10) to further show the impacts of enlarging the local interval. The number of total clients is selected in {100, 500) and the sampling probability of each client being activated per communication round is selected in (0.2, 0.1, 0.02). The prox-weight in FedDyn and the client-level momentum weight in FedCM are both set as 0.1. We report the detailed hyper-parameters selections for each experimental result in the following figures and tables.
5.2 EXPERIMENTAL RESULTS
5.2.
COMPARISON ON INCREASING THE HETEROGENEITY.
To explore the impact of introducing the heterogeneity, we select the three splitting methods on the dataset, including IID, Dirichlet-0.3 and Dirichlet-0.1. On the simple (IFARR10/100 dataset, FedMIM achieves top-1 performance among the three heterogeneous settings. In the IID case on CI FedMIM achieves 36.39% with 4.23% over the FedAvg baseline. The second top per- formance of FedCM is 85.62%. When the heterogeneity is increased to DIR-0.3, FedCM drops from 85.622 62.62%%o to 82.39% with 3.23% loss, while FedMIM drops only 2% and maintains the top-1 accuracy. The other methods like SCAFFOLD, FedDyn and FedSAM are affected at different levels. When we further enlarge the heterogeneity to DIR-O.1, the FedAdam is most affected and its accu- racy drops from 83. $3.19% to 71.75% with approximate 12% loss. On the large CIFAR-100 100 and Tinylmagenet datasets, similar results can be observed. Our proposed FedMIM have the very stable test accuracy. In particular, when the heterogeneity introduced DIR-0.1 on CIFAR-100, Fed- MIM achieves only 1% drops, which is far better than the others with at least 2%. Its performance also better than the test accuracy on DIR-0.3 of most other baselines. In the IID splitting of Tiny- Imagenet, the FedAdam even can achieve the top-1 performance, while when the heterogeneity its performance drops rapidly and even worse than FedAvg. FedMIM adopts the inertial momentum to the local training both on the iteration points and gradients and enhances the local consistency, which can efficiently resist on the heterogeneity. The multi-step makes the gradient changes more smooth during the training, even under the participation of some bad samples of clients whose dataset holds a very large difference, the long-step looking makes the approximation exact and stable for local training, which encourages the efficiency and robustness to the heterogeneity for the FedMIM.
5.2.2 COMPARISON ON ENLARGING THE LOCAL INTERVAL.
To further explore the impact of enlarging the local interval K, we select the three different local intervals to test the performance of our proposed FedMIM and the other baselines. We follow the previous works (Acar et al. 2021 Xu et al. 2021 Kim et al. 2022) to compare the performance under different local epochs E - 1, 5, 10. The total training samples in CIFAR-10/100 is 50,000 and they are split into 100 parts equally with 500 samples from a local client. To fairly compare with the others, we fix the batchsize as 50. which means the local iteration is TrainSammles/Batchsizzz = 10 per epoch. It should be noted that in the proof the local interval K corresponds to the iteration. When the F = 1 with a short local interval, local training do not introduce more local heterogeneity
7

---

Under review as a conference paper at ICLR 2023
Table 1: Test accuracy (%) after 1000 communication rounds on MD., Dirichlet-0.3 (DIR.3), and Dirichlet-O.1 (DIR.1) dataset. We set the number of clients as 100 and set the participation ratio as 0.1. The local interval is fixed as 5 epochs and the batchsize fixed as 50 for all algorithms.
CIFAR-10
CIFAR-I00
TinyImagenet
Method
IID.
DIR.3
DIR.1
IID.
DIR.3
DIR.1
IID.
DIR.3
DIR.
FedAvg
82.16
80.19
75.92
42.77
42.18
40.88
30.31
28.61
27.93
FedAdam
83.19
78.61
71.75
48.92
44.58
42.96
37.04
32.34
29.44
SCAFFOLD
84.43
82.39
78.78
49.11
48.34
46.93
35.09
34.68
34.17
FedCM
85.62
82.57
77.31
51.14
49.27
45.59
34.35
33.34
32.34
FedDyn
84.23
81.44
75.66
46.87
45.77
43.93
33.91
31.97
30.11
FedSAM
84.98
82.13
77.65
48.32
46.24
44.18
35.04
33.15
32.86
Ours
86.39
84.39
80.82
52.83
50.53
49.20
36.47
35.17
34.83
Table 2: Test accuracy (%) after the corresponding proper communication rounds and local epoch
E = 1, 10 respectively. We set the number of clients 100 and the participation ratio 0. 1. The batchsize is fixed as 50 and the heterogeneous dataset is divided as the Dirichlet-0.6 distribution.
CIFAR-10
CIFAR-100
Tinylmagenet
Method
E
E 5
= 10
E 1
E =
E 10
E 1
E =
E = 10
FedAvg
82.76
81.84
81.06
44.95
42.45
40.84
36.28
28.79
24.77
FedAdam
83.56
81.42
80.21
51.50
46.16
44.18
38.25
32.14
21.89
SCAFFOLD
85.03
84.12
83.54
48.35
48.85
43.25
37.21
34.87
26.75
FedCM
86.33
84.89
82.83
52.11
50.42
48.14
40.63
33.70
25.82
FedDyn
85.45
83.96
78.12
49.94
45.80
44.97
39.33
30.55
22.36
FedSAM
84.48
84.17
82.97
49.18
46.79
45.33
38.11
32.94
25.16
ours
86.68
85.18
84.13
55.77
51.37
50.13
40.95
36.82
28.32
to the global view. When the local epochs enlarged to 10, the long local update exacerbates the inconsistency problem and shows a negative impact on the test accuracy. Especially on the large Tinylmagenet dataset, most algorithms fail to converge at T 1000. Thus the test accuracy could be considered as the convergence rate for all the methods. FedMIM achieves the top-1 accuracy on both short epochs and long epochs. the local iteration, the inertial momentum which promotes the local consistency, plays important role in the stochastic gradient estimation. FedMIM obtains the iterative point closer the global iterative point via perturbing the local gradient, which approx- imates the global direction and updates it by one step gradient descent. This allows the local update be corrected not only on the gradient term, but also on the iterative points where the gradient calculated. In the next part, we will discuss the consistency between the baselines and some ablation studies, including the participation ratio, the selection of the a, and B, and the different multi-steps.
5.3 ABLATION STUDIES HYPER-PARAMETER SENSITIVITY
5.3.1 PARTICIPATION RATIO
We test the experiments on the CIFAR-10 dataset under different participation ratios, which are selected from 5%, 20% to test the convergence rate under the setups of fixed local epoch 5 and batchsize 50. The heterogeneity set as DIR-0.1. From the Figure 2 (a), when the heterogeneity is enlarged, the convergence speed of FedAvg loses the most performance. FedAdam, FedCM, and FedDyn show high sensitivity to the participation ratios. SCAFFOLD performs well and maintains the excellent generalization performance via the variance reduction technique under the higher participation option. Beneficial from the inertial momentum, the global direction could be exact estimated. And multi-steps calculation is adopted to further enhance the stability and smooth characteristic in the estimation. Our proposed FedMIM shows a very stable performance both on different heterogeneity and participation ratios, especially on the extreme heterogeneous settings.
8

---

Under review as a conference paper at ICLR 2023
SCAFOLD
SCAFFOLD
Fedbyn
poleey Menee
Pusft
Fedhag
Feddigg
Feeth
PSSCH
FeeDyn
reaDym
Rouals
Participation Ratios.
(b) Local Consistency.
Figure 2: (a) Comparison different participation ratios CIFAR- 10 dataset and (b) comparison on different methods for the local consistency. We fix the other hyper-parametert In (a), L/ shows the ratio equals to 5%/20% In (b), L/ shows the consistency under the CIFARIO/100 dataset.
5.3.2 SELECTION OF MULTI-STEPS
Table 3: Selection of aj and By.
We test the different selection of steps and selection of atj, Bj. The experimental setup is: local epochs 5, total communication rounds 500 and batchsize is fixed 50. When eralization performance. As shown in Table 3 we
02
B1
B2
Accuracy(%)
81.61
0.9
84. 14
FedMIM achieves the best gen- set a, and 3, FedAvg method. And if we set a2 = 0 and By = 0, the FedCM method. shows that the 3, with long history is not good selection for the local clients, due to the expired information be- fore the current time. Local updates will be misled by redundancy of the invalid offset. The adjacent update more relaxed, which can be searched from the last two three steps. In the empirical studies, recommend the selection can be decided by different indicators, a large a, and proper 3, are better.
0.5
83.53
0.8
0.1
84.29
FedMIM degenerates to the
0.3
84.67
0,3
0.6
83.86
FedMIM degenerates
0.6
0.3
0.9
0.1
84,98
0.6
0.3
0.5
0.5
84.42
0.6
0.3
0.1
0.9
84.04
most important. While the ay is
5.3.3 LOCAL CONSISTENCY
We test the consistency during the training as s E, ||x - where x the practical training, the local models can not approach the true local optimal due to the limitation of local interval K, thus all the x k will represent for the dispersion from the global model +1+ To keep the xf, K close to each other can improve the resistance to the local heterogeneity (the idealized case is that all local clients always generate the same parameters per round). Figure 2 (b) shows the empirical results of the consistency on the differrnnt dataset, FedMIM handles the more excellent efficiency on maintaining the local similarity than the other baselines the both CIFAR-10/100.
In
CONCLUSION
In this work, propose novel federated algorithm, named FedMIM, which adopts the multi- step inertial momentum to guide the local training on the heterogeneous clients both on the gra- dient estimation and the iterative point for gradient calculation. We also theoretically prove that the proposed FedMIM achieves O VSKT) under the smoothness assumptions and ((}) under the
(PL) inequality, under the non-convex cases. FedMIM can efficiently improve the local consistency to mitigate the influence from the heterogeneous dataset. We conduct extensive experiments demonstrate the significant performance of our proposed FedMIM the real-world dataset. Furthermore, we learn some ablation studies to verify the stability under different setups.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263 2021.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 2019.
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp. Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learning at scale: System design. Proceedings of Machine Learning and Systems, 1: 374 388, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scatt scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint #xXi:2002..07488 2020.
Yonghai Gong, Yichuan Li, and Nikolaos M Freris. Fedadmm: robust federated deep learning framework with adaptivity to system heterogeneity. arXiv preprint arXiv:2204.03529 2022.
Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. Fedboost: A communication-effiiien
algorithm for federated learning. In International Conference on Machine Learning, pp. 3973- 3983. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pp. 4387- 4398. PMLR, 2020.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:/909 06339, 2019.
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-iid data. In AAAI, pp. 7865-7873, 2021.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurelien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Ad- vances and open problems in federated learning. Foundations and Trends in Machine Learning. 14(1-2):1-21) 2021.
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebas- tian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008. 03606, 2020a.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143. PMLR, 2020b.
Gecho Kim, Jinkyu Kim, and Bohyung Han. Communication-efffcccinn federated learning with ac- celeration of global momentum. arXiv preprint arXiv:2201.03/72, 2022.
Jakub Konečný, Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492. 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Under review as a conference paper at ICLR 2023
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7);3, 2015.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-00 00.550066,,200 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Sys- tems, 2:429-44 450, 2020b.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis- Phili-p Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
Wei Liu, Li Chen, Yunfei Chen, and Wenyi Zhang. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems, 31 35):1175541776 2020.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficeen learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Aritra Mitra, Rayana Jaafar, George Pappas, and Hamed Hassani. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. Advances in Neural Information Processing Systems, 34:1 1406-14419, 2021.
Tomoya Murata and Taiji Suzuki. Birianer reduced local sgd for less heterogeneous federated learning. arXiv preprint arXiv:2102.03/98 2021.
Emre Ozfatura, Kerem Ozfatura, and Deniz Gündüz. Fedade: Accelerated federated learning with drift control. In 2021 IEEE International Symposium on Information Theory (ISIT), pp. 467-472.
IEEE, 2021.
Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. arXiv preprint arXiv:2206.02618, 2022.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295. 2020.
Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Robust and communication-effiiian federated learning from non-iid data. IEEE transactions on neural net- works and learning systems, 31(9):3400-3411 2019.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau en- velopes. Advances in Neural Information Processing Systems, 33:21394-22 21405, 2020.
Quoc Tran Dinh, Nhan H Pham, Dzung Phan, and Lam Nguyen. Feddr-randomized douglas- rachford splitting algorithms for nonconvex federated composite optimization. Advances in Neu- ral Information Processing Systems, 34:30326-3000 30338, 2021.
Stacey Truex, Nathalie Baracaldo, Ali Anwar, Thomas Steinke, Heiko Ludwig, Rui Zhang, and Yi Zhou. A hybrid approach to privacy-preserving federated learning. In Proceedings of the 12th ACM workshop on artificial intelligence and security, pp. 1-11, 2019.
Han Wang, Siddartha Marella, and James Anderson. Fedadmm: A federated primal-dual algorithm allowing partial participation. arXiv preprint arXiv:2203. 2022.
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Slowmo: Improving
communication-efficeeent distributed sgd with slow momentum. arXiv preprint 10.064,9 2019.
Janyy Wang, Zheng Xu. Zachary Garrett, Zachary Charles, Luyang Liu, and Gauri Joshi. Local adaptivity in federated learning: Convergence and consistency. arXiv preprint arXiv:2106.02300, 2021a.
11

---

Under review as a conference paper at ICLR 2023
Jianyu Wang, Zheng Xu, Zachary Garrett, Zachary Charles, Luyang Liu, and Gauri Joshi. Local adaptivity in federated learning: Convergence and consistency. arXiv preprint arXiv:2106.02305, 2021b.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Jing Xu, Sen Wang, Liwei Wang, and Andrew Chi-Chih Yao. Fedem: Federated learning with client-level momentum. arXiv preprint arXiv:2 10874, 2021.
Runhua Xu, Nathalic Baracaldo, Yi Zhou, Ali Anwar, and Heiko Ludwig. Hybridalpha: An efficient approach for privacy-preserving federated learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, pp. 13-23, 2019.
Xinwei Zhang, Mingyi Hong, Sairaj Dhople, Wotao Yin. and Yang Liu. Fedpd: A federated learning framework with optimal rates and adaptivity to non-iid data. arXiv preprint arXiv:2005.
2020.
Haoyu Zhao, Konstantin Burlachenko, Zhize Li, and Peter Richtárik. Faster rates for compressed federated learning with client-variance reduction. arXiv preprint arXiv:21 13097, 2021a.
Haoyu Zhao, Zhize Li, and Peter Richtárik. Fedpage: A fast local stochastic gradient method for communication-cfie -effeicinntt federated learning. arXiv preprint 200.:22000.4755
12


Reviews:
[{"review_id": "iWhV-ZkEXrj", "reviewer": "Reviewer_qspk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new federated learning algorithm, named FedMIM, which adopts the multi-step inertial momentum\non the edge devices and enhances the local consistency for free during the training to improve the robustness of the heterogeneity.\n\nIn terms of theory the paper, the paper proves that FedMIM achieves the $O(1/\\sqrt{SKT})$ convergence rate under the nonconvex setting, where $S$ is the number of selected clients and $K$ is the proper local interval in each communication round. For non-convex function under PL condition, a convergence rate $O(1/T)$ was also proved for FedMIM. \n\nThe paper conduct comprehensive experiments on various real-world datasets and demonstrate the efficacy of the proposed FedMIM against several state-of-the-art baselines.", "summary_of_the_review": "As I mentioned above, the paper is well written and the main contributions of the work are well presented.\n\nI give a score of \"6: marginally above the acceptance threshold\" to this submission. \n\n", "strength_and_weaknesses": "The paper is well written and the main contributions of the work are well presented. I went through the main proofs and the important steps looks also correct.\n\nThe paper is easy to follow and the main contributions are clear. \n\nSuggestion: It would be nice at the point where FedMIMN proposed a pseudocode to explain how one can obtain the classical Local SGD as a special case (if this is possible). \n\nMissing references: Closely related papers are [1] and [2] below. \n\nQuestions: \n1. Also to the best of my knowledge one of the state-of-the-art methods for solving non-convex problems is the $\\tau$-overlap SGP from paper [2]. How your approach is related to this method? \n\n2. SlowMo from the paper Wang et al. (2019), which is already cited in the paper, is also a fast method. Why is not used in the experiments? \n\nMissing References:\n\n[1] Koloskova, Anastasia, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. \"A unified theory of decentralized sgd with changing topology and local updates.\" In International Conference on Machine Learning, pp. 5381-5393. PMLR, 2020.\n\n[2] Assran, Mahmoud, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. \"Stochastic gradient push for distributed deep learning.\" In International Conference on Machine Learning, pp. 344-353. PMLR, 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "c8bzh2kIJh", "reviewer": "Reviewer_YuXz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new federated learning algorithm (i.e., FedMIM),  which adopts the multi-step inertial momentum on the edge devices and enhances the local consistency for free during the training  to improve the robustness of the heterogeneity. Theoretically, it proved that \nthe FedMIM algorithm achieves the best known convergence rate with a linear speedup property with respect to the number of selected clients and proper local interval in each communication round under the nonconvex setting. Empirically, it conducted comprehensive experiments on various real-world datasets and demonstrate the efficacy of the proposed FedMIM against several state-of-the-art baselines.\n", "summary_of_the_review": "This paper proposed a new federated learning algorithm (i.e., FedMIM),  which adopts the multi-step inertial momentum on the edge devices and  enhances the local consistency for free during the training  to improve the robustness of the heterogeneity. Theoretically, it proved that  the FedMIM algorithm achieves the best known convergence rate with a linear speedup property with respect to the number of selected clients and proper local interval in each communication round under the nonconvex setting. Empirically, it conducted\ncomprehensive experiments on various real-world datasets and demonstrate the efficacy of the proposed FedMIM against several state-of-the-art baselines.\n\nSome Comments:\n\n1. The motivation of this paper is not clear. Why could multi-step inertial momentum on the edge devices \n enhance local consistency ?\n\n2. What is the parameter $I$ in the FedMIM algorithm ？\n\n3. How to choose the tuning parameters $\\alpha_j$ and $\\beta_j$ in the FedMIM algorithm ?\n\n4. Reproducibility is a crucial aspect of any work based on experiments. \nYour code should be available to the reviewers throughout all stages of the submission and publicly.", "strength_and_weaknesses": "Strength:\n\nThis paper proposed a new federated learning algorithm (i.e., FedMIM),  which adopts the multi-step inertial momentum on the edge devices and  enhances the local consistency for free during the training  to improve the robustness of the heterogeneity. Theoretically, it proved that \nthe FedMIM algorithm achieves the best known convergence rate with a linear speedup property with respect to the number of selected clients and proper local interval in each communication round under the nonconvex setting. \n\nWeakness:\n\nThe novelty of this paper is limited. This paper basically follows the existing Nesterov' momentum techniques.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SHY1et_NFc", "reviewer": "Reviewer_WgvB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work propose a federated learning algorithm, named FedMIM, which adopts the multi-step inertial momentum on the edge devices and enhances the local consistency for free during the training to improve the robustness of the heterogeneity. Specifically, the authors incorporate the weighted global gradient estimations as the inertial correction terms to guide both the local iterates and stochastic gradient estimation, which can reckon the global objective optimization on the edges’ heterogeneous dataset naturally and maintain the demanding consistent iteration locally.  ", "summary_of_the_review": "federated learning is a great area to study. this work adopts the multi-step inertial momentum on the edge devices to improve the robustness of the heterogeneity.  However, the novelty of the paper is not well-established. ", "strength_and_weaknesses": "strength: multi-step inertial momentum on the edge devices to enhances the local consistency\n\nweakness: lack of novelty compared to the server side momentum (Karimireddy et al., 2020b) or client side momentum Xu et al. (2021) ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject