Explicit Personalization and Local Training: Double Communication Acceleration in Federated Learning
Authors: Kai Yi
Laurent Condat
Peter Richtárik
Venue: Accepted by TMLR ()

Abstract: Federated Learning is an evolving machine learning paradigm, in which multiple clients perform computations based on their individual private data, interspersed by communication with a remote server. A common strategy to curtail communication costs is Local Training, which consists in performing multiple local stochastic gradient descent steps between successive communication rounds. However, the conventional approach to local training overlooks the practical necessity for client-specific personalization, a technique to tailor local models to individual needs. We introduce Scafflix, a novel algorithm that efficiently integrates explicit personalization with local training. This innovative approach benefits from these two techniques, thereby achieving doubly accelerated communication, as we demonstrate both in theory and practice.

Published in Transactions on Machine Learning Research (05/2025)
Explicit Personalization and Local Training: Double Com-
munication Acceleration in Federated Learning
Kai Yi
kai. Koi.yykkaust.ccm www.eiassstueuuu....
Deportment of Computer Science
King Abdullah University of Science and Technology (KAUST)
Laurent Condat
leurent..cmdatt@aaeee
Deportment of Computer Science
King Abdullah University of Science and Technology (KAUST)
POAIA-KAUSS Center of Encellence in Data Science and Artificial Intelligence SEDAIA-KAUSI A1)
Peter Richtárik
yeler.richlrrikkkauss....e
Department of Computer Science
King Abdullah University of Science and Technology (KAUST)
SDA AAASSTKAST Center of Racellence in Data Science and Artificial Intelligence (SDATA-KAUST AT)
Reviewed on OpenReview: https: // openreview. net/ porm/o id- i qVUEuhlaEa
Abstract
Federated Learning is an evolving machine learning paradigm, in which multiple clients perform computations based on their individual private data, interspersed by commu- nication with a remole server. A common strategy to curtail communication costs is Local Training, which consists in performing multiple local stochastic gradient descent steps between successive communicatio rounds. However, the conventional approach to local training overlooks the practical necessity for client-specific personalization, tech- nique to tailor local models to individual needs. We introduce Scafflix, novel algorithm that efficiently integrates explicit personalization with local training. This innovative ap- proach benefits from these two techniques, thereby achieving doubly accelerated communi- cation, as we demonstrate both in theory and practice. The code is publicly available at httpp://ght /gtgggttubb.../ http:////ww.llaamm////ffaa.a.
1 Introduction
Due to privacy concerns and limited computing resources on edge devices, centralized training with all data first gathered in a datacenter often impossible in many real-world applications. So, Federated Learning (FL) has gained increasing interest as a framework that enables multiple clients do local computations, based on their personal data kept private, and to communicate back and forth with a server. FI. is classically formulated an empirical risk minimization problem of the form
min DER
f(x)
M(x))
(ERM)
where fi is the local objective on client i, n is the total number of clients, x is the global model.
Thus, the usual approach is to solve (ERM) and then to deploy the obtained globally optimal model x* arg min (xx all clients. To roduce communication costs between the server and the clients, the practice of updating the local parameters multiple times before aggregation, known as Local Training (LT) (Povey al., 2014; Moritz et al., 2016; McMahan et al., 2017; Lt al., 2020b; Haddadpour & Mahdavi, 2019; Khaled

---

Published in Transactions on Machine Learning Research (05/2025)
et al., 2019; 2020; Karimireddy et al., 2020; Gorbunov et al., 2020a; Mitra et al., 2021), is widely used in FL. LTT, in its most modern form, is communicatoonaccceettiin mechanism, as we detail in Section A.1.
Meanwhile, there is a growing interest in providing personalization to the clients, by providing them more- or-less customized models tailored to their individual needs and heterogeneous data, instead of the one-size- fits-all model x². We review existing approaches to personalization in Section A.2. Tf personalization is pushed to the extreme, every client just uses its private data to learn its own locally-optimal model
z = arg min f((z) #ERd
and no communication at all is needed. Thus, intuilively, more personalization means less communication needed to reach a given accuracy, In other words, personalization is a caanuuiication-acclleattte mecha- nism, like LT.
Therefore, we raise the following question: Is possible to achieve double communnication acceleration in FL by jointly leveraging the acceleration potential of personalization and local truining?
For this purpose, we first have to formulate personalized FL an optimization problem. A compelling interpretation of LT (Hanzely & Richtárik, 2020) is that it amounts to solve implicit personalization objective of the form:
1 a
A
min Fi ERd T
Σ Sa(ss)
+
1
(1)
2m
where 24 E IRd denotes the local model at client E |n] (I...., -= Ei= Tj is the average of these local models, and A 2 0 is the implicit personalization parameter that controls the amount of personalization. When À is small, the local models tend to be trained locally. On the other hand, larger À puts more penalty on making the local models I; close to their mcan F, or equivalently in making all models close to each other, by pushing lowards averaging over all clients. Thus, LT is not only compatible with personalization, but can be actually used to implement it, though implicitly: there a unique parameter À in equation and it is difficult evaluate the amount of personalization for a given value of X.
The more accurate FLIX model for personalized FL was proposed by Gasanov et al. (2022). It consists for every client to first compute locally its personally-optimal model x;, and then to solve the problem
f(z) = 1
min ZERd
ft fasa + (1-aa)=t))
(FLIX)
where E [0, 1] is the explicit and individual personalization factor for client i. At the end, the personalized model used by client is the explicit mixture
x =,2"" (1-aa));;
where xx is the solution to (FLIX). A smaller value of a gives more weight to x;, which means more personalization. The extreme case a; = 0 simply means using fully personalized model without any commu- nication. On the other hand, if a; = 1, the client i uses the global model x* without personalization. Thus, if all a; are equal to 1, there is no personalization at all and (FLIX) reverts to (ERM). So, (FLIX) is a more general formulation of FI. than (ERM). The functions from the fs, so every algorithm appropriate for (ERM) can also be applied to solve (FLIX). Gasanov et al. (2022) proposed an algorithm also called FLIX to solve (FLIX), which is simply vanilla distributed gradient descent (GD) applied to (FLIX).
(FLIX) inherit smoothness and strong convexity
In this paper, we first redesign and generalize the recent Scaffnew algorithm (Mishchenko et al., 2022), which features LT and has an accelerated communication complexity, and propose Individdualizeed-Safleee (i- Scaffnew), wherein the clients can have different properties. We then apply and tune i-Scaffnew for the problem (FLIX) and propose our new algorithm for personalized FI., which we call Scafflix. We answer positively the above question and prove that Scafflix enjoys a doubly accelerated communication complexity, by jointly harnessing the acceleration potential of LT and personalization. That is, its communication complexity
2

---

y is used by the server at Step not needed if a, Published in Transactions on Machine Learning Research (05/2025)
Algorithm Scafflix for (FLIX)
1: input: stepsizes > O,..., > 0; probability E (0, 1}; initial estimates zf...., In E E and
if....,he hg E IRd such that Ei=1 h2 = 0, personalization weights on,..., On
the server, 7 = t Ei= of?;
clients in parallel, x == arg min f for t = 0, 0...... do
5: flip a coin 0f = with probability 0 otherwise] at clients in parallel, do
6: for i = =.=1....
7:
a + (1 0s)ay
estimate of the personalized model #]
8:
compute an estimate of of fii))
1 = - 7i
a. (gt kt)
local SGD step
10:
if = then
send a
7: o to the server, which aggregates z²:= Σ== a? 2/ and broadcasts it to all clients communication, but only with small probability p
11:
12:
x²
13:
h
h h
pa. 1 24)
update of the local control variate
14:
else
15:
1+1 a
16:
hg = h
17:
end if
18:
end for
19: end for
depends on the square root of the condition number of the functions fi and on the og In addition to establishing the new state of the art for personalized FL with our theoretical guarantees, we show by extensive experiments that Scafflix is efficient in real-world learning setups and outperforms existing algorithms.
Our approach is novel and its good performance built on a solid theoretical foundation. We stress that our convergence theorem for Scafflix holds under standard assumptions, without bounded variance or any other restriction. By way of comparison with recent works, pFedGate (Chen et al., 2023) bases its theorem bounded diversity assumption, which is often unrealistic for non-iid FL. Neither FedCR (Zhang et al., 2023) nor FedGMM (Wu et al., 2023) comes with conventional convergence theory. pFedGraph (Ye et al., 2023) and FED-PUB (Baek et al., 2023) also lack a solid convergence analysis.
the
2 Proposed Algorithm Scafflix and Convergence Analysis
We generalize Scaffnew (Mishchenko et al., 2022) and propose wowriddualized-SSfffeee (i-Scaffnew). shown as Algorithm 2 in the Appendix. Its novelty with respect to Scaffnew make use different stepsizes % for the local SGD steps, in order to exploit the possibly different values of L; and μ4, as well as the different properties A; and C of the stochastic gradients. This change not straightforward and requires to rederive the whole proof with different Lyapunov function and to formally endow R2 with different inner product at every client.
We then apply and tune i-Scaffnew for the problem (FLIX) and propose our new algorithm for personalized FL, which we call Scafflix, shown as Algorithm 1.
We analyze Scafflix in the strongly convex case, because the analysis of linear convergence rates in this setting gives clear insights and allows the best of our knowledge, there is no analysis of Scaffnew in the nonconvex setting. But conduct several nonconvex deep learning experiments to show that our theoretical findings also hold in practice.
deepen our theoretical understanding of LT and personalization. And to
3

---

Published in Transactions on Machine Learning Research (05/2025)
Assumption 1 (Smoothness and strong convexity). In the problem (FLIX) (and (ERM) as the particular case 1), we assume that for every i E [n], the function Ji is Lı-smooth and f4-strongly convex, 1 for some Li 2 > 0. This implies that the problem is strongly convex, so that its solution x' exists and is unique.
We also make the two following assumptions on the stochastic gradients of used in Scaffli (and i-Scaffnew particular case with a; = 1).
Assumption (Unbiasedness). We assume that for every 2 0 and e [n], of is an unbiased estimate of f(();; that is,
V (.
To characterize unbiased stochastic gradient estimates, the modern notion of expected smoolhness is well suited (Gower et al., 2019; Gorbunov et al., 2020b):
Assumption (Expected smoothness). We assume that, for every E [n], there exist constants A, 2 L and C, such that, for every 2 0,
|oot Vf(()))) at s , + C₁,
(2)
where Dx(x,x') = f(x) - [(x) -V x²) 0 denotes the Bregman divergence of function yp points I, Rd.
Thus, unlike the analysis Mishchenko et al. (2022, Assumption 4.1), where the same constants are assumed for all clients, since we consider personalization, we individualize the analysis: we consider that each client be different and use stochastic gradients characterized by its own constants A, and C.- This is more representative of practical settings. Assumption 3 is general and covers in particular the following two important cases (Gower et al., 2019):
1. (bounded variance) If of is equal to [fi((() plus a zero-mean random error of variance of (this covers the case of the exact gradient of = Vf((f() with o, = 0), then Assumption is satisfied with A; == Li and C. =
2. (sampling) TI St Ej= Jiz for some T--smooth functions Jig and of = Vees for some jr chosen uniformly at random in [ni], then Assumption 3 is satisfied with A, 2L; and
G = (2, EE= [[;)), [V;(*)))] (this can be extended minibatch and nonuniform sampling).
We now present main convergence result:
Theorem 4 (fast linear convergence). In (FLIX) and Scafflix, suppose that Assumptions 1, 2, 3 hold and that for every E [n). < 7 S A. For every 2 0, define the Lyapunov function
V
1
Tmin
|
7min 1 Σ 2 n 1
-/
n
78
AA function R: - is said to be L-smooth it is differentiable and its gradient is Lipschitz continuous with constant L; that is, for every i E Rd and y Rd, [[V](x) - V/(5)) S Ll|x - -vlli where, here and throughout the paper, the norm the Euclidean norm. f is said to be p-strongly convex if f #ll I1 is convex. We refer to Bauschke Combettes (2017) for such standard notions of convex analysis.
We suppose A: 2 Li. Indeed, have the bias-variance decomposition E e|[lst VA
(() V + E Ello VAGE |
> |VAEE) VA Assuming
the best known smooth-
ness constant of f:, we cannot improve the constant L; such that for every E Rd, [[V VA(E))| (x, Therefore, in equation 2 has be L..
4

---

Published in Transactions on Machine Learning Research (05/2025)
where Yunin mi] Y/- Then Scafflix converges linearly: for every t 0,
E VV S (1
<)
7min 1
n Σ nCC
(4)
where
Ç - min
7p p²) sE|n]
min
(5)
It important to note that the range of the stepsizes %h the Lyapunov function v and the convergence rate in equation 4 equation do not depend on the personalization weights as; they only play a role in the definition of the personalized models at and zÿ- Indcod, the convergence speed essentially depends on the conditioning of the functions x Ja (aar + -x)=;); which are independent from the 0ş. More precisely, let us define, for every i [n],
Li 21 and P4
Fimax
max KG.
icin|
and let us study the complexity of of Scafflix reach e-accuracy, i.e. E[V") S. If, for every i E [n], C₁ 0, A, O(L)), and = 0 = 0 Z,)) the iteration complexity of Scafflix is
o (( KKmax + pr
1 log( Dog(v'=)))
(6)
And since communication occurs with probability p, the communication complexity of Scafflix is
o ((Prmaa + ) log log( (eg(V9--))
(7)
Note that Kmax can be much smaller than Kglobal in the rate of Scaffnew with y A, Thus, Scafflix is much more versatile and adapted to FL with heterogeneous data than Scaffnew.
max, L, min; ju which the condition number that appears
Corollary 5 (case C; = 0). In the conditions of Theorem 4, if 0 (Hims)) and, for every E [n], C; 0, A; = OLL), and = O ) = =,+,), the communication complerity of Scafflix is
① (Vmax Log(TV= (V)-))
(8)
Corollary 6 (general stochastic gradients). In the conditions of Theorem if /mingeta) mi and, for every i E [n],
min ( A,' 2C,
6fmmin
(9)
(or V
1 A. if C2 0), where Jmmi minje(n] M4j the iteration complerity of Scafflix is
(( max max (E(N)
As
C.
))
log( (og(WeEE))
=0 max
max
C2 (E(x] ii VE EpminAli
A
log(w(DE1)
(10)
Effmintt
and its communication complexity is
max
max kE(n
A; max sc[n]
C₁ epminnt
log(
log (00
(11)
A4
I A, = O(L)) uniformly, we have max;e[n] nication complexity of Scafflix is accelerated, it depends on /mmax and ve
A Ps
((/imax) Thus, we see that thanks to LT, the commu-
5

---

Published in Transactions on Machine Learning Research (05/2025)
mushrooms
a6a
w6a
10a
*
10a
10
10
10a
10:11
10a44
10:44
10:17
-
-
10
1e-1
10:00
1e-1
10
1e-1
++
Ie:2
10
-p-
le.3
10ae
le-J
23
1e.4
10.4
1e-4
0
10
20
30
10
20
30
0
10
20
30
109
10a
10:4
10
10:5
103
10
10
10.9.
10
10:13
10:43
10
10
10:18.
20
10a
20.3.
20 Communication rounds
10 Communication rounds
Communication rounds
Figure 1: The objective gap fx**) and the squared gradient norm [[V)(=)] against the number of communication rounds for Scafflix and GD on the problem (FLIX) on class-wise non-iid FL setting, We set all to the same value for simplicity. The dashed line represents GD, while the solid line represents Scafflix. We observe the double communication acceleration achieved through explicit personalization and local training, Specifically, (a) for a given algorithm, smaller ass (i.c. more personalized models) lead to faster convergence; (b) comparing the two algorithms, Scafflix faster than GD, thanks to its local training mechanism.
In the expressions above, the acceleration effect of personalization is not visible: it is "hidden" " in To because every client computes xf but what matters is its personalized model o and || - 2t1l af || xx 11 In particular, assuming that xf 2n = and he = f.?)), we have
0
< Tmin
(1 (1+280 71 p2
I mmax at) 7min 1|
L 12
and we see that the contribution of every client to the initial gap To is weighted by of- Thus, the smaller the a; the smaller y and the faster the convergence. This why personalization an acceleration mechanism in our setting.
3 Experiments
We first consider a convex logistic regression problem show that the empirical behavior of Scafflix is in accordance with the theoretical convergence guarantees available in the convex case. Then, make extensive experiments of training neural networks large-scale distributed datasets.
3.1 Prelude: Convex Logistic Regression
We begin ovaluation by considering the standard convex logistic regression problem with la regularizer. This benchmark problem is takes the form (ERM) with
1 ne. log (1 + exp(- (bij dis)) + ||=r12 n;
f(x)=
Σ
where μ represents the regularization parameter, n4 is the total number of data points present at client i; aij are the training vectors and the bi,j E (-1,1} are the corresponding labels. Every function fi is p-strongly convex and La-smooth with L An; E || + p. We set μ to 0.1 for this experiment. We employ the

---

Published in Transactions on Machine Learning Research (05/2025)
FEMNIST
Shakespeare
0.9
FedAvg
0.5
test accuracy
0.8
test accuracy
FLIX, a= 0.3
Scafflix, a a= 0.3
0.4
0.7
FedAvg
0.3
0.6
FLIX, a a= 0.5
Scafflix, 0.5
0.5
0.2
0
100
200
300
0
250
500
750
1000
communication rounds
communication rounds
Figure 2: Comparative generalization analysis with baselines. We set the communication probability to 0.2. The left figure corresponds to the FEMNIST dalaset with = 0.5, while the right figure corresponds to the Shakespeare dataset with a = 0.3.
mushrooms, a6a, and w6a datasets from the LibSVM library (Chang & Lin, 2011) to conduct these tests. We consider several non-iid splits and present the results on feature-wise non-iid in Figure 1. We discuss the difference among non-iid settings and complementary results in Appendix E.1.
The data is distributed evenly across all clients, and the a set to the same value. The results shown in Figure 1. We can observe the double acceleration effect of our approach, which combines explicit person- alization and accelerated local training. Lower a values, ie. more personalization, yield faster convergence for both GD and Scafflix. Moreover, Scafflix much faster than GD, thanks to its specialized local training mechanism.
3.2 Neural Network Training: Datasets and Baselines for Evaluation
To assess the gencralization capabilitics of Scafflix, undertake comprehensive evaluation involving the training of neural networks using two wedely-recognized large-scale FL datasets.
Datasets. Our selection comprises two notable large-scale FL datasets: Federated Extended MNIST (FEM- NIST) (Caldas et al., 2018), and Shakespeare (McMahan et al., 2017). FEMNIST is a character recognition dataset consisting of 671,585 samples. In line with the methodology described in FedJax (Ro al., 2021), we distributed these samples across 3,400 devices, with each device exhibiting a naturally non-iid characteristic, employ Convolutional Neural Network (CNN) model, featuring two convolutional
For all algorithms, layers and one fully connected layer. The Shakespeare dataset, used for next character prediction tasks, contains total of 16,068 samples, which we distribute randomly across 1,129 devices. For all algorithms applied to this dataset, we use a Recurrent Neural Network (RNN) model, comprising two Long Short- Thort-Term Memory (LSTM) layers and one fully connected layer.
Baselines. The performance of our proposed Scafflix algorithm benchmarked against prominent baseline algorithms, specifically FLIX (Gasanov et al., 2022) and FedAvg (McMahan et al., 2016). The FLIX algo- rithm optimizes the FLIX objective utilizing the SGD method, while FedAvg is designed to optimize the ERM objective. We employ the official implementations for these benchmark algorithms. Comprehensive hyperparancter tuning is carried out for all algorithms, including Scafflix, to ensure optimal results. For both FLIX and Scafflix, local training is required to achieve the local minima for each client. By default, we set the local training batch size at 100 and employ SGD with a learning rate selected from the set C= (10-5, 10-4 .... ,1). Upon obtaining the local optimum, we exccute each algorithm with a batch size of 20 for 1000 communicatin rounds. The model's learning rate is also selected from the C.. All the experiments were conducted a single NVIDIA A100 GPU with 80GB of memory.

---

Published in Transactions on Machine Learning Research (05/2025)
Table Numerical comparison with baselines.
Method
Local Training
Objective
FEMNIST
Shakespeare
FedAvg
Basic
(ERM)
85.25
38.67
FLIX
Basic
Personalized (FLIX)
87.18
43.40
Scaffnew
Accelerated
(ERM)
87.73
51.66
Scafflix
Accelerated
Personalized (FLIX)
89.43
54.21
FEMNIST
FEMNIST
FEMNIST
0.900
0.85
test accuracy
test accuracy
0.88
test accuracy
0.875
0.80
0.850
Scafflix,
0.75
Scafflix, a
0.6
Scafflix,
0.825
Scafflix, a = 0.5
Scafflix, ==
0.70
Scafflix, p= 0.1
Scafflix,
Scafflix,
Scafflix, p 0.2
0.800
Scafflix, a = 0.9
0.4
Scafflix. -
0.65
Scafflix, p- p-
250 500
750 1000
200
400
600
250 500 750 1000
communication rounds
communication rounds
communication rounds
(a)
rS
(c) ps
Figure 3: Key ablation studies: (a) evaluate the influence on personalization factor o, (b) examinate the effect of different numbers of clients participating to communication, (c) compare different values of the communication probability p.
3.3 Analysis of Generalization with Limited Communication Rounds
In this section, we perform in-depth examination of the gencralization performance of Scafflix, particularly in scenarios with a limited number of training epochs. This investigation is motivated by theoretical evidence of the double acceleration property of Scafflix. To that aim, we conduct experiments on both FEMNIST and Shakespeare. These two datasots offer a varied landscape of complexity, allowing for a com- prehensive evaluation of our algorithm. order to ensure a fair comparison with other baseline algorithms, we conducted an extensive search of the optimal hyperparameters for each algorithm. The performance assessment of the generalization capabilities was then carried out on a separate, held-out validation dataset. The hyperparameters that gave the best results in these assessments were selected as the most optimal set.
In order to examine the impact of personalization, we assume that all clients have same select a in {0.1, 0.3, 0.5, 0.7, 0.9). We present the results corresponding to a = 0.1 in Figure 2. Additional comparative analyses with other values of a are available in the Appendix. As shown in Figure 2, is clear that Scafflix outperforms the other algorithms in terms of generalization on both the FEMNIST and Shakespeare datasets. Interestingly, the Shakespoare dataset (ncxt-word prediction) poses a greater challenge compared to the FEMNIST dataset (digit recognition). Despite the increased complexity of the task, Scafflix not only delivers significantly better results but also achieves this faster. Thus, Scafflix is superior both in speed and accuracy.
and we
For additional clarity, we provide numerical results comparing our approach with FedAvg, FLIX, and Scaffnew, as summarized Table To align with Figure 2, we set a 0.5 for FEMNIST and a = 0.3 for Shakespeare, reporting the former after 300 communication rounds and the latter after 1,000 rounds. Notably, the advan- tages of accelerated local training are even more pronounced on the Shakespeare dataset an observation that warrants further investigation.
8

---

Published in Transactions on Machine Learning Research (05/2025)
mushrooms
a6a
w6a
10
1e-01
101
10a
1e-05
10
16:06
10a
10-3
10
1005
104
10:44
[
-X-- 0. 1840
10
-X- 1e+0
10a
10
== 1e-2
10-12
+ a= 1e22
10
1e.
= 1e.3
10-15
1e4
20
60
100
0
10
20
30
10a
10a
10-a
10.5
10a
107.
10:10-
10-7
10-11
10-15
10
10:25
10:20
10
200
10:29
20.2
20
40
60
80
100
0
10
20
30
10
20
30
Communication rounds
Communication rounds
Communication rounds
Figure 4: Inexact approximation of the local optimal.
Figure 5: Comparison between global stepsize (dashed lines) and indi- vidual stepsizes (solid lines).
3.4 Key Ablation Studies
In this section, we conduct several critical ablation studies to verify the efficacy of our proposed Scafflix method. These studics investigate the optimal personalization factor for Scafflix, assess the impact of the number of clients per communication round, and examine the influence of the communication probability p in Scafflix.
Optimal Personalization Factor. In this experiment, we explore the effect of varying personalization factors on the FEMNIST dataset. The results are presented in Figure 3a. We set the batch size to 128 and determine the most suitable learning rate through a hyperparameter search. We consider linearly increasing personalization factors within the set {0.1, 0.3, 0.5, 0.93. An exponential scale for a is also considered in the Appendix, but the conclusion remains the same.
We note that the optimal personalization factor for the FEMNIST dataset is 0.3. Interestingly, personaliza- tion factors that yield higher accuracy also display a slightly larger variance. However, the overall average performance remains superior. This is consistent with expectations as effective personalization may empha- size the representation of local data, and thus, could be impacted by minor biases in the model paramoters received from the server.
Number of Clients Communicating per Round. In this ablation study, we examine the impact of varying the number of participating clients in each commuicatiin round within the Scafflix framework. By default, we set this number to 10. Here, we conduct extensive experiments with different client numbers per round, choosing t from {1,5, 10, 20). The results are presented in Figure 3b. We can observe that Scafflix shows that for larger batch sizes, specifically 10 and 20, demonstrate slightly improved generalization performance.
Selection of Communication Probability p. In this ablation study, we explore the effects of varying the communicatio probability in Scafflix. We select from {0.1, 0.2, 0.5), and the corresponding results are shown in Figure 3c. We can clearly see that a smaller value of p, indicating reduced communication, facilitates faster convergence and superior generalization performance. This highlights the benefits of LT, which not only makes FL faster and more commuiiationn-fliiieen,, but also improves the learning quality.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Inexact Local Optimal. In FL, the primary challenge lies in minimizing communication overhead while effectively managing local computation times. Attaining a satisfactory local optimum (or approximat ion) for each client is both practical and similar to pretraining for finding good initialization, a common practice in fields like computer vision and natural language processing. For instance, in our study of the Shakespeare dataset, distributed across 1,129 devices with over 16,000 samples, a mere 50 epochs of local training per client were necessary achieve optimal results, as demonstrated in Figure This efficiency stands stark contrast to traditional methods, which often require more than 800 communication rounds, each involving multiple local updates.
We further conducted detailed ablation studies on logistic regression to assess the impact of inexact local optimum approximation. A threshold was set such that [[V[ || < indicates a client has reached its local optimum, with the default set to 1e - 6. Our investigation focused on the consequences of using higher values. Appendix Figure 12 details the expected number of local iterations for 100 clients. Notably, an value of le 1 is found to be 23.55 times more efficient than c 1e 6. Additional results for 8 workers with a 0.1 are presented in Figure 4, showing that 1e provides satisfactory approximation. (We anticipate even lower computational cost for finding a local optimum approximation when the data per client is smaller.) Opting for c le - 1 is a viable strategy to reduce computation, while smaller values are advantageous for greater precision. To ensure that our initial x is not already near the optimum, we initialized each element of x! to 100. Additionally, we explored the number of local iterations required for achieving the optimal setting, ranging from [0, 5, 200, 1000), as depicted the right panel Figure 4. These findings underscore the need for balance between performance and computational costs. More comprehensive insights and results are provided in Appendix E.2.
Individual Stepsizes for Each Client. In our experiments, we initially assumed a uniform learning rate for all clients for simplicity. However, to more accurately represent the personalized approach of our method and to align closely with Algorithm 1, we explored different stepsizes for each client. Specifically, we set " = 1/,,, where L, denotes the smoothness constant of the function f; optimizing (FLIX). The impact of this variation demonstrated in Figure 5, which presents results using the mushrooms dataset. We observed that employing individual stepsizes generally enhances performance. This approach, along with global stepsize (indicated by dashed lines in the figure), both contribute to improved outcomes.
3.5 Comparisons with Personalized FL Baselines
While our research primarily seeks to ascertain the impact of explicit personalization and local training on communication costs, we recognize the interest the community for a broad comparative scope. Accordingly, we have included extensive baseline comparisons with other recent FL and particularly personalized FL (pFL) methodologies. A comparative performance analysis on popular datasets like CIFAR100 and FMNIST is presented below:
Table 2: Results of additional baselines.
Method
Ditto
FedSR-FT
FedPAC
FedCR
Scafflix
CIFAR100
58.87
69.95
69.31
78.49
72.37
FMNIST
85.97
87.08
89.49
93.77
89.62
We utilized the public code and adopted the optimal hyper-parameters from FedCR (Zhang et al., 2023). subsequently re-running and documenting all bascline performances under the 'non-iid' setting. Our proposed Scafflix algorithm was reported with a communication probability of p = 0.3 and spanmed 500 communication rounds. We set the personalization factor a at 0.3. Based on the results, when focusing solely on the gencralization (testing) performance of the final epoch, our method is on par with stato-of-the-an approaches such as FedPAC (Xu et al., 2023) and FedCR (Zhang et al., 2023). However, our primary emphasis lies in demonstrating accelerated convergence.
10

---

Published in Transactions on Machine Learning Research (05/2025)
4 Discussion
4.1 Significance and Novelty
While our approach builds on existing "5th- bith-generation' local training methods-mant of which employ control variates and exhibit (sub)linear convergence-it. introduces two key innovations that yield significant theoretical and practical benefits. First, individualized stepsizes allow each client to adapt updates to its own data distribution and local smoothness/ strong convexity constants, thus capturing heterogeneity more precisely than prior work employing single global stepsize. Second, explicit personalization via FI.IX is scamlessly integrated into these local updates, cnsuring that clicnt-spocific models converge faster while also maintaining strong global consensus.
At first glance, these modifications may appear incremental. However, historical precedents-such the adoption of Polyak's adaptive stepsize (Polyak, 1964), dropout (Srivastava et al., 2014), or batch normal- ization (IofTe & Szegedy, 201 (015))demostttaee that small algorithmic changes often unlock disproportionately large improvements. Analogously, our refinements enable a double communication acceleration, evidenced by communication complexity depending on Vemax rather than Kmax Beyond achieving faster theoret- ical rates, this synergy of local training and personalization simultancously reduces the initial error gap, enhancing real-world performance. We believe these contributions-both in handling heterogeneous data and attaining provably accelerated convergence distinguish our work from prior state-of-the-art federated learning algorithms.
4.2 Learning Rate Design
Most existing work on 5th-generation local training methods (e.g., (Mishchenko et al., 2022; Malinovsky et al., 2022; Condat et al., 2023b; Meinhardt et al., 2024)) relies single, global learning rate applied uniformly across all clients. While effective, such an approach does not adapt to varying local data geometry or allow for personalization at the client level. In contrast, our method employs individualized stepsizes % = 1/A,, where A, is a client-specific smoothness parameter (Theorem 4). This design is grounded in theoretical insights indicating that per-client rates can better capture heterogeneity and potentially yield faster convergence in practice.
We acknowledge that this choice of stepsize may appear straight however, it represents deliberate divergence from the convention of uniform, global learning rate. Although more sophisticated adaptive strategies (c.g., adaptive gradient methods) could be considered in future work, here we show that even a relatively simple rule-rooted rooted in individual smoothness constants establishes a strong theoretical and empirical baseline. Our findings suggest that principled, client-specific learning rates open new avenues for further personalization, and we anticipate subsequent studies to refine or extend these ideas.
4.3 Nonconvex Setting and Partial Client Participation
In Theorem 4, we establish linear convergence in the convex setting. To the best of our knowledge, there is no existing analysis of accelerated local training methods under standard assumptions in the nonconvex regime. Addressing this nonconvex case remains a significant challenge and beyond the scope of this work. Instcad, our primary objective is to evaluate the practical effectiveness of our approach. Thorefore, we conduct experiments in both convex and nonconvex settings, and the empirical results confirm the efficiency of our method.
Thcoretical analysis of partial client participation in accelerated local training methods is similarly chal- lenging. For more details on this topic, we refer interested readers to Grudzien et. al. (2022); Condat et al. (2023b). In this paper, we focus on whether double acceleration can be achieved through explicit personal- ization and local training. In Section 3.4, we examine the effect of varying the number of participating clients per round and observe that increasing client participation slightly improves the generalization performance of Scafflix.
11

---

Published in Transactions on Machine Learning Research (05/2025)
5 Conclusion
In the contemporary era of artificial intelligence, improving federated learning to achieve faster conver- gence and reduce communication costs is crucial to enhance the quality of models trained on huge and heterogeneous datasets. For this purpose, we introduced Scafflix, novel algorithm that achieves double communication accoleration by redesigning the objective to support explicit personalization for individual clients, while leveraging a state-of-hhe-rl local training mechanism. We provided complexity guarantees in the convex setting, and also validated the effectiveness of our approach in the nonconvex setting through extensive experiments and ablation studies. Thus, work is step forward the important topic of communicationncfficient federated learning and offers valuable insights for further investigation in the future. Future work could profitably explore the algorithm's 's robustness against adversarial attacks and its adapt- ability to differential privacy in federated settings, expanding its applicability and ensuring its resilience in more challenging scenarios. Additionally, exploring the potential of Scafflix federated fine-tuning of large language models could yield significant insights and contributions, further advancing the field.
Acknowledgments
The rescarch reported in this publication supported by funding from King Abdullah University of Science and Technology (KAUST): i) KAUST Baseline Research Scheme, ii) Center of Excellence for Generative AI, under award number 5940, iii) SDAIA-KAUST Center of Excellence in Artificial Intelligence and Data Science.
References
S. Alam, L. Liu, M. Yan, and M. Zhang. Fedrolex: Model-hettroogeneons federated learning with rolling sub-model extraction. preprint arXiv:2212.01548, 2022.
M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary. Federated learning with personalization layers. preprint 1rXiv:1912.00818, 2019.
J. Back, W. Jeong, J. Jin, J. Yoon, and S. Hwang. Personalized subgraph foderated learning. In Proc. of 40th Inl. Conf. Machine Learning (ICML), PMLR 202, pp. 1396 1415, 2023.
H. H. Bauschke and P. L. Combettes. Conver Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York, 2nd edition, 2017.
D. Bui, K. Malik, J. Goetz, H. Liu, S. Moon, A. Kumar, and K. G. Shin. Federated user representation learning. preprint MXXi:1909.11533: 2019.
S. Caldas, P. Wu, T. Li, J. Konečný, H. B. McMahan, V. Smith, and A. Talwalkar. LEAF: A benchmark for federated settings. preprint arXiv:1812.01097, 2018.
C.C. Chang and C.-J. Lin. LibSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27, 2011.
Chen, L. Yao, D. Gao, B. Ding, and Y. Li. Efficient personalized federated learning via sparse model- adaptation. preprint arXiv:2305.0277 2023.
Y. Chen, X. Qin, J. Wang, C. Yu, and W. Gao. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intelligent Systems, 34):):83 93, 2020.
L. Condat and P. Richtárik. RandProx: Primal-dual optimization algorithms with randomized proximal updates. In Proc. of Int. Conf. Learning Representations (TCLR), 2023.
L. Condat, I. Agarský, and P. Richtárik. Provably doubly accelerated foderated learning: The first thooreti- cally successful combination of local training and compressed communccation preprint Xxi::2210.13277 2022.
12

---

Published in Transactions on Machine Learning Research (05/2025)
L. Condat, I. Agarský, G. Malinovsky, and P. Richtárik. TAMUNA: Doubly accelerated federated learning with local training, compression, and partial participation. preprint arXim:2302.0995 2023a.
L. Condat, G. Malinovsky, and P. Richtárík. TAMUNA: Accelerated federated learning with local training and partial participation. preprint (ff::2300.09332 2023b.
C. T. Dinh, N. H. Tran, and T. D. Nguyen. Personalized federated learning with Moreau envelopes. In Proc. of Conf. Neural Information Processing Systems (NeurIPS), volume 33, pp. 21394-21 21405, 2020.
E. Gasanov, A. Khaled, S. Horváth, and P. Richtárik. Flix: A simple and communictionn-effieett alternative to local methods in federated learning. In Proc. of 24th Int. Conf. Artificial Intelligence and Stalistics (AISTATS); 2022.
E. Gorbunov, F.
Unified theory and new efficient methods. In Proc. of Conf. Neural Information Processing Systems (NewrIPS), 2020a.
Hanzely, and Richtárik. Local SGD:
E. Gorbunov, F. Hanzely, and P. Richtárik. A unified theory of SGD: Variance reduction, sampling, quanti- zation and coordinate descent. In Proc. of 23rd Int. Conf. Artificial Intelligence and Statistics (AISTATS), PMLR 108, 2020b.
R. M. Gower, N. Loizou, X. Qian, A. Sailanbayev, E. Shulgin, and P. Richtárik. SGD: General analysis and improved rates. In Proc. of 36th Inl. Conf. Machine Learning (ICML), PMLR 97, pp. 5200-5200000 2019.
M. Grudzieú, G. Malinovsky, and P. Richtárik. Can 5th Generation Local Training Methods Support Client Sampling? Yes! In Proc. of Int. Conf. Artificial Intelligence and Statistics (AISTATS), April 2023.
Michal Grudzien, Grigory Malinovsky, and Peter Richtárik. Can 5th generation local training methods support client sampling? yes! preprint or Xiv:2211 14970, 2022.
F. Haddadpour and M. Mahdavi. On the convergence of local descent methods in federated learning. preprint arXiv:1910.1444 14425, 2019.
F. Hanzely and P. Richtárik. Federated learning of a mixture of global and local models. preprint arXiv:2002.05516, 2020.
S. Horvath, S. Laskaridis, M. Almeida, I. Leontiadis, S. Venieris, and N. Lane. FjORD: Fair and accurate federated learning under heterogeneous targets with ordered dropout. In Proc. of Conf. Neural Information Processing Systems (NeurIPS), volume 34, pp. 12876-12889, 2021.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456, pmlr, 2015.
P. Kairouz et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1-21,, 2019.
S. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. Suresh. SCAFFOLD: Stochastic controlled averaging for on-device federated learning. In Proc. of Int. Conf. Machine Learning (ICML), 2020.
A. Khaled, K. Mishchenko, and P. Richtárik. First analysis of local GD on heterogencous data. paper 17Xiv:1909.04711 presented at NeurIPS Workshop Federated Learning for Data Privacy and Confi- dentiality, 2019.
A. Khaled, K. Mishchenko, and P. Richtárik. Tighter theory for local SGD on identical and heterogeneous data. In Proc. of 23rd Int. Conf. Artificial Intelligence and Statistics (AISTATS), 2020.
D. Li and J. Wang. Fedmd: Heterogenous federated learning via model distillation. preprint arXiv:1910.08581, 2019.
Q. Li, B. He, and D. Song. Model-contrastive federated learning. In Proc. of IEEE/CVF Conf. Computer Vision and Pattern Recognition, pp. 10713 10713-10722, 2021.
13

---

Published in Transactions on Machine Learning Research (05/2025)
T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in hetero- geneous networks. Proceedings of Machine learning and systems, 2:429-444 -500, 2020a.
X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang. On the convergence of FedAvg on non-IID data. Proc. of Irt. Conf. Learning Representations (ICLR); 2020b.
G. Malinovsky, D. Kovalev, E. Gasanov, L. Condat, and P. Richtárik. From local SGD to local fixed point methods for federated learning. In Proc. of 87th Int. Conf. Machine Learning (TCML), 2020.
G. Malinovsky, K. Yi, and P. Richtárik. Variance reduced Proxskip: Algorithm, theory and application to federated learning. In Proc. of Conf. Neural Information Processing Systems (NeurIPS), 2022.
A. Maranjyan, M. Safaryan, and P. Richtárik. GradSkip: Commuiication local gradient methods with better computational complexity. preprint arXiv:2210.1641 2022.
B. McMahan, E. Moore, D. Ramage, and B. Agūira y Arcas. Federated learning of deep networks using model averaging. preprint arXiv:1602.05629, 2016.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communicttion-cffciine learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Stalistics (ATSTATS), 2017.
Y. Mei, P. Guo, M. Zhou, and V. Patel. Resource-adaptive federated learning with all-in-one neural compo- sition. In Proc. of Conf. Neural Information Processing Systems (NeurIPS), 2022.
Georg Meinhardt, Kai Yi, Laurent Condat, and Peter Richtárik. Prune at the clients, not the server: Accelcrated sparse training in federated learning. arXiv preprint (xXu:2405.2062)) 2024.
K. Mishchenko, G. Malinovsky, S. Stich, and P. Richtárik. ProxSkip: Yes! Local gradient steps provably lead to communication acceleration! Finally! In Proc. of 39th Int. Conf. Machine Learning (ICML), 2022.
A. Mitra, R. Jaafar, G. Pappas, and H. Hassani. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. In Proc. of Conf. Neural Information Processing Systems (NeurIPS). 2021.
P. Moritz, R. Nishihara, I. Stoica, and M. I. Jordan. SparkNet: Training deep networks in Spark. In Proc. of Int. Conf. Learning Representations (ICLR), 2016.
Boris Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics, 4(5):1-17, 1964.
D. Povey, X. Zhang, and S. Khudanpur. Parallel training of DNNs with natural gradient and parameter averaging. preprint 1idi:11100.4555 2014.
J. H. Ro, A. T. Suresh, and K. Wu. FedJAX: Federated learning simulation with JAX. preprint arXiv:2108.0211 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Πya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
J. Wang et al. field guide to federated optimization. preprint arXiv:2107.06917 2021.
Y. Wu, S. Zhang, W. Yu, Y. Liu, Q Gu, D. Zhou, H. Chen, and W. Cheng. Personalized federated learning under mixture of distributions. 2023. preprint arXiv:2305.01068.
J. Xu, X. Tong, and S.-L. Huang. Personalized foderated learning with feature alignment and classifier collaboration. preprint arXiv:2306.1188) 2023.
Yang, H. He, W. Zhang, and X. Cao. Fedsteg: A federated transfer learning framework for secure image steganalysis. IEEE Trans. Network Science and Engineering, 8(2):1084-108) 2020.
14

---

Published in Transactions on Machine Learning Research (05/2025)
R. Ye, Z. Ni, F. Wu, S. Chen, and Y. Wang. Personalized federated learning with inferred collaboration graphs. In Proc. of 40th Tnl. Conf. Machine Learning (TCML), PMLR 202, 2023.
H. Zhang, C. Li, W. Dai, Zou, and H. Xiong, FedCR: Personalized federated learning based on across- client common representation with conditional mutual information regularization. In Proc. of 40th Int. Conf. Machine Learning (ICML), PMLR 202, 2023.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Contents
1 Introduction
1
2 Proposed Algorithm Scafflix and Convergence Analysis
3
3 Experiments
6
3.1 Prelude: Convex Logistic Regression
6
3.2 Neural Network Training: Datasets and Baselines for Evaluation
3.3 Analysis of Generalization with Limited Communication Rounds
8
3.4 Key Ablation Studies
9
3.5 Comparisons with Personalized FL Baselines
10
4
Discussion
11
4.1 Significance and Novelty
11
4.2 Learning Rate Design
4.3 Nonconvex Setting and Partial Client Participation
Conclusion
12
A Related Work
17
A.1 Local Training Methods in Federated Learning
17
A.2 Personalization in FL
17
B Proposed i-Scaffnew Algorithm
18
C From i-Scaffnew to Scafflix
21
D Proof of Corollary
22
E Additional Experimental Results
22
E.1 Evaluating Logistic Regression under Non-IID Conditions
22
E.2 Inexact Approximation of Local Optimal..
27
16

---

Published in Transactions on Machine Learning Research (05/2025)
A Related Work
A.1 Local Training Methods in Federated Learning
Theoretical evolutions of LT in FL have been long-lasting, spanning five generations from empirical results to accelerated communication complexity. The celebrated FedAvg algorithm proposed by McMahan et al. (2017) showed the feasibility of communication-effcien learning from decentralized data. It belongs to the first generation of LT methods, where the focus was on empirical results and practical validations (Povey et al., 2014; Moritz et al., 2016; McMahan et al., 2017).
The second generation of studies on LT for solving (FRM) was based on homogeneity assumptions, such as bounded gradients (3c < to0, [V((()] S G, E R2, i E [nl) (Li et al., 2020b) and bounded gradient diversity ( E.= [z)] S c(((f())ll') (Haddadpour & Mahdavi, 2019). However, these assumptions are too restrictive and do not hold in practical FL settings (Kairouz et al., 2019; Wang et al., 2021).
The third generation of approaches, under generic assumptions on the convexily and smoothness, exhibited sublinear convergence (Khaled et al., 2019; 2020) or linear convergence to neighborhood (Malinovsky et al.,
2020).
Recently, popular algorithms have emerged, such as Scaffold (Karimireddy et al., 2020), S-Local-GD (Gorbunov et al., 2020a), and FedLin (Mitra et al., 2021), successfully correcting for the client drift and enjoying lincar convergence to an exact solution under standard assumptions. However, their communication complexity remains the same as with GD, namely Ox loge")), where K == L/p is the condition number.
Finally, Scaffnew was proposed by Mishchenko et al. (2022), with accelcrated communication complexity O O(Vllog This is a major achievement, which proves for the first time that LT is a communication acceleration mechanism. Thus, Scaffnew is the first algorithm in what can be considered the fifth generation of bT-based methods with accelerated convergence. Subsequent works have further extended Scaffnew with features such as varianco-reduced stochastic gradients (Malinovsky et al., 2022), compression (Condat et al., 2022), partial client participation (Condat et. al., 2023a), asynchronous communication of different clients (Maranjyan et al., 2022), and to general primal dual framework (Condat & Richtárik, 2023). The fifth generation of LT-baaed methods also includes the 5GCS algorithm (Grudzien et al., 2023), based on a different approach: the local steps correspond to an inner loop to compute a proximity operator inexactly. Our proposed algorithm Scafflix generalizes Scaffnew and enjoys even better accelerated communccation complexity, thanks to a better dependence on the possibly different condition numbers of the functions fi.
A.2 Personalization in FL
We can distinguish three main approaches to achieve personalization:
a) One-stage training of a single global model using personalivation algorithms. One common scheme is to design suitable regularizer to balance between current and past local models (Li et al., 2021) or between global and local models (Li et al., 2020a; Hanzely & Richtárik, 2020). The FLIX model (Gasanov et al., 2022) achieves explicit personalization by balancing the local and global model using interpolation. Meta-learning is also popular in this area, as evidenced by Dinh et al. (2020), who proposed a federated meta-learning framework using Moreau envelopes and a regularizer to balance personalization and generalization.
b) Training a global model and fine-tuning every local client or knowledge transfer This approach allows knowledge transfer from a source domain trained in the FL manner to target domains (Li & Wang, 2019), which is especially useful for personalization in healtheare domains (Chen et al., 2020; Yang et al., 2020).
c) Collaborative training between the global model and local models. The basic idca behind this approach is that each local client trains some personalized parts of a large model, such as the last. few layers of a neural network. Parameter decoupling enables learning of task-specific representations for better personal- ization (Arivazhagan et al., 2019; Bui et al., 2019), while channel sparsity encourages each local client to train the neural network with sparsity based on their limited computation resources (Horvath et al., 2021; Alam et al., 2022; Mei et al., 2022).
17

---

Published in Transactions on Machine Learning Research (05/2025)
Algorithm 2 i-Scaffnew for (ERM)
1: input: stepsizes
> O,..., in > 0; probability E (0, 1}; initial estimates z!..... In Ed and
Bi....,,, hg E IRd such that Ei=1 h =
the server, 7 = t -i= 1 7, for = 1.... do
-1
used by server for Step 9
flip coin 0 == {1 with probability p, 0 otherwise) 5: for i 1.=1.... ,..., at clients in parallel, do
6: compute an estimate of of Vf((sf)
7: =f o - (g) 49)
local SGD step
8:
if
then
send 1
7. 2 to the server, which aggregates ==:= Σ 1 " 4y and broadcasts it to all clients communication, but only with small probability p
10:
1-+1= = z²
11:
-+ = h + 2 (x' - 21
update of the local control variate A
12:
else
F 1+1 = Rl
13:
14: h = h!
15:
end if
16:
end for
17:
end for
Despite the significant progress made in FL personalization, many approaches only present empirical results. Our approach benefits from the simplicity and efficiency of the FLIX framework and enjoys accelerated convergence.
B Proposed i-Scaffnew Algorithm
We consider solving (ERM) with the proposed i-Scaffnew algorithm, shown as Algorithm (applying i-Scaffnew discuss subsequently in Section C).
to (FLIX) yields Scafflix,
Theorem (fast lincar convergence). In (ERM) and i-Scaffnew, suppose that Assumptions 1, 2, hold and that for every [n], 0 74 S I. For every 2 0, define the Lyapunov function
1
V
x
x²
Ye h
Vf((=*)11
(12)
p
mm1
Then i-Scaffnew converges linearly: for every N 0,
E T < (1 'T
T
Σ nC
(13)
where
( min min yaps p2)
(14)
iE(n)
Proof. To simplify the analysis of i-Scaffnew, we introduce vector notations: the problem (ERM) can be written as
find x* arg min f(x) s.t. Wx
(15)
XEX
where X' = dd an element x = (Iz) E X' is collection of vectors E Rd, f: En fa(x;), the 3 linear operator W X + X' maps x = (r:)i= to (z, E 15 " for given values ?1 > O,..., Yn > 0
and their harmonic mean 7 Σm )) The constraint Wx 0 means that minus its weighted
18

---

Published in Transactions on Machine Learning Research (05/2025)
average is zero; that is, x has identical components 21 Zn- Thus, equation 15 is indeed equivalent to (ERM). x = (a*)= E X is the unique solution to equation 15, where x² is the unique solution to (ERM).
Moreover, we introduce the weighted inner product in X:: (x,y) + (x, = Ei +o (mi, v.). Then, the Un), with respect to this weighted that 7: (z - zz) 0). Thus, P, as well as W = ld - P, where Id denotes the identity, are self-adjoint and positive linear operators with respoct to the weighted inner product. Morcover, for every x E X,
orthogonal projector P onto the hyperspace (y E X y1
inner product, P: X & (=)=_ with
7/2 E +,;;; (because minimizes || - -xl
1 -
| = |xx| + ||
| ||Wx|
| + I|xx
where x = ( and ã -.E
7, 24.
Let us introduce further vector notations for the variables of i-Scaffnew: for every 2 0 we define the scaled concatenated control variate h' = (nu4)== h+ = (xx4t) with h Vf((*)) ) x (r")_ w' == (w/)== with w/ = of ng. w = (w]) with w = oi - 7: (+t))) h' = he pW& Finally, we denote by Fi the o-algebra generated by the collection of X'valued random variables x², ho , h' and by Ft the o-algebra generated by these variables, well as the stochastic gradients gf.
We can then rewrite the iteration of i-Scaffnew as: &¹ w +h'
if 8r then
x*+1 xx
hl+l = h' -pW**
else
xf x+1 = x²
h1 = h'
end if
We suppose that Σi=1 11 = 0. Then, it follows from the definition of E that EE= 3, 17'
29) = 0,
that for every t 2 0, Em= ht 0: that is, Wh' h
Let t 2 0. We have
E |x
x" x111" |50] =/||=² 11, + (1 - p) 1 11,
with
||xt
x*1 = ||0t - x 132 - -1
Moreover,
|| x² x11 = w w 2 + h' ht 1 + 2(w w', h² h))
w w
h' h"
+ 2(x x", hh - -h"
w' w
hh' h"
1 2 x", h4' h -h" 2(8² x², hi)
W' w
hh' - h"
2 2 - x²,h² - -h" + 2p/88 - xx, ""*
w' w hh' - h*
+ 2(2' x*, h - h²) 2p [W&112
Hence,
E ||x
x" x1122 | [r(]
= =||2* 11 ||
= =||w"= w*11 |hh h" 11 + 2(2* xxx, hi' hh),, + 2 2
19

---

Published in Transactions on Machine Learning Research (05/2025)
On the other hand, we have
E h ++ b'll" |F
=p Ih' 101
+ (1 - p) ||b - h²" 1-11,
and
h h
(h' h*) + (f h²)
hi h"
+ 2(hh h², 'n' -h')7
h h
2((h' h" hn' 16-1i -h²)
||
h
H h
2p h², W(x* - -x)))
||
h
[W&
2p(W (h' - h²), - x*),
h-]
1 x²
[
11
2 2p hh, x¹ - -x>>
Hence,
E | - x"1 |
|1 - -h111 |チ*
+ 12
E
w+ | || 1hhh-h h* 112 2( xx,, fi' - hh) + p W 1
x²,h''
1
Ih' h 10 ||| || 2(h h x² x")
+ p2
= w' w' || + pa (1 - p²) ||h² - 11
(16)
Moreover, for every i E [n],
|w| w/|] 1| = || x² - (r( - vz((x))) [12
= ||ا| - x= ||? - 27/(2 x of - V.((z')) +07 1lo vf((2*)]1
and, by unbiasedness of of and Assumption 2,
E B|ltut w
= ||ا| - x² 1| 2 2n/ - 2¹, V/((aa)) v(((z'))
+ B |1 V/((=*)| |F(
$ ||ا - x* 112 -2%//o - xt, Vfi(z;))
+ C,
It is easy to see that (a) - x², Vi(aa)) V+((z*)) = Dr. (2f x²) + (2* (x)) This yields
B B[lrot - wtll? [F8] < |o| x11 Dr (a", ,x,) 27 Dy, Dr x*) + + A.D (a(,,()))))
+ C2.
In addition, the strong convexity of fa implies that Dj (x*, x²)= 2 4 || x+1²² so that
E w w; 2 [i] s (1 mpm m 100-===-2200000000 -1 ,,,,D,, + +
20

---

Published in Transactions on Machine Learning Research (05/2025)
and since we have supposed Y/ $ Ag.
E
wt will"
|Fo] s (1 mm14)) || =²1 + +r.0,
Therefore,
E
|| - w'1l 11?
F8]
S max(1 -1p-))p;)))) - + Σ 7cC
iE(m)
and
[p Fi] E [ x1 |F8]
| E. ||
1
[ -halt [F/]
+ p22
max
7sp4e) |x
x || + p2 (1 p²) l h- Σ
%c
sEin|
s (c) ( و| + p²
1
l| 1r1l2 + Σ n.
(1 -) + Σ 7c5
(17)
1=1
where
(= min
min mo iE(n]
Using the lower rule, we can unroll the recursion in equation 17 to obtain the unconditional expectation of g++1
C From i-Scaffnew to Scafflix
We suppose that Assumptions 1, 2, 3 hold. We define for every E [n] the function f: R + f( + (1 - a.)))); Thus, (FLIX) takes the form of (ERM) with fi replaced by ft.
We want to derive Scafflix from i-Scaffnew applied to (ERM) with Sa replaced by f.. For this, we first observe that for every i E [n], fi is of sm-oooth and atr:sglynggy convex. This follows easily from the fact that f((x) a, (a + (1 az)x;).
Second, for every t 2 0 and e [n], of is unbiased estimate of Vf((()) a; Vf((z)). Therefore, agg is an unbiased estimate of fi(ff) satisfying
E3 Ilaiot - -i af]
aB f1o VA
|5] < A,D, (zf, + a²C,
Morcover,
(1, = f((1)) - ((TT)) - =
= -a())) - f((x)) (a, (f((="),aa( - x²))
-[(a)) - M(x²) (V)())),
= -2,((,,1'))
Thus, we obtain Scafflix by applying i-Scaffnew to solve (FLIX), viewed as (ERM) with fo replaced by fi, and further making the following substitutions the algorithm: of is replaced by agg a is replaced by agh (so that by in Scafflix converges to (a;) instead of f((tt) a, (.()))) n is replaced by a; 2 (so that the os disappear in the theorem).
21

---

Published in Transactions on Machine Learning Research (05/2025)
Accordingly, Theorem 4 follows from Theorem 7, with the same substitutions and with A,, C and μ4 replaced by of Ai, afC; and af respectively. Finally, the Lyapunov function is multiplied by Tmin/ /nin// to make it independent from when scaling the " by in Corollary 6.
We note that i-Scaffnew is recovered as particular case of Scafflix a, = so that Scafflix indeed more general.
D Proof of Corollary 6
We place ourselves in the conditions of Theorem 4. Let O. We want to choose the 7/ and the number of iterations 7 2 0 such that [[v²] S For this, we bound the two terms (1 <)2 you and Jam Di 7CC, in equation 4 by e/2.
We set p / /minic(m) Yo4, that Ç mi Yops. We have
T 2 {
e 2 log (2V0 ()) = (1 -07 S
(18)
Morcover,
(minneem)) 75) (mingemm) (4))
(vi E [n] s.t. C₁ > 0) n4
E/mmin 20:
Timin Cn
7c S
minje(x) 75P5
2
1=1
Therefore, we set for every i [n]
epmin
76 min
A 2Ca
(or v := 7. if C. = 0), and get from equation 18 that B[[V"] <
after
o
(( max max
(A.
14s "epminnli) ()) log
C₁
log(("())) (VD)-1)
iE(n)
iterations.
E
Additional Experimental Results
E.1
Evaluating Logistic Regression under Non-IID Conditions
Our thorough evaluation investigates the potential for achieving double acceleration through both explicit personalization and efficient local training under varying data distributions. We consider the scenarios outlined below:
IID: Data uniformly distributed across all clients with identical weighting factors, denoted as ag
We induce imbalances in label distribution among clients. The data is bifurcated into positive and negative samples, followed by a tailored sampling technique that incrementally augments the ratio of positive samples relative to negative ones. We define these ratios as Tpos (i 1)/n and Tneg 1
Label-wise Non-IID:
Tpos: where i represents the client index, and is the number of clients.
Feature-wise Non-IID: Variations in feature distribution across clients are introduced by segmenting the features into clusters with the k-means algorithm. The number of clusters corresponds to the client count.
Quantity-wia wise Non-IID: Data volume variance among clients realized. The distribution of data samples per client follows Dirichlet distribution, with a default sctting of 0.5. Notably, a higher a leads to more uniform distribution. At a it resembles uniform distribution, while 1, the distribution becomes skewed, resulting in disparate data volume across workers.
22

---

Published in Transactions on Machine Learning Research (05/2025)
FEMNIST
FEMNIST
0.8
test accuracy
test accuracy
0.8
0.6
0.7
0.4
FedAvg
FedAvg
0.6
FLIX, a=0.1
FLIX, 0.3
Scafflix, a
Scafflix, a 0.3
0.2
0.5
0
100
200
300
100
200
300
communication rounds
communication rounds
FEMNIST
FEMNIST
0.8
0.8
test accuracy
test accuracy
0.7
0.7
0.6
0.6
FedAvg
FedAvg
0.5
FLIX, a=
0.5
FLIX, == 0.9
0.4
Scafflix, 0.7
Scafflix, =
0.4
0
100
200
300
100
200
300
communication rounds
communication rounds
Figure 6: As part of our experimentation on the FEMNIST dataset, we performed complementary ablations by incorporating various personalization factors, represented as a. In the main section, we present the results obtained specifically with a 0.5. Furthermore, we extend our analysis by highlighting the outcomes achieved with a values spanning from 0.1 0.9.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Shakespeare
Shakespeare
0.5
0.5
test accuracy
test accuracy
0.4
0.4
0.3
0.3
FedAvg
0.2
FedAvg
FLIX, a= 0.3
FLIX, == 0.5
0.2
Scafflix, 0.3
0.1
Scafflix, =
250
500
750
1000
0
250
500
750
1000
communication rounds
communication rounds
Shakespeare
Shakespeare
0.5
test accuracy
test accuracy
0.5
0.4
0.4
0.3
FedAvg
0.3
FedAvg
FLIX, a a= 0.7
FLIX, a a
0.2
Scafflix, a 0.7
0.2
Scafflix, a 0.9
750 communication rounds
250
500
1000
0
250
500
750
1000
communication rounds
Figure 7: In our investigation of the Shakespeare dataset, we carried out complementary ablations, consid- cring range of personalization factors denoted The selection strategy for determining the appropriate values remains consistent with the methodology described in the above figure.
FEMNIST
FEMNIST
0.90
0.8
test accuracy
test accuracy
0.85
0.6
Scafflix, 0.1
Scafflix, a 10e
Scafflix, a= 0.3
Scafflix, a 10e
0.80
0.4
Scafflix, a 0.5
Scafflix, a 10c 2
Scafflix, 0.7
Scafflix, 10e
0.2
0.75
Scafflix, 0.9
Scafflix,
0
250
500 750 1000
250 500
750 1000
communication rounds
communication rounds
Figure Ablation studies with different values of the personalization factor a. The left figure the com- plementary experiment lincarly inercasing a with full batch size; the right the figure with exponentially increasing a with default. batch size of 20.
24

---

Published in Transactions on Machine Learning Research (05/2025)
In the main text, Figure illustrates the outcomes for label-wise non-llD. For the sake of completeness, we also include results in Figure 9, Figure 10, and Figure 11 depicting various data partitioning strategies. Across these figures, we consistently observe that Scafflix successfully achieves double acceleration.
mushrooms
a6a
w6a
10"
10a
10a
10
10
10.
10
10a4
10:11
10
*- u= 16+0
10
-06:
1010
10:47
1e10
10
1e4
10:00
10:23
1e:2
+
1e:2
20
-*-
16.3
10
1e-3
10
1e-3
20
30
20
10
10a
10a
10
10.3
10
10
10-7
10.9.
10
10:11
10:11
15
10:5
10:49
10
10.9
10
10
30
10
20
30
10
20
30
Communication rounds
Communication rounds
Communication rounds
Figure 9: Results on IID) splits.
mushrooms
a6a
w6a
10a
10:1
10a
10
10
10
10:13
10:45.
10
1e+0
20
le:
10:99
le:1
10a2
le:1
20
1e3
25
1e-3
10
- a= 1e:3
16.4
10-4
1e-4
0
10
20
30
10
20
30
10
20
30
10a
10:1
-X
10
Vit*" ]|(
10
10:5
10.
1
10
10:41
10
10
10:45
10:7
10
10
10.
10 Communication rounds
Communication rounds
Communication rounds
Figure 10: Feature- wist non-llD.
25

---

Published in Transactions on Machine Learning Research (05/2025)
mushrooms
a6a
w6a
10a
30 K6
10a
10
10:4
10
10a
10:10
10
10:44
10:45.
1010
-X-- e 1610
10
1e-1
10:
10
20
1e-3
20a4
10.3
20.s.
1e:3
1c4
0
10
20
30
10
10"
10
10a
10-
10
10
10-3
10:41
10:11
10:22
15
10:46
10:10
10:19
10:0
20
10
30
20 Communication rounds
10
Communication rounds
Communication rounds
Figure 11: Quantity- wise non-lID.
26

---

Published in Transactions on Machine Learning Research (05/2025)
E.2 Inexact Approximation of Local Optimal
present the expected number of local ileralions to achieve epsilon such that [V < e. We present the results in Figure 12. We can see there is huge difference with respect to the different c. Since in FL, the communication cost is always the bottlencck, for scenarios that local computation is not that expensive, we can run more local iterations to obtain a smaller c. Figure 4, we show on ablations that even choose e -1 1, which can still provide guidance leading acceptable neighborhood. In general, there is neighborhood here. Since in Figure 4, consider the personalization factor 0.1, here conduct further ablations with 0.01 with the results presented in Figure 13.
To visualize the cost of local communications,
a6a
w6a
mushrooms
1e-01
175
16:02
1e-02
1860
200
16-04
10-04
150
16:05
26:05
1e-08
1e-06
Expected of Local iters
Expected of Local Iters
Expected of Local Iters
129
150-
1e:01
1e-02
100
1e:03
150
$3.04
1e-05
100 45 9t
75-
1e:06
te
38.99
50 95
12.85
29.28
522
5
10
15
20
25
5
10
15
20
25
10
15
20
25
Client
Client
Client
Figure 12: Number of local itcrations per client for find approximation oz of the local optimal a such
that [(()) <. The legend is
mushrooms
mushrooms
1010
10
1e-01
1e-03
104
10.6
1e-06
10-10
2
10-14
1
0
10-18
10-14
5
50
20
20.20
200
1000
20
40
60
80
100
20
40
60
80
100
Communication rounds
Communication rounds
Figure 13: Inexact local optimal approximation with 0.01.
27


Reviews:
[{"review_id": "NIsEfUGsIs", "reviewer": "Reviewer_a4cN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The paper explores the theoretically appealing idea of combining local training with explicit personalization for communication-efficient FL.\n- A convergence guarantee is presented for the proposed method, which is rare among recent FL personalization methods.\n\n\n\n**Weaknesses**\n1. **Limited novelty**  \n   The main novelty claimed in the paper—individualized stepsizes in i-Scaffnew—is quite minimal. According to the authors themselves:  \n   > \"We generalize Scaffnew [1] and propose Individualized-Scaffnew (i-Scaffnew), shown as Algorithm 2 in the Appendix. Its novelty with respect to Scaffnew is to make use of different stepsizes γᵢ for the local SGD steps...\"                                \n   - The proposed i-Scaffnew is nearly identical to Algorithm 3 (Decentralized Scaffnew) in [1], with the sole difference being client-specific learning rates.\n   - Scafflix is essentially an application of i-Scaffnew to the FLIX objective instead of ERM, which is a minor adjustment rather than a conceptual innovation.\n   - Moreover, client-specific learning rates are not new in FL. The paper does not propose a novel approach for setting these rates—using γᵢ = 1/Lᵢ is a common heuristic. The authors may consider referencing other designs, such as [2], to provide a more principled solution.\n\n2. **Weak experimental support**\n   - Since the paper’s core contribution is the use of per-client learning rates, the experiments should be centered around validating this choice. However, the only related result appears as a minor point in the ablation study and lacks comparison to alternative approaches.\n   - Without individualized stepsizes, the proposed Scafflix method is just applying Scaffnew to FLIX. Thus, improvements shown in Figures 1–3 mostly reflect the benefits of combining FLIX and Scaffnew, not the novelty of this paper.\n   - The choice of baselines is inadequate. The paper only compares against FedAvg and basic FLIX. There is no direct comparison with Scaffnew, which the proposed method builds upon. Furthermore, stronger baselines such as FedAvg+FLIX, SCAFFOLD, FedProx, and other state-of-the-art personalized FL methods are missing.\n\n3. **Clarity and presentation issues**\n   - The structure of the paper could be improved for clarity. For instance, Algorithm 2 (i-Scaffnew), which underpins the entire method, is placed in the appendix. Given its centrality, it should be moved to the main text. The current structure of Section 2 makes it difficult to understand how i-Scaffnew leads to Scafflix. A clearer explanation of the transition and the role of each component would help improve readability.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n[1] Mishchenko K, Malinovsky G, Stich S, et al. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! ICML 2022.    \n[2] Joseph C É, Thérien B, Moudgil A, et al. Learning Optimizers for Local SGD. International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023. 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes *Scafflix*, a federated learning (FL) algorithm designed to achieve double communication acceleration by integrating local training (LT) with explicit personalization. Scafflix is presented as a generalization of the Scaffnew algorithm, extended to the FLIX objective that incorporates a personalization factor for each client. The authors argue that this combination of LT and personalization leads to faster convergence, both in theory and practice.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Clarify Novelty**: The authors should explicitly state how their proposed algorithm differs from Decentralized Scaffnew [1], beyond using individualized stepsizes. Additionally, they should explain why this small change merits a new algorithm and name.\n\n2. **Expand on Learning Rate Design**: Provide more motivation or theoretical insight for choosing client-specific stepsizes γᵢ = 1/Lᵢ, or consider incorporating more principled or adaptive learning rate strategies.\n\n3. **Improve Experimental Validation**:\n   - Provide a clearer experimental design to isolate the impact of per-client learning rates.\n   - Add comparisons with stronger baselines, especially Scaffnew (on ERM), SCAFFOLD, FedProx, and pFL methods.\n   - Include FLIX + other accelerators (e.g., FedAvg + FLIX) as baselines to separate the effect of the FLIX objective from the algorithmic changes.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "798JbV4xqg", "reviewer": "Reviewer_dZDf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros.\n- he paper addresses the FLIX problem and proposes a new algorithm.\n- It proves the double communication acceleration for proposed algorithm.\n- The paper is clear.\n\nCons.\n- It is unclear whether there is any equivalence or coverage between FLIX and the penalized problem (1).\n- It is not clear if FLIX guarantees solution optimality, considering the underlying landscape might not be flat.\n- The novelty of the proof is unclear, particularly how it differs from the Scaffnew algorithm, and this distinction needs to be highlighted more explicitly.\n- The impact of the parameters $\\alpha$'s on convergence appears somewhat conflicting and requires clearer explanation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers federated learning with personalization. It introduces Scafflix algorithm, modified from Scaffnew algorithm, which achieves doubly double communication acceleration in the strongly convex setting with linear mixed personalization. Experiments show that the proposed algorithm is also effective for neural networks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be helpful to introduce and define double communication acceleration earlier in the introduction (before question) for readers who may not be familiar with this concept.\n\n- The discussion on novelty needs more detail, particularly highlighting the key differences in the proof compared to the Scaffnew algorithm.\n\n- The paper states \"equation 4–equation 5 do not depend on the personalization weights $\\alpha_i$; they only play a role in the definition of the personalized models\". However, since the bound of $\\Psi_0$ depends on $\\alpha_i$, this statement appears somewhat problematic.\n\n- The special case when $\\alpha_i$'s are all 1's and 0's needs to be discussed. When they are all 1's,does the algorithm recover the results for? When they are all 0's, the algorithm seems suboptimal due to certain constants. It would be helpful if the authors could provide some intuition on why the bound includes these constants as in this extreme case, no learning is actually needed.\n\nOverall, the paper is clear and well-written, with its key messages effectively conveyed.", "broader_impact_concerns": "n.a.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BxXPGcSGhE", "reviewer": "Reviewer_xqNi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Convergence results for strongly convergence case. The paper provides a solid convergence analysis for the proposed algorithm under the strongly convex setting.\n2. Extensive numerical results on different choices of the hyper-parameters show the trends of the algorithm on their choices. The comparison between Scafflix, FedAvg, and FLIX also demonstrates that the proposed algorithm outperforms existing algorithms.\n\nWeakness:\n1. Mismatch theoretical analysis and empirical results:\n    1. The convergence analysis cannot generalize to the nonconvex setting, while some experiments use a nonconvex neural network.\n    2. The convergence cannot be generalized to the partial client participation setting. However, in the ablation study, different client numbers in communication are studied without detailed explanation.\n    3. The convergence result covered heterogeneous learning rates and personalization weights for different clients, but the numerical results only investigated the uniform choice of these parameters.\n2. From the final bound, it seems $T = O(1/\\epsilon \\log(1/\\epsilon))$ instead of what the paper claimed $O(log(1/\\epsilon))$?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new algorithm Scafflix, which extends Scaffnew to solve the personalized FL problem (FLIX). The proposed algorithm applies interpolation between the local minimum and the global model, and uses a control variate to regulate the decent direction. Additional communication skipping is used to reduce communication.\n\nThe paper provides a convergence analysis for strongly convex problems, and extensive numerical experiments and ablation studies are conducted to verify the effectiveness of the proposed algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the above weaknesses.\n\nMinor points:\n1. in the definition of $D_{\\varphi}$, $\\varphi$ is not appeared.\n2. Why $\\Psi$ is a  Lyapunov function? The algorithm is not in continuous-time, and the paper does not prove that $\\Psi$ satisfies the definition of a Lyapunov function.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a theoretical convergence analysis that demonstrates the usefulness of local training for communication reduction in the strongly convex case. The experiment results also show clear evidence that the proposed algorithm achieves communication reduction. After the rebuttal, the reviewers' concerns were addressed. Therefore I recommend accepting this paper.
Decision: Accept (poster)