LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication
Authors: Wei Liu
Anweshit Panda
Ujwal Pandey
Haven Cook
George Slota
Naigang Wang
Jie Chen
Yangyang Xu
Venue: Accepted by TMLR ()

Abstract: In the decentralized distributed learning, achieving fast convergence and low communication cost is essential for scalability and high efficiency. Adaptive gradient methods, such as Adam, have demonstrated strong practical performance in deep learning and centralized distributed settings. However, their convergence properties remain largely unexplored in decentralized settings involving multiple local training steps, such as federated learning. To address this limitation,  we propose LoDAdaC, a unified multiple \textbf{Lo}cal Training (MLT) \textbf{D}ecentralized framework with \textbf{Ada}m-type  updates and \textbf{C}ompressed communication (CC). LoDAdaC accommodates a broad class of optimizers for its local adaptive updates, including AMSGrad, Adam, and AdaGrad; it is compatible with standard (possibly biased) compressors such as low-bit quantization and sparsification. MLT and CC enable LoDAdaC to achieve multiplied reduction of communication cost, while the technique of adaptive updates enables fast convergence.  
We rigorously prove the combined advantage through complexity analysis. In addition, experiments on image classification and GPT-style language model training validate our theoretical findings and show that LoDAdaC significantly outperforms existing decentralized algorithms in terms of convergence speed and communication efficiency.

Jie Chen Published in Transactions on Machine Learning Research (04/2026)
LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed commu- nication
Wei Liu Department of Mathematical Sciences Rensselaer Polytechnic Insfitute
budsdqqb kudsdagb@gmaäilcom @gmail com
Anweshit Panda
pandaa2@rpi.co edu
Department of Computer Science
Rensseluer Polylechnic Instilute
Ujwal Pandey
ponden(rrpi edu
Department of Computer Science
Rensselaer Polytechnic Institute
Haven Cook
cookh22rpi.cok edu
Departmend of Computer Science
Rensselaer Polytechnic Institute
George M. Slota
slotag @rpi edu
Department of Computer Science
Rensselacr Polylechnic Institute
Naigang Wang IBM T. J. Watson Research Center
mwongChus.ibm ibw...sssimm..m.. com
wetnfffuureejjee@@ee com
MIT-IBM Watson AT Lob, IBM Research
Yangyang Xu*
auy21 @rrpi edu
Department of Mathematical Sciences Rensselaer Polytecknic Institute
Reviewed on OpenReview: /ttp:r///operevvtte net/ forum? id- 0qoy9usvmm
Abstract
In the decentralized distributed learning, achieving fast convergence and low communication cost is essential for scalability and high efficiency. Adaptive gradient methods, such as Adam, have demonstrated strong practical performance in deep learning and centralized ristributed settings. However, their convergence properties remain largely unexplored in decentralized scttings involving multiple local training steps, such federated learning. To address this limitation, we propose LoDAdaC, unified multiple Local Training (MLT) Decentralized framework with Adam-type updates and Compressed communication (CC). LoDAdaC commodates broad class of optimizers for its local adaptive updates, including AMSGrad, Adam, and AdaGrad; it is compatible with standard (possibly biased) compressors such as low-bit. quant ization and sparsification. MLT and CC enable LoDAdaC to achieve multi- plied reduction of communication cost, while the technique of adaptive updates enables fast convergence. We rigorously prove the combined advantage through complexity analysis. In
"Corresponding author

---

Published in Transactions on Machine Learning Research (04/2026)
addition, experiments on image classification and GPT-style language model training val- idate our theoretical findings and show that LoDAdaC significantly outperforms existing decentralized algorithms in terms of convergence speed and communication efficiency.
1 Introduction
In decentralized learning, multiple agents collaboratively train a model without a central server, by exchang- ing information exclusively with immediate (a.k.a. onc-hop) neighbors. Compared to centralized distributed learning, decentralized learning has betler robustness and scalability. However, communication cost can become bottleneck in decentralized learning, especially when low- bandwidth or wireless communication is performed. This motivates the design of commuicatioon-effeee
decentralized algorithms.
Two widely adopted strategies to reduce the communication burden in distributed learning are Compressed Communication (CC) and Multiple Local Training (MLT). By CC, the agents transmit compressed informa- tion rather than full-precision one, significantly reducing per-round communication cost. Examples include low-bit quantization (Sun al. 2020 Wang et al.) 2018 Bernstein et al.) 2018 Alistarh et al., 2017) and
sparsification (Koloskova et al.] 2019 Stich al.
2018). CC also provides implicit privacy protection: by
transmitting the compressed message, agents inherently obscure precise local data information, thus miti- gating potential privacy risks (Kairouz et al. 2021). On the other hand, MLT, which involves performing several local updates per communication round, has gained popularity in various distributed learning set- tings. Prominent examples include local SGD (Haddadpour et al.] 2019), SGD averaging (Zhang et al. 2015), and, notably, Federated Averaging (FedAvg) (McMahan et al. 2017), a widely employed method in federated learning (FL). Empirically, MLT significantly reduces the number of communication rounds required to achieve target convergence threshold. Moreover, from a privacy perspective, MLT further enhances security by reducing the frequency and amount of sensitive information exchanged among agents, thus limiting potential data leakage (Li et al. 2020 Kairouz et al. 2021
Both CC and MLT have been explored in vanilla and momentum SGD (Singh et al. 2021 Sun et al., 2022 and it is shown in (Singh et al. 2021) that multiplied reduction of communcatiin can be achieved. However, adaptive (i.e., Adam) stochastic methods (Kingma & Ba 2014) exhibit significantly faster convergence than vanilla or momentun SGD on training deep learning models and are now the workhorse for training language models. Hence, is natural to ask the following question:
Can CC and MLT be applied in decentralized adaptive stochastic methods to simultaneously achieve mulliplied reduction of commmmication and fast converyence!
(Q)
1.1 Contributions
This work provides an affirmative answer the question Q based Decentralized framework with Adaptive gradient updates and Compressed communication. Our local update scheme includes both the vector and matrix variants of AdaGrad (Duchi et al. 2011), Adam (Kingma & Ba 2014), AMSGrad (Reddi et al. 2016), and the recently proposed Adam-Mini (Zhang et al. 2024). The integration of CC and MLT enables multiplied reduction of communication cost while adaptive gradient updates further yield fast convergence.
We propose LoDAdaC, a Local training-
A contral technical contribution of our work lies resolving the core analytical challenge introduced by the interaction of MLT, CC, adaptive gradient updates, and decentralized communication: their coupling makes it difficult to derive unified upper bound on the consensus error and stationarity violation. In particular, local adaptive gradient updates introduce noulinearity and dynamically varying gradient scaling, which complicate the analysis even in centralized scttings (Wang et al. 2022b). When combined with decentralized model aggregation, these properties pose significant technical obstacles to convergence analysis. Our analysis carefully disentangles the coupling, leading to tight convergence guarantees under mild conditions and offering the first such results for this challenging setting. By performing K local updates per communication round and utilizing compression operator that compresses one unit of message algorithm needs total communication cost. of ((z2) produce e-stationary solution, thus yielding
n unit with r E (0, (0,1), our
2

---

Published in Transactions on Machine Learning Research (04/2026)
multiplied reduction of communictiin cost. Notably, the result applies uniformly across different choices of compressors and adaptive updates.
In addition, we conduct numerical experiments on two representative tasks, ie., image classification and language model training, to validate the effectiveness of LoDAdaC. Though our complexity result has the same order dependence on e as achieved by existing non-adaptive stochastic methods such as SQuARM- SGD, our numerical results demonstrate that LoDAdaC achieves significant speed up by adaptive gradient updates and significant communication reduction from combining MLT and CC. Specifically, our experiments illustrate that: (i) LoDAdaC equipped with adaptive gradient updates significantly outperforms the baseline decentralized algorithm SQuARM-SGD (that employs momentum gradient update) in terms of convergence speed; (ii) The joint use of MLT and CC reduces the total communication cost dramatically, achieving reductions of over 99% in some scenarios (e.g., see the results yielded by LoDAdaC with K = 50 and Top- k=30% Figures Ba and 3b with nearly no sacrifice of accuracy. These empirical findings align closely with our theoretical results.
1.2 Problem formulation and technical assumptions
We consider docentralized nonconvex stochastic optimization in the form of
min f(x) : Σ f((z)) with f((x) = EEIND, [Fr (z.
(1)
#cRd
Here, n agents, connected via a communication graph G, collectively minimize the objective function f the average of local functions [Li}, each of which is defined as an expectation over data distribution D;. Each agent i E f1.2....,,i n exclusively accesses its local function fa and stochastic gradients (F E.), and collaboration occurs through communication with immediate neighbors.
To perform decentralized computation, each agent i E {1, 2, (1,2....,,) maintains a local copy æ; of the decision variable æ. Let X = [x1, 22.... xn] E Rdxn, The problem I can be equivalently reformulated as
min 1 I s.t. X
(2)
XeRden TI
where W is a mixing (a.k.a, gossip) matrix that governs how agents aggregate local information. Under
Assumption 2iii) given below, imposing the constraint X = XW is equivalent to requiring x = ... C..
i..,, X lies in the consensus subspace.
Throughout the paper, we make the following standard assumptions.
Assumption 1 For each E (1,2,...,nn, n),, the function fa is T--smoohh, i.e., [V](())- Vf(li) T.I yyl, for any y, E Rd, and f is lower bounded, i.e., f mink f(x) > -00.
Assumption 2 For the miring matria W, it holds (i) W is doubly stochastic, i.e., W 2 0, W1 and TW- 1_; (ii) Wi 0 if i and are not neighbors to each other; (mi) Null(W - I) span (1) and P = |W - J| J||2< < 1, where 1 is an all-one vector, the identäly matriz, and J 11
Assumption 2 encodes the standard structural conditions required for decentralized averaging. First, the doubly stochastic property ensures that the mixing operation preserves the network average. Second, the condition W., 0 for any non-neighboring agents i and enforces that communication occurs only between neighboring nodes in the underlying graph G, thereby respecting the locality of the decentralized architecture. Most importantly, the spectral condition p || 2 < 1 guarantees contraction toward consensus and serves as the key quantity controlling the consensus-crres recursion in our analysis. Intuitively, smaller values of p correspond to faster information propagation across the network and hence more rapid agreement among agents. The specific choice of the mixing matrix W depends on the communication topology, and several standard constructions have been proposed the literature (Koloskova et al.] 2019 Mancino-Ball et al., 2023. Nedié al.) 2018). In particular, Xiao & Boyd (2004) showed that one can design an optimal mixing matrix that minimizes P while satisfying the constraints in Assumption 2
3

---

Published in Transactions on Machine Learning Research (04/2026)
1.3 Notations and definitions
We define [T] {0, .......T--.. 1} and use |l-|l to denote the Euclidean norm for vectors and the Frobenius norm for matrices. The spectral norm of matrix A is denoted by || For two vectors a and b of the same dimension, and o b denote componentwise division and multiplication, respectively, while v applies the square-roo operation elementwise to a nonnegalive vector c. X = X(I - J) denotes the consensus error matrix and ā = XX for the average of all local decision variables. E, takes the expectation over the random samples { (E(.2.........))) conditional on the t-th itcrate, while takes the full expectation.
Definition 1.1 We call Q on n-compression operator, if it holds Ee [| 2[+]||] S 1f |||1 for some 7 E (0, and all R.d
The expectation in the above definition is taken with respect to the internal randomness of the compressor; conditioned on the input æ. For deterministic compressors, such as Top-k sparsification, the incquality holds deterministically. Our analysis relies only on the contractive-error property in Definition 1.1 and does not require additional assumptions such as unbiasedness. Examples n-compression operators include Random- k (Stich et al., 2018), Top-k (Aji & Heafield, 2017), and the rescaled quantizations (Chen et al. 2023a): : see more examples in [Chen al. 2023a Koloskova et al. 2019). When r) 0, simplifies to the identity operator.
Definition 1.2 We say that X is an c-stationary point, in expectation, of the decentralized problem E [[)]] + IIX [P] < 2
if
This notion jointly controls stationarity of the averaged model and network disagreement; both must be small for decentralized learning to be practically meaningful.
2 Related work
In this section, we review existing works on distributed stochastic gradient methods (SGMs) in either a centralized or decentralized setting for solving nonconvex problems. Additionally, we review methods developed for distributed learning with MLT and CC.
2.1 Centralized or decentralized (stochastic) adaptive gradient methods
Adaptive SGMs are among the most popular stochastic algorithms for training nonconvex deep learning modols. In practice, adaptive SGMs such as AdaGrad (Duchi et al., 2011), Adam (Kingma & Ba 2014
and AMSGrad (Reddi et al. 2016 are more effective compared to a nonadaptive SGM.
Efforts have been made to integrate adaptive gradient updates into distributed optimization. Hou et (2018)
propose distributed Adam for convex problems, while (Chen et al.] 2020; Zhao et al.) 2022) introduce locally adaptive algorithms for centralized distributed training. The compressed centralized distributed Adam variants are explored in (Chen et al. 2021 2023a. A centralized distributed AMSGrad is studied in (Li et al.] 2022),, and a compressed version is presented in (Wang et al. 2022a). The decentralized Adam variant, DADAM, was introduced in (Nazari et al. 2022 providing convergence results for both convex and nonconvex problems. However, subsequent analysis by CCen et al. 2023b) reveals that DADAM may not converge a stationary point nonconvex settings. To address this limitation, (Chen et al. 2023b Wang
et al. 2025 Liu al. 2025) propose some other decentralized adaptive gradient methods.
Despite these advancements, distributed learning with multiple local adaptive gradient updates has been explored only in centralized setting. Xic et al. (2019) propose AdaAlter, which employs local adaptive updates on the client side. Similarly, Reddi et al. (2020) extend FedAvg by incorporating three types of local adaptive gradient updates to improve optimization performance. More recently, FedLADA (Sun et al. 2023) introduces momentum-corrrette adaptive updates, and FodAMS, along with corrected variant FedCAMS (Wang et al. 2022b), stabilizes local AMSGrad updates to ensure convergence. However, decen- tralized distributed learning with multiple local adaptive gradient updates remains unexplored. Though Gao
4

---

Published in Transactions on Machine Learning Research (04/2026)
& Huang (020) attempt to study a decentralized distributed method with multiple local Adam updates, they conduct analysis only to the case without first-order momentum. In addition, their convergence rate results in Corollary 1 and Corollary 2 are obtained by implicitly assuming B2 = O, i.e, no second-order momentum either.
2.2 MLT in distributed learning
MLT is a simple yet remarkably effective communication-sarvi strategy in distributed learning, where clients perform several local podates-rathes than a single one-between successive communication rounds. A foundational method that employs MLT in centralized distributed learning is FedAvg, with numerous ex- tensions including FedAvg with local momentum (Hsu et al. 2019), server momentum (Sun et al.] 2024), and adaptive FedAvg (Reddi al. 2020). Recent theoretical advancements have clarified why MLT effectively reduces communication complexity in centralized distributed learning (Kairouz et al. 2021 Li et al. 2020
These results have been rigorously established across wide range of local update straegiees including stan-
dard SGD (Haddadpour et al. 2019 Spiridonoff et al. 2021 Stich 2018 Yu et al.) 2019), momentum-basel
methods (Karimireddy et al. 2020 Sun el al.) 2024h, and adaptive gradient methods (Reddi et al. 2020 Xie
et al. 2019).
In decentralized distributed learning, carly work has primarily focused on algorithms using simple local SGD updates. For example, Xing el. al. (2020) propose decentralized federated learning framework for medical applications, operating without central server in dynamic peer-to-peer network. Similarly, Lalitha et al.) (2019) explore decentralized learning using a Bayesian-inspired belief update mechanism over connected networks. Further analyses in (Koloskova et al. 2020 Sun et al.) 2022 Wu et al. 2025 Li et al. 2019 have demonstrated that incorporating MLTT with multiple local SGD updates can also reduce communication complexity in a decentralized distributed setting.
2.3 MLT+CC in distributed learning
Combining MLT and CC, while simultaneously retaining their respective benefits, is notably challenging. Tn a centralized distributed setting, several recent algorithms successfully integrate these strategies, including CompressodScaffhew (Condat al.] 2022). LoCoDL (Condat et al.] 2025), FedCOM (Haddadpour et al., 2021), FedPAQ (Reisizadeh et al.) 2020 and Qeparse-Local-SGD (Basu et al.) 2019). They leverage the advantage of both MLT and CC, achieving multáplied reduction of communication complexity.
Despile these developments, few algorithms leveraging MLT+CC have been proposed in the context of decentralized distributed learning. Extending theoretical guarantees to the decentralized setting introduces substantial challenges due to the absence of a central coordinator. Complications arise from network topology constraints, the need for peer-to-peer communication, and heterogeneily in local data and model states. These factors make the convergence analysis significantly more intricate and have historically limited the rigorous understanding of MLT-CC in decentralized settings.
Among decentralized MCT methods, each individual one covers only part of the design space. DFc- dAvgM from (Sun et al. 2022), SQuARM-SGD from (Singh et al. (021), and TM-DFI, from (Chen et al. 2024 are closely related to our method. DFedAvgM extends decentralized FedAvg by incorporat- ing momentum- sanenttumbbsed local updates and CC. However, DFedAvgM is unable to reduce the order of total communication rounds through MLT and can only mitigate the effect from local variance of stochast gra- dients when no momentum applicd. In addition, using CC will hurt the complexity result of Dvgd to obtain an e-stationary point unless the compression error is controlled in O(e*), which is too-restrictive assumption. Without relying such restrictive assumptions, general convergence result is established to LM-DFI. that incorporates both MLT and CC. However, LM-DFL. is also unable to reduce the order of total communication rounds through MLT. SQuARM-SGD achieves multiplied communication reduction, but its analysis is tailored to momentum SGD and additionally assumes a symmetric mixing matrix.
Compared to existing decentralized methods combining MLT and CC, our contribution transcends a specific optimizer instantiation by providing a unified algorithmic framework and a generalizable convergence analy- sis template. Notably, establish the first rigorous convergence guarantees for federated learning scenarios
5

---

Published in Transactions on Machine Learning Research (04/2026)
MLT Methods
CC
AG
#
CommCost
MLTsave CCsave
PD-SGD Ge & Chang
2023
X
ned
LSGT Li et al.
2019
x
1 ne
/
Avgd
Sun et al. 2022)
/
x
max K
Ke
max 1-s c
(1-4)e 6
X
/
20 o
SQuARM-SGD
(ingh et al.
2021
x
1-n nKet
/
V
nct
LM-DFL
Chen et al.
2024
/
x
max Ko
1
K of
max K
(1-1)
X
x
LoDAdaC (this paper)
V
V
K
Table 1: Comparison between the proposed method and selected approaches that use MLT for nonconvex decentralized distributed learning. "CC" indicates whether compressed communication employed; "AG" denotes the use of adaptive gradient updates; "#Iter" specifies the number of total iterations (per agent) to obtain an e-stationary point of problem (2). see Definition 1.2 "CommCost" refers to the total communication cost, where each communication round incurs unit cost in the absence of compression; "MLTsave" indicates whether the number of communication rounds can be theoretically reduced by employing MLT; and "COsave" reflects whether the total communication cost can be effectively reduced by compression. Here, the O(-) notation is omitted in the lable, assumed to be sufficiently small, and the number of local steps is most O(e "s refers to the compression error given in (Sun et al. 2022 satisfying EE [[|æ: [(x)] S s2d.
employing adaptive gradient methods such as Adam. Moreover, our convergence analysis relies exclusively the contractive-error property of the compression operator, thus naturally extending to accommodate po- tentially biased compressors, including Top-k sparsification. Lastly, proof technique explicitly decouples the intricate interactions among adaptive gradient updates, MLT steps, and decentralized communication, evfectively addressing the core analytical challenge of simultaneously ensuring stationarity and consensus in decentralized distributed training. Detailed comparisons are summarized in Table ! We notice that the complexity result of our method is the same order as that of SQuARM-SGD. However, with adaptive gradient updates, our method is able to achieve significantly faster empirical convergence, in particular for training GPT-style language models; see Section
3 Decentralized adaptive methods with MLT and CC
In this section, we introduce a unified docentralized framework that integrates multiple local adaptive gradicnt updates with compressed communication. It is named LoDAdaC. Also, we provide convergence guarantees for the proposed algorithm under general nonconvex settings.
3.1 A unified algorithmic framework
We present the pseudocode of our framework in Algorithm 1 For simplicity, we take a single randomly sampled data point ş at each iteration. All our theoretical results remain valid by taking a mini-batch of samples.
In addition to Assumptions 2 we make the following assumption, which is standard in the analysis of both distributed and non-distributed adaptive SGMs (Chen et al. 2019 2023b Kingma & Ba 2014 Reddi
et al. 2018 Xu et al. 2023
Assumption 3 The random samples { hii0 are independent. For each and i E (1,2,..., nn. it holds Eight = Vf(())). In addition, there are constants B and B such that l|g! S B, |lg I B for any E (1.2...., nn) and any I, and [x) < B, [.)]ll < Bxx for all æc
The unbiasedness condition E, [g = f (z:) is standard the literature stochastic methods Lan 2020
The bounded gradient condition in Assumption g is stronger than the bounded- variiacce assumptions often used for aon-adaptive methods; can be restrictive for modern deep networks with heavy- gradients. Nevertheless, similar assumptions made in prior work such (Chen et al.] 2023b) for the convergence

---

Published in Transactions on Machine Learning Research (04/2026)
analysis of adaptive methods. Extending the present adaptive+MMT++C analysis under weaker conditions, such generalized smoothness or bounded-variance assumptions, is an important direction for future work.
Algorithm 1: Local training-based Decentralized framework with Adaptive gradient updates and Compressed communication (LoDAdaC)
Input: a > 0, 0 < 31 < 0,0 S 1, a maximum number T communication rounds, number K of local training sleps per communication round, a y-compression operator 2, d-dimension vector-value functions {re}, and a mixing matrix W;
Let æ/ 22
x 2! =h = x = x", and set m, and u," to 0 for cach i. for L= 0, 1,.1...... TK do
for all agents E 21, n in parallel do
5
Obtain one random sample ş and compute a stochastic gradient gf + VE (af, 4)
Let m = f.m!
31)
Let ui = r:(g g 9.):
Update x; _ x
mi
/ 46
if mld(l + 1, K) = 0, then
10
Set x. ++1 x + 2[=, ++
x
and xt
14 +r(E,- W 2(t1).
else
Update xtt1 x, ++ 1 and xtt = xt.
The condition in line 9 of Algorithm I indicates that neighbor communication happens every K iterations, namely, K local updates are performed per communication round. In addition, we ouly need to communicate the compressed vectors to obtain ΣØ=1 Wysztt), as explained below. For each i = 1,2...., let agent maintain a vector y; and initialize as yf xf. Then, for all t 2 0, let y!+1 y/ if mod(t + K) * 0, and + yt Σn= W & 2
++ 2 otherwise. This way, we have 2/ x;
1+ + +> (y) ++ x +1 and
thus enable the reduction of communication cost by only communicating compressed message.
With appropriate parameter 31 and vector function rt, the local update of LoDAdaC in line 8 of Algorithm ! encompasses several well known optimizers as special cases. As we demonstrate in Section 3.2 our theoretical results apply to optimizers listed in Table 2
Optimizer
Description
Vanilla SGD
B1 0 and r. 0, i.c., u² 0 for all and t
Momentum (Heavy-ball) SGD
B1 E (0, and ra 0, i.e., ut 0 for all i and
AMSGrad
ūt 3zu + (1 (2) gf. uf max(u ai)
0. = 0, and B2 E 1)
Adam
u: = 3ru 1-1 + (1 - 32 o gf with 32 E
vTK VTK+1*
Adam-mini
ui 3rui 1-1 + +(1 - B2 mean og), with B2
<TK
VTK+1"
Averaged AdaGrad
ui [+1 Et gi g
Table 2: Representative optimizers of Algorithm with specific selections of 31 and Ta
3.2 Convergence analysis
In this subsoction, we establish the convergence rate results of Algorithm D We first derive a consensus error bound in Lemma 3.1 This bound is essential because it explicitly characterizes the relationship wetween
7

---

generated by Published in Transactions on Machine Learning Research (04/2026)
the consensus error and key algorithmic parameters, including the step size a, MLT steps K, compression error 1/ of Q, and the spectral gap of the communication graph. Such characterization allows us to rigorously analyze how MLT and compression impact the convergence speed. Then we establish a bound in Theorem 3.1 on the objective gradient at averaged points. This bound enables us to show the final convergence rale results of our algorithm with several specific choices of popular adaptive updates. All proofs are given the appendix.
Lemma 3.1 Under Assumptions 1 let 0 K Algorithm 7 salisfies
(1-0)(1-²)) Then the sequence
{x1)*K
TK-1 Σ t=0
56
80
+ 15
E X
S sa'nk²C, where C :
B28
(3)
TK
7(1 p) 71 p)
1 n/2
Theorem 3.1 Suppose that Assumplions hold and ||u] 10 B, for all t 2 0 and i E (1,2,...,, for some By > 0. Let C denote the constant defined in (3] and a, > 0 satisfy
8
p) - n)²)
7
(4)
48L Bu
100
Then the sequence (a")
(K-- 1t=0
generated by Algorithm
satisfies
TK
aTK a2T223BB
a
E |I x 11 I I (x))
8 +
A/B, +8
8/B, 8(1 31)
t=
0437 B2
TK
+
Bu + (v
+ +
E
(1 - -1)²2
1=0
n2
u - +8
V /uu
+
(5)
TKB2 B2 + 612
+ 2
a @TTKKT
no
nd
+
aT?
(vBu 8
+ 2v8
a
TTTnK*C
TK-1
Σ
VBu +8
+
1
8
t= 5(1 - 31)2
5
2/8
For each optimizer listed in Table
2
we are able to show the condition || Bu, A1, V for some constant
is bounded; see Lemma. A.7 Thus by Theorem 3.1 we specify the choice of a and obtain the convergence rate of Algorithm by difterent ways of defining the second momentum term ut in line of Algorithm
B, and that E Σ -
Vut
Theorem 3.2 Under Assumptions 3 let 5 5= 0(1) be universal posilive constant and C be the constand defined in Choose and such that a and > satisfy
40 10 B% +0)
(1 p) n7²)
a
S min
VTK
48I B2 +
100
where 0 = 0(1) is constant. in Table we have
Then for the sequence
{x* [=K)Tk" T-1 It=
generated by Algorithm with any optimizer
TK
|| 112 1 = =0
-
n
nKC
(7)
TK
VnTK
TK
T
8

---

Published in Transactions on Machine Learning Research (04/2026)
3.3 Linear speed up, topology independence, and communication reduction
Based on the convergence rale results in Theorem 3.2 we discuss how the number n of agents, the number K of local updates, and compression ratio 1 - 77 affect the iteration complexity and communication complexity of our algorithm to produce an c-stationary point in expectation.
Linear speed up and topology-indepeedentt step size.
By
and the definition of C
3
nt K
T= Ω
(8)
(1- - 72)
then nkC = 0 ( T
TK = 0 VNTK and we obtain
VnTK
TK-1
(0 VnTKK
E
V2
P+ X
(9)
TK
t=0
Letting be selected from {0,
TK 1 uniformly at random, we have from g that
E [ (T ) 1 = 0 number of local iterations per agent is TK = O
1
Hence, to obtain an e-stationary point in expectation, the total
voTK
Given K, have T O (nka)); when O inequality in (6) holds. Thus in this case, we obtain a linear speed up with respect to and the step size a = (me²) and is independent of and
( the chosen T will satisfy (8) and the first
Multiplied reduction communication cost. For a small enough > O, we can further reduce the order of communication rounds by picking K. Suppose e = (11-))(1-1))) for some E (0, Then
we can choose K = (c- and T T= = 3--3 (<²²)) which satisfies This way, compared performing a single local update, i..., K = 1, we reduce the number of communication rounds by an order of c11+, In addition, by using an 7ccoppression operator, our algorithum only needs 1- n of communication amount compared to using compression. Therefore, the total communication volume required by our algorithm 1-n achieving multiplied reduction of the total communication cost. ne
is +
4 Numerical experiments
In this section, we demonstrate the efficacy of the proposed framework over a set of numerical experiments. We consider three standard benchmarks, including training a convolutional neural network LeNet5 (LeCun et al. 1998) on the MNshionMIIST dataset (Xiao et al. 2017 , a ResNet architecture Fixup-ResNet-20 (Zhang et al. 2019 on the CIFAR-10 dataset (Krizhevsky et al. 2009), and a small-scale 10.7M parameter GPT model, from nanoGPT (Andrej) 2022), the tiny-shakespeare datasct. We will show the performance of LoDAdaC equipped with the following adaptive gradient updates: AdaGrad, Adam, and AMSGrad ho- mogeneously distributed training data. We will compare LoDAdaC against ARM-SMD (Singh et al. 2021), which incorporates compressed communication and local training with a momentum- uaassed SGD, Our methods improve over SQuARM-SGD with the addition an adaptive update. We provide an additional comparison against DADAM (Nazari et al. (2022), which represents methods with decentralived and com- pressod communication but always with single local update per communication round. A final experimental baseline for comparison is CDProxSGT (Yan et. al. 2023). which represents non-adaptive methods with local updates, for the sake of completeness.
We implement all of these methods in Pyrcrc.. For FashionMNIST and CIFAR-10, we run our experiments on a CPU server. This server has two-way 64-core (256 threads) AMD EPYC 7742 CPUs al 2.25GHz and 2TB DDR4 memory. For tiny-shakespeare on nanoGPT, we run the experiments on a separate server with NVIDIA A100 GPUs. Both test systems have Python 3.12.3 and PyTorch 2.7.0+cu126 installed, running on top of Ubuntu 24.04.2 LTS. The code and experimental scripts for our methods are publicly available https://github. /ttea///e/////ggttoeooo/eeteaaan

---

Published in Transactions on Machine Learning Research (04/2026)
O Da
10
Oerth ret6dased
Thantry LREN
DADAM SQUARM
Comseauu Crme
10-4
10-2
DestrisdesA.com
DADAM
20
SOWARMSGB
CDPPeeSGT
0.0
(a) CIFAR-10 optimizer comparison.
8-
Skda0rad bute Stidam
Demnuteccaa
SQJARN-
SQUARM CDPrecoOT
Conernaue Ermer
10
Talikng
Volibadenn
-0-
Epochs
Eacchs
Epects
(b) tiny-shakespeare optimizer comparison.
Figure 1: Optimizer Comparison: Plotted above the training loss, test accuracy, and consensus of CIFAR-10 (top) and the training loss, validation loss, and consensus error of tiny-shakespeare (bottom) with training done on all of the various optimizers.
We will compare training loss, test accuracy, and consensus error, as well as validation loss for the GPT model. We will compare these values relative to the number of communication rounds and the communica- tion volume. We ran significant parametric study, evaluating possible parameters within: local updates K == [1,2,5, 10, 20, 50), optimizers = [AdaGrad, Adam, AMSGrad, DADAM, SQuARM-SGD, CDProxSGT], Top-k compression = [30%, 40%. 50%, 60%, None], agents = [4, 9, 16), topology [ring, 2D-grid), data distribution = [IID, D1.)chlet(1.0), Dirichlet(0.5)) We will show representative selection of these results below on CIFAR-10 and siny-shakespears with FashionMNIST and the rest of the results appearing in the appendix. We use a batch size of 64 for the CIFAR-10 and FashionMNIST datasets and a batch size of 128 for training the GPT model. We initialize the learning rate 0.001 for AdaGrad, Adam, and AMSGrad CIFAR-10 and FashionMNIST and use |3 values of 31 0.9, 82 0.999 for Adam and AMSGrad. We use learning rate of 0.0001 shay-seaespeear. We tune the learning rate to 0.01 for SQuARM-SGD CIFAR-10 and MNIST and 0.005 on tiny-shakespeare - higher learning rates, such as the recom- mended learning rates of 0.1 and 0.2 from (Singh et al. 2021), did not converge with a number of tests when using our Top-k compression operator.
4.1 Optimizer Comparison
We first compare the performance of different optimizers with n agents with a fixed local updates per communication of K respectively. We display these results in Figures la and 1b We compare against SQuARM-SGD, DADAM, and CDProxSGT K in the appendix in Figure 7
and Top-k compression of 40% and 50% for CIFAR-10 and tiny-shakespeare,
baselines for this set of experiments. Similar results are plotted for other values of
We observe that SQuARM-SGD is slightly slower to converge on CIFAR-10 with the given hyperparameters, but it achieves a similar test accuracy. CDProxSGT converges significantly slower and does not achieve an oquivalent test accuracy in the given number of epochs. The performance of both non-adaptive methods terms of validation loss significantly worse on the GPT model, known issue of training language models with non-adaptive momentum SGD methods (Zhao et al. 2025 We note that across these experiments,
10

---

Published in Transactions on Machine Learning Research (04/2026)
Consenan Cite
E-20
0.5
10
Prin
20100
20000
Commutice Bourds
dalinn foanca
(a) CIFAR-10 reduction in communication rounds.
4.0
3.5-
3.5
Consaraua aaa Erop
3.0-
25
1.5-
30000
2000000
Commanssaion Rounc
Commusiction hourds
Communsaioon hounds
(b) shap-see reduction in communication rounds.
Figure 2: Number of Local Updates: Plotted above the training loss, test accuracy, and consensus error of CIFAR-10 (top) and the training loss, validation loss, and consensus error of tiny-shakespeare (bottom) with training done using the Adam optimizer across a number of possible K values from to 50.
Cestl to w20%
1.75
Coseraua Brocr
1.00
877
850
0.29
10*
1
20
Scaled Commonkatiin
Scaled Communiaatoon
Scales Conmurkaaion Ieturee
(a) CIFAR-10 reduction in communication volume.
The 200%
4.0
101
3.5
30-1
25
2.0
2.0
30*
(b) tiny-shakespeare reduction in communication volume.
Figure 3: Communication Volume: Plotted above are the training loss, test. accuracy, and consensus error of CIFAR-10 (top) and the training loss, validation loss, and consensus error of tiny-shakespeare (bottom) with training done using the Adam optimizer across number of possible K and top-k values.
AdaGrad and Adam are generally most performant overall, though the performance all methods contained within our framework is relatively similar. As such, will focus Adam in subsequent. results.
11

---

Published in Transactions on Machine Learning Research (04/2026)
berbubrdidem 4.4,Ping
2.0
Berbbubdddee s,Ring
Bierb.brdkdam N-N4--66600
Conseraa Srree
2.3
30-
0.3
Detiibueedddmmm
0.0
10
Spects
Spoc
tpodu
(a) CIFAR-10 with Adam agent scaling with ring topology.
Disy but mdlddam (e4,6
bucodkdam 2.1
16
but.ckda wow.dammmmmm....mmmm
70-
Tranting Lane
Conarrsua Eroor
2.0-
0.5
10-2
100
Epocts
(b) CIFAR-10 with Adam agent scaling with grid topology.
Figure 4: Larger Agent Counts and Differing Topology: Plotted above are the training loss, test accuracy, and consensus error of CIFAR-10 with training done using the Adam optimizer when scaling to 4, 9, and 16 agents on ring lopology (top) and 2D grid topology (boltom). All experiments were run with K = 20 local updates per communication round.
4.2 Number of Local Updates
Our next set of experiments analyzes the effect of the number of local updates per communication round on training performance. Figures Palaaaaaaaa aaland] give results with Adam training on CIFAR-10 and tiny-shakespeare using Top-k compression of 40% and 50%, respectively. We plot training loss, test accuracy caterracyyyliaaaai /validation loss, K 50. We run all
and consensus error against the total number of communication rounds for K experiments to the same number of epochs, which gives reduction in communication rounds proportional to K. For CTFAR-10, we note only a 1% loss in maximum test accuracy with K = 50 local updates per communication compared to the baseline K = instance. On tiny-sakesseaare, we likewise observe less than a 1% degradation of minimum validation loss when comparing K = 50 to K = 1 local updates per communication. We observed similar results with the same experiments on FNShionMNIST Figure t in the appendix.
4.3 Communication Volume
We continue our experiments by giving the relative proportion of communication volume used by our frame- work with selection of K and Top-k values, compared K baseline without compressed com- munication. Figures 3a and 3b give such comparison using Adam CIFAR-10 and sin--shakespeare. We overall observe a significant reduction in communication volume at relatively low cost to optimization quality. On CIFAR-10, we observe no loss in quality between K = 2 with Top-k 60% to K = 50 with Top-k = 30%, despite a reduction in commmication volume of 50x. Comparing K = 50 with Top = 30% the baseline with no compression or local updates, we note that we use only about 0.6% of the communi- cation volume. The maximum test accuracies across all experiments are within few percent of the baseline. We observe similar results kaay-sha when considering validation loss instead of test accuracy.
12

---

Published in Transactions on Machine Learning Research (04/2026)
2.0
10
1.5
Camoente Eree
10a
0.5
6.0
100000
Communkad MAmzz
Cotman teung
Commurkadon Bounes
(a) CIFAR-10 with non- IID data following Dirichlet 1.0 distribution.
2.0-
16
7.5-
K-1
Kxgz
60-
Consmaa Erio
The
202
0.5
0.
20000
200000
L400000
20000
1400000
Intenng
Jehees
J4t000
Communiation Awants
Commun calinn Sourcc
Communcation Poures
(b) CIFAR-10 with non-lII data following a Dirichlet = 0.5 distribution.
Figure Non-IID Data: Plotted above are the training loss, test accuracy, and consensus error when using the Adam optimizer on CIFAR-10 with non-IID data across K values from 1 to 50. We distribule data following standard Dirichlet distribution process using a 1.0 (top) and 0.5 (bottom).
4.4 Larger Agent Counts and Differing Topology
We demonstrate the linear scaling of our method by running experiments with 4, and 16 agents in ring and a 2D grid communication topology. Given in Figure g are plots of training loss, test accuracy, and consensus error for CIFAR-10 when using Adam optimizer. Similar results for AdaGrad and AMSGrad are displayed in Figure 10 in the appendix. We set K = 20 for all tests with Top-k = 40% compression. The 2D grid topology is defined as 3 x 3 for 9 agents and 4 x 4 for 16 agents. Note that the ring and grid topologies are oquivalent for 4 agents. We obscrve relatively consistent results across all optimizers, with near-lincar scaling in most cases. Minimum achieved training loss and maximum test accuracy are also relatively close, similar to as we previously observed in Figure across varying K values.
4.5 Heterogeneous Training Data
Our final experiments examine non-IID (independent and identically distributed) training data, emulating the client drift due to heterogeneity often observed in real decentralized environments. We use Dirichlet distribution to partition training data. We run experiments with Dirichlet(a) = [0.5, 1.0], using CIFAR-10 10, Adam optimizer, local updates K = [1, 2, 5, 10, 20,50), 50], and Top-k = 40%, with results shown in Figure 5 Our experiments were run for approximately 5x the number of communication rounds as the related IID tests shown in Figure 2a We note that in the less skewed setup with Dirichlet paramoter a = 1.0, our method achieves convergence and test accuracies similar to those Figure 2a though convergence occurs more slowly. Consensus error is also significantly more variable, as expected. However, the tests with larger numbers of local updates still convergence to the same test accuracies of the baselines given by optimizer comparison in Figure la in significantly fewer communication rounds. With more skewed Dirichlet parameter of a 0.5, we note a more reduced rate of convergence, with many lests failing to reach an adequate test accuracy. Further experiments with lower a parameters down to a = 0.1 showed correspondingly slower convergence with many tests failing to converge at all. Overall, these tests empirically demonstrate that our method offers some resilience towards heterogeneous training data, though other methods will likely be required if class label distributions are very significantly skewed.
13

---

Published in Transactions on Machine Learning Research (04/2026)
5 Conclusions and Discussions
We propose a local training based decentralized algorithmic framework with adaptive local update and com- pressed communication. The local update of our framework encompasses several well-known optimizers as special cases, inchuding vanilla SGD, momentum SGD, Adam, AMSGrad, AdaGrad, and Adam-Mini, and the established convergence results apply to all these optimizers. To the best of our knowledge, this is the first work provide theoretical convergence guarantees for adaptive stochastie methods in MLTTbased decentralized nonconvex optimization. Our empirical experiments further highlight the compounded bene- fits of integrating MLT with CC, demonstrating significant reductions in communication overhead without compromising convergence speed or accuracy.
Our current analysis relies on a bounded-gradient assumption, and our language-model experiments are limited to a small-scale transformer benchmark. Extending the framework to weaker assumptions, stronger data heterogeneity, and larger language models natural direction for future work.
Acknowledgements
The authors would like to thank three anonymous reviewers for their valuable comments. This work is partly supported by NSF grant DMS-2208394, ONR grant N000142212573, and also by IBM through the IBM-Renssclaer Future of Computing Research Collaboration.
References
A. F. Aji and K. Heafield. Sparse communication for distributed gradient descent. In EMNLP 2017: Confer- ence on Empirical Methods in Natural Language Processing, pp. 440 445. Association for Computational Linguistics (ACL), 2017.
D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. Qsgd: Commuicatiinn-fficint sgd via gradient quantization and encoding. Advances in neural information processing systems, 30, 2017.
K. Andrej. NanoGPT. https: https://github.com wwt.:/ccc/aaaaaa hwt.buu..cmm/krrpttth/aaaaoooooo
2022. Last Accessed March 2025.
D. Basu, D. Data, C. Karakus, and S. Diggavi. Qsparse-local-sgd:: Distributed sgd with quantization, sparsification and local computations. Advances in Neural Information Processing Systems, 32, 2019.
J. Bernstein, Y. Wang, K. Arizzadenesheli, and A. Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp. 560 569. PMLR, 2018.
C. Chen, Shen, H. Huang, and W. Liu. Quantized adam with error foedback. ACM Transactions Intelligent Systems and Technology (TIST), 12(5):1 26, 2021.
C. Chen, L. Shen, W. Liu, and Z.-Q. Luo. Efficient-adam: Communiatiion-ffiiiettt distributed adam. IEEE Transactions on Signal Processing, 2023a.
L. Chen, W. Liu, Y. Chen, and W. Wang. Commuication--ffiieee design for quantized decentralized federated learning. IEEE Transactions Signal Processing, 72:1175-1188, 2024.
X Chen, M Hong, S Liu, and R Sun. On the convergence of class of adam-type algorithms for non-convex optimization. In 7th International Conference on Learning Representations, ICLR. 2019, 2019.
X. Chen, X. Li, and P. Li. Toward communication efficient adaptive gradient method. In Proceedings of the 2020 ACM-IMS Foundations of Dala Science Conference, pp. 119 128, 2020.
X. Chen, B. Karimi, W. Zhao, and P. Li. On the convergence of decentralized adaptive gradient mothods. In Asian Conference on Machine Learning, pp. 217 232. PMLR, 2023b.
L. Condat, I. Agarský, and P. Richtárik. Provably doubly accelerated foderated learning: The first thooreti- cally successful combination of local training and communication compression. Preprint, orXiv:2210.12 13277, 2022.
14

---

Published in Transactions on Machine Learning Research (04/2026)
L. Condat, A. Maranjyan, and P. Richtárik. LoCoDL: Communication-cfficin distributed learning with local training and compression. In The Thirleenth Internalional Conference on Learning Represenlations, 2025. URL
J. Duchi, Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic opti- mization. Journal of machine learning research, 12(7), 2011.
H. Gao and H. Huang. Adaptive serverless learning. arXiv preprint w:Xi:22088114222 2020.
S. Ge and T.-H. Chang. Gradient tracking with multiple local sgd for decentralized non-convex learning. In 2023 62nd TEEE Conference on Decision and Control (CDC), pp- 138- TEEE, 2023.
F. Haddadpour, M. M. Kamani, M. Mahdavi, and V. Cadambe. Local sgd with periodic averaging: Tighter analysis and adaptive synchronization. Advances in Neural Information Processing Systems, 32, 2019.
F. Haddadpour, M. M. Kamani, A. Mokhtari, and M. Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference on Artificial Intelligence and Statistics, pp. 2350 2358. PMLR, 2021.
L. Hou, R. Zhang, and J. T. Kwok, Analysis of quantized models. International Conference on Learning Representations, 2018.
T. H. Hsu, H. Qi, and M. Brown. Measuring the effects of non-identical data distribution for federated visual 2019.
classification. Preprinl, arXie:1909.00
P. Kairouz, H. B. McMahan, B.
Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Charles, G. Cormode, R. Cummings, et al. Advances and open problems in federated learning. Foundations and trends@ machine learning. 14(1-2):1-210, 2021.
S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference machine learning, pp. 5132 5143. PMLR, 2020.
P. Kingma and J. Ba. Adam: A method for stochastic optimization. Preprint, arXiv: 14X2v:1412.69900 6980, 2014.
A. Koloskova, S. Stich, and M. Jaggi. Decentralized stochastic optimization and gossip algorithms with compressed communication. In International Conference Machine Learning, pp. 3478-3487. PMLR, 2019.
A. Koloskova, N. Loizou, S. Boreiri, M. Jaggi, and S. Stich. A unified theory of decentralized sgd with changing topology and local updates. In International conference on machine learning, pp. 5381-5393. PMLR, 2020.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
A. Lalitha, O. C. Kiline, T. Javidi, and F. Koushanfar. Peer-to-peer federated learning on graphs. Preprint, MXX1:1901.11112 2019.
Guanghui Lan. Pirst-order and stochastic optimization methods for machine learning, volume Springer, 2020.
Y. LeCun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
L. Li, Y. Fan, M. Tse, and K. Lin. A review of applications in federated learning. Computers & Industrial Engineering, 149:106854, 2020.
X. Li, W. Yang, Wang, and Zhang. Communication efficient decentralized training with multiple local updates. stat, 1050:21, 2019.
15

---

Published in Transactions on Machine Learning Research (04/2026)
X. Li, B. Karimi, and P. Li. On distributed adaptive optimization with gradient compression. In International Conference on Learning Representations, 2022.
W. Liu, A. Panda, U. Pandey, C. Brissetle, Y. Shen, G. Slota, N. Wang, Chen, and Y. Xu. Compressed de- centralized momentum stochastic gradient methods for nonconvex optimization. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL http:p///pgaaarooee....rr//raaap wtt.://oruurrdddgghott.itn...e.e
G. Mancino-Ball, Y. Xu, and J. Chen. A decentralized primal-dual framework for non-convex smooth consensus optimization. IEEE Transactions on Signal Processing, 71:525 538, 2023.
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication--ffciee learning of deep works from decentralized data. In Artificial intelligence and slatistics, pp. 1273-1282. PML.R, 2017.
P. Nazari, D. A. Tarzanagh, and G. Michailidis. Dadam: A consensus-basee distributed adaptive gradient method for online optimization. IEEE Transactions Signal Processing, 70:6065-6079, 2022.
A. Nedić, A. Olshevsky, and M. G. Rabbat. Network topology and commuicatioor-comittaaon tradcoffs in decentralized optimization. Proceedings of the IEEE, 106(5):953 976, 2018.
S. Reddi, S. Kale, and S. Kunar. On the convergence of adam and beyond. In International Conference Learning Represeniations, 2018.
S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, Rush, Konečný, S. Kumar, and B. McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2020.
S. J. Reddi, A. Hefuy, S. Sra, B. Poczas, and A. Smola. Stochastic variance reduction for non-convex optimization. International conference on machine learning, pp. 314-323. PMLR, 2016.
A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani. Fedpaq: A communication- efficient federated learning method with periodic averaging and quantization. In International conference on artificial intelligence and slatistics, pp. 2021-2031. PMLR, 2020.
Singh, D. Dala, J. George, and Diggavi. Squarm-sgd: Communication-ellccenn momentum sgd for decentralized optimization. IEEE Journal on Selected Areas in Information Theory, 2(3):954 -999999999 2021.
A. Spiridonoff, A. Olshevsky, and Y. Paschalidis. Commmnication-efficcne sgd: From local sgd to one-shot averaging. Advances in Neural Information Processing Systems, 34:24313 0244313-4326,, 2021.
S. U. Stich. Local sgd converges fast and communicates little. Preprint, ar Niu:1805.09707 2018
S. U. Stich, J.-B. Cordonnier, and M. Jaggi. Sparsified sgd with memory. Advances in neural information processing systems, 31, 2018
J. Sun, X. Wu, H. Huang, and A. Zhang. On the role of server momentum in federated learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 15164 15172, 2024.
Sun, Li, and B. Wang. Decentralized federated averaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4289 4301, 2022.
X. Sun, N. Wang. C. Chen, J. Ni, A. Agrawal, X. Cui, S. Venkataramani, K. El Maghraoui, V. V. Srinivasan, and K. Gopalakrishnan. Ultra-low precision 4-bit training of deep neural networks. Advances in Neural Information Processing Systems, 33:1796 1807, 2020.
Y. Sun, L. Shen, H. Sun, L. Ding, and D. Tao. Efficient federated learning via local adaptive amended optimizer with lincar speedup. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12): 14453-14460 14464, 2023.
N. Wang. J. Choi, D. Brand, C Chen, and K. Gopalakrishnan. Training deep neural networks with bit floating point numbers. Advances in neural information processing systems, 31, 2018.
16

---

Published in Transactions on Machine Learning Research (04/2026)
Y. Wang, L. Lin, and J. Chen. Commuicction--coppesss adaptive gradient method for distributed non- convex optimization. In International Conference on Arlificial Intelligence and Statistics, pp. PMLR, 2022a.
Y. Wang, L. Lin, and J. Chen. Communcation--fficeett adaptive federated learning. In International conference machine learning, pp. 22802 22802-22838. PMT.R, 2022b.
Z. Wang, J. Zhang, X. Wu, and M. Johansson. From promise to practice: Realizing high-performance decentralized training. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevieee hetp://rrttttoormmmmddooooodo10
T. Wu, Z. Li, and Y. Sun. The effectiveness of local updates for decentralized learning under data hetero- geneity. TEEE Transactions on Signal Processing, 2025.
Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. Preprint, woxxix:778.000 077807777, 2017.
L. Xiao and S. Boyd. Fast linear iterations for distributed averaging. Systems & Control Letters, 53(1): 65-78, 2004.
C. Xie, O. Koyejo, I. Gupta, and H. Lin. Local adaalter: Commuiccation--ffiiinntt stochastic gradient descent with adaptive learning rates. Preprint, arXiv:1911.09030 2019.
H. Xing, O. Simeone, and S. Bi. Decentralized federated learning via sgd over wireloss d2d networks. In 2020 IEEE 21st international workshop on signal processing advances in wireless commumications (SPAWC), pp. 1-5. IEEE, 2020.
Y. Xu, Y. Xu, Y. Yan, C. Sutcher-Shepard, L. Grinberg, and J. Chen. Parallel and distributed asynchronous adaptive stochastic gradient methods. Matheralicaa Programming Compulation, 15(3):471 508, 2023.
Y. Yan, J. Chen, P.-.. Chen, X. Cui, S. Lu, and Y. Xu. Compressed decentralized proximal stochastic gradient method for nonconvex composite problems with heterogeneous data. In International Conference on Machine Learning, pp. 39035- 39061. PMLR, 2023.
H. Yu, S. Yang, and S. Zhu. Parallel restarted sgd with faster convergence and less communication: Demys- tilying why model averaging works for deep learning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5693-55 57000, 2019.
H. Zhang, Y. N. Dauphin, and T. Ma. Fixup initialization: Residual learning without normalization. In International Conference on Learning Representations, 2019.
S. Zhang, A. E. Choromanska, and Y. LeCun. Deep learning with clastic averaging sgd. Advances in neural information processing systems, 28, 2015.
Y. Zhang, C. Chen, Z. Li, T. Ding, C. Wu, D. P. Kingma, Y. Ye, Z.-Q. Luo, and R. Sun. Adam-mini: Use fewer learning rates to gain more. Preprint, ftl:2406.16793, 2024.
R. Zhao, D. Morwani, D. Brandfonbrener, N. Vyas, and S. M. Kakade. Deconstructing what makes a good optimizer for autoregressive language models. In The Thirteenth International Conference on Learning Representations, 2025.
W. Zhao, X. Jiao, M. Hu, X. Li, X. Zhang, and P. Li. Paddlebox: Communictiinn-eficiennt terabyte-scale model training framework for online advertising. In 2022 IEEE International Conference on Big Data (Big Data), pp. 1401-1408. IEEE, 2022.
17

---

Published in Transactions on Machine Learning Research (04/2026)
A Convergence analysis of compressed decentralized algorithms with multiple local adaptive gradient updates under nonconvex settings
In this section, we give complete analysis of our decentralixed algorithmic framework. We write the updates of Algorithm 1 in the more compact matrix form for all & [TK],
M 8,M* + (1 G,)
(10)
Update U* 2 0,
(11)
M
Y
(12)
VU--+
X = -X' - aYY
(13)
if mod(t (t + K) 0, then X X' Q -x-
x x++ x X 1 (W - I),
(14)
(W
(15)
clse, X X x =X
(16)
where
G [ 22. ....9........................... [.]. M' = ml m ......., mm] X = ,,...., Ixl
Q X [0 a 1, [(0²] ..... [(a,]]-
We let
= x1, X = XJ = x1 m' M'1, 8' = -Y11-11, Y = YJ.
First establish bounds the sequence (M'), {U') and [Y'}.
Lemma A.1 Under Assumplion
it holds that for any E [TK];
|MI (1 1 vnB < VnB, | vnB8- 1 | s vnB8-1,
(17)
|m B, | I B, | Box Vi E nI......,n.
(18)
Proof. From the update of m, i.e., m = 8im 1-1 + (1- /i, we have that for any t 2 0 and each E n!) m || + (1- Biggilss
+|+(1 || 8, llm +1x B;)
where the second incquality holds from above and noticing m, 0, we obtain
|اgوو= B by Assumption
3 Recursively applying the incquality
Imill (1 + 81 + 89 + +... 8) (1 - B:)B = - (1-)) B B B.
Hence, it holds |m B and || (1 11t VnB. Now by U 2 0, we immediately have | M -u- vBB8- ; and Y 2 Y Y² ||
aB² 8
In addition, we have that for any / 2 0 and each i i (1,2,....
m 110== [1 + (1 - gf 00 < f1 - 11oo - 31) g/ 31 3 +1 - Boc
where the second inequality follows from |1g/ B by Assumption Recursively applying the inequality above and noticing m. 0, we obtain
m ح (1 + 31 + + + +11)) (1 BB) = (1 (1- 1-- Bao S Bre-
The proof is then completed.
The next lemma shows the bound of the consensus error of X.
18

---

Published in Transactions on Machine Learning Research (04/2026)
Lemma A.2 Under Assumptions let p = 1 0 < operator with n 2 0. Then, the following statements hold:
(1-p)(1-2²)
and on n-compression
100
(i) For any E [T], it holds
70)
(1
)P)
E ||
E
x
1 1
16
2
(19)
4
1+
IP 16
(1 1 +
2
E
x
- x
+2 [
2211+
16 7p
a Q²K21
(ii) For any [T], it holds
E
x
(x(1))
( :2
1
PP
X (2]
16
1 - 17
3 T
(1+
1P
| - -x x
+
(1 + 87)
E
(20)
4
16
16
1 1 1- ))
+
3
1 +
+2
a² n222'
PP
(ii) It holds
TK-
E
X [2]
S aZnK² where C
56
80
15
B28
(21)
TK
7p
70
1 - n/?
If in addition 7 @(1 p)(1 -))), have := 1 (1-p) 11-1711
(1-p)10
B²
Proof. (i) From the update rules
13
16
we have, for all 8 E IK -1.
X -
and for the final step in the block of size K, we conduct the compressed communicttion step.
First, using 18 and the inequality (a + b)2 S (1 + m))a + (1 - oi 3 for any m > 0, we have
| 2= x k
ki 1
1=
(22)
<(1 ())l
| (11 + 1/1 o2
Σ yrk+i
1=0
<(1 (1++m)ll
2 + गパ K 22 2B8
As [K - 1].
Next, analyze the case of s = K - 1. By
15 it holds that
x (r+1)K = x (r+1) + - (w I).
Noticing (r+1))] ( 1J from
and J(W 0, we have x (r+1)) = ( ( (++)) -
14
(X (++1) -X(+1) 1).
(23)
19

---

Published in Transactions on Machine Learning Research (04/2026)
Denote W W= (1- wW. For any n2 > O, it then holds
x (x+2)k
2 + (1 1) 17 ( - (w -1|l
5(1 + m22)
x ((+1))=+ 1 J)
<(1 + 12)
(2)
x ( (I (I-J) J)
+ 4 (1 122 (x -
- x((t))k-3))1
(24)
where the second incquality follows from W - 1 2. Recalling 7 1 - p and by (I JJJ- = 0, we have
x ( K 1 [I - J)
<(1 7)
X (++)) - $ (I J)
X (++1) (I J)W
=(1- - 7)
x ++1) 3 a[I J)
- [[I () ]J)
<(1 7)
1 1(1 J)
3 [I - J)
=(1 - 7
(I -
(25)
Substituting 25
into 24 we obtain
x 1
(1- -)] + +4 +1(1+11) 1) 7² x (((t)) - x
S(1 + 92) 777)
x x (I
(26)
For the first term in the RHS of 26 using the bound || ||S VnBBO-2 2, we have
(((+1) (I (I-J) 1 || xx((+)K-1) aI (I-J) |?
x -ax(t)---
(1 m)+m) X 1 + (1 + a² k3
(27)
For the second term in the RHS of (66). using the bound 1s vnB8 1, we have
K-1 yrk+i
x - - x 2 = x x(++)) -xrk ta Σ
i=0
K-1
<(1 (1+mi) =11+m)]
x x - xxx 2
+ (1 + nr )a)
Σ
yrk=i
i=0
<(1 + m) -
-
+ (1 + n a² 2223n888-1
(28)
Plugging 27
28 back into 26 we arrive at
x { 1++
5(1 + 12) - 2 (1+m) X 1 2 +4 +1 m5 > m) X (++1)K
-x
+(11 (11+10 (1) + 4 (1 + (1+ )) (1+ a² K 222n212 --8
(29)
20

---

Published in Transactions on Machine Learning Research (04/2026)
Let 171 = 20 16
and 1/2 = 2. By the definition of 7, 0 < 7p < 1, we then have
1 + y0
(1 1
PP
2
(1 + 12) (1 -0) (1 (1)
(30)
16
+ (1++55) +1 m1) 4 1+ yP
2
? 1
(31)
16
P)
16
((1 12) - 4 (1 1/2 )) 7)) (1 1i S 2
(32)
7P
We then complete the proof of 19 by using the inequalities -00-32 32
in 29
(ii) From
14]
and Definition
for any 77s > 0, it holds that
E
x (r+2))
X
x
= = x o x ( - x - ((+1))
=E
x (r+2))-1
x (x+2)x= - x(r+1jk 2 - -
((+2))
<(1 + ma)EE
X
x X X e
((++))=-1 x ?
+
S(1 <(1 11-
X (x+2))K-
- ((+2))=11) -1 I|
+ (1 + 1/3 )
x (r(+2) - x(+
(33)
For the second term in the RHS of 33 using the bound
| VnBd- 1, we obtain
K
((++))K
1)
2
S <=²
(34)
=
1m0
For the first term in the RHS of (33 by X (r+2)k-1
x 1)K and
| VTBO 2, for any na > 0, we
have
K-1
x -
E
= x
(+1))
a
Σ
-
1=0
K-1
<(1 + 1
((++1)K_ x ((+1))/
(1+ + 1/4 1)
a
(++1)K
1=0
<(1 + 71 7E
-(++))-_ - x
(1+ + m4 2022
(35)
Substituting 34
35
into
33
we have
E (++2)k
x(++))
5(1 13)1? 2
(1 14) - X(+ +(+ ni k)
+ (1 + 13 1)a² 2 B881111
(36)
x ((+1)) x ((+1)) ||
We now bound
By 23), we have that for any 7/5 > 0,
x (++1) - x(+t)xx
E
(x(+1)) - x(+1)K++) (w - II + 7x x 3 J)(W - -117
=E
I)
<(1 7/5) + 2T)² - x ||
+ (1 15 4) 23
1 11+ 12]
17.
(37)
21

---

Published in Transactions on Machine Learning Research (04/2026)
where have used JW J in the equality and (l I) - s >/ W 1| [ + 2y and
||W- - || 2 in the inequality.
For the first term in the RHS of 37 we know from 28 that for any > 0,
X X
1(++
$ (1 + m.)
x ( x + a
(38)
For the second term in the RHS of 37 we know from 27 that
x (r
k
x
( +(I - 1/Il s s1 + m) |x| + (1 a)² 22K2nB=
(39)
Plugging
38
and
39
into 37
we have
E x (r
x x
1 (1 + 15'5 47² (1 m1)) X + (1 + (1+ a²K2nB280)) (2++""")"KBB3'
(1 + 15) (1 + 27)² (1+m) | X - -x-x| 12 + +1++""")2223B3333
(40)
Combining 40
and
36
arrive at
X (r+2)K ((++)) 2
E
<(1 (11+1)
77) (a)m7))
+ n4) (1 + (1+ 47 (1 [1 47 | ||T
+ (1 - 77)7/ + 714 (1 + 15) (1 + 2/)²² (1 +m) ( - -x*x|²
+ (1 77)172111110) (1 + m)) 11) (1 // KB200
+ (1
+ (1+"
+ (1
+ 1 (1 2o
(41)
Let 1/3 = 1/4
175
1-
and 1/1
10 By 7
(1-م)(1-²²) and 17 1, we have 100
3+
(1 11 7P
(1+ 11) + T1) (1 + m1s) (1 + 27)²
S
(1+87)
(42)
4
16
(1 11) + m)) (1 + 155 472 (1 + m1 < 4h2
1
7P
4
(43)
16
1 1/2
16
(1 m0) n (1 + 7/5 )) 17² n. ) s
(44)
7p"
16
(1 (1+m))1711 7/4 (1 + 75) (1 27)² (1 + n.
2
1
(45)
?p
4
(1+ मह ) (1 no 2 1+
(46)
r?
We complete the proof of 20 by using the inequalities
46
in
41
(iii)
Denote Ω
=
x K
+E
x
x ((++))
|"]
21
Then, since 0 < 17 < 1, the two inequalities in 19
and
20
imply
g++1 < Anst" + A1,
(47)
22

---

Published in Transactions on Machine Learning Research (04/2026)
where
Ao
1 +
7P
max
(1-3) TP
+ 447 2
4
4y
11 2
+
3 1²
(1 + 87)
(1+87)
(48)
16
2
1- n7
p/
80
15
A1
+
a n²K21
(49)
7p
- n?2
By P p- it holds that
(1 p)(1 - 12)
2p(1 - x/2)
7p(1 - 172)
7 5
< min
(50)
100
102 32p + 64 + 48p + 16p1²² 128 + 2 112))
Notice that 7 5 7pl 1p
128 12842p"(1-17) yields (1-
72 2
+ 472
4
S -1
22
and 7 S p2+32p++61- 48p 16pn
2pl 1
implies
1-
x
4r 1 + 2) + 3+n (1 + 87) - 1 5g Thus
7:0 Ao 5 (1+
(1-
p7
7P
S
<.
16
8
16
From x7 = x = x.. Vie f1, nn,, we have
|X] 1== 0 and x²= =0 By
36
with r = -1 and (46). we
have
E
x
K
< 2 ( 1 T- 7 a² nB - A 7P
Thus, multiplying both sides of 47 by Al A and summing it over k = 0, o,.,..., 1 gives
16
SAp+ Ω2
A5- * A1 S
22
A1
7p
k=
16
X x 17
-1 A1 + E
(51)
PP
16
18
A1
1+
k²
A1.
(52)
PP
1 n
PPP
Summing up the above incquality for all
0,1,
T - 1, we obtain
Σ r=U
E
K 1
A1. PP
18
(53)
T
Summing 22 with ?/
over s E {0, 1....
..... K - 2),, we derive that, for all [T 1),
1 K-1
x
* 2
K
This implies
1
K
Thus, we have
1 TK-1
K-1
EX
P-=
Σ
E
x
11
Σ
142K3220'3'
56
A
TK
K
pp
t=0
23

---

Published in Transactions on Machine Learning Research (04/2026)
This completes the proof.
To prove the convergence of our algorithm, we define an auxiliary sequence as follows
B1 31 (x' - -0'=x""))
(54)
with x- x" The lemma below shows the difference of two consecutive z-points.
Lemma A.3 Let } be defined
54
It holds that for all t [TK],
2 ++1 z¹ B1
a
m
1-1
1 +o
1
a
9
(55)
B1
i=1
Vui
Vu:= +
i=1
6
where u,
2-
Proof. By
12 16
and
(W I)J= = 0, we have
x
x²
0
m
(56)
im
u
+
8
Thus by
54 we have
91
B1
T
-ー
31
31
81
E
x1
:
B1
1
B1
Q
m
B1
n
m
31
11
1=1
8
1
$1
11
i=1
u
2
+6)
1
a
m
1-1
(1
31
a
m
1-1
1-
31
1=1
V
1-1 +
B1
1=1
u
2-2
+ 8
B1
0
n
1
1
a
g
31
m;
;
u - +
Vul
1-1
+
12
u
8
which is the desired result.
Lemma A.4 Under Assumptions
1 and 3 it holds that for all t t [TK],
I
(V (i:) - v (z))
X 11
(57)
and
IV. 1 {) Q²
(58)
8(1-31)3
Proof. First, by the L-smoothness of f. for each i i {1, 2,
n} and Young's inequality, we have
1
(x)) - vl (o))
<
x क 1
71
24

---

Published in Transactions on Machine Learning Research (04/2026)
which indicates 57 by the definition of X
Also, by the L-smoothness of f, it follows
L2
[V'
"f(=")|| L 1
||
54
112
(1-
p
m: t-1
3
2
m
t-1
B²L22BBB²
B1)²
i=1
ui
2 +
8
31
n
u
+
8(1 -1)
where the last inequality holds by [mm 1mm--|| B from
18
This completes the proof.
Lemma A.5 Under Assumplions 3 it holds that
TK-1 Σ
E
|||
=²11²]
1=0
B2 E (1-1,)
TK-1
Σ 1=0
(59)
1=1
1- +
+
+ 2a
24
T TKB +
61.2
E
P
X
11
6 E
TKK
Σ 1=
||
no
no
t=
Proof. By
555 and Young's inequality, we have
TK -1
E
1xt1 - 2²1?]
0=
TK-1
B1 a
SR.
2
Σ
m:
1=0
$1 n
uu
+
V
/u
+
27
TK-1
+ E
2
Σ
9
(60)
n
u
1
To bound the first term in the RHS of 60
we obtain from
18
that
TK
n
(-
m
72
u
1-2
+
u
5
TK
m
Vui
1-2
+
8
u
1-1
TK-1
(61)
u
2-2
25

---

Published in Transactions on Machine Learning Research (04/2026)
To bound the second term in RHS of (60), we have
TK-
9
21E
n
u
4-1 1
6
TK-1
g
V)((2)))) + ()(()) V + V f(
22
Vut- +8
TK
g
TK-1 1
(f((t))) (
<6E
Σ
Σ
u - +
u ( +
TK-1
V ff(=')
1=0
+
TK
(g) v(((a()))
TK-
(ff((()) (+'('))
<6E
Σ
u +
t=0
u +
TK-1
V ff(z')
/ 1 + 6
1=0
6
TK- 1 n Σ
9 (a((
TK
[V(( vi((z')
E
+ +6E
n
m - +
u - 5
V fx
+ 6E
Σ
Vus 5
TK-1
TK
6
1 Σ (V VA
(=
20
n
TTK-1
+ 8
E
Σ
1=1
4
6L
E
TK-
Σ
X
2
6 E
TK-
Σ
V f(x ||?
(62)
n28
no
where in the last incquality, we have used
57 and the equality holds because
(g)
vf((a))
(g) (z)
E,
ut +
Vul 8
E, [g vf((a)))
Eglg) - v[s(ar)))
0, Vi ド
u +
1 +0
from the fact that gi.g...... gi..... we complete the proof.
9 are conditionally independent of each other. Plugging (61) and
622
into 160
26

---

Published in Transactions on Machine Learning Research (04/2026)
Lemma A.6 Suppose Assumptions and 8 hold, and >/ Bu for all t 2 and i (1, n). It holds
Eg (5
1
9
(63)
71
i=1
u
+
1
I?
(VBu
+
1
Q²BRI3BB
VB, +
1
Vf
x
1?
X 1|2
+
2vB. B, +
n
2/8
8(1-31)² 31)2
2
Proof. By Assumption 3 it holds that
1- + 8 11 v. ()()))
E,
Vf (z'), 1
Σ
g
Σ V)(())))
u
1=1
1
(fi( - V {((3')
Vf (x)) vf(T')
72
Vul-L + +
1 Σ
V=(²')
(x') - vf(z")
+
V /44 +
1=1
V)'z') 1 n2
[f((z)) - vf(x'))
Vf(=")
f((z
(64)
1 +
8
Next we bound each of the four terms in the RHS of 64 For the first term in the RHS of 64 we use Young's inequality and (58) have
(5()') - V)(a') -(())) (Vf((a)) V)
/ut + 5
1
Σ
(I (z') - \)(=')|
|| (x!) - VS. (z'))1)
2
1
Σ
(lVf (x*) - -イ
-3 E²11")
2nvo
i=1
1
L² B²
IX111
L
(65)
2V8
5(1
B1)²
where we have used u! 1-1 2 0 in the first inequality. For the second term in the RHS of 64h, we have
n
Vf(3))
(5(2') Vi(T')
(66)
i=1
u +
VS (3'))1
VB, +
V (2*) - 2 2
4/B,, +0
V (x() )|2
12 +0
(67)
4/B, B, +
811
27

---

Published in Transactions on Machine Learning Research (04/2026)
where the first incquality follows from Young's inequality and u 2 0. For the third term in the RHS of 64p. we have from Young's inequality that
1
(Vi((a)) Vff('))
V
u 1 +
| (3')
VB, +o
V (1.) V.. ( )|²
4yBB B. +
6
|/ (a>')
L2VBB +8
b x 11²
B/BB +0
8
(='()))1 L3 +
(68)
A/B, +8
nd
Since | By for all t 1 0 and i E (1.2...... the last term in the RHS of (64 can be bounded as
1
VS
VB VV (x' (z)
(69)
Substituting 65
69 into 64 and rearranging terms yields the desired result.
Now we are ready to show the main convergence result.
Theorem A.1 Suppose that Assumptions 3 hold, Q is an n-compression operator, and ||ut]l Bu for all t 2 0 and E (1,2...., Let C denote the constant defined in (21), a, 7 0 satisfy
8
(1 - p) n7²)
75
(70)
48L B, +8'
100
Then, holds
TK-1
aTK B²
a
Σ
E || VJ 44) 11 [f (x0) -
AVB, +6
8 8vB, +0 8(1- - 31)
a,s B²
TK
1
1
+ (1 - (1-B))²
(A Bu + d. + all) E
2
+
/ +6
(71)
24
6L2 + no
+ L
TKB2
a TTKK'
no
+
aL?
VBu+ +
T²Tnk*C++
+
TK-1
Σ
L233LLB²
VB, +8 1
n
6
2/8
8(1 - 31)=
8
2/8
t=0
Proof. By the L-smoothness of f, we have
f f( < f (z²) + (V
2411 x²) + 2
2 _ x" -
which together with 55 gives
(z Ss (z') - a v. (z') Σ
g
||
| - =11
+ 2
1=1
u + 8
B1
1
(z))
(z²)
1-1
m
T B1
Vuu
2-2 +
Vul +
28

---

Published in Transactions on Machine Learning Research (04/2026)
Take expectation, sum up over t, and rearrange terms of the above inequality. Noticing =0 = =x,, we have
)
a Σ
TK-1
E
(2')
1 9
Σ
12
S - (=
-
-
TK-1
Σ
|| -(11" =91(²]
n
5
(72)
a
TK-1
Σ E
V (z')
- Σ
1-1
mi
31 t=
1=1
u
+
V
u 1-
Below we bound the inner-product terms on the RHS of 72 First,
81
TK-1
Σ
E
V
(z'). 1
m;
1-1
1
1
1 B1 0=
71
u
1-2
+
V
/ui
1-1
+
8
1 - B1
31
TK-1
Σ
E
V (x') Σ
m
1
8
v
uut-1 +
1
TK-1 Σ t=
+
31
E
- (T' (f
Vi
1
m
1-1
1
1
(73)
31
i=1
V u
+
8
Vul.
+ 8
For the first term in the RHS of
73 we use Young's inequality to have
81
TK -
a
Vi
(T')
m
1-1
o
1
B
t=
u
+ 0
v
|u
8
TK-1
Σ
1
E
V
B/B +8
TK-1 Σ
2,3 B,, +0
E
1
Σ
m
1 Vul
1=0
(1 )1
u
1-2
+ 5
v
8
TK-1
Σ E
[
8/B. Ba +6
TK-1 Σ
233
B + 1
E
(74)
1=0
(1 31
71
2
+ 0
Vu 1-1
+6
29

---

Published in Transactions on Machine Learning Research (04/2026)
where in the last incquality, we have used "Ico S Boo by Lemma it holds
A.1 For the second term in the RHS
of 73
81
TK-
Σ
E
m
1
1
81
Vu 2 +
V Vul
t=0
TK
Σ
1
E ||V (2') - v (=*)
8v Bu +0
TK-1
209 Br +o
1 Σ m
E
1-1
t=
(1 31
Vul 2- +8
V /ug 1-
+ 8
TK
B²
TK-1
BA, + 8
Σ
E
(75)
8
BB +0 8(1 +0 B1)
B1 2
2
+
u
1
where in the last inequality, we have used we oblain
58 and
m 1-1
1l S Boo by Lemma
A.1 Plugging
74
and
75
into
73
a
TK-1
Σ
E
1 m (f
Σ m
1
1
B1
Vul +
1-1
8
TK-1
E [ ||9]
aTK
121228BB2
8v B 8
8v 8/B, 8 5(1 B1)
1=0
4a B²
By +
TK
Σ
Σ
E
(76)
(1-31)²
u +
Vu 1 + 6
0=
Now plugging
59
76
and
63
after taking full expectation into 72 and rearranging terms gives
30
2
TK-1
aTK Q2T28BB22
E
(1) - f (xT))]
+
8V B, +0
8
8 B, + 8(1 -11
ag9 B2
TK
1
+
(4v Bu + +
E
(1 B1)²
u 2
+
V uut + 5
24
6L2
TK-1
X 11/)
+ 2T
TKB2
Σ
nd
nd
t=0
TK-1
L22
VB, +0
VBa + 8
1 2v8
+ a
Σ
B|||
n
8
2/8
5(1 -1)
0
Plug 21 into the incquality above, notice
3a
6La²
2
and rearrange terms. We obtain the
desired result, and complete the proof. 8 BBB,+0
4 4/B,47'
30

---

Published in Transactions on Machine Learning Research (04/2026)
To prove Theorem 3.2
only need to consider the following three settings of {U*}
AMSGrad : Ü = 32 - + +11 (1-31)) 32) G' with Ô-1 = o, U = max (U'
(77)
A
1- + (1 (1-33) G²; U = U
(78)
AdaGrad
U 1
Σ
G G
(79)
[+1
==0
where
lnd of U1 U2 ..... T/] U = [u] u......,"!!
(80)
Notice that when 81 = and 82 1, AMSGrad reduces to the vanilla SGD, and when 81 E (0,1) and f2 = 1, it reduces to the momentum SGD. Consequently, our theoretical guarantecs on AMSGrad naturally extend to these two special cases well. Adam-Mini (Zhang et al.) 2024) can be regarded a special case of Adam by using constant scalar (instead of a vector) for each block of variables the second momentum. Therefore, the results Adam also hold for Adam-Minn
Below we bound || and the summation of the difference between the consecutive terms in the sequence TK-1 for the three optimizers in 77 (79..
( o
Lemma A.7 Let u; = 0 for all i = 1, ,,,,.. the following statements hold.
n. Under Assumption 3 for all 1 0 and i E (1,2....,mh)
(i) For AMGGrad
77 it holds ||e. Do B and
TK 11
d
S
(81)
u
+
Vui
+
8
(it) For Adam in
78
it holds
|/u| 1100 S B and
TK
TKd(1 - 2
(82)
2-2 +8
V vu: +
83
t=
i=1
(iii) For Adagrad in
79
it holds | |ui B²g and
2
n
2d
<
(83)
n
Vu
+
Vul 1-1
+
83
Proof. (1) Noticing a, = 0 and || o g. 2x B2,, we have | (1 -1/+1) B² for each E n..
and t2 0. By ut = max (ū,, ut 14-11 1), it holds
||ui max ( |l || ! -1 1 S max ( 1
1--11
olo0 1/ 2 2looc
< max [||@ oc. ||
llo..... | bloc, || 11o0
max {(1 1 1 B² (1 (8) B² B2...., B2) B² |
1 [00] =11- B²
where the equality holds because B2 (0, 1] and u. 0.
31

---

Published in Transactions on Machine Learning Research (04/2026)
In addition, we have
TK-
Σ
1
1
t=0
2
+
Vul"
+ 8
PK-1
Σ n
1
1
$
2
+ 8
V
/ui
+
8
2
1
TK
Σ
V8
2
+ 8
V
u
+ 8
(84)
TK-1
1
2
+
8
u
1
+
n TK-1
Σ
n
8
1
8
1
nv8
2
+ 0
where
1 holds because ut - 2 and ut 1-11 2 0, and the equality holds because v8
Vu 2
Vu: +88
1-1 is nondecreasing with for each i E {1, u:
nn.. nn..
(ii) Noticing u, = 0 and logf g B we have || -1)tt) B%.
For all 2 11 it holds
U +0
82) g.l, + [u + 8]
j=1
(1- (-) (lui
g gils)
1324
(1 82)g ggli; + 8 /// 81,(1 /Bzu + (1 32)gt oglg + v [ +5,))
d(1
82) Ba
83
where the last incquality follows from 0 < [u s B% and 2 Ig/ogl, S B2. Then the desired incquality holds.
of o of and |l o I B2, it holds | B2, For all t 2 1, it holds (iii) By ut = 1+ Ei
1
d
1
1g g 0
[1
( [ 2 g! og 1b)
2
7B:
g
8
6l /
1
+ 6+
+ +))
32

---

Published in Transactions on Machine Learning Research (04/2026)
where the last inequality follows from 0 2 [u:-1)
I B and 0 S
g
" S Bo Then
TK
t=0
π
Vui u
2-2
+
v
u
+
7
+
v u
8
2dB3
00
83
The proof is then completed.
Now, we prove Theorem 3.2 with its complete statement given as follows.
Theorem A.2 Suppose that Assumplions hold, and Q is n-compression operator. Let 5 O(1) be universal positive constant, C be the constant defined in [11 and a, > 0 satisfy
40 / n BB + +5)
5 48Lv B + 8
.75 (1 p) 27²)
a
$ min
(85)
VTK
100
where 0 = O(1). Then, the following statements hold.
(i) For AMSGrad in
77
il holds
TK
E
|V )|= X ||2
TK
0
- -f LB B + LB2)
+ T L (Boc + L+
1
dB
(86)
VnTK
nKC
L3 (1 + Bxx) + T
L 22 B
nKC
(1 (1+B2))
TK
(ii) For Adam in 78)
with B2 E
1] it holds VTK+1*
vTK
TK
E
|V 2 113
TK
t=
(f(x²) - LB3 LBZ B² + LB²) + T - dB (Boc + L L+1)
(87)
VnTK
nKC
(1+
B2 B (1 (1+
nKC
T
TK
(iii) For AdaGrad 79 the relation
87 holds as well.
Proof. From Lemma A.7 it holds that u I By, Vt, Vi with B, B2 for all the three optimizers in
79
33

---

Published in Transactions on Machine Learning Research (04/2026)
Dividing both sides of 71 by
aTK 0VnTK and rearranging terms, have
Iv B +6
TK-1
a21223/
(x)) 1- S
0VnTK (f(a9)- f*) -
TK
20(1 -
-1)3
4 B à
TK-1 Σ
1 n
1
1
(Av +8+ +8+ E
TK(1 - B1)2
π
1-2
+
1-1
+
24
a L²nK"
B
+
6aL
4aLv B +6 B 4V
B +0
nd
n
8
2V8
8
BB2 B& + 1
4v B +0
81)-
2V8
40 40
1 +8
Adding obtain
21
to the above inequality and replacing a by
S in the resulting inoquality, we
VTK
1 TK-1
16 +
E
VVf
1X 12+
S
OVnTK (f(x") +
TK
72
2T - B1)²
B3
BB2 5
BB +8 +0+
(88)
TK(1
-31)2
2-2
+8
V
f-1
8
160
24
64I2nK
(B + +0)
BB2
B2
+
6L
T(B²
+5)-
B²
VnTK
TK
8
2/8
8
64n0" (B% + 8)
L2
B +
1
16nK (B²
+ +0)
TK
8(1 3-
8
2V8
TK
We now substitute the results in Lemma A.7 to the above inequality. For AMSGrad, we have
TK-
1
d
E
Old).
t=0
n
2
V
|
1
+ 5
8
(ii) For Adam, we know from B2
VTK
that
VTK+i
TK- 1
TKd(1 B2
E
OdB
83
t=0
u
+
u
+
(iii) For Adagrad, we have
2
TK 1
2dB
<
Odd
u
2
Vu
g3
06
Therefore, obtain the desired results.
34

---

Published in Transactions on Machine Learning Research (04/2026)
B The matrix-form adaptive gradient updates
It should be noted that our thcoretical results extend to matrix-form adaptive gradient updates, where the d-dimension real-valued functions } are replace by some d d dimension real-valued functions {rr). Under this formulation, we update the groond-momentnn matrices U as U = rz(g, .......g.) and the model parameter by 2; 1+1 = x a (U?-) + +6) 6) - m
Our theoretical results apply to the matrix-form adaptive gradient method, provided that max,, ((s f1.2.... n) and t z 0, and that the summation Ee E (U d) (U: + +5) remains bounded.
is uniformly bounded for all
A notable example of this framework is the matrix-form AdaGrad method, where U is updated as
U
9:9; for each agent = I....,n.
For this choice of {U:), is not difficult to show that [[st)rls B for all r E {1,2....,dd and E (1,2....,dl d), and
TK-1 a
+ ( (U?--+ + 8)-2 + 2d B
Σ
(89)
83
Therefore, theoretical results extend naturally to the matrix-fort AdaGrad method.
C Examples of nccoppression operators
In this section, provide few concrete examples of compression operators that are r-compression opera- tors. More examples can be found in (Chen et al.) 2023a: Koloskova et al. 2019).
Example C.1 QSGD (Alistarh et al. 2017) compresses Rd by Qaga(x) sign(>))[<=])
s ||||| + where
is uniformly distributed on [0, 1| s is a parameler about compression level. Then Q(x) = ---- 2 with
7 (1+ min dd/s, Vdjs}) is an n-compression operator with n = 1
Example C.2 (xparse(x) Stich et al. 2018 randomly selects out of d coordinates from x, or the k coordinates with the largest values in magnilude from æ. Then 2sparse(ar) is an n-compression operator with
Example C.3 Q goossip @gossip (x)) with probubilily p. Then (x) x)) an n-compression operator with vj = p.
Koloskova et al. 2019 sets @zossip(x)) = x with probability p E [0, 1] and
Additional Numerical Experiments
We include a suite of additional numerical results this section. These results were omitted from the main body of the paper for space considerations. We expand on the results presented in the main body in several ways. Figure includes all omitted FashionMNIST results using the same experiment setup shown in Figures and for CIFAR-10 and siny-shakespeare. However, note that omit CDProxSGT opti- mizer comparisons in this supplement, as its results were not competitive in most experiments, as observed in Figure SQuARM-SGD otherwise represents methods with non-adaptive updates.
We also include additional experiments using expanded parameter settings. Figure repeeass the optimizer comparisons demonstrated in Figure [ with wider variety of local update counts. Figure g repeats the experiments comparing the effect of local update counts on communication rounds shown in Figure 2 with Adam's update, while comparing AdaGrad's and AMSGrad's adaptive updates well. Figure compares
35

---

Published in Transactions on Machine Learning Research (04/2026)
varying values of Top-k compression for all three optimizer variants. We do not show training loss consensus error with these figures for space and clarity, while noting that theses plots would otherwise be consistent with those shown in Figures 2 and 6
D.1 FashionNMIST Results
In Figure we plot the results for the same experiments on PashionNMIST as performed and displayed for CIFAR-10 and tiny-shakespeare compression of 30% used. The primary observations in this figure are consistent as with the prior results. We generally observe little to degradation of accuracy and loss performance, even when including local updates and Top-k compression. In particular, FashionMNIST suffers less in terms of quality impacts when including compression and minimizing communication relative the two other test datascts.
Figures 1 the main body of the paper. For these results, Top-k
Deettbunee--SGradd
Gre
10--
DADAM
Consenouu Entr
Traning Lo53
60
50
Distr ButesAMSGrd
Destr ButesMawirad
Distr butosAsam
10-1
DADAM
10
5.
10.0
12.5
15.0
12.5
12.5
12.5
15.0
17.
Epochs
Epechs
Epochs
(a) FashionMNIST optimizer comparison.
K=1
2.0
Corsonaud Entrr
Taining to5
1.5
K+
K-30
50
WWW
1.0
10-4
3.5
5000
10000
15:00
2500g
S000
10n00
Londs
20100
25rdi
J05000
5000
1000
200000
Comuniation Rounde
Communication Rounds
Commnniaton Rounds
(b) MashionMNIST reduction in comminication rounds.
10
6-50. 30%
C-10, kppkeeoo
Eip-t-hinee
Consengus Entr
10-1
Trakiing Loss
60
MWM VMMAN
10-2
50
2.0
M WWWm
10-
0.5
10
10"a
10-3
10-2
10-1
107
10-4
10-2
10'2
10P
10
10--
10:2
16"
Scaled Clmmunication Valume
Scaled Communicatio Volume
Scaled Communication Velume
(c) FashionMNIST reduction communication volume.
Figure 6: Convergence performance for FashionMNIST: Plotied above are the training loss and test accuracy of FashionMNIST.. The top row compares optimizer performance with Top-k 30% compression and a local update count = 20. The middle row demonstrates the reduction in communication rounds based on the number of local updates with Top-k compression of 30%. The bottom row compares the total communication volume scaled relative the uncompressed baseline with no local updates.
36

---

Published in Transactions on Machine Learning Research (04/2026)
D.2 Additional Optimizer Comparisons
In the main body of the paper, Figures la and 16 compare oplimizer performance for a fixed value of K = 20. Figure repeats these experiments with additional values of K = 2, 5, 10, 50, still using 4 agents. Shown are test accuracy/validatton /validation loss with compression values of 30%, 40%, and 50% for FashionMNISTT, CIFAR- 10, and tiny-shakespeare, respectively. We overall again observe consistent results as with Figure ! where Adam outperforms other optimizers on MNSSi and tiny-shakespeare and AdaGrad outperforms other optimizers on CIFAR-10. Likewise, SQuARM-SGD consistently lacks in generalization performance on the GPT language model with the tin-ssaaesppeare dataset.
D.3 Additional Number of Local Updates Comparisons
In Figure the experiments using Adam in Figures 2a 2b and ob are repeated with the AdaGrad and AMSGrad adaptive updates. For each optimizer and benchmark dataset, we plot the accuracy validation loss that results with local updates of K 2, 5, 10, 20, 50. Again, use Top-k compression values of 30%, 40%, and 50% for FashionMNIST,, CIFAR-10, and tiny-shakespeare. Likewise, we note that accuracy performance is minimally affected by the number of local updates used in these tests.
D.4 Additional Top-k Compression Comparisons
We fixed Top-k compression values for each dataset for the bulk of the experiments demonstrated thus far, to avoid a parametric explosion in the number of results presented. Figure g demonstrates that different choice of Top-k compression in the same order of what was used has minimal impact on the overall optimizer performance and resultant appearance in plots. In Figure 9 we fix = 20 and vary Top-k to 30%, 40%, 50% and 60% for each of our optimizers (Adam, AdaGrad, AMSGrad) and dataset (FashionMNIST,, CIFAR-10, R-, kiny-shakepeeare.. We observe reduction in total used communication volume scaling linearly with Top-k percentages, expected, while accuracy and validation loss relatively consistent across all tests.
D.5 Additional Scaling Comparisons
In Figure I0 we present scaling results for AdaGrad and AMSGrad, as was presented for Adam in Figure 1 in the main body of this paper. We fix K 20 and run with 4, 9, and 16 agents across both ring and 2D grid topologies. As with Adam, note minimal impact to the maximum achieved test accuracy after 250 epochs, demonstrating near-linear speedup for our approach.
37

---

Published in Transactions on Machine Learning Research (04/2026)
DistRuttcAMGGnd
Distr Buteokdnörad
Bittrbbteckden
DADAM
00
60
3.0
DistriudteAMSGrd
40
30
DistibutsAaaGdd
2.5
DiEtIbutSAAcmm
GictibutesAdamm
20
DADAM
20-
DADAM
2.0
SQLARMAARD
55D
10
17.5
160
100
800
500
600
Epochs
Epochs
Epochs
K 2: MNshionMNIST, CIFAR-10, and Shakespeare optimizer comparison.
90
RutedAMSGrod
80
kutadAdaGrad
Distr aidributtoddamm
DAGAM
60
60
SQUARM SGD
Valicaa
40
30
Distributesddam
Diemibbteokdam
20
IADDM
2.0
SOLARM 5G0
SQUARN-SOD
100
100
209
500
600
Epochs
Epochs
Spochs
(b) K MNSSi CIFAR-10, 10, and Shakespeare optimizer comparison.
90
90
ButodAMSGrdd
80-
Distr tutributeodomm
DADAM
60
SQUAIN-SGD
Validaiion
40
DithieutodAMGrrdd
2.5
30
Distibutookdam
20
DADAM
2.9
SQLARM--GD
SQUARN-SGD
100
100
400
500
600
Epochs
Epochs
Epochs
(c) K 10: FaSHionMNIST, CIFAR-10, and Shakespeare optimizer comparison.
4,5
70
EutedAdam
Teut Accurccy
3.5
DADAM
SQUARN SGD
40
Distiba
40
DeemeuteeAMGGrrd
DistibutssAaam
Giitibulechdam
20
DADAM
SQLARM-SGO
SQUARN SGD
30.0
12.3
150
100
250
100
600
Epochs
Epochs
Spochs
(d) K 50: FashionMNIST, CIFAR-10, and Shakespeare optimizer comparison.
Figure 7: Additional optimizer comparisons: Plotted above are accuracy (for FashionMNIST and CIFAR-10) and validation loss (for tiny-shakespeare) for the tested optimizers across a range of local update counts K. For each subplot, MNIS is the right, CIFAR-10 in the middle, and tiny-shakespeare is the right.
38

---

Published in Transactions on Machine Learning Research (04/2026)
90
90
80
80
70
70
Test Accuracy
Test Accuracy
60
50
K=1
K=1
K=2
40
K=2
30
K=5
30
K=5
K=10
K=10
20
K=20
20
K=20
10
K-50
10
K=50
5000
10000
15000
20000
25000
30000
35000
5000
10000
15000
20000
25000
30000
35000
Communication Rounds
Communication Rounds
(a) AdaGrad (left) and AMSGrad (right) on FashionMNIST with Top-k compression of 30%
80
70
Test Accuracy
Accuracy
60
50
40
30
30
20
20
10
10
5000
10000
15000
20000
25000
30000
5000
10000
15000
20000
25000
30000
Communication Rounds
Communication Rounds
(b) AdaGrad (left) and AMSGrad (right)
CIFAR-10 with Top-k compression of 10%.
K=1
K=1
K=2
K=2
K=5
K=5
3.5
Training Loss
K=10
K=20
Validation Loss
3.5
K=10
K=20
3.0
K=50
K=50
2.5
2.0
2.5
1.5
1.0
50000
100000
150000
200000
50000
100000
150000
200000
Communication Rounds
Communication Rounds
(c) AdaGrad (left) and AMSGrad (right) on tiny-shakespeare with Top-k compression 50%.
Figure 8: Additional number of local updates comparisons: Plotted above are accuracy (for Fash- ionMNIST and CIFAR-10) and validation loss (for tiny-shakespcare) using AdaGrad (left) and AMSGrad (right), comparing across a range of local update values K.
39

---

Published in Transactions on Machine Learning Research (04/2026)
80
Test Accurcy
60
50
49
40
K-20, Top k-
Km20.00p K- 30%
K-20, Top k- 30%
K-20, 4-4
K-20.70p
K-20, 40%
20
20
K-20, Top 6060
10
2-p0. k-5
K-20, Top 6060%
10
10:
Scaled Communication Volume
Scaled Commurication Velume
Scaled Communication Volume
(a) Adam (left), AdaGrad (middle), and AMSGrad (right) on FashionMNIST with varying Top-k compression.
86
70
70-
70-
50
50
40
40
30
K-20, Top-k 35%
400%
30
K- k-20.1pp---00 30%
k-200..p---00%
30
K-20, Top-c-
Top-
50%%
20
20
50%
K-20, Top K065
2-20. k-50%
K-20, Top 60%0%
10
10
10-
10
10-4
10-1
10
10-1
10-
30
10-1
10-1
Scalld Communication Volume
Scaled Communication Volume
Scaled Communication Valume
(b) Adam (left), AdaGrad (middle), and AMSGrad (right) on CIFAR- 10 with varying Top-k compression.
K- Thp 3-00%
K-26, Top 3-30%
K-20.0p k- 30%
Ka20 Thp-k
4.0
K-20 k0
4.0
K= Top
K-20 Thp-k-50%
Top-k-n K-20, k-60%
K-20 sep-
3.5
K-20 6-60%
3.5
3.5
K-20.70 k--0%
Weining
3.0
Vallconton
3.0
3.0
2.5
2.5
2.0
1.5
2.0
Scaled Communication Volume
Scaled Commurication Valume
Scaled Communication Volume
(c) Adam (left), AdaGrad (middle), and AMSGrad (right) on shakespaaee with varying Top-k compression.
Additional Top-k compression comparisons: Plotted above are accuracy (for FashionM- NIST and CIFAR-10) and validation loss (for Liny-shakespeare) using Adam (left), AdaGrad (middle), and AMSGrad (right), comparing across range of Top-k compression values.
Figure 9:
40

---

Published in Transactions on Machine Learning Research (04/2026)
Dist 320kkdaoroddt-33./
Distr utokkiarroddn---ee Ring
Consencus Enror
Thiring Loss
2.0
30
Asw.ssnnnN-- Ang
Distri 1it.--sssnnnddd--50
10-1
0.0
250
100
250
Epocto
Epocts
Spuchs
(a) AdaGrad agent scaling with ring topology.
50
10
Distrib weedd 9.0rd
Dierib wwweeeeeonneeeeeeee Grd
Consenous Error
Thinin L295
1.5
60
1.0
Otemnb Dterb..tekdsennddi11111..
250
Epocho
Epecto
Ep och
(b) AdaGrad agent scaling with grid topology.
PorribuueesMMMGGaaddii R,iing
2.0
Trainngtos
3.5
Conoenous Enror
10%
Lo-1
0.5
Distribute dettuuttddessnnd--155m
0.0
100
250
Epocts
Epochs
Spocha
(c) AMSGrad agent scaling with ring topology.
90
Distr DdtrisuessAMGGrr N-4.6r
Detr Ser
Detr MrtieeeesseM 16.Grd
Conseno Enor Entr
Thiking Lo5
60
10"
3.0
30
Listr ibut 15
100
Epocts
Epocts
Epachs
(d) AMSGrad agent scaling with grid topology.
Figure 10: Additional Scaling Results: Plotted above are the training loss, test accuracy, and consensus error of CIFAR-10 10 when scaling to 4, 9, and 16 agents using ring topology and 2D grid topology with the AdaGrad and AMSGrad optimizers. All experiments were run with K = 20 local updates communication round.
41


Reviews:
[{"review_id": "aT33moqORr", "reviewer": "Reviewer_KdrP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposed a novel approach that combines adaptive optimization algorithms with compressed communication for decentralized distributed learning. Theoretical guarantees and empirical results are provided to justify the effectiveness of this approach.\n\nStrength:\n- The paper is well-presented.\n- Theoretical results look sound (attention: I did not check all the proofs, but for some that I did read, there is no evident error).\n- Empirical experiments are chosen and presented properly.  \n\nWeakness:\n- Compared to previous works, the contribution seems incremental. Especially, compared to SQuARM-SGD, the only difference is the local update: one uses Momentum-based methods, while the other uses Adaptive methods.\n- Some terminologies are inconsistent: e.g., the letter A in LoDAaC, sometimes refers to Adam-type (in the abstract), sometimes refers to adaptive (the second sentence in 1.1. Contributions).\n- It is always better to compare algorithms with multiple baselines if possible.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Theoretical guarantees are well-supported by adequate mathematical arguments. While the baselines in the paper are quite limited (only SQuARM-SGD and DADAM), empirical results do support their claims.", "audience_yn": "Yes", "audience_explanation": "The paper might interest the distributed learning community.", "requested_changes": "If the paper is accepted, I recommend that the authors should:\n- Publicly provide their implementation of experiments for reproducible purposes.\n- I think it is better to replace Adam-type by adaptive in the title. Adam is an instance of adaptive optimization methods.\n- Table 2: I am not sure if Adagrad is defined as in the paper. Normally, the running sum is not averaged.\n- It is interesting to report the consensus error in the experiments. I am curious whether all the agents in the experiments converge to the same solution or not.", "broader_impact_concerns": "Not available", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dUBuRma8zS", "reviewer": "Reviewer_eRb6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers a decentralized optimization framework for distributed learning coupled with multiple local updates, compressed communication and adaptive gradient methods, which is close to practical applications and significant for efficient training of large models. Both theoretical results and numerical experiments are provided to validate the advantages of their proposed algorithm LoDAdaC.\n\n**Strengths:**\n- This paper states the motivation behind decentralized optimization and the concepts of multiple local updates and compressed communication in a clear and well-structured manner.\n- The communication cost $O(\\frac{1-\\eta}{K\\epsilon^2})$ derived for LoDAdaC shows the advantage of the combination and the explicit quantitative control by the parameters of MLC and CC.\n- The theoretical results are validated by intensive experiments with LeNet, Resnet, nanoGPT on image or text datasets to show the effects of communications and local updates.\n\n**Weaknesses:**\n- Some points in the assumptions and definitions require further clarification and detail.\n  - The optimization condition in expression (2): $X = XW$. Is this a typo? It would be beneficial to specify the dimensions of $W$ at the outset for clarity.\n  - In Assumption 2, although authors have mentioned construction exist to meet assumption 2, more intuitive insights on the roles of these conditions in the proof should be demonstrated here. The same idea also applies to Definition 1.2.\n  - For Definition 1.1, the condition $\\mathbb{E}_{\\mathcal{Q}} [\\||x-\\mathcal{Q}\\||^2] \\leq \\eta^2 ||x||^2$ takes the expectation over the set of compression operators, however there's no description about the distribution of $\\mathcal{Q}$.\n  - Add more captions for (e)(f) in Figure 1: I think Top-k selection is related with compression ratio $\\eta$. If it is, please make the connection more clearly. I don't quite get the idea behind (e) and (f).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The theoretical results are supported by detailed proofs and validated by intensive experiments.", "audience_yn": "Yes", "audience_explanation": "The paper explores a decentralized optimization framework that closely aligns with practical applications, particularly in the context of training strategies for language models. The authors also present experiments using small-scale nanoGPT models trained on the Tiny Shakespeare dataset. This study makes contributions to both optimization theory and LLM communities.", "requested_changes": "See in Weakness.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sy4HmuOWvb", "reviewer": "Reviewer_r63W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces LoDAdaC, a decentralized learning framework that combines Adam-style adaptive optimization, multiple local updates, and compressed communication. By bringing these three ideas together, previously studied mostly in isolation, the method aims to reduce communication cost while keeping fast convergence. The framework supports a broad set of adaptive optimizers (e.g., AdaGrad, Adam, AMSGrad) and standard compressors, including biased ones, in a decentralized setting.\n\nOn the theory side, the paper gives a convergence analysis for general nonconvex problems with biased compressors, obtaining an iteration complexity of $O(1/\\epsilon^4)$ with explicit dependence on the number of local steps $K$ and the compression ratio $\\eta$. Empirically, LoDAdaC performs well on image tasks and Transformer-based language modeling, achieving up to 99% reduction in communication while outperforming momentum-based baselines such as SQuARM-SGD and matching or improving over DADAM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide a mix of theoretical and empirical evidence to support their claims.\n\n1, Theoretical Convergence: They provide a rigorous convergence analysis showing that LoDAdaC achieves an asymptotic convergence rate comparable to state-of-the-art non-adaptive methods ($O(1/\\epsilon^4)$), while explicating the dependence on the number of local steps ($K$) and compression ratio ($\\eta$).\n\n2, Empirical Validation: Experiments on CIFAR-10, Fashion-MNIST, and Tiny Shakespeare (using a small-scale GPT model) clearly show that the proposed method converges in fewer communication rounds and with less data transfer than baselines like SQuARM-SGD and DADAM.\n\n3, Ablation Studies: The paper includes parameter studies on local step counts and compression rates, supporting the claim that the method is robust across different configurations. While the bounded-gradient assumption is strong and simplifies recent adaptive optimization theory, it is still common in decentralized analyses. The empirical results appear reproducible and distinct enough to validate the core claims of efficiency.", "audience_yn": "Yes", "audience_explanation": "This paper targets core challenges in distributed learning: reducing communication overhead while maintaining fast convergence. As research increasingly focuses on training Transformer-based models over bandwidth-constrained networks, frameworks that successfully unify gradient compression with adaptive optimization are highly relevant. Furthermore, the extension of Adam-type updates to the decentralized setting, supported by rigorous complexity analysis, addresses both a theoretical gap and a practical need for researchers working with architectures where adaptive methods are required for stability.", "requested_changes": "Major Concerns:\n\n1, Could you clarify why the NanoGPT experiment ($\\approx$ 10.7M parameters on Tiny Shakespeare) is described as “large language model training”? This model is several orders of magnitude smaller than what is typically meant by “LLM” today (usually ≥ billions of parameters), so the current wording risks overstating the scale. I recommend replacing “large language model(s)” / “LLM(s)” in the abstract (“e.g., in large language model training”), Introduction (LLMs as the “workhorse” optimizer), and Section 4 with more accurate terms such as “Transformer-based language models” or “small-scale GPT-style language models”, and explicitly acknowledging that the experiments are conducted on a small model.\n\n2, In Definition 1.1, you introduce an η-compression operator and give examples such as Top-$k$ sparsification, which is biased. Could you clarify whether the convergence analysis relies only on the contractive-error property in Definition 1.1, or whether any extra conditions (e.g., unbiasedness/error-feedback behavior) are implicitly needed for such biased compressors? A brief explicit statement here would make the compressor assumptions and the scope of the theory much clearer.\n\n3, In Section 3 you introduce Assumption 3 (bounded gradients) and describe it as “standard” for adaptive SGMs. However, while this assumption is indeed common, it is quite strong and often violated in practice (e.g., heavy-tailed gradients in deep Transformers), and recent analyses of Adam-type methods already work under weaker conditions such as generalized smoothness / bounded variance instead of bounded gradients. Could you soften the wording around “standard” and add a brief discussion that (i) acknowledges Assumption 3 as a strong, often unrealistic assumption, (ii) mentions practical remedies such as gradient clipping, and (iii) situates your analysis relative to these more recent Adam results under weaker assumptions?\n\n4, Could you clarify how you expect LoDAdaC to behave under non-IID (heterogeneous) data, given that all current experiments use homogeneously distributed training data? Real decentralized/federated deployments typically face client drift due to heterogeneity, especially when combining local steps and adaptive updates. It would strengthen the empirical section to include at least one non-IID experiment (e.g., a Dirichlet or label-skewed partition on CIFAR-10); if that is not feasible, I recommend adding a short statement in the problem setup or conclusion acknowledging that the current validation is limited to homogeneous data and briefly discussing how LoDAdaC might interact with client drift and possible remedies.\n\nMinor Concerns:\n\n1, Section 1.1, “LoDAdaC equipped with adaptive gradient updates significantly outperform the baseline…” should be “outperforms”.\n\n2, In the mid of page 5, “siginificantly” should be “significantly”.\n\n3, In the mid of page 11, “Adamini” should be consistently written as “Adam-Mini”.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)