On The Surprising Effectiveness of a Single Global Merging in Decentralized Learning
Authors: Tongtian Zhu
Tianyu Zhang
Mingze Wang
Zhanpeng Zhou
Can Wang
Venue: ICLR 2026 Oral (2026)
Keywords: Decentralized Learning
Model Merging
Training Dynamics
Distributed Training
Primary area: optimization
TL;DR: We discover and theoretically explain why and when a single global parameter merging in decentralized learning can recover the performance of federated learning, even in highly heterogeneous and communication-constrained environments.

Abstract: Decentralized learning provides a scalable alternative to parameter-server-based training, yet its performance is often hindered by limited peer-to-peer communication. 
In this paper, we study how communication should be scheduled over time, including determining when and how frequently devices synchronize. 
Counterintuitive empirical results show that concentrating communication budgets in the later stages of decentralized training remarkably improves global test performance.
Surprisingly, we uncover that fully connected communication at the final step, implemented by a single global merging, can significantly improve the performance of decentralized learning under high data heterogeneity. 
Our theoretical contributions, which explain these phenomena, are the first to establish that the globally merged model of decentralized SGD can match the convergence rate of parallel SGD.
Technically, we reinterpret part of the discrepancy among local models, which were previously considered as detrimental noise, as constructive components essential for matching this rate. 
This work provides evidence that  decentralized learning is able to generalize under high data heterogeneity and limited communication, while offering broad new avenues for model merging research.

Published as a conference paper at ICLR 2026
ON THE SURPRISING EFFECTIVENESS OF A SINGLE GLOBAL MERGING IN DECENTRALIZED LEARNING
Tongtian Zhu", Tianyu Zhang Mingze Wang', Zhanpeng Zhou Can Wang!
'Zhejiang University Mila, Quebec Al Institute Universitéé de Montréal *Peking University SShanghai Jiao Tong University
(raiden, wcang) edu. en Lingzewang@stur edu. on
@zju.edu..
t tianyu. zhang@mimmmmmm la. quebec zzp10128s jtu wtwpssttuudu...o. edu.er en
ABSTRACT
Decentralized learning provides scalable alternative to parameter-serrve-bbseld training, yet its performance is often hindered by limited peer-to-peer communica- tion. In this paper, we study how communication should be scheduled over time, including determining when and how frequently devices synchronize. Counter- intuitive empirical results show that concentrating communication budgets in the later stages of decentralized training remarkably improves global test performance. Surprisingly, we uncover that fully connected communication at the final step, im- plemented by single global merging, can significantly improve the performance of decentralized learning under high data heterogeneity. Our theoretical contributions, which explain these phenomena, model of decentralized SGD can match the convergence rate parallel SGD. Technically, reinterpret part of the discrepancy among local models, which were previously considered as detrimental noise, as constructive components essential for matching this rate. This work provides evidence that decentralized learning is able to generalize under high data heterogeneity and limited communication, while offering broad new avenues for model merging research. Blog post and code are available at & Grokking in Decentralized Learning and ? Code, respectively.
the first to establish that the globally merged
INTRODUCTION
Decentralized learning offers a promising approach to crowdsource computational workloads across geographically distributed compute (Yuan et al., 2022; Borzunov et al., 2023b; Jaghouar et al., 2024). defining characteristic this setting is the reliance on peer-to-peer communication during training, involving the peer-level exchange of model parameters gradients. However, such communication is often constrained practice due to limited bandwidth between geographically distant nodes, making significantly degrade the performance of decentralized
it scarce resource. These constraints learning, both theoretically and empirically (Lian et al., 2017; Koloskova et al., 2020; Vogels et al., 2021). As result, efficiently allocating limited communication resources becomes fundamental challenge in decentralized learning, especially in heterogeneous environments where varying local data distributions intensify communication demands (Martínez Beltrán al., 2023).
To date, most efforts addressing this challenge have focused on optimizing communication allocation the spatial level, particularly through the design of communication graphs (Ying al., 2021: Li et al.. 2022b: Takezawa et al., 2023; Kharrat et al., 2024). In contrast, the temporal allocation of communication, i.c., deciding when and how frequently agents synchronize with others, remains significant yet underexplored direction for improving decentralized learning. Although temporal communication allocation has been studied in federated learning (FL.) (Tang al., 2020), this problem remains largely untouched in the fully decentralized setting, which is fundamenially different due to the lack of a central server for global aggregation (see discussions in Section and Remark 1).
"qqual contribution. 'Corresponding author.

---

Published as a conference paper at ICLR 2026
Avenue Chobbal Test Accuracy
Uedersted Trateing Desesratina Tointy
Avergge Gliskal Toxt Accurscy
6--
Tedeawed Traiiing Braeenlted Traning
R6
02-
6--
03-
Communication Round
Communication Round
(a) CLIP ViT-B/32
(b) ResNet- 18 (w/o pretraining)
(c) Landscape before final merging
Central Server
Local Training
Local Training
Communication
Local Training
Agents
Agents
Communication
Agents
Low Communication
Fully-connected Communication
No Communication
Federated Training
Decentralized Training
Local Training
(d) comparative illustration federated, decentralized, and local training.
Figure 1: (a, b) Global test accuracy (see Definition 1) of CLIP ViT-B/32 (a) and ResNet-18 18 (b) on Tiny ImageNet under 32-agent non-IID setup (Dirichlet a 0.1), where decentralized SGD communicates with one random peer per round with probability 0.2 and performs final global merge. (e) Loss landscape before the final merge for decentralized (d) Illustration of federated, decentralized, and local training. Experimental details are provided in Appendix C.I.
Question: How allocate communication budget in decentralized learning over temporal levels?
To answer this question, we design a series of experiments that allocate communication budgets across different time windows during training (see Figure 2). Specifically, we divide the training process into consecutive windows, each with of fixed number of communication rounds. We assign higher communication budgets selected windows using global synchronization via A. Alleeduce (Sergeev Del Balso, 2018), while keeping communication low otherwise by infrequent synchronization with random peer agents." This reveals how the temporal communication allocation affects performance under constrained budgets. We observe that allocating higher communication budgets toward the later stages of training consistently leads to improved final test performance (see Definition 1). More surprisingly, we observe the remarkable effect of single round of fully-connecee communication.
single global merging of decentralized models, under severely con- strained communication and high data heterogeneity, can significantly improve global test performance.
Surprising Phenomenon:
Our Contributions are summarized below.
Empirical Observations. (1): We highlight the critical role of single global merging in de- centralized training, showing that
achieve performance close federated learning, under severe communication constraints and data heterogeneity (see Figure la, Figure 1b). (2): We observe that limited but non-zero communication preserves challenging pross-iniiaaliztion.. cross-distin dinssdiistiittion "mergeability" of local models throughout training (see Definition 2, Figure Ic,
Agents refer participants in decentralized leaming. "Communication" and "synchromization used interchangeably. Fully connecece communication refers to glohal synchronization via A1 Reduce. In this paper, fully-connected communication is realized through parameter averaging over the models on all agents, namely global merging. visual clarity.
"e 16 agents for landscape visualization
2

---

Published as a conference paper at ICLR 2026
and the blue curve in Figure 2c), which does not hold under complete local training (green curve in Figure la, Figure 1b). These findings remain consistent across datasets, heterogeneity levels, model architectures, and communication topologies (see Appendix C.3), and provide a first systematic study of global merging in decentralized learning.
Theoretical Contributions. We investigate the underlying mechanism that enables the mergeability of local models in decentralized learning. Specifically, we provide the first convergence analysis showing that the globally merged model of decentralized SGD can match the rate of parallel SGD (Theorem and Proposition 2). Furthermore, we offer a theoretical explanation for why limited but nonzero communication can ensure mergeability, and why communication should be concentrated the later stages of training (see Proposition 3).
We anticipate that this work will pave the way for principled decentralized training algorithms capable of generalizing under communication constraints and data heterogeneity, while also advancing model merging research (see discussions in Section 6 and insights in the Q&A Section, Appendix A).
2 RELATED WORK
Temporal Communication Allocation in Parallel, Federated, and Decentralized Learning.
Communication allocation is well-studied in both data-centric parallel learning (Li et al., 2014), and Federated Learning (FL) (McMahan et al., 2017). In parallel learning settings, Gu et al. (2024) proposed novel strategy for scheduling local steps by analyzing the implicit bias of Local SGD (Gu et al., 2023b). FL extends this basee paradigm to handle not identically and independently distributed (non-IID) data, but it critically retains a global model. This reliance on a global model has shaped broad consensus in the FL literature: frequent, etrly-stage communication is considered essential for aligning local models (Wang et al., 2019; Tang et al., 2020).
In contrast, work addresses fully decentralized learning, fundamentally different setting that lacks a central server. Instead of optimizing generic global model, the goal is to make local models generalize to the global distribution. Despite extensive work focusing on communication allocation at the spatial level in decentralized learning (e.g., designing communication topologies) (Ying et al., 2021; Li et al.. 2022b; Takezawa al., 2023; Kharrat et al., 2024). few studies examine temporal allocation: a pioneering IID study (Kong et al., 2021) showed that stronger early alignment to the global average can modestly improve test performance. These findings do not directly transfer to non-IID settings, where C(.) = Lx(.) no longer holds (see Equation (1) and Definition C.2), so they mainly characterize local rather than global generalization (see Definition 1). Due to space
constraints, we refer readers to Appendix B.2 and Appendix B.3 for related work on the implicit bias of decentralized learning, and on the topic of model merging.
NOTATIONS AND PRELIMINARIES
3.1 NON-IID DECENTRALIZED LEARNING
Decentralized learning formalizes distributed learning as optimization problem over connected graph G = (V/ E), where V contains m agents and & denotes the communication links. Each agent k T V samples data from local distribution Dj and maintains local model 0₂ Rd. The objective is to learn a consensus model that minimizes the global population risk (Koloskova et al., 2020):
((0) A 1 Σ EEL~D. [(): Ex)
min BER
KEV
where C(::EE)(((EE)) & (B) denotes the local population risk of 6 on unseen instance & Di-
In practice, the optimization of Equation (1) is performed under the empirical risk minimization framework, leveraging m local datasets S A Um= St. where S (SA.1,.... }..c}} denotes the dataset of agent l sampled from Dr. The resulting optimization problem is given by:
min GER
Ls(l)
L(0; E(0:EM.c)
(2)
REV
3

---

Published as a conference paper at ICLR 2026
To solve the optimization problem in Equation (2), decentralized algorithms minimize the global empirical risk with only local computations and peer-to-pee communication (Tsitsiklis et al., 1986; Nedic & Ozdaglar, 2009). The communication graph is governed by weighted adjacency matrix W E [0, 1| _, sampled from a distribution W(),, where each entry w z reflects the influ- ence of agent 1 on agent k Decentralized learning algorithms operate by alternating between local updates and model aggregation through communication with neighbors, as outlined in Algorithm
Algorithm 1 Decentralized Learning
input Initialize values 0 I&d on each agent V, number of steps T, mixing matrix W
1: in parallel all agents k V, for ....., 1
2: Sample training data < from Da. llt+1) +- Optimizer(0) -5k)
P Local update
3: Send
0 to out-neighbor(s) and receive (0(") JEN..()) from in-neighbor(s) Communication
4: Sample mixing matrix Wt() WW(t), g(++l) EieNa(K) W()0(0)
GGossip averaging
5: end parallel for
Practical Evaluation Metries. In decentralized learning, data heterogeneity and limited training time often prevent full consensus model We therefore propose to adopt average global test accurucy, proxy for global population risk, as the primary metric to quantify how well local models generalize to the global data distribution.
Definition (Average Global Test Accuracy). The average accuracy of agents V is defined as:
Ac [REV) = Σ Ac) where Acc(-)
Σ
BE Aco(;;E)
KEV
LEV
Averuge Accuracy across agents
Test accurocy on the global distribution
Remark 1 (Metric Justification). This metric is specifically designed to address core question in fully decentralized learning: how well do local models (0(.) Jkey, trained with limited seer-tt-peer synchronization, generalize to the global data distribution DD This metric offers a more realistic evaluation for decentralized settings without a global model. See discussions in Appendix C.2.
3.2 MERGEABILITY
Definition 2 (Mergeability under Global Population Risk). A set of local models (0x is globally mergeable if there exist combination weights {WE]KEv E [0, 1] such that:
C
Σ wkRR LEV
Σ KEV
we w(kk)
where C(-) denotes the global population risk.
Definition 2 formalizes the intuition that a linearly interpolated model performs no worse than the original local models. This Definition is inherently non-trivial due to the non-convexity of L.
EMPIRICAL OBSERVATIONS
4.1 INCREASING IMPACT OF COMMUNICATION IN THE LATER STAGES OF TRAINING
To investigate potential solutions of communication scheduling, we explore direct strategy: Con- centrate communication small subset of communication rounds. To this end, we divide the training process into consecutive windows, each consisting of a fixed length of communication rounds.
Our framework incorporates randomized decentralized learning setting where the weighted adjacency matrix (1() change during training (Boyd al., 2006; Koloskova et al., 2020; Vos al., 2023).
4

---

Published as a conference paper at ICLR 2026
Avenug Clobal Test Aocurazy
Fed Aoeeooo
Red Forknwnn
Avengee Gilcbal Test Accuracy
Spece pmngertaal Cotruenrien
Anegue
0/6
Globall
High Communication Time Window
High Communication Time Window
Communication Round
(a)
(b)
(c)
Figure 2: (a, b): Comparisons of global test accuracy (see Definition 1) in decentralized training
of ResNet-18 8 on CIFAR-100 with AdamW, distributed across 16 agents with Dirichlet a = 0.1 (see details in Appendix C.I). Fully-connected communication (i.e., AllReduce) is activated only in specific windows, while low communication with one random peer with probability of 0.2 is used elsewhere. (a): Fully-connected communication 1/10 of total rounds. (b): Fully-connected communication in 1/20 of total rounds. In both, lighter bars show peak accuracy, darker bars show final accuracy. (c): Global test accuracy curves for local models and the globally averaged model (counterfactual) under persistent low communication (blue) and no communication (orange).
Specifically, the communication scheme is as follows: (1) fully-connected communication (see Fig- 1d (b)) is activated only within specific communication windows (i.e., global synchronization via AllReduce (Sergeev & Del Balso, 2018)"); (2) while in all other rounds, each agent communicates only with one random peer with a probability of 0.2 (see "Communication Graph" in Appendix C.I).
As shown in Figure 2, training is divided into 10 (a) and 20 (b) communication windows, respectively, The bars in Figure show both the best global test accuracy achieved during training (lighter-oolored bars) and the final test accuracy at the end of training ddarker-colored bars). Each bar corresponds to one communication window, where fully connected communication is applied only to the rounds within that window, while random peer communication is used in all other rounds. For instance, the inset in Figure 2a presents the complete test accuracy trajectory when fully-connected communication applied during rounds 150 to 180. A consistent trend emerges: allocating communication budgets toward the later stages of training yields substantial improvements, particularly in final test accuracy.
4.2 A SINGLE GLOBAL. MERGING SIGNIFICANTLY IMPROVES GLOBAL TEST PERFORMANCE
In Figure 2b, we reduce the fully-connected communication window length to 10 rounds, yet still observe substantial improvements in global test performance. This observation naturally raises the question: What happens if the fully-comnected window is reduced a single round?
To investigate this, we conduct experiments where fully-connected communication is applied only once, implemented by a single global merging. As shown Figure and Figure 1b, single global merging is sufficient to significantly improve global test performance. Consistent gains are observed across a wide range of settings, including different datasets, architectures, and communication topologies (see additional experiments in Appendix C.3). The significant increase in performance suggests that the potential of decentralized learning might be considerably underestimated.
Comparisons. D-PSGD (Lian et al., 2017) introduced final global merging under IID settings, but did not analyze the performance gap before and after merging. In contrast, we provide a systematic study of this recovery in challenging non-III scenarios. Chen al. (2021) showed the benefit of periodic global averaging, but their method requires frequent global communication (every = 48 steps), whereas we recover performance with only a single merging. We also compare with Skew- Compensated Sparse Push (SCSP) (Aketi et al., 2021), which also includes final global merging
'we note that A11 Reduce can be efficiently realized in decentralized manner such as Ri Ring-ALLReduue.. 6 "Code: https://// /top:://gh/ /ttpttbttbbb.////////////t.... www/cc/////////ee Rai den- 2huu 1CLR- 2026--26--22 2-6----rriingg--- Sengginn-Deee Ln-Decent Learaiing "The term "counterfactual" refers to the fact that glohal merging occurs during decentralized training. Instead, we manually compute the test accuracy of the hypothetical globally averaged model to quantify the "mergeabilit"" local models.
5

---

Published as a conference paper at ICLR 2026
step. While both works aim reduce communication, they differ from ours in methodology and setting: (1 SCSP uses gradient sparsification (top-k gradients) over a fixed topology, whereas we study mergeability under topological sparsification (sparse gossip); (2) SCSP focuses on one local step (H = 1), whereas we show robust mergeability with multiple local steps (e.g.. H = 100) under high heterogeneity. While these works share the broader goal of improving communication efficiency, our work offers a new perspective by investigating the mergeability itself: Why local models retain this property despite extremely limited communication and high data heterogeneity.
Remark 2 (Non-Triee Triviality of the Performance Gain). One may attribute the final improvement to the cumulative effect of sparse gossip. In our setup, each communication round activates one random peer exchange with probability R = 0.2 over T = 300 rounds, yielding about 60 peer exchanges per agent expectation. This may appear analogous, on average, to multiple implicit global aggregations. If this additive interpretation were sufficient to explain the gain, local models should already be better aligned and achieve performance close to that of the post-merge model. However, empirical results challenge this interpretation: shown in Figure Ib, local models under sparse gossip still exhibit poor global test performance, close to the no-communicatee while a single final global merging produces large performance gain. This gap indicates that the effect of single merging is non-trivial, rather than a simple result of accumulated local gossip.
case,
Communication Cost Comparison. Let P be the model size, m the number of agents, and T the number of training rounds. A standard communication cost of (22mPT). In contrast, our decentralized setup has cost of OmRRPT- 2mP)), where R < 2 denotes the expected number of peers per round, and the O(2mP) term arises from final global merging via Rí Ring-Al -
protocol incurs a total
Reduce. We also note that while a global merging may appear less practical in some cases, it can be effectively approximated via multiple rounds of gossip synchronization among local agents (see our supplementary experiments in Appendix C.3.4).
4.3 MERGEABILITY PERSISTS UNDER LIMITED BUT NONZERO COMMUNICATION
A follow-up question is whether the effectiveness of the global merging is specific the end of training. To investigate this, we assess the counterfactual performance of the globally averaged model each training round, depicted by the light-blue curve in Figure 2c. The experiments are conducted under a lower-communicant peer at each round with probability 0.2 (see "Communication Graph" in Subsection C. A consistent superiority of the merged model (light-blue curve) over the local models (dark-blue curve) is observed throughout training, suggesting that local models remain mergeable all stages (see Definition 2).
setting, where each agent communicates with one random
As an ablation, we conduct an experiment where all models are trained locally without any communi- cation (see Figure 2c). In this case, the counterfactual test performance of the globally averaged model remains close to zero (light-orange curve), indicating that without communication, local models are not mergeable. This suggests that mergeability does not arise inherently from the local models themselves. Interestingly, under the low-communication setting, the performance of local models before merging (dark blue curve) remains similar to that in the no-communication case (dark-orange curve). However, after global merging, the models show significant performance improvement. This clear contrast implies that extremely limited but nonzero communication enables mergeability.
Mergeability without Consensus. Prior work gossip algorithms has suggested that local models may converge to a similar state even in minimal communication regimes (Jelasity et al., 2005). In contrast, our work addresses more challenging heterogeneous data setting where we find that local models do not reach a single consensus point, yet remain mergeable. Specifically, we identify an emergent geometric structure where decentralized training guides local models to a ring-like high-loss region surrounding a central low-loss basin (see Figure lc). Notably, this corresponds to a more challenging cross- coossinniiillzztiinn cross-distribution merging scenario, where local models do not start from a shared pretrained checkpoint and are trained on non-IID data distributions.
5 THEORETICAL ANALYSIS
In this section, we examine the underlying mechanisms that enable the mergeability of local models in decentralized learning. As initial step, we conduct fine-grained convergence analysis of the

---

Published as a conference paper at ICLR 2026
globally merged models trained by Decentralized SGD (DSGD). To substantiate the mergeability of local models, we compare the convergence rate of the merged model trained by DSGD with that of parallel SGD. Remarkably, we prove that the merged model in decentralized learning can match the rate of parallel SGD (Dekel et al., 2012; Li et al., 2014). This supports the empirical findings that the merged model can preserve the performance of individual local models (see Definition 2).
5.1 ASSUMPTIONS
We start by introducing the commonly used assumptions (Kong et al., 2021; Koloskova et al., 2020). Assumption 1 (Mixing matrix). Each sample of the (randomized) mixing matrix W E mmxx is doubly stochastic. Moreover, there exists p such that
Ew low O| s (1-r) - 611
VA E Rdxm
Here θ = -0,,...,,m.. + Assumption 2 (Smoothness). The objective function is four-times continuously differentiable (i.e., E C*) and there exist constants Ly 2 0 for q E (1......4... 4) such that:
18.... = 1 11
where 0 Ex On.
|V S Log VOEK..
(5)
We note that given L E C², the boundedness of the Hessian norm (i.e., the case q = 2) implies that L L2-smooth, thereby recovering Assumption D.I with (L = L2).
Assumption 3 (Bounded noise and diversity). There exist o² 2,5² such that for any 02 E (0x |
Eeg | (02s VLn (0+) 00
- VL(00)1 c² (6)
271
where L(0) 1 Σm (0).
Here a measures the local noise level and Ç is a measure of the heterogeneity among agents.
5.2 CONVERGENCE ANALYSIS
Theorem 1 (Non-convex Convergence Rate of DSGD). Suppose Assumption and Assumption hold. Consider decentralized SGD (DSGD) with initializations 0 = - for all k E V, and
- at the t-th step. To achieve an E-stationary point such that 7 EE [ S E, the total number of steps T satisfies:
constant learning rate satisfying 2. Let ((t)
m =k=1 (()) denote the averaged model
T 0 a²
1 (0) (C(O -(*))
mea
where (()) = (n12 -
r(0) + O( with r@)) =
1 ( (e( ) and the consensus distance E? = Tr(T
Remark We note that Theorem gives an implicit bound depending on U1), t (1,2.....5 1}. rather than closed-form expression. It primarily serves to bridge convergence with the per- iteration dynamics of (9), facilitating the subsequent derivation of the conditions on consensus and communication required to recover the parallel SGD rate (see Proposition 2 and Proposition 3).
Comparison and Technical Novelty. As summarized in Table unified analysis by Koloskova pe
et al. (2020) showed that DSGD suffers from additional terms of order O 11-p + vFa+c) in the convergence rate compared to parallel SGD. The core idea behind their analysis to separate the effects of three key factors: the descent force (i.e., the squared gradient norm), gradient noise, and
DDSG refers standard decentralized SGD where the optimizer in Algorithm replaced with SGD.
7

---

Published as a conference paper at ICLR 2026
Table Comparison of non-convex convergence rates for parallel SGD and DSGD, both run with m agents under non-IID data.
Algorithm
Parallel SGD
DSGD (Koloskova el al., 2020)
DSGD (ours)
Rate
o
-
Spric
+
-r-11 Uttt)
me=
me²
p
me
parameter discrepancy among agents. Each of these components is then analyzed and controlled separately. Among them, both the gradient noise and the model discrepancy treated as detrimental to convergence. In contrast, we adopt a new proof framework that leverages the implicit bias of de- centralized learning (see Proposition D.3 (Zhu et al., 2023b) and Appendix B.2). Rather than treating the discrepancy among agents purely as noise, we partially incorporate it as a constructive component essential for matching the rate of parallel SGD. This intuition formalized through the convergence
guarantee provided in Theorem 1, which introduces additional term of O E u(t) In
what follows, we conduct a fine-gr aie-ggained analysis on the sign of U(().
Remark 4 (Reduction to Standard Rates). We consider two special cases where the term U() vanishes because the consensus error is identically zero (E, = 0):
The single-agent case (m = 1);
The fully synchronous Parallel SGD case, where perfect synchronization ensures identical local models (e( g(() for all k).
In both settings, the auxiliary term (++) in Theorem 1 strictly equals zero. Consequently, Theorem 1 naturally recovers the rate of standard (Parallel) SGD, which is of the order
To better characterize how the high-order loss landscape affects the dynamics of U we introduce a new assumption that is theoretically novel yet empirically supported by prior literature.
Assumption 4 (Progressive sharpening). For any positive semi-definite matrix E, the gradient of population risk negatively aligns with the gradient of sharpness. Formally, v0
Rd
V ((O)) <
(7)
By the density of TR, there exists a > such that VC(0) < - - < <0 holds strictly. We refer to 7- A sup (r > 0 VVTVV) -Y1 - degree of progressive sharpening.
VC(O))} <//0)]] as the
Remark 5. Tr(V ((0)E) can be interpreted as "average sharpness" around A; see similar metrics in (Gu et al., 2023a; Zhu et al., 2023b). Assumption 4 reflects a widely observed phenomenon in deep learning: The loss gradient exhibits negative correlation with the gradient of sharpness (Wang et al., 2022; Damian et al., 2023; Cohen et al., 2025). Intuitively, this condition implies that as the optimizer moves to reduce the loss, simultaneously moves in direction that increases the sharpness.
T(0) in (+() remains negative. the following, we establish that this term can dominate the other terms in U(),, thereby ensuring U < O.
Assumption ensures
Tr
Proposition 2. Suppose Assumption and Assumption 4 hold. Assume 17 satisfies r > 1/L2, and assume I C(())) 2 14 > foror all t. Consider the matrix p(t) Σk= (0 o( g(()
T and its trace = Tr(((()). Then, for any fixed m >> O, there exists E > 0 such that
U() (nL2 - VVV
+ O < O.
(8)
Explanations for Assumptions. We assume lower bound on the global gradient norm evaluated at the averaged parameters (()), i... [[())) 2 ft > O. We note that this applies to the gradient on the global data set, which remain significant even if individual local gradients
8

---

Published as a conference paper at ICLR 2026
vanish. The assumption is motivated by the Polyak-Lojasiess [|V 2 ((((0) - which ensures the gradient is bounded from zero before reaching the optimum. Our new assumption formalizes this property for the pre-convergence phase by denoting this lower bound at iteration t as 1 We also note that Assumption 2 requires that the norm of the loss derivatives is bounded up the fourth order, || (())l S Tig for g 1, 2, 3, 4. These bounds are necessary to analyze the interaction between the consensus error Ee and higher-order landscape.
(PL) condition (Polyak, 1963),
At a high level, Proposition 2 highlights the potential of leveraging decentralized training to accelerate distributed training beyond communication efficiency. This theoretical insight aligns with the empirical gains observed in Figure C. Specifically, Proposition 2 implies that satisfying U < 0 necessitates careful control of both y) and = We proceed by discussing the learning rate constraint below, and subsequently analyze the role of = based on which we provide a theoretical justification for allocating more communication in the later stages of training Subsection 5.3.
Remark 6 (Acceleration Under Larger Learning Rate). Note that the coefficient of the term V2(0()) V Tr (V (((()) T()) in Inequality (8) is (1 - 1). To ensure the negativity of this term, we require the condition > L.g. Notably, the resulting interval 1 L.g r < La2
2
coincides exactly with the regime of ossiilatooyy convergence" in classical optimization theory for a quadratic objective C(0) _ H0 (Polyak, 1987, Chapter 1. p. 26).
To provide deeper intuition, we outline the proof sketch of Theorem I below. This analysis establishes a novel descent lemma tailored for decentralized SGD, demonstrating how learning rate control harnesses progressive sharpening to drive acceleration.
Eger [(( s <(((()) - (n-
وا 2
20
Standard Descent
L222 2m
+
(1 n)
V VC((()) TTTO")))VV((((((0) +
+ (9)
>0
<0, progressive sharpening
We note that this third-order effect differs from prior analyses, which were typically established in a near-minima regime (Li et al., 2022c). By contrast, we show that this mechanism emerges whenever local models are inconsistent, i.e., E,
Equation (9) further illuminates the critical role of the consensus violation term E, v (T)) Crucially, the progressive sharpening term scales with OE whereas the higher-order residual error (3)) Consequently, provided E, is properly controlled such that the ( gain dominates the ( error, decentralized SGD can match or even surpass the convergence rate of parallel SGD.
According to Corollary D.2, E [ is bounded by
B < o (1 p2 (1-7))
(12"))
(10)
where the parameter p(with p E (0, 1] reflects connectivity in the communication graph (see Assump- tion 1). larger p indicates better connectivity and faster consensus, while smaller p implies sparse communication graph (i.e., lower communication) and slower information propagation. For example, p corresponds a fully connected topology, enabling perfect communication, whereas p 0 represents the extreme case of complete local training with no communication.
Why Limited but Nonzero Communication Enables Mergeability, Notably, random commu- nication graphs can achieve p ((1), striking a favorable trade-off: they require relatively low communication overhead while still maintaining efficient information mixing due to randomized edge sampling, which ensures rapid decrease of E, (Vos et al., 2023). This is why we adopt random topologies as the primary setup in our experiments: They can satisfy the condition in Proposition even under extremely limited communication, thereby ensuring mergeability (see Figure 1). However, in the case of full local training where p = 0 (see Figure Id), the right-hand side of Equation (10) increases to infinity, indicating that E, may diverge. As consequence, the condition of E, in Propo- sition can no longer be satisfied, which explains why local models after complete local training cannot be reliably merged (see the green curve in Figure 1b).

---

Published as a conference paper at ICLR 2026
5.3 A THEORETICAL EXPLANATION FOR COMMUNICATION ALLOCATION
Proposition highlights the importance of Ei to satisfy Inequality (8). This motivates the question of how small = (or how large p) should be, which we answer in the following sufficient condition.
Proposition 3 (Critical Consensus Edge). Suppose Assumption 1 I-Assumption 4 hold. Assume ri > I. and the consensus error E, S 1 for all I. Then, the following condition ensures that the critical Inequality (8) is satisfied:
24 (1 - p) n/
(nL2 - 1)r" Me
(nL2 1) pe
002 < min
(11)
p²
2(n L4 3/mm
2Ehigh
where Ehigh = 21122223 + / + umkahis 1152 Here, 7² denotes the degree of progressive
sharpening (see Assumption 4), &² denotes the uniform upper bound of the averaged squared local gradient norm (i.e., k= (0 S x²), and μr is the lower bound on the global gradient norm (i.e., (V((0) 24>00 0).
Practical Guidance. Proposition 3 provides guidance for allocating communication ensure ((() < 0, thereby guarantceing the contribution of each step to the cumulative sum in Theorem 1. From Equation (11), note that Φ, x², 7", m, and Ly (q E (1.....44 44) are lime-independent, while the key time-varying factor is the gradient lower bound μt,, which tracks the optimization status of the averaged parameters (0(1) on the global landscape. Under this interpretation, Equation (11) implies term should be dynamically adjusted in response to the changing landscape geometry captured by 14. Analytically, the left-hand side of Equation (11) is strictly
that the communication-relatee
decreasing function of p. while the right-hand side is an increasing function of 14, This implies a fundamental trade-off: more frequent communication (larger p) relaxes the condition, whereas vanishing gradient (smaller pt) tightens the allowable error bound. Specifically,
In the starting phase of training, when the globally averaged model is far from a minimum, the lower bound on gradient norm μ4 is large. This corresponds to a relaxed consensus requirement in Equation (11), which permits low-frequency communication (i.e., smaller p) without significantly impacting the performance of the globally merged model.
Early, High-Gradient Regime:
Late, Low-Gradient Regime: As models approach a solution and training enters convergence phase, the gradient norm f4 decreases. This tightens the constraint in Equation (11), In this regime, frequent communication (i.e., larger p) becomes critical.
We note that this theoretically motivated guidance aligns well with our empirical findings Section that more communication should be concentrated in the later stages of training.
6 IMPLICATIONS AND DISCUSSIONS
Model Merging. The effectiveness of a single merging of decentralized models has significant implications for model merging. recent work showed that pre-trained models occupy large, flat "basic capability basin", within which fine-tuning creates smaller "specific capability basins" (Chen al., 2025). The observed "mergeability" of local models in our paper implies that decentralized learning may guide agents into connected specific capability basins, allowing simple permutation- free merging integrate specialized knowledge. This suggests a practical direction: lightweight synchronization during local training may improve basin connectivity and simplify later merging into a more capable model.
Decentralized Learning. Our work provides promising empirical and theoretical evidence that decentralized learning can generalize under high data heterogeneity and limited communication. class of adaptive, communication
More importantly, our findings could directly motivate efficient decentralized learning algorithms, which dynamically allocate their communication budget by monitoring training dynamics to satisfy the critical consensus edge condition in Equation (11).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re- re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference Learning Representations, 2023.
Sai Aparna Aketi, Amandeep Singh, and Jan Rabaey. Sparse-push: Communication-& energy- efficient decentralized distributed learning over directed & time-varying graphs with non-iid datasets. arXiv preprint arXiv:2102.05715. 2021.
Youssef Allouah, Anastasia Koloskova, Aymane El Firdoussi, Martin Jaggi, and Rachid Guerraoui. The privacy power of correlated noise in decentralized learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 1115-1143, 2024.
Eric Bonabeau, Marco Dorigo, and Guy Theraulaz. Swarm Intelligence: From Natural to Artificial Systems. Oxford University Press, 1999.
Marco Bornstein, Tahseen Rabhani, Evan Z. Wang, Amrit Bedi, and Furong Huang. SWIFT: Rapid decentralized federated learning via wait-free model communication. The Eleventh International Conference on Learning Representations, 2023.
Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 558-568. Association for Computational Linguistics, 2023a.
Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin A Raffel. Distributed inference and fine-tuning of large language models over the internet. In Advances in Neural Information Processing Systems, 2023b.
S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah. Randomized gossip algorithms. IEEE Transactions on Information Theory, 52(6))2508-2555 2530. 2006.
Ying Cao, Zhaoxian Wu, Kun Yuan, and Ali H Sayed. On the trade-off between flatness and optimization in distributed learning. arXiv preprint arXiv:2406.20006, 2024.
CCAF. Cambridge bitcoin electricity consumption index (CBECT). https: www..c...cccccocccccooccccocoooa cbnsi// obeci, 2023.
Huanran Chen, Yinpeng Dong, Zeming Wei, Yao Huang, Yichi Zhang, Hang Su, and Jun Zhu. Understanding pre-training and fine-tuning from loss landscape perspectives. arXiv preprint 1rXiv:2505.17646, 2025.
Lesi Chen, Haishan Ye, and Luo Luo. An efficient stochastic algorithm for decentralized nonconvex- strongly-concave minimax optimization. International Conference Artificial Intelligence and Statistics, 2024.
Xuxing Chen, Minhui Huang, Shiqian Ma, and Krishna Balasubramanian. Decentralized stochastic bilevel optimization with improved per-iteration complexity. In Proceedings of the 40th Interna- tional Conference on Machine Learning, volume 202. pp. 4641-4671 PMLR, 2023.
Yiming Chen, Kun Yuan, Yingya Zhang, Pan Pan, Yinghui Xu, and Wotao Yin. Accelerating gossip sgd with periodic global averaging. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 1791-1802. PMLR, 2021.
Jeremy M Cohen, Alex Damian, Ameet Talwalkar, Zico Kolter, and Jason D Lee. Understanding optimization in deep learning with central flows. In The Thirteenth International Conference on Learning Representations, 2025.
Donato Crisostomi, Mareo Fumero, Daniele Baieri, Florian Bernard, and Emanuele Rodolà. 2m5 Cycle-consistent multi-model merging. In The ehirty-eighth Annual Conference Neural Information Processing Systems, 2024.

---

Published as a conference paper at ICLR 2026
Edwige Cyffers, Aurélien Bellet, and Jalaj Upadhyay. Differentially private decentralized learning with random walks. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 9762-9783 2024.
Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization: The implicit bias of gradient de- scent at the edge of stability, In the Eleventh International Conference on Learning Representations, 2023.
Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research, 13(6):165-202, 2012.
Arthur Douillard, Qixuan Feng, Andrei A Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Mare' Aurelio Ranzato, Arthur Szlam, and Jiajun Shen. Diloco: Distributed low- communication training of language models. arXiv preprint arXiv:2311.0811 2023.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309-1318.
PMLR, 2018.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance linear mode connectivity neural networks. In International Conference on Learning Representations, 2022.
Mathieu Even, Anastasia Koloskova, and Laurent Massoulie. Asynchronous SGD graphs: unified framework for asynchronous decentralized and federated optimization. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, 2024.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 13:5850-5861 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning. pp. 3259-3269. PMLR, 2020.
C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In International Conference on Learning Representations, 2017.
Hongchang Gao and Heng Huang. Fast training method for stochastic compositional optimization 2021.
problems. Advances in Neural Information Processing Systems,
Hongchang Gao, Bin Gu, and My T. Thai. On the convergence of distributed stochastic bilevel optimization algorithms over network. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206, pp. 9238 9281. PMLR, 2023.
Anissa Gardizy and Amir Efrati. Microsoft and OpenAl plot $100 billion stargate Al supercom- puter. The Information, 2024. URL http::/p/////w...eeeeoooooteette htt.-ppppap-----------------rrre
com aaaaaiiticcss htt.:ll----sttaaattt-------rrrro
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
Grand View Research. Ai infrastructure market size, share & growth report, 2030, 2024. URL https://wwe http:://www.gaaddieeeeeeeaa grandviewresearch. www.eeaarrc....aaaaaaaeeeeeeeeee wttp:///ntttttttaaaaaaaaaaea ai- Si-inffaattructuree wwwpuuccttremeeakeeeeeee -aarket-rroor........
Xinran Gu, Kaifeng Lyu, Longbo Huang, and Sanjeev Arora. Why (and when) does local SGD gen- cralize better than SGD? In The Eleventh International Conference Learning Representations, 2023a.
Xinran Gu, Kaifeng Lyu, Longbo Huang, and Sanjeev Arora. Why (and when) does local SGD generalize better than SGD? In International Conference Learning Representations, 2023b.
12

---

Published as a conference paper at ICLR 2026
Xinran Gu, Kaifeng Lyu, Sanjeev Arora, Jingzhao Zhang, and Longbo Huang. A quadratic synchro- nization rule for distributed deep learning. In The Twelfth International Conference on Learning Representations, 2024.
Mert Gurbuzbalaban, Yuanhan Hu, Umut Simsekli, Kun Yuan, and Lingjiong Zhu. Heavy-tail phenomenon in decentralized sgd. arXiv preprini arXiv:2205.0666 2022.
He, L. Nan, and T. Zhu. Imagining democratic, affordable future of foundation models: A decentralised avenue. In Handbook of Blockchain Analytics. Springer, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, 2016.
Lie He, Sai Praneeth Karimireddy, and Martin Jaggi. byzantine-robust decentralized learning via clippedgossip. arXiv preprint arXiv:2202.01545. 2022.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06333 2019.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannanch Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva (eds.). Proceedings of the Thirty-Fourth Conference Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 876-885, AUAI Press, 2018
Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lueas Atkins, Maziyar Panahi, Charles Goddard, et al. Intellect-1 technical report. arXiv preprint arXiv:2412.01152 2024.
Márk Jelasity, Alberto Montresor, and Ozalp Babaoglu. Gossip-based aggregation in large dynamic networks. ACM Trans. Comput. Syst., 23(3):21 August 2005. ISSN 0734-2071.
Salma Kharrat, Marco Canini, and Samuel Horvath. Decentralized personalized federated learning. arXiv preprint XXX::244060065000 2024.
Jari Kolchmainen, Nikolay Blagoev, John Donaghy, Oguzhan Ersoy, and Christopher Nies. Noloco: No-all-reduce low communication training method for large models. arXiv preprint vrXiv:2506.1091 2025.
Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In International Conference on Machine Learning, 2020.
Lingjing Kong. Tao Lin, Anastasia Koloskova, Martin Jaggi, and Sebastian Stich. Consensus control for decentralized deep learning. In Proceedings of the 38th International Conference Machine Learning, 2021.
Alex Krizhevsky, G Hinton, et al. Learning multiple layers of features from tiny images (tech. rep.). University of Toronto, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
Batiste Le Bars, Aurélien Bellet, Mare Tommasi, Erick Lavoie, and Anne-Marie Kermarrec. Refined convergence and topology learning for decentralized sgd with heterogeneous data. In Proceedings of the 26th International Conference Artificial Intelligence and Statistics, 2023.
13

---

Published as a conference paper at ICLR 2026
Batiste Le Bars, Aurélien Bellet, Mare Tommasi, Kevin Scaman, and Giovanni Neglia. Improved stability and generalization guarantees of the decentralized SGD algorithm. Proceedings of the s1st International Conference Machine Learning, 2024.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train-merg: Embarrassingly parallel training of expert language models. In First Workshop on Interpolation Regularizers and Beyond at NeurlPS 2022, 2022a.
Mu Li, David G Andersen, Alexander J Smola, and Kai Yu. Communication efficient distributed machine learning with the parameter server. Advances in Neural Information Processing Systems, 2014.
Shuangtong Li, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. Learning to collaborate in decentralized learning of personalized models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9766-9775, 2022b.
Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss? -a mathematical framework. In International Conference on Learning Representations, 2022c.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In Advances Neural Information Processing Systems, 2017.
Xiangru Lian, Wei Zhang, Ce Zhang, and Liu. Asynchronous decentralized parallel stochastic gradient descent. In International Conference on Machine Learning, 2018
Tao Lin, Sai Praneeth Karimireddy, Sebastian Stich, and Martin Jaggi. Quasi-global momentum: Accelerating decentralized deep learning on heterogeneous data. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Yucheng Lu and Christopher De Sa. Optimal complexity in decentralized training. In Proceedings of the 38th International Conference Machine Learning, 2021.
Kaifeng Lyu. Implicit Bias of Deep Learning Optimization: A Mathematical Examination. PhD thesis, Princeton University, 2024.
Enrique Tomás Martínez Beltrán, Mario Quiles Pérez, Pedro Miguel Sánchez Sánchez, Sergio López Bernal, Gérôme Bovet, Manuel Gil Pérez, Gregorio Martínez Pérez, and Alberto Huertas Cel- drán. Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and challenges. IEEE Communications Surveys & Tutorials, 25(4):2983-3013, 2023.
Michael S Matena and Colin Raffel. Merging models with fisher- averaging. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Michalis Mavrovouniotis, Changhe Li, and Shengxiang Yang. A survey of swarm intelligence for dynamic optimization: Algorithms and applications. Swarm and Evolutionary Computation, 33: 1-17, 2017.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54, pp. 1273-1282. PMLR, 2017.
Abdellah El Mrini, Edwige Cyffers, and Aurélien Bellet. Privacy attacks in decentralized learning. In Proceedings of the s1 International Conference on Machine Learning, 2024.
Giorgi Nadiradze, Amirmojtaba Sabour, Peter Davies, Shigang Li, and Dan Alistarh. Asynchronous decentralized sgd with quantized and local updates. Advances in Neural Information Processing Systems, 2021.
Vaishnavh Nagarajan and Zico Kolter. Uniform convergence may be unable to explain generalizatioonnn in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
14

---

Published as a conference paper at ICLR 2026
Angelia Nedi'e e and Alex Olshevsky. Distributed optimization tine-varying directed graphs. volume 60, pp. 601-615. IEEE, 2014.
Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization.
IEEE Transactions on Automatic Control, 54(1))48-61, 2009.
OpenAI. Announcing the stargate project. 2025.
https: haops://openai.o wwt.ccoom.ii///e/eee/eeee.eeeeee index/
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Konstantin F Pilz, James Sanders, Robi Rahman, and Lennart Heim. Trends in ai supercomputers. arXiv preprint 2r0iv:2504. /6026, 2025.
B. T. Polyak. Gradient methods for minimizing functionals. Zhurnal Vychislitel noi Matematiki i Matematicheskor Fiziki, 3(4):643-653, 1963.
Boris T Polyak. Introduction Optimization. Optimization Software, Inc., Publications Division,
New York, 1987.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763. PMLR, 18 24 Jul 2021.
Sameera Ramasinghe, Thalaiyasingam Ajanthan, Gil Avraham, Yan Zuo, and Alexander Long. Protocol models: Scaling decentralized training with communication--ffiieee model parallelism. arXiv preprint arXiv:2506.0122 01260, 2025.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, patrick gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Leon Bottou, and David Lopez-Pazz Model ratatouille: Recycling diverse models for out-of-distribution generalization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28656, PMLR, 23-29 Jul 2023.
Dominic Richards al. Graph-dependent implicit regularisation for distributed stochastic subgradient descent. Journal of Machine Learning Research, 2020.
Max Ryabinin, Tim Dettmers, Michael Diskin, and Alexander Borzunov. SWARM parallelism: Training large models can be surprisingly communication-efficnn In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 19416-29440, PMLR, 2023.
Ali H. Sayed. Adaptation, Learning, and Optimization over Networks. Now Publishers, 2014.
Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint
2018
Li Shen, Yan Sun, Zhiyuan Yu, Liang Ding, Xinmei Tian, and Dacheng Tao. On efficient training of large-scale deep learning models. ACM Computing Surveys, 57(3). 2024.
Tao Shen, Didi Zhu, Ziyu Zhao, Chao Wu, and Fei Wu. Will Ilms scaling hit the wall? breaking barriers via distributed resources on massive edge devices. arXiv preprint arXiv:2503.0822. 2025.
Abhishek Singha, Charles Lua, Gauri Guptaa, Ayush Chopraa, Jonas Blanca, Tzofi Klinghoffera, Kushagra Tiwarya, and Ramesh Raskara. A perspective on decentralizing ai. 2024.
15

---

Published as a conference paper at ICLR 2026
Ankit Sonthalia, Alexander Rubinstein, Ehsan Abbasnejad, and Seong Joon Oh. Do deep neural network solutions form a star domain? The Thirteenth International Conference on Learning Representations, 2025.
Tao Sun, Dongsheng Li, and Bao Wang. Stability and generalization of decentralized stochastic gradient descent. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Yuki Takezawa, Ryoma Sato, Han Bao, Kenta Niwa, and Makoto Yamada. Beyond exponential graph: Communication- effficienn topologies for decentralized learning via finite-time convergence. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu. D2: Decentralized training over decentralized data. In International Conference on Machine Learning. PMLR, 2018.
Zhenheng Tang, Shaohuai Shi, Wei Wang, Bo Li, and Xiaowen Chu. Communication-effae distributed deep learning: A comprehensive survey. arXiv preprint arXiv:2003.06307, 2020.
J. Tsitsiklis, D. Bertsekas, and M. Athans. Distributed asynchronous deterministic and stochastic gradient optimization algorithms. IEEE Transactions on Automatic Control, 3 (1()))803 812, 1986.
Gal Vardi. On the implicit bias in deep-learning algorithms. Commun. ACM, 66(6):86 93, 2023.
Thijs Vogels, Lie He, Anastasia Koloskova, Sai Praneeth Karimireddy, Tao Lin, Sebastian U Stich, and Martin Jaggi. Relaysum for decentralized deep learning on heterogeneous data. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Thijs Vogels, Hadrien Hendrikx, and Martin Jaggi. Beyond spectral gap: The role of the topology in decentralized learning. Journal of Machine Learning Research, 24(355):1-31, 2023.
Martijn De Vos, Sadegh Farhadkhani, Rachid Guerraoui, Anne marie Kermarree, Rafael Pires, and Rishi Sharma. Epidemic learning: Boosting decentralized learning with randomized communica- tion. In seveth- Conference on Neural Information Processing Systems, 2023.
Jue Wang, Yucheng Lu, Binhang Yuan, Beidi Chen, Percy Liang, Christopher De Sa, Christopher Re, and Ce Zhang. CocktailSGD: Fine-tuning foundation models over 500Mbps networks. In Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.
PMLR, 2023.
Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K. Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 37(6):1205 1221, 2019.
Zixuan Wang, Zhouzi Li, and Jian Li. Analyzing sharpness along GD trajectory: Progressive sharp- ening and edge of stability. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyum Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rehecca Roelofs, Raphael Gontijo-Lopes Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 2022a.
PMLR,
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 7959-7971, June 2022b.
Tongle Wu and Ying Sun. Implicit regularization of decentralized gradient descent for sparse regression. In the ehirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
16

---

Published as a conference paper at ICLR 2026
Wenhan Xian, Feihu Huang, Yanfu Zhang, and Heng Huang. A faster decentralized algorithm for nonconvex minimax problems. Advances in Neural Information Processing Systems, 34: 25865-25877 2021.
Jie Xu, Wei Zhang, and Fei Wang. A(dp) Asynchronous decentralized parallel stochastic gradient descent with differential privacy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES TIES-merging: Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Shuoguang Yang, Xuezhou Zhang, and Mengdi Wang. Decentralized gossip-based stochastic bilevel optimization over communication networks. Advances in Neural Information Processing Systems, 35:238 252, 2022.
Haoxiang Ye and Qing Ling. Generalization error matters in decentralized learning under Byzantine attacks. IEEE Transactions on Signal Processing, 2025.
Bicheng Ying, Kun Yuan, Yiming Chen, Hanbin Hu, Pan Pan, and Wotao Yin. Exponential graph is provably efficient for decentralized deep training. In Advances in Neural Information Processing Systems, 2021.
Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao, Beidi Chen, Percy Liang, Christopher Re, and Ce Zhang. Decentralized training of foundation models in heterogeneous environments. Advances in Neural Information Processing Systems, 2022.
Kun Yuan, Qing Ling, and Wotao Yin. On the convergence of decentralized gradient descent. SIAM Journal on Optimization,
2016.
Liangqi Yuan, Ziran Wang, Lichao Sun. Philip S. Yu, and Christopher G. Brinton. Decentralized federated learning: A survey and perspective. IEEE Internet Things Journal, pp. 1-1, 2024.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Shahryar Zehtabi, Dong-Jun Han, Rohit Parasnis, Seyyedali Hosseinalipour, and Christopher G Brin- ton. Decentralized sporadic federated learning: A unified algorithmic framework with convergence guarantees. In The Thirteenth International Conference on Learning Representations, 2025.
Wei Zhang, Mingrui Liu, Yu Feng, Xiaodong Cui, Brian Kingsbury, and Yuhai Tu. Loss landscape dependent self-adjusting learning rates in decentralized stochastic gradient descent. arXiv preprint arXiv:2112.01433, 2021.
Zhanpeng Zhou, Yongyi Yang. Xiaojiang Yang. Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Zhanpeng Zhou, Mingze Wang, Yuchen Mao, Bingrui Li, and Junchi Yan. Sharpness-aware minimiza- tion efficiently selects flatter minima late in training. In The Thirteenth International Conference on Learning Representations, 2025.
Miaoxi Zhu, Li Shen, Bo Du, and Dacheng Tao. Stability and generalization of the decentralized stochastic gradient descent ascent algorithm. In shirty-seentth Conference on Neural Information Processing Systems, 2023a.
Tongtian Zhu, Fengxiang He, Lan Zhang, Zhengyang Niu, Mingli Song, and Dacheng Tao. Topology- aware generalization of decentralized SGD. In International Conference on Machine Learning. PMLR, 2022.
Tongtian Zhu, Fengxiang He, Kaixuan Chen, Mingli Song, and Dacheng Tao. Decentralized SGD and wverag-ddirection SAM are asymptotically equivalent. In Proceedings of the 40th International Conference on Machine Learning, 2023b
17

---

Published as a conference paper at ICLR 2026
Tongtian Zhu, Wenhao Li, Can Wang, and Fengxiang He. DICE: Data influence cascade in de- centralized learning. In The Thirteenth International Conference on Learning Representations. 2025.
18

---

Published as a conference paper at ICLR 2026
LLM USAGE STATEMENT
We use large language models (LLMs) as writing-assistance tools. Their role is confined to proof- reading and language polishing.
IMPACT STATEMENT
This paper studies the problem of temporal communication allocation in decentralized distributed learning, topic of very high significance in the of communication inntiiicattionnnnnssieee large model training. Specifically, we aim to contribute to the development of coaamunication-fffciiee decentralized learning without compromising performance. The potential positive social impact twofold:
Democratizing Access. For individuals and organizations with constrained infrastructure, our work contributes to the democratization of access to large-scale collaborative training. By reducing communication requirements, we lower the barrier to entry for participating in advanced model development. Such inclusivity can extend the applicability of distributed learning systems edge environments, thereby promoting more equitable contributions to models trained at scale.
Reducing Training Costs. In data center environments, our approach can alleviate communication bottlenecks distributed training. This reduction directly translates to shorter total wall-clock training time, thereby lowering the overall costs and energy consumption associated with large-scale distributed training.
No negative societal impacts are identified.
ETHICS STATEMENT
Our research strictly adheres the ICLR Code of Ethics. The work is foundational, focusing the algorithmic and theoretical properties of decentralized learning, and does not involve human subjects the collection of new sensitive data. All experiments were conducted on publicly available. standard academic datasets. We foresee no direct negative societal impacts; on the contrary, by reducing communication overhead, our findings may contribute positively by democratizing access to large-scale distributed training and lowering the associated resource footprint.
REPRODUCIBILITY STATEMENT
We committed the reproducibility of our research. Our theoretical claims, including all assumptions and their justifications, are presented in Section 5 with complete, step-by-step proofs provided in Appendix D. Comprehensive details for reproducing model architectures, data processing, hyperparameter settings, and communication configurations, well documented in Appendix C.I.
empirical results, including
A LIMITATIONS AND POTENTIAL QUESTIONS
Q: Why use decentralized Adam W in some experiments when the theory is on decentralized SGD?
A:We use decentralized AdamW some of our experiments for its superior performance in Non- IID settings. Crucially, all reported empirical observations remain fully consistent when using decentralized SGD, directly aligning with our theoretical analysis (see Figure and Subsection C.3).
Q: How does the theory explain local models in decentralized learning are globally mergeable?
A: The theoretical explanation for the "mergeability" of local models in decentralized learning is supported by our result that globally merged model converges faster to the optimum than individual local models. Specifically, we provide fine-gne line-grained convergence analysis showing that the model merged from decentralized SGD (DSGD) match the convergence rate of parallel SGD, despite
19

---

Published as a conference paper at ICLR 2026
limited communication. Since the rate of m-agent parallel SGD is superior to that of a single local model, this result transitively justifies the merged model's 's superior performance relative to any individual model, thereby providing theoretical support for mergeability.
Q (Hyperparameter Tuning): How were the baselines luned in terms of hyperparameters?
A: All hyperparametters were tuned via grid search based on global test performance, with the batch size searched over {64, 128 }. For ResNet-18 trained from scratch on Tiny ImageNet, we searched the learning rate over f1 10 5 10-4, 44 x 10-3) 33 for Adam and {1 x 10 3, 5, x 10 3 x 10-2) for SGD. For CLIP B/T-B/32 on Tiny ImageNet, we searched the learning rate over {1 10
10-4, 1 X 10-3 for Adam W and {5 x 10-4, 1 x 10- 5 x 10-3, 1 x 10-2} for SGD. For the optimal hyperparameters selected for our main experiments, please refer to the Implementation Details in Appendix C.I and the additional empirical results in Subsection C.3).
(Comparison with Model Soup): How do initialization schemes affect results? Performance gains from merging have been observed in Model Soup (Wortsman et al., 2022a).
A: We use different initialization schemes and observe consistent performance gains from global merging, whether models start from different random initializations or from a pretrained state. The majority of our experiments use different initializations, demonstrating that local models in decentralized learning can be effectively merged regardless of their starting points. This is quite surprising, as it contrasts with methods like Model Soup, which require models to fine-tuned from an identical pretrained state. Furthermore, our experiments with shared pretrained state confirm that the performance gains hold that setting as well (see Figure and Subsection C.3).
Q (Methodology for Landscape Visualization): Please clarify the methodology for visualizing the loss landscape in Figure 1c, including the basis for the visualization grid.
A: We adopt the visualization tool from (Crisostomi et al., 2024), positioning 16 trained mod- els at the vertices of a regular hexadecagon. Any point within this polygon is an interpo- lated model whose parameters are determined by Wachspress barycentric coordinates; we then evaluate its cross-entropy loss to generate the contour map. Unlike methods that use ran- dom directions, our visualization grid is deterministically defined by the models themselves. allowing a direct investigation of their geometric connectivity. The full implementation is available in their official code repository contours __________ p__models.ipynt gageen//hhbhnaaseerroooookkpppoo Lospoootoooosssssoooouusssss__
Q (Experimental Scope): The empirical findings are restricted visual tasks.
A: Our empirical findings primarily focus on tasks within the vision domain. We note that this is consistent with most existing decentralized learning literature (Lin et al., 2021; Kong et al., 2021; Ying et al., 2021; Vogels al., 2021; Li al., 2022b; Zehtabi et al., 2025). Extending the experimental setup to broader tasks is a meaningful direction for future research.
Q: The finding in Figure (c), that local models eventually converge similar state even with limited communication, was also observed in prior work (Jelasity et al., 2005).
A: our setting, the local models do not, in fact, converge to similar state or a single consensus point. This is because our work addresses more challenging heterogeneous data regime, which differs from the setting in the cited prior work. Instead, we identify an emergent geometric structure where decentralized training guides local models to shared "high-loss ring" surrounding a central low-loss basin (see Figure Ic). Although the models do not reach a consensus, they remain surprisingly mergeable within this region. This geometric arrangement allows their average, i.c., the globally merged model, to fall directly into the low-loss basin. To the best of our knowledge, we are the first to identify this emergent phenomenon in decentralized learning.
20

---

Published as a conference paper at ICLR 2026
B ADDITIONAL BACKGROUND AND RELATED WORK
B.1 DECENTRALIZEI LEARNING
Modern large-scale model training and inference are predominantly conducted within centralized, high-cost data centers. Driven by mounting constraints on computational resources and power availability (Pilz et al., 2025), both academia and industry are increasingly exploring decentralized training approaches (OpenAI, 2025; Grand View Research, 2024). This paradigm, drawing inspiration from swarm intelligence systems (Bonabeau et al., 1999; Mavrovouniotis et al., 2017), offers a more economical and scalable approach by distributing computational tasks across globally distributed nodes, rather than relying solely on a single central server (Yuan et al., 2022: Borzunov et al., 2023b; Jaghouar et al., 2024; Ramasinghe et al., 2025). A notable illustration of the computational potential through decentralization is the Bitcoin system, which sustains workloads equivalent to a 16 GW power draw (CCAF, 2023), surpassing by factor of three the estimated GW consumption of the largest AI supercluster under development (Gardizy & Efrati, 2024; OpenAl, 2025).
To provide context, we summarize key algorithmic and theoretical advances in decentralized learning. While our discussion highlights several notable contributions, it is not exhaustive; readers are referred to recent advances and surveys (Zhu al., 2025; Martínez Beltrán et al., 2023; Singha al., 2024; Yuan et al., 2024; He et al., 2025; Ramasinghe et al., 2025; Kolchmainen et al., 2025).
Algorithmic Progress in Decentralized Learning. The advancement of decentralized learning algorithms has been primarily driven by the need for communication--ffiiieene learning. Decentralized algorithms have been refined to handle a variety of realistic scenarios, including time-varying communication topologies (Nedi' c & Olshevsky, 2014; Koloskova et al., 2020; Ying et al., 2021; Takezawa et al., 2023), asynchronous updates (Lian et al., 2018; Xu et al., 2021; Nadiradze et al., 2021; Bornstein al., 2023; Even et al., 2024), statistical heterogeneity (Tang et al., 2018; Vogels et al., 2021; Le Bars et al., 2023), and robustness to Byzantine failures (He et al., 2022; Ye & Ling, 2025). Moreover, recent works extended beyond standard empirical risk minimization to more structured problem classes, such as compositional (Gao & Huang, 2021), minimax (Xian al., 2021: Zhu et al., 2023a; Chen al., 2024), and bi-level optimization (Yang et al., 2022; Gao et al., 2023; Chen et al., 2023). Additionally, privacy concerns in decentralized learning are also critical, with efforts focusing on differentially privacy (Cyffers al., 2024; Allouah et al., 2024) and data reconstruction attacks (Mrini et al., 2024).
in practical distributed
Theoretical Progress in Decentralized Learning. Foundational work on decentralized optimization (Nedic& & Ozdaglar, 2009; Sayed, 2014; Yuan et al., 2016; Lian et al., 2017) laid the groundwork for understanding convergence. Building on this, Lu & De Sa (2021) proposed a hierarchical abstraction of decentralization, distinguishing it into three layers, providing a unified view across federated and decentralized paradigms, Koloskova et al. (2020) consolidated synchronous decentralized SGD algorithms with changing communication topologies and local updates, and Even et al. (2024) extended the unifying perspective to asynchronous protocols. More recently, Zehtabi et al. (2025) developed these frameworks further by considering the sporadicity of both communication and computations. On the generalization front, Richards et al. (2020) derived stability- bounds for decentralized SGD in convex settings, while Sun et al. (2021) extended these to non-convex objectives, revealing a dependency on the spectral gap of the communication graph. This dependency was subsequently refined by Zhu et al. (2022), who introduced Gaussian weight difference assumption to tighten the bound. Complementary results showed that in convex regimes, the generalization of decentralized SGD matches that of centralized SGD (Le Bars al., 2024), while in non-convex landscapes, decentralization primarily impacts worst-case generalization behavior. To account for unexplained generalization behaviors in decentralized training (Kong et al., 2021; Gurbuzbalaban et al., 2022; Vogels et al., 2023), Zhu et al. (2023b) linked decentralized SGD to random sharpness- aware minimization (SAM), revealing bias toward flatter minima. Notably, akin to our finding that decentralized learning generalizes when allocated high communication late in training, Zhou et al. (2025) has shown that SAM efficiently selects flatter minima in the later stage of training.
Towards Decentralized Training of Foundation Models. Recent advances have shown the feasi- bility of training large-scale foundation models in decentralized environments. DT-FM (Yuan et al., 2022) introduced based scheduling for Transformer training under handwidth-consttring settings, enabling efficient resource allocation. SWARM Parallelism (Ryabinin et al., 2023) scaled decentralized training through resilient pipeline design and adaptive load balancing. CocktailSGID
21

---

Published as a conference paper at ICLR 2026
(Wang et al., 2023) further improved efficiency via a combination of decentralization, gradient sparsi- fication, and quantization for LLM fine-tuning. On the inference side, Petal (Borzunov et al., 2023a) exploited peer-to to-peer networks amortize computational costs actoss heterogeneous nodes. Most recently, Intellect (Jaghouar et al., 2024), building on Diloco (Douillard et al., 2023), leveraged hybrid parallelism, i.e., both data and model parallelism, to collaboratively train models with billions of parameters. NoLoCo (Kolehmainen et al., 2025) further extended Diloco to gossip-type decentralized settings. For a broad survey of large-scale deep learning practice, see Shen al. (2024; 2025).
B.2 IMPLICIT BIAS OF DECENTRALIZED LEARNING
The concept of implicit bias, i.e., the intrinsic preference of learning algorithms for solutions with certain properties, has emerged as a key concept in explaining the empirical success of modern deep learning (Li al., 2022c; Vardi, 2023; Lyu, 2024). Recent studies have highlighted intriguing distinctions between decentralized stochastic gradient descent (DSGD) and its centralized counterpart (CSGD). Gurbuzbalaban al. (2022) demonstrated that under certain conditions, DSGD operating on large, sparse topologies exhibits heavier-tailed parameter distributions compared to CSGD. Zhang et al. (2021) showed that decentralization introduces aandsaap-deepndeet noise, which can improve tolerance to larger learning rates. This observation aligns with findings by Vogels et al. (2023), who revealed that collaboration in decentralized settings permits the use of larger learning rates. Zhu et al. (2023b) first explicitly characterized the implicit bias of decentralized SGD by establishing its connection with random sharpness-awand minimization, proving the existence of flatness bias in decentralized training. Complementing this, Cao et al. (2024) offered detailed analysis of the interplay between flatness and optimization in DSGD, particularly its ability to escape local minima. More recently, Wu & Sun (2024) investigated the implicit regularization properties of decentralized optimization in non-convex sparse regression problems, recovering the convergence rates achieved by gradient descent centralized settings.
Comparison with Zhu et al. (2023b). We note that Zhu et al. (2023b) has highlighted the gener- alization benefits of decentralized learning. but key differences exist in terms of the experimental setup and the insights derived. While Zhu et al. (2023b) focused on IID scenarios and specific cases involving exceptionally large batch sizes, we consider the more realistic non-III setting using standard batch sizes. This shift in focus allows us to uncover phenomena not observed by Zhu et al. (2023b), including insights into communication allocation strategies.
B.3 MODEL MERGING
Mode Connectivity and Model Merging Techniques. Recent works on (Linear) Mode Connectivity have advanced our understanding of the complex loss landscape in neural networks. Freeman & Bruna (2017); Draxler et al. (2018); Garipov et al. (2018); Nagarajan & Kolter (2019); Frankle et al. (2020) discovered that different solutions of deep neural networks can be merged together by simply averaging their parameters. Sonthalia et al. (2025) further showed that the solutions may form a star domain. We note that these phenomena are observed in the following scenarios:
Shared initialization (Frankle et al., 2020; Fort et al., 2020; Zhou et al., 2023). Models are initialized from pretrained checkpoint.
Homogeneous data distribution (Wortsman el al., 2022a). Models are trained homogeneous data distribution.
Permutation (Ainsworth el al., 2023; Entezari et al., 2022). Models are independently trained. The neurons of one model are permuted match the neurons of the other while maintaining a functionally equivalent network.
These findings have inspired a range of model merging techniques for various applications. Izmailov et al. (2018); Matena & Raffel (2022); Rame et al. (2022; 2023); Wortsman et al. (2022a;b) found that merging the parameters of models that start from the same pretrained model and finetune over the same task leads to improved generalization and robustness. Furthermore, Ilharco et al. (2022); Li et al. (2022a); Ilharco et al. (2023): Ortiz-Jimenez et al. (2023): Yadav et al. (2023) showed that merging models that finetune over different tasks enables multi-task abilities.
22

---

Published as a conference paper at ICLR 2026
Comparisons with Model Merging Literature. Our results show that mode connectivity, or mergeability, can still emerge in decentralized learning. even when the local models are initialized differently, trained on highly heterogeneous data, and merged without any permutation. Our findings offer new insights into both model merging techniques and the geometry of the neural network loss landscape, which anticipate will motivate further advances in both areas.
C ADDITIONAL EXPERIMENTS
C.I EXPERIMENTAL SETUPS
Computational Resources. The experiments were conducted computing facility equipped with 80 GB NVIDIA @ A100 GPUs. All implementations are based on PyTorch, and computations are distributed across multiple GPUs for efficiency.
Dataset. We use three widely adopted image classification datasets: CIFAR-10, CIFAR- 100 (Krizhevsky et al., 2009), and Tiny ImageNet (Le & Yang. 2015). CIFAR-10 consists of 60,000 RGB images across 10 classes, while CIFAR-100 R-00 contains 60,000 RGB images across 100 classes. The images in both datasets have spatial resolution of 32 32 pixels. Tiny ImageNet is subset of the ImageNet dataset, comprising 100,000 images drawn from 200 classes, with each image resized to 64 x 64 pixels. It provides mid-scale benchmark that is more challenging than CIFAR datasets but less computationally demanding than training full ImageNet. To incorporate data augmenta- tion, we employ a combination of RandomCrop with 4- 4-pixel padding, RandomHorizonttllFiip and RandAugment with num ops=2 and magnitude=9
Details of Decentralized Learning. We simulate a heterogeneous decentralized learning environ- ment. For our main experiments (Figure la and Figure 1b), we use = 32 agents, while for other experiments, including the sliding window experiments (Figure and the loss landscape visualiza- tions (Figure lc), use 16 for clarity, as a plot with 32 models would be visually crowded. In all configurations, we employ Dirichlet distribution characterized by 0.1 to partition the data among agents. The Dirichlet distribution is commonly used to partition data in federated learning scenarios, as it allows for the control of label distribution skew among agents (Yurochkin et al., 2019; Hsu et al., 2019). A smaller a results in more imbalanced data distributions, where some agents predominantly receive data from limited number of classes, while larger results in more uniform label distributions across agents. This configuration effectively captures the realistic non-llD nature of decentralized learning, where different agents may have access to personalized data reflective of their local environments.
16 agents. The number of agents for the visualization was chosen as
Communication Graph. We evaluate three decentralized communication topologies: random graph, ring graph, and exponential graph. In the random graph setting, during each communication round, each agent selects random subset of its neighbors for gossip averaging. For "R 1", each agent selects exactly one random neighbor in each round. For "R 0.2", each agent selects one neighbor with probability 0.2 and continues local training without communication with probability 0.8. The ring graph enforces fixed cyelic communication structure, while the exponential graph ensures connectivity by allowing agents communicate at exponentially increasing distances.
Communication Rounds and Local Steps. The decentralized learning process is conducted over T = 300 communication rounds. We use a local training step size of H = 100 batches per communication round balance communication and computation costs.
Local Data per Agent. Each agent is assigned a subset of the dataset with fixed size of 4096 samples, drawn according to Dirichlet distribution to simulate realistic non-lID scenarios.
Model Architecture. To ensure a representative comparison across different model families, we adopt ResNet-18 18 (He et al., 2016) and CLIP VIT-B/32 (Radford et al., 2021) backbone architectures in our experiments. ResNet-18 a widely used lightweight convolutional neural network that serves as a canonical example of traditional CNN-based architectures. In contrast, CLIP ViT-B/32 is transformer-basee vision model pre-trained on large-scale image-tex: text pairs. For experiments on Tiny ImageNet, where images are resized to 64x64 pixels, adjust the CLIP visual encoder to handle the lower resolution. With patch size of 32, each image yields visual tokens arranged in a 2x2 grid, plus a [CLS] token, resulting in 5-token input sequence.
23

---

Published as a conference paper at ICLR 2026
Implementation Details. All hyperparameters are tuned through grid search based global test performance (see Definition 1). For experiments using decentralized SGD, the optimal learning rates were found to be 1 x 10-2 for 18 (trained from scratch) and X 10 3 for CLIP ViT- B/32. When using decentralized AdamW, the optimal learning rate is x 10-a for ResNet-18 (both when trained from scratch and fine-tuned from SrageNet-preetrinee weights) and pretrained CLIP ViT-B/32 on Tiny ImageNet. For all experiments, weight decay is set to x 10 and the batch size is selected as 128. The key empirical results remain consistent across these optimizer and hyperparameter choices, indicating that our conclusions are stable and not sensitive to specific hyperparameter configurations.
10 5 for the
Details of Loss Landscape Visualization in Figure 1c. To analyze the geometric connections among models after decentralized training, we visualize the loss landscape spanning their param- eter spaces. We adopt the visualization tool from (Crisostomi et al., 2024), which is specifically designed to analyze the interpolation space within the convex hull formed by a given set of mod- els. In our implementation, we position the 16 trained models the vertices of a regular hex- adecagon. Any point within this polygon represents an interpolated model, whose parameters are a weighted sum of the parameters of the 16 vertex models; the weights determined by the point's Wachspress barycentric coordinates. We then evaluate the erossyy loss of each interpolated model the entire test set generate the final loss contour map, shown in Figure 1c. The http://ciiostom/cccllleossssssis
implementation is available in their notebook loss contours n models. ipynb within the official code repository for (Crisostomi et al., 2024). We note two key aspects of this visualization approach:
merging/bloie
Focus Convex Combinations. For points outside the polygon, one or more of their barycentric coordinates become negative, corresponding to an extrapolation, which is often unstable. This visualization approach is consistent with Definition 2, focusing on the space of convex combinations among the models.
Deterministic Grid vs. Random Directions. Notably, the visualization method differs from approaches that use random directions to probe the landscape of a single model, as our visualization grid is defined directly by the 16 models themselves. This allows geometric connectivity and interpolation properties among this predefined set of models.
directly investigate the
Computational Resource Requirements and Runtime. To enhance accessibility for researchers working with diverse computational environments, our code includes a centralized simulation of decentralized training. This enables the reproduction and extension of our decentralized learning experiments using fewer GPUs. A single decentralized AdamW training experiment with 16 agents using ResNet-18 18 on the Tiny ImageNet dataset requires approximately 15 GB of GPU memory and can be conducted on a single GPU with sufficient memory, such as an NVIDIA V100, RTX 3090, RTX 4090, or A100. On an A A100 GPU, the typical runtime approximately hours for 300 communication rounds, each comprising 100 local steps. For the CLIP ViT-B/32 model, the memory demand rises to about 30 GB, yet it remains feasible on a single A GPU, with a runtime of approximately 12 hours under the same configuration of 300 communication rounds and 100 local steps per round.
C.2 PRACTICAL EVALUATION METRICS
The standard evaluation metric of parallel and federated learning is the accuracy of the global model. Definition C.1 (Test Accuracy of Global Model). The accuracy of the global model 0 is defined as:
Acc(O) A 1 Σ Egg-D, Acc(O:Ex) irl Acc(O0)((:5))
m
In decentralized learning, models are often evaluated in the absence of a full consensus model 0 due to data heterogeneity and limited training time. Two major metrics are adopted in this scenario. Definition C.2 (Average Local Test Accuracy). The average accuracy of agents k V is defined as:
Acc((0ulkev)
EELND, Acc(OniEn) Ex)
if FID 1
Σ LE-D (x)
m
KEV
KEV
Average Test accuracy on the local distribution across ogents
24

---

Published as a conference paper at ICLR 2026
Remark C.1 (Local Generalization). This metric aims to address the following question in decen- tralized learning: how well do local models {0k yev, with the aid of peer-to-peer communication. generalize to their local (personalized) data distribution Dj. This is the standard evaluation metric in personalized decentralized settings, where the goals
optimize local objectíves.
However, in real-world scenarios, local data distributions are often heterogeneous and not guaranteed be IID across agents. In such settings, an important goal understand how well local models, trained on limited local data, generalize to the global data distribution. To account for this, we adopt the following average global test accuracy, a proxy of average global population risk, the primary evaluation metric, which quantifies how well local models generalize to the global distribution.
Definition C.3 (Average Global Test Accuracy). The average accuracy of agents lo is defined as:
Ace (Oo 1 E Acc(Ox), where Acc(.)
BEIDD Aco(;ii)
KEV
LEV
Average Accuricy across agents
Test accuracy on the global distribuaion
Remark C.2 (Global Generalization). This metric is specifically designed to address core research question in fully decentralized learning with non-IID data: how well do local models (0k (oe) trained with limited peer-to-peer synchronization, generalize the global data distri- bution DD? We note that this objective is particularly critical in the highly non-llI scenarios we study, where local models drift significantly apart. Unlike federated learning that measures the performance of a global model, this metric offers a more realistic evaluation for decentralized settings where no central server is present.
C.3 ADDITIONAL EXPERIMENTS
C.3.1 DIFFERENT NUMBER OF AGENTS AND OPTIMIZERS
We conduct additional experiments by varying the number of agents (from 16 to 32) and comparing different optimizers (SGD to AdamW). The effect of single merging remains consistent.
Avronaee Global Test Aecuruy
Trising Decestaliaed Tratzing
Avrerage Global Test Aecuruyy
Trvising Deceralllod Tratnnng Loci Tharring
Leca Teaiting
24-
044
Communication Round
CLIP VIT-B/32
(b) Reseeet-18 (w'o pretraining)
Figure C.1: (a, b): Global test accuracy (see Definition 1) of CLIP ViT-B/32 (a) and ResNet-18 18 (b) trained on Tiny ImageNet using FedAdam (blue), decentralized Adam W (orange), and one-shot FedAdam (green), distributed across 16 agents with high data heterogeneity (Dirichlet = 0.1). Decentralized training involves each agent syncing model parameters with random peer per round with probability of 0.2, with single global merging at the final round (see details in Appendix C.I).
C.3.2 DIFFERENT COMMUNICATION TOPOLOGIES
We also conduct additional experiments with different communication topologies to examine whether the empirical results remain consistent. New observations summarized below.
Models remain mergeable under different numbers of peers. We evaluate two settings (random topology with R 0.2 and R 1; see "Communication Graph" in Appendix C.I). As shown in
25

---

Published as a conference paper at ICLR 2026
Pedersee Trusking Lecn Trimng
Pedenned Traiskig trating
Avennge Giobbal Test Accuracy
Avennge Globbal 'est Accuracy
6.-
0.2
Communication Round
Cummunication Round
(a) CLIP ViT-B/32
(b) ResNet-18 (w/o pretraining)
Figure C.2: (a, b): Global test accuracy (see Definition 1) of CLIP ViT-B/32 (a) and ResNet-18 (b) trained on Tiny ImageNet using F (blue), decentralized Adam (orange), and one-shot FedAdamW (green), distributed across 32 agents with high data heterogeneity (Dirichlet Decentralized training involves each agent syncing model parameters with random peer per round with probability of 0.2, with single global merging at the final round (see details in Appendix C.I).
0.1).
Averge Glohal Test Acourey
Avrengge Global Test Accuracy
Aumagee Prpperts Sronk
04
83
0x-
Communicatinn Round
Communiction Round
Different Number of Peers R
(b) Different Topologies
Figure C.3: Global test accuracy (see Definition 1) of training ResNet-18 18 on Tiny ImageNet, dis- tributed across 16 agents with high heterogeneity (Dirichlet a 0.1; see details in Appendix C.1). We evaluate the effects of different (a) number of peers Rt and (b) communication topologies. Pretrained weights are used only (a).
Figure C.3a, performance improvements are consistently observed, with more significant gains the R = 0.2 case.
Models remain mergeable across different communication topologies. We evaluate two topolo- gies: exponential and ring graphs. As shown in Figure C.3b, both topologies preserve the mergeability of local models, with exponential graphs yielding slightly better test performance for both local and merged models. The trend of mergeability persists across topologies throughout training, though performance may vary.
C.3.3 DIFFERENT HYPERPARAMETER DATASET, AND HETEROGENEITY LEVEL
We further conduct supplementary experiments in which the final global merge is approximated by topology-constrained gossip merging on an exponential graph (Ying et al., 2021).
Summary. Consistent test performance improvement of single global merging range of settings are observed, including different hyperparameter setups, datasets, degree of data heterogeneity, model architectures, optimizers, initialization schemes, and communication topologies.
wide
C.3.4 REALIZING FINAL GLOBAL GOSSIP VIA TOPOLOGY-CONSTRANNE MERGING
We also perform additional experiments where the final global merging is approximated by multiple rounds of gossip merging, i.e., specific exponential topology (Ying al., 2021).
26

---

Published as a conference paper at ICLR 2026
Tiny ImageeNet (m-16 16, alpha - 0.1)
Tiny ImageNet (m-16, alpha 0.1)
Thy ImaggNet alpha 0.1, random topology)
Teattnal pirtuc thr
Musd
Average Global Toan Accuncy
Avenge Avenge Global Tost Accuracy
6-3
Loadd 500 coc-ttuaaae cabrddy
lerage Unuar Juan bntileeded
44
34
144
Average Global
02-
32-
Communiictioon Round
Communication Round
Commumication Round
(a) Different Batch Sizes
(b) Different Learning Rates
(c) Different Initialization Schemes
Figure C.4: Global test accuracy (see Definition 1) of training ResNet-18 18 on Tiny ImageNet with decentralized Adam W, distributed across 16 agents with high heterogeneity (Dirichlet a 0.1; see details in Appendix C.1). We evaluate the effects of different (a) batch sizes (64 vs. 128), (b) learning rates (5 x 10 vs. 1 10 +), and (c) different initialization schemes.
0:3)
(mr52,
Axapp Clobel Ean Actunay
Awarg GlAl Des ACCEPE
enpormi rage
ago
Carmmatiotion Roasd
Corminionion Roas
Commuriatico Read
Commusittion Roasd
(b)
(d)
Figure C.5: Global test accuracy (see Definition 1) for ResNet-18 trained with decentralized A across 32 agents under different levels of data heterogeneity (Dirichlet a = 0.1 (a, c) vs. a = 1.0 (b, d); see Appendix C.I). Results are reported on both CIFAR-100 (a, b) and Tiny ImageNet (c. d).
Acconity
Fokeradd Théisng Lscal Torning
Accrage Global Teat Accuracy
Rasknntd Triisig
Average Accuracy
Trainin Deverralzoed Lacal Traising
Lacil Theriig
4.4
62-
02-
02
Commmniiatioon
Communicatton Round
Communication Round
(a) Round Final Gossip Merging (b) 5 Rounds Final Gossip Merging (c) Round Final Global Merging
Figure C.6: Global test accuracy (see Definition 1) for ResNet- 18 trained with decentralized A on Tiny ImageNet (32 agents). Left: regular decentralized training followed by one round of topology- constrained final gossip merging setting, but with five rounds of final gossip merging to better approximate global aggregation. Right: baseline setting (our original approach), with random communication among all agents during training followed by one perfect global merge.
exponential graph. Middle: the same copology-constraee
Summary. We observe that (1) even single round of topology-come constrained final gossip merging substantially improves global test accuracy, and (2) the resulting performance is comparable to the baseline that uses random communication among all agents followed by perfect global merge.
27

---

Published as a conference paper at ICLR 2026
D THEORY
This section provides the proofs of the main theoretical results presented in this paper. For simplicity, and following the setup in the existing literature, we assume that the sample size of local agents is Wy = n for all k E V.
Lemma D.1 (Consensus Distance Recursion under Local Updates (Kong et al., 2021)). Suppose Assumption 1-Assumption 3 hold. Let g be the local parameter on client k at t-th step, and denote their average by 0C8) m -k=1 gft) Define the consensus distance and the average gradient norm round t by E 1 where Lx = BEgaDD [c(0: Let the learning rate, and o² the variance bound from Assumption 3. Then there exists a constant p > 0 (see Assumption 1) such that for all l 2 0, the following inequality holds:
- k= [10( 10 and of 1 - VV (8(2) )1?
E 1 S 21-2
(1-2)
12(1 12(1-P),7 (
2
(D.1)
P
where the expectation is taken over the stochastic gradients in the t-th update phase.
Proof. For completeness, we provide the proof of Lemma D.1, with minor corrections and additional details. In decentralized SGD (Algorithm with SGD as the local optimizer), each agent A E V
performs at each iteration
(++11
W We V2
In matrix form, letting
( = 10
1 E I ddxm
(((((())())) = cB(
VL (O elt) )]
we have
q(+))= ( W.
The consensus matrix after mixing is
((++)) = ((++)) 111 = ( _11
since 1 W W 1
Thus the consensus distance satisfies
m 2 [| (1 = - VL(0
- ())) (W 1, 1111))1
By Assumption for any θ E Rdxm
Ew OW - -01 s (1 1p)pppl--0p
obtain,
m E.p S 5(1 (1- p) 0 (1 m - v (1- m 11
Applying the inequality ||A B < (1+ a)) + (1 1/ with a = & gives
m - (1 - p) (1 + /)) 1 (I 111 |1
シ) |C(000 6(1 - p) n (
+ (1 0
$ (1 [ m +
1 ; |
p
where we use (1 p/2) < 1 p and (1 + 2/p) < 6/p for p E (0,
28

---

Published as a conference paper at ICLR 2026
We now decompose the stochastic gradient as
,C(0"))(()) - (
so by Young's Inequality, we have
< 2 + + -
Taking expectation over g(t) and invoking Assumption get
[ { = 2 + 20 2
Substituting back and dividing by m yields
B B $ (1 / = +
12(1
1f (0) + 00))
which completes the proof.
Corollary D.2 (Upper Bounds of Consensus Distance (Kong et al., 2021)). Define the consensus
distance and the average gradient norm at round t by = E 100
1( and of
1 Ek= (O( )l2 where (x(0) = E [C(0; Ex) Under the conditions of Lemma D.I. suppose that all iterations t, the gradient norms uniformly bounded by constant o, i.e. of S Φ², Vt {I....,T} T... Then the expected consensus distance satisfies
BEA
S
24 24 -
(o' (0"
12
In the general case where the gradient-norae change slowly, i.e., of S (1 ()) of we have
p1-
E E
p²
The expectation here is taken over the stochastic gradients in the t-th update phase.
Proof. Consider the key recursion from Lemma D.I:
E [ < (1
12 p) 12 (0) +
(1) Special Case: uniformly bounded gradient norms.
Assume of < Φ². Unrolling the above gives
E [ < [
12 (1 - n²20² +
t=0
Since (1 - 2)² 2 we can bound the consensus distance as
12 - p)
2 24 - p?
E E <
12 + +07)
(0² +07)
x P 12
which yields the first claim.
(2) Special Case: slowly changing gradient norms.
の < (1 + P ob41, and since
(1 (1- () (I=))(+++)² s (1-
29

---

Published as a conference paper at ICLR 2026
the consensus distance satisfies
12(1 - -p)np² (0%-1 +
B Σ
E
(1-2)
p)
2
p
1=0
t-1
4)² 12(1 p)n + +
48(1 p)n
4
Σ p²
(02-1
+o
(D.2)
Proposition D.3 (Implicit Bias of Decentralized SGD (Zhu et al., 2023b)). Suppose Assumption hold, the globally averaged model of decentralized SGD (DSGD), defined by 01t) - 0 follows the following gradient descent direction:
Bgtts ppt (ett)Biit++) = - (EWNN(O (2(00 + +c())) t010,
-
where ((t) m
(0 - - B())T Rmxm denotes the consensus distance matrix,
m kk=1
and g(t) = O DE 100 12) denotes the high-order terms. The first expectation
eliminates the randomness from sampled data g(t) = =55 TKkv at step (t).
We can then control the expected squared distance between two consecutive steps of the globally averaged model with Corollary D.4.
Corollary D.4. Under the assumptions in Proposition D.3, the expected squared distance between two consecutive iterates of decentralized SGD can be bounded as follows:
Egco - S
02 + 12
VC( +
Tr (V + 8w0l (D.3)
m
Proof. Denote ((+
-Ee(t) Egct) 0(++1) g(t), We can expand the expected distance follows:
Egi [0(++]) g(t)| 1 Egco ||0(+2)112 + || 2(0(1)) ( ((+1)
Tr (Cov(ō((+1)) + Becs 112 - T(
Ir (Cov(((l+1))
+ B (0(c+)) - -
Tr Cov( vC(O ((o)))
B|goo (p(ct))
( - @(0]]1 (D.4)
1=1
where the second equality follows from the definition of the covariance matrix, namely
Tr (Cov((((+1)) Ego ||(++1) | | @((+2))1
and the final equality is derived from the original update of decentralized SGD (without applying Proposition D.3):
@(++1) 0(t)
Σ
According to the convexity of the vector norm and the fact that
2
172
((0) 1 m
Tr Cov( VC(O :55 e(()))
Egto
EB Beo
VL(((0
(1) (D.5)
m
m
we then complete the proof by applying Proposition D.3 and the bounded noise assumption in Assumption 3.
30

---

Published as a conference paper at ICLR 2026
Corollary D.5. Let P(O) = :. E T B E Rdxdd where ((0)
m. =k== 1 TRd denotes the globally averaged model across m agents. Suppose Assumption
1
hold. Then, for ( ~ (N(,, the expected gradient perturbation satisfies:
B. VC(((() +cl0)] -VC(0
v (ViC(O()))())) (EW--O(TTO [Ra((t))].
(D.6)
where [| is bounded by 111000
Proof. We apply the third-order Taylor expansion to VL around (
V L(((()) (((0())(0)) VL(O + VZL(O() (t) + ( c00 Rs(c!1),
with the remainder:
Ra(e /'' (1-T)² V L(O + (l)e)( (d
Taking expectations over clt) (N(,, since E[e(0] the linear tenm vanishes. The quadratic
term [ [V'L((()) ((0()) c((]) simplifies to Tr (V² ((((t) (+))) due to properties of the Gaussian
distribution. The remainder bound can be bounded as
( %
(1 7
La 10 11 ///
1 / (1 (1-7)"dT. 7T.
6
Since (0 (1-1)) = 1. we have:
[ = La 11
1
L4
10
4 24
For comparison, we restate the convergence rate of DSGD by Koloskova et al. (2020).
Assumption D.1 (L-smoothness), Each population risk L -ELLD. C(O: Ex) for k {I,..., m}
is continuously differentiable, and there is a constant L 2 0 such that:
V (f) (+)) S L OII, A0.0 E Rd
(D.7)
Theorem D.6 (Non-convex Convergence Rate of DSGD (Koloskova et al., 2020)). Under Assump- tion 1, Assumption D. I and Assumption 3, let the learning rate satisfy r S Amnax = ( let
g(t) 1 Em_ g( denote the averaged model at the t-th step. To achieve an E-stationary point I E, the total number of steps T satisfies:
such that Ei= E [
[13]
?
VDo+ +5 1 (C(00) - 2.)
T= O=
mm 2
3
We then provide our main theoretical results as follows.
Theorem D.7 (Non-convex Convergence Rate of DSGD). Suppose Assumption and Assump-
tion 3 hold. Consider decentralized SGD (DSGD) with initializations ( =0(0) for all k V,
and a constant learning rate satisfying 1) < T.g Let (()) Er (t) denote the averaged
model at the t-th step. To achieve an e-stationary point such that ET- [[ S. the total number of steps T satisfies:
T = O
+
EED
ET- U0) (C((1()) - -."),
where (tt)
(n
T()) + OE33), with (
1 Em k= (0(.)
()()
T and the consensus distance E? Tr(r()).
31

---

Published as a conference paper at ICLR 2026
Proof. We structure the proof into several key steps.
Step (A): Descent Force Decomposition.
Based the Lg-smoothness (Assumption D.I) of the loss function & (as implied by Assumption 2)), we can apply the first-order Taylor expansion around g(+) to establish an upper bound for ((((++1)):
(( (((()) (L(( T. (g(c+1)) - 0(0) (B(c+1)-(1))
+ 2 | - -
According Proposition D.3, we have
Egen [0(+)) g(0) 17 (VL(B A ( + +8(1))
where ((t) denotes the variance matrix of c(1) ~ (0, (+)) and σ(() = θ - -m 100
100
denotes the high-order residuals (see Proposition D.3).
Substituting this into the previous bound and taking the expectation with respect to random data sampling yields:
Ega[ [[((+++)]
S (C(00))~V((00))) V((((O) ( - 810) BBer txx -101tt1)-0000 100.p
According to Corollary D.4, we obtain
Egta [ - x(t) | n²0²2 +
> VL((() + ( V + 8100pp
We then decompose the squared norm:
- 4 = || - (((1o).
Combining the previous steps, obtain:
Ege9 ((((+1)) & (((()) - (n 2 )2) V
v [
8
2 (-n + (2) VC((()) T v (( tn VL((0)) 801
0700
L2
m
2
+ 172 L2 (VC(0 + vI(() To 8000_ + n Le
1
(D.8)
2
Tb
We subsequently control terms related to - ( - (O(0) in Equation (D.8).
Step (B): Control Consensus-Relatee Terms
Combining the residual upper bound in Corollary D.5 and with the concavity of (-)3/2 we derive
||000
<
La
3 <
LA
Σ 100000 13)
24
m
24
and thus T3 Liki
vm t Σm
| a 113}
21
Further, by the convexity of square operation, we have
Tj = 1| =
mL 24
Σ
( - g(0) 1333
m k=1
32

---

Published as a conference paper at ICLR 2026
According to Assumption 2, upper-bound T1 as
T V Tr C =
|( r
- L3.
(
1 m
Σ
B
where the colon (:) represents the tensor contraction of the third-order derivative with the matrix r. We can also bound T2 as follows:
T4 S
V
VL
( 100 ||
2
k
L3 1
10
+ In)
1
((0) |s
2 m
k
m
k
L3
10
10 + L1)
10000 - 0003|
2 m
m
For clarity, we consolidate the terms involving Ti to T2 in Equation (D.8):
Accumulated T-terms =(-n+ L2) + T/T3 + L2 (T1 + 8T4 + 4T²).
(D.9)
8
Substituting the Upper bounds for the Accumulated T-terms into Equation (D.8) yields the updated descent incquality:
Egeo ((0(+))) < (((t)) - (1 - |V 11= + 7 + ,'
(D.10)
where 17 (tt) denotes the upper bound for Equation (D.9):
m/LL,
n U - T) +
24
1 + n" و + 2"² (2L1 + 13 + 122mLII2 =/
1152
(n T) + [n + nLa
72 122 + 1/2
r/mLzL?
/mL
V
=
24
2
1152
è A(t) (EE))
H 0
(D.11)
and we recall that the consensus distance E = 1 m Σm |0 - 1 To facilitate subsequent
analysis, we further separate ('() into an Acceleration term (1) plus High-order terms (0)
With ( serving as a unified proxy for the consensus errors, we obtain the new rate as follows.
Step (C): Derive the Convergence Rate
Starting from the descent inequality Equation (D.10):
Egto [C()(+1))] < -(0)) (n- 22 /L(DO) + 2 + o² L2
2
m 2
Taking full expectation and summing over t = O,..., T - 1, we obtain
L2 2
T-1
o² L2
( t=
Σ n
E (0 < ((B(0))
E[C(O())] +n Σ
U
2m
To ensure the descent property of -|| ||| || ||
we have to set 17 1222 that 17 < f, with é 2. Under this condition, and denoting A L(010)
> 0. which in turn implies
L. we obtain
1 T
2A
2 T-1 1 (2
σ² n22
E
U(t) +
(D.12)
T
(2-
e) T
(2 E) m
1=
33

---

Published as a conference paper at ICLR 2026
To ensure this is most it suffices to enforce
n2
2
1
T
U
and
2A
8
(D.13)
(2 (2 2)
3
(2 - C)
3
(2
(2-6)
(2
3
t
To satisfy all three conditions simultaneously, along with a stability condition 1 select accordingly:
we should
{
e (2 (2- Eime L2 20 f and 2
64
6
Σ D
}
min
(2 ()
E)ne
E)E
To ensure valid step-size y7 exists, we substitute these three upper bounds into the condition for T. This yields three distinct lower bounds on the total number of iterations T that must be satisfied. By rearranging the inequality Tn 2 6A require:
6AL2 ((2 ()
18A0 L2
T Σ U
T max
(2 8)2 me?
(2 - E)E
where the first two bounds are derived directly by substituting the first two terms from the minf.) operation for y) into the first lower bound of T.
Therefore, the total number of iterations should be large enough to satisfy all applicable lower bounds. This leads to the sufficient condition:
T
O
A E
A
2
T-1
Σ
(10)
m
8
This condition is sufficient to guarantee
Σ
E
V
C(((()))))
Se.
T
t=
The proof is now complete.
Propusition D.S. Suppose Assumption and Assumption hold. Assume > 1/L2, and assume Σm
[| 2 Mr > for all t. Consider the matrix [(t) = -= (o( (0() T
and its trace Et TT(T(()). Then, for any fixed m > 0, there exists E/ > 0 such that
U A 2 (nL2 V) T TT((V= r()) + < 0. (D.14)
OE?
Remark D.1. We note that Proposition D.8 does not contradict Equation (D.12) when both A and o are zero. The condition = L((())) - C = implies that the models are initialized at an optimal point. In Theorem D.7, we assume that all initializations are identical (0 )(0), Vk E V), so it follows that all models begin the same optimum. Consequently, the consensus error remains zero throughout all iterations, meaning the model covariance matrix ((t) is the zero matrix and its trace E, also zero. Since every component of the term (tt), defined
L4 L VmL n +
1 3 mm L2
(nL2 1)72 U
[n
=
=
1152
A = =(EE)
A mr()) O(E))
(D.15)
Proof. The proof relies on establishing that for sufficiently small E,, the negative leading term A(0) in the decomposition of (+) dominates the higher-order residual term (1), Specifically, A is of order OE=?), while (!) is of order (EE)).
34

---

Published as a conference paper at ICLR 2026
Step (A): Derive Upper Bound on A Let ((t) VL(((() and recall that
T- (1) V Tr ( ((0()) r
Based on the definition of (+)) in Equation (D.15), we have the leading term A (nL2 - 1)T2. To analyze 72, consider the trilinear form defined on the unit sphere. Let 5( be the deviation vectors such that [ Ek= 8 ) and E 1 Ex= _ 112. Define the function F(6) for 5 + 0:
F(6) = V LOL(O) (0, 0
Under Assumption 4, we have A ( 6, ( < O. implying F(6) < 0. Since the unit sphere {6 || = 1 is compact, F attains maximum value M < 0. Let 7 -M > 0. It follows that for any vector 6,
V (6 6, g (0 s -> 1100|.
Substituting this bound into the summation for T2:
To L0 10% (()) 1 Σ (100 = (0
Given the gradient lower bound [ 2 Pe, we obtain Tę S - Consequently, assuming the pre-condition 1/22 - 1 holds, the leading term A(t) satisfies:
A = (n - 1)T; I - (n 1)
(D.16)
This confirms that A provides strictly negative contribution of order O
Step (B): Dominance over Higher-Order Residuals. We now show that U A( H7) for small E, According to Equation (D.15), the residual term is given by
<0
nL2 + [nL
La
mpb222
(1) =
VmL + g 2222L3 +
=3
E
24
1152
P(E.)
Here, P(E,) is a polynomial in E, with positive coefficients, and thus H = OE?)). The condition
(() < is equivalent to (1) (tt)) Using the bound from Equation (D.16), it suffices to show:
P(E))= < (n/22 1)T,,
For E, > 0, we divide both sides by =7 reducing the condition to:
S, P(E)) < (nL2 1)mpe
Let (u)) = u. P(u) for a 2 0. Since P(u) is bounded in a neighborhood of zero, we have:
lim Q(u)= -0- [ nL2 [nL +
La 24
MLL 0.
The term on the right-hand side, C 2 (n1T22 1)7pr, is strictly positive constant. By the continuity of polynomial functions and the intermediate value theorem, there exists threshold 8 > 0 such that for all 0 < E, < 8, the inequality Q(E,) < holds. This implics that for a sufficiently small consensus error E,, the negative drift from (t) dominates the residual H ensuring U(() <. O
Explanation.
The high-level intuition of Theorem D.7 and Proposition D.8 is outlined by the following descent lemma.
Egwo [(O)+)))] <C(0())- (1)
1² La
V VL()
>0
Standard Descent
35

---

Published as a conference paper at ICLR 2026
sf L2
+
(n 1)) progressive sharpening
VL(0(1) (V((V22(0)))))))
2m + +0
>0
(D.17)
Remark D.2. To simultaneously satisfy this requirement (2-2)m S e/3 in Equation (D.13)
oLyLa
and the condition 7) L/L2 needed to maintain the descent property introduced by progressive sharpening, we can appropriately scale the number of agents m. Specifically, for any arbitrarily
d² small target accuracy E, (2-é)m S e/3 is guaranteed to hold as long as the network size satisfies
30² m > (2-1)e* In other words, we can achieve the same convergence rate as parallel SGD by
adapting the network size m to the desired target accuracy E.
Proposition D.9 (Critical Consensus Edge). Suppose Assumption 1 I-Assumption hold. Assume 7i Ty and the consensus error E, S 1 for all t. Then, the following condition ensures that the critical Inequality (8) is satisfied:
24(1 p)n?
(1122 - 1)
(1122 - 11)7"
(62+ < min
(D.18)
p2
2(nL2 21 Wdmm
2Inigh
where Thigh = 22 + m + mmLgl3 Here, " denotes the degree of progressive 1152
sharpening (see Assumption 4), o² denotes the uniform upper bound of the averaged squared local gradient norm (i.e., Ex (O 2 S Φ²),, and Mr is the lower bound on the global gradient norm (i.e., |VL())) 2 p4 > 0).
Proof. The proof establishes that the condition in Equation (11) suffices to guarantee ( < 0.
Recall from the decomposition in Equation (D.15) that ((t) = A + (+)), where ( is the leading descent term and (d represents higher-ordet residuals. We aim to show that the negative drift dominates the error, i..., (1)) < -
Using the lower bound on the gradient norm V 2 Mr derived in Proposition 2, the leading term satisfies (²)) < -nn 1)r" 1, Let K & (nL2 1)n" p4- Under the sharpening regime (nL2 > 1), we have K > 0. Thus, sufficient condition for (1(t) < 0 is:
10 KE
(D.19)
Step (A): Bounding the residual term. Substituting the expansion of (*)) from Equation (D.15) into Equation (D.19) and dividing both sides by E/ (assuming E, 0), the requirement becomes:
(n + b)) VmL1 + 12 + / + 1152
mmL24 =3
E. < K.
AP()))
We define Cin (1 zá /mL, the coefficient of the linear term. Utilizing the assumption that the consensus error is locally bounded (E, S 1), we have E < E, for k: 2 This allows us to upper bound the higher-order polynomial terms using the aggregated coefficient Lighh defined in the proposition:
P(E,) < C.. Liigh
Therefore, suffices to ensure Cia + Eigh < K.
Step (B). To satisfy the inequality above, we employ a budget splitting strategy, requiring both the linear and quadratic components be bounded by half of the descent budget K/2. This yields two separate constraints on E::
K < 20
K
K
K
E <
2 - E,
and
Enigh =2 <
- E
< V 2Enigh
36

---

Published as a conference paper at ICLR 2026
Consequently, E < min 20mm
2iini }.
then ( < 0 holds.
Finally, invoking Corollary D.2, which bounds the consensus enror as E S we see that Equation (11) ensures E, falls within the safery region. This completes the proof.
24(1 ا
(d2 o2).
1
0
37


Reviews:
[{"review_id": "JJmyw7Kzla", "reviewer": "Reviewer_EC7X", "summary": "This paper looks at how to make communication more efficient in decentralized distributed training, where there is no central node and the network connections are random. The authors study when communication should happen during training, checking if communicating more at certain times helps the final shared model perform better.\n\nIn their experiments, they allow heavy communication only during a specific part of training, while at other times, each node talks to just one random neighbor. The results show that increasing communication near the end of training works best.\n\nThe paper also includes a theoretical analysis that proves their method converges. It further shows that training can be faster when certain gradient patterns are present.", "questions": "1 - Can you align your analysis with the dynamic averaging approach in [1], which claims that communication after convergence is not needed, motivating it also through the properties of loss surface; in particular, when models are already in one convex hull their performance will be basically same and aggregation is not needed anymore (I understand that there the setup is federated learning, but I would guess that this is the general understanding of how individually trained models behave).\n\n[1] Kamp, Michael, et al. \"Efficient decentralized deep learning by dynamic model averaging.\" Joint European conference on machine learning and knowledge discovery in databases. Cham: Springer International Publishing, 2018.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes an interesting analysis of mergability of models trained on different (heterogeneous) subsets of data. Theoretical analysis shows that it depends on the sharpness of the merging point, which coincides with the usual intuition of how loss surface behaves. Moreover, proposed approach helps to reduce not needed communication to minimum.", "weaknesses": "Investigation of the communication after convergence is missing - in particular Figure2 demonstrates that continued individual training after full merge leads to reduction of the performance, the question is does it ever stop? Can the model converge to a state that neither merging, no continuing training changes the resulting performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5COLU23lrn", "reviewer": "Reviewer_6EES", "summary": "This paper showed that the performance of decentralized SGD can be improved by performing a single global merging at the end of the training. Then, this paper analyzed the convergence rate and provide the intuition of why the single global merging can improve the performance.", "questions": "See the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* It is an interesting finding that single global merging at the final training phase can significantly improve the accuracy of Decentralized SGD.\n* This paper comprehensively evaluated how the effect of a single global merging affects the performance, depending on when we did it, showing that a single global merging is helpful at the final training phase.", "weaknesses": "* The experimental findings are interesting, but the theoretical explanation provided by this paper is difficult to follow.\n* This paper claims that the convergence rate of Decentralized SGD with a single global merging matches the rate of Parallel SGD in Remark 2, but it is an overclaim. The convergence rate shown in Theorem 1 is not closed-form, which still depends on $\\Xi_t$. Using Assumption 4, this term can disappear, but it is not fair to compare this rate and the convergence rate of Decentralized SGD since these analysis use the different assumptions.\n* This paper claims that $A_t$ would be negative under Assumption 4 and the condition shown in Proposition 3 holds, but it is still unclear to the reviewer what Assumption 4 implies. Can the authors provide several functions that satisfy Assumption 4 or an intuition to understand Assumption 4?\n\nMinor Comments:\n* The statement of theorems and propositions must explicitly mention which assumptions are used. For instance, Theorem 1 and Propositions 2 and 3 seem to use Assumption 1, but it is not explicitly mentioned in their statements. Additionally, Proposition 2 seems to use Assumption 2, but it is not mentioned in the statement.\n* This paper cited the same paper multiple times in the reference. (Kong 2021a and Kong 2021b)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H8TEU7LYvT", "reviewer": "Reviewer_pdQg", "summary": "This paper studies how communication should be scheduled over time to improve global generalization. The authors show that a single global merging in decentralized training can achieve performance close to that of federated learning, and that limited but non-zero communication helps preserve the mergeability of local models throughout training.\n\nIn addition, the authors provide a theoretical explanation for why limited but non-zero communication can ensure mergeability, and why communication should be concentrated in the later stages of training. The investigated problem is novel, and the theoretical analysis appears sound. The findings may further inspire future research on communication scheduling and allocation.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper studies how communication should be scheduled over time to improve global generalization. The authors show that a single global merging in decentralized training can achieve performance close to that of federated learning, and that limited but non-zero communication helps preserve the mergeability of local models throughout training.\n\nIn addition, the authors provide a theoretical explanation for why limited but non-zero communication can ensure mergeability, and why communication should be concentrated in the later stages of training. The investigated problem is novel, and the theoretical analysis appears sound. The findings may further inspire future research on communication scheduling and allocation.", "weaknesses": "Comments:\n\n1.\tThe paper does not discuss the impact of network topology in decentralized learning. In realistic decentralized settings, multiple agents are connected through an underlying topology, which may prevent a single global merging due to communication constraints. Under a general topology, if agents communicate randomly according to the topology and perform full communication within that topology—rather than global merging— in the final stage, would the main conclusions still hold?\n\n2.\tRegarding Proposition 3, could the authors provide more intuition on why communication is needed when the gradient norm is small? Intuitively, when the gradient norm is large, more communication should be beneficial to accelerate optimization; when the gradient norm is small—indicating that the model is approaching a stationary point—less communication should suffice. This seems to contradict the paper’s conclusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VjFAWSmdTO", "reviewer": "Reviewer_jbK5", "summary": "The primary goal of the paper is to investigate optimal strategies for scheduling communication in decentralized learning, specifically focusing on when and how often devices should synchronize during training. The authors' empirical findings reveal that allocating most of the communication budget toward the end of training, particularly by performing a single, fully connected global merging at the final step, can achieve generalization performance comparable to traditional server-based (centralized) training, even under severe communication constraints and high data heterogeneity. On the theoretical side, the paper demonstrates that the globally merged model resulting from decentralized stochastic gradient descent (SGD) can converge as quickly as centralized mini-batch SGD, and provides new insights into why limited but well-timed communication is sufficient for effective model merging and generalization in decentralized settings", "questions": "Suggestions:\n1. Can the authors add server-side training (centralized SGD) curve to the figure 3c? It would be insightful to add the centralized baseline there.\n2. \"Surprisingly, we uncover that fully connected communication at the final step. Low communication throughout training preserves the mergeability of local models.\" -- This phenomenon has already been shown in the literature [1] as Skew-Compensated Sparse Push in [1] but the current paper has no reference to [1]\n\nReferences:\n1. Aketi, S. A., Singh, A., & Rabaey, J. (2021). Sparse-push: Communication-& energy-efficient decentralized distributed learning over directed & time-varying graphs with non-iid datasets. arXiv preprint arXiv:2102.05715.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles the core issues of communication bottlenecks and data heterogeneity in decentralized learning.\n2. The paper investigates how to optimally schedule communication over time and links this to model merging.\n3. The paper provides theoretical guarantees that the merged decentralized model can match or outperform centralized mini-batch SGD.\n4. The paper shows that local models remain mergeable even with limited but nonzero communication.\n5. Demonstrates findings with robust experiments on Tiny ImageNet, CIFAR-100 and CIFAR-10 datasets with up to 32 agents.", "weaknesses": "The decentralized setup in this paper assumes that all-reduce communication is impractical, so presenting a single global merging as an all-reduce operation may not be appropriate. Instead, a more suitable alternative is to perform simple gossip averaging (without additional SGD updates) for n iterations (where n is the number of nodes) when using a doubly stochastic graph structure. However, this shouldn't impact any of the results presented in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)