Achieving Global Flatness in Decentralized Learning with Heterogeneous Data
Authors: Sakshi Choudhary
Sai Aparna Aketi
Kaushik Roy
Venue: Accepted by TMLR ()

Abstract: Decentralized training enables peer-to-peer on-device learning without relying on a central server, but suffers from degraded generalization performance under heterogeneous data distributions due to local overfitting. One strategy to mitigate this is to seek flatter loss landscapes during local optimization at each client. However, with extreme data heterogeneity, local objectives may diverge from the global one, yielding local flatness rather than true global flatness. To mitigate this challenge, we introduce GFlat, a novel decentralized algorithm that enables each client to estimate and incorporate an approximation of the global update direction while seeking a flatter loss landscape locally.
This lightweight strategy allows each client to directly contribute to global flatness without requiring additional communication or centralized coordination.
We theoretically analyze the convergence properties of GFlat and validate its performance through extensive experiments across a range of datasets, model architectures, and communication topologies. GFlat consistently improves generalization in non-IID data settings and achieves up to 6.75\% higher test accuracy compared to state-of-the-art decentralized methods.

Published in Transactions on Machine Learning Research (01/2026)
Achieving Global Flatness in Decentralized Learning with Heterogeneous Data
Sakshi Choudhary
choudh28@purdue.co
Department of Electrical and Computer Engineering
Purdue Universily
Sai Aparna Aketi
aketiayarraa@gmail..cmm com
Central Applied Science
Meta
Kaushik Roy
kaustikk@purduc edu
Department of Electrical and Computer Engineering
Purdue University
Reviewed on OpenReview: /ttps://opennee wpwp///veeeee....e.m net/ forum? 8G32T4RLbX
Abstract
Decentralized training enables poer-to-peer on-device learning without relying tral server, but suffers from degraded generalization performance under heterogeneous data dist.ributions due to local overfitting. One strategy to mitigale this is to seek flatter loss landscapes during local optimization at each client. However, with extreme data hetero- gencity, local objectives may diverge from the global onc, yiclding local flatnoss rather than true global flatness. To mitigate this challenge, we introduce GFlat, a novel recentralized algorithm that enables each client estimate and incorporate an approximation of the global update direction while seeking a flatter loss landscape locally. This lightweight strat- egy allows each client directly contribute to global flatness without requiring additional communication or centralized coordination. We theoretically analyze the convergence prop- erties of GFlat and validate its performance through extensive experiments across range of datasets, model architectures, and communication topologies. GFlat consistently improves generalization in non-IID data settings and achieves up to 6.75% higher test accuracy com- pared to statc-of-the-art decentralized methods.
1 Introduction
Modern deep neural networks are fueled by massive amounts of training data generated edge devices such smartphones, Internet-of-Thiee Things (IoT) sensors, drones, etc. Traditionally, these models are trained in centralized setup by aggregating data from edge devices to the cloud, raising privacy concerns and incurring significant communication costs. To address these challenges, there has been considerable effort towards developing on-device learning algorithms Konečný et. al. (2016); Agarwal & Duchi (011)) Among these, federated learning has emerged as a popular paradigm in which numerous clients collaboratively train a global model by sharing locally computed updates with a central server. However, such setup introduces a single point of failure and demands high network bandwidth for client-server communication Assran al. (2019). These potential issues have spurred interest in decentralized learning, where clients connected via sparse topology and train through peer-to-peer communication without relying server. Decentralized Parallel Stochastic Gradient Descent (DPSGD) Lian et al. (0017) combines SGD with gossip averaging Xiao
& Boyd 2004) and demonstrates that decentralized algorithms can achieve convergence rates comparable
The Porchc implementation can found httpp::/ggttabb.cc///shhhooo...

---

Published in Transactions on Machine Learning Research (01/2026)
(a) SADDLe
(b) GFlat
Figure 1: Loss landscape visualization for (a) SADDLe, which uses only local perturbations, and (b) GFlat, our proposed algorithm that combines local and approximate global perturbations achieve flatter loss land- scapes in decentralized learning. Results are shown for the CIFAR-10 dataset with ResNet-20, distributed across 10 clients with extreme data. heterogeneity.
to their centralized counterparts. Recently, decentralized training has shown promisc in harnessing globally distributed compute resources to train large language models (LLMs) Jaghouar et al. (0024).
To achieve ttate-of-the-art performance, most existing decentralized learning algorithms assume data to independently and identically distributed (IID) across clients Lian et al.] (2017). However, this assumption often helerogeneous
rarely holds in real-world scenarios, where data distributions
non-IID [siih et al.)
(020bb. Understanding and mitigating the impact of non-lll) data in peer-to-peer decentralized learning
remains active area of research Lin et al. (2021): b Aketi et al. 2023b) Esfandiari et al. (2021)
Aketi et al.
(2023a): Tang et al. (2018): Choudhary et al. (0225 Koloskova et al.
2021 Pu & Nedić (2018) 8) Takezawa
et al. 2023 ; Vogels et al. (2021). Some approaches aim to improve model consistency by tracking global
information across clients Lin et al. 20021); Aketi et al. 2002a Koloskova et al. (2021) ; Takezawa et al.
cross-gradient gradients through additional communication (2021). Recent methods attribute poor performance under non- ITD settings to local overfitting and propose to seek flatter loss landscapes at each client indirectly improve global model generalization Choudhary et al. 2025); Shi et al. (2023). However, locally flat loss landscapes do not necessarily imply global flatness, which is ultimately what matters for global model generalization and overall performance.
(2023), while others enhance local gradients by sharing cross
rounds Aketi et al. (2023b); Esfandiari al.
In this work, we alleviate the discrepancy between the flatness of the local and global objective by utilizing global information while seeking flatter loss landscapes at each client. Unlike prior approaches that rely solely on locally computed perturbations estimate sharpness (Choudhary et al. (2025): [hh et al. (2023), propose GFlat, which incorporates an approximation of the global update direction without incurring any computational or communication overhead. Specifically, each client approximates the global perturbation by computing the difference between its current and previous model parameters. This simple yet effective strategy enables clients to estimate global sharpness, resulting in a globally flatter loss landscape compared to relying on local perturbations alone, as illustrated in Figure [ To quantify this improvement, we track the ratio of the largest the fifth largest. Hessian eigenvalue (Amar/ /As), a known proxy for loss curvature Foret et al. 22021) Jastrzebski al. (2020); Golmant et al. (2018). across training epochs. As shown in Figure 2 GFlat consistently achieves flatter global loss surface compared to SADDLe [Chodhaayy et al. 2025)) which uses only local perturbations. Since GFlat modifies only the local optimization step, it be scamlessly integrated into most decentralized lcarning frameworks without altering their communication or aggregation prolocols. To demonstrate this, we also present Q-GFlat, which integrates a Quasi-Global Momentum (QGM) buffer Lin et al. (202I) to further boost performance under data heterogeneity. We present a detailed convergence analysis showing that GFlat achieves convergence rates consistent with state- of- the-art decentralized algorithms. Comprehensive experiments across multiple datasets, models, and graph topologies validate the effectiveness of our approach.
2

---

Published in Transactions on Machine Learning Research (01/2026)
24.6
17.2
S400
60
20
40
38.8
10
10
6.5
Epoch:50
Epoch:100
Epoch:150
Epoch:200
Figure 2: Ratio largest to 5th largest eigenvalue (Amax/ As)) of the Hessian the global averaged model at different stages of training for SADDLe and GFlat with extreme data heterogeneity.
In sunmarr, we make the following contributions:
We introduce GFlat, decentralized learning algorithm that achieves global flatness under hetero- geneous data by injecting locally approximated global information into each client's optimization process, thereby enhancing generalization without extensive compute or communication overhead.
We provide a thcoretical analysis establishing convergence of GFlat a first-order stationary point.
Through extensive experiments various datasets, model architectures, and graph topologies, demonstrate that GFlat results in up to 6.75% better test accuracy compared to current state-of- the-art techniques with varying degrees of data heterogeneity.
2
Related Work
2.1
Data Heterogeneity in Decentralized Learning
Several algorithms have been proposed to address the challenge of non-IID data in decentralized learning Lin
lt a..
2021 0 Aketi et al. (2023b Esfandiari et al.
(2021) b
Aketi et al. (0023a); Tang et al. (2018) Choudhary
et al. (2025): : Koloskova el al. (2021) : Vogels et al.
[2021 Aketi el al. (2024). Tracking mechanisms such as
Gradient Tracking Koloskova al. (2021): Pu Nedić 2018), Momentum Tracking Takezawa al. (2023).
and Gradient Update Tracking Aketi et al. (2023a) track global gradients or model updates minimize the
variation in local gradients across all clients. Cross-Gradienl Aggregation (CGA) Esfandiari el al. 2021 and
2023b) exchange cross-gradients to align local updates between clients. However, most of the above mentioned techniques require an additional communication round. Without incurring any communication overhead, another way to improve performance with non-IID data is to synchronize the momentum buffer at each client through Quasi-Global Moment um (QGM) Lin et al. (2021). However, in the presence of extreme heterogeneity, QGM does not lead to considerable performance improvements Aketi et al. (2022). D2 Tang et. al. (2018) is shown to be agnostic to data heterogeneity, the connectivity between clients. In different vein, SADDLe Choudhary et al. (2025) replaces the local SGD optimizer with Sharpnoss-Aware Minimization (SAM) Foret et al. (2021) to consistently seek locally flat loss landscapes. Although SADDLe leads to impressive improvements in test accuracy, influences global flatness an indirect manner (refer to Figure thereby leaving room for further improvements.
Neighborhood Gradient Mean Aketi et al.
but its convergence requires specific constraints
2.2 Sharpness-Auus Aware Minimization
To improve model generalization, the authors in Forct et al. (2021 proposed Sharpness-Aware Minimization (SAM), an optimizer that simultaneously minimizes loss value as well as sharpness during training, Note that the connection between a flatter loss landscape and better generalization has been a well-studied phenomenon in deep lcarning Keskar compute-efficienss of SAM, several variants have been proposed in the literature Kwon et al. (2021); Liu
al. (2017); Izmailov et al.] (2018). To further improve the performance and
et al. (2022 Du et. al. (2022 Zhao et al. (2022); Mi et al. (2022); Li et. al. (2024) Wu et. al. (2024]
3

---

Published in Transactions on Machine Learning Research (01/2026)
Luo et al. (024)). Another line of research strives to shed light on the theoretical understanding of SAM
by studying its convergence properties Andriushchenko & Flammarion (2022) Si & Yun (2023); Zhang et al. (2024] Oikonomou & Loizou (2025);
Khanh et al. (2024). Furthermore, SAM has been shown to
improve generalization in federated learning sottings Dai ct al. (2023); Qu et al. (022) ; Caldarola et al.
(2022): sharpness docentralized learning remains relatively unexplored. recent theoretical result establishes that
5: Sun el al. (2023 ; Fan et al. p2024;; Lee & Yoon (2024); Caldarola et al. (2025). However, the role of
Decentralized SGD is asymptotically equivalent to average-direction SAM, suggesting deeper connection
between sharpness and decentralized optimization Zhu et al. (2023). Some recent methods have applied SAM
directly to encourage local flatness in decentralized settings Choudhary et al. [2025): Chen et al. (2024) Shi et al. 2023), yet they do not explicitly consider the mismatch between local and global sharpness. To the best of our knowledge, this is the first work to explicitly identify and address the flatness discrepancy between local and global objectives in decentralized learning with heterogeneous data.
3 Background
In this section, we describe the decentralized peer-to- peer learning setup, the flatness- optimizer SAM Forct al. 2021), and discuss its implications towards improving gencralization under data heterogeneity.
In reeeed learning, the goal is to learn a global model by aggregating models trained on locally available data at clients connected in a sparse graph topology modeled as G = ([n]. W), where W is the mixing matrix signifying the graph's 's connectivity. We assume that G is strongly connected i..,, there is a path bet ween each pair of clients Lian el al.] [2017]; Lin et al. (202T). Each entry Way in W implies the effect of client j on client i, and Way = 0 indicates that the and are not connected directly. The knowledge extracted from such private data is shared among peers to minimize the global loss function f(x):
min f(x) = Σ [(x) XERd
Here, fx() is the local loss function at client i. This optimization problem is tackled by combining Stochastic Gradient Descent (SGD) with consensus-based gossip averaging Xiao & Boyd (2004).
Traditional decentralized algorithms like Decentralized Parallel Stochastic Gradient Descent (DPSGD) Lian et al. (0177) assume the data across clients to be distributed in an independent and identical manner (i.e., IID). In particular, each client i in DPSGD maintains model parameters x/, computes local gradient g through SGD over data B Di, and incorporates neighborhood information shown in the following update rule:
Σ JEN())
wiyx
ngj; g) = VVV[((x),B)
(2)
In this work, we focus on non-IID ooo-IID//ettroogeeeous data in the form of skewed label partition, which is more closely aligned with practical learning sccnarios Hsich et al. (2020b). DPSGD performs poorly with non-IID data due to model overfitting and huge variations in local gradients across clients.
An effective approach to improving decentralized learning under non-llI data is to encourage flatter loss landscapes at cach client. Prior work has established a strong correlation between model generalization and the curvature of the loss landscape, i.e., models converging to flatter minima lend to generalize better Keskar et al. (2017); Izmailov et al. (0018) Inspired by this, SADDLe Choudhary et al. (2025) utilizes a flatness-seeking optimizer called Sharpness- Aware Minimization (SAM) at each client. Originally proposed for centralized learning, SAM enhances generalization by minimizing both the loss value and its sharpness through gradient perturbations Foret. et al. (2021). SAM aims to solve the following optimization problem:
min (x) max f(x + { xE2
where p the perturbation radius, and ||-I| denotes the L2-norm. The perturbation radius p controls the size of the neighborhood over which the loss is maximized, guiding the optimizer to converge to flatter regions.
4

---

Published in Transactions on Machine Learning Research (01/2026)
24.6
17.2
60
60
15
Goba
20
Amaad A5
40
10
7.36
10
20
20
1.2
1.3
3.7
Epoch:50
Epoch:100
Epoch:150
0-
Epoch:200
Figure 3: Ratio of largest to 5th largest eigenvalue (Amaz/Ag As) of the Hessian of the local and the global averaged model at different stages of training for SADDLe.
In practice, this min-max objective is approximated via first-order Taylor expansion of f, resulting in the perturbation & being the direction of the gradient Foret et al. 22021). The final gradient is computed by minimizing the perturbed loss function (x) as follows:
Vx(()) R VX(x)xxxx far where g [x())
(4)
SADDLe solves the min-max optimization shown in Equation 3 for the local training at cach client. This results in locally flat model, thus indirectly improving the generalization performance of the aggregated global model. However, due to non-III) data, the local objectives can be very different from the global objective (Equation []. Therefore, merely focusing on minimizing local sharpness cannot effectively lead to a globally flat model. To demonstrate this, we present the local and global sharpness for SADDLe across training epochs for CIFAR-10 distributed with extreme data heterogeneity across clients in Figure 3 We use the ratio of the largest fifth-largest cigenvalue of the Hessian (Amaz/Aa /25) as a proxy for sharpness. The lower the Amaz/As A55 ratio, the flatter the loss landscape Jastrzebski et al. (2020); Foret et al. (2021); Golmant et al. (2018). As shown, SADDLe leads to lower Amax/A Às for the local objective, but this does not directly translate to a low Amaz/As for the global model. To rectify this inconsistency, we propose GFlat, a novel sharpness-aware decentralized algorithm that incorporates an approximated global update into the locally computed SAM perturbation, thereby directly minimizing global sharpness and improving performance under severe data heterogeneity.
4 Methodology
As mentioned carlier, goal is minimize the sharpness of the global model in decentralized learning setup. To achieve this, we first define the objective for global sharpness-awer aware minimization in decentralized learning as follows:
f(x f(x +) = E f((x - + $
min xER
max |
Here, = = (TV](X]] is the global perturbation, approximated via gradient ascent step scaled by the pertur- bation radius p. In a peer-to-pect peer learning scenario, data privacy concerns and communication constraints prohibit the computation of global gradient fx(x) to estimate
v((()
Consequently, gradient computations remain local to each client, relying solely on their respective datasets, Utilizing only these local perturbations Icads to flatter local model, but does not necessitate flatter global model (as shown in Figure 3). One way to circumvent this challenge is to estimate the global update direction locally at each client. We propose to approximate the global gradient through model differences across two consccutive training iterations. Note that such an estimation technique has also been employed in Quasi-Global Global Momentum (QGM) to modify the local momentum to reduce the model discrepancy arising due to heterogeneous data Lin et al. (202)) Based this approximation, the perturbation & at each client
5

---

Published in Transactions on Machine Learning Research (01/2026)
is redefined as the combination of local and global perturbations as follows:
p(s
d
(1 s)
g
where d;= =x= x
(6)
g
Algorithm GFlat: Achieving Global Flatness in Decentralized Learning with Heterogeneous Data
Input: Each client initializes model parameters X,, step size 77, perturbation radius P, scaling factor [0, 1], mixing matrix W = [wiyk.se(Lm)) (i) represents neighbors of i including itself.
procedure TRAIN( ) for Vi
1. for 1,2,..... do
2.
g = f.((xx;; B)) for B E D;
3.
d' x; x
4.
ç = p(s. p( + (1 8) |R
di
5.
E fx( + B))
(1+1/2) = x 78!
6.
SENDRECEIVE(X* (1+1/2))
7.
x E jEA x (1+1/2)
8.
9.
end
return x- E
Here, is scaling factor that defines the emphasis placed global vs local perturbation. Intuitively, in the presence of extreme data heterogeneity, more value should be given to the global perturbation d'/ldd]],, the local gradients overfit the locally available data. Similarly, for milder forms of non-IIDness, it may suffice to emphasize the local perturbation / s//llg/fl over its global counterpart. We observe this in our experiments, where & remains closer to for extreme heterogencity (i.c., a = 0.001). We provide additional discussion on local and global flatness discrepancy Appendix
Algorithm D presents the pseudocode of GFlat. At each training iteration t, GFlat computes the stochastic gradient g at each client i. Then it estimates the global perturbation through model differences (x[-1 -x() as shown in line 3, Algorithm 1 The SAM perturbation & is calculated as weighted combination of local and global perturbations (line 4, Algorithm D All clients calculate g for the perturbed model (xf + {
to perform local updates (line 6). The clients then exchange models with their peers and perform gossip averaging (line 8) Lian et al. (0017) This continues for a predefined set T iterations, and we evaluate the consensus model x Lian et al. (2017); Aketi et al. (2023a). Note that computing d, to estimate the global perturbation requires minimal compute, incurs no communication overhead, and (()) memory each client, where m denotes the number of trainable model parameters. In essence, GFlat approximates the global update direction and incorporates this in local perturbation at each client to target global sharpness without any communication overhcad.
As GFlat modifies the local optimizer, it is effectively complementary to existing decentralized algorithms for tackling data heterogeneity and can be used in synergy with them. To that effect, we present a version of GFlat termed Q-GFlat, which incorporates a Quasi-Globa Momentum (QGM) buffer Lin et al.) (2021) further reduce the model inconsistency. While QGM mimics the global update direction through a modified local momentum, GFlat focuses seeking a flatter minimum for the global model by injecting
global information in the local perturbation (i.e,, ascent step) for the SAM optimizer. Together, these techniques greatly improve the performance in the presence of non-lID data. Please refer to Appendix for implementation details.
5 Convergence Rate Analysis
This section provides the convergence analysis for the proposed GFlat algorithm for a general non-convex loss objective. We state the following standard assumptions:

---

Published in Transactions on Machine Learning Research (01/2026)
Assumption Lipschitz Gradients: Each function fi() is L-smooth ie., (V Vf((i))l s Llly
x|| A x.
Assumption 2 Bounded Variance: The variance of the stochastic gradients is assumed to be bounded.
There exist constants a and d such that
Vs((x;
V
EBDD,
S Vi
(7)
ffi(x; B)] (x)
VV((x) fx()) 0
(8)
Assumption Doubly Stochastic Miring Matria: W is real doubly stochastic matrix with Å (W)
and max (oax())2(WW) IA,, S < 1, where (W) the 7th largest eigenvalue of W and À is constant.
These assumptions align with those used in existing decentralized learning algorithms Lian el. al. 2011
Aketi et al. 2023a b): Esfandiari et al. (2021: :: Choudhary et al. (2024] Lin et al. (2021). Assumption 2
introduces a slight variation of the standard stochastic variance bound (Equation motivated by recent
works exploring flatter loss landscapes in federated learning settings (detailed discussion in Appendix
et al. 2022;; Lee Yoon (2024); Fan et al. (0024). Theorem presents the convergence for our proposed
GFlat algorithm (proof in Appendix B.2).
Theorem Given Assumptions 1-3, let the learning rate satisfy S for all T 2 1, we have:
Then,
T-1
E0
V (x*) (x*)
v 2 (f(x)) f*) + s)² 7/T
2
nL
L2
12n/
+
1=0
(1 (1- vX22
n
(1
vxj2
2n² L2
0²12
2(11 2(1-s)² + +3522+ +3
(1 vA22
(9)
where, 6 maxT maxisign |ब. direction for the GFlat perturbalion 4
v f(x)) [VX(*]]| denoles the worst case error in approzimaling the global update
The result of Theorem 1 shows that the averaged gradient of the consensus (averaged) model x is upper- bounded by the sub-optimality gap f(x²) - f*, the stochastic variance global variance ó due to data heterogencity, the perturbation radius p and the approximation error e. Further, we present a corollary to demonstrate the convergence rale of GFlat. in terms of training iterations Please refer to Appendix B.4 for the detailed proof.
Corollary 2 Suppose that the step size satisfies 17 o( ((V²) and the perturbation radius p = 0 (V+)
For sufficiently large T we have,
[E( 5 [(x9) (1 (1-
82
2 T
(10)
VnT
T3/2
The dominant term in Corollary is (1ss(11))) (1/vnT),, which determines the overall convergence rate of GFlat. This matches the standard convergence behavior observed in decentralized algorithms, as established in existing
literature Lian et al. 2017); Lin et al. (2021) Aketi et al. 2023a Esfandiari et al. (2021).
Remark A closer inspection of Equation 10 reveals that the influence of stochastic variance (a) is
modulated by the factor (1 (1-s). s). In essence, for higher s, GFlat can speed up the convergence by alleviating the impact of stochastic variance. As result, higher values of (iee. closer to 1) lead to reduced variance
7

---

Published in Transactions on Machine Learning Research (01/2026)
contribution, thereby accelerating convergence. Through our experiments, observe that GFlat achieves optimal performance for 0.5 <= <= corroborating this theoretical insight.
Remark 2. The approximation error in estimating the global perturbation (e) appears as a higher-order term in Equation 10 scaled by 1/T, and thus its impact on convergence diminishes as training progresses. Consequently, we observe performance gains reported in Tables 1 even when using an approximation of global perturbation (i.e., de in Equation 6
6 Experiments
6.1 Experimental Setup
We conduct experiments on diverse datasets, model architectures, graph topologies, and sizes. The analysis is presented on - (a) datasets: CIFAR-10, CIFAR-100, Imagenette Husain (0018) and ImageNet Deng et al.
22099; (b) models: ResNet-20, ResNet-18 and MobileNet-v2, (c) graph topologies: undirected ring with peers/client and torus with 4 peers/client with uniform mixing matrix (Figure 7. and (d) graph sizes:
40 clients. We generate disjoint non-llI data partitions across clients using Dirichlet distribution [su et al. (219)), where the class proportions for cach client are drawn from Dir(a). A smaller concentration parameter yields more skewed class distributions, where each client. receives data from fewer dominant classes, resulting in higher statistical heterogeneity (see Figure G The partitions are fixed at initialization and remain unchanged during training. We compare GFlat with DPSGD Lian et al. (2017) and SADDLe Choudhary et al. (2025), and the Q-GFlat variant with QGM Lin et al. [2021) and Q-SADDLe Choudhary et al. (022)). We report the test accuracy the consensus model x averaged over three randomly chosen
seeds. Please refer to Appendix D.4 for details related to training hyperparameters.
6.2 Results
Table 1: Test accuracy of various decentralized algorithms evaluated on CIFAR-10 and Imagenette dis- tributed with different degrees of heterogeneity for various models over a ring lopology. We also include results over the IID baseline, which serves as an upper bound on performance.
CIFAR-10 (ResNet20)
Imagenette (Mobilenet-v2)
Clients
Method
a 0.01
0.001
Q 0.01
0.001
DPSGD (ITD)
90.46
0.33
75.15 0.42
DPSGD
49.17 + 17.38
40.74 + 2.62
40.11 + 4.85
34.50 + 5,88
10
SADDLe
64.58 1 5.63
61.30 1 0.79
45.95 + 3.03
42.37 1 4.30
GFlal (ours)
70.12 + 0.68
62.29 + 0.66
48.91 + 2.51
48.07 + 1.01
DPSGD (IID)
89.46 + 0.02
73.25 + 0.49
DPSGD
40.49 3.06
36.13 5.67
36.45 0.46
30.96 1.55
20
SADDLe
52.14 2.02
47.06 2.25
39.18 1.04
37.72 1.16
GFlat (ours)
56.21 + 1.33
51.76 + 0.68
39.95 + 0.51
39.23 + 1.26
Performance Comparison: Table 1 presents results for GFlat across CIFAR-10 and Imagenette under two different levels of heterogeneity (with lower a indicating higher non-IIDness). As shown, GFlat achieves 1- 5.5% improvement in accuracy over SADDLe, highlighting the benefits of incorporating global perturbation into the local SAM step. Similarly, on Imagenette, GFlat results in up to 5.7% better accuracy compared SADDLe Choudhary et al. (2025). We demonstrate the efficacy of Q-GFlat in Table 2 comparing it against QGM and Q-SADDLe across 10-40 clients connected in ring topology. On CIFAR-10 and CIFAR- 100, Q-GFlat outperforms Q-SADDLe by ~0.6% and ~2.6% respectively. We present additional results Imagenette in Table 3] where Q-GFlat yields an average improvement of 3.8% for 10-20 clients in ring topology. To study the impact of graph topology, results on 20-client torus topology are reported in Table g Both GFlat and Q-GFlat demonstrate significant gains over their respective baselines, SADDLe and Q-SADDLe, achieving an average improvement of 2.2% for a = 0.01, 0.001.

---

Published in Transactions on Machine Learning Research (01/2026)
Test accuracy of various decentralized algorithms evaluated on CIFAR-10 and CIFAR-100 100 dis- tributed with different degrees of heterogeneity over ResNet-20 over ring topology.
Table 2:
CIFAR-I0
CIFAR- 100
Clients
Method
0.01
0.001
0.01
0.001
QGM
77.41 = 8.00
79.48 2.76
48.06 + 4.36
44.16 6.71
Q-SADDLe
87.72 = 1.59
86.33 + 0.24
58.06 + 0.68
56.76 + 0.86
10
Q-GFlat (ours)
88.02 + 1.04
86.73 = 0.20
58.98 + 0.33
58.12 + 0.44
QGM
72.20 = 0.77
62.48 + 8,56
45.23 = 3.26
44.48 + 4.53
Q-SADDLe
84.17 + 0.73
82.81 0.89
52.59 + 0.48
48.20 + 0.93
20
Q-GPlat (ours)
84.61 + 0.70
83.04 0.41
54.34 + 0.65
53.45 + 0.62
QGM
70.46 + 1.14
60.86 0.98
40.15 + 0.90
38.73 1.47
Q-SADDLe
77.49 + 0.83
73.54 1 2.04
43.25 1 1.71
41.99 1.27
40
Q-GPlat (ours)
77.97 + 1.01
75.34 + 1.13
46.37 + 0.79
45.25 + 0.41
Test accuracy of various decentralized algorithms evaluated on Imagenette distributed with different degrees heterogeneity over MobileNet-v2 over a ring topology.
Table 3:
Imagenette (Mobilenet-v2)
Clients
Method
a = 0.01
0.001
QGM
56.30 4.03
45.82 5.99
Q-SADDLe
62.35 3.64
63.18 1.59
10
Q-GPlat (ours)
64.98 0.96
64.89 = 0.56
QGM
52.52 1 4.88
45.67 1 3.10
Q-SADDLe
53.64 + 4.60
53.99 + 3.45
20
Q-GFlat (ours)
60.39 2.40
58.08 = 1.50
Table 4: Test accuracy various docentralized algorithms evaluated on Imagenette distributed with different degrees of heterogeneity torus topology.
Imagenette (Mobilenet-v2)
Clients
Method
0.01
0.001
DPSGD
35.75 + 1.91
27.40 + 1.19
SADDLe
40.55 = 4.19
33.23 + 0.27
20
GFlat (ours)
10.11 2.00
34.63 + 0.83
QGM
48.28 5.88
42.46 1 10.75
Q-SADDLe
58.68 + 3.04
54.78 5.81
Q Q-Gllat (ours)
61.98 1.17
59.19 + 0.39
Across all experiments, we observe that GFlat exhibits greater robustness to random seed variations compared to the base- lines, with significantly lower variance in performance. To further emphasize the effectiveness our approach, we con- duct. experiments on TmageNet using the ResNet-18 18 architec- ture, distributed across 10 clients with three different levels of data heterogeneity. As shown in Table GFlat outperforms SADDLe by 3% on average.
72-
Test Accuracy
69
66
63
0.2
1.0
Scaling Factor (s)
Impact of Scaling Factor: To evaluate the impact of the scaling factor we conduct an ablation study using the CIFAR-10 dataset trained with a ResNet-20 architecture dis- tributed across clients under a highly non-IID setting (a 0.01). As shown in Figure (0.5, 0.7, 13 consistently yields higher test accuracy compared to (0, 0.,,, suggesting that
Figure 5: Impact of scaling factor on per- formance for CIFAR-10 distributed across 10 clients with a 0.01.
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5:
Test accuracy of various decentralized algorithms evaluated on Imagenet distributed with different
degrees of heterogeneity (a) over a ring topology.
Imagenet (ResNet-18)
Clients
Method
0.1
0.01
0.001
DPSGD (IID)
66.73
DPSGD
50.00
45.49
46.77
10
SADDLe
54.63
51.28
51.26
GFlat (ours)
57.54
54.49
52.40
85
89
Test Accuracy
75-
Test Accuracy
65
Q GFlat
55-
GFlat
83
Q-SADDLe
SADDLe
DPSGD (IID)
45-
80
0.001
0.010
1.100
1.000
0.001
0.010
100
1.000
Degree heterogeneity (a)
Degree of heterogeneity (a)
(a) SADDLe GFlat
(b) Q-SADLL vs Q-GFlat
Figure 4: Test accuracy for the CIFAR-10 dataset trained on ResNet-20 20 architecture distributed with varying degrees of data heterogeneity over 20 clients.
larger emphasis on the global direction improves performance. This aligns with the insights gained from our theoretical analysis (Remark 1).
Robustness to Degrees of Data Heterogeneity: We perform an ablation study to analyze the impact of data heterogeneity on the performance of GFlat. Figure g presents the test accuracy of GFlat and Q- GFlat across different values of a, where lower a indicates higher non-IIDness. As observed, both variants consistently outperform their respective baselines. Notably, Q-GFlat and Q-SADDLe even surpass the performance of DPSGD under IID data for = 1, highlighting the effoctiveness of our approach.
7 Limitations and Future Work
GFlat introduces a scaling factor s to balance the relative emphasis on local versus global perturbations. We keep s fixed throughout the training and tune it as a hyperparameter. However, since the alignment between local and global models can vary over training iterations, could be made adaptive. For instance, clients may benefit. from a higher when the divergence between local and gossip-averaged models is large, and lower value when they are well aligned. We leave the design and evaluation of such adaptive strategy to future work. Another promising direction for future work is the integration of computationally efficient variants of SAM Du et al. (002)) Liu et al. (002) into our framework, followed by an analysis of their impact on performance.
8 Conclusion
In this work, we highlight that under severe data hetcrogencity in decentralized training, locally flat loss landscapes at individual clients do not necessarily translate into flainess of the global loss objective. To address this mismatch, we propose GFlat, a novel algorithm that achieves globally flat loss landscapes by incorporating an approximated global direction into the local optimization process. We provide a theoret- ical analysis establish that GFlat achieves convergence rate of (1/VVnT), similar to state-of the-art decentralized algorithms. Furthermore, we validate our approach through extensive experiments conducted
10

---

Published in Transactions on Machine Learning Research (01/2026)
across diverse datasets, model architectures, and graph topologies. Our results show that GFlat improves test accuracy by up to 6.75% over current state-of-the-art decentralized algorithms without any additional communication cost.
9 Acknowledgments
The authors would like to thank Utkarsh Saxena for insightful technical discussions. This work was supported in part by, the Center for the Co-Design of Cognitive Systems (COCOSYS), a DARPA-sponsored JUMP center, the Semiconductor Research Corporation (SRC) and the National Science Foundation (NSF).
11

---

Published in Transactions on Machine Learning Research (01/2026)
References
Alckh Agarwal and John Duchi. Distributed delayed stochastic optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems. volume 24. Curran Associates, Inc., 2011. URI http::
Paper pdf
Sai Aparna Aketi, Sangamesh Kodge, and Kaushik Roy.
Low precision decentralized distributed training over iid and non-iid data. Newral Nelw., 155(C) 460, November 2022. ISSN 0893-6080. doi: 10.1016/ j.nemet.2022.06 03.02.. URL https:///oo..... wrg. 10. 10160/0000001111111111011100./3 .neunet 0.022.08.00
Sai Aparna Aketi, Abolfazl Hashemi, and Kaushik Roy. Global update tracking: A decentralized learning algorithm for heterogeneous data. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advonces in Neural Information Processing Systems, volume 36, pp. 48939 48961. Cur- ran Associates, Inc., 2023a. URL httpps:/pproceedinnnn neur www.ipp.cc/ppppr.....eeeeeeeeeee wwtpapeerfilll/ppppeeeeeeeeeeeee filles/paper2223/5iee 035200a00046111100000000221111-0
Sai Aparna Akeli, Sangamesh Kodge, and Kaushik Roy. Neighborhood gradient mean: An efficient decen- tralized learning method for non-llD data. Thansactions on Mochine Learning Research, 2023b. ISSN 2835-8856. URL https://openreeierr net//forum2dd=wwWaa555...
Sai Aparna Aketi, Sakshi Choudhary, and Kaushik Roy. Averaging rate scheduler for decentralized learning on heterogeneous data. In The Second Tiny Papers Track at ICLR 2024, 2024.
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In International Conference on Machine Learning, pp. 639 668. PMLR, 2022.
Mahmoud Assran, Nicolas Loizon, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning, 2019. URL http::///pppameeeeees.et/trrrrop
Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by sceking flat minima. In European Conference on Computer Vision, pp. 654-672. Springer, 2022.
Debora Caldarola, Pietro Cagnasso, Barbara Caputo, and Marco Ciccone. Beyond local sharpness: Communicationn-fffiieent global sharpness-award aware minimization for federated learning, 2025. URL https:
Simiao Chen, Xiaoge Deng, Dongpo Xu, Tao Sun, and Dongsheng Li. Decentralized stochastic sharpness- aware minimization algorithm. Neural Networks, 176:106325, 2024. ISSN 0893-6080. doi: https://
URL http:///wwcccccccccccccc.c../o/a
508936080240024411
Sakshi Choudhary, Sai Aparna Aketi, Gobinda Saha, and Kaushik Roy. Codec: Commuicationn-fffcien decentralized continual learning. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
URL https://openrrevier .ett/foouumiiddd0000nn..
Sakshi Choudhary, Sai Aparna Aketi, and Kaushik Roy. Saddle: Sharpness-awaan aware decentralized deep learning with heterogeneous data. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), pp. 7720-7730, February 2025.
Rong Dai, Xun Yang, Yan Sun, Li Shen, Xinmei Tian, Meng Wang, and Yongdong Zhang. Fedgamma: Federated learning with global sharpness-awan aware minimization. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-14, 2023. doi: 10.1109/TNNISS.2283..838
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largo-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pottern recognition, pp. 248-255 Iece, 2009.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Jiawei Du, Zhou Daquan, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Newral Information Processing Systems, 2022. URL https://openrevierr wttp://wwww.eettouniiiiiiii wttp/oorurPddd
Yasaman Esfandiari, Sin Yong Tan, Zhanhong Jiang, Aditya Balu, Ethan Herron, Chinmay Hegde, and Soumik Sarkar. Cross-gradient aggrogation for decentralized learning from non-iid data. In International Conference on Machine Learning, pp. 3036 3046. PMLR, 2021.
Ziqing Fan, Shengchao Hu, Jiangchao Yao, Gang Niu, Ya Zhang, Masashi Sugiyama, and Yanfeng Wang. Lo- cally estimated global perturbations are better than local perturbations for federated sharpness-aware min- imization. In Proceedings of the 41st International Conference on Machine Learning, ICML 24. JMLR.org, 2024.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL ittts://openreviee: wttp://ooooommmmddddaaaooeppee
Noah Golmant, Amir Gholami, Michael Mahoncy, and Joseph Gonzalez. pyforch-hessaa-ciggmtinss cf- ficient pytorch hessian eigendecomposition, October 2018. URL htapss///ggthuub..mmmmaagggggg/
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770- 708, 2016.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B. Gibbons. The non-iid data quagmire of decentralized machine learning. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020a.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip B. Gibbons. The non-iid data quagmire of decentralized machine learning. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020b.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arviv preprint ar arXie:1900 Stc:1909.0638 2019.
Hamel Husain. Imagenette
subset of 10 easily classified classes from the imagenet dataset. https:// https://github.com cww.aad.oooooffmomomo com/fastai/imaaeee /imagenettee, 2018.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av-
eraging weights leads to wider optima and better generalization. CoRR, abs/ abs/1803.05407,, 2018. URL
org. awrg/3bs/1803..o http://aaxx..ogg/bb///B833 05400
Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, Max Ryabinin, and Johannes Hagemann.
Intellect-1 technical report, 2024. URL httpss//aarai..orrrbb//44222.2.. 01152
Stanislaw Jastrzebski, Macicj Szymezak, Stanislav Fort, Devansh Arpit, Jacck Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. arXiv preprinl arXiv:2002.098: 09572,, 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Con- ference on Learning Representations, 2017. URL
Pham Duy Khanh, Hoang- Chau Luong, Boris S. Mordukhovich, and Dat Ba Tran. Fundamental convergence analysis of sharpness-aware minimization, 2024. URL htttpp///aarxv..rrr/a//////..11
Anastasīia Koloskova, Tao Lin, and Scbastian U Stich. An improved analysis of gradient tracking for docentralized machine learning. Advances in Neural Information Processing Systems, 34:11422-111 11435, 2021.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Jakub Konečný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization:
Distributed machine learning for on-device intelligence. CoRR, abs/ abs/1610.02527, 2016. URL http:// arxiv. wwt.voorg/abb/111002255
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 5905-5914. PMT.R,
2021. URL https://proceedngs.cu / wttppp/eeeeee/eeeeeeeee
html
Taehwan Lee and Sung Whan Yoon. Rethinking the flat minima searching in federated learning. In Forty- first International Conference Machine Learning, 2024. URL httpe:///peerreview.. wnt./rrroii.ttfffmmmddd
6TM62kp15c
Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly sharpness-aware minimization. In Proceedings of the JEEE/CVP Conference on Compuler Vision and Pattern Recognition (CVPR), 2024.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
Tao Lin, Sai Pranceth Karimiroddy, Scbastian Stich, and Martin Jaggi. Quasi-global momentum: Accelorat- ing racined deep learning heterogeneous data. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6654-6665. PMLR,
18 24 Jul 2021.
Hanxiao Liu, Andy Brock, Karen Simonyan, and Quoc Lc. Evolving normallzation-activation layers. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 13539-13555 Curran As- sociates, Inc., 2020. URL http:::/ppooceding..neette. httppsscccpppppeeeeeeee ffllss/paper/222/fiiie 00005511B000000030666677007
pdf
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness- aware minimization. In Proceedings of the IEEE/CVT Conference Computer Vision and Pattern Recognition, pp. 12360 12370, 2022.
Haocheng Luo, Tuan Truong, Tung Pham, Mehrtash Harandi, Dinh Phung, and Trung Le. Explicit eigenvalue regularization improves sharpness-awar aware minimization. In The stirty--ighhti Annual Conference on Neural Information Processing Systems, 2024. URL httpss///pperreview wwt.://oouunddddeuuuB.......
Pong Mi, Li Shen, Tianhe Ren, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make sharpness- aware minimization stronger: A sparsified perturbation approach. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
wttp:/ewwwweeeeettoi//ei/e/eeee. aeee:fforrmm1dd8____l wNI6ZBDZ
Dimitris Oikonomou and Nicolas Loizou. Sharpness-aware minimization: General analysis and improved
rates. In The Thirteenth International Conference Learning Representations, 2025. URL https:
Shi Pu and Angelia Nedié. A distributed stochastic gradient tracking method. In 2018 IEEE Conference on Decision and Control (CDC), pp. 963 968, 2018. doi: 10. 2tt9//DCC.201..611.1 8.888878.
Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International conference on machine learning, pp. 18250- 18280. PMLR, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp.
2018.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Yifan Shi, Li Shen, Kang Wei, Yan Sun, Bo Yuan, Xueqian Wang, and Dacheng Tao. Improving the model consistency of decentralized federated learning. In International Conference Machine Learning, pp. 31269 31291. PMLR, 2023.
Dongkuk Si and Chulhee Yun. Practical sharpness-aware minimization cannot converge all the way to optima. In Shirth-seventh Conference on Neural Information Processing Systems, 2023. URL https:
//openreview.o //operreview.ne//forrmmm nett/foruummddmmm.]]......
Yan Sun, Li Shen, Shixiang Chen, Liang Ding, and Dacheng Tao. Dynamic regularized sharpness aware min- imization in federated learning: Approaching global consistency and smooth landscape. In International Conference on Machine Learning, pp. 32991-33013. PMLR, 2023.
Yuki Takezawa, Han Bao, Kenta Niwa, Ryoma Sato, and Makoto Yamada. Momentum tracking: Momentum accelcration for decentralized deep learning on heterogencous data. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL http:///pppeeeeeeeeeeeerrrrrraaa
Hanlin Tang, Xiangru Lian, Ming Yan, Ce Zhang, and Ji Liu. de Decentralized training over decentralized data. In Jennifer Dy and Andreas Krause (ods.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4848 4856. PMLR, 10-15 Jul 2018. URL https://proceeding/ mlr. www.mmrrpeeeeeeee hatp///v//ttang8gg/
Thijs Vogels, Lie He, Anastasiaa Koloskova, Sai Praneeth Karimireddy, Tao Lin, Sebastian U Stich, and Mar- tin Jaggi. Relaysum for decentralized deep learning on heterogeneous data. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances én Newral Information Processing Sys- tems, volume 34, pp. 28004-28015. Curran Associates, Inc., 2021.
Tao Wu, Tie Luo, and Donald C. Wunsch. Cr-sam: curvature regularized sharpness-awaad aware minimization. In Proceedings of the Thirty-Fighth AAAI Conference Artificial Intelligence and Thirty-Sirth Conference on Innovative Applicalions of Artificial Intelligence and Pourleenth Symposium Educational Advances in Artificial Intelligence, /AAT24//AA/24/EAA/TT AAAI Press, 2024. ISBN 178-1-57735-88 doi: 10.1609 1111699/aaiiv3886.6 28431. URL https: http::::///oo.o//11..1 w0t./.........//../........... v3816....... 28431
Lin Xiao and Stephen Boyd. Fast linear iterations for distributed averaging. Systems & Control Letters, 53 (1):65-78 78, 2004.
Hao Yu, Rong Jin, and Sen Yang. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. arXiv preprint arXim:1905.088 038117, 2019.
Yihao Zhang, Hangzhou He, Jingyu Zhu, Huanran Chen, Yifei Wang, and Zeming Wei. On the duality between sharpness-awan aware minimization and adversarial training. In ICML, 2024.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning, pp. 26982-26992. PMLR, 2022.
Tongtian Zhu, Fengxiang He, Kaixuan Chen, Mingli Song, and Dacheng Tao. Decentralized SGD and average-direction SAM are asymptotically equivalent. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 43005 43036. PMLR, 23 29 Jul 2023. URL https://proceeddings mlr. ppesss/v022 v202/22222ee.eeee zhu23e. html
15

---

Published in Transactions on Machine Learning Research (01/2026)
A Appendix
B Theoretical Analysis
B.1 Discussion on Stochastic Variance Assumption
As mentioned Section 5 we assume a slight variation of the following standard stochastic variance bound
presented contemporary decentralized learning works:
Eg-D, [Vf((;:) vx(())] o²
(11)
We rewrite our assumption here (Equation
for ease of understanding:
Vf((x; B) [V[(x;
Vf((x) 2 [V][]]] S Vi
EB-D
(12)
Concretely, instead of bounding the norm of the difference between [x( B) and the true gradient (x for cach client, as shown in Equation 11 we bound the difference their directions Lee Yoon (2024): Qui
jet al. (022). We argue that the bound in Equation 11 may be loose and less informative in practice Qu et al. 2022), particularly during the later stages of training, when the gradient norms are small. In such cases,
small variance in magnitude can lead to significant directional variance, potentially causing instability in optimization. This distinction is crucial, as the convergence of the consensus model depends not only on the magnitude of the gradients at each client but also on their alignment. with the true update [VV] are unit vectors, and thus their euclidean distance can be reasonably bounded by the angle between them.
fxx:x)))) directions. Morcover, (x.BII and f((x))
B.2 Convergence Rate Proof
In this work, we solve the optimization problem of minimizing the global loss function f(x) distributed across clients:
min f(x) XE
Σ [(x), where fi(x) EggED, [sx(x; B() Vi
(13)
We begin by providing upper bound the consensus error, defined as 1 Σ. |x] 11 which quantifies the average deviation of local models x from the global averaged model x at training iteration t. The consensus error reflects the effectiveness of gossip-based averaging in decentralized learning and indicates how closely clients are approaching global agreement.
Lemma Given assumptions 1-3, the distance between the average sequence iterate x and the sequence ilerates x (i.e. the consensus error) is given by
2n
X112
s
1'pp(1 (1(11))3000 38" + s) 2
n
(1
vx)²
(14)
4n²
T
212 + 66L22)
Σ
Blll Σ [ + +1
(1
v
1=0
16

---

Published in Transactions on Machine Learning Research (01/2026)
Proof: We introduce the following notations and the subsequent properties et al. 2023bb; Choudhary et al.
Esfandiari et al. (2021
Aketi
(2024
11
n
gi
8m 82
of
(15)
X
G
8i
H'
( + 1), ff2(x) + E2),., VSN(x, + En)
Note that g represents the local stochastic gradient computed using GFlat as shown in Algorithm (line 5). For all the above matrices, we have:
|
| Σm ||a;1| || where a, is the i-th column the matrix A. Therefore,
X' Q
Σ
x 11
(16)
For each doubly stochastic matrix W, the following properties hold true:
QW = WQ:
(I-Q) w(I
For any integer 2 1, () I (VX)² (ll lle the spectrum norm of matrix)
For n arbitrary real square matrices A, E (1.2....,mh n),
A |
(17)
We now proceed to provide a bound on the consensus error. Since X = X - G we have:
X
(18)
Applying the above equation & times, we have:
= - T=1
E GII-Q)
(19)
22
E
X II Q)
= B²2
Σ
7=0
E
G7 [I - Q)
G
[I
Q
(20)
(1-1- Ell
11
E [1
H 1-1
1/2
[-1
(t-
11-1-
द _
Σ
E
v
7=0
17

---

Published in Transactions on Machine Learning Research (01/2026)
(a) follows from the inequality zy (x² + y²) for any two real numbers 2, y-
A - -----
(b) follows from Er
1-VA
We proceed with finding the bounds for |||| [[|]]
n
[ + (ET: VA((X + V + + Vf((x] + Stree) + Vfi(x; + EFrae)
fx( + + V + ( + - + +III
(21)
(X + - Vi(x + Stree I|I
[ Stree)
VX(X + VEX +
111
Here, Strue P g g- is the true perturbation, where g" = vf(x)) - vd((x) denotes the average d. g' is the crror introduced due to the global update approximation gradient across all clients. c |ل:ل| ||
computed through the model differences de and e = maxy maxisiga 41-
Now we bound I, II, and III:
E(IV (ET: Vx(( (T] & 12p²B
8. ||d[]] ||
d/
+ (1 - 8).
8
d
(1
VA(xF)
|']
||d]]
v((({() Vf((() 1"1 <T2100211 - s)
g
= (1 -
(22)
Now, we proceed to bound IT:
11 (x + +
Vf((F + 11 < L'BBIIET - Etruel
-1,p2R (11-
vx((x)
d
vf(x])
vf(xF)
Strue 19
[A(xT)I
[V]((T)]
(23)
vx((F)
d7
Vs(xF)
121
(1)2((1-))
2222
[Vf(x))
|d
VV(())
< <4L' - s)² 222
Here, (a) results from
v(()) [x(
< and
VA(X
11
[V
Finding an upper bound for III:
ft + (True) Vf(x + +
(x{ + STree)= - -/()))
V + [ (24)
< / + (True) - V V |1
33|[|V/ + (True) Vx(((()) +3 - -/(x +
312 p2 +
302 35 3T.2 2== 352 + 6L
(a) follows from Equation g in Assumption
(b) follows from Assumption 1 and the perturbation being bounded by the perturbation radius p.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Putting Equation
22 23 and 24 in Equation 21
E
G
[3]
= 4nL' O'L - s)² + 16nL p²(1 - 8)² + SL'ppp p= $²
Σ
4 + 12n6 - -24nL2p p²
i=1
(25)
+ 4nbB
fx(x
+
n
Putting Equation
25
back in Equation 20
(1 op
E
X
Q)
(AnL''" - 5)² + 16mL (6 s)²+
3
8L3p²² 2
& + 12nd² + +2nnL 2 0² + 4nlil||
[i('' +)
t=1
1=1
(26)
n
4L p² (1 - =] + 16ml2p - 8)² + 1282 + 22133)) + 8l
Σ 6
(1 VX)2
(1- v2)²
1=1
+ 4n7² 2
1-1
(-1--)
V [x + +
r=0
i=1
Summing over L (o...., 1} with E
Xo(I - Q)
=0:
T-1
4nnz
9-11-1
X
Q)
CCT
E[1 Σ
(x + + s
(1 VA)
1=0 ==0
Anaj 2
[ 1
CT +
V + 1)) s
(1 VX)
n
(27)
4nn7²
T-1
CT + (1 - vx)
[|[
[ + $
i=1
where C
nn?²
4I.²
10000"0 4I.2p'
s) s)
16nL2 p2 ()1-s)3++
125 + +)2
211322) +0202+242T2p)
8L''3'311
Σe
-
(1 - vA)2
i=1
Dividing both sides by n:
00/ <
4T. 4T.2P'02 155-8) 5)² + 116122 - s)² + 1252 + 2AT.2/²
(1 - v2)2
(28)
8L 8L"'"'3''TT
( &
+ 1-
4n²
Σ
T-1
[ + +11
(1
2
This directly translates into an upper bound for the consensus error:
2T
(1 s) + 302 4L s)²
n
(1- vXj²
(29)
41²
T-1
2223222 + +6122) +
Σ
Elll Σ
fx +
(1 (イ)
1=0
1=1
19

---

Published in Transactions on Machine Learning Research (01/2026)
We analyze the convergence properties of GFlat based on the following update scheme, derived from
single-step progression of the averaged model x² Ei= xf:
13
x
17
g
(30)
72
To begin, we reiterate Theorem
! presented in Section
5
((I-V))+100 v\) (1- (1-V3)
Theorem Given Assumptions 1-3, let the learning rate satisfy 17 S for all T 2 1, we have:
Then,
81,
E [IVS(1)))))
2 TIT (BIf(x)) f*1 +0) s)² ((1-
4n
nT.
vXj2 [[]= +
T 1=0
n
(31)
12n/2
8,2
+ L2
(1- VI-vX 2(1 5)² + 52 -))-)))++++++ +3) +3s² + 6s² +3
(1 vxP2
Proof: We start with the following property for a L-smooth function f(x):
L
< * x +
t+1
(32)
Ln7
1
Blf(x*t)) E[f(x')
B
E1I1
(33)
2
(a) results from the update rule in Equation 30
We first start with I:
1: [(V((*)
Σ
g
1 M g +
V
+ + - V/(X+ +
=<([V[('))
M +
V + +
(34)
_ ((())(()) Σ (xt + +)) 1 || + E1 n
fl( + <b)1P)
((VS(()))
V + 1
(a) results from Egg = V [x{ + + (b) follows from (a. b) |=[aa|l² + |bbl
||a bb1a]
Simplifying *:
|V 1 V[i(+++00 -11 Σ (x((')
1 Σ
+
n
7
[V - v + + 1 Σ
1. - -x-
(35)
V=1
b 1
(1xx** - x + x 139
20

---

Published in Transactions on Machine Learning Research (01/2026)
(a) follows from Assumption 1. (b) results from the inequity ||a bll2 = ||a||2 1|612
Now, we define an upper bound for 1
|<: - p²||s
di
8)
g
2 0211s
d
Vf(x2
Vf(x))
(1 8)
g
|
gg
|
[V)((x))
[V](())
gg
(36)
$ (382 35 + 3(1
sp
22
6p 8² + 3p2
Putting this result in Equation 35
L Σ +
L n
[V n
[x(x + + -
Σ ( x((1
n
n
(37)
+3I p² + 6L. p²s² + + p
Note that 1
E.=1 2 5 where maxisisn E/.
n
Substituting Equation 37 into Equation
34
1
L
((()((*)
[x( - -
E
x - x(1]
2
2n
1=1
1=1
1=1
(38)
3T22
-3L²p²²² p
Substituting Equation
38
into Equation
E[f(x(+1)] [x)((*)) - V
EE[| Σ Vf(( =
i=1
(39)
E
||x +
(5262 + + 1/2
1
+
L17²
Bll
Σ E
2
2
2m
Rearranging the terms and dividing by 3 0 find the bound for E
2
- Ell Σ {x +
(40)
2262 + 82 -+
1
1, n
Σ
11
+
+ 6T.
+ n
We first bound Elll
g
El| Σ
g
E||
f + +1)))
72
n
L3pPo2(1-s)
Vf(xT + Bl|l
(V + + v
(41)
n²
n
E|
(x) + +
(a) holds because
g (x( { are independent vectors with mean
Yu et al. (2019). and (b) follows
from equation
22
21

---

Published in Transactions on Machine Learning Research (01/2026)
Using the above result in Equation 40
we have:
|||VS(*))1
< (BIS(x*) - [[(((*+)))) )]) - -B11 n1
2
Σ
V
+
+
1
Σ
xx -x11
")
i=1
(42)
1
L2 1222002 s)2 IIl|
6LLP²
+ 8² +
+22
2)
nL
V
+
2
Summing over i E l..., T 1 and dividing by T, we have:
T
Ell
V[
= T/T (f(x9)
- f(x*))
Σ
T-1
Bl1 Σ
1
fx + +53111)
T
T
1=0
2m1
1 T-1
Σ
E
n Σ (1|x -x(111111
1
+6I30²
(x262 +
+
1 2)
(43)
T
2
1=0
11
8)2 8)3
1
ΣΣ
T-1
BlJ
n Σ
V +
n
T
n
Using the result of Lomma
3 in the above Equation and rearranging the terms:
s 2 n/T (S(x)) - [(x))) + n1 nn.- 1+ -1+
4 L²
E [
1 1 vx
+ 4r22L2
[B|]
V +
[I'P²(1 (0011-6)) -s)²0² + 352 + 4I - s)²pp²
n
(1 vxj2
t=0
+ +2L2s²2p's + 6127)
+nI
1.2P002(1 8)2
+ -LL'p"
1822
+
$2 + 2
222
n
2
(44)
2 T/T (f(x*) -f(x)) + (nL
2r2²L2
Σ [Bill
T-1
1
fx
+ (D)PD) +
1 vT
T
1=U
(1p - -sP0² ( L17
nL
(
L2
8²
(1 - vX)² n2
(1 - v\)2
+ +2222 (1 vJj2
2(1 - =))² + 2 +3)
+ 2c² + 6s2² +
When qT. - 1+ 4nj2 50:
1-VX
< 2
(f(x))-)% 100-))+++'-0 +0213 s)² ( 472212 7
E|| [(|
T 1=
T/T
(1-VA)- n
(45)
12n²
81
8²
+ 2²2L2
(((1 9) 522 +3 +3s2 + +6s +3
233
(1- vxp²
(1-
(1 VX-
B.3 Discussion on the Step Size
The condition for Equation 45 to be true is 1b. - 1 + 1,2 that > O, we have:
I-VX S 0. Upon solving this inequality with the fact
V (1 vx22 + 16(1 - VA - -(1 -1- v
n z
(46)
8L
22

---

Published in Transactions on Machine Learning Research (01/2026)
B.4 Proof for Corollary 2
According to Equation
45
the right hand side, there four terms with different coefficients with
respect the step size rj = (VT and perturbation radius p O / We separately investigate each
term:
2 / (J(x)) - 17) = =0 -
Vnn
(47)
L2 p2 - 8)² -s)
(11-V5'
(
4,2
vxj2
nT.
0² 0 0
T n1/2T3/2
(48)
n1
12n
8² = 0 T (1 (1- vx)2
(49)
I?2
2(1 - ()1 +)²+ 2 +3) + +3826 (1 VA)
L2
6s² + 632 0 -p + +
(50)
By omitting n in higher-order terms (since T >> n) and combining the results of all the above equations, the convergence rate is as follows:
T Σ
| (1) s0 (129)-) - (
(1 (1=
8² e
VnTT +
(51)
T
T3/2
T
T
This implics that when T is sufficiently large, GFlat results in convergence rate of O
VnT
C Algorithmic Details
The authors in Lin et al. 2021 propose Quasi-Global Momentum (QGM) buffer, which improves the local momentum acceleration in the presence of data heterogeneity. Specifically, QGM updates the momentum buffer by calculating the difference between two consecutive models, x and xf, to locally approximate the global optimization direction. The update rule for QGM is given by the following equation:
wiglx ngg) + (3mmmm)) mf = jum; + (1 11) x x( (41)
QGM: x
Σ JEN()
(52)
To further enhance performance, we introduce variant of our approach called Q-GFlat. For implementation details, please refer to Algorithm /
23

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm 2
Q-GFlat
Input: Each client e [1,n] initializes model parameters X,, step size y, perturbation radius p, scaling factor E [0, 1], mixing matrix W [ Wii) represents neighbors of i, momentum coefficients B, p.
procedure TRAIN( ) for Vi
1. for 1.2..... T do
2.
g [x; B3) for B ~ D;
3.
m ..m
-1 +g.
4.
d: x x
5.
ç = p(s.Taf) (1 5) mm
d'
6.
g (1+1/2) = (1) !8!
(x( c
)
7.
SENDRECEIVE (x; (1+1/2)
8.
x 1- = E E)E,,(0) ] 1/2)
9.
10.
mf أار 1-1 + (1 p)
11.
end
return x!
D Setup Details
D.1 Datasets and Models
For ImageNet Deng et al. 2009)), the terms of access and details related to the license are available at nttps://image-ntt. org hwtp://wwooaaddoooo php We download Imagenetle Husain (2018) from https://github.
com con/fastal//magenettee which has Apache License 2.0.
All models use Evonorm Liu et al. (2020): Hsich et al. (2020a) as a normalization layer, as it is shown be better suited for decentralized learning with non- data. We use the standard ResNet-18 and ResNet-20 architectures He et al. (2016) with 11M and 0.27M trainable model parameters, respectively. For MobileNet- V2, use the architecture with 2.3M parameters Sandler et al. (2018).
D.2 Non-IID Distribution
Similar to prior works Choudhary et al. (2025): Aketi et al. (2023b a): Lin et al. (2021), we use the Dirichlet distribution simulate different levels of data heterogeneity
our setup Hsu et al. (2019). Figure 0
shows the number of data points from each class of CIFAR-10 that are allocated to different clients for 0.01, 0.001. While 0.01 allows most clients to have samples from two different classes, 0.001 is the most extreme form of non-IIDness, with clients having samples from only one class.
D.3 Graph Topologies
We perform our experiments for two different graph topologies, ring and torus. Ring is one of the most sparsely connected topology and has poers/nette poers/neighbors per client, while torus has Please refer Figure for visualization of these.
Table 6: Learning rate (n), batch size per client, and the number of epochs for all the experiments.
Dataset
CIFAR-10
CIFAR-100
Imagenette
ImageNet
Learning Rate (1))
0.1
0.1
0.01
0.01
Epochs
200
100
100
50
Batch-Size Client
32
20
32
64
24

---

Published in Transactions on Machine Learning Research (01/2026)
9
9
Class Labels
Class Labels
7-
5
5
3
3
5
11 Clients
13
15
17
3
7
11
13
15
17
19
Clients
(a)
0.01
(b) a = 0.001
Figure 6: Test accuracy for the CIFAR-10 10 trained on ResNet-20 architecture distributed with various levels of heterogeneity (a) over 20 clients.
Figure
Ring Graph (left), and Torus Graph (right).
D.4 Hyperparameters
The initial model and all training hyperparameters synchronized the start of training, and communi- cation between clients is assumed to be synchronous. The initial learning rate is set according to Table G with a decay factor of 10 applied at 50% and 75% of training. We use a weight decay of 0.0001, and training proceeds for a fixed number epochs as specified in Table 6 Quasi-Gia Global Momentum (QGM) uses the Nesterov momentum with a coefficient of 0.9. For SADDLe and Q-SADDLe Choudhary et al. (2025). we adopt the same perturbation radius p as reported in the original paper. Additionally, we perform grid search (0.05, 0.2, for and (0.2, 0.5, 0.7, for the scaling factor 8, selecting the best values based performance (see Table 7 for details). As noted in Remark 1 in Section 5 0.5 <= <= 1 yields the best results. Interestingly, when = 1, GFlat becomes more computationally efficient than SADDLe, as it eliminates the ascent step required to compute the local perturbation. We report the test accuracy of the consensus model averaged over three randomly selected seeds. All experiments are conducted using NVIDIA A40 GPUss
Table 7: Porturbation Radius (p) and Scaling Factor (s) for GFlat and Q-GFlat.
Dataset
CIFAR-10
CIFAR- 100
Imagenette
ImageNet
Perturbation Radius (p)
0.1
0.2
0.05
0.05
GFlat: Scaling Factor (s)
0.5
0.7
0.5, 0.7
Q-GFlat: Scaling Factor (s)
0.7,
0.7, 1
E Additional Discussion on Local and Global Flatness
In docentralized learning, each client's objective f((x) captures only a partial view of the global objective f(x) {x)( Under IID data, these objectives are well aligned: shown in Figure the eigenvalue spectra
25

---

Published in Transactions on Machine Learning Research (01/2020)
2.12
2,47
4.52
3.0
2,87
Local
2.0-
1:9
Global
2.5
3.65
1.82
1.5
1.5
1.0
1.0
1.0
0.5
0.0
Epoch: 50
0.0
Epoch1100
Epoch:
0.0
Epoch:200
Figure 8: Ratio of largest to 5th largest eigenvalue (Amaz/As, As) of the Hessian of the local and globally averaged models at 4 different training stages, under an IID data distribution across 10 clients on the CIFAR-10 dataset.
of local and global Hessians are nearly identical, indicating consistent. curvature across clients. However, under non-lll) data, the curvature of these local landscapes can differ dramatically, ie., regions that appear flat for one client may correspond to sharp directions in the global loss. Consequently, optimizing for local flatness alone does not guarantee global flatness, as done in prior methods such as SADDLe Choudhary et al. (2025). In fact, aggregating updates from locally flat but mutually misaligned regions may yield globally sharp solutions that hinder consensus and generalization. This phenomenon is clearly evident in Figure where the discrepancy between local and global flatness widens markedly under heterogeneous data partitions.
We attribute this misalignment to the fact that each client estimates its worst-case perturbation using only its own local gradient gi, which captures limited curvature information. Ideally, one would instead compute perturbations based on the global gradient (g' = 1/n Ei= g)), ensuring that every client's update direction is consistent with the global curvature. While this would align local SAM steps with global flatness, such synchronization fundamentally violates the decentralized setup, requiring full communication among all clients at every iteration.
E.1 Theoretical Analysis
To overcome the above mentioned limitation, we introduce locally approximated global direction d which allows each client to estimate g' using model differences across consecutive training itcrations. This approximation enables clients to incorporate proxy of the global update into their local SAM objectives, effectively aligning local and global flatness without incurring any communication overhead. We refer to the discrepancy between the approximated and true global directions, ||d (aff-1) 7//*|, ||, as the perturbation deviation, which quantifies how closely each client's 's local displacement tracks the global gradient direction. Lemma formalizes this intuition, showing that the perturbation deviation decays asymptotically at rate O(1/T).
Lemma 5 Suppose Assumptions 1-3 hold. Let the stepsize and perturbation radius satisfy = o T and p OVVT). Then
1
- o )
(53)
T
Proof: Recall the parameter update rule and our approximated global direction de from Algorithm
x =
Σ
wiy
wxz(x - -188 -118, 11)
(54)
JENDO)
d =x
x 11
Σ
Wis
g +-1
+ x (x--1- Ewirt(11
(55)
26

---

Published in Transactions on Machine Learning Research (01/2026)
Add and subtract E Wiygj and 17 E Wijgj :
d - 1/ = 17 Σ Wij (g)- 1 8,) + 17
Σ
L 8))
(x- -
Σ wiss(/
Wiygj
+
A
B
(56)
+ Σ
gj 1)
Wiy Bj
D
| "/' s <((A41 - |B] + ||C + |DD]3).
(57)
We start by bounding A. Through Assumption Jensen inequality,
||s 1-/11 8/ll || -|| xi|1 Liddil- Hence, using
|| = 12 E (1) - -B)i| s Σ whyll 2| STRL2 Σ |1d311.
(58)
- and G é 82..82..... gi We will also use the stacked vector notation only to apply I|W 2 non-neighbors simply have Wij- = 0. With standard basis vector e, and
Based on notations in Equation 15 Q == 1
W Qlla S Vλ, we have:
S T Σ |gy) 11²) 1/2
I1²)
BB| = ne, (W) G' < <//l - Qlla l
(59)
For each j, add and subtract (x) and use the inequality I|u u112 - 2||2u|1 + 2|/2|11
|gg 12 = + -V/(x))))V/
(x)
(60)
Summing equation 60 over j = 1,..., n yields
(f) < 2 Σ
|[)((')||
+ 2 Σ
[f (x
(61)
==1
1=1
By L-smoothness (Assumption-l), we have:
-
< Σ |x] 2112 = =L2
(62)
1=1
1|1 = - - 2nd² + 2m V 1(2
(63)
Σ
|
= + 21.2 A.
(64)
1=1
Substituting the above in equation 59
B
< 2
4n(1v + 21 A.
4n
(65)
27

---

Published in Transactions on Machine Learning Research (01/2026)
Now we bound C, where C x w We stack all client models into X ;x.;; E Ruxd, so that the i-th row corresponds to client i. Then C can be written as
C= (I-
(66)
Here, e.- I" is the standard basis vector. Since W is doubly stochastic, have WQ QW Q2 and therefore
- (I-W)= = (I -I- Q) = (W Q) W(I-Q) Q). Q)
C = e (I- W)x = (I- - -) -
(67)
By Assumption 3, W Q S Vx:
||]|| = |ee (W -x----x---1-- -x-1111
Since
|| E, ||رم|| || for the rows a, of A, this yields
|CP| < A Σ | x111 = -
(68)
1=1
Now we bound D with L-smmothness and bounded perturbation:
n > wky (
g gj
//-1)|| L2'p
(69)
Now we bound Ej ||dl1
From equation 55 we add and subtract the unperturbed gradients to obtain:
d 1) Σ UjkKk 1-1
+ (x)
Σ
W x1)
(70)
Wiigk Σ
Wykk (gk
1-1 - -g%) + Σ
(Bk 1t-
-22--g--1 1) (x Σ w)xx
Wik
k
With Aú = Ee Wzngh, By:
= 11 Ei Wikk (gk (g-1-gh), C): Cy: x Ex uisx (-1 Dj == 11 E Wikk
(B 1-1 -gi gk 1-1)
we have,
| < Σ 11,12 + 4 11,1 + Σ |c,1 +4 +4 Σ D311²
BB
(71)
/=1
Σ1AR = ?/ >
& T'WG < 1 Σ 12.
(72)
1=1
since || < 1 for doubly stochastic W. Through equation 64
Σ IIA s n² (In(IV/(<)))+++ 21 +2L2A.).
(73)
For By, let AG', GL-1 Gt (row k is gi g(). Then
11BB11 r'IIAG'' s n² Σ g
gill S n22L3 Σ
||
(74)
28

---

Published in Transactions on Machine Learning Research (01/2026)
Now we bound Cj:
Σ = -11 1---1l(w--- (WW-)) < WW- -Q AA (75)
For Dj. L-smoothness and || S P, we have:
88
1 $ Lp
(76)
Combine equation 78
Eiddi < 4n/ (4n(IV][((')) +8) + +2124.) 4,2 Σ ||dī11 4AA,11 1 +413p²
(77)
(1-41 Σ l|ali s (IV)<(( + 81)21,2, + -
+ 4122
Assuming standard stepsize condition 4n7² S (so (1 - 11-44)22)-1 2) we get
Σ ldgl S 32 ( 02) + 16,, + 82 3A + +
(78)
Substituting the above result in equation 58
|||| < (
+ 16, + 8X72L2 4
(79)
Substituting equation
79
65
68
and
69
in equation 57
A. d 2 (32n'nI² (VV(( 8²) + 16 + - +
(80)
4n/ 4m (V || 82) + 21 A.] + +
Collecting coefficients yields
| - -/g $11 < 1288 (128 1 + 16
(1V + 8²)
C
(81)
(64n L + SAn""12 +
A. + (32n)²1 1,2 + 4) AA, + (32L² + +4)
Cs
C²2
Cp
bd 112 s C, (1V CA +:CAA.1
(82)
with the explicit constants Cg. Cs. C/2, Cp as above. Averaging over total clients n and training iterations T:
T
T
+ ΣΣΣ '12 < C, + Σ (( 8²) + Cs - Σs
+ & 1 7
A Co. (83)
Now similar to main paper, let == O O(vn/T) and p= 0(1/TT)))
29

---

Published in Transactions on Machine Learning Research (01/2026)
From the result of Lemma g
Σ A - (
(84)
From Corollary 2
1 T
V () 0 (
(85)
C, 6 - (7 7
CA 0 77 + 7
1 = (
(86)
C = 0 T -1 = -
1
1
C, - 0
T7
Putting together the above equations, for a fixed << T:
T
Σ ||
1 -
< (())
1 NT
+ +8) + - + (+z)
((7) + of nT
+ ((t) + (+z)
(87)
o
Interpretation. Perturbation deviation quantifies how closely each client's displacement df aligns with the global gradient ng,, and is controlled by the following key factors:
d - 112 S ((I + 6*) C. A. + C..
The bound shows that this deviation is governed by the global gradient magnitude, the consensus error among clients, and the SAM perturbation radius. As training progresses, both the global gradient norm and the consensus error A, decay, while the perturbation effect remains bounded and scales with the decaying learning rate. Consequently, the deviation diminishes at rate O(1/T), indicating that each client's local displacement asymptotically tracks the global descent trend, validating our approximated global direction as faithful proxy without requiring any global synchronization.
Implementation note. In implementation (Algorithm D. each client forms the perturbation direction convex combination of its local and approximated global directions:
d
g!
ş = p(s
(88)
|ddl|
|gg:
where p is the perturbation radius and a controls the emphasis on the global vs local direction.
Since d = x x already represents the model update that includes the step size n, we include 1 only with the global gradient term in Lemma 5 In other words, n7 is implicit in df but explicit in ng". ensuring that both quantities are comparable in scale.
E.2 Empirical Validation
To validate the theoretical result of Lemma we compare our practical implementation of GFlat, which uses the locally approximated global direction d; against its oracle counterpart. GFlat (Oracle) in Table
30

---

Published in Transactions on Machine Learning Research (01/2026)
which directly uses the true global gradient g' for the SAM ascent step. As expected from the bound perturbation deviation, the oracle variant achieves consistently higher accuracy across all settings, serving as an upper bound on the achievable performance under perfect global coordination. However, the small performance gap between the two confirms that our locally approximated direction effectively tracks the global gradient without requiring global synchronization.
Table 8: Test accuracy (%) on CIFAR-10 with ResNet-20 under different levels of heterogeneity (a). GFlat (Oracle) denotes the variant that uses the true global gradient g' in the SAM ascent step and serves as an upper bound on achievable performance, while GFlat uses the locally approximaled global direction.
CIFAR-10 (ResNet20)
Clients
Method
0.01
0.001
GFlat
70.12 + 0.68
62.29 + 0.66
10
GFlat (Oracle)
73.84 + 0.46
66.41 + 0.34
GFlot
56.21 + 1.33
51.76 + 0.68
20
GFlat (Oracle)
57.41 + 0.45
52.09 + 0.94
E.3
Alignment Analysis
We further quantify how well the update directions in GFlat align with the global gradient by mcasuring the deviation s + (1-1s1
(1-s)gt s)gt - g'll averaged over clients. For comparison, we compute the same metric
for SADDLe, which relies solely on local gradients, i.e., lg - gl As shown Figure 9 GFlal exhibits substantially smaller deviation, demonstrating that incorporating the approximated global direction improves curvature alignment across clients and leads to more coordinated updates, along with globally flat model demonstrated in Figure]
3.50
SADOLa
GFiat
3.25
Trtal Devietton
3.00
2.75
2.00
2.25
2.00
1.75
1.50
Figure 9: Total perturbation deviation between each client's update direction and the true global gradient g' CIFAR-10 distributed across 10 clients with a 0.01. The smaller deviation observed for GFlal indicates stronger aligument with the global gradient, validating the theoretical perturbation deviation bound.
31


Reviews:
[{"review_id": "hQFMjMDKek", "reviewer": "Reviewer_ZN39", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents GFlat -- a new decentralized learning algorithm that improves generalization in non-IID data settings through seeking global flatness. Authors propose a simple strategy that improves upon SADDLe -- a method that only achieves local flatness. Both methods define the objective as a sharpness-aware minimization problem for decentralized setting. However, GFlat redefines the perturbation $\\xi$ for each client $i$ as a combination of local and global perturbations. To quantify the flatness authors utilize a proxy for loss curvature -- ratio of largest to the 5th largest eigenvalue $(\\lambda_{max}/\\lambda_5)$. Authors also provide a theoretical analysis of GFlat that shows convergence rate similar to existing literature. Assumptions stated for the analysis are reasonable and standard throughout similar methods. \n\nAuthors conducted experiments on image classification task with 3 DNN models and 2 client topologies, and compare GFlat to SADDLe and DPSGD. Results show consistent improvements over other methods.  \n\nAdditionally, paper has a coherent structure and a comprehensive introduction to the problem and related literature. \n\n---\n\nStrengths:\n1. Problem is clearly defined with a relevant motivation. \n2. Presented methodology has a clear advantage over existing methods which is shown through numerical experiments. \n3. GFlat incurs no additional communication overhead and only requires $\\mathcal{O}(m)$ memory at each client. \n4. Scaling factor $s$ in the redefined perturbation $\\xi$ allows shifting emphasis on global or local perturbation, which makes the method flexible to different degrees of data heterogeneity.   \n\nWeaknesses:\n1. Code used for experiments is not included with the submission which makes it hard to reproduce the numerical results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I have found authors' claims to be accurate based on presented numerical experiments.", "audience_yn": "Yes", "audience_explanation": "I found the topic of this paper to be relevant for TMLR's audience based on other submissions accepted to TMLR that were studying similar research directions.", "requested_changes": "Include some type of reference to the code repository that was used to conduct the numerical experiment.", "broader_impact_concerns": "I have not found any ethical concerns in this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Oe03txx9Y9", "reviewer": "Reviewer_gHAo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper builds on sharpness aware minimization to obtain an optimization algorithm for decentralized learning. Sharpness aware optimization has minimax objective function which aims to obtain a local minimum with largest flat neighborhood, in the sense that function varies slowest in the neighborhood of local minimum. In decentralized learning scenarios, in order to obtain the flat neighborhood around a point, one need to obtain gradients of global objective function. There is a number of factors like privacy, computational and communication overheads which make the computation of global gradients impossible. Therefore, the paper combines the ideas from previous work [1] to estimate and update perturbation using local gradients and model differences. \n\n[1] Lin T, Karimireddy SP, Stich S, Jaggi M. Quasi-global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data. InInternational Conference on Machine Learning 2021 Jul 1 (pp. 6654-6665). PMLR.\n\nStrengths:\nThe paper is very well written and clearly motivates the problem.\n\nWeaknesses:\nThe paper has minimal novelty.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Paper is very well written and well-motivated, and the claims made in the seem to be accurate.", "audience_yn": "Yes", "audience_explanation": "The paper deals with problem of decentralized learning, which is an important practical problem in machine learning.", "requested_changes": "The paper is well organized and clearly written. I have only a few minor suggestions.\nIn Equation (2), the notations $d_i^t$ and $\\boldsymbol{d}_i^t$ seem to represent two distinct quantities; please consider changing one of them to avoid confusion.\nIn addition, it would be helpful to elaborate on the update rule in Step 8 of Algorithm 1 and clarify how it differs from the traditional update rule in Equation (2) for DPSGD.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ulxWVgX86b", "reviewer": "Reviewer_SmoR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces GFlat, a decentralized optimization method intended to achieve \"global flatness\" of the loss landscape when data across clients are heterogeneous (non-IID). The main idea is to modify the Sharpness-Aware Minimization (SAM) optimizer at each client by adding an approximation of the global update direction, estimated from model differences between consecutive iterations. This modification requires no additional communication and is claimed to improve generalization. The authors also provide a convergence analysis and experimental results on several datasets and network topologies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The theoretical section proves a standard convergence rate that does not justify the claimed \"global flatness\" property.  \nMoreover, several notational and conceptual inconsistencies make parts of the analysis difficult to follow.  \nThe exposition often lacks rigor, and key ideas such as how the approximation $d_t = x_{t-1} - x_t$ captures any meaningful global information remain speculative.\n\nThat being said, the experiments show accuracy improvements over SADDLe and other baselines, which seem to indicate that the proposed algorithm has the advertised effects.", "audience_yn": "Yes", "audience_explanation": "The topic: decentralized optimization with non-IID data, is relevant to TMLR’s audience.  \nThe experimental section seems to demonstrate that the authors manage to address that issue.\nHowever, the proposed modification is minor compared to existing SAM variants, and the theoretical content largely reuses standard proofs from decentralized SGD literature.", "requested_changes": "**Critical:**\n\n- Clarify notation and correct typographical errors.  \n   - In Eq. (3), I think that it should be $f_\\rho(x) = \\max_{\\|\\xi\\|\\le\\rho} f(x+\\xi)$.  \n   - Eq. (4): I think that it should read $\\nabla f_\\rho(x) \\approx \\nabla f (x + \\rho g/ \\| g\\|) \\dots$\n   - Eq. (5): missing parentheses to make the max scope clear.  \n   - Eq. (6): use different symbols for $d$ (data) and $d$ (direction). It is currently confusing.\n\n- Discuss why gradient momentum is used only in the first step where it is used to estimate the worst case point locally, and not also when you do the actual update.\n\n- Theorem 1: explain whether $\\epsilon$ is random or deterministic and whether a $\\max_T$ is missing in its definition.  \n\n- Motivation and interpretation: provide a clearer argument for why the local difference $x_t - x_{t-1}$ conveys true global information; as it stands, this is not convincing. Indeed, it aggregates local information about the neighbors, but beyond that, I do not see why it translates to global information.\n\n- Can you clarify a bit more the link between local flatness and global flatness? As for me, it might even be worse than simply saying \"local flatness does not imply global flatness\": aggregating local critical points might not even lead to a global critical point.\n\n**Recommended (non-blocking):**\n\n\n\n- Note that Assumption 2 trivially holds for $\\sigma^2 = 2$ after normalization. \n \n- Include more details on how non-IID partitions are created in the main text.  \n\n- Clarify the DPSGD acronym, which is usually understood as *Differentially Private* SGD, not *Decentralized Parallel* SGD.  \n\n\n- Explain the meaning of \"orthogonal\" in \"orthogonal and compatible\". If it simply means \"independent of communication topology\", say so explicitly.", "broader_impact_concerns": "NA", "additional_comments": "- The paper’s structure and motivation are acceptable, but the writing and technical precision can be improved.  \n- The connection between the proposed perturbation scheme and actual “global flatness” is not substantiated theoretically.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)