Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awareness

## Abstract

Abstract Federated Learning (FL) can be coordinated under the orchestration of a central server to collaboratively build a privacy-preserving model without the need for data exchange.
However, participant data heterogeneity leads to local optima divergence, subsequently affecting convergence outcomes. Recent research has focused on global sharpness-aware minimization (SAM) and dynamic regularization techniques to enhance consistency between global and local generalization and optimization objectives. Nonetheless, the estimation of global SAM introduces additional computational and memory overhead, while dynamic regularization suffers from bias in the local and global dual variables due to training isolation.
In this paper, we propose a novel FL algorithm, FedTOGA, designed to consider optimization and generalization objectives while maintaining minimal uplink communication overhead. By linking local perturbations to global updates, global generalization consistency is improved. Additionally, global updates are used to correct local dynamic regularizers, reducing dual variables bias and enhancing optimization consistency. Global updates are passively received by clients, reducing overhead.
We also propose neighborhood perturbation to approximate local perturbation, analyzing its strengths and limitations. Theoretical analysis shows FedTOGA achieves faster convergence O ⁢ ( 1 / T ) 𝑂 1 𝑇 O(1/T) italic_O ( 1 / italic_T ) under non-convex functions. Empirical studies demonstrate that FedTOGA outperforms state-of-the-art algorithms, with a 1% accuracy increase and 30% faster convergence, achieving state-of-the-art.

## 1 Introduction

The widespread connectivity of mobile terminals has greatly propelled the development of industries related to big data. However, the massive data throughput has led to communication link congestion and increased privacy risks. Consequently, to safeguard data privatization and localization, FL [] has garnered significant attention as a distributed machine learning (ML) method that avoids the need for data exchange. FL employs the “Computation-Then-Aggregation”(CTA) strategy []. Due to the uplink bandwidth constraints on global servers [], FL adopts multiple local training steps and partial participation to alleviate communication bottlenecks. Simultaneously, the diverse data collection preferences of participants [, ] result in conflicts between local optimization, which causes the global loss to converge to a steep local minimum []. As illustrated in Figures 1(a)-1(c), the sharpness of the global model increases dramatically with the enhancement of local heterogeneity. Moreover, this disparity is significantly magnified by increasing synchronization intervals and reducing participation rates [, ].

Figure: (a) Dirichlet 0.1
Refer to caption: extracted/5820635/Figure/Avg_Dirichlet_0.1.png

In response to these challenges, the majority of studies addressing global consistency issues via the Empirical Risk Minimization (ERM)[]. However, when handling highly heterogeneous datasets, global solutions may become trapped in steep local minima, rendering it difficult to provide reliable estimates [] and potentially causing the optimizer to stagnate. Consequently, recent innovations have leveraged Sharpness-Aware Minimization (SAM) [], which seeks to identify a flatter minimum by minimizing the perturbed loss of the model, thereby enhancing generalization capabilities. FedSAM was introduced by incorporating SAM into FL [], and further, momentum-based algorithms were integrated, resulting in the proposal of MoFedSAM. FedGAMMA[] replaced ERM with SAM in Scaffold to enhance its performance and alleviate model bias. FedSpeed[] integrated FedDyn into FedSAM to bolster performance.
However, the approach based on minimizing local sharpness loss fails to capture the flatness of the global loss surface, as depicted in Fig. 1(a). Therefore, [] proposed FedSMOO, which corrects local updates and local perturbations by introducing an additional control variable, as illustrated in Fig. 1(e). Furthermore, FedLESAM [] estimates the global perturbation as the difference between the locally stored historical model from the activation round and the global model received in the current round, thereby avoiding extra computational costs. Nonetheless, FedSMOO introduces additional communication overhead and storage requirements, which can be unacceptable in real-world environments with limited bandwidth. In FedLESAM [], the global perturbation estimate does not encompass additional current local gradient ascent computations, while reducing computational overhead, may lead to insufficient local generalization. Additionally, in both methods, the difference in local perturbation scales may be significant when facing clients with prolonged disconnections, thereby disrupting generalization. Apart from addressing the consistency of perturbation generalization, the consistency of local objective optimization has also been deeply studied, as seen in ADMM-like algorithms [, , ]. However, performance significantly degrades, due to the CTA strategy causing local dual variables to be updated separately on the clients, preventing timely synchronization with global dual variables.

To achieve a reliable, stable, and consistent global model, we propose a novel algorithm named FedTOGA, as illustrated in Fig.1(f). FedTOGA initially guides global update gradients to merge with local perturbations, thereby enhancing local generalization consistency. Simultaneously, it employs global updates to correct the local dynamic regularizer, reinforcing consistency with the global optimization objective. This approach significantly improves performance, even under extreme conditions characterized by highly heterogeneous data or limited client participation.

Specifically, due to the communication interval, the universal SAM optimizer applied on the global server cannot precisely capture the perturbations occurring during local updates on client devices. To maintain local consistency, we introduce the global update gradient as an approximation, replacing the global perturbation estimation used in FedSMOO []. Furthermore, the universal dynamic regularizer on the global server is unable to accurately obtain the dual variable updates during client-side local updates. Hence, we propose leveraging the global update gradient to correct local dual variables, thereby mitigating discrepancies between local and global dual variables and further aligning the global and local objectives. This further aligns with the global and local objectives. To further reduce computational overhead, a method termed Neighborhood Gradient Perturbation has been proposed. When the interval between local training sessions on the client side exceeds one, the client simulates the current perturbation by utilizing cached gradients stored in a gradient register, thereby alleviating computational costs. Unlike FedCM[] and MoFedSAM[], the global update is not treated as a trade-off term with the local perturbation, meaning that their coefficients do not sum to one. As the active local clients converge, they ultimately reach a globally stationary state, characterized by a smooth loss landscape.

Theoretically, FedTOGA is capable of achieving a rapid convergence rate of $O(1/T)$ in non-convex settings. Experimentally, extensive evaluations were conducted on the CIFAR10/100 datasets, demonstrating that FedTOGA achieves faster convergence rates and higher generalization accuracy in practice. These results were obtained in comparison to 17 baseline methods, including FedAvg, FedAdam, FedYogi, SCAFFOLD, FedACG, FedCM, FedDyn, FedDC, FedRCL, FedSAM, MoFedSAM, FedGAMMA, FedSMOO, FedSpeed, FedLESAM, FedLESAM-D, and FedLESAM-S.

- •
We propose a new federated algorithm, FedTOGA, the first global perturbation using the merged global gradient approximation, and the first local dynamic regularizer using global gradient correction, which reduces the uplink communication overhead and reduces the lags in extreme environments while maintaining fast convergence and high generalization.
- •
To alleviate the local computational overhead, we propose for the first time neighborhood perturbation, which is merged or replaced in the local perturbation using gradient registers without increasing the overhead, and analyze the advantages and limitations.
- •
We give a theoretical analysis of the convergence speed. In non-convex scenarios, FedTOGA achieves fast O(1/T) convergence speed. Meanwhile, we conduct extensive numerical studies on the CIFAR-10/100 dataset as well as different neural networks to verify the excellent performance of FedTOGA, especially on highly heterogeneous as well as barren participants far better than the existing methods.

## 2 Preliminaries

This section shows the preliminaries of FL and SAM, and related works can be found in Appendix A.1.

Federated Learning The goal of the FL framework is to build global models that minimize the average experience loss of participating clients:

$$ $\displaystyle\mathop{\arg\min}_{\theta}$ $\displaystyle f(\theta)=\frac{1}{N}\sum_{i\in N}f_{i}(\theta)$ $\displaystyle f_{i}(\theta)\triangleq\mathbbm{E}_{\xi_{i\sim D_{i}}}f_{i}( \theta,\xi_{i})$ (1) $$

Where $f:\mathbbm{R}\rightarrow\mathbbm{R}^{d}$ is denoted as the global objective function, $\theta$ is a model parameter, $N$ is the total number of all the participating clients, and $\xi_{i}$ is a randomly sampled data from the distribution $D_{i}$ subject to data heterogeneity. $f_{i}$ is the loss function for the $i$-th client.

Sharpness Aware Minimization
Many studies[, ] have pointed out that a flat minimum implies a better generalization performance, which possesses greater robustness to model perturbations. To minimize sharpness, [, ] proposed SAM:

$$ $\mathop{\arg\min}_{\theta}\big{\{}F(\theta)=\mathop{\arg\max}_{\|\delta\|\leq \rho}f(\theta+\delta)\big{\}}$ (2) $$

SAM extends the search by a one-step gradient ascent perturbation, and a one-step gradient descent to reduce sharpness and loss. First, calculate the gradient ascent perturbation $\delta=\rho\frac{\nabla f(\theta)}{\|\nabla f(\theta)\|}$. The gradient is then computed after adding the perturbation using the model and updating the model $\tilde{g}=\nabla f(\theta+\delta);\theta=\theta-\eta\tilde{g}$, where $\eta$ is the learning rate.

## 3 Rethink FedSAM and Others

The limitations of SAM in FL systems[] have been widely discussed [, , ], with the main conflict coming mainly from centralized training vs. Distributed Computing Perturbation Differences. The centralized SAM[]training objectives are as follows.

$$ $\max_{\|\delta\|\leq\rho}\mathbbm{E}_{\xi\sim D}f(\theta+\delta,\xi)=\max_{\| \delta\|\leq\rho}\mathbbm{E}_{i}\mathbbm{E}_{\xi_{i}\sim D_{i}}f(\theta+\delta ,\xi_{i})$ (3) $$

where $D=\mathbbm{E}_{i}D_{i}$, some work applies SAM directly to the FL paradigm [, ], and reformulates its goal as.

$$ $\max_{\|\delta_{i}\|\leq\rho}\mathbbm{E}_{i}\mathbbm{E}_{\xi_{i}\sim D_{i}}f_{ i}(\theta_{i}+\delta_{i},\xi_{i})$ (4) $$

where the local model $\theta$, and the perturbation $\delta$ are isolated due to the CTA of FL. As a result, maintaining consistency between the global model and the client model becomes more difficult as the local update interval and the degree of data heterogeneity increase []. In this case, minimizing local sharpness in isolation does not effectively achieve a global flat minimum.

Some recent studies, FedSAM[], MoFedSAM[], FedGAMMA[], FedSpeed[] have not resolved the internal perturbation variance contradiction.MoFedSAM uses momentum to weigh the perturbation gradient against the global gradient to alleviate this problem, FedGAMMA[] uses variance reduction techniques, and FedSpeed[] uses dynamic regularization techniques to alleviate this contradiction. FedSMOO[] notices this contradiction for the first time and will use dynamic regularization to correct the discrepancy between local and global perturbations, FedSOL[] employs perturbation orthogonality to find a consistent direction of perturbation, FedLESAM[] believes that computing the perturbations requires additional computation and therefore opens up additional storage locally to approximate the estimated global perturbations. For a more detailed description of the limitations, please refer to Appendix A.2.

### 3.1 Motivation

Obviously, the above study has some limitations. To increase the consistency of the global perturbations, FedSMOO[] attempts to solve this problem by correcting for the local perturbations, whose local perturbation estimates are computed as $\delta=\rho\frac{\nabla f_{i}(\theta)-\mu_{i}-s}{\|\nabla f_{i}(\theta)-\mu_{i
}-s\|}$. However, it introduces additional computation, which increases the overhead of the clients in FL. To efficiently optimize the global sharpness and reduce the computational burden on the client, FedLESAM[] proposes a strategy of history model staging, which enables the client to add additional local variables to record the history model, thus estimating the global perturbation $\delta=\rho\frac{\theta_{i}^{old}-\theta_{i}^{t}}{\|\theta_{i}^{old}-\theta_{i
}^{t}\|}$. However, as the set of activated clients $S_{t}$ decreases sharply, the perturbation estimation is more affected. FedTOGA estimates the global perturbation by merging in the local perturbation through receiving the proximity global update sent from the server, which does not require additional computation and storage space and ensures the timeliness of the estimation, and at the same time incorporating this global update into the dynamic regularizer [] to continuously find the global minimization.

## 4 Methodology

This section introduces our method, FedTOGA, and provides a preliminary analysis of the three key techniques and demonstrates the flow of the proposed algorithm 1.

Figure: Algorithm 1 FedTOGA Algorithm

### 4.1 Estimate Global Perturbation

As mentioned above, our goal is to efficiently estimate the global perturbations(G-perturbations) of each client without incurring additional storage or computational overhead. To achieve this, we first recall the definition of sharpness-aware minimization in FL:

$$ $\min_{\theta}\big{\{}f=\frac{1}{N}\sum_{i\in N}\max_{\|\delta_{i}\|\leq\rho} \mathbbm{E}_{i}\mathbbm{E}_{\xi_{i}\sim D_{i}}f_{i}(\theta_{i}+\delta_{i},\xi_ {i})\big{\}}$ (5) $$

Therefore, we can obtain that at $t$ round, $k$ moments, the virtual global perturbation variable $\delta^{t}_{k}=\rho\frac{\nabla f(\theta^{t})}{\|\nabla f(\theta^{t})\|}=\rho
\frac{\sum_{i\in N}\nabla f_{i}(\theta^{t}_{k})}{\|\sum_{i\in N}\nabla f_{i}(
\theta^{t}_{k})\|}\simeq\rho\frac{\sum_{i\in S}\nabla f_{i}(\theta^{t}_{k})}{
\|\sum_{i\in S}\nabla f_{i}(\theta^{t}_{k})\|}$. The $\theta_{k}^{t}$ denotes the global model at virtual moment $k$, which is computed as $\theta^{t}_{k}=\frac{1}{M}\sum_{i\in S_{t}}\theta^{t}_{i,k}$.
However, due to the CTA strategy in the FL paradigm, the set of clients does not have effective access to the global model $\theta_{k}^{t}$ at each moment in time, and thus the global perturbation $\delta$ cannot be computed correctly. Inspired by the FedCM[], strategy, we estimate the global update $\Delta^{t}\simeq\nabla f(\theta^{t})$ by passing the global update variable. Finally, we define the update strategy for the global perturbation SAM of FedTOGA as follows:
$\delta_{k}^{t}=\rho\frac{\nabla f(\theta^{t})}{\|\nabla f(\theta^{t})\|}\simeq
\rho\frac{g_{i,k}^{t}+\kappa\Delta^{t}}{\|g_{i,k}^{t}+\kappa\Delta^{t}\|};
\theta_{i,k}^{t}=\theta_{i,k-1}^{t}-\eta_{l}\nabla F_{i}(\theta_{i,k}^{t}+\rho
\delta_{k}^{t})$. The differences between the FedTOGA perturbation strategy and the rest of the similar works can be viewed in Appendix A.2 Tab.4.

### 4.2 Utilize Neighbourhood Perturbation

Besides, as stated by FedLESAM[], local perturbations require additional gradient ascent computation, which may consume additional computational overhead. Therefore, how to estimate the local perturbation without utilizing additional computation? We propose neighborhood perturbation(N-perturbation) for the first time. Specifically, when the client’s local iteration interval exceeds one, the local perturbation gradient $\tilde{g}_{i,k-1}^{t}$ will be recorded by the cache without opening additional storage space. We can get $g_{i,k}\simeq\tilde{g}_{i,k-1}^{t}$. We can further replace the perturbation term in the local SAM optimization and get: $\delta_{i,k}^{t}=\rho\frac{\tilde{g}_{i,k-1}^{t}+\kappa\Delta^{t}}{\|\tilde{g}
_{i,k-1}^{t}+\kappa\Delta^{t}\|}$. This operation allows approximate estimation of local perturbations in environments where the client-side resource is scarce.

Perturbation Fusion?
In the FL paradigm, the edge client SAM only captures the sharpness of a specific small batch of data, which is mitigated effect by the G-Perturbation technique described above to enhance generalization. Let’s further think about whether N-Perturbation may bring additional benefits in addition to alleviating computational overhead. Similar to LookAhead[], it backtracks by perturbing ascent after each gradient descent. Then, our perturbation calculation can be rewritten:
$\delta_{i,k}^{t}=\rho\frac{g_{i,k}+\tilde{g}_{i,k-1}^{t}+\kappa\Delta^{t}}{\|g
_{i,k}+\tilde{g}_{i,k-1}^{t}+\kappa\Delta^{t}\|}$.
A more in-depth discussion of N-perturbation can be found in Appendix A.3.

### 4.3 Global Correction in Dynamic Regularizer

In order to effectively avoid performance degradation and further improve the optimization objective consistency, we also adopt dynamic regularization[] that merges the global update $\Delta$ correction on each local client, which takes the form of an ADMM-like method in order to effectively minimize the global objective $f$. This is the first FL framework that considers local dual variable corrections. First, we consider a centralized global Augmented Lagrangian(AL) function $f_{cen}$ for which the correction introduces a penalty term $\theta=\theta_{i}$ constraint as:

$$ $f_{cen}:\frac{1}{N}\sum_{i\in N}\left\{f_{i}+\left\langle h^{t},\theta^{t}- \theta_{i}^{t}\right\rangle+\frac{1}{2\alpha}\|\theta^{t}-\theta^{t}_{i}\|^{2}\right\}$ (6) $$

In the case of centralized learning, the dual variables are updated promptly, via $h^{t}=h^{t-1}-\frac{1}{\alpha N}\sum_{i}(\theta^{t}_{i}-\theta^{t-1})$. However, FL’s CTA policy makes the dual variables forced to decompose to update locally as follows:

$$ $f_{fed}:\frac{1}{N}\sum_{i\in N}\left\{f_{i}+\left\langle h_{i}^{t},\theta^{t} -\theta_{i}^{t}\right\rangle+\frac{1}{2\alpha}\|\theta^{t}-\theta^{t}_{i}\|^{2 }\right\}$ (7) $$

where the global dual variable $h$ is updated at each communication and the local dual variable $h_{i}$ is locally staged. Thus dual variable differences are amplified as the local interval expands, which has not been addressed in previous studies[, ]. Thus, similar to the strategy for correcting global perturbations (in Sec. 4.1), we employ proximity global update correction to mitigate the difference between the local dual variable and the global dual variable. Specifically, we cause the local dual variables to approximate the global dual variables by adding corrections and obtaining $h_{i}-\beta\Delta^{t}\simeq h$. Locally in each subproblem, we solve the local model $\theta_{i}$ by minimizing the corrected AL function:

$$ $\theta_{i,K}^{t}=\mathop{\min}_{\theta_{i}}\left\{f_{i}-\left\langle h_{i}^{t} -\beta\Delta^{t},\theta^{t}_{i}\right\rangle+\frac{1}{2\alpha}\|\theta^{t}- \theta^{t}_{i}\|^{2}\right\}$ (8) $$

Again, in order not to affect the original SAM we use SGD to solve this problem []. We then update the dual variables locally $h_{i}^{t+1}=h_{i}^{t}-\frac{1}{\alpha}\left(\theta_{i,K}^{t}-\theta_{i,0}^{t}\right)$. After finishing the local training, update $\theta^{t}$ to $\theta^{t+1}$ by solving the equation 7 and start the next iteration.

### 4.4 Overview of FedTOGA

Algorithm1 shows the detailed flow of FedTOGA. First initialize the server-side global model $\theta$. In the global synchronization round $t$, a set $S_{t}$ containing $M$ clients is randomly selected from all clients $N$, and the global model $\theta_{t}$ is sent to the set of authorized clients $S_{t}$ with the global update $\Delta^{t}$ of the $t-1$ round. The client first computes the original gradient $g_{i,k}^{t}$ according to line 16 of the algorithm, and subsequently, computes the SAM gradient $\delta_{i,k}^{t}$ corrected by $\Delta^{t}$ in line 17, with the neighborhood perturbation variable $\tilde{g_{i}}$ being optional. In line 19, we use the formula 8 for local dual variable correction to update the local model $\theta_{i}$, followed by updating the local dual variable $h_{i}$ via line 21. After local training is complete, FedTOGA sends only $\theta_{i}^{t}$ to the server for aggregation. In lines 9-11 of the algorithm, the server updates the global model from $\theta^{t}$ to $\theta^{t+1}$ by minimizing the formula 7. This process is repeated until $T-1$.

## 5 Theoretical Analysis

###### Assumption 1 .

The loss function $f_{i}$ is $L$-Smooth, i.e.,$f_{i}(y)-f_{i}(x)\leq\langle\nabla f_{i}(x),y-x\rangle+\frac{L}{2}\|y-x\|^{2}.$

###### Assumption 2 .

Unbiased and bounded variance of stochastic gradient. The stochastic gradient $\tilde{\nabla}f_{i}(x)=\nabla f_{i}(x,\xi_{i})$ computed by the $i$-th client using mini-batch $\xi$ is an unbiased estimator of $\nabla f_{i}(x)$, i.e.
$\mathbb{E}[\tilde{\nabla}f_{i}(x)]=\nabla f_{i}(x),\mathbb{E}{\|\tilde{\nabla}
f_{i}(x)-\nabla f_{i}(x)\|}^{2}\leq\sigma^{2}_{l}.$

###### Assumption 3 .

Bounded Heterogeneity, for all $x\in\mathbb{R}^{d}$, we establish the following inequality:$\mathbbm{E}\|\nabla f_{i}(x)-\nabla f(x)\|\leq\sigma_{g}$
Besides, the variance of unit gradient is bounded:
$\mathbbm{E}\|\frac{\nabla f_{i}(x)}{\|\nabla f_{i}(x)\|}-\frac{\nabla f(x)}{\|
\nabla f(x)\|}\|\leq\sigma_{g}^{\prime}.$

###### Theorem 1 .

Under Assumption 1-3, For any training interval $t$ on $i$-th client, model divergence satisfies:

$$ $\displaystyle\|\theta_{i,k}^{t}-v_{k}^{t}\|^{2}\leq H_{i}(k)$ (9) $$

where $H_{i}(\tau)\leq\frac{L^{2}\rho^{2}\sigma_{g}^{{}^{\prime}2}+\sigma_{g}^{2}}{2L
^{2}}((1+2\eta_{l}^{2}L^{2})^{\tau}-1)$, $\{v^{t}\}$is a virtual sequence representing the global model.
More Details can be referred to the Appendix.

###### Remark 1 .

The difference between the local model and the global model will be geometrically amplified as the local interval expands, mainly from the model perturbation error and update error, and thus it is reasonable to enhance the consistency
of optimization and generalization objective (in Sec. 4).

###### Theorem 2 .

Under Assumption 1-3. When $\eta_{l}\leq\min\{\frac{c_{1}}{\sqrt{1008L^{2}K+72L^{2}K\beta^{2}}},2\alpha\}$, $c_{1}=\sqrt{1/2-L\alpha\beta^{2}}>0$ and the perturbation learning rate satisfies $\rho=O(1/\sqrt{T})$, and local interval $K>\frac{\alpha}{\eta_{l}}$, let $\Gamma=\frac{1}{2}-1008\eta_{l}^{2}L^{2}K-72\eta_{l}^{2}L^{2}K\beta^{2}-L
\alpha\beta^{2}$ is a positive constant with select the suitable $\eta_{l}$, the auxiliary sequence $\{z^{t}\}$ generated by executing the Algorithm 1 satisfies:

$$ $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\leq$ $\displaystyle\frac{1}{T\alpha\Gamma}\left(\nabla f(z^{0})-f^{*}\right)+\frac{1 6\alpha^{3}L^{2}}{T\Gamma}\mathbbm{E}_{t}\left\|\frac{1}{N}\sum_{i\in N}h_{i}^ {0}\right\|^{2}$ $\displaystyle+\frac{144\alpha L^{2}\eta_{l}^{2}K}{TN\Gamma}\sum_{i\in N} \mathbbm{E}\|h_{i}^{0}\|^{2}+\Upsilon$ (10) $$

where the $f^{*}$ is the optimal of non-convex function $f$, and the term $\Upsilon$ is:

$$ $\Upsilon=\frac{1}{\alpha\Gamma}\left(72\eta_{l}^{2}L^{2}K(24\sigma_{l}^{2}+2 \sigma_{l}^{2}+6L^{2}\rho^{2})+3\alpha L^{2}\rho^{2}\right)$ $$

More Details can be referred to the Appendix.

###### Remark 2 .

When we set the local learning rate $\eta_{l}$ to satisfy $\eta_{l}=O(1/K)$, $\eta_{l}\geq\alpha/K$, and the perturbation learning rate to be $O(1/T)$, FedTOGA can achieve a fast convergence rate of $O(1/T)$ when the local interval $K$ satisfies $K=O(T)$.

###### Remark 3 .

Inspired by the FedSpeed[], the FedTOGA can speed up convergence by increasing the setting of the local interval $K$, which is useful for bandwidth-constrained FL systems. However, the local perturbation learning rate $\rho$ in FedSpeed restricts the upper bound, and our proof slightly relaxes the limitation so that the perturbation learning rate just satisfies $O(1/T)$.

**Table 1: Dirichlet coefficients $u$ selected from $\{0.1,0.6\}$, and $c$ is the Pathological coefficient, i.e., the number of active categories in each client. The two datasets have 100 clients in the upper part with 10% active in each round, 200 clients in the lower part with 5% active in each round.(LeNet)**
| Method | CIFAR10 | CIFAR100 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Partition | Dirichlet | Pathological | Dirichlet | Pathological |  |  |  |  |
| Coefficient | $u=0.6$ | $u=0.1$ | $c=6$ | $c=3$ | $u=0.6$ | $u=0.1$ | $c=20$ | $c=10$ |
| FedAvg [] | $80.28^{\pm 0.14}$ | $74.68^{\pm 0.19}$ | $80.59^{\pm 0.18}$ | $78.10^{\pm 0.23}$ | $47.35^{\pm 0.16}$ | $45.56^{\pm 0.20}$ | $46.46^{\pm 0.20}$ | $43.43^{\pm 0.27}$ |
| FedAdam | $80.39^{\pm 0.17}$ | $71.52^{\pm 0.29}$ | $81.02^{\pm 0.20}$ | $77.88^{\pm 0.23}$ | $48.94^{\pm 0.21}$ | $43.62^{\pm 0.25}$ | $44.86^{\pm 0.25}$ | $41.58^{\pm 0.27}$ |
| FedYogi [] | $80.11^{\pm 0.19}$ | $73.58^{\pm 0.25}$ | $81.08^{\pm 0.21}$ | $78.10^{\pm 0.20}$ | $48.41^{\pm 0.21}$ | $45.44^{\pm 0.22}$ | $46.18^{\pm 0.22}$ | $42.07^{\pm 0.25}$ |
| SCAFFOLD[] | $82.87^{\pm 0.12}$ | $78.00^{\pm 0.16}$ | $83.31^{\pm 0.10}$ | $80.29^{\pm 0.15}$ | $53.68^{\pm 0.21}$ | $50.33^{\pm 0.24}$ | $51.30^{\pm 0.22}$ | $47.71^{\pm 0.22}$ |
| FedACG[] | $82.87^{\pm 0.14}$ | $77.51^{\pm 0.16}$ | $82.86^{\pm 0.12}$ | $80.84^{\pm 0.17}$ | $52.88^{\pm 0.20}$ | $48.72^{\pm 0.23}$ | $50.24^{\pm 0.21}$ | $46.08^{\pm 0.24}$ |
| FedCM[] | $77.04^{\pm 0.30}$ | $62.75^{\pm 0.31}$ | $66.58^{\pm 0.29}$ | $71.20^{\pm 0.33}$ | $43.08^{\pm 0.19}$ | $34.69^{\pm 0.26}$ | $36.27^{\pm 0.18}$ | $28.48^{\pm 0.30}$ |
| FedDyn[] | $82.31^{\pm 0.13}$ | $78.05^{\pm 0.19}$ | $83.13^{\pm 0.18}$ | $79.96^{\pm 0.19}$ | $49.97^{\pm 0.19}$ | $45.85^{\pm 0.29}$ | $47.41^{\pm 0.21}$ | $43.29^{\pm 0.19}$ |
| FedDC[] | $83.58^{\pm 0.14}$ | $78.50^{\pm 0.19}$ | $84.00^{\pm 0.16}$ | $81.72^{\pm 0.17}$ | $51.99^{\pm 0.15}$ | $48.75^{\pm 0.21}$ | $49.53^{\pm 0.19}$ | $44.82^{\pm 0.23}$ |
| FedRCL [] | $77.62^{\pm 0.11}$ | $68.79^{\pm 0.16}$ | $78.28^{\pm 0.15}$ | $76.04^{\pm 0.19}$ | $46.34^{\pm 0.24}$ | $42.28^{\pm 0.17}$ | $44.06^{\pm 0.19}$ | $39.64^{\pm 0.21}$ |
| FedSAM | $81.58^{\pm 0.15}$ | $77.67^{\pm 0.15}$ | $82.15^{\pm 0.17}$ | $79.23^{\pm 0.23}$ | $48.08^{\pm 0.21}$ | $46.86^{\pm 0.26}$ | $46.71^{\pm 0.25}$ | $43.41^{\pm 0.22}$ |
| MoFedSAM [] | $77.17^{\pm 0.12}$ | $66.24^{\pm 0.15}$ | $77.44^{\pm 0.15}$ | $72.15^{\pm 0.19}$ | $43.30^{\pm 0.18}$ | $34.43^{\pm 0.21}$ | $36.50^{\pm 0.19}$ | $29.92^{\pm 0.24}$ |
| FedGAMMA[] | $83.88^{\pm 0.13}$ | $78.61^{\pm 0.15}$ | $83.79^{\pm 0.14}$ | $79.68^{\pm 0.15}$ | $53.94^{\pm 0.20}$ | $49.95^{\pm 0.24}$ | $51.20^{\pm 0.22}$ | $48.11^{\pm 0.29}$ |
| FedSMOO [] | 84.82^±0.15 | 80.06^±0.16 | 85.07^±0.17 | $81.26^{\pm 0.19}$ | 56.57^±0.18 | $52.17^{\pm 0.17}$ | $53.42^{\pm 0.21}$ | $48.12^{\pm 0.19}$ |
| FedSpeed [] | $84.14^{\pm 0.15}$ | $80.16^{\pm 0.16}$ | $84.74^{\pm 0.14}$ | 82.20^±0.19 | $53.96^{\pm 0.19}$ | 52.29^±0.21 | 53.78^±0.18 | 48.33^±0.20 |
| FedLESAM | $80.94^{\pm 0.18}$ | $77.02^{\pm 0.15}$ | $81.79^{\pm 0.18}$ | $78.85^{\pm 0.15}$ | $48.13^{\pm 0.18}$ | $46.55^{\pm 0.21}$ | $46.08^{\pm 0.23}$ | $43.57^{\pm 0.17}$ |
| FedLESAM-D | $83.28^{\pm 0.15}$ | $79.12^{\pm 0.18}$ | $84.20^{\pm 0.19}$ | $80.91^{\pm 0.16}$ | $54.88^{\pm 0.18}$ | $52.08^{\pm 0.22}$ | 54.14^±0.19 | $48.28^{\pm 0.22}$ |
| FedLESAM-S[] | $83.39^{\pm 0.12}$ | $78.23^{\pm 0.17}$ | $83.99^{\pm 0.19}$ | $81.20^{\pm 0.15}$ | $53.29^{\pm 0.15}$ | $50.12^{\pm 0.21}$ | $52.20^{\pm 0.20}$ | $47.29^{\pm 0.17}$ |
| FedTOGA(ours) | 86.01^±0.12 | 82.05^±0.11 | 85.71^±0.13 | 84.00^±0.12 | 57.25^±0.13 | 53.45^±0.13 | 55.49^±0.13 | 51.27^±0.18 |
| FedAvg [] | $77.53^{\pm 0.17}$ | $74.60^{\pm 0.23}$ | $79.21^{\pm 0.25}$ | $76.20^{\pm 0.23}$ | $43.86^{\pm 0.21}$ | $42.70^{\pm 0.24}$ | $42.94^{\pm 0.25}$ | $42.28^{\pm 0.29}$ |
| FedAdam | $79.39^{\pm 0.19}$ | $74.49^{\pm 0.31}$ | $79.53^{\pm 0.23}$ | $76.09^{\pm 0.25}$ | $45.34^{\pm 0.25}$ | $42.79^{\pm 0.23}$ | $43.57^{\pm 0.25}$ | $40.66^{\pm 0.29}$ |
| FedYogi [] | $79.95^{\pm 0.21}$ | $75.29^{\pm 0.25}$ | $79.73^{\pm 0.22}$ | $77.64^{\pm 0.23}$ | $46.67^{\pm 0.25}$ | $43.02^{\pm 0.24}$ | $44.70^{\pm 0.27}$ | $41.33^{\pm 0.30}$ |
| SCAFFOLD[] | $81.18^{\pm 0.15}$ | $76.11^{\pm 0.19}$ | $82.44^{\pm 0.17}$ | $78.52^{\pm 0.17}$ | $51.45^{\pm 0.25}$ | $47.19^{\pm 0.27}$ | $48.26^{\pm 0.28}$ | $46.82^{\pm 0.26}$ |
| FedACG[] | $82.57^{\pm 0.17}$ | $78.47^{\pm 0.20}$ | $82.09^{\pm 0.16}$ | $80.50^{\pm 0.19}$ | $51.96^{\pm 0.24}$ | $49.34^{\pm 0.26}$ | $50.01^{\pm 0.27}$ | $46.82^{\pm 0.25}$ |
| FedCM[] | $76.08^{\pm 0.30}$ | $64.33^{\pm 0.31}$ | $76.64^{\pm 0.29}$ | $68.61^{\pm 0.33}$ | $40.32^{\pm 0.19}$ | $33.05^{\pm 0.26}$ | $34.19^{\pm 0.18}$ | $27.88^{\pm 0.30}$ |
| FedDyn[] | $80.60^{\pm 0.17}$ | $77.53^{\pm 0.21}$ | $81.54^{\pm 0.22}$ | $79.39^{\pm 0.24}$ | $48.40^{\pm 0.20}$ | $45.04^{\pm 0.31}$ | $46.87^{\pm 0.24}$ | $43.04^{\pm 0.29}$ |
| FedDC[] | $81.83^{\pm 0.17}$ | $78.87^{\pm 0.21}$ | $82.44^{\pm 0.17}$ | $80.93^{\pm 0.19}$ | $48.74^{\pm 0.19}$ | $45.11^{\pm 0.26}$ | $45.94^{\pm 0.22}$ | $43.94^{\pm 0.27}$ |
| FedRCL [] | $76.06^{\pm 0.15}$ | $66.88^{\pm 0.19}$ | $76.51^{\pm 0.19}$ | $72.28^{\pm 0.23}$ | $42.05^{\pm 0.27}$ | $38.60^{\pm 0.20}$ | $40.56^{\pm 0.24}$ | $37.28^{\pm 0.26}$ |
| FedSAM | $79.74^{\pm 0.18}$ | $74.69^{\pm 0.19}$ | $79.87^{\pm 0.18}$ | $76.90^{\pm 0.23}$ | $44.78^{\pm 0.25}$ | $43.50^{\pm 0.24}$ | $44.14^{\pm 0.29}$ | $43.36^{\pm 0.25}$ |
| MoFedSAM [] | $76.36^{\pm 0.15}$ | $65.74^{\pm 0.19}$ | $76.74^{\pm 0.17}$ | $70.74^{\pm 0.21}$ | $41.07^{\pm 0.19}$ | $34.11^{\pm 0.23}$ | $35.91^{\pm 0.17}$ | $28.55^{\pm 0.27}$ |
| FedGAMMA[] | $80.89^{\pm 0.17}$ | $75.34^{\pm 0.19}$ | $81.73^{\pm 0.16}$ | $78.74^{\pm 0.19}$ | $49.78^{\pm 0.25}$ | $46.31^{\pm 0.27}$ | $47.91^{\pm 0.26}$ | $45.26^{\pm 0.33}$ |
| FedSMOO [] | 84.17^±0.19 | 80.92^±0.17 | 84.78^±0.19 | 82.79^±0.21 | 52.31^±0.24 | 49.42^±0.20 | $50.59^{\pm 0.21}$ | $46.08^{\pm 0.25}$ |
| FedSpeed [] | $82.76^{\pm 0.19}$ | $79.95^{\pm 0.19}$ | $83.36^{\pm 0.18}$ | $80.72^{\pm 0.22}$ | $49.93^{\pm 0.23}$ | $49.04^{\pm 0.24}$ | 50.61^±0.23 | 46.85^±0.25 |
| FedLESAM | $80.11^{\pm 0.23}$ | $74.35^{\pm 0.22}$ | $78.35^{\pm 0.21}$ | $71.23^{\pm 0.25}$ | $44.35^{\pm 0.19}$ | $43.75^{\pm 0.21}$ | $43.97^{\pm 0.23}$ | $43.21^{\pm 0.22}$ |
| FedLESAM-D | $83.26^{\pm 0.19}$ | $79.89^{\pm 0.20}$ | $83.99^{\pm 0.23}$ | $81.89^{\pm 0.21}$ | $49.77^{\pm 0.20}$ | $45.35^{\pm 0.22}$ | $50.58^{\pm 0.19}$ | $46.55^{\pm 0.21}$ |
| FedLESAM-S[] | $83.76^{\pm 0.17}$ | $79.02^{\pm 0.18}$ | $83.12^{\pm 0.20}$ | $81.57^{\pm 0.21}$ | $49.52^{\pm 0.19}$ | $47.83^{\pm 0.22}$ | $48.21^{\pm 0.20}$ | $45.75^{\pm 0.24}$ |
| FedTOGA(ours) | 84.91^±0.15 | 81.78^±0.17 | 84.90^±0.19 | 83.49^±0.14 | 54.90^±0.16 | 51.00^±0.15 | 53.25^±0.17 | 49.90^±0.21 |

**Table 2: Dirichlet coefficients $u$ selected from $\{0.1,0.6\}$, and $c$ is the Pathological coefficient, i.e., the number of active categories in each client. The CIFAR10 has 100 clients in the left part with 10% active in each round, and 200 clients in the right part with 5% active in each round.(ResNet18)**
| Method | CIFAR10 |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Partition | Dirichlet | Pathological | Dirichlet | Pathological |  |  |  |  |
| Coefficient | $u=0.6$ | $u=0.1$ | $c=6$ | $c=3$ | $u=0.6$ | $u=0.1$ | $c=6$ | $c=3$ |
| FedAvg [] | $79.52^{\pm 0.13}$ | $76.00^{\pm 0.18}$ | $79.91^{\pm 0.17}$ | $74.08^{\pm 0.22}$ | $75.90^{\pm 0.21}$ | $72.93^{\pm 0.19}$ | $77.47^{\pm 0.34}$ | $71.68^{\pm 0.34}$ |
| FedAdam[] | $77.08^{\pm 0.31}$ | $73.41^{\pm 0.33}$ | $77.05^{\pm 0.26}$ | $72.44^{\pm 0.20}$ | $75.55^{\pm 0.38}$ | $69.70^{\pm 0.32}$ | $75.74^{\pm 0.22}$ | $70.49^{\pm 0.26}$ |
| SCAFFOLD[] | $81.81^{\pm 0.17}$ | $78.57^{\pm 0.14}$ | $83.07^{\pm 0.10}$ | $77.02^{\pm 0.18}$ | $79.00^{\pm 0.26}$ | $76.15^{\pm 0.15}$ | $80.69^{\pm 0.21}$ | $74.05^{\pm 0.31}$ |
| FedCM[] | $82.97^{\pm 0.21}$ | $77.82^{\pm 0.16}$ | $83.44^{\pm 0.17}$ | $77.82^{\pm 0.19}$ | $80.52^{\pm 0.29}$ | $77.28^{\pm 0.22}$ | $81.76^{\pm 0.24}$ | $76.72^{\pm 0.25}$ |
| FedDyn[] | $83.22^{\pm 0.18}$ | $78.08^{\pm 0.19}$ | $83.18^{\pm 0.17}$ | $77.63^{\pm 0.14}$ | $80.69^{\pm 0.23}$ | $76.82^{\pm 0.17}$ | $82.21^{\pm 0.18}$ | $74.93^{\pm 0.22}$ |
| FedSAM | $81.46^{\pm 0.12}$ | $77.03^{\pm 0.17}$ | $81.13^{\pm 0.23}$ | $78.30^{\pm 0.24}$ | $78.32^{\pm 0.16}$ | $74.00^{\pm 0.14}$ | $78.75^{\pm 0.27}$ | $75.12^{\pm 0.29}$ |
| MoFedSAM [] | $85.29^{\pm 0.13}$ | $80.25^{\pm 0.17}$ | $84.74^{\pm 0.16}$ | $83.09^{\pm 0.24}$ | $84.76^{\pm 0.20}$ | 80.10^±0.14 | 85.00^±0.27 | 82.13^±0.23 |
| FedGAMMA[] | $82.82^{\pm 0.16}$ | $79.91^{\pm 0.15}$ | $83.51^{\pm 0.18}$ | $77.11^{\pm 0.14}$ | $80.72^{\pm 0.19}$ | $76.70^{\pm 0.14}$ | $81.81^{\pm 0.27}$ | $77.44^{\pm 0.29}$ |
| FedSMOO [] | 86.08^±0.14 | 81.80^±0.18 | 86.38^±0.15 | 82.79^±0.16 | 84.96^±0.19 | $79.76^{\pm 0.19}$ | $84.82^{\pm 0.18}$ | $81.01^{\pm 0.19}$ |
| FedSpeed [] | $86.01^{\pm 0.16}$ | $81.02^{\pm 0.16}$ | $86.09^{\pm 0.19}$ | $82.50^{\pm 0.16}$ | $84.12^{\pm 0.18}$ | $76.74^{\pm 0.14}$ | $84.78^{\pm 0.27}$ | $79.09^{\pm 0.29}$ |
| FedLESAM | $81.04^{\pm 0.19}$ | $76.92^{\pm 0.16}$ | $81.37^{\pm 0.17}$ | $78.21^{\pm 0.21}$ | $77.80^{\pm 0.18}$ | $73.73^{\pm 0.22}$ | $78.44^{\pm 0.20}$ | $74.53^{\pm 0.19}$ |
| FedLESAM-D | $84.27^{\pm 0.14}$ | $80.08^{\pm 0.19}$ | $85.62^{\pm 0.18}$ | $83.00^{\pm 0.22}$ | $82.53^{\pm 0.19}$ | $79.56^{\pm 0.27}$ | 85.04^±0.21 | 81.10^±0.19 |
| FedLESAM-S[] | $84.94^{\pm 0.12}$ | $79.52^{\pm 0.17}$ | $85.88^{\pm 0.19}$ | $82.18^{\pm 0.15}$ | $83.22^{\pm 0.22}$ | $78.69^{\pm 0.17}$ | $85.02^{\pm 0.24}$ | $80.57^{\pm 0.17}$ |
| FedTOGA(ours) | 86.99^±0.13 | 83.16^±0.17 | 87.21^±0.18 | 84.55^±0.15 | 85.21^±0.17 | 81.60^±0.16 | 85.24^±0.19 | 83.25^±0.20 |
| Note: The extended table sees Tab. 7 |  |  |  |  |  |  |  |  |

**Table 3: WALL-CLOCK Time Comparison**
| Method | R(80%) | Cost | R(82%) | Cost |
| --- | --- | --- | --- | --- |
| FedSAM | 481 | 3.6$\times$ | 800+ | 4.7$\times$ |
| MoFedSAM | 167 | 1.2$\times$ | 270 | 1.6$\times$ |
| FedGAMMA | 458 | 3.4$\times$ | 630 | 3.7$\times$ |
| FedSpeed | 262 | 1.9$\times$ | 318 | 1.9$\times$ |
| FedSMOO | 190 | 1.4$\times$ | 253 | 1.5$\times$ |
| FedLESAM-D | 248 | 1.8$\times$ | 418 | 2.5$\times$ |
| FedTOGA | 135 | 1.0$\times$ | 170 | 1.0$\times$ |
| Note: The extended table sees Tab. 8 |  |  |  |  |

###### Assumption 1 .

###### Assumption 2 .

###### Assumption 3 .

###### Theorem 1 .

###### Remark 1 .

###### Theorem 2 .

###### Remark 2 .

###### Remark 3 .

## 6 Experiments

This section describes part of the experimental setup, including the baseline algorithm, dataset, segmentation, and experimental details. We then show performance results on the baseline dataset, followed by extensive further analysis such as visualization as well as parameter sensitivity analysis experiments.

### 6.1 Experimental Setups.

Baselines. We compare FedTOGA with the baseline FedAvg[] and existing SAM-base FL methods, including FedSAM, MoFedSAM[], FedGAMMA[], FedSMOO[], FedSpeed[], with the recent FedLESAM[]. Also, we compare with momentum-based FL algorithms, for example, FedAdam, FedYogi[], FedACG[], FedCM[]. In addition, methods based on local consistency are also included in the comparison, including FedDyn[], SCAFFOLD[] , FedDC[] with FedRCL [].

Dataset and Splits.
The benchmark datasets CIFAR10 and CIFAR100 are utilized in the experiments. We follow the methodologies outlined in [, , ] to simulate client data using Dirichlet and Pathological splits in non-IID scenarios. To emulate real-world conditions where a large number of clients are involved, some of whom may lag, we conducted three experimental setups: First, data is distributed to 100 clients, with 10% of clients participating in training in each round. Second, data is distributed to 200 clients, with 5% of clients participating in each training round.

Experimental Details.
For a fair comparison, we adopt the same experimental setup as in [, ]. We use SGD as the optimizer, with a client learning rate $\eta_{l}$ set to 0.1 and a global learning rate of 1. The weight decay is fixed at $1e^{-3}$. To further assess the generalization capability of our method, we conduct experiments with two models, LeNet and ResNet18[]. For LeNet, the learning rate decays by 0.997 per epoch, whereas for ResNet18[], it decays by 0.998. For CIFAR10, the batch size is set to 50, and the number of local epochs is set to 5. For CIFAR100, the batch size is set to 20, and the number of local epochs is set to 2. In FedTOGA, the local perturbation coefficient $\kappa$ is set to 1, the dual variable coefficient $\beta$ is set to 0.9, and the penalized coefficient $\alpha$ is set to 0.1. Consistent with several other algorithms, the perturbation magnitude $\rho$ is set to 0.1, except for FedSAM, MoFedSAM, and FedLESAM, where it is set to 0.01. Detailed information about the experimental setup can be found in Appendices B.2.

### 6.2 Performance Evaluation

Figure: (a) *
Refer to caption: extracted/5820635/Figure/lr.png

Performance with compared benchmarks.
As shown in Tables 1 and 2, the proposed FedTOGA algorithm shows excellent performance on various heterogeneous datasets in terms of convergence speed and final achieved accuracy. Table 1, which details the test accuracy of the LeNet model, demonstrates that FedTOGA significantly outperforms other algorithms with different heterogeneous data conditions. Specifically, under the Dirichlet-0.1 setting on the CIFAR10 dataset, FedTOGA attains an accuracy of 82.05%, marking a significant improvement of over 7.37% compared to vanilla FedAvg and a 1.99% increase over the second-highest baseline accuracy. Similar results are observed in Table 2 for the ResNet18 model, FedTOGA outperforms all current baseline algorithms.

As seen in Table 3, FedTOGA also exhibits a significant advantage in terms of convergence speed. When reaching 80% accuracy, FedTOGA converges 3.6× faster compared to FedSAM and 1.2× faster compared to the second-best algorithm. Similarly, when reaching 82% accuracy, FedTOGA converges 4.7× faster compared to FedSAM and 1.5× faster compared to the second-best algorithm. This indicates that FedTOGA achieves the target accuracy with significantly reduced computation and communication overhead compared to other methods. The exceptional global consensus capability of FedTOGA enables it to more effectively mitigate the impact of data heterogeneity and enable faster convergence.

Impact of heterogeneity. We use the Dirichlet and Pathological methods for data partitioning. For the Dirichlet distribution, we adopt with variance coefficients $u$ of 0.1 and 0.6. For the Pathological distribution, we use coefficients $c$ of 3 and 6. As shown in Tables 1 and 2, increased data heterogeneity leads to a decrease in accuracy across all algorithms. However, FedTOGA exhibits the smallest accuracy drop. Specifically, for the Resnet18 model, as $u$ changes from 0.6 to 0.1 under the Dirichlet distribution on the CIFAR10 dataset, FedSAM’s accuracy decreases from 81.46% to 77.03%, a 4.43% reduction, while the second-best algorithm, FedSMOO, shows a drop from 86.08% to 81.80%, a 4.28% reduction. In contrast, FedTOGA’s accuracy declines from 86.99% to 83.16%, a 3.83% drop. Similar trends are observed under the Pathological distribution, underscoring FedTOGA’s superior stability and accuracy across varying levels of data heterogeneity.

Impact of partial participation.
We fix all hyperparameters except for the client participation rate to assess its effect on accuracy. As illustrated in Table 2, a reduction in the client participation rate from 10% to 5% results in a modest decline in accuracy across all algorithms. For instance, on the CIFAR10 dataset, under the challenging pathological distribution with $c=3$, FedTOGA’s accuracy decreases marginally from 84.55% to 83.35%, a reduction of just 1.40%, while FedSMOO experiences a sharper decline from 82.79% to 81.01%, a reduction of 1.78%. Similarly, under the Dirichlet distribution with $u=0.1$, FedTOGA’s accuracy decreases from 86.99% to 85.21%, a decrease of 1.78%, whereas FedSMOO’s accuracy drops from 86.08% to 84.96%, a reduction of 1.12%. Notably, despite these reductions, FedTOGA consistently outperforms other algorithms in terms of accuracy, highlighting its robust generalization capability and stability.

Hyperparameters Sensitivity.
We study the sensitivity of the hyperparameters: learning rate decay, penalty coefficient $\alpha$, correction coefficients $\beta$ and $\kappa$, and perturbation coefficient $\rho$. As shown in Figure 2, our extensive experiments demonstrate FedTOGA’s resilience to variations in these hyperparameters. By systematically adjusting each parameter while holding the others constant, we found that FedTOGA remains remarkably stable under changes in learning rate decay and the correction coefficients $\beta$ and $\kappa$. Additionally, the penalty coefficient $\alpha$ and perturbation coefficient $\rho$ effectively maintain robust performance when appropriately selected.

## 7 Conclusion

In this paper, we propose a novel federated learning algorithm, FedTOGA, which for the first time estimates global perturbations by combining global training gradients, and corrects the local dynamic regularizer. This ensures that local optima can be effectively aligned with the global generalization and optimization objectives. FedTOGA facilitates the efficient search for globally consistent flat minima and accelerates convergence without incurring additional local storage or uplink communication overhead. Theoretical analysis guarantees that FedTOGA achieves a fast convergence rate of $O(1/T)$. Extensive experiments were conducted to verify its efficiency and remarkable performance.

## Appendix A Related Works

### A.1 Literature review

In this section, we review the contributions of related works.

Federated Learning
Federated learning gained widespread attention upon its introduction due to its data-exchange-free nature. FedAvg[] As its foundational framework, it allows for collaborative modeling by passing models around without exchanging data[]. However, due to various irresistible factors, the data of cooperative devices show heterogeneous distribution, which makes the modeling effectiveness suffer. Therefore, many studies based on empirical loss minimization have been proposed to solve this problem. FedProx[] employs a simple and intuitive practice, namely, paradigm constraints, which is used to ensure that the local model is never far from the global model. Specifically, it introduces in regular terms during local training to limit the distance of its local model from the global model. SCAFFOLD [], Mime [] uses control variables for local updates, however, its requires greater communication overhead. FedDyn [], FedPD [] considers the inconsistency of the local optimal point with the global optimal point to be a fundamental dilemma, which aligns the locally optimal solution to the global optimal solution via a dynamic regularizer. FedPA [] removes bias from client updates by estimating a global posterior. FedDC [] takes decoupled local as well as updated quantities to mitigate heterogeneity. Furthermore, recent research has shown that this model bias is similar to catastrophic forgetting in continuous learning [, , , ], Clients overriding previously important parameters to learn a new task resulted in disrupting pretask performance. Some studies have mitigated global knowledge collapse by class task recall[, ]. Server momentum[] algorithms also play an important role in FL. [] investigates the convergence failure of ADAM in certain non-convex settings and develops an adaptive optimizer, YOGI, which aims to improve convergence. [] integrates it in a FL framework. FedAvgM [] using Momentum [], while FedACG [] utilizes NAG []. And FedCM[] mitigates local heterogeneity by using the proximity global update gradient applied to the client momentum. Inspired by FedCM[], we approximate the global update gradient $\Delta$ as $\nabla f(\theta)$,to estimate the global perturbation range.

Sharpness-aware Minimization.
Many studies[, ] have pointed out that flat minima imply superior generalization performance, which possesses greater robustness to model perturbations. In order to minimize sharpness[], [, ] proposed sharpness-aware minimization (SAM), and many works [, ] were carried out. Specifically, SAM’s only capture the sharpness of specific small batches of data, and VaSSO[] aims to address this issue. Our FedTOGA can also help solve this problem to some extent, first we add neighborhood perturbations $\tilde{g}_{i,k-1}^{t}$ to help the local SAM optimizer perceive the amount of neighboring batch data perturbations (optionally), and global update perturbations via $\Delta^{t-1}$. In addition, m-SAM [] can be considered to be closely related to federated sharpness minimization. m-SAM [] quantifies the sharpness of batches across m training point batches, averaging out multiple disjoint batches in the generated Updates. The neighborhood global perturbation proposed by our FedTOGA alleviates the problem of local perturbation isolation in the FL paradigm and can be applied to all existing algorithms.

SAM in Federated Learning
In order to extend the generalizability of local models in FL, [, ] introduced SAM into the FL paradigm to propose FedSAM. further, [] combined with FedCM[] to propose MoFedSAM. FedGAMMA[] introduces the variance reduction technique of SCAFFOLD[] into FedSAM and gets some results. And FedSpeed [] uses SAM to optimize FedDyn[]. FedSMOO[] builds on FedSpeed to use dynamic regularization to SAM to estimate global disturbances. FedSOL[] uses the orthogonal idea of continuous learning to make local perturbations close to the global. [] proposes an efficient algorithm, the Local Estimation of Global Perturbations SAM (FedLESAM), which optimizes global sharpness and reduces computation. As we have seen, FedSAM[, ], MoFedSAM, and FedGAMMA[] compute local perturbations and optimize sharpness on client data, which may result the local SAM does not reach the global flat minimum. Several studies have identified this drawback and attempted to address it. FedSOL[] uses local orthogonal solving to limit the range of local perturbations, which can lead to perturbation absences. FedSMOO[] uses dynamic regularity to compute and add corrections, however, requires additional communication and storage overheads. FedLESAM[] believes that additional computation would be burdensome and therefore uses historical storage parameters to estimate global perturbations. However the above solutions may have limitations due to network fluctuations, which we discuss in detail in A.2. Therefore, we propose FedTOGA to estimate the global perturbation using the global update.

### A.2 Existing limitations

FedSMOO[] In the algorithm 2, to solve the model perturbation problem, [] utilizes the local Augmented Lagrangian function to penalize the deviation of the local perturbation from the global perturbation. As training proceeds, the local perturbation is made to approach the global perturbation gradually. However, it needs to open extra storage space on the client side to record $\mu_{i},\tilde{s}_{i}$. Meanwhile, $\tilde{s}_{i}$ needs to be synchronously uploaded to update the global perturbation variable $s$ at the time of aggregation during the communication process, which doubles the communication overhead. Further, this estimation bias will be exacerbated by the server’s strategy of randomly selecting the set of clients $S_{t}$ to mitigate the communication overhead due to communication bottlenecks.

Figure: Algorithm 2 FedSMOO Algorithm

FedLESAM[]
In the algorithm 3, to reduce the computational overhead and estimate the global perturbations, [] utilizes the historical global model record values $\theta^{old}_{i}$ to compare with the latest round’s global model $\theta^{t}$ to estimate the global perturbations. This poses the same problem as the algorithm 2 described above, specifically, in the face of an extreme case where participants will only participate in one global aggregation, FedLESAM will not be able to efficiently estimate the global perturbation variables. Meanwhile, the perturbation scales will vary when the frequency of client participation is different. In addition, since the perturbation estimation does not include the current perturbation computation it may not be possible to accurately estimate the current perturbation direction.

Figure: Algorithm 3 FedLESAM-D Algorithm

**Table 4: Abstract for the SAM-based FL algorithms for solving data heterogeneity, focusing on the basic algorithm, sharpness minimization objective, perturbation computation strategy, additional communication, and storage overhead comparison.**
| Works | Base Algorithm | Minimizing Target | Local Perturbation | Extra Storage | Extra Communication |
| --- | --- | --- | --- | --- | --- |
| FedSAM | FedAvg | Local Sharpness | $\rho\frac{g_{i,k}^{t}}{\|g_{i,k}^{t}\|}$ | $1\times$ | $1\times$ |
| MoFedSAM [] | FedCM | Local Sharpness | $\rho\frac{g_{i,k}^{t}}{\|g_{i,k}^{t}\|}$ | $1\times$ | $1\times$ |
| FedSpeed [] | FedDyn | Local Sharpness | $\rho\frac{g_{i,k}^{t}}{\|g_{i,k}^{t}\|}$ | $2\times$ | $1\times$ |
| FedGAMMA[] | SCAFFOLD | Local Sharpness | $\rho\frac{g_{i,k}^{t}}{\|g_{i,k}^{t}\|}$ | $2\times$ | $2\times$ |
| FedSMOO [] | FedDyn | Local Sharpness<br>With Correction | $\rho\frac{g_{i,k}^{t}-\mu_{i}-s}{\|g_{i,k}^{t}-\mu_{i}-s\|}$ | $3\times$ | $2\times$ |
| FedLESAM(S-D)[] | FedDyn<br>SCAFFOLD | Global Sharpness<br>Estimate | $\rho\frac{\theta_{i}^{old}-\theta_{t}}{\|\theta_{i}^{old}-\theta_{t}\|}$ | $3\times$ | $1\times$ |
| FedTOGA(ours) | FedDyn FedCM | Local With Global<br>Sharpness Estimate | $\rho\frac{g_{i,k}^{t}[\tilde{g}_{i,k-1}^{t}]+\kappa\Delta^{t}}{\|g_{i,k}^{t}[<br>\tilde{g}_{i,k-1}^{t}]+\kappa\Delta^{t}\|}$ | $2\times$ | $1\times$ |

### A.3 Neighbourhood Perturbation strategy

How Neighbourhood Perturbation works?
Let’s take Pytorch as an example. Recall the fact that when the local iteration interval is greater than 1, the gradient register needs to be cleared by (optimizer.zero_grad()) each time the gradient computation is performed, this is to prevent the accumulating gradient to causing errors. Recognizing that using the neighborhood gradient variables in the registers does not need any substantial additional overhead, we use the neighboring gradients temporarily stored in the registers to simulate the current perturbation gradient. We give an example of a local gradient computation to help better understand the workflow. We initialize the model $\theta_{0}$ and a gradient register $G=[\emptyset]$. After the first calculation of the gradient (loss.backward()), the register is updated $G=[g_{1}=\nabla f(\theta_{0})]$.
If SGD is used, then $\theta_{1}=\theta_{0}-\eta g_{1}$, followed by clearing the register $G=[\emptyset]$ before computing the second gradient. If SAM is used, then the perturbation is computed as $\delta_{1}=\rho\frac{g_{1}}{\|g_{1}\|}$, then the gradient register is emptied, and after the gradient is computed again as $\tilde{g}_{1}=\nabla f(\theta+\delta_{1})$, perform model update $\theta_{1}=\theta_{0}-\eta\tilde{g}_{1}$. So the register status changes to $G=[\emptyset]\rightarrow[g_{1}]\rightarrow[\emptyset]\rightarrow[\tilde{g}_{1}]$. When neighborhood perturbation is enabled, gradient calculation and clearing before SAM are no longer required. Therefore, the register status changes to $G=[{g}_{0}]\rightarrow[\emptyset]\rightarrow[{g}_{1}]$, The client will directly use the previously calculated gradient of the gradient register as the perturbation variable. You can observe the gradient calculation and cache change process in Fig. 3.

How Perturbation Fusion works?
With the above technical means of neighborhood perturbation, we can easily merge it in the perturbation computation by not emptying the gradient cache before computing the perturbation rise, then the gradient cache state will change to $G=[\tilde{g}_{0}]\rightarrow[\tilde{g}_{0}+g_{1}]\rightarrow[\emptyset]
\rightarrow[\tilde{g}_{1}]$.

Waht is LOOKAHEAD? LOOKAHEAD[] uses a fast-slow-step mechanism, where a retrospective is performed every $K$ steps forward. The idea behind this is to take a step in the direction of the current gradient update, and then use a set of additional weights (called “slow weights”) to take a step in the same direction, but on a longer time scale. These slow weights are updated less frequently than the original weights, effectively creating a “look ahead” into the future of the optimization process. By incorporating N-perturbation techniques, it forms a lookahead-like updating mechanism that helps the optimizer escape local minima and saddle points more efficiently, leading to faster convergence.

Figure: Figure 3: Illumination of the perturbation technique and its variants
Refer to caption: extracted/5820635/Figure/N-G-P.png

## Appendix B Experiments

### B.1 Introduction of Datasets

**Table 5: Summary of CIFAR10/100**
| Dataset | Total Number | Train Data | Test Data | Class | Size |
| --- | --- | --- | --- | --- | --- |
| CIFAR10 | 60,000 | 50,000 | 10,000 | 10 | 3$\times$32$\times$32 |
| CIFAR100 | 60,000 | 50,000 | 10,000 | 100 | 3$\times$32$\times$32 |

CIFAR10 and CIFAR100 are two datasets widely used in image classification and machine learning research. As shown in Table 5, CIFAR10 contains 60,000 color images in 10 categories with 6,000 images in each category and image size of $32\times 32$ pixels. CIFAR100 is similar to CIFAR10, but it contains 100 categories with 600 images per category, 500 of which are used for training and 100 for testing. These categories are further categorized into 20 major categories, each of which contains five subcategories.

### B.2 Detailed Hyperparameters Selection

To ensure a fair comparison across different datasets, we employed an experimental design consistent with FedGAMMA [], FedSMOO [], and FedLESAM []. ResNet18 [] was selected as the backbone model, utilizing group normalization [] and stochastic gradient descent (SGD). A total of 800 training rounds were conducted, with the initial local learning rate set to $\eta_{l}=0.1$. The global learning rate was maintained at $\eta_{g}=1.0$ for most experiments, except for FedAdam and FedYOGI [] were adjusted to $0.01$. The penalty coefficients $\alpha$ for FedDC [] and FedDyn [] were uniformly set to $0.01$ in the LeNet, consistent with [], but were increased to $0.1$ for ResNet18. In FedACG [], following its prescribed settings, the local penalty coefficient was set to $\mu=0.01$, and the server momentum coefficient $\lambda$ was set to $0.85$. For FedAdam and FedYOGI [], the parameters were set as $\beta_{1}=0.9$, $\beta_{2}=0.99$, and $\tau=1e^{-3}$. The momentum trade-off coefficient for FedCM [] was configured as $\alpha=0.1$. In the SAM-based algorithms, the penalty coefficients for FedSpeed, FedSMOO, FedLESAM-D, and FedTOGA were uniformly set to $\alpha=0.1$. The perturbation coefficients for FedGAMMA, FedSpeed, FedSMOO, FedLESAM(S-D), and FedTOGA were consistently set to $\rho=0.1$ for ResNet18, except for FedSAM and MoFedSAM [] and the vanilla FedLESAM coefficient were set to $0.01$. In the LeNet experiments, the perturbation coefficients $\rho$ for FedGAMMA, FedLESAM, and its variants were set to $0.01$, though in some cases $0.1$ yielded better performance. Weight decay was uniformly set to $1e^{-3}$ across all experiments.In the ResNet18 experiments, the learning rate decay were set to $0.998$ for most methods, except for FedSMOO, FedLESAM and its variants which were set to $0.9995$. In the 200-client case the learning rate decay was set to $0.9995$ (This is not always the case, in some cases a learning rate decay of $0.998$ works better, and we kept only the best results).
In most scenarios, the local perturbation correction coefficient $\kappa$ for FedTOGA was set to 1; however, in cases of increased heterogeneity, $\kappa$ could be slightly enlarged, but not beyond the local interval value $K$. The local dual variable correction coefficient $\beta$ for FedTOGA was chosen within the range of 0 to 1, with $0.8$ or $0.9$ typically performing best on CIFAR10. Generally, the parameter selection range can be determined according to Table 6.

**Table 6: Hyperparameters Selection.**
| Options | SGD-type | Best Selection | proxy-Type | Best Selection |
| --- | --- | --- | --- | --- |
| Local Learning Rate | {0.01,0.1,0.5} | 0.1 | {0.01,0.1,0.5} | 0.1 |
| Global Learning Rate | {0.01,0.1,1.0} | 1.0 | {0.01,0.1,1.0} | 1.0 |
| Learning Rate Decay | {0.995,0.998,0.9995} | 0.998 | {0.997,0.998,0.9995} | 0.9995 |
| SAM Learning Rate | {0.001,0.01,0.1} | 0.01 | {0.001,0.01,0.1} | 0.1 |
| penalized coefficient $\alpha$ | {0.01,0.1,0.2} | 0.1 | {0.01,0.1,0.2} | 0.1 |
| client-level momentum $\alpha$ | {0.01,0.05,0.1} | 0.1 | - | - |
| SAM Perturbation Correction $\kappa$ | - | - | {1,2,4} | 1 |
| Dual variable Correction $\beta$ | - | - | {0.8,0.85,0.9} | 0.9 |
| Server-level momentum $\lambda$ | {0.8,0.85,0.9} | 0.85 | - | - |

Test Experiments: Quadro RTX 6000; NVIDIA-SMI 515.76; Driver Version 515.76; CUDA Version 11.7

### B.3 Distributions of Dirichlet and Pathological Split

Figure: (a) CIFAR10 $\alpha=0.6$
Refer to caption: extracted/5820635/Figure1/CIFAR10_Dirichlet_0.6_100.png

Dirichlet Sampling: The Dirichlet distribution can be thought of as the conjugate prior of a polynomial distribution, and is used to generate weights for a mixture model or to distribute samples in the context of a non-uniform category distribution. By adjusting the parameter $u$, it is possible to generate data ranging from extremely inhomogeneous (near-discrete concentration in a category) to uniformly distributed. The data exhibit a long-tailed distribution, see Fig. 4.

Pathological Sampling: A typical feature of pathological sampling is extreme skewness or anomalies in the data distribution, which may lead to unstable training, convergence difficulties, or severe degradation of model performance. We used it to test and validate the performance of the model under adverse conditions. The data are presented in species isolation, see Fig.4.

The splitting strategy for all data is consistent with FedGAMMA[], FedSMOO[], and FedLESAM[].

### B.4 Evaluation Curves

Figure: (a) Accuracy $u=0.6$
Refer to caption: extracted/5820635/Figure/CIFAR10-Dirichlet-0.6-100-acc.png

As can be seen from the above figures, FedTOGA significantly outperforms other algorithms in scenarios with large heterogeneity (e.g., Dirichlet-0.1 and Pathological-3). Our algorithm still shows stability even when the number of clients decreases(e.g., 200 clients 5% participation). These results are in line with our expectations. We aim to design an algorithm that enhances global consistency while efficiently finding a global flat minimum to improve generalization and reduce the computation and storage requirements of edge nodes.

In addition to the results shown in the main text, we also conducted experiments on CIFAR100. We followed the same parameter settings with FedSMOO, but we found that the baseline algorithm produced fluctuations in performance. Therefore, we report the best performance of previous studies (OLD) and the results of brand new experiments (NEW) in table 7. In order to show the extraordinary performance of FedTOGA, we only report the historical best accuracy of all benchmarks in the main text.

Figure: Table 7: Test accuracy on CIFAR10/100 after 800 rounds under Dirichlet distribution and Pathological splits $u$ is the Dirichlet coefficient selected from $\{0.1,0.6\}$ and $c$ is the Pathological coefficient, which is the number of active categories in each client. The two datasets are divided into 100 clients and 10% of them are active at each round in the upper part, while 200 and 5% in the lower part.(ResNet18)

### B.5 Training Speed

**Table 8: Number of communication rounds required to reach a given accuracy. We recorded the first round of communication to reach a given accuracy to show the improvement in the number of training rounds compared to the other algorithms in the Dirichlet-0.1/0.6 and Pathological-6.0/3.0 settings. We mainly compared the SAM-based FL algorithms benchmark.**
| Partition | Dirichlet | Pathological |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Coefficient | $u=0.6$ | $u=0.1$ | $c=6$ | $c=3$ |  |  |  |  |
| Acc/Rounds | 80% | 82% | 76% | 78% | 80% | 82% | 76% | 78% |
| FedSAM | 491 | 800+ | 587 | 800+ | 443 | 790 | 465 | 691 |
| MoFedSAM [] | 167 | 270 | 303 | 425 | 135 | 253 | 167 | 265 |
| FedGAMMA[] | 458 | 630 | 369 | 591 | 407 | 550 | 701 | 800+ |
| FedSMOO [] | 190 | 253 | 302 | 402 | 205 | 263 | 262 | 322 |
| FedSpeed [] | 262 | 318 | 445 | 530 | 233 | 292 | 349 | 438 |
| FedLESAM | 588 | 800+ | 800+ | 800+ | 620 | 800+ | 497 | 778 |
| FedLESAM-D | 248 | 418 | 369 | 663 | 224 | 376 | 393 | 452 |
| FedLESAM-S[] | 390 | 643 | 529 | 800+ | 348 | 602 | 497 | 800+ |
| FedTOGA(ours) | 135 | 170 | 184 | 226 | 134 | 166 | 158 | 200 |
| Note: The SGD method is not considered. |  |  |  |  |  |  |  |  |

According to the above table 8, we can see that FedTOGA performs far better than the rest of the algorithms. It has the fastest convergence rate while maintaining high accuracy. The SAM optimizer usually slows down the whole training process due to the need to compute additional perturbations to the ascent process, which will be improved by enhancing consistency. MoFedSAM enforces consistency by employing global momentum on each local client and weighting it by a factor a (usually 0.1), which means that local knowledge will be forcibly overwritten by global gradient, which, while speeding up convergence in the early stage, may not be able to take any further from the local knowledge in the later stage. However, it may not be able to draw further effective learning progress from the local in the later stage. FedTOGA corrects local perturbations and dynamic regularizers by guiding global updates, greatly enhancing the consistency of generalization and optimization. Therefore, our method can effectively accelerate the modeling speed and improve the modeling accuracy, especially in the case of large-scale heterogeneous. From Table 9, it can be observed that FedTOGA has a similar local computation time as the SAM-based FL algorithm.

**Table 9: wall clock time(training, loading, evaluation) on CIFAR10 ResNet18 $u=0.6$, 0.1 and 100 clients.**
|  | FedSAM | MoFedSAM | FedGAMMA | FedSpeed | FedSMOO | FedLESAM-D | FedTOGA |
| --- | --- | --- | --- | --- | --- | --- | --- |
| time | 25.71s | 28.73s | 29.88s | 28.98s | 29.67s | 25.70s | 29.12s |

### B.6 Ablation studies

**Table 10: Ablation studies of different modules.**
| SAM | Dynamic Regularization | Dual Correction | SAM Correction | CIFAR10 Acc | CIFAR100 Acc |
| --- | --- | --- | --- | --- | --- |
| ✓ | - | - | - | 81.39% | 48.08% |
| ✓ | ✓ | - | - | 84.14% | 53.79% |
| ✓ | ✓ | ✓ | - | 85.54% | 56.85% |
| ✓ | ✓ | ✓ | ✓ | 86.01% | 57.25% |

We tested the performance of different modules called “SAM”, “Dynamic Regularization”, “Dual Variable Correction” and “SAM Perturbation Correction” modules on the Dirichlet 0.6 partitioned CIFAR-10/100 dataset, LeNet network. The benchmark is FedSAM. After the sequential introduction of the different modules, the CIFAR10 accuracy increased by 2.75%, 4.15%, and 4.62% compared to the FedSAM; the CIFAR100 accuracy increased by 5.71%, 8.77%, and 9.17% compared to the FedSAM.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See weaknesses.
soundness: 3
strengths: + Comprehensive validation: In Sec. 6, FedTOGO is compared against a lot of FL algorithms and is able to outperform all of them.
summary: This paper presents FedTOGA, a Federated Learning (FL) algorithm  designed to prevent the heterogeneity of client data cause the global model to converge to a sharp local minimum. FedTOGA achieves this in a 
communication-efficient way by combining new variants of two techniques: (i) Sharpness-Aware Minimization (SAM) to add perturbations to the training process and (ii) local dynamic regularization. In contrast to existing literature, FedTOGA uses the global gradient update to adjust both the global perturbation and the local regularization. The authors obtain analytic convergence guarantees for their methodology, and show the effectiveness of their approach by conducting extensive experiments.
weaknesses: + In the first contribution bullet point in Sec. 1, the claim about FedTOGA being the first global perturbation technique and first local dynamic regularizer needs to be rephrased to emphasize more on the fact that it is the first to do so using the global update. The current version of this statement overlooks the contributions of FedSMOO, FedLESAM, FedSpeed and papers that use dynamic regularizers, which have developed these ideas but without using global updates.
+ In Sec. 2's Sharpness-Aware Minimization, the parameters \rho and \delta need to clearly defined.
+ In Eq. (4) given in Sec. 2.1, the notation \theta_i seems to imply the different local models for clients in the FL setup. If this is the case, this needs to be clarified by defining \theta_i, which is missing currently. The same comment applies for Eq. (6) in Sec. 4.1, where the minimization is being done over a single vector \theta while there are multiple \theta_i's in the loss function formulation.
+ In Assumption 3 in Sec. 5, can the authors explain why they need a bounded variance of the unit gradient, and why just a bounded variance of the gradient itself is not sufficient? Adding some references which also make this assumption for the unit gradient is recommended.
+ In Sec. 6, does FedTOGO and most of the other FL algorithms perform similarly if the data distribution among clients is IID? Currently all experiments are done in non-IID settings and it would be insightful to 
see how these FL methods compare in IID setups.
+ The global perturbation used in this paper build heavily on prior research, doing a gradient ascent similar in existing algorithms with the addition of some global update information. The same goes for the local regularizer. While it is interesting to see how much accuracy increase can be achieved by making these adjustments, I am concerned about the novelty of this incremental idea. Alongside other issues mentioned above, I do not think this paper is ready to be published at ICLR in its current version.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. Could the authors further explain why their algorithm is better comparing to previous methods in terms of e.g. convergence rate, computational cost, challenge in implementation, etc?

2. What are the technical difficulties of the theoretical analysis, comparing to literature?

3. As in Theorem 2, the choice of learning rate is with order $O(1/\sqrt{T})$ while constants are ignored. However, these neglected constant may significantly influence the actual performances of the algorithms. Could the authors explain why the learning rates are identical for all algorithms? In the above sense, does it cause a fair comparison? If yes, could the authors explain the reason why this renders a fair comparison?
soundness: 2
strengths: The main contribution of the paper is to propose a local and global perturbation-based algorithm (FedTOGA) that enjoys communication and computation efficiency for SAM in FL. Both theoretical convergence guarantee and empirical evaluation of the algorithm are provided.
summary: The paper proposes a global-aware perturbation method for sharpness-aware minimization (SAM) in federated learning (FL). Compared with existing methods, this paper combines the idea of local dynamical regularizer with global updates to mitigate the effect of data heterogeneity. Convergence of the proposed algorithm is theoretically analyzed with the rate $O(1/T)$. Empirical results are shown to justify the performance of the algorithm.
weaknesses: 1. The novelty and contribution of the paper is limited. I think the main advantage of the proposed algorithm that authors claim is to leverage the global and neighborhood perturbation to reduce communication overhead and computational cost. However, this is not clear in the paper. As in Theorem 2 the authors claim that the rate of $O(1/T)$ is achieved when setting $K=O(T)$, which is faster than existing literature. However, FedSMOO also achieves the same rate $O(1/T)$ and linear speedup in the number of clients without any constraint on $K$. Thus, the advantage of FedTOGA is not clear.

2. The analysis tools look quite similar to those in FedSpeed paper. Thus, the technical contribution of this paper seems limited.

3. The presentation of the paper is not good, which makes the reader hard to follow. There are some mistakes and typos. To list a few, in Line 300,  there is no "Line 16" in the algorithm. In Theorem 2, $z_t$ is not defined and in eq. (11) the LHS should be the sum of $\Vert z_t \Vert^2$.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: See in weaknesses.
soundness: 2
strengths: 1. The proposed method is well-motivated, the paper shows that existing methods suffer from the local optima divergence issue, and show how to fix it.
2. The empirical results show that the FedTOGA method is better than other HtFL methods using global sharpness-aware minimization (SAM) and dynamic regularization, as expected.
summary: To address the local optima divergence in Heterogeneous Federated Learning, this paper proposes a FedTOGA method by linking the local dynamic regularizer to global updates to enhance the consistency of optimization and generalization. The method efficiently links local perturbations to global updates and achieves a non-convex convergence rate of $\mathcal{O}(1/T)$. The authors also propose neighborhood perturbation to approximate local perturbation. The authors also provide empirical validations of the theoretical results as well.
weaknesses: 1. In Theorem 2, compared with other methods using global sharpness-aware minimization (SAM) and dynamic regularization, such as [1],  a more distinct summary of the superiority of FedTOGA may be needed.
2. In addition to CIFAR10/100, the authors should also consider the performance of FedTOGA in the TinyImageNet task.
3. In FL research, the local epoch is an important parameter. The authors should study this parameter's impact on performance.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. A technical question: in line 245, according to the motivation of FedCM that $\Delta^t\approx \nabla f(\theta^t)$, why local perturbation is not $\delta_k^t=\rho\frac{(1-\kappa)g_{i,k}^t + \kappa\Delta^t}{\Vert (1-\kappa)g_{i,k}^t + \kappa\Delta^t \Vert}$? It looks like a external momentum in current version. Will the current external momentum setting outperform the original inner momentum form estimation of FedCM?

2. I am confused on line 264. What is the term $g_{i,k}$? Why the fusion term performs as $g_{i,k} + \widetilde{g}_{i,k-1}^t + \kappa\Delta^t$?

3. Line 11 in Algorithm 1, what is the term $g_{i,k}^{t}[\widetilde{g}_{i,k-1}^t]$?

4. What is the conection between the term in line 264 and "lookahead" optimizer? Although the paper claims that this form is an extension of lookahead, the authors could provide the corresponding extended formulas to further explain why this form corresponds to lookahead optimizers.

5. Could the authors compute the variance of the SAM perturbations? In fact, if the global perturbation is approached more closely, the variance of their local perturbations should be smaller. Additionally, the authors could calculate the global perturbation at the beginning of each round and compare whether this approach leads to further improvements.



Some typos:

(1) There are issues with the references in this paper. I suggest that the authors distinguish between the \citet and \citep commands to provide correct citations.

(2) Line 901 "waht" to what
soundness: 3
strengths: 1. The design motivation of this paper is reasonable. Although there are some flaws in the writing, the overall structure is understandable.

2. The experiments are very thorough, with extensive empirical studies conducted under standard settings to validate the efficiency of the proposed techniques.

3. A convergence analysis was conducted for the proposed method to demonstrate that its convergence remains at the same level as the previous works.
summary: This paper proposes an improved optimizer based on SAM, which further enhances the global perspective when applying SAM in federated learning (FL) and maintains global generalization. Both the theoretical analysis and extensive experiments confirm the effectiveness of the proposed method.
weaknesses: 1. The section on Methods (Section 4) is quite disorganized. I suggest that the authors include a notation table to explain all the variables that appear later in the text. I noticed that many variables are introduced without explanation, see questions.

2. The definition of Equation (9) seems somewhat obscure and difficult to understand. The vanilla local objective (like FedDyn) adopts the the augmented alternating direction method of multipliers to solve the consensus problem. The term $h_i$ is the dual variable to balance the dual problem. Performing operations on the dual term seems to affect the consistency solution of the primal problem. I suggest that the authors remove the problem formulation related to Equation (9) and directly introduce the use of certain variables to replace or correct the gradient. Actually, from the optimization perspective, the proposed method still solve eq.(8), but with the novel proposed method (momentum-based gradient estimator and SAM-based local optimizer). If the authors modify the entire Lagrangian function, it still need to be proven that the solution of this function is consistent with the solution of the original problem. I believe this is unnecessary for the techniques proposed in this paper.