FedSMU: Communication-Efficient and Generalization-Enhanced Federated Learning through Symbolic Model Updates
Authors: Xinyi Lu
Hao Zhang
Chenglin Li
Weijia Lu
ZHIFEI YANG
Wenrui Dai
xiaodong Zhang
Xiaofeng Ma
Can Zhang
Junni Zou
Hongkai Xiong
Venue: ICML 2025 poster (2025)
Keywords: Federated learning
efficient communication
enhanced generalization
Primary area: general_machine_learning->everything_else

Abstract: The significant communication overhead and client data heterogeneity have posed an important challenge to current federated learning (FL) paradigm. Existing compression-based and optimization-based FL algorithms typically focus on addressing either the model compression challenge or the data heterogeneity issue individually, rather than tackling both of them. In this paper, we observe that by symbolizing the client model updates to be uploaded (i.e., normalizing the magnitude for each model parameter at local clients), the model heterogeneity, essentially stemmed from data heterogeneity, can be mitigated, and thereby helping improve the overall generalization performance of the globally aggregated model at the server. Inspired with this observation, and further motivated by the success of Lion optimizer in achieving the optimal performance on most tasks in the centralized learning, we propose a new FL algorithm, called FedSMU, which simultaneously reduces the communication overhead and alleviates the data heterogeneity issue. Specifically, FedSMU splits the standard Lion optimizer into the local updates and global execution, where only the symbol of client model updates commutes between the client and server. We theoretically prove the convergence of FedSMU for the general non-convex settings. Through extensive experimental evaluations on several benchmark datasets, we demonstrate that our FedSMU algorithm not only reduces the communication overhead, but also achieves a better generalization performance than the other compression-based and optimization-based baselines.

FedSMU: Communication--fiiieen and Generalization-nnaanned Federated Learning through Symbolic Model Updates
Xinyi Lu
Hao Zhang" 1
Chenglin Li' Weijia Lu Zhifei Yang² Wenrui Dai Xiaodong Zhang?
Xiaofeng M Can Zhang² Junni Zou' Hongkai Xiong'
Abstract
1. Introduction
The signifieant communication overhead and client data heterogeneity have posed an im- portant challenge current federated learning (FL) paradigm. Existing compression-bed based and optimization- FI. algorithms typically focus on addressing either the model compression chal- lenge or the data heterogeneity issue individually, rather than tackling both of them. In this paper, we observe that by symbolizing the client model up- dates to be uploaded (i.e., normalizing the magni- tude for each model parameter at local clients), the model heterogeneity, essentially stemmed from data heterogeneity, can be mitigated, and thereby helping improve the overall generalization perfor- mance of the globally aggregated model at the server. Inspired with this observation, and fur- ther motivated by the success of Lion optimizer in achieving the optimal performance on most tasks in the centralized learning, we propose a new FL algorithm, called FedSMU, which simul- tancously reduces the communication overhead and alleviates the data heterogeneity issue. Specif- ically, FedSMU splits the standard Lion optimizer into the local updates and global execution, where only the symbol of client model updates com- mutes between the client and server. We theoreti- cally prove the convergence of FedSMU for the general non-convex settings. Through extensive experimental evaluations on several benchmark datasets, we demonstrate that our FedSMU algo- rithm not only reduces the communication over- head, but also achieves better generalization performance than the other compression-bust and optimization-based baselines.
Federated learning (FL) is a large-scale machine learning paradigm wherein a multitude of clients, under the orches- tration of a central server, collaboratively learn model without the need of sharing or exchanging any raw client data (McMahan et al., 2017). This paradigm is commonly adopted in data-constrained or data-sensitive environments, such as Internet of things (IoT). healtheare, and finance (Khan al., 2021; Rieke et al., 2020; Yang et al., 2019; Haibo et al., 2023). In essence, FL is distinguished from the traditional distributed learning in the following three major challenges. High communication cost. During each communication round of training, the clients are required to transmit their local model parameters (or updates) the central server for global aggregation. When the number of model parameters becomes significantly large, this transmis- sion process may result in huge bandwidth consumption. Data heterogeneity. Due to the inherently private and per- sonalized nature of federated clients, the datasets across these clients tend to exhibit distinct statistical distributions. Such a data heterogeneity may introduce significant biases into the globally aggregated model, consequently impairing its generalization performance. Partial client participa- tion. In practical scenarios, clients may join leave the FL system at random time intervals. This highly dynamic behavior results in only small subset of clients being active for training during each communication round.
To address these challenges, extensive exploration has been conducted in the FL community, but from different perspec- tives. On one hand, compression-based federated algorithms aim to reduce the amount of data required for model param- eter (or update) transmission. For instance, quantization compression, such as signSGD (Bernstein et al., 201 8a;b), QSGD (Alistarh et al., 2017) and FedPAQ (Reisizadeh et al., 2020), quantize the gradient values into lower-precision inte- gers, thereby reducing the number of transmitted bits. While the sparsity compression methods typically sparsify the gra- dient vector by setting some of its elements to bits, with the aim of reducing the data transmission cost (Wangni et al., 2018; Aji & Heafield, 2017: Lin et al., 2017). However, a direct application of compression methods may
'qqual contribution 'Shanghai Jiao Tong University. Shanghai, China. Unnited Automotive Electronic Systems, Shanghai, China. Correspondence to: Chenglin Li, Wenrui Dai, and Junni Zou <1cl1985@sjus jdd...
fewer
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FedSMU: Communication-Ffiiee and Generalization-nnaanee FL through Symbolic Model Updates
lead to information loss, resulting in decreased model accu- racy (Yu et al., 2022) and slower convergence rates, or even divergence of the training process (Beznosikov et al., 2023). To mitigate these issues, strategies such as error feedback (Richtárik et al., 2021) have been developed, which incor- porate residual errors from previous compression steps into the optimization process. While many algorithms with the error feedback require the full participation of the clients, and if only partial clients are involved, their performance will degrade (Li & Li, 2023).
mance while maintaining the benefits of compression.
We conduct a convergence analysis of FedSMU under the general non-convex settings, and find its conver- gence rate as O (TT) ). where 7 is the total number of communication rounds. This theoretical result matches with the convergence rates of existing FL algorithms.
We conduct series of experiments to demonstrate the superiority of FedSMU. By comparing FedSMU with the other compression- based and optimization based FL algorithms, we show that our FedSMU achieves a higher generalization performance while greatly saving the communication overhead in most cases.
On the other hand, several optimization-base federated algorithms, have been proposed to address the data het- erogeneity issue. For example, SCAFFOLD (Karimireddy et al., 2020) aims to mitigate the client variance by design- ing and iteratively updating the control variates. Though theoretically effective, it incurs doubling the communication overhead. FedGen (Venkateswaran et al., 2023) regulates the local training by transmitting additional generators. Sev- eral adaptive algorithms (Tong et al., 2020) dynamically adjust their learning rates based on the divergence between the local and global models, thereby enhancing the gener- alization performance in federated settings. Most of these optimization-baseee FL algorithms, which mainly aim at mitigating the data heterogeneity, may incur additional com- munication overhead of information exchange for further performance improvement.
2. Related Works
Compression-Efficiee FL. Extensive studies have been dedicated to reducing the amount of data required for gra- dient transmission and thus improving the communication efficiency. Using the unbiased compression method, QSGD (Alistarh et al., 2017), FedPAQ (Reisizadeh et al., 2020) and ECQ-SGD (Wu et al., 2018) compress the gradients up- loaded to the server while keeping the original data integrity and expectation unchanged to save the communication cost. For biased compression, by leveraging the sign operation, signSGD (Bernstein et al., 2018a;b) can compress the gradi- ents up to 1 bit. While the sparsification-based methods like Topk (Stich et al., 201 8: Alistarh et al., 2018), 8), which only keeps the largest K gradients, is another communication- efficiently biased compression method. Other methods, like FedZip (Malckijoo et al., 2021) and Qsparse-local-SGe (Basu et al., 2019), incorporate both the quantization and sparsification. A direct application of biased compression, however, may lead to performance degradation and slower convergence rates due to the bias accumulation (Beznosikov et al., 2023). To address this, optimization techniques have been introduced to mitigate the negative effects of bias. For example, FedEF (Li & Li, 2023) and EF21 (Richtárik et al., 2021) employ error feedback, while MARINA (Gorbunov et al., 2021) and DIANA (Mishchenko et al., 2024) lever- age the compression of gradient differences, both of which enhance the model performance and convergence speed, However, the performance of these algorithms is limited by the client participation rate. In this work, we adopt the sign operation to improve communication efficiency with partial participation of clients, which also helps enhance the generalization capability of the globally aggregated model as shown by Chen et al. (2024; 2021); Foret et al. (2020).
In this paper, we aim to design an algorithm capable of si- multaneously addressing the communication bottleneck and data beterogeneity, without being constrained by the partial client participation issue. To achieve this goal, we first re- visit the fundamental FedAvg algorithm and identify that heterogeneous magnitudes of model updates may result in certain clients' updates being overlooked, thus leading to an unstable and sub-optimal aggregation of the global model. Building upon this observation, we then introduce the con- cept of Magnitude Uniformity (MU) index, which quantifies the clients' contribution to the global model's update. We empirically validate that this MU index is influenced by the degree of data heterogeneity in FL., indicating that a more heterogencous data distribution causes a greater heterogene- ity in the magnitudes of client model updates. Furthermore, heterogeneous client updates may contribute to a decline in the global model's generalization performance. To address this issue and further reduce communication overhead, we motivated to symbolize the model updates as an imme- diate solution, and propose the FedSMU algorithm. Our contributions can be summarized as follows.
We develop a compression-basel FL. method, FedSMU. It uses the sign operation to achieve 1-bit compression and thus greatly saves the communication cost. Simul- tancously, we leverage the design of Lion optimizer (Chen al., 2024) to enhance the generalization perfor-
Generalization- -nhancce FL. In the advancement of FL algorithms, in parallel, various techniques have emerged to improve the generalization performance. By using mo- mentum in FL, one can track the historical information of gradients, suppress the noise and reduce the instability of
2

---

FedSMU: Communication and Seneralization-Enaanee FL through Symbolic Model Updates
Table I. Summary of notations.
ing model updates to alleviate communication, rather than addressing both challenges concurrently. Specifically, most compression-based FL algorithms (Bernstein et al., 201 8a;b; Li & Li, 2023; Wen et al., 2017) significantly reduce the communication cost, with a generalization performance typ- ically comparable to or slightly lower than that of standard FedAvg (McMahan et al., 2017). On the other hand, most optimization-base FL strategies (Karimireddy et al., 2020), which involve exchange of full-precision model updates, and even additional control variables or informative repre- sentations, aim to mitigate the data heterogeneity issue, but at the cost of a huge communication overhead.
T. t
number, index of communication rounds
K. k
number, index of local update stop
n. 71
local, global learning rate momentum coefficients
$1.92
72
weight decay factor
Mi.k
client i's model at round i and step & aggregated server model after round f set of clients with cardinality m
M. m
Ni,n n
set of sampled active clients with cardinality n
model updates. Benefiting from this, methods such as MV- sto-signSGD-SIN (Sun et al., 2023) and FedAdam (Reddi et al., 2020) apply momentum instead of directly updating with gradients, while PR-SGD-N SGD-MMomntumm (Yu et al., 2019) first updates the momentum and then combines the new gradient with weight of the momentum. These methods enhance the model generalization and accelerate conver- gence in FI.. In this work, we employ two sliding average functions to update momentum after calculating the new gradient, a technique introduced by Lion (Chen et al., 2024) to effectively store more historical gradient data. Also, since weight decay regularization has been shown outperform l2 regularization in preventing overfitting and enhancing generalization (Loshchilov, 2017), we leverage weight decay strategy to mitigate the impact of data heterogeneity and further improve generalization performance. The work most closely related to ours is distributed Lion (Liu et al., 2024), which leverages the Lion optimizer to reduce commu- nication overhead by extending it to the distributed setting with the full client participation and iid data. However, it lacks exploration of the partial participation and non-iid data scenarios, which are the major challenges brought by FL.
The recently proposed SCALLION algorithm (Huang et al., 2023) integrates the control variable-based SCAFFOLDD framework with incremental variable compression meth- ods, achieving comparable performance with SCAFFOLD while substantially reducing the upload communication cost. Nonetheless, SCALLION additionally requires to double the download communication overhead for the transmission of control variables. CompressedScaffnew (Condat et al., 2022) and TAMUNA (Condat et al., 2023) also combine the control variates with the model compression. However, these methods rely on the permutation- based compression schemes, which are relatively complex and less flexible. LoCoDL (Condat et al., 2024) extends these two works by supporting a broader class of compressors and demonstrat- ing a convergence acceleration in the convex problems, but it focuses exclusively on the convex setting. Additionally, FedComLoc (Yi et al., 2024) and Sparse-ProxSkip (Mein- hardt al., 2025) make attempts explore the client drift under the non-convex objectives. However, FedComLoc's performance may degrade under the compressed communi- cation due to its reliance on the communication variables, while Sparse-ProxSkip assumes the full client participation, which may not always be feasible in the real-world FL sce- narios. These observation then impose a critical question for the field of compression-effiiie FL: can we design an approach to effectively mitigate both the communication bottleneck and data heterogeneity simultaneously with par- tial participation of clients under the non-convex objectives?
3. Proposed Method
3.1. Notations and Preliminaries
The general optimization problem of federated learning (FL) can be formulated as:
1 R(x), = m 1=1
min f(x) =ERe
3.2. Symbolizing Client Updates
Before answering this question, we revisit the standard FL framework. i.e., FedAvg (McMahan et al., 2017). With Fe- dAvg, clients perform local training using their own datasets that are distributed over clients and non-iid in nature. The server then aggregates these locally trained models to up- date the global model, which subsequently serves as the initial model for the next round of training. However, due to the data heterogeneity, clients' model updates often differ in both the direction and magnitude. Consequently, when model updates from different clients with large deviations are averaged, some updates with relatively small magnitudes
where Fx(x) 2 = RG [Fi(x, {)] represents the local loss function of the i-th client with the data sample & drawn from distribution D,. Under the FL settings, data typically heterogeneous, implying that for two clients and j. the dis- tributions D; and D, can be extremely different. Moreover, the FL systems often operate under limited bandwidth, which renders the communication overhead associated with the exchange of model parameters a significant bottleneck.
Current approaches in FL often prioritize either mitigating data heterogeneity to enhance generalization or compress-
3

---

FedSMU: Communication-Efiiiii and Enteraaliation-Enhancee FL through Symbolic Model Updates
209
2 1.00
56
2
54-
182
180-
52
23
66
FedAug
Fed
46
50
48
FedSMU
FedAvg
55
Fed/vg FodSMU
44
50
Data Heterogernity (Dirichles)
U.S.
0.25
Data Heterogeneity (Dirichlet)
0.6
0.8
lid
45-
3 3
Round [Dircchlet (0.251)
4
(a) MU index vs. heterogeneity
(b) Accuracy heterogeneity
(c) MU communication round
Figure 1. Magnitude uniformity (MU) index and top validation accuracy of FedAvg and FedSMU (ours) on CIFAR-100 with CNN model.
may be overlooked. For instance, we consider three clients, i1 i2 and i3, whose model updates along one dimension are +10, 11, and -1, respectively. In this case, the updates have opposite directions, while the magnitude of client i 's update is much larger than those of clients i2 and iz. Af- ter averaging at the server (i.e., the global model's update becoming +8/3), the contribution of clients i2 and is to the global model's update will be ignored, since the update direction is now dominated by client it. Thus, a direct aver- aging may neglect contributions from the smaller updates and potentially compromise the fairness among clients.
under varying data heterogeneity on the CIFAR- 100 dataset. The experiment involves 100 clients with partial participa- tion rate of 10%. As observed from Figures ((a) and 1(b), for FedAvg, an increase in the data heterogeneity leads to a decrease in the Magnitude Uniformity index, accompanied by a deterioration in the generalization performance. This suggests that with FedAvg. data heterogeneity leads to a significant difference in the magnitude of model updates across clients, resulting in an unstable global aggregation and poorer generalization performance. Additionally, as shown in Figure I(c), the Magnitude Uniformity index tends to rise during the FedAvg training, suggesting that the early stage of an FL. system forces a gradual narrowing on the magnitude difference of model updates across clients.
To address this, and motivated by the Jain's fairness in- dex (Jain et al., 1984), we propose a new metric called the Magnitude Uniformity (MU) index to reflect clients' con- tribution to the global model update. Through empirical analysis, we explore the relationship between this MU index and the local data heterogeneity, which in turn impacts the generalization performance of globally aggregated model.
A straightforward approach to enhance the Magnitude Uni- formity index for FL. is to apply sign operation to the local clients' updates, ensuring that model updates have the uniform magnitude from all the clients. Specifically, after this sign operation, the local model updates for the three clients i1, i2 and ig in the previous example become +1, -1, and -1, respectively. This process guarantees that each client's s model update contributes equally to the globally aggregated model, thereby reducing the impact of model heterogeneity and promoting fairness. By converting the magnitudes of model updates into their respective signs, we actually emphasize the directions of their updates rather than the magnitudes, which could help balance the contributions from different clients' model updates and lead to a more representative and informative global model. Moreover, by symbolizing the updates we can reduce the communication cost to 1 bit per dimension, offering a potential solution to enhancing generalization while saving the communication.
Definition 3.1. (Magnitude Uniformity). We define the magnitude uniformity across m. clients at the 1-th communi- cation round as:
(EIEM
d
|M Erem (B/1) 2: S. = 6. -1ubi: vt,61..
where y. denotes the j-th dimension (d dimensions in total) of client i's model at round t and local step K, and g.d denotes the magnitude of client i's model update in this dimension j at round f. Similar to the Jain's faimess index, higher value of the magnitude uniformity o, indicates more uniform contribution from the clients, thus suggesting a more balanced representation of the clients' data in the global model. Theoretically, such a uniformity may lead to a global model better capturing the information from all the local clients. Consequently, one may raise a question: is this magnitude uniformity index affected by the data hetero- geneity across locally distributed clients, and does it further influence the global model's generalization performance?
In fact, numerous sign-based compression methods (Bern- stein et al., b0;;;;; Wen et al., 2017) have been applied in FL. While theoretically performant, their empirical results often show only marginal improvements or comparable per- formance to FedAvg. Thus, effectively leveraging the sign operation to simultaneously mitigate the communication overhead and enhance generalization in federated learning remains a challenging and unresolved issue. On the other hand, many optimization techniques have been proposed to improve generalization for the centralized learning, such as momentum, Adam, and weight decay. A brute force ap-
Seeking for an answer to this question, we empirically ex- amine the correlation between this Magnitude Uniformity index and the global model's generalization performance

---

FedSMU: Communication-Efficee and Generalization-Enhele Enhanced FL through Symbolic Model Updates
proach could be directly incorporating the sign operations with these optimization strategies in Fl.,, formulating the algorithm design program search to identify federated optimization algorithms that can incorporate sign compres- sion. However, this approach is computationally expensive.
We also notice that inspired by advantages of the Lion op- timizer in the centralized learning, there has been other works, e.g., FedLion proposed by Tang & Chang (2024), in- corporating Lion into the local updates of federated learning, However, Fedl simply uses the vanilla Lion algorithm for the local updates to replace SGD, resulting in a com- munication cost that is even significantly higher than that of FedAvg, as the extra momentum terms need to be trans- mitted. Compared to FedLion, our FedSMU out-stands as follows. 1) Effective utilization of Lion framework. Fe divides the execution of Lion optimizer across the clients and server. In contrast, FedLion merely executes the Lion algorithm locally in parallel as a local optimization strategy, failing to exploit the complete structure of Lion. 2) Communication cost saving. In addition to the model updates, FedLion requires an additional transmission of the full- full-precision momentum terms, resulting in significantly higher communication cost compared to FedSMU, which only necessitates a I-bit communication for each dimension of model updates. This substantial reduction in communica- tion overhead is another key advantage of our FedSMU.
Fortunately, in the context of centralized learning, Lion (EvoLved Sign Momentum) optimizer (Chen et al.. 2024) employs the sign operation compute the updates while overall
tracking momentum, which has demonstrated outstanding performance across various models and tasks. Compared to the simple signSGD (Bernstein et al., 201 2018a;)), Lion leverages the dual momentum tracking and weight de- cay, significantly improving generalization ability of the trained models. Inspired with our observation on the im- pact of Magnitude Uniformity index on FL's generaliza- tion performance, and further motivated by the success of Lion in centralized learning, we thus propose a new fed- erated optimization algorithm aiming at both reducing the communication overhead and enhancing the generalization performance, through symbolizing the client model updates.
3.3. Proposed FedSMU
4. Theoretical Results on Convergence
To leverage the structured design of Lion optimizer and minimize the communication overhead, we propose our FedSMU algorithm for federated learning, which splits the Lion optimizer's s framework of momentum tracking and weight decay to be carried out independently at the server and each client, respectively, as summarized in Algorithm 1.
We now present the convergence analysis of our proposed FedSMU for the general non-convex functions. In general, our analysis is based on the following three standard as- sumptions, which are commonly satisfied by a range of non-convex objective functions.
Specifically, at each communication round f, our proposed FedSMU implements the following steps:
Assumption 4.1. (Lipschitz Gradient). For all i E M. the function (x(x) L-smooth: [[VF,(x) - -Vllil S L | for all x, y E Rd.
1. Participating clients initialize their local models, de- noted as v based on the current global model xz.
2. Each client conducts K steps of local stochastic gradi- ent descent (SGD) to compute the model update si.
Assumption 4.2. (Bounded Variance). For all i e M, the function F. F. {) has a locally-bounded variance of: V[IVF(((rE) V (x))1 of for all x E Rd.
3. Each client symbolically represents its model updates by using the momentum and the sign operations.
Assumption 4.3. (Bounded Gradients). For all i E M, the function F,(x, {) has bounded gradient: [[, G for all x E IRd.
4. The server receives and aggregates these symbolic up- dates, denoted as uf, to update the global model Ø+41 by incorporating the weight decay.
For the non-convex optimization problem, Assumptions 4.1 and 4.2 are standard and widely adopted in various literature of FL (Reddi et al., 2020; Bottou et al., 2018; Reddi et al., 2016; Ghadimi & Lan, 2013; Li & Orabona, 2019). Assump- tion 4.3 is commonly used in the convergence analysis of sign-based methods, such as the distributed signSGD (Sun et al., 2023; Jin al., 2020).
Such a design offers two significant advantages to our FedSMU algorithm. First, it fully leverages the structure of the Lion optimizer. thereby enhancing the generalization performance of the global model. It is also worth noting that in the special case where the number of local update steps and clients are set to 1, our optimizer essentially re- verts to the standard Lion. Second, by transmitting only 1-bit update for each dimension of the model parameters between the clients and server, we substantially reduce the communication overhead in the FL systems.
Theorem 4.4. Under Assumptions 4.1, 4.2, and 4.3, when 0 < 1 S 1 71 = O ((V))) and 31 = O (()))
) we
5

---

FedSMU: Communication and Anmeraalizaiio-Ennaaeee FL through Symbolic Model Updates
Algorithm Federated learning through Symbolic Model
convergence speed and Tmaz and d through experiments in Appendix C.6.
Updates (FedSMU) algorithm.
Server Initialization: 21; Client Initialization: m = 0;
Remark 4.6. The original work of Lion (Chen et al., 2024)
does not include a convergence analysis. Our theoretical analysis provides the relevant convergence rate for the Lion optimizer. Specifically, by setting n Tmaz 1 and K 1, the convergence rate of our FedSMU reduces to that of the Lion optimizer. However, FedLion (Tang & Chang, 2024) cannot be reduced to standard Lion opti- mizer, because it merely parallelizes the execution of Lion optimizer at the client side and incorporates multi multi-precision quantization for communication compression.
for each round t = ... do
sample clients N₁ S .M
for each client E NN in parallel do receive and initialize local model vi.0 2t
for each local step l- 1,2,.... K do vi. -Vi,k- 1/ - 51,-
end
of vi.k Vi,o
/ Sign( m/-1 (1- Bi)g()
5. Experiments
m; = Bamb- 1 (1 B2)gt (for i o No. m
m
We conduct comprehensive comparative experiments to validate the superior performance of FedSMU in sce- narios involving different partial participation rates and data heterogeneity degrees. The additional experi- ments across more scenarios and ablation studies are provided in Appendix C, and the implementable code of our proposed FedSMU algorithm is available https//gghhbbboo//yy68/////eee
send ut, to server end
// at server:
T(+1 = Ze + (1 Ei= ut -221))
broadcast 16+1
end
have:
5.1. Experimental Setup
L(((zo) minJ)
3G
GridTmax
V S
+ nT(1 (2)
+ LT(1 (2)
Models and Dataset. We evaluate our FedSMU and the other baseline algorithms on three real-world visual and lan- guage datasets: CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009) and neural machine translation on Shakespeare, with the same train/test splits as in (Acar et al., 2021). Each client is assigned an uncertain number of classes, and the data within each class varies widely, with the labels of client samples generated according Dirichlet distribution. For instance, using Dirichlet-O.22 proximately 80% of each client's samples belonging to around three or four different classes. We employ the CNN model with LeNet architecture and RNN model both similar to previous studies (McMahan et al., 2017). Furthermore, to demonstrate the applicability of our method more com- plex models and datasets, we evaluate the performance of different FL algorithms using larger model, ResNet18 (He et al., 2016) and ViT-S (Dosovitskiy et al., 2021), and a more challenging dataset, Tiny-ImageNet, a reduced version of the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) (Russakovsky et al., 2015) classification dataset. For additional details on the experimental setup, please refer to Appendix A.
VT
12m
d(2Kg + 1K207 4K2G2)
+ T
1- 32
6Gd
2d
+ vn
+ VT
where V
+ ELIT(
CIFAR- 10, there are ap-
De (oro)) d denotes the dimensions of pa- rameters, Tmax = max( (r') hSism.ISST and T¹ denotes client i's participation interval.
Proof. See Appendix B for the detailed proof.
L
Remark 4.5. The convergence rate of our FedSMU is O V when T is sufficiently large, matching with the convergence rates of existing FL algorithms, such as Fe- dAvg and FedPAQ (Reisizadeh et al., 2020). Note that Tmaa represents the maximum participation interval among all the clients, indicating that larger participation intervals result in slower convergence. Note that d represents the model di- mension and directly influences the rate of convergence, i.c., larger model dimension results in slower convergence. Increasing the number of workers n leads to a tighter error bound. Further in Appendix C.5, we show that though a higher precision quantization can reduce the quantization error, it may slow down the overall convergence rate in some cases. We also verify the relationship between the
Comparison Algorithms. We compare the validation (test) performance of our FedSMU with several other baselines, including the optimization- based FL algorithms such as Fe- dAvg (McMahan et al., 2017), FedLion (Tang & Chang, 2024), and SCAFFOLD (Karimireddy et al., 2020), as well as the compression- based FL algorithms such as FedEF-HS
6

---

FedSMU: Communication-Efficcee and Anneraalzzaion-nnhaneee FL through Symbolic Model Updates
Table Performance comparison under various settings, where smaller Dirichlet parameter indicates higher data heterogeneity, and and H indicate low and high participation rates, respectively. For CIFAR-10 10 and CIFAR-100 100, LeNet model is used, and for Shakespeare, an RNN model is employed. Bold numbers indicate the best performance.
Top-1 Test Accuracy (%)).
Dataset
Setting
FedAvg
SCAFFOLD
SCALLION
FedEF-HS
FedEF-TopK
FedEF-Sine Sign
FedLion
FedSMU
Dir (0.25)-L.
41.44
41.28
42.68
38.31
44.29
37.79
45.09
51.87
CIFAR- 100
Dir (0.6)-L
41.36
45.04
43.28
38.63
44.41
40.34
47.19
53.79
Dir (0.25)-H
42.29
50.49
45.24
37.03
42.69
36.12
48.33
52.35
Dir (0.6)-H
43.44
50.02
45.84
36.09
42.72
38.99
48.85
54.2
Dir (0.25)-L
80.95
81.6
80,91
78.35
80.11
77.87
79.04
80.12
CIFAR- 10
Dir (0.6)-L
82,42
82.36
81.18
79.29
81.73
79.68
80.94
82.48
(0.25)-H
80.6
83.31
81.42
78.17
79.92
78.34
81.61
80.74
Dir (0.6)-H
81.43
84.12
81.75
78.75
81.42
79.38
83.15
82.66
Shakespeare
noniid-H
47.58
51,28
47.86
45.79
46.21
45.00
47.11
47.81
70
45
8 35
35
20
15
20
10
10
Fele Trypt
Gige
0.5
22
18
24
366
0.06
12
1.8
2.4 30
255
12
18
2.6
25
45
35
75
commmunccooo Bis (G8)
conmoun codon bis (GB)
corm CaS0n bis (GB)
con commmmnicaa.coomm
IMB)
(a) Dirichlet0.255 25-CIFARI0
(b) Dirichler0.25 25--2555CCFROO
(c) DirichletO.6-CIIH 6-CFFAR000
Noniid-Shakespent
Figure 2. Convergence performance vs. number of uplink communication bits CIFAR-10. 10, CIFAR -100 and Shakespeare, with 100
clients and 10% participation. For CIFAR-I0 and CIFAR-100, a LeNet model is used, and for Shakespeare, an RNN model is employed.
(Li & Li, 2023), FedEF-TopK (Li & Li, 2023), FedEF-Sign (Li & Li, 2023), and SCALLION (Huang et al., 2023). It is worth noting that FedLion (Tang & Chang, 2024) involves parallel execution of the Lion optimizer on the local clients, requiring the upload of full-precision momentum updates in addition to the compressed model updates. Consequently, the communication overhead of FedLion is higher than Fe- dAvg, even when the model updates are compressed. Ad- ditionally, SCAFFOLD needs twice of the communication cost compared to FedAvg. Though SCALLION uploads the compressed incremental updates, it still results in doubling the communication overhead during the download phase. Since the distributed Lion (Liu et al., 2024) can only applied under the full client participation, we include com- parison with it in Appendix C.9, which also demonstrates the advantages of our FedSMU.
to Appendix A.
5.2. Experimental Results
5.2.1. PERFORMANCE EVALUATION
Experimental results for all the comparison methods under three datasets shown in Table 2 and Figure 2. In most cases, our FedSMU demonstrates superior performance compared to the other baselines (especially compression- based methods) with varying data distributions and client participation rates. The results effectively demonstrate that our FedSMU performs well on both image classification and text prediction tasks. We attribute this improvement to our design, which mimics the Lion optimizer and incorporates symbolic updates, momentum tracking, and weight decay. In contrast, other compression- brasee methods, such TopK and group sign employed by FedEF-TopK and FedEr-Sign, compress the communication traffic but consistently exhibit a poorer generalization performance.
Implementation. We evaluate the performance of the global model on the CIFAR-10, CIFAR-100 and Shakespeare datasets, by utilizing 100 clients with high (H) and low (L.) client participation rates of 10% and 3%, respectively. For the ResNet18 and VIT-S model, we adopt the client number of 10 and the participation rate of 30%. Clients uniformly sampled random without replacement at each round. The learning rates and hyperparameters for all approaches are individually tuned via grid search. For additional details on hyperparameter settings, please refer
Note that our FedSMU generally presents more signifi- performance gain CIFAR-100 for image classifica- tion. For CIFAR-10, though our FedSMU outperforms the compression- based FL algorithms, it is still less effective than the optimization- algorithms, such as FedAvg and SCAFFOLD. Here, we discuss about the possible reason for this slight degradation on CIFAR- In federated hetero-

---

FedSMU: Communication-Fficee and Generalizatio- FL through Symbolic Model Updates
Table Performance comparison under various settings with the ResNet18 network model. Bold numbers indicate the best performance.
Top-1 Test Accuracy (%)).
Dataset
Setting
FedAvg
SCAFFOLD
SCALLION
Fe
FodEF-Topk
FedEF-Sign
FedLion
FedSMU
CIFAR-10
Dir (0.25)
81.74
85.62
83.75
80.43
82.54
83,93
82.44
83.54
CIFAR-100
Dir (0.25)
47.41
48.76
48.15
47.24
48.07
49.03
43.75
49.76
Tiay-emageNet
Dir (0.25)
29.79
33.35
31.91
31.68
31.05
31.17
28.36
33.53
Table 4. Performance comparison under various settings with the ViT-S network model. Bold numbers indicate the best performance.
Top-1 Test Accuracy (%)
Dataset
Setting
FedAvg
SCAFFOLD
SCALLION
FedEF-HS
FedEF-Topp
FedEF-Sign
FedLion
FedSMU
CIFAR-I0
Dir (0.25)
74.24
76.26
75.12
74.11
74.15
74.08
71.75
76.71
CIFAR- 100
Dir (0.25)
41.72
48.27
46.37
51.79
45,28
44.02
43.66
53.24
gencous scenario involving CIFAR-100, which comprises 100 categories as compared to 10 categories for CIFAR- 10, each client typically handles a subset of 13-16 (or 20-25) cat- egories when setting Dirichlet-0.25 (or Dirichlet-O.6). Thus, with such a high degree of heterogeneity incurred on CIFAR- 100, the model updates from clients are more deviated, al- lowing our FedSMU be more effective and demonstrate more significant improvement than on CIFAR-10.
Table S. Top validation accuracy (%) under different participation rate with Dirichlet-0.25 on CIFAR-100 dataset and LeNet model, where NTC indicates the number of total clients, and PR indicates the participation ratc. Bold numbers indicate the best performance.
NTC/PR
50
0.2
100 0.1
150 / 0.066
200 / 0.05
FedSMU
52,39
52.35
51,75
50.22
FodAvg
46.62
42.29
40.93
39.44
SCAFFOLD
52.52
50.49
39.39
37.31
To confirm that our algorithm can maintain good perfor- mance in larger network models, we also conduct evalua- tions on the ResNet18 and ViT-S models. Due to limited computing resources, we set 10 clients in total with a partic- ipation rate of 30%, and show results in Table and Table 4. It can be observed that FedSMU outperforms most base- line methods with ResNet1 8 and VIT-S models. Though it remains slightly inferior to SCAFFOLD on the CIFAR-10 dataset with ResNet18 8 model, it achieves a superior perfor- mance with ViT-S model, further demonstrating its greater advantages in complex network architectures.
SCALLION
48.07
45.24
36.54
35.49
FedEF-HS
42.68
37.03
34.37
31.71
FedEF- TopK
47.25
42.69
40.23
37.41
FedEF-Sign
42.68
36.12
33.94
31.04
FedLion
48.41
48.33
47,81
48.74
deterioration of 7.18%, 15.21%, 12.58%, 10.97%, 9.84%
and 11. 11.64%, respectively. In contrast, FedSMU maintains more stable and superior performance, with only 2. 17% deterioration. These results indicate that our algorithm is minimally impacted by client participation rates and demon- strates greater stability under partial client participation. We attribute this to the use of symbolic operations for the client updates, which effectively leverages each client's update even at very low participation rates. Specifically, when the client participation rate is low, data heterogeneity may cause the update of certain clients to dominate due to larger magni- tudes. Symbolic operations can mitigate this by normalizing the update amplitudes, ensuring that the contributions of all clients are fully considered.
To further evaluate the generalization performance, we de- fine generalization as the test accuracy that algorithm can achieve at the same level of training accuracy. We show that FedSMU continues to demonstrate the best generalization performance, with detailed analysis given in Appendix C.I.
5.2.2. GENERALIZATION VS. PARTICIPATION RATE
We then evaluate the effect of different participation rates on all the algorithms, while keeping the number of participating clients consistent each communication round. Results in Table 5 indicate that FedSMU achieves the highest accuracy in most cases. Specifically, when the number of participat- ing clients is maintained at 10, and when the participation rate decreases from 0.2 to 0.05, FedAvg (McMahan et al., 2017), SCAFFOLD (Karimireddy et al., 2020), SCALLION (Huang et al., 2023), FedEF-HS, FodEF-TopK and FedEF- Sign (Li & Li, 2023)would experience severe performance
5.2.3. GENERALIZATION vs. DATA HETEROGENEITY
We further study the influence of data heterogeneity on the generalization performance of our FedSMU vs. FedAvg and SCAFFOLD. From Table 6, is evident that FedSMU outperforms the other two algorithms. By computing the top accuracy difference between the iid and Dirichlet-0.25 settings in Table 6, we observe a degradation of 3.57%,
8

---

FedSMU: Communication-Ffiiee and Ceneralization-Enanneee FL through Symbolic Model Updates
Table Top validation accuracy (%) under different data hetero- geneity with 100 clients and 10% participation rate CIFAR-100 100 dataset and LeNet model, where Dirichlet-O.. 0.25 indicates the high- est heterogeneity and iid indicates the lowest heterogeneity.
et al., 2024) may further enhance the performance on the modern large language models.
6. Conclusion
Algorithm
Dirichlet-0.25
Dirichlet-0.6
Dirichlet-0.8
In this paper, we have proposed the FedSMU algorithm that could effectively alleviate both the communication cost and data heterogeneity issues of federated learning. The key design the symbolization of local client updates which were introduced to balance the contribution of each client and avoid the dominance by some relatively large update values. We carried out theoretical convergence analysis, and empirically showed that FedSMU converged faster to a higher top accuracy under the same communication cost. Under the condition of a very small partial client participa- tion rate and relatively high data heterogeneity, FedSMU still demonstrated better performance.
FedSMU
$2.35
54.2
54.92
55.92
FedAvg 41.44
43.44
44.21
48.05
SCAFFOLD
50.49
50.02
$3.89
54.78
6.61% and 4.29% in the top accuracy for FedSMU, FedAvg and SCAFFOLD, respectively. Thus, FedSMU is affected less significantly by the degree of data heterogeneity.
Besides, with a horizontal comparison, the improvement of FedSMU over FedAvg is 10.91 %. 10.76%, 10.71% and 7.87% for Dirichlet-0.25. Dirichlet-O.6.. Dirichlet-O.8, and iid distributions, respectively. This indicates that FedSMU achieves a higher performance gain with the increasing de- gree of data heterogeneity. These results also validate that in highly heterogeneous data scenarios, where the difference between clients' model updates becomes greater, FedSMU can alleviate the local model heterogeneity through sym- bolic updates. This promotes the aggregation stability and improves generalization performance of the global model.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants 62320106003, U24A20251, 62401357, 62401366, 62431017, 62125109, 62371 62301 299, 22120106007, in part by the Program of Shanghai Science and Technology Innovation Project under Grant 24BC3300800, and in part by the Al Labora- tory of United Automotive Electronic Systems (UAES) Co. under Grant 2025-3270.
5.2.4. LIMITATIONS
Though our FedSMU effectively enhances the generaliza- tion performance while reducing the communication over- head, it may still have some limitations. First, our com- pression relies on the fixed-precsiion symbol quantization, which might not be optimal for the adaptive scenarios. Ex- ploring adaptive bit quantization further our future inre- search is promising to address this limitation. Second, due to the partial participation inherent in federated learning, the server must broadcast the new global model to initial- ize the newly participating clients at each communication round. This constraint prevents the direct application of our compression techniques to the downloaded global model in FedSMU. We will consider some model lightweight tech-
Impact Statement
Federated learning is crucial machine learning framework that prioritizes the privacy protection. Our work proposes a novel federated compression algorithm aiming at simulta- neously improving the communication efficiency and gen- eralization performance. This approach can be beneficial for the privacy-sensitivy domains, such as healthcare and finance, since transmitting only the sign information makes it more difficult for the attackers reconstruct the original niques, such as mixed precissoe model compression, as promising future research to compress the server-to-client communication in our FedSMU algorithm. Third, though our FedSMU reduces communication costs, it does not necessarily offer an advantage in reducing the communi- cation rounds. This may be due to the noise introduced by the sign operation, which enhances the model's generaliza- tion but meanwhile slows down its convergence. From theoretical perspective, though the generalization proper- ties (Venkateswaran al., 2023) of FedAvg under various assumptions have been extensively examined, such guaran- tees for the compression- based FL approaches remain an open problem. Last, our current focus is only on symbolized local updates with SGD optimizer. However, integrating adaptive optimizers like A to replace SGD (Douillard
data, thereby enhancing the privacy protection level. In addition, by effectively reducing the communication over- head, this work may also enable the practical deployment of federated learning in some extreme scenarios, such as the how-bandwidth networks.
References
Acar, D. A. E., Zhao, Y., Navarro, R. M., Mattina, M., Whatmough, P. N., and Saligrama, V. Federated learn- ing based on dynamic regularization. arXiv preprint arXiv:2111.04263,, 2021.
Aji, A. F. and Heafield, K. Sparse communica- tion for distributed gradient descent. arXiv preprint

---

FedSMU: Communication- and Enborallzation-nnhanee FL through Symbolic Model Updates
arXiv:1704.05021,, 2017.
Condat, L., Maranjyan, A., and Richtárik, P. Locodl: Communication-effiienn distributed learning with lo- cal training and compression. arXiv preprint WrXiv22433.04345 2024.
Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic, M. Qsgd: Communication-efficii sgd via gradient quan- tization and encoding. Advances in neural information processing systems, 30, 2017.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Alistarh, D., Hoefler, T., Johansson, M., Konstantinov, N., Khirirat, S., and Renggli, C. The convergence of sparsi- fied gradient methods. Advances in Neural Information Processing Systems, 31, 2018.
Douillard, A., Feng, Q., Rusu, A. A., Chhaparia, R., Donchev, Y., Kuncoro, A., Ranzato, M., Szlam, A., and Shen, J. Diloco: Distributed low-communication train- ing of language models, 2024, URL httttt//://aax. org / abs 221//31111100 08105.
Basu, D., Data, D., Karakus, C., and Diggavi, S. Qsparse- local-sgd: Distributed sgd with quantization, sparsifica- tion and local computations. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anand- kumar, A. signsgd: Compressed optimisation for non- convex problems. In International Conference Ma- chine Learning, pp. 560-569. PMLR, 2018 8a.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010001412, 2020.
Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization. 23(4):2341 2013.
Bernstein, J., Zhao, J., Arizzadenesheli, K., and Anandku- mar, signsgd with majority vote is communication effi- cient and fault tolerant. arXiv preprint arXiv:1810.05291, 2018b.
Gorbunov, E., Burlachenko, K. P., Li, Z., and Richtárik, P. Marina: Faster non-convex distributed learning with compression. In International Conference on Machine Learning, pp. 3788-377 3798. PMLR, 2021.
Beznosikov, A., Horváth, S., Richtárik, P., and Safaryan, M. On biased compression for distributed learning. Journal of Machine Learning Research, 24(276):1-50. 2023.
Haibo, T., Maonan, L., and Shuangyin, R. Ese: Efficient security enhancement method for the secure aggregation protocol in federated learning. Chinese Journal of Elec- tronics, 32(3):542-555 2023.
Bottou, L., Curtis, F. E., and Nocedal, J. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-314 2018.
Chen, L., Liu, B., Liang, K., and Liu, Q. Lion secretly solves constrained optimization: As Iyарunov predicts. arXiv preprint arXiv:2310.05880 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.1109/CVPR..0166.0
Chen, X., Hsieh, C.-J,, and Gong, B. When vision trans- formers outperform resnets without pre-training strong data augmentations. arXiv preprint 1rxiv22166.01544 2021.
Huang, X., Li, P., and Li, X. Stochastic controlled averaging for federated learning with communication compression. arXiv preprint 2rXiv:2308.08/66 2023.
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham, H., Dong, X., Luong. T., Hsieh, C.-J., Lu, Y., al. Sym- bolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024.
Jain, R. K., Chiu, D.-M. W., Hawe, W. R., et al. A quan- titative measure of fairness and diserimination. Eastern Research Laboratory, Digital Equipment Corporation, Hudson, MA, 21:1, 1984.
Condat, L., Agarsky, I., and Richtárik, P. Provably doubly accelerated federated learning: The first theoretically suc- cessful combination of local training and communication compression. arXiv preprint arXiv:2210.13277 2022.
Jin, R., Huang, Y., He, X., Dai, H., and Wu, T. Stochastic- sign sgd for federated learning with theoretical guarantees. arXiv preprint arXiv:2002. 10940, 2020.
Condat, L., Agarský, I., Malinovsky, G., and Richtárik, P. Tamuna: Doubly accelerated distributed optimization with local training, compression, and partial participation. arXiv preprint arXiv:2302.00832. 2023.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averag- ing for federated learning. In International conference on machine learning, pp. $132-5143. PMLR, 2020.
10

---

FedSMU: Communication-Pffiie and Generalization-Fnin Enhanceed FL through Symbolic Model Updates
Khan, L. U., Saad, W. Han, Z., Hossain, E., and Hong. C. S. Federated learning for internet of things: Recent advances, taxonomy, and open challenges. IEEE Commu- 2021.
Reddi, S. J., Hefny, A., Sra, S., Poczos, B., and Smola, A. Stochastic variance reduction for nonconvex optimization. In International conference machine learning, pp. 3 314- 323. PMLR, 2016.
nications Surveys & Tutorials,
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Reisizadeh, A., Mokhtari, A., Hassani, H., Jadbabaic, A., and Pedarsani, R. Fedpaq: A communication-effiieee federated learning method with periodic averaging and quantization. In International conference artificial intelligence and statistics, pp. 2021-2031, PMLR, 2020,
Li, X. and Li, P. Analysis of error feedback in federated non- convex optimization with biased compression: Fast con- vergence and partial participation. In International Con- ference on Machine Learning, pp. 2023.
PMLR,
Richtárik, P., Sokolov, and Fatkhullin, E121: A new, simpler, theoretically better, and practically faster error feedback. Advances in Neural Information Processing Systems,
Li, X. and Orabona, F. On the convergence of stochastic gradient descent with adaptive stepsizes. In The 22nd international conference on artificial intelligence and statistics, pp. 983-992.100 PMLR, 2019.
2021.
Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A.. Maier-Hein, K., et al. The future of digital health with federated learning. NPJ digital medicine, 3(1):1-7, 2020.
Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, W. J. Deep gradient compression: Reducing the communica- tion bandwidth for distributed training. arXiv preprint 11t2.0171.0888 2017.
Russakovsky, O., Deng. J., Su, H., Krause, J., Satheesh, S., Ma. S., Huang. Z., Karpathy, A., Khosla, A., Bern- stein, M., al. Imagenet large scale visual recognition challenge. International journa of computer vision, 1 115: 211-252, 2015.
Liu, B., Wu, L., Chen, L., Liang, K., Zhu, J., Liang, C.,, Krishnamoorthi, R., and Liu, Q Communication efficient distributed training with distributed lion, arXiv preprint DoXiv:2404.00438, 2024.
Stich, S. U., Cordonnier, J.-B., and Jaggi, M. Sparsified sgd with memory. Advances in neural information processing systems, 31, 2018.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Malekijoo, A., Fadaeieslam, M. J., Malekijou, H., Homayounfar, M., Alizdde--Shabdiz, F., and Rawas- sizadeh, R. Fedzip: A compression framework for communication--eficcenn federated learning. arXiv preprint arXiv:2102.015993 2021.
Sun, T., Wang, Q. Li, D., and Wang, B. Momentum en- sures convergence of signsgd under weaker assumptions. In International Conference on Machine Learning, pp. 33077-33099. PMLR, 2023.
Tang, Z. and Chang, T.-H. Fedlion: Faster adaptive feder- ated optimization with fewer communication. In ICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp. 13316- 13320. IEEE, 2024.
McMahan, B., Moore, F.,, Ramage, D., Hampson, S., and y Arcas, B. A. Communication-eftiiees learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1 1273-1282. PMLR, 2017.
Meinhardt, G., Yi, K., Condat, L., and Richtárik, P. Sparse- proxskip: Accelerated sparse-to-sparse training in fed- erated learning, 2025. URL /tttpp:://axiv....// abs/2405. 20623.
Tong, Q., Liang, G., and Bi, Effective federated adaptive gradient methods with non-iid decentralized data. arXiv preprint 0tti::20000065557 2020.
Mishchenko, K., Gorbunov, E., Takáč, M., and Richtárik, P. Distributed learning with compressed gradient differ- ences. Optimization Methods and Software, pp. 1-16, 2024.
Venkateswaran, P., Isahagian, V., Muthusamy, V., and Venkatasubramannian N. Fedgen: Generalizable federated learning for sequential data. 2023 IEEE 16th Interna- tional Conference on Cloud Computing (CLOUD), pp. 508-318. IEEF, 2023.
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., and McMahan, H. B. Adaptive federated optimization. arxiv preprint arXi::2003.00295, 2020.
Wang. Y., Lin, L., and Chen, J. Communication-effcai adaptive federated learning. In International Conference on Machine Learning, pp. 22802-228222222 -2838. PMLR, 2022.

---

FedSMU: Communication-ffiiceen and Seneralization-nnhaneee FL through Symbolic Model Updates
Wangni, J., Wang, J., Liu, J., and Zhang, T. Gradient spar- sification for communication-effcient distributed opti- mization. Advances in Neural Information Processing Systems, 31, 2018.
Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y., and Li, H. Temgrad: Ternary gradients reduce communi- cation in distributed deep learning. Advances in neural information processing systems, 30, 2017.
Wu, J.. Huang, W., Huang, J., and Zhang. T. Error com- pensated quantized sgd and its applications large-scale distributed optimization. In International conference machine learning, pp. 5325-5355555 53333. PMLR, 2018
Yang, Q.. Liu, Y., Chen, T., and Tong. Y. Federated machine learning: Concept and applications. ACM Transactions Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
Yi, K., Meinhardt, G., Condat, L., and Richtárik, Fedcomloc: Communication-effiiieee distributed train- ing of sparse and quantized models. arXiv preprint arXiv:2403.09904, 2024.
Yu, E., Dong, D., Xu, Y., Ouyang, S., and Liao, Cp-sgd: Distributed stochastic gradient descent with compression and periodic compensation. Journal of Parallel and Dis- tributed Computing, 169:42-57, 2022.
Yu, H., Jin, R., and Yang. S. On the linear speedup analysis of communication efficient momentum sgd for distributed non-convex optimization. In International Conference Machine Learning, pp. 7184-7193. PMLR, 2019.
12

---

FedSMU: Communication-Ffiice and Cenerallzation-Enhaaee FL through Symbolic Model Updates
Appendix
A. Detailed Experimental Setup
We utilize the visual datasets including 110,R-10, CIFAR-100 100 and Tiny-ImageNet. CIFAR-10 10 and CIFAR-I00 are two classic image classification datasets created by the Canadian Institute for Advanced Research (CIFAR). The CIFAR-10 dataset consists of 10 classes of images, with each class containing 6000 32x32-pixel color images. The CIFAR-100 dataset an extension of CIFAR-10, containing 100 classes of images. These 100 classes are divided into 20 superclasses, each containing 5 subclasses. Each subclass contains 600 32 x 32-pixel color images. Both of them comprise 50,000 images for training and 10,000 images for testing. The Tiny-Imageha dataset is reduced version of the ILSVRC classification dataset. It consists of 200 distinct object categories with 64 x 64-pixel color images of 3 channels. Each category includes 500 images for training and 50 for testing.
For CIFAR-10 and CIFAR-100, we employ a LeNet model comprising two convolutional layers with sixty four 5 x 5 filters, two 2 x 2 max pooling layers, two fully connected layers with 384 and 192 neurons, and a softmax layer. We also use a larger network, R confirm that our algorithm still performs well in a larger network. ResNet18 contains 16 convolutional layers. These convolutional layers are distributed across several residual blocks, each containing two 3 x 3 convolutional layers. Additionally, there is 7 x convolutional layer at the beginning of the network. At the end of the network, there is a fully connected layer for output. Vision Transformer (ViT) adapts the Transformer architecture from natural language processing to image classification tasks. In this work, we employ the ViT-Small (ViT-S) variant, which incorporates image patching, positional encoding, and a 12-layer Transformer structure.
For Shakespeare dataset, we employ an RNN model. It consists of the input layer (receiving the input at the current time step), hidden layer (receiving the hidden state from the previous time step along with the input the current time step to compute the new hidden state) and output layer (outputting a result based on the current state of the hidden layer).
All approaches are implemented in PyTorch 1 1.4.0 and CUDA 9.2, with GEFORCE GTX 1080 Ti throughout our experiments.
In most federated learning scenarios, the total number of clients is set to 100 with the participation rate of 0.1, which is a classical experimental setting, like what FedLion (Tang & Chang, 2024) and FMcaagnM et al., 2017) do. Therefore, with the LeNet model, we set 100 clients with participation rates of 0.1 and 0.03 to verify the performance of our algorithm.
However, our computing resources (i.e., GEFORCE GTX 1080 Ti) are insufficient to support experimental setting of 100 clients on the larger network model, we thus reduce the number to 10 clients and set the participation rate 0.3.
We tune the we tune the local learning rate over (1, 0.1, 0.01, 0.001} and set up 5 epochs of local updates with the minibatch B = 50.
over grid to compare the performance of different methods. For local update in all methods,
For our proposed method FedSMU, we tune the parameter B1 and B, over (0.9, 0.99, 0.999], respectively, and set them both to 0.9 for CIFAR- 10, CIFAR- 100 and Tiny-Imageht and 0.95 for Shakespeare. We tune the parameter 7 and 72 over {I, 0. 0.02, 0.018, 0.015, 0.013, 0.01, 0.005, 0.001}. respectively, since they are so sensitive, and set them to 0.015, 0.01 for 10, 0.018, 0.01 for CIFAR- 0.01, 0.01 for ImageNe,, and 0.03, 0.01 for Shakespeare.
For FedSMUMC, we tune the parameter 31 and B2 over {0.9, 0.99, 0.999). respectively, and set them both to 0.9 for CIFAR- 100. We tune the parameter 7/1 and 7/2 over {1, 0.1, 0.01, 0.001), respectively, and set them both 0.01 for CIFAR-100.
For Fed-LocalLin and LonGaloba,,ion, we tune the parameter 31 and B3 over (0.9, 0.99, 0.999], respectively, and set them to 0.9 and 0.99 for CIFAR-100. We tune the parameter 71 and 7/2 over (1, 0.1, 0.01, 0.001), respectively, and set them to 0.001, 0.01 for CIFAR-I00, We tune the parameter fg in Fed-LocalLion over (1, 0.1, 0.01, 0.001} and set them I for CIFAR-100.
B. A Proof of Theorem 4.4
Proof. We set A. = Σ ut = 1 n Σn Sign[3mmt + (1-31)92 319 and ||4.112 = Ei= (Af,, where d is the dimensions of parameters.
Since 7/2 is adjustable, so for each coordinate j. we can assume hr27 S 1 (1 S $1). It has been clarified by Chen et al. (2023) in the Abstract that "Lion is theoretically novel and principled approach for minimizing a general loss Here 7/2 is the weight decay coefficient and z; is the model.
function f(x) while enforcing bound constraint ||0||100 72
13

---

FedSMU: Communication-Effiie and Generalization-nnaanee FL through Symbolic Model Updates
Such assumption has also been used in another algorithm (Liu al., 2024) based Lion optimizer. Thus have A /247// v A In2ar| I 2, and then ||+|| I d and | 720, 2 I 4d.
With Assumption 4.1, we have
f(ac-1) s f(z)) + 211
2111 2z) + | 2zl1
f(x,) + (V)( - + 2 1 A. - -2
f(x)) (v)(z)) ()(()) + (Vf(z)), n - - + Sign[()
(4)
+ 2 4 $1114 0
=(x))) 71 V[ n (Vf(z)) A 72 + Sign (Vf(()) +2Ln7d.
Considering the calculation of A:
A = (VI(z)); A 72 Sign((
1 Σ
(5)
(V
72 + Sign(V/((x))) 4
1=1
For any dimension j assume |/2=11 < 1, and with Assumption 4.3, then we have v)(2() Ex 4 Sigu(V/(<())) s =c] 3GI < Sign(( So.
1247 +
A < 3G1| 11 f(z,) Sign (Vfx(((
(6)
3vaG| n 2 fzz,)) Sig((/\a)))
Substitute Eq. (6) into Eq. (4), we further have
- -m
+m (VS(x;), A. 72 Sign(V/(ax))) +2In/d
A
$ V 33m Vd n1
vf(z)) (ign(Vf(2;))) +2L
G
Taking the expectation of B, with Assumption 4.3 we have
B(B) < E(I 71 vf(z))
71
Σ
otD + E(I Sign V f(()))
71
efll
nK
nKG
< <B([ee)]) +
E
Sign V)(x)))
R "112)
71
n
(8)
t=1
E([leel))
E
Sign (vf(z)))
KG o (1))))
n
i=
d
s E((leel)) + 21
14

---

FedSMU: Communication-Effccce and Anneeralization-nhaanee FL through Symbolic Model Updates
Taking the expectation of Eq. (8), have
- - 3G VdE(leel)) + 6G Vn + 2L7dd.
(9)
Decomposing Et, we have
4
71
V f(x))
+
7/
G
nKG
1=1
111
(E(
71
nKG
(10)
(..)))
+
71
u/)
G
KG
We further define hi = - KC Ek=o (yi kt,k), h4 - (V).
Referring to Algorithm 1, we have of = 820 + (B1 - - + (1 - 31)gt. For each client i, we have
Yi 71 KG
o/ =
KG /
+ 7 32) + + -
=B2(c 71 (F((3,--)) + + -1)(6=1))66-+
1
FF
(11)
G
1 vF(())))
+ +nn(1 - -0))(0
G
Converting the form of Eq. (11), we have
4 = ,26 + t B,)8 + - B1)0 + 71
(F,
FF
(1(
G
(12)
Taking the &g norm of of, with Assumption 4.1, we have
71
||
|VF((.))
(
G
2/11
(F(
(13)
Lmn
||2
G
2LTininVd
G
15

---

FedSMU: Communication-Fffiien and Enteraalizaion-Enhaneee FL through Symbolic Model Updates
Taking the expectation of ||88|| and using the Lemma B.I, we have
B(11))1P) E(I
KG VE(VixiE t VFi(<.))19)
K > 0= B V Fl (V k - VF(())13))
K²G²
(14)
L2K K= El 12ll?
K2G
12 + 16K 16K
G
Taking the & norm of Eq. (12) and using Eq. (13), let T/ = 0, = - Te, max(3shssset1 = -mazi Tel
t - et St 1 and we have
|e/||| s [18ge/_ 11 + || + (1: - 82)8 (1 31)8811
= =11855+2eb)
+ Σ
Bost 11 (1 - -12) Σ
B80
Ti 11/ B1) Σ
B30
j=0
1=0
j=0
2L max
Vinad
d¹
(15)
Σ j=0
82 + (12 8:11l Σ
8220
| 71 -B1)11 Σ
B5 1
G
Ts41
5=0
2T max vinva
$ 88 +1/h+++
(12
8111 Σ
850
(1
81111 Σ
335 |1.
G(1 - B2)
Ty
1=0
j=0
Notice that the random variables (6t) are independent, so E (6) 8/2)
= 0.
Taking the expectation of
E B3 11.
3868 1 and using Eq. (14), we have
E
Σ
83
||
E Σ
B5
115
E(I
Σ
83 112)
T3+1
)=0
E1
32 1
1100
(16)
j=0
1
12 16K2 + 16K2,p62)
B2
G
Taking the expectation of Eq. (15) and substituting Eq. (16) in it, we further have
2L 7nnv G(1 - 32)
+6K3
B||+ا|| S 355 1 +
21 (1-111))))))) 211(1-B1)
B1) 1 B2
(17)
16

---

FedSMU: Communication-Effiien and Generalization-nnaanee FL through Symbolic Model Updates
Recursively iterating it from to T and substituting Eq. (17) into Eq. (10), have
1
f2x0) min S
E Vf(aa)l)))
3GV (GV(B([[E+])) 6G
+
2Lyd
T
II
Vn
f(x0) min IT
DJ 3GVV
-1=
Ei=1 Elleill
d
6G
+ 2L
nf
vn
f(zo) min f
5J
-1=1 Ei=1
gct+1
|
6L 7ind
+ +3GVV
71
nf
B2
1
of + 16K21Pot + 16K21)
+ dGvan(11 -1-111-11 -Bi)
1-
G
(18)
+G n/
+ +Tndd
f(zo) min f
3G
IT
n1 (2)
1 B2
+ 1121 (1)))
(2Ko² 4K20, + 4K222)
1 82
+ 6G vn
+ +Lndd
where Tmax maxm{TTaaa Ma hisimm
Σ m
||/p|| when < T.
Finally, when 21 LVT and 1 - B1 =
VT we complete the proof that
1 B()()(
L(f(ro)
min f)
3Gvdø
Grddmaa
T
VT
nT -22)
LT 32)
12n
d(2Ko7 + 4KK20 + +4K2G2)
(19)
+
T
1- 32
6Gd
2d
Vn
VT
Lemma B.1. Let Assumption 4.1, Assumption 4.2 and Assumption 4.3 hold for Et and P.(:;)) Assume node performs local SGD as
4 = - (VV
with vl.u = Ft. Like the lemma proved in (Sun et al., 2023). since 0 < I 4LK it holds
E Dii xell² < 8K + 16K22 17 oi 16
Proof. Following the proof in (Sun et al.. 2023), note that for any {I...., in client i,
E Vf.e zell = E Ilvi, 17 1/ /VF.(V Si- - x 121
BllybRR-1 v4 (VF. (i
(20)
VE (Vi,e-1) - +VF (V/
-F (xx) + VF; (xx) 2|?
17

---

FedSMU: Communication-Efficeee and Enteraaliztioo-Enhaaeee FL through Symbolic Model Updates
By using the Cauchy's inequality, we have
1 +|l++b1 - (1+ 1+ v
E| + bll2
Bl|all + (1 + +))Bbb|??
17 (VF: (hAAiifis-1) - VP: (witt-1) )) , - 17 (VE: (viA-1) - (V) + VP: (xx)) and
with Vick-1
4/- = 2K- -1 1.
(1 1 2k-1) Bl Bl D2 n (VF: (vi---1:fax-1) - VF; (vLA-=) 11? and 3
We denote R
BK (viA--))
V.F (xx) + VFF (z+) 112. The unbiased expectation property of V.F: (V' ( gives us
(1 (Bl1 - zzll² + 1 || VE (vÉ.A-1:&(,-1)) -VE (vÉ.a-1)l
R =
+ 2K 1
$ 1 + 2K - 1
1
E -llb
mell -
+ nrt
On the other hand, we have the following bound
3 S 4K |VF, (vf.A-1) - V. (xx))² + B VVF (z)))l
I 4L AL' 1 - zel1 + +4Kn G²
When 0 <n ALK
1
2K 1K-17 442 11+ 1
1
1+
K-1
and we can obtain
Vi.k - 23 11?
+ 2K- 1
+ 4L.2 -I
E El xxll² + 2nn²07 + 4K + G
<
1 + K 1
E
+ +21207 + 4Krr00 + G2
The recursion from j = 0 to K yields
K-1
1+ K K-1 )
1 5
E VL, x0l2
Σ
12 + 4kr + + G
1=0
(1+ [1+x-]
1
K
1
$ (K 1)
-1 1 x 12 +
+ + GK
s of + 10 + 16,
where we used the inequality (1+xI)
< 5 holds for any K 2 1.
18

---

FedSMU: Communication-Fficee and Generalization-mmin FL through Symbolic Model Updates
Table Generalization performance comparison under various datasets with LeNet model. Each table entry gives the average test accuracy on different training accuracy levels. "/" means it cannot reach the training accuracy. Bold numbers indicate the hest performance.
Top-1 Test Accuracy (%).
Dataset
Training Accuracy
FedAvg
SCAFFOLD
SCALLION
FedEF-HS
FedEF-Topk
FedEF-Sign
FedLion
FedSMU
83-84
75.14
77.49
75.9
75.63
75.48
75.7
77.22
77.1
CIFAR- 10
85-86
76.17
76.37
76.87
76.83
76.83
77.4
78.3
78.39
87-88
78.56
79.24
77.79
77.85
77.74
/
79.23
79.78
66-67
40.08
45.76
40.56
41.28
/
45.55
49.03
CIFAR-
68-69
40.6
46.16
40.88
/
41.81
/
46.15
50.04
100
70-71
40.9
46.76
41.23
42.34
46.56
50.85
80
70
45
40
accuracy (96)
60:
FedEF-16
FeIEF-IS
FodBR Took
30
Tosk
FodEF Sign
25
RedEF-5ip
Feduion
20
Fedlio
30
FedsHil
FedSHU
20
Feebin
SCARFOLD
10
SCAFFOLD
Fedhig
10
SCALLION
SCALLON
1000 1503 2000 2500
3000
1309
3000 3500
commmunication round
commmunication round
(a) Dirichlet0225-CIAAA 25-CIFARI0
(b) Dirrchlet0.25-CIFA.. 100
Figure 3. Convergence performance vs. number of communication rounds on CIFAR-10 10 and CIFAR-100, with 100 clients and 10% participation, using LeNet model for different algorithms.
C. Additional Experiments
C.1. Measure of Generalization
In above experimental results in main text, generalization refers to an algorithm's ability to achieve top test accuracy, where the test dataset is different from the training dataset. Furthermore, we consider an additional perspective on generalization further evaluate the performance of our FedSMU algorithm. Here, generalization refers model's ability to achieve test performance similar training error levels. Based this definition, compare the validation performance at similar training accuracy levels. The results in Table show that on the CIFAR- 10 and CIFAR-100 datasets with LeNet model, the FedSMU algorithm achieves the highest test accuracy and demonstrates the best generalization performance.
C.2. Convergence Performance vs. Communication Rounds
In Section 5.2.1, considering that FedSMU is compression algorithm, we compare the convergence of different algorithms in terms of communication bits. However, communication rounds are also important, thus in Figure 3, we show the convergence performance terms of communication rounds. Also, we compare the convergence rate using the number of communication rounds required to achieve the target accuracy and the results are presented in the Table
For Shakespeare, our algorithm does require more communication rounds compared to most algorithms. However, for CIFAR-10 and CIFAR-100 it slightly exceeds the number rounds needed by other algorithms. This may be because the distribution of image data is more complex, with each sample containing large amount of pixel information. Training with such highly heterogeneous data results in the gradients that, after taking the sign, introduce noises in the training process, thereby slowing down the convergence. However, from a long-term perspective, these noises can lead to an improved model performance.
C.3. Convergence Performance vs. Wall-Clock Time
We test the wall-clock time needed for each baseline to execute one communication round. Taking CIFAR-100 and participation rate n 0.1 as an example, the average wall-clock time required to execute a round is as follows: FedSMU (10.43 seconds), FedAvg (10.15 seconds), FedEF-HS (10.46 seconds), FedLion (10.63 seconds), SCAFFOLD (10.38 seconds). Experiments demonstrate that in single communication round, our algorithm introduces no significantly
19

---

FedSMU: Communication-Efficiee and Ccneralization--naaneee FL through Symbolic Model Updates
Table Number of communication rounds to achieve a preset target accuracy with 100 clients 10% participation. CIFAR-10 and CIFAR-100 use the LeNet model and Shakespeare uses the RNN network. "f" means it cannot reach the training accuracy. Bold numbers indicate the best performance.
Number of communication rounds achieve a preset target accuracy.
Dataset
Training Accuracy (%)
FedAvg
SCAFFOLD
SCALLION
FedEF-HS
FedEF-TopK
FedFF-Sigg
FedLion
FedSMU
55
26
29
23
41
37
50
21
65
CIFAR-10
60
43
43
29
62
57
81
33
118
(Dir0.25)
65
57
62
48
108
96
111
50
288
35
193
86
286
690
355
794
CIFAR- 100
629
730
882
(Dir0.25)
45
270
3703
632
1811
25
17
12
13
30
20
57
10
11
Shakespeare (noniid)
30
32
19
20
45
36
78
17
20
35
61
27
27
85
68
177
28
48
70
190
60
Fekkog
530
Feskig
40
SCKTOLD
20
SCAFFOLD
30
Fadt-Topk
Fedd Tock
Fedal Topk Fedlice
10
10
0.0 244 48 7.2 12.016416.8
0.06 24 48
5.6 12.0144168
0.96
48 7.2 966 12.0144468 cazon (GB)
0.6 2.4 4. 7.2 96 12.01441664
commmmnncaacn (GB)
conmimu
contouunaate
(a) DIIFRRO
Dir(O.6)-CIFARIO
Dir(0.5))CIFARIO00
(d) Dir(0.6)-CIFAN) 100
Figure 4. Convergence performance vs. number of communication rounds CIFAR-10, 10, CIFAR 100 dataset and LeNet model, with 100 clients and 10% participation.
additional time overhead compared other algorithms. Therefore, the results using wall-clock time metric are similar to those measured by communication rounds. We will not include separate plot here and please refer to Figure and Table 8.
C.4. Convergence Performance Considering Uplink Communications
In Figure 2, only consider uplink (client-to-server communication cost and assume the downlink communication overhead to be the same. Here, we can define the total communication cost per round as presented in (Condat et al., 2022):
Total Communication Uplink Communication + Downlink Communication, c [0, 1].
In practice, due to the factors such as system asymmetry, caching constraints and protocol limitations, the uplink speed is often significantly lower than the downlink speed, as discussed in (Condat et al., 2023). Consequently, many communication- efficient FL studies (Li & Li, 2023; Richtárik et al., 2021) focus merely on minimizing the uplink cost alone.
To have a more comprehensive evaluation of our FedSMU, we followed the setting in (Condat et al., 2023) and set = 0.1, and depicted Figure compare the total communication cost including both the upload and download bits. It shows that FedSMU remains communicaioon-efiiceen even when accounting for the downlink overhead, with a significantly lower total cost compared to the other baselines at comparable accuracy levels.
C.5. Discussion on a-bit
Here, further discuss the extension from the 1-bit compression an a-bit one. First, we would like to acknowledge that for the general quantization- based compression algorithms, higher precision quantization may often lead to faster convergence. However, this may not hold for our FedSMU. This conclusion is based on our analysis, follows.
1) Both experimentally (Figure 1) and intuitively, the symbolic operation (i.e., 1-bit quantization) helps alleviate the heterogeneity of model updates, as all updates have uniform magnitude across all dimensions for each client. Furthermore, reducing model heterogeneity should also intuitively contribute to improving model performance in heterogeneous federated settings.
20

---

FedSMU: Communication-Fffcccni and Ccneralliationn-nhaane FL through Symbolic Model Updates
Table Number communication rounds to achieve a preset target test accuracy with Dirichlet-0.25 on CIFAR-10 dataset with LeNet model. "f" means it cannot reach the test accuracy and bold numbers indicate the smallest rounds.
Number of rounds needed for achieving a target test accuracy.
Test Accuracy(%)
1-bit (FedSMU)
3-bit
8-bit
40
26
65
54
45
33
91
68
50
50
168
119
55
65
285
185
60
118
456
224
65
288
833
342
67.5
399
1260
401
69
507
1967
456
72.3
642
/
643
75
1142
/
1040
77.5
1746
/ 1979
Table 10. Number of communication rounds to achieve preset target test accuracy with Dirichlet-0.25 CIFAR-10 dataset and different CNN model. d, and d2 indicate small and large dimension while and H indicate low and high participation rates. Bold numbers indicate the smallest rounds.
Number of rounds needed for achieving a target test accuracy.
Test Accuracy(%)
d₁, H
d2, H
d₁. L
40
26
30
46
45
33
35
112
50
50
43
193
55
65
226
60
118
111
384
65
288
275
899
67.5
399
409
949
69
507
507
1025
72.3
642
769
1752
75
1142
1185
2221
77.5
1746
1745
2878
2) However, such sign operation (e.g., signSGD) alone does not directly improve generalization in experiments. Inspired by Lion optimizer (Chen et al., 2024) that incorporates the sign operation and then enhances the convergence and generalization to learn in central learning, we introduce Lion's 's structure into federated learning and verify that this combination can indeed improve model generalization.
Consequently, we conclude that in our optimized structure, 1-bit quantization outperforms higher-bit quantization, since multi-bit compression does guarantee that the update amplitude of each client is consistent. Experimental results in Table 9 further validate that for our designed optimization algorithm, using higher- bit compression may not enhance the algorithm's convergence generalization.
C.6. Factors Influencing Convergence Speed
Theoretically, a lower client participation rate (i.e., larger Tmax) leads slower algorithm convergence. Similarly, higher model dimension d also results in slower algorithm convergence. To validate this, we have conducted the following experiments.
All these experiments are done on CIFAR-10 dataset with Dirichlet-0.25. To illustrate the relationship between the model dimension and convergence rate, we use two different CNN models (d, = 797248 and d2 = 17236448) to study the impact of model dimension. Note that here we modify the size of the convolutional layers, keeping the model depth constant. The
21

---

FedSMU: Communication-Effccee and Gcneralization--nhaane FL through Symbolic Model Updates
Table Top accuracy (%) comparison between ablation experiments on CIFAR-100 dataset (Dirichlet-0.25) with LeNet model, where NTC indicates the number of total clients, and PR indicates the participation rate.
NTC PR
FedSMU
FedSMUMC
100 / 0.1
52.35
52.53
100 0.03
51.87
52.14
Table 12. Top validation accuracy (%) on CIFAR-100 dataset with Dirichlet 0.25 and LeNet model, with 100 clients and 10% participation rate.
Top-1 Test Accuracy (%).
Dataset
Setting
FedSMU
Fed-LocalLion
Fed- GlobalLion
FedSMU(12
0)
FedSMU 0)
(fulMUUUrll--recisiin
CIFAR-
Dir (0.25)
$2.35
36.77
47.94
51.34
28.03
42.67
40
35
20
15
10
FedSMU
Globalton LocalLion
0.06
0.6
2.4 3.0
3.6
commmunication bits (GB)
Figure Convergence performance vs. number communication bits on CIFAR- 100 dataset and LeNet model, with 100 elients and 10% participation rates, Dirichlet-0.25 for different ablation algorithms and FedSMU.
number of clients 100 with the participation ratio of 0.1. To illustrate the relationship between participation rate and convergence rate, We use the CNN model with d₁ 797248 and set the number of clients as 100 with different participation ratio ( 0.03 and n rate.
m 0.1, represented in the Table 10 by L and H) to demonstrate the influence of client participation
The experimental results in Table 10 show that when the participation rate is higher (i.e., the Tmar is smaller) and the dimension is smaller, the convergence speed can be faster, which matches with the Theorem 4.4 and is intuitional.
C.7. Variants Solving Momentum Staleness
While our convergence analysis and experimental results demonstrate that FedSMU's performance is less affected by the client participation rate, the momentum of clients may still be extremely stale due to the partial participation in FL. In light of this, we design a variant, named FedSMUMC, to examine the impact of this momentum staleness on the generalization performance. For FedSMUMC, clients upload 1-bit model updates along with extra momentum the full precision. The server then aggregates that momentum update the global momentum and broadcasts momentum for the participating clients. See Appendix for the detail of this algorithm. Results in Table 11 indicate that by uppropriately completing the momentum, additional transmission of momentum with the full precision. Consequently in this sense, the local momentum staleness has a minimum impact on the global model's performance.
the next round as the initial
marginally enhance the model performance, but it necessitates
C.8. Ablation Algorithms
To verify the effectiveness of different FL algorithms built upon the Lion optimizer in terms of the generalization and compression performance, we design additional variants of FL incorporated with Lion, namely Fed-LocalLion and Fed- GlobalLion. Specifically, Fed-LocalLion executes the Lion optimizer locally in parallel at clients, with the server performing model aggregation via weighted summation. On the other hand, Fed-GlobalLion conducts the vamilla SGD locally, treats model aggregation as -euudiggrddeett on the server side, and updates the global model through the Lion optimizer. See
22

---

FedSMU: Communication-Efficee and Anneeralizationnnhaaeee FL through Symbolic Model Updates
Appendix D for the detail of these algorithms.
Our FedSMU consistently outperforms the other variants, illustrated in Figure 5 and Table 12. It is worth noting that FedSMU also integrates additional model compression, whereas these variants require the same communication overhead as FedAvg. This suggests that our FedSMU design effectively harnesses the benefits of Lion, enhancing the generalization while compressing the communication load.
To further assess the necessity of key components FedSMU, we conduct a systematic ablation study by removing specific elements and comparing each pruned variant against the full algorithm. In particular, we examine the effect of excluding the following components: 1) server-side weight decay regularization (?2), 2) client-side gradient sliding average (31), and 3) client-side gradient symbolization.
The experimental results in Table 12 demonstrate that the client-side sliding average plays the most crucial role in achieving a stable and effective training. Additionally, the model update symbolization mechanism itself proves to be more effective than using the full-pree precision updates. This validates our motivation of proposing FedSMU that symbolization balances contributions of the heterogeneous clients by suppressing some extreme update magnitudes, which thus enhances the aggregation stability and leads better generalization.
C.9. Comparison with Distributed Lion (Liu et al., 2024)
Here, we clarify the differences and advantages of our FedSMU compared to the D-Lion (Liu et al., 2024), as follows.
1) Motivation. Our FedSMU can simultaneously mitigate data heterogeneity and reduce communication compression through the symbolic operations. The analysis carried out and verified by experiments (Figure 1). While D-Lion only considers to compress the communication.
2) Scope of application. Our FedSMU can deal with scenarios involving the partial client participation and multiple local updates, whereas D-Lion can not. Performing multiple local updates, in the federated settings, can effectively reduce the communication frequency and thus the overall traffic. Experimental results Table 13 and Table 14 demonstrate that D-Lion fails in such scenarios with low client participation rates and multiple local updates, whereas FedSMU remains robust and performs well under these conditions.
3) Algorithms design. While both algorithms are based on the Lion optimizer, FedSMU fully leverages the structural advantages of the Lion optimizer, including weight decay in the global aggregation. In contrast, D-Lion primarily incorporates the momentum sliding averaging and symbolic operations at local update. This comprehensive utilization of the Lion optimizer structure may explain why the experimental performance of our FedSMU surpasses that of D-Lion.
4) Compatibility with majority vote. We have further extended FedSMU with majority vote, as FedSMU-MV. Experimental results show that FedSMU-MV achieves an accuracy of 47.66% on CIFAR-100, slightly lower than FedSMU's 51.79% under the same settings (number of clients 100, participation rate 0.1, Dirichlet 0.25). This indicates that majority vote is compatible with our algorithm. The slight accuracy drop may result from FedSMU's symbolic model updates. Applying majority vote to the 1-bit results could further suppress some clients' model update information due to the dominant update direction.
Below, we provide the details of the hyperparameters used in our experiments.
To ensure a fair comparison, both algorithms are evaluated on the CIFAR-10 and CIFAR- 100 datasets, using non-llD data (Dirichlet distribution with parameter of 0.25), with total of 10 clients anda batch size of 50.
For FedSMU and FedAvg, we adopt the same parameter settings as outlined in Appendix A.
For D-Lion, we performed a grid search. The learning rate (e) is selected from {0.00005, 0.0005, 0.005, 0.015). the weight decay (A) is chosen from (0.0005, 0.005, 0.001, 0.01} and 31 32 are selected from (0.9,0.99). For Table 13, the selected values are 0.0005, x = 0.001, B1 0.9, B2 0.99. For Table 14, the selected values are 0.015. x = 0.01, 3 = 0.9, 32 = 0.9.
Specifically, from the result in Table 13 and Table 14, we have following observations.
With full participation and one local update (i.c., K = 1 with F), FedSMU performs slightly worse than D-Lion. However, in scenarios with partial participation, FedSMU consistently outperforms D-Lion. This is intuitive, as
23

---

FedSMU: Communication-Fffiienn and Enoeralization-Enhaneee FL through Symbolic Model Updates
Table 13. Performance comparison on CIFAR-10 and CIFAR-100 datasets with LeNet model, where indicate full and partial
participation rates, and K is the number of local updates.
Top-1 Test Accuracy (%).
Dataset
Setting
Algorithm
K = 1 with F
K = with P
K = with F
K = 5 with
FedSMU
32.47
38.35
77.97
75.14
Dir-0.25
D-Lion
34.03
24.58
77.62
34.48
FedAvg
79.64
74.99
72.02
71.48
CIFAR-10
FedSMU
81.84
77.99
82.37
81.71
iid
D-Lion
82
29.06
82.36
44.05
FedAvg
79.53
76.7
76.3
75.84
FedSMU
14.85
20.41
45.69
42.06
Dir-0.25
D-Lion
15.42
3.9
45.54
8.03
FedAvg
44.99
39.34
36.55
36.51
CIFAR-100
FedSMU 50.98
47.18
49.76
49.72
iid
D-Lion 51,46
5.13
50.11
13.07
FedAvg
44.85
41.07
41.38
38.25
Table 14. Performance comparison CIFAR-I0 and CIFAR-100 datasets with LeNet model, where F and P indicate full and partial participation rates, and the number local updates.
Top-1 Test Accuracy (%)
Dataset
Setting
Algorithm
K 100 with F
K 100 with
K 500 with
500 with P
FedSMU
82.24
82.32
82.0
82.08
CIFAR-10
Dir-0.25
D-Lion
82.19
25,86
81.6
51.23
FedSMU
50.15
50.62
46.66
48.2
CIFAR-100
Dir-0.25
D-Lion
49.84
4.05
46.55
16.21
D-Lion does not maintain a complete global model the server and only aggregates the global model updates. Thus in the partial participation settings, asynchronous clients can only save stale global model. result, these clients may receive the global model updates, which, however, cannot be leveraged recover the exact global model of the current round.
With multiple local updates (i.c., K 1), FedSMU mostly outperforms D-Lion. This performance improvement can be attributed to the different approaches weight decay. Specifically, the hyperparameter "22 (denoted as À in D-Lion) controls the weight decay (or La penalty) coefficient. In FedSMU, the regularization is applied to the global model st, potentially mitigating overfitting and thus enhancing generalization. In contrast, D-Lion applies this regularization to the local model - As a result, when the local updates occur multiple times, D-Lion' 's regularization primarily affects the local model, and does not directly improve the generalization capability of the global model. Consequently, when finally evaluating the generalization performance of the global model, FedSMU demonstrates a significant advantage over D-Lion.
In heterogeneous scenarios, the performance of both FedSMU and D-Lion is poorer than that of FedAvg, especially when K is small. This an interesting and somewhat unexpected finding, which we speculate due the data heterogeneity. In the heterogeneous settings, each client samples mini-batch of data for training and performs only single time of update, followed by the application of the sign operation to the model update. Since the local update occurs only once, it introduces substantial sampling variance and inter-client variance. The sign operation, which normalizes the magnitude of updates, may inadvertently amplify this variance between clients, leading to an unstable or even divergent global model aggregation.
24

---

FedSMU: Communication-Fficee and Generalization-nnin Enhancee FL through Symbolic Model Updates
Table 15. Performance comparison under different datasets with LcNet model, where I. and H indicate high participation rates. Bold numbers indicate the best performance.
Top-1 Test Accuracy (%)
Dataset
Setting
FedSMU
EF21
Dir(0.255-L.
80.12
74.43
CIFAR-10
Dir(O.25)-H
80.74
81.51
CIFAR-100
Dir(0.25)-H
52.35
50.07
Table 16. Performance comparison under different datasets with 100 clients and 10% participation rates, Dirichlet-0.25,, LeNet model. Bold numbers indicate the best performance.
Top-1 Test Accuracy (%).
Dataset
FedSMU
FedAMS
FedCAMS
CIFAR-10
80.74
82.47 47.97
80.15
CIFAR-100
52.35
48.3
200
56
2 160
54
1.0
70-
50
FedAvg
48
FedAvg
FecAug
FedSMU
2
Scaffold
SMd
SCAFFOLD
42
53
Data Hetergeewity (Dirchlet)
data neterogenelty (Dirichlet)
Communication Round (Dincheet 10. 2512
(a) MU index vs. heterogeneity
(b) Accuracy vs. heterogeneity
(c) MU vs. communication round
Figure 6. Magnitude uniformity (MU) index and top validation accuracy of FedAvg, SCAFFOLD and FedSMU (ours) CIFAR-100 with LeNet model.
C.10. Comparison with EF21 (Richtárik et al., 2021)
We further explore Error Feedback 2021 (Richtárik et al., 2021) algorithm as state-of-the-art method for Top-K compression and make a comparison with
Experiments on CIFAR-IO and CIFAR-100 are conducted. We set a total of 100 clients with different participation rate (3% and 10%, represented in Table 15 by L and H) and use Dirichlet-0.25. The experimental results shown in Table 15.
On CIFAR-100, FedSMU still shows high performance. While on CIFAR-I0, the accuracy of FedSMU can be higher than EF21 with lower participation. These results strongly demonstrate the superiority of FedSMU in complex image classification tasks, especially under a low client participation rate, which may result from the sign operation promoting the fair contribution clients effectively to the global model update.
C.11. Comparison with Adaptive Algorithms
We compare with two adaptive algorithms (Wang et al., 2022): the optimization-based FedAMS and the compression- based FedCAMS. FedAMS is designed to accelerate the convergence using momentum, while FedCAMS extends FedAMS by further compressing the upload communication. Experiments are conducted on CIFAR-10 and CIFAR- datasets. use total of 100 clients with partial participation ratio of 0.1 and employ a Dirichlet distribution with a concentration parameter of 0.25. The experimental results presented in the Table 16.
The experimental results demonstrate that on the CIFAR-10 10 dataset, FedSMU also outperforms FedCAMS but is slightly inferior to FedAMS, while FedSMU exhibits a superior performance compared to FedAMS and FedCAMS on the CIFAR- 100 dataset. The results strongly demonstrate the superiority of FedSMU in complex image classification tasks, even comparable to the uncompressed federated adaptive algorithm, which may result from promoting the fair contribution of clients effectively to the global model update.
25

---

FedSMU: Communication-Efficce and Ceneralliation-nnhaneee FL through Symbolic Model Updates
C.12. Magnitude Uniformity Index of More Algorithms
We provide Figure show the correlation between Magnitude Uniformity (MU), data heterogeneity, and accuracy of three different algorithms.
The results indicate that with FedAvg, data heterogeneity significantly amplifies the differences in the magnitude of model updates across clients, leading to unstable global aggregation and poorer generalization performance. While SCAFFOLD reduces variance to address these differences, FedSMU directly ensures consistency across all model updates through symbolic operations. Those two approaches enhances Magnitude Uniformity among clients, ultimately improving accuracy.
D. OTHER ALGORITHMS
FedSMUMC, as a variant evaluated in the ablation study of our FedSMU, is shown in Algorithm 2. The basic procedure is equivalent to FedSMU. At each round E [T], subset of clients N₁ S M are active, and the server transmits its current model Fe and glohal momentum Me to these clients. Local clients also additionally transfer mú back to the server (Line 13)
and average them to update the momentum for the next round (Line 16).
Algorithm FedSMUMC
Server Initialization: 21, M1;
for each round = 1, ...T
sample clients N₁ C MM
for each client E N2 in parallel do receive and initialize local model vi_u = 3t
receive momentum M, for each local step k- 1,2, ..., K do Vi.x -Vi,k- 1/ (V
end
si vi,K Vi,0
U, Sign( B1 M2 + (1 B1)g()
m M, + (1 - 32)gt
send u,, mi to server end
// at server:
Mc+1 Ei=1 mi
se+1 = Ft + 71 Dimi ut 7204)
broadcast T1+1, M
end
Fed-LocalLion, variant evaluated in the ablation study of our FedSMU, is shown in Algorithm 3. At each round E [7'] subset of clients Nr C .M are active, and the server transmits its current model a; these clients. Each active client then performs SGD (Line 8) and uses the Lion optimizer to further update model. The server aggregates the local model difference A to compute 2++1-
Sed-GlobalLion. as a variant evaluated in the ablation study our FedSMU, is shown in Algorithm 4. each round a subset of clients N₁ M are active, and the server transmits its current model x; these clients. Each active client then updates the local model (Line 8) and sends the model difference ş to server. The server aggregates the of the global model difference Ge (Line 14) and uses the Lion optimizer to update.
t e [T].
26

---

FedSMU: Communication-Efiiien and Generalization-Emni Enhanced FL through Symbolic Model Updates
Algorithm 3 Fed-Locall.ion
Server Initialization: 21;
Client Initialization: mi = 0;
for each round f 112....7 do
sample clients N, C M
for each client i E Nr in parallel do receive and initialize local model Vi.0 = Ft
for each local step k = 2, K do
v -ViA-1 17
end
gi Vi,K - Vi
u Sign( (3 m + (1-
111-31 31)gi)
m B2m; + (1 - B2)gt (for i & No. mi = mi/1)
Vi Vi,K + +u k2
A = yi - Vi,0
send A to server
end
// at server:
Ie+1 = It + No( Ei= A!)
broadcast Zz+1
end
Algorithm 4 Fred-GlobalLion
Server Initialization: 21, My 0;
for each round t 1,2, ...T do
sample clients N, M
for each client E N2 in parallel do
receive and initialize local model v 3t
for each local step is 2......... do
vi vi. 17 7, FVF
end gi = Vi.K vi,o send of to server end
// at server:
G, = 1 Σn gi
U, = Sign (1- (1 G1)
M, = 32M, + +1 (1
Ft+1 = x, + (U, 7221)
broadcast Z++1
end
27


Reviews:
[{"review_id": "uu1Kget1Ju", "reviewer": "Reviewer_ZtNk", "summary": "In this paper, the authors propose the FedSMU algorithm to address both communication cost and data heterogeneity challenges in federated learning, through sign-based model compression.\n\n## update after rebuttal\n\nI continue to favor acceptance and will leave my rating unchanged.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’d like to thank the reviewer for the comments.\n> Motivation & Contribution\n\nFL involves intertwined challenges where improving one may worsens another. For example, communication-efficient methods may reduce generalization, methods addressing heterogeneity often increase communication, and those balancing both may assume full client participation.\n\nWe aim to design an algorithm addressing both challenges while remaining effective under partial participation. To this end, we introduce the Magnitude Uniformity (MU) index, inspired by Jain’s fairness, to quantify consistency of client contributions. We observe that higher heterogeneity reduces MU, but degrades generalization. To mitigate this, we propose model update symbolization, which normalizes update magnitudes to enhance MU, thus implicitly reducing heterogeneity and communication. To address potential accuracy loss from compression, we integrate a sliding average mechanism (inspired by Lion) for improved stability.\n\nUnlike existing methods that trade off one aspect for another, our core contribution is to jointly improve communication efficiency and generalization, while supporting partial client participation.\n> Few-shot FL\n\nOne-Shot FL typically trains a global model with a single communication round by ensembling and distilling client models using public data [A1, A2]. While FedSMU takes a multi-round paradigm, it shares similar goals and can potentially incorporate One-Shot strategy.\n\nTo our best knowledge, [1, 2] are methodologically different: [1] proposes ACClip for handling heavy-tailed gradients in Transformer, and [2] studies linear Transformers for regression. We show that ACClip is compatible with FedSMU, with provenance improved by FedSMU-ACClip in Table A1 (https://anonymous.4open.science/r/A-7823/F.pdf).\n\n[A1] M. Hasan, et al. \"Calibrated one round federated learning with Bayesian inference in the predictive space,\" AAAI 2024.\n\n[A2] N. Guha, et al. \"One-shot federated learning,\" *arXiv:1902.11175* 2019.\n> Client sampling\n\nWe assumed a uniformly random participation (unbiased sampling), allowing to isolate and study effects of data heterogeneity and communication. \n\nThough client sampling is not our main focus, FedSMU is still compatible with biased sampling. We incorporate it with loss-based client selection [3], and show in Table A2 that FedSMU with biased sampling improves performance.\n> Larger models and larger datasets, e.g., DomainNet and GLUE\n\nWe conducted additional experiments on CIFAR-100 with ViT-Small model. Four strong baselines are compared: FedAvg, SCAFFOLD, FedEF-TopK and FedLion. Table A3 shows that FedSMU consistently outperforms them on this larger model, providing a strong evidence that our sign-based 1-bit quantization approach generalizes well to more complex models.\n\nDue to time constraints, we have not yet evaluated FedSMU with LLMs on RoBERTa or GLUE, but consider this a promising direction for our future work. We will also include a discussion of centralized low-bit quantization methods [5, 6] in Related Work, to better position our contribution within the FL context.\n> Fluctuations in early stages\n\nWe clarify this apparent discrepancy from the following perspectives.\n\n1)Figs. 1(c) and 2 present different metrics. Fig. 1(c) shows the MU index over communication rounds, while Fig. 2 plots the test accuracy against cumulative communication cost. As such, the x-axes differ in both the scale and meaning, and are not directly comparable.\n\n2)In Fig. 1(c), FedSMU maintains a stable MU from the start due to symbolization, while FedAvg gradually improves. In Fig. 2, the early accuracy fluctuations in FedSMU are due to the initial quantization noise when the model is still far from the convergence. These fluctuations would diminish as training progresses and when symbolic updates begin to stabilize the training process.\n> Theoretical assumption\n\nWe acknowledge that recent studies suggest that mini-batch gradients in attention-based models may follow heavy-tailed distributions, thus challenging the standard assumptions in FL, such as the bounded variance. \n\nIn this paper, our theoretical analysis is based on the widely adopted assumptions in centralized and federated learning, particularly for CNNs and LSTMs, which follows the assumption in [A3, A4]. Besides, our evaluation on larger models empirically shows that FedSMU still performs well on ViT-S, which might indicate a practical robustness to the non-Gaussian gradients.  \n\nFinally, we totally agree that better aligning the theory with behaviors of Transformer-based models is an important direction, and will briefly discuss this in the revised theoretical section.\n\n[A3] X. Li,  et al. \"Analysis of error feedback in federated non-convex optimization with biased compression: Fast convergence and partial participation,\" ICML 2023.\n\n[A4] X. Huang, et al. \"Stochastic controlled averaging for federated learning with communication compression,\" *arXiv:2308.08165* 2023.", "claims_and_evidence": "1. The motivation is unclear, particularly the deep rationale connecting the three major challenges. The paper integrates multiple techniques, including model update symbolization/sign operation/1-bit compression, sliding average, MU Index, and model compensation. However, each technique addresses a different challenge, and there is no clear connection between them.\n2. The paper attempts to tackle multiple issues simultaneously, making its main contributions unclear. I strongly recommend that the authors refine the focus of the paper and clearly highlight its core contribution.\n3. Regarding communication efficiency, few-shot or even one-shot FL methods [1][2] have been proposed. The authors should discuss these works in relation to their approach.\n4. Client sampling in FL is another important issue [3][4], but it is not adequately addressed in the paper. From my understanding, partial client participant is one of the biased client sampling. However, the authors only consider smaller sampling rate. A discussion on how $\\textbf{biased}$ client selection impacts the proposed method would be beneficial.\n5. Several low-bit model quantization methods [5][6] have been proposed. Can sign-based 1-bit quantization be effectively applied to larger models? Is the quantization error within an acceptable range when using LLMs?\n6. As shown in Figure 2, FedSMU exhibits large fluctuations in the early stages, which appears to contradict the findings in Figure 1(c). Can the authors elaborate on this discrepancy?\n7. The evaluated datasets and models are not representative of real-world applications. To fully assess the proposed method, I recommend evaluating it on larger models such as ViT and RoBERTa and larger datasets such as DomainNet and GLUE.\n8. The writing needs improvement to meet the standards of a prestigious conference like ICML. The sentence flow is disjointed (e.g., Line 23-24 on the left, Line 25 on the right). Some sentences are overly long (e.g., Line 19-27 on the left). Additionally, there are ambiguous terms, such as “values” in Line 44 on the right—how do smaller values reduce data transmission costs? Do you mean precision? Furthermore, multiple keywords with similar meanings (e.g., symbolic/sign/1-bit compression) should be consolidated for clarity. Sometimes the past tense is used as well.\n9. I feel a big gap between the contribution summary and the paragraph from Line 70-90.\n\n[1] Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., & Sra, S. (2020). Why are adaptive methods good for attention models?. Advances in Neural Information Processing Systems, 33, 15383-15393.\n[2] Ahn, K., Cheng, X., Song, M., Yun, C., Jadbabaie, A., & Sra, S. (2023). Linear attention is (maybe) all you need (to understand transformer optimization). arXiv preprint arXiv:2310.01082.\n[3] Chen, W., Horvath, S., & Richtarik, P. (2020). Optimal client sampling for federated learning. arXiv preprint arXiv:2010.13723.\n[4] Cho, Y. J., Wang, J., & Joshi, G. (2020). Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243.\n[5] Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., ... & Wei, F. (2024). The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764, 1.\n[6] Malekar, J., Elbtity, M. E., & Zand, R. (2024). Matmul or No Matmul in the Era of 1-bit LLMs. arXiv preprint arXiv:2408.11939.", "methods_and_evaluation_criteria": "See Claims And Evidence.", "theoretical_claims": "The paper assumes that the stochastic gradient is an unbiased estimator of the full gradient and that its variance is bounded. However, these assumptions do not align well with recent findings on minibatch gradient distributions in transformer-based models (as I mentioned before that I am concerned the method is not working well with LLMs). Specifically, prior research ([7-8]) has shown that minibatch gradients in attention-based models follow a heavy-tailed distribution rather than a Gaussian-like assumption with bounded variance. This discrepancy raises concerns about the applicability of the theoretical results, as heavy-tailed gradient noise can significantly impact convergence behavior. A discussion on how the proposed method handles such distributions would strengthen the paper.\n\n[7] Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., & Sra, S. (2020). Why are adaptive methods good for attention models?. Advances in Neural Information Processing Systems, 33, 15383-15393.\n[8] Ahn, K., Cheng, X., Song, M., Yun, C., Jadbabaie, A., & Sra, S. (2023). Linear attention is (maybe) all you need (to understand transformer optimization). arXiv preprint arXiv:2310.01082.", "experimental_designs_or_analyses": "See Claims And Evidence.", "relation_to_broader_scientific_literature": "no", "essential_references_not_discussed": "See Claims And Evidence.", "other_strengths_and_weaknesses": "See Claims And Evidence.", "other_comments_or_suggestions": "See Claims And Evidence.", "questions_for_authors": "See Claims And Evidence.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZDLI4p41li", "reviewer": "Reviewer_pLQH", "summary": "This paper proposes a new federated learning algorithm, FedSMU, designed to reduce communication costs and mitigate data heterogeneity. The key idea is to transmit only the sign of local updates for each parameter. Both theoretical analysis and experimental results are provided.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the comments. In the following, we have provided our detailed responses to these comments.\n\n> Comment 1: The motivation and core design of the proposed algorithm are clearly articulated. However, it is unclear what trade-offs are made to achieve communication savings.\n\n**Response:**\n\nWe thank the reviewer for the insightful comment. In our FedSMU, the primary trade-off made to achieve communication efficiency is the loss of precision in the model updates incurred by the 1-bit symbolization, which introduces the quantization noise. This may lead to a slight increase in the number of communication rounds required to reach a target accuracy, as shown in Figure 3 in the Appendix.\n\nHowever, we would like to emphasize that each communication round in FedSMU incurs a significantly lower communication cost compared to other methods. As a result, even with more communication rounds, the overall communication overhead remains substantially lower, as will be further discussed in our response to Question 2. We will also clarify this trade-off in the revised manuscript.\n\n> Question 1: A key concern is that quantizing local updates to one bit may cause the averaged gradient direction to deviate from the steepest descent, potentially increasing the number of communication rounds.\n\n**Response:**\n\nWe appreciate the reviewer’s concern, and acknowledge that quantizing local updates to 1-bit may introduce deviation from the averaged gradient direction, which in turn may result in a larger number of communication rounds needed for convergence, as shown in Figure 3 in the Appendix.\n\nHowever, we would like to emphasize that our FedSMU significantly reduces the communication cost per communication round, which originates from its highly compressed 1-bit update representation. As a result, even if more communication rounds are needed for convergence, the total communication cost remains substantially lower than that of existing methods, as will be further discussed in our response to Question 2.\n\n> Question 2: The paper claims that FedSMU reduces communication costs by transmitting only one bit per local update for each parameter. However, if this leads to more communication rounds, additional overhead—such as more frequent broadcasts—may offset the savings. In Figure 2, does the reported communication cost include both upload and download bits, or only the upload?\n\n**Response:**\n\nWe would like to thank the reviewer for pointing out this issue, which can be addressed as follows.\n\nFirst, we would like to clarify that the communication cost reported in Figure 2 includes only the uplink (client-to-server) transmission for all the comparison algorithms, which will be explicitly stated in the revised manuscript for more clarity.\n\nSecond, the total communication cost per round can be generally estimated by using the following expression, as presented in Literature [B1]:\n\n$Total Communication = Uplink Communication +\\alpha ·Downlink Communication, \\alpha \\in [0,1] $.\n\nIn practice, due to the factors such as system asymmetry, caching constraints and protocol limitations, the uplink speed is often significantly lower than the downlink speed, as discussed in Literature [B2]. Consequently, many communication-efficient FL studies [B3, B4] focus merely on minimizing the uplink cost alone.\n\nTo have a more comprehensive evaluation of our FedSMU, we followed the setting in [B2] and set $\\alpha = 0.1$, and depicted Figure B1 (https://anonymous.4open.science/r/A-7823/F.pdf) to compare the total communication cost including both the upload and download bits. It shows that FedSMU remains communication-efficient even when accounting for the downlink overhead, with a significantly lower total cost compared to the other baselines at comparable accuracy levels.\n\n[B1] Condat, Laurent, Ivan Agarský, and Peter Richtárik. \"Provably doubly accelerated federated learning: The first theoretically successful combination of local training and communication compression,\" *arXiv preprint arXiv:2210.13277* (2022).\n\n[B2] Condat, Laurent, et al. \"Tamuna: Doubly accelerated federated learning with local training, compression, and partial participation,\" *International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS,* 2023.\n\n[B3] Li, Xiaoyun, and Ping Li. \"Analysis of error feedback in federated non-convex optimization with biased compression: Fast convergence and partial participation,\" *International Conference on Machine Learning.* PMLR, 2023.\n\n[B4] Richtárik, Peter, Igor Sokolov, and Ilyas Fatkhullin. \"EF21: A new, simpler, theoretically better, and practically faster error feedback,\" *Advances in Neural Information Processing Systems* 34 (2021): 4384-4396.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "No.", "relation_to_broader_scientific_literature": "This work may contribute to more efficient communication in federated learning.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The motivation and core design of the proposed algorithm are clearly articulated. However, it is unclear what trade-offs are made to achieve communication savings.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. A key concern is that quantizing local updates to one bit may cause the averaged gradient direction to deviate from the steepest descent, potentially increasing the number of communication rounds.\n\n2. The paper claims that FedSMU reduces communication costs by transmitting only one bit per local update for each parameter. However, if this leads to more communication rounds, additional overhead—such as more frequent broadcasts—may offset the savings. In Figure 2, does the reported communication cost include both upload and download bits, or only the upload?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V7xjP9vmWn", "reviewer": "Reviewer_Hhir", "summary": "This paper proposes FedSMU, a federated learning algorithm that improves communication efficiency and generalization. It symbolizes model updates (using sign-based compression) to reduce communication overhead and mitigate data heterogeneity. Inspired by the Lion optimizer, FedSMU splits local updates and global execution, improving generalization. The performance of FedSMU is validated through both theoretical analysis and empirical experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the comments.\n> Impact of component of FedSMU\n\nAs described in Appendix F.9, we had implemented two variants, Fed-LocalLion and Fed-GlobalLion, to evaluate the isolated impact of Lion on the client and server sides. To further assess the necessity of key components in FedSMU, we conduct here a systematic ablation study by removing specific elements and comparing each pruned variant against the full algorithm. In particular, we examine the effect of excluding the following component: 1) server-side weight decay regularization ($\\gamma_2$), 2) client-side gradient sliding average ($\\beta_1$), 3) client-side gradient symbolization.\n\nThe experimental results in Table C1 (https://anonymous.4open.science/r/A-7823/F.pdf) demonstrate that the client-side sliding average plays the most crucial role in achieving a stable and effective training. Additionally, the model update symbolization mechanism itself proves to be more effective than using the full-precision updates. This validates our motivation of proposing FedSMU that symbolization balances contributions of the heterogeneous clients by suppressing some extreme update magnitudes, which thus enhances aggregation stability and leads to better generalization.\n\nMoreover, we observe that directly applying the Lion optimizer on either the client or server side yields suboptimal performance, which indicates that our FedSMU effectively harnesses the benefits of Lion, enhancing the generalization while compressing the communication load.\n> Assumption 4.3\n\nWe acknowledge that this is a slightly stronger assumption, ensuring that both the compressed targets and momentum terms (the moving averages of gradients) in our theoretical analysis are bounded. Moreover, this assumption has been adopted in other federated optimization and sign-based compression studies [C1, C2]. Specifically, in [C1], the authors demonstrate the convergence of distributed SIGNSGD with momentum under their Assumption 4, which is also the bounded gradient assumption.\n\n[C1] Tao Sun, et al. ``Momentum ensures convergence of SIGNSGD under weaker assumptions,'' ICML 2023.\n\n[C2] Sashank Reddi, et al. ``Adaptive federated optimization,'' ICLR 2020.\n> Evaluations on larger models\n\nWe appreciate this suggestion. Following it, we conduct additional experiments using a larger model (Vision Transformer Small (ViT-S) [C3]) on CIFAR-100, with a Dirichlet distribution of 0.25 to simulate non-IID settings.\n\nDue to time constraints, we focus on comparison with the four strong baselines: FedAvg, SCAFFOLD, FedEF-TopK, and FedLion, which demonstrated higher accuracies in our original Table 2.\n\nAs shown in Table A3 (https://anonymous.4open.science/r/A-7823/F.pdf), FedSMU consistently achieves a superior performance with the larger-scale model, further confirming its effectiveness and scalability in more FL scenarios.\n\n[C3] Dosovitskiy, et al. ``An image is worth 16x16 words: Transformers for image recognition at scale,’’ *arXiv:2010.11929* (2020).\n> Adaptive FL as related work\n\nWe thank the reviewer for this valuable suggestion, and will incorporate a discussion of Adaptive FL in the Related Work section, as follows.\n\n“Several adaptive algorithms [C4, C5] dynamically adjust global learning rates based on the divergence between local and global models, thereby enhancing generalization performance in federated settings.”\n\n[C4] Reddi, Sashank, et al. \"Adaptive federated optimization,\" ICLR 2020.\n\n[C5] Tong, et al. \"Effective federated adaptive gradient methods with non-iid decentralized data,\" *arXiv:2009.06557* (2020).\n> Theoretical justification of generalization\n\nWe acknowledge that the current manuscript does not include a theoretical generalization analysis, as our primary focus is on the design of a communication-efficient optimization strategy, for which we provided a convergence analysis. That is, under the general non-convex settings, FedSMU achieves a convergence rate of $\\mathcal{O}(\\frac{1}{\\sqrt{T}})$, where $T$ is the total number of communication rounds. This theoretical result matches with the convergence rates of existing FL algorithms.\n\nThough a theoretical generalization analysis is not included, FedSMU’s generalization ability is thoroughly validated through extensive experiments. Key designs contributing to this include: 1) the symbolization of local updates, which normalizes client contributions and enhances Magnitude Uniformity (MU), helping alleviate data heterogeneity; and 2) the split-Lion optimizer, which decouples updates into local and global components, combining stability and efficiency to further improve generalization.\n> Typos\n\nThank you for your correction, and we will correct it to \"reducing\".\n> Notations\n\nThank you for your suggestion. We will incorporate the missing symbol definitions into Table 1. Specifically, we will clarify that $\\beta_1$ and $\\beta_2$ denote the momentum coefficients, while $\\gamma_2$ represents the weight decay factor.", "claims_and_evidence": "Most claims in the paper are well-supported by theoretical analysis and empirical experiments.", "methods_and_evaluation_criteria": "The proposed method is appropriate for the federated learning setting.", "theoretical_claims": "I did not verify the correctness of the proofs for the theoretical claims. However, Assumption 4.3 is not a weak assumption and may not always hold in practice.", "experimental_designs_or_analyses": "The experimental design is mostly sound, with comparisons across multiple FL baselines. However, a potential limitation is that the experiments primarily focus on small- to medium-scale models (LeNet, ResNet18), leaving uncertainty about FedSMU’s performance on larger architectures.", "relation_to_broader_scientific_literature": "FedSMU combines communication efficiency and generalization improvements in a unified FL framework, which previous works have addressed separately.", "essential_references_not_discussed": "The paper covers most essential references, but it would be better to include Adaptive Federated Learning in the Related Works section for better context.", "other_strengths_and_weaknesses": "The generalization benefits of FedSMU lack theoretical justification.", "other_comments_or_suggestions": "1. In Line 164, there is a typo: \"ruducing\" should be \"reducing\".\n2. Some notations are not explained in the paper, e.g., $\\beta_1$ and $\\beta_2$ are momentum coefficients, and $\\gamma_2$ is the weight decay factor.", "questions_for_authors": "FedSMU splits the standard Lion optimizer into local updates and global execution. How does each component impact FedSMU’s performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aAmO5RtUTF", "reviewer": "Reviewer_AyCK", "summary": "The paper considers for federated learning in the heterogeneous regime, with compression and partial participation. A new algorithm called FedSMU is proposed.\n\n## update after rebuttal\nI think the paper deserves to be accepted and I am confident that the authors will make the recommended changes to make the paper even better.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the comments. In the following, we have provided our detailed responses to these comments.\n\n> Experiments on CIFAR-10\n\nWe thank the reviewer for this observation. Our experimental results indicate that FedSMU achieves a notably better performance on more complex datasets, such as CIFAR-100 and Tiny-ImageNet, while the improvement on CIFAR-10 is relatively marginal. We attribute this to the lower data complexity and heterogeneity in CIFAR-10, which restrains the potential benefit of FedSMU’s core designs, particularly its ability in handling client update imbalance and data heterogeneity. As shown in Table 3 of the original manuscript, FedSMU still outperforms all the baselines on Tiny-ImageNet, highlighting that its advantages become more pronounced in challenging federated scenarios. \n\n> There are however works combining local steps, with control variates to mitigate client drift due to heterogeneity, and compression.\n\nWe thank the reviewer for the insightful comment and for suggesting the relevant literature. We agree that there are existing works that address both the data heterogeneity and communication efficiency, particularly in the convex setting. In response, we will revise our original statement as follows.\n\n“Some existing approaches in FL address either data heterogeneity to improve generalization or communication overhead through update compression. However, it remains a challenge to jointly addressing both of them, especially under the non-convex settings. For example, CompressedScaffnew [D1] and TAMUNA [D2] combine control variates (used to mitigate client drift due to heterogeneity) with the model compression. However, these methods rely on permutation-based compression schemes, which are relatively complex and less flexible. LoCoDL [D3] extends this line of work by supporting a broader class of compressors and demonstrating a convergence acceleration in convex problems, but it focuses exclusively on the convex setting. Our work complements these efforts by proposing a unified approach that simultaneously improves generalization and reduces communication overhead in the more challenging non-convex regime.”\n\n> In the non-convex setting, it is less clear how to mitigate client drift. \n\nIn our work, we propose a new mechanism, symbolization of client updates, which normalizes the magnitude of each model parameter before transmission. This design aims to balance the contribution from each client, reducing the influence of extreme updates caused by local data heterogeneity, and thereby mitigating the model drift during aggregation. The effectiveness of this approach is empirically demonstrated in Figure 1. Moreover, relevant studies, such as FedComLoc [D4] and Sparse-ProxSkip [D5], make attempts to explore client drift under the non-convex objectives. However, FedComLoc’s performance may degrade under the compressed communication due to its reliance on communication variables, while Sparse-ProxSkip assumes the full client participation, which may not always be feasible in real-world FL scenarios.\n\n> Works on distributed low-communication language models\n\nWe thank the reviewer for suggesting this work. We will incorporate DiLoCo [D6] into the Related Work section. This method adopts AdamW for local updates and Nesterov momentum globally, demonstrating a strong empirical performance. While our current focus is on symbolized updates within a communication-efficient FL framework, integrating advanced optimization techniques, such as those in [D6] (especially adaptive optimizers like AdamW), could further enhance the model performance. We consider this as a promising direction for our future work.\n\n[D1] Condat et al. \"Provably doubly accelerated federated learning: The first theoretically successful combination of local training and communication compression,\" arXiv preprint *arXiv:2210.13277* (2022).\n\n[D2] Condat et al. \"Tamuna: Doubly accelerated federated learning with local training, compression, and partial participation,\" *International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS*, 2023.\n\n[D3] Condat et al. \"Locodl: Communication-efficient distributed learning with local training and compression,\" ICLR 2025.\n\n[D4] Yi et al. “FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models,” preprint *arXiv:2403.09904*, 2024.\n\n[D5] Meinhardt et al. “Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning,” preprint *arXiv:2405.20623*, 2024.\n\n[D6] Douillard et al. \"DiLoCo: Distributed Low-Communication Training of Language Models,\" *arXiv:2311.08105*, 2023.", "claims_and_evidence": "The focus in on the non-convex setting, for deep learning. The functions are supposed smooth and with bounded gradients (Assumption 4.3).\n\nThe experiments and comparisons are satisfying and show the merits of FedSMU. The improvement is not large, however, and in some cases FedSMU is even worse than competitors. For instance it is worse than EF21 and FedAMS on CIFAR-10.\n\nThere is a theoretical analysis in Section 4, which is rare for papers focused on empirical performance in deep learning settings. This deserves to be saluted.", "methods_and_evaluation_criteria": "The evaluation is good.", "theoretical_claims": "I did not check the details of the theoretical results but the statements make sense and seem correct.", "experimental_designs_or_analyses": "The experiments seem valid to me.", "relation_to_broader_scientific_literature": "The literature is correctly reviewed. I suggest below some references to add.", "essential_references_not_discussed": "You write that \"Current approaches in FL often prioritize either mitigating data heterogeneity to enhance generalization or compress-\ning model updates to alleviate communication, rather than addressing both challenges concurrently.\"\nThere are however works combining local steps, with control variates to mitigate client drift due to heterogeneity, and compression. \n* CompressedScaffnew in Condat et al. “Provably Doubly Accelerated Federated Learning: The First Theoretically Successful Combination of Local Training and Compressed Communication,” preprint arXiv:2210.13277, 2022. \n* TAMUNA, which extends CompressedScaffnew to partial participation, in Condat et al. “TAMUNA: Doubly Accelerated Federated Learning with Local Training, Compression, and Partial Participation,” preprint arXiv:2302.09832, 2023.\n* LoCoDL, which uses arbitrary unbiased compression, unlike CompressedScaffnew and TAMUNA that use specific sparsification, in Condat et al. “LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression,” ICLR 2025.\n\nThese methods achieve acceleration in the convex setting. In the non-convex setting, it is less clear how to mitigate client drift. This has been studied in \n* Yi et al. “FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models,” preprint arXiv:2403.09904, 2024.\n* Meinhardt et al. “Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning,” preprint arXiv:2405.20623, 2024.\n\nThere is the important paper Douillard et al. \"DiLoCo: Distributed Low-Communication Training of Language Models,\" arXiv:2311.08105, 2023. Instead of SGD local steps, it uses AdamW for the inner iterations, with good empirical performance.", "other_strengths_and_weaknesses": "The paper is well written.", "other_comments_or_suggestions": "No", "questions_for_authors": "I don't have questions. The paper is good overall.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers are in agreement that this paper makes a valuable contribution and should be accepted. We encourage the authors to carefully address the reviewers’ comments when preparing the camera-ready revision.
Decision: Accept (poster)