Low Precision Local Training is Enough for Federated Learning
Authors: Zhiwei Li
Yiqiu LI
Binbin Lin
Zhongming Jin
WEIZHONG ZHANG
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Low Precision Training
Primary area: optimization_for_deep_networks

Abstract: Federated Learning (FL) is a prevalent machine learning paradigm designed to address challenges posed by heterogeneous client data while preserving data privacy.
     Unlike distributed training, it typically orchestrates resource-constrained edge devices to communicate via a low-bandwidth communication network with a central server.  This urges the development of more computation and communication efficient training algorithms. In this paper, we propose an efficient FL paradigm,  where the local  models in the clients  are trained with  low-precision operations and communicated  with the server in low precision format, while only the model aggregation in the server is performed with  high-precision computation. We surprisingly find that   high precision models can be recovered from the  low precision local models with proper aggregation in the server. 
     In this way, both the workload in the client-side and the communication cost can be significantly reduced.   We theoretically show that our proposed  paradigm can converge to the optimal solution as the training goes on, which demonstrates that low precision local training is enough for FL.  Our paradigm can be integrated with existing FL algorithms flexibly. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. Notably, the models trained by our method with the precision as low as 8 bits are  comparable  to those from the  full precision training. As a by-product, we show that low precision local training can relieve the over-fitting issue in local training, which under heterogeneous client data  can cause the client models drift further away from each other and lead to the failure in  model aggregation. Code is released at https://github.com/digbangbang/LPT-FL.

Low Precision Local Training is Enough for Federated Learning
Zhiwei Li Yiqiu Li Binbin Lin Zhongming Jin
Weizhong Zhang lit
Fudan University *Zhejiang University AAlibaba Cloud Computing 4Fullong Inc.
(zwli23, yiqiuli22) @m. fudan. edu en binbinlinézju. edu. cn boangming.jinzmaalll jii.mmalliaaaa-iiiii....iiaiiale com
weizhongzhanggfudan edu, on
Abstract
Federated Learning (FL) is prevalent machine learning paradigm designed address challenges posed by heterogeneous client data while preserving data privacy. Unlike distributed training, it typically orchestrates devices to communicate via a low- now-bandwidth communication network with a central server. This urges the development of more computation and communication efficient training algorithms. In this paper, we propose an efficient FL paradigm, where the local models in the clients trained with low-precision operations and communicated with the server in low precision format, while only the model aggregation in the server is performed with high-precision computation. We surprisingly find that high precision models can be recovered from the low precision local models with proper aggregation in the server. In this way, both the workload in the client-side and the communication cost can be significantly reduced. We theoretically show that our proposed paradigm can converge the optimal solution the training goes on, which demonstrates that low precision local training is enough for FL. Our paradigm can be integrated with existing FL algorithms flexibly. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. Notably, the models trained by our method with the precision as low as 8 bits are comparable to those from the full precision training. As by-product, we show that low precision local training can relieve the over-fitting issue in local training, which under heterogeneous client data can cause the client models drift further away from each other and lead to the failure in model aggregation. Code is released at https:: / tighhu......... com/ wwwd digbangbang/LPTT Lappann//L/LLLLL
cnource-constraingg edge
Introduction
popular privacy preserving machine learning paradigm to collaboratively learn a global model over the decentralized data. In FL paradigm, the clients are responsible for local training and only have access their private datasets, while the server plays an essential role in aggregating the clients' updates into global model. Unlike large-scaled distributed training, FL typically orchestrates resource-constree via a low- www-bandwith communication network with central server. This urges the development of more computation and communication efficient optimization algorithms. The most prevalent approach in FL is developed based local-SGD (26), which is referred to FedAvg. In each communication round, the clients individually train their local models for multiple steps and then send them to the server for aggregation. It can be expected that if longer local training process one uses, the greater communication cost saving one can achieve. However, long time local training
Federated learning (FL) B 15| 22 36
edge devices to communicate
Equul contribution 'Coresponding author: weizzhnnzzhangffuaan edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

could cause the local models drift further away from each other and degrades the aggregated global model's performance or even make the training diverge, especially when the data on the clients are heterogeneous. Therefore, in order to prolong local training processes in FL., extensive efforts have been made in the recent years. For example, the studies [
21 22 modify the local training 16
process by imposing regularization on the client models to enforce them not to drift away from the focuses on refining the global model in the server aggregation process. These methods typically require a large proxy dataset on use it to align the outputs of the global model with that of the client ensemble by knowledge distillation. Others develop handcrafted aggregation rules to reweight the updates based on the statistics of updates or performance on proxy data [31 35 38) or further tune the global model with proxy data in every communication round [5] 24 Although promising experimental results have been reported in the literature, it is still unclear that whether there exists more concise and effective FI. paradigm, which can reduce both the workload in the client-side and the communication cost.
previous global model. Another line of research [5] 6 244
31
35
38)
the server. Some of them [5] 6 24]
In this paper, propose a concise and efficient federated learning paradigm, where the local models in the clients are trained with low precision operations and communicated with the server in low precision format, while only the server-side information integration maintains high-precision computation to ensure the accuracy. Our basic idea is inspired from the Kolmogorov's 's law [10], that is, the sample average converge almost surely to the expected value although the samples always contain noise. Therefore, in the server side, we perform the simple moving average on the received low precision models from the clients recover high precision global model. In this way, both the workload in the client-side and the communication cost can be significantly reduced. We theoretically proved that our proposed paradigm integrated with FedAVG can converge to the optimal solution as the training goes on, which indicates that low precision local training is enough for federated learning. We extend our method various existing FL method show its flexibility. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. Notably, the models trained by our method with the precision as low as 8 bits are comparable to those from the full precision federated learning. Compared with some efficient FL designs, our method can achieve significant savings in training memory overhead, and what is more attractive is that our accuracy performance is even better. Our method exhibits another appealing feature in relieving the over- fitting issue in local training. To be precise, in the local training steps, the models can be easily trained to over-fit the local training data as the local dataset is always insufficient. Under heterogeneous client data, it would further cause the client models drift further away from each other and lead the failure in model aggregation. The experimental results show that our approach can effectively relieve the over-fitting issue since the local training is performed with low precision computation and the expressiveness of the local model is restricted.
Our main contributions are as follows:
We propose an efficient federated learning paradigm that performs low precision computation during local training, saving computational overhead and communication costs, while being able to restore accuracy through high- precision aggregation on the server side.
We theoretically proved that the efficient federated learning algorithm we proposed can achieve convergence rated of (1/TT) under certain assumptions for non-iid situations.
Since the expressiveness of the local model is restricted due to the low precision local training. our approach can relieve the over-fitting issue, which would cause the client models drift further away from each other and lead to the failure in model aggregation when the local dataset are heterogeneous.
The extensive experimental results demonstrate the effectiveness of our approach. Notably, the models trained by our method with the precision as low as bits are comparable to those from the full precision federated learning.
2 Related Work
Federated Learning. Fedrated Learning is first proposed by (26 to realize model training without
sharing client device data. Many works have continued to solve some challenges of FL such as
heterogeneity 16 22 25), privacy (2), communication efficiency [I 18 Also, some works
proposes new FL methods to alleviate data heterogeneity. The vanilla FL method was FedAvg (26).
2

---

FedProx [22] utilizes regularization term while Scaffold [6 sets a control variate to reduce the drift in local training. FedGen [42] and FedFTG [40] maintain generator, the former is used for local data augmentation, while the latter is used for fine-tuning the server.
Efficient Federated Learning. One challenge of FL. the limitation of low bandwidth and computing
resources of client devices. 4 20 assign each client a block mask, resulting in sparse local
models. [28] took the transmission speed into consideration and chose the same method [[2] for uploading, uploading compressed gradients. [ adopts boost training to client-side training
overhead. 28 only transmit the trained head to reduce transmission cost. [3 adopts the idea of
Network Architecture Searching, e.g., each client selects a sub-network. [4] maintains a series of streamlined models in the server, from which the client selects tiny model for training. In [9] the client selects a sub-model of the global model for training. In this paper, we address this issue by using low precision local training.
Preliminary
3.1 Federated Learning
Given N clients with their private datasets Dj: { Uk Uk. [Pel =1...., N,, the optimization objective of FL is always defined as follows:
N
min F(w)
k=1 Ph Fk(w),
where E I&d represents the model parameters, Fs is denoted to be the empirical risk function of client k, i.e., Fww EeED, Ba ((: <) with E(..-) being the loss function and pk Σ Pa denotes the proportion of data contained in client k.
[Py]
Fl. emphasizes data privacy protection and thus the server is not allowed to access these datasets Da directly in model training. The standard method to solve the above training problem of FL is FedAvg [26). which is developed based local SGD. It comprised by two steps, i.e., local training on the clients and model aggregation in the server side. The details are presented below.
In local training. the central server would first randomly select partial clients and broadeasts the latest global model w, to them. We denote the selected clients set as S, and let K = |S, the number of selected clients. Then the client with & E S, would initialize its local model to be w and then performs local training with E(> 1) iterations as follows:
w W
w/ T/4 V.. (w): < St
where w is the weights of the k-th client in step t, et is mini-batch of samples uniformly chosen from 2k. Ni is the step size.
In model aggregation, FedAvg updates the global model to be the weighted average of the received local models, i.e.,
Σ L Gt
Pk w W
W++E
where Ze Ekes, P& normalize the coefficients.
3.2
Block Floating Point Quantization
Fixed point quantization is a standard quantization technique. It uses stochastic rounding to round the numbers up or down random such that E E[Q(z)) I, where Q R IR is the quantization function defined as
((x)
clip 1 clip : 4, w.p.
w.p.
( ( 9
here clip(æ, a, b) = max(min(r, b), .). 5 2-F is the quantization gap represents the distance between successive representable fixed point numbers, 2W-F-1 1---1-2- 2- and | 22 represent the upper and lower limits of the representable numbers, respectively. W is the bit width
3

---

of quantized numbers, and F is the bit width of quantized numbers' fractional part. In order to improve the utilization efficiency of the bit width and better maintain numerical accuracy when data is unevenly distributed, we choose block floating point quantization. Given block of numbers X, it replaces = 2- in fixed point quantization to be å = 2 22 ((W-2-(XX)) where
( coip( X, -2 F-1 -F ,2 2 SW--F-1-
1). -1).
(5)
The shared exponent E(X) is usually set to be the largest exponent in X to avoid overflow 337
Method
In this section, we introduce our low precision federated learning paradigm. It is comprised of two modules,i.c., one the low precision local training to reduce the computation and communication cost, the other is the high precision aggregation with moving average to maintain the model accuracy.
4.1 Low Precision Local Training
In order to reduce the computation and communication cost, we apply block floating point quantization to all clients' device of local training. The simple version of low precision local training is to convert the local training step in Eqn. into
2++ Q( (
Note that the above version, we only quantize the updated parameters. We give this version just for the convenience of the theoretical analysis in Section In practice, quantize the gradient, the activation of each layer, the back bacckpropagation signals, and the momentum in SGD when SGD is adopted as the optimizer. The details are given in Algorithm
Algorithm Low Precision Local Training with All Numbers Quantized
Input: Quantization functions QA, QE, QG. Qu, Qw; Momentum coefficient p; L layers DNN (f1, f2...., fz); Loss function f.
1: ClientUpdatedd k, w
2: Get batch (Ix.je yk.s,) )) from Dj:
3: Forward Propagation:
4:
(at) Tk.s.
5: (at) = (A(((((tt)())) (1) E
6: Backward Propagation:
7:
(et)(2) = V (apprere((aat) (L)) musei)
(et))(-1)= QE( ( (1)))()) ()) VI e [L, L]
8:
Diatjur
(1) = Qa @fillatt) ( (w (00) (e)) A E [],
9:
10: Low Precision SGD Update:
(o/4, )(()) & Qux
(gt ()), Vl E [1, L]
12:
(wo+)) )(0) +- Qw (( m (vft1 ()), A E [1,
13:
Return: wht1
4.2
High Precision Aggregation
Although low precision training can reduce communication and training overhead, it would lead to degradation training accuracy. Inspired from the Kolmogorov's law [IO], that the sample average can converge almost surely to the expected value although the samples always contain noise, we try to recover high- precision solution from the low-precision local model with full precision aggregation process. It is implemented with the following two steps:
Calculate the weighted average of the local models partially reconvert the precision, i.e.,
W++E f
Σ
Pk
w
(7)
gt

---

Since in most cases clients' data is non-iid and quantization causes error, federated learning is harder to converge, however maintaining moving average in the server can significantly alleviate the problem. Formally, we denote w, as the moving average stored in the server, then after aggregation, we update w, as follow
We f Q(Aw(EEwtt (1--A)w A)we)
where À is a coefficient controlling the influence of current weight. This procedure can further compensate the accuracy degradation due to the low precision local training.
In the next round of local training, the quantized model Q(w,) will be distributed to the clients intilize the local models. Our pseudocode Algorithm 2 depicts the process of low precision local training the client device and high precision aggregation on the server. In Algorithm 2 === and F En 2, represents the set of global synchronization steps.
Algorithm/ 2 Federated Learning with Low Precision Local Training
1: Initialize: Wo, Wp f Wo
2: for t = 0, 0.1..... T do 3: if = 0 (mod E) then
4: Sclect K clients from [N] to be S,
5: w/ + Q(w,), S.
6:
end if
7:
for k E St do
8:
w +- Q(w T V V end for
D Client update
9:
10: ift+l 1 E Ig then
Pic w 11: Wi+1 t ExES, qd
> Server update
12: Wi+1 t- Awe + +1 - A)We+1
13: end if
14:
end for
15: Return: Wy
5 Theoretical Analysis
In this section, we give the detailed theoretical results for our low precision FL paradigm. We will first introduce the convergence analysis in the full participation case where all client devices participate (i.e., K N) and then we generalize the results of the analysis to scenarios that are more in line with reality. (i.e., K N). The results demonstrate that we will explore aggregation strategies represented by the aederateeAAveraging Algorithm (or FedAvg) and demonstrate that our proposed low precision FL. framework can converge to the global optimal solution at a rate of (11/T) for non-iid datasets based on strong convexity and smoothness assumptions.
5.1 Assumptions and Notations
We need to make necessary assumptions about the objective function on the clients Fk. k = ,,....N. Assumption is about the smoothness and strong convexity of the loss function and Assumption is about the boundness of the gradients. These assumptions are standard and widely adopted in the
related studies [4 29
39.
Assumption 1. F1, F2,..., EN are L-smooth and p-stroongy functions, which means that for all and w, the following inequalities hold:
L
Fv(v) < (w(w)
w) T [FF(W)
+ v - w| 2
L smooth)
(9)
Fk(v) 2 ( ( VVF((w + V w
2
(μ strong)
(10)
2
where 21-112 represents the square of two norms and k 1, 1,... N.
5

---

Assumption 2. Let Ex be sample that randomly and uniformly sampled from the local data of the k-th client device. For k 1.2.... .N and 0,1, the variance of stochastie gradients in each client device and the expectation of squared two norm of stochastic gradients is bounded:
(11)
(w $
(12)
Degree of Data Heterogeneity. Let F denote the global minimum of the objective function F, and let F represent the minimum of the local objective function F& specifie to the k-th client. We use the metric r 23 taking the form of
F= E PAPE
to measure the degree of heterogeneity of all clients' data distribution. When the data each client device is iid, as the number of samples increase, I' evidently tends towards zero. However, when faced with non-iid situation, I tends towards a positive constant and thus it can be used measure the degree of heterogeneity the data distribution each client device.
5.2 Convergence Analysis: Full Client Device Participation
First we analyze the convergence of the participation of all clients' device in this section. We integrate our low precision framework with FedAvg and train the model for iterations to obtain the Wr, and expect T to be divided by E so that wr is the weight after aggregation.
Theorem 1. Under the Assumptions 7 and
we set 4,
n.' 7 7= max E} and T 2(6-a))
When t satisfying 8² & ni G2, low precision FedAvg with full device participation satisfies:
K E(F(wr) F S T+N
2BB
+
1(7+1)
|[1
w v1
(13)
2
where
B 2= + +1+ + 12 + 16E (2vd8 +3)
1
N 2
+
ok
+ 6LP.
(14)
N2
k=1
5.3 Convergence Analysis: Partial Client Device Participation
Compared full participation situation, the partial participation situation is more in line with the reality. We need to make more assumption on how to choose S..
Assumption Assume S, contains a subset of K indices uniformly sampled from [N] without replacement. In addition, the data is balanced in the sence that p1 aggregation step of FedAvg performs WL+E + 1 Ekes, w Wi+E
PN 1 The
N.
Theorem 2. Under the Assumptions to we choose K. #:' max (86, E} - 1, n 204) and
B = ==2(vāō d
+
(2 do +3) + N² Σk= ok² +6T, C= N-K 8 F2
N-1 K dd When satisfying 8² < G2, low precision FedAvg with partial client device participation satisfies:
2(B+
W1 1(7+1)
B(F(wr)) F
w0
(15)
T+g P
2
Experiments
In this section, conduct extensive experiments verify the effectiveness of our methods in the following five aspects:
When integrated with FedAvg, the models trained by method with the low precision comparable to (if not better than) those from the full precision training. This would also verify our theoretical results [Section[6.]]
Our method effectively relieve the over-fitting issue in FL. See Section (Section 6.2

---

Our paradigm can be integrated with existing FL methods flexibly and preserve the perfor- mance even with low precision local training (Section 6.3)
Ablation studies on the effectiveness of moving average in model aggregation and the transferability over various neural networks (Section 6.4)
Comparison with other efficient FL techniques (Section 6.5
Remark 1. Similar with the existing low precision training studies [32y. We do not give the results on the running time to show the real acceleration. The reason is that achieve real acceleration, we need to implement our method integrated with the professional hardware. Moreover, such implementation standard for the professional hardware platforms.
Benchmark Datasets and Baseline. We conduct experiments over four commonly used datasets: FashionMnist [3], CIFARIO [1)), CIFARIO0 [19] and CINICIO (8] Four commonly used FL
methods: 1) FedAvg | 2) regularization-bsee strategy FedProx [22 3) data-dependent knowl-
edge distillation strategy ABAvg (35] 4) data-free knowledge distillation strategy FedFTG [40 and
FedGen 42) are adopted the baselines.
Configurations. We follow the configurations in the recent studies 226 27 for fair comparison.
To be precise, for FashionMNIST, 10, CINIC10 and CIFAR 100, we run 200 communication rounds with local epoch set to There are clients in total, and the participation ratio in each round is set to 40%. We use Dirichlet distribution to simulate non-iid data distribution and set a to 0.01, 0.04, and 0.16. The smaller a is, the more serious the data hetcrogeneity is. For the network choice, use ConvNet following with 3 layers, and the hidden dimension is set 128. The local learning rate is set to 10-3 with Adam optimizer [17. We report the last round global model's average performance evaluated using the test split of the datasets. For quantization method, adopt the Block FLoating Point Quantization with the number of bits used set to 6, and 32 (without quantization). Some of the other hyperparameter settings included in the Appendix C
6.1 Results on FedAvg
We demonstrate the superior performance of our Low Precision FL method with FedAvg by conduct- ing experiments over comprehensive datasets, various precision and heterogeneity values a.
Heterogeneity. As shown in Table 1 it is as expected that when the heterogeneity goes higher, that is, when decreases, the server performance worsens. Nevertheless, our proposed method can always maintain or improve the performance of the original case (bits 32, w/o. avg) when using quantizaiton bits with moving average, which empirically validates the effectiveness of our proposed method.
Quantization Bits. We conduct experiments on 3 quantization bits: 32, 8, 6 (shown in Table I Figure D. as we observe that in most cases bits enough to hold the performance of full-precision and when the used bits is 6, the server performance begins to decrease due to the low precision level.
Table 1: Results of our method integrated with FedAvg over various levels of heterogeneity and precision. The results with moving average demonstrate that method can match the performance of full-precision federated learning even with all numbers quantized down to 8 bits. The results in the bottom three rows indicates the without moving average, training with low precision would lead to performance degradation.
Averaging
Pres.
FMNIST
CIRARIO
CINCCI
CHRARICO
FMINST
CIEARJ0
CINICIO
CIFARIOC
FMNIST
CIPARIO
CINICIO
CIEARIOO
80.1 0.0.
48.12:2 27
1522002
538.2:11
50.5 2.2
341
50.6 0.6.1.0.1
72.4 2.7.
68.2
42.02:11
arorg
18.3
618-001
0244445
88.
0.0
0.
41.000
150
00.
3.
7..0
0.
72.2 2.0
2..02:14
200:1:12
19
46:414
1474421
4.
u/ ong
6
41.6 4.8
2082008
18.11.1
17
28.8 & L3
6.76
$3.02 1.3
44.6 8 L.]
88.8 1.4
9.02 2.222.2
79.54.2.7
11.1422
1.5-661
039
33
1013 0.5
71.9 000--111
57.0 1.3
3911 6.4
6.2 Quantization Relieves Overfitting
We present the averaged local training loss and the global testing loss over training in Figure can be seen that method can effectively reduce testing losses on the server. The commonality
7

---

0.30
0.50
0.15
0.00
0.
0.20
Degi
Dege
Oeggn
Orpn B00
b4 arg
0.90
100 arg
0:10
8 org
apocs
spect
3.00
3.50
Orig
2.50
2.00
2.50
3.30
4.50
1.50
4.20
1.00
0.90
1.90
2.00
4.50
125
150
200
166
190 200
100
150
200
159
200
(a) FMNIST
(b) CIFARIO
(c) CINICIO
(d) CIFARI00
Figure I: Accuracy and loss of FedAvg with full precision (origin), our method with precsion levels of bit and 6 bit. We set a = 0.01 on all the four datasets. Our method exhibits an effective reduction in fluctuation variance and improves the stability of training. The reason is that compared with the full precision training, our low precision local training can prevent the client models to drift further away from each other and overfit the local datasets, making the aggregation stable.
with the original method is that when each round of communication starts retraining, the training loss of the client will be greatly increased due to the heterogeneity of the data Dx. Subsequentl, due to the highly imbalanced local data categories, the model quickly reached an overfitting state. It can be seen that the beginning of each training round, under our method, the customer's s training loss will not exceed the original training loss. At the same time, with some training steps, our training loss remains above the original training loss, which means our method can alleviate the overfitting problem of local training.
1.0
0.075
0.025
0:00
110
7200
3.0
3.5
2.0
1.8
0.20
0.10
1100
1200
3.0
011
1100
1200
85
4.5
4.0
5.0
0.50
11:00
1200
6.0
Glbel frem Cee
0.8
2.5
1.2
3.0
1.5
0.8
2.5
1.0
1.5
38
25
2.0
1.0
0.4
1.5
1250 Tralingg (FMNIST)
1030
1150
1250
1030
1150
1250
1250
1250
1150
1250
1350
1250
1150
1250
1250
Traiiing
Trahiing forations (CINIC10)
Training (CIFAR100)
8.30 w wag (Durr)) Cibert Trnin lone
Ferbberg Cilwett Truin can
R (Cung) Glabel Taxtl ooe
Faetan Glateel Twxtt coe.
Figure 2: Effectiveness of our method in relieving the over-fitting issue. We present the averaged local training loss and the global test loss over training. We select part of the training procedure (iteration 1000 to 1400) for display, and enlarge part of the picture in the upper right corner to show more details. We can observe that the local training loss of FedAvg (full precision) is siginificantly lower than our method, however its global test loss is much higher than us and fluctuates dramatically.
6.3 Results on Other FL Methods
The four FL methods we used are each representative. ABAvg and FedFTG similar to FedAvg in local training, but the former only performs weight adjustment the server side, while the latter uses knowledge distillation fine-tune the server. FedProx and FedGen are similar FedAvg in the server side, but the former only has regularization constraints on local training, while the latter uses the generator for regularization adjustment in local training. As is demonstrated Table 2 we can see that, regardless of the FL method chosen, our low precision FL algorithm has significant improvement in prediction accuracy compared to the original FL method, especially in dataset CIFAR1O, 10, CINIC10 and CIFAR100.
8

---

Table 2:
The results of our method when integrated with existing FL methods ABAvg. FedProx, FedGen and FedFTG. Our method can improve the effectiveness of FL with certain training cost, and can be applied in both regularization and distillation. The complete result is given in Table ( It shows that integrated with our low precision training technique, these existing methods with the precision of bits can match (or even better than) the performance with full precision.
0.55
Accpging
AllAng
mg
8.6
50.7 8.6
51.1 111-11
14.5
145-18
01:005
100-4004
835
15.14-1
15.54-0.
36.14-3.
12:1435
50.04-277
36.3
703-00
40.34-100
71.34-13
73-34-10
56-14-16
36P--07
125-+03
Terthro
32
811
38
108207
183.4:3.
$3.93.223
46.02 20
80.1 1.0.4
18.0:1007
452207
15.5 18
3:7--81
1:11
66.21-07
003-1-14
17:4--13
07
08:11:0 1:03
700-163
54-11:18
03:102
ony
9.6
62.0:111 11
0.
03
8774002
51.3+0033
25
903
717400
06.7417
120-003
38
50.6
8.8
461-78
10
13
366-+117
36.-2-43
47.34:1..
283504
00..000.4
7384-10
579-5003
41.6--066
6.4 Ablation Study
Moving Average. We conduct ablation study to validate the effectiveness of the moving average for the server, from Table [ the average operation has a significant effect on the server performance over various settings. It is consistent of our expectation that averaging can alleviate the error introduced by quantization and the error by heterogeneity.
Architecture Generalization. We use another network structure, MLP. to verify that our method can be applied to other networks. The MLP was to 3 layers, with the hidden dimension of each layer being 128, and experiments were conducted on FashionMNIST with a of 0.01 and 0.04. From Table 3 our method can still maintain the accuracy, which also demonstrates its generalization ability on other network structures.
Table 3: To display our method generalize to other architectures, the same experiments were also performed on the MLP, using the FashionMNIST. setting a = 0.01 and a = 0.04. Through comparison, it is found that the accuracy under MLP is slightly lower than that of ConvNet. However, our method can still maintain or improve the accuracy on MLP.
lnt w aeg
bit v aeg
32 bil or any
32 bit wio aug
Method
MLP
ConvNet
ML.P
ConvNet
ConvNet
ML.P
ConvNet
FedAvg
73.4
80.1 0.7
72.5
78.0 1.4
74.1
80.7 + 0.6
62.1
79.5 + 2.7
ABAvg
75.1
80.7 1 0.4
69.7 1.9
78.0 1 0.7
76.8 0.7
79.5 1:
58.6
1: 6.9
78.4
2.8
0.01
FedPrux
73.6
78.1
74.7
82.9
0.9
70.9
3.8
FedGen
73.3 1.1
79.9
7.4
78.5 1.3
78.9 1.5
72.2 3.88
80.2
FedFTG
75.1:::0.8
82.0 1.1
74.8 +: 1.2
80.0 +: 0.4
77.1 1: 1.9
82.0 1: 0.7
+: 1.3 63.1
80.6
2.2
FedAvg
09.
83.8 0.2
0.1 0.3
0.2
0.0
78.8
83.1
ABAvg
79.2
88.4 1.2
78.6 166+-00 0.1
86.1
79.1
88.4 0.8
77.2
4- 3.7
83.5 + 2.3
a - 0.04
FedProx
81.7:40.4
86.5
1.2
79.9
79.9
0.2
81.9
8..9
80.8
2.4
86.3
1.5
FedGen
7..0
0.7
0.7
81.9 1.3
79.5
83.6
0.6
83.7 0.8
FedFTG
83.5
87.7 + 0.2
79.2 0.6
81.9
0.3
84.0 0.6
87.6 + 0.6
76.5 4.9
84.6 1.7
6.5
Versus Efficient FL Method
In order to demonstrate that our method can effectively reduce computational costs and memory consumption, we compared it with two personalized model compression FL algorithms. HeteroFL [9] and Split-Mix [14) can both achieve the purpose of reducing training overhead by using smaller models for local training. We use the ConvNet with the same hidden dimension conduct experi- ments, using CIFAR 10 in the process, re-tested a split ratio of 1/16 achieve the same compression ratio as HeteroFL. Figure [3a the vertical dotted line indicates the compression ratio of HeteroFL, more details are in the Appendix E Compared with methods that directly compress model parameters, our method can achieve higher accuracy with lower memory consumption. Because quantization does not bring about the deletion of model parameters, there will be no aggregation problems when the model compression rate too high.
0.01. SplitMix uses split ratio of 1/8 by default. We

---

1926
Method
Type
Acc
Communication cost
Train cost
10
Sp (1/8)
HoteroFL
HeteroF1.
(1/4,1/8)
(1/2)
36.4 + 0.5
19.6
7.3
83.7
Spllmmx (1/66)
(1)
41.0 2.3
39.1
178.4
Bit
SplitMix
(1/2)
40.4++3.6
19.6
89.2
38.9 11/4,1/8)
7.3
33.4
bit
53.3
9.8
51.3
FedAVG
bit
50
Memory (MB)
100
150
200
750
bit
44.7 1.0
6.1
32.1
(a) Training Versus Acc.
(b) Results on CTEARIO with a 0.01.
Figure 3: Results on CIFARIO with 0.01. () denotes the percentage of models on the clients. We use the number of weights, activation, and gradients of local training to approximate the training cost (MB / client) and communication cost (MB / round).
Conclusion
In this paper, we propose efficient FL paradigm, where the local models in the clients trained with low-precision operations and communicated with the server in low precision format, while only the model aggregation in the server is performed with high-precision computation. We theoretically show that our proposed paradigm can converge to the optimal solution as the training goes on, which demonstrates that low precision local training is enough for FL.. Our paradigm be integrated with existing FL algorithms flexibly. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. As by-product, we show that low precision local training can relieve the over-fitting issue in local training.
8 Acknowledgements
Authors acknowledge the support in part by The National Nature Science Foundation of China grant No: 62472097, The National Nature Science Foundation of China grant No: 62273303, Yongjiang Talent Introduction Programme grant No: 2022A-240-G.
10

---

References
[1] Durmus Alp Emre Acar. Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based dynamic regularization. In International Conference on Learning Representations (ICLR 2021), 2021.
[2] Naman Agarwal, Ananda Theertha Suresh, Felix Xinnan X Yu, Sanjiv Kumar, and Brendan McMahan. epsgd: Communication-effie and differentially-prinn -ariivate distributed sgd. Advances in Neural Information Processing Systems, 31, 2018.
[3] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In A. Talwalkar, V. Smith, and M. Zaharia, editors, Proceed- ings of Machine Learning and Systems, volume 1, pages 374-388, 2019.
[4] Daoyuan Chen, Liuyi Yao, Dawei Gao, Bolin Ding, and Yaliang Li. Efficient personalized federated learning via sparse model-adaptation. In International Conference on Machine Learning, pages 5234-5256. PMLR, 2023.
[5] Hong-You Chen and Wii-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. In International Conference on Learning Representations (ICLR 2021), 2021.
[6] Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heterogeneous ensemble knowledge transfer for training large models in federated learning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI) Main Track, 2022.
[7] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pages 2089-2099. PMLR, 2021.
[8] Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic- 10 is not imagenet or c0f arXiv preprint 18x0v:1810.03505, 2018
[9] Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication effi- cient federated learning for heterogeneous clients. In International Conference on Learning Representations (ICLR 2021), 2021.
[10] Rick Durrett. Probability: theory and examples, volume 49, Cambridge university press, 2019.
[11] Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated learning. arXiv: Learn- ing.arXiv: Learning, Feb 2019.
[12] Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mchrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference Artificial Intelligence and Statistics, pages 2350-2358. PMLR, 2021.
[13] Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. Fedboost: A communication- efficient algorithm for federated learning. In International Conference on Machine Learning. pages 9973-3983. PMLR, 2020.
[14] Junyuan Hong. Haotao Wang, Zhangyang Wang, and Jiayu Zhou. Efficient split-mix federated learning for on-demand and in-situ customization. In International Conference on Learning Representations (ICLR 2022), 2022.
[15] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings. al. Advances and open problems in federated learning. Foundations and trends machine learning, 14(1-2):1-210, 2021.

---

[16] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Scaffold stochastic controlled averaging for federated learning. In International Conference On Machine Learning, Vol 119, volume 119. JMIR-JOURNAL. MACHINE LEARNING RESEARCH, 2020.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- lional Conference on Learning Representations (ICLR 2015), 2015.
[18] Jakub Konečný, H.Brendan McMahan, FelixX. Yu, Peter Richtárik, AnandaTheertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv: Learning.a Xivv Learning, Oct 2016.
[19] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[20] Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. Fedmask: Joint computa- tion and communication-effccien personalized federated learning via heterogeneous masking. In Pruceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pages 42-55, 2021.
[21] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10713-10722, 2021.
[22] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
[23] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations (ICLR 2020), 2020.
[24] Tao Lin, Lingjin Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion federated learning. Advances in Neural Information Processing Systems, 33:2351-2388 2020.
[25] Yishay Mansour, Mehryar Mohri, Jae Ro, and AnandaThherrtha Suresh. Three approaches for personalization with applications to federated learning. Cornell University arXiv, Cornell University arXiv,, Feb 2020.
[26] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas coamunication-efffciee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[27] Renjie Pi, Weizhong Zhang, Yueqi Xie, Jiahui Gao, Xiaoyu Wang, Sunghun Kim, and Qifeng Chen. Dynafed: Tackling client data heterogeneity with global dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12177-12186, 2023.
[28) Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication--ffiieeee federated learning with sketching. In International Conference on Machine Learning, pages 8253-8265. PMLR, 2020.
[29] Sebastian U Stich. Local sgd converges fast and communicates little. In International Conference on Learning Representations (ICLR 2019), 2019.
[30] Sehastian U Stich, Jean- -aptitte Cordonnier, and Martin Jaggi. Sparsified sgd with memory. Advances in neural information processing systems, 31, 2018.
[31] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 1177611-762,,,
12

---

[32] Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers, Advances in neural information processing systems, 31, 2018.
[33] Tianchun Wang, Wei Cheng, Dongsheng Luo, Wenchao Yu, Jingchao Ni, Liang Tong, Haifeng Chen, and Xiang Zhang. Personalized federated learning via heterogeneous modular networks. In 2022 IEEE International Conference on Data Mining (ICDM), pages 1197-1202. IEEE, 2022.
[34] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprini arXiv: 777.077777 2017.
[35] Jianhang Xiao, Chunhui Du, Zijing Duan, and Wei Guo. A novel server-side aggregation strategy for federated learning in non-iid situations. In 202/ 20th international symposium on parallel and distributed computing (ISPDC), pages 17 24. IEEE, 2021.
[36] Yueqi Xie, Weizhong Zhang, Renjie Pi, Fangzhao Wu, Qifeng Chen, Xing Xie, and Sunghun Kim. Robust federated learning against both data heterogeneity and poisoning attack via aggregation optimization. arXiv preprint arXiv:2211.05554. 2022.
[37] Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Chris De Sa. Swalp: Stochastic weight averaging in low precision training. In International Conference on Machine Learning, pages 7015-7024. PMLR, 2019.
[38] Yousef Yeganeh, Azade Farshad, Nassir Navab, and Shadi Albarqouni. Inverse distance aggregation for federated learning with non-iid data. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning: Second MICCAI Workshop, DART 2020, and First MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4-8, 2020, Proceedings 2, pages 150-159. Springer, 2020.
[39] Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 5693-5700, 2019.
[40] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling- Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10174-10183, 2022.
[41] Yuchen Zhang, Martin J Wainwright, and John C Duchi. Communication-efiiienn algorithms for statistical optimization. Advances in neural information processing systems, 25, 2012.
[42] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free free knowledge distillation for heteroge- neous federated learning. International conference on machine learning, pages 12878-1222 12889. PMLR, 2021.
13

---

This appendix can be divided into several parts. To be precise:
Section A gives the detailed proof for Theorem
Section gives the detailed proof for Theorem 2 Compared to Theorem Theorem 2 assumes that only subset of client devices participate in the training, which is more line with real-world scenarios.
Cintooucni four FL methods' methods'(ABAre) (ABAvg, FedProx, FedGen, FedFTG) hyperparameters
setting our experiments.
Section D provides detailed experiment results of four FL methods FedProx,
FedGen, FedGG),
presents the comparsion between our low precision FL method and two efficient FL method in training overhead.
Section E
Section
gives Limitation of this paper.
Section gives Broader Impacts of this paper.
A Proofs of Theorem
We analyze our low-precision federated learning algorithm in the setting of full device participation in this section. For the sake of convenience in proving. let's take the value of рı to be N. disregard the moving average, assume that quantization is only applied to model parameters and quantization gap is always 5. Let Zg be set composed of the sequence number of global aggregation steps, i.e., Ig [nE|n 1,2, Suppose the model weights are updated as follows:
v r+1 = =(w Te V.
v (+1 2k-1
if 1 $ IE.
w
i Iz.
where w is an d-dimensional vector, which represents the weight of the k-th client device in step Fs represents the objective function of the k-th device. N is the total number of clients device participating in the training, and T is the total number of steps trained, & represents a batch of data randomly sampled from the k-th client device's local dataset Dj in step i, (w: gl) is stochastic gradient, 7) represents learning rate, Q represents the quantization function, and its formula has already been given in
Notations. To further simplify our proof process, we introduce the following additional definitions:
N
1
1
1 N
Wo Wo,
Σ VF.(
V (w
N
N
N
W, represents the weight of the global model in step t. Obviously, we can deduce Eg. g and
V1+1 = Wr 1/:gt-
Lemma
scalar and can be written as + b8, satisfying -5 T is the quantization gap, represents the distance between successive representable fixed-poin numbers, then we have:
W.P. 1 sign( (w) 8 T Q(w) - w (ign( w, w.p. sign 7 )
where Q(-) is quantization function. And we always can find a suitable i that satisfies |uا| S |w].
Proof. The quantization function is:
Q(w) = [cipp(6] JJ u); clip(0] #1,4 u),
w.p.
10
5
w.p.
12
8
20
5
14

---

When i > 0:
Q(w)
(-ù,
W.p.
12
6i0,
w.p. t
When i <
0 - w w.p.
((w) - w
W, w.p.
5
So have:
w.p. 1 sign ()) 10
Q(w)
sign(w) (8) w.p. sign 2) 8
Next we prove that we always can find a suitable T that satisfies |لا| S |w/. When w 2 ó,we can choose w satisfying
2 O, then we can have |ا] S |w]. When w -8, we can choose a w satisfying Vi S 0, then we also can have |0| S [w] When - < w < 6, we can choose w satisfying w. then we also can have | S [w]. In summary, we always can find a suitable that satisfies | S |w/-
Discussion. In the lemma 2 below, we analyze the convergence of the algorithm under the condition that 82 S. "/ G2, where G² given in assumption 2 and represents the constraint the expected value of the squared two-norm of the stochastic gradient. The right side of the inequality indicates the change in the iterative parameters during the gradient descent process, that is, the size of the gradient multiplied by the size of the learning rate. If the magnitude of this value is already less than the quantization precision, it will cause the gradient descent to fail, and the model parameters will not continue to change. In this case, it makes no sense to continue optimization, so analyzing convergence under this condition is practical.
Lemma 2. assume that TJ S AT. based on assumption and when satisfying 82 S G2 we have:
d
E B|[ w"
2(Vd8
1+
2 )n + (1-11/ BI w 6L
2
N
w 1122 -En, lg 1
w:
N
Proof. When t + 1 & Zg, we can easily derive that W V2+1. So we have:
[e++1= - ww
|i++1
w- 2
Q Qw T)e V (w), w"
N
1=1
W:
ng.
w'
Q(w n. W..
(w T/ V FF (w E(-)
N k=
W. T.
11
V (w VF +
A
1
N
+ +2(W: T/ w'
(Q(w) - T. V.. -(w FF VF.((.t.)))) (l))))
N
A
15

---

( Prepresents inner product operation. Due to BQ(x) z, according to the Law of iterated expecta- tions, EEA3 O. Next we aim bound A2. E.Az 1s:
N (Q(w) - TA V. (w2, (t) - - +)))
N
[W T V.L (w 7h Vw w.w.w.))))) 1111
N
E(Q(W? - VVV. (w.", + (w - 17c Vk, (w/" (2 )),
Q( Tie V (w/,,,/)) - (w ./s V krz , +),)))
N
E Q(w ne F st)) (w - m FR.
N2
In the above equation, we used the law of iterated expectations. The quantization function is:
Qww) clip(8] J. u); clip((] (..,p(111,.,10
w.p.
w
5
5
w.p. 1 -
w
+
w 5
5
w is quantified vector, therefore, its p-th component (w!)), = c 8, с is an integer. We rewrite
nn F (w/, ct), 66 - W. Where b is an integer and |قū| S |l]] So that we can have:
(Q Ti V.VF FrVFF (w st)) (w/ T/e Vi (w) <)))p
sign(w)) + w
w.p. sign(18)a
w
w.p.
The expectation after the square is:
B(Q(w,
n F (w), (w² - 1/1 V (w 11
= . sign(w) 10 w
Bigg(@W) + x² sign w
5
< 20 + 25 |iم
2 [Nr V F (w.
2 + 25
(1 V Fx w/ ( [,,]
The expectation of [Q(w - 11 Fx
(w: 7 V.. <t 1|2 is:
BlQ 1/.
(w: m VVF +
[(E||Q(W? - V. - 113
-
S 2 V Fx (
= 28E 17 V
S 2Váōm E V
+ 2m /
d8 + 1)m E
Fi
(w..et)) D
In the above inequality, we use the incquality ||-I1: $ Vd 11-122 is the dimension of the vector. According to assumption 2) we can get:
E|| Wr+ w 112 s 2(V00 + GG= Ellw, 7:8: 2|
We next aim bound A
W 77:: w²11
16

---

w 1e.B: w -) + 7 1/
W: ne.:
w" 1/ + n
g: 8ell2 +2 T1gr w", gr g1
B1
B2
Ra
Obviously EB3
0 so we can get:
E w. Th. w
El Bll. n.B: -w- w"l1 + En, En, 2|g: gell2
We next aim bound B₁:
w/ Th:B: w
l/w; w w'11 + 7)t || |g
2TJr (wi - w'' N
= -l/w, w" w'11 2/2 + 7)
2 gr 12
277 N
Σ
(wr - w. Fw(
k=1
C2
N
1
21
( R w", (.F(w())
N
To bound C₁ Cz Cz. we need to derive several inequalities using the properties F2, FNN First of all, for k = 1,2, NN, Fg is L-smooth function, get:
F2,
Fs" (x) Fa (w,) (VFx(w?), w/) + _ || w w111
Specifically, when w aVFF(W can get:
Fk" S (w + - - - -V
when a = +. we can get:
(W) Vn s -
(16)
According to the above equation, we can get:
N
N S Σ
N
1g = m
Σ P&
Pk vFl(W) 2Lm Σ (x F)
(17)
In the inequality above, have utilized incquality 16 and the property that 11.112 is convex function.
For A: = 1,2, ......,,, N, Fl is p-strong function, so we can get:
(w, w", ((((W()) < ((F((w)) Fk(w")) 1 w/ w1
(18)
By the inequality:
2 ( 6) S >/lal12 - (r > 0).
(19)
we can get:
22 -2(W: - w/, (w((w()))
-| w11l nel VFl 1|
(20)
7/
w.
w (w FC).
(21)
7/
In the inequality above, we have utilized inequality 220 bound C₁. Cg, C3 to bound B1:
16 again. Now, we can use inequality (16).
7/g:: w
17

---

l| w'11 + m 1o 2n (we- - w", w'
N
1/ w"1l 112 " Bell
2n N
1
Σ
W/ we
t. -w", V(x(()))
kml
S l|
w w1 + 2Le Σ Pk (Fa(W))) - Fb
1
2m
Ww we VFK((w)) + 2T
W h -w", V (w)))
N
N
k=1
k=1
=llw;
w011
mTmp 2
Σ pa (Fe((( - Fk*")
1
Σ
We willi
N
1 N
N
2T
(Fl) Fk(w")) - -7114
1
Σ
1/
w 112
N
N
1=1
N
< <ll,,- w w'11
+ 4Lm 2 Σ ph (Fe - Fi )
+
1
Σ
|
w
N
k=1
N
1
1
21j N
(Fa(w))) - Fa(w")) -Fa
(w
w*)
N
k=1
= (1 m//i]l W. - w0 +
1
Σ W/ will12
N
N
+ 4Lm 2 Σ (F Flr"
2m N
Σ (Fx(w.) - Fk(w")). k=1
D
We set 7/= (1 - 2L))). because 17e S 1L. nt S n
2m. Decompose D we can get:
N
D = =Lmm 2 Σ (F(FF(W)) - Fia) -2m
1
Σ
(F&(w:) - Fn(w"))
N
k=1
k=1
pk Fk" + +2 1 N
(ALm; 2 - 217() Σ (cF) + 4Lme 2
N Σ Fi(w")
N
vt
pk (w!!)) + 4LTx² 2 Σ PMFF*
22T 1:
N
Pk (F(( - F*) + (217 71)
Σ k=1
Pa - -FE"
N
Pa (F F*) + TT.
It should be noted that I
Σ -k) Σ - Fk and it measures the degree of
data heterogeneity between different client devices. We can bound the first term in D:
Pa(FL(w!)- F²) -F*)
Σ ( + Σ (F -F*)
18

---

Ph V FV:((:))) w/ - wee + F(we) p
Ph: VF -| /
wellot + F(W.) - F
7/
k=1
1
IV
(a -
w W. + F(we) FF.
2T}
In the above inequation, we use the inequations (16) So we can bound D and get:
19 and the convexity of Px(()).
D
Ph (F& F²) TL
1=1
N
S 7/t
Pk I Fx")
1
|w w, [011)=((F((w))) (F(w))) F*) 4Tm, 2r
2TTe
N
Ph (F& - FFF)
Ph:
w - -wel -11 7o (F(we) -F") + 4LM 2
217:
k=1
-oo L
pi. (F:(F((w)) F F
7 Σ
pa w w
2T k=1
Σ (w)) - F*) + ALm T
N Ph (F((F(W.))) - F*) + (1ORL + 1 +
=n(n L 1)
7t
Pk.
w
w, 1/
27 k=1
k=1
<6Lm 2 +
Σ pk ll well
When + 1 $ IE, we conclude that:
E w
w" 112 - 2(vd8 + +1m + (1 n.P)EE lw w 11
2
w - 1/2 6L r + Eni
lgg. gg112
N
When + IE, we have w = Q
E
W(+1
w
11
N
2
=
E
Q
x
w²
N
k=1
N
N
N
1
E
Q
N
N
N
V++
w"
1=1
k=1
N
E
Σ
Q(w - V.. w
N
N
N
k=1
k=1
N
SE
Q
N
Σ
V++
+
E
Q(w: V. {f)
19

---

N
N
2E
Q((w - - VVV.(( w
Eg
Eg has been given constraints in the text above that:
E2 S 2(V8 + G2 + - m ||w: w=11
2
will2 + 6T. Bnj ge gx 11
N
K=1
According to the law of iterated expectations. Eg is We next constrain Ej. According to our assumption before Lemma 2, we have:
E EE S Edd S EEd (w,, 22) dn
So when /. + E Læ we have:
d
E B|| - w w0 S 2(V06
+ 11
2 2) G1 + (1 - -nah)BBB w'1 GL
2
+ N
W/ w/ll Enle gg: B:
K=1
In summary, whether t + E IE or &
E Ix, we have:
d )n +(1 - (1-
E B|| - w01 $ 2(v68
+ +
(1-m/))B|ww w' + 6T,
2
2
w: w/ v1112 - -Bn/ l|g. 8:112
+ N K=1
Lemma 3. Based on assumption
we can gel:
gg
ge
1
S
1
k
N2
Proof.
E g:
g = E
VRL -
N
N
WF - VPL((
N2
N
ok 2
N2
Am
Lemma 4. If Ti. is non-inn ient-iicreasing and T). S 22/ F. (t 2 0).eeedd on assumption we can get:
N
1
E w
w 1
(2 +3).
N
20

---

Proof. Since FedAvg requires communication each E steps. Therefore, for any t 2 O, there exists to S t. such that to v E and Wto w for all Also, we use the fact that 77 is non-increasing and π _ 2 Es (t 2 0). then:
Wx w
lww -Wo - -w, w
B Bllw? wioll 11
B w w
Σ
B w
w + w w tw w/ -w
w
N
k=1
E
B|| -
N
Am
N
E
@wW - mm V.. (wh. 7hh (Wf. -
2
Q( Th FFL.(, (w² 17hh (W
N
E
m Fr
w
112)
E Σ
Inequalities
a
S
Σ || 2 and | EX 1 < E || are used in the above proof.
2
We have already proofed in lemma 2 that:
BllQ(ww Mh V ( ch)) - (w 1.5--VAAA < + 12
So have:
El w G
Theorem 1. Under the condition that assumptions 7 we choose k = 1/2 - max E} 1 and 1)+ # When t satisfying 82 I G² our low precision federated learning algorihm with full device participation satisfies:
EF(w.) - F -
<
K
2B
11( + 1((++1
W1 - -w"112)
2 2t+y
t t+g f
2
where
B 2=2(vd8 ++ d
N
G² 16 (3)
Σ ox 2 677.
Proof. Set A, = -E| w. 11 according to lemma we can have:
Aul (1 pm pm) n B.
21

---

Set 17:
11 11 and n S 27 Set B we can choose appropriate 3 and y so that th S min 1L
14..
B max Pp-1 (7 + 1) we will prove A S +. by mathematical induction.
When = obviously it holds. Assume the conclusion holds for some t, it follows that:
A+t $ (1 um)
+ 2 B
$ (1 3p V
JB
t+ t+ +
(1+
t+y v ++
B²B
Bp
(I 71+7)²
(1 (1+
(1 (1+>)²
2
t 8+
Then by the L-smoothness of F..).we have:
L D
EF( F" S
2 2t
We choose 3 = 2. y -max 8K, E} and denote k n²
L
then we have:
B
4B Bp - + (7 + S
8B
= m
Bp-1 1 (7 + 1) <
+ +
1
and
T. v
K (2B +
11 +1
EF(w)) F 1
A1)
2 +
1+7
2
B Proofs of Theorem 2
We analyze our low precision federated learning algorithm in the setting of partial device participation this section.
Updating scheme In real world application scenarios, constrained by communication efficiency and low straggler effect, FedAvg initiates by selecting a random subset S, which length is set to K, and subsequently carries out updates exclusively on this subset. The analysis becomes somewhat complex due the variability of S, every E steps. Nevertheless, this challenge. We envision that FedAvg initiates each epoch by engaging all devices and then relies solely on the parameters updated on subset of these devices to generate the parameters for the subsequent round. It is evident that this method of parameter update is equivalent to the original approach. Then the update of FedAvg with partial devices active can be described as: for all [N).
can employ a thought trick address
++1 = Qww Th. (WF:(w/.f)
vitu Q(E K (
ift + 1 S i S.
w 8+1
We use Es, (.) eliminate the error caused by S. Unless otherwise specified, the meanings and assumptions of the symbols used in this section are the same as in the previous section Lemma 5. Ift + E Ig, we have:
Es, (We+1) V1+1-
Proof. According to the selection method of S;, there is a function:
1. Pii E St) K
I, = 0, P(i E S. = N-K
22

---

According to the aggregation process, we have:
W++1 = Q( 1
Iv
K i=1
So we have:
N
1
Es. (W(+1) = Ess
I.vi+1)
K
i=1
1 N
N
K
P.vi
1+
K
N
(+1
=
Σ
N
1
[+1 VV+1-
1=1
1=1
1=1
Lemma 6. For 1 E IE we have:
N K 8
1s,11411 - wor1l S
E (2 + + dG
N 1 K
Proof. According the aggregation process, we have We+1 Q(* Eies, v(+1).
Es, [w++1 Vet1l22
Es,
Q
K
6-11
K
Σ SES,
K
1+1
V
FES.
YES.
2
2
Es,
Q(
Σ
1
1
Es,
Σ
I(i E Sil(1 vet1)
K
1+1
K2
iES,
iES.
F1
2(Q
Σ
v(+1
V - ve+1)
K
K
H+
K
iES:
iES,
ieS,
F/3
In the lemma 2 F2.
we have proven Fi S dG And obviously we have F3 0, we next aim bound
Es.
- Σt( T(i (v)(v(11-vv))
K2
1
P(i Si) S1) - vet1l12
K2
Pi.j E St)) (vē+1 Ve+1, v241 vet1)
K2
ifj
1
N
| - vetal12
Σ
KN(N KN(N-1) 1) (V/+1-Ve441 Ve41, v241)
K-1
KN
i=1
itj
N 1 11- N
-
K(N 1)
The last step in the above equation uses:
N
Iv - Ve41 11+
+
E(ve
Vesi, V ve+1) 0.
it)
23

---

Therefore:
N
N
We+1 - Ve-1112
V2+1 Weo + -eo -vet1l22 + Fi
K(N -1)
N
N
N
K
1 1it1-lool22 winll2
1
E
+
K(N 1)
N
N
N
K
(1-N)
1
8E ne & (2Vdo + 3) d
K(N 1)
N
We used EE |X - E < I |||| in the inequality above.
Theorem 2. Under the condition that assumptions
we choose K. 7 max (8r, E} Te 2((4g) and B ds +1+ 2) + 16E G (2V + 3) + N² ΣK of + 6 N-K N-1 R (2Vab + 3) + dG?. When satisfying 8² S n G² FedAvg with quantization and partial device participation satisfies:
B[F(w.)] F S
K pl7 p(7+1) 2(B+C)
Wi - 10---="' w(1l²)
7+
2
Proof. We have:
E W141 - w w011 = E || Wi+1 - Vi+1 + vi+1 1 w²11
E We+1 Ve+1l12 + IE || w 11 21E 21E(w:+1
V2+1, Vz+1 w")
G1
G2
We take the expectation for the above formula, G3 vanishes according to lemma 5 If + & Ig. according our parameter update settings, G1 vanishes. We use lemma bound G2:
3
E V w 112 E Wi+1 w 12 (1 Tpp) w" +2
Ift + 1 Ig. We use lemma 6jto bound G1 lemma 2 to bound G2, then we have:
E W| w 11 = EE [| 1 + E V141 w 11
< - 1,]) l| w l1 + v + C).
We use exactly the same method in Theorem and then have:
(2 +
+ u(7 +
B[F(w:)] F
| w" w01 112))
7+
1
2
C Hyperparameters Setting
We will introduce the four FL methods, ABAvg, FedProx. FedGen, FedFTG, and their hyperparame- ters setting in our experiments to prove method's effectiveness.
ABAvg 35 is a data-dependent distillation FL method, which uses validation dataset D. to reweight each collected devices. In our experiment, we set the test dataset as the validation dataset. For FashionMNIST, CIFARIO, 10, CIFAR100, we use the whole test dataset, and for CINIC10. we choose the 20% data of each label.
FedProx 22 an FL method with regularization when training locally. We set the FedProx proximal term μ 0.1 in our experiment.
24

---

FedGen 42 is a data-free distillation FL method, which maintain a generator in the server to generate hidden variables, and transfer the generator to the device side for data enhancement. We set the generator learning rate to 3e - 4, hidden dimension to 32. The generator's training epoch 5.
FedFTG [40] is data-free distillation FL method, which maintain a generator on the server for fine-tuning aggregated server parameters. We set the generator learning rate to le 2, the fine-tuning learning rate to 1e- 4. The totally fine-tune epoch is set to 10, inner epoch for generator is I, inner epoch for server parameter is 5.
All of our models are trained on s GeForce RTX 4090. We found only very small differences when evaluating on these other hardware platforms.
D Low Precision Training with Other FL Methods
We put the four algorithms used here together with Efficient FL. Judging from the Table 4 low- precision local training is sufficient in FL system. We put more emphasis on showing the case of a 0.01, because this case is more consistent with the actual situation of FL, that is, the data heterogeneity of edge devices. Combining four FL methods and four datasets, in the case of a = 0.01, Figure 4 we can find the fact that our algorithm can maintain or even surpass the accuracy of the original algorithm on 8 bit, and at the same time, it can do the same on 6 bit to maintain accuracy.
Table 4: Low precision FL ABAvg, FedProx, FedGen, FedFTG. Our method is universal, whether is regularized locally to constrain updates or finetuned on the server. Our method can maintain the effectiveness of various FL methods with certain training cost, and can be applied in both regularization and distillation.
Method Avaping
Proc
FMNIST
CIFARIO
CINICIO
CIEARIOC
FMNIST
CIFARIO
CINICIO
CHEARI0
EMINST
CIFARIO
CINICIO
SUL DLS
5L12144
4.44
22 0.3
185285
36 022
932
72.5 1.0
162.14
42.5 2:03
780 0.Y
0.
2h. 1-3
184 8.2
85.1 2.8
156208
22
220.8.1 BL
813
2.94
70.2
1.5
23.2 2.0.1
AB/ag
165:09
$14
2.0.5
12.7
2.4
06.1
305 1:3
23.5 1-8
7.15 06
1.
11. 133
89.1 1.0.8
03.5 177
162 22
15.3100
56.0 1.0 25
15.4+65
3.42 8
01
185=21
11
7
81.1+00
41.5+220
00.1
32
75. 25
64.0 4.2
25.4
160204
55.5
2.5
861=811
42.1 2:18
27
00.1 1.0
71.8 1.8
161111
68.32 78
1.0
67.0-2-022
1144
71.8 2
12.0244
29 2.1.
21.202
16.22:002
440:02
25.1 2..1
FedProx
32 kr
$2.9 0.0
02.92.0 0.5
42.4 1.0
18
36.200
81.12:0.55
71.522 0L
07.211
4..
72.6
17.0033
25.74
=00
500-23
0.
60:3---666
M1
51.1
15.1::3.. 3.4
4-.24-16
07
96.3
17.0211
16
99.1+-0.44
72.54-00 D.
160:07
45.5 2.
211:111 24
0.T
10
4.
12
0..
74.0 DLD
03
Pefrin
78.9
07.66 2:2
13.
161 0.8
16.9
29
1..
32Y
91. 1.0.
74.21.10
56.8
05
71.1-3
314-334
1B3 30
2310 4-1
1.1 61
99.-4-0.6
03.14-15
523-13 13
198-07
631-443
33.04-15 18
156661
0..
433
4..34-0.3
79.14-10
573
03
457466
520.1:11 1.1
02
22
NYD 0.22
M2200
18
1-033
73.7 DM
wran
50.0
0..
143202
16T
8-22 0.1.1
00L (L)
25.5 1.11001
71.4 1.1
267
1.4
22.3113 19
631664
796 1.11
37. 13
3F. 130
112185
891 1.8.8
63.0 1.1
185
217
15.2
219
170
23
73. 1:1
31.
21
25 123
03:002
KLL 1.9.
1.14
We also place the algorithms used here, and these pseudocodes are added to our methods. When we conduct experiments, we also follow the algorithm flow from Algorithm to Algorithm We found that although different FL methods will behave differently when the client updates or the server updates, our method can be easily applied them.
E
Efficient FL Detail
We list the detailed results of the Efficient FL used in experiments. We use the same network structure ConvNet and the CIFAR 10 split in 0.01 to conduct experiments. In HeteroFL, a, b. c, d. and e respectively represent 100%, 50%, 25%. 12.5%, and 6.25% channel proportions. In Table 5 for example, al-bl-c1 means that during local training, a, b. c, the three models account for 1:1:1. In SplitMix, a, b, c, d, e, have the same representation. 1/8 and 1/16 of SplitMix represent the minimum model channel ratio after segmentation. We conducted experiments
1/8 and 1/16 respectively.
We calculated the average memory consumption of a single device during each communication round, for each compression method, and also for our method. For memory consumption statistics, we
25

---

record model size, gradient storage. and intermediate activation storage. Table 5 to Table g contains all our experimental results comparing memory savings.
0.50
0.50
0.20
0.90
Origie wmn
Origin 1m
Origin bat or
Origen BR
0.10
opcci
spocn
spocn
spece
0.40
0.90
0.20
01.00
0.90
0.06
30
125
150
200
52
125
150
203
20
100
150
203
52
150
202
0.50
0.50
0.40
0.20
Driguum win
Deigie wono
Deigin
Origen
0.92
0.42
0.80
0.20
0.90
20
122
150
200
52
1E0
150
200
50
100
150
203
100
150
201
(a) FMNIST
(b) CIFARIO
(c) CINICIO
(d) CIFARIO0
Figure 4: Visualization of the accuracy of ABAvg, FedProx, FedGen, FedFTG FL methods when a = 0.01 on 4 datasets. The picture from top to bottom ABAvg, FedProx, FedGen, FedFTG.
Table 5: HeteroFL, al, bl, cl, d1, el represent the percentage of the number of model channels to the original number: 1, 0.5, 0.25, 0.125, 0.625, For example, al-bl-cl means that the and models exist in 1:1:1 ratio.
Model
Sparsity
Accuracy (%)
Memory (MB)
al al-bl
0.00
37.63 + 1.37
205.4
0.25
26.53 2.34
144.5
al-bl-cl
0.41
28.78 + 1.44
109.6
al-bl-ci-d1
0.53
34.45
87.8
al al-bl-cl-dl-el
0.61
25.92
73.1
bl
0.50
36.36
83.7
b1-el
0.62
37.84+1.56
61.8
bl-el-d1
0.70
21.78
48.6
bl-cl-di-el
0.76
26.35 26.35 1.60
40.1
cl
0.75
33.05 1 1.24
39,9
cl-dl
0.81
29.19
31.1
cl-dl-el
0.85
31.55 + 1.56
25.5
dl
0.87
19.61 1.35
22.3
dl-el
0.90
27.95 + 0.77
18.4
el
0.93
26.20 + 0.22
14.5
26

---

Table SplitMix 1/8. Set the smallest model 1/8 of the original model, which is the model. By setting the client's resource limit, for example, al-bl-cl, the client's s resource limit is 1, 1/2, 1/4, then 4, and 2 models can be trained parallel respectively.
Model
Sparsity
Accuracy (%)
Memory (MB)
al
0.00
41.04 2.31
178.4
al-bl
0.25
35.24 = 5.35
133.8
al-bl-el
0.41
32.84 2.50
104.1
al-bi-cl-dl
0.53
31.19 = 33.17
83.6
bl
0.50
40.36 33.66
89.2
bl-el
0.62
38.31 = 2.76
66.9
bl-cl-d1
0.70
30.25 = 4.48
52.0
cl
0.75
39.18 = 4.98
44.6
cl-dl
0.81
38.89 3.50
33,4
dl
0.87
41.80 2.40
22.3
Table 7: SplitMix 1/16. Set the smallest model to 1/16 of the original model, which is the model. By setting the client's resource limit, example, al-bl-c1, the client's resource limit is 1, 1/2, 1/4, then 16, 8, and e models can be trained in parallel respectively.
Model
Sparsity
Accuracy (%)
Memory (MB)
al al-b1
0.00
30.72 + 3.61
232.0
0.25
29.62 2.01
174.0
al-bl-cl
0.41
27.75 + 2.92
135.3
al-bl-cl-d1
0.53
32.08 4.14
108.8
al al-bl-cl-dl-el
0.61
27.09
89.9
0.50
33.73
116.0
bl-el
0.62
32.77
87.0
bl-ci-d1
0.70
25.60 2.81
67.7
bl-el-di-el
0.76
#6.93
54.4
cl
0.75
35.12 2.00
58.0
cl-d1
0.81
32.53 2.22
43.5
cl-dl-e1
0.85
+ 27.33 2.53
33.8
dl
0.87
37.33 + 2.27
29.0
dl-el
0.90
34.04 + 2.68
21.8
el
0.93
31.72 2.15
14.5
Table 8: Low precision FL. We tested the accuracy at different precisions and calculated the memory usage during local training.
Precision
Accuracy (%)
Memory (MB)
8 bit
53.32 + 2.78
51.3
6 bit
49.37 + 0.95
38.5
5 bit
44.67 + 1.04
32.1
Limitations
In fact, what we performed in the experiment was fake quantization, that is, simulated quantization, which means we only focused on the impact of quantization on accuracy. The actual quantization operation requires the cooperation of hardware, and the hardware GPU we used was GeForce RTX 4090, which could not directly perform low precision training, and could not reflect the acceleration effect of our method.
27

---

G
Broader Impacts
FL, while having a positive impact in areas such as data privacy protection and cross-domain collaboration, also faces some potential negative impacts and challenges.
First, in FL, attackers might attempt implant backdoors during the model training process. By introducing specific triggers into the training data, attackers can activate the backdoor after the model deployed, causing the model to output results that the attacker desires. This type of attack poses serious threat to the model's security and needs to be guarded against with strict security measures and auditing processes.
Second, attackers could disrupt the performance of the model by injecting incorrect information into the training data. This data poisoning occur at the client side during the model aggregation process. Data poisoning attacks may lead to the model behaving abnormally under certain conditions, affecting the model'ss reliability and accuracy.
Last, although FL is designed to protect user privacy by training models locally and sharing only model updates rather than raw data, this method is not foolproof. Attackers might infer sensitive information about training data by analyzing the model's update information, such as gradients or weight changes. This kind of privacy leakage can be mitigated with techniques like differential privacy, which may impact the model's performance.
Algorithm 3 Low Precision FL with ABAvg
Input: Quantization functions QA, QF. QG. Qw, Qw; Momentum coefficient p. L layers DNN //,,/,.... Jr.]: Loss function é; Validation dataset D,
1: Initialize: Wo, Wn Wo
2: for /. 0, 1,....
do
3: if 0 (mod E) then
4: Select K clients from [N] to be S,
5: wh Q(w)), S₁
6:
end if
7:
for k E S, do
8:
wha ClientUpdate((, k; w/:)
9:
end for
if E Ig then
11:
Get 0# from testing accuracy of each client A E Sy on D,
Pk Eies,
13: Wi+1 + ExES, PA P4
Wi+1 t Awe + (1-A)
15: end if
end for Return: Wy
17:
19: ClientUpdate((, k, w
20:
Get batch (IR.j. yk.j.) ) from Dj.
21: Forward Propagation:
22:
(at) Fb.s
23:
(a/)() = (a(fi(( (w)) (w)) Ve [1.
24:
Backward Propagation:
25:
(et)(T) = V (((t)en)((aa)))) ya
(ct))(-1) QE( ( (w/) (0), VI E [1
26:
((a)
(1) = Qo( (f(((at)) ( (wo (et) E []
O(w3j()
28:
Low Precision SGD Update:
29:
(ot+1)(0) + Qu (ar(p(z)) (0)) VI [1, L]
30:
(wf+))(0 + Qw (w(((f)()) m ()), Al [1.
31:
Return: w/
28

---

Algorithm Low Precision FL with FedProx
Input: Quantization functions QA, QE. QG. QM. Qw; Momentum coefficient pt layers DNN [2,,/...... Jf.): Loss function é; FedProx proximal term
Initialize: Wo, Wp Wo
2: for = 1. T do
3: if 0 (mod E) then
4: Select K clients from [N] to be
5: w + (w;,), S.
6:
end if
7:
for E S, do
8:
w +- ClientUpdate(1, ki we 1 l w/))
9:
end for
if IE then
11: Wr+1 + Exes, Pe w
90' W1+1 Aw + (1 W
13: end if
end for
15: Return: Wr
16:
17: ClientUpdate(t, k, wh [ w
18: FedProx loss function H. = + 1 11
1 Ii -wfl"
wf
19:
Get batch (Ik.je (k.,,) from Dj
20:
Forward Propagation:
21:
(at)(0) Ib.j.
22:
(at)") = (a( (wt) V/ [1,4]
23:
Backward Propagation:
24:
(et)(2) = V Vaaprass((aa (L))) Mk.ji)
(et) QE( (i (w)))
(ct) VI E [1.
Diatj
(g)))() = Qa ( @fillat) 2 (1o)) (00) (et) + u((w))((()))))))))))))) -
26:
27:
Low Precision SGD Update:
(ot + (Mr((())( + (g) ()), VI [1,
28:
29:
(wf+1) )(0 + Qw ((w))) m ( A4 E [1,A]]
30:
Return: wh
29

---

Algorithm Low Precision FL with FedGen
Input: Quantization functions QA. QF. Qg. QM, Qw: Momentum coefficient p; layers DNN
(J1 .ea...,SI.);; Loss function E; Generator parameter 0; p(y) uniformly initialized; local label
counter Ck-
1: Initialize: Wo, Wo Wo
2: for = 0, I.... T do 3: = 0 (mod E`) then
4: Select clients from [N] to be S
5: Update ck. k St
6:
w. + Q(w,), S₁ end if
7:
8:
for k & S. do
9:
w ClientUpdate((, k. w/: p(y), end for
10:
it + 1 E Ig then
12:
Wi+1 + ExES, Ph w 9r Wi+1 Awe + (1 AjWe+1
13:
14: Server updates p(y) based on {co]NES,
15: Generator updates 0 argmin Ey
Exes, Sr,(z, (w:+)
16: end if
17: end for
18: Return: WT
19:
20: ClientUpdate(t, k, " p(y), 0):
21: Get batch (Ik.je yk.s,) 5.) from Dr. of ~ p(y), at ~ Go(.
22:
Forward Propagation:
23:
(at) Tk.s.
24:
(a = (((ti(a (w ()), VI E [1
25:
Backward Propagation:
26:
(e))(2) = ((op)eer((aat)) Musii)
(et))(-1)- Qu( ( D, (w)))))
(e))00), v [1
Biafja
(1)) = Qo ofillat) ( (w (1)) (E/) + aelfelet (wtt Vl E [1,
O(wz)())
((wt)(2)
29:
Low Precision SGD Update:
(ot, )(1) + Qu (p(xt)))) + (g)) Vl E [1,
31:
1 + Qw (ww)))))) m (1) vl E [1,
32:
Return: wh+1
30

---

Algorithm 6 Low Precision FL with FedFTG
Input: Quantization functions QA. QE Qg. Qm, Qw; Momentum coefficient L layers DNN
(J1,2. f...................... Jt.): Loss function & Generator parameter 0; Server training iteration /; Inner training iteration of the generator and the server 1g1 la' Emd, Eelss Edis is the loss used in (40)
Initialize: Wo, Wy Wo
2: for t = 0, 1..., T
do
3: if't 0 (mod E) then
4: Select K clients from [N] to S,
5: w + k
6: end if
7:
for k E S, do
8:
w/+1 f ClientUpdate(t, k, w!)) end for
9:
if
1 Ig then
11: Wr+1 +- ServerUpdate(O,
W++1 + Awe + AjW++1
13: end if
end for
15:
Return: WT
16:
17: ClientUpdate((, k, wx):
18: Get batch (Ik.je yk.s.) from D3
19:
Forward Propagation:
20:
(at) Tkise
(a/) = (a( (1) V/ [1,4]
21:
22:
Backward Propagation:
23:
(et)(2) = V ((op)eer((aat)) (L))) Me..i)
(et) QE( ( (( (1)) (ct) VI e []
Diaffir-
(g)) = Qo @fillaa) ( (w (0)) (e)) E []
O(w7)
26:
Low Precision SGD Update:
( + (mr((()) (g.) ()), Vl E [1, L]
28:
(wf+1) +- Qw (ww)()) m (1) Al e [1
29:
Return wh+1
30:
31: ServerUpdate(O, (wi+ikkes,
32:
WH+1 = ENES, Pk wh
d 33: Compute (v) ExES,
!Paal Bez DDD [lm. EnES, n%
for 1, 1.,.....T do
35:
Get batch (Z, Y) from ~ No 1) and y (3)
36:
for jor 1.2....... Ig do
37:
Update @ according to min max B.w.((0.)))rp)))))) (mnd dctshats Adia
38:
end for
39:
for j 1, ......... Ia do
Update w according to min max
(mmd Notelels Adcoldiel
wesl
41:
end for end for Return W
43:
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The algorithms mentioned in the introduction and abstract can be found in Section of the article. Theoretical result can be founded in section [ The convergence
proof of algorithm is located in appendix And
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations can be found in appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer: [Yes]
Justification: Theoretical result and its assumptions can be founded in section S The proof of theoretical result located in appendix A and B
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have showed the detail hyperparameters configurations Section and
Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
33

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have added the code to the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the configuration of Section 6 we introduced the hyperparameters and model settings, most of which are adopted from previous work.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Section 6 specified that the average of the last five rounds used as the accuracy, and the standard deviation was calculated. When calculating the training loss of the clients, also introduced the use of 9 clients for separate experiments calculate the mean and standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The hardware we use is mentioned in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have reviewed the Code of Ethics carefully and we preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader Impacts can be founded in appendix G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our study does not carry these risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We used public data and models properly under the license and terms, which were also properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.paperssit
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We didn't 't release some new assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our experiments didn't 't include the crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our experiments didn't 't include the crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
37

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "itKqnXEfUf", "reviewer": "Reviewer_Zy16", "summary": "The paper proposes an efficient Federated Learning (FL) paradigm where local models are trained using low-precision operations and communicated with the central server in low precision format. The aggregation on the server, however, is performed with high-precision computation to ensure accuracy. The authors demonstrate that high-precision models can be recovered from low-precision local models with proper aggregation on the server side. This approach significantly reduces the computational load on client devices and the communication cost. The method is theoretically proven to converge to an optimal solution, even with non-IID data distributions, and extensive experiments show that models trained with low precision (as low as 8 bits) are comparable in performance to those trained with full precision.", "questions": "1. How can we determine the optimal precision level for local training without extensive hyperparameter tuning which is expensive in FL?\n2. Have you explored mixed precision strategy? E.g. using different quantization schema for gradients, activations etc.", "limitations": "The authors have discussed the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method reduces the computational and communication overhead for client devices, which is crucial for resource-constrained environments for large models. The paper also provides theoretical guarantees for convergence to the optimal solution, even with non-IID data distributions.\n2. The method is effective on the datasets and models in the experiments where low precision training has little to no impact on utility.", "weaknesses": "1. The integration of low precision training and high precision aggregation may add complexity to the implementation.The performance improvements are partly dependent on the hardware capabilities, such as the availability of processors supporting low precision operations.\n2. The experiments are limited. Only image datasets are considered. Evaluation on other types of data, like text or tabular, can strengthen the results. \n3. No integration with differential privacy or other privacy protection mechanisms. Federated learning itself is not private and it would be interesting to see what privacy mechanisms are suitable for low precision model updates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The integration of low precision training and high precision aggregation may add complexity to the implementation.** \n\n**A1:** Actually, in our method, the transformation between low and high-precision parameters is performed on the server, and the computation in the clients is standard for the clients supporting low precision operators. As we know, in FL, the server always has rich computation and memory resources and the main challenge in improving the training efficiency comes from the resource constrained clients and the communication cost. Therefore, the increased implementation complexity above would not prevent us from improving the overall training efficiency. \n\n**Q2: The performance improvements are partly dependent on the hardware capabilities, such as the availability of processors supporting low precision operations.** \n\n**A2:** Yes. But we would like to point out some promising  progress in this area. In recent years, deep learning applications urge the development of machine learning accelerators. Lots of accelerators supporting low precision operations have been announced, which make it easy to implement the low precision training/inference algorithms including our approach efficiently in practice.  For details, please refer to the survey [r1]. \n\n\n**Q3:  Only image datasets are considered. Evaluation on other types of data, like text or tabular, can strengthen the results.**\n\n**A3:** Thanks for your suggestion. Actually, we have conducted experiments on most of the datasets used in existing studies. Below, we give the results on the text dataset IMDB and Shakespeare. The results also confirm our conclusion on text data that low-precision local training is sufficient for Federated Learning. Following [r2], we approximate the communication cost by counting the size of the model transmitted from the client to the server in each round, and the training cost by counting the weights, activation, and gradients of local training.\n\n| Method      |IMDB Acc        | Communication cost (MB / round) | Training cost (MB / client) |\n|:----------  |:---------:      | :-----:                          | :-----:                      |\n| FedAVG      | 75.6           | 323.0                           | 3700.8                      |\n| FedAVG+Ours(8 bit)| 74.8           | 80.8                            | 925.2                       |\n\n| Method      |Shakespeare Acc | Communication cost (MB / round) | Training cost (MB / client) |\n|:----------  |:---------:      | :-----:                          | :-----:                      |\n| FedAVG      | 49.1           | 75.5                            | 2927.9                      |\n| FedAVG+Ours(8 bit)| 49.4           | 18.9                            | 731.9                       |\n\n\n**Q4: No integration with differential privacy or other privacy protection mechanisms. Federated Learning itself is not private and it would be interesting to see what privacy mechanisms are suitable for low precision model updates.**\n\n**A4:** We think that theoretically proving what privacy mechanisms are suitable for low precision model updates is too complicated to be done in the author response period due to the variety of privacy mechanisms. Below, we give the results of FL with differential privacy and low-precision local updates. It shows that our method works well with differential privacy. The reason could be that as our quantization process is stochastic, it can be approximately viewed as a differential privacy protection mechanism and therefore they are compatible with each other. \n\n|Method      | Gaussian Epsilon | Acc  |\n|:-          |:-:                |:-    |\n| FedAVG     | 30               | 45.5 |\n| FedAVG+Ours(8 bit)| 30               | 53.9 |\n| FedAVG     | 10               | 43.6 |\n| FedAVG+Ours(8 bit)| 10               | 53.4 |\n| FedAVG     | 5                | 40.8 |\n| FedAVG+Ours(8 bit)| 5                | 53.0 |\n| FedAVG     | 1                | 38.4 |\n| FedAVG+Ours(8 bit)| 1                | 45.8 |\n\nMoreover, we agree with you that theoretically FL is not private. We give the results with standard FL in the manuscript because FL is still one of the main machine learning paradigms to address the challenges on preserving data privacy. The reason could be that as indicated in Figure 4 of [29] recovering the data with details from the received local models is highly nontrivial. \n\n\n**Q5: How can we determine the optimal precision level for local training without extensive hyperparameter tuning, which is expensive in FL?**\n\n**A5:** Our empirical results show that 8-bit precision level works well on all the tasks. In practice, one can tune the precision level during training, e.g., choose 2 sets of clients with different precision levels in some rounds and choose proper precision level  based on the performance of the two aggregated models.\n\n**Q6: Have you explored mixed precision strategy? E.g. using different quantization schema for gradients, activations etc.**\n\n**A6:** Thanks for your suggestion. We performed local mixed precision training on CIFAR10 with $\\alpha=0.01$. We use the triple (x, x, x) to represent the number of bits are used in  quantizing weights, activations, and gradients, respectively. The  results are as follows:\n\n| Mixed precision| Acc      |\n| :--------------|:---------|\n| (32,32,32)     | 52.7±0.6 |\n| (8,8,8)        | 53.3±0.5 |\n| (6,8,8)        | 53.1±0.5 |\n| (8,6,8)        | 50.9±1.0 |\n| (8,8,6)        | 53.2±0.5 |\n| (8,6,4)        | 50.4±1.0 |\n| (6,6,4)        | 50.1±0.7 |\n\nThe experimental results indicate that our method also supports mixed-precision. Results with more mixed precision training methods will be included in the revision if accepted. \n\n    \n\n[r1] Reuther, et al. AI and ML Accelerator Survey and Trends. IEEE HPEC 2022. \n\n[r2] Chen D, et al. Efficient personalized federated learning via sparse model-adaptation. ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "drFSfGb6ae", "reviewer": "Reviewer_5cLQ", "summary": "The paper studies an FL system with data heterogeneity, a topic has been extensively studied in the past few years. The idea is to perform local training with lower precision through applying block floating point quantization. The idea per se is not new, but proving that the convergence can be achieved using low precision local training is an interesting contribution.", "questions": "Please also check my questions in the weakness section.", "limitations": "Appendix F clarifies that the fake quantization is applied in the experiments. I really appreciate learning this information, as one of my question was how the quantization is implemented in PyTorch (as PyTorch only supports int8 inference out of the box). \nRegarding the broader impact, the discussion is about FL in general. The question is if performing lower precision training at clients improves or worsen these privacy and security aspects (e.g. if data poisoning cab be done easier, as the local updates are low precision).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and is easy to follow. The idea is also interesting but it is not necessarily new. The most important contribution is the theoretical proof for convergence. The evaluation results in Section 6 validate the theoretical results.", "weaknesses": "The paper mainly focuses on data heterogeneity in FL systems. What about resource heterogeneity? It would be important to have a discussion (or better some experimental results) on how the proposed solution perform in such setting. Should we use the same quantization level for all clients, or can we adjust the precision according to the resource availability? Also, the current state of the art of quantization in conjunction with Federated Learning is also missing, e.g.:\n\n- FedQNN[1] uses quantized training in FL.\n\n- CoCoFL[2] uses a combination of quantization and freezing for heterogeneous resources in FL.\n\nHow does these SOTA techniques perform compared with the proposed solutions?\n\n[1] Y. Ji and L. Chen, \"FedQNN: A Computation–Communication-Efficient Federated Learning Framework for IoT With Low-Bitwidth Neural Network Quantization,\" in IEEE Internet of Things Journal.\n  \n[2] Kilian Pfeiffer, et al. \"CoCoFL: Communication-and Computation-Aware Federated Learning via Partial NN Freezing and Quantization.\" Transactions on Machine Learning Research., 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:  What about resource heterogeneity? It would be important to have a discussion (or better some experimental results) on how the proposed solution perform in such setting. Should we use the same quantization level for all clients, or can we adjust the precision according to the resource availability?**\n\n**A1:** Thanks for your suggestion.  We give the results in the setting of  HeteroFL [9]. That is, we set three types of resource devices: 1, 1/2, and 1/4, which respectively represent the devices that can train the entire network, 1/2 network, and 1/4 network. We use (x, x, x)  to represent the corresponding bits in three type of devices in training. We let $\\alpha=0.16$. The experimental results are as follows:\n\n| Precision   | Acc  |\n|:----------  |:---- |\n| (32,32,32)  | 55.6 |\n| (8,8,8)     | 55.4 |\n| (8,8,6)     | 54.7 |\n| (8,6,8)     | 52.6 |\n| (6,8,8)     | 50.4 |\n| (8,6,4)     | 52.8 |\n| (4,6,8)     | 43.7 |\n\nThe results above show that our low-precision training mechanism works well in resource heterogeneity applications. They also indicate that when the sum of bit widths used across the three sets of devices is fixed, devices with poorer computational resources should be allocated a smaller number of bit widths. \n\n\n**Q2: How does these SOTA techniques FedQNN and CoCoFL perform compared with the proposed approach?**\n\n**A2:** Thanks for your suggestion. Since the code of FedQNN has not been released,  we give the comparison results with CoCoFL. We follow the setting of CoCoFL and conduct experiments on on Shakespeare and CIFAR10. Following [r3] we calculate the communication cost by counting the size of the model transmitted from the client to the server in each round, and the training cost by counting the weights, activation, and gradients of local training. \n\n| Method      |Shakespeare Acc       | Communication cost (MB / round) | Training cost (MB / client) |\n|:----------  |:---------:            | :-----:                          | :-----:                      |\n| FedAVG      | 49.1                 | 75.5                            | 2927.9                      |\n| CoCoFL      | 49.3                 | 50.3                            | 1951.9                      |\n| FedAVG+Ours(8 bit)| 49.4                 | 18.9                            | 731.9                       |\n\n| Method      |CIFAR10 Acc       | Communication cost (MB / round) | Training cost (MB / client) |\n|:----------  |:---------:        | :-----:                          | :-----:                      |\n| FedAVG      | 84.3             | 21.5                            | 167.7                       |\n| CoCoFL      | 82.0             | 14.3                            | 111.8                       |\n| FedAVG+Ours(8 bit)| 83.1             | 5.4                             | 41.9                        |\n\nThe results above show that  our method has a significant reduction in overhead, because our local training is completely low-precision, and the model transmitted to the server is also low-precision. CoCoFL still trains the unfrozen parameters with full precision, and the transmission is also full-precision, so it will have a relatively larger overhead.\n\n**Q3: Appendix F clarifies that the fake quantization is applied in the experiments. I really appreciate learning this information, as one of my question was how the quantization is implemented in PyTorch.**\n\n**A3:** We adopt the widely used simulation method in the studies [40, r4, r5] of low-precision training, which is approved  by the community as its result is always consistent with that on the real ML accelerators. We have submitted our code in the supplementary materials, which includes the specific implementation of fake quantization. Our approach to quantization involves implementing the quantization of weights, activation values, and gradients during the training process. We have three quantization operations: the first is to implement the quantization of activations after every model module, the second is to quantize the gradients of the parameters after the loss is generated and the gradients are backpropagated, the third is to quantize the weights after the optimizer is updated. The specific implementation process can be referred to in our code.\n\n\n**Q4: If performing lower precision training at clients improves or worsen these privacy and security aspects.**\n\n**A4:** We tested different levels of label flip attack on FedAVG and ABAVG [38] by using MNIST, and the results are as follows. ABAVG maintains a validation dataset on the server side and adjusts the weight of the client aggregation according to the client's accuracy, so it can be used for adversarial attacks. We let $\\alpha=0.1$. Our experimental results show that low precision training can slightly improve the performance. The reason could be that  low precision training prevents the client from over-fitting  the wrong labeled samples, and thus reduces the impacts of the poisonous data. More results will be included in the revision.\n\n| Method            | Label Flip attack rate |Acc     |\n|:-                 |:-:                      |:-      |\n| FedAVG            | 0.4                    | 60.4   |\n| FedAVG+Ours(8 bit)| 0.4                    | 62.6   |\n| FedAVG            | 0.7                    | 22.8   |\n| FedAVG+Ours(8 bit)| 0.7                    | 27.5   |\n| ABAVG             | 0.4                    | 63.1   |\n| ABAVG+Ours(8 bit)| 0.4                    | 64.0   |\n| ABAVG             | 0.7                    | 37.9   |\n| ABAVG+Ours(8 bit)| 0.7                    | 40.6   |\n\n\n\n[r3] Chen D, et al. Efficient personalized federated learning via sparse model-adaptation. ICML 2023.\n\n[r4] Kuzmin, et al. The Power of the Exponent. NeurIPS 2022.\n\n[r5] Nagel, et al. Overcoming Oscillations in Quantization-Aware Training. ICML 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VGilmW8JOz", "reviewer": "Reviewer_vykk", "summary": "The paper proposes a federated learning approach that performs local training on low precision through quantization combined with a high-precision averaging and a moving average at the server. The paper guarantees convergence and empirically compares several levels of low-precision local training to full-precision training on 4 baseline FL methods. It remains unclear, though, what the contribution of the method is: the main focus seems to be on performance in terms of test accuracy, but the experiments do not show a significant improvement over existing methods. The method supposedly improves communication and computation efficiency but is not empirically compared to state-of-the-art methods, such as [1,2,3].\n\nIn their rebuttal, the authors provided novel results that address my concerns about missing baselines. While I remain concerned about the limited novelty and the presentation, I believe that the authors will be able to address these issues to some extent in the next version of the manuscript. Therefore, I have decided to increase my score.\n\n\\\n[1] Liu, Shiyu, et al. \"Quantized SGD in Federated Learning: Communication, Optimization and Generalization.\" International Conference on Neural Information Processing. Singapore: Springer Nature Singapore, 2023.\n\n[2] Kamp, Michael, et al. \"Efficient decentralized deep learning by dynamic model averaging.\" Machine Learning and Knowledge Discovery in Databases: ECML PKDD, 2018.\n\n[3] Reisizadeh, Amirhossein, et al. \"Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization.\" International conference on artificial intelligence and statistics. PMLR, 2020.", "questions": "- It is unclear from the results how much of the benefit stems from quantization and how much from the moving average. Is it correct that the moving average has a very large positive effect regardless of quantization?", "limitations": "Section F in the appendix addresses the limitation of simulating quantization. It does not address issues like computation and communication complexity, the applicability of the approach, or limitations in the empirical evaluation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- convergence guarantees\n- clear explanation of the method", "weaknesses": "- the presentation should be improved (e.g., several typos, such ass \"accept\" instead of \"aspect\", the term \"a avg\" for \"with moving average\" is unintuitive)\n- the ablation study does not clearly show that low precision local training improves performance, since it is combined with a moving average that has a strong positive impact on performance.\n- lack of comparison to baselines\n- unclear use-case for the method\n- the paper does not discuss existing federated learning with quantization literature in sufficient detail.\n- for the non-iid experiments it would be interesting how quantization interacts with local BN layers in FedBN [4]\n\n\\\n[4] Li, Xiaoxiao, et al. \"FedBN: Federated Learning on Non-IID Features via Local Batch Normalization.\" International Conference on Learning Representations, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Contribution in terms of  test accuracy.**\n\n**A1:** It is a misunderstanding. We do not aim to improve  test accuracy, but rather to demonstrate that low precision local training is sufficient for FL and can be used to reduce training and communication cost. Our method, which performs low precision local training, achieves comparable  (or even higher) accuracy with full-precision methods. The improvements on the training efficiency and communication cost can be guaranteed as our low precision operators are standard. \n\n**Q2: Compare with  FedPAQ, Quantized-SGD and dynamic-model-averaging.**\n\n**A2:** Thanks. In our paper, we gave the results with strong baselines HeteroFL [9] and SplitMix [14]. Below is the result on CIFAR10 ($\\alpha=0.01$) of FedPAQ.   \"()\" denotes the percentage of models on the clients. Following [r6] we use the number of   weights, activation, and gradients of local training to approximate training cost.  It shows that we can reduce communication and computational cost  while maintain high accuracy. \n\n| Method                        | Acc     | Communication cost (MB / round) | Training cost (MB / client) |\n| :-                          | :-:   | :-:                       |:-:                        |\n| HeteroFL (1)                  |41.1±2.4 |   39.1                          |  205.4                      |\n| HeteroFL (1/2)                |36.4±0.5 |   19.6                          |  83.7                       |\n| HeteroFL (1/4,1/8)            |29.2±2.1 |   7.3                           |  31.1                       |\n| SplitMix (1)                  |41.0±2.3 |   39.1                          |  178.4                      |\n| SplitMix (1/2)                |40.4±3.6 |   19.6                          |  89.2                       |\n| SplitMix (1/4,1/8)            |38.9±3.5 |   7.3                           |  33.4                       |\n| FedPAQ 8 bit                  |40.7±2.1 |   9.8                           |  205.4                      |\n| FedPAQ 6 bit                  |38.1±0.7 |   7.3                           |  205.4                      |\n| FedPAQ 5 bit                  |35.0±1.3 |   6.1                           |  205.4                      |\n| FedAVG+Ours(8 bit)                    |53.3±2.8 |   9.8                           |  51.3                       |\n| FedAVG+Ours(6 bit)                    |49.4±1.0 |   7.3                           |  38.5                       |\n| FedAVG+Ours(5 bit)                    |44.7±1.0 |   6.1                           |  32.1                       |\n\nWe gave the results of another SOTA CoCoFL in A2 of Reviewer 5cLQ. We have not yet found the code of Quantized-SGD and dynamic-model-averaging, due to the time limitation, we postpone their comparison to revision. \n\n**Q3: Typos.**\n\n**A3:** Thanks. We will improve the writing accordingly. \n\n**Q4: This result  does not clearly show low precision local training improves performance, since it is combined with moving average.**\n\n**A4:** Low-precision training typically tends to decrease instead of improve the accuracy. We introduce it  to reduce communication and computational costs, while  moving average  is adopted to compensate the above accuracy degradation. This point was confirmed in Table 1 and Figure 2. That is that with moving average, the accuracy of the low-precision model can be comparable to full-precision model, and sometimes even better due to its effect on reducing over-fitting risk.\n    \n**Q5: Unclear use-case.**\n\n**A5:** Reducing training and communication cost is an important topic in FL since the clients  in FL are always resource-constrained edge devices  with low-bandwidth communication network. We address this challenge by using low precision local training, which can be implemented efficiently on machine learning accelerators. We note that using accelerators with low precision operators is an new paradigm to accelerate AI models. For details, please refer to the surveys [r7] and [r8]. \n\n**Q6: Discuss FL with quantization literature.**\n\n**A6:** Thanks. We gave  comparison results with SOTA CoCoFL and FedPAQ with quantization in the table in A2 of Reviewer vykk and the table in A2 of reviewer 5cLQ. We will include them and discuss in details in the revision if accepted. \n\n**Q7: How quantization interacts with  FedBN.**\n\n**A7:** Thanks. The results ($\\alpha = 0.01$ ) given below show that our method works well with FedBN. More results will be included in the revision. \n\n| Method        | CIFAR10  | FMNIST   |\n|:-   |:- |:- |\n| FedBN         | 48.7±1.9 | 79.3±0.6 |\n| FedBN+Ours (16 bit) | 49.3±0.4 | 80.2±0.2 |\n| FedBN+Ours (8 bit)  | 48.5±0.3 | 78.6±0.5 |\n\n**Q8: Benefits stem from quantization and moving average.**\n\n**A8:** We would like to clarify  two things: \n1) Quantization is used to reduce the computational and communication cost and typically it tends to produce negative instead of positive  impact on accuracy and we  adopt  moving average to  alleviate this impact. \n2) In some tasks, quantization can reduce the risk of over-fitting leading to higher test accuracy. It is verified by Table 1 and  Fig. 2 in our paper.\n\n**Q9: Does moving average have a large positive effect regardless of quantization?**\n\n**A9:**  Moving average is used to reduce the error introduced by quantization. When full precision local training is adopted, its effect is relatively limited compared with that of low-precision, see Table 1 in our paper.\n\n**Q10: More limitation discussion.**\n\n**A10:** Thanks. For computation and communication, we follow the standard settings in FL. We gave a series of additional experimental results in this rebuttal and we will give more discussion in the revision.   \n\n[r6] Efficient personalized federated learning via sparse model-adaptation. ICML 2023.\n\n[r7]  AI and ML Accelerator Survey and Trends. IEEE HPEC, 2022.\n \n[r8] Advances and Open Problems in Federated Learning. Foundations and Trends in Machine Learning, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "561ttBjztF", "reviewer": "Reviewer_1d4C", "summary": "This paper proposes an efficient federated learning (FL) paradigm, where the local models in the clients are trained with low-precision operations and communicated with the server in low precision format, while only the model aggregation in the server is performed with high-precision computation. The performance is comparable to full-precision training, and sometimes even better since the over-fitting issue in local training is relieved.", "questions": "Justify method novelty against [40] (W1) and answer related questions in W2.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The idea of applying SWALP's low-precision training within each cycle for local training in FL is meaningful and effective.\nS2. There are theoretical analysis on convergence.\nS3. Experiments are comprehensive and demonstrate the effectiveness of the proposed method.", "weaknesses": "W1. The biggest concern is regarding the novelty w.r.t. SWALP [40]. The entire Sec 3.2 and Sec 4.1 are almost the same as in [40]. The only difference seems to be Sec 4.2 but still very similar to the idea of SWA, just the setting of aggregation changes from by cycles to by clients, and a moving average of parameters is used.\n\nW2. Writing needs improvement. For example, there is a typo \"sever\" in Line 108.\nEq (5) is different from its counterpart in [40] where the power was F-1 but now W-F-1. please explain the reason of difference.\nLine 135, missing a space before \"i.e.\"\nEq(7) uses E which is not clear until continuing reading to Line 161 and Algorithm 2 Lines 3-4.\nAlgorithm 2 Line 11, t' is only briefly mentioned in Line 161 without even referring to the used lines.\n\nW3. It would be good to estimate the time reduction with the professional hardware (real acceleration).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The biggest concern is regarding the novelty w.r.t. SWALP [40].** \n\n**A1:** We summarize the differences between our method and SWALP as follows:\n\n1) SWALP is designed for standard sgd in centralized training, our approach focuses on FL.\n2) We show both empirically and theoretically the effecitivenss of our approach in heterogeneous datasets. \n3) We show that besides FedAVG, our method can be integrated with various FL algorithms.\n\n\n**Q2: Typos, e.g., \"sever\" in Line 108 and missing a space before \"i.e.\".**    \n\n**A2:** Thanks. We will correct the typos and improve the writing accordingly.\n\n**Q3: Eq (5) is different from its counterpart in [40] where the power was F-1 but now W-F-1. Please explain the reason of difference.**\n\n**A3:** The reason for this issue is that in Section 3.1 of SWALP, the definitions of W and F are different between the \"Fixed Point Quantization\" section and the \"Block Floating Point (BFP) Quantization\" section. Our definition of W and F is the same as in the \"Fixed Point Quantization\" section. In our paper and \"Fixed Point Quantization\" section, F represents the number of bits occupied by the fractional part, while in the \"Block Floating Point (BFP) Quantization\" section, F stands for the number of bits occupied by the shared exponent.\n\n\n**Q4: Eq(7) uses E which is not clear until continuing reading to Line 161 and Algorithm 2 Lines 3-4.**\n\n**A4:** We provided the definition of E in line 115 of the paper.\n\n**Q5: Algorithm 2 Line 11, t' is only briefly mentioned in Line 161 without even referring to the used lines.**\n\n**A5:** Thank you for your suggestion, we will add further description after t'.\n\n**Q6: It would be good to estimate the time reduction with the professional hardware (real acceleration).**\n\n**A6:** It is difficult to conduct experiments on professional hardware in university, which is expensive and involves hardware programming. Fortunately, we note that our simulation is standard and approved by the community. The reasons are \n1) Such low precision training methods can be implemented on the machine learning accelerators efficiently to achieve real saving in computational and communication cost; \n2) The results would be always consistent with those obtained from simulation. \n\nFor details, please refer to [r9] and [r10].\n\n    \n\n[r9] Kuzmin A, et al. Fp8 Quantization: The Power of the Exponent. NeurIPS 2022.\n\n[r10] Nagel M, et al. Overcoming Oscillations in Quantization-Aware Training. ICML 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers were generally convinced about the communication and computation reduction in the context of FL achieved by the proposed method. They also appreciated both the extent of experiments and the convergence guarantees provided in the paper. On the negative side, there is an overlap with in techniques [40] though, as the authors point out, the application to FL is novel.

Overall, the authors should:
-Include in the paper the additional experiments that they conducted during the rebuttal stage; these can be added in the supplement if there is an issue of space, but hopefully some can surface to the main body of the paper as well.
-Proofread for typos, but also improve presentation w.r.t. aspects indicated by the reviewers.
Decision: Accept (poster)