Title: Low Precision Local Training is Enough for Federated Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Federated Learning

Low Precision Training
Abstract: Federated Learning (FL) is a prevalent machine learning paradigm designed to address challenges posed by heterogeneous client data while preserving data privacy.
     Unlike distributed training, it typically orchestrates resource-constrained edge devices to communicate via a low-bandwidth communication network with a central server.  This urges the development of more computation and communication efficient training algorithms. In this paper, we propose an efficient FL paradigm,  where the local  models in the clients  are trained with  low-precision operations and communicated  with the server in low precision format, while only the model aggregation in the server is performed with  high-precision computation. We surprisingly find that   high precision models can be recovered from the  low precision local models with proper aggregation in the server. 
     In this way, both the workload in the client-side and the communication cost can be significantly reduced.   We theoretically show that our proposed  paradigm can converge to the optimal solution as the training goes on, which demonstrates that low precision local training is enough for FL.  Our paradigm can be integrated with existing FL algorithms flexibly. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. Notably, the models trained by our method with the precision as low as 8 bits are  comparable  to those from the  full precision training. As a by-product, we show that low precision local training can relieve the over-fitting issue in local training, which under heterogeneous client data  can cause the client models drift further away from each other and lead to the failure in  model aggregation. Code is released at https://github.com/digbangbang/LPT-FL.

Low Precision Local Training is Enough for Federated Learning

Abstract

Federated Learning (FL) is a prevalent machine learning paradigm designed to address challenges posed by heterogeneous client data while preserving data privacy. Unlike distributed training, it typically orchestrates resource-constrained edge devices to communicate via a low-bandwidth communication network with a central server. This urges the development of more computation and communication efficient training algorithms. In this paper, we propose an efficient FL paradigm, where the local models in the clients are trained with low-precision operations and communicated with the server in low precision format, while only the model aggregation in the server is performed with high-precision computation. We surprisingly find that high precision models can be recovered from the low precision local models with proper aggregation in the server. In this way, both the workload in the client-side and the communication cost can be significantly reduced. We theoretically show that our proposed paradigm can converge to the optimal solution as the training goes on, which demonstrates that low precision local training is enough for FL. Our paradigm can be integrated with existing FL algorithms flexibly. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. Notably, the models trained by our method with the precision as low as 8 bits are comparable to those from the full precision training. As a by-product, we show that low precision local training can relieve the over-fitting issue in local training, which under heterogeneous client data can cause the client models drift further away from each other and lead to the failure in model aggregation. Code is released at https://github.com/digbangbang/LPT-FL .

1 Introduction

Federated learning (FL) [3, 15, 22, 36] is a popular privacy preserving machine learning paradigm to collaboratively learn a global model over the decentralized data. In FL paradigm, the clients are responsible for local training and only have access to their private datasets, while the server plays an essential role in aggregating the clients' updates into a global model. Unlike large-scaled distributed training, FL typically orchestrates resource-constrained edge devices to communicate via a low-bandwidth communication network with a central server. This urges the development of more computation and communication efficient optimization algorithms. The most prevalent approach in FL is developed based on local-SGD [26], which is referred to as FedAvg. In each communication round, the clients individually train their local models for multiple steps and then send them to the server for aggregation. It can be expected that if longer local training process one uses, the greater communication cost saving one can achieve. However, long time local training

could cause the local models drift further away from each other and degrades the aggregated global model's performance or even make the training diverge, especially when the data on the clients are heterogeneous. Therefore, in order to prolong local training processes in FL, extensive efforts have been made in the recent years. For example, the studies [1, 16, 21, 22] modify the local training process by imposing regularization on the client models to enforce them not to drift away from the previous global model. Another line of research [5, 6, 24, 31, 35, 38] focuses on refining the global model in the server aggregation process. These methods typically require a large proxy dataset on the server. Some of them [5, 6, 24] use it to align the outputs of the global model with that of the client ensemble by knowledge distillation. Others develop handcrafted aggregation rules to reweight the updates based on the statistics of updates or performance on proxy data [31, 35, 38] or further tune the global model with proxy data in every communication round [5, 24]. Although promising experimental results have been reported in the literature, it is still unclear that whether there exists more concise and effective FL paradigm, which can reduce both the workload in the client-side and the communication cost.

In this paper, we propose a concise and efficient federated learning paradigm, where the local models in the clients are trained with low precision operations and communicated with the server in low precision format, while only the server-side information integration maintains high-precision computation to ensure the accuracy. Our basic idea is inspired from the Kolmogorov's law [10], that is, the sample average can converge almost surely to the expected value although the samples always contain noise. Therefore, in the server side, we perform the simple moving average on the received low precision models from the clients to recover a high precision global model. In this way, both the workload in the client-side and the communication cost can be significantly reduced. We theoretically proved that our proposed paradigm integrated with FedAVG can converge to the optimal solution as the training goes on, which indicates that low precision local training is enough for federated learning. We extend our method to various existing FL method to show its flexibility. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. Notably, the models trained by our method with the precision as low as 8 bits are comparable to those from the full precision federated learning. Compared with some efficient FL designs, our method can achieve significant savings in training memory overhead, and what is more attractive is that our accuracy performance is even better. Our method exhibits another appealing feature in relieving the over-fitting issue in local training. To be precise, in the local training steps, the models can be easily trained to over-fit the local training data as the local dataset is always insufficient. Under heterogeneous client data, it would further cause the client models drift further away from each other and lead to the failure in model aggregation. The experimental results show that our approach can effectively relieve the over-fitting issue since the local training is performed with low precision computation and the expressiveness of the local model is restricted.

Our main contributions are as follows:

Wepropose an efficient federated learning paradigm that performs low precision computation during local training, saving computational overhead and communication costs, while being able to restore accuracy through high-precision aggregation on the server side.

We theoretically proved that the efficient federated learning algorithm we proposed can achieve convergence at a rated of O (1 /T ) under certain assumptions for non-iid situations.

Since the expressiveness of the local model is restricted due to the low precision local training, our approach can relieve the over-fitting issue, which would cause the client models drift further away from each other and lead to the failure in model aggregation when the local dataset are heterogeneous.

The extensive experimental results demonstrate the effectiveness of our approach. Notably, the models trained by our method with the precision as low as 8 bits are comparable to those from the full precision federated learning.

2 Related Work

Federated Learning. Fedrated Learning is first proposed by [26] to realize model training without sharing client device data. Many works have continued to solve some challenges of FL such as heterogeneity [16, 22, 25], privacy [2], communication efficiency [11, 18]. Also, some works proposes new FL methods to alleviate data heterogeneity. The vanilla FL method was FedAvg [26].

FedProx [22] utilizes a regularization term while Scaffold [16] sets a control variate to reduce the drift in local training. FedGen [42] and FedFTG [40] maintain a generator, the former is used for local data augmentation, while the latter is used for fine-tuning the server.

Efficient Federated Learning. One challenge of FL is the limitation of low bandwidth and computing resources of client devices. [4, 20] assign each client a block mask, resulting in sparse local models. [28] took the transmission speed into consideration and chose the same method as [12] for uploading, uploading compressed gradients. [13] adopts boost training to client-side training overhead. [7, 28] only transmit the trained head to reduce transmission cost. [33] adopts the idea of Network Architecture Searching, e.g., each client selects a sub-network. [14] maintains a series of streamlined models in the server, from which the client selects a tiny model for training. In [9], the client selects a sub-model of the global model for training. In this paper, we address this issue by using low precision local training.

3 Preliminary

3.1 Federated Learning

Given N clients with their private datasets D k = { ( x k,j , y k,j ) } |D k | j =1 , k = 1 , . . . , N , the optimization objective of FL is always defined as follows:

where w ∈ R d represents the model parameters, F k is denoted to be the empirical risk function of client k , i.e., F k ( w ) = ∑ ξ ∈D k 1 |D k | ℓ ( w ; ξ ) with ℓ ( · , · ) being the loss function and p k = |D k | ∑ N k =1 |D k | denotes the proportion of data contained in client k .

FL emphasizes data privacy protection and thus the server is not allowed to access these datasets D k directly in model training. The standard method to solve the above training problem of FL is FedAvg [26], which is developed based on local SGD. It is comprised by two steps, i.e., local training on the clients and model aggregation in the server side. The details are presented below.

In local training, the central server would first randomly select partial clients and broadcasts the latest global model w t to them. We denote the selected clients set as S t and let K = |S t | be the number of selected clients. Then the client k with k ∈ S t would initialize its local model to be w k t = w t and then performs local training with E ( ≥ 1) iterations as follows:

where w k t is the weights of the k -th client in step t , ξ k t is a mini-batch of samples uniformly chosen from D k , η t is the step size.

In model aggregation, FedAvg updates the global model to be the weighted average of the received local models, i.e.,

where q t = ∑ k ∈S t p k normalize the coefficients.

3.2 Block Floating Point Quantization

Fixed point quantization is a standard quantization technique. It uses stochastic rounding to round the numbers up or down at random such that E [ Q ( x )] = x , where Q : R → R is the quantization function defined as

here clip ( x, a, b ) = max(min( x, b ) , a ) , δ = 2 -F is the quantization gap represents the distance between successive representable fixed point numbers, u = 2 W -F -1 -2 -F and l = -2 W -F -1 represent the upper and lower limits of the representable numbers, respectively. W is the bit width

of quantized numbers, and F is the bit width of quantized numbers' fractional part. In order to improve the utilization efficiency of the bit width and better maintain numerical accuracy when data is unevenly distributed, we choose block floating point quantization. Given a block of numbers X , it replaces δ = 2 -F in fixed point quantization to be δ = 2 -( W -2 -E ( X )) , where

The shared exponent E ( X ) is usually set to be the largest exponent in X to avoid overflow [37].

4 Method

In this section, we introduce our low precision federated learning paradigm. It is comprised of two modules,i.e., one is the low precision local training to reduce the computation and communication cost, the other is the high precision aggregation with moving average to maintain the model accuracy.

4.1 Low Precision Local Training

In order to reduce the computation and communication cost, we apply block floating point quantization to all clients' device of local training. The simple version of low precision local training is to convert the local training step in Eqn.(2) into

Note that in the above version, we only quantize the updated parameters. We give this version just for the convenience of the theoretical analysis in Section 5. In practice, we quantize the gradient, the activation of each layer, the back-propagation signals, and the momentum in SGD when SGD is adopted as the optimizer. The details are given in Algorithm 1.

Algorithm 1 Low Precision Local Training with All Numbers Quantized

Input: Quantization functions Q A , Q E , Q G , Q M , Q W ; Momentum coefficient ρ ; L layers DNN { f 1 , f 2 , . . . , f L } ; Loss function ℓ .

1: ClientUpdate( t, k, w k t ):

2: Get batch ( x k,j t , y k,j t ) from D k

3: Forward Propagation:

4: ( a k t ) (0) = x k,j t

5: ( a k t ) ( l ) = Q A ( f l (( a k t ) ( l -1) , ( w k t ) ( l ) )) , ∀ l ∈ [1 , L ]

6: Backward Propagation:

7: ( e k t ) ( L ) = ∇ ( a k t ) ( L ) ℓ (( a k t ) ( L ) , y k,j t )

8: ( e k t ) ( l -1) = Q E ( ∂f l (( a k t ) ( l -1) , ( w k t ) ( l ) ) ∂ ( a k t ) ( l -1) ( e k t ) ( l ) ) , ∀ l ∈ [1 , L ]

9: ( g k t ) ( l ) = Q G ( ∂f l (( a k t ) ( l -1) , ( w k t ) ( l ) ) ∂ ( w k ) ( l ) ( e k t ) ( l ) ) , ∀ l ∈ [1 , L ]

t 10: Low Precision SGD Update:

11: ( v k t +1 ) ( l ) ← Q M ( ρ ( v k t ) ( l ) +( g k t ) ( l ) ) , ∀ l ∈ [1 , L ]

13: Return: w k

t +1

4.2 High Precision Aggregation

Although low precision training can reduce communication and training overhead, it would lead to a degradation in training accuracy. Inspired from the Kolmogorov's law [10], that is, the sample average can converge almost surely to the expected value although the samples always contain noise, we try to recover high-precision solution from the low-precision local model with a full precision aggregation process. It is implemented with the following two steps:

Calculate the weighted average of the local models to partially reconvert the precision, i.e.,

Since in most cases clients' data is non-iid and quantization causes error, federated learning is harder to converge, however maintaining a moving average in the server can significantly alleviate the problem. Formally, we denote ¯ w t as the moving average stored in the server, then after aggregation, we update ¯ w t as follow

where λ is a coefficient controlling the influence of current weight. This procedure can further compensate the accuracy degradation due to the low precision local training.

In the next round of local training, the quantized model Q ( ¯ w t ) will be distributed to the clients to intilize the local models. Our pseudocode in Algorithm 2 depicts the process of low precision local training on the client device and high precision aggregation on the server. In Algorithm 2, t ′ = t -E +1 and I E = { nE | n = 1 , 2 , · · · } represents the set of global synchronization steps.

5 Theoretical Analysis

In this section, we give the detailed theoretical results for our low precision FL paradigm. We will first introduce the convergence analysis in the full participation case where all client devices participate (i.e., K = N ) and then we generalize the results of the analysis to scenarios that are more in line with reality. (i.e., K < N ). The results demonstrate that we will explore aggregation strategies represented by the FederatedAveraging Algorithm (or FedAvg) and demonstrate that our proposed low precision FL framework can converge to the global optimal solution at a rate of O (1 /T ) for non-iid datasets based on strong convexity and smoothness assumptions.

5.1 Assumptions and Notations

We need to make necessary assumptions about the objective function on the clients F k , k = 1 , . . . , N . Assumption 1 is about the smoothness and strong convexity of the loss function and Assumption 2 is about the boundness of the gradients. These assumptions are standard and widely adopted in the related studies [41, 29, 30, 39].

Assumption 1. F 1 , F 2 , ..., F N are L-smooth and µ -strongly functions, which means that for all v and w , the following inequalities hold:

where ∥·∥ 2 represents the square of two norms and k = 1 , . . . , N .

Assumption 2. Let ξ k t be sample that randomly and uniformly sampled from the local data of the k -th client device. For k = 1 , 2 , · · · , N and t = 0 , 1 , · · · , the variance of stochastic gradients in each client device and the expectation of squared two norm of stochastic gradients is bounded:

Degree of Data Heterogeneity. Let F ∗ denote the global minimum of the objective function F , and let F ∗ k represent the minimum of the local objective function F k specific to the k -th client. We use the metric Γ [23] taking the form of

to measure the degree of heterogeneity of all clients' data distribution. When the data on each client device is iid, as the number of samples increase, Γ evidently tends towards zero. However, when faced with non-iid situation, Γ tends towards a positive constant and thus it can be used to measure the degree of heterogeneity in the data distribution of each client device.

5.2 Convergence Analysis: Full Client Device Participation

First we analyze the convergence of the participation of all clients' device in this section. We integrate our low precision FL framework with FedAvg and train the model for T iterations to obtain the ¯ w T , and we expect T to be divided by E so that ¯ w T is the weight after aggregation.

Theorem 1. Under the Assumptions 1 and 2, we set κ = L µ , γ = max { 8 κ, E } -1 and η t = µ 2( t + γ ) . When t satisfying δ 2 ≤ η 2 t G 2 , low precision FedAvg with full device participation satisfies:

where

5.3 Convergence Analysis: Partial Client Device Participation

Compared to full participation situation, the partial participation situation is more in line with the reality. We need to make more assumption on how to choose S t .

Assumption 3. Assume S t contains a subset of K indices uniformly sampled from [ N ] without replacement. In addition, the data is balanced in the sence that p 1 = · · · = p N = 1 N . The aggregation step of FedAvg performs w t + E ← 1 K ∑ k ∈S t w k t + E

Theorem 2. Under the Assumptions 1 to 3, we choose κ = L µ , γ = max { 8 κ, E }-1 , η t = µ 2( t + γ ) and B = 2( √ dδ +1+ d 2 ) G 2 +16 E 2 G 2 (2 √ dδ +3)+ 1 N 2 ∑ N k =1 σ k 2 +6 L Γ , C = N -K N -1 8 K E 2 G 2 (2 √ dδ + 3)+ dG 2 . When t satisfying δ 2 ≤ η 2 t G 2 , low precision FedAvg with partial client device participation satisfies:

6 Experiments

In this section, we conduct extensive experiments to verify the effectiveness of our methods in the following five aspects:

When integrated with FedAvg, the models trained by our method with the low precision are comparable to (if not better than) those from the full precision training. This would also verify our theoretical results (Section 6.1).

Our method can effectively relieve the over-fitting issue in FL. See Section (Section 6.2).

Our paradigm can be integrated with existing FL methods flexibly and preserve the performance even with a low precision local training (Section 6.3).

Ablation studies on the effectiveness of moving average in model aggregation and the transferability over various neural networks (Section 6.4).

Comparison with other efficient FL techniques (Section 6.5).

Remark 1. Similar with the existing low precision training studies [32], We do not give the results on the running time to show the real acceleration. The reason is that to achieve real acceleration, we need to implement our method integrated with the professional hardware. Moreover, such implementation is standard for the professional hardware platforms.

Benchmark Datasets and Baseline. We conduct experiments over four commonly used datasets: FashionMnist [34], CIFAR10 [19], CIFAR100 [19] and CINIC10 [8]. Four commonly used FL methods: 1) FedAvg [26]; 2) regularization-based strategy FedProx [22]; 3) data-dependent knowledge distillation strategy ABAvg [35] 4) data-free knowledge distillation strategy FedFTG [40] and FedGen [42] are adopted as the baselines.

Configurations. We follow the configurations in the recent studies [26, 42, 27] for fair comparison. To be precise, for FashionMNIST, CIFAR10, CINIC10 and CIFAR100, we run 200 communication rounds with local epoch set to 1. There are 80 clients in total, and the participation ratio in each round is set to 40%. We use Dirichlet distribution to simulate non-iid data distribution and set α to 0.01, 0.04, and 0.16. The smaller α is, the more serious the data heterogeneity is. For the network choice, we use ConvNet following with 3 layers, and the hidden dimension is set to 128. The local learning rate is set to 10 -3 with Adam optimizer [17]. We report the last 5 round global model's average performance evaluated using the test split of the datasets. For quantization method, we adopt the Block FLoating Point Quantization with the number of bits used set to 6, 8 and 32 (without quantization). Some of the other hyperparameter settings are included in the Appendix C.

6.1 Results on FedAvg

We demonstrate the superior performance of our Low Precision FL method with FedAvg by conducting experiments over comprehensive datasets, various precision and heterogeneity values α .

Heterogeneity. As shown in Table 1, it is as expected that when the heterogeneity goes higher, that is, when α decreases, the server performance worsens. Nevertheless, our proposed method can always maintain or improve the performance of the original case (bits = 32, w/o. avg) when using 8 quantizaiton bits with moving average, which empirically validates the effectiveness of our proposed method.

Quantization Bits. We conduct experiments on 3 quantization bits: 32, 8, 6 (shown in Table 1, Figure 1), as we observe that in most cases 8 bits is enough to hold the performance of full-precision and when the used bits is 6, the server performance begins to decrease due to the low precision level.

6.2 Quantization Relieves Overfitting

We present the averaged local training loss and the global testing loss over training in Figure 2. It can be seen that our method can effectively reduce testing losses on the server. The commonality

(a) FMNIST

(b) CIFAR10

(c) CINIC10

(d) CIFAR100

0.80

0.50

0.40

0.15

Test Accuracy

Test Accuracy

0.30

0.10

0.40

0.30

0.20

Origin

Origin

Origin

0.05

Origin

0.20

8 bit w avg

6 bit w avg

0.10

8 bit w avg

6 bit w avg

0.10

8bit w avg

6 bit w avg

8 bit w avg

6 bit w avg

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

3.00

8 bit w avg

Origin

8 bit w avg

Origin

3.50

8bit w avg

Origin

4.80

8 bit w avg

Origin

Test Loss

2.00

2.50

6 bit w avg

2.50

6 bit w avg

6 bit w avg

4.60

6 bit w avg

4.40

1.50

1.00

M

4.20

0.50

1.50

2.00

4.00

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

with the original method is that when each round of communication starts retraining, the training loss of the client will be greatly increased due to the heterogeneity of the data D k . Subsequently, due to the highly imbalanced local data categories, the model quickly reached an overfitting state. It can be seen that at the beginning of each training round, under our method, the customer's training loss will not exceed the original training loss. At the same time, with some training steps, our training loss remains above the original training loss, which means our method can alleviate the overfitting problem of local training.

Client Test Loss

1.4

1.2

1.0

0.8

0.050

0.025

0.075

11001200

3.5

3.0

2.5

2.0

1.6

1.2

0.20

0.10

1100

1200

3.5

3.0

2.5

3.0

2.0

0.2

0.1

1100

1200

4.5

5.5

5.0

4.0

3.0

0.50

1.00

1100

1200

6.5

6.0

5.5

LoSS

Train

0.6

0.4

2.0

1.5

0.8

2.5

1.5

1.0

3.5

2.5

2.0

5.0

Global

0.2

1.0

0.4

2.0

0.5

1.5

1.0

4.5

1050

1150

1250

1350

1050

1150

1250

1350

1050

1150

1250

1350

1050

1150

1250

1350

Training Iterations(FMNIST)

Training Iterations(CIFAR10)

Training Iterations(CINIC10)

Training Iterations(CIFAR100)

8 bit w avg (Ours) Client Train Loss

Fedavg Client Train Loss

8 bit w avg (Ours) Global Test Loss

Fedavg Global Test Loss

6.3 Results on Other FL Methods

The four FL methods we used are each representative. ABAvg and FedFTG are similar to FedAvg in local training, but the former only performs weight adjustment on the server side, while the latter uses knowledge distillation to fine-tune the server. FedProx and FedGen are similar to FedAvg in the server side, but the former only has regularization constraints on local training, while the latter uses the generator for regularization adjustment in local training. As is demonstrated in Table 2, we can see that, regardless of the FL method chosen, our low precision FL algorithm has a significant improvement in prediction accuracy compared to the original FL method, especially in dataset CIFAR10, CINIC10 and CIFAR100.

6.4 Ablation Study

Moving Average. We conduct ablation study to validate the effectiveness of the moving average for the server, from Table 1, the average operation has a significant effect on the server performance over various settings. It is consistent of our expectation that averaging can alleviate the error introduced by quantization and the error by heterogeneity.

Architecture Generalization. We use another network structure, MLP, to verify that our method can be applied to other networks. The MLP was set to 3 layers, with the hidden dimension of each layer being 128, and experiments were conducted on FashionMNIST with α of 0 . 01 and 0 . 04 . From Table 3, our method can still maintain the accuracy, which also demonstrates its generalization ability on other network structures.

6.5 Versus Efficient FL Method

In order to demonstrate that our method can effectively reduce computational costs and memory consumption, we compared it with two personalized model compression FL algorithms. HeteroFL [9] and Split-Mix [14] can both achieve the purpose of reducing training overhead by using smaller models for local training. We use the ConvNet with the same hidden dimension to conduct experiments, using CIFAR10 in the process, α = 0 . 01 . SplitMix uses a split ratio of 1/8 by default. We re-tested at a split ratio of 1/16 to achieve the same compression ratio as HeteroFL. In Figure 3a, the vertical dotted line indicates the compression ratio of HeteroFL, more details are in the Appendix E. Compared with methods that directly compress model parameters, our method can achieve higher accuracy with lower memory consumption. Because quantization does not bring about the deletion of model parameters, there will be no aggregation problems when the model compression rate is too high.

Method

Type

Acc

Communication cost

Train cost

HeteroFL

(1)

41

.

1

±

2

.

4

39

.

1

205

.

4

(1

/

2)

36

.

4

±

0

.

5

19

.

6

83

.

7

(1

/

4

,

1

/

8)

29

.

2

±

2

.

1

7

.

3

31

.

1

SplitMix

(1)

41

.

0

±

2

.

3

39

.

1

178

.

4

(1

/

2)

40

.

4

±

3

.

6

19

.

6

89

.

2

(1

/

4

,

1

/

8)

38

.

9

±

3

.

5

7

.

3

33

.

4

FedAVG

8

bit

53

.

3

±

2

.

8

9

.

8

51

.

3

6

bit

49

.

4

±

1

.

0

7

.

3

38

.

5

5

bit

44

.

7

±

1

.

0

6

.

1

32

.

1

(a) Training cost Versus Acc.

(b) Results on CIFAR10 with α = 0 . 01 .

7 Conclusion

In this paper, we propose an efficient FL paradigm, where the local models in the clients are trained with low-precision operations and communicated with the server in low precision format, while only the model aggregation in the server is performed with high-precision computation. We theoretically show that our proposed paradigm can converge to the optimal solution as the training goes on, which demonstrates that low precision local training is enough for FL. Our paradigm can be integrated with existing FL algorithms flexibly. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. As a by-product, we show that low precision local training can relieve the over-fitting issue in local training.

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations (ICLR 2021) , 2021.

Naman Agarwal, Ananda Theertha Suresh, Felix Xinnan X Yu, Sanjiv Kumar, and Brendan McMahan. cpsgd: Communication-efficient and differentially-private distributed sgd. Advances in Neural Information Processing Systems , 31, 2018.

Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Koneˇ cný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In A. Talwalkar, V. Smith, and M. Zaharia, editors, Proceedings of Machine Learning and Systems , volume 1, pages 374-388, 2019.

Daoyuan Chen, Liuyi Yao, Dawei Gao, Bolin Ding, and Yaliang Li. Efficient personalized federated learning via sparse model-adaptation. In International Conference on Machine Learning , pages 5234-5256. PMLR, 2023.

Hong-You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable to federated learning. In International Conference on Learning Representations (ICLR 2021) , 2021.

Yae Jee Cho, Andre Manoel, Gauri Joshi, Robert Sim, and Dimitrios Dimitriadis. Heterogeneous ensemble knowledge transfer for training large models in federated learning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI) Main Track , 2022.

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning , pages 2089-2099. PMLR, 2021.

Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505 , 2018.

Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Representations (ICLR 2021) , 2021.

Rick Durrett. Probability: theory and examples , volume 49. Cambridge university press, 2019.

Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated learning. arXiv: Learning,arXiv: Learning , Feb 2019.

Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference on Artificial Intelligence and Statistics , pages 2350-2358. PMLR, 2021.

Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. Fedboost: A communicationefficient algorithm for federated learning. In International Conference on Machine Learning , pages 3973-3983. PMLR, 2020.

Junyuan Hong, Haotao Wang, Zhangyang Wang, and Jiayu Zhou. Efficient split-mix federated learning for on-demand and in-situ customization. In International Conference on Learning Representations (ICLR 2022) , 2022.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends® in machine learning , 14(1-2):1-210, 2021.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Scaffold stochastic controlled averaging for federated learning. In International Conference On Machine Learning, Vol 119 , volume 119. JMLR-JOURNAL MACHINE LEARNING RESEARCH, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR 2015) , 2015.

Jakub Koneˇ cný, H.Brendan McMahan, FelixX. Yu, Peter Richtárik, AnandaTheertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv: Learning,arXiv: Learning , Oct 2016.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems , pages 42-55, 2021.

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10713-10722, 2021.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations (ICLR 2020) , 2020.

Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems , 33:2351-2363, 2020.

Yishay Mansour, Mehryar Mohri, Jae Ro, and AnandaTheertha Suresh. Three approaches for personalization with applications to federated learning. Cornell University - arXiv,Cornell University - arXiv , Feb 2020.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Renjie Pi, Weizhong Zhang, Yueqi Xie, Jiahui Gao, Xiaoyu Wang, Sunghun Kim, and Qifeng Chen. Dynafed: Tackling client data heterogeneity with global dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12177-12186, 2023.

Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-efficient federated learning with sketching. In International Conference on Machine Learning , pages 8253-8265. PMLR, 2020.

Sebastian U Stich. Local sgd converges fast and communicates little. In International Conference on Learning Representations (ICLR 2019) , 2019.

Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. Advances in neural information processing systems , 31, 2018.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems , 33:7611-7623, 2020.

Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. Advances in neural information processing systems , 31, 2018.

Tianchun Wang, Wei Cheng, Dongsheng Luo, Wenchao Yu, Jingchao Ni, Liang Tong, Haifeng Chen, and Xiang Zhang. Personalized federated learning via heterogeneous modular networks. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 1197-1202. IEEE, 2022.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Jianhang Xiao, Chunhui Du, Zijing Duan, and Wei Guo. A novel server-side aggregation strategy for federated learning in non-iid situations. In 2021 20th international symposium on parallel and distributed computing (ISPDC) , pages 17-24. IEEE, 2021.

Yueqi Xie, Weizhong Zhang, Renjie Pi, Fangzhao Wu, Qifeng Chen, Xing Xie, and Sunghun Kim. Robust federated learning against both data heterogeneity and poisoning attack via aggregation optimization. arXiv preprint arXiv:2211.05554 , 2022.

Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Chris De Sa. Swalp: Stochastic weight averaging in low precision training. In International Conference on Machine Learning , pages 7015-7024. PMLR, 2019.

Yousef Yeganeh, Azade Farshad, Nassir Navab, and Shadi Albarqouni. Inverse distance aggregation for federated learning with non-iid data. In Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning: Second MICCAI Workshop, DART 2020, and First MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4-8, 2020, Proceedings 2 , pages 150-159. Springer, 2020.

Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 5693-5700, 2019.

Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10174-10183, 2022.

Yuchen Zhang, Martin J Wainwright, and John C Duchi. Communication-efficient algorithms for statistical optimization. Advances in neural information processing systems , 25, 2012.

Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning , pages 12878-12889. PMLR, 2021.

This appendix can be divided into several parts. To be precise:

Section A gives the detailed proof for Theorem 1.

Section B gives the detailed proof for Theorem 2. Compared to Theorem 1, Theorem 2 assumes that only a subset of client devices participate in the training, which is more in line with real-world scenarios.

Section C introduce four FL methods'(ABAvg, FedProx, FedGen, FedFTG) hyperparameters setting in our experiments.

Section D provides detailed experiment results of four FL methods(ABAvg, FedProx, FedGen, FedFTG).

Section E presents the comparsion between our low precision FL method and two efficient FL method in training overhead.

Section F gives Limitation of this paper.

Section G gives Broader Impacts of this paper.

A Proofs of Theorem 1

We analyze our low-precision federated learning algorithm in the setting of full device participation in this section. For the sake of convenience in proving, let's take the value of p k to be 1 N , disregard the moving average, assume that quantization is only applied to model parameters and quantization gap is always δ . Let I E be a set composed of the sequence number of global aggregation steps, i.e., I E = { nE | n = 1 , 2 , · · · } . Suppose the model weights are updated as follows:

where w k t is an d -dimensional vector, which represents the weight of the k -th client device in step t . F k represents the objective function of the k -th device. N is the total number of clients device participating in the training, and T is the total number of steps trained, ξ k t represents a batch of data randomly sampled from the k -th client device's local dataset D k in step t , ∇ F k ( w k t ; ξ k t ) is stochastic gradient, η t represents learning rate, Q represents the quantization function, and its formula has already been given in (4).

Notations. To further simplify our proof process, we introduce the following additional definitions:

w t represents the weight of the global model in step t . Obviously, we can deduce E g t = ¯ g t and ¯ v t +1 = ¯ w t -η t g t .

Lemma 1. If w is a scalar and can be written as w = ˜ w + bδ , satisfying -δ < ˜ w < δ , δ is the quantization gap, represents the distance between successive representable fixed-point numbers, then we have:

where Q ( · ) is quantization function. And we always can find a suitable ˜ w that satisfies | ˜ w | ≤ | w | .

Proof. The quantization function is:

When ˜ w < 0 :

So we have:

Next we prove that we always can find a suitable ˜ w that satisfies | ˜ w | ≤ | w | . When w ≥ δ ,we can choose a ˜ w satisfying ˜ w ≥ 0 , then we can have | ˜ w | ≤ | w | . When w ≤ -δ , we can choose a ˜ w satisfying ˜ w ≤ 0 , then we also can have | ˜ w | ≤ | w | . When -δ < w < δ , we can choose a ˜ w satisfying ˜ w = w , then we also can have | ˜ w | ≤ | w | . In summary, we always can find a suitable ˜ w that satisfies | ˜ w | ≤ | w | .

Discussion. In the lemma 2 below, we analyze the convergence of the algorithm under the condition that δ 2 ≤ η 2 t G 2 , where G 2 is given in assumption 2 and represents the constraint on the expected value of the squared two-norm of the stochastic gradient. The right side of the inequality indicates the change in the iterative parameters during the gradient descent process, that is, the size of the gradient multiplied by the size of the learning rate. If the magnitude of this value is already less than the quantization precision, it will cause the gradient descent to fail, and the model parameters will not continue to change. In this case, it makes no sense to continue optimization, so analyzing convergence under this condition is practical.

Lemma 2. We assume that η t ≤ 1 4 L , based on assumption 1 and 2, when t satisfying δ 2 ≤ η 2 t G 2 we have:

Proof. When t +1 / ∈ I E , we can easily derive that ¯ w t +1 = ¯ v t +1 . So we have:

⟨·⟩ represents inner product operation. Due to E Q ( x ) = x , according to the Law of iterated expectations, E A 3 = 0 . Next we aim to bound A 2 . E A 2 is:

̸

In the above equation, we used the law of iterated expectations.

The quantization function is:

w k t is a quantified vector, therefore, its p-th component ( w k t ) p = c δ , c is an integer. We rewrite [ η t ∇ F k ( w k t , ξ k t ) ] p = bδ + ˜ w . Where b is an integer and | ˜ w | ≤ | w | . So that we can have:

The expectation after the square is:

In the above inequality, we use the inequality ∥·∥ 1 ≤ √ d ∥·∥ 2 2 , d is the dimension of the vector. According to assumption 2 we can get:

We next aim to bound A 1 :

Obviously E B 3 = 0 , so we can get:

We next aim to bound B 1 :

To bound C 1 , C 2 , C 3 , we need to derive several inequalities using the properties of F 1 , F 2 , · · · , F N . First of all, for k = 1 , 2 , · · · , N , F k is a L -smooth function, we can get:

Specifically, when x = w k t -a ∇ F k ( w k t ) , we can get:

when a = 1 L , we can get:

According to the above equation, we can get:

In the inequality above, we have utilized inequality (16) and the property that ∥·∥ 2 2 is a convex function.

For k = 1 , 2 , · · · , N, F k is a µ -strong function, so we can get:

By the inequality:

we can get:

In the inequality above, we have utilized inequality (16) again. Now, we can use inequality (16), (17), (18), (20) to bound C 1 , C 2 , C 3 to bound B 1 :

It should be noted that Γ = N ∑ k =1 p k ( F ∗ -F k ∗ ) = F ∗ -N ∑ k =1 1 N F k ∗ , and it measures the degree of data heterogeneity between different client devices.We can bound the first term in D :

In the above inequation, we use the inequations (16),(19) and the convexity of F k ( · ) . So we can bound D and get:

When t +1 / ∈ I , we can conclude that:

Proof.

Lemma 4. If η t is non-increasing and η t ≤ 2 η t + E , ( t ≥ 0) ,based on assumption 2, we can get:

E 2 has been given constraints in the text above that:

According to the law of iterated expectations, E 3 is 0. We next constrain E 1 . According to our assumption before Lemma 2, we have:

So when t +1 ∈ I E we have:

In summary, whether t +1 ∈ I E or t +1 / ∈ I E , we have:

Lemma 3. Based on assumption 2, we can get:

Proof. Since FedAvg requires a communication each E steps. Therefore, for any t ≥ 0 , there exists a t 0 ≤ t , such that t -t 0 ≤ E -1 and ¯ w t 0 = w k t 0 for all k. Also, we use the fact that η t is non-increasing and η t ≤ 2 η t + E , ( t ≥ 0) , then:

Inequalities ∥ ∥ ∥ ∥ E ∑ i =1 a i ∥ ∥ ∥ ∥ 2 2 ≤ E E ∑ i =1 ∥ a i ∥ 2 2 and E ∥ X -E X ∥ 2 2 ≤ E ∥ X ∥ 2 2 are used in the above proof.

We have already proofed in lemma 2 that:

So we can have:

Theorem 1. Under the condition that assumptions 1, 2, we choose κ = L µ , γ = max { 8 κ, E } -1 and η t = µ 2( t + γ ) . When t satisfying δ 2 ≤ η 2 t G 2 our low precision federated learning algorihm with full device participation satisfies:

where

Proof. Set ∆ t = E ∥ ¯ w t -w ∗ ∥ 2 2 , according to lemma 1, 2, 3, we can have:

Set η t = β t + γ , we can choose appropriate β and γ so that η t ≤ min { 1 µ , 1 4 L } and η t ≤ 2 η t + E . Set v = max { β 2 B βµ -1 , ( γ +1)∆ 1 } , we will prove ∆ t ≤ v t + γ by mathematical induction.

When t = 1 , obviously it holds. Assume the conclusion holds for some t , it follows that:

Then by the L-smoothness of F ( · ) ,we have:

We choose β = 2 µ , γ = max { 8 κ, E } -1 and denote κ = L µ , then we have:

and

B Proofs of Theorem 2

We analyze our low precision federated learning algorithm in the setting of partial device participation in this section.

Updating scheme In real world application scenarios, constrained by communication efficiency and low straggler effect, FedAvg initiates by selecting a random subset S t which length is set to K , and subsequently carries out updates exclusively on this subset.The analysis becomes somewhat complex due to the variability of S t every E steps. Nevertheless, we can employ a thought trick to address this challenge. We envision that FedAvg initiates each epoch by engaging all devices and then relies solely on the parameters updated on a subset of these devices to generate the parameters for the subsequent round. It is evident that this method of parameter update is equivalent to the original approach. Then the update of FedAvg with partial devices active can be described as: for all k ∈ [ N ] ,

We use E S t ( · ) to eliminate the error caused by S t . Unless otherwise specified, the meanings and assumptions of the symbols used in this section are the same as in the previous section.

Lemma 5. If t +1 ∈ I E , we have:

Proof. According to the selection method of S t , there is a function:

According to the aggregation process, we have:

So we have:

Lemma 6. For t +1 ∈ I E we have:

Proof. According to the aggregation process,we have ¯ w t +1 = Q ( 1 K ∑ i ∈S t v i t +1 ) .

In the lemma 2, we have proven F 1 ≤ dG 2 η 2 t . And obviously we have F 3 = 0 , we next aim to bound F 2 .

̸

The last step in the above equation uses:

̸

̸

Therefore:

We used E ∥ X -E X ∥ 2 2 ≤ E ∥ X ∥ 2 2 in the inequality above.

Theorem 2. Under the condition that assumptions 1, 2, 3, we choose κ = L µ , γ = max { 8 κ, E } -1 , η t = µ 2( t + γ ) and B = 2( √ dδ +1+ d 2 ) G 2 +16 E 2 G 2 (2 √ dδ +3) + 1 N 2 ∑ K k =1 σ 2 k +6 L Γ , C = N -K N -1 8 K E 2 G 2 (2 √ dδ + 3) + dG 2 . When t satisfying δ 2 ≤ η 2 t G 2 , FedAvg with quantization and partial device participation satisfies:

Proof. We have:

We take the expectation for the above formula, G 3 vanishes according to lemma 5.

If t +1 / ∈ I E , according to our parameter update settings, G 1 vanishes. We use lemma 1, 2, 3, 4 to bound G 2 :

If t +1 ∈ I E , We use lemma 6 to bound G 1 , lemma 1, 2, 3, 4 to bound G 2 , then we have :

We use exactly the same method as in Theorem 1 and then we have:

C Hyperparameters Setting

We will introduce the four FL methods, ABAvg, FedProx, FedGen, FedFTG, and their hyperparameters setting in our experiments to prove our method's effectiveness.

ABAvg [35] is a data-dependent distillation FL method, which uses a validation dataset D v to reweight each collected devices. In our experiment, we set the test dataset as the validation dataset. For FashionMNIST, CIFAR10, CIFAR100, we use the whole test dataset, and for CINIC10, we choose the 20% data of each label.

FedProx [22] is an FL method with regularization when training locally. We set the FedProx proximal term µ = 0 . 1 in our experiment.

FedGen [42] is a data-free distillation FL method, which maintain a generator in the server to generate hidden variables, and transfer the generator to the device side for data enhancement. We set the generator learning rate to 3 e -4 , hidden dimension to 32. The generator's training epoch is 5.

FedFTG [40] is a data-free distillation FL method, which maintain a generator on the server for fine-tuning aggregated server parameters. We set the generator learning rate to 1 e -2 , the fine-tuning learning rate to 1 e -4 . The totally fine-tune epoch is set to 10, inner epoch for generator is 1, inner epoch for server parameter is 5.

All of our models are trained on s GeForce RTX 4090. We found only very small differences when evaluating on these other hardware platforms.

D Low Precision Training with Other FL Methods

We put the four algorithms used here together with Efficient FL. Judging from the Table 4, lowprecision local training is sufficient in FL system. We put more emphasis on showing the case of α = 0 . 01 , because this case is more consistent with the actual situation of FL, that is, the data heterogeneity of edge devices. Combining four FL methods and four datasets, in the case of α = 0 . 01 , Figure 4, we can find the fact that our algorithm can maintain or even surpass the accuracy of the original algorithm on 8 bit, and at the same time, it can do the same on 6 bit to maintain accuracy.

We also place the algorithms used here, and these pseudocodes are added to our methods. When we conduct experiments, we also follow the algorithm flow from Algorithm 3 to Algorithm 6. We found that although different FL methods will behave differently when the client updates or the server updates, our method can be easily applied to them.

E Efficient FL Detail

We list the detailed results of the Efficient FL used in experiments. We use the same network structure ConvNet and the CIFAR10 split in α = 0 . 01 to conduct experiments. In HeteroFL, a, b, c, d, and e respectively represent 100% , 50% , 25% , 12 . 5% , and 6 . 25% channel proportions. In Table 5, for example, a1-b1-c1 means that during local training, a, b, c, the three models account for 1:1:1. In SplitMix, a, b, c, d, e, have the same representation. 1/8 and 1/16 of SplitMix represent the minimum model channel ratio after segmentation. We conducted experiments on 1/8 and 1/16 respectively.

We calculated the average memory consumption of a single device during each communication round, for each compression method, and also for our method. For memory consumption statistics, we

record model size, gradient storage, and intermediate activation storage. Table 5 to Table 8 contains all our experimental results comparing memory savings.

(a) FMNIST

(b) CIFAR10

(c) CINIC10

(d) CIFAR100

0.80

0.50

0.40

0.20

Origin

Origin

Origin

Origin

0.20

8bit w avg

6 bit w avg

0.10

8 bit w avg

6bit w avg

0.10

8 bit w avg

6 bit w avg

0.05

6bit w avg

8 bit w avg

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

0.80

0.50

0.40

0.20

.15

Origin

Origin

Origin

Origin

0.20

8 bit w avg

6 bit w avg

0.10

6 bit w avg

8bit w avg

0.10

8bit w avg

6 bit w avg

0.05

8bit w avg

6 bit w avg

50

100

150

200

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

0.80

0.50

0.40

0.20

00.60

Origin

Origin

Origin

Origin

0.20

6 bit w avg

8bit w avg

0.10

8bit w avg

6 bit w avg

0.10

8bit w avg

6bit wavg

8 bit w avg

6 bit w avg

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

0.80

0.50

0.40

0.20

0.20

8 bit w avg

6 bit w avg

Origin

0.10

8 bit w avg

6 bit w avg

Origin

0.10

Origin

8 bit w avg

6 bit w avg

S0'0

Origin

8 bit w avg

6 bit w avg

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

F Limitations

In fact, what we performed in the experiment was fake quantization, that is, simulated quantization, which means we only focused on the impact of quantization on accuracy. The actual quantization operation requires the cooperation of hardware, and the hardware GPU we used was GeForce RTX 4090, which could not directly perform low precision training, and could not reflect the acceleration effect of our method.

G Broader Impacts

FL, while having a positive impact in areas such as data privacy protection and cross-domain collaboration, also faces some potential negative impacts and challenges.

First, in FL, attackers might attempt to implant backdoors during the model training process. By introducing specific triggers into the training data, attackers can activate the backdoor after the model is deployed, causing the model to output results that the attacker desires. This type of attack poses a serious threat to the model's security and needs to be guarded against with strict security measures and auditing processes.

Second, attackers could disrupt the performance of the model by injecting incorrect information into the training data. This data poisoning can occur at the client side or during the model aggregation process. Data poisoning attacks may lead to the model behaving abnormally under certain conditions, affecting the model's reliability and accuracy.

Last, although FL is designed to protect user privacy by training models locally and sharing only model updates rather than raw data, this method is not foolproof. Attackers might infer sensitive information about the training data by analyzing the model's update information, such as gradients or weight changes. This kind of privacy leakage can be mitigated with techniques like differential privacy, which may impact the model's performance.

Algorithm 3 Low Precision FL with ABAvg

Algorithm 4 Low Precision FL with FedProx

; L layers DNN

Algorithm 5 Low Precision FL with FedGen

Algorithm 6 Low Precision FL with FedFTG

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The algorithms mentioned in the introduction and abstract can be found in Section 4 of the article. Theoretical result can be founded in section 5. The convergence proof of our algorithm is located in appendix A and B.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations can be found in appendix F.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Theoretical result and its assumptions can be founded in section 5. The proof of theoretical result located in appendix A and B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have showed the detail hyperparameters configurations in Section 6 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have added the code to the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In the configuration of Section 6, we introduced the hyperparameters and model settings, most of which are adopted from previous work.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In Section 6, we specified that the average of the last five rounds was used as the accuracy, and the standard deviation was calculated. When calculating the training loss of the clients, we also introduced the use of 9 clients for separate experiments to calculate the mean and standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The hardware we use is mentioned in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the Code of Ethics carefully and we preserve anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Broader Impacts can be founded in appendix G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our study does not carry these risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We used public data and models properly under the license and terms, which were also properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We didn't release some new assets in this work.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our experiments didn't include the crowdsourcing and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our experiments didn't include the crowdsourcing and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes an efficient federated learning (FL) paradigm, where the local models in the clients are trained with low-precision operations and communicated with the server in low precision format, while only the model aggregation in the server is performed with high-precision computation. The performance is comparable to full-precision training, and sometimes even better since the over-fitting issue in local training is relieved.

STRENGTHS:
S1. The idea of applying SWALP's low-precision training within each cycle for local training in FL is meaningful and effective.
S2. There are theoretical analysis on convergence.
S3. Experiments are comprehensive and demonstrate the effectiveness of the proposed method.

WEAKNESSES:
W1. The biggest concern is regarding the novelty w.r.t. SWALP [40]. The entire Sec 3.2 and Sec 4.1 are almost the same as in [40]. The only difference seems to be Sec 4.2 but still very similar to the idea of SWA, just the setting of aggregation changes from by cycles to by clients, and a moving average of parameters is used.

W2. Writing needs improvement. For example, there is a typo "sever" in Line 108.
Eq (5) is different from its counterpart in [40] where the power was F-1 but now W-F-1. please explain the reason of difference.
Line 135, missing a space before "i.e."
Eq(7) uses E which is not clear until continuing reading to Line 161 and Algorithm 2 Lines 3-4.
Algorithm 2 Line 11, t' is only briefly mentioned in Line 161 without even referring to the used lines.

W3. It would be good to estimate the time reduction with the professional hardware (real acceleration).

QUESTIONS:
Justify method novelty against [40] (W1) and answer related questions in W2.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a federated learning approach that performs local training on low precision through quantization combined with a high-precision averaging and a moving average at the server. The paper guarantees convergence and empirically compares several levels of low-precision local training to full-precision training on 4 baseline FL methods. It remains unclear, though, what the contribution of the method is: the main focus seems to be on performance in terms of test accuracy, but the experiments do not show a significant improvement over existing methods. The method supposedly improves communication and computation efficiency but is not empirically compared to state-of-the-art methods, such as [1,2,3].\n\nIn their rebuttal, the authors provided novel results that address my concerns about missing baselines. While I remain concerned about the limited novelty and the presentation, I believe that the authors will be able to address these issues to some extent in the next version of the manuscript. Therefore, I have decided to increase my score.\n\n\\\n[1] Liu, Shiyu, et al. \"Quantized SGD in Federated Learning: Communication, Optimization and Generalization.\" International Conference on Neural Information Processing. Singapore: Springer Nature Singapore, 2023.\n\n[2] Kamp, Michael, et al. \"Efficient decentralized deep learning by dynamic model averaging.\" Machine Learning and Knowledge Discovery in Databases: ECML PKDD, 2018.\n\n[3] Reisizadeh, Amirhossein, et al. \"Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization.\" International conference on artificial intelligence and statistics. PMLR, 2020.\n\nSTRENGTHS:\n- convergence guarantees\n- clear explanation of the method\n\nWEAKNESSES:\n- the presentation should be improved (e.g., several typos, such ass \"accept\" instead of \"aspect\", the term \"a avg\" for \"with moving average\" is unintuitive)\n- the ablation study does not clearly show that low precision local training improves performance, since it is combined with a moving average that has a strong positive impact on performance.\n- lack of comparison to baselines\n- unclear use-case for the method\n- the paper does not discuss existing federated learning with quantization literature in sufficient detail.\n- for the non-iid experiments it would be interesting how quantization interacts with local BN layers in FedBN [4]\n\n\\\n[4] Li, Xiaoxiao, et al. \"FedBN: Federated Learning on Non-IID Features via Local Batch Normalization.\" International Conference on Learning Representations, 2021.\n\nQUESTIONS:\n- It is unclear from the results how much of the benefit stems from quantization and how much from the moving average. Is it correct that the moving average has a very large positive effect regardless of quantization?\n\nLIMITATIONS:\nSection F in the appendix addresses the limitation of simulating quantization. It does not address issues like computation and communication complexity, the applicability of the approach, or limitations in the empirical evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an FL system with data heterogeneity, a topic has been extensively studied in the past few years. The idea is to perform local training with lower precision through applying block floating point quantization. The idea per se is not new, but proving that the convergence can be achieved using low precision local training is an interesting contribution.\n\nSTRENGTHS:\nThe paper is well written and is easy to follow. The idea is also interesting but it is not necessarily new. The most important contribution is the theoretical proof for convergence. The evaluation results in Section 6 validate the theoretical results.\n\nWEAKNESSES:\nThe paper mainly focuses on data heterogeneity in FL systems. What about resource heterogeneity? It would be important to have a discussion (or better some experimental results) on how the proposed solution perform in such setting. Should we use the same quantization level for all clients, or can we adjust the precision according to the resource availability? Also, the current state of the art of quantization in conjunction with Federated Learning is also missing, e.g.:\n\n- FedQNN[1] uses quantized training in FL.\n\n- CoCoFL[2] uses a combination of quantization and freezing for heterogeneous resources in FL.\n\nHow does these SOTA techniques perform compared with the proposed solutions?\n\n[1] Y. Ji and L. Chen, \"FedQNN: A Computation–Communication-Efficient Federated Learning Framework for IoT With Low-Bitwidth Neural Network Quantization,\" in IEEE Internet of Things Journal.\n  \n[2] Kilian Pfeiffer, et al. \"CoCoFL: Communication-and Computation-Aware Federated Learning via Partial NN Freezing and Quantization.\" Transactions on Machine Learning Research., 2023\n\nQUESTIONS:\nPlease also check my questions in the weakness section.\n\nLIMITATIONS:\nAppendix F clarifies that the fake quantization is applied in the experiments. I really appreciate learning this information, as one of my question was how the quantization is implemented in PyTorch (as PyTorch only supports int8 inference out of the box). \nRegarding the broader impact, the discussion is about FL in general. The question is if performing lower precision training at clients improves or worsen these privacy and security aspects (e.g. if data poisoning cab be done easier, as the local updates are low precision).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an efficient Federated Learning (FL) paradigm where local models are trained using low-precision operations and communicated with the central server in low precision format. The aggregation on the server, however, is performed with high-precision computation to ensure accuracy. The authors demonstrate that high-precision models can be recovered from low-precision local models with proper aggregation on the server side. This approach significantly reduces the computational load on client devices and the communication cost. The method is theoretically proven to converge to an optimal solution, even with non-IID data distributions, and extensive experiments show that models trained with low precision (as low as 8 bits) are comparable in performance to those trained with full precision.\n\nSTRENGTHS:\n1. The proposed method reduces the computational and communication overhead for client devices, which is crucial for resource-constrained environments for large models. The paper also provides theoretical guarantees for convergence to the optimal solution, even with non-IID data distributions.\n2. The method is effective on the datasets and models in the experiments where low precision training has little to no impact on utility.\n\nWEAKNESSES:\n1. The integration of low precision training and high precision aggregation may add complexity to the implementation.The performance improvements are partly dependent on the hardware capabilities, such as the availability of processors supporting low precision operations.\n2. The experiments are limited. Only image datasets are considered. Evaluation on other types of data, like text or tabular, can strengthen the results. \n3. No integration with differential privacy or other privacy protection mechanisms. Federated learning itself is not private and it would be interesting to see what privacy mechanisms are suitable for low precision model updates.\n\nQUESTIONS:\n1. How can we determine the optimal precision level for local training without extensive hyperparameter tuning which is expensive in FL?\n2. Have you explored mixed precision strategy? E.g. using different quantization schema for gradients, activations etc.\n\nLIMITATIONS:\nThe authors have discussed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-20"}