Low Precision Local Training is Enough for Federated Learning

Authors: zhiwei  li
yiqiu  li
binbin  lin
zhongming  jin
weizhong  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 19:09:48

Abstract: Federated Learning (FL) is a prevalent machine learning paradigm designed to address challenges posed by heterogeneous client data while preserving data privacy.
     Unlike distributed training, it typically orchestrates resource-constrained edge devices to communicate via a low-bandwidth communication network with a central server.  This urges the development of more computation and communication efficient training algorithms. In this paper, we propose an efficient FL paradigm,  where the local  models in the clients  are trained with  low-precision operations and communicated  with the server in low precision format, while only the model aggregation in the server is performed with  high-precision computation. We surprisingly find that   high precision models can be recovered from the  low precision local models with proper aggregation in the server. 
     In this way, both the workload in the client-side and the communication cost can be significantly reduced.   We theoretically show that our proposed  paradigm can converge to the optimal solution as the training goes on, which demonstrates that low precision local training is enough for FL.  Our paradigm can be integrated with existing FL algorithms flexibly. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. Notably, the models trained by our method with the precision as low as 8 bits are  comparable  to those from the  full precision training. As a by-product, we show that low precision local training can relieve the over-fitting issue in local training, which under heterogeneous client data  can cause the client models drift further away from each other and lead to the failure in  model aggregation.

Main paper: 
 introduction: 
 Federated learning (FL) [3, 15, 22, 36] is a popular privacy preserving machine learning paradigm to collaboratively learn a global model over the decentralized data. In FL paradigm, the clients are responsible for local training and only have access to their private datasets, while the server plays an essential role in aggregating the clients' updates into a global model. Unlike large-scaled distributed training, FL typically orchestrates resource-constrained edge devices to communicate via a low-bandwidth communication network with a central server. This urges the development of more computation and communication efficient optimization algorithms. The most prevalent approach in FL is developed based on local-SGD [26] , which is referred to as FedAvg. In each communication round, the clients individually train their local models for multiple steps and then send them to the server for aggregation. It can be expected that if longer local training process one uses, the greater communication cost saving one can achieve. However, long time local training could cause the local models drift further away from each other and degrades the aggregated global model's performance or even make the training diverge, especially when the data on the clients are heterogeneous. Therefore, in order to prolong local training processes in FL, extensive efforts have been made in the recent years. For example, the studies [1, 16, 21, 22] modify the local training process by imposing regularization on the client models to enforce them not to drift away from the previous global model. Another line of research [5, 6, 24, 31, 35, 38] focuses on refining the global model in the server aggregation process. These methods typically require a large proxy dataset on the server. Some of them [5, 6, 24] use it to align the outputs of the global model with that of the client ensemble by knowledge distillation. Others develop handcrafted aggregation rules to reweight the updates based on the statistics of updates or performance on proxy data [31, 35, 38] or further tune the global model with proxy data in every communication round [5, 24] . Although promising experimental results have been reported in the literature, it is still unclear that whether there exists more concise and effective FL paradigm, which can reduce both the workload in the client-side and the communication cost.In this paper, we propose a concise and efficient federated learning paradigm, where the local models in the clients are trained with low precision operations and communicated with the server in low precision format, while only the server-side information integration maintains high-precision computation to ensure the accuracy. Our basic idea is inspired from the Kolmogorov's law [10] , that is, the sample average can converge almost surely to the expected value although the samples always contain noise. Therefore, in the server side, we perform the simple moving average on the received low precision models from the clients to recover a high precision global model. In this way, both the workload in the client-side and the communication cost can be significantly reduced. We theoretically proved that our proposed paradigm integrated with FedAVG can converge to the optimal solution as the training goes on, which indicates that low precision local training is enough for federated learning. We extend our method to various existing FL method to show its flexibility. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. Notably, the models trained by our method with the precision as low as 8 bits are comparable to those from the full precision federated learning. Compared with some efficient FL designs, our method can achieve significant savings in training memory overhead, and what is more attractive is that our accuracy performance is even better. Our method exhibits another appealing feature in relieving the over-fitting issue in local training. To be precise, in the local training steps, the models can be easily trained to over-fit the local training data as the local dataset is always insufficient. Under heterogeneous client data, it would further cause the client models drift further away from each other and lead to the failure in model aggregation. The experimental results show that our approach can effectively relieve the over-fitting issue since the local training is performed with low precision computation and the expressiveness of the local model is restricted.Our main contributions are as follows:• We propose an efficient federated learning paradigm that performs low precision computation during local training, saving computational overhead and communication costs, while being able to restore accuracy through high-precision aggregation on the server side. • We theoretically proved that the efficient federated learning algorithm we proposed can achieve convergence at a rated of O(1/T ) under certain assumptions for non-iid situations. • Since the expressiveness of the local model is restricted due to the low precision local training, our approach can relieve the over-fitting issue, which would cause the client models drift further away from each other and lead to the failure in model aggregation when the local dataset are heterogeneous. • The extensive experimental results demonstrate the effectiveness of our approach. Notably, the models trained by our method with the precision as low as 8 bits are comparable to those from the full precision federated learning.FedProx [22] utilizes a regularization term while Scaffold [16] sets a control variate to reduce the drift in local training. FedGen [42] and FedFTG [40] maintain a generator, the former is used for local data augmentation, while the latter is used for fine-tuning the server.Efficient Federated Learning. One challenge of FL is the limitation of low bandwidth and computing resources of client devices. [4, 20] assign each client a block mask, resulting in sparse local models. [28] took the transmission speed into consideration and chose the same method as [12] for uploading, uploading compressed gradients. [13] adopts boost training to client-side training overhead. [7, 28] only transmit the trained head to reduce transmission cost. [33] adopts the idea of Network Architecture Searching, e.g., each client selects a sub-network. [14] maintains a series of streamlined models in the server, from which the client selects a tiny model for training. In [9] , the client selects a sub-model of the global model for training. In this paper, we address this issue by using low precision local training.3 Preliminary 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations can be found in appendix F.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
federated learning: 
 Given N clients with their private datasets D k = {(x k,j , y k,j )}|D k |j=1 , k = 1, . . . , N , the optimization objective of FL is always defined as follows:EQUATIONwhere w ∈ R d represents the model parameters, FL emphasizes data privacy protection and thus the server is not allowed to access these datasets D k directly in model training. The standard method to solve the above training problem of FL is FedAvg [26] , which is developed based on local SGD. It is comprised by two steps, i.e., local training on the clients and model aggregation in the server side. The details are presented below.• In local training, the central server would first randomly select partial clients and broadcasts the latest global model w t to them. We denote the selected clients set as S t and let K = |S t | be the number of selected clients. Then the client k with k ∈ S t would initialize its local model to be w k t = w t and then performs local training with E(≥ 1) iterations as follows:EQUATIONwhere w k t is the weights of the k-th client in step t, ξ k t is a mini-batch of samples uniformly chosen from D k , η t is the step size.• In model aggregation, FedAvg updates the global model to be the weighted average of the received local models, i.e.,EQUATIONwhere q t = k∈St p k normalize the coefficients.Fixed point quantization is a standard quantization technique. It uses stochastic rounding to round the numbers up or down at random such that E[Q(x)] = x, where Q : R → R is the quantization function defined asEQUATIONhere clip(x, a, b) = max(min(x, b), a), δ = 2 -F is the quantization gap represents the distance between successive representable fixed point numbers, u = 2 W -F -1 -2 -F and l = -2 W -F -1 represent the upper and lower limits of the representable numbers, respectively. W is the bit width of quantized numbers, and F is the bit width of quantized numbers' fractional part. In order to improve the utilization efficiency of the bit width and better maintain numerical accuracy when data is unevenly distributed, we choose block floating point quantization. Given a block of numbers X, it replaces δ = 2 -F in fixed point quantization to be δ = 2 -E(X)+W -2 , whereEQUATIONThe shared exponent E(X) is usually set to be the largest exponent in X to avoid overflow [37] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We have added the code to the supplementary materials.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
method: 
 In this section, we introduce our low precision federated learning paradigm. It is comprised of two modules,i.e., one is the low precision local training to reduce the computation and communication cost, the other is the high precision aggregation with moving average to maintain the model accuracy.In order to reduce the computation and communication EQUATIONNote that in the above version, we only quantize the updated parameters. We give this version just for the convenience of the theoretical analysis in Section 5. In practice, we quantize the gradient, the activation of each layer, the back-propagation signals, and the momentum in SGD when SGD is adopted as the optimizer. The details are given in Algorithm 1.Input: Quantization functions Q A , Q E , Q G , Q M , Q W ; Momentum coefficient ρ; L layers DNN {f 1 , f 2 , . . . , f L }; Loss function ℓ. 1: ClientUpdate(t, k, w k t ): 2:Get batch (x k,jt , y k,jt ) from D kForward Propagation:4:EQUATION:(a k t ) (l) = Q A (f l ((a k t ) (l-1) , (w k t ) (l) )), ∀l ∈ [1, L] 6:Backward Propagation:7:(e k t ) (L) = ∇ (a k t ) (L) ℓ((a k t ) (L) , y k,jt ) 8: (e k t ) (l-1) = Q E ( ∂f l ((a k t ) (l-1) ,(w k t ) (l) ) ∂(a k t ) (l-1) (e k t ) (l) ), ∀l ∈ [1, L] 9: (g k t ) (l) = Q G ( ∂f l ((a k t ) (l-1) ,(w k t ) (l) ) ∂(w k t ) (l) (e k t ) (l) ), ∀l ∈ [1, L] 10:Low Precision SGD Update:11:(v k t+1 ) (l) ← Q M (ρ(v k t ) (l) + (g k t ) (l) ), ∀l ∈ [1, L] 12: (w k t+1 ) (l) ← Q W ((w k t ) (l) -η t • (v k t+1 ) (l) ), ∀l ∈ [1, L] 13: Return: w k t+1Although low precision training can reduce communication and training overhead, it would lead to a degradation in training accuracy. Inspired from the Kolmogorov's law [10] , that is, the sample average can converge almost surely to the expected value although the samples always contain noise, we try to recover high-precision solution from the low-precision local model with a full precision aggregation process. It is implemented with the following two steps:• Calculate the weighted average of the local models to partially reconvert the precision, i.e.,EQUATION• Since in most cases clients' data is non-iid and quantization causes error, federated learning is harder to converge, however maintaining a moving average in the server can significantly alleviate the problem. Formally, we denote wt as the moving average stored in the server, then after aggregation, we update wt as followEQUATIONwhere λ is a coefficient controlling the influence of current weight. This procedure can further compensate the accuracy degradation due to the low precision local training.In the next round of local training, the quantized model Q( wt ) will be distributed to the clients to intilize the local models. Our pseudocode in Algorithm 2 depicts the process of low precision local training on the client device and high precision aggregation on the server. In Algorithm 2, t ′ = t -E + 1 and I E = {nE|n = 1, 2w k t ← Q( wt ), k ∈ S t 6: end if 7:for k ∈ S t do 8:w k t+1 ← Q(w k t -η t ∇F k (w k t ; ξ k t )) ▷ Client update 9:end for 10:if t + 1 ∈ I E then 11: w t+1 ← k∈S t ′ p k q t ′ w k t+1 ▷ Server update 12: wt+1 ← λ wt ′ + (1 -λ)w t+1 13:end if 14: end for 15: Return: wT 
theoretical analysis: 
 In this section, we give the detailed theoretical results for our low precision FL paradigm. We will first introduce the convergence analysis in the full participation case where all client devices participate (i.e., K = N ) and then we generalize the results of the analysis to scenarios that are more in line with reality. (i.e., K < N ). The results demonstrate that we will explore aggregation strategies represented by the FederatedAveraging Algorithm (or FedAvg) and demonstrate that our proposed low precision FL framework can converge to the global optimal solution at a rate of O(1/T ) for non-iid datasets based on strong convexity and smoothness assumptions.We need to make necessary assumptions about the objective function on the clients F k , k = 1, . . . , N . Assumption 1 is about the smoothness and strong convexity of the loss function and Assumption 2 is about the boundness of the gradients. These assumptions are standard and widely adopted in the related studies [41, 29, 30, 39] . Assumption 1. F 1 , F 2 , ..., F N are L-smooth and µ-strongly functions, which means that for all v and w, the following inequalities hold:EQUATIONEQUATIONwhere ∥•∥ 2 represents the square of two norms and k = 1, . . . , N .Assumption 2. Let ξ k t be sample that randomly and uniformly sampled from the local data of the k-th client device. For k = 1, 2, • • • , N and t = 0, 1, • • • , the variance of stochastic gradients in each client device and the expectation of squared two norm of stochastic gradients is bounded:EQUATIONEQUATION)Degree of Data Heterogeneity. Let F * denote the global minimum of the objective function F , and let F * k represent the minimum of the local objective function F k specific to the k-th client. We use the metric Γ [23] taking the form ofΓ = F * - N k=1 p k F * k ,to measure the degree of heterogeneity of all clients' data distribution. When the data on each client device is iid, as the number of samples increase, Γ evidently tends towards zero. However, when faced with non-iid situation, Γ tends towards a positive constant and thus it can be used to measure the degree of heterogeneity in the data distribution of each client device.First we analyze the convergence of the participation of all clients' device in this section. We integrate our low precision FL framework with FedAvg and train the model for T iterations to obtain the wT , and we expect T to be divided by E so that wT is the weight after aggregation.Theorem 1. Under the Assumptions 1 and 2, we set κ = L µ , γ = max{8κ, E} -1 and η t = µ 2(t+γ) . When t satisfying δ 2 ≤ η 2 t G 2 , low precision FedAvg with full device participation satisfies:EQUATIONwhere B = 2(EQUATION)Compared to full participation situation, the partial participation situation is more in line with the reality. We need to make more assumption on how to choose S t . Assumption 3. Assume S t contains a subset of K indices uniformly sampled from [N ] without replacement. In addition, the data is balanced in the sence thatp 1 = • • • = p N = 1 N . The aggregation step of FedAvg performs w t+E ← 1 K k∈St w k t+E Theorem 2. Under the Assumptions 1 to 3, we choose κ = L µ , γ = max{8κ, E}-1, η t = µ 2(t+γ) and B = 2( √ dδ +1+ d 2 )G 2 +16E 2 G 2 (2 √ dδ +3)+ 1 N 2 N k=1 σ k 2 +6LΓ, C = N -K N -1 8 K E 2 G 2 (2 √ dδ + 3) + dG 2 . When t satisfying δ 2 ≤ η 2 t G 2, low precision FedAvg with partial client device participation satisfies:EQUATION)6 ExperimentsIn this section, we conduct extensive experiments to verify the effectiveness of our methods in the following five aspects:• When integrated with FedAvg, the models trained by our method with the low precision are comparable to (if not better than) those from the full precision training. This would also verify our theoretical results (Section 6.1). • Our method can effectively relieve the over-fitting issue in FL. See Section (Section 6.2).• Our paradigm can be integrated with existing FL methods flexibly and preserve the performance even with a low precision local training (Section 6.3).• Ablation studies on the effectiveness of moving average in model aggregation and the transferability over various neural networks (Section 6.4).• Comparison with other efficient FL techniques (Section 6.5). Remark 1. Similar with the existing low precision training studies [32] , We do not give the results on the running time to show the real acceleration. The reason is that to achieve real acceleration, we need to implement our method integrated with the professional hardware. Moreover, such implementation is standard for the professional hardware platforms.Benchmark Datasets and Baseline. We conduct experiments over four commonly used datasets: FashionMnist [34] , CIFAR10 [19] , CIFAR100 [19] and CINIC10 [8] . Four commonly used FL methods: 1) FedAvg [26] ; 2) regularization-based strategy FedProx [22] ; 3) data-dependent knowledge distillation strategy ABAvg [35] 4) data-free knowledge distillation strategy FedFTG [40] and FedGen [42] are adopted as the baselines.Configurations. We follow the configurations in the recent studies [26, 42, 27] for fair comparison. To be precise, for FashionMNIST, CIFAR10, CINIC10 and CIFAR100, we run 200 communication rounds with local epoch set to 1. There are 80 clients in total, and the participation ratio in each round is set to 40%. We use Dirichlet distribution to simulate non-iid data distribution and set α to 0.01, 0.04, and 0.16. The smaller α is, the more serious the data heterogeneity is. For the network choice, we use ConvNet following with 3 layers, and the hidden dimension is set to 128. The local learning rate is set to 10 -3 with Adam optimizer [17] . We report the last 5 round global model's average performance evaluated using the test split of the datasets. For quantization method, we adopt the Block FLoating Point Quantization with the number of bits used set to 6, 8 and 32 (without quantization). Some of the other hyperparameter settings are included in the Appendix C. 
results on fedavg: 
 We demonstrate the superior performance of our Low Precision FL method with FedAvg by conducting experiments over comprehensive datasets, various precision and heterogeneity values α.Heterogeneity. As shown in Table 1 , it is as expected that when the heterogeneity goes higher, that is, when α decreases, the server performance worsens. Nevertheless, our proposed method can always maintain or improve the performance of the original case (bits = 32, w/o. avg) when using 8 quantizaiton bits with moving average, which empirically validates the effectiveness of our proposed method.Quantization Bits. We conduct experiments on 3 quantization bits: 32, 8, 6 (shown in Table 1 , Figure 1 ), as we observe that in most cases 8 bits is enough to hold the performance of full-precision and when the used bits is 6, the server performance begins to decrease due to the low precision level.Table 1 : Results of our method integrated with FedAvg over various levels of heterogeneity and precision. The results with moving average demonstrate that our method can match the performance of full-precision federated learning even with all numbers quantized down to 8 bits. The results in the bottom three rows indicates the without moving average, training with low precision would lead to performance degradation. We can observe that the local training loss of FedAvg (full precision) is siginificantly lower than our method, however its global test loss is much higher than us and fluctuates dramatically.The four FL methods we used are each representative. ABAvg and FedFTG are similar to FedAvg in local training, but the former only performs weight adjustment on the server side, while the latter uses knowledge distillation to fine-tune the server. FedProx and FedGen are similar to FedAvg in the server side, but the former only has regularization constraints on local training, while the latter uses the generator for regularization adjustment in local training. As is demonstrated in Table 2 , we can see that, regardless of the FL method chosen, our low precision FL algorithm has a significant improvement in prediction accuracy compared to the original FL method, especially in dataset CIFAR10, CINIC10 and CIFAR100.Table 2 : The results of our method when integrated with existing FL methods ABAvg, FedProx, FedGen and FedFTG. Our method can improve the effectiveness of FL with a certain training cost, and can be applied in both regularization and distillation. The complete result is given in Table 4 .It shows that integrated with our low precision training technique, these existing methods with the precision of 8 bits can match (or even better than) the performance with full precision. Moving Average. We conduct ablation study to validate the effectiveness of the moving average for the server, from Table 1 , the average operation has a significant effect on the server performance over various settings. It is consistent of our expectation that averaging can alleviate the error introduced by quantization and the error by heterogeneity.Architecture Generalization. We use another network structure, MLP, to verify that our method can be applied to other networks. The MLP was set to 3 layers, with the hidden dimension of each layer being 128, and experiments were conducted on FashionMNIST with α of 0.01 and 0.04. From Table 3 , our method can still maintain the accuracy, which also demonstrates its generalization ability on other network structures.Table 3 : To display our method can generalize to other architectures, the same experiments were also performed on the MLP, using the FashionMNIST, setting α = 0.01 and α = 0.04. Through comparison, it is found that the accuracy under MLP is slightly lower than that of ConvNet. However, our method can still maintain or improve the accuracy on MLP. In order to demonstrate that our method can effectively reduce computational costs and memory consumption, we compared it with two personalized model compression FL algorithms. HeteroFL [9] and Split-Mix [14] can both achieve the purpose of reducing training overhead by using smaller models for local training. We use the ConvNet with the same hidden dimension to conduct experiments, using CIFAR10 in the process, α = 0.01. SplitMix uses a split ratio of 1/8 by default. We re-tested at a split ratio of 1/16 to achieve the same compression ratio as HeteroFL. In Figure 3a , the vertical dotted line indicates the compression ratio of HeteroFL, more details are in the Appendix E. Compared with methods that directly compress model parameters, our method can achieve higher accuracy with lower memory consumption. Because quantization does not bring about the deletion of model parameters, there will be no aggregation problems when the model compression rate is too high. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: In the configuration of Section 6, we introduced the hyperparameters and model settings, most of which are adopted from previous work.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this paper, we propose an efficient FL paradigm, where the local models in the clients are trained with low-precision operations and communicated with the server in low precision format, while only the model aggregation in the server is performed with high-precision computation. We theoretically show that our proposed paradigm can converge to the optimal solution as the training goes on, which demonstrates that low precision local training is enough for FL. Our paradigm can be integrated with existing FL algorithms flexibly. Experiments across extensive benchmarks are conducted to showcase the effectiveness of our proposed method. As a by-product, we show that low precision local training can relieve the over-fitting issue in local training.This appendix can be divided into several parts. To be precise:• Section A gives the detailed proof for Theorem 1.• Section B gives the detailed proof for Theorem 2. Compared to Theorem 1, Theorem 2 assumes that only a subset of client devices participate in the training, which is more in line with real-world scenarios.• Section C introduce four FL methods'(ABAvg, FedProx, FedGen, FedFTG) hyperparameters setting in our experiments.• Section D provides detailed experiment results of four FL methods(ABAvg, FedProx, FedGen, FedFTG).• Section E presents the comparsion between our low precision FL method and two efficient FL method in training overhead.• Section F gives Limitation of this paper.• Section G gives Broader Impacts of this paper.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: In Section 6, we specified that the average of the last five rounds was used as the accuracy, and the standard deviation was calculated. When calculating the training loss of the clients, we also introduced the use of 9 clients for separate experiments to calculate the mean and standard deviation.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our study does not carry these risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We used public data and models properly under the license and terms, which were also properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We didn't release some new assets in this work.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our experiments didn't include the crowdsourcing and research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our experiments didn't include the crowdsourcing and research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We analyze our low-precision federated learning algorithm in the setting of full device participation in this section. For the sake of convenience in proving, let's take the value of p k to be 1 N , disregard the moving average, assume that quantization is only applied to model parameters and quantization gap is always δ. Let I E be a set composed of the sequence number of global aggregation steps, i.e.,I E = {nE|n = 1, 2, • • • }.Suppose the model weights are updated as follows:v k t+1 = Q(w k t -η t ∇F k (w k t ; ξ k t )). w k t+1 = v k t+1 if t + 1 / ∈ I E , Q( N k=1 v k t+1 N ) if t + 1 ∈ I E .where w k t is an d-dimensional vector, which represents the weight of the k-th client device in step t. F k represents the objective function of the k-th device. N is the total number of clients device participating in the training, and T is the total number of steps trained, ξ k t represents a batch of data randomly sampled from the k-th client device's local dataset D k in step t, ∇F k (w k t ; ξ k t ) is stochastic gradient, η t represents learning rate, Q represents the quantization function, and its formula has already been given in (4) .Notations. To further simplify our proof process, we introduce the following additional definitions:w0 = w 0 , wt = 1 N N k=1 w k t , vt = 1 N N k=1 v k t , ḡt = 1 N N k=1 ∇F k (w k t ), g t = 1 N N k=1 ∇F k (w k t , ξ k t ).w t represents the weight of the global model in step t. Obviously, we can deduce Eg t = ḡt and vt+1 = wt -η t g t .Lemma 1. If w is a scalar and can be written as w = w + bδ, satisfying -δ < w < δ, δ is the quantization gap, represents the distance between successive representable fixed-point numbers, then we have:Q(w) -w = -w, w.p. 1 -sign( w) w δ , sign( w)δ -w, w.p.sign( w) w δ . where Q(•) is quantization function. And we always can find a suitable w that satisfies | w| ≤ |w| .Proof. The quantization function is:Q(w) = clip(δ⌊ w δ ⌋, l, u), w.p. ⌈ w δ ⌉ -w δ , clip(δ⌈ w δ ⌉, l, u), w.p. 1 -⌈ w δ ⌉ + w δ .When w > 0:Q(w) -w = -w, w.p. 1 -w δ , δ w, w.p. w δ . When w < 0: Q(w) -w = -δ -w, w.p. -w δ , -w, w.p. 1 -w δ . So we have: Q(w) -w = -w, w.p. 1 -sign( w) w δ , sign( w)δ -w, w.p. sign( w) w δ .Next we prove that we always can find a suitable w that satisfies | w| ≤ |w|. When w ≥ δ,we can choose a w satisfying w ≥ 0, then we can have | w| ≤ |w|. When w ≤ -δ, we can choose a w satisfying w ≤ 0, then we also can have | w| ≤ |w|. When -δ < w < δ, we can choose a w satisfying w = w, then we also can have | w| ≤ |w|. In summary, we always can find a suitable w that satisfies | w| ≤ |w|.Discussion. In the lemma 2 below, we analyze the convergence of the algorithm under the condition thatδ 2 ≤ η 2 t G 2, where G 2 is given in assumption 2 and represents the constraint on the expected value of the squared two-norm of the stochastic gradient. The right side of the inequality indicates the change in the iterative parameters during the gradient descent process, that is, the size of the gradient multiplied by the size of the learning rate. If the magnitude of this value is already less than the quantization precision, it will cause the gradient descent to fail, and the model parameters will not continue to change. In this case, it makes no sense to continue optimization, so analyzing convergence under this condition is practical.Lemma 2. We assume that η t ≤ 1 4L , based on assumption 1 and 2, when t satisfying δ 2 ≤ η 2 t G 2 we have:E ∥ wt+1 -w * ∥ 2 2 ≤ 2( √ dδ + 1 + d 2 )η 2 t G 2 + (1 -η t µ)E ∥ wt -w * ∥ 2 2 + 6Lη 2 t Γ + 2 N N K=1 wt -w k t 2 2 + Eη 2 t ∥g t -ḡt ∥ 2 2 .Proof. When t + 1 / ∈ I E , we can easily derive that wt+1 = vt+1 . So we have:∥ wt+1 -w * ∥ 2 2 = ∥v t+1 -w * ∥ 2 2 = 1 N N k=1 Q(w k t -η t ∇F k (w k t , ξ k t )) -w * 2 2 = wt -η t g t -w * + 1 N N k=1 Q(w k t -η t ∇F k (w k t , ξ k t )) - 1 N N k=1 (w k t -η t ∇F k (w k t , ξ k t )) 2 2 = ∥ wt -η t g t -w * ∥ 2 2 A1 + 1 N N k=1 (Q(w k t -η t ∇F k (w k t , ξ k t )) -(w k t -η t ∇F k (w k t , ξ k t ))) 2 2 A2 + 2⟨ wt -η t g t -w * , 1 N N k=1 (Q(w k t -η t ∇F k (w k t , ξ k t )) -(w k t -η t ∇F k (w k t , ξ k t )))⟩ A3 .⟨•⟩represents inner product operation. Due to EQ(x) = x, according to the Law of iterated expectations, EA 3 = 0. Next we aim to bound A 2 . EA 2 is:EQUATIONIn the above equation, we used the law of iterated expectations.The quantization function is:Q(w) = clip(δ⌊ w δ ⌋, l, u), w.p. ⌈ w δ ⌉ -w δ , clip(δ⌈ w δ ⌉, l, u), w.p. 1 -⌈ w δ ⌉ + w δ . w kt is a quantified vector, therefore, its p-th component (w k t ) p = c δ, c is an integer. We rewriteη t ∇F k (w k t , ξ k t ) p = bδ + w.Where b is an integer and | w| ≤ |w|. So that we can have:[Q(w k t -η t ∇F k (w k t , ξ k t )) -(w k t -η t ∇F k (w k t , ξ k t ))] p = -sign( w)δ + w w.p. sign( w) w δ , w w.p. 1 -sign( w) w δ .The expectation after the square is:EQUATION2 . In the above inequality, we use the inequality∥•∥ 1 ≤ √ d ∥•∥ 2 2, d is the dimension of the vector. According to assumption 2 we can get:E∥ wt+1 -w * ∥ 2 2 ≤ 2( √ dδ + 1)η t 2 G 2 + E∥ wt -η t g t -w * ∥ 2 2 . We next aim to bound A 1 : ∥ wt -η t g t -w * ∥ 2 2 = ∥ wt -η t g t -w * -η t ḡt + η t ḡt ∥ 2 2 = ∥ wt -η t ḡt -w * ∥ 2 2 B1 + η t 2 ∥g t -ḡt ∥ 2 2 B2 +2 η t ⟨ wt -η t ḡt -w * , g t -ḡt ⟩ B3 .Obviously EB 3 = 0 , so we can get:E∥ wt -η t g t -w * ∥ 2 2 = E∥ wt -η t ḡt -w * ∥ 2 2 + Eη t 2 ∥g t -ḡt ∥ 2 2 .We next aim to bound B 1 :∥ wt -η t ḡt -w * ∥ 2 2 = ∥ wt -w * ∥ 2 2 + η t 2 ∥ḡ t ∥ 2 2 -2η t ⟨ wt -w * , ḡt ⟩ = ∥ wt -w * ∥ 2 2 + η t 2 ∥ḡ t ∥ 2 2 C1 -2η t 1 N N k=1 ⟨ wt -w t k , ∇F k (w k t )⟩ C2 -2η t 1 N N k=1 ⟨w t k -w * , ∇F k (w k t )⟩ C3 .To bound C 1 , C 2 , C 3 , we need to derive several inequalities using the properties ofF 1 , F 2 , • • • , F N . First of all, for k = 1, 2, • • • , N , F k is a L-smooth function, we can get: F k * ≤ F k (x) ≤ F k (w k t ) + ∇F k (w k t ), x -w k t + L 2 x -w k t 2 2 .Specifically, when x = w k t -a∇F k (w k t ), we can get:F k * ≤ F k (w k t ) + ( L 2 a 2 -a) ∇F k (w k t ) -∇F * k 2 2 .when a = 1 L , we can get:EQUATIONAccording to the above equation, we can get:EQUATIONIn the inequality above, we have utilized inequality (16) and the property thatEQUATION)By the inequality:EQUATIONwe can get:EQUATIONEQUATIONIn the inequality above, we have utilized inequality (16) again. Now, we can use inequality ( 16), ( 17), (18) , (20) to bound C 1 , C 2 , C 3 to bound B 1 : ∥ wt -η t ḡt -w * ∥ = ∥ wt -w * ∥ 2 2 + η t 2 ∥ḡ t ∥ 2 2 -2η t ⟨ wt -w * , ḡt ⟩ = ∥ wt -w * ∥ 2 2 + η t 2 ∥ḡ t ∥ 2 2 -2η t 1 N N k=1 wt -w t k + w t k -w * , ∇F k (w k t ) ≤ ∥ wt -w * ∥ 2 2 + 2Lη t 2 N k=1 p k (F k (w k t ) -F k * ) -2η t 1 N N k=1 wt -w t k , ∇F k (w k t ) + 2η t N k=1 - 1 N w t k -w * , ∇F k (w k t ) = ∥ wt -w * ∥ 2 2 + 4Lη t 2 N k=1 p k (F k (w k t ) -F k * ) + 1 N N k=1 wt -w k t 2 2 -2η t 1 N N k=1 (F k (w k t ) -F k (w * )) -η t µ 1 N N k=1 w k t -w * 2 2 ≤ ∥ wt -w * ∥ 2 2 + 4Lη t 2 N k=1 p k (F k (w k t ) -F k * ) + 1 N N k=1 wt -w k t 2 2 -2η t 1 N N k=1 (F k (w k t ) -F k (w * )) -η t µ 1 N N k=1 (w k t -w * ) 2 2 = (1 -η t µ)∥ wt -w * ∥ 2 2 + 1 N N k=1 wt -w k t 2 2 + 4Lη t 2 N k=1 p k (F k (w k t ) -F k * ) -2η t 1 N N k=1 (F k (w k t ) -F k (w * )) D .We set γ t = 2η t (1 -2Lη t ), because η t ≤ 1 4L , so η t ≤ γ t ≤ 2η t . Decompose D we can get:D = 4Lη t 2 N k=1 p k (F k (w k t ) -F k * ) -2η t 1 N N k=1 (F k (w k t ) -F k (w * )) = (4Lη t 2 -2η t ) N k=1 p k F k (w k t ) + 4Lη t 2 N k=1 p k F k * + 2η t 1 N N k=1 F k (w * ) = -γ t N k=1 p k F k (w k t ) + 4Lη t 2 N k=1 p k F k * + 2η t F * = -γ t N k=1 p k (F k (w k t ) -F * ) + (2η t -γ t ) N k=1 p k (F * -F k * ) = -γ t N k=1 p k (F k (w k t ) -F * ) + 4Lη t 2 Γ.It should be noted that Γ =EQUATIONN F k * , and it measures the degree of data heterogeneity between different client devices.We can bound the first term in D:N k=1 p k (F k (w k t ) -F * ) = N k=1 p k (F k (w k t ) -F k ( wt )) + N k=1 p k (F k ( wt ) -F * ) ≥ N k=1 p k ∇F k ( wt ), w k t -wt + F ( wt ) -F * ≥ - 1 2 N k=1 p k [η t ∥∇F k ( wt )∥ 2 2 + 1 η t w k t -wt 2 2 ] + F ( wt ) -F * ≥ - N k=1 p k [η t L(F k ( wt ) -F k * ) + 1 2η t w k t -wt 2 2 ] + F ( wt ) -F * .In the above inequation, we use the inequations ( 16), (19) and the convexity of F k (•).So we can bound D and get:D = -γ t N k=1 p k (F k (w k t ) -F * ) + 4Lη t 2 Γ ≤ γ t N k=1 p k [η t L(F k ( wt ) -F k * ) + 1 2η t w k t -wt 2 2 ] -γ t (F ( wt ) -F * ) + 4Lη t 2 Γ = γ t η t L N k=1 p k (F k ( wt ) -F k * ) + γ t 2η t N k=1 p k w k t -wt 2 2 -γ t (F ( wt ) -F * ) + 4Lη t 2 Γ = γ t η t L N k=1 p k (F k ( wt ) -F * + F * -F k * ) + γ t 2η t N k=1 p k w k t -wt 2 2 -γ t ( N k=1 p k F k ( wt ) -F * ) + 4Lη t 2 Γ = γ t (η t L -1) N k=1 p k (F k ( wt ) -F * ) + (γ t η t L + 4Lη t 2 )Γ + γ t 2η t N k=1 p k w k t -wt 2 2 ≤ 6Lη t 2 Γ + N k=1 p k w k t -wt 2 2 .When t + 1 / ∈ I E , we can conclude that:E ∥ wt+1 -w * ∥ 2 2 ≤ 2( √ dδ + 1)η 2 t G 2 + (1 -η t µ)E ∥ wt -w * ∥ 2 2 + 2 N N K=1 wt -w k t 2 2 + 6Lη 2 t Γ + Eη 2 t ∥g t -ḡt ∥ 2 2 . When t + 1 ∈ I E , we have w k t+1 = Q( N k=1 v k t+1 N) :E ∥ wt+1 -w * ∥ 2 2 = E Q( 1 N N k=1 v k t+1 ) -w * 2 2 = E Q( 1 N N k=1 v k t+1 ) - 1 N N k=1 v k t+1 + 1 N N k=1 v k t+1 -w * 2 2 = E Q( 1 N N k=1 v k t+1 ) - 1 N N k=1 v k t+1 + 1 N N k=1 Q(w k t -η t ∇F k (w k t , ξ k t )) -w * 2 2 ≤ E Q( 1 N N k=1 v k t+1 ) - 1 N N k=1 v k t+1 2 2 E1 + E 1 N N k=1 Q(w k t -η t ∇F k (w k t , ξ k t )) -w * 2 2 E2 + 2E⟨Q( 1 N N k=1 v k t+1 ) - 1 N N k=1 v k t+1 , 1 N N k=1 Q(w k t -η t ∇F k (w k t , ξ k t )) -w * ⟩ E3E 2 has been given constraints in the text above that:E 2 ≤ 2( √ dδ + 1)η 2 t G 2 + (1 -η t µ)E ∥ wt -w * ∥ 2 2 + 2 N N K=1 wt -w k t 2 2 + 6Lη 2 t Γ + Eη 2 t ∥g t -ḡt ∥ 2 2 .According to the law of iterated expectations, E 3 is 0. We next constrain E 1 . According to our assumption before Lemma 2, we have:E 1 = EE 1 ≤ Edδ 2 ≤ Edη 2 t ∇F k (w k t , ξ k t ) 2 2 = dη 2 t G 2 .So when t + 1 ∈ I E we have:E ∥ wt+1 -w * ∥ 2 2 ≤ 2( √ dδ + 1 + d 2 )η 2 t G 2 + (1 -η t µ)E ∥ wt -w * ∥ 2 2 + 6Lη 2 t Γ + 2 N N K=1 wt -w k t 2 2 + Eη 2 t ∥g t -ḡt ∥ 2 2 .In summary, whether t + 1 ∈ I E or t + 1 / ∈ I E , we have:E ∥ wt+1 -w * ∥ 2 2 ≤ 2( √ dδ + 1 + d 2 )η 2 t G 2 + (1 -η t µ)E ∥ wt -w * ∥ 2 2 + 6Lη 2 t Γ + 2 N N K=1 wt -w k t 2 2 + Eη 2 t ∥g t -ḡt ∥ 2 2Lemma 3. Based on assumption 2, we can get:E∥g t -ḡt ∥ 2 2 ≤ 1 N 2 N k=1 σ k 2 .Proof.E∥g t -ḡt ∥ 2 2 = E 1 N N k=1 (∇F k (w k t , ξ k t ) -∇F k (w k t )) 2 2 = N k=1 1 N 2 E ∇F k (w k t , ξ k t ) -∇F k (w k t ) 2 2 ≤ 1 N 2 N k=1 σ k 2 .Lemma 4. If η t is non-increasing and η t ≤ 2η t+E , (t ≥ 0),based on assumption 2, we can get:1 N N k=1 E wt -w k t 2 2 ≤ 8E 2 η 2 t G 2 (2 √ dδ + 3).Proof. Since FedAvg requires a communication each E steps. Therefore, for any t ≥ 0, there exists a t 0 ≤ t, such that t -t 0 ≤ E -1 and wt0 = w k t0 for all k. Also, we use the fact that η t is non-increasing and η t ≤ 2η t+E , (t ≥ 0), then:EQUATION.Inequalities E i=1 a i 2 2 ≤ E E i=1 ∥a i ∥ 2 2 and E ∥X -EX∥ 2 2 ≤ E ∥X∥ 2 2are used in the above proof.We have already proofed in lemma 2 that:E Q(w k h -η h ∇F h (w k h , ξ k h )) -(w k h -η h ∇F h (w k h , ξ k h )) 2 2 ≤ 2( √ dδ + 1)η 2 h G 2 . So we can have: 1 N N k=1 E wt -w k t 2 2 ≤ 8E 2 η t 2 G 2 (2 √ dδ + 3).Theorem 1. Under the condition that assumptions 1, 2, we choose κ = L µ , γ = max{8κ, E} -1 and η t = µ 2(t+γ) . When t satisfying δ 2 ≤ η 2 t G 2 our low precision federated learning algorihm with full device participation satisfies:EF ( wt ) -F * ≤ L 2 v t + γ ≤ κ t + γ ( 2B µ + µ(γ + 1) 2 ∥w 1 -w * ∥ 2 ).whereB = 2( √ dδ + 1 + d 2 )G 2 + 16E 2 G 2 (2 √ dδ + 3) + 1 N 2 N k=1 σ k 2 + 6LΓ. Proof. Set ∆ t = E∥ wt -w * ∥ 2 2, according to lemma 1, 2, 3, we can have:∆ t+1 ≤ (1 -µη t )∆ t + η t 2 B.Set η t = β t+γ , we can choose appropriate β and γ so thatη t ≤ min{ 1 µ , 1 4L } and η t ≤ 2η t+E . Set v = max{ β 2 Bβµ-1 , (γ + 1)∆ 1 }, we will prove ∆ t ≤ v t+γ by mathematical induction. When t = 1, obviously it holds. Assume the conclusion holds for some t, it follows that:∆ t+1 ≤ (1 -µη t )∆ t + η t 2 B ≤ (1 - βµ t + γ ) v t + γ + β 2 B (t + γ) 2 = t + γ -1 (t + γ) 2 v + [ β 2 B (t + γ) 2 - βµ -1 (t + γ) 2 v] ≤ v t + γ + 1 .Then by the L-smoothness of F (•),we have:EF ( wt ) -F * ≤ L 2 ∆ t ≤ L 2 v t + γ .We choose β = 2 µ , γ = max{8κ, E} -1 and denote κ = L µ , then we have:v = max{ β 2 B βµ -1 , (γ + 1)∆ 1 } ≤ β 2 B βµ -1 + (γ + 1)∆ 1 ≤ 4B µ 2 + (γ + 1)∆ 1 .andEF ( wt ) -F * ≤ L 2 v t + γ ≤ κ t + γ ( 2B µ + µ(γ + 1) 2 ∆ 1 ).We analyze our low precision federated learning algorithm in the setting of partial device participation in this section.Updating scheme In real world application scenarios, constrained by communication efficiency and low straggler effect, FedAvg initiates by selecting a random subset S t which length is set to K, and subsequently carries out updates exclusively on this subset.The analysis becomes somewhat complex due to the variability of S t every E steps. Nevertheless, we can employ a thought trick to address this challenge. We envision that FedAvg initiates each epoch by engaging all devices and then relies solely on the parameters updated on a subset of these devices to generate the parameters for the subsequent round. It is evident that this method of parameter update is equivalent to the original approach. Then the update of FedAvg with partial devices active can be described as: for all k ∈ [N ],v k t+1 = Q(w k t -η t ∇F k (w k t , ξ k t )). w k t+1 = v k t+1 if t + 1 / ∈ S t , Q( k∈St 1 K v k t+1 ) if t + 1 ∈ S t .We use E St (•) to eliminate the error caused by S t . Unless otherwise specified, the meanings and assumptions of the symbols used in this section are the same as in the previous section. Lemma 5. If t + 1 ∈ I E , we have:E St ( wt+1 ) = vt+1 .Proof. According to the selection method of S t , there is a function:I i = 1, P(i ∈ S t ) = K N 0, P(i / ∈ S t ) = N -K N .According to the aggregation process, we have:wt+1 = Q( 1 K N i=1 I i v i t+1 ).So we have:E St ( wt+1 ) = E St ( 1 K N i=1 I i v i t+1 ) = 1 K N i=1 P i v i t+1 = 1 K N i=1 K N v i t+1 = N i=1 1 N v i t+1 = vt+1 .Lemma 6. For t + 1 ∈ I E we have:E St ∥v t+1 -wt+1 ∥ 2 2 ≤ N -K N -1 8 K E 2 η t 2 G 2 (2 √ dδ + 3) + dG 2 η 2 t .Proof. According to the aggregation process,we havewt+1 = Q( 1 K i∈St v i t+1).E St ∥ wt+1 -vt+1 ∥ 2 2 = E St Q( 1 K i∈St v i t+1 ) - 1 K i∈St v i t+1 + 1 K i∈St v i t+1 -vt+1 2 2 = E St Q( 1 K i∈St v i t+1 ) - 1 K i∈St v i t+1 2 2 F1 + 1 K 2 E St N i=1 I(i ∈ S t )(v i t+1 -vt+1 ) 2 2 2⟨Q( 1 K i∈St v i t+1 ) - 1 K i∈St v i t+1 , 1 K i∈St v i t+1 -vt+1 ⟩ F3In the lemma 2, we have proven F 1 ≤ dG 2 η 2 t . And obviously we have F 3 = 0, we next aim to bound F 2 .1 K 2 E St N i=1 I(i ∈ S t )(v i t+1 -vt+1 ) 2 2 = 1 K 2 N i=1 P (i ∈ S t ) v i t+1 -vt+1 2 2 + 1 K 2 i̸ =j P (i, j ∈ S t ) v i t+1 -vt+1 , v j t+1 -vt+1 = 1 KN N i=1 v i t+1 -vt+1 2 2 + i̸ =j K -1 KN (N -1) ⟨v i t+1 -vt+1 , v j t+1 -vt+1 ⟩ = 1 K(N -1) 1 - K N N i=1 v i t+1 -vt+1 2 2 .The last step in the above equation uses:N i=1 v i t+1 -vt+1 2 2 + i̸ =j ⟨v i t+1 -vt+1 , v j t+1 -vt+1 ⟩ = 0.Therefore:E ∥ wt+1 -vt+1 ∥ 2 2 = N K(N -1) 1 - K N E 1 N N i=1 v i t+1 -wt0 + wt0 -vt+1 2 2 + F 1 ≤ N K(N -1) 1 - K N E 1 N N i=1 v i t+1 -wt0 2 2 + F 1 ≤ N K(N -1) 1 - K N 8E 2 η t 2 G 2 (2 √ dδ + 3) + dG 2 η 2 t .We used E ∥X -EX∥ Theorem 2. Under the condition that assumptions 1, 2, 3, we choose κ = L µ , γ = max{8κ, E} -1, η t = µ 2(t+γ) and B = 2(√ dδ + 1 + d 2 )G 2 + 16E 2 G 2 (2 √ dδ + 3) + 1 N 2 K k=1 σ 2 k + 6LΓ, C = N -K N -1 8 K E 2 G 2 (2 √ dδ + 3) + dG 2 . When t satisfying δ 2 ≤ η 2 t G 2, FedAvg with quantization and partial device participation satisfies:E[F ( wt )] -F * ≤ κ γ + t 2(B + C) µ + µ(γ + 1) 2 ∥w 1 -w * ∥ 2 .Proof. We have:E ∥ wt+1 -w * ∥ 2 2 = E ∥ wt+1 -vt+1 + vt+1 -w * ∥ 2 2 = E ∥ wt+1 -vt+1 ∥ 2 2 G1 + E ∥v t+1 -w * ∥ 2 2 G2 + 2E⟨ wt+1 -vt+1 , vt+1 -w * ⟩ G3 .We take the expectation for the above formula, G 3 vanishes according to lemma 5.If t + 1 / ∈ I E , according to our parameter update settings, G 1 vanishes. We use lemma 1, 2, 3, 4 to bound G 2 :E ∥v t+1 -w * ∥ 2 2 = E ∥ wt+1 -w * ∥ 2 2 ≤ (1 -η t µ)E ∥ wt -w ⋆ ∥ 2 2 + η 2 t B.If t + 1 ∈ I E , We use lemma 6 to bound G 1 , lemma 1, 2, 3, 4 to bound G 2 , then we have :E ∥ wt+1 -w * ∥ 2 2 = E ∥ wt+1 -vt+1 ∥ 2 2 + E ∥v t+1 -w * ∥ 2 2 ≤ (1 -η t µ)E ∥ wt -w ⋆ ∥ 2 2 + η 2 t (B + C).We use exactly the same method as in Theorem 1 and then we have:E[F ( wt )] -F * ≤ κ γ + t 2(B + C) µ + µ(γ + 1) 2 ∥w 1 -w * ∥ 2 .We will introduce the four FL methods, ABAvg, FedProx, FedGen, FedFTG, and their hyperparameters setting in our experiments to prove our method's effectiveness.• ABAvg [35] is a data-dependent distillation FL method, which uses a validation dataset D v to reweight each collected devices. In our experiment, we set the test dataset as the validation dataset. For FashionMNIST, CIFAR10, CIFAR100, we use the whole test dataset, and for CINIC10, we choose the 20% data of each label.• FedProx [22] is an FL method with regularization when training locally. We set the FedProx proximal term µ = 0.1 in our experiment.• FedGen [42] is a data-free distillation FL method, which maintain a generator in the server to generate hidden variables, and transfer the generator to the device side for data enhancement.We set the generator learning rate to 3e -4, hidden dimension to 32. The generator's training epoch is 5. • FedFTG [40] is a data-free distillation FL method, which maintain a generator on the server for fine-tuning aggregated server parameters. We set the generator learning rate to 1e -2, the fine-tuning learning rate to 1e -4. The totally fine-tune epoch is set to 10, inner epoch for generator is 1, inner epoch for server parameter is 5.All of our models are trained on s GeForce RTX 4090. We found only very small differences when evaluating on these other hardware platforms.We put the four algorithms used here together with Efficient FL. Judging from the Table 4 , lowprecision local training is sufficient in FL system. We put more emphasis on showing the case of α = 0.01, because this case is more consistent with the actual situation of FL, that is, the data heterogeneity of edge devices. Combining four FL methods and four datasets, in the case of α = 0.01, Figure 4 , we can find the fact that our algorithm can maintain or even surpass the accuracy of the original algorithm on 8 bit, and at the same time, it can do the same on 6 bit to maintain accuracy. We also place the algorithms used here, and these pseudocodes are added to our methods. When we conduct experiments, we also follow the algorithm flow from Algorithm 3 to Algorithm 6. We found that although different FL methods will behave differently when the client updates or the server updates, our method can be easily applied to them.We list the detailed results of the Efficient FL used in experiments. We use the same network structure ConvNet and the CIFAR10 split in α = 0.01 to conduct experiments. In HeteroFL, a, b, c, d, and e respectively represent 100%, 50%, 25%, 12.5%, and 6.25% channel proportions. In Table 5 , for example, a1-b1-c1 means that during local training, a, b, c, the three models account for 1:1:1. In SplitMix, a, b, c, d, e, have the same representation. 1/8 and 1/16 of SplitMix represent the minimum model channel ratio after segmentation. We conducted experiments on 1/8 and 1/16 respectively.We calculated the average memory consumption of a single device during each communication round, for each compression method, and also for our method. For memory consumption statistics, we record model size, gradient storage, and intermediate activation storage. Table 5 to Table 8 contains all our experimental results comparing memory savings. Table 5 : HeteroFL. a1, b1, c1, d1, e1 represent the percentage of the number of model channels to the original number: 1, 0.5, 0.25, 0.125, 0.625. For example, a1-b1-c1 means that the a, b, and c models exist in a 1:1:1 ratio. In fact, what we performed in the experiment was fake quantization, that is, simulated quantization, which means we only focused on the impact of quantization on accuracy. The actual quantization operation requires the cooperation of hardware, and the hardware GPU we used was GeForce RTX 4090, which could not directly perform low precision training, and could not reflect the acceleration effect of our method.FL, while having a positive impact in areas such as data privacy protection and cross-domain collaboration, also faces some potential negative impacts and challenges.First, in FL, attackers might attempt to implant backdoors during the model training process. By introducing specific triggers into the training data, attackers can activate the backdoor after the model is deployed, causing the model to output results that the attacker desires. This type of attack poses a serious threat to the model's security and needs to be guarded against with strict security measures and auditing processes.Second, attackers could disrupt the performance of the model by injecting incorrect information into the training data. This data poisoning can occur at the client side or during the model aggregation process. Data poisoning attacks may lead to the model behaving abnormally under certain conditions, affecting the model's reliability and accuracy.Last, although FL is designed to protect user privacy by training models locally and sharing only model updates rather than raw data, this method is not foolproof. Attackers might infer sensitive information about the training data by analyzing the model's update information, such as gradients or weight changes. This kind of privacy leakage can be mitigated with techniques like differential privacy, which may impact the model's performance.Input:Quantization functions Q A , Q E , Q G , Q M , Q W ; Momentum coefficient ρ; L layers DNN {f 1 , f 2 , . . . , f L }; Loss function ℓ; Validation dataset D v . 1: Initialize: w 0 , w0 ← w 0 2: for t = 0, 1, . . . , T -1 do 3: if t ≡ 0 (mod E) then 4:Select K clients from [N ] to be S t 5:w k t ← Q( wt ), k ∈ S t 6:end if Forward Propagation:22:EQUATION:(a k t ) (l) = Q A (f l ((a k t ) (l-1) , (w k t ) (l) )), ∀l ∈ [1, L] 24:Backward Propagation:25:(e k t ) (L) = ∇ (a k t ) (L) ℓ((a k t ) (L) , y k,jt ) 26: (e k t ) (l-1) = Q E ( ∂f l ((a k t ) (l-1) ,(w k t ) (l) ) ∂(a k t ) (l-1) (e k t ) (l) ), ∀l ∈ [1, L] 27: (g k t ) (l) = Q G ( ∂f l ((a k t ) (l-1) ,(w k t ) (l) ) ∂(w k t ) (l) (e k t ) (l) ), ∀l ∈ [1, L] 28:Low Precision SGD Update:29:(v k t+1 ) (l) ← Q M (ρ(v k t ) (l) + (g k t ) (l) ), ∀l ∈ [1, L] 30: (w k t+1 ) (l) ← Q W ((w k t ) (l) -η t • (v k t+1 ) (l) ), ∀l ∈ [1, L] 31:Return: w k t+1 Algorithm 4 Low Precision FL with FedProx Forward Propagation:Input: Quantization functions Q A , Q E , Q G , Q M , Q W ; Momentum coefficient ρ; L layers DNN {f 1 ,21:EQUATION:(a k t ) (l) = Q A (f l ((a k t ) (l-1) , (w k t ) (l) )), ∀l ∈ [1, L] 23:Backward Propagation:24:(e k t ) (L) = ∇ (a k t ) (L) ℓ((a k t ) (L) , y k,jt ) 25: (e k t ) (l-1) = Q E ( ∂f l ((a k t ) (l-1) ,(w k t ) (l) ) ∂(a k t ) (l-1) (e k t ) (l) ), ∀l ∈ [1, L] 26: (g k t ) (l) = Q G ( ∂f l ((a k t ) (l-1) ,(w k t ) (l) ) ∂(w k t ) (l) (e k t ) (l) + µ((w k t ) (l) -(w k ⌊ t E ⌋E ) (l) )), ∀l ∈ [1, L] 27:Low Precision SGD Update:28:(v k t+1 ) (l) ← Q M (ρ(v k t ) (l) + (g k t ) (l) ), ∀l ∈ [1, L] 29: (w k t+1 ) (l) ← Q W ((w k t ) (l) -η t • (v k t+1 ) (l) ), ∀l ∈ [1, L] 30:Return: w k t+1 Algorithm 6 Low Precision FL with FedFTG Forward Propagation: Input: Quantization functions Q A , Q E , Q G , Q M , Q W ; Momentum coefficient ρ; L layers DNN {f 1 ,EQUATION(a k t ) (l) = Q A (f l ((a k t ) (l-1) , (w k t ) (l) )), ∀l ∈ [1, L] 22:Backward Propagation: (e k t ) (L) = ∇ (a k t ) (L) ℓ((a k t ) (L) , y k,jt )24:(e k t ) (l-1) = Q E ( ∂f l ((a k t ) (l-1) ,(w k t ) (l) ) ∂(a k t ) (l-1) (e k t ) (l) ), ∀l ∈ [1, L] 25: (g k t ) (l) = Q G ( ∂f l ((a k t ) (l-1) ,(w k t ) (l) ) ∂(w k t ) (l) (e k t ) (l) ), ∀l ∈ [1, L] 26:Low Precision SGD Update:27: (v k t+1 ) (l) ← Q M (ρ(v k t ) (l) + (g k t ) (l) ), ∀l ∈ [1, L] 28: (w k t+1 ) (l) ← Q W ((w k t ) (l) -η t • (v k t+1 ) (l) ), ∀l ∈ [1,w t+1 = k∈S t ′ p k q t ′ w k t+1 33: Compute p t ′ (y) ∝ k∈S t ′ |D k | j=1 E (x k,j ,y k,j )∼D k 1 y k,j =y = k∈S t ′ n y k 34:for i = 1, 2, . . . , I doGet batch (Z, Y ) from z ∼ N (0, 1) and y ∼ p t ′ (y)36:for j = 1, 2, . . . , I g do 37:Update θ according to min Justification: The algorithms mentioned in the introduction and abstract can be found in Section 4 of the article. Theoretical result can be founded in section 5. The convergence proof of our algorithm is located in appendix A and B.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The reviewers were generally convinced about the communication and computation reduction in the context of FL achieved by the proposed method. They also appreciated both the extent of experiments and the convergence guarantees provided in the paper. On the negative side, there is an overlap with in techniques [40] though, as the authors point out, the application to FL is novel.

Overall, the authors should:
-Include in the paper the additional experiments that they conducted during the rebuttal stage; these can be added in the supplement if there is an issue of space, but hopefully some can surface to the main body of the paper as well.
-Proofread for typos, but also improve presentation w.r.t. aspects indicated by the reviewers.

Reviews: review: paper_summary: paper_summary: The paper proposes an efficient Federated Learning (FL) paradigm where local models are trained using low-precision operations and communicated with the central server in low precision format. The aggregation on the server, however, is performed with high-precision computation to ensure accuracy. The authors demonstrate that high-precision models can be recovered from low-precision local models with proper aggregation on the server side. This approach significantly reduces the computational load on client devices and the communication cost. The method is theoretically proven to converge to an optimal solution, even with non-IID data distributions, and extensive experiments show that models trained with low precision (as low as 8 bits) are comparable in performance to those trained with full precision.
strength_weakness: strength_weakness: 1. The integration of low precision training and high precision aggregation may add complexity to the implementation.The performance improvements are partly dependent on the hardware capabilities, such as the availability of processors supporting low precision operations.
2. The experiments are limited. Only image datasets are considered. Evaluation on other types of data, like text or tabular, can strengthen the results. 
3. No integration with differential privacy or other privacy protection mechanisms. Federated learning itself is not private and it would be interesting to see what privacy mechanisms are suitable for low precision model updates.
questions: questions: 1. How can we determine the optimal precision level for local training without extensive hyperparameter tuning which is expensive in FL?
2. Have you explored mixed precision strategy? E.g. using different quantization schema for gradients, activations etc.
limitations: limitations: The authors have discussed the limitations in the paper.
paper_summary: paper_summary: The paper studies an FL system with data heterogeneity, a topic has been extensively studied in the past few years. The idea is to perform local training with lower precision through applying block floating point quantization. The idea per se is not new, but proving that the convergence can be achieved using low precision local training is an interesting contribution.
strength_weakness: strength_weakness: The paper mainly focuses on data heterogeneity in FL systems. What about resource heterogeneity? It would be important to have a discussion (or better some experimental results) on how the proposed solution perform in such setting. Should we use the same quantization level for all clients, or can we adjust the precision according to the resource availability? Also, the current state of the art of quantization in conjunction with Federated Learning is also missing, e.g.:

- FedQNN[1] uses quantized training in FL.

- CoCoFL[2] uses a combination of quantization and freezing for heterogeneous resources in FL.

How does these SOTA techniques perform compared with the proposed solutions?

[1] Y. Ji and L. Chen, "FedQNN: A Computation–Communication-Efficient Federated Learning Framework for IoT With Low-Bitwidth Neural Network Quantization," in IEEE Internet of Things Journal.
  
[2] Kilian Pfeiffer, et al. "CoCoFL: Communication-and Computation-Aware Federated Learning via Partial NN Freezing and Quantization." Transactions on Machine Learning Research., 2023
questions: questions: Please also check my questions in the weakness section.
limitations: limitations: Appendix F clarifies that the fake quantization is applied in the experiments. I really appreciate learning this information, as one of my question was how the quantization is implemented in PyTorch (as PyTorch only supports int8 inference out of the box). 
Regarding the broader impact, the discussion is about FL in general. The question is if performing lower precision training at clients improves or worsen these privacy and security aspects (e.g. if data poisoning cab be done easier, as the local updates are low precision).
paper_summary: paper_summary: The paper proposes a federated learning approach that performs local training on low precision through quantization combined with a high-precision averaging and a moving average at the server. The paper guarantees convergence and empirically compares several levels of low-precision local training to full-precision training on 4 baseline FL methods. It remains unclear, though, what the contribution of the method is: the main focus seems to be on performance in terms of test accuracy, but the experiments do not show a significant improvement over existing methods. The method supposedly improves communication and computation efficiency but is not empirically compared to state-of-the-art methods, such as [1,2,3].

In their rebuttal, the authors provided novel results that address my concerns about missing baselines. While I remain concerned about the limited novelty and the presentation, I believe that the authors will be able to address these issues to some extent in the next version of the manuscript. Therefore, I have decided to increase my score.

\
[1] Liu, Shiyu, et al. "Quantized SGD in Federated Learning: Communication, Optimization and Generalization." International Conference on Neural Information Processing. Singapore: Springer Nature Singapore, 2023.

[2] Kamp, Michael, et al. "Efficient decentralized deep learning by dynamic model averaging." Machine Learning and Knowledge Discovery in Databases: ECML PKDD, 2018.

[3] Reisizadeh, Amirhossein, et al. "Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization." International conference on artificial intelligence and statistics. PMLR, 2020.
strength_weakness: strength_weakness: - the presentation should be improved (e.g., several typos, such ass "accept" instead of "aspect", the term "a avg" for "with moving average" is unintuitive)
- the ablation study does not clearly show that low precision local training improves performance, since it is combined with a moving average that has a strong positive impact on performance.
- lack of comparison to baselines
- unclear use-case for the method
- the paper does not discuss existing federated learning with quantization literature in sufficient detail.
- for the non-iid experiments it would be interesting how quantization interacts with local BN layers in FedBN [4]

\
[4] Li, Xiaoxiao, et al. "FedBN: Federated Learning on Non-IID Features via Local Batch Normalization." International Conference on Learning Representations, 2021.
questions: questions: - It is unclear from the results how much of the benefit stems from quantization and how much from the moving average. Is it correct that the moving average has a very large positive effect regardless of quantization?
limitations: limitations: Section F in the appendix addresses the limitation of simulating quantization. It does not address issues like computation and communication complexity, the applicability of the approach, or limitations in the empirical evaluation.
paper_summary: paper_summary: This paper proposes an efficient federated learning (FL) paradigm, where the local models in the clients are trained with low-precision operations and communicated with the server in low precision format, while only the model aggregation in the server is performed with high-precision computation. The performance is comparable to full-precision training, and sometimes even better since the over-fitting issue in local training is relieved.
strength_weakness: strength_weakness: W1. The biggest concern is regarding the novelty w.r.t. SWALP [40]. The entire Sec 3.2 and Sec 4.1 are almost the same as in [40]. The only difference seems to be Sec 4.2 but still very similar to the idea of SWA, just the setting of aggregation changes from by cycles to by clients, and a moving average of parameters is used.

W2. Writing needs improvement. For example, there is a typo "sever" in Line 108.
Eq (5) is different from its counterpart in [40] where the power was F-1 but now W-F-1. please explain the reason of difference.
Line 135, missing a space before "i.e."
Eq(7) uses E which is not clear until continuing reading to Line 161 and Algorithm 2 Lines 3-4.
Algorithm 2 Line 11, t' is only briefly mentioned in Line 161 without even referring to the used lines.

W3. It would be good to estimate the time reduction with the professional hardware (real acceleration).
questions: questions: Justify method novelty against [40] (W1) and answer related questions in W2.
limitations: limitations: N/A

Comments: title: Reply to authors
Rebuttal by Authors (2)
comment: Dear Reviewer 1d4C,

We sincerely appreciate your support and valuable comments on improving our paper. We believe the primary similarity between SWALP and our method is their high-level inspiration from Kolmogorov’s law, which asserts that the sample average can almost surely converge to the expected value despite the presence of noise. We also find it highly appropriate to develop efficient low-precision local training methods for federated learning, given that the clients in many federated learning applications are resource-constrained. Our proposed method is both simple and effective, presenting a new framework/paradigm for accelerating federated learning. We hope it will inspire researchers to create more efficient federated learning algorithms based on low-precision local training in the future. We believe this contribution holds comparable significance to our proposed method itself.

As the deadline for the author-reviewer discussion phase approaches, could you please let us know if any further discussion or clarification is needed? Thank you again for your valuable time; your support is greatly appreciated.

Sincerely,
Authors of Submission 11555
Dear Reviewer Zy16,

We sincerely appreciate for your support and have carefully responded to your constructive comments thoroughly.

Since the deadline for the author-reviewer discussion phase is approaching, would you mind kindly letting us know if any further discussion or clarification is needed? Thank you again for your valuable time.

Sincerely,

Authors of Submission 11555
Dear Reviewer vykk,

We sincerely appreciate your response and  final support! Since the deadline for the author-reviewer discussion phase is approaching, would you mind kindly letting us know if any further discussion or clarification is needed? Thank you again for your valuable time, your support is significant to us. 

Sincerely,

Authors of Submission 11555
Thanks for the responses. I still feel the method a bit similar to SWALP, and writing needs some work to be clear.
I will keep my rating since it was very positive already.
Thanks for the new experiments (I was expecting similar results). Please consider including these new experiments in the paper. I confirm my acceptance score.
Dear authors,

Thank you for your detailed reply and the additional experiments. Please include those points and results in the manuscript. I am leaning towards improving my score after the discussion period with my fellow reviewers.
We would like to sincerely thank the reviewer for the thorough and prompt feedback. We appreciate the opportunity to address and clarify the points raised.
     
**Q5: What if you train the whole model at all devices, but adjust the precision level to the availability of resources at devices?**
  
**A5:** Thanks for your constructive suggestion. Actually, the setting of our previous experiments in the rebuttal is unfair to us rather than to other baselines, as our method is added with much more strict and also unnecessary constraint on the resource. 
    
According to your advice, below we present the experimental results of training all models by adjusting the precision level  to the availability of resources at devices. We now impose three $\textbf{memory}$ resource constraints (1, 1/2, and 1/4)  on the clients. We would like to clarify that in our previous experiments in the rebuttal,  three types of resource devices: 1, 1/2, and 1/4 respectively represent the devices that can train the entire network, 1/2 channels of the network, and 1/4 channels of the network, which is equivalent to the networks with full, 1/4 and 1/16  $\textbf{memory}$ cost.  Here we use the memory reconstraint and let it to be 1, 1/2 and 1/4 just to make the corresponding precision levels be the widely used ones, i.e., 32, 16 and 8 bits. We list four experimental methods:
    1) FedAVG with full resources, where all the devices have the full resources and be able to train the full network.
    2) HeteroFL adjusts the number of parameters of each client to meet the resource constraints for training.
    3) Our method sets the precision levels of the devices under the three constraints to be 32, 16, and 8 respectively to approximately meet the resource constraints and train the full model.
    4) FedAVG refers to the setting that the network which cannot be trained (due to resource constraints) is directly dropped. We tested the results on CIFAR10, setting $\alpha=0.04$ and $0.16$. The detailed result is given in the table below, which verifies the effectiveness of our method.
   
 |Method               |Acc($\alpha=0.04$)|Acc($\alpha=0.16$)|
 |:-|:-:|:-:|
|FedAVG(full resource)|54.4±2.0          |71.9±1.5|
|HeteroFL             |46.3±1.7          |63.4±1.8|
|Ours                 |58.1±0.5          |72.7±0.5|
|FedAVG               |32.4±2.3          |52.8±2.1|
Thanks for the detailed reply. I suggest to integrate the results for Q2 - Q4 in the final version. I have no follow up to these questions. For Q1, I do not see why we need to adapt HeteroFL and adjust the network size at different devices. This could cause some fairness issues, especially when the data is non-i.i.d, as shown in the CoCoFL paper. What if you train the whole model at all devices, but adjust the precision level to the availability of resources at devices?