# FedOpt: Towards Communication Efficiency and Privacy Preservation in Federated Learning
## 6 Related Work and Discussions  
Stochastic Gradient Descent (SGD) is very popular optimisation training technique that supports various DL applications in DNN models. In particular, on one end of the spectrum, SGD can be used to reduce the convergence time in large-scale applications of DL models by using the device-level parallelism [26; 27; 28]. On the other end of the spectrum, SGD can be used to enable and enhance the privacy preserving in DL algorithms [29]. Since the users are require to share the gradient updates, where SGD helps in training of model from the combined data of all the users without revealing the individual's local data to a centralised cloud server [30]. However, despite the tremendous advantages and the extensive applications of SGD based DL, existing research show that, learning the model updates suffers from a massive communication overhead [31; 32]. In order to reduce this communication overhead, a wide variety of methods had been proposed in the past. For example, in [23], the authors proposed a novel approach, i.e., FedAvg, where each user computes the gradient updates by performing multiple epochs of SGD which results into increasing the number of gradient evaluations that causes delay in communication. In order to minimise this communication delay, the authors in [33], use probabilistic-quantisation and random-sparsification. In particular, the authors force the random-sparsity on the users or restrict them in order to learn random-sparse gradient updates (structured and sketched updates) and combine probabilistic quantisation with this sparsification. This method however is not suitable for SGD epochs as it slows down the convergence speed significantly. To overcome this convergence issue, the authors in [34], propose a compression technique; namely, SignSGD that theoretically guarantees the convergence over iid data. This SignSGD quantises each gradient from each user to a binary sign and reduces the bit-size per gradient update by \(\times 32\). This compression on gradients is done by means of a majority vote which may result into  
Figure 8: Computational cost per user with different number of gradients on CIFAR-10 dataset.  
Figure 7: Computational cost per user with different number of gradients on MNIST dataset.  
the loss of important updates. In addition, the compression rate and empirical performance does not reach up-to the convergence requirement in FL.  
On the other hand, several adversaries may violate the private information of users from DL networks [35; 36]. As a result, several privacy preserving DL schemes are proposed in order to eliminate those privacy threats. For example, in [19], the authors proposed the first privacy preserving approach for collaborative DL where, the users share their selective partial gradients with the cloud server in order to ensure the privacy of the training data. However, even sharing a small part of the gradient, the privacy of users can still be compromised [37]. Additionally, in [38], the authors proposed a DL model using additively homomorphic encryption where, all the users share the same decryption key, which may lead to collusion among the users and the cloud server. However, all these approaches suffer from serious privacy threats because of the shared parameters.  
On the contrary, unlike the proposed FedOpt, various privacy preserving centralised training approaches are proposed to overcome the privacy issues. For example, the authors in [39], adopt a two-party computation scheme that utilises two-party computation training model, where the local users separate the sensitive data into two parts before sending it to the two non-collusive central servers. In order to eliminate privacy leakage, [40] used a Gaussian mechanism to update the gradients and proposed a rigid method (e.g., moment accountant) in order to keep record of the entire privacy loss.