# Advanced Optimization Techniques for Federated Learning on Non-IID Data
## 1 Introduction  
Mobile devices have emerged as the primary computational resource for a vast number of users worldwide, and an even greater number of IoT devices are expected to be operational in the coming years. Predictions suggest that by 2025, the global data volume will increase to 180 trillion GBs, with approximately 80 billion nodes likely connected to the Internet [1]. Machine learning models trained on this massive amount of data have the potential to enhance the capabilities of various applications significantly. However, enabling such functionalities on mobile devices often requires data to be shared among servers on a global scale, necessitating the preservation of data security and privacy and the reduction of communication overheads. Consequently, Federated Learning (FL), which retains data on the device and shares only the model, has become increasingly attractive. Federated Learning (FL) represents a critical development in distributed machine learning, as it offers a framework where multiple clients (e.g., mobile phones or organizations) collaborate to train a model under the coordination of a central server while keeping the training data local. The most widely used FL algorithm is Federated Averaging (FedAvg), which assumes that each client is connected to a server. FedAvg trains the global model iteratively, through parallel local model training on clients and aggregation of the global model on the server [2]. This approach not only ensures privacy by eliminating the need for data exchange but also reduces communication costs, which is crucial given the high volume of data managed by these devices. Despite its advantages, FL faces several challenges, primarily related to communication efficiency, system heterogeneity, statistical heterogeneity, and privacy concerns [3]. Typically, client data are collected independently and are likely to be Non-Independent and Identically Distributed (non-IID), which significantly degrades the performance of FedAvg. We show that the accuracy of convolutional neural networks trained with FedAvg on strongly non-IID data can drop significantly, by up to \(~{}3.1\%\) for the MNIST dataset, \(~{}13.4\%\) for the Fashion MNIST dataset, and \(~{}29\%\) for the CIFAR-10 dataset. Addressing these challenges is critical to improving the robustness of FL in real-world applications. To mitigate these challenges, we propose two practical optimization strategies aimed at improving the performance of FL under non-IID conditions. The first strategy focuses on applying the Cyclical Learning Rate (CLR) method to dynamically adjust the learning rate during the federated training process [4]. The second strategy involves a novel approach for data sharing and pre-training on augmented data [5; 6]. This method integrates pre-training on augmented data and data sharing to enhance the efficiency of the FedAvg algorithm, especially in non-IID scenarios. Our experiments demonstrate that these methods can increase the accuracy of FL models by \(~{}36\%\) on the CIFAR-10 dataset, while also reducing the number of required communication rounds by \(5.33\) times. Although Federated Learning (FL) provides a distributed strategy for training models that preserve data privacy, most of the existing research focuses on IID (Independent and Identically Distributed) data. Still, in the real world, data are usually non-IID, resulting in statistical heterogeneity and causing a dramatic drop in the performance of FL algorithms, such as FedAvg. Previous studies have tried to manage data heterogeneity using basic aggregation methods or data collaboration strategies, but they usually do an inadequate job of alleviating accuracy loss or require considerable computational resources. By advancing in the area, this study suggests two novel strategies--Cyclical Learning Rate (CLR) and pre-training on augmented data--that effectively handle the difficulties posed by non-IID data and lead to marked enhancements in convergence speed and communication cost.