# Advanced Optimization Techniques for Federated Learning on Non-IID Data
## 4 Proposed Approach
### Federated Learning with Cyclical Learning Rate  
To enhance the performance of the Federated Averaging (FedAvg) algorithm, we explore the application of the Cyclical Learning Rate (CLR) strategy within the context of Federated Learning (FL), particularly in scenarios involving non-IID data. By replacing the traditional approach of using a fixed learning rate, we investigate how dynamically adjusting the learning rate in cycles can optimize model performance and improve convergence. The concept of cyclical learning rate, introduced by [PERSON] in [PERSON] (2018), suggests that a cyclical learning rate policy can improve accuracy without the need for manual learning rate tuning. This approach allows the learning rate to oscillate between a specified minimum and maximum value, helping the model to escape local minima and converge to better solutions. Non-IID data cause significant gradient variability during local updates due to the imbalance in data distributions across clients. In such scenarios, some clients may have data focused on a small subset of classes, while others may have data from completely different classes. Therefore, each client, optimizes the model based on different objectives, causing weight divergence, a phenomenon where local models diverge from each other and the global model after each communication round. As demonstrated by [PERSON] et al. (2018) [PERSON] et al. (2018), due to this weight divergence, the skewed data across clients can lead to significant performance degradation. In contrast, in IID settings, the client data distributions are similar, resulting in more aligned gradient updates across clients, leading to smoother convergence. However, in non-IID settings, the gradients push the global model in different and often conflicting directions, causing instability and slower convergence. Moreover, the study [PERSON] et al. (2018), highlights that as the degree of heterogeneity increases, gradient conflicts become more pronounced, which amplifies this divergence and further degrades convergence. As a result of this variability, we can run into several issues, including local overfitting and the skewed role of local models in building one global model. Some challenges and issues are:  
* In federated learning (FL), non-IID data can critically degrade performance and converge ([PERSON] et al., 2018). The non-IID case means that the data of each client's datasets may highlight different patterns, so the gradients in local training will be divergent. Experiments with highly skewed data performed with neural networks show that this divergence can lower the accuracy of the global model by up to 55% in some cases [PERSON] et al. (2018). * It also increases communication overhead and has non-IID data that will yield an imbalance in class distributions, which could further impact the convergence and performance of FL. Non-IID data can not only introduce bias but also slow down convergence, regardless of whether batch normalization is used, because the local and global statistical parameters do not align [PERSON] et al. (2018). * More importantly, non-IID data cause fluctuations in historical gradient information, which then results in inconsistent convergence. This is why techniques such as federated gradient scheduling have been suggested for dealing with these challenges by creating IID gradient sets for more stable updates [PERSON] et al. (2018). * Local Overfitting: Clients with other distributions of data are likely to overfit their local models, which, when assembled form the global model, will perform poorly. For example, the simple task of evaluating the performance of multi-modal models concerning uni-modal models showed that the former may be more harmful as a result of this overfitting [PERSON] et al. (2018). * Skewed Contributions: When the clients hold different distributions of data, some of the clients contribute to the global model than others, thus distorting the learning process.