# Advanced Optimization Techniques for Federated Learning on Non-IID Data
## 4 Proposed Approach
### Federated Learning with Cyclical Learning Rate
Consequently, this can intensify the problems associated with the convergence and, in general, the entire model's efficiency ([PERSON] et al., 2018). * These challenges are addressed by proposing strategies of data sharing, optimal node selection, and new aggregation schemes to reduce accuracy and convergence time ([PERSON] et al., 2018). Therefore, complexities arising from non-IID datasets include gradient variations throughout training for the very reasons that such datasets are heterogeneous, samples in non-IID datasets are inherently correlated, and the potential for class imbalance. It is also important to understand these dynamics to design measures for stabilizing corresponding training processes and increasing the quality of models for application to such datasets. As a result, we suggest applying the Cyclical Learning Rate (CLR), which is an adaptive learning rate technique in which the learning rate oscillates between some minimum and maximum values, which was presented by [PERSON] et al. in [PERSON] et al. (2018). Our proposal CLR enabled the model to periodically leap to larger learning rates, useful in escaping certain local minima or tackle points that could be caused by non-IID clients' non-uniform gradient updates. Also, low learning rates are particularly useful to provide more stable convergence to the model while fine-tuning. This approach serves to eliminate direction oscillation that causes limited contribution, which, in turn, helps in the convergence of the model to the right solution through high variance reduction over different gradients and high generalization over different distributions of data. Furthermore, CLR does not require a great number of hyperparameters and adapts the learning rate with cycles, which lets us achieve both exploration (large learning rates) and exploitation (small learning rates). From the experimental evidence also supported in Section 5, it has been shown that incorporating CLR in federated learning on non-IID data has the advantage of increased model accuracy and smaller communication rounds than the use of the fixed learning rate. The state-of-the-art FL and IoT for the large-scale systems, namely FLIBD ([PERSON] et al., 2018), TinyML algorithms ([PERSON] et al., 2018), and distributed Bayesian inference classifiers ([PERSON] et al., 2018), ensures enhanced privacy-preservation, improved efficiency of big data management, and scalability in decentralized systems. Also, federated edge intelligence and edge caching methods have introduced new optimization techniques in resource-limited and non-IID data settings ([PERSON] et al., 2018). To determine the appropriate learning rate boundaries, we utilized the Learning Rate Range Test (LRRT). This method involves gradually increasing the learning rate over a brief training period while monitoring the model's loss. The evaluation loss is then plotted against the learning rate, and two key points are identified: the learning rate where the loss starts decreasing and the learning rate where the loss begins to plateau or increase. These two values serve as the minimum and maximum learning rate boundaries for the cyclical learning rate policy. Various CLR policies have been explored, including triangular, triangular2, and exp_range. These policies differ in how the learning rate changes within each cycle:  
* **Triangular**: the learning rate increases linearly during the first half of the cycle and decreases linearly during the second half. * **Triangular2**: similar to triangular, but the difference in the learning rate is halved at the end of each cycle. * **Exp_range**: the learning rate oscillates between the minimum and maximum values, with each boundary value decaying by an exponential factor. The cycle length is determined by the number of rounds required to complete a full oscillation from the minimum to the maximum learning rate and back again. [PERSON] recommends running for four or more cycles, as this generally yields better performance.