# Advanced Optimization Techniques for Federated Learning on Non-IID Data
## 2 Background
### Adaptive Optimizers  
In federated learning, particularly when dealing with non-IID data distributions, selecting an effective optimization strategy is crucial for ensuring efficient and stable model convergence. While this study focuses on the use of Cyclical Learning Rate (CLR) and pre-training on augmented data, it is important to discuss alternative approaches, such as adaptive optimizers, to provide a comprehensive justification for the methods chosen. One of the primary challenges in federated learning on non-IID data is the heterogeneity of data distributions across clients, which can slow convergence and degrade global model performance. Adaptive optimizers like Adagrad [41] and Adam [42] have been proposed to address these issues by dynamically adjusting the learning rate based on gradient information. These methods compute learning rates for individual parameters using historical gradient data, which helps stabilize training in environments with noisy or non-stationary data. Adagrad, introduced by [PERSON] et al. [41], adapts the learning rate for each parameter by accumulating past squared gradients, allowing it to perform well on sparse or infrequently updated features. This makes Adagrad particularly effective in non-stationary learning environments where certain parameters may need larger updates than others. However, a known limitation of Adagrad is that the learning rate continually decays, which can slow down training in later stages. Adam, developed by [PERSON] and [PERSON] [42], builds on Adagrad by incorporating momentum, where both the first moment (mean of gradients) and second moment (uncentered variance of gradients) are used to adapt learning rates. This combination provides more accurate and stable parameter updates, making Adam one of the most widely used optimizers in machine learning. In centralized settings, Adam has shown fast convergence, especially in tasks with non-stationary objectives. However, in federated learning, particularly under non-IID conditions, the performance of adaptive optimizers like Adam is less consistent. These adaptive optimizers often require accumulating momentum [43] of the previous gradient information for the model update, which may double the uploading communication costs in FL. This is because model training is performed only on local devices and the accumulated gradients (with the same size as the model parameters) also need to be uploaded to the server for aggregation. Research by [PERSON] et al. [44] discusses the bias introduced by pseudo-gradients in Adam when applied in federated environments. In non-IID settings, local updates are aggregated at the server without proper normalization, leading to an unfair bias toward updates from certain clients. This bias can slow down convergence and reduce the fairness of the global model. Furthermore, the paper shows that FedAdam (Adam applied in federated learning) suffers from convergence loss due to the mismatch between local client objectives and the global objective function, particularly in heterogeneous data environments. These issues can result in suboptimal performance and highlight the challenges of using Adam in federated learning [44]. In contrast, the Cyclical Learning Rate (CLR) strategy, proposed by [PERSON] [4], offers significant advantages for federated learning on non-IID data. Unlike adaptive optimizers, CLR oscillates the learning rate between minimum and maximum values, preventing the model from getting trapped in local minima, which is a common issue with biased or skewed data subsets. This cyclical pattern alternates between exploration and fine-tuning phases, leading to more balanced global updates and improved generalization across diverse client distributions. Prior research highlights CLR's effectiveness in enhancing convergence and performance in non-IID federated learning environments [4]. Additionally, CLR practically eliminates the need to tune the learning rate yet achieve near-optimal classification accuracy.