# Advanced Optimization Techniques for Federated Learning on Non-IID Data
## 4 Proposed Approach
### Federated Learning with Data Sharing and Pre-Training on Augmented Data  
To more effectively address the challenges posed by non-IID data in federated learning, we propose another strategy that combines pre-training the global model on augmented data and sharing a small, balanced subset of data among all clients. This strategy aims to mitigate the significant accuracy degradation observed in neural networks trained on highly heterogeneous data by providing a strong starting point for model training. Additionally, we apply the Cyclical Learning Rate (CLR) strategy in conjunction with this pre-training and data-sharing approach to further enhance the model's generalization capabilities in non-IID environments. Our strategy is inspired by insights from previous studies [5; 6]. Specifically, in [5], it is highlighted that pre-training, either with synthetic or decentralized client data, improves the performance of Federated Learning (FL) models by reducing the accuracy gap between FL and centralized learning, while also stabilizing global aggregation, especially in scenarios with non-IID client data. Furthermore, in [6], it is proposed that by creating a small balanced subset of data, training a warm-up model on this subset, and then distributing both the pre-trained model and a portion of the data to clients can help overcome the challenges of data heterogeneity in FL. Building on these insights, we developed a combined approach to improve the performance of FedAvg in non-IID data scenarios. To implement the proposed strategy, we followed these steps:  
1. **Creation of a Balanced Data Subset:** The strategy begins by selecting a balanced subset of the CIFAR-10 dataset. Specifically for this implementation, 10,000 images are selected to uniformly represent all 10 classes of CIFAR-10 (1000 images per class). The remaining 40,000 images are distributed in a non-IID manner among 500 clients, with each client receiving 80 images from two distinct classes (40 from each class). 2. **Distribution of the Balanced Subset Among Clients:** Next, a portion of the balanced subset is randomly distributed to individual clients. Each client is provided with 0.2% of the balanced subset of 10,000 images (20 random images), thus increasing each client's dataset to 100 images. These 20 images are randomly distributed to each client without adherence to IID or non-IID partitioning methods. 3. **Data Augmentation:** Then, multiple data augmentation techniques are applied to this balanced subset. These techniques aim to increase the diversity of the training data and include: * **Random Crop with Padding:** Adds padding to the original images with a random margin between 3 and 7 pixels followed by random cropping to a 32 \(\times\) 32 pixel area. This method introduces spatial variation in the dataset, helping the model recognize objects despite positional changes. * **Horizontal Flip:** Reflects the image along its vertical axis, effectively increasing the dataset size and improving the model's ability to recognize objects regardless of horizontal orientation. * **Brightness and Contrast Adjustment:** Randomly adjusts pixel values within a predefined range (0.8 to 1.2) to simulate different lighting conditions, enhancing the model's robustness to variations in lighting. * **Random Rotation and Scaling:** Applies random rotations between \(-15\) and \(15\) degrees and scaling images within a range of 0.8 to 1.2. This prepares the model to handle objects at various angles and sizes. * **Random Noise:** Adds random pixel values ranging from 8 to 15 to simulate lower-quality images, increasing the robustness of the model in real-world scenarios where perfect image conditions are not always guaranteed. Through the application of these augmentation techniques, as illustrated in Figure 1, an augmented dataset is created, enriched with a broad range of variations of the images from the balanced subset of CIFAR-10.