# Advanced Optimization Techniques for Federated Learning on Non-IID Data
## 3 Methodology
### CNN Models  
This section presents the architectures of the Convolutional Neural Networks (CNNs) applied for federated learning on three different datasets: MNIST, Fashion MNIST, and CIFAR-10. Each architecture is specifically designed to meet the unique challenges of its respective dataset.  
* **MNIST:** The CNN model for the MNIST dataset starts with a reshape layer to convert images from 1D vectors of 784 elements into 28 \(\times\) 28 \(\times\) 1 tensors. It includes two convolutional layers with 5 \(\times\) 5 filters (10 and 20 filters, respectively), each followed by a 2 \(\times\) 2 max-pooling layer. A flattened layer then converts the features into a 1D vector, which connects to a dense layer with 50 neurons. The model ends with an output layer of 10 neurons, representing the 10 classes of digits (0-9) in MNIST. The softmax activation function is used to produce a probability distribution across the classes.
* **Fashion MNIST:** The CNN model for the Fashion MNIST dataset follows a similar approach to that of MNIST but with increased depth, reflecting the added complexity of the images. This model uses three convolutional layers with 16, 32, and 64 filters of size 5 \(\times\) 5. Max-pooling layers with 2 \(\times\) 2 windows are applied after the first two convolutional layers, and another pooling layer follows the last convolutional layer. This arrangement enhances the model's ability to identify the more intricate features in Fashion MNIST images. After flattening the output, the model includes a dense layer with 64 neurons, followed by a softmax output layer with 10 neurons, corresponding to the 10 different clothing categories.
* **CIFAR-10:** The CNN model for the CIFAR-10 dataset is designed to address the challenge of processing higher-resolution color images (32 \(\times\) 32 pixels). It begins with a reshape layer that converts the 1D vectors of 3072 elements into 32 \(\times\) 32 \(\times\) 3 tensors. This is followed by three convolutional layers with 32, 64, and 64 filters of size 3 \(\times\) 3. Two 2 \(\times\) 2 max-pooling layers are included after the first and second convolutional layers to reduce complexity. The features are then flattened into a 1D vector, which feeds into a fully connected layer with 64 neurons. The model concludes with a softmax output layer of 10 neurons, representing the 10 CIFAR-10 classes.  
Across all these models, ReLU (Rectified Linear Unit) was chosen as the activation function due to its ability to introduce non-linearity effectively, while simplifying computations by zeroing out negative values, thus making the models more efficient for training complex image patterns. Sparse categorical cross-entropy was used as the loss function, which is well-suited for multi-class classification problems where each sample belongs to only one of the available classes, and the model must determine the correct class. The optimizer chosen was Stochastic Gradient Descent (SGD), which was valued for its simplicity and effectiveness in image classification tasks. This optimizer iteratively adjusts the model's weights based on the gradient of the loss function, improving the model's accuracy over time.  
Each model is tailored to its dataset, highlighting the versatility of CNNs in image-processing tasks and the importance of selecting the right hyperparameters, especially in federated learning environments.