# Optimizing Activation Function for Parameter Reduction in CNNs on CIFAR-10 and CINIC-10
## 4 Training and Results
### Assessing Model Generalization: A Comparative Analysis Across Activation Functions  
To evaluate the overall generality of our architecture, we train the model on both CIFAR-10 and CINIC-10. In addition, we compare several popular activation functions--such as ReLU, Leaky ReLU, and ELU--with our proposed activation function, EPU.  
Figure 10: Plot of true class vs. predicted labels.  
Figure 9: Visualization of the 2D projection of learned image embeddings using principal component analysis (PCA).  
By incorporating CINIC-10, a dataset known for its increased diversity and complexity relative to CIFAR-10, we aim to examine how well the model generalizes across different data distributions. The comparative analysis focuses on assessing the impact of each activation function on the model's performance and generalization capabilities. Specifically, we analyze whether our proposed EPU function can yield improvements over traditional activation functions by facilitating more robust feature learning and reducing the generalization gap between training and test performance.  
This comprehensive evaluation will help us understand the interplay between dataset complexity, network architecture, and activation function choice, and ultimately guide the development of models that perform reliably in real-world scenarios.  
Table 3 provides the top-1 accuracies on the CIFAR-10 dataset after training for 250 epochs.  
Figure 11 illustrates the test accuracy for different activation functions on the CIFAR-10 dataset. This figure compares the performance of various activation functions, providing insight into their impact on the overall accuracy of the model.  
It can be observed that our proposed function, EPU, initially performs slightly worse than the other activation functions. This is due to the fact that CIFAR-10 contains very few images (i.e., it is a small dataset), and since EPU is fine-tuned during training, achieving its best performance requires either more images or more training iterations.  
These arguments become evident when training on CINIC-10, a dataset that is several times larger than CIFAR-10. With CINIC-10, while the performance of all activation functions tends to decline--likely due to model saturation--the model using EPU continues to improve steadily and achieves the highest accuracy percentage, as shown in both Table 4 and Figure 12.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Activation Function** & **Accuracy (\%)** \\ \hline EPU & 78.32 \\ ReLU & 78.77 \\ LeakyReLU & 79.3 \\ ELU & 79.47 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Top-1 accuracies on CIFAR-10 for 250 epochs.  
Figure 11: Test accuracy for different activation functions on CIFAR-10.  
This outcome not only highlights the scalability of the EPU function with larger datasets but also underscores the importance of adequate data volume and sufficient training iterations for adaptive activation functions. Further experiments with even larger datasets or extended training periods could potentially demonstrate additional advantages of EPU over conventional activation functions.