# Optimizing Activation Function for Parameter Reduction in CNNs on CIFAR-10 and CINIC-10
## 3 Architecture
### The Proposed Architecture  
In this subsection, we present our CNN structure, which contains only three convolutional layers, one fully connected layer, and the output layer. Each convolutional layer follows the same structure, which we refer to as a ConvBlock, visualized in Figure 4.  
As already mentioned, the input image is presented in a two-dimensional format with a size of 32 \(\times\) 32 pixels. Each pixel contains information for three color channels (R, G, and B), with 8 bits allocated for each channel, for a total of 24 bits per pixel. We will write the input information to the first ConvBlock for processing as a tensor with dimensions 32 \(\times\) 32 \(\times\) 3.  
The information is first processed through a convolutional layer (filter) that is configured with a kernel size of 5, a stride equal to 1, and padding set to \"same\". This configuration preserves the spatial dimensions of the image. The output of this layer consists of a c number of (color) channels, which determines the number of features that will be extracted from the input image. After performing the convolution, the output, organized into a tensor with dimensions (h, w, and c)-with h and w representing the spatial dimensions-is fed directly to the proposed activation function EPU for further processing. After convolution, the data are subjected to pooling, specifically max pooling, which is applied with kernel size 2 and stride 2. This reduces the spatial dimensions significantly, which helps to lower the computational complexity and eliminate redundant information while preserving the most important features for the subsequent processing steps. After the convolution and merging operations are completed, the data undergo Batch Normalization. This process normalizes the values in the tensor with respect to the mean and standard deviation, as previously described by Formulas (1)-(3).  
The whole network has a total of three such ConvBlocks. After being processed, the input information (image) of size (32 \(\times\) 32 \(\times\) 3) is resized to (4 \(\times\) 4 \(\times\) 64), where 64 is the value of c, for the last ConvBlock. This final tensor is considered a feature vector which contains (4 \(\times\) 4 \(\times\) 64), 1024 elements in total. Each element of this vector represents a numerical feature extracted from the input image that summarizes its essence and is ready for further processing by the fully connected layer and the output layer of the network. This feature vector is the input to the fully connected layer containing 128 hidden units, this time followed by ReLU activation function instead of EPU. The output from the fully connected layer then goes to the output layer of the network, which is again FF, but this time with only 10 hidden units because that is the number of classes in the dataset. In order  
Figure 4: Overview of so-called ConvBlock.  
to make a prediction, the output from the output layer goes through a softmax activation function that looks like this:  
\[\text{softmax}(x_{i})=\frac{e^{x_{i}}}{\sum_{j=1}^{n}e^{x_{j}}} \tag{18}\]  
It provides the probability for each class. The class with the highest probability is set as the final answer.