# Optimizing Activation Function for Parameter Reduction in CNNs on CIFAR-10 and CINIC-10
## 3 Architecture
### Parameters  
As presented in Section 2, the number of parameters required to achieve similar levels of accuracy has been significantly reduced. By optimizing the number of learnable parameters, this architecture can achieve a performance that is comparable to the most advanced models in the field, while being much more efficient. The number of learnable parameters in the three convolutional layers within each ConvBlock has been reduced to a minimal value. The number of a convolutional layer is calculated using the following formula:  
\[P_{\text{conv}}=(F_{h}\times F_{\text{w}}\times C_{\text{in}}\times C_{\text {out}})+C_{\text{out}} \tag{19}\]  
In this formula \(\text{F}_{\text{h}}\) and \(\text{F}_{\text{w}}\) are the height and width of the kernel, while \(\text{C}_{\text{in}}\), \(\text{C}_{\text{out}}\) correspond to the input and output channels, respectively. The term \(\text{C}_{\text{out}}\) is added to account for the bias parameters, with one bias per output channel.  
This formula helps in determining the number of parameters needed for each convolutional layer, considering factors like the size of the filter, the number of filters, and the input and output dimensions.  
In addition to the convolutional layers, the Batch Normalization layers and the proposed activation function EPU are also parameter-dependent in ConvBlock. The Batch Normalization layer adds two learnable parameters per channel (mean and standard deviation), while EPU includes three additional learnable parameters. This is the result of the way they are configured. Therefore, the total number of parameters in a ConvBlock is the sum of the parameters of all convolutional layers, the parameters from Batch Normalization and the parameters from the activation function:  
\[P_{\text{total}}=P_{\text{conv}}+P_{\text{EPU}}+P_{\text{BN}} \tag{20}\]  
In turn, in a fully connected neural network (FFNN), the number of parameters depends on the number of neurons in each layer and the connections between them. Each layer contains two types of learnable parameters: weights and biases. The weights define the strength of the connections between neurons in different layers, while the biases adjust the output of each neuron to improve the model's ability to fit the data. The number of weight parameters for a certain layer in an FFNN is calculated as follows:  
\[\text{W}_{1}=\text{N}_{\text{in}}\times\text{N}_{\text{out}} \tag{21}\]  
where \(N_{in}\) and \(N_{out}\) are  
Figure 5a illustrates the architecture of a classical perceptron or one (hidden) unit, where IN1 to INn represent the input nodes, w1 to wn correspond to the associated weights, and b1 denotes the bias term. \(\Sigma\)(x) represents the summation function, while f(x) is the activation function that introduces non-linearity.  
For example, Figure 5b considers a connection between two layers where the first layer contains five hidden units and the second layer contains six. The total number of parameters would be calculated as 5 \(\times\) 6 + 6 = 36, accounting for both the weights and biases. In our particular case, the system has two of these connections, as previously mentioned.  
The first connection in our network has an input dimensionality (Nin) of 1024 and an output dimensionality (Nout) of 128. The second connection then takes this 128-dimensional feature representation as its input (Nin = 128) and maps it to an output of size 10 (Nout = 10), corresponding to the number of distinct classes in our dataset.  
As shown in Table 1, the number of parameters in our proposed architecture's network blocks is as follows: