# Optimizing Activation Function for Parameter Reduction in CNNs on CIFAR-10 and CINIC-10
## 3 Architecture  
### Exponential Partial Unit (EPU)  
During the development of neural networks, it was found that their efficient training requires the use of nonlinear activation functions. The most popular of these is ReLU (Rectified Linear Unit) [16]. Its popularity is attributed to its simplicity and efficiency, which are reflected in high accuracy and easy computation. \[ReLU(x)=max(0,\ x), \tag{1}\]  
However, this simplicity can lead to problems. One of them is the so-called exploding gradient, when the error gradients become extremely large, leading to excessively large changes in the model weights. This can make the model unstable--in extreme cases the weights reach huge values or even cause numerical errors that interrupt training. The solution to this problem lies in limiting the activation values to a certain maximum. \[Clipped\ ReLU(x)=min(ReLU(x),\mathrm{a}) \tag{2}\]  
where **a** is the coefficient of clipping. But this solution does not solve the other existing problem known in the community as the \"Dying ReLU\" problem. This means that the weighted sum of the inputs to these neurons consistently results in a negative value, causing the ReLU activation to output zero. Once a neuron becomes inactive, it effectively stops learning since the gradient during backpropagation is zero for negative inputs, due to the zero-derivative generated by the ReLU philosophy. There are also solutions to this, such as Leaky ReLU [17], which adds a small gradient for negative numbers so that the slope is not 0. \[\mathrm{Leaky\ ReLU}(x)=\max(\alpha x,x) \tag{3}\]  
\(\alpha\) is a constant in this equation, usually a small number. Other related solutions include ELU (Exponential Linear Unit) [18] and SeLU (Scaled Exponential Linear Unit) [19]. However, there are activation functions that are trained during training offering additional benefits by allowing the network to optimize the activation parameters depending on the specific characteristics of the data and the training process, resulting in better performance and stability of the models. Examples of such features are PReLU (Parametric ReLU) [20] and LeLeLU (Learnable Leaky ReLU) [21]. This paper presents a new activation function called Exponential Partial Unit (EPU), which is based on the exponential function. Although it does not belong to the ReLU family, it combines low computational complexity and greater flexibility with respect to real data. The formula that describes the EPU is as follows:  
\[\mathrm{EPU(x)=clip}\Big{(}e^{kx},\theta_{\min},\theta_{\max}\Big{)} \tag{4}\]  
where \(e^{kx}\) represents the exponential function with base **e** raised to the power of the scaling factor **k** times the input variable. \(\theta_{min}\) and \(\theta_{max}\) represent the minimum and maximum bounds, respectively. The clip function is defined as  
\[clip\Big{(}e^{kx},\ \ \theta_{min},\ \theta_{max}\Big{)}= \begin{cases}\theta_{min},&if \ e^{kx}<\theta_{min}\\ \ e^{kx},&if\ \theta_{min}<e^{kx}<\theta_{max}\\ \theta_{max},&if\ e^{kx}>\theta_{max}\end{cases}  \tag{5}\]In the proposed function, the parameters k, \(\theta_{min}\) and \(\theta_{max}\) are defined as learnable parameters, allowing them to be dynamically optimized during the training process, all initialized following uniform distribution. The boundary of initialization of these parameters differs from each other. The parameter min is initialized within a negative range, ensuring that it acts as a lower bound when clamping input values, between \(-5\) and \(-1\). \[\theta_{min}\sim\mathcal{U}(-5,1) \tag{6}\]  
Symmetrically, \(\theta_{max}\) is initialized within a positive range from 1 to 5. By enforcing \(\theta_{min}\)\(<\theta_{max}\) at initialization, the function ensures a well-defined input transformation. \[\theta_{max}\sim\mathcal{U}(1,5) \tag{7}\]  
Meanwhile, k is initialized within a small positive range, controlling the scaling factor in the exponential function.