# Optimizing Activation Function for Parameter Reduction in CNNs on CIFAR-10 and CINIC-10
## 3 Architecture
### Regularization  
Oversaturation has always plagued neural network designers. Ordinary users would never realize how much time and effort is required for solving high-variance problems. High variance and high bias are two major components of prediction error. A high-variance model pays too much attention to the training data and fails to perform well on new, unseen data. This can lead to very high scores on the training data, but low scores on validation and tests. On the other hand, a high-bias model pays too little attention to the training data and makes overly simplistic inferences that do not reflect the complexity of the problem well. This leads to large errors in all data distributions. These two types of errors--variance and bias--create a tradeoff called the bias-variance tradeoff, shown in Figure 3. When we try to reduce one error, it often leads to an increase in the other. To find the perfect balance, various regularization methods have emerged. Among the most well-known methods are L2 regularization [27] and Dropout [28]. As presented in [29], L2 shows better results in some cases, while Dropout works better in others. To address the bias-variance tradeoff effectively, we use both methods in this study. The idea behind Dropout is simple: During training some neurons are \"turned off\". At each training step, a random subset of neurons in a layer is selected and temporarily deactivated. These neurons do not contribute to either forward propagation or backpropagation during that step. By deactivating neurons at random, the network is forced to rely on a wider range of neurons rather than depending too much on specific ones. This encourages the model to learn more robust and diverse patterns from the data, making it better at handling new, unseen examples. Figure 3: Bias–Variance Tradeoff in Machine Learning. Blue circles represent training data; red lines show model predictions. Left: overfitting (high variance). Middle: underfitting (high bias). Right: good balance (low bias and variance). The dropout rate is a hyperparameter that determines the probability of a neuron being dropped. For instance, a dropout rate of 0.2 means that 20% of neurons will be deactivated at each training step. Sometimes, an improperly chosen dropout rate can significantly impact the effectiveness of training, potentially deteriorating the model's performance. To compensate for the absence of the dropped neurons, the outputs of the remaining active neurons are scaled up. For instance, if 50% of neurons are deactivated, the outputs of the remaining active neurons are scaled up by a factor equal to the probability of keeping a neuron active. This helps maintain the network's overall functionality and balance during training. During testing, dropout is turned off. All neurons are active, but their weights are adjusted to match the expected behavior during training. In our study, wherever dropout is used, the dropout rate is set to 0.2. Besides Dropout, as mentioned earlier, another method of regularization is used--L2 Regularization also known as Weight Decay. In our case, we limit the complexity of the model with this method by limiting the update of the weights. This function takes away from the complexity of the model by adding the element called the regularization term. The regularization term is added to the underlying cost function, thus affecting the optimization process. The standard cost function J(W), which measures the difference between predicted and actual output values, can be extended by adding the L2 regularization. Thus, the cost function becomes  
\[J=\frac{1}{m}\sum_{l=1}^{m}\mathcal{L}(\hat{y}_{i},y_{i})+\frac{\lambda}{2m} \sum_{l=1}^{L}|w^{[l]}|_{2}^{2} \tag{14}\]  
This addition to the cost function results in smaller weights, which in turn reduces the ability of the model to overfit to specific examples in the training data. \(\lambda\) is a regularization hyperparameter that controls the balance between the model's accuracy on the training data and its complexity.