## 4 Experiments
### Batch Normalization
#### 4.2.2 Evaluation of Different Regularization Techniques  
In this section, we add different types of regularization into the chosen baseline model architectures to examine their effect on the model's generalization performance. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{3}{c}{**(a) CIFAR-10**} \\ \hline  & \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Loss**} & \multicolumn{2}{c}{**Accuracy (\%)**} \\  & & **Train** & **Test** & **Train** & **Test** \\ \hline
1. & NAG & \(5.64\times 10^{-7}\) & 1.516 & 100.00 & 83.68 \\  & + BatchNorm & \(2.34\times 10^{-5}\) & **0.728** & 100.00 & **86.45** \\ \hline
2 & Adam & \(2.00\times 10^{-10}\) & 3.893 & 100.00 & 79.84 \\  & + BatchNorm & \(8.96\times 10^{-5}\) & **2.203** & 100.00 & **82.89** \\ \hline
3. & NAG & \(5.87\times 10^{-7}\) & 2.007 & 100.00 & 80.40 \\  & + BatchNorm & \(2.25\times 10^{-6}\) & **1.633** & 100.00 & **81.21** \\ \hline \multicolumn{5}{c}{**(b) Fashion-MNIST**} \\ \hline  & \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Loss**} & \multicolumn{2}{c}{**Accuracy (\%)**} \\  & & **Train** & **Test** & **Train** & **Test** \\ \hline
1. & Adam & \(5.84\times 10^{-4}\) & 0.920 & 100.00 & **93.27** \\  & + BatchNorm & \(1.70\times 10^{-3}\) & **0.405** & **99.96** & **93.25** \\ \hline
2 & Adam & \(1.27\times 10^{-10}\) & 1.029 & 100.00 & 93.53 \\  & + BatchNorm & \(6.10\times 10^{-5}\) & **0.455** & 100.00 & **93.61** \\ \hline
3. & AdaMax & \(0.00\times 10^{0}\) & 1.790 & 100.00 & **92.58** \\  & + BatchNorm & \(1.20\times 10^{-3}\) & **0.724** & 99.96 & 91.85 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of results obtained with and without Batch Normalization. Figure 5: The effect of Batch Normalization on the loss of baseline models trained on CIFAR-10 dataset. Figure 6: The effect of Batch Normalization on the loss of baseline models trained on Fashion-MNIST dataset. Weight Decay  
Adding \(L_{2}\) and \(L_{1}\) regularization into the baseline models did not, in general, result in the improvement of generalization performance. As we can see in Table 5, on the Fashion-MNIST dataset, neither \(L_{2}\) nor \(L_{1}\) regularization leads to an increase in the test set accuracy. However, applying \(L_{1}\) regularization on the CIFAR-10 leads to increased accuracy and decreased loss on the test data in baseline _Model 1_, while adding \(L_{2}\) regularization has beneficial effect on the performance of baseline _Models 2_ and 3. As parameter \(\lambda\), for both \(L_{2}\) and \(L_{1}\) regularization, the best performing value on the validation set from a predefined set of \(\lambda\) values \(\{10^{-2},10^{-3},10^{-4},10^{-5},10^{-6}\}\) was chosen. If two neighbor values were close in performance on validation data, we additionally investigated the performance of their midpoint on validation data as potential value for the \(\lambda\) parameter. Figures 7 and 8 show that both \(L_{2}\) and \(L_{1}\) regularization reduce all six models' validation loss during the training. In the third model in Figure 7, penalizing models' weights notably slows down the convergence; the model needs more than 300 epochs to reach the loss that the baseline model reached before epoch 50. However, after epoch 200, the penalized model's validation loss falls below the baseline's validation loss despite the slower learning process. From the results in Figure 9 and Table 6, we can gain insight into the effect that added weight penalties have on final model weights. Obtained results justify the name _weight decay_; in both cases, resulting weights of regularized models are closer to 0 than the baseline (_Model 1 with used NAG optimizer_) weights. Most weights of the model that uses \(L_{1}\) regularization are \(\approx\)0; the obtained model has _sparse_ weights.