## 4 Experiments
### Batch Normalization
#### 4.2.2 Evaluation of Different Regularization Techniques
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{3}{c}{**(a) CIFAR-10**} \\ \hline  & \multicolumn{1}{c}{**Model**} & \multicolumn{2}{c}{**Loss**} & \multicolumn{2}{c}{**Accuracy (\%)**} \\  & & **Train** & **Test** & **Train** & **Test** \\ \hline
1. & NAG & \(5.64\times 10^{-7}\) & \(1.516\) & \(100.00\) & \(83.68\) \\  & \(L_{2,\lambda}=5\times 10^{-6}\) & \(1.26\times 10^{-2}\) & \(13.615\) & \(100.00\) & \(83.33\) \\  & \(L_{2,\lambda}=10^{-6}\) & \(2.99\times 10^{-2}\) & \(\mathbf{1.477}\) & \(100.00\) & \(\mathbf{83.83}\) \\ \hline
2. & Adam & \(\mathbf{0.0}\times 10^{-8}\) & \(3.893\) & \(100.00\) & \(79.84\) \\  & \(L_{2,\lambda}=10^{-5}\) & \(2.90\times 10^{-2}\) & \(1.483\) & \(100.00\) & \(\mathbf{79.94}\) \\  & \(L_{2,\lambda}=10^{-6}\) & \(9.27\times 10^{-2}\) & \(\mathbf{1.339}\) & \(100.00\) & \(76.40\) \\ \hline
3. & NAG & \(5.87\times 10^{-7}\) & \(2.007\) & \(100.00\) & \(80.40\) \\  & \(L_{2,\lambda}=10^{-5}\) & \(7.76\times 10^{-2}\) & \(\mathbf{1.569}\) & \(100.00\) & \(\mathbf{80.70}\) \\  & \(L_{1,\lambda}=10^{-6}\) & \(2.15\times 10^{-1}\) & \(1.650\) & \(100.00\) & \(79.99\) \\ \hline \hline \end{tabular}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{3}{c}{**(b) Fashion-MNIST**} \\ \hline  & \multicolumn{3}{c}{**Loss**} & \multicolumn{2}{c}{**Accuracy (\%)**} \\  & & **Train** & **Test** & **Train** & **Test** \\ \hline
1. & Adam & \(5.84\times 10^{-4}\) & \(0.920\) & \(100.00\) & \(\mathbf{93.27}\) \\  & \(L_{2,\lambda}=5\times 10^{-6}\) & \(2.09\times 10^{-2}\) & \(0.593\) & \(100.00\) & \(92.56\) \\  & \(L_{2,\lambda}=5\times 10^{-6}\) & \(5.37\times 10^{-2}\) & \(\mathbf{0.515}\) & \(99.66\) & \(92.05\) \\ \hline
2. & Adam & \(1.27\times 10^{-10}\) & \(1.029\) & \(100.00\) & \(\mathbf{93.53}\) \\  & \(L_{2,\lambda}=5\times 10^{-5}\) & \(3.64\times 10^{-2}\) & \(\mathbf{0.462}\) & \(99.69\) & \(92.11\) \\  & \(L_{2,\lambda}=10^{-6}\) & \(1.51\times 10^{-2}\) & \(0.670\) & \(100.00\) & \(93.20\) \\ \hline
3. & AdaMax & \(0.00\times 10^{9}\) & \(1.790\) & \(100.00\) & \(\mathbf{92.58}\) \\  & \(L_{2,\lambda}=10^{-6}\) & \(2.50\times 10^{-2}\) & \(0.843\) & \(100.00\) & \(92.39\) \\  & \(L_{2,\lambda}=10^{-6}\) & \(1.61\times 10^{-2}\) & \(\mathbf{0.581}\) & \(99.98\) & \(91.99\) \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance of models that use weight decay regularization. Figure 7: Loss learning curves of models that incorporate \(L_{2}\) and \(L_{1}\) weight decay in baseline models trained on CIFAR-10 dataset. Interestingly, the \(L_{1}\) model has the most dispersed weights and the widest range of absolute values of weights. This can be seen as _feature selection_; in each layer, weights that correspond to irrelevant features are set to \(\approx\)0 while the more important features are emphasized by increasing their (absolute) value and, in that way, also increasing their influence on the final output. \(L_{2}\) regularization has a similar effect. The main difference is that in the case of \(L_{2}\) regularization less important weights are \"pulled\" towards 0 but not really set to 0.