## 4 Experiments
### Noise Injection
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{5}{c}{**(a) CIFAR-10**} \\ \hline \multicolumn{5}{c}{**Loss**} \\ \multicolumn{5}{c}{**Model**} \\ \multicolumn{5}{c}{**Train**} \\ \multicolumn{5}{c}{**Test**} \\ \hline
1. NAG & \(5.64\times 10^{-7}\) & 1.516 & 100.00 & 83.68 \\ DataAugm & \(4.51\times 10^{-2}\) & 0.887 & 98.48 & 87.73 \\ DataAugm + BatchNorm & \(1.57\times 10^{-2}\) & 0.748 & 99.49 & **89.54** \\ DataAugm + Dropout & \(1.61\times 10^{-1}\) & **0.640** & 94.74 & 89.27 \\ DataAugm + BatchNorm + Dropout & \(6.38\times 10^{-2}\) & 0.655 & 97.85 & 89.15 \\ \hline
2. Adam & \(2.00\times 10^{-10}\) & 3.893 & 100.00 & 79.84 \\ DataAugm & \(8.06\times 10^{-2}\) & 0.728 & 97.33 & 86.99 \\ DataAugm + BatchNorm & \(3.80\times 10^{-2}\) & 0.673 & 98.74 & 88.54 \\ DataAugm + Dropout & \(1.54\times 10^{-1}\) & 0.649 & 94.74 & 86.10 \\ DataAugm + BatchNorm + Dropout & \(9.06\times 10^{-2}\) & **0.519** & 96.78 & **88.67** \\ \hline
3. NAG & \(5.87\times 10^{-2}\) & 2.007 & 100.00 & 80.40 \\ DataAugm & \(1.88\times 10^{-2}\) & 0.830 & 99.30 & **86.98** \\ DataAugm + BatchNorm & \(1.66\times 10^{-2}\) & 0.966 & 99.41 & 85.00 \\ DataAugm + Dropout & \(5.72\times 10^{-2}\) & **0.713** & 98.04 & 86.19 \\ DataAugm + BatchNorm + Dropout & \(2.48\times 10^{-1}\) & 12723 & 94.88 & 80.58 \\ \hline \hline \multicolumn{5}{c}{**(b) Fashion-MNIST**} \\ \hline \multicolumn{5}{c}{**Model**} \\ \multicolumn{5}{c}{**Train**} \\ \hline
1. [PERSON] & \(5.84\times 10^{-4}\) & 0.920 & 100.00 & 93.27 \\ DataAugm & \(1.18\times 10^{-2}\) & 0.634 & 99.59 & 92.58 \\ DataAugm + BatchNorm & \(5.30\times 10^{-3}\) & 0.431 & 99.81 & 93.49 \\ DataAugm + Dropout & \(8.93\times 10^{-3}\) & **0.267** & 96.82 & 93.92 \\ DataAugm + BatchNorm + Dropout & \(3.79\times 10^{-2}\) & 0.283 & 98.62 & **94.53** \\ \hline
2. [PERSON] & \(1.27\times 10^{-10}\) & 1.029 & 100.00 & 93.53 \\ DataAugm & \(7.70\times 10^{-3}\) & 0.726 & 99.77 & 93.88 \\ DataAugm + BatchNorm & \(4.00\times 10^{-3}\) & 0.443 & 99.87 & 94.17 \\ DataAugm + Dropout & \(4.12\times 10^{-2}\) & 0.312 & 98.59 & 94.32 \\ DataAugm + BatchNorm + Dropout & \(2.09\times 10^{-2}\) & **0.291** & 99.26 & **94.60** \\ \hline
3. [PERSON] & \(0.00\times 10^{6}\) & 1.790 & 100.00 & 92.58 \\ DataAugm & \(4.40\times 10^{-3}\) & 0.758 & 99.88 & 92.78 \\ DataAugm + BatchNorm & \(2.80\times 10^{-3}\) & 0.508 & 99.91 & 92.77 \\ DataAugm + Dropout & \(5.71\times 10^{-2}\) & **0.372** & 97.96 & 92.54 \\ DataAugm + BatchNorm + Dropout & \(3.55\times 10^{-2}\) & 0.409 & 98.72 & 92.20 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Results obtained using Data Augmentation technique. (during the training, we minimize loss instead of maximizing accuracy because the loss function has some \"nice\" properties such as differentiability). The final accuracy obtained by models that use Early Stopping and those that do not are compared in Table 10. Although model accuracy on new data is enhanced in some cases, it is often the case that performance of models that use Early Stopping on test data declines. However, training time significantly reduces. For achieving better final accuracy, we can use the larger values for the patience parameter. In some sense, the Early Stopping method can be seen as the trade-off between the time of training and the final performance of the model. For example, the accuracy of _Model 1_ trained on CIFAR-10 data with _Dropout_ regularization decreases from 87.73% to 84.51% when Early Stopping is used, but training time reduces more than two times. If the training time is not a problem, the model can be trained for longer with saving the parameters that resulted in the best values of a monitored quantity. In Table 10, we can also see how the Data Augmentation technique yields the best accuracy results compared to other previously mentioned \"good performing\" models that incorporate only one regularizer.