## 4 Experiments
### Noise Injection
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**(a) CIFAR-10**} & & \\ \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Loss**} & \multicolumn{2}{c}{**Accuracy (\%)**} \\  & **Train** & **Test** & **Train** & **Test** \\ \hline \(1.\) NAG & \(5.64\times 10^{-7}\) & \(1.516\) & \(100.00\) & \(83.68\) \\ Dropout & \(8.10\times 10^{-2}\) & \(0.754\) & \(97.31\) & \(86.54\) \\ Dropout + inputs & \(1.11\times 10^{-1}\) & \(0.700\) & \(96.51\) & \(85.38\) \\ Dropout + BatchNorm & \(2.00\times 10^{-2}\) & \(0.748\) & \(99.34\) & \(\mathbf{88.25}\) \\ \hline \(2.\) Adam & \(2.00\times 10^{-10}\) & \(3.893\) & \(100.00\) & \(79.84\) \\ Dropout & \(3.49\times 10^{-2}\) & \(0.728\) & \(98.89\) & \(83.64\) \\ Dropout + inputs & \(3.90\times 10^{-2}\) & \(1.128\) & \(98.72\) & \(80.77\) \\ Dropout + BatchNorm & \(9.9\times 10^{-3}\) & \(0.904\) & \(99.66\) & \(\mathbf{86.33}\) \\ \hline \(3.\) NAG & \(5.87\times 10^{-2}\) & \(2.007\) & \(100.00\) & \(80.40\) \\ Dropout & \(1.08\times 10^{-2}\) & \(1.099\) & \(99.63\) & \(80.50\) \\ Dropout + inputs & \(1.22\times 10^{-2}\) & \(\mathbf{1.056}\) & \(99.63\) & \(\mathbf{81.26}\) \\ Dropout + BatchNorm & \(1.22\times 10^{-2}\) & \(522.8\) & \(99.59\) & \(67.22\) \\ \hline \hline \multicolumn{5}{c}{**(b) Fashion-MNIST**} & & \\ \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Loss**} & \multicolumn{2}{c}{**Accuracy (\%)**} \\  & **Train** & **Test** & **Train** & **Test** \\ \hline \(1.\) [PERSON] & \(5.84\times 10^{-4}\) & \(0.920\) & \(100.00\) & \(93.27\) \\ Dropout & \(3.41\times 10^{-2}\) & \(0.471\) & \(98.79\) & \(93.59\) \\ Dropout+ inputs & \(6.76\times 10^{-2}\) & \(\mathbf{0.346}\) & \(97.55\) & \(93.32\) \\ Dropout + BatchNorm & \(1.13\times 10^{-2}\) & \(0.412\) & \(99.66\) & \(\mathbf{93.84}\) \\ \hline \(2.\) Adam & \(1.27\times 10^{-1}\) & \(1.029\) & \(100.00\) & \(93.53\) \\ Dropout & \(1.46\times 10^{-2}\) & \(0.485\) & \(99.55\) & \(93.88\) \\ Dropout + inputs & \(2.11\times 10^{-2}\) & \(0.465\) & \(99.27\) & \(93.21\) \\ Dropout + BatchNorm & \(6.60\times 10^{-3}\) & \(\mathbf{0.434}\) & \(99.77\) & \(\mathbf{94.25}\) \\ \hline \(3.\) AdaMax & \(0.00\times 10^{0}\) & \(1.790\) & \(100.00\) & \(\mathbf{192.58}\) \\ Dropout & \(1.18\times 10^{-2}\) & \(\mathbf{0.576}\) & \(99.60\) & \(92.01\) \\ Dropout + inputs & \(1.18\times 10^{-2}\) & \(0.638\) & \(99.62\) & \(92.04\) \\ Dropout + BatchNorm & \(8.10\times 10^{-3}\) & \(0.612\) & \(99.75\) & \(92.12\) \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Performance of models that employ Dropout regularization. Figure 12: The effect of Dropout on baseline models trained on Fashion-MNIST dataset. Although the accuracy results in Table 8 of the most models that use Dropout with the dropping of input units are lower than those that drop only units in the hidden layers, dropping of input units can play an important role in the generalization performance of the network. To demonstrate how the dropping of input units can positively affect the final performance, we construct a new test set consisting of images from the original CIFAR-10 and Fashion-MNIST test sets with some missing pixel values. The new test sets are obtained by setting 10% of random pixel values from each image to 0. A few examples are shown in Figure 15.  
On the new CIFAR-10 test set, _(NAG) Model 1_, which used the Dropout method without dropping input units, achieves 13.47% accuracy, while final evaluation of the model that dropped inputs with probability \(p\) = 0.1 during the training results with 81.28% accuracy on new data. On the new Fashion-MNIST test set, _(Adam) Model 1_ that incorporates Dropout only on hidden units achieves 83.21% accuracy, while the accuracy of one that drops inputs is equal to 93.17%. Figure 14: The effect of the Batch Normalization on the accuracy of models trained on the Fashion-MNIST dataset that incorporate Dropout regularization. Figure 15: Examples from new CIFAR-10 (top) and Fashion-MNIST (bottom) test sets with missing pixel values.