## 4 Experiments
### Results
#### 4.2.1 Evaluation of Optimization Algorithms  
The following observations about the influence of used optimization algorithm on the behavior and final generalization performance of the CNN model are based on their empirical evaluations on three different model architectures, each trained on two datasets with the nine distinct optimizers reviewed in Section 2. Hyperparameters of optimizers used for training each model are given in Appendix B. Figures 3 and 4 show loss and accuracy learning curves of given models, and the final results on the test and training set are reported in Tables 2 and 3. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model 1** & **Model 2** & **Model 3** \\ \hline Conv 96, \(3\times 3\) & Conv 64, \(3\times 3\) & Conv 96, \(3\times 3\) \\ Conv 96, \(3\times 3\) & Conv 128, \(3\times 3\) & MaxPooling \\ MaxPooling, & MaxPooling & Conv 256, \(3\times 3\) \\ Conv 192, \(3\times 3\) & Conv 128, \(3\times 3\) & MaxPooling \\ Conv 192, \(3\times 3\) & Conv 256, \(3\times 3\) & Conv 384, \(3\times 3\) \\ MaxPooling & MaxPooling & Conv 384, \(3\times 3\) \\ Conv 192, \(3\times 3\) & FC 128 & Conv 256, \(3\times 3\) \\ Conv 192, \(1\times 1\) & FC-Softmax 10 & FC 4096 \\ Conv 10, \(1\times 1\) & & FC 4096 \\ GlobalAveraging & & FC-Softmax 10 \\ FC-Softmax 10 & & \\ \hline \(\approx\)955 K params & \(\approx\)2.1M params & \(\approx\)56 M params \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Baseline model architectures. The following observations are made:  
1. The best test set results, in terms of accuracy, are obtained using the classical Nesterov optimization algorithm and adaptive optimization algorithm Adam and its variant AdaMax. 2. Compared to Nesterov optimization algorithm, Adam and AdaMax show less stable performance (with many \"jumps\") on validation data. The most stable, not necessarily the best, performance on validation data, especially loss, among adaptive optimizers show Adagrad and Adadelta optimization algorithms. \begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline \multicolumn{8}{c}{**Model 1**} & \multicolumn{8}{c}{**Model 2**} & \multicolumn{8}{c}{**Model 3**} \\ \hline \multirow{2}{*}{**Optimizer**} & \multicolumn{2}{c}{**Loss**} & \multicolumn{2}{c}{**Accuracy (\%)**} & \multicolumn{2}{c}{**Loss**} & \multicolumn{2}{c}{**Accuracy (\%)**} & \multicolumn{2}{c}{**Loss**} & \multicolumn{2}{c}{**Accuracy (\%)**} \\  & **Train** & **Test** & **Train** & **Test** & **Train** & **Test** & **Train** & **Test** & **Train** & **Test** & **Train** & **Test** \\ \hline SGD & \(2.25\times 10^{-4}\) & 1.803 & 100.00 & 80.61 & \(7.15\times 10^{-6}\) & 2.688 & 100.00 & 76.49 & \(7.73\times 10^{-5}\) & 2.119 & 100.00 & 76.25 \\ Momentum & \(2.65\times 10^{-6}\) & 1.926 & 100.00 & 83.17 & \(1.36\times 10^{-6}\) & 2.442 & 100.00 & 78.37 & \(8.37\times 10^{-7}\) & 2.171 & 100.00 & 79.23 \\ NAG & \(5.64\times 10^{-7}\) & **1.516** & 100.00 & **83.68** & \(9.79\times 10^{-7}\) & **2.434** & 100.00 & 79.11 & \(5.87\times 10^{-7}\) & **2.007** & 100.00 & **80.40** \\ Adagrad & \(1.38\times 10^{-7}\) & 2.105 & 100.00 & 82.20 & \(3.65\times 10^{-6}\) & 2.586 & 100.00 & 77.49 & \(4.13\times 10^{-5}\) & 2.269 & 100.00 & 77.38 \\ Adadelta & \(4.94\times 10^{-7}\) & 2.218 & 100.00 & 83.65 & \(1.92\times 10^{-7}\) & 2.884 & 100.00 & 79.09 & \(1.32\times 10^{-7}\) & 2.567 & 100.00 & 78.46 \\ RMSProp & \(9.02\times 10^{-2}\) & 9.545 & 99.34 & 82.04 & \(7.59\times 10^{-2}\) & 35.001 & 99.79 & 78.99 & \(4.74\times 10^{-2}\) & 4.064 & 99.39 & 76.37 \\ Adam & \(5.86\times 10^{-5}\) & 1.528 & 100.00 & 82.93 & \(2.00\times 10^{-3}\) & 3.893 & 100.00 & **79.84** & \(2.86\times 10^{-1}\) & 2.517 & 100.00 & 79.07 \\ AdMax & \(6.95\times 10^{-9}\) & 2.341 & 100.00 & 82.83 & \(4.33\times 10^{-4}\) & 3.119 & 99.99 & 78.44 & \(0.00\times 10^{-9}\) & 3.405 & 100.00 & 80.61 \\ Nadam & \(9.63\times 10^{-10}\) & 3.807 & 100.00 & 82.24 & \(5.00\times 10^{-3}\) & 4.925 & 100.00 & 78.65 & \(0.00\times 10^{9}\) & 2.400 & 100.00 & 80.09 \\ \hline \hline \end{tabular}
\end{table}