## 5 Results and Discussions  
In this section, we present the empirical results of the effect of different optimization algorithms on varying the network size (width, depth and width/depth) on three image classification datasets. The results obtained in the study by [14] are also presented as the benchmark to the three CNN architectures with varying network sizes. On the results presented in Tables 8-10, the best-performing results for each optimizer with the different CNN architectures and dataset problems are the bolded values.  
\begin{table}
\begin{tabular}{c c} \hline
**Cats and Dogs** & **Fashion MNIST** & **Natural Images** \\ \hline \hline \multicolumn{3}{c}{Dropout = 0.5} \\ \hline \multicolumn{3}{c}{Pooling layer: MaxPooling 2 \(\times\) 2; stride = 1} \\ \hline \multicolumn{3}{c}{Flattening (Input Layer of Neural Network)} \\ \hline \multicolumn{3}{c}{FC-128 layers} \\ \hline \multicolumn{3}{c}{ReLU (nonlinearity function)} \\ \hline \multicolumn{3}{c}{Dropout = 0.5} \\ \hline Binary Cross entropy & Categorical Cross entropy & Categorical Cross entropy \\ \hline Sigmoid Layer (\(\blacklozenge\)) & Softmax Layer (\(\sigma\)) & Softmax Layer (\(\sigma\)) \\ \hline \end{tabular}
\end{table}
Table 7: _Cont._  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Methods** & **vSGD** & **SGD\({}_{\mathbf{M}}\)** & **SGD\({}_{\mathbf{NM}}\)** & **ADAGrad** & **RMSProp** & **ADAM** & **ADADelta** & **ADAMax** & **NADAM** \\ \hline \multicolumn{3}{c}{Validation Accuracy} \\ \hline \(s\)CNN & 0.643 & 0.745 & 0.736 & 0.675 & 0.848 & 0.829 & 0.624 & 0.826 & 0.855 \\ \(d\)CNN & 0.599 & 0.702 & 0.706 & 0.695 & 0.876 & 0.884 & 0.581 & 0.859 & 0.839 \\ \(w\)CNN & 0.655 & 0.715 & 0.730 & 0.651 & 0.846 & 0.878 & 0.622 & 0.862 & 0.860 \\ \(d\)\(w\)CNN & 0.655 & 0.755 & 0.834 & 0.738 & 0.877 & **0.911** & 0.639 & 0.889 & 0.865 \\ \hline \multicolumn{3}{c}{Validation Convergence Time} \\ \hline \(s\)CNN & 2.625 & 2.375 & 2.250 & 2.500 & 2.125 & 2.625 & 2.625 & 2.250 & 2.625 \\ \(d\)CNN & 0.833 & 2.495 & 2.500 & 1.250 & 0.833 & 0.833 & 1.667 & 1.250 & **0.625** \\ \(u\)CNN & 1.000 & 2.250 & 1.875 & 1.250 & 1.041 & **0.889** & 1.500 & 1.181 & 0.833 \\ \(d\)\(w\)CNN & 1.250 & 2.000 & 2.000 & 2.111 & 1.319 & **0.889** & 1.75 & 1.111 & 0.889 \\ \hline \multicolumn{3}{c}{Validation Loss} \\ \hline \(s\)CNN & 0.670 & 0.525 & 0.531 & 0.649 & 0.429 & 0.400 & 0.687 & 0.437 & 0.362 \\ \(d\)CNN & 0.671 & 0.583 & 0.510 & 0.621 & 0.328 & 0.285 & 0.687 & 0.360 & 0.365 \\ \(u\)CNN & 0.675 & 0.539 & 0.527 & 0.628 & 0.415 & 0.336 & 0.688 & 0.426 & 0.363 \\ \(d\)\(w\)CNN & 0.672 & 0.523 & 0.323 & 0.658 & **0.179** & **0.276** & 0.686 & 0.326 & 0.398 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Results of optimizers with CNN architectures on Cats and Dogs (epoch = 500).  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Methods** & **vSGD** & **SGD\({}_{\mathbf{M}}\)** & **SGD\({}_{\mathbf{NM}}\)** & **ADAGrad** & **RMSProp** & **ADAM** & **ADADelta** & **ADAMax** & **NADAM** \\ \hline \multicolumn{3}{c}{Validation Accuracy} \\ \hline \(s\)CNN & 0.775 & 0.891 & 0.851 & 0.699 & 0.797 & 0.892 & 0.739 & 0.909 & 0.913 \\ \(d\)CNN & 0.249 & 0.613 & 0.688 & 0.406 & 0.741 & 0.852 & 0.145 & 0.782 & 0.912 \\ \(w\)CNN & 0.798 & 0.929 & 0.937 & 0.735 & 0.840 & **0.981** & 0.562 & 0.944 & 0.976 \\ \(d\)\(w\)CNN & 0.498 & 0.758 & 0.725 & 0.589 & 0.222 & 0.963 & 0.236 & 0.911 & 0.948 \\ \hline \multicolumn{3}{c}{Validation Convergence Time} \\ \hline \(s\)CNN & 5.056 & 4.861 & 5.017 & 5.017 & 5.172 & 5.056 & 5.483 & 4.900 & 5.444 \\ \(d\)CNN & 4.470 & 3.633 & 4.658 & 4.594 & 3.533 & 3.056 & 1.486 & 4.233 & 3.500 \\ \(w\)CNN & 3.667 & 4.167 & 3.900 & 7.117 & 7.006 & 6.667 & 7.739 & 6.681 & 5.417 \\ \(d\)\(w\)CNN & 4.161 & 4.900 & 4.008 & 4.239 & 3.900 & **3.083** & 5.056 & 3.864 & 3.864 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Results of optimizers with CNN architectures on Natural Images (epoch = 150).