## 5 Results and Discussions
### Effect of Optimizers with Varying Network Sizes on Fashion MNIST  
Table 10 reports the results obtained for Fashion MNIST. NADAM was the overall best performer in terms of validation accuracy (83.1%) and loss (0.585) on the depth/width CNN model, followed by ADAM (80.4%) in second position, once again at the expense of higher convergence time. However, ADAM recorded better convergence time when compared to its closest rival, NADAM, on the depth CNN model, with lower accuracy and higher loss. ADADelta was the overall worst performer across all the CNN models for this dataset. Figure 3 shows plots of accuracy and loss against epoch on the best-performing configuration with NADAM and increasing CNN depth/width. In Figure A3 in Appendix A.3, Figure A6 in Appendix B.3, and Figure A9 in Appendix C.3, there are more details of all the experimental plots for CNN with varying network sizes for Fashion MNIST. We noticed no significant improvement after about 500 epochs. In this study, the following inferences are drawn based on the results obtained:  
* Overall, all the optimizers showed improved performance, particularly ADAM with an accuracy of 91.1% with the deeper/wider CNN architecture for the two-class Cats and Dogs dataset problem. It is, however, worth noting that ADAMax with an accuracy of 88.9% performed marginally better than NADAM with an accuracy of 86.5% with the deeper/wider CNN architecture on Cats and Dogs. This indicates that the incorporated Nesterov momentum in NADAM had little effect on this dataset problem. * For Natural Images with eight classes, the results show that the wider network architecture generally had better validation accuracy across all the optimizers, especially ADAM with an accuracy of 98.1%, when compared to the simple, deeper, and deeper/wider networks. However, the improved validation accuracy was at the expense of higher convergence time, especially for the adaptive learning rate optimizers. The only exception was ADADelta, which generally performed poorly on this dataset. * It is worth noting that SGD momentum and SGD Nesterov momentum with the wider CNN architecture performed well on Natural Images with accuracies of 92.9% and 93.7%, respectively. This is consistent with previous studies suggesting that vanilla SGD optimizer can be improved with momentum and Nesterov momentum, and that the improved SGD can be competitive with the adaptive learning rate optimizers for some specific tasks, but with additional time to converge. Figure 1: Plot of best-performing ADAM with increasing CNN depth and width on Cats and Dogs. Figure 2: Plot of best-performing ADAM with increasing CNN width for Natural Images. * For Fashion MNIST, NADAM was the better performer with an accuracy of 83.1% on depth/width CNN. However, we noted that for the Fashion MNIST there was generally a reduced performance across all the optimizers combined with the CNN architectures. This could be related to the higher number of classes and the simple network design considered in this study. * The initial results obtained suggest that for the two-class problems, ADAM is a better optimizer to consider with a deeper/wider CNN architecture. For the larger number of classes, ADAM and NADAM with wider networks or deeper/wider network architectures irrespective of dataset size could be considered. However, because of the empirical evidence in the literature that the performance of deeper network tends to worsen after a certain critical point, in addition to increased computational cost due to increased parameterization, a careful balance between depth and width should be applied during the design of deeper/wider CNN architectures. For the majority of tasks, we suggest increasing the width while keeping the depth constant, especially when depth has reached a critical performance point. * The study demonstrated empirically that significant differences do exist in the performance outputs of different optimizers on varying depth, width and depth/width CNN models, depending on the dataset problem.