## 1 Introduction
Secondly, another recent study saw the development of the _Lookahead_ optimization algorithm published by the authors in [6], which iteratively updates two weights, with the faster weights exploring or 'looking ahead', while the slower weights maintain the overall training stability of the neural network. The study was inspired by advances made in neural network loss surface research that allows a robust way to accelerate convergence and training stability All this is achieved with less hyperparameter tuning and minimal computational cost. The backpropagation algorithm is key to fast learning in neural networks [12]. Backpropagation is a term that is used to compute the partial derivatives \(\partial\mathcal{C}/\partial w\) and \(\partial\mathcal{C}/\partial b\) of the cost function \(C\), concerning any weight \(w\) and bias \(b\) in the network [13]. It gives insights into the overall behavior of the neural network in terms of how changing the weights and biases minimizes the cost function, which is determined by the gradients of the cost function. One of the factors that determine the stability and speed of network converge is the optimization algorithm; hence, it is worth examining how each optimizer's update rule could make convergence faster, while at the same time maintaining high accuracy and low loss based on any given dataset-specific problem and the neural network architecture. This paper extends the work in [14] in which the authors performed a comparative evaluation of the seven most commonly used first-order stochastic gradient-based optimization algorithms on a simple convolutional neural network architecture. In this current work, we go further to probe the impact of optimization algorithms on increasing CNN network sizes to understand their performance behavior. Accordingly, we pose the following questions: (1) How do different optimizers impact the learning performance of CNN architectures with variations in width, depth and width/depth on image classification problems? (2) Are there significant differences in the performance outputs? Thus, we formulate a hypothesis for this study that given a non-convex problem, different optimization algorithms can find completely different solutions when initialized from the same point. We achieve this through an empirical study of different CNN architectures with varying network sizes to find out if we observe any discernable learning performance. This leads us to the main contributions of the present paper, summarized as follows:  
1. We implemented three different wider, deeper, and wider/deeper simple CNN architectures. 2. We conducted extensive experiments and analyzed the effects of nine different optimizers on increasing deepness, wideness, and deepness/wideness of the CNN architectures on three benchmark image classification datasets (Cats and Dogs, Natural Images, and Fashion MNIST). 3. We provided insights into the effects of optimizers on CNN depth, width and depth/width architectures to inform optimal problem-specific CNN model design. The remainder of the paper is organized in the following way: Section 2 presents a review of related works; Section 3 describes the different optimizers investigated in this study; and Section 4 presents the methodology of the study, including the experimental setup, summary of the dataset characteristics, and an overview of the CNN architectures studied. The experimental results and discussions are presented in Section 5. Section 6 concludes the paper. For clarity regarding the reminder of this paper, all acronyms adopted and used are provided in Table A1 of Appendix D.