## 1 Introduction  
Neural network optimization algorithms continue to be a well-studied subject among researchers [1, 2, 3, 4, 5, 6, 7]. Training a deep neural network could largely be termed an optimization problem and is usually trained using stochastic gradient descent-based optimization algorithms. The main goals of researchers are to minimize the error function and accelerate convergence to an optimal global solution, with the overall objective of improving the model's performance and its generalization ability. In this regard, optimizers are an important hyperparameter that affect the training performance of deep neural networks. Hence, it is important to choose the right optimizer for any given dataset problem that is being investigated, since the overall objective of neural network training is to minimize the prediction error by locating the global optimum on the loss surface. However, there is no consensus on this, so researchers are left with the task of experimenting with different optimization algorithms. Several well-known optimization algorithms are used in training neural networks and can be broadly categorized into two: the classic stochastic gradient descent (SGD), which uses a static learning rate, and the adaptive stochastic learning rate such as ADAM and ADAGrad, which use adaptive learning rates. The learning rate is a key and central hyperparameter used by the optimizers for training neural networks and controls how fast a given model adapts to the problem it seeks to solve by helping the model to respond to the estimated error or loss each time the weights and biases values are updated. For instance, too large a learning rate could result in too fast a weights update, which may lead to instability in the training process and cause the model to converge prematurely to a suboptimal solution, whereas too small a learning rate may result in very tiny updates to the weights and a longer training process that could cause the model to be stuck in the local optima. Researchers developed the adaptive learning rate optimizers as an improvement to the classic SGD because of the latter's perceived slowness in arriving at an optimal solution and manually tuning the learning rate. However, one of the challenges still faced by these adaptive learning rate optimization algorithms is being trapped in local minima as against the desirable global minima [8], sometimes even exhibiting inferior performance in comparison to the classic SGD in some machine learning settings and problems [8], which has resulted in using a warm-up heuristic to mitigate these effects [9] and other ongoing improvements published in the literature. Among the adaptive learning rate optimizers, ADAM is the most popular among researchers due to its suitability in most problem cases. Because there is no consensus on the right optimizer to choose for a given task, as shown in a comparative study conducted in [10], researchers continue to work to address the issues with ADAM and other well-known optimization algorithms bordering around learning rate, stabilizing training, faster convergence, and improved generalization [6; 11]. There have been two recent developments in optimization research for deep neural networks. First is a study that sought to develop an improved variant of ADAM called rADAM (rectified ADAM) [7]. The authors sought to address the underlying causes of sub-optimal convergence to local optima by rectifying the undesirable large variance of adaptive learning rate usually observed during the early stage of training in various neural network architectures and models. The root cause, they argue, is because of the limited training data at the initial training stage compared to the later stage. They advocated the use of a lower learning rate during the first few training epochs as a solution to alleviate the convergence problem.