## 1 Introduction  
Deep learning methods have yielded significant success in several complicated applications, such as image classification [1, 2, 3] and object detection [4, 5, 6, 7]. It is essential to train a deep convolutional neural network with an excellent optimizer to obtain considerable accuracy and convergence speed. Stochastic gradient descent (SGD) [8] is one of the primary and dominant optimizers that performs well across many applications due to its simplicity [9, 10]. However, SGD scales the gradients uniformly in all directions with a constant learning rate. This strategy leads to a slow training process and sensitivity to tuning learning rates. Recent works proposed various adaptive gradient methods to achieve faster convergence by computing individual learning rates for different gradients to address this problem. Examples of these adaptive methods include AdaGrad [11], RMSProp [12], and Adam [13]. These methods use the exponential moving average (EMA) [14] of the past square gradients to adjust the learning rates. Particularly, Adam is one of the famous adaptive gradient methods, and it has become the alternative optimizer after SGD(M) across many deep learning frameworks because of its rapid training speed [15, 16]. Despite their popularity, there is still a performance gap between the adaptive gradient methods with well-tuned SGD (M) [17]. However, recent theories have shown that Adam suffers from non-convergence issues and weak generalization [15, 18]. An empirical study showed that the parameters of Adam update unstably; its second moment could be out of date, resulting in producinginappropriate learning rates based on the changes in gradients [19]. The second moment of Adam in training has been found to produce extreme learning rates; the extreme or abnormal learning rates encountered during training cause poor generalization performance of Adaptive methods [20]. [PERSON] et al. [18] proposed a variant of Adam called AMSGrad to solve the shortcomings of Adam as mentioned above. Although the authors provide a theoretical guarantee of convergence, AMSGrad only shows better results on training data. However, a study further indicated that AMSGrad does not show noticeable improvement over Adam [21]. Another work to fix the problems of Adam is called AdaBound [20]. AdaBound borrows the truncating theory of gradient clipping [22] to constrain the unreasonable learning rates of Adam by two hand-designed asymptotical functions: the upper bound function \(\eta_{u}=final\_tr*(1+\frac{1}{(1-\gamma)\
eq t})\) and the lower bound function \(\eta_{l}=final\_tr*(1-\frac{1}{(1-\gamma)\
eq t+1})\). These two functions tend to a constant value that transforms [PERSON] to SGD (M) gradually. The experiments in [20] demonstrate that AdaBound maintains a rapid training speed, like Adam, in the early and achieves a considerable generalization, like SGD (M), in the end. Additionally, the parameters of bound functions are not sensitive and default parameters are recommended for all classification tasks. In contrast, the experiments reported in [23] showed that AdaBound does not outperform [PERSON] and others, the poor performance of AdaBound could be caused by the default parameters because the same bound are recommended for all classification task. [PERSON] [24] showed that the bound functions strongly affect the optimizer's behavior and might require careful tuning. A reasonable learning rate range refers to the reasonable learning rates that a current task can converge, the maximum value of this range is considered the largest learning rate that does not cause the disconvergence of training criteria [25]. However, AdaBound is empirically hand-designed for its bounds, which are independent of tasks. So, it remains a problem to AdaBound that its bounds are able to accurately and completely truncate the unreasonable learning rates or not during training, because the unreasonable learning rates of a certain task are not stated explicitly.