# Adam Algorithm with Step Adaptation  
[PERSON]  
1 Institute of Informatics and Telecommunications, Reshetnev Siberian State University of Science and Technology, 31, Krasnoyarskii Rabochii Prospekt, 660037 Krasnoyarsk, Russia; [EMAIL_ADDRESS] (V.K. ); [EMAIL_ADDRESS] (L.K.) [PERSON]  
1  
[PERSON]  
1  
Footnote 1: email: {k.k. and well suited for both data-intensive and parameter-intensive problems. The method is also suitable for solving very noisy problems. Computational results show that Adam works well in practice [14; 15]. Adam converges quickly in most scenarios, but sometimes it may not converge to an optimal solution, especially in cases with noisy or sparse gradients [16; 17]. Tuning its hyperparameters can be critical to the model's performance and is sometimes more sensitive than simpler optimizers. Additionally, for some problems, implementing annealing of the learning rate with Adam is necessary to achieve convergence, which adds another layer of complexity [18]. As we will see further, at a certain level of interference and degree of conditionality of the problem, the Adam algorithm is subject to looping. When solving applied problems, it is difficult to determine the reasons for the impossibility of obtaining a solution. This may be a defect caused by the algorithm or a defect caused by the formulation of the problem. Therefore, it is important to eliminate the defects generated by the lack of convergence of the algorithm. Adaptive methods that adjust the step size have become widespread in large-scale optimization for their ability to converge robustly [19]. In [20], the authors proposed a method where each dimension has its own dynamic rate: large gradients have smaller learning rates and small gradients have large learning rates. The AdaGrad family of methods [16; 21; 22; 23; 24; 25; 26] is based on this principle and shows good results on large-scale learning problems. In [21], instead of accumulating the sum of squared gradients, the author fixed the window of past gradients that are accumulated. In [27; 28], AdaGrad-Norm with a single step size adaptation based on the gradient norm was developed. Theoretical results for the advantage of AdaGrad-like step sizes over the plain stochastic gradient descent in the non-convex setting can be found in [29]. An adaptive optimization algorithm with gradient bias correction (AdaGC) was proposed in [30]. This algorithm improves the iterative direction through the gradient deviation and momentum, and the step size is adaptively revised using the second-order moment of gradient deviation. In [31], AdaBelief is presented to adjust the step size according to the exponential moving average of the noisy gradient. If the observed gradient greatly deviates from the prediction, this method takes a small step; if the observed gradient is close to the prediction, the method takes a large step. AdaDerivative [32], in contrast to AdaBelief, can adaptively adjust step sizes by applying the exponential moving average of the derivative term using past gradient information without the smoothing parameter. In [16], new variants of the Adam algorithm with a long-term memory of past gradients were proposed. The authors in [33] apply the variance reduction technique to construct the adaptive step size in Adam. The Cycle-Norm-Adam (CN-Adam) algorithm based on the cyclic exponential decay learning rate is proposed in [34]. Some well-known adaptive methods are shown in Table 1.  
The Adam algorithm has three main ideas: gradient smoothing, variable scaling and step selection. Our goal is to analyze the effectiveness of the degree of presence of these components in the minimization algorithm. In our proposed algorithms, the way to select the step multiplier was completely excluded. In this paper, we present a method for tuning the step of the Adam method using the fact that in exact one-dimensional descent, the current descent direction is orthogonal to the gradient calculated at the new point.