# Adam Algorithm with Step Adaptation
## 2 Problem Statement and Background  
Let the problem of minimizing the function _f(x)_, \(x\in R^{n}\) be solved. In relaxation minimization methods, successive approximations are constructed using the formulas:  
\[x_{k+1}=x_{k}-h_{k}s_{k},\ \ \ s_{k}\in R^{n}. \tag{1}\]  
Here, \(s_{k}\) is the descent direction, \(h_{k}\) is the step of one-dimensional (line) search, while \(f(x_{k+1})\leq f(x_{k})\). These and other frequently used designations are given in Table 1.  
In exact one-dimensional descent  
\[h_{k}=\underset{h\in R}{\mathrm{argmin}}f(x_{k}-hs_{k}). \tag{2}\] \begin{table}
\begin{tabular}{l c c c} \hline
**Authors** & **Reference** & **Publication Years** & **Method** \\ \hline [PERSON], [PERSON] & [13] & 2015 & Adam \\ [PERSON], [PERSON], [PERSON], & [20] & 2011 & AdaGrad \\ [PERSON] & [21] & 2012 & AdaDelta \\ [PERSON], [PERSON], [PERSON] & [16] & 2018 & AMSGrad \\ [PERSON] et al. & [26] & 2020 & Padam \\ [PERSON], [PERSON], [PERSON] & [27] & 2019 & AdaGrad-Norm \\ [PERSON] et al. & [30] & 2024 & AdaGC \\ [PERSON] et al. & [31] & 2020 & AdaBelief \\ [PERSON], [PERSON], [PERSON] & [32] & 2023 & AdaDerivative \\ [PERSON] et al. & [33] & 2020 & RAdam \\ [PERSON] et al. & [34] & 2024 & CN-Adam \\ \hline \end{tabular}
\end{table} 
Table 1: Some well-known adaptive methods. Denote the current gradient value by \(g_{k}=\
abla f(x_{k})\). In gradient noise conditions, smoothed gradient values are used:  
\[p_{k}=\beta_{1}p_{k-1}+(1-\beta_{1})g_{k},\ p_{k}\in R^{n},\ \beta_{1}\in[0,\ 1). \tag{3}\]  
Previously, in [35; 36], we proposed several algorithms for adapting the step of the gradient method without one-dimensional search. In this study, we extend the ideas of one of these algorithms to a number of gradient methods used for training neural networks. We consider algorithms in which space transformation is carried out using diagonal metric matrices, the diagonal elements of which we will store as a vector. Denote the Hadamard product for vectors \(a\) and \(b\) by \(a\cdot b=a\,\Theta\,b\). In the algorithms presented below, instead of matrices, scaling factors specified as a vector are used, for example, \(b_{k}\in R^{n}\). The descent direction in this case is calculated according to the following rule:  
\[s_{k}=p_{k}\,\Theta\,b_{k}. \tag{4}\]  
In what follows we will assume that we have the ability to calculate the gradient with noise and that the function value is unavailable. Thus, we do not have the ability to use one-dimensional search (2). In gradient algorithms in this case, various types of step multiplier settings are used. Consider the Adam algorithm using the structure of the method with a metric space transformation, where a gradient and a transformed metric matrix are used to form the descent direction. In the Adam algorithm, exponentially smoothed gradients \(p_{k}=\beta_{1}p_{k-1}+(1-\beta_{1})g_{k}\) are used as directions, the scales of which are changed by variables using exponentially smoothed squares of gradients \(b_{k}=\beta_{2}b_{k-1}+(1-\beta_{2})g_{k}^{2}\). The metric matrix is diagonal and its main diagonal elements are formed based on exponentially smoothed squares of the gradient components. The next element is the method for forming a step along the descent direction. In the algorithms presented below, the operations of multiplication, the division of vectors and raising them to a power will mean that there will be operations for the components of the vectors involved in the operation. The addition of a vector and a number will mean adding a number to each component of the vector. The initial values of the parameters in Algorithm 1 are chosen as in [13]. Let us consider the Adam algorithm. ```
1. Set \(\beta_{1}\), \(\beta_{2}\in[0,1)\), \(\epsilon>0\), step size \(\alpha>0\), the initial point \(\mathrm{x}_{0}\in R^{n}\). The algorithm uses parameters \(\beta_{1}=0.9\), \(\beta_{2}=0.999\), \(\epsilon=10^{-8}\), \(\alpha=0.001\). 2. Set \(k=0\), \(p_{k}=0\), \(b_{k}=0\), where \(p_{k}\), \(b_{k}\in R^{n}\). 3. Set \(k=k+1\), \(g_{k}=\
abla f(x_{k})\). 4.