# Adam Algorithm with Step Adaptation
## 3 Algorithms with Step Adaptation
In the case of high interference on the gradient, or essentially non-quadratic functions, instead of (1), we will use a process with the normalization of the descent vector  
\[x_{k+1}=x_{k}-h_{k}\;s_{k}/||\;s_{k}||. \tag{14}\]  
In the following Algorithm 2 (Figure 1), we use the gradient descent along the gradient \(g_{k}\) and step adaptation. ```
1. Set the step adaptation parameter \(\alpha>0\), the initial point \(x_{0}\in R^{n}\) and the initial step \(h_{0}\in R^{1}\). The algorithm uses parameter \(\alpha=0.01\). 2. Set \(k=0\), compute \(g_{k}=\
abla f(x_{k})\). 3. Set \(k=k+1\). 4. Compute the descent direction. \(s_{k}=g_{k-1}/\|g_{k-1}\|\). 5. Find a new approximation of minimum \(x_{k}=x_{k-1}-h_{k-1}s_{k}\). 6. Compute \(g_{k}=\
abla f(x_{k})\). 7. Compute a new search step
7.1 If\(\;|\;(g_{k},s_{k})|\leq 10^{-15}\)then\(h_{k}=h_{k-1}\)else
7.2 If\((g_{k},s_{k})\leq 0\)then\(h_{k}=h_{k-1}/(1+\alpha)\)else
7.3\(h_{k}=h_{k-1}(1+\alpha)\). 8. If the stopping criterion is met, then stop, else go to step 3. In Step 7, the step size of the method is adjusted. Condition 7.1 is necessary for cases where, due to noise or when calculating gradients based on mini-batches, the gradient vanishes. In 7.2, the step decrease is produced when the angle between \(s_{k}\) and \(g_{k}\) is obtuse. In 7.3, the step increase is produced when the angle between \(s_{k}\) and \(g_{k}\) is acute. The small value \(10^{-15}\) in Steps 7.1-7.3 was chosen to exclude step adaptation at zero gradients. The algorithm with a normalization of the descent direction is more stable when minimizing functions with interference and enables a wider range of the step adaptation algorithm parameter \(\alpha\) in (13). Taking into account the variety of functions other than quadratic, where fluctuations in the gradient values can be significant, we will further use the algorithm (14) with step adaptation (13). Here, \(\alpha\) is the adjustable parameter. In the following Algorithm 3 (Figure 2), we use descent along the smooth gradient \(p_{k}\) and step adaptation to select the step. Figure 1: Flowchart of Algorithm 2. In Step 8, the step size of the method is adjusted. Using this algorithm, we will verify the efficiency of space metric transformation for the Adam algorithm. In the Adam algorithm with step adaptation (Algorithm 4), we use components \(p_{k}\) and \(b_{k}\) together to form the descent direction. We called this algorithm Adam with step adaptation, although, to be completely precise, it is Adagrad with elements of momentum and step adaptation. \[s_{k}=\frac{p_{k}}{\sqrt{b_{k}}+\varepsilon_{1}}. \tag{15}\]
```
1. Set \(\beta_{1}\), \(\beta_{2}\in[0,1)\), \(\varepsilon>0\), step adaptation parameter \(\alpha>0\), the initial point \(\mathrm{x}_{0}\in R^{n}\), the initial step \(h_{0}\in R^{1}\). The algorithm uses parameters \(\beta_{1}=0.9\), \(\beta_{2}=0.999\), \(\varepsilon=10^{-8}\), \(\alpha=0.01\). 2. Set \(k=0\), \(p_{k}=0\), \(b_{k}=0\), where \(p_{k}\), \(b_{k}\in R^{n}\). Compute \(g_{k}=\
abla f(\mathrm{x}_{k})\). 3. Set \(k=k+1\). 4. Compute \(p_{k}=\beta_{1}p_{k-1}+(1-\beta_{1})g_{k-1}\). 5. Find a descent direction \(s_{k}=p_{k-1}/\|p_{k-1}\|\). 6. Find a new approximation of minimum \(x_{k}=x_{k-1}-h_{k-1}s_{k}\). 7. Compute \(g_{k}=\
abla f(\mathrm{x}_{k})\). 8. Compute a new search step
8.1 \(\textbf{If }|(g_{k},s_{k})|\leq 10^{-15}\textbf{ then }h_{k}=h_{k-1}\textbf{ else}\)
8.2 \(\textbf{If }(g_{k},s_{k})\leq 0\textbf{ then }h_{k}=h_{k-1}/(1+\alpha)\textbf{ else}\)
8.3 \(h_{k}=h_{k-1}(1+\alpha)\). 9. If the stopping criterion is met, then stop, else go to step 3. ```
**Algorithm 3** (H_MOMENTUM)

In Step 8, the step size of the method is adjusted. Using this algorithm, we will verify the efficiency of space metric transformation for the Adam algorithm. In the Adam algorithm with step adaptation (Algorithm 4), we use components \(p_{k}\) and \(b_{k}\) together to form the descent direction.