# Adam Algorithm with Step Adaptation
## 4 Numerical Results and Discussion
### Application Problem (Regression)
\begin{table}
\begin{tabular}{c c c c c} \hline \hline _Nb_ & _ADAM_ & _H_GR_ & _H_MOMENTUM_ & _H_ADAM_ \\ \hline N & 8088 & 75,539 & 29,697 & 407 \\
1 & 1300 \(\times\) 10\({}^{2}\) & 3887 \(\times\) 10\({}^{2}\) & 2894 \(\times\) 10\({}^{2}\) & 60,993 \\
10 & 4913 \(\times\) 10\({}^{-1}\) & 1509 \(\times\) 10\({}^{2}\) & 6066 \(\times\) 10\({}^{-9}\) & 5961 \\
100 & 11,134 & 8891 & 19,917 & 1332 \\
200 & 10,219 & 8735 & 18,731 & 1087 \\
300 & 9611 & 11,955 & 18,566 & 1028 \\ \hline \hline \end{tabular}
\end{table} 
Table 14: Linear regression without interference for the function \(f_{r}(x,\) [\(a_{max}=10\)]) minimization with gradient information \(f_{rh}(x,\) _Nc, Nb, [\(a_{max}=10\)]), \(n=100\), \(N=1000\), \(n\_{max}=250\),000. Here, we can see that the efficiency of the algorithms increase in the following order: ADAM, H_MOMENTUM and H_GR, H_ADAM. Preference is given to the H_ADAM algorithm only because it solved all the problems. At a low degree of degeneracy (\(a_{max}=10\)), the gradient method H_GR turns out to be more effective compared to the ADAM algorithm. If the H_GR method converges, its results are better compared to other methods. The ADAM method is not operational with small portions of data, even at a low degree of degeneracy. The H_ADAM method works stably. Here, we can see that the efficiency of the algorithms increase in the following order: H_MOMENTUM, H_GR, ADAM and H_ADAM. At a higher degree of degeneracy (\(a_{max}=100\)), the H_ADAM method has better results compared to other methods and retains the ability to train the model with small portions of data for calculating the gradient (\(Nb=1\), \(10\)). The ADAM method is preferable to the H_GR and H_MOMENTUM methods. In general, the imposition of interference did not significantly affect the convergence and convergence rate of the methods. At the same time, the noise did not affect the efficiency of the H_ADAM method. Methods for solving optimization problems with interference under drift conditions are of great importance in robotics and artificial intelligence problems. For these purposes, methods are needed that implement the process of searching and tracking the position of the extremum. In this regard, suitable methods are algorithms with step adaptation, in which the step can take sizes corresponding to the drift speed of the extremum and there is no need to use one-dimensional search. Obviously, methods that allow accelerating gradient descent can also be implemented with step adaptation and are applicable to solving such problems. Consider a function in which the position of the extremum changes along a given trajectory \(x^{*}(m)\):  
\[f_{QX}(x,[a\text{max},\ x*(m)])=\frac{1}{2}\sum_{i=1}^{n}a_{i}\ (x_{i}-x_{i}^{*}(m))^{2},\ \ a_{i}=a\text{max}^{n-1}. \tag{27}\]  
Define the trajectory of the extremum point \(x^{*}(m)\):  
\[x_{1}^{*}(m)=R*\sin(\frac{m}{2\pi R}),\ \ \ x_{n}^{*}(m)=R*\cos(\frac{m}{2\pi R }),\ x_{i}^{*}(m)=0,\ \ \ i=2,3,\ldots,n-1. \tag{28}\]  
When testing methods, \(m\) is equal to the number of calls to the gradient of the function. Relation (28) shows the optimum point moving along a circle in the plane of the first and last components of the vector \(x\) while the other components of the optimum vector are zero. The starting point is \(x_{0}=(10,10,\ldots,10)\). At \(a_{max}=10\), the dynamics of changes in the values of the function (27) during minimization by the studied methods in a logarithmic scale are presented in Figure 6. It is obvious that the ADAM method with a small step is not suitable for solving the problem of chasing a minimum point. Step adaptation methods solve this problem. At \(a_{max}=100\), the dynamics of changes in the values of the function (27) during minimization by the studied methods are presented in Figure 7. Here, the H_ADAM and H_MOMENTUM methods outperform the gradient-based H_GR method. Thus, increasing the degree of the problem conditionality leads to a decrease in the accuracy of tracking the minimum.