# Adam Algorithm with Step Adaptation
## 4 Numerical Results and Discussion
### Preliminary Analysis of Algorithm Parameters
#### 4.1.1 Convergence Rate Analysis  
We conduct an experiment on the following function:  
\[f_{\mathbb{Q}}(x,[\text{amax}])=\frac{1}{2}\sum_{i=1}^{n}a_{i}\ x_{i}^{2},\ \ a_{i}=a\text{max}\frac{i-1}{n-1}.\]  
Eigenvalue \(a_{i}\) of this function has the boundaries \(\lambda_{min}=1\) and \(\lambda_{max}=a_{max}\). Increasing the \(a_{max}\) value leads to an increase of problem conditioning. The initial points are \(x_{0}=(100\), \(100\), \(\ldots\), \(100\)), \(n=100\), \(n\_max=100\),\(000\). The parameters are \(\beta_{1}=0.9\), \(\beta_{2}=0.999\) and \(\alpha=0.01\). Table 2 shows the results of minimization with different values of stopping criterion. The H_GR algorithm most accurately reflects the dependence of the method on the degree of the problem conditioning. With a certain selection of the gradient smoothing parameter \(\beta_{1}\), the other methods will also show an increasing number of iterations as the conditionality worsens. Such a selection should be carried out for the class of applied problems. For the function/\(f_{Q}(x,[a_{max}=10^{4}])\) from Table 2, the trajectory of the function decrease for all the studied methods on a logarithmic scale is shown in Figure 3.  
All step-adaptive methods converge linearly on each section of the trajectory. Since we did not obtain complete information about the convergence of the ADAM algorithm with the number of iterations of 100,000, we will conduct an experiment with a doubled number of iterations (Figure 4). \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{\(a_{\text{max}}\)} & \multicolumn{4}{c}{\(f(x_{k})\)–\(f^{\prime}\)\(\leq\)\(10^{-4}\)} & \multicolumn{4}{c}{\(f(x_{k})\)–\(f^{\prime}\)\(\leq\)\(10^{-10}\)} \\ \cline{2-10}  & **ADAM** & **H_GR** & **H_MOMENTUM** & **H_ADAM** & **ADAM** & **H_GR** & **H_MOMENTUM** & **H_ADAM** \\ \hline
10 & 5307 & 216 & 489 & 607 & 5307 & 944 & 79,147 & 12,679 \\ \(10^{2}\) & 5307 & 720 & 418 & 5143 & 5307 & 1407 & 60,540 & 11,498 \\ \(10^{3}\) & 5307 & 4451 & 854 & 4167 & 5307 & 7892 & 1243 & 4917 \\ \(10^{4}\) & 5307 & 44,442 & 2205 & 891 & 5307 & 78,882 & 3892 & 1572 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Number of iterations for the \(f_{Q}(x,[a_{max}])\) function minimization depending on the \(a_{max}\). Figure 4: Trajectory of the function \(f_{Q}(x,[a_{max}\)=10^{4}]) decrease. The number of iterations (X axis) is given in thousands. Figure 3: Trajectories of the function \(f_{Q}(x,[a_{\text{max}}\)=10^{4}]) decrease. The number of iterations (X axis) is given in thousands. For the ADAM method, the trajectory has a very small segment of high convergence rate. From Figures 3 and 4, we can draw the following conclusions. For step-adaptive methods, the convergence rate is constant along the descent trajectory. The ADAM method has three possible situations. If the step is small then the convergence rate is low, which we observe in the first data segment. If the step is optimal, then the speed is high. If the step is large, then oscillations occur in the minimum neighborhood. Based on the findings, it is necessary to choose the optimal step size for the ADAM method while a narrow range of its high efficiency exists. The graphs can be used to track the required level of criterion optimization, for example, \(10^{-4}\). Our goal is to identify defects in the convergence rate of methods. Comparing the results for the H_ADAM and ADAM methods in Table 2, we can note a sharp deterioration in their convergence rate with increasing optimization accuracy. It is especially important to set overestimated optimization criteria when minimizing functions with noise. In [35], a quadratic increase in the number of iterations is shown depending on the ratio of the noise value to the gradient norm. For example, with a tenfold noise increase, the cost of the number of iterations increases by 100 times. In order to verify the convergence of the method under noise conditions, we also need to set a deep optimization criterion and perform a large number of iterations.