# Adam Algorithm with Step Adaptation
## 4 Numerical Results and Discussion
### Application Problem (Regression)  
We use the problem of estimating the parameters of a linear model as an application problem. When estimating the parameters of a neural network, there may be a significant number of factors that cause the loss of convergence of the learning method. For example, one of them is that the parameters of a neuron fall into the zone where the function ceases to be sensitive to changes in the parameters of this neuron. Special methods are needed to eliminate such shortcomings. The main goal of the optimization (learning) method is to effectively overcome the issues of ill-conditioning in the minimization problem. In this regard, the problem of estimating the linear model parameters enables us to simulate the conditionality of the problem and evaluate the behavior of the learning algorithm depending on the degree of its conditionality. At the same time, the presence of a single minimum point enables us to evaluate the speed qualities of the method. Denote, using \(x^{*}\in R^{n}\), a vector of model parameters that we have to find based on observations of the values of a linear model with these parameters at a number of given points. Set \({x_{i}}^{*}=1\), \(i=1\), \(\ldots\), \(n-1\), \({x_{n}}^{*}=0\), matrix \(m\in R^{N\times n}\), matrix rows \(m_{i}\in R^{n}\) and model values at observation points \(y\in R^{N}\)  
\[m_{i,j}=q\in[-a_{i},a_{i}],\;j=1,2,\ldots,n-1,\;m_{i,n}=1,\;y_{i}=(m_{i},x^{*} ),\;i=1,2,\ldots,N, \tag{23}\]  
where \(q\in[-a_{i},a_{i}]\) is a random uniformly distributed number, \(a_{i}=a\max^{\frac{i-1}{n-1}}\). Scaling the data using the coefficients in (23) determines the degree of conditionality of the problem. The task is to find unknown parameters \(x^{*}\in R^{n}\) given \(m\), \(y\). The solution can be obtained by minimizing the function  
\[f_{r}(x,[a\text{max}])=\frac{1}{2}\!\!\sum_{i=1}^{N}\;\Big{(}(m_{k(i)},x)-y_{k (i)}\Big{)}^{2}. \tag{24}\]  
We will search for a solution using stochastic gradient methods, based on the gradients of a function formed on a randomly selected segment of data. \[f_{rb}(x,Nc,Nb,[a\text{max}])=\frac{1}{2}\!\!\sum_{i=Nc+1}^{Nc+Nb}\;\Big{(}(m_ {k(i)},x)-y_{k(i)}\Big{)}^{2}, \tag{25}\]  
where \(Nc\) is a random number on a segment \(Nc\in[0,\,N-1]\) and \(k(i)=i\bmod N\) is the remainder of division. The first stopping criterion is \(f(x^{k})-f^{*}\leq\epsilon=10^{-10}\). For these purposes, we will compute the function (24) at each step of the algorithm. The second stopping criterion is the maximum number of method iterations \(n\_max\). If the method does not cope with the task within the allotted number of iterations, then we indicate the achieved value of the objective function in the table cell. Tables 13-16 show the results of minimization. \begin{table}
\begin{tabular}{c c c c c} \hline \hline _Nb_ & _ADAM_ & _H_GR_ & _H_MOMENTUM_ & _H_ADAM_ \\ \hline N & 16,672 & 951 & 465 & 429 \\
1 & 4240 & 2465 \(\times\) 10\({}^{-9}\) & 2766 \(\times\) 10\({}^{-8}\) & 74,608 \\
10 & 33,813 & 6911 \(\times\) 10\({}^{-8}\) & 9100 & 5173 \\
100 & 21,740 & 1879 & 2266 & 990 \\
200 & 19,755 & 1882 & 1951 & 723 \\
300 & 18,700 & 1877 & 1878 & 669 \\ \hline \hline \end{tabular}
\end{table} 
Table 16: Linear regression without interference for the function \(f_{r}(x,\) [\(a_{max}=10\)] minimization with gradient information \(f_{rh}(x,\) _Nc, Nb, [\(a_{max}=10\)]), \(n=100\), \(N=1000\), \(c=10\), \(n\_{max}=100\),000. \begin{table}
\begin{tabular}{c c c c} \hline \hline _Nb_ & _ADAM_ & _H_GR_ & _H_MOMENTUM_ & _H_ADAM_ \\ \hline N & 5257 & 1086 & 18,595 & 347 \\
1 & 3638 & 6559 \(\times\) 10\({}^{-4}\) & 90,588 & 15,283 \\
10 & 11,708 & 4266 \(\times\) 10\({}^{-9}\) & 8321 & 2564 \\
100 & 7691 & 1646 & 1984 & 580 \\
200 & 6873 & 1652 & 1704 & 464 \\
300 & 6462 & 1650 & 1666 & 432 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Linear regression without interference for the function \(f_{r}(x,\) [\(a_{max}=10\)] minimization with gradient information \(f_{rh}(x,\) _Nc, Nb, [\(a_{max}=10\)]), \(n=100\), \(N=1000\), \(n\_{max}=250\),000.