# Adam Algorithm with Step Adaptation
## 3 Algorithms with Step Adaptation  
Consider gradient methods for minimizing a quadratic function  
\[f_{Q}(x)=\frac{1}{2}(x,Ax)-(b,x)+c,\quad b,x\in R^{n},\ \ A\in R^{n\times n}. \tag{6}\]  
In the gradient method with a constant optimal step, the step size is determined by the expression  
\[h^{*}=2/(M+m), \tag{7}\]  
where \(M\) and \(m\) are the maximal and minimal eigenvalues of the matrix of second derivatives \(A\). The estimate of the method (1) convergence rate for different ways of selecting the steps (2) and (7) is the same. To estimate step (2), the function derivative along the descent direction is used. The step is determined from the condition of the extremum of the one-dimensional function. The derivative at the optimal point must be zero. Denoted by \(y(h)=-(\
abla f(x_{k}-hs_{k})\), \(s_{k})\), the derivative of function \(\phi(h)=\)\(f(x_{k}-hs_{k})\), with respect to the parameter \(h\), can be computed in a minimization algorithm based on gradient information. If the function is quadratic, then \(y(h)\) is a linear function of \(h\). In [35], the following estimate of the optimal step was obtained  
\[h_{k}^{*}=h_{k}\frac{y(0)}{y(0)-y(h_{k})}. \tag{8}\]  
Estimate (8) is achieved only after obtaining a new point and calculating the gradient of it. To use estimate (8), the following reasoning is required. The gradient method with a constant step converges in a wide range of step size selection, but this range and the optimal step are fixed for a quadratic function, or for strongly convex functions with a Lipschitz gradient [37]. Therefore, in [35], estimate (8) is adopted as an estimate for step (7), towards which the current search step is moved. The simplest way to organize an algorithm of this kind is machine learning. We form a quality functional  
\[\Phi(h)=(h-h^{*})/2. \tag{9}\]  
The gradient minimization method (9) at a certain step is  
\[h_{k+1}=h_{k}-q_{k}\Phi^{\prime}(h_{k})=h_{k}-q_{k}(h_{k}-h_{k}^{*}). \tag{10}\]  
In [35], several methods of organizing update (10) were used. The most suitable for solving minimization problems under interference conditions was the following type of method:  
\[h_{k+1}=h_{k}-q_{k}sign(h_{k}-h_{k}^{*}). \tag{11}\]In (11), possible oscillations in values \(h_{k}-h_{k}\)* are excluded. Process (11) can be organized in another way. For this, we only need information about whether the step \(h_{k}\) value is greater or less than step \(h_{k}\)*. Initially, the derivative is negative. In the case when the boundary of the zero value of the derivative is not reached and \(y(h_{k})<0\), the step should be increased. When the boundary is passed, then \(y(h_{k})>0\) and the step should be decreased. In this case, (11) is rewritten as  
\[h_{k+1}=h_{k}-q_{k}sign(y(h_{k})). \tag{12}\]  
In [35], the following method of step adaptation in Algorithm 1 is used with \(q=(1+\alpha)>1\):  
\[ \begin{split} h_{k+1}=h_{k}(1+\alpha)^{-sign(y(h_{k}))}=h_{k}(1+ \alpha)^{sign(\
abla f(x_{k+1}),s_{k})}\\ =\begin{cases}h_{k}(1+\alpha)if(\
abla f(x_{k+1}),s_{k})>0,\\ h_{k}if(\
abla f(x_{k+1}),s_{k})=0,\\ h_{k}/(1+\alpha)if(\
abla f(x_{k+1}),s_{k})<0\end{cases},\end{split}  \tag{13}\]  
where we made the replacement \(y(h_{k})=-(\
abla f(x_{k}-h_{k}),s_{k})=-(\
abla f(x_{k+1}),s_{k})\). In some cases, the following step adjustment may be effective:  
\[h_{k+1}=h_{k}(1-\alpha\cdot sign(y(h_{k})))=h_{k}(1+\alpha\cdot sign(\
abla f( x_{k+1}),s_{k})).\]  
The step adaptation algorithm will be used to minimize functions with gradient interference. Therefore, for a random variable \(sign(y(h_{k}))\) with \(\mathrm{E}(sign(y(h_{k})))=0\), we must have equality \(h_{k+m}=h_{k}\) for the same number of positive and negative values of \(sign(y(h_{k}))\). Update (13) ensures that the required equality \(h_{k+m}=h_{k}\); that is, the absence of a trend component, with the same number of positive and negative values of \(sign(y(h_{k}))\) on a segment of \(m\) updates.