# Improved Gradient Descent Iterations for Solving Systems of Nonlinear Equations
## 1 Introduction, Preliminaries, and Motivation
### Motivation
#### 1.2.1 Improved Gradient Descent Methods as Motivation
The idea in [58] is usage of a proper diagonal approximation of the Hessian  
\[B_{k}=\gamma_{k}I,\quad\gamma_{k}>0,\gamma_{k}\in\mathbb{R}. \tag{13}\]  
Applying the approximation (13) of \(B_{k}\), the matrix \(H_{k}\) can be approximated by the simple scalar matrix  
\[H_{k}=\gamma_{k}^{-1}I. \tag{14}\]  
In this way, the quasi-Newton line search scheme (11) is transformed into a kind of \(AGD\) iteration, called the \(SM\) method and presented in [58] as  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}-\gamma_{k}^{-1}t_{k}\mathbf{g}_{k}. \tag{15}\]  
The positive quantity \(\gamma_{k}\) is the convergence acceleration parameter which improves the behavior of the generated iterative loop. In [56], methods of the form (15) are termed as improved gradient descent methods (IGD). Commonly, the primary step size \(t_{k}\) is calculated through the features of some inexact line search algorithms. An additional acceleration parameter \(\gamma_{k}\) is usually determined by the Taylor expansion of the goal function. This way of generating acceleration parameter is confirmed as a good choice in [56; 58; 60; 61; 62]. The choice \(\gamma_{k}:=1\) in the \(IGD\) iterations (15) reveal the \(GD\) iterations  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}-t_{k}\mathbf{g}_{k}. \tag{16}\]  
On the other hand, if the acceleration \(\gamma_{k}\) is well-defined, then the step size \(t_{k}:=1\) in the \(IGD\) iterations (15) is acceptable in most cases [63], which leads to a kind of the \(GD\) iterative principle:  
\[\mathbf{g}_{k}^{\mathrm{T}}\mathbf{d}_{k}<0.\]0  
[PERSON] and [PERSON] in [64] proposed two efficient \(IGD\) variants, known as \(BB\) method variants, where the steplength \(\gamma_{k}^{BB}\) was defined as an approximation \(H_{k}=\gamma_{k}^{BB}I\). Therefore, the replacement \(\gamma_{k}^{-1}:=\gamma_{k}^{BB}\) in (17) leads to the \(BB\) iterative rule  
\[\mathbf{g}_{k}^{\mathrm{T}}\mathbf{d}_{k}<0.\]1  
The scaling parameter \(\gamma_{k}^{BB}\) in the basic version is defined upon the minimization of the vector norm \(\min\limits_{\gamma}\|\mathbf{s}_{k-1}-\gamma\mathbf{y}_{k-1}\|^{2}\), which gives  
\[\mathbf{g}_{k}^{\mathrm{T}}\mathbf{d}_{k}<0.\]2  
The steplength \(\gamma_{k}^{BB}\) in the dual method is produced by the minimization \(\min\limits_{\gamma}\|\gamma\mathbf{s}_{k-1}-\mathbf{y}_{k-1}\|^{2}\), which yields  
\[\mathbf{g}_{k}^{\mathrm{T}}\mathbf{d}_{k}<0.\]3The \(BB\) iterations were modified and investigated in a number of publications [65; 66; 67; 68; 69; 70; 71; 72; 73; 74; 75; 76; 77; 78; 79]. The so-called _Scalar Correction_ (SC) method from [80] proposed the trial steplength in (17) defined by  
\[\gamma_{k+1}^{\rm SC}=\left\{ \begin{array}{ll}\frac{\mathbf{s}_{k}^{\rm T} \mathbf{r}_{k}}{\mathbf{y}_{k}^{\rm T}\mathbf{r}_{k}},&\mathbf{y}_{k}^{\rm T }\mathbf{r}_{k}>0\\ \frac{\|\mathbf{s}_{k}\|}{\|\mathbf{y}_{k}\|},&\mathbf{y}_{k}^{\rm T}\mathbf{ r}_{k}\leq 0\end{array} \right.,\;\;\mathbf{r}_{k}=\mathbf{s}_{k}-\gamma_{k} \mathbf{y}_{k}. \tag{20}\]  
The SC iterations are defined as  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}-\gamma_{k}^{\rm SC}\mathbf{g}_{k}.\]  
A kind of steepest descent and \(BB\) iterations relaxed by a parameter \(\theta_{k}\in(0,2)\) were proposed in [81]. The so-called _Relaxed Gradient Descent Quasi Newton_ methods, (shortly \(RGDQN\) and \(RGDQN1\)), expressed by  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}-\theta_{k}t_{k}\gamma_{k}^{-1}\mathbf{g}_{k}, \tag{21}\]  
are introduced in [82]. Here, \(\theta_{k}\) presents the relaxation parameter. This value is chosen randomly within the \((0,1)\) interval in the \(RGDQN\) schemes and by the relation  
\[\theta_{k}=\frac{\gamma_{k}}{t_{k}\gamma_{k+1}}\]  
in the \(RGDQN1\) algorithm.