# Improved Gradient Descent Iterations for Solving Systems of Nonlinear Equations
## 1 Introduction, Preliminaries, and Motivation
### Motivation
#### 1.2.1 Improved Gradient Descent Methods as Motivation  
The most general iteration for solving (7) is expressed as  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+t_{k}\mathbf{d}_{k}. \tag{8}\]  
In (8), \(\mathbf{x}_{k+1}\) presents a new approximation point based on the previous \(\mathbf{x}_{k}\). Positive parameter \(t_{k}\) stays for the steplength value, while \(\mathbf{d}_{k}\) presents the search direction vector, which is generated based on the descent condition  
\[\mathbf{g}_{k}^{\mathrm{T}}\mathbf{d}_{k}<0.\]  
The direction vector \(\mathbf{d}_{k}\) may be defined in various ways. This vital element is often determined using the features of the function gradient. In one of the earliest optimization schemes, the gradient descent method (GD), this variable is defined as negative of the gradient direction, i.e., \(\mathbf{d}_{k}=-\mathbf{g}_{k}\). In the line search variant of the Newton method, the search direction presents the solution to the system of nonlinear equations \(G_{k}\mathbf{d}=-\mathbf{g}_{k}\) with respect to \(\mathbf{d}\), where \(G_{k}:=G(\mathbf{x}_{k})=\bigtriangledown^{2}f(\mathbf{x}_{k})\) denotes the Hessian matrix. Unlike traditional GD algorithms for nonlinear unconstrained minimization, which are defined based on a single step size \(t_{k}\), a class of improved gradient descent (\(IGD\)) algorithms define the final step size using two or more steps size scaling parameters. Such algorithms were classified and investigated in [56]. Obtained numerical results confirm that the usage of appropriate additional scaling parameters decreases the number of iterations. Typically, one of the parameters is defined using the inexact line search, while the second one is defined using the first terms of the Taylor expansion of the goal function. A frequently investigated class of minimization methods that can be applied for solving the problem (7) use the following iterative rule  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}-\theta_{k}t_{k}\mathbf{g}_{k}. \tag{9}\]  
In (9), the parameter \(t_{k}\) represents the step size in the \(k\)th iteration. The originality of the iteration (9) is expressed though the acceleration variable \(\theta_{k}\). This type of optimization scheme with acceleration parameter was originated in [57]. Later, in [58], the authors justifiably named such models as _accelerated gradient descent methods_ (AGD methods shortly). Further research on this topic confirmed that the acceleration parameter generally improves the performance of the gradient method. The Newton method with included line search technique is defined by the following iterative rule  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}-t_{k}G_{k}^{-1}\mathbf{g}_{k}, \tag{10}\]  
wherein \(G_{k}^{-1}\) stands for the inverse of the Hessian matrix \(G_{k}\). Let \(B_{k}\) be a symmetric positive definite matrix such that \(\|B_{k}-G_{k}\|<e\), for arbitrary matrix norm \(\|.\|\) and for a given tolerance \(e\). Further, let \(H_{k}\) be a positive definite approximation of the Hessian's inverse \(G_{k}^{-1}\). This approach leads to the relation (11) which is the quasi-Newton method with line search:  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}-t_{k}H_{k}\,\mathbf{g}_{k}. \tag{11}\]Updates of \(H_{k}\) can be defined as solutions to the quasi-Newton equation  
\[H_{k+1}\mathbf{y}_{k}=\mathbf{s}_{k}, \tag{12}\]  
where \(\mathbf{s}_{k}=\mathbf{x}_{k+1}-\mathbf{x}_{k}\), \(\mathbf{y}_{k}=\mathbf{g}_{k+1}-\mathbf{g}_{k}\). There is a class of iterations (11) in which there is no ultimate requirement for the \(H_{k}\) to satisfy the quasi-Newton equation. Such a class of iterates is known as modified Newton methods [59].