# Improved Gradient Descent Iterations for Solving Systems of Nonlinear Equations
## 2 Multiple Step-Size Methods for Solving SNE
### IGDN Methods for Solving SNE  
Our aim is to simplify the update of the Jacobian \(F^{\prime}(\mathbf{x}_{k}):=J_{k}\). Following (13), it is appropriate to approximate the Jacobian with a diagonal matrix  
\[F^{\prime}(\mathbf{x}_{k})\approx\gamma_{k}I. \tag{25}\]  
Then, \(B_{k}=\gamma_{k}I\) in (5) produces the search direction \(\mathbf{d}_{k}=-\gamma_{k}^{-1}F_{k}\), and the iterations (8) are transformed into  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}-t_{k}\gamma_{k}^{-1}F_{k}. \tag{26}\]  
The final step size in iterations (26) is defined using two step size parameters: \(t_{k}\) and \(\gamma_{k}\). Iterations that fulfill pattern (26) are an analogy of \(IGD\) methods for nonlinear unconstrained optimization and will be termed as \(IGDN\) class of methods. Using the experience of nonlinear optimization, the steplength parameter \(\gamma_{k}\) can be defined appropriately using the Taylor expansion of \(F(\mathbf{x})\):  
\[F_{k+1}=F_{k}+F^{\prime}(\xi_{k})(\mathbf{x}_{k+1}-\mathbf{x}_{k}),\;\xi_{k} \in[\mathbf{x}_{k},\mathbf{x}_{k+1}]\]  
On the basis of (25), it is appropriate to use \(F^{\prime}(\xi_{k})\approx\gamma_{k}I\), which implies  
\[F_{k+1}-F_{k}=\gamma_{k}(\mathbf{x}_{k+1}-\mathbf{x}_{k}). \tag{27}\]  
Using (27) and applying notation (6), one obtains the following updates of \(\gamma_{k}\):  
\[\gamma_{k}=\frac{\mathbf{y}_{k}^{\mathrm{T}}\mathbf{y}_{k}}{\mathbf{y}_{k}^{ \mathrm{T}}\mathbf{s}_{k}}=\frac{\mathbf{s}_{k}^{\mathrm{T}}\mathbf{y}_{k}}{ \mathbf{s}_{k}^{\mathrm{T}}\mathbf{s}_{k}}.\]  
It can be noticed that the iterative rule (26) matches with \(BB\) iteration [64]. So, we introduced the \(BB\) method for solving SNE. Our further contribution is the introduction of appropriate restrictions on the scaling parameter. To that end, Theorem 1 reveals values of \(\gamma_{k}\) which decrease the objective functions included in \(F_{k}\). The inequality \(F_{k+1}\leq F_{k}\) means \((F_{k+1})_{i}\leq(F_{k})_{i}\), \(i=1,\ldots,n\). Theorem 2.1: _If the condition \(\gamma_{k+1}\leq\frac{\gamma_{k}}{t_{k}}\) is satisfied, then the \(IGDN\) iterations (26) satisfy \(F_{k+1}\leq F_{k}\)._  
Proof. : As a consequence of (26) and (27), one can verify  
\[F_{k+1}=F_{k}-t_{k}\gamma_{k+1}\gamma_{k}^{-1}F_{k}=\Big{(}1-t_{k}\gamma_{k+1} \gamma_{k}^{-1}\Big{)}F_{k}. \tag{28}\]In view of \(t_{k},\gamma_{k+1},\gamma_{k}\geq 0\), it follows that \(1-t_{k}\gamma_{k+1}\gamma_{k}^{-1}\leq 1\). On the other hand, the inequality \(1-t_{k}\gamma_{k+1}\gamma_{k}^{-1}\geq 0\) is satisfied in the case \(\gamma_{k+1}\leq\frac{\gamma_{k}}{t_{k}}\). Now, (28) implies \((F_{k+1})_{i}\leq(F_{k})_{i}\), \(i=1,\ldots,n\), which needs to be proven. So, appropriate update \(\gamma_{k+1}\) can be defined as follows:  
\[\gamma_{k+1}=\left\{ \begin{array}{ll}\frac{\mathbf{y}_{k}^{\mathrm{T}} \mathbf{y}_{k}}{\mathbf{y}_{k}\mathbf{z}_{k}}=\frac{\mathbf{z}_{k}^{\mathrm{ T}}\mathbf{y}_{k}}{\mathbf{z}_{k}^{\mathrm{T}}\mathbf{z}_{k}},& \mathbf{y}_{k}^{\mathrm{T}}\mathbf{z}_{k}\geq 0,\\ \frac{\gamma_{k}}{t_{k}},&\mathbf{y}_{k}^{\mathrm{T}}\mathbf{z}_{k}<0.\end{array} \right. \tag{29}\]  
Now, we are able to generate the value of the next approximation in the form  
\[\mathbf{x}_{k+2}=\mathbf{x}_{k+1}-t_{k+1}\gamma_{k+1}^{-1}F_{k+1}. \tag{30}\]  
The step size \(t_{k+1}\) in (30) can be determined using the nonmonotone line search.