# Improved Gradient Descent Iterations for Solving Systems of Nonlinear Equations
## 2 Multiple Step-Size Methods for Solving SNE
### A Class of Accelerated Double Step Size (ADSSN) Methods  
If the steplength \(t_{k}^{2}\) is replaced by another steplength \(l_{k}\) in (35), it can be obtained  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+t_{k}\mathbf{d}_{k}+l_{k}\mathbf{c}_{k}. \tag{39}\]Here, the parameters \(t_{k},t_{k}\geq 0\) are two independent step size values, and the vectors \(\mathbf{d}_{k}\), \(\mathbf{c}_{k}\) define the search directions of the proposed iterative scheme (39). Motivation for this type of iterations arises from [60]. The author of this paper suggested a model of the form (39) with two-step size parameters. This method is actually defined by substituting the parameter \({t_{k}}^{2}\) from (35) with another step size parameter \(l_{k}\). Both step size values are computed by independent inexact line search algorithms. Since we aim to unify search directions, it is possible to use  
\[\mathbf{d}_{k}:=-\gamma_{k}^{-1}F_{k},\quad\mathbf{c}_{k}:=-F_{k}. \tag{40}\]  
The substitution of chosen parameters (40) into (39) produces  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}-\Big{(}t_{k}\gamma_{k}^{-1}+l_{k}\Big{)}F_{k}. \tag{41}\]  
The final step size, \(\Big{(}t_{k}\gamma_{k}^{-1}+l_{k}\Big{)}\), in the iterations (41) are defined combining three step size parameters: \(t_{k}\), \(l_{k}\), and \(\gamma_{k}\). Again, the parameter \(\gamma_{k+1}\) is defined using the Taylor series of the form  
\[F(\mathbf{x}_{k+1})=F(\mathbf{x}_{k})-\gamma_{k+1}\Big{(}t_{k}\gamma_{k}^{-1} +l_{k}\Big{)}F(\mathbf{x}_{k}).\]  
As a consequence, \(\gamma_{k+1}\) can be computed by  
\[\gamma_{k+1}=-\frac{\gamma_{k}F_{k}^{\mathrm{T}}\mathbf{y}_{k}}{(t_{k}+\gamma _{k}l_{k})F_{k}^{\mathrm{T}}F_{k}}.\]  
**Theorem 2**. : _If the condition \(\gamma_{k+1}\leq\frac{\gamma_{k}}{l_{k}+\gamma_{k}l_{k}}\) holds, then the iterations (41) satisfy \(F_{k+1}\leq F_{k}\)._  
Proof. : Taking (27) in conjunction with (41), one can verify  
\[F_{k+1}=F_{k}-\gamma_{k+1}\Big{(}t_{k}\gamma_{k}^{-1}+l_{k}\Big{)}F_{k}=F_{k} \Big{(}1-\gamma_{k+1}\Big{(}t_{k}\gamma_{k}^{-1}+l_{k}\Big{)}\Big{)}.\]  
Clearly, \(\gamma_{k+1}\leq\frac{\gamma_{k}}{l_{k}+\gamma_{k}l_{k}}\) implies \(1-\gamma_{k+1}\Big{(}t_{k}\gamma_{k}^{-1}+l_{k}\Big{)}\geq 0\). The proof follows from \(t_{k}\geq 0,\gamma_{k+1},\gamma_{k}\geq 0\), which ensures \(1-\gamma_{k+1}\Big{(}t_{k}\gamma_{k}^{-1}+l_{k}\Big{)}\leq 1\). In view of Theorem 2, it is reasonable to define the following update for \(\gamma_{k+1}\) in the \(ADSSN\) method:  
\[\gamma_{k+1}=\left\{ \begin{array}{ll}-\frac{\gamma_{k}F_{k}^{\mathrm{T}} \mathbf{y}_{k}}{(t_{k}+\gamma_{k})F_{k}^{\mathrm{T}}F_{k}^{\mathrm{T}}},&\frac {F_{k}^{\mathrm{T}}\mathbf{y}_{k}}{F_{k}^{\mathrm{T}}F_{k}}\
otin(-1,0),\\ \frac{\gamma_{k}}{l_{k}+\gamma_{k}l_{k}},&\frac{F_{k}^{\mathrm{T}}\mathbf{y}_ {k}}{F_{k}^{\mathrm{T}}F_{k}}\in(-1,0).\end{array} \right. \tag{42}\]  
Once the accelerated parameter \(\gamma_{k+1}>0\) is determined, the values of step size parameters \(t_{k+1}\) and \(l_{k+1}\) are defined. Then, it is possible to generate the next point:  
\[F_{k+2}=F_{k+1}-\gamma_{k+2}\Big{(}t_{k+1}\gamma_{k+1}^{-1}+l_{k+1}\Big{)}F_{k +1}.\]  
In order to derive appropriate values of the parameters \(t_{k+1}\) and \(l_{k+1}\), we investigate the function  
\[\Phi_{k+1}(t,l)=F_{k+1}-\gamma_{k+2}\Big{(}\gamma_{k+1}^{-1}\,t+l\Big{)}F_{k+1}.\]  
The gradient of \(\Phi_{k+1}(t,l)\) is equal to  
\[\mathbf{g}(\Phi_{k+1}(t,l))=\big{\{}\Phi_{k+1}(t,l)_{t}^{\prime},\Phi_{k+1}(t, l)_{l}^{\prime}\big{\}}=\Big{\{}-\gamma_{k+2}\gamma_{k+1}^{-1}F_{k+1},- \gamma_{k+2}F_{k+1}\Big{\}}. \tag{43}\]  
Therefore,  
\[\Phi_{k+1}(0,0)=F_{k+1}.\]In addition,  
\[\textbf{g}(\Phi_{k+1}(t,l))=\{0,0\}\Longleftrightarrow F_{k+1}=0. \tag{44}\]  
Therefore, the function \(\Phi_{k+1}(t,l)\) is well-defined. Step scaling parameters \(t_{k}\) and \(l_{k}\) can be determined using two successive line search procedures (31).