## 3 Maximally Decreasing Residual and Automatically Preserving Orthogonality
: _For GMRES, if \(\eta_{k}\
eq 1\) is happened at the \(k\)th step, the orthogonality in Equation (5) cannot be continued after the \(k\)th step, i.e.,_  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\frac{\mathbf{r}_{k}^{T}\mathbf{A}\mathbf{u}_{ k}}{\|\mathbf{A}\mathbf{u}_{k}\|^{2}}\mathbf{u}_{k}. \tag{18}\]1  
_moreover, if \(\eta_{k}<1/2\), the residual does not decrease, i.e.,_  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\frac{\mathbf{r}_{k}^{T}\mathbf{A}\mathbf{u}_{ k}}{\|\mathbf{A}\mathbf{u}_{k}\|^{2}}\mathbf{u}_{k}. \tag{18}\]2  
_For ROGMRES in Equation (18), the residual is absolutely decreased:_  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\frac{\mathbf{r}_{k}^{T}\mathbf{A}\mathbf{u}_{ k}}{\|\mathbf{A}\mathbf{u}_{k}\|^{2}}\mathbf{u}_{k}. \tag{18}\]3  
_and the orthogonality \(\mathbf{r}_{k+1}\cdot(\mathbf{A}\mathbf{u}_{k})=0\) holds._  
Proof. : Applying \(\mathbf{A}\) to \(\mathbf{x}_{k+1}=\mathbf{x}_{k}+\mathbf{V}_{k}\mathbf{y}_{k}\) in Algorithm GMRES yields  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\frac{\mathbf{r}_{k}^{T}\mathbf{A}\mathbf{u}_{ k}}{\|\mathbf{A}\mathbf{u}_{k}\|^{2}}\mathbf{u}_{k}. \tag{18}\]4  
where \(\mathbf{u}_{k}=\mathbf{V}_{k}\mathbf{y}_{k}\). It follows from Equation (23) that  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\frac{\mathbf{r}_{k}^{T}\mathbf{A}\mathbf{u}_{ k}}{\|\mathbf{A}\mathbf{u}_{k}\|^{2}}\mathbf{u}_{k}. \tag{18}\]5  
which can be rearranged to  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\frac{\mathbf{r}_{k}^{T}\mathbf{A}\mathbf{u}_{ k}}{\|\mathbf{A}\mathbf{u}_{k}\|^{2}}\mathbf{u}_{k}. \tag{18}\]6  
and then by Equation (31),  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\frac{\mathbf{r}_{k}^{T}\mathbf{A}\mathbf{u}_{ k}}{\|\mathbf{A}\mathbf{u}_{k}\|^{2}}\mathbf{u}_{k}. \tag{18}\]7  
If \(\eta_{k}\
eq 1\), we can prove Equation (28). Taking the squared norm of Equation (31) and using Equation (23) generates  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\frac{\mathbf{r}_{k}^{T}\mathbf{A}\mathbf{u}_{ k}}{\|\mathbf{A}\mathbf{u}_{k}\|^{2}}\mathbf{u}_{k}. \tag{18}\]8  
If \(\eta_{k}<1/2\), \((1-2\eta_{k})\|\mathbf{A}\mathbf{u}_{k}\|^{2}>0\) and Equation (29) is proven. For ROGMRES, taking the squared norm of Equation (26), we have  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\frac{\mathbf{r}_{k}^{T}\mathbf{A}\mathbf{u}_{ k}}{\|\mathbf{A}\mathbf{u}_{k}\|^{2}}\mathbf{u}_{k}. \tag{18}\]9  
Because \((\mathbf{r}_{k}^{\mathrm{T}}\mathbf{A}\mathbf{u}_{k})^{2}/\|\mathbf{A}\mathbf{ u}_{k}\|^{2}>0\), Equation (30) is proven. The orthogonality \(\mathbf{r}_{k+1}\cdot(\mathbf{A}\mathbf{u}_{k})=0\) was proven in Equation (24) of Lemma 2. Corollary 1: _For GMRES, the orthogonality of the consecutive residual vector is equivalent to the maximality of \(h\) in Equation (22), where_  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\eta\mathbf{u}_{k}; \tag{19}\]0  
Proof: By means of Equation (34), to satisfy the orthogonality condition of the residual vector we require \(\eta_{k}=1\); it implies Equation (37) by the definition in Equation (23). At the same time, Equation (36) changes to  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\eta\mathbf{u}_{k}; \tag{19}\]1  
Inserting \(\eta_{k}=1\) into Equation (22), we have  
\[\mathbf{x}_{k+1}=\mathbf{x}_{k}+\eta\mathbf{u}_{k}; \tag{19}\]2  
of which \(\mathbf{r}_{k}^{\mathrm{T}}\mathbf{A}\mathbf{u}_{k}=\|\mathbf{A}\mathbf{u}_{k }\|^{2}\) in Equation (37) was used. For GMRES, obtaining the maximal value of \(h\) and preserving the orthogonality are the same. Remark 1: _To maintain \(\eta_{k}=1\) in GMRES is a key issue for preserving the orthogonality and for the maximality of reducing the residual.