# An Adaptive Optimization Method Based on Learning Rate Schedule for Neural Networks
## 3 Our Proposed Method  
Our method is based on the momentum method that stops the learning at the moment when it meets the global minimum value of the cost function, which is the moment when learning is well executed. By changing the learning rate, the following is obtained:  
\[\mathbf{w}_{i+1} = \mathbf{w}_{i}-\eta_{i}\mathbf{m}_{i}, \tag{5}\]  
where:  
\[\eta_{i}=\eta_{0}\frac{C(\mathbf{w}_{i})}{\sqrt{\mathbf{v}_{i}+ \epsilon}}, \tag{6}\]  
where \(\eta_{0}\) is a constant, and:  
\[\mathbf{v}_{i}=\sum_{k=1}^{i}\beta_{2}^{k-1}(F(\mathbf{w}_{i-k+1 }))^{2}.\]  
In particular, \(\epsilon\) is a constant introduced to prevent division by zero. **Lemma 1**: _The relationship between \(\mathbf{m}_{i}\) and \(\mathbf{v}_{i}\) is:_  
\[(\mathbf{m}_{i})^{2}\leq\left(\frac{1-\left(\frac{\beta_{1}^{2}} {\beta_{2}}\right)^{i}}{1-\left(\frac{\beta_{1}^{2}}{\beta_{2}}\right)} \right)\mathbf{v}_{i}.\]  
Proof. : The proof can be found in [19]. Using the relationship between \(\mathbf{m}_{i}\) and \(\mathbf{v}_{i}\), from Equation (5), we have:  
\[\implies\quad|\mathbf{w}_{i+1}-\mathbf{w}_{i}|\leq\eta_{0}C( \mathbf{w}_{i})\left(\frac{1-\left(\frac{\beta_{1}^{2}}{\beta_{2}}\right)^{i }}{1-\left(\frac{\beta_{1}^{2}}{\beta_{2}}\right)}\right)^{1/2}. \tag{7}\]  
Here, since \(\epsilon\) is to prevent division by zero, it can be considered as zero in the calculation. Therefore, the sequence \(\{\mathbf{w}_{i}\}\) also converges when the sequence \(\{\eta_{i}\}\) converges to zero. **Theorem 1**: _We need to find the parameter that minimizes the cost function \(C\). The limit value of our constructed sequence \(\{\mathbf{w}_{i}\}\) is a sequence that gives the value of the cost function \(C\) as zero. The closer the cost function \(C\) is to zero, the closer the relationship between \(y_{h,j}\) and \(y_{j}\) (the value of \(y_{h,j}\) made through the ANN approaches the output value \(y_{j}\))._Proof. : After a sufficiently large number \(\tau\), a sufficiently small value \(\eta_{i}\), and using the [PERSON]'s theorem, the following calculation is performed:  
\[C(\mathbf{w}_{i+1})=C(\mathbf{w}_{i})+\frac{\partial C}{\partial\mathbf{w}}( \mathbf{w}_{i})(\mathbf{w}_{i+1}-\mathbf{w}_{i})+O\Big{(}(\mathbf{w}_{i+1}- \mathbf{w}_{i})^{2}\Big{)}, \tag{8}\]  
where part of \(O\Big{(}(\mathbf{w}_{i+1}-\mathbf{w}_{i})^{2}\Big{)}\) represents the second order and is ignored. From Equations (5) and (8), we have:  
\[C(\mathbf{w}_{i+1}) = C(\mathbf{w}_{i})+\frac{\partial C}{\partial\mathbf{w}}( \mathbf{w}_{i})(\mathbf{w}_{i+1}-\mathbf{w}_{i})\] \[= C(\mathbf{w}_{i})-\eta_{0}C(\mathbf{w}_{i})\frac{\partial C}{ \partial\mathbf{w}}(\mathbf{w}_{i})\Bigg{(}\frac{\mathbf{m}_{i}}{\left( \mathbf{v}_{i}+\epsilon\right)^{1/2}}\Bigg{)}\] \[= C(\mathbf{w}_{i})\Bigg{(}1-\eta_{0}\Bigg{(}\frac{\left(\frac{ \partial C}{\partial\mathbf{w}}(\mathbf{w}_{i})\right)^{2}+\mu C(\mathbf{w} _{i})\frac{\partial C}{\partial\mathbf{w}}(\mathbf{w}_{i})+\beta_{1}\frac{ \partial C}{\partial\mathbf{w}}(\mathbf{w}_{i})\mathbf{m}_{i-1}}{\left( \mathbf{v}_{i}+\epsilon\right)^{1/2}}\Bigg{)}\Bigg{)}.\]  
Since \(\beta_{1}<1\), \(\epsilon\
eq 0\), and adjusting \(\eta_{0}\) and \(\mu\):  
\[0<\eta_{0}\Bigg{(}\frac{\left(\frac{\partial C}{\partial\mathbf{w}}(\mathbf{w} _{i})\right)^{2}+\mu C(\mathbf{w}_{i})\frac{\partial C}{\partial\mathbf{w}}( \mathbf{w}_{i})+\beta_{1}\frac{\partial C}{\partial\mathbf{w}}(\mathbf{w}_{i })\mathbf{m}_{i-1}}{\left(\mathbf{v}_{i}+\epsilon\right)^{1/2}}\Bigg{)}<1\]  
Therefore, the following is obtained:  
\[\eta_{i}=\eta_{0}\frac{C(\mathbf{w}_{i})}{\sqrt{\mathbf{v}_{i}+ \epsilon}}, \tag{6}\]0  
where:  
\[\eta_{i}=\eta_{0}\frac{C(\mathbf{w}_{i})}{\sqrt{\mathbf{v}_{i}+ \epsilon}}, \tag{6}\]1  
and \(\lim_{i\to\infty}C(\mathbf{w}_{i})=0\). In Theorem (1), it is possible that \(\delta\) is less than \(1\). Therefore, we explain in more detail that \(\delta\) is less than \(1\) in the following corollary. **Corollary 1**. : \(\delta\) _defined in Theorem (1) is less than 1._  
Proof.