# A Scaling Transition Method from SGDM to SGD with 2 ExpLR Strategy
## 3 TSGD Algorithm  
Based on the above analysis and the basis of the QHM algorithm, we provide a middle ground that combines preferable features from both. This includes the advantages of SGDM with a fast training speed and of SGD with high accuracy. A scaling transition method from SGDM to SGD is proposed--named TSGD. A scaling function \(\rho^{t}\) is introduced to gradually scale the momentum direction to the gradient direction as the iteration, which achieves a smooth and stable transition from SGDM to SGD. The specific algorithm is represented in Algorithm 1.  
```
0: initial parameters: \(\theta_{1}\in\mathcal{F}\), \(\eta\), \(\beta\), \(\rho\).
0: Initialize: \(m_{0}=0\).
1:for\(t=1\) to \(T\)do
2:\(g_{t}=\
abla f_{t}(\theta_{t})\)
3:\(m_{t}=\beta m_{t-1}+(1-\beta)g_{t}\)
4:\(\hat{m}_{t}=(m_{t}-g_{t})\rho^{t}+g_{t}\)
5:\(\theta_{t+1}=\theta_{t}-\eta\hat{m}_{t}\)
6:endfor
```  
**Algorithm 1** A Scaling Transition method from SGDM to **SGD** (TSGD)  
In Algorithm 1, \(g_{t}\) is the gradient of loss function, \(f\) in the \(t\)-th iteration, and \(m_{t}\) is the momentum. \(\hat{m}_{t}\) is the scaled momentum, \(\theta_{t}\) is the optimized parameter, \(\rho^{t}\) is the scaling function, and \(\rho^{t}\in[0,1]\). We provide recommended rules of the hyperparameters \(\rho\) and \(\beta\), which are \(\rho^{T}=0.01\) and \(\beta=0.9\), where \(T\) is the number of iterations, \(T=ceil(samplesize/batchsize)*epochs\).  
[PERSON] et al. (2015) made a detailed convergence analysis on the general form of the QHM algorithm. The TSGD algorithm proposed in this section conforms to the general  
Figure 2: The test accuracy and training loss for ResNet18 on CIFAR10.  
form of the QHM algorithm. Therefore, the TSGD algorithm has the same convergence conclusion as the general form of the QHM algorithm. Theorem 1 gives the convergence theory of the TSGD algorithm based on the literature [35].  
Theorem 1: _Let \(f\) satisfy the condition in the literature [35]. Additionally, assume that \(0\leq\rho^{t}\leq 1\) and the sequences \(\{\eta_{t}\}\) and \(\{\beta_{t}\}\) satisfy the following conditions:_  
\[\sum_{t=0}^{\infty}\eta_{t}=\infty,\sum_{t=0}^{\infty}\eta_{t}^{2}=\infty, \lim_{t\to\infty}\beta_{t}=0,\tilde{\beta}\overset{\Delta}{=}\sup_{t}\beta_{t }<1.\]  
_Then, the sequence \(\{\theta_{t}\}\) that is generated by the TSGD Algorithm 1 satisfies:_  
\[\lim_{t\to\infty}\lVert\
abla f(\theta_{t})\rVert=0.\]  
_Moreover, we have:_  
\[\lim_{t\to\infty}\sup f(\theta_{t})=\limsup_{t\to\infty,\lVert\
abla f(\theta _{t})\rVert\to 0}f(\theta_{t}).\]  
Theorem 1 implies that TSGD can converge, which is similar to SGD-type optimizers. Through exponential decay in step 4 of the TSGD Algorithm, the updated direction smoothly and stably transforms the momentum direction of SGDM to the gradient direction of SGD as iterations. In the early stage of training, the number of iterations is small, \(\rho^{t}\) is close to \(1\), and the updated direction is  
\[\hat{m}_{t}\gets m_{t},\]  
which can speed up the training speed. In the later stage, with the increase of iterations, the number of iterations is large, and \(\rho^{t}\) is close to \(0\). The updated direction is gradually transformed to  
\[\hat{m}_{t}\gets g_{t}.\]  
When the iterative sequence closes to the optimal solution, gradient direction is used to update the parameters and is more accurate. Thus, TSGD has the advantages of both faster speed of SGDM and high accuracy of SGD. TSGD does not need to calculate the second moment of the gradient, which saves computational resources compared to the adaptive gradient descent method.