# A Scaling Transition Method from SGDM to SGD with 2 ExpLR Strategy
## 2 Preliminaries  
**Optimization problem. **\(f:\mathbb{R}^{n}\rightarrow\mathbb{R}\) is the objective function, and \(f(\theta,\zeta)\in C^{1}\) is continuously differentiable, where \(\theta\) is the optimized parameter. Considering the following convex optimization problem [31]:  
\[\min_{\theta\in\mathbb{R}^{n}}f(\theta,\zeta), \tag{1}\]  
where \(\zeta\) is a stochastic variable. Applying the gradient descent method to solve the above minimization problem:  
\[\theta_{t+1}=\theta_{t}-\eta g_{t}, \tag{2}\]  
where \(\eta\) is the constant step size, \(g_{t}\) is the descent direction, and \(\theta\) is the parameter that needs to be updated. **SGD. ** SGD uses the current gradient of the loss function to update the parameters [32]:  
\[g_{t}\leftarrow\
abla f(\theta,\zeta), \tag{3}\]  
where \(f\) is the loss function and \(\
abla f(\theta,\zeta)\) is the gradient of the loss function. **SGDM. ** The momentum method considers the past gradient information and uses it to correct the direction, which speeds up the training. The update rule of the heavy-ball momentum method [33] is  
\[ \begin{split} m_{t}&=\beta m_{t-1}+(1-\beta)\
abla f (\theta,\zeta)\\ g_{t}&\gets m_{t},\end{split}  \tag{4}\]  
where \(m_{t}\) is the momentum and \(\beta\) is the momentum factor. **Efficiency of SGD and SGDM. ** In convex optimization, the gradient descent method can search the optimal global solution of the objective function by the steepest descent. However, it has a series of zigzag paths in the iteration process, which seriously affects the training speed. We can explore the root of this defect using theoretical analysis. Using exact line search to solve the convex optimization problems (1):  
\[ \begin{split}&\varphi(\lambda)=f\Big{(}\theta^{(t)}+\lambda d^{(t)},\zeta\Big{)}\\ & d^{(t)}=-\
abla f\Big{(}\theta^{(t)},\zeta\Big{)}.\end{split}  \tag{5}\]  
In order to find the minimum point along the direction \(d^{(t)}\) from \(x^{(t)}\), let  
\[\varphi^{\prime}(\lambda)=\
abla f\Big{(}\theta^{(t)}+\lambda_{t}d^{(t)}, \zeta\Big{)}^{\top}d^{(t)}=0. \tag{6}\]  
Thus,  
\[-\
abla f\Big{(}\theta^{(t+1)},\zeta\Big{)}^{\top}\
abla f\Big{(}\theta^{(t)},\zeta\Big{)}=0. \tag{7}\]  
We know that \(d^{(t+1)}=-\
abla f\Big{(}\theta^{(t+1)},\zeta\Big{)}\) and \(d^{(t)}=-\
abla f\Big{(}\theta^{(t)},\zeta\Big{)}\) are orthogonal. This shows that the path of the iterative sequence \(\{f(\theta^{t},\zeta)\}\) is a zigzag. When it is close to the minimum point, the step size of each movement will be tiny and seriously affects the speed of convergence as shown in Figure 0(a). [PERSON] et al. [9] proposed SGD with a heavy-ball momentum algorithm to mitigate the zigzag oscillation and accelerate training. The update direction of the SGDM is the EMA [10] of the current gradient and the past gradient to speed up the training speed and reduce the vibration in SGD. The momentum of the SGDM is  
\[m_{t}=\sum_{i=1}^{t}{(1-\beta)\beta^{t-i}g_{i}}. \tag{8}\]  
However, in the later stage of training, there is a defect in using EMA for the gradient. The accumulation of gradients may lead to a faster speed of momentum and not stop in time when it is close to the optimum. This may oscillate in a region around the optimal solution \(\theta^{*}\) and cannot stably converge as shown in Figure 0(b). At the same time, the gradients that are too far may have little helpful information, and the momentum can be calculated using the gradients of the last \(n\) times, such as AdaShift [34]. The negative gradient direction of the loss function is the optimal direction for the current parameter updating. The updating direction is no longer the fastest descent direction when using momentum. When it is close to the optimal solution, the stochastic gradient descent method is more accurate and is more likely to find the optimal point. For more information, we use the ResNet18 to train the CIFAR10 dataset, and the first 75 epochs use the SGDM algorithm (hyperparameter setting: weight_decay = 5 \(\times\) 10\({}^{-3}\)  
Figure 1: The performances of SGD and SGDM.