# A Scaling Transition Method from SGDM to SGD with 2 ExpLR Strategy
## 4 The Warmup Decay Learning Rate Strategy with Double Exponential Functions  
The learning rate (LR) is a crucial hyperparameter to tune for practical training of deep neural networks and controls the rate or speed at which the model learns each iteration [36]. If the learning rate is too large, this may cause oscillation and divergence. On the contrary, training may progress slowly and even stop if the rate is too small. Thus, many learning rate strategies have been proposed and appear to work well, such as warmup [4], decay [37], restart techniques [38], and cyclic learning rates [39]. In this section, we mainly focus on the warmup and decay strategies.  
On the one hand, the idea of warmup was proposed in ResNet [4]. They used 0.01 to warm up the training until the training error was below 80% (about 400 iterations) and then went back to 0.1 and continued training. In the early stage of training, due to many random parameters in the model, if a large learning rate is used, the model may be unstable and fall into a local optimum, which is challenging to fix. Therefore, we use a small learning rate to make the model learn certain prior knowledge and then use a large learning rate for training when the model is more stable. It can use less aggressive learning rates at the start of training [40]. Implementations of the warmup strategy include constant warmup [4], linear warmup [41], and gradual warmup [40].  
On the other hand, decay is also a popular strategy in neural-network training. The decay can speed up the training using a larger learning rate at the beginning and can converge stably using a smaller learning rate after. Not only does this show good results [24; 25; 34; 42] in practical applications but also the learning rate is required to decay in the theoretical convergence analysis, such as \(\eta_{t}=1/t\)[43], \(\eta_{t}=1/\sqrt{t}\)[42]. The specific implementation of learning rate decay is as follows: step decay [26], linear attenuation [44], exponential decay [45], etc.  
To ensure better performance of the training algorithm, we combined the warmup and decay strategies. The warmup strategy was used for a small number of iterations in the early stage of training, and then the decay strategy was used to decrease the learning rate gradually. In this way, the model can learn stably early on, train fast in the middle, and converge accurately later.  
First, an exponential function is used in the warmup stage to gradually increase the learning rate from zero to the upper learning rate as shown in Figure 2(a). The numerical formula is as follows  
\[lr_{warmup}=lr_{u}\big{(}1-\rho_{1}^{t}\big{)}. \tag{9}\]  
Second, an exponential function is also used in the decay stage to gradually decrease the learning rate from the upper learning rate to the lower learning rate as shown in Figure 2(b). The numerical formula is as follows  
\[lr_{decay}=(lr_{u}-lr_{t})\rho_{2}^{t}+lr_{t}. \tag{10}\]  
The main idea of the warmup-decay learning rate strategy is that the learning rate can increase in the warmup stage and decrease in the decay stage. Therefore, we combine the above two processes, and the warmup-decay learning rate strategy with double exponential functions (2 ExpLR) is proposed as shown in Figure 2(c). The numerical formula is as follows  
\[lr_{w-d}=\big{(}(lr_{u}-lr_{t})\rho_{1}^{t}+lr_{t}\big{)}\big{(}1-\rho_{2}^{t }\big{)}. \tag{11}\]  
Similarly, we also provide recommended rules of the hyperparameters \(\rho_{1},\rho_{2}\) as with \(\rho\), that are \(\rho_{1}^{T}=\rho_{2}^{T}=lr_{t}*10^{-1}\). Thus, \(\rho_{1},\rho_{2}\) can be easily calculated using \(\rho_{1}=\rho_{2}=10^{(\lg l_{T}-1)/T}\).  
In Figure 3, we know that 2 ExpLR does not need to set the transition point from warmup to decay manually, compared with other strategies [24; 25; 26]. At the same time, we achieve a smooth and stable transition from warmup to decay through 2 ExpLR. This gives the training algorithm a faster convergence speed and higher accuracy.  
Figure 3: The schematic diagrams of each learning rate strategy.