# A Scaling Transition Method from SGDM to SGD with 2 ExpLR Strategy
## 6 Experiments  
In this section, to verify the performance of the proposed TSGD and TSGD+2 ExpLR, we compared them with other algorithms, including SGDM and Adam. Specifically, we used IRIS, CASIA-FaceV5, and CIFAR classification tasks in the experiments. We set the same random seed through PyTorch to ensure fairness. The architecture we chose for the experiments was ResNet-18. The computational setup is shown in Table 1. Our implementation is available at [[https://github.com/kunzeng/TSGD](https://github.com/kunzeng/TSGD)]([https://github.com/kunzeng/TSGD](https://github.com/kunzeng/TSGD)) (accessed on 20 November 2022).  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline OS & CentOS 8.3 & Dataset storage location & OS File system \\ \hline Processor-CPU & Intel Core i7-6500U & CPU-Memory & 32.0 GB \\ \hline Processor-GPU & Quadro P600 & GPU-Memory & 2.0 GB \\ \hline Programming language & Python 3.7.10 & Development framework & Pytorch 1.7.0 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The computational setup.  
**IRIS and BP neural network.** The IRIS dataset is a classic dataset for classification, machine learning, and data visualization. This dataset includes three iris species with 50 samples of each and some properties of each flower. An excellent way to determine the correct learning rates is to perform experiments using a small but representative sample of the training set, and the IRIS dataset is a good fit. Due to the small number of samples, we used all 150 samples as the training set. The performance of each algorithm is represented by the accuracy and loss value on the training set. We set the relevant parameters using empirical values and a grid search: samplesize = 150, epoch = 200, batchsize = 1, lr(SGD: 0.003, SGDM: 0.05, TSGD: 0.03), up_lr = 0.5, low_lr = 0.005, rho = 0.977237, and rho1 = rho2 = 0.962708. The experimental results show that the method achieves its expected effect. In Figures 4 and 5, the training process of SGDM has more oscillations, and the convergence speed of SGD is slower. However, TSGD is more stable and faster than SGDM and SGD. It is astonishing that the TSGD+2 ExpLR algorithm is stable during training and increases accuracy quickly.  
**CASIA-FaceV5 and ResNet18.** The CASIA-FaceV5 contains 2500 color facial images of 500 subjects. All face images are 16-bit color BMP files, and the image resolution is 640*480. Typical intra-class variations include the illumination, pose, expression, and eye-glasses imaging distance, which provides a remarkable ability to distinguish between different algorithms. We preprocessed it for better training and reshaped it to \(100\times 100\). The parameters were set as: samplesize = 2500, epoch = 100, batchsize = 10, lr = 0.1, \(lr_{u}\) = 0.5, \(lr_{l}\) = 0.005, then \(T\) = _ceil(samplesize/batchsize)_* epoch = 25,000, and thus \(\rho=10^{\left(log(\rho^{T},10)/T\right)}\approx 0.999815\). This is also applicable to \(\rho_{1}=\rho_{2}=0.999696\). The experi  
Figure 4: The accuracy of IRIS-BP-neural-network on the training set.  
Figure 5: The loss of IRIS-BP-neural-network on the training set.  
mental results are in Figures 6 and 7. Although we can observe that the accuracy of the TSGD algorithm does not exceed 1 Adam, it exceeds the SGD algorithm, and the training process is more stable. The main reason may be that the CASIA-FaceV5 dataset has fewer samples and more categories, and each category has only five samples. The Adam algorithm is more suitable for this scenario. The accuracy of the TSGD+2 ExpLR algorithm is much higher than that of other algorithms. The loss value is also much lower than other algorithms, and the training process is more stable.