## 4 Experimental Results
### Discussion about the Hyper-Parameters of the Proposed Semi-Supervised Learning
#### 4.5.5 Different Optimizers for Fine-Tuning  
As noted in Section 4.2, the SGDM optimizer is used instead of Adam in the fine-tuning stage. The following empirical study shows the influence of different optimizers for fine-tuning. The epochs of self-supervised learning were 100, and the remaining setups were the same as in Section 4.5.1. In the fine-tuning stage, the batch size was set to 64. The learning rates of the backbone and the classifier were set to 30 and \(1\times 10^{-3}\), respectively, with a cosine decay scheduler. Random Rotation, Random Scale, and Random PadCrop augmentation strategies were used. The optimizers involved in the experiment included Adam, stochastic gradient descent with momentum (SGDM), stochastic gradient descent (SGD), and RMSProp without the weight decay. All the training was conducted for 100 epochs.  
The results are shown in Table 9. SGDM achieves the best testing accuracy, and the performance of SGD slightly drops. However, Adam and RMSProp show a significant reduction in performance and seem to lose the knowledge transferred from self-supervised learning during fine-tuning. Adam and RMSProp are both adaptive learning rate approaches that can adjust the learning rate of each learnable parameter dynamically. We conjecture that their adaptive learning rate property results in the loss of learned representation. Possible future research is to explore what causes this phenomenon.  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Training Epoch in Self-Supervised Stage**} & \multicolumn{8}{c}{**Number of Labeled Data per Class for Fine-Tuning**} \\ \cline{2-10}  & **1** & **2** & **4** & **8** & **16** & **32** & **64** & **128** \\ \hline
50 & 53.44 & 63.78 & 66.02 & 74.24 & 83.83 & 93.81 & 97.28 & 97.62 \\
100 & 69.81 & 76.04 & **87.74** & **93.95** & **97.15** & **97.56** & **98.03** & **98.84** \\
200 & **76.33** & **86.94** & 87.30 & 92.52 & 94.54 & 96.65 & 97.25 & 98.38 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The test accuracy (%) of the MsvNetLite with different training epochs for self-supervised learning. The bold number is the best result in each column.  
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Data Augmentation Strategies**} & \multicolumn{8}{c}{**Number of Labeled Data per Class for Fine-Tuning**} \\ \cline{2-10}
**in Self-Supervised Learning Stage** & & & & & & & & & \\ \hline
**Rotation** & **Scale** & **PadCrop** & **1** & **2** & **4** & **8** & **16** & **32** & **64** & **128** \\ \hline ✓ & & & 22.28 & 31.33 & 43.66 & 52.26 & 73.06 & 74.35 & 84.31 & 96.18 \\ ✓ & ✓ & & 34.41 & 43.51 & 61.55 & 71.58 & 84.44 & 91.99 & 94.84 & 96.94 \\ ✓ & ✓ & ✓ & **69.81** & **76.04** & **87.74** & **93.95** & **97.15** & **97.56** & **98.03** & **98.84** \\ \hline \hline \end{tabular}
\end{table}
Table 8: The test accuracy (%) of the MsvNetLite with different data augmentation strategies in the self-supervised stage. The ✓ denotes this augmentation method is enabled. The bold number is the best result in each column.