# Automatic Microscopy Analysis with Transfer Learning for Classification of Human Sperm
## 2 Materials and Method
### Implementation  
To better compare with the previous methods, the 5-fold rotation estimation was employed in our approach. The dataset was randomly divided into five sub-samples, of which a single sub-sample was retained as the test data for the verification model, and the other four sub-samples formed the training set. Cross-validation was repeated five times, and each sub-sample was verified once. The final results were averaged five times to achieve the single performance estimation for the proposed model. Another benefit of the 5-fold cross-validation is that it can alleviate overfitting, which is a common problem for training small datasets like HuSHeM. The number of images in the HuSHeM dataset allocated in each fold is summarized in Table 2.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Hyperparameters** & **Value** & **Hyperparameters** & **Value** \\ \hline Fully Connected 1 & 1024 & Dropout 2 & 0.5 \\ Batch Normalization 1 & 1024 & Fully Connected 3 & 4 \\ Dropout 1 & 0.5 & Learning rate & 0.00001 \\ Fully Connected 2 & 256 & Batch size & 64 \\ Batch Normalization 2 & 256 & Training epochs & 2000 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The hyperparameters in the classifier of the proposed network.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Group** & **Data Types** & **Normal** & **Tapered** & **Pyiform** & **Amorphous** \\ \hline \multirow{2}{*}{Group 0} & Train & 43 & 42 & 46 & 42 \\  & Test & 11 & 11 & 11 & 10 \\ \hline \multirow{2}{*}{Group 1} & Train & 43 & 42 & 46 & 42 \\  & Test & 11 & 11 & 11 & 10 \\ \hline \multirow{2}{*}{Group 2} & Train & 43 & 43 & 45 & 42 \\  & Test & 11 & 10 & 12 & 10 \\ \hline \multirow{2}{*}{Group 3} & Train & 43 & 43 & 45 & 41 \\  & Test & 11 & 10 & 12 & 11 \\ \hline \multirow{2}{*}{Group 4} & Train & 44 & 42 & 46 & 41 \\  & Test & 10 & 11 & 11 & 11 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Number of images allocated to each group for the 5-fold cross-validation.  
As the feature extraction part of the original AlexNet and its pre-training parameters remained intact, the code for this part was programmed using the built-in algorithms of the PyTorch document. The feature interpreter and regression part of the proposed model was programmed in PyTorch (1.7.1) with Python (3.7.8). The algorithm was trained on a DELL workstation with a CPU of Intel(R) Xeon(R) Gold 6226R @2.90 GHz and an NVIDIA Quadro RTX 4000 GPU with 8 GB memory using the Jupyter Notebook (6.1.5). The Adam optimizer was chosen to be the learning algorithm with a fixed learning rate of 0.00001 in the proposed model. According to the classification accuracy, the hyperparameters of the model were optimized iteratively. Finally, the best hyperparameter configuration for a specific task here is shown in Table 1.  
During the training process, the parameters of the transfer learning part (i.e., feature extractor) were frozen. Only the parameters in the redesigned prediction layers were trained to accelerate the parameter optimization process and save computing resources. Figure 5a shows how the test accuracy changes with training steps. The test accuracy grows rapidly in the initial part of the training and then rises gradually. A corresponding change is also observed in the loss, as shown in Figure 5b. The loss drops sharply at the beginning and then decreases progressively. However, the test accuracy does not necessarily increase with the training steps, possibly due to overfitting problems. Sometimes, the best test accuracy even appeared in the first half of training. For example, the test accuracy reaches the maximum when the number of training steps is around 900, although the total number of training steps is set to 2000. During the validation process, the test set was enlarged by 64 times to reduce the calculation error caused by the limited test images.