## 4 Results
### Convolutional Neural Network (CNN) Result
\begin{table}
\begin{tabular}{c l l l l l} \hline \hline \multicolumn{6}{c}{**Model Performance with Respect to Learning Rate**} \\ \hline \multicolumn{2}{c}{**Cross-Entropy \& SDG**} & \multicolumn{4}{c}{**Learning Rate**} \\ \hline \multicolumn{2}{c}{**Model**} & **Criteria** & \(lr=1\times 10^{-1}\) & \(lr=1\times 10^{-2}\) & \(lr=1\times 10^{-3}\) & \(lr=1\times 10^{-4}\) \\ \hline \multirow{4}{*}{LeNet} & Train Loss & \(2.78\times 10^{-5}\) & \(2.32\times 10^{-4}\) & \(2.15\times 10^{-3}\) & \(3.52\times 10^{0}\) \\ \cline{2-6}  & Train Score (\%) & 98.5995 & 98.0359 & 87.6746 & 24.3396 \\ \cline{2-6}  & Test Loss & \(8.94\times 10^{-6}\) & \(1.91\times 10^{-4}\) & \(2.15\times 10^{-3}\) & \(1.86\times 10^{0}\) \\ \cline{2-6}  & Test Score (\%) & **99.4086** & 99.0645 & 98.8136 & 46.2164 \\ \hline \multirow{4}{*}{AlexNet} & Train Loss & Inf & \(9.71\times 10^{-1}\) & \(3.02\times 10^{-3}\) & \(9.13\times 10^{-3}\) \\ \cline{2-6}  & Train Score (\%) & 2.2718 & 84.3845 & 98.1127 & 97.8732 \\ \cline{2-6}  & Test Loss & Inf & \(1.69\times 10^{0}\) & \(3.04\times 10^{-3}\) & \(6.04\times 10^{-3}\) \\ \cline{2-6}  & Test Score (\%) & 2.2727 & 81.7914 & 97.6605 & **99.2946** \\ \hline \multirow{4}{*}{VGG} & Train Loss & Inf & \(1.70\times 10^{-3}\) & \(7.83\times 10^{-3}\) & \(8.05\times 10^{-3}\) \\ \cline{2-6}  & Train Score (\%) & 2.2727 & 87.4872 & 98.7905 & 98.1264 \\ \cline{2-6}  & Test Loss & Inf & \(2.56\times 10^{-2}\) & \(6.85\times 10^{-3}\) & \(2.74\times 10^{-3}\) \\ \cline{2-6}  & Test Score (\%) & 2.2727 & 93.5286 & 98.1041 & **99.3759** \\ \hline \multirow{4}{*}{ResNet} & Train Loss & \(3.95\times 10^{-1}\) & \(2.14\times 10^{-2}\) & \(7.55\times 10^{-4}\) & \(1.13\times 10^{-3}\) \\ \cline{2-6}  & Train Score (\%) & 84.8318 & 96.0959 & 99.4000 & 98.9268 \\ \cline{2-6}  & Test Loss & \(3.14\times 10^{-1}\) & \(1.08\times 10^{-2}\) & \(7.47\times 10^{-3}\) & \(7.79\times 10^{-4}\) \\ \cline{2-6}  & Test Score (\%) & 83.6241 & 98.9127 & 99.4386 & **99.7723** \\ \hline \multirow{4}{*}{Transformer (Baseline)} & Train Loss & \(3.78\times 10^{0}\) & \(2.69\times 10^{-1}\) & \(2.27\times 10^{0}\) & \(3.47\times 10^{0}\) \\ \cline{2-6}  & Train Score (\%) & 0.0356 & 0.9609 & 0.5872 & 0.1103 \\ \cline{1-1} \cline{2-6}  & Test Loss & \(3.86\times 10^{0}\) & \(5.27\times 10^{0}\) & \(3.78\times 10^{0}\) & \(3.93\times 10^{0}\) \\ \cline{1-1} \cline{2-6}  & Test Score (\%) & 0.0000 & 0.0282 & **0.0704** & 0.0423 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: All of the models use cross-entropy for the loss and stochastic gradient descent (SDG) for the optimizer. The bold value for the test score represents the best learning rate for performance. All of the loss values are in scientific notation while the scores are in percentage values, where “Inf” represent a very large loss value. LeNet, as an old-fashioned convolutional neural network, shows a slightly worse performance compared with other models. Figure 11 shows the loss curve for LeNet with respect to each of the optimizers. The largest train loss optimizer for the LeNet is Adagrad with a value of \(3.57\times 10^{9}\); the smallest train loss optimizers for the LeNet are Adamax, RAdam, and RMSprop, with a value of \(0.00\times 10^{0}\). The largest test loss optimizer for the LeNet is Adagrad with a value of \(3.69\times 10^{0}\) and the smallest train loss optimizer for the LeNet is Adamax with a value of \(1.91\times 10^{-7}\). The best train score optimizer for LeNet is Adama, with a score of 98.1396% and the worst train score optimizer for LeNet is Adadelta, with a score of 5.0468%. The best test score optimizer for LeNet is RAdam, with a score of 98.4814% and the worst test score optimizer for LeNet is Adadelta, with a score of 3.8445%. The results show that the RAdam optimizer has a better performance in combination with the LeNet, which can be supported by the study by [PERSON] et al. (2018).