## 4 Performance Evaluation
### Experimental Results  
The results of the experiments are shown in Table 1. In addition, Figure 7 shows graphs of the average values in Table 1 of each model. Table 1 shows the results of the validation set and test set at the time when the loss was the lowest with respect to the validation set in 50 epochs. Since the experiments were performed five times each, the mean and standard deviation of the validation set and the test set are respectively shown. Additionally, the overall mean and standard deviation of the validation set and test set are shown too. \"Average (validation/test)\" is the average of five times for each validation set or each test set, respectively. \"Total Average\" is the all average of the validation set and test set. Here, we used accuracy, _F-score_ and Matthews correlation coefficient (MCC) as performance evaluation indicators. Each formula is defined by Equations (18)-(20) as follows:  
\[Accuracy=(TP+TN)/(TP+TN+FP+FN) \tag{18}\]  
\[F-score=2 TP/(2 TP+FP+FN) \tag{19}\]  
\[MCC=\frac{(TP\times TN)-(FP\times FN)}{\sqrt{(TP+FP)\times(TP+FN)\times(TN+FP) \times(TN+FN)}}\, \tag{20}\]  
where \(TP,\ TN,\ FP,\ FN\) are the number of true positives, true negatives, false positives, and false negatives, respectively. Accuracy and _F-score_ is a value between 0 and 1, the closer they are to 1, the better the performance is. For unbalanced data, _F-score_ can perform more accurate performance evaluation than accuracy. Note that the _F1-score_ shown in the experiments is macro-average. The _MCC_ is a correlation coefficient value between \(-1\) and \(+1\), and \(+1\) indicates a perfect prediction, 0 an average random prediction and \(-1\) an inverse prediction. The _MCC_ was calculated for more accurate evaluation, including inverse predictions. In Table 1, accuracy and F-score show the value rounded off the third decimal place, and _MCC_ shows the value rounded off the fourth decimal place. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Layers**} & \multicolumn{1}{c}{**Model**} & \multicolumn{1}{c}{**Units**} & \multicolumn{1}{c}{**Indications**} & \multicolumn{1}{c}{**1**} & \multicolumn{1}{c}{**2**} & \multicolumn{1}{c}{**3**} & \multicolumn{1}{c}{**4**} & \multicolumn{1}{c}{**5**} & \multicolumn{1}{c}{**Average (Validation/Test)**} & \multicolumn{1}{c}{**Total Average**} \\ \hline \multicolumn{1}{c}{} & & & Acc. (\%) & 80.56/69.44 & 77.28/58.33 & 83.33/61.11 & 69.44/66.67 & 61.11/77.78 & 74.44 \(\pm\) 813/66.67 \(\pm\) 6.80 & 70.56 \(\pm\) 8.44 \\ \multicolumn{1}{c}{} & & & F1. (\%) & 80.54/69.42 & 77.14/58.30 & 83.33/57.86 & 69.42/66.56 & 60.99/77.78 & 74.29 \(\pm\) 812/65.98 \(\pm\) 7.43 & 70.14 \(\pm\) 8.82 \\ \multicolumn{1}{c}{} & & & Acc. (\%) & 0.59/61.46 & 0.67/0.202 & 0.39/0.335 & 0.242/0.356 & 0.498 \(\pm\) 0.165/0.340 \(\pm\) 0.130 & 0.430 \(\pm\) 0.167 \\ \cline{2-11} \multirow{3}{*}{LSTM} & & Acc. (\%) & 75.07/70.70 & 75.00/33.53 & 75.07/66.67 & 69.41/66.67 & 55.56/77.78 & 70.70 \(\pm\) 75.48/89.69 \(\pm\) 69.64 \(\pm\) 7.24 \\ \cline{2-11}  & & & Acc. (\%) & 74.08/74.98 & 74.02/83.80 & 74.02/83.69 & 69.41/63.67 & 55.42/77.78 & 69.57 \(\pm\) 73.48/85.70 \(\pm\) 67.81 \(\pm\) 7.42 \\ \multicolumn{1}{c}{} & & & Acc. (\%) & 0.50/50.17 & 0.503/44.07 & 0.390/35.54 & 0.121/0.556 & 0.418 \(\pm\) 0.108/40.05 \(\pm\) 0.136 & 0.411 \(\pm\) 0.150 \\ \cline{2-11} \multirow{3}{*}{} & & Acc. (\%) & 63.89/75.00 & 75.00/55.56 & 69.44/61.61 & 63.89/72.22 & 52.78/83.33 & 65.00 \(\pm\) 7.37/94.94 \(\pm\) 9.94 & 67.22 \(\pm\) 9.02 \\ \cline{2-11}  & & & Acc. (\%) & 64.07/48.35 & 74.02/54.32 & 68.23/99.09 & 63.64/72.14 & 69.63/82.86 & 63.80 \(\pm\) 68.98/8.95 \(\pm\) 10.30 & 66.27 \(\pm\) 9.04 \\ \multicolumn{1}{c}{} & & Acc. (\%) & 0.30/50.07 & 0.54/11.12 & 61.42/0.288 & 0.24/0.247 & 0.40/0.641 & 0.631 \(\pm\) 0.102/0.106 & 0.312 \(\pm\) 0.106 \\ \hline \multirow{11}{*}{GRU} & & Acc.