## 5 Experiment  
Each dataset utilized in this experiment consists of a total of 78,204 data; 50,050 training data, 12,513 validation data, and 15,641 test data. The investigation was carried out 50 times over the randomly selected data sets, the construction ratio of which was maintained. The results presented in this section are based on the average of the 50 runs. The CNN model updates the weights whenever the verification loss is reduced in the learning process. The early termination condition for the training process was set to proceed with 100 additional epochs of training from the point of update. Although all models satisfied the early termination condition in less than 200 epochs, we continued training up to epochs for visual comparison. Figure 5 shows a graph of the training and validation accuracy over different training epochs. These values were obtained by averaging the 50 models as mentioned previously. The training and validation loss is shown in Figure 6. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Layer** & **Type** & **Dimension** & **Kernel Size** & **Parameter \#** \\ \hline
1 & Batch Normalization & 1@32 \(\times\) 16 & \(5\times\) 5 & 4 \\
2 & Convolution & 16@32 \(\times\) 16 & \(5\times\) 5 & 416 \\
3 & Batch Normalization & 16@32 \(\times\) 16 & \(5\times\) 5 & 64 \\
4 & Convolution & 32@32 \(\times\) 16 & \(5\times\) 5 & 12,832 \\
5 & Batch Normalization & 32@32 \(\times\) 16 & \(5\times\) 5 & 128 \\
6 & Convolution & 64@32 \(\times\) 16 & \(5\times\) 5 & 51,264 \\
7 & Maxpooling & 64@10 \(\times\) 5 & \(3\times\) 3 & - \\
8 & Flatten & 3200@1 \(\times\) 1 & - & - \\
9 & Fully Connected & 5@1 \(\times\) 1 & - & 16,005 \\
10 & Output & 5@1 \(\times\) 1 & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summary of the trained CNN model. The performance of each CNN model was investigated by classifying 15,641 test data into five classes. The average confusion matrix over the 50 CNN models is presented in Figure 7. In addition, that of the SVM models can be found in Figure 8. Comparing both matrices reveal that the diagonal components of the CNN model were greater than those of the SVM model, which implies that the CNN models performed better. Five performance metrics, including precision, recall, accuracy, F1 score, and Matthew's correlation coefficient (MCC), were employed to obtain clear insight into the performance comparison. The precision, recall, and accuracy are, respectively, defined as follows [35,36]. Figure 5: Average training accuracy and validation accuracy of the 50 CNN models. Figure 6: Average training loss and validation loss of the 50 CNN models. Figure 7: Average confusion matrix of the 50 CNN models. (**a**) In number and (**b**) in percentage. \[\text{Precision}=\frac{TP}{TP+FP^{\prime}}, \tag{3}\]  
\[\text{Recall}=\frac{TP}{TP+FN}, \tag{4}\]  
and  
\[\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}, \tag{5}\]  
where \(TP\), \(TN\), \(FP\), and \(FN\) stand for true positive, true negative, false positive, and false negative, respectively. As may be observed from Equations (3) and (4), precision and recall indicators evaluate the true predictions. In addition, because predicting false data as false is correct, the accuracy metric evaluates both TP and TN. The data used in this study had an imbalanced ratio by class. In the case of model performance evaluation, simply using accuracy as an evaluation metric can lead to a significant accuracy bias when the data are imbalanced. For this reason, the F1 score and MCC, which can complement this, are employed as additional metrics. The F1 score is the harmonic average of the precision and recall, which is defined as  
\[\text{F1 score}=2\times\frac{\text{Precision}\times\text{Recall}}{\text{ Precision}+\text{Recall}}=\frac{TP}{TP+\frac{1}{2}(FP+FN)} \tag{6}\]  
This can accurately evaluate the performance of an unbiased model even when the data ratio for each class is imbalanced [37].