# Machine Learning Based Representative Spatio-Temporal Event Documents Classification
## 5 Result and Discussion
### Experimental Results  
We compare traditional machine learning methods (GNB, SVM, and RF) and deep-learning-based models (CNN, LSTM, and BiLSTM) to test the classification performance of RSTEDocs. Deep learning basically randomly sets the initial weights of the model. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Label**} & \multirow{2}{*}{**Topic**} & **No.** & **When** & **No.** & **Who** & **Article** \\ \cline{3-6}  & & **No.** & **Where** & **No.** & **What** & \\ \hline \multirow{2}{*}{0} & \multirow{2}{*}{life} & nan & nan & nan & nan & 'Subway pilgrims' on their \\ \cline{3-6}  & & nan & nan & nan & nan & way to work \(\ldots\) \\ \hline \multirow{2}{*}{0} & \multirow{2}{*}{society} & nan & nan & nan & nan & Was Congressman [PERSON] \\ \cline{3-6}  & & nan & nan & nan & nan & Seong-ho \(\ldots\) \\ \hline \multirow{2}{*}{1} & \multirow{2}{*}{society} & 1 & 31 December last year & 1 & citizens & Politicians, please stop \\ \cline{3-6}  & & 1 & Daegu Sports Park & 1 & New Year’s Eve Bell & fighting \(\ldots\) \\ \hline \multirow{2}{*}{1} & \multirow{2}{*}{society} & 2 & Last 31 December & 1 & Daejeon citizens & Citizens of Daejeon \\ \cline{3-6}  & & 2 & Daejeon Jung-gu & 2 & candlelight vigil & celebrate the New Year \(\ldots\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Example of training data. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Split Data** & **Count** & **RSTEDoc** & **Non-RSTEDoc** & **Ratio** \\ \hline Training & 6000 & 1433 & 4567 & 31.38\% \\ Validation & 2000 & 477 & 1523 & 31.32\% \\ Test & 2000 & 477 & 1523 & 31.32\% \\ \hline Total & 10000 & 2387 & 7613 & 31.35\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Training data distribution ratio for model training. Therefore, even if the same data are input to the model, the result may be different each time. Therefore, in order to supplement the reliability of the result value due to the randomness of the weight, each experiment was performed 10 times, and the performance was measured with the mean value. Table 3 presents the performance results (precision, recall, F1 score, accuracy, Area Under the ROC Curve (AUC), and Matthews correlation coefficient (MCC) [34]) of each model. We consider accuracy, F1 score, AUC, and MCC as key metrics for evaluating model performance. The ratio of RSTEDocs and non- RSTEDocs is not equal in the training data. Only about 31% of the total data are RSTEDocs. In this case, even if the classifier is not properly trained, the value of precision increases when most of the documents are judged to be non-RSTEDocs, and the value of recall increases when most of the documents are judged to be RSTEDocs. The F1 score is 0.632 for BiLSTM, higher than CNN's 0.605. Accuracy is 0.835 for BiLSTM, higher than 0.793 for LSTM. AUC of BiLSTM is 0.886, which was higher than 0.830 of LSTM. MCC also shows the highest value of BiLSTM at 0.552. As shown in Table 3, BiLSTM has the best performance as a model of RSTEDoc classifier. Figure 2 shows the comparison of precision, recall, accuracy, F1 score, AUC and MCC performance for each model. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Machine Learning** & **Precision** & **Recall** & **F1 Score** & **Accuracy** & **AUC** & **MCC** \\ \hline Gaussian Naive Bayes & 0.587 & 0.278 & 0.348 & 0.767 & 0.794 & 0.362 \\ Linear SVM & 0.643 & 0.453 & 0.526 & 0.789 & 0.812 & 0.383 \\ Random Forest & 0.738 & 0.188 & 0.324 & 0.781 & 0.786 & 0289 \\ CNN & 0.549 & 0.673 & 0.605 & 0.790 & 0.826 & 0.451 \\ LSTM & 0.584 & 0.463 & 0.516 & 0.793 & 0.830 & 0.528 \\ BiLSTM & 0.675 & 0.593 & 0.631 & 0.835 & 0.886 & 0.552 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of performance evaluation. Figure 2: _Cont._  
AUC is a value that can be a numerical standard in performance evaluation of a binary classification model, and the closer to 1, the closer the graph is to the upper left corner, so it can be a binary classification model with excellent performance.