## 5 Result and Discussion
### Experimental Results
In terms of the F1-score, RepSTDoc_ConvNet yields the highest performance with 0.609. We measured the classification accuracy of human workers on 1400 learning data to verify the challenge of the representative spatio-temporal document classification prob \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Machine Learning** & **Precision** & **Recall** & **F1 score** & **Accuracy** \\ \hline Gaussian Naive Bayes & 0.562 & 0.224 & 0.320 & 0.751 \\ Linear SVM & 0.626 & 0.421 & 0.503 & 0.783 \\ Random Forest & **0.729** & 0.191 & 0.303 & 0.770 \\ ConvNet & 0.525 & 0.591 & 0.556 & 0.762 \\ DocClass\_ConvNet & 0.511 & 0.453 & 0.480 & 0.744 \\ DocClass\_ConvNet\_Mod & 0.614 & 0.496 & 0.548 & **0.794** \\ RepSTDoc\_ConvNet & 0.552 & **0.673** & **0.612** & 0.785 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of evaluation based on the precision, recall, F1 score, and accuracy. Figure 3: Comparison of confusion matrixes (**a**) Linear SVM, (**b**) Random Forest, and (**c**) RepSTDoc_ConvNet. \({}^{\prime}0^{\prime}\) means the nonrepresentative spatio-temporal document and \({}^{\prime}1^{\prime}\) means the representative spatio-temporal document. lem. The 1400 learning data consists of 359 representative spatio-temporal documents and 1041 non-representative spatio-temporal documents. Four workers who participated in building learning data classified representative spatio-temporal documents for 1400 learning data. For each learning data, the number of workers who judged actual representative spatio-temporal documents as representative spatio-temporal documents (True Positive: TP) and the number of workers who judged non-representative spatio-temporal documents (False Negative: FN) were calculated. For one actual representative spatio-temporal document, the ratio was calculated by dividing the number of all four people judged as TP, the number of three or more judged as TP, the number of two or more judged as TP, and the number of one or more judged as TP in Table 5. For each of the 359 representative spatiotemporal documents, the number of documents judged as TP by all 4 people was 189 (52.64%), the number of documents judged as TP by 3 or more people 251 (69.92%), and the number of documents judged as TP by 2 or more people was 310 (89.35%), the number of documents judged as TP by 1 or more people was 332 (92.48%). For one actual nonrepresentative spatio-temporal document, the ratio was also calculated by dividing the number of all 4 people judged as FN, the number of 3 or more people judged as FN, the number of 2 or more people judged as FN, and the number of 1 or more people judged as FN in Table 6. For each of the 1041 nonrepresentative spatio-temporal documents, the number of documents judged as FN by all 4 people was 5 (0.48%), the number of documents judged as FN by 3 or more people was 24 (2.31%), and the number of documents judge as FN by 2 or more people (6.34%), and the number of documents judged as FN by more than one person was 135 (12.97%). First of all, we describe the challenge of the representative spatio-temporal document classification problem through the ratio of documents in which at least three people, more than half of the judges, judged the actual representative spatio-temporal document as the representative spatio-temporal document. About 70% of the three or more people judged the actual representative spatio-temporal document as TP, and the ratio of all four people who judged it as TP was only about 53%, confirming that it is difficult for humans to classify representative spatio-temporal documents from large documents.