# Regional Traffic Event Detection Using Data Crowdsourcing
## 4 Performance Evaluation
As mentioned earlier, the proposed technique uses binary classification to determine data influencing traffic conditions. It then uses multi-class classification to classify the selected data according to the event type. This section demonstrates the importance of removing irrelevant data by comparing and evaluating the results of not using the binary classifier. During multi-class classification without binary classification, irrelevant data are classified as \"others\". In this performance evaluation, cases that perform multi-class classification after binary classification are labeled AB, whereas cases that perform multi-class classification without binary classification are labeled WB. Figure 11 shows the AUC values for AB and WB. The AUC results for the AB and WB models show no differences between AB and WB for the naive Bayes, random forest, and SVC models, which had AUC values of 0.95, 0.97, and 0.98, respectively. In the linear SVC model, the AB result was 0.98, and the WB result was 0.97, indicating that the AB value was higher by 0.1. In the BiLSTM model, the AB model was 0.96, and the WB model was 0.94, corresponding to a 2% difference. In TextCNN, the AB result was 0.96, and the WB result was 0.94, indicating that the AB value was 2% higher in both models. Figure 11: AUC with and without binary classifiers. Figure 10: Multi-class classifier F-measure averages. Figure 12 shows the precision, recall, and F-measures of the AB and WB models. The precision results show that AB improved the performance of all evaluated models. For the linear SVC model, which we selected as the most suitable classifier in earlier parts of the study, the value for WB was 0.79. By contrast, the value for AB was 0.84, which was 5% higher. In terms of recall, WB generally showed higher performance. However, in the case of BiLSTM, AB's performance was 2% higher, whereas TextCNN exhibited the same results for WB and AB. Finally, concerning the harmonic means for precision and recall, random forest WB had a value of 0.73, whereas AB had 0.70, indicating WB's superior performance. By comparison, AB outperformed the rest of the models (excluding random forest). In the case of the linear SVC model, the value for WB was 0.81, whereas that for AB was 0.83, confirming AB's superior performance. Figure 13 shows the F-measures of the six models in the AB and WB cases compared to F-measures without the \"others\" class. In the naive Bayes model, the F-measure values were the same for AB and WB, whereas, in the random forest model, WB had better results than AB. However, regarding the F-measure values of the four models, i.e., excluding the two aforementioned models, those for AB were higher than those for WB. In the case of the linear SVC model, which was selected as the most suitable classifier based on earlier performance evaluation results, the F-measure value for AB was 2% higher than for WB. In the case of WB, data that did not influence traffic conditions were classified as \"others\". Therefore, when we measured the F-measure values without the \"others\" class, the F-measure values for AB increased in all six models. However, we found that the F-measure values increased in the Naive Bayes and BiLSTM models for WB but remained the same or decreased in other models. Thus, we confirmed that accuracy is improved using a binary classifier to determine data relevant to traffic conditions. Figure 12: Precision, recall, and F-measures with and without binary classifiers. Our evaluations verified the performance of the proposed technique, which selects keywords from text data that infer regions and converts them to administrative district names. We performed the analysis based on linear SVC results, which exhibited the best performance after event classification. The total number of data points was 8100. Table 5 shows extraction accuracy percentages based on a random selection of 100 data points from the overall set of 8100 data points. We repeated this process five times and calculated the average accuracy values.