## 4 Experimental Results
### Recall
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **RNN** & **LSTM** & **GRU** & **BERT** & **Average** \\ \hline Original data & 0.931 \(\pm\) 0.013 & 0.951 \(\pm\) 0.015 & 0.952 \(\pm\) 0.012 & 0.956 \(\pm\) 0.013 & 0.948 \(\pm\) 0.013 \\ RandomOverSampler & 0.941 \(\pm\) 0.026 & 0.949 \(\pm\) 0.028 & 0.948 \(\pm\) 0.022 & 0.955 \(\pm\) 0.015 & 0.948 \(\pm\) 0.023 \\ SMOTE & 0.835 \(\pm\) 0.058 & 0.894 \(\pm\) 0.061 & 0.873 \(\pm\) 0.055 & 0.961 \(\pm\) 0.010 & 0.891 \(\pm\) 0.046 \\ Borderline-SMOTE & 0.831 \(\pm\) 0.049 & 0.890 \(\pm\) 0.052 & 0.864 \(\pm\) 0.048 & 0.957 \(\pm\) 0.013 & 0.885 \(\pm\) 0.040 \\ ADASYN & 0.825 \(\pm\) 0.047 & 0.887 \(\pm\) 0.048 & 0.860 \(\pm\) 0.040 & 0.960 \(\pm\) 0.011 & 0.883 \(\pm\) 0.037 \\ Average & 0.873 \(\pm\) 0.039 & 0.914 \(\pm\) 0.041 & 0.900 \(\pm\) 0.035 & 0.958 \(\pm\) 0.012 & 0.911 \(\pm\) 0.032 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Mean and standard deviation of the models’ recall obtained from 33 review data sets for the tourist attractions in Ulsan, where the ratio of irrelevant reviews is about 71.6%. To figure out the reason for the performance degradation, we examined the changes in TP and FP of the models after oversampling was applied. Figure 6 compares the TP and FN values of the models obtained from the restaurant reviews. As shown on the left side of Figure 5, minor performance degradation was observed only in the LSTM model. The TP values decreased, while the FP values increased after oversampling, causing the model's recall to drop from 0.975 to 0.949. Figure 7 compares the TP and FN values of the models obtained from the tourist attraction reviews. In the RNN, LSTM, and GRU models, the TP values decreased when SMOTE, Borderline-SMOTE, and ADASYN were applied. Note that the FN values also drastically increased in these cases. This led to the significant drop of the models' recall. For example, in case of the RNN model with SMOTE, the TP values decreased from 475 to 423, while the FN values increased from 35 to 87. As a result, the recall of the model significantly dropped from 0.931 to 0.835. The situation of the other models, except BERT, was similar. The performance of the BERT model was not affected by oversampling algorithms. In addition, the performance degradation was not so severe when RandomOverSampler was applied. The above results regarding the models' recall can be summarized as follows:  
* When oversampling was applied for the nearly balanced data, such as the review data sets for the restaurants, there was no significant difference in the models' recall. * When oversampling was applied for the skewed data, such as the review data sets for the tourist attractions, the recall of the RNN, LSTM, and GRU models degraded, due to the fact that the TP values of the models decreased, while their FP drastically increased. * The performance of the BERT model was not affected by oversampling algorithms. Figure 5: Comparison of the models’ recall in terms of data imbalance processing algorithms obtained from (**a**) 25 review data sets for the restaurants and (**b**) 33 review data sets for the tourist attractions. Figure 6: Comparison of TP and FN of the models obtained from 25 review data sets for the restaurants where the ratio of irrelevant reviews is about 53.7%.