## 4 Experimental Results
### F1 Scores  
Table 10 shows the mean and standard deviation of the models' F1 scores, obtained from 25 review data sets for the restaurants. In terms of data imbalance processing algorithms, the models using the original data presented the best performance, with an average F1 score of 0.965, while the performance of the models with ADASYN was the worst, with an average of 0.946. In terms of machine learning algorithms, the BERT model provided the best performance, with an average of 0.966, while the LSTM model performed the worst, with an average of 0.933. Among the individual models, the LSTM and GRU models with the original data provided the highest F1 score, at 0.970. Table 11 compares the mean and standard deviation of the models' F1 scores obtained from 33 review data sets for the tourist attractions. Among the data imbalance processing algorithms, the models using the original data presented the best performance, with an average F1 score of 0.957, while the performance of models with ADASYN was the worst, with an average of 0.921. Among the machine learning algorithms, the BERT model provided the best performance, with an average of 0.963, while the RNN model performed the worst, with an average of 0.913. Among the individual models, the BERT model using SMOTE presented the highest F1 score, at 0.964. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **RNN** & **LSTM** & **GRU** & **BERT** & **Average** \\ \hline Original data & 0.954 \(\pm\) 0.027 & 0.970 \(\pm\) 0.016 & 0.970 \(\pm\) 0.017 & 0.966 \(\pm\) 0.020 & 0.965 \(\pm\) 0.020 \\ RandomOverSampler & 0.957 \(\pm\) 0.021 & 0.927 \(\pm\) 0.054 & 0.955 \(\pm\) 0.035 & 0.966 \(\pm\) 0.022 & 0.951 \(\pm\) 0.033 \\ SMOTE & 0.959 \(\pm\) 0.024 & 0.926 \(\pm\) 0.053 & 0.957 \(\pm\) 0.030 & 0.967 \(\pm\) 0.020 & 0.952 \(\pm\) 0.032 \\ Borderline-SMOTE & 0.957 \(\pm\) 0.024 & 0.928 \(\pm\) 0.046 & 0.954 \(\pm\) 0.037 & 0.968 \(\pm\) 0.020 & 0.952 \(\pm\) 0.032 \\ ADASYN & 0.954 \(\pm\) 0.027 & 0.914 \(\pm\) 0.062 & 0.953 \(\pm\) 0.035 & 0.965 \(\pm\) 0.025 & 0.946 \(\pm\) 0.037 \\ Average & 0.956 \(\pm\) 0.024 & 0.933 \(\pm\) 0.046 & 0.958 \(\pm\) 0.031 & 0.966 \(\pm\) 0.021 & 0.953 \(\pm\) 0.031 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Mean and standard deviation of the models’ F1 scores obtained from 25 review data sets for the restaurants in Ulsan, where the ratio of irrelevant reviews is about 53.7%. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **RNN** & **LSTM** & **GRU** & **BERT** & **Average** \\ \hline Original data & 0.945 \(\pm\) 0.011 & 0.961 \(\pm\) 0.013 & 0.961 \(\pm\) 0.012 & 0.962 \(\pm\) 0.011 & 0.957 \(\pm\) 0.012 \\ RandomOverSampler & 0.949 \(\pm\) 0.014 & 0.954 \(\pm\) 0.012 & 0.959 \(\pm\) 0.013 & 0.962 \(\pm\) 0.011 & 0.956 \(\pm\) 0.013 \\ SMOTE & 0.892 \(\pm\) 0.034 & 0.928 \(\pm\) 0.033 & 0.921 \(\pm\) 0.034 & 0.964 \(\pm\) 0.010 & 0.926 \(\pm\) 0.028 \\ Borderline-SMOTE & 0.891 \(\pm\) 0.029 & 0.927 \(\pm\) 0.030 & 0.916 \(\pm\) 0.031 & 0.962 \(\pm\) 0.011 & 0.924 \(\pm\) 0.025 \\ ADASYN & 0.888 \(\pm\) 0.029 & 0.920 \(\pm\) 0.023 & 0.913 \(\pm\) 0.025 & 0.963 \(\pm\) 0.010 & 0.921 \(\pm\) 0.022 \\ Average & 0.913 \(\pm\) 0.024 & 0.938 \(\pm\) 0.022 & 0.934 \(\pm\) 0.023 & 0.963 \(\pm\) 0.010 & 0.937 \(\pm\) 0.020 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Mean and standard deviation of the models’ F1 scores, obtained from 33 review data sets for the tourist attractions in Ulsan, where the ratio of irrelevant reviews is about 71.6%. Figure 7: Comparison of the TP and FN values of the models obtained from 33 review data sets for the tourist attractions where the ratio of irrelevant reviews is about 71.6%. The results of Tables 10 and 11 require further comparative analyses. For example, the performance difference between the LSTM and BERT model using the original data in Table 11 is very small, and it is not clear whether the difference is likely due to a change or to some other factor of interest.