## 4 Experimental Results
### Recall  
Table 8 compares the mean and standard deviation of the models' recall, obtained from 25 review data sets for the restaurants in Ulsan. Among data imbalance processing algorithms, the models using the original data presented the best performance, with an average recall of 0.968, while the performance of the models with ADASYN was the worst, with an average of 0.958. In terms of machine learning algorithms, the BERT model provided the best performance, with an average of 0.969, while the LSTM model performed the worst, with an average of 0.956. Among the individual models, the LSTM model with the original data provided the highest recall, at 0.975. Compared to the precision shown in Table 6, the performance gap between the algorithms was not so significant in this case. Table 9 compares the mean and standard deviation of the models' recall, obtained from 33 review data sets for the tourist attractions. In terms of data imbalance processing algorithms, the models using the original data and RandomOverSampler presented the best performance, with an average of 0.948, while the performance of the models with ADASYN was the worst, with an average of 0.883. Among the machine learning algorithms, the BERT model provided the best performance, with an average of 0.958, while the RNN model performed the worst, with an average of 0.873. Among the individual models, the BERT model using SMOTE provided the highest recall, at 0.961. Figure 5 compares the recall of the models in terms of data imbalance processing algorithms. The figures on the left and right show the recall obtained from the restaurant and tourist attraction data sets, respectively. Note that the patterns of the two figures contrast with those of Figure 2. In the left-hand figure, there was no significant difference in the models' performance. On the other hand, in the right-hand figure, the models' performance was significantly affected by the oversampling algorithms. The RNN model was most affected by oversampling, with its recall dropping from 0.931 to a maximum of 0.825. The recall of the LSTM and GRU models also dropped from 0.951 to 0.887 and from 0.952 to 0.860, respectively, after oversampling. The performance of the BERT model was not affected by the oversampling algorithms. Interestingly, no performance degradation was observed when RandomOverSampler was applied. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **RNN** & **LSTM** & **GRU** & **BERT** & **Average** \\ \hline Original data & 0.953 \(\pm\) 0.033 & 0.975 \(\pm\) 0.019 & 0.974 \(\pm\) 0.019 & 0.969 \(\pm\) 0.023 & 0.968 \(\pm\) 0.023 \\ RandomOverSampler & 0.963 \(\pm\) 0.021 & 0.953 \(\pm\) 0.038 & 0.967 \(\pm\) 0.031 & 0.968 \(\pm\) 0.028 & 0.963 \(\pm\) 0.030 \\ SMOTE & 0.963 \(\pm\) 0.025 & 0.949 \(\pm\) 0.033 & 0.966 \(\pm\) 0.029 & 0.973 \(\pm\) 0.022 & 0.963 \(\pm\) 0.027 \\ Borderline-SMOTE & 0.960 \(\pm\) 0.025 & 0.952 \(\pm\) 0.037 & 0.969 \(\pm\) 0.029 & 0.972 \(\pm\) 0.023 & 0.963 \(\pm\) 0.029 \\ ADASYN & 0.953 \(\pm\) 0.029 & 0.949 \(\pm\) 0.043 & 0.965 \(\pm\) 0.032 & 0.964 \(\pm\) 0.030 & 0.958 \(\pm\) 0.034 \\ Average & 0.959 \(\pm\) 0.027 & 0.956 \(\pm\) 0.034 & 0.968 \(\pm\) 0.028 & 0.969 \(\pm\) 0.025 & 0.963 \(\pm\) 0.028 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Mean and standard deviation of the models’ recall obtained from 25 review data sets for the restaurants in Ulsan, where the ratio of irrelevant reviews is about 53.7%.