## 5 Conclusions and Future Work  
In this paper, we discussed a method to detect irrelevant user reviews efficiently by combining various oversampling and machine learning algorithms. About 35,000 user reviews collected from 25 restaurants and 33 tourist attractions in Ulsan were used for learning, where the ratio of irrelevant reviews in the two kinds of data sets was about 53.7% and 71.6%, respectively. To deal with the data skewness in the collected reviews, data imbalance processing algorithms such as RandomUnderSampler, RandomOverSampler, SMOTE, Borderline-SMOTE, and ADASYN were adopted. To build a model for the detection of irrelevant reviews, RNN, LSTM, GRU, and BERT were used, whose algorithms are known to provide good performance in text processing. By combining these algorithms, a total of 24 detection models were implemented and compared to achieve the best performance. The performance of the models was examined through experiments with the two kinds of review data sets. The results can be summarized as follows:  
* When oversampling algorithms were applied to the nearly balanced data, such as the review data sets for the restaurants where the ratio of irrelevant reviews was about 53.7%, the precision of the LSTM and GRU models significantly degraded. In this case, the F1 scores of the models were heavily influenced by precision. * When oversampling algorithms were applied to the skewed data, such as the review data sets for the tourist attractions where the ratio of irrelevant reviews was about 71.6%, the recall of the RNN, LSTM, and GRU models significantly degraded. In this case, the F1 scores of the models were heavily influenced by recall. * None of the oversampling algorithms improved the performance of the models. The models using the original data provided the best F1 scores for both kinds of review data sets. * The average F1 score of the models for the nearly balanced data was 0.953, which was higher than the average F1 score of the models for the skewed data, whose score was 0.937. * The BERT model provided the best performance, with an average F1 score of 0.965. In addition, its performance was not affected by oversampling algorithms. As summarized, the BERT model can be used properly for the detection of irrelevant reviews. Its superior performance can be explained by the fact that BERT uses pre-trained data for learning, in addition to a given input data. As discussed in Section 3, when using the basic multilingual model of BERT, the number of parameters used for learning in the embedding layer reaches approximately 91,000,000, which is much larger than the 100,000 parameters when using LSTM. This also implies that other pre-trained language models, such as ROBERTA [61], XLNET [62], ALBERT [63], T5 [64], and ElecTra [65], can be used for the efficient detection of irrelevant reviews. Figure 9: Comparison of the models’ balanced accuracy in terms of data imbalance processing algorithms, obtained from (**a**) 25 review data sets for the restaurants and (**b**) 33 review data sets for the tourist attractions. We also observed that conventional oversampling algorithms, such as SMOTE, Borderline-SMOTE, and ADASYN cannot improve the models' performance when processing text reviews. This implies that new records, augmented by oversampling algorithms, cannot properly reflect their original contexts and act as noise when the classification is performed. Interestingly, the performance of the BERT model did not degrade after oversampling. Regarding this, further analysis is required to understand the relationship between how each word is mapped and oversampled when using the algorithms. The discussion in this paper was limited to the detection of irrelevant reviews, especially when the collected reviews have opinions about popular places, i.e., when a search term to collect reviews is the name of a popular tourist attraction, restaurant, or cafe. In this case, irrelevant reviews commonly have opinions about other places near a given target place.