# A Method of Supplementing Reviews to Less-Known Tourist Spots Using Geotagged Tweets
## 5 Classification Experiment
### On-Spot Tweets Detection
#### 5.1.3 Baseline vs. BERT  
In this section, we compare the efficacy of the SVM classifier to a deep learning approach. To do this, we fine-tuned a pre-trained BERT neural language model and used it for the tweet classification task. Next, we compared the performance of the BERT model with that obtained from SVM. Lastly, we evaluate and discuss the results obtained in this experiment.  
In the pre-processing stage, tweets are lowercased. Non-ASCII letters, URLs, @RT: [NAME], are removed. For BERT, texts with a length of less than 4 are discarded. No lemmatization is performed and no punctuation mark is removed since pre-trained embeddings are always used. No stop-word is removed to retain better grasp of the fluency of language.  
We demonstrate the efficacy of the deep bidirectionality of BERT by using the same training dataset used as in the previous experiment, with 1273 geotagged tweets.  
The original BERT-Base uncased model comprises two models, one with 12 transformer layers, 12 self-attention heads, and the other one with 24 encoders, and 16 bidirectional self-attention heads. Both models pre-trained from unlabeled data extracted from the BookCorpus and English Wikipedia words. In this experiment specifically, we used the distilbert-base-uncasedmodel ([[https://huggingface.co/distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased)]([https://huggingface.co/distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased)), accessed on 15 January 2022) version of BERT. Compared to other version of BERT models, a DistilBERT is significantly smaller, consistently faster, while retaining high performance when compared to original BERT model [28]. Training neural language models from scratch is typically time consuming. Even fine-tuning the pre-trained model with a task-specific dataset may take several hours to finish one epoch, as shown by [PERSON] et al. [29]. Thus, in reducing computational time in this experiment, we deploy a ktrain library ([[https://github.com/amaiya/ktrain](https://github.com/amaiya/ktrain)]([https://github.com/amaiya/ktrain](https://github.com/amaiya/ktrain)), accessed on 15 January 2022) which is a lightweight wrapper for tf.keras in TensorFlow 2. It is designed to make the deep learning process more accessible and easier to apply, as described by [PERSON] et al. [30].  
We, therefore, train our model in consecutive 2-epochs. Distilbert-base-uncased model is trained using the same corpus as the original BERT model which includes a concatenation of English Wikipedia and a book corpus in a self-supervised fashion using the BERT base model as a teacher.  
BERT performed well with the same amount of data for the on-spot review tweet identification task when compared to SVM. Table 6 demonstrates the obtained results. The reason for such a high score was most probably due to BERT working best with the context orientation of the sentence, hence simplifying the classification task. This observation suggests that a deep learning approach can show a significant improvement when dealing with limited training data. Pre-trained language models such as BERT have proven to be highly effective for NLP tasks. However, the high demand for computing resources in training such models from scratch hinders their application in practice.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **P** & **R** & **F1** \\ \hline SVM & 0.875 & 0.872 & 0.858 \\ BERT & 0.927 & 0.946 & 0.936 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Classification results obtained from BERT model with comparison to SVM.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**On-Spot** & **Not** & **Ref** \\ \hline
938 & 36 & on-spot \\
120 & 179 & not \\ \hline \hline \end{tabular}
\end{table}
Table 5: Confusion Matrix from SVM classifier.