## 4 A Pilot Study
### Performance of the Proposed Sentiment Analysis System  
Accuracy (A), precision, recall, and F-measure are commonly adopted metrics for evaluating the performances of sentiment analysis [7; 15; 21]. The performance of the proposed sentiment analysis system was evaluated using the F-measure, and the results are shown in Tables 4-6. It is worth mentioning that the proposed system was built based on the VADER and target and emotion dictionaries. The VADER [24] without any further training considering its original parameters fine-tuned during its construction was used as the sentiment extraction in our proposed system. In addition, as there were no parameters in our proposed target and emotion detection steps, there was no need for training the system during the evaluation process. Therefore, the entire dataset was considered in our experiments for testing the proposed method [54]. Tables 4 and 5 present the results of emotion and target detection combined with sentiment analysis resulting in target-oriented sentiment analysis. The results obtained from our proposed system are shown in Table 4, indicating that the proposed system performed better on tweets with positive emotions (e.g., Happy and Love) compared to negative ones. In addition, it is also evident that the proposed system performed better on detecting emotions compared to the National Research Council Canada (NRC) emotion lexicon-based method [54]. NRC emotion lexicon-based method [54] provided better results on tweets without any emotions (None). This is because the NRC method [54] normally provided probability-based results for all emotions and when the probability value for all emotions was zero, this indicated that no emotion was detected for the given tweet. Likewise, Table 5 shows that accommodation, attraction, and weather have higher F-measures, indicating better correlations with manual annotation results. Overall, the proposed system can provide target-oriented sentiment analysis, offering more insight into different aspects of tourism and visitor behaviour. To evaluate the performance of human (manual annotation) vs. machine (our proposed system), we computed the results based on different metrics, including correlation, accuracy (A), precision, recall, and F-measure, as shown in Table 6. From the results shown in Table 6, it is evident that the overall accuracy of the sentiment analysis results at the post level (67.6%) is slightly better than that at the sentence level, with 66% correct prediction. In general, the system performed better on positive and neutral posts than on negative posts. \begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline \multirow{3}{*}{Metric} & \multicolumn{3}{c}{Neutral} & \multicolumn{3}{c}{Positive} & \multicolumn{3}{c}{Negative} & \multirow{3}{*}{Overall} & \multirow{3}{*}{\begin{tabular}{c} Correlation \\ (Proposed \\ System vs. \\ Human) \\ \end{tabular} } \\ \cline{2-2} \cline{6-11}  & \multirow{2}{*}{Precision} & \multirow{2}{*}{Recall} & F- & \multirow{2}{*}{Measure} & \multirow{2}{*}{Precision} & \multirow{2}{*}{Recall} & F- & \multirow{2}{*}{\begin{tabular}{c} Precision \\ (\%) \\ \end{tabular} } & \multirow{2}{*}{Recall} & F- & \multirow{2}{*}{Measure} & \multirow{2}{*}{\begin{tabular}{c} Accuracy \\ (\%) \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} Accuracy \\ (\%) \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} Accuracy \\ (\%) \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} Accuracy \\ (\%) \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} Accuracy \\ (\%) \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} Accuracy \\ (\%) \\ \end{tabular} } & \multirow{2}{*}{\
\begin{tabular}{c} Accuracy \\ (\%) \\ \end{tabular} } & \multirow{2}{*}{
\begin{tabular}{c} Accuracy \\ (\%) \\ \end{tabular} } \\ \hline Sentence & \multirow{2}{*}{53.8} & 89.5 & 67.2 & 90.0 & 55.3 & 68.5 & 42.9 & 41.9 & 42.4 & 66.0 & 0.631 \\ Post & \multirow{2}{*}{57.0} & 89.3 & 69.6 & 89.0 & 57.7 & 70.0 & 32.3 & 35.7 & 33.9 & 67.6 & 0.625 \\ \hline \hline \end{tabular}
\end{table}