# A Sentiment Analysis Model Based on User Experiences of Dubrovnik on the Tripadvisor Platform
## 3 Materials and Methods
### Model Evaluation  
Model evaluation is a process in which selected metrics are used to analyze the effectiveness of an ML model. The choice of metrics depends on the data, the model, and the specific use case. In this study, the following classification metrics were used: the confusion matrix, accuracy, precision, recall, and \(F1\) measure (\(F1\) score) [81; 82; 83]. The confusion matrix is a table that contains key performance data of a machine learning classification model. It is generally used for binary classification problems, where classes are usually denoted as positive and negative, but it can be also used for multi-class classification problems. A binary classification confusion matrix is a \(2\times 2\) table of two rows that represent actual classes and two columns that represent predicted classes. Then, the first row refers to the actual positive class, and it contains two values: true positives (\(TP\))--instances of the positive class correctly classified by the model as positive--and false negatives (\(FN\))--instances of the positive class incorrectly classified as negative. Furthermore, the second row refers to the actual negative class, and it has two values: false \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter \({}^{*}\)** & **Value** \\ \hline Number of epochs & 4 \\ Batch size & 32 \\ Learning rate & 0.00005 \\ Maximum token length & 256 \\ \hline \hline \end{tabular} \({}^{*}\)robertaSentimentFT_Tripadvisor model.  
\end{table} 
Table 4: Model hyperparameters. Figure 6: Distribution of reviews’ token length. The data is displayed in a density histogram with an additional density curve. The bar area indicates the probability of the number of tokens in the reviews, and the density curve shows the distribution. positives (_FP_)--instances of the negative class incorrectly classified as positive--and true negatives (_TN_)--instances of the negative class correctly classified as negative. The sum of the first row of the confusion matrix equals the number of all actual instances of the positive class, and the sum of the second row corresponds to the number of all actual instances of the negative class in the dataset. Also, the sum of the first column is equal to the predicted number of positive instances, and the sum of the second column is equal to the predicted number of negative instances. The main diagonal of the confusion matrix contains _TP_ and _TN_ values, i.e., the numbers of actual instances correctly classified by the model. The accuracy of the model is calculated as a ratio of correct predictions and all predictions:  
\[Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \tag{1}\]  
There are also other metrics which are more useful when the dataset is unbalanced, as it is in our case, such as precision, recall, and the \(F\)1 score, as well as the false-positive rate and false-negative rate. The precision of a model measures how many true-positive predictions are among all predicted instances of a positive class, and it is calculated using _TP_ and _FP_ values, as follows:  
\[Precision=\frac{TP}{TP+FP} \tag{2}\]  
The recall of the model measures how many true-positive predictions are among all actual instances of a positive class, and it is calculated using _TP_ and _FN_ values, as follows:  
\[Recall=\frac{TP}{TP+FN} \tag{3}\]  
The \(F\)1 score, or the \(F\)1 measure, is a harmonic mean between the precision and recall values, and it is calculated as follows:  
\[F1=2\cdot\frac{1}{\frac{1}{Precision}+\frac{1}{Recall}}=\frac{2 TP}{2 TP+FP+FN} \tag{4}\]  
The false-positive rate (_FPR_) measures how many false positives are among all instances of the actual negative class:  
\[FPR=\frac{FP}{FP+TN} \tag{5}\]  
Lastly, the false-negative rate (_FNR_) measures how many false negatives are among all instances of the actual positive class:  
\[FNR=\frac{FN}{TP+FN} \tag{6}\]  
If the classes in the binary classification problem are named differently, e.g., class X and class Y, then we can still find all of the mentioned values with respect to each of the classes.