# Using Aspect-Level Sentiments for Calling App Recommendation with Hybrid Deep-Learning Models
## 4 Results
### Results of Machine-Learning Models  
The machine-learning model results are evaluated in terms of the accuracy, precision, recall, F1 score, and G-mean using BoW, TF-IDF, and Hashing features. Table 12 contains the results of the machine-learning models using the BoW features. The performance of models was good as SVM achieved a significant accuracy of 0.91 and a 0.88 F1 score. This is followed by LR, which obtained a 0.90 accuracy score and a 0.87 F1 score. The good performance of SVM and LR can be attributed to the large feature set. SVM and LR require a large feature set for a good fit, and the used dataset generates a large feature set using the BoW technique, which helps LR to achieve significant results.  
Table 13 contains the results of machine-learning models using the TF-IDF features. The performance of the models is approximately similar to that of BoW, as SVM still has the highest accuracy score, followed by LR. Consistent better performance by the SVM and LR show that the linear models performed well on the dataset used in this study. KNN showed poor performance with BoW and TF-IDF features because it uses matching criteria between training and new data to make a prediction, and its performance was poor when the feature set was large.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Accuracy** & **Precision** & **Recall** & **F1 Score** & **G-Mean** \\ \hline SVM & 0.91 & 0.89 & 0.88 & 0.88 & 0.88 \\ \hline DT & 0.83 & 0.81 & 0.81 & 0.81 & 0.81 \\ \hline RF & 0.85 & 0.83 & 0.79 & 0.81 & 0.81 \\ \hline LR & 0.90 & 0.87 & 0.87 & 0.87 & 0.87 \\ \hline KNN & 0.61 & 0.58 & 0.60 & 0.53 & 0.59 \\ XGBoost & 0.87 & 0.85 & 0.85 & 0.85 & 0.85 \\ \hline \hline \end{tabular}
\end{table}
Table 12: The results of machine-learning models using BoW features.  
The results using hashing features are given in Table 14, which indicates the variation in model accuracy and other metrics. Although the accuracy of the leading performers SVM and LR remains the same, a change in other models is observed. For example, the accuracy of RF reduced from 0.84 with TF-IDF to 0.71 when used with hashing features. On the contrary, KNN showed a marginal improvement of 0.4 in accuracy when hashing features were used. The performance of SVM was superior when used with hashing in comparison with BoW and TF-IDF because the hashing feature gives a more sparse feature set for learning models, which can be significant for linear models, such as SVM and LR. SVM achieved the highest F1 score of 0.90 using the hashing features.  
Figure 3 shows the comparison between machine-learning models using BoW, TF-IDF, and hashing features. RF showed poor performance with hashing features, while the linear models improved their F1 score with hashing features. Table 15 shows the per class score for each machine-learning model.  
\begin{table}
\begin{tabular}{c c c c c c} \hline
**Model** & **Accuracy** & **Precision** & **Recall** & **F1 Score** & **G-Mean** \\ \hline SVM & 0.91 & 0.90 & 0.89 & 0.89 & 0.89 \\ \hline DT & 0.82 & 0.78 & 0.80 & 0.79 & 0.79 \\ \hline RF & 0.84 & 0.83 & 0.76 & 0.79 & 0.79 \\ \hline LR & 0.90 & 0.88 & 0.87 & 0.87 & 0.87 \\ \hline KNN & 0.59 & 0.55 & 0.58 & 0.53 & 0.56 \\ \hline XGBoost & 0.85 & 0.83 & 0.81 & 0.82 & 0.82 \\ \hline \end{tabular}
\end{table}
Table 13: The results of models using TF-IDF features.  
Figure 3: Comparison between model performances using BoW, TF-IDF, and hashing features.