# Using Aspect-Level Sentiments for Calling App Recommendation with Hybrid Deep-Learning Models
## 4 Results
### Experimental Results for Deep-Learning Models
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**SVC**} & \multicolumn{3}{c}{**DT**} \\ \cline{2-10} Features & Class & Precision & Recall & F1 Score & Features & Class & Precision & Recall & F1 Score \\ \hline \multirow{3}{*}{BoW} & 0 & 0.85 & 0.88 & 0.86 & & 0 & 0.75 & 0.73 & 0.74 \\ \cline{2-10}  & 1 & 0.84 & 0.80 & 0.82 & BoW & 1 & 0.78 & 0.83 & 0.80 \\ \cline{2-10}  & 2 & 0.94 & 0.93 & 0.94 & & 2 & 0.88 & 0.88 & 0.88 \\ \hline \multirow{3}{*}{TF-IDF} & 0 & 0.86 & 0.89 & 0.87 & & 0 & 0.73 & 0.71 & 0.72 \\ \cline{2-10}  & 1 & 0.88 & 0.80 & 0.84 & TF-IDF & 1 & 0.75 & 0.81 & 0.78 \\ \cline{2-10}  & 2 & 0.94 & 0.94 & 0.94 & & 2 & 0.87 & 0.87 & 0.87 \\ \hline \multirow{3}{*}{Hashing} & 0 & 0.88 & 0.89 & 0.88 & & 0 & 0.74 & 0.72 & 0.73 \\ \cline{2-10}  & 1 & 0.89 & 0.83 & 0.86 & & 1 & 0.76 & 0.79 & 0.77 \\ \cline{2-10}  & 2 & 0.94 & 0.95 & 0.95 & & 2 & 0.87 & 0.87 & 0.87 \\ \hline \multicolumn{10}{c}{**RF**} & \multicolumn{3}{c}{**XGBoost**} \\ \cline{2-10} Feature & Class & Precision & Recall & F1 Score & Feature & Class & Precision & Recall & F1 Score \\ \hline \multirow{3}{*}{BoW} & 0 & 0.85 & 0.72 & 0.78 & & 0 & 0.82 & 0.80 & 0.81 \\ \cline{2-10}  & 1 & 0.80 & 0.70 & 0.75 & BoW & 1 & 0.83 & 0.85 & 0.84 \\ \cline{2-10}  & 2 & 0.86 & 0.94 & 0.90 & & 2 & 0.90 & 0.91 & 0.91 \\ \hline \multirow{3}{*}{TF-IDF} & 0 & 0.84 & 0.72 & 0.78 & & 0 & 0.91 & 0.78 & 0.80 \\ \cline{2-10}  & 1 & 0.81 & 0.61 & 0.70 & TF-IDF & 1 & 0.79 & 0.69 & 0.73 \\ \cline{2-10}  & 2 & 0.84 & 0.94 & 0.89 & & 2 & 0.87 & 0.91 & 0.89 \\ \hline \multirow{3}{*}{Hashing} & 0 & 0.89 & 0.41 & 0.56 & & 0 & 0.89 & 0.78 & 0.80 \\ \cline{2-10}  & 1 & 0.93 & 0.14 & 0.25 & Hashing & 1 & 0.79 & 0.69 & 0.73 \\ \cline{2-10}  & 2 & 0.68 & 0.98 & 0.80 & & 2 & 0.87 & 0.85 & 0.87 \\ \hline \multicolumn{10}{c}{**LR**} & \multicolumn{3}{c}{**KNN**} \\ \hline Feature & Class & Precision & Recall & F1 Score & Feature & Class & Precision & Recall & F1 Score \\ \hline \multirow{3}{*}{BoW} & 0 & 0.88 & 0.85 & 0.86 & & 0 & 0.63 & 0.31 & 0.42 \\ \cline{2-10}  & 1 & 0.81 & 0.83 & 0.82 & BoW & 1 & 0.29 & 0.74 & 0.42 \\ \cline{2-10}  & 2 & 0.93 & 0.94 & 0.94 & & 2 & 0.81 & 0.73 & 0.77 \\ \hline \multirow{3}{*}{TF-IDF} & 0 & 0.88 & 0.86 & 0.87 & & 0 & 0.58 & 0.45 & 0.51 \\ \cline{2-10}  & 1 & 0.85 & 0.79 & 0.82 & TF-IDF & 1 & 0.26 & 0.62 & 0.36 \\ \cline{2-10}  & 2 & 0.93 & 0.95 & 0.94 & & 2 & 0.82 & 0.65 & 0.73 \\ \hline \multirow{3}{*}{Hashing} & 0 & 0.87 & 0.85 & 0.86 & & 0 & 0.60 & 0.45 & 0.52 \\ \cline{2-10}  & 1 & 0.84 & 0.82 & 0.83 & Hashing & 1 & 0.30 & 0.63 & 0.40 \\ \cline{2-10}  & 2 & 0.93 & 0.95 & 0.94 & & 2 & 0.83 & 0.72 & 0.72 \\ \hline \hline \end{tabular}
\end{table} 
Table 15: The per class scores for the machine-learning models. make more accurate predictions. The architecture of the proposed CNN-GRU is illustrated in Figure 5.  
We used a CNN model with each recurrent architecture. CNN was used at the initial level for feature extraction, and then other models were used for the prediction. It can be observed that the performance of models improved after combining with the CNN. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Accuracy** & **Precision** & **Recall** & **F1 Score** & **G-Mean** \\ \hline CNN & 0.86 & 0.83 & 0.80 & 0.81 & 0.81 \\ \hline RNN & 0.87 & 0.86 & 0.84 & 0.85 & 0.85 \\ \hline CNN-RNN & 0.92 & 0.90 & 0.89 & 0.89 & 0.89 \\ \hline LSTM & 0.92 & 0.91 & 0.89 & 0.90 & 0.90 \\ \hline CNN-LSTM & 0.93 & 0.91 & 0.91 & 0.91 & 0.91 \\ \hline GRU & 0.93 & 0.92 & 0.90 & 0.91 & 0.91 \\ \hline CNN-GRU & 0.94 & 0.93 & 0.91 & 0.92 & 0.92 \\ \hline \hline \end{tabular}
\end{table} 
Table 16: Deep-learning model results. Figure 4: Architecture of deep-learning models. Figure 5: Illustration of the CNN-GRU architecture. The highest accuracy was achieved by the CNN-GRU, which also outperformed the CNN-LSTM ensemble. The CNN-GRU obtained an accuracy score of 0.94, which was the highest of all the machine and deep-learning models used in this study. Its 0.92 F1 score was also the highest among all models.