# Text-Based Emotion Recognition in English and Polish for Therapeutic Chatbot
## 5 Results  
We evaluated four different models for _sentiment_ and _emotion_ recognition, for each of the languages, using the developed CORTEX dataset. The numbers of sentences in individual subsets (train/val/test) are displayed in Table 1. In each experiment, we measured the values of accuracy and support-weighted F1-score (see Table 2). We conducted statistical analyses using the Wilson score interval, for the confidence level set to 90%. We assessed the confidence intervals for F1-score based on the confidence intervals for precision and recall. We also generated confusion matrices, allowing for a more detailed analysis of the results, including the models' mistakes. As seen in the table above, in both experimental setups (_sentiment_ and _emotion_), BERT models outperformed the simpler methods, reaching over 90% accuracy for sentiment classification and almost 80% for emotion classification. The F1-score yielded similar values. The results for the test subset were usually slightly inferior to those for the validation subset; however, the difference was not high. The results obtained for Polish were by a few relative % worse than for English, especially in the case of BERT for _emotion_ recognition. Unsurprisingly, for the 3-class scenario, evaluation metrics reached much higher values than for the 9-class scenario. The baseline models achieved visibly worse results, with SVM being slightly better than Naive Bayes. These classifiers were based on the BoW representation, and, therefore, were not able to express semantic similarity between tokens in a proper way. We observed a slight improvement for the fastText-based classifier applied to token bigrams. In this algorithm, the embeddings were created without the context of the entire input sequence, and this might be why BERT outperformed it by a large margin. The scores for the Polish corpora were worse than for their English counterparts. This difference was significant for the more complex models. Apart from the degraded \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Sentiment**} & **\# Sentences** & \multirow{2}{*}{**Emotion**} & **\# Sentences** & \multirow{2}{*}{**Classes Used in Reference [35]**} \\  & **Train/Val/Test** & & & **Train/Val/Test** & \\ \hline positive & 5985/807/826 & happiness & 2391/310/328 & excited/joyful/grateful/content \\  & & confidence & 1705/237/222 & confident/prepared/hopeful \\  & & other positive & 1889/260/276 & proud/trusting/caring/faithful \\ \hline negative & 8314/1133/1080 & sadness & 2267/322/289 & sad/lonely/disappointed/devastated \\  & & anger & 1804/243/226 & angry/annoved/furious \\  & & fear & 1565/212/207 & afraid/terrified/apprehensive \\  & & guilt & 1576/215/199 & embarrassed/ashamed/guilty \\  & & other negative & 1102/141/159 & jealous/disgusted \\ \hline neutral & 7149/970/953 & neutral & 1787/242/238 & no emotion \\ \hline
**total** & 21,448/2910/2859 & **total** & 16,086/2182/2144 & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Class aggregations and corpus statistics (# sentences) for _sentiment_ and _emotion_ setups. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Experiment**} & \multirow{2}{*}{**Metric**} & \multirow{2}{*}{**Language**} & \multicolumn{4}{c}{**Classifier**} \\ \cline{4-7}  & & & **NB** & **SVM** & **FT** & **BERT** \\ \hline \multirow{4}{*}{Sentiment (3-class)} & \multirow{2}{*}{Accuracy} & en & 85.41 \(\pm\) 1.08 & 86.99 \(\pm\) 1.03 & 88.07 \(\pm\) 0.99 & 93.74 \(\pm\) 0.74 \\  & & pl & 83.00 \(\pm\) 1.15 & 84.89 \(\pm\) 1.10 & 85.59 \(\pm\) 1.08 & 92.24 \(\pm\) 0.82 \\  & & en & 85.44 \(  
quality of the translated sentences, this might also have been caused by the quality of the underlying pretrained embeddings. Nevertheless, as presented in Table 3, both English and Polish models learned to distinguish neutral sentences from emotional ones (the F1-score for the neutral class was around 97%). Positive and negative polarity predictions reached between 88% and 92%, with slightly higher per-class scores for negative polarity.