# Towards Sentiment Analysis for Romanian Twitter Content
## 4 Experiments and Results
### Assessing the Models Performance on Real Cases  
Given that the final purpose in our project is to apply the learned models for inferring the polarity of any Romanian tweet, we manually labeled two small test sets, each one containing 120 distinct tweets. The first one includes tweets specific to the airline industry, comparable with the ones used for training our models, and the second one includes general tweets. We applied on them the best models reported in the previous subsection (i.e., Bernoulli NB for TF-IDF encoding and LSTM for Word2 Vec), the public demo of semitmetric.ro (accessed on 8 September 2022) [21] and the BERT-based classifier.  
Each tweet was manually labeled by five human volunteers. Each one expressed an opinion about the polarity of the tweet and the final sentiment was established to be the one that was selected by the majority. Labeling statistics regarding how humans assessed the polarity is presented in Table 3. We shall note that the labeling task seemed to be a difficult one for the volunteers, as for only 43 tweets (35.8%) in the case of airline industry specific dataset and 47 tweets (39.2%) in the case of general tweets all the 5 volunteers reached a unanimous decision. Furthermore, the distribution of the polarity of the tweets significantly differ from the one of the Twitter US Airline Sentiment Tweets (presented in the last row of Table 3).  
Polarity estimation results on the Romanian dataset with airline industry-specific tweets are presented in Table 4.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Dataset** & **Negative** & **Neutral** & **Positive** & **Unanimous Annotation** \\ \hline Airline industry-specific tweets & 51, 0.465 & 36, 0.300 & 33, 0.275 & 43, 0.358 \\ General tweets & 45, 0.375 & 32, 0.266 & 43, 0.358 & 47, 0.392 \\ Twitter US Airline Sentiment Tweets & 0.63 & 0.21 & 0.16 & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Labeling statistics regarding how humans manually assessed the polarity (number of tweets and percentage).  
Both encodings supply better results with our models (Bernoulli NB and LSTM) than sentimetric.ro. This is expected as we learned our models on tweets specific for the aviation domain. BERT outperforms Bernoulli NB only by a slight margin.  
Table 5 presents the models results on the Romanian general tweets dataset. We notice that Bernoulli NB scores better than sentimetric.ro (accessed on 8 September 2022) in terms of the weighted F1-measure. LSTM scores worse. sentimetric.ro (accessed on 8 September 2022) proved to assess a better polarity on the general domain than on the aviation, which is expected, as we assume that the engine was constructed for a wide usage. BERT proves to be the state of the art, as the margin by which it outperforms our best models is around 5%.  
For both domains, our models' results are worse than those obtained on the translated test set used in Section 4.1, because now the tweets are real ones, not translated, and their target class distribution differs significantly--i.e., from a statistical point of view, sets are extracted from different statistical populations.  
Bernoulli NB is more robust to novel tweets and to a different domains than LSTM and we suppose that this happens because for LSTM the Word2 Vec embedding is learned on our very limited translated dataset and not on the whole Romanian language.