# Towards Sentiment Analysis for Romanian Twitter Content
## 2 Related Work  
SA is a type of text classification employed with the specific objective of inferring affective states and subjective information from text. To perform SA one could adopt either a specific lexicon-based strategy or could use the standard ML-based text classification pipeline [30]. As indicated by the literature [2; 3; 31], classical machine learning algorithms or novel deep learning approaches were applied for inferring SA models on texts. Among classical ML algorithms, popular choices [30] are Bernoulli Naive Bayes (NB) [32], Support Vector Machines (SVM) [33], Random Forest (RF) [34] or the LogisticRegression [35]. For deep learning, all important variants like the standard Deep Neural Network (DNN) [36], the Convolutional Neural Network (CNN) [37], or the Long Short-Term Memory (LSTM) [38] are reported to perform well on text classification tasks. Classical ML methods and the standard DNNs are applied on document-level embeddings such as the well-known TF-IDF [39] or the modern Doc2 Vec [40]. Novel methods of DL, including here architectures composed with CNN and LSTM cells are in general applied on word embeddings such as Word2 Vec [41]. The classical TF-IDF lead to high-dimensionality prediction problems, thus, the literature [3] suggests considering dimensionality reduction schemes, such as principal component analysis (PCA) [42], non-negative matrix factorization (NMF) [43] or Latent semantic analysis (LSA) [44], to make the computation more efficient. In our work, we will experiment with these methods, in the search for a suitable combination to fit our needs. Recently, Google proposes BERT [45] as a state-of-the-art pre-trained model for many NLP tasks. Multilingual BERT, pre-trained also for the Romanian language, is reported to be surprisingly good for cross-lingual model transfer [46]. However, as practice indicates [47], BERT comes with significant time costs for model learning--even on very powerful servers, hence this being in contradiction with the need of our project to frequently retrain the models and to accommodate high loads of messages on short time frames. Just for comparison, we will train a BERT-based classifier using its multilingual version for the Romanian language, to see how well we compare with the state of the art, in both classification performance and training time. Performing SA on microblogging content is seen as a difficult task [12] because one has to deal with bad language [9]. However, for popular languages in the world like English, Spanish or French, plenty of linguistic resources exist such that to enhance SA for microblogging content. For English, we mention BERTweet [10], a large-scale language model trained over a corpus of 850M Tweets, which could be used together with fairseq [13] or transformers [14] for inferring polarity. BERTweet scores an accuracy of 72% on the SemEval2017-Task4A [48] test set, outperforming its competitors RoBERT and XLM-R. [PERSON] et al. [49] reports BERTweet as being the State-of-the-Art on TweetEval benchmark, with a 73% average recall. In France, labeled Twitter data were made available for the participants in the DEFT challenges [15], while in Spain, the TASS workshops collected the efforts to classify Spanish annotated tweets [16]. [PERSON] et al. [50] applied BERT-based models for SA on both English and Italian Twitter datasets and drew conclusions about the importance of individualized preprocessing for exploiting hidden information, a suggestion which we specifically followed in our work. For the Persian language, which is also under-resourced like Romanian, in [51] the lack of any public dataset for sentiment analysis is noted. The authors of this work created a public dataset by collecting 11500 texts and manually labeling each one as either positive, negative or neutral. Around 80% of the texts were collected from an electronic product website, while the rest were collected from Twitter. They applied various classification algorithms and a proposed CNN-LSTM network achieved the highest accuracy of \(\approx\)85%.