<figure><img src="image_1.png" /></figure>

HAL
open science

# Multilingual Sentiment Analysis on Twitter Data
## Towards Enhanced Policy Making

George Manias, Athanasios Kiourtis, Argyro Mavrogiorgou, Dimosthenis Kyriazis

► To cite this version:

George Manias, Athanasios Kiourtis, Argyro Mavrogiorgou, Dimosthenis Kyriazis. Multilingual Sentiment Analysis on Twitter Data Towards Enhanced Policy Making. 18th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2022, Hersonissos, Greece. pp.325-337, 10.1007/978-3-031-08337-2_27. hal-04668667

HAL Id: hal-04668667

https://inria.hal.science/hal-04668667v1

Submitted on 7 Aug 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Multilingual Sentiment Analysis on Twitter Data towards Enhanced Policy Making

George Manias¹[0000-0003-0128-2022], Athanasios Kiourtis¹[0000-0002-1681-3626], Argyro Mavrogiorgou¹[0000-0002-1543-5627] and Dimosthenis Kyriazis¹[0000-0001-7019-7214]

¹ University of Piraeus, Piraeus, Greece
{gmanias, kiourtis, margy, dimos}@unipi.gr

**Abstract.** The great expansion in the usage and popularity of social media platforms during the last decades has led to the production of an enormous real-time volume of social texts and posts, including tweets, that are being produced by users. These collections of social data can be potentially useful and provide useful insights to policymakers to adjust new user-centric policies and regulations. However, extracting and analyzing valuable information and knowledge out of these data is a challenging task as concerns the high multilingualism that describes these data. Thus, both the research and the business communities focus on the utilization of multilingual approaches and solutions to enhance the policy making procedures. To investigate a portion of these challenges this research work performs a comparative analysis of two multilingual sentiment analysis approaches. In this context, three multilingual deep learning classifiers and a zero-shot classification approach were utilized and compared. Their comparison has unveiled insightful outcomes and has a two-fold interpretation. Multilingual deep learning classifiers that have pre-trained and evaluated in monolingual data achieve high performances and transfer inference when applied afterwards in multilingual data. However, the zero-shot classification approach fails to achieve high accuracies in monolingual data as in contrary to when applied on multilingual data.

**Keywords:** Multilingual Sentiment Analysis, Transformers, Multilingual Classifiers, Transfer Learning, Zero-Shot Classification.

## 1 Introduction

During the last decade, social networks have shown a tremendous growth in their usage and popularity. It is worth to mention that in January 2022 the number of people around the world using the social media has been 4.62 billion, 59.5% of the global population, with an increase of 10.1% (+424 million) compared to the same period on 2021 [1]. Especially during the COVID-19 pandemic outbreak social media increasingly assumed a key role in information provision, news spreading and advertising [2] and became one of the most effective digital marketing tools, with more companies embracing the potency of the analysis of social media [3]. As the number of social media users continues to soar, their sentiment is increasingly impacted by tweets and
posts. Thus, marketers and policymakers vividly leverage these networks' emerging power for promoting their products and influencing users' opinions. In fact, social media ad spends stood at approximately 132 billion U.S. dollars in 2021, with spending expected to surpass the 200-billion-dollar mark by 2024 [4]. One of the most popular social media platforms is Twitter, as it is considered one of the main micro-blogging and news sharing platforms [5], while Twitter's potential advertising has greatly increased within the same period [6]. The latter has a two-fold interpretation. In one hand it indicates the increasing need for modern stakeholders to leverage this enormous amount of information to further enhance their policies and advertising strategies for their own products or services. However, as it has been highlighted in recent research works, only a small amount of this large portion of data gives added value, making the retrieval of valuable knowledge out of these data a challenging task [7][8].

What is more, concerning the agricultural and food technology domains a recent survey highlights that only few research works have been conducted with specific focus on the utilization of sentiment analysis in these two domains [9]. More specifically, this research work [9] performs a bibliometric analysis based on Web of Science (WoS) database examining the relationship between sentiment analysis and advertising in several different domains indicating that from a total of 919 examined research works only 26 (2.8%) out of them are related to the agricultural and food technology domains. On top of this, another issue that modern social analysis tools face is highlighted by the aspect of multilingualism that designates these data. While English is the most used language, however relevant brand, product, or service information can be provided in a variety of languages especially when the examined item is of worldwide interest and consumption. The latter limits the use of supervised machine learning models as well as of monolingual sentiment analysis tools to extract and analyze information due to the lack of high-quality annotated and labelled data and corpora in languages other than English to train these specific models. Thus, in recent years, the implementation of multilingual sentiment analysis tools has steadily increased through the utilization of approaches from the domain of Artificial Intelligence (AI) and more specifically from the domain of Deep Learning (DL). To achieve multilingual sentiment analysis one of the most common approaches is based on the translation of the original text to English. Afterwards, a neural sentiment analysis model is applied on the translated data. However, this approach has revealed a significant loss in the sentiment of the examined data [10]. Another approach is to use sentiment lexicon translated into multiple languages. The main drawbacks of this approach are that it requires the involvement of humans in the process of text analysis, as well as that several lexical objects lack of domain interoperability, thus lexicon-based analysis does not have high accuracy yet and its optimization is still an ongoing research topic [11]. In recent years, the performance of multilingual sentiment analysis tools has improved and enhanced through the utilization of approaches and techniques that are mainly based on the implementation of Neural Networks (NNs) and Transformers [12].

To this end, this paper leverages the potentials that these emerging technologies pose to implement a comparative analysis for achieving multilingual sentiment analy-
sis on collected data related to multilingual reviews of wine products. The analysis seeks to highlight the value of the utilization of multilingual sentiment analysis on tweets related to food and agricultural brands and products as a great tool for the modern policy makers. In this context different NNs were exploited coupled with the utilization of pre-trained multilingual sentence embeddings to be used as inputs on the first layers of these NNs. The exploitation and evaluation of pre-trained multilingual sentence embeddings is a key aspect as they have shown remarkable results in the task of multilingual sentiment analysis [13]. What is more, a zero-shot classification approach is also implemented in the context of this paper to further evaluate the application of a model that has not been trained before on data belonging to the same language. The overall evaluation and comparison of the first approach is performed through the training of the proposed NN models to open labelled data and reviews related to wines and then applied to raw unlabeled tweets written in different languages, thus achieving zero-shot and transfer inference in terms of model accuracy. While the zero-shot classification approach is utilized without pretraining on both the labelled data and on the unlabeled multilingual tweets.

The remainder of this paper is structured as follows. Section 2 describes the related work from the domains of multilingual sentiment classifiers and zero-shot classification. The dataset as well as the methodology implementation and the approaches that are followed are being presented in Section 3. Afterwards, Section 4 presents the achieved performances and the experimental results of the utilization of the proposed approaches. Finally, Section 5 concludes the paper and outlines some directions for future works and further enhancements on the proposed methodology.

## 2 Related Work

One of the main challenges facing the modern field of Natural Language Processing (NLP) is the need for the implementation of sophisticated, holistic, and multilingual approaches in modern multilingual and multicultural societies. Therefore, researchers are constantly trying to develop the most comprehensive multilingual systems. To this end, several AI research teams from major pioneers, such as Google AI and Facebook AI Research, have introduced multilingual tools, corpora and sentence encoding models that are able to cover any language, thus overcoming the limitations imposed by the lack of labelled data in all languages [14][15]. The utilization of transfer learning techniques leverages the development of multilingual models [16]. On top of this, recent research works have been applied on a specific form of transfer learning, the zero-shot learning [17]. The latter is utilized when no training data are available for a model that is used for prediction and with the ultimate objective to minimize the overall latency while maintaining robust classification performances. The main idea is to learn a mapping from classes to a vector in such a way that an unseen class in the future can be mapped to the same space and specific information can be retrieved for this unseen class. To this end, several research works take advantage of this approach and apply to several NLP tasks achieving great results [18][19].
Moreover, one of the latest milestones in the NLP field is the introduction of BERT that enables transfer learning with large language models reaching the state-of-the-art for a great number of NLP tasks and applications [20]. In this context several research works have proposed multilingual models based on the utilization of BERT for a wide range of cross-lingual transfer tasks. More specifically, XLM-RoBERTa (XLM-R) is trained on a multi-lingual language modeling objective using only monolingual data [12], while XLM-T framework has based its implementation on an XLM-R model pretrained specifically on millions of tweets in over thirty languages with the scope for using and evaluating multilingual language models in Twitter [21]. On top of this, XLM-T features a set of unified sentiment analysis Twitter datasets in eight different languages. In the same context, several research works have introduced multilingual BERT-based models specifically pre-trained on tweet corpora, thus have focused their implementation on the task of sentiment analysis on tweets [22]-[24]. A cornerstone in the task of multilingual sentiment analysis [25] is the release of the Multilingual Universal Sentence Encoder (MUSE) from the Google AI team [26], which is a sentence encoding model simultaneously trained on multiple tasks and languages and creates a single embedding space common for all languages on which it has been trained on. The alignment in the same latent space provided by such embeddings allows to apply a transfer learning approach, thus the learned knowledge from a single training language can be transferred to other languages. Moreover, authors in [14] introduce a new evaluation framework for multilingual document classification in eight languages and provide the baselines for all language transfer directions using multilingual word and sentence embeddings.

In this paper is implemented a comparative analysis of two different approaches to indicate the maturity and accuracy of these approaches in the task of multilingual sentiment analysis. At first, the approach of Multilingual Sentence Classification (MSC) is analyzed and presented as a tool for retrieving semantically similar text in multiple languages. This approach has a two-fold interpretation, as it also utilizes and evaluates the use of multilingual sentence embeddings coupled with different NNs models. Thanks to the ability of multilingual sentence embeddings to represent meanings of words from different languages in the same vector space, the proposed multilingual sentiment analysis models become almost independent from the input language. Moreover, exploiting the vector alignment of multilingual sentence embeddings, a language-agnostic pipeline is defined that can be trained with any initial labeled data set, and then applied to Tweets written in different languages. Finally, a zero-shot classification approach is followed, and its outcomes are also evaluated and compared with those of the different NNs models of the MSC approach, thus a comprehensive analysis and comparison between these two approaches is.

## 3 Multilingual Sentiment Analysis

The proposed methodology to evaluate multilingual sentiment analysis involves two key approaches, the Multilingual Sentence Classification (MSC) and a zero-shot classification approach. Concerning the first approach, the first step is based on the com-
putation of sentence embeddings. Afterwards, NN models were implemented and
compared for performing the multilingual sentiment analysis task. Hence, a DL ap-
proach is followed and depicted as an activity pipeline in “Fig. 1”. Moreover, a zero-
shot classification approach is also introduced and followed as presented in “Fig. 2”,
and its outcomes were evaluated and compared with the ones of the MSC approach.

**3.1 Multilingual Sentence Classification (MSC)**

The Google Universal Sentence Encoder Multilingual (GUSEM) module is an extension of the MUSE [26] and was introduced for retrieving semantically similar text. Moreover, this tool is specifically optimized for increased semantic retrieval performance as it is designed to be used for tasks such as text classification and cross-lingual text retrieval, among others. Thus, before utilizing the NNs, the sentence embedding representation leveraging the functionalities of the GUSEM module was being created, as it was also used as input in the first layers of the proposed NN models. Under the scopes of this approach three different types of Neural Networks (NNs) were also utilized and more specifically a Vanilla NN (VNN), a Convolutional NN (CNN) and a Long-Short Term Memory (LSTM). The Vanilla NN (MSC-VNN) consisted of 4 layers fully connected. At first an input dense layer was initialized with input the sentence embeddings that have been previously generated. The second layer has a Dropout function of 0.3 to avoid overfitting, and the third layer and the last layer were also Dense layers. Most importantly the last layer has 5 neurons, as the total number of the classes in which the data are classified, and it utilizes a “softmax” activation classifier, as a multiclass classification task was performed in the context of this research work [27]. On top of this, the second NN that was implemented is a Convolutional NN (MSC-CNN) as it has proven that they perform well in text classification tasks [28]. Moreover, an Embedding layer was used as a first layer on this model to leverage the sentence embeddings from the previous step. Though text data are one-dimensional, since they are represented in sentence vectors, 1D convolutional NNs used to extract features from these data. In addition, by adding a GlobalMax-Pooling1D layer it extracts the maximum value from each filter, and the output dimension is just a 1-dimensional vector with length as same as the number of filters that were applied. What is more, Recurrent NNs (RNNs) are a type of NNs that is proven to work well with sequence data [29]. Long Short-Term Memory Networks (LSTMs), a variant of RNNs, are used widely to address the sentiment analysis problems, thus it is also implemented in the context of this paper [30]. All layers except the LSTM layer and the pooling layer, that is not necessary in this model, were the same with those created in the previous MSC-CNN model.

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Multilingual Sentence Classification (MSC) pipeline</figcaption></figure>
## 3.2 Zero-Shot Sentiment Classification (ZSSC)

Recently, zero-shot text classification attracted a huge interest as it is an approach that facilitates the prediction of the target class of a text without having trained on any of the candidate classes/labels [31]. To this end, zero-shot classification is a revolution for unsupervised text classification as it provides more flexibility and demonstrates all the power of transfer learning for more general and unlabeled data. In this context, a zero-shot classification pipeline from Hugging Face library was utilized [32]. More specifically, after the initial text preprocessing the pipeline module and the multilingual model were defined. Hence, the *xlm-roberta-large-xnli* model was selected as the appropriate one to be utilized and evaluated [33], as it is fine-tuned on 100 different languages on the multilingual XNLI dataset [34]. Moreover, as the problem in this research work is a multi-class classification, the *"multi_label"* parameter was placed on *True* during its training phase, thus the scores are independent for each class, but for each of them its probability is computed between 0 and 1. Zero-shot approach was selected as it refers to a problem setup in which a model performs classification on labels it has never seen before, hence it can classify newly fetched and unlabeled twitter data as in the real-world scenario that is examined in the context of this research work. The latter facilitates the training of a model using a limited amount of labeled data that is scarce and not available in many languages, and then the knowledge gained from it to be applied to a wider range of languages and domains.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Zero-Shot Sentiment Classification (ZSSC) pipeline</figcaption></figure>

# 4 Experimental Results

This section presents the experimental results of the utilization of the above presented approaches in labeled datasets and then in unlabeled Twitter data that have been fetched in the context of an evaluation of this research work in a real-world scenario.

## 4.1 Datasets Description

For the purposes of this research work two different datasets have been used and two different sets of experiments were performed. Under the scopes of the training and an initial evaluation of the examined MSC models, a Kaggle dataset has been used and training and testing iterations performed on English-only data [35]. This dataset contains two files with wine reviews, the rating of the wine (measured in points) and other relevant information for wine products. Finally, the trained models were applied in unlabeled multilingual twitter data to evaluate these models on a real-world scenario. Thus, their transfer inference level was evaluated through testing the accuracy of them on multilingual data that are seen first time. These twitter data have been fetched
based on specific wine products as keywords and more specifically based on the "merlot", "cabernet" and "carinena" keywords. With regards to the ZSSC approach the selected pre-trained model (i.e., xlm-roberta-large-xnli) was applied in all set of experiments and datasets without prior training on the data to evaluate its accuracy and performance on monolingual and multilingual data that are seen for first time.

## 4.2 Evaluation Setup and Measures

Experiments presented in this paper were based on implementing the multilingual sentiment analysis task and the presented approaches with the Python programming language and the utilization of TensorFlow 2.0 [36] and Keras libraries [37]. Moreover, the implementation of the code, for both MSC and ZSSC approaches, was performed on Google Colab [38] online platform to facilitate the overall running time through the usage of the GPU option that is available. Moreover, several tools and libraries were utilized to perform several tasks, such as the *sklearn* that was utilized for the computation of the performances of the implemented models, as well as for the splitting of the initial examined dataset to train and test set. Moreover, measures of evaluating the efficiency of NLP systems go beyond the concepts of Precision, Accuracy, Recall and F1-Score. F1-Score has proven to be a better measure to be used and examined, as it provides a balance between Precision and Recall and there is an uneven class distribution (large number of Actual Negatives).

## 4.3 Pre-processing

The initial reviews of the wine products in the Kaggle datasets are provided via a 100-point scale. However, under the scopes of this research work and as the task that is performed is a sentiment analysis problem, thus a transformation should be applied in the rating to also reducing the dimensionality of the problem. The overall range of the review points was ranged from 80-100, thus under the scopes of this research work the **Table 1** was created with the corresponding mapping ratings to also include 5 different classes to express the sentiment and the classification of the wines. Hence, text reviews were analyzed to predict whether a wine is "Mediocre", "Good", "Very Good", "Outstanding", and "Classic" based also on the wine rating information that collected from official wine-related sources [39] and the needed transformations to fit to the examined datasets. One hot encoding approach also utilized to further encode the identified classes for each sentence.

<table><caption>**Table 1.** Description and Mapping of Ratings</caption><thead><tr><td>100-point scale</td><td>Rating</td><td>Number of reviews</td><td>No of Class</td></tr></thead><tbody><tr><td>96-100</td><td>Classic</td><td>1266</td><td>4</td></tr><tr><td>92-95</td><td>Outstanding</td><td>20436</td><td>3</td></tr><tr><td>88-91</td><td>Very Good</td><td>57301</td><td>2</td></tr><tr><td>84-87</td><td>Good</td><td>59437</td><td>1</td></tr><tr><td>80-83</td><td>Mediocre</td><td>12491</td><td>0</td></tr></tbody></table>
Moreover, it was observed that many records occur in classes 1 and 2, while only a few can be observed in class 4. Hence, class weights were used to give more emphasis on the minority classes and manage this imbalance, thus higher weights were attached to less occurred classes (e.g., *class No4: 7*, while *class No1: 1*) to have a higher impact on the training of the NNs. Finally, the preprocessing of raw texts is a crucial and mandatory step as they can contain a high portion of unwanted elements, from plain text, mentions, links, and punctuations, to many other html tags. Hence, all these “noisy data” need to be removed before proceeding to the further processing and analysis of the texts. In the context of this research work, filtering, and removal of punctuations, links, mentions and stopwords were performed in all examined datasets. Following the abovementioned steps of pre-processing results in cleaned texts and tweets, which can be further processed and analyzed to implement the multilingual sentiment analysis task on them.

## 4.4 Results and Discussion

As analyzed in previous sections two different sets of experiments were implemented to evaluate the proposed approaches and models. At first, the dataset that was used from the Kaggle contains two files, the *winemag-data_first150k.csv* and the *winemag-data-130k-v2.csv*. The first of these files was used as the training dataset for the evaluation of the MSC approach. Initially, it was separated at 70% as the training and 30% as the test set and then the two different approaches were utilized. The ZSSC approach is utilized without any prior training in the data. The results in **Table 2** indicate that the three models of the MSC approach achieve good performances in the monolingual multilabel classification task. Of course, further fine-tuning and different settings in the selected layers and the activation functions can be performed to enhance their performances. On the other hand, the ZSSC approach fails to generalize well and performs pure in this task. Utilizing a multilingually pretrained model in a single source-language leads to poor accuracy as forgetting of its multilingual knowledge and underperforming of its functionalities. Moreover, the performance of the ZSSC approach is rather poor due to high similarity between the predicted classes. It is a very hard task to make distinction between *Outstanding*, *Very Good* and *Good* without any prior data. Afterwards, the same MSC trained models were utilized in the second file (*winemag-data-130k-v2.csv*). **Table 3** indicates that all three models of the MSC approach can generalize very well and achieve high performance rates and high transfer learning rate. However, the ZSSC approach also fails to achieve high performances in this experiment, which is in align with the prior results.

<table><caption>**Table 2.** Experimental Results for the Training Dataset</caption><thead><tr><td>Model</td><td>Precision</td><td>Accuracy</td><td>Recall</td><td>F1-Score</td></tr></thead><tbody><tr><td>MSC-VNN</td><td>0.7520</td><td>0.7513</td><td>0.7493</td><td>0.7504</td></tr><tr><td>MSC-CNN</td><td>0.7654</td><td>0.7702</td><td>0.7689</td><td>0.7775</td></tr><tr><td>MSC-LSTM</td><td>0.8103</td><td>0.8079</td><td>0.8032</td><td>0.8142</td></tr><tr><td>ZSSC</td><td>0.5105</td><td>0.5086</td><td>0.5041</td><td>0.5003</td></tr></tbody></table>
<table><caption>**Table 3.** Experimental Results for the Evaluation Dataset</caption><thead><tr><th>Model</th><th>Precision</th><th>Accuracy</th><th>Recall</th><th>F1-Score</th></tr></thead><tbody><tr><td>MSC-VNN</td><td>0.8351</td><td>0.8431</td><td>0.8484</td><td>0.8534</td></tr><tr><td>MSC-CNN</td><td>0.8296</td><td>0.8312</td><td>0.8343</td><td>0.8394</td></tr><tr><td>MSC-LSTM</td><td>0.8678</td><td>0.8756</td><td>0.8631</td><td>0.8732</td></tr><tr><td>ZSSC</td><td>0.5203</td><td>0.5352</td><td>0.5261</td><td>0.5215</td></tr></tbody></table>

Finally, the already implemented models were evaluated in real-world multilingual twitter data that have been fetched based on specific keywords related with wine products and more specifically the keywords “merlot”, “cabernet” and “carinena”. The outcomes of the utilization of these models in multilingual tweets have shown remarkable results and have indicated that the implemented models generalize very well in other languages than the initial in which they were trained (English), thus achieving high transfer inference in terms of model accuracy. **Table 4** depicts five sample tweets from the overall 2000 collected that represent a good performance and classification from the application of these models. In this experiment it is worth to mention that the ZSSC approach has resulted in close or same classification outcomes as the NNs in the MSC approach, and especially as the MSC-VNN. The latter indicates that the ZSSC approach outperforms compared to when utilized in large monolingual sentences, as in the first set of experiments, as it generalizes better via self-training in multilingual data. The latter indicates the need for specific fine-tuning and pretraining of this model in monolingual data and the introducing of weights in the candidate classes.

<table><caption>**Table 4.** Sample Classifications for the Multilingual Tweets</caption><thead><tr><th>Description</th><th>Language</th><th>Model</th><th>Predicted Class</th></tr></thead><tbody><tr><td rowspan="4">Anbefaler en tur til Mendoza for grillet bloddopet bøff og merlot</td><td rowspan="4">Danish</td><td>MSC-VNN</td><td>Outstanding</td></tr><tr><td>MSC-CNN</td><td>Outstanding</td></tr><tr><td>MSC-LSTM</td><td>Classic</td></tr><tr><td>ZSSC</td><td>Outstanding</td></tr><tr><td rowspan="4">Little Merlot Meatball Marinara Murder the perfect evening</td><td rowspan="4">English</td><td>MSC-VNN</td><td>Very Good</td></tr><tr><td>MSC-CNN</td><td>Very Good</td></tr><tr><td>MSC-LSTM</td><td>Outstanding</td></tr><tr><td>ZSSC</td><td>Very Good</td></tr><tr><td rowspan="4">de boassa aqui tomando meu merlot e chega leon e lança dindo eu vomitei no seu computador</td><td rowspan="4">Portuguese</td><td>MSC-VNN</td><td>Good</td></tr><tr><td>MSC-CNN</td><td>Good</td></tr><tr><td>MSC-LSTM</td><td>Good</td></tr><tr><td>ZSSC</td><td>Mediocre</td></tr><tr><td rowspan="4">Para empezar la Semana Tranquilo y Mimando a mi Pequeño Diamante,Salió Sushi 🍃 al estilo Cadiboni, obvio acompañado de una buena Copa de Vino 👤 hoy Un Rosado Merlot Disfrutando de una cena con</td><td rowspan="4">Spanish</td><td>MSC-VNN</td><td>Outstanding</td></tr><tr><td>MSC-CNN</td><td>Classic</td></tr><tr><td>MSC-LSTM</td><td>Classic</td></tr><tr><td>ZSSC</td><td>Very Good</td></tr></tbody></table>
<table><tr><td rowspan="4">Merrez'ca เบอร์ 02 Merlot สีสวยไม่ไหว<br/>โทนส้มอิฐก่าๆ เกาหลี卷ใจ<br/>เนื้อครีมแต่พอเกลียแล้วจะเป็นเนื้อมูสันดุๆ<br/>พิกเม้นต์แน่น ท</td><td rowspan="4">Thai</td><td>MSC-VNN</td><td>Outstanding</td></tr><tr><td>MSC-CNN</td><td>Outstanding</td></tr><tr><td>MSC-LSTM</td><td>Outstanding</td></tr><tr><td>ZSSC</td><td>Outstanding</td></tr></table>

## 5 Conclusion

In this paper, a comparative analysis and evaluation of two different experimentation sets in multilingual sentiment analysis task are presented and examined to specify and indicate the increasing need of utilizing multilingual tasks of NLP. Both approaches were utilized in labeled as well as in unlabeled data that have been fetched based on a real-world scenario in the context of PolicyCLOUD project [40]. The latter provides actionable knowledge to interested stakeholders to further enhance their policies based on the multilingual sentiment analysis task on specific wine products. On top of this, the challenges of performing multilingual sentiment analysis have been examined with emphasis on the evaluation of the role and the impact that pre-trained sentence embeddings and zero-shot classification can have. What is more, the fundamental characteristics of transfer learning approaches are analyzed, as well as the utilization of pre-trained sentence embeddings.

The outcomes of the proposed approaches have a twofold interpretation. The comparison on using the same trained NNs, and more specifically the MSC-LSTM model, in source and target languages by using also a pre-trained sentence embedding model reveals that great achievements have been accomplished and that these models capture semantic information in multiple languages and in high performance. Their utilization in different languages than in the initial language in which they were trained on, enhances the multilingualism and interoperability of these NN models with almost zero losses. Hence, their transfer inference and their application in other languages can be accomplished without significant losses. On the other hand, the overall outcomes and performance rates of zero-shot classification pipeline leads to a conclusion that utilizing a multilingually pretrained model in a single source-language leads to poor accuracy and its pretraining on monolingual data may lead to better results. Moreover, further fine-tuning on this model should be implemented especially in the cases where the texts need to be classified in more than one classes/labels.

To this end, there is a lot of additional work to perform according to the utilization of zero-shot classification, as it is a comparatively new topic and its adaptation in many domain fields needs to be further examined. On top of this, the initial manually labeling of a large portion of examples may enhance its performance and finally achieve higher accuracy than supervised models that have been trained on hundreds of labeled training items. The latter, as also both the proposed approaches in this paper, will be further evaluated and applied in the context of a holistic environment for data-driven policy making as realized by the PolicyCLOUD project [40], where data from four different languages (Bulgarian, Italian, Spanish, and English) will be utilized and processed.
## Acknowledgement

The research leading to the results presented in this paper has received funding from the European Union's funded Project PolicyCLOUD under grant agreement no 870675.

## References

1. Global Social Media Stats | DataReportal – Global Digital Insights, https://datareportal.com/social-media-users, last accessed 2022/02/27.

2. Wiederhold, B. K.: Social media use during social distancing. Cyberpsychology, Behavior, and Social Networking 23(5), 275-276 (2020).

3. Alalwan, A. A.: Investigating the impact of social media advertising features on customer purchase intention. International Journal of Information Management 42, 65-77 (2018).

4. Social media marketing - statistics & facts, https://www.statista.com/topics/1538/social-media-marketing/#dossierKeyfigures, last accessed 2022/02/26.

5. 100 Social Media Statistics For 2022 [+Infographic] | Statusbrew, https://statusbrew.com/insights/social-media-statistics, last accessed 2022/02/25.

6. The Latest Twitter Stats: Everything You Need to Know | DataReportal – Global Digital Insights, https://datareportal.com/essential-twitter-stats, last accessed 2022/02/26.

7. Păvăloiaia, V. D., et al.: Opinion mining on social media data: sentiment analysis of user preferences. Sustainability 11(16), 4459 (2019).

8. Tao, D., Yang, P., Feng, H.: Utilization of text mining as a big da-ta analysis tool for food science and nutrition. Comprehensive reviews in food science & food safety 19(2), 875-894 (2020).

9. Sánchez-Núñez, P., et al: Opinion mining, sentiment analysis and emotion understanding in advertising: a bibliometric analysis. IEEE Access 8, 134563-134576 (2020).

10. Manias, G., et al.: An Evaluation of Neural Machine Translation and Pre-trained Word Embeddings in Multilingual Neural Sentiment Analysis. In: 2020 IEEE International Conference on PIC, pp. 274-283. IEEE (2020).

11. Sadia, A., Khan, F., Bashir, F.: An overview of lexicon-based approach for sentiment analysis. In: IEEC 2018, pp. 1-6 (2018).

12. Conneau, A., et al: Unsupervised Cross-lingual Representation Learning at Scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451. Association for Computational Linguistics (2020).

13. Aydoğan, M., Karci, A.: Improving the accuracy using pre-trained word embeddings on deep neural networks for Turkish text classification. Physica A: Statistical Mechanics and its Applications 541, 123288 (2020).

14. Schwenk, H., Li, X.: A corpus for multilingual document classification in eight languages. arXiv preprint arXiv:1805.09821 (2018).

15. Eriguchi, A., et al.: Zero-shot cross-lingual classification using multilingual neural machine translation. arXiv preprint arXiv:1809.04686 (2018).

16. Tan, C., et al.: A survey on deep transfer learning. In: International conference on artificial neural networks, pp. 270-279. Springer, Cham (2018).

17. Xian, Y., et al.: Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9), pp. 2251-2265 (2018).
18. Artetxe, M., Schwenk, H.: Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. TACL. 7, pp. 597-610 (2019).

19. Sarkar, A., Reddy, S., Iyengar, R. S.: Zero-shot multilingual sentiment analysis using hierarchical attentive network and BERT. In: Proceedings of the 2019 3rd International Conference on Natural Language Processing and Information Retrieval, pp. 49-56 (2019).

20. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Conf. of the NAACL-HLT 2019, pp. 4171–4186 (2019).

21. Barbieri, F., Anke, L. E., Camacho-Collados, J.: Xlm-t: A multilingual language model toolkit for twitter. arXiv preprint arXiv:2104.12250 (2021).

22. Gonzalez, J. A., Hurtado, L. F., Pla, F.: TWilBert: Pre-trained deep bidirectional transformers for Spanish Twitter. Neurocomputing 426, 58-69 (2021).

23. Pota, M., Ventura, M., Catelli, R., Esposito, M.: An effective BERT-based pipeline for Twitter sentiment analysis: a case study in Italian. Sensors 21(1), 133 (2021).

24. Polignano, M., et al.: Alberto: Italian BERT language understanding model for NLP challenging tasks based on tweets. In: 6th Italian Conference on Computational Linguistics, CLiC-it 2019, vol. 2481, pp. 1-6. CEUR (2019).

25. Wehrmann, J., Becker, W. E., Barros, R. C.: A multi-task neural network for multilingual sentiment classification and language detection on twitter. In: Proceedings of the 33rd Annual ACM Symposium on Applied Computing, pp. 1805-1812 (2018).

26. Yang, Y., et al: Multilingual Universal Sentence Encoder for Semantic Retrieval. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 87–94 (2020).

27. Janocha, K., Czarnecki, W. M.: On Loss Functions for Deep Neural Networks in Classification. Schedae Informaticae 25 (2017).

28. Afridi, M. J., Ross, A., Shapiro, E. M.: On automated source selection for transfer learning in convolutional neural networks. Pattern recognition 73, pp. 65-75 (2018).

29. Manaswi, N. K.: Rnn and lstm. In: Deep Learning with Applications Using Python, pp. 115-126. Apress, Berkeley, CA (2018).

30. Muhammad, P. F., Kusumaningrum, R., Wibowo, A.: Sentiment analysis using Word2vec and long short-term memory (LSTM) for Indonesian hotel reviews. Procedia Computer Science 179, pp. 728-735 (2021).

31. Kumar, A., Albuquerque, V. H. C.: Sentiment Analysis Using XLM-R Transformer and Zero-shot Transfer Learning on Resource-poor Indian Language. Transactions on Asian and Low-Resource Language Information Processing 20(5), pp. 1-13, 2021.

32. Hugging Face, https://huggingface.co, last accessed 2022/02/26

33. joeddav/xlm-roberta-large-xnli - Hugging Face, https://huggingface.co/joeddav/xlm-roberta-large-xnli, last accessed 2022/03/02

34. Conneau, A., et al: XNLI: Evaluating cross-lingual sentence representations. In: Proceedings of the EMNLP 2018, pp. 2475-2485 (2020).

35. Wine Reviews | Kaggle, https://www.kaggle.com/zynicide/wine-reviews/home, last accessed 2022/03/05.

36. TensorFlow, https://www.tensorflow.org/, last accessed 2022/02/27.

37. Keras: the Python deep learning API, https://keras.io/, last accessed 2022/02/27.

38. Google Colaboratory, https://colab.research.google.com/, last accessed 2022/03/09.

39. Wine-Searcher, https://www.wine-searcher.com/wine-scores, last accessed 2022/02/28.

40. D. Kyriazis, et al: PolicyCLOUD: Analytics as a Service Facilitating Efficient Data-Driven Public Policy Management. In IFIP International Conference on Artificial Intelligence Applications and Innovations, vol. 583, pp. 141-150, Springer, Cham, 2020.