A Simple Approach to Multilingual Polarity Classification in Twitter

Abstract: Recently, sentiment analysis has received a lot of attention due to the interest in mining opinions of social media users. Sentiment analysis consists in determining the polarity of a given text, i.e., its degree of positiveness or negativeness. Traditionally, Sentiment Analysis algorithms have been tailored to a specific language given the complexity of having a number of lexical variations and errors introduced by the people generating content. In this contribution, our aim is to provide a simple to implement and easy to use multilingual framework, that can serve as a baseline for sentiment analysis contests, and as starting point to build new sentiment analysis systems. We compare our approach in eight different languages, three of them have important international contests, namely, SemEval (English), TASS (Spanish), and SENTIPOLC (Italian). Within the competitions our approach reaches from medium to high positions in the rankings; whereas in the remaining languages our approach outperforms the reported results.

Sentiment analysis is a crucial task in opinion mining field where the goal is to extract opinions, emotions, or attitudes to different entities (person, objects, news, among others). Clearly, this task is of interest for all languages; however, there exists a significant gap between English state-of-the-art methods and other languages. It is expected that some researchers decided to test the straightforward approach which consists in, first, translating the messages to English, and, then, use a high performing English sentiment classifier (for instance, see BIBREF0 and BIBREF1 ) instead of creating a sentiment classifier optimized for a given language. However, the advantages of a properly tuned sentiment classifier have been studied for different languages (for instance, see BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 ).
This manuscript focuses on the particular case of multilingual sentiment analysis of short informal texts such as Twitter messages. Our aim is to provide an easy-to-use tool to create sentiment classifiers based on supervised learning (i.e., labeled dataset) where the classifier should be competitive to those sentiment classifiers carefully tuned by some given languages. Furthermore, our second contribution is to create a well-performing baseline to compare new sentiment classifiers in a broad range of languages or to bootstrap new sentiment analysis systems. Our approach is based on selecting the best text-transforming techniques that optimize some performance measures where the chosen techniques are robust to typical writing errors.
In this context, we propose a robust multilingual sentiment analysis method, tested in eight different languages: Spanish, English, Italian, Arabic, German, Portuguese, Russian and Swedish. We compare our approach ranking in three international contests: TASS'15, SemEval'15-16 and SENTIPOLC'14, for Spanish, English and Italian respectively; the remaining languages are compared directly with the results reported in the literature. The experimental results locate our approach in good positions for all considered competitions; and excellent results in the other five languages tested. Finally, even when our method is almost cross-language, it can be extended to take advantage of language dependencies; we also provide experimental evidence of the advantages of using these language-dependent techniques.
The rest of the manuscript is organized as follows. Section SECREF2 describes our proposed Sentiment Analysis method. Section SECREF3 describes the datasets and contests used to test our approach; whereas, the experimental results, and, the discussion are presented on Section SECREF4 . Finally, Section SECREF5 concludes.
We propose a method for multilingual polarity classification that can serve as a baseline as well as a framework to build more complex sentiment analysis systems due to its simplicity and availability as an open source software. As we mentioned, this baseline algorithm for multilingual Sentiment Analysis (B4MSA) was designed with the purpose of being multilingual and easy to implement. B4MSA is not a naïve baseline which is experimentally proved by evaluating it on several international competitions.
In a nutshell, B4MSA starts by applying text-transformations to the messages, then transformed text is represented in a vector space model (see Subsection SECREF13 ), and finally, a Support Vector Machine (with linear kernel) is used as the classifier. B4MSA uses a number of text transformations that are categorized in cross-language features (see Subsection SECREF3 ) and language dependent features (see Subsection SECREF9 ). It is important to note that, all the text-transformations considered are either simple to implement or there is a well-known library (e.g. BIBREF6 , BIBREF7 ) to use them. It is important to note that to maintain the cross-language property, we limit ourselves to not use additional knowledge, this include knowledge from affective lexicons or models based on distributional semantics.
To obtain the best performance, one needs to select those text-transformations that work best for a particular dataset, therefore, B4MSA uses a simple random search and hill-climbing (see Subsection SECREF14 ) in space of text-transformations to free the user from this delicate and time-consuming task. Before going into the details of each text-transformation, Table TABREF2 gives a summary of the text-transformations used as well as their parameters associated.
We defined cross-language features as a set of features that could be applied in most similar languages, not only related language families such as Germanic languages (English, German, etc.), Romance languages (Spanish, Italian, etc.), among others; but also similar surface features such as punctuation, diacritics, symbol duplication, case sensitivity, etc. Later, the combination of these features will be explored to find the best configuration for a given classifier.
Generally, Twitter messages are full of slang, misspelling, typographical and grammatical errors among others; in order to tackle these aspects we consider different parameters to study this effect. The following points are the parameters to be considered as spelling features. Punctuation (del-punc) considers the use of symbols such as question mark, period, exclamation point, commas, among other spelling marks. Diacritic symbols (del-diac) are commonly used in languages such as Spanish, Italian, Russian, etc., and its wrong usage is one of the main sources of orthographic errors in informal texts; this parameter considers the use or absence of diacritical marks. Symbol reduction (del-d1), usually, twitter messages use repeated characters to emphasize parts of the word to attract user's attention. This aspect makes the vocabulary explodes. We applied the strategy of replacing the repeated symbols by one occurrence of the symbol. Case sensitivity (lc) considers letters to be normalized in lowercase or to keep the original source; the aim is to cut the words that are the same in uppercase and lowercase.
We classified around 500 most popular emoticons, included text emoticons, and the whole set of unicode emoticons (around INLINEFORM0 ) defined by BIBREF8 into three classes: positive, negative and neutral, which are grouped under its corresponding polarity word defined by the class name.
Table TABREF6 shows an excerpt of the dictionary that maps emoticons to their corresponding polarity class.
N-words (word sequences) are widely used in many NLP tasks, and they have also been used in Sentiment Analysis BIBREF9 and BIBREF10 . To compute the N-words, the text is tokenized and N-words are calculated from tokens. For example, let INLINEFORM0 be the text, so its 1-words (unigrams) are each word alone, and its 2-words (bigrams) set are the sequences of two words, the set ( INLINEFORM1 ), and so on. INLINEFORM2 = {the lights, lights and, and shadows, shadows of, of your, your future}, so, given text of size INLINEFORM3 words, we obtain a set containing at most INLINEFORM4 elements. Generally, N-words are used up to 2 or 3-words because it is uncommon to find, between texts, good matches of word sequences greater than three or four words BIBREF11 .
In addition to the traditional N-words representation, we represent the resulting text as q-grams. A q-grams is an agnostic language transformation that consists in representing a document by all its substring of length INLINEFORM0 . For example, let INLINEFORM1 be the text, its 3-grams set are INLINEFORM2 
so, given text of size INLINEFORM0 characters, we obtain a set with at most INLINEFORM1 elements. Notice that this transformation handles white-spaces as part of the text. Since there will be q-grams connecting words, in some sense, applying q-grams to the entire text can capture part of the syntactic and contextual information in the sentence. The rationale of q-grams is also to tackle misspelled sentences from the approximate pattern matching perspective BIBREF12 .
The following features are language dependent because they use specific information from the language concerned. Usually, the use of stopwords, stemming and negations are traditionally used in Sentiment Analysis. The users of this approach could add other features such as part of speech, affective lexicons, etc. to improve the performance BIBREF13 .
In many languages, there is a set of extremely common words such as determiners or conjunctions ( INLINEFORM0 or INLINEFORM1 ) which help to build sentences but do not carry any meaning for themselves. These words are known as Stopwords, and they are removed from text before any attempt to classify them. Generally, a stopword list is built using the most frequent terms from a huge document collection. We used the Spanish, English and Italian stopword lists included in the NLTK Python package BIBREF6 in order to identify them.
Stemming is a well-known heuristic process in Information Retrieval field that chops off the end of words and often includes the removal of derivational affixes. This technique uses the morphology of the language coded in a set of rules that are applied to find out word stems and reduce the vocabulary collapsing derivationally related words. In our study, we use the Snowball Stemmer for Spanish and Italian, and the Porter Stemmer for English that are implemented in NLTK package BIBREF6 .
Negation markers might change the polarity of the message. Thus, we attached the negation clue to the nearest word, similar to the approaches used in BIBREF9 . A set of rules was designed for common negation structures that involve negation markers for Spanish, English and Italian. For instance, negation markers used for Spanish are no (not), nunca, jamás (never), and sin (without). The rules (regular expressions) are processed in order, and their purpose is to negate the nearest word to the negation marker using only the information on the text, e.g., avoiding mainly pronouns and articles. For example, in the sentence El coche no es bonito (The car is not nice), the negation marker no and not (for English) is attached to its adjective no_bonito (not_nice).
After text-transformations, it is needed to represent the text in suitable form in order to use a traditional classifier such as SVM. It was decided to select the well known vector representation of a text given its simplicity and powerful representation. Particularly, it is used the Term Frequency-Inverse Document Frequency which is a well-known weighting scheme in NLP. TF-IDF computes a weight that represents the importance of words or terms inside a document in a collection of documents, i.e., how frequently they appear across multiple documents. Therefore, common words such as the and in, which appear in many documents, will have a low score, and words that appear frequently in a single document will have high score. This weighting scheme selects the terms that represent a document.
The model selection, sometimes called hyper-parameter optimization, is essential to ensure the performance of a sentiment classifier. In particular, our approach is highly parametric; in fact, we use such property to adapt to several languages. Table TABREF2 summarizes the parameters and their valid values. The search space contains more than 331 thousand configurations when limited to multilingual and language independent parameters; while the search space reaches close to 4 million configurations when we add our three language-dependent parameters. Depending on the size of the training set, each configuration needs several minutes on a commodity server to be evaluated; thus, an exhaustive exploration of the parameter space can be quite expensive making the approach useless in practice. To tackle the efficiency problems, we perform the model selection using two hyper-parameter optimization algorithms.
The first corresponds to Random Search, described in depth in BIBREF14 . Random search consists on randomly sampling the parameter space and select the best configuration among the sample. The second algorithm consists on a Hill Climbing BIBREF15 , BIBREF16 implemented with a memory to avoid testing a configuration twice. The main idea behind hill climbing H+M is to take a pivoting configuration, explore the configuration's neighborhood, and greedily moves to the best neighbor. The process is repeated until no improvement is possible. The configuration neighborhood is defined as the set of configurations such that these differ in just one parameter's value. This rule is strengthened for tokenizer (see Table TABREF2 ) to differ in a single internal value not in the whole parameter value. More precisely, let INLINEFORM0 be a valid value for tokenizer and INLINEFORM1 the set of valid values for neighborhoods of INLINEFORM2 , then INLINEFORM3 and INLINEFORM4 for any INLINEFORM5 .
To guarantee a better or equal performance than random search, the H+M process starts with the best configuration found in the random search. By using H+M, sample size can be set to 32 or 64, as rule of thumb, and even reach improvements in most cases (see § SECREF4 ). Nonetheless, this simplification and performance boosting comes along with possible higher optimization times. Finally, the performance of each configuration is obtained using a cross-validation technique on the training data, and the metrics are usually used in classification such as: accuracy, score INLINEFORM0 , and recall, among others.
Nowadays, there are several international competitions related to text mining, which include diverse tasks such as: polarity classification (at different levels), subjectivity classification, entity detection, and iron detection, among others. These competitions are relevant to measure the potential of different proposed techniques. In this case, we focused on polarity classification task, hence, we developed a baseline method with an acceptable performance achieved in three different contests, namely, TASS'15 (Spanish) BIBREF17 , SemEval'15-16 (English) BIBREF18 , BIBREF19 , and SENTIPOLC'14 (Italian) BIBREF20 . In addition, our approach was tested with other languages (Arabic, German, Portuguese, Russian, and Swedish) to show that is feasible to use our framework as basis for building more complex sentiment analysis systems. From these languages, datasets and results can be seen in BIBREF21 , BIBREF3 and BIBREF2 .
Table TABREF15 presents the details of each of the competitions considered as well as the other languages tested. It can be observed, from the table, the number of examples as well as the number of instances for each polarity level, namely, positive, neutral, negative and none. The training and development (only in SemEval) sets are used to train the sentiment classifier, and the gold set is used to test the classifier. In the case there dataset was not split in training and gold (Arabic, German, Portuguese, Russian, and Swedish) then a cross-validation (10 folds) technique is used to test the classifier. The performance of the classifier is presented using different metrics depending the competition. SemEval uses the average of score INLINEFORM0 of positive and negative labels, TASS uses the accuracy and SENTIPOLC uses a custom metric (see BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 ).
We tested our framework on two kinds of datasets. On one hand, we compare our performance on three languages having well known sentiment analysis contests; here, we compare our work against competitors of those challenges. On the other hand, we selected five languages without popular opinion mining contests; for these languages, we compare our approach against research works reporting the used corpus.
Figure FIGREF17 shows the performance on four contests, corresponding to three different languages. The performance corresponds to the multilingual set of features, i.e., we do not used language-dependent techniques.
Figures UID18 - UID21 illustrates the results on each challenge, all competitors are ordered in score's descending order (higher is better). The achieved performance of our approach is marked with a horizontal line on each figure. Figure UID22 briefly describes each challenge and summarizes our performance on each contest; also, we added three standard measures to simplify the insight's creation of the reader.
The winner method in SENTIPOLC'14 (Italian) is reported in BIBREF22 . This method uses three groups of features: keyword and micro-blogging characteristics, Sentiment Lexicons, SentiWordNet and MultiWordNet, and Distributional Semantic Model (DSM) with a SVM classifier. In contrast with our method, in BIBREF22 three external sentiment lexicons dictionaries were employed; that is, external information.
In TASS'15 (Spanish) competition, the winner reported method was BIBREF23 , which proposed an adaptation based on a tokenizer of tweets Tweetmotif BIBREF24 , Freeling BIBREF25 as lemmatizer, entity detector, morphosyntactic labeler and a translation of the Afinn dictionary. In contrast with our method, BIBREF23 employs several complex and expensive tools. In this task we reached the fourteenth position with an accuracy of INLINEFORM0 . Figure UID19 shows the B4MSA performance to be over two thirds of the competitors.
The remaining two contests correspond to the SemEval'15-16. The B4MSA performance in SemEval is depicted in Figures UID20 and UID21 ; here, B4MSA does not perform as well as in other challenges, mainly because, contrary to other challenges, SemEval rules promotes the enrichment of the official training set. To be consistent with the rest of the experiments, B4MSA uses only the official training set. The results can be significantly improved using larger training datasets; for example, joining SemEval'13 and SemEval'16 training sets, we can reach INLINEFORM0 for SemEval'16, which improves the B4MSA's performance (see Table FIGREF17 ).
In SemEval'15, the winner method is BIBREF26 , which combines three approaches among the participants of SemEval'13, teams: NRC-Canada, GU-MLT-LT and KLUE, and from SemEval'14 the participant TeamX all of them employing external information. In SemEval'16, the winner method was BIBREF27 is composed with an ensemble of two subsystems based on convolutional neural networks, the first subsystem is created using 290 million tweets, and the second one is feeded with 150 million tweets. All these tweets were selected from a very large unlabeled dataset through distant supervision techniques.
Table TABREF23 shows the multilingual set of techniques and the set with language-dependent techniques; for each, we optimized the set of parameters through Random Search and INLINEFORM0 (see Subsection SECREF14 ). The reached performance is reported using both cross-validation and the official gold-standard. Please notice how INLINEFORM1 consistently reaches better performances, even on small sampling sizes. The sampling size is indicated with subscripts in Table TABREF23 . Note that, in SemEval challenges, the cross-validation performances are higher than those reached by evaluating the gold-standard, mainly because the gold-standard does not follow the distribution of training set. This can be understood because the rules of SemEval promote the use of external knowledge.
Table TABREF24 compares our performance on five different languages; we do not apply language-dependent techniques. For each comparison, we took a labeled corpus from BIBREF3 (Arabic) and BIBREF21 (the remaining languages). According to author's reports, all tweets were manually labeled by native speakers as pos, neg, or neu. The Arabic dataset contains INLINEFORM0 items; the other datasets contain from 58 thousand tweets to more than 157 thousand tweets. We were able to fetch a fraction of the original datasets; so, we drop the necessary items to hold the original class-population ratio. The ratio of tweets in our training dataset, respect to the original dataset, is indicated beside the name. As before, we evaluate our algorithms through a 10-fold cross validation.
In BIBREF3 , BIBREF2 , the authors study the effect of translation in sentiment classifiers; they found better to use native Arabic speakers as annotators than fine-tuned translators plus fine-tuned English sentiment classifiers. In BIBREF21 , the idea is to measure the effect of the agreement among annotators on the production of a sentiment-analysis corpus. Both, on the technical side, both papers use fine tuned classifiers plus a variety of pre-processing techniques to prove their claims. Table TABREF24 supports the idea of choosing B4MSA as a bootstrapping sentiment classifier because, in the overall, B4MSA reaches superior performances regardless of the language. Our approach achieves those performance's levels since it optimizes a set of parameters carefully selected to work on a variety of languages and being robust to informal writing. The latter problem is not properly tackled in many cases.
We presented a simple to implement multilingual framework for polarity classification whose main contributions are in two aspects. On one hand, our approach can serve as a baseline to compare other classification systems. It considers techniques for text representation such as spelling features, emoticons, word-based n-grams, character-based q-grams and language dependent features. On the other hand, our approach is a framework for practitioners or researchers looking for a bootstrapping sentiment classifier method in order to build more elaborated systems.
Besides the text-transformations, the proposed framework uses a SVM classifier (with linear kernel), and, hyper-parameter optimization using random search and H+M over the space of text-transformations. The experimental results show good overall performance in all international contests considered, and the best results in the other five languages tested.
It is important to note that all the methods that outperformed B4MSA in the sentiment analysis contests use extra knowledge (lexicons included) meanwhile B4MSA uses only the information provided by each contests. In future work, we will extend our methodology to include extra-knowledge in order to improve the performance.
We would like to thank Valerio Basile, Julio Villena-Roman, and Preslav Nakov for kindly give us access to the gold-standards of SENTIPOLC'14, TASS'15 and SemEval 2015 & 2016, respectively. The authors also thank Elio Villaseñor for the helpful discussions in early stages of this research.

Figure and table captions:
Table 1: Parameter list and a brief description of the functionality
Table 3: Datasets details from each competition tested in this work
Figure 1: The performance listing in four difference challenges. The horizontal lines appearing in a) to d) correspond to B4MSA’s performance. All scores were computed using the official gold-standard and the proper score for each challenge.
Table 4: B4MSA’s performance on cross-validation and gold standard. The subscript at right of each score means for the random-search’s parameter (sampling size) needed to find that value.
Table 5: Performance on multilingual sentiment analysis (not challenges). B4MSA was restricted to use only the multilingual set of parameters.

Questions about this paper:
How many texts/datapoints are in the SemEval, TASS and SENTIPOLC datasets?
In which languages did the approach outperform the reported results?

Answers:
Total number of annotated data:
Semeval'15: 10712
Semeval'16: 28632
Tass'15: 69000
Sentipol'14: 6428
Arabic, German, Portuguese, Russian, Swedish