# Evaluating the Effectiveness of Text Pre-Processing in Sentiment Analysis
## 3 Materials and Methods
### Pre-Processing Components  
To assess the impact of pre-processing on the accuracy of Twitter sentiment classifiers, we implemented the next 12 pre-processing components. We will describe them first, and we will later discuss the order in which we execute them. Lowercasing: We convert to lowercase all the characters in the dataset, as opposed to capitalising proper names, initials, and words at the start of the sentence. Capitalisation is complex in Twitter, because users tend to avoid capital letters, as this lends a tone of informality to the conversation that makes the messages more speech-like [50]. Therefore, lowercasing everything seems to be the most practical approach. Removal of URLs and Twitter features: We remove URLs entirely, because we cannot estimate the sentiment of an online resource--web page, picture, or anything else associated with a URL. We also remove the hashtag character, but not the keywords or  
Figure 3: Dataset word cloud based on TF-IDF to determine the most characteristic words. Figure 2: Distribution of positive, negative, and neutral tweets in the experimental dataset per hour. The tweets included in the plot are those annotated with a high confidence sentiment value. phrases comprised within the hashtags--for example, we remove the # character from #happy, but not the word _happy_, because it can have a sentiment-related connotation. We also remove the characters RT at the beginning of the tweets. RT is used to indicate the _re-tweeting_--or re-posting--of someone else's tweets. Removal of unnecessary spaces: Identifying spaces between characters is critical, as spaces are considered word boundaries. Regrettably, splitting a character sequence where spaces occur can also split what should be regarded as a single \"token\". This happens commonly with names--for example, _New York, The Netherlands_, and _Cote d'Ivoire_--but also with a number of borrowed foreign phrases--for instance, _au fait_. Although word segmentation remains an issue, we handle it by removing spaces that only increase the length of the text without adding any meaningful value. Removal of punctuation: Eliminating punctuation before performing sentiment analysis is common. However, some punctuation characters are related to emoticons which express sentiment; thus, their removal may reduce the accuracy of the classification. Indeed, punctuation sequences such as :), :D, :), or <3 are references to emoticons that convey sentiment. As we want to convert the emoticons into words, we remove the punctuation after handling the emoticons. Other researchers have recommended the same approach to speed up the analysis and improve the performance--for instance, see [PERSON]'s work on dimensionality reduction [51]. Negation handling: Negations are words such as _no, not yet_, and _never_, which express the opposite meaning of words or phrases. For the purpose of sentiment analysis, it is common to replace a negation followed by a word with an antonym of the word. For example, the phrase _not good_ is replaced with _bad_, which is an antonym of _good_. Consequently, a sentence such as _the car is not good_ is transformed into _the car is bad_. However, certain negative words, such as _no, not_, and _never_, and negative contractions, such as _mustn't_, _couldn't_, and _doesn't_, are often part of stop-word lists. Thus, we replace all negative words and contractions with nnot, and then we correct this issue after stop-word removal as part of the misspelling correction [52]. Stop-word removal: Extremely common words which are of little value in matching an information need--such as prepositions, definite and indefinite articles, pronouns, and conjunctions--are known as _stop-words_[53]. These words are typically removed to reduce the amount of processing involved in the analysis [54]. The general strategy for constructing a stop-word list is to sort all the terms in the corpus by frequency and then add the most frequent terms, often hand-filtered for their semantic content, to a stop-word list.