# Evaluating the Effectiveness of Text Pre-Processing in Sentiment Analysis
## 3 Materials and Methods
### Pre-Processing Components
The terms in this list are then discarded from any further processing [55]. In the case of sentiment analysis, the utilisation of various stop-word lists is reported in the literature. We have chosen the list used by the NLTK library [56], which is a well-regarded text mining tool. Emoticons and emojis translation: An _emoticon_ is a representation of a facial expression, such as a smile or from, formed by a combination of keyboard characters. An _emoji_ is a small digital image or icon used to express an idea or emotion [57]. Previous studies have shown that emoticons and emojis play a role in both building sentiment lexicons and training classifiers for sentiment analysis [58; 59]. [PERSON] and [PERSON] [58] have concluded that emoticons are strong signals of sentiment polarity. Acknowledging this, we translate emoticons and emojis into their corresponding words using _emot_, the _Open-Source Emoticons and Emoji Detection Library_[60]. Then, :D, which translates into laughing, increases the positive score of a tweet, whereas :-(, which translates into a from, increases the negative score. Acronym and slang expansion: Computer-mediated communication has generated slang and acronyms often referred to as _microtext_[61]. Examples of microtext are expressions such as _\"c u 2 morrow\"_ (see you tomorrow), which are not found in standard English but are widely seen in short message service (SMS) texts, tweets, and Facebook updates. In addition, [PERSON] et al. [62] have explained how important it is to expand acronyms to reveal concealed messages with sentiment repercussions. Additionally, [PERSON] et al. [61] have shown that acronym expansion improves sentiment analysis accuracy by 4%. Therefore, replacing microtext by conventional meanings is indispensable. We perform this via the _SMS and Slang Translator_ dictionary [63]. Spelling correction: Language features are likely to be missed due to misspellings. Using tools that automatically correct misspellings enhances classification effectiveness [64]. Although no spelling correctors is perfect, some of them have demonstrated a reasonably good accuracy [52]. While [PERSON] [51] made use of _AutoMap_[65], a text mining tool developed by CASOS at Carnegie Mellon, we favoured the Python library pysellchecker [66], which employs the _Levenshtein Distance_ algorithm [67]. Tokenisation: This is the process of splitting a piece of text into its parts, called _tokens_, while disposing of certain characters and sequences which are not considered useful [53]. We tokenised the tweets in the dataset using the TweetTokenizer from the NLTK package [56], which we are also using for stop-word removal. Short-word removal: To avoid further noise, we remove all the words that consist of only one or two characters. Such words are unlikely to contribute to the analysis of sentiment and may even be typos. Lemmatisation: There are families of derivationally related words with similar meanings, such as president, presidency, and presidential. The goal of lemmatisation is to reduce inflectional and derivational forms of a word to a common base [53]. Lemmatisation employs vocabulary and morphological analysis to remove inflectional endings and return the base or dictionary form of a word, namely the _lemma_. If lemmatisation encountered the word _saw_, it would attempt to return either _see_ or _saw_ depending on whether it was used as a verb or a noun. Sentiment analysis has used lemmatisation in other languages--for example, Indonesian [68] and Vietnamese [69]. However, in the English language, lemmatisation has been mostly used in classification studies, where it has improved the performance, as indicated by [PERSON] et al. [70]. Thus, we opted to test the use of lemmatisation in sentiment analysis, too.