A Guide To Cleaning Text in Python | by Kurtis Pykes | Towards Data Science

Text is a form of unstructured data. According to Wikipedia, unstructured data is described as “information that either does not have a pre-defined data model or is not organized in a pre-defined manner.” [Source: Wikipedia].
Unfortunately, computers aren’t like humans; Machines cannot read raw text in the same way that we humans can. When we are working with textual data, we cannot go from our raw text straight to our Machine learning model. Instead, we must follow a process of first cleaning the text then encoding it into a machine-readable format.
Let’s cover some ways we can clean text — In another post, I’ll cover ways we can encode text.
When we write, we capitalize various words in our sentence/paragraph for different reasons. For example, we start a new sentence with a capital letter or if something is a noun, we would capitalize the first letter to indicate we are talking about a place/person, etc.
To a human, we can read a text and intuitively tell that “The” which is used at the beginning of a sentence is the same word as “the” which is found later in the middle of the sentence, however, a computer cannot — “The” and “the” are seen as 2 different words by a machine.
Therefore, it’s important to normalize the case of our words so that every word is in the same case and the computer doesn’t process the same word as 2 different tokens.
# Python Exampletext = "The UK lockdown restrictions will be dropped in the summer so we can go partying again!" # lowercasing the texttext = text.lower()print(text)>>>> the uk lockdown restrictions will be dropped in the summer so we can go partying again!
In the majority of natural language tasks, we want our machine learning models to identify the words within a document that provide value to the document. For example in a sentiment analysis task, we want to find the word (or words) that tip the sentiment of the text in one direction or the other.
In the English language (I believe the same would be true for most languages but don’t quote me), there are words that are used more frequently than other words in the language but they do not necessarily add more value to a sentence, hence it is safe to say that we can ignore them by removing the from our text.
Note: Removing stopwords is not always the best idea!
# Importing the libraries import nltkfrom nltk.corpus import stopwordsnltk.download("stopwords")stop_words = set(stopwords.words("english"))print(stop_words)>>>> {'over', 'is', 'than', 'can', 'these', "isn't", 'so', 'my', 'each', 'an', 'between', 'through', 'up', 'where', 'hadn', 'very', "you'll", 'while', "weren't", 'too', 'doesn', 'only', 'needn', 'has', 'just', 'd', 'some', 'into', 've', 'didn', 'further', 'why', 'mightn', 'and', 'haven', 'own', "mightn't", 'during', 'both', 'me', 'shan', "doesn't", 'theirs', 'herself', 'the', 'few', 'our', 'its', 'yourself', 'under', 'at', "you've", 're', 'themselves', 'y', 'ma', 'because', 'him', 'above', 'such', 'we', "wouldn't", 'of', 'from', 'hers', 'nor', "shouldn't", 'a', 'hasn', 'them', 'myself', 'this', 'being', 'your', 'those', 'i', 'if', 'couldn', 'not', 'will', 'it', 'm', 'to', 'isn', 'aren', 'when', 'o', 'about', 'their', 'more', 'been', "needn't", 'had', 'll', 'most', 'against', 'once', 'how', "didn't", "shan't", 'there', 'all', "should've", 'he', "don't", 'she', 'which', 'below', 'on', 'no', 'yourselves', "wasn't", 'shouldn', 'by', 'be', 'have', 'does', "aren't", 'itself', 'same', 'should', 'in', 'before', 'am', "won't", 'having', "you'd", 'mustn', 'for', "that'll", 'that', "couldn't", 'wasn', 'won', "hasn't", 'as', 'until', 'wouldn', "mustn't", 'his', 'ain', "you're", 'out', "she's", 'other', 'are', 't', 'you', 'off', 'yours', 'ourselves', 'himself', 'down', "haven't", 'ours', 'now', "hadn't", 'do', 's', 'her', 'with', "it's", 'then', 'weren', 'any', 'after', 'whom', 'what', 'who', 'but', 'again', 'here', 'did', 'doing', 'were', 'they', 'was', 'or', 'don'}# example texttext = "The UK lockdown restrictions will be dropped in the summer so we can go partying again!"# removing stopwordstext = " ".join([word for word in text.split() if word not in stop_words])print(text)>>>> uk lockdown restrictions dropped summer go partying again!
ASCII formats emojis and other non-ASCII characters as Unicode. Essentially, Unicode is a universal character encoding standard in which each character and symbol in all languages are assigned a code. Unicode is required because it is the only encoding standard that allows us to retrieve or join data using a variety of different languages but the issue is... It’s unreadable in ASCII format.
Note: example code from Python Guides
# creating a unicode stringtext_unicode = "Python is easy \u200c to learn" # encoding the text to ASCII formattext_encode = text_unicode.encode(encoding="ascii", errors="ignore")# decoding the texttext_decode = text_encode.decode()# cleaning the text to remove extra whitespace clean_text = " ".join([word for word in text_decode.split()])print(clean_text)>>>> Python is easy to learn.
Depending on the type of data we are dealing with, we may face various challenges that add noise. For instance, if we are working with data from Twitter, it’s not going to be unusual to find various hashtags and mentions — which refers to a tweet that contains another user's username in Twitter lingo.
If these features are not valuable for the problem that we are attempting to solve then we are better off removing them from our data. However, since we cannot depend on a defined character in many of the instances, we can leverage the power of a pattern-matching tool called Regex to aid us.
import re # removing mentions text = "You should get @BlockFiZac from @BlockFi to talk about bitcoin lending, stablecoins, institution adoption, and the future of crypto"text = re.sub("@\S+", "", text)print(text)>>>> You should get  from  to talk about bitcoin lending, stablecoins, institution adoption, and the future of crypto------------------------------------------------------------------# remove market tickerstext = """#BITCOIN LOVES MARCH 13th A year ago the price of Bitcoin collapsed to $3,800 one of the lowest levels in the last 4 years. Today, exactly one year later it reaches the new all-time high of $60,000 Thank you Bitcoin for always making my birthday exciting"""text = re.sub("\$", "", text)print(text)>>>> #BITCOIN LOVES MARCH 13th A year ago the price of Bitcoin collapsed to  3,800 one of the lowest levels in the last 4 years. Today, exactly one year  later it reaches the new all-time high of 60,000 Thank you Bitcoin for  always making my birthday exciting------------------------------------------------------------------# remove urlstext = "Did someone just say “Feature Engineering”? https://buff.ly/3rRzL0s"text = re.sub("https?:\/\/.*[\r\n]*", "", text)print(text)>>>> Did someone just say “Feature Engineering”?------------------------------------------------------------------# removing hashtags text = """.#FreedomofExpression which includes #FreedomToProtest should be the cornerstone of any democracy. I’m looking forward to speaking in the 2 day debate on the #PoliceCrackdownBill & explaining why I will be voting against it."""text = re.sub("#", "", text)print(text)>>>> .FreedomofExpression which includes FreedomToProtest should be the  cornerstone of any democracy. I’m looking forward to speaking in the 2 day  debate on the PoliceCrackdownBill & explaining why I will be voting against it.------------------------------------------------------------------# remove punctuationimport string text = "Thank you! Not making sense. Just adding, lots of random punctuation."punct = set(string.punctuation) text = "".join([ch for ch in tweet if ch not in punct])print(text)>>>> Thank you Not making sense Just adding lots of random punctuation
When we are doing an NLP task, we may want the computer to be able to understand that “walked”, “walk” and “walking” are all merely different tenses of the same word, otherwise, they’d be treated differently. Stemming and Lemmatization are both techniques used to normalize text in NLP — to further simplify this definition, we are simply reducing a word to its core root.
Definitions according to Wikipedia:
Stemming (Link) — In linguistic morphology and information retrieval, stemming is the process of reducing inflected words to their word stem, base, or root form — generally a written word form.
Lemmatization (Link) — Lemmatisation in linguistics is the process of grouping together the inflected forms of a word so they can be analyzed as a single item, identified by the word’s lemma, or dictionary form.
Though the definitions are quite similar, the way each technique goes about reducing a word is quite different (something I may explore in another article) which can mean the results aren’t consistent between the two techniques.
import nltkfrom nltk.stem.porter import PorterStemmerfrom nltk.stem import WordNetLemmatizerwords = ["walk", "walking", "walked", "walks", "ran", "run", "running", "runs"]-----------------------------------------------------------------# example of stemmingstemmer = PorterStemmer()for word in words:     print(word + " ---> " + stemmer.stem(word))>>>> walk ---> walk     walking ---> walk     walked ---> walk     walks ---> walk     ran ---> ran     run ---> run     running ---> run     runs ---> run------------------------------------------------------------------# example of lemmatizationlemmatizer = WordNetLemmatizer()for word in words:    print(word + " ---> " + lemmatizer.lemmatize(word))>>>> walk ---> walk      walking ---> walking      walked ---> walked      walks ---> walk      ran ---> ran      run ---> run      running ---> running      runs ---> run
In this tutorial, we covered how to clean text in Python.
Specifically, we covered:
Why we clean text
Different ways to clean text
Thank you for reading! Connect with me on LinkedIn and Twitter to stay up to date with my posts about Data Science, Artificial Intelligence, and Freelancing.