How to Clean Your Text Data with Python | Towards Data Science

IntroductionCleaning Text DataSummaryReferences
Introduction
Cleaning Text Data
Summary
References
It should be no surprise that data is most of the time, messy, unorganized, and difficult to deal with. As you work your way into data science from educational practice, you will see that most data is obtained from multiple sources, multiple queries, and that can lead to some unclean data. In some or most situations, you will have to come up with the dataset that will ultimately be used to train your model. There are a few articles out there that focus on numeric data, but I want the focus of this article to be on text data mainly, which coincides with natural language processing. With that being said, here is a simple way to clean your text data in Python, as well as when it would be useful. I will be using the popular dataset from TMBDF 5000 Movie Dataset [2], so that you can follow along.
Removing Stop Words
Another way to interpret ‘stop words’ is by removing unnecessary text. However, it is important to note that when you do use the common stopwords library, you may be removing words that you actually want to keep in. That is why you should think about the list of words that you want to remove first. Some common examples of stop words are ‘the’, ‘of’, etc. The reason you would want to remove these words is if you want to keep the main subject of the words, phrase, sentence, etc. An example would be ‘there was the time where they went to the store in July for the holiday party’. If you removed stop words and a few more unnecessary words, you would get just time, July, holiday, party leftover.
With that being said, let us look at how we can remove some stop words from movie titles:
import pandas as pdimport nltk.corpusnltk.download(‘stopwords’)from nltk.corpus import stopwordsdf = pd.read_csv('...path/tmdb_5000_movies.csv')stop_words = stopwords.words('english')df['clean_title'] = df['title'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop_words)]))
In the code above, we are importing necessary libraries, and then reading our data into a pandas dataframe. From there, we assign the stop words that we will remove from the text of the ‘title’ column, which will show their respective effect in the ‘clean_title’ column. The output is what we see below. In rows 1, 3, and 8, are where the stopwords have been removed as you can see by the before and after, side by side.
In addition to the stop words library from nltk, you can add additional stop words ‘by hand’. In order to do this function, you can simply add a list of strings to stop_words. For example, if we wanted to remove the text ‘3’, as it is not a number in this case, we could add that to a list, as well as the words ‘At’, and the letter ‘v’. It would work like the following below:
stop_words = stopwords.words(‘english’) + [‘At’, ‘v’, ‘3’]# apply the same code as above but assign# a new column to see the differencesdf['clean_title_v2'] = df['title'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop_words)]))
In the screenshot below, you can see how the values in the added list of strings have been removed by the lambda function.
Now that we have shown one way of cleaning text data, let’s discuss the possible applications where this process would be useful to data scientists:
Removing unnecessary words so that you can perform stemming — where you can isolate the root of the words that are left after removal
Similar to above, you can work to isolate just the lemma of the word
Only keeping the necessary words can allow you to more easily tag parts of speech in your data, for example, if you tagged adjectives only, and used that text in your data for your model — like ‘beautiful, ‘amazing’, ‘loud’ would be left to use for predicting your target variable of a movie review. A model would be able to recognize the sentiment more easily from adjectives to discern if a movie review is good or bad, or what the movie would need to improve upon
Creating a summary dashboard from the updated text’s main words, for example, stripping away the numbers in ‘Spider-Man 1’, ‘Spider-Man 2’, and ‘Spider-Man 3’, would allow performing analysis on all Spider-Man movies in general like money made, or the average review rating for those movies
Easier for topic modeling — finding common topics between large values of text
Creating a smaller amount of text so that your modeling can become quicker and cheaper to train and predict
Can be used for removing erroneous text, for example, if people are manually typing in a response, you can clean their responses, as well as if their input is from voice-to-text, you can then automate the removal of words that are commonly interpreted improperly by voice
There are of course more reasons to remove stop words, and clean your text data in general. It is also important to keep in mind that there are emerging algorithms that can handle text data especially well, like CatBoost.
As you can see, just one part of cleaning data can lay the groundwork for further cleaning and processing of data. Hopefully, this change in your data will ultimately make your model more robust and accurate. The examples I gave were small, but you could see how applying this to data where hundreds of sentences could be much more useful.
To summarize, here is how you remove stop words from your text data:
* import libraris* import your dataset* remove stop words from the main library* add individual stop words that are unique to your use case
UPDATE: the word The was not removed as it should be because it was uppercased, so make sure to lowercase all your text before cleaning it. Thank you for the callout, Miia Rämö!
I hope you found my article both interesting and useful. Please feel free to comment down below if you agree with this method of removing stop words — why or why not? Do you prefer a different library? If you do, feel free to comment on the reasons why you prefer one over the other. What disadvantages are there to this library? Do you know of an even newer library that could beat nltk out in terms of efficiency, ease of use, and speed? What other ways do you clean text data? Do you happen to not use a library at all and just remove a specific list you have curated? Thank you for reading my article!
Please feel free to check out my profile and other articles, Matt Przybyla, as well as reach out to me on LinkedIn.
[1] Photo by JESHOOTS.COM on Unsplash, (2019)
[2] Kaggle, TMBDF 5000 Movie Dataset, (2021 — accessed)
[3] Photo by Marylou Fortier on Unsplash, (2020)
[4] M.Przybyla, cleaning text example, (2021)
[5] M.Przybyla, cleaning text example plus adding individual stop words, (2021)