NLP Text Preprocessing: Steps, tools, and examples | by Viet Hoang Tran Duong | Towards Data Science

Text data is everywhere, from your daily Facebook or Twitter newsfeed to textbooks and customer feedback. Data is the new oil, and text is an oil well that we need to drill deeper. Before we can actually use the oil, we must preprocess it so it fits our machines. Same for data, we must clean and preprocess the data to fit our purposes. This post will include a few simple approaches to cleaning and preprocessing text data for text analytics tasks.
We will model the approach on the Covid-19 Twitter dataset. There are 3 major components to this approach: First, we clean and filter all non-English tweets/texts as we want consistency in the data.Second, we create a simplified version for our complex text data.Finally, we vectorize the text and save their embedding for future analysis.
If you want to check out the code: feel free to check out the code for part 1, part 2, and part 3 embedded here. You can also check the whole project blogpost and codes here.
First, to simplify the text, we want to standardize our text into only English characters. This function will remove all non-English characters.
def clean_non_english(txt):    txt = re.sub(r'\W+', ' ', txt)    txt = txt.lower()    txt = txt.replace("[^a-zA-Z]", " ")    word_tokens = word_tokenize(txt)    filtered_word = [w for w in word_tokens if all(ord(c) < 128 for c in w)]    filtered_word = [w + " " for w in filtered_word]    return "".join(filtered_word)
We can even do better by removing the stopwords. Stopwords are common words that appear in English sentences without contributing much to the meaning. We will use the nltk package to filter the stopwords. As our main task is visualizing the common theme of tweets using word cloud, this step is necessary to avoid common words like “the,” “a,” etc. However, if your tasks require full sentence structure, like next word prediction or grammar check, you can skip this step.
import nltknltk.download('punkt') # one time executionnltk.download('stopwords')from nltk.corpus import stopwordsstop_words = set(stopwords.words('english'))def clean_text(english_txt):    try:       word_tokens = word_tokenize(english_txt)       filtered_word = [w for w in word_tokens if not w in stop_words]       filtered_word = [w + " " for w in filtered_word]       return "".join(filtered_word)    except:       return np.nan
For tweets, there is a special feature we need to consider before cleaning: mentions. Your data might have special features like this (or not), and this is case-by-case and not a universal requirement. Hence, know your data well before blindly clean and preprocess!
def get_mention(txt):    mention = []    for i in txt.split(" "):        if len(i) > 0 and i[0] == "@":            mention.append(i)            return "".join([mention[i] + ", " if i != len(mention) - 1 else mention[i] for i in range(len(mention))]
Before, we clean the non-English characters. Now, we remove the non-English texts (semantically). Langdetect is a python package that allows for checking the language of the text. It is a direct port of Google’s language detection library from Java to Python.
from langdetect import detectdef detect_lang(txt):    try:        return detect(txt)    except:        return np.nan
We then filter out all columns then is not “en” language.
All of the codes for part 1 can be found here.
For numerical data, good processing methods are scaling, standardizing, and normalizing. This resource is beneficial in understanding and applies these methods to your data. Within this post's scope, I will not discuss further as other resources have done a great job doing so.
For categorical data, there are numerous approaches. The two nominal approaches are Label Encoder (assigning a distinct number for each label) and One hot encoding (represent with a vector of 0s and 1). More details on the approach to these categorical values can be found here. This resource is very substantial with more types of encoding than these two I mentioned.
This post will introduce some ways to reduce the complexity of data, especially location data. In my dataset, there is a column for location, with the address of the author. However, I cannot perform much analysis with these raw data as they are too messy and complex (having city, county, state, country). Hence, we can standardize the text and reduce it to the “country” level (or state if you are interested). A package to deal with location data is geopy. It can identify the correct address and reformat these locations to standard form. Then, you can choose to keep whichever information you need. For me, the state, country is decent enough.
from geopy.geocoders import Nominatimgeolocator = Nominatim(user_agent="twitter")def get_nation(txt):    try:        location = geolocator.geocode(txt)        x = location.address.split(",")[-1]        return x    except:        return np.nan
Python is awesome with so many packages. I believe you can always find something for your specific need, as I do with my messy location data. Good luck with simplifying these messy data.
The code for part 2 can be found here.
Text vectorization is converting text into vectors of values to represent their meanings. Earlier days, we have one hot encoding method with a vector with a size of our vocabulary, and value 1 wherever the text appear and 0s elsewhere. Nowadays, we have more advanced methods like spacy, GloVe, or even bert embedding. For the scope of this project, I will introduce you to GloVe in python and Jupiter notebooks.
First, we download the embeddings. You can download it manually here or do it directly in your notebooks.
!wget http://nlp.stanford.edu/data/glove.6B.zip!unzip glove*.zip
Then, we create a function to vectorize each data point. The sentence is the mean representation of each word. For empty sentences, we default it to a zeros vector.
def vectorize(value, word_embeddings, dim = 100):    sentences = value.to_list()    sentence_vectors = []    for i in sentences:        if len(i) != 0:            v = sum([word_embeddings.get(w, np.zeros((dim,))) for w in      i.split()])/(len(i.split())+0.001)        else:            v = np.zeros((dim,))        sentence_vectors.append(v)    sentence_vectors = np.array(sentence_vectors)    return sentence_vectors
Finally, we vectorize the whole dataset and save the vectorized numpy array as a file, so we don’t have to go through this process again every time running the code. The vectorized version will be saved as a numpy array in the form of .npy files. Numpy package is convenient in storing and dealing with massive array data.
As my personal standard practice, I try to save all data as separate files after each part to evaluate the data and alter my code more flexibly.
def vectorize_data(data = data, value = 'english_text', dim = 100):    # Extract word vectors    word_embeddings = {}    f = open('glove.6B.{}d.txt'.format(str(dim)), encoding='utf-8')    for line in f:        values = line.split()        word = values[0]        coefs = np.asarray(values[1:], dtype='float32')        word_embeddings[word] = coefs    f.close()    text_vec = vectorize(data[value], word_embeddings, dim)    np.save("vectorized_{}.npy".format(str(dim)), text_vec)    print("Done. Data:", text_vec.shape)    return True
The code for part 3 is here.
Data preprocessing, specifically with text, can be a very troublesome process. A big part of your machine learning engineer workflow will be for these cleaning and formatting data (lucky you if your data is already perfectly clean & kudos to all data engineers for making that happen).
All the code in this post is very abstract and can be applied to many data projects (you only need to change to the name of columns, and all should work properly). In the notebook, I also added exception functions to handle failure cases, ensuring that your code will not crash midway. I hope it helps you with your projects, as much as it helps me in mine.
You can check out the project blog post here.
Good luck with your projects, and let me know if anything needs to change and improve! Thank you, and see you in the next posts!
References:Hale, J. (2020, February 21). Scale, Standardize, or Normalize with Scikit-Learn. Retrieved from https://towardsdatascience.com/scale-standardize-or-normalize-with-scikit-learn-6ccc7d176a02KazAnova. (2017, September 13). Sentiment140 dataset with 1.6 million tweets. Retrieved from https://www.kaggle.com/kazanova/sentiment140Preda, G. (2020, August 30). COVID19 Tweets. Retrieved from https://www.kaggle.com/gpreda/covid19-tweetsRoy, B. (2020, April 25). All about Categorical Variable Encoding. Retrieved from https://towardsdatascience.com/all-about-categorical-variable-encoding-305f3361fd02