Pipeline For Text Data Pre-processing | by Casey Whorton | Towards Data Science

Skip ahead to the actual Pipeline section if you are more interested in that than learning about the quick motivation behind it: Text Pre Process Pipeline (halfway through the blog).
I’ve been looking at performing machine learning on text data but there are some data preprocessing steps that are unique to text data that I was not used to. Because of that, my Python code included a lot of transformation steps where I would wrangle with the data, fit a transformation, then transform the training data, transform the testing data, and then repeat this process for every type of transformation I wanted to do. I remembered reading that Python had a convenient way to wrap up transformations but never had a reason to look into it before now. Usually, I would perform something like a standardization scaling to numeric data or some dummy variable creation and that would be it. Now that I’m in a territory a little new to me, I wanted to keep these transformations straight and learn something new.
import numpy as npimport pandas as pdfrom scipy import sparse as spfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.feature_extraction.text import TfidfTransformerfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import LabelEncoderfrom sklearn.feature_selection import chi2from sklearn.feature_selection import SelectKBest
If you are new to using NLTK as I was about a year ago, this website will get you up and running, and is saved to my favorites because I use it as a reference pretty frequently when using NLTK.
# Import Natural language Toolkit example data and 'stopwords' setfrom nltk.corpus import movie_reviewsfrom nltk.corpus import stopwords
The ‘Movie Reviews’ data has 2000 movie reviews, each with a corresponding category (label) for whether it was negative (‘neg’) or positive (‘pos’).
docs = [(str(movie_reviews.raw(fileid)), category) for category in movie_reviews.categories() for fileid in movie_reviews.fileids(category)]
Just changing the ‘docs’ to a Pandas dataframe now, mostly because I am used to this format. Pipelines of data transformations can apply to more than just Pandas dataframes.
reviews = pd.DataFrame(docs)reviews.columns=('X','y')
The ‘category’ of a movie review is initially ‘neg’ or ‘pos’, changing here to 0 and 1, respectively:
bin_encoder=LabelEncoder()reviews.y=bin_encoder.fit_transform(reviews.y)
At this point, the reviews are in a dataframe, with the X column as the review text and the y column as the target variable, review category. This is more in the style of what I’m currently used to with machine learning projects. Here is a view of the first 5 observations:
reviews.head(5)
X y 0 plot : two teen couples go to a church party ,... 0 1 the happy bastard’s quick movie review \ndamn ... 0 2 it is movies like these that make a jaded movi... 0 3 “ quest for camelot “ is warner bros . ‘ firs... 0 4 synopsis : a mentally unstable man undergoing ... 0
What needs to happen is the X column needs to be tokenized, the stop-words removed, and for each observation to be vectorized in order for the machine to learn how to distinguish positive from negative reviews.
Both NLTK and the Scikit-Learn function CountVectorizer have built-in sets or lists of stopwords which basically serve as a bunch of words that we don’t really want hanging around in our data. Words like ‘a’, ‘of’, and ‘the’ are usually not useful and dominate other words in terms of how often they show up in a sentence or paragraph. Sometimes though, you want to add some custom stopwords of your own, so I do that here with the creation of my list, custom_stopwords:
mystopwords = (stopwords.words())custom_stopwords = ('the','an','a','my','0','''''','!','nt','?','??','?!','%','&','UTC','(UTC)')
Now that the movie reviews are in a dataframe and there is a list of stopwords that can be removed from the movie reviews, I’ll show the text pre process pipeline I used before and the one I made using Sci-kit Learn’s Pipeline.
The idea is to put all of the sentences into a big matrix with columns representing individual words and either a term-frequency or term-frequency-inverse-document-frequency as the observation’s values under a corresponding word.Usually, in a machine learning project, you’ll need to repeat the transformations that you do on the training data to the testing data, so I was explicit in saving the transformations. That is why I had a line to save the TfidfTransform and another line to actually transform the training data. Later, I used that TfidfTransform on the test dataset. I did the same with the CountVectorizer.
count_vect = CountVectorizer(stop_words=mystopwords,lowercase=True)X_train_counts = count_vect.fit_transform(X)tf_transformer = TfidfTransformer(use_idf=True).fit(X_train_counts)X_train_tfidf = tf_transformer.transform(X_train_counts)
Similarly, for selecting the top k features (words) of the dataset, I made sure to save the transform and apply it to the training and test datasets.
chi2(X_res,y_res)k=1000ch2_score = SelectKBest(chi2, k=k)toxic_feature_tran = ch2_score.fit(X,y)X_train_k = ch2_score.fit_transform(X, y)X_test_k = ch2_score.transform(X_test)
Not only was this a hassle, but if I had to repeat this process at any point in time then I would have to make new variables and be sure to keep all of these steps in order and not overwrite anything. It just wasn’t good coding. That is where the Pipeline comes in and cleans everything up.
This is the package you need to download and utilize if your machine learning projects start dealing with several transformations. For each of the transformations in Python that has a ‘fit_transform()’ method (and all of the ones I use here do), we can wrap them up in an actual pipeline that executes them all in order and even go back and view attributes of each of the transformations. Additionally, you can set parameters for each transformation and the syntax for that is in the link I just shared. If anything, using this pipeline has just cleaned up my code a lot and organized my thinking better.
Using these three transformations:
‘count vectorizer’ : Transformation from sentences to all lower-case words, stopwords removed, vectorized
‘chi2score’ : Transformation that selects top k features related to the target based on ChiSquare test statistics
‘tf_transformer’ : Transformation that transforms the vector of top features to tf-idf representation
The pipeline looks like this:
# Using a variable for the top k features to be selectedtop_k_features=1000text_processor = Pipeline([ ('count vectorizer',CountVectorizer(stop_words=mystopwords,lowercase=True)), ('chi2score',SelectKBest(chi2,k=top_k_features)), ('tf_transformer',TfidfTransformer(use_idf=True))])
Now, once this is fit to the training data, the text_preprocessor pipeline has the transform method that does all three of the included transformations in order to the data.
Like any other transformation with a fit_transform() method, the text_processor pipeline’s transformations are fit and the data is transformed. The proc_fit can be used to transform testing data in the same way.
proc_text = text_processor.fit_transform(reviews.X,reviews.y)proc_fit = text_processor.fit(reviews.X,reviews.y)
Returning the original words that ended up in the final 1000 words for a particular comment can still be accomplished by the following two steps:
Find the index of the top 1000 features returned from the ‘chi2score’ transformationFind the ‘feature names’, i.e. the words from the original text
Find the index of the top 1000 features returned from the ‘chi2score’ transformation
Find the ‘feature names’, i.e. the words from the original text
proc_fit.named_steps['chi2score'].get_support(indices=True)[616]
I picked the number 616 from the first transformed movie reviews words. Turns out that this corresponds to feature number 23078 from the original features. (Only 1000 features were left after the ‘chi2score’ transformation, and the numbers of the features were changed.)
Out[49]:23078
What is returned: The word ‘music’ was the word corresponding to feature number 23078 from the original movie review text data that was transformed into count vectors. It is now feature number 616 in the top 1,000 features. We can always just print the entire movie review and see if that word appears there.
proc_fit.named_steps['count vectorizer'].get_feature_names()[23078]'music'print(reviews.iloc[0,0])
Examining the first review, we can see that the word ‘music’ was indeed there, and this word is in the top 1,000 selected features from the entire corpus of movie reviews.
... the studio took this film away from its director and chopped it up themselves , and it shows . there might’ve been a pretty decent teen mind-fuck movie in here somewhere , but i guess “ the suits “ decided that turning it into a music video with little edge , would make more sense . the actors are pretty good for the most part , although wes bentley just seemed to be playing the exact same character that he did in american beauty ...
Usually, there is no need to find individual words in the original data and make sure it has a feature in the new, transformed data. Here, I took the time to go through those steps to make sure that the pipeline I constructed works as intended and so show how all parts of the pipeline can have their attributes displayed.
Unstructured text data can be useful in machine learning projects. That text data needs to be changed into a more structured format that the machine can learn from, usually using pre-built transformations like removing stopwords and tf-idf transformations. Pipelines of transformations that the data needs to go through can be built to simplify code and provide consistency to the text pre-processing stage. Utilizing the Pipeline function can help to wrap up several transformations into one.