Understanding Text Vectorizations I: How Having a Bag of Words Already Shows What People Think About Your Product | Towards Data Science

Do you know that we already (almost) solved sentiment analysis problems before neural networks become ubiquitous?
In the current era of natural language processing (NLP) increasingly relying on deep learning models that generate amazing performances, we have often overlooked the importance of the simplest types of text vectorization techniques — bag of words (BOW) and term frequency-inverse document frequency (TF-IDF). In fact, with these two techniques, we have already been able to predict the sentiment of a given piece of text with more than 80% accuracy. In other words, all we are doing with state-of-art deep learning models are simply trying to squeeze more performance improvement out these classification problems. Almost all of the more complicated models, to some extent, have relied on concepts from BOW and TF-IDF. In this blog post, we will attempt to understand and implement BOW and TF-IDF, with some help from sklearn pipelines.
You might think using pipelines are an extreme overkill. Indeed, implementing pipelines would require extensive utilization in object-oriented programming (OOP), which could potentially make debugging rather difficult. Though it took me two weeks to implement both the BOW and TF-IDF (with some mistakes I made in between, more about this in another blog post), it is definitely worth the time and effort. By using sklearn pipelines, we are able to transform and train/predict with raw data in a single function call. For example, if we want to use sklearn’s TF-IDF vectorizer and train logistic regression model, we just have to use the following lines of code.
from sklearn.feature_extraction.text import CountVectorizerfrom sklearn.linear_model import LogisticRegressionfrom sentiment_analysis.models.model import StreamlinedModellogistic = StreamlinedModel(    transformer_description="Bag of words",    transformer=CountVectorizer,    model_description="logisitc regression model",    model=LogisticRegression,)
TfidfVectorizer and LogisticRegression are module objects from sklearn, while StreamlinedModel is the pipeline object we will implement. As you probably already see, with this structure we will be able to easily swap the above two objects with any transformers and models — This structure will keep the interface universal and make feature extractions a breeze. There are also some other really important reasons we would want to build models in this fashion, but that will be discussed in another blog post.
To implement StreamlinedModel class, we should recall how most of our familiar sklearn packages are used. For example,
logistic = LogisticRegression()logistic.fit(X_train, y_train)logistic.predict(X_test)
Our StreamlinedModel will have the same behavior, therefore we will add the same .fit and .predict methods in the class. Other useful methods include .predict_proba and .score. These methods could simply be wrappers around the same methods in sklearn’s BaseEstimator . For example, the pipeline training model could simply be
We can then build a class method called make_model_pipeline that uses the 4 parameters upon class instantiation and make a sklearn pipeline object.
This self.pipeline is the same object that we can call .fit and .predict etc. That’s it! We have created the StreamlinedModel . Full implementation is here below
With the flexibility of switching the transformer into any transformer of our choice, we are now able to utilize the same interface with our customized BOW and TF-IDF transformers. For the purpose of differentiation, we will call our customized BOW transformer WordFrequencyVectorizer. The same StreamlinedModel could be used in the following way
from models.feature import WordFrequencyVectorizerfrom sklearn.linear_model import LogisticRegressionfrom sentiment_analysis.models.model import StreamlinedModellogistic = StreamlinedModel(    transformer_description="Bag of Words",    transformer=WordFrequencyVectorizer,    model_description="logisitc regression model",    model=LogisticRegression,)
In this blog post, we will focus on implementing the bag of words model and leave the TF-IDF for the next one.
The idea of the BOW model is very simple. We would like to find the fraction of specific words that are strong indicators of positive or negative sentiment. For example, a review contains more positive words like “great”, “fantastic” would be considered more positive than a review with words like “so-so”. Since reviews all have different lengths, we will need to pad each review with 0s for the words that are not contained in the review. For example, if we have the following 3 reviews,
I love dogs, I think they have adorable personalities.I don't like catsMy favorite pet is a bird
We will let each unique word occupy a specific position. Those words that are not contained in the review will be filled in to be 0, while the words that appeared more than once will be incremented to the number of their occurrences.
Now we understand the general idea of how the BOW model works, it’s time to see how we could adapt the idea to our sentiment analysis task. We will start by loading in the reviews from our data source.
The data we will use for this project are the reviews from John Hopkins University’s multi-domain sentiment dataset. Once we download and unzip the data, we will see a series of folders with the following structure.
/reviews     |--/books      |--positive.review      |--negative.review   |--/dvd         |--positive.review      |--negative.review   |--electronics        |--positive.review      |--negative.review   |--kitchen_&_house_supplies        |--positive.review      |--negative.review
We would like to combine all the positive/negative reviews together so that we will not need to reference the individual folders separately. We will create a class called LoadReviews to accomplish this goal. The full implementation of this class is shown below
Since the reviews are stored in XML format, we will have to use BeautifulSoup to parse out the reviews into text strings. The actual text reviews are stored in the tag review_text . Upon reading in the text reviews, we will also filter out the characters that are not Unicode strings. The static method strip_non_printable accomplishes such task. All the processed reviews will be stored in the class attribute reviews. As you probably noticed, our BOW model will only operate on individual word tokens rather than long strings, we will need to take further actions to preprocess the reviews into the proper vectorized format that can be consumed by our BOW transformer.
To begin the preprocessing step, we will first convert the long str format of each review into a tokenized version of the review, which could be easily accomplished by splitting the string by white space. For example, the string “I love nlp” will become the list [“I”, “love”, “nlp”].
tokens = s.split()
Using split will probably result in a common edge case where the word has punctuation attached but no space in between, such as “hello,”. To avoid this word be treated differently as “hello”, we will strip all the punctuations and replace it with whitespace.
s = s.translate(str.maketrans(string.punctuation,                               " " * len(string.punctuation)))
Many words that don’t have strong indications of sentiment, such as, “I”, “they”, “a”, but will still occupy spaces. We will proceed to remove them from the list of unique words so that they no longer occupy a position.
tokens = [t for t in tokens           if t not in stopwords.words("english") + [""]]
In addition, Same words of different forms, such as “work” to “worked”, would need to be treated as the same words. This would require us to stem and lemmatize all the words to its original form, which could be accomplished by using NLTK’s word stemmer.
tokens = [wordnet_lemmatizer.lemmatize(t) for t in tokens]
Lastly, we will also remove the words that are less than 2 characters in length, as they probably provide very little meanings to the sentiment.
tokens = [t for t in tokens if len(t) > 2]
All of these steps could be wrapped into a single utility class named WordTokenizer , implemented as the following.
With all of these components ready, we are now prepared to implement the BOW model.
Since the model will be used as a transformer by the StreamlinedModel object, we will need to follow the sklearn pipeline’s guidelines for building custom transformers. That means the objects will need to have a .transform and a.fit method. The .fit method will take care of the transformations that save their specific parameters from training and apply them during predictions. Specifically, in our case, the list of unique words contained in the dataset will be fitted during training, but not re-fitted again during prediction.
The word_index_mapping will be a property from the object ReviewProcessor that is essentially an enumeration of all the unique words. We will also set the last position of the vector to be “unknown_word”, in case of the word during prediction is not part of the list of unique words.
The .transform method is used to convert the inputted raw text to the word frequency vector.
The get_word_frequency_vector method will take in a review and increment the value at the corresponding index in the vector according to the pre-calculated word_to_index mapping. If the inputted review text contains unknown words, the last index of the word_frequency_vector will be incremented.
The full implementation of the BOW transformer is shown below
Now we have implemented the BOW as a transformer, we could use the StreamlinedModel to build 4 different versions of models and compare the performances.
Just as the StreamlinedModel has the ability to use any transformer, we can also use a different machine learning model by swapping the model argument. For example, previously we have used the logistic regression model by calling the LogisticRegression object, we can change it into the lightGBM model like the following
from models.feature import WordFrequencyVectorizerimport lightgbm as lgbfrom sentiment_analysis.models.model import StreamlinedModellgbm = StreamlinedModel(    transformer_description="Bag of words",    transformer=WordFrequencyVectorizer,    model_description="logisitc regression model",    model=lgb.LGBMClassifier,)
We will try 4 different models and collect their prediction AUC scores to make a bar plot.
Using the logistic regression model could already get us an AUC score of 0.82.
LightGBM model also helped us improve the performance up to 0.888 — showing the superior predictive power of tree models than linear models.
The most interesting results come from the two different naive Bayes model that yielded tremendously different outcomes, as Gaussian naive Bayes performed way worse than multinomial naive Bayes. Since Gaussian naive Bayes models are designed to operate on continuous variables, while word frequencies are only discrete, the subpar performance is expected.
As you probably noticed, the relatively simple multinomial naive Bayes model achieved comparable performance with the rather advanced lightGBM model in this sentiment analysis task. So what how did our model make the decision? Let’s have a closer look.
Since there are multiple reviews got misclassified, we will only visualize the one that yielded the largest distance from the correct label. To do this, we will sort the predicted probabilities print out the first/last one (depending on the label)
# get the indices of the misclassified reviewswrong_positive_inds = np.where((y_test == 1)                              & (y_pred != y_test))[0]wrong_negative_inds = np.where((y_test == 0)                              & (y_pred != y_test))[0]# most wrong positive reviewmost_wrong_positive_index = wrong_positive_inds[    y_prob_lgbm[:, 1][wrong_positive_inds].argmax()]# most wrong negative reviewmost_wrong_negative_index = wrong_negative_inds[    y_prob_lgbm[:, 1][wrong_negative_inds].argmin()]
The most wrong positive review
This is a rather long and detailed positive review that mentioned the strengths and weaknesses of the book. In the weakness section, some negative words like “problems”, “not” are mentioned multiple times, which in turn, making classification on this review more difficult.
Let’s have a look at the most wrong negative review.
This negative review also contains a positive word “interesting” and has no obvious negative words (maybe “outdated” is one negative word), making classifying this review as negative less likely.
So what words exactly does the model think a review is positive/negative? Let’s have a look next.
At this point, we would often raise the question of how to choose between the two best performing model (LightGBM and Multinomial naive Bayes). The answer lies in interoperability. Since naive Bayes relies on counting and probabilities, we can easily figure out which words contribute to the sentiment. LightGBM model, however, is not that simple. Due to a large number of unique words, tree splits are quite challenging to visualize. Are we really out of luck in interpreting them?
Not really. With the help of model explanation package SHAP, we can produce a visualization of feature importance on a single review (more about SHAP in a different post). In the below SHAP visualization graph, red represents the predicted sentiment is closer to 1, while blue represents the predicted sentiment to be 0. Below is an example review that the model predicted to be positive
As the review contains strong positive words like “best” and “enjoyed”, this review is predicted to be a positive one.
The negative reviews normally consist of negative words like “disappointment”, “fails” or “bad”.
Bag of words would probably be the simplest type of text vectorization for sentiment analysis tasks, but it was already quite effective. The project GitHub could be found here. Below is what we have learned from today.
Object-oriented-programming has tremendously improved the flexibility of the entire workflow. With help from sklearn pipelines, we could quickly iterate and test multiple machine learning models with the same BOW transformer.We have to use the right model for the underlying task. The Gaussian naive Bayes model was the only one that could not get over 80% AUC score. It was simply the wrong model to use as it is designed to operate on continuous features.LightGBM wins the race (by a small margin). The tree-based model was able to produce a rather great fit and the best performance as well, although multinomial naive Bayes performed only slightly worse.LightGBM needs to rely on some special explanation package to be understandable, now we are getting it. As model explanation becoming more and more important in our industry, we are often forced to choose simpler models for their interoperability. The existence of SHAP changed all of that.
Object-oriented-programming has tremendously improved the flexibility of the entire workflow. With help from sklearn pipelines, we could quickly iterate and test multiple machine learning models with the same BOW transformer.
We have to use the right model for the underlying task. The Gaussian naive Bayes model was the only one that could not get over 80% AUC score. It was simply the wrong model to use as it is designed to operate on continuous features.
LightGBM wins the race (by a small margin). The tree-based model was able to produce a rather great fit and the best performance as well, although multinomial naive Bayes performed only slightly worse.
LightGBM needs to rely on some special explanation package to be understandable, now we are getting it. As model explanation becoming more and more important in our industry, we are often forced to choose simpler models for their interoperability. The existence of SHAP changed all of that.
In the next post, we will build on what we accomplished with the bag of words model and improve it with a smart weighting scheme. See you next time!