Natural Language Processing: Count Vectorization with scikit-learn | by Hunter Heidenreich | Towards Data Science

If you haven’t already, check out my previous blog post on word embeddings: Introduction to Word Embeddings
In that blog post, we talk about a lot of the different ways we can represent words to use in machine learning. It’s a high level overview that we will expand upon here and check out how we can actually use count vectorization on some real text data.
Today, we will be looking at one of the most basic ways we can represent text data numerically: one-hot encoding (or count vectorization). The idea is very simple.
We will be creating vectors that have a dimensionality equal to the size of our vocabulary, and if the text data features that vocab word, we will put a one in that dimension. Every time we encounter that word again, we will increase the count, leaving 0s everywhere we did not find the word even once.
The result of this will be very large vectors, if we use them on real text data, however, we will get very accurate counts of the word content of our text data. Unfortunately, this won’t provide use with any semantic or relational information, but that’s okay since that’s not the point of using this technique.
Today, we will be using the package from scikit-learn.
Here is a basic example of using count vectorization to get vectors:
from sklearn.feature_extraction.text import CountVectorizer# To create a Count Vectorizer, we simply need to instantiate one.# There are special parameters we can set here when making the vectorizer, but# for the most basic example, it is not needed.vectorizer = CountVectorizer()# For our text, we are going to take some text from our previous blog post# about count vectorizationsample_text = ["One of the most basic ways we can numerically represent words "               "is through the one-hot encoding method (also sometimes called "               "count vectorizing)."]# To actually create the vectorizer, we simply need to call fit on the text# data that we wish to fixvectorizer.fit(sample_text)# Now, we can inspect how our vectorizer vectorized the text# This will print out a list of words used, and their index in the vectorsprint('Vocabulary: ')print(vectorizer.vocabulary_)# If we would like to actually create a vector, we can do so by passing the# text into the vectorizer to get back countsvector = vectorizer.transform(sample_text)# Our final vector:print('Full vector: ')print(vector.toarray())# Or if we wanted to get the vector for one word:print('Hot vector: ')print(vectorizer.transform(['hot']).toarray())# Or if we wanted to get multiple vectors at once to build matricesprint('Hot and one: ')print(vectorizer.transform(['hot', 'one']).toarray())# We could also do the whole thing at once with the fit_transform method:print('One swoop:')new_text = ['Today is the day that I do the thing today, today']new_vectorizer = CountVectorizer()print(new_vectorizer.fit_transform(new_text).toarray())
Our output:
Vocabulary:{'one': 12, 'of': 11, 'the': 15, 'most': 9, 'basic': 1, 'ways': 18, 'we': 19,  'can': 3, 'numerically': 10, 'represent': 13, 'words': 20, 'is': 7,  'through': 16, 'hot': 6, 'encoding': 5, 'method': 8, 'also': 0,  'sometimes': 14, 'called': 2, 'count': 4, 'vectorizing': 17}Full vector:[[1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 2 1 1 1 1 1]]Hot vector:[[0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]Hot and one:[[0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0] [0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0]]One swoop:[[1 1 1 1 2 1 3]]
So let’s use it on some real data! We will check out the 20 News Group dataset that comes with scikit-learn.
from sklearn.datasets import fetch_20newsgroupsfrom sklearn.feature_extraction.text import CountVectorizerimport numpy as np# Create our vectorizervectorizer = CountVectorizer()# Let's fetch all the possible text datanewsgroups_data = fetch_20newsgroups()# Why not inspect a sample of the text data?print('Sample 0: ')print(newsgroups_data.data[0])print()# Create the vectorizervectorizer.fit(newsgroups_data.data)# Let's look at the vocabulary:print('Vocabulary: ')print(vectorizer.vocabulary_)print()# Converting our first sample into a vectorv0 = vectorizer.transform([newsgroups_data.data[0]]).toarray()[0]print('Sample 0 (vectorized): ')print(v0)print()# It's too big to even see...# What's the length?print('Sample 0 (vectorized) length: ')print(len(v0))print()# How many words does it have?print('Sample 0 (vectorized) sum: ')print(np.sum(v0))print()# What if we wanted to go back to the source?print('To the source:')print(vectorizer.inverse_transform(v0))print()# So all this data has a lot of extra garbage... Why not strip it away?newsgroups_data = fetch_20newsgroups(remove=('headers', 'footers', 'quotes'))# Why not inspect a sample of the text data?print('Sample 0: ')print(newsgroups_data.data[0])print()# Create the vectorizervectorizer.fit(newsgroups_data.data)# Let's look at the vocabulary:print('Vocabulary: ')print(vectorizer.vocabulary_)print()# Converting our first sample into a vectorv0 = vectorizer.transform([newsgroups_data.data[0]]).toarray()[0]print('Sample 0 (vectorized): ')print(v0)print()# It's too big to even see...# What's the length?print('Sample 0 (vectorized) length: ')print(len(v0))print()# How many words does it have?print('Sample 0 (vectorized) sum: ')print(np.sum(v0))print()# What if we wanted to go back to the source?print('To the source:')print(vectorizer.inverse_transform(v0))print()
Our output:
Sample 0:From: lerxst@wam.umd.edu (where's my thing)Subject: WHAT car is this!?Nntp-Posting-Host: rac3.wam.umd.eduOrganization: University of Maryland, College ParkLines: 15 I was wondering if anyone out there could enlighten me on this car I sawthe other day. It was a 2-door sports car, looked to be from the late 60s/early 70s. It was called a Bricklin. The doors were really small. In addition,the front bumper was separate from the rest of the body. This isall I know. If anyone can tellme a model name, engine specs, yearsof production, where this car is made, history, or whatever info youhave on this funky looking car, please e-mail.Thanks,- IL   ---- brought to you by your neighborhood Lerxst ----Vocabulary:{'from': 56979, 'lerxst': 75358, 'wam': 123162, 'umd': 118280, 'edu': 50527,  'where': 124031, 'my': 85354, 'thing': 114688, 'subject': 111322,  'what': 123984, 'car': 37780, 'is': 68532, 'this': 114731, 'nntp': 87620,  'posting': 95162, 'host': 64095, 'rac3': 98949, 'organization': 90379,  'university': 118983, 'of': 89362, 'maryland': 79666,  'college': 40998, ... } (Abbreviated...)Sample 0 (vectorized):[0 0 0 ... 0 0 0]Sample 0 (vectorized) length:130107Sample 0 (vectorized) sum:122To the source:[array(['15', '60s', '70s', 'addition', 'all', 'anyone', 'be', 'body',       'bricklin', 'brought', 'bumper', 'by', 'called', 'can', 'car',       'college', 'could', 'day', 'door', 'doors', 'early', 'edu',       'engine', 'enlighten', 'from', 'front', 'funky', 'have', 'history',       'host', 'if', 'il', 'in', 'info', 'is', 'it', 'know', 'late',       'lerxst', 'lines', 'looked', 'looking', 'made', 'mail', 'maryland',       'me', 'model', 'my', 'name', 'neighborhood', 'nntp', 'of', 'on',       'or', 'organization', 'other', 'out', 'park', 'please', 'posting',       'production', 'rac3', 'really', 'rest', 'saw', 'separate', 'small',       'specs', 'sports', 'subject', 'tellme', 'thanks', 'the', 'there',       'thing', 'this', 'to', 'umd', 'university', 'wam', 'was', 'were',       'what', 'whatever', 'where', 'wondering', 'years', 'you', 'your'],      dtype='<U180')]Sample 0:I was wondering if anyone out there could enlighten me on this car I sawthe other day. It was a 2-door sports car, looked to be from the late 60s/early 70s. It was called a Bricklin. The doors were really small. In addition,the front bumper was separate from the rest of the body. This isall I know. If anyone can tellme a model name, engine specs, yearsof production, where this car is made, history, or whatever info youhave on this funky looking car, please e-mail.Vocabulary:{'was': 95844, 'wondering': 97181, 'if': 48754, 'anyone': 18915, 'out': 68847,  'there': 88638, 'could': 30074, 'enlighten': 37335, 'me': 60560, 'on': 68080,  'this': 88767, 'car': 25775, 'saw': 80623, 'the': 88532, 'other': 68781,  'day': 31990, 'it': 51326, 'door': 34809, 'sports': 84538, 'looked': 57390,  'to': 89360, 'be': 21987, 'from': 41715, 'late': 55746, '60s': 9843,  'early': 35974, '70s': 11174, 'called': 25492, 'bricklin': 24160, 'doors': 34810,  'were': 96247, 'really': 76471, ... } (Abbreviated...)Sample 0 (vectorized):[0 0 0 ... 0 0 0]Sample 0 (vectorized) length:101631Sample 0 (vectorized) sum:85To the source:[array(['60s', '70s', 'addition', 'all', 'anyone', 'be', 'body',       'bricklin', 'bumper', 'called', 'can', 'car', 'could', 'day',       'door', 'doors', 'early', 'engine', 'enlighten', 'from', 'front',       'funky', 'have', 'history', 'if', 'in', 'info', 'is', 'it', 'know',       'late', 'looked', 'looking', 'made', 'mail', 'me', 'model', 'name',       'of', 'on', 'or', 'other', 'out', 'please', 'production', 'really',       'rest', 'saw', 'separate', 'small', 'specs', 'sports', 'tellme',       'the', 'there', 'this', 'to', 'was', 'were', 'whatever', 'where',       'wondering', 'years', 'you'], dtype='<U81')]
So, you may be wondering what now? We know how to vectorize these things based on counts, but what can we actually do with any of this information?
Well, for one, we could do a bunch of analysis. We could look at term frequency, we could remove stop words, we could visualize things, and we could try and cluster. Now that we have these numeric representations of this textual data, there is so much we can do that we couldn’t do before!
But let’s make this more concrete. We’ve been using this text data from the 20 News Group dataset. Why not use it on a task?
The 20 News Group dataset is a dataset of posts on a board, split up into 20 different categories. Why not use our vectorization to try and categorize this data?
from sklearn.datasets import fetch_20newsgroupsfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.naive_bayes import MultinomialNBfrom sklearn import metrics# Create our vectorizervectorizer = CountVectorizer()# All datanewsgroups_train = fetch_20newsgroups(subset='train',                                      remove=('headers', 'footers', 'quotes'))newsgroups_test = fetch_20newsgroups(subset='test',                                     remove=('headers', 'footers', 'quotes'))# Get the training vectorsvectors = vectorizer.fit_transform(newsgroups_train.data)# Build the classifierclf = MultinomialNB(alpha=.01)#  Train the classifierclf.fit(vectors, newsgroups_train.target)# Get the test vectorsvectors_test = vectorizer.transform(newsgroups_test.data)# Predict and score the vectorspred = clf.predict(vectors_test)acc_score = metrics.accuracy_score(newsgroups_test.target, pred)f1_score = metrics.f1_score(newsgroups_test.target, pred, average='macro')print('Total accuracy classification score: {}'.format(acc_score))print('Total F1 classification score: {}'.format(f1_score))
Our output:
Total accuracy classification score: 0.6460435475305364Total F1 classification score: 0.6203806145034193
Hmmm... So not super fantastic, but we are just using count vectors! A richer representation would do wonders for our scores!
Hopefully you feel like you learned a lot about count vectorization, how to use it, and some of the potential applications of it!
If you enjoyed reading this article, drop me a comment or maybe donate to my GoFundMe to help me continue with my ML research!
And stay tuned for more word embedding content coming soon!