Feature Extraction, the Moment of Truth in Natural Language Processing
By: Papa Moryba Kouate
Tags: Data Analysis, Data Science, Machine Learning, Naturallanguageprocessing

Part I: Bag of Words

For computers to understand and contextualize human language, the best practice is to convert words into numbers through algorithms constructed with concepts of linear algebra, calculus and statistics.

So if we work with texts, we have to convert texts into numbers in order to guarantee our model the best performance.

The first important concept in Feature Extraction is Bag of Words.

For understanding the meaning of Bag of Words at an intuitive level, think to put into a bag distinct things in disorder. So, the concept of Bag of Words suggests us that at the same way we put things into a bag, we also put tokens into a set in a non-precise order.

For example, we put the tokens of two different documents into our bag or set and we want to know how many times a specific word occurs in each document. To solve this issue, we can create a matrix to compare the occurrence of a word in each document. If that is not clear yet, don’t worry, with an example you will visualize and understand better the concept of Bag of Words.

We have two documents as you can see below.

'blue pill you wake up in your bed and believe whatever you want to believe', 'red pill you stay in wonderland and I show you how deep the rabbit hole goes'

Then, we create a list for these two documents.

txt = [‘blue pill you wake up in your bed and believe whatever you want to believe’,

‘red pill you stay in wonderland and I show you how deep the rabbit hole goes’]

Now we initialize CountVectorizer object, that permits converting a collection of text documents in a vector of token counts. After that we can fit our data.

from sklearn.feature_extraction.text import CountVectorizer vect = CountVectorizer() vect.fit(txt)

out:

CountVectorizer(analyzer='word', binary=False, decode_error='strict',dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',lowercase=True, max_df=1.0, max_features=None, min_df=1,ngram_range=(1, 1), preprocessor=None, stop_words=None,

strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b', tokenizer=None, vocabulary=None)

In the step below we use the attribute vocabulary_ in order to obtain for each word a precise index.

print(vect.vocabulary_)

out:

{'blue': 3, 'pill': 9, 'you': 21, 'wake': 17, 'up': 16, 'in': 8, 'your': 22, 'bed': 1, 'and': 0, 'believe': 2, 'whatever': 19, 'want': 18, 'to': 15, 'red': 11, 'stay': 13, 'wonderland': 20, 'show': 12, 'how': 7, 'deep': 4, 'the': 14, 'rabbit': 10, 'hole': 6, 'goes': 5}

As you can observe each word has an index but they are not in order. To solve this problem, we can use the attribute get_feature_names().

print(vect.get_feature_names())

out:

['and', 'bed', 'believe', 'blue', 'deep', 'goes', 'hole', 'how', 'in', 'pill', 'rabbit', 'red', 'show', 'stay', 'the', 'to', 'up', 'wake', 'want', 'whatever', 'wonderland', 'you', 'your']

Now it is time to create our bag of words, to make it we have to transform our data in the following way.

bag_of_words = vect.transform(texts)

It is better to view in an array form. So:

bag_of_words.toarray()

out:

array([[1, 1, 2, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 2, 1],

[1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 2,0]], dtype=int64)

As you can see above the word “and” with index 0 occurs once for both the first and the second sentence.