Getting Started with Text Vectorization | by Shirley Chen | Towards Data Science

Feel free to follow me on Medium :)
Recently, I completed “Book Recommendations from Charles Darwin” case study on DataCamp. In this project, we will learn how to implement text preprocessing and text vectorization, how to build a book recommendation system using Natural Language Processing(NLP) and detect how closely related Charles Darwin’s books are to each other.
Text Vectorization is the process of converting text into numerical representation. Here is some popular methods to accomplish text vectorization:
Binary Term Frequency
Bag of Words (BoW) Term Frequency
(L1) Normalized Term Frequency
(L2) Normalized TF-IDF
Word2Vec
In this section, we will use the corpus below to introduce the 5 popular methods in text vectorization.
corpus = ["This is a brown house. This house is big. The street number is 1.",          "This is a small house. This house has 1 bedroom. The street number is 12.",          "This dog is brown. This dog likes to play.",          "The dog is in the bedroom."]
Binary Term Frequency captures presence (1) or absence (0) of term in document. For this part, under TfidfVectorizer, we set binary parameter equal to true so that it can show just presence (1) or absence (0) and norm parameter equal to false.
Bag of Words (BoW) Term Frequency captures frequency of term in document. Under TfidfVectorizer, we set binary parameter equal to false so that it can show the actual frequency of the term and norm parameter equal to none.
(L1) Normalized Term Frequency captures normalized BoW term frequency in document. Under TfidfVectorizer, we set binary parameter equal to false so that it can show the actual frequency of the term and norm parameter equal to l1.
(L2) Normalized TFIDF (Term Frequency–Inverse Document Frequency) captures normalized TFIDF in document. The below is the formula for how to compute the TFIDF.
Under TfidfVectorizer, we set binary parameter equal to false so that it can show the actual frequency of the term and norm parameter equal to l2.
Word2Vec provides embedded representation of words. Word2Vec starts with one representation of all words in the corpus and train a NN (with 1 hidden layer) on a very large corpus of data. Here is the two methods that is typically used for training the NN:
Continuous Bag of Words (CBOW) — Predict vector representation of center/target word based on window of context words
Skip-Gram (SG) — Predict vector representation of window of context words based on center/target word
Once we have the embedded vectors for each word and use them for NLP:
Compute cosine similarity between word vectors, create higher order representations using weighted average of the word vectors and feed to the classification task
Python’s spacy package provides pre-trained models we can use to see how w2v works.
import spacynlp = spacy.load("en_core_web_md", disable=['parser', 'ner'])# Get w2v representation of the word 'breakfast'print (nlp('breakfast').vector.size)nlp('breakfast').vector[:10]# Find cosine similarity between w2v representations of breakfast and universenlp('breakfast').similarity(nlp('universe'))   # 0.044292555doc1 = nlp("I like oranges that are sweet.")doc2 = nlp("I like apples that are sour.")doc1.similarity(doc2)     # 0.962154245
Charles Darwin is the most famous scientist in the world. He wrote many other books on a wide range of topics, including geology, plants or his personal life. In this project, we will develop a content-based book recommendation system, which will determine which books are close to each other based on how similar the discussed topics are. Let’s take a look at the books we will use later.
import glob  # glob is a general term used to define techniques to match specified patterns according to rules related to Unix shell.folder = "datasets/"files = glob.glob(folder + "*.txt")files.sort()
As the first step, we need to load the content of each book and check the regular expression to facilitate the process by removing the all non-alpha-numeric characters. We call such a collection of texts a corpus.
import re, ostxts = []titles = []for n in files:    f = open(n, encoding='utf-8-sig')    data = re.sub('[\W_]+', ' ', f.read())     txts.append(data)    titles.append(os.path.basename(n).replace('.txt', ''))[len(t) for t in txts]
And then, for consistency, we will refer to Darwin’s most famous book “On the Origin of Species” to check the results for other given book.
for i in range(len(titles)):    if titles[i] == 'OriginofSpecies':        ori = iprint(ori)      # Index = 15
Next step, we transform the corpus into a format by doing tokenization.
stoplist = set('for a of the and to in to be which some is at that we i who whom show via may my our might as well'.split())txts_lower_case = [i.lower() for i in txts]txts_split = [i.split() for i in txts]texts = [[word for word in txt if word not in stoplist] for txt in txts_split]texts[15][0:20]
For the next parts of text preprocessing, we use a stemming process, which will group together the inflected forms of a word so they can be analyzed as a single item: the stem. In order to make the process faster, we will directly load the final results from a pickle file and review the method used to generate it.
import pickletexts_stem = pickle.load(open('datasets/texts_stem.p', 'rb'))texts_stem[15][0:20]
Bag-of-Words Models (BoW)
First, we need to create a universe of all words contained in our corpus of Charles Darwin’s books, which we call a dictionary. Then, using the stemmed tokens and the dictionary, we will create bag-of-words models (BoW) to represent our books as a list of all uniques tokens they contain associated with their respective number of occurrences.
from gensim import corporadictionary = corpora.Dictionary(texts_stem)bows = [dictionary.doc2bow(text) for text in texts_stem]print(bows[15][:5])
In order to better understand the model, we will transform it into a DataFrame and display the 10 most common stems for the book “On the Origin of Species”.
df_bow_origin = pd.DataFrame()df_bow_origin['index'] = [i[0] for i in bows[15] if i]df_bow_origin['occurrences'] = [i[1] for i in bows[15] if i]df_bow_origin['token'] = [dictionary[index] for index in df_bow_origin['index']]df_bow_origin.occurrences.sort_values(ascending=False).head(10)
TF-IDF Model
Next, we will use a TF-IDF model to define the importance of each word depending on how frequent it is in the text. As a result, a high TF-IDF score for a word will indicate that this word is specific to this text.
from gensim.models import TfidfModelmodel = TfidfModel(bows)model[bows[15]]
Once again, in order to better understand the model, we will transform it into a DataFrame and display the 10 most specific words for the “On the Origin of Species” book.
df_tfidf = pd.DataFrame()df_tfidf['id'] = [i[0] for i in model[bows[15]]]df_tfidf['score'] = [i[1] for i in model[bows[15]]]df_tfidf['token'] = [dictionary[index] for index in df_tfidf['id']]df_tfidf.score.sort_values(ascending=False).head(10)
Now that we have a TF-IDF model on how specific they are to each book, we can measure how related to books are between each other. Therefore, we will use Cosine Similarity and visualize the results as a distance matrix.
from gensim import similaritiessims = similarities.MatrixSimilarity(model[bows])sim_df = pd.DataFrame(list(sims))sim_df.columns = titles sim_df.index = titlessim_df
We now have a matrix containing all the similarity measures between any pair of books from Charles Darwin! We can use barh() to display a horizontal bar plot for which books are the most similar to “On the Origin of Species.”
%matplotlib inlineimport matplotlib.pyplot as pltv = sim_df['OriginofSpecies']v_sorted = v.sort_values()v_sorted.plot.barh()plt.xlabel('Similarity')
However, we want to have a better understanding of the big picture and see how Darwin’s books are generally related to each other. To this purpose, we will represent the whole similarity matrix as a dendrogram, which is a standard tool to display such data.
from scipy.cluster import hierarchyZ = hierarchy.linkage(sim_df, 'ward')chart = hierarchy.dendrogram(Z, leaf_font_size=8, labels=sim_df.index, orientation="left")
Finally, based on the chart we created before, we can conclude that “the variation of animals and plants under domestication” is most related to “On the Origin of Species.”
Source code that created this article can be found in my Github.
Thank you so much for reading my article! Hi, I’m Shirley, currently studying for a Master Degree in MS-Business Analytics at ASU. If you have questions, please don’t hesitate to contact me!
Email me at kchen122@asu.edu and feel free to connect me on LinkedIn!