How to Vectorize Text in DataFrames for NLP Tasks — 3 Simple Techniques | by Eric Kleppen | Towards Data Science

You might have heard that Natural Language Processing (NLP) is one of the most transformative technologies of the decade. It is a fantastic time to get into NLP as new libraries have abstracted away many of the complexities allowing users state of the art results with little model training and little code.
While a computer can actually be quite good at finding patterns and summarizing documents, it must transform words into numbers before making sense of them. Instead of vectorizing the text at the word level like I’ve done in the past, I’ll explore the following techniques to vectorize text in dataframes at the sentence, paragraph or document level:
TextHero: TF-IDFGensim: Doc2VecTensorFlow2: Universal Sentence Encoder 4
As a bonus, I’ll visualize the vectors using t-SNE too! Complete Code is towards the bottom!
Instead of scraping data from one of my favorite sources, I’m going to explore cards from the game Magic: The Gathering. I’ve been into card games almost my entire life and started playing Magic: The Gathering (MTG) around 2nd grade in the mid 90’s. You can play it for free online if you’re interested in trying it!
I was excited to find free datasets of all the cards. To follow the examples, download the AllPrintings SQLITE file from MTGJSON.com
MTGJSON is an open-source project that catalogs all Magic: The Gathering cards in a portable format.
It is easy to connect and load the data into a dataframe since it is already an sqlite file. Follow three steps to load the libraries, data and DataFrame!
1. Import pandas and sqlite3 libraries 2. Connect to the sqlite file 3. Load the data into a pandas DataFrame.
#Import dependenciesimport pandas as pdimport sqlite3#Establish connection to sqlite databaseconn = sqlite3.connect("AllPrintings.sqlite")#load the data into a pandas DataFramedf = pd.read_sql("select * from cards", conn)
Notice I use the pandas read_sql function to generate a dataframe using raw SQL. I can see there are 73 columns and 54033 rows in the dataset.
I am going to reduce the number of columns and the number of rows because I only care about distinct cards. In the game Magic: The Gathering, there are 5 core colors: Black, Blue, Green, Red, and White.
I will explore my favorite color, Green.
df = pd.read_sql("select distinct name, text, convertedManaCost, power, toughness, keywords from cards where borderColor ='black' and colorIdentity = 'G'", conn)
Notice I start the SQL query with SELECT DISTINCT to exclude duplicate cards. This is necessary because some cards have been reprinted in multiple sets over the years.
Removing the duplicates and limiting the dataset to green cards leaves me with 3085 rows and 6 columns.
To clean the data and prepare it for vectorization, I’m going to use the Texthero library as it simplifies cleaning text in DataFrames. To apply the default text cleaning script run hero.clean(pandas.Series).
It runs the following seven functions by default when using clean():
fillna(s) Replace not assigned values with empty spaces.lowercase(s) Lowercase all text.remove_digits() Remove all blocks of digits.remove_punctuation() Remove all string.punctuation (!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~).remove_diacritics() Remove all accents from strings.remove_stopwords() Remove all stop words.remove_whitespace() Remove all white space between words.
fillna(s) Replace not assigned values with empty spaces.
lowercase(s) Lowercase all text.
remove_digits() Remove all blocks of digits.
remove_punctuation() Remove all string.punctuation (!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~).
remove_diacritics() Remove all accents from strings.
remove_stopwords() Remove all stop words.
remove_whitespace() Remove all white space between words.
Instead of using the default clean function, I’m going to build my own pipeline. Using my own pipeline, I can preserve as much card context as possible for similarity:
import texthero as herofrom texthero import preprocessingcustom_pipeline = [preprocessing.fillna,                   #preprocessing.lowercase,                   preprocessing.remove_whitespace,                   preprocessing.remove_diacritics                   #preprocessing.remove_brackets                  ]df['clean_text'] = hero.clean(df['text'], custom_pipeline)df['clean_text'] = [n.replace('{','') for n in df['clean_text']]df['clean_text'] = [n.replace('}','') for n in df['clean_text']]df['clean_text'] = [n.replace('(','') for n in df['clean_text']]df['clean_text'] = [n.replace(')','') for n in df['clean_text']]
Notice I use fillna since some cards do not have text. Notice I decided to exclude remove_brackets from the pipeline. That is because Texthero will remove the data in between the brackets too!
TF-IDF stands for term frequency-inverse document frequency. It is a classical method for weighting the word value instead of simply counting it. It is used to determine how important a word is to a text within a collection documents.
TF-IDF is a bag-of-words (BoW) representation of the text that describes the occurrence of words within a text corpus. It doesn’t account for the sequence of the words. I’ll talk more about the limitations in the next section...
TextHero makes it easy to apply TF-IDF to the text in the dataframe.
df['tfidf'] = (hero.tfidf(df['clean_text'], max_features=3000))
Adding the values to the dataframe is literally 1 line of code! I recommend exploring different numbers of max_features to see how it affects the vectors.
TF-IDF produces a sparse matrix that contains lots of 0’s because of the wide variety of words on the cards.
While TF-IDF is a good starting point to establish a baseline using classical vectorization techniques, it has major limitations. Since it is a bag-of-words representation, it doesn’t keep the words in order and doesn’t account for much semantic meaning. As a sparse matrix, it can also be resource intensive.
To solve the issue of semantic relationships and sparse matrices, models like Word2Vec were created. Previously I’ve written about using the Gensim library to apply the Word2Vec model on text data.
towardsdatascience.com
The Word2Vec model generates vectors for each word, but to calculate a vector for the entire document I could average the vectors for each word in the document. Instead, I’ll use the simpler route: Doc2Vec.
Introduced in 2014, Doc2Vec is a Model that represents Documents, paragraphs or sentences as a Vector. Doc2Vec commonly outperforms simple-averaging of Word2Vec vectors, so it is worth exploring!
At the basic level, the Doc2Vec algorithm gives a document another floating word-like vector, called a doc-vector. It contributes to all training predictions, and is updated like other word-vectors. Check out the full paper by Le and Mikolov to get an understanding of the technical aspects.
Doc2Vec needs to be trained. To train the model, a tag/number needs to be associated with each document of the training. Tagging is done easily using doc2vec.TaggedDocument().
from gensim.models.doc2vec import Doc2Vec, TaggedDocument#tokenize and tag the card textcard_docs = [TaggedDocument(doc.split(' '), [i])              for i, doc in enumerate(df.clean_text)]#display the tagged docscard_docs
Notice I do simple tokenization using .split(‘ ‘) on the cleaned card text.
Next, instantiate a Doc2Vec model.
Set the Vector size to determine how many dimensions will be included. Set the training epochs to determine how many times it will iterate over the training data.
model = Doc2Vec(vector_size=64, min_count=1, epochs = 20)
When setting epochs, consider high numbers tend to reach diminishing returns. Experiment with the hyperparameters! Check out the documentation for a comprehensive list of all model parameters.
Build a vocabulary after instantiating the model, and then train the model.
#instantiate modelmodel = Doc2Vec(vector_size=64, window=2, min_count=1, workers=8, epochs = 40)#build vocabmodel.build_vocab(card_docs)#train modelmodel.train(card_docs, total_examples=model.corpus_count            , epochs=model.epochs)
Building the vocabulary creates a dictionary (accessible via model.wv.vocab) of all of the unique words extracted from training along with the count.
Now that the model has been trained, pass the tokenized text through the model to generate vectors using model.infer_vector.
#generate vectorscard2vec = [model.infer_vector((df['clean_text'][i].split(' ')))             for i in range(0,len(df['clean_text']))]card2vec
Notice the data outputs as numpy array. To add the vectors to the dataframe, use numpy.array().tolist(). This will save them as a list of lists. Then they can be added as a column to the dataframe.
import numpy as np#Create a list of listsdtv= np.array(card2vec).tolist()#set list to dataframe columndf['card2vec'] = dtvdf.head(2)
Notice the card2vec column contains the Doc2Vec vectors. Adding the vectors to the dataframe is a convenient way to store them.
Although the Doc2Vec method is and is done with only a few lines of code, instead of training a Doc2Vec model, there are pre-trained options available requiring even LESS code!
Even simpler than using Doc2Vec, the Universal Sentence Encoder (USE) hosted on Tensorflow-hub (tfhub) is a pre-trained model that encodes text into a 512 dimensional vector. It is trained on a wide variety of text sources.
The Universal Sentence Encoder encodes text into high-dimensional vectors that can be used for text classification, semantic similarity, clustering and other natural language tasks. — Documentation
The latest iterations of the USE models use TensorFlow2. I have used the Tensorflow 1 models in the past to generate wine recommendations, and these new models are MUCH simpler to use and faster to apply.
The model needs to be downloaded from tfhub.
!pip install --upgrade tensorflow_hubimport tensorflow_hub as hub#download the modelembed = hub.load("https://tfhub.dev/google/universal-sentence-encoder-large/5")
Next, run the cleaned_text through the model to generate the vectors. Using the same technique from the Doc2Vec example, transform the vectors from a numpy array into a list of lists.
From there, it is easy to add the vectors to the dataframe. It is that easy!
#generate embeddingsembeddings = embed(df['clean_text'])#create list from np arraysuse= np.array(embeddings).tolist()#add lists as dataframe columndf['use'] = use#check dataframedf.head(2)
Congratulations! You just learned 3 fast and simple ways to vectorize greater-than-word length text in a DataFrame!
Since I went to the trouble of generating the vectors, I might as well explore them! Using the Texthero library, it is easy to apply the t-SNE algorithm to reduce the dimensionality of the vectors and visualize them in 2D space.
The t-SNE (t-distributed stochastic neighbor embedding) is a machine learning algorithm used for visualizing high-dimensional data. The t-SNE technique applies non-linear dimensionality reduction.
df['tsnetfidf'] = hero.tsne(df['tfidf'])df['tsnec2v'] = hero.tsne(df['card2vec'])df['tsneuse'] = hero.tsne(df['use'])
The algorithm is complex and can take a little time to complete!
Next, use Texthero to build scatterplots of the t-sne columns. Texthero uses Plotly under the hood, so the graphs are interactive.
#create scatter plot of tfidfhero.scatterplot(df, col='tsnetfidf', color='convertedManaCost'                 , title="TF-IDF", hover_data = ['name','text'])#create scatter plot of doc2vechero.scatterplot(df, col='tsnec2v', color='convertedManaCost'                 , title="Doc2Vec", hover_data = ['name','text'])#create scatter plot of uni. sent. enc.hero.scatterplot(df, col='tsneuse', color='convertedManaCost'                 , title="U.S.E", hover_data = 'name','text'])
Notice I set the hover_data to name and text. Hovering over a point will reveal the card name and the card text. This way it is easy to compare how the various techniques clustered similar cards together.
Each technique produces different plots, but clumps similar cards together. For example, I find clumps of cards that have trample, or add +1/+1, or tap to add mana.
Natural Language Processing can be a complicated topic, but these techniques should give you a springboard from which you can delve deeper into the field. I presented three simple ways to vectorize text at the paragraph, sentence, or document level:
TextHero: TF-IDFGensim: Doc2VecTensorFlow2: Universal Sentence Encoder 4
Creating vectors from the text prepares it for machine learning tasks like text classification, semantic similarity, and clustering. Learn more about Natural Language Processing and Data Science in my other articles:
medium.com
towardsdatascience.com
Here is the complete code or find it on my github!
github.com
#Import dependenciesimport pandas as pdimport sqlite3import texthero as herofrom texthero import preprocessingfrom gensim.models.doc2vec import Doc2Vec, TaggedDocumentimport numpy as npimport tensorflow_hub as hub#Establish connection to sqlite databaseconn = sqlite3.connect("AllPrintings.sqlite")#load the data into a pandas DataFramedf = pd.read_sql("select * from cards", conn)df = pd.read_sql("select distinct name, text, convertedManaCost, power, toughness, keywords from cards where borderColor ='black' and colorIdentity = 'G'", conn)custom_pipeline = [preprocessing.fillna,                   #preprocessing.lowercase,                   preprocessing.remove_whitespace,                   preprocessing.remove_diacritics                   #preprocessing.remove_brackets                  ]df['clean_text'] = hero.clean(df['text'], custom_pipeline)df['clean_text'] = [n.replace('{','') for n in df['clean_text']]df['clean_text'] = [n.replace('}','') for n in df['clean_text']]df['clean_text'] = [n.replace('(','') for n in df['clean_text']]df['clean_text'] = [n.replace(')','') for n in df['clean_text']]df['tfidf'] = (hero.tfidf(df['clean_text'], max_features=3000))#tokenize and tag the card textcard_docs = [TaggedDocument(doc.split(' '), [i])              for i, doc in enumerate(df.clean_text)]card_docsmodel = Doc2Vec(vector_size=64, min_count=1, epochs = 20)#instantiate modelmodel = Doc2Vec(vector_size=64, window=2, min_count=1, workers=8, epochs = 40)#build vocabmodel.build_vocab(card_docs)#train modelmodel.train(card_docs, total_examples=model.corpus_count            , epochs=model.epochs)#generate vectorscard2vec = [model.infer_vector((df['clean_text'][i].split(' ')))             for i in range(0,len(df['clean_text']))]card2vec#Create a list of listsdtv= np.array(card2vec).tolist()#set list to dataframe columndf['card2vec'] = dtvdf.head(2)#download the modelembed = hub.load("https://tfhub.dev/google/universal-sentence-encoder-large/5")#generate embeddingsembeddings = embed(df['clean_text'])#create list from np arraysuse= np.array(embeddings).tolist()#add lists as dataframe columndf['use'] = [v for v in use]#check dataframedf.head(2)df['tsnetfidf'] = hero.tsne(df['tfidf'])df['tsnec2v'] = hero.tsne(df['card2vec'])df['tsneuse'] = hero.tsne(df['use'])#create scatter plot of tfidfhero.scatterplot(df, col='tsnetfidf', color='convertedManaCost'                 , title="TF-IDF", hover_data = ['name','text'])#create scatter plot of doc2vechero.scatterplot(df, col='tsnec2v', color='convertedManaCost'                 , title="Doc2Vec", hover_data = ['name','text'])#create scatter plot of uni. sent. enc.hero.scatterplot(df, col='tsneuse', color='convertedManaCost'                 , title="U.S.E", hover_data = 'name','text'])
medium.com
If you enjoyed this, follow me on Medium for more
Get FULL ACCESS and help support my content by subscribing
Let’s connect on LinkedIn
Analyze Data using Python? Check out my website
— Eric Kleppen