A Guide To Encoding Text In Python | by Kurtis Pykes | Towards Data Science

The reason we encode text is that computers do not understand characters, words, or sentences; Computers can only process numbers, therefore, if we wish to teach computers to understand natural language then we ought to encode the textual data into a numeric format before inputting it into any machine.
Therefore, text encoding may be defined as the process of converting text into meaningful numeric/vector representations. Within this process, we wish to preserve the context and dependencies between words and sentences so that the machine is capable of detecting patterns associated with the text as well as comprehending the context.
There are various ways to encode text:
One-Hot Encoding
Index-based Encoding
Bag of Words (BOW)
Term Frequency — Inverse Document Frequency (TF-IDF)
Word Embeddings
This isn’t the whole list but it’s enough to get started. Each encoding has its own pros and cons and each one may be better suited for one task over another. The most basic encodings such as BOW do not retain the ordering of words, whereas others leverage neural networks like Word2Vec.
Let’s implement these in Python.
One-Hot encodings represent categorical variables as binary vectors. To implement this the categorical values first ought to be mapped to an integer, then each integer is represented as a binary vector — meaning that the vector would be all 0’s except the index of the integer.
# define the documentscorpus = ["i cant wait to get out of lockdown", "the uk is soon going to be free soon", "linkedin is social media"]# converting text to intergerstoken_docs = [doc.split() for doc in corpus]all_tokens = set([word for sentence in token_docs for word in sentence])word_to_idx = {token:idx+1 for idx, token in enumerate(all_tokens)}# converting the docs to their token idstoken_ids = np.array([[word_to_idx[token] for token in token_doc] for token_doc in token_docs], dtype=object)token_ids_padded = pad_sequences(token_ids, padding="post")token_ids = token_ids.reshape(-1, 1)# convert the token ids to one hot representationone_hot = OneHotEncoder()X = one_hot.fit_transform(token_ids_padded)# converting to dataframeX_df = pd.DataFrame(X.toarray())X_df
Each document is represented by a tensor, and each tensor consists of a sequence of 0’s and 1’s. Although One-Hot encodings preserve word order, as documents get longer we end up with very large sparse vectors.
Index-Based encodings also preserve word order. To implement the index-based encoding, you simply map each word with one index. We’d also need to add paddings to each sentence so that they are all equal in length. Based on the mappings within the dictionary, each document is represented via a sequence of indexes, where each number is the encoding of one word.
import numpy as npimport pandas as pdfrom sklearn.preprocessing import OneHotEncoderfrom tensorflow.keras.preprocessing.sequence import pad_sequences# define the documentscorpus = ["i cant wait to get out of lockdown", "the uk is soon going to be free soon", "linkedin is social media"]# converting text to intergerstoken_docs = [doc.split() for doc in corpus]all_tokens = set([word for sentence in token_docs for word in sentence])word_to_idx = {token:idx+1 for idx, token in enumerate(all_tokens)}# converting the docs to their token idsX = np.array([[word_to_idx[token] for token in token_doc] for token_doc in token_docs], dtype=object)# padding the sequencesX_padded = pad_sequences(X, padding="post")# converting to pandas dfX_df = pd.DataFrame(X_padded) 
Note: The column names represent where the words occur within the text
The downfall of Index-Based encodings is that it introduces numerical distances between texts which aren’t real, for example, why should the word “cat” be indexed as number 1 and “dog” is number 324?
The Bag of Words encoding technique derives its name from the fact that any information or structure of the words in a document is discarded, therefore, it’s like you’ve put the set of words into a bag and given it a shake — it only cares if a word appears in a document and the number of times it appeared. It does not care where it appeared.
import pandas as pdfrom sklearn.feature_extraction.text import CountVectorizer# define the documentscorpus = ["i cant wait to get out of lockdown", "the uk is soon going to be free soon", "linkedin is social media"]# implementing BoWbow = CountVectorizer()bow.fit(corpus)X = bow.transform(corpus)# converting to dataframeX_df = pd.DataFrame(X.toarray(), columns=sorted(bow.vocabulary_))X_df
Although an extremely simple model to grasp and implement, the BoW model falls short in many aspects with the most obvious being the loss of meaning since we are discarding the word order — we lose the context the word was used in.
The issue with providing scores to the frequency of words is that highly frequent words we begin to dominate in the document despite not adding much value to the document. For example, words such as “and”, “the”, “a”, etc. are quite common in the English language so it’s common these words would appear more frequently in documents. TF-IDF recognizes this problem and penalizes the words that are most prominent in all of the documents.
import pandas as pdfrom sklearn.feature_extraction.text import TfidfVectorizer# define the documentscorpus = ["i cant wait to get out of lockdown", "the uk is soon going to be free soon", "linkedin is social media"]# implement tfidftfidf = TfidfVectorizer()tfidf.fit(corpus)X = tfidf.transform(corpus)# convert to dataframeX_df = pd.DataFrame(X.toarray(), columns=sorted(tfidf.vocabulary_))X_df.T
However, since the TF-IDF encoding is based on the BOW encodings, it too does not capture the word position in text, semantics, and co-occurrences in different documents, but it is also very easy to compute.
Word embeddings are a great example of transfer learning. Transfer learning is a research problem in machine learning that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem. [Source: Wikipedia].
To go into the full details of how various word embeddings or transfer learning works is beyond the scope of this article but we will still see how to implement a Word2Vec encoding.
from gensim.models import Word2Vec# define the documentscorpus = ["i cant wait to get out of lockdown", "the uk is soon going to be free soon", "linkedin is social media"]# Word2Vec modelw2v = Word2Vec(min_count=1,               size=20,               window=2,               alpha=0.03)# building the vocabw2v.build_vocab(sentences=corpus)# training the modelw2v.train(sentences=corpus, total_examples=w2v.corpus_count, epochs=10)
Note: A dedicated article will be created for transfer learning in NLP.
The purpose of encoding textual data [and all other kinds of data] is to ensure that the data can properly be consumed by a machine. In this article, I’ve covered 5 different ways to encode your text data. I enjoy using distributed representations, such as word embeddings, when the context really matters, but there are many instances where simpler techniques like Bag-of-words have been effective, such as classification tasks.
Thanks for reading! Connect with me on LinkedIn and on Twitter to stay up to date with my latest post on Artificial Intelligence, Data Science, and Freelancing.