Gentle Introduction To Text Representation - Part - 1 | by Sundaresh Chandran | Towards Data Science

Computers are brilliant when dealing with numbers. They are faster than humans in calculations & decoding patterns by many orders of magnitude. But what if the data is not numerical? What if it's language? What happens when the data is in characters, words & sentences? How do we make computers process our language? How does Alexa, Google Home & many other smart assistants understand & reply to our speech? If you are looking to get answers to some of these questions, this article will be a stepping stone for you in the right direction.
Natural Language Processing is a sub-field of artificial intelligence that works on making machines understand & process human language. The most basic step for the majority of natural language processing (NLP) tasks is to convert words into numbers for machines to understand & decode patterns within a language. We call this step text representation. This step, though iterative, plays a significant role in deciding features for your machine learning model/algorithm.
Text representations can be broadly classified into two sections:
Discrete text representations
Distributed/Continuous text representations
This article will focus on discrete text representations & we will dive into some of the frequently used ones with basic Sklearn implementations.
These are representations where words are represented by their corresponding indexes to their position in a dictionary from a larger corpus or corpora.
Famous representations that fall within this category are:
One-Hot encoding
Bag-of-words representation (BOW)
Basic BOW — CountVectorizer
Advanced BOW — TF-IDF
It is a type of representation that assigns 0 to all elements in a vector except for one, which has a value of 1. This value represents a category of an element.
For example:
If i had a sentence, “I love my dog”, each word in the sentence would be represented as below:
I → [1 0 0 0], love → [0 1 0 0], my → [0 0 1 0], dog → [0 0 0 1]
The entire sentence is then represented as:
sentence = [ [1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1] ]
The intuition behind one-hot encoding is that each bit represents a possible category & if a particular variable cannot fall into multiple categories, then a single bit is enough to represent it
As you may have grasped, the length of an array of word depends on the vocabulary size. This is not scalable for a very large corpus which could contain up to 100,000 unique words or even more.
Lets now implement this using Sklearn:
from sklearn.preprocessing import OneHotEncoderimport itertools# two example documentsdocs = ["cat","dog","bat","ate"]# split documents to tokenstokens_docs = [doc.split(" ") for doc in docs]# convert list of of token-lists to one flat list of tokens# and then create a dictionary that maps word to id of word,all_tokens = itertools.chain.from_iterable(tokens_docs)word_to_id = {token: idx for idx, token in enumerate(set(all_tokens))}# convert token lists to token-id liststoken_ids = [[word_to_id[token] for token in tokens_doc] for tokens_doc in tokens_docs]# convert list of token-id lists to one-hot representationvec = OneHotEncoder(categories="auto")X = vec.fit_transform(token_ids)print(X.toarray())
[[0. 0. 1. 0.] [0. 1. 0. 0.] [0. 0. 0. 1.] [1. 0. 0. 0.]]
scikit-learn.org
Easy to understand & implement
Explosion in feature space if number of categories are very high
The vector representation of words is orthogonal and cannot determine or measure relationship between different words
Cannot measure importance of a word in a sentence but understand mere presence/absence of a word in a sentence
High dimensional sparse matrix representation can be memory & computationally expensive
Bag-of-words representation as the name suggests intutively, puts words in a “bag” & computes frequency of occurrence of each word. It does not take into account the word order or lexical information for text representation
The intuition behind BOW representation is that document having similar words are similar irrespective of the word positioning
The CountVectorizer computes the frequency of occurrence of a word in a document. It converts the corpus of multiple sentences (say product reviews) into a matrix of reviews & words & fills it with frequency of each word in a sentence
Lets see how we can use Sklearn CountVectorizer:
from sklearn.feature_extraction.text import CountVectorizertext = ["i love nlp. nlp is so cool"]vectorizer = CountVectorizer()# tokenize and build vocabvectorizer.fit(text)print(vectorizer.vocabulary_)# Output: {'love': 2, 'nlp': 3, 'is': 1, 'so': 4, 'cool': 0}# encode documentvector = vectorizer.transform(text)# summarize encoded vectorprint(vector.shape) # Output: (1, 5)print(vector.toarray())
[[1 1 1 2 1]]
As you see the word “nlp” occurs twice in the sentence & also falls in index 3. Which we can see as the output of the final print statement
The “weight” of a word in a sentence is its frequency
There are various parameters that can be tweaked as part of the CountVectorizer to get the desired results including text preprocessing parameters like lowercase, strp_accents, preprocessor
A complete list of parameters can be found in Sklearn documentation below:
scikit-learn.org
CountVectorizer also gives us frequency of words in a text document/sentence which One-hot encoding fails to provide
Length of the encoded vector is the length of the dictionary
This method ignores the location information of the word. It is not possible to grasp the meaning of a word from this representation
The intuition that high-frequency words are more important or give more information about the sentence fails when it comes to stop words like “is, the, an, I” & when the corpus is context-specific. For example, in a corpus about covid-19, the word coronavirus may not add a lot of value
To suppress the very high-frequency words & ignore the low-frequency words, there is a need to normalize the “weights” of the words accordingly
TF-IDF representation: The full form of TF-IDF is term frequency-inverse document frequency is a product of 2 factors
Where, TF(w, d) is frequency of word ‘w’ in document ‘d’
IDF(w) can be further broken down as:
Where, N is total number of documents, & df(w) is the frequency of documents containing the word ‘w’
Intuition behind TF-IDF is that the weight assigned to each word not only depends on a words frequency, but also how frequent that particular word is in the entire corpus/corpora
It takes the CountVectorizer discussed in the section above & multiplies it by the IDF score. The resultant output weights for the words from the process is low for very highly frequent words (like stop-words) & very low frequency words (noise terms)
Lets now try to implement this using Sklearn
from sklearn.feature_extraction.text import TfidfVectorizertext1 = ['i love nlp', 'nlp is so cool', 'nlp is all about helping machines process language', 'this tutorial is on baisc nlp technique']tf = TfidfVectorizer()txt_fitted = tf.fit(text1)txt_transformed = txt_fitted.transform(text1)print ("The text: ", text1)# Output: The text:  ['i love nlp', 'nlp is so cool', # 'nlp is all about helping machines process language', # 'this tutorial is on basic nlp technique']idf = tf.idf_print(dict(zip(txt_fitted.get_feature_names(), idf)))
{'about': 1.916290731874155, 'all': 1.916290731874155, 'basic': 1.916290731874155, 'cool': 1.916290731874155, 'helping': 1.916290731874155, 'is': 1.2231435513142097, 'language': 1.916290731874155, 'love': 1.916290731874155, 'machines': 1.916290731874155, 'nlp': 1.0, 'on': 1.916290731874155, 'process': 1.916290731874155, 'so': 1.916290731874155, 'technique': 1.916290731874155, 'this': 1.916290731874155, 'tutorial': 1.916290731874155}
Note the weightage of word ‘nlp’. Since it is present in all the sentences, it is given a low weightage of 1.0. Similarly, the stop-word ‘is’ is also given a comparatively low weightage of 1.22 since it is present in 3 out of 4 sentences given.
Similar to CountVectorizer, there are various parameters that can be tweaked to achieve desired results. Some of the important parameters (apart from the text preprocessing parameters like lowercase, strip_accent, stop_words etc.) are max_df, min_df, norm, ngram_range & sublinear_tf. The impact of these parameters on outputs weights is beyond scope of this article & will be covered separately.
You can find the complete documentation of TF-IDF vectorizer below:
scikit-learn.org
Simple, easy to understand & interpret implementation
Builds over CountVectorizer to penalise highly frequent words & low frequency terms in a corpus. So in a way, IDF achieves in reducing noise in our matrix.
Positional information of the word is still not captured in this representation
TF-IDF is highly corpus dependent. A matrix representation generated out of cricket data cannot be used for football or volleyball. Therefore, there is a need to have high quality training data
Discrete representation is where each word is considered unique & converted into numerical based on various techniques we discussed above. We’ve seen few overlapping advantages & disadvantages across various discrete representations, lets summarise it as a whole
Simple representations that are easy to understand, implement & interpret
Algorithms like TF-IDF can be used to filter out uncommon & irrelevant words easily helping model train & converge faster
The representation is directly proportional to vocabulary size. High vocabulary size can lead to memory constraints
It does not leverage co-occurrence statistics between words. It assumes all words are independent of each other
This leads to highly sparse vectors with few non zero values
They do not capture the context or semantics of the word. It does not consider spooky & scary as similar but as two independent terms with no commonality between them
Discrete representations are widely used across both classical machine learning & deep learning applications for solving complicated use cases like document similarity, sentiment classification, spam classification & topic modeling to name a few.
In the next part, we will discuss distributed or a continuous text representation of text & how it is better (or worse) than discrete representations.
Hope you enjoyed this article. See you soon!
Edit: Part-2 of the series: https://medium.com/@sundareshchandran/introduction-to-text-representations-for-language-processing-part-2-54fe6907868
github.com
Like my article? Buy me a coffee