3 basic Distance Measurement in Text Mining | by Edward Ma | Towards Data Science

In NLP, we also want to find the similarity among sentence or document. Text is not like number and coordination that we cannot compare the different between “Apple” and “Orange” but similarity score can be calculated.
Since we cannot simply subtract between “Apple is fruit” and “Orange is fruit” so that we have to find a way to convert text to numeric in order to calculate it. Having the score, we can understand how similar among two objects.
In my data science work, I tried:
Compare whether 2 article are describing same news
Identifying similar documents
Classifying the category by giving product description
In this article, we will go through 4 basic distance measurements:
Euclidean Distance
Cosine Distance
Jaccard Similarity
Before any distance measurement, text have to be tokenzied. If you do not familiar with word tokenization, you can visit this article.
Euclidean Distance
Comparing the shortest distance among two objects. It uses Pythagorean Theorem which learnt from secondary school.
Score means the distance between two objects. If it is 0, it means that both objects are identical. The following example shows score when comparing the first sentence.
print('Master Sentence: %s' % news_headlines[0])for i, news_headline in enumerate(news_headlines):    score = sklearn.metrics.pairwise.euclidean_distances([transformed_results[i]], [transformed_results[0]])[0][0]    print('-----')    print('Score: %.2f, Comparing Sentence: %s' % (score, news_headline))
Output
Master Sentence: Elon Musk's Boring Co to build high-speed airport link in Chicago-----Score: 0.00, Comparing Sentence: Elon Musk's Boring Co to build high-speed airport link in Chicago-----Score: 1.73, Comparing Sentence: Elon Musk's Boring Company to build high-speed Chicago airport link-----Score: 4.36, Comparing Sentence: Elon Musk’s Boring Company approved to build high-speed transit between downtown Chicago and O’Hare Airport-----Score: 4.24, Comparing Sentence: Both apple and orange are fruit
Cosine Similarity
Determine the angle between two objects is the calculation method to the find similarity. The range of score is 0 to 1. If score is 1, it means that they are same in orientation (not magnitude). The following example shows score when comparing the first sentence.
print('Master Sentence: %s' % news_headlines[0])for i, news_headline in enumerate(news_headlines):    score = sklearn.metrics.pairwise.cosine_similarity([transformed_results[i]], [transformed_results[0]])[0][0]    print('-----')    print('Score: %.2f, Comparing Sentence: %s' % (score, news_headline))
Output
Master Sentence: Elon Musk's Boring Co to build high-speed airport link in Chicago-----Score: 1.00, Comparing Sentence: Elon Musk's Boring Co to build high-speed airport link in Chicago-----Score: 0.87, Comparing Sentence: Elon Musk's Boring Company to build high-speed Chicago airport link-----Score: 0.44, Comparing Sentence: Elon Musk’s Boring Company approved to build high-speed transit between downtown Chicago and O’Hare Airport-----Score: 0.00, Comparing Sentence: Both apple and orange are fruit
Jaccard Similarity
The measurement is refer to number of common words over all words. More commons mean both objects should be similarity.
Jaccard Similarity = (Intersection of A and B) / (Union of A and B)
The range is 0 to 1. If score is 1, it means that they are identical. There is no any common word between the first sentence and the last sentence so the score is 0. The following example shows score when comparing the first sentence.
print('Master Sentence: %s' % news_headlines[0])for i, news_headline in enumerate(news_headlines):    y_compare = calculate_position(transformed_results[i])    x1, x2 = padding(y_actual, y_compare)    score = sklearn.metrics.jaccard_similarity_score(x1, x2)    print('-----')    print('Score: %.2f, Comparing Sentence: %s' % (score, news_headline))
Output
Master Sentence: Elon Musk's Boring Co to build high-speed airport link in Chicago-----Score: 1.00, Comparing Sentence: Elon Musk's Boring Co to build high-speed airport link in Chicago-----Score: 0.67, Comparing Sentence: Elon Musk's Boring Company to build high-speed Chicago airport link-----Score: 0.17, Comparing Sentence: Elon Musk’s Boring Company approved to build high-speed transit between downtown Chicago and O’Hare Airport-----Score: 0.00, Comparing Sentence: Both apple and orange are fruit
You can found out all code from github.
Three methods also have same assumption which is the document (or sentence) are similar if having common words. This idea is very straight forward and simple. It fits some basic cases such as comparing first 2 sentence. However, the score is relative low by comparing first sentence and third sentence although both of them describe same news.
Another limitation is that above methods does not handle synonym scenario. For example buy and purchase, it should have same meaning (in some cases) but above methods will treat both words are difference.
So what is the cue? You may consider to use Word Embedding which introduced by Tomas Mikolov in 2013.
I am Data Scientist in Bay Area. Focusing on state-of-the-art in Data Science, Artificial Intelligence , especially in NLP and platform related.