Similarity Measures — Scoring Textual Articles | by Saif Ali Kheraj | Towards Data Science

Many real-world applications make use of similarity measures to see how two objects are related together. We can use these measures in the applications involving Computer vision and Natural Language Processing, for example, to find and map similar documents. One important use case here for the business would be to match resumes with the Job Description saving a considerable amount of time for the recruiter. Another important use case would be to segment different customers for marketing campaigns using the K Means Clustering algorithm which also uses similarity measures.
Similarities are usually positive ranging between 0 (No Similarity) and 1 (Complete Similarity). We will specifically discuss two important similarity metric namely euclidean and cosine along with the coding example to deal with Wikipedia articles.
Do you remember Pythagoras Theorem?? Pythagoras Theorem is used to calculate the distance between two points as indicated in the figure below.
In the figure, we have two data points (x1,y1) and (x2,y2) and we are interested in calculating the distance or closeness between these two points. To find the distance, we need to first go horizontally from x1 to x2 and then go vertically up from y1 to y2. This makes up a right-angled triangle. We are interested in calculating hypotenuse d which we can calculate easily using the Pythagoras theorem.
where b is the base of the right-angled triangle and p is the perpendicular of the right-angled triangle.
This completes our euclidean distance formula for two points in two-dimensional space.
This defines the euclidean distance between two points in one, two, three or higher-dimensional space where n is the number of dimensions and x_k and y_k are components of x and y respectively.
Python Function to define euclidean distance
def euclidean_distance(x, y):       return np.sqrt(np.sum((x - y) ** 2))
Here x and y are the two vectors.
You can also use sklearn library to calculate the euclidean distance. This function is computationally more efficient.
from sklearn.metrics.pairwise import euclidean_distances
Greater the distance, lower the similarity between the two objects; Lower the distance, higher the similarity between the two objects. To convert this distance metric into the similarity metric, we can divide the distances of objects with the max distance, and then subtract it by 1 to score the similarity between 0 and 1. We will look at the example after discussing the cosine metric.
This is another metric to find the similarity specifically for the documents. This metric is a measure of the angle between x and y as indicated in the diagram and is used when the magnitude of the vector does not matter.
If the angle between v and w is 0 degree, then the cosine similarity =1 (Complete Similarity).
Cosine Formula for dot Product:
‖⋅‖ denotes vector length
θ: Angle between v and w. Here we are interested in measuring the similarity between v and w.
If you want to know how to derive the equation for cosine formula for dot product, please refer to this page: https://proofwiki.org/wiki/Cosine_Formula_for_Dot_Product. This dot product formula is derived from the same Law of Cosine which you have studied in your schools.
Let’s use the notation x and y instead of v and w.
Example:
These two vectors have low similarity explained by the value of 0.11. This value is close to 0 which means that the angle between x and y is close to 90 degrees. If the value would have been close to 1, then this would be very similar objects with an angle close to 0 degrees.
Dividing x and y by their lengths normalizes its length to 1 making it what is known as Unit Vector. This tells that cosine similarity does not take into account the magnitude of x and y. When we need to take into account the magnitude, euclidean might be a better option.
If we already have vectors with a length of 1, cosine similarity can be easily calculated using simple dot product. It is therefore recommended to normalize vectors first to have a unit length to reduce the computation time.
Python Function to define Cosine Similarity
def cosine_similarity(x, y):    return np.dot(x, y) / (np.sqrt(np.dot(x, x)) * np.sqrt(np.dot(y, y)))
The diagram summarizes both the Euclidean and Cosine Metric.
Cosine looks at the angle between the two vectors ignoring magnitude while the Euclidean focuses on the straight line distance taking into account the magnitude of the vector.
Text mining is one of those areas where we can make use of cosine similarity to map similar documents. We can also use it to rank documents with respect to the given vector of words, CV shortlisting is one of those use cases where we can make use of Cosine similarity.
Let’s take the problem where we need to match similar documents, we first create the Document Term Matrix that contains the frequency of terms occurring in a collection of documents. We usually have documents of uneven lengths as in the case of Wikipedia articles. Let’s say the word math appeared more in Document 1 than it does in document 2, cosine similarity, in this case, would be a perfect choice as we are not concerned about the length of the document but the content it contains. If we were to consider the length, then the Euclidean might be a perfect option.
import wikipediaarticles=['Data Mining','Machine Learning','Cricket','Swimming','Tennis']wiki_lst=[]for article in articles:    print(article)    wiki_lst.append(wikipedia.page(article).content)
Here we have just fetched contents from the 5 Wikipedia articles.
from sklearn.feature_extraction.text import CountVectorizercv = CountVectorizer()X=cv.fit_transform(wiki_lst)
Count Vectorizer just converts the collection of documents into the matrix of word counts.
Lets now use the euclidean metric to find out which of our articles are similar. Here we have normalized these euclidean distances between 0 and 1 and then subtracted it by 1 to range it between 0 and 1. Hence, closer it is to 1, higher the similarity.
from sklearn.metrics.pairwise import euclidean_distancesprint("Data Mining and Machine Learning",1-euclidean_distances(X[0],X[1])/np.max((euclidean_distances(X))))print("Data Mining and Cricket",1-euclidean_distances(X[0],X[2])/np.max((euclidean_distances(X))))print("Data Mining and Swimming",1-euclidean_distances(X[0],X[3])/np.max((euclidean_distances(X))))print("Data Mining and Tennis",1-euclidean_distances(X[0],X[4])/np.max((euclidean_distances(X))))
How are these two articles(Data Mining Swimming) most similar? It does not make any sense. Lets now try Cosine Similarity.
from sklearn.metrics.pairwise import cosine_similarityprint("Data Mining and Machine Learning",cosine_similarity(X[0],X[1]))print("Data Mining and Cricket",cosine_similarity(X[0],X[2]))print("Data Mining and Swimming",cosine_similarity(X[0],X[3]))print("Data Mining and Tennis",cosine_similarity(X[0],X[4]))
This is what makes sense to us. Data Mining is very close to Machine Learning.
The Euclidean metric here seems to focus on length rather than the content while cosine concentrated on the content ignoring the magnitude.