Calculate Similarity — the most relevant Metrics in a Nutshell | by Marvin Lüthe | Towards Data Science

Many data science techniques are based on measuring similarity and dissimilarity between objects. For example, K-Nearest-Neighbors uses similarity to classify new data objects. In Unsupervised Learning, K-Means is a clustering method which uses Euclidean distance to compute the distance between the cluster centroids and it’s assigned data points. Recommendation engines use neighborhood based collaborative filtering methods which identify an individual’s neighbor based on the similarity/dissimilarity to the other users.
In this blog post I will take a look at the most relevant similarity metrics in practice. Measuring similarity between objects can be performed in a number of ways.
Generally we can divide similarity metrics into two different groups:
Similarity Based Metrics:
Similarity Based Metrics:
Pearson’s correlation
Spearman’s correlation
Kendall’s Tau
Cosine similarity
Jaccard similarity
2. Distance Based Metrics:
Euclidean distance
Manhattan distance
Similarity based methods determine the most similar objects with the highest values as it implies they live in closer neighborhoods.
Correlation is a technique for investigating the relationship between two quantitative, continuous variables, for example, age and blood pressure. Pearson’s correlation coefficient is a measure related to the strength and direction of a linear relationship. We calculate this metric for the vectors x and y in the following way:
where
The Pearson’s correlation can take a range of values from -1 to +1. Only having an increase or decrease that are directly related will not lead to a Pearson’s correlation of 1 or -1.
Implementation in Python:
import numpy as npfrom scipy.stats import pearsonrimport matplotlib.pyplot as plt# seed random number generatornp.random.seed(42)# prepare datax = np.random.randn(15)y = x + np.random.randn(15)# plot x and yplt.scatter(x, y)plt.plot(np.unique(x), np.poly1d(np.polyfit(x, y, 1))(np.unique(x)))plt.xlabel('x')plt.ylabel('y')plt.show()
# calculate Pearson's correlationcorr, _ = pearsonr(x, y)print('Pearsons correlation: %.3f' % corr)
Pearsons correlation: 0.810
Spearman’s correlation is what is known as non-parametric statistic, which is a statistic who’s distribution does not depend on parameters (statistics that follow normal distributions or binomial distributions are examples of parametric statistics). Very often, non-parametric statistics rank the data instead of taking the originial values. This is true for Spearman’s correlation coefficient, which is calculated similarly to Pearson’s correlation. The difference between these metrics is that Spearman’s correlation uses the rank of each value.
To calculate Spearman’s correlation we first need to map each of our data to ranked data values:
If the raw data are [0, -5, 4, 7], the ranked values will be [2, 1, 3, 4]. We can calculate Spearman’s correlation in the following way:
where
Spearman’s correlation benchmarks monotonic relationships, therefore it can have perfect relationships that are not linear. It can take a range of values from -1 to +1. The following plot clarifies the difference between Pearson’s and Spearman’s correlation.
For data exploration, I recommend calculating both Pearson’s and Spearman’s correlation. The comparison of both can result in interesting findings. If S>P (as shown above), it means that we have a monotonic relationship, not a linear relationship. Since linearity simplifies the process of fitting a regression algorithm to the dataset, we might want to modify the non-linear, monotonic data using log-transformation to appear linear.
Implementation in Python:
from scipy.stats import spearmanr# calculate Spearman's correlationcorr, _ = spearmanr(x, y)print(‘Spearmans correlation: %.3f’ % corr)
Spearmans correlation: 0.836
Kendall’s tau is quite similar to Spearman’s correlation coefficient. Both of these measures are non-parametric measures of a relationship. Specifically both Spearman and Kendall’s coefficients are calculated based on ranking data and not the raw data.
Similar to Pearson’s and Spearman’s correlation, Kendall’s Tau is always between -1 and +1 , where -1 suggests a strong, negative relationship between two variables and 1 suggests a strong, positive relationship between two variables.
Although Spearman’s and Kendall’s measures are very similar, there are statistical advantages to choosing Kendall’s measure in that Kendall’s Tau has smaller variability when using larger sample sizes. However, Spearman’s measure is more computationally efficient, as Kendall’s Tau is O(n2) and Spearman’s correlation is O(nLog(n)).
The first step is to rank the data to ranks:
Then we calculate Kendall’s Tau as:
where sgn takes the sign associated with the differences in the ranked values. We could write
as follows:
The possible results of
are -1, 0, +1. These are summed to give an idea of the proportion of times the ranks of x and y are pointed in the same direction.
Implementation in Python
# calculate Pearson’s correlationfrom scipy.stats import kendalltaucorr, _ = kendalltau(x, y)print(‘Kendalls tau: %.3f’ % corr)
Kendalls tau: 0.695
The cosine similarity calculates the cosine of the angle between two vectors. In order to calculate the cosine similarity we use the following formula:
Recall the cosine function: on the left the red vectors point at different angles and the graph on the right shows the resulting function.
Accordingly, the cosine similarity can take on values between -1 and +1. If the vectors point in the exact same direction, the cosine similarity is +1. If the vectors point in opposite directions, the cosine similarity is -1.
The cosine similarity is very popular in text analysis. It is used to determine how similar documents are to one another irrespective of their size. The TF-IDF text analysis technique helps converting the documents into vectors where each value in the vector corresponds to the TF-IDF score of a word in the document. Each word has its own axis, the cosine similarity then determines how similar the documents are.
Implementation in Python
We need to reshape the vectors x and y using .reshape(1, -1) to compute the cosine similarity for a single sample.
from sklearn.metrics.pairwise import cosine_similaritycos_sim = cosine_similarity(x.reshape(1,-1),y.reshape(1,-1))print('Cosine similarity: %.3f' % cos_sim)
Cosine similarity: 0.773
Cosine similarity is for comparing two real-valued vectors, but Jaccard similarity is for comparing two binary vectors (sets).
In set theory it is often helpful to see a visualization of the formula:
We can see that the Jaccard similarity divides the size of the intersection by the size of the union of the sample sets.
Both Cosine similarity and Jaccard similarity are common metrics for calculating text similarity. Calculating the Jaccard similarity is computationally more expensive as it matches all the terms of one document to another document. The Jaccard similarity turns out to be useful by detecting duplicates.
Implementation in Python
from sklearn.metrics import jaccard_scoreA = [1, 1, 1, 0]B = [1, 1, 0, 1]jacc = jaccard_score(A,B)print(‘Jaccard similarity: %.3f’ % jacc)
Jaccard similarity: 0.500
Distance based methods prioritize objects with the lowest values to detect similarity amongst them.
The Euclidean distance is a straight-line distance between two vectors.
For the two vectors x and y, this can be computed as follows:
Compared to the Cosine and Jaccard similarity, Euclidean distance is not used very often in the context of NLP applications. It is appropriate for continuous numerical variables. Euclidean distance is not scale invariant, therefore scaling the data prior to computing the distance is recommended. Additionally, Euclidean distance multiplies the effect of redundant information in the dataset. If I had five variables which are heavily correlated and we take all five variables as input, then we would weight this redundancy effect by five.
Implementation in Python
from scipy.spatial import distancedst = distance.euclidean(x,y)print(‘Euclidean distance: %.3f’ % dst)
Euclidean distance: 3.273
Different from Euclidean distance is the Manhattan distance, also called ‘cityblock’, distance from one vector to another. You can imagine this metric as a way to compute the distance between two points when you are not able to go through buildings.
We calculate the Manhattan distance as follows:
The green line gives you the Euclidean distance, while the purple line gives you the Manhattan distance.
In many ML applications Euclidean distance is the metric of choice. However, for high dimensional data Manhattan distance is preferable as it yields more robust results.
Implementation in Python
from scipy.spatial import distancedst = distance.cityblock(x,y)print(‘Manhattan distance: %.3f’ % dst)
Manhattan distance: 10.468
This blog post provided an overview of the most relevant similarity metrics used in practice. There is no simple if-then flowchart for choosing the appropriate similarity metric. We will first need to understand and study the data. Then, it is always a case by case decision to find the right way to quantify similarity for a given data science problem.