How to decide the perfect distance metric for your machine learning model | by Harsh Maheshwari | Towards Data Science

Disclaimer: You won’t need a distance metric for every ML model, but if you do then read on to pick the best one.
Distance metrics play a significant role in machine learning and deep learning. Machine learning algorithms like k-NN, K Means clustering, and loss functions used in deep learning depend on these metrics.
Thus, understanding the different types of distance metrics is very important to decide which metric to use when. For example, k-NN often uses euclidean distance for learning. However, what if the data is highly dimensional? Will euclidean distance still be valuable? No, it won’t because, as we know, euclidean distance is not considered a good metric for highly dimensional space(refer to this link for more insight). So I guess you can relate now that knowing your distance measures can help you go from a poor classifier to an accurate model.
In this blog, we will walk through some of the most used Distance metrics and their use case and disadvantages, and how to implement them in python. The ones we will cover are as follows-:
Euclidean Distance
Manhattan Distance
Chebyshev Distance
Minkowski Distance
Hamming Distance
Cosine Similarity
Jaccard Similarity
Sørensen-Dice Index
Euclidean Distance is one of the most commonly used distance metrics. Mathematically it is the square root of the sum of differences between two different data points. Following is the formula for calculating the distance between two k dimension vectors.
Applications/Pros-:
Highly intuitive, easy to understand, and an easy to use metric
It should be used in algorithms like KNN or K Mean, where we have low dimensional data, and straight forward distance between data points is enough to gauge the similarities of these points.
Disadvantages-:
It can be influenced by the scale of input features, which means that the distance computed might get skewed depending on the units of each feature. Hence, it’s essential to normalize the input feature.
Not effective/useful for high dimensional data
Function to calculate Euclidean Distance in python:
from math import sqrtdef euclidean_distance(a, b):    return sqrt(sum((e1-e2)**2 for e1, e2 in zip(a,b)))#ORfrom scipy.spatial.distance import euclideandist = euclidean(row1, row2)print(dist)
It is often referred to as Taxicab distance or City Block distance. Imagine you are in a city block with two sets of parallel roads, horizontal and vertical, as shown. So, if you want to travel from one point to another, you can only move at right angles.
The distance that is then calculated between two points represents the Manhattan distance.
Application/Pros-:
When discrete/binary attributes are present in the dataset, the manhattan distance metric is more effective as it accounts for the path that can be realistically taken from the given values of attributes.
Disadvantages -:
It does not represent optimal distance in the case of floating attributes in our dataset.
For high dimensional data, it works better than Euclidean distance, but it’s still not the best option performance-wise.
Function to calculate Manhattan Distance in python:
def manhattan_distance(a, b):    return sum(abs(e1-e2) for e1, e2 in zip(a,b))#ORfrom scipy.spatial.distance import cityblockdist = cityblock(row1, row2)print(dist)
Chebyshev distance is defined as the maximum difference between two vectors among all coordinate dimensions. In other words, it is simply the maximum distance along each axis
Application/Pros-:
This metric is usually used for logistical problems. For example, to calculate minimum steps required for a vehicle to go from one place to another, given that the vehicle moves in a grid and thus has only eight possible directions (top, top-right, right, right-down, down, down-left, left, left-top)
Disadvantages-:
It can only be used for particular problems. It can’t be applied for any general-purpose problem like euclidean can.
Function to calculate Chebyshev Distance in python:
def chebyshev_distance(a, b):    return max(abs(e1-e2) for e1, e2 in zip(a,b))#ORfrom scipy.spatial.distance import chebyshevdist = chebyshev(row1, row2)print(dist)
Minkowski generalizes all the above-discussed distance metrics like Euclidean, Manhattan, and Chebyshev. It is also called p-norm vector as it adds a parameter called the “p” that allows different distance measures to be calculated. The formula is -:
For p=1 — Manhattan Distance
For p=2 — Euclidean Distance
For p=infinity — Chebyshev Distance
Since this is a more generic distance metric, different values of p will cause different pros and cons. We have already discussed some of them above.
Function to calculate Minkowski Distance in python:
def minkowski_distance(a, b, p):    return sum(abs(e1-e2)**p for e1, e2 in zip(a,b))**(1/p)#ORfrom scipy.spatial import minkowski_distancedist = minkowski_distance(row1, row2)print(dist)
It is the simplest one so far. It is equal to the number of values that are different between two data points. So let’s say the two data points x and y are as follows-:
x = [1,2,3,0,2,4]
y = [1,3,3,0,1,4]
Then hamming distance = 2 as for index (assuming indices start from 0) 1 and 4 the values are different in x and y. It is usually used for calculating the distance between two binary strings.
Advantages/Pros-:
It is generally used in error detection when data is sent from one computer to another computer.
Disadvantages-:
Since it’s used to find the difference between two data points by comparing each value, it is mandatory that they have equal length.
It does not incorporate the actual value/magnitude of the data point. Hence it’s not advised to use when the magnitude of feature plays an important role.
Function to calculate Hamming Distance in python:
def hamming_distance(a, b):    return sum(abs(e1 - e2) for e1, e2 in zip(a, b)) / len(a)#ORfrom scipy.spatial.distance import hammingdist = hamming(row1, row2)print(dist)
It is also one of the most commonly used distance metrics. It is used to find the similarity between two vectors/data points by calculating the cosine angle between them.
For more understanding, if two vectors coincide or have the same orientation, then cosine similarity between them would be 1, and if they are exactly opposite, it would be -1.
Advantages/Pros-:
Works excellent with high-dimensional data and should be used for them ideally.
It is used to find the similarity between many things like text embedding, picture embedding, or fingerprint embedding.
Disadvantages-:
Similar to the hamming distance, the magnitude of the vector is not taken into account in cosine similarity; only their direction is considered.
Function to calculate Cosine Similarity in python:
from numpy import dotfrom numpy.linalg import normdef cosine_similarity(a,b):    return dot(a, b)/(norm(a)*norm(b))#ORfrom scipy.spatial.distance import cosinedist = 1 - cosine(row1, row2)print(dist)
Jaccard similarity is used to calculate similarity, and it emphasizes on the similarity between two finite sample sets instead of vectors. It is defined as the size of the intersection of the sets, divided by the size of the union of these sets.
For example, if two sets have two entities in common and have a total of 7 unique entities, then the similarity will be 2/7.
Advantages/Pros-:
The Jaccard similarity is used as a metric to evaluate image segmentation models in deep learning, where the model has to classify each pixel of the image. It is used to calculate how accurately we have segmented the required entity compared to the ground truth. It can similarly be used in other applications.
Disadvantages-:
Jaccard similarity is highly dependent on the size of the data. Large datasets can significantly impact the similarity, as in this case, the union could increase substantially while the intersection stays low.
Function to calculate Jaccard Similarity in python:
import numpy as npdef jaccard_similarity(x,y):    intersection = np.logical_and(x, y)    union = np.logical_or(x, y)    similarity = intersection.sum() / float(union.sum())    return similarity#ORfrom sklearn.metrics import jaccard_scoresimilarity = jaccard_score(a, b)print(similarity)
The Sørensen-Dice index is very similar to Jaccard similarity. The Sørensen-Dice index is a bit more intuitive because it can be seen as the percentage of overlap between two sets, which is a value between 0 and 1:
The Sørensen-Dice index is also used as a metric for image segmentation.
In this blog, we covered various distance metrics that are widely used in practice. Each metric is advantageous in some situations and disadvantageous in others. Hopefully, this blog will help you decide which metric would better fit your use case.
Follow us on medium for more such content.
Become a Medium member to unlock and read many other stories on medium.