Importance of Distance Metrics in Machine Learning Modelling | by Natasha Sharma | Towards Data Science

A number of Machine Learning Algorithms - Supervised or Unsupervised, use Distance Metrics to know the input data pattern in order to make any Data Based decision. A good distance metric helps in improving the performance of Classification, Clustering and Information Retrieval process significantly. In this article, we will discuss about different Distance Metrics and how do they help in Machine Learning Modelling.
In many real world applications, we use Machine Learning algorithms for classifying or recognizing images and for retrieving information through an Image’s content. For example - Face recognition, Censored Images online, Retail Catalog, Recommendation Systems etc. Choosing a good distance metric becomes really important here. The distance metric helps algorithms to recognize similarities between the contents.
Basic Mathematics Definition(Source Wikipedia),
Distance metric uses distance function which provides a relationship metric between each elements in the dataset.
Some of you might be thinking, what is this distance function? how does it work? how does it decide that a particular content or element in the data has any kind of relationship with another one? Well let’s try and find this out in next couple of sections.
Do you remember studying Pythagorean theorem? If you do, then you might remember calculating distance between two data points using the theorem.
In order to calculate the distance between data points A and B Pythagorean theorem considers the length of x and y axis.
Many of you must be wondering that, do we even use this theorem in machine learning algorithm to find the distance? To answer your question, yes we do use it. In many machine learning algorithms we use the above formula as a distance function. We will talk about the algorithms where it is used.
Now you probably have got an idea what is a distance function? Here is a simplified definition.
Basic Definition from Math.net,
A distance function provides distance between the elements of a set. If the distance is zero then elements are equivalent else they are different from each other.
A distance function is nothing but a mathematical formula used by distance metrics. The distance function can differ across different distance metrics. Let’s talk about different distance metrics and understand their role in machine learning modelling.
There are a number of distance metrics, but to keep this article concise, we will only be discussing a few widely used distance metrics. We will first try to understand the mathematics behind these metrics and then we will identify the machine learning algorithms where we use these distance metrics.
Below are the commonly used distance metrics -
Minkowski distance is a metric in Normed vector space. What is Normed vector space? A Normed vector space is a vector space on which a norm is defined. Suppose X is a vector space then a norm on X is a real valued function ||x||which satisfies below conditions -
Zero Vector- Zero vector will have zero length.Scalar Factor- The direction of vector doesn’t change when you multiply it with a positive number though its length will be changed.Triangle Inequality- If distance is a norm then the calculated distance between two points will always be a straight line.
Zero Vector- Zero vector will have zero length.
Scalar Factor- The direction of vector doesn’t change when you multiply it with a positive number though its length will be changed.
Triangle Inequality- If distance is a norm then the calculated distance between two points will always be a straight line.
You might be wondering why do we need normed vector, can we just not go for simple metrics? As normed vector has above properties which helps to keep the norm induced metric- homogeneous and translation invariant. More details can be found here.
The distance can be calculated using below formula -
Minkowski distance is the generalized distance metric. Here generalized means that we can manipulate the above formula to calculate the distance between two data points in different ways.
As mentioned above, we can manipulate the value of p and calculate the distance in three different ways-
p = 1, Manhattan Distance
p = 2, Euclidean Distance
p = ∞, Chebychev Distance
We will discuss these distance metrics below in detail.
Manhattan Distance:
We use Manhattan Distance if we need to calculate the distance between two data points in a grid like path. As mentioned above, we use Minkowski distance formula to find Manhattan distance by setting p’s value as 1.
Let’s say, we want to calculate the distance, d, between two data points- x and y.
Distance d will be calculated using an absolute sum of difference between its cartesian co-ordinates as below :
where, n- number of variables, xi and yi are the variables of vectors x and y respectively, in the two dimensional vector space. i.e. x = (x1,x2,x3,...) and y = (y1,y2,y3,...).
Now the distance d will be calculated as-
(x1 - y1) + (x2 - y2) + (x3 - y3) + ... + (xn - yn).
If you try to visualize the distance calculation, it will look something like as below :
Manhattan distance is also known as Taxicab Geometry, City Block Distance etc.
Euclidean Distance:
Euclidean distance is one of the most used distance metric. It is calculated using Minkowski Distance formula by setting p’s value to 2. This will update the distance ‘d’ formula as below :
Let’s stop for a while! Does this formula look familiar? Well yes, we just saw this formula above in this article while discussing “Pythagorean Theorem”.
Euclidean distance formula can be used to calculate the distance between two data points in a plane.
Mostly Cosine distance metric is used to find similarities between different documents. In cosine metric we measure the degree of angle between two documents/vectors(the term frequencies in different documents collected as metrics). This particular metric is used when the magnitude between vectors does not matter but the orientation.
Cosine similarity formula can be derived from the equation of dot products :-
Now, you must be thinking which value of cosine angle will be helpful in finding out the similarities.
Now that we have the values which will be considered in order to measure the similarities, we need to know what do 1, 0 and -1 signify.
Here cosine value 1 is for vectors pointing in the same direction i.e. there are similarities between the documents/data points. At zero for orthogonal vectors i.e. Unrelated(some similarity found). Value -1 for vectors pointing in opposite directions(No similarity).
Mahalanobis Distance is used for calculating the distance between two data points in a multivariate space.
According to Wikipedia Definition,
The Mahalanobis distance is a measure of the distance between a point P and a distribution D. The idea of measuring is, how many standard deviations away P is from the mean of D.
The benefit of using mahalanobis distance is, it takes covariance in account which helps in measuring the strength/similarity between two different data objects. The distance between an observation and the mean can be calculated as below -
Here, S is the covariance metrics. We are using inverse of the covariance metric to get a variance-normalized distance equation.
Now that we have a basic idea about different distance metrics, we can move to the next step i.e. machine learning techniques/modelling which use these disatance metrics.
In this section, we will be working on some basic classification and clustering use cases. This will help us in understanding the usage of distance metrics in machine learning modelling. We will start with quick introduction of supervised and unsupervised algorithms and slowly will move on to the examples.
K-Nearest Neighbors(KNN)-
KNN is a non-probabilistic supervised learning algorithm i.e. it doesn’t produce the probability of membership of any data point rather KNN classifies the data on hard assignment, e.g the data point will either belong to 0 or 1. Now, you must be thinking how does KNN work if there is no probability equation involved. KNN uses distance metrics in order to find similarities or dissimilarities.
Let’s take iris dataset which has three classes and see how KNN will identify the classes for test data.
In the #2 image above the black square is a test data point. Now, we need to find which class this test data point belong to, with the help of KNN algorithm. We will now prepare the dataset to create machine learning model to predict the class for our test data.
#Import required libraries#Import required librariesimport numpy as npimport pandas as pdfrom sklearn.model_selection import train_test_splitfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.metrics import accuracy_score#Load the dataseturl = "https://raw.githubusercontent.com/SharmaNatasha/Machine-Learning-using-Python/master/Datasets/IRIS.csv"df = pd.read_csv(url)#quick look into the datadf.head(5)#Separate data and labelx = df.iloc[:,1:4]y = df.iloc[:,4]#Prepare data for classification processx_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=0)
In KNN classification algorithm, we define the constant “K”. K is the number of nearest neighbours of a test data point. These K data points then will be used to decide the class for test data point.(Note this is in a training data set)
Are you wondering that how would we find the nearest neighbours. Well that’s where the distance metric comes into pictures. First, we calculate the distance between each train and test data point and then select the top nearest according to the value of k.
We won’t be creating the KNN from scratch but will be using scikit KNN classifier.
#Create a modelKNN_Classifier = KNeighborsClassifier(n_neighbors = 6, p = 2, metric='minkowski')
You can see in the above code we are using Minkowski distance metric with value of p as 2 i.e. KNN classifier is going to use Euclidean Distance Metric formula.
As we move forward with machine learning modelling we can now train our model and start predicting the class for test data.
#Train the modelKNN_Classifier.fit(x_train, y_train)#Let's predict the classes for test datapred_test = KNN_Classifier.predict(x_test)
Once the top nearest neighbours are selected, we check most voted class in neighbours -
From the above image, can you guess the class for test point? It’s class 1 as it is most voted class.
Through this small example we saw how distance metric was important for KNN classifier. It helped us to get the closest train data points for which classes were known. There is a possibility that using different distance metrics we might get better results. So, in non-probabilistic algorithm like KNN distance metrics plays an important role.
K-means-
In classification algorithms, probabilistic or non-probabilistic we will be provided with labeled data so, it gets easier to predict the classes. Though in clustering algorithm we have no information on which data point belongs to which class. Distance metrics are important part of these kind of algorithm.
In K-means, we select number of centroids that define number of clusters. Each data point will then be assigned to its nearest centroid using distance metric (Euclidean). We will be using iris data to understand the underlying process of K-means.
In the above image #1 as you can see we randomly placed the centroids and in the image #2, using distance metric tried to find their closest cluster class.
import numpy as npimport pandas as pdfrom sklearn.cluster import KMeansimport matplotlib.pyplot as plt#Load the dataseturl = "https://raw.githubusercontent.com/SharmaNatasha/Machine-Learning-using-Python/master/Datasets/IRIS.csv"df = pd.read_csv(url)#quick look into the datadf.head(5)#Separate data and labelx = df.iloc[:,1:4].values#Creating the kmeans classifierKMeans_Cluster = KMeans(n_clusters = 3)y_class = KMeans_Cluster.fit_predict(x)
We will need to keep repeating the assignment of centroids until we have a clear cluster structure.
As we saw in the above example, without having any knowledge about the labels with the help of distance metric in K-Means we clustered the data into 3 classes.
Information Retrieval
In information retrieval we work with unstructured data. The data can be an article, website, emails, text messages, a social media post etc. With the help of techniques used in NLP we can create vector data in a manner that can be used to retrieve information when queried. Once the unstructured data is transformed into vector form, we can use cosine similarity metric to filter out the irrelevant documents from the corpus.
Let’s take an example and understand the usage of cosine similarity.
Create vector form for Corpus and Query-
Create vector form for Corpus and Query-
import mathimport numpy as npimport pandas as pdimport matplotlib.pyplot as pyplotfrom sklearn.metrics.pairwise import cosine_similarityfrom sklearn.feature_extraction.text import TfidfVectorizervectorizer = TfidfVectorizer()corpus = [ 'the brown fox jumped over the brown dog', 'the quick brown fox', 'the brown brown dog', 'the fox ate the dog']query = ["brown"]X = vectorizer.fit_transform(corpus)Y = vectorizer.transform(query)
2. Check the similarities i.e find which document in corpus is relevant to our query-
cosine_similarity(Y, X.toarray())Results:array([[0.54267123, 0.44181486, 0.84003859, 0.        ]])
As you can see from the above example, we queried for word “brown” and in corpus there are only three documents which contain word “brown”. When checked with cosine similarity metric it gave the same results by having >0 values for three document except the forth one.
Through out this article, we got to know about few popular distance/similarity metrics and how these can be used in order to solve complicated machine learning problems. Hope this will be helpful for people who are in their first stage of getting into Machine Learning/Data Science.
Cosine Similarity- Sklearn, TDS article, Wikipedia, ExampleGithub codeDistance Metrics- Math.net, WikiMinkowski Distance Metric- Wiki, Blog, Famous Metrics
Cosine Similarity- Sklearn, TDS article, Wikipedia, Example
Github code
Distance Metrics- Math.net, Wiki
Minkowski Distance Metric- Wiki, Blog, Famous Metrics