K-Nearest Neighbours (kNN) Algorithm: Common Questions and Python Implementation | by Chingis Oinar | Towards Data Science

K-Nearest Neighbours is considered to be one of the most intuitive machine learning algorithms since it is simple to understand and explain. Additionally, it is quite convenient to demonstrate how everything goes visually. However, the kNN algorithm is still a common and very useful algorithm to use for a large variety of classification problems. If you are new to machine learning, make sure you test yourself on an understanding of this simple yet wonderful algorithm. There are a lot of useful sources on what it is and how it works, hence I want to go through 5 common or interesting questions you should know in my personal opinion.
The k-NN algorithm does more computation on test time rather than train time.
That is absolutely true. The idea of the kNN algorithm is to find a k-long list of samples that are close to a sample we want to classify. Therefore, the training phase is basically storing a training set, whereas while the prediction stage the algorithm looks for k-neighbours using that stored data.
Why do you need to scale your data for the k-NN algorithm?
Imagine a dataset having m number of “examples” and n number of “features”. There is one feature dimension having values exactly between 0 and 1. Meanwhile, there is also a feature dimension that varies from -99999 to 99999. Considering the formula of Euclidean Distance, this will affect the performance by giving higher weightage to variables having a higher magnitude.
Read more: Why is scaling required in KNN and K-Means?
The k-NN algorithm can be used for imputing the missing value of both categorical and continuous variables.
That is true. k-NN can be used as one of many techniques when it comes to handling missing values. A new sample is imputed by determining the samples in the training set “nearest” to it and averages these nearby points to impute. A scikit learn library provides a quick and convenient way to use this technique.
Note: NaNs are omitted while distances are calculated.
Example:
from sklearn.impute import KNNImputer# define imputerimputer = KNNImputer() #default k is 5=> n_neighbors=5# fit on the datasetimputer.fit(X)# transform the datasetXtrans = imputer.transform(X)
Thus, missing values will be replaced by the mean value of its “neighbours”.
Is Euclidean Distance always the case?
Although Euclidean Distance is the most common method used and taught, it is not always the optimal decision. In fact, it is hard to come up with the right metric just by looking at data, so I would suggest trying a set of them. However, there are some special cases. For instance, hamming distance is used in case of a categorical variable.
Read more: 3 text distances that every data scientist should know
Why should we not use the KNN algorithm for large datasets?
Here is an overview of the data flow that occurs in the KNN algorithm:
Calculate the distances to all vectors in a training set and store themSort the calculated distancesStore the K nearest vectorsCalculate the most frequent class displayed by K nearest vectors
Calculate the distances to all vectors in a training set and store them
Sort the calculated distances
Store the K nearest vectors
Calculate the most frequent class displayed by K nearest vectors
Imagine you have a very large dataset. Therefore, it is not only a bad decision to store a large amount of data but it is also computationally costly to keep calculating and sorting all the values.
I will break down the implementation into the following stages:
Calculate the distances to all vectors in a training set and store themSort the calculated distancesCalculate the most frequent class displayed by K nearest vectors and make a prediction
Calculate the distances to all vectors in a training set and store them
Sort the calculated distances
Calculate the most frequent class displayed by K nearest vectors and make a prediction
Calculate the distances to all vectors in a training set and store them
It is important to note that there is a large variety of options to choose as a metric; however, I want to use Euclidean Distance as an example. It is the most common metric used to calculate distances among vectors since it is straightforward and easy to explain. The general formula is as follows:
Euclidean Distance = sqrt(sum i to N (x1_i — x2_i)2)
Thus, let’s summarize it with the following Python code. Note: Don’t forget to import sqrt() from “math” module.
Sort the calculated distances
Firstly, we need to calculate all the distances between a single test sample and all the samples in our training set. As distances are obtained, we should sort the list of distances and pick the “nearest” k vectors from our training set by looking at how far they are from a test sample.
Calculate the most frequent class displayed by K nearest vectors and make a prediction
Finally, in order to make a prediction, we should get our k “nearest” neighbours by calling our function I attached above. Thus, the only thing that is left is to count the number of occurrences of each label and pick the most frequent one.
Let’s summarize everything by combining all the function into a separate class object. Here is a more generalized version of the code, please take time to look through it.
Comparison
Let’s compare our implementation with the one provided by scikit learn. I am going to use a simple toy dataset that contains two predictors, which are age and salary. Thus, we want to predict if a customer is willing to purchase our product.
I am going to skip preprocessing since it is not what I want to focus on; however, I used a train-test split technique and applied a StandardScaler afterwards. Anyways, if you are interested, I will provide the source code on Github.
Finally, I will define both models and fit our data. Please refer to the KNN implementation provided above. I am selecting 5 as our default k value. Note that for the latter model the default metric is minkowski, and with p=2 is equivalent to the standard Euclidean metric.
model=KNN(5) #our model model.fit(X_train,y_train)predictions=model.predict(X_test)#our model's predictionsfrom sklearn.neighbors import KNeighborsClassifierclassifier = KNeighborsClassifier(n_neighbors = 5, metric = 'minkowski', p = 2)#The default metric is minkowski, and with p=2 is equivalent to the standard Euclidean metric.classifier.fit(X_train, y_train)y_pred = classifier.predict(X_test)
Results
The figure attached above shows that both models demonstrated identical performance. The accuracy turned out to be 0.93, which is a pretty good result. The figure attached below is a visualization of our test set results. I am providing a single figure since both models are identical. However, I personally suggest using implementations that are provided already since our implementation is simple and inefficient. Moreover, it is just more convenient not to keep writing the exact same code every time.
To sum up, K-Nearest Neighbours is considered to be one of the most intuitive machine learning algorithms since it is simple to understand and explain. Additionally, it is quite convenient to demonstrate how everything goes visually. In this article we answered the following questions:
the k-NN algorithm does more computation on test time rather than train time.
Why do you need to scale your data for the k-NN algorithm?
the k-NN algorithm can be used for imputing the missing value of both categorical and continuous variables.
Is Euclidean Distance always the case?
Why should we not use KNN algorithm for large datasets?
Moreover, I provided a python implementation of the KNN algorithm in order to strengthen your understanding of what happens inside. The full implementation can be found on my Github.