Understanding and using k-Nearest Neighbours aka kNN for classification of digits | by Shubham Gupta | Towards Data Science

In machine learning and statistics, classification is a task of finding the class of a new observation using it’s features/variables. This classification is done using a classifier trained on the basis of training data. Training data is a set of many observations which are correctly labelled with the appropriate class names. Supervised learning involves training with correctly labelled observations hence classification is considered as supervised learning.
Examples of classification:
Labelling an email to ‘spam’ or ‘non-spam’ using email features like words, images and attachments of the email.
Labelling a patient to ‘healthy’ or ‘ill’ using patient’s features like gender, age, weight, blood pressure and observed symptoms.
There are many models which can be used for classification task in machine learning but we will be working with k-Nearest Neighbours as it is simple to use yet powerful.
It is an algorithm which classifies a new data point based on it’s proximity to other data point groups. Higher the proximity of new data point from one group, higher is the likelihood of it getting classified into that group.
Distance between data points is measured by distance metrics like euclidean distance, manhattan distance, minkowski distance, mahalanobis distance, tangential distance, cosine distance and many more.
For data points X and Y with n features:
Minkowski distance formula is:
Minkowski distance when p = 1 is Manhattan distance, when p =2 is Euclidean distance and when p = ∞ is Chebychev distance. Minkowski distance is a generalised form of euclidean distance.
Using distance metric we create a neighbourhood of n closest neighbours to the new data point.
To get the class of new data point, we look at the class groups which have more data points in the created neighbourhood and the class groups which are closer to our new data point compared to other groups in the neighbourhood. Based on these two factors we determine the class of our new data point.
Let’s try understand this better with an example.
Example:
Consider a task of classifying customers into two categories happy and unhappy. You may want to know which customers are unhappy beforehand so that you can prevent them from switching to a competitor service by offering them discounts.
In the image, red represents customers who are unhappy and they have already switched to another competitor whereas blue represents customers happy with us and still using our service.
Now, we have a new customer represented using green circle and we want to know if he is happy or unhappy with our service using kNN algorithm.
If we use 3 neighbours and we are using equal weights for each data point, then we have 2 red points and 1 blue point in the neighbourhood and green point i.e. new data point is classified as red.
If we use 5 neighbours and we are using equal weights for each data point, then we have 3 blue points and 2 red points in the neighbourhood and green point is classified as blue.
If we use 5 neighbours and we are using euclidean distance to calculate weights for each data point, then we have 3 blue points and 2 red points in the neighbourhood. Euclidean distances between data points are denoted using lines.
To calculate weights using euclidean distances we will take inverse of the distances so that closer points have higher weights. For each class we will take sum of calculated weights, and class with higher summed weight becomes predicted class.
Sum of weights for red class:
1/3 + 1/4 = 0.5833
Sum of weights for blue class:
1/5 + 1/8 + 1/6 = 0.4912
Since red class has higher weight, our predicted class for new data point is red class.
In machine learning, before we can use any algorithm, we need to choose the value of hyper-parameters for that model. In case of kNN, important hyper-parameters are:
n_neighbors: Number of neighbours in a neighbourhood.
weights: If set to uniform, all points in each neighbourhood have equal influence in predicting class i.e. predicted class is the class with highest number of points in the neighbourhood. If set to distance, closer neighbours will have greater influence than neighbours further away i.e. class with more points close to new data point becomes predicted class and to do this we take inverse of distance while calculating weights so that closer points have higher weights.
metric: The distance metric to use if we have weights set to distance. Default value is minkowski which is one method to calculate distance between two data points. We can change the default value to use other distance metrics.
p: It is power parameter for minkowski metric. If p=1, then distance metric is manhattan_distance. If p=2, then distance metric is euclidean_distance. We can experiment with higher values of p if we want to.
# kNN hyper-parametrssklearn.neighbors.KNeighborsClassifier(n_neighbors, weights, metric, p)
Trying out different hyper-parameter values with cross validation can help you choose the right hyper-parameters for your final model.
We will be building a classifier to classify hand written digits into one of the class from 0 to 9. The data we will be using is obtained from MNIST database which is a set of 60,000 28×28 pixel black and white images of handwritten individual digits between 0 and 9.
Importing libraries:
# To load MNIST image datafrom sklearn.datasets import load_digits# kNN Classifierfrom sklearn.neighbors import KNeighborsClassifier# Confusion matrix to check model performancefrom sklearn.metrics import confusion_matrix# To split data into training and testing setfrom sklearn.model_selection import train_test_split# For plotting digitimport matplotlib.pyplot as plt
Loading MNIST data of digits:
digits = load_digits()
Transforming data to use with kNN classifier:
# Number of imagesn_samples = len(digits.images)# Changing shape from 28x28 pixel values to a sequence of valuesX = digits.images.reshape((n_samples, -1))# Getting the already known targets for each imagey = digits.target
Creating our training and testing sets:
# Splitting data to train and test setsX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
Creating and training model:
# Creating modelclf = KNeighborsClassifier(n_neighbors=3)# Training modelclf.fit(X_train, y_train)
Getting predictions for test data:
# Predictions for test datapredicted = clf.predict(X_test)
Comparing actual and predicted target values using confusion matrix:
# Print confusion matrixconfusion_matrix(y_test, predicted)
In the matrix, rows represent actual target values where first row is for 0 label, second for 1 label and so on. Similarly, columns represent predictions where first column is for 0 label, second is for 1 label and so on.
Values along the diagonal of the matrix highlighted in yellow are the values which were predicted correctly.
Consider the value highlighted in blue which is 4th column and 9th row. It is a mistake. Our model misclassified 3 as 8.
Overall, our model did a good job in classifying the digits as misclassifications i.e. values other than diagonal are mostly zero or smaller than 2.
Looking at first 10 images and predictions:
# Zip image with predictionimage_with_prediction = list(zip(digits.images, clf.predict(X)))# for first 10 imagesfor pos, (image, prediction) in enumerate(image_with_prediction[:10]):    plt.subplot(3, 4, pos+1) # Create 3x4 grid    plt.axis('off') # no axis    plt.imshow(image, cmap=plt.cm.gray_r) # show image in gray scale    plt.title("Prediction: %i" % prediction) # set title to predicted valueplt.show() # show plot
All the predictions look good except for the 6th image which looks more like 5 but our model thinks it it 9.
When using kNN on your own problems ensure that distances for every feature is scaled according to importance of that feature. In case of house price problem, our features age and house price will have very different scales and you will have to scale down the house price feature to improve model performance.
If your data has large number of dimension then you may want to reduce your features by using feature reduction and feature engineering techniques because data with higher dimension leads to less accuracy.
kNN works well on MNIST dataset because it is a controlled dataset i.e. position of digits is uniform across all the images. Also, the pixel values across all images have similar colour gradients. When dealing with classification problems on images with lot of spatial information i.e. varying position of objects in image and varying colour gradients, you may want to use Convolutional Neural Networks which are built specifically for this kind of task.
kNN can also be used as a regression algorithm i.e. instead of predicting discrete classes it can also be used to predict continuous numbers like house price.
Hope this helped you understand kNN.
Thanks for reading.