K-Nearest Neighbors Classification from Scratch with NumPy | by Levent Baş | Towards Data Science

Welcome to another post of implementing machine learning algorithms from scratch with NumPy. In this post, I will implement K-nearest neighbors (KNN) which is a machine learning algorithm that can be used both for classification and regression purposes. It falls under the category of supervised learning algorithms that predict target values for unseen observations. In other words, it operates on labeled datasets and predicts either a class (classification) or a numeric value (regression) for the test data.
As I have mentioned in my other posts, I won’t be using an already implemented version of the k-nearest neighbor algorithm, I will be implementing it with only NumPy and will be using other libraries for data visualization purposes and creating and using datasets.
Considering there are many useful resources out there that explain the fundamentals of KNN, I want to get into the coding part immediately. Let’s proceed!
We start by importing the libraries and functions that will be used throughout the implementation:
numpy: Obviously, it will be used for numerical computation of multidimensional arrays as we are heavily dealing with vectors of high dimensions.
make_classification: We will use this to create our very own classification dataset. We can decide how many classes and features we want, whether we want samples to be clustered, etc.
In KNN algorithm, we will need a function to calculate the distances between training data points and the data that we would like to classify. Here, I’ve chosen the euclidian distance as it is a widely used one in machine learning applications. One can try using other distance metrics such as Manhattan distance, Chebychev distance, etc.
When we consider the practical aspect of KNN, we try to find the neighbors, the closest data points to the data that we want to classify. Whether a data point is close or not is determined by our euclidian distance function implemented above. Here, the actual value of the distance between data points does not matter, rather, we are interested in the order of those distances.
We pick a number as our hyperparameter (k) before the training process and pick k-nearest neighbors to our data points we want to classify during training. Hence, for instance, when we say 5-nearest neighbors, we mean the first 5 closest data points.
In the kneighbors function above, we find the distances between each point in the test dataset (the data points we want to classify) and the rest of the dataset, which is the training data. We store those distances in point_dist in which each row corresponds to a list of distances between one test data point and all of the training data. Hence, we go over each row, enumerate it and then sort it according to the distances. The reason we enumerate each row is because we don’t want to lose the indices of training data points that we calculated the distances with, since we are going to refer them later.
Consequently, sorted_neigh holds the first k-nearest neighbors of our test data points and they are sorted according to their euclidian distances. We, then, extract indices and distance values from sorted_neigh and return them.
After finding the k-nearest neighbors, we try to predict the classes that our test data points belong to. Here, we have k neighbors and each neighbor has a vote in deciding the class label. However, the voting mechanism may vary according to the chosen criterion. Here, in the predict function above, if weights are chosen as uniform it means that each neighbor has an equal vote (weight) in deciding the class label, irrespective of their distances.
Let’s say we have 5-nearest neighbors of our test data point, 3 of them belonging to class A and 2 of them belonging to class B. We disregard the distances of neighbors and conclude that the test data point belongs to the class A since the majority of neighbors are part of class A. However, if weights are chosen as distance, then this means the distances of neighbors do matter, indeed. Hence, whichever neighbor that is closest to the test data point has the most weight (vote) proportional to the inverse of their distances. Thereby, regarding the aforementioned example, if those 2 points belonging the class A are a lot closer to the test data point than the other 3 points, then, this fact alone may play a big role in deciding the class label for the data point.
In predict function, it is quite easy to predict the label for data points if weights are uniform. First, we get the indices of neighbors and then use those indices to get their corresponding class labels from the training dataset. Each row in neighbors corresponds to the set of neighbors that each test data point has. We then find the occurrences of class labels using numpy’s bincount function and get the index of the maximum occurrence which corresponds to the predicted class label.
Things get a little messier if we have weights chosen as distance. In this case, we find the mean inverse of neighbor distances and calculate class probabilities for each test data point.
I have implemented the score function as a very simple accuracy metric used in classification problems widely. We simply return the percentage of correctly classified labels. It is pretty straightforward!
Now, it is time to create the dataset that we will be testing our knn algorithm upon. We make use of sklearn.dataset’s make_classification function to populate the dataset. Afterward, we normalize each data point by subtracting the mean and then dividing by the standard deviation.
After creating the dataset, we implement a random split for our training phase, hence obtain our training and test sets.
In[104]: kneighbors(X_test)Out[104]: array([[ 85, 405, 370,  63, 694],               [345,  64, 189, 136,  32],               [554, 216, 690, 672,  51],                ...,               [560, 323,  99, 418, 295],               [214, 180, 534, 190, 133],               [ 20, 129, 564, 103, 679]])
It is time to test our code of knn implementation. We get the k-nearest neighbors of our test dataset. Do notice that, each row is related to each data point in our test set and elements in each row correspond to the indices of neighbors of the test data point.
In[105]: predict(X_test)Out[105]: array([0, 1, 2, 1, 0, 2, 1, 1, 0, 0, 2, 1, 1, 0, 0, 1, 0, 0, 0, 1, 2, 2, ..., 2, 1, 0, 2, 1, 2, 1, 0, 0, 1, 1, 0, 0, 2, 0, 2, 1, 2, 0, 1, 2, 0])
As expected, predict function outputs the predicted class labels of the test data.
In [106]: score(X_test, y_test)Out[106]: 0.99
It seems like our implementation did a really good job given the high accuracy score.
After implementing all necessary functions, it is pretty easy to create the class implementation of knn. Here only newly added function is the __init__ function for our class.
Out[110]:              Our Implementation       Sklearn's Implementation                         Accuracy   0.955556                 0.955556
It turns out that the accuracy of our own implementation and sklearn’s implementation looks the same. That’s good news right? We have done a pretty good job with this implementation.
You can also check out my GitHub profile to read the code along a jupyter notebook or simply use the code for implementation.
I’ll be back with more implementations in the future, for sure.
Happy coding!
Questions? Comments? Contact me at leventbas92@gmail.com, through GitHub or LinkedIn.