Algorithms From Scratch: K-Nearest Neighbors | by Kurtis Pykes | Towards Data Science

A non-parametric algorithm capable of performing Classification and Regression; Thomas Cover, a professor at Stanford University, first proposed the idea of K-Nearest Neighbors algorithm in 1967.
Many often refer to the K-NN as a lazy learner or a type of instance based learner since all computation is deferred until function evaluation. Personally, I believe this puts K-Nearest Neighbors towards the less complex end of Machine Learning Algorithms when we begin to conceptualize it.
No matter whether we are doing Classification or Regression style problems the input will consist of the k nearest training examples in the original feature space. However, the output for the algorithm will of-course depend on the type of question — See Terminology section for more on the different outputs.
Link to code generated in the Article...
github.com
K-Nearest Neighbors Classification → The output would determine class membership and the prediction is made by a plurality vote of its neighbors. Therefore, the new instance would be assigned to the class most common amongst the k nearest neighbors.
K-Nearest Neighbors Regression → The output would determine property value for the object. Therefore, the new instance would be classified as the average of the values of the k Nearest Neighbors
Instance-Based Learning → A family of Machine Learning Learning Algorithms that, instead of performing explicit generalization, compares new problem instances with instances seen in training which has been stored in memory. (Source: Wikipedia)
Lazy Learning → A Machine Learning method in which generalization of the training data is, in theory, delayed until a query is made to the system, as opposed to eager learning where the system tries to generalize the training data before receiving queries. (Source: Wikipedia)
Creating the K-NN algorithm is quite simple. The training phase is literally storing feature vectors and labels of the training samples, however we need to determine a positive integer for k. Generally, when we select a large value of k we reduce the effect of noise on the classification, thence making the boundaries between classes less distinctive. Ultimately, the selection of k is largely influenced by the data which means we have no way of knowing until we have tried with the data, yet there are many different heuristics we can use to select k for our data.
Note: To read more about tuning the k hyperparameter see the Hyperparameter Optimization Wikipedia page.
Great, we have selected k. In order to make a prediction on new instances for a classification task, the k closest records (from the training data) to the new observation are identified. Upon evaluation of the of the k neighbors an prediction is made — see K-Nearest Neighbors Classification in the terminology section to see how this is done.
For us to identify the k closest records to a new instance we must take a measure of all the instances. This can be done in various ways, although as a guide many practitioners often use Euclidean distance when we have continuous variables and Hamming distance for discrete variables.
IMAGE HAMMING AND EUCLIDEAN DISTANCE
Chunking the Algorithm
Calculate the Euclidean DistanceLocate NeighborsPredict
Calculate the Euclidean Distance
Locate Neighbors
Predict
Implementation
To implement our K-Nearest Neighbors Classification algorithm, we will use the Iris dataset from Scikit-Learn. In this task we are challenged to predict whether a flower is setosa, versicolor, or virginica given the measurements of the flowers — making it a multiclass classification task.
Note: In this implementation I did not perform any heuristics to select the optimal k — I simply randomly selected a value for k.
import numpy as npimport pandas as pdfrom collections import Counterimport matplotlib.pyplot as plt%matplotlib inlinefrom sklearn.metrics import accuracy_scorefrom sklearn.datasets import load_irisfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.model_selection import train_test_splitiris = load_iris()X, y = iris.data, iris.targetX_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.2, random_state=1810)X_train.shape, y_train.shape, X_test.shape, y_test.shape((120, 4), (120,), (30, 4), (30,))
We now have the data and have used a holdout based cross validation scheme to split the data — If you are unfamiliar with this terminology see the link below.
towardsdatascience.com
The first step is calculating the Euclidean distance between two rows.
def euclidean(x1, x2):    return np.sqrt(np.sum((x1 - x2)**2))
To test this function, I have taken some code from Jason Brownlee that he used to test his distance function. If we have the correct implementation then our outputs should be the same.
# dataset from https://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-scratch/dataset = [[2.7810836,2.550537003,0],[1.465489372,2.362125076,0],[3.396561688,4.400293529,0],[1.38807019,1.850220317,0],[3.06407232,3.005305973,0],[7.627531214,2.759262235,1],[5.332441248,2.088626775,1],[6.922596716,1.77106367,1],[8.675418651,-0.242068655,1],[7.673756466,3.508563011,1]]row0 = dataset[0]for row in dataset:     print(euclidean(np.array(row0), np.array(row)))0.01.32901739152757871.94946466556532471.55914393855405490.53562807219384924.9529406111642152.77899026747829854.33124803802076.598623496953045.084885603993178
And we get the exact same output — feel free to check on the link provided.
As previously mentioned, the k — neighbors to a new observation are the k nearest instances from the training data. Using our distance function, euclidean, we can now calculate the distance between each observation in the training data and the new we’ve been passed and select the k instances from are training data that are closest to our new observation.
def find_neighbors(X_train, X_test, y_train, n_neighbors):         distances = [euclidean(X_test, x) for x in X_train]    k_nearest = np.argsort(distances)[:n_neighbors]    k_nearest_label = [y_train[i] for i in k_nearest]        most_common = Counter(k_nearest_label).most_common(1)[0][0]        return most_common
This function calculates the distances of the new observation to all the rows in the training data and stores it in a list. Next, we find the indexes of the k lowest distances using the NumPy module np.argsort() — see documentation. We then use the indexes to identify the class of the k instances. After that, we count the number of instances in k_nearest_labels list using the Counter function from Pythons in-built modules and return the most common (the label with highest counts). However, we won’t get to see this in action until we make a prediction so lets build our predict function.
def predict(X_test, X_train, y_train, n_neighbors=3):     predictions = [find_neighbors(X_train, x, y_train, n_neighbors)                  for x in X_test]    return np.array(predictions)predict(X_test, X_train, y_train, n_neighbors=3)array([0, 0, 2, 2, 0, 1, 0, 0, 1, 1, 2, 1, 2, 0, 1, 2, 0, 0, 0, 2,           1, 2, 0, 0, 0, 0, 1, 1, 0, 2])
In the predict function we use a list comprehension to find the nearest neighbor for each new instance in the test set and return an array. Using 3 neighbors we get 100% accuracy on our task and we can compared it to scikit-learns implementation to see if we get the same results — which it did.
Note: The Documentation for K-Nearest Neighbors Classifier can be found here
knn = KNeighborsClassifier(n_neighbors=3)knn.fit(X_train, y_train)sklearn_preds = knn.predict(X_test)preds = predict(X_test, X_train, y_train, n_neighbors=3)print(f"My Implementation: {accuracy_score(y_test, preds)}\nScikit-Learn Implementation: {accuracy_score(y_test, sklearn_preds)}")My Implementation: 1.0Scikit-Learn Implementation: 1.0
Pros
Intuitive and Simple
No training step
Can be used both for Classification and Regression (and unsupervised learning)
Easy to implement for multiclass problems
Cons
As the data grows the algorithm becomes slow quite quickly
Sensitive to Outliers
Imbalanced Data causes problems — Can use weighted distances to overcome this.
In this story you’ve learned about the K-Nearest Neighbors algorithm, how to implement the K-Nearest Neighbors classification algorithm from scratch in Python, and the pros and cons of using K-Nearest Neighbors.
Let’s continue the conversation on LinkedIn...