KNN visualization in just 13 lines of code | by Deepthi A R | Towards Data Science

Yes! It’s that simple. Let’s play around with datasets to visualize how the decision boundary changes as ‘k’ changes.
Let’s have a quick review...
K Nearest Neighbor(KNN) algorithm is a very simple, easy to understand, versatile and one of the topmost machine learning algorithms. In k-NN classification, the output is a class membership. An object is classified by a plurality vote of its neighbours, with the object being assigned to the class most common among its k nearest neighbours (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbour.
In KNN, K is the number of nearest neighbours. The number of neighbours is the core deciding factor. K is generally an odd number if the number of classes is 2. When K=1, then the algorithm is known as the nearest neighbour algorithm. This is the simplest case. Suppose P1 is the point, for which label needs to be predicted.
KNN has three basic steps.1. Calculate the distance.2. Find the k nearest neighbours.3. Vote for classes
You can’t pick any random value for k. The whole algorithm is based on the k value. Even small changes to k may result in big changes. Like most machine learning algorithms, the K in KNN is a hyperparameter. You can think of K as a controlling variable for the prediction model.
In this blog, we’ll see how decision boundary changes with k. For this, I’ll be using different types of toy datasets. You can easily download all these datasets from the given link below.
www.kaggle.com
U — Shaped
U — Shaped
In this dataset, we can observe that the classes are in the form of an interlocked U. This is a non-linear dataset. The blue points belong to class 0 and the orange points belong to class 1.
2. Two set concentric circles
In this dataset, we can see that the points form two sets of concentric circles. This is a non-linear dataset.The blue points belong to class 0 and the orange points belong to class 1.
3. XOR
This dataset resembles the 2 variable XOR k-map. This is a non-linear linear dataset.
The blue points belong to class -1 and the orange points belong to class 1.
4. Linearly separable
This is a linear dataset in which the points can be linearly separated.
The blue points belong to class 0 and the orange points belong to class 1.
5. Outliers
In this dataset, we can observe that there are outlier points from both the classes. We’ll see how the presence of outliers can affect the decision boundary. This is a linear dataset.The blue points belong to class 0 and the orange points belong to class 1.
Now that we know how our looks we will now go ahead with and see how the decision boundary changes with the value of k. here I’m taking 1,5,20,30,40 and 60 as k values. You can try with any set of value
To start off, let’s import the libraries.
import matplotlib.pyplot as pltimport pandas as pdfrom sklearn import datasets, neighborsfrom mlxtend.plotting import plot_decision_regions
The following core function required.
def knn_comparison(data, k): x = data[[‘X’,’Y’]].values y = data[‘class’].astype(int).values clf = neighbors.KNeighborsClassifier(n_neighbors=k) clf.fit(x, y)# Plotting decision region plot_decision_regions(x, y, clf=clf, legend=2)# Adding axes annotations plt.xlabel(‘X’) plt.ylabel(‘Y’) plt.title(‘Knn with K=’+ str(k)) plt.show()
You can observe that its just one line of code that is doing ever everything — clf = neighbors.KNeighborsClassifier(n_neighbors=k)
It’s that simple and elegant. Now we just have to load our CSV file and pass it to this function along with k. We’ll do this for all the datasets one by one.
Let’s dive in...
data1 = pd.read_csv(‘ushape.csv’)for i in [1,5,20,30,40,80]:    knn_comparison(data1, i)
data2 = pd.read_csv(‘concertriccir2.csv’)for i in [1,5,20,30,40,60]:    knn_comparison(data2, i)
data3 = pd.read_csv(‘xor.csv’)for i in [1,5,20,30,40,60]:   knn_comparison(data3, i)
data4 = pd.read_csv(‘linearsep.csv’)for i in [1,5,20,30,40,60]:    knn_comparison(data4, i)
data5 = pd.read_csv(‘outlier.csv’)for i in [1, 5,20,30,40,60]:    knn_comparison(data5, i)
In all the datasets we can observe that when k=1, we are overfitting the model. That is, each point is classified correctly, you might think that it is a good thing, well as the saying goes “ too much is too bad”, overfitting essentially means that our model is training way too well to an extent that it negatively impacts the model.
In all the datasets we can observe that when k = 60 (a large number), we are underfitting the model. Underfitting refers to a model that can neither model the training data nor generalize to new data. An underfit machine learning model is not a suitable model. In the case of outlier dataset, for k=60, our model did a pretty good job. How can we tell this? Because we know that those points are outliers and the data is linearly separable, but that’s not the case every time, we cannot be sure. Also in the XOR dataset, the value of k is not affecting the model to a great extent, unlike other models.
Research has shown that there is no optimal value of hyperparameter(k ) that suits all kind of data sets. Each dataset has it’s own requirements. In the case of a small k, the noise will have a higher influence on the result, and a large k will make it computationally expensive. Research has also shown that a small k is most flexible fit which will have low bias but the high variance and a large k will have a smoother decision boundary which means lower variance but higher bias.
Overfitting and underfitting are the two biggest causes for the poor performance of machine learning algorithms.
When K is small, we are restraining the region of a given prediction and forcing our classifier to be “blind” to the overall distribution. A small value for K provides the most flexible fit, which will have low bias but high variance. Graphically, our decision boundary will be more jagged as we observed above. On the other hand, a higher K averages more points in each prediction and hence is more resilient to outliers. Larger values of K will have smoother decision boundaries which mean lower variance but increased bias.
www.datacamp.com/community/tutorials/k-nearest-neighbor-classification-scikit-learnscott.fortmann-roe.com/docs/BiasVariance.htmlrasbt.github.io/mlxtend/user_guide/plotting/plot_decision_regions/#references
www.datacamp.com/community/tutorials/k-nearest-neighbor-classification-scikit-learn
scott.fortmann-roe.com/docs/BiasVariance.html
rasbt.github.io/mlxtend/user_guide/plotting/plot_decision_regions/#references
What are you waiting for? Go ahead and play around with the datasets !!