How Important is the K in KNN Algorithm | by Soner Yıldırım | Towards Data Science

K-nearest neighbors (KNN) is a supervised machine learning algorithm that can be used to solve both classification and regression tasks.
It is based on a very simple idea. The value of a data point is determined by the data points around it.
The value of a data point is determined by the data points around it.
The number of data points that are taken into consideration is determined by the k value. Thus, the k value is the core of the algorithm.
KNN classifier determines the class of a data point by the majority voting principle. If k is set to 5, the classes of 5 closest points are checked. Prediction is done according to the majority class. Similarly, KNN regression takes the mean value of 5 closest points.
In this article, we will examine the importance of the k value for a classification task.
Let’s create a sample classification dataset by using the make_classification function of Scikit-learn.
import numpy as npimport pandas as pdfrom sklearn.datasets import make_classificationX, y = make_classification(     n_samples=1000,     n_features=2, n_informative=2, n_redundant=0, n_classes=2,     class_sep=0.8)
The dataset contains 1000 samples that belong to 2 classes. Let’s also create a scatter plot of the data points (i.e. samples).
import matplotlib.pyplot as pltplt.figure(figsize=(12,8))plt.scatter(X[:,0], X[:,1], c=y)
Selecting the optimal k value is the essential requirement to create a decent and accurate knn model.
If k is selected to be too low, the model becomes too specific and fails to generalize well. It also tends to be sensitive to noise. The model accomplishes a high accuracy on train set but will be a poor predictor on new, previously unseen data points. Therefore, we are likely to end up with an overfit model.
If k is selected to be too large, the model becomes too generalized and fails to accurately predict the data points in both train and test sets. This situation is known as underfitting.
We can now create two different knn models with k values 1 and 50. We will then create scatter plots of the predictions to see the difference.
The first step is to split the dataset into train and test subsets.
from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
The first model is knn with k equals 1.
from sklearn.neighbors import KNeighborsClassifierknn1 = KNeighborsClassifier(n_neighbors=1)knn1.fit(X_train, y_train)predict1 = knn1.predict(X_test)plt.figure(figsize=(12,8))plt.title("KNN with k=1", fontsize=16)plt.scatter(X_test[:,0], X_test[:,1], c=predict1)
We can create the next model just by changing the n_neighbors parameter to 50. Here are the predictions of this model on the test set.
You can see how the generalization changes when k is increased.
What we need is not a too generalized or a too specific model. The goal is to create a robust and accurate model.
An overfit model (too specific) is sensitive to slight changes in the values of data points. Thus, the predictions can change dramatically after very small changes in the data points.
An underfit model (too general) is likely to have poor performance on both train and test subsets.
Unfortunately, there is not a go-to solution to find the optimal k value. It depends on the underlying structure of the dataset. However, there are tools that can help us in finding the optimal k value.
The GridSearchCV function can be used to create, train, and evaluate models with different hyperparameter values. The k is the most important hyperparameter of the knn algorithm.
We will create a GridSearchCV object to evaluate the performance of 20 different knn models with k values changing from 1 to 20. The parameter values are passed to param_grid parameter as a dictionary.
from sklearn.model_selection import GridSearchCVknn = GridSearchCV(     estimator = KNeighborsClassifier(),     param_grid = {'n_neighbors': np.arange(1,21)},     scoring='neg_log_loss',     cv = 5)
You can use any scoring metric of scikit-learn. I have used the log loss which is a commonly used metric for classification tasks.
We can now fit the dataset to the GridSearchCV object. We do not need to split train and test subsets because a 5-fold cross validation is applied.
knn.fit(X, y)
We can access the mean test score of cross validation for each k values.
scores = pd.Series(abs(knn.cv_results_['mean_test_score']))scores.index = np.arange(1,21)
I prefer to save the results in a pandas series to be able to easily plot them. Let’s plot the scores v
scores.plot(figsize=(12,8))plt.title("Log loss of knn with k vales from 1 to 20", fontsize=16)
The loss on the test set does not seem to improve by much after the k value of 10. A good practice is to select the value that the steepness of curve changes.
This plot does not directly tells the optimal k value but it sure narrows down the choices.
The knn algorithm is a widely-used algorithm. It is simple and easy to interpret. Since it does not make any assumption, it can be used to solve non-linear tasks as well.
On the negative side, we can say that the knn algorithm becomes very slow as the number of data points increases because the model needs to store all data points. Due to this reason, it is also not memory efficient.
Finally, it is sensitive to outliers because outliers also have a vote in decisions.
Thank you for reading. Please let me know if you have any feedback.