KNNs &amp; K-Means: The Superior Alternative to Clustering &amp; Classification.
Source: tai_blog

Lets discuss two popular ML algorithms  KNNs and K-Means. Stick around; Ill make this densely packed.   P.S. Im trying out a new thing: I draw illustrations of graphs  etc.  myself  so well also look at some nice illustrations that help us understand the concept.   We will discuss KNNs  also known as K-Nearest Neighbours and K-Means Clustering. They are both ML Algorithms  and well explore them more in detail in a bit.   KNNs: K-Nearest Neighbours. U+1FAC2K-Nearest Neighbors (KNN) is a supervised ML algorithm for classification and regression.   Principle: That similar data points are located close to each other in the feature space.   Quick Primer: What is Supervised? U+1F4A1 supervised refers to a type of learning where the algorithm is trained using labeled data. This means that the input data comes with corresponding output labels that the model learns to predict.So  KNNs is a supervised ML algorithm that we use for Classification and Regression  two types of supervised learning in ML. Lets take a closer look at them:   Regression (Left Graph):The blue dots represent individual data points  each corresponding to a pair of input (x-axis) and output (y-axis) values.The black line running through the data points is the regression line  which represents the models prediction of the output for a given input.   Example:Scenario:Imagine this graph represents data on how study hours (x-axis) impact exam scores (y-axis).   Interpretation:Consider that each blue dot represents a student  with their study hours plotted on the x-axis and their exam score on the y-axis. The regression line shows the predicted exam score based on the number of study hours. For instance  if a student studied for 70 hours  the model might predict a score of around 60 based on the line.   Classification (Right Graph):The red and blue dots represent individual data points that belong to two different categories or classes.The black curve is the decision boundary  which the model uses to separate the two classes. Points on one side of the curve are classified into one category  while points on the other side belong to the other category.Example:Scenario:Imagine this graph represents data on two species of flowers  with the x-axis showing petal width and the y-axis showing petal length. The red dots could represent one species  and the blue dots could represent another.   Interpretation:The model uses the curve to decide the species based on petal dimensions. For instance  if a flower has petal dimensions that fall on the left side of the curve  it would be classified as the red species  and if it falls on the right  it would be classified as the blue species.   In both graphs  the colored dots (data points) illustrate how the model interprets the input data  whether by predicting a continuous outcome (regression) or by categorizing the data into distinct classes (classification).   ClassificationIn Classification  we predict discrete labels or categories for input data. The goal is to assign a class label to new observations based on the training data.   Key AspectsOutput: Discrete categories (e.g.  spam or not spam).Types:   Binary Classification: Involves two classes. For example  determining if an email is spam or not.Multiclass Classification: Involves more than two classes. For example  classifying types of flowers based on features like petal length and width.ExamplesEmail Filtering: Classifying emails as spam or not spam based on their content and metadata.Medical Diagnosis: Predicting whether a patient has a specific disease based on symptoms and test results (e.g.  has disease or does not have disease).Image Recognition: Identifying objects in images  such as classifying images as cat  dog  or bird.RegressionRegression  on the other hand  is used to predict continuous numerical values. Its aim is to model the relationship between input variables and a continuous output.   Key AspectsOutput: Continuous values (e.g.  predicting prices or temperatures).Types: Common types include linear regression and polynomial regression.ExamplesHouse Price Prediction: Estimating the price of a house based on features like size  location  and number of bedrooms.Sales Forecasting: Predicting future sales revenue based on historical sales data and other influencing factors.Temperature Prediction: Forecasting the temperature for a given day based on historical weather data.So  classification is focused on categorizing data into distinct classes  while regression is concerned with predicting continuous outcomes. Pretty cool!   How it WorksChoose KK: Decide on the number of neighbors KK to consider when making predictions.Distance Calculation: For a given data point (the query)  calculate the distance to all other points in the dataset.Sorting: Sort all distances to find the KK closest data points.Voting/Averaging:For classification: The most common class label among the KK neighbors is assigned to the query point.For regression: The average value of the KK neighbors is computed and assigned to the query point.ExampleConsider a scenario where you want to classify whether a new fruit is an apple or an orange based on its color and weight.   Step 1: You choose K=3K=3 (three nearest neighbors).Step 2: For the new fruit  you calculate the distance to all existing fruits in your dataset.Step 3: You find the three closest fruits. Suppose you have two apples and one orange among those neighbors.Step 4: Since the majority are apples  you classify the new fruit as an apple.K-Means ClusteringK-means Clustering is an unsupervised ML algorithm used to partition a dataset into KK distinct clusters based on feature similarity.   The algorithm operates without labeled data  meaning it identifies patterns within the data without prior training.   Quick Primer: What is Unsupervised? U+1F4A1 In unsupervised learning the algorithm is not provided with labeled data and must discover patterns and insights on its own. Since k-Means does not use labeled data  it is categorized as unsupervised learning.How It WorksInitialization: Choose KK  the number of clusters  and randomly select KK initial centroids (the center points of the clusters).Assignment Step: Each data point is assigned to the nearest centroid based on a distance metric  typically Euclidean distance.Update Step: Recalculate the centroids by taking the mean of all data points assigned to each cluster.Iteration: Repeat the assignment and update steps until the centroids no longer change significantly or until a predetermined number of iterations is reached.ExampleImagine a scenario where you are trying to categorize different types of fruits based on their weight and sweetness.   Step 1: You decide to create 2 clusters (K=2). You randomly select two fruits as initial centroids.Step 2: You measure the distance of each fruit to the two centroids and assign each fruit to the nearest centroid.Step 3: After all fruits are assigned  you recalculate the centroids based on the average weight and sweetness of the fruits in each cluster.Step 4: You repeat the assignment and update steps until the clusters stabilize.This process helps you identify groups like sweet and heavy fruits and light and less sweet fruits without needing to know anything about these categories beforehand.   What are the differences between KNN & K-Means?Key DifferencesType of LearningKNN: This is a supervised learning algorithm primarily used for classification and regression tasks. It requires labeled data to train the model.K-means: This is an unsupervised learning algorithm used for clustering. It does not require labeled data and groups data points based on their similarities.ObjectiveKNN: The goal is to predict the class label of a new data point by looking at the k nearest labeled data points in the training set.K-means: The objective is to partition the dataset into k distinct clusters  where each data point belongs to the cluster with the nearest mean (centroid).Input DataKNN: Requires a dataset with known class labels for training.K-means: Works with unlabeled data  grouping similar data points into clusters without any prior knowledge of their labels.Distance CalculationKNN: Computes distances between a new data point and all points in the training set to find the nearest neighbors  typically using metrics like Euclidean or Manhattan distance.K-means: Calculates distances from data points to the centroids of clusters  iteratively updating the centroids based on the mean of the points assigned to each cluster.OutputKNN: Outputs a predicted class label for the new data point based on the majority vote of its nearest neighbors.K-means: Outputs clusters of data points  each represented by a centroid  without any class labels.ParametersKNN: The main parameter is k  which determines how many neighbors to consider for classification or regression.K-means: The parameter k represents the number of clusters to form.Where do we use KNN & K-Means?KNN:Healthcare: KNN is utilized for predicting diseases based on patient data  like assessing the risk of heart attacks or cancer by analyzing gene expressions and other health indicators.   Finance: It plays a significant role in various financial applications  including:   Credit Risk Assessment: Evaluating the creditworthiness of loan applicants by analyzing historical data.Stock Market Forecasting: Predicting stock prices based on economic indicators and company performance.Fraud Detection: Identifying potential money laundering activities by analyzing transaction patterns.Recommendation Systems: KNN is used in recommendation engines where it assigns users to groups based on their behavior  personalizing content suggestions.   Pattern Recognition: Effectively recognizes patterns in data  like classifying handwritten digits or categorizing text documents.   Data Preprocessing: KNN can input missing values in datasets  which provides estimates based on the nearest neighbors of the missing data points.   K-Means Clustering:Market Segmentation: K-Means is commonly used in marketing to segment customers into distinct groups based on purchasing behavior  allowing for targeted marketing strategies.   Image Compression: This algorithm helps in reducing the number of colors in an image by clustering similar colors  which is useful in image processing and storage optimization.   Anomaly Detection: K-Means can identify unusual data points in datasets  which is valuable in fraud detection and network security.   Document Clustering: It is used to group similar documents together  aiding in information retrieval and organization.   Genetic Data Analysis: K-Means assists in clustering genetic data for various research purposes  helping to identify patterns in gene expression.   To Summarize:We discussed two popular ML algorithms  K-Nearest Neighbors (KNN) and K-Means Clustering.   KNN is a supervised learning algorithm used for classification and regression  relying on labeled data to predict outcomes based on the K nearest neighbors. On the other hand  K-Means is an unsupervised learning algorithm used for clustering  which groups data points into K clusters based on feature similarity without labeled data.   Remember  KNN predicts labels or values for new data points  while K-Means identifies clusters in unlabeled data. Use KNN for tasks like disease prediction or recommendation systems  and K-Means for market segmentation or anomaly detection. Both algorithms are versatile and powerful  but their use cases and approaches differ significantly.   Thats it  thanks for reading  happy learning!         References: How I Learnt this ConceptWhat Is The Difference Between KNN and K-means?  YouTube   YouTube Link: Josh Starmer   What K is in KNN and K-Means  Essi Alizadeh (ealizadeh.com)