K-Means and K-Modes Clustering Algorithm
By: Bhanwar Saini
Tags: Data Science, Artificial Intelligence, Clustering, Python, Machine Learning

Photo by Franki Chamaki on Unsplash

This technique is the same as K-Means but more robust towards outliers because of the median, not mean, because K-Means optimizes the squared distances. For example, think of the median salary vs. the mean salary of a large firm when adding a few top executives.

K-medians: instead of taking the mean value of the object in a cluster as a reference point, K-Medians uses absolute deviations (Manhattan Distance) to form k clusters in the data. The centroid of the clusters is the median of the data points in the cluster.

The criteria function for the k median algorithm:

Step for K-Median algorithms:

Select k points as the initial representative objects(i.e., an initial k median)

Repeat this step:

Assign every point to its nearest median

Recompute the median using the median of each feature

until a convergence criterion is satisfied.

Implementation in python:

K-Modes Clustering Algorithm:

K-Means clustering can’t handle non-numerical (categorical) data. But we can map categorical value to 1/0. However, this mapping can’t generate quality clusters for high-dimensional data. Then people requesting the K-Modes method by replacing the means of the clusters with modes, which is called k-modes clustering.

A lot of data in real-world data is categorical, such as gender and profession, and, unlike numeric data, categorical data is discrete and unordered. Therefore, the clustering algorithms for numeric data cannot be used for categorical data. K-Means cannot handle categorical data since mapping the categorical values to 1/0 cannot generate quality clusters for high dimensional data, so instead, we can land onto K-Modes.

The K-Modes approach modifies the standard K-Means process for clustering categorical data by replacing the Euclidean distance function with the simple matching dissimilarity measure, using modes to represent cluster centers, and updating modes with the most frequent categorical values in each of iterations of the clustering process. These modifications guarantee that the clustering process converges to a local minimal result. The number of modes will be equal to the number of clusters required since they act as centroids. The dissimilarity metric used for K-Modes is the Hamming distance from information theory, which can be seen in Fig. Here, x and y are the values of attribute j in objects X and Y. The larger the number of mismatches of categorical values between X and Y is, the more dissimilar the two objects. In the categorical dataset, the mode of an attribute is either “1” or “0,” whichever is more common in the cluster. The mode vector minimizes the sum of the distances between each object in the cluster and the cluster center.

Figure: Hamming distance

Step for K-Modes clustering algorithm:

The K-Modes clustering process consists of the following steps:

Randomly select k unique objects as the initial cluster centers (modes). Calculate the distances between each object and the cluster mode; assign the object to the cluster whose center has the shortest distance. Repeat until all objects are assigned to clusters. Please select a new mode for each cluster and compare it with the previous mode. If different, go back to Step 2; otherwise, stop.

Implementation in python:

Here is code for k-modes clustering in python: