Fuzzy C-Means Clustering with Python | by Yufeng | Towards Data Science

Fuzzy C-means clustering algorithm is an unsupervised learning method. Before learning the details, let me first decipher its fancy name.
So, “fuzzy” here means “not sure”, which indicates that it’s a soft clustering method. “C-means” means c cluster centers, which only replaces the “K” in “K-means” with a “C” to make it look different.
In a clustering algorithm, if the probability of one data point belonging to a cluster can only take the value of 1 or 0, it’s hard clustering. The boundary of a cluster in a hard clustering method can be visualized as a crisp boundary. On the contrary, in a soft clustering method, the probability of one data point belonging to a cluster can take any value between 0 and 1, such as 75%, for which the boundary of a cluster can be visualized as a fuzzy boundary.
Alright, after knowing the concept of a soft clustering method, we can intuitively realize the several important parameters that could possibly drive the clustering. First, of course, the cluster centers. Second, the probability of one point belonging to a specific cluster.
In the Fuzzy c-means (FCM) clustering method, we have two parameters, μ_ij and c_i, and one hyperparameter, m.
μ_ij, membership value, is the probability that the jth data point belongs to the ith cluster, and it is constrained to that the sum of μ_ij over C cluster centers is 1 for every data point j. c_i is the center of the ith cluster (the same dimension with X). And m is the fuzzifier, which controls how fuzzy the cluster boundary should be.
In the example plot above, we are looking at the 5th data point X_5, and suppose we know there are only two clusters and the current cluster centers are c_1 and c_2. μ_25 is the probability of the 5th data point belonging to the 2nd cluster, and μ_15 is the probability of the 5th data point belonging to the 1st cluster. Then, we see that the 5th data point is much closer to c_2 than c_1, so μ_25 (0.6) is larger than μ_15 (0.4). And they meet the constraint that the sum of μ for each data point is 1, where μ_15 + μ_25 = 1.
The example above is just for illustration of the parameters in FCM, but the real values don’t have to be exactly like that.
I choose to show the objective function after introducing the parameters because it will look much clearer here.
You can understand the objective function as a weighted sum of the distance between the data points (X_j) and the cluster centers (C_i). The “distance” term is the L2 norm in the equation above, and in the example (the 5th data point) above, it is exactly the length of the arrows.
If m = 1, then the objective function is simply the probability-weighted sum of distances between data points and cluster centers. What does it mean? It means that the data points that are close to the cluster centers are given higher weights.
In the example above, the distance between the 5th data point and the 2nd cluster center has a larger contribution to the objective function than that between the 5th data point and the 1st cluster center, because μ_25 = 0.6 and μ_15 = 0.4.
Remember we want to minimize the objective function, so it’s good to have small μs for those long distances (between data points and cluster centers).
What if m =2, 3, ...? Then the distances’ contribution difference is becoming smaller and smaller as shown below (μ_15 and μ_25), which are all close to 0.
Therefore, for a very large m, the cluster centers tend to locate at a place with nearly equal distance to all the data points because every point’s distance to centers contributes equally small to the objective function. So, where are the cluster centers? Yes, the center of all the data points.
If m is super large, with all cluster centers located at the centroid of all data points, the clustering will be super “fuzzy” because it doesn’t cluster at all!
So, to make a meaningful clustering, we use m = 2 for most cases.
The constrained parameter optimization could be solved by hand using the Lagrangian function, but we will not cover it in this post. Instead, I am listing the final equations to calculate c_i and μ_ij.
The equation to solve c_i is relatively straightforward, where it represents the weighted average of X to the center of cluster i. Why do we have to divide it by the sum of (μ_ij)^m? That’s because the sum of weights equals 1 only when m=1 (in which case we can remove the entire denominator). When m>1, the sum of weights is the scaler for the weighted sum calculation for cluster centers.
However, the equation of μ_ij doesn’t look that easy to understand, right? If we transform it to the following one, it should be much easier to interpret.
Suppose m=2, then the numerator part is just the reciprocal of the distance between the jth data point to the ith cluster center (the length of the arrow in the illustration plot above). As aforementioned, a larger distance should correspond to a smaller μ_ij, which is also reflected in this equation. The denominator of the equation is the sum of these reciprocals of the distances from x_j to the centers of every cluster.
The full FCM algorithm could be described in the figure below.
The procedure is actually the same as the EM algorithm. If you are interested to read about EM, you can go to the following post.
towardsdatascience.com
The implementation of fuzzy c-means clustering in Python is very simple. The fitting procedure is shown below,
import numpy as npfrom fcmeans import FCMmy_model = FCM(n_clusters=2) # we use two cluster as an examplemy_model.fit(X) ## X, numpy array. rows:samples columns:features
Extract information from the clustering model,
centers = my_model.centerslabels = my_model.predict(X)
That’s it!
Hope the article is useful.