Implementation of Cluster Centroid based Majority Under-sampling Technique (CCMUT) in Python | by Navoneel Chakrabarty | Towards Data Science

In Machine Learning and Deep Learning, Data Imbalance is a major reason of performance degradation of many Classification Algorithms and Deep Neural Networks respectively. Imbalanced Data is referred to as the unequal number of instances or samples belonging to different classses or categories by a significant difference. One such example of Binary Imbalanced Data in which, there are 2 classes but contain unequal number of instances with 76,090 instances with label ‘0’ and that of label ‘1’ are 19668:
To minimize the degree of imbalance, Data Mining and Feature Space Geometry has to be incorporated into the Classical Methodology of solving Machine Learning Classification Problems. There are many Data Mining approaches for Data Balancing. One such important approach is Cluster Centroid based Majority Under-sampling Technique (CCMUT).
In Majority Under-sampling, unimportant (or not-so-important) instances are removed among majority samples. In CCMUT, the demarcation of instances as important and unimportant is done by using the concept of Clustering on Feature-Space Geometry.
Clustering is an Unsupervised Learning Approach. But CCMUT, only uses the concept of finding cluster centroid (clusters are created encircling data-points belonging to the majority class), as already instances are labelled. The cluster centroid is found by obtaining the average feature vectors for all the features, over the data points belonging to the majority class in feature space.
After finding the cluster centroid of the majority class, the instance belonging to the cluster (majority class), which is farthest from the cluster centroid in feature space, is considered to be the most unimportant instance. On the contrary, the instance belonging to the majority class, that is nearest to the cluster centroid in feature space, is considered to be the most important instance.
So, in CCMUT, instances belonging to the majority class are removed on the basis of their importance and number of samples to be under-sampled depends upon the % of Under-sampling or CCMUT.
Python Implementation of CCMUT:
In the Implementation perspective, all the distances in feature space geometry, are considered as Euclidean Distances.
Given below is the Python Implementation of CCMUT in the form of a function, CCMUT() which takes:
Majority Sample Matrix = X% of CCMUT or Under-sampling = f
Majority Sample Matrix = X
% of CCMUT or Under-sampling = f
as arguments and returns the Majority Sample Matrix after under-sampling = X_f.
import numpy as npfrom math import sqrtdef CCMUT(X,f):    # 1. finding cluster centroid....    cluster_centroid = np.sum(X,axis=0)/X.shape[0]    # 2. finding Euclidean Distance from cluster centroid to samples    euclidean = [None]*X.shape[0]    for i in range(0,X.shape[0]):        euclidean[i] = sqrt(sum((cluster_centroid-X[i])**2))    # 3. tracking indices of samples in descending order of distance    indices = list(reversed(sorted(range(len(euclidean)),     key = lambda j: euclidean[j])))    # 4. removing the instances or under-sampling order-wise....    X_f = np.delete(X, indices[:int(f/100*X.shape[0])], axis=0)    # 5. returning the under-sampled Majority Sample Matrix    return X_f
So, calling the CCMUT() with Majority Sample Matrix and % of Under-sampling, the under-sampled Majority Sample Matrix will be obtained. Obviously, after under-sampling, it will not be called a Majority by a large margin or difference.
After that, the Under-sampled Majority Sample Matrix can be concatenated with the Minority Sample Matrix (to be separated, prior to CCMUT) and Shuffled for Predictive Analytics or Predictive Classification Model Development using Machine Learning.
There are also many available researches in which CCMUT has been applied either directly or with certain amendments.
Altınçay, Hakan, and Cem Ergün. “Clustering based under-sampling for improving speaker verification decisions using AdaBoost.” Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). Springer, Berlin, Heidelberg, 2004.Yen, Show-Jane, and Yue-Shi Lee. “Cluster-based under-sampling approaches for imbalanced data distributions.” Expert Systems with Applications 36.3 (2009): 5718–5727.Rahman, M. Mostafizur, and D. Davis. “Cluster based under-sampling for unbalanced cardiovascular data.” Proceedings of the World Congress on Engineering. Vol. 3. 2013.Sobhani, Parinaz, Herna Viktor, and Stan Matwin. “Learning from imbalanced data using ensemble methods and cluster-based undersampling.” International Workshop on New Frontiers in Mining Complex Patterns. Springer, Cham, 2014.Srividhya, S., and R. Mallika. “Cluster concentric circle based under sampling to handle imbalanced data.” Middle East Journal of Scientific Research 24 (2016): 314–319.Anuradha, Nagasuri, and G. Partha Saradhi Varma. “PBCCUT-Priority based Class Clustered under Sampling Technique Approaches for Imbalanced Data Classification.” Indian Journal of Science and Technology 10.18 (2017).Arafat, Md Yasir, Sabera Hoque, and Dewan Md Farid. “Cluster-based under-sampling with random forest for multi-class imbalanced classification.” 2017 11th International Conference on Software, Knowledge, Information Management and Applications (SKIMA). IEEE, 2017.
Altınçay, Hakan, and Cem Ergün. “Clustering based under-sampling for improving speaker verification decisions using AdaBoost.” Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR). Springer, Berlin, Heidelberg, 2004.
Yen, Show-Jane, and Yue-Shi Lee. “Cluster-based under-sampling approaches for imbalanced data distributions.” Expert Systems with Applications 36.3 (2009): 5718–5727.
Rahman, M. Mostafizur, and D. Davis. “Cluster based under-sampling for unbalanced cardiovascular data.” Proceedings of the World Congress on Engineering. Vol. 3. 2013.
Sobhani, Parinaz, Herna Viktor, and Stan Matwin. “Learning from imbalanced data using ensemble methods and cluster-based undersampling.” International Workshop on New Frontiers in Mining Complex Patterns. Springer, Cham, 2014.
Srividhya, S., and R. Mallika. “Cluster concentric circle based under sampling to handle imbalanced data.” Middle East Journal of Scientific Research 24 (2016): 314–319.
Anuradha, Nagasuri, and G. Partha Saradhi Varma. “PBCCUT-Priority based Class Clustered under Sampling Technique Approaches for Imbalanced Data Classification.” Indian Journal of Science and Technology 10.18 (2017).
Arafat, Md Yasir, Sabera Hoque, and Dewan Md Farid. “Cluster-based under-sampling with random forest for multi-class imbalanced classification.” 2017 11th International Conference on Software, Knowledge, Information Management and Applications (SKIMA). IEEE, 2017.