Package: clustersupport
ClusterSupportClusterSupport is a small package designed to enhance scikit-learn's clustering capabilities. The package combines scikit-learn's clustering algorithm classes likeKMeans(),AgglomerativeClustering(), andDBSCAN(),
with additional functions for the analysis and optimisation of clustering results.DependenciesClusterSupport requires the following packages:Analysing clustering resultsClusterSupport inherits clustering classes from scikit-learn and wraps their.fit()methods so that calling.fit()returns an instance of theClusteringResult()class. Let's load in one of scikit-learn's toy datasets to see some of the functionality.from sklearn.datasets import load_boston
data = pd.DataFrame(load_boston()['data'], columns = load_boston()['feature_names'])And then we can call the.fit()method to return an instance of ourClusteringResult()class.import clustersupport as cs
results = cs.KMeans(n_clusters = 3).fit(data)We can calculate a metric of clustering structure, for example the Calinski-Harabasz score or C-index, using:CH_score=results.CH_score()C_index=results.C_index()Currently, ClusterSupport supports the following clustering metrics:Silhouette scoreCalinski-Harabasz scoreC-indexInertia (sum of intra-cluster distances)For another view we can get a summary of the clustering using:summ=results.get_summary(dist_metric='sqeuclidean')which returns a Pandas DataFrame containing values for the number of points in each cluster as well as the average distance
around the cluster mean.clusternavg_dist_around_mean01026.1813668.662386.69The distance metric can be specified with thedist_metricargument. ClusterSupport currently supports'sqeuclidean','euclidean','cosine', and'manhattan'distances.Finally, we can conduct a classifier assessment of the clustering. This involves training a classifier to predict
the cluster to which each data point belongs and assessing the classifier's accuracy on a 'test set' of points that were
not used in the training. This is particularly useful in contexts where the clustering is run on a reduced version of the full
feature space, and we wish to analyse how effectively the clustering captures the full detail of the complete feature space.fromsklearn.decompositionimportPCA# set seed and apply PCA on the datadata_reduced=PCA(n_components=3,random_state=123).fit_transform(data)# fit to the reduced data and save the labelsreduced_fit_labels=cs.KMeans(n_clusters=8,random_state=123).fit(data_reduced).labels_# run fit with the full data setclustering=cs.KMeans(n_clusters=8).fit(data)# run classifier assessment with the reduced fit labelsclf_assessment=clustering.classifier_assessment(classifier='logreg',labels=reduced_fit_labels,roc_plot=True,n=50,save_fig=True,random_state=123)Specifyingroc_plot = Trueuses matplotlib to plot an ROC curve for each cluster so that the user can see how well each cluster is classified.
The function also outputs a Pandas DataFrame with classification metrics (precision, recall, and f1 score) calculated for each cluster.
The AUC for each cluster reflects the classifier's ability to distinguish instances in that cluster, thus providing an estimate of:How well the reduced feature space represents the complete feature spaceHow well-separated each cluster isAnalysing feature importances in clusteringClusterSupport also provides methods for analysing the importance of different features in the cluster, either on at a global or per-cluster level.
These functions are called as methods under the clustering classes inherited from scikit-learn (e.g.clustersupport.KMeans()).Thet_testmethod calculates a t-statistic for each feature in each cluster, calculated as a scaled difference-of-means
between feature values for instances inside the cluster compared to instances outside the cluster. This returns Pandas DataFrame of size
(n_clusters, n_features), with the calculated t-statistic or p-value for the respective cluster/feature combination in each cell.
Welch's two-sample t-test with unequal variances is used for the calculation.feature_t_tests = cs.KMeans(n_clusters = 3).t_test(X = data, output = 'p-value')You can also output the raw t-statistics withoutput = 't-statistic'.
Alternatively you can conduct a non-parametric Mann-Whitney U test to test the ranks of feature values inside/outside each cluster.feature_MW = cs.KMeans(n_clusters = 3).mann_whitney(X = data, output = 'p-value')This also returns a DataFrame of size (n_clusters, n_features), with the calculated statistic or p-value for the cluster/feature combination in each cell.Theleave_one_out()function assesses the global contribution of each feature to the clustering by calculating a global metric like the Calinski-Harabasz score
or the sum of intra-cluster distances (a.k.a inertia).feature_LOO = cs.KMeans(n_clusters = 3).leave_one_out(X = data, metric = 'CH_score')This generates a Pandas Series showing the change in the clustering metric that was calculated when each feature was removed.featurechange_in_CH_score012.371318.716822-8.93893336.74654-9.20392526.19946-3.356187-7.859188-15.04099-21.32391021.8981120.8256123.82213Finally we can build a logistic regression model to calculate a coefficient for each feature in each cluster and return the p-value of the coefficient under the null hypothesis:$$\frac{\beta}{\text{SE}(\beta)} \sim \mathcal{N}(\mu = 0, \sigma^{2} = 1)$$This is done using thelogistic_regression()function, which builds a logistic regression model for each cluster with y = 1 if an instance is in the cluster, and y = 0 if not:feature_LR = cs.KMeans(n_clusters = 3).logistic_regression(X = data, output = 'p-value')which returns a DataFrame of size (n_clusters, n_features), with the calculated value for the cluster/feature combination in each cell.
Types of output can be chosen from 'coef', 'z-score', and 'p-value'.cluster012345678910111200.3988230.01474510.2594530.3987090.210470.3918930.1041190.09952830.3963680.3086280.3081550.3892340.29211910.3583990.3989030.1055710.374340.03277180.306570.3857490.3028240.09474530.009615770.2025490.3697960.16942920.2779880.001473310.1980390.3961210.09521890.358590.1592460.1918150.2995530.1243110.3625650.3905540.354773These p-values are not adjusted.Optimizing clustering hyperparametersClusterSupport also provides functions for optimizing clustering hyperparameters.
Like the feature methods, these functions are called as methods under the clustering classes inherited from scikit-learn (e.g.clustersupport.KMeans())The simplest optimization method iselbow_plot(), which constructs a plot of hyperparameter values compared to a clustering metric.cs.KMeans().elbow_plot(X = data, parameter = 'n_clusters', parameter_range = range(2,10), metric = 'silhouette_score')!Example elbow plotThegap_statistic()method is another function can be used to optimise hyperparameters.
It calculates thegap statisticand its standard errors across a range of hyperparameter values.
For example, to optimise the number of clusters used in K-means clustering, we call the following:gap_statistics = cs.KMeans().gap_statistic(X = data, parameter = 'n_clusters', parameter_range = range(2,10), metric = 'inertia', random_state = 123)Note that then_clustersargument ofKMeans()is not used since the function is iterating through a specified range of hyperparameter values.
Theparameterargument is passed as a string and theparameter_rangeargument should be an iterable containing values of the hyperparameter over which the gap statistic should be calculated.
The function defaults to calculating the gap statistic in terms of changes in inertia but can also be modified to calculate the change in Calinski-Harabasz score, silhouette score, or C-index by changing themetricargument.Tibshirani and colleaguespropose taking the first value of k clusters at which the value of the gap statistic at k clusters is greater than the value for k+1 clusters minus the standard error at k+1 clusters.The function returns a data frame of size(len(parameter_range), 2)which contains the gap statistic for each hyperparameter value and the gap statistic's standard error.n_clustersgap_statisticstandard_error20.014150.0013430.045810.001940.061360.001850.065620.0021260.074780.0024470.1180.003180.117250.0032190.14760.00314We can also use theconsensus_cluster()function to runMonti consensus clusteringover a hyperparameter value range. The function is passed in a similar way to thegap_statistic()function.consensus_data = cs.KMeans().consensus_cluster(X = data, parameter = 'n_clusters', parameter_range = range(2,10), plot = True, random_state = 123)Consensus clustering does not rely on any particular clustering metric. Theplotargument defaults toTrueand causes the function to output the empirical CDFs for consensus values for different hyperparameter values.Monti et alsuggesting picking the number of clusters k at which the largest increase is seen in the area under the CDF between k clusters and k-1 clusters.È˜enbabaoÄŸlu et alsuggest a different method, involving selecting the number of clusters k at which the proportion of unambiguous consensus values (values <0.1 or >0.9) is greatest.Theconsensus_cluster()function returns a Pandas DataFrame of size(len(parameter_range), 2), containing columns for the proportion of unambiguous clusterings and the area under the CDF for every value of the hyperparameter of interest.n_clustersproportion_unambiguous_clusteringsarea_under_cdf210.39630.9990.43140.9750.6450.8270.70460.870.76670.9180.8180.9020.82790.8960.845