Package: clust-learn
Clust-learnA Python package for extracting information from large and high-dimensional mixed-type data through explainable cluster analysis.Table of contentsIntroductionOverall architectureImplementationInstallationVersion and license informationBug reports and future workUser guide & APIData processingData imputationOutliersDimensionality reductionDimensionalityReduction classClass methodsClusteringClustering classClass methodsClassifierClassifier classClass methodsCiting1. Introductionclust-learnenables users to run end-to-end explainable cluster analysis to extract information from large and high-dimensional
mixed-type data, and it does so by providing a framework that guides the user through data preprocessing, dimensionality reduction,
clustering, and classification of the obtained clusters. It is designed to require very few lines of code, and with a strong
focus on explainability.2. Overall architectureclust-learnis organized into four modules, one for each component of the methodological framework presentedhere:data_preprocessingdimensionality_reductionclusteringclassifierFigue 1shows the package layout with the functionalities covered by each module along with the techniques used, the
explainability strategies available, and the main functions and class methods encapsulating these techniques and
explainability strategies.3. ImplementationThe package is implemented with Python 3.9 using open source libraries. It relies heavily onpandasandscikit-learn. Read the complete list of requirementshere.It can be installed manually or from pip/PyPI (see Section4. Installation).4. InstallationThe package is onPyPI. Simply run:pip install clust-learn5. Version and license informationVersion: 0.2.5Author: Miguel Alvarez-Garcia (malvarez.statistics@gmail.com)License: GPLv36. Bug reports and future workPlease report bugs and feature requests through creating a new issuehere.7. User guide & APIclust-learnis organized into four modules:Data preprocessingDimensionality reductionClusteringClassifierFigue 1shows the package layout with the functionalities covered by each module along with the techniques used, the explainability strategies available, and the main functions and class methods encapsulating these techniques and explainability strategies.The four modules are designed to be used sequentially to ensure robust and explainable results. However, each of them is independent and can be used separately to suit different use cases.7.i. Data preprocessingData preprocessing consists of a set of manipulation and transformation tasks performed on the raw data before it is used for its analysis. Although data quality is essential for obtaining robust and reliable results, real-world data is often incomplete, noisy, or inconsistent. Therefore, data preprocessing is a crucial step in any analytical study.7.i.a. Data imputationcompute_missing()compute_missing(df, normalize=True)Calculates the pct/count of missing values per column.Parametersdf:pandas.DataFramenormalize:boolean, default=TrueReturnsmissing_df:pandas.DataFrameDataFrame with the pct/counts of missing values per column.missing_values_heatmap()missing_values_heatmap(df, output_path=None, savefig_kws=None)Plots a heatmap to visualize missing values (light color).Parametersdf:pandas.DataFrameDataFrame containing the data.output_path:str, default=NonePath to save figure as image.savefig_kws:dict, default=NoneSave figure options.impute_missing_values()impute_missing_values(df, num_vars, cat_vars, num_pair_kws=None, mixed_pair_kws=None, cat_pair_kws=None, graph_thres=0.05, k=8, max_missing_thres=0.33)This function imputes missing values following this steps:One-to-one model based imputation for strongly related variables.Cluster based hot deck imputation where clusters are obtained as the connected components of an undirected graphG=(V,E), whereVis the set of variables andEthe pairs of variables with mutual information above a predefined threshold.Records with a proportion of missing values above a predefined threshold are discarded to ensure the quality of the hot deck imputation.Hot deck imputation for the remaining missing values considering all variables together.Parametersdf:pandas.DataFrameData frame containing the data with potential missing values.num_vars:str,list,pandas.Series, ornumpy.arrayNumerical variable name(s).cat_vars:str,list,pandas.Series, ornumpy.arrayCategorical variable name(s).{num,mixed,cat}_pair_kws:dict, default=NoneAdditional keyword arguments to pass to compute imputation pairs for one-to-one model based imputation, namely:For numerical pairs,corr_thresandmethodfor setting the correlation coefficient threshold and method. By default,corr_thres=0.7andmethod='pearson'.For mixed-type pairs,np2_thresfor setting the a threshold on partialetasquare with 0.14 as default value.For categorical pairs,mi_thresfor setting a threshold on mutual information score. By default,mi_thres=0.6.graph_thres:float, default=0.05Threshold to determine if two variables are similar based on mutual information score, and therefore are an edge of the graph from which variable clusters are derived.k:int, default=8Number of neighbors to consider in hot deck imputation.max_missing_thres:float, default=0.33Maximum proportion of missing values per observation allowed before final general hot deck imputation - see step 3 of the missing value imputation methodology in section 2.1.Returnsfinal_pairs:pandas.DataFrameDataFrame with pairs of highly correlated variables (var1: variable with values to impute;var2: variable to be used as independent variable for model-based imputation), together proportion of missing values of variablesvar1andvar2.plot_imputation_distribution_assessment()plot_imputation_distribution_assessment(df_prior, df_posterior, imputed_vars, sample_frac=1.0, prior_kws=None, posterior_kws=None, output_path=None, savefig_kws=None)Plots a distribution comparison of each variable with imputed variables, before and after imputation.Parametersdf_prior:pandas.DataFrameDataFrame containing the data before imputation.df_posterior:pandas.DataFrameDataFrame containing the data after imputation.imputed_vars:listList of variables with imputed variables.sample_frac: float, default=1.0If < 1 a random sample of every pair of variables will be plotted.{prior,posterior}_kws:dict, default=NoneAdditional keyword arguments to pass to thekdeplot.output_path:str, default=NonePath to save figure as image.savefig_kws:dict, default=NoneSave figure options.7.i.b. Outliersremove_outliers()remove_outliers(df, variables, iforest_kws=None)Removes outliers using theIsolation Forest algorithm.Parametersdf:pandas.DataFrameDataFrame containing the data.variables:listVariables with potential outliers.iforest_kws:dict, default=NoneIsolationForest algorithm hyperparameters.Returnsdf_inliers :pandas.DataFrameDataFrame with inliers (i.e. observations that are not outliers).df_outliers :pandas.DataFrameDataFrame with outliers.7.ii. Dimensionality reductionAll the functionality of this module is encapsulated in theDimensionalityReductionclass so that the original data, the instances of the models used, and any other relevant information is self-maintained and always accessible.DimensionalityReduction classdr = DimensionalityReduction(df, num_vars=None, cat_vars=None, num_algorithm='pca', cat_algorithm='mca', num_kwargs=None, cat_kwargs=None)ParameterTypeDescriptiondfpandas.DataFrameData table containing the data with the original variablesnum_varsstring,list,pandas.Series, ornumpy.arrayNumerical variable name(s)cat_varsstring,list,pandas.Series, ornumpy.arrayCategorical variable name(s)num_algorithmstringAlgorithm to be used for dimensionality reduction of numerical variables. By default, PCA is used. The current version also supports SPCAcat_algorithmstringAlgorithm to be used for dimensionality reduction of categorical variables. By default, MCA is used. The current version doesn’t support other algorithmsnum_kwargsdictionaryAdditional keyword arguments to pass to the model used for numerical variablescat_kwargsdictionaryAdditional keyword arguments to pass to the model used for categorical variablesAttributeTypeDescriptionn_components_intFinal number of extracted componentsmin_explained_variance_ratio_floatMinimum explained variance ratio. By default, 0.5num_trans_pandas.DataFrameExtracted components from numerical variablescat_trans_pandas.DataFrameExtracted components from categorical variablesnum_components_listList of names assigned to the extracted components from numerical variablescat_components_listList of names assigned to the extracted components from categorical variablespca_sklearn.decomposition.PCAPCA instance used to speed up some computations and for comparison purposesMethodstransform()Sourcetransform(self, n_components=None, min_explained_variance_ratio=0.5)Transforms a DataFrame df to a lower dimensional space.num_main_contributors(()Sourcenum_main_contributors(self, thres=0.5, n_contributors=None, dim_idx=None, component_description=None, col_description=None, output_path=None)Computes the original numerical variables with the strongest relation to the derived variable(s) (measured as Pearson correlation coefficient).cat_main_contributors(()Sourcecat_main_contributors(self, thres=0.14, n_contributors=None, dim_idx=None, component_description=None, col_description=None, output_path=None)Computes the original categorical variables with the strongest relation to the derived variable(s)(measured as correlation ratio).cat_main_contributors_stats()Sourcecat_main_contributors_stats(self, thres=0.14, n_contributors=None, dim_idx=None, output_path=None)Computes for every categorical variable's value, the mean and std of the derived variables that are strongly related to the categorical variable (based on the correlation ratio)).plot_num_explained_variance()Sourceplot_num_explained_variance(self, thres=0.5, plots='all', output_path=None, savefig_kws=None)Plot the explained variance (ratio, cumulative, and/or normalized) for numerical variables.plot_cat_explained_variance()Sourceplot_cat_explained_variance(self, thres=0.5, plots='all', output_path=None, savefig_kws=None)Plot the explained variance (ratio, cumulative, and/or normalized) for categorical variables.plot_num_main_contributors()Sourceplot_num_main_contributors(self, thres=0.5, n_contributors=5, dim_idx=None, output_path=None, savefig_kws=None)Plot main contributors (original variables with the strongest relation with derived variables) for every derived variable.plot_cat_main_contributor_distribution()Sourceplot_cat_main_contributor_distribution(self, thres=0.14, n_contributors=None, dim_idx=None, output_path=None, savefig_kws=None)Plot main contributors (original variables with the strongest relation with derived variables) for every derived variable.7.iii. ClusteringTheClusteringclass encapsulates all the functionality of this module and stores the data, the instances of the algorithms used, and other relevant information so it is always accessible.Clustering classcl = Clustering(df, algorithms='kmeans', normalize=False)ParameterTypeDescriptiondfpandas.DataFrameData frame containing the data to be clusteredalgorithmsinstance orlistof instancesAlgorithm instances to be used for clustering. They must implement thefitandset_paramsmethodsnormalizeboolWhether to apply data normalization for fair comparisons between variables. In case dimensionality reduction is applied beforehand, normalization should not be appliedAttributeTypeDescriptiondimensions_listList of columns of they input data frameinstances_dictPairs of algorithm name and its instancemetric_stringThe cluster validation metric used. Four metrics available: ['inertia', 'davies_bouldin_score', 'silhouette_score',  'calinski_harabasz_score']optimal_config_tupleTuple with the optimal configuration for clustering containing the algorithm name, number of clusters, and value of the chosen validation metricscores_dictPairs of algorithm name and a list of values of the chosen validation metric for a cluster rangeMethodscompute_clusters()Sourcecompute_clusters(self, n_clusters=None, metric='inertia', max_clusters=10, prefix=None, weights=None)Calculates clusters.
If more than one algorithm is passed in the class constructor, first, the optimal number of clusters
is computed for each algorithm based on the metric passed to the method. Secondly, the algorithm that
provides the best performance for the corresponding optimal number of clusters is selected.
Therefore, the result shows the clusters calculated with the best performing algorithm based on the
criteria explained above.describe_clusters()Sourcedescribe_clusters(self, df_ext=None, variables=None, cluster_filter=None, statistics=['mean', 'median', 'std'], output_path=None)Describes clusters based on internal or externalcontinuousvariables.
For categorical variables usedescribe_clusters_cat().describe_clusters_cat()Sourcedescribe_clusters_cat(self, cat_array, cat_name, order=None, normalize=False, use_weights=False, output_path=None)Describes clusters based on  externalcategoricalvariables. The result is a contingency table.
For continuous variables usedescribe_clusters().compare_cluster_means_to_global_means()Sourcecompare_cluster_means_to_global_means(self, df_original=None, output_path=None)For every cluster and every internal variable, the relative difference between the intra-cluster mean
and the global mean.anova_tests()Sourceanova_tests(self, df_test=None, vars_test=None, cluster_filter=None, output_path=None)Runs ANOVA tests for a given set of continuous variables (internal or external) to test dependency with clusters.chi2_test()Sourcechi2_test(self, cat_array)Runs Chi-squared tests for a given categorical variable to test dependency with clusters.plot_score_comparison()Sourceplot_score_comparison(self, output_path=None, savefig_kws=None)Plots the comparison in performance between the different clustering algorithms.plot_optimal_components_normalized()Sourceplot_optimal_components_normalized(self, output_path=None, savefig_kws=None)Plots the normalized curve used for computing the optimal number of clusters.plot_clustercount()Sourceplot_clustercount(self, use_weights=False, output_path=None, savefig_kws=None)Plots a bar plot with cluster counts.plot_cluster_means_to_global_means_comparison()Sourceplot_cluster_means_to_global_means_comparison(self, use_weights= False, df_original=None, xlabel=None, ylabel=None,
                                              levels=[-0.50, -0.32, -0.17, -0.05, 0.05, 0.17, 0.32, 0.50],
                                              output_path=None, savefig_kws=None)Plots the normalized curve used for computing the optimal number of clusters.plot_distribution_comparison_by_cluster()Sourceplot_distribution_comparison_by_cluster(self, df_ext=None, xlabel=None, ylabel=None, output_path=None, savefig_kws=None)Plots the violin plots per cluster andcontinuousvariables of interest to understand differences in their distributions by cluster.plot_clusters_2D()Sourceplot_clusters_2D(self, coor1, coor2, use_weights=False, style_kwargs=dict(), output_path=None, savefig_kws=None)Plots two 2D plots:
- A scatter plot styled by the categorical variablehue.
- A 2D plot comparing cluster centroids and optionally the density area.plot_cat_distribution_by_cluster()Sourceplot_cat_distribution_by_cluster(self, cat_array, cat_label, order=None, cluster_label=None, use_weights=False, output_path=None, savefig_kws=None)Plots the relative contingency table of the clusters with a categorical variable as a stacked bar plot.7.iv. ClassifierThe functionality of this module is encapsulated in theClassifierclass, which is also responsible for storing the original data, the instances of the models used, and any other relevant information.Classifier classclassifier = Classifier(df, predictor_cols, target, num_cols=None, cat_cols=None)ParameterTypeDescriptiondfpandas.DataFrameData frame containing the datapredictor_colslistofstringList of columns to use as predictorstargetnumpy.arrayorlistValues of the target variablenum_colslistList of numerical columns from predictor_colscat_colslistList of categorical columns from predictor_colsAttributeTypeDescriptionfiltered_features_listList of columns of the input data framelabels_listList of class labelsmodel_Instance ofTransformerMixinandBaseEstimatorfromsklearn.baseTrained classifierX_train_numpy.arrayTrain split of predictorsX_test_numpy.arrayTest split of predictorsy_train_numpy.arrayTrain split of targety_test_numpy.arrayTest split of targetgrid_result_sklearn.model_selection.GridSearchCVInstance of fitted estimator for hyperparameter tuningMethodstrain_model()Sourcetrain_model(self, model=None, feature_selection=True, features_to_keep=[],
			feature_selection_model=None, hyperparameter_tuning=False, param_grid=None,
			train_size=0.8, balance_classes=False)This method trains a classification model.By default, it uses XGBoost, but any other estimator (instance ofscikit-learn.Estimator) can be used.The building process consists of three main steps:Feature Selection (optional)Feature removing highly correlated variables using a classification model and SHAP values
to determine which to keep, and Recursive Feature Elimination with Cross-Validation (RFECV)
on the remaining features.Hyperparameter tuning (optional)Runs grid search with cross-validation for hyperparameter tuning.Notethe parameter grid
must be passed.Model trainingTrains a classification model with the selected features and hyperparameters. By default, an XGBoost
classifier will be trained.Noteboth hyperparameter tuning and model training are run on a train set. Train-test split is performed
usingsklearn.model_selection.train_test_split.hyperparameter_tuning_metrics()Sourcehyperparameter_tuning_metrics(self, output_path=None)This method returns the average and standard deviation of the cross-validation runs for every hyperparameter
combination in hyperparameter tuning.confusion_matrix()Sourceconfusion_matrix(self, test=True, sum_stats=True, output_path=None)This method returns the confusion matrix of the classification model.classification_report()Sourceclassification_report(self, test=True, output_path=None)This method returns thesklearn.metrics.classification_reportinpandas.DataFrameformat.This report contains the intra-class metrics precision, recall and F1-score, together with the global accuracy,
and macro average and weighted average of the three intra-class metrics.plot_shap_importances()Sourceplot_shap_importances(self, n_top=7, output_path=None, savefig_kws=None)Plots shap importance values, calculated as the combined average of the absolute values of the shap values
for all classes.plot_shap_importances_beeswarm()Sourceplot_shap_importances_beeswarm(self, class_id, class_name=None, n_top=10, output_path=None, savefig_kws=None)Plots a summary of shap values for a specific class of the target variable. This usesshap beeswarm plot.plot_confusion_matrix()Sourceplot_confusion_matrix(self, test=True, sum_stats=True, output_path=None, savefig_kws=None)This function makes a pretty plot of an sklearn Confusion Matrix cf using a Seaborn heatmap visualization.plot_roc_curves()Sourceplot_roc_curves(self, test=True, labels=None, output_path=None, savefig_kws=None)Plots ROC curve for every class.8. Citing<>