Package: denmune
DenMune: A density-peak clustering algorithmDenMune is a clustering algorithm that can find clusters of arbitrary size, shapes and densities in two-dimensions. Higher dimensions are first reduced to 2-D using the t-sne. The algorithm relies on a single parameter K (the number of nearest neighbors). The results show the superiority of the algorithm. Enjoy the simplicity but the power of DenMune.Reproducibility & Test DrivesNow you can reproduce all the research experiments, and even share the results and collaborate to the algorithm using our capsule on CodeOcean. Each Capsule is a self-contained computational experiment with computing environment, code, data, version history, and results.also, you may use our repo2docker offered by mybinder.org, which encapsulate the algorithm and all required data in one virtual machine instance. All Jupyter notebooks examples found in this repository will be also available to you in action to practice in this respo2docer. Thanks mybinder.org, you made it possible!Test-driveURLReproduce our code capsule on Code OceanUse our test-drive on MyBinderScientific WorkPaperJournalDataResearchGate statsCoding, Security & MaintenanceCode StyleInstallationCI WorkflowCode CoverageCode ScanningTutorialsReproducible CapsuleRepo2DockerColabkaggleDownloads Statsdownload/weekdownload/monthTotal downloadsBased on the paperPaperMohamed Abbas, Adel El-Zoghabi, Amin Shoukry,DenMune: Density peak based clustering using mutual nearest neighborsIn: Journal of Pattern Recognition, Elsevier,volume 109, number 107589, January 2021DOI:https://doi.org/10.1016/j.patcog.2020.107589Documentation:Documentation, including tutorials, are available on:Watch it in actionThis 30 seconds will tell you how a density-based algorithm, DenMune propagates:Still interested?Watch this10-minillustrative video on:on YoutubeWhen less means moreMost classic clustering algorithms fail in detecting complex clusters where clusters are of different size, shape, density, and being exist in noisy data.
Recently, a density-based algorithm named DenMune showed great ability in detecting complex shapes even in noisy data. it can detect number of clusters automatically, detect both pre-identified-noise and post-identified-noise automatically and removing them.It can achieve accuracy reach 100% in some classic pattern problems, achieve 97% in MNIST dataset. A great advantage of this algorithm is being single-parameter algorithm. All you need is to set number of k-nearest neighbor and the algorithm will care about the rest. Being Non-sensitive to changes in k, make it robust and stable.Keep in mind, the algorithm reduce any N-D dataset to only 2-D dataset initially, so it is a good benefit of this algorithm is being always to plot your data and explore it which make this algorithm a good candidate for data exploration. Finally, the algorithm comes with neat package for visualizing data, validating it and analyze the whole clustering process.How to install DenMuneSimply install DenMune clustering algorithm using pip command from the official Python repositoryFrom the shell run the commandpipinstalldenmuneFrom Jupyter notebook cell run the command!pip install denmuneHow to use  DenMuneOnce DenMune is installed, you just need to import itfromdenmuneimportDenMunePlease note that first denmune (the package) in small letters, while the other one(the class itself) has D and M in capital case.Read dataThere are four possible cases of data:only train data without labelsonly labeled train datalabeled train data in addition to test data without labelslabeled train data in addition to labeled test data#=============================================# First scenario: train data without labels# ============================================data_path='datasets/denmune/chameleon/'dataset="t7.10k.csv"data_file=data_path+dataset# train data without labelsX_train=pd.read_csv(data_file,sep=',',header=None)knn=39# k-nearest neighbor, the only parameter required by the algorithmdm=DenMune(train_data=X_train,k_nearest=knn)labels,validity=dm.fit_predict(show_analyzer=False,show_noise=True)This is an intuitive dataset which has no groundtruth provided#=============================================# Second scenario: train data with labels# ============================================data_path='datasets/denmune/shapes/'dataset="aggregation.csv"data_file=data_path+dataset# train data with labelsX_train=pd.read_csv(data_file,sep=',',header=None)y_train=X_train.iloc[:,-1]X_train=X_train.drop(X_train.columns[-1],axis=1)knn=6# k-nearest neighbor, the only parameter required by the algorithmdm=DenMune(train_data=X_train,train_truth=y_train,k_nearest=knn)labels,validity=dm.fit_predict(show_analyzer=False,show_noise=True)Datset groundtruthDataset as detected by DenMune at k=6#=================================================================# Third scenario: train data with labels in addition to test data# ================================================================data_path='datasets/denmune/pendigits/'file_2d=data_path+'pendigits-2d.csv'# train data with labelsX_train=pd.read_csv(data_path+'train.csv',sep=',',header=None)y_train=X_train.iloc[:,-1]X_train=X_train.drop(X_train.columns[-1],axis=1)# test data without labelsX_test=pd.read_csv(data_path+'test.csv',sep=',',header=None)X_test=X_test.drop(X_test.columns[-1],axis=1)knn=50# k-nearest neighbor, the only parameter required by the algorithmdm=DenMune(train_data=X_train,train_truth=y_train,test_data=X_test,k_nearest=knn)labels,validity=dm.fit_predict(show_analyzer=True,show_noise=True)dataset groundtruthdataset as detected by DenMune at k=50test data as predicted by DenMune on training the dataset at k=50Algorithm's ParametersParameters used within the initialization of the DenMune classdef__init__(self,train_data=None,test_data=None,train_truth=None,test_truth=None,file_2d=None,k_nearest=None,rgn_tsne=False,prop_step=0,):train_data:data used for training the algorithmdefault: None. It should be provided by the use, otherwise an error will raise.train_truth:labels of training datadefault: Nonetest_data:data used for testing the algorithmtest_truth:labels of testing datadefault: Nonek_nearest:number of nearest neighbordefault: 0. the default is invalid. k-nearest neighbor should be at least 1.rgn_tsn:when set to True: It will regenerate the reduced 2-D version of the N-D dataset each time the algorithm run.when set to False: It will generate the reduced 2-D version of the N-D dataset first time only, then will reuse the saved exist filedefault: Truefile_2d: name (include location) of file used save/load the reduced 2-d versionif empty: the algorithm will create temporary file named '_temp_2d'default: Noneprop_step:size of increment used in showing the clustering propagation.leave this parameter set to 0, the default value, unless you are willing intentionally to enter the propagation mode.default: 0Parameters used within the fit_predict function:deffit_predict(self,validate=True,show_plots=True,show_noise=True,show_analyzer=True):validate:validate data on/off according to five measures integrated with DenMune (Accuracy. F1-score, NMI index, AMI index, ARI index)default: Trueshow_plots:show/hide plotting of datadefault: Trueshow_noise:show/hide noise and outlierdefault: Trueshow_analyzer:show/hide the analyzerdefault: TrueThe AnalyzerThe algorithm provide an exploratory tool called analyzer, once called it will provide you with in-depth analysis on how your clustering results perform.Noise DetectionDenMune detects noise and outlier automatically, no need to any further work from your side.It plots pre-identified noise in blackIt plots post-identified noise in light greyYou can set show_noise parameter to False.# let us show noisem=DenMune(train_data=X_train,k_nearest=knn)labels,validity=dm.fit_predict(show_noise=True)# let us show clean data by removing noisem=DenMune(train_data=X_train,k_nearest=knn)labels,validity=dm.fit_predict(show_noise=False)noisy dataclean dataValidationYou can get your validation results using 3 methodsby showing the Analyzerextract values from the validity returned list from fit_predict functionextract values from the Analyzer dictionaryThere are  five validity measures built-in the algorithm, which are:ACC, AccuracyF1 scoreNMI index (Normalized Mutual Information)AMI index (Adjusted Mutual Information)ARI index (Adjusted Rand Index)K-nearest EvolutionThe following chart shows the evolution of pre and post identified noise in correspondence to increase of number of knn. Also, detected number of clusters is analyzed in the same chart in relation with both types of identified  noise.The Scalabilitydata sizetimedata  size: 5000time: 2.3139 secondsdata  size: 10000time: 5.8752 secondsdata  size: 15000time: 12.4535 secondsdata  size: 20000time: 18.8466 secondsdata  size: 25000time: 28.992 secondsdata  size: 30000time: 39.3166 secondsdata  size: 35000time: 39.4842 secondsdata  size: 40000time: 63.7649 secondsdata  size: 45000time: 73.6828 secondsdata  size: 50000time: 86.9194 secondsdata  size: 55000time: 90.1077 secondsdata  size: 60000time: 125.0228 secondsdata  size: 65000time: 149.1858 secondsdata  size: 70000time: 177.4184 secondsdata  size: 75000time: 204.0712 secondsdata  size: 80000time: 220.502 secondsdata  size: 85000time: 251.7625 secondsdata  size: 100000time: 257.563 seconds|The StabilityThe algorithm is only single-parameter, even more it not sensitive to changes in that parameter, k. You may guess that from the following chart yourself. This is of great benefit for you as a data exploration analyst. You can simply explore the dataset using an arbitrary k. Being Non-sensitive to changes in k, make it robust and stable.Reveal the propagationone of the top performing feature in this algorithm is enabling you to watch how your clusters propagate to construct the final output clusters. just use the parameter 'prop_step' as in the following example:dataset="t7.10k"#data_path='datasets/denmune/chameleon/'# train filedata_file=data_path+dataset+'.csv'X_train=pd.read_csv(data_file,sep=',',header=None)fromitertoolsimportchain# Denmune's Paramatersknn=39# number of k-nearest neighbor, the only parameter required by the algorithm# create list of differnt snapshots of the propagationsnapshots=chain(range(2,5),range(5,50,10),range(50,100,25),range(100,500,100),range(500,2000,250),range(1000,5500,500))fromIPython.displayimportclear_outputforsnapshotinsnapshots:print("itration",snapshot)clear_output(wait=True)dm=DenMune(train_data=X_train,k_nearest=knn,rgn_tsne=False,prop_step=snapshot)labels,validity=dm.fit_predict(show_analyzer=False,show_noise=False)Interact with the algorithmclick image to interactThis notebook allows you interact with the algorithm in many aspects:you can choose which dataset to cluster (among 4 chameleon datasets)you can decide which number of k-nearest neighbor to useshow noise on/off; thus you can invesetigate noise detected by the algorithmshow analyzer on/offWe love Jupyter NotebooksNeed to test examples one by one, then here other two optionsUse colab offered by google research to test each example individually.If you are a kaggler like me, then Kaggle, the best workspace where data scientist meet, should fit you to test the algorithm with great experience.Here is a list of Google CoLab & Kaggle notebooks  to practice the use of  the algorithm interactively.DatasetCoLab notebookKaggle notebookHow to use it?Chameleon datasets2D Shape datasetsClustering unlabeled datairis datasetMNIST datasetScoring 97% on MNIST datasetNoise detectionValidationHow does it propagate?Snapshots of propagationScalabilityStabilityk-nearest-evolutionHow to citeIf you have used this codebase in a scientific publication and wish to cite it, please use theJournal of Pattern Recognition articleMohamed Abbas McInnes, Adel El-Zoghaby, Amin Ahoukry, *DenMune: Density peak based clustering using mutual nearest neighbors*
  In: Journal of Pattern Recognition, Elsevier, volume 109, number 107589.
  January 2021@article{ABBAS2021107589,title={DenMune: Density peak based clustering using mutual nearest neighbors},journal={Pattern Recognition},volume={109},pages={107589},year={2021},issn={0031-3203},doi={https://doi.org/10.1016/j.patcog.2020.107589},url={https://www.sciencedirect.com/science/article/pii/S0031320320303927},author={Mohamed Abbas and Adel El-Zoghabi and Amin Shoukry},keywords={Clustering, Mutual neighbors, Dimensionality reduction, Arbitrary shapes, Pattern recognition, Nearest neighbors, Density peak},abstract={Many clustering algorithms fail when clusters are of arbitrary shapes, of varying densities, or the data classes are unbalanced and close to each other, even in two dimensions. A novel clustering algorithm “DenMune” is presented to meet this challenge. It is based on identifying dense regions using mutual nearest neighborhoods of size K, where K is the only parameter required from the user, besides obeying the mutual nearest neighbor consistency principle. The algorithm is stable for a wide range of values of K. Moreover, it is able to automatically detect and remove noise from the clustering process as well as detecting the target clusters. It produces robust results on various low and high dimensional datasets relative to several known state of the art clustering algorithms.}}LicensingThe DenMune algorithm is 3-clause BSD licensed. Enjoy.Task ListUpdate Github with the DenMune sourcecodecreate repo2docker repositoryCreate pip Packagecreate CoLab shared examplescreate documentationcreate Kaggle shared examplesPEP8 compliantContinuous integrationscikit-learn compatiblecreating unit tests (coverage: 100%)generating API documentationcreate reproducible capsule on codeoceancreate conda package (postponed untill NGT has conda installation)