Package: hdbscan-with-cosine-distance
HDBSCANHDBSCAN - Hierarchical Density-Based Spatial Clustering of Applications
with Noise. Performs DBSCAN over varying epsilon values and integrates
the result to find a clustering that gives the best stability over epsilon.
This allows HDBSCAN to find clusters of varying densities (unlike DBSCAN),
and be more robust to parameter selection.In practice this means that HDBSCAN returns a good clustering straight
away with little or no parameter tuning – and the primary parameter,
minimum cluster size, is intuitive and easy to select.HDBSCAN is ideal for exploratory data analysis; it’s a fast and robust
algorithm that you can trust to return meaningful clusters (if there
are any).Based on the paper:R. Campello, D. Moulavi, and J. Sander,Density-Based Clustering Based on
Hierarchical Density EstimatesIn: Advances in Knowledge Discovery and Data Mining, Springer, pp 160-172.
2013Documentation, including tutorials, are available on ReadTheDocs athttp://hdbscan.readthedocs.io/en/latest/.Notebookscomparing HDBSCAN to other clustering algorithms, explaininghow HDBSCAN worksandcomparing performance with other python clustering implementationsare available.How to use HDBSCANThe hdbscan package inherits from sklearn classes, and thus drops in neatly
next to other sklearn clusterers with an identical calling API. Similarly it
supports input in a variety of formats: an array (or pandas dataframe, or
sparse matrix) of shape(num_samples x num_features); an array (or sparse matrix)
giving a distance matrix between samples.importhdbscanclusterer=hdbscan.HDBSCAN(min_cluster_size=10)cluster_labels=clusterer.fit_predict(data)PerformanceSignificant effort has been put into making the hdbscan implementation as fast as
possible. It isorders of magnitude faster than the reference implementationin Java,
and is currently faster than highly optimized single linkage implementations in C and C++.version 0.7 performance can be seen in this notebook.
In particularperformance on low dimensional data is better than sklearn’s DBSCAN,
and via support for caching with joblib, re-clustering with different parameters
can be almost free.Additional functionalityThe hdbscan package comes equipped with visualization tools to help you
understand your clustering results. After fitting data the clusterer
object has attributes for:The condensed cluster hierarchyThe robust single linkage cluster hierarchyThe reachability distance minimal spanning treeAll of which come equipped with methods for plotting and converting
to Pandas or NetworkX for further analysis. See the notebook onhow HDBSCAN worksfor examples and further details.The clusterer objects also have an attribute providing cluster membership
strengths, resulting in optional soft clustering (and no further compute
expense). Finally each cluster also receives a persistence score giving
the stability of the cluster over the range of distance scales present
in the data. This provides a measure of the relative strength of clusters.Outlier DetectionThe HDBSCAN clusterer objects also support the GLOSH outlier detection algorithm.
After fitting the clusterer to data the outlier scores can be accessed via theoutlier_scores_attribute. The result is a vector of score values, one for
each data point that was fit. Higher scores represent more outlier like objects.
Selecting outliers via upper quantiles is often a good approach.Based on the paper:R.J.G.B. Campello, D. Moulavi, A. Zimek and J. SanderHierarchical Density Estimates for Data Clustering, Visualization, and Outlier Detection,
ACM Trans. on Knowledge Discovery from Data, Vol 10, 1 (July 2015), 1-51.Robust single linkageThe hdbscan package also provides support for therobust single linkageclustering algorithm of Chaudhuri and Dasgupta. As with the HDBSCAN
implementation this is a high performance version of the algorithm
outperforming scipy’s standard single linkage implementation. The
robust single linkage hierarchy is available as an attribute of
the robust single linkage clusterer, again with the ability to plot
or export the hierarchy, and to extract flat clusterings at a given
cut level and gamma value.Example usage:importhdbscanclusterer=hdbscan.RobustSingleLinkage(cut=0.125,k=7)cluster_labels=clusterer.fit_predict(data)hierarchy=clusterer.cluster_hierarchy_alt_labels=hierarchy.get_clusters(0.100,5)hierarchy.plot()Based on the paper:K. Chaudhuri and S. Dasgupta.“Rates of convergence for the cluster tree.”In Advances in Neural Information Processing Systems, 2010.InstallingEasiest install, if you have Anaconda (thanks to conda-forge which is awesome!):condainstall-cconda-forgehdbscanPyPI install, presuming you have sklearn and all its requirements installed:pipinstallhdbscanIf pip is having difficulties pulling the dependencies then we’d suggest installing
the dependencies manually using anaconda followed by pulling hdbscan from pip:condainstallcythoncondainstallscikit-learnpipinstallhdbscanFor a manual install get this package:wgethttps://github.com/lmcinnes/hdbscan/archive/master.zipunzipmaster.ziprmmaster.zipcdhdbscan-masterInstall the requirementssudopipinstall-rrequirements.txtorcondainstallscikit-learncythonInstall the packagepythonsetup.pyinstallLicensingThe hdbscan package is 3-clause BSD licensed. Enjoy.