Density-based Clustering with Kernel Diffusion

Finding a suitable density function is essential for density-based clustering algorithms such as DBSCAN and DPC . A naive density corresponding to the indicator function of a unit d-dimensional Euclidean ball is commonly used in these algorithms . Such a density suffers from an inability to capture local features in complex datasets . To tackle this issue , we propose a new kernel diffusion density function , which is adaptive to data of varying local distributional characteristics and smoothness . Furthermore , we develop a surrogate that can be efficiently computed in linear time and space and prove that it is asymptotically equivalent to the kernel diffusion density function . Extensive empirical experiments on benchmark and large-scale face image datasets show that the proposed approach not only achieves a significant improvement over classic density-based clustering algorithms but also outperforms the state-of-the-art face clustering methods by a large margin . 1 INTRODUCTION . Density-based clustering algorithms are now widely used in a variety of applications , ranging from high energy physics ( Tramacere & Vecchio , 2012 ; Rovere et al. , 2020 ) , material sciences ( Marquis et al. , 2019 ; Reza et al. , 2007 ) , social network analysis ( Shi et al. , 2014 ; Khatoon & Banu , 2019 ) to molecular biology ( Cao et al. , 2017 ; Ziegler et al. , 2020 ) . In these algorithms , data points are partitioned into clusters that are considered to be sufficiently or locally high-density areas with respect to an underlying probability density or a similar reference function . We call them density functions throughout this paper . These techniques are attractive to practitioners , due to their nonparametric feature , which leads to flexibility in discovering clusters that have arbitrary shapes , whilst classic methods such as k-means and k-medoids ( Friedman et al. , 2001 ) can only detect convex ( e.g. , spherical ) clusters . Seminal work in the context of density-based clustering includes DBSCAN ( Ester et al. , 1996 ) and DPC ( Rodriguez & Laio , 2014 ) , among many others ( Ankerst et al. , 1999 ; Cuevas et al. , 2001 ; Comaniciu & Meer , 2002 ; Hinneburg & Gabriel , 2007 ; Stuetzle , 2003 ) . Most density-based clustering algorithms implicitly identify cluster centers and assign remaining points to the clusters by connecting with the higher density point nearby . To proceed with these methods it requires a density function , which is usually an estimate of the underlying true probability density or some variants of it . For example , a popular choice is the naive density function that is carried out by simply calculating the number of data points covered in the ε-neighborhood of each x . Note that such densities are not adaptive to different distribution regions . One of the challenging scenarios is when clusters in the data have varying local features , for example , size , height , spread , and smoothness . Therefore , the resulting density function has a tendency to flatten the peaks and valleys in the data distribution , which leads to underestimation of the number of clusters ( see Figure 1 ) . Many heuristics variations of DBSCAN and DPC have been proposed to magnify the local features , thus making the clustering task easier ( Campello et al. , 2013 ; Chen et al. , 2018 ; Ertöz et al. , 2003 ; Zhu et al. , 2016 ) . Most of these methods can be viewed as performing clustering on certain transformations of the naive density function . However , if the naive density function itself is quite problematic in the first place , these methods will become less effective . Moreover , even if we apply adaptive alternatives to modify the classic density functions , there are other contentious issues of the generally used linear kernel density estimator ( KDE ) . It often suffers from severe boundary bias ( Marron & Ruppert , 1994 ) and is acknowledged as computationally expensive . These phenomenon prevent the classic density functions being practically useful and reliable , especially for large-scale and complex clustering tasks . To overcome these problems in density-based clustering algorithms , in this paper we propose a general approach to build the so-called kernel diffusion density function to replace classic density functions . The key idea is to construct the density from a user-specified bivariate kernel that has desired local adaptive properties . Instead of using the naive density function and its variants , we utilize the bivariate kernel to derive a transition probability . A diffusion process is induced by this transition probability , which admits a limiting and stationary distribution . This limiting distribution serves as a plausible density function for clustering with reduced error . Under this framework , we provide examples of symmetric and asymmetric bivariate kernels to construct the kernel diffusion density function , which can tackle clustering complex and locally varying data . We apply the resulting adapted DBSCAN and DPC algorithms to widely different empirical datasets and show significant improvement in each of these analyses . The main contributions of this paper are summarized below : • We introduce new bivariate kernel functions and construct the associated kernel diffusion processes . Based on the diffusion process , we propose a kernel diffusion density function to adapt density-based clustering algorithms such as DBSACN and DPC , which attains accuracy in the presence of varying local features . • We derive a computationally much more efficient surrogate , and show analytically it is asymptotic equivalent to the proposed kernel diffusion density function . • By extensive experiments , we demonstrate the superiority of kernel diffusion density function over naive density function and its variants when applying to DBSCAN and DPC , and show it outperforms state-of-the-art GCN-based methods on face clustering tasks . 2 RELATED WORK . Density-Based Clustering There is vast literature on adapting density-based clustering algorithms to tackle large variations in different clusters in the data . DPC itself is such a refinement of DBSCAN , as it determines cluster centers not only by highest density values but also by taking into account their distances from each other , thus has a generally better performance in complex clustering tasks . Other attempts include rescaling the data to have relative reference measures instead of KDE ( Zhu et al. , 2016 ; Chen et al. , 2018 ) , and using the number of shared-nearest-neighbors between two points to replace the geometric distance ( Ertöz et al. , 2003 ) . Diffusion Maps The technique of diffusion maps ( Coifman et al. , 2005 ; Coifman & Lafon , 2006 ) gives a multi-scale organization of the data according to their underlying geometric structure . It uses a local similarity measure to create a diffusion process on the data which integrates local geometry at different scales along time t. Generally speaking , the diffusion will segment the data into several smaller clusters in small t and group data into one cluster for large t. Applying eigenfunctions at a carefully selected time t leads to good macroscopic representations of the data , which is useful in dimension reduction and spectral clustering ( Nadler et al. , 2005 ) . Face Clustering Face clustering has been extensively studied as an important application in machine learning . Traditional algorithms include k-means , hierarchical clustering ( Sibson , 1973 ) and ARO ( Otto et al. , 2017 ) . Many recent works take advantage of supervised information and GCN models , achieving impressive improvement comparing to traditional algorithms . To name a few , CDP ( Zhan et al. , 2018 ) proposes to aggregate the features extracted by different models ; L-GCN ( Wang et al. , 2019 ) predicts the linkage in an instance pivot subgraph ; LTC ( Yang et al. , 2019 ) generates a series of subgraphs as proposals and detects face clusters thereon ; and GCN ( V+E ) ( Yang et al. , 2020 ) learns both the confidence and connectivity by GCN . In this paper we demonstrate that the proposed density-based clustering algorithm with kernel diffusion , as a general clustering approach , even outperforms theses state-of-the-art methods that are especially designed for face clustering . 3 PRELIMINARIES . 3.1 NOTATIONS . Let the dataset D = { x1 , . . . , xn } ⊂ Rd be n i.i.d samples drawn from a distribution measure F with density f on Rd . Let Fn denote the corresponding empirical distribution measured with respect to D , i.e. , Fn ( A ) = 1n ∑n i=1 1A ( xi ) , where 1A ( · ) denotes the indicator function of set A . We write ||u|| as the Euclidean norm of vector u . Let B ( x , ε ) and Vd denote the d-dimensional ε-ball centered at x and the volume of the unit ball B ( 0 , 1 ) , respectively . Let Nk ( x ) denote the set of k-nearest neighbors of point x within the dataset D . 3.2 DENSITY FUNCTION . Density-based algorithms perform clustering by specifying and segmenting high-value areas in a density function denoted by ρ . Usually , we calculate each of ρ ( xi ) , and then identify cluster centers with ( locally ) highest values . Many popular algorithms such as DBSCAN and DPC employ the following naive density function : ρnaive ( x ) = 1 nεd ∑ y∈D 1B ( x , ε ) ( y ) Vd . ( 1 ) The naive density function ρnaive is actually an empirical estimation of f for carefully chosen ε . It is easy to observe , for clustering purpose we only care about ρnaive ( x ) up to a normalising constant , which makes it simply equivalent to counting the total number of data points in the ε-ball around x . In practice , the data distribution may be very complex and contains varying local features that are difficult to be detected . The naive density in ( 1 ) with the same radius ε for all x usually suffers from unsatisfactory empirical performance , for example , failing to identify small clusters with fewer data points . One possible way to alleviate this problem is through a transformation into the following local contrast ( LC ) function ( Chen et al. , 2018 ) : ρLC ( x ) = 1 n ∑ y∈Nk ( x ) 1ρnaive ( x ) > ρnaive ( y ) . ( 2 ) In this way , ρLC compares the density of each data point with its k-nearest neighbors . To see the benefit of LC , let us consider x to be a cluster center . After local contrasting , ρLC ( x ) is likely to reach the value of k regardless of the size of this cluster . However density functions like ρLC still highly depend on the underpinning performance of ρnaive . This restricts their applications in clustering data with challenging local features . 4 METHODOLOGY . In this section , we present a new type of density-based clustering algorithm , based on the notion of kernel diffusion density function . Towards this end , we will introduce a kernel diffusion density function , which takes account of local adaptability and is well-suited for clustering purpose . We provide details on how to derive this density function from a diffusion process induced by bivariate kernels . We also provide a surrogate density function that is computationally more efficient . 4.1 DIFFUSION PROCESS AND KERNEL DIFFUSION DENSITY . Considering a bivariate kernel function k : D ×D → R+ , such that : • k ( x , y ) is positive semi-definite , i.e. , k ( x , y ) ≥ 0 . • k ( x , y ) is Fn-integrable with respect to both x and y . We define d ( x ) = n ∫ D k ( x , y ) dFn ( y ) as a local measure of the volume at x and define p ( x , y ) = k ( x , y ) d ( x ) . ( 3 ) It is easy to see that p ( x , y ) satisfies the conservation property n ∫ D p ( x , y ) dFn ( y ) = 1 . As a result , p ( x , y ) can be viewed as a probability for a random walk on the dataset from point x to point y , which induces a Markov chain on D with n× n transition matrix P = [ p ( x , y ) ] . This technique is standard in various applications , known as the normalized graph Laplacian construction . For example , we can view D as a graph , L = I −P as the normalized graph Laplacian , and d ( x ) as a normalization factor . For t ≥ 0 , the probability of transiting from x to y in t time steps is given by P t , the t-th power of P . Running the Markov chain forward in time , we observe the dataset at different scales , which is the diffusion process Xt on D. Let ρ ( x , t ) : D × R+ → R+ be the associated probability density , which is governed by the following second-order differential equation with initial conditions : { ∂ ∂tρ ( x , t ) = −Lρ ( x , t ) , ρ ( x , 0 ) = ϕ0 ( x ) , ( 4 ) where ϕ0 ( x ) is a probability density at time t = 0 . In practice we can use any valid choice of ϕ0 ( x ) , e.g . the uniform density . To give an explicit example of the diffusion process , consider a sub-class of k , i.e. , isotropic kernels , where k ( x , y ) = K ( ||x− y||2/h ) for some function K : R → R+ . Here we can dual interpret h as a scale parameter to infer local information and as a time step h = ∆t at which the random walk jumps . Then we can define the forward Chapman-Kolmogorov operator TF as TF ( x ) = n ∫ D p ( x , y ) ϕ0 ( y ) dFn ( y ) . Note that TF is the data distribution at time t = h , thus can be viewed as continuous analogues of the left multiplication by the transition matrix P . Letting h → 0 , the random walk converges to a continuous diffusion process with probability density evolves continuously in t. In this case , we can explicitly write the second-order differential equation in ( 4 ) as : ∂ ∂t ρ ( x , t ) = lim h→0 ρ̂ ( x , t+ h ) − ρ ( x , t ) h = lim h→0 TF − I h ρ ( x , t ) , ( 5 ) where Lh = limh→0 ( TF − I ) /h is the conventional infinitesimal generator of the process . Now we are ready to introduce our kernel diffusion density function . Definition 1 . ( Kernel diffusion density function ) Suppose the Markov chain induced by P is ergodic , we define the kernel diffusion density function as the limiting probability density of the diffusion process Xt , i.e. , ρKD ( x ) = lim t→∞ ρ ( x , t ) . ( 6 ) Intuitively , on the one hand , with increased values of t we expect the diffusion process Xt gradually reveals the geometric structure ( such as high-density regions ) of the data distribution F . To see this , note that the transition probability P reflects connectivity between data . We can interpret a cluster as an underlying geometric structure in which the probability of staying in this region is high during a transition . In the diffusion process , the probability of following a path along a structure usually increases with t , as the involved data points are dense and highly connected . Therefore , the path consists of short and high probability jumps . Whilst paths that do not follow any structure consists of long and low probability jumps , which lowers their overall probability . As a result , geometry structures of F is magnified during the diffusion . On the other hand , by talking certain sophisticated forms of k ( x , y ) that take into account of local adaptivity , we also slow down the diffusion to avoid trivial geometry structures such as one big cluster for all the data points . In this way , we can eventually identify the correct geometry structure at the right scale .

Summary: This paper proposes a different density function for popular density-based clustering algorithms, i.e. truncated symmetric and asymmetric Gaussian kernels for DBSCAN and DPC. It offers some diffusion-based argument for why that is a better density function for clustering than naive function, and provides a computationally more efficient surrogate function as well.  Performance of proposed method on many clustering experiments are provided.