Density-based Clustering with Kernel Diffusion

Finding a suitable density function is essential for density-based clustering algorithms such as DBSCAN and DPC . A naive density corresponding to the indicator function of a unit d-dimensional Euclidean ball is commonly used in these algorithms . Such a density suffers from an inability to capture local features in complex datasets . To tackle this issue , we propose a new kernel diffusion density function , which is adaptive to data of varying local distributional characteristics and smoothness . Furthermore , we develop a surrogate that can be efficiently computed in linear time and space and prove that it is asymptotically equivalent to the kernel diffusion density function . Extensive empirical experiments on benchmark and large-scale face image datasets show that the proposed approach not only achieves a significant improvement over classic density-based clustering algorithms but also outperforms the state-of-the-art face clustering methods by a large margin . 1 INTRODUCTION . Density-based clustering algorithms are now widely used in a variety of applications , ranging from high energy physics ( Tramacere & Vecchio , 2012 ; Rovere et al. , 2020 ) , material sciences ( Marquis et al. , 2019 ; Reza et al. , 2007 ) , social network analysis ( Shi et al. , 2014 ; Khatoon & Banu , 2019 ) to molecular biology ( Cao et al. , 2017 ; Ziegler et al. , 2020 ) . In these algorithms , data points are partitioned into clusters that are considered to be sufficiently or locally high-density areas with respect to an underlying probability density or a similar reference function . We call them density functions throughout this paper . These techniques are attractive to practitioners , due to their nonparametric feature , which leads to flexibility in discovering clusters that have arbitrary shapes , whilst classic methods such as k-means and k-medoids ( Friedman et al. , 2001 ) can only detect convex ( e.g. , spherical ) clusters . Seminal work in the context of density-based clustering includes DBSCAN ( Ester et al. , 1996 ) and DPC ( Rodriguez & Laio , 2014 ) , among many others ( Ankerst et al. , 1999 ; Cuevas et al. , 2001 ; Comaniciu & Meer , 2002 ; Hinneburg & Gabriel , 2007 ; Stuetzle , 2003 ) . Most density-based clustering algorithms implicitly identify cluster centers and assign remaining points to the clusters by connecting with the higher density point nearby . To proceed with these methods it requires a density function , which is usually an estimate of the underlying true probability density or some variants of it . For example , a popular choice is the naive density function that is carried out by simply calculating the number of data points covered in the ε-neighborhood of each x . Note that such densities are not adaptive to different distribution regions . One of the challenging scenarios is when clusters in the data have varying local features , for example , size , height , spread , and smoothness . Therefore , the resulting density function has a tendency to flatten the peaks and valleys in the data distribution , which leads to underestimation of the number of clusters ( see Figure 1 ) . Many heuristics variations of DBSCAN and DPC have been proposed to magnify the local features , thus making the clustering task easier ( Campello et al. , 2013 ; Chen et al. , 2018 ; Ertöz et al. , 2003 ; Zhu et al. , 2016 ) . Most of these methods can be viewed as performing clustering on certain transformations of the naive density function . However , if the naive density function itself is quite problematic in the first place , these methods will become less effective . Moreover , even if we apply adaptive alternatives to modify the classic density functions , there are other contentious issues of the generally used linear kernel density estimator ( KDE ) . It often suffers from severe boundary bias ( Marron & Ruppert , 1994 ) and is acknowledged as computationally expensive . These phenomenon prevent the classic density functions being practically useful and reliable , especially for large-scale and complex clustering tasks . To overcome these problems in density-based clustering algorithms , in this paper we propose a general approach to build the so-called kernel diffusion density function to replace classic density functions . The key idea is to construct the density from a user-specified bivariate kernel that has desired local adaptive properties . Instead of using the naive density function and its variants , we utilize the bivariate kernel to derive a transition probability . A diffusion process is induced by this transition probability , which admits a limiting and stationary distribution . This limiting distribution serves as a plausible density function for clustering with reduced error . Under this framework , we provide examples of symmetric and asymmetric bivariate kernels to construct the kernel diffusion density function , which can tackle clustering complex and locally varying data . We apply the resulting adapted DBSCAN and DPC algorithms to widely different empirical datasets and show significant improvement in each of these analyses . The main contributions of this paper are summarized below : • We introduce new bivariate kernel functions and construct the associated kernel diffusion processes . Based on the diffusion process , we propose a kernel diffusion density function to adapt density-based clustering algorithms such as DBSACN and DPC , which attains accuracy in the presence of varying local features . • We derive a computationally much more efficient surrogate , and show analytically it is asymptotic equivalent to the proposed kernel diffusion density function . • By extensive experiments , we demonstrate the superiority of kernel diffusion density function over naive density function and its variants when applying to DBSCAN and DPC , and show it outperforms state-of-the-art GCN-based methods on face clustering tasks . 2 RELATED WORK . Density-Based Clustering There is vast literature on adapting density-based clustering algorithms to tackle large variations in different clusters in the data . DPC itself is such a refinement of DBSCAN , as it determines cluster centers not only by highest density values but also by taking into account their distances from each other , thus has a generally better performance in complex clustering tasks . Other attempts include rescaling the data to have relative reference measures instead of KDE ( Zhu et al. , 2016 ; Chen et al. , 2018 ) , and using the number of shared-nearest-neighbors between two points to replace the geometric distance ( Ertöz et al. , 2003 ) . Diffusion Maps The technique of diffusion maps ( Coifman et al. , 2005 ; Coifman & Lafon , 2006 ) gives a multi-scale organization of the data according to their underlying geometric structure . It uses a local similarity measure to create a diffusion process on the data which integrates local geometry at different scales along time t. Generally speaking , the diffusion will segment the data into several smaller clusters in small t and group data into one cluster for large t. Applying eigenfunctions at a carefully selected time t leads to good macroscopic representations of the data , which is useful in dimension reduction and spectral clustering ( Nadler et al. , 2005 ) . Face Clustering Face clustering has been extensively studied as an important application in machine learning . Traditional algorithms include k-means , hierarchical clustering ( Sibson , 1973 ) and ARO ( Otto et al. , 2017 ) . Many recent works take advantage of supervised information and GCN models , achieving impressive improvement comparing to traditional algorithms . To name a few , CDP ( Zhan et al. , 2018 ) proposes to aggregate the features extracted by different models ; L-GCN ( Wang et al. , 2019 ) predicts the linkage in an instance pivot subgraph ; LTC ( Yang et al. , 2019 ) generates a series of subgraphs as proposals and detects face clusters thereon ; and GCN ( V+E ) ( Yang et al. , 2020 ) learns both the confidence and connectivity by GCN . In this paper we demonstrate that the proposed density-based clustering algorithm with kernel diffusion , as a general clustering approach , even outperforms theses state-of-the-art methods that are especially designed for face clustering . 3 PRELIMINARIES . 3.1 NOTATIONS . Let the dataset D = { x1 , . . . , xn } ⊂ Rd be n i.i.d samples drawn from a distribution measure F with density f on Rd . Let Fn denote the corresponding empirical distribution measured with respect to D , i.e. , Fn ( A ) = 1n ∑n i=1 1A ( xi ) , where 1A ( · ) denotes the indicator function of set A . We write ||u|| as the Euclidean norm of vector u . Let B ( x , ε ) and Vd denote the d-dimensional ε-ball centered at x and the volume of the unit ball B ( 0 , 1 ) , respectively . Let Nk ( x ) denote the set of k-nearest neighbors of point x within the dataset D . 3.2 DENSITY FUNCTION . Density-based algorithms perform clustering by specifying and segmenting high-value areas in a density function denoted by ρ . Usually , we calculate each of ρ ( xi ) , and then identify cluster centers with ( locally ) highest values . Many popular algorithms such as DBSCAN and DPC employ the following naive density function : ρnaive ( x ) = 1 nεd ∑ y∈D 1B ( x , ε ) ( y ) Vd . ( 1 ) The naive density function ρnaive is actually an empirical estimation of f for carefully chosen ε . It is easy to observe , for clustering purpose we only care about ρnaive ( x ) up to a normalising constant , which makes it simply equivalent to counting the total number of data points in the ε-ball around x . In practice , the data distribution may be very complex and contains varying local features that are difficult to be detected . The naive density in ( 1 ) with the same radius ε for all x usually suffers from unsatisfactory empirical performance , for example , failing to identify small clusters with fewer data points . One possible way to alleviate this problem is through a transformation into the following local contrast ( LC ) function ( Chen et al. , 2018 ) : ρLC ( x ) = 1 n ∑ y∈Nk ( x ) 1ρnaive ( x ) > ρnaive ( y ) . ( 2 ) In this way , ρLC compares the density of each data point with its k-nearest neighbors . To see the benefit of LC , let us consider x to be a cluster center . After local contrasting , ρLC ( x ) is likely to reach the value of k regardless of the size of this cluster . However density functions like ρLC still highly depend on the underpinning performance of ρnaive . This restricts their applications in clustering data with challenging local features . 4 METHODOLOGY . In this section , we present a new type of density-based clustering algorithm , based on the notion of kernel diffusion density function . Towards this end , we will introduce a kernel diffusion density function , which takes account of local adaptability and is well-suited for clustering purpose . We provide details on how to derive this density function from a diffusion process induced by bivariate kernels . We also provide a surrogate density function that is computationally more efficient . 4.1 DIFFUSION PROCESS AND KERNEL DIFFUSION DENSITY . Considering a bivariate kernel function k : D ×D → R+ , such that : • k ( x , y ) is positive semi-definite , i.e. , k ( x , y ) ≥ 0 . • k ( x , y ) is Fn-integrable with respect to both x and y . We define d ( x ) = n ∫ D k ( x , y ) dFn ( y ) as a local measure of the volume at x and define p ( x , y ) = k ( x , y ) d ( x ) . ( 3 ) It is easy to see that p ( x , y ) satisfies the conservation property n ∫ D p ( x , y ) dFn ( y ) = 1 . As a result , p ( x , y ) can be viewed as a probability for a random walk on the dataset from point x to point y , which induces a Markov chain on D with n× n transition matrix P = [ p ( x , y ) ] . This technique is standard in various applications , known as the normalized graph Laplacian construction . For example , we can view D as a graph , L = I −P as the normalized graph Laplacian , and d ( x ) as a normalization factor . For t ≥ 0 , the probability of transiting from x to y in t time steps is given by P t , the t-th power of P . Running the Markov chain forward in time , we observe the dataset at different scales , which is the diffusion process Xt on D. Let ρ ( x , t ) : D × R+ → R+ be the associated probability density , which is governed by the following second-order differential equation with initial conditions : { ∂ ∂tρ ( x , t ) = −Lρ ( x , t ) , ρ ( x , 0 ) = ϕ0 ( x ) , ( 4 ) where ϕ0 ( x ) is a probability density at time t = 0 . In practice we can use any valid choice of ϕ0 ( x ) , e.g . the uniform density . To give an explicit example of the diffusion process , consider a sub-class of k , i.e. , isotropic kernels , where k ( x , y ) = K ( ||x− y||2/h ) for some function K : R → R+ . Here we can dual interpret h as a scale parameter to infer local information and as a time step h = ∆t at which the random walk jumps . Then we can define the forward Chapman-Kolmogorov operator TF as TF ( x ) = n ∫ D p ( x , y ) ϕ0 ( y ) dFn ( y ) . Note that TF is the data distribution at time t = h , thus can be viewed as continuous analogues of the left multiplication by the transition matrix P . Letting h → 0 , the random walk converges to a continuous diffusion process with probability density evolves continuously in t. In this case , we can explicitly write the second-order differential equation in ( 4 ) as : ∂ ∂t ρ ( x , t ) = lim h→0 ρ̂ ( x , t+ h ) − ρ ( x , t ) h = lim h→0 TF − I h ρ ( x , t ) , ( 5 ) where Lh = limh→0 ( TF − I ) /h is the conventional infinitesimal generator of the process . Now we are ready to introduce our kernel diffusion density function . Definition 1 . ( Kernel diffusion density function ) Suppose the Markov chain induced by P is ergodic , we define the kernel diffusion density function as the limiting probability density of the diffusion process Xt , i.e. , ρKD ( x ) = lim t→∞ ρ ( x , t ) . ( 6 ) Intuitively , on the one hand , with increased values of t we expect the diffusion process Xt gradually reveals the geometric structure ( such as high-density regions ) of the data distribution F . To see this , note that the transition probability P reflects connectivity between data . We can interpret a cluster as an underlying geometric structure in which the probability of staying in this region is high during a transition . In the diffusion process , the probability of following a path along a structure usually increases with t , as the involved data points are dense and highly connected . Therefore , the path consists of short and high probability jumps . Whilst paths that do not follow any structure consists of long and low probability jumps , which lowers their overall probability . As a result , geometry structures of F is magnified during the diffusion . On the other hand , by talking certain sophisticated forms of k ( x , y ) that take into account of local adaptivity , we also slow down the diffusion to avoid trivial geometry structures such as one big cluster for all the data points . In this way , we can eventually identify the correct geometry structure at the right scale .

Summary: The authors propose a new density function for density clustering models such as DBSCAN and Density Peaks Clustering (DPC). Given a kernel $K$ and the corresponding normalized random walk transition matrix $P = diag(K\mathbf{1})^{-1}K$, the authors propose to use the density function which corresponds to the stationary distribution attained in the limit. Since this function is expensive to compute, the authors propose a surrogate density function which is easier to compute.  The main experiments compare various density functions for DPC on UCI classification datasets and the DPC on density functions to face detection methods on two face identification datasets.