Hierarchical Clustering: $O(1)$-Approximation for Well-Clustered Graphs

1 Introduction . Hierarchical clustering ( HC ) studies a recursive partition of a data set into clusters of successively smaller size , via an effective binary tree representation . As a basic technique , hierarchical clustering has been employed as a standard package in data analysis , and has comprehensive applications in practice . While traditionally HC trees are constructed through bottom-up ( agglomerative ) heuristics , which lacked a clearly-defined objective function , Dasgupta [ 12 ] has recently introduced a simple objective function to measure the quality of a particular hierarchical clustering and his work has inspired a number of research on this topic [ 3 , 6 , 7 , 8 , 10 , 11 , 20 , 24 ] . Consequently , there has been a significant interest in studying efficient HC algorithms that not only work in practice , but also have proven theoretical guarantees with respect to Dasgupta ’ s cost function . Our contribution . In this work , we present two new approximation algorithms for constructing HC trees that can be rigorously analysed with respect to Dasgupta ’ s cost function . For our first result , we construct an HC tree of an input graph G entirely based on the degree sequence of V ( G ) , and we show that the approximation guarantee of our constructed tree is with respect to the conductance of G , which will be defined formally in Section 2 . The striking fact of this result is that , for any n-vertex graph G with m edges and conductance Ω ( 1 ) ( a.k.a . expander graph ) , an O ( 1 ) -approximate HC tree of G can be very easily constructed in O ( m+ n log n ) time , although obtaining such result for general graphs is impossible under the Small Set Expansion Hypothesis ( SSEH ) [ 6 ] . Our theorem is in line with a sequence of results for problems that are naturally linked to the Unique Games and Small Set Expansion problems : it has been shown that such problems are much easier to solve once ∗The full version of the paper is available at https : //arxiv.org/abs/2112.09055 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the input instance exhibits the high conductance property [ 4 , 5 , 16 , 18 ] . However , to the best of our knowledge , our result is the first of this type for hierarchical clustering . While our first result presents an interesting theoretical fact , we further study whether we can extend this O ( 1 ) -approximate construction to a much wider family of graphs occurring in practice . Specifically , we look at well-clustered graphs , i.e. , the graphs in which vertices within each cluster are better connected than vertices between different clusters and the total number of clusters is constant . This includes a wide range of graphs occurring in practice with a clear cluster-structure , and have been extensively studied over the past two decades ( e.g. , [ 13 , 15 , 23 , 26 ] ) . As our second and main result , we present a polynomial-time O ( 1 ) -approximation algorithm that constructs an HC tree for a well-clustered graph . Given that the class of well-clustered graphs includes graphs with clusters of different sizes and asymmetrical internal structure , our result significantly improves the previous state-of-the-art [ 10 ] , which only holds for graphs generated from stochastic models . At the technical level , the design of our algorithm is based on the graph decomposition algorithm presented in [ 13 ] , which is designed to find a good partition of a well clustered graph . However , our analysis suggests that , in order to obtain an O ( 1 ) -approximation algorithm , directly applying their decomposition isn ’ t sufficient for our purpose . To overcome this bottleneck , we refine the output decomposition via a pruning technique , and carefully merge the refined parts to construct our final HC tree . In our point of view , our presented stronger graph decomposition procedure might have applications in other settings as well . To demonstrate the significance of our work , we compare our algorithm against the previous state-of-the-art with similar approximation guarantee [ 10 ] and well-known linkage heuristics on both synthetic and real-world data sets . Although our algorithm ’ s performance is marginally better than [ 10 ] for the graphs generated from the stochastic block models ( SBM ) , the cost of our algorithm ’ s output is up to 50 % lower than the one from [ 10 ] when the clusters of the input graph have different sizes and some cliques are embedded into a cluster . Related work . Our work fits in a line of research initiated by Dasgupta [ 12 ] , who introduced a cost function to measure the quality of an HC tree . Dasgupta proved that a recursive application of the algorithm for the Sparest Cut problem achieves an O ( log3/2 n ) -approximation , which has been subsequently improved to O ( log n ) and O ( √ log n ) by [ 24 ] and [ 6 , 11 ] . It is known to be NP-hard to find an optimal HC tree [ 12 ] , and SSEH-hard to achieve an O ( 1 ) -approximation for a general input instance [ 6 ] . Cohen-Addad et al . [ 10 ] studied a hierarchical extension of the SBM and showed that a certain SVD projection algorithm together with several linkage heuristics achieves a ( 1 + o ( 1 ) ) -approximation with high probability . We emphasise that our notion of well-clustered graphs generalises the SBM variant studied in [ 10 ] and does not assume the rigid hierarchical structure of the clusters . For another line of related work , Moseley and Wang [ 20 ] studied the dual objective function and proved that Average Linkage achieves a ( 1/3 ) -approximation for the new objective function . Notice that , although this has received significant attention very recently [ 3 , 7 , 8 , 9 , 25 ] , achieving an O ( 1 ) -approximation is tractable under this alternative objective . This suggests the fundamental difference on the hardness of the problem under different objective functions , and is our reason to entirely focus on the Dasgupta ’ s cost function in this work . 2 Preliminaries . Throughout the paper , we always assume that G = ( V , E , w ) is an undirected graph with n vertices , m edges and weight function w : V × V → R≥0 . For any edge e = { u , v } ∈ E we write we or wuv to indicate the similarity weight between u and v. For a vertex u ∈ V , we denote its degree by du , ∑ v∈V wuv and we assume that wmax/wmin = O ( poly ( n ) ) , where wmin ( wmax ) is the minimum ( maximum ) edge weight . We will use dmin , dmax , davg for the minimum , maximum and average degrees in G , where davg , ∑ u∈V du/n . For a nonempty subset S ⊂ V , we define G [ S ] to be the induced subgraph on S and we denote by G { S } the subgraph G [ S ] , where self loops are added to vertices v ∈ S such that their degrees in G and G { S } are the same . For any two subsets S , T ⊂ V , we define the cut value w ( S , T ) , ∑ e∈E ( S , T ) we , where E ( S , T ) is the set of edges between S and T . For any set S ⊆ V , the volume of S is vol ( S ) , ∑ u∈S du and we write vol ( G ) when referring to vol ( V ( G ) ) . We further define the conductance of S by ΦG ( S ) , w ( S , V \ S ) vol ( S ) , and ΦG , min S⊂V vol ( S ) ≤vol ( G ) /2 ΦG ( S ) . We call G an expander graph if ΦG = Ω ( 1 ) . For a graph G , let D ∈ Rn×n be the diagonal matrix defined by Duu = du for all u ∈ V . We denote by A ∈ Rn×n the adjacency matrix of G , where Auv = wuv for all u , v ∈ V . The normalised Laplacian matrix of G is defined as L , I−D−1/2AD−1/2 , where I is the identity n× n matrix . We will denote the eigenvalues of L by 0 = λ1 ≤ · · · ≤ λn ≤ 2 . Hierarchical clustering . A hierarchical clustering ( HC ) tree of a given graph G is a binary tree T with n leaf nodes such that each leaf corresponds to exactly one vertex v ∈ V ( G ) . Let T be an HC tree of some graph G = ( V , E , w ) and let N ∈ T be an arbitrary internal node2 of T . We denote T [ N ] to be the subtree of T rooted at N , leaves ( T [ N ] ) to be the set of leaf nodes of T [ N ] and parentT ( N ) to be the parent of node N in T . In addition , each internal node N ∈ T induces a unique vertex set C ⊆ V formed by the vertices corresponding to leaves ( T [ N ] ) . We will abuse the notation and write N ∈ T for both the internal node of T and its corresponding set of vertices in V . To measure the quality of an HC tree T with similarity weights , Dasgupta [ 12 ] introduced the following cost function : costG ( T ) , ∑ e= { u , v } ∈E we · |leaves ( T [ u ∨ v ] ) | , where u ∨ v is the lowest common ancestor of u and v in T . Trees that achieve a better hierarchical clustering have a lower cost , based on the following consideration : for any pair of vertices u , v ∈ V that corresponds to an edge of high weight wuv ( i.e. , u and v are highly similar ) a “ good ” HC tree would separate u and v lower in the tree , thus reflected in a small size |leaves ( T [ u ∨ v ] ) | . We denote by OPTG the minimum cost of any HC tree of G , i.e. , OPTG = minT costG ( T ) , and use T ∗ to refer to an optimal tree achieving the minimum . We say that an HC tree T is an α-approximate tree if costG ( T ) ≤ α · OPTG . All omitted proofs are deferred to the full version of this paper . 3 Hierarchical clustering for graphs of high conductance . In this section we study hierarchical clustering for graphs with high conductance and prove that , for any input graph G with ΦG = Ω ( 1 ) , an O ( 1 ) -approximate HC tree of G can be simply constructed based on the degree sequence of G. As a starting point , we show that costG ( T ) for any T can be upper bounded with respect to ΦG and the degree distribution of V ( G ) . Lemma 3.1 . It holds for any HC tree T of graph G that costG ( T ) ≤ 94ΦG · min { davg dmin , dmaxdavg } · OPTG . While Lemma 3.1 holds for any graph G , it implies some interesting facts for expander graphs : first of all , when G = ( V , E , w ) satisfies dmax/dmin = O ( 1 ) and ΦG = Ω ( 1 ) , Lemma 3.1 shows that any tree T is an O ( 1 ) approximate tree . In addition , although ΦG plays a crucial role in analysing costG ( T ) as for many other graph problems , Lemma 3.1 indicates that the degree distribution of G might also have significant impact . One could naturally ask the extend to which the degree distribution of V ( G ) would influence the construction of optimal trees . To study this question , we look at the following graph G where all edges have unit weight : ( i ) let G1 = ( V , E1 ) be a constant-degree expander graph of n vertices with ΦG1 = Ω ( 1 ) , e.g. , the ones 2We will always use ( internal ) nodes for the nodes of T and vertices for the elements of the vertex set V . presented in [ 14 ] ; ( ii ) we choose bn2/3c vertices from V to form S , and let K = ( S , S × S ) be a complete graph defined on S ; ( iii ) partition the vertices of V \S into bn2/3c groups of roughly the same size , associate each group to a unique vertex in S and let E2 be the set of edges formed by connecting every vertex in S with all the vertices in its associated group . ( iv ) let G , ( V , E1 ∪ ( S×S ) ∪E2 ) be the union of G1 , K and the edges in E2 , see Figure 1 ( a ) for illustration . By construction , we know that ΦG = Ω ( 1 ) , and the degrees of G satisfy dmax = Θ ( n2/3 ) , dmin = Θ ( 1 ) , and davg = Θ ( n1/3 ) . Therefore , the ratio between costG ( T ) for any HC tree T and OPTG could be as high as Θ ( n1/3 ) . On the other side , it ’ s not difficult to show that the tree T ∗ illustrated in Figure 1 ( b ) , which first separates the set S of high-degree vertices from V \ S at the top of the tree , actually O ( 1 ) -approximates OPTG 3 . This example suggests that grouping vertices of similar degrees first could potentially help reduce costG ( T ) for our constructed T . This motivates us to design the following Algorithm 1 to construct an HC tree . We highlight that the output of Algorithm 1 is uniquely determined by the ordering of the vertices of G according to their degrees , which can be computed in O ( n · log n ) time . Algorithm 1 : HCwithDegrees ( G { V } ) Input : G = ( V , E , w ) with the ordered vertices such that dv1 ≥ . . . ≥ dv|V | ; Output : An HC tree Tdeg ( G ) ; 1 if |V | = 1 then 2 return the single vertex in V as the tree ; 3 else 4 imax : = blog2 ( |V | − 1 ) c ; r : = 2imax ; A : = { v1 , . . . , vr } ; B : = V \A ; 5 Let T1 = HCwithDegrees ( G { A } ) ; T2 = HCwithDegrees ( G { B } ) ; 6 return Tdeg with T1 and T2 as the two children . Theorem 1 . Given any graph G = ( V , E , w ) with conductance ΦG as input , Algorithm 1 runs in O ( m + n log n ) time , and returns an HC tree Tdeg of G that satisfies costG ( Tdeg ) = O ( 1/Φ4G ) · OPTG . Theorem 1 guarantees that , when the input G satisfies ΦG = Ω ( 1 ) , the output Tdeg of Algorithm 1 achieves an O ( 1 ) -approximation . Moreover , as the high-conductance property can be determined in nearly-linear time by computing λ2 ( LG ) and applying the Cheeger inequality , Algorithm 1 presents a very simple construction of an O ( 1 ) -approximate HC tree once the input G is known to have high conductance . Proof sketch of Theorem 1 . The key notion employed in our proof is that of the dense branch which can be informally described as follows : we perform a traversal in T starting at the root nodeA0 and we sequentially travel to the child of higher volume . The process stops whenever we reach a node Ak , for some k ∈ Z≥0 , such that vol ( Ak ) > vol ( G ) /2 and both its children have volume at most vol ( G ) /2 . The sequence of visited nodes by this process is the dense branch of T . Formally , for any HC tree T of G , the dense branch is the path ( A0 , A1 , . . . , Ak ) in T for some k ∈ Z≥0 , such that the following conditions hold : ( 1 ) A0 is the root of T ; ( 2 ) Ak is the node such that vol ( Ak ) > vol ( G ) /2 and both children of Ak have volume at most vol ( G ) /2 . It is important to note that the dense branch of a tree T is unique and it contains all nodes Ai such that vol ( Ai ) > vol ( G ) /2 and only those nodes . At a very high level , the proof of Theorem 1 starts with any optimal HC tree T0 of G and constructs trees T1 , T2 , T3 and T4 such that ( i ) costG ( Ti ) can be upper bounded with respect to costG ( Ti−1 ) for every 1 ≤ i ≤ 4 , ( ii ) the final constructed T4 is exactly the tree Tdeg ( G ) , the output of Algorithm 1 . Combining these two facts allows us to upper bound costG ( Tdeg ) with respect to OPTG . Step 1 : Regularisation . Let ( A0 , . . . , Ak0 ) be the dense branch of any optimal T0 , and let Bi be the sibling of Ai , for all 1 ≤ i ≤ k0 . Let imin = blog2 ( |Ak0 | ) c and imax = blog2 ( |V | − 1 ) c. For simplicity , we assume that the dense branch has size at least 2 , that |A1| ≥ 2imax and that |Ak0 | = 2imin , which ensures that imin ≤ imax . These assumptions allow us to better present the proof techniques by ignoring some corner cases . In the full version of our paper we deal with the 3To see this , notice that cost ( T ∗ ) ≤ cost ( T ∗ [ S ] ) + n · vol ( G1 ) + n2 = Θ ( n2 ) , as the complete subgraph G [ S ] induces a cost of Θ ( ( n2/3 ) 3 ) [ 12 ] . The subgraph G [ S ] also implies that OPTG = Ω ( n2 ) . general case when some of these assumptions do not hold . The goal of this step is to adjust the dense branch of T0 , such that the resulting tree T1 satisfies the following condition : for all i ∈ [ imin , imax ] , there is a node of size exactly 2i along the dense branch of T1 . We achieve this by applying a sequence of operations , each of which creating a new node of exact size 2i for some suitable i . Concretely , let i ∈ [ imin , imax ] be the largest integer such that there is no node of size 2i on the dense branch of T0 . As |Ak0 | = 2imin , there is some node Aj on the dense branch such that |Aj | > 2i and |Aj+1| < 2i . We adjust the branch at Aj as follows : ( i ) we consider a partition of Bj+1 = B1j+1 ∪B2j+1 such that |Aj+1|+ |B2j+1| = 2i ; ( ii ) we replace the node Aj by some newly created node A1j that has children B1j+1 and a new node A 2 j . ( iii ) the two children of A 2 j will be Aj+1 and B 2 j+1 . This adjustment is illustrated in Figure 2 ( a ) , and we repeat this process until no such i exists any more . We call the resulting tree T1 , and the following lemma gives an upper bound of costG ( T1 ) . Lemma 3.2 . Our constructed tree T1 satisfies costG ( T1 ) ≤ ( 1 + 4/ΦG ) · costG ( T0 ) . Step 2 : Compression . With potential relabelling of the nodes , let ( A0 , . . . , Ak1 ) be the dense branch of T1 for some k1 ∈ Z+ satisfying |Ak1 | = 2imin , and Bi be the sibling of Ai . The objective of this step is to ensure that , by a sequence of adjustments , all the nodes along the dense branch are of size equal to some power of 2 . Here , we describe how one such adjustment is performed , and refer the reader to Figure 2 ( b ) for illustration . Let i ∈ [ imin , imax ] be some index such that |Aj1 | = 2i and |Aj2 | = 2i−1 for some j1 < j2 . We compress the dense branch by removing all nodes between Aj1 and Aj2 as follows : The two children of Aj1 will be Aj2 and some new node Cj1 , which has children Cj1+1 and Bj1+1 ; The two children of Cj1+1 will be some new node Cj1+2 and Bj1+2 , etc . The last node Cj2−2 has children Bj2−1 and Bj2 . In addition , we perform one more such adjustment to remove all nodes Aj of size 2imax < |Aj | < n and ensure all nodes ( except potentially A0 ) on the dense branch have size 2i for some i ∈ [ imin , imax ] . We call the resulting tree T2 , and the following lemma bounds the cost of T2 . Lemma 3.3 . Our constructed tree T2 satisfies that costG ( T2 ) ≤ 2 · costG ( T1 ) . Step 3 : Matching . Let ( A0 , . . . , Ak2 ) be the dense branch of T2 , for some k2 ∈ Z+ , and Bi be the sibling of Ai . In this step we transform T2 into T3 , such that T3 is isomorphic to Tdeg ( G ) , which ensures that T3 and Tdeg ( G ) have the same structure . To achieve this , for every 1 ≤ i ≤ k2 we simply replace each T2 [ Bi ] with Tdeg ( G { Bi } ) . We further replace T2 [ Ak2 ] with Tdeg ( G { Ak2 } ) . We call the resulting tree T3 , and bound its cost by the following lemma : Lemma 3.4 . Our constructed tree T3 satisfies that costG ( T3 ) ≤ ( 1 + 4/ΦG ) costG ( T2 ) . Step 4 : Sorting . We assume that ( A0 , . . . , Ak3 ) is the dense branch of T3 for some k3 ∈ Z+ , and we extend the dense branch to ( A0 , . . . , Aimax ) with the same property that for every i ∈ [ k3 , imax ] Ai is the child of Ai−1 with the higher volume , and let S , { B1 , . . . , Bimax , Aimax } . Recall that in Tdeg ( G ) the first r = 2imax vertices of the highest degrees , i.e. , { v1 , . . . , vr } belong to A1 , of which the first 2imax−1 vertices belong to A2 , and so on ; however , this might not be the case for T3 . Hence , in the final step , we prove that T3 can be transformed into Tdeg ( G ) without a significant increase of the total cost . In this step we will swap vertices between the internal nodes B1 , B2 , . . . , Bimax , Aimax in such a way that Aimax will consist of v1 and v2 , Bimax will consist of v3 and v4 , Bimax−1 will consist of v5 up to v8 , etc . We call vertex u misplaced if the position of u is T3 is different from the one in Tdeg ( G ) . To transform T3 into Tdeg ( G ) we perform a sequence of operations , each of which consisting in a chain of swaps focusing on the vertex of the highest degree that is currently misplaced . For the sake of argument , we assume that v1 6∈ Aimax is misplaced , and we apply the following operation to move v1 to Aimax : ( i ) let v1 ∈ Bi0 for some i0 ≥ 1 , and let y be the vertex of the lowest degree among the vertices in S \ { B1 , . . . , Bi0 } . Say y ∈ Bi1 for some i1 > i0 , and swap v1 with y while keeping the structure of the tree unchanged ; ( ii ) repeat the swap operation above until v1 reaches its correct place Aimax . Once the above process is complete and v1 reaches Aimax , we apply a similar chain of swaps to v2 to ensure v2 also reaches Aimax . Then , we sequentially apply the process to v3 and v4 to ensure they reach Bimax , and continue this process until there are no more misplaced vertices . We call the resulting tree T ′3 , and notice that every node X ∈ S in T ′ 3 contains the correct set of vertices . However , the positions of these vertices in T ′3 [ X ] might be different from the ones in Tdeg ( G { X } ) . To overcome this issue , we repeat Step 3 again to the tree T ′ 3 , and this will introduce another factor of ( 1+4/ΦG ) to the total cost of the constructed tree . Importantly , the final constructed tree after this step is exactly the tree Tdeg ( G ) and we bound its cost by the following lemma : Lemma 3.5 . It holds for Tdeg that cost ( Tdeg ) ≤ ( 1 + 24/ΦG ) ( 1 + 4/ΦG ) · cost ( T3 ) . Finally , by combining Lemmas 3.2 , 3.3 , 3.4 and 3.5 we prove the approximation guarantee of Tdeg in Theorem 1 . The runtime of Algorithm 1 follows by a simple application of the master theorem .

Summary: This paper studies the cost function introduced by Dasgupta [13] and present polynomial-time approximation algorithms for well-clustered graphs. The well-clusterable assumptions mostly include an underlying structure of a fixed number of clusters, such that the subgraph induced by each cluster has high conductance, yet the cut between the clusters are sparse. The algorithm design bypasses the popular sparsest-cut subroutine, which was first proposed in Dasgupta [13] and often used by the community when studying this particular objective. The result is also more general than the state-of-the-art constant approximation results for this objective. The proposed algorithm also turns out to have reasonable performance empirically.