Learning Graph Representations by Dendrograms

Hierarchical clustering is a common approach to analysing the multi-scale structure of graphs observed in practice . We propose a novel metric for assessing the quality of a hierarchical clustering . This metric reflects the ability to reconstruct the graph from the dendrogram encoding the hierarchy . The best representation of the graph for this metric in turn yields a novel hierarchical clustering algorithm . Experiments on both real and synthetic data illustrate the efficiency of the approach . 1 Introduction . Many datasets have a graph structure . Examples include infrastructure networks , communication networks , social networks , databases and co-occurence networks , to quote a few . These graphs often exhibit a complex , multi-scale structure where each node belong to many groups of nodes , so-called clusters , of different sizes ( Caldarelli and Vespignani ( 2007 ) ) . Hierarchical graph clustering is a common technique for the analysis of the multi-scale structure of large graphs . Instead of looking for a single partition of the set of nodes , as in usual clustering techniques , the graph is represented by a hierarchical structure known as a dendrogram , which can then be used to find relevant clusterings at different resolutions , by suitable cuts of this dendrogram . While many hierarchical graph clustering algorithms have recently been proposed ( see for instance Newman ( 2004 ) ; Pons and Latapy ( 2005 ) ; Sales-Pardo et al . ( 2007 ) ; Clauset et al . ( 2008 ) ; Lancichinetti et al . ( 2009 ) ; Ronhovde and Nussinov ( 2009 ) ; Huang et al . ( 2010 ) ; Chang et al . ( 2011 ) ; Tremblay and Borgnat ( 2014 ) ; Bonald et al . ( 2018 ) ) , it proves very difficult to evaluate their performance due to the absence of public datasets of graphs with ground-truth hierarchy . A natural approach is then to define some quality metric based on the graph itself , just like modularity is a popular metric for assessing the quality of graph clustering ( Newman and Girvan ( 2004 ) ) . A cost function has been proposed by Dasgupta ( 2016 ) for hierachical graph clustering and has been further analysed and extended by Cohen-Addad et al . ( 2018 ) ; it can be viewed as the expected size of the smallest cluster induced by the hierarchy and containing two nodes sampled at random from the edges of the graph . In this paper , we propose a quality metric based on the ability to reconstruct the graph from the hierarchy . It is equal to the relative entropy of the probability distribution on node sets induced by the hierarchy compared to that induced by independent node sampling . Finding the best graph representation for this metric yields a novel hierarchical clustering algorithm , which can in turn be interpreted in terms of modularity . In the next section , we introduce sampling distributions of nodes and node sets induced by the graph , which play a key role in our approach . We then formalize the problem of graph representation by a dendrogram . Our quality metric follows from the characterization of the optimal solution in terms of graph reconstruction . The corresponding hierarchical graph clustering algorithm is then presented and interpreted in terms of modularity . The results are illustrated by some experiments on both real and synthetic data . 2 Sampling distribution . Consider a weighted , undirected , connected graph G = ( V , E ) of n nodes , without self-loops . Let w ( u , v ) be equal to the weight of edge { u , v } , if any , and to 0 otherwise . We refer to the weight of node u as : w ( u ) = ∑ v∈V w ( u , v ) . We denote by w the total weight of nodes : w = ∑ u∈V w ( u ) = ∑ u , v∈V w ( u , v ) . Observe that the weight of each edge is counted twice in this sum . Similarly , for any sets A , B ⊂ V , let w ( A , B ) = ∑ u∈A , v∈B w ( u , v ) , and w ( A ) = ∑ u∈A w ( u ) . Node sampling . The weights induce a probability distribution on node pairs : ∀u , v ∈ V , p ( u , v ) = w ( u , v ) w , with marginal distribution : ∀u ∈ V , p ( u ) = ∑ v∈V p ( u , v ) = w ( u ) w . This joint probability distribution is symmetric in the sense that p ( u , v ) = p ( v , u ) for all u , v ∈ V . This is the relative frequency of moves from node u to node v by a random walk in the graph , with transition probability : ∀u , v ∈ V , p ( v|u ) = p ( u , v ) p ( u ) = w ( u , v ) w ( u ) . Observe that p ( u , u ) = 0 for all u ∈ V due to the absence of self-loops in the graph . Node set sampling . For any partition P of V , the weights induce a probability distribution on P : ∀A , B ∈ P , p ( A , B ) = w ( A , B ) w , with marginal distribution : ∀A ∈ P , p ( A ) = ∑ B∈P p ( A , B ) = w ( A ) w . Again , this joint probability distribution is symmetric in the sense that p ( A , B ) = p ( B , A ) for all A , B ∈ P. This is the relative frequency of moves from A to B by the random walk . 3 Representation by a dendrogram . Assume the graph G is represented by a dendrogram , that is a rooted binary tree T whose leaves are the nodes of the graph , V . We denote by int ( T ) the set of internal nodes of the tree T ( all nodes except leaves ) . For each i ∈ int ( T ) , a positive number d ( i ) is assigned to node i , corresponding to its height in the dendrogram . We assume that the dendrogram is regular in the sense that d ( i ) ≥ d ( j ) if i is an ancestor of j in the tree . For each i ∈ int ( T ) , there are two subtrees attached to node i , with sets of leaves A and B ; these sets uniquely identify the internal node i so that we can write i = ( A , B ) = ( B , A ) and d ( i ) = d ( A , B ) = d ( B , A ) . Ultrametric . The dendrogram defines a metric on V : for each u , v ∈ V , we define d ( u , v ) = 0 if u = v and d ( u , v ) = d ( i ) otherwise , where i ∈ int ( T ) is the closest common ancestor of u and v in the corresponding tree . This is an ultrametric in the sense that : ∀u , v , x ∈ V , d ( u , v ) ≤ max ( d ( u , x ) , d ( v , x ) ) . Conversely , any ultrametric defines a dendrogram , which can be built from bottom to top by successive merges of the closest nodes , where the distance of the node resulting from the merge of two nodes u and v , which we denote by u ∪ v , to any other node x is defined by : d ( u ∪ v , x ) = max ( d ( u , x ) , d ( v , x ) ) . Nested clustering . The tree T induces a clustering C of the set of nodes , with A ∈ C if and only if A is the set of leaves of a subtree of T . This is a nested clustering in the sense that : ∀A , B ∈ C , A ⊂ B or A ∩B = ∅ . Although C is not a partition of V , we get a probability distribution on cluster pairs by restricting the sampling to cluster pairs A , B such that ( A , B ) ∈ int ( T ) : ∑ A , B : ( A , B ) ∈int ( T ) p ( A , B ) = ∑ i∈int ( T ) ∑ u , v : i=u∧v p ( u , v ) = ∑ u , v∈V p ( u , v ) = 1 , where u ∧ v denotes the closest common ancestor of u and v in the tree T . Graph representation . To assess the quality of the dendrogram , we address the issue of the reconstruction of the graph from the dendrogram . In general , some information about the relative node weights may be known in addition to the dendrogram . Thus we introduce some probability distribution π on V representing this prior information : the distribution π is uniform in the absence of such information and is equal to the sampling distribution p for a perfect knowledge of the relative node weights . Now we look for the best representation of G by a dendrogram d in the sense of the reconstruction of G from d , π , which can be viewed as the autoencoding scheme : G 7−→ d , π 7−→ Ĝ , where Ĝ is the reconstruction of G. Since d ( u , v ) can be interpreted as a distance between u and v , its inverse corresponds to a similarity . Accounting for the prior information about the relative weights of u and v , we define the weight ŵ ( u , v ) between nodes u , v ∈ V in the graph Ĝ as : ŵ ( u , v ) = π ( u ) π ( v ) d ( u , v ) 1 { u6=v } . ( 1 ) Denoting by ŵ the total weight , ŵ = ∑ u , v∈V ŵ ( u , v ) , ( 2 ) we get the following node pair sampling distribution associated with Ĝ : p̂ ( u , v ) = ŵ ( u , v ) ŵ . ( 3 ) The distance between graphs G and Ĝ can then be assessed through the Kullback-Leibler divergence between the respective sampling distributions , D ( p||p̂ ) = ∑ u , v∈V p ( u , v ) log p ( u , v ) p̂ ( u , v ) , which can be written as the relative entropy D ( p||p̂ ) = H ( p , p̂ ) −H ( p ) , where H ( p , p̂ ) = − ∑ u , v∈V p ( u , v ) log p̂ ( u , v ) and H ( p ) = − ∑ u , v∈V p ( u , v ) log p ( u , v ) are the cross-entropy between the probability distributions p and p̂ and the entropy of the probability distribution p , respectively . Observe that H ( p , p̂ ) ≥ H ( p ) , with equality if and only if p̂ = p. Optimization problem . Minimizing the Kullback-Leibler divergence D ( p||p̂ ) in p̂ is equivalent to minimizing the cross-entropy H ( p , p̂ ) in p̂ . In view of ( 1 ) and ( 3 ) , we have H ( p , p̂ ) = H ( p , π ) + ∑ u , v∈V p ( u , v ) log d ( u , v ) + log ( ŵ ) , where π here denotes the bivariate probability distribution π ( u , v ) = π ( u ) π ( v ) . Thus the problem reduces to minimizing the cost function : J ( d ) = ∑ u , v∈V p ( u , v ) log d ( u , v ) + log ∑ u , v∈V π ( u ) π ( v ) d ( u , v ) , over all ultrametrics d defined on V . Observe that J ( αd ) = J ( d ) for any α > 0 , so that the best ultrametric is defined up to a multiplicative constant . Using the fact that d ( u , v ) = d ( i ) , where i = u ∧ v is the closest common ancestor of u and v in the tree , we get : J ( d ) = ∑ A , B : ( A , B ) ∈int ( T ) p ( A , B ) log d ( A , B ) + log ∑ A , B : ( A , B ) ∈int ( T ) π ( A ) π ( B ) d ( A , B ) , ( 4 ) with π ( A ) = ∑ u∈A π ( u ) . The problem of the best representation of G by a dendrogram d now reduces to the optimization problem : arg min d J ( d ) , ( 5 ) over all ultrametrics d on V .

Summary: This paper suggests a new metric for assessing the quality of hierarchical clustering. Dasgupta recently suggested the first such metric with interesting properties. This has encouraged a number of recent works about designing algorithms that work well for the metric and other similar metrics. This paper suggests a new metric for evaluating hierarchical clustering of given graph data.