End-to-End Learning of Probabilistic Hierarchies on Graphs

We propose a novel probabilistic model over hierarchies on graphs obtained by continuous relaxation of tree-based hierarchies . We draw connections to Markov chain theory , enabling us to perform hierarchical clustering by efficient end-to-end optimization of quality metrics such as Dasgupta cost or Tree-Sampling Divergence ( TSD ) . We show that our model learns rich , high-quality hierarchies present in 12 real world graphs , including a large graph with 2.3M nodes . Our model consistently outperforms recent as well as strong traditional baselines such as average linkage . We demonstrate that our model also obtains competitive results on link prediction despite not being trained on this task , highlighting the quality of the hierarchies discovered by our model . 1 INTRODUCTION . Clustering is a fundamental problem in unsupervised learning , both in theory and practice . In contrast to traditional ‘ flat ’ clustering , hierarchical clustering brings several advantages . It enables us to naturally analyze ( Zhao & Karypis , 2002 ) and visualize ( Himberg et al. , 2004 ) a given dataset on different scales . Further , different downstream tasks may require different granularities . Hierarchical clustering lets us pick the desired granularity after learning . It can also be used for personalized recommendation ( Zhang et al. , 2014 ) or to solve record linkage tasks ( Wick et al. , 2012 ) . In an influential paper , Eisen et al . ( 1998 ) perform hierarchical clustering on gene expression data and argue that it could be used to discover the yet-unknown meaning of certain genes , which could lead to advances in medicine . Besides traditional vector data , real-world graphs are often scale-free and hierarchically organized ( Ravasz & Barabási , 2003 ; Barabási & Pósfai , 2016 ) . This makes them interesting candidates for uncovering the underlying hierarchy via hierarchical clustering . Thus , the hierarchical clustering method we introduce is particularly useful for analyzing scale-free real-world networks such as Web graphs , citation networks , flight networks , or biological networks . Related work . We can group hierarchical clustering algorithms into discrete and continuous optimization methods . Discrete optimization methods can themselves be decomposed into agglomerative approaches and divisive approaches . Agglomerative methods – like the famous linkage algorithms ( Gower & Ross , 1969 ; Jardine & Sibson , 1968 ; Bonald et al. , 2018 ) – follow a bottom-up approach and iteratively aggregate the two most similar clusters w.r.t . a given similarity measure . Divisive methods follow a top-down approach and recursively split larger clusters into smaller ones using , e.g. , k-means ( Steinbach et al. , 2000 ) or spectral clustering ( Charikar & Chatziafratis , 2016 ) . While Moseley & Wang ( 2017 ) ; Kamvar et al . ( 2002 ) have shown connections with explicit objective functions , those methods are mainly based on heuristics to avoid exhaustive combinatorial optimization . On the other hand , continuous optimization methods circumvent this issue by ( i ) relaxing the discrete tree structure , ( ii ) defining an explicit quality metric for hierarchies , and ( iii ) optimizing it using gradient-based optimizers . Chierchia & Perret ( 2019 ) ; Monath et al . ( 2017 ; 2019 ) ; Chami et al . ( 2020 ) focus on minimizing the Dasgupta cost ( Dasgupta , 2016 ) ( see Eq . ( 1 ) ) to fit an ultrametric , hyperbolic embeddings , or a probabilistic model over cluster assignments . Monath et al . ( 2017 ) learns a probabilistic assignment of the samples to leaves of a tree with a fixed binary structure . The method of Monath et al . ( 2019 ) requires approximation for the computation of lowest common ancestors ( LCA ) probabilities and regularization losses . Beyond the Dasgupta cost , Charpentier & Bonald ( 2019 ) propose the Tree Sampling Divergence ( TSD ) , a quality metric for hierarchies on graphs . Both metrics have the benefit to be internal metrics , i.e. , they do not require ground truth information to evaluate the quality of a hierarchy . Most of the aforementioned methods focus on vector data . Contributions . In this work , we focus on hierarchical clustering in homophilic graphs , where the similarity of two nodes is made explicit by the presence or absence of an edge . We ( 1 ) propose a probabilistic model over hierarchies via continuous relaxation of a tree ’ s parent assignment matrices . We ( 2 ) theoretically analyze the model by drawing connections to absorbing Markov chains , which allows ( 3 ) efficient and exact computation of relevant quantities ( e.g. , LCA probabilities ) . This enables us to ( 4 ) learn hierarchies on graphs by efficient , end-to-end optimization of quality metrics such as Dasgupta cost and the TSD score . Our extensive experimental evaluation on 12 real-world graphs , including a massive graph with more than 2M nodes and 60M edges , highlights the effectiveness of our Flexible Probabilistic Hierarchy model ( FPH ) . It outperforms all baselines , traditional and recent , on the quality of the learned hierarchies measured both by TSD and Dasgupta cost . Remarkably , FPH also performs competitively on link prediction despite not being trained on this task . 2 HIERARCHICAL GRAPH CLUSTERING PRELIMINARIES We define a graph G = ( E , V ) with n nodes and m undirected edges . We let w ( vi , vj ) be equal to the weight of edge ( vi , vj ) if they are connected and to 0 otherwise , and w ( vi ) = ∑ j w ( vi , vj ) to be equal to the weight of node vi . This allows us to define the edge distribution P ( vi , vj ) ∝ w ( vi , vj ) normalized over all edges s.t . ∑ ij P ( vi , vj ) = 1 and the node distribution P ( vi ) ∝ w ( vi ) s.t . ∑ i P ( vi ) = 1 , which are later needed to compute the quality metrics of learned hierarchies . Discrete hierarchical clustering . We define a hierarchical clustering T of a graph G as a tree structured partitioning of the nodes V . We denote the nodes of the tree T with n′ ∈ N internal nodes by { x1 , ... , xn+n′ } . The tree nodes can be decomposed into the leaf nodes { v1 , ... , vn } corresponding to the nodes of the input graph , and the internal nodes { z1 , ... , zn′ } representing clusters of leaf nodes ; overall { x1 , ... , xn+n′ } = { v1 , ... , vn } ∪ { z1 , ... , zn′ } . Since any tree is a directed acyclic graph ( DAG ) , we impose w.l.o.g . a topological ordering on the internal nodes of the tree in the remainder of this work . That is , zi can not be a parent of zj if i < j ; zn′ denotes the root node . We can uniquely define a tree as T = ( Â , B̂ ) with two adjacency matrices , Â ∈ { 0 , 1 } n×n′ and B̂ ∈ { 0 , 1 } n′×n′ . Â describes the connections from the leaves to the internal nodes , i.e . Âi , k = 1 if internal node zk is the parent of leaf node vi . B̂ describes the connections between the internal nodes , i.e . Bk , l = 1 if internal node zl is the parent of internal node zk . Since in a tree any node has exactly one parent , Â is row-stochastic and B̂ is row-stochastic ; except its last row B̂n′ corresponding to the root , whose sum is 0 . We denote by vi ∧ vj the internal node which is the lowest common ancestor ( LCA ) of vi , vj in the tree . The maximum possible number of internal nodes is n′ = n− 1 . Probabilistic hierarchical clustering . The central idea of FPH is to define a probabilistic tree via continuous relaxation of the binary-valued adjacency matrices Â , B̂ . I.e. , we define A , B such that A ∈ [ 0 , 1 ] n×n′ and B ∈ [ 0 , 1 ] n′×n′ while keeping the row-stochasticity constraints , i.e. , ∑n′ j Ai , j = 1 for 1 ≤ i ≤ n , ∑n′ l Bk , l = 1 , 1 ≤ k < n′ , and for the root node we have∑n′ l Bn′ , l = 0 . This describes a probabilistic hierarchy ; the probability of an internal node zj to be a parent of a leaf or an internal node is Aij : = p ( zj |vi ) or Bij : = p ( zj |zi ) , respectively . See Fig . 1 for a visualization . Our key technical contribution are efficient closed-form expressions for the lowest common ancestor probabilities that arise when sampling discrete trees from the probabilistic one . 2.1 INTERNAL METRICS FOR HIERARCHICAL CLUSTERING . Internal metrics are commonly used to evaluate clustering methods , and they have the important advantage of not requiring external labels . For hierarchical clustering on graphs , two internal metrics have been proposed to assess the quality of a discrete hierarchy T given an input graph G : Dasgupta cost ( Das . ) ( Dasgupta , 2016 ) , and Tree-Sampling Divergence ( TSD ) ( Charpentier & Bonald , 2019 ) . While the Dasgupta cost can be used for graphs and vector data , TSD is tailored specifically to graphs . Das ( T ) = ∑ vi , vj∈V P ( vi , vj ) · c ( vi ∧ vj ) , TSD ( T ) = KL ( p ( z ) ||q ( z ) ) = ∑ z p ( z ) log p ( z ) q ( z ) ( 1 ) For Das. , c ( z ) = |z| denotes the number of leaves for which the internal node , i.e. , z = vi ∧ vj , is an ancestor . Intuitively , the cost is large if vi and vj are connected by an edge and their lowest common ancestor contains many leaves , i.e. , is close to the root . Thus , Dasgupta favors leaves connected by an edge to have LCAs low in the hierarchy . For Das. , lower scores indicate better hierarchies . TSD defines two distributions over the internal nodes , p ( z ) = ∑ vi , vj : z=vi∧vj P ( vi , vj ) , induced by the edge distribution , and q ( z ) = ∑ vi , vj : z=vi∧vj P ( vi ) P ( vj ) , induced by the independent node distribution . Intuitively , we expect the graph distribution p ( z ) ( induced by edge sampling ) and the null model distribution q ( z ) ( induced by independent node sampling ) to differ significantly if the tree T indeed represents the hierarchical structure of the graph , leading to high TSD scores . Continuous relaxation . To enable end-to-end gradient-based optimization of Dasgupta or TSD , we replace the discrete hierarchy T = ( Â , B̂ ) with its continuous relaxation T = ( A , B ) . This leads to probabilistic parent assignments of leaves to internal nodes . By itself , this does not help us in hierarchical clustering as it is not clear how to compute the lowest common ancestor probabilities when sampling discrete hierarchies from the relaxed one , i.e. , the probability that an internal node is the lowest common ancestor of two leaves . Our main theoretical contribution is to derive closed-form expressions for the lowest common ancestor probabilities in Sec . 3 . This gives rise to continuous versions of TSD and Dasgupta ( see equations in Appendix A.1 ) , which we will refer to as Soft-TSD and Soft-Das. , respectively . For the first time , this allows us to directly optimize for hierarchical clustering quality metrics in an end-to-end fashion instead of proxy losses or heuristic algorithms . For optimization over hierarchies , we either minimize the Dasgupta cost or maximize TSD : min A , B Soft-Das ( T = ( A , B ) ) or max A , B Soft-TSD ( T = ( A , B ) ) , ( 2 ) where A ∈ [ 0 , 1 ] n×n′ , B ∈ [ 0 , 1 ] n′×n′ are the continuous relaxations of the parent assignment matrices as described in Sec . 2 . Note that for both metrics we recover the same score as their discrete versions in the case of a deterministic probabilistic model , i.e . when A , B are binary-valued .

Summary: The paper proposes a probabilistic model of tree-based hierarchical clustering by considering a continuous relaxation. The authors establishes the theorical connections between tree-sampling and the Markov chain that is specifically constructed. This allows an end-to-end gradient-based learning to optimize the internal metrics of graph clustering, namely the Dasgupta cost and Tree Sampling Divergence. This method allows efficient and effective modeling of the graph clustering. Extensive experiments results on different types of graph, including some of the large scale ones, suggests the proposed method behave favorably.