End-to-End Learning of Probabilistic Hierarchies on Graphs

We propose a novel probabilistic model over hierarchies on graphs obtained by continuous relaxation of tree-based hierarchies . We draw connections to Markov chain theory , enabling us to perform hierarchical clustering by efficient end-to-end optimization of quality metrics such as Dasgupta cost or Tree-Sampling Divergence ( TSD ) . We show that our model learns rich , high-quality hierarchies present in 12 real world graphs , including a large graph with 2.3M nodes . Our model consistently outperforms recent as well as strong traditional baselines such as average linkage . We demonstrate that our model also obtains competitive results on link prediction despite not being trained on this task , highlighting the quality of the hierarchies discovered by our model . 1 INTRODUCTION . Clustering is a fundamental problem in unsupervised learning , both in theory and practice . In contrast to traditional ‘ flat ’ clustering , hierarchical clustering brings several advantages . It enables us to naturally analyze ( Zhao & Karypis , 2002 ) and visualize ( Himberg et al. , 2004 ) a given dataset on different scales . Further , different downstream tasks may require different granularities . Hierarchical clustering lets us pick the desired granularity after learning . It can also be used for personalized recommendation ( Zhang et al. , 2014 ) or to solve record linkage tasks ( Wick et al. , 2012 ) . In an influential paper , Eisen et al . ( 1998 ) perform hierarchical clustering on gene expression data and argue that it could be used to discover the yet-unknown meaning of certain genes , which could lead to advances in medicine . Besides traditional vector data , real-world graphs are often scale-free and hierarchically organized ( Ravasz & Barabási , 2003 ; Barabási & Pósfai , 2016 ) . This makes them interesting candidates for uncovering the underlying hierarchy via hierarchical clustering . Thus , the hierarchical clustering method we introduce is particularly useful for analyzing scale-free real-world networks such as Web graphs , citation networks , flight networks , or biological networks . Related work . We can group hierarchical clustering algorithms into discrete and continuous optimization methods . Discrete optimization methods can themselves be decomposed into agglomerative approaches and divisive approaches . Agglomerative methods – like the famous linkage algorithms ( Gower & Ross , 1969 ; Jardine & Sibson , 1968 ; Bonald et al. , 2018 ) – follow a bottom-up approach and iteratively aggregate the two most similar clusters w.r.t . a given similarity measure . Divisive methods follow a top-down approach and recursively split larger clusters into smaller ones using , e.g. , k-means ( Steinbach et al. , 2000 ) or spectral clustering ( Charikar & Chatziafratis , 2016 ) . While Moseley & Wang ( 2017 ) ; Kamvar et al . ( 2002 ) have shown connections with explicit objective functions , those methods are mainly based on heuristics to avoid exhaustive combinatorial optimization . On the other hand , continuous optimization methods circumvent this issue by ( i ) relaxing the discrete tree structure , ( ii ) defining an explicit quality metric for hierarchies , and ( iii ) optimizing it using gradient-based optimizers . Chierchia & Perret ( 2019 ) ; Monath et al . ( 2017 ; 2019 ) ; Chami et al . ( 2020 ) focus on minimizing the Dasgupta cost ( Dasgupta , 2016 ) ( see Eq . ( 1 ) ) to fit an ultrametric , hyperbolic embeddings , or a probabilistic model over cluster assignments . Monath et al . ( 2017 ) learns a probabilistic assignment of the samples to leaves of a tree with a fixed binary structure . The method of Monath et al . ( 2019 ) requires approximation for the computation of lowest common ancestors ( LCA ) probabilities and regularization losses . Beyond the Dasgupta cost , Charpentier & Bonald ( 2019 ) propose the Tree Sampling Divergence ( TSD ) , a quality metric for hierarchies on graphs . Both metrics have the benefit to be internal metrics , i.e. , they do not require ground truth information to evaluate the quality of a hierarchy . Most of the aforementioned methods focus on vector data . Contributions . In this work , we focus on hierarchical clustering in homophilic graphs , where the similarity of two nodes is made explicit by the presence or absence of an edge . We ( 1 ) propose a probabilistic model over hierarchies via continuous relaxation of a tree ’ s parent assignment matrices . We ( 2 ) theoretically analyze the model by drawing connections to absorbing Markov chains , which allows ( 3 ) efficient and exact computation of relevant quantities ( e.g. , LCA probabilities ) . This enables us to ( 4 ) learn hierarchies on graphs by efficient , end-to-end optimization of quality metrics such as Dasgupta cost and the TSD score . Our extensive experimental evaluation on 12 real-world graphs , including a massive graph with more than 2M nodes and 60M edges , highlights the effectiveness of our Flexible Probabilistic Hierarchy model ( FPH ) . It outperforms all baselines , traditional and recent , on the quality of the learned hierarchies measured both by TSD and Dasgupta cost . Remarkably , FPH also performs competitively on link prediction despite not being trained on this task . 2 HIERARCHICAL GRAPH CLUSTERING PRELIMINARIES We define a graph G = ( E , V ) with n nodes and m undirected edges . We let w ( vi , vj ) be equal to the weight of edge ( vi , vj ) if they are connected and to 0 otherwise , and w ( vi ) = ∑ j w ( vi , vj ) to be equal to the weight of node vi . This allows us to define the edge distribution P ( vi , vj ) ∝ w ( vi , vj ) normalized over all edges s.t . ∑ ij P ( vi , vj ) = 1 and the node distribution P ( vi ) ∝ w ( vi ) s.t . ∑ i P ( vi ) = 1 , which are later needed to compute the quality metrics of learned hierarchies . Discrete hierarchical clustering . We define a hierarchical clustering T of a graph G as a tree structured partitioning of the nodes V . We denote the nodes of the tree T with n′ ∈ N internal nodes by { x1 , ... , xn+n′ } . The tree nodes can be decomposed into the leaf nodes { v1 , ... , vn } corresponding to the nodes of the input graph , and the internal nodes { z1 , ... , zn′ } representing clusters of leaf nodes ; overall { x1 , ... , xn+n′ } = { v1 , ... , vn } ∪ { z1 , ... , zn′ } . Since any tree is a directed acyclic graph ( DAG ) , we impose w.l.o.g . a topological ordering on the internal nodes of the tree in the remainder of this work . That is , zi can not be a parent of zj if i < j ; zn′ denotes the root node . We can uniquely define a tree as T = ( Â , B̂ ) with two adjacency matrices , Â ∈ { 0 , 1 } n×n′ and B̂ ∈ { 0 , 1 } n′×n′ . Â describes the connections from the leaves to the internal nodes , i.e . Âi , k = 1 if internal node zk is the parent of leaf node vi . B̂ describes the connections between the internal nodes , i.e . Bk , l = 1 if internal node zl is the parent of internal node zk . Since in a tree any node has exactly one parent , Â is row-stochastic and B̂ is row-stochastic ; except its last row B̂n′ corresponding to the root , whose sum is 0 . We denote by vi ∧ vj the internal node which is the lowest common ancestor ( LCA ) of vi , vj in the tree . The maximum possible number of internal nodes is n′ = n− 1 . Probabilistic hierarchical clustering . The central idea of FPH is to define a probabilistic tree via continuous relaxation of the binary-valued adjacency matrices Â , B̂ . I.e. , we define A , B such that A ∈ [ 0 , 1 ] n×n′ and B ∈ [ 0 , 1 ] n′×n′ while keeping the row-stochasticity constraints , i.e. , ∑n′ j Ai , j = 1 for 1 ≤ i ≤ n , ∑n′ l Bk , l = 1 , 1 ≤ k < n′ , and for the root node we have∑n′ l Bn′ , l = 0 . This describes a probabilistic hierarchy ; the probability of an internal node zj to be a parent of a leaf or an internal node is Aij : = p ( zj |vi ) or Bij : = p ( zj |zi ) , respectively . See Fig . 1 for a visualization . Our key technical contribution are efficient closed-form expressions for the lowest common ancestor probabilities that arise when sampling discrete trees from the probabilistic one . 2.1 INTERNAL METRICS FOR HIERARCHICAL CLUSTERING . Internal metrics are commonly used to evaluate clustering methods , and they have the important advantage of not requiring external labels . For hierarchical clustering on graphs , two internal metrics have been proposed to assess the quality of a discrete hierarchy T given an input graph G : Dasgupta cost ( Das . ) ( Dasgupta , 2016 ) , and Tree-Sampling Divergence ( TSD ) ( Charpentier & Bonald , 2019 ) . While the Dasgupta cost can be used for graphs and vector data , TSD is tailored specifically to graphs . Das ( T ) = ∑ vi , vj∈V P ( vi , vj ) · c ( vi ∧ vj ) , TSD ( T ) = KL ( p ( z ) ||q ( z ) ) = ∑ z p ( z ) log p ( z ) q ( z ) ( 1 ) For Das. , c ( z ) = |z| denotes the number of leaves for which the internal node , i.e. , z = vi ∧ vj , is an ancestor . Intuitively , the cost is large if vi and vj are connected by an edge and their lowest common ancestor contains many leaves , i.e. , is close to the root . Thus , Dasgupta favors leaves connected by an edge to have LCAs low in the hierarchy . For Das. , lower scores indicate better hierarchies . TSD defines two distributions over the internal nodes , p ( z ) = ∑ vi , vj : z=vi∧vj P ( vi , vj ) , induced by the edge distribution , and q ( z ) = ∑ vi , vj : z=vi∧vj P ( vi ) P ( vj ) , induced by the independent node distribution . Intuitively , we expect the graph distribution p ( z ) ( induced by edge sampling ) and the null model distribution q ( z ) ( induced by independent node sampling ) to differ significantly if the tree T indeed represents the hierarchical structure of the graph , leading to high TSD scores . Continuous relaxation . To enable end-to-end gradient-based optimization of Dasgupta or TSD , we replace the discrete hierarchy T = ( Â , B̂ ) with its continuous relaxation T = ( A , B ) . This leads to probabilistic parent assignments of leaves to internal nodes . By itself , this does not help us in hierarchical clustering as it is not clear how to compute the lowest common ancestor probabilities when sampling discrete hierarchies from the relaxed one , i.e. , the probability that an internal node is the lowest common ancestor of two leaves . Our main theoretical contribution is to derive closed-form expressions for the lowest common ancestor probabilities in Sec . 3 . This gives rise to continuous versions of TSD and Dasgupta ( see equations in Appendix A.1 ) , which we will refer to as Soft-TSD and Soft-Das. , respectively . For the first time , this allows us to directly optimize for hierarchical clustering quality metrics in an end-to-end fashion instead of proxy losses or heuristic algorithms . For optimization over hierarchies , we either minimize the Dasgupta cost or maximize TSD : min A , B Soft-Das ( T = ( A , B ) ) or max A , B Soft-TSD ( T = ( A , B ) ) , ( 2 ) where A ∈ [ 0 , 1 ] n×n′ , B ∈ [ 0 , 1 ] n′×n′ are the continuous relaxations of the parent assignment matrices as described in Sec . 2 . Note that for both metrics we recover the same score as their discrete versions in the case of a deterministic probabilistic model , i.e . when A , B are binary-valued .

Summary: The paper proposes a probabilistic clustering method based on two losses: Dasgupta loss and Tree-sampling divergence. A scalable sampling-based optimization is proposed. The modes is optimized with the projected gradient descent.