Unsupervised Learning of Graph Hierarchical Abstractions with Differentiable Coarsening and Optimal Transport

1 INTRODUCTION . A proliferation of graph neural networks emerged as competitive alternatives to graph kernels for various graph applications ; e.g. , theorem proving ( Wang et al. , 2017 ) and chemoinformatics ( Jin et al. , 2017 ; Fout et al. , 2017 ; Schütt et al. , 2017 ) . These models learn sophisticated feature representations of a graph and its constituents ( i.e. , nodes and edges ) through layers of feature transformation . Among them , a broad array of work is convolutional , extending convolution filters in the spatial domain to the spectral domain or to local neighborhoods ( Bruna et al. , 2014 ; Henaff et al. , 2015 ; Duvenaud et al. , 2015 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Chen et al. , 2018 ; Velic̆ković et al. , 2018 ; Ying et al. , 2018a ; Liao et al. , 2019 ; Xu et al. , 2019b ) ; whereas a few others are recurrent , which treat the representation of a graph node as the state of a dynamical system , being recurrently updated ( Scarselli et al. , 2009 ; Li et al. , 2016 ; Gilmer et al. , 2017 ; Jin et al. , 2017 ) . Several convolution architectures ( Xu et al. , 2019b ; Morris et al. , 2019 ) are connected to the Weisfeiler–Lehman ( WL ) graph isomorphism test because of the resemblance in iterative node ( re ) labeling . They are stated to be as expressive as WL in isomorphism tests , rendering strong competitors to WL graph kernels ( Shervashidze et al. , 2011 ) inspired by the same procedure . An image analog of graph neural networks is convolutional neural networks , whose key components are convolution and pooling . The pooling operation reduces the spatial dimensions of an image and forms a hierarchical abstraction through successive downsampling . For graphs , a similar hierarchical abstraction is particularly important for maintaining the structural information and deriving a faithful feature representation . A challenge , however , is that unlike image pixels that are spatially regular , graph nodes are irregularly connected and hence pooling is less straightforward . Several graph neural networks perform pooling in a hierarchical manner . Bruna et al . ( 2014 ) build a multiresolution hierarchy of the graph with agglomerative clustering , based on -covering . Defferrard et al . ( 2016 ) and Fey et al . ( 2018 ) employ Graclus that successively coarsens a graph based on the heavy-edge matching heuristic . Simonovsky & Komodakis ( 2017 ) construct the hierarchy through a combined use of spectral polarity and Kron reduction . These neural networks build the graph hierarchy in the preprocessing step , which defines in advance how pooling is performed given a graph . No learnable parameters are attached . In fact , most practical coarsening methods to date are built on mathematical heuristics ; how they affect the structure and properties of the graph is less understood ( Loukas & Vandergheynst , 2018 ) . Recently , hierarchical abstractions as a learnable neural network module surfaced in the literature of graph representation learning . Representative approaches include DIFFPOOL ( Ying et al. , 2018b ) , GRAPH U-NET ( Gao & Ji , 2019 ) , and SAGPOOL ( Lee et al. , 2019 ) . In the first approach , a soft clustering of nodes is parameterized and learned . The next graph in the hierarchy is thus a complete graph of the clusters . In the second approach , the top nodes according to some parameterized ordering are selected and the induced subgraph becomes the next graph in the hierarchy . The third approach is similar to the second one , except that the ordering is computed through self-attention . All approaches treat the learnable hierarchy as part of the neural network ( in conjunction with a predictive model ) , which is trained with a downstream task in a ( semi- ) supervised manner . In this work , we propose an unsupervised approach , called OTCOARSENING , that produces a hierarchical abstraction of a graph independent of downstream tasks . Therein , node features for the graphs in the hierarchy are derived simultaneously , so that they can be used for different tasks through training separate downstream predictive models . OTCOARSENING consists of two ingredients : a parameterized graph coarsening strategy in the algebraic multigrid ( AMG ) style ; and an optimal transport that minimizes the structural transportation between two consecutive graphs in the hierarchy . The “ OT ” part of the name comes from Optimal Transport . We show that this unsupervised approach produces meaningful coarse graphs that are structure preserving ; and that the learned representations perform competitively with supervised approaches in graph classification and regression . The contribution of this work is threefold . First , for unsupervised learning we introduce a new technique based on hierarchical abstraction through minimizing discrepancy along the hierarchy . Second , key to a successful hierarchical abstraction is the coarsening strategy . We develop one motivated by AMG and empirically show that the resulting coarse graphs qualitatively preserve the graph structure . Third , we demonstrate that the proposed technique , combining coarsening and unsupervised learning , performs comparably with supervised approaches but is advantageous in practice facing label scarcity . 2 RELATED WORK . Hierarchical ( a.k.a . multilevel or multiscale ) methods are behind the solutions of a variety of problems , particularly for graphs . Therein , coarsening approaches are being constantly developed and applied . Two active areas are graph partitioning and clustering . The former is often used in parallel processing , circuit design , and solutions of linear systems , among many others . The latter appears in descriptive data analysis . Several representative developments are discussed here . It is not intended to be a full account of the overwhelming literature and long history . Many of the graph hierarchical approaches consist of a coarsening and an uncoarsening phase . The coarsening phase successively reduces the size of a given graph , so that an easy solution can be obtained for the smallest one . Then , the small solution is lifted back to the original graph through successive refinement in the reverse coarsening order . For coarsening , a class of approaches applies heave-edge matching heuristics ( Hendrickson & Leland , 1995 ; Karypis & Kumar , 1998 ; Dhillon et al. , 2007 ) . In the conceptual level , nodes connected by a heavily weighted edge are grouped into a node in the coarse graph , so that the edge is protected from partitioning . The use of matching heuristics was not much analyzed until recently . Loukas and coauthors show that for certain graphs , the principal eigenvalues and eigenspaces of the coarsened and the original graph Laplacians are close under randomized matching ( Loukas & Vandergheynst , 2018 ; Loukas , 2019 ) . On the other hand , in the uncoarsening phase , refinement can be done in several ways . One approach is Kernighan-Lin refinement ( Kernighan & Lin , 1970 ) , which is commonly applied in spectral partitioning and spectral clustering methods ( Shi & Malik , 2000 ; Luxburg , 2007 ) , whether or not done in a multilevel fashion . Another approach uses kernel k-means , as in Dhillon et al . ( 2007 ) . Another class of coarsening approaches selects a subset of nodes from the original graph . Call them coarse nodes ; they form the node set of the coarse graph . Other nodes are aggregated with weights to the coarse nodes in certain ways , which , simultaneously define the edges in the coarse graph . Many of these methods were developed akin to algebraic multigrid ( AMG ) ( Ruge & Stüben , 1987 ) , as also is this work . In AMG , the set C of coarse nodes is initialized as empty . Then , each node in the complement of C is investigated in some order ; if its coupling to the current C is sufficiently weak , the node is moved to C. The coupling may be defined based on edge weights ( Kushnir et al. , 2006 ) , diffusion distances ( Livne & Brandt , 2012 ) , or algebraic distances ( Ron et al. , 2011 ; Chen & Safro , 2011 ; Safro et al. , 2014 ) . The aggregation weights are defined accordingly . In this work , the selection of the coarse nodes and the aggregation weights are parameterized and learned instead . Besides the AMG style , the dominant eigenvector of the graph Laplacian has also been used for selecting coarse nodes ( Shuman et al. , 2015 ) , who however use a combination of Kron reduction ( Dörfler & Bullo , 2013 ) and graph sparsification to define the edges of the coarse graph . Hierarchical graph representation is emerging in graph deep learning . Representative approaches include DIFFPOOL ( Ying et al. , 2018b ) , GRAPH U-NET ( Gao & Ji , 2019 ) , and SAGPOOL ( Lee et al. , 2019 ) . Cast in the above setting , DIFFPOOL is more similar to the first class of coarsening approaches , whereas GRAPH U-NET and SAGPOOL more similar to the latter . All methods are supervised , as opposed to ours . Our work is additionally drawn upon optimal transport , a tool recently used for defining similarity of graphs ( Vayer et al. , 2019 ; Xu et al. , 2019a ) . In the referenced work , Gromov–Wasserstein distances are developed that incorporate both node features and graph structures . Moreover , a transportation distance from the graph to its subgraph is developed by Garg & Jaakkola ( 2019 ) . Our approach is based on a relatively simpler Wasserstein distance , whose calculation admits an iterative procedure more friendly to neural network parameterization . 3 METHOD . In this section , we present the proposed method OTCOARSENING , beginning with the two main ingredients : coarsening and optimal transport , followed by a summary of the computational steps in training and the use of the results for downstream tasks . 3.1 AMG-STYLE COARSENING . The first ingredient coarsens a graph G into a smaller one Gc . For a differentiable parameterization , an operator will need be defined that transforms the corresponding graph adjacency matrix A ∈ Rn×n into Ac ∈ Rm×m , where n and m are the number of nodes of G and Gc respectively , with m < n. We motivate the definition by algebraic multigrid ( Ruge & Stüben , 1987 ) , because of the hierarchical connection and a graph-theoretic interpretation . AMG also happened to be referenced as a potential candidate for pooling in some graph neural network architectures ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ) . 3.1.1 BACKGROUND ON ALGEBRAIC MULTIGRID . AMG belongs to the family of multigrid methods ( Briggs et al. , 2000 ) for solving large , sparse linear systems of the form Ax = b , where A is the given sparse matrix , b is the right-hand vector , and x is the unknown vector to be solved for . For simplicity , we assume throughout thatA is symmetric . The simplest algorithm , two-grid V-cycle , consists of the following steps : ( i ) Approximately solve the system with an inexpensive iterative method and obtain an approximate solution x′ . Let r = b−Ax′ be the residual vector . ( ii ) Find a tall matrix S ∈ Rn×m and solve the smaller residual system ( STAS ) y = ST r for the shorter unknown vector y . ( iii ) Now we have a better approximate solution x′′ = x′ + Sy to the original system . Repeat the above steps until the residual is sufficiently small . In practice , it is unlikely that the residual system ( STAS ) y = ST r in the second step , though smaller , can be solved exactly , if the original system Ax = b can not be . Hence , one naturally appeals to recursion . That is , one solves ( STAS ) y = ST r only approximately , obtains the residual , constructs a further smaller residual system , and proceeds recursively , until when a sufficiently small residual system can be solved exactly and inexpensively . The matrix of the residual system , STAS , is called the Galerkin coarse-grid operator . One may show that step ( ii ) , if solved exactly , minimizes the energy norm of the error x−x′′ over all possible corrections from the range of the matrix S. Decades of efforts on AMG discover practical definitions of S that both is economic to construct/apply and encourages fast convergence . We depart from these efforts and define/parameterize an S that best suites graph representation learning .

Summary: This paper proposes an unsupervised hierarchical approach for learning graph representations. The proposed architecture is constructed by unrolling k-steps of a parametrized algebraic multigrid approach for minimizing the Wasserstein metric between the graph and its representation. The node distance (transport cost) used in the Wasserstein metric is also learned as an L2 distance between the embeddings of some graph embedding function. The approach is compared against 6 other state of the art approaches on 5 graph classification tasks, showing significant improvements 4 of them.