Unsupervised Learning of Node Embeddings by Detecting Communities

1 INTRODUCTION . Graphs are a natural representation of relations between entities in complex systems , such as social networks or information networks . To enable inference on graphs , a graph embedding may be learned . It comprises node embeddings , each being a vector-based representation of a graph ’ s node that incorporates its relations to other nodes ( Goyal & Ferrara , 2018 ; Hamilton et al. , 2017b ) . While supervised node embeddings have received a lot of attention , most real-world graphs are not labelled , which calls for unsupervised learning techniques . The main principle in unsupervised learning of node embeddings is that “ similar ” nodes have close embeddings in the embedding space . The similarity of nodes is often defined based on their distance in a graph , e.g. , based on their co-occurrence probability in a random walk ( Goyal & Ferrara , 2018 ; Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) . Recently , it was also argued that two nodes should be similar , if they are similar to a graph summary representation ( Veličković et al. , 2018 ) . In this work , we argue that node embeddings shall not only be of high quality for inference tasks , but shall also be meaningful . That is , they shall directly provide insights into interesting structures in a graph in order to avoid a potentially biased post-analysis step , e.g. , through clustering of the embeddings . We therefore assess node similarity from the perspective of node communities , where the dimensions of embeddings are some unknown communities instead of some unknown latent features as in traditional techniques . Considering a community as a set of densely connected nodes with sparse connections to outside nodes , the homophily principle is restated as follows : Nodes with similar community membership characteristics shall have close embeddings . Specifically , for each node , we incorporate membership information as a probability distribution over a set of communities . Then , nodes are similar , if they are both likely and unlikely to be part of the same communities . Since communities are generally unknown , we propose to minimize the mincut loss for unsupervised learning of communities and node embeddings simultaneously . Mincut loss leverages the principle that communities are well-separated if there are few connections between them ( Fortunato , 2010 ) . It is theoretically motivated as its optimal closed-form solution can be found , while its variant , the normalized cut , is a well-studied problem . Aiming at a realisation of the above idea , we propose Deep MinCut ( DMC ) , a neural network approach to minimize mincut loss . We learn node embeddings to sample one-hot vectors that represent the assignment of nodes to communities . The vectors are drawn from distributions parameterized by continuous node embeddings using Gumbel-Softmax ( Jang et al. , 2016 ; Maddison et al. , 2016 ) . This renders the process differentiable and , thus , enables joint learning of embeddings and communities . We demonstrate the applicability of DMC in various applications . In node classification , our node embeddings turn out to outperform traditional embedding techniques ( Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ; Veličković et al. , 2018 ) , while also revealing the graph ’ s community structure . In community detection , by stacking mincut losses , we are able to learn a hierarchy of communities , e.g. , when generating word embeddings , we can link words to topics , and topics to abstract themes . 2 RELATED WORK . Graph embedding constructs a low-dimensional model of the nodes of a graph that incorporates its structure ( Hamilton et al. , 2017b ; Goyal & Ferrara , 2018 ) . Embedding techniques can be classified into shallow ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) and deep approaches ( Hamilton et al. , 2017a ; Kipf & Welling , 2016 ; Wu et al. , 2019 ) . Shallow approaches rely on an embedding lookup table to map nodes to embeddings . On the other hand , deep models construct a node ’ s embedding by performing aggregation of its neighbours ’ embeddings . Unsupervised node embeddings . While different models may be employed to embed nodes ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) , the majority of unsupervised learning techniques leverage a contrastive loss function , such as skipgram loss ( Hamilton et al. , 2017b ) or infomax loss ( Veličković et al. , 2017 ) . The encoder is trained such that , given a scoring function , a high score is given to positive samples , whereas negative samples receive a low score . For skipgram loss , the positive samples are nodes that are close in a random walk , while the negative samples are randomly selected from other graph nodes . A drawback of the random-walk objective is that it can only capture local information around a node ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) . For infomax loss , positive samples are nodes in the original graph , whereas negative samples are nodes in randomly corrupted graphs . While infomax loss can capture the global structure , its performance is highly dependent on the corruption strategy ( Veličković et al. , 2018 ) . Since embeddings also need to be learned for the corrupted graphs , it further suffers from high training time . While our proposed method also considers the global graph structure , it differs in that our approach is nonconstrastive , i.e. , it does not require unnecessary learning of negative samples . Moreover , due to the nature of our loss function , we are able to learn the node embeddings as well as their clusters . Community detection is a well-studied problem with many applications ( Fortunato , 2010 ; Fortunato & Hric , 2016 ) . While numerous techniques for community detection have been proposed , we focus on those that generate node embeddings , such as spectral methods ( Newman , 2006b ; a ; White & Smyth , 2005 ) . Spectral methods operate either on the modularity matrix ( Newman , 2006b ) or the Laplacian matrix ( White & Smyth , 2005 ) . While embeddings may be learned by reconstructing these matrices ( Wang et al. , 2016 ) , existing methods leverage matrix factorization , which does not scale to large graphs . Closest to our work is ( Nazi et al. , 2019 ) , which proposes a partition loss function for graph partitioning . As the work focuses on graph partitioning , the loss function aims for balanced partitions based on the number of nodes and , therefore , is not applicable in our setting . 3 EMBEDDINGS AND COMMUNITY DETECTION . Graphs . We consider a directed , weighted graph G = { V , F } with nodes V = { vi } and edges F = { ( vi , vj ) | vi ∈ V ∧ vj ∈ V } , each edge ( vi , vj ) being assigned a weight s ( vi , vj ) ∈ R. Such a graph can also be represented by its adjacency matrix A of size n× n , where each row and column represents a node in G and a cell Aij denotes the edge weight . Note that we allow self-loops in the graph , but not multi-edges between nodes . Also , edge weights can be negative . We also consider attributed graphs in which nodes have features . We denote the node features matrix as F ∈ Rn×D . Communities . We denote by C = { C1 , C2 , · · · , Ck } the set of k disjoint communities of graph G , where ⋃k i=1 Ci = V and ∀ Ci 6= Cj , Ci ∩ Cj = ∅ . The assignment of nodes to communities is captured by a membership matrix P ∈ { 0 , 1 } n×k with rows representing nodes in G and columns representing communities in C. As each node is only assigned to one community , the rows of P are one-hot vectors , where Pij = 1 if node vi is assigned to community Cj . Assuming the membership matrix P is already known , the number of cross-connections between communities Ci and Cj can be captured by the non-diagonal elements of the adjacency matrix C of the quotient graph , where the nodes are communities : C = PTAP ( 1 ) On the other hand , elements Cii capture the number of connections within community Ci . Mincut loss . While community detection is a well-studied problem , there is no consensus on the precise notion of a community ( Fortunato , 2010 ) . A common principle is that well-separated communities have more connections inside than across communities . Hence , communities are detected by minimizing the number of connections between them , as captured by the following loss function : LP ( A ) = − k∑ i Cii = −Tr ( PTAP ) ( 2 ) where Tr ( X ) is the trace of matrix X . We call the loss function in Equation 2 mincut loss , as it aims to minimize the number of connections between communities . Degenerated cases . Minimizing Equation 2 may lead to degenerated cases , where all nodes are assigned to one community while the others are empty ( Fortunato , 2010 ) . In practice , there are two solutions to this problem . If there is prior knowledge on the communities ( e.g. , they shall have equal size ) , a respective constraint is added to the mincut loss . Another approach is to minimize the normalized cut ( Shi & Malik , 2000 ; Zhang & Rohe , 2018 ) , which is defined as follows : ncut ( C ) = k∑ i=1 cut ( Ci , C−i ) assoc ( Ci , C−i ) = k∑ i=1 PT : ,i ( D−A ) P : ,i PT : ,iDP : ,i = Tr ( PT ( D−A ) P PTDP ) = Tr ( PTLP PTDP ) where C−i denotes the set of communities except Ci and assoc ( Ci , C−i ) is the total degree of nodes in community Ci . Then , the normalized cut ( normcut ) loss can be captured as follows : LP ( A ) = Tr ( PTLsymP PTP ) ( 3 ) where Lsym = I −D−1/2AD1/2 is the symmetric Laplacian matrix with D be the degree matrix of A and and L = D−A is the Laplacian matrix . Graph-like data . For ease of presentation , mincut loss is formulated based on graphs . Yet , it can be applied to any problem comprising a set T of items and a kernel function k : T × T → R that assigns weights to item pairs . This creates a kernel matrix K that can be considered as the adjacency matrix of the items . Mincut loss is designed to separate items into subsets such that the connection strength between every pair of subsets is minimized , which also means the coherence of each subset is maximized . Hence , mincut loss is applicable to a wide range of unsupervised problems . 4 DEEP MINCUT . We first discuss the spectral approach to find optimal solutions for normcut and mincut loss , as it provides a baseline technique for comparison . We then introduce Deep MinCut along with an efficient training process based on minibatches .

Summary: This paper is reporting an unsupervised approach to learn node embeddings and communities simultaneously by minimizing the mincut loss function. This approach programs the data through encoder to generate membership likehood matrix H, and then generates corresponding membership matrix P using node selection. The matrix P and adjacency matrix are coupled to generate community adjacency matrix to minimize the cutting. However, despite such attractive points, the novelty and strength of this study is not outstanding enough for publication in ICLR. Details comments are as follows,