Deep Graph Infomax

1 INTRODUCTION . Generalizing neural networks to graph-structured inputs is one of the current major challenges of machine learning ( Bronstein et al. , 2017 ; Hamilton et al. , 2017b ; Battaglia et al. , 2018 ) . While significant strides have recently been made , notably with graph convolutional networks ( Kipf & Welling , 2016a ; Gilmer et al. , 2017 ; Veličković et al. , 2018 ) , most successful methods use supervised learning , which is often not possible as most graph data in the wild is unlabeled . In addition , it is often desirable to discover novel or interesting structure from large-scale graphs , and as such , unsupervised graph learning is essential for many important tasks . Currently , the dominant algorithms for unsupervised representation learning with graph-structured data rely on random walk-based objectives ( Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ; Tang et al. , 2015 ; Hamilton et al. , 2017a ) , sometimes further simplified to reconstruct adjacency information ( Kipf & Welling , 2016b ; Duran & Niepert , 2017 ) . The underlying intuition is to train an encoder network so that nodes that are “ close ” in the input graph are also “ close ” in the representation space . While powerful—and related to traditional metrics such as the personalized PageRank score ( Jeh & Widom , 2003 ) —random walk methods suffer from known limitations . Most prominently , the random-walk objective is known to over-emphasize proximity information at the expense of structural information ( Ribeiro et al. , 2017 ) , and performance is highly dependent on hyperparameter choice ( Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ) . Moreover , with the introduction of stronger ∗Work performed while the author was at Mila . †CIFAR Fellow encoder models based on graph convolutions ( Gilmer et al. , 2017 ) , it is unclear whether randomwalk objectives actually provide any useful signal , as these encoders already enforce an inductive bias that neighboring nodes have similar representations . In this work , we propose an alternative objective for unsupervised graph learning that is based upon mutual information , rather than random walks . Recently , scalable estimation of mutual information was made both possible and practical through Mutual Information Neural Estimation ( MINE , Belghazi et al. , 2018 ) , which relies on training a statistics network as a classifier of samples coming from the joint distribution of two random variables and their product of marginals . Following on MINE , Hjelm et al . ( 2018 ) introduced Deep InfoMax ( DIM ) for learning representations of highdimensional data . DIM trains an encoder model to maximize the mutual information between a high-level “ global ” representation and “ local ” parts of the input ( such as patches of an image ) . This encourages the encoder to carry the type of information that is present in all locations ( and thus are globally relevant ) , such as would be the case of a class label . DIM relies heavily on convolutional neural network structure in the context of image data , and to our knowledge , no work has applied mutual information maximization to graph-structured inputs . Here , we adapt ideas from DIM to the graph domain , which can be thought of as having a more general type of structure than the ones captured by convolutional neural networks . In the following sections , we introduce our method called Deep Graph Infomax ( DGI ) . We demonstrate that the representation learned by DGI is consistently competitive on both transductive and inductive classification tasks , often outperforming both supervised and unsupervised strong baselines in our experiments . 2 RELATED WORK . Contrastive methods . An important approach for unsupervised learning of representations is to train an encoder to be contrastive between representations that capture statistical dependencies of interest and those that do not . For example , a contrastive approach may employ a scoring function , training the encoder to increase the score on “ real ” input ( a.k.a , positive examples ) and decrease the score on “ fake ” input ( a.k.a. , negative samples ) . Contrastive methods are central to many popular word-embedding methods ( Collobert & Weston , 2008 ; Mnih & Kavukcuoglu , 2013 ; Mikolov et al. , 2013 ) , but they are found in many unsupervised algorithms for learning representations of graphstructured input as well . There are many ways to score a representation , but in the graph literature the most common techniques use classification ( Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ; Kipf & Welling , 2016b ; Hamilton et al. , 2017b ) , though other scoring functions are used ( Duran & Niepert , 2017 ; Bojchevski & Günnemann , 2018 ) . DGI is also contrastive in this respect , as our objective is based on classifying local-global pairs and negative-sampled counterparts . Sampling strategies . A key implementation detail to contrastive methods is how to draw positive and negative samples . The prior work above on unsupervised graph representation learning relies on a local contrastive loss ( enforcing proximal nodes to have similar embeddings ) . Positive samples typically correspond to pairs of nodes that appear together within short random walks in the graph—from a language modelling perspective , effectively treating nodes as words and random walks as sentences . Recent work by Bojchevski & Günnemann ( 2018 ) uses node-anchored sampling as an alternative . The negative sampling for these methods is primarily based on sampling of random pairs , with recent work adapting this approach to use a curriculum-based negative sampling scheme ( with progressively “ closer ” negative examples ; Ying et al. , 2018a ) or introducing an adversary to select the negative examples ( Bose et al. , 2018 ) . Predictive coding . Contrastive predictive coding ( CPC , Oord et al. , 2018 ) is another method for learning deep representations based on mutual information maximization . Like the models above , CPC is also contrastive , in this case using an estimate of the conditional density ( in the form of noise contrastive estimation , Gutmann & Hyvärinen , 2010 ) as the scoring function . However , unlike our approach , CPC and the graph methods above are all predictive : the contrastive objective effectively trains a predictor between structurally-specified parts of the input ( e.g. , between neighboring node pairs or between a node and its neighborhood ) . Our approach differs in that we contrast global / local parts of a graph simultaneously , where the global variable is computed from all local variables . To the best of our knowledge , the sole prior works that instead focuses on contrasting “ global ” and “ local ” representations on graphs do so via ( auto- ) encoding objectives on the adjacency matrix ( Wang et al. , 2016 ) and incorporation of community-level constraints into node embeddings ( Wang et al. , 2017 ) . Both methods rely on matrix factorization-style losses and are thus not scalable to larger graphs . 3 DGI METHODOLOGY . In this section , we will present the Deep Graph Infomax method in a top-down fashion : starting with an abstract overview of our specific unsupervised learning setup , followed by an exposition of the objective function optimized by our method , and concluding by enumerating all the steps of our procedure in a single-graph setting . 3.1 GRAPH-BASED UNSUPERVISED LEARNING . We assume a generic graph-based unsupervised machine learning setup : we are provided with a set of node features , X = { ~x1 , ~x2 , . . . , ~xN } , whereN is the number of nodes in the graph and ~xi ∈ RF represents the features of node i . We are also provided with relational information between these nodes in the form of an adjacency matrix , A ∈ RN×N . While A may consist of arbitrary real numbers ( or even arbitrary edge features ) , in all our experiments we will assume the graphs to be unweighted , i.e . Aij = 1 if there exists an edge i→ j in the graph and Aij = 0 otherwise . Our objective is to learn an encoder , E : RN×F × RN×N → RN×F ′ , such that E ( X , A ) = H = { ~h1 , ~h2 , . . . , ~hN } represents high-level representations ~hi ∈ RF ′ for each node i . These representations may then be retrieved and used for downstream tasks , such as node classification . Here we will focus on graph convolutional encoders—a flexible class of node embedding architectures , which generate node representations by repeated aggregation over local node neighborhoods ( Gilmer et al. , 2017 ) . A key consequence is that the produced node embeddings , ~hi , summarize a patch of the graph centered around node i rather than just the node itself . In what follows , we will often refer to ~hi as patch representations to emphasize this point . 3.2 LOCAL-GLOBAL MUTUAL INFORMATION MAXIMIZATION . Our approach to learning the encoder relies on maximizing local mutual information—that is , we seek to obtain node ( i.e. , local ) representations that capture the global information content of the entire graph , represented by a summary vector , ~s . In order to obtain the graph-level summary vectors , ~s , we leverage a readout function , R : RN×F → RF , and use it to summarize the obtained patch representations into a graph-level representation ; i.e. , ~s = R ( E ( X , A ) ) . As a proxy for maximizing the local mutual information , we employ a discriminator , D : RF × RF → R , such that D ( ~hi , ~s ) represents the probability scores assigned to this patch-summary pair ( should be higher for patches contained within the summary ) . Negative samples for D are provided by pairing the summary ~s from ( X , A ) with patch representations ~̃hj of an alternative graph , ( X̃ , Ã ) . In a multi-graph setting , such graphs may be obtained as other elements of a training set . However , for a single graph , an explicit ( stochastic ) corruption function , C : RN×F × RN×N → RM×F × RM×M is required to obtain a negative example from the original graph , i.e . ( X̃ , Ã ) = C ( X , A ) . The choice of the negative sampling procedure will govern the specific kinds of structural information that is desirable to be captured as a byproduct of this maximization . For the objective , we follow the intuitions from Deep InfoMax ( DIM , Hjelm et al. , 2018 ) and use a noise-contrastive type objective with a standard binary cross-entropy ( BCE ) loss between the samples from the joint ( positive examples ) and the product of marginals ( negative examples ) . Following their work , we use the following objective1 : L = 1 N +M N∑ i=1 E ( X , A ) [ logD ( ~hi , ~s ) ] + M∑ j=1 E ( X̃ , Ã ) [ log ( 1−D ( ~̃ hj , ~s ) ) ] ( 1 ) This approach effectively maximizes mutual information between ~hi and ~s , based on the JensenShannon divergence2 between the joint and the product of marginals . As all of the derived patch representations are driven to preserve mutual information with the global graph summary , this allows for discovering and preserving similarities on the patch-level—for example , distant nodes with similar structural roles ( which are known to be a strong predictor for many node classification tasks ; Donnat et al. , 2018 ) . Note that this is a “ reversed ” version of the argument given by Hjelm et al . ( 2018 ) : for node classification , our aim is for the patches to establish links to similar patches across the graph , rather than enforcing the summary to contain all of these similarities ( however , both of these effects should in principle occur simultaneously ) .

Summary: This paper describes an approach for unsupervised learning of node features on a graph (with known structure), so that learned local representations represent community information that has high mutual info with a graph-level summary. The general idea is they apply InfoMax to graphs via graph convolutional networks (GCN), and report impressive results, including rivaling supervised learning methods for node classification. The 3 experiments are on paper topic classification, social network modeling, and protein classification.