GraphSAD: Learning Graph Representations with Structure-Attribute Disentanglement

1 INTRODUCTION . Representing nodes or entire graphs with informative low-dimensional feature vectors plays a crucial role in many real-world applications and domains , e.g . user analysis in social networks ( Tan et al. , 2011 ; Yan et al. , 2013 ) , relational inference in knowledge graphs ( Bordes et al. , 2013 ; Trouillon et al. , 2016 ; Sun et al. , 2019 ) , molecular property prediction in drug/material discovery ( Gilmer et al. , 2017 ; Wu et al. , 2018 ) and circuit response prediction in circuit design ( Zhang et al. , 2019 ) . Recently , Graph Neural Networks ( GNNs ) ( Kipf & Welling , 2017 ; Velickovic et al. , 2018 ; Xu et al. , 2019 ) have shown their superiority in many different tasks . In general , the essential idea of these methods is to learn effective node representations ( or graph representations with an additional graph pooling ) through aggregating the attributes of each node and its neighbors in an iterative and nonlinear way . For an attributed graph , GNNs commonly encode the information of its graph structure and node attributes into a single representation . This might be problematic , since the semantic space of graph structure and node attributes might not be well aligned , and these two types of information could be useful for different tasks . For example , predicting the health condition of a user mainly depends on his/her profile information , and the social network does not provide too much meaningful information ; in another case , the prediction of a user ’ s social class mainly relies on his/her social network structure . Therefore , a more reasonable solution is to disentangle these two types of information into two distinct sets of representations , and the importance of which can be further determined by downstream tasks . Such disentangled representation has been proved to be beneficial to model ’ s generalization ability and interpretability ( Chen et al. , 2016 ; Higgins et al. , 2017 ; Alemi et al. , 2017 ) . Recently , DisenGNN ( Ma et al. , 2019 ) studied disentangled node representation learning by grouping the neighbors of each node to different channels , and each channel corresponds to a different latent factor . In other words , DisenGNN focuses on disentangling the various latent factors of graph structure . By contrast , our work intends to disentangle the representations of graph structure and node attributes , which is orthogonal to their work and also more general . In this paper , we aim to learn node/graph representations with Structure-Attribute Disentanglement ( GraphSAD ) . As a naive trial , we first attempt to conduct disentanglement in the input space , named as Input-SAD , which separates a graph into a structure and an attribute component and then encodes these two components respectively . However , since graph structure and node attributes are not completely independent , it is better to suppress the dependency of these two factors in the embedding space , instead of directly separating the input graph . Inspired by this fact , we propose to distill a graph ’ s structure and attribute information into the distinct channels of embedding vectors , named as Embed-SAD . Concretely , for each node embedding , half of its elements capture the graph structure through edge reconstruction , and the other half extracts the attribute information by minimizing the mutual information with the structure counterpart and , at the same time , preserving semantic discriminability . In addition , we devise a metric to quantitatively evaluate graph representation ’ s structure-attribute disentanglement , denoted as SAD-Metric , which measures the sensitivity of a model when varying either the graph structure or node attributes of an input graph . We summarize our contributions as follows : • We study structure-attribute disentangled node/graph representation learning through separating graph structure and node attributes in either the input or the embedding space . • We design a quantitative metric to measure the extent of structure-attribute disentangle- ment , which is novel on its graph-specific data processing scheme . • Through combining the proposed disentangling techniques with various GNNs , we empir- ically verify our method ’ s superior performance on both the node and graph classification benchmark datasets . Also , we analyze the disentangled graph representations via the proposed metric and qualitative visualization . 2 PROBLEM DEFINITION AND PRELIMINARIES . 2.1 PROBLEM DEFINITION . We study learning node representations ( e.g . social networks ) or whole-graph representations ( e.g . molecular graphs ) of attributed graphs . Formally , we denote an attributed graph as G = ( V , E , A ) . V denotes the set of nodes . E = { ( u , v , tuv ) } is the set of edges with tuv as the type of the edge connecting node u and v ( e.g . different types of bonds in molecular graphs ) . A = { Av|v ∈ V } represents the set of node attributes . Our goal is to learn meaningful representations for each node or the whole graph . Existing GNNs typically mix both the graph structure and node attributes into a unified representation through neural message passing . However , in practice , these two types of information may encode different semantics and be useful for different tasks . Take the prediction on social networks as an example . When predicting the social class of users , the graph structure plays a more important role than user attributes , while user attributes are definitely more informative than graph structure when forecasting users ’ health conditions . It is therefore desirable to disentangle the information of graph structure and node attributes into different sets of representations and use the downstream task to determine their importance . Specifically , we define our problem as follows : Node/Graph Representation Learning with Structure-Attribute Disentanglement . Given an attributed graph G = ( V , E , A ) , we aim to learn node ( or whole-graph ) representations by disentangling the semantics of graph structure S = { V , E } and node attributes A into two distinct sets of representations , i.e . zv = [ zv , S , zv , A ] ( or zG = [ zG , S , zG , A ] ) . The importance of the two kinds of representations is further determined by the downstream task such as node or graph classification . 2.2 PRELIMINARIES . Graph Neural Networks ( GNNs ) . A GNN maps each node v ∈ V to an embedding vector zv and also encodes the entire graph G as vector zG . For an L-layer GNN , the L-hop information surrounding each node is captured via a neighborhood aggregation mechanism . Formally , the l-th GNN layer can be defined as : z ( l ) v = COMBINE ( l ) ( z ( l−1 ) v , AGGREGATE ( l ) ( { ( z ( l−1 ) v , z ( l−1 ) u , tuv ) : u ∈ N ( v ) } ) ) , ( 1 ) where N ( v ) is the set of node v ’ s neighbors , tuv denotes edge attribute , z ( l ) v denotes the representation of v at the l-th layer , and z ( 0 ) v is initialized by the node attribute Av . Using all the node embeddings in a graph , the entire graph ’ s embedding can be derived by a permutation-invariant readout function : zG = READOUT ( { zv|v ∈ V } ) . ( 2 ) Mutual Information Estimator . Mutual information ( MI ) quantifies the mutual dependency between two random variables . Some recent works ( Belghazi et al. , 2018 ; Hjelm et al. , 2019 ) studied neural-network-based MI estimators . Among which , the Noise-Contrastive Estimation ( NCE ) ( Gutmann & Hyvärinen , 2010 ; 2012 ) was first employed as a lower bound of MI by van den Oord et al . ( 2018 ) , and we also adopt this estimator in our method for its effectiveness and concision . In practice , for two random variables x1 and x2 , given one positive pair ( x+1 , x + 2 ) ∼ p ( x1 , x2 ) and K distractors ( x+1 , x2 , j ) ∼ p ( x1 ) p ( x2 ) ( j = 1 , 2 , · · · , K ) , the NCE estimation of MI is defined as : INCE ( x+1 , x + 2 , { x2 , j } Kj=1 ) = log ( K+1 ) +log exp ( T ( x+1 , x + 2 ) ) exp ( T ( x+1 , x + 2 ) ) + ∑K j=1 exp ( T ( x+1 , x2 , j ) ) , ( 3 ) where T ( · , · ) is a parameterized discriminator function which outputs a scalar value for a pair of input samples , and its architecture is detailed in Sec . 5.1 . 3 LEARNING GRAPH REPRESENTATIONS WITH STRUCTURE-ATTRIBUTE DISENTANGLEMENT . 3.1 INPUT-SAD : STRUCTURE-ATTRIBUTE DISENTANGLEMENT FOR INPUTS . As an initial attempt , we seek to learn structure-attribute disentangled node/graph representations by separating a graph into a structure and an attribute component and then encoding them respectively , as shown in Fig . 1 ( a ) . Concretely , given an attributed graph G = ( V , E , A ) , these two components are constructed and encoded as follows . The structure component extracts the graph structure and forms another graph GS = ( VS , ES , AS ) , in which the node and edge sets remain unchanged , i.e . VS = V , ES = E , and the out-degree of each node serves as its attribute , i.e . AS = { d ( v ) |v ∈ VS } ( d ( · ) denotes the out-degree function ) . A GNN maps this component to a δ-dimensional embedding space : ( zV , S , zG , S ) = GNN ( VS , ES , AS ) , ( 4 ) where zV , S = { zv , S |v ∈ V } ∈ R|V|×δ denotes the node embeddings derived only by the graph structure , and zG , S ∈ Rδ is the embedding of the entire structure component . The attribute component is formed as a feature matrix U ∈ R|V|×D , where the feature vector Uv ∈ RD is a D-dimensional embedding of node attribute Av . For this component , a fully-connected network and a readout function ( e.g . mean pooling in our implementation ) are used for encoding : zV , A = FCN ( U ) , zG , A = READOUT ( zV , A ) , ( 5 ) where zV , A = { zv , A|v ∈ V } ∈ R|V|×δ denotes the attribute embeddings of the nodes in graph G , and zG , A ∈ Rδ embeds the whole attribute component . The complete information of graph G is restored by concatenating the structure and attribute embedding for each node and for entire graph : zV = { [ zv , S , zv , A ] | v ∈ V } ∈ R|V|×2δ , zG = [ zG , S , zG , A ] ∈ R2δ , ( 6 ) where [ · , · ] denotes the concatenation operation . Upon these concatenated node/graph embeddings , the prediction task ( e.g . node/graph classification ) is performed by a task-specific network C , which defines the supervised loss Lsup for model optimization : min GNN , FCN , C Lsup . ( 7 )

Summary: This paper presents a novel method called Embed-SAD (as well as Input-SAD) to learn graph/node representations to disentangle structure and attribute information. Input-SAD is a simple baseline that tries to get structure-attribute disentanglements by individually processing graph structures and node attributes. For structure, the original node attibutes are replaced by out-degrees only, and passed to GNNs, while for attibutes, the node attibutes are passed to fully-connected networks. Embed-SAD is a more elaborate method to disentangle the GNN embeddings by posing two types of additional losses, i.e., the edge-reconstruction loss for structures, and the Noise-Contrastive Estimation (NCE) loss to maximize the mutual information against the structure-encoding vectors, in addition to the original loss for supervision. The paper also develops an interesting evaluation metric called SAD-Metric where node attibutes or graph structures are exclusively perturbed for each graph, and prediction for whether that perturbation is for structure or for attibutes made by the element-wise absolute differences between embedded vectors before and after the perturbation. This SAD-Metric can quantify the extent to which the obtained representation can detect which perturbation, that for structures or that for attibutes, is made for each sample graph. The experimental results also demonstrated that the structure-attibute disentanglement by Embed-SAD learning strategy actually improved the prediction performance of many off-the-shelf GNNs over many different graph- or node-level tasks.