GRAPHSAD: LEARNING GRAPH REPRESENTATIONS
Year: 2020

Abstract:
Graph Neural Networks (GNNs) learn effective node/graph representations by aggregating the attributes of neighboring nodes, which commonly derives a single representation mixing the information of graph structure and node attributes. However, these two kinds of information might be semantically inconsistent and could be useful for different tasks. In this paper, we aim at learning node/graph representations with Structure-Attribute Disentanglement (GraphSAD). We propose to disentangle graph structure and node attributes into two distinct sets of representations, and such disentanglement can be done in either the input or the embedding space. We further design a metric to quantify the extent of such a disentanglement. Extensive experiments on multiple datasets show that our approach can indeed disentangle the semantics of graph structure and node attributes, and it achieves superior performance on both node and graph classification tasks.

heading: 1 INTRODUCTION
text: Representing nodes or entire graphs with informative low-dimensional feature vectors plays a crucial role in many real-world applications and domains, e.g. user analysis in social networks (Tan et al., 2011; Yan et al., 2013), relational inference in knowledge graphs (Bordes et al., 2013; Trouillon et al., 2016; Sun et al., 2019), molecular property prediction in drug/material discovery (Gilmer et al., 2017; Wu et al., 2018) and circuit response prediction in circuit design (Zhang et al., 2019). Recently, Graph Neural Networks (GNNs) (Kipf & Welling, 2017; Velickovic et al., 2018; Xu et al., 2019) have shown their superiority in many different tasks. In general, the essential idea of these methods is to learn effective node representations (or graph representations with an additional graph pooling) through aggregating the attributes of each node and its neighbors in an iterative and nonlinear way.
For an attributed graph, GNNs commonly encode the information of its graph structure and node attributes into a single representation. This might be problematic, since the semantic space of graph structure and node attributes might not be well aligned, and these two types of information could be useful for different tasks. For example, predicting the health condition of a user mainly depends on his/her profile information, and the social network does not provide too much meaningful information; in another case, the prediction of a user’s social class mainly relies on his/her social network structure. Therefore, a more reasonable solution is to disentangle these two types of information into two distinct sets of representations, and the importance of which can be further determined by downstream tasks. Such disentangled representation has been proved to be beneficial to model’s generalization ability and interpretability (Chen et al., 2016; Higgins et al., 2017; Alemi et al., 2017).
Recently, DisenGNN (Ma et al., 2019) studied disentangled node representation learning by grouping the neighbors of each node to different channels, and each channel corresponds to a different latent factor. In other words, DisenGNN focuses on disentangling the various latent factors of graph structure. By contrast, our work intends to disentangle the representations of graph structure and node attributes, which is orthogonal to their work and also more general.
In this paper, we aim to learn node/graph representations with Structure-Attribute Disentanglement (GraphSAD). As a naive trial, we first attempt to conduct disentanglement in the input space, named as Input-SAD, which separates a graph into a structure and an attribute component and then encodes these two components respectively. However, since graph structure and node attributes are not completely independent, it is better to suppress the dependency of these two factors in the embedding space, instead of directly separating the input graph. Inspired by this fact, we propose to distill a
graph’s structure and attribute information into the distinct channels of embedding vectors, named as Embed-SAD. Concretely, for each node embedding, half of its elements capture the graph structure through edge reconstruction, and the other half extracts the attribute information by minimizing the mutual information with the structure counterpart and, at the same time, preserving semantic discriminability. In addition, we devise a metric to quantitatively evaluate graph representation’s structure-attribute disentanglement, denoted as SAD-Metric, which measures the sensitivity of a model when varying either the graph structure or node attributes of an input graph.
We summarize our contributions as follows:
• We study structure-attribute disentangled node/graph representation learning through separating graph structure and node attributes in either the input or the embedding space. • We design a quantitative metric to measure the extent of structure-attribute disentangle-
ment, which is novel on its graph-specific data processing scheme. • Through combining the proposed disentangling techniques with various GNNs, we empir-
ically verify our method’s superior performance on both the node and graph classification benchmark datasets. Also, we analyze the disentangled graph representations via the proposed metric and qualitative visualization.

heading: 2 PROBLEM DEFINITION AND PRELIMINARIES

heading: 2.1 PROBLEM DEFINITION
text: We study learning node representations (e.g. social networks) or whole-graph representations (e.g. molecular graphs) of attributed graphs. Formally, we denote an attributed graph as G = (V, E ,A). V denotes the set of nodes. E = {(u, v, tuv)} is the set of edges with tuv as the type of the edge connecting node u and v (e.g. different types of bonds in molecular graphs). A = {Av|v ∈ V} represents the set of node attributes.
Our goal is to learn meaningful representations for each node or the whole graph. Existing GNNs typically mix both the graph structure and node attributes into a unified representation through neural message passing. However, in practice, these two types of information may encode different semantics and be useful for different tasks. Take the prediction on social networks as an example. When predicting the social class of users, the graph structure plays a more important role than user attributes, while user attributes are definitely more informative than graph structure when forecasting users’ health conditions. It is therefore desirable to disentangle the information of graph structure and node attributes into different sets of representations and use the downstream task to determine their importance. Specifically, we define our problem as follows:
Node/Graph Representation Learning with Structure-Attribute Disentanglement. Given an attributed graph G = (V, E ,A), we aim to learn node (or whole-graph) representations by disentangling the semantics of graph structure S = {V, E} and node attributes A into two distinct sets of representations, i.e. zv = [zv,S , zv,A] (or zG = [zG,S , zG,A]). The importance of the two kinds of representations is further determined by the downstream task such as node or graph classification.

heading: 2.2 PRELIMINARIES
text: Graph Neural Networks (GNNs). A GNN maps each node v ∈ V to an embedding vector zv and also encodes the entire graph G as vector zG . For an L-layer GNN, the L-hop information surrounding each node is captured via a neighborhood aggregation mechanism. Formally, the l-th GNN layer can be defined as:
z(l)v = COMBINE (l) ( z(l−1)v ,AGGREGATE (l) ({( z(l−1)v , z (l−1) u , tuv ) : u ∈ N (v) })) , (1)
where N (v) is the set of node v’s neighbors, tuv denotes edge attribute, z(l)v denotes the representation of v at the l-th layer, and z(0)v is initialized by the node attribute Av . Using all the node embeddings in a graph, the entire graph’s embedding can be derived by a permutation-invariant readout function:
zG = READOUT ({ zv|v ∈ V }) . (2)
Mutual Information Estimator. Mutual information (MI) quantifies the mutual dependency between two random variables. Some recent works (Belghazi et al., 2018; Hjelm et al., 2019) studied neural-network-based MI estimators. Among which, the Noise-Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2010; 2012) was first employed as a lower bound of MI by van den Oord et al. (2018), and we also adopt this estimator in our method for its effectiveness and concision. In practice, for two random variables x1 and x2, given one positive pair (x+1 ,x + 2 ) ∼ p(x1,x2) and K distractors (x+1 ,x2,j) ∼ p(x1)p(x2) (j = 1, 2, · · · ,K), the NCE estimation of MI is defined as:
INCE ( x+1 ,x + 2 , {x2,j}Kj=1 ) = log(K+1)+log
exp ( T (x+1 ,x + 2 ) )
exp ( T (x+1 ,x + 2 ) ) + ∑K j=1 exp ( T (x+1 ,x2,j) ) , (3) where T (·, ·) is a parameterized discriminator function which outputs a scalar value for a pair of input samples, and its architecture is detailed in Sec. 5.1.

heading: 3 LEARNING GRAPH REPRESENTATIONS WITH STRUCTURE-ATTRIBUTE DISENTANGLEMENT

heading: 3.1 INPUT-SAD: STRUCTURE-ATTRIBUTE DISENTANGLEMENT FOR INPUTS
text: As an initial attempt, we seek to learn structure-attribute disentangled node/graph representations by separating a graph into a structure and an attribute component and then encoding them respectively, as shown in Fig. 1(a). Concretely, given an attributed graph G = (V, E ,A), these two components are constructed and encoded as follows.
The structure component extracts the graph structure and forms another graph GS = (VS , ES ,AS), in which the node and edge sets remain unchanged, i.e. VS = V , ES = E , and the out-degree of each node serves as its attribute, i.e. AS = {d(v)|v ∈ VS} (d(·) denotes the out-degree function). A GNN maps this component to a δ-dimensional embedding space:
(zV,S , zG,S) = GNN(VS , ES ,AS), (4)
where zV,S = {zv,S |v ∈ V} ∈ R|V|×δ denotes the node embeddings derived only by the graph structure, and zG,S ∈ Rδ is the embedding of the entire structure component.
The attribute component is formed as a feature matrix U ∈ R|V|×D, where the feature vector Uv ∈ RD is a D-dimensional embedding of node attribute Av . For this component, a fully-connected network and a readout function (e.g. mean pooling in our implementation) are used for encoding:
zV,A = FCN(U), zG,A = READOUT(zV,A), (5)
where zV,A = {zv,A|v ∈ V} ∈ R|V|×δ denotes the attribute embeddings of the nodes in graph G, and zG,A ∈ Rδ embeds the whole attribute component.
The complete information of graph G is restored by concatenating the structure and attribute embedding for each node and for entire graph:
zV = { [zv,S , zv,A] | v ∈ V } ∈ R|V|×2δ, zG = [zG,S , zG,A] ∈ R2δ, (6)
where [·, ·] denotes the concatenation operation. Upon these concatenated node/graph embeddings, the prediction task (e.g. node/graph classification) is performed by a task-specific network C, which defines the supervised loss Lsup for model optimization:
min GNN,FCN,C
Lsup. (7)

heading: 3.2 EMBED-SAD: STRUCTURE-ATTRIBUTE DISENTANGLEMENT FOR EMBEDDINGS
text: The explicit separation of an input graph into structure and attribute components forces the independent encoding of graph structure and node attributes. However, these two factors are not completely independent. For example, in a social network, the social connections of a person can provide useful information about his/her character, and vice versa. Therefore, the representations derived by Input-SAD may not fully capture the structure and attribute information of a graph. To tackle this shortcoming, we seek to perform encoding on the original graph and distill its structure and attribute information into distinct channels of embedding vectors, as illustrated in Fig. 1(b).
For an attributed graph G = (V, E ,A), a GNN is employed to map the graph to a 2δ-dimensional embedding space: (zV , zG) = GNN(V, E ,A), (8) where zV = {zv|v ∈ V} ∈ R|V|×2δ denotes node embeddings, and zG ∈ R2δ is the embedding of entire graph. We further divide these embeddings into two channels:
zG = [zG,S , zG,A], zv = [zv,S , zv,A], ∀v ∈ V, (9)
where zv,S , zv,A (zG,S , zG,A) ∈ Rδ are the structure and attribute embedding of node v (the whole graph G). In order to distill the structure and attribute information to the corresponding channel, we propose two learning schemes.
Learning structure embedding by edge reconstruction. The node embeddings fully capturing the graph structure are supposed to be capable of reconstructing the edges of this graph. Specifically, using the structure embeddings of a pair of nodes, we expect to predict the existence and type of the edge between them, which defines the reconstruction constraint for learning structure embeddings:
Lrecon = −Eu∼PV ,v∼PV Nt∑ i=0 1[tuv=i] · log p(y = i|zu,S , zv,S), (10)
where PV denotes the uniform distribution over V , 1[tuv=i] is the indicator function judging whether the edge (u, v) belongs to type i (i = 0 denotes there is no edge), Nt is the number of different edge types, and p(y|zu,S , zv,S) is modeled by a neural network F which is detailedly discussed in Sec. B. This objective function relates to the ones proposed in VGAE (Kipf & Welling, 2016) and GraphSAGE (Hamilton et al., 2017), while it additionally constrains the reconstruction of edge type, which enables the structure embeddings to adequately capture the information of graph structure.
Learning attribute embedding by mutual information (MI) minimization. Now that the structure embedding is obtained, we would like to derive the attribute embedding which extracts the information of node attributes and suppresses that of graph structure. To achieve this goal, we employ a neural-network-based MI estimator (i.e. the NCE estimator in Sec. 2.2) to estimate and, simultaneously, minimize the dependency between structure and attribute embedding.
In specific, we denote the structure and attribute latent factor as two random variables, zS and zA, and regard the structure and attribute embedding of each node as the sample from the corresponding marginal distribution, i.e. zv,S ∼ p(zS), zv,A ∼ p(zA) (v ∈ V). For computing the NCE estimation of MI, we define the structure and attribute embedding of the same node as a positive pair, i.e. (zv,S , zv,A) ∼ p(zS , zA) (v ∈ V), and the embedding pair constituted by two different nodes serves as a distractor, i.e. (zv,S , zw,A) ∼ p(zS)p(zA) (v 6= w, v, w ∈ V). With these notions, the estimated MI between two latent factors is defined as:
IS,A = Ev∼PV ,wj∼PV\{v} INCE ( zv,S , zv,A, {zwj ,A}Kj=1 ) , (11)
Algorithm 1: Evaluation procedure for SAD-Metric. Input: Evaluation set D = {Gi}Ni=1, the model to be evaluatedM. Output: The evaluation score for SAD-Metric. Initialize the counter: c = 0 ; for i=1 to N do
yGi ← RandomSample({0, 1}) ; # Sample a binary label if yGi = 0 then G′i ←ModifyAttribute(Gi) ; # Modify the attribute factor of a graph else G′i ←ModifyStructure(Gi) ; # Modify the structure factor of a graph zGi ←M(Gi), zG′i ←M(G ′ i) ; # Infer graph embeddings ∆zGi = |zGi − zG′i | ; # Compute embedding difference Predict the binary label ŷGi = arg maxy p(y|∆zGi) ; if ŷGi = yGi then
c← c+ 1 ;
Return Acc = c/N × 100% ; # Compute prediction accuracy as SAD-Metric
where PV and PV\{v} denote the uniform distribution over the node set with and without node v, and INCE(·, ·, ·) is the NCE estimation function defined in Eq. 3. Through minimizing this estimated MI, both the linear and nonlinear dependency between structure and attribute embeddings can be suppressed, which facilitates structure-attribute disentangled node/graph representations. Note that, such learning mechanism relates to the information bottleneck (IB) principle (Tishby et al., 2000; Tishby & Zaslavsky, 2015; Alemi et al., 2017), while, compared with IB, the proposed approach intends to separate two different sources of information instead of pursuing the maximal compression about the input.
Model optimization. We perform the prediction task (e.g. node/graph classification) by appending a task-specific networkC upon the disentangled node/graph embeddings (i.e. zV or zG), which defines a supervised loss Lsup intended to be minimized. This supervised task also guarantees that the meaningful semantic information is not eliminated by the MI minimization scheme. For structureattribute disentanglement, on one hand, the reconstruction loss Lrecon is minimized to distill the information of graph structure into structure embeddings. On the other hand, the MI minimization is conducted in an adversarial manner, in which the discriminator T (defined in Eq. 3) is trained to maximize IS,A, while the GNN encoder seeks to minimize that term. The overall objective is:
min GNN,F,C max T Lsup + λ1Lrecon + λ2IS,A, (12)
where λ1 and λ2 are the trade-off parameters balancing between different objectives.

heading: 3.3 SAD-METRIC: STRUCTURE-ATTRIBUTE DISENTANGLEMENT METRIC
text: In order to quantify the extent of structure-attribute disentanglement achieved by various models, we devise a classifier-based metric to measure the learnt graph representations. Inspired by a previous work (Higgins et al., 2017), we focus on two desired properties of the disentangled representations: (1) independence: a representation vector is expected to be divided into several channels whose interdependency is as low as possible; (2) interpretability: each of these channels corresponds to a single latent factor of the data.
To derive such a metric, for a given graph G, we first sample a binary label yG from the uniform distribution over {0, 1}. According to this label, either the graph’s structure or attribute is modified under the other factor fixed (yG = 0: fix G’s structure; yG = 1: fix G’s attribute), which forms the counterpart graph G′. In practice, we modify a graph’s structure through randomly dropping one of its edge (Rong et al., 2019), and the graph’s attribute is modified by randomly altering the attribute of a node (implementation details are stated in Sec. B). Using the model to be evaluated, the embeddings of graph G and G′ (i.e. zG and zG′ ) are inferred, and we denote the absolute difference between these two embeddings as ∆zG (i.e. ∆zG = |zG − zG′ |). When the structure and attribute
information are disentangled in graph embeddings (i.e. independence and interpretability holds), the elements corresponding to the fixed factor should possess lower values in ∆zG , which makes it easier to predict yG with a low capacity classifier (e.g. linear classifier) upon ∆zG . Based on this fact, we employ the prediction accuracy of yG on a set of graphs as the structure-attribute disentanglement metric (SAD-Metric). The whole evaluation procedure is summarized in Algorithm 1.
In order to measure the extent of structure-attribute disentanglement in node embeddings, we further design a node-centric metric, named as node-SAD-Metric. The detailed definition and experimental results for this metric are presented in Sec. E.

heading: 3.4 THEORETICAL ANALYSIS
text: In this section, we theoretically illustrate that the disentanglement of structure and attribute representation is able to ease the burden of model optimization by shrinking the solution space.
For an attributed graph G = (V, E ,A), we can regard each type of node attribute as an attribute node, which transforms graph G into another form, G = (V,VA, E , EA) (VA: the set of all attribute nodes, EA: the edges connecting normal and attribute nodes), as shown in Fig. 2(a). This graph can be divided into two parts: (1) a bipartite graph reflecting attribute information, GA = (V,VA, EA) (Fig. 2(b)) and (2) an unattributed graph depicting graph structure, GS = (V, E) (Fig. 2(c)). We first give the definitions of topological space and the spaces for graph GA and GS . Definition 1. A topological space T = ( X,N (x) ) is composed of a set X and a neighborhood function N (x) mapping each x ∈ X to a subset of X . Definition 2. The topological space for graph GA is TA = ( V ∪ VA,NA(v) ) , and the topological
space for graph GS is TS = ( V,NS(v) ) .
We consider two ways of graph embedding. The first way embeds graph GA and GS to a common embedding space, i.e. learning a function f : TA × TS → Z, while the second way embeds two graphs to separate embedding spaces, i.e. learning a function f̃ : TA × TS → ZA × ZS . When the dimension of embedding space is identical in these two ways, we propose that the dimension of the solution space of function f̃ is smaller.
Proposition 1. If it holds that dim(Z) = dim(ZA) + dim(ZS), we have dim(f̃) 6 dim(f).
The detailed proof of this proposition is provided in Sec. A. Proposition 1 tells that the structureattribute disentanglement of graph representation narrows the solution space of the model, which enables us to train the graph encoder more effectively.

heading: 4 RELATED WORK
text: Graph Representation Learning. The early efforts towards learning low-dimensional embeddings of nodes/graphs focused on optimizing the objectives induced by random walk statistics (Perozzi et al., 2014; Tang et al., 2015; Grover & Leskovec, 2016) or matrix factorization (Cao et al., 2015; Wang et al., 2016). By contrast, the Graph Neural Networks (GNNs) (Scarselli et al., 2008) derive embedding vectors via a neighborhood aggregation mechanism. Gilmer et al. (2017) suggested that most GNNs perform a Message Passing and a Readout phase, and different techniques (Kipf & Welling, 2017; Hamilton et al., 2017; Velickovic et al., 2018; Ying et al., 2018; Zhang et al., 2018; Xu et al., 2019) have been explored to enhance the effectiveness of these two phases. Un-
like these methods which mix the information of graph structure and node attributes into a single representation, our approach aims to disentangle these two factors in node/graph representations.
Learning Disentangled Representations. A disentangled representation is expected to separate the distinct and informative factors of variation in the data (Bengio et al., 2013). Some previous works sought to achieve this goal under the guidance of weak supervision (Hinton et al., 2011; Kulkarni et al., 2015; Siddharth et al., 2017). On another line of research, representation disentanglement is pursued by various unsupervised/self-supervised techniques (Chen et al., 2016; Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018). For graph-structured data, a recent work (Ma et al., 2019) disentangled the latent factors of graph structure via a neighborhood routing mechanism. The proposed structure-attribute disentanglement is orthogonal to their work and also more general.
Measuring Disentangled Representations. The quantitative measurement of representation disentanglement is essential to compare different disentangling algorithms. Given the true factors of variation, various disentanglement metrics have been designed based on classifier (Karaletsos et al., 2016; Higgins et al., 2017; Kim & Mnih, 2018), mutual information estimation (Chen et al., 2018; Ridgeway & Mozer, 2018) or distribution entropy (Eastwood & Williams, 2018). The proposed SAD-Metric follows the embedding-based evaluation protocol as previous works, while it is novel on its data processing scheme which is tailored for graph-structured data.

heading: 5 EXPERIMENTS

heading: 5.1 EXPERIMENTAL SETUP
text: Model configurations. For all approaches evaluated in Secs. 5.2 and 5.3, we equip them with a GCN (Kipf & Welling, 2017) (hidden units’ dimension: 300, readout function: mean pooling) with two/three layers for node/graph classification, respectively. The performance on other GNNs, i.e. GraphSAGE (Hamilton et al., 2017), GAT (Velickovic et al., 2018) and GIN (Xu et al., 2019), is reported in Sec. 5.5. For the Input-SAD model, a two-layer fully-connected network (hidden units’ dimension: 300, activation function: ReLU) is adopted to encode the attribute component of graph. For the Embed-SAD model, the discriminator of the NCE estimator is built as an encoder-and-dot architecture: T (x1,x2) = f(x1)Tf(x2), where f(·) is modeled by two linear layers and a ReLU nonlinearity in between, and it projects the original feature vector to an inner-product space.
Training details. In node classification experiments, an Adam optimizer (Kingma & Ba, 2015) (learning rate: 1 × 10−3) is employed to train the model for 1000 epochs, and, for graph classification tasks, we use an Adam optimizer (learning rate: 1 × 10−3, batch size: 32) to perform optimization for 100 epochs. For the negative sampling in NCE estimator (Eq. 3), we utilize all the nodes other than the selected one as negative samples (i.e. K = |V| − 1), while, on three large networks (i.e. PubMed, Coauthor-CS and Coauthor-Physics), 3000 nodes serve as negative samples. Unless otherwise stated, trade-off parameters λ1 and λ2 are set as 1 and 0.1 (sensitivity analysis is in Sec. 5.5), which is determined by the grid search on the validation sets of three citation networks.
Performance comparisons. We combine the proposed Input-SAD and Embed-SAD model with four kinds of GNNs (GCN, GraphSAGE, GAT and GIN) to verify their effectiveness. Furthermore, we compare our method with five existing approaches that seek to promote GNNs’ representation learning capability, i.e. Multi-task (Tran, 2018), GraphMix (Verma et al., 2019), DropEdge (Rong et al., 2019), DisenGNN (Ma et al., 2019) and InitRes (Chen et al., 2020) (detailed settings are in Sec. B). Among which, Multi-task and DisenGNN relate to our method: the former performs link prediction and node/graph classification simultaneously; the latter disentangles node representations based on different generative causes of edges.

heading: 5.2 EXPERIMENTS OF NODE CLASSIFICATION
text: Datasets. We employ three citation networks (i.e. Cora, CiteSeer and PubMed (Sen et al., 2008)) and two larger coauthor networks (i.e. Coauthor-CS and Coauthor-Physics (Shchur et al., 2018)) for semi-supervised node classification (20 labeled nodes per category). The node attributes of three citation networks are the bag-of-words representation of the document, and the nodes in two coauthor networks are featured by the paper keywords for each author’s papers. Edge attributes are not included in these datasets. The details about dataset statistics, dataset split and evaluation protocol are provided in Sec. C.
Results. In Tab. 1, we report the performance of different methods on five standard node classification benchmarks. Based on a two-layer GCN model, the proposed Embed-SAD achieves the highest test accuracy on four of five tasks among all approaches. The performance of Input-SAD is inferior on these tasks, which, we think, is mainly ascribed to its failure in fully capturing the structure and attribute information of each node.

heading: 5.3 EXPERIMENTS OF GRAPH CLASSIFICATION
text: Datasets. For graph classification, we adopt eight single-task/multi-task classification datasets in MoleculeNet (Wu et al., 2018), and, following previous findings (Chen et al., 2012), a scaffold split scheme is utilized for dataset split. For each molecular graph sample, the type and chirality of atom serve as the node attributes, and the type and direction of bond constitute the edge attributes. For evaluation, five independent runs with distinct random seeds are conducted, and the average performance is reported. More dataset statistics are in Sec. C.
Results. Tab. 2 shows the comparisons between our methods and existing techniques. Embed-SAD outperforms DisenGNN, a closely related work, on six of eight datasets, and a 1.1% performance gain is achieved in terms of average ROC-AUC, which illustrates the effectiveness of structureattribute disentanglement. Input-SAD performs well on this graph classification benchmark and obtains superior performance on SIDER and BACE datasets, which, we think, is because the graph pooling operation is able to supplement the missing structure or attribute information of each node from other nodes.

heading: 5.4 QUANTITATIVE EVALUATION OF STRUCTURE-ATTRIBUTE DISENTANGLEMENT
text: Evaluation details. We employ the same datasets and dataset split scheme as in Sec. 5.3. We use a GCN model with random parameters (random-GCN) and a GCN model pre-trained by graph classification (GCN-baseline) as baseline models. Three disentanglement models (DisenGNN, InputSAD and Embed-SAD) also establish appropriate priors by performing graph classification on the training set. A linear binary classifier is built upon the model to be evaluated, and the classifier is
trained with the graphs in the training set and evaluated by test graphs. All results are averaged over five independent runs.
Results. We report the SAD-Metric score of five methods in Tab. 3. The poor performance of random-GCN shows that it can hardly disentangle the structure and attribute information of a graph, and the graph classification pre-training endows the GCN-baseline model with better disentanglement capability. Compared to DisenGNN, the proposed Input-SAD and Embed-SAD model better disentangle graph’s structure and attribute information. The Embed-SAD model achieves the best disentanglement of graph representations on seven of eight datasets.

heading: 5.5 ANALYSIS
text: Effect of two objectives on Embed-SAD. In Tabs. 1 and 2, we analyze the effect of reconstruction loss and structure-attribute mutual information minimization. When removing any of these two objectives, the model’s performance is impaired, which demonstrates their complementary relation.
Results of different GNNs. In Fig. 3(a), we combine three representation disentanglement techniques with four kinds of GNNs, and the average test ROC-AUC over eight datasets of MoleculeNet is plotted (detailed results are in Sec. D). Embed-SAD performs best on all four configurations, and the performance of Input-SAD and DisenGNN is comparable with each other.
Sensitivity of trade-off parameters λ1, λ2. Figs. 3(b), (c) show the performance of Embed-SAD on citation networks using different trade-off weights. We can observe that stable performance gain is obtained when the values of λ1 and λ2 are around 1 and 0.1, respectively.
Visualization. In Figs. 4(a), (b), (c), we visualize the absolute values of the correlations between the learnt graph embedding’s elements on BBBP dataset. Among three models, Embed-SAD suppresses the correlation between structure and attribute embedding to the greatest extent. We further visualize the embedding distributions using t-SNE (Maaten & Hinton, 2008) in Figs. 4(d), (e). Both Input-SAD and Embed-SAD separate two kinds of embeddings into distinct spaces, and attribute embeddings possess stronger semantic discriminability compared to the structure counterparts.

heading: 6 CONCLUSIONS AND FUTURE WORK
text: We study node/graph representation learning with Structure-Attribute Disentanglement (GraphSAD) in both the input and the embedding space. We further design a quantitative metric to measure such disentanglement. On node and graph classification benchmark datasets, we empirically verify our method’s superior performance over existing techniques.
Our future explorations will involve improving the learning manners for structure-attribute disentangled representations, evaluating the proposed models on more tasks (e.g. regression-based tasks) and disentangling graphs in other ways.

heading: A PROOF OF PROPOSITION 1
text: Considering the search of embedding function as a combinatorial search problem (Bruynooghe, 1981; Chen et al., 2001; Kotthoff, 2016), we can derive the dimension of the solution space for two types of embedding functions, i.e. f : TA × TS → Z and f̃ : TA × TS → ZA × ZS , as follows:
dim(f) =
( dim(TA) + dim(TS)
dim(Z)
) , (13)
dim(f̃) = ( dim(TA) dim(ZA) ) · ( dim(TS) dim(ZS) ) . (14)
Under the condition that dim(Z) = dim(ZA) + dim(ZS), the basic property of combinations gives the following inequality:(
dim(TA) dim(ZA)
) · (
dim(TS) dim(ZS)
) 6 ( dim(TA) + dim(TS)
dim(Z)
) , (15)
which deduces that: dim(f̃) 6 dim(f). (16)
This conclusion shows that the solution space of the graph embedding function with structureattribute disentanglement is narrower than that of the entangled counterpart.

heading: B MORE IMPLEMENTATION DETAILS
text: Modeling conditional distribution for edge reconstruction. For the graphs that only provide the existence of edges (i.e. Nt = 1), we utilize an inner-product decoder for edge prediction:
p(y = 0|zu,S , zv,S) = 1− σ(zTu,S zv,S), p(y = 1|zu,S , zv,S) = σ(zTu,S zv,S), (17)
where σ(·) denotes the sigmoid function. For the graphs owning more than one type of edges (i.e. Nt > 1), we concatenate the embedding vectors of two nodes and use a linear classifier to predict the type of the edge between them:
p(y|zu,S , zv,S) = F ( [zu,S , zv,S ] ) , (18)
where F : R2δ → RNt+1 is the edge prediction function modeled by a linear classifier network. Attribute modification for SAD-Metric. To modify the attribute of a graph, we randomly select a node in this graph and alter its attribute. Specifically, for a graph sample (i.e. molecule) in the datasets of MoleculeNet, we modify its attribute by randomly selecting an atom and resetting the atom’s type and chirality to other valid values. Such technique can be applied to any graph dataset in which node attributes are discretely represented.
Detailed settings for compared methods. In the experiments, we compare our method with five existing techniques that aim to improve graph representation learning, i.e. Multi-task (Tran, 2018), GraphMix (Verma et al., 2019), DropEdge (Rong et al., 2019), DisenGNN (Ma et al., 2019) and InitRes (Chen et al., 2020). The detailed settings of these approaches are as follows:
• Multi-task. Node embeddings are employed for both node classification and link prediction, and the loss term for link prediction possesses the weight of 0.1.
• GraphMix. In this approach, parameter α controls the Beta distribution from which mixup ratios are sampled, and we fix this parameter as 1.0 in all experiments.
• DropEdge. During training, 10% edges of each graph are randomly dropped to mitigate the over-fitting and over-smoothing problems.
• DisenGNN. For each GNN layer, the feature vector of each node is divided into 5 channels, and 5 iterations of neighborhood routing are performed.
• InitRes. This method constructs a residual connection from the initial node representations to each GNN layer. We set the residual ratio as 0.2, such that the final representation of each node retains at least 20% of the input feature.

heading: C MORE EXPERIMENTAL DETAILS
text: Citation networks. In Tab. 4, we provide the detailed information about three citation networks, in which 20 labeled nodes per category are used for training, and there are 500 and 1000 nodes for validation and test, respectively. We adopt the standard dataset split proposed in Yang et al. (2016) for all experiments on citation networks. All the reported results are averaged over 100 independent runs using different random seeds.
Coauthor networks. The statistics of two coauthor networks are listed in Tab. 4. Following Shchur et al. (2018), we use 20 labeled nodes per category as the training set, 30 nodes per category as the validation set, and the rest as the test set. The reported accuracy is averaged over 30 random train/validation/test splits, and, for each split, 50 independent runs are performed.
Molecule datasets. In Tab. 5, we present the detailed statistics of eight molecule datasets in MoleculeNet (Wu et al., 2018). On each dataset, a model intends to predict one or multiple properties of various molecules, where the prediction of each property is a binary classification task.

heading: D MORE RESULTS ON MOLECULENET
text: In Tabs. 6, 7 and 8, we combine different techniques with three GNNs (i.e. GraphSAGE (Hamilton et al., 2017), GAT (Velickovic et al., 2018) and GIN (Xu et al., 2019)) and evaluate various models’ performance on MoleculeNet, in which we set the number of attention heads as 5 for GAT. For all the three experiments, the Embed-SAD model outperforms other methods in terms of average test ROC-AUC. The Input-SAD model achieves superior performance on ClinTox and BACE datasets, and its overall performance is comparable with DisenGNN.

heading: E NODE-SAD-METRIC: NODE-CENTRIC STRUCTURE-ATTRIBUTE DISENTANGLEMENT METRIC
text: E.1 DEFINITION
The node-SAD-Metric measures the extent of structure-attribute disentanglement in node embeddings. Similar with the graph-level SAD-Metric, this node-level disentanglement metric evaluate two properties of node embeddings, i.e. independence and interpretability (detailed definitions referring to Sec. 3.3). In this case, after structure and attribute modification, the nodes of a graph are classified into three types: (1) the nodes whose one-hop structure is modified, (2) the nodes whose attribute is modified and (3) the nodes without one-hop structure or attribute modification. In practice, we first randomly drop 20% edges in a graph and then randomly modify the attributes of 20% of the rest nodes whose one-hop structure have not been modified. Similar as in SAD-Metric (Sec. 3.3), we employ the absolute difference between the node embeddings before and after the above modification to perform a three-way classification. Upon on such embedding difference, a
linear classifier is trained to classify which type of modification is performed on a node, and the prediction accuracy of such node classification task serves as the node-SAD-Metric.
E.2 EXPERIMENTAL RESULTS
Setups. As in Sec. 5.4, we use the MoleculeNet dataset (Wu et al., 2018) and the scaffold split scheme (Chen et al., 2012) for this experiment. Also, the settings of five studied models are identical with those in Sec. 5.4. A linear classifier is trained with the graphs in the training split and evaluated by the graphs in the test split. All results are averaged over five independent runs using different random seeds.
Results. Tab. 9 reports the structure-attribute disentanglement performance of five models in terms of node-SAD-Metric. The random-GCN baseline can poorly disentangle the structure and attribute information in node embedding, and GCN-baseline performs better by pre-training on graph classification task. Among three methods for disentangled representation learning, the Embed-SAD model achieves the best performance on seven of eight tasks.

Review summaries:
This paper presents a novel method called Embed-SAD (as well as Input-SAD) to learn graph/node representations to disentangle structure and attribute information. Input-SAD is a simple baseline that tries to get structure-attribute disentanglements by individually processing graph structures and node attributes. For structure, the original node attibutes are replaced by out-degrees only, and passed to GNNs, while for attibutes, the node attibutes are passed to fully-connected networks. Embed-SAD is a more elaborate method to disentangle the GNN embeddings by posing two types of additional losses, i.e., the edge-reconstruction loss for structures, and the Noise-Contrastive Estimation (NCE) loss to maximize the mutual information against the structure-encoding vectors, in addition to the original loss for supervision. The paper also develops an interesting evaluation metric called SAD-Metric where node attibutes or graph structures are exclusively perturbed for each graph, and prediction for whether that perturbation is for structure or for attibutes made by the element-wise absolute differences between embedded vectors before and after the perturbation. This SAD-Metric can quantify the extent to which the obtained representation can detect which perturbation, that for structures or that for attibutes, is made for each sample graph. The experimental results also demonstrated that the structure-attibute disentanglement by Embed-SAD learning strategy actually improved the prediction performance of many off-the-shelf GNNs over many different graph- or node-level tasks.

This paper focuses on disentangling embeddings of the structure and the attribute of graph. The authors' key idea is that the structure and attribute information should be split in GNN. Based on this, the authors try to disentangle the structure embedding and the attribute embedding. With two different components, two different kinds of embeddings can be captured at the input stage.   In addition, these two different kinds of embeddings can be obtained by reconstructing the edge and minimizing the mutual information. At last, the authors propose a metric to evaluate the disentanglement. The models in this paper outperform baselines in node classification and graph classification task.