GL-Disen: Global-Local disentanglement for unsupervised learning of graph-level representations
Authors: Thilini Cooray
Ngai-man Cheung
Wei Lu
Venue:  (2021)
Keywords: Unsupervised Graph Representations
Disentanglement Learning
GNN
Unsupervised Learning

Abstract: Graph-level representation learning plays a crucial role in a variety of tasks such as molecular property prediction and community analysis. Currently, several models based on mutual information maximization have shown strong performance on the task of unsupervised graph representation learning. In this paper, instead, we consider a disentanglement approach to learn graph-level representations in the unsupervised setting. Our work is the first to study disentanglement learning for graph-level representations. Our key observation is that the formation of many real-world graphs is a complex process with global and  local generative factors. We hypothesize that disentangled representations which capture these global and local generative factors into independent latent units can be highly beneficial. Specifically, for graph-level representation learning, our disentanglement approach can alleviate distraction due to local variations of individual nodes or individual local neighbourhoods. We propose a VAE based learning algorithm to disentangle the global graph-level information, which is common across the entire graph, and local patch-level information, which varies across individual patches (the local subgraphs centered around the nodes). Through extensive experiments and analysis, we show that our method achieves the state-of-the-art performance on the task of unsupervised graph representation learning.


Under review as a conference paper at ICLR 2021
GL-DISEN: GLOBAL-LOCAL DISENTANGLEMENT FOR UNSUPERVISED LEARNING OF GRAPH-LEVEL REPRE- SENTATIONS
Anonymous authors Paper under houble-blind review
ABSTRACT
Graph-level representation learning plays crucial role in variety of tasks such molecular property prediction and community analysis. Currently, several mod- els based on mutual information maximization have shown strong performance on the task of unsupervised graph representation learning. In this paper, instead, we consider a disentanglement approach to learn graph-level representations the unsupervised setting. Our work is the first to study disentanglement learn- ing for graph-level representations. Our key observation that the formation of many woal-world graphs is complex process with global and local generative fac- tors. We hypothesize that disentangled representations which capture these global and local generative factors into independent latent units can be highly benefi- cial. Specifically, for graph-level representation learning, our disentanglement approach can alleviate distraction due to local variations of individual nodes individual local neighbourhoods, We propose a VAE based learning algorithm to disentangle the global graph-level information, which is common across the entire graph, and local patch-level information, which varies across individual patches (the local subgraphs centered around the nodes). Through extensive experiments and analysis, we show that our method achieves the state-of- th--tr-rrttttt performance on the task of unsupervised graph representation learning.
INTRODUCTION
Graph structured data has been very useful in representing a variety of data types including social networks (Newman & Girvan 2004), protein-protein interactions Krogan et al. (206), scene graphs (Krishna et al.) 2016), customer purchasing patterns (Bhatia et al., 2016) and many more. Graph Neural Networks (GNNs) have recently become the prominent approach for representing graph structured data (Li et al. 2016 Gilmer et al. (2017] Kipf & Welling 2017]] Velickovic et (018 Xu al. 2019). GNNs capable of representing graphs in a permutation invariant manner, enabling information propagation among neighbours and mapping graphs to low dimensional spaces.
In this work, we focus on graph-level representation learning. Graph-level representation learning is crucial for tasks like molecular property identification (Duvenaud et al.) 2015)) and community classification based on the patterns of discussion threads (Yanardag & Vishwanathan 2013), and they are useful for applications such as drug discovery and recommendation systems. Availability of task specific labels plays significant role in graph representation learning as much as its role in other domains such images, text and speech. However, due to many specialized fields which graphs are utilized (e.g.. biological sciences, quantum mechanics), collecting labels has become very expensive as it needs expert knowledge (Sun et al. 2020). Therefore, unsupervised learning of graph representation is crucial.
Recent state-of-the-arr art unsupervised graph representation learning methods (un et al. 2020 Has- sani & Khasahmadi 2020) are based on Infomax principle by Linsker (198)). These methods learn the graph representation by maximizing the mutual information between the representation of the entire graph and the representations of individual patches of the graph. Here we follow (Velickovid
al. 2019: Sun et al. 2020) and define patches local subgraphs centered around node. This

---

Under review as a conference paper at ICLR 2021
approach allows the graph level representation to capture the globally relevant information from the patch representations (Sun et al., 2020).
Global-local disentanglement. We propose novel approach for graph-level representation learn- ing. Our observation is that many graphs are generated using multplee heterogeneous factors, with different factors providing different information. Specifically, the formation of many real-world graphs is driven by graph-level factors and node/patch-leve factors. For example, an online dis- cussion thread can be represented a graph where nodes represent users who have participated in the discussion thread, and edges represent interaction between the users in the thread (Yanardag & Vishwanathan, 2013). Graph-level representation of such communication graphs can be used to classify the sub sub-community (e.g. subreddits on Reddit) of that discussion thread. However, the for- mation of these communication graphs driven by global graph-level factors (e.g., the topic of the discussion-then -hhrrad) and local node-level factors (e.g., characteristies of individual users engaging in on-line discussion). The graph formed with complex process involving complicated interactions between global graph-level factors and local node/patch- levell factors.
has been discussed in the literature that disentangling these generative factors can benefit many tasks in different domains (Bengio et al. 2013) Ridgeway 2016). This is because disentanglement enables to separate out explanatory generative factors which cause variations in data and facilitates selection of only those factors which are well suited for the downstream task. Importantly, removing the irrelevant factors from the prediction process increases the robustness of models (Ma et al, 2019).
Based on the above discussion, we hypothesize that graph representation learning that disentangles the graph-level and node/patch-level factors can be useful for many graph analysis tasks. In particu- lar, the disentangled graph-level representation can be powerful for graph-level inference. Therefore, in this work, we propose GL-Disen: global graph level local node/patch level disentanglement method for graph level representation learning.
Disentanglement learning a novel direction for GNNs, and it has not been studied for graph level representations learning. Existing work have only focused on disentangling the factors which forms each neighbourhood, based on supervision from downstream tasks (Ma et al. 2019 Liu al. 2020
Yang et al. (020) and disentangling node and edge features in attributed graphs (Guo et al. 2020
To summarize, our contributions are:
We propose GL-Disen: novel global graph-level and local node/patch-level disentangling model. To the best of our knowledge, this the first work of applying unsupervised disen- tangled learning for graph level representation learning.
We conduct extensive experiments to verify that our model learns meaningful disentan- gled global and local representations for graphs. The disentangled global representation achieves outstanding performance in graph classification.
2 RELATED WORK
Here we review the most relevant work on unsupervised graph level representation learning to ours.
Reviews of disentangle learning and other unsupervised graph learning methods are in Appendix [A]
The most recent family of graph embedding methods are based on contrastive learning. Main idea is to train an encoder model make it learn the contrast in between representation which captures the structural and statistic information provided by original data and a negative sample. InfoGraph by Sun al. (2020) was the first graph level embedding model which utilized contrastive learning and this method was inspired by Infomax principle based Deep Graph Infomax (DGI) (Velickovic et al., 2019.. It draws negative samples from other graphs and sum pooling used as the readout func- tion. Multi-view contrastive (CMV) learning method by Hassanii & Khasahmadij (2020) enhances InfoGraph by introducing multi-view based data augmentation mechanism which uses contrastive learning to maximize mutual information among multiple structural views of the input graph. On the other hand, Graph Contrastive Coding (GCC) (Qiu et al. 2020) utilizes contrastive learning for learning universal graph embeddings which can be transferred to multiple downstream tasks. Info- max principle based and contrastive learning based methods have produced the best performance for graph embedding models far.
2

---

Under review as a conference paper at ICLR 2021
3 GL-DISEN METHODOLOGY
3.1 GRAPH GENERATION PROCESS
We follow the general framework of (Higgins et al. 2017). However, our specific disentanglement method is quite different, as will be discussed. Let D = {G. G, be the set that consists of graphs and their ground truth generative factors for global and local level. Each graph G = (V, A), contains a set of nodes V and A is the adjacency matrix. Gj and L represent two sets of generative factors: G contains global factors gr C G common for the entire graph and Ip C L represents local factors which can differ from local patch to patch. In our model, gr and It are conditionally independent given G, where p(gr, lr|G) = p(gr G)) p(lr]G). We assume that the graph G is generated using true world generator which uses the ground truth generative factors: pGlgg 1r) = Gen(gr, Ir).
3.2
GLOBAL GRAPH LEVEL AND LOCAL PATCH LEVEL DISENTANGLEMENN
Our goal is to develop an unsupervised deep graph generative model which can learn the joint distribution of graph G, the set of generative factors Z, using only the samples from G. This should be learnt in a way that the set of latent generative factors can generate the observed graph G, such that p(G = p(G|gr, Ip) Gen (rr lr). A suitable approach to fulfill this objective is maximize the marginal log-likelihood for the observed graph G over the whole distribution of latent factors Z.
max E pu (z) (G|Z))
(1)
For an observed graph G. the inferred posterior probability distribution of the latent factors Z can be described as qu G). However, the graph generation process we described in Section 3.1 assumes two independent sets of generative factors representing global and local level information relevant for a graph. Therefore we consider model where the latent factor set Z can be divided into two independent latent factor sets as Z (Zg: Zr). Zg represents the latent factors which capture the global generative factors of G and Z captures the local counterpart. Therefore we can rewrite our inferred posterior distribution as follows:
g (Z[G) (o Ze G) go(Zg|G)gg
We discuss in detail the two posteriors: 4s (ZylG) and qø (Z G). The graph G consists of [V] number of nodes. In a graph data structure, each node not isolated. They are connected with its neighbours and propagates information. Therefore, we use the term patch to indicate the local neighbourhood centered at each node where the node interacts with. Therefore, go (Zg and (Z G') are the posterior distributions of all these [V] patches. However, if we consider the global latent posterior, is common for all [V] patches, as the graph G was originally generated with gr common for all V. Hence, propose to use single latent Zg to capture the global generative factors common for all patches. In particular, we use go (2gg()) to model this single posterior. On the other hand, the factors which contribute generate each patch can vary significantly. Therefore in this model we assume the local latent factors are independent Therefore, we update Eq. 2 as:
[V]
g (Z = 96 (Zg. Z/GG) (ZglGG) [Z
(3)
Here 21(1) is the latent factor that captures the local generative factors for a patch centered at node Now, our objective is make sure the latent factors sampled from global and local latent pos- terior distributions can capture the global and local generative factors g, and I, respectively in a disentangled manner. Note that, we aim to only disentangle global latent factors from local latent factors in this work. This is because, since the intention of the global latent Z, is to capture all the global factors for graph level representation, entanglement among individual factors in either global latent or local latent is not being focused. Thus, this is different from (Higgins et al. 2017). To
In the evaluation assess the validity of this assumption.
3

---

Under review as a conference paper at ICLR 2021
enforce the disentangling nature between these two latent factors, first, try match each of them to their respective priors p(zy) and p(zi) separately. We select unit Gaussians (N 1)) as priors. This leads to following constrained optimization problem (Higgins et al. 2017)
max EGNG 2, ( (Glz:g Zril
s.t. KL(go(Zg) ()) p(Zz)) <e (Z) p(Z)) <9
(4)
where and 1 are strengths of each constraint. Following (Higgins al. 2017). Eq. can be written
to obtain the variational evidence lower bound (ELBO) of Graph Variational Autoencoder (GVAE)
(Kipf Welling 2016) (Here we call this as GVAE because our input a graph) with two separate latent representations with additional coefficients as follows:
F(0, d, a; 76 G. Zi) 2 C(O, d; G, 2g. Z,, 0,
(u 2,10) loo pa 2n. Zi)l
(5)
(L p(z))) KT((s (Z G) p(z))
we can expand the KL. divergence term [L (Z G) || p(z)) and rewrite our objective function for single graph G as:
Based on Eq 3
C(,, d: G, Zi 7) = E, Dog Z,)
a K (2
(6)
(z(gs(zz()))) p(z(()))
The training process maximizes this lower bound for all the graphs in a minibatch G, from the full dataset G:
[Co
1
Lo.0 (G.) =
C(O, 6; Zz. Zi, 7,
(7)
G
3.3 GL-DISEN ARCHITECTURE IN DETAIL
Figure | depicts the proposed GL-Disen model. This is a variation of GVAE where we utilize N-layer GNN the encoder. th layer of GNN be defined in general as
(((1)
(1--))) (n evu : N(1) :ex))=u (()))
h. (m) (OMBINE (n-1,, (h.-1), AGGREGATEE
where (T) the feature vector of patch centered at node E V at the th layer after propagating information from its neighbours u E [Nv).. e is the feature vector of the edge between u and where (v, u) A. h!! is often initialized with node features. We use the term GNN indicate any network which use layers described in Eq. g
Then utilize tions for global and local level factors. Let H (h. (N) i be the output of GNN after N layers. Hg = GNN (H A) and log a, = GNN ( A) is used to generate pa- rameters for the Gaussian posterior distributions for global level latent factors; qa (z))()))) (HH, (i), diag(a (i)), Vi e {1... [1... Same way we obtain parameters for the local level latent posterior distributions; (s()(())))
separate of GNN layers produce parameters for posterior distribu-
(i)
Vi (I...IV]) where
4

---

Under review as a conference paper at ICLR 2021
V12)
ACCUM
(
Input graph: (G- (V))))
Local posterions
gi2(()
q(2/(())
Sampling latent factors from
posteriors
GNN Encoder
700 - 9 q (TG)
(2) (
7(3) (()
Fg
2g
Decoder
Reconstriited
Gruph (G'
Figure 1: GL-Disen architecture: Given input graph G, first send it through GNN ob- tain individual global and local latent posterior distributions (Ex: g(zg (1) G), (Z)())) for each patch. Then all individual global posterior distributions g Gi) are sent through ACCUM pro- cess to accumulate into a single posterior g(zz/G) for the global latent, using the procedure defined 9 Sampled local latent factors from their respective local posteriors are combined with the global latent Zg. and this becomes the input to the decoder to reconstruct the graph per Eq 10 Overall model is trained by maximizing the objective function Eg
μ4 = GNN p. (H-V A) and log oj GNN (H A). By utilizing GNN layers for this purpose, GL-Disen allows each patch centered at node to disentangle and propagate only the disentangled information to obtain a better posterior distribution for each type of latent.
distributions for both global and local latents (z) and (z) G), Vi E {1... (VI). we utilize the accumulation step proposed by [Bouchacourt et al. (2018) We model the the single posterior distribution (z(zg as the product of posteriors ( G), E [V]. Then, the distribution parameters of (2, G))
After obtaining posterior
calculated follows (Bouchacourt
2018):
IVI
V Σ 1=1
(1), 1
pg (1) (1)
Then global and local latent generative factors are sampled from their respective posterior distribu- tions (Zg (o(Zg]G) and 2i(i (z) Vi E (1.../V|) and sent through decoder for reconstructing the graph G. Note that the global latent factor 2g is only sampled once for the en- tire graph using go GT). We select simple 2-layer feed-forward neural network with non-linear
activations as our decoder D for the experiments.
graph reconstruction can be done via both node reconstruction adjacency matrix reconstruction. Hence we can rewrite the log-likellne likelihoodd of generating G (first term of Eq. 6 as the combination of generating V and A as follows:
2() (x,
log po (Vil2g, (1))
(10)
M
v
+ (x,, (.).
log pe (Aij Zg. 21(),, z(())
After training GL-Disen in an unsupervised manner, we utilize the latent global z, ~ go (z G) as the representation which summarizes graph G in downstream tasks such graph classification.
Difference between GL-Disen and 3- VAE (Higgins et al. 2017). While our work is largely in- spired by 3-VAE and we apply 3-VAR ideas to graphs, would like highlight one key difference
5

---

Under review as a conference paper at ICLR 2021
which is critical for this whole work. In particular, 3- VAE baseline camnot discover global factors tomatically: 3- B-V discovers independent latent factors, but it is not possible for baseline 3- B-V to understand if these learned factors global / non-global in the unsupervised setting. Usually, some manual inspection performed on the learned latent variables. E.g., for images, one needs to perform traversal of individual latent variables one by one, and observe their effects (e.g., change in azimuth). See Figure 2 of (Higgins et al. 2017). However, as many graphs represent very special- ized knowledge, e.g. protein-protein interaction, it is difficult understand the observed effects and what factors are global/local. Instead, in our work, we add on top of 3- BAV accumulation step for the GNN encoder outputs of vertices belonging the same graph (see Fig This forces the emergence of the global factors common information across all the patches. This mechanism is critical for our idea extract representation for the whole graph, and we are able to capture global factors without priori knowledge of the generative factors.
4 EXPERIMENTS
Here we discuss our main experiment and analysis results. Additional experiments and analysis are discussed in the Appendix.
Table 1: Mean 10-fold cross validation accuracy on graph classification. Results in bold indicate the best accuracy. Underlined results show the second best performances.
DATASET
MUTAG
PTC-MR
IMDE-BIN
IMDB-MUL
RED-BIN
RED-MUL-5U
node2vec
72.6 10.2
58.6 1 8.0
sub2vee
61.1 15.8
60.0
1.5
36.7 0.8
71.5 + 0.4
36.7 0.4
graph2vec
83.2 - 9.6
60.2
11.1
50.4 0.9
75.8 - 1.0
47.9 0.3
InfoGraph
89.0 = 1.1
1..7
0.9
49.7
82.5 1.4
53.5 = 1.0
CMV
89.7 + 1.1
1.7
74.2 0.7
51.2 0.5
44.5 0.6
GCC
72.0
49.4
89.8
$3.7
GVAE(baseline)
87.7 + 0.7
61.24
70.7 + 0.7
49.3
87.1
52.8 + 0.2
(ours)
90.7 = 0.6
67.9 = 0.8
74.7
= 0.6
52.1 = 0.4
90.9 + 0.3
54.9 - 0.1
4.1
QUANTITATIVE ANALYSIS ON GRAPH CLASSIFICATION
Firstly, evaluate the effectiveness of learnt disentangled graph level representations from GL-Disen model on downstream graph classification task.
4.1.1 BASELINES AND RESULTS
We compare our proposed GL-Disen with latest models unsupervised graph representation learning, which do not employ exhaustive feature selection (Ex: enumerating through paths sub- trees) or cand-crafted filters in this section (Refer Appendix [F] for a comparison with kernel meth- ods). They only node features and adjacency matrix GNNs propagate information and learn graph representations. We compare with all existing work to the best of our knowledge, which are; node2vec (Grover Leskoved 2016). sub2vec (Adhikari et (2018). graph2vec (Narayanan al, 2017). InfoGraph (Sun et al. 2020). CMV (Hassani & Khasahmadi 2020) and GCC (Qiu et al., 2020 We also GVAE (Kipf & Welling 2016) baseline indicate the performance of en- tangled representations. We use the same evaluation procedure followed by existing work (Yanardag
Vishwanathan 2015 Sun et al.) 2020; Hassani Khasahmadi] (0200) for a fair comparison. 10- fold cross validation accuracy is used to report the performance and the mean accuracy and standard variation of repeated runs used as the final result. Complete details of our experiment setup
included the Appendix
Performance comparison of GL-Disen with state-of-the-an unsupervised graph representation learn- ing models are reported in Table I We used 6 commonly used datasets; MUTAG, PTC-MR, IMDB- BINARY. IMDB-MU MULTI, REDDIT-BINARY and REDDIT-MULTI. Appendix Dccotainss complete details. For existing work, report results from previous papers. Our global graph level and local patch level disentanglement based GL-Disen model has been able to surpass all latest GNN based unsupervised methods. It important mention that, although CMV (Hassani & Khasahmadi

---

Under review as a conference paper at ICLR 2021
2020 uses additional diffusion matrix to expand the neighbourhood information propagation and GCC pretrains (Qiu et al., 2020) its models with larger datasets, our GL-Disen which only uses ad- jacency matrix without any pretraining has been able to surpass them. This shows the effectiveness of disentangling and removing non-global factors for global level downstream tasks.
4.2
QUALITATIVE ANALYSIS ON GL-DISEN
In order to qualitatively evaluate GL--Disen, we focus on two main aspects. First is, whether GL- Disen actually have the ability to disentangle global and local level factors into independent repre- sentations. Second is, whether the inferred global latents correspond to the fixed global generative factors of the graph.
50
14
82
(a) Disentanglment ability of GL-Disen GVAE
(b) Patch-wise MAPD for global(left) and local(right) factors
Figure 2: Analysis on GL-Disen: (a) Disentanglement ability of learnt global and local latent factors by GL-Disen via absolute values of correlation compared with GVAE, which does not perform global/local disentanglement. (b) Inter-patch MAPD among global and local latent factors. Lower MAPD for global factors indicates the global factor representations disentangled by GL-Disen is indeed shared among the entire graph, unlike local factors which are local to certain patches.
4.2.1 ARE THE LEARNT GLOBAL AND LOCAL LATENT FACTORS BY GL-DISEN DISENTANGLED?
To answer this question, we measure the similarity and dependence between global 2g and local 2 latent representations for a given graph G. Following Ma et al. (2019), we calculate correlation between disentangled global latent representation and local latent representations for graphs from MUTAG (Kriege & Mutzel, 2012) dataset and visualize in Figure aa. Following UDR, (Duan al. 2020), we use Spearman's correlation calculate the similarity/ correlation matrix. As the reference to demonstrate the difference, we use the output correlation for the same graph from non- disentangling model (GVAE) (we divide its single latent representation halves and consider as z, and local zi). Entry (i, j) of the correlation matrix indicate the absolute correlation value between zgG (1) and z (j). Note that, only for analysing purposes we sample z (zgg)()) (i)
The diagonal of the correlation matrix shows the correlation between global latent and local latent learnt by GL-Disen for each patch of the graph. We can observe that the correlation values di- agonal is very low, closed to 0.0 for GL-Disen, while correlations between local and global latent variables learned by GVAE have higher values. This shows that the global and local latent represen- tations output by GL--Disen is capable of likely capturing mutually exclusive information showing its disentangling ability.
4.2.2 ARE THE INFERRED GLOBAL LATENTS CORRESPOND TO THE GLOBAL GENERATIVE FACTORS OF THE GRAPH?
In order to verify the learned global latents Zg indeed map to the underlying global factors used for generating graphs, we consider the scenarios when global generative factors are unknown (real- world data) and known (synthetic data).
Real-world data based experiments. For real-world graphs, a priori knowledge of the genera- tive factors is usually not available (e.g. molecular graphs) However, following (Higgins et al. 2017). since global generative factors for all the patches of the same graph are fixed, we ex- pect latent variables corresponding global generative factors have small variance. There-
7

---

Under review as a conference paper at ICLR 2021
0.23
0.79
422
30
46
CM
180
Craph
Stughiid
(a) p
(b) p = 0.5
(c)
Figure 3: Variation of (Blue) and 2 (Orange) of each graph generated with different p values
fore, we check if global latent variables extracted from each patch should be similar to each- other than their local counterparts. Since multiple global generative factors may contribute for the graph generation and the information propagation is different from patch to patch, we cannot assume that the extracted global factors to be exactly the same. But they should be more similar than extracted local latent factors. We evaluate this using the Mean Absolute Pairwise Difference (MAPD) measure used by Higgins (2017) for the disentanglement metric proposed in 3 VAE. MAPD matrix for global latents is calculated as MAPD, = MEAN( (,(()) % [ ) and (1. MEAN (z( (i) ) used for local latents. We average all the dimen- sions of the latent.
Figure shows that the produced global level latent representations
similar each other for the entire graph compared to local level representations. Following the argument of Higgins et al.) (2117). if the representations are disentangled meaningfully. i.c. independent and interpretable, then, there can only be small variances in the inferred latents that correspond to the fixed generative factors. Under our model, the global factors for a given graph are fixed, therefore, we examine the variance of the inferred latents 2, (f). Vi (1... [V] for given graph (and we show that indeed these 2,(2) which correspond the fixed global factors of the given graph has small variance).
0.8
0.5
2.14
-1.50
1299
-121
0,579
(a) Impact on p gen with the increase of ng
(b) Visualization of generated graphs with the in- crease of Zg
Figure 4: Impact analysis of global latent representation zy the generation process of GL-Disen and recovering the global generative factor (a) plots how the distribution of the edge density probability p gen (the recovered p) changes with the increase of z, value. (b) visualizes generated graphs where in each row local latent representation Zy is fixed and in each column Zg is fixed. This shows that 2g has strong negative correlation with the global generative factor.
Synthetic graph based experiments we utilized a synthetic dataset to evaluate this scenario. Synthetic graph dataset was generated using Erdos-Renyi (ER) mode Erdos Renyi (1660). The ER(n, p) graphs are synthetic graphs with two global generative factors: number of nodes and parameter p E [o, 1] for the synthetic graph to include an edge (i,j) for < i < S n with prob- ability p. There is randomness in each generated graph for single value due to many different edge combinations can represent p. In our experiments, we focus on parameter p. as n is too easy to learn. Therefore,
create training dataset of ER(n, p) with fixed generated 4000 graphs in our dataset where 3000 used for training and the remaining 1000 for testing in which adjacency matrix is the only input for GL-Disen. We use simple 2 layer GL--Disen
50 and varying We
8

---

Under review as a conference paper at ICLR 2021
model with hidden dimension size 2 and dimensions of 2g and z are 1. We analyze two aspects from this experiment.
First is, whether our 2, corresponds to the fixed global generative factor p while z is not. To achieve this, we analyzed how 2y and 2; vary with fixed p. Figure 3 shows seatter plots for 3 different values of p. From the testing set, we selected the set of graphs generated using the given p and we sampled 2g as the global latent representations for each graph. Then we calculated an accumulated single local latent representation (2/) for each graph using 21 on Eq to observe local latent factor variation with p. We plotted the values of 2g and z for each graph for the given in Figure 3 We can see that while values of bg are scattered within a very small range (almost similar to a constant) when the global generative factor p is fixed, z has varied lot. This shows us that indeed the global latent representation 2 in GL-DDseen has been able to map accurately to the global generative factor while the local latent representations have not. Next, we calculated the Pearson r correlation as well as co-variance to understand how 2g changes with p. We obtained that 2 has very strong negative correlation with p of value 0.93 and the co-variance is 0.248. We further confirm this from the generative process of GL-DDseen in Fig. & T0 where we qualitatively showcase how global and local latent factors effect the graph generation process of GL-Disen.
4.3
CAN NODE LEVEL TASKS BENEFIT FROM GL-DISEN TOO?
In this section we evaluate the impact of disentanglement on node level tasks see whether node level tasks only benefit from local information or not.
Table 2: Mean test set accuracy on node classification. Results in bold indicate the best accuracy for each dataset. Value in brackets in the last row indicates the value which gave the best performance.
DATASET
CORA
CITESEER
PUBMED
GVAE (Kipf & Welling 2016)
77.8 1 0.4
60
61.5 0.4 33.7 + 10.9
Disentangled Graph only
14.4 + 8.7
15.7 + 4.7
Disentangled Node only
75.5 + 0.6
62.2 + 0.4
72.4 + 0.3
Combined
78.8 + 0.4 (1)
0.4) 67.8 + 0.2 (1 = 0.45)
77.4 = 0.4 (1) = 0.9)
To find this out, we updated our GL--Disen decoder D, with a gating mechanism, where our gate value 77 determine the contribution from local patch representation (i). combined input of 2; (i)
z: (i) + (1 - .7) z, is used as input to decoder D during training time and also used as the input to our node level downstream--tsk node classification. GVAB (Kipf & WWeling 2016) which produces entangled representations is out baseline. We use the citation network benckmark (Cora, CiteSeer and PubMed) BarMMMed)Yynn&ddda et al. (2016) for our experiments. Table the performance of GL-Disen on node-level task when only graph level information is utilized ("Graph only" + rj = 0), when only patch information is used ("Node only" + 1j = 1) and the best performance obtained by combining I contains detailed discussion). We can observe that for all datasets, combining
both (Appendix both information has been beneficial over using only patch level information. Donnat et al. (2018) has stated that identifying distant nodes with similar neighbourhood structures is a strong fact for node level task performance and we believe our combined method increases the performance due to the fact that Zy (the disentangled global representation) has been able to capture those kind of similarities across the entire graph compared to "Disentangled node only" model which solely relies on its local neighbourhood similarities.
5 CONCLUSIONS
We introduced a disentanglement learning based approach for unsupervised graph level represen- tation learning. Our assumption was that disentangling global level information shared among the entire graph, from local level information which are unique for patches, is beneficial for graph level tasks as it removes irrelevant information which can act as noise. We proposed VAE based GL-Disen for this purpose. From both quantitative and qualitative empirical results, we showcased the validity of our assumption and the effectiveness of our model. We achieve new state-of the-art performance for unsupervised graph representation learning task.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B. Aditya Prakash. Sub2vec: Feature learn- ing for subgraphs. In Advances in Knowledge Discovery and Data Mining 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC. Australia, June 3-6, 2018, Proceedings, Part II, URL https:://///o// http:///oooor////////o
doi: 00.1007/978-3-31--99000 pp. 170 182, 2018.
Yunsheng Bai, Hao Ding, Yang Qiao, Agustin Marinovic, Ken Gu, Ting Chen, Yizhou Sun, and Wei Wang. Unsupervised inductive graph-level representation learning via graph-graph proximity, In Proceedings of the Twenty-Eighuh International Joint Conference doi: 10.24963/ijcai 2019, Macao, China, August 10-16, 2019, pp. 1988-1994, 1994, 2019. URL
Artificial Intelligence, IJCAI
24963/ fca.: 20199/ 2019/275
Baldi and K. Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2:53-58, 1989.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Do. Intell., (5(8): 1828-1888, 2013. doi: 10.1 1091 TPAMI.2013.00.
https: org/ /0. 11 TPAMI PWW.PPAAM.......... 2013.50
K. Bhatia, K. Dahiya, H. Jain, A. Mittal, Y. Prabhu, and M. Varma. The extreme classificattinn repos- itory: Multi-label datasets and code, 2016. URL http: htt////annnaaaaaaaaaaaaaaaaaaaaa wwwpd/owwwddds/////oooooooooo/s/
XC/XM XC/XMMeepos itory............... .etooyytyllll...................
K. M. Borgwardt and H. P. Kriegel. Shortest-path kernels on graphs. In Fifth IEEE International Conference on Data Mining ((CDM)05), pp. pp--, 2005.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence. (AAAI-18), the 30th innovative Applications of Artificial Intelligence (1)AI-18), and the 8th AAAI Sympoxium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp- 2095-2102, 2018, URL https:/ /www aaai, org/ ocs/ index. php/ AAAI/AAAIIII AAAI18/
paper/view 16521
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in f3-vae. 04 2018.
Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Trans. doi: 0..1445/1661189116611
Intell. Syst. Technol. 23):27::-27:277, 2011.
URL https:
7/
1961199
Dexiong Chen, Laurent Jacob, and Julien Mairal. Convolutional kemel networks for graph- structured data. CoRR, abs/2003.05 189, 2020. URL
051 89
Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders.
In Advances in Neu- ral Information Processing Systems 31: Annual Conference
Neural Informa- 'e
tion Processing Systems 2018, NewrIPS 2018, 3-8 December 2018, Monir
al, Canada, pp. 2615-2625, 2018. URL http://papers. mips.pppp... wttppppc.ccccc/////cccppppp
7527-isolating htt.::oolaaaigggssrr-oe--o---o wwu.ces-uuesoooooo of
Guillaume Desjardins, Aaron C. Courville, and Yoshua Bengio. Disentangling facta of variation via generative entangling. CoRR, abs/ / 1210.5474, 2012. URL bttp://arxiv.orq//a
121 0. 5474.
Claire Donnat, Marinka Zitnik, David Hallac, and Jure Leskovec. Learning structural node embed- dings via diffusion wavelets. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining. KDD 2018, London, UK, August 19-23, 2018 pp. 1320 1329, 2018 doi: 10.1145/3219819932200 321 981 3220025
URL hattps://de http:////...011...1111111 1145/
10

---

Under review as a conference paper at ICLR 2021
Sunny Duan, Loic Matthey, Andre Saraiva, Nick Watters, Christopher Burgess, Alexander Lerch- ner, and Irina Higgins. Unsupervised model selection for variational disentangled representa- tion learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. URL BRlLhhtps://// https:///ppeeevvieieee ew. net./ / forum? d= SyxL2TNEvr
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learn- ing molecular fingerprints. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 2224 2232. Curran Associates, Inc., 2015. URL http: htttp::/papess.niitt wip..cccppp..........cccccppppp. pdf
Paul Erdos and Alfred Renyi. On the evolution of random graphs. Publ, Math. Inst. Hungary, Acad. Sci., 5:17-61, 1960.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. CoRR, abs/1903.02422 2019. UtLtttp:://arxi org/oroggss..o. /1tp/1933.0 02428
Thomas Gärtner, Peter Flach, and Stefan Wrobel. On graph kernels: Hardness results and efficient alternatives. In Computational Learning Theory and Kernel Machines, 16th Annual Conference on Computational Learning Theory and 7th Kernel Workshop, COI.T/Kernel 2003, Washington, DC, USA, August 24-27, 2003, Proceedings, pp. 129 143, 2003. doi: 10. 1001 URL https: /tts://doos wtw.:o/org/1..10101000000001 97 540
978-3- 778-3-540-0 540-54006---991000
Justin Gilmer, Samuel S. Schoenholz, Patrick E. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1263-1272, 2017. URL UtLhttp: /tLptttp://prrcceddingg mlr www.ppress//vi
wtw.:/pp.esss//ee/eeeeeoooo wmer7///aammrrrraaammmmmm.aaaaaa html
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, pp. 249-256, 2010. CRLLAtttp:/ppoceeeeee
mlr prwps/ressssspcc wttpe/epsssss//.se.tte.vtv htmll
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Min- ing. San Francisco, CA, USA, August 13-17, 2016, pp. 855 864, 2016. doi: 10.1145/2939672. 2939754. BRRLhtpps:..
2t..o..000/...111999999999222 29397554
Xiaojie Guo, Liang Zhao, Zhao Qin, Lingfei Wu, Amarda Shehu, and Yanfang Ye. Interpretable deep graph generation with node-edge so-disentanglemmnt. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23- 27, 2020, pp. 1697-1707, 2020. URL J http:: acm. orr.arr......................... httg://oo/11....11.. /11///...1111339448888 3403221
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. Proceedings of International Conference on Muchine Learning, pp. 3451-3461. 2020.
Irina Higgins, Lo " e Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In 5th International Conference on Learning Representa- tions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017. URL https: wpw.reviee..eeeeeeeemee net forum www.ooondd-yyttuuaee..
Geoffrey E. Hinton and Richard S. Zemel. scription length and helmholtz free energy.
Autoencoders, minimum de- In Advances in Neural In- formation Processing Systems 6. [7th NIPS Conference. Denver. Colorado, USA, 1993]. pp. 3-10. 1993.
http://apprr.nnips.cccp/ppaaaapp URL .
798 Sos-aauuonncoddrs--ee nimmmiiiumm---mmmmmm descri haw.:aaddeerrrptttoooaaoaaaaaaaa htw.acppp.....o.ooooaaaaaaaaaaaa

---

Under review as a conference paper at ICLR 2021
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the 35th Interna- tional Conference on Machine Learning, ICML 2018, Stockholmsm "a ssan, Stockholm, Sweden, July 10-15, 2018, pp. 2663-2663, 2018. URL http: http::/pprocecingg. .ww.ingg...rr...ommm. press/v8C/ /ttp
kiml 8b ht.l
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL htllttttt//axv..coom abtp///....11.1./.// 6980
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Truck Pruceedings, 2014. URL httphhttp///axii http:///axvvv..o.oe.o abb/ / 1312 6114
Thomas N. Kipf and Max Welling. Variational graph auto-encoders. CoRR, URL http://arxiv org abs/1611 07308
2016.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France,
April 24-26, 2017, Conference Track Proceedings, 2017. URL
forum?ld=S forum?1G=SJU4ayyg]
Risi Kondor and Horace Pan. The multiscale laplacian graph kernel. In Advances in Neural Infor- mation Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. -982-2999, 2016. URL/attp://papers https//cccpppaooeeaoaa--aamaaaaa Lawpelle--apaaaaaangaaaaea kernel
Nils M. Kriege and Petra Mutzel. Subgraph matching kernels for attributed graphs. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26- July 1, 2012, 2012.
(4appers//2..cdae pdf
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual Connecting language and vision using crowdsourced dense image annotations. 2016.
genome:
URL httts: org/ wwwpaaas......0.. 07332
Nevan Krogan, Gerard Cagney, Haiyuan Yu, Gouqing Zhong, Xinghua Guo, Alexandr Ignatchenko, Joyce Li, Shuye Pu, Nira Datta, Aaron Tikuisis, Thanuja Punna, José Peregrín-Alvaree Michael Shales, Xin Zhang, Mike Davey, Mark Robinson, Alberto Paccanaro, James Bray, Anthony She- ung. and Jack Greenblatt. Global landscape of protein complexes in the yeast saccharomyces cerevisiae. Nature, 440:637-43, 04 2006. doi: 10. 1010///auurr0046700.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track net worfooorumm//m www.mmmdddddkkkdeeeemmm
Proceedings, 2018. URL
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard S. Zemel. Gated graph sequence neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL pectp://arxie
org/ arg/abs/ 511 05493
Ralph Linsker. Self-organization in perceptual network. computer, 2103):105-117, 1988. doi: 1091/09/..36. http:httpp:::/o...rggg//
Yanbei Liu, Xiao Wang, Shu Wu, and Zhitao Xiao. Independence promoted graph disentangled networks. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Phirty-Secone Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-1 2020, pp. 4916-4919999 4923, 2020. URL htttt:t/////// wtw.aiaoo//o/////a/aaaaoaooao//a //nddxxx.
php/AAAT/ /
12

---

Under review as a conference paper at ICLR 2021
Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolutional networks. In Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 4212-4221, 2019. URL http:// proceedi ngs ml wwwpsrrsvvvv /mma 9a. html
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. CoRR,
2017. http: wtgg/bb//1717 05005
M. E. Newman and Girvan. Finding and evaluating community structure in networks. Phys. Rev. F, 69:026113 13, Feb 2004. doi: 10. 10.1103/PhyReevE.69.022
httpps://ing aps..
org/doi/10 1103 1wt.111//////e....... 69. 026113
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Deiito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. arXiv preprint vrXiv:2006.09963 2020.
Karl Ridgeway. A survey of inductive biases for factorial representation-lcariig abs/1612.05299, 2016. [Bll://:pp///xxx.voor//aaass 012.//2912000999999 05299
CoRR,
A. Sanfeliu and K. Fu. A distance measure between attributed relational graphs for pattern recog- nition. IEEE Transactions Systems, Man, and Cybernetics, SMCC13(3)3353-3623 1983. doi:
10.1109/SSMC..193..6633 19111391166.1
Nino Shervashidze, S. V. Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten M. Borgwardt. Efficient graphlet kernels for large graph comparison. Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, AISTATS 2009, Clearwater Beach, Florida, USA, April 16-18, 2009, pp. 488-495, 2009. URL http: http:::/ppocceeinggs. amw.inggssrrr....mmmm.m...m...mm mllr. press/ v5 ht/ahrrashidzee0a.hha html
Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M. Borgwardt. Weisfeiler-lehman graph kernels. J. Mach. Learn. Res., 12:2539 2011. URL
http::/dl. acm. owg/ waarg//itaae tatiiatiioo........ wfmpid=20700000
Fan- Yun Sun, Jordan Hoffmaan,, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. In 8th In- ternational Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
net wer./oruumteteeeee w
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li 'o, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URRLAttps://ppezeviiee
net wttp::eeeeeeotttttteeeeroe
Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL
forum?id=rkl2911i
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL pettps::/oppeneeeee 6y556iA5Km
Pinar Yanardag and S. V. N. Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, Sydney, NSW, Australia, August 10-13, 2015, pp. 1-355-1374, 2015. doi: 10.11457778325.27884411 URL
https://66f..0//100 .ww.koorg/o00 01t.//..1115288888 27834177
13

---

Under review as a conference paper at ICLR 2021
Yiding Yang, Zunlei Feng, Mingli Song. and Xinchao Wang. Factorizable graph convolutional networks. CoRR, abs/2010.05421, 2020. /ltt:pp://aaxi..oor//////////// 05421
Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learn- ing with graph embeddings. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pp. 40-48, 2016. URL
http://proceedings mlr. wwwss/v48/yaaeeeee
14

---

Under review as a conference paper at ICLR 2021
A RELATED WORK
A.1 UNSUPERVISED GRAPH LEVEL REPRESENTATION LEARNING
Kernel Methods Graph kernels are widely used for representing graph structured data over decades. Main idea of graph kernels is to first find out best sub-structures which the graphs can be divided into and then enumerate and count the occurrences of these sub-structures represent them as high dimentional feature vector. Most common substructures are walks (Gärtner et al.) 2003), shortest paths (Borgwardt Kriegel 2005), subtrees (Shervashidze et al. 2011), or graphlets (Shervashidze et al. 2009). Then the graph kernels are defined to calculate pairwise substructure similarity between two given graphs. Earlier graph kernels decoupled the process for kernel based data representations and task based model training into two parts. Therefore unlike GNNs, kernels can neither produce task dependent features, nor can train end to end. However kernels have more expressive power and regularize properly than GNNs. More recent kernel based models like GCKN (Chen et al. 2020) have tried to combine the best of both kernal and GNN worlds by extending convolution kernel networks. However, still these kernel based methods rely enumeration of the substructure occurrences in graphs, where they gain better expressive power at the cost of efficiency. Kernel based methods which use walk kernels are the most similar to GNN as GNN also uses walks for information propagation. Although doesn't It belong to the kernel category, the recent work by Bai al. (2019) also follows a pair-wiss wise graph similarity caluculation mechanism based on proximity calculations using Graph Edit Distance (Sanfeliu & Fu 1983) for unsupervised graph representation learning.
Other Methods Another set of methods were proposed inspired by the word2vec skip-gram model from natural language processing to encode neighbourhood information to a vectorized represen- tation and consider it as the graph embedding. First model was node2vec (Grover & Leskovec, 2016) and sub2vec (Adhikari et al., 2018) uses random walks to identify each node's neighbour- hood and encode that to a latent vector represent each node or sub- wub-graph respectively. graph2vec kernel (Shervashidze et al. 201 to calculate non- linear substructures opposed to linear sub-structures from sub2vec to vectorize full graphs. However these methods are completely dependent of neighbourhood information and unable to utilize node features.
Naraaanan et al., 2017) uses Weisfeiler-Lehman
A.2 DISENTANGLED REPRESENTATION LEARNING
Autoencoder (Baldi & Hornik 1989 Hinton & Zemel 1993) based approaches has proven to be
one of the most effective methods in representation leaning (Bengio et al., 2013) which provides compact and meaningful representations without any supervision. These learnt latent representations can be utilized for downstream tasks such as classification or clustering. Variational Autoencoders (Kingma & Welling 2014) further enhance these capabilities by supporting variational inference. Disentangled learning (Desjardins et al. (0022) is focused on learning the model to assign different factors used for object composition different dimension of the latent representation. VAEs were proven to be capable of disentangling the latent features after regularizing its objective function (Higgins et al. 2017 Burgess et al. 2018). Later more variations of VAE (Chen al. 2018 Kim & Mnih 2018 Kumar et al. 2018) have been proposed for better disentangled learning.
Another line of disentangled learning was proposed based on grouped observations by Bouchhccurrr et al. (2018) which makes the models leam the semantics of provided grouping. These methods aim at disentangling content (group) and style (individual element) level information where the content is common for the entire group while style is independent for each element in group. Group level acts as weak supervision where the disentanglement is only learnt based on the fact that all element belong to the same group without actually knowing the group label. They propose a multi- layer VAE (ML- VAE) to achieve this and they show the effectiveness of their method based on its generalizability unseen groups and controllability over latent space.
Inspired by this work, we propose to apply natural grouping among nodes (as they belong to the same graph via edges) to disentangle global graph level factors from local factors.
15

---

Under review as a conference paper at ICLR 2021
A.3 DISENTANGLED REPRESENTATION LEARNING FOR GNN
In this section, we discuss disentangled representation learning for GNN. We refer the reader to the Appendix for review of general disentangled representation learning.
Disentangled learning is a very novel direction for graph representation learning domain as only very few work are currently available. DisenGCN (Ma et 2019) was the first method proposed to disentangle the node representation which dynamically extracts each factor caused for each neigh- bour form an edge with current node. IPGDN (Liu al. 2020) proposes another disentangled node representation which focuses not only on disentangling factors which relates the node with its neighbours, but also on making those factors as independent as possible. Both of these methods are guided by the supervision of downstream node classification tasks. FactorGCN (Yang et al.] 2020) another supervised disentangling model which uses a factoring mechanism at the input level to disentangle input features which are propagate among neighbour nodes to separate factors and form them as separate graphs. Then they use a discriminator to make the factors as independent as possible. Then each of these graphs task supervision. NED-VAL VAE (Guo et al., 2020) an unsupervised disentangling model which dis- entangles node and edge features from attributed graphs.
separately sent through GNN and aggregated together.
Compared to these methods, we follow a different level of disentanglement in an unsupervised man- ner. Our GL-Disen focuses on disentangling global level factors common for the entire graph from local level factors specific to patches. We do not aim at disentangling individual factors in each level. All existing disentangle methods including DisenGCN(Ma et al. 2019), IPGDN et al., 2020). FactorGCN(Yann (ang et al.] (2020) and NED-VAE with all its variations (Guo et al. 2020) can be identified as disentangling individual factors local level as they either factorize the neighbour- hoods or disentangle node and edge features. They do not aim at disentangling and separating out factors common for the entire graph.
B PSEUDO CODE OF OUR PROPOSED GL-DISEN
Algorithm GL-Disen Algorithm
1: for Each epoch do
2: Sample a minibatch of graphs G5
3: for G E Gg do
4: procedure ENCODER(G)
5:
Encode G E (V, into (i),()) o())))) vi end procedure Accumulate q(2, (i) o), Vi (1... [V]
6:
7:
8:
Obtain a single global level latent posterior go (2, GG)
Eq.
9:
procedure DECODER(qs ((
vi e1. VV}
10:
Sample single global latent factor Zy ~ q (z, G) for the entire G
11:
Sample individual local latent factors for each substructure
12:
for i E {1 [V do 21) 2 (G)
13:
14:
end for
Decode zy and {z (
reconstruct the graph G using p(l/z (
16:
end procedure end for
18:
Update 0, & by taking gradient step of (o,)(Gs), Vo, C(G,, 0,
> Eq.
19:
end for
C MODEL. COMPLEXITY ANALYSIS
We like to discuss the time and space complexity of GL-Disen compared to our baseline GVAE Kipf & Welling 2016). Most of the computation complexity comes from the GNN encoder (Eq where the time and space complexity is ((V2) for a single GNN layer with number of nodes the graph and for GNN with N layers, it becomes O(V2N), Only difference between GL--Disen encoder and
16

---

Under review as a conference paper at ICLR 2021
GVAE encoder is that due disentangling, and GL-Disen encoder requires output of two different parameter sets for global and local posterior distributions instead of one as in baseline GVAE. There- fore we need an additional GNN layers. Since it is constant addition, the overall complexity stays at O(V2N) scale. The decoder complexity for both GVAF and GL--Disen is O(V2) with adjacency reconstruction being the dominant component TOP. The two additional steps GL--Disen have for the disentanglement are as follows (in between encoder and decoder): (A) Accumulating using Eq and (B) combining global and local samples feed to the decoder. Both step (A) and (B) are linear operations during both training and inference with the complexity of OVV) in both time and space. Compared to the high complexity of the GNN encoder and decoder common for both GVAE and ours, this linear increment to disentanglement is not significant.
D EVALUATION DATASETS
We select six commonly used graph classification benchmark datasets as follows: MUTAG (Kriege Mutzel 2012) dataset contains mutagenic aromatic and heteroaromatic nitro compounds while PTC dataset (Kieee & Mutzel 2012) consists of chemical compounds reported for carcinogenicity of rats. Apart from these biomformatics datasets, next we evaluate on four social network datasets Yanardag & Vishwanathan 2015) namely
REDDIT-BINARY and REDDIT-MULLT-SK IMDB datasets contain information about movies where the nodes are ac- tors/actresses and they are connected by edges if they have acted in the same movie. IMDB-BINARY contains two genres of movies: Action and Romance. Multi-class version of IMDB dataset contains movies from Comedy, Romance and Sci-Fi genres. Reddit datasets were created using threads in dif- ferent subreddits. Nodes in each graph are users who responded to that particular thread and edges formed when each graph as question/answer based community or a discussion-base community and REDDIT- MULTI-5K labels graphs into 5 labels according their subreddit namely, worldnews, videos, Ad- viceAnimals, aww and mildlyinteressingg Table contains statistics of all these datasets. Statistics of the benchmark datasets are in Table
respond to another user's comment. REDDIT-BINARY dataset labels
Table 3: Dataset statistics used for graph level tasks
DATASET
MUTAG
PTC-MR
IMDB-BIN
IMDB-MUL
RED-BIN
RED-MUL-SK
# Graphs
188
344
1000
1500
2000
4999
Avg. Nodes
17.93
14.29
19.77
13.0
429.63
508.52
Avg. Edges
19.79
14.69
96.53
65.94
497.75
594.87
Classes 2
2
For node classification task, we used the citation network datasets; Cora, CiteSeer and PubMed)
Yang et al. (2016). Statisties of the datasets are in Table
Table 4: Dataset statistics used for node level tasks
DATASET
CORA
CITESEER
Graphs
# Features
1433
3703
500
# Nodes
2485
2110
19717
# Edges
5069
3668
44324
Classes
E EXPERIMENT SETUP
We use the same evaluation procedure followed by existing work YYaaardag & Vishwanathan 2015
Sun al. 2020 Hassani & Khasahmadi] 2020) for fair comparison. 10-fold cross validation curacy is used to report the performance and the mean accuracy and standard variation of 5 repeated runs is used the final result. Mean Squared Error and Binary Cross Entropy losses are used to
17

---

Under review as a conference paper at ICLR 2021
calculate node and adjacency matrix reconstruction enror. After GL-Disen model is trained in un- supervised manner, the classification accuracies are calculated using LIBSVM (Chang & Lin 2011) which the C parameter training folds of data.
selected from {1, 10', 102 10 10*3 using cross-validation from the
Following InfoGraph (Sun et al. 2020), Graph Isomorphism Network (GIN) (Xu al. 2019) is used as the GNN encoder GL-Disenn. We initialize GL-DDisen model using Xavier initialization (Glorot & Bengio 2010) and train the model using Adam optimizer (Kingma Ba 2015). Initial learning rate, number of GNN layers and batch size were set to 0.001, 2 and 128 respectively. Number of training epochs and hidden dimension were chosen from {20, 30, 40, 50), {32, 128, 512]. Refer to the Appendix for the selected hyper-paune parameter sets for each dataset.
For node classification task, we follow DGI (Velickovic et 2019) and report the mean classifica- tion accuracy with standard deviation on the test nodes after 50 runs of training followed by linear model. We use a single layer GCN as the encoder and the hidden dimensions are 512.
Model implementations and data loading were done using Pytorch (Paszke et al. 2017) and Pytorch Geometric (Fey & Lenssen 2019) for our experiments.
F COMPARISON OF KERNEL BASED MODELS WITH GL-DISEN FOR GRAPH CLASSIFICATION
In this section compare the performance of GL-Disen with another line of unsupervised
graph representation learning; kernel based methods. We use 7 different models of kernel methods
for this; Random Walk (RW) (Gärtner et al. 2003), Shortest Path (SP) (Borgwardt & Kriegel 2005),
Graphlet Kernel (GK) (Shervashidze et al. 2009). Seisfeiler-Lehmane Subtree kernel (WL) (Sher-
vashidze al.) 201T), Deep Graph Kernels (DGK) (Yanardag & Vishwanathan 2015 Multi-Scale
Laplacian (MLG) (Kondor & Pan, 2016) and most recent Graph Convolutional Kernet Network
(GCKN) (Chen 20200. We only compare with GCKN walk kernel method as it is the closest feature aggregation for GNNs.
Table 5: Mean 10-fold cross validation accuracy comparison graph classification of Kernel meth- ods with GL-Disen. Results in bold indicate the best accuracy for each category. Underlined results show the second best performances.
DATASET
MUTAG
PIC-MR
IMDB-BIN
IMDE-MUL
RED-BIN
RED-MUL-5K
Kerne Methods
83.7 = 1.5
57.9 1.3
50.7 + 0.3
34.7 0.2
SP
85.2 - 2.4
58.2 2.4
55.6 0.2
38.0
+ 0.3
64.1
39.6 + 0,2
GK
81.7 - 2.1
+.4
65.9 1.0
43.9 0.4
+..
41.0 + 0.2
WL
80.7 3.0
0.5
72.3 3.4
68.8 1 0.4
46.1 + 0.2
DGK
87.4 2.7
60.1
6.6
44.6
78.0 0.4
41.3
0.2
MLG
87.9 - 1.6
63.3 1.5
66.6 0.3
41.2 0.0
GCKN-walk
92.8 = 6.1
65.9 + 2.0
75.9 + 3.7
53.4 4.7
Ours
GVAE(baselinc)
87.7 = 0.7
61.2. 1.8
70.7 = 0.7
49.3 0.4
87.1
= 0.1
52.8 = 0.2
GL-Disen(ours)
90.7 = 0.6
67.9 + 0.8
74.7 + 0.6
52.1 + 0.4
90.9 - 0.3
54.9 = 0.1
GCKN [Chen et al. 2020) reports superior performance to ours in 3 datasets although our results are very comparable to them. One of the major aspect of kernel methods is they use manual processes (graph traversals like depth first search) to find all possible paths for substructures like random walks, trees or graphlets. Then they compare all those pairs of paths in each pair of graphs to calculate kernel values to find similarities. This is a very expensive operation. However for small graphs this gives better results as it covers all possible neighbourhoods. However the GCKN (Chen et al. 2020 mentions, when there are very large dense graphs, they are unable to extend this method. This can be reason that kernel based methods do not evaluate on denser datasets like Reddit. On the other hand, GNNs achieve efficiency by eliminating from manual path and graph to graph pairwise comparison and reducing neighborhoods for only random walks. However even with
18

---

Under review as a conference paper at ICLR 2021
limited neighbourhood, could that GNNs specially with our disentanglement mechanism have been able to achieve almost similar performance.
G ADDITIONAL ANALYSIS TO PROVIDE MORE EVIDENCE THAT Zy IS ESSENTIAL TO CAPTURES GLOBAL LEVEL INFORMATION
Table [ shows that GL-Disen is able surpass all existing GNN based unsupervised methods for graph classification. However, important to compare the performance of disentangled global representation against disentangled local representations well entangled representations ver- ify that the most crucial information for downstream global level tasks (Ex: graph classification) are indeed captured by global latent representation Zg.
For this, the first experiment we conducted was to separately evaluate how the downstream graph classification accuracy gets effected when using disentangled latent global representations, disen- tangled latent local representations and entangled latent representations. We used the same config- urations as the original GL-Disen where 2g was sent to the SVM We the same accumulation function mention in Eq to to obtain the single posterior distribution and sample a representation for both disentangled latent local and entangled latent before feeding them to SVM. For obtaining en- tangled representations, we use a vanilla GVAE.
Next, we examine whether local factors are completely irrelevant for global level tasks. To analyse this, we feed a linear combination of latent global representation and latent local representation to the SVM classifier. This experiment was conducted using the best pre-trained models according to Section E representation. We use the following function to obtain the final graph representation Zg to be sent to the SVM. We use the same accumulation function mention in 9 for obtaining single posterior distribution to sample z from.
We set gating hyper-parametee À to determine the contribution comes from latent local
2g = zi + (1 - A) Zg
(1I)
0.91
global only
CL-Daun local only
Entiggled
0.90
global combined
Mean Accuscyy
0.89
0.88
0.87
Figure 5: Comparison of the impact of disentangled local only, disentangled global-local combined and entangled (GVAE) latent representations for graph classification performance against disentan- gled global only Zy representation. (best viewed in colour)
Figure 5 visualizes the impact of local latent factorsz, and entangled latent factors (latent repre- sentation from vanilla GVAE) against global latent factors Zg for graph classification task. Each dotted line indicates the accuracy value obtained when each of these latent representations are indi- vidually sent to SVM for graph classification. We can observe that global latent representation has achieved the best individual performance surpassing local only and entangled. GL-Disen local only performance has been very close to entangled performance obtained from GVAE. We believe this is because, although global information are included in GVAE's latent representation (in entangled manner), each observation is treated in an IID manner. Hence those global information might be getting suppressed in order give space for latent factors which enhances reconstruction ability of the model. Solid blue line shows how the graph classification accuracy changes with increasing X. We observe when the is increasing (more contribution coming from local latent representation),
19

---

Under review as a conference paper at ICLR 2021
the classification performance drops. Therefore we can conclude that the global latent 2g achieves the best performance. And there is no gain in incorporating disentangled local latent z except some random variations.
H How CAN THIS METHOD PROVE THAT EACH FACTOR NECESSARY FOR THE GENERATIVE PROCESS?
In what follows, we discuss how we validate that our learned local and global latent variables carry critical information for the generative process.
As we discussed, our main focus of GL--Disen is to separate out the group of generative factors as local and global, i.e. global/local disentanglement, and that is sufficient for our task. There is no need for us to explicitly separate each one of global and local factors from those sets individually.
To evaluate the necessity of our local/global latent variables, first we calculated the node feature reconstruction error for MUTAG dataset and obtained the following results. MSE when both global and local factors fed to the decoder is 0.03256 and it increases to 0.03654 when global factors removed (local only). When only global factors are fed to the decoder (global only), the error further increases 0.08329. From these errors, we can observe that local latent factors have the largest impact on the generation of the node features. This is expected as global factors are common for all the patches for given graph. Therefore, to reconstruct the node features (which differ from node to node), local factors are crucial. However, we can observe from the difference of full model and local only errors, that our model does not ignore the global factors during node feature generation. Hence showing it is also necessary.
Next, we show that our learned global latent variables carry critical graph level information in the generative process. We refer to Sec. 4.2.2 Synthetic graph based experiments and Fig. 4 on the updated manuscript.
Fig (b), we show generated sample graphs using disentangled global and local factors. In each row of Fig Fig b),, the local latent factors are fixed and in each column the global factors are fixed. When we consider a single row, we could observe that, the edge density of the graph changes with the change of global factors. Although 2 rows have two structurally different graphs (nodes have different neighbourhoods), the global factor has been able to change the edge density of those 2 in a similar manner. If only local factors are necessary, then every graph in the same row should look alike. This shows that graph level generative information is captured by global latent variables. Therefore the global latent variables are necessary for the generative process.
Further evidence that our learned global latent variables carry critical graph level information comes from the evaluation on graph classification task. In Appendix G we evaluate the impact of different combinations of global/local latent (Eq. [1] on graph level task performance. We observe that using only learned global latent variables (X- 0) achieves the best performance in graph level classifica- tion. On the other hand, when À = 1 in Eq 11 i.e., only local factors are used for graph classification, the performance drops significantly. This shows that global latent variables carry critical graph level information in the generative process. We remark that these global/local representations are learned unsupervised settings; then the representations are tested in SVM classifiers.
I How DOES NODE LEVEL TASK PERFORMANCE GETS EFFECTED WITH GLOBAL LEVEL INFORMATION?
As a detailed look at the Table we indicate in detail how the node level performance gets effected with all the values of 17 in Figure 6 and Table 6 for the Citeseer dataset. We like to elaborate that the input to the decoder during unsupervised model training (1) 2i( + (1 n)) 2g) is the same which goes to the SVM classifier during the inference and downstream task performance evaluation. Therefore, although we show the downstream task performance here, it has a strong relatability with the GL-Disen decoding process. We could see that when 7 = O, when only global level disentangled information is used for decoding process, GL-Disen is unable to learn local patch level disentangled information. Therefore during inference of GL-Disen and evaluation of downstream node classification, the performance is very low. This is expected as all nodes in the graph will have the same features, as the global latent representation common to all the nodes. But when disentangled local information start to get incorporated in the decoding process with the increase of
20

---

Under review as a conference paper at ICLR 2021
7) value, observe the increase of downstream task performance. But after 17 0.45, again the performance has started to degrade upto = 1 (when GL-Disen does not learn global graph level information). This shows that to obtain the best performance, both global and local disentangled information crucial for node level tasks and GL.-Disen provides the capability of controlling the contribution of each factor obtain the optimal performance.
70
60
Mean Accuracy
50
30
20
0.0
0.2
0.4
0.6
0.8
1.0
n
Figure 6: Impact Mean test set accuracy Node classification with different 7) CiteSeer.
Table 6: Mean test set accuracy change on Node classification with different on CiteSeer.
7
MEAN ACCURACY %
0.0
15.7
0.05
44.7
0.1
67.6
0.15
65.6
0.2
66.8
0.25
65.9
0,3
63.9
0.35
64.1
0.4
64.5
0.45
67.8
0.5
64.8
0.55
64.8
0.6
64.7
0.65
64.6
0.7
64.0
0.75
63.1
0.8
64.8
0.85
62.6
0.9
63.1
0.95
62.8
1.0
62.2
21

---

Under review as a conference paper at ICLR 2021
J ABLATION STUDY
J.1 IMPACT OF THE RATIO BETWEEN THE NUMBER OF FACTORS USED FOR GLOBAL AND LOCAL LATENT REPRESENTATIONS
In this study analyse the impact of the number of factors allocate for storing global latent Z and local latent (i) has on the downstream task. We consider the number factors for each latent the dimension of the vectors each of them kept in the GL-DDisen. Therefore when changing the number of factors allocated for each latent, we change the vector size. This is done as a ratio. For an example, if the ratio is 3:1, that means the dimensionality of 2g is three times the dimensionality of (i). When we input for the decoder also, we maintain this ratio.
Global to Local Ratio
Accuracy + S.D.
3:1
86.5 1 0.8
1:1
90.7 + 0.6
1:3
88.7 + 0.6
0:1
87.7 = 0.7
Above table shows the impact of the change of factor sizes for global and local latents have MUTAG dataset's classification accuracy. When both latents have same size, the size was 128 in this trained model. We observe that the accuracy drops when most of the information comes from global latent as the reconstruction error increases by being unable to reconstruct the graph in the decoder. When there is dimension for global, that is the vanilla GVAE where all information is entangled and considered local. This shows that allowing equal amount of factors is the beneficial for our GL-Disen.
J.2 IMPACT OF USING DEGREE FEATURES
Social network datasets (IMDB, REDDIT) we are using do not have input node features. Therefore existing work has taken two approaches provide synthetic node features. First is using stant vector and the second is to use node's degree count as a feature. Following table shows the performances of GL-Disen when different types of synthetic features are used.
Synthetic feature
IMDB-BINARY
IMDB-MULTI
REDDIT-BINARY 90.9
REDDIT-MULTISKK
Constant
72.5 + 0.3
48.3 + 0.3
54.9 0.1
Out degree count
74.7 0.6
52.1 + 0.4
85.5
52.4 = 0.2
We can observe that, only IMDB benefits from degree features while Reddit achieves best perfor- mance when constant feature is used.
J.3
IMPACT OF HIDDEN SIZE
Following table shows how hidden size effect the performance of GL-Disen.
Hidden size
MUTAG
PTC-MR
IMDB-BIN
IMDB-MUL
REDD-BIN
RED-MUL-SK
32
90.54 0.3
63.7 + 1.3
0.6
49.9 0.1
90.9 +
54.9 0.1
128
90.7 = 0.6
65.9 - 1.2
72.7 0.4
51.7 0.5
90.2 - 0.2
52.4
512
89.8 0.8
67.9 + 0.8
74.7 0.6
52.1 1 0.4
89.3 + 0.2
51.64 0.2
J.4 SELECTED FINAL SET OF INPPER-PARAMETES
Following has the final set of syper-parameters selected for reported results. We observed that reconstructing only node features was beneficial for MUTAG and PTC as they had node features. For other datasets, only reconstructed adjacency matrices.
22

---

Under review as a conference paper at ICLR 2021
Hidden size
MUTAG
PTC-MR
IMDB-BIN
IMDB-MULTI
RED-BIN
RED-MULTI-SK
a
le-6
le-9
le+3
10
let4
le+4
le-6
le-9
le43
10
le+4
le44
10
10
le44
1
le+3
le+3
epochs
20
30
40
30
50
50
Hidden size
128
512
512
512
32
32
Degree
No
No
Yes
Yes
No
No
Reconstruction
Node
Node
Adj
Adj
Adj
Adj
K MORE SAMPLE PLOTS FOR DISENTANGLEMENT ANALYSIS
K.1 SAMPLES FOR DISENTANGLEMEN BETWEEN GLOBAL AND LOCAL LATENT FACTORS SAMPLES FROM LEARNT POSTERIOR DISTRIBUTIONS BY GL-DISEN
Figure provides examples to showcase the ability of our proposed GL-Disen model's capa- bility of producing mutually exclusive information for its global and local latents. This shows that GL-Disen actually disentangle two latent factors.
K.2 SAMPLES FOR SHOWING THE SIMILARITY OF GLOBAL LATENT FACTORS ACROSS THE GRAPH COMPARED TO LOCAL LATENT FACTORS
In this section, we provide more qualitative proof that the latent factors our GL--Disen model maps as global are indeed globally common across the graph. Figure g and Figure 9 compares the pair- wise similarity of global latent factors and local latent factors separately in two matrices for multiple different graphs from MUTAG dataset. We can see that compared to local level inter-node latent rep- resentation similarity, global level similarity is very high as the Mean Absolute pairwise difference low.
L GENERATED DENSER GRAPHS FROM GL-DISEN FOR SYNTHETIC DATASET
In the main paper have only incorporated graph generated with very small probabilities as dense graphs are hard to visualize. In Figure 10 we have incorporated denser graphs.
M DETAILED COMPARISON OF GL-DISEN WITH SOME RELATED WORK
The graph-graph proximity (Bai et al., 2019 is not a disentanglement learning based method. It has proposed an unsupervised learning mechanism where they first obtain graph embedding using multihead attention to aggregate nodes in each layer and concatenate K layers together. The unsu- pervised loss they are using is based on graph proximity there they have to calculate the proximity measure for each and every pair of graphs in the dataset obtain the distance matrix( Fig 1.(b) of Bai et al. 2019)). Compared to this graph pair-wise comparison, GL.-Disen uses a disentanglement mechanism individually for each graph where forces the model make the global level informa- tion and local level information in the graph to be independent. This implicitly pushes the model towards learning similar global representations of graphs in the dataset. As shown Fig. 4 of our updated manuscript, even without this expensive pairwise comparison, global level similarities has been captured by our GL-Disen. In Fig. 4(b) we show that our global representation (zy) has been able to map different global generative factor values (global generative factor is the feature that sim- ilarize/group graphs in this dataset) to different latent values. Hence during inference it has been able to apply it back and increase the edge density of any random graph. Since they have randomly split the dataset for train, validation and test splits while we conduct cross validation, we are unable to compare the performance.
Apart from FactorGCN (Yang et al. 2020) being a supervised model (section 3.4) while GL-Disen unsupervised, there two main differences disentangle the generative factors, they do not determine whether those are local or global. In work intention is factorize the graph into separate levels of information which are,
have with them. First is that, although they
23

---

Under review as a conference paper at ICLR 2021
1.0
1.0
0.8
0.8
Local taleet factor
0.6
0.8
Intent
12
0.4
0.4
10
16
0.2
0.2
18
20
0.0
19
0.0
1011 12111215 1616616666666666 18192021
0
2
9
10
11
12
14
15
16
17
18
(a) Sample graph
(b) Sample graph
1.0
0.8
Local islent factor
8 6
0.6
0.5
18171615141122111
0.4
0.4
0.2
0.2
0.0
0.3
9
10
11
12
13
14
16
16
17
18
9
10
12
13
14
15
16
17
18
(c) Sample graph 3
(d) Sample graph
1.0
1.0
2
0.8
0.8
Local latert factorr
5
0.6
0.8
0.4
0.4
16111 15 1221222 11 10
0.2
0.2
15
0.0
0.9
10
11
12
13
14
15
2
3
5
7
9
10
latent factor
factor
(c) Sample graph 5
(f) Sample graph 6
Figure 7: Correlation matrices show the dependence among global and local latent factors. Very low correlation close to 0.0 shows that these is very less correlation between these two factors. These sample graphs from MUTAG dataset showcase the capability of GL-Disen producing mutually exclusive information for its global and local latent factors.
information common to the entire graph and information specific to local patches. FactorGCN has no mechanism to ensure any extracted factor global. Second difference is they disentangle the input features before the neighbourhood propagation while disentangle after the propagation.
24

---

Under review as a conference paper at ICLR 2021
This is due the difference of objectives we have with FactorGCN. We want to extract global level information common for the entire graph, the more suitable design decision for us is to disentangle after neighbourhood propagation allowing the graph to share information with its neighbours and form local patches. This mechanism also helps us when some datasets like Reddit and IMDB which do not have any node features to extract more meaningful information guidance from any supervision as well. We are unable to compare directly with results provided in FactorGCN paper
method does get
a supervised method and ours is unsupervised.
NED-VAE (Guo et al. 2020) also does not disentangle factors common for the entire graph (global factors) from the factors specific for each local patch. Their unsupervised disentangle mechanism aims at disentangling node features, edge features and node-edge joint. Using the loss function term (Sec 4.2.2 (Guo al. 2020) they try to make node, edge and node-edge joint features independent of each-other. We do not impose such restrictions in our model as we only want to separate out features common for the entire graph and features specific to local patches. Our model has the flexibility of using either node or edge or joint features and extract globally relevant information from any of these, while also separating out local features which are specific for patches. Their node- edge joint representation is like combination of both nodes and edges In Fig (Guo al.) 2020), third column samples, it seems like while node and edge factors have disentangled graph features (first two columns of Fig 3), node-edge joint factor has entangled them again (Both edge density and node values change with it). We believe this is expected, as it is mainly used to inform node and edge decoders about the structure of that particular individual graph (since node and edge encoders have no other mechanism to share information among them). Basically it captures the uniqueness of each individual graph with respect to node and edge features and structures, while our global latent tries to capture the common global generative factors which the entire graph dataset was generated from. For GL-Disen, we demonstrate Appendix G that removing individual node/edge/patch specific local information using disentanglement and separating out global factors is beneficial for graph level tasks such as classification. This work has only focused on graph generation. Hence discriminative performance downstream tasks have not being assessed in the paper.
25

---

Under review as a conference paper at ICLR 2021
Global level latent
Local level latent
1.30
1.00
0.75
0.75
0.25
0.25
0.30
0.00
(a) Sample Graph
Global level latent
Local level latent
11 10
11
15 1516 1311
12
0.3
0.3
15
15 1414133
0.0
0.0
10
12
13
15
10
13
14
(b) Sample Graph 2
Global latent
level
1.25
1.25
1.00
1.00
11 10
0.50
0.50
12
12
0.25
0.25
13
23
14
15
(c) Sample Graph 3
Figure 8: Part 1 Mean Absolute Pairwise Difference among global latent factors and local latent factors for two graphs from MUTAG. These matrices clearly show that the inter-node latent repre- sentation difference for produced global latent factors is very low compared local latent factors. This indicates the global factor representations disentangled by GL-Disen is indeed shared among the entire graph, unlike local factors which local certain nodes groups.
26

---

Under review as a conference paper at ICLR 2021
Global level latent
Local level latent
10
16 16 16 1312
16 1815144 14 1512
0.0
0.0
12
16
(a) Sample Graph 4
Global level latent
Local level latent
12
1.2
0.9
0.9
0.0
0.0
7
10
10
(b) Sample Graph 5
Global
level
1.00
1.00
0.75
0.75
0.50
0.50
12 11 10
0.25
11
18
12
(c) Sample Graph
Figure 9: Part Mean Absolute Pairwise Difference among global latent factors and local latent factors for two graphs from MUTAG. These matrices clearly show that the inter-node latent repre- sentation difference for produced global latent factors is very low compared local latent factors. This indicates the global factor representations disentangled by GL-Disen is indeed shared among the entire graph, unlike local factors which local certain nodes groups.
27

---

Under review as a conference paper at ICLR 2021
Figure 10: Visualizes denser generated graphs where in each row local latent representation 2, is fixed and each column 2g is fixed. This shows that 2, has a strong negative correlation with the global generative factor.
28


Reviews:
[{"review_id": "vsuMMcuTShW", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I think the idea of the paper is interesting. The writing is well and easy to read. However, it does not meet the condition of acceptance from my point of view. I have some concerns with its characterization of the literature.\n\n- Some important related work is missing. It seems authors ignore talking about some literature of unsupervised graph representation learning, such as [1], [2], etc. Also, they do not make a performance comparison with the methods above in experiments.\n[1] Contrastive Multi-View Representation Learning on Graphs. ICML 2020\n[2] Self-supervised Training of Graph Convolutional Networks. Arxiv 2020\n\n- Disentangling the global and local generative factors graph representation learning is important. However, the authors didn't explain the definition of “Global” and “Local” factors clearly. It would also be better if they can show an example of global/local factors when generating graph.\n\n- The experiments are missing. I have some concerns as follows. What is the best number of generative factors which is important for this method? Can this method occur mode collapse and how to valid or prevent it? How can this method prove that each factor is necessary for the generative process? What is the real meaning of each factor? How about the time/space complexity of this method? More experiments or discussions should be conducted to answer these questions.\n\nBased on the above reasons, this paper can have much more improvement.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "R5l4n8GZ5UA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a VAE-type generative model approach to characterize the hidden factors, with a divided focus on the global and local reconstructions. The claim is that the learnt hidden representations are disentangled (which is not defined clearly) using two reconstruction terms. The setting of the problem adopts the graph VAE setting in [1,2] (which I think the authors should mention in the related work), and the ELBO & local aggregation (convolution) approaches used in this paper are relatively standard in the generative modelling and graph representation learning domain. \n\nApart from the limited novelty, which would not have affected my evaluation if it solves the problem as claimed, I have several major concerns about this paper:\n\n1. The notion of disentanglement is not well-defined in the first place. In the VAE setting where the hidden factors are stochastic, does disentanglement refer to independence? Or they are orthogonal under a specific measure induced by the graph itself? The claims made by the authors can never be examined rigorously (the visual results do not constitute supportive evidence as I shall discuss later). \n\n2. There is no guarantee that the so-called global and local factors are not confounded. Both the global and local reconstruction terms involve the two types of factors. Given the high expressivity of deep learning models, the local factors can easily manage both tasks, or the global factors are merely enhancing the signals of the local factors. There no mechanism to prevent the cross-terms during the optimization, so the learning process of the global and local factors confounded as a result of how the authors design the objective function.\n\n3. Unclear interpretation of the visual results. It seems that the visual results showcase a similar pattern among the local and global factors, despite the difference that the signal is stronger for the local factors (which is evident as they play a more critical role in the objective). In the absence of a clear definition of disentanglement, more persuasive numerical results and interpretations are needed.\n \n\n[1] Kipf T N, Welling M. Variational graph auto-encoders[J]. arXiv preprint arXiv:1611.07308, 2016.\n[2] Xu, Da, et al. \"Generative graph convolutional network for growing graphs.\" ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PFKFeGqZfVx", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  \nThis paper proposes an unsupervised graph-level representation learning method considering global-local disentanglement. Specifically, the authors propose a GL-Disen model based on graph VAE architecture to jointly learn global and local representations for a graph. The global information is shared across the whole graph while the local information varies from patch to patch, corresponding to common and local factors, respectively. Empirical experimental results show that the learned representation achieves superior performance in the downstream graph classification task, and analyses demonstrate the learned representations exhibit some disentangle property.   \n\nPros:  \n1. Unsupervised graph representation learning considering global and local disentanglement seems to be a novel problem.  \n2. The proposed method generalizes disentangled VAE into graph data to disentangle common factors from the local ones. The formulations and model descriptions are clear.  \n3. Experiments, including both qualitative analysis and quantitative results, demonstrate the effectiveness of the learned global factors in downstream tasks.  \n\nCons and questions:  \nMy major concern lies in the insufficiency of experiments. Specifically:  \n1. The disentanglement part is modified from Beta-VAE. Since normal VAE is adopted in graphs (e.g., Variational Graph Auto-Encoders by Kipf and Welling), the authors need to compare these methods to demonstrate the improvement is actually from the disentanglement part rather than the VAE structure.   \n2. Although the authors demonstrate the effectiveness of disentanglement in downstream tasks (i.e., graph classification), it is unclear whether these global factors have intuitive explanations on some of the datasets, e.g., the showcases of molecular graphs in Duvenaud et al., 2015, or the authors may adopt some synthetic datasets.  \n3. Since both the global and local node representations are disentangled, I am curious whether the local node representations can also be validated in some downstream node-level tasks.   \n4. Figure 2 in Section 4.2.1 is not entirely convincing since there is no reference line of how much correlation a non-disentangled method will have (e.g., in Ma et al., 2019, the authors compare the disentangled method with GCN).   \n\nOther questions:  \n5. How the proposed method can handle the mode collapse problem, i.e., only a few latent factors learn useful information?  \n6. As shown in Table 1, though the proposed method outperforms other GNNs, it does not always compare favorably to kernel-based methods such as GCKN. The authors may want to further elaborate on the pros and cons of using GNN vs. kernel-based methods.  \n7. There lacks a discussion on the complexity of the proposed method.  \n8. The technical contribution is somewhat limited since both Beta-VAE and graph VAE are known in the literature. It would be more interesting if the authors can integrate local-global disentanglement with local neighborhood disentanglement in Ma et al. 2019 to derive a more novel architecture.  \n\nI will be happy to improve my scores if authors can address the above questions.  \n  \n  \n=========  \n  \nI have updated my score considering the paper has improved its quality after the revision (adding more experiments/baselines, comparison with the literature, etc.).\n\n\n=========   \nNew updates: following the new comments of Reviewer 4, I also briefly check the code in the supplementary material and find it indeed seems to have the consistency problem (i.e., not reconstructing graph edges as mentioned in the paper). Thus, I am also wondering how the authors implement Graph-VAE in the rebuttal phase and whether the improvement of their proposed method over Graph-VAE is really from disentanglement or the differences in the autoencoder. Based on this potentially serious problem, I reinstate my original score and think the paper should be clarified before acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KU2nbhXmxhW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a disentanglement learning based approach for unsupervised graph level representation learning. They assume that disentangled representations which capture these global and local generative factors into independent latent units can be highly beneficial for graph level tasks. The extensive experiments and analysis  show that our method achieves the state-of-the-art performance on the task of unsupervised graph representation learning.\n\n===========\nStrengths:\n1. The paper is well written and the disentangling factors can benefit the unsupervised graph representation learning.\n2. The performance of this work is good compared with the state-of-the-art baselines. The source code is also available.\n3. The related work is sufficient to understand the motivation of this work. The \n\n=====\nWeakness:\n1. The idea is not very novel. For example, two important assumptions 1) a  global and local factor for graph analysis 2) local latent factors are independent. \nThose two assumptions actually have been explored in unsupervised learning tasks. For example, the follow paper[1] exactly disentangle local and global information into two separate sets of latent variables within the VAE framework.  It seems that migrating this idea under graph is straightforward. The paper is more like a mixture of [1] and (Higgins et al., 2017), and GCN\n\n[1] Charakorn, Rujikorn, et al. \"An Explicit Local and Global Representation Disentanglement Framework with Applications in Deep Clustering and Unsupervised Object Detection.\" arXiv preprint arXiv:2001.08957 (2020).\n\n2. In Figure4, It seems that the GL-Disen global has very good accuracy. The GL-disen global-local combines only outperform GL-Disen global within a very small range of \\lambda but with large fluctuation. Does that mean the local factor contribution little to the overall performance?\n\n\nIn Conclude,the authors propose a VAE based learning algorithm to disentangle the global graph-level information. The overall presentation is good. The similar ideas have been explored in unsupervised learning. The novelty of this work is thus not very impressive.", "extra_scores": {}, "extra_text": {}}, {"review_id": "22cInD8Aeh_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed to disentangle the global level information from the local level one to reduce the effect of the irrelevant information. The proposed method outperforms several state-of-the-arts on multiple datasets for graph classification. Overall, I like the idea of applying unsupervised disentangled learning to graph level representation learning. Some concerns are on the experimental study and missing references.\n\nStrong Points:\n1. Disentanglement learning is a cutting-edge field and has gained much attention in recent years. It is true that global and local features often entangle together when we learn graph representations. The problem is real and important.\n\n2. The architecture of the model is easy to understand and reasonable.\n\n3. The experimental study is comprehensive, including both qualitative analysis and quantitative analysis. The experimental setup instructions and pseudo-codes are very clear, making the algorithm easy to be reproduced.\n\nWeak Points:\n1. Performing experiments only on graph classification tasks weakens the significance of the paper. It is common for graph representation learning methods to be tested on other tasks, such as graph similarity/distance computation and graph-level clustering, in order to draw a general and convincing conclusion.\n\n2. Some important references are missing. The authors should discuss and compare with them.\nOn graph-level representation learning:\n- Bai et al. Unsupervised Inductive Graph-Level Representation Learning via Graph-Graph Proximity. IJCAI 2019.\nOn disentangled representation learning:\n- Yang et al. Factorizable Graph Convolutional Networks. NIPS 2020.\n- Guo et al. Interpretable Deep Graph Generation with Node-edge Co-disentanglement. KDD 2020.\n\n3. The paper mentioned that the global and local latent generative factors are sampled from their respective posterior distributions. More details are expected.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors designed a disentanglement mechanism for global and local information of graphs and proposed a graph representation method based on it. I agree with the authors that 1) considering the global and local information of graphs jointly is reasonable and helpful (as shown in the experiments) and 2) disentanglement is different from independence. 

However, the concerns of the reviewers are reasonable --- Eq. (2) and the paragraph before it indeed show that the authors treat the global and the local information independently. Moreover, the disentanglement of the global information (the whole graph) and the local information (the patch/sub-graph) is not well-defined. In my opinion, for the MNIST digits, the angle and the thickness (or something else) of strokes can be disentangled (not independent) factors that have influences on different properties of the data. In this work, if my understanding is correct, the global and the local factors just provide different views to analyze the same graphs and the proposed method actually designs a new way to leverage multi-view information. It is not sure whether the views are disentangled and whether the improvements are from "disentanglement". 

If the authors can provide an example to explain their "disentanglement" simply as the MNIST case does, this work will be more convincing. Otherwise, this work suffers from the risk of overclaiming.
Decision: Reject