Title: Disentangled Generative Graph Representation Learning
Venue: NeurIPS 2024
Accepted: no
Keywords: Probabilistic Inference

disentangled representation learning

graph representation learning
Abstract: Recently, generative graph models have shown promising results in learning graph representations through self-supervised methods. However, most existing generative graph representation learning (GRL) approaches rely on random masking across the entire graph, which overlooks the entanglement of learned representations. This oversight results in non-robustness and a lack of explainability. Furthermore, disentangling the learned representations remains a significant challenge and has not been sufficiently explored in GRL research.
Based on these insights, this paper introduces  **DiGGR** (**Di**sentangled **G**enerative **G**raph **R**epresentation Learning), a self-supervised learning framework. DiGGR aims to learn latent disentangled factors and utilizes them to guide graph mask modeling, thereby enhancing the disentanglement of learned representations and enabling end-to-end joint learning.
Extensive experiments on 11 public datasets for two different graph learning tasks demonstrate that DiGGR consistently outperforms many previous self-supervised methods, verifying the effectiveness of the proposed approach.

1

Abstract

Recently, generative graph models have shown promising results in learning graph representations through self-supervised methods. However, most existing generative graph representation learning (GRL) approaches rely on random masking across the entire graph, which overlooks the entanglement of learned representations. This oversight results in non-robustness and a lack of explainability. Furthermore, disentangling the learned representations remains a significant challenge and has not been sufficiently explored in GRL research. Based on these insights, this paper introduces DiGGR ( Di sentangled G enerative G raph R epresentation Learning), a self-supervised learning framework. DiGGR aims to learn latent disentangled factors and utilizes them to guide graph mask modeling, thereby enhancing the disentanglement of learned representations and enabling end-to-end joint learning. Extensive experiments on 11 public datasets for two different graph learning tasks demonstrate that DiGGR consistently outperforms many previous self-supervised methods, verifying the effectiveness of the proposed approach.

1 Introduction

Self-supervised learning (SSL) has received much attention due to its appealing capacity for learning data representation without label supervision. While contrastive SSL approaches are becoming increasingly utilized on images [Chen et al., 2020] and graphs [You et al., 2020], generative SSL has been gaining significance, driven by groundbreaking practices such as BERT for language [Devlin et al., 2018], BEiT [Bao et al., 2021], and MAE [He et al., 2022a] for images. Along this line, there is a growing interest in constructing generative SSL models for other modalities, such as graph masked autoencoders (GMAE). Generally, the fundamental concept of GMAE [Tan et al., 2022] is to utilize an autoencoder architecture to reconstruct input node features, structures, or both, which are randomly masked before the encoding step. Recently, various well-designed GMAEs have emerged , achieving remarkable results in both node classification and graph classification [Hou et al., 2022, Tu et al., 2023, Tian et al., 2023].

Despite their significant achievements, most GMAE approaches typically treat the entire graph as 27 holistic, ignoring the graph's latent structure. As a result, the representation learned for a node tends 28 to encapsulate the node's neighborhood as a perceptual whole, disregarding the nuanced distinctions 29 between different parts of the neighborhood [Ma et al., 2019, Li et al., 2021, Mo et al., 2023]. 30 For example, in a social network G , individual n is a member of both a mathematics group and 31 several sports interest groups. Due to the diversity of these different communities, she may exhibit 32 different characteristics when interacting with members from various communities. Specifically, 33 the information about the mathematics group may be related to her professional research, while the 34 information about sports clubs may be associated with her hobbies. However, the existing approach 35 overlooks the heterogeneous factors of node n , failing to identify and disentangle these pieces of 36

Factors

Factors

Factors

Factors

Factors

Factors

Factors

Factors

Factors

Factors

Factors

Factors

Factors

Factors

Factor Analysis

Probability Bar

Maximum Probability

Different node types

(a) Applying previous methods to GMAE

(b) Latent factor learning

information effectively [Hou et al., 2022]. Consequently, the learned features may be easily influenced 37 by irrelevant factors, resulting in poor robustness and difficulty in interpretation. 38

To alleviate the challenge described above, there is an increasing interest in disentangled graph 39 representation learning [Bengio et al., 2013, Li et al., 2021, Ma et al., 2019, Mo et al., 2023, Xiao 40 et al., 2022], which aims at acquiring representations that can disentangle the underlying explanatory 41 factors of variation in the graph. Specifically, many of these methods rely on a latent factor detection 42 module, which learns the latent factors of each node by comparing node representations with various 43 latent factor prototypes. By leveraging these acquired latent factors, these models adeptly capture 44 factor-wise graph representations, effectively encapsulating the latent structure of the graph. Despite 45 significant progress, few studies have endeavored to adapt these methods to to generative graph 46 representation learning methods, such as GMAE. This primary challenge arises from the difficulty of 47 achieving convergence in the latent factor detection module under the generative training target, thus 48 presenting obstacles in practical implementation. As shown in Fig.1(a), directly applying the previous 49 factor learning method to GMAE would make the factor learning module difficult to converge, 50 resulting in undistinguished probabilities and misallocation of similar nodes to different latent factor 51 groups. 52

53

54

55

56

57

58

59

60

61

62

63

To address these challenges, we introduce Di sentangled G enerative G raph R epresentation Learning ( DiGGR ), a self-supervised graph generation representation learning framework. Generally speaking, DiGGR learns how to generate graph structures from latent disentangle factors z and leverages this to guide graph mask reconstruction, while enabling end-to-end joint learning. Specifically, i ) To capture the heterogeneous factors in the nodes, we introduce the latent factor learning module. This module models how edges and nodes are generated from latent factors, allowing graphs to be factorized into multiple disentangled subgraphs. ii ) To learn a deeper disentangled graph representation, we design a factor-wise self-supervised graph representation learning framework. For each subgraph, we employ a distinct masking strategy to learn an improved factor-specific graph representation. Evaluation shows that the proposed framework can achieve significant performance enhancement on various node and graph classification benchmarks.

64

65

66

67

68

69

70

71

72

The main contributions of this paper can be summarized as follows:

We utilized the latent disentangled factor to guide mask modeling. A probabilistic graph generation model is employed to identify the latent factors within a graph, and it can be jointly trained with GMAE through variational inference.

Introducing DiGGR ( Di sentangled G enerative G raph R epresentation Learning) to further capture the disentangled information in the latent factors, enhancing the disentanglement of the learned node representations.

Empirical results show that the proposed DiGGR outperforms many previous self-supervised methods in various node- and graph-level classification tasks.

2 Related works 73

Graph Self-Supervised Learning: Graph SSL has achieved remarkable success in addressing label 74 scarcity in real-world network data, mainly consisting of contrastive and generative methods. Con75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

trastive methods, includes feature-oriented approaches[Hu et al., 2019, Zhu et al., 2020, Veliˇ ckovi´ c et al., 2018], proximity-oriented techniques [Hassani and Khasahmadi, 2020, You et al., 2020], and graph-sampling-based methods [Qiu et al., 2020]. A common limitation across these approaches is their heavy reliance on the design of pretext tasks and augmentation techniques. Compared to contrastive methods, generative methods are generally simpler to implement. Recently, to tackle the challenge of overemphasizing neighborhood information at the expense of structural information [Hassani and Khasahmadi, 2020, Veliˇ ckovi´ c et al., 2018], the Graph Masked Autoencoder (GMAE) has been proposed. It applies a masking strategy to graph structure [Li et al., 2023a], node attributes [Hou et al., 2022], or both [Tian et al., 2023] for representation learning. Unlike most GMAEs, which employ random mask strategies, this paper builds disentangled mask strategies.

Disentangled Graph Learning: Disentangled representation learning aims to discover and isolate the fundamental explanatory factors inherent in the data [Bengio et al., 2013]. Existing efforts in disentangled representation learning have primarily focused on computer vision [Higgins et al., 2017, Jiang et al., 2020]. Recently, there has been a surge of interest in applying these techniques to graph-structured data [Li et al., 2021, Ma et al., 2019, Mercatali et al., 2022, Mo et al., 2023]. For example, DisenGCN [Ma et al., 2019] utilizes an attention-based methodology to discriminate between distinct latent factors, enhancing the representation of each node to more accurately reflect its features across multiple dimensions. DGCL [Li et al., 2021] suggests learning disentangled graph-level representations through self-supervision, ensuring that the factorized representations independently capture expressive information from various latent factors. Despite the excellent results achieved by the aforementioned methods on various tasks, these methods are difficult to converge in generative graph SSL, as we demonstrated in the experiment of Table.3. Therefore, this paper proposes a disentangled-guided framework for generative graph representation learning, capable of learning disentangled representations in an end-to-end self-supervised manner.

3 Proposed Method

In this section, we propose DiGGR ( Di sentangled G enerative G raph R epresentation Learning) for self-supervised graph representation learning with mask modeling. The framework was depicted in Figure 2, comprises three primary components: Latent Factor Learning (Section 3.2), Graph Factorization (Section 3.2) and Disentangled Graph Masked autoencder (Section 3.3). Before elaborating on them, we first show some notations.

3.1 Preliminaries

A graph G can be represented as a multi-tuple G = { V, A, X } with N nodes and M edges, where | V | = N is the node set, | A | = M is the edge set, and X ∈ R N × L is the feature matrix for N nodes with L dimensional feature vector. The topology structure of graph G can be found in its adjacency matrix A ∈ R N × N . z ∈ R N × K is the latent disentangled factor matrix, and K is the predefined factor number. Since we aim to obtain the z to guide the mask modeling, we first utilize a probabilistic graph generation model to factorize the graph before employing the mask mechanism. Given the graph G , it is factorized into { G 1 , G 2 , ..., G K } , and each factor-specific graph G k consists of its factor-specific edges A ( k ) , node set V ( k ) and node feature matrix X ( k ) . Other notations will be elucidated as they are employed.

3.2 Latent Factor Learning

In this subsection, we describe the latent factor learning method. In this phase, our objective is to derive factor-specific node sets { V (1) , V (2) , ..., V ( K ) } and adjacency matrices { A (1) , A (2) , ..., A ( K ) } , serving as basic unit of the graph to guide the subsequent masking. The specific approach involves modeling the distribution of nodes and edges, utilizing the generative process developed in EPM [Zhou, 2015]. The generative process of EPM under the Bernoulli-Poisson link [Zhou, 2015] can be described as:

where K is the predefined number of latent factors, and u and v are the indexes of the nodes. Here, M uv is the latent count variable between node u and v ; γ k is a positive factor activation level indicator, which measures the node interaction frequency via factor k ; z uk is a positive latent variable for node

Graph Factorization

via

Mask

Mask

Mask

Feature Reconstruction

Random Mask phase

𝑳𝒐𝒔𝒔

GNN

FeedBack

𝒌

𝝀

𝑮

Masked Node

Hidden

Latent Factor Learning

Reconstruction-based

Representation Learning

GAE

GAE

GAE

GAE

Disentangled Graph Mask

u , which measures how strongly node u is affiliated with factor k . The prior distribution of latent 126 factor variable z uk is set to Gamma distribution, where α and β are normally set to 1. Therefore, the 127 intuitive explanation for this generative process is that, with z uk and z vk measuring how strongly 128 node u and v are affiliated with the k -th factor, respectively, the product γ k z uk z vk measures how 129 strongly nodes u and v are connected due to their affiliations with the k -th factor. 130

131

132

133

134

135

136

137

138

139

140

141

142

Node Factorization: Equation 1 can be further augmented as follows:

where M ukv represents how often nodes u and v interact due to their affiliations with the k -th factor. To represent how often node u is affiliated with the k -th factor, we further introduce the latent count M uk · = ∑ v = u M ukv . Then, we can soft assign node u to multiple factors in { k : M uk · } ≥ 1 , or hard assign node u to a single factor using arg max ( M uk · ) . However, our experiments show that

k

soft assignment method results in significant overlap among node sets from different factor group, diminishing the distinctiveness. Note that previous study addressed a similar issue by selecting the top-k most attended regions [Kakogeorgiou et al., 2022]. Thus, we choose the hard assign strategy to factorize the graph node set V graph into factor-specific node sets { V (1) , V (2) , · · · , V ( K ) } .

Edge Factorization: To create factor-specific edges A ( k ) for a factor-specific node set V ( k ) , a straightforward method involves removing all external nodes connected to other factor groups. This can be defined as:

Besides, the global graph edge A can also be factorized into positive-weighted edges [He et al., 143 2022b] for each latent factor as: 144

Applying Equation 4 to all pairs of nodes yields weighted adjacency matrices { A ( k ) } k k =1 , with A ( k ) 145 corresponding to latent factor z k . Note that A ( k ) has the same dimension as A and Equation 4 146

̸

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

presents a trainable weight for each edge, which can be jointly optimized through network training, showcasing an advantage over Equation 3 in this aspect. Therefore, we apply Equation 4 for edge factorization.

Variational Inference: The latent factor variable z determines the quality of node and edge factorization, so we need to approximate its posterior distribution. Denoting z u = ( z u 1 , ..., z uK ) , z u ∈ R K + , which measures how strongly node u is affiliated with all the K latent factors, we adopt a Weibull variational graph encoder [Zhang et al., 2018, He et al., 2022b]:

where GNNEPM ( · ) stands for graph neural networks, and we select a two-layer Graph Convolution Networks ( i.e. , GCN [Kipf and Welling, 2016a]) for our models; k u , λ u ∈ R K + are the shape and scale parameters of the variational Weibull distribution, respectively. The latent variable z u can be conveniently reparameterized as:

The optimization objective of latent factor learning phase can be achieved by maximizing the evidence lower bound (ELBO) of the log marginal likelihood of edge log p ( A ) , which can be computed as:

where the first term is the expected log-likelihood or reconstruction error of edge, and the second term is the Kullback-Leibler (KL) divergence that constrains q ( z u ) to be close to its prior p ( z u ) . The analytical expression for the KL divergence and the straightforward reparameterization of the Weibull distribution simplify the gradient estimation of the ELBO concerning the decoder parameters and other parameters in the inference network.

3.3 Disentangled Grpah Masked Autoencoder

With the latent factor learning phase discussed in 3.2, the graph can be factorized into a series of factor-specific subgraphs { G 1 , G 2 , ..., G K } via the latent factor z . To incorporate the disentangled information encapsulated in z into the graph masked autoencoder, we proposed Disentangled Graph Masked Autoencoder in this section. Specifically, this section will first introduce the latent factor-wise GMAE and the graph-level GMAE.

3.3.1 Latent Factor-wise Grpah Masked Autoencoder

To capture disentangled patterns within the latent factor z , for each latent subgraph G k = ( V ( k ) , A ( k ) , X ( k ) ) , the latent factor-wise GMAE can be described as:

where ¯ X ( k ) is the masked node feature matrix for the k -th latent factor, and ˜ X d denotes the reconstructed node features. GNNenc ( . ) and GNNdec ( . ) are the graph encoder and decoder, respectively; H ( k ) d ∈ R N × D are factor-wise hidden representations, and H d = H (1) d ⊕ H (2) d · · · ⊕ H ( K ) d . After the concatenation operation ⊕ in feature dimension, the multi factor-wise hidden representation becomes H d ∈ R N × ( K · D ) , which is used as the input of GNN dec ( . ) .

Regarding the mask opeartion, we uniformly random sample a subset of nodes ¯ V ( k ) ∈ V ( k ) and mask each of their features with a mask token, such as a learnable vector X [ M ] ∈ R d . Thus, the node feature in the masked feature matrix can be defined as:

The objective of latent factor-wise GMAE is to reconstruct the masked features of nodes in ¯ V ( k ) 182 given the partially observed node signals ¯ X ( k ) and the input adjacency matrix A ( k ) . Another crucial 183 component of the GMAE is the feature reconstruction criterion, often used in language as cross184 entropy error [Devlin et al., 2018] and in the image as mean square error [He et al., 2022a]. However, 185

texts and images typically involve tokenized input features, whereas graph autoencoders (GAE) do 186 not have a universal tokenizer. We adopt the scored cosine error of GraphMAE [Hou et al., 2022] as 187 the loss function. Generally, given the original feature X ( k ) and reconstructed node feature ˜ X ( k ) , the 188 defined SCE is: 189

where ¯ V = ¯ V (1) ∪ ¯ V (2) ... ∪ ¯ V ( K ) and Equation 10 are averaged over all masked nodes.The scaling factor γ is a hyper-parameter adjustable over different datasets. This scaling technique could also be viewed as adaptive sample reweighting, and the weight of each sample is adjusted with the reconstruction error. This error is also famous in the field of supervised object detection as the focal loss [Lin et al., 2017].

Graph-level Graph Mask Autoencoder: For the node classification task, we have integrated graph-level GMAE into DiGGR. We provide a detailed experimental analysis and explanation for this difference in Appendix A.1.2. The graph-level masked graph autoencoder is designed with the aim of further capturing the global patterns, which can be designed as:

¯ X is the masked node feature matrix, whose mask can be generated by uniformly random sampling a subset of nodes ˜ V ∈ V , or obtained by concatenating the masks of all factor-specific groups ˜ V = ¯ V (1) ∪ ¯ V (2) ... ∪ ¯ V ( K ) . The global hidden representation encoded by GNNenc ( . ) is H g , which is then passed to the decoder. Similar to Equation 10, we can define the graph-level reconstruct loss as:

which is averaged over all masked nodes.

3.4 Joint Training and Inference

Benefiting from the effective variational inference method, the proposed latent factor learning and dsientangled graph masked autoencoder can be jointly trained in one framework. We combine the aforementioned losses with three mixing coefficient λ d , λ g and λ z during training, and the loss for joint training can be written as

Since Weibull distributions have easy reparameterization functions, these parameters can be jointly trained by stochastic gradient descent with low-variance gradient estimation. We summarize the training algorithm at Algorithm 1 in Appendix A.4. For downstream applications, the encoder is applied to the input graph without any masking in the inference stage. The generated factor-wise node embeddings H d and graph-level embeddings H g can either be concatenated in the feature dimensions or used separately. The resulting final representation H can be employed for various graph learning tasks, such as node classification and graph classification. For graph-level tasks, we use a non-parameterized graph pooling (readout) function, e.g. , MaxPooling and MeanPooling to obtain the graph-level representation.

Time and space complexity: Let's recall that in our context, N , M , and K represent the number of nodes, edges, and latent factors in the graph, respectively. The feature dimension is denoted by F , while L 1 , L 2 , L 3 , and L 4 represent the number of layers in the latent factor learning encoder, the latent factor-wise GMAE's encoder, the graph-level GMAE's encoder, and the decoder respectively. In DiGGR, we constrain the hidden dimension size in latent factor-wise GMAE's encoder to be 1 /K of the typical baseline dimensions. Consequently, the time complexity for training DiGGR can be expressed as O (( L 1 + L 2 + L 3 ) MF +( L 1 + L 2 /K + L 3 ) NF 2 + N 2 F + L 4 NF 2 ) , and the space complexity is O (( L 1 + L 2 + L 3 + L 4 ) NF + KM +( L 1 + L 2 /K + L 3 + L 4 ) F 2 ) , with O (( L 1 + L 2 /K + L 3 + L 4 ) F 2 ) attributed to model parameters. We utilize the Bayesian factor model in our approach to reconstruct edges. Its time complexity aligns with that of variational inference in SeeGera Li et al. [2023b], predominantly at O ( N 2 F ) ; Therefore, the complexity of DiGGR is comparable to previous works.

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

4 Experiments

We compare the proposed self-supervised framework DiGGR against related baselines on two fundamental tasks: unsupervised representation learning on node classification and graph classification . We evaluate DiGGR on 11 benchmarks. For node classification, we use 3 citation networks (Cora, Citeseer, Pubmed [Yang et al., 2016]), and protein-protein interaction networks (PPI) [Hamilton et al., 2017]. For graph classification, we use 3 bioinformatics datasets (MUTAG, NCI1, PROTEINS) and 4 social network datasets (IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY and COLLAB). The specific information of the dataset and the hyperparameters used by the network are listed in the Appendix A.2 in table 5 and 6. We also provide the detailed experiment setup in Appendix A.2 for node classification (4.1) and graph classification (4.2)

4.1 Node Classification

The baseline models for node classification can be divided into three categories: i ) supervised methods, including GCN [Kipf and Welling, 2016a] , DisenGCN[Ma et al., 2019], VEPM[He et al., 2022b] and GAT [Velickovic et al., 2017]; ii ) contrastive learning methods, including MVGRL [Hassani and Khasahmadi, 2020], InfoGCL [Xu et al., 2021], DGI [Veliˇ ckovi´ c et al., 2018], GRACE [Zhu et al., 2020], BGRL [Thakoor et al., 2021] and CCA-SSG [Zhang et al., 2021]; iii ) generative learning methods, including GraphMAE [Hou et al., 2022], GraphMAE2[Hou et al., 2023], Bandana[Zhao et al., 2024], GiGaMAE[Shi et al., 2023], SeeGera[Li et al., 2023b], GAE and VGAE [Kipf and Welling, 2016b]. The node classification results were listed in Table 1. DiGGR demonstrates competitive results on the provided dataset, achieving results comparable to those of supervised methods.

4.2 Graph Classification

Baseline Models We categorized the baseline models into four groups: i ) supervised methods, including GIN [Xu et al., 2018], DiffPool[Ying et al., 2018] and VEPM[He et al., 2022b]; ii ) classical graph kernel methods: Weisfeiler-Lehman sub-tree kernel (WL) [Shervashidze et al., 2011] and deep graph kernel (DGK) [Yanardag and Vishwanathan, 2015]; iii ) contrastive learning methods, including GCC [Qiu et al., 2020], graph2vec [Narayanan et al., 2017], Infograph [Sun et al., 2019], GraphCL [You et al., 2020], JOAO [You et al., 2021], MVGRL [Hassani and Khasahmadi, 2020], and InfoGCL [Xu et al., 2021]; 4) generative learning methods, including graph2vec [Narayanan et al., 2017], sub2vec [Adhikari et al., 2018], node2vec [Grover and Leskovec, 2016], GraphMAE [Hou et al., 2022], GraphMAE2[Hou et al., 2023], GAE and VGAE [Kipf and Welling, 2016b]. Per graph classification research tradition, we report results from previous papers if available.

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

Performance Comparison The graph classification results are presented in Table 2. In general, we find that DiGGR gained the best performance among other baselines on five out of seven datasets, while achieving competitive results on the other two datasets. The performance of DiGGR is comparable to that of supervised learning methods. For instance, the accuracy on IMDB-B and IMDB-M surpasses that of GIN and DiffPool. Moreover, within the reported datasets, our method demonstrates improved performance compared to random mask methods like GraphMAE, particularly on the IMDB-M, COLLAB, and PROTEINS datasets. This underscores the effectiveness of the proposed method.

4.3 Exploratory Studies

Visualizing latent representations To examine the influence of the learned latent factor on classification results, we visualized the latent disentangled factor z , which reflects the node-factor affiliation, and the hidden representation H used for classification. MUTAG is selected as the representative for classification benchmarks. We encodes the representations into 2-D space via t-SNE [Van der Maaten and Hinton, 2008]. The result is shown in Figure 3(a), where each node is colored according to its node labels. The clusters in Figure 3(a) still exhibit differentiation in the absence of label supervision, suggesting that z obtained through unsupervised learning can enhance node information and offer a guidance for the mask modeling. We then visualize the hidden representation used for classification tasks, and color each node according to the latent factor to which it belongs. The results are depicted in Figure 3(b), showcasing separability among different color clusters. This illustrates the model's ability to extract information from the latent factor, thereby enhancing the quality of the learned representations.

Task-relevant factors To assess the statistical correlation between the learned latent factor and the task, we follow the approach in [He et al., 2022b] and compute the Normalized Mutual Information (NMI) between the nodes in the factor label and the actual node labels. NMI is a metric that ranges from 0 to 1, where higher values signify more robust statistical dependencies between two random variables. In the experiment, we utilized the MUTAG dataset, comprising 7 distinct node types, and the NMI value we obtained was 0.5458. These results highlight that the latent factors obtained through self-supervised training are meaningful for the task, enhancing the correlation between the inferred latent factors and the task.

Disentangled representations To assess DiGGR's capability to disentangle the learned representation for downstream task, we provide a qualitative evaluation by plotting the correlation of the

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

node class  0

(a)

z

, node label

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

node class  0

(b)

H

, factor label

(a) GraphMAE

(b) DiGGR

node representation in Figure 4. The figure shows the absolute values of the correlation between the elements of 512-dimensional graph representation and representation obtained from GraphMAE and DiGGR, respectively. From the results, we can see that the representation produced by GraphMAE exhibits entanglement, whereas DiGGR's representation displays a overall block-level pattern, indicating that DiGGR can capture mutually exclusive information in the graph and disentangle the hidden representation to some extent. Results for more datasets can be found in Appendix A.3.

Why DiGGR works better: To validate that disentangled learning can indeed enhance the quality of the representations learned by GMAE, we further conduct quantitative experiments. The Normalized Mutual Information (NMI) is used to quantify the disentangling degree of different datasets. Generally, the NMI represents the similarity of node sets between different factor-specific graphs, and the lower NMI suggests a better-disentangled degree with lower similarity among factor-specific graphs. The NMI between latent factors and the corresponding performance gain (compared to GraphMAE) are shown in the Table.3. As the results show, DiGGR's performance improvement has a positive correlation with disentangled degree, where the better the disentangled degree, the more significant the performance improvement. For methods relying on Non-probabilistic Factor Learning, the NMI tends to approach 1. This is attributed to the challenges faced by the factor learning module in converging, thereby hindering the learning of distinct latent factors. The presence of confused latent factors offers misleading guidance for representation learning, consequently leading to decreased performance.

5 Conclusions

In this paper, we propose DiGGR (Disentangled Generative Graph Representation Learning), designed to achieve disentangled representations in graph masked autoencoders by leveraging latent disentangled factors. In particular, we achieve this by two steps: 1) We utilize a probabilistic graph generation model to factorize the graph via the learned disentangled latent factor; 2) We develop a Disentangled Graph Masked Autoencoder framework, with the aim of integrating the disentangled information into the representation learning of Graph Masked Autoencoders. Experiments demonstrate that our model can acquire disentangled representations, and achieve favorable results on downstream tasks.

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

References

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems , 33: 5812-5823, 2020.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022a.

Qiaoyu Tan, Ninghao Liu, Xiao Huang, Rui Chen, Soo-Hyun Choi, and Xia Hu. Mgae: Masked autoencoders for self-supervised learning on graphs. arXiv preprint arXiv:2201.02534 , 2022.

Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. GraphMAE: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 594-604, 2022.

Wenxuan Tu, Qing Liao, Sihang Zhou, Xin Peng, Chuan Ma, Zhe Liu, Xinwang Liu, and Zhiping Cai. Rare: Robust masked graph autoencoder. arXiv preprint arXiv:2304.01507 , 2023.

Yijun Tian, Kaiwen Dong, Chunhui Zhang, Chuxu Zhang, and Nitesh V Chawla. Heterogeneous graph masked autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 9997-10005, 2023.

Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolutional networks. In International conference on machine learning , pages 4212-4221. PMLR, 2019.

Haoyang Li, Xin Wang, Ziwei Zhang, Zehuan Yuan, Hang Li, and Wenwu Zhu. Disentangled contrastive learning on graphs. Advances in Neural Information Processing Systems , 34:2187221884, 2021.

Yujie Mo, Yajie Lei, Jialie Shen, Xiaoshuang Shi, Heng Tao Shen, and Xiaofeng Zhu. Disentangled multiplex graph representation learning. In International Conference on Machine Learning , pages 24983-25005. PMLR, 2023.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.

Teng Xiao, Zhengyu Chen, Zhimeng Guo, Zeyang Zhuang, and Suhang Wang. Decoupled selfsupervised learning for graphs. Advances in Neural Information Processing Systems , 35:620-634, 2022.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265 , 2019.

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 , 2020.

Petar Veliˇ ckovi´ c, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint arXiv:1809.10341 , 2018.

Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International conference on machine learning , pages 4116-4126. PMLR, 2020.

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gcc: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1150-1160, 2020.

Jintang Li, Ruofan Wu, Wangbin Sun, Liang Chen, Sheng Tian, Liang Zhu, Changhua Meng, Zibin Zheng, and Weiqiang Wang. What's behind the mask: Understanding masked graph modeling for graph autoencoders. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1268-1279, 2023a.

Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster) , 3, 2017.

Wentao Jiang, Si Liu, Chen Gao, Jie Cao, Ran He, Jiashi Feng, and Shuicheng Yan. Psgan: Pose and expression robust spatial-aware gan for customizable makeup transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5194-5202, 2020.

Giangiacomo Mercatali, André Freitas, and Vikas Garg. Symmetry-induced disentanglement on graphs. Advances in neural information processing systems , 35:31497-31511, 2022.

Mingyuan Zhou. Infinite edge partition models for overlapping community detection and link prediction. In Artificial intelligence and statistics , pages 1135-1143. PMLR, 2015.

Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. What to hide from your students: Attention-guided masked image modeling. In European Conference on Computer Vision , pages 300-318. Springer, 2022.

Yilin He, Chaojie Wang, Hao Zhang, Bo Chen, and Mingyuan Zhou. A variational edge partition model for supervised graph representation learning. Advances in Neural Information Processing Systems , 35:12339-12351, 2022b.

Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. Whai: Weibull hybrid autoencoding inference for deep topic modeling. arXiv preprint arXiv:1803.01328 , 2018.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016a.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Xiang Li, Tiandi Ye, Caihua Shan, Dongsheng Li, and Ming Gao. Seegera: Self-supervised semiimplicit graph variational auto-encoders with masking. In Proceedings of the ACM web conference 2023 , pages 143-153, 2023b.

Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. Graph attention networks. stat , 1050(20):10-48550, 2017.

Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, and Xiang Zhang. Infogcl: Informationaware graph contrastive learning. Advances in Neural Information Processing Systems , 34: 30414-30425, 2021.

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veliˇ ckovi´ c, and Michal Valko. Large-scale representation learning on graphs via bootstrapping. arXiv preprint arXiv:2102.06514 , 2021.

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S Yu. From canonical correlation analysis to self-supervised graph neural networks. Advances in Neural Information Processing Systems , 34:76-89, 2021.

Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. Graphmae2: A decoding-enhanced masked self-supervised graph learner. In Proceedings of the ACM Web Conference 2023 , pages 737-746, 2023.

Ziwen Zhao, Yuhua Li, Yixiong Zou, Jiliang Tang, and Ruixuan Li. Masked graph autoencoder with non-discrete bandwidths. arXiv preprint arXiv:2402.03814 , 2024.

Yucheng Shi, Yushun Dong, Qiaoyu Tan, Jundong Li, and Ninghao Liu. Gigamae: Generalizable graph masked autoencoder via collaborative latent space reconstruction. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , pages 2259-2269, 2023.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016b.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems , 31, 2018.

Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research , 12(9), 2011.

Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 1365-1374, 2015.

Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. arXiv preprint arXiv:1707.05005 , 2017.

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semisupervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv:1908.01000 , 2019.

Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning , pages 12121-12132. PMLR, 2021.

Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash. Sub2vec: Feature learning for subgraphs. In Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part II 22 , pages 170-182. Springer, 2018.

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , pages 855-864, 2016.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) , 2(3):1-27, 2011.

Reid Andersen, Fan Chung, and Kevin Lang. Local graph partitioning using pagerank vectors. In 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06) , pages 475-486. IEEE, 2006.

460

461

462

463

464

465

466

467

468

469

470

471

A Appendix / supplemental material

Optionally include supplemental material (complete proofs, additional experiments and plots) in appendix. All such materials SHOULD be included in the main submission.

A.1 Ablation Study

A.1.1 Number of factors

One of the crucial hyperparameters in DiGGR is the number of latent factors , denoted as K . When K = 1 DiGGR degenerates into ordinary GMAE, only performing random masking over the entire input graph on the nodes. The influence of tuning K is illustrated in Figure 5. Given the relatively small size of the graphs in the dataset, the number of meaningful latent disentangled factor z is not expected to be very large. The optimal number of z that maximizes performance tends to be concentrated in the range of 2-4.

1

2

4

8

16

Factor Number K

82.5

83.5

84.5

85.5

86.5

Accuracy (%)

1

2

4

8

16

Factor Number K

86

87

88

89

90

Accuracy (%)

(b) MUTAG

1

2

4

8

16

Factor Number K

72.5

73.0

73.5

74.0

74.5

Accuracy (%)

(c) Citeseer

1

2

4

8

16

Factor Number K

72

74

76

78

80

Accuracy (%)

(d) IMDB-B

A.1.2 Representation for downstream tasks

We investigate the impact of various combinations of representation levels on downstream tasks. As 472 illustrated in Table 4, for the node classification task, both H d and H g are required, i.e. , concatenating 473 them in feature dimension, whereas for the graph classification task, H d alone is sufficient. This 474 difference may be due to the former not utilizing pooling operations, while the latter does. Specifically, 475 the graph pooling operation aggregates information from all nodes, providing a comprehensive 476 view of the entire graph structure. Thus, in node classification, where the node representation has 477 not undergone pooling, a graph-level representation ( H g ) is more critical. In contrast, in graph 478 classification, the node representation undergoes pooling, making disentangled information H d more 479 effective.

480

481

482

483

484

485

486

487

A.2 Implementation Details

Environment All experiments are conducted on Linux servers equipped with an 12th Gen Intel(R) Core(TM) i7-12700, 256GB RAM and a NVIDIA 3090 GPU. Models of node and graph classification are implemented in PyTorch version 1.12.1, scikit-learn version 1.0.2 and Python 3.7.

Experiment Setup for Node Classification The node classification task involves predicting the unknown node labels in networks. Cora, Citeseer, and Pubmed are employed for transductive learning, whereas PPI follows the inductive setup outlined in GraphSage [Hamilton et al., 2017]. For evaluation,

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

we use the concatenated representations of H d and H g in the feature dimension for the downstream task. We then train a linear classifier, report the mean accuracy on the test nodes through 5 random initializations. The graph encoder GNN enc ( . ) and decoder GNNdec ( . ) are both specified as standard GAT [Velickovic et al., 2017].We train the model using Adam Optimizer with β 1 = 0 . 9 , β 2 = 0 . 999 , ϵ = 1 × 10 8 , and we use the cosine learning rate decay without warmup. We follow the public data splits of Cora, Citeseer, and PubMed.

Experiment Setup for Graph Classification The graph classification experiment was conducted on 7 benchmarks, in which node labels are used as input features in MUTAG, PROTEINS and NCI1, and node degrees are used in IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, and COLLAB. The backbone of encoder and decoder is GIN [Xu et al., 2018], which is commonly used in previous graph classification works. The evaluation protocol primarily follows GraphMAE [Hou et al., 2022]. Notice that we only utilize the factor-wise latent representation H d for the downstream task. Subsequently, we feed it into a downstream LIBSVM [Chang and Lin, 2011] classifier to predict the label and report the mean 10-fold cross-validation accuracy with standard deviation after 5 runs. We set the initial learning rate to 0.0005 with cosine learning rate decay for most cases. For the evaluation, the parameter C of SVM is searched in the sets { 10 3 , ..., 10 } .

Data Preparation The node features for the citation networks (Cora, Citeseer, Pubmed) are bag-ofwords document representations. For the protein-protein interaction networks (PPI), the features of each node are composed of positional gene sets, motif gene sets and immunological signatures (50 in total). For graph classification, the MUTAG, PROTEINS, and NCI1 datasets utilize node labels as node features, represented in the form of one-hot encoding. For IMDB-B, IMDB-M, REDDIT-B, and COLLAB, which lack node features, we utilize the node degree and convert it into a one-hot encoding as a substitute feature. The maximum node degree is set to 400. Nodes with degrees surpassing 400 are uniformly treated as having a degree of 400, following the methodology of GraphMAE[Hou et al., 2022]. Table 5 and Table 6 show the specific statistics of used datasets.

Details for Visualization MUTAG is selected as the representative benchmark for visualization in 4.3. The MUTAG dataset comprises 3,371 nodes with seven node types. The distribution is highly skewed, as 3,333 nodes belong to three types, while the remaining four types collectively represent less than 1.2% of the nodes. For clarity in legend display, we have visualized only the nodes belonging to the first three types.

A.3 Disentangled Representations Visualization

We chose PROTEINS and IMDB-MULTI as representatives of the graph classification dataset, and followed the same methodology as in Section 4.3 to visualize their representation correlation matrices on GraphMAE, and community representation correlation matrices on DiGGR, respectively. The feature dimensions of PROTEINS and IMDB-MULTI are both 512 dimensions, and the number of communities is set to 4.

The result is presented in Figure 6. We can see from the results that the graph representations of 524 GraphMAE are entangled. In contrast, the correlation pattern exhibited by DiGGR reveals four 525 distinct diagonal blocks. This suggests that DiGGR is proficient at capturing mutually exclusive 526 information within the latent factor, resulting in disentangled representations. 527

(a) PROTEINS, GraphMAE

(b) PROTEINS, DiGGR

(c) IMDB-MULTI, GraphMAE

(d) IMDB-MULTI, DiGGR

1.0

1.0

0.8

0.8

0.6

0.6

0.4

-0.4

-0.2

-0.2

1.0

1.0

0.8

0.8

-0.6

0.6

-0.4

0.4

-0.2

-0.2

A.4 Training Algorithm 528

Algorithm 1 The Overall Training Algorithm of DiGGR

1: Input : Graph G = { V, A, X } ; latent factor number K .

2: Parameters : Θ in the inference network of Latent Factor Learning phase, Ω in the encoding network of DiGGR, Ψ in the decoding network of DiGGR.

3: Initialize Θ , Ω , and Ψ ;

4: for iter = 1,2, · ·· do

5: Infer the variational posterior of z u based on Eq. 5;

6: Sample latent factors z u from the variational posterior according to Eq. 6;

7: Factorize the graph G into K factor-wise groups {G ( k ) } K k = 1 by node and edge factorization methods;

8: Encoding {G ( k ) } K k = 1 via latent factor-wise Graph Masked Autoencoder according to Eq. 8;

9: Encoding G via graph-level graph masked autoencoder according to Eq. 11;

10: Calculate ∇ Θ , Ω , Ψ L (Θ , Ω , Ψ ; G ) according to Eq. 13, and update parameters Θ , Ω , and Ψ jointly.

11: end for =0

A.5 Broader Impacts 529

This paper presents work whose goal is to advance the field of Machine Learning. There are many 530 potential societal consequences of our work, none which we feel must be specifically highlighted 531 here. 532

533

534

535

A.6 Limitations

Despite the promising experimental justifications, our work might potentially suffer from limitation:

Although the complexity of the model is discussed in Section 3.4, and it is comparable to previously published work, extending DiGGR to extremely large graph datasets remains challenging at this stage 536

due to the incorporation of an additional probabilistic model into the generative graph framework. 537

538

One potential solution to this problem could be utilizing PPR-Nibble [Andersen et al., 2006] for efficient implementation, a method that has proven effective in some graph generative models [Hou 539

et al., 2023]. This approach will be pursued in our future work. 540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We listed our main contribution in the last paragraph of Section 1

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We've discuss about the limitations in Appendix A.6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper is not focus on theoretical explanations and assumptions

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have listed the specific settings of the experiment in the appendix A.2 of the paper, including the datasets used and the hyperparameter settings of the model. We have also uploaded the code in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have uploaded the code in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have listed the specific exprimental setup in Appendix A.2

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In the main experiment 4, we ran 5 different random seeds for each dataset and reported the average results and variances in the table6 and 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We listed the experiment environment in AppendixA.2

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with the NeurIPS Code of Ethics in every respect for this paper. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed in appendix A.5

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: this paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All these are properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper introduces DiGGR (Disentangled Generative Graph Representation Learning), a self-supervised learning framework that aims to guide graph mask modeling through disentangled latent factors to enhance the disentanglement of learned representations. Extensive experiments across 11 public datasets for node and graph classification tasks demonstrate the framework's effectiveness, significantly outperforming many existing self-supervised methods.

STRENGTHS:
Innovative Approach: The DiGGR framework innovatively utilizes disentangled latent factors to guide graph mask modeling, a novel contribution in generative graph representation learning that significantly enhances the model's explainability and robustness.
Comprehensive Experiments: The paper conducts extensive experiments on multiple datasets and tasks, showing significant performance improvements over existing methods, thus providing strong empirical support for the proposed approach.

WEAKNESSES:
Complexity and Scalability: The framework appears computationally complex, which might limit its scalability to very large graphs or real-time applications. Unfortunately, this aspect is not extensively discussed in the paper.
Lack of Theoretical Analysis: While the empirical results are strong, the paper lacks a detailed theoretical analysis of why the disentanglement process improves performance, which could provide deeper insights into the method’s efficacy and limitations.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
See Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework called DiGGR, aimed at improving the robustness and explainability of generative graph models by addressing the issue of entangled graph representations.\n\nSTRENGTHS:\n1. The paper tells the story in an easy-to-read way, and the whole paper is quite easy to follow.\n2. The problem of disentangled learning is a very popular yet important task.\n3. The paper conducts comprehensive experiments to evaluate their method.\n\nWEAKNESSES:\n1. Lack of novelty. Graph disentangled learning is not a new task. There are tons of existing methods for disentangled representation learning, such as those maximizing KL divergence or minimizing mutual information between two sets of representations. A lot of related works such as [1], [2], [3] and [4] are not discussed. Also node factorization is not a new idea, such as node clustering in [3].\n\n[1] Disentangled graph collaborative filtering. SIGIR 2020.\n[2] Disentangled Graph Convolutional Networks. ICML 2019.\n[3] Deep Generative Model for Periodic Graphs. NeurIPS 2022.\n[4] Disentangled contrastive learning on graphs. NeurIPS 2021.\n\n2. The motivation of the proposed method is not clear to me. For example, why should we use mask? Also, why the proposed method sticks to GAE, not VGAE or other types of GNN, such as GCN, GIN or GAT?\n\nQUESTIONS:\n1. Line 21: Does \"other modalities\" mean the GMAE it self, or another different kind of graph? What's the difference between GMAE and other famouse GNN such as GCN, GIN, GAT, etc?\n2. Line 37, Why should we disentangle different pieces of information of a single node? Will this help improve GNN's expressiveness?\n3. Line 120: The fulle name of EPM should be introduced before using its abbreviation.\n4. The paper mentions \"mask strategies\" for quite a few times, but I'm still confused about the purpose of masking. The paper just mentions that the proposed approach uses masking but does not tell readers the intuition behind it. Will masking produce more expressive graph representation? Will it help disentanglement? \n5. What's the difference between the proposed approach and a lot of existing approachs such as [1], [2], [3] and [4]? It seems that the proposed one factorizes nodes and edges and finally pull them together, while other techniques may rely on KL divergence or mutual information. I don't think those works are well discussed by the paper.\n\n[1] Disentangled graph collaborative filtering. SIGIR 2020.\n[2] Disentangled Graph Convolutional Networks. ICML 2019.\n[3] Deep Generative Model for Periodic Graphs. NeurIPS 2022.\n[4] Disentangled contrastive learning on graphs. NeurIPS 2021.\n\n6. I understand that the number of latent groups can be pre-defined, but can we theoretically guarantee that those groups are identifiable in terms of the learned representation? As shown in Figure1, are those groups permutational invariant for their positions?\n\nLIMITATIONS:\nYes. Limitations have been discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
The work proposes a disentangled generative self-supervised learning method for graphs. The authors introduce a latent factor learning module to capture the heterogeneous factors in the nodes. The proposed method factorizes the graph into factor-specific subgraphs, and jointly trains a disentangled Graph MAE applying distinct masks for each subgraph. Experimental results demonstrate that DiGGR outperforms traditional methods that treat the graph holistically, without accounting for its latent structure.

STRENGTHS:
1. The proposed method first explores a factorization method for generative graph SSL. 
2. The authors provide extensive experimental results and analysis on both node and graph-level tasks to show the improved effectiveness, interpretability, and generalization by using the proposed method.

WEAKNESSES:
- The computation complexity of the proposed method is quite high. Could the author pride training time comparison to the baseline methods to help us get a sense of the real complexity?
- Could the author provide more insights on how to find an optimal factor number K according to the statistics of diverse datasets? This might be useful for real-world applications.

QUESTIONS:
Please see the weakness part above.

LIMITATIONS:
Yes, the authors discussed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a self-supervised learning framework DiGGR, aimed at enhancing the disentanglement of learned graph representations. The authors argue that existing generative graph models tend to overlook the entanglement of learned representations, leading to non-robust and non-explainable models. DiGGR addresses this by introducing a latent factor learning module and a disentangled graph masked autoencoder, allowing for factor-wise graph representations. The framework is tested on various benchmarks, demonstrating its effectiveness in outperforming previous self-supervised methods.\n\nSTRENGTHS:\n1. This paper studies an interesting research problem that is disentangled graph representation learning. This research problem is very hot recently.\n\n2. The model design is easy to understand. The paper provides a detailed explanation of the proposed model.\n\n3. The experiments demonstrate the effectiveness of the model. The performance improvement on some comparisons seems to be significant.\n\nWEAKNESSES:\n1. One of my concerns is from the novelty. I think the model design is a little similar to the works [1-2]. The authors should make more comprehensive discussions to show the differences between them.\n\n2. The experiments ignore some recent or related contrastive baselines [1-4] for comparisons.  The improvements on some datasets seem to be not significant. \n\n3.  More large-scale benchmarks should also be considered, e.g., OGB. The experimental settings are not very clear for reproducing the results.\n\n[1] Disentangled contrastive learning on graphs. NeurIPS 2021.\n\n[2] Disentangled Graph Contrastive Learning With Independence Promotion. TKDE 2022.\n\n[3] Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations. TNNLS 2023.\n\n[4] MA-GCL: Model Augmentation Tricks for Graph Contrastive Learning. AAAI 2023.\n\nQUESTIONS:\nCould you provide the experimental settings for reproducing the results (e.g., hyper-parameter configurations for the model and baselines)?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}