Disentangled Generative Graph Representation Learning

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:11:09

Abstract: Recently, generative graph models have shown promising results in learning graph representations through self-supervised methods. However, most existing generative graph representation learning (GRL) approaches rely on random masking across the entire graph, which overlooks the entanglement of learned representations. This oversight results in non-robustness and a lack of explainability. Furthermore, disentangling the learned representations remains a significant challenge and has not been sufficiently explored in GRL research.
Based on these insights, this paper introduces  **DiGGR** (**Di**sentangled **G**enerative **G**raph **R**epresentation Learning), a self-supervised learning framework. DiGGR aims to learn latent disentangled factors and utilizes them to guide graph mask modeling, thereby enhancing the disentanglement of learned representations and enabling end-to-end joint learning.
Extensive experiments on 11 public datasets for two different graph learning tasks demonstrate that DiGGR consistently outperforms many previous self-supervised methods, verifying the effectiveness of the proposed approach.

Main paper: 
 introduction: 
 Self-supervised learning (SSL) has received much attention due to its appealing capacity for learning data representation without label supervision. While contrastive SSL approaches are becoming increasingly utilized on images [Chen et al., 2020] and graphs [You et al., 2020] , generative SSL has been gaining significance, driven by groundbreaking practices such as BERT for language [Devlin et al., 2018] , BEiT [Bao et al., 2021] , and MAE [He et al., 2022a] for images. Along this line, there is a growing interest in constructing generative SSL models for other modalities, such as graph masked autoencoders (GMAE). Generally, the fundamental concept of GMAE [Tan et al., 2022] is to utilize an autoencoder architecture to reconstruct input node features, structures, or both, which are randomly masked before the encoding step. Recently, various well-designed GMAEs have emerged , achieving remarkable results in both node classification and graph classification [Hou et al., 2022 , Tu et al., 2023 , Tian et al., 2023] .Despite their significant achievements, most GMAE approaches typically treat the entire graph as holistic, ignoring the graph's latent structure. As a result, the representation learned for a node tends to encapsulate the node's neighborhood as a perceptual whole, disregarding the nuanced distinctions between different parts of the neighborhood [Ma et al., 2019 , Li et al., 2021 , Mo et al., 2023] .For example, in a social network G, individual n is a member of both a mathematics group and several sports interest groups. Due to the diversity of these different communities, she may exhibit different characteristics when interacting with members from various communities. Specifically, the information about the mathematics group may be related to her professional research, while the information about sports clubs may be associated with her hobbies. However, the existing approach overlooks the heterogeneous factors of node n, failing to identify and disentangle these pieces of Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.Figure 1 : The number of latent factors is set to 4. In Fig. 1 (a), the probabilities of nodes belonging to different latent groups are similar, resulting in nodes of the same type being incorrectly assigned to different factors. In contrast, Fig. 1(b) shows that the probabilities of node-factor affiliation are more discriminative, correctly categorizing nodes of the same type into the same latent group.information effectively [Hou et al., 2022] . Consequently, the learned features may be easily influenced by irrelevant factors, resulting in poor robustness and difficulty in interpretation.To alleviate the challenge described above, there is an increasing interest in disentangled graph representation learning [Bengio et al., 2013 , Li et al., 2021 , Ma et al., 2019 , Mo et al., 2023 , Xiao et al., 2022] , which aims at acquiring representations that can disentangle the underlying explanatory factors of variation in the graph. Specifically, many of these methods rely on a latent factor detection module, which learns the latent factors of each node by comparing node representations with various latent factor prototypes. By leveraging these acquired latent factors, these models adeptly capture factor-wise graph representations, effectively encapsulating the latent structure of the graph. Despite significant progress, few studies have endeavored to adapt these methods to to generative graph representation learning methods, such as GMAE. This primary challenge arises from the difficulty of achieving convergence in the latent factor detection module under the generative training target, thus presenting obstacles in practical implementation. As shown in Fig. 1 (a), directly applying the previous factor learning method to GMAE would make the factor learning module difficult to converge, resulting in undistinguished probabilities and misallocation of similar nodes to different latent factor groups.To address these challenges, we introduce Disentangled Generative Graph Representation Learning (DiGGR), a self-supervised graph generation representation learning framework. Generally speaking, DiGGR learns how to generate graph structures from latent disentangle factors z and leverages this to guide graph mask reconstruction, while enabling end-to-end joint learning. Specifically, i) To capture the heterogeneous factors in the nodes, we introduce the latent factor learning module. This module models how edges and nodes are generated from latent factors, allowing graphs to be factorized into multiple disentangled subgraphs. ii) To learn a deeper disentangled graph representation, we design a factor-wise self-supervised graph representation learning framework. For each subgraph, we employ a distinct masking strategy to learn an improved factor-specific graph representation. Evaluation shows that the proposed framework can achieve significant performance enhancement on various node and graph classification benchmarks.The main contributions of this paper can be summarized as follows:• We utilized the latent disentangled factor to guide mask modeling. A probabilistic graph generation model is employed to identify the latent factors within a graph, and it can be jointly trained with GMAE through variational inference.• Introducing DiGGR (Disentangled Generative Graph Representation Learning) to further capture the disentangled information in the latent factors, enhancing the disentanglement of the learned node representations.• Empirical results show that the proposed DiGGR outperforms many previous self-supervised methods in various node-and graph-level classification tasks.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We listed our main contribution in the last paragraph of Section 1 Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Graph Self-Supervised Learning: Graph SSL has achieved remarkable success in addressing label scarcity in real-world network data, mainly consisting of contrastive and generative methods. Con-trastive methods, includes feature-oriented approaches [Hu et al., 2019 , Zhu et al., 2020 , Veličković et al., 2018] , proximity-oriented techniques [Hassani and Khasahmadi, 2020, You et al., 2020] , and graph-sampling-based methods [Qiu et al., 2020] . A common limitation across these approaches is their heavy reliance on the design of pretext tasks and augmentation techniques. Compared to contrastive methods, generative methods are generally simpler to implement. Recently, to tackle the challenge of overemphasizing neighborhood information at the expense of structural information [Hassani and Khasahmadi, 2020, Veličković et al., 2018] , the Graph Masked Autoencoder (GMAE) has been proposed. It applies a masking strategy to graph structure [Li et al., 2023a] , node attributes [Hou et al., 2022] , or both [Tian et al., 2023] for representation learning. Unlike most GMAEs, which employ random mask strategies, this paper builds disentangled mask strategies.Disentangled Graph Learning: Disentangled representation learning aims to discover and isolate the fundamental explanatory factors inherent in the data [Bengio et al., 2013] . Existing efforts in disentangled representation learning have primarily focused on computer vision [Higgins et al., 2017 , Jiang et al., 2020] . Recently, there has been a surge of interest in applying these techniques to graph-structured data [Li et al., 2021 , Ma et al., 2019 , Mercatali et al., 2022 , Mo et al., 2023] .For example, DisenGCN [Ma et al., 2019] utilizes an attention-based methodology to discriminate between distinct latent factors, enhancing the representation of each node to more accurately reflect its features across multiple dimensions. DGCL [Li et al., 2021] suggests learning disentangled graph-level representations through self-supervision, ensuring that the factorized representations independently capture expressive information from various latent factors. Despite the excellent results achieved by the aforementioned methods on various tasks, these methods are difficult to converge in generative graph SSL, as we demonstrated in the experiment of Table .3. Therefore, this paper proposes a disentangled-guided framework for generative graph representation learning, capable of learning disentangled representations in an end-to-end self-supervised manner.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We've discuss about the limitations in Appendix A.6 Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed method: 
 In this section, we propose DiGGR (Disentangled Generative Graph Representation Learning) for self-supervised graph representation learning with mask modeling. The framework was depicted in Figure 2 , comprises three primary components: Latent Factor Learning (Section 3.2), Graph Factorization (Section 3.2) and Disentangled Graph Masked autoencder (Section 3.3). Before elaborating on them, we first show some notations.A graph G can be represented as a multi-tuple G = {V, A, X} with N nodes and M edges, where |V | = N is the node set, |A| = M is the edge set, and X ∈ R N ×L is the feature matrix for N nodes with L dimensional feature vector. The topology structure of graph G can be found in its adjacency matrix A ∈ R N ×N . z ∈ R N ×K is the latent disentangled factor matrix, and K is the predefined factor number. Since we aim to obtain the z to guide the mask modeling, we first utilize a probabilistic graph generation model to factorize the graph before employing the mask mechanism.Given the graph G, it is factorized into {G 1 , G 2 , ..., G K }, and each factor-specific graph G k consists of its factor-specific edges A (k) , node set V (k) and node feature matrix X (k) . Other notations will be elucidated as they are employed.In this subsection, we describe the latent factor learning method. In this phase, our objective is to derive factor-specific node sets {V (1) , V (2) , ..., V (K) } and adjacency matrices {A (1) , A (2) , ..., A (K) }, serving as basic unit of the graph to guide the subsequent masking. The specific approach involves modeling the distribution of nodes and edges, utilizing the generative process developed in EPM [Zhou, 2015] . The generative process of EPM under the Bernoulli-Poisson link [Zhou, 2015] can be described as:EQUATIONwhere K is the predefined number of latent factors, and u and v are the indexes of the nodes. Here, M uv is the latent count variable between node u and v; γ k is a positive factor activation level indicator, which measures the node interaction frequency via factor k; z uk is a positive latent variable for node The input data successively passes three modules described in Sections 3.2 and 3.3: Latent Factor Learning, Graph Factorization, and Disentangled Graph Mask Autoencoder. Graph information will be first processed through Latent Factor Learning and Graph Factorization, the former processed the input graph to get the latent factor z; the latter performs graph factorization via z, such that in each factorized subgraph, nodes exchange more information with intensively interacted neighbors. Hence, during the disentangled graph masking phase, we will individually mask each factorized subgraph to enhance the disentanglement of the obtained node representations. u, which measures how strongly node u is affiliated with factor k. The prior distribution of latent factor variable z uk is set to Gamma distribution, where α and β are normally set to 1. Therefore, the intuitive explanation for this generative process is that, with z uk and z vk measuring how strongly node u and v are affiliated with the k-th factor, respectively, the product γ k z uk z vk measures how strongly nodes u and v are connected due to their affiliations with the k-th factor.Node Factorization: Equation 1 can be further augmented as follows:EQUATIONwhere M ukv represents how often nodes u and v interact due to their affiliations with the k-th factor.To represent how often node u is affiliated with the k-th factor, we further introduce the latent count M uk• = v̸ =u M ukv . Then, we can soft assign node u to multiple factors in {k : M uk• } ≥ 1, or hard assign node u to a single factor using arg max k (M uk• ). However, our experiments show that soft assignment method results in significant overlap among node sets from different factor group, diminishing the distinctiveness. Note that previous study addressed a similar issue by selecting the top-k most attended regions [Kakogeorgiou et al., 2022] . Thus, we choose the hard assign strategy to factorize the graph node set V graph into factor-specific node sets{V (1) , V (2) , • • • , V (K) }.Edge Factorization: To create factor-specific edges A (k) for a factor-specific node set V (k) , a straightforward method involves removing all external nodes connected to other factor groups. This can be defined as:EQUATIONBesides, the global graph edge A can also be factorized into positive-weighted edges [He et al., 2022b] for each latent factor as:EQUATIONApplying Equation 4to all pairs of nodes yields weighted adjacency matrices{A (k) } k k=1 , with A (k)corresponding to latent factor z k . Note that A (k) has the same dimension as A and Equation 4presents a trainable weight for each edge, which can be jointly optimized through network training,showcasing an advantage over Equation 3 in this aspect. Therefore, we apply Equation 4 for edge factorization.Variational Inference: The latent factor variable z determines the quality of node and edge factorization, so we need to approximate its posterior distribution. Denotingz u = (z u1 , ..., z uK ), z u ∈ R K + ,which measures how strongly node u is affiliated with all the K latent factors, we adopt a Weibull variational graph encoder [Zhang et al., 2018 , He et al., 2022b] :EQUATIONwhere GNN EPM (•) stands for graph neural networks, and we select a two-layer Graph Convolution Networks (i.e., GCN [Kipf and Welling, 2016a] ) for our models; k u , λ u ∈ R K + are the shape and scale parameters of the variational Weibull distribution, respectively. The latent variable z u can be conveniently reparameterized as:EQUATIONThe optimization objective of latent factor learning phase can be achieved by maximizing the evidence lower bound (ELBO) of the log marginal likelihood of edge log p(A), which can be computed as:L z = E q(Z | A,X) [ln p (A | Z)] - N u=1 E q(zu | A,X) ln q(z u | A, X) p(z u ) (7)where the first term is the expected log-likelihood or reconstruction error of edge, and the second term is the Kullback-Leibler (KL) divergence that constrains q(z u ) to be close to its prior p(z u ). The analytical expression for the KL divergence and the straightforward reparameterization of the Weibull distribution simplify the gradient estimation of the ELBO concerning the decoder parameters and other parameters in the inference network.With the latent factor learning phase discussed in 3.2, the graph can be factorized into a series of factor-specific subgraphs {G 1 , G 2 , ..., G K } via the latent factor z. To incorporate the disentangled information encapsulated in z into the graph masked autoencoder, we proposed Disentangled Graph Masked Autoencoder in this section. Specifically, this section will first introduce the latent factor-wise GMAE and the graph-level GMAE.To capture disentangled patterns within the latent factor z , for each latent subgraphG k = (V (k) , A (k) , X (k)), the latent factor-wise GMAE can be described as:EQUATIONwhere X(k) is the masked node feature matrix for the k-th latent factor, and Xd denotes the reconstructed node features. GNN enc (.) and GNN dec (.) are the graph encoder and decoder, respectively;H (k) d∈ R N ×D are factor-wise hidden representations, andH d = H (1) d ⊕ H (2) d • • • ⊕ H (K) d. After the concatenation operation ⊕ in feature dimension, the multi factor-wise hidden representation becomes D) , which is used as the input of GNN dec (.).H d ∈ R N ×(K•Regarding the mask opeartion, we uniformly random sample a subset of nodes V (k) ∈ V (k) and mask each of their features with a mask token, such as a learnable vector X [M ] ∈ R d . Thus, the node feature in the masked feature matrix can be defined as:EQUATIONThe objective of latent factor-wise GMAE is to reconstruct the masked features of nodes in V (k)given the partially observed node signals X(k) and the input adjacency matrix A (k) . Another crucial component of the GMAE is the feature reconstruction criterion, often used in language as crossentropy error [Devlin et al., 2018] and in the image as mean square error [He et al., 2022a] . However, texts and images typically involve tokenized input features, whereas graph autoencoders (GAE) do not have a universal tokenizer. We adopt the scored cosine error of GraphMAE [Hou et al., 2022] as the loss function. Generally, given the original feature X (k) and reconstructed node feature X(k) , the defined SCE is:EQUATION)whereV = V (1) ∪ V (2) ... ∪ V (K)and Equation 10are averaged over all masked nodes.The scaling factor γ is a hyper-parameter adjustable over different datasets. This scaling technique could also be viewed as adaptive sample reweighting, and the weight of each sample is adjusted with the reconstruction error. This error is also famous in the field of supervised object detection as the focal loss [Lin et al., 2017] .Graph-level Graph Mask Autoencoder: For the node classification task, we have integrated graph-level GMAE into DiGGR. We provide a detailed experimental analysis and explanation for this difference in Appendix A.1.2. The graph-level masked graph autoencoder is designed with the aim of further capturing the global patterns, which can be designed as:EQUATIONX is the masked node feature matrix, whose mask can be generated by uniformly random sampling a subset of nodes Ṽ ∈ V , or obtained by concatenating the masks of all factor-specific groupsṼ = V (1) ∪ V (2) ... ∪ V (K) .The global hidden representation encoded by GNNenc(.) is H g , which is then passed to the decoder. Similar to Equation 10, we can define the graph-level reconstruct loss as:EQUATIONwhich is averaged over all masked nodes.Benefiting from the effective variational inference method, the proposed latent factor learning and dsientangled graph masked autoencoder can be jointly trained in one framework. We combine the aforementioned losses with three mixing coefficient λ d , λ g and λ z during training, and the loss for joint training can be written asEQUATIONSince Weibull distributions have easy reparameterization functions, these parameters can be jointly trained by stochastic gradient descent with low-variance gradient estimation. We summarize the training algorithm at Algorithm 1 in Appendix A.4. For downstream applications, the encoder is applied to the input graph without any masking in the inference stage. Time and space complexity: Let's recall that in our context, N , M , and K represent the number of nodes, edges, and latent factors in the graph, respectively. The feature dimension is denoted by F , while L 1 , L 2 , L 3 , and L 4 represent the number of layers in the latent factor learning encoder, the latent factor-wise GMAE's encoder, the graph-level GMAE's encoder, and the decoder respectively.In DiGGR, we constrain the hidden dimension size in latent factor-wise GMAE's encoder to be 1/K of the typical baseline dimensions. Consequently, the time complexity for training DiGGR can be expressed as O((L 1 + L 2 + L 3 )M F + (L 1 + L 2 /K + L 3 )N F 2 + N 2 F + L 4 N F 2 ), and the space complexity is O((L 1 + L 2 + L 3 + L 4 )N F + KM + (L1 + L2/K + L3 + L 4 )F 2 ), with O((L 1 + L 2 /K + L 3 + L 4 )F 2) attributed to model parameters. We utilize the Bayesian factor model in our approach to reconstruct edges. Its time complexity aligns with that of variational inference in SeeGera Li et al. [2023b] , predominantly at O(N 2 F ); Therefore, the complexity of DiGGR is comparable to previous works. [Velickovic et al., 2017] 83.00 ± 0.70 72.50 ± 0.70 79.00 ± 0.30 97.30 ± 0.20 DisenGCN [Ma et al., 2019] 83.7 73.4 80.5 -VEPM [He et al., 2022b] 84.3 ± 0.1 72.5 ± 0.1 82.4 ± 0.2 -MVGRL [Hassani and Khasahmadi, 2020] 83.50 ± 0.40 73.30 ± 0.50 80.10 ± 0.70 -InfoGCL [Xu et al., 2021] 83.50 ± 0.30 73.50 ± 0.40 79.10 ± 0.20 -DGI [Veličković et al., 2018] 82.30 ± 0.60 71.80 ± 0.70 76.80 ± 0.60 63.80 ± 0.20 GRACE [Zhu et al., 2020] 81.90 ± 0.40 71.20 ± 0.50 80.60 ± 0.40 69.71 ± 0.17 BGRL [Thakoor et al., 2021] 82.70 ± 0.60 71.10 ± 0.80 79.60 ± 0.50 73.63 ± 0.16 CCA-SSG [Zhang et al., 2021] 84.20 ± 0.40 73.10 ± 0.30 81.00 ± 0.40 73.34 ± 0.17 76.30 ± 0.20 66.80 ± 0.20 75.80 ± 0.40 -Bandana [Zhao et al., 2024] 84.62 ± 0.37 73.60 ± 0.16 83.53 ± 0.51 -GiGaMAE [Shi et al., 2023] 84.72 ± 0.47 72.31 ± 0.50 --SEEGERA [Shi et al., 2023] 84.30 ± 0.40 73.00 ± 0.80 80.40 ± 0.40 -GraphMAE [Hou et al., 2022] 84.20 ± 0.40 73.40 ± 0.40 81.10 ± 0.40 74.50 ± 0.29 GraphMAE2 [Hou et al., 2023] 84 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We have uploaded the code in the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We compare the proposed self-supervised framework DiGGR against related baselines on two fundamental tasks: unsupervised representation learning on node classification and graph classification.We evaluate DiGGR on 11 benchmarks. For node classification, we use 3 citation networks (Cora, Citeseer, Pubmed [Yang et al., 2016] ), and protein-protein interaction networks (PPI) [Hamilton et al., 2017] . For graph classification, we use 3 bioinformatics datasets (MUTAG, NCI1, PROTEINS) and 4 social network datasets (IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY and COLLAB). The specific information of the dataset and the hyperparameters used by the network are listed in the Appendix A.2 in table 5 and 6. We also provide the detailed experiment setup in Appendix A.2 for node classification (4.1) and graph classification (4.2)The baseline models for node classification can be divided into three categories: i) supervised methods,including GCN [Kipf and Welling, 2016a] , DisenGCN [Ma et al., 2019] , VEPM [He et al., 2022b] and GAT [Velickovic et al., 2017] ; ii) contrastive learning methods, including MVGRL [Hassani and Khasahmadi, 2020] , InfoGCL [Xu et al., 2021] , DGI [Veličković et al., 2018] , GRACE [Zhu et al., 2020] , BGRL [Thakoor et al., 2021] and CCA-SSG [Zhang et al., 2021] ; iii) generative learning methods, including GraphMAE [Hou et al., 2022] , GraphMAE2 [Hou et al., 2023] , Bandana [Zhao et al., 2024] , GiGaMAE [Shi et al., 2023] , SeeGera [Li et al., 2023b] , GAE and VGAE [Kipf andWelling , 2016b] . The node classification results were listed in Table 1 . DiGGR demonstrates competitive results on the provided dataset, achieving results comparable to those of supervised methods.Baseline Models We categorized the baseline models into four groups: i) supervised methods, including GIN [Xu et al., 2018] , DiffPool [Ying et al., 2018] and VEPM [He et al., 2022b] ; ii) classical graph kernel methods: Weisfeiler-Lehman sub-tree kernel (WL) [Shervashidze et al., 2011] and deep graph kernel (DGK) [Yanardag and Vishwanathan, 2015] ; iii) contrastive learning methods,including GCC [Qiu et al., 2020] , graph2vec [Narayanan et al., 2017] , Infograph [Sun et al., 2019] ,GraphCL [You et al., 2020] , JOAO [You et al., 2021] , MVGRL [Hassani and Khasahmadi, 2020] ,and InfoGCL [Xu et al., 2021] ; 4) generative learning methods, including graph2vec [Narayanan et al., 2017] , sub2vec [Adhikari et al., 2018] , node2vec [Grover and Leskovec, 2016] , GraphMAE [Hou et al., 2022] , GraphMAE2 [Hou et al., 2023] , GAE and VGAE [Kipf and Welling, 2016b] . Per graph classification research tradition, we report results from previous papers if available. Performance Comparison The graph classification results are presented in Table 2 . In general, we find that DiGGR gained the best performance among other baselines on five out of seven datasets, while achieving competitive results on the other two datasets. The performance of DiGGR is comparable to that of supervised learning methods. For instance, the accuracy on IMDB-B and IMDB-M surpasses that of GIN and DiffPool. Moreover, within the reported datasets, our method demonstrates improved performance compared to random mask methods like GraphMAE, particularly on the IMDB-M, COLLAB, and PROTEINS datasets. This underscores the effectiveness of the proposed method.Visualizing latent representations To examine the influence of the learned latent factor on classification results, we visualized the latent disentangled factor z, which reflects the node-factor affiliation, and the hidden representation H used for classification. MUTAG is selected as the representative for classification benchmarks. We encodes the representations into 2-D space via t-SNE [Van der Maaten and Hinton, 2008] . The result is shown in Figure 3 Task-relevant factors To assess the statistical correlation between the learned latent factor and the task, we follow the approach in [He et al., 2022b] and compute the Normalized Mutual Information (NMI) between the nodes in the factor label and the actual node labels. NMI is a metric that ranges from 0 to 1, where higher values signify more robust statistical dependencies between two random variables. In the experiment, we utilized the MUTAG dataset, comprising 7 distinct node types, and the NMI value we obtained was 0.5458. These results highlight that the latent factors obtained through self-supervised training are meaningful for the task, enhancing the correlation between the inferred latent factors and the task.To assess DiGGR's capability to disentangle the learned representation for downstream task, we provide a qualitative evaluation by plotting the correlation of the .3. As the results show, DiGGR's performance improvement has a positive correlation with disentangled degree, where the better the disentangled degree, the more significant the performance improvement. For methods relying on Non-probabilistic Factor Learning, the NMI tends to approach 1. This is attributed to the challenges faced by the factor learning module in converging, thereby hindering the learning of distinct latent factors. The presence of confused latent factors offers misleading guidance for representation learning, consequently leading to decreased performance. 
conclusions: 
 In  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We have listed the specific exprimental setup in Appendix A.2 Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: In the main experiment 4, we ran 5 different random seeds for each dataset and reported the average results and variances in the table6 and 5.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We listed the experiment environment in AppendixA.2 Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We conform with the NeurIPS Code of Ethics in every respect for this paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We have discussed in appendix A.5 Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. 
 
 Appendix: Optionally include supplemental material (complete proofs, additional experiments and plots) in appendix. All such materials SHOULD be included in the main submission.One of the crucial hyperparameters in DiGGR is the number of latent factors, denoted as K. When K = 1 DiGGR degenerates into ordinary GMAE, only performing random masking over the entire input graph on the nodes. The influence of tuning K is illustrated in Figure 5 . Given the relatively small size of the graphs in the dataset, the number of meaningful latent disentangled factor z is not expected to be very large. The optimal number of z that maximizes performance tends to be concentrated in the range of 2-4. We investigate the impact of various combinations of representation levels on downstream tasks. As illustrated in Table 4 , for the node classification task, both H d and H g are required, i.e., concatenating them in feature dimension, whereas for the graph classification task, H d alone is sufficient. This difference may be due to the former not utilizing pooling operations, while the latter does. Specifically, the graph pooling operation aggregates information from all nodes, providing a comprehensive view of the entire graph structure. Thus, in node classification, where the node representation has not undergone pooling, a graph-level representation (H g ) is more critical. In contrast, in graph classification, the node representation undergoes pooling, making disentangled information H d more effective. Experiment Setup for Node Classification The node classification task involves predicting the unknown node labels in networks. Cora, Citeseer, and Pubmed are employed for transductive learning, whereas PPI follows the inductive setup outlined in GraphSage [Hamilton et al., 2017] . For evaluation, we use the concatenated representations of H d and H g in the feature dimension for the downstream task. We then train a linear classifier, report the mean accuracy on the test nodes through 5 random initializations. The graph encoder GNN enc(.) and decoder GNN dec (.) are both specified as standard GAT [Velickovic et al., 2017] .We train the model using Adam Optimizer with β 1 = 0.9, β 2 = 0.999, ϵ = 1 × 10 8 , and we use the cosine learning rate decay without warmup. We follow the public data splits of Cora, Citeseer, and PubMed.Experiment Setup for Graph Classification The graph classification experiment was conducted on 7 benchmarks, in which node labels are used as input features in MUTAG, PROTEINS and NCI1, and node degrees are used in IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, and COLLAB. The backbone of encoder and decoder is GIN [Xu et al., 2018] , which is commonly used in previous graph classification works. The evaluation protocol primarily follows GraphMAE [Hou et al., 2022] . Notice that we only utilize the factor-wise latent representation H d for the downstream task. Subsequently, we feed it into a downstream LIBSVM [Chang and Lin, 2011] classifier to predict the label and report the mean 10-fold cross-validation accuracy with standard deviation after 5 runs. We set the initial learning rate to 0.0005 with cosine learning rate decay for most cases. For the evaluation, the parameter C of SVM is searched in the sets {10 3 , ..., 10}.The node features for the citation networks (Cora, Citeseer, Pubmed) are bag-ofwords document representations. For the protein-protein interaction networks (PPI), the features of each node are composed of positional gene sets, motif gene sets and immunological signatures (50 in total). For graph classification, the MUTAG, PROTEINS, and NCI1 datasets utilize node labels as node features, represented in the form of one-hot encoding. For IMDB-B, IMDB-M, REDDIT-B, and COLLAB, which lack node features, we utilize the node degree and convert it into a one-hot encoding as a substitute feature. The maximum node degree is set to 400. Nodes with degrees surpassing 400 are uniformly treated as having a degree of 400, following the methodology of GraphMAE [Hou et al., 2022] . Table 5 and Table 6 show the specific statistics of used datasets.Details for Visualization MUTAG is selected as the representative benchmark for visualization in 4.3. The MUTAG dataset comprises 3,371 nodes with seven node types. The distribution is highly skewed, as 3,333 nodes belong to three types, while the remaining four types collectively represent less than 1.2% of the nodes. For clarity in legend display, we have visualized only the nodes belonging to the first three types.We chose PROTEINS and IMDB-MULTI as representatives of the graph classification dataset, and followed the same methodology as in Section 4.3 to visualize their representation correlation matrices on GraphMAE, and community representation correlation matrices on DiGGR, respectively. The feature dimensions of PROTEINS and IMDB-MULTI are both 512 dimensions, and the number of communities is set to 4. The result is presented in Figure 6 . We can see from the results that the graph representations of GraphMAE are entangled. In contrast, the correlation pattern exhibited by DiGGR reveals four distinct diagonal blocks. This suggests that DiGGR is proficient at capturing mutually exclusive information within the latent factor, resulting in disentangled representations. Encoding {G (k) } K k=1 via latent factor-wise Graph Masked Autoencoder according to Eq. 8;9:Encoding G via graph-level graph masked autoencoder according to Eq. 11;10:Calculate ∇ Θ,Ω,Ψ L(Θ, Ω, Ψ; G) according to Eq. 13, and update parameters Θ, Ω, and Ψ jointly. 11: end for=0This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.Despite the promising experimental justifications, our work might potentially suffer from limitation:Although the complexity of the model is discussed in Section 3.4, and it is comparable to previously published work, extending DiGGR to extremely large graph datasets remains challenging at this stage due to the incorporation of an additional probabilistic model into the generative graph framework.One potential solution to this problem could be utilizing PPR-Nibble [Andersen et al., 2006] for efficient implementation, a method that has proven effective in some graph generative models [Hou et al., 2023] . This approach will be pursued in our future work.• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The scores of this paper are mixed. After reading the paper and the authors' rebuttals, I decided to reject this paper due to its incremental contributions to methodology and performance. 

Firstly, as the reviewers pointed out, there are many existing disentangled graph representation learning methods, while few of them are treated as baselines in this work, which makes the novelty of the proposed method questionable. Although the authors highlighted the differences between the proposed method and the existing ones, the claims were not verified by sufficient experiments. Especially, a comprehensive comparison with the method in "Deep Generative Model for Periodic Graphs. NeurIPS 2022" is necessary because the main difference between the proposed method and this one is the distribution assumption of latent factors. 

Secondly, considering the change in the averaged result and the corresponding standard deviation, the performance improvements caused by the proposed method are incremental in most situations. The additional results shown in the rebuttal phase did not resolve this concern successfully. In addition, according to the figures shown in the paper and the pdf file provided in the rebuttal phase, the disentangled latent factors seem to imply clustered nodes, which is questionable for representing heterophilic graphs (and in my opinion, it might be the reason for the incremental performance improvements, and I strongly suggest the authors add analytic experiments to verify whether such a correlation exists and what a kind of impact it has on the model performance). 

In addition, there are many typos in the paper, e.g., the "grpah" in the titles of Sections 3.3 and 3.3.1.

In summary, although no comprehensive discussions happened in the rebuttal phase (As AC, I am sorry to the authors, but I have tried my best to remind the reviewers), based on the analysis above, I suggest the authors polish this work and resubmit it in the future.

Reviews: review: paper_summary: paper_summary: The paper proposes a self-supervised learning framework DiGGR, aimed at enhancing the disentanglement of learned graph representations. The authors argue that existing generative graph models tend to overlook the entanglement of learned representations, leading to non-robust and non-explainable models. DiGGR addresses this by introducing a latent factor learning module and a disentangled graph masked autoencoder, allowing for factor-wise graph representations. The framework is tested on various benchmarks, demonstrating its effectiveness in outperforming previous self-supervised methods.
strength_weakness: strength_weakness: 1. One of my concerns is from the novelty. I think the model design is a little similar to the works [1-2]. The authors should make more comprehensive discussions to show the differences between them.

2. The experiments ignore some recent or related contrastive baselines [1-4] for comparisons.  The improvements on some datasets seem to be not significant. 

3.  More large-scale benchmarks should also be considered, e.g., OGB. The experimental settings are not very clear for reproducing the results.

[1] Disentangled contrastive learning on graphs. NeurIPS 2021.

[2] Disentangled Graph Contrastive Learning With Independence Promotion. TKDE 2022.

[3] Augmentation-Free Graph Contrastive Learning of Invariant-Discriminative Representations. TNNLS 2023.

[4] MA-GCL: Model Augmentation Tricks for Graph Contrastive Learning. AAAI 2023.
questions: questions: Could you provide the experimental settings for reproducing the results (e.g., hyper-parameter configurations for the model and baselines)?
limitations: limitations: n/a
paper_summary: paper_summary: The work proposes a disentangled generative self-supervised learning method for graphs. The authors introduce a latent factor learning module to capture the heterogeneous factors in the nodes. The proposed method factorizes the graph into factor-specific subgraphs, and jointly trains a disentangled Graph MAE applying distinct masks for each subgraph. Experimental results demonstrate that DiGGR outperforms traditional methods that treat the graph holistically, without accounting for its latent structure.
strength_weakness: strength_weakness: - The computation complexity of the proposed method is quite high. Could the author pride training time comparison to the baseline methods to help us get a sense of the real complexity?
- Could the author provide more insights on how to find an optimal factor number K according to the statistics of diverse datasets? This might be useful for real-world applications.
questions: questions: Please see the weakness part above.
limitations: limitations: Yes, the authors discussed the limitations.
paper_summary: paper_summary: The paper introduces a framework called DiGGR, aimed at improving the robustness and explainability of generative graph models by addressing the issue of entangled graph representations.
strength_weakness: strength_weakness: 1. Lack of novelty. Graph disentangled learning is not a new task. There are tons of existing methods for disentangled representation learning, such as those maximizing KL divergence or minimizing mutual information between two sets of representations. A lot of related works such as [1], [2], [3] and [4] are not discussed. Also node factorization is not a new idea, such as node clustering in [3].

[1] Disentangled graph collaborative filtering. SIGIR 2020.
[2] Disentangled Graph Convolutional Networks. ICML 2019.
[3] Deep Generative Model for Periodic Graphs. NeurIPS 2022.
[4] Disentangled contrastive learning on graphs. NeurIPS 2021.

2. The motivation of the proposed method is not clear to me. For example, why should we use mask? Also, why the proposed method sticks to GAE, not VGAE or other types of GNN, such as GCN, GIN or GAT?
questions: questions: 1. Line 21: Does "other modalities" mean the GMAE it self, or another different kind of graph? What's the difference between GMAE and other famouse GNN such as GCN, GIN, GAT, etc?
2. Line 37, Why should we disentangle different pieces of information of a single node? Will this help improve GNN's expressiveness?
3. Line 120: The fulle name of EPM should be introduced before using its abbreviation.
4. The paper mentions "mask strategies" for quite a few times, but I'm still confused about the purpose of masking. The paper just mentions that the proposed approach uses masking but does not tell readers the intuition behind it. Will masking produce more expressive graph representation? Will it help disentanglement? 
5. What's the difference between the proposed approach and a lot of existing approachs such as [1], [2], [3] and [4]? It seems that the proposed one factorizes nodes and edges and finally pull them together, while other techniques may rely on KL divergence or mutual information. I don't think those works are well discussed by the paper.

[1] Disentangled graph collaborative filtering. SIGIR 2020.
[2] Disentangled Graph Convolutional Networks. ICML 2019.
[3] Deep Generative Model for Periodic Graphs. NeurIPS 2022.
[4] Disentangled contrastive learning on graphs. NeurIPS 2021.

6. I understand that the number of latent groups can be pre-defined, but can we theoretically guarantee that those groups are identifiable in terms of the learned representation? As shown in Figure1, are those groups permutational invariant for their positions?
limitations: limitations: Yes. Limitations have been discussed in the paper.
paper_summary: paper_summary: The paper introduces DiGGR (Disentangled Generative Graph Representation Learning), a self-supervised learning framework that aims to guide graph mask modeling through disentangled latent factors to enhance the disentanglement of learned representations. Extensive experiments across 11 public datasets for node and graph classification tasks demonstrate the framework's effectiveness, significantly outperforming many existing self-supervised methods.
strength_weakness: strength_weakness: Complexity and Scalability: The framework appears computationally complex, which might limit its scalability to very large graphs or real-time applications. Unfortunately, this aspect is not extensively discussed in the paper.
Lack of Theoretical Analysis: While the empirical results are strong, the paper lacks a detailed theoretical analysis of why the disentanglement process improves performance, which could provide deeper insights into the method’s efficacy and limitations.
questions: questions: See Weaknesses.
limitations: limitations: See Weaknesses.

Comments: title: Further reply
We sincerely appreciate your efforts in reviewing the manuscript and hope our response will address your concerns effectively.
reviewer response
comment: Dear Reviewer DL9g,

I hope this email finds you well. I wanted to follow up on the responses I provided two days ago to the remaining questions and concerns you raised regarding our submission .

As the deadline is approaching, I wanted to kindly ask if the clarifications have addressed your concerns. If there are any outstanding issues, please let me know, and I would be happy to address them as quickly as possible.

If the responses were satisfactory, I would greatly appreciate it if you could reconsider the evaluation of our submission.

Thank you again for your time and thoughtful feedback.

Best regards,

Paper 17865 Authors
### **Q4**:
This is an insightful question, and we would like to explain it from the following two perspectives:

1. We have not overlooked permutation invariance in the task. Rather than explicitly enforcing it in the model, we take a data-driven approach to determine permutation invariance within the latent groups identified by DiGGR. Specifically, in our model, we define the log-likelihood of the adjacency matrix as the optimization function (see Equation 7). 
According to the theorem in [3], the gradient of the log-likelihood estimation, $\nabla_A \log p(A)$, is permutation equivariant, meaning that the log-likelihood function $\log p(A)$ is inherently permutation invariant. By optimizing latent factor learning using the ELBO of $\log p(A)$, we implicitly enforce permutation invariance in our latent factor.

2. Our model is primarily designed for classification tasks, where no operations in the network involve permutation, and all the adjacency matrices stick to a specific and intrinsic node ordering.
If the model is extended to graph generation tasks in the future, we can adopt the approach in [1], which uses a fixed permutation to constrain the adjacency matrix A to a specific node ordering, thereby limiting the model's space complexity to $Q(n^2)$.

[1] Deep Generative Model for Periodic Graphs. NeurIPS 2022.

[2] Multi-objective Deep Data Generation with Correlated Property Control. NeurIPS 2022.

[3] Permutation Invariant Graph Generation via Score-Based Generative Modeling. AISTATS 2020.

[4] NON-NEGATIVE CONTRASTIVE LEARNING. ICLR 2024.

[5] Nonnegative Matrix Factorization: A Comprehensive Review. TKDE 2013.

[6] Lee, Daniel D., and H. Sebastian Seung. "Learning the parts of objects by non-negative matrix factorization." nature 401.6755 (1999): 788-791.

[7] WHAI: WEIBULL HYBRID AUTOENCODING INFERENCE FOR DEEP TOPIC MODELING. ICLR 2018

[8] A variational edge partition model for supervised graph representation learning. NeurIPS 2022

[9] Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." arXiv preprint arXiv:1312.6114 (2013).

[10] Latent dirichlet allocation. NeurIPS 2001

[11] Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology 58.1 (1996): 267-288.

[12] Learning structured output representation using deep conditional generative models. NeurIPS 2015
### **Q1 and Q2**: 
Thank you for your valuable advice. We will incorporate the benefits of using the Gamma distribution in the next revision. Essentially, we model $z$ as a Gamma distribution rather than a Gaussian distribution for the following reasons:

1. Compared to the Gaussian distribution, the Gamma distribution possesses **non-negative** and **sparse** properties, both of which we leverage in our approach.

a) Based on the non-negative characteristic, the latent factor learning can be converted into **non-negative** matrix factorization (NMF). Unlike canonical matrix factorization, NMF ensures that different features will not cancel one another out when calculating feature similarity [4]. To boot, the non-negativity constraint will lead to sort of sparseness naturally, which is proved to be a highly effective representation distinguished from both the completely distributed and the solely active component description [5]. According to [6], such property inherently benefits the learning of disentangled representation, which is unattainable with canonical matrix factorization under a Gaussian distribution.

b) The sparsity induced by the Gamma distribution is akin to imposing an additional sparsity constraint on NMF. This constraint aids in enhancing the uniqueness of the decomposition while enforcing a locality-based representation, making the factorization sparser and improving its robustness against potential offsets caused by additive noise [6].

 
2. **Training Stability of Gamma**: We adopted the variational inference method proposed in [7], using the Weibull distribution to approximate the Gamma distribution for reparameterization. This method has been shown to enable stable training in graph networks [8]. We want to emphasize that the sparsity introduced by the Gamma distribution does not compromise the model's performance. On the contrary, recent studies [4, 5, 7] have shown that the increased feature sparsity enhances the network's nonlinearity, leading to improved model expressiveness and generalization ability. (Note that non-negative values are common in deep learning models, particularly as outputs from non-linear functions like ReLU.  These non-linear functions can enhance the model's expressive power and generalization ability by increasing its nonlinearity, rather than bringing information loss.)

3. **Why Choose Gamma instead of L1**: 
Gamma distribution introduces probabilistic sparsity, which has the following two benefits : 

a)Sparsity can be controlled more flexibly by adjusting parameters such as the shape and scale of the Gamma distribution[9]. Unlike the L1 norm, which forces some parameters to zero, this method allows for a balanced trade-off between sparsity and information retention[10], reducing the risk of information loss due to excessive sparsification.
 
b)The optimization process of Gamma's probabilistic sparsity is smoother than that of the L1 norm[12], which causes non-smoothness and issues like "dead neurons" as parameters approach zero[11]. The sparsity introduced by Gamma can improve convergence and reduce the risk of local minima or unstable solutions[9]. 

c) Probabilistic sparsity can provide nonlinearity, analogous to a Relu activation function, which is often helpful for the model's expressiveness. However, the L1 norm often hurts the original model's expressiveness.

### **Q3**
Similar to other works [1, 2, 8], we evaluate the expressiveness of our framework using various downstream task metrics. In Tables 1 and 2 of this paper, we use accuracy for comparison. In the global response, we specifically use the Gamma latent variable model for classification and compare it with the Gaussian latent variable model from [1] on downstream tasks. These comparisons demonstrate the expressiveness of the gamma latent variable model.  In future revisions, we will also include experimental results on the link prediction task to more comprehensively validate the gamma latent variable model's expressiveness.  


| Model   | IMDB-BINARY (%) | MUTAG (%)   |
|---------|-----------------|-------------|
| PGD-VAE | 54.37 ± 0.21     | 84.06 ± 0.52|
|  Gamma latent variable model | 70.79 ± 0.34     | 85.12 ± 0.64|
Thank the author or the time on rebuttal. I think they partially addressed my concern. As a result, I'm willing to increase my score. Here are the remaining questions:

1. I don't the treating $z$ as a distribution has any difference from existing works that rely on VAE, such as [1] and [2], since VAE also treats latents as random varibles that are supposed to follow Gaussian distribution. Basically, latents are sampled from Gaussian in VAE. 

2. I don't see obvious benefits of modeling latents using Gamma distribution, as even for regular matrix that has values that can be either negative or positive, we can still perform matrix factorization, such as SVM. I would suggest author to clearly explain this in the paper. Also, I think with Gamma distribution assumption, e.g. the last layer of encoder, need to produce non-negative values then. Will that cause any instability during training? Also, it would be great if author could explain the benefits brought by the sparsity of Gamma distribution. Will this cause loss of information? Why not just use L1 norm to enforce the sparsity?

3. For Q2, are there mathematical metrics to measure the expressiveness of GNN? If so, I believe involving those metrics in experiments would be more persuasive.

4. Ignoring permutation invariance will introduce huge complexity for model parameters to well capture all kinds of graphs. If there are $n$ nodes of a graph, I guess it's $O(n!)$ to permute those nodes for a graph generation task.

[1] Deep Generative Model for Periodic Graphs. NeurIPS 2022.

[2] Multi-objective Deep Data Generation with Correlated Property Control. NeurIPS 2022.
Thank you for your suggestion. We will include the official dataset link and the corresponding references in future revisions.