Disentangled and Self-Explainable Node Representation Learning

Authors: Simone Piaggesi, André Panisson, Megha Khosla
Primary area: interpretability and explainable AI
Keywords: Node representation learning, explainable ai, disentangled learning

Abstract:
Node representations, or embeddings, are low-dimensional vectors that capture node properties, typically learned through unsupervised structural similarity objectives or supervised tasks. While recent efforts have focused on explaining graph model decisions, the interpretability of $\textit{unsupervised}$ node embeddings remains underexplored. To bridge this gap, we introduce DiSeNE ($\textbf{Di}$sentangled and $\textbf{Se}$lf-Explainable $\textbf{N}$ode $\textbf{E}$mbedding), a framework that generates self-explainable embeddings in an unsupervised manner. Our method employs disentangled representation learning to produce dimension-wise interpretable embeddings, where each dimension is aligned with distinct topological structure of the graph. We formalize novel desiderata for disentangled and interpretable embeddings, which drive our new objective functions, optimizing simultaneously for both interpretability and disentanglement. Additionally, we propose several new metrics to evaluate representation quality and human interpretability. Extensive experiments across multiple benchmark datasets demonstrate the effectiveness of our approach.

Decision: reject
Reviewer scores: 5, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730326370541

thread_type:
official_review

<thread_object>
creator: reviewer_y413

<content>
summary:
This paper introduces a framework that generates self-explainable embeddings in an unsupervised manner. The method employs disentangled representation learning to produce dimension-wise interpretable embeddings, where each dimension is aligned with distinct topological structure of the graph. The paper drives new objective functions and optimizes simultaneously for both interpretability and disentanglement. Additionally, the paper proposes new metrics to evaluate representation quality and human interpretability.

strengths:
1. The paper formalizes new and essential criteria for achieving disentangled and explainable node representations
2. The paper introduces novel evaluation metrics to help quantifying the goodness of node representation learning

weaknesses:
1. The paper introduces comprehensibility as a metric for evaluating explanations. However, how does the method perform on widely-used explanation metrics like AUC, Fidelity-, and Fidelity+? I strongly recommend that the authors also consider including these popular metrics to thoroughly assess the method's performance.

2. The paper proposes using comprehensibility and sparsity to enhance human interpretability. This raises a natural question: do these metrics truly support human understandability? I strongly recommend that the authors conduct human evaluations to validate that these metrics align with human comprehension.

3. The baselines are too weak. The authors compare with the node embedding methods: DeepWalk, Deep AE, DISE-GAE, DISE-FACE. I suggest the authors consider more modern methods to have the node embedding, such as normalizing flows [R1], diffusion models [R2], self-supervised methods [R3].

[R1] RicciNet: Deep Clustering via A Riemannian Generative Model, WWW 2024

[R2] Directional diffusion models for graph representation learning, NeurIPS 2024

[R3] GraphMAE: Self-Supervised Masked Graph Autoencoders, KDD 2022.

4. The method is self-explainable, but how effective is the visualization of its explanations? I suggest that the authors include a case study to show the visualization.

5. The paper claims utility for downstream tasks like link prediction and node classification. However, the baselines used for evaluating these tasks are relatively weak. I recommend that the authors consider incorporating more widely-used contrastive learning methods as baselines, such as [R4], to assess whether the proposed node embeddings achieve superior performance in these downstream tasks.

[R4]. Provable Training for Graph Contrastive Learning, NeurIPS 2024.

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601029482

thread_type:
official_review

<thread_object>
creator: reviewer_x1gw

<content>
summary:
The paper introduces DISENE, a novel approach to self-explanatory node embedding, addressing the increasing need for interpretability in graph representation learning. DISENE’s design emphasizes disentangled representations, allowing each embedding dimension to independently capture specific, non-overlapping structural features of the graph. This self-explanatory capability is achieved through a combination of disentangled representation learning and the embedding of interpretability directly within the model architecture, moving beyond traditional post-hoc interpretability approaches.

strengths:
1. Innovative Self-Explanatory Node Embedding Method: This paper introduces the DISENE model, which leverages disentanglement and self-explanatory design to enable each embedding dimension to correlate directly with specific structural features within the graph. 
2. Disentangled Feature Representation: DISENE ensures that each embedding dimension independently captures a unique graph structure feature, minimizing overlap across dimensions. This disentangled representation improves both the interpretability and robustness of the embeddings, providing more granular structural information for downstream applications that require feature independence.

weaknesses:
1. Reliance on Interpretability Metrics with Ambiguous Definitions: The paper relies heavily on various interpretability metrics, such as consistency and sparsity, to evaluate model performance. However, these metrics lack clear, systematic definitions in the text. For instance, Section 4.2: Evaluation Metrics introduces these terms conceptually but fails to provide precise mathematical definitions or specific calculation methodologies. This lack of clarity may lead to inconsistencies in metric implementation across studies, thereby impacting the reproducibility of the results and the fairness of comparisons. For interpretability metrics to serve as robust evaluation tools, a more rigorous definition and operationalization are essential.
2. Ambiguity in Interpretability Metric Definitions: While the paper presents novel interpretability metrics, it leaves substantial gaps in detailing how these metrics are quantified. Section 3.4: Explanation and Evaluation discusses the interpretability aspects of the model but lacks the necessary formalism and computation process, making it challenging to assess the reliability and generalizability of these interpretability metrics. This ambiguity undermines the objective measurement of the model’s self-explanatory capabilities, calling into question the effectiveness of the proposed interpretability criteria.
3. Lack of Competitive Edge in Link Prediction and Node Classification Tasks: DISENE exhibits only moderate performance in standard tasks like link prediction and node classification, without showcasing significant improvements. As reported in Table 3: Link Prediction Results and Table 4: Node Classification Results, DISENE’s accuracy and AUC scores do not exhibit a marked advantage over other established models. This highlights a trade-off between interpretability and predictive power, suggesting that while DISENE makes strides in interpretability, it may fall short in delivering competitive performance in traditional graph embedding tasks, limiting its overall efficacy in practical applications.
4. Limited Dataset Scope, Focused on Small-Scale Citation Networks: The paper’s empirical analysis relies heavily on established citation networks such as Cora, CiteSeer, and PubMed (Section 4.1: Datasets). While these datasets are standard in graph embedding research, they are relatively small and structurally simple, which raises concerns about the model’s generalizability to larger, more complex graph structures. The limited dataset scope restricts the evaluation of DISENE’s performance on diverse, real-world graphs and challenges the model’s claims of broad applicability.
5. Outdated Baseline Comparisons: The baseline models selected for comparison—primarily traditional graph embedding methods such as DeepWalk, node2vec, and GraphSAGE (Section 4.3: Baselines)—do not reflect the recent advancements in graph representation learning. Given the rapid evolution of graph neural networks, the lack of comparisons with modern methods, including GAT, Graph Transformers, and Graph Autoencoders, as well as other interpretable models, undermines the robustness of the empirical validation. Comparisons with more recent and sophisticated models would provide a stronger benchmark, allowing for a more accurate assessment of DISENE’s strengths and weaknesses in both interpretability and performance.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620895304

thread_type:
official_review

<thread_object>
creator: reviewer_pgtm

<content>
summary:
In this paper, the authors propose a method for interpreting node embeddings. Their method is based on several desiderata in terms of graph structures. In addition, they also propose several new criteria to evaluate the explainability. Experimental results seem to validate their method on these criteria.

strengths:
- The authors propose novel and interesting perspectives to consider node embedding interprebility. New metrics are also introduced to measure these perspectives.

- Experiment results seem to be promising.

weaknesses:
- The proposed method for node interprebility is based on graph structures but semantic information is omitted, as verified by experiments that in GNN methods, node embeddings are initialized by identity matrix. However, in real-world applications this may not be true. For example, in social networks, if each node represents a user, it does not make sense that the user information is not considered in initial features. Whether or not the proposed method can be applied to semantic-rich graph scenarios are not discussed in the paper.

- The three key desiderata is not well explained. In corresponding paragraphs, the authors directly derive how they achieve these goals without explaining them. The definition of Equation 1 does not make sense to me as well. Say when we have $\phi_{d_1} (u_1, v_1; h) > 0$, it only means the the dimension $d_1$ is more important for embeddings of $u_1$ and $v_1$ (to compute the edge likelihood as in the paper, but what is the physical meaning of edge likelihood by the way?) than $d_1$ to other node embeddings with edges. Why could we use it to assign edges to dimensions? The dimension is shared across all node embeddings but the edge is not.

- While the authors propose several metrics for evaluation, I am confused why they do not compare their methods with previous metrics and representative node classification explanation methods such as GNN Explainer. If there is a discrepancy, please specify.

Ying, Zhitao, et al. "Gnnexplainer: Generating explanations for graph neural networks." Advances in neural information processing systems 32 (2019).

questions:
See the weakness above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716167057

thread_type:
official_review

<thread_object>
creator: reviewer_lzef

<content>
summary:
This paper introduces **DISENE (Disentangled and Self-Explainable Node Embedding)**, a novel framework designed to generate self-explainable and disentangled node embeddings in graph-based learning tasks. The method leverages disentangled representation learning to ensure that each dimension of the learned embeddings corresponds to a unique topological substructure in the input graph, aiming to improve the interpretability of learned representations. The authors propose new evaluation metrics, such as overlap consistency and dimensional sparsity, to quantify the degree of disentanglement and interpretability of the embeddings.

strengths:
The paper presents a well-structured approach with clearly defined disentanglement objectives and proposed evaluation metrics. The experiments cover multiple datasets and tasks, providing a comprehensive evaluation of the proposed method.

weaknesses:
1. While the paper claims to generate "self-explainable" embeddings, the interpretability largely relies on proposed metrics (e.g., overlap consistency) without providing direct, human-interpretable explanations for individual embedding dimensions. The method lacks intuitive explanations for individual nodes or substructures, limiting the practical utility of the "self-explainable" claim, particularly for non-technical users or domain experts who might expect more direct insights from the embeddings.

2. The paper lacks comprehensive comparisons with other interpretability-focused methods in graph learning, such as **GNNExplainer** or **PGExplainer**, which provide more fine-grained explanations at node and edge levels. A detailed comparison with these methods in terms of interpretability (not just performance) would strengthen the paper's claims. Additionally, comparisons with simpler feature-based interpretability approaches would help highlight the unique advantages of the proposed method.

3. **Evaluation metrics may be subjective**: While novel, the new evaluation metrics are somewhat subjective and not widely used in the graph learning community. Calibrating these metrics against more established interpretability evaluation standards or providing more detailed justification for why these metrics are appropriate would be helpful. For instance, how does overlap consistency relate to human ability to interpret the learned embeddings? More human-centric evaluations, such as user studies or expert assessments, could help validate the practical interpretability of the embeddings.

questions:
1. How does DISENE compare to graph interpretability methods like GNNExplainer in terms of explainability? A direct comparison with methods like GNNExplainer or PGExplainer would be helpful, not just in terms of performance but also in the quality of explanations provided. Can DISENE provide interpretable explanations at node or edge level similar to these methods?

2. How does DISENE scale to larger graphs? Could the authors provide more details about the computational complexity of the method? Specifically, how does the disentanglement objective affect training time and memory usage, and how does the method scale to larger graphs?

3. How do the proposed metrics relate to human interpretability? Since these metrics are novel, it would be useful to understand how well they align with human judgments of interpretability. While the proposed metrics help quantify disentanglement, it would be helpful to provide visualizations or examples where specific embedding dimensions correlate with interpretable graph structures (e.g., communities, topics). This would help demonstrate the practical interpretability of the method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732204307956

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Initial response

comment:
Dear Reviewers, 

We sincerely appreciate the time and effort you have invested in providing valuable insights and suggestions.
We acknowledge that many of your comments focus on clarifying our **desiderata** and **evaluation metrics**, which form the core of our paper. To facilitate further discussions, we will start by outlining key clarifications in these areas.
In the coming days, we will provide a detailed rebuttal addressing each of your concerns individually and upload a revised version of the paper for your review. In the meantime, we kindly request you to review the following key clarifications and share any further questions or feedback they may raise.
We are confident that addressing these critical aspects will effectively resolve many of the concerns you have highlighted.


**Clarification on metrics and a perspective on how they relate to existing metrics** 

Based on the reviewers' suggestions, we provide a detailed explanation of how our evaluation metrics relate to and extend existing metrics in the context of explainability. Unlike prior works, which primarily focus on explaining **model decisions**, our approach is tailored to explain **model encodings**—a novel perspective. Here's how our metrics align with and build upon previous approaches:
1. **Comprehensibility** measures how well the explanations of embedding dimensions align with human-interpretable structures, such as community structures in the graph. These are often the most intuitive units for understanding a graph.
   * We evaluate comprehensibility using the **F1-score**, quantifying how closely the explanations match human-understandable ground truth. 
   * This approach is analogous to the **accuracy metric** used in GNNExplainer and subsequent works, where explanations for model decisions are compared to planted substructures in the graph, perceived as human-readable justifications for a node’s label.
2. **Sparsity** quantifies the explanation size. As we produce soft masks we use entropy for this quantification. Smaller the entropy shorter the explanation. Motivation for using entropy for explanation size quantification can be found in one of the existing works, **Zorro** [1].
3. **Overlap Consistency** and **Positional Coherence** serve as metrics to evaluate the **faithfulness** of explanations. In our context, faithfulness reflects how well the embeddings align with the explanation structures they are intended to represent. We measure this alignment as follows:
   * **Overlap Consistency:**
This metric assesses whether the correlation between two embedding dimensions is mirrored in their corresponding explanations. For example, if two embedding dimensions are correlated, their explanation substructures should also overlap. This ensures a direct alignment between the relationships in the embedding space and the structural explanations derived from them.
   * **Positional Coherence:**
This metric evaluates whether the feature value of a node in a specific embedding dimension corresponds to its spatial relationship with the explanation substructure for that dimension. Simply put, if a node $u$ has a high value in embedding dimension $d$, it should be positioned closer to the substructure that explains dimension $d$. This ensures that the spatial arrangement of nodes in the graph reflects the embeddings' latent features, reinforcing consistency between the embeddings and their explanations.
4. **Plausibility** evaluates how closely the explanations align with human reasoning, a concept also referred to as "Plausibility" in existing works like **Bagel** [2]. Unlike **Comprehensibility**, which measures the alignment of explanation substructures with community structures in the graph (independent of task-specific information), plausibility focuses on explanations for decisions made in downstream tasks.
The process involves two key steps:
   * **Training on Downstream Tasks:**
Using the interpretable embeddings, we train a model to perform a downstream task. This model can either leverage feature interpretability techniques, such as **SHAP**, or use inherently interpretable models, such as logistic regression.
   * **Extracting and Evaluating Explanations:**
The substructures corresponding to the most important features (identified by the interpretability techniques) or the explanations directly returned by the interpretable model serve as the final explanations. These explanations are then compared with the available human rationale behind the decisions, and the alignment is quantified to compute the **Plausibility Score**.

[1] Funke, Thorben, et al. "Zorro: Valid, sparse, and stable explanations in graph neural networks." IEEE Transactions on Knowledge and Data Engineering 35.8 (2022): 8687-8698.

[2] Rathee, Mandeep, et al. "Bagel: A benchmark for assessing graph neural network explanations." arXiv preprint arXiv:2206.13983 (2022).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732692923688

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We sincerely thank the reviewers for their insightful comments, which have significantly contributed to improving our work. In response, we have submitted a revised version of the paper, and addressed all concerns through detailed point-by-point responses.

In the revised manuscript, we have substantially updated the Appendix to incorporate new experiments on node classification explanations with graph-based explainers, additional images for visualizing the explanations, and technical discussions as requested by the reviewers. We strongly believe that these additions better support the proposed methodology, extending the initial submission with new empirical evidence.

In the new experiments, we observed that computing reliable Plausibility scores in Table A5 requires focusing on node instances with correctly classified labels. This is because accurate explanations are contingent upon the correctness of the model's decisions. Based on this observation, we also updated the results presented in Table 3 of the main paper. However, we note that these updates resulted in only minor changes to the previously reported scores, because of the high overall accuracy of the classifiers as illustrated in Figure A6.

To further strengthen the new experiments, we have introduced two additional synthetic datasets to analyze node classification explanations, Tree-Grids and Tree-Cliques. Compared to the previously used BA-Cliques and ER-Cliques, Tree-Cliques introduces a tree structure as scaffolding, while Tree-Grids presents also a different ground-truth motif. These datasets have been also incorporated into the experiments of the main paper. This addition impacts Table 3 and the corresponding discussion on the same page, providing additional evidence to support our findings. Besides, our initial global response provides required clarfifications on key desiderata and evaluation metrics.

We are confident that these revisions address the reviewers’ concerns and further enhance the quality of our work. We kindly ask reviewers to increase their score if our rebuttal has addressed their main concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733133720384

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion Period

comment:
Dear Reviewers,

We believe that we have addressed all of your concerns. As the discussion period is coming to an end, please let us know any of your remaining concerns so that we can provide timely answers. 

Best,
Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a new framework named DISENE (Disentangled and Self-Explainable Node Embedding) for generating self-explainable and disentangled node embeddings in graph-based learning tasks. Reviewers agreed that this paper brings an interesting perspective by considering node embedding interoperability and also presents new evaluation metics. However, reviewers raised many concerns regarding the newly proposed metrics, the selection of baselines, limited datasets, etc. Overall, the current version of this work is not ready for publication at ICLR.