SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jialin Chen
jialin.chen@yale.edu
Shirley Wu
shirwu@cs.stanford.edu
Abhijit Gupta
abhijit.gupta@yale.edu
Rex Ying
rex.ying@yale.edu
The widespread deployment of Graph Neural Networks (GNNs) sparks significant interest in their explainability, which plays a vital role in model auditing and ensuring trustworthy graph learning. The objective of GNN explainability is to discern the underlying graph structures that have the most significant impact on model predictions. Ensuring that explanations generated are reliable necessitates consideration of the in-distribution property, particularly due to the vulnerability of GNNs to out-of-distribution data. Unfortunately, prevailing explainability methods tend to constrain the generated explanations to the structure of the original graph, thereby downplaying the significance of the in-distribution property and resulting in explanations that lack reliability. To address these challenges, we propose D4Explainer, a novel approach that provides in-distribution GNN explanations for both counterfactual and model-level explanation scenarios. The proposed D4Explainer incorporates generative graph distribution learning into the optimization objective, which accomplishes two goals: 1) generate a collection of diverse counterfactual graphs that conform to the in-distribution property for a given instance, and 2) identify the most discriminative graph patterns that contribute to a specific class prediction, thus serving as model-level explanations. It is worth mentioning that D4Explainer is the first unified framework that combines both counterfactual and model-level explanations. Empirical evaluations conducted on synthetic and real-world datasets provide compelling evidence of the state-of-the-art performance achieved by D4Explainer in terms of explanation accuracy, faithfulness, diversity, and robustness.<sup>1</sup>
Graph neural networks (GNNs) have rapidly gained popularity recently due to their ability to model relational data [1, 2]. However, when it comes to critical decision-making and high-stake applications, such as healthcare, finance, and autonomous systems, the explainability of GNNs is fundamental for humans to understand the model's decision-making logic and build trust in the deployment of GNNs in real-world scenarios [3, 4, 5].
Counterfactual and model-level explanations. Existing methods mainly focus on factual and instance-level explanations [6, 7, 8, 9, 10, 11], while the significance of counterfactual and model-level explanations are equally noteworthy, yet under-explored. Counterfactual explanation considers
<sup>1</sup>The code is available at https://github.com/Graph-and-Geometric-Learning/D4Explainer
"What-if" scenarios of model predictions, addressing the question of how slight adjustments to the input graph can lead to different model predictions [12, 13, 14]. Model-level explanation, on the other hand, aims to generate the most discriminative graph pattern for a target class, thus shedding light on the overall decision-making behavior and internal functioning of the model [15, 16]. Counterfactual and model-level explanations present a distinct challenge concerning the distribution constraint imposed on generated explanations. An explanation that is faithful and reliable should adhere to the distribution of the underlying dataset. This becomes particularly crucial in real-world scenarios where domain-specific rules exist, such as in drug design and molecule generation. In such cases, explanations should conform to the true distribution of the dataset [16, 17, 18].
However, the existing methods typically extract explanatory subgraphs from the input graph, ignoring additional possible edges. This prevailing paradigm heavily relies on the out-of-distribution (OOD) effect to influence the model's prediction. To illustrate this point, in Figure 1, we show the t-SNE projection of the Tree-Cycle dataset, where graphs are labeled as Tree or Cycle based on whether they present the corresponding structures. Specifically, CF-GNNExplainer [13] generates counterfactual explanations for a node with Tree label by removing its neighbor edges. While the explanation doesn't maintain any discriminative information on the Cycle class, it could still be predicted as Cycle with high probability due to the OOD effect, making the explanation unreliable.
On the other hand, generating in-distribution
graphs is challenging, due to the difficulty of encoding complex graph distributions, e.g., the distribution of node degrees, cycle counts and edge homogeneity. Recently, graph diffusion models have shown to be a powerful technique to encode such complex distribution on graphs [19, 20], which trains a powerful denoising model that progressively removes noise from a random noise graph and then tractably recovers in-distribution samples.
Figure 1: t-SNE Projection of Tree-Cycle dataset, where Cycle is a counterfactual motif for Tree.
Proposed work. Inspired by the success of graph diffusion models, we propose a novel GNN explainability approach, D4Explainer, in-Distribution GNN explanations via Discrete Denoising Diffusion. Through a forward diffusion process that progressively introduces random noise, we enable D4Explainer to optimize for alternative and diverse explanations based on multiple noisy versions of the given graph. A powerful denoising model is trained to remove noise and eliminate redundant edges that are irrelevant to the target property, thereby ensuring the model's robustness. By employing a carefully designed loss function that incorporates both the preservation of the counterfactual property and generative graph distribution learning, D4Explainer is capable of generating in-distribution counterfactual explanations. As highlighted in green in the bottom left of Figure 1, D4Explainer adds essential edges that complete the truly counterfactual motif, i.e., Cycle. With a slight modification to the loss function, D4Explainer can also perform model-level explanations for a specific target class.
Empirical experiments on eight synthetic and real-world datasets show that D4Explainer achieves state-of-the-art performance in both counterfactual and model-level explanations, with a strong counterfactual accuracy ( $>80\%$ ) when only  $5\%$  of the edges are modified. Maximum mean discrepancy (MMD) metrics show that the distribution of explanations generated by D4Explainer is the closest to the original distribution of the dataset, compared with all baselines. D4Explainer obtains the highest Top- $K$  accuracy in the robustness evaluation, which further illustrates that D4Explainer is capable of generating consistent explanations with the presence of noise.
Our contributions are in three-folds: (1) A novel approach to generate in-distribution, diverse and robust explanations is proposed, which leverages the denoising diffusion model to capture the underlying distributions of explanation graphs; (2) D4Explainer explores counterfactual explanations in a larger search space by allowing adding edges, which provides high-level understandings of how edge addition helps to create truly counterfactual motifs; (3) D4Explainer represents the first
framework that unifies counterfactual and model-level explanations, providing faithful explanations for both settings.
Explainability of GNNs Compared with the explainability methods in image domain [21, 22, 23, 24, 25, 26, 27], explainability in GNNs [3] remains a challenging problem due to the discrete structure of graphs. Here, we focus on post-hoc and model-agnostic explanations. Non-parameterized methods rely on gradient-like signals [28, 29], relevant walks [30, 31], perturbation [9, 32, 33, 34] to identify important node/edge features or graph structures as explanations, without learnable parameters. Score-based explainability [6, 8, 7, 35] formulate a trainable model to obtain the importance scores on node features or edge as the explanations by maximizing the mutual information between the explanatory subgraph and the target prediction. Counterfactual explanation methods find minimal perturbation to the graph instance such that the prediction changes. However, most existing methods [13, 36] only consider edge deletion on the original graph without any distribution constraints, thus easily creating out-of-distribution samples and overfitting the noise over each individual instance. CLEAR [37] is the only explainer that also considers adding edges in generating counterfactual explanations. However, the intrinsic effect of edge addition to counterfactual properties is under-explored by CLEAR. Generation-based explanations is a recently popular trend that trains graph generators to generate GNN explanations. Existing works train policy networks for the sequential graph generation process based on the reinforcement learning approach [38, 10, 15] or explicitly parameterize the distribution of model-level explanations [16]. The differences of our method are (1) we prevent explicit modeling and sequential decision-making learning but incorporate the generative graph distribution learning implicitly into the training procedure and (2) the more stable and robust generative backbone i.e., diffusion model ensures better properties of the generated explanations, e.g., diversity and robustness.
Graph Diffusion Models Denoising diffusion probabilistic models [39, 40, 41] are shown to be powerful for a wide range of generative tasks, including images [42], language [43], and discrete graph domain [19, 20, 44]. Recent work [20] proposes to use discrete noise for the forward Markov process without relying on continuous Gaussian perturbations. Another related work [19] formulates the diffusion process on the categorical node and edge attributes and successfully generates real and in-distribution graphs. Recently, the score-based model [45] and stochastic differential equations formulation have been applied to the field of graph generation [46, 47]. These related works highlight the effectiveness of diffusion models for graph denoising and generation tasks. In our paper, we design the pipeline of the diffusion-based model for explanation task scenarios, as well as devise a novel classifier-guided sampling algorithm for model-level explanations.
Counterfactual explanation. Given an instance (i.e., a node or a graph) and a well-trained GNN, the goal of counterfactual explanation is to identify the minimal modification to the original instance that alters GNN's prediction [12, 13, 36]. Without loss of generality, we consider the explanation problem for the graph classification task. Formally, let  $f$  denote a well-trained GNN classifier to be explained,  $\hat{Y}_G$  denote the label of graph  $G$  predicted by  $f$ . The counterfactual explanation  $G^c$  satisfies that  $\hat{Y}_{G^c} \neq \hat{Y}_G$ , while the difference between  $G^c$  and  $G$  is minimal. This problem is usually formulated as an optimization problem that minimizes the mutual information between  $G^c$  and  $\hat{Y}_G$  [6, 13].
Model-level explanation. Model-level explanation aims to identify recurring and discriminant graph patterns that can trigger a specific prediction from the model  $f$  [15, 16]. Formally, given a class  $C_i \in \{C_1, \dots, C_l\}$ , model-level explanation for the target class  $C_i$  can be defined as  $G^m = \operatorname{argmax}_G P_f(C_i|G)$ , where  $P_f(C_i|G)$  denotes the probability for the class  $C_i$  predicted by the GNN  $f$ , given the graph  $G$ . See Appendix B for more descriptions of the explanation task setting.
3.2 Discrete Diffusion Process for Graph
Forward diffusion process. In this work, we focus on discrete structural diffusion and leave the diffusion over continuous features in future work. Let  $t \in [0,T]$  denote the timestep of the diffusion process, which is also a noise level indicator. Let  $A_{t}$  denote the one-hot version of the adjacency matrix at timestep  $t$ , where each element  $a_{t}^{ij}$  is a 2-dimensional one-hot encoding of the presence or absence of the  $ij$ -th element in the adjacency matrix. The forward diffusion process is a Markov chain with a transition matrix  $Q_{t} \in \mathbb{R}^{2 \times 2}$ , that progressively transforms the input graph into pure noise. Mathematically, the forward diffusion process can be written as  $q(a_{t}^{ij}|a_{t-1}^{ij}) = \mathrm{Cat}(a_{t}^{ij};P = a_{t-1}^{ij}Q_{t})$ , where  $\mathrm{Cat}(\boldsymbol{x};\boldsymbol{P})$  is a categorical distribution over the one-hot vector  $\boldsymbol{x}$  with probability vector  $\boldsymbol{P}$ . The multi-step diffusion has a closed form as  $q(a_{t}^{ij}|a_{0}^{ij}) = \mathrm{Cat}(a_{t}^{ij};\boldsymbol{P} = a_{0}^{ij}\bar{Q}_{t})$ , where  $\bar{Q}_{t} = \prod_{i=1}^{t}\bar{Q}_{i}$ . See Appendix C for detailed derivation.
Graph-level expression. The forward diffusion process is identically and independently performed over each edge in the full adjacency matrix. Therefore, the graph-level diffusion  $q(G_{t}|G_{t - 1})$  is the product of element-wise categorical distributions as
$$
q \left(G _ {t} \mid G _ {t - 1}\right) = \prod_ {i j} q \left(\boldsymbol {a} _ {t} ^ {i j} \mid \boldsymbol {a} _ {t - 1} ^ {i j}\right) \text {a n d} q \left(G _ {t} \mid G _ {0}\right) = \prod_ {i j} q \left(\boldsymbol {a} _ {t} ^ {i j} \mid \boldsymbol {a} _ {0} ^ {i j}\right) \tag {1}
$$
Denoising diffusion models have shown a powerful ability to recover complex distributions accurately [48, 41, 20, 44], by leveraging the diffusion process to capture intricate dependencies and generate samples that exhibit high-quality in-distribution property and diversity.
4 Proposed Method: D4Explainer
D4Explainer is designed for two distinct explanation scenarios: counterfactual explanation and model-level explanation. In counterfactual explanation (Sec. 4.1), D4Explainer employs a Forward diffusion process to create a sequence of noisy versions and trains a Denoising model to effectively capture the desired distribution of counterfactual graphs. For model-level explanation (Sec. 4.2), D4Explainer trains a Denoising model to recover the underlying original distribution and leverages a well-trained GNN to progressively enhance the explanation confidence during the reverse sampling. An overview is shown in Figure 2. The notation used throughout this work is summarized in Appendix A.
(a) Diffusion Model for Counterfactual Explanation
Figure 2: Overview of D4Explainer. (a) Diffusion Model for counterfactual explanations. The diffusion process  $q(G_{t}|G_{t - 1})$  transforms an input graph  $G_{0}$  to the pure noise  $G_{T}$ . Then the Denoising Model  $p_{\theta}(\cdot)$  outputs the clean graph  $\tilde{G}_0$  given a noisy graph  $G_{t}$ , under the constraints of the counterfactual loss  $\mathcal{L}_{cf}$  and the distribution loss  $\mathcal{L}_{dist}$ . (b) Reverse Sampling for model-level explanations. We leverage a well-trained GNN to select a temporary graph with the highest confidence score from the candidate graphs and obtain  $G_{t - 1}^{r}$  from  $G_{t}^{r}$  recursively until we achieve the final model-level explanation  $G_0^r$ .
(b) Reverse Sampling for Model-level Explanation
4.1 Counterfactual Explanation Generation
Forward diffusion process. We build on the discrete diffusion process over graphs as introduced in Sec. 3.2. The forward Diffusion Process enables D4Explainer to optimize with a sequence of perturbed graphs  $\{G_0, G_1, \dots, G_T\}$  with increasing levels of noise, which essentially enable D4Explainer to thoroughly explore possible counterfactual explanations for the given graph.
Denoising model. To generate a counterfactual graph that closely resembles the input graph, the Denoising Model  $p_{\theta}(G_0|G_t)$  takes as input the noisy adjacency matrix  $A_{t}$  corresponding to a noisy graph  $G_{t}$ , the node features of the original graph  $X_0$ , noise level indicator  $t$ , and then predicts the dense adjacency matrix. Through sampling from the dense adjacency matrix with the reparameterization trick [49], we arrive at the discrete adjacency matrix  $\tilde{A}_0$  and the corresponding explanation graph  $\tilde{G}_0$ . The Denoising Model is set as an extension of the provably powerful Graph Network (PPGN) [50]. To incorporate time information, an MLP module is employed to process the noise level indicator  $t$  and learn time-related latent features, thereby enhancing the denoising capability. The edge features, node features, and time-related latent features are concatenated and updated by the powerful layers (PPGN). We refer to Appendix D.1 for a complete and detailed description of the PPGN used in our D4Explainer.
Loss function. Different from traditional graph generation tasks [20, 19, 47], counterfactual explanations necessitate both counterfactual property and proximity to the original graph. To address these challenges, we propose a specifically designed loss function that simultaneously optimizes these two properties. Instead of iteratively recovering the intermediate noisy graph  $G_{t}$  in the traditional manner, we employ a re-weighted version of the evidence lower bound (ELBO) on the negative log-likelihood that directly reconstructs the initial distribution at  $t = 0$  in our distribution-learning term  $\mathcal{L}_{dist}$ . The re-weighting strategy prioritizes more challenging denoising tasks at larger timesteps:
$$
\mathcal {L} _ {\text {d i s t}} = - \mathbb {E} _ {q (G _ {0})} \sum_ {t = 1} ^ {T} \left(1 - 2 \cdot \bar {\beta} _ {t} + \frac {1}{T}\right) \mathbb {E} _ {q (G _ {t} | G _ {0})} \log p _ {\theta} \left(G _ {0} \mid G _ {t}\right), \tag {2}
$$
where  $\bar{\beta}_t$  is the transitioning probability (the off-diagonal element in the transition matrix  $\bar{Q}_t$ ) and  $q(G_0)$  is the distribution of the training dataset. The distribution loss  $\mathcal{L}_{dist}$  is equivalent to the cross-entropy loss between  $G_0$  and  $p_{\theta}(G_0|G_t)$  over the full adjacency matrix, which guarantees the proximity of generated counterfactual explanations to the original graph. To optimize the counterfactual property, we design a specific counterfactual loss  $\mathcal{L}_{cf}$  as follows,
$$
\mathcal {L} _ {c f} = - \mathbb {E} _ {q (G _ {0})} \mathbb {E} _ {t \sim [ 0, T ]} \mathbb {E} _ {q (G _ {t} | G _ {0})} \mathbb {E} _ {p _ {\theta} (\tilde {G} _ {0} | G _ {t})} \log \left(1 - f (\tilde {G} _ {0}) [ \hat {Y} _ {G _ {0}} ]\right), \tag {3}
$$
where  $f$  is the well-trained GNN classifier,  $f(\tilde{G}_0)[\hat{Y}_{G_0}]$  denotes the probability for the original label  $\tilde{Y}_{G_0}$  predicted by  $f$ , given the generated graph  $\tilde{G}_0$ . Our total loss function is formulated as  $\mathcal{L}(\theta) = \mathcal{L}_{dist} + \alpha \mathcal{L}_{cf}$ , where  $\alpha$  is a hyper-parameter that balances the counterfactual and indistribution properties. Achieving the desired counterfactual property while maintaining proximity to the true data distribution involves a trade-off. For instance, making drastic modifications to the original graph may easily alter the model's prediction, but it can also lead to an explanation that deviates significantly from the original graph. The distribution loss  $\mathcal{L}_{dist}$  and the counterfactual loss  $\mathcal{L}_{cf}$  together encourage the denoising model to eliminate redundant edges that are irrelevant to the counterfactual property while reconstructing the original edges to preserve the true distribution.
Working principle of D4Explainer. D4Explainer not only preserves the in-distribution property but also introduces diversity and robustness to the generated counterfactual explanations. Diversity enables the explainer to provide multiple alternative explanations for model predictions, while robustness ensures consistent effectiveness of the explanations even in the presence of noise. Existing explainers often optimize a singular explanation per instance, leading to overfitting on noise and bias attribution issues [51]. On the contrary, D4Explainer's objective is to search for counterfactual graphs within the distribution of the original graphs, adhering to the constraints imposed by  $\mathcal{L}_{dist}$  and  $\mathcal{L}_{cf}$ . Through an iterative process of adding noise and removing counterfactual-irrelevant edges, D4Explainer captures the underlying distribution of counterfactual explanations. This denoising strategy also enhances the robustness of D4Explainer. Moreover, the inherent stochasticity in the forward processes introduces diversity into the generated explanations.
4.2 Model-level Explanation
Motivation. The goal of model-level explanation is to generate class-wise graph patterns. Let  $C$  denote the target class. Each reverse sampling step  $q_{C}(G_{t - 1}^{r}|G_{t}^{r})$  can be formulated as a conditional generation satisfying the following equation,
$$
q _ {C} \left(G _ {t - 1} ^ {r} \mid G _ {t} ^ {r}\right) \propto p _ {\theta} \left(\tilde {G} _ {0} \mid G _ {t} ^ {r}\right) q \left(G _ {t - 1} ^ {r} \mid \tilde {G} _ {0}\right) f (C \mid \tilde {G} _ {0}), \tag {4}
$$
where  $f(C|\tilde{G}_0)$  can be computed by the target class probability predicted by the well-trained GNN  $f$ , conditioned on the given graph  $\tilde{G}_0$ . Existing sampling methods [20, 41] cannot perform conditional sampling in the discrete context, as we cannot sample all possible  $\tilde{G}_0$  to obtain  $f(C|\tilde{G}_0)$  and then compute the normalized probabilities. To overcome these challenges, we propose to utilize the well-trained GNN as guidance toward the target class. At each step, we generate a set of candidates by  $p_{\theta}(\tilde{G}_0|G_t^r)$  and refer to the GNN to select a temporarily optimal  $\tilde{G}_0$  with the highest  $f(C|\tilde{G}_0)$ .
Multi-step sampling. We repeat the sampling steps and progressively increase the explanation confidence (i.e.,  $f(C|\tilde{G}_0))$  in the process. Figure 3 shows an empirical visualization of the reverse generation process for the house motif. We observe that the temporary graph  $\tilde{G}_0$  gets closer to the target motif with increasing explanation confidence  $p$  during the reverse sampling process.
The proposed model-level explanation generation utilizes a denoising model trained with a similar procedure as Sec. 4.1 (Figure 2(a)). The difference is that the training loss is only  $\mathcal{L}_{dist}$ , since  $\mathcal{L}_{cf}$  leads to a counterfactual graph that changes the label. To start with, given a predefined number of nodes  $N$  in the target explanation, we randomly sample an Erdős–Rényi graph with  $N$  nodes and edge probability  $\frac{1}{2}$  as  $G_T^r \sim \mathcal{B}_{N,1/2}$ . Then we sample a set of candidates from the distribution  $p_\theta(G_0|G_T^r)$ . The well-trained GNN computes the explanation confidences for these candidates and selects the temporary explanation  $\tilde{G}_0$  with the highest score.
Then, we sample  $G_{T - 1}^{r}$  through the same Diffusion Process  $G_{T - 1}^{r}\sim q(G_{T - 1}|\tilde{G}_{0})$  as Equation 1. Sampling steps iteratively reverse the chain until we obtain the final model-level explanation  $G_0^r$  after  $T$  steps. Apart from explanation confidence  $f(C|\tilde{G}_0)$ , model-level explanations should also satisfy sparsity and succinctness. It is worth noting that the proposed algorithm is capable of preserving the sparsity level similar to the training graphs in the generated explanations. For real-world datasets that are densely self-connected, it is suggested to plug regularization constraints in the selection policy for the temporary explanation at each step. The complete sampling algorithm is shown in Appendix D.4.
Figure 3: Visualization of the temporary  $\tilde{G}_0$  at  $t = T;3T / 4;T / 2;T / 4$  and the terminal modellevel explanation for BA-3Motif (house motif). Different node colors indicate different labels.
4.3 Complexity Analysis