SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Following previous setup [36], we identify the  $K$  most relevant edges in the original counterfactual explanation and compute the fraction of these edges present in the explanation of its noisy version, denoted by Top-  $K$  Accuracy. We apply noise by randomly adding or removing edges with probability  $\sigma$ . We restrict that  $\sigma < 0.1$ , as the noise of larger  $\sigma$  may cause the noisy graph to switch the predicted label. Top-  $K$  accuracy w.r.t. noise levels over three molecular datasets are shown in Fig. 9. As shown in Figure 9, we observe that D4Explainer outperforms all baselines on BBBP and performs comparably to PGExplainer on Mutag and IGExplainer on NCI1. To keep consistent, we show Top-  $K$  Accuracy with noise levels from  $0\%$  to  $10\%$  for three datasets. However, with more than  $5\%$  noise, only  $65\%$  of perturbed noisy graphs have the same label as the original one for NCI1, much smaller than BBBP and Mutag. The high sensitivity of NCI1 to noise explains the drop in robustness as noise increases past  $\sigma = 0.05$ . Overall, results in Figure 9 demonstrate D4Explainer's strong ability to generate consistently effective counterfactual explanations despite the noise.
E.8 Model-level Explanation
Table 10 shows the quantitative comparison of probability and density in the model-level explanations generated by XGNN and D4Explainer. We generate 100 model-level explanations for each dataset and compute the average probability (i.e., explanation confidence) and average density. As can be observed from Table 10, D4Explainer outperforms XGNN over both metrics on four datasets in general. Particularly, D4Explainer achieves almost  $100\%$  explanation confidence on three synthetic datasets with an appropriate  $N$ , i.e., the number of nodes in the desired explanation. In many real-world scenarios, the ground truth model-level explanations are not unique. That is, we can hardly know the exact discriminative graph structure and feature that the GNNs learned for prediction. The appropriate  $N$  might require domain-specific knowledge, while we can test with different  $N$  and select the one that achieves the highest explanation confidence.
Table 11 reports the sensitivity of  $K$  and  $T$ , which denote the number of candidates and number of iterations in the reverse sampling algorithm for model-level explanations, respectively. Similarly, we utilize probability and density to quantitatively measure the properties of the generated explanations.
Table 10: Quantitative comparison in terms of probability and density with different numbers of nodes
BA-3Motif	Mutag	Tree-Grid	Tree-Cycle
# nodes	5	6	7	6	7	8	8	9	10	5	6	7
D4Explainer	Prob.	0.997	1.000	0.998	0.832	0.856	0.920	0.832	0.994	0.991	0.991	0.995	0.989
Density	0.313	0.327	0.294	0.278	0.327	0.315	0.369	0.372	0.379	0.400	0.381	0.343
XGNN	Prob.	0.632	0.883	0.834	0.523	0.824	0.875	0.752	0.836	0.902	0.968	0.989	0.992
Density	0.552	0.444	0.433	0.537	0.479	0.437	0.421	0.406	0.439	0.400	0.390	0.367
Table 11: Hyperparameter sensitivity in model-level explanation generation
hyper-parameters	BA-3Motif (N=5)	Mutag (N=6)	Tree-Grid (N=9)	Tree-Cycle (N=6)
Prob.	Density	Prob.	Density	Prob.	Density	Prob.	Density
(1) K = 10, T = 50	0.899	0.3152	0.799	0.314	0.901	0.400	0.987	0.372
(2) K = 20, T = 10	0.798	0.3277	0.524	0.284	0.897	0.383	0.991	0.388
(3) K = 20, T = 50	0.967	0.3126	0.812	0.295	0.994	0.372	0.994	0.361
(4) K = 20, T = 100	0.997	0.3133	0.832	0.278	0.972	0.427	0.992	0.325
(5) K = 30, T = 50	0.972	0.2972	0.823	0.287	0.994	0.355	0.997	0.361
From the table, we can observe that when  $T$  is small (e.g.,  $T = 10$ ), the probability  $p$  is relatively lower (see experiments 2, 3, 4), indicating that the quality of the generated model-level explanations is sub-optimal. This result further emphasizes the effectiveness of multi-step sampling, which progressively increases the explanation confidence. Additionally, we observe that the probability  $p$  increases as  $K$  increases, under the same conditions of  $T$  (see experiments 1, 3, 5). This suggests that increasing the number of candidates helps to obtain a good-quality model-level explanation within fewer steps. In our implementation, we ensure  $K \geqslant 20$  and  $T \geqslant 50$  by default for a balance between the quality and time complexity.
F Discussions
Limitations. In this paper, we explore the application of denoising diffusion models in generating counterfactual and model-level explanations for Graph Neural Networks (GNNs). While D4Explainer has shown promising results in terms of various metrics, including explanation accuracy, robustness, diversity, etc. It still introduces unique challenges and limitations. Firstly, the computational complexity of training D4Explainer on large-scale graph structures poses scalability concerns. Additionally, the reliance on the underlying GNN architecture can limit the generalizability of the explanations across different GNN models. Furthermore, in model-level explanations, high-quality explanations rely on an appropriate number of nodes, which might require domain-specific knowledge.
Broader Impacts. The social impact of this work is twofold. On one hand, the ability to generate counterfactual explanations for GNNs can enhance transparency and interpretability, empowering users to understand and trust the decisions made by these models. By shedding light on the features and interactions that contribute to specific predictions, this work can facilitate the identification of biases, discriminatory patterns, and vulnerabilities present in GNNs. However, it is crucial to acknowledge the limitations and potential risks associated with using generated explanations as they might not always capture the complete causal relationships present in complex graph structures. Consequently, relying solely on these explanations may lead to unintended consequences, such as reinforcing existing biases or making incorrect assumptions about the model's behavior.
Future Works. Moving forward, several important avenues for future research emerge from this study. First, addressing the scalability challenges associated with training denoising diffusion models on large-scale graph structures is a crucial direction. Developing efficient training algorithms, exploring parallelization strategies, and investigating graph-specific optimizations can significantly improve the applicability of D4Explainer to real-world large-scale graphs. Secondly, an interesting future direction is to consider the node attributes and edge attributes during the explanation generation, e.g., by performing diffusion processes over continuous features. Moreover, future work should address the potential risks associated with unintended consequences, biases, and misuse of explanations. Developing guidelines and frameworks for responsible and accountable use of generated explanations is crucial, particularly in high-stakes domains such as healthcare, finance, and criminal justice.