SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junfeng Fang $^{1}$ , Wei Liu $^{1\dagger}$ , Yuan Gao $^{1}$ , Zemin Liu $^{2}$ , An Zhang $^{2}$ , Xiang Wang $^{1*}$ , Xiangnan He $^{1*}$
<sup>1</sup>University of Science and Technology of China, <sup>2</sup>National University of Singapore
{fjf,lw0118,yuangao}@mail.ustc.edu.cn, {zeminliu,an_zhang}@nus.edu.sg,
{xiangwang1223, xiangnanhe}@gmail.com
This work studies the evaluation of explaining graph neural networks (GNNs), which is crucial to the credibility of post-hoc explainability in practical usage. Conventional evaluation metrics, and even explanation methods — which mainly follow the paradigm of feeding the explanatory subgraph to the model and measuring output difference — mostly suffer from the notorious out-of-distribution (OOD) issue. Hence, in this work, we endeavor to confront this issue by introducing a novel evaluation metric, termed OOD-resistant Adversarial Robustness (OAR). Specifically, we draw inspiration from adversarial robustness and evaluate post-hoc explanation subgraphs by calculating their robustness under attack. On top of that, an elaborate OOD reweighting block is inserted into the pipeline to confine the evaluation process to the original data distribution. For applications involving large datasets, we further devise a Simplified version of OAR (SimOAR), which achieves a significant improvement in computational efficiency at the cost of a small amount of performance. Extensive empirical studies validate the effectiveness of our OAR and SimOAR. Code is available at https://github.com/MangoKiller/SimOAR_OAR.
Post-hoc explainability has manifested its extraordinary power to explain graph neural networks (GNNs) [1, 2, 3, 4]. Given a GNN-generated prediction for a graph, it aims to identify an explanatory subgraph, which is expected to best support the prediction and make the decision-making process more credible, fair, and understandable [5, 6, 7]. However, the reliable evaluation of explanation quality remains a key challenge. As a primary solution, Human supervision seeks to justify whether the explanations align with human knowledge [8, 9], but it is often too subjective, thus hardly providing quantifiable assessments. Another straightforward solution is quantitatively measuring the agreement between the generated and ground-truth explanations, such as Precision and Recall [10, 11]. Unfortunately, access to the ground truth is usually unavailable and labor-extensive, thereby limiting the scope of evaluations based on this method.
Recently, a compromised paradigm — Feature Removal [12, 13] — has been prevailing to quantitatively evaluate the explanation's predictive power as compared to the full graph, without exploiting the human supervision and ground truth. The basic idea is to first remove the unimportant features and feed the remaining part (i.e., explanatory subgraph) into the GNN, and then observe how the prediction changes. The prediction discrepancy instantiates Accuracy [8] and Fidelity [9] of the
$^{\dagger}$ Liu Wei is equal contribution to this paper.
*Corresponding authors. Xiang Wang and Xiangnan He are also affiliated with Institute of Artificial Intelligence, Institute of Dataspace, Hefei Comprehensive National Science Center.
Figure 1: Pipelines and flaws of different evaluation methods. In the "Input" graph,  $-\mathrm{NH}_2$  is considered as the ground truth explanation for its mutagenicity. Best viewed in color.
explanation, reflecting "how accurate and faithful the explanation is to recover the prediction of the input graph". Despite the prevalence, these removal-based metrics usually come with the caveat of the out-of-distribution (OOD) issue [14, 13]. Specifically, as the after-removal subgraphs are likely to lie off the distribution of full graphs [15, 16], the GNN is forced to handle these off-manifold inputs and easily gets erroneous predictions [17, 18]. Take Figure 1 (a) as an example. For the full molecular graph, the GNN classifies it as "mutagenic", which is reasonable due to the presence of mutagenic  $-\mathrm{NH}_2$  group; whereas, when taking the subgraph, i.e., non-mutagenic C-Cl group solely as the input, the GNN surprisingly maintains its output "mutagenic". Clearly, the prediction on the explanatory subgraph might be skeptical, which easily deteriorates the faithfulness of the removal-based evaluations.
In sight of this, recent efforts [18, 17] are beginning to mitigate the OOD issue via the Generation-based metrics. Instead of directly feeding the to-be-evaluated subgraph into the target GNN, they use a generative model [19, 20] to imagine and generate a new full graph conditioned on the subgraph. These methods believe that the generation process could infill the subgraph and pull it closer to the original graph distribution. As Figure 1 (b) shows, comparing the predictions on this new graph and the original graph could be the surrogate evaluation. While intuitively appealing, the generative models easily inherit the data bias and inject it into the infilling part. Considering Figure 1 (b) again, since molecules in the Mutagenicity dataset comprising non-mutagenic chloride  $(- \mathrm{Cl})$  always carry amino  $(- \mathrm{NH}_2)$ , a generative model is prone to capture this occurrence bias and tend to infill  $- \mathrm{NH}_2$  with the  $- \mathrm{Cl}$ -involved subgraphs. This bias not only exerts on the generations but also makes the evaluation inconsistent with the GNN behavior: in the generative model,  $- \mathrm{Cl}$  is assigned with more "mutagenic" scores as it usually accompanies the mutagenic partner  $- \mathrm{NH}_2$ ; in contrast, the GNN finds no "mutagenic" clues in  $- \mathrm{Cl}$ . In a nutshell, the generation-based metrics show respect to the data distribution somehow but could be inconsistent with GNNs' behavior and lose control of the infilling part.
Scrutinizing these removal- and generation-based metrics (as summarized in Figure 1), we naturally raise a question: "Can we design a metric that respects the data distribution and GNN behavior simultaneously?" To this end, we draw inspiration from adversarial robustness [21, 22] and propose a new evaluation framework, OAR (OOD-resistant Adversarial Robustness), to help reliably assess the explanations. As shown in Figure 1 (c), OAR encapsulates two components: constrained attack and OOD reweighting, which respect the GNN behavior and data distribution, respectively. Specifically:
- Intuitively, perturbations on label-irrelevant features should be ineffective to the GNN prediction, while those on label-relevant features are supposed to be impactful and destructive to the prediction [22, 21]. Hence, for the original input graph, the attack model performs perturbations constrained on the complementary part of its explanation. This perturbing game aims to naturally take control of the "infilling" process, making the explanatory subgraph less influenced by its infilling.
- Having obtained a set of perturbed graphs, the reweighting component estimates the OOD score of each perturbed graph, which reflects the degree of distribution shift from the original data
distribution. Then, we feed these graphs into GNN and reweight their predictions with OOD scores. The sum of weighted predictions can quantitatively evaluate the importance of the target subgraph.
We validate the effectiveness of OAR in evaluation tasks across various start-of-the-art explanation methods, datasets, and backbone GNN models. OAR has manifested surprising consistency with the metrics like Precision, Recall and Human Supervision. Furthermore, to better generalize to the large datasets, we also provide a Simplified version of OAR (SimOAR) achieving significant improvements in computational efficiency at the expense of a small amount of performance degradation. Our main contributions can be summarized as follows:
- We propose a novel metric, OAR for evaluating GNNs' explainability, which tries to resolve the limitation of current removal- and generative-based evaluations by taking both data distribution and GNN behavior into account (Section 2.2).
- We provide a simplified version of OAR, SimOAR for better generalization to the evaluation tasks involving large datasets, which greatly shortens the execution time while only sacrificing a small amount of performance (Section 2.3).
- Experimental results demonstrate that our OAR/SimOAR outperforms the current evaluation metrics by a large margin, and further validate the high efficiency of SimOAR (Section 3).
In this section, we propose an evaluation method for the explainability of GNNs from the perspective of adversarial robustness. We start with the notation of GNNs and its explainability in Section 2.1. After that, we detail our evaluation metric, OAR via three progressive steps (Section 2.2). In Section 2.3, we provide a simplified version of OAR called SimOAR for applications demanding more efficient execution.
Graph neural networks (GNNs). GNNs have achieved remarkable success due to their powerful representation ability. Without loss of generality, we focus on the graph classification task in this work: a well-trained GNN model  $f$  takes a graph  $\mathcal{G}$  as the input and outputs its probabilities  $\pmb{y}$  over classes  $\{1,\dots,C\}$ , i.e.,  $\pmb{y} = f(\mathcal{G}) \in \mathbb{R}^C$ . Typically,  $\mathcal{G}$  is an undirected graph involving the node set  $\mathcal{V}$  and the edge set  $\mathcal{E}$ . We first introduce the feature of node  $v_{i} \in \mathcal{V}$  as a  $d$ -dimensional vector and collect the features of all nodes into  $X \in \mathbb{R}^{|\mathcal{V}| \times d}$ . Then we define an adjacency matrix  $A \in \mathbb{R}^{|\mathcal{V}| \times |\mathcal{V}|}$  to describe graph topology, where  $A_{uv} = 1$  if the edge connecting nodes  $u$  and  $v$  exists, i.e.,  $(u,v) \in \mathcal{E}$ , otherwise  $A_{uv} = 0$ . Based on these,  $\mathcal{G}$  can be alternatively represented as  $\mathcal{G} = (\pmb{A},\pmb{X})$ .
Explainability for GNNs. Upon the GNN model  $f$ , explanation techniques of GNNs generally study the underlying relationships between their outputs  $y$  and inputs  $\mathcal{G}$ . They focus on explainability w.r.t. input features, aiming to answer "Which parts of the input graph contribute most to the model prediction?" Towards this end, explainers typically assign an importance score to each input feature (i.e., node  $v_{i}$  or edge  $(v_{i}, v_{j})$ ) to trace their contributions. Then they select the salient part (e.g., a subset of nodes or edges with top contributions) as the explanatory subgraph  $\mathcal{G}_{s}$  and delete the complementary part  $\mathcal{G}_{\bar{s}} = \mathcal{G} \backslash \mathcal{G}_{s}$ . We formulate the explanation method as  $h$  and yield the above process as  $\mathcal{G}_{s} = h(\mathcal{G}, f)$ .
2.2 OOD-Resistant Adversarial Robustness
Retrospecting the removal- and generation-based evaluations, we emphasize that both these classes come with inherent limitations. Specifically, Removal-based metrics pay less heed to the data distribution thus forcing GNNs to handle off-manifold instances, while generation-based metrics are inconsistent with GNN behavior and lose control of the infilling part. Fortunately, in this section, we claim that it is possible to get the best of and avoid the pitfalls of both worlds — removal-based and generation-based metrics — by taking both GNN behavior and data distribution into account.
To meet these challenges, we elaborate our evaluation metric, OOD-resistant adversarial robustness (OAR) via three progressive steps: in the first step, we formulate the adversarial robustness tailored for GNNs explanations, which naturally conforms to the GNN behavior; in the second step, we introduce a tractable and easy-to-implement objective of above adversarial robustness; in the third step, we introduce an elaborate OOD reweighting block which confines the overall evaluation process to the original data distribution.
STEP 1: Formulation of Adversarial Robustness. We prioritize the introduction of adversarial robustness in machine learning [21, 22, 23] that motivates our method. Concretely, given a machine
Figure 2: The pipeline of OAR, which takes both model behavior and data distribution into account.
learning model, an input  $x$  and a subset of the input  $x_{s} \subseteq x$ , the adversarial robustness of  $x_{s}$  denotes the minimum perturbation leading to the wrong prediction, on the condition that perturbation is only imposed on  $x_{s}$  [22]. Inspired by this idea, we define the adversarial robustness of GNN explanation  $\mathcal{G}_s$ , and formulate it as the minimum adversarial perturbation  $\delta$  on the structure of complementary subgraph  $\mathcal{G}_{\bar{s}}$ . More formally,
Definition 1. Given a GNN model  $f$ , an input graph  $\mathcal{G} = (\mathbf{A},\mathbf{X})$  with prediction  $\mathbf{y}$  and explanation  $\mathcal{G}_s$ , suppose that  $\mathcal{G}' = (\mathbf{A}',\mathbf{X}')$  is the graph generated by adding and deleting edges in  $\mathcal{G}$ , the adversarial robustness  $\delta$  of explanation  $\mathcal{G}_s$  is defined as:
$$
\begin{array}{l} \delta_ {\mathcal {G} _ {s}} = \min  _ {\boldsymbol {A} ^ {\prime}} \sum_ {u \in \mathcal {V}} \sum_ {v \in \mathcal {V} \backslash u} | A _ {u v} - A _ {u v} ^ {\prime} | \\ s. t. \underset {i} {\arg \max } f \left(\mathcal {G} ^ {\prime}\right) _ {i} \neq \underset {i} {\arg \max } \boldsymbol {y} _ {i}, \quad \sum_ {u \in \mathcal {V} _ {s}} \sum_ {v \in \mathcal {V} _ {s} \backslash u} | A _ {u v} - A _ {u v} ^ {\prime} | = 0, \tag {1} \\ \end{array}
$$
where  $\mathcal{V}$  and  $\mathcal{V}_s$  are the node sets of  $\mathcal{G}$  and  $\mathcal{G}_s$ , respectively.
Definition 1 identifies the quality of explanation  $\mathcal{G}_s$  as the difficulty of reversing the prediction by perturbing features not belonging to  $\mathcal{G}_s$  solely. That is, when  $\mathcal{G}_s$  is fixed, the more difficult it is to fool the model by perturbing its complementary, the more important  $\mathcal{G}_s$  is. The key intuition behind this inference is: if an explanation comprises most of the label-relevant features, it is conceivably hard to change the prediction by manipulating the remaining features that are label-irrelevant. Thus, according to Definition 1, we can find that: a good explanation would yield high adversarial robustness  $\delta$  and vice versa, which naturally conforms to the GNN behavior.
It seems that adversarial robustness is the feasible metric to evaluate the GNNs' explanations. However, there are still two matters standing in the way of its adoption: 1) Is its objective (i.e., Equation (1)) tractable and easy to implement? 2) Does it respect the data distribution?
STEP 2: Finding a Tractable Objective. To answer the first question, we argue that Equation (1) is hard to realize and sometimes even intractable owing to two possible reasons:
- The primary reason is that adversarial attacks may fail to find any adversarial example, since a solution satisfying two conditions in Equation (1) simultaneously may not exist. In other words, if the explanation  $\mathcal{G}_s$  is precise enough, it is almost impossible to reverse the prediction via manipulating features in the complementary part  $\mathcal{G}_{\bar{s}}$  which are mainly label-irrelevant.
- It is notoriously hard to search for the minimum adversarial perturbation  $\delta$  in most cases. Current attack methods [24, 25] typically turn to find an alternative sub-optimal solution. Thus, leveraging these methods could introduce additional bias and threaten the fairness of evaluation.
To address these issues and make the evaluation objective tractable and easy to implement, we first formulate the inference derived from Definition 1:
Proposition 1. When the high quality explanation  $\mathcal{G}_s$  is anchored (fixed), perturbations restricted to the complementary part  $\mathcal{G}_{\bar{s}}$  have a weak influence on the model prediction.
While Definition 1 evaluates the explanations via the adversarial robustness  $\delta$ , Proposition 1 indicates a more straightforward way to the tractable evaluation objective from its dual perspective. To be more specific, Definition 1 quantifies perturbation on  $\mathcal{G}_{\bar{s}}$  causing change of prediction; conversely, Proposition 1 quantifies change of prediction caused by perturbation on  $\mathcal{G}_{\bar{s}}$ . More formally,
Definition 2. Given a GNN model  $f$ , an input graph  $\mathcal{G} = (\mathbf{A},\mathbf{X})$  with prediction  $\mathbf{y}$  and explanation  $\mathcal{G}_s$ , suppose that  $\mathcal{G}' = (\mathbf{A}',\mathbf{X}')$  is the graph generated by adding and deleting edges in  $\mathcal{G}$ , the approximate adversarial robustness  $\delta^*$  of  $\mathcal{G}_s$  is defined as:
$$
\delta_ {\mathcal {G} _ {s}} ^ {*} = \mathbb {E} _ {\mathcal {G} ^ {\prime}} \left(f \left(\mathcal {G} ^ {\prime}\right) _ {c} - \boldsymbol {y} _ {c}\right)
$$
$$
s. t. c = \underset {i} {\arg \max } \boldsymbol {y} _ {i}, \quad \sum_ {u \in \mathcal {V} _ {s}} \sum_ {v \in \mathcal {V} _ {s} \backslash u} | A _ {u v} - A _ {u v} ^ {\prime} | = 0, \tag {2}
$$
where  $\mathcal{V}_s$  refers to the node set of  $\mathcal{G}_s$ ;  $c$  denotes the predicted class of  $\mathcal{G}$  by model  $f$ ;  $f(\mathcal{G}')_c$  represents the probability value of  $f(\mathcal{G}')$  for the given class  $c$ .
Stemmed from Definition 2, the evaluation method to quantify the adversarial robustness of GNN explanations is more explicit and computationally convenient. As shown in Figure 2, for the to-be-evaluated subgraph  $\mathcal{G}_s$  (i.e., C-Cl in red dotted box), we anchor it and randomly perturb the remain part  $\mathcal{G}_{\bar{s}}$  to get graphs  $\mathcal{G}'$  (i.e., molecules in green dotted box). Then we compare the expectation of the prediction of  $\mathcal{G}'$  with the prediction of the original graph  $\mathcal{G}$ . If they are close, most features in  $\mathcal{G}_{\bar{s}}$  must be label-irrelevant. Hence, we can assign high quality for the explanation  $\mathcal{G}_s$ .
So far, there is only one question left: how to ensure that the aforementioned evaluation process respects the data distribution?
STEP 3: OOD Reweighting Block Tailored for GNNs. Before elaborating our OOD block, we first retrospect that in most scenarios, a tiny perturbation would not induce large distribution shifts along the input space, thanks to the approximate continuity of input features (e.g., image pixels in computer vision). Unfortunately, the structural features of GNNs' inputs — adjacency matrix comprising of 0s and 1s — are discrete thus only one perturbation (e.g., adding or deleting an edge) could induce large distribution shifts, and further violate the underlying properties, such as node degree distribution [26], graph size distribution [27] and domain-specific constraints [28].
Thus, it is pivotal to construct an OOD reweighting block for assessing whether the generated graph  $\mathcal{G}'$  deviates from the data manifold. This block is expected to assign an "OOD score" — the degree of distribution shift between  $\mathcal{G}'$  and original graph  $\mathcal{G}$  — to each  $\mathcal{G}'$ . However, it is non-trivial to quantify the degree of OOD [29]. Inspired by the great success of graph anomaly detection [30, 31, 32, 33, 34], we treat the OOD instance as the "anomaly" since it is isolated from the original data distribution, and naturally employ the common usage module of anomaly detection — variational graph auto-encoder (VGAE) [19] containing an encoder and a decoder — to instantiate our OOD reweighting block. Additionally, the great success of diffusion generative models has been recognized recently [35, 36], and thus this domain is deferred for future investigation.
Concretely, as shown in Figure 2, the preparation for evaluations is training the VGAE model on the dataset  $\mathcal{D}$  where input graph  $\mathcal{G}$  is sampled. After that, we can leverage the reconstruction loss to approximate the degree of OOD for any generated instance  $\mathcal{G}'$ . To be more specific:
- Given  $\mathcal{G}' = (\mathbf{A}',\mathbf{X}')$ , the encoder first learns a latent matrix  $\mathbf{Z}$  according to  $\mathbf{A}'$  and  $\mathbf{X}'$ , where row  $\mathbf{z}_i$  corresponds to the node  $v_i'$  in  $\mathcal{G}'$ . Note that  $\mathbf{z}_i$  is assumed to follow the independent normal distributions with expectation  $\boldsymbol{\mu}_i$  and variance  $\sigma_i^2$ . Formally:
$$
q \left(\boldsymbol {Z} \mid \boldsymbol {A} ^ {\prime}, \boldsymbol {X} ^ {\prime}\right) = \prod_ {i = 1} ^ {| \mathcal {V} ^ {\prime} |} q \left(\boldsymbol {z} _ {i} \mid \boldsymbol {A} ^ {\prime}, \boldsymbol {X} ^ {\prime}\right) = \prod_ {i = 1} ^ {| \mathcal {V} ^ {\prime} |} \mathcal {N} \left(\boldsymbol {z} _ {i} \mid \boldsymbol {\mu} _ {i}, \operatorname {d i a g} \left(\boldsymbol {\sigma} _ {i} ^ {2}\right)\right), \tag {3}
$$
where  $\mu$  and  $\sigma$  are parameterized by two two-layer GCNs [37] called  $\mathrm{GCN}_{\mu}$  and  $\mathrm{GCN}_{\sigma}$ .
- Then, the decoder recovers the adjacency matrix  $A'$  based on  $Z$ :
$$
p \left(\boldsymbol {A} ^ {\prime} \mid \boldsymbol {Z}\right) = \prod_ {i = 1} ^ {| \mathcal {V} ^ {\prime} |} \prod_ {j = 1} ^ {| \mathcal {V} ^ {\prime} |} p \left(A _ {i j} ^ {\prime} \mid \boldsymbol {z} _ {i}, \boldsymbol {z} _ {j}\right), \tag {4}
$$
$$
\mathrm {w i t h} p (A _ {i j} ^ {\prime} = 1 \mid \boldsymbol {z} _ {i}, \boldsymbol {z} _ {j}) = \sigma (\boldsymbol {z} _ {i} ^ {\top} \boldsymbol {z} _ {j}),
$$
where  $\sigma (\cdot)$  is the logistic sigmoid function.
- The OOD score of  $\mathcal{G}'$  is given by the normalized reciprocal of the reconstruction loss  $\mathcal{L}_{\text{recon}}(\mathcal{G}')$ ,
$$
\mathcal {L} _ {\text {r e c o n}} \left(\mathcal {G} ^ {\prime}\right) = - \log p \left(\boldsymbol {A} ^ {\prime} \mid \boldsymbol {Z}\right), \text {w i t h} \boldsymbol {Z} = \boldsymbol {\mu} = \operatorname {G C N} _ {\boldsymbol {\mu}} \left(\boldsymbol {A} ^ {\prime}, \boldsymbol {X} ^ {\prime}\right). \tag {5}
$$
Since VGAE is trained on the dataset  $\mathcal{D}$ ,  $\mathcal{G}'$  straying far from the data distribution of  $\mathcal{D}$  would get the high reconstruction loss  $\mathcal{L}_{\text{recon}}$ . Thus, as the reciprocal of  $\mathcal{L}_{\text{recon}}$ , the OOD score of  $\mathcal{G}'$  must be low. Conversely, if  $\mathcal{G}'$  is in distribution, it would gain a high OOD score because it is easy to be reconstructed. Based on this, our OOD block can mitigate the impact of OOD instances. Specifically, the OOD score is utilized as the weight of each prediction when calculating the expectation of the generated graph's prediction. This allows for the marginalization of instances with low OOD scores, as shown in the gray dotted box of Figure 2.
Overall evaluation process. As the last piece of the OAR puzzle, i.e., OOD reweighting block has been instantiated, let's revisit Figure 2 and summarize the overall process of OAR:
1. Before we evaluate the explanatory subgraph  $\mathcal{G}_s$ , the OOD reweighting block (i.e., VGAE) is trained on the dataset  $\mathcal{D}$  where input graph  $\mathcal{G}$  is sampled.
2. Then, we fix the  $\mathcal{G}_s$  and randomly perturb the structure of the complementary part  $\mathcal{G}_{\bar{s}}$  to get  $\mathcal{G}'$ .
3. Each  $\mathcal{G}'$  is fed into GNN  $f$  and VGAE simultaneously to audit prediction and OOD score. Both GNN's behavior and data distribution are taken into consideration in this step.
4. At last, according to the predictions and their weights (i.e., OOD scores), we calculate the weighted average of the generated graphs' predictions. The closer this average is to the original prediction of  $\mathcal{G}$ , the higher the quality of the explanation  $\mathcal{G}_s$  is.
The pseudocode and the tricks to expedite computations are detailed in Appendix A.
2.3 A simplified version of OAR