SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Property 9. (Redundancy consideration). Given an explanation  $G_{ex} \subseteq G$ , if  $I(f(\cdot), G_{ex} \setminus \{i\}, G) > I(f(\cdot), G_{ex}, G)$  holds, the new explanation  $G_{ex}' = G_{ex} \setminus \{i\}$  will be chosen.
Taking the above properties into account, we propose our expansion-based Monte Carlo tree search for important substructure initialization which aims at providing important connected components of the graph. The detailed Algorithm is presented in Appendix C. Given a graph  $G$  to be explained, the node in the MCTS is defined as  $N_{i}$  which contains the following variables:
$$
N _ {i}: \left\{G _ {\text {s u b}} ^ {i}, T _ {i}, R _ {i}, A _ {i}, C _ {i}, W _ {i} \right\}
$$
where  $G_{sub}^{i}$  denotes the corresponding substructure of graph  $G$  for node  $N_{i}$  in the search tree.  $T_{i}$  is the visiting time of node  $N_{i}$  in the search tree,  $R_{i}$  refers to the importance (reward) of substructure  $G_{sub}^{i}$ .  $A_{i}$  represents the action set of node  $N_{i}$ .  $C_{i} = \{N_{j}\}_{j=1}^{|A_{i}|}$  represents a set of children with respect to node  $N_{i}$ , and we denote  $N_{j}$  as one of the child nodes obtained through the action  $a_{j} \in A_{i}$  at parent node  $N_{i}$ .  $W_{i}$  means the sum of the children's rewards.
The expansion-based MCTS is initialized with an empty set,  $N_0 = \emptyset$ . At the beginning of each iteration, our method will randomly choose an unvisited node from the graph, or choose the node with the highest reward if all nodes have been visited. Then, the tree will be iteratively expanded according to the node-wise tree expansion until it reaches a leaf node. Specifically, given an MCTS node  $N_i$ , it will only choose the child node within 1-hop neighbors of the associate  $G_{sub}^i$  to expand, which means that the action set is topology dependent. Therefore, the substructure of any nodes in the search tree will be a connected component. During the child node selection, the reward of each child node  $R_j$  of  $N_i$  will be calculated following Equation (2). Finally, the chosen action is decided following the child selection strategy:
$$
a ^ {\star} = \arg \max  _ {a _ {j}} \frac {W _ {j}}{T _ {j}} + \beta R _ {j} \frac {\sqrt {\sum_ {k \in C _ {i}} T _ {k}}}{1 + T _ {j}} \tag {9}
$$
where  $\beta$  is a hyperparameter for balancing exploration-exploitation trade-off [4]. We define the maximum substructure size as  $\gamma$ , and for any  $G_{sub}^{i} \subseteq G$ ,  $|G_{sub}^{i}| \leq \gamma$  always holds. After reaching the maximum substructure size, the reward of associated leaf nodes is backpropagated up the tree along the search path, updating all the information stored in each node of the path. The important substructure set includes all nodes in MCTS after performing  $M_{1}$  times iterations.
4.2 Explanation Exploration
In this section, we propose another expansion-based MCTS for exploring high-explainable explanations. The detailed algorithm is provided in Appendix C. Different from the previous expansion-based MCTS provided in Section 4.1 we propose a slight modification on it such that we change the action set from node level to substructure level. As Section 3.2 discussed, this can be useful not only to obtain more flexible explanation results on real-world cases but also to provide the higher potential to approximate the theoretically optimal Shapley-based explanation.
The action set of MCTS in this phase is built upon the substructure set derived from the important substructure initialization phase. Similar to the previous MCTS, at the beginning of each iteration, it will randomly select an unvisited substructure from the set, or choose the substructure with the highest reward to expand if all substructures have been visited. Afterward, the substructure-wise tree expansion also follows the Equation 2 and 9 to develop the tree and provide the explanation candidate. The action set corresponding to each node of MCTS in the phase is the whole substructure set. To further accelerate the exploration, we filter the unimportant substructures by only keeping the top K important substructure in the action set.
Notice that calculating all possible combinations of either substructures or nodes can definitely obtain the optimal Shapley-based explanation. Nevertheless, such node-wise or substructure-wise brute force methods lead to  $O(2^{|V|})$  computational complexity, which is an NP-hard problem. We provide a feasible solution to approximate the optimal Shapley-based explanation in polynomial time  $\mathcal{O}(M_1\gamma |V|^2 + M_1\gamma |V| \times |E| + M_2Kt_s\frac{|V| \times (1 - sparsity)}{\gamma}$ , where  $M_1$  and  $M_2$  denote maximum number of iterations of the first phase and second phase respectively,  $\gamma$  is the maximum substructure size,  $K$  refers to the size of the important substructure set and  $t_s$  is the time budget of MCTS in the second phase. We leave the derivation of the time complexity in Appendix B.4.
Table 2: Comparison of our SAME and other baseline using fidelity.
Methods\Dataset	Graph classification	Node classifier.
Molecular graph	Semantic graph	Synthetic graph
BBBP	MUTAG	Graph-SST2	Graph-SST5	BA-2Motifs	BA-Shapes
Grad-CAM [22]	0.226±0.036	0.261±0.018	0.257±0.056	0.229±0.042	0.472±0.010	-
GNNExplainer [35]	0.148±0.041	0.188±0.031	0.143±0.041	0.170±0.046	0.442±0.026	0.154±0.000
PGExplainer [19]	0.197±0.043	0.156±0.004	0.219±0.040	0.207±0.036	0.431±0.011	0.135±0.020
GNN-LRP [25]	0.111±0.040	0.253±0.030	0.103±0.042	0.131±0.057	0.146±0.010	0.155±0.000
SubgraphX [38]	0.433±0.073	0.379±0.030	0.262±0.027	0.283±0.042	0.493±0.003	0.181±0.005
GStarX [40]	0.117±0.043	0.656±0.096	0.183±0.050	0.186±0.050	0.476±0.014	-
SAME	0.489±0.034	0.702±0.125	0.373±0.042	0.393±0.022	0.549±0.004	0.214±0.000
Relative Improve	12.9%↑	7.01%↑	42.3%↑	38.9%↑	11.3%↑	18.2%↑
Note: The fidelity results are averaged across different sparsity from 0.5 to 0.8. The quantitative results are presented in the form of mean  $\pm$  std. The previous SOTA results on different datasets are marked with an underline. Relative Improve denotes the relative improvement of our SAME method over the SOTA methods.
Our objective of the experiments is to understand the following two questions. 1) Are the explanations provided by SAME more informative and faithful compared to other methods under the same test conditions? 2) Can SAME provide a more human-intuitive explanation than others? To this end, we perform extensive quantitative and qualitative analysis to evaluate the explanatory power of SAME, following previous literature [38][40]. The SAME is compared with various competitive baselines and shows state-of-the-art (SOTA) results in all the cases.
Dataset. The experiments are conducted on six datasets with diverse categories, including molecular graphs (e.g., BBBP [33] and MUTAG [7]), sentiment graphs (e.g., Graph-SST2 and Graph-SST5 [29]) and synthetic Barabási-Albert graphs (e.g., BA-2Motifs [19] and BA-Shapes [35]). We conduct a node classification task for the BA-Shapes dataset, and graph classification tasks for the rest five datasets. More detailed descriptions of datasets are provided in Appendix D.
Metrics. In this work, we use several criteria [37] to evaluate our approach: (1) Sparsity quantifies how compact are the explanations, and further facilitates fair comparison by restricting the different explanations to the same size. (2) Fidelity determines how informative and faithful are the explanations by removing the selected nodes. (3) Inv-Fidelity measures the explanations from the same aspect as fidelity while it keeps the selected nodes. (4) Harmonic fidelity [40] normalizes fidelity by sparsity and takes a harmonic mean to make different explanations comparable with a single metric. We leave detailed mathematical definitions of the above metrics in Appendix E.
Experimental setup. In the important substructure initialization phase, we set the MCTS iteration number  $M_{1}$  to 20. The exploration-exploitation trade-off  $\beta$  is set to 5 for BBBP and 10 for other datasets. The substructure size  $\gamma$  has different settings in different datasets. In the explanation exploration phase, we set the hyperparameter  $K = 7$  for important substructure filtering,  $M_{2} = 10$  for the MCTS iteration number, and the other hyperparameters of MCTS remain the same as the previous phase. We follow [37,40] to set other baselines hyperparameters. All methods are implemented in PyTorch [21] and PyG [10]. Our experiments are conducted on a single Nvidia V100 GPU with an Intel Xeon Gold 5218 CPU. We leave the detailed settings in Appendix F.
Table 3: Comparison of inference time (in seconds) on different datasets.
Dataset Methods	BBBP	MUTAG	Graph-SST2	Graph-SST5	BA-2Motifs	BA-Shapes
Grad-CAM 22	0.16	0.23	0.39	0.44	0.14	-
GNNExplainer 35	7.56	1.96	7.64	19.39	1.89	2.72
PGExplainer 19	0.15	0.21	0.35	0.43	0.12	0.13
GNN-LRP 25	2.37	1.97	5.84	5.47	3.30	51.77
SubgraphX 38	26.72	151.75	36.48	71.32	85.50	162.80
GStarX 40	84.54	25.24	30.64	54.49	77.99	-
SAME	7.86	5.67	6.06	8.83	8.19	14.08
Note: The PGExplainer needs training before inferring the explanation.
Sentence: "a carefully structured scream of consciousness that is tortured and unsettling -- but unquestionably alive."
Figure 3: Comparison of the explanations on Graph-SST2 with GCN classifier.
5.1 Quantitative Analysis
To validate the overall explainability performance, we compare the proposed SAME with a series of competitive baselines under different metrics. Table2 shows the averaged fidelity under different sparsity (i.e., sparsity=[0.5,0.6,0.7,0.8]). The proposed SAME significantly outperforms the previous state-of-the-art on both real-world and synthetic datasets. Specifically, the performance improvement of SAME is  $12.9\%$  on BBBP,  $7.01\%$  on MUTAG,  $42.3\%$  on Graph-SST2,  $38.9\%$  on Graph-SST5,  $11.3\%$  on BA-2Motifs and  $18.2\%$  on BA-Shapes. Notably, we also demonstrate reliable improvements of SAME over previous SOTA methods in terms of harmonic fidelity at different sparsities, with an average improvement of  $1.92\%$  on the graph classification task. This result implies SAME has a higher ability to discover important components in the graph than all baseline methods. As for the inv-fidelity metric, the explanatory power of SAME is competitive compared with the previous SOTA. Detailed results for harmonic fidelity and inv-fidelity are in Appendix G.1. Since the proposed SAME is a model-agnostic method, it works well with GAT and GIN. More comparisons with GraphSVX [8] and OrphicX [16] are provided in Appendix G.3.
The computational cost of SAME and other baselines on different datasets are summarized in Table 3. We show that SAME consistently achieves much lower computational cost compared to GStarX and SubgraphX, reflecting its efficiency and robustness. This further verifies that expansion-based MCTS in SAME can work effectively in various scenarios.
Figure 3 presents the visualization comparison of the explanations on sentiment graphs. The nodes of adjectives and adverbs are considered to be important for they reveal the attitude of a sentence. Thus, the optimal explanation here is therefore the word or phrase with a positive meaning. In this sense,
Figure 4: Comparison of the explanations on MUTAG with GIN classifier.
Figure 5: Comparison of the explanations on BA-2Motifs with GCN classifier.
SAME can better capture the adjectives-or-adverbs-like graph structures than other baselines. For instance, SubgraphX focuses on adjectives and adverbs but fails to capture the "but" word which bears significant weight under the contrasting relationship. Intuitively, without the "but", the contribution of "tortured" and "unsettling" should be negative. GStarX achieves to identify words that are consistent with the opinion such as "carefully" and "alive," yet overlooks the crucial transitional relationship between "alive" and "unsettling". Figure 4 shows the visualization of explanations on molecular graphs. The ground truth explanations (i.e., functional group  $-\mathrm{NO}_2$ ) of MUTAG are labelled by human experts. We see that SAME is able to provide the explanations the same as the ground truth. We also illustrate the explanation of the synthetic graph in Figure 5. The ground-truth label of all the graphs in BA-2Motifs is a 5-node-house-structure motif. Results show that SAME exactly finds the ground-truth explanation compared to other baselines. We leave more comparisons in Appendix G.2.
Structure-aware Shapley-based Multipiece Explanation provides strong explainability over GNN models, while this ability is limited by only using the single connected substructure. Moving forward from the theoretical deduction, we propose the SAME method for explaining GNN, a novel perturbation-based method that is aware of input graph structure, feature interactions, and multi-grained importance. Experimental results demonstrate that our SAME consistently outperforms SOTA methods on multiple benchmarks by a large margin with various metrics and provides a more understandable explanation.
Limitations. In the implementation of SAME, the Shapley value is obtained through approximation following [38]. Under this approximation, the fairness axioms discussed in Section [3.1] no longer hold. This is also identified as an unsolved issue for the Shapley value in machine learning by [23]. In addition, the scalability of SAME on large graphs can also become a potential challenge. As the time complexity shown in Section [4] the time overhead caused by the increase in the number of nodes will become very expensive when scaling up the size of the input graph.
This work was funded in part by the National Key R&D Program of China (2021YFF1200804), National Natural Science Foundation of China (62001205), Shenzhen Science and Technology Innovation Committee (2022410129, KCXFZ2020122117340001), Shenzhen-Hong Kong-Macao Science and Technology Innovation Project (SGDX2020110309280100), Guangdong Provincial Key Laboratory of Advanced Biomaterials (2022B1212010003).
We would like to extend our sincere appreciation to Dr. Jialin Liu for her insightful suggestions on the manuscript composition and the elucidation of definitions.
[37] H. Yuan, H. Yu, S. Gui, and S. Ji. Explainability in graph neural networks: A taxonomic survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[38] H. Yuan, H. Yu, J. Wang, K. Li, and S. Ji. On explainability of graph neural networks via subgraph explorations. In International Conference on Machine Learning, pages 12241-12252. PMLR, 2021.
[39] H. Zhang, B. Wu, X. Yuan, S. Pan, H. Tong, and J. Pei. Trustworthy graph neural networks: Aspects, methods and trends. arXiv preprint arXiv:2205.07424, 2022.
[40] S. Zhang, Y. Liu, N. Shah, and Y. Sun. Gstarx: Explaining graph neural networks with structure-aware cooperative games. In Advances in Neural Information Processing Systems, 2022.
[41] Y. Zhang, D. Defazio, and A. Ramesh. Relex: A model-agnostic relational model explainer. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 1042–1049, 2021.
Appendix to “SAME: Uncovering GNN Black Box with Structure-aware Shapley-based Multipiece Explanation”
A Notations 1
B Theoretical Analysis 2
B.1 Definitions of Desire GNN Explanation Technique Properties 2
B.2 Discussion about Four Fairness Properties 3
B.3 Detailed Proof of Regret Bound 3
B.4 Analysis of Time Complexity 4
B.4.1 Important Substructure Initialization Phase 4
B.4.2 Explanation exploration phase 5
C Detailed Pseudo Code of SAME 5
D Dataset Description 5
E Metrics 8
F Detailed Experimental Settings 8
F.1 Computation Details 8
F.2 Hyperparameter Settings 8
G Additional Results 9
G.1 Results under Other Metrics 9
G.2 Additional Visualization Results 9
G.3 Additional Quantitative Analysis 11
This appendix comprises both theoretical and experimental materials and is structured as follows. Section A presents a notation table that includes all notations used in this paper. Section B provides the full theoretical analysis. Section C presents detailed algorithms related to Section 3 in the manuscript. Section D-F show the detailed settings of the experiments in this work. Section G provides additional quantitative and qualitative results.
Table S1 contains the notations organized into three sections, each pertaining to different aspects of our paper. These include:
- Graph: These symbols are related to graph theory, representing elements such as graph nodes, edges, and other attributes of graphs.
- MCTS: These symbols represent elements related to the Monte Carlo Tree Search (MCTS) algorithm, such as MCTS nodes, actions, and rewards.
- Importance score: These symbols are used to denote values associated with measuring the importance of explanations in our explanation algorithm.
Table S1: Notations of the graph (see top part), MCTS (see middle part) and importance score (see bottom part).
Notation	Description
G	The graph set to be explained. For any input graph G ∈ G, G = (V, X, E), V = {v1, v2, ..., vn} denotes node set, E ∈ Rn×n denotes edge set and X ∈ Rn×d denotes node feature set.
Gex	The set of the explanation candidates for a given graph G. Gex = {Gexi}m i=1, where Gex ⊆ G, m is the number of explanation candidates
Gi sub	Each Gi sub is a connected component, and is called a substructure
Gi com	Each composite Gi sub contains more than two nodes or substructures.
Gi ex	Each Gi ex refers to a possible final explanation, i.e., Gex = {Gsubj}l i j=1, where li is the number of substructures in explanation Gi ex
Ti	The number of visit for node i in MCTS
Ri	The reward of node i in MCTS
Ci	The child set of node i in MCTS
Ai	A set of actions of MCTS node i to reach its children
Wi	The reward of all children for MCTS node i
N	The node set of the corresponding MCTS of the input graph G. Each node of MCTS can be denoted as Ni ∈ N, where Ni is a structure containing {Gex, Ti, Ri, Ai, Ci, Wi}.
π	The strategy to choose the best child in MCTS.
Pi	A set of players (including all nodes in the input G).
f(·)	A well-trained GNN model.
I(·,·,·)	The importance score, where we employ Shapley value here in this paper.
|·|	The size of a set.
B.1 Definitions of Desire GNN Explanation Technique Properties
The following content is a detailed definition of desired properties in Table 1 of the main manuscript.
- Graph-level tasks: the GNN explanation method can handle the graph classification/regression tasks.
- Node-level tasks: the GNN explanation method can handle the node classification/regression tasks.
- Feature interactions: given a graph  $G$  to be explained, when measuring the importance of the explanation result  $G_{ex}$ , the GNN explanation method can consider the influence of  $\{G / G_{ex}\}$  on the importance of  $G_{ex}$ .
- Structure awareness: given a graph  $G$  to be explained, when measuring the importance of the explanation result  $G_{ex}$ , the GNN explanation method is sensitive to the topology of the given input graph  $G$ .
- Multipiece explanation: the GNN explanation method can provide the explanation  $G_{ex}$  that can be composed of one or more connected components.
- Node-wise importance: given any node  $v_{i} \in G$ , its importance  $I(f(\cdot), v_{i}, G)$  can be considered by the GNN explanation method.
- Substructure-wise importance: given any substructure  $G_{sub}^{i} \subseteq G$ , it can be directly calculated as a whole through the GNN explanation method to obtain its importance  $I(f(\cdot), G_{sub}^{i}, G)$ . Note that any substructure is a connected component which has more than one node.
- Composite-wise importance: given any composite  $G_{com}^{i} \subseteq G$  consisting of more than two nodes or substructures, it can be directly calculated as a whole through the GNN explanation method to obtain its importance  $I(f(\cdot), G_{com}^{i}, G)$ .
- Priority-based integration: given an explanation  $G_{ex}^{j} \subseteq G$  with any size, the node  $v_{i} \in \{G / G_{ex}^{j}\}$  will be added by the GNN explanation method on  $G_{ex}^{j}$  to get a new explanation, if and only if for any node  $v_{k} \in \{G / G_{ex}^{j}\}$  except  $v_{i}$ ,  $I(f(\cdot), \{v_{i} \cup G_{ex}^{j}\}, G) \geq I(f(\cdot), \{v_{k} \cup G_{ex}^{j}\}, G)$  always holds.
- Redundancy consideration: given an explanation  $G_{ex}^{j} \subseteq G$ , if there exist a node  $v_{i} \in \{G / G_{ex}^{j}\}$  that satisfies  $I(f(\cdot),\{G_{ex}^{j} / v_{i}\}, G) \geq I(f(\cdot),G_{ex}^{j},G)$ , the GNN explanation technique would not choose  $G_{ex}^{j}$  as the final explanation.
Brain Networks Associated with Specific Cognitive Task
Functional Groups Associated with Molecular Property
Figure S1: Examples of the characteristics and properties within a graph or node.
Social Groups Associated with Single Person
As Figure S1 shows, the characteristics and properties within a graph or node tend to be jointly influenced by more than one high-order connected community of the graph. The "multipiece explanation" property is aware of such an attribute by considering whether multiple disconnected pieces are allowed to appear together in the explanation output. For example, when "A" and "B" are two orthogonal connected components and the ground truth explanation for a prediction is "A and B", then the GNN explanation method has a probability of providing the correct "A and B" explanation.
We call a GNN explanation method that has the "structure-awareness" property if it can process the structural information of the input graph, whether through explicit mechanisms, as demonstrated by GstarX, or implicit modalities, as exemplified by SubgraphX and this study.
B.2 Discussion about Four Fairness Properties
Efficiency demonstrates that the aggregated importance of individual substructures is equivalent to the prediction of GNN on the entire substructure set. Symmetry and Dummy take into account the equal importance of substructures with the same interactions and the substructures that are completely unimportant, respectively. Monotonicity ensures that the explanation method is consistent in the trend of results between different well-trained GNN models given the same dataset.
B.3 Detailed Proof of Regret Bound
The proper definition of the regret phenomenon that the algorithm or the function  $f$  encounters must be considered to make the objective clear. Prior studies have recognized two principal definitions:
We recall the Lemma 1 in the main manuscript as follows.
Lemma 1. Let  $d$  be the  $\nu$ -near-optimality dimension (where  $\nu$  is defined in Assumption 3 of the main manuscript), and  $C$  is the corresponding constant. Then  $|I_h| \leq C \delta(h)^{-d}$ .
Proof. From Assumption 3 in the main manuscript, each cell  $(h,i)$  contains a ball of radius  $\nu \delta(h)$  centered at  $x_{h,i}$ . Consequently, if  $|I_h| = |\{x_{h,i} \in X_{\delta(h)}\}|$  exceeds  $C\delta(h)^{-d}$ , this would imply the existence of more than  $C\delta(h)^{-d}$  disjoint  $l$ -balls of radius  $\nu \delta(h)$ , each centered within  $X_{\delta(h)}$ . This assertion would contradict the definition of  $d$ .
We recall the Theorem 1 in the main manuscript as follows.
Theorem 1. Let us write  $h(n)$  the smallest integer  $h$  such that  $C\sum_{l = 0}^{h}\delta (l)^{-d}\geq n$ . Then the loss of algorithm is bounded as  $r_n\leq \delta (h(n))$ .
Proof. Consider the tuple  $(h_{\max}, j_{\max})$  to be the node with the maximum depth that the algorithm expands up to the  $n$ th round. Given that the algorithm's expansion is limited to nodes within the set  $I$ , we can provide that the total number of nodes expanded, denoted by  $n$ , adheres to the following condition:
$$
\begin{array}{l} n = \sum_ {l = 0} ^ {h _ {\max }} K ^ {l} \sum_ {j = 0} ^ {K ^ {l} - 1} \mathbb {1} \left\{\left(h, j\right) \text {h a s b e e n e x p a n d e d} \right\} \\ \leq \sum_ {l = 0} ^ {h _ {\max }} | I _ {l} | \\ \leq C \sum_ {l = 0} ^ {h _ {\max}} \delta (l) ^ {- d}, \\ \end{array}
$$
from Lemma 1. Now from the definition of  $h(n)$  we have  $h_{\max} \geq h(n)$ . Finally, since node  $(h_{\max}, j_{\max})$  has been expanded, we have that  $(h_{\max}, j_{\max}) \in I$ , thus  $f(x(n)) \geq f(x_{h_{\max}, j_{\max}}) \geq f^{\star} - \delta(h_{\max}) \geq f^{\star} - \delta(h(n))$ .