SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SAME: Uncovering GNN Black Box with Structure-aware Shapley-based Multipiece Explanation
Ziyuan Ye $^{1,2,*}$ , Rihan Huang $^{1,3,*}$ , Qilin Wu $^{1,4}$ , Quanying Liu $^{1,\dagger}$ $^{1}$ Southern University of Science and Technology,  $^{2}$ The Hong Kong Polytechnic University,  $^{3}$ King Abdullah University of Science and Technology,  $^{4}$ Carnegie Mellon University {ziyuanye9801, rihanhuang.work, kyrinwu}@gmail.com, liuqy@sustech.edu.cn
Post-hoc explanation techniques on graph neural networks (GNNs) provide economical solutions for opening the black-box graph models without model retraining. Many GNN explanation variants have achieved state-of-the-art explaining results on a diverse set of benchmarks, while they rarely provide theoretical analysis for their inherent properties and explanatory capability. In this work, we propose Structure-Aware Shapley-based Multipiece Explanation (SAME) method to address the structure-aware feature interactions challenges for GNNs explanation. Specifically, SAME leverages an expansion-based Monte Carlo tree search to explore the multi-grained structure-aware connected substructure. Afterward, the explanation results are encouraged to be informative of the graph properties by optimizing the combination of distinct single substructures. With the consideration of fair feature interactions in the process of investigating multiple connected important substructures, the explanation provided by SAME has the potential to be as explainable as the theoretically optimal explanation obtained by the Shapley value within polynomial time. Extensive experiments on real-world and synthetic benchmarks show that SAME improves the previous state-of-the-art fidelity performance by  $12.9\%$  on BBBP,  $7.01\%$  on MUTAG,  $42.3\%$  on Graph-SST2,  $38.9\%$  on Graph-SST5,  $11.3\%$  on BA-2Motifs and  $18.2\%$  on BA-Shapes under the same testing condition. Code is available at https://github.com/same2023neurips/same
Graph neural networks (GNNs) have demonstrated a powerful representation learning ability to deal with data in non-Euclidean space. However, the explanation techniques for deep learning models on images and text cannot directly apply to understand GNNs [27,18,6]. There is a gap in the understanding of how GNNs work, which largely limits GNNs' application in many fields.
Many GNN explanation techniques aim to examine the extent to which GNNs depend on individual nodes and edges of the graph [22, 35, 19, 24, 37]. However, graph features within nodes and edges contribute different amounts of information when considered individually than contextualized with topology [37, 39]. Therefore, discovering the most important explanation with one or more connected components given an input graph and a well-trained GNN raises the additional challenge of handling structure-aware feature interactions. Recently, a number of studies, including GNN-LRP [25], SubgraphX [38] and GStarX [40] have endeavored to address this issue to some extent. Although many of the current GNN explanation techniques have empirically achieved state-of-the-art
*Equal contribution, co-first author.
explainability performance, the design of new GNN explanation techniques mainly relies upon empirical intuition, iterative experiments, and heuristics principles. To the best of our knowledge, the explanatory capability, potential limitations and inherent properties of GNN explanation techniques have not yet been thoroughly studied from a theoretical perspective.
In this work, we propose a novel Structure-Aware Shapley-based Multipiece Explanation (SAME) technique for fairly considering the multi-level structure-aware feature interactions over the graph by introducing expansion-based Monte Carlo tree search (MCTS). Our construction is inspired by the recently proposed perturbation-based GNN explanation methods [38, 40], which have proven effective for providing explainability from a cooperative game perspective. We summarize the main differences between SAME and previous work in Table The main contributions and novelties of this work include the following. (1) Theoretical aspect: i) We review the characteristics of previous methods [22, 35, 19, 25, 38, 40] and highlight several desired properties (see Table that can be considered by explanation methods for GNNs. ii) We provide the loss bound of the MCTS-based explanation techniques and further verify the superiority of our expansion-based MCTS in SAME compared to previous work [38] in an intuitive manner (Sec. 3.2). (2) Empirical aspect: Our experiments cover both real-world and synthetic datasets. The results show that i) SAME outperforms previous SOTA with fidelity and harmonic fidelity metrics under the same testing condition (Sec. 5.1). ii) SAME qualitatively achieves a more human-intuitive explanation compared to previous methods across multiple datasets (Sec. 5.2).
Table 1: Comparison of the properties of different GNN explanation techniques.
Methods
Properties	Grad
-CAM	GNN Explainer	35 PGExplainer	19 GNN
-LRP	SubgraphX	38 GStarX	40	SAME(Ours)
Graph-level tasks	✓	✓	✓	✓	✓	✓	✓	✓
Node-level tasks	✓	✓	✓	✓	✓	✓	✓
Feature interactions	✓	✓	✓	✓
Structure awareness	✓	✓	✓	✓	✓
Multipiece explanation	✓	✓	✓	✓	✓	✓	✓
Node-wise importance	✓	✓	✓	✓
Substructure-wise importance	✓	✓	✓
Composite-wise importance	✓	✓	✓	✓	✓
Priority-based integration	✓	✓	✓	✓
Redundancy consideration	✓	✓	✓	✓	✓
Note: Feature interactions and structure awareness are discussed in Sec. 3.1 Multipiece explanation is provided in Sec. 3.2 Multi-grained importance (node / substructure / composite), priority-based integration and redundancy consideration are presented in Sec. 4 Detailed mathematical definitions are provided in Appendix B.1.
Improving the explainability of GNN models in a post-hoc fashion has been a theme in deep graph learning. An intuitive way to explain a well-trained GNN is to trace the gradient in the models [322], where the larger gradient indicates the higher importance of node or edge of the graph. Previous work has also studied the decomposition-based methods [26,25,9] which decompose the final prediction into several terms and mark them as the important scores for input features. Another line of GNN explanation techniques lies in perturbation-based methods [19,31,11,34] which usually obtain a mask for the input graph in various ways to identify the important input feature. SubgraphX [38], as one of a perturbation-based method, samples the subgraphs from the input graph by the pruning-based MCTS and finds the most important one via the Shapley value. However, the pruning-based MCTS in SubgraphX leads to a much larger search space and thus causes higher computational costs. Moreover, SubgraphX can only provide a single connected explanation for each graph, which limits its explanatory power in many scenarios that require multipiece explanation. Most recently, GStarX [40] scores node importance based on the Hamiache-Navarro (HN) value. Although GStarX also fairly considers the structure-aware feature interactions, it fails to account for the multi-grained importance, which might result in a suboptimal explanation. For other categories in GNN explanation techniques, including surrogate methods [30,41,8,12], generation-based methods [36,15,32], and counterfactual-based methods [17,2,16], we refer readers to a recent survey [37].
3 Theoretical Motivation
Notation and Preliminaries. The well-trained target GNN to be explained can be formulated as  $f: \mathcal{G} \to \mathcal{V}$ , where  $\mathcal{G}$  denotes the space of input graphs and  $\mathcal{V}$  refers to the related label space. A graph can be denoted as  $G = (V, X, E)$ , where  $V \in \mathbb{R}^{n \times n}$  represents node set,  $X \in \mathbb{R}^{n \times d}$  is the node feature set and  $E \in \mathbb{R}^{n \times n}$  denotes edge set. Given a well-trained GNN model  $f(\cdot)$  and an input graph  $G$ , the goal of GNN explanation is to find the most important explanation  $G_{ex}^*$  from  $G$ . Formally, this can be defined as an optimization problem that maximizes the importance of the explanation  $G_{ex}$  for a given graph  $G$  with  $n$  nodes using an importance scoring function  $I(f(\cdot), G_{ex}, G)$ :
$$
G _ {e x} ^ {*} = \underset {G _ {e x} \subseteq G} {\arg \max } I (f (\cdot), G _ {e x}, G), \tag {1}
$$
where each explanation  $G_{ex}^{i}$  has  $n_i$  nodes, and the other nodes not in the explanation can be expressed as  $\{G\backslash G_{ex}^{i}\} = \{v_{j}\}_{j = n_{i} + 1}^{n}$ . It is noteworthy that each explanation  $G_{ex}^{i}$  might contain one or more substructures (i.e., connected components).
3.1 Structure-aware Shapley-based Explanations Satisfies Fairness Axioms
Unlike grid-like images or sequence-like texts, graphs have more complex and abstract structures. The importance scoring function of explanations determines the reliability and explanatory power of the GNN explanation method. Therefore, we present the idea that a comprehensive assessment of the importance of an explanation should consider the feature interactions under the constraints of the input graph's topology.
Shapley value [28], originating from cooperative game theory, is the unique credit allocation scheme that satisfies the fairness axioms. This concept is similar to the importance scoring function for explanation with the consideration of feature interactions. Some previous work have brought Shapley value into deep learning explanation methods [18, 5, 14]. In our study, the importance assessment of explanation is treated as a cooperative game, where the explanation  $G_{ex}^{i}$  and all nodes not in the explanation  $\{G \backslash G_{ex}^{i}\}$  are the players in the game. Therefore, when scoring the importance of any explanation  $G_{ex}^{i}$ , a set of players participating in the game can be denoted as:
$$
P _ {i} = \{G _ {e x} ^ {i}, \underbrace {v _ {n _ {i} + 1} , v _ {n _ {i} + 2} , \ldots , v _ {n}} _ {\{G \backslash G _ {e x} ^ {i} \}} \}.
$$
Inspired by the close connection between feature interactions and Shapley value, we define several desirable properties of importance scoring function for explanation according to fairness axioms:
Property 1. (Efficiency). The sum importance of all players  $p_j$  in  $P_i$  is the same as the improvement of GNN  $f(\cdot)$  on  $P_i$  over an empty set,  $\sum_{j=1}^{|P_i|} I(f(\cdot), p_j, G) = f(P_i) - f(\emptyset)$ .
Property 2. (Symmetry). For any explanation  $G_{ex}^{i}$ , if there exist two other players  $p_j, p_k \in \{P_i / G_{ex}^i\}$  that satisfy  $f(G_{ex}^i \cup p_j) = f(G_{ex}^i \cup p_j)$ , then  $I(f(\cdot), p_j, G) = I(f(\cdot), p_k, G)$ .
Property 3. (Dummy). If a player  $p_j$  makes no contribution to GNN  $f(\cdot)$ , i.e.  $f(G_{ex}^i \cup p_j) = f(G_{ex}^i)$  holds for any  $G_{ex}^i$ , then  $I(f(\cdot), p_j, G) = 0$ .
Property 4. (Monotonicity). Consider two well-trained GNN models  $f_{1}(\cdot)$  and  $f_{2}(\cdot)$ , given an explanation  $G_{ex}^{i}$ , if for any player  $p_j$ ,  $f_{1}(G_{ex}^{i} \cup p_{j}) - f_{1}(p_{j}) \geq f_{2}(G_{ex}^{i} \cup p_{j}) - f_{2}(p_{j})$  always holds, then  $I(f_{1}(\cdot), G_{ex}^{i}, G) \geq I(f_{2}(\cdot), G_{ex}^{i}, G)$ .
Shapley value can well satisfy the above four fairness Properties [14] However, the Shapley-based importance scoring function attempts to take all nodes in the graph except explanation  $G_{ex}^{i}$  into the cooperation game, which not only ignores the topological information of the input graph but also brings huge computational costs. Fortunately, [38] alleviate this issue by modifying the Shapley-based importance scoring function as 'k-hop Shapley', which also makes it structure-aware:
$$
I (f (\cdot), G _ {e x} ^ {i}, G) = \sum_ {p _ {i} \subseteq \left\{P _ {i, k h o p} \backslash G _ {e x} ^ {i} \right\}} \frac {| p _ {i} | ! \left(| P _ {i , k h o p} | - | p _ {i} | - 1\right) !}{| P _ {i , k h o p} | !} \left(f \left(p _ {i} \cup G _ {e x} ^ {i}\right) - f \left(p _ {i}\right)\right), \tag {2}
$$
where  $I(f(\cdot), G_{ex}^{i}, \mathcal{G})$  denotes the weighted sum of the marginal contribution of explanation  $G_{ex}^{i}$  and  $P_{i,khop} = \{G_{ex}^{i}, v_{n_i + 1}, v_{n_i + 2}, \ldots, v_{n_i + k_i}\}$  includes the nodes within the  $G_{ex}^{i}$  as well as the nodes in the k-hop neighbors of  $G_{ex}^{i}$ .
3.2 Structure-aware Shapley-based Multipiece Explanation Provide Strong Explainability
The characteristics and properties within a graph or node tend to be jointly influenced by more than one high-order connected community of the graph. This implies that the appropriate explanation within this context requires the GNN explanation method to provide the multiple connected substructures simultaneously.
To solve the above challenge, we first define the mathematical formalization of the search processes on graphs by utilizing the MCTS-based GNN explanation methods. Then we propose a mathematically coherent framework to explore the explanatory power of MCTS-based GNN explanation methods using computational methods for the multi-armed bandit problem in the MCTS algorithm [20, 13].
Our mathematical framework is based on the hierarchical partitioning of the MCTS search space  $\mathcal{X}$ . More precisely, the smoothness of the MCTS search space  $\mathcal{X}$  can be defined by the inequality  $|f(x_{i}) - f(x_{j})|\leq l(x_{i},x_{j})$ , where  $l(x_{i},x_{j})$  refers to the Lipschitz continuity between any two substructures  $x_{i}$  and  $x_{j}$  in  $\mathcal{X}$ . It serves as a critical metric to ascertain the boundedness of the change in the explanation method  $f(\cdot)$  concerning the change in input substructures. Assuming that the function  $f(\cdot)$  is Lipschitz continuous and the  $l$  is given, an evaluation of the function  $f(\cdot)$  at any point  $x_{t}$  enables us to define an upper bounding function  $B_{t}(x)$  for  $f(\cdot)$ . This upper bounding function can be refined after each evaluation of  $f(\cdot)$ :
$$
\forall x \in \mathcal {X}, f (x) \leq B _ {t} (x) \stackrel {\text {d e f}} {=} \min  _ {1 \leq s \leq t} [ f (x _ {s}) + l (x, x _ {s}) ], \tag {3}
$$
where metric  $l$  satisfies the following Assumptions [13] In the context of computational uncertainties associated with MCTS, the evaluation strategy in [3] offers the potential to describe specific numerical estimates within an undefinable space.
Assumption 1. (Local smoothness). There exists at least one stage-optimal substructure  $x^{\star} \in \mathcal{X}$  of  $f(\cdot)$  (i.e.  $f(x^{\star}) = \sup_{x \in \mathcal{X}} f(x)$ ) and  $\forall x \in \mathcal{X}, f(x^{\star}) - f(x) \leq l(x, x^{\star})$  holds.
Assumption 2. (Decreasing diameters). There exists a decreasing sequence  $\delta(h) > 0$ , such that for any depth  $h \geq 0$  and for any cell  $\mathcal{X}_{h,i}$  of depth  $h$ ,  $\sup_{\mathcal{X}_{h,i}} l(\mathcal{X}, \mathcal{X}_{h,i}) \leq \delta(h)$  holds.
Assumption 3. (Well-shaped cells). There exists  $\nu >0$  such that for any depth  $h\geq 0$ , any cell  $\mathcal{X}_{h,i}$  contains a  $l$ -ball of radius  $\nu \delta (h)$  centered in  $x_{h,i}$ .
With the given assumptions, the search space  $\mathcal{X}$  can be partitioned into  $K^H$  subsets (i.e., cells)  $\mathcal{X}_{h,i}$  using a  $K$ -ary tree of depth  $H$ , where  $0 \leq h \leq H, 0 \leq i \leq K^{h-1}$ . Based on this partitioning method, the MCTS process can be treated as the expansion of this  $K$ -ary tree. The root of  $K$ -ary tree (i.e., cell  $\mathcal{X}_{0,0}$ ) corresponds to the whole search space  $\mathcal{X}$ . Each cell  $\mathcal{X}_{h,i}$  corresponds to a node  $(h,i)$  of the tree, where  $h$  denotes the depth of the tree and  $i$  refers to the index. Each node  $(h,i)$  possesses  $K$  children nodes  $\{(h+1,i_k)\}_{1 \leq k \leq K}$  s.t. the associated cells  $\{\mathcal{X}_{h+1,i_k}, 1 \leq k \leq K\}$  form a partition of the parent's cell  $\mathcal{X}_{h,i}$ . Consequently, expanding one node requires adding one of its  $K$  children to the current tree, which corresponds to subdividing the cell  $\mathcal{X}_{h,j}$  into  $K$  children cells  $\{\mathcal{X}_{h+1,j_1}, \ldots, \mathcal{X}_{h+1,j_K}\}$ . Assume that there exists a decreasing sequence  $\delta(h) \geq 0$  that satisfies  $\sup_{\mathcal{X}_{h,i}} l(\mathcal{X}, \mathcal{X}_{h,i}) \leq \delta(h)$  for any  $\mathcal{X}_{h,i}$ . The decreasing sequence  $\delta(h)$  ensures that each cell size reduces with increasing depth.
The search space  $\mathcal{X}$  can be divided through the above partitioning method according to  $\delta(h)$ , with respect to  $l$ -open balls. Let  $\mathbf{T}_t$  denote nodes of the current tree, and  $\mathbf{L}_t$  denotes the incoming leaves of  $\mathbf{T}_t$  to be expanded at round  $t$ . Recalling our earlier definition of  $B_t(h)$  which was derived from the Lipschitz continuity, we can now generalize it to a new representation that connects to  $\delta(h)$ . Formally, it is expressed as:
$$
b _ {h, i} = f \left(x _ {h, i}\right) + \delta (h). \tag {4}
$$
Based on this, we now consider which nodes will be expanded during the search. Note that Assumption  $\boxed{2}$  implies that the b-value of any cell contains  $x^{\star}$  upper bounds  $f^{\star}$ . In other words, for any cell  $\mathcal{X}_{h,i}$  such that  $x^{\star} \in \mathcal{X}_{h,i}$ ,
$$
b _ {h, i} = f \left(x _ {h, i}\right) + \delta (h) \geq f \left(x _ {h, i}\right) + l \left(x _ {h, i}, x ^ {\star}\right) \geq f ^ {\star}. \tag {5}
$$
This means that a leaf  $(h,i)$  of a  $K$ -ary tree will never be expanded if  $f(x_{h,i}) + \delta(h) < f^{\star}$ . Therefore, under this partitioning strategy, the only set of nodes that will be expanded could be defined as  $I \stackrel{\text{def}}{=} \cup_{h \geq 0} I_h$ , which could be stated as
$$
I _ {h} \stackrel {\text {d e f}} {=} \left\{\text {n o d e s} \{h, i \} \text {s u c h t h a t} f \left(x _ {h, i}\right) + \delta (h) \geq f ^ {\star} \right\}. \tag {6}
$$
In order to derive a loss bound, we now define a measure of the quantity of near-optimal states, called near-optimality dimension. For any  $\epsilon > 0$ , the set of  $\epsilon$ -optimal states can be defined as
$$
\mathcal {X} _ {\epsilon} := \{x \in \mathcal {X}, f (x) \geq f ^ {\star} - \epsilon \}. \tag {7}
$$
Definition 1. ( $\eta$ -near-optimality dimension). The  $\eta$ -near-optimality dimension is the smallest  $d \geq 0$  such that there exists  $C \geq 0$ , for all  $\epsilon > 0$ , the maximum number of disjoint  $l$ -balls of radius  $\eta \epsilon$  with centers in  $\mathcal{X}_{\epsilon}$  is less than  $C \epsilon^{-d}$ .
Definition represents the number of near-optimal states for a function  $f(\cdot)$  around its optimal solution. It is important to note that  $d$  is not an intrinsic property of  $f(\cdot)$  as we are packing near-optimal states using  $l$ -balls. Instead, it characterizes both  $f(\cdot)$  and  $l$  and depends on the constant. In order to relate this measure to the algorithmic details, we also need to correlate it with the characteristics of the partitioning, specifically the shape of the cells. That is, the near-optimality dimension  $d$  is dependent on a particular constant, which will be determined in accordance with the parameter  $\nu$  as defined in Assumption 3
Lemma 1. Let  $d$  be the  $\nu$ -near-optimality dimension, and  $C$  the corresponding constant. Then  $|I_h| \leq C\delta(h)^{-d}$ .
Building upon Lemma we further analyze the loss of the MCTS-based GNN explanation methods across  $n$  iterations in Theorem
Theorem 1. Let us write  $h(n)$  the smallest integer  $h$  such that  $C\Sigma_{l=0}^{h}\delta(l)^{(-d)} \geq n$ , then the loss of the MCTS-based GNN explanation methods is bounded as:
$$
r _ {n} \leq \delta (h (n)) \tag {8}
$$
The loss  $r_n$  reflects the gap between the obtained and optimal explanations over  $n$  iterations. We aim to bound this loss by  $\delta(h(n))$ , illustrating that refining the partition of the search space  $\mathcal{X}$  reduces the loss, thus better approximating the optimal explanation. We provide a complete description of the mathematical properties and theorem proofs of the framework in Appendix B.2.
Figure 1: Illustration of ground truth explanation and the possible sub-optimal explanations provided by pruning-based MCTS explanation techniques.
Leveraging the mathematical underpinnings provided above, as demonstrated in Figure[1] we employ an example to contextualize the theoretical insights within the GNN explanation. The ground truth
explanation is highlighted in yellow which includes two connected components. When searching for explanations starting from any node in the two components through the pruning-based MCTS, the nodes in other components are accessible only via the unimportant node which is highlighted in blue. In this situation, given sparsity constraints, the pruning-based MCTS can only generate the suboptimal explanation. As a consequence, regardless of the search trajectory adopted, the diameter of the  $l$ -ball remains unyielding, failing to converge to the  $\eta$ -near optimality numerical solution. Therefore, it is necessary to design an explanation method that can accurately retain important nodes while avoiding irrelevant nodes, thus increasing the likelihood of discovering the optimal explanation. This ambition resonates with previously proposed loss bound  $r_n \leq \delta(h(n))$ , emphasizing the need for advanced exploration to reduce losses and approximate the optimal explanation with higher precision.
Figure 2: Overview of Structure-Aware Shapley-based Multipiece Explanation (SAME) method. (a) Important substructure initialization phase aims at searching the single connected important substructure. (b) Explanation exploration phase provides a candidate set of explanations by optimizing the combination of different important substructures. (c) The comparison of the final explanation with the highest importance score from the candidate set with the optimal explanation.
4 Structure-aware Shapley-based Multipiece Explanation Method
As we discussed in Section 3 the structure-aware Shapley-based multipiece explanation provides a potential way to effectively uncover the GNN black box. In order to approximate the optimal Shapley-based explanation, we propose a two-phase framework, Structure-aware Shapley-based Multipiece Explanation (SAME) method, which is composed of (1) an important substructure initialization phase and (2) an explanation exploration phase, as shown in Figure. 2
In the first phase (Section 4.1), we extend an expansion-based Monte Carlo tree search as an important substructure initializer to generate connected components not only of high importance but also of multi-grained diversity. In the second phase (Section 4.2), we apply the important substructure set as an action set in another expansion-based Monte Carlo tree search to explore potential explanations.
4.1 Important Substructure Initialization
In this section, we propose an expansion-based MCTS approach for important substructure initialization. It is intuitive that a favorable initialization should not exclude important substructures at any scale and should not include redundant substructures. Formally, these can be defined as:
Property 5. (Node-wise importance). Given an input graph  $G$  to be explained, for any node  $v_{i} \in G$ , its  $I(f(\cdot), v_{i}, G)$  importance will be considered.
Property 6. (Substructure-wise importance). Given an input graph  $G$  to be explained, for any substructure  $G_{sub_i} \subseteq G$ , its importance  $I(f(\cdot), G_{sub_i}, G)$  will be considered.
Property 7. (Composite-wise importance). Given an explanation  $G_{com}^{i} \subseteq G$  consisting of one or more substructures, its importance  $I(f(\cdot), G_{com}^{i}, G)$  will be considered.
Property 8. (Priority-based integration). Given an explanation  $G_{ex}^{j}$  with any size, the node  $v_{i} \in \{G \backslash G_{ex}^{j}\}$  will be added on  $G_{ex}^{j}$  to get a new explanation  $G_{ex}^{k}$ , if and only if for any  $v_{l} \in \{G \backslash (G_{ex}^{j} \cup v_{i})\}$ ,  $I(f(\cdot), G_{ex}^{j} \cup v_{i}, G) > I(f(\cdot), G_{ex}^{j} \cup v_{l}, G)$  holds.