Towards Multi-Grained Explainability for Graph Neural Networks

Abstract: When a graph neural network (GNN) made a prediction, one raises question about explainability: “Which fraction of the input graph is most inﬂuential to the model’s decision?” Producing an answer requires understanding the model’s inner workings in general and emphasizing the insights on the decision for the instance at hand. Nonetheless, most of current approaches focus only on one aspect: (1) local explainability, which explains each instance independently, thus hardly exhibits the class-wise patterns; and (2) global explainability, which systematizes the globally important patterns, but might be trivial in the local context. This dichotomy limits the ﬂexibility and effectiveness of explainers greatly. A performant paradigm towards multi-grained explainability is until-now lacking and thus a focus of our work. In this work, we exploit the pre-training and ﬁne-tuning idea to develop our explainer and generate multi-grained explanations. Speciﬁcally, the pre-training phase accounts for the contrastivity among different classes, so as to highlight the class-wise characteristics from a global view; afterwards, the ﬁne-tuning phase adapts the explanations in the local context. Experiments on both synthetic and real-world datasets show the superiority of our explainer, in terms of AUC on explaining graph classiﬁcation over the leading baselines. Our codes and datasets are available at https://github.com/Wuyxin/ReFine.


Reviews and author responses:

This paper aims to develop an explainability approach to graph neural networks. The authors introduce such a method coined ReFine which in fact combines local and global aspects. They also provide numerical experiments proving the superiority of their approach, even if only one part of ReFine is used.   Almost all explainability methods focus on local aspects. Then some (but in fact very few) approaches exist which consider global aspects such as characteristics of an entire class. When it now comes to graph neural networks, only some approaches in each of those groups does exist. Thus it is very nice to see that this paper aims to develop a joint approach, which is quite original. The paper is well-written, and the method well-explained as also carefully compared to the state-of-the-art approaches. What is particularly intriguing to see is that even already one part of ReFine, namely the global part, to some extent already outperforms other approaches. Including also the fine-tuning part sets ReFine further apart from existing work, showing its superiority. 

To my mind, this is a very nice paper, which introduce a conceptually new approach that is extremel promising. 


The authors addressed my comments successfully, and I will keep my (positive) rating. No, those issues were not mentioned. It would be indeed important to critically assess not only when this explainability approach succeeds, but also which its limitations are.

This paper develops a method for generating multi-grained explanations for graph neural networks. The basic idea is to first derive high-level explanations for each class and then fine-tune these explanations to obtain low-level explanations for each instance. The experiments show that the developed methods show good performance on various datasets.  1.	Originality: to my knowledge, developing multi-grained explanations is a novel task for understanding GNN, although there exist methods that generate multi-grained explanations for general black-box models [1]. I suggest that the authors discuss [1] in the related work part. The technologies used in this paper are not new and I have some questions about the method design. I will explain this in detail later.

[1] Karthikeyan Natesan Ramamurthy, Bhanukiran Vinzamuri, Yunfeng Zhang, and Amit Dhurandhar. Model agnostic multilevel explanations. In NeurIPS, 2020

2.	Quality & Clarity: the paper is difficult to understand due to confusing/unexplained notations. I also have questions about the method design and whether the developed method fits the initial motivation.

a.	Confusing/unexplained notations: What do “attentive graphs” refer to? I do not see an explanation/definition in the text around this notion. The authors mention that “the class-wise saliency map M^(c) can be generated from Equation (4)”, but Eq. (4) contains only M (not M^(c)).  How do the authors introduce the information of c in Eq. (4)? Another question is about G’: it is mentioned in Eq. (8), but I do not see a clear explanation for it. Also, what is the specific formulation for G^(c)_exp? I only see a vague description about how to derive G^(c)_exp: “with … S^(c), we are able to extract the subgraph G^(c)_exp”. Finally, I can see that S^(c) is more “discrete” (by using selection) and “random”, but how is S^(c) more local?

b.	Method design:  Is G_att^(c) the same for different instances in the same class? I do not think so because X is different for different instances. If G_att^(c) is different for different instances, why can we say that the class-wise attribution module is a “pretraining” step that learns class-wise (but not instance-wise) explanations? Is it achieved only by using L_cts? L_cts is not enough for me because it only forces embeddings for the extracted structures (explanation) to be similar, instead of directly forcing the extracted structures to be similar. I do not see how the design of S^(c) (with selection and randomness) enables the derivation of instance-level explanations. In my understanding, Sec. 3.2 is more “local” compared with Sec. 3.1 majorly because Sec. 3.2 removes L_cts. Is it true? If not, what is the key difference between global/local explanations? Can the authors more clearly explain the rationale for the method design?

3.	Significance: the submission addresses a difficult and novel task and presents interesting results. I believe the work can have good impact if the concerns about its clarity and method design are well addressed.

4.	Minor issues: “Figure 3.1” -> “Figure 2", “explnatory” -> “explanatory”

===========Update after rebuttal===========

I have read the authors' response. I find that most of my concerns remained and would like to keep my score unchanged. The authors have made great effort to respond to my concerns, but unfortunately, I do not think they really understand some of my questions. For example, I am asking about the formulation *mathematical representation) of G^(c)_exp, while they say that they have explained that G^(c)_exp is the explanatory subgraph. That does not really answer my question. The authors say that we can infer that “attentive graphs” is the widely known saliency map because they have mentioned that "Such saliency map can further result in an attentive graph". If saliency map is exactly the same with the attentive maps, why do we need two names, and why use the phrase "further result"?  Yes

The paper proposes a multi-grained GNN explainer framework with the following components: (i) global explainability pre-training aiming at enabling the model to capture important patterns across multiple instances, and (ii) local explainability fine-tuning targeting at customizing the explanatory subgraph for each instance individually. Numerical results show the proposed method overpasses competitors at a significant margin.  (Originality)
The substructure loss is borrowed from PGExplainer (https://arxiv.org/abs/2011.04573), and the contrastive loss is not original, but the combination of them resulting in a new framework is novel to me.

(Quality)
(i) Introduction is well-written revealing the significance and position of the work, that the local explanations fall short in systematizing the prototypical shared patterns, while the globally important substructure may not be essential in local, thus combining them leads to the best world.

However, the execution of the global explanation is a little confused that not matches the description. The paper interprets the PGExplainer loss that maximizing the mutual information between the GNN’s predictions and the underlying structure is local (as used in the local explanation). Then why it is also used in the global explanation (with contrastive loss as auxiliary)? According to the schematic illustration (fig 2), contrastive loss should be treated as the main component in the global explanation. Furthermore, what is the difference between $L_1$ and $L_2$ (specifically, $G_{att}$ and $G_{exp}$)? Is it one continuous but the other discrete in edges?

(ii) One mistake in eq (5) that $\alpha=\sigma(\text{MLP}(\cdot, \cdot))$. Here $\sigma$ should be removed if you refer to PGExplainer eq (4), check the range of their $w_{ij}$.

(iii) About the order of global first then local training, could author provide an ablation that switches the order? Not sure that the best choice should be global first since lots of explainability work the prediction is built on top of local explanations, which is "local first".

Missing citations on contrastive learning on graphs, e.g.
https://arxiv.org/abs/1908.01000
https://proceedings.neurips.cc/paper_files/paper/2020/hash/3fe230348e9a12c13120749e3f9fa4cd-Abstract.html Not applicable.

 Thank you for the comments. We offer the classifications below to solve your concerns and sincerely hope you can reconsider your final score.

**[Comment 1. Discussion of MAME [1]]**

Thanks for bringing MAME [1] to our attention. Below please find the relations and differences between our ReFine and MAME.

1. **Relation**. ReFine and MAME are proposed to obtain the global and local view of model workings and achieve multilevel explainability.
2. **Differences**. First, the explanation of MAME is to hire three terms (for local explanation, explanation sparsity, and group constraint of local explanations), and it adjusts the strength of the group constraint to generate different levels of explanations. Whereas our ReFine introduces a new pre-training & fine-tuning paradigm, where pre-training gets the global view, and fine-tuning enhances the local view. Second, MAME is not proposed to generate GNN explanations, while our ReFine is specific to the GNN explanations.


**[Comment 2. Notation Clarity]**

We will provide a more precise explanation of notations in our final version.

* As we claimed in line 92 ("Such saliency map can further result in an attentive graph"), the attentive graph is the well-known saliency map, i.e., soft masks on features.
* As mentioned at line 113 ("Next we introduce the construction of each class-wise attributor, while we omit the superscript for conciseness"), we omit the superscript "c" for the notations in Equations (3), (4), and (5). We will add them back during revision.
* $\mathcal{G}' $ is an iterated variable, which is the graph instance different from the graph of interest $\mathcal{G}$. Clearer explanations for this notation will be added in our revision.
* As we stated at line 94, $\mathcal{G}^{(c)}_{exp}$ stands for the explanatory subgraph.
* $S^{(c)}$ is generated by $\mathcal{H}$ which is trained by optimizing on the local fidelity optimization, i.e., Equation (10).

**[Comment 3. Method Design]**

We offer the clarifications about the key model design below, and summarize the comparison of pre-training and fine-tuning phases in the table below.

* $G^{(c)}_{att}$ is specific for each graph instance but derived from the class-wise attributor model, which is shared across all graph instances within the same class $c$. Hence, the class-wise attributor has a global view and understanding.
* Minimizing the objective of class-wise attributors ($\mathcal{L}_{1}$) is maximizing a lower bound of conditional mutual information between the saliency maps and the GNN's predictions, conditioning on the class labels. Hence, it can capture some common agreements (including similar structures, properties) within the same classes.
* Minimizing the objective of contrastive learning ($\mathcal{L}_{cts}$) is maximizing a lower bound of mutual information between latent graph representations of two graphs within the same class. Hence, it encourages the distinctions between different classes, emphasizing the discriminative characteristics.
* Minimizing the objective of fine-tuning ($\mathcal{L}_{2}$) is maximizing a lower bound of the mutual information between the explanatory subgraph and target prediction of individual instances. Hence, it is able to achieve the local fidelity.
* $\mathbf{S}^{(c)}$ is a dropping version of $\mathbf{M}^{(c)}$


| Phase                              | Pre-training (Attributor Module)        | Fine-tuning (Selector Module) |
|------------------------------------|-----------------------------------------|-------------------------------|
| Module Inputs                      | $\\{\mathcal{G}\mid Y=c\\}$                              | $\mathcal{G}$                           |
| Module parameters                  | GNN in Eq. 3 and MLP in Eq. 5             | GNN in Eq. 3 and MLP in Eq. 5   |
| Module parameters’ initialization  | Xaiver                                  | pre-trained parameters        |
| Module outputs (i.e., graph masks) | $\mathbf{M}^{(c)}$                               | $\mathbf{S}^{(c)}$                     |
| Module loss                        | class-wise loss $(\mathcal{L_{1}}$ and $\mathcal{L_{cts}})$ | instance-wise loss $(\mathcal{L}_{2})$  |


**[Comment 3. Method Design]**

We sincerely appreciate your comments and will revise them in our updated manuscript.


**Reference:**
1. Karthikeyan Natesan Ramamurthy, Bhanukiran Vinzamuri, Yunfeng Zhang, and Amit Dhurandhar. Model agnostic multilevel explanations. In NeurIPS, 2020

 Thanks so much for your appraisal of our manuscript. We're very glad you rate our work as well-motivated and novel. Furthermore, we find the questions raised in your feedback very insightful, pushing us to improve our paper further.

**[Comment 1. Execution of Global Explainability]**

We apologize and we will present it more clearly in our final version. Here are our responses: According to the taxonomy in the Introduction, we actually categorize PG-Explainer in the group of global explainability since it aims at the globally important patterns across multiple instances (achieved via a universal GNN module). In particular, at lines 56-59, we claim that "It is also worth mentioning that, although the general understanding of GNN predictions has been considered in a recent work PGExplainer, it is only exploited to train a generative probabilistic model shared across all the explained instances, rather than dissecting and modeling the class-wise knowledge explicitly.".

Furthermore, differentiating from PG-Explainer that hires a universal GNN module to maximize "the mutual information between the GNN's predictions and the underlying structure", we have two key components towards the global explainability: (1) minimizing the objective of class-wise attributors is maximizing a lower bound of conditional mutual information between the saliency maps and the GNN's predictions, conditioning on the class labels; and (2) minimizing the objective of contrastive learning is maximizing a lower bound of mutual information between latent graph representations of two graphs within the same class.

**[Comment 2. Contrastive Loss]**

As for the contrastive loss, we do treat it as an important component towards the global explanations. In particular, in our experiments, we set $\gamma=1$ in Equation (6), which indicates the equal importance of minimizing the class-wise attributors ($\mathcal{L_1}$) and the contrastive loss ($\mathcal{L_{cts}}$).

**[Comment 3. Difference between Two Losses]**

The key difference between $\mathcal{L_1}$ and $\mathcal{L_2}$ lies in two parts: (1) In $\mathcal{L_1}$ attributor models is random initialized, while in $\mathcal{L_2}$ attributor models are pre-trained; (2) $G_{att}$ assigns an attentive mask on the adjacency matrix, while $G_{exp}$ selects the top-K edges and remains these edges' importance scores. Assuming there as 5 edges in the input graph, $G_{att}$ could be [0.9, 0.1, 0.2, 0.7, 0.5], and $G_{exp}$ could be [0.8, 0, 0, 0, 0.6] when the budge of explanatory edges is 2.


**[Comment 4. Reparameterization Trick]**

Thank you very much for the useful suggestions, and we appreciate the detailed comments about our equations. We sincerely apologize for the incorrectness of $\alpha=\sigma(\text{MLP}(\cdot,\cdot))$ in Equation (5). It differs from our actual formulation and code, which is included in the link. We will correct it as $\alpha=\text{MLP}(\cdot,\cdot)$ in our updated manuscript. Thanks again for your useful feedback.

**[Comment 5. Order of Pre-training & Fine-tuning]**

It is a good idea. However, in the current paradigm, it is impossible to switch the order of pre-training and fine-tuning. Because the pre-trained parameters are required when training locally, and $G_{att}$ cannot be built on the top of $G_{exp}$. Hence we leave this switch in future work.

**[Comment 6. Missing Citations]**

Thanks for pointing out some missing related work. We will add them during revision.

 Thank you for the appraisal of our work. We genuinely value your positive comments on novelty.  As for the limitations, we have mentioned the limitation concerning the explanation conciseness in line 314-316. 

 Thank you for your feedback. We offer the clarifications below to resolve your concerns.

**[Comment 1. Difference from GEM [1]]**

Thanks for bringing the GEM paper [1] to our attention. Although the ICML 2021 conference date and the date officially released in ArXiv are after the NeurIPS submission, we find the paper studying the GNN explainability and will definitely compare it in our final version. Below please see our clarifications of our differences from them:

1. **Paradigm.** Distinct from GEM that follows the conventional training paradigm to implicitly capture the global and local views from all graph instances, ReFine adopts the novel pre-training & fine-tuning paradigms to explicitly model class-wise knowledge and local fidelity. As a result, our ReFine is able to output the explanations with a global view and local view simultaneously, whereas GEM fails.
2. **Method.** Take a close look at the attribution module. Technically, upon node representations, GEM applies the inner product on the representations of the ending nodes to get the edge attribution; differently, our Refine uses a MLP on node representations and the reparameterization trick to get the edge attribution. 

**[Comment 2. Time Complexity]**

Thanks for the suggestions. We will provide the experiment and analysis of time complexity in our final version. We summarize the time costs in the tables below. The first table shows the training time cost (in second) per epoch, while the second table shows the inference time cost (in second) per instance. Clearly, our ReFine is more efficient than GNNExplainer and is computationally comparable to PG-Explainer.


| |Mutagenicity | VG-5    | MNIST   | BA-3motif |
|-----|-----|---------|---------|-----------|
| PG-Explainer | 36.4 | 39.0 | 170.6|  25.9  |
| ReFine (fine-tuning) | 39.1| 43.5| 175.1  |27.8 |


| |Mutagenicity | VG-5    | MNIST   | BA-3motif |
|-----|-----|---------|---------|-----------|
| GNNExplainer |2.03 | 1.88| 0.637 | 1.11|
| PG-Explainer | 0.030 |0.035 | 0.040 | 0.032|
| ReFine (fine-tuning) | 0.821  |0.583|  0.535 |0.423 |

**Reference:**

[1] Generative Causal Explanations for Graph Neural Networks. ICML, 2021.

his paper exploits the pre-training and fine-tuning techniques to develop the explainer and generates multi-grained explanations from a global perspective and a local perspective. The proposed method combines contrastive learning into class-wise generative probabilistic models. Experiments on both synthetic and real-world datasets  are conducted
  **Strengths:** Previous works rarely consider the explanations of GNN from the perspective of multi-grained and contrastive learning. These ideas are primarily used in the field of images before. It is a good direction for improvement to explain GNN from a global and local perspective and shorten the distance within the class. In the experimental section, quantitative and qualitative analyses are performed, results show that the proposed method is effective.

**Weaknesses:** 
1: The class-aware attribution module uses the output of the decoder as an importance score and selects the node of top-p as the subgraph of explanation. The module looks quite similar to [1]. More discussion on the differences are expected.

2: Time complexity analysis and experiments are missing.

[1] Generative Causal Explanations for Graph Neural Networks. ICML, 2021.

**Correctness:** The paper is generally correct except for a typo. In line 98, “Figure 3.1” should be referred to as Figure 2 in the paper.

**Clarity:** The paper is well written and easy to read.

**Relation to Prior Work:** The related work is comprehensive in general and can be investigated from the perspective of global explainability and local explainability. No

Meta-review: In this paper, the authors propose an explainability approach to graph neural networks (GNN). The fundamental idea is to first derive high-level explanations for each class and then fine-tune these explanations to obtain low-level explanations for each instance. Reviewers agree that the problem in this paper is important. During the discussion, we carefully consider the novelty of the proposed method over the recent works (NeurIPS 2020) and we agree that the GNN formulation has some merits. However,  there still exist some concerns in experiments for publication. Thus, I encourage authors to revise the paper based on reviewer's comments and resubmit it to a future venue.

Decision: accept