Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper for the first time adopts motif extraction as a pre-process step to generate the candidate set for explanation, and employs an attention-based method to identify the most important motifs as explanations.

strength_and_weaknesses: **Strength**
- Using motifs as explanations is a new and motivating idea
- The proposed algorithm is simple and effective
- The writing is easy to follow

**Weaknesses**
- The method heavily relies on the quality of motif extraction rules, whose complexity should also be discussed
- Some details need to be explained as commented in my summary

clarity,_quality,_novelty_and_reproducibility: The writing and presentation is in general clear and easy to follow. The novelty lies in the using motifs to provide explanations, which triggers a simple (with limited technical contribution) yet effective method. The codes are provided for reproducibility.

summary_of_the_review: This paper provides a new way to explain GNNs via motifs. The method is simple, practical and effective, while the idea is straight-forward with limited technical contribution. I also have the following questions.
- The method requires a motif extraction step, which heavily relies on domain knowledge. As a fair comparison, what if directly applying the general motif extraction algorithm for all datasets, including the molecule datasets?
- The complexity of the motif extraction algorithm should be discussed, especially when the graph is large.
- There is a contradiction in the method description: $y$ is the predicted label based on Figure 1, but in Algorithm 1 it represents the ground truth label of the graph. Which one is adopted in the actual design?
- How large is the motif list for different datasets, compared with the candidate set of SubgraphX?
- How are the datasets split for training and testing?
- The fidelity metric defined in Eq. (10) is unclear: the output of the feature extractor $\Psi$ should be the embedding, then what is the meaning of the subscript $y_i$? Presumably, it should be based on the prediction accuracy or probability (e.g. output of the classifier $\xi$) as in [1].
- The reported evaluation metric in Table 1 should be explained in the title to avoid confusion: for BA-2Motif and BA-Shape datasets, the metric is accuracy.
- In section 2, “donate” -> “denote”

[1] Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji. Explainability in graph neural networks: A taxonomic survey. arXiv preprint arXiv:2012.15445, 2020b.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to explain predictions of graph neural networks (GNNs). Existing GNN explanation methods identify important edges, nodes, or arbitrary substructures but fail to consider recurrent and interpretable substructures. This work proposes MotifExplainer to explain GNNs by identifying important motifs. Empirical studies on both synthetic and real-world datasets demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Strengths
1. Using the attention mechanism for motif-based explanation seems like an efficient and reasonable approach.
2. Diverse experiments are done to demonstrate the superiority of the proposed approach.
3. This paper is generally written well, and there is not much confusion in understanding it.

Weakesses
1. The proposed approach requires domain-specific motifs as its inputs. This makes the approach less generalizable, and its improved accuracy as the result of adding domain-specific knowledge into the algorithm, rather than the result of technical improvements. One important goal of GNN explanation is to find new, meaningful substructures from the graph which play a key role in the prediction, which give novel insights and observations about the given graph. This work does not have such an advantage.
2. The technical contribution is limited. Using attention for GNN explanation is a reasonable approach, but is not novel. This paper presents the approaches for node and graph classification in separate subsections, but they look very similar and can be presented together. Other parts of the proposed approach, such as motif extraction and embedding, also look trivial.
3. Many parts of the proposed approach are not clearly presented, even though they are the core parts of technical contributions. See the questions below.

Questions
1. Section 3.2: What happens if the same edge is contained in multiple motifs?
2. Section 3.2: What is the complexity and actual time for extracting motifs?
3. Section 3.3: How do we keep edges from the target node to the motif?
4. Section 3.3: How do we mask features of irrelevant nodes?
clarity,_quality,_novelty_and_reproducibility: - Clarity: This paper is generally written well, except that the technical details of the proposed approach are missing.
- Quality: The technical contributions seem to be limited.
- Novelty: This paper solves an interesting problem which is to consider domain-specific motifs in GNN explanation.
- Reproducibility: Code is not given, but sufficient information for reproducibility is presented.

summary_of_the_review: It is a reasonable approach to use known (or domain-specific) motifs to better explain the decisions of GNNs. However, I think the technical contributions of this paper are limited in two ways: a) the proposed approach does not work without pre-defined motifs, which is common in real-world cases, and b) the proposed approach seems to be a combination of existing techniques.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes MotifExplainer, a method that identifies important motifs - recurrent and statistically significant patterns in graphs - to use as explanations for a Graph Neural Network (GNN) prediction task. The method first extracts motifs in a graph using domain-specific motif extraction rules (specified in the paper). It encodes a motif embedding of the extracted motifs using a pre-trained GNN. These motif embeddings serve as keys to the graph embedding (query) in the attention mechanism applied to learn attention weights by minimizing the loss between the prediction of the attention model and the original predictions.  Experimental results on seven datasets show that MotifExplainer can highlight relevant motifs in the graphs (quantitatively and qualitatively). 

strength_and_weaknesses: Strengths:
- The presented idea is interesting and well-explained. A simpler framework - motif extraction+attention - is trained to mimic the performance of a complicated GNN model. The learned attention weights highlight the relevant extracted motif. 

- The paper provides motif extraction rules for different application domains.

- The paper presents qualitative and quantitative results on synthetic and real-world datasets and reports state-of-the-art performance. 

- The reported computational time for the proposed method is lower than existing models.
 
Weaknesses:
The main weakness of the paper is in the experimental design, as some of the choices are not well defined. 
 
- First, given the importance of the motif extractor scheme for this work, the paper needs to provide a sufficient description (example visualizations) of how much the extractions based on the rules vary from the ground truth. Specifically, the line “note that neither NH2 nor NO2 is explicitly included in our motif extraction rules” is useful. However, how similar/different are these motifs to the construction, like a single input example?  
 
- While the included baselines are state-of-the-art methods, the presented framework seems to have some ideological similarities to GraphLIME [1] method. Is there a reason this was not included in the comparisons?
 
- All the methods (proposed and baselines) have hyperparameters that they can be quite sensitive to. Was sufficient hyperparameter tuning performed to select the best explanation? For example, why K=5 was chosen? If not, is that fair?
 
- My understanding is that fidelity scores should be low (less difference in prediction change when removing unimportant features). Why are the higher scores in Table 1 (for BA datasets) highlighted in bold?  

Minor points:

- The values for K are not specified in Table 6,7,8 in the appendix
 
- (This point  is on a broader level) The line - “a fundamental criterion for explanations is that they must be human explainable.” While I agree with this statement, it is unclear what "human explainable" means outside the context of image and text. If a GNN model can make an accurate prediction using 4 edges of the house motif instead of 5 - do we necessarily want the explainer method to highlight the full motif with 5 edges? I can see the possibility of a disconnect between what the model is learning and what the human expects the model to learn. Maybe the paper could clarify this.

clarity,_quality,_novelty_and_reproducibility: The motivation, explanation, and illustration of MotifExplainer components (Figure 1) are clear.
 
The experiment setup could be improved to support the claims of the paper better
 
The paper is the first to apply the attention mechanism to explain GNNs from a motif-level perspective and one of few methods to consider subgraphs to explain GNNs.
 
The description of the model is clear, but the paper lacks hyperparameter tuning and settings

summary_of_the_review: MotifExplainer uses domain-specific motif extraction rules and attention to identify importation motifs. Changing the motif extraction rules accordingly allows for easy application of the method to various domains. The experimental design requires more extensive descriptions (especially about the extracted motifs and design choices) to support the claim that MotifExplainer is a state-of-the-art explanation method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies explainability of graph neural networks (GNNs). The authors are particularly interested in using motifs to generate the explanantion, since motifs are more meaningful and statistically significant in many graph mining tasks. Based on this motivation, the authors propose MotifExplainer, which first extracts motifs with domain knowledge and then leverages attention mechanism to measure the contribution of each motif. And the motifs with the highest attention scores are used as the proxy to find the most important motifs. Experimental results show that the proposed MotifExplainer not only outperforms all baseline methods quantitatively and qualitatively, but also is the fastest method during inference.
strength_and_weaknesses: Strengths:
- Motif is more meaningful in many graph mining tasks, thus the motivation is clear and reasonable.
- The paper is overall well-written and easy to follow.
- Both quantitative and qualitative analysis demonstrate the efficacy of the proposed model.

Concerns/Questions:
- For a node classification task, suppose that the target node is 'far away' from the motif and is not directly connected to any node in the motif. How should we keep the connectivity between the target node and the motif?
- The goal of explainability is to open the black box of deep learning models. More importantly, there was a hot debate in NLP domain on whether we should trust attention as a proxy of explanation. I assume such problem could exist in GNN as well. As such, how should we trust the explanation generated by another black-box model (attention layer here)? 
- Why should we use attention score rather than directly using the contribution of each motif to find important subgraphs (i.e., the loss $f(y, y')$)?


========================
The author have addressed most of my concerns. I have raised my score. But in general, I still think some evaluation (whether quantitative or qualitative) without using those black-box module is essential.
clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written and clear. The motivation is clear and the proposed method is novel.
summary_of_the_review: The paper is well motivated and easy to follow. My major concern on the technical side is the use of attention mechanism, which is a black box in its nature and controversial to be used as a proxy for finding explanation in NLP tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presented an explanation model on motif of the graph. The motif of choice is generated with domain knowledge. Following by an attention model, the attention weights is utilized for the explanation for the GNN from the motif. Compared with state of the art, the method showed high performance. The human readability of the motif is also a plus of the paper. However, application for this method on other domain would be limited owing to the lock of the domain knowledge.
strength_and_weaknesses: Strength:
1. This is the first method to introduce attention mechanism to explain the importance of the motif. It does showed certain level of improvement compared with previous methods.
2. The input of domain knowledge do boost the performance increase for the explanation.
3. They also introduce different scores to cover the explanation, for which their method showed good performance.  

Weakness:
1. Introducing domain knowledge generated motif would be great but also limited the applicability of this methods. It also introduced certain bias in the performance evaluation as other methods do not have the domain knowledge input. Even though this paper provided a way to generate motif. But only counting the cycle or edge, does not fit my impression of "motif" and do limited the power of the method. The ablation study also illustrate the important of the domain knowledge input. For that, I think it significantly hurt the novelty of this paper.
2. The method could also introduce some background of the attention mechanism in neural network explanation.
3. This model only performed the test on vanilla GNN with three layers. I think more experiment should be conducted. For example, different GNN architecture. Different hyperparameters like the number of layers, the width of the GNN. I would guess the attention mechanism would have high variance if the architecture is changed. 



 

 
clarity,_quality,_novelty_and_reproducibility: It is clear written. The novelty of the paper is limited by the generalization to other domains without domain knowledge. I would also recommend more experiments for the GNN of interest, i.e., hyperparameters, model architectures. 
summary_of_the_review: This paper presented the motif explanation and showed empirical performance increase over state of the art models. But some limitation also applied for the method to generalize to other domain.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes MotifExplainer, a method that identifies important motifs - recurrent and statistically significant patterns in graphs - to use as explanations for a Graph Neural Network (GNN) prediction task.  It employs an attention-based method to identify the most important motifs as explanations. It encodes a motif embedding of the extracted motifs using a pre-trained GNN. These motif embeddings serve as keys to the graph embedding (query) in the attention mechanism applied to learn attention weights by minimizing the loss between the prediction of the attention model and the original predictions. Empirical studies on both synthetic and real-world datasets demonstrate the effectiveness of the proposed method. However, application for this method on other domain would be limited owing to the lack of the domain knowledge. The reviewers raised concerns over the method’s dependence on the quality of motif extraction rules, high computational complexity, dependence on domain knowledge that limits generalizability, limited technical contribution and novelty, lacking a sufficient description/visualization of how much the extractions based on the rules vary from the ground truth. Additionally, one reviewer pointed to the need that some evaluation (whether quantitative or qualitative) without using those black-box module is essential. Another reviewer pointed to the need to experiment MotifExplainer on different GNN architectures. The authors addressed some of those comments. The AC sympathize with the reviewers on the unaddressed comments. Overall, a more comprehensive revision of the paper addressing all the mentioned issues is still needed to improve the paper and pass the bar for acceptance. 
justification_for_why_not_higher_score: The reviewers raised concerns over the method’s dependence on the quality of motif extraction rules, high computational complexity, dependence on domain knowledge that limits generalizability, limited technical contribution and novelty, lacking a sufficient description/visualization of how much the extractions based on the rules vary from the ground truth. Additionally, one reviewer pointed to the need that some evaluation (whether quantitative or qualitative) without using those black-box module is essential. Another reviewer pointed to the need to experiment MotifExplainer on different GNN architectures. The authors addressed some of those comments. The AC sympathize with the reviewers on the unaddressed comments. Overall, a more comprehensive revision of the paper addressing all the mentioned issues is still needed to improve the paper and pass the bar for acceptance.
Decision: Reject