Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors improve the graph explainability by two methods. First is introducing the contrastive loss in the embedding space. The second is a random sampling to improve the original greedy top-k selection. The experiments are well done on different benchmarks with numerical results as well as qualitative visualization.
main_review: * From Table 1, it seems the performance improvement are largely from the contrastive loss.  Recently, contrastive loss has been demonstrated to be effective in different applications, including graph learning, which makes the contribution relatively less novel. On the other hand, could you explain more how do you implement the class-wise attributor? Did you just simply train c attributors for each class? if so, is it a fair comparison that you use more parameters? 

* Although we see some improvements from Refine-FT to ReFine in Table 1. However, if I understand correctly, PGM is very close to ReFine-CT, except for the finetuning. Comparing PGM and ReFine-CT, we don't see much advantage of having finetuning. Could you comment on this?

* The description from line 167 to line 181 could be improved. How do you sample "random edges"? How do you decide "\rho"? What do you mean by decrease exponentially with t?  How sensitive are those parameters? It's known that the the greedy selection is suboptimal in many problem by discarding the dependency. Having so many degree of freedom allows us to find some sweet spot to be better than greedy. Some ablation and robustness study is necessary to make the proposed finetuning steps convincing. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a generative model to explain graph neural networks. The framework contains two steps: pre-training and fine-tuning. The first pre-training step trains edge probabilities in the adjacency matrix to capture class-wise saliency, followed by a contrastive learning by encouraging the similarity between two different adjacency matrices when they have the same class output. The second fine-tuning step employs  stochastic dropout to the edges based on the probabilities learned in the pre-training step while maximizing the mutual information between sampled subgraphs. Experiments on four datasets show consistent improvement over recent competing baselines, including SA, GNNExplainer, PGExplainer, and PGM-Explainer.
main_review: Strengths 
* The task being addressed here is of high impact on how to explain learned graph neural networks, and can potentially help the community design better graph architecture in the future when we have deeper understanding of what the graph networks learn.
* Thorough literature survey and background knowledge. I especially like Sec 4.1 where the authors give a one-sentence summary of each recent competing method. This really helps the readers get up to speed and delve deep into the core experimental results.
* Strong performance in terms of usual metrics: ACC-AUC and recall. Figure 3 and 4 are also helpful in comprehending what the proposed loss functions capture qualitatively. Sec 4.2 is also well written in a sense that it gives a detailed walk through of the experimental results.

Weaknesses
* Despite the good metric numbers in tables, I still do not fully capture why the proposed two-stage pipeline can capture the claimed global patterns and local features. It seems like the only justification in the submission is in Figure 3 column 4, where the authors claim the explainer graph does not cross the MNIST digit and background. The authors did not perform the same pre-training / fine-tuning qualitative comparison in Figure 4 on Mutagenicity dataset. Based on the presented material, it is hard to fully believe that the proposed fine-tuning step can really capture the high local accuracy.
* Based on the formulation in Eq. 9, the proposed fine-tuning for local explainability is essentially a stochastic version of edge dropout based on the learned probability matrix in the pre-training step. Therefore, it is understandable that dropout would give a reasonable amount of performance boost, especially when the training data is relatively small. This is also reflected in Table 1 that most of the performance boost is contributed by the contrastive learning. Again, the overall observation does not tell the story that pre-training captures the global patterns while fine-tuning (edge dropout) captures the local patterns.
* Why does the fine-tuning (edge dropout) works well on MNIST but not that significant on other datasets?
* Figure 3 does not speicicy which column represents what digits. It is extremely hard for readers to understand it.
* Typo: line 55: show it effectiveness -> its, line 86: one extra period in the end of the sentence.

limitations_and_societal_impact: I appreciate the authors discussing the limitation of the proposed method on the mapping between captured knowledge and the structure representation [13]. It would be great for the authors to further address the weaknesses mentioned above, especially on why the fine-tuning stage actually captures local patterns.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: 
This paper proposed a pretrain-fine tune framework for multi-grained GNN explaination. The experimental results show the proposed method to be effective.

main_review: Advantages:
1.	Experiments on multiple datasets are conducted to evaluate the method
2.	The proposed method is reasonable and easy-to-understand.
3.	The method is practical and easy for implementation.

Disadvantages:
1.	The proposed method is a combination of existing techniques. Specifically, the pre-training phase seems to be adopting PGExplainers for each class and adding a loss function to constrain the difference between classes. The fine-tuning phase is adopting simulated annealing algorithm to select subgraphs and use the subgraph to further train the model. This limits the technical contribution of this work.
2.	The authors give no theoretical analysis to their model. As the authors only made limited adjustments to existing works, even though the method is effective, more theoretical insights should be shown to provide understanding on how each part improves the performance and give inspirations to the research community. 
3.	There are typos in the paper, such as ``Figure 3.1’’ in line 98.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes an instance-level explanation method called ReFine for graph neural networks. ReFine first generates a probability map over the edges in a similar way as PGExplainer. Then, it uses contrastive loss to encourage the graph embedding of explanations to be class distinctive. Next, the usage of the selection module makes the soft selection of the edges in the explanations.
main_review: This work has a clear logic flow and strong motivations for each component in this method. 

The usage of contrastive loss is creative and full of insight. And the ablation study in Table 1 also shows the effectiveness of this loss. 

For the experiments, it provides solid results on both synthetic datasets and real-world datasets with different graph neural networks. 

Enhancement: This work only provides the experiments on the graph classification task, while I believe it can also be applied to the node classification task. However, it would be better to also conduct experiments for node classification.

The major concern is the technical contribution. Compared with PGExplainer, the only difference is the proposed loss functions. Then the technical contribution is somehow limited. 

I have a question about this work.

1. Through the experiment in Table 1, finetuning procedure shows its power. But I am still confused about the details of the fine-tuning procedure described in Section 3.2. Could you provide more details and insights about this fine-tuning procedure?

Overall, I think the idea is interesting and this paper is solid while the novelty is somehow limited. Hence, I would recommend a weak acceptance.

limitations_and_societal_impact: YES
review_title: null

Meta-review: The reviews for this paper were overall positive, commenting on the high impact, strong performance,  good motivations, and easy to implement method. After the authors helped address some of the reviewers’ concerns (particularly around novelty), all reviewers are recommending acceptance.  

We recommend to take the reviewers' comments and suggestions into account while preparing the camera ready final version of the paper.  Particularly any intuition on why edge dropout helps local explainability would be beneficial.
Decision: Accept (Poster)