Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a method to offer explanations for a multitask prediction model with a single explainer. The method provides explanations in cases where the GNN is trained in a self-supervised manner, and the resulting representations are used in future downstream tasks. 

main_review: The paper provides an interesting framework to tackle graph explanation. 

(1) Competitive baseline

Other methods, such as PGExplainer, are training multiple explainers on different downstream tasks and evaluating each explainer on different downstream tasks. Should they have more expression power to achieve an upper bond for one universal explainer for all tasks or at least can recover the results of one universal explainer?  However, existing methods showed much worse results in Fig3. I speculate the reason maybe that many aspects of the two methods are different. Therefore, it will be interesting to see under exactly the same setup of Task-Agonistic, but one trainer per task, if the proposed method doesn't lead to much information drop then it demonstrates the good performance. 

(2) Novelty
The paper is quite similar with graph pretraining to learn a useful embedding and contrastive learning with mutual information , but with additional explanation part. Can the author recap on the major novelty contribution?  

(3) Intuition of design of p 
Page 4 sec 3.2 "We introduced a masking vector p  to indicate specific dimensions of embeddings on which to maximize MI. During
explanation, we obtain the masking vector from the importance vector computed by any downstream
explainer T_down. As no downstream importance vector is available at training, we sample the masking vector p randomly". 

How can the randomly selection of p during training help with specific dimension of embedding from importance vector during testing? 


summary_of_the_review: The paper is tackling an interesting problem -- explanation of graph. However, some results are a bit counter-intuitive.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose TAGE, a task-agnostic explanation method for explaining GNNs. TAGE explains GNN embedding models without downstream tasks and allows the explanation of multi-task models. This paper maximizes the mutual information of masked graph embedding and masked subgraph embedding as the objective function. The regularization term is added to restrict the number of edges. Empirical results focus on three aspects: (1) Improvement in fidelity scores brought by TAGE; (2) Visualizations on explanations to the GNN model; and (3) Comparison of computational time cost among GNN explainers.

main_review: Strengths:

1. This paper considered an interesting setting where there is no downstream task, or the target model to be explained is a multi-task model. Specifically, the entire pipeline consists of two explainers: an embedding explainer and a downstream explainer, which can be trained in isolation. The proposed method uses the downstream explainer to get dimension importance and highlight the important dimensions. 

2. The presentation of the paper is in general clear.

3: It is a novel setting that is different from existing ones.


Weaknesses:

1. The technical contribution of this paper is limited. The proposed method is somewhat incremental compared with the existing method PGExplainer. Many design details are directly borrowed from PGExplainer. For example, the design of the embedding explainer and the form of the regularization term are the same.

2. There is very little intuition provided to justify the practical significance of the explanation provided by the embedding explainer if there is no downstream task. I would suggest the authors include some examples of the generated explanations, which is essential to determine whether the claimed fidelity brings meaningful results in practice.

3. Some key baselines are missing. For example, Gem [1] is also a learning-based explainer published in ICML2021. However, the author did not discuss and compare theirs with this work. I'd like to see the comparisons between the proposed method and [1].

[1] Wanyu Lin, Hao Lan, Baochun Li. Generative Causal Explanations for Graph Neural Networks. ICML 2021.


4. The proofs of Equations 2 and 3 are insufficient. Besides, in Equation 3, the definition of the positive and negative samples is unclear. The sample, positive sample, and negative sample in conditional contrastive loss should be specified for graph-level or node-level tasks. 


5. Regarding the method itself, the target model of the embedding explainer seems limited to node-level embedding models instead of graph/node-level embedding models. Specifically, graph embeddings are required when training the embedding explainer. When it comes to the implementation details of embedding explainers, node embeddings are required. Although node embeddings can be simply compressed to graph embeddings, the reversion is difficult. Therefore, an embedding explainer targeting a graph-level embedding model will fail to predict node-pair importance scores.

6. "For all methods, we vary the threshold for selecting important nodes or edges and compare how fidelity scores change over sparsity scores on each task and dataset.''

How to set the threshold is not clear. 

7. The empirical evidence could be more substantial.

-	In essence, an idea explainer should be human-understandable. However, by looking at the Visualization in Figure 4, model explainability is not clear. Visualizing a few instances is not convincing in terms of model explainability. Further analyses are expected. 


-   About the evaluation of universal explanation ability, experiments results with downstream tasks are missing, which are indispensable for demonstrating the effectiveness of TAGE.

-   On the experimental design, some scenarios do not satisfy the multi-task setting defined in general. For example, predicting the 121 binary labels for nodes in the PPI dataset is a multi-label classification task and cannot be regarded as 121 independent tasks. It is unfair to compare with the baseline PGExplainers trained and eval on this problematic experimental setting. Therefore, the results in Table 2 do not justify the use of TAGE.

-   In Figure 3, on Task0 of the PPI data set, the fidelity scores of the proposed method are much lower than those of DeepLIFT. I would like to see a reasonable elaboration on this underperformance.


Minor points:

In figure 2, the expression of $G_s$ is wrong. It should be written as $\tau_\theta(p, G)$, the \eps operation should be done after using the GNN model.

In Section 3, $z_i$ (or $z_j$) denotes a graph embedding and a node embedding successively, which is ambiguous. If the author could distinguish between the notations of two different embeddings, this section would be more readable.

summary_of_the_review: The technical contribution of this paper is limited. The proposed method is somewhat incremental compared with the existing method PGExplainer.  The paper lacks rigorous empirical analysis, especially in terms of studying the interpretability aspect.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is motivated by the fact that existing task-specific explainers are too expensive to be applied to generating explanations for a model trained for multi-tasks. They decompose the typical end-to-end learning-based GNN explainer into two parts: the embedding explainer $\mathcal{T}_{\mathcal{E}}$ and the downstream explainer $\mathcal{T}_d$ ($d$ for down). 
The downstream explainer maps embeddings into importance scores $m$ for all embedding dimensions given a trained model and certain inputs. The embedding explainer is associated with the embedding model $\mathcal{E}$, which maps a given graph into a subgraph of high importance, conditioned on the embedding dimension importance.

Finally, $\mathcal{T}_{\mathcal{E}}$ is trained via a self-supervised manner, and they use a gradient explainer for $\mathcal{T}_d$. In the experiments, TAGE outperforms the SOTA explainers w.r.t fidelity, sparsity, and especially time cost.

main_review: **Strengths**
1.	The motivation is technically a good angle. Indeed extra costly training for generating explanations is a bottleneck for the applications. 
2.	The paper is easy to understand.
3.	The method is simple but effective. 

**Weakness**
1.	This paper mainly focuses on explaining multi-task models, which somehow limits the applicability.
2.	Why does the author use $\boldsymbol{p}\otimes \mathcal{E}(\mathcal{T}_\theta(\boldsymbol{p}, G))$ but $p\otimes\mathcal{E}(\mathcal{T}_\theta(\boldsymbol{p},G))$. If there is a performance gap between this two formulations, I wonder how each of them affect the quality of generated explanations.
3.	During the self-training of the embedding model, $p$ is sampled from a multivariate Laplace distribution, while later, the input is the conditional embeddings generated by the gradient. The distributions of the two groups of inputs could be different numerically and thus may affect the specific performance of the embedding model. Can the authors comment on this a bit? 
4.	Some typos: last row of page 5 “as input” should be “an input”; In Section 4.1, “include four graph classiﬁcation tasks” should be “include three graph classiﬁcation tasks”.

summary_of_the_review: I think this work is overall good, however, some technical details need to be clarified and the proposed model can only be necessarily applied to a limited range of explainers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this manuscript, a new explainer for GNNs is proposed. The newly proposed method aims to provide a task-agnostic explanation for the embedding GNNs rather than a specific downstream task. The motivation of the proposed method is that the modern GNNs are typically trained in a two-stage manner, where the embedding GNN is trained in the first stage and then the task-specific lightweight MLP. The proposed method is trained based on the MI between the whole graph and sub-graph generated by the explainer.
main_review: The proposed scenario where we typically need to use two-stage GNNs is interesting and makes sense.

The proposed method is simple and somehow can achieve the desired goal.

There are several concerns.

1, The setup of the explainer in this manuscript is a little different from the traditional ones. In the field of DNN (CNN or GNN) explanation, we may want to make the explanation only rely on the input sample and the model, which is most of the previous works have done. However, the proposed explainer needs one to train on the whole dataset used for the model to be explained. This may lead to several problems. 1) the explainer itself will rely on the training data, which may bring the same dataset bias to the explanation. 2) the proposed explainer, under its setup, is more like an input distillation model, where the input graph will be shrunk during the inference time.

2, The proposed method seems to be a general method for the DNN models, not limited to the GNN. I'm wondering if the authors have tried to apply the proposed method to the CNN or transformer?

3, For the evaluation, it would be interesting to apply the proposed method on the generated synthetic datasets like that used in GNNExplainer, where the ground truth explanations are available.
summary_of_the_review: The main concerns are about the setup of the explainer and the evaluation.
review_title: null

Meta-review: Summary of the paper and the reviews:
The authors propose a method to explain the GNN predictions in a task-agnostic setting, meaning that the method can be applied to a new downstream task without fine-tuning. The task is formulated as to predict the important subgraph given the input graph and the ground truth label. The learning algorithm optimizes the mutual information between the embedding of the subgraph and the original graph. The experiment shows the quantitative improvement measured by the fidelity score, the qualitative visualization of highlighted subgraphs and comparison of the cost w.r.t the baseline GNN explainer models. 
Strength: 
1) The task-agnostic setting is novel.
2) The proposed method shows improvement in the fidelity score with a reduced training cost over the baseline models
Weakness: 
1) The proposed objective requires additional justification. To optimize the intractable mutual information objective, the authors propose JSE and InfoNCE as upper bound estimations of mutual information, but the negative sampling technique in the proposed method is not fully justified.
2) During training, the authors simulate the task-specific importance vector by sampling masks from a Laplace distribution. During testing, the importance vector is obtained by gradient-based approach. Further analysis is needed to quantify the effect of this training/testing discrepancy.  
3) In the empirical experiment, the proposed task-agnostic model outperforms the task-specific baselines. Why should such an outcome happen? The reason needs additional analysis but is not provided in the current paper. Moreover, the qualitative results of a few examples are not sufficiently convincing for the reported empirical success.

Summary of the discussions and the decision by reviewers: One reviewer asked for a justification of the negative sampling approaches used to approximate the mutual information objective. While the authors described their implementation design in their rebuttal, the theoretical justification of the method was not enough. Two reviewers raised the question about how randomly sampling importance masks during training could affect the downstream tasks performance, which was not fully addressed in the rebuttal. Other than that, the experimental concerns about new baselines and datasets were well addressed by the authors.

Recommendation: The paper has received borderline review scores (5, 5, 5, 6). Although the authors addressed some of the concerns in their rebuttal on the experimental design and added important baselines, more convincing justifications/analysis for the proposed method are still missing. Therefore, the reviewers didn’t raise their scores. Based on the above concerns, the recommendation is to reject.
Decision: Reject