Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes MEGAN, a multi-explanation graph attention network model that leverages attention mechanism to produce node and edge attribute explanations along multiple channels for graph classification and prediction tasks. The proposed approach is explanation co-training: along with the prediction task, and explanation task is trained where, based on node importances, an importance value is generated per channel. Channels are defined based on the task: for regression, the explanation task is whether the the importance value is greater than y_channel, or less than y_channel, whereas for classification, the number of channels is the number of possible output classes C. 

The models are compared with GNNExplainer model over datasets RbMotifs, Solubility and MovieReviews datasets. The metric used is Area under ROC curve (AUROC), which measures similarity to ground truth explanations. Another metric used is fidelity metric, which looks at faithfulness to predicted output. Experiments are also run on synthetic datasets. MEGAN reports better R^2, node and EDGE AUC and Fidelity metrics and lower MSE.
strength_and_weaknesses: Strengths:
The paper is well written and the problem is well motivated. 
The proposed architecture is outlined is good detail. 
Experiments are run on real world and synthetic datasets.

Weaknesses:
The notions of channels for explanations and how explanations are defined needs to be described in greater detail.
Experiments report performance at various percentiles of metrics, but statistical significance is missing.
Proposed approach is compared only with a single baseline approach, need more approaches to be covered in prior work and compared with. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written, easy to read. The proposed architecture sues co-training which is novel but overall appears to be incremental in contribution. Hyperparameters for training procedure and architecture details for each of the experiments are needed in more detail for reproducibility. 
summary_of_the_review: While the problem is well motivated and the paper is well written, with experiments on several real and synthetic datasets, the notion of channels for explanations seems rather limited and needs to be better described. The performance improvements over metrics for explanations needs statistical significance to be reported.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: 
This paper introduces  a multi-explanation graph attention network architecture by useing attention mechanisms to produce node and edge attribution explanations along multiple channels for graph classification and regression tasks. Experiments demonstrate the effectiveness of the proposed model. 

strength_and_weaknesses: Strength:
1. The research problem is very important and this paper provides different aspects to provide explanations in GNNs. 
2. This paper is easy to follow and clearly written.  


Weaknesses

There are some concerns regarding this paper.


1. The novelty of this paper is limited. The proposed framework of multi-explanation graph attention network is straightforward by using attention mechanisms to produce node and edge attribution explanations along multiple channels for graph classification and regression tasks. 
It would be better if they detail more their novelty. Also, it's unclear the contributions of this work.

2. This work misses some advanced related works and baselines in their experiments. This work mainly compared with GNNExplainers, and this work can be more solid if they can compare some advanced GNN explanations methods.


3. Evaluation in GNNs explanations are not convincing enough.  Maybe more real-world datasets and evaluation metrics can enhance the effectiveness of the proposed method. 

clarity,_quality,_novelty_and_reproducibility: The novelty and contributions of this work is limited. 
summary_of_the_review: Strength:
1. The research problem is very important and this paper provides different aspects to provide explanations in GNNs. 
2. This paper is easy to follow and clearly written.  


Weaknesses

There are some concerns regarding this paper.


1. The novelty of this paper is limited. The proposed framework of multi-explanation graph attention network is straightforward by using attention mechanisms to produce node and edge attribution explanations along multiple channels for graph classification and regression tasks. 
It would be better if they detail more their novelty. Also, it's unclear the contributions of this work.

2. This work misses some advanced related works and baselines in their experiments. This work mainly compared with GNNExplainers, and this work can be more solid if they can compare some advanced GNN explanations methods.


3. Evaluation in GNNs explanations are not convincing enough.  Maybe more real-world datasets and evaluation metrics can enhance the effectiveness of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new explainable graph attention network model. The highlight of this model is its multi-channel explanability. Along with outputing node predictions, the model also generates extra edges and nodes importance scores across preset channels. To along these importance scores with multi-explanation channels, the authors introduce a new way of training the model, named explanation co-training. It adds an explanation step, where the model will be penalized when the importance scores are not aligned with the ground truth label. The authors evaludated their model in one synthetic dataset and two real-world datasets and showed the proposed model gets great empirical results as well as reasonable explanations.
strength_and_weaknesses: Strength:
1. The paper is well written and well organized. The motivation of multi-explanation is explained clearly. It's not hard to follow how the authors designes the model to have multiple explanability channel in the methodology part. 
2. The idea of multi-explanation is novel and fits the graph idea well. The authors did a great job on analyzing potential issues of multi-channel  explanations and proposing a novel solution (co-training) to address some of the issues. In section 5, the authors thoroughly discussed the limitations of the proposed model.
3. The expermental results look promising. The authors showed that the explanations generated by the propose model were reasonable and aligned with the ground truth. It also outperforms the baseline model on many aspects.
4. The authors released their codes anonymously and included detailed comments and docs about codes itself and how to re-run the experiments.

Questions:
1. If you train the model twice with same datasets but different random seeds, do the same explanation channels (by index) in each run align with each other? I think this is an important aspect to discuss when you have multiple explanation channels. If it's stable, the same channel (by index) should refer to the same underlying latent space. If it's not stable, is there a way we can make it stable, or we can somehow find a mapping between new channels and old channels? For example, someone uses this model and spends quite a lot of resources to figure out what each explanation channel roughly means, then they get some new graphs data and need to re-train the model, how can they make use of the understandings of old explanation channels? Would like to see some discussions on this.
2. In the movie review datasets, the authors mentioned "our model’s accuracy is consistent with that achieved by other GNNS
as reported by ...". It would be nice to include the metrics somewhere. Maybe I missed it, but I couldn't find them in the main body or the appendix.
3. Would it be possible to run MEGAN over non graph datasets? Namely just graphs with zero edges. Just curious how MEGAN would perform comparing with those non-graph based explanability models.
clarity,_quality,_novelty_and_reproducibility: I think the paper explains its idea clearly and the model, expermental results, and analysis are well presented. The contributioin of this paper is clear and novel. Though I didn't run the release codes, it makes me confident about it's reproducibility.
summary_of_the_review: I think this paper is well organized and the idea will be beneficial to other researchers in this field.



---- update after seeing other reviewers' comments and authors' response
Overall I still think this is a good paper. Other reviewers called out that comparisons with many other GNN explanable models are missing and the experimental sections could've been more thorough. I think their points made sense, so I'm lowering my score to marginal accept for now.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes the multi-explanation graph attention network (MEGAN), which is an attention-based self-explaining model for graph regression and classification. MEGAN features multiple explanation channels independent of the task specifications. The edge explanations are given as the edge importance tensor, which is calculated from the concatenation of attention logit tensors. The node importance tensor, which represents node explanations, is then given as the product of node importance embeddings and the pooled edge importance tensor. The paper first evaluates MEGAN on a synthetic graph regression dataset, and further demonstrates the advantages of multi-channel explanations on two real-world datasets: the prediction of water solubility of molecular graphs and sentiment classification of movie reviews. The authors claim that MEGAN produces explanations consistent with human intuition.
strength_and_weaknesses: - Strength
    - The paper proposes an attention-based self-explaining model for graph regression and classification, which features multiple explanation channels independent of the task specifications. 
    - The paper is well organized in illustrating the overall framework and the process of how to generate multi-channel explanations. It is easy to follow the technical details.
- Weaknesses
    - The only baseline in the experiments is GNNExplainer, which was published 3 years ago, and there are quite a few new models offering better performance on explaining graph-based prediction tasks. More importantly, GNNExplainer is a post-hoc method for explaining blackbox GNN models, but MEGAN is a self-explaining models including both prediction and explanation modules. The baseline is too weak, and it is not fair to compare a post-hoc method with a self-explaining method. The paper should compare MEGAN with some more appropriate baselines, which are also self-explaining methods, such as (Gao, et al., 2021), (Zhang, et al., 2022) and (Magister et al., 2022).
        - (Gao, et al., 2021) GNES: Learning to Explain Graph Neural Networks
        - (Zhang, et al., 2022) ProtGNN: Towards Self-Explaining Graph Neural Networks
        - (Magister et al., 2022) Encoding Concepts in Graph Neural Networks
    - For evaluating the interpretability of MEGAN, quantitative analysis is only conducted on the synthetic data set. More experiments should be done on real-world datasets as well. 
    - BAShapes and TreeCycles are synthetic datasets most widely used as benchmark of GNN explanation tasks. However, this paper only uses the RbMotifs dataset created by the authors themselves in the experiments. For fairness of experiments, the authors need to give an explanation for this issue as well.
- Question
    - For the explanation co-training, the reference value y_c seems to be unspecified. How do we define this value in training as it seems to affect the model's effectiveness.

clarity,_quality,_novelty_and_reproducibility: - Clarity of the paper is good, and it is easy for readers to follow the content.
- In addition to illustrating what MEGAN is doing in each step, it will be helpful if the paper can offer more convincing insights on why MEGAN is designed this way.
- Probably there will be better reproducibility when the source codes are available.

summary_of_the_review: The framework of MEGAN is well illustrated in this paper, but in the experiments the baseline is too weak and not appropriate.  It is not fair to compare a post-hoc method (GNNExplainer) with a self-explaining method (MEGAN).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposes  an attention-based self-explaining model for graph regression and classification. The main idea is to multiple explanation channels to produce explanations for graph prediction tasks. The work is specifically motivated by the need for explanations of different polarities for regression problem, hence the need for multiple channels accounting for different polarities. 
Experiments are reported on synthetic and real datasets and the proposed method appears to show good performance.
Strength:
Well written paper on an important topic.
experiments on both synthetic and real data produce promising results
Weakness:
The main idea of using multi-channel attention for explanation is straight-forward with limited technical novelty.
Limited experimental results. Authors have included additional baselines based on reviewers comment. That definitely improved the qualify of the evaluation. But the experiments on real data is very cursory. The explanation evaluation is qualitative and anecdotal. The performance comparison was only detailed in the appendix.  As evaluation on real data is a very critical part, the authors should do the due diligence to have a cohesive and complete treatment in the main body of the paper and only defer less important part to the appendix. 


justification_for_why_not_higher_score: Limited technical novelty and the evaluation of explanation on real data is qualitative and anecdotal.
Decision: Reject