Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a framework for learning attention weights on the edges of the graph through a graph attention framework. The learned attention is causally-guided, which means that it aims at aligning the attention mechanism with the causal effects of interventions on the graph connectivity. The causal effect of an edge is measured through the impact it has in the final task, thus it is integrated in a loss function of the training phase. The authors show that the proposed framework improves the performance on node classification tasks in standard datasets. Moreover, the learned attention seems to be more interpretable.
strength_and_weaknesses: Strength: The proposed idea is relatively intuitive and valid. The paper is generally easy to follow. 

Weaknesses: Parts of the paper (e.g., definition of causality, discussion of experimental results) are superficially treated (please refer to the detailed comments below)
clarity,_quality,_novelty_and_reproducibility: Parts of the paper that are related to the algorithm should be more clearly written. The approach is somehow novel but it should be better and more precisely stated and motivated. The experiments should be easily reproducible is the authors eventually make the code available. 
summary_of_the_review: Below are a few comments that should be addressed by the authors, and could hopefully improve the quality:

1) The causal attention regularization scheme is not clear. A summary of the algorithm should be provided, or a schematic overview that would help the reader understand how Eq (5) is integrated into (3). For example, $\sigma$ in Eq 5 is not defined. Please make the algorithm as accessible as possible to readers.

2) Is there a reason why the regularisation is applied only to attention networks? Could it be applied to any other architecture as well by adding a regularisation term that represents the auxiliary supervised prediction task that aim to align the weights of the networks with the causal effect of removing specific edges or nodes on the graph?

3) The definition of causality is vague as it captures more correlation of a specific edge with the outcome than causality. Can you comment/clarify that part?

4) How do you define R? It seems that it is an important parameter of your algorithm, and one of the reasons why it does not scale on real graphs. 

5) While the method is tested in different datasets, the results are not sufficiently described. Can you comment on the performance of homophily/heterophily datasets? Is there any specific pattern? Does this improved attention scheme help in one or another direction?

6) The neighbor voting regularization scheme is not clear. Please make it precise and clear. 

7) The graph pruning approach needs also more details. Why is that method more interesting than classical pruning approaches? 

8) While the intuition of the paper is reasonable, there is no clear conclusion on the usefulness of the method. The authors touch upon different aspects such as 1) generalisation 2) interpretability 3) graph pruning, but the results are not strong enough in any of these aspects. The postdoc analysis of the interpretability weight on the ogbn-arxiv is also relatively vague, as some of the statements could be subjective. 

9) The authors should include F1 scores in the experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a regularization framework for training graph attention networks. The proposed method is based on the idea of performing edge interventions to determine the relative importance of an edge to specific prediction tasks. Extensive experiments show that the new regularization framework helps improve generalization while at the same time makes the attention coefficients more interpretable.
strength_and_weaknesses: Pros: 
- The paper is well structured, well motivated and nicely written. The presentation is clear, the idea is interesting, and the results are easy to follow.

Cons/Questions:
- The test accuracies reported in Table 11 in Appendix A.6 are much lower than what are typically achieved by GNNs. For example, the test accuracy for Cora is usually higher than 0.8 for most existing GNN methods with 1-2 layers, but in Table 11 the test accuracy for GAT is only 0.58. Could the authors comment on what differences had caused such a large gap between existing results and the results reported in this work?
- For semi-supervised node classification task, usually only a small fraction (e.g. 5% or less) of nodes are labelled. In this case, does it mean that the regularization loss in Equation 3 is computed only on the training nodes, and consequently only the attention coefficients for the training nodes are regularized? If this was true, then the regularization would have no effect on the test nodes. I think I must have misunderstood something, could the authors help clarify the effect of regularization on attention coefficients for nodes whose labels are unknown?
clarity,_quality,_novelty_and_reproducibility: Clarity is good. The presentation is clear and the concepts are easy to follow.

Quality is good. The experiments are extensive and well designed.

The idea to regularize attention coefficients based on edge interventions is interesting, but not very surprising, given prior work that study similar problems. The proposed regularization framework is incremental.
summary_of_the_review: The paper proposes an interesting regularization framework to improve the generalizability and interpretability of graph attention networks. The idea is interesting and the paper is well written. However the overall method development is incremental and I still have some questions regarding the empirical results. I would increase the score if the authors properly address those questions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The Paper argues that existing studies to improve the generalizability of attention mechanism in GNN are independent of the actual prediction without regard for the training objective. Thus the paper leverages active interventions on nodes’ neighborhoods by deleting some specific edges to align graph attention training with the causal impact of these interventions on task performance. Experiments on three graph attention architectures across an eight-node classification task show improvements.
strength_and_weaknesses: Strengths:
1. The paper conducts abundant experiments on a broad range of datasets, which shows consistent performance to demonstrate the effectiveness of CAR.
2. Qualitative analyses are provided to suggest that the attention-weight changes produced by CAR are intuitive and interpretable.  

Weaknesses:
1. The paper shares a lot of common idea with the work “Causal Attention for Interpretable and Generalizable Graph Classification, KDD22”, but lack discussion and comparison with it.
2. The paper does not provide an analysis and discussion of the causal effect score, which is a very important component of the work. Is it theoretical solid in its current form? And experiments are also required to prove its effectiveness compared to other estimations of the causal effect.
3. Equation 5, p_ij^{n} can be greater than 1 in practice, since removing some noisy edges can also improve the performance. And d(n) >1 will make the value C greater when d(n) increases. However, the author claims that “per-edge attention weights are expected to be smaller for higher-degree nodes”.

clarity,_quality,_novelty_and_reproducibility: Clarity: Good

Quality: Fair

Novelty: Limited

Reproducibility: Good
summary_of_the_review: The proposed method is effective and easy to follow. However, the novelty needs to justify based on related works. And an analysis and discussion of the causal effect score is also required.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper claims that existing attention-based GNNs are vulnerable to spurious correlations, thereby hampering the generalizability of the model. To improve the generalization ability of the attention-based GNNs, the authors propose a causal inference approach. Specifically, they use a causally-inspired regularization scheme (CAR).  It can be applied to any attention-based GNN architecture. They conduct experiments on node classification tasks, and find the accuracy improvements and loss reductions.
strength_and_weaknesses: Strength:
1.	They conduct extensive experiments to demonstrate the effectiveness of their work.
2.	The issue of spurious correlation in GNNs is significant.

Weaknesses:
1. The authors claim that they can solve the problem of spurious correlations. But where is the spurious correlation in their tasks? The authors do not give any detailed descriptions or examples, nor do they show examples of failures of mainstream approaches.
2. Does the example in Figure 1 exist? The authors use the example of Figure 1 to introduce confounders, but it is doubtful whether there is a corresponding phenomenon in the real-world dataset.
3. The authors claim that they can improve ANY attention-based GNNs, but only GAT-like networks are introduced in the experiments. But there exist numerous attention-based methods for GNNs [1-5]. It is questionable then whether it will work on other types of attention mechanisms.
4. For equation (5), the authors only give the regularization term they propose, but do not explain it from a theoretical and causal point of view. For equation (4), the author does not give a detailed definition of $L^p$.
5. How the edge intervention process is implemented, the author does not give a detailed algorithm process or model architecture diagram to explain in detail.
6. For experiments, the author reproduces the statement that they use a public data splitting method. However, compared to the public results, the performance of node classification is questionable, for example, the performance of GAT on Cora is probably around 80-83, based on mainstream packages, such as Pyg or DGL.
7. The authors claim that their method can improve generalization by overcoming spurious correlations, but lack many baseline methods, such as [6].

[1] Gated Graph Sequence Neural Networks, ICLR 2016
[2] How to find your friendly neighborhood: Graph attention design with self-supervision, ICLR 2020
[3] Self-attention graph pooling, ICML 2019
[4] Understanding Attention and Generalization in Graph Neural Networks, NeurIPS 2019
[5] Causal Attention for Interpretable and Generalizable Graph Classification, KDD 2022
[6] Handling Distribution Shifts on Graphs: An Invariance Perspective, ICLR 2022

clarity,_quality,_novelty_and_reproducibility: The description of the method is confusing. It is difficult to understand how to implement the method from the current manuscript. 
Theoretical proofs and methodological explanations are missing.

summary_of_the_review: The authors claim that they can solve the spurious correlation problem, but give no examples of their task or failures of existing methods. The description and theoretical justification of the method in this paper are insufficient.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) This paper introduces a regularization scheme for training graph attention networks that aligns graph attention with the causal impact of interventions on graph edges. With extensive experiments on eight node classification tasks and three graph attention architectures,  the proposed regularization scheme is shown to help improve model generalization and interpretability compared to without the regularization.
(b) The proposed idea is intuitive and improves performance when applied to several graph attention models. 
(c) As noted by several reviewers, the main idea of this work is somewhat incremental, limited in novelty. The main evaluation compares training with and without the proposed regularization term. While this does show improved performance, it is not clear whether the approach will actually improve upon the state-of-the-art as the reported results are over large range of (suboptimal) parameter settings.
justification_for_why_not_higher_score: Limited novelty and inconclusive as to whether it will provide benefits over the current state of the art.
Decision: Reject