Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors describe a limitation of the GAT, claiming that the ranking of the attention scores is unconditional on the query node. The paper proposes a method named dynamic graph attention by modifying the order of operations. 

main_review: The explanations and justifications of the proposed methods can be improved. Why is it a problem that, for any query node, the attention function is monotonic with respect to the neighbor (key) scores? As currently presented, the dynamic attention defined in Definition 3.2 seems a convenient choice. Why is it necessary that every key can be selected using a query? Also, if a key is important, why does any query want to ignore this key or decay the key’s score. I am really doubting the method, but just feel the explanations and justifications can be clearer.

The simple fix captured by Equation 7 was claimed to be able to compute dynamic attention as defined in the paper. Is there a deeper reason for changing the order of local operations? Why LeakyReLU first is better?

Some results show that GATv2 seems to outperform GAT on a set of benchmark datasets, but are mixed in other datasets. In node prediction task in Section 4.4, GATv2 sometimes does better with 1 attention head, and other times better with 8 attention heads. Given it is able to compute dynamic attention, shouldn’t more attention heads be better? In link prediction in Section 4.6, no-attention-based methods outperform attention-based methods. Why is attention not needed in these datasets? Also, why do you think some datasets require dynamic attention? Gaining deeper insights to these questions will be really helpful. 



summary_of_the_review: The proposed method has potential, but the methods needs better explanations and justifications; and the results seems limited and mixed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes the limitation of GAT by pointing out that GAT computes a limited kind of attention: static attention.
This paper then introduces a simple fix by modifying the order of operations and proposes GATv2: a dynamic attention variant.
Experiments show that GATv2 can outperform GAT.
main_review: The experiments show some improvement, but the theory has flaws:

The theory assumes that there is a fixed set of keys shared by all queries,
but this assumption cannot fit GAT. 
In GAT, the softmax normalization is applied to each node's adjacent neighbors,
which means each node will only attend to its adjacent neighbors.

The example in Figure 1 and Section 4.1 is a **complete** bipartite graph.
In this special case, queries have the same set of keys, and the theory works,
but the special example is far from the real-world datasets. 
How do you define static and dynamic attention when different nodes have different key sets?
summary_of_the_review: The empirical improvement cannot be properly established on static and dynamic attention
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author mainly redesign the Graph Attention Network, and they show that the new model can capture the dynamic attention instead of static attention in the original GAT.
main_review: Strengths:

1. a new type of GAT model, that can have very borader impact.

2. The design is super simple as shown in Eq.(6) and Eq(7).

3. They also show some mathmatical analysis, and the results is good.

Weakness:

1. The theorem is not clear for me, J_max leads to maximal values of its attentions distribution. Why says that the \alpha compute only static attention?

2. Any example for he continues function \withhat{g} in Eq.(9)?
summary_of_the_review: In summary, I think the algorithm is this paper is simple yet effective, just swap some of operators in the orignal graph attention network. The impacts are broad. The differences of the dynamic attention and static attention can be further discussed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes GATv2, the improved variant of the graph attention network (GAT). To demonstrate that GATv2 is more expressive than the original GAT, the authors (1) give theoretical justifications with the notion of static/dynamic attention, (2) conduct experiments on the synthetic dataset which GAT cannot learn, but GATv2 can, (3) demonstrate that GATv2 outperforms GAT on various real-world node-, link-, and graph-level prediction benchmarks.

main_review: The novelty of the proposed model is very simple; the authors just switch the order of two operations (multiplication with attention weights and the activation). However, I recommend accepting this paper since the implications from its theoretical and empirical results can be significant to the graph neural network community.

Many recent papers are designing variants of attention and applying them for various domains, but no paper clearly answers why the attention looks like what they proposed and why it works. We cannot say this paper is totally free from this criticism, but it can be the first step to finding the optimal attention architecture. I believe that future research of designing attentions can leverage the principle and practices in this work.

*Questions:* First, the authors state that DPGAT (in Transformer) is strictly weaker than GATv2. Then, can we get a better version of the Transformer if we replace its attention with GATv2’s? Here, I am not demanding the full experiments on GATv2-Transformer on NLP tasks, just asking what GATv2 authors think about this. Second, what properties of graphs characterize the simplicity of the task (or the necessity of different rankings of each node)? That is, what properties of datasets cause the different performance gap between GATv2 and GAT by datasets (besides an average degree in link prediction)? For example, the proposed DictionaryLookup task seems to be simple but requires using different rankings. GATv2 outperforms GAT on ogbn-arxiv with a small margin (<0.3%p), but on ogbn-proteins with a relatively large margin (nearly 6.0%p on one head). Third, the classical citation benchmarks are considered to be easy-to-overfit, but we know GAT does not outperform baselines on PubMed. How does GATv2 perform on the PubMed dataset with the public split? 

*Suggestions:* It would be nice if the authors report the statistical significance test results on the tables. 

summary_of_the_review: The proposed GATv2 model is simple but more expressive than the original GAT. The authors’ claim is justified both theoretically and empirically. I would recommend the acceptance since the principles and practices presented by this paper can be useful for future research to design attentional models.
review_title: null

Meta-review: This paper argues that the widely adopted graph attention networks (GAT) have a shortcoming that with the static nature of the attention mechanism, they may fail to represent certain graphs. This paper presents an alternative, GATv2, a simple variant with the same time complexity as GAT but with more expressivity, able to represent the graphs that GAT fails to. This is shown both empirically and theoretically, with various tasks on synthetic as well as standard benchmark graphs. 

GATs are of high interest to the ICLR community, and this paper makes fundamental progress in how attention works in GNNs. This is one of the few papers that present both empirical and theoretical analyses, and these findings will motivate others in the community to make further advances in this field.
Decision: Accept (Poster)