Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper carries out several kinds of analysis on the GAT networks of Velickovic (2018), which augment GNN updates with multihead self attention. Three standard attention types are compared, on several different datasets, and differences between uniform attention and learned attention are reported. An experiment is carried out where low-attention edges are pruned.

While understanding the value of attention is important, this paper leaves many questions open. First, since the graphs studied in this paper are, if not generally sparse to begin with at least they only include connections that are meaningful, the sparsification experiment is a bit hard to understand. One particular extension would improve things: adding random edges (can the model learn to prune them out?), but learning sparse attention (see e.g., Maruf et al., 2019) rather than thresholding seems to be a reasonable point of comparison.

Overall this paper would be more valuable if a clear and concise recommendation could be given regarding how to use or understand attention; but the lack of a consistent pattern of results makes any obvious narrative hard to support. I would encourage the authors to continue this line of work so that it can be used to provided guidance to those who would like to make more effective use of GNNs.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper analyzes attention in graph neural networks. It makes two major claims:
(1) Datasets have a strong influence on the effects of attention. The attention in citation networks are more uniform, but they behave differently in protein or molecule graphs. 
(2) With attention-based graph sparsification, it is possible to remove a large portion of edges while maintaining the performance.

I have some concerns about this paper: (1) the analysis lacks theoretical insights and does not seem to be very useful in practice; (2) the proposed method for graph sparsification lacks novelty and the experiments are not thorough to validate its usefulness; (3) the writing of this paper is messy, missing many details.

In the analysis part (section 5), the choices of probing metrics seem arbitrary and lack theoretical insights. The authors used the L1 norm, but it seems not appropriate for the tasks here, e.g. KL divergence is preferred to measure the distributional discrepancy, entropy for concentration etc. Many important details are missing or not clear, for example, in Table 2, which head/layer is used for computing the attention, and what does “GCN vs learned” mean? The maximum pairwise difference is not clearly defined. The meta graph classification (section 5.2) only considers a synthetic dataset. Overall, I feel the analysis didn’t present too many interesting observations, and I cannot see too much potential value in applications (even for the graph sparsification task in this paper, its correlation with the analysis is quite weak).

In section 6, it explores whether it is possible to remove part of the edges from the graph while maintaining the classification performance. It is an interesting task, but the method proposed in this paper is not realistic and lacks novelty. In 6.1 and 6.2, it needs to train a GAT first to get the attention scores, then remove edges according to attention scores and train another GAT. In this way, it doesn’t reduce the computational requirement, as it still trains a full model to get the attention. Only in 6.3 it presents a realistic setting, where the attention scores are derived from a small GAT, and train another GNN on the sparsified graph. But the paper didn’t explain why it is possible to get reliable attention scores with a small GAT, and the experiment is only on one dataset. Does it apply to other datasets (citation network, molecules) and settings (transductive, inductive)? So far the experiments are not enough to be considered as a valid contribution.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents an empirical study of the attention mechanism in the graph attention networks (GAT).  The study reveals that the attention patterns largely depend on the dataset, on some datasets they are sharp, but on others the attention patterns are almost uniform and not so different from the uniform aggregation weights in GNNs that does not have attention.  The authors further tried to utilize these findings and attempted to do attention-based graph sparsification, and showed that they can get a similar level of performance with only a fraction of the edges in the original graph if they do the sparsification based on the attention weights.

Given the popularity of the GAT model in the graph neural networks literature and the effectiveness of the attention mechanism in many deep learning architectures, the empirical study presented in this paper focused on attention is valuable.  The experiments are clearly motivated and executed, which I appreciate.

As this is an empirical paper, one (maybe) problem with it is that the findings presented aren’t that surprising in hindsight - of course the attention patterns should be data dependent, and doing attention-based graph sparsification seems like an obvious thing that should work.  The results on dataset-dependent attention patterns may have told us more about the datasets rather than the GAT model.

There are a few presentation issues that need clarification:
- sec 5.1: it is not clear from the text what \alpha^{static} is.  As mentioned earlier there are multiple possible static attention weights (GCN vs GraphSAGE).
- sec 5.1: I found it strange to normalize the discrepancy score by 2|V|.  The number of terms in the sum should be 2|E| where |E| is the number of edges as each edge is counted twice.  Normalizing by 2|V| does not guarantee the score is in [0, 1] as claimed in the paper.
- sec 5.1: “Besides, these attention do not get concentrated on self loops based on relatively stable values.” -- I don’t see why having stable attention values can show there’s no concentration on self-loops.
- Figure 5 left: looks like the curves can’t reach the right end, this means 1 <= k <= 8 is probably not a good range.
- Table 5 is a bit confusing.  From the text my understanding is that a GAT is trained first, and then do sparsification, and then train another GraphSAGE on the sparsified graph to do prediction.  It’s not immediately clear why the GAT performance in Table 5 is so bad, while the GraphSAGE performance is just way better.  After reading this a few times I realized a much smaller GAT is used (with much smaller hidden size) while the GraphSAGE model is always using a large hidden size.  I think this part needs some improvement.

Overall I liked the empirical study and think the community can benefit from this paper.
review_title: Official Blind Review #1

Meta-review: This paper suggests that datasets have a strong influence on the effects of attention in graph neural networks and explores the possibility of transferring attention for graph sparsification, suggesting that attention-based sparsification retains enough information to obtain good performance while reducing computational and storage costs. 

Unfortunately I cannot recommend acceptance for this paper in its present form. Some concerns raised by the reviewers are: the analysis lacks theoretical insights and does not seem to be very useful in practice; the proposed method for graph sparsification lacks novelty; the experiments are not thorough to validate its usefulness. I encourage the authors to address these concerns in an eventual resubmission.

Decision: Reject