Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors propose an enhanced GAT model named SuperGAT by adding link prediction as an auxiliary task when conducting node classification and compare different methods of attention forms. The authors have also conducted evaluation based on both synthetic datasets and real-world datasets to analyze how different attention methods perform on various data and task types. In general, the paper is written well and easy to follow. However, there are still several issues the authors need to address:

First, the authors need to better justify the novelty of the proposed method. The claimed self-supervision task can be considered as general link prediction task where the attention weights is used as features. Though they authors propose two new types of attention forms, i.e., 1) scaled dot-product and 2) mixed GO and DP, they are normalized and combined version of existing attention mechanisms. I suggest the authors to better justify the novelty of the proposed technique and how they differ from existing work.

Another major concern is the experiment settings. It is good to see that the authors raise several research questions to guide experiment design. However, some assumptions in these research questions are questionable. For example, in RQ1, the authors claim that “ideal node representation can be generated by aggregating only neighbors with the same label”. As the neighborhood information of a node can be also informative when predicting node labels in certain cases, a good node representation does not necessarily need to only aggregates neighbors of the same labels, which makes the proposed method that uses KL divergence to compare label-agreement and graph attention questionable. I suggest the authors to provide more justification on this assumption. RQ2’s primary goal is to understand how different graph attention methods perform for the link prediction task, it would be better if the authors can justify why they didn’t conduct experiments where only link prediction (self-supervised) loss is used and discard the node classification task. In RQ3, the authors hypothesize that “different graph attention will have different abilities to model graphs under various homophily and average degree”. This is probably true. However, given the so many graph properties (e.g., degree distribution, graph diameter, and average clustering coefficient) and model configuration (e.g., # of layers and task type), it is unclear why the authors choose these two controlled variables and why they believe they are the most important ones. I suggest the authors provide more rationale on how they choose the controlled variables and how other factors may impact model performance.

Another minor question is that why do the authors add an activation function for $e_{ij, DP}$ in Eqn. 4, given that $e_{ij, DP}$ is already a dot product that indicates the weight of a link. It would be better if the authors can elaborate more on the design rationale.

In summary, I think the authors focuses on an interesting problem but need to further address the issues listed above.

review_title: R4

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
===========
The paper provides an interesting direction for improving the Graph Attention Networks. More specifically, the authors propose a self-supervised graph attention network (SuperGAT) designed for noisy graphs. They encode positive and negative edges so that SuperGAT learns more expressive attention. They focus on two characteristics that influence the effectiveness of attention and self-supervision: homophily and average degree. They show the superiority of their method (4 variations) by comparing it with many state-of-the-art methods, in 144 synthetic datasets (with varying homophily and average node degree) and 17 real-world datasets (again with various ).


Reasons for score:
===========
Overall, I vote for accepting. I find very interesting the idea of using self-supervision to improve graph attention networks and the experiments are nicely done and convincing. The authors do an impressive work to include as much information and results as they can in the given space.


Strengths:
===========
- The paper is about an interesting problem in the ICLR community. Graph Attention Networks have gained a lot of attention in the recent years from researchers in the field of graph and node representations with applications in node classification and link prediction. The idea of adding the advancements in recent direction of self-supervised learning to improve the learnt representations seems very promising.
- The authors have done a great job in the structure and the presentation of the paper. The paper is well-written and especially the sections Experiments and Results are well-structured and contain a lot of packed details in the design and the outcome of the experiments. More specifically, Figure 4 stands out as in a very limited space contains the information for the best performed model in all 144+17 graphs!
- The contributions of this work include the proposed method (GANs with self-supervision), but also an analysis for the selection of the best model depending on two important features of the graph (homophily and average degree).
- The authors compare their method with all state-of-the-art methods and also four variations of their own model and exhaust their evaluation by testing the performance in 144 synthetic graphs and 17 real-world datasets (including the benchmark datasets that are usually being used in this domain).


Weaknesses:
===========
- The proposed method uses two known graph attention mechanism as building blocks, they use negative sampling and they add cross-entropy loss for all node labels and self-supervised graph attention losses for all layers. These building blocks and mechanisms are known in the literature, and as a result the proposed method adds incremental novelty compared to the related works.
- In Appendix, A.3, the description and discussion for t-SNE plots is limited or absent. It would be better to add more details to it, as for example why this is a good representation and how the representations improve or not based on the hyperparameters. Also, how the results in the subfigures differ in terms of representations. It is difficult to get any insights from these plots.


Questions during rebuttal:
===========
- Overall, my recommendations for more analysis and insights of the results are all responded from the Appendices. I would like a comment and clarification from the authors regarding Appendix A.3 Figure 5 (t-SNE plots), even though it is not in the main paper submission.
- My understanding is that the authors are going to release the code upon acceptance, is this correct? In the repository that the code will be released, it will be useful to also add links to all 17 public datasets to ease research in the field. 

review_title: Recommendation for Acceptance

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new attention mechanism SuperGAT (with various flavours) for graph neural networks that is self-supervised. They exploit the presence/absence of an edge between a pair of nodes to guide the attention. The authors then make the observation that the homophily and average degree of a graph influence the design of the attention mechanism.  Extensive experiments are shown, where the various versions of SuperGAT are tested on 17 real-world dataset and many synthetic ones, and these results are compared against other state-of-the-art models (including the original GAT work). 

The paper is well-written and it is clear that the authors have done extensive experimentation to test their hypothesis. 
However, the paper has some weaknesses that I try to summarize below.
- To obtain SuperGAT, the authors have made some tweaks made to the original GAT formulation. These tweaks are minor and are not surprising or inspired from a deep/novel insight.  

- The choice of studying two graph properties homophily and average degree seem arbitrary. What is the reasoning behind these properties? Were there other properties (e.g. diameter, degree sequence, ...etc) that the authors have studied that did not yield good results? This is particularly of interest due to the fact that experiments do not entirely support/explain the importance of these two properties, such as in the case of Flickr and Crocodile datasets.

- The above reasons would still not be a major disadvantage had the experimental results shown strong superiority of SuperGAT over previous models. In all the experiments that the authors perform, SuperGAT’s performance is only slightly better than older models (e.g. in Table 2, difference between an accuracy of 72.5 vs 72.6 can hardly be considered superior). And even then, SuperGAT does not have the best performance across all datasets.

Proposition 1 and its proof are interesting contributions of this work, but they may not be enough. Perhaps instead of concentrating on performance superiority, the authors can look at the explainability aspect of their proposed architecture and look deeper into other graph properties that may guide the design/use of self-supervision.

Another minor comment: on Page 3, the authors say “if the number of nodes is large, it is not efficient to use all possible negative cases”. The number of negative cases is a function of the number of edges and not the number of nodes. If a graph contains all possible edges, then the number of all possible negative cases is zero, no matter how many nodes there are. 

review_title: The authors present a new self-supervised attention mechanism for graph neural networks and conduct extensive experiments. The analysis and the design however lack some depth and it is uncertain how much this work enhances our understanding (or ability) to use attention in graphs. 

final_score: rating: 7: Good paper, accept
review_content: ********Summary
In this paper, they introduced self-supervised graph attention network (SuperGAT), which is claimed to perform well in noisy graphs. They used information in the edges as an indicator of importance of relations in the graph, then they learn the relational importance using self-supervised attention. After learning the attentions values using their self-supervised method, they can predict the likelihood of an edge between nodes. They worked on two popular attention mechanisms: GO and DP, and showed in their experiments DP has better performance than GO for link prediction task. And Go has better performance in label-agreements between nodes. The other question they answered in their experiments was what the best attention model is to choose. They introduce a recipe based on two graph characteristics: homophily and average degree.

********Positives
- One thing I liked about this paper was thorough and neat experiments. I enjoyed the way they designed their experiments by mentioning several important questions followed by their answered backed up with their experiments. They used two attention mechanism as the base, then applied their method on these two methods for link prediction and label-agreements tasks and compare their results.

- I also liked that they examined their recommendation for the choice of attention model on real world datasets, and their answer for real-word data was almost similar to synthetics data.


- The paper was well-organized and well-written. They clearly explained their method.

********Notes
- I would recommend adding a picture showing their architecture and compare it with other two attention models

- I sort-of understand the reading as to why "GO learns label-agreement better than DP." Based on the argument on page 6. A strong argument would be helpful to explain why "DP predicts edge presence better than GO."

- (minor note:)No need the parenthesis in this sentence line 8: "Interestingly, for datasets (CS, Physics, Cora-ML, and Flickr) in which"

********Reason to accept
I am in general positive about this paper. The innovation is not significant; however, their experiments were interesting, and they prove how well their method works well empirically. I think this research will be useful for people in this area.

******* After Rebuttal
I have read the author's response

review_title: interesting method for graph attention network; thorough experiments; need to add a picture describing the general framework

Meta-review: Two reviewers are very positive about this paper and recommend acceptance, one indicates rejection and one is on the fence. Although all referees appreciate the extensive experiments and analysis presented in the paper, their main concerns are related to the limited superiority of the method wrt state of the art [R1], seemingly arbitrary choices and questionable assumptions [R4]. The rebuttal adequately addresses R1's concerns by highlighting statistical significance of the results, and partially covers R4's concerns. Although the proposed approach may be perceived as incremental [R1, R2, R3, R4], the authors argue that introducing self-supervision to graph attention is not trivial, and emphasize their findings on how/when this is beneficial. Moreover, R2 and R3 acknowledge that the contribution of the paper holds promise, is worth exploring, and may be useful to the research community. Most reviewers are satisfied with the answers in the rebuttal. After discussion, three referees lean towards acceptance and the fourth reviewer does not oppose the decision. I agree with their assessment and therefore recommend acceptance. Please do include your comments regarding the choice of average degree and homophily in the final version of paper.
Decision: Accept (Poster)