Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper introduces a neural network architecture to operate on graph-structured
data named Graph Attention Networks.
Key components are an attention layer and the possibility to learn how to
weight different nodes in the neighborhood without requiring spectral decompositions
which are costly to be computed.

I found the paper clearly written and very well presented. I want to thank
the author for actively participating in the discussions and in clarifying already
many of the details that I was missing.

As also reported in the comments by T. Kipf I found the lack of comparison to previous
works on attention and on constructions of NN for graph data are missing.
In particular MoNet seems a more general framework, using features to compute node
similarity is another way to specify the "coordinate system" for convolution.
I would argue that in many cases the graph is given and that one would have
to exploit its structure rather than the simple first order neighbors structure.

I feel, in fact, that the paper deals mainly with "localized metric-learning" rather than
using the information in the graph itself. There is no
explicit usage of the graph beyond the selection of the local neighborhood.
In many ways when I first read it I though it would be a modified version of
memory networks (which have not been cited). Sec. 2.1 is basically describing
a way to learn a matrix W so that the attention layer produces the weights to be
used for convolution, or the relative coordinate system, which is to me a
memory network like construction, where the memory is given by the neighborhood.

I find the idea to use the multi-head attention very interesting, but one should
consider the increase in number of parameters in the experimental section.

I agree that the proposed method is computationally efficient but the authors
should keep in mind that parallelizing across all edges involves lot of redundant
copies (e.g. in a distributed system) as the neighborhoods highly overlap, at
least for interesting graphs.

The advantage with respect to methods that try to use LSTM in this domain
in a naive manner is clear, however the similarity function (attention) in this
work could be interpreted as the variable dictating the visit ordering.

The authors seem to emphasize the use of GPU as the best way to scale their work
but I tend to think that when nodes have varying degrees they would be highly
unused. Main reason why they are widely used now is due to the structure in the
representation of convolutional operations.
Also in case of sparse data GPUs are not the best alternative.

Experiments are very well described and performed, however as explained earlier
some comparisons are needed.
An interesting experiment could be to use the attention weights as adjacency
matrix for GCN.

Overall I liked the paper and the presentation, I think it is a simple yet
effective way of dealing with graph structure data. However, I think that in
many interesting cases the graph structure is relevant and cannot be used
just to get the neighboring nodes (e.g. in social network analysis).
review_title: Very interesting work, but the graph structure is not fully exploited

final_score: rating: 5: Marginally below acceptance threshold
review_content: This is a paper about learning vector representations for the nodes of a graph. These embeddings can be used in downstream tasks the most common of which is node classification.

Several existing approaches have been proposed in recent years. The authors provide a fair and almost comprehensive  discussion of state of the art approaches. There are a couple of exception that have already been mentioned in a comment from Thomas Kipf and Michael Bronstein. A more precise discussion of the differences between existing approaches (especially MoNets) should be a crucial addition to the paper. You provide such a  comparison in your answer to Michael's comment. To me, the comparison makes sense but it also shows that the ideas presented here are less novel than they might initially seem. The proposed method introduces two forms of (simple) attention. Nothing groundbreaking here but still interesting enough and well explained. It might also be a good idea to compare your method to something like LLE (locally linear embedding). LLE also learns a weight for each of neighbors of a node and computes the embedding as a weighted average of the neighbor embeddings according to these weights. Your approach is different since it is learned end-to-end (not in two separate steps) and because it is applicable to arbitrary graphs (not just graphs where every node has exactly k neighbors as in LLE). Still, something to relate to. 

Please take a look at the comment by Fabian Jansen. I think he is on to something. It seems that the attention weight (from i to j) in the end is only a normalization operation that doesn't take the embedding of node i into account.  

There are two  issues with the experiments.

First, you don't report results on Pubmed because your method didn't scale. Considering that Pubmed has less than 20,000 nodes this shows a clear weakness of your approach. You write (in an answer to a comment) that it *should* be parallelizable but somehow you didn't make it work. We have to, however, evaluate the approach on what it is able to do at the moment. Having a complexity that is quadratic in the number of nodes is terrible and one of the major reasons learning with graphs has moved from kernels to neural approaches. While it is great that you acknowledge this openly as a weakness, it is currently not possible to claim that your method scales to even moderately sized graphs. 

Second, the experimental set-up on the Cora and Citeseer data sets should be properly randomized. As Thomas pointed out, for graph data the variance can be quite high. For some split the method might perform really well and less well for others. In your answer titled "Requested clarifications" to a different comment you provide numbers randomized over 10 runs. Did you randomize the parameter initialization only or also the the train/val/test splits? If you did the latter, this seems reasonable. In Kipf et al.'s GCN paper this is what was done (not over 100 splits as some other commenter claimed. The average over 100 runs  pertained to the ICA method only.) 
review_title: Good basic idea with several weaknesses in the technical exposition and the experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper has proposed a new method for classifying nodes of a graph. Their method can be used in both semi-supervised scenarios where the label of some of the nodes of the same graph as the graph in training is missing (Transductive) and in the scenario that the test is on a completely new graph (Inductive).
Each layer of the network consists of feature representations for all of the nodes in the Graph. A linear transformation is applied to all the features in one layer and the output of the layer is the weighted sum of the transformed neighbours (including the node). The attention logit between node i and its neighbour k is calculated by a one layer fully connected network on top of the concatenation of the transformed representation of node i and transformed representation of the neighbour k. They also can incorporate the multi-head attention mechanism and average/concatenate the output of each head.

Originality:
Authors improve upon GraphSAGE by replacing the aggregate and sampling function at each layer with an attention mechanism. However, the significance of the attention mechanism has not been studied in the experiments. For example by reporting the results when attention is turned off (1/|N_i| for every node) and only a 0-1 mask for neighbours is used. They have compared with GraphSAGE only on PPI dataset. I would change my rating if they show that the 33% gain is mainly due to the attention in compare to other hyper-parameters. [The experiments are now more informative. Thanks]
Also, in page 4 authors claim that GraphSAGE is limited because it samples a neighbourhood of each node and doesn't aggregate over all the neighbours in order to keep its computational footprint consistent. However, the current implementation of the proposed method is computationally equal to using all the vertices in GraphSAGE.

Pros:
- Interesting combination of attention and local graph representation learning. 
- Well written paper. It conveys the idea clearly.
- State-of-the-art results on three datasets.

Cons:
- When comparing with spectral methods it would be better to mention that the depth of embedding propagation in this method is upper-bounded by the depth of the network. Therefore, limiting its adaptability to broader class of graph datasets. 
- Explaining how attention relates to previous body of work in embedding propagation and when it would be more powerful.
review_title: Well written paper, lack of novelty

Meta-review: The authors appear to have largely addressed the concerns of the reviewers and commenters regarding related work and experiments. The results are strong, and this will likely be a useful contribution for the graph neural network literature.
Decision: Accept (Poster)