Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposed the temporal graph attention layer which aggregates in-hop features with self-attention and incorporates temporal information with Fourier based relative positional encoding. This idea is novel in GCN field. Experimental results demonstrate that the TGAT which adds temporal encoding outperforms the other methods. Overall this paper addressed its core ideas clearly and made proper experiments and analysis to demonstrate the superiority against existing counterparts.

There are some things need to be further answered. The baselines compared in this paper seems to be too weak. For example, how does T-GraphSage (GraphSAGE+Temporal encoding) work? How does the single-head variant of TGAT work? How does the original GAT work plus temporal encoding (as I notice TGAT uses self-attention which is similar but may not be equivalent to original GAT attention formulation, are they equivalent or not?)
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Summary: This paper addresses the problem of representation learning for temporal graphs. That is, graphs where the topology can evolve over time. The contribution is a temporal graph attention (TGAT) layer aims to exploit learned temporal dynamics of graph evolution in tasks such as node classification and link prediction. This TGAT layer can work in an inductive manner unlike much prior work which is restricted to the transduction setting. Specifically, a temporal-kernel is introduced to generate time-related features, and incorporated into the self-attention mechanism. The results on some standard and new graph-structured benchmarks show improved performance vs a variety of baselines in both transduction and inductive settings. 

Pros: 
+ Dynamic graphs are an important but challenging data structure for many problems. Improved methods in this area are welcome. 
+ Dealing with the inductive setting is an important advantage. 
+ Clear performance improvements on prior state of the art is visible in both transductive+inductive settings and node+edge related tasks.

Cons+Questions:
1. Technical significance: Some theory is presented to underpin the approach, but in practice it seems to involve concatenating or adding temporal kernels element-wise to the features already used by GAT. In terms of implementation the concatenation in Eq 6 seems to be the only major change to GAT. I’m not sure if this is a major advance.  
2. Insight. The presented method apparently improves on prior work by learning something about temporal evolution and exploiting it in graph-prediction tasks. But it's currently rather black-box. It would be better if some insight could be extracted about *what* this actually learns. What kind of temporal trends exist in the data that this method has learned? And how are they exploited in by the prediction tasks?
3. Writing. The English is rather flaky throughout. One particular recurring frustration is the use of the term “architect” which seems wrong. Probably “architecture” is the correct alternative. 
4. Clarity of explanation. The paper is rather hard to follow and ambiguous. A few specific things that are not explained so well: 
4.1. Eq 1+2 is not a sufficiently clear and self-contained recap of prior work. 
4.2. Symbol d_T used at the start of Sec 3.1 seems to be used without prior definition making it hard to connect to previous Eq1+2. 
4.3 The claim made about alternative approaches (Pg4) “Reparameterization is only applicable to local-scale distribution family, which is not rich enough”. Seems both too vague and unjustified. 
4.4 The relationship between $t_i$ and the neighbours of the target node in Eq. 6 is not very clear.

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The major contribution of this paper is the use of random Fourier features as temporal (positional) encoding for dynamic graphs. These encodings are concatenated with standard node embeddings in transformer-like attention calculations for graph message passing. The reader finds that the proposed approach is interesting.

Experimental results are also favorable.

Concern: Whereas the use of random Fourier features (RFF) is well justified, a limitation is that it is based on a stationarity assumption. Thus, it may be less applicable to nonstationary structural changes. To cope with nonstationarity, a straightforward idea is to parameterize the temporal encoding by using neural networks rather than RFF. In the authors' approach, the RFF is in a sense parameterized, because the frequencies omega are learned. Nevertheless, the stationarity limitation persists.

Question: In the ablation study, what exactly is "the original positional encoding"? Are they learned embedding vectors? Since the authors consider continuous time rather than discrete time, how many embedding vectors are there?


review_title: Official Blind Review #1

Meta-review: The major contribution of this paper is the use of random Fourier features as temporal (positional) encoding for dynamic graphs. The reviewers all find the proposed method interesting, and believes that this is a paper with reasonable contributions. One comment pointed out that the connection between Time2Vec and harmonic analysis has been discussed in the previous work, and we suggest the authors to include this discussion/comparison in the paper.
Decision: Accept (Poster)