Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a recurrent and autorgressive architecture to model temporal knowledge graphs and perform multi-time-step inference in the form of future link prediction. Specifically, given a historical sequence of graphs at discrete time points, the authors build sequential probabilistic approach to infer the next graph using joint over all previous graphs factorized into conditional distributions of subject, relation and the objects. The model is parameterized by a recurrent architecture that employs a multi-step aggregation to capture information within the graph at particular time step. The authors also propose a sequential approach to perform multi-step inference. The proposed method is evaluated on the task of future link prediction across several baselines, both static and dynamic, and ablation analysis is provided to measure the effect of each component in the architecture.

The authors propose to model temporal knowledge graphs with the key contribution being the sequential inference and augmentation of RNN with multi-step aggregation. The paper is well written in most parts and provides adequate details with some exceptions. I appreciate the extended ablation analysis as it helps to segregate the effect of each component very clearly. However, there are several major concerns which makes this paper weaker:

- The paper approaches temporal knowledge graphs in discrete-time fashion where multiple events/edges are available at each time step. While this is intuitive, the authors fail to position the paper in light of various existing discrete-time approaches that focus on representation learning over evolving graphs [1,2,3,4,5]. Related work mentions [1] learns evolving representations but all these methods can do future link prediction and hence this is a big miss for the paper. A discussion and comparison with these approaches is certainly required as most of static and dynamic baselines currently compared also focus on learning representations, hence that is not a valid argument to miss comparison. 

- The baselines tested by the authors are either support static graphs, supports interpolation or supports continuous time data. However, as the authors explicitly propose a discrete time model starting from Section 3, it is important to perform experiments on atleast few of the discrete time baselines to demonstrate the efficacy of the proposed method. For instance, authors can augment relation as extra feature or use their encoders and optimization function to perform experiments e.g. Evolve-GCN  only require to replace GCN with R-GCN.

- From the ablation it is clear that aggregation is the most important component as without it, the performance drops much closer to ConvE which is a static baseline and significantly worse than other RE-Net variants. However, the aggregation techniques are not novel contributions but augmentation to the RNN architecture. Hence it is important to show how augmenting aggregation module with other baselines (for instance, ConvE and TA-DistMult)) and the above mentioned discrete baselines would affect the performance of these baselines. 

- While the authors describe attentive Pooling Aggregator, the experiments only show mean aggregator and multi-step one. Is there a reason Attentive pooling is not used for any experiments? 
-It appears that global vector H_t is not playing significant role based on ablation study. Can the authors explain why that si the case? Also, what aggregation is used to compute H_t? Is it sum over all previous h_t's?

- Algorithm 1 is not very clearly explained. When the authors mention that they only use one sample, does that mean a single subject is sampled at each time point t'? If so, how do you ensure the coverage is good across subjects in the newly generated graph? I admit I am not clear on this and would recommend the authors to elaborate in response and also in the paper. Also, the inference computation complexity is concerning. While it seems fine for the provided dataset, most real-world graphs have billion of nodes and I all of E, L and D would be larger for such graphs. This seems to put a strict limitation on scalability of inference module. 

- It is not clear what is the difference between RE-NET and RE-NET w. GT. Could the authors elaborate this more? It seems the authors do not update history when they perform RE-NET w/o multi-step. However, in the RE-NET w. GT, where is the ground truth history used in Algorithm 1?

- The time span expansion for WIKI and YAGO is very unnatural and it is not clear if these experiments provide any value. For instance, can the authors show that in multi-step inference scheme, they can actually predict events at multiple time points corresponding to time span events in actual dataset? As multiple triplets can appear at consecutive time points, the current modification just makes them equivalent which doesn't seem correct. 

I am willing to revisit my score if the above concerns are appropriately addressed and requested experiments are provided.

[1] Evolve-GCN: Evolving Graph Convolutional Networks for Dynamic Graphs, Pareja et. al.
[2] DynGEM: Deep embedding method for dynamic graphs, Goyal et. al.
[3] dyngraph2vec: Capturing network dynamics using dynamic graph representation learning, Goyal et. al.
[4] Dynamic Network Embedding by Modeling Triadic Closure Process, Zhou et. al.
[5] Node Embedding over Temporal Graphs, Singer et. al.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper presents the Re-Net model which sequentially generates a temporal knowledge graph (TKG) in an autoregressive fashion by taking both global and local information into account.
The generation of TKG is motivated via a joint distribution problem which is then parametrized by the usage of a recurrent event encoder.
In addition to past information, the encoder aggregates local as well as global information for which the authors propose three different aggregation schemes build upon the works of, e.g., attentive pooling and the RGCN model.
In an in-depth-evaluation study, the performance of the proposed model is evaluated on five different datasets on which it consistently improves upon the state-of-the-art.
An ablation study shows the benefits of all proposed features of Re-Net, e.g., the usage of more sophisticated aggregation schemes, the impact of using global information, and the number of RGCN layers.

As far as I know, the proposed method is a novel and clever (though not ground-breaking) contribution to the field of performing global structure inference over TGKs.
The paper is well-written but is partially becoming a little hard to comprehend due to its overloaded notation, e.g., $h_t(s)$ vs. $h_s^l$ and $N(s)_t$ vs. $N_t^{(s)}$ vs. $N_t^{(s,r)}$, and could be improved by a more rigorous formulation, e.g., for $N(s)_t$ or $c_s$ (which should also depend on r).

1. Re-Net evolves the embeddings of entities and performs predictions via negative log likelihood. Hence, the model seems to be limited to predict events between entities which have been already seen during training and does not generalize to unseen entities. In addition, by applying a softmax classifier your model does not seem to be able to scale to large knowledge graphs? Are those observations correct and how could they be resolved?

2. As far as I understood, the formulation of $N^{(s,r)}$ is not needed for defining the mean and attentive pooling aggregators since you are aggregating information independent of the relation type. However, the current formulation could confuse readers (including me).

3. Algorithm 1 could be made more clear since the sampled number of M subjects does not get mentioned again. I guess the top-k triples are picked across all M samples and not individually? In addition, the sampling of subjects should relate to Equation 5 instead of Equation 4.

4. In Figure 3 it is not clear why the accuracy does not decrease with temporal distance from the training examples. It would be helpful to interpret and clarify the results in more detail.

5. I was not able to fully comprehend your complexity analysis. For example, it is not clear what $|E|$ means (I guess the maximum number of triples in a time step?). In addition, it seems that you are still dependent on computing node embeddings for all entities in your graph, even if you only report runtimes for computing a single example. In my opinion, there is a $L \cdot |E|$ term missing in your complexity analysis for computing RGCN across the whole graph. Please clarify!

6. The results of using the attentive aggregation scheme should be included into Tables 1 and 2.

7. Since Figure 5c signalizes that Re-Net can effectively leverage larger receptive field sizes, how does it perform when increasing the number of layers further?

------------------------
Update after the rebuttal:

I would like to thank the authors for answering my questions and clarifying several issues. The raised questions were not critical for my overall rating, which remains unchanged (6: Weak Accept).
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper properly applied several technique from RNN and graph neural networks to model dynamically-evolving, multi-relational graph data. There are two key component: a RNN to encode temporal information from the past event sequences, and a neighborhood aggregator collects the information from the neighbor nodes. The contribution on RNN part is design the loss and parameterizes the tuple of the graph. The contribution of the second part was adapting Multi-Relational Aggregator to this network. The paper is well-written. Although I'm familiar with the dataset, the analysis and comparison seems thorough. 

I'm leaning to reject or give borderline for this paper because (1) This paper is more like an application paper. Although the two component is carefully designed, the are more like direct application. I'm not challenge this paper is not good for the target task. But from the point of view on Machine learning / deep learning, there is not much insight from it. The technical difficult was more from how to make existing technique to fit this new problem.  This "new" problem seems more fit to data mining conference. (2) The experiments give tons of number but it lack of detailed analysis, like specific win/loss case of this model. As a more application-side paper, these concrete example can help the reader understand why this design outperform others. For example, it can show what the attention weights look like, and compare to the proposed aggregator. 

Some questions:
[This question is directly related to my decision] Does this the first paper to apply autoregressive to knowledge graph? from related work, the answer is no. Can the author clarify more on this sentence? 

"In contrast, our proposed method, RE-NET, augments a RNN with message passing procedure between entity neighborhood to encode temporal dependency between (concurrent) events (i.e., entity interactions), instead of using the RNN to memorize historical information about
the node representations."

The paper give complexity of this algorithm but no comments about how it compare with other method and how practical it is.

It lacks of some details for the model:
(1) what is the RNN structure? 
(2) For the aggregator, what is the detailed formulation of h_o^0? 
 
review_title: Official Blind Review #1

Meta-review: The paper proposes a recurrent and autorgressive architecture to model temporal knowledge graphs and perform multi-time-step inference in the form of future link prediction. However, the reviewers feel that the papers are more of a straight application of current techniques. Furthermore, a better presentation of the experimental section will also help improve the paper.  
Decision: Reject