Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: **Summary:** This paper proposes xERTE, a comprehensive set of strategies (i.e. a temporal relational attention mechanism and a human-mimic representation update scheme, temporal neighborhood sampling and pruning, etc.) for link forecasting in temporal knowledge graphs (tKGs). Experiments on real-world tKGs show significant improvements and better explainability on KG forecasting. 

**Reason to accept:**
(1) Well motivated problem formation and clear definition. (2) Well illustrated model architecture and model details of major components and technical details, the reverse representation update is particularly interesting; (3) Extensive experiments on multiple tKG datasets are conducted. Insights, ablation studies on model variants, and case studies are also provided.

**Reason to reject:**
Model complexity analysis and inference time. Clarity and some details about experiments and sampling techniques.

**Comments & Questions:**
(1) The approach is technically sound. One concern is about model complexity and inference time. The workflow of xERTE seems complicated with multiple attention calculations and representation updates and it would be better to provide the inference time for query (regarding different steps) to show the scalability and whether it is applicable for large-scale tKG.
 
(2)  Several questions regarding the experiments need clarification: 
(i) Is $l$ (step number) or $L$ explicitly given in the paper or manually set up? What is the number of steps for inference in the test set? Also, what is the difference between the expansion step and the inference step? 
(ii) How do you make predictions by static KG embedding models by “compressing temporal knowledge graphs into a static, cumulative graph by ignoring the time information”? 
(iii) What does “raw MRR/Hits@k” score mean? Does it refer to the raw/filtered settings? If so, why not use filtered evaluation, which seems more reasonable? (iv) The performance of TcomplEx, which is claimed as one baseline, is not reported. 
(v) Why are the performance of TA-TransE and T-TransE even worse than static TransE? It was not the case using similar datasets in [4].
(vi) How are all entity embeddings (l=0) and predicated embeddings initialized? Are entity embeddings from 4.4 Dynamic embeddings?  Are all dimension parameters explicitly listed in the paper? 

(3) The uniform and time-sensitive sampling are fairly reasonable. However, many KGs (including YAGO used in this paper) have given ontology with specific entity types, domain and range for the edges, which are helpful for KG inference [1,2,3]. Why choose timestamp-weighted sampling instead of sampling by relations or entity types which also significantly limit the subgraph size and are intended to pick high-related relations and important entities. 

(4) Can xERTE apply to an inductive setting, in which new entities are emerging as tKG develops as a commonly observed case? A related concern is how the facts in these datasets are split into train/validation/test? 

(5) As one major contribution to explainability, it is suggested to provide analysis about justification on temporal model capability. One possible way to do so is to provide how the representation of one node evolves during the inference graph. 

(6) In Section 4.4, what are stationary embeddings and time-variant embeddings? Do we assume both are given in the dataset or trained? What is the contribution here to adopt dynamic embeddings?

**Minor issues:**
(1) Almost all figures are not quite visible and the text in the subfigures needs to be enlarged.
(2) Page 3, Inference Graph, Line 7: Incomplete sentence.

**References: **
[1] Ma, S, et al. "Transt: Type-based multiple embedding representations for knowledge graph completion." ECML, 2017.
[2] Lv, X., et al. "Differentiating Concepts and Instances for Knowledge Graph Embedding." Proceedings of the 2018 Conference on EMNLP. 2018.
[3] Hao, J., et al. "Universal representation learning of knowledge bases by jointly embedding instances and ontological concepts." KDD, 2019.
[4] Garcia-Duran, Alberto, Sebastijan Dumančić, and Mathias Niepert. "Learning Sequence Encoders for Temporal Knowledge Graph Completion." EMNLP, 2018.

review_title: Interesting paper on subgraph inference over temporal KG. Some clarifications needed.

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
**[UPDATE, 30 Nov]: Rating raised after reading the authors rebuttal.**

The paper presents a knowledge graph embedding model that learns latent representation of a temporal knowledge graph to predict unseen events.

The problem addressed in the paper is relevant for the community, and the angle proposed by the authors is interesting, i.e. tackling learning from temporal graphs with an eye to interpretability.

The paper is well structured, although some sections lack clarity and at times makes reading difficult (e.g. 4.3). 

Related work is comprehensive and gives a good coverage of recent temporal KGE works.

The choices made in the architecture (which is grounded on the so-called “inference graph”) seem reasonable. 

It is worth mentioning that is the first KGE paper I see that includes a user evaluation survey. Given the narrative emphasises explainability, this is certainly appreciated. 

Experiments use agreed-upon datasets and evaluation protocol.

Nevertheless, there are some shortcomings. I have some doubts and questions for the authors:

1. Why not replacing Table 1 with Table 7 (appendix)? Reporting raw metrics in the main paper is less meaningful that the time-aware filtered settings.
2. Baselines: experiments in table 1 do not include recent SOTA-level models such as DE-Simple [Goel 2020] and TNTComplex [Lacroix 2020] - this despite both works are present in the bibliography. Given that xERTE claims to beat SOTA, I believe it is necessary to add those two models in the experiments to make sure SOTA is beaten, in particular using the time-aware filtered settings (Table 7, appendix).
3. The paper does not include experiments on training time. It will be interesting to empirically validate claims of scalability. Training time is a deal breaker for time-aware models, so reporting figures would really add value to the work.
4. The reverse representation update mechanism in 4.3 lacks clarity. Could you please re-phrase and provide a bit more clarity? I fail to see how what described “mimics human behaviour”.
5. In a note on page 3 you say reciprocal triples are added. How does this impact training time, given that in practice you double the size of the training set?
6. In 4.1, how is the number of expansion sets L chosen? Is that an hyperparameter? In that case, it would have been interesting to see some experiments of predictive power and training time.
7. What is the role of the \gamma hyperparameter in 4.2? Have you provided some experiments to show what happens with ranges of different values?
8. Have you run ablation studies on the attention mechanism? (i.e. turning it on/off).
9. How is the time encoding \Phi defined (4.4)? Could you please clarify?
10. Unfortunately, the human evaluation is dismissed in the appendix. While I appreciate the effort of running a user survey,  the summary proposed in 5.3 lacks clarity and this piece of your work would deserve more prominence.
11. Human evaluation: I have doubts on using only 7 questions in the questionnaire. I wonder how meaningful results are. Besides, the questionnaire does not compare against a baseline, so it is hard to tell if the users are really satisfied with the content of the inference graph.
12. Human evaluation: could you add some details on the population? (i.e. the 53 users) 

Minor comments:
- figures 2,3 are too small.
- content of user survey description in I.1 may partially break anonymity.
review_title: Time awareness for link prediction in knowledge graphs with an eye on interpretability 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper addresses the problem of link forecasting in temporal knowledge graphs with special emphasis on generating explainable predictions. The explainability component of their work involves producing a subgraph of the knowledge graph (called the inference graph) that serves as a context of the query being asked, and on which the prediction decision was based. 

The inference graph is generated by sampling a subset of prior edges in the L-hop neighborhood of the query node. The authors try different sampling schemes to select a subset of edges. They then use a GNN aggregation scheme to produce a time-away representation of each node. This representation is generated by taking into account the “relevance” of nodes and edges to the query by employing a relational attention mechanism. Each node is then given a plausibility score, and the node with highest plausibility score is selected to be the answer to the query.  

The authors run experiments on standard temporal datasets and have appropriate ablation studies. They compare their work to existing methods, and their experiments show that their model performs significantly better than state of the art.
Link forecasting in temporal knowledge graphs is an important problem that arises in many real-life applications, and being able to explain these predictions is equally important. The model proposed by the authors contain the right components for tackling the problem: subgraph sampling based on time, attention mechanism based on the role of a given entity in the query, message passing to learn from neighborhood information for each node, ...etc. 

The writing of the paper however is hard to follow; notation is sometimes incomplete or undefined, mathematical formulations are often unnecessarily complex. The pieces were described in a somewhat topsy-turvy order and It was difficult putting together all the pieces of the algorithm. Some further details below:
- Page 3, paragraph 3: what is tKG? There is possibly a missing word in the sentence before-last of the paragraph “with v_q to .”

- In Definition 2, the definition of N_v and \bar{N_v} is hard to follow, even though the concept is simple. Consider saying it in plain words first, then state the definition of N_v without the union operator (by using an “or” in the conditional)

- There is a lot of repetition in the text. E.g. mention of “sampling the neighbors” a few times in the text until you encounter it in Section 4.5. Perhaps discuss sampling first?)

- In Equation 2 (and elsewhere), what is P_{vw}?

- Section 4.3 was hard to follow. The motivation for computing representations in a reverse manner seems to stem only from computational considerations, but it is not clear whether this reverse update mechanism produces the same representation as the standard “forward” update would. The section in the appendix on the topic did not shed more light to this process. 

- In the same paragraph, the sentence starting with “Since we only feed…” is confusing:  is the current inference step m or l ?
Section 4.6: It would be less confusing to put the definition of a_e_i^l before that of a_v^l. Also, at which point is this “plausibility score” computed?

- There is a lot happening in Figure 1 but the figure is not referenced from the text. Why are there two identical modules? Perhaps more explanation would be helpful. 

Overall, the paper has a lot of promise and the work conducted looks solid. The authors are encouraged to make another attempt at making their work clearer and understandable.

review_title: Temporal Knowledge Graph forecasting is an important problem that rises in many applications, and the authors have used an array of attention mechanisms and subgraph sampling techniques to tackle the problem. The clarity of their presentation however needs improvement, and some details further clarifications.

final_score: rating: 1: Trivial or wrong
review_content: In Appendix I.1 it says "We are machine learning scientists from the University of Munich...", which violates the rule of anonymity. Please let me know if I should continue on reviewing this paper. Thanks!
review_title: This paper violates the rule of anonymity

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
Authors have presented a method to forecast future links on temporal knowledge graphs (KGs). They use attention mechanisms to extract a query-dependent subgraph. According to the authors, this extracted subgraph provides a graphical explanation of the prediction. Authors have performed an ablation study to denote the effect of different components (e.g., updating the representation of nodes, time encoding, sampling strategy) in their method. They have tested the performance of their approach on 3 datasets and have shown that their approach outperforms other baselines in terms of Hits and MRR.

Although the method proposed in this work outperforms other baselines, the novelty and contribution of this work is still not completely clear to me. Major contributions are listed in the last paragraph; however,  I believe some of them are derived with straightforward changes to previous works. As an example, authors have mentioned they have provided the "first" explainable model based on attention mechanisms for temporal KGs.

As far as I have understood, by "explainability" authors mean providing the graphical explanation or in other words, providing relevant arguments in the extracted subgraph. The most relevant paper to this line of work is by Xu et al. 2019 which is also cited by authors. Authors explicitly mention that the work by Xu et al. deals with static KGs whereas their method works with temporal KGs as well. To me, the base of both approaches are the same since Xu et al. 2019 also models a reasoning process by constructing a sub-graph and I believe this issue is limiting the novelty of this work. It would be great if authors can show an example in which the work by Xu et al. does not provide a proper explainability while their approach does.

I also recommend that authors define the metrics they have used (i.e., Hits and MRR) either in footnotes or the supplementary material.

The writing of the paper could be improved. Examples are but not limited to:
- Page 3, 2nd paragraph, "with $v_q$ to$
- Page 5, 1st paragraph, "we need pass messages"
- Page 5, 1st paragraph, "$(l − m)$-aeay"
- Page 5, 1st paragraph,  "at l-th inference step"
- Page 6, 1st paragraph, "an unique"

review_title: Review of the paper

Meta-review: The paper has received 4 positive reviews all supporting the acceptance of the paper. The authors have provided a strong rebuttal and have addressed the reviewers' concerns. Please make sure to include all reviewer feedbacks in the camera-ready version. 
Decision: Accept (Poster)