Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Review:
This paper extends the ComplEx model (Trouillon et al., 2016) for completing temporal knowledge bases by augmenting it with timestamp embeddings. Besides, based on the assumption that these timestamp representations evolve slowly over time, the paper introduces this prior as a regularizer. Also, the paper adds a non-temporal component to the model to deal with static facts in knowledge bases. The proposed model has been evaluated using the current benchmark temporal event datasets, showing state-of-the-art performance.
This paper could be weakly accepted because the paper introduces new regularization schemes based on the tensor nuclear norm for tensor decomposition over temporal knowledge graphs, which could be a significant algorithmic contribution. Additionally, the submission empirically studies the impact of regularizations for the proposed model, and weight different regularizers according to the joint marginal of timestamps and predicates, which achieves good experimental results.
Feedback to improve the paper:
1. For novelty, the paper does not clearly point out that Ma et al. (2018) already augmented the ComplEx model with time embedding for completing temporal knowledge graphs.
2. The paper does not point out whether the units of the timestamps affects the model and how to adjust the model accordingly. For example, the time granularity of the ICEWS dataset is 24 hours. If we switch the unit of timestamps from hours to days, do the results of the proposed model change? If yes, how to peak the best time unit for a given dataset?
3. For the Wikidata, the author reports the filtered Mean Reciprocal Rank of the conducted experiments, where the author provides not only the overall score but also the temporal MRR. However, the paper does not provide information about error bars as well as the unfiltered version of the experiment results. Since the results of TComplEx and TNTComplEx are only slightly better than ComplexE, the reviewer doubts whether the proposed model can really beat the ComplEx model when considering the error bars. Also, NT-MRR and T-MRR are not clearly defined.
4. The submission proposes a new large-scale temporal event dataset, and states that this dataset is more realistic than the current benchmarks. However, the reviewer does not find any argument in the paper to support this statement.
References:
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In International Conference on Machine Learning, pp. 2071–2080, 2016.
Yunpu Ma, Volker Tresp, and Erik A Daxberger. Embedding models for episodic knowledge graphs. Journal of Web Semantics, pp. 100490, 2018.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The work in this paper is focused on the task of knowledge base completion, dealing specifically with temporal relations, which is quite important in practice, and also not as well studied in literature. Specifically, the authors have 3 main contributions in this paper:
1. They present an order 4 tensor factorization for dealing with temporal data, which is a nice extension of the work in Lacroix 2018. The authors introduce different forms of regularization to extend to the order 4 tensors. Inspired by previous work, they produce different regularization strategies for order 4 tensor by unfolding the modes to reduce it ot an order 3 formulation. They also describe a regularization term for smoothing the temporal embeddings. 
3. Finally, the authors mine wikidata for temporal relations and contribute a dataset based on wikidata that is much larger than existing datasets for this task.


Overall, I think this paper is  interesting as it provides an incremental extension of ComplEx to the temporal case, and the experiments to support the formulation show improvements in MRR. However, the experiments around standard benchmarks as well as the data produced by the authors do not always support the hypothesis that modeling temporal dimension in the proposed formulation is a win for the KB completion task.  For example, the use of auxiliary losses  for enforcing temporal constraints makes the overall performance worse. This is mentioned in a section, but I think it deserves a more thorough explanation. Also, there is no mention about statistical significance of the results, and so it is hard to judge the claim of these being SOTA as made by the authors. For example, on the wikidata dataset produced, the ComplEx model outperforms the proposed models.



Strengths:
1. The work in this paper is quite well motivated and the modeling formulation is clear and easy to follow. The authors did a great job in citing relevant work and walking through the model formulation as well as the various regularization terms introduced.
2. The authors compare their work to the previous SOTA - the ComplEx models for multiple datasets, including their own released datasets. They present a clear set of experiments to demonstrate the effectiveness of their approach both on non-temporal and temporal relations.
3. There is a dataset being released and also code, which should aid in reproducibility of the results (though I have not tested the code).

Areas to be addressed:
1. The work seems to assume that the time is discretized by year. However, it is unclear how one would deal with a KB where the temporal relations can change on the order of weeks (example popular movies). For that matter, how would the model be modified to deal with heterogenous time scales in the evolution of the relations in the KB? Can the authors add some clarification as well as explanations for how this would be addressed in this formulation?
2. While there is a section devoted to different regularization, from Table 3, the impact of choosing the right regularizer seems minimal in terms of MRR. Also, it seems that the results in Table 3 are comparable to the “ranks multiplied by 10” setting in Table 2. What is the reason for this choice? 
3. For Table 4, why is there a performance difference between static and non-static relations? It would help if the authors could provide some more error analyses to dive into this performance difference - is it just data imbalance or inherent difficulty in the task for temporal relations. 
4. Section 6 talks about enforcing another auxiliary loss, however, these results are not part of the table. I would urge the authors to add this to Table 4. Also, the loss in MRR mentioned is it the average loss or does enforcing this auxiliary loss also influence performance on T-MRR as well? If so, might it be that the auxiliary loss is too strong and might need to be penalized? Did the authors try penalizing the auxiliary loss? Finally, the graph in Figure 2 is hard to follow when printed in black and white. What would the plot look like in comparison to a model that does not enforce the auxiliary function for the same example? I would recommend re-working Section 6 to provide some more details about the performance of the models both across the various datasets as well as error analyses of a few examples compared across TNTComplex, TNTComplex + auxiliary loss + Baseline Complex. Having some representative examples would make it easy to understand where these models differ in their performance.
5. In the results, can the authors include metrics like filtered MRR as well as Hits@10, similar to the one suggested by Bordes 2013? This would make it easier to compare against previous literature results which all seem to be reporting on these metrics.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors study an important problem, i.e., time-aware link prediction in a knowledge base. Specifically, the authors focus on predicting the missing link in a quadruple, i.e., (subject, predicate, ?, timestamp). In particular, the authors design a new tensor (order 4) factorization based method with proper regularization terms shown in Eqs.(4-6).

The authors also prepare a dataset which may be helpful for further study on this topic, which is highly appreciated.

Empirical studies on three public datasets show the effectiveness of the proposed model over the state-of-the-art tensor factorization based methods that are published very recently.


Some comments:

1 The technical novelty is a bit limited in compared with those 3-order tensor factorization based methods.

2 For the time-aware link prediction problem, some deep learning based methods may also perform well, especially those capturing sequential signals, e.g., RNN and their variants. The authors are encouraged to include those works in the related work and in the empirical studies.

Minor:

Typo: 'ICEWS05-15'

review_title: Official Blind Review #1

Meta-review: The authors propose a new algorithm based on tensor decompositions for the problem of knowledge base completion. They also introduce new regularisers to augment their method. They also propose an new dataset for temporal KB completion.  

All the reviews agreed that the paper addresses an important problem and presents interesting results. The authors diligently responded to reviewer queries and addressed most of the concerns raised by the reviewers. 

Since all the reviewers are in agreement, I recommend that this paper be accepted. 
Decision: Accept (Poster)