SlowGuess/NAACL_2021_OCR huggingface dataset:
Dataset	YAGO11k	Wikidata12k
Metric	MR	Hits@1	MR	Hits@1
TransE*	1.70	78.4	1.35	88.4
TransH*	1.53	76.1	1.40	88.1
HoIE*	2.57	69.3	2.23	84.0
t-TransE*	1.66	75.5	1.97	74.2
Hyte*	1.23	81.2	1.13	92.6
RTFE-TransE	1.43	84.1	1.88	73.7
RTFE-RDGCNtype	1.19	93.7	1.77	82.9
RTFE-RDGCN	1.11	95.0	1.36	83.2
RTFE-RDGCNrel	1.10	96.4	1.20	92.3
Table 3: Relation prediction. For RTFE-RDGCNtype,  $\lambda$  is set to 0, which only considered type distance. For RTFE-RDGCN,  $\lambda$  is set to 0.2.
$G_{t},(s,r^{\prime},o)\notin G\}$  .We call this variant RTFE-  $\mathrm{RDGCN}_{rel}$  , which improves the performance of relation prediction on Wikidata12k compared with RTFE-RDGCN (with the improvement of  $9.1\%$  in Hits@1).
5.4 Extensibility validation
In order to verify the influence of pre-trained static features on RTFE's entire TKG completion, we divide timestamps into four time intervals and perform pre-training of RTFE on them respectively. Then the pre-trained static features of these time
intervals are used as inputs to RTFE to test the performance of entity prediction at all timestamps.
The experimental results are presented in Figure 3. Although a complete SKG is not provided for pre-training, RTFE still remains a similar performance, which verifies the framework's extensibility for future timestamps. So RTFE can be extended to future timestamps to some extent, without the retraining of former timestamps, which shows good lightness and immediacy.
5.5 Ablation study
In this subsection, we explore the effects of preliminary training and recursive training. w/o pretrain refers to RTFE without preliminary training for static futures (i.e., only recursive training). As shown in Table 4, RTFE generally outperforms w/o pre-train, which verifies the significance of preliminary training. For TKGE models, w/o pre-train is generally competitive with them. With preliminary training, RTFE gets a good starting point. So RTFE outperforms the original TKGE models.
However, it's interesting to find that on GDELT w/o pre-trains outperform RTFE-DE-SimpleE and RTFE-TComplEx. This can be attributed to the special denseness of this dataset. RTFE can fit each timestamp well with its enough facts and recursive
Dataset	Wikidata12k	YAGO11k	ICEWS14	ICEWS05-15	GDELT
Metric	MRR	Hits@1	MRR	Hits@1	MRR	Hits@1	MRR	Hits@1	MRR	Hits@1
RTFE-RotatE	36.2	22.1	23.7	17.1	45.1	30.1	35.9	12.5	35.7	24.8
w/o pre-train	27.5	21.2	20.1	16.4	33.0	17.6	35.7	16.0	32.5	20.3
DE-SimpleE	-	-	-	-	52.6	41.8	51.3	39.2	23.0	14.1
RTFE-DE-SimpleE	-	-	-	-	57.3	49.4	58.7	50.6	42.9	35.9
w/o pre-train	-	-	-	-	48.5	42.6	53.1	46.8	44.7	37.9
TComplEx	37.2	26.6	25.2	18.1	54.7	44.5	58.3	48.0	21.7	12.8
RTFE-TComplEx	45.4	36.6	25.8	18.8	59.2	50.3	64.5	55.3	29.7	21.2
w/o pre-train	41.9	34.4	26.5	20.0	54.4	41.6	63.2	54.1	34.2	25.9
Table 4: Ablation study. w/o pre-train refers to RTFE without preliminary training for static futures.
training. Pre-training provides a local optimum of all timestamps. In this case, it's not as good as the random initialization of the first timestamp
There are two kinds of facts in the TKGs: continuous facts and discrete events. Continuous facts have temporal attributes (since... is true, until... ends) like (Obama, President, United States, 2009-2016). And discrete events have temporal attributes (happens at... ) like (Obama, Inaugurated as, President, 2009).
In recent years, some work (Jiang et al., 2016a; Esteban et al., 2016; Tresp et al., 2017; Trivedi et al., 2017; Garcia-Duran et al., 2018; Jain et al., 2020; Ma et al., 2019; Xu et al., 2019; Jin et al., 2019; Liu et al., 2019; Wang and Li, 2019; Tang et al., 2020; Goel et al., 2019; Xu et al., 2020; Jain et al., 2020; Lacroix et al., 2020;) began to use the time information to improve the KG completion or directly complete the TKG. Based on the fact or event they dealt with, we state representative TKGE methods as follows.
(1) Event completion: DE (Goel et al., 2019) made the entity embedding into a function DEEMB that takes the time point as a variable. While DE transplanted SKG embedding methods to TKGs, it didn't involve recent GNN-based SKG embedding methods. TComplEx (Lacroix et al., 2020) presented an extension of Complex (Trouillon et al., 2016) by adding timestamp embedding into decomposition of tensors of order 4. ATiSE (Xu et al., 2019) incorporated time information into entity/relation representations by using Additive Time Series decomposition.
(2) Event prediction: (Esteban et al., 2016) trained an event prediction model by using background information provided by KG and recent events. RE-NET (Jin et al., 2019) models the event
sequence as a temporal joint probability distribution. The method is trained on historical data and then, by sampling from the probability distribution, predicts the events of the future timestamp graph. GHNN (Han et al., 2020) used Hawkes process to capture the dynamic of evolving graph sequences. Glean (Deng et al., 2020) incorporated both relational and world contexts to capture historical information.
(3) Continuous fact completion: (Jiang et al., 2016a; Jiang et al., 2016b) used the order of relations and temporal consistency constraints to improve completion but did not make the embedding space directly contain time information. (García-Durán et al., 2018) used RNN to learn the representation of temporal relations, but did not consider that the embedding of entities should also change over time. Hyte (Dasgupta et al., 2018) represented timestamps as hyperplanes, and projected the entities and relations onto these hyperplanes. Then, the facts of all timestamps are learned jointly using a translation-based score function.
We propose a framework RTFE for TKG completion. We have transplanted SKGE models to TKGs and enhance the performance of existing TKGE models. Experiments show that on five TKG datasets RTFE outperformed baselines and is extensible for future timestamps to some extent.
In the future, we will further deal with discrete events. Since events with adjacent timestamps are correlated, we plan to modify RTFE so that it can learn correlations (especially causality) of events. By modeling spatio-temporal dependency of TKG, events in future timestamps can be forecasted. Besides, we plan to deal with the task of predicting time validity of facts (Leblay and Chekol, 2018).
This work was supported in part by the National Science Foundation of China (Grant No. 61902034); Engineering Research Center of Information Networks, Ministry of Education.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787-2795.
Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. 2018. Hyte: Hyperplane-based temporally aware knowledge graph embedding. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2001-2011.
Songgaojun Deng, Huzefa Rangwala, and Yue Ning. 2020. Dynamic knowledge graph based multi-event forecasting. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1585-1595.
Cristóbal Esteban, Volker Tresp, Yinchong Yang, Stephan Baier, and Denis Krompaß. 2016. Predicting the co-evolution of event and knowledge graphs. In 2016 19th International Conference on Information Fusion (FUSION), pages 98-105. IEEE.
Alberto García-Durán, Sebastijan Dumančić, and Mathias Niepert. 2018. Learning sequence encoders for temporal knowledge graph completion. arXiv preprint arXiv:1809.03202.
Sankalp Garg, Navodita Sharma, Woojeong Jin, and Xiang Ren. 2020. Temporal attribute prediction via joint modeling of multi-relational structure evolution. arXiv preprint arXiv:2003.03919.
Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker, and Pascal Poupart. 2019. Diachronic embedding for temporal knowledge graph completion. arXiv preprint arXiv:1907.03143.
Zhen Han, Yunpu Ma, Yuyi Wang, Stephan Gunnemann, and Volker Tresp. 2020. Graph hawkes neural network for forecasting on temporal knowledge graphs. In 8th Automated Knowledge Base Construction (AKBC).
Jinjing Huang, Wei Chen, An Liu, Weiqing Wang, Hongzhi Yin, and Lei Zhao. 2020. Cluster query: a new query pattern on temporal knowledge graph. World Wide Web, 23(2):755-779.
Prachi Jain, Sushant Rathi, Soumen Chakrabarti, et al. 2020. Temporal knowledge base completion: New algorithms and evaluation protocols. arXiv preprint arXiv:2005.05035.
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Long papers), pages 687-696.
Tingsong Jiang, Tianyu Liu, Tao Ge, Lei Sha, Baobao Chang, Sujian Li, and Zhifang Sui. 2016a. Towards time-aware knowledge graph completion. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1715-1724.
Tingsong Jiang, Tianyu Liu, Tao Ge, Lei Sha, Sujian Li, Baobao Chang, and Zhifang Sui. 2016b. Encoding temporal information for time-aware link prediction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2350-2354.
Woojeong Jin, He Jiang, Meng Qu, Tong Chen, Changlin Zhang, Pedro Szekely, and Xiang Ren. 2019. Recurrent event network: Global structure inference over temporal knowledge graph. arXiv preprint arXiv:1904.05530.
Seyed Mehran Kazemi and David Poole. 2018. Simple embedding for link prediction in knowledge graphs. In Advances in neural information processing systems, pages 4284-4295.
Timothée Lacroix, Guillaume Obozinski, and Nicolas Usunier. 2020. Tensor decompositions for temporal knowledge base completion. arXiv preprint arXiv:2004.04926.
Julien Leblay and Melisachew Wudage Chekol. 2018. Deriving validity time in knowledge graph. In *Companion Proceedings of the The Web Conference* 2018, pages 1771-1776.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Y Liu, and X Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion, 2015. Google Scholar Google Scholar Digital Library Digital Library.
Yu Liu, Wen Hua, Kexuan Xin, and Xiaofang Zhou. 2019. Context-aware temporal knowledge graph embedding. In International Conference on Web Information Systems Engineering, pages 583-598. Springer.
Yunpu Ma, Volker Tresp, and Erik A Daxberger. 2019. Embedding models for episodic knowledge graphs. Journal of Web Semantics, 59:100490.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pages 593-607. Springer.
Avirup Sil and Silviu Cucerzan. 2014. Towards temporal scoping of relational facts based on wikipedia data. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 109-118.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197.
Xiaoli Tang, Rui Yuan, Qianyu Li, Tengyun Wang, Haizhi Yang, Yundong Cai, and Hengjie Song. 2020. Timespan-aware dynamic knowledge graph embedding by incorporating temporal evolution. IEEE Access, 8:6849-6860.
Volker Tresp, Yunpu Ma, Stephan Baier, and Yinchong Yang. 2017. Embedding learning for declarative memories. In European Semantic Web Conference, pages 202-216. Springer.
Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. 2017. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. arXiv preprint arXiv:1705.05742.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. International Conference on Machine Learning (ICML).
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Aaai, volume 14, pages 1112-1119. Citeseer.
Zhihao Wang and Xin Li. 2019. Hybrid-te: Hybrid translation-based temporal knowledge graph embedding. In 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pages 1446-1451. IEEE.
Yuting Wu, Xiao Liu, Yansong Feng, Zheng Wang, Rui Yan, and Dongyan Zhao. 2019. Relation-aware entity alignment for heterogeneous knowledge graphs. arXiv preprint arXiv:1908.08210.
Chengjin Xu, Mojtaba Nayyeri, Fouad Alkhoury, Jens Lehmann, and Hamed Shariat Yazdi. 2019. Temporal knowledge graph embedding model based on additive time series decomposition. arXiv preprint arXiv:1911.07893.
Chengjin Xu, Mojtaba Nayyeri, Fouad Alkhoury, Hamed Shariat Yazdi, and Jens Lehmann. 2020. Tero: A time-aware knowledge graph embedding via temporal rotation. arXiv preprint arXiv:2010.01029.
Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang. 2020. Learning hierarchy-aware knowledge graph embeddings for link prediction. In AAAI, pages 3065-3072.
Type	Dataset	#Entity	#Relation	#Timestamp	#Train	#Validation	#Test
Fact	YAGO11k	10623	10	60	203858	21763	21159
Wikida12k	12554	24	77	239928	18633	17616
Event	ICEWS14	7128	230	365	72826	8941	8963
ICEWS05-15	10488	251	4017	368962	46275	46092
GDELT	500	20	366	2735685	341961	341961
Table 5: Statistics on YAGO11k, Wikidata12k, ICEWS14, ICEWS05-15, and GDELT.
Model	Entity dim	Relation dim	Batchsize	Neg ratio	Leanring rate	Pre-train epochs	Epochs per time
TransE	300	300	#triplets/100	1	0.01	≤ 1000	≤ 200
TransD	50	50	200	1	0.0001	5000	200
RGCN	500	500×500	1400	10	0.001	≤ 6000	≈ 2000
RDGCN	150	300	Full batch	10;50	0.001	1000	100;200
RotatE	2000	1000	1024	256	0.0001	6000	300
DE-SimplE	100	100	512	500	0.001	500	100
TComplEx	256	256	1000	0	0.01	50	20
Table 6: Parameter settings of embedding models.
In YAGO11k and Wikidata12k, each fact was formatted as  $(s, r, o, \text{start_date}, \text{end_date})$ . Following the step of Hyte to divide the timestamps, we used a year with more than 300 occurrences as the timestamp boundary. So the timestamp could be formatted as (year, the next year with more than 300 occurrences). Then, the facts were divided into these timestamps to get the set of quadruples  $(s, r, o, t)$ , where  $t$  represents the timestamp of the fact (A original fact can appear in multiple adjacent timestamps after decomposition). The statistics of the five datasets are illustrated in Table 5.
B Examples of feature  $X$  in RTFE
Feature  $X$  refers to the input vectors of entities/relations. E.g., for RGCN (Schlichtkrull et al., 2018), feature  $X$  refers to the input hidden state  $h^{(0)}$  at layer 0; For TransE (Bordes et al., 2013), feature  $X$  refers to the embeddings.
C Parameter settings
The same parameters are used for the SKGE method and its corresponding RTFE version. Their main parameters are shown in Table 6. Besides, for TransE, the number of batches is set to 100. For RGCN, dropout ratio is set to 0.2; the number of GCN layer is set 2 and a res-net layer is added between the two RGCN layers. For RDGCN,  $\lambda$  is set to 0.2.