SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 2: Link prediction results on ICEWS14 and ICEWS05-15. *: results are taken from (García-Durán et al., 2018). ‰: results are taken from (Lacroix et al., 2020). Dashes: results are not reported in the responding literature. Other results are taken from the original papers. The best results among all models are written bold.
a time-aware link prediction query  $(s,r,?,T)$ , we first generate the candidate list  $\mathcal{C} = \{(s,r,o',T):o'\in \mathcal{E}\}$ . Following the time-wise filtered setting used in most previous TKGE-related work, e.g., TComplEx (Lacroix et al., 2020), we then remove the candidate quadruples appearing in the train  $\Omega_{\mathrm{train}}$ , valid  $\Omega_{\mathrm{train}}$  and test set  $\Omega_{\mathrm{train}}$  from the candidate list. The filtered candidate list is denoted as  $\overline{\mathcal{C}} = \{\omega :\omega \in \mathcal{C},\omega \notin \Omega_{\mathrm{train}}\cup \Omega_{\mathrm{valid}}\cup \Omega_{\mathrm{test}}\}$ . We get the rank of test quadruple  $(s,r,o,T)$  among the candidate quadruples  $\overline{\mathcal{C}}$  by sorting their scores. We use Mean Reciprocal Rank (MRR) and Hits@N as evaluation metrics. The Mean Reciprocal Rank (MRR) is the average of the reciprocal values of all computed ranks. The percentage of testing quadruples which are ranked lower than N is considered as Hits@N.
We compare our models with the state-of-the-art KGE model, ComplEx-N3 (Lacroix et al., 2018) and several existing TKGE approaches including TTransE (Leblay and Chekol, 2018), HyTE (Dasgupta et al., 2018), TA-TransE, TA-DistMult (García-Durán et al., 2018), ATiSE (Xu et al., 2019), TeRo (Xu et al., 2020a), DE-SimpleE (Goel et al., 2020), TIMEPLEX(base) (Jain et al., 2020) and TComplEx (Lacroix et al., 2020). We do not use the complete TIME-PLEX model and the TNTComplEx model as baselines since the former incorporates additional temporal constraints for some specific relations and the latter is designed for modelling a KG where some facts involve time information and others do not. Among the existing TKGE
approaches, TComplEx achieves state-of-the-art results on TKG completion.
5.5 Experimental Setup
We implement our proposed model TeLM in PyTorch. We use the Adagrad optimizer with a learning rate of 0.1 to train both models. The batch size  $b$  is fixed as 1000. The regularization weights  $\lambda_{\omega}$  and  $\lambda_{\mathcal{T}}$  are tuned in a range of  $\{0, 0.001, 0.0025, 0.005, 0.0075, 0.01, \ldots, 0.1\}$ . To avoid too much memory consumption, we follow the setting in (Lacroix et al., 2020) to make the maximum embedding no more than 2000. The above experimental setup is also used for evaluating TComplEx on YAGO11k and Wikidata12k. Notably, the time granularity parameters  $u$  and  $tr$  are also regraded as hyperparameters for TeLM as mentioned in the previous section. The optimal hyperparameters for TeLM are as follows:  $\lambda_{\omega} = 0.0075$ ,  $\lambda_{\mathcal{T}} = 0.01$ ,  $u = 1$  on ICEWS14;  $\lambda_{\omega} = 0.0025$ ,  $\lambda_{\mathcal{T}} = 0.1$ ,  $u = 1$  on ICEWS05-15;  $\lambda_{\omega} = 0.025$ ,  $\lambda_{\mathcal{T}} = 0.001$ ,  $tr = 100$  on YAGO11k;  $\lambda_{\omega} = 0.025$ ,  $\lambda_{\mathcal{T}} = 0.0025$ ,  $tr = 1$  on Wikidata12k. The optimal embedding dimension is  $k = 2000$  in all cases. The training processes of a TeLM model with  $k = 2000$  on ICEWS14, YAGO11K and Wikidata12k all cost less than half an hour with a GeForce RTX 2080 GPU. On ICEWS05-15, It takes about 2 hours to train a 2000-dimensional TeLM model.
6.1 Link Prediction
Table 2 and 3 list the link prediction results of our models and all baseline models on four TKG
YAGO11k	Wikidata12k
Metrics	MRR	Hits@1	Hits@3	Hits@10	MRR	Hits@1	Hits@3	Hits@10
ComplEx-N3†	.181	.115	-	.311	.248	.143	-	.489
TTransE°	.108	.020	.150	.251	.172	.096	.184	.329
HyTE†	.136	.033	-	.298	.253	.147	-	.483
TA-DistMult†	.155	.098	-	.267	.230	.130	-	.461
TeRo	.187	.121	.197	.319	.299	.198	.329	.507
ATiSE	.185	.126	.189	.301	.252	.148	.288	.462
TIME-PLEX(base)	.184	.110	-	.319	.324	.220	-	.528
TComplEx△	.185	.127	.183	.307	.331	.233	.357	.539
TeLM	.191	.129	.194	.321	.332	.231	.360	.542
Table 3: Link prediction results on YAGO11k and Wikidata12k. †: results are taken from (Jain et al., 2020). °: results are taken from (Xu et al., 2019). △: results are obtained from our experiments. Dashes: results are not reported in the responding literature. Other results are taken from the original papers. The best results among all models are written bold.
datasets. As shown in Table 2, TeLM surpasses all baseline models on ICEWS datasets regarding all metrics. Compared to TComplEx, TeLM obtains the improvements of 1.5 MRR points on ICEWS14 and 1.8 MRR points on ICEWS05-15.
TA-TransE is not included in Table 3 since there is no literature reporting the results of TA-TransE on YAGO11k and Wikidata12k and the performances of TA-TransE are worse than most baseline models on other TKG datasets. The results of DE-SimplE on YAGO11k and Wikidata12k can not be obtained since DE-SimplE mainly focuses on event-based datasets and cannot model time intervals or time annotations missing moth and day information which are common in YAGO and Wikidata. On YAGO11k, TeLM outperforms all baseline models other than TA-TransE and DE-SimplE regarding MRR, Hits@1 and Hits@10, though performs slightly worse than TeRo on Hits@3. Additionally, TeLM also achieves the state-of-the-art results except the the Hits@1 of TComplEx is 0.1 point higher than TeLM.
6.2 Effect of Linear Temporal Regularizer
We compare the performances of the TeLM model trained with various temporal regularizers mentioned before, e.g., the smoothing temporal regularizer, the projective temporal regularizer, the 3-order autoregressive temporal regularizer, and our proposed linear temporal regularizer. As shown in Figure 2, the TeLM model trained with the linear temporal regularizer outperforms the TeLM model trained with other temporal regularizer on ICEWS14. Compared to the smoothing temporal regularizer, the linear temporal regularizer improves MRR by 0.2 point and Hits@1 by 0.3 point.
Figure 2: Results of TeLM trained with different temporal regularizers on ICEWS14.
And the linear temporal regularizer is also less sensitive to the temporal regularization weight  $\lambda_{\mathcal{T}}$  amongst the range of  $\{0.001, \dots, 0.1\}$  since its bias component is learned during the training process and thus can be partly adaptive to different  $\lambda_{\mathcal{T}}$ .
In Figure 3, In we show 2-d PCA projections of the 2000-dimensional time embeddings of TeLM models trained with/without a linear temporal regularizer. Adjacent time embeddings of TeLM trained without the temporal regularization naturally come together. However, the time embeddings representing time points in different months are not well divided. By contrast, time embeddings of TeLM trained with the linear temporal regularizer are forming good clusters in chronological order. Overall, the linear temporal regularizer provides good geometric meanings of time embeddings by effectively retaining the time sequence information in temporal KGs and thus improves the performances
Figure 3: The figure illustrates 2-d PCA projection of the 2000 dimensional time embeddings which are obtained after training TeLM on ICEWS14 with a smoothing temporal regularizer and a linear temporal regularizer. Time points in different months are represented with different colors
(d)
Figure 4: Results of TeLM with different time granularities and embedding dimensions on TKGs.
of TeLM.
6.3 Effect of Time Granularity and Embedding Dimension
In this work, we analyze the effect of the change of the time granularity on the performance of our model. As mentioned in the previous section, we adopt two different time-division approaches for event-based datasets, i.e., ICEWS datasets, and time-wise KGs involving time intervals, i.e., YAGO11k as well as Wikidata12k. As shown in Figure 4(a), on ICEWS14 where time distribution of facts is relatively uniform, the performance of TeLM decreases with the time unit  $u$  increasing, since representing time with a small time granularity can provide more abundant time information.
On the other hand, Figure 4(b) illustrates that using the smallest time granularity is non-optimal for YAGO11k due to the long-tail property of time data. An appropriate minimum threshold used for generating time steps, e.g.,  $tr = 100$ , can improve the link prediction results of TeLM by alleviating the effect of the long-tail property of time data and decrease the memory usage with fewer time steps. Meanwhile, using overly coarse-grained time units always leads to low performances since the time information is not fully expressed in these cases.
Figure 4(c) and (d) show that the performances on ICEWS14 and YAGO11k of TGeomE2 improve with the increasing of the embedding dimension in a range of  $k = \{20,50,100,200,500,1000,2000\}$ . TeLM with
$k = 500$  has fewer adjustable parameters than TComplEx with  $k = 1740$  used in (Lacroix et al., 2020) but performs closely (0.612 vs 0.61 on MRR). It will still be interesting to explore the performances of TeLM models with higher-dimensional embeddings, e.g., Ebisu et al. (2018) use 10000-dimensional embeddings for TorusE, although it would bring more memory pressure.
We propose a new time-aware approach for TKG completion, TeLM, which performs 4th-order tensor factorization of a temporal knowledge graph using multivector embeddings for knowledge graph representation and a linear temporal regularizer for learning time embeddings. Compared to real-valued and complex-valued embeddings, multivector embeddings provide better generalization capacity and richer expressiveness with higher degree of freedom for TKGE. Moreover, the linear temporal regularizer provides better geometric meanings for time embeddings and improves the performances of TeLM compared to the temporal smoothness. Additionally, two time division methods are used for different types of TKG datasets to study the effect of the time granularity on TKG completion. Our proposed models trained with the linear temporal regularizer achieve the state-of-the-art results on time-wise link prediction over four well-known datasets involving various forms of time information, e.g., time points, begin or end time, and time intervals. Experimental results also show that choosing a reasonable time division method with an appropriate time granularity is helpful for TKG completion.
This work is supported by the EC Horizon 2020 grant LAMBDA (GA no. 809965), the CLEOPA-TRA project (GA no. 812997) and the China Scholarship Council (CSC).
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. The semantic web, pages 722-735.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM
SIGMOD international conference on Management of data, pages 1247-1250. AcM.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems, pages 2787-2795.
Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. 2018. Hyte: Hyperplane-based temporally aware knowledge graph embedding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2001-2011.
Takuma Ebisu and Ryutaro Ichise. 2018. Toruse: Knowledge graph embedding on a lie group. In Thirty-Second AAAI Conference on Artificial Intelligence.
Fredo Erxleben, Michael Gunther, Markus Krötzsch, Julian Mendez, and Denny Vrandecic. 2014. Introducing wikidata to the linked data web. In International Semantic Web Conference, pages 50-65. Springer.
Alberto García-Durán, Sebastijan Dumančić, and Mathias Niepert. 2018. Learning sequence encoders for temporal knowledge graph completion. arXiv preprint arXiv:1809.03202.
Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker, and Pascal Poupart. 2020. Diachronic embedding for temporal knowledge graph completion. In AAAI.
Prachi Jain, Sushant Rathi, Soumen Chakrabarti, et al. 2020. Temporal knowledge base completion: New algorithms and evaluation protocols. arXiv preprint arXiv:2005.05035.
Seyed Mehran Kazemi and David Poole. 2018. Simple embedding for link prediction in knowledge graphs. In Advances in neural information processing systems, pages 4284-4295.
Timothee Lacroix, Guillaume Obozinski, and Nicolas Usunier. 2020. Tensor decompositions for temporal knowledge base completion. arXiv preprint arXiv:2004.04926.
Timothée Lacroix, Nicolas Usunier, and Guillaume Obozinski. 2018. Canonical tensor decomposition for knowledge base completion. International Conference on Machine Learning (ICML).
Jennifer Lautenschlager, Steve Shellman, and Michael Ward. 2015. Icews event aggregations.
Julien Leblay and Melisachew Wudage Chekol. 2018. Deriving validity time in knowledge graph. In *Companion of the The Web Conference* 2018 on The Web Conference 2018, pages 1771-1776. International World Wide Web Conferences Steering Committee.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference on artificial intelligence.
Farzaneh Mahdisoltani, Joanna Biega, and Fabian M Suchanek. 2013. Yago3: A knowledge base from multilingual wikipedias. In CIDR.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
Mojtaba Nayyeri, Chengjin Xu, Sahar Vahdati, Nadezhda Vassilyeva, Emanuel Sallinger, Hamed Shariat Yazdi, and Jens Lehmann. 2020. Fantastic knowledge graph embeddings and how to find the right space for them. In International Semantic Web Conference, pages 438-455. Springer.
Mojtaba Nayyeri, Chengjin Xu, Yadollah Yaghoobzadeh, Hamed Shariat Yazdi, and Jens Lehmann. 2019. Toward understanding the effect of loss function on the performance of knowledge graph embedding.
Uriel Singer, Ido Guy, and Kira Radinsky. 2019. Node embedding over temporal graphs. arXiv preprint arXiv:1903.08889.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. arXiv preprint arXiv:1606.06357.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):2724-2743.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In AAAI, pages 1112-1119. CiteSeer.
Chengjin Xu, Mojtaba Nayyeri, Fouad Alkhoury, Jens Lehmann, and Hamed Shariat Yazdi. 2019. Temporal knowledge graph embedding model based on additive time series decomposition. arXiv preprint arXiv:1911.07893.
Chengjin Xu, Mojtaba Nayyeri, Fouad Alkhoury, Hamed Shariat Yazdi, and Jens Lehmann. 2020a. Tero: A time-aware knowledge graph embedding via temporal rotation. arXiv preprint arXiv:2010.01029.
Chengjin Xu, Mojtaba Nayyeri, Yung-Yu Chen, and Jens Lehmann. 2020b. Knowledge graph embeddings in geometric algebras. arXiv preprint arXiv:2010.00989.
Yonghui Xu, Shengjie Sun, Yuan Miao, Dong Yang, Xiaonan Meng, Yi Hu, Ke Wang, Hengjie Song, and Chuanyan Miao. 2020c. Time-aware graph embedding: A temporal smoothness and task-oriented approach. arXiv preprint arXiv:2007.11164.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575.
Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon. 2016. Temporal regularized matrix factorization for high-dimensional time series prediction. In Advances in neural information processing systems, pages 847-855.
Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019. Quaternion knowledge graph embeddings. In Advances in Neural Information Processing Systems, pages 2731-2741.