SlowGuess/NAACL_2021_OCR huggingface dataset:
Chengjin Xu
University of Bonn / Germany
xuc@iai.uni-bonn.de
Yung-Yu Chen
s6ynchen@uni-bonn.de
Mojtaba Nayyeri
nayyeri@iai.uni-bonn.de
Fraunhofer IAIS/ Germany
jens.lehmann@iais.fraunhofer.de
Representation learning approaches for knowledge graphs have been mostly designed for static data. However, many knowledge graphs involve evolving data, e.g., the fact (The President of the United States is Barack Obama) is valid only from 2009 to 2017. This introduces important challenges for knowledge representation learning since the knowledge graphs change over time. In this paper, we present a novel time-aware knowledge graph embedding approach, TeLM, which performs 4th-order tensor factorization of a Temporal knowledge graph using a Linear temporal regularizer and Multivector embeddings. Moreover, we investigate the effect of the temporal dataset's time granularity on temporal knowledge graph completion. Experimental results demonstrate that our proposed models trained with the linear temporal regularizer achieve the state-of-the-art performances on link prediction over four well-established temporal knowledge graph completion benchmarks.
Numerous large-scale knowledge graphs (KGs) including DBpedia (Auer et al., 2007), FreeBase (Bollacker et al., 2008) and WordNet (Miller, 1995) have been established in recent years. Such KGs abstract knowledge from the real world into a complex network graph consisting of billions of triples. Each triple is denoted as  $(s,r,o)$ , where  $s$  is the subject entity,  $o$  is the object entity, and  $r$  is the relation between the entities.
Knowledge graph completion is one of the main challenges in the KG field since most KGs are incomplete. To tackle this problem, knowledge graph embedding (KGE) approaches embed entities and relations into a low-dimensional embedding space
and measure the plausibility of triples by inputting embeddings of the entities and their relation to a score function (Wang et al., 2017). For instance, ComplEx (Trouillon et al., 2016) has been proven to be a highly effective KGE model, where entities and relations are represented as complex embeddings, and the score of a triple  $(s,r,o)$  is computed with the asymmetric Hermitian dot product.
Some KGs involve temporal facts, e.g., the triple (Barack Obama, presidentOf, USA) is only valid in a specific time period [2009, 2017]. Temporal KGs like Wikidata (Erxleben et al., 2014), YAGO3 (Mahdisoltani et al., 2013) and ICEWS (Lautenschlager et al., 2015) incorporate time information into triples. Triples attached with time information are represented as quadruples, shaped like  $(s,r,o,T)$ , where  $T$  denotes the times-tamp. Traditional KGE models disregard time information, leading to an ineffectiveness of performing link prediction on TKGs involving temporary relations, e.g., (?, presidentOf, USA, 2010). Recent researches show that the temporal knowledge graph embedding (TKGE) models, which encode time information in their embeddings, have better performances on link prediction over TKGs than traditional KGE models (Dasgupta et al., 2018; García-Durán et al., 2018; Xu et al., 2019; Goel et al., 2020; Lacroix et al., 2020).
In this paper, we present a novel temporal KG embedding approach TeLM. We move beyond the complex-valued representations and introduce more expressive multivector embeddings from 2-grade geometric algebras to model entities, relations, and timestamps for TKGE. At a high level, our approach performs 4th-order tensor factorization of a temporal KG, using the asymmetric geometric product. The geometric product provides a
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2569-2578 June 6-11, 2021. ©2021 Association for Computational Linguistics
greater extent of expressiveness compared to the complex Hermitian operator.
Specially, each relation is represented as a pair of dual multivector embeddings used to handle the beginning and the end of the relation. In this way, TeLM can adapt well to datasets where time annotations are represented in the various forms: time points, begin or end time, time intervals.
Moreover, we develop a new linear temporal regularization function for time representation learning which introduces a bias component in the temporal smoothing function and empirically study the effect of the time granularity for a TKG dataset on the performance of our models.
Experimental results on four well-established TKG datasets show that our approach outperforms the state-of-the-art TKGE models, and the linear temporal regularization function improves the performance of our model compared to three common temporal regularization functions.
Tensor decomposition-based KGE approaches have led to good results in static KG completion. Such approaches (Yang et al., 2014; Trouillon et al., 2016; Kazemi and Poole, 2018; Zhang et al., 2019; Xu et al., 2020b) model a static KG as a low-dimensional 3rd-order tensor and consider knowledge graph completion as a tensor decomposition problem. A typical tensor decomposition model ComplEx (Trouillon et al., 2016) has been proven to be fully expressive with complex embeddings. Apart from tensor decomposition approaches, distance-based KGE models are also commonly used for KG completion. However, distance-based KGE models like TransE (Bordes et al., 2013) and its variants (Wang et al., 2014; Lin et al., 2015; Nayyeri et al., 2019, 2020) have been proven to have limitations in modeling various relation patterns which does not lead to the state-of-the-art results on the current benchmarks.
The above KGE approaches achieve satisfactory results on link prediction over static KGs. Recent research on TKG completion shows that the inclusion of time information can improve the performances of KGE models on TKGs. TTransE (Leblay and Chekol, 2018), HyTE (Dasgupta et al., 2018), ATiSE and TeRo (Xu et al., 2019, 2020a) propose scoring functions which incorporate time representations into a distance-based score function in different ways. Further
more, RTGE (Xu et al., 2020c) introduces the concept of temporal smoothness to optimize and learn the hyperplanes of adjacent time intervals jointly on the basis of HyTE. García-Durán et al. (2018) utilize recurrent neural networks to learn time-aware representations of relations and use standard scoring functions from the existing KG embedding model, e.g. TransE and DistMult. DE-SimplE (Goel et al., 2020) uses diachronic entity embeddings to represent entities at different time steps and exploit the same score function as SimplE to score the plausibility of a quadruple. TIMEPLEX (Jain et al., 2020) and TComplEx (Lacroix et al., 2020) extend the time-agnostic ComplEx model in different ways. Among them, TComplEx performs a 4th-order tensor decomposition of a TKG using the quadraticomial Hermitian product which involves the embedding of timestamp  $T$ . Similarly to RTGE, TComplEx also uses the temporal smoothness to improve its performance. Thanks to the strong expressiveness provide by the complex embeddings and the 4th-order tensor decomposition, TComplEx achieves state-of-the-art results on TKG completion.
3 Geometric Algebras
In this section, we provide a brief introduction to the 2-grade Geometric Algebra  $\mathbb{G}^2$ . The contents are sufficient to understand the rest of the work.
Members of  $\mathbb{G}^2$  are called 2-grade multivectors. The multivector space  $\mathbb{G}^2$  is built with vectors from the vector space  $\mathbb{R}^2$ . Let  $\{e_1, e_2\}$  be an orthonormal basis of  $\mathbb{R}^2$ . The multivector space  $\mathbb{G}^2$  is based on two rules:  $e_1e_1 = e_2e_2 = 1$  and  $e_1e_2 = -e_2e_1$ . The multivector space  $\mathbb{G}^2$  is 4-dimensional with basis:
1 spans 0-vectors, scalars,
$$
\left\{e _ {1}, e _ {2} \right\} \text {s p a n s 1 - v e c t o r s , v e c t o r s , a n d}
$$
$$
\left\{e _ {1} e _ {2} \right\} \text {s p a n s 2 - v e c t o r s , b i v e c t o r s .}
$$
A 2-grade multivector  $M \in \mathbb{G}^2$  can be written as  $M = a_0 + a_1e_1 + a_2e_2 + a_{12}e_1e_2$ . Noteworthy, the unit bivectors from  $\mathbb{G}^2$  has similar algebraic properties as the imaginary unit  $\mathbf{i}$ , i.e.,
$$
(e _ {1} e _ {2}) ^ {2} = - e _ {1} e _ {1} e _ {2} e _ {2} = - 1 = \mathbf {i} ^ {2}.
$$
Thus, the complex numbers  $C \in \mathbb{C}$  can be embedded into a subalgebra of  $\mathbb{G}^2$  which are formed with scalars and bivectors. In other words, a 2-grade multivector  $M = a_0 + a_{12}e_1e_2$  consisting of a
scalar plus a bivector is isomorphic to a complex number  $C = a_0 + a_{12}\mathbf{i}$ .
The norm of a multivector  $M \in \mathbb{G}^2$  is equal to the root of the square sum of real values of its all elements. Taking a 2-grade multivector as an example, its norm is defined as:  $||M|| = \sqrt{a_0^2 + a_1^2 + a_2^2 + a_{12}^2}$ .
Geometric algebra also introduces a new product geometric product denoted as  $\times_{n}$  where  $n$  is the grade of multivectors, as well as three multivector involutions, space inversion, reversion and Clifford conjugation.
The geometric product of two 2-grade multivectors comprises of multiplications between scalars, vectors and bivectors. The product of two 2-grade multivectors  $M_{a} = a_{0} + a_{1}e_{1} + a_{2}e_{2} + a_{12}e_{1}e_{2}$  and  $M_{b} = b_{0} + b_{1}e_{1} + b_{2}e_{2} + b_{12}e_{1}e_{2}$  from  $\mathbb{G}^2$  is equal to
$$
\begin{array}{l} M _ {a} \times_ {2} M _ {b} = a _ {0} b _ {0} + a _ {1} b _ {1} + a _ {2} b _ {2} - a _ {1 2} b _ {1 2} \\ + \left(a _ {0} b _ {1} + a _ {1} b _ {0} - a _ {2} b _ {1 2} + a _ {1 2} b _ {2}\right) e _ {1} + \tag {1} \\ \end{array}
$$
$$
\begin{array}{l} \left(a _ {0} b _ {2} + a _ {1} b _ {1 2} + a _ {2} b _ {0} - a _ {1 2} b _ {1}\right) e _ {2} \\ + \left(a _ {0} b _ {1 2} + a _ {1} b _ {2} - a _ {2} b _ {1} + a _ {1 2} b _ {0}\right) e _ {1} e _ {2}, \\ \end{array}
$$
The Clifford conjugation of a 2-grade multivector  $M$  is a subsequent composition of space inversion  $M^{*}$  and reversion  $M^{\dagger}$  as  $\overline{M} = M^{\dagger *}$ , where space inversion  $M^{*}$  is obtained by changing  $e_i$  to  $-e_i$  and reversion is obtained by reversing the order of all products i.e. changing  $e_1e_2$  to  $e_2e_1$ . Thus, the Clifford conjugation of an 2-grade multivector  $M_{a} = a_{0} + a_{1}e_{1} + a_{2}e_{2} + a_{12}e_{1}e_{2}$  is computed as  $\overline{M_a} = a_0 - a_1e_1 - a_2e_2 - a_{12}e_1e_2$ . Note that the product of a multivector  $M_{a}$  and its conjugation  $\overline{M}_2$  is a scalar, i.e., given a 2-grade multivector  $M_{a} = a_{0} + a_{1}e_{1} + a_{2}e_{2} + a_{12}e_{1}e_{2}$ , we have
$$
M _ {a} \times_ {2} \overline {{M _ {a}}} = a _ {0} ^ {2} - a _ {1} ^ {2} - a _ {2} ^ {2} + a _ {1 2} ^ {2}, \tag {2}
$$
producing a real number.
4.1 TeLM Model
Let  $\mathcal{E}$  denote the set of entities,  $\mathcal{R}$  denote the set of relations. A TKG denoted as  $\Omega$  is a collection of numerous quadruples shaped like  $(s,r,o,T)$  where  $s,o\in \mathcal{E},r\in \mathcal{R}$  and  $T$  denotes the timestamp. The timestamp  $T$  can be represented as various forms, e.g., a time interval  $[t_b,t_e]$ , a begin time  $[t_s, - ]$  or an end time  $[- , t_{e}]$  and a time point  $t$ . A time point  $t$  can be denoted as a special time interval  $[t_b,t_e]$
where  $t = t_b = t_e$ . We extend the relation set  $\mathcal{R}$  of a TKG to a pair of dual relation sets,  $\mathcal{R}_b$  and  $\mathcal{R}_e$ . A relation  $r_b \in \mathcal{R}_b$  is used to handle the begin of relation  $r$ , meanwhile a relation  $r_e \in \mathcal{R}_e$  is used to handle the end of relation  $r$ . By doing this, we score a fact  $(s, r, o, [t_b, t_e])$  as the mean value of scores of two quadruples,  $(s, r_b, o, t_b)$  and  $(s, r_e, o, t_e)$  which represent the begin and the end of this fact respectively, i.e.,  $f(s, r, o, [t_b, t_e]) = \frac{1}{2} (f(s, r_b, o, t_b) + f(s, r_e, o, t_e))$ . For a fact missing the begin time or the end time, e.g.,  $(s, r, o, [t_b, -])$  or  $(s, r, o, [-, t_e])$ , the score of this fact is equal to the score of the quadruple involving the known time, i.e.,  $f(s, r, o, [t_b, -]) = f(s, r_b, o, t_b)$ ,  $f(s, r, o, [-, t_e]) = f(s, r_e, o, t_e)$ . We construct a set of time steps  $\mathcal{T}$  for a TKG. For any time  $t$  appearing in the TKG, we can find a time step  $\tau \in \mathcal{T}$  to represent  $t$ . The time set  $\mathcal{T}$  changes with time granularity of the TKG.
Our approach TeLM embeds a TKG in a multiple-dimensional 2-grade multivector space  $\mathbb{G}^{2\times k}$  where  $k$  is the dimension of embeddings, and score a quadruple with an element-wise geometric product. TeLM embeds each entity, relation and time step as a  $k$ -dimensional 2-grade multivector embedding  $\mathbf{M}$  where each component is a multivector, i.e.,  $\mathbf{M} = [M_1,\dots ,M_k],i = 1,\dots ,k,M_i\in$ $\mathbb{G}^2$  . We can define the score function of TeLM as
$$
f (s, r, o, t) = \left\langle \mathbf {S c} \left(\mathbf {M} _ {s} \otimes_ {2} \mathbf {M} _ {r \tau} \otimes_ {2} \overline {{\mathbf {M} _ {o}}}\right), \mathbf {1} \right\rangle , \tag {3}
$$
where  $\tau$  is the time step corresponding to time  $t$ ,  $\mathbf{M}_{r\tau} = \mathbf{M}_r\otimes_2\mathbf{M}_\tau$ ,  $\mathbf{M}_s$ ,  $\mathbf{M}_r$ ,  $\mathbf{M}_o$  and  $\mathbf{M}_{\tau}$  denote the  $k$ -dimensional multivector embeddings of  $s$ ,  $r$ ,  $o$  and  $\tau$  respectively.  $\otimes_{2}$  denotes the elementwise geometric product between 2-grade multivector embeddings, e.g.,  $\mathbf{M}_r\otimes_2\mathbf{M}_{\tau} = [M_{r_1}\times_2 M_{\tau_1},\dots ,M_{r_k}\times_2 M_{\tau_k}]$ .  $\mathbf{Sc}(\cdot)$  denotes the real-valued vector of the scalar component of a multivector embedding,  $\mathbf{1}$  denotes a  $k\times 1$  vector having all  $k$  elements equal to one,  $\overline{\mathbf{M}}$  denotes the element-wise conjugation of multivectors i.e.  $\overline{\mathbf{M}} = [\overline{M}_1,\ldots ,\overline{M}_k]$ . and  $\langle a,b\rangle \coloneqq \sum_{k}a_{k}b_{k}$  is the dot product.
In our approach, the total number of parameters increases linearly with embedding dimension  $k$ , i.e., the space complexity of a TeLM model is  $\mathcal{O}(k)$ . Since the score is computed with an asymmetric quadranomial geometric product between  $k$ -dimensional multivector embeddings, the time complexity is also equal to  $\mathcal{O}(k)$ , which are the same as some common KGE models, e.g., TransE and DistMult.
Using full multiclass log-softmax loss function and N3 regularization has been proven to be helpful in boosting the performances of tensor decomposition-based (T)KGE models (Lacroix et al., 2018; Xu et al., 2020b; Lacroix et al., 2020; Jain et al., 2020). In this work, we follow such setting for TeLM and utilize the reciprocal learning for simplifying the training process.
For each relation  $r$ , we create an inverse relation  $r^{-1}$  and create a quadruple  $(o, r^{-1}, s, t)$  for each training quadruple  $(s, r, o, t)$ . At the evaluation phase, queries of the form  $(?, r, o, t)$  are answered as  $(o, r^{-1},?, t)$ . By doing this, the multiclass log-loss of a training quadruple  $\omega = (s, r, o, t)$  can be defined as follows,
$$
\begin{array}{l} \mathcal {L} _ {\omega} = - \log \left(\frac {\exp (f (s , r , o , t))}{\sum_ {s ^ {\prime} \in \mathcal {E}} \exp (f (s ^ {\prime} , r , o , t))}\right) \\ - \log \left(\frac {\exp \left(f \left(o , r ^ {- 1} , s , t\right)\right)}{\sum_ {o ^ {\prime} \in \mathcal {E}} \exp \left(f \left(o ^ {\prime} , r ^ {- 1} , s , t\right)\right)}\right) \tag {4} \\ + \lambda_ {\omega} \sum_ {i = 1} ^ {k} (| | M _ {s _ {i}} | | _ {3} ^ {3} + | | M _ {r \tau_ {i}} | | _ {3} ^ {3} + | | M _ {o _ {i}} | | _ {3} ^ {3}), \\ \end{array}
$$
where  $\lambda_{\omega}$  denotes the N3 regularization weight.
4.3 Temporal Regularization
A common approach to leverage the temporal aspect of temporal graphs is to use time as a regularizer to impose a smoothness constraint on time embeddings. RTGE (Xu et al., 2020c) and TComplEx (Lacroix et al., 2020) introduce the temporal smoothness between hyperplanes and embeddings of adjacent time steps, respectively, based on the assumption that the neighboring time steps should have close representations. The smoothing temporal regularizer is defined as,
$$
\mathcal {L} _ {\mathcal {T}} = \sum_ {i = 1} ^ {n _ {\tau} - 1} \left| \left| \mathbf {M} _ {\tau_ {i + 1}} - \mathbf {M} _ {\tau_ {i}} \right| \right| _ {p} ^ {p}, \tag {5}
$$
where  $n_{\tau}$  is the number of time steps and  $p = 3$  in this work since we use N3 regularization.
Apart from the basic temporal smoothness, various temporal regularization methods are used for learning temporal embeddings. Singer et al. (2019) add a rotation projection to align the neighboring temporal embeddings. The loss of such projective temporal regularization can be defined as,
$$
\mathcal {L} _ {\mathcal {T}} = \sum_ {i = 1} ^ {n _ {\tau} - 1} \left| \left| \mathbf {M} _ {\tau_ {i + 1}} - \mathbf {M} _ {w} \otimes_ {2} \mathbf {M} _ {\tau_ {i}} \right| \right| _ {p} ^ {p}, \tag {6}
$$
where  $\mathbf{M}_w$  is the rotation embedding. Yu et al. (2016) propose an autoregressive temporal regularizer based on the assumption that the change of temporal embeddings fits an AR model. This autoregressive temporal regularizer is defined as,
$$
\mathcal {L} _ {\mathcal {T}} = \sum_ {i = 1} ^ {n _ {\tau} - m} \left| \left| \mathbf {M} _ {\tau_ {i + m}} - \sum_ {j = 0} ^ {m - 1} \mathbf {M} _ {j} \otimes_ {2} \mathbf {M} _ {\tau_ {i + j}} \right| \right| _ {p} ^ {p}, \tag {7}
$$
where  $m = 3$  is the order of the AR model used in our work, and  $\mathbf{M}_j$  denote the weight of the embeddings of previous time steps which are learned during the training process.
In this work, we develop a novel linear temporal regularizer by adding a bias component between the neighboring temporal embeddings, which can be defined as,
$$
\mathcal {L} _ {\mathcal {T}} = \sum_ {i = 1} ^ {n _ {\tau} - 1} | | \mathbf {M} _ {\tau_ {i + 1}} - \mathbf {M} _ {\tau_ {i}} - \mathbf {M} _ {b} | | _ {p} ^ {p}. \tag {8}
$$
where  $\mathbf{M}_b$  denotes the bias embedding which are randomly initialized and then learned from the training process. This linear regularizer promotes that the difference between embeddings of two adjacent time steps is smaller than the difference between embeddings of two distant time steps, i.e.,  $||\mathbf{M}_{\tau_{i + m}} - \mathbf{M}_{\tau_i}|| > ||\mathbf{M}_{\tau_{i + 1}} - \mathbf{M}_{\tau_i}||$  when  $m\gg 1$ . This formulation can be helpful for effectively clustering and ordering time embeddings  $\mathbf{M}_{\tau_i}$ .
The total loss  $\mathcal{L}_b$  of a training batch  $\Omega_{b}$  is the sum of the quadruple loss and the temporal regularization term, i.e.,
$$
\mathcal {L} _ {b} = \frac {1}{b} \sum_ {\omega \in \Omega_ {b}} \mathcal {L} _ {\omega} + \lambda_ {\mathcal {T}} \mathcal {L} _ {\mathcal {T}}. \tag {9}
$$
where  $\lambda_{\mathcal{T}}$  denotes the coefficient of the temporal regularizer. In this work, we use the linear temporal regularizer for TeLMand compare its performance with other three temporal regularizers.
To compare our model with baselines, we used the following three datasets, namely ICEWS14, ICEWS05-15, and YAGO11k, released by (Das-gupta et al., 2018) and (García-Durán et al., 2018).
ICEWS14 and ICEWS05-15 are the two most common TKG benchmarks extracted from the large-scale event-based database, Integrated Crisis Early Warning System (ICEWS) (Lautenschlager
Dataset	#Entities	#Relations	Period(year)	#Train	#Valid	#Test
ICEWS14	6,869	230	201	72,826	8,941	8,963
ICEWS05-15	10,094	251	2005-2015	368,962	46,275	46,092
YAGO11k	10,623	10	-431-2844	16,406	2,050	2,051
Wikidata12k	12,554	24	19-2020	32,497	4,062	4,062
et al., 2015). ICEWS is a repository that contains political events with specific time annotations, e.g. (Barack Obama, Make a visit, Ukraine, 2014-07-08). It is noteworthy that time annotations in ICEWS are all time points. ICEWS14 contains events in 2014, and ICEWS05-15 contains events occurring between 2005-2015. These two datasets are filtered by only selecting the most frequently occurring entities in the graph.
YAGO3 (Mahdisoltani et al., 2013) and Wikidata (Erxleben et al., 2014) are two temporal KGs where time annotations are represented in various forms, i.e., time points like [2003-01-01, 2003-01-01], beginning or end time like [2003, ##], and time intervals like [2003, 2005]. YAGO15k, Wikidata11k (García-Durán et al., 2018), YAGO11k and Wikidata12k (Dasgupta et al., 2018) are subsets of YAGO3 and Wikidata. In YAGO15k and Wikidata11k, time information is represented as either begin time or end time of each fact and some facts do not include time annotations. In this paper, we focus on performing link prediction on time-aware facts where time annotations are represented as various forms. Based on this consideration, we use YAGO11k and Wikidata12k as datasets, where all of facts involve time annotations.
The statics of datasets are listed in Table 1. All datasets can be downloaded from https: //github.com/soledad921/ATISE.
5.2 Time granularity
In the previous work (García-Durán et al., 2018; Goel et al., 2020; Lacroix et al., 2020), the time granularity of ICEWS14 and ICEWS05-15 was set as 1 day. For YAGO11k and Wikidata12k, Das-gupta et.al (2018) and Xu et.al (2019) dropped the month and day information. They took care of the unbalance that might occur in terms of number of facts in a particular interval by clubbing neighboring years which are less frequently mentioned into the same time step and applying a minimum threshold of 300 facts per interval during construction. To illustrate, in Wikidata12k, there were time steps like [1596-1777], [1791-1815] with a large span as
the facts occurring on those years were relatively less in KG. The years like 2013 being highly frequent were self-contained. This setting was used to alleviate the effect of the long-tail property of time data in YAGO11k and Wikidata12k. As shown in Figure 1, the time distribution of facts in ICEWS14 is relatively uniform, while the frequency distribution of time data in YAGO11k has a long tail.
Figure 1: Time distribution of facts in TKGs.
In this work, we study the effect of time granularity on TKG completion. For ICEWS datasets, we test our model with different time units, denoted as  $u$ , in a range of  $\{1, 2, 3, 7, 14, 30, 90 \text{ and } 365\}$  days. Dasgupta et al. (2018) and Xu et al. (2019) applied a minimum threshold of 300 triples per interval during construction for YAGO11k and Wikidata12k. We follow their time-division approaches for these two datasets and test different minimum thresholds, denoted as  $tr$ , amongst  $\{1, 10, 100, 1000, 10000\}$  for grouping years into different time steps. The change of time granularity will reconstruct the set of time steps  $\mathcal{T}$ . To illustrate, the total number of time steps in ICEWS14 is 365 with  $u = 1$ . When the time unit  $u$  changes from 1 to 2, the set of time steps  $\mathcal{T}$  will be reconstructed and include 188 different time steps. In YAGO11k, there are totally 388 different time steps when  $tr = 1$ . Years like -453, 100 and 2008 are taken as independent time steps. When  $tr$  for YAGO11k rises to 100, the number of time steps drops to 118 and years between -431 and 100 are clubbed into a same time step.
We evaluate our models on link prediction over the above-mentioned TKG benchmarks. To perform
ICEWS14	ICEWS05-15
Metrics	MRR	Hits@1	Hits@3	Hits@10	MRR	Hits@1	Hits@3	Hits@10
ComplEx-N3°	.47	.35	.54	.71	.49	.37	.55	.73
TTransE*	.255	.047	-	.601	.271	.084	-	.616
HyTE	.297	.108	.416	.655	.316	.116	.445	.681
TA-TransE	.275	.095	-	.625	.299	.096	-	.668
TA-DistMult	.477	.363	-	.686	.474	.346	-	.728
DE-SimplE	.526	.418	.592	.725	.513	.392	.578	.748
ATiSE	.550	.436	.629	.750	.519	.378	.606	.794
TeRo	.562	.468	.621	.732	.586	.469	.668	.795
TIME-PLEX(base)	.589	.499	-	.761	.632	.542	-	.802
TComplEx	.61	.53	.66	.77	.66	.59	.71	.80
TeLM	.625	.545	.673	.774	.678	.599	.728	.823