Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations
Authors: Zhen Han
Ruotong Liao
Beiyan Liu
Yao Zhang
Zifeng Ding
Jindong Gu
Heinz Koeppl
Hinrich Schuetze
Volker Tresp
Venue: Submitted to ICLR 2023 (2023)
Keywords: structured and unstructured knowledge integration
temporal knowledge embedding
pre-trained language model
temporal reasoning

Abstract: World knowledge exists in both structured (tables, knowledge graphs) and unstructured forms (texts). Recently, there have been extensive research efforts in the integration of structured factual knowledge and unstructured textual knowledge. However, most studies focus on incorporating static factual knowledge into pre-trained language models, while there is less work on enhancing temporal knowledge graph embedding using textual knowledge. Existing integration approaches can not apply to temporal knowledge graphs (tKGs) since they often assume knowledge embedding is time-invariant. In fact, the entity embedding in tKG embedding models usually evolves over time, which poses the challenge of aligning temporally relevant textual information with entities. To this end, we propose Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations (ECOLA), which uses tKG quadruple as an implicit measure to temporally align textual data and the time-evolving entity representations and uses a novel knowledge-text prediction task to inject textual information into temporal knowledge embedding. ECOLA jointly optimizes the knowledge-text prediction objective and the temporal knowledge embedding objective, and thus, can simultaneously take full advantage of textual and structured knowledge. Since existing datasets do not provide tKGs with aligned textual data, we introduce three new datasets for training and evaluating ECOLA. Experimental results on the temporal knowledge graph completion task show that ECOLA outperforms state-of-the-art tKG embedding models by a large margin.

Under review as a conference paper at ICLR 2023
ENHANCED TEMPORAL KNOWLEDGE EMBEDDINGS WITH CONTEXTUALIZED LANGUAGE REPRESENTA-
TIONS
Anonymous authors Paper under double-blind review
ABSTRACT
World knowledge exists in both structured (tables, knowledge graphs) and un- structured forms (texts). Recently, there have been extensive research efforts the integration of structured factual knowledge and unstructured textual knowl- edge. However, most studies focus on incorporating static factual knowledge into pre-trained language models, while there is less work on enhancing tempo- ral knowledge graph embedding using textual knowledge. Existing integration approaches can not apply temporal knowledge graphs (tKGs) since they often assume knowledge embedding is time-invariant. In fact, the entity embedding in (KG embedding models usually evolves over time, which poses the challenge of aligning lemporally relevant textual information with entities. To this end, we pro- pose Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations (ECOLA), which uses tKG quadruple as an implicit measure to temporally align textual data and the time-evolving entity representations and uses novel knowledge-text prediction task to inject textual information into temporal knowledge embedding. ECOLA jointly optimizes the knowledge-texx prediction objective and the temporal knowledge embedding objective, and thus, can simulta- neously take full advantage of textual and structured knowledge. Since existing datasets do not provide tKGs with aligned textual data, we introduce three new datasets for training and evaluating ECOLA. Experimental results on the temporal knowledge graph completion task show that ECOLA outperforms state-of-the-art tKG embedding models by large margin.
INTRODUCTION
Knowledge graphs (KGs) have long been considered effective and efficient way to store structural knowledge about the world. knowledge graph consists of a collection triples (s, p, o), where s (subject entity) and (object entity) correspond to nodes and p (predicate) indicates the edge type (relation) between the two entities. Common knowledge graphs (Toutanova et al.) 2015 Dettmers et al. 2018) assume that the relations between entities are static connections. However, the real world, there are not only static facts and properties but also time-evolving relations associated with the entities. For example, the political relationship between two countries might worsen because of trade fights. To this end, temporal knowledge graphs (IKGs) (Tresp et al., 2015)) were introduced that capture temporal aspects of relations by extending triple quadruple, which adds timestamp or time interval to describe when the relation is valid, e.g. (Argentina, deep comprehensive strategic partmership with, China, 2022). Extensive studies have been focusing on learning temporal knowledge embedding (Leblay & Chekol 2018 Han et al. 2020c), which not only helps infer missing links in tKGs but also benefits various knowledge-related downstream applications, such as temporal question answering (Saxena et al. 20211b.
However, knowledge graph embedding often suffers from the sparseness knowledge graphs. For example, the tKG model proposed by Han al. (2020a) performs much better on the dense tKG than the sparse one. To address this problem, some recent studies incorporate textual information to enrich knowledge embedding. KEPLER (Wang et al. 2021) learns the representation of entity by encoding the entity description with a pre-trained language model (PLM) and optimizing the knowledge embedding objective. KG-Bert (Yao al. 2019) takes entity and relation descriptions of

---

Under review as a conference paper at ICLR 2023
Pofer Ooour
Redetow
2001 1000-00022 18
The Mete Foundaaton ltunched moserch award, which was offoaily of the strmars
Give avards
ASTA SScusianneey Hied Outactey oosccl retreted Mark.
202 1-08-10
Metn
Today is Citicber 24, ciin't espect Mena instagram photi-etaring for dibblle devicces
Acquire 22012-06-00
2019-6-12
Sug
was reportor bal Movee addeet apvaned Sil Coart.
J TIITKK
O
Instagram
Figure 1: An example of a temporal knowledge graph with textual event descriptions.
triple as the imput of PLM and turns knowledge graph completion into sequence classification problem. However, they do not take the temporal nature and the evolutionary dynamics of know Enowledge graphs into account. In tKG embedding models, the entity representations usually evolve over time as they involve in different events at different timestamps. Taking financial crises as an example, companies are more likely involved in events such laying off employees. But when the economy recovers, companies hire staff again rather than cut jobs. Thus, the entities should also be able to drift their representations over time manage the changes. Therefore, given an entity, should be taken into account which textual knowledge is relevant to it at which timestamp. We name this challenge as temporal alignment between texts and tKG, which to establish correspondence between textual knowledge and their temporal knowledge graph depiction. This is one of the challenges that existing approaches cannot handle due to their limitation of assuming knowledge embedding is static and using the time-invariant description of entity to enhance its representation. Thus, they are not appropriate in temporal knowledge graph scenarios where temporal alignment is required. The other challenge is that temporal knowledge embedding models learn the entity representations of time, which exposes another limitation of existing approaches that their architectures cannot be naturally combined with (KG models. Therefore, it is not clear how to enhance lemporal knowledge embedding with textual data.
function
To this end, we propose Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations (ECOLA), which uses temporally relevant textual knowledge to enhance the time- dependent knowledge graph embedding and ensures that the enhanced knowledge embedding pre- serves the temporal nature. Specifically, we solve the temporal alignment challenge by using tKG quadruples as an implicit measure. We pair a quadruple with its relevant textual data, e.g., event descriptions, which corresponds to the temporal relations between entities at specific time. Then we use the event description to enhance the representations of entities and predicate involved in the given quadruple. In particular, we encode entities and predicates by tKG embedding models and encode texts using token embedding.... Given a quadruple-tex pair, we concatenate the embedding of entities, predicate, and textual tokens and feed them into pre-trained language model. We introduce a novel knowledge-text prediction (KTP) task to inject textual knowledge into temporal knowledge embedding. The KTP task is an extended masked language modeling task, which randomly masks words in texts and entity/predicates in quadruples. With the help of the KTP task, ECOLA would be able to recognize mentions of the subject entity and the object entity and align semantic relationships in the text with the predicate in the quadruple. Thus, the model can take full advantage of the abundant information from the textual data, which especially helpful for embedding entities and predicates prediction
that only appear in a few quadruples. ECOLA jointly optimizes the knowledge-te and temporal knowledge embedding objectives. Since our goal is to develop an approach that can generally improve any potential tKG models, we combine the model with different benchmark tKG embedding models (Goel et al., 2020: Han et al. 2020cc; 2021). For training ECOLA, we need datasets with temporal KG quadruples and aligned textual event descriptions, which is unavailable in existing temporal KG benchmark datasets. Thus, we construct three new temporal knowledge graph datasets by adapting two existing datasets, i.e., GDELT (Leetaru & Schrodt (013)) and Wiki (Dasgupta et al. 22018) and an event extraction dataset (Li et al. 2020). To make a fair comparison with other temporal KG embedding models and keep fast inference, we only take the enhanced
2

---

Under review as a conference paper at ICLR 2023
temporal knowledge embedding to perform the temporal KG completion task at test time but do not use any textual descriptions of test quadruples.
To summarize, our contributions knowledge graph representation models with textual knowledge via pre-trained language models. ECOLA shows its superiority on the temporal KG completion task and can be potentially combined with any temporal KG embedding model. (ii) We are the first to address the challenge of enhancing temporal knowledge embedding with temporally relevant textual information while preserving the time-evolving properties of entity embedding. (iii) To train the integration models, we construct three datasets, which align each quadruple with relevant textual description, by adapting three existing temporal KG completion datasets. Extensive experiments show that ECOLA is model-agnostic and enhances temporal KG embedding models with up to 287% relative improvements the Hits @ metric.
follows: (i) We propose ECOLA that enhances temporal
2 PRELIMINARIES AND RELATED WORK
Temporal Knowledge Graphs Temporal knowledge graphs are multi-relational, directed graphs with labeled timestamped edges between entities (nodes). Let & and P represent a finite set of entities and predicates, respectively. (KG contains a collection of events written as quadruples. A quadruple q = (e.: P: eo, t) represents a timestamped and labeled edge between a subject entity e, E & and an object entity e, E & at a timestamp / E TT Let F represents the set of all true quadruples, i.e., real events in the world, the temporal knowledge graph completion (tKGC) is the task of inferring JF based on a set of observed facts O which is a subset of J. Specifically, tKGC is to predict either a missing subject entity (?, P. cos 1) given the other three components or a missing object entity (es, P. ?, t). We provide related works on temporal knowledge representations in Appendix
Joint Language and Knowledge Models Recent studies have achieved great success in jointly learning language and knowledge representations. Yamada et al. (2016) and Ganea & Hofmann (2077) use entity linking to map entities and words into the same representation space. Inspired by the success of contextualized language representation, Zhang et al. (2019) and Peters et al.) (2019) focus on enhancing language models with external knowledge. They separately pre-train the entity embedding with knowledge embedding models, e.g., TransE Bordes al.. (2013), and inject the pre-trained entity embedding into PLMs, while fixing the entity embedding during training PLMs. Thus, they are not real joint models for learning the knowledge embedding and language embedding simultaneously. Yao et al. (2019). Kim et al. (2020). and Wang et al. (202T) learn to generate entity embeddings with pre- trained language models (PLMs) from entity deseriptions. Moreover, HH et al.) (2019), (Sun et al. (2020), and Liu et al. (2220) exploit the potential of contextualized knowledge representation by constructing subgraphs structured knowledge and textual data instead of treating single triples as training units. Nevertheless, none of these works consider the temporal aspect of knowledge graphs, which makes them different from our proposed ECOLA.
ECOLA
In this section, we present the overall framework of ECOLA, including the model architecture in Section 3.1 3.3 novel training task designed for aligning knowledge embedding and language representation in Section 3.4 and the training procedure in Section 3.5 As shown in Figure 2 ECOLA implicitly incorporates contextualized language representations into temporal knowledge embeddings by jointly optimizing the knowledge-ter text prediction loss and the temporal knowledge embedding loss. Note that, at inference time, we only take the enhanced temporal knowledge embeddings to perform the temporal KG completion task without using any textual data for preventing information leakage and keep fast inference speed.
3.1 EMBEDDING LAYER
In tKG embedding models, entity representations evolve over time. Thus, the key point of enhancing -ime-dependent entity representation e,(t) is to find texts that relevant to the entity at the time of interest t. To this end, we use tKG quadruples (e.g.. (c,, p. ej. t)) as an implicit measure for the alignment. We pair quadruple with its relevant textual data and use such textual data enhance
3

---

Under review as a conference paper at ICLR 2023
Word Prediction
Entlyy Prediction
Predicate Prediction
Fouth rewing Aips umeats
Transfomerr Encoder
(199)) unidity tailie, spredicee mudio, Japan, 2015-23-11 Vocie deeniggnn TrAdA
(both Farci, downgrekks mude tcs with Japars 2010
moeing digin Wem Uar snared codiigg pormess on
News Sources
Temporal Knowledge Graph
South Koroa downgrades trade with Japan
Figure 2: Model architecture. ECOLA jointly optimizes the knowledge-ted prediction (KTP) objective and the temporal knowledge embedding (tKE) objective.
Inpaz
South
frodde Bas wirth
90-2
En Enllasiing
Mppor
Fimbwertigg
Type
fousee
Seave
Fouss
Furts
Syetrie
Eusry
Tut
Euc
Figure 3: ECOLA input representations. Following textual tokens, the last four tokens in input correspond to quadruple from temporal knowledge graph. The last one is the timestamp / for incorporating temporal information into entity representation ez(1). Here, denotes subword token embedding, and denote entity and predicate embedding, respectively.
the entity representation (t). Therefore. training sample is pair quadruple from temporal KGs and its corresponding textual description, which are packed together into a sequence. As shown Figure the input embeddings are sum of token embedding, type embedding, and position embedding. For token embedding, maintain three lookup tables for subwords, entities, and predicates, respectively. For subword embedding, we first tokenize the textual description into sequence of subwords following Bert (Devlin al. 2018) and use WordPiece embeddings (Wul et al.] 2016) with 30,000 token vocabulary. As the yellow tokens shown in the Figure 3 We denote embedding sequence of subword tokens (wi, .... wa). In contrast to subword embedding, the embeddings for entities and predicates are directly learned from scratch, similar to common knowledge embedding methods. We denote the entity embedding and predicate embedding and respectively, as the blue tokens shown in Figure 3 We separate the knowledge tokens, i.c., entities and predicates, and subword tokens with a special token [SEP]. To handle different token types, we add type embedding to indicate the type of each token, i.e., subword, entity, and predicate. For position embedding. we assign each token an index according to its position in the input sequence and follow Devlin et al. (2018) to apply fully-learnable absolute position embeddings.
3.2 TEMPORAL KNOWLEDGE ENCODER
As shown in Figure the input embedding for entities and predicates consists of knowledge token embedding, type embedding, and position embedding. In this section, we provide details of the temporal knowledge embedding objective.
temporal embedding function defines entity embedding as a function that takes an entity and a timestamp input and generates time-dependent representation. There is line of work exploring temporal embedding functions. Since we aim to propose a model-agnostic approach, we combine ECOLA with three temporal embedding functions, i.e., DyERNIE-Fuclid (Han et al.) 2020c). UTEE
4

---

Under review as a conference paper at ICLR 2023
(Han et al. 2021), and DE-SimplE (Goel et al. 2020). In the following, refer to DyERNIE. Euclid as DyERNIE and take as an example to introduce our framework. Specifically, the entity representation is derived from initial embedding and velocity vector
(t) e,
PyER Ve,l,
where d represents the initial embedding that does not change over time, and Ve, is an entity- specific velocity vector. The combination with other temporal embedding functions are discussed in Section The score function measuring the plausibility of quadruple is defined as follows,
DyER (et, P.
d(P O ( e PyER (t) + +) + b, + by,
(1)
where and p represent the diagonal predicate matrix and the translation vector of predicate p. respectively, d denotes the Fuclidean distance, and b;, bj are scalar biases of the subject and object e; and Bg respectively. By learning tKE, we generate M negative samples for each positive quadruple in batch. We choose the binary cross entropy the temporal knowledge embedding objective
-1 N
LIKE
(yk log(ps) + - yı:) log(1 - -PE));
Æ-1
where N is the sum of positive and negative training samples, yA represents the binary label indicating whether training sample is positive or not, P&c denotes the predicted probability o (00 PyFR), and a-)) represents the sigmoid function.
3.3 MASKED TRANSFORMER ENCODER
To encode the input sequence, we use the pre-trained contextual language representation model Bert (Devlin et al. 2018). Specifically, the encoder feeds a sequence of N tokens including entities, predi- cates, and subwords into the embedding layer introduced in Section to get the input embeddings and then computes T. layers of d-dimensional contextualized representations. Eventually, we get contextualized representation for each token, which could be further used to predict masked tokens.
3.4 KNOWLEDGE TEXT PREDICTION TASK
To incorporate textual knowledge into temporal knowledge embedding, we use the pre-trained language model Bert to encode the textual description and propose knowledge-text prediction task to align the language representations and the knowledge embedding. The knowledge-text prediction task is extension of the masked language modeling (MLM) task. As illustrated in Figure 2 given a pair of a quadruple and the corresponding event description, the knowledge-texx prediction task to randomly mask some of the input tokens and train the model predict the original index of the masked tokens based on their contexts. As different types of tokens are masked, we encourage ECOLA to learn different capabilities:
Masking entities. To predict an entity token in the quadruple, ECOLA has the following ways to gather information. First, the model can detect the textual mention of this entity token and determine the entity; second, if the other entity token and the predicate token are not masked, the model can utilize the available knowledge token to make a prediction, which is similar to the traditional semantic matching-basee temporal KG models. Masking entity nodes helps ECOLA align the representation spaces of language and structured knowledge, and inject contextualized representations into entity embeddings.
Masking predicates. To predict the predicate token in the quadruple, the model needs to detect mentions of subject entity and object entity and classify the semantic relationship between the two entity mentions. Thus, masking predicate tokens helps the model integrate language representation into the predicate embedding and map words and entities into a common representation space.
Masking subwords. When subwords are masked, the objective is similar to traditional MLM. The difference is that ECOLA not only considers the dependency information in the text but also the entities and the logical relationship in the quadruple. Additionally, we initialize the encoder with the pre-trained BERT buse. Thus, masking subwords helps ECOLA keep linguistic knowledge and avoid catastrophic forgetting while integrating contextualized representations into temporal knowledge embeddings.
5

---

Under review as a conference paper at ICLR 2023
In each quadruple, the predicate and each entity have a probability of 15% be masked. Similarly, we mask 15% of the subwords of the textual description at random. We ensure that entities and the predicate cannot be masked the same time in single training sample, where conduct an ablation study in Section 6 to show the improvement of making this constraint. When a token is masked, we replace it with (1) the [MASK] token 80% of the time, (2) randomly sampled token with the same type as the original token 10% of the time, (3) the unchanged token 10% of the time. For each masked token, the contextualized representation in the last layer of the encoder is used for three classification heads, which are responsible for predicting entities, predicates, and subword tokens, respectively. At last, cross-entropy loss CKTP is calculated over these masked tokens.
Although we focus on generating informative knowledge embeddings in this work, joint models often benefit both the language model and the temporal KG model. Unlike previous joint models (Zhang et al., 2019 Peters et al. 2019), we do not modify the Transformer encoder architecture, e.g., adding entity linkers or fusion layers. Thus, the language encoder enhanced by external knowledge can be adapted to wide range of downstream tasks as easily as Bert. We evaluate the enhanced language model on the temporal question answering task and report the results Appendix C
3.5 TRAINING PROCEDURE AND INFERENCE
We initialize the transformer encoder with the pre-trained language model BERT buse I and the knowl- edge encoder with random vectors. Then we use the temporal knowledge embedding (tKE) objective LIKE to train the knowledge encoder and use the knowledge-text prediction (KTP) objective CKTP to incorporate temporal factual knowledge and textual knowledge in the form of a multi-task loss:
L LIKE + XLTTP,
where À is a hyperparameter to balance tKE loss and KTP loss. Note that those two tasks share the same embedding layer of entities and predicates. At inference time, aim answer link prediction queries, e.g., (e., ?, t). Since there is no textual description at inference time, we take the entity and predicate embedding as input and use the score function of the knowledge encoder, e.g., Equation | to predict the missing links. Specifically, the score function assigns a plausibility score to each quadruple, and the proper object can be inferred by ranking the scores of all quadruples [(e., ej. e3 E E) that are accompanied with candidate entities.
4 MODEL VARIANTS
ECOLA is aggostic and can enhance different temporal knowledge embedding models. Besides ECOLA-DyERNIE, we introduce here additional variants of ECOLA.
ECOLA-DE enhances the tKG embedding model DE-SimplE, which applies the diachronic embedding (DE) function (Goel et al. 2020). DE-function defines the temporal embeddings of entity at timestamp / as
& yd, a., [m] if
e DE PE
(3)
ae, [m] sin[w.,Int + [n]) else.
Here, [t) denotes the th element of the embeddings of entity at time t. a We,; be, e Rdd are entity-specific vectors with learnable parameters, d is the dimensionality of the embedding, and y E [0. represents the portions of the time-independent part. Besides, it use SimplE (Kazemi & Poole 2018) as the score function temporal knowledge embedding.
ECOLA-UTEE enhances UTEE Han et al. 2022 that learns shared temporal encoding function for all entities to deal with the overfitting problem of the diachronic approach (Goel et al. (0200) on sparse datasets. Compared to ECOLA-DE, ECOLA-UTEE replaces Equation 3 with follows:
UTEE (t) [e, a sin(ws + b)], é, E RTd; a. be R (1-7)d
where ē; denotes entity-speciffe time-invariant part, the amplitude vector a, frequency vector w, and bias b are shared among all entities, denotes the concatenation operator, and 7 [0, 1].

---

Under review as a conference paper at ICLR 2023
5 DATASETS
The training procedure of ECOLA requires both temporal knowledge graphs and textual descriptions. Given a quadruple (ea,P) P, co. f). the key point to find texts that are temporally relevant to e, and e, at I. Existing temporal KG datasets do not provide such information. To facilitate the research on integrating textual knowledge into temporal knowledge embeddings, we reformat three existing datasets, ie., GDELT D and Wik for evaluating the proposed integration method. We show the statistics of the datasets in Table 2 in the appendix. Due to the limited size of upload files, we only attach DuEE and subsets of GDELT and Wiki in the supplementary material and will publish the complete version after acceptance.
GDELT is an initiative knowledge base storing events across the globe connecting people and organizations, e.g., (Google, consult, the United States, 2018/01/06). For each quadruple, GDELT provides the link to the news resource which the quadruple is extracted from. We assume each sentence that contains both mentions of subject entity and object entity is relevant to the given quadruple, and thus, temporally aligned with the subject and object at the given timestamp. We pair each these sentences with the given quadruple to form a training sample. This process similar to the distant supervision algorithm Mintz et al. (2009) in the relation extraction task, which assumes that, given a relationship between two entities, any sentence containing these two entities would express this relation. In total, the dataset contains 5849 entities, 237 predicates, 2403 timestamps, and 943956 quadruples with accompanying sentences.
DuFE is originally a human-annotated dataset for event extraction containing 65 event types and 121 argument roles. Each sample contains sentence and several extracted event tuples. We reformat DuEE by manually converting event tuples into quadruples and then pairing the quadruples with their corresponding sentence.
Wiki temporal knowledge graph dataset proposed by Leblay & Chekol (218)) containing temporal facts from the Wikidata (Vrandečić & Krötzsch (014). Different from GDELT and DuEE, time annotations in Wiki are represented as time intervals, e.g.. (Savonranta, instance of, municipality of Finland, 1882 2009). Following the post-processing by Dasgupta et al. (2018), we discretize the time span into 82 different timestamps. We align each entity to its Wikipedia page and extract the first abstract section its description. To construct the relevant textual data of each quadruple, combine the subject entity description, relation, and object description into a sequence, separated by [SEP] token between two neighboring parts. In this case, the knowledge-text prediction task let subject entity see the descriptions of its neighbors at different timestamps, and thus, preserving the temporal alignment between time-dependent entity representation and textual knowledge.
EXPERIMENTS
We evaluate the enhanced temporal knowledge embedding on the temporal KG completion task. Specifically, we take the entity and predicate embedding ECOLA-DYERNIE and use Equation to predict missing links. To make fair comparison with other temporal KG embedding models, we do not use any textual descriptions at test time.
Baselines We include both static and temporal KG embedding models. From the static KG embedding models, use TransE (Bordes et al.) 2013), DistMult (Yang et al.) 2014), and SimplE (Kazemi & Poole 2018). These methods ignore the time information. From the temporal KG embedding models, we compare our model with several state-of-the-a methods, including AITSEE (Xu et al. 2019). TNTCompIEELacrroi et al. (220)). DyERNIE (Han et al. (2020c), TeRO (Xu et al. 2020), and DE-SimplE (Goel et al. 2020). We provide implementation details in Appendix and attach the source code in the supplementary material.
'https://ai.ban baidu. com/broad/downloud
'https:/wwww.ww httt:///wwwwikiiataaa Waiiii/iiaattaaainnnnn For a fair comparison with other baselines, we choose Fueldd
7

---

Under review as a conference paper at ICLR 2023
Evaluation protocol For each quadruple q (e.. Co, in the test set Gtest. create two queries: (es, P; ?.t) and (?,p, co t). For each query, the model ranks all possible entities & according to their scores. Following the filtered setting in Han al. (2020)), remove all entity candidates that correspond to true triples from the candidate list apart from the current test entity. Let Rank( (s) and Rank(eo) represent the rank for 6s and 6, of the queries respectively, we evaluate our models using standard metrics across the link prediction literature: mean reciprocal rank (MRR): Rankte_)) and Hits@k(k (1,3, 10)): the percentage of times that the true entity candidate appears in the top is of ranked candidates.
2 2/ Eyggi... Rank(e,) Rank(e.))
Table 1:
Temporal link prediction results: Mean Reciprocal Rank (MRR, %) and Hits @ 1/3(%). The results of the proposed fusion models (in bold) and their counterpart KG models are listed together.
Datasets
GDELT filtered
Wiki filtered
DuFE filtered
Model
MRR
Hits@1
Hits@
MRR
Hits@1 1
@its@
MRR
Hits@! I
Hits
TransE
8.08
0.00
8.33
27.25
16.09
33.06
34.25
4,45
60.73
SimplE
10.98
4.76
10.49
20.75
16.77
23.23
51.13
40.69
58,30
DistMult
11.27
4.86
10.87
21.40
17.54
23.86
48.58
38.26
55.26
TeRO
6.59
1.75
5.86
32.92
21.74
39.12
54.29
39.27
63.16
ATiSE
7.00
2.48
6.26
35.36
24.07
41.69
53.79
42.31
59.92
INTCompIEx
8.93
3.60
8.52
34.36
22.38
40.64
57.56
43.52
65.99
UTEE
9.76
4.23
9.77
26.96
20.98
30.39
53.36
43.92
60.52
ECOLA-UTEE
19.11 +
15.29
19.46 +
38.35
30.56
42.11 +
60.36
46.55 :
69.22 :
00.16
00.38
00.05
00.22
00.18
00.14
00.36
00.51
00.93
DyERNIE
10.72
4.24
10,81
23,51
14.53
25.21
57.58
41,49
70,24
ECOLA-DyERNIE
19.99 +
16.40 +
19.78 +
41.22 =
33.02 :
45.00 +
59.64 +
46,35 :
67.87 +
00.05
00.09
00.03
00.04
00.06
00.27
00.20
00.18
00.53
Quantitative Study Table D reports the tKG complction results on the test sets, which are averaged over three trials. The error bars of the ECOLA models are also provided. Firstly, we can see that ECOLA-UTER, improves its baseline temporal KG embedding model, UTEE, by large margin, demonstrating the effectiveness of our fusing strategy. Specifically, ECOLA-UTEE enhances UTEE on GDELT with a relative improvement of 95% and 99% in terms of mean reciprocal rank (MRR) and Hits @ 3, even nearly four times better in terms of Hits @ 1. Thus, its superiority clear on GDELT, which is the most challenging dataset among benchmark tKG datasets, containing nearly one million quadruples and more than two hundred relations. Secondly, ECOLA-UTEE and ECOLÁ-DE generally outperform UTEE and DE-SimplE on the three datasets, demonstrating that ECOLA is model-agnostic and able to enhance different tKG embedding models. Besides, in the DuFE dataset, ECOLA-DYERNIE achieves better performance than DyERNIE in Hits @ and MRR, but the gap reverses in Hits @ 3. The reason could be that ECOLA-DyERNIE is good at classifying hard negatives using textual knowledge, and thus has a high Hits @ I; however, since DuEE is much smaller than the other two datasets, ECOLA-DAS -COLA-DYERNIE may overfit in some cases, where the ground truth is pushed away from the top rank.
We compare the performance of DE-Simp.E, ECOLA-DE, and ECOLA-SF on GDELT in Figure 4a ECOLA-SF is the static counterpart of ECOLA-DE, where we not consider the temporal alignment while incorporating textual knowledge. Specifically, ECOLA-SF integrates all textual knowledge into the time-invariant part of entity representations. We provide more details of ECOLA-SF in Appendix B We can see ECOLA-DE significantly outperforms DE-SimplE and
ECOLA-SF in terms of MRR and Hits@ 1. In particular, the performance gap between ECOLA-DE and ECOLA-SF is significant, demonstrating the temporal alignment between time-dependent entity representation and textual knowledge is more powerful than the static alignment.
Ablation study Figure 4b shows the results of different masking strategies on GDELT. The first strategy called Masking E+R+ W which allows to simultancously mask predicate, entity, and subword tokens in the same training sample. The second strategy is Masking E/R+ W, where we mask 15% subword tokens in the language part, and either entity predicate in the knowledge tuple. In other words, simultancously masking an entity and predicate in a training sample is not allowed. In the third strategy called Masking F/R/W, for each training sample, choose to mask either subword
8

---

Under review as a conference paper at ICLR 2023
ECOLA-DE
GOOLA-SF
Simplll
Masiing Safler
Masking ErtiWW
Masking
Spe Drieodd
Withou: Drieed.
20
20
10
12
MRR
Hr91
MAR
HiHP1
MAA
HR@10
(a)
(b)
(c)
Figure 4: Ablation Study. (a) Temporal alignment analysis. We compare De-SimplE, ECOLA-DE. and ECOLA-SF in terms of MRR(%) and Hits@ 1(%) on GDELT. (b) Masking strategy analysis. We compare ECOLA-DE with different masking strategies and show the results of MRR(%) and Hits@ (%) on GDELT. (c) Type embedding analysis. We compare ECOLA-DE with/without type embedding and show the results of MRR(%) (6) and Hits 1/10(%) on GDELT.
tokens, an entity, or the predicate. The experimental results show the advantage of the second masking strategy, indicating that remaining adequate information in the knowledge tuple helps the model to align the knowledge embedding and language representations. Figure 4c demonstrates the ablation study on the type embedding, which differentiates among subword tokens, entity. and predicate of the input. We can observe that removing type embedding leads to a considerable performance gap on GDELT, indicating that providing distinguishment between subwords, entities, and predicates helps the model to better understand different input components and different prediction tasks.
Qualitative Analysis To investigate why incorporating textual knowledge can improve the tKG embedding models' performance, we study the test samples that have been correctly predicted by the fusion model ECOLA-DE but wrongly by the (KG model DE-SimplE, language representations help overcome the incompleteness of the tKG by leveraging knowledge from augmented textual data. For example, there is a test quadruple (US, host visir, 19-11-14) with ground truth R.T. Erdogan. The training set contains quite relevant quadruple, i.e., (Turkey, intend to negotiate with, US, 19-11-11). However, the given tKG does not contain information indicating that the entity R.T. Erdoğan is a representative of Turkey. So it is difficult for the tKG model DE-SimplE to infer the correct answer from the above-mentioned quadruple. In ECOLA-DE. the augmented textual data do contain such information, c.g. "The president of Turkey, R.T. Erdogan, inaugurated in Aug. 2014.", which narrows the gap between R.T. Erdogan and Turkey. Thus, by integrating textual information into temporal knowledge embedding, the enhanced model gain additional information which the knowledge base does not include. Another example is relevant the entity Charles de Gaulle. To infer the test quadruple (Charles de Gaulle, citizenship of, 1958) with ground truth French 5th Republic, it is noticed that in the training set of ECOLA-DE, we have quadruple (Charles de Gaulle, president of. French 4th Republic, 1957) with supporting textual data "Charles de Gaulle was the last president of French 4th Republic, and French 5th Republic emerged from the collapse of the 4th Republic in 1958. ", which shows that the entity representation of Charles de Gaulle is enhanced by the evolving history of France and is temporally closer to French 5th Republic at the query timestamp 1958.
observed that
7 CONCLUSION
We propose ECOLA enhance temporal knowledge embedding using textual knowledge. Specif- ically, we enhance time-evolving entity representations with temporally relevant textual data by encoding the textual data using a pre-trained language model and introducing a novel knowledge-text prediction task to align the temporal knowledge and language representation into the same semantic space. Besides, we construct three datasets that contain paired structured temporal knowledge and unstructured textual descriptions, which can benefit future research fusing temporal structured and unstructured knowledge. Extensive experiments show ECOLA is aggost- and can improve many temporal knowledge graph models by large margin.

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement About datasets, Since the size of GDELT and Wiki exceeds the limit of upload file size allowed by ICLR, we only upload DuEE. partial Wiki and partial GDELT (short version with 1000 samples) in the supplementary material due to the and will publish the complete dataset of Wiki and GDELT after acceptance. Besides, we provide the description of the data We provide the dataset statistics in Table 2 in appendix. Additionally, we provide our source code in the supplementary material.
processing steps in Section 5
REFERENCES
Antoine Bordes, Nicolas Usunier, Alberto Farcia-Duran Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. Hyte: Hyperplane-basee based temporally aware knowledge graph embedding. In Proceedings the 2018 conference on empirical methods in natural language processing, pp. 2001-2011 2018.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAl conference on artificial intelligence, volume 32, 2018.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXivv/810.000 1810.04805. 2018.
Octavian-Fugen Ganea and Thomas Hofmann. Deep joint entity disambiguation with local neural attention. arXiv preprint arXiv: (ow::7700049.0 2017.
Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker, and Pascal Poupart. Diachronic embedding for temporal knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3988-3995, 2020.
Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. Explainable subgraph reasoning for forecasting on temporal knowledge graphs. In International Conference on Learning Representations, 2020a.
Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. xerte: Explainable reasoning on temporal knowledge graphs for forecasting future links. arXiv preprint arXiv:2012. 15537, 2020b.
Zhen Han, Yunpu Ma, Peng Chen, and Volker Tresp. Dyernie: Dynamic evolution of riemannian manifold embeddings for temporal knowledge graph completion. arXiv preprint arXiv:201 1,03984, 2020c.
Zhen Han, Gengyuan Zhang, Yunpu Ma, and Volker Tresp. Time-dependent entity embedding is not all you need: A re-evaluation of temporal knowledge graph completion models under a unified framework. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8104 8118, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 111633///12211emmm//m.. main.639, URL
wwwethooooy..... www.lllyy...ooooooo.ee.o 2021 wmmll-pppee ma i nn. 639
Bin He, Zhou, Jinghui Xiao, Qun Liu, Nicholas Jing Yuan, Tong Xu, et al. Integrating graph contextualized knowledge into pre-trained language models. arXiv preprint arXiv: arXiv:1912.001 1912.0017, 2019.
Seyed Mehran Kazemi and David Poole. Simple embedding for link prediction in knowledge graphs. Advances in neural information processing systems, 31, 2018
Bosung Kim, Taesuk Hong, Youngjoong Ko, and Jungyun Seo. Multi-task learning for knowledge graph completion with pre- trained language models. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 1737-1743, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 142266980, 2014.
10

---

Under review as a conference paper at ICLR 2023
Timothée Lacroix, Guillaume Obozinski, and Nicolas Usumier. Tensor decompositions for temporal knowledge base completion. arXiv preprint 20X1:20004.44226. 2020.
Julien Leblay and Melisachew Wudage Chekol. Deriving validity time in knowledge graph. In Companion Proceedings of the The Web Conference 2018, pp- 1771-1776, 2018.
Kalev Leetaru and Philip A Schrodt. Gdelt: Global data events, location, and tone, 1979-2012. In ISA annual convention, volume 2, pp. 1-49. Citeseer, 2013.
Xinyu Li, Fayuan Li, Lu Pan, Yuguang Chen, Weihua Peng, Quan Wang, Yajuan Lyu, and Yong Zhu. Duce: a large-scale dataset for chinese event extraction in real-world scenarios. In CCF International Conference on Natural Language Processing and Chinese Computing, pp. 534-545. Springer, 2020.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2901-2908, 2020.
Yunpu Ma. Volker Tresp, and Erik A Daxberger. Embedding models for episodic knowledge graphs. Journal of Web Semantics, 59:100490, 2019.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 1003-1011, 2009.
Matthew E Peters, Mark Neumann, Robert L. Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. Knowledge enhanced contextual word representations. arXiv preprint 0rXiv:1909.04664 2019.
Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. Question answering over temporal knowledge graphs. In Proceedings the 59th Annual Meeting of the Association for Computational Linguistics, 2021a.
Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. Question answering over temporal knowledge graphs. arXiv preprint arXiv:2106.015// 2021b.
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. Colake: Contextualized language and knowledge embedding. arXiv preprint arXiv:2010.00309, 2020.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1499-1509, 2015.
Volker Tresp, Cristóbal Esteban, Yinchong Yang, Stephan Baier, and Denis Krompaß. Learning with memory embeddings. arXiv preprint arXiv:1511.0797 2015.
Denny Vrandečié and Markus Krötzsch. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10)778-85 85, sep 2014. ISSN 0001-0782, doi: 10.1145//629489, URL https://doi org/10.1145/262993
Xiaozii Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang. Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176-194, 2021.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's 's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint 16v.1669.08814 2016.
Chengjin Xu, Mojtaba Nayyeri, Fouad Alkhoury, Hamed Shariat Yazdi, and Jens Lehmann. Temporal knowledge graph embedding model based on additive time series decomposition. arXiv preprint 1911:/91 2019.
11

---

Under review as a conference paper at ICLR 2023
Chengjin Xu, Mojtaba Nayyeri, Fouad Alkhoury, Hamed Shariat Yazdi, and Jens Lehmann. TeRo: A time-aware knowledge graph embedding via temporal rotation. In Proceedings of the 28th International Conference Computational Linguistics, pp. 1583- 1593, Barcelona, Spain (Online), December 2020. International Committee on Computational I Linguistics. doi: 10. 8653/v1/20200
coling-main 133...139. URL httpss//:///a htt.::p:://.c.aattooooooooogo/// wwt.logg..rg//20000 wow.-ngmannnnnnonooooooo 1.39
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. Joint learning of the em- bedding of words and entities for named entity disambiguation, arXiv preprint 2016.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv: 1rXfv:1412.6575 2014.
Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion. arXiv preprint 1rXiv:1909003/990 2019.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:/905.07/22) 2019.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
Table 2: Datasets Statistics
Dataset
Entities
# Predicates
# Timestamps
# training set
# validation set
# test set
GDELT
5849
237
2403
755166
94395
94395
DUEE
219
41
629
1879
247 19374
247
WIKI
10844
23
82
233525
19374
Table 3: Hyperparameter settings of ECOLA and baselines.
Parameters
Embedding dimension
Negative Sampling
Learning rate
Batch Size
Datasets
GDELT
DuFF
GDELT
DuFFE
GDELT
DuFFi
GDELT
DaEF
TransE
768
768
768
200
100
100
5c-4
5e-4
5c-4
256
128
256
SimplE
768
768
768
200
100
100
5c-4
5e-4
Se-4
256
128
256
TTransE
768
768
200
5.2e-4
5.2e-4
256
256
TNTComplEx
768
768
768
200
100
100
1,5e-4
1,5e44
1.5e-4
256
256
256
DE-SimplE
768
768
200
5c-4
5e-4
128
256
SCOLA-SF
768
768
768
200
le-4
2e:5
1e-4
64
16
64
ECOLA-DE
768
768
768
200
200
200
20-5
2e-5
2e-5
8
ECOLA-UTEE
768
768
768
200
200
200
20-5
20-5
ECOLA-dyERNIE
768
768
768
200
200
200
26-5
e-4
2e-5
4
A RELATED WORK OF TEMPORAL KNOWLEDGE EMBEDDING
Temporal Knowledge Embedding (IKE) is also termed Temporal Knowledge Representation Learning (TKRL), which is to embed entities and predicates of temporal knowledge graphs into oww-dimesiional vector spaces. TKRL is expressive and popular paradigm underlying many KG models. To capture temporal aspects, each model either embeds discrete timestamps into a vector space learns time-dependent representations for each entity. Ma al. (019)) developed extensions of static knowledge graph models by adding timestamp embeddings to their score functions. Besides, HyTE (Dasgupta et al., 2018) embeds time information in the antity-relation space by learning temporal hyperplane to each timestamp and projects the embeddings of entities and relations onto simestamp-specific hyperplanes. Later, Goel et al. (2020) equipped static models with a diachronic entity embedding function which provides the characteristics of entities at any point in time and achieves strong results. Moreover, [Han et al. (2020c) introduced non-Euclidean embedding approach that learns evolving entity representations in product of Riemannian manifolds. It is the first work to contribute to geometric embedding for tKG and achieves state-of oftte-of-th-arr the-art performances on the benchmark datasets. In particular, ECOLA is model-agmostic which means any temporal KG embedding model can be potentially enhanced by training with the knowledge-tex task.
B ECOLA-SF: AN ABLATION STUDY ON STATIC FUSION
We compare the effectiveness of enhancing temporal knowledge embedding and enhancing static knowledge embedding. In particular, we only feed the static part of entity embeddings into PLM to perform the knowledge-text prediction task. We refer it ECOLA-SF (StaticFusion).
ECOLA-SF is the static counterpart of ECOLA-DE, where we do not apply temporal knowledge embedding to the knowledge-tess prediction objective LKTP. Specifically, we randomly initialize an embedding vector ē; é Rd for each entity e; E E, where é; has the same dimension as the token embedding pre-trained language models. Then we learn the time-invariant part é; knowledge-tews prediction task. For the tKE objective, we have the following temporal knowledge embedding.
(F(() [W]ee[[]] if 11sn 7d, a., [n] sin(w., [n][+ b., [m]) else,
where (FF()) Rd is an entity embedding containing static and temporal embedding parts. agy Wey b.; E Rd-ad are entity-specific vectors with learnable parameters. Waf Rdxad is
13

---

Under review as a conference paper at ICLR 2023
Model
Hits (%)
Hits 10 (%)
CRONKGQA
25.8
52.0
ECOLA-CRONKGQA
27.5
54.4
Table 4: Performance of ECOLA enhaned language model in tKBQA task.
matrix with learnable weights. Note that ef only plays a role in LIKE, and we use static embedding é; instead e (t) in LKTP.
C ENHANCEMENT ON LANGUAGE REPRESENTATIOOS
Although our work focuses on enhancing temporal knowledge embeddings with contextualized language representations, joint models often benefit both the language model and the tKG model due to the mutual information exchange between language and tKGs during joint training. To study ECOLA's enhancement on the language model, we selected temporal question answering as downstream task to show that the proposed ECOLA can also benefit the language model.
Temporal Question Answering over Temporal Knowledge Graphs (TKGQA) Natural questions often include temporal constraints, e.g., who was the US president before Jimmy Carter? To deal with such challenging temporal constraints, temporal question answering over temporal knowledge base, formulated as TKGQA task, has become trendy since tKGs help to find entity timestamp answers with support of temporal facts.
Performance Gain on TKGQA Saxena al. (2021a) introduced the TKGQA dataset CRONQUES- TIONS containing natural temporal questions with different types of temporal constraints and an accompanying temporal knowledge graph (IKG). They proposed baseline called CRONKGQA that uses a pre-trained language model (BERT) to understand the implicit representation of temporal constraints in temporal questions followed by scoring function for answer prediction. We enhance the language encoder in CRONKGQA with the proposed ECOLA approach, i.e., we find temporal relevant texts for quadruples in the supporting tKG given in CRONQUESTIONS and train the language model and the tKG model jointly with the proposed KTTP task. Then we plug the enhanced language model back into CRONK GQA and name the enhanced model ECOLA- SCOLA-CRONK GQQ. The models 113)): the percentage of times that the true entity
are evaluated with standard metrics Hits@k(k or time candidate appears in the top of ranked candidates. As shown in Table 4 empirical results show that our proposed ECOLA enhances the language model with 7.4 % relative improvements regarding precision on CRONQUESTIONS, demonstrating the benefits of ECOLA the language model.
D IMPLEMENTATION
We use the datasets augmented with reciprocal relations to train all baseline models. We tune hyperparameters our models using the random search and report the best configuration. Specifically, set the loss weight À to be 0.3, except for ECOLA-DE model trained Wiki dataset where is set to be 0.001. We use the Adam optimizer (Kingma & Ba (2044). We use the implementation of DE-SimplE ATiSE/TeRC We the code for TNTCopmlEx from the (KG framework (Han et al. 2021). We implement TTransE based on the implementation of TransE in РуКЕEN We provide the detailed settings of hyperparameters of each baseline model and ECOLA in Table in the appendix.
THE AMOUNT OF COMPUTE AND THE TYPE OF RESOURCES USED
We run our experiments on an NVIDIA A40 with memory size of 48G. We provide the training time of our models and some baselines in Table 5 Note that there are no textual descriptions at
14

---

Under review as a conference paper at ICLR 2023
inference time, and take the entity and predicate embedding imput and the score function of KG models to predict the missing links. Thus, the inference time of ECOLA (e.g., ECOLA-DE) and its counterpart KG model (c.g.. DE-SimplE) is the same.
Table 5: The runtime of training procedure (in hours).
Dataset
GDELT
DuEE
Wiki
DE- DE-SimplE
17
0.5
5.0
ECOLA-DE
24.0
16.7
43.2
UTEE
67.3
0.5
11.3
ECOLA-UTEE
36.0
12.8
45.6
DyERNIE
25
0.1
5.9
ECOLA- DEERNE
23.8
10.8
67.2
15


Reviews:
[{"review_id": "swXx3Fv8jK", "reviewer": "Reviewer_r4WS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- This paper studies temporal knowledge graph (tKG) embeddings and proposes a new method that leverages textual knowledge/language models to overcome the sparsity of KGs. The authors identify two challenges: aligning temporally relevant textual information with KG entities, and incorporating temporal information into joint text-KG modeling. The paper then proposes the ECOLA method that addresses these two challenges: it creates new datasets that align tKG and text using using tKG quadruples, and uses a knowledge-text prediction (KTP) task to combine textual knowledge and temporal knowledge quadruples from tKG. The proposed method outperforms existing tKG embedding methods that do not use textual information. ", "summary_of_the_review": "Overall, the paper studies an important problem and develops a method that works well. There is some room for improvements, but I did not find critical weaknesses. Overall I think this paper is above acceptance threshold.", "strength_and_weaknesses": "Strength\n- The paper is generally well-written and easy to understand \n- The problem studied is well-motivated. Static KGs are often studied in existing works, but a lot of knowledge is inherently time-dependent, motivating temporal KG representations. The use of textual information for temporal KG is an important direction for coping with the sparsity of tKGs.\n- The paper identifies clear technical challenges (alignment of text and KG tuples, and modeling of temporal information), and proposes reasonable solutions to them.\n- Experiments are solid, conducted on three datasets, and show that the proposed method provides significant improvement in performance.\n\nWeakness / suggestion for improvement\n- The authors mention that at inference time, the method does not use textual descriptions for test quadruples, but it still achieves improved performance. I wonder if the authors could provide a bit more discussions about the intuition behind this? - e.g. Is this because the test entities were seen during training and the text information used during training already provided extra knowledge for these test entities? A related question is, would the proposed method work if an unseen entity is given at test time? I think it would be great if the authors could add these discussions to the paper.\n\nQuestion\n- Did the author also consider masking and predicting the temporal information in the KTP task? If not, might that be an interesting thing to try?\n\nTypo\n- Page 2 at “a novel knoweldge-text prediction (KTP) task”: “knoweldge” -> “knowledge”", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SmjEYbrF3k-", "reviewer": "Reviewer_REq6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a temporal KG-enhanced language model. \n\nThe idea is to combine pretrained temporal knowledge embeddings with the language model as input tokens, and the language model will take temporal-aware embeddings as input. \nThe authors propose masking the predicates, entities, or subwords for pretraining the model. \nThe authors convert three KGs into the pretraining corpus to find relevant text with temporal KG knowledge. Experiments show that the pretrained model has improved performance on link prediction for the three temporal KGs.", "summary_of_the_review": "In all, I feel the paper focuses on an important area, but the novelty and contribution are not enough for a conference like ICLR.", "strength_and_weaknesses": "Strength: \n\nThe paper fills in the gap on using temporal KG to enhance the language model. The results look solid and bring consistent improvement in link prediction.\n\nWeakness: \n1. The method in the paper lacks novelty. It basically reformulates the previous KG+LM methods to the temporal KG+LM setting. Actually, the method (and even the figures) looks a lot like the KEPLER paper (Wang et al., 2019), with the KG embeddings replaced by tKG embeddings, and the transE loss replaced by the corresponding tKG loss.\n2. The method is only tested on link prediction tasks. I would expect this model to work in more diverse settings (i.e., any task that LM can be used for). It would be great if the model could work for settings like question answering, event extraction, etc. I'm also interested in how the tKG-pretraining affects general NLU performance like on GLUE.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IHy2tgDKib", "reviewer": "Reviewer_LrUQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to use contextualized language representation to enhance the temporal knowledge graph embedding. It combines tKGE losses with knowledge-text-prediction loss during training. As such, the proposed method can be combined with arbitrary tKGE methods. Experimental results showed significant improvements in various temporal knowledge graph completion tasks.", "summary_of_the_review": "In summary, this paper proposed a framework that enhances the temporal knowledge graph embedding with pre-trained language models and jointly optimizes the TKGC task with knowledge-text prediction loss (KTP). The proposed model has demonstrated decent experimental improvements in the temporal knowledge graph completion tasks.  However, this paper lacks sufficient details on the model structure. It is not clear how the pre-trained transformer model combines with tKG model. \n\n\nBesides, the analysis section of the paper can also be improved. For example, in the qualitative analysis part, the examples only showed that ECOLA-DE benefited from the extra relevant textual training data. Whereas this finding is somewhat superficial, and it is not directly related to the proposed KTP loss. Further analysis can be conducted on the underlying reasons behind ECOLA’s strong empirical performance.\n\n==============================\nUpdates on Nov 30. \n\nI appreciate the clarification of the authors, which help me understand some parts better. The authors also mentioned that they will update their paper accordingly. Therefore, I made an adjustment to my recommendation score. Overall, the paper's technique contribution is not significant enough, though the proposed method for combining the two sources improved the performance. ", "strength_and_weaknesses": "Strength: This paper proposed a technique to enhance the temporal knowledge graph embeddings by contextualized language models. The motivation is reasonable and experiment results have demonstrated decent improvements for the temporal KG completion tasks.\n\nWeakness: \n1. Enhancing knowledge graph embeddings by contextualized language model embedding itself has been widely explored, even for enhancing temporal knowledge graph embedding [1]. \n\n2. It is unclear how the transformer encoder (BERT) combines with the tKG model in Figure 2. From section 3.1, we know the quadruple representation can be obtained by the BERT encoder. However, the introduction of DyERNIE is confusing. Since the embedding of DyERNIE is derived from a static embedding and a velocity vector, how does this equation link to the notations in section 3.1? The title of section 3.1 is ‘Embedding layer’, whereas the final representation of BERT should be dynamic contextualized embedding since the BERT encoder is updated by the KTP (MLM-like) loss. In figure 3, the time expression 2019.08.02 is an input to the transformer, but it seems to be treated differently compared to the other words. The notation tau mentioned in figure 3 was not explained elsewhere, whereas tau is related to the most important temporal aspect of this model. \n\n\n[1] Mavromatis, Costas, et al. \"Tempoqr: temporal question reasoning over knowledge graphs.\" Proceedings of AAAI. 2022.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AHGz984IwIX", "reviewer": "Reviewer_cTPN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly addresses the problem of learning the representations of temporal knowledge graphs (tKGs) which contain time-evolving relations between the subject and object. To alleviate the sparsity of the tKG and learn better embeddings, the authors propose to extend the MLM objective so that during training, the knowledge quadruples are paired with corresponding texts and some of the tokens or entities are masked. The model is trained to predict the mask tokens and the mask entities or relationships. This way the model can build latent connections between two embeddings spaces.", "summary_of_the_review": "Overall the paper is clear and well-motivated. It has strong empirical results and the results can support the claim that the proposed method is model-agnostic. Some analysis is conducted and provides some insights. Some of the remaining research questions need to be answered (see questions below).\n\nQuestion:\n1. What do you mean by “we solve the temporal alignment challenge by using tKG quadruples as an implicit measure. ” Can you be more specific with the term “implicit measure”?\n\n2. Based on the results we can see that the ECOLA mainly leverages the language model to improve the KG embeddings. I wonder if the performance can be further improved if larger language models are leveraged since BERT base is relatively small.\n\n3. Does the quality of the text significantly affect the model’s performance? If in another domain there is no high-quality paired text, can the proposed method still improve the KG embeddings?", "strength_and_weaknesses": "Strength:\n\n1. The paper is well-written and easy to follow. The motivation to consider time-evolving relations in KGs is clear. Leveraging the rich textual information to enhance the embeddings of knowledge graph is interesting.\n\n2. Propose a simple yet effective method to improve the representations of tKGs and the experiment results show the ECOLA can improve embeddings from various embedding methods and achieve large improvement.\n\nWeakness:\n\n1. Novelty is the main concern. The main method ECOLA extends the MLM objective to predict both masked tokens and entities or relationships. The Model Variants are merely combinations of ECOLA with different previous knowledge embedding methods.\n\n2. Not clear what benefits the language model gain after the joint training. The paper can be stronger if the author can show that the proposed ECOLA can benefit the language model as well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject