Temporal Knowledge Graph Reasoning Based on N-tuple Modeling
Authors: Zhongni Hou
Xiaolong Jin
Zixuan Li
Long Bai
Saiping Guan
Yutao Zeng
Jiafeng Guo
Xueqi Cheng
Venue: EMNLP 2023 Findings (2023)
Keywords: Knowledge graph
n-ary temporal knowledge graph
graph convolution network

Abstract: Reasoning over Temporal Knowledge Graphs (TKGs) that predicts temporal facts (e.g., events) in the future is crucial for many applications. The temporal facts in existing TKGs only contain their core entities (i.e., the entities playing core roles therein) and formulate them as quadruples, i.e., (subject entity, predicate, object entity, timestamp). This formulation oversimplifies temporal facts and inevitably causes information loss. Therefore, we propose to describe a temporal fact more accurately as an n-tuple, containing not only its predicate and core entities, but also its auxiliary entities, as well as the roles of all entities. By so doing, TKGs are augmented to N-tuple Temporal Knowledge Graphs (N-TKGs). To conduct reasoning over N-TKGs, we further propose N-tuple Evolutional Network (NE-Net). It recurrently learns the evolutional representations of entities and predicates in temporal facts at different timestamps in the history via modeling the relations among those entities and predicates. Based on the learned representations, reasoning tasks at future timestamps can be realized via task-specific decoders. Experiment results on two newly built datasets demonstrate the superiority of N-TKG and the effectiveness of NE-Net.

Temporal Knowledge Graph Reasoning Based on N-tuple Modeling
Zhongni Hou! 1.2
Xiaolong Jin 122, Zixuan Li Long Bai", Saiping Guan', Yutao Zeng', Jiafeng Guo and Xueqi Chengl?
School of Computer Science and Technology, University of Chinese Academy of Sciences; "CAS Key Laboratory of Network Data Science and Technology.
Institute of Computing Technology, Chinese Academy of Sciences; Teeneett Inc. wiw.aaiioooonniiiuaannaa lizixuan, haiiiuan,bbilooggeeeo... guansaiiinnnnnasaeeeaneee ac. cn (guojiafeng, cxq)@ict ac. cn; yutao, www.yuttaazeegguuoooooooooo com com
(houzhongni 18 jinxiaolong,
Abstract
(America, Demand,
(America, Criticlze, Vietnam, &
Prance, Consult England, f
Ira.
Reasoning over Temporal Knowledge Graphs (TKGs) that predicts temporal facts (e.g., events) the future is crucial for many applica- tions. The temporal facts in existing TKGs only contain their core entities (i.e., the entities play- ing core roles therein) and formulate them as quadruples, i.e. (subject entity, predicate, ob- ject entity, timestamp). This formulation over- simplifies temporal facts and inevitably causes information loss. Therefore, we propose to de- scribe a temporal fact more accurately as an n- tuple, containing not only its predicate and core entities, but also its auxiliary entities, as well as the roles of all entities. By so doing, TKGs are augmented to N-tuple Temporal Knowledge Graphs (N-TKGs), To conduct reasoning over N-TKGs, we further propose N-tuple Evolu- tional Network (NE-Net). It recurrently learns the evolutional representations of entities and predicates in temporal facts at different times- tamps in the history via modeling the relations among those entities and predicates. Based on the learned representations, reasoning tasks at future timestamps can be realized via task- specific decoders. Experiment results on two newly built datasets demonstrate the superiority of N-TKG and the effectiivneess of NE-Net.
Hold Visit. England,
(lemanyy Consult, Franze,
(America, Sunction. Iran. h
(a) Quadraples TKG.
Demand,
(Criticize,
(Consult:
Critic: America.
Critic: America.
Inquirer: Hunnc,,
Tarpet Iran.
Target: Vieezam.
Advisor: England. sy)
Demand Content: Oil,,
Content: Human Right.
(Sanction.
(Hold a Visit.
(Consult
Sunctioner.Ameriia
Visitor: France.
Client: Germany.
Recipiens: Iran.
Host: England.r)
Consultant: Franco.ry)
Operation: Expor..))
(b) N-toples N-TKG.
Figure I: A comparison of quadruples in TKG and n- tuples in N-TKG.
observed history. Such task can be applied to many practical applications, including disaster relief (Sig- norini et al., 201 1), financial analysis (Bollen et al., 201 I), etc.
Actually, besides its core entities (i.c., subject entity and object entity), temporal fact often in- volves other auxiliary roles and the corresponding arguments (i.e., entities). As illustrated Figure I, the "Demand" event involves not only the core roles and entities, but also the auxiliary role "De- mand Content" and its corresponding entity "Oil". Moreover, the number of roles in temporal facts may be variable. For example, the "Hold a Visit" event contains two roles, while the "Criticize" event has three. Above all, the existing quadruple-based formulation cannot describe temporal facts accu- rately, thus limiting the applications TKGs.
1 Introduction
Knowledge Graphs (KGs), which represent facts in the form of triples (Bordes et al., 2013; Dettmers et al., 2018), i.e., (subject entity, predicate, object entity), have attracted increasing research attention in recent years. As the validity of facts can change over time, Temporal Knowledge Graphs (TKGs) extend triples into quadruples (Han et al., 2020b; Park et al., 2022), i.e., (subject entity, predicate, ob- ject entity, timestamp), to represent temporal facts, such as events. Reasoning over TKGs aims to an- swer queries about future temporal facts, such as (America, Sanction, 2, 2024-1-100 based the
Motivated by these, we propose to describe temporal facts more accurately as n-tuples and correspondingly augment TKGs as N-tuple Tem- poral Knowledge Graphs (N-TKGs), so as not to cause information loss. Specifically, cach n- tuple in N-TKGs is denoted in form of (predicate, role,:entity:
timestamp). For example, the event "Consult" in Figure/ is described as (Con- sult, Client: Germany, Consultant: France, Consult Content: Iran, tz).
"Corresponding authors.

---

Similar to that of TKGs, reasoning over N-TKGs is an important task for their practical upplications. However, existing methods for the reasoning task on either TKGs with quadruples or static KGs with n-tuples have limitations when facing N-TKGs. In more detail, the reasoning methods for quadruple- based TKGs (Dasgupta et al., 2018; Goel et al., 2020; Jin et al., 2020; Han et al., 2020b; Park et al., 2022) cannot be directly applied to n-tuples and have to take adaptation measures. Those reason- ing methods for static n-tuple KGs (Rosso al., 2020; Guan et al., 2020; Liu et al., 2021; Guan et al., 2019) cannot capture temporal information contained in those facts at different timestamps.
tional representations via capturing the relations among entities and predicates formed upon con- current facts and simultaneously emphasizing the relations between core entities. NE-Net further utilizes task-specific decoders to address different reasoning tasks; (3) Experiments on the two new datasets demonstrate the superiority of N-TKG and the effectiveness of NE-Net.
Related Works
Static N-tuple KG Reasoning. Static n-tuple KG reasoning aims to infer the missing elements of given n-tuple. Existing methods can be di- vided into three categories, namely, hyperplane methods, multi-linear methods and neural meth- ods. Hyperplane methods (Wen al., 2016; Zhang et al., 2018) project entities into relation hyper- planes to calculate the plausibility scores for n- tuples. Multi-linear methods (Liu et al., 2021) ap- ply the multi-linear product to measure the plausi- bility scores. Neural methods (Guan et al., 2019; Galkin et al., 2020) leverage CNN or GCN to cap- ture the relatedness scores of role-entity pairs in n-tuples. Particularly, Neulnfer (Guan al., 2020) and HINGE (Rosso et al., 2020) notice that differ- ent elements in n-tuples are of different importance and propose to represent an n-tuple as a main triplet along with auxiliary role-entity pairs. However, the occurrence time of temporal facts is viewed as a common kind of information about the facts and is usually ignored by the above methods (Rosso et al., 2020). Therefore, these methods cannot model the temporal behavioral patterns across adjacent timestamps and are not effective enough to exploit historical data for future predictions.
To solve the above problems, propose model called N-tuple Evolutional Network (NE- Net) to conduct reasoning tasks over N-TKGs, NE- Net consists of an entity-predicate encoder and task-specific decoders. The entity-predicate en- coder is used to learn the evolutional representa- tions of entities and predicates in different temporal facts. Specifically, at each timestamp, it employs an
unit to capture the relations among entities and predicates formed upon con- current facts. It also adopts a core-entity unit to emphasize and more directly model the relations between core entities (in terms of predicates) of individual temporal facts, as such relations contain the most primary information of the facts. Next, an aggregation unit is utilized to integrate the outputs of these two units so to obtain more accurate rep- resentations of entities and predicates. Note that the encoder employs a recurrent mechanism to auto- regressively learn the evolutional representations from the facts at temporally adjacent timestamps, and thus implicitly reflects temporal behavioral pat- terns of entities in their evolutional representations.
TKG Reasoning. There are two different task set- tings for TKG reasoning. namely, interpolation and extrapolation (Li al., 2022c; Han et al., 2021b; Sun al., 2021; Li et al., 2022b,a). TKG rea- soning under the former setting to infer miss- ing elements of facts at known timestamps (i.e.. the history) (Sadeghian et al., 2016; Esteban et al., 2016; Han et al., 2020a; Leblay and Chekol, 2018). Under the interpolation setting, HyTE (Dasgupta et al., 2018) extends the idea of TransH (Wang et al., 2014) and associates each timestamp with corresponding hyperplane. DE-DistMult (Goel et al., 2020) and DE--SmpllE (Goel al., 2020) both utilize a diachronic embedding for entities and relations, dividing the representations of enti- ties into a static segment and a varying seg-
NE-Net finally leverages task-specific decoders to conduct different reasoning tasks, namely, pred- icate reasoning and entity reasoning, respectively, based on the learned representations of entities and predicates. addition, as there is no N-TKG dataset publicly available, we construct two new datasets, named NWIKT and NICE, to facilitate the research on reasoning over N-TKGs.
In summary, contributions are follows: (1) We propose use n-tuples to describe temporal facts more accurately, and further enhance TKGs as N-TKGs; (2) We propose NE-Net to conduct the reasoning tasks over N-TKGs. It leverages an entity-pree prediccte encoder to learn accurate evolu-

---

Ennity-Predicae Ficuuder
Tink specific Decoders
Predlcatte Rezocrring
Guarry
COmPOAN
COmpOEN
Huri
Eniiyy Reasoning Decoder
IR
(Cassuli
Aiviio
uxb
and
(Sasction.
Respiar::
Opention Oilio
R
Figure 2: An illustrative diagram of the proposed NE-Net model for N-TKG reasoning.
ment. TKG reasoning under the latter setting, which this paper focuses is to infer missing elements of facts at future timestamps (Han al.. 2020b, 2021a: Park al., 2022). RE-Net (Jin et al., 2020) and REGCN (Li al., 2021b) both adopt a relation- aware GCN and recurrent com- ponent to capture temporal associations within the history. CyGNet (Zhu et al., 2020) applics time- aware copy-generation network model the repet- itive pattern of frequently occurring facts. Fo- cusing the explainable TKG reasoning meth- ods, TTTer (Sun et al., 2021) and CluSTeR (Li al., 2021a) employ reinforcement learning to adaptively search informational facts the history. More recently, CEN (Li et al., 2022b) utilizes a length-aware CNN to mine complex temporal pat- terns of different lengths. CENET (Xu et al., 2023) employs the contrastive learning strategy to identify potential entities from historical and non non-historical dependency. the above methods, the facts all in the form of quadruples. Therefore, they cannot directly model the temporal associations involving auxiliary entities.
it plays the role as Pi C Vp: and i. are the total number of role-entity pairs and the timestamp of the n-tuple f. respectively, Note that in any fact E Fr, E1 and e2 are the two core entities, corre- sponding to its subject and object ones, respectively. Moreover, in this paper, the same type of facts is denoted with the same predicate. Therefore, pred also denotes the type of f.
Predicate reasoning. It aims to predict the type of the fact (i.e., predicate) that will occur on the given entity C2 the next timestamp 1. In other words, it aims answer the query like (?,-:-....,
Cxy ....-... -, L 1+1). Here, only the target entity ex is available and it can be any entity in the fact. Entity reasoning. It aims to predict the unknown entity playing the given role Pq in a fact of the type specified by predicate pred at the next timestamp Ê + 1. Formally, this task aims answer the query like (pred, pi :e,, P2 :0: Py:?,... ... Pu. : Cas 1). Here, all Pi and e (1 < and A q) are given.
The NE-Net Model
As illustrated in Figure 2, NE-Net consists of an entity-predicall encoder and task-specific decoders. The entity-predicate encoder recurrently models the relations among entities and predicates gener- ated by concurrent facts the history and outputs the evolutional representations of all entities and predicates. Based on the learned representations, task specifc decoders are adopted to conduct dif- ferent reasoning tasks.
3 Problem Formulation
N-TKG. An N-TKG & can be formalized as se- quence of KGs with timestamps, i.e., 9 (91,92, ...... The KG, Gr, at timestamp can be denoted as G: - (Vpred, Vent, Vp. Ft), where Vpred, Vont, Vp F, are the sets of predicates, en- tities. roles and temporal facts (hereinafter referred as facts if not causing any confusion) occurring timestamp 6, respectively. Each fact f F, denoted as (pred, P1 01: .... Pe: Ct . On5 t), where pred E Vpred is its predicate; each e; E Vent (1 < i S n) is an entity involved this fact, where
4.1
The Entity- -Preicctr Encoder
There are mainly two kinds of information in his- torical facts, namely, the explicit relations among entities and predicates in concurrent facts at the

---

same timestamp, and the implicit temporal behav- ioral patterns of entities reflected in different times- tamps. The entity-predicate encoder incorporates these two kinds of information into the represen- tations of entities and predicates, via recurrently modeling the KG sequences. In this paper, we con- sider the history at the latest m timestamps, i.e., (9-m+11 9t-m+2, Gt), where t is the current timestamp.
ber of predicate instance nodes in the EPG, each of which corresponds to a specific fact. By do- ing, the EPG at timestamp T can be formulated as G, = (Vens U Viread E,). where Vent- Vired and Fir are the sets of entity nodes, predicate instance nodes, and edges, respectively. Here, each predi- cate instance node is only associated with the enti- ties in the corresponding fact. And, there is a map- ping function (p+): Viredd Vprred which maps the predicate instance node v to its type (p()) E Vpred to indicate which kind of predicate belongs to. Each edge eij- = (ei, Ug) E E, links the entity node 2 and the predicate instance node by via the corresponding role p. Note that for those entities and predicates not involved in any fact at timestamp self-loop edges are added.
4.1.1 Entity-Predicate Modeling in Concurrent Facts
In this paper, by denoting temporal fact f as (pred, p1:e1, Pi: Et; : en, t), its predicate pred forms natural relations with its all entities 6₁ (1 S i < n) specified by Pt. respectively. Such relations between the entities and predicate pre- serve the relatively complete information of the fact and are referred to as the entity-predicant re- lations. Among these relations, those between the predicate and core entities are more important than others, they reflect the primary information of the fact and should thus be given more attention. Besides these relations within a fact, by sharing the same entity, the predicates and entities in different facts, being concurrent in the same timestamp, may form more complicated structures.
Upon the EPG, the EP unit works with two steps, namely, the message-passingg step and the pred- aggregation step. It finally updates and outputs the entity representation matrix Hr and the predicate representation matrix H pred at timestamp т.
step employs a CompGCN (Vashishth et al., 2019) with wi layers, to update the representations of entities and predicate instances. Specifically, the representation of the node v at layer E [0,w] 1] is obtained follows:
The message-
To capture the above relations and structures so as to finally learn the representations of entities and predicates, the encoder first utilizes an Entity- Predicate unit ("EP unit" in Figure 2) to model the relatively complete entity-predicate relations in concurrent facts. It then utilizes a Core-Entity unit ("CE unit" in Figure 2) to highlight the re- lations between core entities. Finally, the above relational information captured by the EP and CE units is integrated via an Attention baasdd Aggre- gation unit ("AA unit" in Figure 2) to obtain more accurate evolutional representations of entities and predicates.
h hottl= 1
Σ +W] (1)
(u.p.))CG,
where h denotes the representation of node v obtained after / layers: p denotes the representation of the role p W and W are the parameters the 1-th layer; vi(-) is an activation function. The representations of predicate instance nodes at the first layer are obtained by looking up the input evo- lutional representation matrix of predicates gred according to its type. In particularly, at the first timestamp m + 1, the randomly initialized pred- icate representation matrix H Hpred is used as the in- put. The representations of entity nodes at the first layer are calculated by the CE unit, which will be introduced in the following. The representations of roles are obtained from the role representation ma- trix Hp, which is randomly initialized and shared across timestamps.
The Entity-Predicee Unit. In order to em- ploy the EP unit to model the relatively complete prediy- relations in all concurrent facts at timestamp T (L m < 7 < m), we first need to construct an Ennity-Prediice
Graph ("EPG" in Fig- ure 2) based on them. To do so, for each fact, we denote its predicate and all entities the nodes of the EPG and the entity-predicate relations as its edges. As aforesaid, in this paper, the facts of the same type use the same predicate. There- fore, in order to avoid confusion between different facts of the same type, we create the same num-
At timestamp т, the pred-aggregation step up- dates the evolutional representation of a predicate hired by aggregating the information of all related nodes. For a predicate pred, after the message passing step, its different predicate instance nodes have different representations due to the different

---

involving entities. NE--Net utilizes the mean pool- ing (MP) operation to summarize the information of its predicate instance nodes h pred
The Aftention-based Aggregation Unit. To integrate the information in the above two kinds of relations, i.e., Hent and Hents the attention- based aggregation unit utilized to learn impor- tance weights of the two kinds of information, and adaptively combine them in order to obtain the final evolutional representations of entities Hent at timestamp Specifically, take the entity as example, its evolutional representation h is calculated as follows:
Upreed MP
Σ
Riggred
(2)
vE(Ulppo)==pred)
Besides, for a predicate pred, to preserve the in- herent semantic information, NE-Net further in- corporates the initial representation Apreds into its evolutional representation h hpeed at timestamp T as:
h = Mi aj Ri,ent
(5)
Ripred g(Pepred hprea) Walhered (eprodd; (3)
where ha,ent and h are the updated representa- tions of entity i outputted by the above two units, aitent and âs,ent measure the importance weights of two representations. Taking the representation example, NE-Net first gets its impor-
where W2 is learnable parameter matrix and denotes the concatenation operation.
The Core-Entity Unit. As aforementioned, the most important information in a fact is the rela- tion between its two core entities in terms of the predicate, which is more important than other rela- tions. To emphasize these relations between core entities more directly, this unit views predicates edges, and constructs the Core Entity Graph ("CEG" in Figure 2) for concurrent facts times- tamp т. Specifically, the CEG at timestamp T can be formulated as G, = (Vent, E,), where Vent and Fr, are the sets of core entity nodes and edges, re- spectively. Each edge e = (ut, pred, v&) E Er links the two core entity nodes v/ and 2) corre- sponding to certain fact occurring at timestamp T via its predicate pred of an n-tuple. Similarly, for those entities not involved in any fact, self-loop edges are added.
h h tance value पे. Tentt
W./en = f(h = qtanh(WWaaa (Wyhi + b). (6)
where Wa is the weight matrix, q is the learnable importance gate. Similarly, w.nnnt =f fh
After obtaining the importance values of each representation, NE-Net normalizes them to get the final importance weights, i.e., a.,nne and d. via the softmax function.
4.1.2 Entity-Predicate Modeling across Different Timestamps
The temporal patterns hidden in the historical facts of a specific entity implicitly reflect its behavioral trends and preferences. As the temporal facts at a specific timestamp are already modeled in the lutional representations of entities and predicates via the above three units, NEiNet directly uses the evolutional representations of entities and predi- cates learned from timestamp 7 1, namely, H pred as the input of interaction modeling step for concurrent facts. By recurrently model- ing on KGs at adjacent timestamps, the temporal behavioral patterns of entities can be modeled.
Based on the constructed CEG T, and the input representation matrix of entities H at timestamp T 1, this unit leverages an RGCN (Schlichtkrull et al., 2018) with w2 layers to encode the relations between core entities into the updated entity representation matrix H T
and H
ii (CCN G.),
(4)
where HTT denotes the updated entity representa- tion matrix obtained at the [-th layer. The represen- tations of entity nodes at the first layer are obtained by looking up the input evolutional representation matrix of entities H 1 For the first timestamp t f- m + 1, the randomly initialized entity repre- sentation matrix Hent used as the input. As the relations between core entities maintain the most primary information of temporal facts, NE-Net di- rectly uses the output of this unit, i.e., H as the input entity representation matrix of the entity- predicate unit to emphasizing such information.
4.2 Task-specific Decoders
Based on the evolutional representations of enti- ties and predicates at timestamp t, NE-Net utilizes task-specific decoders conduct different types of reasoning tasks.
4.2.1 Predicate Reasoning Decoder
Give the query of predicate reasoning (?,-i-.....- Cay...--: : -, + 1), NE-Net multiplies the evolu- tional representation of e, at timestamp with the evolutional representations of predicates H
pred

---

generate the conditional probability vector p(predle,, G1a):
pi(predle, e, G1x)) represents the probability score of the i-th predicate. Similarly, the objective of the entity reasoning task is to minimize the following loss:
p(predles: Gra) = o (H))))))
(7)
where (-) is the sigmoid function.
[Vene--1
4.2.2 Entity Reasoning Decoder
Leans
ant 3641, logpsel
(11)
To deal with queries having a varied number of role-entity pairs and capture the relations among elements within the query, NE-Net reorganizes role-entity pairs in query sequence and then designs a Transformer-basee decoder for the en- tity reasoning task. Specifically, given query (pred, P1: cl,-... Pq :?, .... Pn Cny (++), NE-Net first replaces the missing entity with a special to- ken [MASK], and linearize the query as: X- (pred, P1,,1,.... Pq. [MKS en) To iden- tify which kind of element the model is dealing with, each element æ; E X is assigned with a type type: {0, 1). where 0 represents a predicate, represents role, and 2 represents entity. The input representation of the i-th element in the se- quence, i... h.. is calculated by:
t=u FEFF+1
pred, Pt Cir Po: Pe: Cns 91+),
Experiments
5.1 Experimental Setup
5.1.1 Datasets
Since there is no N-TKG dataset available for ex- periments, build two datasets, namely NWIKI and NICE.
About NWIKI: (1) We downloaded the Wikidata dump dump' and extracted the facts which contain the timestamp information and involve human enti- ties; (2) To extract n-tuple facts, we selected predi- cates with more than 3 role types and retained their corresponding facts; (3) We filtered out entities of low frequency, as their behaviors are difficult learn and predict for most data-driven based meth- ods. Correspondingly, facts involving these entities were also filtered out; (4) The fact related to the "Position Held" predicate comprise over 50% of the dataset. However, the filtering operation in the previous step could impair connectivity. To main- tain robust connection with other predicates, we retained facts that associated with other predi- cates and share core entities with the facts related to the "Position Held" predicate: (5) Following RE- NET (Jin et al., 2020), all facts were split into the training set, the validation set, and the test set by a proportion of 50%:100%:109 according to the time ascending order.
h a + type,
(8)
where af is generated by looking up the evolu- tional representation matrices of entities and predi- cates, i.c., He and H. Hr and the role representa- tion matrix, i... Нp type; is obtained by looking up the learnable type representation matrix H Hyype- Then, all input representations are fed into a stack of L successive Transformer blocks (Vaswani et al.. 2017). Based on the output of Transformer, the final representation of [MASKJ, i.e., hg can be obtained, and is multiplied with the evolutional rep- resentations of entities H to obtain a probability vector over all entities.
ple pred, pI pr - Cn.. = oHHinhh) (9)
About NICE: (1) We collected the raw event records from ICEWS from Jan 1, 2005 to Dec 31, 2014; (2) From these raw event records, we extracted core entities, timestamps, and partial aux- iliary information, i.e., the occurrence places. (3) We observed that the predicate names contain addi- tional auxiliary information. For example, the pred- icate names "Cooperate Economically" and "En- gage in Judicial Cooperation" contain the specified cooperation aspects of the general predicate "En- gaged in Cooperation", namely "Economic" and
where (-)) is the sigmoid function.
4.3 Model Learning
Given a fact f = (pred, Pi:e1, Pn en, ++1) E
Fi+1, let denotes any entity in f, T denotes the number of timestamps in the training set. The ob- jective of the predicate reasoning task is to mini- mize the following cross-entropy loss:
T-1
Mareal-1
Ipreet
spred e, (10)
FEFF1
20171 wwwp:i/wwwwrggeelell/iiiibeeeee /hups://datavese
where Voro is a 0/1 value, denoting whether the i-th predicate occurs on at timestamp t +
htw.://ddaaaeeee..aaaaddd harvard. had.raarddddddaaaveee//ceww

---

#Train
#Valid
#Test
Dataset
1/roo
Vool
TEmestamps
Time Interval
Binary
Nary
Overall
Binary
Nary
Overall
Binary
Nary
Overall
NWIKI
22
17,481
20,686
87,711 322,692
108,397
1,847
12.523
14,370
2,438
13,153
15,591
205
year
NICE
20
10,860
46.176
368,868
5.395
40,907
5,268
40.891
46,302
46.159
4,017
24 hours
Table 1: The statistics of the two proposed datasets, NWIKI and NICE.
NWTKI
SICF
soning methods include DE-DistMult (Goel et al., 2020), DE-SimplE (Goel al., 2020), HyTE (Das- gupta et al., 2018), RE-NET (Jin al., 2020), CyGNet (Zhu al., 2020), REGCN (Li al., 2021b), GHT (Sun et al., 2022). CEN (Li et al.. 2022b) and TiRGN (Li et al., 2022a). To facilitate this kind of methods to N-TKG, we convert the historical n-tuples into a KG sequence. In each KG, we view both entities and predicate as nodes, and link entity nodes with fact nodes via roles. Such KG sequence is taken the input of TKG reason- ing methods.
Recall
MELNN
$3.20
55.76
25.33
BRENN
$4.00
61:62
56.11
5.91
27:23
3:45
MLARAM
$3.50
61:14
55.55
8.48
45.52
12.23
DNN
F-CC
73,20
75.27
72.04
16.70
39.87
27.48
RENET
76.17
93:60
$3.30
43.62
20
18.82
NE.Net
78.34
94.86
$4.29
32.43
85.04
45.78
Table Experimental results on predicate reasoning.
"Militarily", respectively. To obtain such auxil- iary information, we designed rule templates to extract them from the specified predicate names and merged these specified predicates into the gen- eral ones. (4) Considering the absence of role in- formation in the raw event records, we manually assigned role names to each predicate. (5) Similar to NWIKI, NICE was also split into the training set, the validation set, and the test set by proportion of 80%:10%::0%%
5.1.4 Implementation Details
For the evolution encoder, the history length m on two datasets set to 2; the number of Transformer blocks for NICE and NWTKT is set to 1 and 2, respectively. For all datasets, the numbers, Wi and w2, of GCN layers in the entity-predicate and core- entity units are set to 4 and 2, respectively; the number of self-attention heads is set to 4.
5.1.2 Evaluation Metrics
We employ FI, recall, and precision as metries for the predicate reasoning task. We adopt Hits (1, 3, 10} and MRR as metrics for the entity reasoning task. Following Han et al. (2021a), we report the results under the time filter setting.
5.2 Experimental Results
5.2.1 Results Predicate Reasoning
To analyze the effectiveness of NE-Net on predi- cate reasoning task, we compare NE-Net with base- lines on both datasets. The results are presented in Table 2. It is shown that NE-Net outperforms baselines across all datasets in terms of F1 and Re- call. Notably, the counting- the worst performance both datasets, indicating the importance of modeling the relations among entities and predicates at different timestamps in the predicate reasoning task. can be particularly observed that NE-Net shows lower performance on the precision metric in NWIKI, as compared with RENET. This is because RENET focuses on target entity to retrieve historical facts, enabling it to utilize a longer history, while NE-Net solely considers the history in the latest timestamps.
5.1.3 Baselines
For the predicate reasoning task, we compare NE- Net with temporal reasoning models, including RENET (Jin et al., 2019) and T-GCN (Zhao et al., 2019). Following Deng et al. (2019), we further compare NE-Net with DNN, MLKNN (Zhang and Zhou, 2007), BRKNN (Spyromitros et al., 2008), MLARAM (Deng et al., 2019). These four models are simple DNN or KNN-based methods, which use basic count features to conduct reasoning tasks. For the entity reasoning task, we compare NE- Net with static n-tuple KG reasoning methods and TKG reasoning methods. The static n-tuple KG reasoning methods include: NALP (Guan et al., 2019), NeuInfer (Guan et al., 2020), HypE (Fatemi al., 2021), HINGE (Rosso al., 2020), Hy- Transformer (Yu and Yang. 2021), RAM (Liu et al., 2021). Since these methods are not able to model the temporal information, we construct a cumula- tive graph for all the training facts. The TKG rea-
methods have
5.2.2 Results Entity Reasoning
To investigate the effectiveness of NE-Net, we di- vide the test set into core and auxiliary categories (denoted as C and AUX) based on the role to be predicted, and conduct the entity reasoning task

---

NWIKT
NICE
Model
HiG1
H6r
H
MRR
HGR
H
H 10
MRR
AUX
C
AUIX
AIX
C.
AIIX
C:
AUXX
AIXX
AUXX
AUX
NALP
10.59
7.54
10.59
10.84
22.55
15.17
14.86
10.25
14,66
39.82
26.
09.48
43.4
19.87
25.96
46,92
Nenlofer
19.83
6.07
25.22
8.62
28.
11.91
23.08
8.14
13.77
14.48
28.
18, 30
47.56
29.59
24.78
19.32
HINGE
10.85
23.47
14.05
25.91
19.88
21.74
13.83
15.88
04
27.85
42.88
43.52
24.89
RAM
$1.42
22.68
33.36
26.01
34.36
28. 14
32.63
24.72
8.37
14.48
16.41
23.26
27. 15
47.64
14.38
23:45
HypF
24.91
19.83
25.75
19
25.20
63.35
Hy- Tr-nnssformo
33,40
18.36
35.85
23.60
37.84
27.65 27.60
34.97
21.66
28.51
61.02
44.49
82.46
61.11
91,38
59.47
72.71
DE-DisiMult
16.10
DE-SimplE
10.87
16.41
19:14
13:87
11.53
21.86
34.90
19.30
Hy
21.82
RENET
33,56
38.41
41.28
36.57
33.43
47.77
63.06
43.32
CyNet
64.71
53.12
36.81
REGEN
46.25
155.13
72.31
56.78
37.33
53.85
68.27
48,03
GHIT
37.78
CEN
30.28
45,20
61.04
40.61
33.32
49.29
64,65
43,98
TIGNN
61.10
NE.Net
66.87
46.45
76.08
66.01
80.29
77.32
72.03
$7.68
38.36
68.55
54.18
$8.16
69.99
94.61
48.98
79.06
Table 3: Experimental results on the entity reasoning task for predicting core and auxiliary entities, respectively.
on two datasets. Comprehensive results on these two categories are presented in Table 3. Especially, we only report the results of TKG reasoning meth- ods on the core category, as they only focus on quadruples.
Preddgg
NE-Net
C
66.25
59.47
70.94
57.02
72,03
NWIKI
AUX
53.97
40.69
56.52
48.03
57.68
All
54.23
48.73
62.69
51.88
63.82
C
47.21
47.86
48.71
43.74
48.98
AUX
78.78
75.58
79.06
All
64.16
64.34
65.09
61.07
65.37
As shown in Tables 3, NE-Net consistently sig- nificantly outperforms static n-tuple KG reason- ing methods across all metrics on both datasets. This can be attributed to the capacity of NE-Net to model the rich relations among entities and predi- cates at different timestamps. On the core category, we observe that TKG reasoning methods under the interpolation setting, perform worse than NE-Net. These methods focus on predicting facts occurring at known timestamps, and not capture the rela- tions between entities and predicates within newly- emerged temporal facts. Moreover, NE-Net shows better performance than TKG reasoning methods under the extrapolation setting. Different from these methods, NE-Net learns the representations of the predicates they dynamic evolve, and em- phasizes the relations between core entities within facts containing various numbers of auxiliary en- tities. Besides, NE-Net can capture information of elements within queries. In short, these results convincingly suggest that NE-Net is effective in conducting the reasoning task over N-TKGs.
Table 4: MRR results by different variants of NE-Net.
Dataset
30%
60%
100%
C
48.24
67.85
71.25
72.03
NWTKI
AUX
10.44
34.39
52.41
57.68
AIl
26.62
48.71
60.47
63,82
C
47.65
47.91
48.37
48.98
NICE
AUX
77.86
78.09
78.47
79.06
All
64.10
64.34
64.76
65.36
Table 5: MRR results of NE-Net modeling different ratios of auxiliary information.
tions. Notably, -CE has more significant impact on the core category when compared with that on the auxiliary category, as CEGss predominantly em- phasize entities playing core roles in n-tuples. Fur- thermore, -CE has a greater performance drop on NWIKI in comparison to NICE. Notably, NWIKI has a higher proportion of facts that involve three or more role-entity pairs (46.32%), as compared to NICE (30.97%). As a result, NWIKI exhibits richer relations among entities and predicates. -CE makes it more difficult for the model to distinguish the most important information contained in the relations among entities in NWIKI.
5.3 Ablation Study
To verify the necessity of the attention-based ag- gregation unit (denoted as -AA), we simply use the max pooling operation to integrate the outputs of the entity-predicate unit and the core-entity unit. It can be seen that -AA yields worse results com- pared to NE-Net, which demonstrates the necessity of adaptively integrating the information learned from the entity- predicte unit and the core-entity unit. Also, the removal of the
To study the effectiveness of each module of NE- Net, we conduct ablation studies on all datasets. The results summarized in Table 4. It can be observed that removing the core-entity unit (denoted -CE) results in worse performance on both datasets, which illustrates that addition- ally capturing the relations between core entities can help NE-Net learn more precise representa-

---

History at
History at
Query at +
Answer
(Disapprove,
(Consult,
(Engage in Diplomatic Cooperation.
Opponent: Barack Obama,
Consulter:
Cooperator: Barack Ohama.
Proporent: Head of Government (Kenyu),
Comsulte:: Mulatu Teshone,
Eshiopia
Cooperator:
Disappooee Way: Denounce, Kenya)
Consult Way: Engage in Negotiation, Fihiopin)
Cooperation way: Praise Endorse)
History at
History at
Query
Answer
(Express Intent Cooperate,
Volunteer: Iran,
Cosperation Target: South Africa,
(Engage Diplomatic Cooperation.
(Provide
Cosperate Content: Engage Diplomatic Cooperation)
Cosperaton: Iran.
Provider: Iran.
(Express Intent to Cooperate,
Cooperator: South Africa,
Recipient:
Volunteer:
Cooperation way: Sign Agroement)
Content: Humanitarian Aid)
Cosperarion Target: Yemen,
Cooperati Content: Provide Humamitatian Aid)
Table 6: Case studies on the predictions of NE-Net.
operation (denoted as -PredAgg) also results in worse performance than NE-Net, which demon- strates that aggregating the influence of predicate instances and the inherent representations can help learn better representations of predicates. To verify the necessity of the Transformer-based decoder on the entity reasoning task, we replace this decoder with ConvTranst (Dettmers al., 2018), 8),, denoted as Trans. It can be seen that removing the trans- former decoder leads to decrease in performance, which demonstrates the effectiveness of our entity reasoning decoder.
5.5 Case Studies
We present two cases in Table 6 where NE-Net cor- rectly predicts the answer entity, In this table, t 1 and f represent historical timestamps, while ( represents the query timestamp. In the first case, the correct answer to the query plays as an auxiliary entity in the latest historical fact. is hard for the model to predict the correct answer without such auxiliary information. In the second case, the aux- iliary entity "Humanitarian Aid" helps the model establish a connection between the query and the most informative historical fact, and thereby en- abling more accurate predictions.
5.4 Detailed Analysis
Conclusions
In this paper, we proposed to utilize n-tuples to represent temporal facts more precisely, and corre- spondingly enhanced TKGs N-TKGs. We fur- ther introduced model called NE-Net, conduct reasoning over N-TKGs. NE-Net learns evolutional representations of entities and predicates, via mod- cling the relations among entities and predicates, and highlighting the relations among core entities. Further, it adopts task-specific decoders to conduct different reasoning tasks. Experimental results on two new datasets show the superiority of N-TKG and the effectiveness of NE-Net.
To illustrate the superiority of N-TKGs, we inves- tigate the performance of NE-Net with different ratios of auxiliary information. We provide NE- Net with histories containing varying ratios (0%, 30%, 60%,100%) of auxiliary information. Here, the auxiliary information in the history is randomly selected, and we run NE-Net three times and report the averaged results.
From Table 5, it can be observed that the per- formance of NE-Net is positively correlated with the amount of auxiliary information utilized. This suggests that providing more precise descriptions of temporal facts can enhance the performance of TKG reasoning, thereby demonstrating the superi- ority of the proposed N-TKGs. Additionally, we notice that the ratio of auxiliary information has greater impact NWIKI than NICE. Actually, the types of auxiliary roles in NICE are limited, while in NWIKI, there are more diverse types of auxiliary roles, including occupation, employer, winners, and more. These richer types of auxiliary roles in NWIKT can better help with predictions.
Acknowledgments
The work is supported by the National Natural Sci- ence Foundation of China under grant U1911401, the National Key Research and Development Project of China, the JCJQ Project of China, Bei- jing Academy of Artificial Intelligence under grant BAA12019ZD0300 and the Lenovo-CAS Joint Lab Youth Scientist Project. We thank anonymous reviewers for their insightful comments and sug- gestions.

---

8 Limitations
Saiping Guan, Xiaolong Jin, Jiafeng Guo, Yuanzhuo
Wang, and Xueqi Cheng. 2020. Neuinfer. Knowl- edge inference on n-ary facts. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 6141-6151.
The proposed NE-Net model cannot handle tem- poral facts that involve roles with multiple enti- ties, such as the temporal fact "(Attack, Attacker: (entity,, entityz), Victim: fentitys, entity,), ), 2026- 10-01>". Furthermore, it is unable to model long history, the use of deep GCN layers can bring the over-smoothing problem.
Saiping Guan, Xiaolong Jin, Yuanzhuo Wang, and Xueqi Cheng. 2019. Link prediction n-ary re- lational data. In The World Wide Web Conference, pages 583-593.
Zhen Han. Peng Chen, Yunpu Ma, and Volker Tresp. 2020a. Dyernie:
Dynamic evolution of riemannian manifold embeddings for temporal knowledge graph completion. arXiv preprint S.kiv:2011.039993
References
Johan Bollen, Huina Mao, and Xiaojun Zeng. 2011. Twitter mood predicts the stock market. Journal of computational science, 2(1):1-8.
Zhen Han. Peng Chen, Yunpu Ma, and Volker Tresp. 2020b. Explainable subgraph reasoning for forecast- ing on temporal knowledge graphs. In International Conference on Learning Representations.
Antoine Bordes, Nicolas Usunier, Alberto Gareia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In Advances in neural information processing systems, pages 2787-2795.
Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp. 2021a. Learning neural ordinary equa- tions for forecasting future links on temporal knowl- edge graphs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 8352 -3644.
Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. 2018 Hyte: Hyperplane-basse based tem- porally aware knowledge graph embedding. In Pro- ceedings of the 2018 Conference Empirical Meth- ods in Natural Language Processing, pages 2001- 2011.
Zhen Han, Zifeng Ding, Yumpu Ma. Yujia Gu. and Volker Tresp. 2021b. Temporal knowledge graph forecasting with neural ode. arXiv preprint arXiv:2101.01151
Songgaojun Deng, Huzefa Rangwala, and Yue Ning. 2019. Learning dynamic context graphs for predict- ing social events. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1007-1016.
Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. 2020. Recurrent event network: Autoregressive struc- ture inference over temporal knowledge graphs. In EMNLP.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowl- edge graph embeddings. In Thirty-Second AAAI Con- ference Artificial Intelligence.
Woojeong Jin, Changlin Zhang, Pedro Szekely, and Xi- ang Ren. 2019. Recurrent event network for reason- ing over temporal knowledge graphs. arXiv preprint arXiv:1904.055
Cristóbal Esteban, Volker Tresp, Yinchong Yang, Stephan Baier, and Denis KrompaB. 2016. Predicting the co-evolution of event and knowledge graphs. In 2016 19th International Conference on Information Fusion (FUSION), pages 98-105. IEEF
Julien Leblay and Melisachew Wudage Chekol. 2018. Deriving validity time in knowledge graph. In Com- panion Proceedings of the The Web Conference 2018, pages 1771-1776. International World Wide Web Conferences Steering Committee.
Bahare Fatemi, Perouz Taslakian, David Vazquez, and David Poole. 2021. Knowledge hypergraphs: pre- diction beyond binary relations. In Proceedings of the Twenty-Nine International Conference on Inter- national Joint Conferences on Artificial Intelligence, pages 2191-2197.
Yujia Li, Shiliang Sun, and Jing Zhao 2022a. Tirgn: time-guided recurrent graph network with local- global historical patterns for temporal knowledge graph reasoning. In Proceedings of the Thirty- International Joint Conference Artificial Intelli- gence, LJCAI 2022, Vienna, Austria, 23-29 July 2022. pages 2152-2155 ijcai. org.
Mikhail Galkin, Priyansh Trivedi, Gaurav Maheshwari. Ricardo Usbeck, and Jens Lehmann. 2020. Message passing for hyper-relational knowledge graphs. arXiv preprint arXiv:2009. 10847.
Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu. Yong Zhu, Long Bai. Wei Li, Jiafeng Guo, and Xueqi Cheng, 2022b. Complex evolutional pattern learning for temporal knowledge graph rea- soning, arXiv preprint 22X3v:203..07782.
Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker. and Pascal Poupart. 2020. Diachronic embedding for temporal knowledge graph completion. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3988-3995.
Zixuan Li, Zhongni Hou, Saiping Guan, Xiaolong Jin, Weihua Peng. Long Bai, Yajuan Lyu, Wei Li, Jiafeng Guo, and Xueqi Cheng. 2022c. Hismatch: Historical

---

structure matching based temporal knowledge graph reasoning. arXiv preprint arXiv:2210.09708.
Haohai Sun, Jialun Zhong, Yunpu Ma, Zhen Han, and Kun He. 2021. Timetraveler: Reinforcement learning for temporal knowledge graph forecasting. arXiv preprint
Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng Guo, Yuanzhuo Wang. and Xueqi Cheng. 202 Search from history and reason for future: Two-stage reasoning on temporal knowledge graphs. arXiv preprint arXi::2 106.00327.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. 2019. Composition- multi- relational graph convolutional networks. In Interna- tional Conference on Learning Representations.
Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. 202 Temporal knowledge graph reason- ing based on evolutional representation learning. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Infor- mation Retrieval, pages 408-417.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Zhen Wang. Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by trans- lating on hyperplanes. In Proceedings of the AAAI conference on artificial intelligence, volume 28.
Yu Liu, Quanming Yao, and Yong Li. 2021. Role-
aware modeling for n-ary relational knowledge bases. Proceedings of the Web Conference 2021, pages 2660-2671.
Jianfeng Wen, Jianxin Li, Yongyi Mao, Shini Chen, and Richong Zhang. 2016. On the representation and embedding of knowledge bases beyond binary relations. arXiv preprint 1rXiv:1604.088
Namyong Park, Fuchen Liu, Purvanshi Mehta, Dana Cristofor, Christos Faloutsos, and Yuxiao Dong. 2022. Jointly modeling event time and network structure for reasoning over temporal knowledge graphs. Proceedings of the Fifteenth ACM Interna- tional Conference on Web Search and Data Mining, pages 794 803.
Evokg:
Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. 2023. Tem- poral knowledge graph reasoning with historical con- trastive learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 37, pages 4765 4773.
Paolo Rosso, Dingqi Yang. and Philippe Cudré- Mauroux. 2020. Beyond triplets: hyper-relational knowledge graph embedding for link prediction. In Proceedings of The Web Conference 2020, pages 1885-1896.
Donghan Yu and Yiming Yang. 2021. Improving hyper-relational knowledge graph completion. arXiv preprint arXiv: 204.000000.........
Min-Ling Zhang and Zhi-Hua Zhou. 2007. MI-knn: A lazy learning approach to multi-label learning. Pat- tern recognition, 40(7):2038 2044
Ali Sadeghian, Miguel Rodriguez, Daisy Zhe Wang, and Anthony Colas. 2016. Temporal reasoning over event knowledge graphs. In Workshop on Knowledge Base Construction, Reasoning and Mining.
Richong Zhang. Junpeng Li, Jiajie Mei, and Yongyi Mao. 2018. Scalable instance reconstruction in knowledge bases via relatedness affiliated embed- ding. In Proceedings of the 2018 World Wide Web Conference, pages 1185-1194.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolu- tional networks. In European Semantic Web Confer- ence, pages 593-607. Springer.
I Ling Zhao, Yujiao Song, Chao Zhang, Yu Liu, Pu Wang, Tao Lin, Min Deng. and Haifeng Li. 2019. T-gen: A temporal graph convolutional network for traffic prediction. IEEE Transactions Intelligent Trans- portation Systems. 21(9):3888333
Alessio Signorini. Alberto Maria Segre, and Philip M Polgreen. 2011. The of twitter track lev- els of disease activity and public concern in the us during the influenza hlnl pandemic. PloS one,
69467
Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan Cheng, and Yan Zhan. 2020. Learning from his- tory: Modeling temporal knowledge graphs with quential
Eleftherios Spyromitros, Grigorios Tsoumakas, and Toannis Vlahavas. 2008. An empirical study of lazy multilabel classification algorithms. In Hellenic conference on artificial intelligence, pages 401 406-406. Springer.
networks. arXiv preprint arXiv:2012.08492.
Haohai Sun. Shangyi Geng, Jialun Zhong, Han Hu, and Kun He. 2022. Graph hawkes transformer for ex- trapolated reasoning on temporal knowledge graphs. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 7481-7493.


Reviews:
[{"review_id": "yZ7OCrT6CU", "reviewer": "Reviewer_vaso", "summary": "", "questions": "1. In Section 5.1.1, why did you filter out the entities of low frequency?\n\n2. Since NE-Net models include recurrent modeling (Modeling Interactions in Temporal Adjacent Facts) and Transformer modules, what is the time complexity of NE-Net?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the effort you have invested in reviewing our work. We are honored to address the concerns you've raised.\n\n**Q1**: The claim that existing quadruple-based formulation cannot describe temporal facts accurately is not convincing, which means the papers should provide experiments to compare how the N-tuple-based structure is better than the quadruple-based structure to support the claim.\n\n**A1**: Thank you for this valuable suggestion. As shown in the last block of Table 3, we present the results of NE-Net with different ratios of auxiliary information (0%, 30%, 60%, 100%). NE-Net (0%) denotes the NE-Net that models quadruple-based facts, while NE-Net (100%) denotes the NE-Net that models N-tuple-based facts (taking into account all auxiliary information considered). The superiority of NE-Net (100%) comparing to NE-Net (0%) justifies the necessity of introducing N-tuple based temporal facts. Furthermore, it can be noted that as the ratio of auxiliary information increases, so does the performance, further highlighting the superiority of the N-tuple-based structure.\n\nTo clarify the aforementioned discussion, we will reorganize Table 3 and Section 5.2.2. Specifically, we will segregate the results in the last block into a separate table and provide a more comprehensive analysis of the results in the paper.\n\n**Q2**: (1) The newly constructed datasets determine the upper border of model performance, but the descriptions of how to construct these two datasets are not clear, especially for the selection preference of predicates as the number of predicates is too small (22 for NWIKI, 20 for NICE). (2) In Section 5.1.1, why did you filter out the entities of low frequency?\n\n**A2**: We reorganized the construction process of the dataset and rewrote it as follows:\n\nThe NWIKI dataset was constructed as follows: (1) We downloaded the Wikidata dump until November 20, 2017, and extracted the facts which contain the timestamp information and involve human entities; (2) To extract n-tuple predicate instances, we selected predicates with more than 3 role types and retained their corresponding predicate instances; (3) We filtered out entities of low frequency, as their behaviors are difficult to learn and predict for most data-driven based methods. Correspondingly, facts involving these entities were also filtered out; (4) The instances of the \"Position Held\" predicate comprise over 50% of the dataset. However, the filtering operation in the previous step could impair connectivity. To maintain a robust connection with other predicates, we retained facts that are associated with other predicates and share core entities with the \"Position Held\" instances; (5) Following RE-NET [1], all facts were split into the training set, the validation set, and the test set by a proportion of 80%:10%:10% according to the time ascending order. \n\nThe NICE dataset was constructed as follows: (1) We collected the raw event records from ICEWS [2] from Jan 1, 2005 to Dec 31, 2014; (2) From these raw event records, we extracted core entities, timestamps, and partial auxiliary information, i.e., the occurrence places. (3) We observed that the predicate names contain additional auxiliary information. For example, the predicate names “Cooperate Economically” and “Engage in Judicial Cooperation” contain the specified cooperation aspects of the general predicate “Engaged in Cooperation”, namely “Economic” and “Militarily”, respectively. To obtain such auxiliary information, we designed rule templates to extract them from the specified predicate names and merged these specified predicates into the general ones. (4) Considering the absence of role information in the raw event records, we manually assigned role names to each predicate. (5) Similar to NWIKI, NICE was also split into the training set, the validation set, and the test set by a proportion of 80%:10%:10%.\n\nRefer to the question **“the selection preference of predicates as the number of predicates is too small (22 for NWIKI, 20 for NICE)”**: Despite the ample variety of predicates in Wikidata, the number of predicates is limited when it comes to predicates that contain n-tuple (N>3) roles and are temporally associated. The widely used quadruple-based WIKI dataset, derived from the same source, also encompasses only 24 distinct types of predicates [1]. As for the NICE dataset, the third step of the construction process reduces the number of predicates to 20: The predicates in ICEWS systems are organized in a hierarchical manner, with a maximum number of 256 fine-grained types [3]. As we extracted additional auxiliary information from the specified predicate names and merged the corresponding specified predicates into more general ones, there are only 20 kinds of predicates in NICE. For example, we merged the predicates “Cooperate Economically” and “Engage in Judicial Cooperation” into the predicate “Engaged in Cooperation” after the extraction of the role value \"Cooperate Aspect”.\n\nRefer to the question **“In Section 5.1.1, why did you filter out the entities of low frequency?”**:\nThe presence of low frequency entities poses challenges for data-driven methods to effectively learn and predict their behaviors. Furthermore, these predicate instances involving these entities may influence the evaluation of the model abilities. Similar filtering operation is also adopted in the construction process widely used TKG datasets, YAGO and WIKI [4].\n\n**Q3**: Some SOTA baselines published in early 2023 are missing (see Missing References section).\n\n**A3**: Thank you for this valuable suggestion. We compare the proposed NE-Net with three new and representative TKG reasoning methods, namely GHT [5], CEN [6], and TiRGN [7]. The corresponding results are as follows:\n\n|       | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI |  NICE | NICE |  NICE | NICE |  NICE | NICE |  NICE | NICE |\n|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:----:|:-----:|:----:|:-----:|:----:|:-----:|:----:|\n|       |  H@1  |  H@1  |  H@3  |  H@3  |  H@10 |  H@10 |  MRR  |  MRR  |  H@1  |  H@1 |  H@3  |  H@3 |  H@10 | H@10 |  MRR  |  MRR |\n| Model |   C   |  AUX  |   C   |  AUX  |   C   |  AUX  |   C   |  AUX  |   C   |  AUX |   C   |  AUX |   C   |  AUX |   C   |  AUX |\n|  GHT  | 30.71 |   -   | 37.78 |   -    | 39.94 |    -   | 34.57 |    -    | 30.28 |   -   | 45.20 |   -   | 61.04 |   -   | 40.61 |   -   |\n|  CEN  | 48.47 |   -    | 62.97 |   -    | 70.10 |   -    | 56.89 |    -   | 33.32 |   -   | 49.29 |   -   | 64.65 |    -  | 43.98 |   -   |\n| TiGRN | 50.61 |   -    | 68.24 |    -   | 81.13 |   -    | 61.10 |    -   | 34.82 |   -   | 51.54 |   -   | 66.47 |    -  | 45.77 |   -   |\n\nAs we can see, NE-Net still outperforms these baselines across all metrics on both datasets. We will manage to add other missing baselines and references in a later version.\n\n**Q4**: Since NE-Net models include recurrent modeling (Modeling Interactions in Temporal Adjacent Facts) and Transformer modules, what is the time complexity of NE-Net?\n\n**A4**: To illustrate the efficiency of the proposed NE-Net, we analyze the computational complexity of its evolutional interaction encoder. We view the computational complexities of the interaction aggregation unit as constants. The time complexity of the EP unit is $O(\\omega_1|\\mathcal{E}_1|)$, where $|\\mathcal{E}_1|$ is the maximum number of role edges in the concurrent facts. The time complexity of the CE unit is $O(\\omega_2|\\mathcal{E}_2|)$, where $\\mathcal{E}_2$ denotes the maximum number of concurrent facts in the history. As we recurrently model on historical KGs at adjacent timestamps for $m$ times, the time complexity for the evolutional interaction encoder is finally $O(m\\omega_1|\\mathcal{E}_1|+m\\omega_2|\\mathcal{E}_2|)$.\n\n[1] Jin W, Qu M, Jin X, et al. Recurrent event network: Autoregressive structure inference over temporal knowledge graphs[J]. arXiv preprint arXiv:1904.05530, 2019.\n\n[2] Ward M D, Beger A, Cutler J, et al. Comparing GDELT and ICEWS event data[J]. Analysis, 2013, 21(1): 267-297.\n\n[3] Halterman A, Bagozzi B, Beger A, et al. PLOVER and POLECAT: A New Political Event Ontology and Dataset[J]. 2023.\n\n[4] Dasgupta S S, Ray S N, Talukdar P. Hyte: Hyperplane-based temporally aware knowledge graph embedding[C]//Proceedings of the 2018 conference on empirical methods in natural language processing. 2018: 2001-2011.\n\n[5] Sun H, Geng S, Zhong J, et al. Graph Hawkes Transformer for Extrapolated Reasoning on Temporal Knowledge Graphs[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 7481-7493.\n\n[6] Li Z, Guan S, Jin X, et al. Complex Evolutional Pattern Learning for Temporal Knowledge Graph Reasoning[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022: 290-296.\n\n[7] Li Y, Sun S, Zhao J. Tirgn: time‐guided recurrent graph network with local‐global historical patterns for temporal knowledge graph reasoning[C]//Proceedings of the Thirty‐First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23‐29 July 2022. ijcai. org, 2022: 2152-2158.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper holds the view that the existing quadruple-based formulation cannot describe temporal facts accurately, thus limiting the applications of TKGs, which is a new perspective of TKG reasoning.\n\n2. The authors build two new N-tuple-based datasets to support its claims and model, which will be of some help to the TKG community.", "reasons_to_reject": "1. The claim that existing quadruple-based formulation cannot describe temporal facts accurately is not convincing, which means the papers should provide experiments to compare how the N-tuple-based structure is better than the quadruple-based structure to support the claim.\n\n2. The newly constructed datasets determine the upper border of model performance, but the descriptions of how to construct these two datasets are not clear, especially for the selection preference of predicates as the number of predicates is too small (22 for NWIKI, 20 for NICE).\n\n3. Some SOTA baselines published in early 2023 are missing (see Missing References section).", "questions_to_authors": "1. In Section 5.1.1, why did you filter out the entities of low frequency?\n\n2. Since NE-Net models include recurrent modeling (Modeling Interactions in Temporal Adjacent Facts) and Transformer modules, what is the time complexity of NE-Net?", "paper_topic_and_main_contributions": "The paper proposes a new temporal KG data structure: N-tuple, two curated N-tuple-based datasets, and a sophisticated reasoning model NE-Net. The experimental results show its superiority over existing models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Table 3: CyNet -> CyGNet\n2. TiGRN -> TiRGN"}, "extra_text": {"Missing_References": "1. Xu, Yi, et al. \"Temporal knowledge graph reasoning with historical contrastive learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 4. 2023.\n\n2. Li, Yujia, Shiliang Sun, and Jing Zhao. \"Tirgn: time‐guided recurrent graph network with local‐global historical patterns for temporal knowledge graph reasoning.\" Proceedings of the Thirty‐First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23‐29 July 2022. ijcai. org, 2022.\n\n3. ..."}}, {"review_id": "ugrqI8SJrL", "reviewer": "Reviewer_NCMC", "summary": "", "questions": "1. See Reasons to Reject 3. How is the role-entity pair constructed? \n\n2. How do the TKG reasoning baselines reason with full auxiliary information (line 531)? Is the auxiliary entity modeled the same as the core entity? If so, is it possible for them to predict auxiliary entities?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the effort you have invested in reviewing our work. We are honored to address the concerns you've raised.\n\n**Q1**: There are no baselines and references in related works after 2021. The baselines may be outdated.\n\n**A1**: Thank you for this valuable suggestion. We compare our model with three new and representative TKG reasoning methods, namely GHT [1], CEN [2], and TiRGN [3]. The corresponding results are as follows:\n\n|       | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI |  NICE | NICE |  NICE | NICE |  NICE | NICE |  NICE | NICE |\n|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:----:|:-----:|:----:|:-----:|:----:|:-----:|:----:|\n|       |  H@1  |  H@1  |  H@3  |  H@3  |  H@10 |  H@10 |  MRR  |  MRR  |  H@1  |  H@1 |  H@3  |  H@3 |  H@10 | H@10 |  MRR  |  MRR |\n| Model |   C   |  AUX  |   C   |  AUX  |   C   |  AUX  |   C   |  AUX  |   C   |  AUX |   C   |  AUX |   C   |  AUX |   C   |  AUX |\n|  GHT  | 30.71 |   -   | 37.78 |   -    | 39.94 |    -   | 34.57 |    -    | 30.28 |   -   | 45.20 |   -   | 61.04 |   -   | 40.61 |   -   |\n|  CEN  | 48.47 |   -    | 62.97 |   -    | 70.10 |   -    | 56.89 |    -   | 33.32 |   -   | 49.29 |   -   | 64.65 |    -  | 43.98 |   -   |\n| TiGRN | 50.61 |   -    | 68.24 |    -   | 81.13 |   -    | 61.10 |    -   | 34.82 |   -   | 51.54 |   -   | 66.47 |    -  | 45.77 |   -   |\n\nAs we can see, NE-Net still outperforms these baselines across all metrics on both datasets. We will manage to add other missing baselines and references in a later version.\n\n**Q2**: The experiments lack case studies. There is only the analysis of final performance with varying percentages of auxiliary entities. A case study would explain how auxiliary entities help with the reasoning. \n\n**A2**: Thank you for your constructive comment. We show two cases in the following table, where NE-Net correctly predicts the answer entity. In this table, $t-1$ and $t$ represent historical timestamps, while $t+1$ represents the query timestamp. In the first case, the correct answer to the query plays as an auxiliary entity in the latest history fact. The model is hard to predict the correct answer without such auxiliary information. In the second case, the auxiliary entity \"Humanitarian Aid\" helps the model establish a connection between the query and the most informative historical fact, and thereby enabling more accurate predictions.\n\n| Case number | History at t-1 | History at t | Query at t+1 | Answer |\n|:---:|:---|:---|:---|:---:|\n| Case 1 | (Disapprove, Opponent: Barack Obama,   Proponent: Head of Government (Kenya),   Disapprove Way: Denounce,    Place: Kenya) | (Consult,      Consulter: Barack Obama,     Consulted: Mulatu Teshome,     Consult Way: Engage in Negotiation,     Place: Ethiopia)  | (Engage in Diplomatic Cooperation,     Cooperator: Barack Obama,     Cooperator: ?,     Cooperation way: Praise or Endorse) | Ethiopia |\n| Case 2 | (Express Intent to Cooperate,     Volunteer: Iran,     Cooperation Target: South Africa, Cooperate Content: Engage in Diplomatic Cooperation); (Express Intent to Cooperate,     Volunteer: Iran,      Cooperation Target: Yemen, Cooperate Content: Provide Humanitarian Aid) | (Engage in Diplomatic Cooperation,     Cooperator: Iran,    Cooperator: South Africa,     Cooperation way: Sign Formal Aggrement) | (Provide Aid,     Provider: Iran,     Recipient: ?,     Aid Content: Humanitarian Aid) | Yemen |\n\n**Q3**: The dataset construction is not clearly explained. For example, in 5.1.1, how is the (3) step for NWIKI conducted? How is the \"type-specific role-entity pair and the occurrence place\" obtained in the (2) step for NICE? How is the role-entity pair constructed?\n\n**A3**: Refer to the question **“how is the third step in NWIKI conducted”**: During the building process of NWIKI, we filtered out entities of low frequency, as their behaviors are difficult to learn and predict for most data-driven based methods. Correspondingly, facts involving these entities were also filtered out. After that, we notice that the instances of the \"Position Held\" predicate comprise over 50% of the dataset. However, the filtering operation in the previous step could impair connectivity. To maintain a robust connection with other predicates, we retained facts that are associated with other predicates and shared core entities with the \"Position Held\" instances.\n\nRefer to the question **“how to obtain the type-specific role-entity pair and the occurrence place and how to construct the role-entity pair in NICE”**: There is partial auxiliary information, i.e., the occurrence places, in the raw event records provided by ICEWS. And we observed that the predicate names of ICEWS contain additional auxiliary information. For example, the predicate names “Cooperate Economically” and “Engage in Judicial Cooperation” contain the specified cooperation aspects of the general predicate “Engaged in Cooperation”, namely “Economic” and “Militarily”, respectively. To obtain such auxiliary information, we designed rule templates to extract them from the specified predicate names and merged these specified predicates into the general ones. Since role information is absent in the raw event records, we manually assigned role names to each predicate.\n\n**Q4**: How do the TKG reasoning baselines reason with full auxiliary information (line 531)? Is the auxiliary entity modeled the same as the core entity? If so, is it possible for them to predict auxiliary entities?\n\n**A4**: We apologize for omitting details about the baseline reproduction process. As illustrated in the paper, we converted the concurrent facts in the history into graph sequences, where the predicate instances are treated as nodes. Such operation is similar to the CVT node in knowledge graphs [4] and allows each predicate instance to connect with various auxiliary entities. By decomposing the predicate instances of each graph into multiple quadruples, each auxiliary entity therein can be modeled in the same way as that of the core entity, and thus existing TKG reasoning methods can predict auxiliary entities. However, we observe that including the auxiliary entities for these methods will lead to worse performance on core entity prediction. For example, the MRR result of REGCN on the core entity prediction decreases from 56.78 to 21.52. We guess that it is because the future predicate instances are unseen and cannot be learned since each predicate instance only appears once. As a result, we abandon the prediction of auxiliary entities and focus on the prediction of core entities. We will report this part of results in the next version.\n\n[1] Sun H, Geng S, Zhong J, et al. Graph Hawkes Transformer for Extrapolated Reasoning on Temporal Knowledge Graphs[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 7481-7493.\n\n[2] Li Z, Guan S, Jin X, et al. Complex Evolutional Pattern Learning for Temporal Knowledge Graph Reasoning[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022: 290-296.\n\n[3] Li Y, Sun S, Zhao J. Tirgn: time‐guided recurrent graph network with local‐global historical patterns for temporal knowledge graph reasoning[C]//Proceedings of the Thirty‐First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23‐29 July 2022. ijcai. org, 2022: 2152-2158.\n\n[4] Wen J, Li J, Mao Y, et al. On the representation and embedding of knowledge bases beyond binary relations[J]. arXiv preprint arXiv:1604.08642, 2016.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The n-tuple augmented TKGs are novel and complement the existing research. \n\n2. The paper is well-written.\n\n3. The experiments on two datasets demonstrate the effectiveness of NE-Net.", "reasons_to_reject": "1. There are no baselines and references in related works after 2021. The baselines may be outdated.\n\n2. The experiments lack case studies. There is only the analysis of final performance with varying percentages of auxiliary entities. A case study would explain how auxiliary entities help with the reasoning.\n\n3. The dataset construction is not clearly explained. For example, in 5.1.1, how is the (3) step for NWIKI conducted? How is the \"type-specific role-entity pair and the occurrence place\" obtained in the (2) step for NICE?", "questions_to_authors": "1. See Reasons to Reject 3. How is the role-entity pair constructed? \n\n2. How do the TKG reasoning baselines reason with full auxiliary information (line 531)? Is the auxiliary entity modeled the same as the core entity? If so, is it possible for them to predict auxiliary entities?", "paper_topic_and_main_contributions": "The paper proposes N-tuple augmented temporal knowledge graph (N-TKGs) to accurately describe the temporal fact of TKGs. The paper proposes a NE-Net to model the interaction among concurrent facts and temporal patterns of N-TKGs. The experiments on predicate prediction and entity prediction demonstrate the effectiveness of the method and outperform the static n-tuple baselines and temporal quadruple baselines. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "References after 2021. For example:\n\nStatic n-ary knowledge graph reasoning methods:\n\nPolygonE: Modeling N-ary Relational Data as Gyro-Polygons in Hyperbolic Space\n\n\nTKG reasoning methods:\n\nhttps://aclanthology.org/2022.findings-emnlp.458.pdf\n\nhttps://aclanthology.org/2022.findings-emnlp.542.pdf"}}, {"review_id": "ZOefVUnhj5", "reviewer": "Reviewer_ivcT", "summary": "", "questions": "1.As mentioned in weakness, what is a predicate instance and predicate type?\n\n2. As mentioned in weakness, why the time window m is set to only 2 in experiments?\n\n3. In line 500, the authors mentioned they use time filter protocol for evaluation. I’d like to know more about the time-aware filtering protocol implemented in NTKGs. For example, in core entity prediction, does it filter out facts with the same core entity, predicate and timestamps or it filter out facts whose entities, predicate and timestamp are completely the same as facts in datasets? How it is implemented during AUX prediction?\n\n4. Refer to weakness (2), why some recent and competitive models are not evaluated? Especially, the improvements on NICE are marginal. \n\n5. Refer to weakness (3)\n\n6. Refer to weakness (5)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the effort you have invested in reviewing our work. We are honored to address the concerns you've raised.\n\n**Q1**: The paper is not easy to follow, especially in section 4. In section 4.1.1, the author mentioned “predicate instance” and “predicate type” which are not properly defined or illustrated and as a result, makes the method introduction really hard to follow. What is a predicate instance and predicate type?\n\n**A1**: The predicate type is a concept in the knowledge base, which becomes a predicate instance when its arguments are specified [1][2]. For example, “(attack, attacker: person A, victim: person B)” and “(attack, attacker: person C, victim: person D)” are two distinct predicate instances. However, they both belong to the \"attack\" predicate type. We will describe these two terms more clearly in the latter version.\n\n**Q2**: The results are not convincible enough. Some recent and competitive TKG extrapolation models and N-tuple KG reasoning models like: TLogic, xERTE, CENET, Hyper2 and GRAN are not included. Especially, the improvements on NICE are marginal.\n\n**A2**: Refer to **the performance of the competitive TKG extrapolation models**, we compare our model with three new and representative TKG reasoning methods, namely GHT [3], CEN [4], and TiRGN [5]. The corresponding results are as follows:\n\n|       | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI | NWIKI |  NICE | NICE |  NICE | NICE |  NICE | NICE |  NICE | NICE |\n|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:----:|:-----:|:----:|:-----:|:----:|:-----:|:----:|\n|       |  H@1  |  H@1  |  H@3  |  H@3  |  H@10 |  H@10 |  MRR  |  MRR  |  H@1  |  H@1 |  H@3  |  H@3 |  H@10 | H@10 |  MRR  |  MRR |\n| Model |   C   |  AUX  |   C   |  AUX  |   C   |  AUX  |   C   |  AUX  |   C   |  AUX |   C   |  AUX |   C   |  AUX |   C   |  AUX |\n|  GHT  | 30.71 |   -   | 37.78 |   -    | 39.94 |    -   | 34.57 |   -    | 30.28 |   -   | 45.20 |   -   | 61.04 |   -   | 40.61 |   -   |\n|  CEN  | 48.47 |   -    | 62.97 |   -    | 70.10 |   -    | 56.89 |    -   | 33.32 |   -   | 49.29 |   -   | 64.65 |    -  | 43.98 |   -   |\n| TiGRN | 50.61 |   -    | 68.24 |    -   | 81.13 |   -    | 61.10 |    -   | 34.82 |   -   | 51.54 |   -   | 66.47 |    -  | 45.77 |   -   |\n\nAs we can see, NE-Net still outperforms these baselines across all metrics on both datasets. We will manage to add other missing baselines and references in a later version.\n\nRefer to **the marginal improvements on NICE**, this may be caused by the limited auxiliary role types in NICE. The detailed analysis is provided in lines 578-583.\n\n**Q3**: The necessity of introducing NTKG reasoning models is not well supported. Because, for N-tuple KGs, they can represent temporal facts by adding a role “timestamp” and the corresponding value is the date. In this way, static N-tuple models can handle temporal N-ary facts. Thus, when evaluating static N-tuple models, except for the trivial experiments the author having done in this paper, treating timestamps of facts in NTKGs as auxiliary role-value pairs and then running static models on them would make more sense.\n\n**A3**: To verify the capability of existing N-tuple KG models in performing temporal reasoning tasks, we have conducted supplementary experiments following the suggested setting. Specifically, we kept the time information in n-tuple facts, and retrained NALP, HINGE, as well as Hy-Transformer on NWIKI for predicting the core entities. The results are presented in the following table, where the models suffixed with \"w. time\" indicating the training process with time information:\n\n| model\\metrics | H@1 | H@3 | H@10 | MRR |\n|:---:|:---:|:---:|:---:|:---:|\n| NALP | 10.59 | 10.59 | 22.52 | 14.86 |\n| NALP w. time | 11.91 | 14.62 | 18.36 | 14.16 |\n| HINGE | 19.10 | 23.47 | 25.91 | 21.74 |\n| HINGE w. time | 11.59 | 15.63 | 19.71 | 14.49 |\n| Hy-transformer | 33.40 | 35.85 | 37.84 | 34.97 |\n| Hy-transformer w. time | 29.93 | 32.25 | 34.05 | 31.42 |\n\nWe observed that these existing N-tuple KG reasoning methods have shown comparable, and even worse performance when compared to the performance not considering time information. Since the N-tuple KG reasoning approaches view the timestamp information as regular entities, the significant temporal order characteristic is omitted. On the contrary, the proposed NE-Net takes the temporal order into consideration, and experimental results verify its effectiveness in the temporal reasoning task.\n\n**Q4**: One of the motivations of proposing NE-Net is to model the temporal patterns of historical facts. The window m is to only 2 which does not comply with the motivation and makes the utilization of temporal information questionable.\n\n**A4**: We performed grid search on the length of time window (1, 6). The optimal lengths of history $m$ on both datasets are experimentally set to 2. The value of $m$ is not too large, as the facts at the most recent timestamps carry greater weight. This is substantiated by several quadruple-based TKG reasoning models [4][6] which confirm that the optimal historical length should not be excessively large in TKG reasoning datasets. For instance, RE-GCN demonstrates an optimal history lengths of 1 and 3 in the ICEWS14 and WIKI datasets, which are constructed from the same source data as the proposed NWIKI and NICE datasets [6]. We provide an ablation study to prove the effectiveness of using temporal information. Specifically, we conduct experiments only using the score function with the randomly initialized learnable embeddings without using history. The corresponding MRR results are as follows: \n\n|  | NWIKI | NWIKI | NWIKI | NICE | NICE | NICE |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| variants | C | AUX | ALL | C | AUX | ALL |\n| NE-Net (no his) | 27.30 | 12.80 | 19.01 | 43.90 | 78.40 | 62.69 |\n| NE-Net (w. his) | 72.03 | 57.68 | 63.82 | 48.98 | 79.06 | 65.37 |\n\nThe performance gap verifies the effectiveness of the proposed NE-Net using temporal information.\n\n**Q5**: It is unclear how many parameters this complex model uses and how it compares with existing approaches.\n\n**A5**: With regards to the number of parameters, the proposed NE-Net consists of 15.8M weights on the NWIKI, while REGCN, one of the most related baselines, consists of 10.2M weights. Such different scales of parameters can be attributed to variations in the GCN layer configurations and the choice of decoders in each model.\n\n**Q6**: In line 500, the authors mentioned they use the time filter protocol for evaluation. I’d like to know more about the time-aware filtering protocol implemented in NTKGs. For example, in core entity prediction, does it filter out facts with the same core entity, predicate and timestamps or it filter out facts whose entities, predicate, and timestamp are completely the same as facts in datasets? How it is implemented during AUX prediction?\n\n**A6**: In the core entity prediction, the facts having completely the same elements as the query fact except for the query elements are filtered out. Take a typical query $(pred, role_1: e_1, role_2: ?, role_3:e_3, t)$ with answer $o_1$ in the test set as an example, and assume there are two other facts occurring at timestamp $t$, namely, $f_1 = (pred, role_1: e_1, role_2: o_2, role_3:e_3, t)$ and $f_2 = (pred, role_1: e_1, role_2: o_3, role_3:e_4, t)$, only $o_2$ in $f_1$ can perfectly match the query. Therefore, $f_1$ is the only one that is filtered out. The same filtering process applies to the auxiliary entity prediction.\n \n\n[1] Finch S E, Finch J D, Huryn D, et al. An approach to inference-driven dialogue management within a social chatbot[J]. arXiv preprint arXiv:2111.00570, 2021.\n\n[2] Leppanen M. Towards an abstraction ontology[J]. Frontiers in Artificial Intelligence and Applications, 2007, 154: 166.\n\n[3] Sun H, Geng S, Zhong J, et al. Graph Hawkes Transformer for Extrapolated Reasoning on Temporal Knowledge Graphs[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 7481-7493.\n\n[4] Li Z, Guan S, Jin X, et al. Complex Evolutional Pattern Learning for Temporal Knowledge Graph Reasoning[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022: 290-296.\n\n[5] Li Y, Sun S, Zhao J. Tirgn: time‐guided recurrent graph network with local‐global historical patterns for temporal knowledge graph reasoning[C]//Proceedings of the Thirty‐First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23‐29 July 2022. ijcai. org, 2022: 2152-2158.\n\n[6] Li Z, Jin X, Li W, et al. Temporal knowledge graph reasoning based on evolutional representation learning[C]//Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 2021: 408-417.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is the first to combine TKGs with N-tuple KGs resulting in NTKGs and it also provides two NTKG datasets for evaluations.\n\n2. The proposed NE-Net considers and makes use of the main characteristics of TKG extrapolation and N-tuple KG reasoning. ", "reasons_to_reject": "1. The paper is not easy to follow especially in section 4. In section 4.1.1, the author mentioned “predicate instance” and “predicate type” which are not properly defined or illustrated and as a result, makes the method introduction really hard to follow.\n\n2. I think the results are not convincible enough. Some recent and competitive TKG extrapolation models and N-tuple KG reasoning models like: TLogic, xERTE, CENET, Hyper2 and GRAN are not included.\n\n3. I think the necessity of introducing NTKG reasoning models is not well supported. Because, for N-tuple KGs, they can represent temporal facts by adding a role “timestamp” and the corresponding value is the date. In this way, static N-tuple models can handle temporal N-ary facts. Thus, when evaluating static N-tuple models, except for the trivial experiments the author having done in this paper, treating timestamps of facts in NTKGs as an auxiliary role-value pairs and then running static models on them would make more sense.\n\n4. One of the motivations of proposing NE-Net is to model the temporal patterns of historical facts. However, in section 5.1.4, the window m is to only 2 which does not comply with the motivation and makes the utilization of temporal information questionable.\n\n5.It is unclear how many parameters this complex model uses and how it compares with existing approches.", "questions_to_authors": "1.As mentioned in weakness, what is a predicate instance and predicate type?\n\n2. As mentioned in weakness, why the time window m is set to only 2 in experiments?\n\n3. In line 500, the authors mentioned they use time filter protocol for evaluation. I’d like to know more about the time-aware filtering protocol implemented in NTKGs. For example, in core entity prediction, does it filter out facts with the same core entity, predicate and timestamps or it filter out facts whose entities, predicate and timestamp are completely the same as facts in datasets? How it is implemented during AUX prediction?\n\n4. Refer to weakness (2), why some recent and competitive models are not evaluated? Especially, the improvements on NICE are marginal. \n\n5. Refer to weakness (3)\n\n6. Refer to weakness (5)", "paper_topic_and_main_contributions": "This paper addresss the problem of information loss in traditional temporal knowledge graphs due to the simple quadruple form of facts representation and attempt to model dynamic facts with n-tuples associated with timestamps yielding N-TKGs. To infer missing fasts in N-TKGs, this paper also proposes a N-TKGs reasoning model, NE-Net, which incorpoorates three units in the encoding peroid to capture interactions between entities involed in various facts and facts at adjacent timestamps. In the decoding stage, the model uses different task-specific decoders to deal with tasks including relation predictions and entity predictions. This paper also provides two N-TKGs datasets built uppon Wikidata and ICEWS for evaluations. Experimental results on the two newly built datatsets show that the proposed method is superior to several baseline models of TKG and NKG reasoning.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "In Table 3, CyGNet, “G” missed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel strategy for modelling temporal knowledge graphs, based on n-tuples associated with timestamps. Further, the authors introduce two novel datasets for N-tuple modelling. Experimental results on these new datasets show that the proposed strategy is superior to current alternatives. 

While the initial version lacked clarity around certain areas, including the construction process for the two datasets, the authors have addressed this in their rebuttals. The authors have also provided additional evaluation results with more recent baselines, showing continued competitiveness. 

One concern mentioned by several reviewers is the necessity of moving to an N-tuple format rather than the previously used quadruple format; although the introduced datasets are useful, and the introduced models outperform contemporaries on these, this argument may not be sufficiently well-supported.
Decision: Accept-Findings