NeuSTIP: A Neuro-Symbolic Model for Link and Time Prediction in Temporal Knowledge Graphs
Authors: Ishaan Singh
Navdeep Kaur
Garima Gaur
Mausam .
Venue: EMNLP 2023 Main (2023)
Keywords: Information Extraction
Neuro-Symbolic Knowledge Graph Completion
Temporal Knowledge Graph Completion
TL;DR: We develop a novel neuro-symbolic TKGC method that represents the temporal information of TKGs in a temporal rule language it defines, and ascribes a novel confidence score to a candidate answer, which can be an entity or a time interval.

Abstract: Neuro-symbolic (NS) models for knowledge graph completion (KGC) combine the benefits of symbolic models (interpretable inference) with those of distributed representations (parameter sharing, high accuracy). While several NS models exist for KGs with static facts, there is limited work on temporal KGC (TKGC) for KGs where a fact is associated with a time interval. In response, we propose a novel NS model for TKGC called NeuSTIP, which performs link prediction and time interval prediction in a TKG. NeuSTIP learns temporal rules with Allen predicates, which ensure temporal consistency between neighboring predicates in the rule body. We further design a unique scoring function that evaluates the confidence of the candidate answers while  performing link and time interval predictions by utilizing the learned rules. Our empirical evaluation on two time interval based TKGC datasets shows that our model shows competitive performance on link prediction and establishes a new state of the art on time prediction.

NeuSTIP: A Neuro-Symbolic Model for Link and Time Prediction in Temporal Knowledge Graphs
Ishaan Singh Navdeep Kaur Garima Gaur
Mausam
Indian Institute of Technology, Delhi (ishaanyuvraj, navdeepkjohal, 27 mausamêcse. iitd.ac. ac.
Abstract
KGs, a key KGC task is link prediction, viz. given query r(s, ?), predict o for which the fact r(s, o) holds the real-world. This problem is fairly well-studied and has been tackled in many differ- ent ways. Existing works can be broadly catego- rized into GNN-based solutions (Zhu et al., 2021; Vashishth et al., 2020), LM-based approaches (Yao et al., 2019; Wang et al., 2022), KG embedding- based models (Sun et al., 2019; Trouillon et al., 2016), and neuro-symbolic (NS) solutions (Yang et al.. 2017: Qu et al., 2021). Of special interest to us are NS approaches, which bring together human- interpretable deduction capabilities of symbolic models with parameter sharing and other benefits of distributed representations in neural models.
Neuro-symbolic (NS) models for knowledge graph completion (KGC) combine the benefits of symbolic models (interpretable inference) with those of distributed representations (pa- rameter sharing. high accuracy). While several NS models exist for KGs with static facts, there is limited work on temporal KGC (TKGC) for KGs where a fact is associated with a time inter- val. In response, propose novel NS model for TKGC called NeuSTIP, which performs link prediction and time interval prediction in TKG. NeuSTIP learns temporal rules with Allen predicates, which ensure temporal con- sistency between neighboring predicates the rule body. We further design unique scor- ing function that evaluates the confidence of the candidate answers while performing link and time interval predictions by utilizing the learned rules. Our empirical evaluation on two time interval based TKGC datasets shows that our model shows competitive performance on link prediction establishes new state of the art time prediction.
Temporal KG completion (TKGC) extends KGC task to TKGs. In addition to link prediction, it in- volves an additional task of time- interall prediction given a query r(s, o,?). infer the time interval when the fact holds true. The majority of exist- ing solutions are KG-embedding based (Dasgupta et al., 2018; Jain al., 2020; Xu et al., 2020), and do not incorporate symbolic rules. NS approaches have received very limited attention for TKGC. TLogic and ALRE-IR (Liu al., 2021; Mei et al., 2022) study NS-TKGC, but for time-instant KGs (i.e., KGs where a fact associated with an instant, not an interval). TILP (Xiong et al., 2023) is a very recent model for time- lime-interval TKGC, but it can only do link prediction. To the best of our knowl- edge, no NS-TKGG approach exists that performs both link and time-interval prediction.
1 Introduction
Knowledge Graphs (KGs) are factual information repositories, where each fact is encoded as r(s, o), where and are the real-world entities and is the relationship between them. For instance, the fact presidentOf(Joe Joe Biden, USA) represents the fact that Joe Biden is the president of USA. Temporal KGs extend these to entity-entity relations that have a temporal facet, for e.g., workedAt Einstein, ETH Zurrch, [1112, In this work, we study Temporal KGs that maintain temporal facts, r(s, o, T), annotating each fact r(s, o) with the time period T for which it holds.
A key challenge in building an NS-TKGC model is to design unified rule language, which is intu- itive and interpretable to humans, and also effective on both prediction tasks. Secondly, ideally, the con- fidence of rule should be computed based on both the statistical properties of the rule groundings and also the similarity scores of latent representations (which not the case in existing models).
While being a popular source of structured in- formation, KGs are often incomplete. To this end, the problem of enriching KGs by inferring miss- ing information is formulated as a KG completion (KGC) task. In the context of static (non-temporal)
Contributions:
We propose NeuSTIP (Neuro

---

Symbolic Link and Time Interval Prediction), the first comprchensive NS framework for TKGC, which is effective on both link prediction and time interval prediction tasks. It uses an intuitive rule language that integrates the complete set of Allen algebra relations and KG relations, with the goal of enforcing temporal consistency between neigh- boring predicates in the rule body. It also uses novel way compute confidence of temporal rules that combines both symbolic and embedding infor- mation. Moreover, to the best of our knowledge, it is the first NS-TKGC model to perform time- interval prediction. We evaluate the performance of NeUSTIP using two benchmark time interval KGC datasets, WIKIDATA 12k and YAGO1 Ik (Dasgupta et al., 2018). We find that hybrid of NeuSTIP with KG-embedding model TimePlex (Jain et al., 2020) obtains best results on both prediction tasks in both datasets. We release the NeuSTIP! code- base for further research.
model respectively to integrate time information in GNN. GNN-based models are generally compu- tationally expensive (Luo et al., 2022), and do not scale well to large datasets.
Neuro-Symbolic (Rule-based) Models: These models combine neural embeddings with explicit rule learning and inference, getting benefits of both interpretable inference of symbolic models and pa- rameter sharing of neural models. TLogic (Liu et al., 2021) and ALRE-IR (Mei al., 2022) are NS link-prediction models designed for time-instant TKGs. Probably the closest to our work is very recent model TILP (Xiong et al., 2023) that per- forms all possible constrained walks on time inter- val datasets to learn temporal logic rules and adopts attention mechanism to score each rule. There are two key differences between TILP and NeuSTIP. Firstly, TILP's rules require expressing temporal relations between all pairs of intervals mentioned in the rule body. This blows up the number of rules, forcing TILP to use only three Allen rela- tions. In contrast, NeuSTIP encodes temporal rela- tions only between pairs of neighboring intervals in rule path, and uses all 13 Allen relations. Our experiments show that NeuSTIP performs better inferences compared to TILP. Secondly, and more importantly. TILP is developed only for link predic- tion, whereas NeuSTIP introduces specific score and loss functions for time-interval prediction.
2 Related Work
There are three main types of TKGC models in literature: purely embedding-based multi-hop rea- soning based, and rule-based.
Embedding-baseed Models: All these models learn embeddings of entities and relations and de- fine an associated scoring function to assess the validity of temporal fact. Earlier methods incor- porated time within entity embeddings (Dasgupta al., 2018; García-Durán et al., 2018), and used Trase scoring (Bordes et al., 2013). Recent methods (Messner et al., 2022: Sadeghian et al., 2021) adapt static KGE models such as RotatF (Sun et al., 2019), BoxE (Abboud et al., 2020) for TKGC task. Other approaches such as TransE- TAE (Jiang et al., 2016), Timeplex (Jain et al., 2020) explicitly learn time embeddings, and addi- tionally model temporal constraints between pairs of tuples in the TKG. These models are generally less interpretable, due to their complete reliance on latent embeddings.
2.1 Time-Interval Prediction
Time prediction in TKG is relatively underexplored. Existing research includes Know-Evolve (Trivedi et al., 2017) and GHNN (Han et al., 2020), which perform time instant prediction by modeling a given TKG fact as a point process. Time-interval prediction is studied in embedding- such Time2Box (Cai et al., 2021) and Time- plex (Jain et al., 2020) models they develop novel TKGEE-based scoring functions for the task. To the best of our understanding. no NS TKGC method exists for time-interval prediction, a gap that NeuSTIP aims to fill.
models
Multi-hop Reasoning Models: Such models exploit neighborhood information of an entity by employing distinct graph neural network architec- tures (Kipf and Welling, 2017). Models such TeMP (Wu et al., 2020), RE-NET (Jin et al., 2020), XERTE (Han et al., 2021a), CyGNet (Zhu et al., 2020) exploit self-attention/GRU,, RNN, time- aware attention mechanism, and Copy-Generation
Preliminaries
A knowledge graph maintains set of entities E, and relations R. and each fact r(s, o) is denoted by directed edge from subject entity entity o, with the label r. A temporal KG (TKG) IC additionally maintains the time domain, which is suitably discretized into a set T of discrete time
object
"https://ggite cttpt/uubcomm/aaaaaa

---

pF, TI
MMFFC
Jadon
WBI, T3
David
PF,
MUFC
Jadon
WBI,T3
London
WBi - A1, pF, A2, pF WBI
wBi,
IMt, T4
wBi,
A4, iMt, A5,
David
David
Victoris
London
IMtt
wBi,
Mictoria
FF: playsFor, wBi: wasBornin, IMT: isMarrieToo
Allen relations
Figure I: Extracting rules using the target fact w Bi( David, London, T6): Traversing KG (left) to all paths (center) from subject entity David object entity London; Extracting one rule per grounding (center) interleaving Allen relations A, that specify relations between time intervals of neighboring KG relations in rule body.
instants, with two special elements min and mar denoting the minimum and maximum time instants attainable in the TKG. A temporal fact r(s, o, T), where T [ts, te] denotes the time interval during which the relation was true (to, te E T) is repre-
4.1 Mining of Temporal Rules
NeuSTIP learns first-order logic rules of the form:
< A (a (T; T(11) Ch(c1, (1,,7)
Ari(ei,ett1,
C++1, T++1)
(I)
edge (r.T), between
sented in the TKG entities and (see Figure 1 (left)). For ease of model design, a common pre-processing step in- crements the TKG by adding inverse relations (and therefore inverse edges) to the graph (Jain al., 2020). Any TKG Completion model evaluated using (1) link prediction queries: tail prediction, r(s,?. ?) and head prediction, (o,? (T), and (2) time interval prediction queries: r(s, o, ?).
where and r; denote the Allen relations and KG relations respectively, and ej and T; are variables that will ground in the entity set & and time interval T x T: m the rule length. The rule body can be seen as a path from 61 to cm+1, where each KG relation r; shares the object entity with the subject entity of relation 1i+1, 1 S < m. Simi- larly, Allen relations specify relations between two consecutive time intervals, with the first Allen predicate a1 (T1, T2) of the rule body being rela- tion between the time interval in the rule head and the first time interval mentioned in the body. Refer to Appendix B for examples of temporal rules.
3.1 Background Allen Algebra
Our model is based on Allen's interval calcu- lus (Allen, 1983), which is formal system to represent relations between time intervals. defines 13 exhaustive and pairwise-disjoint rela- tion types. E.g., given two time intervals T1 = [A1s, [1e] and 72 = [12g, (2)). the Allen relation overlaps(T1, T2) holds if fl, < t2, tle < t2, (See Appendix A for more details). To avoid ambi- guity, refer to relations between time intervals Allen relations and relations R between en- tities KG relations. Our model uses all 13 Allen relations its rule language.
NeuSTIP mines temporal rules by, for each known fact rn o, T) E K, finding ground paths in TKG from 8 to o of length up to max limit. For instance, as shown in the center of Figure 1, for the fact w wii(Daviic London, 76), all paths from David to London are found. Each ground path gets converted to exactly one first-order rule (by replacing entities and time intervals with vari- ables), based on the specific KG relations in the path, and the specific Allen relations between each pair of time intervals (as shown in Figure I). We highlight that our rule extraction process does not involve sampling walks, rather we use all walks departure from existing non-temporal rule min- ing approaches (Qu et al., 2021). We choose this because our preliminary analysis suggested that TKGs are sparser than non-temporal KGs, hence, sampling or random walks may miss important rules.
4 The Proposed NeuSTIP Model
We first describe NeuSTIP's rule language and its algorithm for mining rules. We then discuss its model for scoring a candidate answer, for both link and time interval prediction queries (See Figure 2). Further, we describe the loss function and inference procedures.
4.2 Scoring Candidate Answers
NeuSTIP first finds candidate answers c and then scores each answer. For a link prediction query

---

rales, ?, [bo.dal) ratei-ea. ?) query
Time Prediction
path score
tle t2:
Temporal Knowledge Graph
OR
Link Prediction M
Ground the rules/ Generate paths
((path) [o]
A(11, +2,)
Temporal
OR
considered
All paths
rule
score
(ol,, Cm-11 Ti)
((pohh) ) [c]
A (T, T1t1)
Arvle, Prei. B....T+41)) Ti+1))
GRU
PCA
(Lb)) ) ((pahh)
patheP
Vembl
[5][c]
stta [
score(c, Ln}
considered
All rules
Allen Predicates
elemb Mlstat
scorelc, Lz)
score(c)
Figure 2: Overview of NeuSTIP evaluating link prediction (e, 1, ?T) and time cnntictioo-rrlee ea, ?) queries: Given a TKG and the generated rules (as shown Figure 1), NeuSTIP first (middle) grounds the relevant rules and finds candidate answer c; later computes rule score (middle) and corresponding path score (right) based the type (link or time) of the query. The score rule computation involves evaluating embedding- score (emo(Lz) (c))[c] and statistical measure based esta (L, [c]] For scoring paths, link prediction queries treat all the paths similarly and assign score of whereas for time prediction NeuSTIP relies priors (difference in start time and end time of pair of relations) learned from the data.
?,, finding candidates is straightforward: find all relevant rules, identify each rule's ground- ings, and mark groundings of variable cm+1 as candidate answer entities. However, for interval prediction (s, o, ?), grounds all the relevant rules, by ignoring the first Allen relation (Ti, T2). This is because it cannot be priori ground, since Ti is unknown (the intent is to predict it). It then identifies all time intervals T1, which consistent with the predicate a1 (Ti, T2), and outputs those as candidate answers. For example, 0 before and T₃ = [1990, 2000), then will output intervals [4, 4e] such that l6 < le < 1990.
Lj consistent with e. Note that candidate answer can be arrived by firing multiple rules in the rule set C. Further, a given rule Lj E can be grounded to different paths which when followed in TKG arrive at candidate Our scoring function (in Equation 2) takes into account the significance of all such rules and the corresponding paths that lead to the generation of candidate e by aggregat- ing rule score rh( [T)][[] and path score @ppath) [-]- Since e is of three types, the score of each rule Lj has three components [o)) 4 (b,),)) [ta) and J [te) used to score object entities, start time instants and end time instants, respectively. Simi- larly, the path scores are also specified using three components. We now explain NeuSTIP model that computes the rule score (LL))]]] and path score ((pat)][]]
For interval prediction queries, NeuSTIP scores start and end time instants separately. Its scor- ing function is motivated by RNNLogic (Qu et al., 2021). and extended to temporal setting: the score of candidate E {o, tb, te) is computed as
4.2.1 Rule Score
score(c) Σ score(c, =
NeuSTIP uses novel neuro-symbolic rule scoring function (Equation 4) - combines a rule embed- ding based score &emmb and statistical measure- based score Visatt to compute vb.
LEEC
Σ Σ (Lj)(c ([path)[c]
(2)
LEEL pathEp
Here, L is the set all first-order rules Lj that gen- erate answer c, and P is the set ground paths of
Embedding based rule score: NeuSTIP learns three embeddings for a KG rule head relation r:

---

r0, and r used for scoring for object entity, start time instants, and end time instant candidates, respectively. It also computes an embedding Lj for each rule L (motivated by Mei et al. (2022)) by passing the sequence of Allen and KG relations in Ly's 's rule body through a Gated Recurrent Unit (GRU) (Cho et al., 2014). For candidate answer depending upon its type (denoted object: O, start time: B, end time: E), the rule score of T. is computed as the cosine similarity between the rule embedding Ljj and the relevant rule head embed- ding r (E {1%
Notice from Equation 1, relations rh and ri will capture two facts about the same subject e1. There- fore, given the fact r'1 (c(, e2 [12g, (2e)), NeuSTIP calculates path score for start time candidate tl, in query Th (c1, Cm 11, ?) as,
((path) = Nt 12olu o (5)
The same approach is used to estimate (path)
B.B. 1 1.
4.3 Loss Function
e/mb [2)[c] = sim(Li rh)
(3)
NeuSTIP is trained by minimizing two loss func- tions LtP and Lyp for link prediction and interval prediction, respectively. The score of each candi- date object entity score(o) is normalized to P(o) by computing softmax over the entity set & Sim- ilarly, the scores of start time and end time candi- dates are normalized over the time instant set T. For a given query, we denote the set of known cor- rect answers as D and the set of other answers as N. We further use set S to denote the subset of N with higher score than that of the gold entity. For link prediction, the loss LLP is computed as
Further details on GRU model are in Appendix C.
Statistical measure based score: NeuSTIP lever- ages PCA score for each rule as an additional mea- sure of its accuracy. This symbolic rule confidence metric acts as prior of the rules and helps in in- formed initialization of the rule score. PCA score estimates the fraction of entities predicted by the rule that are known be true at training time (Galárraga et al., 2013). See Appendix D for more information. We denote by Vastt [c)] based on which type of entity being predicted by the rule. NeuSTIP computes the aggregate rule score as:
Eees( (P(e) ((()))((o)) (6)
Σ P(m)
Σ oCD
S
+ [c) Voom (L))(c] sttat [c]
(4)
Similarly, for time interval prediction let D de- note the set of known true intervals. We construct two sets D, and De with the start time instants and end time instants of intervals in D. Further, N,-
4.2.2 Path Score
For link prediction, the model exploits relatively simple approach and sets ((path)[o) = 1 for each path that reaches the target answer o. This implics that the score of candidate entities is dependent on the total number of groundings and the quality of the corresponding rules. A low-quality rule with less number of groundings cannot generate high- scoring candidate entity.
T Dj and N =T D.. Because time instants lb and te numerical, NeuSTIP uses loss function that captures differences between true and other times via LTP as follows:
Σ Σ (P(ms) - P(to)) d(ns. to) MED, m,N
For time prediction, rule can only be partially ground. Notice the first Allen relation a specifies the Allen relation between the time interval Ti of Th and T2 of r1 (See Equation 1). Since Ti is what we intend to predict, thus condition a1 (Ti, can- not be ground. However, Allen relation A1 and time interval T2 guide NeuSTIP (discussed in Sec- tion 4.2) to find its candidate start time t1, and end time Ale of time interval T1. For scoring the candidates, NeuSTIP computes two Gaussian dis- tributions, A WW B.) and (u /f.)) ( they represent the start/end time difference of pairs of KG relations (r, r') with the same subject entity.
Σ (P(n.) P(to)) * d d(neste)
fcCDe RcCN
where d(.,-) -) is the time instant distance function. We define the distance function in Appendix E.
4.4 Inference
At the test time, perform link prediction by ranking all objects o based on score(o). For time- interval prediction for query r(s, o, ?). NeuSTIP constructs a T x matrix, whose each entry is ((t)))((tt)+N))) +NW - tt//p,""r oindt) where P(t..) and P(t.) are the probabilities defined above, and

---

oetite is the Gaussian distribution over the duration of time-intervals of target relation r in the query. After populating the matrix with data, matrix TI in order to ensure te 2 io- We return the predicted time interval as [tg, tz] = arg ma (TI).
Table 1: Performance of models for link prediction task on WIKIDATAI2K and YAGO1
we consider its apper-triangun
Model
WIKIDATATZK
YAGOTIK
MRR
He
H@ 10
MRR
H@
H@ 10
Neural-LF
18.25
9.08
38.48
10.01
4,01
18.45
AnyBURL
19.08
10.30
3.78
TLogic
25,36
17.54
44.24
15.45
11.80
23.09
TILPP-bse
31.14
21.52
50.77
18.80
13.36
30.89
4.5 Extended Model with Timeplex
TILP
33.28
23.42
52.89
24.11
16.67
41,49
ComplEx
24.82
14.30
48.90
18.14
11.46
31.11
Inspired by RNNLogiC, we further ensemble our model with embedding-basee model, Time- Plex (Jain et al., 2020) integrate the complemen- tary features of NeuSTIP, our rulc-based model and an embedding- model as follows:
TA-ComplEEx
22.78
12.69
46.00
15.24
9.36
26.26
HyTE
25.28
14.70
48.26
13.55
3.32
29.81
DE DE-SimplE
25.29
14.68
49.05
15.12
8.75
26.74
INI-Complex
30.10
18.01
31.28
TimePlex (Buse)
32.38
22.03
52.79
18.35
10.99
31.86
TimePiex
33.35
22.78
53.20
23.64
16.92
36.71
NeuSTIP (Base)
31.98
23.81
35.15
NeUSTIP w/ KGE
34.78
24.38
53.75
25.23
18.45
37.76
ens_score(c) = score(c) + 1 ( Timplex(c) (8)
where E {or l., and r is learnable parameter. score(c) is NeuSTIP score (Equation 2).
2016). TA-ComplEx (García-Durán al., 2018). HyTE (Dasgupta et al., 2018), DE-SimplE (Goel et al., 2020), TNT-Complex (Lacroix et al., 2020), and TimePlex (Jain et al., 2020). Apart from ComplEx (Trouillon et al., 2016), all the other embedding-based solutions are proposed especially for temporal KGs. The original TimePlex model itself has two versions base model and the full model. The full version adds two temporal consis- tency gadgets, modeling relation recurrence, and typical duration distributions between two relations. In the same vein, TILP also has two variants, and its full model additionally introduces temporal fea- tures such as recurrence, and duration distribution into the model. We compare against both versions of the TimePlex and TILP models.
5 Experiments
5.1 Experimental Setup
Datasets and Metrics: We evaluate the pro- posed model on two standard time-interval TKGC datasets: WIKIDATAI2K and YAGOlIk (Das- gupta et al., 2018). For both datasets, we consider the temporal granularity to be 1 year. The dataset statistics are reported in Appendix F. For link pre- diction, report the standard metries of Mean Reciprocal Rank (MRR), Hits @1 I, and Hits 10 and use time-aware filtered measures in each model (Jain al., 2020). For time interval prediction, we employ the aeIOU metric (Jain et al., 2020) as:
acTOU(Tev ,TT)= mar(1, (1, (T" n (TT)) (To w Ter)
For interval prediction, NeuSTIP is the first neuro-symbolic model there is no existing NS model to directly compare against. We resort to using only embedding- basedd models for compar- isons: HyTE, TNT-Complex, and TimePlex.
such that 7 and 7 are gold and predicted time intervals, vol) refers to the size of the time interval (which would be in terms of number of years for these datasets), Tee ( TP" refers to overlap in time- interval, 7 77 is the smallest single contiguous interval containing Tev and TPr.
We report results from two variants of our pro- posed model, NeuSTIP (base) and NeuSTIP with KGE. The base model trained exclusively with the proposed temporal rules and corresponding can- didate score, as in Equation 2. In NeuSTIP with KGE variant, we integrate the state-of oftte-of-theaan art KG embedding model, Timeplex (full) using an ensem- ble (sec Section 4.5). The specifics of the training procedure of NeuSTIP and hyperparameter settings are reported in Appendix G. H and I. For all com- parisons, where possible we report published re- sults since our datasets are standard and exact splits have been used as is in earlier works.
Baselines: Link prediction TKGC relatively well-studied. We compare against 10 embedding and wawpee///neurosyymollle solutions. Specif- ically, we choose a total of 4 rule based models Ne (Yang et al., 2017) and AnyBurl (Meil- icke et al., 2019) handling static (non-temporal) KG, and TLogic (Liu et al., 2021) (time-instants) and TILP (Xiong et al., 2023) (time-intervals) working with temporal KGs. In embedding based approaches, we consider ComplEx (Trouillon et al.,

---

Table 2: Performance of models for time interval pre-
Table 3: Link prediction performance of NeuSTIP and variants, NeuSTIP-TR (w/o temporal constraints), NeuSTIP-PCA (w/o PCA score)
diction task WIKIDATA 12k and YAG
Dataset
WIKIDATA
YAGOTIK
WIKIDATATER
YAGOTIK
Model
acIOU
acIOU
Model
MRR
H@1
H@
MRR
H@ 10
HyTE
5.41
5,41
NeaSTIP
31,98
22.11
50.31
23.81
17.26
35.15
INT-Complex
23.35
8.40
NeuSTIP- TR
26.00
15.60
47.14
18,90
11.68
30.40
Timeplex (Base)
26.20
14.21
NeaSTIP-PeC PCA
29.69
20.10
47.77
22.14
16.07
32.72
Timeplex
26.36
20.03
NeuSTIP (Base)
27.36
22.49
NeuSTIP w/ KGE
28.21
27.83
human-interpretablee
Q5. How important is it to consider all the 13
5.2 Results and Observations
Allen predicates in our proposed model?
Q6. What is the effect of rule length on model performance?
Link Prediction: We report the performance of NeuSTIP and other baseline methods in Table 1. We observe that the performance of our NeuSTIP (base) is comparable with the base model of Time- Plex for WIKIDATAI2K, while the gap is more pronounced on YAGO1 as even our base model outperforms Tiee (base) on all the three met- rics with a gain of over MRR pts. Similarly, our base model is comparable to the base model of the best temporal rule-based model (TILP- P-ase) for WIKIDATAI2K whereas our base model out- performs it all three metrics the YAGO1 lk dataset. We also observe that our 'NeuSTIP w/ KGE' model, outperforms the state-of-the-art mod- els on out of 3 metrics on YAGO1 1k, and on all metrics on WIKIDATAI2K.
For these analysis questions, unless otherwise stated, we use NeuSTIP (base) model this allows us to directly assess the impact of design choices made in our model.
In order understand the in- cremental contribution of each component (Q1), we conduct an ablation study by removing three components from NeuSTIP: Allen predicates (TR), PCA score (PCA) (Bquation 4) and duration dis- tribution (INTV) (Section 4.4). Please note that duration distribution is employed only during time interval prediction, it will not change the results of link prediction. Likewise, Allen predicates (TR) are critical in predicting potential time intervals in our algorithm (Section 4.2.2) and cannot be absent from the model while doing time prediction.
Ablation study:
Time Interval Prediction Table reports the results of time interval prediction. We observe that the performance of our base model (NeuSTIP (Basc)) is better than that of all models both datasets. Our 'NeuSTIP w/ KGE' model yields further improvements, outperforming the state-of- the-art Timeplex model on YAGO1 Ik by strong margin of more than aeIOU pts. Overall, our work establishes a new state of the art for time- interval prediction in TKGC.
Table compares the link prediction perfor- mance of NeuSTIP model ablations. We observe that the presence Allen relation constraints is critical for our model - it aids performance sub- stantially in both datasets. We hypothesize that without them, low precision rules get too many groundings, which confuse the model. Next, the absence of PCA score (NeuSTIP PCA) also hurts performance, as PCA acts as informed prior the rule score.
6 Analysis
We investigate several research questions to gain further insights into our model:
Table 4 shows the effect of removing the du- ration distribution (INTV) and PCA score (PCA) on the time interval prediction of the model. We observe that both components help the model, al- though the contributions are not huge. See Ap- pendix K for more details on ablation study.
Q1. What is the effect of each component of
NeuSTIP on link and time-interval prediction?
Q2. How does model perform in the inductive learning setting where the train and the test entities are disjoint?
Inductive Setting: is generally believed that NS models perform well in inductive settings. To verify this for NeuSTIP (Q2), we conduct an ex- periment where the training and testing entities are
Q3. How does NeuSTIP perform when there limited training data available?
Q4. Are the rules generated by our model

---

Table 4: Interval prediction performance of NeuSTII and its variants, NeuSTIP-ING (w/o interval distribu- tion), NeuSTIP-PCA (w/o PCA score)
Datase:: WIKIDATA12K
KouSTi (boes)
limepiex ibeset
WIKIDATAIZK
YAGOIIk
Model
acIOU
actOU
NeuSTIP
27.36
22.49
NeuSTTI- INTV
26,80
20.24
NeuSTTP- PCA
26.91
20.63
0.30
0.35
Table 5: Link prediction results in Inductive setting
0:33
0.29
Thining Data
WIKIDATAI2K
YAGOIIk
Model
MRR
H@1 H@10
MRR
H@1
H@10
Figure Limited training data experiment WIKIDATA
NeuSTIP
25.30
16.03
44.46
20.01
14.10
31.31
TLogic
16.05
10.86
26.76
7.68
5.25
13.20
Table 7: Link prediction and Time prediction perfor-
mance on All vs integrated (ITG) Allen Predicates
Table 6: Interval prediction results in Inductive setting
WIKIDATA12K
YAGO11k
Model
Model
MRR
Hiel
1000
actOL
MRR
H481
10010
act(U
acIOU
acIOU
NeuS
31388
22.11
30.31
17.36
23.51
17.36
38.15
22.49
NeuSTIP
25.96
23.47
NeuSTIP (TG))
29.49
18.78
50.18
23.37
Timeplex
models are opaque in nature. Here, we illustrate one real example from the YAGO1 dataset it
disjoint. In order to generate the data for this exper- iment, we randomly select a subset of the temporal facts from the test data. Then, we remove any such temporal facts from the train data, which share common entity with these test temporal facts. (Ap- pendix L). We perform link and time interval pre- diction on the newly generated data in Table 5 and
shows the reasoning behind predicting the correct entity/time interval by the rules of NeuSTIP model: Query: (Franz Dahlem, isAffiliatedTo ? [1920, 1946))
Communist Party of Germany The rule that grounds the gold object:
Correct Answer:
6. We utilize Tlogic and Timeplex as baselines for link and time interval prediction respectively. It can be clearly seen that our model outperforms Time- plex by a significant margin on acIOU, because embedding- baseed models generally struggle in the inductive setting. Further, NeuSTIP outperforms TLogic all metries for link prediction supporting the hypothesis that NeuSTIP generalize well to new data.
isAffiliateedT((1, E2, T1) + During(T1, T2) A isMarriedTo(E1, E3, T2) A Contains
T3) A isAffiliatedTo E3, E2, T3)
The groundings: E1: Franz Dahlem, E2: Communist Party of Germany, Dahlem, T1: [1920,1946), T2: [1899, 1974), T3: [1920,1946]
Kathe
The above rule provides an explanation of why Franz Dahlem was affiliated to Communist Party of Germany at given time interval by reasoning that his wife Kathe Dahlem was also affiliated the party during their marriage. (See Appendix N).
Limited training data: NS models have another advantage that they can learn in the presence of less amount of training data (Q3). We test this in Figure 3 WIKIDATA 12k dataset where pare NeUSTIP (base) against Timeplex (base) on test data while training the models varied train- ing data sizes. We observe that when limited data is available (e.g. 10% data) NeuSTIP's rules are still able to capture the patterns in the data, while Timeplex struggles to perform well (Appendix M).
Importance of All Allen predicates: Recall that TILP is a recent model, which only uses three Allen relations in temporal rules: 'before', 'after' and a new aggregate relation 'touching', which combines all other Allen relation into one. In Q5, we wish to probe the value of considering all 13 Allen relations in NeuSTIP's rules, and compare it to NeuSTIP version which uses the exact three relations used in TILP. We call this model NeuSTIP (ITG). The com- parisons for link and time prediction of NeuSTIP and NeuSTIP (ITG) are provided Table 7.
Human interpretability of logical rules: One advantage of our temporal rule-based model that the predictions are in a human- camaan-nterrprttbbll form (Q4), whereas the predictions of embedding-baat based

---

Table 8: Effect rule length on link and time interval prediction for YAGO1IK Ik and WIKIDATAI I2K. Max RL denotes maximum Rule Length that was considered.
ration S 5), and long relations (duration >> 5), We find NeuSTIP oulperforms baselines in all but one case (instant relations) reiterating the importance of finding groundings (Appendix P) in a rule-based model. Further details are in Appendix O.
YAGOIIK
Max RI
MRR
Hits
Hits@ 10
aclOU
3
23.81
17.26
35.15
22.49
2
10.95
10.43
11.65
19.23
1
9.39
9.17
9.58
18.37
7
Conclusions
WIKIDATAI2K
Max RL
MRR
Hits@
Hits@10 10
acIOU
In this paper, we develop a novel neuro- -uro-symbolic TKGC method that represents the temporal infor- mation of TKGs in temporal rule language it defines, and uses rule groundings ascribe confi- dence score to a candidate answer, which can be an entity or a time interval. For that, it defines a novel scoring function. And, it also uses novel loss formu- lation for training. The key novelty of the proposed formulation is that it can perform both link pre- diction and time interval predictions in the neuro- symbolic setting, Compared to previous methods, our model has made substantial improvements in both link prediction and time interval prediction performance over two benchmark datasets. Fur- thermore, we show that our model generalizable to new entities, is human- cuman-interpretable, and works well with limited data while answering a given link prediction and time prediction query. We release our code and datasets for further research.
3
31.98
22.11
50.31
27.36
2
23.43
17.26
32.96
21.64
1 23.26
17.06
32.83
21.55
We observe that integrating many Allen relations into one has a profound impact on H@! for both datasets, as the rules lose their preciseness. They become more generic and cause other entities to get many groundings, confusing the model. We fur- ther notice that integrating Allen predicates has far- reaching consequences on time prediction because Allen predicates play crucial role in deciding and scoring the start and end time of time intervals in NeuSTIP. On further looking at data, we notice that Allen predicates that yield deterministic start/end points, such as equals, finishes, are quite im- portant for good performance in time interval pre- diction. They should not be aggregated together, that makes the scores assigned to start/end in- stances get distributed over larger range, which affects the model performance.
In the future, we will explore combinations of NeuSTIP with other orthogonal paradigms such as models that learn box embeddings instead of vector embeddings (Cai et al., 2021), and models based on graph neural network (Zhu et al., 2021). Exploring different methods of ensembling our rule-based and embedding-basse models (apart from linear ensembling) can be another future work direction.
Rule Length: We perform experiments to study the effect of rule length on the performance of NeuSTIP (Q6). We define a rule of rule length i as one whose rule body consists of i KG rela- tions and Allen relations. A rule set of maximum (max.) rule length i consists of all rules that have length from 1 to in it. For efficiency purposes, restrict the maximum rule length to 3. Our results for link and time interval prediction are presented in Table 8.
Acknowledgements
This work is supported by IBM AI Horizons Net- work, grants from Google, Verisk, and Huawei, and the Jai Gupta chair fellowship by IIT Delhi. We also acknowledge travel support from the Profes- sional Development Allowance (PDA) fund of IIT Delhi. We thank the II Delhi HPC facility for its computational resources. We sincerely thank Ananjan Nandi for useful discussions during the course of the research.
From the table, we conclude that the rules of length 3 perform significantly better than the rules of length 2 because length 2 rules fail to capture the entity chains (A + - B) in cyelic rules such as Tb (A, B, T1) <- A1 (T1, T2), r(A, D, T2), AT2, T3), r " C, T3), AT33 T4), r(C, B, T4) in its body which are important while finding the path from the head to the tail of temporal fact.
Limitations
To analyse the model further, we performed ad- ditional experiments on the effect of NeuSTIP per- formance on different types of temporal relations instant relations (duration = O), short relations (du-
One limitation of our model, like mule-based models in general, is that they cannot directly capture the
Pht / ttw./ com/dairr-itd//eeeee

---

numeric features present in the data. Although we exploited the gadgets subsumed in Timeplex (Jain et al., 2020) to capture numeric features present in TKGs, capturing these features directly inside our model is an interesting future direction. Further, our proposed model currently deals with closed path rules only and could benefit more if open paths can be exploited for time prediction.
'13, page 413-422, New York, NY, USA. Association for Computing Machinery.
Alberto García- Sebastijan Dumancic, and Math- ias Niepert. 2018. Learning Sequence Encoders for Temporal Knowledge Graph Completion. In Con- ference on Empirical Methods in Natural Language Processing.
Rishab Goel, Seyed Mchran Kazemi, Marcus Brubaker, and Pascal Poupart. 2020. Diachronic Embedding for Temporal Knowledge Graph Completion. In AAAI.
Ethics Statement
We anticipate no substantial ethical issues arising due to our work on link prediction and time interval prediction for Neuro-Symbolic TKGC.
Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. 2021a. Explainable Subgraph Reasoning for Fore- casting on Temporal Knowledge Graphs. In ICLR.
Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp. 2021b. Learning Neural Ordinary Equations for Forecasting Future Links Tempo- ral Knowledge Graphs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 3352-8364, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
References
Ralph Abboud, ismail Īlkan Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. 2020. BoxE:
A Box Embedding Model for Knowledge Base Completion. In NeuRIPS.
James F. Allen. 1983. about Temporal Intervals. 26(11):832-84 843..
Maintaining Knowledge Commun. ACM,
Zhen Han, Yunpu Ma, Yuyi Wang. Stephan Günne- mann, and Volker Tresp. 2020. Graph Hawkes Neu- ral Network for Forecasting on Temporal Knowledge Graphs. In AKBC.
Antoine Bordes, Nicolas Usumier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi- relational Data. In NeurlPS, pages 2787-2795.
Prachi Jain, Sushant Rathi, Mausam, and Soumen Chakrabarti. 2020. Temporal Knowledge Base Com- pletion: New Algorithms and Evaluation Protocols. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 3733-3747, Online. Association for Computa- tional Linguistics.
Ling Cai, Krzysztof Janowicz, Bo Yan. Rui Zhu, and Gengchen Mai. 2021. Time in Box: Advane- ing Knowledge Graph Completion with Temporal Scopes. In Proceedings of the 1 1Ith on Knowledge Capture Conference, K-CAP '21, page 121-128, New York, NY, USA. Association for Computing Machinery.
Tingsong Jiang, Tianyu Liu, Tao Ge, Lei Sha, Baobao Chang, Sujian Li, and Zhifang Sui. 2016. Towards Time-Aware Knowledge Graph Completion. In Pro- ceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1715-1724, Osaka, Japan. The COL- ING 2016 Organizing Committce.
Kyunghyun Cho. Bart van Merriënboer, Caglar Gul- cehre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. 2014, Learn- ing Phrase Representations using RNN Encoder- Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar. Association for Com- putational Linguisties.
Woojeong Jin, Meng Qu. Xisen Jin, and Xiang Ren. 2020. Recurrent Event Network: Autoregres- sive Structure Inference over Temporal Knowledge Graphs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6669-6683, Online, Association for Computational Linguistics.
pages
Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. 2018 HyTE: Hyperplane-based Temporally aware Knowledge Graph Embedding. In Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pages 2001-2011, Brussels, Belgium. Association for Com- putational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Luis Antonio Galárraaa, Christina Teffioudi, Katja Hose, and Fabian Suchanek. 2013. AMIT: Association Rule Mining under Incomplete Evidence in Ontolog- ical Knowledge Bases. In Proceedings of the 22nd International Conference on World Wide Web, WWW
Thomas N. Kipf and Max Welling. 2017. Semi- Supervised Classification with Graph Convolutional Networks. Proceedings of the Sth International Conference on Learning Representations, ICLR '17.

---

Timothée Lacroix, Guillaume Obozinski, and Nicolas Usunier. 2020. Tensor Decompositions for Tempo- ral Knowledge Base Completion. In International Conference on Learning Representations.
Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. 2017. :volvevolve: Deep Temporal Rea- soning for Dynamic Knowledge Graphs. In Proceed- ings the 34th International Conference on Machine Learning- Volume 70, 17, page 3462-3471. JMLR.org.
Yushan Liu, Yunpu Ma, Marcel Hildebrandr, Mitchell Joblin, and Volker Tresp. 2021. TLogic: Temporal Logical Rules for Explainable Link Forecasting on Temporal Knowledge Graphs. In AAAI
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. 2016. Complex Embeddings for Simple Link Prediction. In ICML, pages 2071 2080-
Jin Luo, Hong Shen, YanFeng Hu, and Chen Peng. 2022. TaCE: Time-aware Convolutional Embedding Learn- ing for Temporal Knowledge Graph Completion. In under review.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. 2020. Composition- Multi- relational Graph Convolutional Networks. In Inter- national Conference on Learning Representations.
Xin Mei. Libin Yang, Xiaoyan Cai, and Zuowei Jiang. 2022. An Adaptive Logical Rule Embedding Model for Inductive Reasoming over Tem Temporal Knowledge Graphs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7304-7316, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Mod- els. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pages 4281 22814429, Dublin, Ireland. Association for Computational Linguistics.
Christian Meilicke, Melisachew Wudage Chekol, Daniel Ruffinelli, and Heiner Stuckenschmidt. 2019. Any- time Bottom-up up Rule Learning for Knowledge Graph Completion. In Proceedings of the Twenty-Fighth Artificial Intel-
Jiapeng Wu, Meng Cao, Jackie Chi Kit Cheung, and William L. Hamilton. 2020. TeMP: Temporal Mes- sage Passing for Temporal Knowledge Graph Com- pletion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5730-5746, Online, Association for Computational Linguistics.
International Joint Conference ligence, IJCAI-19, pages 3137-3143. International Joint Conferences on Artificial Intelligence Organi- zation.
Johannes Messner, Ralph Abboud, and ismail ilkan Ceylan. 2022. Temporal Knowledge Graph Com- pletion using Box Embeddings. In AAAL, volume abs/2109.08970.
Siheng Xiong, Yuan Yang, Faramarz Fekri, and James Clayton Keree, 2023. TILP: Differentiable Learning of Temporal Logical Rules on Knowledge Graphs. In ICLR.
Ananjan Nandi, Navdeep Kaur, Parag Singla, and Mausam 2023. Simple Augmentations of Log- ical Rules for Neuro-Symbolic Knowledge Graph Completion. In Proceedings of the s1 Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 256-269 Toronto, Canada. Association for Computational Linguisties.
Chengjin Xu, M. Nayyeri. Fouad Alkhoury. Hamed Shariat Yazdi, and Jens Lehmann. 2020. TeRo: A Time-aware Knowledge Graph Embedding via Temporal Rotation. In International Conference on Computational Linguistics.
Fan Yang. Zhilin Yang, and William W Cohen. 2017. Differentiable Learning of Logical Rules for Knowl- edge Base Reasoning. In NeuRIPS, volume 30. Cur- ran Associates, Inc.
Meng Qu. Junkun Chen, Lascal A. C. Xhonneux, Yoshua Bengio, and Jian Tang. 2021. RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs. In ICLR, pages 1-21.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. KG- BERT: BERT for Knowledge Graph Completion. AAAI
Ali Sadeghian, Mohammadreza Armandpour, Anthony Colas, and Daisy Zhe Wang. 2021. ChronoR: Rota- tion Based Temporal Knowledge Graph Embedding. AAAI.
Yuyu Zhang, Xinshi Chen, Yuan Yang. Arun Rama- murthy, Bo Li, Yuan Qi, and Le Song. 2020. Efficient Probabilistic Logic Reasoning with Graph Neural Networks. In International Conference on Learning Representations.
Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. 2019. DRUM: End- To-End Differentiable Rule Mining On Knowledge Graphs. In NeuRIPS, volume 32. Curran Associates, Inc.
Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan Cheng, and Yan Zhan. 2020. Learning from History: Modeling Temporal Knowledge Graphs with Sequen- tial Geny-Geenr Networks. In AAAI Conference on Artificial Intelligence.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Yun Nie, and Jian Tang. 2019. RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In ICLR.

---

Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhon- neux, and Jian Tang. 2021. Neural Bellman- Ford Networks: A General Graph Neural Network Frame- work for Link Prediction. Advances in Neural Infor- mation Processing Systems, 34.
ure In this said example, pF,pF wBi, 1MT denote abbreviations for the relations playsfor, playFor wasBornIn, isMarriedTo respec- tively. The relations A1 to A5 represent Allen relations. In order to learn a rule that is based upon target temporal fact wBi(David, London, Т6) the model would first obtain walk David (iMt,T4), Victoria (vBi,75), London on GAW- This walk would further be expressed in the logical form as iMt(David, Victoria, T wBi (eiiiiiiiia London, T5).
A Allen's Interval Calculus
The temporal facts considered in our setting en- code time intervals in the last argument, requiring the use of Allen relations as formal technique that captures the temporal relations between the time intervals present in the data. As discussed in Section 3.1, we utilize all the 13 Allen relations possible between any two time intervals in our tem- poral rules. In this section, we describe all 13 Allen relations in Figure 4 in detail. Each of the relations in Allen algebra calculus set of rules. For example if we have one time interval X [Xstort; Xend] and another time in- terval Y [Ystart, Yema] then the Allen relation before exists between them, i.. before(X, (X, Y) iff xstart < Xond < Sstartt < Yond- Similarly, the constraints for each of the 13 Allen relations is defined in the last column named 'Chronological Sequence' of Figure 4.
In the next step (b), when Allen relations are introduced into the walk, the corre- sponding example would be expressed as TT,, T4) A (David, Victoria, T4) A A5(T4 T5) A wBi (Bicttooria,, London, T5). Please note that these Allen relations denote one of the 13 relations in Allen Algebra calculus. The final rule after introducing variables is wBi(A, B, C) t A4 F) A iMt F) A AF G) A wBi(D, B, G). This rule is expressed without the entity and time interval variables in Figure (rrihht) wBi A4 A 1Mt A5 A wBi.
be written
C GRU for Embeddiig--based Rule Score
B The Details and an Example of Temporal Logic Rule Extraction
In this section, we provide the details of GRU uti- lized in Section 1.2. NeuSTIP learns umique em- bedding L representation for the body ^/ Arr of a given rule Lj. Motivated by ARLE-IR model (Mei et al., 2022), we employ Gated Re- current Unit (GRU) model (Cho et al., 2014) to learn Lg embedding. At time f, the input of the form x; = [Ar rı] is fed to GRU where A, and rt are the embedding vectors of t-th Allen pred- icate At and KG relation Tt in rule Lý's body. GRU unit utilizes the following functions in order to generate the hidden-layer embedding h, at time
We elaborate on temporal logic rule extraction de- fined in Section 4.1 along with an example. In order to carry out all-walks, it expresses TKG as graph GAw wherein each quadruple is expressed as (2.T) o denoting an edge (I; T) between entities s and (see Figure (left)). NeuSTIP mines all walks over Gaw in three steps: (a) First, begin- ning at s, it performs agnostii walks of length m on dл such that the final node of the walk is o (Figure 1 (middle)). It then expresses these walks in logical form. At this stage, these all walks exclusively consist of KG relations. (b) Next, it introduces Allen relations into the all-walks cap- tured thus far, in order to bind the time intervals existing between neighboring KG relations in the walk. Further, a special Allen relation is introduced to bind the time interval of target temporal fact and the first KG relation in the walk. (c) At the final step, it substitutes the constants with variables to generalize grounded rule into final rule (Fig. (right)).
rt o (W,- Xt + U, h + br)
(9)
z: a (Wz X; + Ug Ug- + b.)
(10)
n. tanh (Wn X + +ro h + bm)
(11)
h (1 z @ + z h
(12)
where rt is the reset gate that allows the hidden state to discard information that is insignificant in the future and z; is the update gate that controls how much information from h is carried over to hr- The final hidden state embedding hm af- ter sequential steps of GRU represents the path embedding Lj of given rule L,.
Our example is based upon a fragment of the YAGOlIk dataset which is shown in Fig-

---

Allen Predicates
Pictoral Example
Chronological Sequence
Inverse Relations
Relations
before(X, Y)
after(Y,X)
Xaae Mena start end
X
equals(X, Y)
equals(V,X)
Muart start Mnn and
Y
meets(X,
met by(Y, X)
Xyme Kend start end
overlaps(X, Y)
overlapped by(Y, X)
Mtarrt sare Xand Yend
X
contains(X, Y)
during(Y,X)
Manee Ytar Yend" Xand
Y
X
starts(X, Y)
started by(Y, x)
X start Xend < Yend
Y
X
finishes(X, Y)
finished by(Y, X)
stant < Xeart Xand = and
Figure 4: This figure lists all the 13 relations Allen algebra calculus (Allen, 1983). The pictorial example the third column is for the relations in the first column
D PCA Score Metric for Temporal Data
4.3 the model sorts the years in T in increasing order and assign a unique id to each of them. The difference d is then taken between those ids. The difference is then divided by the maximum differ- ence between any two ids, in order to follow the constraint that 0 S d (.) 1
Here we explain PCA score utilized in Section 4.2.1. PCA metric (Galárraga et al., 2013) is based on the Partial Closed World assumption according to which if we know one object for a given s and T in a temporal fact (s, o, T) then we know all the o' for that and T. If we consider temporal rule Is, o, T), the PCA score of this rule for link prediction, thst (L)) is:
F
Data Statisties
Ly to be
The details of the datasets used for experimentation in Section are provided in Table 9. We utilize two standard TKG datasets YAGOI and WIKI- DATAI2K for our experimentation, Both these datasets are time interval- datasets. We uti- lize the standard train, valid and test splits for these datasets in experiments.
#(s. o, B, T) > O o, T): B, > O 2,
Here, N(s, o, T) denotes the path in the body B of the rule Lj. This implies that we divide the number of positive examples satisfied by the rule by the total number of (s, o, T) satisfied by the rule such that Fn (s, o', T) is a positive example for some o'. Similarly, we define the PCA score for start time instance the Vatat (L))(tt)) as
Table 9: Statistics of Temporal KG datasets
Features
YAGOII
WIKIDATAI2K
#Entities
10622
12554
#Relations
10
24
#Instants
251
237
(s. s. o. tb) Meel > A St. T. (,, o. (bo, tz]) # o, t.): |Ncl > A FT' : TP
#Intervals
6651
2564
# #Training
16408
32497
#Validation
2051
4062
Neb| is a notation for [N(s, B, tb)). This im-
#Test
2050
4062
plics that we divide the number of positive exam- ples P satisfied by the rule by the total number of
G Experimental Details for NeuSTIP
(s, T) satisfied by the rule such that (s, T') is positive example for some T'.
For all the results reported for the proposed model in Table and 2, we optimize parameters of the loss function defined in Section 4.3 with an Adam optimizer (Kingma and Ba, 2015) while decreasing
E Distance Computation between Time Instances
order to find the distance between two time instances ta and to iec. d(ta tb), in Section

---

the learning rate by Cosine Annealing ensuring that the minimum learning rate at any time during the training does not fall below the one-fifth of its ini- tial value. To get the best results for link prediction, we train our model for 5000 epochs. Likewise, for time interval prediction, we train the model for 2000 epochs. We set dimensionality for all the Allen relation embeddings, KG relation embed- dings, and the rule head embeddings (which have the same dimension as the hidden dimension of the GRU) to be 32. Besides, we set the maximum rule length as for both datasets.
rule head, we disallow this temporal fact to re- oceur in the body of the rule avoid mining cyclic rules. Further, there are some cases in time prediction where we have 0 groundings for all the instances with respect to both start and end scores. In such cases, for given rela- tion I, we predict I_stari as mean hean_starr([r) and t_end as mnean_starlly [rran_star[r] and mean_offset(r] are the average start and the average offset of intervals for the re- lation I, computed in terms of the assigned ids (as explained Appendix E).
mean_ _fffee[[[[] Here,
Further, 7) in Equation 8 a learnable parameter that is trained along with the rule score (Ly)) [c] in Equation 4. The initial value of is chosen hyperparameter and is tuned over the dev set selecting the best value of MRR for link prediction and acIOU for time prediction.
While computing @(path) (Eqn. 5) for the start and end time instances in time interval prediction, there are also cases when т = (see Eqn 1). For such cases, the Gaussian distributions are computed for the difference between successive occurrences of start/end time for fixed (s,r). In such cases, given the fact ri ri(e), e2; [t2,, f22)). NeuSTIP calcu- lates path score for start time candidate (l in query ra (e1, cm+1, ?) as,
During the training of the model, we select the best validation model for link prediction based on the MRR metric and the best validation model for time interval prediction based on the aeIOU metric. Further details of the hyperparameters adopted for all the experiments are provided in Appendix H and more details about restrictions on the ruleset are explained in Appendix I.
((path)(tIp) = ((ab((tl, +220)
(13)
Here, abs represents absolute value. The same approach is used to estimate ((path)[t1e]. Additionally, the loss function for Time In- terval prediction (see Section 4.3) is normalized for each positive start instance by dividing it by Σ En,eN, d(ns, t4)) for given ts. and similarly for each positive end instance.
H Hyper-Paraneter Settings for NeuSTIP
For both link and time interval prediction in Table I and 2, we set our learning rate to be le - for both datasets. Table 10 lists the values of the coefficient 1) (in Equation 8) which we multiply to the overall KGE score while ensembling with our based model for the two datasets. For link prediction, we consider 7 from the set (0, le-3, 1e-2, }, and select the best value of 1) based on MRR the dev set. For time interval prediction, we consider 7) from the set (0, 1e-3, 1e-2, 1e-1, 1) and deter- mine the best value of based on aeIOU on the validation set.
Statistical Richness of the Rules
We compute metrics such as the number of rules, and the average number of groundings per rule as a measure for computing the statistical richness of the rules obtained in our model. The proposed methodology of performing all walks in NeuSTIP generated 8186 rules for YAGOI 1k and 31,807 rules for WIKIDATAI2K dataset. The reason for obtaining moderate number of rules while we con- sider all walks on TKGC the sparsity of temporal KGCs as already discussed in Section 4.1. Further, the average number of groundings per rule con- sidering the train set is 17.87 for YAGOI Ik, and 885.21 for WIKIDATA 12k, which is sufficiently high.
Table 10: The best hyperparameter settings for link and time prediction over TKG datasets
Hyper-parane parameterre (Link prediction)
YAGOLKK
WIKIDATAI2K
le-3
le-2
77 (Time prediction)
le-1
le-1
I More Implementation Details
K Link Prediction for Ablation Study
During the process of rule extraction in Section (4.1), given the temporal fact [n o, T) in the
This section provides the details of the experimen- tal setup for the ablation study on link prediction

---

performed in Table in Section 6 (Q1). In order to perform link prediction for the models: NeuSTIP, NeuSTIP- TR and NeuSTIP-PCA, we utilize the same initial value of hyperparameters as reported in Table 10 in Appendix H. To train the model with rules that are without the Allen relations (NeuSTIP- TR), we essentially treat the absence of Allen re- lations special 'NOREL' constraint, and feed the embedding of this constraint as input xy (Appendix C) the GRU at every step instead of giving the Allen relation embeddings as imput.
like the number of random walks and time window size, leads better model performance. There- fore. for both datasets, we increased the number of random walks to 20/ from its default value of 200 and set the time window size to 1000. Simi- lar to NeuSTIP, we generated rules of lengths 1,2, and 3 using the exponential transition distribution of TLOGIC that performs better than the uniform distribution.
M Limited Data Study
This section is complementary to the limited data study conducted in Section 6 (Q3) as we perform limited data study for the YAGO1 1K dataset here. We consider different percentages of the original data (10% 100%) and train NeuSTIP (base) on this data and the model using the standard test data. We compare our model against the Time- plex (base) model in Figure 5, As can be seen, limited training data areas (10% of total training data), the gap between the time-prediction perfor- mance of our model and Timeplex becomes quite significant. While embedding-basse -baed models are expected to perform well in the data-scarce sce- nario, the strategy of NeuSTIP handle the cases where no relevant rule is grounded helps in improv- ing the model performance significantly. Such a scenario is often encountered in the limited data setting. To this end, when no rule is grounded for query r(s, o, ?), NeuSTIP uses a priori knowledge of the mean of start time instants 1g and the aver- age duration (intu) of relation and to predict the time interval [romg % gang + intu].
L Experimental Setup for Inductive Learning Study
This section explains the experimental setup for the inductive learning study (Q2) performed in Section 6. Our experimental setup is motivated by a similar study performed in DRUM model (Sadeghian et al., 2019).
Data generation: In order generate our induc- tive test data, we randomly select subset of the temporal facts from the standard test data, our experiments, we chose 50% of the standard test data. We train our model on one set of inductive training data that helps the model in learning rules along with their confidence. It is to be noticed that if we utilize the inductive training data as back- ground knowledge the test time, then because of the disjoint set of entities the train and the test time, we would not be able to obtain any ground- ings of the rules. To overcome this, we employ the standard train data and validation data as the background knowledge at the test time in order to ground the body of the rules. The key point here is that the rules along with their confidences that are learned on one set of entities can be applied to different set of entities as well since the rules generalizable and are not bound to fixed entities.
Dataset: Yagal1K
0239
KeuSTIP (bons) timeplex (base)
We compared NeuSTIP with TLOGIC (Liu et al., 2021) which addresses the problem of link fore- casting on temporal knowledge graphs. Unlike our solution, TLOGIC works with the time-instant dataset. Therefore, for a comparable setup, we used our datasets only while treating the start time instant of the interval the timestamp of the re- spective fact. We used the publicly available TLOGIC implementation and ran using the same training data and background knowledge as that for NeuSTIP. The ablation study of TLOGIC sug-
0.:33
0.002
0.09
0.66
Traning Data
Figure 5: Limited training data experiment on YAGOIIK
N Human interpretability of Rules
This section provides more examples of the inter- pretability feature (Q4) of NeuSTIP put forward in
Pht com iu-yyshan/TLoogi gested that the higher value of the hyperparameters,

---

Section 6. Next, we provide another example of an interpretable rule from the YAGO1IK dataset for time prediction as below.
Empire, E3: Finland, E4: Mikkeli Province. T2: [1917.presentl, T3: [1917ppreeent], T4: [0,1917]
Query: (Donna Hanover, isMarriedTo, Rudy Giuliani. ?)
The above rule explains that Turku is part of the country Finland since 1917, and another place Mikkeli Province is also part of Finland since 1917. Just before this, Mikkeli Province was part of Russian Empire, the rule pro- vides us the temporal information that Turku was also part of Russian Empire just before being a part of Finland.
Correct Answer: [1984, 2002]
Rule grounding gold start and gold end:
isMarriedTo E1, E2, T1) < (qu T2) A
isMarriedTo 1 (E1, E2, T2)
The Grounding: E1: Donna Hanover, E2: Rudy Giuliani, T2: [1984,2002]
The above rule provides the temporal information that since Rudy was married to Donna for given time interval, Donna was also married to Rudy for the exact same time interval due to the symmetric nature of the isMarriedTo relation.
Time Interval Prediction for Distinct Relation Classes
Table 12: aeIOU across relation classes in YAGO1IK and WIKIDATA 12k datasets
Next, we present two examples from WIKI- DATA12k dataset on how the rules of NeuSTIP provide human interpretable predictions. We begin by considering an example of link prediction in WIKIDATAI2K dataset.
YAGOIIK
Model
Instant
Short
Long
NeuSTIP (base)
9.32
23.93
58.71
NeuSTIP w/ KGE
19.03
24.59
60.43
Timeplex
18.39
20.63
24.80
WIKIDATAI2K
Query: (Ammerschwihr, liate, ?, [1920,
Model
Instant
Short
Long
present])
NeuSTIP (base)
35.44
25.98
22.78
Correct Answer:
Haut-Rhin
NeuSTIP w/ KGE
36.97
26.11
27.49
Timeplex
34.51
24.80
20.93
The rule that grounds the gold object:
MetBy(T1, T2) liate (E1, E3, T2) Equals T3) liate 1(33, E4 т3) A Meets (T3, T4) A liate E4, E,,T4)
Siate(E1, E2, T1)
The goal of this experiment is to observe the perfor- mance of the proposed model for different classes of relations present in the TKG. Motivated by Time- plex model (Jain al., 2020), we categorize the relations into three classes: Instant, Short and Long. Instant relations are those whose start time and the end time coincide in a given time in- terval. Each relation in the Short category has an average time duration of less than five years. Each relation in the Long category has an average time duration greater than five years. A category- wise detail of relations in YAGO1 and WIKI- DATA12K datasets is shown in Table 11.
Groundings: E1: Ammerschwihr, E2: Haut- Rhin, E3: Upper Alsace, E4: Soultzmatt, T1
[1920, present], T2: [1871, 1920], T3: [1871, 1920), T4: [1920, present]
Here, the abbreviation liate in WIKIDATAI2K stands for located in the administrative territorial entity. The above rule pro- vides an explanation of why Ammerschvihr is lo- cated in the administrative entity of Haut-Rhin since 1920, by reasoning that Ammerschvihr and Soultzmatt were both part of the entity Upper Alsace for the same time interval just before 1920, and Soultznn tzmat became a part of Haut-Rhin in year 1920. The next example explains time predic- tion WIKIDATAI2K by NeuSTIP ruleset.
The detailed results of NeuSTIP (base), NeuSTIP w/ KGE, and Timeplex across different relation classes are presented in Table 12. As we observe NeuSTIP (base) considerably outperforms Timeplex in five out of six cases. This supports the hypothesis that our proposed model captured the patterns present in different relation classes by its learned rules. Further, NeuSTIP w/ KGE con- sistently outperforms the Timeplex model. Please note that the reason for the performance degrada- tion of NeuSTIP (base) for Instant category YAGO1 LK is that there are no groundings available
Query: (Turku, country, Russian Empire,??
Correct Answer: [1809,1917
Rule that grounds the gold start and gold end: country [E1, E2, T1) Meets T2) A countr (E1, E3, T2) Equals(T2, T3) A country- (( E3, E4, T3) A MT, T4) A country(E4, E2 T4)
Rule Grounding: E1: Turku, E2: Russian

---

Table 11: Categorization of YAGO1 Ik and WIKIDATA12K relations into Instant, Short and Long categories.
YAGOlIk Relation-wise Category
Instant
Short
Long
diedIn, WasBomin
playsFor, graduatedFrom
isMarneTo,, isAfiliittedo, owns, worksAt, created, hasWonPrize
WIKIDATAT2K Relation-wise Calegory
Instant
Short
Long
nominatedFFor,
InemeerOrSporte Team, award
positionHeld, locatedin.kmminssttattii http://tattttiirrrrrrrrraaaainit spouse instanceOf, employer, memberOf, countryfChiizeshhpp.
winner, academic
Reeceveed, significantveet,
Degree
iiimmedddministraiieeeoooo educatedAt
titeeOHChessPorson, memberOffPllticalParyy residence, country ContainsAAmminiitatiieeeeent Eerroraiiiier.ririiiilnnnntttt capitalOr.
IMAStatsAnnOORRank, besigugeDssignatonn headOtGovernment
for many temporal facts lying in this category. Ac- cordingly, the explanation for the high performance in Long category of YAGOI that the model discovers perfect rules for a considerable amount of temporal facts which results a performance boost. For instance, for the relation isMarri that belongs to the Long category, the model dis- covers the inverse of isMarriedTo in the rule set and yields better performance by exploiting the symmetric nature of this relation. The other Long relations where we perform better than TimePlex are hasWonPrize and isAffiliatedDo relations, there are a substantial number of tempo- ral facts for which closed walk groundings exist, and hence the target start time and end time are often grounded in these cases, leading to better performance.
obtain a positive score in Equation 5 for the true start time and true end time, then we will assign the temporal fact to GR, otherwise we assign to NGR. Here, of course, the path also has to be from the head 's' to the tail "o' in the rule body. The effect of rule groundings on link prediction and time interval prediction presented in Table 13.
Table 13: Effect of rule groundings on link and time prediction for YAGOIIK and WIKIDATAI2K datasets.
YAGOTIK
these
Category GR
MRR
Hits@1
Hits@10
acIOU
33.57
24.48
49.36
30.07
NGR
0.018
0.0
0.0
9.86
WIKIDATAI2K
Category GR
MRR
Hits@1
His@@ 10
acIOU
36.03
24.55
50.31
29.65
NGR
0.016
0.0
0.0
14.89
P Effect of Groundings on the Model Performance
As can be observed from the table, the perfor- mance of the model gets disappointingly low when the model can not discover the groundings of tar- get tail in a given rule set (NGR class). Based on this study, we conclude that the performance of the model critically depends upon whether a path ex- ists from the head to tail in a given TKG in order to generate non-zero scores for a temporal fact. This is typical behavior of KSC models and similar behavior has been observed in the past for static NS- KGC models such as RNNLogic (Qu et al., 2021), ExpressGNN (Zhang al., 2020) models. How- ever, this effect can be alleviated to some extent by rule augmentation techniques recently proposed in Nandi et al. (2023). Contrastingly, the TKGE models would always generate a non-zero score for a given temporal fact because they employ a scoring function that composes the embeddings to generate the resulting score without fail.
This empirical study is motivated by the ques- tion: how critical are the rule groundings in order to achieve superior performance in the proposed model? In order conduct this study, we train the NeuSTIP (base) in a typical setting. However, at the test time, we categorize our test temporal facts into two categories; GR and NGR. For a link pre- diction query (,,,, GR encodes those temporal facts for which the true tail answer 'o' is reached by following the grounded path of at least one rule's s body while beginning the path at the temporal fact head 's', otherwise the temporal fact is assigned NGR category. Likewise, for time prediction query r(s.o,?), GR encodes those temporal facts for which the true start time '4p' and the true end time tte are grounded by some rule. Specifically, if we

---

Effect of the first Allen Relation on Model Performance
formance as compared the OVERALL mode. This is because the head Allen relation does not as directly influence the entity reached by the rule during link prediction as compared to the time pre- diction. The impact on link prediction seems to be governed by the total number of rules, as more rules help in grounding the gold entity because more paths are available to ground them. Because of this, when the rules are divided into DET and NONDET categories, fewer rules are available for link prediction, resulting in a drop in performance. In summary, using both sets of relations finds com- petitive performance in all settings.
Table 14: NeuSTIP model's 's performance based on first
Allen relation di in rule language.
YAGO1Ik
Allen Relation
MRR
aclOU
DET
18.62
22.55
NONDET
17.50
16.62
OVERALL
23.81
22.49
WIKIDATAI2K
Allen Relation
MRR
ac1OU
DET
23.45
27.31
NONDET
25.30
21.97
OVERALL 31.98
27.36
R Influence of Ensembling on the Proposed Model
It is very difficult to study the impact of Allen rela- tions present in given rule in our current rule lan- guage in general because multiple Allen relations are present one rule. Further, Allen relations are interleaved between the KG relations to bind the time of two neighboring KG relations in the rule body. However, the first Allen relation dj present in rule body (Equation 1) has its unique contri- bution to the score computation, especially in the case of time interval prediction discussed in the paper. Hence, for our study, we divide Allen predi- cates into two categories: DET refers to the Allen predicates that fix either the start or the end time of the time interval for e.g.: finishedby, meets, metby, starts, startedby, equals and NON- DET refers to the Allen predicates where both ends are free, for example: before, after, overlaps. during, contains, everlappedby
Our goal here to study the question: when we ensemble NeuSTIP with Timeplex in Table 1, how do they influence each other? In order to answer this, we considered the ranks of NeuSTIP (base), Timeplex and an ensemble (NeuSTIP w/ KGE) individually for each test quadruple obtained during link prediction and performed comparative study to understand how the KGE models contribute the performance of NeuSTIP.
Our major conclusion after analysis is that NeuSTIP mainly fails for the cases when no rule gets fired for a given quadruple for the gold entity resulting in a zero score in Equation 2 in paper. For instance, we found 796 such quadruples (out of total 4102 test instances) in YAGO1 Ik and 609 quadruples (out of 8121 quadruples in test data) in WIKIDATAI2K, where our model generated zero score. For such instances, the average rank of our model was 5327 for YAGOI Ik, and 6307 for WIKIDATA 12k while the average rank of Time- plex for such cases was 1654.51 for YAGOI and 1420.07 for WIKIDATA 12k. Therefore, our model relies ensembling for the cases when it obtains groundings for any rules learnt by NeuSTIP for given quadruple. However, for the cases when the model obtains the groundings, performs bet- ter than the corresponding Timeplex and Ensemble (NcuSTIP w/ KGE) model. For instance, when NeuSTIP obtained its groundings, its average rank is 51.1, while the average rank of Timeplex for such cases is 357.54, and is 169.70 for ensemble for YAGOlIk datasets. Similarly, the average rank of NeuSTIP for cases when groundings were found is 38.09 while the average rank of Timeplex for such cases 103. 27, and ensemble for such cases
We, then, categorize the rules into two categories based upon whether the first Allen relation of the rule falls into DET or NONDET category while ignoring the rules in the other category and pre- dicting scores in Equation 2 based on rules in a given category. We also consider a third category- OVERALL- that constitutes the results in Table 1 and 2 of the paper that are obtained with all rules considered. The results for the link prediction and time interval prediction in three categories are pre- sented in Table 14.
Results DET category are better than NON- DET category for link prediction for YAGO1 Ik dataset and the results of NONDET category are better than DET category for link prediction for WIKIDATA12K dataset. For time interval predic- tion, DET alone can help the model in achieving performance similar OVERALL model perfor- mance for time prediction. Further, for link pre- diction, using only one of them reduces the per-

---

is 88.65 for WIKIDATAI2K, Hence, we conclude that NeuSTIP relies on ensembling when it finds no grounding but outperforms the ensembling and the Timeplex model when it obtains the groundings for the candidate entity in link prediction for the rules learnt by NeuSTIP.
Experimental Setup of NeuSTIP (base): We first sorted YAGOI I/WIKIDATAI2Z, datasets accord- ing to the start time of time intervals, to make them suitable for forecasting ensuring the time train and test sets are disjoint. Specifically, we list the possible values of the start time for train, valida- tion, and test phases in YAGOI IK/WIKIAAA 12k datasets in the Table 16. The size of YAGOI Ik dataset (number of quadruples) after this split (with- out adding inverses) is: 16,806 (train). 1823 (valid), 1880 (test). Likewise, the size of WIKIDATA 12k dataset after this split is 33,368 (train), 3649 (valid), 3604 (test) without adding inverses. We then conducted our experiment with the following constraints:
S Influence of Ensembling on Contemperancous Models
The significant performance gain of NeuSTIP af- ter ensembling it with Timeplex (a pure neural approach) directed us to study the impact of the ensembling approach on the other baseline mod- cls. To that end, we experimented (results in Table 15 below) with the state-of-the-ar baseline model TILP (Xiong et al., 2023) ensembled with the Time- plex for different values of eta hyperparameter that influence the weightage of both approaches (sce Equation 8). In our preliminary attempt at ensem- bling, we observed different trend compared to what we noted in NeuSTIP, i.e., the ensembling did not improve the performance of TILP. This counter-intuitive observation points towards the so- phisticated process of ensembling. An in-depth analysis of ensembling methods for leveraging the goodness of symbolic and neural approaches can be an interesting future research direction.
(a) during the rule generation phase, we ensure that the start time of any interval oc- curring in the body of the rule is strictly less than that of the quadruple used in the head. Conse- quently, our model finds fewer rules. (b) Similarly during grounding the rules, while performing pa- rameter learning, we maintain the above condition. (c) While running inference on the test set, this condition is automatically ensured due to the way in which the dataset is split. It is important to note that due to fewer rules and lesser groundings found in this split, our model's performance also reduces. Results of NeuSTIP (base) for Link Forecasting: obtained the final results of link forecasting by employing time-aware filtering (Jain et al., 2020) that we have employed in all the experiments our paper. The results are given Table 17.
Table 15: Effect of Ensembling TILP with Timeplex.
ETA
10c-3
10c-5
10c-6
10c-8
0
H@1
10.09
11.16
11.26
11.80
16.81
H@ 14.19
16.86
17.47
18.79
41.39
MRR
11.69
13.24
13.50
14.25
25.32
Table 17: Link Forecasting for NeuSTIP (Base)
YAGOTIk
T Future Link Forecasting using NeuSTIP Model
MRR 7.08
H
H@ 10
3.38
13,86
WIKIDATAI2K
MRR
Hel
H@
Our goal here is to perform future link forecasting on NeuSTIP (base) such that time instances in train and test sets are chronologically ordered and are disjoint. We now explain the experimental setup followed by the results.
13.82
7.45
26.07
Link Forecasting for TANGO model: We now explain link forecasting TANGO model (Han et al., 2021b) which is a past model in literature from which our link forecasting experiment has been motivated. Please note that the results ob- tained below are not directly comparable to the results of link forecasting in NeuSTIP (base) pro- vided above because NeuSTIP is time-interval dataset and TANGO a iimee-nntance dataset, their time-awar aware filtering techniques are different from each other. We are providing these results only for completeness.
Table 16: Possible range of start time instances in the datasets for link forecasting
YAGO1IK
Train start times range
Mmiin 2007]
Valid start times range
[2008. 2011]
Test start times range
12012, touae]
WIKIDATAI2K
Train start times range
Imiin 2009]
Valid start times range
[2010, 2012]
Test start times range
[2013 3, Imax]

---

Table 18: Results of TANGO model for link forecasting in YAGOI Ik and WIKIDATAI2K datasets.
on YAGO1 1A datasets for different filtering settings. For the definition of the raw, time- aware, and time-unaware filtering please refer TANGO model (Han et al., 202
YAGO1IK
Model
MRR
H@1
H
Raw
2,073
0.638
4.681
NeuSTIP vs TANGO comparison: Lastly, we obtained the results of NeuSTIP (Base) for link forecasting for raw (without filtering) settings that the results of TANGO and NeuSTIP can be compared directly for settings in Table 19.
Time Aware filter
2.089
0.665
4.681
Time Unaware filter
2.726
1.197
5.346
WIKDDATAI2K
Model
MRR
H@1
H@ 10
Raw
4.718
2.79
8.075
Time Aware filter
4.83
2.917
8.131
Time Unaware filter
6.39
4.735
8.963
Table 19: Comparison TANGO and NeuSTIP (hase)
model for link forecasting in YAGO1 and WIKI- DATAI2K 12k datasets for raw filtering setting.
Experimental Setup of TANGO model: We ob- tained results of link forecasting for the TANGO model on the YAGO1 1k/WIKIDATA 12k datasets. We further (a) consider the start time of all the quadruples, since the TANGO model is inherently designed for time instance datasets and cannot directly work with time-interval datasets as our model. (b) We utilized the publicly available code", and ran on the default set of hyperparameters af- ter pre-processing the datasets in exactly the same way as the model does for ICEWS05-15 15 dataset.
YAGOIIK
Model
MRR
H@1
H TO
NeuSTIP (Base)
5.289
0.0
12.978
TANGO
2.073
0.638
4.681
WIKIDATAI2K
Model
MRR
HΦl
NeuSTIP (Base)
9.455
0.0
23,557
TANGO
4.718
2.79
8.075
As can be observed, our model outperforms TANGO for link forecasting on both YAGOI Ik and WIKIDATAI2K on MRR and H@ 10.
Below, in Table 18, we provide the results of applying the TANGO model for link forecasting
htttps://g thub. com/ wwwm//eeporaaKTeee////ooo


Reviews:
[{"review_id": "f9q37VJH9d", "reviewer": "Reviewer_Yomj", "summary": "", "questions": "Question A: How can the model predict future links?\nQuestion B: what is the computational complexity of different parts of the model?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback on our paper. We address your comments below. \n\n**Q1:  The performance of the core of the method (NeuSTIP-base) is mainly lower than the corresponding competitors. The most performance gain is obtained by the ensembling approach. In this case, it is essential to know how will be results of other models after being ensembled with TimePlex. Besides, as the ensemble version of the model is the best model, it is important to compare it with other ensemble approaches.**\n\n**Q1, Part 1: The performance of the core method (NeuSTIP-base) is mainly lower than the corresponding competitors.**\n\n**Link prediction:** The best baseline models (no additional gadgets) that can be directly compared against our model in Table 1 are TILP-base in the Neuro-Symbolic Temporal models category and Timeplex (base) in the embedding-based models category. When compared to TILP-base, we observe that NeuSTIP (base) model is competitive on WIKIDATA12k dataset and is 5 pt and more than 4 pt better than TILP-base on MRR and Hits@10 on YAGO11k dataset.  When compared with Timeplex (Base), our base model is still performing better than Yago11k on all metrics, however, we agree with the reviewer that it performs slightly worse than Timeplex (base) on WIKIDATA12k dataset. We wish to point out that it has been noticed in the past literature that when directly compared, embedding based models outperform Neuro-Symbolic models because neuro-symbolic models sometimes suffer from low-coverage of rules which results in zero score for some triples in KG [3].\n\n**Time prediction:** Our base model outperforms Timeplex (base), the best baseline model, on both the datasets for time-interval prediction. In fact, for YAGO11k dataset, aeIOU score of our model is 8 point higher than Timeplex (base)’s score. \n\n**Part 2: The most performance gain is obtained by the ensembling approach. In this case, it is essential to know how will be results of other models after being ensembled with TimePlex. Besides, as the ensemble version of the model is the best model, it is important to compare it with other ensemble approaches.**\n\nThe reviewer has raised a valid point. We experimented (results in table below) with the state-of-the-art baseline model TILP by ensembling it with the embedding based approach Timeplex (the same we used for our ensembled approach) for different values of eta hyperparameter.  We observed a different trend compared to what we noted in NeuSTIP, i.e., the ensembling did not improve the performance of TILP (reported in table below). This counterintuitive observation points towards the sophisticated process  of ensembling.\n\n| ETA     |  10E-3   |  10E-5  |  10E-6   |  10E-8  |   0     | \n|---------|----------|---------|----------|---------|---------|\n|  H@1    |   10.09  |  11.16  |   11.26  |   11.80 |  16.81  |\n|  H@10   |    14.19 |  16.86  |   17.47  |   18.79 | 41.39   | \n|  MRR    |  11.69   |  13.24  |   13.50  |   14.25 | 25.32   | \n\n\n**Other Ensembling Approaches:** Right now we have ensembled NeuSTIP (base) with Timeplex linearly wherein score_ensemble = score(NeuSTIP) + eta*score(Timeplex) where eta is a learnable parameter. We could also explore more advanced ensemble approaches such as in Rivas-Barragan [4] that exploit the distribution of the score to ensemble the models together. We could also learn eta itself by exploiting the distribution of the score as in [5]. We would perform some experiments on different ensembles in the final draft of the paper.\n\n**Q2: It is not clear if the obtained rules are statistically rich and also how is it computationally expensive to use all walks.**\n\n**Statistical Richness of the rules:** We compute multiple metrics such as number of rules, average number of groundings per rule as a measure for statistical richness of the rules. The proposed methodology of performing all walks in NeuSTIP generated 8186 rules for YAGO11k and  31,807 rules for WIKIDATA12k dataset. The reason for obtaining a moderate number of rules while we consider all walks on TKGC is the sparsity of temporal KGCs as already discussed in the paper (section 4.1). Further, the average number of groundings per rule considering the train set is 17.87 for YAGO11k, and 885.21 for WIKIDATA12k, which is sufficiently high.\n\n**Computationally Expensive:** Total Run-time required by our model during the rule generation step for YAGO11k is 16.49s and for WIKIDATA12k is 278.68s on a single CPU of Linux server. The configuration of the Linux server where the rules were generated is Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz. \n\n**Q3: The model may be influenced by low-quality paths when the path score is set to 1 for link prediction. Not all paths are informative and helpful for prediction.**\n\nWe agree with the reviewer that assigning path score as 1 to all the path during link prediction does not distinguish the informative paths from the futile paths. In fact, motivated by RNNLogic [2], we considered assigning a real number score between 0 and 1 to each path. RNNLogic extends the RotatE model to paths by composing the embeddings of all the relations along the path. However,  finding a suitable ‘compositional’ embedding model in the presence of time-intervals is a research task in itself, as past literature on this is limited. \n\nAlso, please note that for a fixed rule, the entity reached by a higher number of paths would be ascribed a higher confidence. The cumulative score also includes the rule score (Equation 2).\n\n**Q4: It is not clear why only $r_{h}$ and $r_{1}$ are considered in equation 5, and why other relations in the path are not considered.**\n\nThe primary reason for learning the normal distribution between $r_{h}$ and only $r_{1}$ can be attributed to the rule language designed in Equation (1). As can be seen from the language, the target relation $r_{h}$ shares its subject only with the $r_{1}$ relation and hence considering $(r_{h}, r_{1})$ leads to learning the normal distribution between two relations that share a subject entity. For example let one grounding of head $r_{h}$ be $\\textit{(Joe Biden, Born in, USA, 1942)}$ and further $r_{1}$ has grounding $\\textit{(Joe Biden, Graduated From, University of Delaware, 1965)}$, then Gaussian distribution would consider the difference (1942-1965 = -23) as one of the point to compute the mean $\\mu_{r_{h},r_{1}}$ in the normal difference only because both the relations contain the information about a shared entity - Joe Biden. \n\n**Q5:  A time distribution given each relation is modeled by a Gaussian. However, a relation may present different frequencies in different time periods and the model may not capture various degrees of temporality with a single Gaussian distribution.**\n\nYes, we agree with the reviewer that a single Gaussian distribution may not capture the different frequencies in different time periods. We can, in practice, use mixture models to capture the various degrees of temporality, but that would add to the parameters and the model complexity. Exploring more ways to assign the scores to the candidate start and end time instances obtained after Allen predicate filtering is an avenue for further research.\n\n**Q6: The writing of some parts of the paper should be improved, e.g., Page 5, like 402, what is vol, or section 4.2.2 should be improved to be clearer.**\n\n We thank the reviewer for pointing out the typos in the paper. We will fix them in the final version of the paper. Vol in line 402 refers to the size of the time interval. $(T^{ev} ⋂ T^{pr})$ refers to overlap in time-interval [2]. $T^{ev}⋓T^{pr}$ is the smallest single contiguous interval (hull) containing all of $T^{ev}$ and $T^{pr}$ .E.g.,[1,2] ⋓ [30,40]=[1,40]. \n\nWe request the reviewer to help us understand the part of section 4.2.2 where the writing needs to be improved. It is quite clear to us. We will make the best effort to improve the understandability of section 4.2.2 in the final draft of the paper, if we understand the confusion. For better understanding of the reviewer, we explain the high-level idea of Section here: In path score computation for time interval prediction, we score the start time $t_{b}$ and end time $t_{e}$ independently. To compute the path score of start time, we first compute the mean and the variance of the difference of start time between $r_{h}$ and $r_{1}$ for the cases where they share the subject entity. Then we compute the normal distribution between difference of time $t_{2b}$ which is obtained by partial grounding of a given rule and one of the potential answer $t_{1b}$ for the start time and return this normal distribution as the score $\\phi(path)[t_{1b}]$.\n\n**Q7: There are several temporal KGEs e.g., [1] that can handle inductive settings in both time (future link prediction) and entity. Those methods might be compared to the proposed method in table 5.**\n\nIn our Table 5, we perform inductive link prediction, but without considering forecasting tasks. Model [1] seems to only work for the link forecasting case. So, it can’t be directly applied to our experimental setting of Table 5 – hence our model can not be directly compared to the TANGO model [1].\n\n**Question A: How can the model predict future links?** \n\nOn your suggestion, we performed link forecasting on NeuSTIP(base) whose experimental setup and the results are explained below:\n\n**Experimental Setup:**  We sorted YAGO11k/WIKIDATA12k datasets according to the start time of the time intervals, to make it suitable for forecasting ensuring the time in train and the test sets are disjoint as explained soon. \n\n**Running NeuSTIP (base):**\n\n1.) For the rule generation phase, we ensure that the start time of any interval occurring in the body of the rule is strictly less than that of the quadruple used in the head. Consequently, our model finds fewer rules.\n\n2.) Similarly during grounding the rules, while performing parameter learning, we maintain the above condition.\n\n3.) While running inference on the test set, this condition is automatically ensured due to the way in which the dataset is split. It is important to note that due to fewer rules and lesser groundings found in this split, our model's performance also reduces.\n\n**The time range of datasets for link forecasting:**\n\nWe list the possible values of the start time for train, validation and test phases in YAGO11k/WIKIDATA12k  datasets below:\n\nYAGO11k:\n\nTrain start times range : [T_min, 2007]\n\nValid start times range : [2008, 2011]\n\nTest start times range: [2012, T_max]\n\nWIKIDATA12k:\n\nTrain start times range : [T_min, 2009]\n\nValid start times range : [2010, 2012]\n\nTest start times range: [2013, T_max]\n\n**Dataset size:**\n\nThe size of dataset (number of quadruples) for YAGO11k after this split comes out to be (without adding inverses):  Train: 16,806, Valid: 1823, Test: 1880\n\nThe size of dataset (number of quadruples) for WIKIDATA12k after this split comes out to be (without adding inverses): Train: 33,368, Valid: 3649, Test: 3604\n\n**Results of NeuSTIP (base) for link forecasting:**\n\nWe obtained the final results of link forecasting by employing time aware filtering [2] that we have employed in all the experiments in our paper. The results are given below:\n\nYAGO11k:         MRR: 7.08, H@1: 3.38, H@10: 13.86\n\nWIKIDATA12K: MRR: 13.82, H@1: 7.45, H@10: 26.07\n\n**Link Forecasting for TANGO model[1]:**\n\nWe also obtained results of link forecasting for the TANGO model[1] on the YAGO11k/WIKIDATA12k datasets. Please note that the results obtained below are not directly comparable to the results of link forecasting to NeuSTIP results provided above because NeuSTIP is a time-interval dataset and TANGO is a time-instance dataset, their time-aware filtering techniques are different from each other. We are providing these results only for completeness. \n\n**Experimental Setup:** \n\n1.) Here, we consider the start time of all the quadruples, since the TANGO model is inherently designed for time instance datasets and cannot directly work with Time Interval datasets as our model.\n\n2.) We pick the code from https://github.com/TemporalKGTeam/TANGO, and run it on the default set of hyperparameters after pre-processing the datasets in exactly the same way as which the model does for ICEWS05-15 dataset.\n\n**Results for Link forecasting for TANGO Model:**\n\nBelow, we provide the results of applying the TANGO model for link forecasting on YAGO11k/WIKIDATA12k datasets for different filtering settings.\n\n YAGO11k (Raw):                               MRR: 2.073,  H@1: 0.638,  H@10: 4.681\n\n YAGO11k (Time Aware filter):           MRR: 2.089, H@1: 0.665, H@10: 4.681  \n\n YAGO11k (Time Unaware filter):       MRR: 2.726, H@1: 1.197, H@10: 5.346 \n\nWIKIDATA12k (Raw):                         MRR: 4.718, H@1: 2.79, H@10: 8.075\n\nWIKIDATA12k (Time Aware filter):     MRR: 4.83, H@1: 2.917, H@10: 8.131\n\nWIKIDATA12k (Time Unaware filter): MRR: 6.39, H@1: 4.735, H@10: 8.963\n\nFor the definition of the RAW, time aware and Time unaware filtering please refer to TANGO model [1]. \n\n**NeuSTIP vs TANGO comparison:** Lastly, we obtained the results of NeuSTIP for link forecasting(LF) for raw (without filtering) settings so that the results of TANGO and NeuSTIP can be compared directly for raw settings. \n\nNeuSTIP link forecasting results for WIKIDATA12k (Raw): MRR: 9.455, H@1: 0.0, H@10: 23.557\n\nTANGO link forecasting results for WIKIDATA12k (Raw):   MRR: 4.718, H@1: 2.79, H@10: 8.075\n\nNeuSTIP link forecasting results for YAGO11k (Raw):        MRR: 5.289, H@1: 0.0, H@10: 12.978\n\nTANGO link forecasting results for YAGO11k (Raw):      MRR: 2.073,  H@1: 0.638,  H@10: 4.681\n\nAs can be observed, our model outperforms TANGO for link forecasting on both YAGO11k and WIKIDATA12k on MRR and H@10. \n\n**Question B: What is the computational complexity of different parts of the model?**\n\nThis is the complexity analysis for various components in our model:\n\n 1.) **Rule Extraction:** Here, the time taken is essentially equal to the number of walks of length less than or equal to max rule length $L$ in the Knowledge Graph. The complexity of this is $N_{walk}(1) + N_{walk}(2) + … N_{walk}(L)$. Since at every entity $\\textit{E}$ occurring in the walk, we explore all its neighbors, this can be loosely bound to $G_{train}(\\delta)^{L}$, where $\\delta$ is the maximum degree of an entity.\n\n2.) **Finding rule groundings for Parameter Learning:**\n \nLet $N_{rule}$ be max number of rules for a tuple in $G_{train}$, $N_{path}$ be maximum number of paths following a given rule for a quadruple.\n\nTime Complexity for grounding for Link prediction comes out as $G_{train}N_{rule}N_{path}L$\n\nFor time prediction, the query is $(h,r,t,?)$. Essentially for every path, there is some corresponding starting time interval $E$ occurring in the first predicate of the rule body. Maximum such $E$ would be bounded by the number of tuples of the form $(h, r_{body1}, \\*, \\*)$, which is bounded by the degree of node $h$. Among the paths, we only select those ones which end at the entity $t$.\n\nFor each $E$, we use the first Allen predicate in the body of the rule to perform filtering, let $T_{startmax}$ and $T_{endmax}$ be the maximum number of possible candidates for any $E$ after Allen filtering. \n\nFor each such candidates, we use the distribution scores, which are computed only once for a fixed difference and a fixed pair of relations, (essentially this component would contribute $O(TR^2)$ time)\n\nOverall Time complexity is given as $G_{train}N_{rule}(N_{path}L + \\delta(T_{startmax} + T_{endmax}) ) + TR^2$\n\nTime taken to pre-compute pairwise difference distribution:\n\nFor a fixed entity $E$, all the facts would be considered in pairs with each other, which will contribute to some pair of $(r1,r2)$.\nHence the total complexity is given as $O(E(\\delta)^2)$\n\n3.) **Time taken for PCA score**:\n \nGiven a quadruple $(h,r,?,T)$ for link prediction, we obtain all the set of reached entities following all possible rules. Among these, the set of reached entities which are an answer of $(h,r,?,T)$ will contribute to the numerator of PCA, rest will contribute to the denominator of PCA.\nLet $N_{cand}$ be the maximum number of answer entities.\n\nWe essentially maintain a mask for each query of the size of $E/T$, which is set to 1 for the answer entities/instances, and this is used in computing the PCA score.\n\nLink complexity is given as $O(G_{train}(E + N_{rules}N_{cand}))$\n\nSimilarly for PCA of time, complexity comes out as:  $O(G_{train}(T + N_{rules} (T_{startmax} + T_{endmax}))$\n\n4.) **Time Taken for Inference** :\n\nWe obtain rule score from GRU, find path score through the groundings , let $N_{path}$ be the maximum number of paths for a given query in the background KB by following a rule.\n\nLink complexity is given as $G_{test}N_{rule}N_{path}L$\n\nTime complexity is given as $G_{test}N_{rule}(N_{path}L +\\delta*(T_{startmax} + T_{endmax}) )$\n\nFor link, time prediction, we get the scores using the groundings and aggregating on all rules (where rule scores are obtained by pre-computed PCA, passing through GRU).\nIn Link Prediction, we simply produce a ranked list of entities using the obtained scores. ($(O(E))$ time per query)\n\nFor Time Prediction, after obtaining the scores, we take $O(T^{2})$ time to predict the best interval per query.\n\nReferences:\n\n[1] Zhen Han,Zifeng Ding,Yunpu Ma,Yujia Gu, Volker Tresp, “Learning Neural Ordinary Equations for Forecasting Future Links on Temporal Knowledge Graphs”, EMNLP 2021\n\n[2] Prachi Jain, Sushant Rathi, Mausam, Soumen Chakrabarti, “Temporal Knowledge Base Completion: New Algorithms and Evaluation Protocols”, EMNLP 2020\n\n[3] Ananjan Nandi, Navdeep Kaur, Parag Singla, Mausam, “Simple Augmentations of Logical Rules for Neuro-Symbolic Knowledge Graph Completion”, ACL 2023 (Short Paper)\n\n[4] Daniel Rivas-Barragan and others, Ensembles of knowledge graph embedding models improve predictions for drug discovery, Briefings in Bioinformatics, Volume 23, Issue 6, November 2022\n\n[5] Yinquan Wang,Yao Chen, Zhe Zhang, Tian Wang, “A probabilistic ensemble approach for knowledge graph embedding”, Neurocomputing,volume 500, 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "--The model is founded on top of Allen algebra which is a well-defined system for modeling temporal relations.\n\n-- The proposed approach can utilize temporal rules and combine them with embedding to perform link prediction.\n\n-- The proposed method is designed by a chain of reasonable steps.", "reasons_to_reject": "-- The performance of the core of the method (NeuSTIP-base) is mainly lower than the corresponding competitors. The most performance gain is obtained by the ensembling approach. In this case, it is essential to know how will be results of other models after being ensembled with TimePlex. Besides,  as the ensemble version of the model is the best model, it is important to compare it with other ensemble approaches. \n\n-- It is not clear if the obtained rules are statistically rich and also how is it computationally expensive to use all walks.\n\n-- The model may be influenced by low-quality paths when the path score is set to 1 for link prediction. Not all paths are informative and helpful for prediction.\n\n-- It is not clear why only r_h and r_1 are considered in equation 5, and why other relations in the path are not considered.\n\n-- A time distribution given each relation is modeled by a Gaussian. However, a relation may present different frequencies in different time periods and the model may not capture various degrees of temporality with a single Gaussian distribution.\n\n-- The writing of some parts of the paper should be improved, e.g., Page 5, like 402, what is vol, or section 4.2.2 should be improved to be clearer. \n\n-- There are several temporal KGEs e.g., [1] that can handle inductive settings in both time (future link prediction) and entity. Those methods might be compared to the proposed method in table 5.", "questions_to_authors": "Question A: How can the model predict future links?\nQuestion B: what is the computational complexity of different parts of the model?", "paper_topic_and_main_contributions": "Link prediction and time prediction are among the popular solutions for temporal knowledge graph completion. Previous works are categorized as embedding-based, multi-hop reasoning-based, and rule-based approaches where most of them perform link prediction and a few of them address the problem of time prediction, especially prediction of time intervals. \nThis paper proposes a rule-based approach to perform both link prediction and time-interval prediction on temporal knowledge graphs. It first introduces Allen algebra as a formal system representing relations between time intervals.  To mine the rules, it first finds the ground paths from subject to object and then converts them to first-order rules. \nThe score of each candidate's answer (entity, start time, end time) is obtained based on two individual scores: rule score, and path score. The rule score itself is based on embedding with the gated recurrent unit and also statistical measure-based scores. Path score is additionally used for time interval prediction by modeling time gaps between head and body relations using Gaussian distribution. \nExperimental results on Wikidata12k and Yago11k show the proposed method outperforms other competitors in link and time interval prediction, especially by ensembling with KGE models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Han, Zhen, et al. \"Learning neural ordinary equations for forecasting future links on temporal knowledge graphs.\" Proceedings of the 2021 conference on empirical methods in natural language processing. 2021."}}, {"review_id": "l60FVY9TIy", "reviewer": "Reviewer_N5Ch", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our approach innovative, and experiments sound. We address your comments below\n\n**Q1:  I am not sure if there are any “previous ideas” which share the same idea with this paper.**\n\n\nTo the best of our knowledge, our work is the second work to use Allen algebra after TILP model [2] and the first one to use all the 13 Allen predicates and show their value in the context of time-interval prediction. The formula in Equation (2) is inspired from the score function in RNNLogic[1]. However, we extend the formula to temporal setting by performing both link and time interval prediction in temporal domains. \n\n**Q2: I just staring from the expression in this article to conduct an analysis of its grammar and readability.Formula 2 is quite challenging, especially when initially reading it, as the derivation details for rule embedding and path embedding are not yet understood. Consequently, the comprehension of the formula is not clear. Considering this, it might be beneficial to rearrange the order of formulas 3, 4, and 5.**\n\nWe will add the following sentences to explain the definition of rule score $(\\psi(L_{j})[c])$ and path score $(\\phi(path)[c])$ after the Equation (2) in the final draft of the paper: Equation (2) indicates that the candidate answer ‘c’ can be arrived at by firing multiple rules in the rule set $\\mathcal{L}$ and for one rule $L_{i} \\in \\mathcal{L}$, multiple paths can be followed in the graph $\\mathcal{G_{AW}}$ to arrive at candidate $c$. Further the importance of each rule in the final score is captured by rule score $(\\psi(L_{j})[c])$ and the significance of each path in a given rule is encoded by the path score $(\\phi(path)[c])$ parameter. \n\nAs you know, there are two styles of describing a complex system: the top-down approach and bottom-up approaches. Generally, first giving the big picture and then going into the details is better for readability. That's why we followed the top-down approach to the writing and ordered the current equations as (2),(3),(4),(5). We will try to improve the readability by further giving further information about $\\psi(L_{j})[c]$ and $\\phi(path)[c]$ by giving the forward references in the paragraph that follows Equation (2). \n\n**Q3: Btw, is this expression method in Part 6 acceptable? I am not entirely certain about its writing style.**\n\nIf you are pointing out to Q1 to Q6 in line 479 to 488 that we wrote the questions one after the other, then we did that to save space. If the paper gets accepted, we would use the extra space to write each question in a new line.\n\n**Q4: At the end of part 6, the article mentions the possibility of mixing multiple Allen relations, which may confuse the model and result in performance issues. Some Allen relations have a more significant impact on the results compared to others, but there is limited mention of experiments and validations in this regard, should further experimental evidence be supplemented.**\n\n In order to answer this question of the reviewer, we performed an additional experiment.\n\n**Experiment:** It is very difficult to study the impact of Allen relations present in a given rule in our current rule language, because multiple Allen relations are present in one rule. Further, Allen relations are interleaved between the KG relations to bind the time of two neighboring KG relations in the rule body. However, the first Allen relation $\\textit{a1}$ has its unique contribution to the score computation, especially in the case of Time interval prediction as discussed in the paper. Hence we  first divide Allen predicates into two categories: $\\textit{DET}$ refers to the the Allen predicates which fix either the start or the end time of the time interval for e.g.: inverse_finishes, meets, inverse_meets, starts, inverse_starts, equals and $\\textit{NONDET}$ refers to the Allen predicates where both ends are free, for example: before, after, overlaps, during, inverse_during, inverse_overlaps. Then we categorize the rules into two categories based upon whether the first Allen relation of the rule falls into $\\textit{DET}$ or $\\textit{NONDET}$ category while ignoring the rules in the other category and predicting scores in Equation (2) based on rules in a given category. The results for the link prediction and time interval prediction in two categories along will the results obtained the all rules considered ($\\textit{OVERALL}$) in Table 1 and 2 of the paper are presented below \n\n\n| Dataset     | Allen Relation|  MRR     | aeIOU    | \n|-------------|---------------|----------|----------|\n|             |     DET       |  18.62   |  22.55   |\n|  Yago11k    |    NONDET     |  17.50   |  16.62   |\n|             |    OVERALL    |  23.81   |  22.49   |\n|-------------|---------------|----------|----------|\n|             |     DET       |  23.45   |  27.31   |\n| WIKIDATA12k |    NONDET     |  25.30   |  21.97   |\n|             |    OVERALL    |  31.98   |  27.36   |\n|-------------|---------------|----------|----------|\n\nResults in $\\textit{DET}$ category are better than $\\textit{NONDET}$ for link prediction for YAGO11k and the results of $\\textit{NONDET}$ category are better than $\\textit{DET}$ category for link prediction for WIKIDATA12k dataset.  For time interval prediction, $\\textit{DET}$ alone can help us in achieving similar to our $\\textit{OVERALL}$ model performance for time prediction. Further, for link prediction, using only one of them reduces the performance as compared to the $\\textit{OVERALL}$ mode. This is because the head Allen relation does not as directly influence the entity reached by the rule during link prediction as compared to the time prediction. Impact on link prediction seems to be governed by the total number of rules, as more rules help in grounding the gold entity because paths are available to ground them. Because of this reason when the rules are divided into $\\textit{DET}$ and $\\textit{NONDET}$ categories, lesser numbers of rules are available for link prediction which results in drop of performance. In summary, using both sets of relations finds competitive performance in all settings.\n\n**Q5: There is another question: The best results are obtained by the combined model of NeuSTIP and Tcomplex-like embedding models. How can we determine if the experimental performance is mainly contributed by NeuSTIP? Alternatively, when combining the two models, how do they influence each other when presented with a specific question?**\n\nWe can see that our NeuSTIP base model by itself gives competitive performance to the best embedding based model (TimePlex) as demonstrated in Tables (1) and (2) in the paper. If we perform ablation on any one model from the ensembled model, the performance gets worse.\n\nFurthermore, in order to answer the above question, we considered the ranks of NeuSTIP, Timeplex and an ensemble (NeuSTIP w/ KGE) individually for each test quadruple obtained during link prediction and performed a comparative study to understand how the KGE models contribute to the performance of NeuSTIP.\n\nOur major conclusion after analysis is that NeuSTIP mainly fails for the cases when no rule gets fired for a given quadruple for the gold entity resulting in zero score in Equation (2) in paper. For instance, we found \n796 such quadruples (out of total 4102 test instances) in YAGO11k and 609 quadruples (out of 8124 quadruples in test data) in WIKIDATA12k, where our model generated zero score. For such instances, the average rank of our model was 5327 for YAGO11k, and  6307 for  WIKIDATA12k while the average rank of Timeplex for such cases was 1654.51 for YAGO11k and 1420.07 for WIKIDATA12k. Therefore our model relies on ensembling for the cases when it obtains no groundings for any rules learnt by NeuSTIP for a given quadruple.\n\nHowever, for the cases when the model obtains the groundings, it performs better than the corresponding Timeplex and Ensemble (NeuSTIP w/ KGE) model. For instance, when NeuSTIP obtained its grounding, its average rank is 51.1, while the average rank of Timeplex for such cases is 357.54, and is 169.70 for ensemble for YAGO11k datasets. Similarly, the average rank of NeuSTIP for cases when groundings were found is 38.09 while the average rank of Timeplex for such cases is 103.27, and ensemble for such cases is 88.65 for WIKIDATA12k. Hence we conclude that NeuSTIP relies on ensembling when it finds no grounding but outperforms the ensembling and the Timeplex model when it obtains the groundings for the candidate entity in link prediction for the rules learnt by NeuSTIP. \n\n\n**Q6: Can the explanation of the model's human interpretability be made clearer?**\n\nWith human interpretability, we mean that (i) rules are human readable, (ii) we always know how (or why) we arrive at a certain answer. For example, consider the grounding of the rule provided in the Section 6: \n\nisAffiliatedTo(Franz_Dahlem, Communist_Party_of_Germany, [1920,1946]) ←During([1920,1946],[1899,1974])∧ isMarriedTo(Franz_Dahlem, Kathe_Dahlem\n,[1899,1974])∧Contains([1899,1974],[1920,1946])∧isAffiliatedTo(Kathe_Dahlem, Communist_Party_of_Germany, [1920,1946])\n\n1. In order to understand this grounding, we request the reviewer to first consider the head (isAffiliatedTo) and the first two relations (during and isMarriedTo) in the rule body. By considering this partial rule grounding, we can conclude that person/entity Franz_Dahlem is affiliated to party/entity Communist_Party_of_Germany during the time when he is married to person/entity Kathe_Dahlem. Here the affiliation time interval ([1920,1946]) is subsumed by the married time interval which means the entire time he was affiliated to the party, he was married.\n2. Now consider the last two relations in the body - contains and isAffiliatedTo. Also note that the Allen relations ‘contains’ and ‘during’ are inverse of each other. This partial rule grounding implies that the entire time ‘Kathe_Dahlem’ was affiliated to Communist_Party_of_Germany, she was in her marriage time interval (the marriage time is being provided by the first argument of contains).\n\nCombining these reasonings we can conclude that Franz_Dahlem is affiliated to Communist_Party_of_Germany (the rule head) at a certain time because he was married to a woman (point 1) who was also affiliated to the same party (point 2) at around the same time interval.         \n\nReferences: \n\n[1]  Meng Qu, Junkun Chen, Louis-Pascal A. C. Xhonneux, Yoshua Bengio, and Jian Tang. RNNLogic:  Learning Logic Rules for Reasoning on Knowledge Graphs. In ICLR, 2021.\n\n[2] Siheng Xiong, Yuan Yang, Faramarz Fekri, and James Clayton Kerce. TILP: Differentiable Learning of Temporal Logical Rules on Knowledge Graphs. In ICLR. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea presented in this article can be considered quite innovative. It introduces a neural-symbolic TKGC method that enhances interpretability, building upon the traditional KG embedding-based KGC approach. In comparison to other methods that utilize learning temporal logic rules and attention mechanisms, the proposed NeuSTIP model can handle two types of problems: link prediction and time interval prediction. .Also this paper discusses a novel approach for ranking candidate answers, proposing two scoring functions: one based on rule embedding and relevant rule head embedding, where the rule score is computed using the inner product. Additionally, the path score is generated using a Gaussian distribution. The combination of these scoring functions is utilized to filter the results. Furthermore, to incorporate temporal features, the final scoring function incorporates information from models such as Tcomplex embeddings. These innovative aspects represent relative significant advancements in the field. It’s might be a good paper with some novel ideas.", "reasons_to_reject": "I am not sure if there is any “previous ideas” which share same idea with this paper.So, I just staring from the expression in this article to conduct and analysis of it’s grammar and readability.Formula 2 is quite challenging, especially when initially reading it, as the derivation details for rule embedding and path embedding are not yet understood. Consequently, the comprehension of the formula is not clear. \n\nConsidering this, it might be beneficial to rearrange the order of formulas 3, 4, and 5.\" Btw, is this expression method in Part 6 acceptable? I am not entirely certain about its writing style. At the end of part 6, the article mentions the possibility of mixing multiple allen relations, which may confuse the model and result in performance issues. Some Allen relations have a more significant impact on the results compared to others, but there is limited mention of experiments and validations in this regard, should further experimental evidence be supplemented.\n\nThere is another question: The best results are obtained by the combined model of NeuSTIP and Tcomplex-like embedding models. How can we determine if the experimental performance is mainly contributed by NeuSTIP? Alternatively, when combining the two models, how do they influence each other when presented with a specific question?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses work on exploring methods for temporal KGC (temporal knowledge graph completion). The paper draws the motivation for the work that most knowledge graph completion work make use of Graph algorithm like GNN, language model based solutions and knowledge-graph embedding based solutions. These methods, including the neural-symbolic approaches,  posses characteristics of unpredictable time intervals and poor human interpretability. The author/s then proposes a method that  use an intuitive rule language that integrates the complete set of allen algebra relation and KG relations, with the ability of computing confidence of temporal rules that combines both symbolic and embedding information , the author proposes a brand new NS-TKGC model which can perform time interval prediction. It also provide novel scoring function which combine rule score, path score and time information, it’s not bad. Last, the 2 benchmarks they provided, WIKIDATA12k and YAGO11k are challenging, which is good.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1.Can the explanation of the model's human interpretability be made clearer?\n\n2.Can the formulation of Formula 2 be adjusted in relation to the order of Formulas 3, 4, and 5? Before understanding Formulas 3, 4, and 5, the comprehension of the content in Formula 2 is not very clear. \n\n3.In the beginning of Part 6, the description of each component of Q should employ a more refined writing style to clearly convey the issues that need to be addressed. \n\n4.Human interpretability of logical rules can be expressed with a more structured approach, avoiding the mixing and confusion between logical rules and textual expressions. Enhancing readability will be beneficial in this context.\n\n5.Some small typos, like past tense"}}, {"review_id": "axFaSV3XBG", "reviewer": "Reviewer_juCQ", "summary": "", "questions": "In the end, the authors mention that they will explore combinations of NeuSTIP with box embeddings and GNN. I wonder how these can benefit each others. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We answer your comments below:\n\n**Q1: The paper lacks a figure illustrating the overall idea of the proposed method. I can only get the overall idea after I finish the reading of the whole methodology section. It would be significantly improve the readability and presentation of the paper if the authors could add one \"bigmap\" figure in the paper.**\n\nYour suggestion about “bigmap” figure is excellent because it would improve the readability of the paper. We will add it to the paper. As per conference rules, we cannot provide a figure in the rebuttal but provide a high level idea of how the figure would look like.\n\nOur figure will consist of three vertical blocks. In the first block (block 1) NeuSTIP would learn the temporal logical rules from a given TKG, in the second block (block 2) it will ground the rules for a given query and also compute the rule score there. In the third block (block 3), it would compute the path score for both link and time-interval prediction and compute the final score in Equation 2.\n\nSpecifically, in the first block the model would take the TKGC as input and produce the rules in Equation (1). These rules would be fed to the second vertical block in the figure. This block would also take the query as input and would ground the rules by taking rules, query and TKGC as input. From the groundings, one link (arrow) would go to block three where the path score for each path would be computed in two sub paths, one for the link prediction and one for the time prediction. The other link (arrow) from the groundings would stay in block 2, where rule score would be computed by two units GRU and PCA score. At the bottom of block 3, the final rule score and the path score will be fed to a unit that represents the score in Equation (2) and this will be set as an output of Block 3. \n\n**Q2: In the end, the authors mention that they will explore combinations of NeuSTIP with box embeddings and GNN. I wonder how these can benefit each other.**\n\nFrom this we mean that, in addition to ensembling NeuSTIP with Timeplex which is an embedding based model, we could also ensemble our model with other type of models such as Temporal Box Embedding model such as BoxTE[1], although this model is time-instance based model and we would need to modify the model for time interval prediction. The major benefit of this model would be that while Timeplex considers entities and relations as points in multi-dimensional hyper-space, box embeddings based models consider entities or relations as boxes in hyper-space and can hence encapsulate more information inside them. It would be interesting to see how this affects the performance of the model.\n\nWe can also ensemble our NeuSTIP model with GNN based models  such as xERTE model [2]. Our proposed model benefits from Timeplex the most when a given quadruple is not covered by any rule in the proposed model. In those cases, Timeplex helps in performance improvement but it is still not using the path information because it's fundamentally an embedding based model. We could benefit from graph based models in such cases with the additional advantages being that we would  exploit paths to compute scores in such cases.\n\nReferences: \n\n1.) Johannes Messner, Ralph Abboud,Ismail Ilkan Ceylan, “Temporal Knowledge Graph Completion using Box Embeddings”, AAAI 2022.\n\n2.) Zhen Han, Pen Chen, Yunpu Ma, Volker Tresp, “Explainable Subgraph Reasoning for Forecasting on Temporal Knowledge Graphs”, ICLR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes the first neural-symbolic temporal KG completion (TKGC) approach named NeuSTIP that performs  both link and time-interval prediction. Existing neural-symbolic methods either focus on time-instant KGs or only do link prediction. \n2.  The proposed method first considers enforcing temporal consistency between neighboring predicates in the rule body, by integrating the complete set of Allen algebra relations and KG relations. In my opinion, such modeling is novel in TKGC. \n3. The authors also propose a hybrid of NeuSTIP and a KG-embedding model TimePlex (Jain et al., 2020). Such hybrid model obtains best results on both prediction tasks.\n4. In general, neural-symbolic TKGE seems to be a good choice and has many advantages, e.g., it is human-interpretable, and works well with limited data. ", "reasons_to_reject": "The paper lacks a figure illustrating the overall idea of the proposed method. I can only get the overall idea after I finish the reading of the whole methdology section. It would be significantly improve the readability and presentation of the paper if the authors could add one \"bigmap\" figure in the paper.\n", "questions_to_authors": "In the end, the authors mention that they will explore combinations of NeuSTIP with box embeddings and GNN. I wonder how these can benefit each others. ", "paper_topic_and_main_contributions": "This paper works on temporal knowledge graph completion, i.e., predicting missing links or time periods based on existing factual knowledge.  The main novelty of the proposed method is that it uses an intuitive rule language to integrate the Allen algebra relations and KG relations, which aims at enforcing temporal consistency between neighboring predicates in the rule body. The advantages of the method is that it can perform link and time interval predictions in the neuro-symbolic setting, and the prediction is more human-interpretable. Besides, the method also works well with limited data. Empirical evaluations verify these advantages. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Missing period in line 658"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agreed that this paper presents an innovative approach to temporal knowledge graph completion. The method's combination of Allen intervals, neurosymbolic scoring, interpretable rule mining were recognized as strong contributions. Sufficient detail was provided to be able to reproduce the results. Several reviewers pointed out issues with presentation, writing, and clarity that could be improved. While the absolute performance of the proposed method for KGC in isolation does not represent a new SotA result, time interval prediction does improve over prior models and the authors propose an ensembling approach to improve KGC quality. During rebuttal discussion, the authors provided ancillary results that helped to quantify improvements and provide details to support their conclusions. The ideas in this paper and its goal of predicting both time intervals and tail entities are likely to generate significant interest for a small community interested in TKG modeling.
Decision: Accept-Main