TR-Rules: Rule-based Model for Link Forecasting on Temporal Knowledge Graph Considering Temporal Redundancy
Authors: Ningyuan Li
Haihong E
Shi Li
Mingzhi Sun
Tianyu Yao
Meina Song
Yong Wang
Haoran Luo
Venue: EMNLP 2023 Findings (2023)
Keywords: Knowledge Graph
Temporal Knowledge Graph
Link forecasting
Temporal Rules

Abstract: Temporal knowledge graph (TKG) has been proved to be an effective way for modeling dynamic facts in real world. Many efforts have been devoted into predicting future events i.e. extrapolation, on TKGs. Recently, rule-based knowledge graph completion methods which are considered to be more interpretable than embedding-based methods, have been transferred to temporal knowledge graph extrapolation. However, rule-based models suffer from temporal redundancy when leveraged under dynamic settings, which results in inaccurate rule confidence calculation. In this paper, we define the problem of temporal redundancy and propose TR-Rules which solves the temporal redundancy issues through a simple but effective strategy. Besides, to capture more information lurking in TKGs, apart from cyclic rules, TR-Rules also mines and properly leverages acyclic rules, which has not been explored by existing models. Experimental results on three benchmarks show that TR-Rules achieves state-of-the-art performance. Ablation study shows the impact of temporal redundancy and demonstrates the performance of acyclic rules is much more promising due to its higher sensitivity to the number of sampled walks during learning stage.

TR-Rules:
Rule-based Model for Link Forecasting on Temporal Knowledge Graph Considering Temporal Redundancy
Ningyuan Li', Haihong Li Shi?, Mingzhi Sun', Tianyu Yao!,
Meina Song', Yong Wang', Haoran Luo!
Beijing University of Posts and Telecommunications:
National Computer Network Emergency Response Technical Team/Coordination
Center of China; Lianyann Guorong (Beijing) Technology Co., Ltd.
jason.ningguan..li ehaihong. sunmingzhi, yaotianyu, mnsong, luohaoran bupt.edu.cn shili @ cert.org.en, Lungyong@oceanumm
Abstract
Cyelic Rules
Temporal knowledge graph (TKG) has been proved be an effective way for modeling dy- namic facts in real world. Many efforts have been devoted into predicting future events ie. extrapolation, TKGs. Recently, rule-based knowledge graph completion methods which are considered to more interpretable than embedding- based methods, have been trans- ferred to temporal knowledge graph extrapola- tion. However, rule-based models suffer from temporal redundancy when leveraged under dy- namic settings, which results in inaccurate rule confidence calculation. In this paper, we de- fine the problem of temporal redundancy and propose TR-Rules which solves the temporal redundancy issues through simple but effec- tive strategy. Besides, to capture more informa- tion lurking TKGs, apart from cyclic rules, TR-Rut RR-R also mines and properly leverages acyclic rules, which has not been explored by existing models. Experimental results on three benchmarks show that TR-Rules achieves state- of-the-art performance. Ablation study shows the impact of temporal redundancy and demon- strates the performance of acyclic rules is much more promising due to its higher sensitivity to the number of sampled walks during learning stage.
(arond to regutiare, ra
Oiteadd comporate T4)
(Make a visit, 73)
(Convutt, T5)
(Make statement, TI)
(Muke a visil, T2)
(Elected as president, Tm
(Bost visit, T2)
Acyclic Rules
Figure I: An example subgraph of Temporal knowledge graph involving political facts and there are cyclic and acyelic rules existing in it which provide crucial infor- mation for link forecasting.
dependent on time, i.e., facts in KGs may not stand perpetually. Thus, TKGs are introduced resolve this limitation, where each fact is represented by a quadruple (s,r,o,t) with timestamp incorporated. Link forecasting or extrapolation on TKG which aims at predicting possible links at future times- tamps, has been one of the main tasks being stud- ied. Recently, embedding-baase models like RE- NET(Jin al., 2019), CyGNet(Zhu al., 2021). have been proposed and perform well for extrapola- tion on TKGs. However, compared with rule-based models, embedding- based models are believed lacking interpretabilityy which leads to limitations on their downstream applications. TLogic(Liu et al., 2022) is the first model which leverages rule- based methods TKG link forecasting and gains competitive results well as more explainable pre- dicting process since it generates human readable rules. It mines cyclic rules by performing temporal random walk on TKGs first, and then extracting and generalizing walks into symbolic rules.
1 Introduction
Knowledge Graphs (KGs) store various real-world facts in the form of (s.r,0) in which s and repre- sent subject and object real-world entities while denotes relation between and o. KGs are critical in many downstream applications such as question answering (Hao et al., 2017), recommender sys- tems (Hildebrandt et al., 2019) and information retrieval (Liu et al., 2018). Traditional knowledge graph can be viewed as a static knowledge base, however, plausibility of most real-world facts are
"Corresponding Author
'Our cnde available at httpps//ggithh com/ SsconLee-22/TR-- RR-Rules

---

Nevertheless, existing models only mine cyclic rules in TKGs and ignore acyclic rules which, also contain important information for link prediction. Besides, previous works do not notice that under temporal settings, (s,r,o) which has to be unique in static KGs, might appears repeatedly with different timestamps. And this temporal property can cause the model to obtain inappropriate confidence of rules through conventional confidence calculation algorithm, which further leads iccrrect predic- tions. We call this issue Temporal Redundancy.
tively and experimental results prove the ef- fectiveness of TR-Reees in solving temporal redundancy and demonstrate that the perfor- mance of acyclic rules more promising due to its higher sensitivity to the number of sam-
pled walks.
Related Work
2.1 TKG Extrapolation
Most of the existing TKG extrapolation models are embedding- models which aim at learn- ing representations of entities, relations as well as timestamps in vector space and then obtain scores of facts through proposed score functions to mea- sure the plausibility of those facts being true. RE- NET (Jin et al., 2019) employs RNN-based encoder and EGCN-based encoder capture sequential and structural information of facts for future events prediction. CyGNet (Zhu et al., 2021) introduces eopy-generation mechanism to learn more infor- mation from repetitive facts in history. HIP (He et al., 2021) proposes three score functions to pass information from temporal, structural and repet- itive patterns. CluSTeR (Li al., 2021) adopts reinforcement learning to search clues from his- tory for link forecasting. CENET (Xu al., 2022) focuses more on the non-historical facts and uses contrastive learning to distinguish whether histori- cal or non-historical information is more important for events prediction.
In this paper, we first define Temporal Redun- dancy and give an intuitive illustration on why Tem- poral Redundancy causes inappropriate confidence calculation. Then we propose TR- TR-Rules, rule- based TKG link forecasting model which resolves Temporal Redundancy through simple but effec- tive aggregation strategy. TR-Rules aggregates those matched rule bodies before any new rule heads appear as one body during the rule support counting process. Unlike previous works merely mine cyclic rules on TKG, by performing tempo- ral random walk on TKG, TR-Rules also mines acyclic rules which is crucial for link prediction but unexplored by existing models. Experimental results show that, without considering temporal re- dundancy, simply using acyclic rules together with cyclic rules gives worse performance, which the probable reason why acyclic rules have not been involved and studied by existing models. Ablation study also demonstrates that acyclic rules are more sensitive to the number of random walks sampled for rule mining, which consequently results in more promising performance. Our contributions are as follows:
Apart from embedding- bassdd models, XERTE (Han al., 2021) and TLogic (Liu al., 2022) provide more explainable predictions. XÉRTE per- forms extraction on subgraphs around the queries to model structural dependencies and temporal dy- namics. TLogic mines temporal rules via adopt- ing temporal random walks on TKGs, which are applied to forecast plausible events at future times- tamps afterwards.
We discover and define the problem of tempo- ral redundancy which results in inappropriate rule confidence calculation.
We propose TR-Rules, a rule-based temporal knowledge graph extrapolation model which resolves temporal redundancy with simple algorithm and, to our best knowledge, is the first model that mines acyclic rules on tempo- ral knowledge graphs.
2.2 Rule-based KG Completion
AMIE (Galárraga et al., 2013) proposes one of the earlist rule mining systems that learns closed rules on knowledge bases. AnyBURL (Meilicke et al., 2019) is a random- random-walk- baak- model which transforms sampled walks from KG into general rules and then applies obtained rules to predict missing facts. SAFRAN (Ot al., 2021) focuses on the effect of rules redundancy on aggregation especially noisy-or aggregation, and proposes an approach to cluster rules before application.
We evaluate TR-Rules on three benchmarks and the results show that TR-Rules achieves state-of-the-art performance.
We also study the performance of acyclic rules and cyclic rules of different length respec-

---

3 Preliminaries and Notations
For better illustration, we give examples of cyclic and acyclic rules, both of which are rea- sonable and critical in link forecasting. For facts of (Ameriaca, Intent to negotiate, China, T1) and ( (America, Make a visit, China, T2) where Ti < T2, we can generalize them into cyclic rules of length 1:
3.1 TKG Extrapolation
In this paper, we use & R. and T to represent all the entities, relations and timestamps respectively, While [8] and |R| denote the number of entities and relations. TKG be considered as sequence of graphs i.e. G (GI...Gn) in which G; [(s, r, o, 14)] where 5, &, с and 4 E T. TKG extrapolation is to answer a query (s, 1s,7,?,t, tq or (?,r, (4) based on the previously observed facts {G < t. Normally, the model generates a ranked list of candidate entities according to their plausibility.
(X, Make visit, Y, Ti.))
(X, Intent to negotiate, Y, Tb)
From facts (Jason, Major, Education, Ti) and (Jason, Job, Teacher, 72), a reasonable tem- poral acyclic rule can be generalized which is:
(X, Job, Teacher, Th) <
3.2 Random Walk on TKG
(X, Major, Education, Th)
Following the definition given by TLogic, non- increasing random walk TKG with length e N can be defined as:
3.4 Confidence Estimation
Generally, temporal rules may not always be tau- tology. Thus, need to estimate the confidence of each temporal rule before application. Existing rule-based TKG extrapolation model TLogic lever- ages the standard confidence which is widely used in static KG completion models and take times- tamps into consideration. Take a cyclic rule R: (E, The E1 Ti+1) + A, Tbi, E,41, T;) for example. The body support of R can be defined the number of rule body instances, i.c, the number of sequences ((e), rb1, 62, l),,.., (et, Tbt, et+1, 44) where (cs, Tbi, Ci+1, f4) 9 and t; S f++1 for C [0, 1]. And the rule support of R is defined as the number of body instances whose rule head stands at future timestamp f4r i.e., (E1, Th et41, 4111) E G. The confidence is calcu- lated as
(lecaa, ea, (et, Te Fe, (eu with 4 1t.....6
The non-increasing restriction ensures random walks only go back in time forming chains comply- ing with temporal causality.
3.3 Temporal Rules
Here, we give the definition of temporal cyclic and acyclic rules. Let E; and T; be variables and ri, e; represent fixed constants (fixed constant means that it has to be specific entity or relation). A temporal cyclic rule can be represented as:
(Ei, En 1, 1at1) (Es i rbi, E:),
rule_sup
con - body_sup +
(1)
where 1 > Tn Ti.
And temporal acyclic rule can be defined as:
where is hyper-paramedt for smoothing and weight for rules of different types and differ- ent lengths. This confidence estimation algorithm works for static KG completion but when it comes to temporal settings, it suffers from Temporal Re- dundancy which will be defined and discussed in the next section.
(Ea The eh, Th + (Ea, Tb. eb, Tb)
where T3 > Tb. en and e, are fixed entities. It is no- table that TR-Rules, just as the definition given above, we confine the length of acyclic rules to be 1. The left side of a rule is called rule head and the right side of rule is called rule body. A tempo- ral rule in the above forms indicates that when the rule body stands( the variable constraints and temporal restrictions), the rule head at future timestamp TRI3 will be true. When applied, each variable in rule head or rule body can be instanti- ated with specific entities or relations.
4 Proposed Method
In this section, we propose TR-Rules, rule-based model for TKG extrapolation, which adopts both cyclic and acyelic rules mining and resolves tempo- ral redundancy. First, we give the definition Tem- poral Redundancy and illustrate how TR-Rules

---

Rule: (X, Make visit, (X, Intend negotiate,
Temporal Redundanccy
Rule_sup +1
Rule, +1
No valiid body instance window3
Rule sup
Rule sup +1 Valid body Instunce
Rule sup Valid body Instanee
Valid body Instance
Mikee xisit
Inlend
Male
Intend
Mike
orgotlate
negreiato
release persons
Regediate
change policy
visit
T6
Windon Confilence
Standard Confidencu in green
Figure 2: Illustration of the issue of Temporal Redundancy and window confidence.
resolves it. Then we introduce the rule learning module of TR-Rules where specifically we show the acycle rules mining algorithm of TR-Rules. Fi- nally, the rule application module of TR-Rules is introduced to show how we combine cyclic and acyclic rules together to predict future events.
tion of standard confidence, the result should be 1. Because there are body instances (The US encoy, Incoy, to negotiate, China) at (Ti Ty. TT). which means the body support is and for body instance at Tl, there are head instances Intent to negotiate holding true at (T2, Ty) making it counted
rule support, for body instances at TT), head instance T9 makes them counted as rule support. Thus the rule support is 3 either and ignoring the pa- rameters for smoothing, the confidence of this rule
4.1 Temporal Redundancy
As mentioned above, standard confidence counts the number of body instances whose rule heads stand at future timestamps. It works well for static KGs, however, it is notable that in TKGs triplet can appear more than once, which means that is possible the same body instance can be counted multiple times for rule support only it holds true at different timestamps before the timestamp where its corresponding rule head stands. Espe- cially, when there are multiple body instances hold- ing true between any two temporally adjacent rule head instances, call this phenomenon Temporal Redundancy.
However, in fact, the head instance (The US envoy, Make a visit, China) stand- ing at T7 might be the result of the two intentions at (T3, Ta) together. In other word, only sin- gle intention may not necessarily results in visit, which intuitively contradicts to the confidence be- ing 1. Since when the confidence of rule being 1, means that the rule head will surely hold true at future timestamps as long as there exists valid body instances. From this point of view, standard confidence calculation is not suitable for temporal settings.
In figure 2, Temporal Redundancy takes place between T2 and T7. When Temporal Redun- dancy happens, the confidence of rules is over estimated, because many interactions between facts have strongly connections with the ac- cumulation of the causes. In figure 2, we take a cyclic rule: Make visit,Y) + (X, Intent to negotiate, Y) which is mined from ICEWS14 and some relevant facts for example. In this case, when estimating the confidence of this rule, suppose there is no other relevant facts, according to the defini-
In TR-Rules, we solve this issue by proposing simple but effective window based algorithm for confidence estimation called window confidence. Instead of focusing counting the number of body instances whose rule heads stand in future the rule support, window confidence calculates the rule support based on the rule head instances. During the confidence estimation of each rule, we sort the timestamps all the rule head instances and

---

divide the timeline into windows by those times- tamps. Each window starts timestamp 0 or a timestamp of rule head instances mentioned above and ends at the timestamp next to its start point or the last timestamp. Like in figure 2, we obtain 3 windows according to the rule head instances. Starting from the first window, if there exists at least one body instance, then we add 1 to the rule support and shift to the next window. We can see that by using window confidence, the confidence of the example rule above is y since only window I and window 2 involve valid body instances. What window confidence intends to do is that aggregate the temporal redundancy inside windows with a designed function. In Tu-Rules, the function we utilize just aggregate all body instances inside window as one. From this point of view, the stan- dard confidence applied in TLogic trivial ver- sion of window confidence which associates with no aggregation function.
appears multiple times in rule, then it has be replaced by the same variable.
In TR-Rulcs, when sampling the new edge from current node, we use the exponentially weighted transition distribution which can be defined as:
exp(4 Le) P(l:te, EmpETe exp(lv - (c)
where C,te represent the current edge and the timestamp of current edge and To denotes a set of possible timestamps associated with available edges in the next step:
(, E G and < te) step 1, (el(obje: r, r, E G and S te) step n, (4(oobje, r,e,)) E G and t S (e) otherwise.
(2)
where obje represents the object entity of the cur- rent edge, e E and G denotes the whole TKG. This distribution gives higher probability when t is close to le which makes the random walker more likely to select closer edges during sampling, This is based on the intuition that closer facts might contribute more to predictions compared with facts that happen at further timestamps.
4.2 Rule Learning
In this section we introduce the learning procedures of cyclic and acyclic rules respectively. Our cyclic rules mining algorithm is similar to the rule learn- ing stage in TLogic. We first randomly sample a quadruples (e1, Th, En+1, tx+1) which is consid- ered to be the rule head. Then we perform random walks traversing back in time and starting from the entity Cu+1. For the first step, to satisfy the temporal restriction mentioned in section 3.3, the random walker is confined to sample edges con- necting to the current node at timestamps tn where fn+1 > t.- Then, in the following steps, the walker samples adjacent edges whose timestamps & sat- isfies 4 < fi+1 where 2 < i < 1 iteratively. When it comes to the last step. the walker is re- stricted to sample valid edge that connects e so that all the walks together with the rule head form cyclic path. If there is no available edges during any steps, the current random walk will stop and we will start a new one by randomly sampling a rule head. Since we traverse back in time during the learning procedure, when generating tempo- ral rules, we need to rebuild the path in reverse order by substituting relations in edges with their inverse and exchanging the objects and subjects which gives: (e1, rf. e2, (1)...(en, r'u en-1, 1n+ Finally, we replace the entities and timestamps with variables and the generated rules with length n are specific to relation The Specifically, if an entity
In TulRs, we also propose a learning algo- rithm for acyclic rules, Similarly, we first sample a random quadruples (Ch, Th, Gt ta)) to be the rule head. Then, unlike the cyclic mining procedure, starting from Cb, the random walkers are made to select edge adjacent to e₁ which is (ch, Tb Gj (444) where < th. Then replace en, with variable and set ei, to be fixed. Consequently, an acyclic rule specific to rı is obtained.
When estimating confidence, we first sample a fixed number of instanced paths and leverage the window confidence mentioned above. The output of the rule learning module is a set of rules with their corresponding confidence.
4.3 Rule Application
Given a query Q (s, ?,t),, we first find the rules whose head relations match with r in Q and sort them in descending order according to their confi- dence. Then, for each rule, we traverse the TKGs to find matching body instances which satisfy the temporal constraints and the object entity in each rule head is considered to be one of the answer candidates if there exist valid body instances. For a candidate answer c generated by rule R. we lever- age the score function f proposed in TLogic to

---

measure the plausibility of (s,7; C, being true:
and test according to the timestamps in ascending order, which do not overlap with each other. The statistics of these three datasets are listed Table
[RR, c) aa- ca-cons(1 ( (1-a) -tt)))extpx(t((B11111 (1 c)))
(3)
(R,, c) denotes all the body instances of rule Rt that leads to can- didate and t1 (6) means the earliest timestamp in the body instance b. If there are multiple body in- stances, then 1 returns the one closest to t. This score function considers not only the rule confi- dence but also the time difference. Obviously, the confidence part enables candidates derived by rules with higher confidence to receive higher scores. And the time difference part assigns higher scores to candidates generated by close body instances, which is based on the phenomenon that edges in rules incline hold true when the time difference is low. Normally, one candidate can be generated by multiple rules, we use Noisy-OR to aggregate scores of e obtained by different rules. The final scores of c is calculated as:
where a, X are
Metries and Implementation Details The met- rics use evaluate TR-Rules are: mean reciprocal rank (MRR) and Hits @ 1/3/10. For N testing queries, the MRR is computed as MRR = N Ei rank, and Hits is calculated as Hits@K
I
I(rank: K), where I is the indicator function. In this paper, we use the time- aware filtering protocol proposed in XERTE(Han et al., 2021), which filters out all the true entities [o](s; r, 0, E G.} given query except the answer entity of query Compared with the traditional filtering protocol that filters out all the true entities long as the triplet (s, o) holds true at any timestamps, time-aware filtering is more reasonable.
r,,
As for the implementation details, we learn cyclic rules with length of 1,2,3 and acyclic rules. We set the the number of sampled walks dur- ing rules learning stage to 200 for cyclic rules on all datasets and for acyclic rules, we it 5000 on ICEWS14 and 1000 on ICEWS0515 and ICEWS18. During the confidence estimation pro- cess, we sample 500 body instances for all kinds of rules. The smoothing parameter e is 3 and we filter out rules with confidence less than 0.01 and body instances number less than 2. We set w in (1) to 1 for cyclic rules in all lengths and 0.5 for acyclic rules. a, À in (3) are set to 0.5 and for all three datasets. Following TLogic, we also time windows during rule applications which filter out facts not in the period spanning from ty Wsise to ty. where ty represents the timestamp of the query and Wsize denotes the time window size. We Wsize to 200 for ICEWS18, 1000 for ICEWS0515 and use all facts on ICEWS14, which is the same TLogie does.
score(c) = 1- -I s)
(4)
SEC
where C. represents all the scores of candidate derived from multiple rules. The intuition behind Noisy-OR is that the results represent the possibil- ity that at least one rule holds for candidate e. is possible that no rule learned can answer the given query. In this paper, we follow the simple baseline proposed in TLogic which will generate candidate answers according to the object distribu- tion in the training set.
5 Experiments
5.1 Experimental Setup
Datasets evaluate TR- RR-Rules on three datasets: ICEWS14, ICEWS18 and ICEWS0515. All of the three datasets subsets of Integrated Crisis Early Warning System (Boschee et al., 2015), which store political events take place in 2014, 2018 and from 2005 to 2015 respectively. Under TKG extrapola- tion settings, all datascts are split into train, valid
Baseline Methods In this paper, we select both static and temporal state-of-the-art models as base- lines. As for static models we select Dismult(Yang et al., 2014), ComplEx(Trouillon et al., 2016) and AMMlllice et al., 2019). Temporal mod- els include TTranSE (Leblay and Chekol, 2018), TA- DistMult (Garcfa-Durán et al., 2018), DE- SimplE(Goel et al., 2020), TNTComplEx (Lacroix et al., 2020), RE-Net (Jin et al., 2019), CyGNet (Zhu et al., 2021), XERTE (Han et al., 2021),
Nevin
Nouti
Nr
N...
N.
Nume
ICEWS14
63685
13823
13222
7128
230
365
ICEWS0515
522958
69224
69147
251
4017
ICEWSI8
373018
45995
49545
23033
256
304
Table I: Statistics of ICEWS14, ICEWS05-15 and ICEWS18

---

ICEWSI4
ICEWS05-15
ICEWS18
Model
MRR
Hits
H
Hits@10 10
MRR
0its@1 I
Hits@:3
Hits@ 10
MRR
Hits@1
Hiis@13
Hits@ 10
Dismult
27.67
18.16
31.15
46.96
28.73
19.33
32.19
47.54
10.17
4.52
10.33
21.25
ComplEx
30.84
21.51
34.48
49.58
31,69
21
35.74
52.04
21.01
11.87
23.47
39.87
AnyBURI
29.67
21.26
33.33
46.73
32.05
23.72
35.45
50.46
22.77
15.10
25.44
38.91
Thrans
13.43
3.11
17.32
34.55
15.71
5.00
19.72
38.02
8.31
1.92
8.56
21.89
TA-Dismult
26.47
30.22
24.31
14.58
27.92
44.21
16.75
8.61
18.41
SimplE
49.11
38.99
52.75
19.30
11.53
21.86
TNTComplEx
23.35
36.03
49.13
27.54
19.52
30.80
42.86
21.23
13.28
24.02
36,91
RE-Net
38.28
28.68
41.34
54.52
42.97
31.26
46.85
63.47
28.81
19.05
32.44
47.51
CyGiNet
32,73
23.69
36.31
50.67
34.97
25.67
39.09
52.94
24.93
15,90
28.28
42.61
CENET
37.16
27.78
41.16
55.49
27.14
18.58
29.99
44.15
XERTE 40.79
32.70
45.67
$7.30
46.62
37.84
52.31
63.92
29.31
21.03
33.51
46.48
LLogic 43.04
33.56
48.27
61.23
46.97
36.21
53.13
67.43
29.82
20.54
33,95
48.53
TR-Rules
43.32
33.96
48.55
61.17
47.64
37.06
$3.80
67.57
30.41
21,10
34.58
48,92
Table Results MrTRRlless on ICEWS14, ICEWS05-15 and ICEWS18. Best results are bold and second best results underlined.
TLogic(Liu et al., 2022) and CE-NET(Xu et al., 2022). All the results except CENET are taken from (Liu et al., 2022) and we evaluate CENET on ICEWS0515 and ICEWS18 using aime- filtering protocol.
ble explanation that, first, we mine more acyclic rules, the superiority of window confidence is am- plified. In addition, without the cyclic restriction, the matched body instances of acyclic rules are slightly less likely to be relevant to the correspond- ing head instances. In other words, acyclic rules suffer more from temporal redundancy due its acyclic forms. However, the aggregation operation in window confidence perfectly alleviate this is- sue. This also explains why TR-Rules(C+A) which just additionally involves acyclic rules comparing to TLogic, gives worse results than TLogic does. In other words, under temporal settings, window confidence provides the proper way to make use of acyclic rules through fixing the bias in confidence calculation caused by temporal redundancy, As discussed in TLogic, we can see that only cyclic rules of length I can give competitive performance on all datasets.
5.2 Main Results
Table 2 reports results of TR-Rules and all base- line models under time-aware filtering protocol on three datasets. We can see that TR-Rules achieves state-of-the-art performance in all met- rics except Hits@ 10 on ICEWS14 and Hits@ on ICEWS0515 where TR-Rules gives the sec- ond best results. Table 3 presents some high con- fidence rules of variant types mined by TR-Rules, which demonstrates the better interpretability of TR-Rules.
5.3 Ablation Study
To study the contribution of acyclic rules and the impact of Temporal Redundancy on different kinds of rules, we perform ablation study on three datasets and the results are given in Table 4. As for the notations, Cl means we only mine eyclic rules of length one, A means only mine acyclic rules, C means we only mine cyclic rules but in- cluding length of 1,2.3 and R means we use win- dow confidence for solving Temporal Redundancy, TR-Rules(C) is equivalent to TLogic which mines cyclic rules in TKGs.
5.4 Acyelic Rules Performance Analysis
We also study the performance of both kinds of rules on ICEWS14 when the number of sampled walks during rule learning are set to different val- ues. figure 3, the orange line displays the performance of acyclic rules with standard con- fidence being utilized, the blue line is the perfor- mance of acyelic rules with window confidence being utilized and the green line is the performance of cyclic rules with standard confidence. As we can see in figure 3, the performance of acyclic rules with whichever confdence calculation algo- rithms improves dramatically as the number of sampled walks grows. The improvements are up to 14.89 (Hits@ 10) and 10.07% (MRR). How- ever, in comparison, the performance of cyclic rules changes no more than 0.2% as the number of walks
We can see that the introduction of window con- fidence that addresses Temporal Redundancy im- proves the performance of TR-Rules in most cases including mining only cyclic rules or only acyclic rules or both. Besides, window confidence im- proves the performance acyclic rules more than that of cyclic rules on three datasets. The possi-

---

Rullss
Confideuce
Type
(X. Engage negobattons,, Y.Tr) (X, Express intent euse sanction, Y.To)
0.80
Cycliel
(X, Conduct strike, (rriee,.TT)) X.TT) <- (X. Break diplomatic relations, Y.
0.82
Cyclic3
(Y Expel peacekeepers" X, Thg) A (X, Break diplomatic relations, T.Ta))
(x, Reduce relation, ion, South Korea, Tx) + (x, Express ontent to ease sunction, North Korea, Is)
0.86
Acyclic
Table 3: High confidence rules mined by TR-Rules
ICEWS14
ICEWS05-15
ICEWS18
Model
MRR
Hitx@1
Hits@
Hits@
MRR
Hitv@0
Hitv@83
Hitv@1
MRR
Hits@1
Hits@0
Hity@
TLogic
43.04
33.56
48.27
46.97
36.21
53.13
29.82
33,95
48.53
Rules(CI)
40.57
30.86
46.32
59.16
44.98
34.18
51.42
65.57
27.73
31.70
46.64
(R-Ru)es(CI+R)
40.79
31.16
46.57
59.21
45.76
35.07
52.26
66.10
27.83
18.74
31,90
46.68
TR Rules(A)
31.14
24.60
35.11
43.75
30,28
23.88
34,21
42.72
17.02
11.86
19.51
27.85
TR-Ru Rules(A+R)
31.98
25.33
35.97
44,83
30.85
24.56
34,57
43.11
17.28
12.11
19.81
28.21
TR- Rules(C+R)
43.03
33.54
48.33
61.07
47.71
37.10
53.91
67.72
29.87
20.55
33,94
48.63
TR-Rules(C+A)
41.58
32,54
46,56
59.23
46.08
35.44
52,24
66,25
29.56
20,36
33.68
48.12
TR-Rules
43.32
33.96
48.55
61.17
47.64
37.06
53.80
67.57
30.41
21.10
34.58
48.92
Table 4: Ablation study of TR- TR-Rules on ICEWS14, ICEWS05-15 and ICEWS18. Best results are in bold and second best results are underlined.
Elcribatie doctionn
grows. The possible reason is that acyclic rules involve fixed entities (e.g. South Korea instead of Country) while entities are replaced with variables in cyclic rules. Hence, acyclic rules need to cap- ture enough information in TKGs by more samples and cyclic rules need less sampled walks because only some specific walks can be generalized into a paradigm. The performance of cyelic rules even get slightly worse because 200 sampled walks are enough for mining strong and general rules and as sampling more walks, more rare but accidental rules are obtained which might affect the inference accuracy, Thus, we can conclude that acyclic rules are more sensitive the number of walks and yield promising results.
Randdrr
Figure 4: The confidence distribution of rules mined by TR-Rules(windoww and (lgggi((ta)d)rr) on ICEWS0515.
line increases which means the impact of Temporal Redundancy also grows as the number of sampled walks gets larger and window confidence resolves it effectively. Figure 4 describes the distribution of confidence of rules calculated by standard and window confidence algorithm. As we can see in the figure the left which displays the confidence distribution of all rules, generally, confidence com- puted by window confidence algorithm inclines be in lower confidence section. This phenomenon coincides with the intuition that by solving tem- poral redundancy, TR-Rules gives lower but more practical confidence under temporal settings, which can be viewed as refinements for temporal rules and the original inflated confidence is corrected. It demonstrates that our method indeed works and verifies the correctness of our motivations. The rest two figures display the confidence distribution of acyelic rules and cyclic rules respectively. We can see that the difference between these two distribu-
40.0
15.0
130-00
43
502
32.5
42-
055
Aryt Ioaaliio corfderce
35.
acytlime reta confidere
connacece
1000
2000
3000
<ooo
5002
1000
2002
2000
<d0o
soba
Samberofoangee milks
Sunberofsanplea ealls
Figure 3: Results of cyclic rules and acyclic rules with different number of sampled walks on ICEWS14.
5.5 Temporal Redundancy Impact Analysis As we have discussed in section 5.3, window confi- dence benefits the performance of both eyclic rules and acyclic rules. In figure 3, we can clearly see the difference between the orange line and blue

---

tions larger in the middle figure which supports our conclusion of acyclic rules suffer more from temporal redundancy. It further demonstrates that our proposed window confidence serves as an ef- fective way to utilize acyclic rules TKGs for extrapolation task.
embedding-based models, our model has better in- terpretability since it can generate cuman-readable
rules. Besides, the inference process of TR-Rules are visible, which means that users can see the
whole progress of how the model gives answers in- cluding which rules are utilized and how the scores of candidates are aggregated. With these superi- orities, TR-Rules can be applied into downstream applications which require high interpretabilities and reliabilities. Thus, our work complies with the ethnies policy.
6
Conclusion
In this paper, we define the problem of Temporal Redundancy and propose TR-Rules, rule-based TKG extrapolation model which solves Temporal Redundancy by replacing standard confidence with window confidence. TR-Rules also firstly mines acyclic rules in TKGs which are proved to have more promising performance compared with cyclic rules because of their higher sensitivity to the num- ber of sampled random walks. Experimental results show that TR-Rules achieves state-of-the of-thh-rrt art per- formance in most metrics on three datasets. As for future work, it promising to explore more sophisticated aggregation methods for window con- fidence calculation, such as employing cluster or machine learning methods.
References
Elizabeth Boschee, Jennifer Lautenschlager. Sean O'Brien, Steve Shellman, James Starz, and Michael Ward. 2015. Icews coded event data. Harvard Data- verse, 12.
Luis Antonio Galárraga, Christina Teflioudi, Katja Hose, and Fabian Suchanek. 2013. Amie: association rule mining under incomplete evidence ontological knowledge bases. In Proceedings of the 22nd in- lernational conference World Wide Web, pages 413-422.
Alberto Gareía-Durán, Sehastijan Dumančié, and Math- ias Niepert. 2018. Learning sequence encoders for temporal knowledge graph completion. arXiv preprint 1rX.vv:809.032000
Limitations
As we discussed in Section 4.1, the aggregation function in window confidence is quite simple. It cannot model the interactions among body in- stances in different windows which might result inaccurate confidence in some cases. Besides, for rule-based models, especially those mandom-walk based models like TR-Rules and TLogic, it is im- possible to mine very long rules due to the unaf- fordable time consumption. In Ru-Rules, we mine cyclic rules of length 1,2,3 and acyclic rules of length 1. Thus, as discuss in conclusion, de- veloping more sophisticated aggregation functions in window confidence might yields better perfor- mance. Moreover, employing some other rule learn- ing algorithms other than random walk sampling might obtain more high quality rules.
Rishab Goel, Seyed Mehran Kazemi, Mareus Brubaker, and Pascal Poupart. 2020. Diachromic embedding for temporal knowledge graph completion. In Proceed- ings of the AAAI conference on artificial intelligence, volume 34, pages 3988-3995.
Zhen Han. Peng Chen, Yumpu Ma, and Volker Tresp. 2021. Explainable subgraph reasoning for forecast- ing on temporal knowledge graphs. In International Conference on Learning Representations.
Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, and Jun Zhao. 2017. An end- to-end model for question answering over knowledge base with cross-attention combining global knowl- edge. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers), pages 221-231.
Yongquan He, Peng Zhang, Luchen Liu, Qi Liang, Wenyuan Zhang, and Chuang Zhang, 2021. Hip net- work: Historical information passing network for ex- trapolation reasoning on temporal knowledge graph. In IJCAI, pages 1915-1921.
Acknowledgements
This work is supported by the National Science Foundation of China (Grant No. 62176026) and Beijing Natural Science Foundation (M22009).
Marcel Hildebrandt, Swathi Shyam Sunder, Serghei Mogoreanu, Mitchell Joblin. Akhil Mehta, Ingo Thon, and Volker Tresp. 2019. A recommender sys- tem for complex real-world applications with nonlin- ear dependencies and knowledge graph context. The Semantic Web: 16th International Conference,
Ethics Statement
this paper, we propose TR-Rules, a rulc- based TKG extrapolation model. Compared with

---

ESWC 2019, Portorot, Slovenia, June 2-6, 2019, Pro- ceedings 16, pages 193. Springer.
the AAAI Conference on Artificial Intelligence, vol-
ume 35, pages 4732-4740
Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. 2019. Recurrent event network: Autoregressive structure inference over temporal knowledge graphs. arXiv preprini urXiv:990..05530.
Timothée Lacroix, Guillaume Obozinski, and Nico- las Usunier. 2020. Tensor decompositions for tem- poral knowledge base completion. arXiv preprint arXiv:2004.04922
Julien Leblay and Melisachew Wudage Chekol. 2018. Deriving validity time in knowledge graph. In Com- panion proceedings of the the web conference 2018, pages 1771-1776.
Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Ji- afeng Guo, Yuanzhuo Wang, and Xueqi Cheng. 2021. Search from history and reason for future: Two-stage reasoning on temporal knowledge graphs arXiv preprint 20X.::2106000222
Yushan Liu, Yunpu Ma. Marcel Hildebrandt, Mitchell Joblin, and Volker Tresp. 2022. Tlogic: Temporal logical rules for explainable link forecasting on tem- poral knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4120-
Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2018. Entity-duet neural ranking: Un- derstanding the role of knowledge graph seman- tics in neural information retrieval. arXiv preprint 1xXYv:880500791.
Christian Meilicke, Melisachew Wudage Chekol, Daniel Ruffinelli, and Heiner Stuckenschmidt. 2019, Any- time bottom-up rule learning for knowledge graph completion. In IJCAL, pages 3137-3143.
Simon Ott, Christian Meilicke, and Matthias Samwald. 2021. Safran: An interpretable, base- link pre- diction method outperforming embedding models. arXiv preprint 1090v:210..08002.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Interna- tional conference on machine learning, pages 2071- 2080. PMLR.
Yi Xu, Junjie Ou, Hui Xu, and Luoyi Fu. 2022. Tem- poral knowledge graph reasoning with historical con- trastive learning. arXiv preprint arXiv:2211.1090
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng, 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint 141::141.
Cunchao 7hu, Muhao Chen, Changjun Fan, Guangquan Cheng, and Yan Zhang. 2021. Learning from history: Modeling temporal knowledge graphs with sequen- tial copy-gennration networks. In Proceedings of


Reviews:
[{"review_id": "Wh5mHGU8DX", "reviewer": "Reviewer_uQVS", "summary": "", "questions": "A. Could the authors provide the rationale behind the selection of these baselines?\nB. Have the authors considered conducting multiple experiments and averaging the results?\nC. Have the authors considered releasing the paper’s code publicly?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough feedback on our paper. We are pleased that the reviewer recognizes the merits of our work and acknowledges the contributions we have made to the field. We would like to address the concerns and questions below.\nwe are sorry about the grammatical mistakes and blurry figure and we will correct them in the final version if accepted. Thank you for your suggestions.\n\nReply A:\nThe first three models (Dismult, ComplEx, AnyBURL) are competitive link prediction models for static KG which do not consider temporal information. The reason that we include them is that we want to show TR-Rules performs much better than static models since TR-Rules captures temporal information by temporal rules sampling.\n\n(TTransE, TA-Dismult, DE-SimplE, TNTComplEx) these four models are designed for interpolation task on TKGs which are not effective in predicting future events and, instead, they focus on inferring missing facts in the past. The reason of selecting these models is that we want to show the superiority of TR-Rules over these interpolation models because TR-Rules mines temporal rules which can be considered as causality or logic between potential future events and existing facts. As a result, we can say TR-Rules is suitable for TKG extrapolation task.\n\n(RENET, CyGNet, xERTE, CENET, TLogic) these models are for extrapolation task on TKGs. We include these models to show that our model TR-Rules indeed achieves state-of-the-art performance on extrapolation on TKGs. Especially the comparison with TLogic demonstrates the effectiveness of the unprecedented use of acyclic rules and the proposed window confidence for solving temporal redundancy.\n\nReply B:\nI completely understand your concern and here are my explanations: the main results provided in TR-Rules are the average of 5 rounds of experiments and the ablation study results are the average of 3 rounds of experiments. The following table shows the max differences of MRR in 5 rounds of experiments. Besides, stable performance is one of the property of rule-based models. As TLogic points out that running the same model with the same hyperparameter settings results in very small deviation, which coincides with conditions of TR-Rules. I'm sorry about the omissions of relevant settings and we will consider add these informations into the final version if possible. The following table displays the differences between multiple results of TR-Rules.\n\nMax Differences of MRR (in percent):\n|Datasets| ICEWS14 | ICEWS0515 | ICEWS18 | \n| --- | --- | --- | --- | \n|Results|43.32|47.64|30.41|\n|Max Difference|-0.11|+0.13|-0.09|\n\n\nReply C:\nYes, we have submitted our source code in the supplementary materials and we will definitely release our source code if this paper is accepted.\n\nWe kindly request that you increase your rating to acceptance, especially for the soundness score, in light of the improvements and illustrations we have made. A higher rating is an encouragement and affirmation of our work in this field. Thank you again for your thorough feedbacks. If you still have any concerns, please feel free to comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a new method which is TR-Rules, a rule-based model for extrapolating temporal knowledge graphs. This innovative approach effectively addresses temporal redundancy through a straightforward algorithm. Notably, TR-Rules is the pioneering model to extract acyclic rules from temporal knowledge graphs.\n2. This paper includes multiple baselines and conducts extensive experiments, with the obtained results sufficiently validating the scientific and rational nature of the proposed approach.\n3. The primary contribution of this paper is to address Temporal Redundancy in TKG within the field of natural language, offering significant research significance.", "reasons_to_reject": "1. Some formulas are missing labels, such as the formulas on line 171 and line 208, etc.\n2. The figures in the paper are a bit blurry. Please consider replacing them with clearer ones.\n3. Some sentences contain grammatical mistakes or are not complete sentences, such as, in line 370, the word \"learning\" should be preceded by an article such as \"a\" or \"the\".\n4. The baselines in the paper lack corresponding explanations, which makes it difficult for me to understand the reasons behind their selection. Therefore, I kindly request the author to provide a brief introduction to these baselines.", "questions_to_authors": "A. Could the authors provide the rationale behind the selection of these baselines?\nB. Have the authors considered conducting multiple experiments and averaging the results?\nC. Have the authors considered releasing the paper’s code publicly?", "paper_topic_and_main_contributions": "he topic of this paper is closely related to the temporal knowledge graph (TKG), which has been proven to be an effective method for modeling dynamic facts in the real world. Many efforts have been dedicated to predicting future events, known as extrapolation, on TKGs. Recently, rule-based knowledge graph completion methods, considered more interpretable than embedding-based approaches, have been extended to address temporal knowledge graph extrapolation. However, in dynamic scenarios, rule-based models suffer from temporal redundancy, leading to inaccurate rule confidence calculations. As a response, the authors define the temporal redundancy problem and propose TR-Rules, a model that effectively resolves this issue through a straightforward yet powerful strategy. Moreover, to harness more latent information within TKGs, TR-Rules explores acyclic rules via temporal random walks, a facet unexplored by existing models. Experimental results on three benchmarks demonstrate that TR-Rules achieve state-of-the-art performance. Ablation studies highlight the impact of temporal redundancy and underscore the greater potential of acyclic rules, as their performance is more sensitive to the number of sampled walks during the learning phase.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In line 370, the word \"learning\" should be preceded by an article such as \"a\" or \"the\".\n2. In line 592, the word \"first\" seems redundant and should be removed.\n3. Apart from the two minor errors mentioned above, I didn't identify any other grammar mistakes. However, I recommend the authors consider seeking assistance from a proofreading service or using ChatGPT to polish the paper."}}, {"review_id": "7J6fP7AJBz", "reviewer": "Reviewer_cXsZ", "summary": "", "questions": "A.Is the acyclic rules first proposed by this paper? if not, there should have references and more analysis about how the acyclic rules were used in previous methods. \\\nB. Figure 3 is confusing. The green line is not explained. The result of the blue line is not consistent with the number in Table 2. \\\nC. In Table 2, TR-Rules (C+A) are equivalent to TLogic with acyclic rules. However, its performance is bad than TLogic, which is weird and needs more analysis. \\\nD. Section 5.4 is confusing. Line 536, ‘the performance of acyclic rules improves dramatically as the number of sampled walked grows’, how does it convey? Where is the acyclic rule in Figure 3?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback on our paper. We appreciate your dedication to ensuring the quality and clarity of our work and feel grateful that you recognize the contribution of proposed methods. \n\nHowever, probably because of the organization of this paper, we think that the contribution of unprecedentedly leveraging acyclic rules on TKG extrapolation task is slightly underestimated. Also, in terms of this, the proposed window confidence can be viewed as an exploration of how to use acyclic rules properly under temporal settings.\n\nHere, we give an intuitive example to illustrate the necessity and benefits of using acyclic rules on TKG. In fact, acyclic rules can be replaced by cyclic rules as long as there are plenty knowledge lurking in TKGs, however it is really costly and not feasible to build TKGs to include “plenty” of knowledge. Take a case from ICEWS14 for example:\n\nAcyclic rule:\n\n(X, Reduce Relation, South Korea, T1) ← (X, Express intent to ease sanction, North Korea, T2)\n\nCan be considered equal to cyclic rules:\n\n(X, Reduce Relation, Y, T1) ← (X, Express intent to ease sanction, Z, T2) and (Z, Is Hostile To, Y, T3)\n\nHowever, ICEWS14 does not contain relation “is hostile to” or any relations explicitly describing status between countries is already broken (relation “Reduce Relation” serves like an action rather than a description of already in bad status), since it is a dataset recording diplomatic events and activity. As a result, equivalent cyclic rules can not be obtained under this case. However, the acyclic rule itself implicitly contains the knowledge about “North Korea is in a bad relation with South Korea”\n\nThus, intuitively the introduction of acyclic rules should make improvements since they serve as another form of knowledge. But as you mentioned in question C, the  results do not coincide with expectations. This is the main factor that motivates us discovering the existence of Temporal Redundancy. Because, as mentioned in section 5.3, paragraph 2, without the cyclic restriction and confined to length of 1, the matched body instances of acyclic rules repeat more in a time window and some of them might be less relevant to the head instances. In other words, acyclic rules suffer more from Temporal Redundancy. Simply combining acyclic rules with cyclic rules “disturbs” the inference of cyclic rules rather than contributes to more effective inference. Because under temporal settings, the standard confidence calculation cannot correctly evaluates rules' plausibility especially acyclic. The proposed window confidence alleviates this issue and makes them work together. Hence, we can say that if a model intends to capture knowledge in the acyclic form under temporal settings, the window confidence is an effective solutions.\nBesides, we also discover that, due to its higher sensitivity to the number of sampled walks, acyclic rule gives promising performance under temporal settings, which has not been discussed before even on static KGs.\n\nAs for the reason to reject B, following your insightful suggestions, we generate the distribution of confidence scores of TLogic and TR-Rules on two datasets. The results are as follows: (numbers denote the proportion ) \n\nICEWS14 All Rules:\n|Models| 0-0.1 | 0.1-0.2 | 0.2-0.3 | 0.3-0.4 | 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |\n| --- | --- | --- | --- | --- | --- | --- |--- | --- | --- | --- |\n|TLogic|0.212|0.172|0.162|0.130|0.155|0.079|0.055|0.025|0.010|0.001|\n|TR-Rules|0.427|0.214|0.153|0.083|0.078 |0.024|0.013|0.006|0.002|0.001|\n\nICEWS0515 All Rules:\n\n|Models| 0-0.1 | 0.1-0.2 | 0.2-0.3 | 0.3-0.4 | 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |\n| --- | --- | --- | --- | --- | --- | --- |--- | --- | --- | --- |\n|TLogic|0.312|0.173|0.130|0.097|0.097|0.064|0.053|0.039|0.026|0.009|\n|TR-Rules|0.570|0.176|0.099|0.056|0.048|0.021|0.013| 0.008|0.005|0.002|\n\nThe above table shows that on both datasets, in comparison, confidence generated by TR-Rules incline to be in  lower confidence section. This phenomenon coincides with the intuition that by solving temporal redundancy, TR-Rules gives lower but more practical confidence under temporal settings, which can be viewed as refinement for temporal rules and the original inflated confidence is corrected. It demonstrates that our method indeed works and verifies the correctness of our motivations.\n\nWe also investigate the difference between distribution of the confidence score of ONLY acyclic rules. From the table below we can see that, the difference of acyclic rules is quite larger than the difference of all rules shown above. This is a solid support for our claim which is “acyclic rules suffer more from temporal redundancy”.\n\nICEWS14 ONLY Acyclic:\n|Models| 0-0.1 | 0.1-0.2 | 0.2-0.3 | 0.3-0.4 | 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |\n| --- | --- | --- | --- | --- | --- | --- |--- | --- | --- | --- |\n|TLogic|0.159|0.172|0.171|0.141|0.170|0.087|0.060|0.027|0.010|0.001|\n|TR-Rules|0.361|0.226|0.171|0.095|0.092|0.028|0.016|0.008|0.002|0.0001|\n\n\nICEWS0515 ONLY Acyclic: \n|Models| 0-0.1 | 0.1-0.2 | 0.2-0.3 | 0.3-0.4 | 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |\n| --- | --- | --- | --- | --- | --- | --- |--- | --- | --- | --- |\n|TLogic|0.215|0.168|0.143|0.113|0.118|0.079|0.068|0.050|.034|0.012|\n|TR-Rules|0.520|0.181|0.108|0.064|0.059|0.026|0.017|0.011|0.009|0.004|\n\n\nReply to Question A:\n\nWe are the first to explore using Acyclic rules on TKG extrapolation, which we think is one of the factors that  makes TR-Rules not an incremental work to TLogic.  As you mentioned in question C, experimental results show that simply using acyclic rules and cyclic rules together with standard confidence does not give better performance under temporal settings, which probably is the reason why no one has ever attempted to utilize acyclic rules for TKG extrapolation. Our proposed method enables these two types of rules to work together and give better performance under temporal settings.\n\nReply to Question B:\n\nI'm sorry about the confusion and really appreciate the time that you committed  into checking the experiments which helps us polish this section. We will the make the caption and illustration more clear. Here is my illustration of figure3 for you which will be added into the full paper later: \nThe main purpose of Figure 3 is to show that, compared with cyclic rules, the performance of Acyclic rules is much more sensitive to the number of sampled walks during rule learning stage. This conclusion means that acyclic rules can give promising performance and can be more effective on larger TKGs. \n\nThe x axis denotes the number of sampled walks during rule learning stage. \n\nThe orange line represents the performance of ONLY acyclic rules with standard confidence as the number of sampled walks grows; \n\nThe blue line represents the performance of ONLY acyclic rules with window confidence as the number of sampled walks grows;\n\nThe green line represents the performance of ONLY cyclic rules of length 1 with standard confidence as the number of sampled walks grows. \n\nWith these illustrations, the conclusions are supported more sufficiently. Conclusions including:\n\ni. the performance of acyclic rules improves dramatically as the number of sampled walks grows. \n\nii. Compared with cyclic rules, acyclic rules are more sensitive to the number of sampled walks which yields promising performance.\n\niii. The impact of Temporal Redundancy also intensifies as the number of walks grows since the difference between the orange line and the blue line increases .\n\nMore relevant analysis can be found in line 541 to line 547.\n\nReply to Question C:\n\nIndeed, the utilization of acyclic rules should gain improvements intuitively. However, as we claimed in section 5.3, paragraph 2, acyclic rules suffer more from temporal redundancy and the table of distribution provided for Reason to reject 2 displays a more significant change in the distribution of acyclic rules. The confidence of rules are generally overestimated by using standard confidence under temporal settings. As a result, the inference of cyclic rules is influenced by the more affected acyclic rules. \n\n\nReply to Question D:\n\nI really appreciate the time that you committed to thoroughly check experiments . I think the explanations provided for question B can answer these questions and help you follow Section 5.4 better. If you still have any lingering doubts, please comment on this rebuttal. Thank you.\n\nWe kindly request that you increase your rating to acceptance if you are satisfied with the illustrations and clarification we have made . A higher rating is an encouragement and affirmation of our work in this field. Thank you again for your insightful feedbacks which really help us to polish this paper. If you still have any concerns, please feel free to comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. This paper investigates an interesting problem of TKG forecasting which has been neglected by previous works. \\\nB. The proposed method is easy to understand and a lot of effort has been done in the figures to improve the readability of the paper. \\\nC. The code is provided.\n", "reasons_to_reject": "A. The contribution of this paper is incremental. The proposed strategy is more likely to be a trick or implementation skill in calculating confidence scores. \\\nB. There should be more experimental studies to verify the motivations, such as the distribution of confidence scores obtained by TLogic and TR-Rules. \\\nC. There are many errors in Section 5, where careful further polishing is required.\n", "questions_to_authors": "A.Is the acyclic rules first proposed by this paper? if not, there should have references and more analysis about how the acyclic rules were used in previous methods. \\\nB. Figure 3 is confusing. The green line is not explained. The result of the blue line is not consistent with the number in Table 2. \\\nC. In Table 2, TR-Rules (C+A) are equivalent to TLogic with acyclic rules. However, its performance is bad than TLogic, which is weird and needs more analysis. \\\nD. Section 5.4 is confusing. Line 536, ‘the performance of acyclic rules improves dramatically as the number of sampled walked grows’, how does it convey? Where is the acyclic rule in Figure 3?\n", "paper_topic_and_main_contributions": "This paper studies the link forecasting methods on temporal knowledge graphs. They propose a window confidence algorithm to solve the temporal redundancy problem of rule-based TKG forecasting methods and simultaneously model the cyclic and acyclic rules.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3PS5fL4Jgq", "reviewer": "Reviewer_FUMh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedbacks on our paper. We appreciate your crucial suggestions and the recognition of proposed window confidence for solving temporal redundancy.\n\nHowever, probably because of the organization of this paper, we think the contribution of successfully using acyclic rules is slightly underestimated. Here we would like to reclaim and draw your attention to the contribution of unprecedentedly using acyclic rules on TKG. \nAlso, in terms of this, the proposed window confidence can be viewed as an exploration of how to use acyclic rules properly under temporal settings.\n\nHere, we give an intuitive example again to illustrate the necessity and advantages of using acyclic rules on TKG. In fact, sometimes acyclic rules can have equivalent cyclic rules as long as there are plenty knowledge lurking in TKGs, however it is really costly and not feasible to build TKGs to include “plenty” of knowledge in all aspect. Take a case from life for example:\n\nAcyclic rule:\n\n(X, Job, Teacher, T1) ← (X, Major in, Education, T2)\n\nCan be considered equal to cyclic rules:\n\n(X, Job, Y, T1) ← (X, Major in, Z, T2) and (Z, Is Required, Y, T3)\n\nHowever, for a TKG that records people's activities, it probably does not contain relation “is required” or facts like (Education major, is Required, Y, Tx). As a result, equivalent cyclic rules can not be obtained under this case. The acyclic rule itself implicitly contains the knowledge about “Education major is required for Teacher”. \nThus, In most cases, the building cost and incompleteness of TKGs makes the utilization of acyclic rules irreplaceable. \n\nThus, intuitively the introduction of acyclic rules should make improvements since they serve as another form of knowledge. But no one has ever managed to use acyclic rules properly for TKG extrapolation. Perhaps, as shown in ablation study, simply using acyclic rules and cyclic rules together with standard confidence gives worse performance. In this paper, we discover an effective way to utilize these type of rules containing important knowledge. We argue that acyclic rules suffer more from Temporal Redundancy. Simply combining acyclic rules with cyclic rules “disturbs” the inference of cyclic rules rather than contributes to more effective inference. The proposed window confidence alleviates this issue and makes them work together. Hence, we can say that if a model intends to capture knowledge in the acyclic form under temporal settings, the window confidence is an effective solutions.\n\nBesides, we also discover that, due to its higher sensitivity to the number of sampled walks, acyclic rule gives promising performance under temporal settings, which has not been discussed before even on static KGs.\n\n\n\nReply for the first Question:\n\nAs we've mentioned above, we think the proper utilization of acyclic rules on TKG extrapolation is one of the most important points making TR-Rules not an incremental work to TLogic. As shown in ablation study, simply using acyclic rules and cyclic rules together with standard confidence gives worse performance, which perhaps is the reason why no one has ever managed to use acyclic rules under temporal settings. However, acyclic rules indeed contain irreplaceable information for extrapolation. From this point of view, we can say we are  the first to discover how to use acyclic rules properly. The proposed window confidence makes it successful by solving temporal redundancy which is a key factor preventing acyclic rules from making sense under temporal settings. Meanwhile, window confidence also benefits cyclic rules as well.\nWe genuinely appreciate that you mentioned the issue of generalization which is not included during writing and it really opens our mind to evaluate the contribution of TR-Rules from a new perspective. \nAs for the generalization ability, the window confidence proposed by TR-Rules can be applied to any rule-based method as long as it involves confidence calculation under temporal settings. The confidence calculation is independent from the rule-mining stage and the rule applying stage. No matter what novel rule mining strategies or effective aggregation methods for rule application the models introduce, window confidence can yield more appropriate confidence than standard confidence. Thank you again for providing us with a new view of evaluation!\n    \nReply for the second Question:\n\nI totally understand your concern. The followings are the table of multiple runs results and the distribution of rules mined by TLogic and TR-Rules which verifies the stability and demonstrates the effectiveness of TR-Rules since the overall confidence inclines to be lower as original rules with implausible high confidence are now assigned with more accurate confidence which means rules are refined. I think this can prove the effectiveness of solving temporal redundancy. Meanwhile, thanks to the window confidence, acyclic rules can provide appropriate information which makes positive contributions to inference.\n\nICEWS14 All Rules:\n|Models| 0-0.1 | 0.1-0.2 | 0.2-0.3 | 0.3-0.4 | 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |\n| --- | --- | --- | --- | --- | --- | --- |--- | --- | --- | --- |\n|TLogic|0.212|0.172|0.162|0.130|0.155|0.079|0.055|0.025|0.010|0.001|\n|TR-Rules|0.427|0.214|0.153|0.083|0.078 |0.024|0.013|0.006|0.002|0.001|\n\nICEWS0515 All Rules:\n\n|Models| 0-0.1 | 0.1-0.2 | 0.2-0.3 | 0.3-0.4 | 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |\n| --- | --- | --- | --- | --- | --- | --- |--- | --- | --- | --- |\n|TLogic|0.312|0.173|0.130|0.097|0.097|0.064|0.053|0.039|0.026|0.009|\n|TR-Rules|0.570|0.176|0.099|0.056|0.048|0.021|0.013| 0.008|0.005|0.002|\n\nThe above table shows that on both datasets, in comparison, TR-Rules gives lower but more practical confidence under temporal settings, since confidence generated by TR-Rules locals more in lower confidence section. This results complies to the intuition that by solving temporal redundancy the original inflated confidence is fixed and the temporal rules are more solid for inferencing. It demonstrates that our method indeed works and verifies the correctness of our motivations.\n\n\nTo verify our claim which is “acyclic rules suffer more from temporal redundancy”, we also scrutinize the difference between distribution of the confidence score of ONLY acyclic rules. From the table below we can see the dramatical change in distribution of Acyclic rules compared with that of all rules (table above) .\n\nICEWS14 ONLY Acyclic:\n|Models| 0-0.1 | 0.1-0.2 | 0.2-0.3 | 0.3-0.4 | 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |\n| --- | --- | --- | --- | --- | --- | --- |--- | --- | --- | --- |\n|TLogic|0.159|0.172|0.171|0.141|0.170|0.087|0.060|0.027|0.010|0.001|\n|TR-Rules|0.361|0.226|0.171|0.095|0.092|0.028|0.016|0.008|0.002|0.0001|\n\n\nICEWS0515 ONLY Acyclic: \n|Models| 0-0.1 | 0.1-0.2 | 0.2-0.3 | 0.3-0.4 | 0.4-0.5 | 0.5-0.6 | 0.6-0.7 | 0.7-0.8 | 0.8-0.9 | 0.9-1.0 |\n| --- | --- | --- | --- | --- | --- | --- |--- | --- | --- | --- |\n|TLogic|0.215|0.168|0.143|0.113|0.118|0.079|0.068|0.050|.034|0.012|\n|TR-Rules|0.520|0.181|0.108|0.064|0.059|0.026|0.017|0.011|0.009|0.004|\n\nMax Differences of MRR (in percent):\n|Datasets| ICEWS14 | ICEWS0515 | ICEWS18 | \n| --- | --- | --- | --- | \n|Results|43.32|47.64|30.41|\n|Max Differences|-0.11|+0.13|-0.09|\n\nThe stablity of rule-based methods has been shown in TLogic (standard deviation of results is 0.0012). This condition also coincides with TR-Rules's. \n\nWith extra experiments above, we can say the results of TR-Rules are not accidental and the proposed window confidence calculation indeed manages to solve temporal redundancy.\n\nWe kindly request you to raise your ratings if you are satisfied with the illustrations and experiments that we made. A higher rating is an encouragement and affirmation of our work in this field. Thank you again for your thorough as well as insightful feedbacks. If you still have any concerns, please feel free to comment. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written, and the experiments are well-conducted. The work also has good reproducibility as the authors provide the code. The results analysis is also interesting.", "reasons_to_reject": "Firstly, the generalization ability of the proposed method is unclear. It would be interesting to know if the method could be used in other rule-based or learning-based methods. As the proposed method is an extension to TLogic, it seems a bit incremental. Providing more insights into the generalization ability of the proposed method could strengthen the paper. \n\nSecondly, the improvements over TLogic in Table 2 are not very significant. Conducting a student t-test on multiple runs could help verify the significance of the results. This could provide readers with a better understanding of the effectiveness of the proposed method compared to TLogic.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This manuscript investigates the problem of link prediction in temporal knowledge graphs. The authors propose a simple and effective method to address the limitation of temporal redundancy in rule-based methods. ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method which is TR-Rules, a rule-based model for extrapolating temporal knowledge graphs. This innovative approach effectively addresses temporal redundancy through a straightforward algorithm. However, the reviewers also state the technical contribution is incremental, more experimental studies are expected to support the claims.
Decision: Accept-Findings