TFLEX: Temporal Feature-Logic Embedding Framework for Complex Reasoning over Temporal Knowledge Graph
Authors: Xueyuan Lin
Haihong E
Chengjin Xu
Gengxian Zhou
Haoran Luo
Tianyi Hu
Fenglong Su
Ningyuan Li
Mingzhi Sun
Venue: NeurIPS 2023 poster (2023)
Keywords: Temporal Knowledge Graph Reasoning
Temporal Knowledge Graph Embedding
Temporal Knowledge Graph
Temporal Logic
Knowledge Graph Reasoning
Knowledge Graph Embedding
Knowledge Graph
Machine Learning
TL;DR: We propose a novel embedding-based framework for complex logical reasoning over temporal knowledge graph.

Abstract: Multi-hop logical reasoning over knowledge graph plays a fundamental role in many artificial intelligence tasks.  Recent complex query embedding methods for reasoning focus on static KGs, while temporal knowledge graphs have not been fully explored.  Reasoning over TKGs has two challenges: 1. The query should answer entities or timestamps; 2. The operators should consider both set logic on entity set and temporal logic on timestamp set.

To bridge this gap, we introduce the multi-hop logical reasoning problem on TKGs and then propose the first temporal complex query embedding named Temporal Feature-Logic Embedding framework (TFLEX) to answer the temporal complex queries.  Specifically, we utilize fuzzy logic to compute the logic part of the Temporal Feature-Logic embedding, thus naturally modeling all first-order logic operations on the entity set.  In addition, we further extend fuzzy logic on timestamp set to cope with three extra temporal operators (**After**, **Before** and **Between**).

Experiments on numerous query patterns demonstrate the effectiveness of our method.

TFLEX: Temporal Feature-Logic Embedding Framework for Complex Reasoning over Temporal Knowledge Graph
Xueyuan Lin
Haihong E Chengjin Xu
Gengxian Zhou'
Haoran Luo
Tianyi Hu'
Fenglong Su Ningyuan Li Mingzhi Sun'
Beijing University of Posts and Telecommunications
University of Bonn
National University of Defense Technology
1inxy59emai12 sysu. edu. en, shaihong@bupt. edu. en, xucliai wuw.uniiionnn mmii-bonn.. de 2. 2.gengxlan@sel8 qmul aac. gmul.ac.uk. uk, (luohaoran, hutianyi } @buppt edd. on bufenggiongl@enud:: hwwp:ennuudeduueeeeooooe edu. en, Ilason.ningyuue
sunmingzhi) @bupt. edu.
Abstract
Multi-hop logical reasoning over knowledge graph plays fundamental role in many artificial intelligence tasks. Recent complex query embedding methods for reasoning focus static KGs, while temporal knowledge graphs have not been fully explored. Reasoning over TKGs has two challenges: 1. The query should answer entities timestamps: 2. The operators should consider both set logic on entity set and temporal logic on timestamp set. To bridge this gap, we introduce the multi-hop logical reasoning problem TKGs and then propose the first temporal complex query embedding named Temporal Feature-Logic Embedding framework (TFLEX) answer the temporal complex queries. Specifically, we utilize fuzzy logic to compute the logic part of the Temporal Feature-Logie embedding, thus naturally modeling all first-order logic operations the entity set. In addition, we further extend fuzzy logic on timestamp set to cope with three extra temporal operators (After, Before and Between). Experiments on numerous query patterns demonstrate the effectiveness of our method.
Introduction
Multi-hop logical reasoning over knowledge graphs (KGs) a fundamental issue in artificial intelli- gence. It aims to find the answer entities for first-order logic (FOL) query which involves logical operators (existential quantification 3, conjunction A, disjunction'V and negation-). Generally, the query is parsed into computation graph, according to which subgraph matching is executed on KG to find the answers. The computation graph a directed acyclic graph (DAG) whose nodes represent entity sets, and edges represent logical operators acting entity However, results are inevitably incorrect as KGs are incomplete and noisy. Besides, the computation complexity will spiral for large-scale KGs large queries. Therefore, people propose to embed query into oow-dimensional space to solve the problem.
Query embedding (QE) learns the embeddings of queries and entities, so that answer entities are close its queries embedding space. has attracted arising attention, as low-dimension embeddings can model implicit dependency and reduce computation. There follows series of QE methods, including GQE[I], Query2box(2], BetaE[3], ConE[4], etc. However, existing works only focus
Corresponding Authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Temporal Complex Query
[[[]] = =V; 3T,, To,
be Leeeeletee_ss__e (François Hollande, President of France, T.) A After Ta)
step_down_fee frrannrascoiss Hollande, President of France, Tb) ^ Before Ta)A make_a a_ visit Jinping, V1, To) A -make a visrt Obama, V2, T.)
Computation Graph
73
C1
C2
Cs
After
And
G3
C4
TimeAnd
d6
T4
Not
Before
61 François Hollande
#1 be elected
President of France os Xi Jinping
Fs : make visit
François Hollande C5
P22 stop down from e4
President of France eq : Barack Obama 74 make visit
Figure 1: A typical multi-hop temporal complex query and its computation graph: "During François Hollande was the president of France, which countries did Xi Jinping visit but Barack Obama did not visit?". In the computation graph, there are entity set (blue circle), timestamp set (green triangle), time set projection (green arrow), entity set projection (blue arrow) and logical operators (red rectangle).
on queries over static KGs. These methods can neither handle an entity query involving temporal information and operators, answer the timestamp set for temporal query.
Temporal knowledge graph (TKG) augments triples in static knowledge graphs with temporal information, represented as <head, relation, tail, timestamp> named fact. For example, the fact <Angela Merkel, make a visit, China, 2010-07-16> specifies the time when the event happens. Generally speaking, TKGs are more close to real world than static KGs, because most knowledge needs to be updated with time, and static KGs cannot express this change. In recommendation systems, TKGs are used to model user behaviors, which includes liking, buying, reading, commenting and so on. In financial applications, TKGs involve stock holding behaviors, trading behaviors, and financial events. These applications require reasoning over TKGs to answer complex queries. However, recent researches in TKGs focus on temporal link prediction, which is simply single-hop. A complex logical query involving multiple facts for multi-hop reasoning is not fully explored yet.
To fill the gap, we introduce a temporal multi-hop logical reasoning task over TKGs. The task aims to answer temporal complex queries, which have two main distinctions from existing queries over static KGs. Firstly, the answer sets for queries over TKGs are either entity sets or timestamp sets, while that for existing queries over static KGs can only be entity sets. Secondly, is included in the query, temporal operators such as After, Before should be considered apart from FOL operators. To understand this new task, we provide the definition of temporal complex query in Section 3. In addition, an example of temporal complex query is shown in Figure 1. This example query pertains to the financial event of China's 's visit and may be of interest financial analysts.
temporal information
According to the definition of the temporal complex query, we then generate datasets of temporal complex queries on three popular TKGs, and propose the first temporal complex query embedding framework, Temporal Feature-Logic Embedding framework (TFLEX) to answer these queries. In our framework, embeddings of objects (entity, query, timestamp) are divided into two parts, the entity part, and the timestamp part. The entity part handles the FOL operations over entities, while the timestamp part copes with the temporal operations over timestamps. Each part is further divided into feature and logic components. On the one hand, the computation of the logic components follows fuzzy logic, which enables our framework to handle all FOL operations. On the other hand, feature components are mingled and transformed under the guidance of logic components, thereby integrating logical information into the feature. Moreover, we extend fuzzy logic to support extra temporal operations (After, Before and Between) to handle temporal operations in the queries.
The contributions of our work are summarized as follows: (1) For the first time, the definition of the task of multi-hop logical reasoning over TKGs given. (2) We propose the first multi-hop logical reasoning framework TKGs, namely Temporal Feature-Logic Embedding framework (TFLEX),

---

which supports all FOL operations and extra temporal operations (After, Before and Between). (3) We generate three new TKG datasets for the task of multi-hop logical reasoning. Experiments on three generated datasets demonstrate the efficacy of the proposed framework. The source code of our framework and the datasets are available online
2 Related Work
Complex Query Embedding. It learns embeddings of queries and entities, and the answer entities are close to queries in the embedding space. Existing methods utilize a lot of objects to create the embeddings, such as (1) probability distribution [3, 5] (2) geometric object [1, 4, 6, (3) fuzzy logic [8-10] and (4) others [11-13]. However, existing embedding-bend beasbeee methods are considered on static KGs. They cannot utilize temporal information the TKGs, and therefore cannot handle temporal queries on temporal KG. Firstly, static query embeddings (QEs) are built over (s. r, o) triples instead of (s, r, o, t) quartets, thus ignoring the timestamps for temporal complex reasoning. The second reason is the order property of timestamps, which is on the contrary that entities are unordered, leading to that static QEs are unable to handle Before and After temporal logic. In addition, we also notice the semantic conflict in experiments (see section 5.2) when concatenating the geometric embedding (static QE) with the fuzzy embedding (that can handle temporal logic) to promote the static QE to temporal one. Therefore, it is challenging for static QEs to utilize temporal information the TKGs.
Temporal Knowledge Graph Completion (TKGC). It aims at inferencing new facts in the TKGs. Existing TKGC methods could be categorized (I) tensor decomposition [14-16], (2) timestamp- based transformation [17-21]. (3) dynamic embedding [22-25], (4) Markov process models [26, 27]. (5) autoregressive models [28-30], (6) others [31-33] and so on. Among these works, most of them only confined to the one-hop link prediction task, also known as one-hop reasoning. Some works [25, 28-30, 32] can perform multi-hop reasoning via path consisting of connected quartets. But none of them could answer logical queries that involve multiple logical operations (conjunction, negation and disjunction). In this paper, we focus on the temporal complex query answering task, which is more challenging than TKGC task.
3 Definitions
Temporal Knowledge Graph (TKG) G = [FF consists of entity set V. relation set R. timestamp set T and fact set F = object-timestane (s, r, o, quartets. Without loss of generality, G is first-order logic knowledge base, where each quartet (8, denotes atomic formula r(s, o, f), with a predicate and its arguments.
X V x T containing pubject-predicant
Multi-hop Logical Reasoning over TKG is the task to answer Temporal Complex Query when given a TKG G. We focus on Existential Positive First-Order (EPFO) query [34] over TKG. namely Temporal Complex Query g, which is categorized into entity query and timestamp query. Formally, the query q consists of target variable A, a non-variable anchor entity V₂ C V. non-variable anchor timestamp set Tg S T, bound variables V/, V/2 and Tı.... Tí. logical operations (existential quantification 3, conjunction A, disjunction V, identity 1, negation -), and extra temporal operations (After, Before). After(t1, t2) means to is after t,. while Before(t1, t2) means f2 is before 4. Inspired by [2, 3], the disjunctive normal form (DNF) of temporal query y is defined as:
q[A] = A : ..,,... Vn, Ti................ Ti: (e) A .... A A v... V (eT A ... A (
where = fo rV.. Vo or A, T) or f rV, or A, Vo. T) or g(T,, T,) q entity query, e= f=ror(V, Vo, T or or (T,, T,) or (T, A) or g(A,T) if i timestamp query
with V2. V4 U (V., .VA), T.T.TT E T. U (Tr,".. .. ,TTh.
r R E (1, {After, Before)
In the equation, the DNF is disjunction of m conjunctions, where of AfArrAee, denotes a conjunction between n, logical atoms, and each e: denotes a logical atom. We ignore indices in the definition
#ttpp::/ www.u...ommmiiii httpaa...........uuuuastt.....o.
3

---

of e/ to keep the formula clean. The goal of answering the query is to find the set of entities (or timestamps) [q] that satisfy the query, such that [q] iff q [ holds true, where [q] is the answer set of the query 4-
Following existing static query embedding works, we introduce Computation Graph, which is a directed acyclic graph (DAG) representing the structure of temporal complex query. Its nodes represent entity/iiestamp sets S C V. UV U T. UT, while directed edges represent logical or relational operations acting on these sets. A computation graph specifies how the reasoning of the query has proceeded on the TKG. Starting from anchor sets, we obtain the answer set after applying operations iteratively on non-answer sets according to the directed edges in the computation graph. The edge types on the computation graph are defined as follows:
1. Relational Projection P. Given an entity set S S V. a timestamp set S S T(or an entity set S2 S for entity projection) and relation r R. projection operation maps S₁ and S2 t'.t) (u, (v, v', F}. is entity projection
to another set: S' (ESES, {e'
(isss,, [t](u, t) E F), P is timestamp projection
2. Intersection I (Union Ut etc.). Given entity sets or timestamp sets {Si, ), the intersection (union, etc.) operation computes logical intersection (union, etc.) of these sets 1 S,, etc.).
3. Complement/Negation C. The complement set of a given set S is V -, SCV T S. SCT
4. Extended temporal operators f. Given a timestamp set S, extended operators compute
{ (t' for some t' TT, > max(S)); f After
certain set of timestamps S': S'
{L' for some 1' E T < min(S)), f is Before
In order to efficiently compute the answer set of temporal complex query, we consider embedding the query set into low-dimensional vector space, where the answer set is also represented by continuous embedding vector. Formally, the Temporal Query Embedding V, of query g is a continuous embedding vector, generated by executing operations according to the computation graph, starting from the temporal embeddings of anchor entity or timestamp sets. The Temporal Query Answer to the query q is the entity v (or timestamp f) whose embedding (or t) has the smallest distance dist(v, Vg) (or dist(t, Vg)) to the embedding of query q.
Method
In this section, replace the variables in the query formulation with temporal feature- em- beddings, and perform logical operations via neural networks. We first introduce the temporal feature-logic embedding for entities, timestamps, and queries in Section 4.1. Afterwards, we intro- duce logical operators in Section 4.2 and how to train the model in Section 4.3.
4.1 Temporal Feature-Logic Embeddings for Queries and Entities
In this section, we design temporal embeddings for queries, entities and timestamps. In general, the answers to queries may entities or timestamps. Therefore, we consider part of the embedding as an entity part to answer entities, while the other is the timestamp part to answer timestamps. Formally, the embedding of query set [q] is V, = -qf. qf q qi) where of E R2d entity feature, of E [0, 1Jd is entity logic, q/ E Rd is time feature, of E [0, 1] is time logic respectively, d is the embedding dimension. The parameter Qi is the uncertainty q: (1 qi of the feature, according to fuzzy logic. An entity e E V is a special query without uncertainty. We propose to represent an entity as the query with logic part O, which indicates that the entity's 's uncertainty is O. Formally, the embedding of entity v is v = (vf. 0, O, 0), where vf E Rd is the entity feature part and 0 is a d-dimensional vector with all elements being O. Similarly, the embedding of timestamp t is t = (0, 0, t/y 0) with entity part and time logic being O.
Attention that we introduce vector logic, which is a type of fuzzy logic over vector space, to cope with logical transformation in the vector space. Fuzzy logic a generalization Boolean logic, such that the truth value of logical atom is a real number in [0, 1]. In comparison, generalization

---

of a real number, the truth value in vector logic is a vector [0. 1| the logical operations in vector logic as (A)(A), OR(V),NOT( and so on, which receive one or multiple vectors and output one vector as answer. For more details about fuzzy logic, please refer to Appendix A.I.
/ in the semantic space. We denote
4.2 Logical Operators for Temporal Feature-Logic Embeddings
In this section, we introduce the designed neural logical operators, including projection, intersection, complement, and all other dyadic operators.
Projection Operator Pe and Pt. The goal of operator Pe is to map an entity set to another entity set under given relation and given timestamp, while operator Pe outputting timestamp set given relation and two entity queries. We define a function Pa V. V. + V in the embedding space to represent Pe, and Pe Vg r, V. + V for P. respectively. To implement Pe and Pt. first represent relations translations relational embedding r = (ry, rf, r'y rf). We follow the assumption of translation-ban
query embeddings and assign each relation with
methods: do ~ qe + r + t. As a comparison, static KGE TransE [35] has o R s + т, and temporal KGE TTranSE [36] has = 8t + The addition represents semantic translation starting from the source entity set, following the relation and timestamp conditioning, ending at the target entity set. Therefore, define P. and P. as:
Pc(V g. r, V,) = (MMP. (V, + V() P.(V 01 r, Vaz) (MLP)((V + +r vo2))
(1)
where MLP: R + Rid is a multi-layer perception network (MLP), + is element-wise addition and g is an activate function to generate of E [0. of E [0. We use MLP and activation function g(-) to make projection operator output valid query embedding, which shows the closure property of the operator. Pe and P. do not share parameters so the MLPs are different. We define as:
g(x) x(0 ox|o 2d)); x[2d: 3d]: (x(3d 4d)]
where x(O d) is the slice containing element r; of vector with index 0 i < d, a()) is Sigmoid function and [: is the concatenation operator.
Dyadic Operators. There are two types of dyadic operators for our framework. One for entity set and the other for timestamp set. Each type includes intersection (AND), union (OR), the symmetric difference (XOR), etc. With the help of fuzzy logic, our framework can model all dyadic operators directly. Below we take unified way to build these operators.
We start from intersection operators Ie (on entity set) and Iz (on timestamp set). The goal of intersection operator I. (I,) is represent [q] nn-1199 based on their entity parts (times- tamp parts). Suppose that Vg. q, = (q. T. L.
q(,)) is temporal feature-logic embedding for [a:]- We notice that there exists Alignment Rule in the process of reasoning. When perform- ing entity set intersection Ie, we should also perform intersection on timestamp parts in order to align the entities into the same time set. The same also holds for timestamp set intersection I, and all other dyadic operators. Therefore, we firstly define the intersection operators as follows: Alignment Rule
Fuzzy Logic
n
Io(V q::
V
og Li
AÑD i=1
Σ
B B
AND (qh))
I(V gi qm
aggi
n AND
(..) Σ Blai,
AND (gt.)
Alignment Rule
Fuzzy Logic
where AND is the conjunction in fuzzy logic, a; and 3 are attention weights. of logic, we compute af eet and gr via the following attention mechanism:
To notice the changes
(MPP (L (DD)
(MLP, ( d()))
1
(3)
Σー MMPP (i
Σラ MMLP ((
5

---

Before
After
d
df di dt
d
di
df +
Figure 2: The computation of time part in temporal operators Before and After.
where MLP Rdd + Rd are MLP networks, [ is concatenation. The first self-attention neural network will learn the hidden information from entity logic and leverage to entity feature, while the second one gathers logical information from time logic to time feature. Note that the computation of entity logic, and time logic obeys the law of fuzzy logic, without any extra learnable parameters. In this way, all dyadic operators can be generated from fuzzy logic in our framework. Due to space limitation, we present the union, exclusive or, implication operators and so on in Appendix A.3.
Complement Operators: C. and C, The aim of Ce is to identify the complement of query set [q] such that [-q] = V P'll while C, aims to calculate the complement [ T [[]] by the time parts. Suppose that V, = (q.. Li qy-gf) we define the complement operator Ce and Ct as:
(V₂V)) = (fan (4))), qy, qy.
Cv(V,) = (q7, qi fuu NOT(()))
(4)
where (gr) = (aanhMMMPs((4 (f))), ). Shoe (hollg))) = tanhhaanna (MLP.((gf) af) are feature negation functions, two MLP3,4 3,4 I22d Rd are MLP networks, NOT is negation in fuzzy logic.
Temporal Operators: After Ar, Before B, and Between D,, The operator After A. V, + V (Before B V, + ViD aims to deduce the timestamps after(before) a given fuzzy time set [4]] Let V, = (qy, of Zy. gf), we define A, and B; as:
+ of 1f of
1
Ac = (af. oi
(V,))) (ff. f o)
(5)
2
2
The entity part does not change after computation because temporal operator only affects the time part (time feature q) and time logic 4.) The motivation of computation is illustrated in Figure 2. Since
of is the uncertainty of time feature qf. the time part can be viewed as an interval [q] g' q3 gil
whose center is a' and half-length qf- The interval is covered by [q' 1, q6 + 1] because the probability df < 1. Then, after interval [a' di df + qf is the interval [a] + of. df + 1] whose center is q + -2
1+ and half-length is - 1-q which gives the time part of embedding A (V,).
Similarly, the time part of embedding B. (Vg)) is o3 1+タ (time feature) and --qî (time logic), which are generated from [q] - 1, 1, - of before !q' - gi qj + 4i)- The operator Between D. : Vg V9, + V inferences the time set after [q1] and before [oe]. Therefore, we define Between D, as D.(V Voz) = I.AA (V.,), (V to output the time between [[]]] and [[a]].
2
4.3 Learning Temporal Feature-Logie Embeddings
We expect the model to achieve high scores for the answers to the given query q. and low scores for o' & [g]. Therefore, we firstly define a distance function to measure the distance between given answer embedding and query embedding, and then we train the model with negative sampling loss.
Distance Function Given an entity embedding v (vf. 0,0,0, a timestamp embedding
(0, O, fy; 0) and a query embedding V, = (q qi g of) ). we define the distance d between the answer and the query q as the sum of the feature distance (between the feature parts) and the logic part (to expect uncertainty to be 0). If the query answers entities, the distance is d' dv: V,)) loy of |l + of Otherwise, the distance is d' (t; V₂) = l|t/ d/ + qf for queries answering timestamp set, where || || is the L norm and + is element-wise addition. The distance function aims to optimize two losses. One is to push the answers to the neighbor of query in the embedding

---

space. It corresponds to the term L1 distance between answer and query. The other is to reduce the uncertainty of the query (the probability interpretation of the logic part), to make the answers more accurate.
Loss Function Given training set of queries, we optimize negative sampling loss
k k Σ
L = - -lo (1 d(v: Vg))
oldo -7)
(6)
where 7 > 0 a fixed margin, is the number of negative entities, and (()) is the sigmoid function.
When query is answering entities (timestamps), E [q]] is positive entity (timestamp), 2g & [9] is the i-th negative entity (timestamp).
5 Experiments
In this section, we evaluate the ability of TFLEX to reason over TKGs. We first introduce experimental settings in Section 5.1, and then present the experimental results in Section 5.2.
5.1 Experimental Settings
Datasets and Query Generation Experiments are performed on three new datasets generated from standard benchmarks for TKGC: ICEWS14 [37], ICEWS05-15 [37], and GDELT-500 [38] (with statistics in Appendix B. We predefined 40 kinds of complex queries for each dataset. The definition of the 40 kinds of complex queries and the query generation process details are described in Appendix B.2. We consider 27 kinds of queries for training and all 40 kinds for evaluation and testing. Please refer to Appendix B.3 for summaries of the final datasets.
To briefly summarize the results, we aggregate groups of queries that can be answered: entities (avg.). timestamps (avg,), entities with negation (avge,c.). ), timestamps with negation (avg,,c, ), entities with unseen union (avg,u,)) )). timestamps with unseen union (avg )). and other hybrid unseen structures (avg,). These groups inspired by the experiment settings of existing static QEs [1-4]. The detail that which query belongs to which group will be shown in Appendix B.7. Note that the training set only involves 4 groups of queries: avg.. avg, c. and avgr.c."
Evaluation Given a test query q. for each non-trivial answer v E [9l - valva of the query rank it against non-answer entities V [I]]eesa (or non-answer timestamps T [g] text if the query is answering timestamps). Then we calculate Mean Reciprocal Rank (MRR) based on the rank. The higher, the better. Please refer to Appendix B.5 for the definition of MRR.
5.2
Main Results
For each group, we report the average MRR in Table 1. The raw MRR results all query structures for each dataset in detail are presented in Appendix B.7.
How well can TFLEX answer temporal complex queries?. We compare TFLEX with state-of-the- art query embeddings Query2box [2], BetaE [3] and ConE [4] on answering entities. Existing query embeddings only handle FOL entity set, but unable to cope with temporal logic over timestamp set. Therefore, the results of these three methods have to obtained by ignoring the timestamps, so that the avg,, a avg(u,) ) and avg, of three methods are zeros. Comparing the results of these methods in Table 1, we can see that TFLEX outperforms all the baselines on all the metrics. These results demonstrate that TFLEX can perform reasoning over TKGs well, at least better than the existing query embeddings.
Ablation on entity part. The variant X(ConE) replaces the entity part of TFLEX with ConE [4] to handle the logic over entity sets. In the entity part. ConE is geometric while TFLEX is fuzzy. The variant performs even worse than TFLEX and ConE. The dropping MRR indicates that the entity part plays an important role in the framework. Considering the performance of ConE, we think there is semantic conflict between the time part of TFLEX and the entity part of ConE. Simply combining static QEs with dynamic QEs is not clever way to achieve the best performance.
7

---

Table I: Average MRR results for different groups of temporal complex queries. X denotes the variant of TFLEX. X(ConE) replaces the entity part with ConE [4]. FLEX ablates the time part. X-1F merges entity feature and timestamp feature into feature. X-logic removes the logic part.
Dataset
Metries
Query2box
BetaE
ConE
TFLEX
X(ConE)
FLEX
X-1F
X-logic
avg.
25.06
37.19
41.94
56.79
40.93
43.67
56.89
56.64
avge.c. avg. aygr,c (14) (c4) avg. AVG
36.69
44.88
50.82
42.15
44.41
49.78
51.17
17.56
16.41
18.77
18.03
36.37
35.24
37.73
36.39
ICEWS14
19.95
26.47
35.74
25.46
29.25
34.48
34.68
26.24
24.07
28.04
26.36
28.03
26.65
29.31
28.61
35.93
30.13
36.43
35.98
avg.
24.00
31.33
40.93
48.99
36.29
38.96
49.90
44,80
avg..c. avgr a (Ht.)
29.70
43.52
46.17
38.12
42.10
46.11
41.92
4.39
4.41
4.43
3.29
30.16
29.49
30.26
28.34
ICEWS05-15
21.54
43.02
54.37
36.37
44.38
54.05
45.36
((4,) avg. AVG
28.69
26.40
27.70
23.39
24.26
21.69
24.41
21.95
33.72
27.54
33.98
29.86
avg. avg..C. avg. avgr. ( ( avg.
9.67
14.75
18.44
19.60
17.83
19.07
17.92
17.36
11.15
12.67
13.52
12.34
13.35
12.13
12.11
5.38
3.16
5.49
5.75
GDELT-500
6.31
3.93
6.50
6.86
7.41
7.44
6.92
6.71
6,35
6.59
6.80
6.17
6.17
6.47
6.64
AVG
9.32
8.17
8.86
8.92
Ablation on time part. The variant FLEX removes the time part of the temporal feature-logic embedding. Then, FLEX only answer entities. The results show that FLEX slightly outperforms ConE. However, the performance of FLEX is worse than TFLEX with large margin on all the datasets. Therefore, conclude that the time part also plays important role in the framework.
Ablation on feature part. If we remove the entity and timestamp feature, the embeddings of entities and timestamps will crash to zeros. Instead, consider another way explore the impact of the feature part. Noticing that some TKGC approaches [36, 39] embed entities and timestamps into the same semantic space, propose X-1F merge the entity and timestamp features into one feature. Compared with TFLEX, X-1F achieves higher scores on ICEWSI4 14 and ICEWS05-15, but lower on GDELT. The results imply that unifying the feature of entity and timestamp is potentially beneficial in some datasets.
Ablation on logic part. The variant X-logic removes both entity logic and time logic. It achieves lower scores than TFLEX on all the datasets. This because the logic part is responsible for reasoning over TKGs. Removing the logic part results in that neural logical operators completely rely on neural network to learn the logic, which is not enough to handle various temporal complex queries.
Out-of-data reasoning. The results on avg ). avg avg, support that the framework can reason over unseen entity logic, unseen time logic as well as their hybrid. The entity union and time union operators are not included in the training set, but the framework still handle them well.
Sensitive analysis. (L) The selection of hyperparameters (embedding dimension d, the margin 7) has substantial influence the effectiveness of TFLEX. We train the model with embedding dimension (300, 400, 500, 600, 700, 800, 900, 1000), the margin y E (5, 10, 15, 20, 25, 30, 35, 40). The best performance is achieved when d 800 and 15. Too small and too large d, y both lead to worse results, reported in Appendix B.6. (2) Besides, we also investigate the stability of TFLEX. We train and test for five times with different random seeds and report the error bars in Appendix B.6. The small standard variances demonstrate that the performance of TFLEX is stable.
Comparison between TFLEX and TKGC methods. It's natural to compare TFLEX with SOTA TKGC methods, since all of them can answer one-hop temporal queries. We present the compar- ison results in Table 2. We can that TFLEX is competitive with translatio-basee methods

---

Table 2: TKGC Results (%) ICEWS14, ICEWS05-115 15, and GDELT. The results from top to bottom are organized as static KGEs, timestamp-basee transformation TKGEs, tensor decomposition, autoregressive models and ours. Best results are in bold. indicate the results taken from [40, 17]. Other results are the best numbers reported in their respective paper.
ICEWS14
ICEWS05-15
GDELT
Model
MRR
Hir@1
Hir@3
Hit@ 10
MRR
Hit@
Hit@s
Hit@
MRR
Hit@1
H
TransE"
28.0
63.7
29.4
9.0
66.3
11.3
0.0
DistMult" [41]
43.9
32.3
67.2
45.6
33.7
69.
19.6
11.7
20.8
34.8
SimplE" [42]
45.8
34.1
51.6
68.7
47.8
35.9
53.9
70.8
12.4
22.0
36.6
ConT [43] [36]
18.5
11.7
20.5
31.5
16.3
10.5
18.9
27.2
14.4
8.0
15.6
26.5
25.5
HyTE [44]
29.7
10.8
41.6
65.5
31 6
11.6
44,5
68.1 1
11.8
0.0
16.5
32.6
TA-DistMult [45]
47.7
36,3
68.6
47.4
34,6
72.8
20.6
21.9
DE-TransE: [46]
32.6
12.4
46.7
68.6
31 4
10.8
45.3
68.5
12.6
0.0
18.1
35.0
DE-DistMult [46]
50.1
39,2
56.9
70.8
48.4
36.6
54.6
71.8
21.3
22.8
DE-SimplE [46]
52.6
41.8
59.2
72:5
51.3
39.2
57.8
74.8
23.0
14.1
24.8
40.3
ChronoR [17]
62.5
54.7
66.9
77.3
$9.6
TuckERT [39]
$9.4
51.8
64.0
73.1
62.7
55.0
67.4
76.9
45.3
61.4
TuckERTNT [39]
60.4
52.1
65.5
75.3
63.8
55.9
68.6
78.3
38.1
28.3
41.8
57.6
RGCRN [47, 40]
33.3
24.0
36.5
$1.5
35.9
26.2
40.0
$4.6
18.6
11.5
19.8
32.4
CyGNed
34.6
RE-NET [28]
35.7
25,9
40.1
54.8
36.8
26.2
41.8
$7.6
19.6
12.0
20.5
33.8
RE GCN' [40]
37.7
27.1
42.5
58.8
38.2
27.4
43.0
59.9
19.1
11.9
20.4
33.1
43.9
31.4
8.6
TFLEX
48.2
35.7
56.5
72.3
43.0
30.0
49.8
69.5
18.5
10.1
19.6
34.9
Ww
W
betore
attrr
Table 3: MRR of Pe on ICEWS14, ICEWS05- 15, and GDELT-500. The best results are in bold.
Simiariy Score
Results of TTransE and HyTE are taken from Goel
al. [46].
bPt P.(s..
Datasets
TTransE*
HyTE
TFLEX-Ip
TFLEX
Pt-Ris, (.
APP...0)....
ICEWS14
25.5
29.7
42.9
48.2
ICEWS05-15
27.1
31.6
40.6
43.0
Timastama (0=2014-111, 365-2014:12-31
GDELT1500
Figure Score distributions of Pt, bPt and aPt.
(ConT [43], TTranSE [36], HyTE [44), etc.), but it doesn't outperform SOTA TKGC methods like ChronoR [17] and TuckERT [39]. However, the result doesn't 't affect the novelty and contribution of this paper. Please note that the projection operator of TFLEX is as simple as TTransE, not further optimized for TKGC tasks only. Upgrading the projection operator outperform SOTA TKGC methods remains a future work.
Necessity of training on temporal complex queries. Our experiments demonstrate that training on complex queries is necessary to achieve the best performance. We compare with translation- based Pa operators (TTransE [36], HyTE [44] and TFLEX's variant TFLEX-Ip) using only one- Pe queries for training. We choose TTranSE and HyTE because our projection operator is also translation- based (Pa(V q, V.) V, + V.). From the result table 3, we observe that TFLEX achieves the best performance when comparing with these translation-baser baselines on all datasets. Besides, compared with TFLEX-1p. TFLEX achieves 7.9% relative improvement on average on MRR, which demonstrates that training on complex queries could improve the one-hop svery-answering ability.
Effectiveness of neural temporal operators. We found that the temporal operators A, and B, change the semantic predicted timestamp embedding logically. We randomly select an event (s, r,o) o) from ICEWS14 and consider three temporal queries Pt, bPt and aPt. Then, Pt(= Pe(s, o)) predicts the date when this event happened. And bPt(=B, r, o))) should predict the date before t, while aPt(=.A, (PP. o))) is supposed to predict the time after t. The output of three queries are time embeddings. Because
time embedding is fuzzy, we score to all possible Timestamps, and visualize the normalized similarity score distribution over all days in a year, from 0 to 365 in ICEWS14. The higher the score, the more possible the predicted date is the day. We plot the three score distributions in Figure 3. For each distribution, we highlight the periods of the highest scores

---

with a colored background. These colored blocks represent the most likely happening time interval of the event. We observe that the order of colored blocks (corresponding to the predictions of bPt, Pt, and aPt) matches the logical meanings of these operators (Before the event, On the event, After the event). The visualization shows that neural temporal operators perform the time transformation correctly. More examples are attached in Appendix D.
Explaining answers with temporal feature-logic framework. We take query Pe2 for example. Given temporal query Pe2: gf [[[]] V,, 3Va, ri (ei, Va,4) A (2 V2, f2), let's try an example which can be written as: On 2014-04-04 who consulted the man who was appealed or requested by the Head of Government (Latvia) on 2014-08-01? In this example, e1 is "Head of Government (Latvia)", r1 is "Make an appeal or request", 1 is "2014-08-01", 12 is "consult and l/2 is "2014- 04-04". Then we use TFLEX execute the query and get answers. We classify the answers into easy, hard and wrong ones. The easy answer is the correct answer that appears in the training set, while the hard answer the correct answer that exists in the testing set instead of training set.
We present five most likely answers for interpretation in Figure 4. From the table we observe that TFLEX ranks easy answers "François Hollande", "Taavi Rõivas" and hard answer "Andris Berzins" higher than wrong answers "Angela Merkel" and "Head of Government (Latvia)". This shows that TFLEX successfully finds the hard answer by performing complex reasoning, and distinguishes the right answers from the wrong ones.
We provide 36 examples in Appendix E, including the visualization and intermediate explanation of answers for each query structure.
Query Sentence
On 2014-04-04, who consulted the man who was appealed to or requested by Head of Government (Lavvia) on 2014-08-01?
Temporal Query
3V. (e,,, V2, A (Va, Vg,
Rank
Query Answers
Correctness
Answer Type
6L:
Head Government atvia))
TL:
Make an appeal or request
François Hollande
Easy
2014-08-01
Taavi Rnivas
Rasy
comoutt-1
Jyrki Katainen
Hard
2014-04-04
Angela Merkel
X
Head of Govemment (Latvia)
x
Figure 4: Intermediate variable assignments and ranks for example Pe2 query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
6 Conclusion
In this paper, we firstly define a temporal multi-hop logical reasoning task on temporal knowledge graphs. Then we generate three new TKG datasets and propose the Temporal -eature-Logie embed- ding framework, TFLEX, to handle temporal complex queries in datasets. Fuzzy logic is used to guide all FOL transformations the logic part of embeddings. We also further extended fuzzy logic to implement extra temporal operators (Before, After and Between). To the best of our knowledge, TFLEX is the first framework support hop complex reasoning on temporal knowledge graphs. Experiments on benchmark datasets demonstrate the efficacy of the proposed framework in handling different operations in complex queries.
Acknowledgments and Disclosure of Funding
This work is supported by the National Science Foundation of China (Grant No. 62176026) and Beijing Natural Science Foundation (M22009); Engineering Research Center of Information Net- works, Ministry of Education. We would like to express our sincere gratitude our corresponding authors, Prof. Haihong E and Dr. Chengjin Xu, for their guidance, expertise, and unwavering support throughout the project. Furthermore, we would like to thank Fenglong Su, Gengxian Zhou, Tianyi Hu, Ningyuan Li, Mingzhi Sun, and Haoran Luo for their individual contributions and collaboration various aspects of this project. Their expertise and dedication have significantly enriched the final outcome. Additionally, we extend our thanks to the anonymous reviewers for their time and efforts in reviewing our work. Their constructive feedback and comments have been instrumental in improving the overall clarity and rigor of this research.
10

---

Broader Impact
Multi-hop reasoning makes the information stored in TKGs more valuable. With the help of multi-hop reasoning, we can digest more hidden and implicit information in TKGs. will broaden and deepen KG applications in various fields, such as question answering, recommend systems, and information retrieval. It may also bring about risks exposing unexpected personal information on public data.
Finance temporal knowledge graph is a good example to illustrate the broader impact of multi-hop reasoning, In the financial field, the information stored in TKGs is very sensitive. The one-hop reasoning can complete the hidden financial transaction, while the multi-hop reasoning can help to detect the fraud. The After operator could also be used predict the future financial transaction. People may take the advantages of logical reasoning to digest financial factor to obtain excess returns.
Military temporal knowledge graph is another example. With the help of multi-hop logical reasoning. we may predict the future military strategy of country. Besides, with the fuzzy completion of the hidden military information, we can also detect the hidden militarily moves, which may save thousands of soldiers' lives.
The last example is social temporal knowledge graph. The behaviors of people are left and stored in TKGs. With the help of logical reasoning, we may predict a short future of a person. For example, the query may answer the evolution of user profiles: how long may person transfer another role, from student to worker, becoming a parent, or being a grandparent. Tracking the user's identity change can provide super benefits for merchants' advertising. Detecting the role of person is also helpful to provide more personalized services.
However, we should agree that the multi-hop reasoning is still at an extremely early stage, though it may bring about risks. Therefore, we should pay more attention to the security of privacy of users at the mean time when we explore the technology of multi-hop reasoning over TKGs.
TKGs and the
Limitation & Future Work
In this section, we list 4 limitations and talk about the possible future work.
The temporal operators are not enough.. We define three extra temporal operators (Before, After and Between) in query generation. However, there exists more temporal operators in the real world. For example, Allen [49] defined 13 types of temporal relations represented by two intervals, including before/after, during/contains, overlaps/overlappeddbb by, meets/met-by by, starts/started- by, finishes/fnished- by and equal. In the future we would like to promote these temporal operators to TKGs.
Limitation 1:
The temporal embedding could improve.. this paper, the embedding of the timestamp is randomly initialized and finally learned by the model via logical advantages. Such embedding ignores the order of different timestamps. The order property is learned by the After and Before operators, which may be not enough. We recall that in the field NLP, positional embeddings also have order features, which may be used for the construction timestamp embeddings.
Limitation 2:
The query generation is time-consuming. There are 40 predefined query structures in our query generation module. Each query structure has 10k+ queries for training. With the scale of TKGs increasing, the number of queries will also increase, even 40x faster. Therefore, we need to find a more efficient way to generate queries for large scale TKGs.
Limitation 3:
Limitation 4: The MRR and Hits@k k are weak.. The MRR and Hits@ k evaluation metric may not inflect the performance of multi-hop reasoning. We observe that some queries have lots of answers. When the number of answers is larger than k, the MRR and Hits @k will be low, even if all answers are correctly ranked at top. Because the right answers that ranked after k are labeled as false. The Hits@k decreases with the increase number of right answers, which is not reasonable. This disadvantage prevents us from comparing the performance across different datasets. this paper. GDELT much denser, and the count of right answers is larger than the other two datasets. Therefore, the MRR of GDELT lower than the other two datasets. can also be verified that on the average MRR, group avg, is lower than group avg.- The reason is that the number of right answers of group avg, is larger than that of group avg., which can be seen from the statistic of average answer count of queries in Table 10. In the future, there should be more reasonable evaluation metric for multi-hop reasoning.
11

---

References
[1] Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, and Jure Leskovec. Embedding logical queries on knowledge graphs. Advances in Neural Information Processing Systems, 31:2026-2037, 2037, 2018.
[2] H Ren, W Hu, and Leskovec. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. In International Conference on Learning Representations (ICLR), 2020.
[3] Hongyu Ren and Jure Leskovee. Beta embeddings multi-hop logical reasoning in knowledge graphs. Neural Information Processing Systems (NeurIPS), 2020..
[4] Zhangiu Zhang, Jie Wang. Jiajun Chen, Shuiwang Ji, and Feng Wu. Cone: Cone embeddings for multi-hop reasoning knowledge graphs. Advances in Neural Information Processing Systems, 34, 2021.
[5] Nurendra Choudhary, Nikhil Rao, Sumeet Katariya, Karthik Subbian, and Chandan Reddy. Probabilistic entity representation model for reasoning over knowledge graphs. Advances in Neural Information Processing Systems, 34, 2021.
[6] Nurendra Choudhary, Nikhil Rao, Sumeet Katariya, Karthik Subbian, and Chandan K Reddy. Self- supervised hyperboloid representations from logical queries over knowledge graphs. In Proceedings of the Web Conference 2021, pages 1373-1384, 1384, 2021.
[7] Lihui Liu, Boxin Du, Heng Ji, Xiang Zhai, and Hanghang Tong. Neural-answering logical queries knowledge graphs. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1087-1097, 2021.
[8] Xueyuan Lin, Haihong E, Gengxian Zhou, Tianyi Hu, Li Ningyuan, Mingzhi Sun, and Haoran Luo. Flex: Feature-logic embedding framework for complex knowledge graph reasoning, ArXiv, abs/2205.11039 2023. URL ht ttttpp::. /api..... wew.aticscholle......... org/ CorpuaID 248986747.
[9] Erik Arakelyan, Daniel Daza, Pasquale Minervini, and Michael Cachez. Complex query answering with neural link predictors. arXiv preprint 1..iv:2011.004550 2020.
[10] Francois P. S. Laus, Prithviraj Sen, Pavan Kapanipathi, Ryan Riegel, Ndivhuwo Makondo, Thabang Lehese, and Alexander G. Gray. Logic emheddings for complex query answering. ArXiv, abs/2103.00418 2021,
[11] Haitian Sun, Andrew Amold, Tania Bedrax Weiss, Fernando Pereira, and William W Cohen. Faithful embeddings for knowledge base queries. Advances in Neural Information Processing Systems, 33, 2020.
[12] Dinesh Garg, Shajith Ikbal, Santosh K Srivastava, Harit Vishwakarma, Hima Karanam, and L Venkata Subramaniam. Quantum embedding of knowledge for reasoning. Advances in Neural Information Processing Systems, 32:5594-5604, 2019.
[13] Santosh Kumar Srivastava, Dinesh Khandelwal, Dhiraj Madan, Dinesh Garg, Hima Karanam, and L Venkata Subramaniam. Inductive quantum embedding. Advances in Neural Information Processing Systems, 33, 2020.
[14] Lifan Lin and Kun She. Tensor decomposition-bune based temporal knowledge graph embedding. 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), pages 969-975, 2020. URL https: / /aa.pp.........//../ semanticscholas org/ wtw.:opww:::20001111.1
[15] Timothée Lacroix, Guillaume Obozinski, and Nicolas Usunier. Tensor decompositions for temporal knowledge base completion. ArXiv, abs/2004, 04496, 2020. URL ht /ppp. org/ / Corpus 21490000
[16] Pengpeng Shao. Guohua Yang, Dawei Zhang. Jianhua Tao, Feihu Che, and Tong Liu. Tucker decomposition- based temporal knowledge graph completion. Knowl. Based Syst., 238: 1078107741, 2020. URL https: / //api. . semant www.antccccccc....cccoc org Co:
[17] Ali Reza Sadeghian. Mohammadreza Armandpour, Anthony Colas, and Daisy Zhe Wang. Chronor: Rotation based temporal knowledge graph embedding. In AAAÍ Conference on Artificial Intelligence, 2021.
URL https: / ap1. www.nticsccheeooccccccccccccccce Ctt./ooppps 222222666...222......
[18] Julien Leblay and Melisachew Wudage Chekol. Deriving validity time in knowledge graph. Companion Proceedings of the The Web Conference 2018, 2018. URL https: //api. semant iescchllsse org/ CorpusID:138466
12

---

[19] Wessel Radstok and Melisachew Wudage Chekol. Leveraging static models for link prediction in tem- poral knowledge graphs. 2021 IEEE 33rd International Conference Tools with Artificial Intelligence (ICTAI), pages 1034-1041, 2021. URL https:/ /api. www.nticscoooooioooo....o org CorpusID:
235670110.
[20] Chengjin Xu, M. Nayyeri, Fouad Alkhoury, Hamed Shariat Yazdi, and Jens Lehmann. Tero: A time- aware knowledge graph embedding via temporal rotation. In International Conference on Computational Linguistics, 2020. URI. https: /api. semant i cscho ar wttgg//orppip ID:: 2221 24 934.
[21] Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. Hyte: Hyperplane- based temporally aware knowledge graph embedding. In Proceedings of the 2018 conference on empirical methods natural language processing, pages 2001-2011, 1, 2018.
[22] Chengjin Xu, M. Nayyeri, Fouad Alkhoury, Hamed Shariat Yazdi, and Jens Lehmann. Temporal knowledge graph completion based on time series gaussian embedding. In International Workshop on the Semantic Web, 2019, URL https: / api. semant iommannicscholar....
Crrg/oorpsID:: 218900858
[23] Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. Dyernie: Dynamic evolution of riemannian manifold embeddings for temporal knowledge graph completion. ArXiv, 0tss2201103996 2020. URL
https:/ /tpp.ppp..ppp semant iiwctticscccsscc. wwtg httpg//orpusssp 22 223242444.
[24] Rakshit S. Trivedi, Hanjun Dai, Yichen Wang. and Le Song. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In International Conference on Machine Learning, 2017. URL https:
// //api. semant cscholar. org/Corpus ID: 8040343.
[25] Jiapeng Wu, Mengyao Cao, Jackie Chi Kit Cheung, and William L. Hamilton. Temp: Temporal message passing for temporal knowledge graph completion. In Conference Empirical Methods in Natural Language Processing, 2020. URL https: /ttss://aaaa /apl. semant iescholas org/ CorpusID: 222 77069.
[26] Youri Xu, Haihong E, Meina Song, Wenyu Song, Xiaodong Lv, Wang Haotian, and Yang Jinrui. Rtfe: A recursive temporal fact embedding framework for temporal knowledge graph completion. In North American Chapter of the Association for Computational Linguistics, 2020. URL https: /aapi. www.nticscoooee.ee........c org/ CorpusID 222067007.
[27] Siyuan Liao, Shangsong Liang, Zaiqiao Meng, and Qiang Zhang. Learning dynamic embeddings for temporal knowledge graphs. Proceedings of the 14th ACM International Conference on Weh Search and Data Mining, 2021. URL https: / /api. semant icscholar. org/ CorpussI : 23212 6110.
[28] Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. Recurrent event network: Autoregressive structure inference over temporal knowledge graphs. In Conference on Empirical Methods in Natural Language Processing, 2019. URL https: / api. semant wcscholaccclllla org/ CorpusID: 222205878
[29] Zixuan Li. Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo. Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. Temporal knowledge graph reasoning based on evolutional representation learning. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021. URL https: / appi.
semant ccschlaas ar. org/ httg///oruusssu ID: : 233324265.
[30] Zhen Han, Zifeng Ding, Yunpu Ma. Yujia Gu, and Volker Tresp. Learning neural ordinary equations for forecasting future links on temporal knowledge graphs. In Conference Empirical Methods in Natural Language Processing, 2021. URL https:/ attps::/api..... semanticschola aa....oro .org Corpus ID: 237304520.
[31] Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. Explainable subgraph reasoning for forecasting on temporal knowledge graphs. In International Conference on Learning Representations, 2021. URL
https:/ /api. semant escholaz: orgg Saarrg/CorpsssD.io : 614395.
[32] Jaehun Jung. Jinhong Jung, and U Kang. Learning to walk across time for interpretable temporal knowledge graph completion. Proceedings the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021. URL https: api. semanticschi cerccorrrrr.... lar wrt.er////ooooo cttp//orrpiii:33333
[33] Ruijie Wang, Zheng Li, Dachun Sun, Shengzhong Liu, Jinning Li, Bing Yin, and Tarek Abdelzaher. Learning to sample and aggregate: Few-shot reasoning over temporal knowledge graphs. Advances in Newral Information Processing Systems, 35:16863 1886663616677, 2022.
[34] Nilesh Dalvi and Dan Suciu. Efficient query evaluation on probabilistic databases. The VLDB Journal, 16: 523-544, 2007.
13

---

[35] Antoine Bordes, Nicolas Usunier, Alberto Carcia-Durán, Jason Weston, and Oksana Yakhnenko. Translat- ing embeddings for modeling multi-relational data. In NIPS 2013., 2013.
[36] Tingsong Jiang, Tianyu Liu, Tao Ge, Lei Sha, Baobao Chang. Sujian L., and Zhifang Sui. Towards time- aware knowledge graph completion. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Pupers, pages 1715-1724, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL http::/ http:///aaaaattoooooo/oo / org/ wtg///6-11/61
[37] Elizabeth Boschee, Jennifer Lautenschlager, Sean O Brien,, Steve Shellman, James Starz, and Michael Ward. ICEWS Coded Event Data, 2015. URL https: : /doi. org/10.
[38] Kalev Leetaru and Philip Schrodt. Gdelt: Global data on events, location, and tone. ISA Annual Convention, 2013.
[39] Pengpeng Shao, Guohua Yang, Dawei Zhang. Jianhua Tao, Feihu Che, Tong Liu. Tucker decomposition- based temporal knowledge graph completion. Knowl. Based Syst. 238:1078411 2020.
[40] Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. Temporal knowledge graph reasoning based on evolutional representation learning. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021. URL https:
ap1 semant icscholar. org/ CorpusID 233324265.
[41] Bishan Yang. Scott Wen-tau Yih, Xiaodong He. Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of the International Conference on Learning Representations (ICLR) 2015, May 2015.
[42] Seyed Mehran Kazemi and David Poole. Simple embedding for link prediction in knowledge graphs In Neural Information Processing Systems. 2018. URL httpt::/ Corpus DD 3674966.
org/
[43] Yunpu Ma, Volker Tresp, and Erik Daxberger. Embedding models for episodic knowledge graphs J. Web Semant., 59, 2018. URL https::///// / /api. somant i cscholar. org/ htt/CCrrpussDD 54444869.
[44] Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. HyTE:
Hyperplane- temporally aware knowledge graph embedding. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2001-2011, 1, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 1188655/11DD1 1225. URL https:s//ccllathhhheeee
org/
D18 1225.
[45] Alberto García-Durán Sehastijan Dumancic, and Mathias Niepert. Learning sequence encoders for temporal knowledge graph completion. In Conference on Empirical Methods in Natural Language Processing, 2018. URL https: semant icscholas
Ittp:/Corps::::211:1 521834444
[46] Rishab Goel, Seyed Mehran Kazemi, Mareus Brubaker, and Pascal Poupart. Diachronic embedding for temporal knowledge graph completion. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3988-3995, 2020.
[47] Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In International Conference on Neural Information Processing, 2016. URL https: : /api.. semant innaniccchllar org arg/CorpusID:222 26677744..
[48] Cunchao 7hu, Muhao Chen, Changjun Fan, Guangquan Cheng, and Yan Zhan. learning from history: Mod- eling temporal knowledge graphs with sequential gepereeneottion networks. In AAAI Conference on Artifi- cial Intelligence, 2020. URL https: / api. semant inmlaantisccinn icscholar wttp/Corppsss1:22221111222222
[49] James F. Allen. Maintaining knowledge about temporal intervals. Commun. ACM, 26(11):832-843, nov 1983. ISSN 0001-0782. doi: 10.1145/182.358444, URL ht ttt::///coi. org/0/........... /ttg//..11.//1/.....1111 1115/1822. 358434.
[50] Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
14

---

Appendix
More Details about Neural Operators
A.1
Fuzzy Logic
Fuzzy logic is generalization of Boolean logic, which is based on the idea that the truth of statement can be expressed number between 0 and 1. It is mathematical framework for reasoning with imprecise information. In fuzzy logic, there are also logical operators, such as conjunction, disjunction, negation, implication, equivalence, and exclusive or. Most popular fuzzy logic operators are defined as follows (a, b E [0, 1)):
(I) conjunction: a A b = min(a, b). disjunction: a V = max(a, b).
- exclusive (3) implication: a b max(1 a, b). equivalence: a + min(a, b) + max(1 a, 1 b).
(2) negation:
b min(a, b) + (1 a, b).
However, the fuzzy logic operators are defined over real space, not suitable for reasoning over vector space. We expect that the fuzzy logic operators embedding space of knowledge graph is a vector space. In order to cope with tensor computation in neural networks, introduce the fuzzy logic operators over vector space in the following, which is called vector logic.
be executed by matrix computation. Because the
A.2 Vector Logic
Vector logic is an elementary logical model based on matrix algebra. In vector logic, true values are mapped to the vector, and logical operators executed by matrix computation.
Truth Value Vector Space V2. A two-valued vector logic uses two d-dimensional 2 2) column vectors & and 77 to represent true and false in the classic binary logic. The two vectors ø and 77 are real-valued, normally orthogonal to each other, and normalized vectors, i.e,, | 1, || = Truth value vector space generated by V/2 {3, rj., and operations on vectors in truth value space is based on scalar product.
Operators. The basic logical operators are associated with their own matrices by vectors in truth- value vector space. Two common types of operators are monadic and dyadic.
(1) Monadic Operators are functions: 1/2 + V22 Two examples are Identity I ss" + An' and Negation N ris 5o7" such that f5 8, Tn 7., N N ग. Note that the truth tables of and N fulfill the real logical rules identity and negation in classic boolean logic.
(2) Dyadic operators are functions: V/2 @ V/2 - V2. where & denotes Kronecker product. Dyadic operators include conjunction C, disjunction D, implication IMPL, equivalence E, exclusive or XOR, etc. For example, the conjunction between two logical propositions (p A q) is performed by C(ū & 7), where C 5 3) + 7ī @ 7 + n & 8)² + nū @ 77"". can be verified that C(ñ & 8) C π ग. Dyadic operators which correspond to
C(8 3) C logical operations in classic binary logic are defined by its formulation to perform logical operations on truth value vectors. Their associated matrices has d? rows and d columns.
Many-valued Two-dimensional Logic. Many-valued logic is introduced to include uncertainties in the logic vector. Weighting 5 and ń by probabilities, uncertainties are introduced: f f= că 8ū, where e, [0, 1], c + 8 = 1. Besides, operations vectors be simplified to computation the scalar of these vectors. For example, given two vectors i = as + "r, a' + 3' n, we have:
NOT( = NN = (1 - a.88 + an7
(OT(a) = 5" N 1
OR(ī, 17) = (7
+ a' ao' AND(a, = aa'
OR(a, a') = D'"
AND(1,, i7, (7 =C ()
IMPL(ū, 7) -L(7
IMPL(a, a') --"
a(1 a') XOR(a, = 5T X a + a' -2aa
XOR (7, 77 =x(70
15

---

A.3 Neural Dyadic Operators
In this section, we show that all dyadic operators are generated from fuzzy logic in our framework. Below we take union operator for example. We define entity union operator Ue and time union operator 24 according Alignment Rule 4.2 follows:
Fuzzy Logic
Alignment Rule
n
Vc(V 91 ... Van) = a
( f=1
AND
m=1
V. aggi
AND(
Σ B.d.,
(
Alignment Rule
Fuzzy Logic
where AND, OR are the conjunction, disjunction operators in fuzzy logic respectively, a and 3 are attention weights, designed in intersection operators. In the time part of entity union operator U. and the entity part of time union operator U4, we follows Alignment Rule 4.2 to perform intersection. Besides, please aware that each operator owns its MLPs and parameters. These operators do not share parameters with each other.
For queries to intersection/union, the AND and OR be written as:
AND (g..) II
qua
i=1
(R(q.) = Σ 4u.1
Σ
9u19 +
9u999,94 +... + (-1)" Igijj<knn
Σ
II
1=1
IEKjEnn
The equations come from the definition AND and OR vector logic. The cost of OR operator's implementation is not as expensive as it seems to be. In fact, it's O d) complexity, where d is the embedding dimension. We provide the proof in next section.
Other dyadic operators can be generated in the same way. Since listing all operators is boring, we provide the last example. Then, the entity implication operator Le and time implication operator C, are defined as follows:
Fuzzy Logic
Alignment Rule
Lo(Vn Van.)
agKi
n IMPL
Σ B.g.. AND(g..)
L((V Va..)
AND(
Σ
Bigi
IMPL (g
Alignment Rule
Fuzzy Logic
A.4 Theoretical Analysis
To understand why the feature-logic framework works, we have the following proposition, which shows that the designed intersection and union operators obey commutative law of real logical operations. The proofs are presented in next section in Appendix A.5:
Proposition 1. Commutativity: Given Temporal Feature-Logic embedding Vg Vgu. we have
Ie (V Vo.) = IV Va.) and (Vaa, Va.) Ue (V%.. n. Vo.), IVV Va.) (V Van) and (e Vog) uc(Vo Vaa).
A.5 Proofs
In this part, we prove that TFLEX has the property commutativity. Besides, we also show that the computation complexity of OR operation is linear to the number of queries.
16

---

Proof. (Proposition 1) Commutativity:
For the intersection operations, as the calculations of I, and I, are identical, here, we only prove that Ze complies with commutative law. The entity feature part and the time feature part of the result are computed as weighted summation of each query's corresponding parts. Since addition commutative and the attention weights do not concern the order of calculations, both feature parts' calculations are commutative.
Then, we discuss the logic parts. The logic parts only include the AND in fuzzy logic which, essentially, is just the multiplication of each element by the definition provided above. Because multiplication is surely commutative, the calculation of either entity logic part time logic part is commutative. Thus the intersection operation Ie (I,) is commutative.
As for Ue and Ur their feature parts have the same form of weighted summation as the intersection operations do. Thus, the feature parts of both Ue and le comply to commutative law. Also, the time logic part of Me and the enfity logic part of (4, solely concern AND operator which has been proved commutative before. The OR operator, by definition, gives the aggregation of different accumulative parts each of which is commutative itself. Also, the multiplications in each of the summations are commutative. Hence, OR operation invariant to the order of calculations, which finally gives the calculations of entity logic part of Ue and the time logic part Uf are commutative. Then we can naturally affirm that Ue and Ue are commutative as well.
Proof. Additionally, we prove that the cost of OR operator's implementation is not as expensive seems to be. The process of computation can be described as follows:
Algorithm 1: OR
Input: queries {q1...na}
Output: the union of queries
1: Let u = 0.
2: For q In (n)
3: u = u + u
4: EndFor
5: return
As implement OR step by step series of queries, the loop goes n-1 times in total. Assuming the embedding dimension of a query is d, we can have the cost of OR is O(r*d).
B More Details about Experiments
In this section, show more details about experiments. Firstly, we introduce the origin datasets in Section B.I. Then, we describe the details on how we define the query structure and how to sample queries in Section B.2. With the generated queries, construct three datasets and report their statistics in Section B.3. We also show the details of our model implementation in Section B.4 and evaluation protocol in Section B.5. Besides, show experimental data for sensitive analysis in Section B.6. Finally, we present the detail metrics of main results in Section B.7.
B.1 Details of Origin Datasets
To construct the reasoning dataset, we use three datasets of temporal KGs that have become standard benchmarks for TKGC: ICEWS14 [37], ICEWS05-15 [37], and GDELT-500 [38]. The two subsets of ICEWS are generated by Boschee al. [37]: ICEWSI4 14 corresponding to the facts in 2014 and ICEWS05-15 corresponding to the facts between 2005 and 2015. GDELT-500 generated by Leetaru and Schrodt [38] corresponds to the facts from April 1, 2015, to March 31, 2016. The statistics of the dataset are shown in Table 8.
17

---

Table 4: Definition of query structures in static query embeddings [2 [2-4]. Operators defined entity sets only, including Pe, And, Or, and Not. 'e', 'r,, 'n', 'u' denote entity, relation, negation and union, respectively. The braket 'O" denotes relational projection or intersection operation.
Query Name
Query Structure Definition
1p
('e',
2p
(r',(r,)
3p
(f,
2i
((e,(t.),
3i
(C,
ip
(E)
pi
(CC,(T,
2in
3in
(CC
inp
((C))
pin
((E.00,,1)).CC((7711))
pni
(CC,0
2u-DNF
(Ce', ((e,))))))((,))))))))))))))
up-DNF
(CC))
2u-DM
up-DM
Table Basic functions. Qe is entity set and Qt is timestamp set. & the of all entities, is the set of all timestamps and F is the set of triples.
Name
Symbol
Input
Output
And
And
Qe1, Qe2
Qe1 nQe2
Or
Or
Qe1 Qe2 Qe
Qe1 UQez
Not
Not
E Qe
EntityProjection
Pe
Qe, r, Qt
{ols E Qe,t Qt. (s.,,,,)) F}
TimeProjection
Pt
Qei, Qe2
{tss E Qe1, E Qez, (8,1, F
TimeAnd TimeAnd
Qt1. Qtz
Qt1 n nQz2
TimeOr
TimeOr
Qhi, Qt2
Qh1 UQtz
TimeNot
TomeNot
Q
T- QI
before
before
Q
(11 min (Q1)
after
after
Q
(41> max Q1))
B.2 Details of Query Generation
The Design of Query Structure Schema. Previous query embedding methods share the same query structure schema (Table 4) perform complex logical reasoning over static knowledge graph. When it comes to dynamic knowledge graph (TKG), the static schema is not sufficient to capture the temporal reasoning. In this paper, we propose a novel query structure schema for reasoning. The schema is composed of basic functions and query structures. The basic functions are defined in Table The query structures are shown in Table 6 with visualization Figure and Figure 6. The query structures are based on the basic functions: And, Or, Not, EntityProjection. TimeProjection. TimeAnd, TimcOr, TimeNot, Before, and After. Of course, users can define more basic functions query structures to extend the schema to multi-modal KGs, hyper-relation KGs and so on. It is flexible schema overall.
In order to keep similar experiment settings with previous static query embeddings. the query struc- tures are further aggregated into groups, shown Table 7. The groups avg. avg. avgg,c, avgr,c, are for training and testing. Besides, extra groups avg avg avg, are for evaluation and testing only.
In implementation, define the basic functions and query structures functions in python, which are named Complex Query Function (CQF). There are lots of advantages of using python functions to define query structures. Firstly, the functions are easy to understand and debug. Secondly, the functions are easy to be reused in the dataset-sampling process and model-training process. Thirdly,
18

---

Table 6: Definition query structures temporal query embeddings. Operators on entity sets include Pe, And, Or, and Not. Operators on timestamp sets include Pt, TimeAnd, TimeOr, TimeNot, after, and before. The prefix "a" and "b" represent "after" and "before" respectively. The prefix "s" or "o" mean that the sub-query is a subject query or object query respectively.
Type
Query Name
Query Structure Definition
Pe
re
Pe2
ro, 12)
entity multi-hop
Pe3
Pe(Pe(PessI, rl, 12, 12, 12), 13)
e2i
And(Pa(sI, tl), Pe(s2, 12, 12))
e3i
tnd(Pe(sl,rI,,)), (), Pe(s2, r2, 12), Pe(e3, r3, (3))
e2i N
And(Pe(sI, tl), Not(Pe(s2, (2))) And(Pe(sI,rI. (I), Pe(s2,
e3LN
Not(Pe(s3, 13, (3)))
entity not
e2i
Pe(And(Po(s1, r1, tl), Not(Pe(s2, r2, (2))), r3, rn tI), r2. (2), Not(Pe(s2, 13, ()))
e2i PeN
e2i_NPe
And(Noc(Pa(P<(s1,r) rl, tl), r2, 12)), Pe(s2, 13, tă))
e2u
Or(Pess1, Pe(s,, r2, (2))
entity union
Pe Pe_e2u e2u
Pe(On(PessI, Pe(s2, r2, (2)). r3, (3)
Pt
P,
aPt
after(Pu(s, n)
bPt
before(P(s.....))
Pe
Pe(sl, 12, ol))
time multi-hop
Pt sPe Pt
Pl(Pe(sl, Pt(22, r2, o3), o2)
Pt_oPe Pt
Pt(s1, rl, Pe(s2, 12, Pt(s3, ol)))
12i
rimeAnd(P(ssI,r,, ol), Pl(s2, r2, o2))
13i
TimcAnd(P(5l,,ll..om ol), Pt(s2, r2, o2), P3(s3, 03))
121 N
TimeAAn(P((sI,rI..00) ol), TimeNou(P(s2,,2..000
ti_
rimeAnd(P(ssl,rl, ol), Pt(s2, r2, o2), TimeNot r3, o3)))
time not
Pe N
rl. TimeA 12, (I), r3, ol). TimeNot(Pt(33, 14, o2))))
_Pt
TineAAnddTTnneeWooPPPeceeil.l rl, tl), r2, ol)), Pt(s2, r3, o2))
12i PIN
fimcAnn//P((esss.r1..)))... ol), TimeNot(Pt(2,, r3, o2))
12u
TimeOr(P(ss,rrl,oo)) Pt(s2, r2, o2))
time union
12u
Pt(s3, r3, o2)))
between
TimeAnndiafier(Pisss,.. r1, ol)), hcfore(Pi(22, r2, o2)))
e2i Pe
1l, (2), Pe(s2, (2, r3, (3))
Pe_e2i
rl, (2), r3, t3)
12i Pe
TimeAAud(P(PP(sI)),,,) (1), r2, o1), Pt(s2, r3, o2))
Pe t2i
r2,5), r2, ol, s3, r3, o2))
Pe aPt
affer ol)))
hybrid multi-hop
bPt
be ol)))
Pe at2i
after(22222 r2, 83, 13, o2))
Pe bt2i
Pe(sl, beforee((22ss22
o2))
Pt_sPe
Pt(Pe(sl, tl), ol)
Pt oPe
Pl(sl, rl. Pe(s2,
Pt_se2i
Pt(e2i(s), rl, tl, s2, 12, o3)
Pt Pcoc2i
Pi(sI, rl, e2i(22, (3.
19

---

Pe
23 NPc
Not
PeN
And
B
Pd
Pe_eeUPe_PY
fs
Not
PL
4
221
Aad
Aad
And
4
<2a
GL Pe
Or
And
Pe_e2n
Pa
6
Or
Aad
75
PL IPc
Peldii
TimeAnd
PL
Figure 5: Visualization of temporal query structures answering entity sets. These structures are basic functions that can be used to construct more complex temporal query structures.
20

---

12_N
TimeAnd
TimrAnd
BUUN
TimeAnd
Ts
TimeNot
12
cs
TimeOr
PLki
Pe
And
R
Timetr
Pe_aPI
Afler
Po.bPt
F
TL
(N_NPt
F
Before
F2
TimrAnd
@J PIN
Pc. bx2
Time-And
TimeNot
C
CS
TimeAnd
Before
Pe_BLPP NPt
between
After
TimeAnd
TimeNat
Refore
Figure 6: Visualization of temporal query structures answering timestamp sets. These structures basic functions that can be used to construct more complex temporal query structures.
21

---

Table 7: The query structures aggregated into groups. The groups inspired by the experiment settings in static query embeddings [1-4]. The groups a,g,, avg,, avg..c avgr.c, are for training and testing. Besides, extra groups a g a for evaluation and testing only.
1p
2p
3p
2i
3i
av a
Pe
Pe2
Pe3
e2i
e3i
Pt, aPt, bPt
Pe Pt
Pt sPe Pt, Pt oPe Pt
12i
t3i
2in
3in
inp
pin
pni
av
e2i N
e3i N
Pe e2i N
e2i PeN
e2i NPe
aV
t2i _ N
t3i
Pe [2i_
12i
PtN
12i NPt
2u
avg(u4)
e2u
Pe e2u
12u
Pe_12u
pi
avg.
e2i_Pe Pe
Pe_e2i
Pe _Pt
Pe Pet
t2i_Pe Pe
Pe_t2i
Pe at2i
Pe bt2i
Pt sPe
Pt_oPe
between
Pt_sei
Pt_oe2i
the functions are easy to be extended to support more complex query structures and reimplement the existing query embedding methods.
As is introduced in Section 4, we use the computation graph to represent the reasoning process. To get the computation graph of CQF, Syntax Tree (AST), which is a more friendly readable subset of computation graph. In this way, executing the computation graph is equivalent executing the CQF in python interpreter. Since have the god privileged access to the AST, we can modify the interpreter to support various dynamic reasoning processes. Below we show how unify (1) the reasoning of ground-truth, (2) the sampling of anchors, and (3) the reasoning of embedding aabeedding-baed methods by modifying the python interpreter.
the python interpreter to parse the function to Abstract
Ground- Truth Reasoning Interpreter.
In the reasoning of ground-truth, we need get all real answers of a query under given TKG. To achieve this, the first we need to do is to implement the basic functions. We wrap the python built-in set to QuerySet class to store entities and timestamps. The basic functions are implemented as lambda function in python, with input and output QuerySet. Then, the basic functions are registered as symbols to the python interpreter. When executing the CQF, the python interpreter will call the corresponding basic functions get the final answer. The answers are finally generated from subgraph matching, which depends on the ground truth TKG. We show pseudocode of the ground-truth reasoning interpreter in Figure 7.
Anchors Sampling Interpreter.
The anchor, which may be entity, relation or timestamp, is the input argument of the CQF. The aim of the anchors sampling interpreter is to get the possible anchors and answers of query under given TKG. For the sampled anchors, we expect the answer set not empty. Since the answers could be obtained from the ground-truth reasoning interpreter, we only focus on the sampling of anchors in the anchors sampling interpreter.
Given the standard split of edges into training (Firain, validation (Fuati) and test (Fres) sets, we append inverse relations and double the number of edges in the graph. Then we create three graph: Guain (V, R, T, Funin ). Gvalid (V, R, T, Fu + Fvalid J. = (V, R, T F + Fvalid + Fress. Given a query q. let [q] iqin [[]] valid and [4]]eat denote a set of answers (entities or timestamps) obtained by subgraph matching of q Guain. Gralia and Gues For each query q, the reasoning process starts from anchor nodes and computes the final answer set via subgraph matching.
For the basic function Pe and Pt, we simply use all the triples and extract the entities and timestamps as the anchors. These anchors are supposed to have no empty answers under Pe and Pt. However, when it comes to some hard queries, such as e2i and e3i, the random sampled anchors may have empty answers. Because the TKG sparse and incomplete, the intersection of two query sets has high probability to be empty. The empty answers are meaningless for model- model-training and damage to
22

---

1. predefined query airazfures and the Interpreier CamplestueryPucctiosss
"Po2": "daf Pe2(e1, rl, tl, r2, t2): return Pa(Po(al, r1, t1), r2, t2)", #
"Pa3": 'daf Pe3(a1,
r2, 13, t3): return Pa(Pa(Pa(et, "aPe"; 'def aP, r, o): return Timeffuer(PS(e, o))", A for after "bPt": deef bPt(e, F, o); return limeBefore(Pt(e, o))", for before
t1), r2, 12), r3, ta)",
class Interpreter:
def ___lnf syabels: Dict[atr, Anyl):
self aayrbll synbola
for k, CDF in Comples@ueryuuttioon iteess):
(CQF)
def (aelf, lime: str) -> Any: return ast ast. (ral)
# GroundTrush Rensowing Interpreler
quarySat sat
EatitySet TineSet set class brundetseeeenctgeeeeeprrrrrreeee def __init__. TKG):
Pe defaultdiet ( (uubddasdefaelllliee deeeudddeeeeeeellleeeeteeee Pt defsulvdict (lanbaa (fankddsseffuuuudddeeeee))) for s, r, o, in IXG.F: F 34 the set of all facts Pe(e] add)
Pt[c] add())
add(t)
symbols =<
"Pa": lanbda (qa, r, qt): Entitysaerraduuce [Pa[a] [v] [t] for t in product (qa, qt}]))
"Pu"; lanbda (us, r, 90): (reducc)
[Pt [e] [o] for 8, o in product qo)]))
"And": lanbda (ql, 92): (ntitySet(q1 92)
"Or': lanbda (qt, Q2): BatitySet(q] q2)
"Kat"; larhda (qe): Intitysat.(TKG. qe)
of all entseses
"Tinetnd"; lanbda (g1, g2): (imoSet & g2)
"Tinaür": lanbda (ql, 92): (imeSar q2)
"TineNot': lanbda (qt): TimeSet qt) # 7 the set of all Esnestamps
"TineBefore": lanbda (qt): TimeSet((t for in TSG.T if t ain(qt)])
"Tinedfter": lanbda (qt): (imeSet([t
in TKG.T if msx(qt)])
super Sn.....ttt... (aymbols)
TXG.P [AAngela Nerkel, make weast, Ohina, 2010-07-16)3 (TKG)
anever p. ("ae(" Angela () "nake a visie', ('2010-07-18 EntitySat( ('China')
Figure 7: Python-style pseudocode of ground-truth reasoning interpreter.
the performance of data-sampling process. To solve this problem, we use the following two strategies accelerate the sampling.
(1) Inverse Sampling: We randomly sample an entity as objective, denoted ca- Then we sample subjective entity cx Cag relation F1, r2 and timestamp 1 f2, according to the fact (e., 1, Co, &i) and (Eez, 72, co 12). The sampled anchors are (e.,, T1, 1 41; Eags T2, f2). These anchors under e2i have the answer 6, least, which asserts that the answer set is not empty. Such sampling method that samples the answer first and then samples the anchors called Inverse Sampling.
(2) Bi-directional Sampling: For long multi- hop queries, such as Pe2, the complexity of random sampling is O(N21., where is the entity count and L the length of the query. Pe2 has L 2. contains an anchor-sampling complexity of O(N²) and a ground -ouud reasoning (to get answers) complexity of O ONN The origin sampling to sample one subjective entity, get the objective as answers next subjective anchors, again get the next objective answers. The final answers recursively depend on the initial subjective entity, leading to low performance. To accelerate, we sample an sample in two directions. One is forward, em
entity at the middle of AST, denoted c. Then as the subjective entity, to sample relation r'2 and timestamp l2. The other backward, 6m as the objective entity, to sample subjective anchor et, relation and timestamp 41 The sampled anchors are (e1, F1, f1: T2, t2). The final answer set is asserted not empty, because it at least contains all the answers of Pe(em. T2, tz). Be aware that the two directions independent, which in parallel. Such sampling method that samples the answer in two directions is called Bi-directional Sampling, which can reduce the computation complexity to ONN composed of a sampling complexity of O 4) and ground-truth reasoning complexity of (
be sampled
23

---

python run, reessoning tna, interpreter-Py
your dataseti data /data//FLEE/aaaa" use enbeddingg waapddnnnnrrretigggeooooo reasoning interpreter(
(TTFLE, ("TFLEX, dine0o garma15", device="cuda:1"):
data alreedy prepared, ing
step
0.0701355565907799
interpreter
you: samplei taastttakkaaaeeee namee""21",
e2ilel,
t2):
Pnde,, ri, Pelež, r2,
sample e2iiel, r1, t1, +2. r2. t2) data:
quer les: [29. 155, 47, 29, 372, 471,
answer: (2669, 1751), hard answer: (2083)
bot: [1 [229 155, 47, 29, 372. 47). [2069. 1751), (2083, 2669, 1751)]]
you: enb eleentity ceantity__oooeeeee token(291 291; enb_ wwwrelattiooo token 155]; emb, pol.ttstiissttnt taken(
you: emb #zvent kit_
tokent ttk_n
291; enb enb_r r2treltt ton tokeni 372); tnrrt [oken(47])
you: enb 41 Peçemb e1, emb, r1, (m)_tt)
you: ene Q2 Pel exb c2, enb r2. eno_ta
your eno
Andi emb q1, enb q2)
you: enbedd ing_iingnnnnn enw.eenwwweeee eetrrreettteee eeeeenttient emb topke3)
bot: III
7.56644860011
'Head Government (D) i)out 6-48099465533965551 ( ('Ethlopla", 4,45724291202392611
your use groundtruth,
wewprruuuuhrreooiitgooo reasoning, interpreter()
bot: ing groundrruth
your groundtruth, weanetruthannweer.o
groundtruth (atrrrthhhssreee1] groundtruth (22033)
you:
the
the set:
Figure 8: The screenshot of a real running interpreter.
Embedding Reasoning Interpreter.
On the contrary of the ground-truth reasoning interpreter which has 'set' as the input and output of the CQFs, the embedding reasoning interpreter, the input and output CQFs are 'embedding vectors' The embedding- -bated method for reasoning over TKG is called Temporal Query Embedding. The computation complexity of the embedding reasoning interpreter OLL + N). where L is the length of the query and N is the entity count. The complexity consists of a muery-emeddding complexity of (L) and a scoring-to-answe complexity of O(N). The embedding reasoning is much faster than the ground-truth reasoning, which has computation complexity of O(N4).
To implement the embedding reasoning interpreter, we just need replace the basic functions with neural networks. The symbols dict to pass to the interpreter is
"Pe": Pe: "Pt": Pt,
"And": Ie, "Or": Ue "Not": Ne.
(8)
"TimeAnd": I, "TimeOr": Us "TimeNot": Nr,
"TimeAfter": Ar, "TimeBefore": B.
The embedding vectors are learned as is introduced in Section 4. Unlike the ground-truth reasoning, the embedding reasoning is fuzzy. The output of the embedding reasoning interpreter is an embedding vector, which is fuzzy set representation. To get the final answer set, we need to use the distance function (in Section 4.3) to score candidate answers. this way, the embedding vector is converted to the final answer set. The final answers are given in the ranking list, where each answer is followed by its distance to the query. Usually the top-k answers are accepted as the final answers.
Note that the interpreter is flexible, and easy to implement and extend. In order to reproduce static query embeddings [2-4] over dynamic knowledge graphs, the only thing to do is implement the symbols dict and the distance function.
We hope that the design of interpreter is helpful for the future research of temporal query embedding. Screenshot.
Additionally, we show the screenshot a real running interpreter in Figure 8. In the screenshot, we randomly select example of e2i query. Then, use the embedding reasoning interpreter get the final answer set step by step. The final answer set is shown in the ranking list, where each answer is followed by its similarity score the query. Finally, the bot correctly predicts the hard answer which only exists in the test set!
B.3 Details of Generated Datasets
Finally, we generate three datasets for the task of temporal complex reasoning using the process in Section B.2, The statistics of the generated datasets are listed below. Table 9 show the count of query

---

Table 8: Statistics ICEWS14, ICEWS05-15, and GDELT-500.
Dataset
Entities
Relations
Timestamps
[Taining]]
Validation
Test
Total Edges
ICEWS14
7,128
230
365
72,826
8,941
8,963
90,730
ICEWS0S-15
10,488
251
4,017
368,962
46,275
46,092
479,329
GDELT-500
20
366
2,735,685
341,961
341,961
3,419,607
Table 9: Queries count for each dataset.
ICEWS14
ICEWS05-15
GDELT-500
Query Name
Train
Validate
Test
Train
Validate
Test
Train
Validate
Test
Pe
66783
8837
8848
344042
45829
45644
1115102
273842
273432
Pe2
72826
3482
4037
368962
10000
10000
2215309
10000
10000
Pe3
72826
3492
4083
368962
10000
10000
2215309
10000
e2i
72826
3305
3655
368962
10000
10000
2215309
10000
10000
e3i
72826
2966
3023
368962
10000
10000
2215309
10000
10000
Pt
42690
7331
7419
142771
28795
28752
687326
199780
199419
aPt
13234
4411
4411
68262
10000
10000
221530
10000
10000
bPt
13234
4411
4411
68262
10000
10000
221530
10000
10000
Pe
7282
3385
3638
36896
10000
10000
221530
10000
10000
Pte
13234
5541
68262
10000
10000
221530
10000
10000
oP Pt
13234
5480
6242
68262
10000
(8000
221530
10000
10000
121
72826
5112
6631
368962
10000
10000
2215309
10000
10000
13i
72826
3094
3296
368962
10000
10000
2215309
10000
10000
e2i N
7282
2949
2975
36896
0000
10000
221530
10000
10000
e3i_N
7282
2913
2914
36896
00000
1000
221530
10000
10000
Pe e2i eiiN
7282
2968
3012
36896
10000
10000
221530
10000
10000
eli PeN
7282
2971
3031
36896
10000
10000
221530
10000
10000
NPe
7282
3061
3192
36896
10000
10000
221530
10000
10000
12i
7282
3135
3328
36896
10000
10000
221530
10000
10000
13i N
7282
2924
2944
36896
10000
10000
221530
10000
100000
Pe Pe_1Zi_N
7282
3031
3127
36896
10000
10000
221530
10000
100000
12i PIN
7282
3300
3609
36896
10000
I 10000
221530
10000
10000
12LNPt
7282
4873
5464
36896
10000
10000
221530
10000
10000
e2u
2913
10000
10000
10000
10000
e2u
2913
10000
10000
10000
10000
12u
2913
2913
10000
10000
10000
10000
Pe_12u
2913
2913
10000
10000
10000
10000
between
7282
2913
2913
36896
10000
I 0000
221530
10000
10000
e2i_Pe Pe
2913
2913
10000
1 10000
10000
10000
Pe e2i
2913
2913
10000
10000
10000
10000
t2i_Pe
2913
2913
10000
1000
10000
10000
1000
10000
10000
aPt
7282
4134
4733
68262
10000
1000
221530
10000
10000
Pe_at2i
7282
4607
5338
36896
10000
10000
221530
10000
10000
Pt_
7282
4976
5608
36896
10000
10000
221530
10000
10000
se2i
7282
3226
3466
36896
10000
10000
221530
10000
10000
Pe bPt
7282
3970
4565
36896
10000
10000
221530
10000
10000
bt2b
7282
4583
5386
36896
10000
10000
221530
10000
10000
oPe
7282
3321
3621
36896
10000
10000
221530
10000
10000
Pt_oe2i
7282
3236
3485
36896
10000
10000
221530
10000
10000
structures in the split of training, validation and testing. The average answers count of each query structure reported in Table 10. The number of queries for each dataset is shown in Table 11.
B.4 Implementation Details
We use the Embedding Reasoning Interpreter as introduced Appendix B.2. implement our model with PyTorch and use Adam [50] as a gradient optimizer. For each experiment, we use one GTX 1080 graphic card. The hyperparameters for each dataset are shown in Table 12. Note that we do not perform hyperparameter tuning for each dataset. Instead, the same hyperparameters
25

---

Table 10: Average answers count for each dataset. All numbers are rounded to two decimal places.
ICEWS14
ICEWS05-15
GDELT-500
Query Name
Train
Validate
Test
Train
Validate
Test
Train
Validate
Test
Pe
1.09
1.01
1.01
1.07
1.01
1.01
2.07
1.21
1.21
Pe2
1.03
2.19
2.23
1.02
2.15
2.19
2.61
6.51
6.13
Pe3
1.04
2.25
2.29
1.02
2.18
2.21
5.11
10.86
10.70
e21
1.02
2.76
2.84
1.01
2.36
2.52
1.05
2.30
2.32
e3i
1.00
1.57
1.59
1.00
1.26
1.26
1.00
1.20
1.35
Pt
1.71
1.22
1.21
2.58
1.61
1.60
3.36
1.66
1.66
aPt
177.99
176.09
175.89
2022.16
2003.85
1998.71
156.48
155.38
153.41
bPI
181.20
179.88
179.26
1929.98
1923.75
1919.83
160.38
159.29
157.42
Pe_P
1.58
7.90
8.62
2.84
18.11
20.63
26.56
42.54
41.33
Pt_ Pt
1.79
7.26
7.47
2.49
13.51
10.86
4.92
14.13
12.80
Pt Pt_oPe
1.75
7.27
7.48
2.55
13.01
14.34
4.62
14.47
12.90
t2i
1.19
6.29
6.38
3.07
29.45
25.61
1.97
8.98
7.76
13i
101
2.88
3.14
1.08
10.03
10.22
1.06
3.79
3.52
1.02
2.14
1.01
2.05
4.58
e3LN
1.00
1.00
1.37
e2LN
1.02
2.16
8.54
8.12
PeN
1.02
2.17
8.36
NPe
1.12
2.87
1
1.15
3.31
3.44
1.21
4.06
4.20
2.91
8.78
7.56
B3 N
1.00
1.02
1.03
1.01
1.02
1.02
1.15
3.19
3.20
Pe 12i N
1.08
2.59
2.70
1.08
2.47
2.62
4.10
12.02
11.37
12i PLN
1.41
5.22
5.47
1.70
8.10
8.11
4.56
12.56
11.32
t2i NPE
8.14
25.96
26.23
66.99
154.01
147.34
17.58
35.60
32.22
e2u
3.12
3.17
2.38
2.40
5.04
5.41
Pe c2u
2.38
2.44
1.24
1.25
9.39
10.78
l2u
4.35
4.53
5.57
5.92
9.70
10.51
Pe 12u
2.72
2.83
1.24
1.28
9.90
11.27
between
122.61
120.94
120.27
1407.87
1410.39
1404.76
214.16
210.99
207.85
e2i_e
1.00
1.10
Pe e2i
1.32
5.49
1.01
1.44
Pe_t2i
1.14
1.16
1.07
1.08
2.01
2.20
Pe_aPt
4.67
16.73
16.50
18.68
43.80
46.23
49.31
66.21
68.88
Pe_at2i
7.26
22.63
21.98
30.40
60.03
53.18
88.77
101.60
101.88
Pt_
8.65
28.86
29.22
71.51
162.36
155.46
27.55
45.83
43.73
Pt se2i
1.31
5.72
6.19
1.37
9.00
9.30
2.76
8.72
7.66
Pe bPt
4.53
17.07
16.80
18.70
45.81
48.23
67.67
84.79
83.00
Pe_bt
7.27
21.92
21.23
30.31
61.59
64.98
88.80
100.64
100.67
Pt_oPe
1,41
5.23
5.46
1.68
8.36
8.21
3.84
11.31
10.06
Pt_oeii
1.32
6.51
7.00
1.44
10.49
10.89
2.55
8.17
7.27
Table 11: Number of queries. Pe represents query answering (s.r, Pt represents query answering (s,r,o,?). QoF. represents the query of entities (except Pe). Qo' represents query of timestamps (except Pt). nlp represents query that Pe or Pt.
Training
Validation
Test
Dataset
Pt
QoE
QoT
Pe
Pt
nlp
Pe
Pt
nlp
ICEWS14
273.710
27.371
59,078
8,000
66,990
66,990
10,000
66,990
66,990
10,000
1CEWS05-15
149,689
14,968
20,094
5,000
66,990
22,804
10,000
66,990
66,990
10,000
GDELT-500
107,982
10,798
16,910
4,000
66,990
17,021
10,000
66,990
66,990
10,000
26

---

Table 12: Hyperparameters on each dataset. d is the embedding dimension, is the batch size, the negative sampling size, 7 is the parameter in the loss function, m is the maximum training step, and is the learning rate.
Dataset
d
?
1
ICEWS14 14
800
512
128
15
300k
10
ICEWS05-15
800
512
128
30
300k
1 x
10
GDELT-500
800
512
128
30
300k
x 10
37.0
35
ICEWS14
ICEWS14
36.5
MRR
MRR
35.5
35.0
34.5
400
800
1000
10
20
30
40
embedding dimension
the margin in loss function
(a) The impact of embedding dimension MRR.
(b) The impact of the margin 7 MRR.
Figure 9: The impact of embedding dimensionality d and the margin on MRR.
found in sensitive analysis on ICEWS14 (Section B.6) for all datasets. Besides, the source code is available at Gitbub f
We cite these projects $ $ and thank them for their great contributions!
B.5
Evaluation
Given the standard split of edges into training (Firain), validation (Fvatia) and test (Fres) sets, we append inverse relations and double the number of edges in the graph. Then we create three graph:
Irain = (V, R, Funin, Gvalid = (V, R, Fvaiia
= (V, R, T, Foriin + F + Fres)- Given query q. let [g] [g]] valid. and [9]]est denote set of answers (entities or timestamps) obtained by subgraph matching of q on Guain Gvalid and Gees- For a test query q and its non-trivial answer set [g]t Iq]lealid: denote the rank of each answer v rank(u,). The mean reciprocal rank (MRR) is MRR(g) = ||l|| Σn.ev rank(es) and Hits at K (Hits @KissK) K) is Hits@ Hits@K(g))
[1. nsk
[To En,r ((:)nk((s)), where f(n) = 0, n> K
B.6 Experiment Data for Sensitive Analysis
We conduct experiments on ICEWS14 to analyze the impact of the embedding dimensionality d and the margin 7 on the performance of TFLEX.
Impacts of Embedding Dimensionality Our experiments indicate that the selection of embed- ding dimension has substantial influence on the effectiveness of TFLEX. We train TFLEX with (300, 500, 600, 700, 800, 900, 1000} and plot results based on the
embedding dimension validation set, as shown in Figure 9a. With the increase of d, the model performance (indicated by 800. Therefore, we assign 800 as the best setting.
MRR) increases rapidly and reaches its top
TFLEX: http:s/s//gttaa..cc//h/eeeaaaaaa stanford/ Ketrr///eeaaoninnngg
*GQE, Query2hox, BetaE: htttt//ttttttt////tttti.bb.oomm http://hhh..ccoooooaaaaaaaaaarea
27

---

Table 13: The mean values and standard variances of TFLEX's MRR results ICEWS14.
Pe
Pe2
Pe3
e2i
e31
48.21
37.27
33.53
69.24
95.70
#0.37
#0.32
+1142
40.46
Pt
aPt
bPt
Pe Pt
Pt sPe
Pt
Pt oPe Pt
12i
13i
20.87
3.00
2.96
14.27
9.57
9.46
27.49
$2.84
20.34
20.50
#0057
10066
20.51
40.63
c2i N
Pe
PeN
NPe
45.55
99.56
34.74
35.63
38.61
+0.12
#0.50
#0.44
#0.4
#0.20
12i_N
t3i_N
Pe_12i_N
12i PtN
12 NPt
25.38
98.91
34.05
11.42
12.07
20013
1:040
e2u
Pe e2u
29.20
42.28
#0.12
-0.43
t2u
Pe_tZu
30.73
21.74
10.54
110.3
cii Pe 98.86
Pe e2i 36.77
Pe aPt 8.66
Pe bPt 9,74
Pe at2i 7.90
Pe b12i 7.78
10.3
10042
10.61
1033
10.39
1.039
12i Pe 96.62
Pe 12i 64.50
PPe 4.32
Pt oPe 10.58
Pt se2i
Pt oe
between
8.20
7.95
2.57
10.12
Impacts of Parameter We train TFLEX with parameter y E {5, 10, 15, 20, 30, and plot MRR results in Figure 9b. Too small and too large y both get bad results, while 7 15 in the middle is the best. Therefore, we choose 15.
Error Bars of Main Results order to evaluate how stable the performance of TFLEX is, we five times with random seeds {1, 10, 100, 1000, 10000} and report the error bars of these results. Table 13 shows the error bar of TFLEX's MRR results on ICEWS14. Overall, the standard variances small, which demonstrates that the performance of TFLEX is stable.
B.7 Detail Metrics of Main Results
We report the MRR results on all query structures for each dataset in Table 14 (ICEWS14), Table 15 (ICEWS05-15) and Table 16 (GDELT-500). Query2box [2] can only answer queries without negation. BetaE [3] and ConE [4] can answer queries with negation, except temporal logic. ConE(temporal) and X(ConE) the variants of ConE, aiming exploring the right way to promote the static query embeddings to temporal ones in Section C. FLEX, X(ConE), X-IF, X-logic are the variants of TFLEX, introduced in the main body the paper.
C Exploration of Static Query Embeddings Toward Dynamic
In this section, we present more research questions that aim to explore the right way to promote the static query embeddings to temporal ones. Since ConE [4] is a strong baseline for complex logical reasoning over static knowledge graphs, we conduct exploration experiments by introducing variants of ConE. We present the overall results for each dataset in Table 17, detailed results in Table 14 (ICEWS14), Table 15 (ICEWS05-15) and Table 16 (GDELT-500). Overall, we are interested in the following research questions:
28

---

Table 14: MRR results on ICEWS14. The group aven. wraps to two rows. AVG denotes average performance under all query types.
Model
avg.
Pe
Pe2
Pe3
e21
e31
Query2box
25.06
25.81
17.29
12.97
16.00
53.25
BctaE
37.19
39.52
23.77
17.50
23.77
81,38
ConE
41.94
42.55
30.90
24.78
26.52
84.93
CoeEotemporal)
42.23
24.99
28.98
84.17
FLEX
43.67
45.25
33.07
27.22
27.62
85.18
TFLEX
56.79
48.21
37.27
33.53
69.24
95.70
X(ConE)
40.93
40.58
28.84
24.31
30.93
79.98
X-IF
56.89
37.51
32.61
71.25
94.47
X-logic
37.42
31.23
73.11
93.45
avEr
aPt
bPt
Pe_P Pt
Pt sPe
Pt Pt_oPe__
(2i
13i
TELEX
17.56
20.87
3,00
2.96
14.27
9.57
9.46
27.49
52.84
X(ConE)
16.41
19.45
3.06
3.01
13.96
8.79
8.67
25.01
49.36
X-IF
18.77
21.94
3.04
2.99
16.69
10.73
10.44
29.66
54.71
X-logic
18.03
21.33
3.10
3.05
15.76
9.85
9.94
28.57
52.63
av8c.c.
e2i_N
e3i
Pe e2i
BetaF:
36.69
30.44
87.93
19.94
22.80
22.36
ConE
44.88
40.98
99.12
29.22
29.84 84.84
25.22
ConExtemporal)
44.94
41.45
29.48
30.71
24.09
FLEX
44.41
38.30
31.18
30.49
22.84
TFLEX
50.82
45.55
99.56
34.74
35.63
38.61
X(ConE)
42.15
39.10
96.25
26.92
27.59
20.89
X-IF
49.78
42.31
99.31
34.89
34.77
37.61
X-logic
51.17
44.65
99.25
35.30
36.31
40.33
avgr.c,
12i
t3L
12i PtN
NPt
TFLEX
36.37
25.38
98.91
34.05
11.42
12.07
X(ConE)
35.24
26.90
30,30
9.44
10.75
X-IF
37.73
26.40
98.82
37.85
12.38
13.23
X-logic
36.39
25.68
98.69
33.30
11.34
12.93
e2u
Pe e2u
BetaE
19.95
18.61
21.28
ConE
26.47
21.84
31.11
ConE(temporal)
27.63
23.01
32,26
FLEX
29.25
24.05
34.46
TFLEX
35.74
29.20
42.28
X(ConE)
25.46
20.26
30.67
X-IF
34.48
30.04
38.93
X-logic
34.68
29.44
39.92
avE(u.]
12u
Pe. 12u
TFLEX
26.24
30.73
21.74
X(ConE)
24.07
27.63
20.51
X-IF
28.04
33.91
22.16
X-logic
26.36
31.21
21.52
avg.
e2i
Pe. e21
PPt
Pe Pe_at21
bt2i
TFLEX
28.03
98.86 86
36.77
8.66
9.74
7.90
7.78
X(ConE)
26.65
87.31
28.72
11.01
10.40
11.21
11.27
X-1F
29.31
98.87
36.00
10.04
10.21
8,30
8.06
X-logic
28.61
97.64
36.96
10.09
10.27
8.76
8.93
AVG
[2 Pe
P2i
Pt PUsPe
Pt oPe
Pt_seii
Pt Pt_ocii oc2i
between
TFLEX
35,93
96.62
64.50
4.32
10.58
8.20
7.95
2.57
X(ConE)
30.13
95.40
63.04
3.77
8.82
6.44
5.76
3.32
X-IF
36.43
97.24
70.32
4.88
10.73
12.24
11.55
2.55
X-logie
35.98
96.35
64.31
4.97
10.35
10.59
10.35
2.34
29

---

Table 15: MRR results ICEWS05-15. The group avgr wraps to two performance under all query types.
AVG denotes average
Model
ave.
Pe
Pe2
Pe3
c2i
c3i
Query2box
24.00
25.94
16.62
14.22
17.68
45.52
Betal
31.33
35.78
21.47
18.18
18.10
63.11
ConE
40.93
42.67
29.39
24.79
26.15
81.64
ConE(temporal)
40.74
42.64
29.30
24.76
25.35
81.65
FLEX
38.96
41.60
29.80
24.58
24.37
74.46
TFLEX
48.99
43.04
36.28
33.89
90.60
X(ConE)
36.29
39,90
26.62
22.85
22.52
69.56
X-IF
49.90
43.07
36.87
34.96
40.85
93.76
X-Iogic
44.80
40.57
31.47
30.67
35.80
85.47
avg.
Pt
aPt
bPt
Pe_Pt
Pt_ PP_sPe_PP Pt_oPe_P
t2i
TFLEX
4.39
10.62
0.38
0.38
7.29
2.63
1.83
3.85
8.15
X(ConE)
4.41
10,98
0.38
0.38
6.75
2.16
1.63
4.22
8.81
10.71
0.38
0.38
7.40
1.86
3,80
8.16
X-logic
3.29
8.57
0.38
0.39
6.37
1.69
1.41
2.49
5.06
avge.c.
e2i_N
e3i_N
Pe e2L_
e2i PeN
e2i_Nee
BetaF
29.70
21.68
71.08
20.31
21.11
14.30
ConE
43.52
43.04
96.08
28.41
28.75
21.30
ConEttempoaal)
43.34
42.85
96.08
28.33
28.62
20.83
FLEX
42.10
36.22
97.47
27.93
27.23
21.64
TFLEX
46.17
41.34
96.69
34.29
34.63
23.88
X(ConE)
38.12
33.70
88.83
24.80
25.60
17.64
X-IF
46.11
38.06
96.82
35.76
36.20
23.73
X-logic
41.92
36.57
91.16
30.37
20.07
avgr.,
t2i_N
t3L
Pe (2i
PLN
12i NPt
TFLEX
30.16
16.09
98.40
31.39
3.35
X(ConE)
29.49
16.55
98.46
28.23
2,53
X-IF
30.26
16.05
98.37
32.07
3.27
1.56
X-logic
28.34
12.94
96.03
29.47
2.18
1.07
aVE F
e2u
Pe_e2u
Betal
21.54
20.95
22.13
ConE
43.02
37.21
48.83
ConEttemporal)
43.14
37.05
49.24
FLEX
44.38
35.72
53.04
TFLEX
54.37
52.99
55.75
X(ConE)
36.37
29.89
42.86
54.05
52.47
55.64
X-logic
45.36
43.84
46.88
avg(u))
12u
Pe 12u
TFLEX
28.69
44.99
12.39
X(ConE)
26.40
40.35
12.46
X-IF
27.70
43.38
12.02
X-logic
23.39
37.07
9.71
uvg.
Pe
Pe_e2i
Pe_aPt
Pe_ 4.56
Pe_at2i
Pe_bt2i
TFLEX
24.26
94.23
57.16
5.07
4.38
3.93
X(ConE)
21.69
81.36
38.20
5.63
5.03
5.80
5.36
X-IF
24.41
94.26
55.97
5.14
3.95
X-Iogic
21.95
87.48
50.11
4.28
3.95
3.96
3.55
AVG
Pt_SPe
oPe
Pe_sei
PL_oei
between
TFLEX
33.72
92.25
48.35
0.46
2.82
0.98
1.02
0.22
((CoF))
27.54
91.92
43.83
0.58
2.18
0.95
1.02
0.14
X-IF
33.98
92.21
50.77
0.46
2.89
1.13
1.23
0.17
X-logic
29.86
86.17
40.78
0.62
1.98
1.10
1.21
0.16
30

---

Table 16: MRR results GDELT-500. The group avg, wraps to two rows. AVG denotes average performance under all query types.
Model
avg.
Pe
Pe2
Pe3
e21
e31
Query2box
9.67
10.70
4.73
3.41
12.28
17.22
BctaE
14.75
14.02
5.76
4.71
18.61
30.67
ConE
18.44
16.65
6.18
4.70
23.11
41.55
ConExtemporal)
18.51
17.61
5.62
24.52
40.22
FLEX
19.07
17.72
5.78
4.67
24.30
42.85
TFLEX
19.60
18.50
5.76
4.68
25.94
43.14
X(ConE)
17.83
17.05
5.76
4.66
24.02
37.66
X-IF
17.92
16.18
22.41
39.80
X-logic
17.36
22.45
37.47
avEr
aPt
bPt
Pe_P Pt
Pt sPe
TToPP__P
t2i
13i
TELEX
5.38
6.49
3.25
3.27
6.16
3.06
2.88
6.78
11.17
X(ConE)
3.16
2.15
3.39
3.33
6.96
2.29
2.31
2.34
2.49
X-IF
5,49
6.62
3.30
3.30
5.84
3.07
3.05
7.08
11.66
X-logic
5.75
7.15
3.34
3.31
5.95
3.14
3.06
7.52
12.54
avBc.c,
e2i_N
e3i
Pe e2i
BetaF:
11.15
11.22
24.17
5.53
5.46
9.39
ConE
12.67
12.69
29.09
5.47
5.42
10.66
ConExtemporal)
12.67
13.42
11.63
FLEX
13.35
13.57
30.25
11.99
TFLEX
13.52
14.46
30.09
5.51
5.19
12.36
X(ConE)
12.34
13.42
26.28
5.54
5.35
11.14
X-IF
12.13
11.77
27.63
5,70
5.40
10.14
X-logic
12.11
12.32
27.00
10.01
avgr.,
t2i N
t3i
Pe t2i
12i PtN
2i NPt
TFLEX
6.31
5.92
9.01
9.30
2.76
4,56
X(ConE)
2.43
10.00
X-IF
6.50
5.90
9.70
2.85
4.90
X-logic
6.86
6.51
10.94
8.78
2.94
5.14
avg av8().)
e2u
Pe.e2u
BetaE
6.20
4.64
7.75
ConE
6.96
4.68
9.25
ConE(temporal)
7,30
4.50
10.09
FLEX
7.44
4.47
10.42
TFLEX
7.58
10.55
X(ConE)
7.41
4.64
10.19
X-IF
6.92
4.80
9.04
X-logic
6,91
4.74
9.09
avE(ы.]
12u
Pe 12u
TFLEX
6.71
9.03
4.38
X(ConE)
6.35
10.39
2.31
X-IF
6.59
8.60
4.58
X-logic
6.80
4.76
avg.
e2i Pe
Pe. e2l
Pe. Pe_at2i
bt2i
TFLEX
6.17
17.53
6.38
4.73
4.58
4.47
4.58
X(ConE)
6.17
17.53
6,44
6.45
6.18
6.02
5,97
X-IF
6.47
18.13
6.82
5.48
5.35
5.19
5,31
X-logic
6.64
17.45
6.79
5.37
AVG
(2i Pe
Pei
Pt PP
oPe
Pt_seii
Pt oc2i
between
TFLEX
9,32
15.83
3.11
2.93
3.07
1.85
X(ConE)
8.17
2.45
17.97
2.70
2.24
2.20
2.29
1.76
X-IF
8.86
8.80
15.13
2.92
2.72
3.11
3.29
1.87
X-logie
8.92
10.28
15.28
3.27
3.11
3.23
3.36
1.79
31

---

RQ1:
How about incorporating temporal information into the static query embedding, without considering time logic?
RQ2:
Further, how about enhancing the static query embedding with temporal logic?
To address research question (RQ1), we propose variant of ConE called (omE(temporal) that incorporates temporal information into the static query embedding. The only difference between ConE and Comp((emporal) is the projection operator used. In ConE(temporal), the projection operator incorporates temporal information through a technique called TTransE, which uses the temporal embedding of timestamps. In terms of formulation, the projection operator for ConE(temporal) is given by Pe(V (Vg. Vi)= = g( MLP (V, + Vt)), while for ConE it is given by Pe(V q: V.) g( MLP (Va q r)). Note that r, V, cone embeddings.
From Table 17, we observe that there is slightly improvement in terms of MRR when comparing ConE(temporal) with ConE. It shows that the temporal information can help to improve the perfor- mance of static query embedding on answering entities over TKGs. However, the improvement not significant, which indicates that simply aware of the temporal information is not enough to promote the static query embedding to efficient temporal one.
Next, we wonder if it could help when we utilize more queries involving time logic, which provides more temporal information. Therefore, we address research question (RQ2) by proposing a variant of ConE and TFLEX, called X(ConE), that enhances the static query embedding with temporal logic. X(ConE) replaces the entity part of temporal feature-logic embedding with ConE, is introduced in Section 5.2. In the other view, X(ConE) is based on ConE, using cone embedding concatenated with the time part of temporal feature-logic embedding, which can handle temporal logic.
From Table 17, to our surprise, X(ConE) performs worse than ConE and TFLEX. The concatenation of cone embedding and feature-logic embedding does not help to improve the overall performance of temporal complex reasoning. This is because cone embedding is geometric, while temporal feature-logic embedding fuzzy. We think there a mismatch or semantic conflict between the two embeddings, which leads to the poor performance of X(ConE).
Be aware that we do not deny the importance of temporal logic. In fact, temporal logic is very important for temporal complex reasoning (See Ablation on time part in Section 5.2). However, think that it is not good idea promote the static query embedding to temporal one by simply concatenating the geometric embedding with the fuzzy embedding. The right way to promote remains an open question.
Table 17: Average MRR results for TFLEX and the variants of ConE. AVG, denotes average of avg.r avg..c. and avg. AVG denotes average of all groups.
Dataset
Model
avg,
C.,e...
avBu.
avg,
avgr,c.
aVEr4
avg,
AVG.
AVG
ConE
41.94
44.88
26.47
37.76 38.27
Conp(temporal)
42.23
44.94
27.63
ICEWS14
X(ConE)
40.93
42.15
25.46
16.41
35.24
24.07
26.65
36.
30.13
TFLEX
56.79
50.82
35.74
17.56
36.37
26.24
28.03
47.78
35.93
ConE
40.93
43.52
43,02
42.49
Cone(temporal)
40.74
43.34
43.14
42.4
ICEWS05-15
X(ConE)
36.29
38.12
36.37
4.41
29.49
26.40
21.69
36.93
27.54
TFLEX
48.99
46.17
54.37
4.39
30.16
28.69
24.26
49.84
33.72
ConF
18.44
12.67
6.96
12.69
ConE(temporal)
18.51
12.67
7.30
12.83
GDELT-500
X(ConE)
17.83
12.34
7.41
3.16
3.93
6.35
6.17
12.53
8.17
TFLEX
19.60
13.52
7.58
5.38
6.31
6.71
6.17
13.57
9.32
D Visualization of Semantic Changes by Neural Temporal Operators
In this section, we present more visualization of the semantic changes of temporal operators in TFLEX. The examples are randomly selected in the test set of ICEWS14. The visualization is shown in Figure 10, Figure| 1 and Figure 12. We attach the anchors and the ground-truth answers in the corresponding caption of figures. From the figures, we observe that the model correctly ranks the hard answers top when answering query Pt. can also be seen from the figure that the semantic changes of temporal operators aPt and bPt are intuitive and reasonable.
32

---

before
after
Similarity Score
bPt=/B((P(Ssr,.))) r, 0))
Pt=
APt= 0))
Timestamp (0=2014-01-01, 301-244--131))
Figure 10: Score distributions of Pt, bPt and aPt. where is Citizen (India), r is Criticize or
denounce, o Sadhu (India), ground-truth answer Pt 2014-03-27.
before
a after
Similarity Score
bPt=B,(P.(s, 0))
Pt=P,(s,r,0
aPtM..((Hss,, r,))
120
302
300
Timestamp (0=2014-01-01, 365-201--1-31)
Figure 11: Score distributions of Pt, bPt and aPt, where s Citizen (India), r is Use unconventional violence, is Chief Secretary Chandra, ground-truth answer of Pt is 2014-05-11.
bPt=B((P(Ss.r.O) Pt=P((s,r,o)
biefo
fter
Similarity Score
aPt=AA(P(s,,..o r,
100 120
220
230
900
200
Timestamp (0=2014-01-01. 365=2014-12-311
Figure 12: Score distributions of Pt. bPt and aPt, where s is Mohammad Javad Zarif, r is Appeal for economic cooperation, o is United Arab Emirates, ground-truth answer of is 2014-04-14.
33

---

E Explaining Answers with Temporal Feature-Logic Framework
In this section, we present the explanation of answers given by TFLEX. Below we take the query "e2i" as an example. The temporal query "e2f" is defined as follows: v[V]] = V₁, 3V, (e1, Va, (1) A (e V2. Let's consider the specific query: "Who was consulted by Mohammad Javad Zarif on 2014-04-07 and consulted Mohammad Javad Zarif on 2014-04-07?" In this example, e1 "Mohammad Javad Zarif", r1 "consulted by". f "2014-04-07", r2 "consulted", and t2 "2014- 04-07". We use TFLEX to execute the query and obtain the answers. The answers are categorized into easy, hard, and wrong ones. Table 18 presents the five most likely answers and their rankings for interpretation. From the table, we observe that TFLEX ranks the answers "Mohammad Javad Zarif", "Sebastian Kurz", "China" and "Catherine Ashton" high, while ranking the wrong answer "Iran" low. TELEX even infers hard answer "Catherine Ashton" with higher score than two easy answers "Sebastian Kurz" and "China". This demonstrates that TFLEX successfully identifies the correct answers using complex reasoning and distinguishes them from the wrong ones.
eii Pe(20), Pe_ e__e2i()1), Navigation: Pt(13), Pe(14), Pe2(15), Pe3(16), Pe P((17), e2i(18), e3i(19), e2i
Pe_ ___22i(2),, 12i(23), t3i(24), t2i (e(5),,, Pt (e_se,(())), Pt oe2i(27), Di_NPt(28), t2i PIN(29), Pe (2i PtPe NPt(30). e2i NPe(31), e2i PeN(32), Pe e2i Pe NPe(33), e2i N(34), e3i N(35), e2u(36), Pe_ e_____33) Pt sPe(38), Pt oPe(39), 12i N(40), t3i N(41), 12u(42), Pe 12u(43), Pe aPt(44), Pe (Pt)), Pe Pe_at2i(46), Pe bt2((47), between(48).
Query Sentence
When did North Atlantic Treaty Organization consult Taavi Röi- vas?
Query DNF
[[ Vi,r(cr, 02, V4)
Rank
Query Answers
Correctness
Answer Type
RL North Atberic Treaty Organinninn
1
2014-11-20
Hard
Consult
2
2014-12-03
Taass Biivas
3
2014-06-16
4
2014-06-25
5
2014-10-13
x
Figure 13: Intermediate variable assignments and ranks for example Pt query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Query Sentence
Philippines denounced or criticized who on 2014-04-01?
Pe Pe(ei,mi,t))
Query DNF
[V.] (e, V2,4,)
Rank
Query Answers Correctness
Answer Type
1
China
Hard
61 Philippincs
2
Japan
x
F1 Criticizo or denounce
3
Malaysia
x
f1
2014-04-01
4
Philippines
X
5
Iran
Figure 14: Intermediate variable assignments and ranks for example Pe query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Query Sentence
On 2014-04-04, who consulted the man who was appealed to or requested by the Head of Government (Larvia) on 2014-08-01?
Pr (o(Phlourrut))rs.t))
Temporal Query
Vo
Vs, RV, (e,, Va.41) A (2, V2)
Rank
Query Answers
Correctness
Answer Type
Head (Laivia)
T1: Make an appeal or request
François Hollande
Fasy
2014-08-01
Thavi Rūivas
Easy
comeut-1 2014-04-04
Jyrki Katainen
Hard
Angela Merkel
X
Head of Guvermment (Latvia)
x
Figure 15: Intermediate variable assignments and ranks for example Pe2 query. Correctness indicates whether the answer belongs to the ground-trun -trnhhh-rtthhh set of answers.
34

---

Query Sentence
Who was criticized or denounced on 2014-08-26 0-6 by the person who made a statement on 2014-02-05 the place where the presidential movement made an appeal or request 2014-04- 24?
Temporal Query
a (,, Vs,ts)
V4, Vo, (e11, Vo.(1) A (Vaa Vo. A
Pronddeniall Mivannc
Viiee wn wesgeelaen
2014
Rank
Query Answers
Correctness
Answer Type
Medi stalerenn
4: 30144008
Government (France)
Basy
CHRAcO BEDARS 20146633
6:
2
France
x
François Hollande
Hard
Angelu Merkel
X
Citizen (International)
x
Figure 16: Intermediate variable assignments and ranks for example Pe3 query. Correctness indicates whether the answer belongs to the ground-truth touundtruuhh set of answers.
Query Sentence
Who arrested, detained, or charged the citizen (Thailand) with legal action after Police (Gresce) arrested, detained, or charged the illegal immigrant (Greece) with legal action?
Temporal Query
a(Vi]
37, (y V4 T2) A raleg. Co. Ta)
Rank
Query Answers
Correctness
Answer Type
Arrrott deture, cherycutth o
Paki (incoon
Military (Thailand)
Easy
T2l Araet dasaa charge with lagatattoo
Municipal Court (Thailand)
Easy
4g:
Tlagi berigra
Thailand
Hard
National Council for Peace and Order of Thailand Police (Camhodia)
Figure 17: Intermediate variable assignments and ranks for example Pe Pt query. Correctness indicates whether the answer belongs to the ground truth set of answers.
Query Sentence
Who was consulted by Mohammad Javad Zarif on 2014-04-07 and consulted Mohammad Javad Zarif on 2014 2014-04-07?
And
Temporal Query
o
V =
Vilee V4,4) A ralez, V2)
Rank
Query Answers
Correctness
Answer Type
Malurmead
Mohammad Javad Zarif
Fasy
f,2
2014-04-07
Consult
2
Cathrine Ashion
Hard
69.
Mohammad lavad Zarif
Schastian Kurz
Easy
Comouit
China
Fasy
Irun
x
Figure 18: Intermediate variable assignments and ranks for example e2i query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Query Sentence
Where was Joseph Robinette Biden discussed by telephone 2014 4. 10.05 05 and visited by Saud bin Faisal bin Abdul-Aze on 2014-01-202
Temporal Query
V Vi,rife,, Ve,tt) t2) ra(es, Vi,ta)
FL1 Jocp Rubbor Billes
Rank
Query Answers
Correctness
Answer Type
UAE Arned Forces
Easy
Jusch Bukiuaa
United Arab Emirates
Hard
6.
Diccss by tylipio
2014-1000
Middle East
X
Morocco
Abdel Fattah Al-Sisi
X
Figure 19: Intermediate variable assignments and ranks for example e3i query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
35

---

Query Sentence
Where was occupied territory on 2014-03-01 was visited on 2014-06-04 by the person who Ohaid Humaid Al Tayer made statement on 2014-06-11?
dus Mi
Temporal Query
d V4, 3V. (e), V2,41) A Aa V2, A (2 Vi,
Rank
Query Answers Correctness
Answer Type
F1T Ohaid Horall Al
Iran
Easy
China
X
6.7
Nora (Tios))
Iray
X
Ocopp srmoay
France
X
Afghanistan
X
Figure 20: Intermediate variable assignments and ranks for example e2i_Pe _e query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Query Sentence
Where consulted 2014-03-25 by the place where Labor Union (Croatia) made appeal or request on 2014-02-03 and was consulted by Government (Croatia) on 2014-03-25?
Temporal Query
[o] =
V,, ri(e,, Vo.41) A (2(2,, Va. (2) raV, Vs, Th)
Rank
Query Answers
Correctness
Answer Type
Labor Union (Croatia)
Hard
61
Osceree (Conle)
Business (Croatia)
x
Govermment (Croatia)
Easy Fasy
Ministry (Croatia)
Branko Greie
Figure 21: Intermediate variable assignments and ranks for example Pe_e2i query. Correctness indicates whether the answer belongs the ground-truth set of answers.
Query Sentence
Who signed formal agreement with Govermeet (South Africa) at the time when Gazprom signed formal agreement with China and Mzukisi Fatycla made statement Police (South Africa)?
Temporal Query
V
Vi, 77, (e V2, Tw) A ra(e2, f3, T.) A Ag((s: T,)
Mivommnn di Nima
Rank
Query Answers
Correctness
Answer Type
7/1 Slps Joreeeaooooreec
South Africa
Easy
Chase
Tom Motsoahae Thabane
441 Scadad
China
Bank (China)
Henry Rotich
x
Figure 22: Intermediate variable assignments and ranks for example Pe_ 12i query. Correctness
indicates whether the answer belongs to the ground-trus trunh- set of answers.
di Din Pt)
Query Sentence
At what time did Barack Obama negotiate with and consult Xi Jinping?
Temporal Query
[
1r. Tr) A Argles ca. (b))
Tianckad
Rank
Query Answers Correctness
Answer Type
42
Jaginn
2014-11-15
Easy
111
regottiations
2014-03-24
Fasy
425
2014-11-12
Easy
422
727
lirping
2014-11-11
Easy
Bardsk Utura
d
2014-11-14
Easy
Figure 23: Intermediate variable assignments and ranks for example t2i query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
36

---

Query Sentence
When did Japan engage diplomatic cooperation with South Korea, Mohammad Javad Zarif make statement Iran, and François Hollande make statement to France?
Tiee Awatl Plae Mww.i Mol
Temporal Query
77:)
Tarrles, Tr) Ara(cs, ca, Tt) A (cs. Tr)
Rank
Query Answers Correctness
Answer Type
Eqpara dgiin
2014-09-18
Hard
Scatt liree
2014-09-20
Easy
N11
Maki level Taet
2014-08-22
Fasy
61
4
2014-07-01
Hard
2014-08-16
x
Figure 24: Intermediate variable assignments and ranks for example 13i query. Correctness indicates whether the answer belongs to the ground-trute set of answers.
Query Sentenec
When did Activist (Somalia) fight with small and light weapons with Armed Gang (Somalia) while Citizen (Oman) en- gage in a symbolic act with person with whom finance Economy Commerce Trade Ministry (Oman) shared intelli- gence or information on 2014-01-06?
aUt Www.aatPe//eeeuniiiiiiia Mariiall
Temporal Query
777]] = V7, Va (e( V2.51) A r2(V,, ca, 75) A ra(ea, c4. 75)
Rank
Query Answers
Correctness
Answer Type
2014-01-30
Easy
2014-02-05
x
2014-01-29
x
2014-02-07
2014-02-04
x
Figure 25: Intermediate variable assignments and ranks for example 12i_Pe query. Correctness
indicates whether the answer belongs to the ground-truth set of answers.
PUKE Pa H))
Query Sentence
When did Tran negotiate with the country that visited Iran on 2014-03-09?
Temporal Query
TT:
Tr, TV. ri(ei, Ve, A ra(ez, V2, A Aa (;)
Rank
Query Answers Correctness
Answer Type
Tha
712
2014-03-10
Easy
2014-03-08
Easy
6- 201405-99
2014-06-16
Easy
757
4
2014-03-13
Easy
2014-03-09
Easy
Figure 26: Intermediate variable assignments and ranks for example Pt_se2i query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Piad Plo.r.ttoouuffcco counchi)
Query Sentence
When did Armenia cooperate militarily with the country that couperates militarily with Armenia 2014-12-08 and 2014-07- 02?
Ami
Temporal Query
V
V., V..Wa.rr Va, Tr) A r2(e2, Va.4, (1) A ra(es, Va)
Arar le
Rank
Query Answers
Correctness
Answer Type
Deprecellees
2014-12-08
Easy
6-
20141240
2014-07-02
Easy
2014-05-06
Fasy
4
2014-10-22
Easy
2014-07-17
x
Figure 27: Intermediate variable assignments and ranks for example Pt_ Pt_oe2i query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
37

---

Query Sentence
When did Barack Ohama express intent to meet or negotiate with Japan, and that day, Catherine Ashton 't consulted by the person Edgars Rinkevics expressed intent meet negotiate with 2014-04-22?
Temporal Query
2 Ts, 3v. miee,, V e2, Arsss,
Rank
Query Answers
Correctness
Answer Type
2014-04-17
Easy
Ape
2014-04-18
Easy
Aood
2014-02-13
Easy
2014-04-05
Easy
2014-02-11
x
Figure 28: Intermediate variable assignments and ranks for example 2i_NPt query. Correctness
indicates whether the answer belongs to the ground-trute truth set of answers.
Query Sentence
When was the citizen (Greece) made appeal request hy the country that made an appeal or request to the Party President (Greece), and on that day, Angela Merkel did not express intent to meet or negotiate with Evo Morales?
Trw.h4 swnft
Temporal Query
TT:b] = Tr, aV. Pr(ei, Ve, Arz( 62, -r)ssrssss,.,,1
Rank
Query Answers Correctness
Answer Type
2014-05-16
Easy
2014-07-24
Easy
3
2014-05-25
Fasy
4
2014-04-29
Easy
2014-07-29
Easy
Figure 29: Intermediate variable assignments and ranks for example (2i PtN query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Query Sentence
Who discussed by telephone with Sergey Viktorowich Lavrov person
the day when Sen was threatencal arrested, detained, or charged with legal action by the Buddhist (Cambodia) on 2014-01-30, 30, and on that day, Education (Colom- hia) made statement Eric Gamer?
Temporal Query
= Vy, 3V, To rifer, Vy,T)) A ralea, Ka.di) A
r 9g. 72) A 1e Pa)
Rank
Query Answers
Correctness Answer Type
John Kerry
Hard Easy
2
Cabinet/ / Council of Ministers/ / Advisons (United States)
3
Calherine Ashton
x
Sorgey Vikturovich Lavrov
5
Joseph Robincite Biden
x
Figure 30: Intermediate variable assignments and ranks for example Pe_12i_ N query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Query Sentence
Who visited Iran on 2014-04 (44--04-2 29 and did not made statement at the place where Carlos Saul Menem made a statement on 2014- 09-20?
Temporal Query
V ra V,,ts)
3V. Vb, rele, Virti) A -r2(V., Vo, A
Rank
Query Answers
Correctness
Answer Type
Santos Edelmar Lopez
Easy
2
Simon Gass
Easy
Sebastian Kurz
Easy
Catherine Ashton
Iran
Figure 31: Intermediate variable assignments and ranks for example N2i_NPe query. Correctness indicates whether the answer belongs the ground -otuhhtruthhae set of answers.
38

---

Query Sentence
Who was commented pessimistically on 2014-03-13 by the person who expressed accord Energy Departmen//Mnistry (Turkey) on 2014-03-27 and not visited by Mainland Affairs Council on 2014-02-11?
Temporal Query
[[V]] ra(ea, Vs,ta)
Vi, 3., Ma Va,4)) A r2(Ve, V4, TeJA
Ganug this Manneeonhiiiia (Tanee
F11
Rank
Query Answers
Correctness
Answer Type
Member of the Judiciary (Turkey) Lawwer/Attomg Develoy (Turkey)
Easy
Halbal
Hard
61
3
Justice and Development Party
4
Other Authorities / Officials (Turkey)
X
Party Member (Turkey)
Figure 32: Intermediate variable assignments and ranks for example e2i_PeNN query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Query Sentence
On 2014-10-21, who negotiated with the person who consulted with the Treasury/Fiancee Ministry (Eritrea) on 2014-09-11 17 but
did not arrest, detain, or charge Drug Gang (Beltrán -Leyva Car- tel) with legal action on 2014-10-16?
Temporal Query
v[Vi] Va(V. Vi.ta)
V4, 3V, V2, 7i (e Va,tiAA
r2(e2. Va, V2)
Rank
Query Answers
Correctness
Answer Type
Sneycing Kotme Codd diaaiiuecceooo wihiuddction
South Korea
Easy
Japan
201-4
China
V
Hard
4
North Korea
X
Thailand
Figure 33: Intermediate variable assignments and ranks for example Pe Pe_e2i_N query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Al, Asal r...). (all
Query Sentence
Who rallied opposition against Rotswana 2014-03-11 but not empathetically commented upon by Ciiizen (Brunei) on 2014- 03-26?
ABE
Temporal Query V[V
Va,ra 1444)) A -r2(e, Vista)
Dearress
Rank
Query Answers
Correctness
Answer Type
Relly
201443-11
Industry (Botswana)
Easy Hard
Ciious (liaao
2
Citizen (Botswana)
cusdaak senace
3
Richard Sezihera
X
4:
2014-22-3
Emplooyentt Ministry (Botswanu)
5
Paul Kagame
Figure 34: Intermediate variable assignments and ranks for example e2i N query. Correctness indicates whether the answer belongs the ground-trutt -tuth set of answers.
Query Sentence
Who was praised or endorsed by Envoy (Kazakhstan) on 2014- 05-19 and signed a formal agreement with RIA Novosti on 2014- 12-12 but not signed a formal agrecment with Goveroor (South Korca) 2014-05-09?
JUN Anttttt Pellars Mil
Temporal Query V([
Vi,ri(e;, V2, 4, A raleg, Vs, fy)A As(es, V2,
Eany Sinnkher
Rank
Query Answers
Correctness
Answer Type
1
Iran
Easy
Als Nevod
2
Irag
X
411 Doree Norvicoo
21s1212
3
China
4
Afghanistan
5
North Atlantic Treaty Organization
Figure 35: Intermediate variable assignments and ranks for example e3i e3i_N query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
39

---

Query Sentence
Who consulted Foreign Affairs (Unuguay) on 2014-11-24 or granted asylum for Julian Paul Assange on 2014-06-19?
Temporal Query
Vi V. V2,62)
Rank
Query Answers
Correctness
Answer Type
CL Fareigs Affice (Tlnggay))
Foreign Affairs (Ecuudor)
Easy
f1:
Cinusit
Mexico
Basy
9,1
2916-11-24
661 Jullan Pall Amazge
Government (Mexico)
Grani onyfum
Guatemala
3511 BN:
Other Authorities Officials (Mexico)
Figure 36: Intermediate variable assignments and ranks for example c2u query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Pluta PeOMtPiinuts 66 Plle Pilrrci
Query Sentence
On 2014-05-24, who occupied the land praised endorsed the 11-18 or requested
Lower House of Parliament (France) on 2014-11-18
appealed by Newspaper (Afghanistan) 2014-05-05!
Temporal Query
[V-]
14 aV, (ri(e), 6) Vrg(e2, V.. Ara((.)))).... 14,
411 Lome Dhamad
Rank
Query Answers
Correctness
Answer Type
601
Ncmppen Eherian
Armed Gang (Afghanistan)
Easy
Armed Band (Afghanistan)
Guapp tombeyy"
3
Military Personnel Special (Afghanistan)
Hard
(Talibban)
Afghanistan
x
Figure 37: Intermediate variable assignments and ranks for example Pe_e2u query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
IPe (P(Pe(e).7(kk))maa))))
Query Sentence
When was Iraq visited by the person who visited The Middle East on 2014-09-15?
Temporal Query
77 = Tr, 3., Pr(ei, Vo. (va, 62, Ty)
Middle Tast
Rank
Query Answers Correctness
Answer Type
PLT Make wisis""
2014-09-13
Easy
8y
2014-03-15
2014-06-25
Fasy
Make
2014-06-26
Easy
2014-06-27
Easy
2014-06-24
Easy
Figure 38: Intermediate variable assignments and ranks for example Pt_sPe query. Correctness
indicates whether the answer belongs the ground-truth set of answers.
PurPe Pelesrr.
Query Sentence
At what time did France meet the person who made statement Humphrey 'third' location?
Temporal Query
TT:
Tr, IV.. (i(e1, Vr) A ra(e2, Va,t1))
Rank
Query Answers
Correctness
Answer Type
Meet third location
2014-11-21
Hard
Peter Humphray
2014-11-25
Easy
T> Make statementt
2014407-31
2014-11-20
Easy
2014-09-23
Easy
2014-09-21
Easy
Figure 39: Intermediate variable assignments and ranks for example Pt Pt_oPe query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
40

---

Query Sentence
When did Ahu Bakr Abdullah al-Qirbi discuss with Mohammad Javad Zarif by telephone, and that day Senate (Slovakia) did not reject Justice Departnen//Ministryy (Slovakia)?
Temporal Query
04:: = 1e,,,,,,, Tb) A--2(es 64.25)
Rank
Query Answers
Correctness
Answer Type
(herm
2014-01-19
Easy
1g1
Kayen
2014-01-18
Hard
4g7
Drpar and Manurry
2014-11-B 1-02
x
4
2014-02-02
x
2014-12-13
x
Figure 40: Intermediate variable assignments and ranks for example r2i_N N query. Correctness
indicates whether the answer belongs the ground-trufe -ounnd-truth set of answers.
Query Sentence
When did the Police (Egypt) use lactics of violent repression against protesters (Egypl) and the Police (Ukraine) use tactics of violent repression against protesters (Fgypt), and on that day Information Communication / Transparency Ministry (China) did not consult Warren Truss?
Temporal Query
g/2:] = Tile,, 62, Tb) A Aafes, ea Px) A mraees, co TT)
Arbe Jayel
Rank
Query Answers Correctness
Answer Type
2014-01-21
Easy
Hanley Chine
2014-01-13
x
Yose
2014-01-22
x
4
2014-02-20
x
5
2014-01-14
X
Hard
Figure 41: Intermediate variable assignments and ranks for example t3i_ N query. Correctness indicates whether the answer belongs to the ground fround-truth set of answers.
<d PMws +i)
Query Sentence
When did citizens (Botswana) obstruct passage and block Nonofo Molefhi or Zheng Zeguang make statements about China?
Temporal Query
T
Tr.r,e,,, Tr) Vrg(es, Ca. Pz)
Seadt Mokehi
Rank
Query Answers Correctness
Answer Type
Caloe (lunnra
2014-04-09
Easy
Zhang Zaguang
2014-05-21
Easy
Mike siteeel
2014-05-20
V
Hard
44
Cless
4
2014-05-22
x
2014-05-22
x
Figure 42: Intermediate variable assignments and ranks for example 12u query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Query Sentence
Who was appealed requested by Citizen (Nigeria) on the day when the University of Cape Town consulted News Editor (Nige- ria) or State Security Court (United Arab Emirates) arrested, detained, or charged Muslim (United Arah Emirates) with legal action?
Fille Www.n Pouneelll
Temporal Query
q/V/| V1, "T, (er, Vs, T.)(rafez.es. T.)vrs(ea, es, T.))
Ciker Bigero Ulevenry etCipe Time Cerai
Rank
Query Answers
Correctness
Answer Type
Government (Nigeria)
Easy
Tal drrae ddeis, entorrrrttttlllatiieee
Mullini wal Eninae)
Assemblies (Nigeria)
Hard
Cas Ara Cinkna
Other Authorities Officials (Nigeria)
Head of Government (Nigeria)
Citizen (Nigeria)
Figure 43: Intermediate variable assignments and ranks for example Pe_ 12_ query. Correctness indicates whether the answer belongs to the ground-trute set of answers.
41

---

Pr.ath Polt
Query Sentence
Who made statement citizens (India) before Representatives (France) consulted Stephane Le Foll?
Temporal Query
[/V]] (2 63, T..)
V4, 3T. To. ril Vs, Tb) A After(Ta, Tb) A
After
117
Rank
Query Answers
Correctness
Answer Type
Court Judge (India)
Easy
Governor (India)
Easy
Manohar Lai Khattar
Easy
Vasundhara Raje
Easy
Head of Govermment (India)
Figure 44: Intermediate variable assignments and ranks for example Pe_aPt query. Correctness indicates whether the answer belongs the ground-truth set of answers.
Query Sentence
Who expressed intent to meet or negotiate with the Head Gov- ermment (Bangladesh) after South Korea Criticized or denounced Sadako Ogata?
Pran Mulortligge///aa.eiilll
Temporal Query
V4, 3T, (e Vt) A Before(Ta. Tb)
r2(e2 T.)
Rank
Query Answers
Correctness
Answer Type
TLE
Citicbsi donceace
4g
Sedaio Opila
Japan
Easy
T5:
Abpresu fnderttomedd regetiotee
North Korea
Easy
41
load of Geeemaaaaaattaaaaadd
China
Easy
South Korea
x
Kim Jong-Un
Figure 45: Intermediate variable assignments and ranks for example Pe_bP bPt query. Correctness
indicates whether the answer belongs to the ground ground-truth set of answers.
Query Sentence
Who criticized denounced by North Korea on the day before Admiral (Russia) made statement for Chuck Hagel and Ministry (Madagascur) made statement for Government (Mada- gascar)?
Temporal Query
VT Th, Files, Vs,Ta)) A After(Ta Ts) A r2(e2. cs. Ta) A ro.
Rank
Query Answers
Correctness
Answer Type
Tide
1
South Korea
Easy Easy
2
Japan
3
North Korea
Head of Government (South Korea)
Easy Easy
5
John Kerry
Figure 46: Intermediate variable assignments and ranks for example Pe_ e__at2i query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
Query Sentence
Who was declined commented by Citizen (India) after the day when the employee (Mozambique) consulted the Presidential Candidate (Mozambique). and Evo Morales made statement to the Military (Bolivia)?
Temporal Query
BT, Th (ei. Vs,T)) A Before(Ta. Th) A P2(e2: C3. Ars
[
Cikan bookl
Rank
Query Answers
Correctness
Answer Type
Media (India)
Easy
141
Member of Parliament (India)
X
Court Judge (India)
V
Easy
4
Head of Government (India)
X
S
Governor (India)
X
Figure 47: Intermediate variable assignments and ranks for example Pe_bt2i query. Correctness indicates whether the answer belongs to the ground-truth set of answers.
42

---

Query Sentence
What was the day hetween the day that Tsai Ing accused the Party Member (Taiwan) and the foreign atfairs (United States) appealed for diplomatic cooperation (such policy support) from the other Authoritissofficials (Romania)?
Temporal Query
7T =
Tr, 3f, Tr En (11,,, To) A r2(es.e.. To) A After(T,, Tv) A Before(Th, Tr)
Rank
Query Answers
Correctness
Answer Type
Appeod for aepleeete unglppiil
2014-07-19
Easy
Aoos
2014-07-2 29
42
2014-06-10 10
Easy
2014-08-22
d5
2014-06-25
Easy
Figure 48: Intermediate variable assignments and ranks for example between query. Correctness indicates whether the answer belongs to the ground- woound-rruthhao set of answers.
43


Reviews:
[{"review_id": "WHQVCbgJp4", "reviewer": "Reviewer_AEhV", "summary": "Authors present new embedding framework called TFLEX to embed complex temporal queries over temporal knowledge graphs to perform multi-hop reasoning with time constraints on TKGs. Authors present the overall embedding framework using Fuzzy logic to model complex logical queries and extending fuzzy logic to include three temporal operators After, Before and Between. It presents new benchmark datasets to evaluate embedding on TKGs and show experimental results to show the benefits of proposed approach.", "questions": "Can this solve complex query answering equally well on other bench mark tasks and how does it fare against the prior art?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "paper presents a new embedding framework and dataset for complex temporal queries, which can benefit the community in extending research in this direction. ", "weaknesses": "Not sure if this framework can handle complex queries without temporal constraints on par with other complex query embedding methods?\nsuppose if we ignore the temporal aspect of current framework and run the standard benchmarks on complex query handling does it work as well as other methods in the literature. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe appreciate the reviewer's constructive feedback on our paper. We are pleased that the reviewer recognizes the merits of our work and acknowledges the contributions we have made to the field. We would like to address the concerns and questions below.\n\n> Not sure if this framework can handle complex queries without temporal constraints on par with other complex query embedding methods? suppose if we ignore the temporal aspect of current framework and run the standard benchmarks on complex query handling does it work as well as other methods in the literature. Can this solve complex query answering equally well on other bench mark tasks and how does it fare against the prior art?\n\nIt's an interesting question. And of course, the answer is yes. Ignoring the time parts of TFLEX degenerates to the variant FLEX in our experiments.\nWe report the results of FLEX on the standard datasets FB237, FB15k and NELL, comparing to famous QE baselines (GQE, Query2box, BetaE, ConE) in the tables below.\nFrom the tables, we observe that FLEX achieves competitive performance compared to the four baselines. This is similar to the situation on temporal complex queries over TKGs.\nWe attribute the improvement to the fuzzy logic operators, which can handle complex queries better than the geometric operators in QE baselines.\n\nTable 1. MRR results for answering queries without negation ($\\exists$, $\\land$, $\\lor$) on FB15k, FB237 and NELL. The best results are in bold. **AVG** denotes average performance.\n\n| **Dataset** | **Model** | **1p**   | **2p**   | **3p**   | **2i**   | **3i**   | **pi**   | **ip**   | **2u**   | **up**   |  **AVG** |\n| :---------- | :-------- | :------- | :------- | :------- | :------- | :------- | :------- | :------- | :------- | :------- | -------: |\n| FB15k       | GQE       | 53.9     | 15.5     | 11.1     | 40.2     | 52.4     | 27.5     | 19.4     | 22.3     | 11.7     |     28.2 |\n|             | Q2B       | 70.5     | 23.0     | 15.1     | 61.2     | 71.8     | 41.8     | 28.7     | 37.7     | 19.0     |     40.1 |\n|             | BetaE     | 65.1     | 25.7     | 24.7     | 55.8     | 66.5     | 43.9     | 28.1     | 40.1     | 25.2     |     41.6 |\n|             | ConE      | 73.3     | 33.8     | 29.2     | 64.4     | 73.7     | 50.9     | 35.7     | **55.7** | 31.4     |     49.8 |\n|             | FLEX      | **77.1** | **37.4** | **31.6** | **66.4** | **75.2** | **54.2** | **42.4** | 52.9     | **34.3** | **52.4** |\n|             |\n| FB237       | GQE       | 35.2     | 7.4      | 5.5      | 23.6     | 35.7     | 16.7     | 10.9     | 8.4      | 5.8      |     16.6 |\n|             | Q2B       | 41.3     | 9.9      | 7.2      | 31.1     | 45.4     | 21.9     | 13.3     | 11.9     | 8.1      |     21.1 |\n|             | BetaE     | 39.0     | 10.9     | 10.0     | 28.8     | 42.5     | 22.4     | 12.6     | 12.4     | 9.7      |     20.9 |\n|             | ConE      | 41.8     | 12.8     | 11.0     | 32.6     | 47.3     | 25.5     | 14.0     | 14.5     | 10.8     |     23.4 |\n|             | FLEX      | **43.6** | **13.1** | **11.1** | **34.9** | **48.4** | **27.4** | **16.1** | **15.4** | **11.1** | **24.6** |\n|             |\n| NELL        | GQE       | 33.1     | 12.1     | 9.9      | 27.3     | 35.1     | 18.5     | 14.5     | 8.5      | 9.0      |     18.7 |\n|             | Q2B       | 42.7     | 14.5     | 11.7     | 34.7     | 45.8     | 23.2     | 17.4     | 12.0     | 10.7     |     23.6 |\n|             | BetaE     | 53.0     | 13.0     | 11.4     | 37.6     | 47.5     | 24.1     | 14.3     | 12.2     | 8.5      |     24.6 |\n|             | ConE      | 53.1     | 16.1     | 13.9     | 40.0     | **50.8** | 26.3     | 17.5     | 15.3     | 11.3     |     27.2 |\n|             | FLEX      | **57.8** | **16.8** | **14.7** | **40.5** | **50.8** | **27.3** | **19.4** | **15.6** | **11.6** | **28.2** |\n\nTable 2. MRR results for answering queries with negation on FB15k, FB237, and NELL. The best results are in bold. **AVG** denotes average performance.\n\n| **Dataset** | **Model** | **2in**  | **3in**  | **inp**  | **pin**  | **pni**  |  **AVG** |\n| :---------- | :-------- | :------- | :------- | :------- | :------- | :------- | -------: |\n| FB15k       | BetaE     | 14.3     | 14.7     | 11.5     | 6.5      | 12.4     |     11.8 |\n|             | ConE      | 17.9     | 18.7     | 12.5     | 9.8      | 15.1     |     14.8 |\n|             | FLEX      | **18.0** | **19.3** | **14.2** | **10.1** | **15.2** | **15.4** |\n|             |\n| FB237       | BetaE     | 5.1      | 7.9      | 7.4      | 3.6      | 3.4      |      5.4 |\n|             | ConE      | 5.4      | 8.6      | 7.8      | **4.0**  | **3.6**  |      5.9 |\n|             | FLEX      | **5.6**  | **10.7** | **8.2**  | **4.0**  | **3.6**  |  **6.5** |\n|             |\n| NELL        | BetaE     | 5.1      | 7.8      | 10.0     | 3.1      | 3.5      |      5.9 |\n|             | ConE      | 5.7      | 8.1      | 10.8     | 3.5      | 3.9      |      6.4 |\n|             | FLEX      | **5.8**  | **9.1**  | **10.9** | **3.6**  | **4.1**  |  **6.7** |\n"}, {"review_id": "2xRfbjH8l2", "reviewer": "Reviewer_BpBt", "summary": "This paper studied the multi-hop logical reasoning problem on temporal knowledge graphs and proposed the first temporal complex query embedding framework named Temporal Feature-Logic Embedding framework(TFLEX).Firstly, they defined the task of multi-hop logical reasoning over TKGs. Secondly, they designed the first multi-hop logical reasoning framework which utilizes fuzzy logic to compute the logic part and extend fuzzy logic on timestamp set for supporting all FOL operation and extra temporal operations(After, Before and Between). At last, they generated three new TKG datasets for the task of multi-hop logical reasoning. Experiments on benchmark datasets demonstrate the efficacy of the proposed framework in deal with different operations in complex queries.", "questions": "1.\tIt is recommended that the author compare the methods of relevant articles with the improvements of the author's article to highlight the efficiency of their methods\n2.\tIt is suggested that the author can further optimize the content of the paper to streamline the appendix content, and simplify the subsection: Explaining answers with the framework and Experimental analysis into the main body of the paper to better demonstrate its experiments and principles.\n3.\tIt is suggested that Pseudocode should be used for interpretation, which is helpful for reading and analysis and is more standardized.\n", "limitations": "The author thoroughly analyzed its limitations in the article, including insufficient time operators, improved time embedding, long query generation time, MRR and Hits@k Weak. Also proposed its plan for future improvement work", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThey creatively conducted research multi-hop logical reasoning problem on the TKGs and proposed the first TFLEX framework to answer the temporal complex queries.\n2.\tthey relatively proposed using fuzzy logic to handle temporal feature-logical embedding and expanded extra temporal operations.\n3.\tThey provided three new TKG datasets and compared them with multiple benchmarks for experimental verification in order to better validate the reliability of their framework.\n4.\tThey had clear explanations and explanations of the definitions and methods used, with reasonable classification and appropriate illustrations.\n5.\tThere are more detailed compensations and explanations in the appendix to better explain the details of the method and experiment. \n6.\tThere are also corresponding open source code and datasets.\n", "weaknesses": "1.\tIn the section on related work, there is a lack of appropriate explanations for the problems and shortcomings of the methods cited in the relevant articles, as well as a comparison with current work.\n2.\tThe appendix section provides some overly detailed supplements to the overall concept, such as a more concise description of fuzzy logic.\n3.\tIt is not recommended to directly apply source code for explanation and display in the appendix section.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe appreciate the reviewer's time and effort in evaluating our manuscript. We have carefully considered the provided feedback and would like to address each point of concern:\n\n> Lack of comparison with related work:\n\n(see pdf in the global rebuttal section)\n\nWe acknowledge the reviewer's comment regarding the need for more explicit discussions about the problems and shortcomings of the methods cited in the related articles. In the pdf in global rebuttal section, we include a comprehensive comparison between the methods outlined in the related work and our proposed approach. This will not only highlight the limitations of existing methods but also underscore the advancements offered by our approach.\n\n> Overly detailed supplements in the appendix:\n\nWe understand the reviewer's concern about the detailed supplements in the appendix. While we believe that these supplemental explanations contribute to a deeper understanding of the concepts, we will revisit the appendix section to ensure that the content remains concise and directly relevant to the main concepts presented in the paper.\n\n> Using pseudocode for interpretation:\n\nWe appreciate the suggestion to use pseudocode for better interpretation. In the revised manuscript, we will include pseudocode to present the key algorithms and operations in our TFLEX framework. This will enhance the readability and facilitate a standardized understanding of our proposed methodology.\n\n> Streamlining appendix content and simplifying subsections:\n\nWe take the reviewer's point about streamlining the appendix content and simplifying certain subsections. To enhance the readability and coherence of the paper, we will integrate the content from the subsections \"Explaining answers with the framework\" and \"Experimental analysis\" into the main body of the paper. This adjustment will help our readers better understand the experimental setup and principles underlying our work.\n\n----\n\nIn conclusion, we are genuinely thankful for the reviewer's insightful feedback, which has guided us towards refining our manuscript. We are committed to addressing each concern and ensuring that the revised paper comprehensively presents our work while effectively addressing the points raised.\nThank you for your time and consideration.\n"}, {"review_id": "PJJs6aXDwm", "reviewer": "Reviewer_SQqW", "summary": "This paper proposes a method to learn on temporal knowledge graphs, using a combination of fuzzy logic with a temporal extension and node embeddings. To test the method, three new datasets were generated.\n\nThe choices made for the model seem logical and are over all well motivated in the paper (e.g. figure 2). The final model was tested on multiple data sets and compared to several non-trivial baselines. Furthermore, ablation studies were performed to analyze the effect of several components of the TFLEX model.\n\nWhile this paper has several strong points when it comes to presentation (e.g. figure 2 and the colors in line 164), there are some lacking areas. Firstly, having to much mathematical notation spread throughout the text itself (e.g. lines 173-180) compromises readability. Secondly, given the novelty of the model, I would have liked to see a detailed diagram representing the model.\n\nThis paper provides a novel approach for learning a temporal graphs and provides three data sets for this task. Research on temporal graphs is somewhat limited (as seen by the limitations of the baselines), making this a welcome addtion to the sub-area of machine learning on graphs.\n\n\n", "questions": "To me, it is not clear why we would want up to n compositions of the identity, negation, before and after operations (see line 105).\n  In lines 124-125 you seem to assume a query only has one answer. Is this only for training purposes, or is this a limitation of the model?", "limitations": "The model seems to be limit the answer set of queries to just one answer (the embedding closest to the query embedding).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper performs a analysis by motivating choices and thoroughly evaluating the model. The figures that are present are very clear.", "weaknesses": "Some of the math could have been beter separated from the text, making the paper more readable.  There is not figure to visually convey the model itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful assessment of our paper. We would like to address the points raised in the review and provide clarification and additional information to enhance the overall understanding and appreciation of our work.\n\n> To me, it is not clear why we would want up to n compositions of the identity, negation, before and after operations (see line 105).\n\nWell, we also find that it is not necessary. We will remove it in the revision. For now, you can refer to our response to reviewer NJDL, where we present a revised version of the definition.\n\n> In lines 124-125 you seem to assume a query only has one answer. Is this only for training purposes, or is this a limitation of the model?\n\nTo clarify, a query has an answer set instead of one answer only. Each answer in the answer set is named Temporal Query Answer. For each answer, its embedding should be close to the query embedding.\n\nOnce again, we appreciate the opportunity to receive feedback and engage in this scholarly discourse, and we are dedicated to delivering an enhanced version of our paper that aligns with the reviewer's and the broader academic community's expectations."}, {"review_id": "yuLJmeUeEW", "reviewer": "Reviewer_NJDL", "summary": "The authors introduce an embedding-based method for answering complex, i.e., multi-hop, queries on temporal knowledge graphs. The Temporal Feature-Logic Embedding framework (TFLEX) uses fuzzy logic to model first-order logic operations on the entity and timestamp sets. The queries, with answers being either entities or timestamps, are embedded via four components, including feature and logic vectors for both entities and timestamps. The correct answer to the query is determined as the entity or timestamp whose embedding is closest to the query embedding according to a predefined distance function. Based on existing benchmark datasets, three new datasets were created that include a variety of complex queries. The proposed method TFLEX is compared against state-of-the-art query embedding methods, where TFLEX outperforms all baselines. Ablation studies show the effectiveness of the particular design of the method.", "questions": "The definitions section and method section contain the formulation for the two cases entity query and timestamp query and include a lot of notation, which impedes the readability. It is sometimes not clear, which equations are needed for which query type. Since both cases are similar in many ways, one suggestion would be to focus on one case, add more details/examples/descriptions, and include the other case in the appendix or mention the differences in a separate section. \n\nMinor comments:\nFigure 1: “Obama” -> “Barack Obama” (to be consistent with the other names)\nLine 21: The statement “results are inevitably incorrect” is rather strong. Depending on size and content of the KG, results might be correct.\nTable 1: The best results could be marked in bold to make it easier to identify the best method.", "limitations": "The authors addressed the limitations and corresponding future work adequately in a separate section. Possible broader impact and application areas are also stated in a designated section.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper seems to be the first one to address the multi-hop logical reasoning problem on temporal knowledge graphs, where queries consist of disjunctions of atomic formulas. The authors introduce this new task, which can find application and are relevant in many domains.\n- The proposed method supports FOL operations as well as temporal operations and is able to perform multi-hop reasoning. \n- Three new datasets are generated, including 40 kinds of complex queries, which can be used for further benchmark experiments.\n- The experiments and ablation studies are extensive and confirm the effectiveness of the proposed method.\n- The source code and datasets are available online, which supports reproducibility and further evaluation.", "weaknesses": "The main weakness concerns the clarity and preciseness of the technical content, which makes it difficult to follow and to understand the methodology, thereby making it challenging to judge the soundness. Here are some examples: \n- Figure 1: The query example \"During François Hollande was the president of France, which countries did Xi Jinping visit but Obama did not visit?\" is given. The logical formulation states that the answer is an entity (country) so that there exists a timestamp T_1 when Xi visited this country and there exists a timestamp T_2 when Obama did not visit this country. This does not seem to reflect the query correctly. The answer should be a country so that there exists a timestamp T_1 when Xi visited this country and for all T_2, Obama did not visit this country. Also, the picture might be easier to understand if the entities for Hollande (e_1, e_3) and France (e_2, e_4) are represented by one node instead of two.  \n- Line 87: The fact set should be a subset of the complete graph. The set on the right-hand side of the equation, however, denotes the complete graph (all possible quadruples).\n- The definitions for the entity query and the timestamp query are difficult to understand. Maybe it is possible to first introduce atomic formulas, then literals, then conjunctions, and last the disjunctive normal form. Some query examples with corresponding query structures could make it more comprehensible. Why should there be the same number (k) of bound variables V_i and T_i? There is only one relation r in the query definition, but is it not common to have several relations (see Figure 1)? Should “Between” also be one of the possibilities for f?\n- The method builds on fuzzy logic, which is important for modeling the operators. A short introduction to fuzzy logic directly in the paper (instead of being in the appendix) would help understanding. \n- The MLPs in Equation 3 are different, so indices could be added (as in later cases) to make it clearer.\n- Line 160: The Alignment Rule is mentioned and is part of the subsequent equations. Since both formulas (for I_e and I_t) look exactly the same, how are the AND operators calculated differently?\n- Line 131: ||q|| should be a query set, so is V_q the embedding of all queries or just one query? \n- Existing literature for temporal KG completion does not only include the two groups tensor decomposition and translation. There are also methods based on dynamic (time-dependent) embeddings, logical rules, autoregressive models, Markov processes, … (see [1] for a survey from January 2022).\n[1] Temporal Knowledge Graph Completion: A Survey. Borui Cai, Yong Xiang, Longxiang Gao, He Zhang, Yunfeng Li, Jianxin Li. https://arxiv.org/abs/2201.08236", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe would like to express our sincere gratitude for the thoughtful feedback. Below, we address the comments and concerns in a comprehensive manner.\n\n> Figure 1: The logical formulation and the representation of entities in the query example.\n\nThe example logical formulation is designed to correctly handle the negation by applying it to the entity set (country) rather than the timestamp set (when François Hollande was the president of France). The set of timestamp already indicates that for-all timestamps in the set. By doing so, we ensure to find the country that meets the specific time constraint for Xi's visit and Obama’s non-visit.\n\nAs for the representation of entities for Hollande and France, such representation allows each node corresponds to the term in the above query formulation. We believe that using separate nodes (e_1, e_3, e_2, e_4) enhances clarity and avoids potential confusion in the interpretation of the query.\n\n> The definitions for temporal complex query.\n\nThis is a valuable review. Below, we first revise the definitions, then response to the comments. If some parts are still unclear, please let us know. We appreciate your feedback.\n\nTemporal Knowledge Graph (TKG) $G = \\{\\mathcal{V}, \\mathcal{R}, \\mathcal{T}, \\mathcal{F}\\}$ consists of entity set $\\mathcal{V}$, relation set $\\mathcal{R}$, timestamp set $\\mathcal{T}$ and fact set $\\mathcal{F} = \\{ (s,r,o,t) \\} \\subseteq \\mathcal{V}\\times\\mathcal{R}\\times\\mathcal{V}\\times\\mathcal{T}$ containing subject-predicate-object-timestamp $(s,r,o,t)$ quartets.\nWithout loss of generality, $G$ is a first-order logic knowledge base, where each quartet $(s,r,o,t)$ denotes an atomic formula $r(s, o, t)$, with $r$ a binary predicate and $s, o, t$ its arguments.\n\nWe focus on Existential Positive First-Order (EPFO) query [1] over TKG, namely Temporal Complex Query $q$, which is categorized into entity query and timestamp query. Formally, the query $q$ consists of a target variable $A$, a non-variable anchor entity set $V_a \\subseteq \\mathcal{V}$, a non-variable anchor timestamp set $T_a \\subseteq \\mathcal{T}$, bound variables $V_1,\\cdots,V_k$ and $T_1, \\cdots, T_l$, logical operations (existential quantification $\\exists$, conjunction $\\land$, disjunction $\\lor$, identity $1$, negation $\\lnot$), and extra temporal operations ($\\textbf{After}$, $\\textbf{Before}$).\nInspired by [2,3], the disjunctive normal form (DNF) of query $q$ is defined as:\n$$\n\\begin{aligned}\n  q[A] = A\n  &\\\\;\\\\;:\\exists V_1, \\cdots, V_k, T_1, \\cdots, T_l : (e_{1}^{1} \\land \\cdots \\land e_{n_1}^{1}) \\lor \\cdots \\lor (e_{1}^{m} \\land \\cdots \\land e_{n_m}^{m})\\\\\\\\\n  \\text{where}\n  & \\\\;\\\\; e = f \\circ r(V_s, V_o \\text{ or } A, g(T))\\text{ or }f \\circ r(V_s \\text{ or } A, V_o, g(T))\\\\;\n      \\text{ if }q\\text{ is entity query,}\\\\\\\\\n  & \\\\;\\\\; e = f \\circ r(V_s, V_o, g(T \\text{ or } A)) \\\\;\n      \\text{ if }q\\text{ is timestamp query} \\\\\\\\\n  \\text{with}\n  & \\\\;\\\\; V_s, V_o \\in V_a \\cup \\\\{V_1, \\cdots, V_k\\\\}, T\\in T_a \\cup \\\\{T_1, \\cdots, T_l\\\\}, r\\in\\mathcal{R}, f\\in \\\\{1, \\lnot\\\\}, g\\in\\\\{\\textbf{After}, \\textbf{Before}\\\\}\\\\\\\\\n\\end{aligned}\n$$\n\nIn the equation, the DNF is a disjunction of $m$ conjunctions, where $e_{1}^{j} \\land \\cdots \\land e_{n_j}^{j}$ denotes a conjunction between $n_j$ logical atoms, and each $e_i^j$ denotes a logical atom. We ignore indices in the definition of $e_i^j$ to keep the formula clean. The goal of answering the query $q$ is to find the set of entities (or timestamps) $[q]$ that satisfy the query, such that $A\\in [ q]$ iff $q[A]$ holds true, where $[ q]$ is the answer set of the query $q$.\n\n- [1] Efficient query evaluation on probabilistic databases.\n- [2] Query2box: Reasoning Over Knowledge Graphs In Vector Space Using Box Embeddings.\n- [3] Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs.\n\n> Should “Between” also be one of the possibilities for f?\n\nWe do not include Between in the definition of f for two reasons. Firstly, Between receives *two* inputs, different from any *unary* f in {1, time negation, After, Before}. Secondly, Between is not atomic, because `Between(t1, t2) = TimeAnd(After(t1), Before(t2))`.\n\n> A short introduction to fuzzy logic in the main body of the paper.\n\nWe will add the following introduction to fuzzy logic and vector logic in the method section.\n\nTo cope with logical transformation in the vector space, we introduce vector logic, which is a type of fuzzy logic over vector space.\nFuzzy logic is a generalization of Boolean logic, such that the truth value of a logical atom is a real number in $[0, 1]$. In comparison, as a generalization of a real number, the truth value in vector logic is a vector $[0, 1]^d$ in the semantic space. We denote the logical operations in vector logic as $\\textbf{AND}(\\land), \\textbf{OR}(\\lor), \\textbf{NOT}(\\lnot)$, and so on, which receive one or multiple vectors and output one vector as answer. For more details about fuzzy logic, please refer to Appendix A.\n\n> The indices for MLPs to make it clearer. Line 160: how are the AND operators calculated differently?\n\nIt should be clarified that all operators don't share parameters, which means all MLPs are different. We spend efforts to keep the design of operators simple, thus proposing the same structure with trainable parameters for all dyadic operators. So, $\\mathcal{I}_e$ and $\\mathcal{I}_t$ is the same on logic parts, but not the same on the feature parts. We would revise the notation as follows to make it clearer:\n- $\\mathcal{P}_e(\\mathbf{V}_q,\\mathbf{r},\\mathbf{V}_t)         = g(\\textbf{MLP}_0^e(\\mathbf{V}_q+\\mathbf{r}+\\mathbf{V}_t))$\n- $\\mathcal{P}_t(\\mathbf{V}\\_{q_1},\\mathbf{r},\\mathbf{V}\\_{q_2}) = g(\\textbf{MLP}_0^t(\\mathbf{V}\\_{q_1}+\\mathbf{r}+\\mathbf{V}\\_{q_2}))$\n- $\\alpha_i = …\\textbf{MLP}_1 → \\alpha^{e,t}_i = …\\textbf{MLP}^{e,t}_1$\n- $\\beta_i = …\\textbf{MLP}_2 → \\beta^{e,t}_i = …\\textbf{MLP}^{e,t}_2$\n\n> Existing literature for temporal KG completion\n\nsee global rebuttal."}, {"review_id": "dSZak40Gw5", "reviewer": "Reviewer_Zrqw", "summary": "This paper proposes TFLEX, a framework that reasons over temporal knowledge graphs (TKG). It takes a complex query about either an entity or a timestamp and a set of constraints as input. The query is then converted into a directed acyclic graph (DAG), which is a computation graph that projects the query into an embedding space. The paper also defines operations that happens in the graph, including projection, intersection, complement, and other set/temporal logic operations (e.g. After, Before, and Between). \n\nWhen the query embedding is produced, its distances between all candidate entity/timestamp embeddings are computed. The loss function is designed to minimise the distance between the embedding of the query and that of the correct entity/timestamp and maximise the distance between the query and the incorrect candidates.\n\nThe framework was evaluated on three standard datasets with modifications. The assessment process involves predefined complex queries on top of each dataset, including 27 types for training and 40 types for test. Several variants of the proposed framework were compared, as well as a few previous works on embedding query for static knowledge graphs. ", "questions": "Please see my points above. ", "limitations": "The paper has a good discussion about its limitations in page 11.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This paper discusses complex query embedding in temporal knowledge graphs, which is an important yet under-investigated topic. \n- The paper provides a definition to the problem and some attempts to solves the problem.", "weaknesses": "Related Works\n- From my understanding, TKGC is an important subset of the task (Temporal Complex Query) defined in this paper. There are many TKGC works missing in the related work part. For example, [1] and [2]. \n- The discussion on complex query embedding works is insufficient. The paper should provide a more substantial explanation as to why existing works on static KG cannot be easily adapted for temporal KGs. \"They cannot utilize temporal information\" is a result, not a reason. \n\nMethod\n- In line 146 the paper introduces \"the relational embedding\" without saying how it is defined. Is it a trainable parameter? How is it initialised?\n- The definition to operators appears arbitrary. In equation (3), $V_q + r + V_t$ are adding the embedding for entities, relations, and timestamps together. What is the semantics of addition here?\n- Again, in equation (7), something like $q_f^t+\\frac{1+q_l^t}{2}$ is in essence adding the entity feature with its a truth value. What is the semantics of addition between these two values?\n- Later in the definition of distance function (line 199), $q_l^e + ||v_f^e-q_f^e||_1$ is adding a truth value with a L1-norm. What is the semantics of addition here?\n\nExperiments:\n- The method is evaluated on 40 types of generated queries from three datasets for TKGC. The paper did no discuss the reason for choosing these specific 40 types or how complete these types are. Nor has the it discussed whether TFLEX can generalise to queries outside of those 40 types.\n- There is no comparison with TKGC works. Since the TKGC task aims at completing a simpler query, it is naturally a subtask of what is proposed in this work. In other words, a (s, r, ?, t+1) completion task can be converted to a query and TFLEX should be able to handle it. I would expect at least a separate table comparing TFLEX and state-of-the-art TKGC on a completion task. Otherwise, it is difficult to assess how well TFLEX captures temporal information.\n\nMiscellaneous:\n- There are technical details being moved to appendix due to page limit. Please note that the page limit is nine content pages, not eight. The current main text is not self-contained to provide readers a high-level idea about the benefits of the framework. For example, as I pointed out above, many definitions and designs about the operators lack intuition or rationale, and look like something randomly popping out.\n- The source code provided in the supplementary material and the link in the paper reveal a non-anonymous email \"l****9@sysu.edu.cn\". While I believe this is an honest mistake, I urge the author to remove this information.\n\n---\nReference\n\n[1] Chronor: Rotation based temporal knowledge graph embedding, AAAI 2021.\n[2] Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning, SIGIR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and detailed feedback. Below please find responses to individual comments/questions:\n\n## Related Works\n\n> TKGC works in the related work part.\n\nsee global rebuttal.\n\n> The discussion on complex query embedding works is insufficient.\n\nTo begin with, we have many discussions about the static-to-dynamic difficulty. Apart from the related work, we first mention the difficulty in line 29-30 (Introduction Section), then discuss static QEs’ poor results and analyze the reasons in line 230-232 (Experiment Section), and even further explore the right way to promote the static query embeddings to temporal ones in line 670-707 (a full page for exploration in Appendix C).\n\nWe would like to add the below sentences in the revision to make it more comprehensive.\n\n```\nFirstly, static query embeddings (QEs) are built over (s, r, o) triples instead of (s, r, o, t) quartets, thus ignoring the timestamps for temporal complex reasoning. The second reason is the order property of timestamps, which is on the contrary that entities are unordered, leading to that static QEs are unable to handle Before, After temporal logic. Therefore, it is challenging for static QEs to utilize temporal information in the TKGs.\n```\n\n## Method\n\n> the relational embedding.\n\nYes, it’s trainable, and randomly initialised. Besides, entity embedding and timestamp embedding are also trainable and randomly initialised.\n\n> The semantics of addition in equation 3.\n\nThe equation 3 follows the assumption of translation-based methods: $q_o \\approx q_s + r + t$. As a comparison, static KGE TransE has $o\\approx s+r$, and temporal KGE TTransE has $o \\approx s + r + t$. The addition represents a semantical translation starting from the source entity set, following the relation and timestamp conditioning, ending at the target entity set.\n\n> What is the semantics of addition between these two values in equation 7?\n\nThe entity feature and its truth value together form a fuzzy interval in the semantic space.\n\nThose entities, whose entity features are covered by the fuzzy interval, are viewed as the answers of the query.\n\n> What is the semantics of addition in the distance function (line 199)?\n\nThe distance function aims to optimize two losses. One is to push the answers to the neighbor of query in the embedding space. It corresponds to the term L1 distance between answer and query. The other is to reduce the uncertainty of the query (the probability interpretation of the logic part), to make the answers more accurate. We use element-wise addition to combine the two losses.\n\n## Experiments:\n\n> The paper did no discuss the reason for choosing these specific 40 types or how complete these types are. Nor has the it discussed whether TFLEX can generalise to queries outside of those 40 types.\n\nIn Appendix B.2, we discuss why to choose these types in the dataset generation section in detail. We mention that it is in order to keep a similar experiment setting with previous static query embeddings. We also present the comparison of query types between temporal and static ones in Table 5.\n\nAbout generalization, please be aware that TFLEX is **trained on 27 types and evaluated on 40 types**. The extra 13 types contain various unseen query structures that do not exist in the training set. The experiment result **Out-of-data reasoning** (line257-258) shows the generalization ability of TFLEX. Therefore, it needs not to introduce more types of queries.\n\n> There is no comparison with TKGC works. I would expect at least a separate table comparing TFLEX and state-of-the-art TKGC on a completion task.\n\nWe don’t agree with the opinion that we ignore all the TKGC baselines. Actually, we compare to two translation-based TKGC methods (TTransE and HyTE), present the results in Figure 4, and discuss in paragraph **Necessity of training on temporal complex queries** (line264-269). We choose TTransE and HyTE because our projection operator is also translation-based ($\\mathcal{P}_e(\\mathbf{V}_q,\\mathbf{r},\\mathbf{V}_t)  \\propto \\mathbf{V}_q+\\mathbf{r}+\\mathbf{V}_t$). Such comparison is fair enough to determine how well complex queries improve the performance of translation-base methods. Introducing other TKGC baselines leads to unfair comparison, and unable to investigate whether training on temporal complex queries is necessary or not.\n\nAnyway, we present the Table 1 in pdf in global rebuttal section, which is more complete, comparing TFLEX and SOTA TKGC methods. The table shows that TFLEX is competitive with translation-based methods, but it doesn't outperform the SOTA TKGC methods like ChronoR and TuckERT.\n\nHowever, the result doesn't affect the novelty and contribution of this paper. Please note that the projection operator of TFLEX is as simple as TTransE, not further optimized for TKGC tasks only. Upgrading the projection operator to outperform SOTA TKGC methods remains a future work. Besides, previous work GQE[1] did not compare to any KGC methods when the paper proposed complex logical reasoning task against KGC for the first time. And the following works including Query2box[2], BetaE[3], ConE[4], none of them compares to KGC methods.\n\n- [1] Embedding Logical Queries on Knowledge Graphs.\n- [2] Query2box: Reasoning Over Knowledge Graphs In Vector Space Using Box Embeddings.\n- [3] Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs.\n- [4] ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs.\n\n\n## Miscellaneous:\n\n> The presentation of the content.\n\nThanks for pointing out. We will move more content to the main body in the revision.\n\n> Non-anonymous email in the supplementary material.\n\nSorry for the mistake. The email belongs to a third-party package author. We will remove it.\n"}]

Author rebuttal:

Dear Reviewers,

We would like to express our sincere gratitude to the reviewers for their thoughtful and constructive feedback on our submission. We greatly appreciate the time and effort dedicated to evaluating our work, and we are excited to engage in this rebuttal process to address the raised concerns and comments.

Below, we provide the response for the points that mostly concerned.

> [@BpBt, @NJDL, @Zrqw] Existing literature for temporal KG completion and comparison with related TKGC works.

We select some related TKGC works that we have already read before or cited by the survey (Temporal Knowledge Graph Completion: A Survey.) mentioned by reviewer NJDL. We would like to add the following sentences to the related work section.

```md
TKGC task aims at inferencing new facts in the TKGs.
Existing TKGC methods could be categorized to (1) tensor decomposition [1,2,3], (2) timestamp-based transformation [4,5,6,7,8], (3) dynamic embedding [9,10,11,12], (4) Markov process models [13,14], (5) autoregressive models [15,16,17], (6) others [18,19] and so on. Among these works, most of them only confined to the one-hop link prediction task. Some works [12,15,16,17,19] can perform multi-hop reasoning via a path consisting of connected quartets. But they cannot answer logical queries that involve multiple logical operations involving conjunction, negation and disjunction. In this paper, we focus on the temporal complex query answering task, which is more challenging than TKGC task.
```

> [@Zrqw] A separate table comparing TFLEX and state-of-the-art TKGC on a completion task.

we attach the table in the pdf.

> [@BpBt, @NJDL, @Zrqw] Improvement of readability

In order to improve the readability of the paper, we plan to revise as follows:


1. More readable notations in the formulas. More explanations about the symbols. [#our response to NJDL and Zrqw].
2. A leading sentence in the method section.
   ```md
   In this section, we replace the variables in the query formulation with temporal feature-logic embeddings, and perform logical operations via neural networks. We first introduce the temporal feature-logic embedding for entities, timestamps, and queries in Section 4.1. Afterwards, we introduce logical operators in Section 4.2 and how to train the model in Section 4.3.
   ```
3. A short introduction to fuzzy logic to be presented in method section. Please refer to our response to NJDL for detail. [#our response to NJDL]
4. A brief introduction to query types in the dataset setting in experiment section [#our response to Zrqw].
5. Streamlining appendix content and simplifying subsections. [#our response to BpBt]

------

In conclusion, we once again thank the reviewers for their valuable insights and feedback. We have carefully considered all the comments and suggestions and have made corresponding revisions to improve the quality and clarity of our work. We believe that our paper contributes significantly to the field by presenting a novel approach for complex logical reasoning over temporal knowledge graphs. We are confident that the revisions we have made address the reviewers' concerns. We look forward to the opportunity to present our findings and engage in insightful discussions with the community.
Thank all for the time and consideration.

------

References:

- [1] Tensor Decomposition-Based Temporal Knowledge Graph Embedding.
- [2] Tensor decompositions for temporal knowledge base completion.
- [3] Tucker decomposition-based Temporal Knowledge Graph Completion.
- [4] ChronoR: Rotation Based Temporal Knowledge Graph Embedding
- [5] Deriving validity time in knowledge graph.
- [6] Leveraging Static Models for Link Prediction in Temporal Knowledge Graphs.
- [7] TeRo: A Time-aware Knowledge Graph Embedding via Temporal Rotation.
- [8] HyTE: Hyperplane-based Temporally aware Knowledge Graph Embedding.
- [9] Temporal Knowledge Graph Completion Based on Time Series Gaussian Embedding.
- [10] DyERNIE: Dynamic Evolution of Riemannian Manifold Embeddings for Temporal Knowledge Graph Completion.
- [11] Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs.
- [12] TeMP: Temporal Message Passing for Temporal Knowledge Graph Completion.
- [13] RTFE: A Recursive Temporal Fact Embedding Framework for Temporal Knowledge Graph Completion.
- [14] Learning Dynamic Embeddings for Temporal Knowledge Graphs.
- [15] Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs.
- [16] Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning.
- [17] Learning Neural Ordinary Equations for Forecasting Future Links on Temporal Knowledge Graphs.
- [18] Explainable Subgraph Reasoning for Forecasting on Temporal Knowledge Graphs.
- [19] Learning to Walk across Time for Interpretable Temporal Knowledge Graph Completion.


Meta-review: The paper introduces TFLEX, a novel embedding framework to embed complex temporal queries over temporal knowledge graphs to perform multi-hop reasoning with time constraints on TKGs. TFLEX uses fuzzy logic to model complex logical queries and extends to include three temporal operators (After, Before and Between). They provided three new TKG datasets and compared them with multiple benchmarks for experimental verification in order to better validate the reliability of their framework. As a result, TFLEX is considered by reviewers a valuable addition to the field of learning temporal graphs.
Decision: Accept (poster)