Title: TFLEX: Temporal Feature-Logic Embedding Framework for Complex Reasoning over Temporal Knowledge Graph
Venue: NeurIPS 2023
Accepted: yes
Keywords: Temporal Knowledge Graph Reasoning

Temporal Knowledge Graph Embedding

Temporal Knowledge Graph

Temporal Logic

Knowledge Graph Reasoning

Knowledge Graph Embedding

Knowledge Graph

Machine Learning
Abstract: Multi-hop logical reasoning over knowledge graph plays a fundamental role in many artificial intelligence tasks.  Recent complex query embedding methods for reasoning focus on static KGs, while temporal knowledge graphs have not been fully explored.  Reasoning over TKGs has two challenges: 1. The query should answer entities or timestamps; 2. The operators should consider both set logic on entity set and temporal logic on timestamp set.

To bridge this gap, we introduce the multi-hop logical reasoning problem on TKGs and then propose the first temporal complex query embedding named Temporal Feature-Logic Embedding framework (TFLEX) to answer the temporal complex queries.  Specifically, we utilize fuzzy logic to compute the logic part of the Temporal Feature-Logic embedding, thus naturally modeling all first-order logic operations on the entity set.  In addition, we further extend fuzzy logic on timestamp set to cope with three extra temporal operators (**After**, **Before** and **Between**).

Experiments on numerous query patterns demonstrate the effectiveness of our method.

TFLEX: Temporal Feature-Logic Embedding Framework for Complex Reasoning over Temporal Knowledge Graph

Abstract

Multi-hop logical reasoning over knowledge graph plays a fundamental role in many artificial intelligence tasks. Recent complex query embedding methods for reasoning focus on static KGs, while temporal knowledge graphs have not been fully explored. Reasoning over TKGs has two challenges: 1. The query should answer entities or timestamps; 2. The operators should consider both set logic on entity set and temporal logic on timestamp set. To bridge this gap, we introduce the multi-hop logical reasoning problem on TKGs and then propose the first temporal complex query embedding named Temporal Feature-Logic Embedding framework (TFLEX) to answer the temporal complex queries. Specifically, we utilize fuzzy logic to compute the logic part of the Temporal Feature-Logic embedding, thus naturally modeling all first-order logic operations on the entity set. In addition, we further extend fuzzy logic on timestamp set to cope with three extra temporal operators ( After , Before and Between ). Experiments on numerous query patterns demonstrate the effectiveness of our method.

1 Introduction

Multi-hop logical reasoning over knowledge graphs (KGs) is a fundamental issue in artificial intelligence. It aims to find the answer entities for a first-order logic (FOL) query which involves logical operators (existential quantification ∃ , conjunction ∧ , disjunction ∨ and negation ¬ ). Generally, the query is parsed into computation graph, according to which subgraph matching is executed on KG to find the answers. The computation graph is a directed acyclic graph (DAG) whose nodes represent entity sets, and edges represent logical operators acting on entity sets. However, results are inevitably incorrect as KGs are incomplete and noisy. Besides, the computation complexity will spiral for large-scale KGs or large queries. Therefore, people propose to embed query into low-dimensional space to solve the problem.

Query embedding (QE) learns the embeddings of queries and entities, so that answer entities are close to its queries in the embedding space. It has attracted arising attention, as low-dimension embeddings can model implicit dependency and reduce computation. There follows a series of QE methods, including GQE[1], Query2box[2], BetaE[3], ConE[4], etc. However, existing works only focus

Temporal Complex Query

Computation Graph

And

?

And

Not

TimeAnd

After

Before

be elected as

President of France

Xi Jinping

make a visit

step down from

President of France

make a visit

on queries over static KGs. These methods can neither handle an entity query involving temporal information and operators, nor answer the timestamp set for a temporal query.

Temporal knowledge graph (TKG) augments triples in static knowledge graphs with temporal information, represented as < head, relation, tail, timestamp > named fact. For example, the fact < Angela Merkel, make a visit, China, 2010-07-16 > specifies the time when the event happens. Generally speaking, TKGs are more close to real world than static KGs, because most knowledge needs to be updated with time, and static KGs cannot express this change. In recommendation systems, TKGs are used to model user behaviors, which includes liking, buying, reading, commenting and so on. In financial applications, TKGs involve stock holding behaviors, trading behaviors, and financial events. These applications require reasoning over TKGs to answer complex queries. However, recent researches in TKGs focus on temporal link prediction, which is simply single-hop. A complex logical query involving multiple facts for multi-hop reasoning is not fully explored yet.

To fill the gap, we introduce a temporal multi-hop logical reasoning task over TKGs. The task aims to answer temporal complex queries, which have two main distinctions from existing queries over static KGs. Firstly, the answer sets for queries over TKGs are either entity sets or timestamp sets, while that for existing queries over static KGs can only be entity sets. Secondly, as temporal information is included in the query, temporal operators such as After , Before should be considered apart from FOL operators. To understand this new task, we provide the definition of temporal complex query in Section 3. In addition, an example of a temporal complex query is shown in Figure 1. This example query pertains to the financial event of China's visit and may be of interest to financial analysts.

According to the definition of the temporal complex query, we then generate datasets of temporal complex queries on three popular TKGs, and propose the first temporal complex query embedding framework, Temporal Feature-Logic Embedding framework (TFLEX) to answer these queries. In our framework, embeddings of objects (entity, query, timestamp) are divided into two parts, the entity part, and the timestamp part. The entity part handles the FOL operations over entities, while the timestamp part copes with the temporal operations over timestamps. Each part is further divided into feature and logic components. On the one hand, the computation of the logic components follows fuzzy logic, which enables our framework to handle all FOL operations. On the other hand, feature components are mingled and transformed under the guidance of logic components, thereby integrating logical information into the feature. Moreover, we extend fuzzy logic to support extra temporal operations ( After , Before and Between ) to handle temporal operations in the queries.

The contributions of our work are summarized as follows: (1) For the first time, the definition of the task of multi-hop logical reasoning over TKGs is given. (2) We propose the first multi-hop logical reasoning framework on TKGs, namely Temporal Feature-Logic Embedding framework (TFLEX),

which supports all FOL operations and extra temporal operations ( After , Before and Between ). (3) We generate three new TKG datasets for the task of multi-hop logical reasoning. Experiments on three generated datasets demonstrate the efficacy of the proposed framework. The source code of our framework and the datasets are available online *.

2 Related Work

Complex Query Embedding . It learns embeddings of queries and entities, and the answer entities are close to queries in the embedding space. Existing methods utilize a lot of objects to create the embeddings, such as (1) probability distribution [3, 5] (2) geometric object [1, 2, 4, 6, 7] (3) fuzzy logic [8-10] and (4) others [11-13]. However, existing embedding-based methods are considered on static KGs. They cannot utilize temporal information in the TKGs, and therefore cannot handle temporal queries on a temporal KG. Firstly, static query embeddings (QEs) are built over (s, r, o) triples instead of (s, r, o, t) quartets, thus ignoring the timestamps for temporal complex reasoning. The second reason is the order property of timestamps, which is on the contrary that entities are unordered, leading to that static QEs are unable to handle Before and After temporal logic. In addition, we also notice the semantic conflict in experiments (see section 5.2) when concatenating the geometric embedding (static QE) with the fuzzy embedding (that can handle temporal logic) to promote the static QE to temporal one. Therefore, it is challenging for static QEs to utilize temporal information in the TKGs.

Temporal Knowledge Graph Completion (TKGC) . It aims at inferencing new facts in the TKGs. Existing TKGC methods could be categorized to (1) tensor decomposition [14-16], (2) timestampbased transformation [17-21], (3) dynamic embedding [22-25], (4) Markov process models [26, 27], (5) autoregressive models [28-30], (6) others [31-33] and so on. Among these works, most of them only confined to the one-hop link prediction task, also known as one-hop reasoning. Some works [25, 28-30, 32] can perform multi-hop reasoning via a path consisting of connected quartets. But none of them could answer logical queries that involve multiple logical operations (conjunction, negation and disjunction). In this paper, we focus on the temporal complex query answering task, which is more challenging than TKGC task.

3 Definitions

Temporal Knowledge Graph (TKG) G = {V , R , T , F} consists of entity set V , relation set R , timestamp set T and fact set F = { ( s, r, o, t ) } ⊆ V × R × V × T containing subject-predicateobject-timestamp ( s, r, o, t ) quartets. Without loss of generality, G is a first-order logic knowledge base, where each quartet ( s, r, o, t ) denotes an atomic formula r ( s, o, t ) , with r a predicate and s, o, t its arguments.

Multi-hop Logical Reasoning over TKG is the task to answer Temporal Complex Query q when given a TKG G . We focus on Existential Positive First-Order (EPFO) query [34] over TKG, namely Temporal Complex Query q , which is categorized into entity query and timestamp query. Formally, the query q consists of a target variable A , a non-variable anchor entity set V a ⊆ V , a non-variable anchor timestamp set T a ⊆ T , bound variables V 1 , · · · , V k and T 1 , · · · , T l , logical operations (existential quantification ∃ , conjunction ∧ , disjunction ∨ , identity 1 , negation ¬ ), and extra temporal operations ( After , Before ). After ( t 1 , t 2 ) means t 2 is after t 1 , while Before ( t 1 , t 2 ) means t 2 is before t 1 . Inspired by [2, 3], the disjunctive normal form (DNF) of temporal query q is defined as:

In the equation, the DNF is a disjunction of m conjunctions, where e j 1 ∧· · ·∧ e j n j denotes a conjunction between n j logical atoms, and each e j i denotes a logical atom. We ignore indices in the definition

of e j i to keep the formula clean. The goal of answering the query q is to find the set of entities (or timestamps) J q K that satisfy the query, such that A ∈ J q K iff q [ A ] holds true, where J q K is the answer set of the query q .

Following existing static query embedding works, we introduce Computation Graph , which is a directed acyclic graph (DAG) representing the structure of temporal complex query. Its nodes represent entity/timestamp sets S ⊆ V a ∪ V ∪ T a ∪ T , while directed edges represent logical or relational operations acting on these sets. A computation graph specifies how the reasoning of the query has proceeded on the TKG. Starting from anchor sets, we obtain the answer set after applying operations iteratively on non-answer sets according to the directed edges in the computation graph. The edge types on the computation graph are defined as follows:

Relational Projection P . Given an entity set S 1 ⊆ V , a timestamp set S 2 ⊆ T (or an entity set S 2 ⊆ V for entity projection) and a relation r ∈ R , projection operation maps S 1 and S 2 to another set: S ′ = { ∪ ( v ∈ S 1 ,t ∈ S 2 ) { v ′ | ( v, r, v ′ , t ) ∈ F} , P is entity projection ∪ ( v ∈ S 1 ,v ′ ∈ S 2 ) { t | ( v, r, v ′ , t ) ∈ F} , P is timestamp projection

Intersection I (Union U , etc.). Given entity sets or timestamp sets { S 1 , · · · , S n } , the intersection (union, etc.) operation computes logical intersection (union, etc.) of these sets ∩ n i =1 S i ( ∪ n i =1 S i , etc.).

Complement/Negation C . The complement set of a given set S is ¯ S = { V S, S ⊆ V T S, S ⊆ T

Extended temporal operators f . Given a timestamp set S , extended operators compute a certain set of timestamps S ′ : S ′ = { { t ′ | for some t ′ ∈ T , t ′ > max( S ) } , f is After { t ′ | for some t ′ ∈ T , t ′ < min( S ) } , f is Before

In order to efficiently compute the answer set of a temporal complex query, we consider embedding the query set into a low-dimensional vector space, where the answer set is also represented by a continuous embedding vector. Formally, the Temporal Query Embedding V q of a query q is a continuous embedding vector, generated by executing operations according to the computation graph, starting from the temporal embeddings of anchor entity or timestamp sets. The Temporal Query Answer to the query q is the entity v (or timestamp t ) whose embedding v (or t ) has the smallest distance dist ( v , V q ) (or dist ( t , V q ) ) to the embedding of query q .

4 Method

In this section, we replace the variables in the query formulation with temporal feature-logic embeddings, and perform logical operations via neural networks. We first introduce the temporal feature-logic embedding for entities, timestamps, and queries in Section 4.1. Afterwards, we introduce logical operators in Section 4.2 and how to train the model in Section 4.3.

4.1 Temporal Feature-Logic Embeddings for Queries and Entities

In this section, we design temporal embeddings for queries, entities and timestamps. In general, the answers to queries may be entities or timestamps. Therefore, we consider a part of the embedding as an entity part to answer entities, while the other is the timestamp part to answer timestamps. Formally, the embedding of query set J q K is V q = ( q e f , q e l , q t f , q t l ) where q e f ∈ R d is entity feature, q e l ∈ [0 , 1] d is entity logic, q t f ∈ R d is time feature, q t l ∈ [0 , 1] d is time logic respectively, d is the embedding dimension. The parameter q l is the uncertainty q l ⃗ s +(1 -q l ) ⃗ n of the feature, according to fuzzy logic. An entity v ∈ V is a special query without uncertainty. We propose to represent an entity as the query with logic part 0 , which indicates that the entity's uncertainty is 0 . Formally, the embedding of entity v is v = ( v e f , 0 , 0 , 0 ) , where v e f ∈ R d is the entity feature part and 0 is a d -dimensional vector with all elements being 0. Similarly, the embedding of timestamp t is t = ( 0 , 0 , t t f , 0 ) with entity part and time logic being 0 .

Attention that we introduce vector logic, which is a type of fuzzy logic over vector space, to cope with logical transformation in the vector space. Fuzzy logic is a generalization of Boolean logic, such that the truth value of a logical atom is a real number in [0 , 1] . In comparison, as a generalization

of a real number, the truth value in vector logic is a vector [0 , 1] d in the semantic space. We denote the logical operations in vector logic as AND ( ∧ ) , OR ( ∨ ) , NOT ( ¬ ) , and so on, which receive one or multiple vectors and output one vector as answer. For more details about fuzzy logic, please refer to Appendix A.1.

4.2 Logical Operators for Temporal Feature-Logic Embeddings

In this section, we introduce the designed neural logical operators, including projection, intersection, complement, and all other dyadic operators.

Projection Operator P e and P t . The goal of operator P e is to map an entity set to another entity set under a given relation and a given timestamp, while operator P t outputting timestamp set given relation and two entity queries. We define a function P e : V q , r , V t ↦→ V ′ q in the embedding space to represent P e , and P t : V q 1 , r , V q 2 ↦→ V ′ q for P t respectively. To implement P e and P t , we first represent relations as translations on query embeddings and assign each relation with relational embedding r = ( r e f , r e l , r t f , r t l ) . We follow the assumption of translation-based methods: q o ≈ q s + r + t . As a comparison, static KGE TransE [35] has o ≈ s + r , and temporal KGE TTransE [36] has o t ≈ s t + r . The addition represents a semantic translation starting from the source entity set, following the relation and timestamp conditioning, ending at the target entity set. Therefore, we define P e and P t as:

where MLP : R 4 d → R 4 d is a multi-layer perception network (MLP), + is element-wise addition and g is an activate function to generate q e l ∈ [0 , 1] d , q t l ∈ [0 , 1] d . We use MLP and activation function g ( . ) to make projection operator output a valid query embedding, which shows the closure property of the operator. P e and P t do not share parameters so the MLPs are different. We define g as:

where x [0 : d ] is the slice containing element x i of vector x with index 0 ≤ i < d , σ ( · ) is Sigmoid function and [ · ; · ] is the concatenation operator.

Dyadic Operators . There are two types of dyadic operators for our framework. One for entity set and the other for timestamp set. Each type includes intersection (AND), union (OR), the symmetric difference (XOR), etc. With the help of fuzzy logic, our framework can model all dyadic operators directly. Below we take a unified way to build these operators.

We start from intersection operators I e (on entity set) and I t (on timestamp set). The goal of intersection operator I e ( I t ) is to represent J q K = ∩ n i =1 J q i K based on their entity parts (timestamp parts). Suppose that V q i = ( q e i,f , q e i,l , q t i,f , q t i,l ) is temporal feature-logic embedding for J q i K . We notice that there exists Alignment Rule in the process of reasoning. When performing entity set intersection I e , we should also perform intersection on timestamp parts in order to align the entities into the same time set. The same also holds for timestamp set intersection I t and all other dyadic operators. Therefore, we firstly define the intersection operators as follows:

where AND is the conjunction in fuzzy logic, α i and β i are attention weights. To notice the changes of logic, we compute α e,t i and β e,t i via the following attention mechanism:

qi

Before

After

qi

一

一

+1

q

where MLP e,t 1 , 2 : R 2 d → R d are MLP networks, [ · ; · ] is concatenation. The first self-attention neural network will learn the hidden information from entity logic and leverage to entity feature, while the second one gathers logical information from time logic to time feature. Note that the computation of entity logic, and time logic obeys the law of fuzzy logic, without any extra learnable parameters. In this way, all dyadic operators can be generated from fuzzy logic in our framework. Due to space limitation, we present the union, exclusive or, implication operators and so on in Appendix A.3.

Complement Operators: C e and C t . The aim of C e is to identify the complement of query set J q K such that J ¬ q K = V\ J q K , while C t aims to calculate the complement J ¬ q K = T \ J q K by the time parts. Suppose that V q = ( q e f , q e l , q t f , q t l ) , we define the complement operator C e and C t as:

where f e not ( q f ) = tanh( MLP 3 ([ q e f ; q e l ])) , f t not ( q t f ) = tanh( MLP 4 ([ q t f ; q t l ])) are feature negation functions, two MLP 3 , 4 : R 2 d → R d are MLP networks, NOT is negation in fuzzy logic.

Temporal Operators: After A t , Before B t and Between D t . The operator After A t : V q ↦→ V ′ q (Before B t : V q ↦→ V ′ q ) aims to deduce the timestamps after(before) a given fuzzy time set J q K . Let V q = ( q e f , q e l , q t f , q t l ) , we define A t and B t as:

The entity part does not change after computation because temporal operator only affects the time part (time feature q t f and time logic q t l ). The motivation of computation is illustrated in Figure 2. Since q t l is the uncertainty of time feature q t f , the time part can be viewed as an interval [ q t f -q t l , q t f + q t l ] whose center is q t f and half-length is q t l . The interval is covered by [ q t f -1 , q t f + 1] because the probability q t l < 1 . Then, after interval [ q t f -q t l , q t f + q t l ] is the interval [ q t f + q t l , q t f + 1] whose center is q t f + 1+ q t l 2 and half-length is 1 -q t l 2 , which gives the time part of embedding A t ( V q ) . Similarly, the time part of embedding B t ( V q ) is q t f -1+ q t l 2 (time feature) and 1 -q t l 2 (time logic), which are generated from [ q t f -1 , q t f -q t l ] before [ q t f -q t l , q t f + q t l ] . The operator Between D t : V q 1 , V q 2 ↦→ V ′ q inferences the time set after J q 1 K and before J q 2 K . Therefore, we define Between D t as D t ( V q 1 , V q 2 ) = I t ( A t ( V q 1 ) , B t ( V q 2 )) to output the time between J q 1 K and J q 2 K .

4.3 Learning Temporal Feature-Logic Embeddings

We expect the model to achieve high scores for the answers to the given query q , and low scores for v ′ / ∈ J q K . Therefore, we firstly define a distance function to measure the distance between a given answer embedding and a query embedding, and then we train the model with negative sampling loss.

Distance Function Given an entity embedding v = ( v e f , 0 , 0 , 0 ) , a timestamp embedding t = ( 0 , 0 , t t f , 0 ) and a query embedding V q = ( q e f , q e l , q t f , q t l ) , we define the distance d between the answer and the query q as the sum of the feature distance (between the feature parts) and the logic part (to expect uncertainty to be 0). If the query answers entities, the distance is d e ( v ; V q ) = ∥ v e f -q e f ∥ 1 + q e l . Otherwise, the distance is d t ( t ; V q ) = ∥ t t f -q t f ∥ 1 + q t l for queries answering timestamp set, where ∥ · ∥ 1 is the L 1 norm and + is element-wise addition. The distance function aims to optimize two losses. One is to push the answers to the neighbor of query in the embedding

space. It corresponds to the term L1 distance between answer and query. The other is to reduce the uncertainty of the query (the probability interpretation of the logic part), to make the answers more accurate.

Loss Function Given a training set of queries, we optimize a negative sampling loss

where γ > 0 is a fixed margin, k is the number of negative entities, and σ ( · ) is the sigmoid function. When query q is answering entities (timestamps), v ∈ J q K is a positive entity (timestamp), v ′ i / ∈ J q K is the i -th negative entity (timestamp).

5 Experiments

In this section, we evaluate the ability of TFLEX to reason over TKGs. We first introduce experimental settings in Section 5.1, and then present the experimental results in Section 5.2.

5.1 Experimental Settings

Datasets and Query Generation Experiments are performed on three new datasets generated from standard benchmarks for TKGC: ICEWS14 [37], ICEWS05-15 [37], and GDELT-500 [38] (with statistics in Appendix B.1). We predefined 40 kinds of complex queries for each dataset. The definition of the 40 kinds of complex queries and the query generation process details are described in Appendix B.2. We consider 27 kinds of queries for training and all 40 kinds for evaluation and testing. Please refer to Appendix B.3 for summaries of the final datasets.

To briefly summarize the results, we aggregate groups of queries that can be answered: entities (avg e ), timestamps (avg t ), entities with negation (avg e, C e ), timestamps with negation (avg t, C t ), entities with unseen union (avg {U e } ), timestamps with unseen union (avg {U t } ), and other hybrid unseen structures (avg x ). These groups are inspired by the experiment settings of existing static QEs [1-4]. The detail that which query belongs to which group will be shown in Appendix B.7. Note that the training set only involves 4 groups of queries: avg e , avg t , avg e, C e , and avg t, C t .

Evaluation Given a test query q , for each non-trivial answer v ∈ J q K test -J q K valid of the query q , we rank it against non-answer entities V J q K test (or non-answer timestamps T J q K test if the query is answering timestamps). Then we calculate Mean Reciprocal Rank (MRR) based on the rank. The higher, the better. Please refer to Appendix B.5 for the definition of MRR.

5.2 Main Results

For each group, we report the average MRR in Table 1. The raw MRR results on all query structures for each dataset in detail are presented in Appendix B.7.

How well can TFLEX answer temporal complex queries? . We compare TFLEX with state-of-theart query embeddings Query2box [2], BetaE [3] and ConE [4] on answering entities. Existing query embeddings only handle FOL on entity set, but unable to cope with temporal logic over timestamp set. Therefore, the results of these three methods have to be obtained by ignoring the timestamps, so that the avg t , avg t, C t , avg {U t } and avg x of three methods are zeros. Comparing the results of these methods in Table 1, we can see that TFLEX outperforms all the baselines on all the metrics. These results demonstrate that TFLEX can perform reasoning over TKGs well, at least better than the existing query embeddings.

Ablation on entity part . The variant X(ConE) replaces the entity part of TFLEX with ConE [4] to handle the logic over entity sets. In the entity part, ConE is geometric while TFLEX is fuzzy. The variant performs even worse than TFLEX and ConE. The dropping MRR indicates that the entity part plays an important role in the framework. Considering the performance of ConE, we think there is semantic conflict between the time part of TFLEX and the entity part of ConE. Simply combining static QEs with dynamic QEs is not a clever way to achieve the best performance.

Ablation on time part . The variant FLEX removes the time part of the temporal feature-logic embedding. Then, FLEX can only answer entities. The results show that FLEX slightly outperforms ConE. However, the performance of FLEX is worse than TFLEX with a large margin on all the datasets. Therefore, we conclude that the time part also plays an important role in the framework.

Ablation on feature part . If we remove the entity and timestamp feature, the embeddings of entities and timestamps will crash to zeros. Instead, we consider another way to explore the impact of the feature part. Noticing that some TKGC approaches [36, 39] embed entities and timestamps into the same semantic space, we propose X-1F to merge the entity and timestamp features into one feature. Compared with TFLEX, X-1F achieves higher scores on ICEWS14 and ICEWS05-15, but lower on GDELT. The results imply that unifying the feature of entity and timestamp is potentially beneficial in some datasets.

Ablation on logic part . The variant X-logic removes both entity logic and time logic. It achieves lower scores than TFLEX on all the datasets. This is because the logic part is responsible for reasoning over TKGs. Removing the logic part results in that neural logical operators completely rely on neural network to learn the logic, which is not enough to handle various temporal complex queries.

Out-of-data reasoning . The results on avg {U e } , avg {U t } , avg x support that the framework can reason over unseen entity logic, unseen time logic as well as their hybrid. The entity union and time union operators are not included in the training set, but the framework can still handle them well.

Sensitive analysis . (1) The selection of hyperparameters (embedding dimension d , the margin γ ) has a substantial influence on the effectiveness of TFLEX. We train the model with embedding dimension d ∈ { 300 , 400 , 500 , 600 , 700 , 800 , 900 , 1000 } , the margin γ ∈ { 5 , 10 , 15 , 20 , 25 , 30 , 35 , 40 } . The best performance is achieved when d = 800 and γ = 15 . Too small and too large d, γ both lead to worse results, reported in Appendix B.6. (2) Besides, we also investigate the stability of TFLEX. We train and test for five times with different random seeds and report the error bars in Appendix B.6. The small standard variances demonstrate that the performance of TFLEX is stable.

Comparison between TFLEX and TKGC methods . It's natural to compare TFLEX with SOTA TKGC methods, since all of them can answer one-hop temporal queries. We present the comparison results in Table 2. We can see that TFLEX is competitive with translation-based methods

1.0

before

afteri

8'0

Score

0.6

Similarity

0.4

bPt=Bt(Pt(s,r,o))

0.2

Pt=Pt(s,r,o)

aPt=At(Pt(s,r,o))

0.0

0

50

Timestamp (0=2014-1-1, 365=2014-12-31)

100

150

200

250

300

350

(ConT [43], TTransE [36], HyTE [44], etc.), but it doesn't outperform the SOTA TKGC methods like ChronoR [17] and TuckERT [39]. However, the result doesn't affect the novelty and contribution of this paper. Please note that the projection operator of TFLEX is as simple as TTransE, not further optimized for TKGC tasks only. Upgrading the projection operator to outperform SOTA TKGC methods remains a future work.

Necessity of training on temporal complex queries . Our experiments demonstrate that training on complex queries is necessary to achieve the best performance. We compare with translation-based P e operators (TTransE [36], HyTE [44] and TFLEX's variant TFLEX-1p ) using only one-hop Pe queries for training. We choose TTransE and HyTE because our projection operator is also translation-based ( P e ( V q , r , V t ) ∝ V q + r + V t ). From the result table 3, we observe that TFLEX achieves the best performance when comparing with these translation-based baselines on all datasets. Besides, compared with TFLEX-1p , TFLEX achieves 7.9% relative improvement on average on MRR, which demonstrates that training on complex queries could improve the one-hop query-answering ability.

Effectiveness of neural temporal operators . Wefound that the temporal operators A t and B t change the semantic of predicted timestamp embedding logically. We randomly select an event ( s, r, o ) from ICEWS14 and consider three temporal queries Pt , bPt and aPt . Then, Pt ( = P t ( s, r, o ) ) predicts the date t when this event happened. And bPt (= B t ( P t ( s, r, o )) ) should predict the date before t , while aPt (= A t ( P t ( s, r, o )) ) is supposed to predict the time after t . The output of three queries are time embeddings. Because the time embedding is fuzzy, we score it to all possible Timestamps, and visualize the normalized similarity score distribution over all days in a year, from 0 to 365 in ICEWS14. The higher the score, the more possible the predicted date is the day. We plot the three score distributions in Figure 3. For each distribution, we highlight the periods of the highest scores

with a colored background. These colored blocks represent the most likely happening time interval of the event. We observe that the order of colored blocks (corresponding to the predictions of bPt , Pt , and aPt ) matches the logical meanings of these operators (Before the event, On the event, After the event). The visualization shows that neural temporal operators perform the time transformation correctly. More examples are attached in Appendix D.

Explaining answers with temporal feature-logic framework . We take query Pe2 for example. Given temporal query Pe2 : q [ V ? ] = V ? , ∃ V a , r 1 ( e 1 , V a , t 1 ) ∧ r 2 ( V a , V ? , t 2 ) , let's try an example which can be written as: On 2014-04-04, who consulted the man who was appealed to or requested by the Head of Government (Latvia) on 2014-08-01? In this example, e 1 is "Head of Government (Latvia)", r 1 is "Make an appeal or request", t 1 is "2014-08-01", r 2 is "consult -1 ", and t 2 is "201404-04". Then we use TFLEX to execute the query and get answers. We classify the answers into easy, hard and wrong ones. The easy answer is the correct answer that appears in the training set, while the hard answer is the correct answer that exists in the testing set instead of training set.

We present five most likely answers for interpretation in Figure 4. From the table we observe that TFLEX ranks easy answers "François Hollande", "Taavi Rõivas" and hard answer "Andris Berzins" higher than wrong answers "Angela Merkel" and "Head of Government (Latvia)". This shows that TFLEX successfully finds the hard answer by performing complex reasoning, and distinguishes the right answers from the wrong ones.

We provide 36 examples in Appendix E, including the visualization and intermediate explanation of answers for each query structure.

6 Conclusion

In this paper, we firstly define a temporal multi-hop logical reasoning task on temporal knowledge graphs. Then we generate three new TKG datasets and propose the Temporal Feature-Logic embedding framework, TFLEX, to handle temporal complex queries in datasets. Fuzzy logic is used to guide all FOL transformations on the logic part of embeddings. We also further extended fuzzy logic to implement extra temporal operators ( Before , After and Between ). To the best of our knowledge, TFLEX is the first framework to support multi-hop complex reasoning on temporal knowledge graphs. Experiments on benchmark datasets demonstrate the efficacy of the proposed framework in handling different operations in complex queries.

Acknowledgments and Disclosure of Funding

This work is supported by the National Science Foundation of China (Grant No. 62176026) and Beijing Natural Science Foundation (M22009); Engineering Research Center of Information Networks, Ministry of Education. We would like to express our sincere gratitude to our corresponding authors, Prof. Haihong E and Dr. Chengjin Xu, for their guidance, expertise, and unwavering support throughout the project. Furthermore, we would like to thank Fenglong Su, Gengxian Zhou, Tianyi Hu, Ningyuan Li, Mingzhi Sun, and Haoran Luo for their individual contributions and collaboration in various aspects of this project. Their expertise and dedication have significantly enriched the final outcome. Additionally, we extend our thanks to the anonymous reviewers for their time and efforts in reviewing our work. Their constructive feedback and comments have been instrumental in improving the overall clarity and rigor of this research.

Broader Impact

Multi-hop reasoning makes the information stored in TKGs more valuable. With the help of multi-hop reasoning, we can digest more hidden and implicit information in TKGs. It will broaden and deepen KG applications in various fields, such as question answering, recommend systems, and information retrieval. It may also bring about risks exposing unexpected personal information on public data.

Finance temporal knowledge graph is a good example to illustrate the broader impact of multi-hop reasoning. In the financial field, the information stored in TKGs is very sensitive. The one-hop reasoning can complete the hidden financial transaction, while the multi-hop reasoning can help to detect the fraud. The After operator could also be used to predict the future financial transaction. People may take the advantages of logical reasoning to digest financial factor to obtain excess returns.

Military temporal knowledge graph is another example. With the help of multi-hop logical reasoning, we may predict the future military strategy of a country. Besides, with the fuzzy completion of the hidden military information, we can also detect the hidden militarily moves, which may save thousands of soldiers' lives.

The last example is social temporal knowledge graph. The behaviors of people are left and stored in TKGs. With the help of logical reasoning, we may predict a short future of a person. For example, the query may answer the evolution of user profiles: how long may a person transfer to another role, from student to worker, becoming a parent, or being a grandparent. Tracking the user's identity change can provide super benefits for merchants' advertising. Detecting the role of a person is also helpful to provide more personalized services.

However, we should agree that the multi-hop reasoning is still at an extremely early stage, though it may bring about risks. Therefore, we should pay more attention to the security of TKGs and the privacy of users at the mean time when we explore the technology of multi-hop reasoning over TKGs.

Limitation & Future Work

In this section, we list 4 limitations and talk about the possible future work.

Limitation 1: The temporal operators are not enough. . We define three extra temporal operators ( Before , After and Between ) in query generation. However, there exists more temporal operators in the real world. For example, Allen [49] defined 13 types of temporal relations represented by two intervals, including before/after, during/contains, overlaps/overlapped-by, meets/met-by, starts/startedby, finishes/finished-by and equal. In the future we would like to promote these temporal operators to TKGs.

Limitation 2: The temporal embedding could improve. . In this paper, the embedding of the timestamp is randomly initialized and finally learned by the model via logical advantages. Such embedding ignores the order of different timestamps. The order property is learned by the After and Before operators, which may be not enough. We recall that in the field of NLP, positional embeddings also have order features, which may be used for the construction of timestamp embeddings.

Limitation 3: The query generation is time-consuming. . There are 40 predefined query structures in our query generation module. Each query structure has 10k+ queries for training. With the scale of TKGs increasing, the number of queries will also increase, even 40x faster. Therefore, we need to find a more efficient way to generate queries for large scale TKGs.

Limitation 4: The MRR and Hits@k are weak. . The MRR and Hits@k evaluation metric may not inflect the performance of multi-hop reasoning. We observe that some queries have lots of answers. When the number of answers is larger than k, the MRR and Hits@k will be low, even if all answers are correctly ranked at top. Because the right answers that ranked after k are labeled as false. The Hits@k decreases with the increase number of right answers, which is not reasonable. This disadvantage prevents us from comparing the performance across different datasets. In this paper, GDELT is much denser, and the count of right answers is larger than the other two datasets. Therefore, the MRR of GDELT is lower than the other two datasets. It can also be verified that on the average MRR, group avg t is lower than group avg e . The reason is that the number of right answers of group avg t is larger than that of group avg e , which can be seen from the statistic of average answer count of queries in Table 10. In the future, there should be a more reasonable evaluation metric for multi-hop reasoning.

References

Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, and Jure Leskovec. Embedding logical queries on knowledge graphs. Advances in Neural Information Processing Systems , 31:2026-2037, 2018.

H Ren, W Hu, and J Leskovec. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. In International Conference on Learning Representations (ICLR) , 2020.

Hongyu Ren and Jure Leskovec. Beta embeddings for multi-hop logical reasoning in knowledge graphs. Neural Information Processing Systems (NeurIPS) , 2020.

Zhanqiu Zhang, Jie Wang, Jiajun Chen, Shuiwang Ji, and Feng Wu. Cone: Cone embeddings for multi-hop reasoning over knowledge graphs. Advances in Neural Information Processing Systems , 34, 2021.

Nurendra Choudhary, Nikhil Rao, Sumeet Katariya, Karthik Subbian, and Chandan Reddy. Probabilistic entity representation model for reasoning over knowledge graphs. Advances in Neural Information Processing Systems , 34, 2021.

Nurendra Choudhary, Nikhil Rao, Sumeet Katariya, Karthik Subbian, and Chandan K Reddy. Selfsupervised hyperboloid representations from logical queries over knowledge graphs. In Proceedings of the Web Conference 2021 , pages 1373-1384, 2021.

Lihui Liu, Boxin Du, Heng Ji, ChengXiang Zhai, and Hanghang Tong. Neural-answering logical queries on knowledge graphs. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1087-1097, 2021.

Xueyuan Lin, Haihong E, Gengxian Zhou, Tianyi Hu, Li Ningyuan, Mingzhi Sun, and Haoran Luo. Flex: Feature-logic embedding framework for complex knowledge graph reasoning. ArXiv , abs/2205.11039, 2023. URL https://api.semanticscholar.org/CorpusID:248986747 .

Erik Arakelyan, Daniel Daza, Pasquale Minervini, and Michael Cochez. Complex query answering with neural link predictors. arXiv preprint arXiv:2011.03459 , 2020.

Francois P. S. Luus, Prithviraj Sen, Pavan Kapanipathi, Ryan Riegel, Ndivhuwo Makondo, Thabang Lebese, and Alexander G. Gray. Logic embeddings for complex query answering. ArXiv , abs/2103.00418, 2021.

Haitian Sun, Andrew Arnold, Tania Bedrax Weiss, Fernando Pereira, and William W Cohen. Faithful embeddings for knowledge base queries. Advances in Neural Information Processing Systems , 33, 2020.

Dinesh Garg, Shajith Ikbal, Santosh K Srivastava, Harit Vishwakarma, Hima Karanam, and L Venkata Subramaniam. Quantum embedding of knowledge for reasoning. Advances in Neural Information Processing Systems , 32:5594-5604, 2019.

Santosh Kumar Srivastava, Dinesh Khandelwal, Dhiraj Madan, Dinesh Garg, Hima Karanam, and L Venkata Subramaniam. Inductive quantum embedding. Advances in Neural Information Processing Systems , 33, 2020.

Lifan Lin and Kun She. Tensor decomposition-based temporal knowledge graph embedding. 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI) , pages 969-975, 2020. URL https://api.semanticscholar.org/CorpusID:229701707 .

Timothée Lacroix, Guillaume Obozinski, and Nicolas Usunier. Tensor decompositions for temporal knowledge base completion. ArXiv , abs/2004.04926, 2020. URL https://api.semanticscholar. org/CorpusID:214390104 .

Pengpeng Shao, Guohua Yang, Dawei Zhang, Jianhua Tao, Feihu Che, and Tong Liu. Tucker decompositionbased temporal knowledge graph completion. Knowl. Based Syst. , 238:107841, 2020. URL https: //api.semanticscholar.org/CorpusID:226965423 .

Ali Reza Sadeghian, Mohammadreza Armandpour, Anthony Colas, and Daisy Zhe Wang. Chronor: Rotation based temporal knowledge graph embedding. In AAAI Conference on Artificial Intelligence , 2021. URL https://api.semanticscholar.org/CorpusID:232269660 .

Julien Leblay and Melisachew Wudage Chekol. Deriving validity time in knowledge graph. Companion Proceedings of the The Web Conference 2018 , 2018. URL https://api.semanticscholar.org/ CorpusID:13846713 .

Wessel Radstok and Melisachew Wudage Chekol. Leveraging static models for link prediction in temporal knowledge graphs. 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI) , pages 1034-1041, 2021. URL https://api.semanticscholar.org/CorpusID: 235670110 .

Chengjin Xu, M. Nayyeri, Fouad Alkhoury, Hamed Shariat Yazdi, and Jens Lehmann. Tero: A timeaware knowledge graph embedding via temporal rotation. In International Conference on Computational Linguistics , 2020. URL https://api.semanticscholar.org/CorpusID:222124934 .

Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. Hyte: Hyperplane-based temporally aware knowledge graph embedding. In Proceedings of the 2018 conference on empirical methods in natural language processing , pages 2001-2011, 2018.

Chengjin Xu, M. Nayyeri, Fouad Alkhoury, Hamed Shariat Yazdi, and Jens Lehmann. Temporal knowledge graph completion based on time series gaussian embedding. In International Workshop on the Semantic Web , 2019. URL https://api.semanticscholar.org/CorpusID:218900866 .

Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. Dyernie: Dynamic evolution of riemannian manifold embeddings for temporal knowledge graph completion. ArXiv , abs/2011.03984, 2020. URL https://api.semanticscholar.org/CorpusID:226262324 .

Rakshit S. Trivedi, Hanjun Dai, Yichen Wang, and Le Song. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In International Conference on Machine Learning , 2017. URL https: //api.semanticscholar.org/CorpusID:8040343 .

Jiapeng Wu, Mengyao Cao, Jackie Chi Kit Cheung, and William L. Hamilton. Temp: Temporal message passing for temporal knowledge graph completion. In Conference on Empirical Methods in Natural Language Processing , 2020. URL https://api.semanticscholar.org/CorpusID: 222177069 .

Youri Xu, Haihong E, Meina Song, Wenyu Song, Xiaodong Lv, Wang Haotian, and Yang Jinrui. Rtfe: A recursive temporal fact embedding framework for temporal knowledge graph completion. In North American Chapter of the Association for Computational Linguistics , 2020. URL https: //api.semanticscholar.org/CorpusID:222067007 .

Siyuan Liao, Shangsong Liang, Zaiqiao Meng, and Qiang Zhang. Learning dynamic embeddings for temporal knowledge graphs. Proceedings of the 14th ACM International Conference on Web Search and Data Mining , 2021. URL https://api.semanticscholar.org/CorpusID:232126110 .

Woojeong Jin, Meng Qu, Xisen Jin, and Xiang Ren. Recurrent event network: Autoregressive structure inference over temporal knowledge graphs. In Conference on Empirical Methods in Natural Language Processing , 2019. URL https://api.semanticscholar.org/CorpusID:222205878 .

Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. Temporal knowledge graph reasoning based on evolutional representation learning. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2021. URL https://api.semanticscholar.org/CorpusID:233324265 .

Zhen Han, Zifeng Ding, Yunpu Ma, Yujia Gu, and Volker Tresp. Learning neural ordinary equations for forecasting future links on temporal knowledge graphs. In Conference on Empirical Methods in Natural Language Processing , 2021. URL https://api.semanticscholar.org/CorpusID: 237304520 .

Zhen Han, Peng Chen, Yunpu Ma, and Volker Tresp. Explainable subgraph reasoning for forecasting on temporal knowledge graphs. In International Conference on Learning Representations , 2021. URL https://api.semanticscholar.org/CorpusID:235614395 .

Jaehun Jung, Jinhong Jung, and U Kang. Learning to walk across time for interpretable temporal knowledge graph completion. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , 2021. URL https://api.semanticscholar.org/CorpusID:236979995 .

Ruijie Wang, Zheng Li, Dachun Sun, Shengzhong Liu, Jinning Li, Bing Yin, and Tarek Abdelzaher. Learning to sample and aggregate: Few-shot reasoning over temporal knowledge graphs. Advances in Neural Information Processing Systems , 35:16863-16876, 2022.

Nilesh Dalvi and Dan Suciu. Efficient query evaluation on probabilistic databases. The VLDB Journal , 16: 523-544, 2007.

Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS 2013. , 2013.

Tingsong Jiang, Tianyu Liu, Tao Ge, Lei Sha, Baobao Chang, Sujian Li, and Zhifang Sui. Towards timeaware knowledge graph completion. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , pages 1715-1724, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL https://aclanthology.org/C16-1161 .

Elizabeth Boschee, Jennifer Lautenschlager, Sean O'Brien, Steve Shellman, James Starz, and Michael Ward. ICEWS Coded Event Data, 2015. URL https://doi.org/10.7910/DVN/28075 .

Kalev Leetaru and Philip A. Schrodt. Gdelt: Global data on events, location, and tone. ISA Annual Convention , 2013.

Pengpeng Shao, Guohua Yang, Dawei Zhang, Jianhua Tao, Feihu Che, and Tong Liu. Tucker decompositionbased temporal knowledge graph completion. Knowl. Based Syst. , 238:107841, 2020.

Zixuan Li, Xiaolong Jin, Wei Li, Saiping Guan, Jiafeng Guo, Huawei Shen, Yuanzhuo Wang, and Xueqi Cheng. Temporal knowledge graph reasoning based on evolutional representation learning. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2021. URL https://api.semanticscholar.org/CorpusID:233324265 .

Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of the International Conference on Learning Representations (ICLR) 2015 , May 2015.

Seyed Mehran Kazemi and David L. Poole. Simple embedding for link prediction in knowledge graphs. In Neural Information Processing Systems , 2018. URL https://api.semanticscholar.org/ CorpusID:3674966 .

Yunpu Ma, Volker Tresp, and Erik A. Daxberger. Embedding models for episodic knowledge graphs. J. Web Semant. , 59, 2018. URL https://api.semanticscholar.org/CorpusID:54444869 .

Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. HyTE: Hyperplane-based temporally aware knowledge graph embedding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 2001-2011, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1225. URL https://aclanthology.org/ D18-1225 .

Alberto García-Durán, Sebastijan Dumancic, and Mathias Niepert. Learning sequence encoders for temporal knowledge graph completion. In Conference on Empirical Methods in Natural Language Processing , 2018. URL https://api.semanticscholar.org/CorpusID:52183483 .

Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker, and Pascal Poupart. Diachronic embedding for temporal knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3988-3995, 2020.

Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In International Conference on Neural Information Processing , 2016. URL https://api.semanticscholar.org/CorpusID:2687749 .

Cunchao Zhu, Muhao Chen, Changjun Fan, Guangquan Cheng, and Yan Zhan. Learning from history: Modeling temporal knowledge graphs with sequential copy-generation networks. In AAAI Conference on Artificial Intelligence , 2020. URL https://api.semanticscholar.org/CorpusID:229180723 .

James F. Allen. Maintaining knowledge about temporal intervals. Commun. ACM , 26(11):832-843, nov 1983. ISSN 0001-0782. doi: 10.1145/182.358434. URL https://doi.org/10.1145/182. 358434 .

Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR) , 2015.

Appendix

A More Details about Neural Operators

A.1 Fuzzy Logic

Fuzzy logic is a generalization of Boolean logic, which is based on the idea that the truth of a statement can be expressed as a number between 0 and 1. It is a mathematical framework for reasoning with imprecise information. In fuzzy logic, there are also logical operators, such as conjunction, disjunction, negation, implication, equivalence, and exclusive or. Most popular fuzzy logic operators are defined as follows ( a, b ∈ [0 , 1] ):

conjunction : a ∧ b = min( a, b ) . disjunction : a ∨ b = max( a, b ) .

negation : ¬ a = 1 -a . exclusive or : a ⊕ b = min( a, 1 -b ) + min(1 -a, b ) .

implication : a → b = max(1 -a, b ) . equivalence : a ↔ b = min( a, b ) + max(1 -a, 1 -b ) .

However, the fuzzy logic operators are defined over real space, not suitable for reasoning over vector space. We expect that the fuzzy logic operators can be executed by matrix computation. Because the embedding space of knowledge graph is a vector space. In order to cope with tensor computation in neural networks, we introduce the fuzzy logic operators over vector space in the following, which is called vector logic.

A.2 Vector Logic

Vector logic is an elementary logical model based on matrix algebra. In vector logic, true values are mapped to the vector, and logical operators are executed by matrix computation.

Truth Value Vector Space V 2 . A two-valued vector logic uses two d -dimensional ( d ≥ 2 ) column vectors ⃗ s and ⃗ n to represent true and false in the classic binary logic. The two vectors ⃗ s and ⃗ n are real-valued, normally orthogonal to each other, and normalized vectors, i.e. , ∥ ⃗ s ∥ = 1 , ∥ ⃗ n ∥ = 1 . Truth value vector space are generated by V 2 = { ⃗ s, ⃗ n } , and operations on vectors in truth value space is based on scalar product.

Operators . The basic logical operators are associated with their own matrices by vectors in truthvalue vector space. Two common types of operators are monadic and dyadic.

Monadic Operators are functions: V 2 → V 2 . Two examples are Identity I = ⃗ s⃗ s T + ⃗ n⃗ n T and Negation N = ⃗ n⃗ s T + ⃗ s⃗ n T such that I⃗ s = ⃗ s, I⃗ n = ⃗ n, N⃗ n = ⃗ s, N⃗ s = ⃗ n . Note that the truth tables of I and N fulfill the real logical rules of identity and negation in classic boolean logic.

Dyadic operators are functions: V 2 ⊗ V 2 → V 2 , where ⊗ denotes Kronecker product. Dyadic operators include conjunction C , disjunction D , implication IMPL, equivalence E , exclusive or XOR, etc. For example, the conjunction between two logical propositions ( p ∧ q ) is performed by C ( ⃗ u ⊗ ⃗ v ) , where C = ⃗ s ( ⃗ s ⊗ ⃗ s ) T + ⃗ n ( ⃗ s ⊗ ⃗ n ) T + ⃗ n ( ⃗ n ⊗ ⃗ s ) T + ⃗ n ( ⃗ n ⊗ ⃗ n ) T . It can be verified that C ( ⃗ s ⊗ ⃗ s ) = ⃗ s, C ( ⃗ s ⊗ ⃗ n ) = C ( ⃗ n ⊗ ⃗ s ) = C ( ⃗ n ⊗ ⃗ n ) = ⃗ n . Dyadic operators which correspond to logical operations in classic binary logic are defined by its formulation to perform logical operations on truth value vectors. Their associated matrices has d 2 rows and d columns.

Many-valued Two-dimensional Logic . Many-valued logic is introduced to include uncertainties in the logic vector. Weighting ⃗ s and ⃗ n by probabilities, uncertainties are introduced: f ⃗ = ϵ⃗ s + δ⃗ n , where ϵ, δ ∈ [0 , 1] , ϵ + δ = 1 . Besides, operations on vectors can be simplified to computation on the scalar of these vectors. For example, given two vectors ⃗ u = α⃗ s + β⃗ n, ⃗ v = α ′ ⃗ s + β ′ ⃗ n , we have:

A.3 Neural Dyadic Operators

In this section, we show that all dyadic operators are generated from fuzzy logic in our framework. Below we take union operator for example. We define entity union operator U e and time union operator U t according to Alignment Rule 4.2 as follows:

where AND , OR are the conjunction, disjunction operators in fuzzy logic respectively, α i and β i are attention weights, as designed in intersection operators. In the time part of entity union operator U e and the entity part of time union operator U t , we follows Alignment Rule 4.2 to perform intersection. Besides, please be aware that each operator owns its MLPs and parameters. These operators do not share parameters with each other.

For n queries to intersection/union, the AND and OR can be written as:

The equations come from the definition of AND and OR in vector logic. The cost of OR operator's implementation is not as expensive as it seems to be. In fact, it's O ( n ∗ d ) complexity, where d is the embedding dimension. We provide the proof in next section.

Other dyadic operators can be generated in the same way. Since listing all operators is boring, we provide the last example. Then, the entity implication operator L e and time implication operator L t are defined as follows:

A.4 Theoretical Analysis

To understand why the feature-logic framework works, we have the following proposition, which shows that the designed intersection and union operators obey commutative law of real logical operations. The proofs are presented in next section in Appendix A.5:

Proposition 1. Commutativity : Given Temporal Feature-Logic embedding V q a , V q b , we have I e ( V q a , V q b ) = I e ( V q b , V q a ) and U e ( V q a , V q b ) = U e ( V q b , V q a ) , I t ( V q a , V q b ) = I t ( V q b , V q a ) and U t ( V q a , V q b ) = U t ( V q b , V q a ) .

A.5 Proofs

In this part, we prove that TFLEX has the property of commutativity. Besides, we also show that the computation complexity of OR operation is linear to the number of queries.

Proof. (Proposition 1) Commutativity :

For the intersection operations, as the calculations of I e and I t are identical, here, we only prove that I e complies with commutative law. The entity feature part and the time feature part of the result are computed as a weighted summation of each query's corresponding parts. Since addition is commutative and the attention weights do not concern the order of calculations, both feature parts' calculations are commutative.

Then, we discuss the logic parts. The logic parts only include the AND in fuzzy logic which, essentially, is just the multiplication of each element by the definition provided above. Because multiplication is surely commutative, the calculation of either entity logic part or time logic part is commutative. Thus the intersection operation I e ( I t ) is commutative.

As for U e and U t , their feature parts have the same form of weighted summation as the intersection operations do. Thus, the feature parts of both U e and U t comply to commutative law. Also, the time logic part of U e and the entity logic part of U t solely concern AND operator which has been proved commutative before. The OR operator, by definition, gives the aggregation of different accumulative parts each of which is commutative itself. Also, the multiplications in each of the summations are commutative. Hence, OR operation is invariant to the order of calculations, which finally gives the calculations of entity logic part of U e and the time logic part of U t are commutative. Then we can naturally affirm that U e and U t are commutative as well.

Proof. Additionally, we prove that the cost of OR operator's implementation is not as expensive as it seems to be. The process of computation can be described as follows:

As we implement OR step by step on a series of n queries, the loop goes n-1 times in total. Assuming the embedding dimension of a query is d , we can have the cost of OR is O( n*d ).

B More Details about Experiments

In this section, we show more details about our experiments. Firstly, we introduce the origin datasets in Section B.1. Then, we describe the details on how we define the query structure and how to sample queries in Section B.2. With the generated queries, we construct three datasets and report their statistics in Section B.3. We also show the details of our model implementation in Section B.4 and evaluation protocol in Section B.5. Besides, we show more experimental data for sensitive analysis in Section B.6. Finally, we present the detail metrics of main results in Section B.7.

B.1 Details of Origin Datasets

To construct the reasoning dataset, we use three datasets of temporal KGs that have become standard benchmarks for TKGC: ICEWS14 [37], ICEWS05-15 [37], and GDELT-500 [38]. The two subsets of ICEWS are generated by Boschee et al. [37]: ICEWS14 corresponding to the facts in 2014 and ICEWS05-15 corresponding to the facts between 2005 and 2015. GDELT-500 generated by Leetaru and Schrodt [38] corresponds to the facts from April 1, 2015, to March 31, 2016. The statistics of the dataset are shown in Table 8.

B.2 Details of Query Generation

The Design of Query Structure Schema . Previous query embedding methods share the same query structure schema (Table 4) to perform complex logical reasoning over static knowledge graph. When it comes to dynamic knowledge graph (TKG), the static schema is not sufficient to capture the temporal reasoning. In this paper, we propose a novel query structure schema for reasoning. The schema is composed of basic functions and query structures. The basic functions are defined in Table 5. The query structures are shown in Table 6 with visualization in Figure 5 and Figure 6. The query structures are based on the basic functions: And , Or , Not , EntityProjection , TimeProjection , TimeAnd , TimeOr , TimeNot , Before , and After . Of course, users can define more basic functions or query structures to extend the schema to multi-modal KGs, hyper-relation KGs and so on. It is a flexible schema overall.

In order to keep a similar experiment settings with previous static query embeddings, the query structures are further aggregated into groups, shown in Table 7. The groups avg e , avg t , avg e, C e , avg t, C t are for training and testing. Besides, extra groups avg {U e } , avg {U t } , avg x are for evaluation and testing only.

In implementation, we define the basic functions and query structures as functions in python, which are named Complex Query Function (CQF) . There are lots of advantages of using python functions to define query structures. Firstly, the functions are easy to understand and debug. Secondly, the functions are easy to be reused in the dataset-sampling process and model-training process. Thirdly,

?

Pe

?

Pt

?

Pe2

?

Pe3

Pe_Pt

And

?

e2i

And

?

e3i

And

?

Pe_e2i

Pt_lPe

Pt_rPe

?

And

TimeAnd

Pe_t2i

And

?

e2i_PeN

Not

And

?

And

Not

Pe_e2i_Pe_NPe

And

?

e2i_N

And

Not

And

?

e3i_N

And

Not

?

And

?

e2i_Pe

And

?

e2i_NPe

Not

Or

?

e2u

And

?

Pe_e2u

Or

?

?

TimeAnd

?

TimeAnd

?

t2i

t3i

TimeAnd

?

TimeAnd

?

TimeNot

TimeAnd

?

t2i_PtN

TimeNot

TimeAnd

Pe_t2i_PtPe_NPt

TimeNot

?

TimeAnd

?

TimeNot

TimeAnd

?

TimeNot

t2i_N

t3i_N

And

?

And

?

And

And

Pt_le2i

Pt_re2i

TimeOr

?

t2u

?

TimeOr

Pe_t2u

Pe_aPt

After

Pe_bPt

Before

TimeAnd

?

After

TimeAnd

Pe_at2i

TimeAnd

?

Before

TimeAnd

Pe_bt2i

TimeAnd

?

After

Before

between

?

?

the functions are easy to be extended to support more complex query structures and reimplement the existing query embedding methods.

As is introduced in Section 4, we use the computation graph to represent the reasoning process. To get the computation graph of a CQF, we use the python interpreter to parse the function to Abstract Syntax Tree (AST), which is a more friendly readable subset of computation graph. In this way, executing the computation graph is equivalent to executing the CQF in python interpreter. Since we have the god privileged access to the AST, we can modify the interpreter to support various dynamic reasoning processes. Below we show how to unify (1) the reasoning of ground-truth, (2) the sampling of anchors, and (3) the reasoning of embedding-based methods by modifying the python interpreter.

Ground-Truth Reasoning Interpreter .

In the reasoning of ground-truth, we need to get all real answers of a query under a given TKG. To achieve this, the first we need to do is to implement the basic functions. We wrap the python built-in set to QuerySet class to store entities and timestamps. The basic functions are implemented as lambda function in python, with input and output as QuerySet. Then, the basic functions are registered as symbols to the python interpreter. When executing the CQF, the python interpreter will call the corresponding basic functions to get the final answer. The answers are finally generated from subgraph matching, which depends on the ground truth in TKG. We show the pseudocode of the ground-truth reasoning interpreter in Figure 7.

Anchors Sampling Interpreter .

The anchor, which may be entity, relation or timestamp, is the input argument of the CQF. The aim of the anchors sampling interpreter is to get the possible anchors and answers of a query under a given TKG. For the sampled anchors, we expect the answer set not empty. Since the answers could be obtained from the ground-truth reasoning interpreter, we only focus on the sampling of anchors in the anchors sampling interpreter.

Given the standard split of edges into training ( F train), validation ( F valid) and test ( F test) sets, we append inverse relations and double the number of edges in the graph. Then we create three graph: G train = {V , R , T , F train } , G valid = {V , R , T , F train + F valid } , G test = {V , R , T , F train + F valid + F test } . Given a query q , let J q K train , J q K valid, and J q K test denote a set of answers (entities or timestamps) obtained by subgraph matching of q on G train , G valid and G test. For each query q , the reasoning process starts from anchor nodes and computes the final answer set via subgraph matching.

For the basic function Pe and Pt , we simply use all the triples and extract the entities and timestamps as the anchors. These anchors are supposed to have no empty answers under Pe and Pt . However, when it comes to some hard queries, such as e2i and e3i , the random sampled anchors may have empty answers. Because the TKG is sparse and incomplete, the intersection of two query sets has a high probability to be empty. The empty answers are meaningless for model-training and damage to

the performance of data-sampling process. To solve this problem, we use the following two strategies to accelerate the sampling.

Inverse Sampling : We randomly sample an entity as objective, denoted e o . Then we sample subjective entity e s 1 , e s 2 , relation r 1 , r 2 and timestamp t 1 , t 2 , according to the fact ( e s 1 , r 1 , e o , t 1 ) and ( e s 2 , r 2 , e o , t 2 ) . The sampled anchors are ( e s 1 , r 1 , t 1 ; e s 2 , r 2 , t 2 ) . These anchors under e2i have the answer e o at least, which asserts that the answer set is not empty. Such sampling method that samples the answer first and then samples the anchors is called Inverse Sampling .

Bi-directional Sampling : For long multi-hop queries, such as Pe2 , the complexity of random sampling is O ( N 2 L ) , where N is the entity count and L is the length of the query. Pe2 has L = 2 . It contains an anchor-sampling complexity of O ( N L ) and a ground-truth reasoning (to get answers) complexity of O ( N L ) . The origin sampling is to sample one subjective entity, get the objective as answers as next subjective anchors, again get the next objective answers. The final answers recursively depend on the initial subjective entity, leading to low performance. To accelerate, we sample an entity at the middle of AST, denoted e m . Then we sample in two directions. One is forward, e m as the subjective entity, to sample relation r 2 and timestamp t 2 . The other is backward, e m as the objective entity, to sample subjective anchor e 1 , relation r 1 and timestamp t 1 . The sampled anchors are ( e 1 , r 1 , t 1 ; r 2 , t 2 ) . The final answer set is asserted not empty, because it at least contains all the answers of Pe ( e m , r 2 , t 2 ) . Be aware that the two directions are independent, which can be sampled in parallel. Such sampling method that samples the answer in two directions is called Bi-directional Sampling , which can reduce the computation complexity to O ( N L + L 2 ) , composed of a sampling complexity of O ( N L 2 ) and a ground-truth reasoning complexity of O ( N L ) .

Embedding Reasoning Interpreter .

On the contrary of the ground-truth reasoning interpreter which has ' set ' as the input and output of the CQFs, in the embedding reasoning interpreter, the input and output of CQFs are ' embedding vectors '. The embedding-based method for reasoning over TKG is called Temporal Query Embedding . The computation complexity of the embedding reasoning interpreter is O ( L + N ) , where L is the length of the query and N is the entity count. The complexity consists of a query-embedding complexity of O ( L ) and a scoring-to-answer complexity of O ( N ) . The embedding reasoning is much faster than the ground-truth reasoning, which has a computation complexity of O ( N L ) .

To implement the embedding reasoning interpreter, we just need to replace the basic functions with neural networks. The symbols dict to pass to the interpreter is

The embedding vectors are learned as is introduced in Section 4. Unlike the ground-truth reasoning, the embedding reasoning is fuzzy. The output of the embedding reasoning interpreter is an embedding vector, which is a fuzzy set representation. To get the final answer set, we need to use the distance function (in Section 4.3) to score to candidate answers. In this way, the embedding vector is converted to the final answer set. The final answers are given in the ranking list, where each answer is followed by its distance to the query. Usually the top-k answers are accepted as the final answers.

Note that the interpreter is flexible, and easy to implement and extend. In order to reproduce static query embeddings [2-4] over dynamic knowledge graphs, the only thing to do is to implement the symbols dict and the distance function.

We hope that the design of interpreter is helpful for the future research of temporal query embedding.

Screenshot .

Additionally, we show the screenshot of a real running interpreter in Figure 8. In the screenshot, we randomly select an example of e2i query. Then, we use the embedding reasoning interpreter to get the final answer set step by step. The final answer set is shown in the ranking list, where each answer is followed by its similarity score to the query. Finally, the bot correctly predicts the hard answer which only exists in the test set!

B.3 Details of Generated Datasets

Finally, we generate three datasets for the task of temporal complex reasoning using the process in Section B.2. The statistics of the generated datasets are listed below. Table 9 show the count of query

structures in the split of training, validation and testing. The average answers count of each query structure are reported in Table 10. The number of queries for each dataset is shown in Table 11.

B.4 Implementation Details

We use the Embedding Reasoning Interpreter as is introduced in Appendix B.2. We implement our model with PyTorch and use Adam [50] as a gradient optimizer. For each experiment, we use one GTX1080 graphic card. The hyperparameters for each dataset are shown in Table 12. Note that we do not perform hyperparameter tuning for each dataset. Instead, we use the same hyperparameters

37.0

ICEWS14

36.5

36.0

MRR

35.5

35.0

34.5

400

600

800

1000

embeddingdimension

(a) The impact of embedding dimension on MRR.

35

ICEWS14

34

一

MRR

33

32

31

10

20

30

40

the marginy in loss function

(b) The impact of the margin γ on MRR.

found in sensitive analysis on ICEWS14 (Section B.6) for all datasets. Besides, the source code is available at Gitbub † . We cite these projects ‡ § , and thank them for their great contributions!

B.5 Evaluation

Given the standard split of edges into training ( F train), validation ( F valid) and test ( F test) sets, we append inverse relations and double the number of edges in the graph. Then we create three graph: G train = {V , R , T , F train } , G valid = {V , R , T , F train + F valid } , G test = {V , R , T , F train + F valid + F test } . Given a query q , let J q K train , J q K valid, and J q K test denote a set of answers (entities or timestamps) obtained by subgraph matching of q on G train , G valid and G test. For a test query q and its non-trivial answer set v ∈ J q K test -J q K valid , we denote the rank of each answer v i as rank ( v i ) . The mean reciprocal rank (MRR) is MRR ( q ) = 1 || v || ∑ v i ∈ v 1 rank ( v i ) and Hits at K (Hits@K) is Hits@K ( q ) =

B.6 Experiment Data for Sensitive Analysis

We conduct experiments on ICEWS14 to analyze the impact of the embedding dimensionality d and the margin γ on the performance of TFLEX.

Impacts of Embedding Dimensionality Our experiments indicate that the selection of embedding dimension has a substantial influence on the effectiveness of TFLEX. We train TFLEX with embedding dimension d ∈ { 300 , 400 , 500 , 600 , 700 , 800 , 900 , 1000 } and plot results based on the validation set, as shown in Figure 9a. With the increase of d , the model performance (indicated by MRR) increases rapidly and reaches its top at d = 800 . Therefore, we assign 800 as the best setting.

Impacts of Parameter γ We train TFLEX with parameter γ ∈ { 5 , 10 , 15 , 20 , 25 , 30 , 35 , 40 } and plot MRR results in Figure 9b. Too small and too large γ both get bad results, while γ = 15 in the middle is the best. Therefore, we choose γ = 15 .

Error Bars of Main Results In order to evaluate how stable the performance of TFLEX is, we run five times with random seeds { 1 , 10 , 100 , 1000 , 10000 } and report the error bars of these results. Table 13 shows the error bar of TFLEX's MRR results on ICEWS14. Overall, the standard variances are small, which demonstrates that the performance of TFLEX is stable.

B.7 Detail Metrics of Main Results

We report the MRR results on all query structures for each dataset in Table 14 (ICEWS14), Table 15 (ICEWS05-15) and Table 16 (GDELT-500). Query2box [2] can only answer queries without negation. BetaE [3] and ConE [4] can answer queries with negation, except temporal logic. ConE(temporal) and X(ConE) are the variants of ConE, aiming at exploring the right way to promote the static query embeddings to temporal ones in Section C. FLEX, X(ConE), X-1F, X-logic are the variants of TFLEX, introduced in the main body of the paper.

C Exploration of Static Query Embeddings Toward Dynamic

In this section, we present more research questions that aim to explore the right way to promote the static query embeddings to temporal ones. Since ConE [4] is a strong baseline for complex logical reasoning over static knowledge graphs, we conduct exploration experiments by introducing variants of ConE. We present the overall results for each dataset in Table 17, detailed results in Table 14 (ICEWS14), Table 15 (ICEWS05-15) and Table 16 (GDELT-500). Overall, we are interested in the following research questions:

RQ1 : How about incorporating temporal information into the static query embedding, without considering time logic?

RQ2 : Further, how about enhancing the static query embedding with temporal logic?

To address research question ( RQ1 ), we propose a variant of ConE called ConE(temporal) that incorporates temporal information into the static query embedding. The only difference between ConE and ConE(temporal) is the projection operator used. In ConE(temporal), the projection operator incorporates temporal information through a technique called TTransE, which uses the temporal embedding of timestamps. In terms of formulation, the projection operator for ConE(temporal) is given by P e ( V q , r , V t ) = g ( MLP ( V q + r + V t )) , while for ConE it is given by P e ( V q , r , V t ) = g ( MLP ( V q + r )) . Note that V q , r , V t are cone embeddings.

From Table 17, we observe that there is slightly improvement in terms of MRR when comparing ConE(temporal) with ConE. It shows that the temporal information can help to improve the performance of static query embedding on answering entities over TKGs. However, the improvement is not significant, which indicates that simply aware of the temporal information is not enough to promote the static query embedding to efficient temporal one.

Next, we wonder if it could help when we utilize more queries involving time logic, which provides more temporal information. Therefore, we address research question ( RQ2 ) by proposing a variant of ConE and TFLEX, called X(ConE) , that enhances the static query embedding with temporal logic. X(ConE) replaces the entity part of temporal feature-logic embedding with ConE, as is introduced in Section 5.2. In the other view, X(ConE) is based on ConE, using cone embedding concatenated with the time part of temporal feature-logic embedding, which can handle temporal logic.

From Table 17, to our surprise, X(ConE) performs worse than ConE and TFLEX. The concatenation of cone embedding and feature-logic embedding does not help to improve the overall performance of temporal complex reasoning. This is because cone embedding is geometric, while temporal feature-logic embedding is fuzzy. We think there is a mismatch or semantic conflict between the two embeddings, which leads to the poor performance of X(ConE) .

Be aware that we do not deny the importance of temporal logic. In fact, temporal logic is very important for temporal complex reasoning (See Ablation on time part in Section 5.2). However, we think that it is not a good idea to promote the static query embedding to temporal one by simply concatenating the geometric embedding with the fuzzy embedding. The right way to promote remains an open question.

D Visualization of Semantic Changes by Neural Temporal Operators

In this section, we present more visualization of the semantic changes of temporal operators in TFLEX. The examples are randomly selected in the test set of ICEWS14. The visualization is shown in Figure 10, Figure 11 and Figure 12. We attach the anchors and the ground-truth answers in the corresponding caption of figures. From the figures, we observe that the model correctly ranks the hard answers at top when answering query Pt . It can also be seen from the figure that the semantic changes of temporal operators aPt and bPt are intuitive and reasonable.

1.0

before

after

0.8

Score

0.6

Similarity

0.4

bPt=Bt(Pt(s, r, o))

0.2

Pt=Pt(s,r,o)

aPt=At(Pt(s,r,o))

0.0

0

50

100

150

200

250

300

350

Timestamp (0=2014-01-01,365=2014-12-31)

1.0

before

lafter

0.8

Score

0.6

Similarity

0.4

bPt=Bt(Pt(S,r,o))

0.2

Pt=Pt(s,r,o)

aPt=At(Pt(s,r,o))

0.0

0

50

100

150

200

250

300

350

Timestamp (0=2014-01-01,365=2014-12-31)

1.0

bPt=Bt(Pt(s, r,o))

b'efol

fter

Pt=Pt(S,r,o)

0.8

Score

aPt=At(Pt(S,r,o))

0.6

Similarity

0.4

0.2

0.0

0

50

100

150

200

250

300

350

Timestamp (0=2014-01-01,365=2014-12-31)

E Explaining Answers with Temporal Feature-Logic Framework

In this section, we present the explanation of answers given by TFLEX. Below we take the query " e2i " as an example. The temporal query " e2i " is defined as follows: q [ V ? ] = V ? , ∃ V a , V b , r 1 ( e 1 , V a , t 1 ) ∧ r 2 ( e 2 , V b , t 2 ) . Let's consider the specific query: "Who was consulted by Mohammad Javad Zarif on 2014-04-07 and consulted Mohammad Javad Zarif on 2014-04-07?" In this example, e 1 = "Mohammad Javad Zarif", r 1 = "consulted by", t 1 = "2014-04-07", r 2 = "consulted", and t 2 = "201404-07". We use TFLEX to execute the query and obtain the answers. The answers are categorized into easy, hard, and wrong ones. Table 18 presents the five most likely answers and their rankings for interpretation. From the table, we observe that TFLEX ranks the answers "Mohammad Javad Zarif", "Sebastian Kurz", "China" and "Catherine Ashton" high, while ranking the wrong answer "Iran" low. TFLEX even infers a hard answer "Catherine Ashton" with higher score than two easy answers "Sebastian Kurz" and "China". This demonstrates that TFLEX successfully identifies the correct answers using complex reasoning and distinguishes them from the wrong ones.

Navigation: Pt(13), Pe(14), Pe2(15), Pe3(16), Pe_Pt(17), e2i(18), e3i(19), e2i_Pe(20), Pe_e2i(21), Pe_t2i(22), t2i(23), t3i(24), t2i_Pe(25), Pt_se2i(26), Pt_oe2i(27), t2i_NPt(28), t2i_PtN(29), Pe_t2i_PtPe_NPt(30), e2i_NPe(31), e2i_PeN(32), Pe_e2i_Pe_NPe(33), e2i_N(34), e3i_N(35), e2u(36), Pe_e2u(37), Pt_sPe(38), Pt_oPe(39), t2i_N(40), t3i_N(41), t2u(42), Pe_t2u(43), Pe_aPt(44), Pe_bPt(45), Pe_at2i(46), Pe_bt2i(47), between(48).

Pe3

Presidential Movement

Make an appeal or request

2014-04-24

?

2014-02-05

Criticize or denounce

2014-08-26

Who was criticized or denounced on 2014-08-26 by the person

who made a statement on 2014-02-05 at the place where the

presidential movement made an appeal or request on 2014-04-

24?

Temporal Query

q

[

V

?

]

=

V

?

,

∃

V

a

, V

b

, r

1

(

e

1

, V

a

, t

1

)

∧

r

2

(

V

a

, V

b

, t

2

)

∧

r

3

(

V

b

, V

?

, t

3

)

Rank

Query Answers

Correctness

Answer Type

1

Government (France)

✔

Easy

2

France

✘

-

3

✔

Hard

4

✘

-

5

Citizen (International)

✘

-

?

Pe_Pt

Citizen (Thailand)

Police (Greece)

Illegal Immigrant (Greece)

Arrest, detain, or charge with legal action

Who arrested, detained, or charged the citizen (Thailand) with

legal action after Police (Greece) arrested, detained, or charged

the illegal immigrant (Greece) with legal action?

Temporal Query

q

[

V

?

] =

V

?

,

∃

T

a

, r

1

(

e

1

, V

?

, T

a

)

∧

r

2

(

e

2

, e

3

, T

a

)

Rank

Query Answers

Correctness

Answer Type

1

Military (Thailand)

✔

Easy

2

Municipal Court (Thailand)

✔

Easy

3

Thailand

✔

Hard

4

National Council for Peace and Order of Thailand

✘

-

5

Police (Cambodia)

✘

-

Consult

And

?

e2i

2014-04-07

2014-04-07

Temporal Query

q

[

V

?

] =

V

?

, r

1

(

e

1

, V

?

, t

1

)

∧

r

2

(

e

2

, V

?

, t

2

))

Rank

Query Answers

Correctness

Answer Type

1

✔

Easy

2

✔

Hard

3

✔

Easy

4

China

✔

Easy

5

Iran

✘

-

And

?

e3i

2014-10-05

Discuss by telephone

2014-10-05

Make a visit

2014-01-20

2014-01-20?

Temporal Query

q

[

V

?

] =

V

?

, r

1

(

e

1

, V

?

, t

1

)

∧

r

2

(

e

2

, V

?

, t

2

)

∧

r

3

(

e

3

, V

?

, t

3

)

Rank

Query Answers

Correctness

Answer Type

1

UAE Armed Forces

✔

Easy

2

United Arab Emirates

✔

Hard

3

Middle East

✘

-

4

Morocco

✘

-

5

✘

-

e2i_Pe

And

?

Make statement

2014-06-11

Make a visit

2014-06-04

Naval (Yemen)

Occupy territory

2014-03-01

Where was occupied territory on 2014-03-01 and was visited on

statement on 2014-06-11?

Temporal Query

q

[

V

?

] =

V

?

,

∃

V

a

, r

1

(

e

1

, V

a

, t

1

)

∧

r

2

(

V

a

, V

?

, t

2

)

∧

r

3

(

e

2

, V

?

, t

3

)

Rank

Query Answers

Correctness

Answer Type

1

Iran

✔

Easy

2

China

✘

-

3

Iraq

✘

-

4

France

✘

-

5

Afghanistan

✘

-

And

?

Pe_e2i

Labor Union (Croatia)

Make an appeal or request

2014-02-03

Government (Croatia)

2014-03-25

Consult

2014-03-25

Where was consulted on 2014-03-25 by the place where Labor

Union (Croatia) made an appeal or request on 2014-02-03 and

was consulted by Government (Croatia) on 2014-03-25?

Temporal Query

q

[

V

?

] =

V

?

,

∃

V

a

, r

1

(

e

1

, V

a

, t

1

)

∧

r

2

(

e

2

, V

a

, t

2

)

∧

r

3

(

V

a

, V

?

, T

3

)

Rank

Query Answers

Correctness

Answer Type

1

Labor Union (Croatia)

✔

Hard

2

Business (Croatia)

✘

-

3

Government (Croatia)

✔

Easy

4

Ministry (Croatia)

✔

Easy

5

Branko Grcic

✘

-

t2i

TimeAnd

?

Xi Jinping

Xi Jinping

At what time did Barack Obama negotiate with and consult Xi

Jinping?

Temporal Query

q

[

T

?

] =

T

?

, r

1

(

e

1

, e

2

, T

?

)

∧

r

2

(

e

3

, e

4

, T

?

))

Rank

Query Answers

Correctness

Answer Type

1

2014-11-15

✔

Easy

2

2014-03-24

✔

Easy

3

2014-11-12

✔

Easy

4

2014-11-11

✔

Easy

d 5

2014-11-14

✔

Easy

TimeAnd

?

t3i

Japan

Engage in diplomatic cooperation

South Korea

Make statement

France

When did Japan engage in diplomatic cooperation with South

Temporal Query

q

[

T

?

] =

T

?

, r

1

(

e

1

, e

2

, T

?

)

∧

r

2

(

e

3

, e

4

, T

?

)

∧

r

1

(

e

5

, e

6

, T

?

)

Rank

Query Answers

Correctness

Answer Type

1

2014-09-18

✔

Hard

2

2014-09-20

✔

Easy

3

2014-08-22

✔

Easy

4

2014-07-01

✔

Hard

5

2014-08-16

✘

-

TimeAnd

?

Finance / Economy / Commerce / Trade Ministry (Oman)

Share intelligence or information

2014-01-06

Citizen (Oman)

Activist (Somalia)

Armed Gang (Somalia)

When did Activist (Somalia) fight with small arms and light

weapons with Armed Gang (Somalia) while Citizen (Oman) en-

gage in a symbolic act with the person with whom finance /

Economy / Commerce / Trade Ministry (Oman) shared intelli-

gence or information on 2014-01-06?

Temporal Query

q

[

T

?

] =

T

?

,

∃

V

a

, r

1

(

e

1

, V

a

, t

1

)

∧

r

2

(

V

a

, e

2

, T

?

)

∧

r

3

(

e

3

, e

4

, T

?

)

Rank

Query Answers

Correctness

Answer Type

1

2014-01-30

✔

Easy

2

2014-02-05

✘

-

3

2014-01-29

✘

-

4

2014-02-07

✘

-

5

2014-02-04

✘

-

Pt_le2i

And

?

And

Iran

2014-03-09

Iran

Host a visit

2014-03-09

Engage in negotiation

Iran

When did Iran negotiate with the country that visited Iran on

2014-03-09?

Temporal Query

q

[

T

?

] =

T

?

,

∃

V

a

, r

1

(

e

1

, V

a

, t

1

)

∧

r

2

(

e

2

, V

a

, t

2

)

∧

r

3

(

V

a

, e

3

, T

?

)

Rank

Query Answers

Correctness

Answer Type

1

2014-03-10

✔

Easy

2

2014-03-08

✔

Easy

3

2014-06-16

✔

Easy

4

2014-03-13

✔

Easy

5

2014-03-09

✔

Easy

And

?

And

Pt_re2i

Armenia

Cooperate militarily

Armenia

Cooperate militarily

2014-12-08

Armenia

Cooperate militarily

2014-07-02

When did Armenia cooperate militarily with the country that

cooperates militarily with Armenia on 2014-12-08 and 2014-07-

02?

Temporal Query

q

[

V

?

] =

V

?

,

∃

V

a

, r

1

(

e

1

, V

a

, T

?

)

∧

r

2

(

e

2

, V

a

, t

1

)

∧

r

3

(

e

3

, V

a

, t

2

)

Rank

Query Answers

Correctness

Answer Type

1

2014-12-08

✔

Easy

2

2014-07-02

✔

Easy

3

2014-05-06

✔

Easy

4

2014-10-26

✔

Easy

5

2014-07-17

✘

-

TimeAnd

?

TimeNot

Edgars Rinkevics

Express intent to meet or negotiate

2014-04-22

Consult

Japan

When did Barack Obama express intent to meet or negotiate with

Japan, and on that day, Catherine Ashton wasn't consulted by the

person Edgars Rinkevics expressed intent to meet or negotiate

with on 2014-04-22?

Temporal Query

q

[

T

?

] =

T

?

,

∃

V

a

, r

1

(

e

1

, V

a

, t

1

)

∧¬

r

2

(

V

a

, e

2

, T

?

)

∧

r

3

(

e

3

, e

4

, T

?

)

Rank

Query Answers

Correctness

Answer Type

1

2014-04-17

✔

Easy

2

2014-04-18

✔

Easy

3

2014-02-13

✔

Easy

4

2014-04-05

✔

Easy

5

2014-02-11

✘

-

TimeAnd

?

t2i_PtN

TimeNot

Party President (Greece)

2014-11-03

Make an appeal or request

Citizen (Greece)

When was the citizen (Greece) made an appeal or request by the

country that made an appeal or request to the Party President

(Greece), and on that day, Angela Merkel did not express intent

to meet or negotiate with Evo Morales?

Temporal Query

q

[

T

?

] =

T

?

,

∃

V

a

, r

1

(

e

1

, V

a

, t

1

)

∧

r

2

(

V

a

, e

2

, T

?

)

∧¬

r

3

(

e

3

, e

4

, T

?

)

Rank

Query Answers

Correctness

Answer Type

1

2014-05-16

✔

Easy

2

2014-07-24

✔

Easy

3

2014-05-25

✔

Easy

4

2014-04-29

✔

Easy

5

2014-07-29

✔

Easy

TimeAnd

Pe_t2i_PtPe_NPt

TimeNot

?

Buddhist (Cambodia)

Arrest, detain, or charge with legal action

2014-04-30

Threaten

Education (Colombia)

the day when Hun Sen was threatened by the person who was

arrested, detained, or charged with legal action by the Buddhist

(Cambodia) on 2014-04-30, and on that day, Education (Colom-

bia) made a statement to Eric Garner?

Temporal Query

q

[

V

?

]

=

V

?

,

∃

V

a

, T

a

, r

1

(

e

1

, V

?

, T

a

)

∧

r

2

(

e

2

, V

a

, t

1

)

∧

r

3

(

V

a

, e

3

, T

a

)

∧ ¬

r

4

(

e

4

, e

5

, T

a

)

Rank

Query Answers

Correctness

Answer Type

1

✔

Hard

2

Cabinet / Council of Ministers / Advisors (United States)

✔

Easy

3

✘

-

4

✘

-

5

✘

-

And

?

e2i_NPe

Not

Make statement

2014-09-20

2014-09-20

Iran

Host a visit

2014-04-29

Who visited Iran on 2014-04-29 and did not made a statement at

the place where Carlos Saul Menem made a statement on 2014-

09-20?

Temporal Query

q

[

V

?

]

=

V

?

,

∃

V

a

, V

b

, r

1

(

e

1

, V

a

, t

1

)

∧

¬

r

2

(

V

a

, V

b

, t

2

)

∧

r

3

(

e

2

, V

?

, t

3

)

Rank

Query Answers

Correctness

Answer Type

1

✔

Easy

2

✔

Easy

3

✔

Easy

4

✘

-

5

Iran

✘

-

̸

And

?

e2i_PeN

Not

Energy Department/Ministry (Turkey)

2014-03-27

Make pessimistic comment

2014-03-13

Mainland Affairs Council

Make a visit

2014-02-11

Who was commented pessimistically on 2014-03-13 by the

person who expressed accord to Energy Department/Ministry

(Turkey) on 2014-03-27 and not visited by Mainland Affairs

Council on 2014-02-11?

Temporal Query

q

[

V

?

]

=

V

?

,

∃

V

a

, V

b

, r

1

(

e

1

, V

a

, t

1

)

∧

r

2

(

V

a

, V

?

, T

b

)

∧

r

3

(

e

2

, V

?

, t

3

)

Rank

Query Answers

Correctness

Answer Type

1

Member of the Judiciary (Turkey)

✔

Easy

2

Lawyer/Attorney (Turkey)

✔

Hard

3

Justice and Development Party

✘

-

4

Other Authorities / Officials (Turkey)

✘

-

5

Party Member (Turkey)

✘

-

̸

And

?

And

And

Not

Pe_e2i_Pe_NPe

Treasury/Finance Ministry (Eritrea)

2014-09-17

Drug Gang (Beltrán -Leyva Cartel)

2014-10-16

2014-10-21

On 2014-10-21, who negotiated with the person who consulted

with the Treasury/Finance Ministry (Eritrea) on 2014-09-17 but

did not arrest, detain, or charge Drug Gang (Beltrán -Leyva Car-

tel) with legal action on 2014-10-16?

Temporal Query

q

[

V

?

]

=

V

?

,

∃

V

a

, V

b

, r

1

(

e

1

, V

a

, t

1

)

∧

r

2

(

e

2

, V

a

, t

2

)

∧

r

3

(

V

a

, V

?

, t

3

)

Rank

Query Answers

Correctness

Answer Type

1

South Korea

✔

Easy

2

Japan

✘

-

3

China

✔

Hard

4

North Korea

✘

-

5

Thailand

✘

-

Envoy (Kazakhstan)

Praise or endorse

2014-05-19

RIA Novosti

Sign formal agreement

2014-12-12

Governor (South Korea)

2014-05-09

And

?

e3i_N

And

Not

Who was praised or endorsed by Envoy (Kazakhstan) on 2014-

05-19 and signed a formal agreement with RIA Novosti on 2014-

12-12 but not signed a formal agreement with Governor (South

Korea) on 2014-05-09?

Temporal Query

q

[

V

?

] =

V

?

, r

1

(

e

1

, V

?

, t

1

)

∧

r

2

(

e

2

, V

?

, t

2

)

∧ ̸

r

3

(

e

3

, V

?

, t

3

)

Rank

Query Answers

Correctness

Answer Type

1

Iran

✔

Easy

2

Iraq

✘

-

3

China

✘

-

4

Afghanistan

✘

-

5

North Atlantic Treaty Organization

✘

-

Or

?

e2u

Foreign Affairs (Uruguay)

2014-11-24

2014-06-19

Who consulted Foreign Affairs (Uruguay) on 2014-11-24 or

Temporal Query

q

[

V

?

] =

V

?

, r

1

(

e

1

, V

?

, t

1

)

∨

r

2

(

e

2

, V

?

, t

2

)

Rank

Query Answers

Correctness

Answer Type

1

Foreign Affairs (Ecuador)

✔

Easy

2

Mexico

✔

Easy

3

Government (Mexico)

✔

Hard

4

Guatemala

✘

-

5

Other Authorities / Officials (Mexico)

✘

-

TimeAnd

?

TimeNot

t2i_N

Discuss by telephone

Senate (Slovakia)

Reject

Department/Ministery (Slovakia)

Javad Zarif by telephone, and on that day Senate (Slovakia) did

not reject Justice Department/Ministery (Slovakia)?

Temporal Query

q

[

T

?

] =

T

?

, r

1

(

e

1

, e

2

, T

?

)

∧ ¬

r

2

(

e

3

, e

4

, T

?

)

Rank

Query Answers

Correctness

Answer Type

1

2014-01-19

✔

Easy

2

2014-01-18

✔

Hard

3

2014-11-02

✘

-

4

2014-02-02

✘

-

5

2014-12-13

✘

-

TimeAnd

?

TimeNot

t3i_N

Police (Egypt)

Use tactics of violent repression

Protester (Egypt)

Opposition Activist (Ukraine)

Police (Ukraine)

Information / Communication / Transparency Ministry

Consult

(China)

When did the Police (Egypt) use tactics of violent repression

against protesters (Egypt) and the Police (Ukraine) use tactics

of violent repression against protesters (Egypt), and on that day

Information / Communication / Transparency Ministry (China)

Temporal Query

q

[

T

?

] =

T

?

, r

1

(

e

1

, e

2

, T

?

)

∧

r

2

(

e

3

, e

4

, T

?

)

∧ ¬

r

3

(

e

5

, e

6

, T

?

)

Rank

Query Answers

Correctness

Answer Type

1

2014-01-21

✔

Easy

2

2014-01-13

✘

-

3

2014-01-22

✘

-

4

2014-02-20

✘

-

5

2014-01-14

✘

Hard

TimeOr

?

t2u

Citizen (Botswana)

Make statement

China

When did citizens (Botswana) obstruct passage and block Nonofo

Molefhi or Zheng Zeguang make statements about China?

Temporal Query

q

[

T

?

] =

T

?

, r

1

(

e

1

, e

2

, T

?

)

∨

r

2

(

e

3

, e

4

, T

?

)

Rank

Query Answers

Correctness

Answer Type

1

2014-04-09

✔

Easy

2

2014-05-21

✔

Easy

3

2014-05-20

✔

Hard

4

2014-05-22

✘

-

5

2014-05-22

✘

-

?

TimeOr

Pe_t2u

Citizen (Nigeria)

Make an appeal or request

University of Cape Town

Consult

News Editor (Nigeria)

Muslim (United Arab Emirates)

State Security Court (United Arab Emirates)

Who was appealed or requested by Citizen (Nigeria) on the day

when the University of Cape Town consulted News Editor (Nige-

ria) or State Security Court (United Arab Emirates) arrested,

detained, or charged Muslim (United Arab Emirates) with legal

action?

Temporal Query

q

[

V

?

] =

V

?

,

∃

T

a

, r

1

(

e

1

, V

?

, T

a

)

∧

(

r

2

(

e

2

, e

3

, T

a

)

∨

r

3

(

e

4

, e

5

, T

a

))

Rank

Query Answers

Correctness

Answer Type

1

Government (Nigeria)

✔

Easy

2

Assemblies (Nigeria)

✔

Hard

3

Other Authorities / Officials (Nigeria)

✘

-

4

Head of Government (Nigeria)

✘

-

5

Citizen (Nigeria)

✘

-

Pe_aPt

After

?

Citizen (India)

Stephane Le Foll

Representatives (France)

Who made a statement for citizens (India) before Representatives

(France) consulted Stephane Le Foll?

Temporal Query

q

[

V

?

]

=

V

?

,

∃

T

a

, T

b

, r

1

(

e

1

, V

?

, T

b

)

∧

After

(

T

a

, T

b

)

∧

r

2

(

e

2

, e

3

, T

a

)

Rank

Query Answers

Correctness

Answer Type

1

Court Judge (India)

✔

Easy

2

Governor (India)

✔

Easy

3

✔

Easy

4

✘

Easy

5

Head of Government (India)

✘

-

Pe_bPt

Before

?

South Korea

Criticize or denounce

Head of Government(Bangladesh)

Who expressed intent to meet or negotiate with the Head of Gov-

ernment (Bangladesh) after South Korea Criticized or denounced

Temporal Query

q

[

V

?

]

=

V

?

,

∃

T

a

, T

b

, r

1

(

e

1

, V

?

, T

b

)

∧

Before

(

T

a

, T

b

)

∧

r

2

(

e

2

, e

3

, T

a

)

Rank

Query Answers

Correctness

Answer Type

1

Japan

✔

Easy

2

North Korea

✔

Easy

3

China

✔

Easy

4

South Korea

✘

-

5

✘

-

TimeAnd

?

After

TimeAnd

Pe_at2i

North Korea

Criticize or denounce

Admiral (Russia)

Make statement

Ministry (Madagascar)

Make statement

Government (Madagascar)

Who was criticized or denounced by North Korea on the day

before Admiral (Russia) made a statement for Chuck Hagel and

Ministry (Madagascar) made a statement for Government (Mada-

Temporal Query

q

[

V

?

]

=

V

?

,

∃

T

a

, T

b

, r

1

(

e

1

, V

?

, T

b

))

∧

After

(

T

a

, T

b

)

∧

r

2

(

e

2

, e

3

, T

a

)

∧

r

3

(

e

4

, e

5

, T

a

)

Rank

Query Answers

Correctness

Answer Type

1

South Korea

✔

Easy

2

Japan

✔

Easy

3

North Korea

✔

-

4

Head of Government (South Korea)

✔

Easy

5

✔

Easy

TimeAnd

?

Before

TimeAnd

Pe_bt2i

Citizen (India)

Decline comment

Presidential Candidate (Mozambique)

Employee (Mozambique)

Make statement

Military (Bolivia)

Who was declined commented by Citizen (India) after the day

when the employee (Mozambique) consulted the Presidential

Candidate (Mozambique), and Evo Morales made a statement to

the Military (Bolivia)?

Temporal Query

q

[

V

?

]

=

V

?

,

∃

T

a

, T

b

, r

1

(

e

1

, V

?

, T

b

)

∧

Before

(

T

a

, T

b

)

∧

r

2

(

e

2

, e

3

, T

a

)

∧

r

3

(

e

4

, e

5

, T

a

)

Rank

Query Answers

Correctness

Answer Type

1

Media (India)

✔

Easy

2

Member of Parliament (India)

✘

-

3

Court Judge (India)

✔

Easy

4

Head of Government (India)

✘

-

5

Governor (India)

✘

-

TimeAnd

?

After

Before

between

Other Authorities / Officials (Romania)

Foreign Affairs (United States)

Accuse

Party Member (Taiwan)

What was the day between the day that Tsai Ing-wen accused the

Party Member (Taiwan) and the foreign affairs (United States)

appealed for diplomatic cooperation (such as policy support) from

the other Authorities/officials (Romania)?

Temporal Query

q

[

T

?

]

=

T

?

,

∃

T

a

, T

b

, r

1

(

e

1

, e

2

, T

a

)

∧

r

2

(

e

3

, e

4

, T

b

)

∧

After

(

T

a

, T

?

)

∧

Before

(

T

b

, T

?

)

Rank

Query Answers

Correctness

Answer Type

1

2014-07-19

✔

Easy

2

2014-07-29

✔

-

3

2014-06-10

✔

Easy

4

2014-08-21

✔

-

d 5

2014-06-25

✔

Easy

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes TFLEX, a framework that reasons over temporal knowledge graphs (TKG). It takes a complex query about either an entity or a timestamp and a set of constraints as input. The query is then converted into a directed acyclic graph (DAG), which is a computation graph that projects the query into an embedding space. The paper also defines operations that happens in the graph, including projection, intersection, complement, and other set/temporal logic operations (e.g. After, Before, and Between). \n\nWhen the query embedding is produced, its distances between all candidate entity/timestamp embeddings are computed. The loss function is designed to minimise the distance between the embedding of the query and that of the correct entity/timestamp and maximise the distance between the query and the incorrect candidates.\n\nThe framework was evaluated on three standard datasets with modifications. The assessment process involves predefined complex queries on top of each dataset, including 27 types for training and 40 types for test. Several variants of the proposed framework were compared, as well as a few previous works on embedding query for static knowledge graphs.\n\nSTRENGTHS:\n- This paper discusses complex query embedding in temporal knowledge graphs, which is an important yet under-investigated topic. \n- The paper provides a definition to the problem and some attempts to solves the problem.\n\nWEAKNESSES:\nRelated Works\n- From my understanding, TKGC is an important subset of the task (Temporal Complex Query) defined in this paper. There are many TKGC works missing in the related work part. For example, [1] and [2]. \n- The discussion on complex query embedding works is insufficient. The paper should provide a more substantial explanation as to why existing works on static KG cannot be easily adapted for temporal KGs. \"They cannot utilize temporal information\" is a result, not a reason. \n\nMethod\n- In line 146 the paper introduces \"the relational embedding\" without saying how it is defined. Is it a trainable parameter? How is it initialised?\n- The definition to operators appears arbitrary. In equation (3), $V_q + r + V_t$ are adding the embedding for entities, relations, and timestamps together. What is the semantics of addition here?\n- Again, in equation (7), something like $q_f^t+\\frac{1+q_l^t}{2}$ is in essence adding the entity feature with its a truth value. What is the semantics of addition between these two values?\n- Later in the definition of distance function (line 199), $q_l^e + ||v_f^e-q_f^e||_1$ is adding a truth value with a L1-norm. What is the semantics of addition here?\n\nExperiments:\n- The method is evaluated on 40 types of generated queries from three datasets for TKGC. The paper did no discuss the reason for choosing these specific 40 types or how complete these types are. Nor has the it discussed whether TFLEX can generalise to queries outside of those 40 types.\n- There is no comparison with TKGC works. Since the TKGC task aims at completing a simpler query, it is naturally a subtask of what is proposed in this work. In other words, a (s, r, ?, t+1) completion task can be converted to a query and TFLEX should be able to handle it. I would expect at least a separate table comparing TFLEX and state-of-the-art TKGC on a completion task. Otherwise, it is difficult to assess how well TFLEX captures temporal information.\n\nMiscellaneous:\n- There are technical details being moved to appendix due to page limit. Please note that the page limit is nine content pages, not eight. The current main text is not self-contained to provide readers a high-level idea about the benefits of the framework. For example, as I pointed out above, many definitions and designs about the operators lack intuition or rationale, and look like something randomly popping out.\n- The source code provided in the supplementary material and the link in the paper reveal a non-anonymous email \"l****9@sysu.edu.cn\". While I believe this is an honest mistake, I urge the author to remove this information.\n\n---\nReference\n\n[1] Chronor: Rotation based temporal knowledge graph embedding, AAAI 2021.\n[2] Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning, SIGIR 2021.\n\nQUESTIONS:\nPlease see my points above.\n\nLIMITATIONS:\nThe paper has a good discussion about its limitations in page 11.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce an embedding-based method for answering complex, i.e., multi-hop, queries on temporal knowledge graphs. The Temporal Feature-Logic Embedding framework (TFLEX) uses fuzzy logic to model first-order logic operations on the entity and timestamp sets. The queries, with answers being either entities or timestamps, are embedded via four components, including feature and logic vectors for both entities and timestamps. The correct answer to the query is determined as the entity or timestamp whose embedding is closest to the query embedding according to a predefined distance function. Based on existing benchmark datasets, three new datasets were created that include a variety of complex queries. The proposed method TFLEX is compared against state-of-the-art query embedding methods, where TFLEX outperforms all baselines. Ablation studies show the effectiveness of the particular design of the method.\n\nSTRENGTHS:\n- The paper seems to be the first one to address the multi-hop logical reasoning problem on temporal knowledge graphs, where queries consist of disjunctions of atomic formulas. The authors introduce this new task, which can find application and are relevant in many domains.\n- The proposed method supports FOL operations as well as temporal operations and is able to perform multi-hop reasoning. \n- Three new datasets are generated, including 40 kinds of complex queries, which can be used for further benchmark experiments.\n- The experiments and ablation studies are extensive and confirm the effectiveness of the proposed method.\n- The source code and datasets are available online, which supports reproducibility and further evaluation.\n\nWEAKNESSES:\nThe main weakness concerns the clarity and preciseness of the technical content, which makes it difficult to follow and to understand the methodology, thereby making it challenging to judge the soundness. Here are some examples: \n- Figure 1: The query example \"During François Hollande was the president of France, which countries did Xi Jinping visit but Obama did not visit?\" is given. The logical formulation states that the answer is an entity (country) so that there exists a timestamp T_1 when Xi visited this country and there exists a timestamp T_2 when Obama did not visit this country. This does not seem to reflect the query correctly. The answer should be a country so that there exists a timestamp T_1 when Xi visited this country and for all T_2, Obama did not visit this country. Also, the picture might be easier to understand if the entities for Hollande (e_1, e_3) and France (e_2, e_4) are represented by one node instead of two.  \n- Line 87: The fact set should be a subset of the complete graph. The set on the right-hand side of the equation, however, denotes the complete graph (all possible quadruples).\n- The definitions for the entity query and the timestamp query are difficult to understand. Maybe it is possible to first introduce atomic formulas, then literals, then conjunctions, and last the disjunctive normal form. Some query examples with corresponding query structures could make it more comprehensible. Why should there be the same number (k) of bound variables V_i and T_i? There is only one relation r in the query definition, but is it not common to have several relations (see Figure 1)? Should “Between” also be one of the possibilities for f?\n- The method builds on fuzzy logic, which is important for modeling the operators. A short introduction to fuzzy logic directly in the paper (instead of being in the appendix) would help understanding. \n- The MLPs in Equation 3 are different, so indices could be added (as in later cases) to make it clearer.\n- Line 160: The Alignment Rule is mentioned and is part of the subsequent equations. Since both formulas (for I_e and I_t) look exactly the same, how are the AND operators calculated differently?\n- Line 131: ||q|| should be a query set, so is V_q the embedding of all queries or just one query? \n- Existing literature for temporal KG completion does not only include the two groups tensor decomposition and translation. There are also methods based on dynamic (time-dependent) embeddings, logical rules, autoregressive models, Markov processes, … (see [1] for a survey from January 2022).\n[1] Temporal Knowledge Graph Completion: A Survey. Borui Cai, Yong Xiang, Longxiang Gao, He Zhang, Yunfeng Li, Jianxin Li. https://arxiv.org/abs/2201.08236\n\nQUESTIONS:\nThe definitions section and method section contain the formulation for the two cases entity query and timestamp query and include a lot of notation, which impedes the readability. It is sometimes not clear, which equations are needed for which query type. Since both cases are similar in many ways, one suggestion would be to focus on one case, add more details/examples/descriptions, and include the other case in the appendix or mention the differences in a separate section. \n\nMinor comments:\nFigure 1: “Obama” -> “Barack Obama” (to be consistent with the other names)\nLine 21: The statement “results are inevitably incorrect” is rather strong. Depending on size and content of the KG, results might be correct.\nTable 1: The best results could be marked in bold to make it easier to identify the best method.\n\nLIMITATIONS:\nThe authors addressed the limitations and corresponding future work adequately in a separate section. Possible broader impact and application areas are also stated in a designated section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to learn on temporal knowledge graphs, using a combination of fuzzy logic with a temporal extension and node embeddings. To test the method, three new datasets were generated.\n\nThe choices made for the model seem logical and are over all well motivated in the paper (e.g. figure 2). The final model was tested on multiple data sets and compared to several non-trivial baselines. Furthermore, ablation studies were performed to analyze the effect of several components of the TFLEX model.\n\nWhile this paper has several strong points when it comes to presentation (e.g. figure 2 and the colors in line 164), there are some lacking areas. Firstly, having to much mathematical notation spread throughout the text itself (e.g. lines 173-180) compromises readability. Secondly, given the novelty of the model, I would have liked to see a detailed diagram representing the model.\n\nThis paper provides a novel approach for learning a temporal graphs and provides three data sets for this task. Research on temporal graphs is somewhat limited (as seen by the limitations of the baselines), making this a welcome addtion to the sub-area of machine learning on graphs.\n\nSTRENGTHS:\nThis paper performs a analysis by motivating choices and thoroughly evaluating the model. The figures that are present are very clear.\n\nWEAKNESSES:\nSome of the math could have been beter separated from the text, making the paper more readable.  There is not figure to visually convey the model itself.\n\nQUESTIONS:\nTo me, it is not clear why we would want up to n compositions of the identity, negation, before and after operations (see line 105).\n  In lines 124-125 you seem to assume a query only has one answer. Is this only for training purposes, or is this a limitation of the model?\n\nLIMITATIONS:\nThe model seems to be limit the answer set of queries to just one answer (the embedding closest to the query embedding).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the multi-hop logical reasoning problem on temporal knowledge graphs and proposed the first temporal complex query embedding framework named Temporal Feature-Logic Embedding framework(TFLEX).Firstly, they defined the task of multi-hop logical reasoning over TKGs. Secondly, they designed the first multi-hop logical reasoning framework which utilizes fuzzy logic to compute the logic part and extend fuzzy logic on timestamp set for supporting all FOL operation and extra temporal operations(After, Before and Between). At last, they generated three new TKG datasets for the task of multi-hop logical reasoning. Experiments on benchmark datasets demonstrate the efficacy of the proposed framework in deal with different operations in complex queries.\n\nSTRENGTHS:\n1.\tThey creatively conducted research multi-hop logical reasoning problem on the TKGs and proposed the first TFLEX framework to answer the temporal complex queries.\n2.\tthey relatively proposed using fuzzy logic to handle temporal feature-logical embedding and expanded extra temporal operations.\n3.\tThey provided three new TKG datasets and compared them with multiple benchmarks for experimental verification in order to better validate the reliability of their framework.\n4.\tThey had clear explanations and explanations of the definitions and methods used, with reasonable classification and appropriate illustrations.\n5.\tThere are more detailed compensations and explanations in the appendix to better explain the details of the method and experiment. \n6.\tThere are also corresponding open source code and datasets.\n\nWEAKNESSES:\n1.\tIn the section on related work, there is a lack of appropriate explanations for the problems and shortcomings of the methods cited in the relevant articles, as well as a comparison with current work.\n2.\tThe appendix section provides some overly detailed supplements to the overall concept, such as a more concise description of fuzzy logic.\n3.\tIt is not recommended to directly apply source code for explanation and display in the appendix section.\n\nQUESTIONS:\n1.\tIt is recommended that the author compare the methods of relevant articles with the improvements of the author's article to highlight the efficiency of their methods\n2.\tIt is suggested that the author can further optimize the content of the paper to streamline the appendix content, and simplify the subsection: Explaining answers with the framework and Experimental analysis into the main body of the paper to better demonstrate its experiments and principles.\n3.\tIt is suggested that Pseudocode should be used for interpretation, which is helpful for reading and analysis and is more standardized.\n\nLIMITATIONS:\nThe author thoroughly analyzed its limitations in the article, including insufficient time operators, improved time embedding, long query generation time, MRR and Hits@k Weak. Also proposed its plan for future improvement work", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
Authors present new embedding framework called TFLEX to embed complex temporal queries over temporal knowledge graphs to perform multi-hop reasoning with time constraints on TKGs. Authors present the overall embedding framework using Fuzzy logic to model complex logical queries and extending fuzzy logic to include three temporal operators After, Before and Between. It presents new benchmark datasets to evaluate embedding on TKGs and show experimental results to show the benefits of proposed approach.

STRENGTHS:
paper presents a new embedding framework and dataset for complex temporal queries, which can benefit the community in extending research in this direction.

WEAKNESSES:
Not sure if this framework can handle complex queries without temporal constraints on par with other complex query embedding methods?
suppose if we ignore the temporal aspect of current framework and run the standard benchmarks on complex query handling does it work as well as other methods in the literature.

QUESTIONS:
Can this solve complex query answering equally well on other bench mark tasks and how does it fare against the prior art?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07