TECHS: Temporal Logical Graph Networks for Explainable Extrapolation Reasoning
Year: 2023

Abstract:
Extrapolation reasoning on temporal knowledge graphs (TKGs) aims to forecast future facts based on past counterparts. There are two main challenges: (1) incorporating the complex information, including structural dependencies, temporal dynamics, and hidden logical rules; (2) implementing differentiable logical rule learning and reasoning for explainability. To this end, we propose an explainable extrapolation reasoning framework TEemporal logiCal grapH networkS (TECHS), which mainly contains a temporal graph encoder and a logical decoder. The former employs a graph convolutional network with temporal encoding and heterogeneous attention to embed topological structures and temporal dynamics. The latter integrates propositional reasoning and first-order reasoning by introducing a reasoning graph that iteratively expands to find the answer. A forward message-passing mechanism is also proposed to update node representations, and their propositional and first-order attention scores. Experimental results demonstrate that it outperforms state-of-the-art baselines.

heading: 1 Introduction
text: Knowledge Graphs (KGs) are widely used in intelligent systems (Ji et al., 2022; Mao et al., 2022; Zhu et al., 2023), where knowledge is commonly represented by triplets in the form of (s, r, o). The limit of conventional KGs is that real-world knowledge usually evolves over time. For example, a fact (Donald Trump, presidentOf, USA) is incorrect now because Joe Biden has been the new president of the USA since 2021. For more comprehensive representations of knowledge, Temporal Knowledge Graphs (TKGs) (Liang et al., 2022) are proposed by introducing time information (time point or interval) via quadruplets, i.e., (s, r, o, t). Then, the former example is defined as (Donald Trump, presidentOf, USA, 2017/01/20-2021/01/20).
∗ Corresponding author.
TKGs are usually incomplete (Cai et al., 2022; Liang et al., 2022). Many studies predicted future facts, based on past facts, namely TKG forecasting or extrapolation reasoning. Figure 1a shows the task that predicts facts at time ti with the facts at ti−2 and ti−1. A model should not only learn topology dependencies, i.e., the neighbor information of an entity (like Barack Obama at ti−2), but also learn temporal dynamics, i.e., the variations of properties of an entity over time (e.g., Angela Merkel evolves during ti−2 to ti−1). Thus, temporal embedding methods, e.g., TNTComplEx (Lacroix et al., 2020) and CyGNet (Zhu et al., 2021) were proposed. However, these blackbox methods fail to explain their predictions. An explainable method, xERTE (Han et al., 2021) conducted instanced propositional reasoning. However, the model is not scalable, as the evidence is entity-dependent, e.g., related to Barack Obama and other entities in Figure 1a. If we can learn the entity-independent rule in Figure 1b for the query
1281
(Barack Obama, makeStatement, ?, ti) in Figure 1a, the correct answer South Korea will be easily obtained after rule grounding.
Motivated by the fact that TKGs have many hidden logical rules to achieve explainable and accurate predictions, TLogic (Liu et al., 2022) searched first-order logical rules and used them for reasoning. However, this two-step pipeline method may cause error propagation issues. Generally, there are two main challenges for explainable extrapolation reasoning on TKGs: (1) TKGs contain diverse information, e.g., structural dependencies, temporal dynamics, and hidden logical rules that are difficult to incorporate together and achieve full coverage; (2) Logical rule representations are discrete and symbolic, resulting in the natural gap between logical rules and the continuous computation of neural networks. Thus, implementing differentiable logical rule learning and reasoning is not directly achievable (Yang et al., 2017).
To address above issues, we propose a unified framework TEemporal logiCal grapH networkS (TECHS). It first utilizes a graph convolutional network (GCN) to embed topological structures and temporal dynamics. To determine the weights of different edges between entities, a generic time encoding and a heterogeneous attention mechanism is introduced. Then, a logical decoder is proposed to integrate propositional and first-order reasoning to find the answer. A reasoning graph that contains both query entity and entity-time pair nodes is used to constantly expand over iterations. We update propositional and first-order attention weights as well as node representations via a novel forward message-passing mechanism. Finally, nodes’ attention weights with the same entity are aggregated as the answer indicator. Besides, first-order logical rules can be induced by a novel Forward Attentive Rule Induction (FARI) algorithm using learned first-order attention weights.
Our contributions are summarized as follows: (1) A unified framework TECHS is proposed to conduct explainable extrapolation reasoning on TKGs. To our best knowledge, this is the first study to jointly model structural dependencies, temporal dynamics, and propositional and first-order reasoning. (2) We integrate propositional and first-order reasoning in a logical decoder, where a forward message-passing is proposed to update their attention weights and node representations to achieve explainability. First-order logical rules are induced
by a novel FARI algorithm. (3) Extensive experiments verify the effectiveness of each module and the superiority over state-of-the-art baselines.

heading: 2 Related Work
text: The studies of extrapolation reasoning can be categorized into the following three trends. Static Embedding. By omitting time information in fact quadruplets, general KG embedding methods can be utilized for TKGs, such as TransE (Bordes et al., 2013), DistMult (Yang et al., 2015) and ComplEx (Trouillon et al., 2016). However, these methods simply consider the structural dependency in TKGs and ignore the temporal dynamics. Temporal Embedding. TTransE (Leblay and Chekol, 2018) expanded TransE to the temporal setting by fusing temporal information in relation embeddings. Similarly, TA-DistMult and TATransE (García-Durán et al., 2018) learned relation representations with time information and calculated quadruplet plausibility by DistMult and TransE. Differently, DE-SimplE (Goel et al., 2020) proposed diachronic entity embedding which contained static segment and time-varying segment. Upon ComplEx, TNTComplEx (Lacroix et al., 2020) learned complex-valued embeddings for the entity, relation and time. RE-Net (Jin et al., 2020) learned the global representations of the time subgraph and the local representations of nodes on it. CyGNet (Zhu et al., 2021) introduced a timeaware copy-generation mechanism to model the probability of existing facts, occurring in the future and predicted whether new facts would emerge. However, the aforementioned methods are all in black-box fashion and lack of explainability. Explainable Reasoning. xERTE (Han et al., 2021) proposed a human-understandable reasoning strategy, introducing an expanding query-relevant subgraph to achieve explainability. TITer (Sun et al., 2021) conducted reasoning from a query node and sequentially transferred to a new node related to the prior on TKGs until the answer was founded. Upon AnyBURL (Meilicke et al., 2019) that sampled paths to learn first-order rules in static KGs, TLogic (Liu et al., 2022) learned temporal logical rules with confidences via a temporal random walk. The candidate scores were obtained by rule applications in TKGs. However, xERTE and TITer conducted propositional reasoning by an end-toend framework that had limited scalability, as its reasoning process was query-specific. Although
TLogic learned query-independent first-order logical rules, its pipeline method might cause error propagation and performance degradation.

heading: 3 Preliminaries
text: A TKG can be represented as G = {E ,R, T ,F}, where E , R and T denote the set of entity, relation and time, respectively. F ⊂ E×R×E×T is the fact collection. Each fact is a quadruplet, such as (s, r, o, t) where s, o ∈ E , r ∈ R and t ∈ T . For a query (s̃, r̃, ?, t̃) in testing, the model needs to predict an answer entity õ, based on the facts that occur earlier than t̃, i.e., t̃ > max(Ttrain).
Logical reasoning in KGs can be categorized as: propositional and first-order. Propositional reasoning, generally known as multi-hop reasoning (Ren and Leskovec, 2020; Zhang et al., 2021, 2022a), is entity-dependent that usually reasons over queryrelated paths to obtain an answer. First-order reasoning is entity-independent, using first-order logical (FOL) rules for different entities (Zhang et al., 2022b), describing causal knowledge in the form of body to head, e.g., premise⇒conclusion, where new facts can be deduced, given observed ones. For efficient and explainable reasoning on TKGs, we define the FOTH rule and the reasoning graph. Definition 1. First-order Temporal Horn (FOTH) Rule: Based on Horn rules (Lin et al., 2022) on static KGs, atoms in FOTH rule body are connected transitively by shared variables. Meanwhile, rule body and rule head have the same start and end variables. Time growth also needs to be satisfied, i.e., time sequence is increasing and the time in the rule head is the maximum. For example, the following rule ϵ, ∃X,Y,Z r1(X,Y ) : t1∧r2(Y,Z) : t2 ⇒ r(X,Z) : t is a FOTH rule with length 2 if t1 ⩽ t2 < t. X , Y and Z are variables that can be instantiated as entities of TKGs by rule grounding. Noticeably, for rule learning and reasoning, t1, t2 and t are virtual time variables that are only used to satisfy the time growth and do not have to be instantiated. To represent the rule certainty, each rule is assigned with a confidence value ϵ ∈ [0, 1]. Definition 2. Reasoning Graph: For a query (s̃, r̃, ?, t̃), we introduce a reasoning graph G̃ = {O,R, F̃} for propositional and first-order reasoning. O is a node set that consists of nodes in different iteration steps, i.e., O=O0 ∪ O1 ∪ · · · ∪ OL. O0 only contains a query entity s̃ and others consist of nodes in the form of entity-time pairs. (nli, r̄, n l+1 j ) ∈ F̃ is an edge that links nodes at
two neighbor steps, i.e., nli∈Ol, nl+1j ∈Ol+1 and r̄∈R. The reasoning graph is constantly expanded by searching for posterior neighbor nodes. For start node n0= s̃, its posterior neighbors are N (n0)= {(ei, ti)|(s̃, r̄, ei, ti) ∈ F ∧ ti < t̃}. For a node in following steps nli = (ei, ti) ∈ Ol, its posterior neighbors are N (nli) = {(ej , tj)|(ei, r̄, ej , tj) ∈ F∧ti ⩽ tj∧tj<t̃}. Its prior parents are Ñ (nli)= {(nl−1j , r̄)|nl−1j ∈ Ol−1∧ (nl−1j , r̄, nli) ∈ F̃}. An example reasoning graph with two steps is shown in Figure 1c. To take prior nodes into account at the current step, an extra relation self is added. Then, nli = (ei, ti) can be obtained at the next step as nl+1i = (ei, ti) (ti is the minimum time if l = 0).

heading: 4 Methodology
text: There are three key technical parts in TECHS: temporal graph encoder, logical decoder, and extrapolation prediction. Figure 2 shows its architecture.

heading: 4.1 Temporal Graph Encoder
text: Generally, GCNs follow an iterative messagepassing strategy to continuously aggregate information from neighbor nodes. As conventional GCNs cannot model time information, we propose a temporal graph encoder. The generic time encoding (Xu et al., 2020) is introduced to embed times in TKGs as it is fully compatible with attention to capture temporal dynamics, which is defined
as: et= √
1 dt [cos(w1t+ b1), · · · , cos(wdtt+ bdt)].
[w1, · · · , wdt ] and [b1, · · · , bdt ] are trainable parameters for transformation weights and biases. dt is the dimension of time embedding. Based on it, a temporal GCN is proposed by fusing neighbor information with the heterogeneous attention:
hk+1o =W k h1h k o +
∑
(s,r,t)∈N̂ (o)
αks,r,o,tW k h2m k s,r,t, (1)
where W denotes a transformation matrix. N̂ is the neighbor set. mks,r,t is the message information of neighbors that contains subject, relation and time representations, which is given by:
mks,r,t=W k m1 [ (hks + et)⊙ (gkr + et) ] . (2)
h and g are the entity and relation embeddings, respectively. ⊙ is the element-wise product of two embedding vectors. αks,r,o,t is a heterogeneous attention value to determine the importance of a current temporal edge. It is obtained by the correlation
between time, relation and the current entities:
aks,r,o,t=σ ( (αk)⊤Wka[et∥gkr∥(hks − hko)] ) , αks,r,o,t= exp(aks,r,o,t)∑
(s′,r′,t′)∈N̂ (o) exp(a k s′,r′,o,t′)
, (3)
where σ is LeakyReLU (Xu et al., 2015). ∥ is concatenation. αk is the attention vector to be learned.
Finally, the relation embedding is updated by gk+1r = Wkrgkr . At the last layer K, the representation matrix H, G and E of entity, relation and time are obtained, then feeding into the logical decoder.

heading: 4.2 Logical Decoder
text: For decoding the answer for query (s̃,r̃,?,t̃), we introduce an iterative forward message-passing mechanism in a continuously expanding reasoning graph, regulated by propositional and first-order reasoning. In the reasoning graph, we set three learnable parameters for each node nli to guide the computation: node embedding nli, hidden FOTH embedding onli and reasoning attention βnli . The start node n
0=s̃ is initialized as its embedding hs̃. A hidden FOTH representation on0 for n0 is initialized as a query relation embedding gr̃. The attention weight βn0 for n0 is initialized as 1. The node ni=(ei, ti) are firstly represented by the linear transformation of GCN embeddings: ni=Wn[hei∥eti ]. Constant forward computation is required in the reasoning sequence of the target, whether conducting multi-hop propositional reasoning or first-order logic reasoning. Thus, forward message-passing is proposed to pass information (i.e., representations and attention weights) from the prior nodes to their posterior
neighbor nodes. The computation of each node is contextualized with prior information that contains both entity-dependent and entity-independent parts, reflecting the continuous accumulation of knowledge and credibility in the reasoning process.
Specifically, to update node embeddings in step l+1, its own feature and the information from its priors are integrated:
nl+1j =W l n1nj +
∑
(nli,r̄)∈Ñ (n l+1 j )
βnli,r̄,n l+1 j Wln2mnli,r̄,nl+1j , (4)
where mnli,r̄,nl+1j is the message from a prior node to its posterior node, which is given by the node and relation representations:
mnli,r̄,nl+1j =W l m2[n l i∥gr̄∥nj ]. (5)
This updating form superficially seems similar to the general message-passing in GCNs. However, they are actually different as ours is in a one-way and hierarchical manner, which is tailored for the tree-like structure of the reasoning graph.
The attention weight βnli,r̄,nl+1j for each edge in a reasoning graph contains two parts: propositional and first-order attention. As propositional attention is entity-dependent, we compute it by the semantic association of entity-dependent embeddings between the message and the query:
e1 nli,r̄,n l+1 j = SIGMOID(Wlp[mnli,r̄,nl+1j ∥q]), (6)
where q = Wq[hs̃∥gr̃∥et̃] is the query embedding. As first-order reasoning focuses on the interaction
among entity-independent relations, we first obtain the hidden FOTH embedding of an edge by fusing the hidden FOTH embedding of the prior node and current relation representation via a gated recurrent unit (GRU) (Chung et al., 2014). Then, the firstorder attention is given by:
onli,r̄,nl+1j =GRU(gr̄, onli), e2 nli,r̄,n l+1 j = SIGMOID(Wlfonli,r̄,nl+1j ). (7)
Furthermore, the overall reasoning attention can be obtained by incorporating propositional and firstorder parts to realize the complementarity of these two reasoning methods. Since the prior node with high credibility leads to faithful subsequent nodes, the attention of the prior flows to the current edge. Then, the softmax normalization is utilized to scale edge attentions on this iteration to [0,1]:
enli,r̄,n l+1 j =βnli (e1 nli,r̄,n l+1 j +λe2 nli,r̄,n l+1 j ),
βnli,r̄,n l+1 j
= exp(enli,r̄,n l+1 j ) ∑
(nl i′ ,r̄ ′)∈Ñ (nl+1j ) exp(enl i′ ,r̄ ′,nl+1j
) ,
(8)
where λ is the weight for balancing the two reasoning types. Finally, the FOTH representation and attention of a new node nl+1j are aggregated from edges for the next iteration:
onl+1j = ∑
(nli,r̄)∈Ñ (n l+1 j )
βnli,r̄,n l+1 j onli,r̄,nl+1j ,
βnl+1j =
∑
(nli,r̄)∈Ñ (n l+1 j )
βnli,r̄,n l+1 j
. (9)
Insights of FOTH Rule Learning and Reasoning. In general, the learning and reasoning of first-order logical rules on KGs or TKGs are usually in twostep fashion (Galárraga et al., 2013, 2015; Qu and Tang, 2019; Zhang et al., 2019; Qu et al., 2021; Vardhan et al., 2020; Liu et al., 2022; Cheng et al., 2022; Lin et al., 2023). First, it searches over whole data to mine rules and their confidences. Second, for a query, the model instantiates all variables to find all groundings of learned rules and then aggregates all confidences of eligible rules. For example, for a target entity o, its score can be the sum of learned rules with valid groundings and rule confidences can be modeled by a GRU. However, this is apparently not differentiable and cannot be optimized by an end-to-end manner. Thus, our model conducts the transformation of merging multiple
rules by merging possible relations at each step, using first-order attention as:
So= ∑
γ∈Γ βγ
= ∑ γ∈Γ f [ GRU(gγ,h, gγ,b1 , · · · , gγ,b|γ|)]
≈ L∏
l=1
∑ nj∈Ol f̄l [ GRU(gr̄, o l nj )) ] .
(10)
βγ is the confidence of rule γ. gγ,h and gγ,bi are the relation embeddings of head h and i-th body bi of this rule. f̄l is for the attention calculation. In this way, the differentiable process is achieved. This is an extension and progression of Neural-LP (Yang et al., 2017) and DURM (Sadeghian et al., 2019) on TKGs. Figure 3 intuitively illustrates such transformation. Finally, the real FOTH rules can be easily induced to constantly perform attention calculation over the reasoning graph, which is summarized as FARI in Algorithm 1.

heading: 4.3 Extrapolation Prediction
text: After attention weights for nodes in the last decoding step L have been obtained, we can aggregate node attentions with the same entity to get the entity score: So= ∑ nLi =(o,ti) βnLi . All entity scores can be normalized into [0,1] by ŷo= So∑ p Sp
. Compared with the true label yo, the model can be optimized by a binary cross-entropy loss:
L=− ∑
o
yo log(ŷo)+(1−yo)(1−log(ŷo)). (11)
The number of nodes may explode in the logical decoder as it shows an exponential increase to
Algorithm 1: FARI for FOTH rules. Input: the reasoning graph G̃, attentions e2. Output: the FOTH rule set Γ.
1 Init Γ = ∅, B(n0s̃) = [0, []], D0[n0s̃] = [1, B(n0s̃)]; 2 for l=1 to L of decoder iterations do 3 Initialize node-rule dictionary Dl; 4 for node nlj in Ol do 5 Set rule body list B(nlj) = [] ; 6 for (nl−1i , r̄) of Ñ (nlj) in Ol−1 do 7 Prior e2i,l−1, B(n l−1 i ) = Dl−1[nl−1i ]; 8 for weight ϵ, body γb in B(nl−1i ) do 9 ϵ′ = e2i,l−1 · e2nl−1i ,r̄,nlj ; 10 γ′b = γb.add(r̄), B(nlj).add([ϵ ′, γ′b]) ;
11 e2j,l = sum{[ϵ ∈ B(nlj)]} ; 12 Add nlj : [e 2 j,l, B(n l j)] to Dl ; 13 Normalize e2j,l of n l j in Ol using softmax; 14 for nLi in OL do 15 e2i,L, B(n L i ) = DL[nLj ] ; 16 for ϵ, γb in B(nLi ) do 17 Γ.add([ϵ, γb[1](X,Y1) : t1 ∧ · · · ∧
γb[L](YL−1, Z) : tL ⇒ r̃(X,Z) : t])
18 Return rule set Γ.
reach |N (ni)|L by iterations. For computational efficiency, posterior neighbors of each node are sampled with a maximum of M nodes in each iteration. For sampling M node in the reasoning graph, we follow a time-aware weighted sampling strategy, considering that recent events may have a greater impact on the forecast target. Specifically, for a posterior neighbor node with time t′, we compute its sampling weight by exp(t
′−t̃)∑ t̄ exp(t̄−t̃) for the query
(s̃,r̃,?,t̃), where t̄ denotes the time of all possible posterior neighbor nodes for a prior node. After computing attention weights for each edge in the same iteration, we select top-N among them with larger attention weights and prune others. As we add an extra self relation in the reasoning graph, the FARI algorithm can obtain all possible rules (no longer than length L) by deleting existing atoms with the self relation in induced FOTH rules.

heading: 5 Experiments and Results

heading: 5.1 Datasets and Experiment Setup
text: We conduct experiments on five common TKG datasets for extrapolation reasoning, i.e., ICEWS14, ICEWS18, ICEWS0515, WIKI (Leblay and Chekol, 2018) and YAGO (Mahdisoltani et al., 2015), which are the union ones of model xERTE, TITer and TLogic. The first three are all the
subsets of Integrated Crisis Early Warning System (O’brien, 2010). The last two contain massive real facts that are distinguished by years. The statistics of these five datasets are detailed in Table 1.
For training and testing, we add an inverse relation for each relation in TKGs. Thus, for the head entity prediction of query (?, r̃, õ, t̃), we can predict results by its variant (õ, r̃−1, ?, t̃). For testing, time-filter setting is used in which all correct entities at the query time except for the true query object are filtered out from answers. For entities out of the final iteration of the reasoning graph, we set their scores as 0. Mean reciprocal rank (MRR) and Hits@k (H@k for abbreviation, k is 1, 3 or 10) are selected as evaluation metrics, where larger values denote better performance. The above settings are all in line with baselines for equal comparison.
We introduce fourteen baselines in three technical trends: (1) Static Embedding: TransE (Bordes et al., 2013), DistMult (Yang et al., 2015) and ComplEx (Trouillon et al., 2016). (2) Temporal Embedding: TTransE (Leblay and Chekol, 2018), TA-DistMult (García-Durán et al., 2018), TA-TransE (García-Durán et al., 2018), DESimplE (Goel et al., 2020), TNTComplEx (Lacroix et al., 2020), RE-Net (Jin et al., 2020) and CyGNet (Zhu et al., 2021). (3) Explainable Reasoning: xERTE (Han et al., 2021), TITer (Sun et al., 2021), AnyBURL (Meilicke et al., 2019) and TLogic (Liu et al., 2022). When conducting experiments, the default max number of sampled nodes and selected edges are 600 and 100, respectively. The learning rate, GCN layers, GCN dimensions, iteration steps, decoder dimensions and first-order weight λ are set to 0.001, 2, 200, 3, 50 and 0.65 by default. Adam algorithm (Kingma and Ba, 2015) is utilized to optimize the model parameters. When conducting experiments, out model is implemented in DGL (Wang et al., 2019) and PyTorch (Paszke et al., 2019), and trained on a single GPU of NVIDIA Tesla V100 with 32G memory.

heading: 5.2 Comparison Results
text: In each dataset, we run five times with different random seeds and report their mean results in Ta-
ble 2 and Table 3. As shown, our TECHS has achieved advanced performance. Compared with static embedding and temporal embedding models, e.g., the strongest RE-Net, our metrics have been greatly improved by 5.6%, 5.91%, 8.02% and 7.43% in ICEWS14. The performance of TECHS is also competitive with the explainable reasoning methods. It outperforms xERTE, TITer and AnyBURL by 3.09%, 2.15% and 14.21% MRR in ICEWS14, respectively. It demonstrates TECHS makes up for the shortcomings of simply using
propositional reasoning or static first-order logical rules on TKGs. Finally, compared with the state-of-the-art TLogic, TECHS also shows certain improvements, i.e., achieving better performance on all twelve metrics of ICEWS14, ICEWS0515 and ICEWS18 datasets. TECHS has an average improvement of 0.92%, 1.65% and 1.26% on these three datasets. Besides, TECHS yields 0.48%, 3.37%, 1.77% and 2.12% improvements in MRR and Hits@10 metrics in WIKI and YAGO datasets, compared with the state-of-the-art TITer. In summary, the results show the superiority of our model that conducts temporal graph embedding as well as integrates propositional and first-order reasoning.

heading: 5.3 Ablation Studies
text: To verify the effectiveness of each module in TECHS, ablation studies are carried out in Table 4. For “w/o time”, we remove the time embedding in the GCN. “w/o emd” means we remove the whole GCN encoder module and perform random initialization for embeddings. For the logical decoder, “w/o PR” or “w/o FO” means that we remove propositional or first-order attention in Eq. 8 when computing nodes’ attention for the ablation of the corresponding reasoning pattern. We analyze the results from the following two aspects: First, both topology structures and time dynamics in GCN embeddings contribute to extrapolation reasoning. When only removing time information, the metrics decrease slightly compared with the whole GCN ablation, e.g., 0.44% vs. 1.43% MRR drops in ICEWS14. Second, for logical reasoning, both propositional and first-order logic reasoning
is important. Propositional reasoning has a bigger impact in ICEWS14 than first-order reasoning (3.54% vs. 1.89% Hits@10 drops), while they have roughly the same effect in ICEWS0515 and ICEWS18 (3.56% vs. 3.43%, 2.89% vs. 2.23% Hits@10 drops). This may be due to the different topology structures of different datasets, resulting in different logical reasoning patterns. In summary, ablation studies show that structural dependencies and temporal dynamics as well as propositional and first-order reasoning all bring positive gains.

heading: 5.4 Hyperparameter Analysis
text: We run our model with different hyperparameters to explore weight impacts in Figure 4. Figure 4a shows the changes in the performance of models with different sampling hyperparameters M and N, where small values would lead to great performance decline. This is because fewer nodes and edges lead to insufficient and unstable training, respectively. When increasing M and N, the GPU memory of the model will increase rapidly in Figure 4b, especially for M. We also record the average training time of one epoch with different M and N in Figure 4c. Its overall trend is consistent with Figures 4a and 4b. In general, TECHS is time efficient as the running time is between 0.2 and 1 hour.
Figure 4d shows the impact of different weights when using first-order reasoning, where smaller weights show worse results, generally. Thus, the FOTH rule is functional for extrapolation reasoning on TKGs. Different contextualized, e.g., vanilla RNN, GRU, LSTM (Hochreiter and Schmidhuber, 1997) for FOTH rule learning and reasoning are compared in Figure 4e, where GRU outperforms the other two competitors. RNN performs worst, showing that simple models are not competent enough for discrete structures of FOTH rules.
To explore the effects of decoder iterations on model performance, we carry out experiments with iteration L=1, 2, 3, 4 in ICEWS14, ICEWS0515 and ICEWS18. As Figure 4f shows, the performance generally improves with the iteration increasing. The metrics of L=3 and L=4 are similar, which shows that the answer is usually in the adjacent hops of the target entity. Larger hops bring more candidates, which may affect model performance, e.g., Hits@10 values drop when L=4 in ICEWS14 and ICEWS18. Therefore, L=3 is selected as the default setting in our experiments.

heading: 5.5 Case Study for Explainable Reasoning
text: Figure 5 visualizes two reasoning graphs on ICEWS14 and ICEWS0515, showing the extrapolation reasoning process of TECHS. The propositional attention weights of nodes are listed nearby them, which represent the propositional reasoning score of each node at the current step. For example, the uppermost propositional reasoning path from Massoud Barzani to Iran: 2014-08-26 in case B learned a large attention score for the correct answer Iran. Generally, nodes with more prior neighbors or larger prior attention weights significantly impact subsequent steps and the prediction of final entity scores. From both reasoning cases, we induce several FOTH rules using the FARI algorithm. Some typical ones with their confidence scores are shown in Table 5. For example, the rule [7] with lower confidence is learned for the prediction of the false candidate Iraq in case B. These attentions and FOTH rules demonstrate the explainability of our model. Besides, we observe that propositional and first-order reasoning have an
incompletely consistent effect. Thus, they can be integrated to jointly guide the reasoning process, leading to more accurate reasoning results.

heading: 6 Conclusion
text: To effectively integrate complex information on TKGs and implement differentiable logical reasoning, this work proposes TECHS which mainly contains a temporal graph encoder and a logical decoder. The former utilizes the temporal encoding and heterogeneous attention to embed structural dependencies and temporal dynamics. The latter realizes differentiable rule learning and reasoning by continuously conducting forward message-passing in the proposed reasoning graph. Finally, FOTH rules can be easily induced by a novel FARI algorithm. In the future, we will explore mining more types of rules on TKGs, such as numerical rules (Wang et al., 2020), and expand to the scenario of inductive reasoning (Pan et al., 2022).

heading: 7 Limitations
text: Due to the massive combination of relations and times on TKGs, balancing the model performance and efficiency is challenging. Our model TECHS performs well as Section 5.2 and 5.4 discussed. However, there is also a limitation. TECHS is a
two-step approach that can be further improved if we can fuse logical reasoning in the graph encoder like ConGLR (Lin et al., 2022). The model will be more efficient for computational space and time.

heading: Acknowledgments
text: This work was supported by National Key Research and Development Program of China (2022YFC3303600), National Natural Science Foundation of China (62137002, 62293553, 62250066, 62176207, 62192781, and 62250009), Innovative Research Group of the National Natural Science Foundation of China (61721002), “LENOVO-XJTU” Intelligent Industry Joint Laboratory Project, Foundation of Key National Defense Science and Technology Laboratory (6142101210201), Project of China Knowledge Centre for Engineering Science and Technology, Natural Science Basic Research Program of Shaanxi (2023-JC-YB-293), the Youth Innovation Team of Shaanxi Universities, XJTU Teaching Reform Research Project “Acquisition Learning Based on Knowledge Forest”.
Ethical Statement
We honor the ethical code set out in the ACL Code of Ethics.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
3 A2. Did you discuss any potential risks of your work? 4.3
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 5.4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5.1
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No. We follow the same experimental setting and result presentation of previous studies for equal comparison. We run five times with different random seeds and report their mean results for each dataset.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No. We did not use such packages.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will explore mining more types of rules on TKGs, such as numerical rules (Wang et al. , 2020), and expand to the scenario of inductive reasoning (Pan et al. , 2022)