SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xueyuan Lin<sup>1</sup> Haihong E<sup>1*</sup> Chengjin Xu<sup>2*</sup> Gengxian Zhou<sup>1</sup>  
Haoran Luo<sup>1</sup> Tianyi Hu<sup>1</sup> Fenglong Su<sup>3</sup> Ningyuan Li<sup>1</sup> Mingzhi Sun<sup>1</sup>  
<sup>1</sup> Beijing University of Posts and Telecommunications  
<sup>2</sup> University of Bonn
<sup>3</sup> National University of Defense Technology linxy59@mail2.sysu.edu.cn, ehaihong@bupt.edu.cn, xuc@iai.uni-bonn.de z.gengxian@se18.qmul.ac.uk, {luohaoran, hutianyi}@bupt.edu.cn sufenglong18@nudt.edu.cn, {jason.ningyuan.li, sunmingzhi}@bupt.edu.cn
Multi-hop logical reasoning over knowledge graph plays a fundamental role in many artificial intelligence tasks. Recent complex query embedding methods for reasoning focus on static KGs, while temporal knowledge graphs have not been fully explored. Reasoning over TKGs has two challenges: 1. The query should answer entities or timestamps; 2. The operators should consider both set logic on entity set and temporal logic on timestamp set. To bridge this gap, we introduce the multi-hop logical reasoning problem on TKGs and then propose the first temporal complex query embedding named Temporal Feature-Logic Embedding framework (TFLEX) to answer the temporal complex queries. Specifically, we utilize fuzzy logic to compute the logic part of the Temporal Feature-Logic embedding, thus naturally modeling all first-order logic operations on the entity set. In addition, we further extend fuzzy logic on timestamp set to cope with three extra temporal operators (After, Before and Between). Experiments on numerous query patterns demonstrate the effectiveness of our method.
Multi-hop logical reasoning over knowledge graphs (KGs) is a fundamental issue in artificial intelligence. It aims to find the answer entities for a first-order logic (FOL) query which involves logical operators (existential quantification  $\exists$ , conjunction  $\land$ , disjunction  $\lor$  and negation  $\neg$ ). Generally, the query is parsed into computation graph, according to which subgraph matching is executed on KG to find the answers. The computation graph is a directed acyclic graph (DAG) whose nodes represent entity sets, and edges represent logical operators acting on entity sets. However, results are inevitably incorrect as KGs are incomplete and noisy. Besides, the computation complexity will spiral for large-scale KGs or large queries. Therefore, people propose to embed query into low-dimensional space to solve the problem.
Query embedding (QE) learns the embeddings of queries and entities, so that answer entities are close to its queries in the embedding space. It has attracted arising attention, as low-dimension embeddings can model implicit dependency and reduce computation. There follows a series of QE methods, including GQE[1], Query2box[2], BetaE[3], ConE[4], etc. However, existing works only focus
Temporal Complex Query
$$
q \left[ V _ {?} \right] = V _ {?}, \exists T _ {a}, T _ {b}, T _ {c}:
$$
be_elected_as(Francois Hollande, President of France,  $T_{a}$ )  $\wedge$  After $(T_{a}, T_{c}) \wedge$
step_down_from(Francois Hollande, President of France,  $T_{b}$ )  $\wedge$  Before $(T_{b}, T_{c})\wedge$
make_a.visit(Xi Jinping,  $V_{?},T_{c})\wedge \neg$  make_a.visit(Barack Obama,  $V_{?},T_{c}$
Computation Graph
Figure 1: A typical multi-hop temporal complex query and its computation graph: "During François Hollandé was the president of France, which countries did Xi Jinping visit but Barack Obama did not visit?" In the computation graph, there are entity set (blue circle), timestamp set (green triangle), time set projection (green arrow), entity set projection (blue arrow) and logical operators (red rectangle).
$e_1$  : François Hollande
$e_3$  : François Hollande
$r_1$  : be elected as
$r_2$  : step down from
$e_2$  : President of France
$e_4$  : President of France
$e_5$  : Xi Jinping
$e_6$  : Barack Obama
$r_3$  : make a visit
$r_4$  :makeavisit
on queries over static KGs. These methods can neither handle an entity query involving temporal information and operators, nor answer the timestamp set for a temporal query.
Temporal knowledge graph (TKG) augments triples in static knowledge graphs with temporal information, represented as <head, relation, tail, timestamp> named fact. For example, the fact <Angela Merkel, make a visit, China, 2010-07-16> specifies the time when the event happens. Generally speaking, TKGs are more close to real world than static KGs, because most knowledge needs to be updated with time, and static KGs cannot express this change. In recommendation systems, TKGs are used to model user behaviors, which includes liking, buying, reading, commenting and so on. In financial applications, TKGs involve stock holding behaviors, trading behaviors, and financial events. These applications require reasoning over TKGs to answer complex queries. However, recent researches in TKGs focus on temporal link prediction, which is simply single-hop. A complex logical query involving multiple facts for multi-hop reasoning is not fully explored yet.
To fill the gap, we introduce a temporal multi-hop logical reasoning task over TKGs. The task aims to answer temporal complex queries, which have two main distinctions from existing queries over static KGs. Firstly, the answer sets for queries over TKGs are either entity sets or timestamp sets, while that for existing queries over static KGs can only be entity sets. Secondly, as temporal information is included in the query, temporal operators such as After, Before should be considered apart from FOL operators. To understand this new task, we provide the definition of temporal complex query in Section 3. In addition, an example of a temporal complex query is shown in Figure 1. This example query pertains to the financial event of China's visit and may be of interest to financial analysts.
According to the definition of the temporal complex query, we then generate datasets of temporal complex queries on three popular TKGs, and propose the first temporal complex query embedding framework, Temporal Feature-Logic Embedding framework (TFLEX) to answer these queries. In our framework, embeddings of objects (entity, query, timestamp) are divided into two parts, the entity part, and the timestamp part. The entity part handles the FOL operations over entities, while the timestamp part copes with the temporal operations over timestamps. Each part is further divided into feature and logic components. On the one hand, the computation of the logic components follows fuzzy logic, which enables our framework to handle all FOL operations. On the other hand, feature components are mingled and transformed under the guidance of logic components, thereby integrating logical information into the feature. Moreover, we extend fuzzy logic to support extra temporal operations (After, Before and Between) to handle temporal operations in the queries.
The contributions of our work are summarized as follows: (1) For the first time, the definition of the task of multi-hop logical reasoning over TKGs is given. (2) We propose the first multi-hop logical reasoning framework on TKGs, namely Temporal Feature-Logic Embedding framework (TFLEX),
which supports all FOL operations and extra temporal operations (After, Before and Between). (3) We generate three new TKG datasets for the task of multi-hop logical reasoning. Experiments on three generated datasets demonstrate the efficacy of the proposed framework. The source code of our framework and the datasets are available online * .
Complex Query Embedding. It learns embeddings of queries and entities, and the answer entities are close to queries in the embedding space. Existing methods utilize a lot of objects to create the embeddings, such as (1) probability distribution [3, 5] (2) geometric object [1, 2, 4, 6, 7] (3) fuzzy logic [8-10] and (4) others [11-13]. However, existing embedding-based methods are considered on static KGs. They cannot utilize temporal information in the TKGs, and therefore cannot handle temporal queries on a temporal KG. Firstly, static query embeddings (QEs) are built over (s, r, o) triples instead of (s, r, o, t) quartets, thus ignoring the timestamps for temporal complex reasoning. The second reason is the order property of timestamps, which is on the contrary that entities are unordered, leading to that static QEs are unable to handle Before and After temporal logic. In addition, we also notice the semantic conflict in experiments (see section 5.2) when concatenating the geometric embedding (static QE) with the fuzzy embedding (that can handle temporal logic) to promote the static QE to temporal one. Therefore, it is challenging for static QEs to utilize temporal information in the TKGs.
Temporal Knowledge Graph Completion (TKGC). It aims at inferencing new facts in the TKGs. Existing TKGC methods could be categorized to (1) tensor decomposition [14-16], (2) timestamp-based transformation [17-21], (3) dynamic embedding [22-25], (4) Markov process models [26, 27], (5) autoregressive models [28-30], (6) others [31-33] and so on. Among these works, most of them only confined to the one-hop link prediction task, also known as one-hop reasoning. Some works [25, 28-30, 32] can perform multi-hop reasoning via a path consisting of connected quartets. But none of them could answer logical queries that involve multiple logical operations (conjunction, negation and disjunction). In this paper, we focus on the temporal complex query answering task, which is more challenging than TKGC task.
Temporal Knowledge Graph (TKG)  $G = \{\mathcal{V},\mathcal{R},\mathcal{T},\mathcal{F}\}$  consists of entity set  $\mathcal{V}$ , relation set  $\mathcal{R}$ , timestamp set  $\mathcal{T}$  and fact set  $\mathcal{F} = \{(s,r,o,t)\} \subseteq \mathcal{V} \times \mathcal{R} \times \mathcal{V} \times \mathcal{T}$  containing subject-predicate-object-timestamp  $(s,r,o,t)$  quartets. Without loss of generality,  $G$  is a first-order logic knowledge base, where each quartet  $(s,r,o,t)$  denotes an atomic formula  $r(s,o,t)$ , with  $r$  a predicate and  $s,o,t$  its arguments.
Multi-hop Logical Reasoning over TKG is the task to answer Temporal Complex Query  $q$  when given a TKG  $G$ . We focus on Existential Positive First-Order (EPFO) query [34] over TKG, namely Temporal Complex Query  $q$ , which is categorized into entity query and timestamp query. Formally, the query  $q$  consists of a target variable  $A$ , a non-variable anchor entity set  $V_{a} \subseteq \mathcal{V}$ , a non-variable anchor timestamp set  $T_{a} \subseteq \mathcal{T}$ , bound variables  $V_{1},\dots ,V_{k}$  and  $T_{1},\dots ,T_{l}$ , logical operations (existential quantification  $\exists$ , conjunction  $\land$ , disjunction  $\lor$ , identity 1, negation  $\neg$ ), and extra temporal operations (After, Before). After  $(t_1,t_2)$  means  $t_2$  is after  $t_1$ , while Before  $(t_1,t_2)$  means  $t_2$  is before  $t_1$ . Inspired by [2, 3], the disjunctive normal form (DNF) of temporal query  $q$  is defined as:
$$
q [ A ] = A: \exists V _ {1}, \dots , V _ {k}, T _ {1}, \dots , T _ {l}: \left(e _ {1} ^ {1} \wedge \dots \wedge e _ {n _ {1}} ^ {1}\right) \vee \dots \vee \left(e _ {1} ^ {m} \wedge \dots \wedge e _ {n _ {m}} ^ {m}\right)
$$
where  $e = f \circ r(V_s, V_o$  or  $A, T)$  or  $f \circ r(V_s$  or  $A, V_o, T)$  or  $g(T_i, T_j)$  if  $q$  is entity query,
$e = f\circ r(V_{s},V_{o},T$  or  $A$  ) or  $g(T_{i},T_{j})$  or  $g(T,A)$  or  $g(A,T)$  if  $q$  is timestamp query
with  $V_{s}, V_{o} \in V_{a} \cup \{V_{1}, \dots, V_{k}\}$ ,  $T, T_{i}, T_{j} \in T_{a} \cup \{T_{1}, \dots, T_{l}\}$ ,
$$
r \in \mathcal {R}, f \in \{1, \neg \}, g \in \{\text {A f t e r}, \text {B e f o r e} \}
$$
In the equation, the DNF is a disjunction of  $m$  conjunctions, where  $e_1^j \wedge \dots \wedge e_{n_j}^j$  denotes a conjunction between  $n_j$  logical atoms, and each  $e_i^j$  denotes a logical atom. We ignore indices in the definition
*https://github.com/LinXueyuanStdio/TFLEX
of  $e_i^j$  to keep the formula clean. The goal of answering the query  $q$  is to find the set of entities (or timestamps)  $[[q]]$  that satisfy the query, such that  $A \in [[q]]$  iff  $q[A]$  holds true, where  $[[q]]$  is the answer set of the query  $q$ .
Following existing static query embedding works, we introduce Computation Graph, which is a directed acyclic graph (DAG) representing the structure of temporal complex query. Its nodes represent entity/timestamp sets  $S \subseteq V_{a} \cup V \cup T_{a} \cup T$ , while directed edges represent logical or relational operations acting on these sets. A computation graph specifies how the reasoning of the query has proceeded on the TKG. Starting from anchor sets, we obtain the answer set after applying operations iteratively on non-answer sets according to the directed edges in the computation graph. The edge types on the computation graph are defined as follows:
1. Relational Projection \(\mathcal{P}\). Given an entity set \(S_{1} \subseteq \mathcal{V}\), a timestamp set \(S_{2} \subseteq \mathcal{T}\) (or an entity set \(S_{2} \subseteq \mathcal{V}\) for entity projection) and a relation \(r \in \mathcal{R}\), projection operation maps \(S_{1}\) and \(S_{2}\) to another set: \(S' = \left\{ \begin{array}{ll} \cup_{(v \in S_{1}, t \in S_{2})} \{v' | (v, r, v', t) \in \mathcal{F} \}, & \mathcal{P}\) is entity projection \\ \(\cup_{(v \in S_{1}, v' \in S_{2})} \{t | (v, r, v', t) \in \mathcal{F} \}, & \mathcal{P}\) is timestamp projection \end{array} \right.\)
2. Intersection  $\mathcal{I}$  (Union  $\mathcal{U}$ , etc.). Given entity sets or timestamp sets  $\{S_1, \dots, S_n\}$ , the intersection (union, etc.) operation computes logical intersection (union, etc.) of these sets  $\cap_{i=1}^{n} S_i$  ( $\cup_{i=1}^{n} S_i$ , etc.).
3. Complement/Negation  $\mathcal{C}$ . The complement set of a given set  $S$  is  $\bar{S} = \left\{ \begin{array}{ll} \mathcal{V} - S, & S \subseteq \mathcal{V} \\ \mathcal{T} - S, & S \subseteq \mathcal{T} \end{array} \right.$
4. Extended temporal operators  $f$ . Given a timestamp set  $S$ , extended operators compute a certain set of timestamps  $S'$ :  $S' = \begin{cases} \{t' | \text{for some } t' \in \mathcal{T}, t' > \max(S)\}, & f \text{ is After } \\ \{t' | \text{for some } t' \in \mathcal{T}, t' < \min(S)\}, & f \text{ is Before } \end{cases}$
In order to efficiently compute the answer set of a temporal complex query, we consider embedding the query set into a low-dimensional vector space, where the answer set is also represented by a continuous embedding vector. Formally, the Temporal Query Embedding  $\mathbf{V}_q$  of a query  $q$  is a continuous embedding vector, generated by executing operations according to the computation graph, starting from the temporal embeddings of anchor entity or timestamp sets. The Temporal Query Answer to the query  $q$  is the entity  $v$  (or timestamp  $t$ ) whose embedding  $\mathbf{v}$  (or  $\mathbf{t}$ ) has the smallest distance  $\text{dist}(\mathbf{v}, \mathbf{V}_q)$  (or  $\text{dist}(\mathbf{t}, \mathbf{V}_q)$ ) to the embedding of query  $q$ .
In this section, we replace the variables in the query formulation with temporal feature-logic embeddings, and perform logical operations via neural networks. We first introduce the temporal feature-logic embedding for entities, timestamps, and queries in Section 4.1. Afterwards, we introduce logical operators in Section 4.2 and how to train the model in Section 4.3.
4.1 Temporal Feature-Logic Embeddings for Queries and Entities
In this section, we design temporal embeddings for queries, entities and timestamps. In general, the answers to queries may be entities or timestamps. Therefore, we consider a part of the embedding as an entity part to answer entities, while the other is the timestamp part to answer timestamps. Formally, the embedding of query set  $\llbracket q\rrbracket$  is  $\mathbf{V}_q = (\pmb {q}_f^e,\pmb {q}_l^e,\pmb {q}_f^t,\pmb {q}_l^t)$  where  $\pmb {q}_f^e\in \mathbb{R}^d$  is entity feature,  $\pmb {q}_l^e\in [0,1]^d$  is entity logic,  $\pmb {q}_f^t\in \mathbb{R}^d$  is time feature,  $\pmb {q}_l^t\in [0,1]^d$  is time logic respectively,  $d$  is the embedding dimension. The parameter  $\pmb{q}_l$  is the uncertainty  $\pmb {q}_l\vec{s} +(1 - \pmb {q}_l)\vec{n}$  of the feature, according to fuzzy logic. An entity  $v\in \mathcal{V}$  is a special query without uncertainty. We propose to represent an entity as the query with logic part 0, which indicates that the entity's uncertainty is 0. Formally, the embedding of entity  $v$  is  $\mathbf{v} = (v_f^e,\mathbf{0},\mathbf{0},\mathbf{0})$ , where  $\boldsymbol {v}_f^e\in \mathbb{R}^d$  is the entity feature part and 0 is a  $d$ -dimensional vector with all elements being 0. Similarly, the embedding of timestamp  $t$  is  $\mathbf{t} = (\mathbf{0},\mathbf{0},t_f^t,\mathbf{0})$  with entity part and time logic being 0.
Attention that we introduce vector logic, which is a type of fuzzy logic over vector space, to cope with logical transformation in the vector space. Fuzzy logic is a generalization of Boolean logic, such that the truth value of a logical atom is a real number in  $[0,1]$ . In comparison, as a generalization
of a real number, the truth value in vector logic is a vector  $[0,1]^d$  in the semantic space. We denote the logical operations in vector logic as  $\mathbf{AND}(\land),\mathbf{OR}(\lor),\mathbf{NOT}(\neg)$ , and so on, which receive one or multiple vectors and output one vector as answer. For more details about fuzzy logic, please refer to Appendix A.1.
4.2 Logical Operators for Temporal Feature-Logic Embeddings
In this section, we introduce the designed neural logical operators, including projection, intersection, complement, and all other dyadic operators.
Projection Operator  $\mathcal{P}_e$  and  $\mathcal{P}_t$ . The goal of operator  $\mathcal{P}_e$  is to map an entity set to another entity set under a given relation and a given timestamp, while operator  $\mathcal{P}_t$  outputting timestamp set given relation and two entity queries. We define a function  $\mathcal{P}_e: \mathbf{V}_q, \mathbf{r}, \mathbf{V}_t \mapsto \mathbf{V}_q'$  in the embedding space to represent  $\mathcal{P}_e$ , and  $\mathcal{P}_t: \mathbf{V}_{q_1}, \mathbf{r}, \mathbf{V}_{q_2} \mapsto \mathbf{V}_q'$  for  $\mathcal{P}_t$  respectively. To implement  $P_e$  and  $P_t$ , we first represent relations as translations on query embeddings and assign each relation with relational embedding  $\mathbf{r} = (r_f^e, r_l^e, r_f^t, r_l^t)$ . We follow the assumption of translation-based methods:  $q_o \approx q_s + r + t$ . As a comparison, static KGE TransE [35] has  $o \approx s + r$ , and temporal KGE TTransE [36] has  $o_t \approx s_t + r$ . The addition represents a semantic translation starting from the source entity set, following the relation and timestamp conditioning, ending at the target entity set. Therefore, we define  $\mathcal{P}_e$  and  $\mathcal{P}_t$  as:
$$
\mathcal {P} _ {e} \left(\mathbf {V} _ {q}, \mathbf {r}, \mathbf {V} _ {t}\right) = g \left(\mathbf {M L P} _ {0} ^ {e} \left(\mathbf {V} _ {q} + \mathbf {r} + \mathbf {V} _ {t}\right)\right)
$$
$$
\mathcal {P} _ {t} \left(\mathbf {V} _ {q _ {1}}, \mathbf {r}, \mathbf {V} _ {q _ {2}}\right) = g \left(\mathbf {M L P} _ {0} ^ {t} \left(\mathbf {V} _ {q _ {1}} + \mathbf {r} + \mathbf {V} _ {q _ {2}}\right)\right) \tag {1}
$$
where  $\mathbf{MLP}:\mathbb{R}^{4d}\to \mathbb{R}^{4d}$  is a multi-layer perception network (MLP),  $+$  is element-wise addition and  $g$  is an activate function to generate  $\pmb{q}_l^e\in [0,1]^d$ $\pmb {q}_l^t\in [0,1]^d$  . We use MLP and activation function  $g(.)$  to make projection operator output a valid query embedding, which shows the closure property of the operator.  $\mathcal{P}_e$  and  $\mathcal{P}_t$  do not share parameters so the MLPs are different. We define  $g$  as:
$$
g (\mathbf {x}) = [ \mathbf {x} [ 0: d ]; \sigma (\mathbf {x} [ d: 2 d ]); \mathbf {x} [ 2 d: 3 d ]; \sigma (\mathbf {x} [ 3 d: 4 d ]) ] \tag {2}
$$
where  $\mathbf{x}[0:d]$  is the slice containing element  $x_{i}$  of vector  $\mathbf{x}$  with index  $0\leq i < d$ ,  $\sigma (\cdot)$  is Sigmoid function and  $[\cdot ;\cdot ]$  is the concatenation operator.
**Dyadic Operators.** There are two types of dyadic operators for our framework. One for entity set and the other for timestamp set. Each type includes intersection (AND), union (OR), the symmetric difference (XOR), etc. With the help of fuzzy logic, our framework can model all dyadic operators directly. Below we take a unified way to build these operators.
We start from intersection operators  $\mathcal{I}_e$  (on entity set) and  $\mathcal{I}_t$  (on timestamp set). The goal of intersection operator  $\mathcal{I}_e(\mathcal{I}_t)$  is to represent  $\llbracket q\rrbracket = \cap_{i=1}^{n}\llbracket q_i\rrbracket$  based on their entity parts (timestamp parts). Suppose that  $\mathbf{V}_{q_i} = (\pmb{q}_{i,f}^e, \pmb{q}_{i,l}^e, \pmb{q}_{i,f}^t, \pmb{q}_{i,l}^t)$  is temporal feature-logic embedding for  $\llbracket q_i\rrbracket$ . We notice that there exists Alignment Rule in the process of reasoning. When performing entity set intersection  $\mathcal{I}_e$ , we should also perform intersection on timestamp parts in order to align the entities into the same time set. The same also holds for timestamp set intersection  $\mathcal{I}_t$  and all other dyadic operators. Therefore, we firstly define the intersection operators as follows:
$$
\mathcal {I} _ {e} \left(\mathbf {V} _ {q _ {1}}, \dots , \mathbf {V} _ {q _ {n}}\right) = \left(\sum_ {i = 1} ^ {n} \boldsymbol {\alpha} _ {i} ^ {e} \boldsymbol {q} _ {i, f} ^ {e}, \quad \underset {i = 1} {\overset {n} {\mathrm {A N D}}} \{\boldsymbol {q} _ {i, l} ^ {e} \}, \sum_ {i = 1} ^ {n} \beta_ {i} ^ {e} \boldsymbol {q} _ {i, f} ^ {t}, \underset {i = 1} {\overset {n} {\mathrm {A N D}}} \{\boldsymbol {q} _ {i, l} ^ {t} \}\right)
$$
$$
\mathcal {I} _ {t} \left(\mathbf {V} _ {q _ {1}}, \dots , \mathbf {V} _ {q _ {n}}\right) = (\sum_ {i = 1} ^ {n} \boldsymbol {\alpha} _ {i} ^ {t} \boldsymbol {q} _ {i, f} ^ {e}, \underset {i = 1} {\overset {n} {\mathrm {A N D}}} \{\boldsymbol {q} _ {i, l} ^ {e} \}, \sum_ {i = 1} ^ {n} \beta_ {i} ^ {t} \boldsymbol {q} _ {i, f} ^ {t}, \underset {i = 1} {\overset {n} {\mathrm {A N D}}} \{\boldsymbol {q} _ {i, l} ^ {t} \})
$$
where AND is the conjunction in fuzzy logic,  $\alpha_{i}$  and  $\beta_{i}$  are attention weights. To notice the changes of logic, we compute  $\alpha_{i}^{e,t}$  and  $\beta_{i}^{e,t}$  via the following attention mechanism:
$$
\boldsymbol {\alpha} _ {i} ^ {e, t} = \frac {\exp (\mathbf {M L P} _ {1} ^ {e , t} ([ \boldsymbol {q} _ {i , f} ^ {e} ; \boldsymbol {q} _ {i , l} ^ {e} ])))}{\sum_ {j = 1} ^ {n} \exp (\mathbf {M L P} _ {1} ^ {e , t} ([ \boldsymbol {q} _ {j , f} ^ {e} ; \boldsymbol {q} _ {j , l} ^ {e} ]))}, \quad \beta_ {i} ^ {e, t} = \frac {\exp (\mathbf {M L P} _ {2} ^ {e , t} ([ \boldsymbol {q} _ {i , f} ^ {t} ; \boldsymbol {q} _ {i , l} ^ {t} ])))}{\sum_ {j = 1} ^ {n} \exp (\mathbf {M L P} _ {2} ^ {e , t} ([ \boldsymbol {q} _ {j , f} ^ {t} ; \boldsymbol {q} _ {j , l} ^ {t} ]))} \tag {3}
$$
Figure 2: The computation of time part in temporal operators Before and After.
where  $\mathbf{MLP}_{1,2}^{e,t}:\mathbb{R}^{2d}\to \mathbb{R}^d$  are MLP networks,  $[\cdot ;\cdot ]$  is concatenation. The first self-attention neural network will learn the hidden information from entity logic and leverage to entity feature, while the second one gathers logical information from time logic to time feature. Note that the computation of entity logic, and time logic obeys the law of fuzzy logic, without any extra learnable parameters. In this way, all dyadic operators can be generated from fuzzy logic in our framework. Due to space limitation, we present the union, exclusive or, implication operators and so on in Appendix A.3.