SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiaxin Bai  
Department of CSE  
HKUST  
jbai@connect.ust.hk
Xin Liu  
Department of CSE  
HKUST  
xliucr@cse.ust.hk
Weiqi Wang  
Department of CSE  
HKUST  
wwangbw@cse.ust.hk
Chen Luo
Amazon.com Inc cheluo@amazon.com
Yangqiu Song*  
Department of CSE  
HKUST  
yqsong@cse.ust.hk
Querying knowledge graphs (KGs) using deep learning approaches can naturally leverage the reasoning and generalization ability to learn to infer better answers. Traditional neural complex query answering (CQA) approaches mostly work on entity-centric KGs. However, in the real world, we also need to make logical inferences about events, states, and activities (i.e., eventualities or situations) to push learning systems from System I to System II, as proposed by Yoshua Bengio. Querying logically from an EVentuality-centric KG (EVKG) can naturally provide references to such kind of intuitive and logical inference. Thus, in this paper, we propose a new framework to leverage neural methods to answer complex logical queries based on an EVKG, which can satisfy not only traditional first-order logic constraints but also implicit logical constraints over eventualities concerning their occurrences and orders. For instance, if we know that Food is bad happens before PersonX adds soy sauce, then PersonX adds soy sauce is unlikely to be the cause of Food is bad due to implicit temporal constraint. To facilitate consistent reasoning on EVKGs, we propose Complex Eventuality Query Answering (CEQA), a more rigorous definition of CQA that considers the implicit logical constraints governing the temporal order and occurrence of eventualities. In this manner, we propose to leverage theorem provers for constructing benchmark datasets to ensure the answers satisfy implicit logical constraints. We also propose a Memory-Enhanced Query Encoding (MEQE) approach to significantly improve the performance of state-of-the-art neural query encoders on the CEQA task.
Querying knowledge graphs (KGs) can support many real applications, such as fact-checking and question-answering. Using deep learning methods to answer logical queries over KGs can naturally leverage the inductive reasoning and generalization ability of learning methods to overcome the sparsity and incompleteness of existing KGs, and thus has attracted much attention recently, which are usually referred to as Complex Query Answering (CQA) [37, 26, 38]. As the computational complexity of answering complex logical queries increases exponentially with the length of the query [37, 26], brute force search and sub-graph matching algorithms [29, 30, 28] are unsuitable
*Prof. Yangqiu Song is a visiting academic scholar at Amazon.
Queries	Type	Interpretations
q1 = V2..3V:Interact(V2, V) ∧ Assoc(V, Alzheimer) ∧ Assoc(V, MadCow)	Entity	Find the substances that interact with the proteins associated with Alzheimer's and Mad cow disease.
q2 = V2..Precedence(Food is bad,PersonX add soy sauce) ∧ Reason(Food is bad, V2)	Eventuality	Food is bad before PersonX add soy sauce. What is the reason for food being bad?
q3 = V2..Precedence(V2, PersonX go home) ∧ ChosenAlternative(PersonX go home,PersonX buy an umbrella)	Eventuality	Instead of buying an umbrella, PersonX go home. What happened before PersonX go home?
Figure 1: Complex query examples and corresponding interpretations in natural language.  $q_{1}$  is a query on an entity knowledge graph, while  $q_{2}$  and  $q_{3}$  are queries on an eventuality knowledge graph.
for processing complex queries. To overcome these challenges, various techniques, such as query encoding [23] and query decomposition [2], have been proposed. These techniques enable effective and scalable reasoning on incomplete KGs and facilitate the processing of complex queries.
Most of the existing work in this field has primarily focused on entity-centric KGs that only describe entities and their relationships. As Yoshua Bengio described in his view<sup>2</sup> of moving from System I to System II [16-18, 25, 13], we need to equip machine learning systems with logical, sequential reasoning, and other abilities. Particularly, such a system requires the understanding of how actions (including events, activities, or processes) interact with changes in distribution which can be reflected by states. Here we can summarize events, activities, and states as a linguistic term, eventualities (or situations), according to the linguistics literature [33, 4]. As with many other KG querying tasks, querying eventuality-centric knowledge graphs can also support many applications, such as providing references for making logical and rational decisions of intuitive inferences or eventual planning. This requires the CQA models to perform reasoning at the eventuality level. To provide resources for achieving eventuality-level reasoning, recently constructed KGs, such as ATOMIC [41, 24], Knowlywood [43], and ASER [52, 53], tend to use one or more discourse relations to represent the relationships between eventuality instances. For example, PersonX went to the store and PersonX bought some milk are two simple eventuality instances, with the latter being a possible consequence of the former. The construction of these EVentuality-centric Knowledge Graphs (EVKGs) thoroughly maps the relationships between eventualities and enables us to reason about eventuality instances and their relationships using logical queries, thereby facilitating a more comprehensive approach to modeling complex relationships than traditional knowledge graphs.
Aside from the importance of querying EVKGs, reasoning on EVKG also significantly differs from that on an entity-centric KG because eventualities involve considering their occurrences and order. In entity-centric KGs, as shown in Figure 1  $q_{1}$ , the vertices represent entities such as Alzheimer or Mad Cow Disease, and truth values are assigned to the edges between entities to indicate their relationships. For example, the statement Assoc(Beta - amyloid, Alzheimer) is true. In contrast, during the reasoning process on EVKG, the eventualities may or may not occur, and determining their occurrence is a crucial part of the reasoning. For instance, given ChosenAlternative(PersonX go home, PersonX buy umbrella) in Figure 1  $q_{2}$ , it implicitly suggests that "PersonX go home" occurs, while "PersonX buy umbrella" does not. Moreover, there are relationships that explicitly or implicitly describe the order of occurrences, such as temporal and causal relations. For example, Reason(PersonX study hard, PersonX pass exam) indicates the causality between "PersonX pass the exam" and "PersonX study hard," which also implies that "PersonX pass the exam" occurs after "PersonX study hard." When multiple edges are presented in a given situation, it is essential to ensure that there are no contradictions regarding the occurrence of these eventualities. For example, in Figure 1  $q_{3}$ , ChosenAlternative(PersonX go home, PersonX buy umbrella) ∧ Succession(PersonX go home, PersonX buy umbrella) is contradictory because the former suggests that PersonX did not buy an umbrella, while the latter implies otherwise.
To enable complex reasoning on eventuality knowledge graphs, we formally define the problem of complex eventuality query answering (CEQA). CEQA is a more rigorous definition of CQA on EVKG that consider not only the explicitly given relational constraints, but also the implicit logical constraints on the occurrences and temporal order of eventualities. The implicit constraints are derived from the relational constraints and can be further divided into two types: occurrence constraints and temporal constraints. Incorporating these implicit constraints into complex query answers drastically
$^{2}$ http://www.iro.umontreal.ca/~bengioy/AAAI-9feb2020.pdf
Figure 2: Complex eventuality queries with their implicit temporal and occurrence constraints
changes the nature of the reasoning process. Unlike conventional CQA, the reasoning process of CEQA is defeasible because when additional knowledge is presented, the original reasoning could be weakened and overturned [19]. For example, we showed in Figure 2,  $\text{PersonX adds soy sauce}$  is a possible answer to the query What is the reason for food being bad. However, if more knowledge is given, like  $\text{Food is bad}$  is before  $\text{PersonX adds soy sauce}$ , then it cannot be the proper reason anymore due to temporal constraints. However, all the existing methods for CQA cannot incorporate additional knowledge to conduct defeasible reasoning in CEQA.
To address this problem, we propose the method of memory-enhanced query encoding (MEQE). In the MEQE method, we first separate the logic terms in a query into two categories, computational atoms and informational atoms. Computational atoms, like Reason(Food is bad,  $V_{?}$ ), contains at least one variable in their arguments, and informational atoms, like Precedence(Food is bad, PersonX add soy sauce), does not contain variables. For the computational atoms, following previous work, we construct the corresponding computational graph to recursively compute its query embedding step-by-step. For the informational atoms, we put them into a key-value memory module. For each of the informational atoms, its head argument is used as the memory key, and its relation type and tail arguments are used as memory values. In the query encoding process, after each operation in the computational graph, a relevance score is computed between the query embedding and memory heads. This relevance score is then used to retrieve the corresponding memory values of the corresponding relation and tail. Then these memory values are aggregated, adjusted, and added back to the query embedding. By doing this, the query encoder is able to leverage implicit logical constraints that are given by the informational atoms. We evaluate our proposed MEQE method on the eventuality knowledge graph, ASER, which involves fourteen types of discourse relations between eventualities. Experiment results show that our proposed MEQE is able to consistently improve the performance of four frequently used neural query encoders on the task of CEQA. Code and data are publicly available<sup>3</sup>.
2 Problem Definition
In this section, we first introduce the definitions of the complex queries on entity-centric and eventuality-centric KGs. Then we give the definition of implicit logical constraints and the informational atomics that specifically provide such constraints to the eventuality queries.
2.1 Complex Queries
Complex query answering is conducted on a KG:  $\mathcal{G} = (\mathcal{V},\mathcal{R})$ . The  $\mathcal{V}$  is the set of vertices  $v$ , and the  $\mathcal{R}$  is the set of relation  $r$ . The relations are defined in functional forms to describe the logical expressions better. Each relation  $r$  is defined as a function with two arguments representing two vertices,  $v$  and  $v'$ . The value of function  $r(v,v') = 1$  if and only if there is a relation between the vertices  $v$  and  $v'$ .
In this paper, the queries are defined in conjunctive forms. In such a query, there are logical operations such as existential quantifiers  $\exists$  and conjunctions  $\land$ . Meanwhile, there are anchor eventualities  $V_{a} \in \mathcal{V}$ , existential quantified variables  $V_{1}, V_{2}, \ldots, V_{k} \in \mathcal{V}$ , and a target variable  $V_{?} \in \mathcal{V}$ . The query
<sup>3</sup>https://github.com/HKUST-KnowComp/CEQA
Table 1: The discourse relations and their implicit logical constraints.  $\eta (V)$  is True if and only if  $V$  occurs.  $\tau (V)$  indicates the happening timestamp of  $V$ . Meanwhile, the instance-based temporal logic operator  $\prec ,\succ$  or  $=$  means  $V_{1}$  is before, after, or at the same time as  $V_{2}$ .
Discourse Relations (ei)	Semantics	ImplicitConstraints
OccurrenceConstraints (o_i)	TemporalConstraints (t_i)
Precedence(V1,V2)	V1 occurs before V2.	η(V1) ∧ η(V2)	τ(V1) ∝ τ(V2)
Succession(V1,V2)	V1 occurs after V2 happens.	η(V1) ∧ η(V2)	τ(V1) ∝ τ(V2)
Synchronous(V1,V2)	V1 occurs at the same time as V2.	η(V1) ∧ η(V2)	τ(V1) = τ(V2)
Reason(V1,V2)	V1 occurs because V2.	η(V1) ∧ η(V2) ∧ (η(V1) ← η(V2))	τ(V1) ∝ τ(V2)
Result(V1,V2)	V1 occurs as a result V2.	η(V1) ∧ η(V2) ∧ (η(V1) → η(V2))	τ(V1) ∝ τ(V2)
Condition(V1,V2)	If V2 occurs, V1.	η(V1) → η(V2)	τ(V1) ∝ τ(V2)
Concession(V1,V2)	V2 occurs, although V1.	η(V1) ∧ η(V2)	-
Constrast(V1,V2)	V2 occurs, but V1.	η(V1) ∧ η(V2)	-
Conjunction(V1,V2)	V1 and V2 both occur.	η(V1) ∧ η(V2)	-
Instantiation(V1,V2)	V2 is a more detailed description of V1.	η(V1) ∧ η(V2)	-
Restatement(V1,V2)	V1 restates the semantics of V2.	η(V1) ← η(V2)	-
Alternative(V1,V2)	V1 and V2 are alternative situations.	η(V1) ∨ η(V2)	-
ChosenAlternative(V1,V2)	Instead of V2 occurs, V1.	η(V1) ∧ ¬η(V2)	-
Exception(V1,V2)	V1, except V2.	¬η(V1) ∧ η(V2) ∧ (¬η(V2) → η(V1))	-
is written to find the answers  $V_{?} \in \mathcal{V}$ , such that there exist  $V_{1}, V_{2}, \ldots, V_{k} \in \mathcal{V}$  satisfying the logical expression:
$$
q \left[ V _ {?} \right] = V _ {?}. \exists V _ {1}, \dots , V _ {k} := e _ {1} \wedge e _ {2} \wedge \dots \wedge e _ {m}. \tag {1}
$$
Each  $e_i$  is an atomic expression in any of the following forms:  $e_i = r(v_a, V)$ , or  $e_i = r(V, V')$ . Here  $v_a$  is an anchor eventuality, and  $V, V' \in \{V_1, V_2, \dots, V_k, V_\varphi\}$  are distinct variables.
2.2 Complex Eventuality Queries
For complex eventuality queries, they can also be written in the form of a conjunctive logical expression as Eq. (1). Differently, each atomic  $e_i$  can all be in the form of  $e_i = r(v_i, v_j)$ , where  $v_i, v_j \in V$  are given eventualities. These atomics, which do not include variables, are called informational atomics, because they only provide implicit constraints.
The relations  $r$  in CEQA are discourse relations, and they exert implicit constraints over the eventualities, and these constraints can be categorized into occurrence constraints and temporal constraints. Suppose the occurrence and temporal constraints derived from the  $i$ -th atomic  $e_i$  is denoted as  $o_i$  and  $t_i$ . Then complex eventuality query, including its implicit constraints can be written as
$$
q \left[ V _ {?} \right] = V _ {?} \cdot \exists V _ {1}, \dots , V _ {k} := \left(e _ {1} \wedge \dots \wedge e _ {m}\right) \wedge \left(o _ {1} \wedge \dots \wedge o _ {m}\right) \wedge \left(t _ {1} \wedge \dots \wedge t _ {m}\right). \tag {2}
$$
The constraints brought from each type of discourse relations are presented in Table 1. Further justifications of the derivation process are given in the Appendix B.
2.2.1 Occurrence Constraints
The occurrence constraints determine whether certain eventuality occurs or not. For instance, consider Figure 2 (A), where the logical query means that Instead of buying an umbrella, PersonX goes home. What occurred before PersonX went home? If we rely solely on relational constraints, as in the conventional definition of CQA, the answers are only determined by the latter part of the query, What happened before PersonX went home? Consequently, PersonX buys an umbrella could be a solution to this query. However, within the query, there is an information atomic saying, instead of buying an umbrella, PersonX goes home, which denies the occurrence of PersonX buying an umbrella. To formally express such constraint, we use the function  $\eta(V)$ . If eventuality  $V$  occurs, then  $\eta(V) = \text{True}$ , otherwise it is False. As depicted in Figure 2, the occurrence constraint of this query comprises the terms  $\eta(V_{?}) \land \neg \eta(\text{PersonX buy umbrella})$ . In this case,  $V_{?}$  cannot be PersonX buys an umbrella or there is an contradiction.
Most discourse relations assume the occurrence of the argument eventualities, for example, Precedence, Conjunction, and Reason. However, there are also relations that do not imply the occurrence of the arguments, such as Condition and Restatement. Moreover, the Exception and ChosenAlternative relations restrict certain eventualities from happening. For instance, in the
$q = V_{?}$ .  $\exists$  V: Succession(PersonX complains, V)  $\land$  Succession(PersonX leaves restaurant, V)  $\land$  Reason(V,  $V_{?})\land$  Precedence(Food is bad, PersonYadds soy sauce)  $\land$  ChosenAlternative(PersonY adds ketchup, PersonY adds vine
Figure 3: An example complex eventuality query with the computational and informational atomics.  $V$  is something that happens before a person complains and leaves the restaurant, according to the KG, the  $V$  could be either Service is bad or Food is bad. If  $V_{?}$  is the reason of  $V$ , then according to the graph,  $V_{?}$  could be either Staff is new, PersonY adds ketchup, PersonY adds soy sauce, and PersonY adds vinegar. However, in the query we also know that PersonY adds vinegar does not happen, and PersonY adds soy sauce happens after the Food is bad, thus cannot be the reason for Food is bad. The conflict here is causality implies precedence.
case of ChosenAlternative(PersonX read books, PersonX play games), it implies that PersonX reads books:  $\eta(\text{PersonX read books})$ , and does not play games:  $\neg \eta(\text{PersonX play games})$ . Another example is Exception(Room is empty, PersonX stay in room), which implies that the room is not empty and PersonX is present in the room. Furthermore, if PersonX is not in the room, then the room is empty. This can be formally expressed as  $\neg \eta(\text{Room is empty}) \land \eta(\text{PersonX stay in room}) \land (\neg \eta(\text{PersonX stay in room}) \rightarrow \eta(\text{Room is empty}))$ . For a comprehensive overview of the occurrence constraints, please refer to Table 1.
2.2.2 Temporal Constraints
The temporal constraints reflect the order of occurrence of the eventualities. As shown in Figure 2 (B), the complex query on the eventuality knowledge graph can be interpreted as  $Food$  is bad before  $PersonX$  adds soy sauce. What is the reason for food being bad? If we only considered the relational constraints, like in the conventional setting of CQA, then  $PersonX$  adds soy sauce is a possible answer. However, in the definition of CEQA, the answer  $PersonX$  adds soy sauce is incorrect because the food is bad already occurred before  $PersonX$  added soy sauce, but something that occurs later is impossible to be the reason for something that previously occurred. Formally, we use the expression of temporal logic  $\succ$ ,  $\prec$ , and  $=$  to describe the temporal order between two eventualities [22].  $\tau(A) \prec \tau(B)$  means  $A$  occurs before  $B$ , and  $\tau(A) = \tau(B)$  means they happen at the same time, and  $\tau(A) \succ \tau(B)$  means  $A$  occurs after  $B$ . For example in Figure 2 (B), the temporal constraint is represented by  $\tau(Food \text{ is bad}) \prec \tau(PersonX \text{ add soy sauce}) \wedge \tau(Food \text{ is bad}) \succ \tau(V_{\ref{eq:1}})$ , which can be interpreted as  $Food$  is bad is before  $PersonX$  adds soy sauce and  $V_{\ref{eq:1}}$  is before  $Food$  is bad. Because of this,  $V_{\ref{eq:1}}$  cannot  $PersonX$  adds soy sauce, otherwise there exists a contradiction.
The temporal relations  $\mathrm{Precedence}(A,B)$ ,  $\mathrm{Succession}(A,B)$ , and  $\mathrm{Synchronous}(A,B)$  naturally describes the temporal constraint. Meanwhile, previous studies also assume that causation implies precedence [40, 10, 54]. With this assumption, the temporal constraints can also be derived from relations like Reason and Result. The descriptions of temporal constraints are given in Table 1.
3 Memory-Enhanced Query Encoding
In this section, we first introduce the method of query encoding, and then introduce how to use the memory module to represent the informational atomics to conduct reasoning on EVKGs.
Figure 4: The example computational graph and the memory-enhanced query encoding process.
3.1 Computational Graph and Query Encoding
Figure 3 and 4 show that there is a computational graph for each query. This computational graph is a directed acyclic graph (DAG) that consists of nodes and edges representing intermediate encoding states and neural operations, respectively. By recursively encoding the sub-queries following the computational graph, the operations implicitly model the set operations of the intermediate query results. The set operations are defined as follows: (1) Relational Projection: Given a set of vertices  $A$  and a relation  $r \in R$ , the relational projection operation returns all eventualities that hold the relation  $r$  with at least one entity  $e \in A$ . This can be expressed as:  $P_r(A) = \{v \in \mathcal{V} \mid \exists v' \in A, r(v', v) = 1\}$ ; (2) Intersection: Given sets of eventualities  $A_1, \ldots, A_n \subseteq \mathcal{V}$ , the intersection computes the set that is the subset to all of the sets  $A_1, \ldots, A_n$ . This can be expressed as  $\bigcap_{i=1}^{n} A_i$ .
Various query encoding methods are proposed to recursively encode the computational graph. However, the query embeddings of these methods can be translated into  $d$ -dimensional vectors. As shown in Figure 4, the computations along the computation graph start with the anchor eventualities, such as PersonX complains. Suppose the embedding of an anchor  $v$  is denoted as  $e_v \in R^d$ . Then, the initial query embedding is computed as  $q_0 = e_v$ . As for the relational projection operation, suppose the  $e_{rel} \in R^d$  is the embedding vector of the relation rel. The relation projection  $F_{proj}$  is expressed as
$$
q _ {i + 1} = F _ {p r o j} \left(q _ {i}, e _ {r e l}\right), \tag {3}
$$
where the  $q_{i}$  and  $q_{i + 1}$  are input and output query embeddings for this relational projection operation. Meanwhile, for the Intersection operations, suppose there are  $k$  embeddings of sub-queries,  $q_{i}^{(1)}, q_{i}^{(2)}, \ldots, q_{i}^{(k)}$ , as the input for this operation, then the output can be expressed as:
$$
q _ {i + 1} = F _ {\text {i n t e r}} \left(q _ {i} ^ {(1)}, q _ {i} ^ {(2)}, \dots , q _ {i} ^ {(k)}\right), \tag {4}
$$
where the  $F_{inter}$  is a neural network that is permutation-invariant to the input sub-query embeddings adopted from the backbone models [23, 5, 1, 12].
3.2 Memory-Enhanced Query Encoding
The computational graph is capable of encoding computational atoms presented in the logical expression. However, informational atoms can influence the reasoning outcomes by introducing implicit temporal or occurrence constraints. As depicted in Figure 3, the absence of informational atoms results in two false answers from the knowledge graph. When informational atoms are included, providing implicit constraints, the only two correct answers can be derived.
Based on this observation, we propose using a memory module to encode the constraint information provided by the informational atoms. Suppose that there are  $M$  informational atoms in the query. Their head embeddings, relation embeddings, and tail embeddings are represented as  $c_h^{(m)}, c_r^{(m)}$ , and  $c_t^{(m)}$  respectively. For each operator output  $q_i$  from the computational graph, we compute its relevance score  $s_{i,m}$  towards each head eventuality  $m$ ,
$$
s _ {i, m} = <   q _ {i}, c _ {h} ^ {(m)} >. \tag {5}
$$
Then we use the  $s_{i,m}$  to access the values from the constraint relation and tails, and then aggregate the memory values according to the relevance scores
$$
v _ {i} = \sum_ {m = 1} ^ {M} s _ {i, m} \left(c _ {r} ^ {(m)} + c _ {t} ^ {(m)}\right). \tag {6}
$$
Table 2: The dataset details for CEQA. #Ans. reports the number of answers that are proved to be not contradictory by theorem provers. #Contr. Ans. reports the number of answers that can be searched from the ground truth KG, but are contradictory due to the occurrence or temporal constraints.
Data Split	#Types	OccurrenceConstarints	Temporal Constraints
#Queries	#Ans.	#Contr. Ans.	#Queries	#Ans.	# Contr. Ans.
Train	6	124,766	5.02	1.53	35,962	5.02	1.15
Validation	15	30,272	7.68	1.75	23,905	9.17	1.44
Test	15	30,243	8.40	1.81	24,226	11.40	1.50
Finally, as shown in Figure 4, the constraint values are added back to the query embedding after going through a feed-forward layer FFN, and this process is described by