Complex Query Answering on Eventuality Knowledge Graph with Implicit Logical Constraints
Authors: Jiaxin Bai
Xin Liu
Weiqi Wang
Chen Luo
Yangqiu Song
Venue: NeurIPS 2023 poster (2023)
Keywords: Knowledge Graph
Complex Query Answering
Eventuality Graph

Abstract: Querying knowledge graphs (KGs) using deep learning approaches can naturally leverage the reasoning and generalization ability to learn to infer better answers. Traditional neural complex query answering (CQA) approaches mostly work on entity-centric KGs. However, in the real world, we also need to make logical inferences about events, states, and activities (i.e., eventualities or situations) to push learning systems from System I to System II, as proposed by Yoshua Bengio. Querying logically from an EVentuality-centric KG (EVKG) can naturally provide references to such kind of intuitive and logical inference. Thus, in this paper, we propose a new framework to leverage neural methods to answer complex logical queries based on an EVKG, which can satisfy not only traditional first-order logic constraints but also implicit logical constraints over eventualities concerning their occurrences and orders. For instance, if we know that *Food is bad* happens before *PersonX adds soy sauce*, then *PersonX adds soy sauce* is unlikely to be the cause of *Food is bad* due to implicit temporal constraint. To facilitate consistent reasoning on EVKGs, we propose Complex Eventuality Query Answering (CEQA), a more rigorous definition of CQA that considers the implicit logical constraints governing the temporal order and occurrence of eventualities. In this manner, we propose to leverage theorem provers for constructing benchmark datasets to ensure the answers satisfy implicit logical constraints. We also propose a Memory-Enhanced Query Encoding (MEQE) approach to significantly improve the performance of state-of-the-art neural query encoders on the CEQA task.

Complex Query Answering on Eventuality Knowledge Graph with Implicit Logical Constraints
Jiaxin Bai Department of CSE HKUST
Xin Liu Department of CSE HKUST
Weiqi Wang Department of CSE HKUST wwangbwucse ust hk
j Obai@connect.oom cettmnecttutttttt ust.hk
xliucr@cse.c ust. hk
Chen Luo Amazon.com Inc
Yangqiu Song* Department of CSE HKUST yqsong@cse. ust. hk
cheluo@amazon.co com
Abstract
Querying knowledge graphs (KGs) using deep learning approaches can naturally leverage the reasoning and generalization ability learn infer better answers. Traditional neural complex query answering (CQA) approaches mostly work on entity-centric KGs. However, in the real world, also need make logical inferences about events, states, and activities (i.e., eventualities situations) to push learning systems from System System II, proposed by Yoshua Bengio. Querying logically from an EVVenuulity-centtrr KG (EVKG) can naturally provide references such kind intuitive and logical inference. Thus, this paper, we propose a new framework to leverage neural methods to answer complex logical queries based on an EVKG, which can satisfy not only traditional first-order logic constraints but also implicit logical constraints over eventualities concerning their occurrences and orders. For instance, if we know that Food is bad happens before PersonX adds soy sauce, then PersonX adds soy sauce is unlikely to be the cause of Food is bad due to implicit temporal constraint. To facilitate consistent reasoning EVKGs, propose Complex Eventuality Query Answering (CEQA), rigorous definition CQA that considers the implicit logical constraints governing the temporal order and occurrence of eventualities. In this manner, propose to leverage theorem provers for constructing benchmark datasets to ensure the answers satisfy implicit logical constraints. We also propose Memory- Query Encoding (MEQE) approach to significantly improve the performance of state-of-the-ar neural query encoders on the CEQA task.
Introduction
Querying knowledge graphs (KGs) can support many real applications, such fact-checking and question-an question-answering. Using deep learning methods to answer logical queries over KGs can naturally leverage the inductive reasoning and generalization ability of learning methods to overcome the sparsity and incompleteness of existing KGs, and thus has attracted much attention recently, which are usually referred as Complex Query Answering (CQA) (37 26 38 As the computational complexity of answering complex logical queries increases exponentially with the length of the query 26), brute force search and sub- uub-graph matching algorithms 29 are unsuitable
Prof. Yangqiu Song is visiting academic scholar at Amazon.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Queries
Type
Interpretations
41 V, Interact(V,, VV
Entity
Find the substances that interact with the proteins
Assac(V, MadCow)
associated with Alzheimer's and Mad cow discase.
fi V,, Precedence(Faod bod,PersonX add soy sauce)
Eventuality
Food is bad before PersonX add soy sauce. What the for food being bad?
Reason bod, V,)
fl V,. Precadencn(V,, Personk go home)
Eventuality
Instead of buying an umbrellaa, PersonX home.
Chouen@termativee(Prran go Home,Persond buy on umbrello)
What happened before PersonX go home?
Figure I: Complex query examples and corresponding interpretations in natural language. 91 is a query on an entity knowledge graph, while 92 and @3 are queries on an eventuality knowledge graph.
for processing complex queries. To overcome these challenges, various techniques, such as query encoding [23] and query dccomposition [2]. have been proposed. These techniques enable effective and scalable reasoning on incomplete KGs and facilitate the processing of complex queries.
Most of the existing work in this field has primarily focused on entity-centric KGs that only describe entities and their relationships. As Yoshua Bengio described in his viewNof of moving from System I to System [16 [8 25 [3], we need equip machine learning systems with logical, sequential reasoning, and other abilities. Particularly, such system requires the understanding of how actions (including events, activities, or processes) interact with changes in distribution which can be reflected by states. Here we can summarize events, activities, and states as a linguistic term, eventualities (or situations), according to the linguistics literature 33 A As with many other KG querying tasks, querying eventuality- references for making logical and rational decisions of intuitive inferences or eventual planning. This requires the CQA models to perform reasoning at the eventuality level. To provide resources for achieving eventuality-level reasoning, recently constructed KGs, such as ATOMIC 41 24 Knowlywood [43), and ASER (52 53), tend to use or more discourse relations to represent the relationships between eventuality instances. For example, PersonX went to the store and PersonX bought some milk are two simple eventuality instances, with the latter being possible consequence of the former. The construction of these EVentuality-centric Knowledge Graphs (EVKGs) thoroughly reason about eventuality instances
knowledge graphs can also support many applications, such as providing
maps the relationships between eventualities and enables and their relationships using logical queries, thereby facilitating a more comprehensive approach to modeling complex relationships than traditional knowledge graphs
Aside from the importance of querying EVKGs, reasoning on EVKG also significantly dif- fers from that on entity-centric KG because eventualities involve considering their occur- rences and order. In entity-centric KGs, as shown in Figure [ q1. the vertices represent en- tities such Alzheimer or Mad Cow Disease, and truth values assigned to the edges between entities to indicate their relationships. For example, the statement Assoc amyloid, Alzheimer) is true. In contrast, during the reasoning process EVKG, the even- tualities may or may not occur, and determining their occurrence is a crucial part of the rea- soning. For instance, given ChosenAlternativell (PersonX go home, Per buy umbrella) in Figure [ q2. it implicitly suggests that "PersonX go home" occurs, while "PersonX buy umbrella" does not. Moreover, there are relationships that explicitly or implicitly describe the order of occurrences, such as temporal and causal relations. For example, (Prrsonn study hard, PersonX pass ezam) indicates the causality between "PersonX pass the exam" and "PersonX study hard," which also implies that "PersonX pass the exam" occurs after "PersonX study hard." When multiple edges are presented in given situation, tial to ensure that there are no contradictions regarding the occurrence of these eventualities. For PersonxX X go home, Person X buy umbrella) A
essen-
example, in Figure 1 f3. ProosenAleerattie Succession (PersonXX X go home, Person.X buy umbrella) is contradictory because the former suggests that PersonX did not buy an umbrella, while the latter implies otherwise.
To enable complex reasoning on eventuality knowledge graphs, we formally define the problem of complex eventuality query answering (CEQA). CEQA is a more rigorous definition of CQA on EVKG that consider not only the explicitly given relational constraints, but also the implicit logical constraints on the occurrences and temporal order of eventualities. The implicit constraints are derived from the relational constraints and can be further divided into two types: occurrence constraints and temporal constraints. Incorporating these implicit constraints into complex query answers drastically
http:://ww.wr htw.://wwaaeeeeeee/////errr//o/a htt.:aaaaabbbooaa///e//aeee Satpg//oooooAAA---t00000000.00
2

---

[A]Query:
g= Precedeeccc, Personk go home) ^ Chosenalternativeeeerssss go Persoe, buy an umbreilo )
Constraints:
((V))
< <PPersonX home) A (VVV) (Persou go home) A (PersouX go home) A [y( Persont buy an umbreila)
Temporal:
r(V) x [PPorsonX go home)
Occurrence:
n(V)) A ( go home) A - (Person buy on umbrella)
(8) Query:
V,, P Precedence(Food is bad, PersonX odds soy sauce) A Reason[Food is bad, V.).
[Food is bad) A ((PersooXX odds soy souce) || is bodi A ( A (n(VV) - n(Food bad))
Constraints:
A t(Food bod) x r|PersonX adis soy souce) A rFood is bad) > x(V.)
Temporal:
r/Food bad] < rPerssnnX adds soy sauce) A I(Food bod) > x(V))
Ocourrence:
i/Food bad) A ipPersont adds soy sauce) A.j(Food is bad) A (VV)) A (n(V)) - ((Food bad))
Figure 2: Complex eventuality queries with their implicit temporal and occurrence constraints
changes the nature of the reasoning process. Unlike conventional CQA, the reasoning process of CEQA is defeasible because when additional knowledge is presented. the original reasoning could be weakened and overturned [1]]. For example, we showed in Figure [ PersonX adds soy sance is a possible answer to the query What is the reason for food being bad. However, if more knowledge is given, like Food is bad is before PersonX adds soy sauce, then cannot be the proper reason anymore due to temporal constraints. However, all the existing methods for CQA cannot incorporate additional knowledge conduct defeasible reasoning in CEQA.
To address this problem, we propose the method of memory-enhanced query encoding (MEQE). In the MEQE method, we first separate the logic terms in query into two categories, computational atomics and informational atomics. Computational atomics, like Reason(Foodd is bad, Vz),, contains at least one variable in their arguments, and informational atom- ics, like Precedence( Fooo is bad, Person, add soy sauce). does not contain variables. For the computational atomics. following previous work, we construct the corresponding computational graph to recursively compute its query embedding step-by-step. For the informational atomics, we put them into a key-value memory module. For each of the informational atomics, its head argument is used as the memory key, and its relation type and tail arguments are used as memory values. In the query encoding process, after each operation in the computational graph, relevance score is computed between the query embedding and memory heads. This relevance score is then used to retrieve the corresponding memory values of the corresponding relation and tail. Then these memory values are aggregated, adjusted, and added back to the query embedding. By doing this, the query encoder is able to leverage implicit logical constraints that are given by the informational atomics. We evaluate proposed MEQE method on the eventuality knowledge graph, ASER, which involves fourteens types of discourse relations between eventualities. Experiment results show that our proposed MEQE is able to consistently improve the performance of four frequently used neural query encoders on the task of CEQA. Code and data are publicly available
2 Problem Definition
In this section, we first introduce the definitions of the complex queries on entity-centric and eventuality-centrie KGs. Then we give the definition of implicit logical constraints and the informa- tional atomics that specifically provide such constraints to the eventuality queries.
2.1 Complex Queries
Complex query answering is conducted on KG: = (V,, R). The V is the set of vertices v, and the R is the set of relation r. The relations are defined in functional forms to describe the logical expressions better. Each relation is defined as function with two arguments representing two vertices, e and v'. The value of function r(v, v') = 1 if and only if there is a relation between the vertices e and v'.
In this paper, the queries are defined in conjunctive forms. In such a query, there are logical operations such as existential quantifiers 3 and conjunctions A. Mcanwhile, there are anchor eventualities V. E V, existential quantified variables V1, V2, V/g E V, and a target variable Va E V. The query
3

---

Table 1: The discourse relations and their implicit logical constraints. 77V) is True if and only if V occurs. r(V) indicates the happening timestamp of V. Meanwhile, the instance-based temporal logic
operator %, means V1 is before, after, at the same time as V2.
Discourse Xelations (ex)
Semuntics
Implicit Constraints Occunence Constraints (o,)
Temoral Constraints (42)
Precedence 11, Vg)
Vi occurs before Va.
m(V) A (V2)
r(V)) < r(V2)
Succession V1 Vg)
1 occurs afte V/2 happezs.
((V1) A V2)
T(V) +(12)
Syachronoual (V. Vg)
2(V1) v(V2)
<(V)) x(1))
Beason( V1, V2)
1/1 oecurs becous V2.
2(V)) A ((())) (n(1) =(V2))
r(V) r(Vs]
Result(Vs, 1/2)
result If Va occurs. V1.
m(V,) ((12) (n((1) =(V2))
r(V1) +(V2)
Condstion V1, Vg)
m(V) - n(V2)
T(V)) +(V))
Coscanston[11 V2))
V/2 occurs although V1. 1/2 occurs. but V1-
A
Constrast(Vi, V2)
A((1))
Cosjunctioa[V V2)
V and V2 both octur occur.
Irstantination([.
is mire detiiled deseription of V1
o(V1) 4(12))
Restaatesent
V22-
9(K1) +33)
Alternative(ii 12)
Vi V/ are alternative situatins..
m(V1) v/12)
#oosnd1sarrativve V22
Instedd of V/ accurs V1.
9(V1) -m(
Reception(V] V2)
encque Vg.
7N(K1 A A
(2) (-1(1)) (((1))
is written to find the answers V22 E V, such that there exist V1, V2,,.. E satisfying the logical expression:
q [V[] = V3.3%,, V22 = = Ae2 em-
Each & is an atomic expression in any of the following forms: e; = r(v, r(,, Here Va is an anchor eventuality, and V, V.V" E (V, V2.... VR, V2} are distinct variables.
2.2 Complex Eventuality Queries
For complex eventuality queries, they can also be written in the form of a conjunctive logical expression Eq. (1) Differently, each atomic e can all be in the form of e =r(,, ey), where u/ E V are given eventualities. These atomics, which do not include variables, are called informational atomics, because they only provide implicit constraints
The relations in CEQA are discourse relations, and they exert implicit constraints over the eventual- ities, and these constraints be categorized into occurrence constraints and temporal constraints. Suppose the occurrence and temporal constraints derived from the i-th atomic e; is denoted as o, and t,. Then complex eventuality query, including its implicit constraints can be written
[/>]] V ..., V& (e1
cm A (or A... A am) A (t1 tm).
(2)
The constraints brought from each type of discourse relations are presented in Table Further justifications of the derivation process are given in the Appendix B
[
2.2.1 Occurrence Constraints
The occurrence constraints determine whether certain eventuality occurs or not. For instance, consider Figure 2 (A), where the logical query means that Instead of buying an umbrella, PersonX goes home. Whut occurred before PersonX went home? If we rely solely on relational constraints, conventional definition of CQA, answers are only determined by the latter part of the query, What happened before PersonX went home? Consequently, PersonX buys umbrella could be solution to this query. However, within the query, there is an information atomic saying, instead of buying umbrella, PersonX goes home, which denies the occurrence of PersonX buying umbrella. To formally express such constraint, we use the function vV)). If eventuality V occurs, then 17( VV- True, otherwise it is False. As depicted in Figure 2 the occurrence constraint of this query comprises the terms (V V3) A (Persso bay umbrella). In this case, Vy cannot be PersonX buys unbrella or there is contradiction.
in the
Most discourse relations assume the occurrence of the argument eventualities, for example, Precedence, Conjunction, and Reason. However, there are also relations that do not imply the occurrence of the arguments, such as Condi tion and Restatement. Moreover, the Exception and chosenAlternatiin relations restrict certain eventualities from happening. For instance, in the

---

q v, Successioni/erseeee complolns, V) Susccesssion///rssnn leaves restaurant V) Reason(V, V
PrecedencelFoode is bod, PersonY adds souce) A Chosen/Aterrnttive//eeaas adds ketchup, PersonY adds vinegor) Eventuality Knowledge Graph:
Computational Atomics:
Succession
Reason
Succession
PersonX complains
Service
itaffis
complains Succession
PersonX
Reason
is bad
Reason
ChosenAl ternative Live
V
V
Persont addo
Reason ketchup
Persoux leaves
Answer: V, (Stalf new, PersonY adds ketchup,
Succession
restaurant
PersonY adds adds soy sauce, PersonY adds vinegar).
Precedence
Informational Atomics:
PersonX leaves
Food is bad
Reason son
PersonY Y adds
Precedence
ChosenAlternativee
restaurant
say sauce
Foodis
PersonY adds soy sauce
PersonY Y adds
PersonY adds vinegar
Person Y adds
bad
ketchup
vinegar
Answer: V,E (Stall is PersunY adds ketchup)
An example complex eventuality query with the computational and informational atomics. V is something that happens before person complains and leaves the restaurant, according to the KG, the V could be either Service bad or Food bad. V3 the reason of V. then according to the graph, V/2 could be either Staff is new, PersonY adds ketchup, PersonY adds soy sauce, and the query we also know that PersonY adds vinegar does not
Figure 3:
PersonY adds vinegar. However, happen, and PersonY adds soy sauce happens after the Food is bud, thus cannot be the reason for Food is bud. The conflict here is causality implies precedence.
case of Chosennlreernaaive@PPPssso read books, Person play games), it implies that Per- sonX reads books: n(PersonX read books), and does not play games: "n(Person play games). Another example is Exception( Room is empty, PersonX stay in room), which implies that the room is not empty and PersonX is present in the room. Furthermore, the room, then the room is empty. This can be formally expressed -my Room és empty) A (PPesson stay in room) A ( (-m((er PPrrsnoX stay in room) + Roooo is empby))) For com- prehensive overview of the occurrence constraints, please refer Table [
PersonX is not in
2.2.2 Temporal Constraints
The temporal constraints reflect the order of occurrence of the eventualities. As shown in Figure 2 (B), the complex query on the eventuality knowledge graph can be interpreted as Food is bad before PersonX adds soy sauce. What is the reason, for food being bad? If we only considered the relational constraints, like in the conventional setting of CQA, then PersonX adds soy sauce is a possible answer. However, in the definition of CEQA, the answer PersonX adds soy sauce is incorrect because the food is bad already occurred before PersonX added soy sauce, but something that occurs later is impossible to be the reason for something that previously occurred. Formally, we use the expression temporal logic >, <, and = to describe the temporal order between two eventualities 22.. 7 rA) < r(B) means A occurs before B, and 7(A) r(B) means they happen at the same time, and T T(A) > T(B) means occurs after B. For example in Figure (B), the temporal constraint is represented by r(Food is bad) < 7 [[PersonX add soy sauce) FFFood is bad) TV%), which can be interpreted as Food is bad is before PersonX adds soy sauce and V/ is before Food is bad. Because of this, V/2 cannot PersonX adds soy sauce, otherwise there exists contradiction.
The temporal relations Precedence( A, B), Succession(A. (A, B), and Synchronous(( (A, B) naturally describes the temporal constraint. Meanwhile, previous studies also assume that causation implies precedence [40] J0 [44,, With this assumption, the temporal constraints can also be derived from relations like Reason and Result. The descriptions of temporal constraints are given in Table [
3 Memory-Enhanced Query Encoding
In this section, we first introduce the method of query encoding, and then introduce how to use the memory module to represent the informational atomics conduct reasoning EVKGs.
5

---

where the qi and q++1 are imput and output query embeddings for this relational projection operation. Meanwhile, for the Intersection operations, suppose there are i embeddings of sub-queries, Computational Graph
Constraint Memory
Reason
Key
Value
Intersection
Foudly Buel
PrRcnsnnce
Permon mids sey
Succession
Succession
thesenkiton
O Persank compleins
Personi leares restaurant
Figure 4: The example computational graph and the memory- enhanced query encoding process.
3.1 Computational Graph and Query Encoding
Figure 3 3nd show show that there is computational graph for each query. This computational graph is directed acyclic graph (DAG) that consists of nodes and edges representing intermediate encoding states and neural operations, respectively. By recursively encoding the sub-queries following the computational graph, the operations implicitly model the set operations of the intermediate query results. The set operations are defined follows: (1) Relational Projection: Given set of vertices A and relation r E R, the relational projection operation returns all eventualities that hold the relation with at least one entity E A. This can be expressed as: (,(AA {r V 3v' A, r,',,, v) 1}; (2) Intersection: Given sets of eventualities A1.... An S V. the intersection computes the set that is the subset to all of the sets AI..... A,, This can be expressed n A4.
Various query encoding methods are proposed to recursively encode the computational graph. How- ever, the query embeddings of these methods can be translated into d-dimensional vectors. As shown in Figure 4 the computations along the computation graph start with the anchor eventualities, such as PersonX complains. Suppose the embedding of anchor is denoted as e, E Rd. Then, the initial query embedding is computed as go = cv. As for the relational projection operation, suppose the erel Rd is the embedding vector of the relation rel. The relation projection Por is expressed as
gi+1 Frroj (4: Eret),
(3)
... (2) (k) as the input for this operation, then the output can be expressed as:
ai+1 Finter (q 2 .. .....
where the Fitee is neural network that is permutation-invarian to the input sub-query embeddings adopted from the backbone models 23
12
3.2 Mehare-Enhanced Query Encoding
The computational graph is capable encoding computational atomics presented in the logical expression. However, informational atomics can influence the reasoning outcomes by introducing implicit temporal or occurrence constraints. As depicted in Figure 3 the absence of informational atomics results in two false answers from the knowledge graph. When informational atomies included, providing implicit constraints, the only two correct answers can be derived.
Based on this observation, we propose using a memory module to encode the constraint information provided by the informational atomics. Suppose that there are M informational atomics in the query.
Their head embeddings, relation embeddings, and tail embeddings are represented as cj cr
(m) (m)
and of (m) respectively. For each operator output qi from the computational graph, we compute its relevance score 8i towards each head eventuality
$1,m =< q:: ch
(5)
Then the st.m to access the values from the constraint relation and tails, and then aggregate the memory values according to the relevance scores
M
81 (m) + cg (11)).
01
(6)

---

Table 2: The dataset details for CEQA, #Ans. reports the number of answers that are proved to be not contradictory by theorem provers. #Contr. Ans. reports the number of answers that can be searched from the ground truth KG, but are contradictory due to the occurrence or temporal constraints.
Occureenceonnstarints
Temporal Constraints #Ans.
Data Split
#Types
#Queries
#Ans.
#Contr. Ans.
#Queries
Contr. Ans.
Train
124.766
5.02
1.53
35,962
5.02
1.15
Validation
15
30,272
7.68
1.75
23,905
9.17
1.44
Test
15
30,243
8.40
1.81
24,226
11.40
1.50
Finally, as shown in Figure 4 the constraint values are added back to the query embedding after going through a feed-forward layer FFN, and this process is described by
= q: + FF)(::).
3.3 Learning Memory-Enhanced Query Encoding
To train the model, compute the normalized probability of being the correct answer to query by applying the sof tmax function to all similarity scores:
p(q. v)
(8)
E (
where <,, >> denotes the dot product of two vectors, when is the query embedding after the last operation. cross-entropy loss used maximize the log probabilities of all correct answer pairs:
C
log Dgg((q ,00))
where (g(), v(0) denotes one of the positive query-answer pairs, and N is the total number of them.
4 Experiments
To ensure fair comparison of various methods for the CEQA problem, we generated a dataset by sampling from ASER (53), the largest eventuality knowledge graph, which encompasses fourteen types of discourse relations. The division of edges within each knowledge graph into training, validation, and testing sets was performed in an 8:1:1 ratio, as illustrated in Table [5] The training graph Gtrain. validation graph Gunt, and test graph Grest were constructed using the training edges, training+validaan edges, and lished configuration outlined in prior research by [37. Moreover, we conducted evaluations using different reasoning models, consistent with settings previous studies.
edges, respectively, following the estab-
4.1 Query Sampling with Theorem Prover
We employ the sampling algorithm proposed by 37 with the conjunctive query types outlined in [46] Specifically, for the training dataset, we sample queries that have a maximum of two anchor nodes, while for the validation and test sets, we select queries containing up three anchor eventualities. The query types in our framework reflect the structure the computational graph and are represented using Lisp-like format [46] 7 Once the query-answer pairs sampled, randomly select up to three edges that share common vertices with the reasoning chain of the query-answer pairs. These selected edges are then used the informational atomics for the corresponding query. Subsequently, we employ the z3 prover [15] to filter the queries. We retain only those queries where the informational atomies incorporate effective implicit constraints, ensuring the presence of meaningful constraints in the data. The detailed query types and their numbers of answer with/without contradictions are shown in Table 6 in which the p is for projection, the i is for intersection, and is for eventuality.
In detail, for each eventuality present on the reasoning path towards an answer the complex query, we create corresponding boolean variable in the z3 prover. We then incorporate the relevant
7

---

Table 3: Experiment results of different query encoding models. In this experiment, we compare the performance of the query encoder with or without the memory-enhanced query encoding method.
DecurreneConsstraints Hit@3
Temporal Constraints
Average Hit@3
Models
Hit@1
MRR
Hit@
Hit@3
MRR
Hit@1
MRR
GQE
8.92
14.21
13.09
9.09
14.03
12.94
9.12
14.12
13.02
+ MEQE
10.20
15.54
14.31
10.70
15.67
14.50
10.45
15.60
14.41
Q2P
14.14
19.97
18.84
14.48
19.69
18.68
14.31
19.83
18.76
+ MEQE
15.15
20.67
19.38
16.06
20.82
19.74
15.61
20.74
19.56
Nerual MLP
13.03
19.21
17.75
13.45
19.06
17.68
13.24
19.14
17.71
+ MEQE
15.26
20.69
19.32
15.91
20.63
19.47
15.58
20.66
19.40
FuzzQE 11.68
18.64
17.07
11.68
17.97
16.53
11.68
18.31
16.80
+ MEQE
14.76
21.12
19.45
15.31
21.01
19.49
15.03
21.06
19.47
occurrence constraints based the relations between these eventualities, as outlined in Table and feed them into the z.3 prover. If the result returndd by the prover is unsat, it indicates a contradiction in the reasoning process. Regarding temporal constraints, we follow similar approach. We create corresponding floating variables that represent the timestamps of the occurrence of the eventualities. We then establish constraints on the temporal order by utilizing floating operators such >, or < between the variables. By doing so, for each query, we establish a corresponding linear program. Once again, the prover outputs unsat, signifies contradiction, namely, there no solution for the timestamps of these events. Queries that have no contradictory answers and queries where all the answers are contradictory are discarded. The remaining queries are then categorized into two types: queries with occurrence constraints and queries with temporal constraints. Table presents the average number of contradictory and non-contradictory answers per query.
4.2 Baselines and Metrics
In this section, we introduce several baseline query encoding models that use different neural network architectures to parameterize the operators the computational graph and recursively encode the query into various embedding structures: (1) GQE (33 uses vectors to encode complex queries; (2)
Q2P uses multiple vectors to encode queries; (3) Neural MLP [ use MLP as the operators; (4)
FuzzQE uses fuzzy logic to represent logical operators.
To define the evaluation metrics, we use q to represent a testing query, and Gual and Gtest to represent the validation and testing knowledge graphs, respectively. We use [4] val and [g]eest to represent the answers query q on Gool and Grest, respectively. Eq. (10) shows how compute the metrics. When the evaluation metric is @it K, m(() is defined as m(() (r) = 1[r < K]. where m(r) if S K, and m(rr = otherwise. For mean reciprocal ranking (MRR), m(() is defined as m(r) +
Evelglena Maaeea///lle /gloar m(rank(v))
metric(g)
(10)
[g) Desst// [gleat]
During the training process, the testing graph Grest is unobserved. In the hyper-parameters selection process, we use the same metrics as Eq. (10), replace the graphs Grest
/ Gvat with Goat Sata/rrnn-
4.3 Details
To ensure fair comparisons, we replicate all the models under a unified framework. We use the same number embedding sizes of three hundred for all models and use grid-search to tune the hyperparameters of the learning rate ranging from (0.002, 0.001, 0.0005, 0.0002, 0.0001) and batch size ranging from {128, 256, 512). All the experiments can be run on NVIDIA RTX3090 GPUs. Experiments repeated three times, and the averaged results reported.

---

Table 4: The Hit@3 and MRR on different query types with various number of anchor nodes.
Query Type
Metric
GOE
MEQF
Base:
MRQQ
Neural MLP MEQF
Rose.
PuzzjE
MRQQ
16x03
13.76
13.74
15.93
15.23
18.02
MRR
11.86
12.75
16.90
18.35
15.31
16.51
14.38
16.58
34.48
44.65
43.71
MRR
30.53
32,80
39.79
34.77
35.02
35.16
36.92
36.53
Hinge
12.40
12.42
15.22
15.96
15.03
15.09
15.36
16.45
MRR
11.46
11.38
14.36
15.25
14.21
14.74
14.82
15.36
Hinons
14.16
14.87
17.49
19.86
17.06
19.07
16.59
18.65
MRR
13.16
13.19
16.48
18.89
15.49
18.27
14.69
17.22
Hiners
14.63
18,02
25.67
26.17
23.93
24.34
18.58
26.31
16.95
24.38
25.13
23.41
17.72
24.92
17.20
20.63
22.52
22.92
23.22
23.99
22.67
24.53
21.76
21.51
Hiness
24.66
28.11
45.10
48.62
47.14
47.56
MRR
22.57
24.22
40.14
36.70
40.95
1i:@3 MRR
17.06
11.81
12.38
17.00
16.44
16.86
17.42
15.88
17:24
22.06
19.70
MRR
15.62
17.59
20.76
21.60
20.45
22.19
17.52
21.70
16:03
16.23
19.75
24.45
25.59
23.39
25.45
22.33
25.63
MRR
15.05
18.36
23.30
24.15
21.60
24.26
20.87
24.00
Hiness
20.43
23.08
34.52
36.44
36.56
42.00
35.88
41.50
MRR
19.26
21.74
31.91
33.45
32.46
37.41
33.74
36,65
Hix
13.29
15.05
20.08
20.87
21.79
22.94
19:65
22.81
MRR
12.34
14,04
19.31
19.81
19:57
21.65
17.85
21.37
Hin0s
15.64
17.67
22.63
25.10
22.97
24.50
20.22
25.44
23.13
23.04
21.93
23.22
4.4 Experiment Results
Table presents the results of the main experiment, which compares different query encoding models with and without MEQE. The table includes the performance metrics of Hit@1 1, Hir@3, and MRR for both occurrence constraints and temporal constraints, along with the average scores across all categories. The experimental results demonstrate that our proposed memory-enhanced query encoding (MEQE) model consistently improves the performance of existing query encoders in complex eventuality query answering. We conduct experiments encoders, and the MEQE model, leveraging the memory model depicted in Figure 4 outperforms the baselines. The MEQE models differ structurally from the bascline models by incorporating memory module that contains informational atomics. By reading this memory module, MEQE effectively incorporates implicit constraints from these atomics, leading improved performance.
four commonly used query
Additionally, we observed that combining MEQE with the Q2P [5] model yields the best average performance across three metrics: Hit@| 1, Hit @ 3, and MRR. Furthermore, average, MEQE enhances the Hit@1 1 metric by 17.53% and the Hit@3 3 metric by 9.53%. The greater improvement in the Hit@1 metric suggests that the model's 's ability to accurately predict the top-ranked answer has improved more significantly compared to predicting answers within the top three rankings. Moreover, MEQE demonstrates 13.85% improvement in performance queries with temporal constraints and an 11.15% improvement on occurrence constraints. This indicates that MEQE is particularly effective handling temporal constraints compared occurrence constraints.
Table 4 displays the @it@3 and MRR results of various types of complex queries. The table demon- strates the superiority of MEQE over the baseline models across different query types. Furthermore, the table indicates that, on average, MEQE achieves an improvement 8. and 11.6% respectively. This suggests that MEQE is particularly adept handling queries with multiple eventualities.
5 Related Work
Complex query answering is a task in deductive knowledge graph reasoning, where system model required to answer a logical query on an incomplete knowledge graph. Query encoding [23] is fast and robust method for addressing complex query answering, Various query embedding methods

---

utilize different structures to encode logical KG queries, enabling them to handle different types of logical queries. The GQE method, introduced by Hamilton al. [23], represents queries as vector
representations to answer conjunctive queries. Ren et al. [37 employed hyper-rectangles to encode and answer existential positive first-order (EPFO) queries. Simultaneously. Sun al. [42] proposed the use of centroid-sketer representations to enhance the faithfulness of the query embedding method for EPFO queries. Both conjunctive queries and EPFO queries are subsets first-order logic (FOL) queries. The Beta Embedding (36] is the first query embedding method that supports a comprehensive set of operations Moreover, Zhang al. [55] utilized cone embeddings to encode FOL queries. Meanwhile, there are also neural-sye -ymralli methods for query encoding. Xu et al. [49 proposes an entangled neural- symbolic method, ENeSy, for query encoding. Wang al. 47 propose using pre- pre-trained knowledge graph embeddings and one-hop message passing to conduct complex query answering. Additionally, Yang et al. (50) propose using Gamma Embeddings to encode complex logical queries. Finally, Liu et al. (27) propose pre-training on the knowledge graph with kg- kg-transformer and then fine-tuning on the complex query answering. Recently, Bai et al. 77] proposes to use sequence encoders to encode the linearized computational graph of complex queries, Galkin et al. [20) propose to conduct inductive logical reasoning on KG, and Zhu al. (56) proposes GNN-QE
FOL by encoding entities and queries into probabilistic Beta distributions.
conduct reasoning on
KG with message passing on the knowledge graph. Meanwhile, Bai et al. [6] formulate the problem of numerical CQA and propose the corresponding query encoding method of NRN.
Another approach to addressing complex knowledge graph queries is query decomposition 2 In this research direction, the probabilities of these atomic queries modeled using link predictors, and then an inference time optimization is used to find the answers. In addition, an alternative to query encoding and query decomposition is proposed by Wang et al. [77. They employ message passing on one-hop atomic queries to perform complex query answering. A recent neural search based method called QTO is introduced by Bai et al. [8), which has shown impressive performance in complex question answering (CQA). Theorem proving is another deductive reasoning task applied to knowledge graphs. Neural theorem proving methods [39] 3ll 32 have been proposed to tackle the incompleteness of KGs by using embeddings to conduct inference on missing information.
Limitation
Although our experiments demonstrate that MEQE improves the performance of existing models on the CEQA task, the evaluation is conducted on specific benchmark datasets constructed with theorem provers from the largest general-domain eventuality graph ASER (53). The generalizability of the proposed approach to specific professional fields may require further investigation and evaluation.
Conclusion
In this paper, we introduced complex eventuality query answering (CEQA) as more rigorous definition of complex query answering (CQA) for eventuality knowledge graphs (EVKGs). We ad- dressed the issue of implicit logical constraints on the occurrence and temporal order of eventualities, which had not been adequately considered in the existing definition of CQA. To ensure consistent reasoning, we leveraged theorem provers to construct benchmark datasets that enforce implicit logical constraints on the answers. Furthermore, we proposed constraint with (MEQE) to enhance the performance of ofate-of-the-arr neural query encoders on the CEQA task. Our experiments showed that MEQE significantly improved the performance of existing models on the CEQA task. Overall, work provides more comprehensive and effective solution to the complex query-answentt problem on eventuality knowledge graphs.
query encoding
8 Acknowledgments
The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (1621 1520 and 16205322) from RGC of Hong Kong. We also thank the support from the UGC Research Matching Grants (RMGS20EG01-D. RMGS20CRIL, RMGS20CRI2. RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08).
10

---

References
[1] Alfonso Amayuelas, Shuai Zhang, Susie Xi Rao, and Ce Zhang. Neural methods for logi- cal reasoning over knowledge graphs. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL httips:/opeenreviett nww/forunffodddeeoooooooo tegeouUVVRt tgcAoUVHRIS
[2] Erik Arakelyan, Daniel Daza, Pasquale Minervini, and Michael Cochez. Complex query answering with neural link predictors. In 9th International Conference on Learning Repre- sentattonn. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
https://opearevieit wtw.://rrrriii...nnefrrr...m net/forumm - htte:/forumidddMoseeee Nosdod
[3] Nicholas Asher. Reference to abstract objects in discourse, volume 50. Springer Science Business Media, 1993.
[4] Emmon Bach. The algebra events. Linguistics and philosophy, 9(1::5-16, 1986.
[5] Jiaxin Bai, Zihao Wang, Hongming Zhang, and Yangqiu Song. Query2Particles: Knowledge graph reasoning with particle embeddings. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2703-2714 14, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/2000 htt.:/...n2i2n..nn.....
URL https
//aclanthology.oo org 2022. findings naacl. 207
[6] Jiaxin Bai, Chen Luo, Zheng Li, Qingyu Yin, Bing Yin, and Yangqiu Song. Knowledge graph reasoning over entities and numerical values. In Ambuj K. Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos. Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye, editors, Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pages 57-68. ACM, 2023. doi: 10. 145/ 3580305.3599399. URL hhttps://daat org/ 1ttg//0..0 100...115//88800 35915////3580355553339999999999 3599399
[7] Jiaxin Bai, Tianshi Zheng, and Yangqiu Song. Sequential query encoding for complex query answering on knowledge graphs. Transactions on Machine Learning Research, 2023. ISSN 2835-8856 URL
[8] Yushi Bai, Xin Lv, Juanzi Li, and Lei Hou. Answering complex logical queries knowledge graphs via query tree optimization. arXiv preprint arXiv:2212.099567 2022.
[9] Peru Bhardwaj, John D. Kelleher, Luca Costabello, and Declan O' O'Sullivan. Adversarial attacks on knowledge graph embeddings via instance attribution methods. In Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 8225-8239. Association for Computational I. Linguistics, 2021. doi: 10. 7/doi. org/ 18653//1//02/2 emnlp cennl-mmain 648
Moens,
http:////221.eepppmooee m44n.6... URL https
[10] Mario Bunge. Causality and modern science. 1979.
[11] Chunkit Chan, Jiayang Cheng. Weiqi Wang, Yuxin Jiang, Tianging Fang, Xin Liu, and Yangqiu Song. Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations. CoRR, abs/2304. 14827, 2023. doi: 10.48550/ARXIV22044 148277. URL org/10. 48550/ arXiv. 2304. 14827
https:://oi.
[12] Xuelu Chen, Ziniu Hu, and Yizhou Sun. Fuzzy logic based logical query answering knowledge graphs. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual
Event, February 22 March 2022, pages 3939-3948. AAAI Press, 2022. URL https
//ojs. aaai org/indez. httpp//AAA/tttttlleeie/e////////
[13] Brendan Conway-Smine -mith and Robert I. West. System-1 and system-2 realized within the common model of cognition. Proceedings http://ceur-ws. org ISSN, 1613:0073, 2022.
11

---

[14] Hanjun Dai. Hui Li, Tian Tian, Xin Huang, Lin Wang, Jun Zhu, and Le Song. Adversarial attack on graph structured data. Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML. 2018, Stockholmsmüsssor, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1 1123-1132. PMLR, 2018. htt.htttpppppcccednime wtt.esssmm.rprr.sseempemmm
mlr prwpse vttss/vv///aaa dai 18b. html
[15] Leonardo De Moura and Nikolaj Bjømerr Z3: An efficient smt solver. In Tools and Algorithms for the Construction and Analysis of Systems: 14th International Conference, TACAS 2008, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2008, Budapest, Hungary, March 29-April 6, 2008. Proceedings 14, pages 337 -400. Springer, 2008.
[16] Jonathan St BT Evans. Heuristic and analytic processes in reasoning. British Journal of Psychology, 75(4):451-468, 1984.
[17] Jonathan St BT Evans. In two minds: dual-procee -rooces accounts of reasoning. Trends cognitive sciences, 7(10):454-45, 2003.
[18] Jonathan St BT Evans. Dual-procsssing accounts of reasoning, judgment, and social cognition. Annu. Rev. Psychol. 59:255-22777 2008.
[19] Yu Feng, Ben Zhou, Haoyu Wang. Helen Jin, and Dan Roth. Generic temporal reasoning with differential analysis and explanation. CoRR, abs/2212. 10467, 2022. doi: 10.48550/arXiv2221... 10467. URL httphttps://o..... htt.://..rrrr11....55/////.1.11. 2212.
[20] Michael Galkin, Zhaocheng Zhu, Hongyu Ren, and Jian Tang. Inductive logical query answer- ing in knowledge graphs. In NewrlPS, 2022. http cwwpppr..cpppprrrccc files/
html
[21] Laura Giordano and Camilla Schwind. Conditional logic of actions and causation. Artif. Intell., 157(1-2):239-77, 2004. doi: (0.10166)..ttin22044444.000 URL https://doi... org/10
2004 04. 009
[22] Valentin Goranko and Antje Rumberg. Temporal Logic. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2022 edition, 2022.
[23] William L. Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, and Jure Leskovec. Em- bedding logical queries on knowledge graphs In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pages 2030 2018 URL #B50335cca1334004666663663e3 htt.:e/aaeeee00022----bttaaaaa.a wbstract..ttet neuriss.. wwwpariss.ccppppppeeeeee ccc/pappe//011//aaaeee
[24] Jena D. Hwang, Chandra Bhagavatula. Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. (comet-) atomic 2020: On symbolic and neural commonsense knowledge graphs. In Thirty-Fifih AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 6384 6392. AAAI Press, 2021. URL https:///jss aaai
org/ brgg/nndx.cr php/
[25]
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
[26] Lihui Liu, Boxin Du, Heng Ji, ChengXiang Zhai, and Hanghang Tong. logical queries on knowledge graphs. In Feida Zhu, Beng Chin Ooi, and Chunyan Miao, editors, KDD) '21: The 27th ACM SIGKDD Conference Mining, Virtual Event, Singapore, August 14-18, 2021, pages 1087-1097. ACM, 2021. doi: 0L114453444754.4446777 URL https: /oi. org /0. 1145 1145/3447548 347777447734443477775
Knowledge Discovery and Data
12

---

[27] Xiao Liu, Shiyu Zhao, Kai Su, Yukuo Cen, Jiezhong Qiu, Mengdi Zhang, Wei Wu, Yuxiao Dong. and Jie Tang. Mask and reason: Pre-training knowledge graph transformers for complex logical queries. In Aidong Zhang and Huzefa Rangwala, editors, KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14- 18, 2022, pages 11 1 130. ACM, 2022. doi: 10. 10.1145/33346883339477 URL https:/ : htt.//oo.oggg///o/oo//o org 10. 10411114//555
[28] Xin Liu and Yangqiu Song. Graph convolutional networks with dual message passing for subgraph isomorphism counting and matching. In Thirly-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intel- ligence, IAAI 2022, The Twelveth Symposium on Educational Advances Artificial Intelligence. EAAI 2022 Virtual Event, February 22 - March 1, 2022, pages 7594-7602. AAAI Press, 2022. doi: V111//AA7... URL wwwph:tpp///c / org/ 10 1609/ aaaii v3617 20725
[29] Xin Liu, Haojie Pan, Mutian He, Yangqiu Song, Xin Jiang, and Lifeng Shang. Neural subgraph isomorphism counting. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 1969-1969. ACM, 2020. doi: 10.1 14113494486..44033333
URL CRI.httss: //doi orgo/o/////......oo /1t1//1.1115/3394494444433333330
[30] Xin Liu, Jiayang Cheng, Yangqiu Song, and Xin Jiang. Boosting graph structure learning with dummy nodes. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 13704-13716 PMLR, 2022. URL httpss//ppooceedingg mll
gress/v162/1iu222 html
[31] Pasquale Minervini, Matko Bosnjak, Tim Rocktäschel, Sebastian Riedel, and Edward Grefen- stette. Differentiable reasoning on large knowledge bases and natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Inno- vative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 5182 5190. AAAI Press, 2020. URL https://ojs aaai org/index iindex pbp/AAAA//article/rieeeee
[32] Pasquale Minervini, Sehastian Riedel, Pontus Stenetorp, Edward Grefenstette, and Tim Rock- täschel. Learning reasoning strategies in end-to-end differentiable proving. In Pascal Hitzler and Md. Kamruzzaman Sarker, editors, Neuro-Symbolic Artificial Intelligence: The State of the Art, volume 342 of Frontiers in Artificial Intelligence and Applications, pages 280-293, IOS Press, 2021, doi: 10.3233/FA1A210355 URLhttps: org/10 wttg///.////////.. 32033//AA///A23035
[33] Alexander P. D. Mourelatos. Events, processes, and states. Linguistics and Philosophy, 415-434, 01 1978.
[34] Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco, May 2008. European Language Resources Association (ELRA). URL http:// www. wrw. conf org.:pprceednngs///r00000 htt.:///ddd/55__ppp/// pdf
[35] Rashmi Prasad, Nikhil Dinesh, Alan Lec, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L. Webber. The penn discourse treebank 2.0. In LREC, 2008
[36] Hongyu Ren and Jure Leskovec. Beta embeddings for multi- hopp--hopppo logical reasoning in knowledge graphs. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-
12, 2020, virtual, 2020. URL, httpp:t/////oooeenngggs hwwprsseurrip...p ceeurips.cc/ppperreee wwtp:/ppppppppp/eee/eeep Aww.aaat.tbtraat.ttttaaat
13

---

[37] Hongyu Ren, Weihua Hu, and Jure Leskovec. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. In 8th International Conference on Learning Represen- lations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreviex. wwtp:e/oooooodddddlgll.........
[38] Hongyu Ren, Mikhail Galkin, Michael Cochez, Zhaocheng Zhu, and Jure Leskovec. Neural graph reasoning: Complex logical query answering meets graph databases, 2023.
[39] Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Con- ference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3788- 3808, 2017. URL https::/pproceddnnggg wwwpeeiiinnnnnnuuiiie www.neurrppp.e www.eeee....ppppeeeeeeeeee
Awwrrstttsttttttateeeeeeeeeaeeet html
[40] Bertrand Russell. On the notion of cause. In Proceedings of the Aristotelian society, volume 13, pages 1-26. JSTOR, 1912.
[41] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Han- nah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. ATOMIC: an atlas of ma- chine commonsense for if-then reasoning. In The Thirty-Third AAAI Conference Ar- tificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial In- telligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, URL
2019, pages 3027-3035. AAAI Press, 2019. doi: 10. wtt.:/..........1.113000033 https: /tt:i//oiii.. org/10. 10. 1609/ aaai. 098101.33013002
[42] Haitian Sun, Andrew O. Amold, Tania Bedrax- Weiss, Fernando Pereira, and William W. Cohen. Faithful embeddings for knowledge base queries. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Aisuan-Tiet Lin, editors, Advances in Neural Informa- tion Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, neurips html
NeurlPS 2020, December 6-12, 2020, virtual, 2020. cc/ (aapeer/0020/aaaee http:///000000111111070000000000
[43] Niket Tandon, Gerard de Melo, Abir De, and Gerhard Weikum. Knowlywood: Mining ac- tivity knowledge from hollywood narratives. In James Bailey, Alistair Moffat, Charu C. Aggarwal, Maarten de Rijke, Ravi Kumar, Vanessa Murdock, Timos K. Sellis, and Jef- frey Xu Yu, editors, Proceedings of the 24th ACM International Conference on Informa- tion and Knowledge Management, CIKM 2015, Melbourne, VIC, Australia, October 19 doi: 10. 00.1145//80644162886688 URL https:
23, 2015, pages 223-232, ACM, 2015.
7/doi.org//1 111111111111/1 280688
[44] Weiqi Wang, Tianqing Fang, Wenxuan Ding, Baixuan Xu, Xin Liu, Yangqiu Song, and Antoine Bosselut. CAR: comceptuulizaaionnzuumnne augmented reasoner for zero-shot commonsense question answering. CoRR, (tbb/2335.148699 2023, doi: 1044850/ARXIV.2305... 14869. URL https
//doi 1ffoi.org//0..o 48..0.4855//488 arwiv. 2305. 14869
[45] Weiqi Wang. Tianqing Fang. Baixuan Xu, Chun Yi Louis Bo, Yangqiu Song, and Lei Chen. CAT: contextualized conceptualization and instantiation framework for commonsense reasoning. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), ACL 2023, Toronto. Canada, July 9- 14, 2023, pages 13111-131 Association for Computational Linguistics, 2023. doi:
1/20 18663/v1/2023 acl loll-ong... 733.
URL httpss/tt/////oorrr////
[46] Zihao Wang, Hang Yin, and Yangqiu Song. Benchmarking the combinatorial generalizabil- ity of complex query answering on knowledge graphs. In Joaquin Vanschoren and Sii-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL http:e/:////atteteeeeemcceeeeee proceeeiigsooooeongg...... neurips. wttpris....cppppeeee wwtppppee//22//aae/ecp fetbaaaa66g9ffaaaffffcoo22o26663 wwwp:esst----ttttttt-----ee-o html
14

---

[47] Zihao Wang. Yangqiu Song, Ginny Y. Wong, and Simon See. Logical message passing networks with one-hop inference on atomic formulas. CoRR, abs/2301.08859, 2023. doi: 10.485550/rXxw.2300 1..50000XXX..200088888., 08859. https//s:://do. org/10. 2.00..8550//r/// arrvv.. 2wt.riiiii.i.......m www.:3232180888 .88859
[48] Bonnie Lynn Webber and Aravind K. Joshi. Anchoring a Lexicalized Tree- Free-Adjoining Grammar for discourse. In Discourse Relations and Discourse Markers, 1998. URL
https://aclanthoolggs org/ww/// 2rgg/w8--00 W 0315
[49] Zezhong Xu, Wen Zhang, Peng Ye, Hui Chen, and Huajun Chen. Seural-symbolic entangled framework for complex query answering. CoRR, abs/2209 007799, 2022. doi: 10.48550/arXiv. 2209.08779. URL httpt://p///.o...o org/ 118550/arxiv> 2209 2009008779
[50] Dong Yang, Peijun Qing, Yang Li, Haonan Lu, and Xiaodong Lin. Gammae: Gamma embeddings for logical queries on knowledge graphs. CoRR. abs/2210.155778, 2022. doi: 10.485500arXx..221015577
|Rthttps://aol org/10. /t///10.88555 48550/ arXiv. 2210. 15578
[51] Xiaoyu You, Beina Sheng. Daizong Ding, Mi Zhang. Xudong Pan, Min Yang, and Fuli Feng. Mass: Model-agnostic, semantic and stealthy data poisoning attack on knowledge graph em- bedding. In Ying Ding, Jie Tang, Juan Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben, editors, Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023, pages https:::/// org/ 10. 1145/ 3543507 3583203
ACM, 2023. doi: 10.1 00.1145/3543507.358900
[52] Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-KK Ki Leung. ASER: A large-scale eventuality knowledge graph. In Yennun Huang, Irwin King, Tie-Yan Liu, and Maarten van Steen, editors, WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 201-211. ACM / IW3C2, 2020, doi: 10.1 10.1145/3366423333800000 URL https://doi.n / 1145/3366423,... 3380107
[53] Hongming Zhang, Xin Liu, Haojie Pan, Haowen Ke, Jiefu Ou, Tianqing Fang, and Yangqiu Song. ASER: towards large-scale commonsense knowledge acquisition via higher-order selectional preference over eventualities. Artif. Intell., 309: 103740, 2022. doi: 10. (0.1016/j.rrtit.22222 103740.
URL htt.:tttt:///o..coom org/ 1016/ artint......... 2022. 103740
[54] Jiayao Zhang, Hongming Zhang, Weijie J. Su, and Dan Roth. ROCK: causal inference prin- ciples for reasoning about commonsense causality. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning. ICML 2022. 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 26750-26771, PMLR, 2022. URL htttps://proceedingg/ mlr ppess/v16// 1atpa////2////haa2aam httll
[55] Zhanqiu Zhang, Jie Wang, Jiajun Chen, Shuiwang Ji, and Feng Wu. Cone: Cone em- beddings for multi-hop reasoning over knowledge graphs. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, edi- tors, Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 19172-19183, 2021. URL /tt.https///rreeednggs a0
neurips.cc. httpsscccpppp/eeeeeeee att.://a....c.....btat.t...... html
[56] Zhaocheng Zhu, Mikhail Galkin, Zuobai Zhang, and Jian Tang. Neural-symbolic models for logical queries on knowledge graphs In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference Ma- chine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 27454-27478. PMLR, 2022. URL mlr www..ppresssvvve1 /htpe/hh22cccc. html
[57] Daniel Zügnerr, Amir Akbarnejad, and Stephan Günnemann. Adversarial attacks on neural networks for graph data. In Sarit Kraus, editor, Proceedings the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 6256-6250. orcai.org, 2019. doi: 10. 10.24963/jjcaiiioo 20199/87 24963/1jcai 1ttpss//jcai.011/8777
URL //doi org/
15

---

A Broader Impact
This paper is the first work discussing how to conduct logical reasoning over knowledge graphs that describe events, states, and actions, known as eventualities. The proposed method, MEQE, is capable of effectively and efficiently answering logical queries over eventuality knowledge graphs.
The experiments were conducted on publicly available knowledge graphs, eliminating any data privacy concerns. However, one possible concern is that our proposed reasoning method is susceptible to adversarial attacks 14 [7 [
and data poisoning [5T] on knowledge graph reasoning systems, which may result in unintended outcomes for users.
B Logical Constraints from Discourse Relations
In this paper, we utilize discourse structures based the early work by Asher []. where discourse relations are considered as predicates that involve two abstract objects, such as events, states, and propositions [88.. We have adopted the discourse relation definitions from the Penn Discourse Tree- bank (PDTB) 34 which consist of four general classes: Temporal, Comparison, Cont ingennge and Expansi on. Each general class comprises various types, and the logical constraints are derived based on the semantic meaning of these discourse types.
The Temporal class is used when there is temporal relationship between the described situations in the arguments. It includes Precedence AA, B), Succession(A, B), and Synchronous (A, B). In Temporal relations, we employ the temporal logic expressions > <, and represent the temporal order between two eventualities [22] A < B denotes that A occurs before B, A = B implies that they happen simultaneously, and A > B indicates that A occurs after B.
The Contingency class used when one of the described situations in and B causally influences the other. It encompasses Reason, Result, and Condit fondition. Reason describes cause-and-effect relationship between two eventualities. We use the conditional operator > [21] to represent conditional and causal relations. Reason B, A), Result(A (A, B), and Conditio on B, A) can all implies A >>>>> B, indicating that A causes B [21]. Moreover, Reason and Result also imply they both occur.
The Comparison class depicts a discourse relation between A and B to to highlight significant differences between the two situations. Semantically, it indicates that the underlying values of A and B are independent of the connective (35]. Therefore, we simply use A A B to represent both sub-types of Contrast(A, B) and Consession(A, B), signifying that both eventualities indeed occur.
Expansion class describes those relations that expand the discourse and move its narratives or exposition forward. The Con junnjjjniioo/i A, B) is used to indicate new situations that provide new information in B that is related to the situation described in A. The logical formulation from the conjunction can be expressed as A A B. Mcanwhile, the instantiationAA (A, B) relation also requires both arguments
hold [35]. Thus it can also be described by the expression A B. Exception: inpption(/4 [A, B) indicates that B specifies an exception to the generalization specified by A. In other words, A is false because B is true, but if B were false, would be true. The semanties of an exception is expressed in -A A B A (-B + A). Restatement B) describes the relationship that the semantics of restates the semantics of A. So the A and B hold true the same time A <> B. Alternative(AA (A, B) relationship applies when two eventualities describe alternative situations. The semantics of Alternative (A,,, B) is A B. ChosenAlternativer A, B) means that two alternatives A and B are given, but the first one A is not chosen. Its semantic meaning is represented as (A V B) A AA
The Expansi on class encompasses relations that expand the discourse and advance its narratives or exposition 335. Conjuncti (A,, B) is used indicate new situations in B that provide related information to the situation described in A. The logical formulation from conjunction can be expressed as A A B. Similarly, Instantiation B) also requires both arguments hold (35) and can be described by the expression A A B. Exception(AA A, B) indicates that B specifies an exception to the generalization specified by A. In other words, A is false because false, A would be true. The semantics of an exception can be expressed as - A B A (-B + A). Restatement A, B) describes a relationship where the semanties of restates the semantics of Therefore, A and B hold true simultancously, represented as A <> B. Alternative(A.. B) applies
true, but if B were
16

---

Table 5: The basic information about the ASER-50K used for the experiments, and its standard training, validation, and testing edges separations.
Dataset
Relation Types
Entities
Training
Validation
Testing
All Edges
ASER5OK
14
54,557
113,608
13,860
13,784
141,252
Table 6: A breakdown of the detailed query types is provided in the training, validation, and testing sets, along with corresponding statistics. Specifically, we report the number of samples, the number of con-connrray answers, and the number of answers that satisfy the relational constraints but contradictory due to occurrence or temporal constraints.
Split
Type
Depths
Queries
Cleunnne
Cantr.
Queries
Temporal Constrints
Contr. Ans.
(p-te))
112
(p-(xte)))
2
21,010
6.03
2,09
1,876
6.36
1.36
Tm..
(p.6i.gp.e)).fpp.cc (pui))))
2
40.728
1.63
15,941
7.44
2
(i.pp.e))..p.ppxeee0)
2
1
18,088
3,048
4.93
1.78
1.07
1.38
1,940
81
4.10
1.60
1,10
1.00
16,009
(p-(e))
1,36
4.77
(p-(pile))
2,317
12.82
3.33
965
13.57
(p.dk/p..ell...//el)))
2
(i.pp.ee).pp.p/eell)
2
2,130
821
8.01
2.85
1.59
1.19
877
71
8.07
2.08
1.21
1.37
(x:p4pp.el1::.404400)
2
2.391
2.32
2.338
12:30
Val.
2
2,452
10.02
1.73
2,618
12.57
1.57
10.68
Bxc-ipuceh.p..e))xx.....))
2.30
1.522
7.64
1.52
977
8.39
1.43
Bxi.feccp.eelllpppppeee./.......
1,327
7.89
1.59
2,368
10.86
1,39
11.64
aad.ep.cehxppeeee.........e
1,068
2,329
7.73
1.51
2,399
10.73
1.42
2,335
9.30
2,07
2,568
12.19
1.45
[p-te)
1,091
4.83
1,38
50
6.78
1.18
(p-(pue1))
2,261
3.39
954
(p....p.eel.ppeeil)
2,425
11:77
2.20
2.434
17.13
1.95
(..p/e))ip.fe1))
8.pp.e)..p.pp.el0)
2
2,093
1.65
845
10.30
[xpp.p..e))..ppppde1))
2
2,402
10.89
2,30
2,315
15.11
1,53
Ts.
(p.b@kk.p.eehhppeeeh.....pp
2
2.386
11.26
1.81
2.648
15.95
1.77
2
2,368
9.07
1,02
2,470
13.00
1.47
bad.pppeehxpppppee)xp....ll)
2
2,782
9.13
1.72
2,420
12:97
1.40
2
2,346
10.09
1.93
2,607
13.89
1.64
kdd.ppdelxxp.eexxp.p...ee)>
1.63
2.423
2
2,321
10.13
2.17
2,558
14.66
1.57
when two eventualities describe alternative situations. The semantics of AlternativeAA, B) A B. ChosenAlternativer B) means that two alternatives, A and B, are given, but only the first one A is chosen. Its semantic meaning is represented as (A V B) -B
C Differences Between Commonsense Reasoning and Eventuality Reasoning
Our task is different from other QA or implicit reasoning tasks in several ways. Firstly, it has broader scope, encompassing various relationships, including non-common sense discourse relations found in Treebank 2.0, which is even challenging for large language models [[]]. This resource provides additional relations, which include four general types: temporal (before/after), contingency (because/result), comparison (but/although), and expansions (and//oreexeepiinsttaad) In contrast, common sense relations mainly focus on the first two types relations: contingency and temporal. The occurrence constraints discussed in this paper primarily exist in the expansion type, which does not appear in common sense KG but exists in the event KG. This makes our task more complex, and it cannot be effectively addressed using common question guestion-answering answering methods [44).
17

---

Moreover, our main focus is on complex query answering, where queries center around intricate relationships between eventualities. Unlike existing common sense knowledge graphs (CSKGs), which typically handle relations involving two events in a triple, our task involves multiple events within single query-answer pair. This presents a challenge in formulating our task as either a knowledge graph completion (KGC) or question-answering (QA) task, as such formulations would require discarding most query constraints, reducing complexity, and simplifying it into basic query answering task. While commonsense knowledge may play role in answering our queries, it is not as prevalent as other tasks 451 Additionally, our task does not heavily rely on the semantic information of the query itself; instead, it relies on learning graph structures to perform query answering and reasoning. We utilize the inherent structure of the graph rather than relying solely on natural language processing. Finally, there are several complex query-answering tasks that share similar settings with the one in our paper, such as the EFO-1 benchmarks [46].
D Knowledge Graph Details
The eventuality knowledge graph, ASER-50K, is derived from a sub-sample of ASER2. O ASER2.1 includes the Co-Occurrence relations, which indicate that two eventualities co-occur in two consec- utive sentences in the original text. However, in this paper, we exclude the co-occurrence relation to focus on discourse relations. To remove noise from ASER 2.1, we eliminate edges with low frequen- cies and retain only those with frequency higher than two. The ASER graph is constructed using an extractive method from natural language text, which may result in the inclusion of eventualities with high frequency but vague semantics, such as PersonX know and PersonX think. To address this issue, we remove the most frequent one hundred vertices and retain the remaining densest vertices. The resulting ASER-50K dataset comprises 54,557 eventualities and 141,252 edges. Subsequently, we randomly partition the edges into training, evaluation, and testing sets in an 8:1:1 ratio. The numbers of edges in each set are presented in Table
The query types in our framework reflect the structure of the computational graph and are represented using a Lisp-like format 46 77. For instance, the query (i, (p, (e)), (p, (e)) represents a query with two anchor eventualities, each having relational projection, and the answer eventualities are the intersection of these two projection results. Additionally, this query type is also referred to as 2i in related work [37 36 However, our naming approach is more flexible and can be extended to accommodate more complex query structures. We sample our queries based on the query types, limiting them to a maximum of three anchors and a maximum depth of two. Specifically, in the training set, we only sample queries with maximum of two anchors. Further details regarding the query types in the training, validation, and testing sets can found in Table
E
Query Sampling Algorithm
The query sampling algorithm employed in this study is based on the work by Ren et al. (37. We replicate the sampling algorithm and provide the pseudo-code for the sampling process in Algorithm I Our focus in this paper is on conjunctive logical queries derived from eventuality knowledge graphs. As result, the query sampling process involves only the operations of relational projections and intersections. Given knowledge graph G and query type T. we initiate the query generation process by starting with a random node v. The goal is to recursively construct a query that has v as its answer, following the structure specified by T. During each recursive step, we examine the last operation in the query. If the operation is a projection, we randomly select one of its predecessors that holds the corresponding relation to e, which will serve as the answer to the sub-query. The recursion is then applied to node u and the sub-query type of T. Similarly, for intersection, we recursively apply the process to their respective sub-queries on the same node v. The recursion continues until the current node contains an anchor entity, at which point the process terminates. This recursive approach allows us to systematically construct queries that satisfy the given query type T and have v as the desired answer.
18

---

Algorithm The algorithm used for sampling a complex query from a knowledge graph starting from random vertex e from the knowledge graph G with query structure T.
Require: G a knowledge graph.
function SAMPLEQUERY(T,
T is arbitrary node of the computation graph.
is an arbitrary knowledge graph vertex I' opperation p then
u + SAMPLE(( [u, u)is an edge in G})
ypl + type of (u, u) in G
ProjectionTType
SubQuery t- SAA-ALLLUUURRTTTCIII
return (Projection'''pp, ype, RelT SubQuery)
else if if Ioperratin then
I Intersection Result (i)
for T T.Children do
SubQuery + SAMPLEQUERY((Tchiid v)
IndersecttinnResulle hew.:oneeeeeut..ssSsccCcCCcCcaa
end for return Intersection Result else if I lpoopaaaiinn then
return (e, Tvalue)
end if end function
Table 7: Ablation Studies on Constraints and Feed-Forwart Network in MEQE.
Occurrence Constraints Hit@
Temporal Constraints
Average Hit@3
Models
Hit@3
MRR
Hit@
Hit@3
MRR
Hit@1
GQE
8.92
14.21
13.09
9.09
14.03
12.94
9.12
14.12
13.02
+ MEQE
10.20
15.54
14.31
10.70
15.67
14.50
10.45
15.60
14.41
MEQE Constraints
8.29
12.87
11.62
8.80
13.02
12.17
8.54
12.95
11.90
MEQR- FFN
0.67
1.17
1.13
0.74
1.23
1.12
0.70
1.19
1.08
Q2P
14.14
19.97
18.84
14.48
19.69
18.68
14.31
19.83
18.76
MEQE
15.15
20.67
19.38
16.06
20.82
19.74
15.61
20.74
19.56
MEQE Constraints
14.16
20.00
18.86
14.72
19.92
18.79
14.44
19.96
18.82
MEQE FFN
12.77
16.63
15.89
12.74
16.83
14.75
12.76
16.73
15.32
Nerual MLP
13.03
19.21
17.75
13.45
19.06
17.68
13.24
19.14
17.71
+ MEQE
15.26
20.69
19.32
15.91
20.63
19.47
15.58
20.66
19.40
MEQE Constraints
13.33
19.15
17.94
13.49
19.18
14.48
13.41
19.16
18.08
MEQE- FFN
10.35
14.67
13.71
10.94
14.67
12.74
10.64
14.67
14.53
FuzzQE
11.68
18.64
17.07
11.68
17.97
16.53
11.68
18.31
16.80
MEQE
14.76
21.12
19.45
15.31
21.01
19.49
15,03
21.06
19.47
MEQE Constraints
12.69
19.92
17.68
13.53
18.25
17.91
13.11
19.08
17.80
MEQE- -FN
9.81
15.26
14.46
10.17
15.37
14.87
9.99
15.31
14.66
Further Ablation Study on the memory module and FFN layer
To demonstrate the effectiveness of the relevance and the feed-forward module, conducted an ablation study on our proposed MEQE method, and the results are presented below.
When removed the added the relations and tails embedding to the query embedding. the performance was negatively impacted. This is because the query embedding more likely to have higher similarity to the answers that should be excluded. This effect was more significant in the GQE model, as the GQE model uses the simplest operation the relations projection.
network, shown in the rows of "MEQE FFN" and directly
We also conducted another ablation study by replacing the constraints with random triples so that there are no contradictory answers in the rows of "MEQQ--Costtraitts" We observed that the performance of the model comparable the baseline model. This indicates that the performance improvement gained from the constraints instead of the structural changes of the query encoder.
19

---

Logical Query:
q- V,, Successton/PProon compliin, / Someereennnneeeeeeeeee My ReasontV, V. Precedence/food bod, Personl adis say souce) Cnoseevalleemmeeeeeee//eooooo adus ketchup, Pezsonl vinegor)
(VV (Persnn
compleiss) (ooploaimmmmmnn)
A x(V) (ntoooooo
AT(V) < =(Persen) rertauzad) (VV) A A (/erson) (erres)es
Implicit Constraints:
Agll) An(VV) (H(V)) - 7000) A x(V) z(V)
AqlFood is bad) piPesson adds souce) 1iFood is bad) t|PersonY' adiss say soyce)
(Persoot
(Person adds vinegor)
Query Types:
(p, (1, (p, (p.)))
Personk
Succession
Intersection
Reason
Service bool Bad, ..]
Computational Groph:
Personk kaves restaurant
Succossion
(Service is bad,
new,
Persont adds ketchup.
(Peronn receives call. Service bad,
PerontE-able vinagees
Occurrence Constraints
Auxino adde say-
Temporal Constraints
ChosenAllernatiiie
Pracedance
Informational Atomics:
Rood is had
Person Y adds 509 SaUVe
ketchup
adds vinegar
Figure 5: The example provided showcases complex eventuality query along with its implicit constraints, query type, computational graph and atomics visualization.
These experiments prove two things. First, the relevance score is effective in finding the corresponding constraints. Second, the feed-forward layer is useful and necessary to adjust the direction of the memory contents to incorporate into the query embedding.
G Detailed Example of Complex Eventuality Query
Sgsurdde a detailed example of a complex eventuality query. This query corresponds to the query type (p, (i, (p, (e)) (p, (e)))), and its corresponding computational graph is depicted. The implicit constraints of the atomics in the logical query are derived according to the discourse relations. When the computational graph is executed on the eventuality knowledge graph, without considering the logical constraints, there would be four potential answers: Staff is new, PersonY adds ketchup, PersonY adds vinegar, and PersonY adds soy sauce.
However, the answer PersonY adds vinegur is contradictory due to occurrence constraints, as one of the informational atomics indicates that PersonY Y adds vinegar did not occur. Furthermore, the answer PersonY adds soy sauce is contradictory due to temporal constraints, as it occurs after Food is bad, indicating that PersonY adds soy sauce cannot be the reason for Food is bad.
20


Reviews:
[{"review_id": "CsTZKoVybu", "reviewer": "Reviewer_DAya", "summary": "The paper proposes a new task of complex event knowledge graph completion, curates the datasets from existing event knowledge graph, and designs a solution. However, the tasks motivation is not very clear regarding the formal logic form as each event is still in the form of natural language, especially considering the event chains are likely break down along with increasing reasoning paths. The authors need to further discuss the different and importance of the proposed task compared with commonsense reasoning or commonsense knowledge graph completion. \n\nWeaknesses\n\n1. The paper is hard to follow and needs further polish. For example,\n    1. line 53-60, what ‘s the difference between entity-centric and event centric KGC? The given example is not convincing and seems artificial. If the event does not occur, it may not be necessary to included in the KG. So, the difference lies on the Open-world assumption or closed world assumption, not entity- and event-centric.\n    2. What is the motivation to formulate the commonsense knowledge as the formal logic form? For commonsense knowledge, it has a great chance to hold true with probability. That is, along with the increasing paths, the reasoning chains are more likely false.\n2. The proposed task looks similar with commonsense reasoning. So, it is necessary to discuss and compare with the datasets and baseline methods for commonsense question answering or commonsense knowledge graph completion.\n3. What is the quality of the curated benchmark?", "questions": "see summary", "limitations": "see summary", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "see summary", "weaknesses": "see summary", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "|  | Commonsense KG |  | Eventuality KG |\n|---|---|---|---|\n| KG  | Atomic | ConceptNet | ASER |\n| Node | Event-Centric (mostly) | Entity-Centric (mostly) | Eventuality |\n| Edges | If A then B;  | Entity relations  | If A then B;  |\n|  | A because B; | … | A because B; |\n|  | A as a result B;  |  | A as a result B;  |\n|  | A before B;  |  | A before B;  |\n|  | A after B. |  | A after B. |\n|  |  |  | Narrative Relations:  |\n|  |  |  | **Although A, B;** |\n|  |  |  | **A but B;** |\n|  |  |  | **A and B;** |\n|  |  |  | **A for example B;** |\n|  |  |  | **A in other words B;** |\n|  |  |  | **A or B;** |\n|  |  |  | **Instead of A, B;** |\n|  |  |  | **A except B;** |\n\n\nFirst of all, we would like to clarify that **not all relations between events/eventualities are commonsense relations**.  As we have explained in the section 2 about  the scope of relations we adopt in this paper is discourse relations, which include four general types: **temporal** (before/after), **contingency** (because/result), **comparison** (but/although), and **expansions** (and/or/except/instead). The commonsense relations, on the other hand, mainly focus on the first two types of relations: **contingency** and **temporal**. The occurrence constraints discussed in this paper primarily exists in the **expansion** type which do not appear in commonsense KG but exists in the KG. \n\nBased on this clarification, we will address your concerns: \n\nRe W1.1:\n\nAs shown in line 33-34 and line 44-45, the major difference is that the vertices of entity-centric KG are entities, and the vertices of event-centric KGs are events. On simple way to distinguish events from entities is that, the events include verbs that that can be associated with ture/false values to indicate whether that events happens in a specific situation, but the entities are all nouns. \n\nMeanwhile, the open-world assumption means the missing edges in a knowledge graph are unknown instead of false. Regardless of what kind of KG is, this assumption is always adopted in almost every task for knowledge graph, like KG completion, complex query answering, and rule mining. \n\nRe W1.2\n\nYes, as you have said, commonsense knowledge is something that mostly happens in a situation. But the level of commonsense modeling is not down to the level of our modeling of events, in which we care about under a specific situation, the occurrence of events.  This is why we need logical reasoning in our task. \n\n\nRe: W2\n\nDifference between Commonsense Question Answering and Complex Eventuality Query Answering (CEQA). \n\nCommonsenseQA is a benchmark dataset for NLP models to test the ability to reason with commonsense knowledge, and it's a multiple choice problem. The commonsense QA problems do not necessarily include events and the relations between events. Here is an example:\n\n* Where can I stand on a river to see water falling without getting wet?\n* A. waterfall,  B. bridge,  C. valley, D. stream, E. bottom\n\n\nCommonsense KG completion is the task of given head and relation, predicting the tail node. The evaluation is a ranking task. \n\nFor ConceptNet:\n* Question: (bacteria, causes, V) \n* Answers: [tooth decay, infection in cut]\n\nFor Atomic:\n* Question: (X repels Y's attack, Xattribute, ?)\n* Answers: [X is strong, X is skilled, X is brave]\n\nOur task differs from traditional question answering tasks because our task encompasses a broad range of relationships, including non-common sense relations as seen in Treebank 2.0, which provides numerous additional relations. Some complex queries include relations that only exist at eventuality level, such as co-ocurence, conjunction, contradiction, cannot be effectively simulated by commonsense question answering. As a result, our focus lies in complex query answering, where queries primarily revolve around intricate connections between eventualities. Unlike existing common sense knowledge graphs (CSKGs) that can only replicate relations involving two events in a triple, our task involves far more than two events in a single query-answer pair. This poses a challenge in formulating our task as either a knowledge graph completion (KGC) or question-answering (QA) task, as such formulations would necessitate discarding most query constraints, diminishing complexity, and transforming it into a simpler query answering task. Although commonsense knowledge may come into play when answering our queries, it is not as prevalent as in other tasks. Instead of relying on human-level commonsense reasoning, our task also doesn’t rely on the semantic information of the query, but rather rely on learning graph structures to perform query answering and reasoning, utilizing the structure of the graph itself instead of natural language. Thus, we believe that our task is significantly different and methods solving those tasks cannot be simply mitigated. \n\nRe: W3 \n\nOur benchmark dataset is derived directly from ASER without any complex transformations. ASER has been demonstrated to possess exceptional eventuality and discourse extraction quality, with an accuracy of over 90%. This means that the extracted information is highly accurate and representative of the original semantic meanings. As a result, we are confident in the reliability of our benchmark's query-answer pairs.\n \nWe used the Amazon Mechanical Turk platform to conduct human annotations and assess the plausibility of each query-answer pair. The workers received thorough training, including detailed instructions and qualification rounds to ensure their accuracy level was above 90%. After annotating 1200 query-answer pairs, we calculated the statistics and found that 86.5% of the answers were considered plausible. These results align with our expectations and provide validation for our hypothesis.\n\nWe assure you that we will carefully consider these discussions and incorporate them into our paper's final version. Additionally, we plan to conduct case studies to further demonstrate the quality of our benchmark.\n\n\n\n\n\n"}, {"review_id": "8YeuM4o17I", "reviewer": "Reviewer_ktzN", "summary": "This paper proposes an approach to address the challenge of complex query answering on eventuality knowledge graphs by integrating implicit logical constraints. The authors introduce the task of complex eventuality query answering (CEQA), which requires considering the occurrence and temporal order of eventualities. Methodologically, the paper encodes the edges of the knowledge graph containing these constraints as key-value pairs, which are then integrated into the attention mechanism. The authors extracted eligible data from the ASER dataset and conducted experiments combining their proposed method with various query encoding models, demonstrating improved performance.", "questions": "- In Figure 3, please provide a brief explanation of the confliction of the \"Precedence\" relation in the two subfigures.\n- Could you provide a concise overview of the structure of the permutation-invariant neural network for the intersection operation in Section 3.1?\n- Consider reducing the number of method-independent first-order logic examples and representations in order to allocate more space for comprehensive experimental and methodological analyses of the defined categories in the paper.\n", "limitations": "I have no concerns about the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper's motivation is solid, aiming to incorporate logical information from eventualities into complex query answers.\n- The method serves as an effective additional information exploitation approach, adaptable to various query encoding models.\n- The constructed CEQA dataset can provide insights for related tasks.", "weaknesses": "- This paper dedicates excessive description to the introduction. The use of multiple representations of first-order logic may be misleading, as the actual encoding is relational.\n- The work assumes the existence of a knowledge graph and constructs datasets using theorem provers, limiting its generalizability to other tasks.\n- The paper introduces two types of constraints and presents contradictory answers in the data, but lacks analysis of these categories in the methods and experiments.  While Table 2 provides statistics on constraints and contradictory answers, conducting further experiments specifically targeting these categories would provide valuable insights into the functioning of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! I would like to address your concerns one by one.\n\nRe: W1\n\nWe clarify that the proposed problem is formally defined in logical form. Meanwhile, relational encoding is part of the query encoding method. It is possible that there will be other methods for this task that do not use relations encoding. \n\nMoreover, the definition in logical form is necessary, because we are using the logical expression as input to the theorem prover to verify the correctness of the answers. This is an important part of our problem definition.\n\n\nRe: W2\n\nIn this paper, our proposed method is a logical query answering method on KG, so we do need a knowledge graph as it is part of our problem definition.\n\nHowever, we argue that it is a novel and generalizable idea to use theorem prover for verifying the consistency of the relations among multiple events. This is because events can also be obtained from other sources, for example text, instead of sampling from a KG.  \n \nFor a specific example, we can use this idea to verify the logical consistency of the stories generated from language models. In doing so, we can first extract events from the stories, and use discourse parsers to create edges among the events. Then we can apply the theorem prover on this graph of events to verify whether there are contradictions on occurrence and the order of occurrence of events. \n\n\nRe:W3\n\nWe would like to clarify that we do indeed include the corresponding results for the two types of constraints and contradictory answers in Table 3. This table has nine columns, with the three columns on the left side showing the results with occurrence constraints, the three columns in the middle showing the results on temporal constraints. The last three columns show the averaged results.\n\n\n\nRe: Q1\n\nIn table 3, V is something that happens before a person complains and leaves the restaurant, according to the KG, the V could be either “Service is bad” or “Food is bad”. If V? is the reason of V, then according to the graph, V? could be either “Staff is new”, “PersonY adds ketchup”, “PersonY adds soy sauce”, and “PersonY adds vinegar”. \n\nHowever, in the query we also know that  “PersonY adds vinegar” does not happen, and “PersonY adds soy sauce” happens after the “Food is bad”, thus cannot be the reason for “Food is bad”. The conflict here is causality implies presidency, and an specific event cannot happen both before and after another event. \n\n\nRe: Q2\n\nThe relation projections and intersection operations are adopted from backbone models. For different backbone bone models,  we have different parametrization of Intersection/Union. But basically, they are all deep-set functions [1].\n\n* GQE [2] model uses a feed-forward layer, followed by an average pooling, and then followed by a matrix multiplication. \n* Query2Particles [3] use self-attentions for intersection modeling:\n* FuzzQE [4] uses element-wise fuzzy logic operations on embedding space to do intersections and unions. \n* Neural-MLP [5] uses MLP-Mixer as an intersection module. \n\nAll the baseline models have their unique designs, but they share one thing in common, their intersection operations are all invariant to the permutation. They are all special types of deep set functions [1]. We will include the formula of them to the appendix of this paper. \n \n\nRe: Q3\n\nThank you for your advice, we will move parts of the logical definition to the appendix, so that we can include the newly added experiment results to the paper.\n\n\nReference: \n\n[1] Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., & Smola, A. J. (2017). Deep sets. Advances in neural information processing systems, 30.\n\n[2] Hamilton, Will, et al. \"Embedding logical queries on knowledge graphs.\" Advances in neural information processing systems 31 (2018).\n\n[3] Jiaxin Bai, Zihao Wang, Hongming Zhang, and Yangqiu Song. 2022. Query2Particles: Knowledge Graph Reasoning with Particle Embeddings. In Findings of the Association for Computational Linguistics: NAACL 2022. Association for Computational Linguistics, Seattle, United States, 2703–2714.\n\n[4] Chen, Xuelu, Ziniu Hu, and Yizhou Sun. \"Fuzzy logic based logical query answering on knowledge graphs.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 4. 2022.\n\n[5] Alfonso Amayuelas, Shuai Zhang, Susie Xi Rao, and Ce Zhang. Neural methods for logi333 cal reasoning over knowledge graphs. In The Tenth International Conference on Learning 334 Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL 335 https://openreview.net/forum?id=tgcAoUVHRIB.\n"}, {"review_id": "zVXcUMJORu", "reviewer": "Reviewer_dKK5", "summary": "This work aims to conduct complex logical query task over eventuality-centric KG (EVKG) and propose the Complex Eventuality Query Answering (CEQA) setting that considers the implicit constraint of the temporal order and occurrence of eventualities. The authors also propose a memory-enhanced query encoding method and achieve state-of-the-art performance on the CEQA task.", "questions": "please refer to the weakness\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. very interesting and important research problem\n2. the proposed MEQE module enhances the state-of-the-art query encoder on the CEQA task\n3. the paper is overall well-organized and well-written", "weaknesses": "1. the novelty of memory-enhanced module is limited since the memory mechanism has been proposed in many prior works such as [1] \n2. Recently, path-based KG reasoning methods such as QE-GNN [2] have achieved much research progress and show stronger reasoning ability than embedding-based methods. However, the proposed MEQE seems cannot incorporate the path-based method that didn't learn the representation for each node and relation in KGs. \n\n[1] Rossi, E., Chamberlain, B., Frasca, F., Eynard, D., Monti, F., & Bronstein, M. (2020). Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637.\n[2] Zhu, Z., Galkin, M., Zhang, Z., & Tang, J. (2022, June). Neural-symbolic models for logical queries on knowledge graphs. In International Conference on Machine Learning (pp. 27454-27478). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Re W1:\n\nThank you for pointing out this reference, and we will cite the corresponding papers. Meanwhile, we would like to argue that, we are the first work to use memory modules in the problem of query encoding, and we are the first work to propose using memory modules to encode the logical constraints during the query encoding process. This is a novel idea worth noticing in the community of knowledge graph reasoning. \n\nRe W2:\n\nYes, we admit that the current design of our approach cannot be directly used together with GNN-QE. In the current research context, it is the only method that uses GNN over the underlying knowledge graph.\n\nHowever, GNN-QE obtained superior performance at a high computational cost. Here are the reasons:\n\n1. The projection operation of GNN-QE relies on GNN operations, and with time complexity $O(|V|d^2 + |E|d)$, where $|V|$ is the number of vertices, $|E|$ is the number of edges, and d is GNN hidden size. The inference time grows linearly with the size of KG. However, the projection operations of other QE methods are of $O(d^2)$, and they are independent of KG size. \n\n2. The query representation size of GNN-QE is $|V|$ instead of $d$ as previous methods. In previous work, their query embedding size is around 15,000 (the number of vertices),  while other QE methods have the query embedding size at around 300-400. \n\nMoreover, for the previous query encoding methods, they can be scaled-up to the graph with 86,000,000 nodes [1]. Yet, we are not confident that the QE-GNN can also scale up to large graphs.  Meanwhile, we also evaluate the performance of FuzzQE, another fuzzy logical based method over the fixed query embedding size. \n\n[1] Ren, H., Dai, H., Dai, B., Chen, X., Zhou, D., Leskovec, J., & Schuurmans, D. (2022, August). Smore: Knowledge graph completion and multi-hop reasoning in massive knowledge graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 1472-1482).\n\n"}, {"review_id": "1JYVRnJucq", "reviewer": "Reviewer_Ne3e", "summary": "The paper proposed a reasoning task \"Complex Eventuality Query Answering (CEQA)\". CEQA is performed over EVKG and is different from traditional CQA over entity-centric KG. Authors of the paper clearly explain the new task, and further discussed a memory-augmented method to improve models' performance on CEQA.", "questions": "Please see above.", "limitations": "The paper proposed a novel reasoning task potentially useful for building more powerful and general reasoning systems. The paper has made substantial contribution in proposing new tasks, but is limited in modeling and experiments.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper made two contributions. First, it proposed the CEQA task and discussed its difference from the traditional CQA task. Then, the paper proposed a memory augmentation method for query encoding.\n\nThe authors clearly discussed the CEQA task and emphasized its importance in logical reasoning.", "weaknesses": "I have a few quick comments of the paper.\n\nMaybe it worth briefly discussing System 1 and 2 for the completeness of the paper. Readers may be unfamiliar with the terms. \n\nPlease consider discussing your dataset created from ASER earlier in the paper and maybe go through a few examples. This task is new, to my knowledge, to many readers including myself. Maybe also discuss ASER a bit more. How is ASER constructed? \n\nI have another concern about the quality of ASER, the backbone database of the proposed task. Some statements and/or reasoning can be ambiguous or debatable. For example, someone can say, \"PersonX adds soy sauce\" is the cause of \"Food is bad\" due to some other implicit information in addition to the temporal one. How did ASER (or other similar database resolve this problem)?\n\nHow is ASER related and different from Commonsense QA? Both require reasoning with implicit information. Are there other relevant tasks in addition to the ones you described in the paper. Please include them in the Related Work section.\n\nAre you able to experiment with a few more datasets? I understand this is a new task with limited resources. However, experiment results in this paper is limited to fully prove the effectiveness of your proposed method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and efforts for reviewing this paper.\n\nW1 System 1 and System 2: \n\nIn short, the theory of System 1 and System 2 reasoning, proposed by Daniel Kahneman, suggests that human thinking can be divided into two systems. System 1 operates automatically and intuitively, making quick judgments and performing routine tasks effortlessly. System 2, on the other hand, engages in deliberate and analytical thinking, requiring conscious effort. In the context of logical query answering, System 2 reasoning is engaged when we need to carefully analyze the question, consider relevant information, and apply logical rules to arrive at a correct answer. We will incorporate brief discussions in our introduction later.\n\n\n\nW2 curated benchmark and ASER:\n\nWe will add a subsection presenting more detailed statistics, knowledge format, and examples of ASER and our constructed benchmark. For your information, ASER is constructed via two main steps: extraction and conceptualization. The extraction step involves extracting eventualities (events or situations) from a collection of large corpora. Then, in the conceptualization step, the authors use a graph-based approach to represent the extracted eventualities and their relations as a knowledge graph. This involves identifying and linking semantically related eventualities and relations to form a large-scale eventuality knowledge graph.\n\nW3 : \n\nThe question posed is a good one, and to answer it, we need to understand how the edges of ASER are constructed from text using information extraction methods. When a statement like \"PersonX adds soy sauce\" is identified as the cause of \"Food is bad,\" and both phrases appear in the text corpus above a certain frequency, they are recorded in the ASER graph. Similarly, phrases like \"PersonX sleeps\" before and after \"PersonX takes a shower\" can both be plausible high-frequency edges that are stored in the KG, despite being contradictory in a specific situation.\nIt's important to note that, in a random subgraph of ASER or a similar database, not all edges can hold simultaneously in a specific situation. This is a characteristic of a database describing events and activities, and not a shortcoming of the database itself. The KG is designed to provide all possible answer candidates, while logical verification is left to the reasoning process. This is the main motivation behind the approach taken in this paper.\n\n\nW4 Difference with commonsense QA:\n\nAs we discussed with Reviewer DAya, our task different with other QA or implicit reasoning tasks. Our task is distinct from traditional question answering tasks due to its wide scope, encompassing various relationships, including non-common sense relations found in Treebank 2.0. This resource provides additional relations, making our task more complex. In particular, we encounter queries that involve relationships unique to eventuality levels, such as co-occurrence, conjunction, and contradiction. These intricate connections cannot be effectively addressed using common sense question answering methods. Our main focus is on complex query answering, where queries center around intricate relationships between eventualities. Unlike existing common sense knowledge graphs (CSKGs), which typically handle relations involving two events in a triple, our task involves multiple events within a single query-answer pair. This presents a challenge in formulating our task as either a knowledge graph completion (KGC) or question-answering (QA) task, as such formulations would require discarding most query constraints, reducing complexity, and simplifying it into a basic query answering task. While commonsense knowledge may play a role in answering our queries, it is not as prevalent as in other tasks. Additionally, our task does not heavily rely on the semantic information of the query itself; instead, it relies on learning graph structures to perform query answering and reasoning. We utilize the inherent structure of the graph rather than relying solely on natural language processing. \nAs for relevant tasks, there are several complex query answering tasks that share similar settings with the one in our paper, like the EFO1 benchmarks [2]. \nWe will make sure these differences are more clearly discussed in our paper.\n\nW5 More dataset:\n\nThank you for your suggestion. Incorporating additional datasets would undoubtedly strengthen our paper. However, at this stage, we have been unable to identify other viable datasets. The main challenge lies in finding knowledge bases or graphs that encompass both (i) eventualities, including states, actions, and events, as nodes, and (ii) comprehensive relationships between these eventualities, as edges. We will remain vigilant and explore any new resources that become available in the future.\n\n\nReference: \n\n[1] Daniel, K. (2017). Thinking, fast and slow.\n\n[2] Wang, Z., Yin, H., & Song, Y. (2022). Benchmarking the Combinatorial Generalizability of Complex Query Answering on Knowledge Graphs. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 (NeurIPS Datasets and Benchmarks 2021).\n"}, {"review_id": "nwhy8ph2YE", "reviewer": "Reviewer_YBho", "summary": "The paper extends the traditional complex query-answering task into an eventuality-centric complex query-answering task to understand the reasoning at the eventuality level. Specifically, the paper divides the discourse rations into two types of implicit constraints: occurrence constraints and temporal constraints. The occurrence constraints determine whether certain eventuality happens or not. The temporal constraints provide the order of the eventualities' occurrence. In addition, the paper proposes a new memory-enhanced query encoding to reason based on eventuality-centric knowledge graphs. The model first introduces a computational graph that encodes queries, including operations such as relational projection and intersections. The paper proposes a memory-enhanced encoding component that utilizes a memory module to encode constraint information. The operation output is used as a query to access each head eventuality based on the relevance score. The paper then uses an attentional aggregation to sum over the constraint relation and tails. The final model is optimized based on similarity scores with cross-entropy loss. ", "questions": "1. In section 3.1, what function is used for relation projection and intersection operation?\n2. In section 3.2, what relevance score is used for Equation 5?\n3. Has the paper also tested the new model for answers without any contradictions?", "limitations": "The paper provides a limitation section and broader impact in the Appendix. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper introduces a new eventuality-centric complex query-answering task to better model the reasoning at the eventuality level. The paper proposes a new way to provide discourse relations with two implicit logic constraints. The idea of occurrence constraints and temporal constraints is interesting.\n2. The paper introduces a new memory-enhanced query encoding to update the query representation with relevance-based constraint representations.\n3. The paper tests the new framework with a new dataset sampled from ASER. The model shows strong performance over multiple different baselines. The paper also provides code and dataset construction details in the Appendix. The paper also includes a case study in the Appendix.", "weaknesses": "Some parts of the paper are not very clear:\n1. In section 3.1, what function is used for relation projection and intersection operation? The paper said that the intersection is a permutation-invariant neural network. However, it needs to be clarified in detail to readers. \n2. In section 3.2, what relevance score is used for Equation 5? Suppose the paper used a semantic relevance score such as cosine similarity. In that case, the motivation for this part is unclear because the constraints with higher similarity might not be the ones with closed relevance. Moreover, Figure 4 needs to be clarified. The paper needs to briefly explain the right part (constraint memory in Figure 4.) I suggest going through the walkthrough example in section 3.2. The code does not include a ReadMe file.\n3. In section 4.1, the paper only focuses on the answer with constraints. Has the paper also tested the new model for answers without any contradictions?\n4. The abbreviation in Table 4 needs to be clarified. It seems that p is the projection, i is the intersection. However, those abbreviations are not clarified in the caption. Readers also find it hard to figure out what e represents. The analysis of Table 4 is superficial. The paper needs to add more qualitative analysis with more concrete examples. The paper needs to conduct an ablation study to show the contribution of each component. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Re W1 & Q1:\n\nThe relation projections and intersection operations are adopted from backbone models. For different backbone bone models, we have different parametrization of Intersection/Union.\n\nGQE [1] model uses a feed-forward layer, followed by an average pooling, and then followed by a matrix multiplication for intersection. The relational projection is modeled by using a matrix multiplication.\nQuery2Particles [2] use self-attentions for intersection modeling, and the gated transition function for relational projection. \nFuzzQE [3] uses element-wise fuzzy logic operations on embedding space to do intersections and unions. Meanwhile it use a feed forward layer, a layer normalization, and a sigmoid function for relational projection. \nNeural-MLP [4] uses MLP-Mixer as an intersection module and the relation projection module.\n\n\n\nRe W2 & Q2:\n\nAll existing query encoders use the similarity between the query embedding and answers embedding to retrieve answers. In our case, this similarity score is an inner product. Because of this, we can use the same similarity measure to compute the relevance between the query embedding and the head eventuality embedding as a relevance score to the constraint. If the constraint is relevant to the query embedding, then relation and tail information of this constraint are added into the query representation. \n\nHowever, if a constraint is relevant, we want to exclude the information of its relation and tail from the query embedding, we are motivated to add a feed-forward layer to adjust the direction of the memory value, which contains the relation and tail embedding.  \n\nFor example, in the example of Figure 4, the “Food is bad” in the constraint memory has a high relevance score to the “the events happens before X complains and leaves the restaurant”. \n\nThen, when the constraint of “Food is bad before PersonY adds soy sauce” is given, we add the information of relation type “Precedence” and tail node “PersonY adds soy sauce” from memory value to the query embedding.  We are motivated to do this because we want to exclude the answer of  “PersonY adds soy sauce” if the following relation projection is “Reason”, “Condition”, or “Succession”, because we want to avoid temporal contradiction. \nWe further prove the correctness of the intuition behind this idea by the ablation study conducted in Re W4. \n\n\nRe W3 & Q3:\n\nWe do not fully this question, because “the answers with constraints” has the same meaning as “the answers without contradictions”. We filtered out the contradictory answers according to the theorem prover, and only kept the answers that satisfy the logical constraints in the dataset construction process. \n\n\n| Models |  | Occurence Constaints |  |  | Temporal Constraints |  |  | Average |  |  |  |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|---|---|\n|  | Hit@1 | Hit@3 | MRR | Hit@1 | Hit@3 | MRR | Hit@1 | Hit@3 | MRR |  |  |\n| GQE | 8.92 | 14.21 | 13.09 | 9.09 | 14.03 | 12.94 | 9.12 | 14.12 | 13.02 |  |  |\n| + CMQE | 10.20 | 15.54 | 14.31 | 10.70 | 15.67 | 14.50 | 10.45 | 15.60 | 14.41 |  |  |\n| + CMQE - Constraints | 8.29 | 12.87 | 11.62 | 8.80 | 13.02 | 12.17 | 8.54 | 12.95 | 11.90 |  |  |\n| + CMQE - FFN | 0.67 | 1.17 | 1.13 | 0.74 | 1.23 | 1.12 | 0.70 | 1.19 | 1.08 |  |  |\n| Q2P | 14.14 | 19.97 | 18.84 | 14.48 | 19.69 | 18.68 | 14.31 | 19.83 | 18.76 |  |  |\n| + CMQE | 15.15 | 20.67 | 19.38 | 16.06 | 20.82 | 19.74 | 15.61 | 20.74 | 19.56 |  |  |\n| + CMQE - Constraints | 14.16 | 20.00 | 18.86 | 14.72 | 19.92 | 18.79 | 14.44 | 19.96 | 18.82 |  |  |\n| + CMQE - FFN | 12.77 | 16.63 | 15.89 | 12.74 | 16.83 | 14.75 | 12.76 | 16.73 | 15.32 |  |  |\n| Nerual MLP | 13.03 | 19.21 | 17.75 | 13.45 | 19.06 | 17.68 | 13.24 | 19.14 | 17.71 |  |  |\n| + CMQE | 15.26 | 20.69 | 19.32 | 15.91 | 20.63 | 19.47 | 15.58 | 20.66 | 19.40 |  |  |\n| + CMQE - Constraints | 13.33 | 19.15 | 17.94 | 13.49 | 19.18 | 14.48 | 13.41 | 19.16 | 18.08 |  |  |\n|  + CMQE - FFN | 10.35 | 14.67 | 13.71 | 10.94 | 14.67 | 12.74 | 10.64 | 14.67 | 14.53 |  |  |\n| FuzzQE | 11.68 | 18.64 | 17.07 | 11.68 | 17.97 | 16.53 | 11.68 | 18.31 | 16.80 |  |  |\n| + CMQE | 14.76 | 21.12 | 19.45 | 15.31 | 21.01 | 19.49 | 15.03 | 21.06 | 19.47 |  |  |\n| + CMQE - Constraints | 12.69 | 19.92 | 17.68 | 13.53 | 18.25 | 17.91 | 13.11 | 19.08 | 17.80 |  |  |\n| + CMQE - FFN | 9.81 | 15.26 | 14.46 | 10.17 | 15.37 | 14.87 | 9.99 | 15.31 | 14.66 |  |  |\n\n\n\nRe W4: \n\nYes, for the query types notations, the “p” is for projection, the “i” is for intersection, and “e” is for eventuality. We will include the detailed introduction of query types and abbreviations in the paper.\n\nWe conducted an ablation study on our proposed MEQE method to prove the effectiveness of the relevance score and the feed-forward To prove the effectiveness of the relevance score and the feed-forward module mentioned in Re W2, we conducted an ablation study on our proposed MEQE method, and here are the results: When we removed the feed-forward network and directly added the relations and tails embedding to the query embedding, the performance was negatively affected. This is because the query embedding is more likely to have higher similarity to the answers that should be excluded. This effect was more significant in the GQE model, as the GQE model uses the simplest element-wise addition as the relations projection.\n\nWe conduct another ablation of replacing the constraints to random triples so that there are no contradictory answers. Then we observed that the performance of the baseline models is comparable with the MEQE model. This indicates that the performance improvement is gained from the constraints instead of the structural changes of the query encoder. \n\nThis experiment proves two things. First, the relevance score is effective in finding the corresponding constraints. Second, the feed-forward layer is useful and necessary to adjust the direction of the memory contents to incorporate into the query embedding.\n\n\n"}]

Author rebuttal:

Meta-review: I support accepting this paper. All reviewers besides Reviewer DAya support accepting the paper. 

I read the author response and think the authors have sufficiently answered Reviewer DAya's concerns (the reviewer did not reply to the response). In particular,

-As the authors state, their paper is substantially different from common sense reasoning:  "Some complex queries include relations that only exist at eventuality level, such as co-ocurence, conjunction, contradiction, cannot be effectively simulated by commonsense question answering. "

-The authors addressed the concern about the quality of the benchmark by discussing the quality of ASER as well as the results of an Amazon Mechnical Turk evaluation.
Decision: Accept (poster)