SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our task is different from other QA or implicit reasoning tasks in several ways. Firstly, it has a broader scope, encompassing various relationships, including non-common sense discourse relations found in Treebank 2.0, which is even challenging for large language models [11]. This resource provides additional relations, which include four general types: temporal (before/after), contingency (because/result), comparison (but/although), and expansions (and/or/except/instead). In contrast, common sense relations mainly focus on the first two types of relations: contingency and temporal. The occurrence constraints discussed in this paper primarily exist in the expansion type, which does not appear in common sense KG but exists in the event KG. This makes our task more complex, and it cannot be effectively addressed using common sense question-answering methods [44].
Moreover, our main focus is on complex query answering, where queries center around intricate relationships between eventualities. Unlike existing common sense knowledge graphs (CSKGs), which typically handle relations involving two events in a triple, our task involves multiple events within a single query-answer pair. This presents a challenge in formulating our task as either a knowledge graph completion (KGC) or question-answering (QA) task, as such formulations would require discarding most query constraints, reducing complexity, and simplifying it into a basic query answering task. While commonsense knowledge may play a role in answering our queries, it is not as prevalent as in other tasks [45]. Additionally, our task does not heavily rely on the semantic information of the query itself; instead, it relies on learning graph structures to perform query answering and reasoning. We utilize the inherent structure of the graph rather than relying solely on natural language processing. Finally, there are several complex query-answering tasks that share similar settings with the one in our paper, such as the EFO-1 benchmarks [46].
D Knowledge Graph Details
The eventuality knowledge graph, ASER-50K, is derived from a sub-sample of  $\mathrm{ASER2.1^4}$ . ASER2.1 includes the Co-Occurrence relations, which indicate that two eventualities co-occur in two consecutive sentences in the original text. However, in this paper, we exclude the co-occurrence relation to focus on discourse relations. To remove noise from ASER 2.1, we eliminate edges with low frequencies and retain only those with a frequency higher than two. The ASER graph is constructed using an extractive method from natural language text, which may result in the inclusion of eventualities with high frequency but vague semantics, such as PersonX know and PersonX think. To address this issue, we remove the most frequent one hundred vertices and retain the remaining densest vertices. The resulting ASER-50K dataset comprises 54,557 eventualities and 141,252 edges. Subsequently, we randomly partition the edges into training, evaluation, and testing sets in an 8:1:1 ratio. The numbers of edges in each set are presented in Table 5.
The query types in our framework reflect the structure of the computational graph and are represented using a Lisp-like format [46, 7]. For instance, the query  $(\mathbf{i},(\mathbf{p},(\mathbf{e})),(\mathbf{p},(\mathbf{e})))$  represents a query with two anchor eventualities, each having a relational projection, and the answer eventualities are the intersection of these two projection results. Additionally, this query type is also referred to as 2i in related work [37, 36]. However, our naming approach is more flexible and can be extended to accommodate more complex query structures. We sample our queries based on the query types, limiting them to a maximum of three anchors and a maximum depth of two. Specifically, in the training set, we only sample queries with a maximum of two anchors. Further details regarding the query types in the training, validation, and testing sets can be found in Table 6.
E Query Sampling Algorithm
The query sampling algorithm employed in this study is based on the work by Ren et al. [37]. We replicate the sampling algorithm and provide the pseudo-code for the sampling process in Algorithm 1. Our focus in this paper is on conjunctive logical queries derived from eventuality knowledge graphs. As a result, the query sampling process involves only the operations of relational projections and intersections. Given a knowledge graph  $G$  and a query type  $T$ , we initiate the query generation process by starting with a random node  $v$ . The goal is to recursively construct a query that has  $v$  as its answer, following the structure specified by  $T$ . During each recursive step, we examine the last operation in the query. If the operation is a projection, we randomly select one of its predecessors  $u$  that holds the corresponding relation to  $v$ , which will serve as the answer to the sub-query. The recursion is then applied to node  $u$  and the sub-query type of  $T$ . Similarly, for intersection, we recursively apply the process to their respective sub-queries on the same node  $v$ . The recursion continues until the current node contains an anchor entity, at which point the process terminates. This recursive approach allows us to systematically construct queries that satisfy the given query type  $T$  and have  $v$  as the desired answer.
$^{4}$ https://hkust-knowcomp.github.io/ASER/html/index.html
Algorithm 1 The algorithm used for sampling a complex query from a knowledge graph starting from a random vertex  $v$  from the knowledge graph  $G$  with query structure  $T$ .
Require:  $G$  is a knowledge graph. function SAMPLEQUERY(T,v)  $T$  is an arbitrary node of the computation graph.  $v$  is an arbitrary knowledge graph vertex if T operation  $= p$  then  $u\gets$  SAMPLE({u|(u,v)is an edge in G}) RelType  $\leftarrow$  type of  $(u,v)$  in  $G$  ProjectionType  $\leftarrow p$  SubQuery  $\leftarrow$  SAMPLEQUERY(T(child,u) return (ProjectionType,RelType,SubQuery) else if TOperation  $= i$  then IntersectionResult  $\leftarrow (i)$  for child  $\in T$  .Children do SubQuery  $\leftarrow$  SAMPLEQUERY(T(child,v) IntersectionResult.PUSHBACK(child,v) end for return IntersectionResult else if TOperation  $= e$  then return  $(e,T.$  value) end if end function
Table 7: Ablation Studies on Constraints and Feed-Forward Network in MEQE.
Models	Occurrence Constraints	Temporal Constraints	Average
Hit@1	Hit@3	MRR	Hit@1	Hit@3	MRR	Hit@1	Hit@3	MRR
GQE	8.92	14.21	13.09	9.09	14.03	12.94	9.12	14.12	13.02
+ MEQE	10.20	15.54	14.31	10.70	15.67	14.50	10.45	15.60	14.41
+ MEQE - Constraints	8.29	12.87	11.62	8.80	13.02	12.17	8.54	12.95	11.90
+ MEQE - FFN	0.67	1.17	1.13	0.74	1.23	1.12	0.70	1.19	1.08
Q2P	14.14	19.97	18.84	14.48	19.69	18.68	14.31	19.83	18.76
+ MEQE	15.15	20.67	19.38	16.06	20.82	19.74	15.61	20.74	19.56
+ MEQE - Constraints	14.16	20.00	18.86	14.72	19.92	18.79	14.44	19.96	18.82
+ MEQE - FFN	12.77	16.63	15.89	12.74	16.83	14.75	12.76	16.73	15.32
Nerual MLP	13.03	19.21	17.75	13.45	19.06	17.68	13.24	19.14	17.71
+ MEQE	15.26	20.69	19.32	15.91	20.63	19.47	15.58	20.66	19.40
+ MEQE - Constraints	13.33	19.15	17.94	13.49	19.18	14.48	13.41	19.16	18.08
+ MEQE - FFN	10.35	14.67	13.71	10.94	14.67	12.74	10.64	14.67	14.53
FuzzQE	11.68	18.64	17.07	11.68	17.97	16.53	11.68	18.31	16.80
+ MEQE	14.76	21.12	19.45	15.31	21.01	19.49	15.03	21.06	19.47
+ MEQE - Constraints	12.69	19.92	17.68	13.53	18.25	17.91	13.11	19.08	17.80
+ MEQE - FFN	9.81	15.26	14.46	10.17	15.37	14.87	9.99	15.31	14.66
F Further Ablation Study on the memory module and FFN layer
To demonstrate the effectiveness of the relevance score and the feed-forward module, we conducted an ablation study on our proposed MEQE method, and the results are presented below.
When we removed the feed-forward network, as shown in the rows of "MEQE - FFN" and directly added the relations and tails embedding to the query embedding, the performance was negatively impacted. This is because the query embedding is more likely to have a higher similarity to the answers that should be excluded. This effect was more significant in the GQE model, as the GQE model uses the simplest operation as the relations projection.
We also conducted another ablation study by replacing the constraints with random triples so that there are no contradictory answers in the rows of "MEQE-Constraints". We observed that the performance of the model is comparable to the baseline model. This indicates that the performance improvement is gained from the constraints instead of the structural changes of the query encoder.
Figure 5: The example provided showcases a complex eventuality query along with its implicit constraints, query type, computational graph and atomics visualization.
These experiments prove two things. First, the relevance score is effective in finding the corresponding constraints. Second, the feed-forward layer is useful and necessary to adjust the direction of the memory contents to incorporate into the query embedding.
G Detailed Example of Complex Eventuality Query
Figure 5 provides a detailed example of a complex eventuality query. This query corresponds to the query type  $(\mathfrak{p},(\mathfrak{i},(\mathfrak{p},(\mathfrak{e})),(\mathfrak{p},(\mathfrak{e}))))$ , and its corresponding computational graph is depicted. The implicit constraints of the atomics in the logical query are derived according to the discourse relations. When the computational graph is executed on the eventuality knowledge graph, without considering the logical constraints, there would be four potential answers: Staff is new, PersonY adds ketchup, PersonY adds vinegar, and PersonY adds soy sauce.
However, the answer PersonY adds vinegar is contradictory due to occurrence constraints, as one of the informational atomics indicates that PersonY adds vinegar did not occur. Furthermore, the answer PersonY adds soy sauce is contradictory due to temporal constraints, as it occurs after Food is bad, indicating that PersonY adds soy sauce cannot be the reason for Food is bad.