SlowGuess/NAACL_2022_OCR huggingface dataset:
Xiaomeng Jin, Manling Li, Heng Ji
{xjin17, manling2, hengji}@illinois.edu
Event schema depicts the typical structure of complex events, serving as a scaffolding to effectively analyze, predict, and possibly intervene in the ongoing events. To induce event schemas from historical events, previous work uses an event-by-event scheme, ignoring the global structure of the entire schema graph. We propose a new event schema induction framework using double graph autoencoders, which captures the global dependencies among nodes in event graphs. Specifically, we first extract the event skeleton from an event graph and design a variational directed acyclic graph (DAG) autoencoder to learn its global structure. Then we further fill in the event arguments for the skeleton, and use another Graph Convolutional Network (GCN) based autoencoder to reconstruct entity-entity relations as well as to detect coreferential entities. By performing this two-stage induction decomposition, the model can avoid reconstructing the entire graph in one step, allowing it to focus on learning global structures between events. Experimental results on three event graph datasets demonstrate that our method achieves state-of-the-art performance and induces high-quality event schemas with global consistency. $^{1}$
Event Schema (Chambers and Jurafsky, 2008, 2009; Balasubramanian et al., 2013; Nguyen et al., 2015; Modi et al., 2016; Li et al., 2021) is induced from historical events to describe the common or stereotypical evolution pattern of complex events. Figure 1b shows an example schema of complex event "IED-bombing" (Improvised Explosive Device), where multiple events are inter-connected via temporal links (e.g., TRANSPORT happens after ASSEMBLE) and arguments (e.g., the transporting ENTITY is the weapon that is being assembled; the
transporting DESTINATION is controlled by the as-tembler, i.e., the AGENT). It enables a descriptive analysis of inter-event structures, as well as the prediction of future events over temporal-based and argument-based structures.
A number of methods have been proposed for learning event schemas from instance event graphs, called event schema induction, which fall into three categories. The set-based (Chambers, 2013; Sha et al., 2016; Huang et al., 2016) and sequence-based (Granroth-Wilding and Clark, 2016; Rudinger et al., 2015) methods treat a complex event as a set or a linear sequence of atomic events, respectively. However, they fail to capture the multi-dimensional evolution of real-world complex events, i.e., multiple events may precede or follow one event. Graph-based methods (Li et al., 2020, 2021), instead, adopt graphs to formulate event schemas. A graph model is usually learned from instance event graphs through generating the schema event by event. Although graph-based methods are theoretically superior to the first two categories, existing graph-based methods are limited to modeling only the local structure of event graphs, i.e., the first-order dependency of an event node with respect to its neighbors, while ignoring the high-order and global dependencies among atomic events in the entire graph.
However, modeling the global structure of event graphs is crucial to event schema induction. The global structure enables the model to be aware of the position of each event node in the entire graph. It allows the model to better comprehend the role of a specific event in the complex event. For example, in Figure 1b, there are three TRANSPORT events in the schema, but they differ regarding the item being transported, i.e., bombs, the injured, or suspects. The global structure context enables the model to differentiate the position of the three TRANSPORT events and predict their neighbor events precisely. Moreover, when the model only has access to local
<sup>1</sup>The programs, data and resources are made publicly available for research purpose in https://github.com/tracyjin/DoubleGAE.git
Human Language Technologies, pages 2013 - 2025
The September 11 attacks were a series of four coordinated terrorist attacks. Four commercial airliners traveling from the northeastern U.S. to California were hijacked mid-flight by 19 al-Qaeda terrorists. Each group had one hijacker who took over control of the aircraft. Their explicit goal was to crash each plane into a prominent American building, causing mass casualties and partial or complete destruction of the targeted buildings. The attacks resulted in 2,977 fatalities, over 25,000 injuries, and substantial long-term health consequences.
(a) Event instance graph
(b) Event schema graph
Figure 1: An illustrative example of (a) an instance event graph of 9/11 Attacks extracted from a news article and (b) graph-based event schema of IED bombings. We use circles to denote events and squares to represent event arguments (entities). The solid blue arrows represent temporal links between events (i.e., event skeleton), the solid yellow lines represent argument roles between events and entities, and the dashed blue lines represent entity-entity relations.
structure information during the generation process, the schema generated may only be consistent at the local level, but not at the global level. For example, the schema may keep repeating the sequence of ASSEMBLE - TRANSPORT - ATTACK. From this perspective, the global structure information can be viewed as the supervision that guides the entire generation process to be globally consistent.
To capture the global structure information of complex events, in this paper, we propose a new event schema induction approach using double graph autoencoders. Graph Autoencoder (Salha et al., 2019; Zhang and Chen, 2018) is known to be able to preserve the structural information of an entire graph in the embedding space. Therefore, our key idea is to use graph autoencoders to capture the
global dependency among nodes in event graphs.
Specifically, our model contains two graph autoencoders that are organized in a hierarchical manner: (1) To model the skeleton of an event graph, we design a high-level variational graph autoencoder for directed acyclic graphs (DAGs). Event skeleton is a subgraph of an event graph, which consists of salient events and their temporal orders, representing the fundamental structure of event evolution. (2) With the event skeleton as the global context, we decorate entity nodes in the skeleton by introducing a low-level graph autoencoder based on Graph Convolutional Network (GCN). It takes as input an expanded event skeleton and reconstructs the original event graph by adding coreferential entities and entity-entity relations.
These two graph autoencoders decompose the process of event schema induction into two steps, thereby avoiding the need to reconstruct the entire graph directly, and improving the schema learning efficiently at both the high level (event schema skeleton) and the lower level (entity-entity relations).
We conduct extensive experiments on three event graph datasets. The experimental results demonstrate that our proposed method achieves state-of-the-art performance on event schema induction. Additionally, we show in a case study that the event schema generated by our method is more reasonable and globally consistent.
We make the following novel contributions:
- We propose a two-stage global structure aware schema induction framework, providing a global context of event skeleton to determine inter-event interactions via arguments.
- We introduce a double graph autoencoder that preserves the global structural information, allowing the model to capture high-order dependencies between nodes.
- We propose a comprehensive set of metrics for structure-aware comparison between schema graphs and instance graphs.
- Our method significantly outperforms baselines, demonstrating the effectiveness of considering global structure context in event schema induction.
Our data resources come from news and Wikipedia articles that describe a series of complex events.
Figure 2: The architecture of our model.
We extract events, entities, as well as their relations using Information Extraction (IE) tools (Wu et al., 2008; Li et al., 2011; Hogenboom et al., 2013; Lin et al., 2020), then construct instance event graphs. An instance event graph consists of two types of nodes: event nodes and entity nodes, and we use  $t_i$  to represent the type of node  $i$ . Similarly, we use  $t_{ij}$  to denote the edge type between node  $i$  and node  $j$ . Accordingly, there are three types of edges in an instance graph: (1) the event-event temporal link  $(i,j)$ , which represents a temporal order that event  $j$  happens after event  $i$ , with  $t_i$  and  $t_j$  indicating their event types, such as TRANSPORT; (2) the event-entity argument link  $(i,a,m)$ , which represents that event  $i$  has an argument entity  $m$ , which plays the argument role  $t_{im} = a$ , such as AGENT; (3) the entity-entity relation  $(m,r,n)$ , which represents that there is a relation  $t_{mn} = r$  between entity  $m$  and entity  $n$ , such as AFFILIATION.
Given a set of instance graphs  $\{G_1, G_2, \dots\}$  that belong to the same topic, our goal is to learn a schema  $S$  that summarizes the instance graphs and represents their underlying common evolution pattern. Note that different from instance graphs, nodes (events and entities) in the schema  $S$  are not instantiated but represented by their types.
We design two graph autoencoders, where the first autoencoder deals with the high-level skeleton of an event graph and the second autoencoder focuses on
on the low-level arguments of an event graph.
As shown in the upper side of Figure 2, the high-level autoencoder, which is specially designed for directed acyclic graphs (DAGs), takes an instance event skeleton as input, and encodes the event skeleton as a probability distribution in the embedding space. Then it reconstructs the input skeleton by feeding a vector sampled from the distribution into the decoder.
After the event skeleton is reconstructed, we decorate the entity nodes according to the pre-defined argument roles of each event (as shown on the right side of Figure 2). However, the entity-entity relations and coreference links among arguments are still missing in the expanded event skeleton. Therefore we introduce a low-level graph autoencoder to take as input an expanded event skeleton and reconstruct the original event graph (as shown in the lower part of Figure 2). The low-level graph autoencoder employs Graph Convolutional Network (GCN) to encode each node into an embedding vector, then predicts the type of an entity-entity relation based on the learned entity embeddings.
3.2 Event Skeleton Generation
An instance event graph can contain up to hundreds of nodes, but the majority are entity nodes that are associated with event nodes. Therefore, as shown in the upper left of Figure 2, our first step is to extract the event skeleton  $G_{S}$  from the instance graph  $G$ , which serves as the backbone of  $G$ .
For a given instance event graph  $G$ , we use a
graph neural network (GNN) based variational autoencoder to process its event skeleton  $G_{S} \subseteq G$ . Traditional GNNs learn the node representations by aggregating information from their neighbors iteratively, then apply a readout function to all node representations and output the representation of the entire graph (Xu et al., 2019). However, off-the-shelf GNNs are not suitable for modeling event skeleton, because event skeleton is a DAG whose nodes follow an intrinsic partial order, whereas existing GNN models focus more on capturing the local structure of a graph.
Encoding. To capture the global structure of event skeleton  $G_{S}$ , inspired by Zhang et al. (2019), we design a new GNN architecture for the encoder, in which messages can only pass forward following event-event temporal orders. Specifically, for a given event node  $i$  in the event skeleton  $G_{S}$ , its representation  $\mathbf{s}_i$  is computed by:
$$
\mathbf {s} _ {i} = \operatorname {A G G} \left(\left\{\mathbf {s} _ {j} \mid (j, i) \in G _ {S} \right\} \cup \mathbf {t} _ {i}\right), \tag {1}
$$
where  $\mathbf{t}_i$  is the one-hot event type vector of an event node  $i$ , and we use it as the initial feature for the event.  $\mathrm{AGG}(\cdot)$  is the aggregate function. In Eq. (1), we only consider predecessors as neighbors, allowing the model to capture the temporal flow of the event graph. Considering that predecessors contribute differently to predicting the current event node, we utilize a self-attention function as the aggregate function to characterize the importance of different predecessors:
$$
\mathbf {s} _ {i} = \sigma \left(\mathbf {W} _ {1} \sum_ {j: (j, i) \in G _ {E}} \operatorname {S o f t M a x} \left(\alpha_ {i j}\right) \mathbf {s} _ {j} + \mathbf {W} _ {2} \mathbf {t} _ {i}\right), \tag {2}
$$
where  $\alpha_{ij} =$  LeakyReLU  $\left(\mathbf{w}^{\top}\left[\mathbf{W}_{1}\mathbf{s}_{j},\mathbf{W}_{2}\mathbf{t}_{i}\right]\right)$  is (unnormized) attention weight,  $\mathbf{w}$  is a learnable vector,  $\mathbf{W}_1$  and  $\mathbf{W}_2$  are learnable matrices,  $[\cdot ,\cdot ]$  denotes concatenate operation, and  $\sigma (\cdot)$  is a nonlinear activation function.
It is worth noting that according to Eq. (1),  $\mathbf{s}_i$  can only be computed after its predecessors' representations have been computed, which implies that the encoder computation sequence naturally follows the topological order of the event skeleton. Specifically, we first create a dummy event node START that has an outgoing link to each event node that has no predecessor, and a dummy event node END with an incoming link from each event node that has no successor. Then we compute the representations of all event nodes according to
their topological order. Finally, the representation of END, i.e.,  $\mathbf{s}_{\mathrm{END}}$  , is taken as the output of the encoder.
Decoding. The decoder follows a GNN architecture similar to the encoder, as presented in Algorithm 1. The input is the encoder output  $\mathbf{s}_{\mathrm{END}}$  for  $G_{S}$ , and our goal is to reconstruct the skeleton graph  $G_{S}^{\prime}$ . Specifically, we first obtain the posterior approximation  $p(\cdot | G_{S})$  by calculating the mean vector  $\mu$  and the diagonal covariance matrix  $\Sigma$  via two Multi-Layer Perceptrons (MLPs) (line 2). Then we sample a vector  $\mathbf{s}_G$  from the Gaussian distribution  $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$  as the global representation of the skeleton graph, which will be used throughout the following generation process (line 3). The representation of the generated graph  $\mathbf{g}$  is initialized as an all-zero vector (line 4).
Next, the decoder generates a DAG node by node (line 5-17). When generating a new event node  $i$ , the decoder computes the event type distribution for node  $i$  to obtain a sampled type  $t_i$  (line 6). The distribution is learned based on the concatenation of  $\mathbf{s}_G$  and  $\mathbf{g}$ , summarizing the input skeleton graph and the generated graph. Based on the value of  $t_i$ , the decoder performs one of the following two steps:
- If the generated event node  $i$  is END, then the decoder will stop the generation procedure, and connect all nodes that do not have any predecessor to the END node (line 8-10).
- Otherwise, the decoder uses another MLP to predict the edge probability of node  $i$  with existing nodes, and add the generated edges into the graph (line 12-15). After the generation step for node  $i$  is completed, the decoder then computes the representation  $\mathbf{s}_i$  for node  $i$  according to Eq. (2) (line 16), and updates the generated graph representation  $\mathbf{g}$  by  $\mathbf{s}_i$  (line 17). The updated  $\mathbf{g}$  will be further used to generate a new node for the next iteration.
Different from existing event schema induction methods that are only able to model the local structural information of event graphs, our method encodes the global information of an event graph as the global graph representation  $\mathbf{s}_G$ , and further applies this global state to guide the entire generation process. This enables our model to fully capture the structural information of event graphs, which has been shown to be extremely important for event schema induction.
3.3 Entity-Entity Relation Completion
After generating the event skeleton, we aim to complete the generated event schema graph by adding back entities as well as edges associated with these entities. We take advantage of the event ontology, where each event has a predefined set of argument roles. For example, an ATTACK event has an ATTACKER argument role whose type can be PERSON (PER), as well as a TARGET argument role whose type can be LOCATION (LOC). Therefore, we first expand the event skeleton by adding the predefined event arguments back to each event, as shown in the right of Figure 2. Yet, such an expanded event skeleton  $G_{E}$  is not the final event schema, because: (1) entity-entity relations are missing, e.g., the LOCATED_IN relation between a PER entity and a LOC entity in Figure 1b; (2) entities in  $G_{E}$  can be coreferential, which require to be merged. For example, in Figure 1b, the DESTINATION of the TRANSPORT event is also the PLACE where the ATTACK event happens.
We formulate the above two cases as a unified entity-entity edge missing problem, by treating coreference as a special type of entity-entity relation between unmerged nodes. To predict missing entity-entity relations, we design a low-level
graph autoencoder. It reads the expanded event skeleton  $G_{E}$  as input, and then reconstructs the original event graph  $G$  by adding entity-entity relations back, as shown in the lower part of Figure 2. During the inference stage, the learned graph autoencoder can therefore take a generated event skeleton as input, and output a comprehensive event schema graph.
Encoding. Different from the high-level autoencoder whose purpose is to generate event skeleton, the low-level autoencoder is to reconstruct the relations between entities, therefore, the low-level graph autoencoder is expected to be nonprobabilistic, and therefore we adopt Graph Convolutional Networks (GCN) (Kipf and Welling, 2017) as our encoder. Let  $\mathbf{s}_i^k$  denote the representation at iteration  $k$  for the node  $i\in G_{E}$ , which can be either an event or an entity. The encoder updates node representations iteratively for  $k = 1,\dots ,K$ , where  $K$  is the layer number of the GCN encoder:
$$
\mathbf {s} _ {i} ^ {k} = \sigma \left(\mathbf {W} ^ {k} \sum_ {j \in \mathcal {N} (i) \cup \{i \}} \alpha_ {i j} \mathbf {s} _ {j} ^ {k - 1}\right). \tag {3}
$$
Here  $\mathcal{N}(\cdot)$  denotes the set of neighbors of node  $i$ ,  $\alpha_{ij} = 1 / \sqrt{|\mathcal{N}(i)|\cdot|\mathcal{N}(j)|}$ ,  $\mathbf{W}^k$  is a learnable matrix, and  $\sigma$  is an activation function.  $\mathbf{s}_i^0$  is initialized as the one-hot type vector of node  $i$ , i.e.,  $\mathbf{s}_i^0 = \mathbf{t}_i$ . All edges in  $G_E$ , including event-event temporal links and event-entity argument links, are treated as undirected when counting neighbors, due to our focus on the local graph structure. The encoder output is the final representation  $\mathbf{s}_i^K$  of node  $i \in G_E$ .
Decoding. The decoder takes as input the final node representations  $\mathbf{s}_i^K$  from the encoder, and reconstructs entity-entity relations in the original event graph  $G$ . The entity-entity relation can be one of the following three cases: no relation, coreference, or predefined entity-entity relation types in the ontology. Therefore, we create two new relation types NO-RELATION and CO-REFERENCE, and add them into the original set of entity-entity relations as the prediction target. Specifically, the decoder predicts the relation type  $t_{ij}$  of two given entities  $i$  and  $j$  by a MLP:
$$
\hat {t} _ {i j} = \mathrm {M L P} _ {\text {e n t i t y - r e l}} \left(\left[ \mathbf {s} _ {i} ^ {K}, \mathbf {s} _ {j} ^ {K} \right]\right), \tag {4}
$$
where  $\hat{t}_{ij}$  denotes the predicted relation type. It is noteworthy that NO-RELATION dominates the prediction targets, making the classification problem highly unbalanced. To address this issue, we divide
the task into two steps: predicting the existence of edges and then predicting the type of a known edge, to optimize the learning effectiveness.
3.4 Training and Generation
Training. The training objective for the high-level variational graph autoencoder consists of a reconstruction loss and a regularization loss:
$$
L _ {h i g h} = \operatorname {D i s t} \left(G _ {S}, G _ {S} ^ {\prime}\right) + \mathrm {K L} \left(\mathcal {N} (\boldsymbol {\mu}, \boldsymbol {\Sigma}), \mathcal {N} (\mathbf {0}, \mathbf {1})\right), \tag {5}
$$
where the first term  $\mathrm{Dist}(G_S, G_S')$  measures the distance between the input event skeleton  $G_S$  and the reconstructed event skeleton  $G_S'$ . We sum up the negative log-likelihood of each decoding step by forcing them to generate the ground truth event type or edge at each step. The second term  $\mathrm{KL}(\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma}), \mathcal{N}(\mathbf{0}, \mathbf{1}))$  forces the distribution output by the encoder to be close to the standard Gaussian distribution, to ensure a smooth embedding space.
The loss function for the low-level GCN graph autoencoder is the cross-entropy loss between predicted and ground-truth entity-entity relations:
$$
L _ {l o w} = \sum_ {\text {e v e n t} i, j} \operatorname {C E L o s s} \left(\hat {t} _ {i j}, t _ {i j}\right). \tag {6}
$$
Schema Generation. We are able to generate the event schema by decoding the trained model. We first sample a global graph representation from  $\mathcal{N}(\mathbf{0},\mathbf{1})$  in the embedding space of the high-level variational graph autoencoder. It is then fed into the decoder to generate a schema skeleton with event nodes only. Afterwards, the skeleton is further fed into the low-level graph autoencoder to predict coreferential entities and entity-entity relations.
We conduct experiments in the scenario of IED bombings following the state-of-the-art graph-based schema induction literature (Li et al., 2021). Specifically, three subtypes of complex events for IED are considered, including General IED, Car bombing IED, and Suicide IED. To construct the dataset, we select a set of Wikipedia articles related to IED bombings and identify the references in each Wikipedia article, then collect news articles from those references  $^{2}$ . We use RESIN (Wen et al.,
2021; Du et al., 2022), a state-of-the-art IE system, to extract event mentions, relations, and entity mentions from these news articles, and perform entity coreference resolution. We also do human curation to correct obviously erroneous event-event links (e.g., a temporal link indicating that an INJURE event happens after a DIE event for one person). The statistics of these three datasets are summarized in Table 1.
We compare our proposed method with graph schema induction baselines:
(1) Temporal Event Graph Model (TEGM) (Li et al., 2021). It is the state-of-the-art graph schema induction model, which generates the entire event graph using an auto-regressive graph generation model. We compare with it to explore the effectiveness of the two-stage framework that generates event skeleton first and provides a global context for argument generation.
(2) Frequency-Based Sampling (FBS). To examine the effectiveness of our graph autoencoder to generate event skeleton, we compare with a frequency-based baseline. It constructs the event schema based on the frequency of event-event temporal links in the training data. Specifically, for each possible pair of event types  $(t,t^{\prime})$ , we compute the number of edges in training graphs whose two associated events exactly have type  $t$  and  $t^\prime$ . After that, we construct a schema graph in which each node corresponds to one event type, and there is no edge in the schema graph initially. Then at each timestamp, we sample one pair of event types according to their frequency, and add this sampled edge into the schema graph. The procedure is repeated until we detect a cycle in the schema graph after adding a new edge.