SlowGuess/NAACL_2021_OCR huggingface dataset:
Zixuan Zhang and Heng Ji
{zixuan11, hengji}@illinois.edu
The tasks of Rich Semantic Parsing, such as Abstract Meaning Representation (AMR), share similar goals with Information Extraction (IE) to convert natural language texts into structured semantic representations. To take advantage of such similarity, we propose a novel AMR-guided framework for joint information extraction to discover entities, relations, and events with the help of a pre-trained AMR parser. Our framework consists of two novel components: 1) an AMR based semantic graph aggregator to let the candidate entity and event trigger nodes collect neighborhood information from AMR graph for passing message among related knowledge elements; 2) an AMR guided graph decoder to extract knowledge elements based on the order decided by the hierarchical structures in AMR. Experiments on multiple datasets have shown that the AMR graph encoder and decoder have provided significant gains and our approach has achieved new state-of-the-art performance on all IE subtasks<sup>1</sup>.
Information extraction (IE) aims to extract structured knowledge as an information network (Li et al., 2014) from unstructured natural language texts, while semantic parsing attempts to construct a semantic graph to summarize the meaning of the input text. Since both of them focus on extracting the main information from a sentence, the output information networks and semantic graphs have a lot in common in terms of node and edge semantics. In an example shown in Figure 1, many knowledge elements in the information network can be perfectly matched to certain nodes in the semantic graph with similar semantic meanings. Moreover, these two types of graphs may also be similar with regard to network topology. Specifically, the nodes that are
Figure 1: Comparison of the AMR graph generated from pre-trained AMR parser and information network from IE for the same sentence from ACE05: Scott Peterson now faces death penalty because of murdering his wife Laci and their unborn son at their house.
neighbors or connected via a few hops in the semantic graph are also likely to be close to each other in the corresponding information network. In Figure 1 we can see that "Scott Peterson", which acts as a shared argument for two event triggers "murdering" and "faces", is also directly linked to two main predicates murder-01 and face-01 in the semantic graph. From a global perspective, an information network can be approximately considered as a subgraph of semantic parsing, where the IE nodes are roughly a subset of the nodes in the semantic graph while maintaining similar inter-connections.
To further exploit and make use of such similarities for information extraction, we propose an intuitive and effective framework to utilize information from semantic parsing to jointly extract an in
<sup>1</sup>The programs are publicly available for research purpose at https://github.com/zhangzx-uiuc/AMR-IE.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 39-49 June 6-11, 2021. ©2021 Association for Computational Linguistics
formation network composed of entities, relations, event triggers and their arguments. We adopt Abstract Meaning Representation (AMR) (Banarescu et al., 2013) which contains rich semantic structures with fine-grained node and edge types as our input semantic graphs. Compared with previous IE models, our proposed model mainly consists of the following two novel components.
AMR-Guided Graph Encoding. The AMR graph topology can directly inform the IE model some global inter-dependencies among knowledge elements, even if they are located far away in the original sentence. Such a property makes it easier for the IE model to capture some non-local long-distance connections for relation and event argument role labeling. We design a semantic graph aggregator based on Graph Attention Networks (GAT) (Velickovic et al., 2018) to let the candidate entity and event trigger nodes to aggregate neighborhood information from the semantic graph for passing message among related knowledge elements. The GAT architecture used in our model is specifically designed to allow interactions between node and edge features, making it possible to effectively leverage the rich edge types in AMR.
AMR-Conditioned Graph Decoding. A large number of nodes in these two types of graphs share similar meanings, which makes it possible to obtain a meaningful node alignment between information networks and semantic graphs. Such an alignment provides potential opportunities to design a more organized way in the decoding part of a joint IE model. Instead of using sequential decoding as in previous models like OneIE (Lin et al., 2020), where the types of knowledge elements are determined in a left-to-right order according to their positions in the original sentence, we propose a new hierarchical decoding method. We use AMR parsing as a condition to decide the order of decoding knowledge elements, where the nodes and edges are determined in a tree-like order based on the semantic graph hierarchy.
Experiment results on multiple datasets show that our proposed model significantly outperforms state-of-the-art on all IE subtasks.
We focus on extracting entities, relations, event triggers and their arguments jointly from an input sentence to form an information network. Note that the AMR graphs in our model are not required
to be ground-truth but are generated by pretrained AMR parsers. Therefore, we do not incorporate additional information and our problem settings are identical to typical joint information extraction approaches such as DyGIE++ (Wadden et al., 2019) and OneIE (Lin et al., 2020). Given an input sentence  $S = \{w_{1}, w_{2}, \dots, w_{N}\}$ , we formulate our problem of joint information extraction as follows.
Entity Extraction Entity extraction aims to identify word spans as entity mentions and classify them into pre-defined entity types. Given the set of entity types  $E$ , the entity extraction task is to output a collection  $\mathcal{E}$  of entity mentions:
$$
\mathcal {E} = \left\{\varepsilon_ {i} = \left(a _ {i}, b _ {i}, e _ {i}\right) \mid a _ {i} \leqslant b _ {i}, e _ {i} \in E \right\}
$$
where  $a_{i},b_{i}\in \{1,2,\dots ,N\}$  denote the starting and ending indices of the extracted entity mentions, and  $e_i$  represents the entity type in a type set  $E$ . For example, in Figure 1, the entity mention "Scott Peterson" is represented as  $(0,1,\text{PER})$ .
Relation Extraction The task of relation extraction is to assign a relation type to every possible ordered pair in the extracted entity mentions. Given the identified entity mentions  $\mathcal{E}$  and pre-defined relation types  $R$ , the set of relations is extracted as
$$
\mathcal {R} = \left\{r _ {i} = (\varepsilon_ {i}, \varepsilon_ {j}, l _ {i j} ^ {r}) \mid l _ {i j} ^ {r} \in R, \varepsilon_ {i}, \varepsilon_ {j} \in \mathcal {E} \right\}
$$
where  $\varepsilon_{i}$  and  $\varepsilon_{j}$  are entity mentions from  $\mathcal{E}$  and  $i,j\in \{1,2,\dots ,|\mathcal{E}|\}$ . An example relation mention is ("their", "son", PER-SOC) in Figure 1.
Event Extraction The task of event extraction includes extracting event triggers and their arguments. Event trigger extraction is to identify the words or phrases that most clearly indicate the occurrence of a certain type of event from an event type set  $T$ , which can be formulated as:
$$
\mathcal {T} = \{\tau_ {i} = (p _ {i}, q _ {i}, t _ {i}) \mid p _ {i} \leqslant q _ {i}, t _ {i} \in T \}
$$
where  $p_i, q_i \in \{1, 2, \dots, N\}$  denotes the starting and ending indices of the extracted event mentions, and  $t_i$  represents an event type in  $T$ . Given the pre-defined set of event arguments  $A$ , the task of event argument extraction is to assign each trigger and entity pair an argument role label to indicate if an entity mention acts as some certain role of the event, which is formulated as extracting an argument set  $\mathcal{A}$
$$
\mathcal {A} = \left\{\alpha_ {i} = (\tau_ {i}, \varepsilon_ {j}, l _ {i j} ^ {a}) \mid l _ {i j} ^ {a} \in A, \tau_ {i} \in \mathcal {T}, \varepsilon_ {j} \in \mathcal {E} \right\}
$$
where  $\tau_{i}$  and  $\varepsilon_{j}$  are previously extracted event and entity mentions respectively, and  $l_{ij}^{a}$  denotes the event argument role label.
Information Network Construction All of these extracted knowledge elements form an information network  $G = (V, E)$  (an example is shown in Figure 1). Each node  $v_{i} \in V$  is an entity mention or event trigger, and each edge  $e_{i} \in E$  indicates a relation or event argument role. Thus our problem can be formulated as generating an information network  $G$  given an input sentence  $S$ .
Given an input sentence  $S$ , we first use a pretrained transformer-based AMR parser (Fernandez Astudillo et al., 2020) to obtain the AMR graph for  $S$ . We then use RoBERTa (Liu et al., 2019) to encode each sentence to identify entity mentions and event triggers as candidate nodes. After that, we map each candidate node to AMR nodes and enforce message passing using a GAT-based semantic graph aggregator to capture global inter-dependency between candidate nodes. All the candidate nodes and their pairwise edges are then passed through task-specific feed-forward neural networks to calculate score vectors. During decoding, we use the hierarchical structure in each AMR graph as a condition to decide the order in beam search and find the best candidate graph with the highest global score.
We employ a transformer based AMR parser (Fernandez Astudillo et al., 2020) pre-trained on AMR 3.0 annotations $^2$  to generate an AMR graph  $G^{a} = (V^{a},E^{a})$  with an alignment between AMR nodes and word spans in an input sentence  $S$ . Each node  $v_{i}^{a} = (m_{i}^{a},n_{i}^{a}) \in V^{a}$  represents an AMR concept or predicate, and we use  $m_{i}^{a}$  and  $n_{i}^{a}$  to denote the starting and ending indices of such a node in the original sentence. For AMR edges, we use  $e_{i,j}^{a}$  to denote the specific relation type between nodes  $v_{i}^{a}$  and  $v_{j}^{a}$  in AMR annotations.
Embeddings for AMR Relation Clusters To reduce the risk of over-fitting on hundreds of fine-grained AMR edge types, we only consider the edge types that are most relevant to IE tasks, and manually define  $M = 12$  clusters of AMR edge types as shown in Table 1. Note that each  $ARGx$
relation is considered as an individual cluster since each  $ARGx$  indicates a distinct argument role. For each edge type cluster, we randomly initialize a  $d_{E}$  dimensional embedding and obtain an embedding matrix  $\pmb{E} \in \mathbb{R}^{M \times d_{E}}$ , which will be optimized during the training process.
Categories	AMR relation types
Spatial	location, destination, path
Temporal	year, time, duration, decade, weekday
Means	instrument, manner, topic, medium
Modifiers	mod, poss
Operators	op-X
Prepositions	prep-X
Core Roles	ARG0, ARG1, ARG2, ARG3, ARG4
Others	Other AMR relation types.
Table 1: Manually defined AMR relation clusters for IE, where each  $ARGx$  is treated as an individual cluster.
3.2 Entity and Event Trigger Identification
We first identify the entity mentions and event triggers as candidate nodes from an input sentence. Similar to (Lin et al., 2020), we adopt feed forward neural networks constrained by conditional random fields (CRFs) to identify the word spans for entity mentions and event triggers.
Contextual Encoder Given an input sentence  $S = \{w_{1}, w_{2}, \dots, w_{N}\}$  of length  $N$ , we first calculate the contextual word representation  $x_{i}$  for each word  $w_{i}$  using a pre-trained RoBERTa encoder (Liu et al., 2019). If one word is split into multiple pieces by the RoBERTa tokenizer, we take the average of the representation vectors for all word pieces as the final word representation.
CRFs based Sequence Tagging After obtaining the contextual word representations, we use a feedforward neural network  $FFN$  to compute a score vector  $\hat{y}_i = FFN(\boldsymbol{x}_i)$  for each word, where each element in  $\hat{y}_i$  represents the score for a certain tag in the tag set<sup>3</sup>. The overall score for a tag path  $\hat{z} = \{\hat{z}_1, \hat{z}_2, \dots, \hat{z}_N\}$  is calculated by
$$
s (\hat {\pmb {z}}) = \sum_ {i = 1} ^ {N} \hat {y} _ {i, \hat {z} _ {i}} + \sum_ {i = 1} ^ {N + 1} P _ {\hat {z} _ {i - 1}, \hat {z} _ {i}},
$$
where  $\hat{y}_{i,\hat{z}_i}$  is the  $\hat{z}_i$ -th element of the score vector  $\hat{\pmb{y}}_i$ , and  $P_{\hat{z}_{i - 1},\hat{z}_i}$  denotes the transition score from tag  $\hat{z}_{i - 1}$  to  $\hat{z}_i$  from an estimizable matrix  $\pmb{P}$ . Similar to (Chiu and Nichols, 2016), the training
$^{2}$ https://catalog.ldc.upenn.edu/LDC2020T02
3We use BIO tagging scheme to tag word spans.
objective for node identification is to maximize the log-likelihood  $\mathcal{L}^I$  of the gold tag-path  $z$ .
$$
\mathcal {L} ^ {I} = s (\boldsymbol {z}) - \log \sum_ {\hat {\boldsymbol {z}} \in Z} e ^ {s (\hat {\boldsymbol {z}})} \tag {1}
$$
We use separate CRF-based taggers for entity and event trigger extraction. Note that we do not use the specific node types predicted by the CRF taggers as the final output classification results for entities and triggers, but only keep the identified entity and trigger spans. The final types of entities and triggers are jointly decided with relation and argument extraction in the subsequent decoding step. Specifically, we will obtain the collections of entity spans  $\{(a_i,b_i)\}_{i = 1}^{|\mathcal{E}|}$  and trigger spans  $\{(p_i,q_i)\}_{i = 1}^{|T|}$  during this step, where  $a_{i},b_{i},p_{i},q_{i}$  denote the starting and ending indices of the word spans.
3.3 Semantic Graph Aggregator
To make the best use of the shared semantic features and topological features from the AMR parsing for the input sentence, we design a semantic graph aggregator, which enables the candidate entity nodes and event nodes to aggregate information from their neighbors based on the AMR topology.
Initial Node Representation Each entity node, trigger node or AMR node is initialized with a vector representation  $h_i^0$  by averaging the word embeddings for all the words in their spans. For example, given an entity node  $(a_i, b_i)$ , its representation vector is calculated by
$$
\pmb {h} _ {i} ^ {0} = \frac {1}{| b _ {i} - a _ {i} + 1 |} \sum_ {k = a _ {i}} ^ {b _ {i}} \pmb {x} _ {k}
$$
where  $\pmb{x}_k$  is the word representation from the RoBERTa encoder.
Node Alignment We first try to align each identified entity node and trigger node to one of the AMR nodes before conducting message passing. Take an entity node with its span  $(a_{i},b_{i})$  as an example. Given the set of AMR nodes  $\{(m_i^a,n_i^a)\}_{i = 1}^{|V^a |}$ , we consider  $b_{i}$  as the index of the head word of the entity node, and aim to find  $(m_{i*}^a,n_{i*}^a)$  that covers  $b_{i}$  as the matched AMR node for  $(a_{i},b_{i})$ , that is, such a node satisfies  $m_{i*}^a\leqslant b_i\leqslant n_{i*}^a$ . If no nodes can be matched to  $(a_{i},b_{i})$  in this way, we turn to search for the nearest AMR node:
$$
i ^ {*} = \arg \min  _ {k} \left(| b _ {i} - m _ {k} | + | b _ {i} - n _ {k} |\right),
$$
Figure 2: An illustration for node alignment and message passing. For unmatched entity or trigger nodes, such as  $h_5^l$ , we add a new node with feature  $h_5^l$  and link this new node to the nearest node of  $h_5^l$ .
where  $(m_{i^*}^a, n_{i^*}^a)$  is the AMR node with the shortest distance to the entity node  $(a_i, b_i)$ . We also conduct alignment for event trigger nodes in the same way.
Heterogeneous Graph Construction After obtaining the matched or nearest AMR node for each identified entity mention and event trigger, we construct a heterogeneous graph with initialized node and edge features as follows. Given an AMR graph  $G^{a} = (V^{a},E^{a})$ , we consider the following three cases to initialize feature vectors for each node  $v_{i}^{a}$ :
- Node  $v_{i}^{a}$  has been matched to an entity mention or event trigger. We take the representation vector of the matched node (instead of  $v_{i}^{a}$ ) as the initialized feature vector.
- Node  $v_{i}^{a}$  is not matched to any identified nodes but labeled as the nearest node for an entity mention or event trigger, e.g.,  $(a_{i}, b_{i})$ . We add a new node in the AMR topology with the representation vector of  $(a_{i}, b_{i})$ , and link this new node from  $v_{i}^{a}$  with an edge type Others defined in Table 1.
- Node  $v_{i}^{a}$  is neither matched nor acted as the nearest node to any entities (triggers). We use its own node representation as the initialized feature vector.
For each edge  $e_{i,j}^{a}$ , we first map it to an AMR relation cluster according to Table 1 and then look up for its representation  $e_{i,j}$  from the embedding matrix  $\mathbf{E}$ . We use  $h_{i}^{0}$  to represent the initial feature for each node. An illustration for this step is shown in Figure 2.
Attention Based Message Passing Inspired from Graph Attention Networks (GATs) (Velickovic et al., 2018), we design an  $L$ -layer attention based message passing mechanism on an AMR graph topology to enable the entity and trigger
nodes to aggregate neighbor information. For the node  $i$  in layer  $l$ , we first calculate the attention score for each neighbor  $j \in \mathcal{N}_i$  based on node features  $h_i^l, h_j^l$  and edge features  $e_{i,j}^l$ .
$$
\alpha_ {i, j} ^ {l} = \frac {\exp \left(\sigma \left(f ^ {l} [ \mathbf {W h} _ {i} ^ {l} : \mathbf {W} _ {e} \boldsymbol {e} _ {i , j} : \mathbf {W h} _ {j} ^ {l} ]\right)\right)}{\sum_ {k \in \mathcal {N} _ {i}} \exp \left(\sigma \left(f ^ {l} [ \mathbf {W h} _ {i} ^ {l} : \mathbf {W} _ {e} \boldsymbol {e} _ {i , k} : \mathbf {W h} _ {k} ^ {l} ]\right)\right)}
$$
where  $\mathbf{W}$ ,  $\mathbf{W}_e$  are trainable parameters, and  $f^l$  and  $\sigma(\cdot)$  are a single layer feed-forward neural network and LeakyReLU activation function respectively. Then the neighborhood information  $h^*$  can be calculated by the weighted sum of neighbor features.
$$
\boldsymbol {h} ^ {*} = \sum_ {j \in \mathcal {N} _ {i}} \alpha_ {i, j} ^ {l} \boldsymbol {h} _ {i} ^ {l}
$$
The updated node feature is calculated by a combination of the original node feature and its neighborhood information, where  $\gamma$  controls the level of message passing between neighbors, and  $\mathbf{W}^*$  denotes a trainable linear transformation parameter.
$$
\boldsymbol {h} ^ {l + 1} = \boldsymbol {h} ^ {l} + \gamma \cdot \mathbf {W} ^ {*} \boldsymbol {h} ^ {*} \tag {2}
$$
We select the entity and trigger nodes from the graph and take their feature vectors  $\boldsymbol{h}_i^L$  from the final layer as the representation vectors that have aggregated information from the AMR graph (as Fig. 2 illustrates). We use  $\boldsymbol{h}_i^e$  and  $\boldsymbol{h}_i^t$  to denote the features of each entity and trigger respectively.
3.4 Model Training and Decoding
In this subsection, we introduce how we jointly decode the output information network given the identified entity and trigger nodes with their aggregated features  $h_i^e$  and  $h_i^t$ . We design a hierarchical decoding method that incorporates the AMR hierarchy as a condition to decide a more organized order for decoding knowledge elements.
Maximizing Scores with Global Features Similar to OneIE (Lin et al., 2020), we use task-specific feed-forward neural networks to map each node or node pair into a score vector. Specifically, we calculate four types of score vectors  $s_i^e$ ,  $s_i^t$ ,  $s_{i,j}^r$  and  $s_{i,j}^{a}$  for entity, trigger, relation, and argument role extraction tasks respectively, where the dimension of each score vector is identical to the number of classes in each task.
$$
\begin{array}{l} \boldsymbol {s} _ {i} ^ {e} = F F N ^ {e} (\boldsymbol {h} _ {i} ^ {e}), \quad \boldsymbol {s} _ {i} ^ {t} = F F N ^ {t} (\boldsymbol {h} _ {i} ^ {t}), \\ \boldsymbol {s} _ {i, j} ^ {r} = F F N ^ {r} \left(\left[ \boldsymbol {h} _ {i} ^ {e}: \boldsymbol {h} _ {j} ^ {e} \right]\right), \\ \boldsymbol {s} _ {i, j} ^ {a} = F F N ^ {a} \left(\left[ \boldsymbol {h} _ {i} ^ {t}: \boldsymbol {h} _ {j} ^ {e} \right]\right). \\ \end{array}
$$
Therefore, the total score  $c(G)$  is formulated as
$$
c (G) = \sum_ {i = 1} ^ {| \mathcal {E} |} \pmb {s} _ {i} ^ {e} + \sum_ {i = 1} ^ {| \mathcal {T} |} \pmb {s} _ {i} ^ {t} + \sum_ {i = 1} ^ {| \mathcal {E} |} \sum_ {i = 1} ^ {| \mathcal {E} |} \pmb {s} _ {i, j} ^ {r} + \sum_ {i = 1} ^ {| \mathcal {T} |} \sum_ {i = 1} ^ {| \mathcal {E} |} \pmb {s} _ {i, j} ^ {a}.
$$
We inherit the approach of using global features in OneIE (Lin et al., 2020) to enforce the model to capture more information on global interactions. The global score  $g(G)$  for an information network  $G$  is defined as the sum of local score  $c(G)$  and the contribution of global features  $f_{G}$ .
$$
g (G) = c (G) + \boldsymbol {u} \cdot \boldsymbol {f} _ {G} \tag {3}
$$
where  $\pmb{u}$  is a trainable parameter. The global feature vector  $f_{G}$  is composed of binary values indicating whether the output graph possesses some interdependencies among knowledge elements (e.g., an attacker is likely to be a person being arrested). We use the global feature categories identical to (Lin et al., 2020) during training, and the overall training objective is to maximize the identification log-likelihood, the local score  $s(G)$  while minimizing the gap on the global score between ground-truth  $\hat{G}$  and predicted information network  $G$ .
$$
\max  \mathcal {L} ^ {I} + c (\hat {G}) - (g (\hat {G}) - g (G)).
$$
Hierarchical Ordered Decoding Given the output score vectors for all nodes and their pairwise edges, the most straightforward way is to output an information network  $G$  with the highest global score  $g(G)$ . Due to the utilization of global features, searching through all possible information networks could incur exponential complexity, thus we take a similar approach based on beam search used in (Lin et al., 2020). Compared with OneIE (Lin et al., 2020), we creatively incorporate the AMR hierarchy to decide a more organized decoding order instead of a simple left-to-right order based on the word positions in the original sentence. Specifically, given the nodes and their alignments with AMR, we sort up these nodes according to the positions of their aligned AMR nodes in a top-to-down manner, that is, the aligned AMR node which is nearest to the AMR root node needs to be decoded first. We illustrate the decoding order in Fig. 3 using an example. We use  $\mathcal{U} = \{v_1, v_2, \dots, v_K\}$  to denote the sorted identified trigger and entity nodes, and similar to (Lin et al., 2020), we add these nodes step by step from  $v_1$  to  $v_K$ , and in each step, we obtain all possible subgraphs by enumerating the types of the new
Figure 3: An illustration of ordered decoding, where  $\tau_{1}$  and  $\tau_{2}$  are identified triggers while each  $\varepsilon_{i,j}$  is identified entity. In this example, the order of beam search decoding is:  $\tau_{1},\tau_{2},\varepsilon_{1,1},\varepsilon_{2,1},\varepsilon_{1,2},\varepsilon_{2,2},\varepsilon_{2,3}$ .
node and pairwise edges with other existing nodes. We only keep the top  $\theta$  subgraphs in each step as candidate graphs to avoid exponential complexity before finally select the graph with the highest global score  $g(G)$  in step  $K$  as the output.
ACE-2005 Automatic Content Extraction (ACE) 2005 dataset<sup>4</sup> provides fine-grained annotations for entity, relation, and event extraction. We use the same preprocessing and data split as in OneIE (Lin et al., 2020) and DyGIE++ (Wadden et al., 2019) to obtain the ACE05-E corpus with 18,927 sentences. Following (Lin et al., 2020), we keep 7 entity types, 6 relation types, 33 event types, and 22 event argument roles.