SlowGuess/NAACL_2022_OCR huggingface dataset:
Minh Van Nguyen<sup>1</sup>, Bonan Min<sup>2</sup>, Franck Dernoncourt<sup>3</sup>, and Thien Huu Nguyen<sup>1,4</sup>
$^{2}$  Raytheon BBN Technologies, USA
$^{3}$  Adobe Research, San Jose, CA, USA
4 VinAI Research, Vietnam
{minhnv, thien}@cs.uoregon.edu,
bonan.min@raytheon.com, dernonco@adobe.com
Event trigger detection, entity mention recognition, event argument extraction, and relation extraction are the four important tasks in information extraction that have been performed jointly (Joint Information Extraction - JointIE) to avoid error propagation and leverage dependencies between the task instances (i.e., event triggers, entity mentions, relations, and event arguments). However, previous JointIE models often assume heuristic manually-designed dependency between the task instances and meanfield factorization for the joint distribution of instance labels, thus unable to capture optimal dependencies among instances and labels to improve representation learning and IE performance. To overcome these limitations, we propose to induce a dependency graph among task instances from data to boost representation learning. To better capture dependencies between instance labels, we propose to directly estimate their joint distribution via Conditional Random Fields. Noise Contrastive Estimation is introduced to address the maximization of the intractable joint likelihood for model training. Finally, to improve the decoding with greedy or beam search in prior work, we present Simulated Annealing to better find the globally optimal assignment for instance labels at decoding time. Experimental results show that our proposed model outperforms previous models on multiple IE tasks across 5 datasets and 2 languages.
To extract structured information from unstructured text, a typical information extraction (IE) pipeline involves four major tasks: event trigger detection (ETD), event argument extraction (EAE), entity mention recognition (EMR), and relation extraction (RE). Previous work has performed such IE tasks via pipelined approaches (Li et al., 2013; Chen et al., 2015; Du and Cardie, 2020; Li et al., 2020), where a model for one task uses output predictions from other models performing other tasks.
Consequently, errors from the predictions can be propagated between the models in the pipeline.
Recently, ETD, EMR, EAE, and RE have been solved jointly in a single model, i.e., Joint Information Extraction - JointIE (Wadden et al., 2019; Lin et al., 2020; Nguyen et al., 2021a; Zhang and Ji, 2021), to avoid error propagation and leverage dependency between prediction instances of the four IE tasks (i.e., event trigger, entity mention, relation, and event argument candidates in a sentence). For example, if a Person entity mention is a Victim argument for a Die event, it is likely that the same entity mention is also a Target argument for an Attack event in the same sentence. To implicitly exploit instance dependency for representation learning, Wadden et al. (2019) and Lin et al. (2020) employ a shared encoder to obtain representation vectors to classify instances of different IE tasks. Later work heuristically captures dependency between IE task instances via explicitly connecting the task instances that share an entity mention or event trigger (Nguyen et al., 2021a) or aligning the task instances that share text spans with some nodes on a semantic graph (Zhang and Ji, 2021) to aid representation learning. While natural, these manual designs for dependency between task instances might not be optimal for representation learning of JointIE.
In addition to representation learning, at the prediction level, previous work tends to factorize the joint distribution of labels for all the task instances in JointIE into the product of label distributions for each individual instance (i.e., performing local normalization), thus hindering the ability to fully exploit the interactions of instance labels across IE tasks. (Lin et al., 2020) and (Zhang and Ji, 2021) mitigate this problem by decoding instance labels with handcrafted global features while (Nguyen et al., 2021a) focuses on encoding label interactions via consistency regularization over global type dependency graphs. However, these approaches still
Human Language Technologies, pages 4363 - 4374
Figure 1: Overview of the three stages in our proposed model: i) identifying task instances, ii) inducing instance dependency, and iii) joint modeling and decoding of instance labels. Each node represents an instance for one of the four IE tasks, and edges (with weights  $>0.3$ ) between nodes represent induced instance dependency.
assume a factorization of the joint label distribution for prediction instances, thus unable to fundamentally address the label dependency encoding issue. Recently, some works have attempted to directly model the joint distribution of instance labels by reformulating JointIE tasks as text generation problems using state-of-the-art pre-trained seq2seq models, e.g., BART or T5 (Lewis et al., 2020; Raffel et al., 2020). In such generative models, text spans and labels for task instances are generated by the decoder in an autoregressive fashion to encode label dependency for joint distribution computation (Lu et al., 2021; Hsu et al., 2021). Unfortunately, this approach needs to assume an order of the task instances to be decoded (e.g., from left to right) that disallows later instances in the order to interfere/correct predictions for earlier instances, causing suboptimal performance for JointIE.
In this work, we aim to overcome these issues by inducing dependency between the task instances for JointIE from data to boost representation learning, and directly modeling the joint distribution of the labels for all the task instances to fully enable label interactions. To this end, we consider each task instance as a node in a fully connected dependency graph; the weight for each edge is then learned to capture the dependency level between two corresponding instances. Note that this is different from prior work (Nguyen et al., 2021a; Zhang and Ji, 2021) that heuristically designs sparser dependency graphs with disconnected task instance pairs, thus failing to explore all possible interactions between instance pairs for optimal representations. In our method, the induced dependency graph for instance nodes is then employed by Graph Convolutional Networks (GCNs) (Kipf and Welling,
2017; Nguyen and Grishman, 2018) to enhance the representation for each instance node with information from all the other nodes according to their dependency levels. Afterwards, the enhanced instance representations and the induced dependency graph are utilized to estimate the joint distribution of instance labels via Conditional Random Fields (CRFs) (Lafferty et al., 2001). This formulation enables us to approximately maximize the intractable joint likelihood of the ground-truth instance labels via Noise Contrastive Estimation (NCE) (Gutmann and Hyvarinen, 2012), which converts the maximization problem into the nonlinear logistic regression discriminating between the true labels and the noise labels.
Finally, previous work for JointIE has employed a greedy or beam search for decoding instance labels, which is not optimal due to their greedy nature. In this work, we propose a novel decoding algorithm for JointIE via Simulated Annealing (SA) (Kirkpatrick et al., 1983), which has been shown to be able to approximate the global optimum of a function (Kirkpatrick et al., 1983; Van Laarhoven and Aarts, 1987). Experimental results show that our proposed model for JointIE significantly outperforms previous models on multiple tasks with large margins across 5 datasets and 2 languages.
Given an input sentence, ETD aims to predict text spans and event types for event triggers based on a predefined set of event types, e.g., "Attack" and "Transport" (Lai et al., 2020). Similarly, EMR seeks to determine text spans and entity types (e.g., "Person", "Organization") for entity mentions in the sentence (Nguyen et al., 2016b). Different from the
first two tasks, EAE and RE involves predictions for a pair of objects at a time. Given an event trigger and an entity mention, EAE aims to predict the argument role (e.g., "Victim") of the entity mention for the event trigger (Veyseh et al., 2020c). An argument role can be "Not-an-argument" indicating that the entity mention is not an argument for the trigger. For RE (Veyseh et al., 2020a,b), the task focuses on the classification of relation (e.g., "Work for") for a given pair of entity mentions. There is also a special type "No-relation" to specify no relation between two entity mentions. As such, we call the union set  $C$  of the predefined event types, entity types, argument roles, and relation types as the information types (excluding "Not-an-argument" and "No-relation").
To capture dependency among task instances for JointIE, an approach is to obtain all text spans for entity/event mention candidates along with their possible pairs to form the nodes for a dependency graph to improve representation learning. However, this approach will retain many text spans for non-entity/event mentions to introduce noise into the modeling. It will also entail a large dependency graph that can hinder the efficiency of the model. To this end, our model for JointIE first identifies text spans for entity mentions and event triggers. Afterwards, all possible pairs of entity-entity and entity-entity mentions are considered to identify positive pairs for event arguments and relations respectively. The detected entity mentions, event triggers, event arguments, and relations are called task instances that should be classified to obtain corresponding information types in  $C$ . In our model, a dependency graph among the detected task instances will be learned to provide inputs for GCNs to compute dependency-enhanced representations for the task instances. Finally, the enhanced representations will be used to compute a joint distribution over labels for all the task instances to train our model. We will also employ Simulated Annealing to achieve the global optimum for label assignment of the task instances in the decoding phase.
3.1 Identifying event and entity mentions
Given an input sentence  $\mathbf{w} = [w_1, \ldots, w_N]$  with  $N$  words, we identify its event triggers and entity mentions by solving two corresponding sequence
tagging problems for event and entity mentions. In particular, we use the BIO tagging schema to assign two labels to each word in  $\mathbf{w}$  to mark the text spans of event triggers and entity mentions, i.e.,  $\{\text{"B-TRIGGER", "I-TRIGGER", "O"}\}$  labels for event triggers, and  $\{\text{"B-ENTITY", "I-ENTITY", "O"}\}$  labels for entity mentions. The pre-trained transformer-based language model BERT (Devlin et al., 2019) is first utilized to obtain the contextualized embeddings for the words in the sentence:  $\mathbf{X} = \mathbf{x}_1, \ldots, \mathbf{x}_N = \mathrm{BERT}([w_1, \ldots, w_N])$ .
Next, the vector sequence  $\mathbf{X}$  is sent to two different CRF layers (Lafferty et al., 2001; Chiu and Nichols, 2016) to compute two distributions for the tag sequences of  $\mathbf{w}$  for event triggers and event mentions. The negative log-likelihoods  $L_{t}$  and  $L_{e}$  for golden trigger and entity tag sequences are then obtained to be included in the overall training loss. At test time, the Viterbi algorithm (Forney, 1973) is employed to determine the best tag sequences for event triggers and event mentions in  $\mathbf{w}$ .
Let  $V^t$  and  $V^e$  be the sets of text spans for event triggers and entity mentions respectively in  $\mathbf{w}$  (i.e., golden spans in the training time and predicted spans in the test time). To prepare for the next components, we compute the representations vectors  $\mathbf{z}_i^t$  and  $\mathbf{z}_j^e$  for each event trigger/instance  $t_i \in V^t$  and entity mention/instance  $e_j \in V^e$  respectively by averaging over the contextualized embeddings of the words inside the spans.
3.2 Identifying event arguments and relations
Given the detected event triggers and entity mentions, we obtain a representation vector  $\mathbf{z}_{ij}^{a}$  for each pair of event-entity mentions  $a_{ij} = (t_i,e_j)$  (i.e.,  $t_i\in V^t$ $e_j\in V^e$ ), and a representation vector  $\mathbf{z}_{ij}^{r}$  for each pair of entity-entity mentions  $r_{ij} = (e_i,e_j)$  (i.e.,  $e_i,e_j\in V^e$ ) via:
$$
\begin{array}{c} \mathbf {z} _ {i j} ^ {a} = F F N _ {a} ^ {d o w n} (\operatorname {c o n c a t} (\mathbf {z} _ {i} ^ {t}, \mathbf {z} _ {j} ^ {e})) \text {a n d} \mathbf {z} _ {i j} ^ {r} = \\ F F N _ {r} ^ {d o w n} (\operatorname {c o n c a t} (\mathbf {z} _ {i} ^ {e}, \mathbf {z} _ {j} ^ {e})). \end{array}
$$
Here, we use the feed-forward networks  $FFN_{a}^{down}$  and  $FFN_{r}^{down}$  to make sure that  $\mathbf{z}_i^t$ ,  $\mathbf{z}_j^e$ ,  $\mathbf{z}_{ij}^a$ , and  $\mathbf{z}_{ij}^r$  have the same dimensionality. Next, the pair representation vectors  $\mathbf{z}_{ij}^a$  and  $\mathbf{z}_{ij}^r$  are sent into two different feed-forward networks followed by sigmoid activations to compute the possibilities for being positive examples for event arguments and relations of  $a_{ij}$  and  $r_{ij}$  respectively:  $p_{ij}^a = \sigma (\mathrm{FFN}^a (\mathbf{z}_{ij}^a))$ , and  $p_{ij}^r = \sigma (\mathrm{FFN}^r (\mathbf{z}_{ij}^r))$ . Here,  $p_{ij}^a \in (0,1)$  is the probability for the entity mention  $e_j$  being an actual argument for the
event trigger  $t_i$  while  $p_{ij}^r \in (0,1)$  is the likelihood that there exists a relation of interest between the entity mentions  $e_i$  and  $e_j$ . At training time, we obtain the negative log-likelihoods  $L_a$  and  $L_r$  for the golden event argument and relation identification to be included in the overall loss function for minimization. At test time, the event-entity pair  $a_{ij}$  and entity-entity pair  $r_{ij}$  are retained as positive examples for event arguments and relations if their likelihoods  $p_{ij}^a$  and  $p_{ij}^r$  are greater than 0.5.
For convenience, let  $V^a$  and  $V^r$  be the sets of positive event-entity pairs  $a_{ij}$  (called argument instances) and entity-entity pairs  $r_{ij}$  (called relation instances) respectively. Also, let  $V = V^t \cup V^e \cup V^a \cup V^r$  be the set of all detected event, entity, argument, and relation instances. For each instance  $v_i \in V$ , we will use  $\mathbf{v}_i$  for its corresponding instance representation (i.e., from  $\mathbf{z}_i^t$ ,  $\mathbf{z}_j^e$ ,  $\mathbf{z}_{ij}^a$ , or  $\mathbf{z}_{ij}^r$ ).
3.3 Inducing Instance Dependency
Given the detected event, entity, argument, and relation instances in  $V$ , it remains to predict the information types in  $C$  for the instances to solve JointIE. While it is possible to directly employ the instance representations  $\mathbf{v}_i$  for label prediction, our goal is to exploit instance dependency in IE to enhance the representation vector for one instance with the information from other instances to facilitate type prediction. In particular, using the instances  $v_i$  in  $V$  as the nodes in a dependency graph  $G$ , we aim to enrich instance representations by feeding them into a GCN model. As such, instead of assuming a heuristic manually-designed dependency graph among the instances as in previous work (Zhang and Ji, 2021; Nguyen et al., 2021a), we propose to automatically learn the dependency graph  $G$  for the instances in  $V$ . To this end, our dependency graph  $G$  is a fully connected graph among the nodes in  $V$  where a weight  $\alpha_{ij} \in (0,1)$  is learned for each edge to quantify the dependency between the instances  $v_i$  and  $v_j$  in  $V$ . In this work, we present two sources of information that can be used for determining the dependency between the task instances: (i) semantic and (ii) syntactic information.
Semantic Information: The semantic-based weight  $\alpha_{ij}^{sem}$  for the edge between  $v_{i}$  and  $v_{j}$  quantifies their relatedness/dependency based on semantic information, i.e., via the representation vectors  $\mathbf{v}_i$  and  $\mathbf{v}_j$ :  $\alpha_{ij}^{sem} = FFN^{sem}(\mathrm{concat}(\mathbf{v}_i,\mathbf{v}_i))$ . Here,  $FFN^{sem}$  is a feed-forward network with the sigmoid function in the end.
Syntactic Information: The syntax-based weight  $\alpha_{ij}^{syn}$  for the edge between  $v_{i}$  and  $v_{j}$  is computed in a similar way as  $\alpha_{ij}^{sem}$ . In particular, for each word  $w_{k} \in \mathbf{w}$ , we retrieve the dependency relation  $d_{k}$  between  $w_{k}$  and its governor in the dependency tree of  $\mathbf{w}$ , which is generated by the Trankit's dependency parser (Nguyen et al., 2021b). We then obtain the embedding  $\mathbf{m}_{k}$  of  $d_{k}$  for  $w_{k}$  by looking up the learnable dependency embedding matrix  $\mathbf{M}$ . Afterwards, the syntax-based representation vector  $u_{i}$  for the instance  $v_{i} \in V$  is computed via:  $u_{i} = \max\text{-pool}_{w_{k} \in SPAN_{v_{i}}}(\mathbf{m}_{k})$ . Here,  $SPAN_{v_{i}}$  involves the words in the corresponding text span of  $v_{i}$  in  $\mathbf{w}$  if  $v_{i}$  is an event trigger or entity mention instance. Otherwise,  $SPAN_{v_{i}}$  contains the words inside the text spans of the involving event triggers and entity mentions in the pair for  $v_{i}$ . As such, we compute the syntax-based dependency weight  $\alpha_{ij}^{syn}$  for  $v_{i}$  and  $v_{j}$  via:  $\alpha_{ij}^{syn} = FFN^{syn}(\text{concat}(\mathbf{u}_{i}, \mathbf{u}_{i}))$  where  $FFN^{syn}$  is also a feed-forward network with the sigmoid function in the end. Finally, we combine the semantic- and syntax-based weights to obtain the overall dependency weight  $\alpha_{ij}$  for  $v_{i}$  and  $v_{j}$  in  $V$ :  $\alpha_{ij} = (\alpha_{ij}^{sem} + \alpha_{ij}^{syn}) / 2$ .
3.4 Enhancing Representations with GCNs
To enhance the representation vectors for the instances  $v_{i} \in V$ , a GCN model with  $K$  layers is applied over the induced dependency graph  $G$  to compute richer representations for the instances:
$$
\mathbf {h} _ {i} ^ {k} = \operatorname {R e L U} \left(\frac {\sum_ {v _ {j} \in V} \alpha_ {i j} \mathbf {W} ^ {k} \mathbf {h} _ {j} ^ {k - 1} + \mathbf {b} ^ {k}}{\sum_ {v _ {j} \in V} \alpha_ {i j}}\right), 1 \leq k \leq K \tag {1}
$$
Here,  $h_i^k$  is the representation for the instance  $v_{i}$  at the  $k$ -th layer of the GCN  $(\mathbf{h}_i^0\equiv \mathbf{v}_i)$ , and  $\mathbf{W}^k,\mathbf{b}^k$  are trainable weight and bias for the layer.
In this way, representation information from all the other instances  $v_{j}$  ( $j \neq i$ ) will be incorporated into the enhanced representation vector for  $v_{i}$  according to their learned dependency weights. Finally, the last layer's representation  $\mathbf{h}_i^K \equiv \mathbf{h}_i$  (we omit  $K$  for simplicity) is used to compute the score vector  $\mathbf{s}_i \in \mathbb{R}^{|C|}$  for  $v_{i}$ , where  $\mathbf{s}_i[c]$  measure the possibility for  $v_{i}$  to have the  $c$ -th label in the label set  $C$ :  $\mathbf{s}_i = FFN^{score}(\mathbf{h}_i)$  ( $FFN^{score}$  is a scoring feed-forward network). The score vectors  $\mathbf{s}_i$  will later be used for modeling the joint distribution of the labels for all the instances in  $V$ .
3.5 Computing Joint Distribution of Labels
Let  $Y$  be the set of labels  $y_{i}$  for the instances  $v_{i}$  in  $V$ . To infer the labels for the instances in  $V$ , we
need to estimate the joint distribution  $P(Y|\mathbf{w},V)$ . In previous work (Wadden et al., 2019; Lin et al., 2020; Nguyen et al., 2021a; Zhang and Ji, 2021), JointIE methods mostly focus on learning representations for the task instances to compute a label distribution for each instance  $v_{i}$  for prediction:  $P(y_{i}|\mathbf{w},V)\coloneqq \mathrm{softmax}(\mathbf{s}_{i})$ . This practice essentially implies the following factorization for  $P(Y|\mathbf{w},V)$ :  $P(Y|\mathbf{w},V) = \prod_{y_i\in Y}P(y_i|\mathbf{w},V)$ . As a result, this factorization assumes the independence of the instance labels, thus unable to fully capture beneficial label dependency for IE tasks.
To address this issue, we directly estimate the joint distribution  $P(Y|\mathbf{w},V)$  so that the dependency between instance labels can be facilitated to improve prediction performance. To this end, we formulate the joint distribution  $P(Y|\mathbf{w},V)$  with Conditional Random Fields (Lafferty et al., 2001):
$$
P (Y | \mathbf {w}, V) = \frac {1}{Z (V)} \prod_ {\left(v _ {i}, v _ {j}\right)} \psi_ {i j} \left(y _ {i}, y _ {j}, V\right) \tag {2}
$$
where  $\psi_{ij}(y_i,y_j,V)$  is a positive potential function defined on the edge  $(v_{i},v_{j})$  of the dependency graph  $G$ , and  $Z(V) = \sum_{Y^{\prime}\in C_V}\prod_{(v_i,v_j)}\psi_{ij}(y_i',y_j',V)$  is the normalization term to make sure that  $P(Y|\mathbf{w},V)$  is a valid probability distribution ( $C_V$  is the set of all possible label assignments  $Y$  for the instances in  $V$ ). Considering the instance information, the instance dependency, and the label dependency, we propose the potential function as:
$$
\psi_ {i j} \left(y _ {i}, y _ {j}, V\right) := \exp \left(\mathbf {s} _ {i} \left[ y _ {i} \right] + \mathbf {s} _ {j} \left[ y _ {j} \right] + \alpha_ {i j} \pi_ {y _ {i} \leftrightarrow y _ {j}}\right) \tag {3}
$$
where  $\mathbf{s}_i[y_i]$  is the local score for instance  $v_{i}$  being assigned with the label  $y_{i}$ ,  $\alpha_{ij}$  is the induced dependency weight for the edge  $(v_{i},v_{j})$  in  $G$ , and  $\pi_{y_i\leftrightarrow y_j}$  is a learnable transition score indicating the dependency between the labels  $y_{i}$  and  $y_{j}$ . With this formulation, we can derive the joint distribution  $P(Y|\mathbf{w},V)$ :
$$
P (Y | \mathbf {w}, V) = \frac {\exp (s (Y))}{\sum_ {Y ^ {\prime} \in C _ {V}} \exp (s \left(Y ^ {\prime}\right))} \tag {4}
$$
$$
s (Y) = \gamma \sum_ {v _ {i} \in V} \mathbf {s} _ {i} [ y _ {i} ] + \sum_ {\left(v _ {i}, v _ {j}\right)} \alpha_ {i j} \pi_ {y _ {i} \leftrightarrow y _ {j}} \tag {5}
$$
is the global score for the label assignment/configuration  $Y$  of the instances.  $\gamma$  is a hyperparameter to balance the local and transition scores.
To train the model, we need to maximize the joint likelihood in Equation (4) for the
golden label configuration  $Y^{*}$ . However, this requires the computation of the normalization term  $\sum_{Y' \in C_V} \exp(s(Y'))$ , which is intractable. To overcome this issue, we employ Noise Contrastive Estimation (NCE) (Gutmann and Hyvarinen, 2012; Mikolov et al., 2013). NCE converts the maximization problem into the nonlinear logistic regression that discriminates between the golden label configurations and the noise label configurations. In particular, the maximization of  $P(Y^{*}|\mathbf{w}, V)$  is done with NCE via minimizing the contrastive loss:
$$
L _ {N C} = - \log \sigma \left(s \left(Y ^ {*}\right)\right) - \sum_ {n = 1} ^ {N _ {n o i}} \mathbb {E} _ {Y _ {n} ^ {\prime} \sim P _ {n o i}} \left[ \log \sigma \left(- s \left(Y _ {n} ^ {\prime}\right)\right) \right] \tag {6}
$$
where  $\sigma$  is the sigmoid function and  $N_{\text{noi}}$  is the number of noise configurations  $Y_{n}^{\prime}$  drawn from  $P_{\text{noi}}$ , assumed to be a uniform distribution. Intuitively, the minimization of  $L_{NC}$  increases the global score  $s(Y^{*})$  for the true label configuration  $Y^{*}$  while decreasing the global scores  $s(Y^{\prime})$  for the noise label configurations  $Y^{\prime}$  to appropriately train the model. To the end, the overall loss function to train our model is:  $L = L_{t} + L_{e} + L_{a} + L_{r} + L_{NC}$ .
Algorithm 1: Simulated Annealing Search
3.6 Joint Decoding via Simulated Annealing
At inference time, we need to search for the configuration  $\hat{Y}$  that has the highest global score  $s(\hat{Y})$  in  $C_V$ :  $\hat{Y} = \operatorname{argmax}_{Y' \in C_V} s(Y')$ . A brute-force search for  $\hat{Y}$  cannot be done as the search space  $C_V$  is exponentially large ( $|C_V| = |C||V|$ ). Previous work has made several attempts to deal with this issue. (Wadden et al., 2019) and (Nguyen et al., 2021a) simply perform greedy decoding for each