SlowGuess/ACL_2020_OCR huggingface dataset:
Amir Pouran Ben Veyseh<sup>1</sup>, Franck Dernoncourt<sup>2</sup>, Dejing Dou<sup>1</sup> and Thien Huu Nguyen<sup>1,3</sup>
$^{1}$ Department of Computer and Information Science, University of Oregon, Eugene, Oregon, USA
$^{2}$ Adobe Research, San Jose, CA, USA
$^{3}$ VinAI Research, Hanoi, Vietnam
{apouranb, dou, thien}@cs.uoregon.edu
franck.dernoncourt@adobe.com
This paper studies the task of Relation Extraction (RE) that aims to identify the semantic relations between two entity mentions in text. In the deep learning models for RE, it has been beneficial to incorporate the syntactic structures from the dependency trees of the input sentences. In such models, the dependency trees are often used to directly structure the network architectures or to obtain the dependency relations between the word pairs to inject the syntactic information into the models via multi-task learning. The major problems with these approaches are the lack of generalization beyond the syntactic structures in the training data or the failure to capture the syntactic importance of the words for RE. In order to overcome these issues, we propose a novel deep learning model for RE that uses the dependency trees to extract the syntax-based importance scores for the words, serving as a tree representation to introduce syntactic information into the models with greater generalization. In particular, we leverage Ordered-Neuron Long-Short Term Memory Networks (ON-LSTM) to infer the model-based importance scores for RE for every word in the sentences that are then regulated to be consistent with the syntax-based scores to enable syntactic information injection. We perform extensive experiments to demonstrate the effectiveness of the proposed method, leading to the state-of-the-art performance on three RE benchmark datasets.
One of the fundamental tasks in Information Extraction (IE) is Relation Extraction (RE) where the goal is to find the semantic relationships between two entity mentions in text. Due to its importance, RE has been studied extensively in the literature. The recent studies on RE has focused on deep learning to develop methods to automatically induce sen
tence representations from data (Zeng et al., 2014; Nguyen and Grishman, 2015a; Verga et al., 2018). A notable insight in these recent studies is that the syntactic trees of the input sentences (i.e., the dependency trees) can provide effective information for the deep learning models, leading to the state-of-the-art performance for RE recently (Xu et al., 2015; Guo et al., 2019; Tran et al., 2019). In particular, the previous deep learning models for RE has mostly exploited the syntactic trees to structure the network architectures according to the word connections presented in the trees (e.g., performing Graph Convolutional Neural Networks (GCN) over the dependency trees (Zhang et al., 2018)). Unfortunately, these models might not be able to generalize well as the tree structures of the training data might significantly differ from those in the test data (i.e., the models are overfit to the syntactic structures in the training data). For instance, in the cross-domain setting for RE, the domains for the training data and test data are dissimilar, often leading to a mismatch between the syntactic structures of the training data and test data. In order to overcome this issue, the overall strategy is to obtain a more general representation of the syntactic trees that can be used to inject the syntactic information into the deep learning models to achieve better generalization for RE.
A general tree representation for RE is presented in (Veyseh et al., 2019) where the dependency trees are broken down into their sets of dependency relations (i.e., the edges) between the words in the sentences (called the edge-based representation). These dependency relations are then used in a multitask learning framework for RE that simultaneously predicts both the relation between the two entity mentions and the dependency connections between the pairs of words in the input sentences. Although the dependency connections might be less specific to the training data than the whole tree structures,
8021
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8021-8032 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
the major limitation of the edge-based representation is that it only captures the pairwise (local) connections between the words and completely ignores the overall (global) importance of the words in the sentences for the RE problem. In particular, some words in a given sentence might involve more useful information for relation prediction in RE than the other words, and the dependency tree for this sentence can help to better identify those important words and assign higher importance scores for them (e.g., choosing the words along the shortest dependency paths between the two entity mentions). We expect that introducing such importance information for the words in the deep learning models might lead to improved performance for RE. Consequently, in this work, we propose to obtain an importance score for each word in the sentences from the dependency trees (called the syntax-based importance scores). These will serve as the general tree representation to incorporate the syntactic information into the deep learning models for RE.
How can we employ the syntax-based importance scores in the deep learning models for RE? In this work, we first use the representation vectors for the words from the deep learning models to compute another importance score for each word (called the model-based importance scores). These model-based importance scores are expected to quantify the semantic information that a word contributes to successfully predict the relationship between the input entity mentions. Afterward, we propose to inject the syntax-based importance scores into the deep learning models for RE by enforcing that the model-based importance scores are consistent with the syntactic counterparts (i.e., via the KL divergence). The motivation of the consistency enforcement is to promote the importance scores as the bridge through which the syntactic information can be transmitted to enrich the representation vectors in the deep learning models for RE.
In order to implement this idea, we employ the Ordered-Neuron Long Short-Term Memory Networks (ON-LSTM) (Shen et al., 2019) to compute the model-based importance scores for the words in the sentences for RE. ON-LSTM extends the popular Long Short-Term Memory Networks (LSTM) by introducing two additional gates (i.e., the master forget and input gates) in the hidden vector computation. These new gates controls how long each neuron in the hidden vectors should be activated across different time steps (words) in the sentence
(i.e., higher-order neurons would be maintained for a longer time). Based on such controlled neurons, the model-based importance score for a word can be determined by the number of active neurons that the word possesses in the operation of ON-LSTM. To our knowledge, this is the first time ON-LSTM is applied for RE in the literature.
One of the issues in the original ON-LSTM is that the master gates and the model-based importance score for each word are only conditioned on the word itself and the left context encoded in the previous hidden state. However, in order to infer the importance for a word in the overall sentence effectively, it is crucial to have a view over the entire sentence (i.e., including the context words on the right). To this end, instead of relying only on the current word, we propose to obtain an overall representation of the sentence that is used as the input to compute the master gates and the importance score for each word in the sentence. This would enrich the model-based importance scores with the context from the entire input sentences, potentially leading to the improved RE performance of the model in this work.
Finally, to further improve the representations learned by the deep learning models for RE, we introduce a new inductive bias to promote the similarity between the representation vectors for the overall sentences and the words along the shortest dependency paths between the two entity mentions. The intuition is that the relation between the two entity mentions of interest in a sentence for RE can be inferred from either the entire sentence or the shortest dependency path between the two entity mentions (due to the demonstrated ability of the shortest dependency path to capture the important context words for RE in the prior work (Bunescu and Mooney, 2005)). We thus expect that the representation vectors for the sentence and the dependency path should be similar (as both capture the semantic relation) and explicitly exploiting such similarity can help the models to induce more effective representations for RE. Our extensive experiments on three benchmark datasets (i.e., ACE 2005, SPOUSE and SciERC) demonstrate the effectiveness of the proposed model for RE, leading to the state-of-the-art performance for these datasets.
RE has been traditionally solved by the feature-based or kernel-based approaches (Zelenko et al.,
8022
2003; Zhou et al., 2005; Bunescu and Mooney, 2005; Sun et al., 2011; Chan and Roth, 2010; Nguyen and Grishman, 2014; Nguyen et al., 2015c). One of the issues in these approaches is the requirement for extensive feature or kernel engineering effort that hinder the generalization and applicability of the RE models. Recently, deep learning has been applied to address these problems for the traditional RE approaches, producing the state-of-the-art performance for RE. The typical network architectures for RE include the Convolutional Neural Networks (Zeng et al., 2014; Nguyen and Grishman, 2015a; dos Santos et al., 2015; Wang et al., 2016), Recurrent Neural Networks (Nguyen and Grishman, 2016; Zhou et al., 2016; Zhang et al., 2017; Nguyen et al., 2019a), and self-attention in Transformer (Verga et al., 2018). The syntactic information from the dependency trees has also been shown to be useful for the deep learning models for RE (Tai et al., 2015; Xu et al., 2015; Liu et al., 2015; Miwa and Bansal, 2016; Peng et al., 2017; Zhang et al., 2018; Guo et al., 2019; Tran et al., 2019; Song et al., 2019; Veyseh et al., 2019). However, these methods tend to poorly generalize to new syntactic structures due to the direct reliance on the syntactic trees (e.g., in different domains) or fail to exploit the syntax-based importance of the words for RE due to the sole focus on edges of the dependency trees (Veyseh et al., 2019).
The RE problem can be formulated as a multi-class classification problem. Formally, given an input sentence  $W = w_{1},w_{2},\ldots ,w_{N}$  where  $w_{t}$  is the  $t$ -th word in the sentence  $W$  of length  $N$ , and two entity mentions of interest at indexes  $s$  and  $o$  ( $1 \leq s < o \leq N$ ), our goal is to predict the semantic relation between  $w_{s}$  and  $w_{o}$  in  $W$ .
Similar to the previous work on deep learning for RE (Shi et al., 2018; Veyseh et al., 2019), we first transform each word  $w_{t}$  into a representation vector  $x_{t}$  using the concatenation of the three following vectors: (i) the pre-trained word embeddings of  $w_{t}$ , (ii) the position embedding vectors (to encode the relative distances of  $w_{t}$  to the two entity mentions of interest  $w_{s}$  and  $w_{o}$  (i.e.,  $t - s$  and  $t - o$ ), and (iii) the entity type embeddings (i.e., the embeddings of the BIO labels for the words to capture the entity mentions present in  $X$ ). This word-to-vector transformation converts the input sentence  $W$  into a sequence of representation vec
tors  $X = x_{1},x_{2},\ldots ,x_{N}$  to be consumed by the next neural computations of the proposed model.
There are three major components in the RE model in this work, namely (1) the CEON-LSTM component (i.e., context-enriched ON-LSTM) to compute the model-based importance scores of the words  $w_{t}$ , (2) the syntax-model consistency component to enforce the similarity between the syntax-based and model-based importance scores, and (3) the similarity component between the representation vectors of the overall sentence and the shortest dependency path.
3.1 CEON-LSTM
The goal of this component is to obtain a score for each word  $w_{t}$  that indicates the contextual importance of  $w_{t}$  with respect to the relation prediction between  $w_{s}$  and  $w_{o}$  in  $W$ . In this section, we first describe the ON-LSTM model to achieve these importance scores (i.e., the model-based scores). A new model (called CEON-LSTM) that integrates the representation of the entire sentence into the cells of ON-LSTM will be presented afterward.
ON-LSTM: Long-short Term Memory Networks (LSTM) (Hochreiter and Schmidhuber, 1997) has been widely used in Natural Language Processing (NLP) due to its natural mechanism to obtain the abstract representations for a sequence of input vectors (Nguyen and Nguyen, 2018b, 2019). Given the input representation vector sequence  $X = x_{1},x_{2},\ldots ,x_{N}$ , LSTM produces a sequence of hidden vectors  $H = h_{1},h_{2},\ldots ,h_{N}$  using the following recurrent functions at the time step (word)  $w_{t}$  (assuming the zero vector for  $h_0$ ):
$$
f _ {t} = \sigma \left(W _ {f} x _ {t} + U _ {f} h _ {t - 1} + b _ {f}\right)
$$
$$
i _ {t} = \sigma \left(W _ {i} x _ {t} + U _ {i} h _ {t - 1} + b _ {i}\right)
$$
$$
o _ {t} = \sigma \left(W _ {o} x _ {t} + U _ {o} h _ {t - 1} + b _ {o}\right) \tag {1}
$$
$$
\hat {c} _ {t} = \tanh  \left(W _ {c} x _ {t} + U _ {c} h _ {t - 1} + b _ {c}\right)
$$
$$
c _ {t} = f _ {t} \circ c _ {t - 1} + i _ {t} \circ \hat {c} _ {t}
$$
$$
h _ {t} = o _ {t} \circ \operatorname {t a n h} \left(c _ {t}\right)
$$
where  $f_{t}, i_{t}$  and  $o_{t}$  are called the forget, input and output gates (respectively).
In order to compute the importance score for each word  $w_{t}$ , ON-LSTM introduce into the mechanism of LSTM two additional gates, i.e., the master forget gate  $\hat{f}_{t}$  and the master input gate  $\hat{i}_t$  (Shen et al., 2019). These gates are computed and inte
8023
grated into the LSTM cell as follow:
$$
\hat {f} _ {t} = c u m m a x (W _ {\hat {f}} x _ {t} + U _ {\hat {f}} h _ {t - 1} + b _ {\hat {f}})
$$
$$
\hat {i} _ {t} = 1 - c u m m a x (W _ {\hat {i}} x _ {t} + U _ {\hat {i}} h _ {t - 1} + b _ {\hat {i}})
$$
$$
\bar {f} _ {t} = \hat {f} _ {t} \circ \left(f _ {t} \hat {i} _ {t} + 1 - \hat {i} _ {t}\right) \tag {2}
$$
$$
\bar {i} _ {t} = \hat {i} _ {t} \circ \left(i _ {t} \hat {f} _ {t} + 1 - \hat {f} _ {t}\right)
$$
$$
c _ {t} = \bar {f} _ {t} \circ c _ {t - 1} + \bar {i} _ {t} \circ \hat {c} _ {t}
$$
where  $cummax$  is an activation function defined as  $cummax(x) = cumsum(softmax(x))^1$ .
The forget and input gates in LSTM (i.e.,  $f_{t}$  and  $i_{t}$ ) are different from the master forget and input gates in ON-LSTM (i.e.,  $\hat{f}_{t}$  and  $\hat{i}_{t}$ ) as the gates in LSTM assume that the neurons/dimensions in their hidden vectors are equally important and that these neurons are active at every step (word) in the sentence. This is in contrast to the master gates in ON-LSTM that impose a hierarchy over the neurons in the hidden vectors and limit the activity of the neurons to only a portion of the words in the sentence (i.e., higher-ranking neurons would be active for more words in the sentence). Such hierarchy and activity limitation are achieved via the function  $\text{cumax}(x)$  that aggregates the softmax output of the input vector  $x$  along the dimensions. The output of  $\text{cumax}(x)$  can be seen as the expectation of some binary vector of the form  $(0, \dots, 0, 1, \dots, 1)$  (i.e., involving two consecutive segments: the 0's segment and the 1's segment). At one step, the 1's segments in the gate vectors represent the neurons that are activated at that step. In ON-LSTM, a word  $w_{i}$  is more contextually important than another word  $w_{j}$  if the master gates for  $w_{i}$  have more active neurons than those for  $w_{j}$ . Consequently, in order to compute the importance score for the word  $w_{t}$ , we can rely on the number of active neurons in the master gates that can be estimated by the sum of the weights of the neurons in the master gates in ON-LSTM. Following (Shen et al., 2019), we employ the hidden vectors for the master forget gate in ON-LSTM to compute the importance scores for the words in this work. Specifically, let  $\hat{f}_{t} = \hat{f}_{t1}, \hat{f}_{t2}, \dots, \hat{f}_{tD}$  be the weights for the neurons/dimensions in  $\hat{h}_{t}$  (i.e.,  $D$  is the dimension of the gate vectors). The model-based importance score mod $t$  for the word  $w_{t} \in W$  is then obtained by:  $\text{mod}_t = 1 - \sum_{i=1..D} \hat{f}_{ti}$ . For convenience, we also use  $H = h_1, h_2, \dots, h_N$  to denote the hidden
$^1$  cumsum  $(u_{1},u_{2},\ldots ,u_{n}) = (u_{1}^{\prime},u_{2}^{\prime},\ldots ,u_{n}^{\prime})$  where  $u_{i}^{\prime} = \sum_{j = 1..i}u_{j}$
vectors returned from the application of ON-LSTM over the input representation vectors  $X$ .
Introducing Sentence Context into ON-LSTM
One limitation of the ON-LSTM model is that it only relies on the representation vector of the current word  $x_{t}$  and the hidden vector for the left context (encoded in  $h_{t-1}$ ) to compute the master gate vectors and the model-based important score for the word  $w_{t}$  as well. However, this score computation mechanism might not be sufficient for RE as the importance score for  $w_{t}$  might also depend on the context information on the right (e.g., the appearance of some word on the right might make  $w_{t}$  less important for the relation prediction between  $w_{s}$  and  $w_{o}$ ). Consequently, in this work, we propose to first obtain a representation vector  $x_{t}' = g(x_{1}, x_{2}, \ldots, x_{N})$  that has the context information about the entire sentence  $W$  (i.e., both the left and right context for the current word  $w_{t}$ ). Afterward,  $x_{t}'$  will replace the input representation vector  $x_{t}$  in the computation for the master gates and importance score at step  $t$  of ON-LSTM (i.e., in the formulas for  $\hat{f}_{t}$  and  $\hat{i}_{t}$  in Equation 2). In this way, the model-based importance score for  $w_{t}$  will be able to condition on the overall context in the input sentence.
In this work, we obtain the representation vector  $x_{t}^{\prime}$  for each step  $t$  of ON-LSTM based on the weighted sum of the transformed vectors of the input representation sequence  $x_{1}, x_{2}, \ldots, x_{N}$ :  $x_{t}^{\prime} = \sum_{i} \alpha_{ti} (W_{x} x_{i} + b_{x})$ . The weight  $\alpha_{ti}$  for the term with  $x_{i}$  in this formula is computed by:
$$
\alpha_ {t i} = \frac {\exp \left(\left(W _ {h} h _ {t - 1} + b _ {h}\right) \cdot \left(W _ {x} x _ {i} + b _ {x}\right)\right)}{\sum_ {j = 1} ^ {N} \exp \left(\left(W _ {h} h _ {t - 1} + b _ {h}\right) \cdot \left(W _ {x} x _ {j} + b _ {x}\right)\right)} \tag {3}
$$
where  $W_{h}, b_{h}, W_{x}$  and  $b_{x}$  are the learnable parameters. Note that in this formula, we use the ON-LSTM hidden vector  $h_{t-1}$  from the previous step as the query vector to compute the attention weight for each word. The rationale is to enrich the attention weights for the current step with the context information from the previous steps (i.e., encoded in  $h_{t-1}$ ), leading to the contextualized input representation  $x_{t}'$  with richer information for the master gates and importance score computations in ON-LSTM. The proposed ON-LSTM with the enriched input vectors  $x_{t}'$  is called CEON-LSTM (i.e., Context-Enriched ON-LSTM) in this work.
8024
3.2 Syntax-Model Consistency
As mentioned in the introduction, the role of the model-based importance scores obtained from CEON-LSTM is to serve as the bridge to inject the information from the syntactic structures of  $W$  into the representation vectors of the deep learning models for RE. In particular, we first leverage the dependency tree of  $W$  to obtain another importance score  $syn_{t}$  for each word  $w_{t} \in W$  (i.e., the syntax-based importance score). Similar to the model-based scores, the syntax-based scores are expected to measure the contextual importance of  $w_{t}$  with respect to the relation prediction for  $w_{s}$  and  $w_{o}$ . Afterward, we introduce a constraint to encourage the consistency between the model-based and syntax-based importance scores (i.e., mod  $t$  and  $syn_{t}$ ) for the words via minimizing the KL divergence  $L_{import}$  between the normalized scores:
$$
\overline {{m o d}} _ {1}, \ldots , \overline {{m o d}} _ {N} = s o f t m a x (m o d _ {1}, \ldots , m o d _ {N})
$$
$$
\overline {{s y n}} _ {1}, \dots , \overline {{s y n}} _ {N} = s o f t m a x (s y n _ {1}, \dots , s y n _ {N})
$$
$$
\mathcal {L} _ {\text {i m p o r t}} = - \Sigma_ {i} \overline {{\operatorname {m o d} _ {i}}} \log \frac {\overline {{\operatorname {m o d} _ {i}}}}{\overline {{\operatorname {s y n} _ {i}}}} \tag {4}
$$
The intuition is to exploit the consistency to supervise the model-based importance scores from the models with the syntax-based importance scores from the dependency trees. As the model-based importance scores are computed from the master gates with the active and inactive neurons in CEON-LSTM, this supervision allows the syntactic information to interfere directly with the internal computation/structure of the cells in CEON-LSTM, potentially generating representation vectors with better syntax-aware information for RE.
To obtain the syntax-based importance scores, we take the motivation from the previous work on RE where the shortest dependency paths between the two entity mentions of interest have been shown to capture many important context words for RE. Specifically, for the sentence  $W$ , we first retrieve the shortest dependency path  $DP$  between the two entity mentions  $w_{s}$  and  $w_{o}$  and the length  $T$  of the longest path between any pairs of words in the dependency tree of  $W$ . The syntax-based importance score  $syn_{t}$  for the word  $w_{t} \in W$  is then computed as the difference between  $T$  and the length of the shortest path between  $w_{t}$  and some word in  $DP$  in the dependency tree (i.e., the words along DP will have the score of  $T$ ). On the one hand, these
syntax-based importance scores are able to capture the importance of the words that is customized for the relation prediction between  $w_{s}$  and  $w_{o}$ . This is better suited for RE than the direct use of the edges in the dependency trees in (Veyseh et al., 2019) that is agnostic to the entity mentions of interest and fails to encode the importance of the words for RE. On the other hand, the syntax-based importance scores  $syn_{t}$  represent a relaxed form of the original dependency tree that might have a better chance to generalize over different data and domains for RE than the prior work (i.e., the ones that directly fit the models to the whole syntactic structures (Zhang et al., 2018) and run the risk of overfitting to the structures in the training data).
3.3 Sentence-Dependency Path Similarity
In this component, we seek to further improve the representation vectors in the proposed deep learning model for RE by introducing a novel constraint to maximize the similarity between the representation vectors for the overall input sentence  $W$  and the words along the shortest dependency path  $DP$  (i.e., inductive bias). The rationale for this bias is presented in the introduction.
In order to implement this idea, we first obtain the representation vectors  $R_W$  and  $R_{DP}$  for the sentence  $W$  and the words along  $DP$  (respectively) by applying the max-pooling operation over the CEON-LSTM hidden vectors  $h_1, h_2, \ldots, h_N$  for the words in  $W$  and  $DP$ :  $R_W = \max\_pooling_{w_i \in W} \{h_i\}$  and  $R_{DP} = \max\_pooling_{w_i \in DP} \{h_i\}$ . In the next step, we promote the similarity between  $R_W$  and  $R_{DP}$  by explicitly minimizing their negative cosine similarity², i.e., adding the following term  $L_{path}$  into the overall loss function:
$$
L _ {p a t h} = 1 - \cos \left(R _ {W}, R _ {D P}\right) \tag {5}
$$
3.4 Prediction