SlowGuess/NAACL_2022_OCR huggingface dataset:
Xinnian Liang $^{1*}$ , Shuangzhi Wu $^{2}$ , Mu Li $^{2}$  and Zhoujun Li $^{1\dagger}$
$^{1}$ State Key Lab of Software Development Environment, Beihang University, Beijing, China
2Tencent Cloud Xiaowei, Beijing, China
{xnliang, lizj}@buaa.edu.cn; frostwu@tencent.com, limugx@qq.com;
Relation extraction is a key task in Natural Language Processing (NLP), which aims to extract relations between entity pairs from given texts. Recently, relation extraction (RE) has achieved remarkable progress with the development of deep neural networks. Most existing research focuses on constructing explicit structured features using external knowledge such as knowledge graph and dependency tree. In this paper, we propose a novel method to extract multi-granularity features based solely on the original input sentences. We show that effective structured features can be attained even without external knowledge. Three kinds of features based on the input sentences are fully exploited, which are in entity mention level, segment level, and sentence level. All the three are jointly and hierarchically modeled. We evaluate our method on three public benchmarks: SemEval 2010 Task 8, Tacred, and Tacred Revisited. To verify the effectiveness, we apply our method to different encoders such as LSTM and BERT. Experimental results show that our method significantly outperforms existing state-of-the-art models that even use external knowledge. Extensive analyses demonstrate that the performance of our model is contributed by the capture of multi-granularity features and the model of their hierarchical structure. Code and data are available at https://github.com/xnliang98/sms.
Relation extraction (RE) is a fundamental task in Natural Language Processing (NLP), which aims to extract relations between entity pairs from given plain texts. RE is the cornerstone of many downstream NLP tasks, such as knowledge base construction (Ji and Grishman, 2011), question answering (Yu et al., 2017), and information extraction (Fader et al., 2011).
Most recent works focus on constructing explicit structured features using external knowledge such as knowledge graph, entity features and dependency tree. To infuse prior knowledge from existing knowledge graph, recent works (Peters et al., 2019a; Wang et al., 2020b,a) proposed some pretrain tasks to help model learn and select proper prior knowledge in the pre-training stage. Baldini Soares et al. (2019); Yamada et al. (2020); Peng et al. (2020) force model learning entity-related information via well-designed pre-train tasks. Zhang et al. (2018); Guo et al. (2019); Xue et al. (2020); Chen et al. (2020) encode dependency tree with graph neural network (Kipf and Welling, 2017) (GNN) to help RE models capture non-local syntactic relation. All of them achieve a remarkable performance via employing external information from different structured features.
However, they either need time-consuming pretraining with external knowledge or need an external tool to get a dependency tree which may introduce unnecessary noise. In this paper, we aim to attain effective structured features based solely on the original input sentences. To this end, we analyze previous typical works and find that three kinds of features mainly affect the performance of RE models, which are entity mention level<sup>1</sup>, segment<sup>2</sup> level and sentence level features. Sentence level and entity mention (Baldini Soares et al., 2019; Yamada et al., 2020; Peng et al., 2020) level features were widely used by previous works but segment level feature (Yu et al., 2019; Joshi et al., 2020) does not get as much attention as the previous two features. These three level features can provide different granularity information from input sentences for relation prediction (Chowdhury and Lavelli, 2012; Kim). However, recent works did not consider them at the same time and ignored
${}^{1}$  entity mentions contain the entity itself and co-references of it.
2 continuous words in sentence (n-gram)
*Contribution during internship at Tencent Inc.
Human Language Technologies, pages 5088 - 5098
Figure 1: An Example for relation extraction which shows the hierarchical structure between entity mention level and segment level features.
the structure and interactive of them.
We employ a simple example in Figure 1 to show the hierarchical and joint structure of the previous three granularities features. The hierarchical structure is between mention level and segment level features. This example gives sentence and entity pairs ("Steven Jobs", "Apple"). We can find that the relation "the_CEO_of" of given entity "Apple" and entity "Steven Jobs" is built upon the core segment "the CEO of" between the entity mention "it" (i.e. co-reference of entity "Apple") and entity "Steven Jobs". To extract relation from this example, RE models need to first capture mention level features of given entities and then catch core segment level feature "the CEO of" which is related to mention level features. Finally, RE models can easily predict the relation with previous two-level hierarchical features. Besides, sentence-level semantic features can assist RE models to predict the relation of examples without an explicit pattern of entity mentions and segments.
Following previous intuitive process, we propose a novel method which extracts multi-granularity features based solely on the original input sentences. Specifically, we design a hierarchical mention-aware segment attention, which employs a hierarchical attention mechanism to build association between entity mention level and segment level features. Besides, we employ a global semantic attention to get a deeper understanding of sentence level features from input sentence representation. Finally, we aggregate previously extracted multi-granularity features with a simple fully-connected layer to predict the relation.
To evaluate the effectiveness of our method, we combine our method with different text encoders (e.g. LSTM and BERT) and results show that our method can bring significant improvement for all of them. Compared with models without using external knowledge, SpanBERT with our method can
achieve a new state-of-the-art result on Semeval 2010 Task 8, Tacred and Tacred Revisited. It is deserved to mention that the performance of our model is very competitive with the state-of-the-art models, which employ large-scale extra training data or information. We also do many analyses to demonstrate that features from representation of input itself are enough for the sentence-level RE tasks and multi-granularity features with hierarchical structure are crucial for relation prediction.
The structure of our model and details of each component is shown in figure 2. We can see the overall architecture in the middle. It is divided into three components from bottom to top: 1) A text encoder which is employed to obtain text vector representations; 2) A multi-granularity hierarchical feature extractor which can exploit effective structured features from text representations; 3) A feature aggregation layer which aggregate previous multi-granularity features for relation prediction. In this section, we will introduce details of three components.
Firstly, we formalize the relation extraction task. Let  $\mathbf{x} = \{x_{1}, x_{2}, \ldots, x_{n}\}$  be a sequence of input tokens, where  $x_{0} = [\mathrm{CLS}]$  and  $x_{n} = [\mathrm{SEP}]$  are special start and end tokens for BERT-related encoders. Let  $s_{1} = (i,j)$  and  $s_{2} = (k,l)$  be pairs of entity indices. The indices in  $s_{1}$  and  $s_{2}$  delimit entities in  $\mathbf{x}$ :  $[x_{i},\dots,x_{j-1}]$  and  $[x_{k},\dots,x_{l-1}]$ . Our goal is to learn a function  $P(r) = f_{\theta}(\mathbf{x}, s_{1}, s_{2})$  where  $r \in \mathcal{R}$  indicates the relation between the entity pairs, which is marked by  $s_{1}$  and  $s_{2}$ .  $\mathcal{R}$  is a pre-defined relation set.
2.1 Encoder Layer
We first employ a text encoder (e.g. BERT) to map tokens in input sentences into vector representations which can be formalized by Equ. (1).
$$
\mathbf {H} = \{h _ {0}, \dots , h _ {n} \} = f _ {\text {e n c o d e r}} (x _ {0}, \dots , x _ {n}) \tag {1}
$$
Where  $\mathbf{H} = \{h_0,\dots ,h_n\}$  is the vector representation of input sentences.
Our work is built upon  $\mathbf{H}$  and does not need any external information. We employ a max-pooling operation to obtain shallow features of entity pairs and input sentences.  $h_{e_1} = \mathrm{Maxpooling}(h_{i:j})$  and  $h_{e_2} = \mathrm{Maxpooling}(h_{k:l})$  are the representations of entity pairs.  $h_g = \mathrm{Maxpooling}(\mathbf{H})$  is the vector representation of input sentences which contains global semantic information.
Figure 2: Middle: The structure of our proposed multi-granularity hierarchical feature extractor. Left: Details of global semantic attention (sentence level feature) and feature aggregation layer. Right: Details of mention attention (entity mention level feature) and mention-aware segment attention (segment level feature).
2.2 Multi-Granularity Hierarchical Feature Extractor
The multi-granularity hierarchical feature extractor is the core component of our method and it consists of three attention mechanisms for different granularity features extraction: 1) mention attention which is designed to entity mention features of given entity pairs; 2) mention-aware segment attention which is based on the entity mention features from previous mention attention and aim to extract core segment level feature which is related to entity mentions; 3) global semantic attention which focuses on the sentence level feature.
2.2.1 Mention Attention
The structure of mention attention is shown in the right bottom of Figure 2. To capture more information about given entity pairs from input sentences, we extract entity mention level features by modeling the co-references (mentions) of entities implicitly. We employ a mention attention to capture information about entity 1 and 2 respectively. Specifically, we can use the representation of an entity as a query to obtain the entity mention feature from  $\mathbf{H}$  by Equ. (2).
$$
h _ {e _ {1}} ^ {\prime} = \operatorname {S o f t m a x} \left(\frac {\mathbf {H} \cdot h _ {e _ {1}}}{\sqrt {d}}\right) \cdot \mathbf {H} \tag {2}
$$
$$
h _ {e _ {2}} ^ {\prime} = \operatorname {S o f t m a x} \left(\frac {\mathbf {H} \cdot h _ {e _ {2}}}{\sqrt {d}}\right) \cdot \mathbf {H}
$$
Where  $d$  is the dimension of vector representation and used to normalize vectors. Then,  $h_{e_1}'$  and  $h_{e_2}'$  model the mentions of given entity pairs implicitly and contain more entity semantic information than  $h_{e_1}$  and  $h_{e_2}$ .
2.2.2 Mention-Aware Segment Attention
The structure of mention-aware segment attention is shown in the right top of Figure 2. And the mention-aware segment attention is a hierarchical structure based on the entity mention features  $h_{e_1}'$  and  $h_{e_2}'$  from mention attention.
Before introducing mention-aware segments attention, we first introduce how to get the representations of segments. We employ convolutional neural networks (CNN) with different kernel sizes to obtain all n-gram segments in texts, which can effectively capture local n-gram information with Equ. (3).
$$
\mathbf {H} _ {t} = \operatorname {C N N} _ {t} (\mathbf {H}), t \in \{1, 2, 3 \} \tag {3}
$$
Where  $t$  is the kernel size of CNN and is empirically set as  $t \in \{1,2,3\}$  which means extract 1-gram, 2-gram, and 3-gram segment level features.
Intuitively, the valuable segments should be highly related to given entity pairs, which can help the model to decide the relation of given entity pairs. Entity mention features  $h_{e_1}'$  and  $h_{e_2}'$  contain comprehensive information of given entity pairs
and  $\mathbf{H}_t$  contain 1,2,3-gram segment level features. We can extract mention-aware segment level features by simply linking them with attention mechanisms by Equ. (4).
$$
h _ {m} ^ {t} = \operatorname {S o f t m a x} \left(\frac {\mathbf {H} _ {t} \cdot \left(W _ {m} \left[ h _ {e _ {1}} ^ {\prime} ; h _ {e _ {2}} ^ {\prime} \right]\right)}{\sqrt {d}}\right) \cdot \mathbf {H} _ {t} \tag {4}
$$
Then, we get  $\{h_m^t\}_{t = 1,2,3}$  which capture different granularity segments features.
2.2.3 Global Semantic Attention
The structure of global semantic attention is shown in the left bottom of Figure 2. Previous works always directly concatenate vector representation  $[h_{e_1}; h_{e_2}; h_g]$  as the global semantic feature of input text. We argue this is not enough to help model capture deeper sentence level semantic information for RE. Different from them, to obtain better global sentence-level semantic feature, we employ an attention operation called global semantic attention which use the concatenation of  $[h_{e_1}; h_{e_2}; h_g]$  as query to capture deeper semantic feature from context representation  $\mathbf{H}$  by Equ. (5).
$$
h _ {s} = \operatorname {S o f t m a x} \left(\frac {\mathbf {H} \cdot \left(W _ {s} \left[ h _ {e 1} ; h _ {e 2} ; h _ {g} \right]\right)}{\sqrt {d}}\right) \cdot \mathbf {H} \tag {5}
$$
Where  $W_{s} \in R^{d \times 3d}$  is a linear transform matrix, and  $d$  is a hidden dimension of vectors. The concatenation of  $[h_{e_1}; h_{e_2}; h_g]$  is used as a query of the attention operation, which can force the extracted global semantic representation  $h_{s}$  contain entity mention related sentence level feature.
2.3 Feature Aggregation Layer
The structure of the feature aggregation layer is shown in the left top of Figure 2. We aggregate previous multi-granularity features by Equ. (6).
$$
h _ {o} = \operatorname {R e L U} \left(W _ {a} \left[ h _ {s}; h _ {e _ {1}} ^ {\prime}; h _ {e _ {1}} ^ {\prime}; h _ {m} ^ {1};; h _ {m} ^ {2};; h _ {m} ^ {3} \right]\right) \tag {6}
$$
Where  $W_{a} \in R^{6d \times d}$  is a linear transform matrix and ReLU is a nonlinear activation function.
2.4 Classification
Finally, we employ a softmax function to output the probability of each relation label as follows:
$$
P (r | \mathbf {x}, s _ {1}, s _ {2}) = \operatorname {S o f t m a x} \left(W _ {o} h _ {o}\right) \tag {7}
$$
The whole model is trained with cross entropy loss function. We call the multi-granularity hierarchical feature extractor: SMS (relation extraction with Sentence level, Mention level and mention-aware Segment level features).
Tacred	Semeval
lr	3e-5	2e-5
warmup steps	300	0
batch size	64	32
V100 GPU	4x	1x
epochs	4	10
max length	128	128
Table 1: Hyper-parameters used in training.
We evaluate the performance of our method on Semeval 2010 Task 8, Tacred and Tacred Revisited datasets.
SemEval 2010 Task 8 (Hendrickx et al., 2010) is a public dataset which contains 10,717 instances with 9 relations. The training/validation/test set contains 7,000/1,000/2,717 instances respectively.
$\mathbf{Tacred}^3$  is one of the largest, most widely used crowd-sourced datasets for Relation Extraction (RE), which is introduced by (Zhang et al., 2017), with 106,264 examples built over newswire and web text from the corpus used in the yearly TAC Knowledge Base Population (TAC KBP) challenges. The training/validation/test set contains 68,124/22,631/15,509 instances respectively. It covers 42 relation types including 41 relation types and a no Relation type and contains longer sentences with an average sentence length of 36.4.
Tacred Revisited<sup>4</sup> was proposed by (Alt et al., 2020) which aims to improve the accuracy and reliability of future RE method evaluations. They validate the most challenging 5K examples in the development and test sets using trained annotators and find that label errors account for  $8\%$  absolute F1 test error, and that more than  $50\%$  of the examples need to be relabeled. Then, they relabeled the test set and released the Tacred Revisited dataset.
The setting of hyper-parameters is shown in table 1. Following the implementation details mentioned in (Zhang et al., 2017), we employ the "entity mask" strategy and the "multi-channel" strategy during experiments. The former means replacing each subject entity (and object entity similarly) in the original sentence with a special [SUBJ-(NER)] token. All of our reported results are the mean of 5 results with different seeds, which are ran
<sup>3</sup>https://nlp.stanford.edu/projects/tacred/
<sup>4</sup>https://github.com/DFKI-NLP/tacrev
Model	Tacred	Tacred Revisited
P(ΔP)	R(ΔR)	F1(ΔF1)	P(ΔP)	R(ΔR)	F1(ΔF1)
1	LSTM	62.5	63.4	62.9	67.7	73.1	70.3
PA-LSTM*	65.7	64.5	65.1	73.6	72.8	73.2
SA-LSTM	68.1	65.7	66.9	78.3	72.5	75.4
C-GCN*	69.9	63.3	66.4	76.8	71.4	74.1
C-AGGCN*	71.9	63.4	67.5	78.2	70.5	74.3
2	TRE	-	-	67.4	-	-	75.3
BERT-base	68.1	67.7	67.9	69.4	75.8	72.6
BERT-large	69.2	69.4	69.3	75.1	74.8	75.0
BERT+LSTM	73.3	63.1	67.8	74.1	73.9	74.0
SpanBERT-base	67.6	68.6	68.1	69.1	78.2	73.7
SpanBERT-large	70.8	70.9	70.8	77.8	78.3	78.0
DG-SpanBERT-large*	71.4	71.6	71.5	79.2	78.6	78.9
3	MTB†	-	-	71.5	-	-	-
KnowBERT-W+W‡	71.6	71.4	71.5	79.0	79.6	79.3
KEPLER‡	70.4	73.0	71.7	-	-	-
K-Adapter‡	70.1	74.0	72.0	-	-	-
LUKE†	70.4	75.1	72.7	79.7	80.6	80.2
4	LSTM+SMS	72.8(+10.3)	64.6(1.2)	68.4(+5.5)	80.8(+3.1)	72.2(-0.9)	75.9(+5.6)
SpanBERT-base+SMS	72.6(+5.0)	68.4(-0.2)	70.5(+2.4)	79.1(+10.0)	77.7(-0.5)	78.3(+4.6)
SpanBERT-large+SMS	72.2(+1.4)	71.6(+0.7)	71.9(+1.1)	79.3(+1.5)	80.4(+2.1)	79.8(+1.8)
BERT-base+SMS	69.4(+1.3)	70.2(+2.5)	69.7(+1.8)	77.0(+7.6)	80.1(+4.3)	78.5(+5.9)
BERT-large+SMS	70.7(+1.5)	69.1(-0.3)	69.9(+0.6)	78.9(+3.8)	79.2(+4.4)	79.1(+4.1)
Table 2: Results on Tacred and Tacred Revisited. Bold means the best results in each block. Underline means the best results in block 1, 2, and 4. * means that the model employs dependency tree information. †means that the model employs extra training data to pre-train the model. ‡means the model employs knowledge graphs.
domly selected. We evaluate the models on Ta-cred with the official script $^5$  in terms of the Macro-F1 score and on Semeval with the official script semeval2010_task8 scorer-v1.2.pl.
When employing LSTM as the encoder, we employ a single-layer bidirectional LSTM with the hidden dimension size set to 200, we set dropout after the input layer and before the output layer with  $p = 0.5$ . We use stochastic gradient descent (SGD) with epochs of 30, learning rate of 1.0, decay weight of 0.5 and batch sizes of 50 to train the model. The latter is to augment the input by concatenating it with part-of-speech (POS) and named entity recognition (NER) embeddings. Glove (Pennington et al., 2014) embedding with 300-dimension is used for initializing word embedding layers in LSTM+SMS. NER embedding, POS embedding and position embedding are randomly initialized with 30-dimension vectors from uniform distribution.
3.3 Comparison Models
We mainly compare with models which are based on pre-trained language models (e.g. BERT). We reproduce the results of BERT and SpanBERT to evaluate the improvement of our method. We also compared other models with pre-trained lan
guage models. TRE (Alt et al., 2019), which uses the unidirectional OpenAI Generative PreTrained Transformer (GPT) (Radford et al., 2019). BERT-LSTM (Shi and Lin, 2019), which stacks bidirectional LSTM layer to BERT encoder. DGSpanBERT, which replaced the encoder of CGCN (Zhang et al., 2018) with SpanBERT and achieved the new state-of-the-art result without extra training data. MTB (Baldini Soares et al., 2019), which incorporates an intermediate "matching the blanks" pre-training on the entity-linked text based on English Wikipedia. KnowBERT-W+W (Peters et al., 2019b), which is an an advanced version of KnowBERT. KEPLER (Wang et al., 2020b), which integrates factual knowledge with the supervision of the knowledge embedding objective. K-Adapter (Wang et al., 2020a), which consists of a RoBERTa model and an adapter to select adaptive knowledge. LUKE (Yamada et al., 2020), which is trained with a new pre-training task which involves predicting randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia and introduce a new entity-aware attention mechanism.
In order to further prove the effectiveness of our SMS, we use bi-directional LSTM as encoder, and compare with models which do not use pre-trained language models. We choose two sequence-based models. PA-LSTM (Zhang et al., 2017), which
<sup>5</sup>https://github.com/yuhaozhang/tacred-relation
employ Bi-LSTM to encoder the plain text and combine with position-aware attention mechanism to extract relation. PA-SLTM is the benchmark of Tacre. SA-LSTM (Yu et al., 2019), which employ CRF to learn segment-level attention and is the best sequence-based model of Tacre.
We also compare our model with two other dependency-based models which make use of GCN (Kipf and Welling, 2017) to capture semantic information from the dependency tree. C-GCN (Zhang et al., 2018), which applies pruning strategy and GCN to extract features from tree structure for relation extraction. C-AGGCN (Guo et al., 2019), which introduces self-attention to build a soft adjacent matrix as input of Dense GCN to learn tree structure features.
3.4 Results on Tacred and Tacred Revisited
We first report the results or our model on Tacred and Tacred Revisited on Table 2. Compared models are divided into three categories: 1) models with Bi-LSTM encoder in block 1; 2) models with pretrained models in block 2; 3) models with external knowledge in block 3. The results of our model are reported in block 4. We use * to mark models with dependency trees which are obtained with external tools. We use †to mark models which use external training data to pre-train the model and ‡to mark models which employ knowledge graphs to pre-train or fine-tune the model. Models with †and ‡require external data and we do not directly compare them.
3.4.1 Compare with Pre-trained models
We can see that our SMS can bring at least 0.6 and up to 5.5 F1 score improvement for the original encoder on Tacred dataset. On the Tacred Revisited dataset, our SMS can bring at least 1.8 and up to 5.9 F1 score improvement for the original encoder. Overall, different encoders with SMS all can obtain remarkable improvement on both datasets. This proves that our SMS really captures effective features from input sentence representations, which can not get directly from the representations. Compared with models which employ pre-trained models without external knowledge (i.e. training data or knowledge graph) in block 2, pretrained models with our SMS in block 4 overall perform better and SpanBERT-large+SMS achieve new state-of-the-art results on both datasets. In addition, we can see that the performance of SpanBERT-large+SMS is better than MTB, KnowBERT-W+W, and KEPLER
Models	SemEval F1(ΔF1)
LSTM	82.7
LSTM+Attention	84.0
TRE	87.1
BERT-base	87.9
BERT-large	88.8
SpanBERT-base	88.2
SpanBERT-large	89.4
R-BERT (Wu and He, 2019a)	89.3
MTB †	89.5
KnowBert-W+W ‡	89.1
LSTM+SMS	86.8(+4.1)
BERT-base+SMS	88.4(+0.5)
BERT-large+SMS	89.8(+1.0)
SpanBERT-base+SMS	88.5(+0.3)
SpanBERT-large+SMS	90.3(+0.9)
Table 3: Results on SemEval 2010 task8.  $\dagger$ means that the model employs external knowledge to pre-train the model.  $\ddagger$ means the model employs a knowledge base.
in block 3 and is competitive with K-Adapter and LUKE.
The increase of F1 score is more conspicuous on Tacred Revisited compared with Tacred. This phenomenon is further evidence that existing models have neared the upper limit of Tacred, which have many mistrabeled examples. Besides, we can see that models based on SpanBERT all have a pretty good performance. This phenomenon proves the importance of segment level features.
3.4.2 Compare with LSTM-based models
To further evaluate the effectiveness of our method SMS, we specially combine SMS with LSTM encoder. We can observe that our model also outperforms the model with LSTM encoder in block 1. Dependency-based models with graph neural networks (C-GCN and C-AGGCN) have a remarkable performance on Tacred and models which focus on segments (SA-LSTM) have a better performance on Tacred Revisited. This phenomenon means that directly modeling the segment level feature can not effectively overcome the noise from mislabeled examples and the introduction of graph structure with dependency trees can help models tackle some influence from wrong examples in the dataset itself.