SlowGuess/ACL_2020_OCR huggingface dataset:
Daoyuan Chen $^{1}$  Yaliang Li $^{1}$  Kai Lei $^{2}$  Ying Shen $^{3*}$
1Alibaba Group
$^{2}$ School of Electronics and Computer Engineering, Peking University
$^{3}$ School of Intelligent Systems Engineering, Sun Yat-Sen University
1{daoyuanchen.cdy, yaliang.li}@alibaba-inc.com
2leik@pkusz.edu.cn, 3sheny76@mail.sysu.edu.cn
Distant supervision based methods for entity and relation extraction have received increasing popularity due to the fact that these methods require light human annotation efforts. In this paper, we consider the problem of shifted label distribution, which is caused by the inconsistency between the noisy-labeled training set subject to external knowledge graph and the human-annotated test set, and exacerbated by the pipelined entity-then-relation extraction manner with noise propagation. We propose a joint extraction approach to address this problem by re-labeling noisy instances with a group of cooperative multiagents. To handle noisy instances in a fine-grained manner, each agent in the cooperative group evaluates the instance by calculating a continuous confidence score from its own perspective; To leverage the correlations between these two extraction tasks, a confidence consensus module is designed to gather the wisdom of all agents and re-distribute the noisy training set with confidence-scored labels. Further, the confidence are used to adjust the training losses of extractors. Experimental results on two real-world datasets verify the benefits of re-labeling noisy instance, and show that the proposed model significantly outperforms the state-of-the-art entity and relation extraction methods.
The extraction of entities and relations has long been recognized as an important task within natural language processing, as it facilitates text understanding. The goal of the extraction task is to identify entity mentions, assign predefined entity types, and extract their semantic relations from text corpora. For example, given a sentence "Washington is the president of the United States of America",
an extraction system will find a PRESIDENT_OF relation between PERSON entity "Washington" and COUNTRY entity "United States of America".
A major challenge of the entity and relation extraction task is the absence of large-scale and domain-specific labeled training data due to the expensive labeling efforts. One promising solution to address this challenge is distant supervision (DS) (Mintz et al., 2009; Hoffmann et al., 2011), which generates labeled training data automatically by aligning external knowledge graph (KG) to text corpus. Despite its effectiveness, the aligning process introduces many noisy labels that degrade the performance of extractors. To alleviate the introduced noise issue of DS, extensive studies have been performed, such as using probabilistic graphical models (Surdeanu et al., 2012), neural networks with attention (Zeng et al., 2015; Lin et al., 2016) and instance selector with reinforcement learning (RL) (Qin et al., 2018; Feng et al., 2018).
However, most existing works overlooked the shifted label distribution problem (Ye et al., 2019), which severely hinders the performance of DS-based extraction models. Specifically, there is a label distribution gap between DS-labeled training set and human-annotated test data, since two kinds of noisy labels are introduced and they are subject to the aligned KG: (1) False Positive: unrelated entity pair in the sentence while labeled as relations in KG; and (2) False Negative: related entity pair while neglected and labeled as NONE. Existing denoising works assign low weights to noisy instances or discard false positives while not recovering the original labels, leaving the shifted label distribution problem unsolved.
Moreover, most denoising works assume that the target entities have been extracted, i.e., the entity and relation extraction is processed in a pipe-lined manner. By extracting entities first and then classifying predefined relations, the entity extraction
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5940-5950 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: Overview of the proposed method. A group of multiagents are leveraged to evaluate the confidence of noisy instances from different extraction views. Base extractors are refined by iteratively training on the redistributed instances with confidence-scored labels.
errors will be propagated to the relation extractor, introducing more noisy labels and exacerbating the shifted label problem. Besides, there are some correlations and complementary information between the two extraction tasks, which are under-utilized but can provide hints to reduce noises more precisely, e.g., it is unreasonable to predict two COUNTRY entities as the relation PRESIDENT_OF.
In this paper, to reduce the shifted label distribution gap and further enhance the DS-based extraction models, we propose a novel method to re-label the noisy training data and jointly extract entities and relations. Specifically, we incorporate RL to re-label noisy instances and iteratively retrain entity and relation extractors with adjusted labels, such that the labels can be corrected by trial and error. To leverage the correlations between the two extraction tasks, we train a group of cooperative multiagents to evaluate the instance confidence from different extraction views. Through a proposed confidence consensus module, the instances are re-labeled with confidence-scored labels, and such confidence information will be used to adjust the training loss of extractors. Finally, the performances of extractors are refined by exploring suitable label distributions with iterative re-training.
Empirical evaluations on two real-world datasets show that the proposed approach can effectively help existing extractors to achieve remarkable extraction performance with noisy labels, and the agent training is efficient with the help of correlations between these two extraction tasks.
In this research, we aim to refine entity extractor and relation extractor trained with DS, by incorporating a group of cooperative multiagents. Formally, given a DS training corpus  $D = \{s_1,\dots ,s_n\}$ , an entity extractor  $\theta_e^\prime$  and a relation extractor  $\theta_r^\prime$  trained on  $D$  are input into the multiagents. The agents re-distribute  $D$  with confidence-scored labels and output two refined extractors  $\theta_e^*$  and  $\theta_r^*$  using the adjusted labels.
Towards this purpose, we model our problem as a decentralized multiagents RL problem, where each agent receives local environmental observation and takes action individually without inferring the policies of other agents. It is hard to directly evaluate the correctness of adjusted noisy labels since we do not know the "gold" training label distributions suitable to the test set. Nonetheless, we can apply RL to indirectly judge the re-labeling effect by using performance scores on an independent validation set as rewards, which is delayed over the extractor re-training. Further, the decentralization setting allows the interaction between the distinct information of entity and relation extractors via intermediate agents.
As shown in Figure 1, a group of agents acts as confidence evaluators, and the external environment consists of training instances and classification results of extractors. Each agent receives a private observation from the perspective of entity
5941
extractor or relation extractor, and makes an independent action to compute a confidence score of the instance. These actions (confidence scores) will then be considered together by the confidence consensus module, which determines whether the current sentence is positive or negative and assigns a confidence score. Finally, the updated confidence are used to retrain extractors, the performance score on validation set and the consistent score of the two extractors are combined into rewards for agents.
The proposed method can be regarded as a post-processing plugin for existing entity and relation extraction model. That is, we design a general framework of the states, actions and rewards by reusing the inputs and outputs of the extractors.
2.2 Confidence Evaluators as Agents
A group of cooperative multiagents are used to evaluate the confidence of each instance. These multiagents are divided into two subgroups, which act from the perspective of entity and relation respectively. There can be multiple agents in each subgroup for the purpose of scaling to larger observation space and action space for better performance. Next, we will detail the states, actions and rewards of these agents.
States The states  $S_{e}$  for entity-view agents and  $S_{r}$  for relation-view agents represent their own viewpoint to evaluate the instance confidence. Specifically, entity-view agents evaluate sentence confidence according to three kinds of information: current sentence, the entity extraction results (typed entity) and the noisy label types. Similarly, relation-view agents make their decisions depending on the current sentence, the relation types from relation extractor and the noisy label types from DS.
Most entity and relation extractors encode the semantic and syntactic information of extracted sentences into low-dimension embeddings as their inputs. For entity types and relation types, we also encode them into embeddings and some extractors have learned these vectors such as CoType (Ren et al., 2017). Given reused extractors, we denote the encoded sentence vector as s, the extracted type vector as  $\mathbf{t}^{\mathrm{e}}$  and  $\mathbf{t}^{\mathrm{r}}$  for entity and relation respectively, and DS type vectors as  $\mathbf{t}_{\mathbf{d}}^{\mathrm{e}}$  and  $\mathbf{t}_{\mathbf{d}}^{\mathrm{r}}$  for entity and relation respectively. We reuse the sentence and type vectors of base extractors to make our approach lightweight and pluggable. Finally, we average the extracted and DS type embeddings to decrease the size of observation space, and con
catenate them with the sentence embedding s to form the states  $\mathbf{S}^{\mathrm{e}}$  and  $\mathbf{S}^{\mathrm{r}}$  for entity/relation agents respectively as follows:
$$
\mathbf {S} ^ {\mathrm {e}} = \mathbf {s} \| \left(\mathbf {t} ^ {\mathrm {e}} + \mathbf {t} _ {\mathbf {d}} ^ {\mathrm {e}}\right) / 2, \quad \mathbf {S} ^ {\mathrm {r}} = \mathbf {s} \| \left(\mathbf {t} ^ {\mathrm {r}} + \mathbf {t} _ {\mathbf {d}} ^ {\mathrm {r}}\right) / 2, \tag {1}
$$
Note that we have encoded some semantics into the type vectors, e.g., the margin-based loss used in CoType enforces the type vectors are closer to their candidate type vectors than any other non-candidate types. Intuitively, in the representation spaces, the average operation leads in the midpoint of extracted type vector and DS type vector, which partially preserves the distance property among the two vectors and other type vectors, so that helps form distinguishable states.
Actions To assign confidence in a fine-grained manner and accelerate the learning procedure, we adopt a continuous action space. Each agent uses a neural policy network  $\Theta$  to determine whether the current sentence is positive (conform with the extracted type  $t_i$ ) or negative ("None" type) and computes a confidence score  $c$ . We model this action as a conditional probability prediction, i.e., estimate the probability as confidence given by the extracted type  $t_i$  and the current state  $\mathbf{S}$ :  $c = p(positive|t_i, \Theta, \mathbf{S})$ . We adopt gated recurrent unit (GRU) as policy network, which outputs the probability value using sigmoid function. A probability value (confidence score) which is close to 1/0 means that the agent votes a sentence as positive/negative with a high weight.
To handle huge state spaces (e.g., there are thousands of target types in our experimental dataset) and make our approach scalable, here we divide and conquer the state space by using more than one agent in entity-view and relation-view groups. The target type set is divided equally by agent number and each agent only is in charge of a part of types. Based on the allocation and DS labels, one sentence is evaluated by only one relation agent and two entity agents at a time, meanwhile, the other agents are masked.
Re-labeling with Confidence Consensus To leverage the wisdom of crowds, we design a consensus strategy for the evaluated confidence from multiagents. This is conducted by two steps: gather confidence and re-label with confidence score. Specifically, we calculate an averaged score as  $\bar{c} = c_{sum} / 3$ , where  $c_{sum}$  is the sum of all agent.   
confidences and the dividing means three agents
5942
evaluated the present sentence due to the above masking action strategy. Then we label the current sentence as negative ("None" type) with confidence  $C = 1 - \bar{c}$  if  $\bar{c} \leq 0.5$ , otherwise we label the current sentence as positive (replace noisy label with extracted type) with confidence  $C = \bar{c}$ . This procedure can be regarded as weighted voting and re-distribute the training set with confidence-scored labels as shown in the right part of Figure 1, where some falsely labeled instances are put into intended positions or assigned with low confidences.
Rewards The reward of each agent is composed of two parts: shared global reward  $g$  expressing correlations among sub-tasks, and separate local rewards restricting the reward signals to different three agents for different sentences (recall that we evaluate each sentence by different agents w.r.t their responsible types). Specifically, the global reward  $g$  can give hints for denoising and here we adopt a general, translation-based triple score as used in TransE (Bordes et al., 2013)  $g = ||\mathbf{t}_1 + \mathbf{t_r} - \mathbf{t_2}||$ , where  $\mathbf{t_1}, \mathbf{t_r}$  and  $\mathbf{t_2}$  are embeddings for triple  $(E_1, R, E_2)$  and pre-trained by TransE. The score is used to measure the semantic consistency of each triple and can be easily extended with many other KG embedding methods (Wang et al., 2017). As for the separate local reward, we use F1 scores  $F_1^e$  and  $F_1^r$  to reflect the extractor performance, which are gained by entity extractor and relation extractor on an independent validation dataset  $^1$  respectively. Finally, to control the proportions of two-part rewards, we introduce a hyper-parameter  $\alpha$ , which is shareable for ease of scaling to multiple agents as:
$$
r _ {e} = \alpha * F _ {1} ^ {e} - g, r _ {r} = \alpha * F _ {1} ^ {r} - g. (2)
$$
2.3 Model Learning
2.3.1 Loss Correction for Extractors
With the evaluated confidence and re-labeled instances, we adjust the training losses of entity extractor and relation extractor to alleviate the performance harm from noise and shifted label distribution. Denote the original loss of extractor as  $\ell$ , the new loss  $\ell'$  is adjusted by an exponential scaling factor  $\lambda$  and confidence  $C$  as:  $\ell' = C^{\lambda} \ell$ . Intuitively, a small confidence score  $C$  and a large  $\lambda$  indicate that the current instance has almost no impact on the model optimization. This can alleviate
side-effects caused by noises and prevent the gradient being dominated by noisy labels, especially for those with divergent votes since the averaging in confidence consensus module leads to a small  $C$ .
2.3.2 Training Algorithm
Pre-training Many RL-based models introduce pre-training strategies to refine the agent training efficiency (Qin et al., 2018; Feng et al., 2018). In this study, we pre-train our models in two aspects: (1) we first pre-train entity and relation extractors to be refined as environment initialization, which is vital to provide reasonable agent states (embeddings of sentences and extracted types). (2) we then pre-train the policy networks of agents to gain a preliminary ability to evaluate confidence. In order to guide the instance confidence evaluation, we extract a small part of the valid data. The relatively clean DS type labels of the valid data are used to form states. The binary label is assigned according to the valid data and the policy networks are pre-trained for several epochs. Although the binary labels from valid data are not exactly the continuous confidence, the policy networks gain a better parameter initialization than random initialization by this approximate training strategy.
To gain a relatively clean data, we randomly select  $20\%$  data from the original training set, extract them using pretrained CoType model and retain only one instance for each sentence whose DS label is the same as the extracted label.
5943
Iterative Re-training With the pre-trained extractors and policy networks, we retrain extractors and agents as Algorithm 1 detailed. The agents refine extractors in each epoch and we record parameters of extractors that achieve best F1 performance. For each data batch, entity and relation extractor perform extraction, form the states  $\mathbf{S}^{\mathrm{e}}$  and  $\mathbf{S}^{\mathrm{r}}$  as Equation (1), and send them to entity and relation agents respectively. Then agents take actions (evaluate confidence) and redistribute instance based on confidence consensus module (Section 2.2). Finally extractors are trained with confidence and give rewards as Equation (2).
Curriculum Learning for Multiagents It is difficult to learn from scratch for many RL agents. In this study, we extend the curriculum learning strategy (Bengio et al., 2009) to our cooperative multiagents. The motivation is that we can leverage the complementarity of the two tasks and enhance the agent exploration by smoothly increasing the policy difficulty. To be more specific, we maintain a priority queue and sample instances ordered by their reward values. Once the reward of current sentence excesses the training reward threshold  $r_{threshold}$  or the queue is full, we then learn agents policies using Proximal Policy Optimization (PPO) (Schulman et al., 2017) algorithm, which achieves good performances in many continuous control tasks. Algorithm 2 details the training procedure.
Datasets We evaluate our approach on two public datasets used in many extraction studies (Pyysalo et al., 2007; Ling and Weld, 2012; Ren et al., 2017): Wiki-KBP: the training sentences are sampled from Wikipedia articles and the test set are manually annotated from 2013 KBP slot filling task; BioInfer: the dataset is sampled and manually annotated from biomedical paper abstracts. The two datasets vary in domains and scales of type set, detailed statistics are shown in Table 1.
Datasets	Wiki-KBP	BioInfer
#Relation / entity types	19 / 126	94 / 2,200
#Train Mr / Me	148k / 247k	28k / 53k
#Test Mr / Me	2,948 / 1,285	3,859 / 2,389
Table 1: Datasets statistics.  $M^r$  and  $M^e$  indicates relation and entity mentions respectively.
Baselines For relation extraction, we compare with both pipe-lined methods and joint extraction methods: MintZ (Mintz et al., 2009) is a feature-based DS method using a logistic classifier; MultiR (Hoffmann et al., 2011) models noisy DS labels with multi-instance multi-label learning; DS-Joint (Li and Ji, 2014) jointly extracts entities and relations using structured perceptron; FCM (Gormley et al., 2015) introduces a neural model to learn linguistic compositional representations; PCNN (Zeng et al., 2015) is an effective relation extraction architecture with piece-wise convolution; CoType (Ren et al., 2017) is a state-of-the-art joint extraction method leveraging representation learning for both entity and relation types; RRL-PCNN (Qin et al., 2018) is a state-of-the-art RL-based method, which takes PCNN as base extractor and can also be a plugin to apply to different relation extractors; ARNOR (Jia et al., 2019) is a state-of-the-art de-noising method, which proposes attention regulation to learn relation patterns; BA-fix-PCNN (Ye et al., 2019) greatly improves the extraction performance by introducing  $20\%$  samples of the test set and estimate its label distribution to adjust the classifier of PCNN.
For entity extraction methods, we compare with a supervised type classification method, HYENA (Yosef et al., 2012); a heterogeneous partial-label
5944
Wiki-KBP	BioInfer
Methods	S-F1	Ma-F1	Mi-F1	S-F1	Ma-F1	Mi-F1
HYENA	0.26	0.43	0.39	0.52	0.54	0.56
FIGER	0.29	0.56	0.54	0.69	0.71	0.71
WSABIE	0.35	0.55	0.50	0.64	0.66	0.65
PLE	0.37	0.57	0.53	0.70	0.71	0.72
CoType	0.39	0.61	0.57	0.74	0.76	0.75
MRL-CoType	0.42±7.2e-3	0.64±1.1e-2	0.60±8.3e-3	0.77±6.5e-3	0.79±1.3e-2	0.78±7.4e-3
(improvements)	(+7.69%)	(+4.92%)	(+5.26%)	(+4.05%)	(+3.95%)	(+4.00%)
Table 2: NER performance on two datasets, 3-time average results with standard deviations are reported.
Wiki-KBP	BioInfer
Methods	Precision	Recall	F1	Precision	Recall	F1
MintZ	0.296	0.387	0.335	0.572	0.255	0.353
MultiR	0.325	0.278	0.301	0.459	0.221	0.298
DS-Joint	0.444	0.043	0.078	0.584	0.001	0.002
FCM	0.151	0.500	0.301	0.535	0.168	0.255
ARNOR	0.453	0.338	0.407	0.589	0.382	0.477
BA-Fix-PCNN	0.457	0.341	0.409	0.587	0.384	0.478
RRL-PCNN	0.435	0.322	0.392	0.577	0.381	0.470
PCNN	0.423	0.310	0.371	0.573	0.369	0.461
MRL-PCNN (improvements)	0.461±2.5e-3 (+8.98%)	0.325±2.3e-3 (+4.83%)	0.407±1.4e-3 (+9.70%)	0.590±1.1e-3 (+2.97%)	0.386±2.3e-3 (+4.61%)	0.483±2.8e-3 (+4.77%)
CoType	0.348	0.406	0.369	0.536	0.424	0.474
MRL-CoType (improvements)	0.417±1.9e-3 (+19.83%)	0.415±1.6e-3 (+2.22%)	0.416±1.7e-3 (+12.74%)	0.595±2.1e-3 (+11.01%)	0.437±1.8e-3 (+3.01%)	0.498±2.0e-3 (+5.63%)
Table 3: End-to-end relation extraction performance, 3-time average results with standard deviations are reported.
embedding method, PLE (Ren et al., 2016); and two DS methods FIGER (Ling and Weld, 2012) and WSABIE (Yogatama et al., 2015).
Multiagents Setup To evaluate the ability of our approach to refine existing extractors, we choose two basic extractors for our Multiagent RL approach, CoType and PCNN, and denote them as MRL-CoType and MRL-PCNN respectively. Since PCNN is a pipe-lined method, we reuse a pre-trained and fixed CoType entity extractor, and adopt PCNN as base relation extractor to adapt to the joint manner. For the CoType, we use the implementation of the original paper  $^{2}$ , and adopt the same sentence dimension, type dimension and hyper-parameters settings as reported in (Ren et al., 2017). For the PCNN, we set the number of kernel to be 230 and the window size to be 3. For the KG embeddings, we set the dimension to be 50 and pre-train them by TransE. We use Stochasitic Gradient Descent and learning rate scheduler with cosine annealing to optimize both the agents and extractors, the learning rate range and batch size is set to be [1e-4, 1e-2] and 64 respectively.
We implement our RL agents using a scalable RL library, RLlib (Liang et al., 2018), and adopt 2/8 relation agents and 2/16 entity agents for Wiki
KBP/BioInfer datasets respectively, according to their scales of type sets. For the multi-agents, due to the limitation of RL training time, we set the PPO parameters as default RLlib setting and perform preliminary grid searches for other parameters. For the PPO algorithm, we set the GAE lambda parameter to be 1.0, the initial coefficient for KL divergence to be 0.2. The loss adjusting factor  $\lambda$  is searched among  $\{1,2,4\}$  and set to be 2, the reward control factors  $\alpha$  is searched among  $\{2e - 1,1,2,4\}$  and set to be 2. For all agents, the dimensions of GRU is searched among  $\{32,64\}$ , and the setting as 64 achieved slightly better performance than setting as 32, while the larger dimension setting leads to higher memory overhead for each agent. Hence we set it to be 32 to enable a larger scale of the agents.
3.2 Effectiveness of Multiagents
3.2.1 Performance on Entity Extraction
We adopt the Macro-F1, Micro-F1 and Strict-F1 metrics (Ling and Weld, 2012) in the entity extraction evaluation. For Strict-F1, the entity prediction is considered to be "strictly" correct if and only if when the true set of entity tags is equal to the prediction set. The results are shown in Table 2 and we can see that our approach can effectively refine the base extractors and outperform all baseline
$^{2}$ https://github.com/INK-USC/DS-RelationExtraction
5945