SlowGuess/ACL_2020_OCR huggingface dataset:
Wiki-KBP	BioInfer
Settings	Precision(%)	Recall(%)	F1(%)	Precision(%)	Recall(%)	F1(%)
Curriculum	41.7±0.19	41.5±0.16	41.6±0.17	59.5±0.21	43.7±0.18	49.8±0.20
Joint (w/o curriculum)	41.3±0.22	40.9±0.20	41.1±0.21	58.7±0.24	42.6±0.19	48.5±0.23
Separate (w/o joint)	38.8±0.24	40.5±0.27	38.4±0.25	54.7±0.27	41.3±0.23	47.6±0.26
Table 4: Ablation results of the MRL-CoType for end-to-end relation extraction.
methods on all metrics. Note that the refinements on BioInfer is significant (t-test with  $p < 0.05$ ) even though the BioInfer has a large entity type set (2,200 types) and the base extractor CoType has achieved a high performance (0.74 S-F1), which shows that our agents are capable of leading entity extractors towards a better optimization with noisy.
3.2.2 Performance on Relation Extraction
Another comparison is the end-to-end relation extraction task, we report the precision, recall and F1 results in Table 3 and it illustrates that:
(1) Our method achieves best F1 for Wiki-KBP, outperforms all baselines on all metrics for BioInfer data, and significantly refines both the two base extractors, PCNN and CoType (t-test with  $p < 0.05$ ), demonstrating the effectiveness of our approach.
(2) The improvements for CoType are larger than PCNN. Since CoType is a joint extraction model and leverages multi-agents better than the single-task extractor with fixed entity extractor. This shows the benefit of correlations between the two extraction tasks.
(3) Using the same base relation extractor, the MRL-PCNN achieves significantly better improvements than RRL-PCNN (t-test with  $p < 0.05$ ). Besides, the precision of RRL-PCNN method is relatively worse than recall, which is mainly caused by the noise propagation of entity extraction and its binary discard-or-retain action. By contrast, our model achieves better and more balanced results by leveraging the cooperative multiagents with fine-grained confidences.
(4) The MRL-PCNN gains comparable performance with BA-Fix-PCNN, which leverages the additional information from the test set to adjust softmax classifier. This verifies the effectiveness and the robustness of the proposed RL-based relabeling method to reduce the shifted label distribution gap without knowing the test set.
To evaluate the impact of curriculum learning strategy and joint learning strategy of our method, we compare three training settings: curriculum learn
Figure 2: Smoothed average rewards on Wiki-KBP data for two agents of MRL-CoType. The light-colored lines are un-smoothed rewards.
ing, standard training procedure as described in Section 2.3; joint multiagents training without curriculum learning (randomly sample training instances); and separate training without the participation of other agents using a pipeline manner, i.e., train an entity agent with only entity extractor and train a relation agent with only relation extractor.
The end-to-end relation extraction results are reported in Table 4. The curriculum setting and the joint setting achieve much better results than the separate training setting. This shows the superiority of cooperative multi-agents over single view extraction, which evaluates confidence with limited information. Besides, the curriculum setting achieves better results than the joint setting, especially on the BioInfer data, which has a larger type set and is more challenging than Wiki-KBP. This indicates the effectiveness of the curriculum learning strategy, which enhances the model ability to handle large state space with gradual exploration.
Training efficiency is an important issue for RL methods since the agents face the exploration-exploitation dilemma. We also compare the three settings from the view of model training. Figure 2 reports the average rewards for an entity agent and a relation agent on Wiki-KBP respectively. A high average reward indicates that the agent is trained
5946
Figure 3: Proportions of re-labeled instances for MRL-CoType. "N-to-P" denotes the instances are re-labeled from negative to positive. "divergent" means that entity agents and relation agent have different evaluations about whether the instance is positive or negative.
effectively since it made valuable decisions and received positive feedback. From it we have the following observations: (1) The curriculum setting and the joint setting gain better performance than the separate training, which is consistent with the end-to-end extraction results. The improvement comes from the mutual enhancement among agents, since the correlations between the two tasks can restrict the reward signals to only those agents involved in the success or failure on the task; (2) The curriculum learning achieves higher rewards than the other two settings with fewer epochs, since that the convergence to local optimum can be accelerated by smoothly increasing the instance difficulty, and the multiagents provide a regularization effect.
3.4 Re-labeling Study
To gain insight into the proposed method, we conduct a statistic on the final re-labeled instances. Figure 3 reports the results and shows that our approach identifies some noisy instances including both positives and negatives, and leverage them in a fine-grained manner comparing with discard-or-retain strategy. Besides, the instances which are re-labeled from negatives to positives take a larger proportion than those with inverse re-labeling assignments, especially on Wiki-KBP data. This is in accordance with the fact that many noisy labels are "None" in DS setting. Note that some instances are re-labeled with divergent evaluations between entity-view and relation-view agents, which are usually get low confidence through the consensus module and have a small impact on the optimization with damping losses.
We further sample two sentences to illustrate the re-labeling processes. On Table 5, the first sentence has a noisy relation label None, while the relation extractor recognizes it as country_of_birth relation. Based on the extracted type, the relation-view agent evaluates it as a confidential positive instance
due to the typical pattern "born in" in the sentence. The entity-view agents also evaluate it as positive with relatively lower confidence, and finally the sentence is re-labeled as positive by the consensus module. For the second sentence, agents disagree that it is positive. With the help of diverse extraction information, the consensus module re-labels the instance with low confidence score, and further alleviates the performance harm by loss damping.
Many entity and relation extraction methods have been proposed with the pipelined fashion, i.e., perform named entity recognition (NER) first and then relation classification. Traditional NER systems usually focus on a few predefined types with supervised learning (Yosef et al., 2012). However, the expensive human annotation blocks the large-scale training data construction. Recently, several efforts on DS and weak supervision (WS) NER extraction have been made to address the training data bottleneck (Yogatama et al., 2015; Yang et al., 2018). For relation extraction, there are also many DS methods (Mintz et al., 2009; Min et al., 2013; Zeng et al., 2015; Han and Sun, 2016; Ji et al., 2017; Lei et al., 2018) and WS methods (Jiang, 2009; Ren et al., 2016; Deng et al., 2019) to address the limitation of supervised methods. Our method can be applied for a large number of those extractors as a post-processing plugin since the DS and WS usually incorporate many noises.
A recent work CrossWeigh (Wang et al., 2019) estimates the label mistakes and adjusts the weights of sentences in the NER benchmark CoNLL03. They focus on the noises of supervised "gold standard" labels while we focus on the noises of automatically constructed "silver standard" labels. Moreover, we deal with the noises by considering the shifted label distribution problem, which is overlooked by most existing DS works. In Ye et al. (2019), this issue is analyzed and authors improve performance significantly by using the distribution information from test set. In this paper, we propose to use RL to explore suitable label distributions by re-distributing the training set with confidence-scored labels, which is practical and robust to label distribution shift since we may not know the distribution of test set in real-world applications.
Another extraction manner is joint extraction, such as methods based on neural network with parameter sharing (Miwa and Bansal, 2016), represent
5947
Sentence 1, False Negative, Label: (Bashardost[/person], None, Ghazni[/location])	Entity Extractor	Relation Extractor	Entity Agents	Relation Agent	Confidence Consensus
Bashardost, an ethnic Hazara, was born in Ghazni province to a family of government employees.	Bashardost[/person]Ghazni[/location]	country_of_birth	0.772 0.729	0.896	Positive (0.799)
Sentence 2, False Positive, Label: (profilin[/Protein], POS_ACTIONPhysical, actin[/Protein])
Acanthamoeba profilin affects the mechanical properties of nonfilamentous actin.	profilin[/None]actin[/Protein]	None	0.373 0.791	0.236	Negative (0.533)
Table 5: Confidence evaluations on two noisy instances using MRL-CoType.
tation learning (Ren et al., 2017) and new tagging scheme (Zheng et al., 2017). However, these works perform extraction without explicitly handling the noises. Our approach introduces multiagents to the joint extraction task and explicitly model sentence confidence. As for the RL-based methods, in Zeng et al. (2018), RL agent is introduced as bag-level relation predictor. Qin et al. (2018) and Feng et al. (2018) use agent as instance selectors to discard noisy instances in sentence-level. Different from adopting a binary action strategy and only focus on false positives in these works, we adopt a continuous action space (confidence evaluation) and handle the noises in a fine-grained manner. The binary selection strategy is also adopted in a related study, Reinforced Co-Training (Wu et al., 2018), which uses an agent to select instances and help classifiers to form auto-labeled datasets. An important difference is that they select unlabeled instances while we evaluate noisy instances and relabel them. More recently, HRL (Takanobu et al., 2019) uses a hierarchical agent to first identifies relation indicators and then entities. Different from using one task-switching agent of this work, we leverage a group of multiagents, which can be a pluggable helper to existing extraction models.
To deal with the noise labels and accompanying shifted label distribution problem in distant supervision, in this paper, we propose a novel method to jointly extract entity and relation through a group of cooperative multiagents. To make full use of each instance, each agent evaluates the instance confidence from different views, and then a confidence consensus module is designed to re-label noisy instances with confidence. Thanks to the exploration of suitable label distribution by RL agents, the confidence are further used to adjust the training losses of extractors and the potential harm caused by noisy instances can be alleviated.
To demonstrate the effectiveness of the proposed method, we evaluate it on two real-world datasets and the results confirm that the proposed method can significantly improve extractor performance and achieve effective learning.
This work is supported by the National Natural Science Foundation of China (No.61602013), and the Shenzhen General Research Project (No. JCYJ20190808182805919).
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning., pages 41-48.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems., pages 2787-2795.
Yang Deng, Yaliang Li, Ying Shen, Nan Du, Wei Fan, Min Yang, and Kai Lei. 2019. Medtruth: A semi-supervised approach to discovering knowledge condition information from multi-source medical data. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 719-728.
Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. 2018. Reinforcement learning for relation classification from noisy data. In Thirty-Second AAAI Conference on Artificial Intelligence.
Matthew R Gormley, Mo Yu, and Mark Dredze. 2015. Improved relation extraction with feature-rich compositional embedding models. Proceedings of the 2015 joint conference on empirical methods in natural language processing and computational natural language learning, pages 1774-1784.
Xianpei Han and Le Sun. 2016. Global distant supervision for relation extraction. In Proceedings of the AAAI Conference on Artificial Intelligence.
5948
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 541-550.
Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2017. Distant supervision for relation extraction with sentence-level attention and entity descriptions. In Thirty-Second AAAI Conference on Artificial Intelligence.
Wei Jia, Dai Dai, Xinyan Xiao, and Hua Wu. 2019. Arnor: Attention regularization based noise reduction for distant supervision relation classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Jing Jiang. 2009. Multi-task transfer learning for weakly-supervised relation extraction. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics, pages 1012-1020.
Kai Lei, Daoyuan Chen, Yaliang Li, Nan Du, Min Yang, Wei Fan, and Ying Shen. 2018. Cooperative denoising for distantly supervised relation extraction. In Proceedings of the 27th International Conference on Computational Linguistics., pages 426-436.
Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In Proceedings of the 52th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 402-412.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. 2018. RLlib: Abstractions for distributed reinforcement learning. In Proceedings of the 35th annual international conference on machine learning.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2124-2133.
Xiao Ling and Daniel S Weld. 2012. Fine-grained entity recognition. In Twenty-Sixth AAAI Conference on Artificial Intelligence, volume 12, pages 94-100.
Bonan Min, Ralph Grishman, Li Wan, Chang Wang, and David Gondek. 2013. Distant supervision for relation extraction with an incomplete knowledge base. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 777-782.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the
47th Annual Meeting of the Association for Computational Linguistics, pages 1003-1011.
Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1105-1116.
Sampo Pyysalo, Filip Ginter, Juho Heimonen, Jari Björne, Jorma Boberg, Jouni Järvinen, and Tapio Salakoski. 2007. Bioinfer: a corpus for information extraction in the biomedical domain. BMC bioinformatics, 8(1):50.
Pengda Qin, Weiran Xu, and William Yang Wang. 2018. Robust distant supervision relation extraction via deep reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Xiang Ren, Wenqi He, Meng Qu, Clare R Voss, Heng Ji, and Jiawei Han. 2016. Label noise reduction in entity typing by heterogeneous partial-label embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining., pages 1825-1834.
Xiang Ren, Zeqiu Wu, Wenqi He, Meng Qu, Clare R Voss, Heng Ji, Tarek F Abdelzaher, and Jiawei Han. 2017. Cotype: Joint extraction of typed entities and relations with knowledge bases. In WWW, pages 1015-1024.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D Manning. 2012. Multi-instance multi-label learning for relation extraction. In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, pages 455-465.
Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, and Minlie Huang. 2019. A hierarchical framework for relation extraction with reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):2724-2743.
5949
Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. 2019. Crossweigh: Training named entity tagger from imperfect annotations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5157-5166.
Jiawei Wu, Lei Li, and William Yang Wang. 2018. Reinforced co-training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1252-1262.
Yaosheng Yang, Wenliang Chen, Zhenghua Li, Zhengqiu He, and Min Zhang. 2018. Distantly supervised NER with partial annotation learning and reinforcement learning. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2159-2169.
Qinyuan Ye, Liyuan Liu, Maosen Zhang, and Xiang Ren. 2019. Looking beyond label noise: Shifted label distribution matters in distantly supervised relation extraction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3832-3841.
Dani Yogatama, Daniel Gillick, and Nevena Lazic. 2015. Embedding methods for fine grained entity
type classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 291-296.
Mohamed Amir Yosef, Sandro Bauer, Johannes Hoffart, Marc Spaniol, and Gerhard Weikum. 2012. Hyena: Hierarchical type classification for entity names. Proceedings of the 21th International Conference on Computational Linguistics, pages 1361-1370.
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In Proceedings of the 2015 joint conference on empirical methods in natural language processing and computational natural language learning, pages 1753-1762.
Xiangrong Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018. Large scaled relation extraction with reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence.
Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. 2017. Joint extraction of entities and relations based on a novel tagging scheme. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1227-1236.