BERTBooster: A knowledge enhancement method jointing incremental training and gradient optimization

International Journal of Intelligent SystemsVolume 37, Issue 11 p. 9390-9403 RESEARCH ARTICLE BERTBooster: A knowledge enhancement method jointing incremental training and gradient optimization Wenchao Jiang, Wenchao Jiang orcid.org/0000-0002-6300-1962 School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangzhou, ChinaSearch for more papers by this authorJiarong Lu, Jiarong Lu orcid.org/0000-0002-9703-4084 School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorTiancai Liang, Tiancai Liang orcid.org/0000-0001-5928-7286 Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangzhou, ChinaSearch for more papers by this authorXiao Hong, Corresponding Author Xiao Hong [email protected] orcid.org/0000-0002-0823-0877 School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China Correspondence Hong Xiao, School of Computers, Guangdong University of Technology, 510006 Guangzhou, China. Email: [email protected]Search for more papers by this authorJianfeng Lu, Jianfeng Lu orcid.org/0000-0001-6834-1539 School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, ChinaSearch for more papers by this authorTao Wu, Tao Wu orcid.org/0000-0002-6212-4741 Guangdong Provincial Corps Hospital of Chinese People's Armed Police Force, Guangzhou, ChinaSearch for more papers by this author Wenchao Jiang, Wenchao Jiang orcid.org/0000-0002-6300-1962 School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangzhou, ChinaSearch for more papers by this authorJiarong Lu, Jiarong Lu orcid.org/0000-0002-9703-4084 School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, ChinaSearch for more papers by this authorTiancai Liang, Tiancai Liang orcid.org/0000-0001-5928-7286 Institute of Artificial Intelligence and Blockchain, Guangzhou University, Guangzhou, ChinaSearch for more papers by this authorXiao Hong, Corresponding Author Xiao Hong [email protected] orcid.org/0000-0002-0823-0877 School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, China Correspondence Hong Xiao, School of Computers, Guangdong University of Technology, 510006 Guangzhou, China. Email: [email protected]Search for more papers by this authorJianfeng Lu, Jianfeng Lu orcid.org/0000-0001-6834-1539 School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, ChinaSearch for more papers by this authorTao Wu, Tao Wu orcid.org/0000-0002-6212-4741 Guangdong Provincial Corps Hospital of Chinese People's Armed Police Force, Guangzhou, ChinaSearch for more papers by this author First published: 22 August 2022 https://doi.org/10.1002/int.22998Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onEmailFacebookTwitterLinkedInRedditWechat Abstract The knowledge-enhanced BERT model solves the problem of lacking knowledge in downstream tasks by injecting external expertize, and achieves higher accuracy compared with BERT model. However, owning to large-scale external knowledge is utilized into knowledge-enhanced BERT, some shortcomings comes such as information noise, lower accuracy and weak generalization ability, and so on. To solve this problem, a knowledge enhancement method BERTBooster which combines incremental learning and gradient optimization is proposed. BERTBooster disassembles the input text corpus into entity noun sets through entity noun recognition, and uses the incremental learning task denoising entity auto-encoder to create an incremental task set of entity nouns and external knowledge triples. Furthermore, BERTBooster introduces a new gradient optimization algorithm ChildTuningF into BERT model to improve the generalization ability. BERTBooster can effectively improve the factual knowledge cognition ability of CAGBERT model and improve the accuracy of the model in downstream tasks. Experiments are carried out on six public data sets such as Book_Review, LCQMC, XNLI, Law_QA, Insureace_QA, and NLPCC-DBQA. The experimental results show that the accuracy rate in downstream tasks is increased by 0.65% on average after using BERTBooster on CAGBERT. CONFLICT OF INTEREST The authors declare no conflict of interest. Open Research DATA AVAILABILITY STATEMENT Research data are not shared. REFERENCES 1Devlin J, Chang MW, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805; 2018. doi:10.48550/arXiv.1810.04805 2Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Advances in neural information processing systems. 2017:5998-6008. 3Sun yi, Hang-ping Qiu, Zheng yu, Chao-ran Zhang, Chao He. A survey of knowledge enhancement methods for natural language pre-training models. J. Chin. Inform. Process. 2021; 35(7): 10-29. 4Zhang Z, Han X, Liu Z, et al. ERNIE: enhanced language representation with informative entities [C]. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics; 2019: 1441-1451. 5Peters ME, Neumann M, Logan RL IV, et al. Knowledge enhanced contextual word representations [C]. In: Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); 2019. 6Mikolov T, Chen K, Corrado G, et al. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. 2013. doi:10.48550/arXiv.1301.3781 7Bordes A, Usunier N, Garcia-Duran A, et al. Translating embeddings for modeling multi-relational data [J]. Adv Neural Inf Process Syst. 2013; 26. 8Gan W, Sun Y, Sun Y. Knowledge structure enhanced graph representation learning model for attentive knowledge tracing. Int J Intell Syst. 2022; 37(3): 2012-2045. 9Liu W, Zhou P, Zhao Z, et al. K-BERT: enabling language representation with knowledge graph. AAAI. 2020:2901-2908. 10Yu-ze Li, Xin Luan, Zun-wang Ke, Zhe Li, Shou-er·Si La-Mu Wu. A survey of pre-training language models for knowledge perception. Comput Eng. 2021:1-21. doi:10.19678/j.issn.1000-3428.0060823 11De-guang Chen, Jin-lin Ma, Zi-ping Ma, jie Zhou. Summary of natural language processing pre-training technology. J Front Comput Sci Technol. 2021; 15(8): 1359-1389. 12Vincent P, Larochelle H, Bengio Y, et al. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th International Conference on Machine learning; 2008:1096-1103. 13Sun Y, Wang S, Li Y, et al. Ernie: enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223; 2019. doi:10.48550/arXiv.1904.09223 14Yun W, Zhang X, Li Z, Liu H, Han M. Knowledge modeling: a survey of processes and techniques. Int J Intell Syst. 2021; 36(4): 1686-1720. 15Jiang Nan, Jie Wen, Li Jin, Liu Ximeng, Jin, Di. GATrust: a multi-aspect graph attention network model for trust assessment in OSNs. IEEE Trans Knowl Data Eng. 2022; 1. doi:10.1109/TKDE.2022.3174044 16Hinton GE, Srivastava N, Krizhevsky A, et al. Improving neural networks by preventing co-adaptation of feature detectors. Comput Sci. 2012; 3(4): 212-223. 17Tseng HY, Chen YW, Tsai YH, et al. Regularizing meta-learning via gradient dropout. Proceedings of the Asian Conference on Computer Vision; 2020. 18Xu R, Luo F, Zhang Z, et al. Raise a child in large language model: towards effective and generalizable fine-tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing; 2021:9514-9528. 19Keskar NS, Nocedal J, Tang PTP, et al. On large-batch training for deep learning: generalization gap and sharp minima. 5th International Conference on Learning Representations, ICLR 2017; 2017. 20Sun X, Zhang Z, Ren X, et al. Exploring the vulnerability of deep neural networks: a study of parameter corruption. Proceedings of the AAAI Conference on Artificial Intelligence, Vol 35; 2021:11648-11656. 21Foret P, Kleiner A, Mobahi H, et al. Sharpness-aware minimization for efficiently improving generalization. International Conference on Learning Representations; 2020. 22Duan N, Tang D Overview of the NLPCC 2017 shared task: open domain Chinese question answering. National CCF Conference on Natural Language Processing and Chinese Computing. Springer, Cham, 2017:954-961. 23Xu B, Xu Y, Liang J, et al. CN-DBpedia: a never-ending Chinese knowledge extraction system. International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Springer, Cham; 2017:428-438. Volume37, Issue11November 2022Pages 9390-9403 ReferencesRelatedInformation