Bidirectional difference locating and semantic consistency reasoning for change captioning

International Journal of Intelligent SystemsVolume 37, Issue 5 p. 2969-2987 RESEARCH ARTICLE Bidirectional difference locating and semantic consistency reasoning for change captioning Yaoqi Sun, Yaoqi Sun orcid.org/0000-0001-8874-241X School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, ChinaSearch for more papers by this authorLiang Li, Corresponding Author Liang Li [email protected] orcid.org/0000-0002-1943-8219 Institute of Computing Technology, CAS, Beijing, China Correspondence Liang Li, Institute of Computing Technology, CAS, Beijing 100086, China. Email: [email protected] Bolun Zheng, School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China. Email: [email protected]Search for more papers by this authorTingting Yao, Tingting Yao orcid.org/0000-0002-8262-7004 School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, ChinaSearch for more papers by this authorTongyv Lu, Tongyv Lu orcid.org/0000-0002-9137-1944 School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, ChinaSearch for more papers by this authorBolun Zheng, Corresponding Author Bolun Zheng [email protected] orcid.org/0000-0001-8788-1725 School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China Correspondence Liang Li, Institute of Computing Technology, CAS, Beijing 100086, China. Email: [email protected] Bolun Zheng, School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China. Email: [email protected]Search for more papers by this authorChenggang Yan, Chenggang Yan orcid.org/0000-0003-1204-0512 School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, ChinaSearch for more papers by this authorHua Zhang, Hua Zhang orcid.org/0000-0001-8493-7383 School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, ChinaSearch for more papers by this authorYongjun Bao, Yongjun Bao Data and Intelligence Department, JD.com, Beijing, ChinaSearch for more papers by this authorGuiguang Ding, Guiguang Ding School of Software, Tsinghua University, Beijing, ChinaSearch for more papers by this authorGregory Slabaugh, Gregory Slabaugh Digital Environment Research Institute (DERI), Queen Mary University of London, London, UKSearch for more papers by this author Yaoqi Sun, Yaoqi Sun orcid.org/0000-0001-8874-241X School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, ChinaSearch for more papers by this authorLiang Li, Corresponding Author Liang Li [email protected] orcid.org/0000-0002-1943-8219 Institute of Computing Technology, CAS, Beijing, China Correspondence Liang Li, Institute of Computing Technology, CAS, Beijing 100086, China. Email: [email protected] Bolun Zheng, School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China. Email: [email protected]Search for more papers by this authorTingting Yao, Tingting Yao orcid.org/0000-0002-8262-7004 School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, ChinaSearch for more papers by this authorTongyv Lu, Tongyv Lu orcid.org/0000-0002-9137-1944 School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, ChinaSearch for more papers by this authorBolun Zheng, Corresponding Author Bolun Zheng [email protected] orcid.org/0000-0001-8788-1725 School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China Correspondence Liang Li, Institute of Computing Technology, CAS, Beijing 100086, China. Email: [email protected] Bolun Zheng, School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China. Email: [email protected]Search for more papers by this authorChenggang Yan, Chenggang Yan orcid.org/0000-0003-1204-0512 School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, ChinaSearch for more papers by this authorHua Zhang, Hua Zhang orcid.org/0000-0001-8493-7383 School of Automation, College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, ChinaSearch for more papers by this authorYongjun Bao, Yongjun Bao Data and Intelligence Department, JD.com, Beijing, ChinaSearch for more papers by this authorGuiguang Ding, Guiguang Ding School of Software, Tsinghua University, Beijing, ChinaSearch for more papers by this authorGregory Slabaugh, Gregory Slabaugh Digital Environment Research Institute (DERI), Queen Mary University of London, London, UKSearch for more papers by this author First published: 19 January 2022 https://doi.org/10.1002/int.22821Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Change captioning is an emerging task to describe the changes between a pair of images. The difficulty in this task is to discover the differences between the two images. Recently, some methods have been proposed to address this problem. However, they all employ unidirectional difference localization to identify the changes. This can lead to ambiguity about the nature of the changes. Instead, we propose a framework with bidirectional difference localization and semantic consistency reasoning to describe the image changes. First, we locate the changes in the two images by capturing bidirectional differences. Then we design a decoder with spatial-channel attention to generate the change caption. Finally, we introduce semantic consistency reasoning to constrain our bidirectional difference localization module and spatial-channel attention module. Extensive experiments on three public data sets show that the performance of our proposed model outperforms the state-of-the-art change captioning models by a large margin. Volume37, Issue5Special Issue: Intelligent Information Processing and Services in Media ConvergenceMay 2022Pages 2969-2987 RelatedInformation