Multistage semantic‐aware image inpainting with stacked generator networks

International Journal of Intelligent SystemsVolume 37, Issue 2 p. 1599-1617 RESEARCH ARTICLE Multistage semantic-aware image inpainting with stacked generator networks Yongpeng Ren, Yongpeng Ren orcid.org/0000-0003-1441-3810 College of Computer Science, Chengdu University of Information Technology, Chengdu, ChinaSearch for more papers by this authorHongping Ren, Hongping Ren orcid.org/0000-0001-7257-8547 College of Computer Science, Chengdu University of Information Technology, Chengdu, ChinaSearch for more papers by this authorCanghong Shi, Canghong Shi orcid.org/0000-0002-5188-6230 College of Computer Science, Xihua University, Chengdu, ChinaSearch for more papers by this authorXian Zhang, Xian Zhang orcid.org/0000-0003-4697-9480 College of Computer Science, Chengdu University of Information Technology, Chengdu, ChinaSearch for more papers by this authorXi Wu, Xi Wu orcid.org/0000-0002-7129-4961 College of Computer Science, Chengdu University of Information Technology, Chengdu, ChinaSearch for more papers by this authorXiaojie Li, Corresponding Author Xiaojie Li lixj@cuit.edu.cn orcid.org/0000-0003-3341-4034 College of Computer Science, Chengdu University of Information Technology, Chengdu, China Correspondence Xiaojie Li, College of Computer Science, Chengdu University of Information Technology, 610225 Chengdu, China. Email: lixj@cuit.edu.cnSearch for more papers by this authorJiancheng Lv, Jiancheng Lv orcid.org/0000-0001-6551-3884 College of Computer Science, Sichuan University, Chengdu, ChinaSearch for more papers by this authorJiliu Zhou, Jiliu Zhou orcid.org/0000-0003-4659-8549 College of Computer Science, Chengdu University of Information Technology, Chengdu, ChinaSearch for more papers by this authorImran Mumtaz, Imran Mumtaz orcid.org/0000-0003-4294-2971 University of Agriculture Faisalabad, Faisalabad, PakistanSearch for more papers by this author Yongpeng Ren, Yongpeng Ren orcid.org/0000-0003-1441-3810 College of Computer Science, Chengdu University of Information Technology, Chengdu, ChinaSearch for more papers by this authorHongping Ren, Hongping Ren orcid.org/0000-0001-7257-8547 College of Computer Science, Chengdu University of Information Technology, Chengdu, ChinaSearch for more papers by this authorCanghong Shi, Canghong Shi orcid.org/0000-0002-5188-6230 College of Computer Science, Xihua University, Chengdu, ChinaSearch for more papers by this authorXian Zhang, Xian Zhang orcid.org/0000-0003-4697-9480 College of Computer Science, Chengdu University of Information Technology, Chengdu, ChinaSearch for more papers by this authorXi Wu, Xi Wu orcid.org/0000-0002-7129-4961 College of Computer Science, Chengdu University of Information Technology, Chengdu, ChinaSearch for more papers by this authorXiaojie Li, Corresponding Author Xiaojie Li lixj@cuit.edu.cn orcid.org/0000-0003-3341-4034 College of Computer Science, Chengdu University of Information Technology, Chengdu, China Correspondence Xiaojie Li, College of Computer Science, Chengdu University of Information Technology, 610225 Chengdu, China. Email: lixj@cuit.edu.cnSearch for more papers by this authorJiancheng Lv, Jiancheng Lv orcid.org/0000-0001-6551-3884 College of Computer Science, Sichuan University, Chengdu, ChinaSearch for more papers by this authorJiliu Zhou, Jiliu Zhou orcid.org/0000-0003-4659-8549 College of Computer Science, Chengdu University of Information Technology, Chengdu, ChinaSearch for more papers by this authorImran Mumtaz, Imran Mumtaz orcid.org/0000-0003-4294-2971 University of Agriculture Faisalabad, Faisalabad, PakistanSearch for more papers by this author First published: 24 September 2021 https://doi.org/10.1002/int.22687Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Deep learning has been widely applied into image inpainting. However, traditional image processing methods (i.e., patch-based and diffusion-based methods) generally fail to produce visually natural contents and semantically reasonable structures due to ineffectively processing the high-level semantic information of images. To solve the problem, we propose a stacked generator networks assisted by patch discriminator for image inpainting by multistage. In the proposed method, our generator network mainly consists of three-layer stacked encoder-decoder architecture, which could fuse different level feature information and achieve image inpainting via a coarse-to-fine hierarchical representation. Meanwhile, we split the masked image into different patches in each layer, which could effectively enlarge the receptive field and extract more useful features of images. Moreover, the patch discriminator is introduced to judge the patches of inpainting image are real or fake. In this way, our network can effectively utilize the semantic information to complete a fine result. Furthermore, both perceptual loss and style loss are used to improve the inpainting results in verse. Experimental results on Places2 and Paris StreetView illustrate that our approach could generate high-quality inpainting results, and our method is more effective than the existing image inpainting methods. Volume37, Issue2February 2022Pages 1599-1617 RelatedInformation