Retracted: Jointly network image processing: multi‐task image semantic segmentation of indoor scene based on CNN

IET Image ProcessingVolume 14, Issue 15 p. 3689-3697 THIS ARTICLE HAS BEEN RETRACTEDFree Access Retracted: Jointly network image processing: multi-task image semantic segmentation of indoor scene based on CNN Retraction(s) for this article Retraction: IPR Special Issue Retraction Statement Volume 17Issue 1IET Image Processing pages: 301-301 First Published online: August 22, 2022 Li Huang, Li Huang College of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of ChinaSearch for more papers by this authorMeiling He, Meiling He College of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of ChinaSearch for more papers by this authorChong Tan, Corresponding Author Chong Tan tanchong0116@163.com Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of ChinaSearch for more papers by this authorDu Jiang, Du Jiang Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Hubei Key Laboratory of Mechanical Transmission and Manufacturing Engineering, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Research Center for Biomimetic Robot and Intelligent Measurement and Control, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of ChinaSearch for more papers by this authorGongfa Li, Gongfa Li orcid.org/0000-0002-2695-2742 Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Hubei Key Laboratory of Mechanical Transmission and Manufacturing Engineering, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Research Center for Biomimetic Robot and Intelligent Measurement and Control, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Institute of Precision Manufacturing, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of ChinaSearch for more papers by this authorHui Yu, Hui Yu School of Creative Technologies, University of Portsmouth, Portsmouth, PO1 2UP UKSearch for more papers by this author Li Huang, Li Huang College of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of ChinaSearch for more papers by this authorMeiling He, Meiling He College of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of ChinaSearch for more papers by this authorChong Tan, Corresponding Author Chong Tan tanchong0116@163.com Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of ChinaSearch for more papers by this authorDu Jiang, Du Jiang Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Hubei Key Laboratory of Mechanical Transmission and Manufacturing Engineering, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Research Center for Biomimetic Robot and Intelligent Measurement and Control, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of ChinaSearch for more papers by this authorGongfa Li, Gongfa Li orcid.org/0000-0002-2695-2742 Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Hubei Key Laboratory of Mechanical Transmission and Manufacturing Engineering, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Research Center for Biomimetic Robot and Intelligent Measurement and Control, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of China Institute of Precision Manufacturing, Wuhan University of Science and Technology, Wuhan, 430081 People's Republic of ChinaSearch for more papers by this authorHui Yu, Hui Yu School of Creative Technologies, University of Portsmouth, Portsmouth, PO1 2UP UKSearch for more papers by this author First published: 10 December 2020 https://doi.org/10.1049/iet-ipr.2020.0088Citations: 47AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Image semantic segmentation has always been a research hotspot in the field of robots. Its purpose is to assign different semantic category labels to objects by segmenting different objects. However, in practical applications, in addition to knowing the semantic category information of objects, robots also need to know the position information of objects to complete more complex visual tasks. Aiming at a complex indoor environment, this study designs an image semantic segmentation network framework of joint target detection. Using the parallel operation of adding semantic segmentation branches to the target detection network, it innovatively implements multi-vision task combining object classification, detection and semantic segmentation. By designing a new loss function, adjusting the training using the idea of transfer learning, and finally verifying it on the self-built indoor scene data set, the experiment proves that the method in this study is feasible and effective, and has good robustness. 1 Introduction In recent years, we have clearly noticed the rapid development of Artificial Intelligence (AI), from AlphaGo's defeat of human professional Go players to restaurant smart food service robots, face recognition, unmanned driving etc. We can always feel that AI technology is around us, enriching people's lives and leading the social progress [[1]-[3]]. Computer Vision technology is an important direction in the field of AI research. In short, its purpose is to equip computers with human wisdom eyes to teach computers how to know the world. Computer vision technology is a multidisciplinary research field that uses computers to obtain a high level of understanding of images from images or digital video. Image semantic segmentation refers to the annotation and segmentation of different objects in the image with different colours, and the annotation of semantic information of each pixel in the image, which belongs to the category of image semantic annotation [[4]]. Image semantic segmentation plays an important role in scene understanding, scene recognition, object detection and tracking6. In the scene understanding, we must know what content the scene contains to get the output of image information, and the category and position relationship of the target object in the scene need the image semantic segmentation method to obtain. Target recognition detection and semantic segmentation are complementary. Target recognition is to get the location information of the object, while semantic segmentation can get the content information as well as the location information [[5]-[7]]. Therefore, the combination of target detection and semantic segmentation can achieve more refined visual tasks, which have been widely used in mobile robots, intelligent security, driverless, industrial vision and other fields [[8]-[10]], as shown in Fig. 1. (i) Mobile robot field: a large number of service robots currently appearing, such as handling robots, home service robots etc. image semantic segmentation technology can analyse the scene they are in, move to the target location, and interact with the scene. Fig. 1Open in figure viewerPowerPoint Application of image semantic segmentation in different fields (ii) Intelligent security field: using scene semantic segmentation technology to identify the behaviour mode of the scene and effectively monitor the scene, it can effectively detect abnormal behaviour modes and prevent the occurrence of unexpected situations. (iii) Unmanned driving field: through real-time semantic segmentation (annotation) analysis of traffic scenes, the current road conditions can be obtained, providing accurate road detection information for unmanned driving and assisted driving systems, which can greatly improve the safety of unmanned driving and assisted driving. (iv) Industrial vision field: in the process of industrial production, industrial robots are equipped with vision systems, and the semantic segmentation model can be used to detect products and accurately capture products, thereby improving production efficiency. (v) Virtual reality field: the application of image semantic segmentation technology in the field of virtual reality helps to restore the real information in the scene, thereby bringing a better sense of experience to the player. The main content of this paper is the semantic segmentation of indoor scene images. Compared with the semantic segmentation of a single object, the semantic segmentation of scene images is more difficult. Indoor scenes exist many problems like abundant semantic classes, mutual occlusion between objects, uneven lighting, and similarity of different objects etc. It also incorporates a variety of computer vision technologies such as object detection, recognition, and image segmentation [[11], [12]], making indoor scene semantic segmentation becoming a challenging research direction in image semantic understanding, which is more challenging and difficult than outdoor scenes. The basic goal of indoor scene semantic segmentation is to densely provide a predefined semantic category label for each pixel in a given indoor scene image (or a frame in a video of shooting an indoor scene). As shown in Fig. 2, it is an example image of an indoor scene corresponding to the manually labelled image. Fig. 2Open in figure viewerPowerPoint Semantic segmentation sample of indoor scene (a) Sample image, (b) Corresponding semantic annotation information With the development of deep learning, deep convolutional neural networks (CNNs) have broken through the limitations in object classification, object recognition and object segmentation in the field of image classification, and have shown excellent performance in the semantic segmentation task of images [[13]]. In computer vision research works, whether based on basic theoretical research or applied research, semantic segmentation of images has become a very popular research direction, and many studies have obtained exciting results. However, the study of image semantic segmentation will face different task challenges in different scenarios [[14]]. For example, due to the complexity and variability of indoor scenes, it often leads to different expressions of the same scene, and different scenes may have striking similarities. At the same time, there are a large number of semantic categories in the indoor scene, and the objects are occluded and overlapped. The weak recognition of low-level visual features and the influence of illumination etc. [[15]-[17]], make the semantic segmentation based on indoor scenes to be further studied. The work arrangement of this paper is as follows: Section 1 introduces the research background and significance of this paper, and fully explains the research significance of this paper. Section 2 summarises the traditional image segmentation and image semantic segmentation based on deep learning, and makes a comparative analysis of the research methods and methods used in each development stage of the technology. In Section 3, a multi-task semantic segmentation network framework is designed, and each part of the model is elaborated in detail. Section 4 is the multi-task semantic segmentation experiment based on indoor scene, and analyses the experimental results. Section 5 is the conclusion and future outlook. 2 Related work In the high-level image understanding, the semantic segmentation of images is an important research task, and it has become one of the important research directions in the field of computer vision for decades. The traditional image segmentation mainly includes threshold segmentation method, edge-based segmentation method, region-based segmentation method and clustering segmentation method [[18]-[22]]. Threshold segmentation generally only considers the greyscale information of the image [[23]], so it is only suitable for processing relatively simple images, such as binary images. If some images with similar features and small differences in boundary grey values are segmented, it is difficult to guarantee the segmentation accuracy and the application range is limited. Edge-based segmentation mainly performs image segmentation by detecting edges containing different semantic information. Sobel operator [[24]], Laplacian operator [[25]] and Canny edge detection [[26]] are used to complete the segmentation task by convolving the image. When the image has less noise and the edge grey value changes greatly, a better segmentation effect is obtained, and vice versa. The region-based threshold segmentation is mainly performed by the grey level of the pixel, and the local spatial information of the image is utilised to overcome the image segmentation space small in other methods [[27]]. The essence of the clustering method is to transform the image segmentation problem into a cluster analysis of pattern recognition. The pixels with similar grey values in the image are aggregated into the same class, and then the images are represented as different regions by clustering [[28]]. Image semantic segmentation based on traditional methods mainly uses artificially designed feature extractors to extract low-level features of the image for image segmentation. The output image does not achieve the effect of semantic annotation segmentation, so the image semantic segmentation technology in this period can only be called image segmentation, the demand for real-time semantic segmentation cannot be met at all. Before the introduction of deep learning, the semantic segmentation algorithm completes two main tasks of image segmentation and recognition mainly by constructing a conditional random field model (CRFs). CRFs model was proposed by Lafferty et al. [[29]]. It was first used in text analysis, after continuous development research, it became one of the most successful probabilistic graph models for solving image semantic segmentation problems. Since the CNN can extract the features of the image from a large amount of training sample data, it has been well applied in the problems of image classification/recognition/detection/segmentation and achieved good results. In object detection, Girshick et al. [[30]] proposed R-CNN in 2014 and successfully applied deep learning to target detection. R-CNN follows the idea of traditional target detection, and only partially improves it. It extracts a series of candidate regions that may be objects in advance, and extracts features on candidate regions, which greatly reduces the amount of calculation. After R-CNN, Girshick [[31]] proposed Fast R-CNN in 2015 to Ren et al. [[32]] proposed Faster R-CNN in 2015, and gradually realised end-to-end network training, which not only simplified the network structure but also improved the detection speed and algorithm performance. Meanwhile, Chen et al. [[33]] proposed Fully Convolutional Networks (FCNs) in 2015. FCNs use the convolutional layer instead of the fully connected layer. Through the upsampling and feature layer cropping operations, the problem that the input image size is inconsistent with the predicted image is solved, and the end-to-end image semantic segmentation is realised. Chen et al. [[34]] proposed Deeplab, which uses hole convolution to perform upsampling-intensive prediction tasks, introduces Spatial Pyramid Pooling Structure to segment target objects in multiple sizes, combines DCNN and probability map models to improve object boundary detection. Lin et al. [[35]] proposed a feature pyramid structure to extract multi-scale feature information, and through different levels of feature fusion, the information of small objects can be more completely reflected, thereby improving the accuracy of target detection. In summary, although there are many methods for images semantic segmentation based on deep learning, there are still certain problems in the versatility, segmentation efficiency, accuracy, and visual effects of indoor scene semantic segmentation. Existing methods only perform single-task image segmentation. While in indoor scene understanding, the robot needs to know the position information of the object in addition to the semantic information of the object to better analyse the environment. Therefore, semantic segmentation of indoor scenes based on deep learning needs further study. It can be seen from the above CNN that after the introduction of deep learning, the semantic segmentation of images forms a general framework, which is mainly based on region segmentation. The semantic segmentation process of most images is shown in Fig. 3. Therefore, this paper proposes a multi-task semantic segmentation method combining image semantic segmentation with object detection. The image segmentation and target detection are divided into two branches for parallel computing and shared feature extraction network. The detected network training can quickly determine the target position in the image and select the regional proposal, and then perform pixel-level image semantic segmentation based on this. Fig. 3Open in figure viewerPowerPoint General flow based on deep learning image semantic segmentation 3 Multi-task semantic segmentation of joint target detection 3.1 Fully convolutional networks Conventional CNNs usually connect several fully connected layers after the convolutional layer, and map the feature map generated by the convolutional layer into a fixed-length eigenvector [[36]-[38]]. However, for image semantic level segmentation, the CNN in the form of output vector cannot satisfy the image semantic segmentation task. Therefore, FCNs are proposed to solve the problem of image semantic segmentation. Unlike the classic CNN, which uses a fully connected layer to obtain a fixed-length feature vector (full connection layer + softmax output) after convolution, the FCN can accept input images of any size and the deconvolution layer is used to upsample the last convolved feature map to restore it to the same size as the input image [[39]], thus producing a prediction for each pixel. At the same time, the spatial information in the original input image is preserved. Finally, the pixel-by-pixel classification is performed on the upsampled feature map to realise the end-to-end image semantic segmentation. This method can solve the semantic segmentation task more easily than the traditional method, as shown in Fig. 4. Fig. 4Open in figure viewerPowerPoint CNNs and fully CNNs In the research of scene image understanding, image semantics segmentation can annotation semantic label on every pixel in the target image, thus realising the pixel-level classification of scene image, and completing scene image from low-level features research to high-level image semantics understanding [[40], [41]]. Image semantic understanding is more intuitive than target detection, and the information is richer. It not only includes the shape and size of the object, but also includes the label and position information of the object, and more comprehensively realises the analysis of the scene image. As an important branch of scene understanding, the target detection algorithm can complete the detection of the target position and the specific number in the target image, but cannot detect the object area like the sky, ground, grass and other irregular shapes [[42]]. Although image semantics segmentation can realise the segmentation of the detected objects, it cannot distinguish the individual objects belonging to the same kind of objects, nor can it obtain the specific number of objects [[43]]. To remedy the shortcomings of the above two and better understand the image, this paper proposes a multi-task image segmentation method by combining target detection and image semantics segmentation. The method can perform pixel-level semantic segmentation on the target while achieving target detection, and overcomes the shortcomings of a single task, through experimental verification, good results can be achieved in the target individuals with large differences and small target objects. 3.2 Multi-task semantic segmentation algorithm design Since the traditional semantic segmentation can only interpret the pixel-level semantic information of the target and cannot determine the relative position information of the target, while in the construction of the actual semantic map, the position information of the target is required to draw an accurate scene map. Therefore, this paper designs a multi-task semantic segmentation algorithm (MSSA-RCNN) based on improved FCN, which combines target detection and semantic segmentation. The flow of MSSA is shown in Fig. 5. It can be seen that the MSSA-RCNN algorithm is mainly composed of two parts: target detection framework and FCN-based semantic segmentation branch. The first part of the target detection branch uses the Faster-RCNN algorithm based on the candidate region idea to achieve target detection in semantic segmentation. The second part of semantic segmentation branch, based on FCN, introduces RoI Align to remove the quantisation operation, which better solves the problem of regional mismatch in RoI Pooling twice quantisation so that the pixels in the original image and the pixels in the output image are completely aligned to avoid pixel errors while improving accuracy. Fig. 5Open in figure viewerPowerPoint Algorithm flow of multitasking semantic segmentation The specific process is: the image is input to the CNN to obtain the feature map. The deeper the network layer is, the richer the image features extracted. This paper uses the ResNet101 network, which adds a residual module to the VGG network to make the network deeper and the ability to extract features stronger [[44]-[46]]. Then use a feature pyramid network (RPN) to extract the region candidate bound on the feature map, and use another CNN to further perform feature extraction on the feature map, combined with the region candidate bound to obtain the target region of interest, and finally in each target region RoI Pooling is used to extract features, and the fully connected layer is used to predict the category of the objects in the frame and the regression of the bounding box to complete the detection of the target object. The semantic segmentation branch selects the region of interest and introduces the RoI Align layer to enable each RoI to generate a fixed-size feature map. It uses bilinear interpolation to find a more accurate area [[47]]. Finally, up-sampling the obtained feature map to change the fully connected layer into a fully convolutional layer, reconstruct the spatial information of the image, thus complete the semantic segmentation of the image, and finally realise the multi-task output of target detection and image semantic segmentation. (i) Backbone network: In this paper, a standard neural network ResNet is used as the feature extractor. It is based on the VGG network and adds the residual module [[48]], which makes the feature extraction more powerful. Fig. 6Open in figure viewerPowerPoint Structure of ResNet As shown in Fig. 6, it mainly uses a shortcut connection [[49]]. Define the underlying mapping as G ( x ), so that the superimposed non-linear layer adapts to another mapping H ( x ) = G ( x ) − x, so the original feature mapping is modified as H ( x ) + x, and the whole process can be expressed as follows: y = H ( x , { ω i } ) + x (1) where x , y represent the input and output of the convolution layer, respectively, and function H ( x { ω i } ) is the residual mapping that can be learned. In the residual network, there are two structures as shown in Figs. 7 (1) and (2). Fig. 7 (1) is ResNet50, which is mainly used in the residual network with fewer convolution layers; Fig. 7 (2) is ResNet101, which is mainly used in the network with more convolution layers [[50]]. (ii) Regional proposal network (RPN): RPN is a lightweight neural network, which mainly uses a sliding window to scan the region to find the target in the image [[51]]. After RPN network scanning, a series of rectangular boxes will be output in the area, and RPN will generate two outputs for each rectangular box, object category (foreground or background) and rectangular box fine-tuning. Using RPN network prediction, the best rectangular box containing the target object can be selected. If there are multiple rectangular boxes overlapping with each other, the rectangular box with the highest foreground score will be retained. Non-maximum suppression (NMS) algorithm is used to fine-tune its position and scale to get the final regional proposal, and then it will be transferred to the next stage [[52], [53]]. The principle of the NMS algorithm is shown in Fig. 8. The four types of objects in the picture are the chair, book, table, and keyboard. Each type of object has N overlapping rectangular frames containing objects. The target detection finally outputs the target's position and prediction confidence (scores). The NMS algorithm screens the rectangle frame according to the score from high to low, and the detection frame with a lower score will be discarded under a certain threshold intersection-over-union (IoU) condition. Repeat until you find the optimal target rectangle [[54], [55]]. Fig. 7Open in figure viewerPowerPoint Two structures of residual network Fig. 8Open in figure viewerPowerPoint Schematic diagram of the NMS algorithm IoU is a concept commonly used in target detection, which represents the overlap rate between the generated candidate bound and the ground truth bound, that is, the ratio of their intersection and union. The most ideal situation is that their ratio is equal to 1, that is, completely overlapping, as shown in Fig. 9. In the figure, IoU defines the overlapping degree of the bounding boxes of the two regions C and G. In short, the overlapping area between the two boxes (the green part in the figure) accounts for the total area of the two boxes. The formula is defined as (): IOU = area C ∩ area G area C ∪ area G (2) The MSSA in this paper is to use IoU to determine the overlap between these candidate bounds and the real artificial labelled bounds after obtaining a large number of target candidate bounds. The threshold is set to 0.5. If IoU > 0.5, these candidate bounds are the default target objects. Otherwise, it should be considered as the background area. Fig. 9Open in figure viewerPowerPoint IoU schematic diagram 3.3 RoI Pooling and RoI align In common two-level detection frameworks (such as Fast RCNN, Faster RCNN etc.), RoI Pooling is used to pool the corresponding region into a fixed-size feature map according to the location coordinates of the preselected box, to facilitate subsequent classification and boundary box regression operations. The position of the preselection box is usually obtained by model regression, which is generally a float point number, but the pooled feature map requires a fixed size, so RoI Pooling has two quantisation processes [[56]], as shown in Fig. 10 is the Faster RCNN detection framework. Two quantification process of RoI Pool will lead to the deviation of the pixel, taking the feature scaling step size 32 as an example, the deviation of 0.1 pixels on the feature image is the deviation of 3.2 pixels on the original image, so the deviation of 0.8 pixels in Fig. 9, in the original picture, the difference of 30 pixels is approached, which will have a huge impact on the detection of small target objects. Fig. 10Open in figure viewerPowerPoint Schematic diagram of RoI Pooling To solve the shortcomings of RoI Pooling, this paper introduces the RoI Align layer, which eliminates the strict quantisation operation and uses the bilinear interpolation method to obtain the image values on the coordinates of the floating point pixels, and transforms the entire feature aggregation process into a continuous operation [[57], [58]], as shown in Fig. 11. Specifically, RoI Align traverses each candidate region, keeps the floating-point boundary without quantisation, and then divides the candidate region into k×k units, and the boundary of each unit is also not quantised. Finally, four fixed coordinate positions are calculated in each unit, and the values of these four positions are calculated by bilinear interpolation to perform the ma