Semantic combined network for zero‐shot scene parsing

IET Image ProcessingVolume 14, Issue 4 p. 757-765 Research ArticleFree Access Semantic combined network for zero-shot scene parsing Yinduo Wang, Yinduo Wang School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorHaofeng Zhang, Corresponding Author Haofeng Zhang zhanghf@njust.edu.cn orcid.org/0000-0002-4039-7618 School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorShidong Wang, Shidong Wang School of Computing Sciences, University of East Anglia, Norwich, UKSearch for more papers by this authorYang Long, Yang Long Department of Computer Science, Durham University, Durham, UKSearch for more papers by this authorLongzhi Yang, Longzhi Yang Department of Computer and Information Science, Northumbria University, Newcastle upon Tyne, UKSearch for more papers by this author Yinduo Wang, Yinduo Wang School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorHaofeng Zhang, Corresponding Author Haofeng Zhang zhanghf@njust.edu.cn orcid.org/0000-0002-4039-7618 School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, People's Republic of ChinaSearch for more papers by this authorShidong Wang, Shidong Wang School of Computing Sciences, University of East Anglia, Norwich, UKSearch for more papers by this authorYang Long, Yang Long Department of Computer Science, Durham University, Durham, UKSearch for more papers by this authorLongzhi Yang, Longzhi Yang Department of Computer and Information Science, Northumbria University, Newcastle upon Tyne, UKSearch for more papers by this author First published: 19 February 2020 https://doi.org/10.1049/iet-ipr.2019.0870AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Recently, image-based scene parsing has attracted increasing attention due to its wide application. However, conventional models can only be valid on images with the same domain of the training set and are typically trained using discrete and meaningless labels. Inspired by the traditional zero-shot learning methods which employ auxiliary side information to bridge the source and target domains, the authors propose a novel framework called semantic combined network (SCN), which aims at learning a scene parsing model only from the images of the seen classes while targeting on the unseen ones. In addition, with the assistance of semantic embeddings of classes, the proposed SCN can further improve the performances of traditional fully supervised scene parsing methods. Extensive experiments are conducted on the data set Cityscapes, and the results show that the proposed SCN can perform well on both zero-shot scene parsing (ZSSP) and generalised ZSSP settings based on several state-of-the-art scenes parsing architectures. Furthermore, the authors test the proposed model under the traditional fully supervised setting and the results show that the proposed SCN can also significantly improve the performances of the original network models. 1 Introduction In the last few decades, image-based scene analysis has become one of the fundamental tasks of computer vision. It can be used in a large number of applications such as image editing and autonomous navigation, and attracts an increasing number of researchers to endeavour in it. The purpose of this task is to classify each pixel in the image by assigning a discrete tag to it. Recently, due to the rapid development of deep convolutional neural networks (CNNs) [1, 2], many efforts have been devoted to employ CNNs to improve the performance of scene parsing, such as fully convolutional network (FCN) [3], pyramid scene parsing network (PSPNet) [4] and semantic image segmentation with deep convolutional nets (DeepLab) [5], which can significantly outperform traditional feature extraction methods such as conditional random field (CRF) [6]. However, in the era of big data, an increasing number of a new scene categories are emerging. The traditional full supervision scene analysis methods are suffering from a serious problem, i.e. when a new category of image appears the model should be retrained by adding the samples of the new class. For example, if we have trained a fully supervised model with the images captured from a town scene, when the model is applied in a city scene, there will be many unseen categories such as 'skyscraper', which makes the model inoperable. Therefore, we need to acquire a large number of images from the new categories and combine them into the original data set to train a new model, which is a labour cost and difficult to be implemented. Conventional Zero-Shot Learning (ZSL) [7-10] is inspired by the behaviour of our human beings that when we meet new categories, we always employ some intermediate information to build up a bridge from seen to unseen categories. Therefore, some semantic vectors such as attribute annotated by experts [11], have been utilised as the auxiliary information to achieve the purpose of recognising novel categories. As shown in Fig. 1b, following the same strategy of conventional ZSL, zero-shot scene segmentation is first proposed by [12], which aims to train models only dependent on the annotated data within seen classes, but can be applied on a disjoint data set of unseen classes. In addition, inspired by conventional generalised ZSL (GZSL), as shown in Fig. 1c, in this paper we introduce a more realistic task generalised zero-shot scene parsing (GZSSP), which enlarges the search spaces into both seen and unseen classes. Fig. 1Open in figure viewerPowerPoint An illustration of task ZSSP and our introduced GZSSP (a) Traditional scene parsing, (b) ZSSP, (c) GZSSP Moreover, most of the state-of-the-art scene parsing networks adopt the same strategy that the labels of all pixels are discrete and barely have relations with each other. The labels used during training are always integers, e.g. the category 'person' is annotated as 1, and 'rider' as 2. With such operation, it can be easily found that the difference between the label 'person' and the label 'rider' is same as that between 'person' and 'building'. However, we all know that the difference between 'person' and 'rider' is much smaller than that between 'person' and 'building' in realistic scenarios. Therefore, we argue that the labels are related, which should be adopted in scene parsing networks. In order to solve the aforementioned problem, we propose a novel deep architecture called semantic combined network (SCN), which combines the word embedding of classes into classification networks. Specifically, we no longer consider labels as meaningless values, but instead exploit a higher-dimensional attribute vector for each label, such as Word2Vec [13], which allows us to be able to work on both zero-shot scene parsing (ZSSP) and GZSSP settings at the same time. Therefore, employing the relationship between classes is much better to improve the performance of the corresponding original network. In this paper, based on several state-of-the-art architectures such as FCN, PSPNet and DeepLab, we utilise the proposed SCN to solve both ZSSP and GZSSP tasks. During the experiments, we randomly split the classes of Cityscapes [14] into two disjoint parts, including seen classes and unseen classes, and conduct extensive experiments on them. Besides, due to the combination of semantic information, we also test our SCN under the traditional fully supervised setting. The contributions of this work are as follows: • We propose a novel and effective SCN, which integrates the labels' semantic embeddings (SEs) into scene parsing network to solve the more challenging task, ZSSP. • Inspired by GZSL, we introduce a more realistic task, GZSSP, which enlarges the search space from seen classes to all classes. • Instead of directly employing meaningless and unrelated labels, we combine the semantic information of each class into several state-of-the-art networks, and make them feasible for new ZSSP and GZSSP tasks. • Extensive experiments are conducted on all the three settings, including ZSSP, GZSSP and conventional fully supervised learning, and the experimental results show that our SCN can not only well solve the tasks of ZSSP and GZSSP, but also significantly improve the performance of the state-of-the-art methods under the conventional supervised setting. The main content of this paper is organised as follows: in Section 2 we briefly introduce the existing methods for scene parsing and ZSL. Section 3 describes the proposed method in detail. Section 4 gives the experimental results of the comparison with existing methods for ZSSP, GZSSP, and traditional scene parsing. Finally, we conclude this paper in Section 5. 2 Related work 2.1 Scene parsing and semantic segmentation Scene parsing, based on semantic pixel-wise segmentation, is an ongoing hot topic research [15-17]. Conventional methods often utilise contextual information of raw data or hand-craft feature for scene parsing, such as CRF [6], which combines multi-scale component to refine the performance. Based on CRF, Lucchi et al. [18] tried to make full use of global image-level prior to scene parsing. Based on some early efforts such as VGG [19], GoogLeNet [20], AlexNet on object classification with CNNs, FCN [3] replaces the last fully connected layer of them with series of deconvolution layers to classify each pixel, and achieves great success. After realising CNNs have such great potential and effect in this field, a large number of researchers have been endeavouring in it. Some works such as DilateNet [21] and DeepLab [5] are dedicated to enlarging the receptive field of convolutional layers to improve the performance, and PSPNet [4] has made a big progress by integrating the contexts of different scales. To refine the results, some methods such as Deeplab [5] and CRFasRNN [22] enhance the post-processing of the networks to improve segmentation accuracy. 2.2 Zero shot learning ZSL can be simply classified into inductive ZSL and transductive ZSL. Inductive ZSL methods are only trained with labelled seen data and unlabelled unseen data is strictly inaccessible, while transductive ZSL, which is first proposed by Fu et al. [23], learns a model with both labelled seen and unlabeled unseen data. For inductive ZSL, since the visual attribute learning proposed by [11], many researchers conduct their work on establishing a visual to attribute projection to find the relationship between seen and unseen classes. Early efforts like DAP [24] learns probabilistic attribute classifiers to predict the label. ALE [25], SJE [26], and DEVISE [27] project features into a SE space by employing bilinear compatibility functions. CONSE [28] and semantic similarity embedding [29] exploit seen classes to improve the performance and reduce the usage of manual attributes by constructing the attributes of unseen classes. Near recently, Long et al. [30] and Li et al. [31] propose a generative framework that constructs a projection from attributes of seen classes to visual features, and then utilise the attributes of unseen classes to synthesise unseen visual features, finally a supervised model is trained for all classes. For transductive ZSL, semi-supervised framework [32] learns a multi-class classification model on all classes jointly with both labelled data and unlabeled data as input. Song et al. in [33] designed two independent objective functions for seen data and unseen data; respectively, and integrates them into a whole framework during the training phase so as to employ the powerful feature extraction ability of deep architecture. 2.3 Generalised zero shot learning ZSL assumes that the ascription of test data is known in advance, thus search space of the nearest neighbour can be restricted on only unseen classes. Chao et al. [34] first argued that under more practical situations, the convention ZSL is unreasonable, because we cannot obtain the knowledge whether the test data belongs to unseen classes or seen classes beforehand in most circumstances. Therefore, they propose the new task – GZSL, which assumes that the search space of the nearest neighbour should be extended to both seen and unseen classes. Recently, Zhang et al. in [35] considered GZSL problem as a triple verification problem and a novel optimisation of regression and compatibility function is proposed to solve this problem. Subsequently, Xian et al. [36] put forward a new standard split of several popular data sets for GZSL testing, and release a benchmark of some recent ZSL methods, which makes the later researchers more convenient and has greatly promoted the development of ZSL. 2.4 Semantic embeddings Current ZSL methods always rely on the intermediate attributes, which represent the SEs of both seen and unseen classes. Conventional attributes [37, 38] are usually annotated by experts with real values. This type of annotation needs experts' prior knowledge, and will cost a lot of manpower. Therefore, recent methods such as Attributes2Classname [39] start to use Word2Vec [13] to automatically generate attributes based on the data set 'WikiPedia'. However, this type of textual description often leads to serious performance degradation because it is not directly related to visual appearance. Another semantic attribute representation is based on similarity, which can be annotated by humans [40, 41] or the textual descriptions [29, 42]. 3 Methodology In this section, we will first give the definitions of our tasks on three different settings, including ZSSP, GZSSP and fully supervised learning, and then describe the proposed SCN in details. 3.1 Problem definitions 3.1.1 Preliminaries Given a set of images X, including training part and testing part , and all the classes of the images are denoted as , where represents the set of seen classes, and stands for the set of unseen classes. To fulfil the requirement of the tasks, we divide all the image pixels into four parts: training pixels of seen classes in are represented by and test pixels of seen classes in are denoted as . Similarly, the training pixels in unseen classes are denoted as and the test pixels of unseen classes are represented as . Now, we can formally describe the scene parsing tasks with the following three types. 3.1.2 Zero shot scene parsing In ZSSP task, the given data only includes the pixels of seen classes in and their corresponding labels S during training, we need to train a function , which can be transferred to unseen classes to predict the labels of . It should be noted that the search space is restricted to unseen classes only. 3.1.3 Generalised ZSSP Similar to ZSSP, only is given for GZSSP task in training phase to learn a function to predict the labels of both and . In addition, different from ZSSP, we are supposed to be ignorant of whether the test pixel belongs to seen classes or unseen classes, so the search scope of this task should be conducted on both seen and unseen classes. 3.1.4 Traditional fully supervised scene parsing Given the pixels of all classes and their corresponding labels D, the purpose of this task is to train an optimal projection model to assign each pixel with a label, and then the trained model can be applied on test data . 3.2 Model architecture Since our SCN can be compatible with many architectures such as FCN, for the sake of simplicity of description, we here first take FCN as an example and the extensions on PSPNet and DeepLab will be described later. The whole architecture of our proposed SCN is illustrated in Fig. 2. Since SCN adopts the semantics of labels, we divide our framework into two parts, one is feature extraction network (FEN) to extract the high dimensional feature from input images, another is semantic embedded network (SEN), which is exploited to compute the SEs from labels and generate the category probability. Fig. 2Open in figure viewerPowerPoint An illustration of the framework of our proposed network. Besides the traditional CNN-based scene parsing architectures, we add another branch to extract semantic information of labels to meet the ZSSP and GZSSP task. In the Word2Vec part, we employ an open-source pre-trained word embedding model to directly extract the word vectors for each class as the input to the following computation. During attribute map construction, the inner product of a pixel's visual attribute and each class's semantic attribute are calculated to generate the similarity of this pixel to all classes (a) Word concepts, (b) Input image, (c) Feature map, (d) Attribute map, (e) Final prediction, (f) Word2Vec 3.2.1 Feature extraction network In our framework, as shown in the grey block of Fig. 2, an image x is first input into the FEN to generate high dimensional features for each pixel. Here, the FEN can be replaced by many other scene parsing networks, such as FCN, PSPNet and DeepLab, by removing the final softmax layer. In FEN, the input images first pass through a VGG module and generate small scale feature, and then several deconvolutional blocks are added to scale the feature map to the same size as the input image, where the channel number equals the number of categories. Additionally, in order to obtain image information with different resolutions, some skip connections are added in the FEN, which can be seen in the grey block of Fig. 2. We first de-convolute the last layer of VGG into the same size of 'conv4', and then add the result and 'conv4' in element-wise to generate 'deconv4', which is subsequently de-convoluted into the same size of 'conv3', and added by 'conv3' in element-wise to get 'deconv3'. Subsequently, the 'deconv3' is up-sampled and convoluted to get the feature map with the same size as the original input image. For the convenience of being replaced by FCN or other architectures, we add an extra convolutional layer to generate the final coarse-category prediction map with the same channels as the number of categories. The final prediction map is denoted as . 3.2.2 Semantic embedded network SEN, the critical part of our network, is designed to extract the deep discriminative semantic representation information of labels to assist the classification of our method. The blue block in Fig. 2 is the framework of our SEN, which can be divided into two parts, one is utilised to extract semantic information of labels and another is for generating the image semantics in our network. In the first part, once we have obtained the ground-truth of the images, we will know the exact name of each category. Through the procedure of Word2Vec, which is a network pre-trained from large text corpora, we can obtain a M-dimensional vector by taking a label name as input. Therefore, for every single class , we can generate a M-dimensional vector by Word2Vec, and finally achieve an dimensional semantic matrix for all classes. In the second part, in order to make the coarse-category map to be consistent with dimension of the extracted semantic vectors, we convolute the coarse-category prediction map , which is the final output of FEN with channels, with a kernel to generate a high dimensional semantic map , where, the dimensionality of is , and has its dimensionality of . So far, each pixel in the image has an M-dimensional vector to represent its deep semantic information. Thus, the probability of each pixel belongs to a certain category can be calculated by applying an Inner Product and Softmax, which can be represented as, (1)Where is the probability of the pixel belongs to the th class. is the th entry of the matrix . 3.3 Settings 3.3.1 ZSSP setting It is noteworthy that the pixel of unseen classes should not be included during ZSSP training. Since and might be found in the same image, we construct a mask to block the pixels belong to unseen classes when calculation the cross-entropy loss to fulfil this task. We only choose the part of seen classes of semantic matrix for train, and denote it as , which is subsequently multiplied with the SEs of seen classes. We use the same formula as (1) to calculate the probability of each pixel by replacing and with and , respectively. And the cross-entropy loss function can be defined for ZSSP task as follows, (2)where, is the corresponding label of . During testing, we extract unseen classes semantics and denote them as a matrix . Given a test pixel in an image x, we take x to pass through our SCN, where the word semantics uses , and obtain the final probability of the pixel for all unseen classes U. The prediction of ZSL is calculated as (3) For the seen part of test images , we can easily predict their labels with the same process of the fully supervised task. 3.3.2 GZSSP setting In ZSSP setting, we previously assumed that the test images can be divided into seen part and unseen part, thus for the unseen part, we just need to find the nearest neighbour only on unseen classes. However, ZSSP is usually unreasonable in realistic scenarios because we cannot know whether the pixel belongs to seen or unseen classes in advance. Therefore, generalising the classification on all classes is necessary, and we call this task GZSSP. In this task, we calculate three metrics, test seen accuracy , test unseen accuracy and harmonic accuracy H. and represents class mean average precision (mAP) from test seen data and test unseen data respectively and H is defined as (4) 3.3.3 Fully supervised scene parsing setting For the traditional fully supervised scene parsing setting, the sample pixels of both seen classes and unseen classes can be obtained during training, and we also train the whole framework using cross-entropy loss. The loss function can be represented as follows: (5)where is the corresponding label of , and is cth entry of the one-hot vector label of pixel . During testing, we predict the label of a pixel by finding its nearest class based on the probability of different class embeddings, which can be easily generated from the last layer of our SCN with an argmax method, (6) 3.4 Other types of FEN Since our SCN is an extended architecture based on conventional deep scene parsing models by adopting the word SEs, thus it is convenient to substitute the FEN with other scene parsing frameworks such as PSPNet and DeepLab, and the replaced models can be found in Fig. 3, where SE represents semantic embedded. For the SCN with PSPNet architecture, the upper branch in Fig. 3, we calculate the final model loss after the inner product of feature semantics and word embeddings of classes. And the lower branch in Fig. 3 is SE-Deeplab, instead of fusing all four atrous convolution layers directly to generate a prediction map, we project them to semantic space and then fuse the output results, which is multiplied by the word embeddings afterwards to yield the class probability of each pixel. Fig. 3Open in figure viewerPowerPoint An illustration of task ZSSP and our introduced GZSSP (a) SE-PSP, (b) SE-Deeplab 4 Experiments 4.1 Data set We employ the data set Cityscapes [14] in our experiments. Cityscapes is a semantic urban scene parsing data set, and it is released recently for pixel-wise scene parsing and instance annotation. The data set is comprised of a large, veritable set of stereo video sequences, among which there are 5000 images annotated with pixel-level high quality annotations. In the data set, there are 19 predefined categories containing both stuff and objects. In our experiment, 3475 images are assigned as train set and the left 1525 images are treated as a test set. For the tasks of ZSSP and GZSSP, we randomly pick five classes as unseen classes and the remaining 14 as seen classes, and the results recorded in Tables 1 and 2 are the average values of ten executions. Table 1. Experiment of SCN for ZSSP with five unseen classes. Bold font is for the best result Method PA, % mAP, % SE-FCN 85.91() 53.93() SE-PSP 90.41() 59.76 () SE-Deeplab 91.54() 62.58() Table 2. Average result of SCN for GZSSP with five unseen classes. Bold font is for the best result Method ts, % tr, % H SE-FCN 42.83 54.98 48.15 SE-PSP 42.74 68.90 52.76 SE-Deeplab 45.03 64.35 52.98 4.2 Evaluation metrics For evaluation, there are four kinds of metrics used in our scene parsing experiments. For evaluating the precision of the tasks ZSSP and GZSSP, mAP, pixel-wise accuracy (Pixel Acc.), and H, which is introduced above, are employed in our experiments. Besides, for traditional fully supervised scene parsing task, we utilise the mean of class-wise intersection over union (mIoU). 4.3 Implementation details Since details are always important for training a neural network, we describe the settings of our SCN in this paragraph. Besides, for the convenience of description and comparison, we name our frameworks as SE-FCN (SE for Semantic Embedded), SE-PSP and SE-Deeplab, respectively. We train our models with SEN fixed and FEN fine-tuned. Noted that during the experiment of SE-FCN, because the size of the original image is , we need bytes to store the generated matrix of feature semantics, which is too big for a single GPU such as GTX 1080Ti. Therefore, we resize the original input images to of their original height and width by bi-linear interpolation, which might lead to performance degradation for information loss during interpolation, but we still observe a significant performance improvement compared to their original methods, which can prove the effectiveness of our architecture. In addition, we use a constant learning rate, which is set to , and the iteration time is set to for SE-FCN and for SE-PSP and SE-Deeplab. We use Adam optimiser for SE-FCN and the momentum is set to 0.9, while SE-PSP and SE-Deeplab use Momentum optimiser and the momentums are both set to 0.9. Due to the limited physical memory on our GPU card, we set the batch-size to 1 during training. Hyper-parameter that controls the weight of regularisation in SE-PSP and SE-Deeplab is set to 0.4, which is the same as that in the original PSPNet and Deeplab. 4.4 Zero shot scene parsing In this section, we mainly focus on the performance of our SCN on the setting of ZSSP. Here, we do not compare with other ZSL methods on this task, because our concentration is not focused on the design of ZSL algorithms but the application scenario of ZSSP task, and this method can be considered as a baseline for coming researchers. In our experiment, since there is not a standard public split for seen–unseen classes, we random-repeatedly choose five classes as unseen classes, which are not employed during the following training phase. We calculate the averages of mAP and PA of ten executions, and record the average of them in Table 1, and the visual results are shown in Fig. 4. It should be noted that since we have random-repeatedly picked out five classes as unseen classes, thus the unseen classes in each raw in Fig. 4 may be different. In addition, the same experiment setting and calculation is exploited in the experiments for GZSSP in the following section. Fig. 4Open in figure viewerPowerPoint Visual results of ZSSP. The rows from top to bottom are the prediction with SE-FCN, SE-DeepLab, and SE-PSPNet, respectively (a) Ground truth, (b) Seen classes, (c) Unseen classes As shown in Table 1, we classify the pixels into five unseen Categories and the results obtained by different structures have only a little difference. Among them, SE-Deeplab has the best classification result, and can reach 62.58% for mAP, and it also has the smallest fluctuation range for the different splits of unseen classes. Since there is a big unbalance between the number of pixels of each category, which often leads to poor predictions in a category with fewer pixels, and finally causes the degradation of the all class mAP value. However, although the mAP in Table 1 is not so high, our method can still make a good prediction. As shown in Fig. 4, we can find that although there is some noise in prediction, our method classifies most of the pixels correctly, e.g. 'terrain' and 'rider' in the first row, 'rider' and 'building' in the second row, and 'car' and 'sidewalk' in the last one. 4.5 Generalised zero shot scene parsing GZSSP is a more practical task in realistic scenarios. We use the same settings as that in ZSSP, but different searching strategies for classification. In ZSSP, the search range is fixed on five unseen classes, while in GZSSP it is relaxed to all 19 classes. Since H is calculated by and , here we directly compute the final average results of (short for ), (short for ) and H, which are recorded in Table 2, and show the visual category maps in Fig. 5. Fig. 5Open in figure viewerPowerPoint Some examples of our GZSSP. The first and second row are predictions from SE-PSP, and the last row is from SE-Deeplab (a) Image, (b) Ground truth, (c) Unseen classes, (d) All classes From Table 2, we can discover that our SCN can obtain good resul