Dilated Convolutional Pixels Affinity Network for Weakly Supervised Semantic Segmentation

Chinese Journal of ElectronicsVolume 30, Issue 6 p. 1120-1130 Original Research PaperFree Access Dilated Convolutional Pixels Affinity Network for Weakly Supervised Semantic Segmentation ZHANG Zhe, ZHANG Zhe zhangzhe18@mails.jlu.edu.cn College of Computer Science and Technology, Jilin University, Changchun, 130012 China Key Laboratory for Symbol Computation and Knowledge Engineering of National Education Ministry, Changchun, 130012 ChinaSearch for more papers by this authorWANG Bilin, WANG Bilin College of Computer Science and Technology, Jilin University, Changchun, 130012 China Key Laboratory for Symbol Computation and Knowledge Engineering of National Education Ministry, Changchun, 130012 ChinaSearch for more papers by this authorYU Zhezhou, Corresponding Author YU Zhezhou yuzz@jlu.edu.cn College of Computer Science and Technology, Jilin University, Changchun, 130012 China Key Laboratory for Symbol Computation and Knowledge Engineering of National Education Ministry, Changchun, 130012 ChinaSearch for more papers by this authorLI Zhiyuan, LI Zhiyuan College of Computer Science and Technology, Jilin University, Changchun, 130012 ChinaSearch for more papers by this author ZHANG Zhe, ZHANG Zhe zhangzhe18@mails.jlu.edu.cn College of Computer Science and Technology, Jilin University, Changchun, 130012 China Key Laboratory for Symbol Computation and Knowledge Engineering of National Education Ministry, Changchun, 130012 ChinaSearch for more papers by this authorWANG Bilin, WANG Bilin College of Computer Science and Technology, Jilin University, Changchun, 130012 China Key Laboratory for Symbol Computation and Knowledge Engineering of National Education Ministry, Changchun, 130012 ChinaSearch for more papers by this authorYU Zhezhou, Corresponding Author YU Zhezhou yuzz@jlu.edu.cn College of Computer Science and Technology, Jilin University, Changchun, 130012 China Key Laboratory for Symbol Computation and Knowledge Engineering of National Education Ministry, Changchun, 130012 ChinaSearch for more papers by this authorLI Zhiyuan, LI Zhiyuan College of Computer Science and Technology, Jilin University, Changchun, 130012 ChinaSearch for more papers by this author First published: 01 November 2021 https://doi.org/10.1049/cje.2021.08.007 This work is supported by the Development Project of Jilin Province of China (No.20200801033GH, No.2020122328JC), Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University (No.20180520017JH), and the Fundamental Research Funds for the Central Universities, JLU. AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract This paper studies semantic segmentation primarily under image-level weak-supervision. Most state-of-the-art technologies have recently used deep classification networks to create small and sparse discriminatory seed regions of each interest target as pseudo-labels for training segmentation networks, which achieve inferior performance compared with the fully supervised setting. We propose a Dilated convolutional pixels affinity network (DCPAN) to localize and expand the seed regions of objects to bridge this gap. Although introduced dilated convolutional units enable capture of additional location information of objects, it falsely highlighted true negative regions as dilated rate enlarge. To address this problem, we properly integrate dilated convolutional units with different dilated rates and self-attention mechanisms to obtain pixel affinity measure matrix for promoting classification network to generate high-quality object seed regions as pseudo-labels; thus, the performance of the segmentation network is boosted. Furthermore, although our approach seems simple, our method obtains a competitive performance, and experiments show that the performance of DCPAN outperforms other state-of-art approaches in weakly-supervised settings, which only use image-level labels on the Pascal VOC 2012 dataset. I. Introduction Deep convolutional neural networks (DCNNs) have recently achieved considerable success in fully supervised semantic segmentation of images[1,2], which require a significant quantity of full-labeled training images. However, full-labeled images need to annotate every pixel of the image, which results in a large time-consuming and laborious. The bottleneck in semantic segmentation research has become the creation of training data sets. Therefore, how to obtain pixels-level annotations of a given image cheaply and more efficiently is a promising direction for semantic segmentation in the future, such as under bounding boxes[3], scribbles[4], points[5], and image-level[6-11] supervision. This paper mainly study weakly supervised image semantic segmentation, which only using the weakest image-level annotations. It is worth noting that finely annotating an image with pixel-level labels takes about 1.5h[12], but annotating an image with image-level labels takes about 1s per class[13]. Unlike pixel-level label make image semantic segmentation model to learn about object boundary and relationship between their components in the image, the image-level label only indicates objects of class labels appear in the map without any information about boundary and location. The critical issue, therefore, is to create the correspondence between image-level labels and information of position. By using the Class activation maps[14] (CAM) produced by the deep classification network, which can highlight the most discriminative location of objects in the image, most weakly supervised methods of image semantic segmentation [15-19] solve this problem by adopting these location maps as pseudo pixel-level labels to train fully supervised semantic segmentation methods. However, as shown in Fig.1(a), the CAM-based localization maps focus only on a small and sparse discriminative region of objects, e.g., the head of a dog, which deviates from the semantic segmentation requirement for the creation of complete and dense object seed regions. In order to enhance the efficiency of weakly surprised segmentation methods, it is necessary to highlight other parts of the object non-discriminative region in an image and produce denser and complete object seed regions as pseudo-labels for training semantic segmentation networks. Fig. 1Open in figure viewerPowerPoint Comparisons of CAMs producedby deep classification network with different setting for a given input map. (a) represents CAMs produced by conventional classification network; (b) presents CAMs generated by classification network with dilated convolutional units; (c) shows the CAMs predicted by our DCPAN And for that, we propose a Dilated convolutional pixels affinity network (DCPAN) to address this problem, motivated by the most important advantage of dilated convolution[2,20], which can effectively expand the receptive fields of convolutional kernels and without any additional parameters or amount of computation, meanwhile utilizing self-attention mechanism to catch long-range contextual dependence between feature maps from the standard convolutional unit (dilated rate = 1) and different dilated convolutional units (dilated rate = 4, 8). Recently, dilated convolution and self-attention mechanism has been used for processing the problem of fully-supervised semantic segmentation[2,20-22] and making the performance of semantic segmentation network improved which proved the validity of these approaches. So that we introduce DCPAN with vary dilated rates into the deep classification network to produce denser object seed regions, as shown in Fig.1(c). And then adopt these object seed regions, which overlap more complete and denser regions of objects as pseudo labels for training semantic segmentation networks. Concretely, we start with a deep classification network[23-25] with multiple dilated rates of convolutions unit to produce seed region. In normal, initial object seed regions produced by the deep classification network are small and sparse but with high precision, not enough as pseudo labels to train the image semantic segmentation network. But through dilated convolutions unit to enlarge the receptive fields of convolutional kernels, the more surrounding parts of discriminative regions can be perceived. More non-discriminative parts of objects can be found be able to produce denser seed regions. However, with the rise of dilated rate, convolutional kernels will achieve larger receptive fields that can perceive more background information rather than objects, resulting in some true negative object seed regions being falsely highlighted, as shown in Fig.1(b). To improve this limitation, we introduce a self-attention mechanism into the dilated convolutional unit to prevent background regions around small size objects and some irrelated regions of objects are falsely highlighted, which capture long-range contextual dependence between any locations of feature maps produced by standard convolutional unit and different dilated convolutional units respectively to enhance the relevant object regions and suppress the other irrelated regions. Finally, we use the output of the classification network augmented by the proposed dilated convolutional pixels affinity method, which produces denser and complete object seed regions as pseudo annotations for the image semantic segmentation network training. Moreover, in the experiments, we testify that our method is validated on the challenging PASCAL VOC 2012 Segmentation benchmark[26] for image-level weakly-supervised image semantic segmentation method. The key contributors to our work can be listed as follows: We propose a dilated convolutional pixels affinity network, which enable deep classification network effectively produce high-quality object seed regions as pseudo annotations for training semantic segmentation network. We test the efficacy of proposed DCPAN in generating high-quality seed object regions and the reasonableness of experimental decision-making in the semantic segmentation. Our approach only uses image-level labels on the PASCAL VOC 2012 benchmark, which attain state-of-the-art results for weakly-supervised semantic segmentation and obtain competitive performance for other weakly-supervised methods using stronger supervision or extra information. The remainder of this paper is structured as follows: First, we present related work in Section II, and we introduce the specifics of our approach in Section III. In the experiment, Section IV gives an in-depth analysis of the dilated convolutional pixels affinity network. Finally, we draw the conclusion in section V. II. Related Works The research of semantic segmentation[1,2,20,21,27] has been achieved great process in the last few years, which mainly due to advance of learning framework[28] based on DCNNs. The work of Long et al.[1] is the initial to introduce Fully convolutional neural networks (FCNs) framework into semantic segmentation. Follow up this work, various methods have been introduced to improve the precision of semantic segmentation by fusing multi-scale feature maps[21], enlarging the receptive fields[2,20], implementing a deeper encoder-decoder structure network[27]. However, training a DCNN-based image semantic segmentation model would require a considerable number of pixel-level annotations, which will take a lot of time-consuming and laborious. For this reason, there are many semantic segmentation approaches with various types of weak supervision that have been proposed, which can efficiently reduce the effects of pixel-level annotations, e.g., bounding boxes[3], scribbles[4], and points[5], etc. Nevertheless, these types of weak supervision still require human efforts during label procedure. This article concentrates only on the weakest supervision, image-level labels, which require minimal or no human efforts for annotation. Image-level annotations as supervision have been widely used for semantic segmentation since it is easy to obtain. Papandreou et al.[29] suggested a method based on the expectation-maximization algorithm to predict semantic foreground and background pixels dynamically. The Log-sum-exp (LSE) approach was introduced by Pinheiro et al.[6], which places more weight on the pixels that are important during training for classifying images. Qi et al.[7] adopted the proposal of objects from MCG[30] to improve semantic segmentation performance, which forms a unified framework including object location and segmentation branches. Wei et al.[8] suggested a Simple-to-complex (STC) architecture to strengthen the segmentation network gradually. Recently, Class activation mapping (CAM) has been proposed by Ref.[14], which can localize initial objects seed region from image-level annotations and utilize it as localization cues to train semantic segmentation network. Kolesnikov et al.[15] proposed a Seed, expand, and constrains (SEC) approach, which is the first use of location cues produced by CAM as supervision to train segmentation network, that integrates seeding loss for location cues, expansion loss for expanding seed region, and constrain-to-boundary loss for constraining seed region to object boundaries into a unified framework. However, the SEC can only obtain small and sparse seed regions of objects that are not enough as pseudo labels to train semantic segmentation networks. To resolve this problem, Wei et al.[9] derived the multiple classification networks, which start with a small seed region, to progressively discover new and complement the object region for training semantic segmentation network. Xiang et al.[19] proposed a bottom-up and top-down structure, in which the bottom-up step digs out common object features from the initial seed region produced by the classification network and expand object seed regions by saliency-guided method, where these object localization regions are used as pseudo labels for training image semantic segmentation network in the top-down step. Jiwoon et al.[17] proposed an AffinityNet to predict pixel-level semantic affinity, estimate pixels of adjacent attribute the same class, and then use random walk with semantic affinity matrix from AffinityNet to expand initial seed region produced by CAM[14]. Zilong et al.[16] suggested a deep-seed region growth strategy mixing conventional Seed region growing (SRG) with deep convolutional neural networks, which could dynamically and gradually expand the initial seed regions throughout the training of its semantic segmentation network. CIAN[31] proposed a cross-image affinity approach that would capture long-range contextual dependencies between two distinct images contain objects of the same class, thereby improving the performance of a weakly supervised semantic segmentation network. However, methods mentioned above of weakly-supervised semantic segmentation all starting with initial small and sparse seed regions, which is disadvantageous for the performance of semantic segmentation networks. There is a group of approaches aims to produce denser and more accurate initial seed region as pseudo annotations for training segmentation network. Jungbeom et al.[10] proposed a FickleNet, which starts with a feature map produced by a general classification network, to generate a denser initial seed region using random combinations of hidden units in a convolutional neural network. Yude Wang et al.[32] proposed a Self-supervised equivariant attention mechanism (SEAM) to improve the prediction consistency for given an image with various transformed, which boost classification network to produce high-quality object seed regions. Inspired by dilated convolution[2,20], which can effectively expand the receptive fields of convolutional kernels and without any additional parameters or amount of computation for improving the performance of the fully supervised semantic segmentation model[1], Wei et al.[11] proposed Multi-dilated convolution (MDC) utilizes several convolutional blocks with different dilated rates to expend object seed region. However, some true negatives will be falsely highlighted with gradually increasing the dilated rates. In this paper, we adaptively integrate dilated convolutional units with a self-attention mechanism to capture pixel affinity measure matrix for boosting the performance of the classification network, which can enhance the relevant object regions and suppress the other irrelated regions, and finally enabling producing more accurate and denser object seed regions. III. Approach In this part, we provide a thorough explanation of the proposed dilated convolutional pixel affinity network to produce more complete and dense object seed regions under image-level annotations as pseudo-labels for training semantic segmentation. First of all, we will introduce in producing initial object localization cues through a deep classification network. Then, we will describe to refine the seed cues by dilated convolutional pixels affinity method and introduce how to fuse pixels affinity measure matrices produced by the proposed method with multi-dilated convolutional units. Finally, we will be using these high-quality seed cues as pseudo labels to train a semantic segmentation network. 1. Seeds generation from classification network In order to achieve seed cues under the supervision of the image-level, we adopt the deep convolutional classificationnetwork to produce object localization maps. As we know, image-level annotations only perceive there are objects in the image, do not provide any location information of objects. Recently, by training a classification network to produce feature maps under the supervision of image-level labels, the location information of the discriminative target regions in the image can be obtained[14]. However, the localization maps produced by the classification networks only focus on the small and sparse discriminative regions of objects. Hence, we use dilated convolutional units to expand the receptive field of the convolutional kernel and do not produce any computationally expensive characteristics to perceive more accurate and dense seed regions. The proposed framework employs a deep classification network with dilated convolutional units to localize discrimination object regions, as shown in Fig.2. As the backbone for feature extraction, we use the modified VGG-16[24] model, which has been pre-trained in the ImageNet[33] database. In this framework of proposed the network, the last two pooling layers are removed to improve the resolution of the feature map, and the dilated convolution unit is equipped on the conv5_3 layer to perceive more object seed regions. Then, the object seed regions are further optimized by dilated convolutional pixels affinity method. Next, Global average pooling (GAP) was applied on the last feature maps to produce tensor as image representation and classified by a full-connected layer under image-level labels. Finally, for trained network, class activation maps of object discriminative regions, which denoted M c , is calculated by M c ( x , y ) = ∑ k w k c f k ( x , y ) (1) where w ∈ c × k indicate weight matrix of classification, in which c is the number of object classes and k is the channel dimension of the last feature map, and w k c is the element of w which denote the contributes of f k ( x , y ) for class c. f k ( x , y ) indicate the activation at localization ( x , y ) of k channel on the last feature map before the global average pooling layer. Fig. 2Open in figure viewerPowerPoint Overall framework of the proposed dilated convolutional pixels affinity network We can also notice the seed cues in the background. Inspired by Ref.[34], classes with the highest probability tend to highlight discrimination object regions, and classes with the lowest probability tend to highlight irrelevant regions of the object. Thus, we sort the classes probability series from lowest to highest, i.e., 1 , … , c . The activation map of the background, which is obtained as follows: M bg = ∑ c b ( c ) M c (2) where b(c) is balanced fusion function, which is given by b ( c ) = 1 , if c ≤ p - 1 , if c ≥ q 0 , otherwise (3) which just consider the p activation maps of low probability classes and the q activation maps of high probability classes. Finally, we denote object regions and background regions by setting the hard threshold to M c and M bg respectively. 2. Dilated convolutional pixels affinity model Although, a classification network with dilated convolutional units can effectively enlarge the receptive fields to generate more accurate and denser object regions. But, it can also perceive more irrelevant object information, with gradually rising the dilated rates, leading to some true negative regions being falsely highlighted. We propose the Dilated convolutional pixel affinity model (DCPAM) to solve this issue, which uses the advantages of the self-attention mechanism[22] to catch the long-range contextual dependence between the feature maps produced by the standard convolution unit ( i . e . , d = 1 ) and the dilated convolution unit with different dilated rates ( i . e . , d = 2 to n). The detail of DCPAM in Fig.3 are described as following. Fig. 3Open in figure viewerPowerPoint Illustration of dilated convolutioinal pixels affinity method with different dilated rates DCPAM enables standard convolutional feature maps from all locations to perceive dilated convolutional feature maps of other locations, enhancing the pixels with the same class and suppressing the pixels that belong to different classes. The feature maps generated by the standard convolutional unit and dilated convolutional unit are normally obtained as DCPAM input, respectively denoted as S ∈ R C × H × W and D ∈ R C × H × W . Then, the latest feature maps are further reshaped into R C × N , where the total number of pixel locations is N = H × W . Then, we obtain the affinity measure matrix A ∈ R N × N by performing the matrix product between the reshaped feature maps S and D, and the each position feature representation of A is defined as follows: a i , j = exp S i T D j (4) where i and j is the index of pixel position from reshaped feature S and D, and S i and D j is the feature value of i and j position respectively. a i , j measure the i position of standard convolution feature map impact on the j position of the dilated convolution feature map. Next, we normalize a i , j by applying a softmax layer to guarantee that the sum of all weights of dilated convolution feature value is one unit, and the detail is as follows: a ¯ i , j = a i , j ∑ j = 1 N a i , j (5) We aim to retrieve supplementary features of the enlarged receptive field from dilated convolutional Dj to standard convolutional Si. For this purpose, we implement a matrix multiplication between A and reshaped feature map of the standard convolutional unit, and all affinity score a ¯ i , j from different pixel positions of dilated convolutional to standard convolutional feature map are summed together. Then we reshape the above result to R C × H × W . Finally, to obtain an augmented feature map E ∈ R C × H × W , we multiply it with the weight parameter λ and perform the element-wise summation of the standard convolutional feature map S, which is defined as follow: E i = λ ∑ j = 1 N a ¯ i , j S j + S i (6) where λ is set to 0 in the initial stage, and gradual-learned during the training process. From Eq.(6), it can be informed that the i position of E is the sum of feature value affinity measure matrix produced by DCPAM at every position and original features produced by standard convolutional unit at i position. Consequently, we can take advantage of enlarging the receptive field without any consumption to propagate information from small discriminate object regions to other object regions and selectively strengthen features according to the affinity measure matrix, which avoids propagating information to irrelevant object regions. As shown in Fig.3, there are two dilate convolution units with different dilated rate, which can perceive different regions in one image to enhance the original feature map produced by the standard convolutional unit. It can simply be augmented to implement relationships among muti-dilated convolutional units and a standard convolutional unit. Specifically, given the standard convolutional unit and multi-dilated convolutional units, which are defined as m d ∣ d = 2 , … , N , where d is dilated rate. We obtained affinity measure matrix A by computing the affinity score from muti-dilate convolutional units to the standard convolution unit according to Eqs.(4) and (5), which denoted as A d ∣ d = 1 , … , N . Next, we fuse multiple affinity measure matrixes by conducting an average element-wise operation, which denoted as follows: A = average d ∈ { 1 , … , N } A d (7) Finally, as shown in Eq.(6), we utilize A to enhance the initial feature map produced by standard convolution unit, and we adopt augmented feature map E as input of Eqs.(1) and (2) produce high-quality object localization maps as pseudo-labels for training semantic segmentation methods. Meanwhile, there many other fuse approaches are also available, e.g., max function. The detailed discussion will be present in Section IV experiments. 3. Training the segmentation network After obtaining the high-quality object seed cues from the location maps produced by the classification network, we can utilize them as supervision for training semantic segmentation networks. We adopt the balanced seeding loss[10,16] and together utilize the advantage of auxiliary segmentation mask[9,11] produced by inferring semantic segmentation network under the image-level annotations of ground-truth in an online manner, which considering the unbalanced distribution seed cues of foreground and background, meanwhile utilizing ignored pixels of object location maps and gain robustness to falsely labeled pixels. Specifically, let I denote the space of training images and for any training image i ∈ I . The label set is denoted as C = c fg ∪ c bg , where c fg and c bg are the foreground label and background label set respectively. We aim to train a semantic segmentation network f ( i ; θ ) , in which θ is the optimizable parameter, that enables to predict the conditional probability of any label c ∈ C at any location u of the feature map which denoted as f ( i ; θ ) . Use H c to represent a set of object regions corresponding to class produced by classification network, and H ^ c denoted the auxiliary segmentation mask online produced by segmentation network. Then the balanced seeding loss and auxiliary seeding loss are present as follow: L seed = - 1 ∑ c ∈ c fg H c ∑ c ∈ c fg ∑ u ∈ H c log f u , c ( i ; θ ) - 1 ∑ c ∈ c bg H c ∑ c ∈ c bg ∑ u ∈ H c log f u , c ( i ; θ ) (8) where • is the cardinality of pixels, L seg = - 1 ∑ c ∈ C H ^ c ∑ c ∈ C ∑ u ∈ H ^ c log f u , c ( i ; θ ) (9) Furthermore, we also adopt Lboundary used in Refs.[10,15,16] as constraint loss function, which can encourage the segmentation masks produced by the segmentation network to match up with object boundary. Then, we define Lboundary as the mean KL-divergence between outputs of the segmentation network and CRF[35], which denoted as follow: L boundary = 1 N ∑ u = 1 N ∑ c ∈ C R u , c ( i , f ( i ; θ ) ) log R u , c ( i , f ( i ; θ ) ) f u , c ( i ; θ ) (10) where R u , c ( i , f ( i ; θ ) ) is the output of fully-connected CRF with i and segmentation masks produced by segmentation network as input. Finally, we minimize the total loss function L to optimize the segmentation network: L = L seed + L seg + L boundary (11) IV. Experiments 1. Experimental setup 1) Dataset and evaluation metrics The proposed approach is evaluated on the PASCAL VOC 2012 image segmentation benchmark dataset. There are 20 foreground classes and one background class in the PASCAL VOC dataset[26]. It is divided into three components: the training set (1464 images), the validation set (1449 images) and the test set (1456 images). We adopt augmented the training set with 10582 images from Ref.[36] by following common experimental protocol[9-11]. In our experiment, only image-level labels are required for training, and the mean Intersection over union (mIoU) is used as a metric to measure the efficiency of our approach in image datasets of 21 classes. The detailed analysis of the method experiment is only implemented on the validation set in which the ground truth is available. We compare our approach with other state-of-the-art methods in validation and test set. 2) Training/testing details We use object seeds region produce by proposed DCPAN as pseudo pixel-labels for training semantic segmentation network.