Structured consistency loss for semi-supervised semantic segmentation

The consistency loss has played a key role in solving problems in recent studies on semi-supervised learning . Yet extant studies with the consistency loss are limited to its application to classification tasks ; extant studies on semi-supervised semantic segmentation rely on pixel-wise classification , which does not reflect the structured nature of characteristics in prediction . We propose a structured consistency loss to address this limitation of extant studies . Structured consistency loss promotes consistency in inter-pixel similarity between teacher and student networks . Specifically , collaboration with CutMix optimizes the efficient performance of semi-supervised semantic segmentation with structured consistency loss by reducing computational burden dramatically . The superiority of proposed method is verified with the Cityscapes ; The Cityscapes benchmark results with validation and with test data are 81.9 mIoU and 83.84 mIoU respectively . This ranks the first place on the pixel-level semantic labeling task of Cityscapes benchmark suite . To the best of our knowledge , we are the first to present the superiority of state-of-the-art semi-supervised learning in semantic segmentation . 1 INTRODUCTION . With the deep learning approach of Krizhevsky et al . ( 2012 ) introduced in ILSVRC-2012 ( Russakovsky et al. , 2015 ) , deep learning based image processing has quickly spread among the field . Deep learning has evolved into supervised learning , where the performance of the network is greatly affected by the quantity and quality of labeled data provided . However , preparing labeled data is time-consuming and expensive as it is manually prepared . When it comes to semantic segmentation , it requires pixel-by-pixel annotation making it particularly costly to prepare labeled data for processing . Currently , various techniques such as active learning ( Mackowiak et al. , 2018 ) , interactive segmentation ( Maninis et al. , 2018 ) , weakly-supervised leaning ( Lee et al. , 2019 ) and more are developed and presented to solve the labeling cost problem in semantic segmentation . In this study , we are aimed at bypassing the labeling cost problem with semi-supervised learning technique . Semi-supervised learning is a technique to improve network performance with relatively a smaller number of labeled data when a large number of unlabeled data is also available . In reality , a simple build up of unlabeled data can be easily collected from various data sources such as web crawling , vehicle logging and more , and what makes data preparation costly is the manual labelling process to make the collected data useful in learning process . That is , semi-supervised learning technique best suits the data needs of semantic segmentation in real-world research environment . For this reason , our research designed the semi-supervised learning approach suitable for semantic segmentation . Prior studies apply various versions of semi-supervised learning technique to image classification task and report significant improvement in the performance . Unlike classification , semantic segmentation is a task that performs structured prediction . Specifically , prediction of classification results in a class vector , while semantic segmentation makes predictions per regional location and predicts structural characteristics of regions . We can also learn from prior studies that structural relationship between pixels is important in semantic segmentation ; Liu et al . ( 2019 ) discussed the distillation of between-pixel relationships and Xie et al . ( 2018 ) discussed the neighbor-pixel distillation of features . Together , we pinpoint that it is important to adequately modify the semi-supervised learning to be applied in semantic segmentation . In this paper , we propose a semi-supervised learning with structured consistency loss in semantic segmentation . Structured consistency loss reflects the structural relationship between pixels in semantic segmentation . It regularizes the inter-pixel relationships consistent , thereby allowing the network to learn more powerful generalization capabilities . We propose a method incorporating CutMix ( Yun et al. , 2019 ) in data augmentation , achieving a significantly lower GPU memory utilization by limiting the observation region at each CutMix box . This allows us to process the final prediction map at the size of the original image ; we no longer must reduce the size of the final prediction map to lower the GPU memory utilization , unlike it was with pair-wise knowledge distillation which requires feature distillation with the lowest resolution . Our method achieved the first place with mIOU 83.84 in the Cityscapes benchmark pixel-level semantic labeling task ( Cordts et al. , 2016 ) . This is the first study showing that the semi-supervised learning achieved the state-of-the-art performance in semantic segmentation using the Cityscapes benchmark . Note that our semi-supervised learning technique could be used in parallel with other researches for further improvements since our contribution is not in network architecture , but in learning techniques . 2 RELATED WORKS . Semantic Segmentation . The early period of semantic segmentation approaches were mostly based on Fully Convolutional Network ( FCN ) ( Sermanet et al. , 2013 ; Long et al. , 2015 ) . Improving earlier segmentation models , the loss of spatial information is mediated with the use of encoderdecoder architecture ( Noh et al. , 2015 ; Ronneberger et al. , 2015 ; Badrinarayanan et al. , 2017 ) or dilated convolution ( a.k.a . Atrous Convolution ) expanding the receptive field ( Chen et al. , 2014 ; Yu & Koltun , 2015 ) . Improving the localization performance , ( Chen et al. , 2017 ) applied the Atrous Spatial Pyramid Pooling ( ASPP ) in semantic segmentation and PSPNet ( Zhao et al. , 2017 ) proposed a feature pyramid pooling module to gather global contextual information around the image object or stuff . More recently , Chen et al . ( 2018 ) suggested well-organized architecture with combining encoder-decoder architecture and dilated convolution . Many subsequent methods that achieved state-of-the-art performance have followed this structure ( Takikawa et al. , 2019 ; Zhuang et al. , 2018 ; Li et al. , 2019 ) . Zhu et al . ( 2019 ) adopted DeeplabV3Plus ( Chen et al. , 2018 ) with WideResNet38 ( Zagoruyko & Komodakis , 2016 ) as the backbone network . The inspiring results were generated due to labelling of video image using temporal information , not by architectural improvements . Moreover , boundary label relaxation also eased the strictly labeling in boundary issue and class uniform sampling dealt the class imbalance problem . To make our contribution clearly visible , we employed those architecture and methods except for video label generation . Semi-Supervised learning . In recent years , semi-supervised method has become a very prominent theme , but a lot of researches were limited to classification task in earlier times . They used a loss function computed on unlabeled data and encouraged the model to generalize better to unseen data in same domain . Grandvalet & Bengio ( 2005 ) proposed the entropy minimization loss showing that decision boundary tends to lie on low density region of class distribution . Consistency loss which encourages the model to produce the same output distribution when its inputs are perturbed is suggested in Laine & Aila ( 2016 ) . Consistency regularization loss played a breakthrough role which facilitated researches in this field . To employ consistency loss efficiently , Tarvainen & Valpola ( 2017 ) proposed teacher network with exponential-moving-average ( EMA ) network and generated better guessed label . More recently , MixMatch ( Berthelot et al. , 2019 ) combined entropy minimization , consistency loss , and MixUp ( Zhang et al. , 2017 ) regularization for the better generalization . Furthermore , result of Xie et al . ( 2019 ) nearly matched the performance of models trained on the full sets of CIFAR-10 with only using 10 % of this dataset , thanks to sophisticated augmentation method with realistic noise . Structured Prediction . Semantic segmentation has a nature of structured prediction ; prediction result has a equal shape of input image and prediction vector of each pixel has a strong correlation with each other , especially close one . There were some researches reflected this characteristics by a loss term , or architectural perspective . In Xie et al . ( 2018 ) , local pair-wise ( 8-neighbors ) distillation was used to make efficient feature distillation . They transferred the knowledge of zero-order , and first-order by consistency map . Liu et al . ( 2019 ) showed impressive progress of feature distillation using pair-wise distillation . They calculated total inter-pixel similarities in a specific feature map and distilled the feature of teacher to student by similarities . The knowledge distillation and consistency loss differ in the underlying philosophy . The purpose of knowledge distillation is to enable an efficient network to learn powerful network predictions or feature extraction capabilities for the same image , while the goal of consistency loss is to enhance robustness against disturbances by forcing the same prediction . However , the implementation detail of both methods are same . That is , using the distance function as the cost function . In this paper , we employed the knowledge distillation technique introduced in Liu et al . ( 2019 ) to our structured consistency loss . Most recently , consistency loss with CutMix ( Yun et al. , 2019 ) for semantic segmentation was introduced in French et al . ( 2019 ) . They investigated the difference between classification and semantic segmentation in terms of low density region ; it lies in boundary of class distribution in classification , while it is distributed in object boundary pixels of each image in semantic segmentation . For this reason , they selected CutMix instead of MixUp to conserve the local-structural identity of image in CutMix box . We also used CutMix augmentation to give different perturbed images to teacher and student while maintaining the structural information . 3 METHODOLOGY . Since semantic segmentation can be considered as a classification problem performed pixel by pixel , it is reasonable to try to apply semantic segmentation in the same way to consistency loss , widely used in semi-supervised classification . Indeed , several studies have made improvements in this methodology . However , semantic segmentation has a characteristic of structured prediction unlike general classification . For this reason , if the existing consistency loss is used as it is , it is difficult to expect high performance improvement while ignoring the nature of the structured prediction of semantic segmentation . Therefore , structured consistency loss is used to encourage network to predict properly focusing on inter-pixel relationship in local patch using cosine similarity . Local patches are implemented with random CutMix augmentation scheme . Looking at too narrow range of receptive field can be ineffective in identifying pixel-to-pixel relationships , while looking at too wide range increases computational complexity and hinder network ’ s learning . Therefore , precise tuning of the CutMix box size is essential . Experimental details would be explained in section 4.1 . 3.1 ARCHITECTURE . Our main architecture is divided into two parts , supervised training with labeled image and unsupervised training with unlabeled image . These images are used to compose mini-batch in each step in order to calculate labeled and unlabeled loss simultaneously . Formally , the combined loss Ltot for semi-supervised learning is computed as : Ltot = Lx + Lu ( 1 ) Lx = −log ∑ C∈Nw P ( C ) ( 2 ) Where Lx is general cross entropy loss with boundary label relaxation for semantic segmentation with labeled image x. Lu is a loss with unlabeled image u . In equation ( 2 ) , Lx is composed of P ( ) with softmax probability of each class where Nw is the set of classes within a w by w window boundary region , respectively . Note that Lx reduces to the standard cross entropy loss with one-hot label when w equals one . CutMix . The CutMix augmentation was originally designed for classification task . It mixes samples by cutting rectangular patch and pasting from one sample into another . Following as done by Liu et al . ( 2019 ) , we use it for semantic segmentation by generating N patches with random size and position . The total area of patches would be approximately half of the image dimension H , W to make balance between two images u and x. Semi-supervised . In Figure 1 , our main architecture of semi-supervised approach is described . First , we generate úa and úb by applying random augmentation used in Zhu et al . ( 2019 ) except CutMix . And then create CutMix image um from the augmented images . The teacher network takes the images as input and generates prediction results ( pt ( úa ) , pt ( úb ) ) , and also generates CutMix image pmt ( úa , úb ) from the two predictions . Student network use the result as a guessed label introduced by Berthelot et al . ( 2019 ) . The student network gets the CutMix image as input during training and produces prediction ps ( um ) . In the end , we calculate two consistency losses using the prediction of student and teacher networks each . Note that teacher and student network have same network architecture . The weights of teacher is not from its own weights using gradient update , but from the student weights by EMA manipulation . These losses give the student network ability to predict similar result as the teacher networks by only looking at local patches where teacher network receives the overall image information . Unlabeled loss term is separated into two parts : consistency loss and structured consistency loss . Consistency loss is a conventional loss used for classification task and performed pixel-wise manner while structured consistency loss is a novel approach to figure out inter-pixel relationship within local patch . Formula of unlabeled loss is as follows : Lu = λcLc + λscLsc ( 3 ) Lc = 1 H ×W ∑ i∈T ||ps ( um , i ) − pmt ( ´ua , i , ´ub , i ) ||2 ( 4 ) Where Lc is conventional consistency loss which is average of pixel-wise squared L2 loss used in classification task . We will deep dive into Lsc in next section . λc , λsc are weight for each loss term . T = { 1 , 2 , ... , H ×W } denotes a set of all the pixel indices in the image . ps ( um , i ) represents prediction of student network for ith pixel of CutMix unlabeled image um . pmt ( ´ua , i , ´ub , i ) represents CutMix prediction of teacher network for ith pixel for unlabeled augmented images úa , úb .

Summary: - This paper introduces a structured consistency loss for semantic segmentation under the semi-supervised learning paradigm. Specifically, authors propose to include this term to force consistency between the predictions of two networks (a teacher and a student). To evaluate their method, authors employ images from the cityscapes dataset. Results show that the proposed method achieves competitive results compared to state-of-the-art architectures.