Structured consistency loss for semi-supervised semantic segmentation
Authors: JongMok Kim
Joo Young Jang
Hyunwoo Park
Venue:  (2020)
Keywords: semi-supervised learning
semantic segmentation
structured prediction
structured consistency loss

Abstract: The consistency loss has played a key role in solving problems in recent studies on semi-supervised learning. Yet extant studies with the consistency loss are limited to its application to classification tasks; extant studies on semi-supervised semantic segmentation rely on pixel-wise classification, which does not reflect the structured nature of characteristics in prediction. We propose a structured consistency loss to address this limitation of extant studies. Structured consistency loss promotes consistency in inter-pixel similarity between teacher and student networks. Specifically, collaboration with CutMix optimizes the efficient performance of semi-supervised semantic segmentation with structured consistency loss by reducing computational burden dramatically. The superiority of proposed method is verified with the Cityscapes; The Cityscapes benchmark results with validation and with test data are 81.9 mIoU and 83.84 mIoU respectively. This ranks the first place on the pixel-level semantic labeling task of Cityscapes benchmark suite. To the best of our knowledge, we are the first to present the superiority of state-of-the-art semi-supervised learning in semantic segmentation.

Under review as a conference paper at ICLR 2020
STRUCTURED CONSISTENCY Loss FOR SEMI- SUPERVISED SEMANTIC SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
The consistency loss has played key role in solving problems in recent stud- ies semi-supervised learning. Yet extant studies with the consistency loss are limited to its application to classification tasks; extant studies on semi-supervised semantic segmentation rely on pixel-wise classification, which does reflect the structured nature of characteristics in prediction. We propose a structured con- sistency loss address this limitation of extant studies. Structured consistency loss promotes consistency in inter-pixel similarity between teacher and student networks. Specifically, collaboration with CutMix optimizes the efficient per- formance of semi supervisd semantic segmentation with structured consistency loss by reducing computational burden dramatically. The superiority of proposed method is verified with the Cityscapes; The Cityscapes benchmark results with validation and with test data are 81.9 mIoU and 83.84 mloU respectively. This ranks the first place the pixel-level semantic labeling task of Cityscapes bench- mark suite. To the best of our knowledge, we are the first to present the superiority of state-of-the-art semi-supervised learning in semantic segmentation.
INTRODUCTION
With the deep learning approach of KKizhevsky et al.] (2012) introduced in ILSVRC-2012 (Rus- sakovsky al.] (2155), deep learning based image processing has quickly spread among the field. Deep learning has evolved into supervised learning, where the performance of the network is greatly affected by the quantity and quality of labeled data provided. However, preparing labeled data is time-consuming and expensive as it is manually prepared. When it comes to semantic segmenta- tion, it requires pixel-by-pin bxxeeel annotation making it particularly costly to prepare labeled data for processing. Currently, various techniques such active learning (Mackowiak et al. 2018), interac- tive segmentation (Maninis et al. 2018). weakly- superviee leaning (Lee et al. 2019) and more are developed and presented to solve the labeling cost problem in semantic segmentation. In this study, we are aimed at bypassing the labeling cost problem with semi-supervised learning technique.
Semi-supervised learning is technique improve network performance with relatively smaller number of labeled data when large number of unlabeled data is also available. In reality, a simple build up of unlabeled data can be easily collected from various data sources such as web crawling. vehicle logging and more, and what makes data preparation costly is the manual labelling process to make the collected data useful in learning process. That semi-supervised learning technique best suits the data needs of semantic segmentation in real-world research environment. For this reason, our research designed the semi-supervised learning approach suitable for semantic segmentation.
Prior studies apply various versions of sumi-super learning technique to image classification task and report significant improvement in the performance. Unlike classification, semantic segmen- tation is task that performs structured prediction. Specifically, prediction of classification results in a class vector, while semantic segmentation makes predictions per regional location and predicts structural characteristics of regions. We can also learn from prior studies that structural relationship between pixels is important in semantic segmentation; Liu et al. (2019) discussed the distillation of netween-pixel relationships and Xie al. (2018) discussed the neighbor-pise distillation of fea- tures. Together, we pinpoint that it is important to adequately modify the sume-supe learning to be applied in semantic segmentation.

---

Under review as a conference paper at ICLR 2020
In this paper, we propose semi-supervised learning with struetured consistency loss in seman- tic segmentation. Structured consistency loss reflects the structural relationship between pixels in semantic segmentation. It regularizes the inter-pixel relationships consistent, thereby allowing the network to learn more powerful generalization capabilities. We propose a method incorporating CutMix (Yun et al. (019) in data augmentation, achieving significantly lower GPU memory uti- lization by limiting the observation region at each CutMix box. This allows us to process the final prediction map the size of the original image;
longer must reduce the size of the final predic- tion map to lower the GPU memory utilization,, unlike it was with pair- wise knowledge distillation which requires feature distillation with the lowest resolution.
Our method achieved the first place with mIOU 83.84 in the Cityscapes benchmark pixel-level se- mantic labeling task (Cordts al. 2016). This is the first study showing that the semi-supervised learning achieved the state-of-the-an performance in semantic segmentation using the Cityscapes benchmark. Note that our semi-supervised learning technique could be used in parallel with other researches for further improvements since our contribution is not in network architecture, but in learning techniques.
2 RELATED WORKS
Semantic Segmentation. The early period of semantic segmentation approaches were mostly based on Fully Convolutional Network (FCN) (Sermanet et al. 2013 Long et al. 2015). Improving earlier segmentation models, the loss of spatial information is mediated with the use of encoder-
decoder architecture (Noh et al.) 2015 Ronneberger et al., 2015 Badrinarayanan et al.
2017) or
dilated convolution (a.k.a. Atrous Convolution) expanding the receptive field (Chen et al. 2014 Yu
& Koltun 2015). Improving the localization performance, (Chen et al. 2017) applied the Atrous Spatial Pyramid Pooling (ASPP) in semantic segmentation and PSPNet (Zhao et al., 2017) proposed a feature pyramid pooling module to gather global contextual information around the image object
or stuff.
More recently, Chen et al. 2018) suggested well-organized architecture with combining encoder-decoder architecture and dilated convolution. Many subsequent methods that achieved state-of-the-an performance have followed this structure (Takikawa et al., 2019 Zhuang et al., 2018 Li al.) (019). Zhu et al. (2019) adopted DeeplabV (Chen et al., 2018) with WideResNet38
Zagoruyko & Komodakis 2016) as the backbone network. The inspiring results were generated due to labelling of video image using temporal information, not by architectural improvements. Moreover, boundary label relaxation also eased the strictly labeling in boundary issue and class uniform sampling dealt the class imbalance problem. To make our contribution clearly visible, we employed those architecture and methods except for video label generation.
Semi-Supervised learning. In recent years, semi remi-supervised method has become very prominent theme, but a lot of researches were limited to classification task in earlier times. They used a loss function computed on unlabeled data and encouraged the model generalize better to unseen data in same domain. Grandvalet & Bengio (2005) proposed the entropy minimization loss showing that decision boundary tends to lie on low density region of class distribution. Consistency loss which encourages the model to produce the same output distribution when its inputs are perturbed is suggested in Laine Aila (2016). Consistency regularization loss played breakthrough role which facilitated researches in this field.
To employ consistency loss efficiently, Tarvainen & Valpola (017)) proposed teacher network with ppponentialmooving--veeee (EMA) network and generated better guessed label. More recently, Mix (Berthelot af.] (019) combined entropy minimization, consistency loss, and MixUp Zhang et al. (0017) regularization for the better generalization. Furthermore, result of [ie et al. 2019) nearly matched the performance of models trained on the full sets of CIFAR-10 with only using 10 % of this dataset, thanks to sophisticated augmentation method with realistic noise.
Structured Prediction. Semantic segmentation has nature of structured prediction; prediction result has a equal shape of input image and prediction vector of each pixel has a strong correlation with each other, especially close one. There were some researches reflected this characteristics by
2

---

Under review as a conference paper at ICLR 2020
loss term, or architectural perspective. In Xie al. (2018). local pair-wise (8-neighbors) distillation was used to make efficient feature distillation. They transferred the knowledge of zero-order, and first-order by consistency map. Liu et al. (2019) showed impressive progress of feature distillation using pair-wise distillation. They calculated total inter-pixel similarities in a specific feature map and distilled the feature of teacher student by similarities.
The knowledge distillation and consistency loss differ in the underlying philosophy. The purpose of knowledge distillation to enable an efficient network learn powerful network predictions or feature extraction capabilities for the same image, while the goal of consistency loss is to enhance robustness against disturbances by forcing the same prediction. However, the implementation detail of both methods are same. That is, using the distance function as the cost function. In this paper, we employed the knowledge distillation technique introduced Liu et al. (2019) to our structured consistency loss.
Most recently, consistency loss with CutMix (Yun et al. 2019) for semantic segmentation was intro- duced in French et al. (2019). They investigated the difference between classification and semantic segmentation terms low density region; it lies in boundary of class distribution in classification, while it is distributed in object boundary pixels of each image in semantic segmentation. For this reason, they selected CutMix instead of MixUp to conserve the local-structural identity of image in CutMix box. We also used CutMix augmentation to give different perturbed images to teacher and student while maintaining the structural information.
3 METHODOLOGY
Since semantic segmentation can be considered as a classification problem performed pixel by pixel, is reasonable try apply semantic segmentation the same way consistency loss, widely used in semi-supervised classification. Indeed, several studies have made improvements in this methodology. However, semantic segmentation has a characteristic of structured prediction unlike general classification. For this reason, if the existing consistency loss is used as it is, it is difficult expect high performance improvement while ignoring the nature of the structured prediction of semantic segmentation.
Therefore, structured consistency loss used to encourage network to predict properly focusing on inter-pixel relationship in local patch using cosine similarity. Local patches are implemented with random CutMix augmentation scheme. Looking at too narrow range of receptive field can be ineffective in identifying pixel-to-pxel relationships, while looking at too wide range increases computational complexity and hinder network's 's learning. Therefore, precise tuning of the CutMix box size is essential. Experimental details would be explained in section 4.1
3.1 ARCHITECTURE
Our main architecture is divided into two parts, supervised training with labeled image and unsuper- vised training with unlabeled image. These images are used compose mini-batch in each step in order to calculate labeled and unlabeled loss simultaneously. Formally, the combined loss Ltot for semi-supervised learning computed as:
Liot Lxx + Lu
(I)
Lx -log ΣP(C)
CEN.
Where Lx is general cross entropy loss with boundary label relaxation for semantie segmentation with labeled image L. is a loss with unlabeled image In equation (2), E. is composed of P() with softmax probability of each class where Mr, is the set of classes within a w by w window boundary region, respectively. Note that C, reduces the standard cross entropy loss with one-hot label when w equals one.
The CutMix augmentation was originally designed for classification task. mixes samples by cutting rectangular patch and pasting from one sample into another. Following done
CutMix.
3

---

Under review as a conference paper at ICLR 2020
Ua
Pe ( uua
Teacher Network (EMA)
Student Network
ps (um)
Pmn ( uu Up
Um
Consistency Loss
cutmix
cutmix
Structured Consistency
up
Teacher Network (EMA)
Figure Main architecture describing semi-supervised approach with CutMix augmentation for semantic segmentation. Two losses, consistency and structured consistency, calculated from prediction of teacher and student networks each.
by Liu et al. 2019), we use for semantic segmentation by generating patches with random size and position. The total area of patches would be approximately half of the image dimension H, W to make balance between two images u and x.
Semi-supervised. In Figure ! our main architecture of semi-supervised approach is described. First, we generate via and vís by applying random augmentation used in Zhu et al. (2019) except CutMix. And then create CutMix image umm from the augmented images. The teacher network takes the images as input and generates prediction results (Pr().)) Pt (166), and also generates CutMix image Pimx (1a,, vi,) from the two predictions. Student network use the result as guessed label introduced by (Berthelot al. (2019). The student network gets the CutMix image input during training and produces prediction pa (um). In the end, we calculate two consistency losses using the prediction of student and teacher networks each. Note that teacher and student network have same network architecture. The weights of teacher is not from its own weights using gradient update, but from the student weights by EMA manipulation. These losses give the student network ability to predict similar result as the teacher networks by only looking at local patches where teacher network receives the overall image information.
Unlabeled loss term is separated into two parts: consistency loss and structured consistency loss. Consistency loss a conventional loss used for classification task and performed pixel-wise manner while structured consistency loss is a novel approach to figure out inter-pixel relationship within local patch. Formula of unlabeled loss as follows:
C. AcCo + AecLoc
L. W
(1v.()m.)) - Pma (Wa,s; u6,;)
(4)
KE
Where La is conventional consistency loss which is average of pixel-wise squared L.2 loss used in classification task. We will deep dive into Cae in next section. Xo Asc are weight for each loss term. T {1, ..., H W} denotes prediction of student network for ith pixel of CutMix unlabeled image un Pont (ua.s W6,s) represents CutMix prediction of teacher network for ith pixel for unlabeled augmented images ula zég
of all the pixel indices in the image. Ps (um,)) represents
3.2 STRUCTURED CONSISTENCY LOSS
As explained in section we used pair- wise knowledge distillation technique suggested by Liu et al. 2019) into structured consistency loss can be written
4

---

Under review as a conference paper at ICLR 2020
an
L
Stredtared Consistency
Seructured Consistancy
(b)
Figure 2: Structured consistency loss calculation procedure described visually. CutMix box size of by 3 is used for simplicity. Note that the each pair of cosine similarity (af, a with same color are used to calculate the loss. (a) Masking the covered region (denoted as black box) by another CutMix box since it doesn't 'tt include uncovered box related information. (b) Random elimination of cosine similarities when the number of similarities exceed pair-
04j = pT. P// ((Ipllllllless))
(5)
1
ΣΣ AET JET
|aat - afyy
(6)
H X W)
Where aly and atj denote the similarity between the ith pixel and the jth pixel produced from the teacher and student network each and p: represents prediction vector of ith pixel.
In the previous work, prediction vector (p) was calculated the feature stage rather than the pre- diction stage due to feature distillation effects. On the contrary, structured consistency loss must be calculated at the image domain because the goal to get the same prediction from the perturbed image. In order to do that, there are issues overcome calculate structured consistency loss in image domain; The calculation of structured loss consumes a lot of GPU memory usage since the complexity is (H W)²
feasible in image domain with the same size the input.
Nevertheless, because our main architecture uses CutMix augmentation, calculating the structured consistency loss inside the CutMix box reduces computing complexity approximately N times. To be specific, CutMix augmentation allows limitation of calculation into the CutMix box, thereby enabling efficient GPU memory usage. Moreover, by using CutMix, the network can learn the ability to make accurate predictions with limited local information which predicts similar to the prediction with global area. The structured consistency loss with CutMix is described below:
an,ij = PT. Pn. (Ipp. Pll
N
Lee Σ
Σ
(8)
SET JET
Where T, = 21,2,,.... H, W.) denotes a set of all the pixel indices wth CutMix box except for the region covered by other CutMix box. an.ij represents the similarity between the ith pixel and the jth pixel in wth CutMix box and N the number of CutMix boxes.
To achieve state-of-the-art performance, it is necessary to revise the loss term because input image with high resolution and complex network architecture cause the out memory problem. In order to handle this issue, the number of CutMix boxes for calculating structured consistency loss and the maximum number pairs for calculating losses each box are restricted Nioz and N Npairs respectively, make algorithm realizable. To be specific, didn't CutMix box with dotted
5

---

Under review as a conference paper at ICLR 2020
line borders while only considered Noor CutMix box with straight line for structured consistency
loss in Figure [ In Figure aa), we eliminated the region covered by another CutMix box, and also randomly dropped out similarity vectors excessing over N pair in Figure Lb.. Finally, we derived the feasible structured consistency loss as follows:
Ts n,drop
Droop(((..))vi Ta, Vj Tnh, Apair)
Lec
(10)
The n,drop (K.))TT m.drop
Nhort1
The customized function (Xop(X, n) randomly clips X to when exceeding In the process of attaching the CutMix box, box attached earlier is likely to be covered by the box attached behind, so the loss is calculated using posterior Nbor of boxes. In addition, the loss is not calculated through masking on the area covered by another box. By doing this, only those pixels that are structural- relevant within the CutMix box can learn about the relationship to each other.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Network Structures. We adopt a state-of-the-art segmentation architecture from Zhu et al. (2019). Deeplaby3+ with WideResNet38.. Output stride of encoder is set to and that of the low level feature transferred to decoder is set to 2. The teacher network used to generate guessed label is made from EMA model (Tarvainen & Valpola, 2017). Inference with test and validation sets are also done with EMA model. The parameter value of EMA weight is set to 0.999 and it is averaged out every training steps.
Training Procedure. We use a SGD optimizer with polynomial learning rate policy. Following the setup of Zhu al. (2019) the initial learning rate is 0.002, power set to 1.0, weight decay of 0.001 and momentum of 0.9. We use synchronized batch normalization with a batch size of per GPU, for labeled data and the other for unlabeled one, 8 V100 GPUs. The training epoch is set to 175 based on the labeled data. For unlabeled data, we adopt CutMix augmentation additionally. From the empirical results, we the hyper-parameters N, Nhon, Npair Åo Asc to 32, 16, 9000, 20, 3 respectively. We use training skills, class uniform sampling and boundary label relaxation, to get strong baseline network.
Cityscapes. Cityscapes widely accepted validation medium among recent studies, thanks to its high-quality labeled image database. Cityscapes offers pixel-level annotations of 5,000 images and coarse annotations of 20K images. The reliable pixel-level annotations are also split into three subsets for training, validation and test, each consisting of 2,975, 500 and 1,525 images respectively, test set to gauge the performance our methodology pre-
and we used these training, validation sented in this paper. We also use coarsely annotated images for class uniform sampling to overcome the imbalance between classes. Those images with coarse annotations are also used, ignoring the annotations associated with, unlabeled images in unlabeled training phase for greater learning exposure.
4.2 RESULTS
The experimental results from our semi-supervised learning technique with structured consistency loss are summarized in Table [ Our experiments were conducted on Cityscapes test images with multi-scale strategy (0.5, 1.0 and 2.0), horizontal-flip. and overlapping-tile methods. Our perfor- mance result ranks the first the Cityscapes benchmark, even after including the results of unpub- lished studies. We also want emphasize that the result with only training set exhibits the highest performance result when compared with performance results of published methods. Graphical re- sults the Cityscapes test presented in Figure

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison of Per-class mloU results of recent methods Cityscapes with state- of-the-art method. Both published and unpublished methods are included. Our results exhibit the best overall performance. Note that the results for "Ours" is obtained from training data only, and
"Ours+" is obtained from training and validation data.
Mathod
med
s.walk
huild
wall
Fence
pole
[-light
I-iign
vag
Iomain
sky
persoon
rider
car
Irack
has
Irsin
motor
bike
miool
Gused SUNN
98.7
87.4
94.2
01.9
66.6
72.9
79.6
82.5
94.3
73.3
96.2
$8.3
14.2
96.6
77.2
90.2
85.2
72.6
794
82.8
77.4
Net
9538
87.7
94.2
05.0
66.7
731
19.3
824
94.2
72.9
90.0
58.4
76.2
96.3
19.8
89.0
53.2
74.1
79.9
83.3
Video Propagation
988
87.8
94.2
64.0
65.0
724
79.0
82.8
94.2
74.0
96.1
88.2
75.4
96.5
78.8
94.00
91.6
73.7
79.0
83.5
88.2
94.3
04.3
05.3
72.
19.3
96.1
58.4
75.6
96.6
93.7
91.2
73.7
19.4
Tencen Lab
9666
86.9
94.1
63.5
650
70.7
17.7
80.2
94.0
73.1
95.
87.8
14.5
96.
82.8
94.5
50.4
74.0
77.5
82.9
GOCF
988
87.8
94.1
66.00
66.1
51.1
78.4
82.2
94.1
74.5
955.
88.0
74.0
96.5
79.9
12.4
50.8
31.8
78.4
83.2
77.9
opensog-ggrop group
988
68.3
94.3
66.9
66.7
733
80.2
83.0
94.2
74.1
96.0
88.5
75.8
96.5
78.5
91.88
50.2
73.4
70.3
83.7
Ouns+
98.9
88.4
94.3
65.9
728
19.5
8300
71.3
96.1
88.6
75,9
96.6
19.3
93.8
91.5
74.8
79.7
83.8
Figure 3: Qualitative results of our method on the Cityscapes test set. Pictures depict predicted segmentation masks.
We believe the observed improvement in performance is attributed to (a) the use coarsely labeled images in unlabeled training phase and (b) the effectiveness of structured consistency loss added to semi-supervised learning in semantic segmentation. In recent years, many researchers have at- tempted to achieve the close performance obtained from learning with the entire labeled data by using a combination of only a small portion of the labeled data set and a large unlabeled data set. Yet the use of coarsely labeled data has been limited in extant studies; they are only used in the early training stage, or sporadically used to aid the segmentation of images with rare classes.
We believe that our approach reflects the real-world environment where the available training re- sources are mostly unlabeled; the combination of a small labeled data set and a large unlabeled data set to achieve the performance equivalent to using large labeled data is the way we all want to go. We are a step closer to this goal with the structured consistency loss applied to semi-supervised learning in semantic segmentation.
4.3 ABLATION STUDY
Validation with Baseline. In this section, we conducted additional experiments to demonstrate the superiority and effectiveness of our proposed method in semi-supervised learning. We first perform supervisory baseline, and the result is summarized in the first row of Table The baseline result exhibits a reasonable performance, as a decent amount of recent techniques are already included; pre-training with Mapillary, boundary label relaxation, and class uniform sampling are included in the baseline. By adding consistency loss to the baseline, the mloU increases to 81.55% from 81 81.18%, showing the practical implication of the semi-supervised learning using the CutMix in semantic seg- mentation. mloU increases further with addition of structured consistency loss to 81. 81.90%, reflecting the significance of our proposed method.
7

---

Under review as a conference paper at ICLR 2020
The qualitative results have been visually provided in Figure 4 allowing gauge the qualitative performance. Thanks to inter- aite-pixel consistency processed with the structured consistency loss, our proposed method exhibits less errors in small and visually casy-to-confuse region(s).
Exponential-mooiin-averaaa (EMA) Application. Networks with EMA well-known for their great generalization ability (Tarvainen Valpola 2017 Berthelot et al. 2019). Hence must be beneficial apply EMA both training (teacher network) and inference (validation test), Nevertheless, not all recent studies apply EMA to the weight of teacher network, but use the same weight as that of student network. Therefore, we conduct an experiment to examine the performance variability depending on application of EMA. The experiment result is sumamrized in T When the EMA weight is used in the teacher network, mloU increases O. 135% on average, while the use of EMA weight in validation seems to have no significant effect on mloU. Since the use of EMA weight teacher network and validation yields the greatest mloU, we proceed to use the EMA weight in both.
Table 2: Comparison of mloU results supervised-only (baseline), with consistency loss, with consistency loss and structured con- sistency loss on Cityscapes validation images.
Table 3: Comparison of mloU depending on whether ssponentialmooving-avene (EMA) is applied to both/either teacher network and/or validation Cityscapes image.
Method
mloU (%)
Gain (%)
EMA
mloU (%)
Baseline
81.17
0.0
Teacher
Validation
+ Consistency loss
X
X
81.77
81.55
+0.38
X
81.74
+ Structured Consistency loss
O
X
81.88
81.90
+0.35
O
0
81.90
Figure 4: Qualitative results the validation data of Cityscapes. From the left to the right original images, supervised-only. addition of consistency loss, addition of structured consistency loss, and ground truth.
5 CONCLUSION
In this paper, we propose the structured consistency loss in semi-supervised learning for semantic segmentation. With CutMix augmentation, the structured consistency loss fully exploits the rela- tionship among local regions and enhance the network generalization. This technique could also be simultaneously applied the general network. Our proposed method achieves the state-of-the- art premier performance in the Cityscapes benchmark suite, not only ranking the first place only among publication results, but also among all results including unpublished results. Therefore, we confident that the semi-supervised learning is also highly effective to solve practical real world problems under data- insufficiency, when accompanied with CutMix augmentation as well as the structured consistency loss. Despite of superior performance exhibited by method, the random sampling using N Npair: Nowr the proposed method still has a room for optimization. leave this future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions pattern analysis and machine intelligence, 39(12) (9(1222481-225
2017.
David Berthelot, Nicholas Carlini, lan Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint
arXiv:1905.02249, 2019.
Blang-Chiel Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Se- mantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7066 2014.
Liang-Chich Chen, George Papandreou, lasonas Kokkinos. Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Liang- Ching-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Geoff French, Timo Aila, Samuli Laine, Michal Mackiewicz, and Graham Finlayson. Consis- tency regularization and cutmix for semi-supervised semantic segmentation. arXiv preprint arXiv:I906.000 2019.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances neural information processing systems, pp. 529-536, 2005.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint 1010:0266 2016.
Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon. Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5267-5276, 2019.
Xiangtai Li, Li Zhang, Ansheng You, Maoke Yang, Kuiyuan Yang, and Yunhai Tong. Global aggre- gation then local distribution in fully convolutional networks. arXiv preprint arXiv:1909.07223 2019.
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2604-2613, 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother. -ereels-cost-effeccivee region-based active learning for semantic segmentation. arXiv preprint 1.X0:1810009926.1. 2018.
Maninis, Sergi Caelles, Jordi Pont- Just-, and Luc Van Gool. Deep extreme cut: From extreme points to object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 616-625, 2018

---

Under review as a conference paper at ICLR 2020
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for seman- tic segmentation. In Proceedings of the IEEE international conference on computer vision, pp. 1520-1528, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-244-022 241. Springer. 2015.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211225 252, 2015. doi: 10. 12007/s11263-011 1263-015-8811
Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013.
Towaki Takikawa, David Acuna, Varun Jampani, and Sanja Fidler. Gated-senn: Gated shape cnns for semantic segmentation. arXiv preprint arXiv: 2019.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve cemi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195-1204 2017.
Jiafeng Xie, Bing Shuai, Jian-Fang Hu, Jingyang Lin, and Wei-Shi Zheng. Improving fast segmen- tation with teacher-student learning. arXiv preprint arXiv: 2018.
Qizhe Xie. Zihang Dai, Eduard Hovy, Minh- Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv: 1904. 12848, 2019.
Fisher and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Interna- tional Conference on Computer Vision (ICCV), 2019.
Sergey Zagoruyko and Nikos Komodakis.
Wide
residual networks.
arXiv preprint
1055v:1605.07146, 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz, mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp-
2881-2890, 2017.
Yi Zhu, Karan Sapra, Fitsum A Reda, Kevin J Shih, Shawn Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 8856- 8865, 2019.
Yueqing Zhuang, Fan Yang, Li Tao, Cong Ma, Ziwei Zhang, Yuan Li, Huizhu Jia, Xiaodong Xie, and Wen Gao. Dense relation network: Learning consistent and context-aware representation for semantic image segmentation. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3698-3702. IEEE, 2018.
10


Reviews:
[{"review_id": "SklrTN5TYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- key problem: semi-supervised semantic segmentation;\n- contributions: 1) combine the CutMix data augmentation of Yun et al 2019 with the standard consistency loss of  and the  structured consistency loss of Liu et al 2019, 2) state of the art results on Cityscapes.\n\nRecommendation: Reject\n\nKey reason 1: limited novelty.\n- What is the major difference with French et al 2019? Is it just the addition of the structured consistency loss of Liu et al? Figure 1 is almost identical to Figure 3 in French et al 2019.\n- What is the major novelty / insight besides the direct combination of the aforementioned papers and others like MixMatch (Berthelot et al 2019)?\n- Is the main contribution the faster computation of the structured consistency loss by limiting it to the CutMix boxes?\n\nKey reason 2: limited experimental validation.\n- The performance improvements are all <0.4%: are they statistically significant? What is the variance w.r.t. the random seed (important as the method is heavily relying on sampling random transformations)?\n- I would suggest conducting experiments on more than just one dataset, e.g., by adding COCO and/or Mapillary Vistas.\n- What is the performance of the approach when using only a subset of the labels (e.g., 10%, 25%, 50%) vs the full Cityscapes training set?\n\nKey reason 3: writing could be significantly improved.\n- There are numerous grammatical mistakes and unclear sentences, e.g., \"It regularizes the inter-pixel relationships consistent\", \"a lot of researches were\".\n- Figures 2a and 2b are identical.\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyeIOlXTFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- This paper introduces a structured consistency loss for semantic segmentation under the semi-supervised learning paradigm. Specifically, authors propose to include this term to force consistency between the predictions of two networks (a teacher and a student). To evaluate their method, authors employ images from the cityscapes dataset. Results show that the proposed method achieves competitive results compared to state-of-the-art architectures.\n- Contributions are unclear. The only contribution seems to be the ‘first study that employs semi-supervised learning and achieves state-of-the-art performance in semantic segmentation using the Cityscapes’. First, this contribution is insufficient to be considered as a technical contribution. And second, I feel authors mislead the reader with this assertion. \n- The paper is poorly written with many senseless sentences and weird word compositions (e.g., ‘our research designed the semi-supervised learning approach suitable for semantic segmentation’ or ‘In this study, we are aimed at bypassing the labeling cost problem with semi-supervised learning technique.’ or ‘It regularizes the inter-pixel relationships consistent, thereby allowing the network to learn more powerful generalization capabilities’ just to give few examples). This makes the reading of this work particularly difficult, as well as the understanding on some parts.\n- Authors fail to compile relevant works, particularly in semi-supervised segmentation, leading to a very weak related work section. Particularly in semi-supervised segmentation, more than one hundred papers have been published since 2015 only in major conferences on vision (CVPR,ICCV,ECCV) and learning (ICLR, ICML, NeurIPS). None of them, however, are listed in the related work section. Authors should significantly improve this section.\n- Which is the difference with French et al. (2019) and Liu et al. (2019)? Both seem to be using CutMix for semantic segmentation.\n- Where the labeled images are used? Authors show that labeled images and their corresponding ground truth masks are employed in eq 1) and 2). Nevertheless, it is never explained in which moment these are employed. Are they used to pre-traine the student and teacher networks? Are they mixed with unsupervised images during training? Please clarify. \n- Please add reference for the cross entropy loss with boundary label relaxation.\n- Furthermore, this paper contains many grammatical errors.\n- Even though the idea is interesting, I am inclined to reject this paper because there are many unclear details, the related work is insufficient and the paper needs a thorough proof-reading in english. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes to combine the CutMix data augmentation of Yun et al 2019 with the standard consistency loss of  and the  structured consistency loss of Liu et al 2019 and applies the resulting approach to the Cityscapes dataset.  The reviewers were unanimous that the paper is not suitable for publication at ICLR due to a lack of novelty in the method.  No rebuttal was provided.
Decision: Reject