Semi-supervised Semantic Segmentation using Auxiliary Network
Authors: Wei-Hsu Chen
Hsueh-Ming Hang
Venue:  (2020)
Keywords: deep learning
semi-supervised segmentation
semantic segmentation
CNN

Abstract: Recently, the convolutional neural networks (CNNs) have shown great success on semantic segmentation task. However, for practical applications such as autonomous driving, the popular supervised learning method faces two challenges: the demand of low computational complexity and the need of huge training dataset accompanied by ground truth. Our focus in this paper is semi-supervised learning. We wish to use both labeled and unlabeled data in the training process. A highly efficient semantic segmentation network is our platform, which achieves high segmentation accuracy at low model size and high inference speed. We propose a semi-supervised learning approach to improve segmentation accuracy by including extra images without labels. While most existing semi-supervised learning methods are designed based on the adversarial learning techniques, we present a new and different approach, which trains an auxiliary CNN network that validates labels (ground-truth) on the unlabeled images. Therefore, in the supervised training phase, both the segmentation network and the auxiliary network are trained using labeled images. Then, in the unsupervised training phase, the unlabeled images are segmented and a subset of image pixels are picked up by the auxiliary network; and then they are used as ground truth to train the segmentation network. Thus, at the end, all dataset images can be used for retraining the segmentation network to improve the segmentation results. We use Cityscapes and CamVid datasets to verify the effectiveness of our semi-supervised scheme, and our experimental results show that it can improve the mean IoU for about 1.2% to 2.9% on the challenging Cityscapes dataset.

Under review as a conference paper at ICLR 2020
SEMI-SUPERVISER SEMANTIC SEGMENTATION USING AUXILIARY NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, the convolutional neural networks (CNNs) have shown great success semantic segmentation task. However, for practical applications such as au- tonomous driving, the popular supervised learning method faces two challenges: the demand of low computational complexity and the need of huge training dataset accompanied by ground truth. Our focus in this paper is semi-supervised learning for semantic segmentation. We wish to use both labeled and unlabeled data in the training process. A highly efficient semantic segmentation network is our plat- form, which achieves high segmentation accuracy with a low model size and high inference speed. We propose a supervisde learning scheme to improve seg- mentation accuracy by including extra images without labels. While most existing semi-supervised learning methods are designed based on the adversarial learning techniques, we present new and different approach, which trains auxiliary CNN network that validates labels (ground-truth) on the unlabeled images. In the supervised training phase, both the segmentation network and the auxiliary network are trained using the labeled images. Then, in the unsupervised training phase, the unlabeled images are segmented and subset of image pixels are picked up by the auxiliary network; and then they are used as ground truth to train the seg- mentation network. Thus, at the end, all dataset images can be used for retraining the segmentation network to improve the segmentation results. We use Cityscapes and Cam Vid datasets to verify the effectiveness of our semi-supervised scheme. Our experimental results show that to 2.9% on the challenging Cityscapes dataset.
improve the mean IoU for about 1.2%
INTRODUCTION
Semantic Segmentation, which identifies the category label of each pixel, is an important task in computer vision. A number of convolutional neural network (CNN) based semantic segmentation systems have been developed in recent years such as [Cen et al. (201 b 2018): Long et al. (2013); Zhao et al. (2017); RRonneeennnnereeeeeeeeeeeeereeee et al. (2015). For practical applications such as autonomous driving, there a high demand for real-time processing and sufficient training data. Hence, different research directions have been explored. For example, some researchers proposed various efficient network structures (Chen et al. 2019; Paszke et al. 2016 [Poude et al. 2018, Lo et al., 2018 PPudel et al.) Zhao al. 2018 Romera al. 2017 Li et 2019), and the others focus on the weakly- or semi semi-supervised learning schemes (Hung et al.) 2018 Lin et al. 2016 Bearman
2019
Yu al. 2018
al.
2016
Qi et al. 2016 Rajchl al. 2016; Pathak al. 2015
Papandreou et al.
Hong
et al.
2015 Luc et al. 2016; Dai et al.
2015:
Pathak et al. 2014 Zhu et al.) 2019 Mittal et al.
2019
In this study, we first design an efficient segmentation network inherited from Chen et al. (2019) as our baseline model, and then propose a semi-supervised learning scheme. There are three training stages in our system. In the first stage, the segmentation network is trained using the labeled images. Then, in the second stage, an auxiliary network trained using the segmentation network model trained in the previous stage and the labeled data to generate the confidence map. Inspired by Hung et al. (2018), we use the concept confidence map to assign a confidence score to each segmented pixel the unlabeled images. In their work (Hung et al. 2018), the confidence map is the output of a GAN discriminator network (Goodfellow et al. 2014). where the discriminator network learns to distinguish between the segmentation map and the ground truth map. However, find that

---

Under review as a conference paper at ICLR 2020
the trusted (high confidence) regions in their confidence map are mostly located the large target objects, and thus the effectiveness of remi-supervised learning is limited. To obtain more reliable small object labels, we adopt different approach in generating the confidence map. In our approach, the confidence map is generated by an auxiliary (CNN) network, which is trained using the proposed auxiliary loss function. We carefully design the auxiliary loss function such that it leads confidence map, particularly, on the small objects. In the third stage, the unlabeled images are used as inputs the proposed system generate the labels. Therefore, we can use both originally labeled and unlabeled images to retrain the segmentation network model to achieve a better performance in the end.
reliable
In summary, our main contributions of this work are as follows. First, Based on DSNet-fast (Chen jtt al. 2019), we design a powerful segmentation network, which achieves a very good balance between speed and accuracy. It produces 73.9% mean IoU on the Cityscapes testing set with speed of 73 FPS on single GTX 1080Ti card. Second, We propose a semi-supervised learning scheme with the notion of auxiliary network, which can be used to annotate the unlabeled images. Third, with the help of auxiliary network, our semi-supervised learning method can include the unlabeled images in training, which improves the segmentation accuracy.
2 RELATED WORK
Recently, CNNs have been widely used in many fields of computer vision. For the semantic seg- mentation task, FCN (Long et al.] 2015) is a pioneer. replaces the fully-connected layers of the classification network (Krizhevsky et al.) 2012) with the convolution layers, and thus it can gener- ate dense labels on the input image of the same size. Then, SegNet (Badrinarayanan et al. 2017) was subsequently proposed, which uses symmetric encoder-decoder architecture for feature map down-sampling and up-sampling. The U-Net (Ronneberger et al. 2015) introduces the concate- nation operation to up-sample the features with different levels. PSPNet (Zhao et al. (017) and DeepLab (Chen et al. 201 20]7a/b] 2018) propose the atrous spatial pyramid pooling (ASPP) module to integrate multi-scale features. There are many other studies improving the segmentation results; however, for practical applications, high inference speed network is very desirable. For example, ENet (Paszke al. 20166) real-time segmentation model with good segmentation results. ICNet Zhao et al.] 2018) and BiSeNet (Yu et al. 2018) were recently proposed, and they aim at better balance between speed and accuracy.
Another challenge in training a semantic segmentation network is the need of a large amount of la- beled data (ground truth). The labeling cost of pixel-level annotation is extremely expensive. Hence, in recent years, the study of the weakly- and semi-supervised learning approaches to tackle this prob- lem attracted a lot of attention. For the weakly supervised methods, the pixel-level annotation is no longer used to train the segmentation network, but instead, other forms of annotations obtained at low costs are used for training. Several different types of weakly-supervision have been studied, in- cluding bounding box supervision (Rajchl et al. 2016 Dai et al. 2015), scribbles supervision (Lin) ctt al. (0166). point supervision (Bearman et al. 2016). and image- supervision (Qi et al.] 2016 Pathak al., 2014).. Particularly, the development of image-level supervision is the most popular lar one. Pathak et al. (2015) convert image-level labels to restrict the distribution of CNN output. Pa-
pandreou al. (2015) combine image-level labels using EM algorithms to train the segmentation model. Wei et al. (2018) propose a generic classification network, which adopts the convolutional blocks with different dilated rates to generate dense object localization maps. It can produce reli- able segmentation masks for training the segmentation model. For semi-supervised learning, [Hong al. (2015) propose a method to separately train the classification and segmentation networks, and then pass the information from the classification network to the segmentation network to reduce the search space for effective segmentation. Lue et al. (2016) employ an adversarial network to enhance the segmentation quality. Recently, Zhu et al. (2019) propose video prediction model to generate new samples for improving segmentation accuracy. Morcover Hung et al. (2018) propose self-taught learning approach based on the dense pixel-level probability maps produced by adver- sarial network. MMtttll et al. (2019) describe the dual-branch method including a semi eemi-supervised classification branch to learn from unlabeled data.
In this study, we propose an auxiliary network, which can determine the credibility of each pixel on the segmentation map of unlabeled images and then the trusted pixels can be used as additional
2

---

Under review as a conference paper at ICLR 2020
Segmentation Network
Image
Segmentation Map Class Probability Map
Auxiliary
Network
Confidence Map
Generated Annotation
Figure The process of generating annotations for unlabeled image.
ground truth data in training. Different from the previous works, we focus on the quality of the generated labels. We believe that the supervision signals of small objects are more important than the larger ones. Thus, we assign different weights to the generated tags of different categories, especially biased towards the small objects. In training the auxiliary network, we propose an auxiliary loss function containing two critical terms to achieve this objective.
3 METHODOLOGY
In this section, first describe the general framework of our semi-supervised semantic segmenta- tion scheme, and then we describe the architecture and design concepts of the proposed segmentation network and auxiliary network.
3.1 SEMI-SUPERVISED LEARNING USING AUXILIARY NETWORK
There are in total three training stages in our semi-supervised learning scheme. In the first stage, simply train the segmentation network using the labeled images. In this paper, we simply use the typical cross entropy loss in the supervised training of segmentation network. Then in the second stage, an auxiliary network is trained using the proposed auxiliary loss function. Both the originally labeled images and the previously trained segmentation network used in this stage. Our auxiliary network takes the class probability maps of size x W x C produced by the segmentation network input, where and W are the height and width of the image size and C' denotes the number of classes. The auxiliary network outputs a confidence map of size H x W x 1. Each pixel of the confidence map a probability value between and 1, which represents the credibility of that pixel on the segmentation map. In the third stage, we feed the unlabeled images into the segmentation network pre-trained the first stage to obtain the class probability map, and then pass this map through the auxiliary network to obtain the confidence map. Based on the probability value of each pixel of the confidence map, we can determine the segmentation reliability of each pixel on the segmentation map. Then, select the pixcls with high reliability as the annotated data points. And we ignore the pixels with lower reliability in the next-phase training. We set a threshold Taux on the confidence map to separate the high reliability pixels from the low reliability pixels. Hence, mask the error- error-prone pixels on the output segmentation map and produce the reliable annotated maps for the unlabeled images. Finally, since all the images now have labels, retrain the segmentation network using all them, and produce a more accurate segmentation model. Figure shows the process for generating annotated labels for the images without ground truth.
Loss function for auxiliary network To train the auxiliary network, we create another auxiliary ground truth map of dimension H W x Assuming that we have the original labeled ground truth class map and the estimated (segmented) class map produced by the segmentation network, the pixel value on the auxiliary ground truth map is set to if the class at that pixel on the estimated segmentation map matches that the ground truth class map; and it is set 0. if not. Different
3

---

Under review as a conference paper at ICLR 2020
from the typical binary cross entropy loss, assign weights to the terms in the loss function for auxiliary network, which defined below.
Laux Σ Whyohhhh Wh. an (x=)) + Vis.o) log(1 On, (x")))
where yh equals to the estimate class on the segmentation map at pixel (ht w) is identical to the class value on the ground truth label map at the same pixel, and yl.w equals 0 if they are different. In equation [ (x") denotes the confidence map output at pixel (h, w), when the auxiliary network takes the class probability map x" input. W h,w represents the class weighting value at pixel (h, w) the confidence map, and it is defined as 1/In(c + Pelass), inspired by ENet (Paszke et al. 2016), where we set equal to 1.04. The class weighting at different locations of confidence map is decided by the ground truth label map. For example, when the class at the pixel (h, w) on ground truth label map is class i, the class weight at that pixel on the confidence map is the weight value associated with class i. Class weighting term is used tackle the data imbalance issue. That is, some classes having few number of pixels are often unfavored in training. With class weighting, our auxiliary network performs well not only on the big objects but also on the small objects. Moreover, for different classes, the imbalance issue between positive and negative samples are quite different. Thus, we propose a weight term Th." which is defined as #negative pirelctaes! positive pizelctaze. and it calculated using the auxiliary network ground truth. Similarly, this weight term is location variant depending the ground truth label map. Both W h, and Th are the critical terms in the proposed auxiliary loss function. There were several versions tested and the current version gives the best result.
3.2 NETWORK ARCHITECTURE
3.2.
SEGMENTATION NETWORK
We adopt the DSNet-fast (Chen et al. 2019) as our backbone model, but we made some modifica- tions. Based on our observations, once the receptive field of the network is large enough to cover the entire image, making good use of the shallow-layer features more effective than increasing the network depth for the semantic segmentation task. Thus, we redesigned the initial block and the early block of DSNet-fast, and reduced the CNN layers in the deeper block to achieve better accuracy and inference speed. The overall architecture shown in Figure 2
Guature size
L/a
1/4
1/16
1/2
Encoder
Inital Biccis
Blocis
Blocks (192)
Block
Togot
Cony
Cony
Decoder
Uppcomple 7x
Up-campie
Uppesample
Decomy 4% (A (kcloous)
(96)
Outyat
Figure 2: The architecture the proposed segmentation network. "NB": non- non-bottleneck unit. "B": bottleneck unit. "T": transition layer. The number of output channels for each block are marked inside parentheses.
In the encoder part, adopt the same bottleneck unit and transition layer in DSNet-fast, and pro- pose new initial block and non- bon-bottleneck units. All our core units are shown in Figure 3 In the initial block, after applying the 3x3 convolution with stride 2 on the input image, we split into two branches. The 1x1 convolution applied to the left branch to integrate features and reduce the number of channels, and on the right branch, stack of two 3x3 convolutions is used to learn the
4

---

Under review as a conference paper at ICLR 2020
Images
Input (C)
Inpul (C)
2
1xl Cony (dogg)
3xc Cous (45)
3x3 Conv (C)
Ixl Cons (16)
bxl Corv (16)
3x3 Cony (gr)
Concatenation (64)
Concatenaiion (C116)
Concatenatioon [C
Av2. pociing
Figure 3: Proposed core units. (a) initial block. (b) non-bottleneck unit. (c) bottleneck unit. "C":: the number of channels from input. "gr": growth rate, which is set to 32 in all our bottleneck unit. The number of channels for each layer are marked inside parentheses.
features. Furthermore, the feature maps of these two branches are concatenated and followed by an average pooling with stride 2 to produce the output. Since the 1 x convolution does not take the spatial information into account, the feature map concatenated from the two branches have different receptive fields, and thus it helps the network to extract features different scales. On the other hand, due to dimension reduction made by the 1 convolution operation, the left branch helps the right branch to produce more feature channels under the constraint of fixed number of concate- nated feature maps. Similarly, we use the same concept to design our non-bottleneck unit. We find that this design strategy is indeed helpful in producing better segmentation results similar model complexity.
In the decoder part, we employ 3x3 convolutions with {24, 32, 16, 24] channels after the Blocks 1-4, respectively, to reduce the computational complexity. Furthermore, the up-sampling and con- catenation operations are adopted, same as DSNet-fast, to integrate the feature maps at different spatial levels. Finally, the concatenated feature map is up-sampled by a factor of 4 using a decon- volution kernel to produce the estimated segmentation map. All the convolution layers are followed by batch normalization (Toffe & Szegedy 2015) and ReLU activations.
3.2.2 AUXILIARY NETWORK
Our auxiliary network mainly consists five convolution layers, one atrous spatial pyramid pooling (ASPP) module, and deconvolution layer. The overall architecture is shown in Table [ auxiliary network architectures have been tested before we settle on the current design, which is quite simple but provides good results.
A few
Table 1: Auxiliary network architecture.
Input size
Block
Output size
512x1024x3
3x3 Conv, stride 2
256x512x32
256x512x32
3x3 Conv stride
256x512x64
256x512x64
3x3 Conv stride 2
128x256x 128
128x256x 128
3x3 Conv, stride
128x256x:56
128x256x256
3x3 Conv stride
128x256x256
128 x25xxxx62
ASPP
128x256x 160
128 256x 160
8x8 Deconv 4x
512x 1024x
5

---

Under review as a conference paper at ICLR 2020
Since find that detailed spatial information is vital in the auxiliary network, feature map size only 1/4 of the original image size before the deconvolution layer. We use the 3x3 convolution layers at the beginning. and then it is followed by single atrous spatial pyramid pooling module similar to DeepLab-vv3 (Chen et al. 2018 to extract multi-scale information as well as enlarging the receptive field. For the ASPP (Atrous Spatial Pyramid Pooling) module of our auxiliary network, we use the dilated convolution with dilated rate {1. 2, 4. 8, 16] for branch 1-5 respectively, and do use the image pooling branch. Each branch in our ASPP module has 32 feature channels. Finally, the deconvolution kernel is applied after the ASPP module to rescale the output to the input size and generate the dense confidence map; moreover,
followed by the sigmoid function to limit the output between 0 and 1. Same as the segmentation network, all the convolution layers are followed by batch normalization (loffe & Szegedy 2015) and ReLU activations.
4 EXPERIMENTAL RESULTS
In this section, we first introduce two datasets, Cityscapes (Cordts al. 2016) and CamVid (Brostow et al., 2008), that are used to evaluate proposed method. Both are popular road scene datasets. Then, we describe our training details for the segmentation and the auxiliary networks. At the end, we show the comparisons with the state-of-the the-the-rt methods as well as some visual results.
4.1 DATASETS
Cityscapes The Cityscapes dataset is road scene dataset with the image resolution of 1024 x2048, and it provides 19 object classes for evaluation. There are two forms of annotation data in its database, namely, fine and coarse. Only the fine annotation set is used in our experiments, where it contains 2975, 500, 1525 images for training, validation, and testing respectively. We re- size the image to 512x 1024 for training and testing due to the hardware limitation; however, for evaluation on the testing set, we restore the resolution of segmentation results to the original size for fair comparison.
Vid The Cam dataset is also a road scene dataset, but its image resolution (about 360 480) is much smaller than that of the Cityscapes dataset. It provides 11 semantic classes, and contains 367, 101, 233 images for training, validation, and testing.
4.2 IMPLEMENTATION DETAILS
We use PyTorch framework to implement our method, and we measure the network speed on single GTX 1080Ti GPU, The popular metric, mean IoU, is used to compute the segmentation performance. In addition, the encoder part of our segmentation network has been pre-trained on ImageNet (Deng et al., (009) to produce better initial parameters.
Segmentation network To train the segmentation network for the Cityscapes dataset, stochastic gradient decent (SGD) optimization with momentum 0.9, weight decay 0.0001, and batch size 4. We adopt the poly learning rate policy described in Chen et al.) (2017a) with power 0.9 in all our experiments and the initial learning rate is set to 0.05 here. We train it for 200 epochs in total. For the Cam Vid dataset, the SGD optimization is also used but with momentum 0.9, weight decay 0,0005, and batch size 8. In addition, the initial learning rate is set to 0,075, and we train it for 150 epochs in total.
the
For both Cityscapes and Cam Vid datasets, we adopt the class weighting scheme same as in Paszke al. (2016) with an additional haper-parameter с set 1.1, and the data augmentation strategies are employed to boost the performance. Our data augmentation strategies include random horizontal flip. random pixel translation, and random scaling. For the Cityscapes dataset, the scaling factors in random scaling are (0.75, 1.0, 1.25, 1.5, 1.75], and for the Cam Vid dataset, they are {0.8, 1.0, 1.3, 1.7).
Auxiliary network To train the auxiliary network for Cityscapes dataset, we use the Adam op- timization (Kingma & Ba 2014) with momentum 0.9, weight decay 0.0001, batch size and the

---

Under review as a conference paper at ICLR 2020
Table 2: Comparison with the state-of-the-arr methods the Cityscapes testing set. The efficiency- based methods are adopted for comparison. "f": inference speed using Titan X GPU, "4" inference speed using Titan XP GPU. "tt": inference speed using GTX 1080Ti GPU.
Method
Training dataset
Mcan IoU (%)
Speed (FPS)
Params
ENet (Paszke et al.
2016
Fine
58.3
76.9
0.37 M
ContextNet (Poudel et
2018
Fine
66.1
18.3
0.85 M
EDANet (Lo et al.
2018
Fine
67.3
81.3f
0.68 M
Fast-SCNN (Poudel et al.
2019)
Fine, Coarse
68.0
75.31
1.11 M
BiSeNet Yu et al.
2018
Fine
68.4
105.8*
5.8 M
DSNet-fast (Chen al.
2019)
Fine
69.1
52.91
3.0 M
ICNet Zhao et al.
2018
Fine
69.5
30.31
6.68 M
ERFNet [Romera et al.
2017
Fine
69.7
41.7f
2.1 M
DF1-Seg-d8 (Li et 2019
Fine
71.4
136.9
DF2-Seg2 (Li et al.
2019)
Fine
75.3
56.3tt
Ours
Fine
73.9
55.81
73.2
2.11 1 M
initial learning rate is set to 0.0005. For the CamVid dataset, different from the former, the weight decay for Adam optimization is set to 0.0005, and batch size is set to 8.
The training duration 50 epochs for both datasets, and we use the proposed auxiliary loss function in training the auxiliary network. The data augmentation strategies for training auxiliary network are same as described the above.
4.3 PERFORMANCE EVALUATION
Evaluation on Cityscapes dataset In order evaluate the performance of the proposed segmen- tation network, we test it on the Cityscapes testing set, and compare the results with the state-of-the art networks. It is worth noticing that we do not adopt any testing technology such as multi-scale testing in the evaluation process. From Table we find that our segmentation network can achieve a good trade-off between speed and accuracy. achieves 73.9% mean IoU with fewer parameters and higher inference speed comparing to the other segmentation nets.
Moreover, to evaluate the proposed semi-supervised learning scheme, similar to that work (Hung et al.) (018), we randomly sample half of images from the training set and call them the labeled data, and the other half are the unlabeled data. We conduct the experiments on the Cityscapes val- idation set. First of all, we train the segmentation and auxiliary network using the labeled images as deseribed Section 3.1 and then we measure the confidence map performance. In this process, feed the validation set images the entire network, and then mask the error-prone pixels of the segmentation map by the confidence map. Then, compare the masked segmentation map (generated annotations) with its true ground truth to calculate the accuracy. We three performance met- rics for evaluation, and the experimental results are shown in Table 3 where Taue represents the threshold set for the confidence map. The mean IoU metric is the mloU calculated by the gener- ated annotations and the true ground truth without considering the masked pixels (by the confidence map). The average class semi ratio the average value the "class semi ratio" of all classes. To compute the class semi ratio for class i, we only consider the region of pixels that belong to class i on the ground truth label map. The denominator of class semi ratio for class is the number of all pixels; and the numerator is the number of pixels not masked by the confidence map, the estimated segmentation map. calculating the selected pixels, we first count the pixels of the esti- mated segmentation map that are selected by the confidence map and are used as ground truth (GT) in training. Then, we compute its ratio with respect to the entire image. From Table 3 we can find accurate, but the selected pixels
that with higher threshold, the pixels selected GT

---

Under review as a conference paper at ICLR 2020
Table 3: The measuring results for generated annotations the Cityscapes validation set using half of training data.
Taux
Mcan IoU (%)
Average class semi ratio (%)
Selected pixels (%)
0
70.3
T00
100
0.7
83.7
75.6
86.1
0.8
85.6
71.2
83.3
0.9
88.2
64.2
78.0
Table 4: The per-class IoU for generated annotations the Cityscapes validation set using half of training data.
IoU(%) (%) Class semi ratio (%)
Class
T Touz
0.0
Toun = 0.7
Taun = 0.8
Taus = 0.9
Road
97.4/ 100
99.0795.1
99.2 / /95.0
99.4793.4
Sidewalk
80.5 100
91.07 83.0
92.17 79.9
75.5
Building
90.9/ 100
96.5 85.4
97.17 82.5
97.8 77.4
Wall
49.2/ 100
62.1 67.99
63.8 / 62.8
66.7 54.6
Fence
52.8/ 100
67.5 68.5
70.0 63.0
63.0
73.8 54.8
Pole
59.4/ 100
78.0/72.2
62.0/66.3
87.57 56.0
Traffic light
63.57 100
81.6774.4
85.2768.7
89.9759.3
Traffic sign 71.9/ 100
88.0 79.5
90.3/75.1
/
Vegetation
91.3/ 100
97.47 86.1
97.97
98.6/78.7
Terrain
59.4/ T00
74.1/720
76.3 67.7
80.1/60.4
Sky
93.5/ 100
98.6/91.4
98.8 / 89.5
99.2/
Person
75.5/ 100
90.6 / 80.5
92.8 75.8
95.2/67.6 67.6
Rider
53.87 TO0
77.1/
75.3759.0
81.076688
Car
92.7 100
98.6/ 89.4
98.9/87.4
99.3 83.9
Truck
68.7 100
86.7 70.88
88.0/67.4
/9.9/62.3
Bus
70.9 100
89.1 70.6
90.8 67.3
9
Train
51.3/ 100
52.99
73.1/47.4
76.07
Motorcycle
42.7/ 100
62.37 52.4
65.1744.5
68.3 33.3
Bicycle
70.1 100
86.7 / 76.88
89.9 70.7
93.6/60.6
fewer; this is trade-off. In addition, shown in Table confidence map also produce good results on the small objects with a higher threshold. Thus, the generated annotations used for retraining segmentation network have good quality.
After generating the annotated labels for 1/2 unlabeled images, we retrain the segmentation network using all images. The experimental results of selecting pixels based different threshold values shown in Table The proposed semi-supervised learning scheme can improve the performance for about 1.0% 1.2%. If T auw is set to O, it means that all the pixels of the estimated segmentation map are used as ground truth for unlabeled image, which is obviously unreasonable and it decreases
Table 5: The proposed semi-supervised learning segmentation results on the Cityscapes validation based different threshold values on the confidence map.
Data amount
Labeled data
Unlabeled data
Touz
Mean IoU (%)
1/2
1/2
0
69.9
1/2
1/2
0.7
71.3
1/2
1/2
0.8
71.4
1/2
1/2
0.9
71.5
1/2
1/2
70.3
8

---

Under review as a conference paper at ICLR 2020
Table 6: Comparison with Hung al. 2018) and Mittal et al. (2019) the Cityscapes validation
sct.
Data amount (labeled data unlabeled data)
Method
1/8 + 7/8
1/4 3/4
1/2 + 1/2
+ 0
(Hung et al.
2018) baseline
55.5
59.9
64.1
66.4
Hung et
2018)
+ adversarial learn.
57.1
61.8
64.6
67.7
Hung et al.
2018)
+ adversarial learn. semi
58.8
62.3
65.7
(Mittal ct al.
2019) baseline
56.2
60.2
66.0
(Mittal al.
2019) adversarial learn. semi
59.3
61.9
65.8
Ours baseline
56.8
63.0
66.
69.6
Ours + semi
60.7
65.5
67.7
Ours baseline (using proposed segment. net)
57.7
65.5
70.3
74.4
Ours semi (using proposed segment. net)
60.2
68.4
71.5
Figure 4: Sample results on the Cityscapes validation set using half of training data. From left to right: (a) image (b) ground truth (c) segmentation results without semi-supervised learning (d) segmentation results with sumi-supereised learning
the accuracy. Since Touz at 0.9 gives the best results, we use the same value in the following experiments.
To verify the effectiveness of our approach, we conduct experiments with different amount of data, and shows the comparison with Hung al. (2018) and Mittal et (209 in Table 6 For the fair comparison, we also adopt the Deeplab abbv2 framework as our backbone segmentation net model. Under the same settings that using 1/2, 1/4, and 1/8 training images as labeled data and using the rest of images unlabeled data, our semi-supervised learning method can lead to higher perfor- mance improvements. It can improve the mean IoU by 3.9,, 2.5%, and 1.6% for 1/8, 1/4, and 1/2 training datasets, respectively. Particularly, compared with Hung al. (2018), if we focus learning mech- the performance differences between the schemes with and without improvement. Furthermore, anism, semi-supervised learning approach offers quite significant when our proposed segmentation network is used as the backbone segmentation net, it increases the mean IoU for another 1.2% 2.9%. At the end, we show some sample visual results in Figure 4
Evaluation Cam Vid dataset We also evaluate scheme the Cam Vid dataset. As the results shown Table / our semi-supervised learning method can boost the performance of the segmentation network, and it can achieve 1.3% improvement mean IoU.

---

Under review as a conference paper at ICLR 2020
Table 7: Comparisons the Cam testing set.
Data amount
Method
Labeled data
Unlabeled data
Mean IoU (%)
Ours baseline (using proposed segment. net)
1
71.8
Ours baseline (using proposed segment. net)
1/2
1/2
69.3
Ours semi (using proposed segment. net)
1/2
1/2
70.6
5 CONCLUSIONS
In this paper, we first propose a highly efficient segmentation network as our platform, and then we design a superisseee learning scheme using an auxiliary network. The auxiliary network is used to verify the estimated segmentation map and to generate annotations on the unlabeled images. Equipped with the carefully designed auxiliary loss funetion in training, the auxiliary network per- forms well not only on the large objects but also the small objects. It shows that the unlabeled images together with the generated annotations (labels) can be used to retrain the segmentation network for better segmentation quality. Our experimental results the Cityscapes and Cam Vid datasets demonstrate the effectiveness the proposed method.
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions pattern analysis and machine intelligence, (0(12):2481-4499) 2017.
Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. What's the point: Semantic segmentation with point supervision. In European conference computer vision, pp. 549-565.
Springer, 2016.
Gabriel Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recog- nition using structure from motion point clouds. In European conference computer vision, pp- 44-57. Springer, 2008.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2017a
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.055: 2017b.
Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Ping-Rong Chen, Hsueh-Ming Hang, Sheng-Wa Wei Chan, and Jing-Jhih Lin. Dsnet: An efficient cnn for road scene segmentation. arXiv preprint Xiv: met.:/9940..2..... 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolu- tional networks for semantic segmentation. In Proceedings of the IEEE International Conference
Computer Vision, pp. 1635-1644 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248 255.255. leee, 2009.
10

---

Under review as a conference paper at ICLR 2020
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-262222 2680, 2014.
Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled deep neural network for semi- supervised semantic segmentation. In Advances neural information processing systems, pp- 1495-1503, 2015.
Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen-Yu Lin, and Ming-Hsuan Yang. Adversarial learning for semi-supervised semantic segmentation. arXiv preprint arXiv:1802.07933 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Xin Li, Yiming Zhou, Zheng Pan, and Jiashi Feng. Partial order pruning: for best speed/accuracy trade-off in neural architecture search. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9145-9153, 2019.
Di Lin, Jifeng Dai, Jiaya Jia, Kaiming Hc, and Jian Sun. Scribblesup: Scribble-supervised convolu- tional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3159-3167, 2016.
Shao- Yuan Lo, Hsueh-Ming Hang, Sheng- Wei Chan, and Jing-Jhih Lin. Efficient dense modules of asymmetric convolution for real-time semantic segmentation. arXiv preprint 11X9::180900632. 2018.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Pauline Luc, Camille Couprie, Soumith Chintala, and Jakob Verbeek. Semantic segmentation using adversarial networks. arXiv preprint 1xXiv:6611.08408 2016.
Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox. Semi-supervised semantic segmentation with high-and low-level consistency. arXiv preprint arXiv:1908.05770 2019.
Papandreou, L-Ch Chen, K Murphy, and AL Yuille. Weakly-and semi-supervised learning of denn for semantic image segmentation. arXiv preprint
2015.
Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello. Enet: A deep neu- ral network architecture for real-time semantic segmentation. arXiv preprini arXiv:1606.0000 2016.
Deepak Pathak, Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional multi- class multiple instance learning. arXiv preprint 11Xiv:1412.7144, 2014.
Deepak Pathak, Philipp Krahenbuhl, and Trevor Darrell. Constrained convolutional neural networks for weakly supervised segmentation. Proceedings of the IEEE international conference on computer vision, pp. 1796-1804 2015.
Rudra PK Poudel, Ujwal Bonde, Stephan Liwicki, and Christopher Zach. Contextnet: Exploring context and detail for semantic segmentation in real-time. arXiv preprint a (00e::8000.5450 04555. 2018.
Rudra PK Poudel, Stephan Liwicki, and Roberto Cipolla. Fast-senn: fast semantic segmentation network. arXiv preprint arXiv: 100.::9902.045000 2019.

---

Under review as a conference paper at ICLR 2020
Xiaojuan Qi, Zhengzhe Liu, Jianping Shi, Hengshuang Zhao, and Jiaya Jia. Augmented feedback in semantic segmentation under image level supervision. In European Conference on Computer Vision, pp. 90-105. Springer, 2016.
Martin Rajchl, Matthew CH Lee, Ozan Oktay, Konstantinos Kamnitsas, Jonathan Passerat- Palmbach, Wenjia Bai, Mellisa Damodaram, Mary A Rutherford, Joseph V Hajnal, Bernhard Kainz, et al. Deepeut: Object segmentation from bounding box annotations using convolutional neural networks. IEEE transactions on medical imaging, 36(2)6774-68,, 2016.
Eduardo Romera, José M Alvarez, Luis M Bergasa, and Roberto Arroyo. Efficient convnet for real- time semantic segmentation. In 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 1789-1794.
IEEE, 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng, and Thomas S Huang. Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7268- 7277, 2018.
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilat- eral segmentation network for real-time semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 325-341, 2018.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2881 2890, 2017.
Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 405 420, 2018.
Yi Zhu, Karan Sapra, Fitsum A Reda, Kevin J Shih, Shawn Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8856-
8865, 2019.
12


Reviews:
[{"review_id": "Skx3g4UIKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission introduces a semi-supervised method using auxiliary network for improved semantic segmentation. The authors modify a previous work as their main network architecture and use another small network as auxiliary branch. The framework can work in a semi-supervised setting since they can use confidence map to annotate unlabeled images to train the network. \n\nI give an initial rating of weak reject because (1) novelty in architecture design is trivial (2) the way of using unlabeled images is not new (3) experiments are not supportive (3) performance is not comparative to state-of-the-art. I will illustrate more as below.\n\n1. Architecture is not novel. As I mentioned, the authors adopt DSNet-fast as their main branch with minor modifications. And they use a simplified DeepLab architecture as their auxiliary branch. There is no ablation study or strong motivation to design this network. \n\n2. Using an auxiliary branch to measure confidence and try to get more labeled data is not new. There are many previous literature exploring similar ideas. Please clarify the differences between the literature and this submission in either introduction or related work section. \n    (1) Universal Semi-Supervised Semantic Segmentation, ICCV 2019\n    (2) Improving Semantic Segmentation via Video Propagation and Label Relaxation, CVPR 2019\n    (3) Semi-Supervised Semantic Segmentation with High- and Low-level Consistency, Arxiv 2019\nEspecially for (2) and (3), (2) use unlabeled images with confidence map, (3) has two branches as well. Both of them has results on Cityscapes. \n\n3. In table2 , speed comparison is not intuitive. The methods are evaluated on too many kinds of hardware and can not be directly compared. It is good to re-evaluate all algorithms on a single hardware, or remove this column. \n\n4. The experiment setting is too simple. Divide existing training data into half will make it hard to compare with other approaches. All the experiments in this submission can only show you are better than the baselines, but can't convince me your approach actually works. Because the training distribution is similar, it is easy for the auxiliary network to generalize. I would suggest the authors to use all the training data, and bring additional unlabeled images into picture to see what will happen. In addition, if you use all training data, you can make fair comparisons to many literatures and demonstrate the effectiveness of your approach.\n\n5. What is \"ours\" and \"ours with proposed network\" in table 5, please clarify.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1ejMTzTtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- This paper proposes a semi-supervised learning strategy for semantic segmentation of road scenes. Specifically, authors propose to include an auxiliary network that will predict the confidence (at pixel-level) of the predictions on unlabeled images. These confidence values will be used to  generate a new auxiliary ground-truth to retrain the network using the unlabeled images.\n- Even though the idea is somehow interesting and results seem to improve with respect to the baselines, this paper is very similar to standard semi-supervised learning approaches for natural images that employ image proposals (i.e., EM-based methods). In those works, unlabeled images are segmented with the network trained on labeled images, generating some proposals. These proposals are later employed as a fake ground truth to re-train the network employing both labeled and unlabeled images. The only difference in this work is to employ the virtual confidence map to mask-out some pixels (those with lowest confidence values).  \n- Related work section is extremely weak. Authors merely mention few papers (some other relevant papers are missing), and throw a sentence for each one, without making connections between works. This makes difficult to place their work among the literature (e.g., which limitations of previous approaches the current method intend to address?). Authors should significantly improve this section.\n- I am not sure about the fact that employing only the probability maps as input to generate a confidence map is reliable, if no other information is employed. These predictions (e.g., confidence map) will be based only on the probabilities obtained by the first network. While this is already a good indicator of the confidence of the network to make those predictions, I believe that input images should also be included. The intuition behind this is that there may exist some regions with similar probabilities (from first network), which are incorrectly classified (leading to 0-masked pixels on the auxiliary ground-truth) in some cases, while correctly classified in other situations (leading to 1-masked pixels on the auxiliary ground-truth).  \n- Eq.1) is basically the standard cross-entropy loss, with the difference of the weighting terms.\n- a_{h,w} is the softmax of the auxiliary network, isn’t it?\n- Further, authors threshold the values of the confidence map to generate the new auxiliary ground truth. Why not to use the raw values so that each pixel is weighted differently according to its importance? \n- Authors make some claims which were never demonstrated. For example, they mention that the proposed approach performs better on small targets than previous approaches. Nevertheless, only mean results (over all the classes) are shown. To this end authors should report per-class performances, instead of the mean.\n- Furthermore, authors make several over claims, misleading information. For example, they mentioned that they proposed a highly efficient segmentation method. Nevertheless, from Table 2 it can be observed that the proposed method ranks in the middle in terms of both speed and parameters, compared to other state-of-the-art models. Similarly, authors mention that their model is equipped with a carefully designed auxiliary loss function during training, while they basically employ a standard cross-entropy weighted by some values to account for imbalance between classes and between positive and negative pixels within the same class.   \n- The paper contains many grammatical errors.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlYwvx1qH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focused on the problem of semantic segmentation. The author first proposed an efficient segmentation network. Then a semi-supervised learning scheme with an auxiliary network is introduced to annotate the unlabeled images thus help boost the segmentation accuracy.\n\nClarity:\nThere is no novelty for the proposed fast segmentation network detailed in Sec 3.2. The auxiliary network for predicting the confidence map may look interesting, while the experimental results are not convincing.\n\nExperiments:\n1. The design of the fast segmentation network in this paper is boring and not much related to the title.\n\n2. The experimental results regarding the effectiveness of the auxiliary network are based on a weak backbone, some stronger backbones like DeepLab and PSPNet should be included.\n\n3. What the results would be if using all labeled data and the newly added unlabeled data? Can the newly unlabeled data with labels from the auxiliary network help boosting the performance?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a semi-supervised learning approach to handle semantic classification (pixel-level classification). The approach extends Hung et al. 18, using a confidence map generated by an auxiliary network, aimed to improve the identification of small objects.

The reviews state that the paper novelty is limited compared to the state of the art; the reviewers made several suggestions to improve the processing pipeline (including all images, including the confidence weights). 
The reviews also state that the paper needs be carefully polished. 

The area chair hopes that the suggestions about the contents and writing of the paper will help to prepare an improved version of the paper. 

Decision: Reject