Semi-supervised semantic segmentation needs strong, high-dimensional perturbations
Authors: Geoff French
Timo Aila
Samuli Laine
Michal Mackiewicz
Graham Finlayson
Venue:  (2020)
Keywords: computer vision
semantic segmentation
semi-supervised
consistency regularisation

Abstract: Consistency regularization describes a class of approaches that have yielded ground breaking results in semi-supervised classification problems. Prior work has established the cluster assumption\,---\,under which the data distribution consists of uniform class clusters of samples separated by low density regions\,---\,as key to its success. We analyze the problem of semantic segmentation and find that the data distribution does not exhibit low density regions separating classes and offer this as an explanation for why semi-supervised segmentation is a challenging problem. 
We then identify the conditions that allow consistency regularization to work even without such low-density regions. 
This allows us to generalize the recently proposed CutMix augmentation technique to a powerful masked variant, CowMix, 
leading to a successful application of consistency regularization in the semi-supervised semantic segmentation setting and
reaching state-of-the-art results in several standard datasets.

Under review as a conference paper at ICLR 2020
SEMI-SUPERVISED SEMANTIC SEGMENTATION NEEDS STRONG, HIGH-DIMENSIONAL PERTURBATIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
Consistency regularization describes a class of approaches that have yielded ground breaking results in semi-supervised classification problems. Prior work has established the cluster assumption - under which the data distribution con- sists of uniform class clusters of samples separated by low density regions- -s as key its success. We analyze the problem of semantic segmentation and find that the data distribution does not exhibit low density regions separating classes and offer this as an explanation for why semi-supervised segmentation a challeng- ing problem. We then identify the conditions that allow consistency regularization to work even without such low-density regions. This allows us to generalize the recently proposed CutMix augmentation technique to powerful masked variant, CowMix, leading to a successful application of consistency regularization in the semi-supervised semantic segmentation setting and reaching state-of-the-art re- sults in several standard datasets.
INTRODUCTION
Semi-supervised learning offers the tantalizing promise of training a machine learning model with limited amounts of labelled training data and large quantities of unlabelled data. These situations of- ten arise in practical computer vision problems where large quantities of images are readily available and generating ground truth labels acts as bottleneck due to the cost and labour required.
Consistency regularization (Sajjadi et al., 2016b Laine & Aila 2017 Miyato et al. 2017 Oliver jet al., 2018) describes a class seme-supervised learning algorithms that have yielded state-of- the-art results in semi-supervised classification, while being conceptually simple and often easy to implement. The key idea is to encourage the network to give consistent predictions for unlabeled inputs that are perturbed in various ways.
The effectiveness of consistency regularization is often attributed to the smoothness assumption (Miyato et al. [2017] Luo et al.) 2018) cluster assumption (Chapelle & Zien 2005 (Sajaadi al.) 2016a]]Shr (hhu et al. 2018 Verma et al. 2019). The smoothness assumption states that samples close to each other likely have the same label. The cluster assumption special case of the smooth- ness ussumption states that decision surfaces should lie in low density regions, not crossing high density regions. This typically holds in classification tasks, where most successes of consistency regularization have been reported so far.
On a high level, semantic segmentation classification, where each pixel is classified based on its neighborhood. It is therefore intriguing that consistency regularization has not been clearly ben- eficial in this context. We make the observation that the distance between patches centered on neighboring pixels varies smoothly even when the class of the center pixel changes, and thus there are no low-density regions on class boundaries. This alarming observation leads us to investigate the conditions that can allow consistency regularization to operate even in these conditions. Our key insight is that this is indeed possible, and that previous attempts have yielded little success primarily because the perturbations they used were not strong enough, and especially not high-dimensional enough.
We show that flexibly masked variant of CutMix YYun al. 2019), which call CowMix based on the mask appearance, does realize significant gains in supervuee semantic segmentation. This result clearly signposts a direction where further improvements are likely to be available.

---

Under review as a conference paper at ICLR 2020
2
BACKGROUND
Our work relates to prior art in three areas: recent regularization techniques for classification, semi- supervised classification with a focus on consistency regularization, and semantic segmentation.
2.1 MIXUP, CUTOUT, AND CUTMIX
The MixUp regularizer of Zhang et al. 20118) improves the performance of supervised image, speech and tabular data classifiers by using interpolated samples during training. The inputs and target labels of two randomly chosen examples are blended using the same randomly chosen factor.
The Cutout regularizer VeVeiss & Taylor (2017) augments an image by masking a rectangular region to zero. The recently proposed CutMix regularizer of Yun et al. (2019) combines aspects of Mp and Cutout, cutting a rectangular region from and pasting it over A. MixUp, Cutout, and CutMix improve supervised classification performance, with CutMix outperforming the other two.
2.2
SEMI-SUPERVISED CLASSIFICATION
The II-model of Laine & Aila 2017) passes each unlabeled sample through classifier twice, ap- plying two realizations of stochastic perturbation process, and minimizes the difference between the resulting class probability predictions. Their temporal model maintains per-sample moving average of historical predictions and encourages subsequent predictions to be consistent with the average. Sajjadi et al. (2016b) similarly encourage consistency between the current and historical predictions. MMyato et al. (2017) improve the results by replacing the stochastic perturbations with adversarial directions, thus focusing on perturbations that are closer to the decision boundaries.
The mean teacher model of Tarvainen & Valpola (201)) encourages consistency between predictions of a student network and a teacher network. The teacher's weights are an exponential moving average (Polyak & Juditsky 1992) of those of the student, leading to faster convergence and improved results. (French et al. (2018) adapt the mean teacher approach for domain adaptation.
Interpolation consistency training (ICT) (Verma et al. 2019) and MixMatch (Berthelot et al. 2019) both combine MixUp (Zhang et al. 2018) with consistency regularization. ICT uses the mean teacher model and applies MixUp to unsupervised samples, blending input images along with teacher class predictions to produce blended input and target to train the student. MixMatch stochastically perturbs each sample multiple times and averages the predictions to produce unsuper- vised targets; MixUp is applied to labeled as well as unlabeled samples.
2.3 SEMANTIC SEGMENTATION
Long al. (2015) finetune pre-trained VGG-16 (Simonyan & Zisserman 2014) image classifier to produce a dense set of predictions for overlapping input windows. They effectively transform the image classifier into fully convolutional network that be used segment input images of arbitrary size. Various methods were proposed for increasing the localization accuracy of the results (Long al.) 2015 Chen et al., 2014; Mostajabi decoder networks (Badrinarayanan et al. 2015 Ronneberger et al. 2015) led to a solution where the output resolution natively matches the input. In these recent methods the encoder downsamples the input progressively, similarly to image classifiers, the decoder performs progressive upsampling, and skip connections route data between the matching resolutions of the two networks, improving the ability to accurately segment fine details.
al. (2044) until the introduction of encoder-
number of approaches for semi-supervised semantic segmentation use additional data. Kalluri et al. (2018) use data from two datasets from different domains, maximizing the similarity between per-class embeddings from each dataset. Stekovic et al. (2018) use depth images and enforced ge- ometric constraints between multiple views of a 3D scene. Relatively few approaches operate in strictly semi superviee setting. Hung et al. (2018) employ adversarial learning. using a discrimina- tor network that distinguishes real from predicted segmentation maps to guide learning. Perone & CChen- Adad (2018) apply consistency regularization to a MRI volume dataset and their method is the only successful application of consistency regularization to segmentation that are aware of.
2

---

Under review as a conference paper at ICLR 2020
(a) Example image
(b) Avg. distance to neighhor, patch size 15x15
(c) Avg. distance to neighbor, patch size 225 x25
In segmentation task, low-density regions rarely correspond to class boundaries. (a) An image from Camvid dataset. (b) Average 1,² distance between raw pixel contents of patch centered pixel p and four overlapping patches centred on the immediate neighbors of p. using 15x 15 pixel patches. (c) Same for more realistic receptive field size of 225 x 225 pixels. Dark blue indicates large inter-patch distance and therefore low density region, white indicates distance of The red lines indicate segmentation ground truth boundaries.
Figure I:
CONSISTENCY REGULARIZATION FOR SEMANTIC SEGMENTATION
Consistency regularization adds consistency loss term Leons the loss that is minimized during training OOliver et al. 2018). In classification task, cons measures distance d(-,-) between the predictions resulting from applying a neural network fo to an unsupervised sample æ and a perturbed version & of the same sample, i.e., Teans d fo(ā)). The perturbation used to generate á
depends on the variant of consistency regularization used. A variety of distance measures d(...)
have been used, e.g., squared distance (Laine & Aila 2017) or cross-entropy (Miyato et al. 2017).
Athiwaratkun al. (2019) analyze simplified version of the II-model (Laine Aila 2017) in which perturbation consists of additive Gaussian noise so that å = x + eh,, where h N (0, /) and d(.s.) :) is squared Euclidean distance. For small constant e, the expected value of the consistency loss term cons is approximately proportional to the square of the Frobenius norm of the Jacobian Jo (x) of the networks outputs with respect to its inputs:
E|Leons] = E [[] fo + eh) fe(=)]]] ] =
(1)
Thus, minimizing Leans directly flattens the decision function in the vicinity of unsupervised sam- ples. This illustrates clearly the mechanism by which consistency regularization encourages the network to move the decision boundary and its surrounding region of high gradient into gions low sample density.
3.1 WHY SEMI-SUPERVISED SEMANTIC SEGMENTATION IS CHALLENGING
We attribute the infrequent success of consistency regularization in semantic segmentation problems to the observation that low density regions in input data do not align well with class boundaries. As illustrated Figure [ the cluster assumption is clearly violated: how much the raw pixel content of the receptive field of one pixel differ from the contents of the receptive field of a neighboring pixel has effectively no correlation with whether the patches' center pixels belong to the same class or not. For the cluster assumption to hold, we would require the distances between the contents of receptive fields to large between classes and small within classes, which is not the case here.
The lack of variation in the patchwise distances is easy to explain from a signal processing perspec- tive. With patch shape B, the 7,2 distance between the pixel content of two overlapping patches centered at, say, horizontally neighboring pixels can be written / B * (Axx)² where denotes convolution and Axx is the horizontal gradient of the input image æ. The squared gradient image is thus low-pass filtered by a B-shaped box filter, which suppresses any fine details and leads to smoothly varying sample density across the image. We test this experimentally in Appendix
Our decision compare the pixel content of image patches is motivated by the fact that the perturbations used to generate from x in prior work operate in raw pixel (input) space. The success of virtual adversarial training (Miyato al. 2017) in semi-supervised classification in which f is
3

---

Under review as a conference paper at ICLR 2020
Isotropic perturbation
Constrained perturbation
S
(a) Low density region separating classes
(b) No low density region
(c) Distance map and contours
(d) Constrain dist. map contours
Figure 2: Toy 2D semi-supervised classification experiments. Blue and red circles indicate super- vised samples from class and respectively. The field of small black dots indicate unsupervised samples. The learned decision function is visualized by rendering the probability of class in green; the soft gradation represents the gradual change in predicted class probability. (a, b) Semi-supervised learning with and without low density region separating the classes. The dotted orange line in (a) shows the decision boundary obtained with plain supervised learning. (c) Rendering of the distance the true class boundary with distance map contours. Strong colours indicate greater distance to class boundary. (d) Decision boundary learned when samples are perturbed along distance contours in (c). The magenta line indicates the true class boundary. Appendix explains the setup detail.
an adversarial example chosen to maximise the difference in prediction from that of illustrates the significance of the input space distribution in semi- supersseee learning.
3.2 CONSISTENCY REGULARIZATION WITHOUT CLUSTER ASSUMPTION
Let us analyze simple toy example of learning to classify 2D points in a semi- fashion. Figure Z illustrates the setup where cluster assumption holds and there is gap between the unsu- pervised samples belonging to the two different classes. The perturbation used for the consistency loss a simple Gaussian nudge to both coordinates, and as expected, the learned decision boundary settles neatly between the two clusters.
In Figure pp. the cluster assumption violated and there are no density differences the set of unsupervised samples. In this case, the consistency loss does more harm than good good even though it successfully flattens the neighborhood of the decision function, it does so also across the true class boundary. In order for the consistency regularization to be net win, it would have to perturb the samples as much as possible, but at the same time avoid crossing the true class boundary.
In Figure Zb. plot the contours of the distance to the true class boundary, suggesting potentially better mechanism for perturbation. Indeed, when perturbations are done only along these contours, the probability of crossing the true class boundary is negligible compared the regularization po- tential in the remaining dimension. Figure 2 shows that the resulting learned decision boundary aligns well with the true class boundary.
Low-density regions provide an effective signal that guides consistency regularization by providing areas into which a decision boundary can settle. This illustrative toy example demonstrates an alter- native mechanism; the orientation of the decision boundary can be constrained to lie parallel to the directions of perturbation. We therefore argue that consistency regularization can be successful even when the cluster assumption is violated, if the following guidelines are observed: 1) the perturba- tions must be varied in order to cover as much the input space the same class as possible, 2) they must be high-dimensional in order to sufficiently constrain the orientation of the decision boundary in the high-ding sigh-dimessiona space of natural imagery, 3) the probability of a perturbation crossing the true class boundary must be very small compared to the amount of exploration in other dimensions, and 4) the perturbed inputs should be plausible, i.c., they should not be grossly outside the manifold of real inputs.
4

---

Under review as a conference paper at ICLR 2020
a
(x.)
Teacher network
mix Xp))
fo, (mix Xx)))
mix(So, (xa)ffo(xx)) (xx))
Mix
Mix
Loss
Xb
fo,(
Student network
licons
Teacher network
Figure 3: Illustration of mixing regularization for semi- supervisee semantic segmentation with the mean teacher framework. and b2 denote the weights of the student and teacher networks. respec- tively. The arbitrary mask M is omitted from the argument list of function mir for legibility.
3.3 CUTOUT AND CUTMIX FOR SEMANTIC SEGMENTATION
If we consider the classical augmentation-lis perturbations such translation, rotation, scaling, and brightness/contrass changes, is evident that these have low chance of confusing the output class (Athiwaratkun et al. 2019) but they also provide very little variation. Note that in context of semantic segmentation, geometric transformations need be applied in reverse for the result image before computing the loss (Ji et al.) 2018). As such, translation turns into no-op, unlike classification tasks where remains a useful perturbation (see Appendix D.I.I for more details). Adding noise is another questionable perturbation strategy although nigh-dimensional, such per- turbations are very unlikely to lie on the manifold natural images.
Out of previously proposed perturbation methods for consistency regularization, identify CutOut and CutMix as promising candidates for semantic segmentation as they provide a large variety of possible outputs and are class preserving. Both approaches use mask M with randomly chosen rectangular region. Our masks have M inside the rectangle, and M = O otherwise.
CutOut. To apply CutOut in semantic segmentation task, we mask the input pixels with M and disregard the consistency loss for pixels masked to 0 by M. Using square distance as the metric, we have T.co = ||M @ (Je(M ④ 2r) fe(=))] where denotes elementwise product.
CutMix. CutMix requires two input images that we shall denote T, and Tg that we mix with the mask M. Following ICT (Verma al. (2019)) we run both input images as well as the mixed variant through the network. The consistency loss is then taken between the segmentation of the mixed image and the mix between the segmented input images. To simplify the notation, let us define function miz b, M) (1 M) @ + M @ b that selects the output pixel based on mask M. We can now write the consistency loss for segmentation CutMix as
L Leons
|| mér (x.),, fo())), M) fo min((a, 26, M)) 13
So far we have assumed that both original and perturbed images are segmented using the same network fa. In this sense, our approach is similar to the II-network Laine & Aila (2017), although mean teacher (Tarvainen & Valpola. 2017) is known to outperform it in image classification tasks. Our preliminary tests indicated that the same true for semantic segmentation, and therefore all the experiments in this paper use the mean teacher framework. Specifically, in CutOut, unperturbed image æ using the teacher network and the perturbed image M O x using the student network. In CutMix sor--folllee following Verma et al. (2019) we similarly segment the original images Ia and 24 using the teacher network, and the mixed image using the student network. The computation is illustrated in Figure
segment the
To analyze how much variation these perturbations provide, we note that the original CutOut al- ways masks out fixed-size square random position, so the resulting mask has only two degrees
5

---

Under review as a conference paper at ICLR 2020
64
Figure 4: Example CowOut/CowMix masks with 0.5 and varying in 384x388 pixel image.
of freedom. The original CutMix has three degrees of freedom in choosing the position and size of the rectangle (the aspect ratio is fixed), and in addition it replaces the contents of the rectangle from another training image, providing one further degree of freedom. Probably more importantly, the purtuu-perturbed images are rather implausible because real data rarely contains axis-aligned constant-color rectangles, whereas the content-filled rectangles of CutMix are not nearly as conspic- uous. In semantic segmentation, we deviate from the original methods in that we choose all four parameters of the rectangle at random in order to obtain as much variation as possible.
3.4 COWOUT AND COwMIX
The use of a rectangular mask restricts the dimensionality of the perturbations that CutOut and CutMix produce. Intuitively, more complex mask that has more degrees of freedom should provide better exploration of the plausible input space. We propose combining the semantic CutOut and CutMix regularizers introduced above with novel mask generation method, giving rise two regularization methods that we dub CowOut and CowMix due to the Friesian cow -like texture of the masks.
To generate a mask with given proportion of pixels having M = O, we start by sampling Gaussian noise image (0, I). convolve with a Gaussian smoothing kernel, and threshold the result at r erf (2p - 1) 2S, + Sn, where Sn and S. are the mean and standard deviation of the smoothed noise, respectively. The standard deviation a of the Gaussian smoothing kernel determines the average size of features in the mask and is drawn from log-uniform distribution: log (a min: maax). Figure shows example masks with varying values a.
To estimate how much variation these masks contain, we first observe that if smoothing were performed before thresholding, we would obtain binary per-pixel Bernoulli mask. This has an exponential amount of variation with respect to the area of the receptive field: a receptive field with area A has 2 possible masks. Applying the smoothing filter has similar effect as zooming the mask -f if the smoothing constant is doubled, variation (see, e.g., Figure between the amount of variation, i.e., the strength of the resulting perturbation, and plausibility of the masked/mixed image. If too little filtering is performed, we obtain a huge amount of variation but the perturbed images are unrealistic, and with too much filtering the images are very realistic but the amount of variation is small. As a geometric interpretation. we hypothesize that the more varied masks help the network to cope with occlusions better than the rectangular masks that can mimic occlusions only along horizontal and vertical edges.
expect to obtain roughly 4x fewer "bits" of vs. a = 16). The suitable value for a is thus an empirical tradeoff
Considering potential future work, we note that it should be possible to improve the results further with more principled method for choosing so that its distribution is automatically adapted to the training data.
4 EXPERIMENTS
We will now describe describe experiments and main results. We will start by describing the training setup, followed by an investigation of various perturbation methods the context of semi- supervised semantic segmentation, and conclude with comparison against the bate-of-the-and

---

Under review as a conference paper at ICLR 2020
Sup. baseline Std. aug 48 6666
ICT
Cutout
CutMix
CowOut
CowMix
Fully sup.
46.24% 2.21 48.29% 2,01
53.09% 255 50.95%
52.93%
55.06%
64. 19% a04
Table 1: Measurements and ablation on CAMVID test set. Our results are mean intersection-over- union (mloU) presented as mean + std. dev computed from 10 runs. Training was run for 300 epochs, except in (a) standard augmentation and (b) CutMix experiments where convergence failures were observed. For those experiments, the results were taken after (a) 50 and (b) 100 epochs.
4.1 TRAINING SETUP
We use two segmentation networks in our experiments: 1) U-Net (Ronneberger et al. 2015 with ResNet-50 (He et al. 2016) based encoder that was pre-trained using ImageNet and a randomly ini- tialized decoder (Appendix D.2), and 2) DeepLab v2 network (Chen al. 2017) based ResNet- 101 and pre- trained for semantic segmentation using the COCO (Lin et al. 2014) dataset, as used by Hunng al. (0118).
We minimise the sum of the supervised cross-entropy loss and consistency loss Loons with weights of and 10 respectively. We use the ean teacher algorithm (Tarvainen & Valpola (017). as detailed in Appendix [.3] We replace the sigmoidal ramp-up of the consistency regularization weight (Laine)
& Aila 2017 Tarvainen & Valpola 2017) with the average thresholded confidence of the teacher
network (see Appendix D.3.4). which increases as the training progresses (French et al. 2018). Our
implementation uses the Py PoTorc framework and will be made available.
4.2 COMPARISON OF PERTURBATION METHODS
The CAMVID (Brostow et al. 2008) training set consists of 367 images. We chose 10 subsets of 30 labeled images that used for the supervised loss, while all training images were used to compute the consistency loss. The U-net setup is used in this test.
The results for different perturbation methods are given in Table Perturbations based on standard data augmentation (flips, rotations, brightness, etc., see Appendix D.I) and Interpolation Consis- tency Training (ICT) resulted in no measurable improvement in the mean IoU score compared the baseline of supervised training using only the labeled samples. The Cutout and CutMix experiments used masks with single random rectangle, and led to clear improvements over the baseline.
For CowOut and CowMix, we generated the masks using a E [4, 16). For CowOut, we found that choosing the masked pixel proportion randomly with p (0(0, 1) produced the best results, whereas for CowMix it was optimal to always use p = 0.5. CowMix led to the best result in this experiment, bridging approximately half of the gap between the supervised baseline and the fully supervised reference result.
We tuned all syper-parameters using the CAMVID dataset due to small size and fast run-time. We found that these hyper-parameten also worked well for other, larger datasets.
4.3 RESULTS ON CITYSCAPES AND PASCAL VOC
We will now compare our results against the state-of-the-ard in semi-supervised semantic segmenta- tion, which is currently the adversarial training approach of Hung et al. (2018). We use two datasets in our experiments. CITYSCAPES consists urban scenery and has 2975 images in its training set. PASCAL VOC 2012 (Everingham et al. 2012) is more varied, but includes only 1464 training images, and thus we follow the lead of Hung et and augment using SEMANTIC BOUND- ARIES (Hariharan et al. 2011), resulting in 10582 training images. The practical benefits of semi- supervised learning are realised by reducing the required quantity supervised training data as much possible, hence we tested our approch using less supervised data than in Hung al. (2018) We note that we generated CowMix masks using a E [8, 32] due to the larger visual features present in CITYSCAPES and PASCAL VOC.
Our results are given in Tables and as mean intersection-oven ievetiioon-ve--uiiiin (mloU) percentages, where higher is better. The supervised baseline results between Hung et al. and Deepl aa implementa-
7

---

Under review as a conference paper at ICLR 2020
Labeled samples
100
372 (12.5%
744 (25
1488 (50 %
2975 (AlI)
Hung al. 2018
Adversarial training with COCO-pretraied Deeplah v2 network
Baseline
55.5%
59,9%
64.1% 1
66.4%
Sumi-supervised
58.8%
62.3%
65.7%
Della
3.5
2.4
1.6
Our results: Same COCCO-pretrained DeepLab v2 network
Baseline
44.34% 1.81
56.02% 0.80
60.90% 0.70
65.05% 0.7.
67.79% 0.3.2
ICT
55.30%
CutOut
54,25% :: 0.69
CutMix
57.64% 56 0.91
CowOut
58.09%
CowMix
49.01% 2.58
60.53% 0.29
64.10% 2082
66.51 %6 0:45
69.03% 0:27
Deelta
4.67
4.51
3.20
1.24
Our results: U-Net and randomly initialized decoder
Baseline
43.83 0.99
54.76% 0.50
60.35% 093
64.93% 0.32
68.34% 0.76
CowMix
51.98% 2.76
61.48% 1.84
64.85% 0
26.90%=0..3
69.57%
Delta
8.15
6.72
4.50
1.97
1.23
Table 2: Performance (mloU) CITYSCAPES validation set, each computed from 5 runs. The results for Hung et al. (2018) are from their paper.
# Labels
100
200
400
800
2646 (25%)
10582 (All)
Hung et al.
2018
Adversarial training with COCO-prctrainod DeepLab v2 network
Baseline
39.22%
46.57%
55.65%
62.54%
68.41
72.50%
Semi-sup.
38.82% 391
49.40% 1.00
60.29% 2.25
66.45% 0.60
71.27% 0.26
-Delta
0.40
2.83
4.64
3.91
2.86
Our results: Same COCO-pretrainee DeepLab v2 network
Baseline
41.17% 1.96
48.96%
1.70
58. 18.18%%-5 1.16
64.51 % 0.85
70.16% 0.35
73.32% 2.19
CowMix
52.10% 1.35
57.84% 1.55
64.18% 1
67.84%:00
70.99% 0.83
73.43% 0006
Delta
10.93
8.88
6.00
3.33
0.11
Our results: U-Net and randomly initialized decoder
Baseline
24.91%
2.13
2.13
33.66% + 1.64
41.80% + 1.15
49.18% 1.61
60.58% 0.77
65.97% 1.11
CowMix
41.00%
49.42% 1.74
54.20% 1.12
54.67%
63.48%
65.54% 4.99
Delta
16.09
15.76
12.4
5.49
2.90
-0.43
Table 3: Performance (mloU) on augmented PASCAL VOC validation set, each computed from runs.
tion are based on the same setup, provided by the authors, but differ slightly in practice due to the different choice of optimizer, etc.
We can see that using the same DeepLab setup, CowMix outperforms the adversarial training ap- proach in both datasets, with the exception of PASCAL when using large number of labeled sam- ples. The difference is particularly significant when only a small number of labeled samples is available, e.g., 52% vs 38% mloU with 100 labeled samples in PASCAL.
When using our U-Net architecture in which the decoder has not been pre-trained we were un- able successfully apply the adversarial approach. U-Net improves the performance of CowMix on CITYSCAPES and continues to yield significant gains relative to the supervised baseline in the case of PASCAL. While the performance on the more varied PASCAL dataset significantly benefits the pre- treitrainig of DeepLab v2, it also provides good test environment for semi-supervised learning algorithms when the decoder randomly initialized.
4.4 DISCUSSION
We attribute the fact that the adversarial approach (Hung al. (2018) ineffective with small num- bers of labeled samples to the requirements imposed by its discriminator network. A small set of ground truth labels lacks the variation necessary to effectively train the discriminator to distinguish
8

---

Under review as a conference paper at ICLR 2020
ground truth from predicted segmentation maps, preventing it from effectively guiding the segmen- tation network. In contrast, consistency regularization minimizes variation prediction over class preserving perturbation, effectively propagating labels between unlabelled samples. It therefore does not impose similar requirements on the size of the labeled data set.
5 CONCLUSIONS
We have shown that consistency regularization is viable solution for semi-supervised semantic segmentation, despite the lack low-density regions between classes. The proposed CowMix reg- ularization leads to very high-dimensional perturbations that enable state-of-the-art results, while being considerably easier to implement and use than the previous methods based on adversarial training. Even better and more varied perturbation strategies an obvious avenue for future work. Additionally, it could be fruitful to investigate when the CowMix approach could also be beneficial in the context of classification.
REFERENCES
Ben Athiwaratkun, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many con- sistent explanations of unlabeled data: Why you should average. In International Conference Learning Representations, 2019.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. CoRR, 0os/1511.00566 2015.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raf- fel. Mixmatch: A holistic approach to semi-supervised learning. CoRR, abs/1905.00229, 2019.
Gabriel J. Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes in video: A
high-definition ground truth database. Pattern Recognition Letters, 30(2):88-97, 2008
Olivier Chapelle and Alexander Zien. Semi-supervised classification by low density separation. In AISTATS, volume 2005, pp. 57-64, 2005.
Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. CoRR, abs/1412.7062, 2014.
Liang- Lhng-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE transactions pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Terrance Dririe and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. CoRR, his/1708.04552.. 2017.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results, 2012.
Geoff French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In International Conference on Learning Representations, 2018.
Bharath Hariharan, Pablo Arbeléez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. International Conference on Computer Vision, pp- 991-998, 1.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Wei-Chih Hung, Yi-Hsuan Tsai, Van-Ting Liou, Yen- Yu Lin, and Ming- Hsuan Yang. Adversarial learning for semi-supervised semantic segmentation. CoRR, abs/1802.072 802.079334 2018.

---

Under review as a conference paper at ICLR 2020
Xu Ji, João F. Henriques, and Andrea Vedaldi. Invariant information distillation for unsupervised image segmentation and clustering. CoRR, (oss/1807.0663,, 2018.
Tarun Kalluri, Girish Varma, Manmohan Chandraker, and CV Jawahar. Universal semi-supervised
semantic segmentation. CoRR, abs/1811.10323, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference on Learning Representations, 2017.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755, 2014.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431-3440, 2015.
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors teacher graphs for semi-supervised learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 8896-8905, 2018.
Takeru Miyato, Schi-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial train- ing: a regularization method for supervised and semi-supervised learning. arXiv preprint arXiv:1704.03976 2017.
Mohammadreza Mostajabi, Payman Yadollahpour, and Gregory Shakhnarovich. Feedforward mantic segmentation with zoom-out features. CoRR, abs/1412.0777, 2014.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic eval- uation of semi-super learning algorithms. In International Conference Learning Repre- sentations, 2018.
Christian Perone and Julien Cohen-Adad. Deep semi-supervised segmentation with weight- averaged consistency targets. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 12-19. Springer, 2018.
Boris Polyak and Anatoli Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838-88)) 1992.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed-
ical image segmentation. In International Conference on Medical Image Computing and Computer-Asiisted Intervention, pp. 234-241, 2015.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Mutual exclusivity loss for semi-supervised deep learning. In 23rd IEEE International Conference on Image Processing, ICIP 2016, 2016a.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems, pp. 1163-113 171, 2016b.
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A DIRT-t approach to unsupervised domain adaptation. In International Conference on Learning Representations, 2018.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. CoRR, bass/149..10 409.1556, 2014.
Sinisa Stekovic, Friedrich Fraundorfer, and Vincent Lepetit. S4-net: Geometry-consistent semi- supervised semantic segmentation. CORR, abs/1812.1001 2018
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight- averag-averaged consis- tency targets improve supi-super deep learning results. Advances in Neural Information Processing Systems, pp. 1194-1204, 2017.
10

---

Under review as a conference paper at ICLR 2020
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation con- sistency training for semi-supervised learning. CoRR, abs/1903.03822 00822, 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. CoRR, abs/1905.04899, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- risk minimization. In International Conference on Learning Representations, 2018.

---

Under review as a conference paper at ICLR 2020
Cityscapes training class Different class
Frequency (norma lizee lized)
200
400
600
800
1000
L distance between neighbouring patches
Figure 5; Histogram of L distance between pairs of patches of size 225 x 225 centred on neigh- boring pixels, split between those with the same class in blue and pairs that lie either side of class boundary in orange. Computed from all patches in the CITYSCAPES dataset whose central pixel has valid ground truth class.
A
2D TOY EXPERIMENTS
The neural networks used in our 2D toy experiments are simple classifiers in which samples are 2D 3, points ranging from -1 to 1. Our networks are multi-layer perceptrons consisting of 3 hidden layers of 512 units, each followed by a ReLU non-linearity. The final layer is 2-unit classification layer. We use the mean teacher (Tarvainen with binary cross-entropy as the consistency loss function, a consistency loss weight of 10 and confidence thresholding (French et al. 2018) with threshold of 0.97.
Valpola (017)) semi-supervised learning algorithm
The ground truth decision boundary was derived from a hand-drawn 512x512 pixel image.
The distance map shown in Figure [c) was computed using scipy scipy..o nd i mammaagee... morpho logy. di stance transfor consformn edt, with distances negated for regions assigned to class Each pixel the distance map therefore has a signed distance the ground truth class boundary. This distance map used to generate the countours seen as lines in Figure 2 and used to support the constrained consistency regularization experiment illustrated in Figure d).
The constrained consistency regularization experiment described Section 3.2 required that sam- ple should be perturbed to å such that they truth decision boundary. This was achieved by drawing isotropic perturbations from a normal dis- trubtion & mining the distances m(x) and m.) from x and á to the ground truth boundary (using a pre- computed distance map) and discarding the perturbation by masking consistency loss for x 0 if [m(f) m(x)] > 0.016 (0.016 & 4 pixels in the source image).
the same similar distance to the ground
+ h where h NO 0.117) (0.117 ss 30 pixels in the source image), deter-
B DISTRIBUTION OF PATCHES IN THE CITYSCAPES DATASET
we observed that if we extract overlapping neighboring patches from an image, where the patches are centered on neighboring pixels and compute the I,² distance between the raw pixel content neighboring patches, we see that the distances vary smoothly across the image. This is due to the fact that the distance between these patches can be efficiently computed by applying a box/uniform filter to the squared gradient image. This equivalence between the patch-to to-hhtch distance and uniform filter (that suppresses high frequencies and therefore the fine details necessary for low- densityy regions/ large inter- inter-patch distances to lie along class/objec/teexture boundaries in an image) suggests that these distance maps will be smooth for natural images in general.
In Section 3.1
12

---

Under review as a conference paper at ICLR 2020
Cityscapes training set distance ratio historgram
100
80
Frequency
60
40
20
0
0
7
Distance ratio n:b
Figure 6: Histogram the ratio of the distances 5. where the L pixel content distance from the query patch Q, drawn from image I, the nearest neighbour / most similar patch drawn from image I such that + j and 6 the distance from the query patch Q; to the patch P; centred on neighbouring pixel that has different ground truth class. Put simply, this histogram illustrates that the closest patch to Q; found in any other image in CITYSCAPES is typically at around 3x the distance as immediate neighbour that lies the other side of class boundary.
We will now use the CITYSCAPES dataset to experimentally test our observations concerning the distribution of image patches. Figure 5 illustrates the distributions of distances between pairs of patches centred on neighbouring pixels that have the same class in comparison to pairs of patches centred on pixels that lie either side a class boundary. The higher proportion of lower pixel content distances between neighboring patches that have the same class can be explained by the presence of large lightly textured regions such as sky that have few class boundaries and patches with similar pixel content.
Further analysis of the CITYSCAPES dataset revealed that in terms of pixel content distance, the nearest neighbours patch other patches centred neighboring and nearby pixels. We chose 1000 pairs of patches Q; and P; where i I 1000 from CITYSCAPES whose central pixels either side of class boundary. For each query patch Q: we computed the distance all other patches in the entire dataset. Figure [ shows that the closest neighbour extracted from another image is typically around 3x the distance from Q as the immediate neighbor P..
B. EFFICIENT COMPUTATION OF SLIDING WINDOW PATCH DISTANCES
Computing the distance from patch Q, every patch centred on each pixel in the complete CITYSCAPES dataset would be prohibitively expensive if performed in a brute-force fashion. There are 2975 training images CITYSCAPESwith each image having a resolution of 1024 x 512 pixels (downsampled by factor of 2 from the original resolution, used in our segmentation experi- ments). Centering a patch on each pixel results in 1,559,756, 755,7556,80 patches.
Given query patch Q and patch A where A is extracted from an image /, the squared distance ||A Q11 can be expanded to Q-Q+A-A A-A-AAA- 2(Q 100---2-Q-00 A). Q-Q is constant, A A can be computed for all patches in 7 in sliding window fashion using the box filter trick from Section 3.1 A where B is box-filter kernel, 2 is the element-win wide squared image and represents convolution. Q can be computed using correlation; given that the correlation kernel Q is typically large, we FFT-based convolution.
* T2
C
PSEUDOCODE
The mask generation function is given the form of Python/PyTorch code Listing ! and the training functions for semi-supervised CutOut and CutMix given in Listings and respectively.
13

---

Under review as a conference paper at ICLR 2020
The latter incorporate the mean teacher framework (Trvainen & Valpola 2017) and confidence thresholding (French et al. 2018) as used our experiments. In keeping with Yun et al. (2019); DeVries & Taylor (2017); p Verma et al. (2019), mask generator allows the proportion of pixels that come from each source image to vary. Although we found via experimentation that for CowMix the best proportion was always p 0.5, varying the proportion is beneficial for CowOut as detailed in Section 4.2
import numpy as np From www.y..niiagg.....m
import gaussian filter from scipy.special import ertinv
def generate_mixing_mas (img_size, sigma_min, sigma_max, p_min, p_max): I Randomly draw sigma from log-uniform distribution
sigma pp.pppeppp (np. rn (np. logissigaimminn np, np. (sigm____a))) np.randem. uniformmp min, p_max) # Randomly draw proportion p
mp.randem. normal (size=img_size 10 Ns gaussian_filtest (M, sigma)
Generate noise image
# Smooth with Gaussian
11 # Compute threshold
12 t erfinv (p+2 - 1) (2++0.5) Nt.std() + Ns.mean
return moosse_smootl Listing I: Python/NumPy code for cow mask generation
astypp (float) # Apply threshold and return
def cutout_less, (x, mask, teacher_meder student_modell.
"**x input image
shape (batch, chan, H, W), mask mixing mask of shape (batch, 1, H, W)"*"
# Apply teacher model and softmax to get per-pixel class probability y_t = seftmax (teacher_modee) moceeermmodeleeeeee (x), dim=1)
Apply student model to masked image
y_s softmax (student_mode) (x * mask), dim=1)
# Confidence thresholding factor
conficence = y_t.max (dim=1) Dimension 1 is class prob.
conf_fac (confidence. > 0.97) me
10 Consistency is squared error between student and teacher preds for masked pixels only, modulated with confidence factor
11 return (squared diff (ym_t. t, ym_s) * m) mean conf fac Listing 2: PyTorch code for CutOut CowOut for segmentation
def cutmix ses__osssxxaa xb, mask, teacher_model, student_model)) """xa, xb are input image pair each of shape (batch, chan, B, W), mask is mixing mask of shape (batch, 1, H, *)***
Apply teacher model and softmax to get per-pixel class probability ya_t soitmax (tetcher_ttaccr__mmdde (xa), dim=1) yb_t softmax (teacher_mode) (yb), dim-1)
# Mix images and teacher predictions mask ym ym t ya_t (1 mask) yb_t mask # Apply student model to mixed image ymss softmax (student bectdeet__mddlll___eee model (xm), dim=1) i Confidence thresholding facter
(1 mask)
12 confidence ym_t. max (dim=1) Dimension 1 class prob.
13 fac (confidence 0.97). m)ean()
14 # Consistency is squared error between student and teacher preds, modulated with confidence factor
15 return squared_dite _f (ym_t, ym_s) mean( conf_fac
Listing 3: PyTorch code for CutMix / CowMix for segmentation
D SEMANTIC SEGMENTATION EXPERIMENTS
D.1 DATA AUGMENTATION
Our data augmentation scheme that we used for standard augmentation based perturbation consists of an affine transformation composed of horizontal flips, translation in the range [-4,4 4] pixels,
14

---

Under review as a conference paper at ICLR 2020
ResNet-50 features
conv5_ 1/32 2048 chn
conv4
conv3
Foom Resnet
Legend
1/1 1024 chn
% res
% res
Comfle-C taye; lerne, ffters
512 chn
256 chn
Tronposed com,, upommpee
Decoder
Concatenate
layers
Conv3-192
Conv3-128
Conv3-96
Conv3-64
Conv3-48
Conv3-32
TConv-128 1/26
TConv-96
TConv-54
TConv-48
TConv-32
Conv1-C logits
res
Full
Figure 7: ResNet-50 based U-Net decoder architecture.
uniform scaling in the range [0.8, 1.25] and rotation in the range [-15", 15%). We also modify the brightness and contrast by adding value b (O,, 0.1) and scaling by factor ~ e ((0,,)(1.)),
D.1.1 AFFINE TRANSFORMATION BASED DATA AUGMENTATION FOR SEMANTIC SEGMENTATION
In Section 3.3 we stated that for the purpose of semantic segmentation, all geometric transformations need to be applied in reverse for the result image before computing the loss and that translation ef- fectively becomes no-op. This occurs in similar manner in both the supervised and unsupervised pathways.
Let us assume that the image æ is augmented by applyng the affine transformation matrix M; & transform(r, A). We apply our segmentation network fu to get our predictions Upred fo()). We note that the pixel-wise predictions Upred were generated from the transformed image and are therefore are not aligned with our ground truth annotations y. There are two ways in which we can compute the supervised loss Leei we can apply the transformation A the ground truths transform(Js (y, A) to align them with Upred and compute I.ee cross croos_entropy(1i entropyy@preed. (fpred, V) or we can apply the inverse transformation A to our predictions to bring them into alignment with the original image Ypred transform [preed A-1). Should the transformation A include translation, it will be countered by applying the equivalent translation to the pixel-wise ground truth in the first case or will be undone by applying A-1 to the predictions the second case.
We must adopt similar strategy when computing the unsupervised loss term Leons when using affine augmentation as perturbation. the image x transformed using two different affine transfor- mations A and B producing & and T. Our network fe will generate predictions Upred and Upred that are not aligned with one another. In order to compute the consistency loss Leons we must align the pixel-wise predictions with one another. This can be done by transforming Upred into the same frame (aansform(jpraee AB-1).
of reference as Upred by applying the transformation AB-1; Upred
We would like to note that these constraints limit the utility of translations used as an augmentation for semantic segmentation. Classification problems are typically translation invariant; applying translation will not change the target class. This not true for semantic segmentation which is translation variant. For example, consider applying translation to patch that centered pixel that very close to a class boundary that separates a car from a road an image. The translated patch may have different class. Computing either supervised or unsupervised loss without countering this translation will either cause the network to learn from erroneous training data.
D.2 U-NET NETWORK ARCIITECTURE
Our U-Net network shown in Figure 7 and Table
15

---

Under review as a conference paper at ICLR 2020
Description
Resolution channels
ResNet-50 layer conv5_x x 32 res, 2048 chn
X 2048
4
Conv x 3 x 192
x 192
T
TransposeConv
x 128
I 128
1 Concat with ResNet-50 layer conv4. x T6 res, 1024 chn Conv x 128
X 1152
128
TransposeCony x x 96
96
Concat with ResNet-50 layer conv3 x res, 512 chn Conv x x
608
96
TransposeCony x x 64
X 64
Concat with ResNet-50 layer conv2 x res, 256 chn Conv x 3 x 64
320
X 64
TransposeCony 4 x x 48
48
Conv x
48
TransposeCony x x 32
X 32
Conv x 3 x 32
X 32
Conv x x C
X C'
Table 4: ResNet-50 based U-Net decoder. C' is the number of target classes.
D.3 TRAINING DETAILS
D.3.1 EXPERIMENTS USING U-NET ARCHITECTURE
In keeping with Long al. 2013) use batch size of We freeze the batch normalization layers within the ResNet encoder, using the pre-trained running mean and variance rather than computing per-batch mean and variance during training. We use the Adam Kingma & Ba (205) optimization algorithm with a learning rate of 1 10-4 As per the mean teacher algorithm Tarvainen & Valpola (2177), after each iteration the weights wy of the teacher network are updated to be the exponential moving average of the weights W, of the student: wy ar
at)w,, where a₁ 0.99.
The Cityscapes images were downsampled to half resolution (1024 x 512) prior to use, as in Hung et al. (2018). When using Cityscapes we trained for 100,000 iterations using batch size of 1.
We tuned our approach and selected hyper-parameters using the CAMVID dataset due to its small size and fast run-ting timtime, after which we applied the same parammerrsettrss to Cityscapes. We did not run the augmentation based perturbation experiments on the Cityscapes dataset due to the long run-time involved.
D.3.2 EXPERIMENTS USING DEEPLAB V2 ARCHITECTURE
We found that we had to decrease the learning rate to x 10-5 to get good performance with DeepLab v2. We note that the PyTorch implementation of DeepLab v3 worked best with 1 x 10-4. We believe that this is because the U-Net and DeepLab v3 networks accept input images with zero- mean unit-variance, where Deeplab v2 accepts images whose values in the range 0 to 255 with only mean subtraction.
For the PASCAL VOC experiments, we extracted 321 321 random crops and used batch size of 10, in keeping with Hung el al. (2018). For the CITYSCAPES experiments we used full image crops and batch size of 2.
D.3.3 ABLATION
The augmentation based perturbation experiments performed on the CAMVID dataset were trained for 50 epochs. The CutMix and CowOut experiments were trained for 100.
D.3.4 CONFIDENCE THRESHOLDING
French et al. (2018) apply confidence thresholding, in which they mask consistency loss to 0 for samples whose confidence predicted by the teacher network is below threshold of 0.968. In the
16

---

Under review as a conference paper at ICLR 2020
Chycnpeet val detion oe care au (8)
Daidin
20
66
56-
72
50-
16
65-
66-
15.5
Tamrsin
Penson
Rair
70
46
20
50
1048
Mekridde
Moydee
Figure 8: Visualization of semi-supervised segmentation on CITYSCAPES, 100 supervised samples
context of segmentation, we found that this masks pixels close to class boundaries they usually have a low confidence. These regions are often large enough encompass small objects, preventing learning and degrading performance. Instead we modulate the consistency loss with the proportion of pixels whose confidence is above the threshold. This values grows throughout training, taking the place of the sigmoidal ramp-up used in Laine & Aila (2017); Tarvainen & Valpola 2017).
D.4 DETAILED PERFORMANCE TABLES
The detailed per-class performance on the CITYSCAPES dataset are presented in Table and visual- ized in Figure ad the augmented PASCAL VOC dataset are presented 6 visualized
in Figure
D.5 HYPER PAEERR-RRAMEERRR CHOICE
We explored the hyper-parameter parameters for CowMix using the CAMVID dataset. We present our results in the form of bar plots. Black and green bars at left and right end are the performance of the baseline and fully supervised setups respectively while blue bars show response to rrper-parameter values. The performance is strong when consistency weight has value in the range of to 30, with 10 being the optimum choice, as seen in Figure 10
The EMA value used update the teacher network had little effect, in Figure
The a used for Gaussian smoothing yielded good results when between and 8, but was best when drawn from log 2/(4 16), seen in Figure 12
17

---

Under review as a conference paper at ICLR 2020
Pascal AUG validation set dess 1991
Lee Jardine
6
20
Taciomann
21
23
28-
42-
23
Figure 9: Visualization of supervupe segmentation on augmented PASCAL VOC dataset, 100 supervised samples
CamVic val. miU (%)
62
55-
02
200
200
1080
Conextency
Figure 10: Effect of consistency weight hyper- sypee-parramttr. In each bar, the central line is the mean, with the extents of the bar placed standard deviation each side.
18

---

Under review as a conference paper at ICLR 2020
CamVid- - vil. micU (%)
72-
55-
Bnsinn
Eaily sup
Figure 11: Effect of mean teacher EMA hyper-parameter.
CaniVid - val. miu (%)
65-
55-
1-16
32
Fubl
Smoothing oen and Rue
Figure 12: Effect CowMix o min and O max hyper-parameters. A single value indicates that a was fixed, while range (e..g 4-16) indicates log 16).
19

---

Under review as a conference paper at ICLR 2020
OVERALL
Road
Sidewalk
Building
DeepLab Baseline
44.34% + 1.61
94.21% 1221%11 0.18
60.29% 1.09
83.06% 1: 0.09
DeepLab CowMix
49.01% 2.58
95.69% 1.2.4
68.62%
1.11
85.47% 0.30
DeepLab Full
07.79%+0.33
97.13%
77.74%
89.29% 0.07
U-net Baseline
+
95.12% 0.25
65.94%
1.74
83.78% = 0.34
U-net CowMix
51.98% 2.76
96.61% % 0.20
74.59% + 1.57
86.81% % = 0.27
U-net Full
68.34% 0.76
97.57% 0.02
80.26% 0.16
90.54% 1: 0.12
Wall
Fence
Pole
Traffic light
DaepLab Baseline
13.51% 3.39
14.86% 2.63
33.31% 0.49
22.20% 1: 5.68
DeepLab CowMix
15.85% 4.84
19.82% 4.89
36.66% 0.79
28. 14% 7.10
DeepLab Full
46.12% 1.51
45.09% 0.88
43,91% 0.14
49.49% = 0.73
U-net Baseline
10.65% 2.28
1.67 12.84%
1.09 46.25%
39.59%
26.29% 4.40
U-net CowMix
13.91% 5.01
18.87%
0.50
36.94% 1
U-net Full
44.22% 2.36
47.52% 0.69
56.29% 0.4.4
58.68% + 1.28
Traffic sign
Vegetation
Terrain
Sky
DecpLab Baseline
39.65% + 2.08
84,42% 0,440
37.48% + 1.94
87.30% = 0.75
DeepLab CowMix
45.30%
86.60% 0.200
40.82%
3.44
90.88% 0.24
DeepLab Full
62.67%
89.42%
57.59%
0.79
92.80% 1
U-net Baseline
40.83% 3.27
86.54% 0.27
41.02%
3.02
87.47% 1
U-net CowMix
55.89% 1: 3.44
88.73% 0.05
45.08%
1 4.4.
91.12% 1 1.31
U-net Full
70.43% + 0.55
90.96%
0.11 0.12
59.19% 0.80
94.05% = 0.18
Person 55.76% = 0.93
Rider
Car
Truck
DeepLab Baseline
17.59%
3,70
83.70% 1.14
17.46% - 23.15% 1 13.60
DeepLab CowMix
61.39% 11 0.39
23.38%
3.80
86.33% 1.50
DeepLab Full
69.89% 1: 0.24
47.37%
0.76
91.52%
0.0.8
67.77% 1 1.65
U-net Baseline
57.75% 1. 1.81
14.69%
2.56
85.13% 1 0.49
5.86% 1: 2.83
U-net CowMix
66.86% + 0.90
26.85%
4.78
88.42% 1.57
17.77% = 10.13
n-net Full
75.18%
50.23% 0.96
92.23% 0.27
54.06% 2.89
Bus
Train
Motorcycle
Bicycle
DeepLab Baseline
24.11% 1: 12.62
14.49% 1: 7.71
7.40% 5.6.63
51.69% 1: 1.24
DeepLab CowMix
24.52% 1: 20.60
28.07%
14.68
13.86% 1
56.69% 1 1.10
DeepLab Full
78.46% 0.86
65.02%
4.88
50.41%
1.75
66.36% 0.13
U-net Baseline
15.42% 5.23
7.65% 3.57
6.31% 3.76
49.90% = 62.63% 0.88
U-net CowMix
25.80% + 16.62
20.02% 9.65
24.43% + 10.97
U-net Full
68.56% 1: 3.08
51.44% 6,32
46.26% J: 1.31
70.74% 1: 0.41
Table Per-class performance on CITYSCAPES dataset, 100 supervised samples
20

---

Under review as a conference paper at ICLR 2020
OVERALL
Background
Person
Bird
Adv. Bascline
39.22%
87.00% 0.89
55.00% 9.88
1: 24.40%
Adv. Semi-sup.
38.82% 3.91
88.00% + 0.63
66.00% 9.770
25.60%
12.22
Adv. Full
72.50% 0.27
93.00% = 0.00
87.00%
0.66
39.80% = 0.40
DeepLab Baseline
41.17% 1.96
89.18% 0.51
57.57% 7.40
31.07% = 2.66
DeepLab CowMix
52.10% 1.35
91.27% 0.43
75.01% 4.77
34.73%
DeepLab Full
73.32% 0.19
93.39%
0.03
86.13% 0.39
39.36% + 0.39
U-net Baseline
24.91%
86.85% 0.57
37.48% 4
18.15% 4.85
U-net CowMix
41.00% 2,70
89.99% 0.32
67 67.28% 5:266
32.59% = 4.92
U-net Full
65.97% 1.11
92.50% 0.19
80.44% 1.45
37.74% = 1.01
Cat
Cow
Dog
Horse
Adv. Baseline
44.80% + 18.54
27.40% 8.64
40.80% 16.27
53.40% 27.22
Adv. Semi-sup.
58.80% 26.95
7.80% 8.47
25.20% +00%:-27 21.17
59.80% 30.72
Adv. Full
85.20% + 0.98
60.40% 1.62
78. 78.40% 0.49
90.00% - 0.63
DeepLab Baseline
53.06% + 12.50
29.79% 3.06
40.19% 11.65
52.14% 26.38
DeepLab CowMix
75.49% 2. 2.11
51.59% 6.62
58.94% 2.76
53.47% + 29.03
DeepLab Full
85.84% 0.91
0.52 63.73%
76.76% 0.44
91.58% 1 0.17
U-net Baseline
26.02% 1: 10.70
11.06% 0.69
17.34% J: 10.69
28.56% 1 15.70
U-net CowMix
58.26% 7.16
32.37%
6.23
44.96% 4.58
47.54%
1 24.69
U-net Full
77.76%
60.25% 1.48
64.85% 1.60
80.54%
2.36
Sheep
Aeroplane
Bicycle
Boat
Adv. Baseline
60.00% +: 5.76
62.40% 4.45
10.60% 3.01
26.80% : 14.23
Adv. Semi-sup.
73.80% +: 1.72
56.40% 22.29
1.20% 1.17
18.00% : 20.94
Adv. Full
84.40% +: 0.49
87.80% 0.40
33.40% 1: 0.49
81.20% 1 0.40
DeepLab Baseline
57.98% 7.31
60.11%
14.76% 2.50
24.58% = 15.00
DeepLab CowMix
72.81% 8.10
74.37%
19.83% 2.28
28. 28.86% 17.15
DeepLab Full
84.97% 0.46
87.92% 0.42
34.19% + 0.94
81.80% = 0.78
U-net Baseline
28.57% 6.63
40.27% 1: 6.37
9.18% 3.09
7.68% 1 5.95
U-net CowMix
60.13% 1: 10.67
49.20% 19.50
20.10%
2.95
13.99% 1 13.67
U-net Full
76.83% 3.57
83.25% 0.81
30.99% 2.60
58.35% 1:
Bus
Car 58.40% 1: 16.80
Motorbike
Train
Adv. Baseline
9.60% + 10.59
53 80% = 6.91
37.40% 5.57
46.60% = 5.99
Adv. Semi-sup.
1.60% 3.20
38.60% 26.44.4
68.40% 2.94
Adv. Full
51.60% 1. 1.20
81.00% 0.63
78.20% 0.75
79.40% 1 0.49
DeepLab Baseline
15.86% 10.62
49.61% %
36.08%
52.30% 1:
DeepLab CowMix
25.41% + 14.08
67.77%
5.12
52.66% 10.88
67.04% 3.32
DeepLab Full
52.38% 1.46
81.93% 0.46
80.22% + 0.61
79.96% 0.46
U-net Baseline
4.00% + 4.00
3.04 33.70%
13.26% 6.16
24.99% = 8.00
U-net CowMix
14.15% + 8.28
47.94%
27.13% 8.12
57.67% 4.95 73.71%
U-net Full
36.78% + 11.81
1.66 73.91%
64.44% 3.96
Bottle
Chair
Dining table
Potted plant
Adv. Baseline
68 28.80% 0.75
4.20% = 5.38
29.60% 24.42
8.00% = 7.67
Adv. Semi-sup.
79.40% + 0.80
0.60% = 0.80
33.00% 28.93
1.00% = 1.10
Adv. Full
83.00% 0.00
57.40% 0.80
79.80% 0.75
43.60%
DeepLab Baseline
70.93% 2.15
9.44% 9.85
28.80% 23.66
12.51%
DcepLab CowMix
78.10% 0.339
14.53% 17.63
32.57% 28.5.
16.88% + 15.62
DeepLab ab Full
84.45% 0.11
56.66%
1.03
82.06% 0.0.6
46,21 9. 1.11
U-net Baseline
61.96% 1.97 1.97
7.75% 8.12
13,44%
11.28
5.69% + 5.38
U-net CowMix
75.26% 2.51
12.34% 10.32
17.43% 14.52
12.55% + 12.38
U-net Full
82.71% 0.54
53.68% 5.34
70.51% 2:2.05
40.19% + 2.09
Sofa
TVmonitor
Adv. Bascline
34.40% 4: 28.58
39.80% 8.13
Adv. Semi-sup.
35.00% 29.45
18.20% 12.51
Adv. Full
82.20% 0.75
64.60% 2.50
DeepLab Baseline
34.95% 28.67
43.72% 9.62
Lab CowMix
39.51% 32.95
63.18% 1.10
Lab Full
81.98% 100%%+0. 0.13
68. 155%++. 1.80
U-net Baseline
17.64% 16.75
29.58%
7.32
U-net CowMix
33.16% 27.70
46.93% 6.07
U-net Full
75.06% 1: 1.67
70.86% 1.11
Table 6: Per-class performance on augmented PASCAL VOC dataset, 100 supervised samples
21


Reviews:
[{"review_id": "SJlKzMliKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work analyzes the consistency regularization in semi-supervised semantic segmentation. Based on the results on a toy dataset, this work proposes a novel regularization for semi-supervised semantic segmentation, which is named CowMix.\n\nPros:\n-- The proposed CoxMix is easy to understand and implement.\n-- The experimental results seem to benefit from this proposed CoxMix at a first glance.\n\nCons:\nThe writing is not clear. Sometimes I have to make a ``guess\" about the technical details. For example:\n-- Other than L_{cons}, is there any other Loss term utilized in this work? Based on Figure 3, it seems only L_{cons} is utilized. If so, is it a waste not to use the label training data (although very few) to calculate a cross-entropy loss?\n\n-- It seems the experimental setting in this submission follows the settings in Hung, 2018. However, for the experiment on VOC 2012 validation set, Hung tested their method on 1/8 1/4 1/2 of labeled data (Table 1). While in this submission, Table 3 shows the results on label data of 100, 200, 400, 800, 2646(25%). The split ratios seem different from Hung's work, which confuses me.\n\n-- \"Note that in context of semantic segmentation, all geometric transformations need to be applied in reverse for the result image before computing the loss (Ji et al., 2018). As such, translation turns into a no-op, unlike in classification tasks where it remains a useful perturbation.\" \n    Is there any experimental result to support this claim?\n\n-- It is a little hard for me to fully understand Figure 2. For example, how to get 2. (c)? What is the meaning of the word \"gap\" here?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1eLF2upFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provided first provided analysis for the problem of semantic segmentation. Through a few simple example, the authors suggested that the cluster assumption doesn’t hold for semantic segmentation. The paper also illustrated how to perturb the training examples so that consistency regularization still works for semantic segmentation. \nThe paper also introduce a perturbation method that can achieve high dimensional perturbation, which achieve solid experimental results.\n\nThe analysis part seems interesting and innovative to me. But it is very qualitative and I'm not fully convinced that the analysis on 2d example can actually carry over to high dimensional spaces for images. I also don't quite see the connection between the toy example and the proposed perturbation method. For example, why the proposed perturbation method has the property of \"the probability of a perturbation crossing the true class boundary must be very small compared to the amount of exploration in other dimensions\"?\n\nThe proposed algorithm is an extension of the existing cutout and cut mix. The way to generate new mask is a very smart design to me. This should be the most important contribution of the paper.\n\nThe writing of the paper is very clear and easy to follow. The experimental results look very convincing overall and proposed algorithm does show very promising results. \n\nTo sum up, the paper is an ok paper from the practical perspective, but the analysis in the paper wasn't strong enough to me.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklLvzGCFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n# Summary\n\nThis paper proposes a method for semi-supervised semantic segmentation. \nThe authors tackle this problem through consistency regularization, a successful technique in image classification, that encourages the network to give consistent predictions on unlabeled samples which are perturbed in multiple ways. The authors argue that the cluster assumption (to which effectiveness of consistency regularization has been partially attributed) does not hold in semantic segmentation. Thus, in order to enable class boundaries to become low-density regions and then better guide contrastive regularization, the authors argue that a stronger perturbation must be inserted. To this effect they first propose looking at CutOut and CutMix types of methods. They improved upon them by putting forward a variant of CutMix, coined CowMix, with more degrees of freedom and using flexible masks instead of rectangular ones. CowMix is evaluated on the Cityscapes and PascalVOC 2012 datasets in the semi-supervised regime and showing encouraging results.\n\n\n# Rating\nI find the paper and the advanced ideas of interest for the community and I consider they are novel. I'm currently on the fence between Weak Accept and Weak Reject, mostly due to incomplete evaluations and support for claims made in the introduction regarding the infeasibility of contrastive regularization methods for semantic segmentation. I would be happy to upgrade my rating if authors addressed these concerns.\n\n\n# Strong points\n- The paper is well written and mostly clear with a good coverage and positioning w.r.t. related work. The authors illustrate well the reasoning and the choices they have made. The author provide plenty of ablation studies (e.g., per class statistics) and implementation details, improving significantly the reproducibility of the contribution.\n- The flexible masking technique that is advanced here is novel and experimentally seems effective.\n- This work is among the few that address semi-supervised semantic segmentation in a non-adversarial manner, so I would give it some novelty credit.\n- I appreciate the evaluation protocol of averaging across multiple runs.\n\n# Weak points\n\n## Unclear aspects\n- The authors argue that consistency regularization has had little success so far in semantic segmentation problems since low density regions in input data do not align well with class boundaries. It would be useful to provide a reference to this claim or at least validate it experimentally on a large dataset.\n\n- In Figure 1, it is not clear on which features where the distances between patches computed? Is it on raw pixels or intermediate feature maps from a CNN? \nIf the distances are made over raw pixels, I find it difficult to make the connection between distances in the pixel space and distances in the class space.\nAre the neighbor patches overlapping with the central/query patch?\n\n## Experiments\n- The authors compare against other methods on the CamVid dataset. CamVid is a small and relatively limited dataset (~367 images for training from the streets of Cambridge). I'm worried that this dataset might not be enough to conclude and emphasize the benefits of this method over other semi-supervised techniques. For instance CowOut does not seem do be above CutOut, while CutMix has convergence problems and low scores. \nThe other experiments on Cityscapes and Pascal VOC are certainly interesting, but the method is compared only against Hung et al. which a different family of methods and the subset baseline (which is useful but not enough). I think this work would benefit from an additional baseline in the style of contrastive regularization methods, e.g. ICT, and eventually CutOut, to support the initial arguments regarding the limitations of these methods in semantic segmentation and respectively the effectiveness of the flexible masks over the rectangular ones in this setup.\n\n\n# Suggestions for improving the paper:\n1) It would be useful to include other semi-supervised baselines, e.g. ICT, and the baseline perturbation CutMix on larger experiments, in order to better emphasize the contributions of this work.\n\n2) Did the authors try the flexible masking on image classification? How is it expected to perform over ICT, MixUp or MixMatch?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for semi-supervised semantic segmentation through consistency (with respect to various perturbations) regularization. While the reviewers believe that this paper contains interesting ideas and that it has been substantially improved from its original form, it is not yet ready for acceptance to ICLR-2020. With a little bit of polish, this paper is likely to be accepted at another venue.
Decision: Reject