Title: SEMI-SUPERVISED SEMANTIC SEGMENTATION NEEDS STRONG, HIGH-DIMENSIONAL PERTURBATIONS

Abstract: Consistency regularization describes a class of approaches that have yielded ground breaking results in semi-supervised classification problems. Prior work has established the cluster assumption -under which the data distribution consists of uniform class clusters of samples separated by low density regions -as key to its success. We analyze the problem of semantic segmentation and find that the data distribution does not exhibit low density regions separating classes and offer this as an explanation for why semi-supervised segmentation is a challenging problem. We then identify the conditions that allow consistency regularization to work even without such low-density regions. This allows us to generalize the recently proposed CutMix augmentation technique to a powerful masked variant, CowMix, leading to a successful application of consistency regularization in the semi-supervised semantic segmentation setting and reaching state-of-the-art results in several standard datasets.

1. INTRODUCTION:

Semi-supervised learning offers the tantalizing promise of training a machine learning model with limited amounts of labelled training data and large quantities of unlabelled data. These situations often arise in practical computer vision problems where large quantities of images are readily available and generating ground truth labels acts as a bottleneck due to the cost and labour required.

Consistency regularization (Sajjadi et al., 2016b; Laine & Aila, 2017; Miyato et al., 2017; Oliver et al., 2018) describes a class of semi-supervised learning algorithms that have yielded state-ofthe-art results in semi-supervised classification, while being conceptually simple and often easy to implement. The key idea is to encourage the network to give consistent predictions for unlabeled inputs that are perturbed in various ways.

The effectiveness of consistency regularization is often attributed to the smoothness assumption (Miyato et al., 2017; Luo et al., 2018) or cluster assumption (Chapelle & Zien, 2005; Sajjadi et al., 2016a; Shu et al., 2018; Verma et al., 2019) . The smoothness assumption states that samples close to each other are likely to have the same label. The cluster assumption -a special case of the smoothness assumption -states that decision surfaces should lie in low density regions, not crossing high density regions. This typically holds in classification tasks, where most successes of consistency regularization have been reported so far.

On a high level, semantic segmentation is classification, where each pixel is classified based on its neighborhood. It is therefore intriguing that consistency regularization has not been clearly beneficial in this context. We make the observation that the distance between patches centered on neighboring pixels varies smoothly even when the class of the center pixel changes, and thus there are no low-density regions on class boundaries. This alarming observation leads us to investigate the conditions that can allow consistency regularization to operate even in these conditions. Our key insight is that this is indeed possible, and that previous attempts have yielded little success primarily because the perturbations they used were not strong enough, and especially not high-dimensional enough.

We show that a flexibly masked variant of CutMix (Yun et al., 2019) , which we call CowMix based on the mask appearance, does realize significant gains in semi-supervised semantic segmentation. This result clearly signposts a direction where further improvements are likely to be available.

2. BACKGROUND:

Our work relates to prior art in three areas: recent regularization techniques for classification, semisupervised classification with a focus on consistency regularization, and semantic segmentation.

2.1. MIXUP, CUTOUT, AND CUTMIX:

The MixUp regularizer of Zhang et al. (2018) improves the performance of supervised image, speech and tabular data classifiers by using interpolated samples during training. The inputs and target labels of two randomly chosen examples are blended using the same randomly chosen factor.

The Cutout regularizer of DeVries & Taylor (2017) augments an image by masking a rectangular region to zero. The recently proposed CutMix regularizer of Yun et al. (2019) combines aspects of MixUp and Cutout, cutting a rectangular region from B and pasting it over A. MixUp, Cutout, and CutMix improve supervised classification performance, with CutMix outperforming the other two.

2.2. SEMI-SUPERVISED CLASSIFICATION:

The Π-model of Laine & Aila (2017) passes each unlabeled sample through a classifier twice, applying two realizations of a stochastic perturbation process, and minimizes the difference between the resulting class probability predictions. Their temporal model maintains a per-sample moving average of historical predictions and encourages subsequent predictions to be consistent with the average. Sajjadi et al. (2016b) similarly encourage consistency between the current and historical predictions. Miyato et al. (2017) improve the results by replacing the stochastic perturbations with adversarial directions, thus focusing on perturbations that are closer to the decision boundaries.

The mean teacher model of Tarvainen & Valpola (2017) encourages consistency between predictions of a student network and a teacher network. The teacher's weights are an exponential moving average (Polyak & Juditsky, 1992) of those of the student, leading to a faster convergence and improved results. French et al. (2018) adapt the mean teacher approach for domain adaptation.

Interpolation consistency training (ICT) (Verma et al., 2019) and MixMatch (Berthelot et al., 2019) both combine MixUp (Zhang et al., 2018) with consistency regularization. ICT uses the mean teacher model and applies MixUp to unsupervised samples, blending input images along with teacher class predictions to produce a blended input and target to train the student. MixMatch stochastically perturbs each sample multiple times and averages the predictions to produce unsupervised targets; MixUp is applied to labeled as well as unlabeled samples. Long et al. (2015) finetune a pre-trained VGG-16 (Simonyan & Zisserman, 2014) image classifier to produce a dense set of predictions for overlapping input windows. They effectively transform the image classifier into a fully convolutional network that can be used to segment input images of arbitrary size. Various methods were proposed for increasing the localization accuracy of the results (Long et al., 2015; Chen et al., 2014; Mostajabi et al., 2014) until the introduction of encoderdecoder networks (Badrinarayanan et al., 2015; Ronneberger et al., 2015) led to a solution where the output resolution natively matches the input. In these recent methods the encoder downsamples the input progressively, similarly to image classifiers, the decoder performs progressive upsampling, and skip connections route data between the matching resolutions of the two networks, improving the ability to accurately segment fine details.

2.3. SEMANTIC SEGMENTATION:

A number of approaches for semi-supervised semantic segmentation use additional data. Kalluri et al. (2018) use data from two datasets from different domains, maximizing the similarity between per-class embeddings from each dataset. Stekovic et al. (2018) use depth images and enforced geometric constraints between multiple views of a 3D scene. Relatively few approaches operate in a strictly semi-supervised setting. Hung et al. (2018) employ adversarial learning, using a discriminator network that distinguishes real from predicted segmentation maps to guide learning. Perone & Cohen-Adad (2018) apply consistency regularization to a MRI volume dataset and their method is the only successful application of consistency regularization to segmentation that we are aware of. 

3. CONSISTENCY REGULARIZATION FOR SEMANTIC SEGMENTATION:

Consistency regularization adds a consistency loss term L cons to the loss that is minimized during training (Oliver et al., 2018) . In a classification task, L cons measures a distance d(•, •) between the predictions resulting from applying a neural network f θ to an unsupervised sample x and a perturbed version x of the same sample, i.e., L cons = d(f θ (x), f θ (x)). The perturbation used to generate x depends on the variant of consistency regularization used. A variety of distance measures d(•, •) have been used, e.g., squared distance (Laine & Aila, 2017) or cross-entropy (Miyato et al., 2017) . Athiwaratkun et al. (2019) analyze a simplified version of the Π-model (Laine & Aila, 2017) in which perturbation consists of additive Gaussian noise so that x = x + h, where h ∼ N (0, I) and d(•, •) is a squared Euclidean distance. For small constant , the expected value of the consistency loss term L cons is approximately proportional to the square of the Frobenius norm of the Jacobian J f θ (x) of the networks outputs with respect to its inputs:

EQUATION

Thus, minimizing L cons directly flattens the decision function in the vicinity of unsupervised samples. This illustrates clearly the mechanism by which consistency regularization encourages the network to move the decision boundary -and its surrounding region of high gradient -into regions of low sample density.

3.1. WHY SEMI-SUPERVISED SEMANTIC SEGMENTATION IS CHALLENGING:

We attribute the infrequent success of consistency regularization in semantic segmentation problems to the observation that low density regions in input data do not align well with class boundaries. As illustrated in Figure 1 , the cluster assumption is clearly violated: how much the raw pixel content of the receptive field of one pixel differ from the contents of the receptive field of a neighboring pixel has effectively no correlation with whether the patches' center pixels belong to the same class or not. For the cluster assumption to hold, we would require the distances between the contents of receptive fields to be large between classes and small within classes, which is not the case here.

The lack of variation in the patchwise distances is easy to explain from a signal processing perspective. With patch shape B, the L 2 distance between the pixel content of two overlapping patches centered at, say, horizontally neighboring pixels can be written as B * (∆ x x) 2 , where * denotes convolution and ∆ x x is the horizontal gradient of the input image x. The squared gradient image is thus low-pass filtered by a B-shaped box filter, which suppresses any fine details and leads to smoothly varying sample density across the image. We test this experimentally in Appendix B.

Our decision to compare the raw pixel content of image patches is motivated by the fact that the perturbations used to generate x from x in prior work operate in raw pixel (input) space. The success of virtual adversarial training (Miyato et al., 2017) in semi-supervised classification -in which x is an adversarial example chosen to maximise the difference in prediction from that of x -illustrates the significance of the input space distribution in semi-supervised learning.

3.2. CONSISTENCY REGULARIZATION WITHOUT CLUSTER ASSUMPTION:

Let us analyze a simple toy example of learning to classify 2D points in a semi-supervised fashion. Figure 2a illustrates the setup where cluster assumption holds and there is a gap between the unsupervised samples belonging to the two different classes. The perturbation used for the consistency loss is a simple Gaussian nudge to both coordinates, and as expected, the learned decision boundary settles neatly between the two clusters.

In Figure 2b , the cluster assumption is violated and there are no density differences in the set of unsupervised samples. In this case, the consistency loss does more harm than good -even though it successfully flattens the neighborhood of the decision function, it does so also across the true class boundary. In order for the consistency regularization to be a net win, it would have to perturb the samples as much as possible, but at the same time avoid crossing the true class boundary.

In Figure 2c , we plot the contours of the distance to the true class boundary, suggesting a potentially better mechanism for perturbation. Indeed, when perturbations are done only along these contours, the probability of crossing the true class boundary is negligible compared to the regularization potential in the remaining dimension. Figure 2d shows that the resulting learned decision boundary aligns well with the true class boundary.

Low-density regions provide an effective signal that guides consistency regularization by providing areas into which a decision boundary can settle. This illustrative toy example demonstrates an alternative mechanism; the orientation of the decision boundary can be constrained to lie parallel to the directions of perturbation. We therefore argue that consistency regularization can be successful even when the cluster assumption is violated, if the following guidelines are observed: 1) the perturbations must be varied in order to cover as much of the input space in the same class as possible, 2) they must be high-dimensional in order to sufficiently constrain the orientation of the decision boundary in the high-dimensional space of natural imagery, 3) the probability of a perturbation crossing the true class boundary must be very small compared to the amount of exploration in other dimensions, and 4) the perturbed inputs should be plausible, i.e., they should not be grossly outside the manifold of real inputs. 

3.3. CUTOUT AND CUTMIX FOR SEMANTIC SEGMENTATION:

If we consider the classical augmentation-like perturbations such as translation, rotation, scaling, and brightness/contrast changes, it is evident that these have a low chance of confusing the output class (Athiwaratkun et al., 2019) but they also provide very little variation. Note that in context of semantic segmentation, all geometric transformations need to be applied in reverse for the result image before computing the loss (Ji et al., 2018) . As such, translation turns into a no-op, unlike in classification tasks where it remains a useful perturbation (see Appendix D.1.1 for more details).

Adding noise is another questionable perturbation strategy -although high-dimensional, such perturbations are very unlikely to lie on the manifold of natural images.

Out of previously proposed perturbation methods for consistency regularization, we identify CutOut and CutMix as promising candidates for semantic segmentation as they provide a large variety of possible outputs and are class preserving. Both approaches use a mask M with a randomly chosen rectangular region. Our masks have M = 1 inside the rectangle, and M = 0 otherwise.

CutOut. To apply CutOut in a semantic segmentation task, we mask the input pixels with M and disregard the consistency loss for pixels masked to 0 by M . Using square distance as the metric, we have

L cons = ||M (f θ (M x) -f θ (x))|| 2

, where denotes an elementwise product.

CutMix. CutMix requires two input images that we shall denote x a and x b that we mix with the mask M . Following ICT (Verma et al. (2019) ) we run both input images as well as the mixed variant through the network. The consistency loss is then taken between the segmentation of the mixed image and the mix between the segmented input images. To simplify the notation, let us define function mix (a, b, M ) = (1 -M ) a + M b that selects the output pixel based on mask M . We can now write the consistency loss for segmentation CutMix as

EQUATION

)

So far we have assumed that both original and perturbed images are segmented using the same network f θ . In this sense, our approach is similar to the Π-network Laine & Aila (2017), although mean teacher (Tarvainen & Valpola, 2017) is known to outperform it in image classification tasks.

Our preliminary tests indicated that the same is true for semantic segmentation, and therefore all the experiments in this paper use the mean teacher framework. Specifically, in CutOut, we segment the unperturbed image x using the teacher network and the perturbed image M x using the student network. In CutMix -following Verma et al. ( 2019) -we similarly segment the original images x a and x b using the teacher network, and the mixed image using the student network. The computation is illustrated in Figure 3 .

To analyze how much variation these perturbations provide, we note that the original CutOut always masks out a fixed-size square at a random position, so the resulting mask has only two degrees In semantic segmentation, we deviate from the original methods in that we choose all four parameters of the rectangle at random in order to obtain as much variation as possible.

σ = 2 σ = 4 σ = 8 σ = 16 σ = 32 σ = 64

3.4. COWOUT AND COWMIX:

The use of a rectangular mask restricts the dimensionality of the perturbations that CutOut and CutMix can produce. Intuitively, a more complex mask that has more degrees of freedom should provide better exploration of the plausible input space. We propose combining the semantic CutOut and CutMix regularizers introduced above with a novel mask generation method, giving rise to two regularization methods that we dub CowOut and CowMix due to the Friesian cow -like texture of the masks.

To generate a mask with a given proportion p of pixels having M = 0, we start by sampling a Gaussian noise image N (0, I), convolve it with a Gaussian smoothing kernel, and threshold the result at τ = erf -1 (2p -1) √ 2S σ + S µ , where S µ and S σ are the mean and standard deviation of the smoothed noise, respectively. The standard deviation σ of the Gaussian smoothing kernel determines the average size of features in the mask and is drawn from a log-uniform distribution: σ ∼ log U(σ min , σ max ). Figure 4 shows example masks with varying values of σ.

To estimate how much variation these masks contain, we first observe that if no smoothing were performed before thresholding, we would obtain a binary per-pixel Bernoulli mask. This has an exponential amount of variation with respect to the area of the receptive field: a receptive field with area A has 2 A possible masks. Applying the smoothing filter has a similar effect as zooming the mask -if the smoothing constant σ is doubled, we can expect to obtain roughly 4× fewer "bits" of variation (see, e.g., Figure 4 , σ = 8 vs. σ = 16). The suitable value for σ is thus an empirical tradeoff between the amount of variation, i.e., the strength of the resulting perturbation, and plausibility of the masked/mixed image. If too little filtering is performed, we obtain a huge amount of variation but the perturbed images are unrealistic, and with too much filtering the images are very realistic but the amount of variation is small. As a geometric interpretation, we hypothesize that the more varied masks help the network to cope with occlusions better than the rectangular masks that can mimic occlusions only along horizontal and vertical edges.

Considering potential future work, we note that it should be possible to improve the results further with a more principled method for choosing σ so that its distribution is automatically adapted to the training data. 

4.1. TRAINING SETUP:

We use two segmentation networks in our experiments: 1) U-Net (Ronneberger et al., 2015) with a ResNet-50 (He et al., 2016) based encoder that was pre-trained using ImageNet and a randomly initialized decoder (Appendix D.2), and 2) DeepLab v2 network (Chen et al., 2017) based on ResNet-101 and pre-trained for semantic segmentation using the COCO (Lin et al., 2014) dataset, as used by Hung et al. (2018) .

We minimise the sum of the supervised cross-entropy loss and consistency loss L cons with weights of 1 and 10 respectively. We use the ean teacher algorithm (Tarvainen & Valpola, 2017) , as detailed in Appendix D.3. We replace the sigmoidal ramp-up of the consistency regularization weight (Laine & Aila, 2017; Tarvainen & Valpola, 2017) with the average thresholded confidence of the teacher network (see Appendix D.3.4), which increases as the training progresses (French et al., 2018) . Our implementation uses the PyTorch framework and will be made available.

4.2. COMPARISON OF PERTURBATION METHODS:

The CAMVID (Brostow et al., 2008) training set consists of 367 images. We chose 10 subsets of 30 labeled images that were used for the supervised loss, while all training images were used to compute the consistency loss. The U-net setup is used in this test.

The results for different perturbation methods are given in Table 1 . Perturbations based on standard data augmentation (flips, rotations, brightness, etc., see Appendix D.1) and Interpolation Consistency Training (ICT) resulted in no measurable improvement in the mean IoU score compared to the baseline of supervised training using only the labeled samples. The Cutout and CutMix experiments used masks with a single random rectangle, and led to clear improvements over the baseline.

For CowOut and CowMix, we generated the masks using σ ∈ [4, 16] . For CowOut, we found that choosing the masked pixel proportion randomly with p ∼ U(0, 1) produced the best results, whereas for CowMix it was optimal to always use p = 0.5. CowMix led to the best result in this experiment, bridging approximately half of the gap between the supervised baseline and the fully supervised reference result.

We tuned all hyper-parameters using the CAMVID dataset due to its small size and fast run-time. We found that these hyper-parameters also worked well for other, larger datasets.

4.3. RESULTS ON CITYSCAPES AND PASCAL VOC:

We will now compare our results against the state-of-the-art in semi-supervised semantic segmentation, which is currently the adversarial training approach of Hung et al. (2018) . We use two datasets in our experiments. CITYSCAPES consists of urban scenery and has 2975 images in its training set. PASCAL VOC 2012 (Everingham et al., 2012) is more varied, but includes only 1464 training images, and thus we follow the lead of Hung et al. and augment it using SEMANTIC BOUND-ARIES (Hariharan et al., 2011) , resulting in 10582 training images. The practical benefits of semisupervised learning are realised by reducing the required quantity of supervised training data as much as possible, hence we tested our approch using less supervised data than in Hung et al. (2018) . We note that we generated CowMix masks using σ ∈ [8, 32] due to the larger visual features present in CITYSCAPES and PASCAL VOC.

Our results are given in Tables 2 and 3 as mean intersection-over-union (mIoU) percentages, where higher is better. The supervised baseline results between Hung et al. and tion are based on the same setup, provided by the authors, but differ slightly in practice due to the different choice of optimizer, etc.

We can see that using the same DeepLab setup, CowMix outperforms the adversarial training approach in both datasets, with the exception of PASCAL when using a large number of labeled samples. The difference is particularly significant when only a small number of labeled samples is available, e.g., 52% vs 38% mIoU with 100 labeled samples in PASCAL.

When using our U-Net architecture -in which the decoder has not been pre-trained -we were unable to successfully apply the adversarial approach. U-Net improves the performance of CowMix on CITYSCAPES and continues to yield significant gains relative to the supervised baseline in the case of PASCAL. While the performance on the more varied PASCAL dataset significantly benefits the pre-training of DeepLab v2, it also provides a good test environment for semi-supervised learning algorithms when the decoder is randomly initialized.

4.4. DISCUSSION:

We attribute the fact that the adversarial approach (Hung et al. (2018) ) is ineffective with small numbers of labeled samples to the requirements imposed by its discriminator network. A small set of ground truth labels lacks the variation necessary to effectively train the discriminator to distinguish ground truth from predicted segmentation maps, preventing it from effectively guiding the segmentation network. In contrast, consistency regularization minimizes variation in prediction over class preserving perturbation, effectively propagating labels between unlabelled samples. It therefore does not impose similar requirements on the size of the labeled data set.

5. CONCLUSIONS:

We have shown that consistency regularization is a viable solution for semi-supervised semantic segmentation, despite the lack of low-density regions between classes. The proposed CowMix regularization leads to very high-dimensional perturbations that enable state-of-the-art results, while being considerably easier to implement and use than the previous methods based on adversarial training. Even better and more varied perturbation strategies are an obvious avenue for future work.

Additionally, it could be fruitful to investigate when the CowMix approach could also be beneficial in the context of classification. 

A 2D TOY EXPERIMENTS:

The neural networks used in our 2D toy experiments are simple classifiers in which samples are 2D x, y points ranging from -1 to 1. Our networks are multi-layer perceptrons consisting of 3 hidden layers of 512 units, each followed by a ReLU non-linearity. The final layer is a 2-unit classification layer. We use the mean teacher (Tarvainen & Valpola, 2017) semi-supervised learning algorithm with binary cross-entropy as the consistency loss function, a consistency loss weight of 10 and confidence thresholding (French et al., 2018) with a threshold of 0.97.

The ground truth decision boundary was derived from a hand-drawn 512×512 pixel image.

The distance map shown in Figure 2 (c) was computed using scipy.ndimage.morphology.distance transform edt, with distances negated for regions assigned to class 0. Each pixel in the distance map therefore has a signed distance to the ground truth class boundary. This distance map was used to generate the countours seen as lines in Figure 2 (c) and used to support the constrained consistency regularization experiment illustrated in Figure 2(d) .

The constrained consistency regularization experiment described in Section 3.2 required that a sample x should be perturbed to x such that they are at the same -or similar -distance to the ground truth decision boundary. This was achieved by drawing isotropic perturbations from a normal distrubtion x = x + h where h ∼ N (0, 0.117) (0.117 ≈ 30 pixels in the source image), determining the distances m(x) and m(x) from x and x to the ground truth boundary (using a precomputed distance map) and discarding the perturbation -by masking consistency loss for x to 0if |m(x) -m(x)| > 0.016 (0.016 ≈ 4 pixels in the source image).

B DISTRIBUTION OF PATCHES IN THE CITYSCAPES DATASET:

In Section 3.1 we observed that if we extract overlapping neighboring patches from an image, where the patches are centered on neighboring pixels and compute the L 2 distance between the raw pixel content of neighboring patches, we see that the distances vary smoothly across the image. This is due to the fact that the distance between these patches can be efficiently computed by applying a box/uniform filter to the squared gradient image. This equivalence between the patch-to-patch distance and a uniform filter (that suppresses high frequencies and therefore the fine details necessary for low-density regions / large inter-patch distances to lie along class/object/texture boundaries in an image) suggests that these distance maps will be smooth for natural images in general. , where n is the L 2 pixel content distance from the query patch Q i drawn from image I j to the nearest neighbour / most similar patch drawn from image I k such that k = j and b is the distance from the query patch Q i to the patch P i centred on a neighbouring pixel that has a different ground truth class. Put simply, this histogram illustrates that the closest patch to Q i found in any other image in CITYSCAPES is typically at around 3× the distance as an immediate neighbour that lies on the other side of a class boundary.

We will now use the CITYSCAPES dataset to experimentally test our observations concerning the distribution of image patches. Figure 5 illustrates the distributions of distances between pairs of patches centred on neighbouring pixels that have the same class in comparison to pairs of patches centred on pixels that lie either side of a class boundary. The higher proportion of lower pixel content distances between neighboring patches that have the same class can be explained by the presence of large lightly textured regions such as sky that have few class boundaries and patches with similar pixel content.

Further analysis of the CITYSCAPES dataset revealed that in terms of pixel content distance, the nearest neighbours to a patch are other patches centred on neighboring and nearby pixels. We chose 1000 pairs of patches Q i and P i where i ≤ 1000 from CITYSCAPES whose central pixels lie either side of a class boundary. For each query patch Q i we computed the distance to all other patches in the entire dataset. Figure 6 shows that the closest neighbour extracted from another image is typically around 3× the distance from Q i as the immediate neighbor P i .

B.1 EFFICIENT COMPUTATION OF SLIDING WINDOW PATCH DISTANCES:

Computing the distance from patch Q i to every patch centred on each pixel in the complete CITYSCAPES dataset would be prohibitively expensive if performed in a brute-force fashion. There are 2975 training images in CITYSCAPESwith each image having a resolution of 1024 × 512 pixels (downsampled by a factor of 2 from the original resolution, as used in our segmentation experiments). Centering a patch on each pixel results in 1,559,756,800 patches.

Given a query patch Q and a patch A where A is extracted from an image I, the squared distance

A -Q 2 can be expanded to Q•Q+A•A-2(Q•A). Q•Q is a constant, A•

A can be computed for all patches in I in sliding window fashion using the box filter trick from Section 3.1; A • A = B * I 2 where B is a box-filter kernel, I 2 is the element-wide squared image and * represents convolution. Q • A can be computed using correlation; given that the correlation kernel Q is typically large, we use FFT-based convolution.

C PSEUDOCODE:

The mask generation function is given in the form of Python/PyTorch code in Listing 1, and the training functions for semi-supervised CutOut and CutMix are given in Listings 2 and 3, respectively. The latter incorporate the mean teacher framework (Tarvainen & Valpola, 2017) and confidence thresholding (French et al., 2018) as used in our experiments. In keeping with Yun et al. (2019) ; DeVries & Taylor (2017); Verma et al. (2019) , our mask generator allows the proportion of pixels that come from each source image to vary. Although we found via experimentation that for CowMix the best proportion was always p = 0.5, varying the proportion is beneficial for CowOut as detailed in Section 4.2.

1 import numpy as np 2 from scipy.ndimage.filter import gaussian_filter 3 from scipy.special import erfinv Listing 1: Python/NumPy code for cow mask generation 1 def cutout_loss(x, mask, teacher_model, student_model):

2 """x is input image of shape (batch, chan, H, W), mask is mixing mask of shape (batch, 1, H, W)""" In Section 3.3 we stated that for the purpose of semantic segmentation, all geometric transformations need to be applied in reverse for the result image before computing the loss and that translation effectively becomes a no-op. This occurs in a similar manner in both the supervised and unsupervised pathways.

Let us assume that the image x is augmented by applyng the affine transformation matrix M ; x = transform(x, A). We apply our segmentation network f θ to get our predictions ŷpred = f θ (x). We note that the pixel-wise predictions ŷpred were generated from the transformed image x and are therefore are not aligned with our ground truth annotations y. There are two ways in which we can compute the supervised loss L ce ; we can apply the transformation A to the ground truths ŷ = transform(y, A) to align them with ŷpred and compute L ce = cross entropy(ŷ pred , y) or we can apply the inverse transformation A -1 to our predictions to bring them into alignment with the original image y pred = transform(ŷ pred , A -1 ). Should the transformation A include a translation, it will be countered by applying the equivalent translation to the pixel-wise ground truth in the first case or will be undone by applying A -1 to the predictions in the second case.

We must adopt a similar strategy when computing the unsupervised loss term L cons when using affine augmentation as perturbation. If the image x is transformed using two different affine transformations A and B producing x and x. Our network f θ will generate predictions ŷpred and ỹpred that are not aligned with one another. In order to compute the consistency loss L cons we must align the pixel-wise predictions with one another. This can be done by transforming ỹpred into the same frame of reference as ŷpred by applying the transformation AB -1 ; ȳpred = transform(ỹ pred , AB -1 ).

We would like to note that these constraints limit the utility of translations used as an augmentation for semantic segmentation. Classification problems are typically translation invariant; applying a translation will not change the target class. This is not true for semantic segmentation which is translation variant. For example, consider applying a translation to a patch that is centered on a pixel that is very close to a class boundary that separates a car from a road in an image. The translated patch may now have a different class. Computing either supervised or unsupervised loss without countering this translation will either cause the network to learn from erroneous training data.

D.2 U-NET NETWORK ARCHITECTURE:

Our U-Net network is shown in Figure 7 In keeping with Long et al. (2015) we use a batch size of 1. We freeze the batch normalization layers within the ResNet encoder, using the pre-trained running mean and variance rather than computing per-batch mean and variance during training. We use the Adam Kingma & Ba (2015) optimization algorithm with a learning rate of 1 × 10 -4 . As per the mean teacher algorithm Tarvainen & Valpola (2017) , after each iteration the weights w t of the teacher network are updated to be the exponential moving average of the weights w s of the student: w t = α t w t + (1 -α t )w s , where α t = 0.99.

1 4 × 320 Conv 3 × 3 × 64 1 4 × 64 TransposeConv 4 × 4 × 48 1 2 × 48 Conv 3 × 3 × 48 1 2 × 48 TransposeConv 4 × 4 × 32 1 × 32 Conv 3 × 3 × 32 1 × 32 Conv 1 × 1 × C 1 × C

The Cityscapes images were downsampled to half resolution (1024 × 512) prior to use, as in Hung et al. (2018) . When using Cityscapes we trained for 100,000 iterations using a batch size of 1.

We tuned our approach and selected hyper-parameters using the CAMVID dataset due to its small size and fast run-time, after which we applied the same hyper-parameters to Cityscapes. We did not run the augmentation based perturbation experiments on the Cityscapes dataset due to the long run-time involved.

D.3.2 EXPERIMENTS USING DEEPLAB V2 ARCHITECTURE:

We found that we had to decrease the learning rate to 1 × 10 -5 to get good performance with DeepLab v2. We note that the PyTorch implementation of DeepLab v3 worked best with 1 × 10 -4 . We believe that this is because the U-Net and DeepLab v3 networks accept input images with zeromean unit-variance, where DeepLab v2 accepts images whose values are in the range 0 to 255 with only mean subtraction.

For the PASCAL VOC experiments, we extracted 321 × 321 random crops and used a batch size of 10, in keeping with Hung et al. (2018) . For the CITYSCAPES experiments we used full image crops and a batch size of 2.

D.3.3 ABLATION:

The augmentation based perturbation experiments performed on the CAMVID dataset were trained for 50 epochs. The CutMix and CowOut experiments were trained for 100. D.3.4 CONFIDENCE THRESHOLDING French et al. (2018) apply confidence thresholding, in which they mask consistency loss to 0 for samples whose confidence as predicted by the teacher network is below a threshold of 0.968. In the 

:

EXPERIMENTSWe will now describe describe our experiments and main results. We will start by describing the training setup, followed by an investigation of various perturbation methods in the context of semisupervised semantic segmentation, and conclude with a comparison against the state-of-the-art.

Figures and Tables:

Figure 1: In a segmentation task, low-density regions rarely correspond to class boundaries. (a) An image from Camvid dataset. (b) Average L 2 distance between raw pixel contents of a patch centered at pixel p and four overlapping patches centred on the immediate neighbors of p, using 15×15 pixel patches. (c) Same for a more realistic receptive field size of 225×225 pixels. Dark blue indicates large inter-patch distance and therefore a low density region, white indicates a distance of 0. The red lines indicate segmentation ground truth boundaries.

Figure 2: Toy 2D semi-supervised classification experiments. Blue and red circles indicate supervised samples from class 0 and 1 respectively. The field of small black dots indicate unsupervised samples. The learned decision function is visualized by rendering the probability of class 1 in green; the soft gradation represents the gradual change in predicted class probability. (a, b) Semi-supervised learning with and without a low density region separating the classes. The dotted orange line in (a) shows the decision boundary obtained with plain supervised learning. (c) Rendering of the distance to the true class boundary with distance map contours. Strong colours indicate greater distance to class boundary. (d) Decision boundary learned when samples are perturbed along distance contours in (c). The magenta line indicates the true class boundary. Appendix A explains the setup in detail.

Figure 3: Illustration of mixing regularization for semi-supervised semantic segmentation with the mean teacher framework. θ s and θ t denote the weights of the student and teacher networks, respectively. The arbitrary mask M is omitted from the argument list of function mix for legibility.

Figure 4: Example CowOut/CowMix masks with p = 0.5 and varying σ in a 384×384 pixel image.

(a) standard augmentation and (b) CutMix experiments where convergence failures were observed. For those experiments, the results were taken after (a) 50 and (b) 100 epochs.

Figure 5: Histogram of L 2 distance between pairs of patches of size 225 × 225 centred on neighboring pixels, split between those with the same class in blue and pairs that lie either side of a class boundary in orange. Computed from all patches in the CITYSCAPES dataset whose central pixel has a valid ground truth class.

Figure 6: Histogram of the ratio of the distances n b, where n is the L 2 pixel content distance from the query patch Q i drawn from image I j to the nearest neighbour / most similar patch drawn from image I k such that k = j and b is the distance from the query patch Q i to the patch P i centred on a neighbouring pixel that has a different ground truth class. Put simply, this histogram illustrates that the closest patch to Q i found in any other image in CITYSCAPES is typically at around 3× the distance as an immediate neighbour that lies on the other side of a class boundary.

def generate_mixing_mask(img_size, sigma_min, sigma_max, p_min, p_max): 6 Randomly draw sigma from log-uniform distribution 7 sigma = np.exp(np.random.uniform(np.log(sigma_min), np.log(sigma_max))) (p * 2 -1) * (2 ** 0.5) * Ns.std() + Ns.mean() 13 return (noise_smooth > t).astype(float) # Apply threshold and return

Apply teacher model and softmax to get per-pixel class probability Consistency is squared error between student and teacher preds for masked pixels only, modulated with confidence factor 11 return (squared_diff(ym_t, ym_s) * m).mean() * conf_fac Listing 2: PyTorch code for CutOut / CowOut for segmentation 1 def cutmix_loss(xa, xb, mask, teacher_model, student_model): 2 """xa, xb are input image pair each of shape (batch, chan, H, W), mask is mixing mask of shape (batch, 1, H, W)""" 3 Apply teacher model and softmax to get per-pixel class probability 4 ya_t = softmax(teacher_model(xa), dim=1) 5 yb_t = softmax(teacher_model(yb), dim=1) Consistency is squared error between student and teacher preds, modulated with confidence factor 15 return squared_diff(ym_t, ym_s).mean() * conf_fac Listing 3: PyTorch code for CutMix / CowMix for segmentation D SEMANTIC SEGMENTATION EXPERIMENTS D.1 DATA AUGMENTATION Our data augmentation scheme that we used for standard augmentation based perturbation consists of an affine transformation composed of horizontal flips, translation in the range [-4, 4] pixels,

Figure 7: ResNet-50 based U-Net decoder architecture.

66% ± 1.80 46.24% ± 2.21 48.29% ± 2.01 53.09% ± 2.56 50.95% ± 2.49 52.93% ± 1.35 55.06%± 1.74 64.19% ± 0.41 Measurements and ablation on CAMVID test set. Our results are mean intersection-overunion (mIoU) presented as mean ± std .dev computed from 10 runs. Training was run for 300 epochs, except in

Table content: <table><tr><td>Sup. baseline Std. aug (a)</td><td>ICT</td><td>Cutout</td><td>CutMix (b)</td><td>CowOut</td><td>CowMix</td><td>Fully sup.</td></tr><tr><td>48.</td><td/><td/><td/><td/><td/><td/></tr></table>

our DeepLab implementa-Under review as a conference paper at ICLR 2020 Performance (mIoU) on CITYSCAPES validation set, each computed from 5 runs. The results forHung et al. (2018) are from their paper.

Table content: <table><tr><td colspan="2">Labeled samples</td><td>100</td><td/><td colspan="2">372 (12.5%)</td><td colspan="2">744 (25%)</td><td>1488 (50%)</td><td>2975 (All)</td></tr><tr><td/><td/><td colspan="6">Hung et al. (2018): Adversarial training with COCO-pretrained DeepLab v2 network</td></tr><tr><td>Baseline</td><td/><td>-</td><td/><td colspan="2">55.5%</td><td colspan="2">59.9%</td><td>64.1%</td><td>66.4%</td></tr><tr><td colspan="2">Semi-supervised</td><td>-</td><td/><td colspan="2">58.8%</td><td colspan="2">62.3%</td><td>65.7%</td><td>-</td></tr><tr><td>-Delta</td><td/><td>-</td><td/><td>3.5</td><td/><td/><td>2.4</td><td>1.6</td><td>-</td></tr><tr><td/><td/><td colspan="6">Our results: Same COCO-pretrained DeepLab v2 network</td></tr><tr><td>Baseline</td><td/><td colspan="2">44.34% ± 1.61</td><td colspan="2">56.02% ± 0.80</td><td colspan="2">60.90% ± 0.70</td><td>65.05% ± 0.71</td><td>67.79% ± 0.32</td></tr><tr><td>ICT</td><td/><td/><td/><td colspan="2">55.30% ± 1.14</td><td/></tr><tr><td>CutOut</td><td/><td/><td/><td colspan="2">54.25% ± 0.69</td><td/></tr><tr><td>CutMix</td><td/><td/><td/><td colspan="2">57.64% ± 0.91</td><td/></tr><tr><td>CowOut</td><td/><td/><td/><td colspan="2">58.09% ± 1.37</td><td/></tr><tr><td>CowMix</td><td/><td colspan="2">49.01% ± 2.58</td><td colspan="2">60.53% ± 0.29</td><td colspan="2">64.10% ± 0.82</td><td>66.51% ± 0.45</td><td>69.03% ± 0.27</td></tr><tr><td>-Delta</td><td/><td>4.67</td><td/><td>4.51</td><td/><td/><td>3.20</td><td>1.46</td><td>1.24</td></tr><tr><td/><td/><td colspan="6">Our results: U-Net and randomly initialized decoder</td></tr><tr><td>Baseline</td><td/><td colspan="2">43.83% ± 0.99</td><td colspan="2">54.76% ± 0.50</td><td colspan="2">60.35% ± 0.93</td><td>64.93% ± 0.32</td><td>68.34% ± 0.76</td></tr><tr><td>CowMix</td><td/><td colspan="2">51.98%± 2.76</td><td colspan="2">61.48%± 1.84</td><td colspan="2">64.85%± 0.26</td><td>66.90%± 0.22</td><td>69.57%± 0.43</td></tr><tr><td>-Delta</td><td/><td>8.15</td><td/><td>6.72</td><td/><td/><td>4.50</td><td>1.97</td><td>1.23</td></tr><tr><td># Labels</td><td>100</td><td/><td>200</td><td/><td>400</td><td/><td>800</td><td>2646 (25%)</td><td>10582 (All)</td></tr><tr><td/><td colspan="7">Hung et al. (2018): Adversarial training with COCO-pretrained DeepLab v2 network</td></tr><tr><td>Baseline</td><td colspan="2">39.22% ± 2.08</td><td colspan="2">46.57% ± 1.34</td><td colspan="2">55.65% ± 0.88</td><td>62.54% ± 0.45</td><td>68.41% ± 0.29</td><td>72.50% ± 0.27</td></tr><tr><td>Semi-sup.</td><td colspan="2">38.82% ± 3.91</td><td colspan="2">49.40% ± 1.00</td><td colspan="2">60.29% ± 2.25</td><td>66.45% ± 0.69</td><td>71.27%± 0.26</td><td>-</td></tr><tr><td>-Delta</td><td>0.40</td><td/><td>2.83</td><td/><td>4.64</td><td/><td>3.91</td><td>2.86</td><td>-</td></tr><tr><td/><td colspan="7">Our results: Same COCO-pretrained DeepLab v2 network</td></tr><tr><td>Baseline</td><td colspan="2">41.17% ± 1.96</td><td colspan="2">48.96% ± 1.70</td><td colspan="2">58.18% ± 1.16</td><td>64.51% ± 0.85</td><td>70.16% ± 0.35</td><td>73.32% ± 0.19</td></tr><tr><td>CowMix</td><td colspan="2">52.10%± 1.35</td><td colspan="2">57.84%± 1.55</td><td colspan="2">64.18%± 1.79</td><td>67.84%± 0.88</td><td>70.99% ± 0.41</td><td>73.43%± 0.06</td></tr><tr><td>-Delta</td><td>10.93</td><td/><td>8.88</td><td/><td>6.00</td><td/><td>3.33</td><td>0.83</td><td>0.11</td></tr><tr><td/><td colspan="5">Our results: U-Net and randomly initialized decoder</td><td/></tr><tr><td>Baseline</td><td colspan="2">24.91% ± 2.13</td><td colspan="2">33.66% ± 1.64</td><td colspan="2">41.80% ± 1.15</td><td>49.18% ± 1.61</td><td>60.58% ± 0.77</td><td>65.97% ± 1.11</td></tr><tr><td>CowMix</td><td colspan="2">41.00% ± 2.70</td><td colspan="2">49.42% ± 1.74</td><td colspan="2">54.20% ± 1.12</td><td>54.67% ± 5.66</td><td>63.48% ± 0.26</td><td>65.54% ± 4.99</td></tr><tr><td>-Delta</td><td>16.09</td><td/><td>15.76</td><td/><td>12.4</td><td/><td>5.49</td><td>2.90</td><td>-0.43</td></tr><tr><td colspan="8">Table 3: Performance (mIoU) on augmented PASCAL VOC validation set, each computed from 5</td></tr><tr><td>runs.</td><td/><td/><td/><td/><td/><td/></tr></table>

and Table 4.

Table content: <table><tr><td>Description</td><td>Resolution × channels</td></tr><tr><td>ResNet-50 layer conv5 x 1 32 res, 2048 chn Conv 3 × 3 × 192 TransposeConv 4 × 4 × 128 Concat with ResNet-50 layer conv4 x 1 16 res, 1024 chn Conv 3 × 3 × 128 TransposeConv 4 × 4 × 96 Concat with ResNet-50 layer conv3 x 1 8 res, 512 chn Conv 3 × 3 × 96 TransposeConv 4 × 4 × 64 Concat with ResNet-50 layer conv2 x 1 4 res, 256 chn</td><td>1 32 × 2048 1 32 × 192 1 16 × 128 1 16 × 1152 1 16 × 128 1 8 × 96 1 8 × 608 1 8 × 96 1 4 × 64</td></tr></table>

ResNet-50 based U-Net decoder. C is the number of target classes.

Table content: <table><tr><td>D.3 TRAINING DETAILS</td></tr><tr><td>D.3.1 EXPERIMENTS USING U-NET ARCHITECTURE</td></tr></table>

Per-class performance on CITYSCAPES dataset, 100 supervised samples 80% ± 18.54 27.40% ± 8.64 40.80% ± 16.27 53.40% ± 27.22 Adv. Semi-sup. 58.80% ± 26.95 7.80% ± 8.47 25.20% ± 21.17 59.80% ± 30.72 Adv. Full 85.20% ± 0.98 60.40% ± 1.62 78.40% ± 0.49 90.00% ± 0.63 DeepLab Baseline 53.06% ± 12.50 29.79% ± 3.06 40.19% ± 11.65 52.14% ± 26.38 DeepLab CowMix 75.49% ± 2.11 51.59% ± 6.62 58.94% ± 2.76 53.47% ± 29.03 80% ± 23.68 12.51% ± 9.71 DeepLab CowMix 78.10% ± 0.39 14.53% ± 17.63 32.57% ± 28.15 16.88% ± 15.62 95% ± 28.67 43.72% ± 9.62 DeepLab CowMix 39.51% ± 32.95 63.18% ± 1.10 DeepLab Full 81.98% ± 0.13 68.15% ± 1.80 U-net Baseline 17.64% ± 16.75 29.58% ± 7.32 U-net CowMix 33.16% ± 27.70 46.93% ± 6.07 U-net Full 75.06% ± 1.67 70.86% ± 1.11

Table content: <table><tr><td/><td>OVERALL</td><td>Road</td><td>Sidewalk</td><td>Building</td></tr><tr><td colspan="2">DeepLab Baseline 44.34% ± 1.61</td><td>94.21% ± 0.18</td><td>60.29% ± 1.09</td><td>83.06% ± 0.09</td></tr><tr><td colspan="2">DeepLab CowMix 49.01% ± 2.58</td><td>95.69% ± 0.24</td><td>68.62% ± 1.11</td><td>85.47% ± 0.30</td></tr><tr><td>DeepLab Full</td><td>67.79% ± 0.32</td><td>97.13% ± 0.09</td><td>77.74% ± 0.36</td><td>89.29% ± 0.07</td></tr><tr><td>U-net Baseline</td><td>43.83% ± 0.99</td><td>95.12% ± 0.25</td><td>65.94% ± 1.74</td><td>83.78% ± 0.34</td></tr><tr><td>U-net CowMix</td><td>51.98% ± 2.76</td><td>96.61% ± 0.20</td><td>74.59% ± 1.57</td><td>86.81% ± 0.27</td></tr><tr><td>U-net Full</td><td>68.34% ± 0.76</td><td>97.57% ± 0.02</td><td>80.26% ± 0.16</td><td>90.54% ± 0.12</td></tr><tr><td/><td>Wall</td><td>Fence</td><td>Pole</td><td>Traffic light</td></tr><tr><td colspan="2">DeepLab Baseline 13.51% ± 3.39</td><td>14.86% ± 2.63</td><td>33.31% ± 0.49</td><td>22.20% ± 5.68</td></tr><tr><td colspan="2">DeepLab CowMix 15.85% ± 4.84</td><td>19.82% ± 4.89</td><td>36.66% ± 0.79</td><td>28.14% ± 7.10</td></tr><tr><td>DeepLab Full</td><td>46.12% ± 1.51</td><td>45.09% ± 0.88</td><td>43.91% ± 0.14</td><td>49.49% ± 0.73</td></tr><tr><td>U-net Baseline</td><td>10.65% ± 2.28</td><td>12.84% ± 1.67</td><td>39.59% ± 1.09</td><td>26.29% ± 4.40</td></tr><tr><td>U-net CowMix</td><td>13.91% ± 5.01</td><td>18.87% ± 3.07</td><td>46.25% ± 0.50</td><td>36.94% ± 5.18</td></tr><tr><td>U-net Full</td><td>44.22% ± 2.36</td><td>47.52% ± 0.69</td><td>56.29% ± 0.47</td><td>58.68% ± 1.28</td></tr><tr><td/><td>Traffic sign</td><td>Vegetation</td><td>Terrain</td><td>Sky</td></tr><tr><td colspan="2">DeepLab Baseline 39.65% ± 2.08</td><td>84.42% ± 0.40</td><td>37.48% ± 1.94</td><td>87.30% ± 0.75</td></tr><tr><td colspan="2">DeepLab CowMix 45.30% ± 1.48</td><td>86.60% ± 0.20</td><td>40.82% ± 3.46</td><td>90.88% ± 0.24</td></tr><tr><td>DeepLab Full</td><td>62.67% ± 0.39</td><td>89.42% ± 0.04</td><td>57.59% ± 0.79</td><td>92.80% ± 0.08</td></tr><tr><td>U-net Baseline</td><td>40.83% ± 3.27</td><td>86.54% ± 0.27</td><td>41.02% ± 3.02</td><td>87.47% ± 1.40</td></tr><tr><td>U-net CowMix</td><td>55.89% ± 3.44</td><td>88.73% ± 0.05</td><td>45.08% ± 4.41</td><td>91.12% ± 1.31</td></tr><tr><td>U-net Full</td><td>70.43% ± 0.55</td><td>90.96% ± 0.12</td><td>59.19% ± 0.80</td><td>94.05% ± 0.18</td></tr><tr><td/><td>Person</td><td>Rider</td><td>Car</td><td>Truck</td></tr><tr><td colspan="2">DeepLab Baseline 55.76% ± 0.93</td><td>17.59% ± 3.70</td><td>83.70% ± 1.14</td><td>17.46% ± 8.96</td></tr><tr><td colspan="2">DeepLab CowMix 61.39% ± 0.39</td><td>23.38% ± 3.80</td><td>86.33% ± 1.50</td><td>23.15% ± 13.60</td></tr><tr><td>DeepLab Full</td><td>69.89% ± 0.24</td><td>47.37% ± 0.76</td><td>91.52% ± 0.08</td><td>67.77% ± 1.65</td></tr><tr><td>U-net Baseline</td><td>57.75% ± 1.81</td><td>14.69% ± 2.56</td><td>85.13% ± 0.49</td><td>5.86% ± 2.83</td></tr><tr><td>U-net CowMix</td><td>66.86% ± 0.90</td><td>26.85% ± 4.78</td><td>88.42% ± 1.57</td><td>17.77% ± 10.13</td></tr><tr><td>U-net Full</td><td>75.18% ± 0.40</td><td>50.23% ± 0.96</td><td>92.23% ± 0.27</td><td>54.06% ± 2.89</td></tr><tr><td/><td>Bus</td><td>Train</td><td>Motorcycle</td><td>Bicycle</td></tr><tr><td colspan="3">DeepLab Baseline 24.11% ± 12.62 14.49% ± 7.71</td><td>7.40% ± 5.63</td><td>51.69% ± 1.24</td></tr><tr><td colspan="5">DeepLab CowMix 24.52% ± 20.60 28.07% ± 14.68 13.86% ± 11.17 56.69% ± 1.10</td></tr><tr><td>DeepLab Full</td><td>78.46% ± 0.86</td><td>65.02% ± 4.88</td><td>50.41% ± 1.75</td><td>66.36% ± 0.13</td></tr><tr><td>U-net Baseline</td><td>15.42% ± 5.23</td><td>7.65% ± 3.57</td><td>6.31% ± 3.76</td><td>49.90% ± 1.60</td></tr><tr><td>U-net CowMix</td><td colspan="2">25.80% ± 16.62 20.02% ± 9.65</td><td colspan="2">24.43% ± 10.97 62.63% ± 0.88</td></tr><tr><td>U-net Full</td><td>68.56% ± 3.08</td><td>51.44% ± 6.32</td><td>46.26% ± 1.31</td><td>70.74% ± 0.41</td></tr></table>

Per-class performance on augmented PASCAL VOC dataset, 100 supervised samples

Table content: <table/>

Peer review (iclr2020):
# Summary This paper proposes a method for semi-supervised semantic segmentation. The authors tackle this problem through consistency regularization, a successful technique in image classification, that encourages the network to give consistent predictions on unlabeled samples which are perturbed in multiple ways. The authors argue that the cluster assumption (to which effectiveness of consistency regularization has been partially attributed) does not hold in semantic segmentation. Thus, in order to enable class boundaries to become low-density regions and then better guide contrastive regularization, the authors argue that a stronger perturbation must be inserted. To this effect they first propose looking at CutOut and CutMix types of methods. They improved upon them by putting forward a variant of CutMix, coined CowMix, with more degrees of freedom and using flexible masks instead of rectangular ones. CowMix is evaluated on the Cityscapes and PascalVOC 2012 datasets in the semi-supervised regime and showing encouraging results. # Rating I find the paper and the advanced ideas of interest for the community and I consider they are novel. I*m currently on the fence between Weak Accept and Weak Reject, mostly due to incomplete evaluations and support for claims made in the introduction regarding the infeasibility of contrastive regularization methods for semantic segmentation. I would be happy to upgrade my rating if authors addressed these concerns. # Strong points - The paper is well written and mostly clear with a good coverage and positioning w.r.t. related work. The authors illustrate well the reasoning and the choices they have made. The author provide plenty of ablation studies (e.g., per class statistics) and implementation details, improving significantly the reproducibility of the contribution. - The flexible masking technique that is advanced here is novel and experimentally seems effective. - This work is among the few that address semi-supervised semantic segmentation in a non-adversarial manner, so I would give it some novelty credit. - I appreciate the evaluation protocol of averaging across multiple runs. # Weak points ## Unclear aspects - The authors argue that consistency regularization has had little success so far in semantic segmentation problems since low density regions in input data do not align well with class boundaries. It would be useful to provide a reference to this claim or at least validate it experimentally on a large dataset. - In Figure 1, it is not clear on which features where the distances between patches computed? Is it on raw pixels or intermediate feature maps from a CNN? If the distances are made over raw pixels, I find it difficult to make the connection between distances in the pixel space and distances in the class space. Are the neighbor patches overlapping with the central/query patch? ## Experiments - The authors compare against other methods on the CamVid dataset. CamVid is a small and relatively limited dataset (~367 images for training from the streets of Cambridge). I*m worried that this dataset might not be enough to conclude and emphasize the benefits of this method over other semi-supervised techniques. For instance CowOut does not seem do be above CutOut, while CutMix has convergence problems and low scores. The other experiments on Cityscapes and Pascal VOC are certainly interesting, but the method is compared only against Hung et al. which a different family of methods and the subset baseline (which is useful but not enough). I think this work would benefit from an additional baseline in the style of contrastive regularization methods, e.g. ICT, and eventually CutOut, to support the initial arguments regarding the limitations of these methods in semantic segmentation and respectively the effectiveness of the flexible masks over the rectangular ones in this setup. # Suggestions for improving the paper: 1) It would be useful to include other semi-supervised baselines, e.g. ICT, and the baseline perturbation CutMix on larger experiments, in order to better emphasize the contributions of this work. 2) Did the authors try the flexible masking on image classification? How is it expected to perform over ICT, MixUp or MixMatch?