Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 

# Summary

This paper proposes a method for semi-supervised semantic segmentation. 
The authors tackle this problem through consistency regularization, a successful technique in image classification, that encourages the network to give consistent predictions on unlabeled samples which are perturbed in multiple ways. The authors argue that the cluster assumption (to which effectiveness of consistency regularization has been partially attributed) does not hold in semantic segmentation. Thus, in order to enable class boundaries to become low-density regions and then better guide contrastive regularization, the authors argue that a stronger perturbation must be inserted. To this effect they first propose looking at CutOut and CutMix types of methods. They improved upon them by putting forward a variant of CutMix, coined CowMix, with more degrees of freedom and using flexible masks instead of rectangular ones. CowMix is evaluated on the Cityscapes and PascalVOC 2012 datasets in the semi-supervised regime and showing encouraging results.


# Rating
I find the paper and the advanced ideas of interest for the community and I consider they are novel. I'm currently on the fence between Weak Accept and Weak Reject, mostly due to incomplete evaluations and support for claims made in the introduction regarding the infeasibility of contrastive regularization methods for semantic segmentation. I would be happy to upgrade my rating if authors addressed these concerns.


# Strong points
- The paper is well written and mostly clear with a good coverage and positioning w.r.t. related work. The authors illustrate well the reasoning and the choices they have made. The author provide plenty of ablation studies (e.g., per class statistics) and implementation details, improving significantly the reproducibility of the contribution.
- The flexible masking technique that is advanced here is novel and experimentally seems effective.
- This work is among the few that address semi-supervised semantic segmentation in a non-adversarial manner, so I would give it some novelty credit.
- I appreciate the evaluation protocol of averaging across multiple runs.

# Weak points

## Unclear aspects
- The authors argue that consistency regularization has had little success so far in semantic segmentation problems since low density regions in input data do not align well with class boundaries. It would be useful to provide a reference to this claim or at least validate it experimentally on a large dataset.

- In Figure 1, it is not clear on which features where the distances between patches computed? Is it on raw pixels or intermediate feature maps from a CNN? 
If the distances are made over raw pixels, I find it difficult to make the connection between distances in the pixel space and distances in the class space.
Are the neighbor patches overlapping with the central/query patch?

## Experiments
- The authors compare against other methods on the CamVid dataset. CamVid is a small and relatively limited dataset (~367 images for training from the streets of Cambridge). I'm worried that this dataset might not be enough to conclude and emphasize the benefits of this method over other semi-supervised techniques. For instance CowOut does not seem do be above CutOut, while CutMix has convergence problems and low scores. 
The other experiments on Cityscapes and Pascal VOC are certainly interesting, but the method is compared only against Hung et al. which a different family of methods and the subset baseline (which is useful but not enough). I think this work would benefit from an additional baseline in the style of contrastive regularization methods, e.g. ICT, and eventually CutOut, to support the initial arguments regarding the limitations of these methods in semantic segmentation and respectively the effectiveness of the flexible masks over the rectangular ones in this setup.


# Suggestions for improving the paper:
1) It would be useful to include other semi-supervised baselines, e.g. ICT, and the baseline perturbation CutMix on larger experiments, in order to better emphasize the contributions of this work.

2) Did the authors try the flexible masking on image classification? How is it expected to perform over ICT, MixUp or MixMatch?

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper provided first provided analysis for the problem of semantic segmentation. Through a few simple example, the authors suggested that the cluster assumption doesn’t hold for semantic segmentation. The paper also illustrated how to perturb the training examples so that consistency regularization still works for semantic segmentation. 
The paper also introduce a perturbation method that can achieve high dimensional perturbation, which achieve solid experimental results.

The analysis part seems interesting and innovative to me. But it is very qualitative and I'm not fully convinced that the analysis on 2d example can actually carry over to high dimensional spaces for images. I also don't quite see the connection between the toy example and the proposed perturbation method. For example, why the proposed perturbation method has the property of "the probability of a perturbation crossing the true class boundary must be very small compared to the amount of exploration in other dimensions"?

The proposed algorithm is an extension of the existing cutout and cut mix. The way to generate new mask is a very smart design to me. This should be the most important contribution of the paper.

The writing of the paper is very clear and easy to follow. The experimental results look very convincing overall and proposed algorithm does show very promising results. 

To sum up, the paper is an ok paper from the practical perspective, but the analysis in the paper wasn't strong enough to me.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This work analyzes the consistency regularization in semi-supervised semantic segmentation. Based on the results on a toy dataset, this work proposes a novel regularization for semi-supervised semantic segmentation, which is named CowMix.

Pros:
-- The proposed CoxMix is easy to understand and implement.
-- The experimental results seem to benefit from this proposed CoxMix at a first glance.

Cons:
The writing is not clear. Sometimes I have to make a ``guess" about the technical details. For example:
-- Other than L_{cons}, is there any other Loss term utilized in this work? Based on Figure 3, it seems only L_{cons} is utilized. If so, is it a waste not to use the label training data (although very few) to calculate a cross-entropy loss?

-- It seems the experimental setting in this submission follows the settings in Hung, 2018. However, for the experiment on VOC 2012 validation set, Hung tested their method on 1/8 1/4 1/2 of labeled data (Table 1). While in this submission, Table 3 shows the results on label data of 100, 200, 400, 800, 2646(25%). The split ratios seem different from Hung's work, which confuses me.

-- "Note that in context of semantic segmentation, all geometric transformations need to be applied in reverse for the result image before computing the loss (Ji et al., 2018). As such, translation turns into a no-op, unlike in classification tasks where it remains a useful perturbation." 
    Is there any experimental result to support this claim?

-- It is a little hard for me to fully understand Figure 2. For example, how to get 2. (c)? What is the meaning of the word "gap" here?

review_title: Official Blind Review #1

Meta-review: This paper proposes a method for semi-supervised semantic segmentation through consistency (with respect to various perturbations) regularization. While the reviewers believe that this paper contains interesting ideas and that it has been substantially improved from its original form, it is not yet ready for acceptance to ICLR-2020. With a little bit of polish, this paper is likely to be accepted at another venue.
Decision: Reject