Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper extends semantic segmentation models with a contrastive loss on "representations" at different locations in the image. A separate "representation head" is built from the intermediate features (between the "encoder" and "decoder") of the fully-convolutional segmentation network (DeepLabV3+). The "regional contrast" (ReCo) loss then encourages self-similarity between the representations of pixels belonging the same class and less similarity between representations for pixels of different classes. The submission investigates adding the ReCo loss both fully supervised and semi-supervised semantic segmentation training on Pascal VOC, CityScapes, and SUN RGB-D benchmarks.
main_review: **Strengths**

i) Very illustrative experiments.

One nice additional experiment is the plotting of dendograms over the class representation. These provide some evidence that the representations make sense, independent of the effect on end-to-end accuracy.

There are also very extensive ablation & hyperparameter studies. The effect of the "active sampling" method is particularly interesting.

ii) Submission includes well-documented code.

The README gives full setup instructions and some sample training commands. Code for necessary setup to reproduce results, such as dataset preprocessing, is provided. The code is commented to an okay extent, and the names make sense. It is fairly straightfoward to find missing details in the code and resolve any confusions.

Minor comment: Code organization could be improved, especially in having a file "module_list.py" with a fairly generic name that includes classes & functions with very disparate purposes.

iii) Strong accuracy results.

The authors set up an experimental baseline with the state of the art in data augmentation, and extensively use all the available similar "tricks" to improve accuracy. The ReCo loss successfully improves accuracy on top of these as well.

The baselines (on which the submission still improves) are solid overall: "most of the time surpassing, the performance reported in each publication." For example, the authors report 72.9 mIoU for ClassMix on 5% of labels in PASCAL VOC, vs 67.77 mIoU reported in Table 2 of Olsson et al. This is unsuprising because this submission uses DeepLabV3+ as the baseline network while Olsson et. al. use DeepLabV2. But it is good that the authors resolve this confound and report the best version of baseline methods for comparison.

**Weaknesses**

iv) No empirical comparisons to other representation-based methods.

Unless I missed it, I don't see experimental comparison to what seems to be the most closely-related works: those under the "Contrastive Learning" subsection. Even if the other works don't e.g. emphasize semi-supervised learning, I would guess at least a partial comparison could be made given that the submission does also support fully-supervised training in the code. (As it naturally would, given that the upper bound of the % of artificially-sampled labels to use is 100%.)

It's less than ideal to see a submission propose both a new benchmark (as described in the "Semi-Supervised Segmentation Benchmark Redesign" subsection) and a new method simultaneously, with the method not also evaluated on existing simpler benchmarks.

v) Lack of ablation on the "Mean Teacher" component.

The final method proposed in the submission includes a union of a lot of the techniques in the literature for semi-supervised segmentation. So it would take quite a lot of experiments, on top of the already-extensive ones that were done, to consider ablating everything. One relatively ablation that was elided was using the original network as for pseudo-labels.

**Minor Comments**

  * Weird grammar in "better performance from a variety choice of semi-supervised baselines" on page 8. I'm not entirely sure what the intended meaning is here.
  * which -> that above Figure 8
  * Instructions on setting up CityScapes dataset for experiments seems to assume that zip files add an enclosing directory:
    For example, with the basic `unzip` command on ubuntu, gtFine_trainvaltest.zip unpacks into gtFine/. The cityscapes_preprocess.py script, though, seems to assume it goes in gtFine_trainvaltest/gtFine/. This then also fails silently with empty train_label_list, causing failures later. Documenting the expected cityscapes directory layout in README.md is a common practice, I recommend it here. See for example:

https://github.com/google-research/deeplab2/blob/main/g3doc/setup/cityscapes.md

I also recommend some assertions on the input in cityscapes_preprocess.py to catch these kinds of errors.

  * A possibly slightly closer comparison for contrastive loss in semantic segmentation is "Contrastive Learning for Label-Efficient Semantic Segmentation" by Zhao et al. Like (Wang et al., 2021a) it is roughy concurrent work (also published in ICCV 2021. Zhao et. al. also considers semi-supervised learning (though not as much of a focus as in the submission).
  * Suggest repeating description from Figure 8 that "Brighter color represents closer... relationship" in Appendix E as well (assuming the figures there have the same meaning).
summary_of_the_review: This is an interesting approach, and the authors provide extensive high-quality experiments that cover both end-to-end accuracy and illustrating how the method works.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work presents a new semi-supervised learning framework based on a pixel-level contrastive learning with active query/key sampling. For a given query feature with a class c, the positive key sample is generated as a mean vector of all samples with the class c, while the negative key sample is sampled by considering the pair-wise relationship between classes in (3) such that more samples are from hard cases. The active query is adaptively sampled by considering rare queries in order to overcome the class imbalance in semantic segmentation as in (4). The proposed contrastive loss, call ReCo, is applied to semi-supervised semantic segmentation task with two cases: full labels and partial labels. Extensive results demonstrate the outstanding performance when using ReCo loss within existing semi-supervised learning methods.
main_review: * Pros
1) The active query and key sampling seems to be effective in optimizing the contrastive loss with little computational overhead.
2) The exhaustive comparative study is provided. The performance was validated by re-implementing all existing methods in the same backbone architectures and training strategies for a fair comparison.
3) The performance gain over the existing methods is impressive in the semi-supervised semantic segmentation task.

* Cons
1) The sampling idea is rather simple. While the query is sampled using the pair-wise similarity measure between classes, the hard key is simply selected by thresholding the prediction confidence.
a) The pair-wise similarity is computed within mini-batches, meaning this may be unable to reflect the global distribution over entire datasets when sampling the active query.
b) The hard key sample is also selected with the simple thresholding. The performance may heavily depend on how to setup the threshold, and more importantly it is not considered that the threshold may be varying with respect to classes.

2) The pseudo labels for unlabeled images are generated by simply applying the threshold. This simple idea poses a risk of still generating unreliable pseudo labels. It would be better to refer to recent approaches proposed in the self-training of unsupervised domain adaptation (UDA).

summary_of_the_review: The proposed query/key sampling may be useful for the semi-supervised semantic segmentation task, and the idea was verified using extensive experiments. However, the simple thresholding used for sampling negative key and generating pseudo labels may be often sensitive when deployed in real-world dataset.

* Post-rebuttal: The rebuttal letter has addressed some concerns in the original comments. The additional ablation study in the appendix supports the effectiveness of computing the pair-wise similarity in mini-batches. The argument on the the fixed threshold is also reasonable. Thus, I improve my rating score from '6: marginally above the acceptance threshold' to '8: accept, good paper'.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed to implement  the contrastive learning loss into the semi-supervised semantic segmentation framework. The method can achieve high quality semantic segmentation results by using only a few labeled samples (e.g 5 examples of each semantic class).
main_review: Strengths:
* The proposed method can achieve high-quality semantic segmentation results with only 5 examples of each semantic class.
* The presentation and organization are clear and easy to follow.
* The figures in the paper look nice. 
* Would be definitely accepted if it's presented six month ago.

Weakness:
* The similar idea have been presented in existing publications [1, 2] (appeared on arXiv before May 2021). [1] is missing in the relate work. 

[1] Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank. Alonso et al. ICCV 2021. 

[2] Exploring cross-image pixel contrast for semantic segmentation. Wang et al. ICCV 2021

No considering the later one:

[3] Looking beyond single images for contrastive semantic segmentation. Zhang et al. NeurIPS 2021.
summary_of_the_review: The contrastive learning for semi-supervised semantic segmentation has been presented in [1]. 
The differences/novelties compared to [1] are not significant enough for a new publication. 

- Post-rebuttal:
I slightly improve my rating score from 3 to 5 (marginally below the acceptance threshold). 
ICCV paper [1] is released on Arxiv in May, accepted in August and officially published in October.
Paper [1] must be addressed and compared in the paper. The author must make a clear statement on the differences and improvements (compared with [1]).

- For the rebuttal: "[1] and [2] both apply contrastive learning with stored feature banks, but we sample features on-the-fly".
The momentum memory bank has been shown effective in [MoCo, 1,2]. It boosts the scope for selecting the positive & negative pairs. "Sample features on-the-fly" can't be argued as an advantage over existing work without supporting experiments. Intuitively, this limits the selection of the contrastive pairs into the current mini-batch and would lead to worse accuracy. Experiments must be done to support this argument. 

- After further discussion, 
I would like to raise the rating score to 6 (marginally above the acceptance threshold).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes ReCo, a regional contrastive learning method for semi-supervised semantic segmentation. The query and key pixel sampling methods are proposed for efficient learning. The proposed method showed state-of-the-art level performances in various settings and various datasets. 
main_review: === Strength ===
1. The proposed method is reasonable and well-motivated.

2. The paper is very well-written and easy to understand.

3. The experimental results and analysis presented by the authors align well with the authors' claims.

4. The proposed method showed state-of-the-art level performances in various settings and various datasets.


=== Weakness ===
1. Novelty. 
- The contrastive approach proposed in this paper seems to have slightly modified the contrastive learning technique used in classification. In addition, the mean-teacher method has been considered by many existing semi-supervised learning methods.

2. Experiment. 
- The authors provide only the results of ReCo+ClassMix, and do not provide the results of ReCO itself. 
- Baselines in Table 1 seem weak. Recent methods [ref1, ref2, ref3] should be compared.
- The strong baseline [ref2] is missing. [ref2] produces 72.4 (1/16), 74.6 (1/8), and 76.3 (1/4) with ResNet-101 DeepLab v3+. 

3. Experimental settings
- Partial label setting is also important, but the weak+semi-supervised setting seems more important in this field, because it provides more diverse and strong baselines. I recommend the authors to conduct a weak+semi supervised setting experiment and compare it with strong baselines [ref1, ref2, ref3, ref4].


[ref1] Ouali, Yassine, Céline Hudelot, and Myriam Tami. "Semi-supervised semantic segmentation with cross-consistency training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[ref2] Lai, Xin, et al. "Semi-Supervised Semantic Segmentation With Directional Context-Aware Consistency." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[ref3] Zou, Yuliang, et al. "Pseudoseg: Designing pseudo labels for semantic segmentation."ICLR. 2021.

[ref4] Lee, Jungbeom, Eunji Kim, and Sungroh Yoon. "Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

4. Minor

- For background query pixels, is a single mean vector sufficient as a positive key?

- In the active negative key sampling method, once a hard class is sampled, are negative key pixels randomly selected among the pixels of that class?  
summary_of_the_review: Although this paper lacks novelty, I think that it will provide a good future direction for many researchers in this field, if it is verified that the proposed method generally works in more settings. If my concerns related to experiments are resolved, I would consider upgrading the score.
review_title: null

Meta-review: An interesting paper, with non-trivial results. The reviewers all agree that the paper is above bar (with two of them indicating strong vote for acceptance). The simplicity of the proposed approach (noted by some of the reviewers) is in my view a positive. Overall, a worthy contribution.
Decision: Accept (Poster)