Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents the data collection strategy for tackling contrastive representation leaning on semantic segmentation task. The core idea is the auxiliary labels, which guide the pixel-level training sample collection across images. In addition, the authors provide a metric, i.e., PmIoU, to evaluate the quality of the generated auxiliary labels. The experimental results show that the model training guided by the auxiliary labels can be better than the previous methods.
main_review: [Strengths]    
+ The paper is well-written and easy to follow.
+ This paper is well organized and with good descriptions.
+ The mentioned literature is sufficient.
+ This model performance is good.

[Weaknesses]    
For the contributions
- Basically, the core idea is the auxiliary-label generation, which collects diverse training samples across images by leveraging K-means without annotations. However, the generation steps of feature embedding and embedding clustering are not novel compared to the previous methods, e.g., [49, 51, A].
- The metric PmIoU is designed for evaluating the quality of the auxiliary labels. However, the need for ground-truth annotations may limit its usage on datasets without annotations.

For the implementations:
- There are several implementation details described in section 3. Hence, it is not easy to realize that how many benefits are derived from the auxiliary-label generation.

For the experiments:
- It is better to discuss the relation between the superpixel size and the positive-pair partition.

Related work:  
[A] Xiao Zhang, Michael Maire: Self-Supervised Visual Representation Learning from Hierarchical Grouping. NeurIPS 2020

[Justification of rating]    
The proposed method is quite engineering yet has good results. My main concern is that the proposed method is lacking novelty. The proposed auxiliary-label generation for collecting diverse training samples across images has few highlights compared to previous methods.

[Post-rebuttal Updates] 
The author's feedback provides more experimental results to address my concerns. Indeed, the authors demonstrate the cross-image correspondences help contrastive semantic segmentation. However, the cross-image correspondences are built upon many engineering steps and may not be easy to carry out. In sum, I agree that the work is slightly above the acceptance rate for its extension experiments, and I would like to update my final rating to “6: Marginally above the acceptance threshold.”

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Contrastive learning requires positives and negatives, and their definitions have an impact on the transfer of contrastive representations to supervised tasks. This work contributes a new definition of positives for the purpose of semantic segmentation, where the positives come not only *within* an image but *across* images. The intra-image (within) and inter-image (across) positives are derived by clustering spatially-local features across a dataset. These initial clustering features can be drawn from image-wise contrastive learning, transfer from another supervised dataset, or an oracle that is trained on some or all of the target dataset. The clustering is a straightforward but effective application of k-means; the insight is in choosing the number of desired clusters. In this work an over-clustering is desired, so that clusters correspond to "auxiliary" labels that are not necessarily the same as the supervised task labels, and more fine-grained for contrasting. This difference in labeling separates auxiliary labels from pseudo-labels, which are predictions of the supervised task labels, and underlines that auxiliary labeling can be totally unsupervised unlike pseudo-labeling. A simple metric comparing auxiliary labels to ground truth segmentations is proposed ("proxy mIoU") as a surrogate of full training as a means to cross-validate auxiliary labeling design choices.

Experiments compare the proposed method with existing contrastive learning methods (DenseContrast, MoCo, PixPro) and one recent semi-supervised semantic segmentation method (PseudoSeg, ICLR'21) on the standard datasets of Cityscapes and ADE20K plus the older, smaller dataset of NYUDv2. The downstream semantic segmentation results for all methods are measured with the DeepLabV3 architecture with a ResNet-50 backbone for standardized fair comparison. The proposed method improves on the state-of-the-art, as well as the common standard of supervised ImageNet pre-training, on all three datasets vs. both the contrastive and semi-supervised methods. The margin of improvement is largest with the least supervision, but even the fully-supervised case improves by 1-2 points. Ablations examine different choices of representation for auxiliary labeling and a variety of design choices and hyperparameters for contrastive learning.
main_review: This work demonstrates a role for clustering as supervision for contrastive pre-training of semantic segmentation. It mixes supervision and self-supervision to improve semantic segmentation accuracy, and unsurprisingly results improve with more supervision. That said, it does improve on other contrastive (MoCo, PixPro) and semi-supervised (pseudo-labeling) methods while surpassing fine-tuning from initialization by ImageNet classification.

Originality:
- This is the first work to propose and examine cross-image contrastive learning of spatially-local representations, to the best of my knowledge.
- There is no discussion of nor comparison with contrastive learning driven by hierarchical segmentation (Self-Supervised Visual Representation Learning from Hierarchical Grouping by Zhang & Maire, NeurIPS'20). This related work groups positives within images, and grades their relationship by distance between regions, but does not mine positives across images, as done by the submission. This comparison would gauge how much more it helps to define positives across images.
- Could the same auxiliary label approach not be of use for contrastive learning on images? It seems like this could bridge clustering-based self-supervision (see papers by Caron et al. like SwAV) and contrastive learning. Perhaps the use of clustering in this work should be related to such kinds of self-supervision, and also hybrids like PCL (Prototypical Contrastive Learning, Ji et al. ICLR'21). Coming at this from another angle, the proposed method could be cast as the local analogue of these clustering contrastive methods, where the clustered features are local instead of global.
- The proposed proxy metric is quite simple, and not so different from metrics used for box/mask proposals such as ABO (see Selective Search by Uijlings et al.).

Quality:
- Existing contrastive methods are already superior to ImageNet pre-training, so this extension of contrastive learning cannot take the credit for that. It does however improve more still, so that certainly counts. Indeed, the results show 4-6 point gaps over the second best contrastive or semi-supervised method.
- Auxiliary labels from contrastive methods are close to oracle auxiliary labels from segmentation supervision on the target dataset; see Table 2. This is a good sanity check.
- The clustering for auxiliary labeling is done on very little data (5000 images) relative to the size of these datasets. Does this matter? The experiments do not cover this choice, nor the choice of excluding outliers by cluster distance quantiles. It seems like these could be important choices for applying the proposed method.
- CityScapes and ADE20K are common and contemporary datasets for semantic segmentation. NYUDv2 is less so, as it is smaller and older with <1000 training and testing images. However, more datasets are better than fewer, so this is fine.

Clarity:
- The illustrative figures 1-4 adequately communicate the motivation, difference with prior work, and overview of the method's steps.
- Consider naming your method, for ease of reference. Perhaps "AuxCo" for auxiliary contrastive learning, as an example.
- Which feature is used for the auxiliary loss, when an intermediate feature is also used?
- The paper asserts this work is "designed from first principles" for semantic segmentation and therefore does better. This statement is unscientifically vague. Please describe the positive designs or characteristics only present in this work and/or the negative qualities hampering other work. If these cannot be specified then drop this part.

Significance:
- Annotation for segmentation is particularly burdensome, and this work improves results across all amounts of annotation over the state-of-the-art, and shows it across a sufficient set of datasets and mostly complete set of baselines.
- Contrastive baselines that make use of clustering are notably missing, such as SwAV or PCL. Without them, the results do not indicate if any kind of clustering will do, or if the proposed method is needed to make the most of contrastive pre-training for semantic segmentation.
- The experiments could be more informative about the factors underlying the method's success. For instance, basic questions like how performance varies with 1. the number of auxiliary labels, 2. the amount of unlabeled images, and 3. the source of unlabeled data (the target data, ImageNet, another segmentation dataset, ...)  are totally unanswered.
- It is unclear if the proposed method would still be of use without access to a massive amount of unlabeled data. All of the experiments assume a larger source of unlabeled data: NYUDv2 and CityScapes include 100k+ unlabeled video frames and ADE20K is supplemented with the 10ks of images in COCO. That is, how would this do when pre-training by auxiliary label contrastive learning on the standard training set alone and then fine-tuning on a subset of the labels? Although annotation is by far the more expensive effort, data collection is nevertheless not free.

Decision:
There are certainly positives, but this work is held back by the disconnect with contrastive pre-training for segmentation (see Zhang & Maire, 2020) and clustering contrastive learning (SwAV, PCL, etc.) and the lack of analysis on what is needed to make the method work. For the rebuttal, please discuss these alternative contrastive learning schemes and the significance of the proposed method in regimes with less than ~100k images. Such regimes are relevant to medical imaging and remote sensing for example.

Other Feedback
- Figure 1: Specify if the pseudo-labels and auxiliary labels are real, and derived from particular methods, or simply illustrative. (Either is fine, but it's better to be precise).
- Figure 3: There is a right arrow to nowhere from the contrastive loss. Remove it?
- Figure 5: There are different ranges for each axis, but IoU and the proposed proxy both have ranges of [0, 1], so why not plot them accordingly?
- For multi-scale feature extraction, why not choose features from different layers instead of resizing the input images? Feature pyramids are common and effective, and doing so would reduce the computation needed.

**Post Response**: The comparison with [A] is favorable and the missing analyses were adequately addressed. This settles the issues that pointed to rejection, so this paper should be considered for acceptance.
limitations_and_societal_impact: The limitations and negatives are addressed. The method requires a lot of computation, as other contrastive learning methods do, and this is noted in the conclusion. This work does not have any specific negative impacts related to its particulars, but it does include a generic admission of potential bias.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose a a method to learn pixel-level representations based on contrastive learning. The method rely on pre-trained feature extractor (trained unsupervised, on image-level and pixel-level supervision) to generate ‘auxiliary labels’ . The auxiliary labels, computed through K-means, are used to find positive pairs at a pixel-level. The authors show that the proposed approach achieve improved results in semantic segmentation accuracy when compared to ImageNet pretraining and other contrastive approaches.

main_review: ### Originality
The idea of learning dense (pixel-level) representation in an unsupervised way have been explored before in the literature (as mentioned by authors). Moreover, the idea of utilizing clustering in latent representation to find positive samples have also been explored (as mentioned by authors). 

The difference between the proposed approach and other methods is the fact that, in this paper, the authors use a pre-trained network as feature extractor to the auxiliary labels. This poses a big issue to the model, as we can’t consider the model to be fully unsupervised if it uses a feature extractor that has been trained on labeled data.

### Quality
- The method is not fully unsupervised as the model rely on a feature extractor that has been trained with supervised data (except if considering the feature extractor trained on unsupervised data).  Therefore, it is not a fair comparison to compare the proposed approach with fully unsupervised methods (as it is done in Table 1).
- It seems that the feature extractor utilized in this method (on Table 1) was trained using supervised semantic segmentation labels. (DeepLab v3 This should be considered as the first baseline to compare the method with. The authors did not compare with this method.
- Another problem I find with this paper is the use of many different ad-hoc design choices. The performance of the method might be very dependant on many different parameters: (i) the number of clustering used, (ii) the confidence weight assigned to “type of positive pair”,  (iii) the post-processing technique, (iv) the feature extractor considered, etc.  Although some ablation study was provided, it is still very difficult to be sure where the improvement in performance really comes from.

### Clarity
The paper is in general well written and easy to follow.

### Significance
- I am not sure if any of the results shown are significant, since the proposed method rely on feature extractor that utilized labeled data (except if pretrained with MoCo).
- When training on a given dataset (eg CityScapes) and utilizing a feature extractor trained on Cityscapes as well, 
- Does the feature extractor used to train the model (DeepLabV3 trained on the dataset itself) uses the whole supervised dataset? If so, it does not make sense to mention that the segmentation model was trained “only on 10%” of the data if the feature extractor was trained on the full data.

---
### Post-rebuttal updates
The authors were able to respond to most my issues and I update my final rating to 6 (specially my issue with using labeled data from another datasets in order to train the proposed model).  I believe the manuscript is slightly above acceptance rate, but would be fine it is finally rejected. My main issue is that (i) the work requires a lot of engineering steps and ad-hoc choices in order to work and, more importantly,  (ii) it utilizes labeled data in order to train the per-pixel representations.
limitations_and_societal_impact: The authors have addressed limitations and societal impact.
review_title: null

Meta-review: 3 expert reviewers recommend acceptance but not without a little bit of hesitation.
The paper has a good experimental section and really drives home the point about the value of obtaining positives through correspondence across different scenes for contrastive learning (of segmentation in this case). The actual techniques proposed to achieve this were not the most elegant and there are some closely related papers, but overall seems like a valuable paper. 
Decision: Accept (Poster)