Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work focuses on the semi-supervised semantic segmentation. To solve the large intra-class variation problem, this work attempts to regularize the distribution of within-class features. The proposed methods is mainly based on the student-teacher framework with a well-designed prototype-based predictor and a widely-used linear predictor. It encourages the consistency between the linear predictor and prototype-based predictor to regularize the distribution of within-class features. Extensive experiments have been conducted to validate the effectiveness of this method. The proposed method achieves promising performance on different datasets and settings.
strengths_and_weaknesses: Strengths:
1. The idea is simple and clear.
2. Comprehensive experiments have conducted to validate the improvements of the proposed methods.


Weakness:
1. The motivation and the principle are not clear. Why do you use the prototype-learning predictor? What are the inherent differences between the linear predictor and the prototype-based predictor? Why does this work? Line 124-130 only demonstrates the differences on the practice. It is better to analyze or provide more reasons behind the regularization.
2. It is better to provide more details about the prototype initialization and updating, which are confused now. What does the learned prototype represent? How to perform clustering on them to find out internal sub-classes? What is the definition of the distance between the  pixel  and the prototypes?
3. Why are the improvements on the blender setting of the PASCAL VOC 2012 incremental?
4. It is better to perform the ablation studies about the CutMix. What is the performance about using other strong augmentations?
questions: 1. Please demonstrate more details about how the regularization between prototype-based predictor and linear predictor works.
2. Please provide more details about the prototype initialization and updating.
3. Please conduct ablative experiments about the CutMix.
limitations: This work discussed the limitation on the model architectures. 
Maybe it is necessary to consider the potential data distribution unbalance problem.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper is about semi-supervised semantic image segmentation. A moment distillation method is used and the authors proposed to use pixels only with confidence larger than 0.8 as supervision.
Experiments are done on Pascal VOC12 and Cityscapes.

strengths_and_weaknesses: [Strengths]

1. The experimental results seems pretty good from Table 1 and Table 3.  
2. The combination of cutmix and distillation is interesting.


[Weakness]

1. Ablation study of the choice of confidence threshold is absent.
2. Some details are missing. For example, L189 'in a fully-supervised way for several epochs', what the exactly hyper parameters are used?
3. Eevey
questions: 1. I thinks an Algorithm describing the global process is necessary. Otherwise it is difficult to reproduce the experiment results.

2. Please explain “EMA update” in Figure 1.

3. Is Prototype similar to assign more than one template in classifier layers? For example, if there are 21 category, the classifier layer is set to  512x48.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a novel approach to regularize the distribution of intra-class features to ease label propagation difficulty. In particular, the proposed method adopts a standard linear predictor and a prototype-based predictor and encourages the consistency between predictions from two predictors. Experimental results validate the effectiveness of the proposed method.
strengths_and_weaknesses: Strength:

The paper is easy to follow. The proposed method is effective. Experimental results are promising.

Weakness:

(1) Limited novelty. From my point of view, the paper only proposes a consistency regularization method on the prediction head. The contribution is quite limited.

(2) Prototype generation. How to deal with false prediction on unlabeled images?

(3) Experimental results. The results of previous methods on Cityscapes dataset seem strange to me. They are different from the original results of the papers. 

(4) Feature visualization. Although intra-class compactness is improved, inter-class discrimination is weakened. Will this have bad influence on segmentation?
questions: Please see weakness above.
limitations: Yes, the authors have addressed the limitations and potential negative social impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a consistency based scheme for semi-supervised semantic segmentation, where they employ a student-teacher framework and enforce consistency between predictions made by a prototypical classifier and a linear classifier. By selecting multiple prototypes for each class, it is hypothesized that the limitation of intra-class distances can be addressed. 
strengths_and_weaknesses: -- Strengths

- The idea of using complementary training signals from linear classifier and prototypical classifier to enforce consistency is interesting.
- Apart from a few required clarifications (see below), the paper is well written and easy to follow, and the intuitions and concepts have been explained well. 
- The performance on reported datasets is extremely strong. I am indeed surprised that a method as simple as this results in such strong performance and large improvements. 

-- Weaknesses

- While the abstract and the intuition is built upon the fact that this method reduces "intra-class" discrepancies, it has not been explained or demonstrated that this is indeed a limitation in prior methods.  It has also not been shown that this method better alleviates this issue compared to prior methods. For example, is there a concrete metric that helps us understand that existing methods really perform poorly on intra-class differences, while your method does better? The tSNE plots are inconclusive (the highlighted cluster already looks compact in first two plots as well, and the cluster for label "4" still forms two distinct clusters using your method, same as "supervised-only"). 

- There are really two parts to this framework - one is FixMatch style teacher-student model with weak-strong data augmentation, while another is the proposed consistency loss. Each of these have to evaluated separately to better understand the benefits. Currently, ablations are only present on the consistency part. 

-- Minor
- Authors are encouraged to use the published versions in citations instead of the arxiv-only versions in an updated version.


questions: - Overall, while I appreciate the simplicity and effectiveness of the method, it is not clear why this method has to work. What part of your pipeline is the most effective and makes the most difference compared to prior work? This analysis is not present in the paper. I am willing to raise my rating if this is cleared.

- What is done at test-time? If I understand correctly, you use the student model along with the classifier based predictions at test-time? How does the performance vary if we use the prototype based prediction instead?

- How is the teacher model initialized and updated?

- From fig2, any reason why 3-4 prototypes should work better instead of just one? Wouldn't the intra-class separation be least when all features are clustered around one center rather than distributed around 3-4 centers?

- Are the prototypes computed in the feature space? If so, I believe that the feature space for segmentation models is of lesser resolution - how do you get pixel level labels at this reduced resolution to compute class wise prototypes?

- It would also be interesting to visualize the prototypes themselves along with the features in the same space, and know what is actually being learnt by the prototypes and the classifier. I am more curious to know how they are different. Assume a case when all prototypes belonging to a class match the classifier weight vector for that class. All the conditions would be satisfied, but the consistency criterion would be trivial in that case.
limitations: The societal impact has been addressed. The limitations section could be better. For example, the benefits from your method seem limited if more unlabeled data is available. Do you have an intuition why?
review_title: null

Meta-review: metareview: This paper proposes a teacher-student scheme for semi-supervised semantic segmentation. A consistency regularization is setup between a prototypical classifier and a linear classifier and different augmentation degrees (weak vs. strong) are applied to the teacher and student networks. On the positive side, the reviewers have found the ideas in this paper simple and strong in practice and they have indicated that the proposed setting is interesting. While the novelty of this paper may seem incremental since consistency regularization, in general, is heavily explored in semi-supervised training, the proposed setting is new for the semantic segmentation problem. One of the main criticisms of this submission is that it consists of many moving parts that are not well motivated and how they are orchestrated during training is missing from the original submission. After careful discussion, I believe that the merits of this submission outweigh the issues, and I am happy to recommend this paper for acceptance. 

Last but not least, I strongly recommend the authors bring the algorithms to the main (if possible), provide additional implementation details, and make their code publicly available. 
confidence: Less certain
Decision: Accept