Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper uses the pre-trained large model (CLIP) to transfer the language knowledge to the unseen labels for zero-shot semantic segmentation. The idea is reasonable and coherent with previous works that distill knowledge from pre-trained models. The experiments also show good results on several benchmarks in a zero-shot setting. However, the method is not novel and leads to many problems.
main_review: Strength:

- This paper proposes to use language knowledge (especially semantic similarities, I guess) to label the pixels of new classes for zero-shot semantic segmentation. The idea of distilling knowledge from pre-trained models is reasonable and becomes a new trend for many zero-shot settings.
- The experiment results seem good, which outperforms the current SOTA on a zero-shot setting and is on par with some few-shot results. 

Weaknesses:

- The method is not that novel. It just computes the semantic similarity between the language embedding from CLIP and visual embedding from DPT, and adopts cross-entropy as supervision. Such a simple design cannot address some critical issues in zero-shot semantic segmentation, as mentioned in Failure cases in Sec. 5.1. The authors notice the issues but leave them to the readers.

- The result comparison in the experiment is not fair. This work uses a stronger backbone (ViT) in the image encoder than the previous works (RN-101). Is there any way to get a fair comparison?
summary_of_the_review: I like the idea of utilizing pre-trained models to transfer the knowledge for zero-shot semantic segmentation, and I think it would be the main trend for many other zero-shot settings. However, this paper only adopts a naive method to solve the problem, leaving some critical issues to the readers. So my initial rating is borderline.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes Language driven Semantic segmentation (LSeg) for semantic segmentation. Essentially, LSeg embeds text labels and image pixels into a common space, and assigns the closest label to each pixel. LSeg is flexible and can dynamically handle arbitrary label sets on the fly with varying length, content, and order. The paper demonstrates that LSeg achieves comparable performance as state of the art few-shot semantic segmentation networks on FSD-1000 even though it is used in zero-shot setting. When a fixed label set is used based on the data set (all labels in the training set), it also matches the accuracy of traditional segmentation algorithms on ADE-20k. LSeg uses text encoder from CLIP-ViT-B/32 which is frozen during training while the weights of the image decoder (DPT with a ViT-L/16) is updated to maximize the correlation between the text embedding and the image pixel embedding of the ground truth class of the pixel. Spatial regularization is applied at the end that also up samples the predictions to the original input resolution. 
main_review: The biggest strength of the proposed approach is its simplicity and its flexibility since it can dynamically handle arbitrary label sets on the fly with varying length, content, and order. 

Weaknesses:
1.	While the performance on FSD-1000 and ADE-20k are impressive, it is compared against just 1-shot. Also, the gap is larger in Pascal-5i and COCO-20i with less mIoU and FB-IoU for LSeg.
2.	The strongest baseline uses ResNet-101 which has much fewer parameters (~45M) than the backbone used by LSeg (307M). So, it does not seem like a fair comparison.


summary_of_the_review: Experiments section is weak. Most of the tables contain much weaker and older models for comparison (even VGG16). The biggest concern is that ViT-L has 307M parameters while the largest backbone used in comparisons is ResNet-101 which has about 45M parameters. ViT-B has 86M which is still larger than ResNet-101 but will at least be a fairer comparison that using ViT-L for image decoder.  

However, the flexibility of the approach is powerful. Lseg can dynamically handle arbitrary label sets on the fly with varying length, content, and order.

Hence the recommended rating is “marginally above” the acceptance threshold.

=====

The reviewer thanks the authors for their response. Glad to see that LSeg still shows improvements when using ResNet101 as backbone. Updated the rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is about semantic image segmentation, the template weights of each category is generated by language text.
main_review: This method is similar to image classification of CLIP on ImageNet-1k, where the template weights of each class derived from language text, and also called zero-shot learning. As a comparison, this paper v.s. CLIP can be treated as FCN v.s. VGG. I have some questions about this paper
1. In Figure 2, F_{H1} the orange cube is a typo? Should be F_{1W}? 
2. In Figure 2, the size is also not correct. The input image H*W*C, the feature should not be that size as stated in the paragraph of bottom of Page 3. 
3. In Figure 2, what does C stands for? Input image have C channel, I think C=3 for RGB image. Then Why Text embedding also C channel?
4. In Figure 2, what is the real method to compute training loss? In Equation (2) it computes a global average of softmax_{y_{ij}}(F_{ij})? What does this expression mean? Besides I think an explicit notation of F_{ij} = {f_{ij1}, …f_{ijK}} is necessary. 
5. In Table 1, Table 2, Table 3, LSeg with ResNet101 is missing. In Table 6 it said Text Encoder is RN50x16. I think this kind of statement is very strange and an explanation in caption is absolutely necessary.

summary_of_the_review: The clarity of presentation needs improvement. More ablation study necessary.


==========================================

I have read the revised version and still think it is not the final version. For example

1 In Equation(2), what does y_ij means? An image has several labels therefore y_ij is a vector? In this formula it seems it is a scalar. 

2 In the revision paper, I still cannot find the training loss. softmax is not a loss objective. A objective function should be something like

                                                                                         min f(x)

3 In comparison, although they are both ResNet-101, but ResNet-101 from CLIP is abosolutely better than other ResNet-101.

===================================

I have read the response and now think the comparison is OK.
From Figure 2 I think every pixel has more than one label. But in Equation (2) it seems y_ij is a scalar.
review_title: null

Meta-review: The paper presents an approach to semantic segmentation based on text embedding of class labels. This enables zero-shot semantic segmentation with class labels that were not seen during training. I appreciate the new ablation against a ResNet-101 backbone. I don't find the similarity with CLIP substantial, and I recommend that the paper is accepted.
Decision: Accept (Poster)