Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper leverages CLIP for Zero-shot segmentation, which is a very hot topic currently. The authors proposed a CLIP-retrieval-based way to build gallery candidates for the semantics segmentation class, and then use dense-clip to generate reference image embedding. Then they proposed  several way to boost the performances, including language-guided co-segment and context elimination to remove the bias of background. Experiments show the proposed method achieves state-of-the-art performance.
strengths_and_weaknesses: Strength:
- Outperforming State-of-the-art performance.


Weaknesses:
- 1. The paper is not well written. The authors make it hard to understand even for some very clear concepts. 
- 2. Novelty is somehow limited. Technical contribution is not enough. It is just like to find a prototype for a class and then use it for normal clip inference. Although there are some modifications, such as context elimination, however, these are more like tricks which does not have technical depth. 
- 3. Why the used numbers in Table 2 are different for DenseCLIP [92] (Table 1 and Table 3 in original DenseCLIP [92])?


questions: See weakness
limitations: Yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a retrieve and co-segment approach that leverages a pretrained image-text model (e.g. CLIP) for unsupervised semantic segmentation. The results on existing benchmarks are good compared to other unsupervised segmentation approaches.


strengths_and_weaknesses: Strength
* Retrieve and co-segment is an intuitive and reasonable approach for unsupervised segmentation.
* Using image-text models e.g. CLIP for retrieval makes sense and is effective.
* Adaptation to target distribution by training on pseudo labels is reasonable and effective.
* Performance of ReCo+ seems better than existing unsupervised segmentation approaches at system level.

Weakness
* Compared with the unsupervised segmentation approaches, I think ReCo has a clear advantage by using CLIP which makes the approaches not directly comparable. CLIP has seen lots of image-text pairs and acquired reasonable pixel localization ability, while existing approaches such as PiCIE have no access to this kind of knowledge. In addition, if I understand correctly, ReCo has access to the category names of the target dataset while existing approaches do not.
* ReCo uses ViT-L/14 for retrieval, which is larger and stronger than the models used by existing works (e.g. ResNet18 of PiCIE). How does the performance of ReCo compare to existing unsupervised segmentation methods if we use smaller CLIP models (e.g. R50 or ViT-B/32) for retrieval and inference?
* How does the performance change if you pick more than one seed pixel per image?
* The steps to identify seed pixels (L158-168) seem highly heuristics-based. Alternatively, would clustering approaches work there?  
questions: See weakness.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses the task of zero-shot segmentation in images by leveraging powerful large-scale pretrained vision-and-language models such as CLIP. Interestingly, the proposed approach does not require costly and time-consuming pixel-wise annotations for training. Instead, it uses CLIP to select groups of relevant images that correspond to the natural language queries, based on nearest neighbors. Next, it uses pretrained visual encoders to identify seed pixels in the images that have strong support across the entire group of relevant images. These seed pixels are used to compute a reference feature for each language query to produce a segmentation attention map for each new query image, which is further refined by another segmentation mask that is computed by CLIP.
strengths_and_weaknesses: Strengths: 

1) The paper is largely well-written and easy to follow. In particular, the mathematical definitions that are provided are very helpful for understanding the proposed approach.

2) The proposed approach is theoretically sound and intuitive. While it is not entirely original due to the existence of approaches including DenseCLIP, the idea of discovering common spatial regions that occur in images containing the same concept is very interesting. More importantly, it leverages the large-scale pretrained CLIP model to retrieve related images for a language query. This allows the proposed approach to be trained on any unlabeled image sets.

3) The task of image segmentation often requires fine-grained pixel-wise annotations which is an especially costly process. Being able to leverage powerful and large-scale pretrained models to circumvent this process is especially significant. Coupled with the empirical evidence that it outperforms state-of-the-art approaches, this can be an important area of research, given the availability of increasingly larger multimodal datasets such as LAION-5B.

Weaknesses:

It would be helpful to see some qualitative visualizations of co-segmentation with seed pixels. Given that these seed pixels are used to compute a reference embedding for new query images and concepts during inference time, it seems to be a very important component of the proposed approach. It may help a reader to determine if the regions selected by the seed pixels are consistent across most images that contain a concept.
questions: 1) During inference time, are you always computing the reference feature for a given image and query concept on the fly, or do you store a dictionary of precomputed reference features for a selected list of concepts?
limitations: Yes, the authors have addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a method for zero-shot transfer in semantic segmentation. To solve this problem, it first performs a image-text retrieval by CLIP to get image archive. Then it use a pre-trained encoder to perform co-segmentation. During inference, it combines the results from reference image embedding and Dense CLIP to get the final segmentation results.
strengths_and_weaknesses: Strengths:
* The proposed pipeline which combines retrieval and co-segmentation is novel.
* It outperforms the compared methods significantly.

Weakness:
* The method is complicated and requires two encoders during inference, which slow down the speed. I want to know the comparisons in FPS when compared to other methods.
* Missing important citations: there are some concurrent works for open vocabulary semantic segmentation[1, 2], which are not cited. It would be better if related discussions are included.

[1] A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model. Arxiv.

[2] Decoupling Zero-Shot Semantic Segmentation. CVPR 2022.

questions: see weakness
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper utilizes the CLIP model for zero-shot transfer. At first, they leverage the CLIP to dynamically curate training sets from unlabelled images for arbitrary collections of concept names and leverage the robust correspondences offered by modern image representations to co-segment entities among the resulting collections. The synthetic segment collections are then employed to construct a segmentation model whose knowledge of concepts is inherited from the scalable pre-training process of CLIP. In this way, the proposed method could perform unsupervised segmentation approaches while inheriting the convenience of nameable predictions and zero-shot transfer.
strengths_and_weaknesses: strengths:

1, The paper is well written and easy to understand.
2, Leveraging the CLIP for unsupervised segmentation is interesting. 
3, The proposed training pipeline is reasonable.
3, The experiments are sufficient to show the effectiveness of the proposed method.

weaknesses:

1, The whole training pipeline seems a little complex. For example, the proposed method should utilize CLIP to filter some candidate images from numerous unlabeled data. And the identification of seed pixels includes four steps 
2, Also, the adjacency matrix A is the computation cost and is sensitive to k in the first step of the identification of seed pixels.
questions: I only have one question about the fourth step in identifying seed pixels. Could the author clarify them?
limitations: I am very curious about the potential of employing the proposed method in instance segmentation. And the whole training pipeline is a little complicated.
review_title: null

Meta-review: metareview: After author response and the discussion the paper received 1x borderline reject, 1x borderline accept, 3x weak accept [note that one reviewer mentioned the score increase only in the discussion].

The main strength are:
- Overall novel framework for zero-shot segmentation
- Strong performance
- The authors revised the paper and addressed many/most of the reviewer's concerns/suggestions in the author response.

I recommend acceptance, with the expectation
* the authors provide the additional revisions as promised
* If possible address the comment of reviewer 1QtT "what if remove Eq. (3)? It seems P^c_{new} is already good enough from Figure 2."

confidence: Certain
Decision: Accept