Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes MaskCLIP, an open-vocabulary segmentation solution that could naturally deal with arbitrary categories using CLIP’s text representation. The newly proposed RMA module enables MaskCLIP to effectively leverage the prior knowledge contained in the CLIP visual encoder. Thus, only trained on COCO, MaskCLIP shows competitive performance on cross-dataset validation and open-vocabulary prediction.
strength_and_weaknesses: **Strength**

The author builds a reasonable pipeline for open-vocabulary panoptic segmentation that could be trained in a data-efficient way.
The author designs RMA, which could effectively use the CLIP visual representation to support open-vocabulary recognition and mask refinement.
The qualitative and quantitative results are both competitive.


**Weaknesses**

**1)** Lack of experiments to support the claim

**a)** The performance of COCO is not reported. Getting a good trade-off between closed and open-vocabulary settings is also important. 

**b)** RMA is the core contribution with a complicated structure. However, the author does not carry out ablation studies for the detailed structure of RMA. They could not prove the design of RMA is effective. 

**c)** Only report the experiment results for one backbone, which is not convincing.

**d)** The title is panoptic segmentation, however, the experiment related to panoptic segmentation is only Table2 with 4 lines,  which is far from sufficient. The authors should at least include more naive baselines, like using different kinds of methods to fuse the CLIP representation.

**2)** Lack of explanations 

**a)** How to convert the segmentation model to class-agnostic? If you just remove the class head, how to rank and remove duplicated masks? 

**b)** What kind of Prompt is used is not mentioned, which is important.

**c)** The implementation of Mask Refinement is not mentioned. 

**d)** The loss functions are not mentioned. 

**e)** According to Section 4.2, the model is trained for only 10k iterations. However, the normal config on coco is 12 or 36 epochs, and Mask2former trains for 50 epochs.

clarity,_quality,_novelty_and_reproducibility: The motivation and the core idea of this paper are clear. However, it lacks many explanations for the details. 
The novelty of this paper is limited. The core component RMA seems ok, but it brings many parameters and computations. However, the author does not prove it could beat simple alternatives.

summary_of_the_review: The overall structure is clear, the writing needs further polishing with more explanations of details, and the contributions are overclaimed without sufficient support. The manuscript seems not well prepared and needs further investigation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on Open-Vocabulary Panoptic Segmentation where the goal is to conduct panoptic segmentation for arbitrary categories classes (open-vocabulary). A method called MaskCLIP is proposed to build on top of canonical background and instance mask representation with a cascade mask proposal and refinement process. Specifically, a Relative Mask Attention (RMA) module is presented on top of a ViT CLIP model to integrate mask tokens for semantic and instance level segmentations. Experiments are conducted on both open-vocabulary panoptic segmentation and semantic segmentation tasks including ADE20K and PASCAL datasets, which shows improvements over the baseline model or some existing methods. 
strength_and_weaknesses: Strength:

+ It is reasonable to adapte CLIP module to better take care of mask inputs by applying and learning a mask attention module. The idea makes sense to me. 

+ The proposed method shows improvements over the baseline model on both panoptic segmentation and semantic segmentation tasks. It also achieves better performance than some existing open-vocabulary semantic segmentation methods.  

+ Ablations studies and analysis are conducted to help understand contribution of each part of the model. 


Concerns:

- The key component of the proposed method is RMA module, which is a kind of general idea for adapting CLIP for Open-Vocabulary Panoptic Segmentation. There is no specific designs for panoptic segmentation. Therefore, it is not clear to me why authors emphasize panoptic segmentation where there is no existing methods for direction comparison.

-  For panoptic segmentation, in addition to very basic CLIP baseline, it would be more convincing to benchmark SOTA Open-Vocabulary Segmentation method, e.g. OpenSeg or others by using the same level of supervision to enable comparisons for panoptic segmentation.

-  Authors claimed efficiently for the proposed method and strength than avoids the student-teacher training process. However, evidences are not sufficient to support such claim. For example, there is no inference time comparison between the proposed method and existing method in Table 3. There is also no experiments to compare the proposed method and student-teacher training methods, e.g. [1]. Does the proposed method running faster than [1] or achieve better performance than [1]?

[1] Open-vocabulary object detection via vision and language knowledge distillation, ICLR 2022

- Are comparisons in Table 3 fair? the proposed method rely on both semantic and instance level annotations (panoptic segmentation) to conduct training. Is this the same case for those Open-Vocabulary Semantic Segmentation models?

- Authors mention many time that the proposed method does not need finetuning. Does the proposed method perform better than finetuning CLIP on masked data? 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. The source code is not provided. 
summary_of_the_review: The proposed method has encouraging results for Open-Vocabulary Semantic Segmentation. However, many major claims are not well supported by the current experiments. Please refer to the Strength And Weaknesses part for more details.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper works on open-vocabulary panoptic segmentation using pretrained CLIP weights. The authors first propose a new panoptic segmentation framework that first do class-agnostic mask proposal using an existing frameworks (MaskRCNN and Mask2Former), and then use the predicted mask to interact with the CLIP visual backbone via a relative mask attention module (RMA). The final classification prediction (for any classes) is the dot-product between the refined mask features and the fixed CLIP language embedding. Experiments show the proposed framework outperform existing semantic segmentation methods on ADE20K and Pascal with a good margin (Table. 3), and the proposed mask attention module played an important role in the improvements.
strength_and_weaknesses: Strength

+ The task of open-vocabulary segmentation is important, and the authors proposed a reasonable (and unified) framework for open-vocabulary segmentation panoptic, semantic, and instance.

+ I appreciate that the evaluation is done in an cross-dataset setting, which can justify the generalization ability of the proposed framework.

+ The proposed mask attention module gives a good improvements in Table 2, Table 3, and Table 4.

+ The overall framework outperforms strong state-of-the-art semantic segmentation methods including LSeg, OpenSeg, and ALIGN as shown in Table 3.

Weaknesses

- While the paper aims at open-vocabulary segmentation, I feel the idea of using a mask attention module is not directly related to open-vocabulary --- it seems that this module could also improve standard segmentation, with the use of the CLIP visual encoder as an feature refiner. Please correct me if I am wrong, otherwise it would be better to also provide experiments on close-vocabulary segmentation.

- The overall framework looks computationally expensive. It adds an additional ViT-L backbone from CLIP visual encoder, after an already-expensive Mask2Former module. It will be nice if the authors can discuss the FLOPs or latency for better understanding the trade-off.

- Except for semantic segmentation experiments, the authors did not compare to other papers, and the numbers compared to the close-vocabulary setting is low. This makes these two settings not calibrated and less convincing. I understand that there are no/ few works reporting numbers on this specific setting, both some other comparable settings might be viable. E.g., the LVIS base/ novel split in ViLD.


clarity,_quality,_novelty_and_reproducibility: The paper is well-written, in good quality, and novel. It's currently hard to reproduce without code due to the complexity of the system.
summary_of_the_review: This paper propose a valid framework for an important and challenging problem. The author shows state-of-the-art performance on one of the benchmarks (semantic segmentation), but is not super convincing on others (panoptic and instance segmentation). There are also some concerns on the complexity of the added component. My current rating is a borderline accept (mostly for the cross-dataset evaluation setup), but might change (increase or decrease) after rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a Transformer-based approach MaskCLIP to address open-vocabulary panoptic segmentation. It uses the existing Mask2former to propose Masks of objects/stuff, and label these detected Mask regions by sending them into CLIP. Instead of sending these mask regions one by one, it aims to modify the image-encoder (ViT in this paper) of CLIP to deal with the mask regions of the same image in parallel. To achieve this, Mask Class Tokens that are copied from class token are proposed to represent corresponding mask regions, the masks proposed by mask2former is used to mask the attention matrix in cross-attention, by which each mask class token focuses on a specific mask region. Then the paper proposes a Relative Mask Attention to 1) update the attention matrix of the mask class token and 2) refine segmentation masks. It achieves good performance on open-vocabulary panoptic/semantic segmentation.
strength_and_weaknesses: **Strength**:

1) The idea of Mask Class Tokens is interesting and useful. Previous works usually crop the detected mask regions and then send the cropped regions to CLIP image encoder, which means the CLIP image encoder is called many times and processes the patches of the same image many times. With Mask Class Tokens, the CLIP could deal with the mask regions of the same image in parallel in a single forward. And experiments prove its efficiency.
2) The experiments show the proposed MaskCLIP achieves good results on open-vocabulary panoptic/semantic segmentation.

**Weaknesses**:

1) The paper is not well presented. It is a little hard to understand the ideas, especially Eq. (1). A detailed explanation of the four parts in Eq. (1) may help the reader to better understand.
2) Some key details are missing. What if $f_2$ in Figure 2? The text says "$f_1$ and $f_2$ are two downsampling networks...sharing the same architecture", does this mean $f_1$ is ViT also?
3) If yes to the above question, so the image-encoder of CLIP is called #mask times, and the forward process time should be the same as sending mask regions to CLIP one by one (or in a batch), which is not efficient. 
4) Could you please provide the inference speed, like GFLOP, of w/ and w/o RMA, w/ and w/o Mask Class Tokens?
5) The proposed approach is **not real open-vocabulary**. The claimed "open-vocabulary" is based on Mask2former training data, so the mask2former can only propose masks like in training data. For example, the is class **Person** in COCO but no class **Leg**, so is the target-of-the-interest is **Leg**, the mask2former cannot provide a mask for **Leg**, so your approach should also fail in labeling **Leg**.
6) Your approach is actually unified to semantic/instance/panoptic segmentation, why emphasize panoptic in the title and abstract?
7) **[Training data]** Do you use all the categories of COCO panoptic segmentation? Do you set some of them as novel categories?
clarity,_quality,_novelty_and_reproducibility: **Clarity**: a little weak, some details are missing and contributions are not well presented.

**Quality**: this paper is good work, at least the Mask Class Tokens. Not sure about the Relative Mask Attention (RMA), I hope the authors could address my concerns on RMA.

**Novelty**: the idea of Mask Class Tokens is novel.

**Reproducibility**: it's easy to reproduce the idea of mask class tokens.
summary_of_the_review: Overall, the proposed Mask Class Token is interesting and useful, while the proposed Relative Mask Attention missing essential details and remains some concerns. The performance on open-vocabulary panoptic/semantic segmentation is good. I give a borderline score and am willing to raise the score if authors demonstrate more significance of the RAM.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: *Summary*: The authors propose a framework for open-vocabulary panoptic segmentation that leverages a pre-trained vision-language model (CLIP). The framework consists of three stages - (1) predicting class agnostic masks (2) refining the mask feature by RMA (relative mask attention) interactions with the CLIP visual model and (3) using the CLIP language embeddings for open-vocabulary classification. The framework is evaluated on the ADE20K, Pascal and COCO (author comments) benchmarks.

*Strengths* - (1) As also noted by two reviewers, the proposed framework is generic enough without specific design choices for panoptic segmentation. (2) The RMA module proposed in this work is an interesting and empirically sound (Table 2/3) way to use the CLIP visual encoder. This is particularly useful in combination with Mask class tokens, since compared to prior work like ViLD, it does not require separate forward passes of each crop. (3) Empirical analysis on cross-dataset performance is quite good and demonstrates that the model is quite robust. 

*Weaknesses*: (1) The method is evaluated only on panoptic segmentation benchmarks, rather than standard segmentation benchmarks. As noted earlier, the method does not have any panoptic specific ideas or design decisions. This seems to be purely an experimental design choice that is not addressed even after multiple reviewers bring it up. Lack of such experiments makes it hard to compare this work in a setting where prior open-vocabulary segmentation methods report their results. (2) The closed-world performance of this method is low, which makes it practically less useful. (3) The reviewers agree that the method is quite complex with marginal empirical gains.
justification_for_why_not_higher_score: - All the reviewers raised concerns about the complexity of the work, the choice of only evaluating on panoptic segmentation, and the weak performance on closed-world classes.
- The author rebuttal does not address these three points well.
- Even the most positive reviewer (GGkA), finally admitted that they are not willing to champion the paper and are NOT excited about the work.
Decision: Reject