Open-Vocabulary Panoptic Segmentation MaskCLIP
Authors: Zheng Ding
Jieke Wang
Zhuowen Tu
Venue: Submitted to ICLR 2023 (2023)
Keywords: open-vocabulary
panoptic segmentation
semantic segmentation
CLIP

Abstract: In this paper, we tackle an emerging computer vision task, open-vocabulary panoptic segmentation, that aims to perform panoptic segmentation (background semantic labeling + foreground instance segmentation) for arbitrary categories of text-based descriptions in inference time. We first build a baseline method by directly adopting pre-trained CLIP models without finetuning nor distillation. We then develop MaskCLIP, a Transformer-based approach with a Relative Mask Attention (RMA) module. The RMA is an encoder-only module that seamless integrates mask tokens with a pre-trained ViT CLIP model for semantic/instance segmentation and class prediction. MaskCLIP learns to efficiently and effectively utilize pre-trained dense/local CLIP features within the RMA that avoids the time-consuming student-teacher training process. We obtain encouraging results for open-vocabulary panoptic/instance segmentation and state-of-the-art results for semantic segmentation on ADE20K and PASCAL datasets. We show qualitative illustration for MaskCLIP with online custom categories.

Under review as a conference paper at ICLR 2023
OPEN- VECABLOAA PANOPTIC/UNIVERSAL SEGMEN- TATION WITH MASKCLIP
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, tackle emerging computer vision task, open-vocabulary panop- tic segmentation, that aims to perform panoptic segmentation (background se- mantic labeling foreground instance text-based descriptions in inference time. We first build a baseline method by directly adopting pre-trained CLIP models without finetuning nor distillation. We then develop MaskCLIP, a Transformer-basee approach with a MaskCLIP Vi- sual Encoder, which an encoder-only module that seamless integrates mask tokens with pre-trained VIT CLIP model for semantic/instance segmentation and class prediction. MaskCLIP learns efficiently and effectively utilize pre-trained dense/local CLIP features within the MaskCLIP Visual Encoder that avoids the time-consuming student-teacher training process. We obtain encouraging results for open-vore vocabulary panoptic/instance segmentation and state-of- the-arte art results for semantic segmentation on ADE20K and PASCAL datasets. We show qualitative illustration for MaskCLIP with online custom categories.
segmentation segmentation) for arbitrary categories of
INTRODUCTION
Panoptic segmentation (Kirillov et al. 2019b) or image parsing (Tu et al. 2003) integrates the task of semantic segmentation (Tu 20088 for background regions (e.g. "stuff" like "road", "sky"") and
instance segmentation (He et al. (017)) for foreground objects (e.g. "things" such as "person". "table"). Existing panoptic segmentation methods (Kirillov al. 2019b a et al. 2019 Xiong et al. 2019 Lazarow et al. 2020) and instance segmentation approaches (He et al. 2017) deal with a fixed set of category definitions, which are essentially represented by categorical labels without semantic relations. DEtection TRansformer (DETR) (Carion et al.) (020)) is a pioncering work that
builds a Transformer-basse architecture for both object detection and panoptic segmentation.
The deep learning field is moving rapidly towards the wooed/oord//eoooshho settings (Bendale & Boult
2015) where computer vision tasks such as classification (Radford et al. 2021), object detection
(Li al.
20226 Zareian et al. 2021 Zang al. 2022 Gu ||022 Cai et al. 2022), semantic
labeling (Li et al. 2022a Ghiasi et al. 2022), and image retrieval (Bendale & Boult
2015 Hinami &
Satoh 2018 Zareian al. 2021] Hinami & Satoh 2018, Kamath al. 2021) perform recognition
and detection for categories beyond those in the training set.
In this paper, we take the advantage of the existence of pre-trained CLIP image and text embedding models (Radford et al.) (021), that are mapped to the same space. We first build a baseline method for open-vocabulary panoptic segmentation using CLIP models without training. We then develop new algorithm, MaskCLIP, that is a Transformer based approach efficiently and effectively utilizing pre-trained dense/local CLIP features without heavy re-training. The key component of MaskCLIP is a Relative Mask Attention (RMA) module that seamlessly integrates the mask tokens with pre- trained ViT-based CLIP backbone. MaskCI.IP is distinet and advantageous compared with existing approaches in three aspects: 1) A canonical background and instance segmentation representation by the mask token representation with a unique encoder-only strategy that tightly couples a preitrained CLIP image feature encoder with the mask token encoder. 2) MaskCLIP avoids the challenging student-teacher distillation processes such as OVR-CNN (Zareian et al.) (021) and ViLD (Gu et al.) 2022) that face limited number of teacher objects to train; 3) MaskCLIP also learns to refine masks beyond simple pooling in e.g. OpenSeg (Ghiasi et al. 2022).
The contributions of work are listed follows.

---

Under review as a conference paper at ICLR 2023
We develop new algorithm, MaskCLIP, to perform open-vocabulary panoptic segmentation building on top of canonical background and instance mask representation with a cascade mask proposal and refinement process.
We device the MaskCLIP Visual Encoder under an encoder-only strategy by tightly coupling a pre-trained CLIP image feature encoder with the mask token encoder, to allow for the direct formulation of the mask feature representation for semantic/instance segmentatoon-reeieeennt and class prediction. Within the MaskCLIP Visual Encoder, there a new module called Relative Mask Attention (RMA) that performs mask refinement.
MaskCLIP expands the scope of the existing CLIP models to open-vocabulary panoptic seg- mentation by demonstrating encouraging and competitive results for open-vocabulary panoptic. instance, and semantic segmentation.
RELATED WORK
Table 1: Comparison for recent vecen-ocabulary approaches object detection. semantic segmentation. instance segmentation, and panoptic segmentation. GLIP (Li et al. (002b); OVR- CVR-CNN (Zareian et al. (02
ViLD (uu et al. 2022); RegionCLIP (Zhong et al. 2022; OV-DETR Zang et al. 2022 LSeg (Li et al. 2022ay;
OPenSeg (Ghiast al. 2022); DenseCLIP (Rao al. 2022); XPM (Huynh et al. 2022 * indicates that the
corresponding method loosely following the definition. Dense Clip features refer to the use pixel-wise/loge
features. Note that OpenSeg uses its ALIGN (Jia et al. 2021
which is an alternative to CLIP.
Task
Method
Arbitrary Online
Segmentation
Dense CLIP
Training
Annotation
Inference
semantic
instance
Teatures
data
type
GLIP OVR-CNN
FeeODDs GeMü, Cap24M
labels bbox captions
Object Det.
COCO buxe, CCMM
bbox captions lahels bbox captions
ViLD
RogionCLIP
CCIN COCO
LSeg
labels segmentations
Semantic Seg.
OpenSeg
COCO, Eeo..aoolllaSSaraie
masks + captions
DenseCLIP
com
labels segmentations
Instance Sog.
XPM
COCO, COCO.CCIM
labels + masks + captions
Panoptic Sog.
MaskCLIP (ours)
labels + masks
Open vocabulary. The open vocabulary setting is gaining increasing popularity lately as traditional fully supervised setting cannot handle unseen classes during testing, while real world vision applica- tions like scene understanding, self driving and robotics are commonly required to predict unseen classes. Previous open-vocabulary attempts have been primarily made for object detection. ViL.D Guu al. 2022) trains a student model to distill the knowledge of CLIP. RegionCI.IP (Zhong et al. 2022 finetunes the pretrained CLIP model to match the image areas with corresponding texts. OV DETR
(Zang et al. 2022) uses CLIP as an external model to obtain the query embedding from CLIP model. Recently there is also work made for open-vocabulary semantic segmentation (Ghiasi al. 2022
Panoptic segmentation. Existing panoptic segmentation (or image parsing (Tu et al. 2005g) methods (Kirillov et al. 2019b Li et al. 2019 Xiong et al. 2019 Lazarow et al. 2020) perform training and testing based on a fixed set of category labels. Open-set panoptic segmentation (Hwang et al. 202T)
is an exemplar based approach that requires categories to be known in advance, which is narrower than the open-vocabulary setting where categories of interest can be freely specified in inference.
Open- vecabblcayu panoptic segmentation: emerging task. As open-set, open- wen-world, zero-shot, and open-vocabulary are relatively new concepts that have no commonly accepted definitions, thus, different algorithms are often not directly comparable with differences in problem definition/setting.. training data, and testing scope. Table [ gives summary for the recent open-vocabulary applications. XPM (Huynh et al., 2022) utilizes vision- languaee cross modal data to generate pseudo- mask supervision to train student model for instance segmentation, and thus, it may not be fully open- vocabulary to allow for arbitrary object specifications in the inference time. LSeg (Li et al. 2022a) also has limited -pcabulcay aspect as the learned CNN image features in LSeg are not exposed to representations beyond the training labeling categories. OpenSeg (Ghiasi et al. 2022) is potentially applicable for instance/panoptic segmentation, but OpenSeg is formulated to be trained on captions which lack instance- level information that is fundamental for panoptic segmentation. The direct image feature pooling strategy in OpenSeg is potentially another limiting factor towards the open- vocabulary panoptic segmentation. Nevertheless, no results for open vecebuuuay panoptic/instance segmentation reported in (Ghiasi al, 2022).
2

---

Under review as a conference paper at ICLR 2023
CLIP model distillation/reuse. After its initial release, the CLIP model (Radford et al. 2021 that is learned from large-scale scale image-text paired captioning datasets has received a tremendous amount of attention. Some other similar vision- -asoonlanguuge models have also been proposed later e.g. ALIGN
(Jia et al., 2021), GLIP (Li et al. 2022b). Many algorithms have been developed lately (Zang et al.
2022; Wang al. 2022 Zhong al. 2022 Luo al. 2021 Patashnik et al. 202T Shen et al. 2022 trying knowledge distillation from the CLIP model to benefit the down-stream tasks one way or the other by leveraging the rich semantic language information paired in the images. Here, we directly adopt the backbone of CLIP image model to train for open- neee-vocabulary panoptic segmentation. There have been attempts (Rao al., 2022] Zhou et al., 2022) that use the dense CLIP features to represent pixel-wise feature as teacher model to train student model for semantic segmentation.
METHOD
Category Narsaz [perore deek ..)
CLIF Text Fnonder
MaskCI Visaa Encaler
Class
Maicc
Nawark
Curpe
Refined
Figure I: Illustration of the pipeline. Our pipeline contains stages. The first stage is class agnostic mask proposal network the second stage is built on the pretrained CLIP VIT model. All weights of CLIP ViT model during training are fixed. Arrows in orange denote weight sharing. The embeddings' weights of Mask Class Tokens are shared by Class Tokens in the CLJPP VIT model and are fixed. RMA represents Relative Mask Attention which is built based on the CLIP VIT attention layer. RMA contains all the weights from CLIIP ViT attention layer which all fixed during training. Additional weights are added in RMA for further mask information utilization and mask refinement. The demo image we use here is from ADE20K (Zhou et al. 2019).
Our pipeline, shown in Figure I contains two stages. The first stage is class agasosanostic mask proposal network. The second stage is MaskCLIP Visual Encoder which is built on the CLIP (Radford et al. 2021) VIT architecture. It takes the images and the coarse masks from the first stage the input and will output refined masks along with the corresponding dense image features for further classification using the text embeddings from the CLIP text encoder.
3.1 CLASS-AGNOSTIC MASK PROPOSAL NETWORK
Our Class-Agnostic Mask Proposal Network is built on instance/segmentatioon models such as MaskRCNN(He et al.) 2017) and Mask2Former(Chens et al., 2022). To make the model class- agnostic, we remove the class supervision during training. The classification head thus becomes binary classification for either positive or negative in these models.
3.2 MASKCLIP VISUAL ENCODER
Similar to CLIP, our MaskCLIP Visual Encoder also predicts the image features. Unlike the CLIP Visual Encoder that only uses on class token to output the feature of the whole image. Our MaskCLIP Visual Encoder uses another M Mask Class Tokens to output the partial/dense features for each corresponding area of the image given the masks. The Mask Class Tokens use attention masks and Relative Mask Attention obtain the partial/dense features which we discuss in the following two parts,
3.2.1 MASK CLASS TOKENS
In order to obtain dense image features for the corresponding masks bounding boxes for further recognition or distillation, an easy way to do this is simply masking or cropping the image and then sending the obtained image the pretrained image encoder. This method has been widely used in
3

---

Under review as a conference paper at ICLR 2023
several open vocabulary object detection methods (Zhong et al. 2022 Gu et al. 2022 that it's not computation efficient (N masks/boxes will lead to N images and they will be computed through the image encoder independently) and also loses the ability to the global image context information which very important for recognizing some objects and stuff. For masking, another problem is that masks in different shapes and simply masking the image will cause the resulted image to have transparent background which usually doesn't exist in real images that are used for training large language-vision models e.g. CLIP.
The problem
To solve this, we propose Mask Class Tokens for efficient feature extraction from images without los- ing the global image context information. In the original CLIP ViT-based visual encoder framework, the input of the network is N image tokens and 1 class token. The final output of the class token will be used for the relation computation with the text embeddings. Our newly introduced M Mask Class Tokens will be alongside with the image tokens and the class token. The embeddings' weights of the Mask Class Token are provided by the class token the pretrained CLIP ViT model and are fixed. Each Mask Class Token will output corresponding dense image feature similar to the class token which outputs the feature of the whole image. To achieve this, we design an attention mask as following
M F(N+1)((++))
T(N+I)XM TM
MMXX FMXI
in which M the number of Mask Class Tokens, N the number of image tokens, Tmxn is an False matrix and M' is defined as following:
m x n True matrix, Fmxn kn is
False
mask; contains at least one pixel in patch,
Ma, =
True otherwise.
where True means that this position is masked out i.e. not allowed to attend and False otherwise.
In our mask attention matrix M, F(x+T)x (N++) shows the N Image Tokens and one Class Token are attending each other as in the original CLIP. T(N+1)x shows that the N Image Tokens and one Class Token are not attending the M Mask Class Tokens. M shows that the Mask Class Tokens are attending the Image Tokens given the corresponding masks. FMx1 shows that the M Mask Class Tokens are attending the Class Token. Ta M shows that the M Mask Class Tokens are not interacting with each other.
In this way, each Mask Class Token will learn from the corresponding mask area of the images. The image tokens are also interacting with each other which means the global information won't ' lose. And it's also very efficient since we don't t need to do redundant computing for each mask or finetune the pretrained model. However, the mask information are not fully utilized and they cannot be refined either. But we will see in the experiments later that simply adpoting Mask Class Tokens to the pretrained CLIP model without any finetuning will already serve competitive baseline.
3.2.2 RELATIVE MASK ATTENTION
Irage
Tokom
Mox
Madl
f
Maci Pathh Takens
Mark Clars
Output
Tokens
Mark Class
Toker
Figure 2: Relative Mask Attention. Our Relative Mask Attention mechanism adds another attention matrix A'M,-N to original attention matrix. The newly added attention matrix is computed using the Image Tokens and the Mask Patch Tokens. The mask patch tokens are acquired by patchifying the masks using the similar way for the images as showed here. Moreover, the masks are refined by using M, in Eq. which is computed by Image Tokens and Mask Class Tokens.
4

---

Under review as a conference paper at ICLR 2023
To further utilize the mask information and refine the coarse masks, we propose Relative Mask Attention mechanism in our transformer. Our key design principle is try not to change the CLIP features directly this would destroy the learned relationship between the image features and text features in the CLIP model. Therefore, we adopt a way to only change the attention matrix in the transformer learn better linear combination of the values in the attention layers according the mask information. As in Figure our proposed Relative Mask Attention Mechanism only changes the attention matrix and refines the masks. M, is defined in Eq. 5 A.M.NN is defined in Eq. fax is the class-agnostic mask proposal network. f1 and /2 are two downsampling networks that encode the images/masks to image tokens/mask patch tokens sharing the same architecture. fr. is two-layer convolutional network that maps the attention matrix to mask residual.
Similar relative positional encoding, we use relative attention mechanism here. Let D be the dimension of the token embedding, for each Mask Class Token TMCC E ED with a corresponding mask K2 E TR H whose shape is the same as the image, we use a similar way as for the images to get mask patch tokens 7MP E MRMx in the computation of the attention. In our attention matrix, the Mask Class Tokens attending image tokens part will then be as following:
D
AM..N: (dqm MT)) + ΦK... (T
AM,-N: 0q (I ØK (T + A.M.-N
(4)
2VD
where T'M E RND is image tokens, MC E MxD is Mask Class Tokens, TP E RMXNXD is Mask Patch Tokens σQ. ỘK, &Q... ỘK,, are linear transformations, element-wise product and De (+) is the sum of the embedding dimension. Φx... (TMM E RNND will first be broadcast to RMXNx before doing element-wise production.
The attention will also in turn be used for the refinement of the masks. The vanilla attention can be seen as a relationship between each mask area and all the image patches. Thus we utilize this to help our coarse masks be more accurate. The updating process of the masks is as following:
M, = o + f-(oq(TMC) o Φx (TM)))
(5)
where Me, M, E N denotes the coarse mask and refined mask respectively, fr is a learnable non-linear function that maps the attention matrix to mask residual, and o are sigmoid and inverse sigmoid functions respectively.
4 EXPERIMENTS
In this part, we train our proposed MaskCLIP method using COCO Lin et al. 2014) training data and test other datasets (ADE20K (Zhou et al. 2019 2017), PASCAL Context (Mottaghi et al. [0014), LVIS) under the open vocabulary setting. Due to the novel setting of the open vocabulary panoptic segmentation task, we also compare our performance with previous methods. Apart from the quantitative results, we also provide qualitative results to validate our method that has good ability to learn dense image features and can support user-specified arbitrary categories.
open vocabulary semantic segmentation
4.1 DATASETS
COCO: COCO (Lin et al., 2014) includes 133 classes where classes are things and classes are stuff or background. There are 118k training images and 5k validation images. In our experiments, first train the class-agnostic mask proposal network on COCO training dataset using the annotations of panoptic masks. Then we train models on COCO training images in supervised manner.
ADE20K: ADE20K (Zhou et al. 2019 2017) contains 20,210 images and annotations for training and 2000 images and annotations for validation. It serves both panoptic segmentation and semantic segmentation. The full version (A-847) (Zhou et al.] 2019) includes 847 classes and the short version (A-)50) Zhou et al. 2017) includes 150 classes. We the validation set in ADE20K for
5

---

Under review as a conference paper at ICLR 2023
testing without any training this dataset in which we can test model's capability of open vocabulary segmentation.
PASCAL Context:
PASCAL Context (Mottaghi et al., 2014) contains 10,103 per-pixel annotations
for images of PASCAL VOC 2010 (Everingham et al.,, where 4998 for training and 5105 for validation. The full version (P-459) includes 459 classes and the short version includes 59 classes. This dataset serves as another benchmark testing our model's open vocabulary segmentation abiltiy.
LVIS: LVIS (Gupta et al. 2019) contains 100,170 images for training and 19,809 images for validation. It extends COCO (Lin et al., 2014) but contains 1,203 categories. It is considered as one of the most challenging benchmark for instance segmentation because of its large vocabulary, long-tailed distribution, and fine-graine classification. We report our model's 'ss performance of open vocabulary instance segmentation on the validation dataset.
4.2 IMPLEMENTATION DETAILS
Class-Agnostic Mask Proposal Network. In our first stage, we train a class-agnostic mask proposal network using MaskRCNN (He et al. 2017) and Mask2Former (Cheng et al.) (022) on COCO training data. The experiment setting use for MaskRCNN is R50-FPN-Ix. The backbone use in Farkerormer is ResNet-50. All the training setting follows the default in their models.
CLIP Baseline. We design our first baseline by directly using the class-agnostic mask proposal network from the first stage and the pretrained CLIP model. We mask the images according to the masks from the class-agnostic mask proposal network and send the masked images to the CLIP model to get classification results. The pretrained CLIP model we use is ViT-L/14 336px and the text inputs we use are simply the category names defined by each dataset. Those two settings keep the same with the following two methods for fair comparison.
Mask w/o RMA Baseline. Our second baseline is based on the Mask Class Tokens which doesn't 't use the Relative Mask Attention mechanism. Instead of masking the images and sending the resulted images directly to the CLIP model for feature extraction, we use Mask Class Tokens to acquire the corresponding dense image features. The obtained image features will then be used for further open vocabulary classification.
The two baselines above don't 't need any training in the second stage and can be used to directly perform the open vocabulary tasks. We will demonstrate that the second baseline better feature extraction in both quantitative results and qualitative results under the open vocabulary setting and show the effectiveness and efficiency of the proposed Mask Class Tokens.
MaskCLIP. In our MaskCLIP method, we still use the CLIP ViT-L/14 @ 336px pretrained model as with the previous two. This model has 24 attention layers and we add Relative Mask Attention in four of them which is 6, 12, 18, 24. We use AdamW (Loshchilov & Hutter 2019] our optimizer and the learning rate is set to 0.0001. We train our model on COCO training data for 10k iterations with batchsize of 8. The training takes around 3h on 8 Nvidia A5000 GPUs.
Loss Function. The loss function is L = Avehae + Adice + Ance where Lee is the loss for classification, Ldice and Live the losses for mask localization. In experiments, We set Ase 2, Adice 5, Ance = 5.
In next three parts, we evaluate our methods on open vocabulary panoptic, instance segmentation, and semantic segmentation tasks. The class-agnostic mask proposal networks we use in those methods trained using Mask2Former other than noted.
4.3 OPEN- VOCABULOY PANOPTIC SEGMENTATION
First, we compare our MaskCLIP with the two baselines on ADE20K validation set under the open vocabulary panoptic segmentation setting. The results are presented in Table 2 As can be seen from the table, the MaskCLIP w/o RMA baseline performs better on all the metrics in panoptic segmentation setting which demonstrates that our feature extraction method is better than the vanilla way in this setting. It extracts the features without the need to changing the input and can simultancously extract multiple mask area features easily. For 100 masks' feature extraction in single image, the CLIP baseline takes ~38 single 3090 GPU while the MaskCIIIP w/o RMA

---

Under review as a conference paper at ICLR 2023
baseline only takes -0.6s which is -4x faster. Our MaskCLIP beats both baselines significantly utilizes the accurate mask information and refines the masks during the feature extraction process.
Table 2: Results open-vocabulary panoptic segmentation using the ADE20K validation dataset. th and
represent thing and stuff classes respectively.
Method
PQ1
PQ f
PQ
SQ
SQ
SQ"
RQ 1
RQth 1
RQ 1
CLIP Baseline
8.207
8.473
7.675
53.124
52,661
54,048
10.534
10.883
9.835
MaskCLIP w/o RMA
9,565
8.922
10.852
62.507
62,268
62.985
12.645
11.758
14.418
MaskCLIP (MaskRCNN)
12.860
11.242
16.095
64,008
64.183
63.658
16.803
14.968
20.473
MaskCLIP
15.121
13.536
18.290
70.479
70.021
71.396
19.211
17,488
22.737
In this part, we show two sets of images to demonstrate our model capability. The first is the qualitative results on ADE20K. We compare method with the two baselines in Figure 3 It can be seen that our method performs much better than the two baselines. The results from the first column show that due to the lack of global information, CLIP baseline failes to predict the floor. Instead it predicts skyscraper. This is an easy case but if only floor area is provided it does have some similarities with the wall of skyscraper. While the MaskCLIP w/o RMA baseline and MaskCLIP model can predict the floor correctly with the the global image context information.
Imput Image
CLIP Baseline line
MaskCLIF wio RMA
MaskCLIP
Figure 3: Qualitative results on ADE20K panoptic segmentation. The images are taken from the ADE20K validation We the class names directly from the ADE20K 150 classes the text inpputs. Three images are presented here using our MaskCLIP model along with the two baselines.
The second set images we're presenting are in Figure 4 These figures show our capability of specifying any arbitrary classes in performing panoptic segmentation task. The results show that though we train a new model based on the CLIP model without any distillation methods, we can still preserve the CLIP image features very well. Our model doesn't 't have clear bias towards the base classes the training set and could tell the difference very well that have no chance to learn in the COCO training: e.g toy vs real and filled empty.

---

Under review as a conference paper at ICLR 2023
(a) "loy cabbic": "real rubbitt".
(b) "taose", "donkey"; "dky" "grass"
(c) 'emupty bottle,, "iilled bottle". "doar", "wall", 'ground"
"backgrondd
Figure 4: User-specified class panoptic segmentation. The labels above are the text inputs we used for testing the images. Texts in bold novel classes i.c. don't ton't exist in the labels of COCO training data. (a) Our model able to distinguish object properties of real rabbit and toy rabbit. potential for fine-grained classifications and not have bias toward the base classes. (c) results show that it can tell the difference between the filled status and empty status of bottles.
This example shows that model is
4.4
OPEN- VOCABULARY SEMANTIC SEGMENTATION/LABEINE
We also use our method to compare with open open-vocabulary semantic segmentation as in Table The setting is similar, they all train COCO panoptic training set and test on ADE20K validation set. On the four datasets we test, MaskCLIP reaches the state-of-the-art results on three them with only P-59 slightly lower.
Table 3: Results on open-vocabulary semantic segmentation. A-150 and A-847 represent the ADE20K dataset with 1.0 classes and 847 classes respectively. P-459 and P-59 represents PASCAL Context dataset with 459 classes and 59 classes respectively. All results use the mloU metric. All methods presented here don't use data other than COCO for training.
Method
COCO Training Data
A-150 f
A-847
P-459 f
P-59 f
ALIGN (Jia et al. 2021
None
10.7
4.1
15.7
ALIGN proposals
al. 2021
Masks
12.9
5.8
22.4
LSeg+ (Li al. 2022a
Masks Labels
18.0
3.8
7.8
46.5
OpenSeg (Giast
2022
Masks + Captions
21.1
6.3
9.0
42.1
CT.IP Baseline
Masks
13.8
5.2
3.2
25.3
MaskCLIP w/o RMA
Masks
14.9
5.6
5.3
26.1
MaskCLIP (MaskRCNN)
Masks + Labels
22.4
6.8
9.1
41.3
MaskCLIP
Masks Labels
23.7
8.2
45.9
To compare with previous methods, also provide semantic segmentation comparison in Figure Results on ALIGN++ and OpenSeg are directly from (Ghiasi et al. (2022) and we run the same image using our MaskCLIP model. be seen that due the open vocabulary setting, some similar classes may be mistakenly classified e.g. all three methods predict the house in this image while the ground truth building.
Image
+
OpenSeg
MaskCLIP
Figure 5: Comparison on open voca-uocyy semantic segmentation. The input image and the results for
ALIGN+ ++, OpenSeg are from (Ghiasi et (022).
4.5 OPEN-VOCARUTLAR INSTANCE SEGMENTATION
Cross Dataset Setting. We present the results open vocabulary instance segmentation in Table under the cross-dataset setting. Since instance segmentation can be regarded as "thing-only" panoptic segmentation, directly apply our model trained segmentation task. MaskCLIP with different class-agnostic mask proposal networks perform better than CL.IP Baseline and MaskCLIP w/o RMA in general.
panoptic dataset to the instance
8

---

Under review as a conference paper at ICLR 2023
Table 4: Results on open-vocabulary instance segmentation using the ADE20k validation dataset and the LVIS validation dateset under the cross-dataset sefting.
ADE20K
LVIS
Method
AP1 f
APo0 f
API5
AP
Ap50
API5
CLIP Baseline
3,974
6,090
4.288
4.989
7.244
5.227
MaskCLIP w/o RMA
4.263
6.696
4.402
5.762
8.202
6.169
MaskCLIP (MaskRCNN)
6.164
12.072
5.775
6,431
12.753
5.777
MaskCLIP
5.989
9.739
6.209
8,404
12.190
8.810
COCO Split Setting. Besides the cross-dataset setting, we also follow the COCO Split Setting in XPM(Huynh et al.. (202)) perform the instance segmentation in Table On the generalized setting which is more challenging setting, we outperforms previous results in base, target and all categories. On the constrained setting, we also show competitive results in both base and target categories.
Table 5: Results on open-vocabulary instance segmentation under the COCO split setting.
Constrained
Generalized
Methad
Base
Target
Base
Target
Soft- TeatTTeehhrrX Xu et al. 2021
41.8
14.8
41.5
9.6
33.2
Unbiased Sabiased-Teacherr Luu ef 2021
41.8
15.1
41.4
9.8
33.1
XPMnh et al.
2022
42.4
24.0
41.5
21.6
36.3
MaskCI.IP
42.8
23.2
42.6
21.7
37.2
5 ABLATION STUDY
Incorporating GT Masks. Since our model can decouple the mask proposal process and the classification process, we could also use the ground truth mask proposals which can be regarded as "perfect" mask proposal network in our method. In this way, we can eliminate the effects of quality of the mask proposals and inspect the method's classification capabilities. Table We can see that the performance could gain lot from the "perfect" mask proposals. And our MaskCLIP method also outperforms OpenSeg in this setting.
Table 6: Incorporating GT Masks. Results on using GT masks as mask proposals for open- vocabulary panoptic segmentation and semantic segmentation.
PQ
mloU f
OpenSeg (Ghiasi et al. 2022 MaskCLIP
21.1
15.1
23.7
OpenSeg GT masks (Ghiasi et al. 2022
27.5
MaskCLIP GT masks
35.8
31.7
Mask Refinement. In our Relative Mask Attention part, the attention layer will use the accurate mask information to learn better attention matrix and the mask will also use the attention information to gradually refine itself. this ablation study, we only let the attention matrix learn from the mask without any mask refinement. And we get the results in Table Since the SQ reflects the segemtation quality, we care more about SQ here. can be seen that MaskCLIP performs slightly better than that without the mask refinement which demonstrates the effectivity of the mask refinement.
Table Ablation Study on Mask Refinement. Results on ADE20K validation set are reported here. Both methods are trained on COCO and tested on ADE20K validation dataset.
PQ t
PQ
PO +
SQ
SQ
SQ
MaskCLIP w/o mask refinement MaskCLIP
13.624
13.253
14.368
66.361
67.715
63.653
15.121
13.536
18.290
70.479
70.021
71.396
6 CONCLUSION
In this paper, we have presented a new algorithm, MaskCLIP, to tackle emerging computer vision task, open-vocatee vocabuocayulary panoptic segmentation. MaskCLIP a Transformer-based approach using mask queries with the ViT-based CLIP backbone to efficiently and effectively utilize pre-trained dense/local CLIP features. MaskCLIP consists Relative Mask Attention (RMA) module that is seamlessly integrated with pre-trained CLIP. MaskCLIP is distinct compared with existing approaches in open- vocabbuayy semantic segmentation/objer detection by building an integrated encoder module for segmentation mask refinement and image feature extraction with pre-trained

---

Under review as a conference paper at ICLR 2023
CLIP image model. Encouraging experimental results open- vocabulay punoptiiksemaanticnnstteee segmentation have been obtained.
REFERENCES
Abhijit Bendale and Terrance Boult. Towards open world recognition. In CVPR, 2015.
Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rahul Bhotika, and Stefano Soatto. X-detr: A versatile architecture for instance-wise vision-language tasks. arXiv preprint arXiv:2204. 056666, 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, pp. 213-229, 2020.
Bowen Cheng. Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1290-1299, June 2022.
M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.pascal
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In ECCV, 2022.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. ICLR, 2022.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, pp. 5356-5364, 2019.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pp. 2961-2969, 2017.
Ryota Hinami and Shin ichi Satoh. Discriminative learning of open-vocabulary object retrieval and localization by negative phrase augmentation. In EMNLP. 2018.
Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, and Ehsan Elhamifar. Open-vocabulary instance segmentation via robust cross-modal pseudo-labeling, CVPR, 2022.
Jaedong Hwang, Seoung Wug Oh, Joon-Youn Young Lee, and Bohyung Han. Exemplar-based open-set panoptic segmentation network. In CVPR, pp. 1175- 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916, 2021.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In CVPR, pp. 1780-1790, 2021.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019a.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In CVPR, 2019b.
Justin Lazarow, Kwonjoon Lee, Kunyu Shi, and Zhuowen Tu. Learning instance occlusion for panoptic segmentation. In CVPR, 2020.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. In ICLR, 2022a.
10

---

Under review as a conference paper at ICLR 2023
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang. Lu Yuan, Lei Zhang, Jenq- Beng-Neng uae Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language- -igge pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10965-10975, June 2022h.
Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, and Xingang Wang. Attention-guided unified network for panoptic segmentation. In CVPR, 2019.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Yen-Cheng Liu, Chih- Yao Ma, Zijian He. Chia- Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. arXiv preprint arXiv:2102.09480, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2019.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end end video clip retrieval. arXiv preprint (xXiv:2004008600 2021.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2014.
Seong-Whan Whan Lee, Sanja Fidler, Raquel
Or Patashmik, Zongze Wu, Eli Shechtman, Daniel Cohen Or. and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In CVPR, pp. 2085-2094, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 9748-8766 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, 2022.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and- language tasks? In ICLR, 2022.
Zhuowen Tu, Auto-context and its application to high-level vision tasks. In CVPR, 2008.
Zhuowen Tu, Xiangrong Chen, Alan L Yuille, and Song-Chun Zhu. Image parsing: Unifying segmentation, detection, and recognition. International Journal of computer vision, 13-2):113-140, 2005.
Zhecan Wang, Noel Codella, Yen-Chur Chen, Luowei Zhou, Jianwei Yang, Xiyang Dai, Bin Xiao, Haoxuan You, Shih-Fu Chang, and Lu Yuan. Clip-td: Clip targeted distillation for vision-language tasks. arXiv preprint arXiv:2201.052 0252299 2022.
Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In CVPR, 2019.
Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end semi-supervised object detection with soft teacher. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3060-3069, 2021.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. arXiv:2203.1 /176, 2022.
Alireza Zareian, Kevin Dela Rosa. Derek Hao Hu, and Shih-Fu Chang. detection using captions. In CVPR, pp. 44393-14402. 2021.
object

---

Under review as a conference paper at ICLR 2023
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liuniann Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region- based language-image pretraining. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 16793-16803, June 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding scenes through ade20k dataset. International Journal of Computer Vision, 127(3):302-33) 321, 2019.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022.
12

---

Under review as a conference paper at ICLR 2023
A CLIP BASELINE DETAILS
Here we provide more details our CLIP Baseline. Given RGB image I R with height H and width W and list of category names with C' classes, we precompute the text embedding of the category names as & E RCDD, The mask proposal network fm outputs N masks M IR N For each mask: the cropped image region is the dlement-wise product between the binary mask M; and the image I, i.e. R, E H x3, the visual embedding V E R of the cropped region is computed by the visual encoder where D) is the hidden dimension; the final classification score V; FC is the softmax over the dot product between the visual embedding V, and the text embedding T. A formal algorithm is described as and visualization of this is shown as 6
Algorithm CLIP Baseline
Require: Mask proposal network fm. CLIP visual encoder fo, CLIP text encoder ft-
Given image I E R#x and list T containing C' category names.
= f(()).
.M Im(I.
for / = 1,2,... N do
R, MM ④ I.
V. (R,)).
V softmax(E V.).
end for
CLIP Text Encoder
Text Embed
Category Names: ('Person', 'Desk'... ...]
CLIP Visual Encoder
Class-Agnostic Mask Proposal Network
Classification Results
Figure 6: Illustration of the CLIP baseline.
B ABLATION ON USING RELATIVE MASK ATTENTION IN DIFFERENT LAYERS
In this part we conduct an ablation study on using different layers for relative mask attention. Since our pretrained CLIP model is fixed during the whole training procedure, whether each layer would help the final results remains a question. We use four different kinds of layers combination of the layers in this part and provide the results in Table g We can see that the last layer is key part of our results since the features are gradually learned through all the attention layer. Though the last four layers' features should the best, the performance wouldn't be better if Relative Mask Attention only used the last four layers. This is also reasonable since the network should not have the accurate mask information too late.
C MORE VISUALIZATION RESULTS ON ARBITRARY CATEGORIES
In this part, we provide more visualization results on user-specified class discoveries in Figure We select some very close text prompts such "four-leg animal" and "two-leg animal": "car". "truck" and "SUV" and find that our method can still classify them. We also show another result which is "person identification" in Figure (c) which shows our model preserve the dense/local CLIP features rather well.
13

---

Under review as a conference paper at ICLR 2023
Table 8: Ablation Study on Relative Mask Attention Layers in different layers. All the meth- ods are trained on COCO and tested on ADE20K validation dataset. The pretrained CLIP ViT- L/14@336p model has 24 layers and we replace four of them with our relative mask attention to fully utilize the accurate mask information and refine the masks.
Different Layers
PQ
PQ
PQ
1, 7, 13, 19 3, 9, 15, 21, 22, 23,
11.241
10.519
12.686
11.372
10.141
13.835
14.673
14.048
15,922
6, 12, 18, 24
15.121
13.536
18.290
fourleg animabackgroune
two-leg animal
"four-leg unimal"
"two-leg animal"".
"car" "truck", "SUV" "road", "skyy"
"Persom: Obama". ", "Person: Biden"
"background"
"Person: Trump"
Figure 7: More qualitative restuls on user-specified class. The labels above are the text prompts we used for testing the images. Texts in bold novel classes i.c. don't 't exist in the labels of COCO training data.
14

---

Under review as a conference paper at ICLR 2023
D QUALITATIVE RESULTS ON ADE-20K INSTANCE SEGMENTATION
We also show qualitative results on ADE20K comparing the CLIP Baseline, MaskCLIP w/o RMA and MaskCLIP. As shown in Figure MaskCLIP much better than the CLIP Baseline and MaskCLIP w/o RMA. Since the mask proposal network is class-agnostic, some masks that are not objects would actually be predicted in which case the classification later will be very important as may be classified as some object classes in the dataset. The visualization results of MaskCLIP contains fewer non-object masks and are more accurate in class prediction.
Input
(b) CLIP Baseline
(c) MaskCLIP w/o RMA
(d) MaskCLIP
Figure 8: Qualitative results on ADE20K instance segmentation.
COCO VALIDATION RESULTS
In this part, we provide the COCO validation results. The model trained on COCO panotpic training data and evaluated on COCO validation data. This results provide more comparison on model's
Table 9: Results on COCO validation dataset. Panoptic and semantic segmentation tasks are both evaluated here which is not under the open vocabulary setting since all the classes are base classes. Results of ALIGN, ALIGN++ and OpenSeg are cited from Ghiasi et al. (2022). RMA refers to the Relative Mask Attention (RMA) module shown in Figure
Panoptic PQ
Semantic mloU
PQ
POSt
ALIGN et al. (2021)
15.6
ALIGN w/ proposa Jia et al. (021)
17.9
LSeg+Li et al. (202a
55.1
OpenSeg Ghiasi et 2022
36.1 1
CLIP Baseline
17.47
23.47
8.42
23.3
MaskCLIP w/o RMA
15.32
20.12
8.07
19.5
MIskCLIP
30.89
34.78
25.02
47.6
15

---

Under review as a conference paper at ICLR 2023
F EFFICIENCY ANALYSIS.
We also provide the FLOPs for differnt models in Table TO We use an input resolution of 640x640 here. The CLIP visual encoder we ViT-L/14 0TEL/14@3360 @366xx As we can see, our MaskCLIP w/o RMA and MaskCLIP's FLOPs are much lower than the CLIP Baseline which needs to send each image to the CLIP Visual Encoder.
Table 10: FLOPs for differnt models.
Method
FLOPs
Mask2Former
79G
CLIP Visual Encoder
233G
CLIP Baseline
23382G
MaskCLIP w/o RMA
352G
MsakCLIP w/ RMA
542G
16


Reviews:
[{"review_id": "wHC2H06vaqS", "reviewer": "Reviewer_GGkA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on open-vocabulary panoptic segmentation using pretrained CLIP weights. The authors first propose a new panoptic segmentation framework that first do class-agnostic mask proposal using an existing frameworks (MaskRCNN and Mask2Former), and then use the predicted mask to interact with the CLIP visual backbone via a relative mask attention module (RMA). The final classification prediction (for any classes) is the dot-product between the refined mask features and the fixed CLIP language embedding. Experiments show the proposed framework outperform existing semantic segmentation methods on ADE20K and Pascal with a good margin (Table. 3), and the proposed mask attention module played an important role in the improvements.", "summary_of_the_review": "This paper propose a valid framework for an important and challenging problem. The author shows state-of-the-art performance on one of the benchmarks (semantic segmentation), but is not super convincing on others (panoptic and instance segmentation). There are also some concerns on the complexity of the added component. My current rating is a borderline accept (mostly for the cross-dataset evaluation setup), but might change (increase or decrease) after rebuttal.", "strength_and_weaknesses": "Strength\n\n+ The task of open-vocabulary segmentation is important, and the authors proposed a reasonable (and unified) framework for open-vocabulary segmentation panoptic, semantic, and instance.\n\n+ I appreciate that the evaluation is done in an cross-dataset setting, which can justify the generalization ability of the proposed framework.\n\n+ The proposed mask attention module gives a good improvements in Table 2, Table 3, and Table 4.\n\n+ The overall framework outperforms strong state-of-the-art semantic segmentation methods including LSeg, OpenSeg, and ALIGN as shown in Table 3.\n\nWeaknesses\n\n- While the paper aims at open-vocabulary segmentation, I feel the idea of using a mask attention module is not directly related to open-vocabulary --- it seems that this module could also improve standard segmentation, with the use of the CLIP visual encoder as an feature refiner. Please correct me if I am wrong, otherwise it would be better to also provide experiments on close-vocabulary segmentation.\n\n- The overall framework looks computationally expensive. It adds an additional ViT-L backbone from CLIP visual encoder, after an already-expensive Mask2Former module. It will be nice if the authors can discuss the FLOPs or latency for better understanding the trade-off.\n\n- Except for semantic segmentation experiments, the authors did not compare to other papers, and the numbers compared to the close-vocabulary setting is low. This makes these two settings not calibrated and less convincing. I understand that there are no/ few works reporting numbers on this specific setting, both some other comparable settings might be viable. E.g., the LVIS base/ novel split in ViLD.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_DbCKvaW0V", "reviewer": "Reviewer_MZxt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a Transformer-based approach MaskCLIP to address open-vocabulary panoptic segmentation. It uses the existing Mask2former to propose Masks of objects/stuff, and label these detected Mask regions by sending them into CLIP. Instead of sending these mask regions one by one, it aims to modify the image-encoder (ViT in this paper) of CLIP to deal with the mask regions of the same image in parallel. To achieve this, Mask Class Tokens that are copied from class token are proposed to represent corresponding mask regions, the masks proposed by mask2former is used to mask the attention matrix in cross-attention, by which each mask class token focuses on a specific mask region. Then the paper proposes a Relative Mask Attention to 1) update the attention matrix of the mask class token and 2) refine segmentation masks. It achieves good performance on open-vocabulary panoptic/semantic segmentation.", "summary_of_the_review": "Overall, the proposed Mask Class Token is interesting and useful, while the proposed Relative Mask Attention missing essential details and remains some concerns. The performance on open-vocabulary panoptic/semantic segmentation is good. I give a borderline score and am willing to raise the score if authors demonstrate more significance of the RAM.", "strength_and_weaknesses": "**Strength**:\n\n1) The idea of Mask Class Tokens is interesting and useful. Previous works usually crop the detected mask regions and then send the cropped regions to CLIP image encoder, which means the CLIP image encoder is called many times and processes the patches of the same image many times. With Mask Class Tokens, the CLIP could deal with the mask regions of the same image in parallel in a single forward. And experiments prove its efficiency.\n2) The experiments show the proposed MaskCLIP achieves good results on open-vocabulary panoptic/semantic segmentation.\n\n**Weaknesses**:\n\n1) The paper is not well presented. It is a little hard to understand the ideas, especially Eq. (1). A detailed explanation of the four parts in Eq. (1) may help the reader to better understand.\n2) Some key details are missing. What if $f_2$ in Figure 2? The text says \"$f_1$ and $f_2$ are two downsampling networks...sharing the same architecture\", does this mean $f_1$ is ViT also?\n3) If yes to the above question, so the image-encoder of CLIP is called #mask times, and the forward process time should be the same as sending mask regions to CLIP one by one (or in a batch), which is not efficient. \n4) Could you please provide the inference speed, like GFLOP, of w/ and w/o RMA, w/ and w/o Mask Class Tokens?\n5) The proposed approach is **not real open-vocabulary**. The claimed \"open-vocabulary\" is based on Mask2former training data, so the mask2former can only propose masks like in training data. For example, the is class **Person** in COCO but no class **Leg**, so is the target-of-the-interest is **Leg**, the mask2former cannot provide a mask for **Leg**, so your approach should also fail in labeling **Leg**.\n6) Your approach is actually unified to semantic/instance/panoptic segmentation, why emphasize panoptic in the title and abstract?\n7) **[Training data]** Do you use all the categories of COCO panoptic segmentation? Do you set some of them as novel categories?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IljDeqobRS", "reviewer": "Reviewer_TB5t", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on Open-Vocabulary Panoptic Segmentation where the goal is to conduct panoptic segmentation for arbitrary categories classes (open-vocabulary). A method called MaskCLIP is proposed to build on top of canonical background and instance mask representation with a cascade mask proposal and refinement process. Specifically, a Relative Mask Attention (RMA) module is presented on top of a ViT CLIP model to integrate mask tokens for semantic and instance level segmentations. Experiments are conducted on both open-vocabulary panoptic segmentation and semantic segmentation tasks including ADE20K and PASCAL datasets, which shows improvements over the baseline model or some existing methods. ", "summary_of_the_review": "The proposed method has encouraging results for Open-Vocabulary Semantic Segmentation. However, many major claims are not well supported by the current experiments. Please refer to the Strength And Weaknesses part for more details. ", "strength_and_weaknesses": "Strength:\n\n+ It is reasonable to adapte CLIP module to better take care of mask inputs by applying and learning a mask attention module. The idea makes sense to me. \n\n+ The proposed method shows improvements over the baseline model on both panoptic segmentation and semantic segmentation tasks. It also achieves better performance than some existing open-vocabulary semantic segmentation methods.  \n\n+ Ablations studies and analysis are conducted to help understand contribution of each part of the model. \n\n\nConcerns:\n\n- The key component of the proposed method is RMA module, which is a kind of general idea for adapting CLIP for Open-Vocabulary Panoptic Segmentation. There is no specific designs for panoptic segmentation. Therefore, it is not clear to me why authors emphasize panoptic segmentation where there is no existing methods for direction comparison.\n\n-  For panoptic segmentation, in addition to very basic CLIP baseline, it would be more convincing to benchmark SOTA Open-Vocabulary Segmentation method, e.g. OpenSeg or others by using the same level of supervision to enable comparisons for panoptic segmentation.\n\n-  Authors claimed efficiently for the proposed method and strength than avoids the student-teacher training process. However, evidences are not sufficient to support such claim. For example, there is no inference time comparison between the proposed method and existing method in Table 3. There is also no experiments to compare the proposed method and student-teacher training methods, e.g. [1]. Does the proposed method running faster than [1] or achieve better performance than [1]?\n\n[1] Open-vocabulary object detection via vision and language knowledge distillation, ICLR 2022\n\n- Are comparisons in Table 3 fair? the proposed method rely on both semantic and instance level annotations (panoptic segmentation) to conduct training. Is this the same case for those Open-Vocabulary Semantic Segmentation models?\n\n- Authors mention many time that the proposed method does not need finetuning. Does the proposed method perform better than finetuning CLIP on masked data? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7bowM2N8Ts", "reviewer": "Reviewer_a6DX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes MaskCLIP, an open-vocabulary segmentation solution that could naturally deal with arbitrary categories using CLIP’s text representation. The newly proposed RMA module enables MaskCLIP to effectively leverage the prior knowledge contained in the CLIP visual encoder. Thus, only trained on COCO, MaskCLIP shows competitive performance on cross-dataset validation and open-vocabulary prediction.", "summary_of_the_review": "The overall structure is clear, the writing needs further polishing with more explanations of details, and the contributions are overclaimed without sufficient support. The manuscript seems not well prepared and needs further investigation.", "strength_and_weaknesses": "**Strength**\n\nThe author builds a reasonable pipeline for open-vocabulary panoptic segmentation that could be trained in a data-efficient way.\nThe author designs RMA, which could effectively use the CLIP visual representation to support open-vocabulary recognition and mask refinement.\nThe qualitative and quantitative results are both competitive.\n\n\n**Weaknesses**\n\n**1)** Lack of experiments to support the claim\n\n**a)** The performance of COCO is not reported. Getting a good trade-off between closed and open-vocabulary settings is also important. \n\n**b)** RMA is the core contribution with a complicated structure. However, the author does not carry out ablation studies for the detailed structure of RMA. They could not prove the design of RMA is effective. \n\n**c)** Only report the experiment results for one backbone, which is not convincing.\n\n**d)** The title is panoptic segmentation, however, the experiment related to panoptic segmentation is only Table2 with 4 lines,  which is far from sufficient. The authors should at least include more naive baselines, like using different kinds of methods to fuse the CLIP representation.\n\n**2)** Lack of explanations \n\n**a)** How to convert the segmentation model to class-agnostic? If you just remove the class head, how to rank and remove duplicated masks? \n\n**b)** What kind of Prompt is used is not mentioned, which is important.\n\n**c)** The implementation of Mask Refinement is not mentioned. \n\n**d)** The loss functions are not mentioned. \n\n**e)** According to Section 4.2, the model is trained for only 10k iterations. However, the normal config on coco is 12 or 36 epochs, and Mask2former trains for 50 epochs.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject