Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
Authors: Qihang Yu
Ju He
Xueqing Deng
Xiaohui Shen
Liang-Chieh Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: open-vocabulary panoptic segmentation
panoptic segmentation
vision and language
TL;DR: A simple single-stage open-vocabulary panoptic segmentation framework that advances state-of-the-art performance while running 7.5x and 6.6x faster for training and testing

Abstract: Open-vocabulary segmentation is a challenging task requiring segmenting and recognizing objects from an open set of categories in diverse environments. One way to address this challenge is to leverage multi-modal models, such as CLIP, to provide image and text features in a shared embedding space, which effectively bridges the gap between closed-vocabulary and open-vocabulary recognition.
Hence, existing methods often adopt a two-stage framework to tackle the problem, where the inputs first go through a mask generator and then through the CLIP model along with the predicted masks. This process involves extracting features from raw images multiple times, which can be ineffective and inefficient. By contrast, we propose to build everything into a single-stage framework using a _shared **F**rozen **C**onvolutional **CLIP** backbone_, which not only significantly simplifies the current two-stage pipeline, but also remarkably yields a better accuracy-cost trade-off. The resulting single-stage system, called FC-CLIP, benefits from the following observations: the _frozen_ CLIP backbone maintains the ability of open-vocabulary classification and can also serve as a strong mask generator, and the _convolutional_ CLIP generalizes well to a larger input resolution than the one used during contrastive image-text pretraining. Surprisingly, FC-CLIP advances state-of-the-art results on various benchmarks, while running practically fast. Specifically, when training on COCO panoptic data only and testing in a zero-shot manner, FC-CLIP achieve 26.8 PQ, 16.8 AP, and 34.1 mIoU on ADE20K, 18.2 PQ, 27.9 mIoU on Mapillary Vistas, 44.0 PQ, 26.8 AP, 56.2 mIoU on Cityscapes, outperforming the prior art under the same setting by +4.2 PQ, +2.4 AP, +4.2 mIoU on ADE20K, +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes, respectively. Additionally, the training and testing time of FC-CLIP is 7.5x and 6.6x significantly faster than the same prior art, while using 5.9x fewer total model parameters. Meanwhile, FC-CLIP also sets a new state-of-the-art performance across various open-vocabulary semantic segmentation datasets. Code and models are available at https://github.com/bytedance/fc-clip

Convolutions Die Hard: Open-Vocabulan Segmentation with Single Frozen Convolutional CLIP
Qihang Yu', Ju He², Xueqing Deng', Xiaohui Shen', Liang- Chieh Chen' "ByteDance
The Johns Hopkins University
Abstract
Open-vocabulary segmentation is challenging task requiring segmenting and recognizing objects from an open set of categories in diverse environments. One way address this challenge is leverage multi-modal models, such CL.IP, to provide image and text features in a shared embedding space, which effec- tively bridges the gap between closed-vocabulary and open-vocabulary recognition. Hence, existing methods often adopt a two-stage framework to tackle the problem, where the inputs first go through mask generator and then through the CLIP model along with the predicted masks. This process involves extracting features from raw images multiple times, which can be ineffective and inefficient. By contrast, we propose to build everything into a single-stage framework using shared Frozen Convolutional CLIP backbone, which not only significantly simplifies the current two-stage pipeline, but also remarkably yields better accuracy-cost trade-off. The resulting single-stage system, called FC-CLIP, benefits from the following observations: the frozen CLIP backbone maintains the ability of open-vocabulary classification and can also serve as a strong mask generator, and the convolutional CLIP generalizes well larger input resolution than the one used during con- trastive image-text pretraining. Surprisingly, FC-CLIP advances state-of-the-art results on various benchmarks, while running practically fast. Specifically, when training on COCO panoptic data only and testing in zero-shot manner, FC-CLIP achieve 26.8 PQ, 16.8 AP, and 34.1 mloU ADE20K, 18.2 PQ, 27.9 mIoU on Mapillary Vistas, 44.0 PQ, 26.8 AP, 56.2 mloU on Cityscapes, outperforming the prior art under the same setting by +4.2 PQ, +2.4 AP, +4.2 mloU ADE20K, +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes, respectively. Addi- tionally, the training and testing time of FC-CLIP is 7.5x and 6.6x significantly faster than the same prior art, while using 5.9x fewer total model parameters. Meanwhile, FC-CLIP also
new state-of-the-art performance across various open-vocabulary semantic segmentation datasets. Code and models are available at https://gi wtw.ttttt....bb//tteeeeeteet htt.h//...comyytytdddee///------
Introduction
44 is a complex computer vision task that aims predict of non- overlapping masks, each with its corresponding class label. It combines the tasks of semantic segmentation [37 and instance seementation 34 making challenging problem solve. Many
Panoptic segmentation
methods [43
87
18 83
51
93
20
94. (3) have been proposed to tackle this problem, and
significant progress has been made in terms of panoptic quality (PQ). However, due to the high cost of annotating such fine-grained dataset [4 22], the number of semantic classes is typically limited few dozens or hundreds. This restriction hinders the further application of existing approaches to real-world settings, where the number of possible semantic classes is unlimited.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

VIT-L14 @224pH
VIT--14 @448pk
VIF-L14 6672pM
VIT-L-4 G606GN
VITA-14 @0120pb
ComNeXtL 2224pM
C @448pp
ConvNoX:LL 6677pp
@896px
LonNNeXi @1120px
Figure I: k-means visualization on top of frozen CLIP backbone features w.r.t different input resolutions. Both ViT-based and CNN-based CLIP produces semantic- meeaniii-- features. However, when scaling up the input resolutions. we note that ViT-based CLIP features turn noisier, while CNN-based ones are smoother and generalize better. The smoother feature map is preferable for mask-pooling modules in our design.
To overcome the limitations of closed- vocabulary segmentation, vreabuocayylary segmentation 48
90 29 25
has been proposed. These approaches uses text embeddings of category names [97]. represented in natural language, as label embeddings, instead of learning them from the training dataset. By doing so, models can classify objects from wider vocabulary, which improves their ability to handle a broader range of categories. To ensure that meaningful embeddings are provided, a pretrained text encoder (23 70 57 69 is typically used. This encoder can effectively capture the semantic meaning of words and phrases, which is critical for open-vocabulary segmentation.
Multi-modal models, such as CLIP 69 and ALIGN 40 have shown promise for open-vocabulary segmentation due to their ability to learn aligned image-text feature representations from large-scale Internet data 744.. SimBaseline [90] and OVSeg [52) are two recent methods that use a two-stage
framework to adapt CLIP for open- vocabulaay segmentation. In these methods, images are first processed by a heavy mask generator [36 20 to obtain mask proposals, and then each masked image crop is generated and fed into frozen CLIP model for classification. MaskCLIP [25] extends this approach apen-vocabulay
panoptic segmentation, but additionally leverages mask proposals as attention masks in the CLIP backbone to efficiently avoid multiple forwarding processes for the masked crops. More recently, ODISE [89 employs a stable diffusion UNet [72 71 as a frozen backbone for mask generator, which significantly boosts the state-of-the oftte-of-the-att performance. However, despite these advances, they still rely on a two-stage framework, where the mask generator and CLIP classifier extract features from raw images separately, resulting in inefficiency and ineffectiveness.
natural question thus arises as to whether is possible to unify the mask generator and CLIP classifier into single-slage framework for open-vocabulary seymentation. Sharing the feature extractor between them straightforward solution, but poses two challenges. First, fine-tuning CLIP backbone can disrupt the alignment between image and text features, resulting in much worse performance on vocaaul-y categories. Existing methods [90] 52 25 89 rely on another separate backbone for mask generator, increasing model size and computational costs. Second, CLIP models are typically pretrained on relatively lower-resolution inputs, while dense prediction tasks require much higher resolution for optimal performance. This makes it difficult to directly apply CLIP -rett-prt backbones to downstream dense prediction tasks, particularly ViT-based CLIP models (26). where careful treatments required (e.g., side adapter [17 91], or cost aggregation [oo 21).. Consequently, existing methods [25] 89 perform mask segmentation and CLIP classification different input scales, leading sub-optimal performance.
To alleviate the two challenges, we propose to build both mask generator and CL.IP classifier on top of shared Frozen Convolutional CLIP backbone, resulting in single-stage framework FC-CLIP. Its

---

design is based on the following observations. The frozen CLIP backbone ensures that the pretrained image-text feature alignment intact, allowing -uc-b--vocabulary classification. can also serve as a strong mask generator by appending a lightweight pixel decoder and mask decoder 220 94]. The convolutional CLIP, based on a Convolutional Neural Network (CNN) 447. empirically shows a better generalization ability compared to ViT-based CLIP (6). when the input size scales up. This echoes the success of fully convolutional networks [60] in dense prediction tasks. Both observations are critical for developing single-stage framework, but they have been overlooked and undiscovered by existing two-stage pipelines [25] 89]. In Fig. | we visualize the learned visual representation of
ViT-based and CNN-based CLIP via k- m-mans clustering [59]. As shown in the figure, the features learned by LNNNbased CLIP are more robust across different input sizes.
Surprisingly, the adoption of a single frozen convolutional CLIP as the shared feature extractor results in an extremely simple yet effective design. Specifically, the single-stage FC-CLIP consists of three modules built upon a shared frozen convolutional CLIP backbone: a class-agnostic mask generator, an in- on-vocabulary classifier, and an out-of- vocabuaaryy classifier (see Fig. for comparison between pipelines). The proposed method not only enjoys simple design, but also comes with very low cost for both training and testing. As a comparison, our model has only 238M frozen parameters and 21M trainable parameters, against the state-of-the-art work ODISE [89] that has 1494M frozen and 28M trainable parameters. Furthermore, our model training only takes 25.6 V100 GPU days, which model also runs 6.6x faster. Although FC-CLIP enjoys a simple design, still outperforms previous methods across multiple datasets. Trained on COCO panoptic dataset only, FC-CLIP surpasses prior sfate-of-the- art ODISE 89 significantly in a zero-shot manner. Specifically, FC-CLIP achieves 26.8 PQ (+3.44, 18.2 PQ (+4,0), and 44.0 PQ (+20. on ADE20K, Mapillary Vistas, and Cityscapes, respectively.
is 7.5x faster compared to ODISE's s 192 V100 GPU days. During inference,
As panoptic segmentation unifies semantic and instance segmentation, FC-CLIP naturally extends to open-vocabulary semantic and instance segmentation. With the same model trained on COCO panoptic data only (i.e., no task-specific fine-tuning). FC-CLIP achieves state-of-the-a performance on open-vocabulary instance and semantic segmentation. Specifically, FC-CLIP achieves 16.8 AP on ADE20K, surpassing the ofatc-of-art ODISE [89] by +2.4. FC-CLIP also outperforms the state-of-art specialized open- vecabbocabulary semantic segmentation model SAN [9] by +1.1 and +1.1 mloU on the challenging ADE20K- -47220K8477 (A-847) and PASCAL-Contex--455 (PC-459) benchmarks, respectively.
In summary, through the lens of careful re-design of existing two-stage open- neen-vocaylaary segmen- tation models, we establish a simple, strong, and fast baseline for the community. The proposed FC-CLIP adopts single-stage framework by exploiting shared frozen convolutional CLIP, which not only advances the state-of-the-ar performances on multiple benchmarks, but also enjoys a practi- cally fast training and inference speed. We hope our study will inspire future research on efficient single-stage open-vocabulary segmentation models.
2 Related Work
Vision-language models target at encoding vision and language jointly in a fusion model. Early extract visual representations by pretrained object detectors and fine-tune on downstream tasks with language supervision. Recently, with the breakthrough of large language models 23 rapid progress has been made in this field. CLIP (69 and ALIGN 40 demonstrate that pretraining dual-encoder models with contrastive objectives on large-scale noisy image-text pairs can learn representation with cross-modal alignment ability and show strong performance in zero-shot downstream tasks. The following works [95] 92) further confirm these points and achieve impressive results in zero-shot transfer learning such as open- opea-voccaylary image recognition.
works 78
16. 98
Closed-vocabulary segmentation can be divided into three types according to the semantics of the grouping pixels, i.e. semantic, instance and panoptic segmentation. Semantic segmentation interprets high-level category semantic concepts. Prior works [9] 72 10
[3 28 96, 86 99. 30 mainly treat this task as a per-pixel classification problem and build their models on top of the idea of FCN (60]. Instance segmentation groups foreground pixels into different object instances. Starting from Mask R-
11
(66 mainly address this task with mask classification, where a set of bounding boxes and binary masks are predicted. Panoptic segmentation seeks for holistic scene understanding including both stuff and things. The pioneering work 44 and prevalent
CNN 36), prior works [22 56 12
80 84
ones (55 43 87
18
50 82
67] decompose the problem into various proxy tasks and merge the
3

---

two staget
naive single-stages high-resoluttion image
FC-CLIP (ours):
low-resolution image
high-resoluttion image
high-resolution image
VIT based CLIP
backbone
A CLIP
CNN-based CLIP
A mask
classifier & mask generator
classifier & mask generator
classifier
classifier
generator
in-vocab class
out-vacab
class
mask
class
mask
mask
class
frozen
trainable
Figure 2: Comparisons between open-vocabulary panoptic segmentation pipelines. Left: Existing methods [25] [9] adopt a two-stage pipeline, where the first stage employs a high-resolution image to generate class-agnostic masks, and the second stage feeds both the fow-resolution image and predicted masks to a frozen CLIP backbone for open-vocabulary recognition. This incurs heavy computation, as image features extracted multiple times. Middle: A naive single-stage framework builds everything together and fine-tunes the CLIP backbone, breaking the pretrained alignment between images and texts. Right: Our single-stage framework FC-CLIP employs a shared frozen convolutional CLIP, where "frozen CLIP" maintains the open-vocabulary recognition and can serve strong mask generator, and "convolutional CLIP" generalizes well to large input sizes. Note that the predicted masks are used for CLIP recognition in all three schemes (not shown for simplicity).
results in the end. Recently, following DETR most works K present end-to-end solutions based on the idea of mask classification. Standing on their shoulden our proposed method builds top of the pixel decoder and mask decoder of Former 20 by additionally exploiting the open- vocabuulay recognition ability from CLIP (69].
76
19
39
49
Open- vocabulary segmentation aims segmenting arbitrary classes including those that can not be 88)
accessed during the training procedure. Priors works [48
29 90 52
24
[oT ( 104 62 102 [2]
perform apen-vocabulary semantic segmentation through leveraging large pretrained dision-languag
models 669 40 71. Recently, MaskCLIP (25) presents two-stage pipeline, which consists of a
class-agnostic mask generator and frozen CLIP 69 encoder for cross modal alignment, and thus
expands the scope of the CLIP models into open-vocabulary panoptic segmentation. ODISE [89] digs out the innate potential of pretrained text-image diffusion models 771 in terms of the ability to present open concepts in the representation space for performing strong open- open-vocabulary panoptic segmentation. FreeSeg (68] encodes multi-granularity concepts into a compact textural abstraction, enabling generalizability to arbitrary text description. Unlike those methods, we propose single- stage framework by exploiting single frozen convolutional CLIP backbone, resulting in a simpler, faster, and stronger model than existing works.
We also note that the pioneering work F-VLM [46] builds open- vocabulary detection framework top of frozen CLIP backbone. However, FC-CLIP differs from it with a totally different observation and motivation. Specifically, our work was initially motivated by the state-of-art open-vocabulary segmentation model ODISE [89], which found that the CLIP backbone extracts noisier features than diffusion models (Figure B. 1. in (89), leading to inferior segmentation results (which justifies their adoption of diffusion models). Their observation motivated us to look deeply into the problem. Interestingly, our discoveries show that both ViT-based (used by ODISE [89] and CNN-based CLIP can produce semantic-meam meaningful features. However, when scaling up the input resolutions, we discover that ViT-based CLIP features turn noisier, while CNN-based ones are smoother and generalize better across input sizes. F-VLM [6) also empirically found that frozen CLIP can provide meaningful features for object detection. However, they did not choose CNN-based CLIP on purpose and thus did not compare carefully between ViT-based and CNNN-asee CLIP backbones. the other hand, in our paper, we have provided careful ablation studies on ViT-based and CNN- based CLIP, where we observe that even though both ViT-based and CNN-based CLIP initially have comparable performance at resolution 224, CNN-based CLIP shows better and more robust performance when input resolution scales up.

---

image
prediction
mask decoder
CNN-
00-0 mask prediction
pixel decoder
based CLIP
mask pooling
class embeddings
geometric ensemble
B embeddings
frozen
class prediction
in-vocab branch
mask
out-vocab branch
pooling
class embeddings
Figure Overview of FC-CLIP, which contains three main components: mask generator, an in- vocabuualry (in-vocab) classifier, and an out-of- -ut-ol-vocabulary (out-vocab) classifier. All components build on top shared frozen covolutional CLIP backbone. The pixel decoder and mask decoder follow the design of Fask2Former, and generate class-agnostic masks. The in- in-vocabulary classifier yields the class embeddings by mask mask-pooling over final pixel features from pixel decoder. During testing, FC-CLIP additionally exploits the out-of-vocabulary classifier by mask-pooling over frozen CLIP backbone features, and the final class prediction is obtained by geometric ensembling both classifiers. Note that the text embeddings are obtained by feeding category names into CLIP text encoder, which are done beforehand and cached memory, thus causing no additional costs. Also, the class agnossic mask proposals are fed to the mask pooling modules (not shown for simplicity).
3 Method
In this section, we first define the problem of open-vocabulary segmentation. We then introduce the existing two-stage pipeline, followed by our proposed single-stage framework FC-CLIP.
Problem Definition Open-vocabulary segmentation aims to segment the image I e "xxx33 into of masks with associated semantic labels:
{ = {(m,, G))
(1)
The K ground truth masks m, {0, 13 HX contain the corresponding ground truth class label c. During training, fixed set of class labels Cerain is used, while during inference, another set of categories Ctest is used. In the open-vocabulary setting, Ctest may contain novel categories unseen during training, i.e., Ctrain + Ctest- We follow previous works (2) and assume the availability of the category names of Ctest (represented natural language) during testing.
adopt a two-stage pipeline for open-vocabulary segmentation. The first stage contains class-agnostic mask generator M with parameters 0. that generates set of N mask proposals {m.) E ig. XXX given the
Two-Stage Open- Vocabulary Segmentation Existing works 90 52 25 89
input image
(mi 1 M(IOMM)
(2)
In the second stage, a CLIP adapter P takes both image and mask proposals {m, as inputs, where the latter input is used guide the frozen CLIP model CLIP* (" denotes frozen). The adapter performs mask classification through forwarding processes with either masked crops [90] 52 or
masked attention 25 89]:
N P(I, ( CLIP").
(3)
where ( E NXICI refers to the predicted class probabilities for the N predicted masks, Ce (Ctrain, )test} depending on training or testing phase, and |C] is the category size.
Although this framework has achieved impressive gpen-vocay segmentation performance, it has two limitations. First, the image features are extracted twice, once for mask generation and the other for mask classification. The double feature extractions incur heavy computation, making it costly to scale up backbone parameters. Second, the mask generator often requires high-resolution inputs (e.g., 1024 x 1024), whereas the CLIP model is usually pretrained with lower-resolution images (e.g..
5

---

224 x 224). The two-stage pipeline thus needs to feed high- reghuuesolttion images into the mask generator and low-resolution images into the CLIP classifier, making the model inefficient.
Naïve Single-Stage Open-Vone Vocabulary Segmentation To avoid increasing the model size and computational cost of duplicate feature extractions, one may naïvely formulate everything together into a single-stage framework F. where both mask generator and mask classifier share the same CLIP-pretrained backbone C'LIP (not frozen) for extracting features from an input image
Lm,,C,. = F(I CLIP, BMr)
However, we empirically discover that fine-tuning this naive single-stage framework causes a mis- alignment between image and text features in the pretrained CLIP model, leading to sub-optimal performance, especially for novel unseen classes. It also increases the training costs by 2.1x to 52.8 GPU days. Interestingly, our experiments also show that a frozen CLIIP backbone can pro- vide sufficient features for mask generation, while preserving the image-text aligned representation. Nevertheless, we still face another challenge, where CLIP models are usually pretrained on low- resolution images (c.g., 224 x 224), whereas segmentation models prefer higher-resolution inputs (e.g., 800 x 1333 for COCO, or 1024 x 2048 for Cityscapes). This discrepancy results in the signifi- cant performance degradation, when applying a frozen CLIP on large input images. Digging into the details, we found that it is related to the popular VIT (26] backbone used in CLIP that does not transfer well to different input sizes, which could be alleviated by extra careful designs (e.g., side adapter [17 [9] or cost aggregation [101] 211). On the other hand, CNN-based CLIP models (such as ResNet [35 and ConvNeXt (58) exhibit better generalization ability to different input sizes, due to their fully convolutional nature 66). Additionally, the CNN-based CLIP backhone, extracting multi-scale feature maps, can be used as a simple plug-in module into modern vooabul-cc segmentation models (20) 194. Motivated by the observations, we thus propose FC-CLIP, simple yet effective single-stage open-vocabulary segmentation framework built entirely on single frozen convolutional CLIP backbone CLLIPNN
(ms ] = F(;; CICCIIP CENIPENN,,MM 0m)
(S)
The proposed FC-CLIP leverages the semantic features of frozen CNse CLIP backbone for both mask generation and CLIP classification. Unlike previous works 90 52 25| 89] which often train a separate mask generator and ignore the potential reuse of CLIP's s semantic features, we incorporate the CNN-based CLIP backbone into the ofate-of-the-ar segmentation method Mask2Former [20]. We note that FC-CLIP is a general meta-architecture that can build on top of several modern segmentation methods (20 144. Our approach offers several advantages. By freezing and sharing the backbone features, our model is significantly more efficient during both training and testing (i.e., avoiding feature duplication). The CNsebased CLIP backbone not only transfers well to different input resolutions (from its pretrained image size), but also generates multi-scale feature maps, seamlessly compatible with modern segmentation methods 20 94), At a high level, FC-CLIP consists of three components: class-agnostic mask generator, in-vocabulary classifier, and vocabuulyy classifier. We detail each component below.
FC-CLIP
Class-Agnostic Mask Generator Following Mask2Former 200, we use a pixel decoder enhanced with multi-scale deformable attention [103] to improve the features extracted from the frozen CNN- based CLIP backbone. The enhanced pixel features, together with a set of object queries 7 83 are then passed through a series of mask decoders, where each consists of masked cross-attention 20
self-attention [81]. and a forward network. The resulting segmentation logits are obtained by performing a matrix multiplication between the object query and pixel features. The predicted masks are matched with ground-truth masks in a one-to-one manner through Hungarian matching [45] and are supervised accordingly. Moreover, as the number of object queries is often greater than the number of labeled masks, only subset of predicted masks are optimized through this matching process. We apply no penalty to the remaining unmatched proposals, which ensures that more mask proposals are obtained.
In- nn-Vobbullayy Classifier Once the mask proposals predicted, they are classified with category text embedding in a contrastive manner, where the class embeddings for each mask and category text embeddings are projected into common embedding space. That is, the predicted class probability by in-vocabulary classifier is defined as follows: Vi w I,...,N
G = softmar( [cos(Vi, t1), cos(vi, t2), ..., cos(vi, kci)))
(6)

---

where T is learnable temperature parameter with initialization of 0.07 to control the sharpness of the distribution, cos is cosine distance measurement. v; the class embeddings for i-th predicted mask, which is obtained by mask pooling over the final pixel features from pixel decoder, similar to [29]. t; the category name's s text embeddings of class j. which is obtained by feeding the category name to text encoder. Note that these category text embeddings only need to be generated once. They are then kept in memory to serve as text classifiers, and thus it incurs negligible additional cost during training. This forms our in- vocabbuuylrry classifier.
Vorabularry Classifier During inference, however, we notice that using the in- in-vocabulary classifier alone fails to generalize to completely novel unseen classes, as the model is only trained on finite set of categories and thus could not recognize diverse novel concepts. To address this issue. we introduce an out-of-vocabulaa classifier, which applies mask pooling to the frozen CLIP backbone features, aiming to borrow the pretrained (intact) open-vocabulary recognition ability from CLIP. Unlike the other two-stage methods [90] 52, 25 [9]. where one or multiple forward processes of CLIP are needed, the adopted out-of- -ocabuualy classifier introduces marginal additional costs, since the backbone features already extracted (and only lightweight
is performed). The predicted class probability by tittof--occbuarry classifier čj, is then obtained in a manner similar to Eq. 6 by replacing V; with the mask-pooled features over frozen CLIP backbone features. This classifier strictly maintains the original CLIP feature distribution, allowing us to better recognize brand new categories. Note that the out-of-vocabulatt classifier is only performed during testing.
Combining In- and Out-of-Vocabullat Vocabulary Classifiers Following prior works employ geometric ensemble to fuse the classification scores between in-vocabulary and out-of- vocabulary classifiers. That is, Vj = 1,...:ICI
29 46 89), we
4 ( (2,sn ()) (i)) jf E Ctrasn (c (1)) (G),ou(()) otherwise
where (5) denotes the j-th element of 2, and the underscripts in and out refer to in-vocabulary and out-of-vocabule classifier, respectively. o E [0. 1) balance the predictions between in- and out-of-vocabulary classifiers for seen and novel unseen categories.
Experimental Results
Herein, we provide implementation details of FC-CLIP in Sec. 4.1 After setting the stage, we introduce our main results, compared with state-of-the-art methods and ablations studies in Sec. 4.2
4.1 Implementation Details
pretrained on LAION-2B (74 dataset. On top of the CLIP backbone, we build the mask generator, following Mask2Former (20] Nine mask decoders are employed to generate the class-agnostic masks by taking as inputs the enhanced pixel features and set of object queries. For in- in-ooaauulary classification, following (29.. the class embeddings are obtained by mask-pooling the pixel features from the pixel decoder's 's final output. Afterwards. the classification logits (before softmax) is obtained by matrix multiplication between the predicted class embeddings and categories' text embeddings.
Architecture We use ConyNeXt-Large CLIP 58 69 backbones from OpenCLIP 38
Training Strategy We follow (20 and adopt the same training recipe and losses without any special design. The training is optimized with AdamW [41 61) optimizer and weight decay 0.05. We use crop size of 1024 x 1024. We employ the learning rate 1 x 10-4 and multi-step decay schedule. The training batch size is 16, and the model is trained for 50 epochs on COCO panoptic training set
Inference Strategy During inference, the shorted side of input images will be resized to 800 while ensuring longer side not exceeds 1333. For Cityscapes and Mapillary Vistas, we increase the shorter side size to 1024. We adopt mask-wise merging scheme [20] for the mask predictions. The voccablrv classifier is only performed during inference by mask-pooling over the frozen CLIP backbone features. The final classification results then obtained by geometrie ensembling
in- and out-of-vocabulary classifiers [3 46 89 as in Eq. 77. where we default a = 0.4 and
7

---

Table I: Open-vocabulary panoptic segmentation performance ADE20K. The proposed FC-
CLIP demonstrates better performances than prior arts, while using much fewer frozen parameters. We provide more results in the supplementary material
zero-shot test dataset
training dataset
params (M)
ADE20K
COCO
method
frozen
trainable
PQ
AP
mloU
PQ
AP
mloU
MaskCLIP [25]
304
15.1
23.7
FreeSeg 68
16.3
6.5
24.6
ODISE 89
1494
28
22.6
14.4
29.9
55.4
46.0
65.2
ODISE
89
(caption)
1494
23.4
13.9
28.7
45.6
38.4
52.4
FC-CLIP (ours)
200
21
26.8
16.8
34.1
54.4
44.6
63.7
Table 2: Open-vocabulary panoptic segmentation performance on street-view datasets. The proposed FC-CLIP demonstrates better transferability to street-view dataset
zero-shot test dataset
Mapillary Vistas
Cityscapes
method
PQ
SQ
RQ
mloU
PQ
SQ
RQ
AP
mloU
ODISE [89]
14.2
61.0
17.2
23.9
75.3
29.0
FC-CLIP (ours)
18.2
57.7
22.9
27.9
44.0
75.4
53.6
26.8
56.2
3 = 0.8. Following prior arts, we also adopt prompt engineering from 22 89 and prompt templates If not specified, FC-CLIP is only trained on COCO panoptic dataset
from 33
52]
(4 Following
prior works [29
89 we zero-shot evaluate the model on ADE20K [100]. Cityscapes [22] and
Mapillary Vistas
(64 for open- open-vocabulary panoptic segmentation. We also report open-vocabulay
semantic segmentation results on those datasets along with PASCAL datasets [27] 63 The panoptic
segmentation results are evaluated with the panoptic quality (PQ) 44 Average Precision (AP), and
mean petereectionovvr-uniii (mloU), and semantic segmentation is evaluated with mloU [27]. Note
that all results are obtained with the same single checkpoint trained on COCO panoptic data only.
4.2 Results
We summarize the main results for open- vocabulayyulary panoptic segmentation and semantic segmentation in Tab. [T. Tab. and Tab. 3 where train FC-CLIP on COCO train set with panoptic annotation and evaluate it on various datasets in a zero-shot manner.
veca-Vocayylary Panoptic Segmentation Evaluation on ADE20K In Tab. 1 we compare our FC-CLIP with other state-of-the-art methods on ADE20K [00), the main test-bed of zero-shot open-vocabulary panoptic segmentation. As shown in the table, method achieves significantly better performance compared to MaskCLIP (25), with +11.7 PQ. +00.8 AP and +10.4 mloU, even though use fewer frozen (-66M) and trainable (-42M) parameters. When compared to the concurrent methods FreeSeg (68 and ODISE [89) the advantage of FC-CLIP persists. FC-CLIP is +10.5 PQ, +10.3 AP, and +9.5 mloU better than FreeSeg without using COCO-Stufi annotations [S] (which contains more semantic classes than COCO- Panoppaan....). Our PQ, AP, mloU score are also +4.2, +2.4, +4.2 higher than ODISE under the same training settings. Compared to ODISE with caption [5 for supervision, our model still outperforms it by +3.4 PQ, setting a new state-of-the-art record. Meanwhile, it is noticeable that model has 6.3x (5.9x) significantly fewer frozen (total) parameters compared to ODISE, which utilizes strong large backbone from stable diffusion 71 for feature extraction.
Open-Vone voca-Vocyyylrry Panoptic Segmentation Evaluation on Street-Viey Datasets In Tab. 2 we evaluate on Cityscapes and Mapillary Vistas, which focus street driving scenes. Compared to ofte-oof-the-a method ODISE, FC-CLIP achieves better performances on both datasets. Specifically, it outperforms ODISE by +4.0 PQ and +20.1 PQ Mapillary Vistas and Cityscapes, respectively. Notably, FC-CLIP has a slightly lower SQ, which indicates our mask generator is actually weaker than the one in ODISE, which utilizes much larger backbone.
Vocabuacyyy Semantic Segmentation Evaluation Although our model was trained on COCO panoptic data only, it also performs well on open vocabulay semantic segmentation. In Tab. 3

---

Table 3: Open-vocabulary semantic segmentation performance. The proposed FC-CLIP also
demonstrates state-of-the-ar performances on open-vocabulary semantic segmentation
mloU
method
training dataset
A-847 PC-459 A-150
PC-59
PAS-21 21 PAS-20
SPNet 185
Pascal VOC
127
24.3
18.3
7S3NNet [
Pascal VOC
19.4
38.3
LSeg [88
Pascal VOC 27
47.4
GrT 88
GCC [75 YFCC [79
4.3
4.9
10.6
25.9
50.7
52.3
SimBaseline [90]
COCO Stuff [5]
15.3
74.5
ZegFormer 24
COCO Stuff S
16.4
73.3
LSeg+ 48
29
COCO Stuff
3.8
7.8
18.0
46.5
OVSeg 52
COCO Stuff
12.4
29.6
55.7
94.5
SAN 9
COCO Stuff
13.7
17.1
33.3
60.2
95.5
OpenSeg 29
COCO Panoptic COCO Caption
6.3
9.0
21.1
42.1
ODISE (caption)
COCO Panoptic + COCO Caption
11.0
13.8
28.7
55.3
82.7
MIs
COCO Panoptic
8.2
10.0
23.7
45.9
ODISE 89
COCO Panoptic
11.1
14.5
29.9
S7.3
84.6
FC-CLIP (ours)
COCO Panoptic
14.8
18.2
34.1
58.4
81,8
95.4
Table 4: FPS comparison. All results are obtained with one V100 GPU, CUDA 11.6 and PyTorh 1.13, by taking the average runtime on the entire validation set, including post-processing time
method
ADE20K
COCO
ODISE [9)
0.41
0.39
FC-CLIP (ours)
2.71 (6.61x) 2.76 (7.08x)
report our model's performance various benchmarks against other open-vocabulary segmentation models, where FC-CLIP shows an overall superior performance. Specifically, with the same training annotations used, FC-CLIP outperforms MaskCLIP by +6.6, +8.2, +10.4, +12.5 mloU across A-847, PC-459, A-150, and PC-59, respectively. Compared to methods with caption annotations, FC-CLIP persists its advantages, where it outperforms ODISE (caption) by +3.8, +4.4, +5.4. +3.1 mloU across datasets A-847, PC-459, A-150, PC-59 respectively. Against other open-vocabulary semantic segmentation methods, model maintains its advantages across different datasets, despite being trained solely with panoptic annotations. Furthermore, it demonstrates comparable performance to saate-of-the-art open-vocabulary semantic segmentation methods, which utilize the COCO -ttfff dataset as their training set. The COCO- -ooof- dataset comprises 171 classes, 38 more classes than COCO-Panoptic and offers highly desirable annotations for semantic segmentation tasks. It is worth mentioning that these methods build their approach on top of ViT-L (with extra designs (91). resulting in significantly larger model size compared to our deployed ConvNeXt-L. (304M vs. 198M). Despite the disparity in model size. FC-CLIP remains competitive in terms of performance. Specifially, FC-CLIP outperforms state-of-the-un art open- vocabulary semantic segmentation method SAN 19 by 1.1 and 1.1 mIoU on the challenging A-847 and PC-459 datasets.
Inference Speed We provide comparison of FPS (frames per second) in Tab. The proposed FC-CLIP not only demonstrates superior performances, but also enjoys a significant fast inference time: FC-CLIP runs 6.61 and 7.08x faster than ODISE evaluated on ADE20K and COCO datasets, respectively.
Training on ADE20K and Evaluating on COCO We further validate the effectiveness of FC-CLIP by using a different training datasct. Specifically, we follow [68 89 to train our model on ADE20K dataset with panoptic annotation, and evaluate it COCO panoptic dataset. As shown in Tab. 5 FC-CLIP outperforms FreeSeg (68] by +0055 PQ. and ODISE [8] by +2.0 PQ on COCO dataset. Notably, our model actually has a lower SQ (-1.4) compared to ODISE, which utilizes a much larger backbone and thus has a stronger mask generator. Nevertheless, FC-CLIP still outperforms ODISE significantly with simple yet effective design.
Fine-tuning CLIP Backbone Harms Performance on Novel Vocabularies We validate the necessity of freezing CLIP backbone to ensure a better generalization to novel vocabularies. We compare the performance of trainable CL.P variant and frozen CLIP variant in Fig. where we use the same mask proposals to ensure fair comparison. Specifically, compare the performance

---

Table 5: Results training on ADE20K panoptic and evaluating on COCO panoptic val The proposed FC-CLIP performs better than prior arts, even in the different setting (i.e., trained on ADE20K and zero-shot evaluated on COCO)
zero-shot test dataset
training dataset
COCO
ADE20K
method
PQ
SQ
RQ
PQ
SQ
RQ
FreeSeg 68
16.5
72.0
21.6
ODISE
89
25.0
79.4
30.4
31.4
77.9
36.9
FC-CLIP (ours)
27.0
78.0
32.9
41.9
78.2
50.2
Tranable CLP
Frusen CLIP
bualiin
arsie martine
ansristtnd
slager
Jani
Figure 4: Trainable CLIP vs. Frozen CLIP, with per-class PQ analysis. We show common classes (labeled in green) shared by COCO and ADE20K, and 10 novel classes (labeled in red) that are only ADE20K. The frozen CLIP demonstrates much better recognition ability for novel classes, while performing similarly for the seen classes.
10 seen classes, which are shared by both COCO and ADE20K (e.g., person, sky), and 10 unseen classes, which are only included in ADE20K dataset (e.g., arcade machine, dishwasher). As shown in the figure, tuning CLIP backbone leads to worse performance on unseen concepts, which breaks the CLIP feature alignment and thus loses its recognition ability much wider vocabulary.
Conclusion
In this work, we have presented FC-CLIP, a simple yet effective single-stage framework for open- vocabulary segmentation. FC-CLIP shows great potential by building everything top of shared frozen convolutional CLIP backbone, which not only significantly reduces training and testing costs, but also establishes strong baseline on multiple benchmarks. Our study demonstrates how to better adapt a pretrained CLIP model for downstream dense prediction tasks, which we hope will shed the light on unleashing CLIP's potential for other various downstream tasks.
L Limitations FC-CLIP presents simple single-stage open- open-vocabulary segmentation framework with state-of-the-arr performance. We note that there exist some interesting research topics to be explored in the near future, such as better unleashing CLIP's potential in both mask segmentation and classification, how to deal with conflict or overlapping vocabularies (e.g., cat vs. cat head), etc.
Broader Impact FC-CLIP shows great potential for segmenting and naming every object in the scene, which could facilitate many applications including intelligent home assistants, robots, self- driving, etc. Yet it relies on CLIP model pre- pre-trained on the Internet data that may be biased, which calls for future research for calibration to avoid misuse.
10

---

References
[1] lean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In NeurlPS, 2022.
[2) Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jac Lec, Yolact: Real-time instance segmentation. In ICCV, 2019.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
[4] Maxime Bucher, Tuan-Hung Hun-Huug Vu, Matthicu Cord, and Patrick Pérez. Zero-shot semantic segmentation. In NeurIPS, 2019.
[5] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
[6] Zhaowei Cai and Nuno Vasconcelos. Cascade r-onn: Delving into high quality object detection. In CVPR, 2018.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[8] Kai Chen, Jiangmiao Pang, Jiaqi Wang. Xiong, Xiaoxiao Li, Shuyang Sun. Wansen Feng. Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In CVPR, 2019.
[9] Liang-Chior Chen, George Papandrcou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuillc. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR, 2015.
[10] Liang- Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, fully connected erfs. IEEE TPAMI, 2017.
[11] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolu- tion for semantic image segmentation. arXiv:1706.05580 055587 2017.
[12) Liang- Lang-Chich Chen, Alexander Hermans, George Papandreou, Florian Schroff, Peng Wang, and Hartwig Adam. Masklab: Instance segmentation by refining object detection with semantic and direction features. CVPR, 2018.
[13] Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[14] Liang-Chich Chen, Huiyu Wang, and Siyuan Qiao. Scaling wide residual networks for panoptic segmen- tation. arXiv:2011.11675 2020.
[15] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitncck. Microsoft coco captions: Data collection and evaluation server. arXiv: 1504.00335 2015.
[16] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation lcarning. In ECCV, 2020.
[17] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2023.
[18] Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu. Thomas S Huang. Hartwig Adam, and Liang-Chich Chen. Ceeoptic-Deepp.att A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation. In CVPR, 2020.
[19] Bowen Cheng, Alexander Schwing, and Alexander Kirillov. Per-pixel elassification not all you need for semantic segmentation. In NewrIPS, 2021.
[20] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, 2022.

---

[21] Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee. Anurag Arnab, Paul Hong- suck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. arXiv:2303. 2023.
[22] Marius Cordts, Mohamed Omran, Schastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Be- nenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. CVPR, 2016.
[23] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, In NAACL, 2019.
[24] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In CVPR, 2022.
[25]
Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskelip. In ICML, 2023.
[26] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[27] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJC,, 88:303-333 338, 2010.
[28] Jun Fu, Jing Liu, Haijic Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019.
[29] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In ECCV, 2022.
[30] Jiaqi Gu, Hyoukjun Kwon, Dilin Wang, Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, and David Z Pan. Multi-scale high-resolution vision transformer for semantic segmentation. In CVPR, 2022.
[31] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022.
[32] Xiuye Gu, Yin Cui, Jonathan Huang, Abdullah Rashwan, Xuan Yang. Xingyi Zhou, Golnaz Ghiasi, Weicheng Kuo, Huizhong Chen, Liang-Chieh multi-dataset multi-task segmentation model. NewrIPS, 2023.
Chieh Chen, and David A Ross. Dataseg: Taming a universal
[33] Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang. Jan Kautz, and Derek Hoiem. Contrastive learning for weakly supervised phrase grounding. In ECCV, 2020.
[34] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation, In ECCV, 2014.
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[36] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[37] Xuming He, Richard Zemel, and Miguel Á Carreira-Perpiñãan. Multiscale conditional random fields for image labeling. CVPR, 2004.
[38] Gabriel Ilharco. Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021. URL httpee////// org/10, 5281 202/////eoo.... 5143773
[39] Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In CVPR, 2023.
[40] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe Lc, Hun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and languae representation learning with noisy text supervision. JCML, 2021.
[41] Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[42] Alexander Kirillov, Evgeny Levinkov. Bjoern Andres, Bogdan Savchynskyy, and Carsten Rother. In- stancecut: from edges to instances with multicut. In CVPR, 2017.
12

---

[43] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR. 2019.
[44] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019.
[45] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83997, 1955.
[46] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vim: Open-vocabulary object detection upon frozen vision and language models. In ICLR, 2023.
[47] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2) 23244, 1998.
[48] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladien Koltun, and René Ranftl. Language-drivon semantic segmentation. In ICLR, 2022.
[49] Feng Li, Hao Zhang, Shilong Liu, Lei Zhang, Lionel Ni, Heung-Yeun Yeung Shum, et al. Mask dino: Towards unified transformer- based framework for object detection and segmentation. In CVPR, 2023.
[50] Qizhu Li, Xiaojuan Qi, and Philip HS Torr. Unifying training and inference for panoptic segmentation. In CVPR, 2020.
[51] Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Tong Lu, and Ping Luo. Panoptic segformer. In CVPR, 2022.
[52] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, 2023.
[53] James Liang, Tianfei Zhou, Dongfang Liu, and Wenguan Wang. segmentation. In ICML. 2023.
Clustseg: Clustering for universal
[54] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[55] Huanyu Liu, Chao Peng, Changqian Yu, Jingho Wang, Xu Liu, Gang Yu, and Wei Jiang. An end-to-end network for panoptic segmentation. In CVPR, 2019.
[56] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In CVPR, 2018
[57] Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoee, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv:1907.11692 2019.
[58] Zhuang Liu, Hanzi Mao, Chao-Yusa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. convnct for the 2020s. In CVPR, 2022.
[59] Stuart Lloyd. Least squares quantization pem. IEEE transactions information theory, 1892)112--337, 1982.
[60] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmen- tation. In CVPR, 2015.
[61] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[62] Chaofan Ma, Yuhuan Yang, Yanfeng Wang, Ya Zhang, and Weidi Xie. Open-vocabulay, semantic segmentation with frozen vision-language models. In BMVC, 2022.
[63] Roozbeh Mottaghi, Xianjic Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lec, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014.
[64] Gerhard Neuhold. Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017.
[65] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In ECCV, 2020.
13

---

[66] Siyuan Qiao, Liang-Chies Chen, and Alan Yuille. Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. In CVPR, 2021.
[67] Siyuan Qiao, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Vip-deeplab: Learning visual perception with depth-aware video panoptic segmentation. In CVPR, 2021.
[68] Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang. Shilei Wen, Xin Pan, et al. Freeseg: Unified, universal and open-vocabulary image segmentation. In CVPR, 2023.
[69] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. ICML, 2021.
[70] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits transfer learning with a unified text-to-text transformer. TMLR, 21(1):5485-555 5551, 2020.
[71] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[72] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[73] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 115:21 1-252, 2015.
[74] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
[75] Piyush Sharma, Nan Ding, Schastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2566-2565, 2018.
[76] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.
[77] Shuyang Sun. Weijun Wang, Qihang Yu, Andrew Howard, Philip Torr, and Liang-Chice Chen. Remax: Relaxing for better training on efficient panoptic segmentation. NewrIPS, 2023.
[78] Hao Tan and Mohit Bansal. Lxmert: In EMNLP, 2019.
Learning cross-modality encoder representations from transformers.
[79] Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59 (2):64-73, 2016.
[80] Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In ECCV, 2020.
[81] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
[82] Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chich DeepLab: Stand-Alone Axial-Altention for Panoptic Segmentation. In ECCV, 2020.
Chich Chen. Axial-
[83] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Max-deeplah: End-to-end panoptic segmentation with mask transformers. In CVPR, 2021.
[84] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. SOLOv2: Dynamic and fast instance segmentation. In NeurlPS, 2020.
[85] Yonggin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynop Akata. Semantic projection network for zero-and few-label semantic segmentation. In CVPR, 2019.
[86] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NewrIPS, 2021.
14

---

[87] Yuwen Xiong, Renjie Liao. Hengshuang Zhao. Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In CVPR, 2019.
[88] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR. 2022.
[89] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with lext-to-image diffusion models. In CVPR, 2023.
[90] Mengde Xu, 7heng Zhang. Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model. In ECCV, 2022.
[91] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. In CVPR, 2023.
[92] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Scyedhosscini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. TMLR, 2022.
[93] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, 2022.
[94] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chied Chen. k-means Mask Transformer. ECCV, 2022.
[95] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang. Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv:2111.11 1.l.:1211.1142, 2021.
[96] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-comtextual representations for semantic segmenta- tion. In FCCV, 2020.
[97] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, 2021.
[98] Pengehuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, I Lijuan Wang. Yejin Choi, and Jianfeng Gao. VinvI: Revisiting visual representations in vision-language models. In CVPR, 2021.
[99] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequeecee perspective with transformers. In CVPR, 2021.
[100] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[101] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022.
[102] Ziqin Zhou, Yinjic Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In CVPR, 2023.
[103] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
[104] Xueyan Zou, Zi- Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Jianfeng Wang. Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jac Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language. In CVPR, 2023.
15

---

Appendix In the following supplementary materials, we present additional experimental results pertaining to the design of FC-CLIP. Our supplementary analysis also includes comparisons against other methods that specifically address open-vocabulary semantic segmentation, ensemble methods, and hyperparameter tuning. Furthermore, we provide quantitative comparison between ViT-based CLIP and CNN-based CLIP across varying input sizes, along with additional visualizations and comprehensive dataset details.
6 Additional Experimental Results
Fine-tuning or Freezing CLIP Backbone in FC-CLIP In this study, we provide a comprehensive analysis of the impact of fine-tuning or freezing the CLIP backbone in our framework. We specifically focus on the PQ and PQunsee metries, which evaluate the performance for classes that overlap and do not overlap between the training and testing datasets, respectively. To determine whether a class is seen or unseen, adopt the prompt engineering technique described in [29], which provides synonyms or subcategories of classes. Specifically, if any category name test dataset overlaps with category name in training dataset, we consider it seen class; otherwise unseen. As discussed in the main paper, the proposed FC-CLIP contains three components: a class-agnostic mask classifier. We thus explore using
generator, an in-vocabulary classifier, and an frozen or trainable CLIP for each component, and summarize the results in Tab. 6 To ensure fair comparison, all "trainable" modules utilize the same weights, resulting in identical mask proposals and in- on-vocabulary classification results. Moreover, we note that the first row in Tab. 6 with trainable mask generator and vocabulayr classifier, can be considered as an approximation OpenSeg (9 in our framework. Our findings reveal that an in onvooabbllrry classifier built upon trainable CLIP backbone achieves a higher PQueen score (37.9 compared 32.4), but experiences a decrease in PQ"meeen (2.6 compared to 12.6) compared to a frozen aut-of-vocabulay classifier. Consequently, model that incorporates a trainable CLIP backbone for all components yields a PQ of 24.1, which is 2.7 lower than our final model (last row) that relies on a single frozen CLIP backbone. Using trainable mask generator and in- -o-cuabbllrry classifier, along with frozen out-of -oc-vocayllary classifier boosts the performance but requires maintaining one trainable and one frozen CLIP weights, resulting in 2x more backbone parameters, In summary, our observations demonstrate that building the entire framework upon frozen CLIP backbone is not only effective but also efficient, providing better balance between PQreen and PQ metrics.
Table 6: Effects of fine-tuning or freezing the CLIP backbone for each module in FC-CLIP. Building all three modules upon single frozen CLIP backbone attains best performance. Note that our mask generator and in- in-vocabulary classifier use the same backbone following (20 22 94), and thus is infeasible (denoted as N/A) for the setting the 2nd last row. Our final setting is labeled in gray
mask generator -n-ooaabllary classifier oft-of-vocabulayy classifier
PQ
PQ
PQumsen
trainable
trainable
17.7
37.9
2.6
trainable
frozen
21.1
32.4
12.6
trainable
trainable
trainable
24.1
38.9
13.1
trainable
trainable
frozen
25.4
40.0
14.6
trainable
frozen
frozen
N/A
N/A
N/A
frozen
frozen
frozen
26.8
39.5
17.3
Evaluation with Grounding PQ and Grounding mIoU It is worth emphasizing that despite the
absence of grounding loss [33 97 29 89 during training, our model exhibits exceptional grounding segmentation capabilities. Tab. 7 presents the grounding PQ and grounding mloU scores of FC- CLIP, following the evaluation methodology outlined in [29]. In this evaluation, we exclusively employ truth classes as text query inputs to assess the effectiveness of concept grounding. Compared to OpenSeg 29), FC-CLIP achieves a substantial performance improvement, with notable enhancements of +11.6, +9.1. +13.1, and +17.7 on A-847, PC-459, A-150, and PC-59, respectively. Even when compared to OpenSeg trained with the Localized Narrative dataset 65 which enables training on a significantly larger vocabulary, FC-CLIP still surpasses it with improvements of +8.0, +2.2, +8.6 and +13. on A-847, PC-459, A-1 and PC-59, respectively, underscoring the grounding proficiency of FC-CLIP.
16

---

Table 7: Grounding segmentation performance. The proposed FC-CLIP also demonstrates state-
of-the-art performances on grounding segmentation. MV: Mapillary Vistas
grounding PQ
grounding mloU
method
ADE20K Cityscapes MV
A-847
PC-459
A-150 PC-59 PAS-21 PAS-20
ALIGN 40
29
17.8
21.8
25.7
34.2
ALIGN w/ proposal 40
29
17.3
19.7
25.3
32,0
1Seeg+
48
29
10.5
17.1
30.8
56.7
OpenSeg
29
21.8
32.1
41.0
57.2
OpenSeg
29
w/ I. Narr
25.4
39.0
45.5
61.5
FC-CLIP (ours)
38.4
48.1
21.5
33.4
41.2
54.1
74.9
88.7
98.5
Table 8: Ensemble methods comparison with zero-shot evaluation (PQ) on ADE20K. Our method is robust to different ensemble methods (arithmetic and geometric). The results show that it is preferable to bias towards using the in- no-vocabblary classifier for seen classes and the -ot-bl-vocabulary classifier for unseen classes. Our final setting (a = 0.4, 3 0.8) is labeled in gray method
arithmetic
geometric
(a 0.0, 8 0.0)
17.8
17.8
(a 100, = 1.0)
21.9
21.9
(a 0.0, 8 1.0)
25.3
25.3
(a 1.0, 0.0)
17.5
17.5
(a 0.5, 8 0.5)
25.0
25.3
(a 0.5, 0.6)
25.6
26.4
(a 0.5, 8 0.7)
25.5
26.7
(a 0.5, 3 0.8)
25.4
26.6
(a 0.4, = 0.6)
25.1
25.6
(a 0.4, 0.7)
25.6
26.4
(a 0.1. / = 0.8)
25.6
26.8
(a 0.4, 0.9)
25.4
25.8
Table 9: Quantitative results of ViT-based CLIP and CNN-based CLIP when input size (denoted
as "res") varies for panoptic segmentation on COCO and ADE20K. All results are obtained by applying CLIP directly as a mask classifier with the same mask proposals from ODISE 89
COCO PQ @res
ADE20K PQ @res
CLIP backbone
224 448 672 896 1120
224 448 672 896 1120
ViT-L/14
19.3 22.5 20.6 18.5 14.9
11.9
13.7
12.6 11.6 9.1
ConvNeXttLL
17.3 23.5 27.0 28.6
29.3
9.3 12.8
14.8 16.0 15.9
Ensemble In- Vocabulary and Out-of- Vocabulary Classifiers In Tab. we present experiments conducted to evaluate the impact of ensemble methods and ensemble parameters on the performance of the in-vocabulary and out-of-vocabulart classifiers. Specifically, methods: arithmetic and geometric. The arithmetic method involves a linear combination of the in-vocabulary classifier and the out-of-vocabulary classifier, while the geometric method is defined as shown in Equation (7) of main paper. It is worth noting that FC-CLIP exhibits robustness to different ensemble methods, with both methods displaying consistent trend within the explored hyper-paramett ranges. However, the geometric ensemble consistently outperforms the arithmetic ensemble by slight margin. Additionally, we observe that preference given values of a < 0.5 and / 2 0.5, which biases the model towards using the in-vocabulary classifier for seen classes and the -oocbl-voyayulary classifier for unseen classes. We also explore extreme cases, including = 0.0 and / = 0.0 (i.e., exclusively utilizing the in- nn-vocbuulary classifier for every class), a = 1.0 and [ = 1.0 (i.e., exclusively utilizing the aut-of-ocabbllyy classifier for every class), 0.0 and 3 = 1.0 (i.e., using the in in-vocabulary classifier for seen classes and the out-of voc-ol-vocabulary classifier for unseen classes), and a = 1.0 and 3 0.0 (i.e., using the aut-of-vocabulary classifier for seen classes and the in-vocabulary classifier for unseen classes). The results align with our observations that it preferable to bias towards the in-vocabulary classifier for seen classes and the oft-of- vocabuaryy classifier for unseen classes.
examine
ensemble
17

---

Table 10: Quantitative results of ViT-based CLIP and CNN-based CLIP when input size (denoted as "res") varies for ImageNet-1k classification.
Accuracy @res
CLIP backbone
224 336 448 560 672 784 896
ViT-L/14
75.3 74.3 71.3 67.5 63.1 58.5 53.9
ConvNeXt-L
75.1 77.1 76.8 74.2 69.8 65.6 58.4
Table 1I: Open-vocabulary segmentation performance with different backbones and segmenta- tion frameworks. All models are trained on COCO and tested on the other datasets in zero-shot manner. MV: Mapillary Vistas. *; KMaX-DeepLab with multi-scale deformable attention [03
panoptic datasets (PQ)
semantic datasets (mloU)
method
backbone
ADE Cityscapes
MV
A-847
PC-459
PC-59
PAS-21
FC-CLIP
R50 35 69]
17.9
40.3
15.9
7.1
12.9
50.5
75.9
FC-CLIP
R101
35
69
19.1
40.9
16.7
7.7
12.3
48.9
77.6
FC-CLIP
R50x4
21.8
42.2
17.4
13.1
54.0
79.0
FC-CLIP
R50x16 16 69
22.5
42.0
17.8
10.3
15.7
56.4
80.7
FC-CLIP
R50x64
69
22.8
42.7
18.2
10.8
16.2
55.7
80.3
FC-CLIP w/ KMaX
ConvNeXI-L
[58
38
24.5
43.0
17.0
11.4
15.0
57.4
84.7
FC-CLIP w/ kMaX"
ConvNeXt-L
58
38
26.4
40.2
17.4
13.6
17.5
57.1
81.2
FC-CLIP
ConvNeXt-L.
38
26.8
44.0
18.2
14.8
18.2
58.4
81.8
Quantitative ViT-based CLIP vs. CNN-based CLIP when Input Size Scales Training our model solely with ViT-based CLIP, without any additional modifications [101] 17,91121 is infeasible. Furthermore, applying VIT to large input sizes is computationally expensive. Therefore, to evaluate the effects of using ViT- CNN-based CLIP in framework, incorporate them into out-of- vocabulary classifier, which is performed only during inference. To ensure a fair comparison, we use the mask proposals and disable the geometric ensemble scheme. We also perform experiment on the ImageNet (73) benchmark to ensure a comprehensive comaprison. In Tab. and Tab. TO] we conduct an ablation study analyze the impact of different input resolutions for CLIP models. We consider both ViT-based (ViT-L/14) and CNN-based (ConvNeXI-L) CLIP models. By employing them as zero-shot classifiers and varying the input resolutions, we observe that CNN-based CLIP demonstrates superior generalization ability the input size scales up. Specifically, we observe that the ViT-L/14 CLIP has higher PQ and Accuracy at a lower resolution (i.e.. input size 224). but suffers from higher resolution, which leads existing two-stage methods [90, 52 25 91 89 to adopt different input resolutions for mask generator and classifier branches. On the contrary, FC-CLIP provides simple solution by adopting a CNN-based CL.IP that generalizes well to different input sizes.
FC-CLIP with Different Backbones and Different Segmentation Frameworks Though we majorly report FC-CLIP results with ConvNeXt-L (58 69 backbone Mask2Former [20] frame- work. We note that FC-CLIP can be easily incorporated with different backbones and segmentation frameworks. Specifically, we experiment FC-CLIP with different backbones (e.g., ResNet (35) and different segmentation architecture (e.g., L.MaXDDeepl [94)). As shown in Tab. I1 FC-CLIP demonstrates superior performance across different backbones and frameworks.
Visualization
We provide visualization on ADE20K val set in Fig.
7 Datasets Information and Licenses
The datasets we used for training and/or testing FC-CLIP are described as follows.
We train FC-CLIP on COCO data with panoptic annotation [44] We follow the 2017 splits which include 118k images for train split and 5k images for val split. If not specified, we train our model on the COCO train split and report results on val set of various datasets.
COCO:
License: Creative Commons Attribution 4.0 License
URL: httpp:://oot https://coooodtaaset. org #home
18

---

cop
Ward
calmet
Figure 5: Visualization examples of FC-CLIP ADE20K val set. FC-CLIP is trained on COCO panoptic training set and zero-shot evaluated on ADE20K validation set.
19

---

covers a wide range of indoor and outdoor scenes, with 2k val images. We evaluate FC-CLIP on both the version with 847 classes (A-847) and the more widely-used version with 150 frequent categories (A-150).
ADE20k:
ADE20k TOO
License: Creative Commons BSD-3 License
URL: httpt::///toousssstsssss cswicc mimm wdwe http://iiiii//atattttt////220
Cityscapes: Cityscapes [22] focuses on semantic understanding of urban street scenes, We use the fine data includes 500 images for validation set.
License:
This dataset is made freely available to academic and non-academic entities for non- commercial purposes such as academic research, teaching, scientific publications, personal experi- mentation.
URL: https::///wttt citysaapes-aaaast.ccom
Mapillary Vistas: Mapillary Vistas 664 is large-scale traffic-related dataset, including 2k images for validation purposes.
License: Creative Commons Attribution NonCommercial Share Alike (CC BY-NC-SA) license
URL: https:://ww.ttt www.p/paar...co//attttttttteeeee
covers a wide variety of indoor and outdoor scenes and includes 5k val images. We evaluate FC-CLIP on both its full version (PC-459) with 459 classes and the more common version (PC-59) with 59 classes.
Pascal Context: Pascal Context 663
URL: httpt:t//ww.w.... cB. stanftrddd edw. httt:///oocbbe/paaaalooottteete
Pascal VOC: Pascal VOC contains 1.5k val images with 20 foreground classes and
background class. Due to the ambiguity in definition of "background", we assign the background class to the pixels predicted as PC-59 categories that are not in Pascal VOC following [29]. which leads to PAS-21. We also evaluate the model with background class excluded, which leads to PAS-20.
URL: robots. ac. uk wwwpacaa/////
20


Reviews:
[{"review_id": "nNoNSBhveG", "reviewer": "Reviewer_N4iz", "summary": "In this submission, the authors proposed a new method for open-vocabulary panoptic segmentation. In the open-vocabulary segmentation setting, the model is trained on seen category annotations and tested on unseen categories. The frozen features of CLIP/ALIGN have been demonstrated to be effective in new category generalization.  To leverage the representation of CLIP, prior works typically use the two-stage framework, one stage forwards high-resolution images for mask generation with vanilla networks, and the other stage inputs low-resolution images for mask classification. The proposed a shared Frozen Convolutional CLIP backbone (FC-CLIP) which unifies the pipelines into a single stage. It exploits frozen CLIP with ConvNeXt image encoder as the backbone network. FC-CLIP is simple and yet effective, surpass prior state-of-the-art on many open-vocabulary segmentation tasks.", "questions": "1. I noted authors used a different resolution setting, long side 1281. Is there any ablation on this design compared to Mask2Former 1024 short-side resize used in other works? Besides, regarding the inference time comparison, I would suggest authors use the same input size for all the models. \n2. In Table 1 of supplementary material, I think the last row should be swapped with the second last row.", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This submission explains the motivation and method very well. I like Figure 2 very much, which clearly shows the differences between FC-CLIP and prior works. \n2. The proposed FC-CLIP simplifies the multiple forwards of the image encoder into just one forward pass, which saves the computation cost. \n3. The proposed model outperforms many prior works and more quantitative results are also provided in the supplementary material.", "weaknesses": "1. In the related work, I would suggest authors discuss more about the relationship with F-VLM, which also uses the frozen Convolution CLIP as the shared image encoder. What are the major differences between the designs of F-VLM and FC-CLIP? \n2. The ablation study on CLIP model type is missing. In Figure 1, the authors demonstrate that CNN-based CLIP should get better features than ViT-based CLIP. I truly appreciate that the authors did compare ViT-L/14 with ConvNeXt-L in Table 5 of the supplementary material. I would suggest authors also compares other CNN-based CLIP like R50x4, R50x16, R50x64 used in F-VLM. \n3. Table 5 in supplementary material shows a very interesting experiment, which shows increasing resolution from 224->448, ViT-L still outperforms ConvNeXt-L. But when increasing resolution to 672, ViT-L/14 accuracy drops significantly, which is slightly counter intuitive. Is this PQ evaluated with geometric mean or not? And I would also recommend evaluating zero-shot ImageNet classification accuracy at different resolution inputs, which could better justify whether CNN-based CLIP outperforms ViT-based CLIP when changing resolutions. \n4. Authors state \"training model solely with ViT-based CLIP is infeasible\", probability due to the GPU memory constraint. To compare more fairly with CNN-based CLIP, I would suggest authors use a sliding window to extract features for ViT-based CLIP, e.g. sliding a 224x224 or 336x336 window over 1024x1024 input image to extract the ViT-based CLIP features, which should still have the capability of zero-shot classification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive feedback and address the concerns below.\n\n>***W1: Relationship with F-VLM***\n\nPlease refer to **C1: Relationship to F-VLM**\n\n>***W2: Other CNN-based CLIP***\n\nWe thank the reviewer for the valuable suggestion. We provide results with different CNN backbones below:\n\n| CLIP      | R50 (38.3M) | R101 (56.3M) | R50x4 (87.1M) | R50x16 (167.3M) | ConvNeXt-L (199.8M) |\n|-----------|:-----------:|:-----------:|:------------:|:---------------:|:-------------------:|\n| ADE20K PQ |     15.5    |     17.1    |     18.5     |       20.4      |         24.5        |\n\nAs shown in the table above, the performance increases as the model size increases. Using ConvNeXt-L backbone in our setting achieves the best performance, echoing that ConvNeXt is a more modern design of CNN. We note that prior arts use a very strong backbone and CLIP model (e.g., StableDiffusion UNet + ViT-L/14 as in ODISE), which makes it hard to compare against them with much smaller models. Therefore, we adopt ConvNeXt-L in the end (note that ConvNeXt-L has 199.8M parameters, while ViT-L has 304.3M).\n\n>***W3: Table 5 in Supplementary***\n\nWe thank the reviewer for the question and suggestion. We address each concern below.\n\n**Evaluation setting**: We herein provide more details regarding the mask classification evaluation. The experiments are our early exploration to verify the effect of using CNN-based and ViT-based CLIP as mask classifiers at different input resolutions, with ODISE’s codebase. Specifically, we adopt ODISE’s mask proposals (thus the mask proposals are kept the same across settings, and ***not trained*** for any specific CLIP model), but replace the mask classification results using either a LAION-2B pretrained ViT-L/14 or ConvNeXt-L CLIP backbone. For ViT-L/14, the classification pipeline follows MaskCLIP [24] and ODISE with attention masking. For ConvNeXt-L, we employ the simple mask pooling to obtain classification logits for each mask. For simplicity and a fair comparison, there is ***no*** geometric mean used in this ablation study.\n\n**Zero-shot ImageNet accuracy**: We appreciate the suggestion and perform a similar experiment on ImageNet in the following table.\n\n| IN1k Acc   |  224 |  336 |  448 |  560 |  672 |  784 |  896 |\n|------------|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n| ViT-L/14   | 75.3 | 74.3 | 71.3 | 67.5 | 63.1 | 58.5 | 53.9 |\n| ConvNeXt-L | 75.1 | 77.1 | 76.8 | 74.2 | 69.8 | 65.6 | 58.4 |\n\nWe observe the same trend (as in segmentation) that while both ViT-based and CNN-based CLIP can yield a strong zero-shot performance at smaller resolutions, CNN-based CLIP generalizes better when scaling up input resolutions.\n\nWe would like to emphasize that the scenario of applying the CLIP model for image classification on ImageNet and mask classification on COCO is slightly different. ImageNet contains object-centric images, while COCO has objects with diverse scales. As a result, scaling up input resolution from 224 to 448 improves the recognition accuracy (as small objects are more visible) for **both** ViT-based and CNN-based CLIP models on COCO (see Table 5 in Supplementary), but not on ImageNet.\n\n\n>***W4: ViT-based CLIP backbone in a sliding window manner***\n\nWe thank the reviewer for the question and suggestion. We note that using ViT-based CLIP backbone presents several technical challenges, including, but not limited to: GPU memory, absence of multi-scale features, resolution discrepancy between upstream pretraining and downstream segmentation, etc. Even with the suggested sliding window method, a careful re-design is still needed to handle problems such as overlapping pixels, ensemble class tokens at each window, a side-adapter to generate multi-scale features, etc. As noted in the paper, we think a CNN-based CLIP is a straightforward and effective solution to these problems, allowing us to build a simple, strong, and effective single-stage open-vocabulary segmentation framework. That being said, we agree with the reviewer that adapting ViT models is an interesting and promising research problem, especially considering that ViT usually demonstrates a better property of model scaling up.\n\n>***Q1: Resolution difference***\n\nWe thank the reviewer for the question. We note that FC-CLIP is a meta architecture that can build upon several different segmentation frameworks. In the submission, we follow kMaX-DeepLab to resize longer edge to 1281 and pad shorter edge to 1281, which gives a similar effective size (if only considering non-padded pixels) as Mask2Former which resizes shorter edge to 800 and longer side to 1333. To further clarify the resolution issue, we also provide results of building FC-CLIP upon Mask2Former in the following table. Note that ODISE is also built upon Mask2Former and thus FC-CLIP-Mask2Former further provides a comprehensive comparison with ODISE.\n\n|                     | frozen params (M) | trainable params (M) |             ADE20K        | Mapillary Vistas | Cityscapes |\n|---------------------|:-----------------:|:--------------------:|:---------------------:|:----------------:|:----------:|\n| ODISE               |        1494       |          28          | 22.6 / 23.4 (caption) |       14.2       |    23.9    |\n| FC-CLIP-kMaX        |        200        |          34          |          24.5         |       17.0       |    43.0    |\n| FC-CLIP-Mask2Former |        200        |          21          |          26.8         |       18.2       |    44.0    |\n\nAs shown in the table above, we observe FC-CLIP even achieves better performances, when we switch to the same segmentation framework as ODISE (i.e., use Mask2Former).\n\n>***Q2: Swap rows in Table 1 of sup***\n\nThanks for the suggestion. We will look into it in the revision.\n"}, {"review_id": "jYPharLyaj", "reviewer": "Reviewer_GVRs", "summary": "This work proposes a new approach to open-vocabulary panoptic segmentation that unifies the mask generator and CLIP classifier into a single-stage framework. This is achieved by sharing the feature extractor between them, which presents two challenges: disrupting the alignment between image and text features during fine-tuning and the need for higher resolution inputs for dense prediction tasks. The authors address these challenges by adopting the shared Frozen CNN-based CLIP backbone. The resulting FC-CLIP model achieves state-of-the-art performance on several benchmarks while being more efficient and effective than previous methods.", "questions": "The proposed method is well-motivated and novel. However, some key experiments are lack. More details please refer to the weakness part.", "limitations": "The authors have briefly discussed the limitations and future work of their work. However, there are some limitations required to discussed.  \n\nThe FC-CLIP model relies on a CLIP model pre-trained on the Internet data that may be biased, which calls for future research for calibration to avoid misuse. \nThey could provide a more detailed discussion of these issues. For example, they could discuss the potential biases in the pre-trained CLIP model and how these biases could impact the performance of the FC-CLIP model. They could also discuss the potential negative societal impact of the FC-CLIP model, such as its use in surveillance systems or other applications that could infringe on privacy rights.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper's approach to unifying the mask generator and CLIP classifier into a single-stage framework is a novel contribution to the field of open-vocabulary panoptic segmentation. \n\n2. The paper is well-written and clearly presents the problem of open-vocabulary panoptic segmentation, the challenges of the current two-stage pipeline, and the proposed FC-CLIP model. \n\n3. The paper is well-organized and clearly presents the problem, methodology, and results. The authors provide a detailed explanation of the FC-CLIP model, making it easy to understand the proposed approach. The paper's figures and tables are well-designed and provide a clear visualization of the results.", "weaknesses": "\n1. Absence of Ablation Study: The paper lacks an ablation study to conduct a thorough analysis of the impact of different components or design choices of the FC-CLIP model on its performance. For instance, the inclusion of an in/out-vocabulary classifier and the combine strategy, if adopted within the naive single-stage framework, what would be its effect on performance?\n\n2. The experimental results are not convincing enough. The FC-CLIP model has a higher number of trainable parameters compared to ODISE. However, it remains uncertain whether the observed performance improvement can be attributed solely to the increased parameter count. (2) Table 2 highlights significant improvements achieved by the FC-CLIP model on the Cityscapes dataset. It would be beneficial to provide additional explanations or discussions to elucidate the reasons behind these improvements. (3) While FC-CLIP demonstrates noticeable enhancements on the ADE20K dataset, the improvements on the COCO dataset appear to be comparatively smaller.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive feedback and address the concerns below.\n\n>***W1: Missing ablation study on different components/design choices***\n\nWe have **already provided the asked ablation studies** on model designs in the Table 1 and Table 4 of Supplementary, e.g., combining in/out-of-vocabulary classifiers, as well as the effect of them in the naive single-stage framework.\nTo clarify the confusion, we quote the results from supplementary here again:\n\n**1. Combining in-/out-of-vocabulary classifiers and hyper-parameters:**\n\n| ensemble (alpha, beta) | arithmetic ensemble | geometric ensemble |\n|:------------------------:|:-------------------:|:------------------:|\n| (0.0, 0.0)             |         17.6        |        17.6        |\n| (1.0, 1.0)             |         19.4        |        19.4        |\n| (0.0, 1.0)             |         22.8        |        22.8        |\n| (1.0, 0.0)             |         15.7        |        15.7        |\n| (0.5, 0.5)             |         23.2        |        23.4        |\n| (0.4, 0.6)             |         23.1        |        23.3        |\n| (0.4, 0.7)             |         23.7        |        24.2        |\n| (0.4, 0.8)             |         24.0        |        24.5        |\n| (0.4, 0.9)             |         23.9        |        24.1        |\n\n**2. Effect of each module:**\n\n| mask generator | in-vocab classifer | out-of-vocab classifier |  $PQ$  | $PQ_{seen}$ | $PQ_{unseen}$ |\n|:--------------:|:------------------:|:-----------------------:|:----:|:-------------:|:---------------:|\n|    trainable   |      trainable     |            -            | 16.2 |      33.9     |       2.9       |\n|    trainable   |          -         |          frozen         | 19.4 |      28.5     |       12.6      |\n|    trainable   |      trainable     |        trainable        | 19.6 |      34.9     |       8.2       |\n|    trainable   |      trainable     |          frozen         | 23.8 |      36.0     |       14.7      |\n|     frozen     |       frozen       |          frozen         | 24.5 |      37.4     |       14.9      |\n\nFor the naive single-stage models with in-/out-of-vocabulary classifiers, please refer to the third row **(trainable, trainable, trainable)**, which leads to 19.6 PQ with a performance degradation by -4.9 PQ from our final setting in the last row ($PQ_{unseen}$ degrades most by -6.7 PQ). The 4th row **(trainable, trainable, frozen)** provides the result by employing another frozen off-the-shelf CLIP as out-of-vocabulary classifier and leads to a comparable PQ score 23.8 (-0.7 PQ), but we note that this actually leads to the same model costs as two-stage models, with the need of additional weights and extra forwarding pass. We will add additional elaboration to the results to avoid confusion.\n\n>***W2.1: Improvement comes from more trainable parameters?***\n\nWe note that ODISE has a great advantage over our method with *1294M* more frozen parameters. We only have *6M* more trainable parameters, which is mainly due to the differences between kMaX-DeepLab and Mask2Former. We think that the 7.5 times more frozen parameters should have more impacts on performance compared to small additional 6M trainable parameters. That being said, to further address the question, we also build FC-CLIP on top of Mask2Former, which not only has fewer (-7M) trainable parameters (note that ODISE has another trainable MLP for diffusion model's text embeddings), but also has a better performance than ODISE, indicating the improvement comes from our simple and effective meta architecture design.\n\n|                     | frozen params (M) | trainable params (M) |         ADE20K        | Mapillary Vistas | Cityscapes |\n|---------------------|:-----------------:|:--------------------:|:---------------------:|:----------------:|:----------:|\n| ODISE               |        1494       |          28          | 22.6 / 23.4 (caption) |       14.2       |    23.9    |\n| FC-CLIP-kMaX        |        200        |          34          |          24.5         |       17.0       |    43.0    |\n| FC-CLIP-Mask2Former |        200        |          21          |          26.8         |       18.2       |    44.0    |\n\n>***W2.2: Improvement on Cityscapes***\n\nFC-CLIP has a better improvement on the street-view datasets such as Cityscapes and Mapillary Vistas than ODISE. We think that it is because ODISE relies on latent diffusion models for feature extraction, which performs a VQ tokenization on the input image. The VQ tokenizer was trained for image generation purposes and thus may not handle complex street view very well. On the contrary, FC-CLIP has a simpler and more effective design, which generalizes well to such datasets.\n\n>***W2.3: Improvement on COCO***\n\nIn Table 1 of the main paper, the COCO dataset is used as the only training dataset for both ODISE and FC-CLIP. Therefore, its results only indicate the closed-vocabulary performance and thus closely related to the model's capacity to fit the training dataset instead of its ability to novel datasets in the open-vocabulary scenario. In this case, ODISE has a much larger model size and thus is expected to fit COCO better than FC-CLIP (as shown in the paper L281 to  L284, we also observe that ODISE can provide better mask proposals, thanks to its much larger model size). However, FC-CLIP generalizes much better to other datasets in a zero-shot manner. \n\n>***CLIP biases and limitations***\n\nWe thank the reviewer for bringing up the bias and limitation in the pre-trained CLIP model, which may also impact FC-CLIP, we will add related limitation discussion in a revision per suggested."}, {"review_id": "nTqYutfNOF", "reviewer": "Reviewer_YCke", "summary": "The authors propose an approach based on CLIP model and extend it for zero-shot semantic segmentation. The authors argue that pervious works solve the problem with a two-stage approach, which first generates mask predictions using one backbone and then extract features from another backbone using CLIP model is suboptimal and inefficient. They present an approach using a frozen CLIP image encoder as backbone and generate mask and prediction with trainable pixel and mask decoders. They validate the effectiveness of proposed method on several commonly used semantic segmentation benchmarks and achieve solid performance.", "questions": "1. Please clarify the contributions of proposed methods.\n\n2. Please justify the different between proposed approach and the previous work [28]", "limitations": "The limitation is discussed in the paper to some extent and I don't have major concerns on the limitation.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Extending CLIP model to semantic segmentation in an efficient and effective approach is a recently popular and active research problem in Computer Vision. The authors present an approach to address this problem.\n\n2. The proposed technique is technically sound.\n\n3. The authors conduct experiments comparing to the latest previous work, ODISE, and shows that the performance is comparable while enjoy faster inference speed.", "weaknesses": "1. The contribution is incremental. The proposed approach in very similar to OpenSeg [28], with just different selection of backbone, mask generation and classification of in-vocabulary classification. The minor difference in the selections of components is not significant for the top-tier conference. Comparing to [28], it is unclear where the improvement come from. It could be because of the selection of pixel decoder features for in-vocabulary categories, or the different selection of mask generation or the input resolution. There is no ablation to understand this and neither of these changes are significant contributions. \n\n2. The naive single-stage baseline presented in the paper is not reasonable. As CLIP backbone is fine-tuned without considering text encoder, it naturally loses the capability of open-vocabulary classification. The baseline is not reasonable but created in a way to makes the contributions of proposed method appear meaningful\n\n3. The technical details presented in this paper are not self-contained. Details as follow:\n- The description of naive single-stage baseline is vague. What is the mask generator used in the baseline? Does the classifier use text embedding or something else?\n- The description of class-agnostic mask generator presented in the paragraph starting at line 192 is vague and lack of details. What is the pixel decoder with axial attention? What is the kMax mask decoders? What is the k-means cross-attention? How is Hungarian matching used? How is the subset of predicted masks selected during the matching process? As mask generation is a critical component in the proposed method, it needs to be properly described even if they are introduced in the previous work to make the paper self-contained and to justify the contribution of the proposed method.\n\n- Abuse of notation. In line 136, sum{m_i} <= 1^{HxW} is not properly defined. I am guessing it means every entry in the matrix is smaller or equal to 1. This is a minor issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and address the concerns below.\n\n>***W1, Q2: Comparison to OpenSeg***\n\nIn the submission, we have already carefully discussed the limitations of naive single-stage methods such as OpenSeg, and how FC-CLIP differs from them. We summarize them again below for reference:\n\n1. As discussed in the Supplementary L19 to L23, we note that methods like OpenSeg jointly fine-tune the whole model, which leads to a worse generalization ability to novel concepts. While in this paper, we have provided in-depth analysis and experiments to validate the importance of adopting a frozen CNN-based CLIP backbone for better open-vocabulary segmentation performance.\n\n2. We respectfully disagree that *”Comparing to [28], it is unclear where the improvement come from. …. There is no ablation to understand this and neither of these changes are significant contributions.\"* In fact, in the Supplementary, we have compared a naive single-stage baseline which can be considered as a reproduction of OpenSeg (as their code is not open-sourced) in our framework (Supp. Table 1). We quote the results from Supplementary here:\n\n| mask generator | in-vocab classifer | out-of-vocab classifier |  $PQ$  | $PQ_{seen}$ | $PQ_{unseen}$ |\n|:--------------:|:------------------:|:-----------------------:|:----:|:-------------:|:---------------:|\n|    trainable   |      trainable     |            -            | 16.2 |      33.9     |       2.9       |\n|    trainable   |          -         |          frozen         | 19.4 |      28.5     |       12.6      |\n|    trainable   |      trainable     |        trainable        | 19.6 |      34.9     |       8.2       |\n|    trainable   |      trainable     |          frozen         | 23.8 |      36.0     |       14.7      |\n|     frozen     |       frozen       |          frozen         | 24.5 |      37.4     |       14.9      |\n\nAs shown in the table above, the first row **(trainable, trainable, -)** can be considered as a reproduction of OpenSeg in our framework. Both our final model (last row) and our \"OpenSeg\" baseline (1st row) use the same backbone and segmentation frameworks thus involve no \"selections of components\".\n\nBesides, we also disagree that *\"improvement is from better selection of modules\".* When compared to prior state-of-the-art ODISE, FC-CLIP has a much smaller backbone and simpler design, and can actually have an even better performance when switching to ODISE’s segmentation framework Mask2Former.\n\n>***W2: The naive single-stage baseline not reasonable. Fine-tuning text encoder?***\n\nWe respectfully disagree with the reviewer. To the best of our knowledge, most open-vocabulary segmentation methods (e.g., SAN, ODISE) do not fine-tune the text encoder. This is natural, as the COCO Panoptic dataset contains only 133 classes, where freezing the text encoder avoids the catastrophic forgetting problem. Our pipeline closely follows them. We do not see that freezing the text branch is improper, considering segmentation datasets only contain limited training vocabularies. To the best of our knowledge, *only* the concurrent work OpenSeeD (ICCV 2023) fine-tunes the text encoder when trained on COCO and Objects365 datasets. However, they achieve 19.7 PQ on ADE20K, which is similar to the performance of our all trainable baseline 19.6 PQ (only trains on COCO). We also note that the single-stage method OpenSeg is not open-sourced. But, we have closely followed it to build our solid single-stage baseline, which is not *\"created in a way to make the contributions of the proposed method appear meaningful\".* \n\n>***W3.1: Mask generator and classifier for naive single-stage baseline***\n\nAs shown in the Figure 2 of our paper, we aim to provide a system-wise comparison among different open-vocabulary segmentation methods, involving no specific requirements on Mask Generator or Classifier. To answer this question, we have built a baseline mimicking OpenSeg (as their code/model is not open-sourced) in Table 1 of Supplementary, which uses the same mask generator and classifier as our final model (kMaX-DeepLab segmentation framework and CLIP text embeddings).\n\n>***W3.2: Technical details regarding model architecture and training***\n\nWe respectfully disagree with the reviewer. The proposed FC-CLIP is a meta architecture, which can be built on top of several segmentation frameworks ***without*** any change (as shown in our response to Q1 of Reviewer N4iz, where we also experiment with Mask2Former). We do not provide any further details of the adopted kMaX-DeepLb segmentation framework, as they are not the main focus of this work (and we did not make any change to them). Providing additional detailed descriptions of them obscures the focus of this work: a general open-vocabulary segmentation pipeline with a ***frozen convolutional CLIP***. We believe that the provided descriptions and references in the draft are already sufficient for the readers to get the context. Additionally, we promise to fully open-source our training/testing codes, allowing the community to check every detail. If the reviewer still disagrees with us, we would like to get the second opinion from either the other reviewers or the area chair. If they also think so, we are more than happy to add more details for those segmentation frameworks in the main paper.\n\n>***W3.3: Notation***\n\nThe notation appropriately explains the property of panoptic segmentation, where the predicted masks do not overlap with each other. This notation is standard for panoptic segmentation and is aligned to the panoptic segmentation works [40, 72]. That being said, we thank reviewer for suggestion, and we will make it clear.\n\n>***Q1: Contributions of proposed methods.***\n\nPlease refer to common concerns **C2: Novelty/Contribution**\n"}, {"review_id": "uUHpUTrHP1", "reviewer": "Reviewer_5KPy", "summary": "This paper proposes to build a one-stage open-vocabulary detector with a frozen language encoder (CLIP) and acvhieves good performance.", "questions": "Described above.", "limitations": "Described above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Strong performance. Performance on many benchmarks is better than previous models.\n2. Simple framework. One stage does look more simple to use.", "weaknesses": "1. The novelty is limited. The proposed design does not involve enough novelty.\n2. From my point of view, one stage and two stage methods do not differ that much. Why one-stage method is much better than previous methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive feedback and address the concerns below.\n\n> ***W1: Novelty***\n\nPlease refer to common concerns **C2: Novelty/Contribution**\n\n> ***W2: One-stage v.s. two-stage methods***\n\nA one-stage framework can re-use a shared feature extractor across different modules, which not only leads to a simpler framework but also achieves much better accuracy-cost trade-off. Compared to the state-of-the-art two-stage method ODISE, FC-CLIP demonstrates a better performance, while using only 15.4% total parameters and runs 4.4x faster. The proposed one-stage model FC-CLIP significantly simplifies the system design without the need to consider separate modules in the pipeline (e.g., only need to consider one backbone instead of multiple backbone combinations). \n\n"}, {"review_id": "C5xUNM7Ywu", "reviewer": "Reviewer_VW2t", "summary": "The paper tackles the open-vocabulary panoptic segmentation with a frozen CLIP. Unlike previous two-stages works, the paper uses the same backbone from the frozen CLIP for the mask generator and classifier. The single-stage framework accelerates the training and inferring speed and achieves SOTA performance.", "questions": "1. In Open-CLIP, they have three configures of ConvNext-Large:\n\nConvNext-Large (D) @ 256x256 /w augreg on LAION-2B with a top-1 of 75.9%\n\nConvNext-Large (D) @ 320x320 fine-tune of 256x256 weights above for ~2.5B more samples on LAION-2B, top-1 of 76.6%\n\nConvNext-Large (D) @ 320x320 soup of 3 fine-tunes of 256x256 weights above on LAION-2B, top-1 of 76.9%\n\nWhich one did the author use? The ConvNext-L trained with 320x320 is naturally more suitable for high-resolution inputs than ViT-L/14 trained with 224x224.", "limitations": "In limitations, the author mentioned \"how to deal with conflict or overlapping vocabularies (e.g. cat vs. cat head)\". Does this problem occur in the experiment of the paper? What will happen to FC-CLIP if it faces the problem?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The single-stage framework proposed in the paper is concise and time-friendly for training and inference. The writing of the paper is clear and can be easily followed. The paper shows the convolutional CLIP has a better generalization ability to higher-resolution inputs than ViTs. FC-CLIP achieves SOTA performance on multiple open-vocabulary panoptic segmentation benchmarks.", "weaknesses": "1. The novelty of the paper is weak. The idea of using the frozen CLIP for open-vocabulary prediction is the same as F-vlm. The main difference is that paper uses the framework of F-vlm to tackle the panoptic segmentation problem and substitute the Mask R-CNN heads in F-vlm with kMaX-DeepLab. Regarding innovation, I am not saying that the author must come up with some fancy innovations. But compared with the predecessors' work, what targeted improvements and discoveries have been made to new problems? Compared to F-vlm, I did not see any special design for the panoptic segmentation problem.\n\n2. The claim that (frozen) CNN-based CLIP is better than ViT-based CLIP for dense prediction is not well verified. In the paper, the author verified the claim in Figure 1 with k-means visualization of these two types of CLIP. But whether the smoother feature correlates with the dense prediction performance is unclear. Many works [2][3] use ViT-based CLIP as the backbone of dense prediction tasks. Although the author compares ViT-based CLIP and CNN-based CLIP in Table 5 of Supp, the mask proposals are optimized with the CNN feature (on seen classes). It is more rigorous to use the ground truth mask to evaluate the classification performance of these two types of CLIP. The experiment of training a mask proposal on top of ViT-based CLIP is also needed to evaluate the mask proposal performance of ViT-based CLIP. The results of the 1281 resolution (used in the main paper) are missing in Table 5 of Supp.\n\n3. Eq. 7 might have some typos. 'i' is the index of the mask, and 'i' should be substituted with the index of the class. Eq. 7 might be copied from F-vlm because in F-vlm,  'i' denotes the index of the class, which is correct. The author should check the formula more carefully.\n\n\n[1] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vlm: Open-vocabulary object detection upon frozen vision and language models. ICLR, 2023.\n\n[2] Ma, Chao et al. “Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models.” British Machine Vision Conference (2022).\n\n[3] Zhou, Ziqi et al. “ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation.” ArXiv abs/2212.03588 (2022): n. pag.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the constructive feedback and address the concerns below.\n\n> ***W1: Novelty, comparison to F-VLM***\n\nPlease refer to **C1: Relationship to F-VLM** and **C2: Novelty/Contribution**\n\n> ***W2: ViT-based or CNN-based CLIP for dense prediction***\n\nWe thank the reviewer for the questions, and address the concerns carefully below.\n\n**Many works [2][3] use ViT-based CLIP**: We thank the reviewer for the extra references, which we will add in the revision. We note that those papers [2][3] adopting ViT-based CLIP work *specifically* for semantic segmentation only, which often does not require a very high input resolution as it does not need to predict object-level masks with different scales (COCO dataset particularly contains many small objects). To be concrete, a typical ADE20K semantic segmentation setting is to use $512 \\times 512$ input resolution, while for COCO panoptic segmentation, it is usually $800 \\times 1333$ or $1281 \\times 1281$. The ViT-L/14 (with output stride 14) is sufficient under the relatively lower resolution $512 \\times 512$. Additionally, we note that state-of-the-art open-vocabulary semantic segmentation method SAN [24] also observes that ViT performs undesirable under higher resolution and thus they have to adopt a two-stage framework to feed different resolution images to the model. We quote from their paper: *\"Accurate semantic segmentation needs high-resolution images, but the released ViT CLIP models are designed for low-resolution images (e.g., $224 \\times 224$) and directly apply to high-resolution images giving a poor performance. To alleviate the conflicts in input resolutions, we use low resolution images in the CLIP model and high-resolution images in the side adapter network.\"*\n\n**Table 5 of Supp, the mask proposals are optimized with the CNN feature; mask proposals from ViT-based CLIP**: Regarding the evaluation of directly applying CNN-based and ViT-based CLIP as mask classifier, we use the **ODISE model as mask generator**, which is **not specifically trained for either CNN-based or ViT-based CLIP (but used as a separate module to ensure fairness)**, and thus the mask proposals were not optimized with CNN-based CLIP. Finally, we would like to note that false positive/negative proposals are a common issue in most segmentation models, and thus evaluating the CLIP model with a real mask proposal model (instead of ground-truth masks) is also practically important and reasonable. We will revise the draft to make it clearer.\n\n**Results of 1281 resolution are missing in Table 5 of Supp**: We thank the reviewer for the suggestion. We note that either 1280 or 1281 is not divisible by 14, and thus it is not applicable to use ViT-L/14. We provide more results under resolution 224 to 1120 below, where the 1120 is expected to provide an approximation result of 1280.\n\n|            |      | COCO |  PQ  | @res |      |    |      | ADE20K |  PQ  | @res |      |\n|------------|:----:|:----:|:----:|:----:|:----:|----|:----:|:------:|:----:|:----:|:----:|\n|            |  224 |  448 |  672 |  896 | 1120 | \\| |  224 |   448  |  672 |  896 | 1120 |\n| ViT-L/14   | 19.3 | 22.5 | 20.6 | 18.5 | 14.9 | \\| | 11.9 |  13.7  | 12.6 | 11.6 |  9.1 |\n| ConvneXt-L | 17.3 | 23.5 | 27.0 | 28.6 | 29.3 | \\| |  9.3 |  12.8  | 14.8 | 16.0 | 15.9 |\n\n> ***W3: Equation typos***\n\nWe sincerely thank the reviewer for pointing out the typos. We will fix them in the revision. Additionally, we would like to note that the geometric ensemble is not a specific design proposed by F-VLM. Instead, it is a common trick employed in many prior works such as [SAN, OVSeg, SimSeg, ViLD, ODISE].\n\n> ***Q1: ConvNeXt-L CLIP backbone comparison***\n\nWe thank the reviewer for the question. We use the ConvNeXt-L with \"laion2b_s29b_b131k_ft_soup\" weight in OpenCLIP. However, we respectfully disagree that ConvNeXt-L enjoys an advantage by pre-training with a higher resolution. In fact, as we work on object-level prediction, the dense feature map is required, instead of a globally pooled vector. Additionally, we note that only the last layer of the CLIP model can be used directly for classification due to the pre-training objective. Consequently, the ConvNeXt-L backbone leads to a ***32x*** downsampled feature map (i.e., $10 \\times 10$, if input size is $320 \\times 320$), while ViT-L/14 has a much larger feature map (i.e., $16 \\times 16$, even if input size is $224 \\times 224$), which actually favors ViT-L/14 when using them as dense feature extractors. This explains why ViT-L/14 performs better than CovnNeXt-L when input size is $224 \\times 224$ in Table 5 of Supplementary.\n\n> ***Limitation: Overlapping vocabularies***\n\nWe thank the reviewer for the question. We note that this is a common problem for most existing open-vocabulary segmentation works. It also exists in current benchmarks, e.g., in ADE20K, there are three different but semantically similar classes: *chair*, *armchair*, and *swivel chair*. FC-CLIP simply relies on CLIP to make predictions among the overlapping vocabularies. We agree with the reviewer that how to resolve such problems (e.g., building hierarchical vocabulary space) is an interesting future direction.\n"}]

Author rebuttal:
We appreciate all reviewers for their valuable suggestions, and we address the common concerns as follows. For the remaining concerns, please see the individual post for each reviewer.

>***C1: From reviewers N4iz W1, VW2t W1, Relationship to F-VLM***

We thank the reviewers for the suggestion. We note that F-VLM is a pioneering work that builds an open-vocabulary detection framework on top of a frozen CLIP backbone. However, FC-CLIP differs from it with a totally different observation and motivation, as detailed below.

Our work was initially motivated by the state-of-art open-vocabulary segmentation model ODISE, which found that the CLIP backbone extracts noisier features than diffusion models (see Figure B. 1. in ODISE paper), leading to inferior segmentation results (which justifies their adoption of diffusion models). Their observation motivated us to look deeply into the problem. Interestingly, our discoveries show that both ViT-based (used by ODISE) and CNN-based CLIP can produce semantic-meaningful features. However, when scaling up the input resolutions, we discover that ViT-based CLIP features turn noisier, while CNN-based ones are smoother and generalize better across input sizes.

Concurrently, F-VLM also empirically found that a frozen CLIP can provide meaningful features for object detection; however, they did not choose CNN-based CLIP on purpose and thus did not compare carefully between ViT-based and CNN-based CLIP backbones. On the other hand, in our paper, we have provided careful ablation studies on ViT-based and CNN-based CLIP in the Table 5 of Supplementary, where we observe that even though both ViT-based and CNN-based CLIP initially have comparable performance at resolution 224, CNN-based CLIP shows better and more robust performance when input resolution scales up. These important studies are missing in F-VLM.

Finally, even though we would like to provide more technical implementation differences from F-VLM, we note that in their github repository, they only released a demo and testing code (we note that their inference code was just open-sourced on **8/7/2023**, and their best backbone R50x64 is not available), which makes it hard to provide an in-depth comparison and also prevents the community from reproducing their results (please see F-VLM's OpenReview public comment, where people reported failing to reproduce the results). On the contrary, we promise to fully release all the training/testing codes of FC-CLIP (and best backbone) to facilitate the research in this area.

>***C2: From reviewers YCke Q1, VW2t W1, 5KPy W1, Novelty/Contribution***

To be concrete, we summarize our contributions/novelties as follows:

1. To the best of our knowledge, FC-CLIP is the first work that provides an in-depth analysis on adapting different types of CLIP model for downstream open-vocabulary segmentation tasks that require higher resolution inputs, while prior works (e.g., MaskCLIP, ODISE) simply favor a frozen CLIP model or Diffusion Model without looking into the difference between CNN-based and ViT-based CLIP models.

2. We look into the problem of extending the CLIP model for open-vocabulary segmentation, and identify the important problem of resolution discrepancy between the pre-training stage (image-text contrastive learning) and fine-tuning stage (open-vocabulary segmentation), and propose a simple and effective solution by adopting a frozen CNN-based CLIP model.

3. FC-CLIP is a simple and effective meta architecture that can be easily adopted on top of different segmentation frameworks. It not only achieves significantly better performance compared to prior state-of-the-art methods but also enjoys a much lower training/testing cost.

Meta-review: This work proposes an improved method for open-vocabulary panoptic segmentation, which results in significant improvements in accuracy, and training and inference speeds (7x) over the existing SOTA. The key research insight that enabled this is the authors' finding that the image encoder of a ConvNext-based CLIP model scales better to high resolution input images versus a ViT-based backbone, which, in turn, results in better accuracy for segmentation tasks.

The majority of the reviewers were positive (A, WA, WA, BA) except one Reviewer ***YCk*** (R). The latter's major concerns were (a) the lack of empirical comparisons to the existing work OpenSeg [28] and (b) lack of sufficient novelty. The AC and SAC discussed this paper in detail.

OpenSeg [28]'s code is not available and the authors have made an attempt to compare against a close proxy to OpenSeg [28] and shown their method to be better. The paper provides a clear technical novel insight that convolutional backbones from the CLIP encode can help address the issue of scaling up with image resolution, resulting in both significant improvements in accuracy and efficiency over the state-of-the-art for open-vocabulary segmentation. Hence, there is sufficiently novelty and strong experimental results in this work, which justify accepting it for NeurIPS 2023.

The AC and SAC's final decision is to accept this work. Congratulations!

The authors are strongly advised to incorporate the changes in the final manuscript that they promised in the rebuttal.
Decision: Accept (poster)