SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qihang Yu $^{1}$ , Ju He $^{2}$ , Xueqing Deng $^{1}$ , Xiaohui Shen $^{1}$ , Liang-Chieh Chen $^{1}$ $^{1}$ ByteDance  $^{2}$ The Johns Hopkins University
Open-vocabulary segmentation is a challenging task requiring segmenting and recognizing objects from an open set of categories in diverse environments. One way to address this challenge is to leverage multi-modal models, such as CLIP, to provide image and text features in a shared embedding space, which effectively bridges the gap between closed-vocabulary and open-vocabulary recognition. Hence, existing methods often adopt a two-stage framework to tackle the problem, where the inputs first go through a mask generator and then through the CLIP model along with the predicted masks. This process involves extracting features from raw images multiple times, which can be ineffective and inefficient. By contrast, we propose to build everything into a single-stage framework using a shared Frozen Convolutional CLIP backbone, which not only significantly simplifies the current two-stage pipeline, but also remarkably yields a better accuracy-cost trade-off. The resulting single-stage system, called FC-CLIP, benefits from the following observations: the frozen CLIP backbone maintains the ability of open-vocabulary classification and can also serve as a strong mask generator, and the convolutional CLIP generalizes well to a larger input resolution than the one used during contrastive image-text pretraining. Surprisingly, FC-CLIP advances state-of-the-art results on various benchmarks, while running practically fast. Specifically, when training on COCO panoptic data only and testing in a zero-shot manner, FC-CLIP achieve 26.8 PQ, 16.8 AP, and 34.1 mIoU on ADE20K, 18.2 PQ, 27.9 mIoU on Mapillary Vistas, 44.0 PQ, 26.8 AP, 56.2 mIoU on Cityscapes, outperforming the prior art under the same setting by +4.2 PQ, +2.4 AP, +4.2 mIoU on ADE20K, +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes, respectively. Additionally, the training and testing time of FC-CLIP is  $7.5 \times$  and  $6.6 \times$  significantly faster than the same prior art, while using  $5.9 \times$  fewer total model parameters. Meanwhile, FC-CLIP also sets a new state-of-the-art performance across various open-vocabulary semantic segmentation datasets. Code and models are available at https://github.com/bytedance/FC-clip.
Panoptic segmentation [44] is a complex computer vision task that aims to predict a set of non-overlapping masks, each with its corresponding class label. It combines the tasks of semantic segmentation [37] and instance segmentation [34], making it a challenging problem to solve. Many methods [43, 87, 18, 83, 51, 93, 20, 94, 53] have been proposed to tackle this problem, and a significant progress has been made in terms of panoptic quality (PQ). However, due to the high cost of annotating such a fine-grained dataset [54, 22], the number of semantic classes is typically limited to a few dozens or hundreds. This restriction hinders the further application of existing approaches to real-world settings, where the number of possible semantic classes is unlimited.
ViT-L-14 @224px
ViT-L-14 @448px
ViT-L-14 @672px
ViT-L-14 @896px
ViT-L-14 @1120px
ConvNeXt-L @224px
Figure 1:  $k$ -means visualization on top of frozen CLIP backbone features w.r.t. different input resolutions. Both ViT-based and CNN-based CLIP produces semantic-meaningful features. However, when scaling up the input resolutions, we note that ViT-based CLIP features turn noisier, while CNN-based ones are smoother and generalize better. The smoother feature map is preferable for mask-pooling modules in our design.
ConvNeXt-L @448px
ConvNeXt-L @672px
ConvNeXt-L @896px
ConvNeXt-L @1120px
To overcome the limitations of closed-vocabulary segmentation, open-vocabulary segmentation [48, 90, 29, 25] has been proposed. These approaches use text embeddings of category names [97], represented in natural language, as label embeddings, instead of learning them from the training dataset. By doing so, models can classify objects from a wider vocabulary, which improves their ability to handle a broader range of categories. To ensure that meaningful embeddings are provided, a pretrained text encoder [23, 70, 57, 69] is typically used. This encoder can effectively capture the semantic meaning of words and phrases, which is critical for open-vocabulary segmentation.
Multi-modal models, such as CLIP [69] and ALIGN [40], have shown promise for open-vocabulary segmentation due to their ability to learn aligned image-text feature representations from large-scale Internet data [74]. SimBaseline [90] and OVSeg [52] are two recent methods that use a two-stage framework to adapt CLIP for open-vocabulary segmentation. In these methods, images are first processed by a heavy mask generator [36, 20] to obtain mask proposals, and then each masked image crop is generated and fed into a frozen CLIP model for classification. MaskCLIP [25] extends this approach to open-vocabulary panoptic segmentation, but additionally leverages mask proposals as attention masks in the CLIP backbone to efficiently avoid multiple forwarding processes for the masked crops. More recently, ODISE [89] employs a stable diffusion UNet [72, 71] as a frozen backbone for mask generator, which significantly boosts the state-of-the-art performance. However, despite these advances, they still rely on a two-stage framework, where the mask generator and CLIP classifier extract features from raw images separately, resulting in inefficiency and ineffectiveness.
A natural question thus arises as to whether it is possible to unify the mask generator and CLIP classifier into a single-stage framework for open-vocabulary segmentation. Sharing the feature extractor between them is a straightforward solution, but it poses two challenges. First, fine-tuning CLIP backbone can disrupt the alignment between image and text features, resulting in a much worse performance on out-of-vocabulary categories. Existing methods [90, 52, 25, 89] rely on another separate backbone for mask generator, increasing model size and computational costs. Second, CLIP models are typically pretrained on relatively lower-resolution inputs, while dense prediction tasks require a much higher resolution for optimal performance. This makes it difficult to directly apply CLIP-pretrained backbones to downstream dense prediction tasks, particularly ViT-based CLIP models [26], where careful treatments are required (e.g., side adapter [17, 91], or cost aggregation [101, 21]). Consequently, existing methods [25, 89] perform mask segmentation and CLIP classification at different input scales, leading to sub-optimal performance.
To alleviate the two challenges, we propose to build both mask generator and CLIP classifier on top of a shared Frozen Convolutional CLIP backbone, resulting in a single-stage framework FC-CLIP. Its
design is based on the following observations. The frozen CLIP backbone ensures that the pretrained image-text feature alignment is intact, allowing out-of-vocabulary classification. It can also serve as a strong mask generator by appending a lightweight pixel decoder and mask decoder [20, 94]. The convolutional CLIP, based on a Convolutional Neural Network (CNN) [47], empirically shows a better generalization ability compared to ViT-based CLIP [26], when the input size scales up. This echoes the success of fully convolutional networks [60] in dense prediction tasks. Both observations are critical for developing a single-stage framework, but they have been overlooked and undiscovered by existing two-stage pipelines [25, 89]. In Fig. 1, we visualize the learned visual representation of ViT-based and CNN-based CLIP via  $k$ -means clustering [59]. As shown in the figure, the features learned by CNN-based CLIP are more robust across different input sizes.
Surprisingly, the adoption of a single frozen convolutional CLIP as the shared feature extractor results in an extremely simple yet effective design. Specifically, the single-stage FC-CLIP consists of three modules built upon a shared frozen convolutional CLIP backbone: a class-agnostic mask generator, an in-vocabulary classifier, and an out-of-vocabulary classifier (see Fig. 2 for comparison between pipelines). The proposed method not only enjoys a simple design, but also comes with a very low cost for both training and testing. As a comparison, our model has only  $238\mathrm{M}$  frozen parameters and  $21\mathrm{M}$  trainable parameters, against the state-of-the-art work ODISE [89] that has  $1494\mathrm{M}$  frozen and  $28\mathrm{M}$  trainable parameters. Furthermore, our model training only takes  $25.6\mathrm{V}100$  GPU days, which is  $7.5\times$  faster compared to ODISE's  $192\mathrm{V}100$  GPU days. During inference, our model also runs  $6.6\times$  faster. Although FC-CLIP enjoys a simple design, it still outperforms previous methods across multiple datasets. Trained on COCO panoptic dataset only, FC-CLIP surpasses prior state-of-the-art ODISE [89] significantly in a zero-shot manner. Specifically, FC-CLIP achieves  $26.8\mathrm{PQ}(+3.4)$ ,  $18.2\mathrm{PQ}(+4.0)$ , and  $44.0\mathrm{PQ}(+20.1)$  on ADE20K, Mapillary Vistas, and Cityscapes, respectively.
As panoptic segmentation unifies semantic and instance segmentation, FC-CLIP naturally extends to open-vocabulary semantic and instance segmentation. With the same model trained on COCO panoptic data only (i.e., no task-specific fine-tuning), FC-CLIP achieves state-of-the-art performance on open-vocabulary instance and semantic segmentation. Specifically, FC-CLIP achieves 16.8 AP on ADE20K, surpassing the state-of-art ODISE [89] by +2.4. FC-CLIP also outperforms the state-of-art specialized open-vocabulary semantic segmentation model SAN [91] by +1.1 and +1.1 mIoU on the challenging ADE20K-847 (A-847) and PASCAL-Context-459 (PC-459) benchmarks, respectively.
In summary, through the lens of a careful re-design of existing two-stage open-vocabulary segmentation models, we establish a simple, strong, and fast baseline for the community. The proposed FC-CLIP adopts a single-stage framework by exploiting a shared frozen convolutional CLIP, which not only advances the state-of-the-art performances on multiple benchmarks, but also enjoys a practically fast training and inference speed. We hope our study will inspire future research on efficient single-stage open-vocabulary segmentation models.
Vision-language models target at encoding vision and language jointly in a fusion model. Early works [78, 16, 98] extract visual representations by pretrained object detectors and fine-tune on downstream tasks with language supervision. Recently, with the breakthrough of large language models [23, 3], rapid progress has been made in this field. CLIP [69] and ALIGN [40] demonstrate that pretraining dual-encoder models with contrastive objectives on large-scale noisy image-text pairs can learn representation with cross-modal alignment ability and show strong performance in zero-shot downstream tasks. The following works [95, 1, 92] further confirm these points and achieve impressive results in zero-shot transfer learning such as open-vocabulary image recognition.
Closed-vocabulary segmentation can be divided into three types according to the semantics of the grouping pixels, i.e. semantic, instance and panoptic segmentation. Semantic segmentation interprets high-level category semantic concepts. Prior works [9, 72, 10, 11, 13, 28, 96, 86, 99, 30] mainly treat this task as a per-pixel classification problem and build their models on top of the idea of FCN [60]. Instance segmentation groups foreground pixels into different object instances. Starting from Mask R-CNN [36], prior works [42, 56, 12, 6, 2, 8, 80, 84, 66] mainly address this task with mask classification, where a set of bounding boxes and binary masks are predicted. Panoptic segmentation seeks for holistic scene understanding including both stuff and things. The pioneering work [44] and prevalent ones [55, 43, 87, 18, 50, 82, 14, 67] decompose the problem into various proxy tasks and merge the
Figure 2: Comparisons between open-vocabulary panoptic segmentation pipelines. Left: Existing methods [25, 89] adopt a two-stage pipeline, where the first stage employs a high-resolution image to generate class-agnostic masks, and the second stage feeds both the low-resolution image and predicted masks to a frozen CLIP backbone for open-vocabulary recognition. This incurs heavy computation, as image features are extracted multiple times. Middle: A naive single-stage framework builds everything together and fine-tunes the CLIP backbone, breaking the pretrained alignment between images and texts. Right: Our single-stage framework FC-CLIP employs a shared frozen convolutional CLIP, where "frozen CLIP" maintains the open-vocabulary recognition and can serve as a strong mask generator, and "convolutional CLIP" generalizes well to large input sizes. Note that the predicted masks are used for CLIP recognition in all three schemes (not shown for simplicity).
results in the end. Recently, following DETR [7], most works [83, 76, 19, 20, 51, 93, 94, 39, 49, 77] present end-to-end solutions based on the idea of mask classification. Standing on their shoulders, our proposed method builds on top of the pixel decoder and mask decoder of Mask2Former [20] by additionally exploiting the open-vocabulary recognition ability from CLIP [69].
Open-vocabulary segmentation aims at segmenting arbitrary classes including those that can not be accessed during the training procedure. Priors works [48, 29, 90, 52, 24, 88, 101, 91, 104, 62, 102, 32] perform open-vocabulary semantic segmentation through leveraging large pretrained vision-language models [69, 40, 71]. Recently, MaskCLIP [25] presents a two-stage pipeline, which consists of a class-agnostic mask generator and a frozen CLIP [69] encoder for cross-modal alignment, and thus expands the scope of the CLIP models into open-vocabulary panoptic segmentation. ODISE [89] digs out the innate potential of pretrained text-image diffusion models [71] in terms of the ability to present open concepts in the representation space for performing strong open-vocabulary panoptic segmentation. FreeSeg [68] encodes multi-granularity concepts into a compact textural abstraction, enabling generalizability to arbitrary text description. Unlike those methods, we propose a single-stage framework by exploiting a single frozen convolutional CLIP backbone, resulting in a simpler, faster, and stronger model than existing works.
We also note that the pioneering work F-VLM [46] builds an open-vocabulary detection framework on top of a frozen CLIP backbone. However, FC-CLIP differs from it with a totally different observation and motivation. Specifically, our work was initially motivated by the state-of-art open-vocabulary segmentation model ODISE [89], which found that the CLIP backbone extracts noisier features than diffusion models (Figure B. 1. in [89]), leading to inferior segmentation results (which justifies their adoption of diffusion models). Their observation motivated us to look deeply into the problem. Interestingly, our discoveries show that both ViT-based (used by ODISE [89]) and CNN-based CLIP can produce semantic-meaningful features. However, when scaling up the input resolutions, we discover that ViT-based CLIP features turn noisier, while CNN-based ones are smoother and generalize better across input sizes. F-VLM [46] also empirically found that a frozen CLIP can provide meaningful features for object detection. However, they did not choose CNN-based CLIP on purpose and thus did not compare carefully between ViT-based and CNN-based CLIP backbones. On the other hand, in our paper, we have provided careful ablation studies on ViT-based and CNN-based CLIP, where we observe that even though both ViT-based and CNN-based CLIP initially have comparable performance at resolution 224, CNN-based CLIP shows better and more robust performance when input resolution scales up.
Figure 3: Overview of FC-CLIP, which contains three main components: mask generator, an in-vocabulary (in-vocab) classifier, and an out-of-vocabulary (out-vocab) classifier. All components build on top of a shared frozen covolutional CLIP backbone. The pixel decoder and mask decoder follow the design of Mask2Former, and generate class-agnostic masks. The in-vocabulary classifier yields the class embeddings by mask-pooling over final pixel features from pixel decoder. During testing, FC-CLIP additionally exploits the out-of-vocabulary classifier by mask-pooling over frozen CLIP backbone features, and the final class prediction is obtained by geometric ensembling both classifiers. Note that the text embeddings are obtained by feeding category names into a CLIP text encoder, which are done beforehand and cached in memory, thus causing no additional costs. Also, the class-agnostic mask proposals are fed to the mask pooling modules (not shown for simplicity).
In this section, we first define the problem of open-vocabulary segmentation. We then introduce the existing two-stage pipeline, followed by our proposed single-stage framework FC-CLIP.
Problem Definition Open-vocabulary segmentation aims to segment the image  $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$  into a set of masks with associated semantic labels:
$$
\left\{y _ {i} \right\} _ {i = 1} ^ {K} = \left\{\left(m _ {i}, c _ {i}\right) \right\} _ {i = 1} ^ {K}. \tag {1}
$$
The  $K$  ground truth masks  $m_{i} \in \{0,1\}^{H \times W}$  contain the corresponding ground truth class label  $c_{i}$ . During training, a fixed set of class labels  $C_{train}$  is used, while during inference, another set of categories  $C_{test}$  is used. In the open-vocabulary setting,  $C_{test}$  may contain novel categories unseen during training, i.e.,  $C_{train} \neq C_{test}$ . We follow previous works [25, 89] and assume the availability of the category names of  $C_{test}$  (represented in natural language) during testing.
Two-Stage Open-Vocabulary Segmentation Existing works [90, 52, 25, 89] adopt a two-stage pipeline for open-vocabulary segmentation. The first stage contains a class-agnostic mask generator  $\mathcal{M}$  with parameters  $\theta_{\mathcal{M}}$  that generates a set of  $N$  mask proposals  $\{\hat{m}_i\}_{i=1}^N \in \mathbb{R}^{N \times H \times W}$ , given the input image  $\mathbf{I}$ :
$$
\left\{\hat {m} _ {i} \right\} _ {i = 1} ^ {N} = \mathcal {M} (\mathbf {I}; \theta_ {\mathcal {M}}). \tag {2}
$$
In the second stage, a CLIP adapter  $\mathcal{P}$  takes both image I and mask proposals  $\{\hat{m}_i\}_{i = 1}^N$  as inputs, where the latter input is used to guide the frozen CLIP model  $CLIP^{*}$  (\* denotes frozen). The adapter performs mask classification through forwarding processes with either masked crops [90, 52] or masked attention [25, 89]:
$$
\left\{\hat {c} _ {i} \right\} _ {i = 1} ^ {N} = \mathcal {P} \left(\mathrm {I}, \left\{\hat {m} _ {i} \right\} _ {i = 1} ^ {N}; C L I P ^ {*}\right), \tag {3}
$$
where  $\{\hat{c}_i\}_{i=1}^N \in \mathbb{R}^{N \times |C|}$  refers to the predicted class probabilities for the  $N$  predicted masks,  $C \in \{C_{train}, C_{test}\}$  depending on training or testing phase, and  $|C|$  is the category size.
Although this framework has achieved impressive open-vocabulary segmentation performance, it has two limitations. First, the image features are extracted twice, once for mask generation and the other for mask classification. The double feature extractions incur heavy computation, making it costly to scale up backbone parameters. Second, the mask generator often requires high-resolution inputs (e.g.,  $1024 \times 1024$ ), whereas the CLIP model is usually pretrained with lower-resolution images (e.g.,
$224 \times 224$ ). The two-stage pipeline thus needs to feed high-resolution images into the mask generator and low-resolution images into the CLIP classifier, making the model inefficient.
Naïve Single-Stage Open-Vocabulary Segmentation To avoid increasing the model size and computational cost of duplicate feature extractions, one may naively formulate everything together into a single-stage framework  $\mathcal{F}$ , where both mask generator and mask classifier share the same CLIP-pretrained backbone  $CLIP$  (not frozen) for extracting features from an input image  $\mathbf{I}$ :
$$
\left\{\hat {m} _ {i}, \hat {c} _ {i} \right\} _ {i = 1} ^ {N} = \mathcal {F} (\mathbf {I}; C L I P, \theta_ {M}). \tag {4}
$$
However, we empirically discover that fine-tuning this naive single-stage framework causes a misalignment between image and text features in the pretrained CLIP model, leading to sub-optimal performance, especially for novel unseen classes. It also increases the training costs by  $2.1 \times$  to 52.8 GPU days. Interestingly, our experiments also show that a frozen CLIP backbone can provide sufficient features for mask generation, while preserving the image-text aligned representation. Nevertheless, we still face another challenge, where CLIP models are usually pretrained on low-resolution images (e.g.,  $224 \times 224$ ), whereas segmentation models prefer higher-resolution inputs (e.g.,  $800 \times 1333$  for COCO, or  $1024 \times 2048$  for Cityscapes). This discrepancy results in the significant performance degradation, when applying a frozen CLIP on large input images. Digging into the details, we found that it is related to the popular ViT [26] backbone used in CLIP that does not transfer well to different input sizes, which could be alleviated by extra careful designs (e.g., side adapter [17, 91], or cost aggregation [101, 21]). On the other hand, CNN-based CLIP models (such as ResNet [35] and ConvNeXt [58]) exhibit better generalization ability to different input sizes, due to their fully convolutional nature [60]. Additionally, the CNN-based CLIP backbone, extracting multi-scale feature maps, can be used as a simple plug-in module into modern closed-vocabulary segmentation models [20, 94]. Motivated by the observations, we thus propose FC-CLIP, a simple yet effective single-stage open-vocabulary segmentation framework built entirely on a single frozen convolutional CLIP backbone  $CLIP_{CNN}^{*}$ :
$$
\left\{\hat {m} _ {i}, \hat {c} _ {i} \right\} _ {i = 1} ^ {N} = \mathcal {F} \left(\mathbf {I}; C L I P _ {C N N} ^ {*}, \theta_ {M}\right). \tag {5}
$$
FC-CLIP The proposed FC-CLIP leverages the semantic features of a frozen CNN-based CLIP backbone for both mask generation and CLIP classification. Unlike previous works [90, 52, 25, 89], which often train a separate mask generator and ignore the potential reuse of CLIP's semantic features, we incorporate the CNN-based CLIP backbone into the state-of-the-art segmentation method Mask2Former [20]. We note that FC-CLIP is a general meta-architecture that can build on top of several modern segmentation methods [20, 94]. Our approach offers several advantages. By freezing and sharing the backbone features, our model is significantly more efficient during both training and testing (i.e., avoiding feature duplication). The CNN-based CLIP backbone not only transfers well to different input resolutions (from its pretrained image size), but also generates multi-scale feature maps, seamlessly compatible with modern segmentation methods [20, 94]. At a high level, FC-CLIP consists of three components: class-agnostic mask generator, in-vocabulary classifier, and out-of-vocabulary classifier. We detail each component below.