CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction
Authors: Size Wu
Wenwei Zhang
Lumin Xu
Sheng Jin
Xiangtai Li
Wentao Liu
Chen Change Loy
Venue: ICLR 2024 spotlight (2024)
Keywords: open-vocabulary object detection
open-vocabulary image segmentation
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Open-vocabulary dense prediction tasks including object detection and image segmentation have been advanced by the success of Contrastive Language-Image Pre-training (CLIP). CLIP models, particularly those incorporating vision transformers (ViTs), have exhibited remarkable generalization ability in zero-shot image classification. However, when transferring the vision-language alignment of CLIP from global image representation to local region representation for the open-vocabulary dense prediction tasks, CLIP ViTs suffer from the domain shift from full images to local image regions. In this paper, we embark on an in-depth analysis of the region-language alignment in CLIP models, which is essential for downstream open-vocabulary dense prediction tasks. Subsequently, we propose an approach named CLIPSelf, which adapts the image-level recognition ability of CLIP ViT to local image regions without needing any region-text pairs. CLIPSelf empowers ViTs to distill itself by aligning a region representation extracted from its dense feature map with the image-level representation of the corresponding image crop. With the enhanced CLIP ViTs, we achieve new state-of-the-art performance on open-vocabulary object detection, semantic segmentation, and panoptic segmentation across various benchmarks. Models and code are released at https://github.com/wusize/CLIPSelf.

Published as a conference paper at ICLR 2024
CLIPSELF: VISION TRANSFORMER DISTILLS ITSELF FOR OPEN-VOCABULARY DENSE PREDICTION
Size Wu'
Wenwei Zhang'
Lumin Xu²
Sheng Jin
Xiangtai Li
Wentao Liu 4.5
Chen Change Loy'
S-Lab, Nanyang Technological University 2 The Chinese University of Hong Kong
The University of Hong Kong SenseTTme Research and Tetras. 'Shanghai Al Laboratory
i ze000 de ntu. edu.sg
ABSTRACT
Open-vocabulary dense prediction tasks including object detection and image seg- mentation have been advanced by the success of Contrastive Language-Ima Pre-training (CLIP). CLIP models, particularly those incorporating vision trans- formers (ViTs), have exhibited remarkable generalization ability in zero-shot age classification. However, when transferring the vision- vision-language alignment CLIP from global image representation to local region representation for the open-vocabulary dense prediction tasks, CLIP ViTs suffer from the domain shift from full images to local image regions. this paper, we embark on an in-depth analysis of the region-langen language alignment in CLIP models, which is essential for downstream open-vocabulary dense prediction tasks. Subsequently, we propose approach named CLIPSelff, which adapts the image-level recognition ability of CLIP VIT to local image regions without needing any region-text pairs. CLIP- Self empowers ViTs distill itself by aligning a region representation extracted from its dense feature map with the image-level representation of the correspond- ing image crop. With the enhanced CLIP ViTs, we achieve new state-of-th the art performance on open- neen-vocabulary object detection, semantic segmentation, and panoptic segmentation across various benchmarks. Models and code are released at hnttps://qithui com was wtt.aiiieccLLLPllll
INTRODUCTION
Dense prediction tasks, including object detection Girshick 2015 Ren et al, 2015) and segmen- tation (Hariharan et al. 2014, Kirillov et al., 2019), , have been significanlly advanced in the era of deep neural networks. However, traditional detection and segmentation models trained to recog- nize only a fixed set of object categories. Such design restricts the real-world applications of these models where infinite visual concepts exist. Therefore, open-vocabulary object detection (Zareian et al. 202I) and image segmentation (Ghiasi et al., 202I). which require the detection and segmen- lation models to recognize and localize visual concepts unseen in the training datasets, are gaining increasing attention from the community.
Recent open-vocabulary approaches (Gu al. 2021 Xu al. 2022 Kuo al. 2023 Xu al.
2023b) are typically inspired by the Contrastive Language-Iman Pre- Pre-training (CLIP) RRafford al.) 202T). As depicted in Fig. [a), CLIP models, particularly the variants incorporating vision transformers (ViTs), have demonstrated impressive generalization capabilities in image classifica- tion tasks, achieving exceptional zero-shot performance. To enable apen-vocabulary object detection and segmentation, is crucial to transfer the vision- languaelagguage alignment of CLIP models, especially the powerful ViT-based variants, from full images to local image regions.
In light of this, we conducted preliminary experiment to evaluate the region-language alignment of CLIP's dense features and assess their competence in local object recognition. As shown in Fig. [bb, two approaches were tested using the region boxes annotated in the COCO dataset (Lin ct al. 2014). The first approach, referred to as 'Image Crop', directly inputs the image crops that enclose the regions into CLIP and utilizes the image-level features for classification. The second

---

Published as a conference paper at ICLR 2024
(a) Image Classification ImageNet (b) Region Classifivation COCO
(0) K-Means Visualization of Dense Feature Map
Emage
RN50x64
ViT-L/14
80.4
Image Crop
Feature
73.1
73.6
66
66.9
RN500664 VIT-L114
RN50X64 VIT-L/14 RN50X64 VIHL/14
Figure 1: (a) CLIP ViTs exhibit excellent zero-shot ability on image classification compared with CLIP CNNs. (b) To classify regions, CLIP ViT is as effective CLIP CNN by separately clas- sifying the image crop of each region. However, it struggles when extracting region representation from the dense feature map for recognition. (c) The K-Means results of the CLIP ViT's dense fea- ture are much noisier, demonstrating the inferiority of CLIP viT's dense representation.
IE
CLIP's Image Eicodder
IE
IE
Dense Feature Teach
Dense Feature
Teach
TE
CLIP's Text Encoder
A monitor
+ TE
IE
Reyboard
Text
Patch
Parumeteess
Embeddings
Emboddings
Frozen
(4) Using region-text puirs
(b) ACLIPSelf
Figure 2: (a) Using region-text pairs fine-tune CLIP for dense prediction tasks. These pairs either manually annotated or generated via matching between region proposals and parsed image captions. (b) Our CLIPSelf does not rely the association between text descriptions and regions, and only uses CLIP ViT's representations of image patches to learn the dense features.
approach, referred to as 'Dense Feature', first obtains the dense feature map of the input image and then extracts region representations from the feature map for recognition. Our findings reveal that the dense features of the Convolutional Neural Network (CNN) based model (RN50x64) exhibit high proficiency in region classification, surpassing the approach that uses image crops. This suggests the potential of directly applying CNN-based CLIP models to open-vocabulary dense prediction tasks. In contrast, the ViT-based model (ViT-L/14) struggles with region recognition when using its dense features, despite achieving satisfactory accuracy when utilizing image-level representations of the corresponding image crops. Furthermore, the K-Means visualization of the feature maps in Fig. [Tc) provides qualitative evidence of the inferior dense representation of ViT-based CLIP mod- els. Compared to CNN models, CLIP ViT lacks local inductive bias, hindering the smooth transfer from representing pixels of whole image representing pixels of local image region. Indeed, each spot on the dense feature map of the CLIP ViT tends to encode the global image, which is desired for local region recognition. Relevant analyses are provided in Sec. A.I Meanwhile, notable progress has been made in building open-vocabulary object detectors based on frozen CLIP CNNs (Kuo al. 2023 Wu et al. 2023c Xuet al., 2023c), demonstrating competitive performance. However, applying CLIP VIT's to dense prediction tasks has proven challenging and less straight- forward (Zhou al. 2022a Ding al., 2023) Xu et al., 2023b). Therefore, we aim to develop an effective and general solution address the limitations of CLIP ViTs' dense representation.
One intuitive approach to enhance CLIP ViTs for dense prediction tasks involves fine-tuning CLIP using region-text pairs as illustrated in Z which establishes a direct alignment between region and language representations. However, annotating sufficient region-text pairs for training a robust region representation is resource-intensive. To address this challenge, RegionCLIP (Zhong et al. 2022) has explored the generation of pseudo labels by matching object nouns extracted from image captions with region proposals, thereby forming region-text pairs. While eliminating the need for exhaustive annotation, this approach suffers from the noisy matching between regions and object nouns. Compared with the probably imprecise text description (object noun) of region, the image-
'The dense feature map of a CL.IP VIT is obtained following Zhou et al. 2022a which modifies the output layer of VIT for better laxel-language alignment.
2

---

Published as a conference paper at ICLR 2024
level representation of the image crop enclosing the region could serve more reliable teacher to guide the enhancement of the region representation in the context of CLIP ViTs as shown in Fig. [
In this paper, we present CLIPSelf, a self-distillation approach that obviates the necessity of paired data associating text descriptions with image regions. Fig. (()) provides an overview of how CLIP- Self derives representation conducive dense prediction through self-distillation. Specifically, CLIPSelf fine-tunes CLIP ViTs by maximizing the cosine similarities between the region represen- tations pooled from the dense feature map and the image representations of the corresponding image crops. The regions in our method can be obtained by partitioning an image into a grid of m x n patches. The unique design of CLIPSelf eliminates the need for complex region-text matching process or the acquisition of additional labeled region-tex pexoontexttoo pairs while effectively bridging the gap between dense and image representations of the CLIP ViTs for region recognition. Consequently, CLIPSelf significantly strengthens the vision- vision-language alignment of the CLIP ViTs' dense features, benefiting their application to downstream open- vocabulary dense prediction tasks.
The effectiveness of CLIPSelf is validated on open-vocabulary object detection and image seg- mentation benchmarks. For open-vocabllary object detection, we established a two-stage baseline based on frozen CLIP ViTs, and the fine-tuned models achieved state-of-the-art performance on OV-COCO and OV-LVIS benchmarks, as well as on the transfer detection benchmark. For open- vocabulary semantic and panoptic segmentation, CLIPSelf also yields non-trivial improvements to current state-of-the-art methods, such as Cat-Seg (Cho et al. 2023) and ODISE (Xu et al. 2023a).
2 RELATED WORK
Open- Vocabulary Dense Prediction. These directions primarily include object detection Zareian ict al. 202T, Gu et al. 2021 Kuo et al. 2023) and image segmentation (Ghiasi et al.) 2021 Xu
iet al. 2023a; 2022). aiming recognize local visual concepts of arbitrary categories described by texts. The impressive vision-language alignment brought by Contrastive Image-Language Pre- training (CLIP) (Radford et al.] 2021) has inspired numerous studies explore the downstream application of CLIP features to these tasks. For open-vocabulary detection, a series of works (Gu et al. 2 et. 2023a) distill knowledge from the CLIP models recognize novel object con- cepts. There are also works (Kuo et al. 20233 Wu et al.) 2023c that directly build object detectors upon frozen CLIP CNNs. For apen-vocabular segmentation, the typical two-stage approaches (Xu eet al. 2022; Ding et al. 2022 Xu ct al. 2023a) first generate class-agnostic mask proposals and then classify the proposals with CLIP. Particularly, concurrent work ZeroSeg (Chen et al., 2023) distills CLIP's representation on image patches to the semantic segmentation model, resembling CLIPSelf in the labelling-free free nature. However, we would like to categorize ZeroSeg into downstream appli- cations of CLIP that transfer the knowledge of CLIP to specific dense prediction models. In contrast, CLIPSelf, which facilitates the transfer knowledge from image to local regions within the CLIP ViTs in self-distillation manner, is posited between the upstream image- mexte-texttt pre-training and the downstream applications, and generally applicable to various downstream tasks.
Vision-Language Alignment for Images and Regions. Vision-language pre-training has given risc to models with aligned image and text representations (Frome et al. 2013 Jayaraman & Grauman Radford al. (2021) Recent studies on contrastive vision- language pre-tnaining (Jia et al. 2021 Radford et al. 2021 Zhai et al. 2022) have significantly
2014 Jia et al. 2021 Kim et al.
202T
improved the generalization ability of recognition models. In particular, CLIP models (Radford) et al. (02T) that are pre-trained on billion-scale image-text pairs have exhibited impressive zero- shot classification performance on a wide range of datasets. Motivated by the success aligning image and text representations, many studies (Zhong et al. [022]] Li al. 2022at Liu et al.) 2023
Zhang et al. 2022b Mukhoti et al. 2023 Wu et al.] 2023b) have sought to achieve vision-language
alignment at the local regions of images for dense prediction tasks. Some works learn region-text
alignment using annotations in visual grounding datasets (Liu et al.] [2023] Krishna et al. 2017
Plummer et al. 2015) generating pseudo region-text pairs Zhong et al. (2022 Wu et al. (2023b) approaches (Gupta et al.) 2020; Mukhoti et al. 2023) that indi-
There are also weakky-superrvise rectly align region and language representations using only image-text pairs. Different from these studies, CLIPSelf facilitates the transfer of a CLIP ViT's global vision- Languag alignment to local regions by self-distillation, process that circumvents associating regions with texts.
3

---

Published as a conference paper at ICLR 2024
Vision Transformers in Open- Vocabulary Learning. For open-vocabulary or zero-shot image recognition, vision transformer (ViT) based vision-language models have demonstrated superior ca- pability (Radford al. 2021 Sun et al. 2023). However, ViT's have shown inferior
alignment in the context of open-vocabulan ary dense prediction, despite their success in standard dense prediction tasks (Liu et al. 2021 Zhang al. 202T;] et al. 2022b) Liu et al. 2022 Zhang al.) 2022ab. Recent studies (Zhou et al. 2022a| Ding et al., 2023) have attempted to improve the CLIP ViTs' vision-language alignment on the dense features by modifying the output layer of ViTs or employing masked attention. However, such attempts have yielded sub-optimal results. There are also intection-orientte Swin-based foundation models that learn agiionlanguage grounding from region-text pairs, e-g.. GLIP (Li et al. 2022a; Zhang et al. 2022b) and Grounding DINO (Liu jet al. 2023). However, these works only treat Swin as a visual backbone and separately learn the region-language grounding on a cross-modality head, without explicitly exploiting vision-language alignment in the representation of Swin Transformers. In the more relevant open- vocabulary de- tection literature, few works (Minderer al. 2022; Kim al. 2023b a) have sought to craft the vision-language pre-training of ViT-based open- ggen-vocabulary detectors by scaling up image-level su- pervision provoking region awareness in the architecture of ViTs. For instance, CFM-VIT (Kim) et al.) 2023a) adopts random feature masking and window attentions in the ViTs to improve localiza- tion ability. However, such works still fall short of explicit region-language alignment, because the vision-language pre-training is conducted only on image-text pairs. To the best of our knowledge, CLIPSelf is the first work that explicitly injects strong vision-lang language models. Furthermore, an enhanced variant of CLIP ViT with local window attention has been explored to further validate the promising applicability of CLIPSelf.
alignment into the ViT-based
3 METHODOLOGG
3.1 IMAGE REPRESENTATION V.S. DENSE REPRESENTATION
ViT-based CLIP model comprises a series of residual attention blocks. We briefly explain how the image and dense representations are obtained from the last residual attention block.
CLIP's Image Representation. The input to the last residual attention block a (Io, 31, .... = representing a class embedding xp and h w image embeddings {ara (1,2, h x w). A residual attention block z (2)Attn(xr) can be written as:
q = -Eb, (x)) & Emb, (ar), v- = Emb, (x)
y = + Proj Proj(SoftMax( gk
)u),
FFN(Y),
where es constant, Proj represents projection layer, Emb comprises a layer norm and a projection layer, and FFN stands for a feed-forward network. Finally, the updated class embedding is used to represent the whole image: Tinuge
2 20
CLIP's Dense Representation. To extract the dense feature map from a CLIP ViT, we follow (Zhou et al., 2022a) to slightly modify the last residual attention block. Specifically, we keep the projection layers, layer norms, and FFNs while discarding the self-attention. The modified residual attention block z' ModifiedResAttt((f can be written as:
Emb, (x), y' = x + Proj(v), = + FFN(3').
We discard the class embedding z' [0) and reshape the image embeddings z'(1 : h x w/) into an h x w feature map Xdense. from which we can extract representations for boxes masks by RolAlign or mask pooling (He et al. 2017).
Discussion and Motivation of CLIPSelf. Although the dense features have been used in existing works (Zhou et al. 2022a Wu et al. 2023a; Cho et al. 2023) to extract box or mask representations for dense prediction tasks, we make the first attempt to provide representation. Specifically, we compare the recognition ability of image representation and dense representation by using them classify the region boxes annotated in the COCO dataset (Lin al.) 2014). Given an image and a region box annotation, the dense representation of the region can be
in-depth analysis of the dense
Thhre is linear output layer following the last attention module. We ignore for brevity.
4

---

Published as a conference paper at ICLR 2024
(a) Motivation of CLIPSelf. Image Represeniation v.s. Dense Representation
(b) Our Method: CLIPSell
100
100
VIT-8/16
mAAe
VI--//14
Pooling
LE
80
Stadent
Feature
M.N 14SAAK
Sample
Initialize
Student Finhekdings
60
50
IE
MALIOR Coniee Steslerty
20
Irput Imagee Size
Input Image Sipe
640
224
672
896
Top5
Topl
Image Crop
Dense Feature
Image Putches
Teacher
Teacher Embeddings
Figure 3: (a) Region classification using image representation (blue) and dense representation
(green) of CLIP ViTs. The y-axis stands for the mean accuracy (mAcc). The x-axis the input image size for obtaining dense feature maps (green). The input size for image representation (blue) of the image crops is fixed at 224 x 224 for ViT-B/16 and 336 x 336 for ViT-L/14. (b) CLIPSelf randomly splits an image into patch regions for self-distillation Then it aligns the region represen- tation pooled (by RoLAlign) from the dense feature map of the student to the corresponding image representation of the Teacher. Teacher: the original CLIP ViT; Student: the fine-tuned CLIIP VřT.
extracted from &denso by pooling (RolAlign). For image representation, we crop the region box from the image first and then send it to the CL.IP model to obtain Timage-
As shown in Fig. 3 the image representation (blue) outperforms the dense representation (green) on both Topl and Top5 classification accuracy by a considerable margin across all input sizes. is noticeable that the performance of ViTs' dense representation does not grow with the input image size, which hinders the use of the models for downstream tasks like object detection and image segmentation, where large image resolution is always desired. Please refer to Sec. A for more relevant analyses and discussions on this phenomenon. Based on the results in Fig. a(), we believe the region representations extracted from the dense feature map can be improved by aligning them to the image representations of the corresponding image crops.
3.2 CLIPSELF
We propose a simple self-distillation approach, CLIPself, that allows CLIP ViTs to teach themselves by aligning the region representations pooled from dense feature maps to the image representations of the corresponding image crops as shown in Fig. b). We denote the original CLIP model as Teacher and the fine-tuned CLIP model as Student. The weights of the Teacher are fixed, while
the weights of the Student are initialized with the weights of the Teacher. To train the Student, we partition the image into regions and align the Student's s dense representations of the regions to the corresponding image representations of the Teacher.
Image Patches as Regions. For self-distillation purposes, we divide an image into grid m n patches. During each training iteration, and n are randomly selected from the set {1, M) to allow different patch sizes. Empirically, we set M = in our implementation. Experiments show that such simple patch sampling scheme already enhance the Student's dense representation remarkably. Moreover, the grid image patches are more effective in covering background content (referred to as 'stuff") than region proposals generated by external models, which predominantly focus on foreground objects (referred to as 'thing'). This is validated by the results of classifying stuff masks in Tab. 2
Self-Distillation. Given an image, we can obtain the dense feature map Xdense from the Student as described Sec. 3.1 Given the m n patch regions, the Student embedding of patch Pi (i (0, ... m 1),j E (O.... ..., n 1}) can be denoted as denss Pooling( (Xidesee pks), Correspondingly. the teacher embedding Wi is obtained by sending PV to the Teacher for image representation. Consequently, the self-distillation loss to align student and teacher embeddings can be written as:
L m-1n-1
ij dense densol
immaee
m X
1=0 1=
5

---

Published as a conference paper at ICLR 2024
Table I: Ablation study on the design choices of CLIPSelf. The row with blue color is our default
choice in the main experiment. The experiments are on ViT-B/16 from EVA-CLIP. #1 in all the sub-tables stands for the original EVA-CLIP ViT. #5* (c) stands for the sanity check.
(a) Number of image patches
(b) Number of learnable layers
(c) Input image size of Student
Training Patch Split
Mcan Accuracy
Learnable Layers
Mcan Accuracy
Input
Mcan Accuracy
Top1
Top5
Top1
Top5
Image Size
Top1
Top5
18.2
33.2
18.2
33.2
18.2
33.2
M=4
71.3
90.1
45.0
71.1
320
46.5
70.1
M=
72.1
91.3
3
59.4
82.3
640
67.1
87.7
M=8
71.6
91.1
4
9
68.7
88.7
1024
72.1
91.3
M=10
70.0
90.2
S
12
72.1
91.3
5*
1024
52.3
76.4
3.3 APPLICATION TO OPEN-VOCABURE VOCABULARY DENSE PREDICTION
For open-vocabulary object detection, build two-stage detector frozen CLIP ViT backbone and only train the detection heads following F-VLM (Kuo et al. [023). And we replace the backbone with the CLIPSelf fine-tuned model. As the detection task primarily targets at foreground objects, we keep using region proposals as an option for implementing CLIPSelf in the context of open- vocabulary object detection. Please refer to the appendix AA.3 for the details of the detector. For semantic segmentation, our fine-tuned CLIP ViTs can serve as better initialization for the backbones of Cat-Seg (Cho et al. 2023). For panoptic segmentation, our fine-tuned ViT can be applied the inference stage of ODISE (Xu et al. 2023a) to improve open- vocablocyy ability.
4 EXPERIMENTS
4.1 ABLATION STUDY OF CLIPSELF
Experiment Setting. To train CLIPSelf, use 8 A100 GPUs and set the batch size 2 on each GPU. We train the models for 6 epochs using the AdamW (Loshchilov & Hutter] (017)) optimizer with learning rate of 1e-5 and weight decay of 0.1. By default, we use the images in train2017 split of COCO dataset (Lin et al. 2014), which are exactly the training images of most downstream open-vocabulary benchmarks. All experiments in Sec. [4.] are conducted on the ViT-B/16 from E.VA-CIIP (Sun et al. 2023) considering its high efficiency and capacity, and we assume using image patches for self-distillation unless otherwise stated. The mean accuracy (mAcc) of classifying region boxes annotated in COCO's va12017 split is used as the indicator for evaluation.
Design Choices. We conduct ablation studies on design choices of CLIPSelf, namely patch split, trainable layers, and input size of the Student. For the patch split, image is split into grid of m X n patches, where m and n are randomly sampled from {1, M). In Tab. Ia we show the results when M is set 4, 6, and 8. The results support the highest accuracy. For the number of trainable layers, we experiment with updating the last 3, 6, 9, and 12 attention layers of ViT-B/16. As shown in Tab. Tb the region classification accuracy consistently improves with more trainable layers. Therefore, we choose to update all the attention modules in our implementation of CLIPSelf. For the input size of the Student, we experiment with different input image sizes (320 x 320, 640 x 640, 1024 x 1024). The input to the Teacher is fixed 224 x 224. For non-square images, we pad zero values the bottom and right of the images after color normalization. As shown in Tab. IC training with larger images for the Student improves the performance of region classification. Therefore, we set 1024 x 1024 as the default input size for the Student. However, considering the memory cost, we do not further increase the image size. For the input size and trainable layers on ViT-I/14, please refer to the appendix A.2
choice of M = 6 as achieves
Sanity Check on Input Image Size. The aforementioned ablation study input image size of the Student reveals the efficacy of training with larger inputs, which raises a question on whether the improvement of CLIPSelf is merely contributed to training the Student with higher image resolution instead of the region-wing assess to what extent the dense representation is enhanced by merely increasing the image size. Specifically, we train the Student with input size 1024 x 1024 using the image-level supervision from COCO Caption dataset (Chen et al. 2015). As shown in Tab. [c(#5), image-level supervision
supervision in the self-distillation. Therefore, we add a sanity check to

---

Published as a conference paper at ICLR 2024
Table 2: Enhancement of dense representation. We report the Top1 and Top5 mean accuracy on classifying boxes and panoptic masks (thing and stuff).
Boxes
Thing Masks
Stuff Masks
#
Model
Method
Region Proposals
Top1
Top5
Top1
Top5
Top1
Top5
1
ViT-B/16
18.2
33.2
20.6
36.5
18.4
43.5
2
VIT-B/16
CLIPSelf
x
72.1
91.3
74.4
91.8
46.8
80.2
3
VIT-B/I6
CLIPSelf
/
74.0
92.6
76.3
92.8
36.8
75.0
Images
Original
Ours
Images
Original
Ours
Figure 4: K-Means visualization of the dense feature maps of CLIP ViT. We show the raw images, the K-Means results of the original model, and those of our fine-tuned model by CLIPSelf.
with large input size only improves the Top1 mAce to 52.1%, lagging behind the result of our self- distillation approach (72.1%) by a considerable margin.
4.2 ENHANCEMENT OF DENSE REPRESENTATION BY CLIPSELF
Quantitative Evaluation. We conduct a comprehensive evaluation on the enhancement of CLIP VIT's dense representation. In addition to the region box classification introduced in Sec. 4.1 we also report the mAcce of classifying panoptic masks (thing and stuff) annotated in COCO Panoptic dataset (Kirillov al., 2019). The mask embeddings for classification are extracted from the dense feature maps of the CLIP VIT by mask pooling (He et al. (017). As shown in Tab. 2 &#2). CLIP- Self not only improves the ViTs' recognition ability for region boxes but also for panoptic masks, which establishes CLIPSelf as general solution to both open- vocabulay object detection and open- vocabulary image segmentation. Further results of various ViT variants are in the appendix A.2
Using Region Proposals. We also implement CLIPSelf using region proposals generated by gion proposal network (RPN) trained on COCO's aaatrrinn00 split. To satisfy the vocabulayy setting, the RPN is trained solely using annotations of the 48 base object categories defined in OV- COCO. As shown in Tab. #3), leveraging region proposals boosts the classification accuracy for foreground instances, including object boxes and thing masks. However, this approach exhibits duced proficiency in recognizing background contents (stuff masks) since the training of CLIPSelf has primarily emphasized foreground instances. Consequently, the utilization of region proposals is considered an alternative option only for open-vocabulary object detection.
Qualitative Results. We present visualizations of the CLIP ViT's dense representation by employ- ing K-Means clustering (Lloyd 1982) on the dense feature maps, where pixels of high cosine simi- larities are grouped into clusters. For clarity, we discard clusters with only a few pixels. As depicted in Fig. E our fine-tuned CLIP VIT demonstrates improved accuracy in gathering pixels belonging to the same object into single cluster, e-g., the 'human face' and the 'hat' the bottom right example. Additionally, our CLIPSelf model exhibits a reduction in false positive clusters, which either cover a small portion of an object or include pixels from different objects. The improved K-Means cluster results serve visible evidence of the enhancement of the CLIP VIT's dense representation.
4.3 APPLICATION TO OPEN- SHEN-VOCABULARY TASKS
Experiment Setting. We employ the refined CLIP ViTs to open-vocabulary dense prediction tasks. To ensure a fair comparison on each open-vocabulary benchmark, we implement CLIP- Self using the training set of the corresponding benchmark. Concretely, for open- vocabulaya de- tection on OV-COCO benchmark, the train2017 split of COCO dataset (Lin et al., 2014] is used for self-distillation. For the OV-LVIS benchmark, use the images from the train split of LVIS v1.0 (Gupta et al. 2019). For open-vocabulary image segmentation tasks, we use COCO's t Praind crain2013 7 split training dataset for both the semantic and panoptic segmentation benchmarks.
7

---

Published as a conference paper at ICLR 2024
Table 3: Results on open- open-vocabulary object detection. 'L', 'B' and 'H' in ViT-based methods stand for base, large and huge model sizes. '/16' and '/14' stand for the downsample ratio of input images.
(a) OV-COCO benchmark
(b) OV-LVIS benchmark
Method
Backhone
A
Method
Backbone
mAP,
ViLD (Gu et al. 2021
RN50
27.6
VILD GG ct al.
2021
RN50
16.6
Detic Zhou et al.. 2022b
RN50
27.8
OV-DETR Zang el al 2022
RNS0
17.4
F-VLM Kuo el al. 2023
RN50
28.0
BARON-KD (Wü
2023a
RN50
22.6
OV-DETR Zang al. 2022
RN50
29.4
CORA+ Wu al. 2025c
RN50x4
28.1
BARON-KD (WD
2U25a
RN50
34.0
F-VLM Kuo
A1P.5
RN50x64
32.8
CORA [Wu et 20236
RN50x4
41.7
PromprOVD (Song Bang 2023
VIT-B/16
23.1
CORA+ Wu et al. 202550
RN50x4
43.1
OW-VIT MMnnterrr cl al. 2022
VIT-I/14
25.6
Pro (Song & Bang
2023
VTT-B/16
30.6
RO-VIT Kum ct at. 20225
VIT-L/16
32.4
RO-VIT (Kim et all 20256
VIT-L/16
33.0
CFM-VIT KKm et 20258
VT-L//1
33.9
CFM-ViT KTm al 2023a
VIT-I/16
34.1
RO-VIT Kimre 20251)
VIT-H/16
34.1
F-VIT
VIT-B/16
17.5
F.VIT
VI--//16
11.5
F-VI++CLIPSeI
VIT-B/16
37.6
C-VITCCLIPSII
VIT-B/16
25.3
F-VIT
ViT-L/14
24.7
F-VIT
VIT-L/14
24.2
F-VIT+CLIPSeI
VIT-L/14
44.3
C-VITCCIPPSeI
VIT-1/14
34,9
Table Results on open-vocabulary semantic segmentation.
ADE-150
ADE-847
PASCAL Context
Method
Model
mloU
mAce
mlot.
mAce
mloU
mAcc
SAN et al. 2023b
VIT-B/16
27.5
45.6
10.1
21.1
53.8
73.0
SAN Xu ef al. 2025b
VIT-L/14
32.1
50.7
12.4
25.2
57.7
77.6
Cat-Ség (Cho et al.
2023
ViT-B/16
27.2
41.2
8.4
16.6
57.5
74.0
Cat-Seg (Cho et al.
2025
VIT-L/14
31.5
46.2
10,8
20.5
62.0
78.3
CLt-Seg-CLIIPSelf
VIT-B/16
29.0
46.0
9.3
20.1
58.0
75.3
Cat-Seg-CI.IPSod IPPeef
VIT-L/14
34.5
54.8
12,4
25.4
62.3
80.7
Table 5: Results on open-vocabulary panoptic segmentation. means the results are obtained by
running ODISE's officially released code and model.
Score
COCO Panoptic
ADE20K
Method
Model
CLIP
Pred
PQ
mAP
mIoU
PQ
mAP
mIoU
ODISE Xu 2023a
ViT-L/14
27.6
26.2
23.7
15.3
9.8
17.3
ODISE Xu 20238
ViT-L/14
/
45.3
38.1
52.3
22.9
13.4
28.5
CLISE
VIT-L/14
35.1
30.9
36.7
19.5
10.6
24.5
ODISE+CLIPSeI
VIT-L/14
/
45.7
38.5
52.3
23.7
13.6
30.1
Open- Vocabulcyy Object Detection. Following F-VLM (Kuo et al. 2023)) which freezes the CLIP
backbone, introduce two-stage detector baseline built frozen CLIP ViTs, called F-VI7. To extract multi-scale feature maps from the frozen backbone, we interpolate the feature maps from the intermediate layers of ViTs. We use the ViTs from EVA-CLIP (Sun et al. experiments for their efficiency and high capacity. The results are shown in Tab. By replacing the
2023) for our main
CLIP ViTs refined by CLIPSelf, the performance is significantly improved (44.3 vs 24.7 APanvel on OV-COCO and 34.9 vs 24.2 mAP, on OV-LVIS) and our detectors achieves new state-of-the-ar results the two benchmarks. More details and experimental results can be found in Sec. A.3
Semantic Segmentation. We apply CLIPSelf fine-tuned models to Cat- Seg (Cho al.] 2023). where the dense features of CLIP VITs (ViT-B/16 and ViT-L/14 from OpenAl) are used in a cost-aggregation module. The segmentation model is trained on COCO Stuff (Caesar et al.) 2018) and evaluated on ADE20K (Zhou et al., 2017) (ADE-150 and ADE- 847) and PASCAL Context (Mottaghi et al., 2014) datasets. As shown in Tab. CLIPSelf leads to performance improvement across all the test datasets.
Open-Vocabulle
Open-Vocabulary Panoptic Segmentation. We first reproduce ODISE Xu et al. 2023aa using the original CLIP model (ViT-L/14 from OpenAl) by running its officially released model and code Subsequently, we apply our fine-tuned model during the inference stage of ODISE, where the CLIP score and the score predicted by the mask generator are fused to classify the panoptic masks The model is trained on the COCO Panoptic (Lin et al.) 2014h dataset and evaluated on ADE20K (Zhou ct al. 2017) dataset. Tab. 5 presents the results of using the CLIP score only and using the fused score. The observed improvements in Tab. indicate the enhanced recognition capability of ViT's dense representation for recognizing panoptic masks.
8

---

Published as a conference paper at ICLR 2024
Table 6: Comparison with using region-fext pairs. Models are from EVA-CLIP (Sun et
2023
Region Proposals
Boxes
Thing Masks
Stuff Masks
OV-COCO
Model
Method
Topl
Top5
Topl
Top5
Top1
Top5
50
5P
ViT-B/16
Region-Tess
71.1
90.7
73.7
91.4
34.2
68.6
34.4
54.2
ViT-B/16
CLIPSelf
74.0
92.6
76.3
92.8
36.8
75.0
37.6
54.9
Table 7: Exploring local window attention (denoted 'WindowAttn', The 'GlobalAttn' means the
global attention used in the original CLIP ViTs. Models are from EVA-CLIP (Sun et al. 2023)
Boxes
Thing Masks
Stuff Masks
OV-COCO
Model
CLIPSelf
Attention
Top1
Top5
Top1
Top5
Top1
Top5
APS
VrT-B/16
GlobalAttn
18.2
33.2
20.6
36.5
18.4
43.5
17.5
41.0
VIT-B/16
Window Attn
34.7
60.3
40.6
64.8
30,9
61.2
19.4
48.5
ViT-B/16
/
Global
72.1
91.3
74.4
91.8
46.8
80.2
33.6
54.2
VIT-B/16
V
WindowAttn
73.3
91.7
74,9
92.1
48.6
81.0
33.6
55,9
Table 8: Performing diff-istiillation on CC3M (Sharma et al.
2018
The model (ViT-B/16 from
EVA-CLIP (Sun et al., 2023) ) is trained on CC3M for epoch.
Boxes
Thing Masks
Stuff Masks
OV-COCO
OV-LVIS
Model
CLIPSelf
Topl
Top5
Topl
Top5
Topl
Top5
APgeet
AP
mAP.
mAP,
mAP,
VIT-B/16
18.2
33.2
20.6
36.5
184
43.5
17.5
41.0
11.5
12.3
20.6
VIT-B/16
72.1
91.5
74.5
92.0
49.5
81.6
35.8
54.6
26.6
21.7
29.2
4.4 DISCUSSION
Comparison with Using Region-Texx Pairs. To demonstrate the advantage of CLIPSelf, we com- pare with the method using region-text pairs. We follow the principle of RegionCLIP (Zhong al.) 2022) that matches region proposals with object nouns to generate region-text pairs to implement the compared method. For fair comparison, CLIPSelf also employs region proposals. More details on the compared method are in Sec. A.4 As indicated in Tab. 6 (#1 and #2), CLIPSelf outperforms the method using noisy region- pairs by a large margin.
To verify CLIPSelf's applicability to different models, we exploit architectures beyond plain ViTs (global attention). Unfortunately, there are no public CLIP-like ViTs equipped with local attentions, e.g. Swin Transformer (Liu et al.) 2021). Relevant methods like GLIP (Li jet al. 2022a) are built under different benchmarks and simply treat Swin as the backbone without explicitly exploring its region-language alignment. Therefore, attention based ViTs from the current CLIP ViTs. Specifically, we replace the original global atten- tion with 4 x 4 window attentions. For the global image token (class token), replicate it when splitting the image into windows and average the class token of each window when merging the windows. This modification improves region recognition and apen-vocabulary detection (#1 and #2), but lags largely behind using CLIPSelf for self-distillation (#2 and #3). Importantly, CLIPSelf can consistently improve the CLIP model with window attention (#2 and #4). Therefore, believe the effectiveness of CLIPSelf can be extrapolated to a wider range of model architectures.
Beyond Plain ViTs.
opt for developing local window
Training on CC3M. The experiments above mainly use the downstream COCO dataset for self- distillation to ensure fair comparison with prior methods, especially distillation- based and frozen CLiP-based approaches. We also consider conducting self-distillation on the out-of-domain CC3M dataset (Sharma et al.] (2018). As shown in Tab. 8 the model fine-tuned on CC3M exhibits consistent improvement on zero-shot box and mask recognition as well vocaablayy object detection.
CONCLUSION
In this paper, we undertake comprehensive analysis the dense representation of ViT-based CLIP models. Based on this analysis, we introduce CLIPSelf, which fine-tunes CLIP ViTs' dense rep- resentation via self-distillation without relying on region-text pairs. The fine-tuned CLIP ViTs sig- nificantly surpass the performance of the original models when applied to open-vocabulary object detection and image segmentation. Furthermore, we validate CLIPSelf's promising applicability by exploring local window attentions beyond plain ViTs and implementing CLIPSelf on web data (CC3M). In conclusion, our CLIPSelf serves as a straightforward yet effective solution to enhance the dense representation of CLIP ViTs, which is critical to open- open-vocabulary dense prediction tasks.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENTS
This research is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: ASSG3-PhD-2223-08-05555 the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contri- bution from the industry partner(s).
REFERENCES
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Sean Chang Culatana, and Mohamed Elhoseiny. Exploring open-vocabulary semantic seg- mentation from clip vision encoder distillation only. In Int. Conf. Comput. Vis., 2023.
Xinlei Chen, Hao Fang, Tsung-Yr Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and captions: Data collection and evaluation server. arXiv
C Lawrence Zitnick. Microsoft preprint arXiv:1504.0033 2015.
Seokju Cho, Heeseong Shin, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Amab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary mantic segmentation. arXiv preprint mrXiv:2303.17997 2023.
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In IEEE Conf. Comput. Vis. Pattern Recog.. pp. 11583-11592 2022.
Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary universal image segmentation with maskelip. In ICML, 2023.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL Visual Object Classes (VOC) Challenge. Int. J. Comput, Vis., 2010.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. Adv. Neural Inform. Process.
Syst., 2013.
Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-bow labels. In Eur. Conf. Comput. Vis., 2022.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Open-vocabulary image segmentation. arXiv preprint arXiv:2112. 2021.
Ross Girshick. Fast r-cnn. In Int. Conf. Comput. Vis., 2015.
Xiuye Gu, Tsung-Yr Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. Int. Conf. Learn. Represent., 2021.
Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In IEEE Conf. Comput. Vis. Puttern Recog., 2019.
Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem. Con- trastive learning for weakly supervised phrase grounding. In Eur. Conf. Comput. Vis., 2020.
Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In Eur. Conf. Comput. Vis., 2014.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Int. Conf. Comput. Vis., 2017.
10

---

Published as a conference paper at ICLR 2024
Dinesh Jayaraman and Kristen Grauman. Zero-shot recognition with unreliable attributes. In Adv. Neural Inform. Process. Syst., 2014.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
Dahun Kim, Anelia Angelova, and Weicheng Kuo. Contrastive feature masking open-vocabulary vision transformer. In Int. Conf. Comput. Vis., 2023a.
Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for open-vocabulary object detection with vision transformers. In IEEE Conf. Comput. Vis. Pattern Recog., 2023b.
Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Vision-and-language transformer without con-
volution or region supervision. In ICML, 2021.
Alexander Kirillov, Kaiming He, Ross B. Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. IEEE Conf. Comput. Vis. Pattern Recog., 2019.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 2017.
Weicheng Kuo, Yin Cui, Xiuye Gu, A. J. Piergiovanni, and Anelia Angelova. F-VIM: Open- vocabulary object detection upon frozen vision and language models. Int. Conf. Learn. Represent., 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In IEEE Conf. Comput. Vis. Pattern Recog., 2022a.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. In Eur. Conf. Comput. Vis., 2022b.
Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object detection. In ICLR, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects in context. In Eur. Conf. Comput. Vis., 2014.
Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su. Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0544 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Inf. Conf. Comput. Vis., 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Stuart Lloyd. Least squares quantization pem. IEEE transactions on information theory, 28(2): 129-137, 1982.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn. Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani. Zhuoran Shen, et al. Sim- ple open-vocabulary object detection with vision transformers. Eur. Conf. Comput. Vis., 2022.

---

Published as a conference paper at ICLR 2024
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmen- tation in the wild. In JEEE Conf. Comput. Vis. Pattern Recog. pp. 891-898, 2014.
Jishnu Mukhoti, Tsung- Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser- Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting eegion-to-phrase correspondences for richer image- to-sentence models. In Int. Conf. Comput. Vis., 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, and Fahad Shah- baz Khan. Bridging the gap between object and image-level representations for open-vocabulary detection. In NeurlPS, 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Adv. Neural Inform. Process. Syst., 28, 2015.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Int. Conf. Comput. Vis., 2019.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernyme,, image alt-text dataset for automatic image captioning. In ACI, 2018.
Hwanjun Song and Jihwan Bang. Prompt-guided transformers for end-to-end open-vocabulary ob- ject detection. arXiv preprint 230.::3333.13886 2023.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 2023.
Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. Object-aware distillation pyramid for open-vocabulary object detection. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object detection. In IEEE Conf. Comput. Vis. Pattern Recog. 2023a.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Wentao Liu, and Chen Change Loy. Clim: Con- trastive eiguuag--mmage mosaic for region representation. arXiv preprint arXiv:2312.1 2023b.
Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In IEEE Conf. Comput. Vis. Pattern Recog., 2023c.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., 2023a.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In Eur. Conf. Compul. Vis., 2022.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. IEEE Conf. Comput. Vis. Pattern Recog., 2023b.
Shilin Xu Xu, Xiangtai Li, Size Wu, Wenwei Zhang, Yining Li, Guangliang Cheng, Yunhai Tong, Kai Chen, and Chen Change Loy. Dst-det: Simple dynamic self-training for open-vocabulary object detection. arXiv pre-print, 2023c.
12

---

Published as a conference paper at ICLR 2024
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary DETR with conditional matching. Eur. Conf. Comput. Vis., 2022.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In IEEE Conf. Comput. Vis. Pattern Recog.. 2021.
Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. LiT: Zero-shot transfer with moceed-maage text tuning. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit: Semantic segmentation with plain vision transformers. Adv. Neural Inform. Process. Syst., 2022a.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. Adv. Neural Inform. Process. Syst., 2022b.
Zixiao Zhang, Xiaoqiang Lu, Guojin Cao, Yuting Yang, Licheng Jiao, and Fang Liu. Vit-yolo: Transformer-based yolo for object detection. In Int. Conf. Comput. Vis. Worksh. 2021.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. RegionCLIP: Region- -baoed language-iman image pretraining. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 633-641, 2017.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from CLIP. In Eur: Conf. Comput. Vis., 2022a.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In Eur. Conf. Comput. Vis., 2022b.
13

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1
CLIP MODELS' DENSE REPRESENTATION
RNS0X4
(OpenAI)
VIT-B
1000
100
99
100
20
20
20
224
1024
640
280
1024
540
Isput tmage sizn
Inper image smt
Irper Image site
VIT-L/14 (OpesAI)
VT--/14 (EVA-CCPP
88
60
60
20
Irpart Image vbo
Tepar Image Xco
Ieput Image 20o
Top5
Top1
Image Crop
Dense Feature
Figure Al: Region classification using CLIP Models. The x-axis of the figures stands for the input size obtain dense features. The input size for the image-level representation of the image crops is fixed at 288 x 288 for RN50x4, 448 x 448 for RN50x 64, 224 x 224 for ViT-B/16 and 336 x 336 for ViT-L/14.
Table A1: Retrieval experiment on images and regions. The images and regions are obtained from COCO's va12017 split.
Image Retrieval
Region Retrieval
Model
CLIPSelf
R@1
R@5
R@ 10
R@1 l
R@5
R@ 10
VIT-B/16
X
67.0
85.6
91.4
34.7
60.5
71.4
VIT-B/16
28.4
50.3
62.5
57.1
80.1
85,8
VIT-L/14
X
46.1
63.6
71.4
45.2
66.6
74.6
VIT-L/14
37.8
25.5
61.1
49.4
69.8
76.0
Region Recognition. We conduct comprehensive analysis of the dense representation of CLIP models in the context of region recognition. Specifically, we evaluate the Top1 and Top5 mean accuracy (mAce) of region classification using the dense features of CLIP models with varying in- put image sizes. Fig. All Allllusttates the results obtained. For CNN-based models, we observe that their dense features are highly effective for region recognition. In fact, the performance of region recognition using dense feature maps surpasses that of utilizing image-level representations of the corresponding image crops, particularly when the input image size exceeds 640 x 640. The remark- able alignment between regions and language exhibited by CLIP CNNs enables their straightforward application to downstream open-vocabulary dense prediction tasks. However, the trend opposite for CLIP Vision Transformers (ViTs). The dense features of ViTs exhibit inferior performance in region classification across all input image sizes. Moreover, the accuracy tends decrease as the input image size is further increased. Meanwhile, employing ViTs' image-level representation of the regions' image crops yields satisfactory results, indicating that it can serve as a reliable teacher for refining the dense representation. These findings shed light the contrasting behaviors of CLIP CNNs and ViTs, emphasizing the potential benefits of leveraging image-level representations to refine the dense representation of ViTs.
Are Dense Features Encoding Global Images? We conduct a retrieval experiment to answer this question. Specifically, first encode images and regions annotated in COCO' validation split. The region embeddings are obtained by cropping the region boxes and sending to the CLIP model. Then extract dense feature maps of the images and let each location in the feature map retrieve the
14

---

Published as a conference paper at ICLR 2024
Table A2: Different extraction methods to obtain dense features.
#
Model
Source
Method
Boxes
Thing Masks Top5
Stuff Masks
Top1
Top5
Top1
Top1
Top5
1
ViT-B/16
OpenAl
Image Crops
63.8
83.7
2
ViT-B/16
OpenAl
Zhou al. 2022a
29.3
51.6
33.5
56.0
25.9
50.9
3
ViT-B/16
OpenAl
Masked Attention
28.2
46.7
29.6
47.5
32.2
61.6
4
ViT-B/16
OpenAl
CLIPSelf
67.4
88.4
69.4
88.5
43.4
76.9
Table A3: The enhanced dense representations for recognizing boxes, thing masks, and stuff masks.
Model
Source
CLIPSelf
Boxes
Thing Masks
Stuff Masks
Top1
Top5
Topl
TopS
Topl
Top5
ViT-B/16
OpenAl
X
29.3
51.6
33.5
56.0
25.9
50.9
ViT-B/16
OpenAl
67.4
88.4
69.4
88.5
43.4
76.9
ViT-B/16
EVA-CLIP
X
18.2
33.2
20.6
36.5
18.4
43.5
VIT-B/16
EVA-CLIP
72.1
91.3
74.4
91.8
46.8
80.2
VIT-L/14
OpenAl
21.4
45.9
28.3
52.0
11.8
27.9
VIT-L/14
OpenAl
68.9
89.6
70.0
88.0
35.5
71.5
VIT-L/14
EVA-CLIP
56.7
78.0
59.0
79.8
20.8
41.9
VIT-L/14
EVA-CLIP
77.1
93.3
78.7
93.7
44.4
78.3
image and region it belongs by calculating cosine similarity between the location feature and the encoded image or region features. For each retrieval, we provide 50 samples (1 positive and 49 negative samples) and calculate recall at 1, 5, and 10, respectively. As shown in Tab. Al[#1), the dense features are well matched with the corresponding images, indicating that each location on the dense feature map tends to encode global image representation. This observation coincides with the K-Means visualization Fig. and Fig. 4 where the clustering results of the original CLIP ViT quite diffused.
Masked Attention. We follow the practice in (Zhou et al. 2022a to extract dense feature maps of CLIP VITs. One might question whether applying masked-attention operation could be better cure for the inferior dense-level representation. However, as presented in Tab, [3] our ex- periments indicate that such approach improves the recognition of stuff masks but worsens the classification of object boxes and thing masks. Furthermore, the accuracy improvement for stuff masks achieved through the masked-attention attention operation lags significantly behind the performance of CLIPSelf (#4).
A.2 CLIPSELF
Input Image Sizes & Trainable Layers. For the Student model, which extracts dense represen- tations, we set the image size of input images 1024 1024 for ViT-B/I6 and 896 896 for ViT-L/14. As for the Teacher model, which takes cropped image patches as input, we set the input size 224 224 for ViT-B/16 and 336 336 for ViT-L/14. In the training of CLIPSelf, update the 12 attention layers of ViT-B/16 and the 24 attention layers of ViT-L/14.
Enhancement of Dense Representation. To summarize the improvements achieved by CLIPSelf in the context of classifying boxes and masks, we provide summary in Tab, A3 The results clearly demonstrate that CLIPSelf effectively enhances the dense representation of all variants of ViTs.
From Images to Regions. As shown in the retrieval experiment in Tab, AI #2), the models refined by CLIPSelf yield dense features that better match the corresponding regions instead of images, revealing the transfer of knowledge from global image representation to local region representation.
A.3 OPEN-VOCABULANE OBJECT DETECTION
Benchmark Details. The open-vocabular, COCO (OV-COCO) benchmark, proposed OV- RCNN (Zareian et al., 2021), splits 65 object categories in COCO dataset (Lin al. 2014) into 48 base categories and 17 novel categories. The open- vocabuacyy LVIS (OV-LVIS) benchmark, pro- posed in ViL.D (Gu al. 2021) the 337 rare categories in LVVIS v1.0 (Gupta et al. 2019) dataset
15

---

Published as a conference paper at ICLR 2024
Table A4: Design choices of the open-vocabulary object detector. The 'Backbone LR' represents the scalar multiplied by the base learning rate (1e-4) in training. 'Inter' means using outputs of ViTs' intermediate layers.
#
Model
FPN
Layers
Backbone LR
A
30
AP50
VIT-B/16
SimpleFPPN
Last
x0.0
23.1
46.6
40.4
ViT-B/16
Standard
Last
x0.0
26.5
47.0
VIT-B/16
Standard
Inter
x0.0
33.6
54.2
48.8
4
ViT-B/16
Standard
Inter
x0.01
31.1
56,6
49.9
5
ViT-B/16
Standard
Inter
x0.1
21.8
58.6
48.8
Table A5: Results of OpenAl models on OV-COCO and OV-LVIS benchmarks.
OV-COCO
OV-LVIS
Method
Model
APi
AP50
mAP,
mAP.
mAP,
mAP
F-VIT
VIT-B/16
16,0
36.9
31.4
8.3
11.4
19.7
14.1
C-VII+CLIPSeII
ViT-B/16
29.8
46.9
42.5
21.6
16.2
23.8
20.1
F-VIT
VIT-L/14
9.2
44.3
35.2
10.7
19.6
26.3
F-VITCCLPPSef
VIT-L/14
31.3
49.2
44.6
24.4
21.1
27.5
24.2
Table A6: Detailed comparison OV-COCO benchmark. t means using learnable category prompts for region classification. stands for methods that obtain open-vocabulary ability from both CLIP visual model and additional image annotations (e.g., image captions).
Method
Backbone
A AP
APh
AP30
OV-RCNN (Zariian al.
2021
RN50
17.5
41.0
34,9
RegionCLIP Zhong
2022
RN50
26.8
54.8
47.5
RegionCLIP Zhong el al.
2022
RN50
31.4
57.1
50.4
RegionCLIP Zhong ct al.
2022
RN50x4
39.3
61.6
55.7
ViLD (Gu et al. 202T
RN50
27.6
59.5
51.2
OV-DETK (Zang ct
2022)
RN50
29.4
61.0
52.7
PR-OVD (Gao
2022
RN50
30.8
46.1
42.1
Detic [Zhou al. 20226
RN50
27.8
$1.1
45.0
OC-OVD (Rasheed el al.
2022
RNS0
36.6
54.0
49.4
VLDet (Lin et al. [2023
RN50
32.0
50.6
45.8
F-VLM (Kuo el al. 2023
RN50
28.0
39.6
BARON-Cap (Wu
2023a
RNSO
33.1
$4.8
49.1
BARON- KDD Wu et
2023a
RN50
34.0
60.4
53.5
BARON KD Wu
ct al.
2023a
RNS0
42.7
54.9
51.7
OADP Wang et al.
20233
RN50
35.6
55.8
50.5
CORA Wu et al. 025c
RN50
35.1
35.5
35.4
CORA et 2025e
RN50x4
41.7
44.5
43.8
CORA+ Wu a 202.c
RN50x4
43.1
60.9
56.2
RO-ViT Kim al. 20235
ViT-B/16
30.2
41.5
RO-ViT Kim et al.
20233
ViT-L/16
33.0
47.7
CFM-ViT Kim ct
2023a
ViT-L/16
34.1
46.0
F-VIT
VIT-B/16
17.5
41.0
34,9
F-ViT+Region-TE Text
ViT-B/16
34.4
54.2
49.0
F-ViT+CLIPSed (Image Patch)
ViT-B/16
33.6
54.2
48.8
R-VIT+CLIPSeIf (Region Proposal)
ViT-B/16
37.6
54.9
50.4
F-VIT
ViT-L/14
24.7
53.6
46.0
R-ViT-Region-Tex Text
ViT-L/14
38.7
59.6
54.1
R-ViT+CI.IPSef (Image Patch)
ViT-I/114
38.4
60.6
54.8
F.VIT+ILIPSef (Region Proposal)
ViT-L/14
44.3
64.1
59.0
as novel categories. For evaluation, we follow previous works to use box AP at IoU 0.50 on novel categories (APnovel) as the main metric OV-COCO, and the mean mask AP rare categories (mAP,) as the main metric on OV-LVIS.
Implementation Details. To train the detector. set the batch size to on each GPU and employ the AdamW optimizer with a learning rate of le-4 and weight decay 0.1. We train the models for 3 epochs the OV-COCO benchmark and 48 epochs the OV-LVIS benchmark. In the main
16

---

Published as a conference paper at ICLR 2024
Detailed comparison on OV-LVIS benchmark. means using learnable category prompt for region classification. stands for methods that obtain open-vocabulary ability from both CLIP visual model and additional image annotations (e.g.. image captions).
Table A7:
Method
Backhone
mAP,
mAP,
mAP,
mAP
RegionC III (Zhong al.
2022
RN50
17.1
27.4
34.0
28.2
RegionCI.IP Zhoong
2022
RN50x4
22.0
32.1
36.9
32.3
Detic Zhou al.
20226
RN50
24.9
32.4
Detic (hou et al. 2022b
SwinB
33.8
47.0
VLDet (Lin ct al. 2023
RN50
21.7
29.8
34.3
30.1
VLDet (Lin et al
2025
SwinB
26.3
39.4
41.9
38.1
ViLD (uu
2021
RN50
16.6
24.6
30.3
25.5
OV-DETR iZang
2022
RN50
17.4
25.0
32.5
26.6
DetPro (Du ct al. 2202
RN50
19.8
25.6
28.9
25.9
BARON-KD Wu et al.
202.a
RN50
22.6
27.6
29.8
27.6
OADP (Wang al. 2023
RN50
21.7
26.3
29.0
26.6
OC-OVD (Rasheed
2022
RN50
21.1
25.0
29.1
25.9
R-VLM Kuo
RN50
18.6
24.2
F-VLM Kuo ct al.
2023
RN50x4
26.3
28.5
F-VLM Kuo el al. 2023
RN50x16 16
30.4
32.1
F-VLM Kuo cf al 2023
RN50x64
32.8
34.9
CORA Wu al. 2023c
RN50x4
22.2
CORA+ Wu el
202.5c
RN50x4
28.1
OWL- (Kim al. 20023b
ViT-L/14
25.6
34.7
RO- VIT Kim ct al 20236
ViT-B/16
28.0
30.2
RO-VIT Kim cl al 20233b
ViT-L/16
32.1
34.0
RO-VIT Kim et al. 20233b
ViT-H/I6
34.1
35.1
CFM-V vii
Kim et z023a
VIT-L/16
33,9
36.6
F-T
VIT-B/I6
11.5
12.3
20.6
15.4
F R-ViT+CLIPSeIf (Image Patch)
VIT-B/16
25.3
21.8
29.1
25.2
E-VIT
ViT-L/14
24.2
27.9
31.5
28.7
F-VIT+LLIPSeI (Image Patch)
VIT-L/14
34.9
34.6
35.6
35.1
experiments using ViTs from EVA-CLIP (Sun et al. 2023). the input image size is set as 896 x 896 for ViT-L/14. For ViT-B/16, the input image size is 640 640 on OV-COCO and 1024 1024 on OV-LVIS. For potential non-square inputs, we pad zero values to the bottom and right of the images after color normalization.
F-VIT Architecture. In this section, we examine the design choices of the baseline open-vocabulary detector, V-V To build an object detector based on ViTs, a popular approach is ViTDet (Li al,) 20226), which utilizes simple Feature Pyramid Network (FPN) and only employs feature maps from the last attention layer. The design ViTDet based on two observations: (1) the high ca- pacity of ViT allows it to learn detection-related features without additional parameters except standard FPN, and (2) the output of the last attention layer is trained to provide task-specific repre- sentations for object detection. However, in the course of adapting CLIP ViTs for open- open-vocabulary object detection where we would like to freeze the backbone to retain the vision-languare alignment of CLIP (Kuo et al. 2023), these presuppositions
longer hold since do not update the VIT to learn the task-specific features. As shown in Tab. A4 using the ViTDet-like architecture (#1) yields the worst results. Transitioning
a standard FPN (#2) slightly improves the performance. Then, we utilize the feature maps from multiple intermediate layers of the ViT (43) instead of solely relying on the output of the last layer. This design significantly enhances performance for both novel and base categories. Specifically, we interpolate the feature maps from layers [3, 5. 11] of ViT-B/16 with relative scales [ 1 TE T/ the input image size. For ViT-L/14, we interpolate the feature maps from layers [6, 10, 14, 23] with relative scales 1 I. to the input image size. 7, T 28
Unfreeze the Backbone. We explore the possibility of updating the parameters of the ViT backbone in our approach. Tab. A4 (#4&#5) presents the impact on performance when gradually increasing the learning rate of the backbone. Notably, while the AP50 for base categories improves, the perfor- mance for novel categories decreases. Based on these observations, we opt to maintain fixed ViT backbone, the frozen architecture already achieves satisfactory results on base categories.
17

---

Published as a conference paper at ICLR 2024
Table A8: Transfer evaluaaion the L.VIS-traing detector on PACAL VOC, COCO and Objects365.
Pascal VOC so
COCO
Objects365 AP
Method
AP75
AP
AP
AP
AP
AP75
ViLD
2021
36.6
55.6
39.8
11.8
18.2
12.6
DetPro
al. 2022
74,6
57.9
34.9
53.8
37.4
12.1
18.8
12,9
BARON [Wu et al.
2023a
76.0
58.2
36.2
55.7
39.1
13.6
14.5
RO-ViT
Kim el al. 202.bb
17.1
19.5
F-VLM
39.8
61.6
43.8
17.7
27.4
19.1
CFM-VTT Kim et al.
2023a
18.7
28.9
20.3
CLLPPl
77.6
59.8
40.5
63.8
44.3
19.5
20.7
OVD results with OpenAl models. In the main experiments of open-vocabulary detection, we em- ploy ViTs from EVA-CLIP (Sun al. (023), given their favorable performance and high efficiency. However, it is worth noting that CLIPSelf can also be applied for other ViT variants. We report the results using ViTs from OpenAI (Radford et al. (021) Tab. [S The input size is 640 x 640 for ViT-B/16 and 672 x 672 for VIT-1/14. We only use image patches for self-distillation. All the other settings remain the same as the experiments on EVA-CLIP ViTs.
Comprehensive System-Level Comparison. To provide thorough and comprehensive system- level comparison of open-vocabulary object detection methods, we present detailed results for both the OV-COCO and OV-LVIS benchmarks in Tab. Anland Tab. A7 respectively.
Transfer Evaluation. We evaluate the detector (ViT-L-/14) trained on OV-LVIS on the validation split of PASCAL VOC (Everingham et al. (010)), COCO (Lin et al. (014) and Objects365 v1 (Shao ct al. 2019) datasets. Our approach consistently outperforms all previous methods in the transfer evaluation.
A.4 USING REGION-TEXT PAIRS
We adopt the methodology proposed in RegionCLIP (Zhong et al. 2022 to fine-tune CLIP ViTs using pscudo-labelled region-text pairs. Specifically, we parse object nouns from the COCO Caption dataset (Chen et al., 2015) and generate region proposals using RPN trained COCO's s 48 base categories. To establish correspondence between object nouns and region proposals, the original RegionCLIP extracts region features from the dense feature maps of CLIP CNNs. Differently, use the image-level features of the image crops enclosing the regions, considering the inferior dense representation of the original CLIP ViT. For fair comparison, also fine-tune the model the train2017 split of COCO dataset (Lin et al. 2014) for the same 6 epochs.
B BROADER IMPACT
Our work contributes to an in-depth analysis of the dense representation of ViT-based CLIP mod- els and introduces CLIPSelf, an effective approach to unleash the power CLIP ViTs for open- vocabulary dense prediction. We are the first to build open-vocabulary object detectors upon frozen CLIP VIT backbones and achieve state-of-the-ar performance. As transformers are becoming in- creasingly popular unified architecture for both vision and language tasks, it is of great signifi- cance to successfully adapt the generalization ability of CLIP ViTs from image classification tasks to dense prediction tasks. Experiments also validate the effectiveness of our CLIPSelf beyond plain ViTs, demonstrating promising applicability of CL.IPSelf on a wider range of model architectures.
C LIMITATION
Our work is built upon the pre-trained CLIP models. Therefore, the performance of CLIPSelf is greatly influenced by the visual-language alignment in the original CLIP models. How to further enhance both image and dense representation of pre-trained vision-language models will be an inter- esting research topic. Moreover, as there are no public CLIP-like vision- languallagguage models with the Swin Transformer (Liu al. 2021) backbone, CLIPSelf on the Swin-based models. Recently, there are some detection-orienter Swin-based foun- dation models that learn region-language grounding from region-text pairs, e-g., GLIP (Li et al.
not able to directly validate the effectiveness of
18

---

Published as a conference paper at ICLR 2024
We envision that CLIPSelf has the potential to further enhance these works. The self-distillation can be a strong supplement or replacement of the pseudo labelling process in these methods. However, due to resource limitation and lack of further study on the vision and language representations within these architectures, the empirical validation is not provided in this work. We plan to explore this in future work.
2022a Zhang al. 2022b) and Grounding DINO (Luu et al. 2023
D VISUALIZATION
Open-vocabulary Object Detection. We present qualitative results for open-vocabulary object de- tection on the OV-COCO benchmark in Fig. [A2] The red boxes indicate novel categories, while the blue boxes represent base categories. These visualizations offer insights into the model's perfor- mance and its ability to detect novel objects.
Vpen-Vocabulary Image Segmentation. We present visualizations of open-vocabulary semantic segmentation results in Fig. A3] The segmentation model is trained on COCO Stuff and evaluated on the ADE20K dataset (Zhou et al. 2017).
19

---

Published as a conference paper at ICLR 2024
KEBOUTTA
umbrelaao
Figure A2: Visualization of object detection results. The red boxes are for the novel categories and the blue boxes are for the base categories.
IROR
Figure A3: Visualization of image segmentation. The images from ADE20k (Zhou al. 2017
20


Reviews:
[{"review_id": "rOrBXuzPY7", "reviewer": "Reviewer_yRsy", "summary": "The paper proposed to address the problem of adapting vision Transformers pre-trained with image-language alignment (e.g., CLIP) for dense prediction tasks (e.g., object detection, semantic segmentation). The key idea is to distill features from the pre-trained Transformer model itself, in which region representation from a dense feature map matches those from cropped regions (using the pre-trained model). The proposed method was evaluated on multiple open-vocabulary object detection and semantic segmentation benchmarks. The results are solid.", "questions": "If the explanation described in the weaknesses section is true or partially true, it brings two additional questions. \n\n* First, do those new localized representations (after distillation) still preserve the zero-shot recognition capacity of CLIP? One interesting experiment is to evaluate those features directly for zero-shot object detection / segmentation (e.g., see Table 4 in RegionCLIP). \n\n* Second, is the proposed method a remedy to a particular design (ViT with global self-attention)? This question is perhaps more fundamental. What if a CLIP model is trained using ViT with local self-attention (e.g., using Swin Transformers)? Will they still benefit from the method? This is much harder to validate directly, as there is no pre-trained models available. Yet one possible alternative is to modify the pre-trained ViT by replacing global attention with local ones, and then train for those open-vocabulary tasks. (Indeed, MaskCLIP can be considered as a special case of this modification, where the global attention at the end is replaced by a local attention of size 1x1.) It will be interesting to see how the proposed method compares to this alternative.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper addressed an important and trendy topic on adapting large-scale pre-trained vision-language models (i.e., foundation models). \n\n* The paper is well written. Key concepts are clearly described. Technical details are easy to follow. \n\n* The core idea is well motivated and quite intuitive. \n\n* The experiments are extensive (three open-vocabulary dense prediction tasks across multiple datasets + ablations). The results are impressive.", "weaknesses": "It is somewhat unclear why the proposed method, which is conceptually simple, can work well. My bet is that the proposed self-distillation encourages localized visual features, i.e., features on the dense map capture local region representations (instead of global image representations). This is partially backed up by Fig 1 (b). The gap between dense features and cropped image features is negligible for CNN (where the features are localized), yet rather significant for ViT. Part of the reason lies in the design of ViT using global attention, where the dense features are diffused in a way that each spot encodes a global image representation. There might be ways to validate this explanation. For example, one possibility is a retrieval experiment using features on the dense maps. Features on dense maps from pre-trained CLIP ViT might yield similar images instead of similar regions, while features after distillation may better retrieve regions. It will be great if the authors can comment on this reasoning here and discuss the key intuition in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OMp55GZRz4", "reviewer": "Reviewer_BRE4", "summary": "In this work, the authors proposed a simple yet effective method for learning fine-grained visual-language representations for open-vocabulary detection and segmentation tasks. Given that the original CLIP-based VIT models have a great degradation of performance for zero-shot fine-grained image understanding, the authors explored a cheap way of refining the visual representation from the original CLIP vision encoder by distilling the image-patch representation to regional dense feature representations. Through such a simple strategy, the region-level visual-semantic representation ability is immediately unleashed. According to the extensive evaluations of different tasks and settings, the refined representations significantly outperform the original ones and other state-of-the-art methods. Overall, I think the proposed method called CLIPSelf is a very simple yet effective way to refine the existing models, which could be a good way of learning better representations for open-vocabulary tasks under low budgets.", "questions": "Overall,I have two main questions:\n\n1. How to distinguish this work from ZeroSeg regarding the core learning algorithm?\n\n2. Are the improvements over CLIP baselines sourced from in-domain training? What if we use another dataset like CC3M for the distillation?\n\nOne minor question:\n\nHow to adapt the VIT models to different input image resolutions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method CLIPSelf, distilling CLIP models for fine-grained representation is simple yet effective. It is motivated by an interesting observation in the study of CLIP representation for image-level tasks to region-level tasks. The author also observed that cropping image patches significantly outperform feature pooling for ViT vision encoders. As a result, the authors proposed to distill the fine-grained understanding of visual content in image patches to the regional representations in the hidden feature space.\n\n2. The authors conducted extensive experiments to validate the effectiveness of the proposed methods, spanning from open-vocabulary object detection to semantic segmentation and then panoptic segmentation. It shows that the refined visual-semantic representations by CLIPSelf bring significant gain over the baseline model and also outperform strong previous works.", "weaknesses": "1. There are some related works that already explored the way of distilling patch-level representations to regional feature-level representations, such as ZeroSeg proposed in \"Exploring Open-Vocabulary Semantic Segmentation without Human Labels. Chen et al. ICCV 2023\". The overall methodology proposed in this work resembles the one in ZeroSeg, in that both attempted to distill the CLIP knowledge from patch to feature map and for open-vocabulary segmentation tasks.\n\n2. It turns out that the authors only used the COCO training set for distillation. A doubt on this setting is that it is probably very favorable to downstream tasks like COCO detection and segmentation, and ADE20K as well. Though this in-domain distillation brings a significant boost, the authors should have more studies and evaluations on more downstream datasets from various domains and also try using non-COCO training sets for distillation, which seems to be more realistic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MAw5ItcqbI", "reviewer": "Reviewer_CnGz", "summary": "This paper presents a simple approach to improve the transferability of CLIP ViT models to open-vocabulary object detection and image segmentation.\nWhile CLIP ViT shows strong performance in image classification (as compared to CLIP ConvNets),\nits region-level representation is known to underperform in dense prediction tasks.\nThe authors first demonstrate this shortcoming through a simple experiment,\nthen propose a self-distillation approach named CLIPSelf to remedy this.\nIn their experiments, the authors apply the CLIPSelf procedure to fine-tune the CLIP ViT checkpoints made public by OpenAI,\nand then use the fine-tuned weights for open-vocabulary object detection and image segmentation.\nEmpirical results show that visual representations from CLIPSelf are better than CLIP for dense prediction tasks,\nand CLIP ViTs fine-tuned by CLIPSelf procedure are highly competitive on open-vocabulary detection benchmarks like COCO and LVIS.", "questions": "Minor questions and suggestions:\n\n- Figure 3 (b) non-standard word use: \"initiate\" -> \"initialize\"\n- Figure A2 caption is incorrect: \"image segmentation\" -> \"object detection\"", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Note:**\nI reviewed this paper for the NeurIPS 2023 conference.\nI will contextualize my current review based on the changes the authors have made since their previous submission.\nThe main idea of CLIPSelf has been unchanged since NeurIPS 2023 submission --\nthe authors have significantly revamped the experimental study with more suitable benchmarks and baselines,\nand added more implementation details in the draft.\n\nI identified some key strengths of the proposed approach in the NeurIPS 2023 submission, they still apply to the current submission:\n\n- The proposed approach (CLIPSelf) is conceptually simple and shows large improvements over CLIP on multiple evaluations shown in the paper.\n- CLIPSelf does not require any form of annotations beyond just images, this relaxation simplifies the data and modeling pipeline.\n- Authors have performed comprehensive evaluations with multiple downstream tasks involving dense predictions to show the versatility of the proposed approach.\n\nI find notable other strengths in the current submission:\n\n- Empirical results on open-vocabulary detection are quite strong. CLIPSelf outperforms prior works that use equivalent frozen backbones (e.g. F-VLM, CORA).\n- More notably, CLIPSelf outperforms the recent work of RO-ViT which proposes training a new backbone catered to downstream task _from scratch_.\n  I find this result particularly appealing as it shows that CLIPSelf can lightly adapt a wide range of publicly available checkpoints that were trained with relatively less consideration for the particular task of open-vocabulary object detection.\n- The writing clarity has also improved in this submission.", "weaknesses": "My review mainly contrasts the previous submission (NeurIPS 2023) with the current ICLR 2024 submission --\nsince the main story of the paper is mostly unchanged, I do not find any significant outstanding concerns.\n\nI have a few outstanding concerns and suggestions, which may further improve the paper.\nI am curious to hear the authors' thoughts on these and am willing to engage in the discussion:\n\n1. **Is self-distillation on task dataset (COCO) necessary?**\nMain CLIPSelf experiments use images from COCO/LVIS dataset to perform self-distillation.\nThe authors should consider disentangling the impact of self-distillation in isolation,\nas compared to self-distillation using images from the target downstream task.\nIf CLIPSelf does not _require_ COCO images, then it strengthens the contribution --\nthe authors may call for future works to \"fine-tune\" their CLIP ViT for an additional epoch to improve their out-of-the-box usability for dense prediction tasks.\nIt would be even better if CLIP ViTs retain their original zero-shot classification and retrieval performance after such self-distillation.\nIf the authors wish to do this experiment in the rebuttal, they may consider a large dataset of diverse unlabeled images\n(e.g. images from datasets with approx. 10-20 million samples like Conceptual Captions, RedCaps, YFCC-15M, Segment Anything dataset, etc.)\n\n2. **Accounting for the 'padding' patches during distillation.**\nCurrent state-of-the-art object detectors using plain ViTs (e.g. ViTDet and SAM) typically pad input images to a square.\nAny non-square input images are made square by adding zero pixels to the right and bottom after color normalization.\nPre-trained CLIP ViTs have not seen such patches so they encounter a train-test mismatch if they are kept frozen in the object detector.\nHave the authors considered randomly adding padding regions to image patches while performing self-distillation?\nI believe that accounting for this detail will ease the adaptation of CLIP ViTs for open-vocabulary object detection.\n\n**Summary:**\nThe authors have diligently incorporated the reviewer feedback from the previous conference submission along with other significant changes.\nIn my opinion, these changes have strengthened the technical contributions and improved the presentation of this paper.\nI have a few other outstanding concerns but the technical strengths already outweigh them.\nI believe this paper is ready to be a part of the published literature -- I am leaning towards an acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ksW67CfnM0", "reviewer": "Reviewer_xSsC", "summary": "This paper proposes CLIPSelf, which is a method to adapt ViT based CLIP model trained on full images to dense prediction tasks. CLIPSelf distill a student ViT model that has good region representation from its dense feature map from a teacher ViT model that only has good full image features. Specially, CLIPSelf aligns the dense feature maps with the teacher model at crops of an image. \n\nThe proposed methods show good performance in standard benchmarks for open-vocabulary object detection, semantic segmentation and panoptic segment. It is reported to be stronger in the main metrics of the respective benchmarks to be stronger than many recent works, under the same or similar settings.\n\nOne interesting advantage of this work is that it does not need region-text pairs for training per se, unlike some recent methods such as RegionCLIP. Nevertheless, combining CLIPSelf (as pretraining step) with RegionCILP (region-text pairs for finetuning) results in further improvements for open-vocabulary detection on OV-COCO.", "questions": "- I would like to see comparisons with GLIP, GLIP v2 and Grounding DINO.\n- I would like to see if the proposed method can only improve on ViT based backbones.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strengths of the paper are\n\n* The paper is clearly written. It has sufficient details in the main text to reproduce the method.\n* The motivation of the paper is clearly presented. The paper includes an interesting analysis on the zero-shot image classification capability using image feature and region features, using CLIP ViT and CLIP CNN, respectively. The analysis shows that CLIP ViT has much better image features for zero-shot classification, but much worse dense features for the same task. This suggests that CLIP ViT has the potential to be much stronger than its CNN counterpart, but requires additional work to adapt the full image features to dense features.\n* The proposed CLIPSelf is fairly simple yet seems to be quite effective in improving the dense feature of CLIP models. This is evident in the various results on standard benchmarks from Table 3-7. The proposed CLIPSelf method seems to be among the strongest methods in standard settings in the listed benchmarks, and can improve existing methods when the CLIP pre-trained with the CLIPSelf target replaces vanilla CLIP models.\n* The details in the design of CLIPSelf have been tested carefully by ablation studies. For example, the design to use randomly sampled patches in distillation is validated by the results of classifying stuff masks.", "weaknesses": "The main weaknesses of the paper are\n\n* This paper does not include any studies on windowed attention based methods, such as Swin V2. This seems to have weakened the case for CLIPSelf, due to following reasons.\n\n  * In a sense, the analysis that compares CILP ViT and CLIP CNN can also be interpreted as \"CNN is more suited for dense prediction tasks\". Recent empirical studies do overwhelmingly show that transformer based model outperforms CNN based models in many computer vision tasks, including dense prediction tasks. From the analysis, it appears that CILPSelf is most helpful with backbones that does not have translation equivariance built-in. But many (if not most) best models for dense predictions tasks still use backbones that have this useful properties for dense predictions (e.g. Swin V2). In fact, that is the choice for some of the best recent open-vocabulary detectors, such as GLIP (v2) and grounding DINO. The choice to only study ViT backbones leave two obvious questions open: (1) Can CLIPSelf similarly benefit Swin-based backbones? (2) How does the current method compare to SOTA methods based on Swin v2, such as GLIP and grounding DINO? \n  * The square focus on ViT backbones also makes it harder, in certain cases presented in the experiments section, to interpret the comparison with prior methods. For example, in Table 3, CLIPSelf uses either ViT-B/16 or ViT-L/14 backbones. However, most of the cited methods are either from CNN backbone or Swin V2. In a few instances where the prior methods are based on ViT, the particular ViT variant is different from the two with CILPSelf.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)