Refining CLIP's Spatial Awareness: A Visual-Centric Perspective
Authors: Congpei Qiu
Yanhao Wu
Wei Ke
Xiuxiu Bai
Tong Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Self-distillation; CLIP; Open-vocabulary dense prediction
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We refine CLIP’s region-language alignment by enhancing its spatial awareness, improving performance from both visual-centric and vision-language perspectives.

Abstract: Contrastive Language-Image Pre-training (CLIP) excels in global alignment with language but exhibits limited sensitivity to spatial information, leading to strong performance in zero-shot classification tasks but underperformance in tasks requiring precise spatial understanding. Recent approaches have introduced Region-Language Alignment (RLA) to enhance CLIP's performance in dense multimodal tasks by aligning regional visual representations with corresponding text inputs. However, we find that CLIP ViTs fine-tuned with RLA suffer from notable loss in spatial awareness, which is crucial for dense prediction tasks. To address this, we propose the Spatial Correlation Distillation (SCD) framework, which preserves CLIP's inherent spatial structure and mitigates above degradation. To further enhance spatial correlations, we introduce a lightweight Refiner that extracts refined correlations directly from CLIP before feeding them into SCD, based on an intriguring finding that CLIP naturally capture high-quality dense features. Together, these components form a robust distillation framework that enables CLIP ViTs to integrate both visual-language and visual-centric improvements, achieving state-of-the-art results across various open-vocabulary dense prediction benchmarks.

Published as a conference paper at ICLR 2025
REFINING CLIP'S SPATIAL AWARENESS: A VISUAL- CENTRIC PERSPECTIVE
Congpei Qiu' Yanhao Wu' Wei Ke' Xiuxiu Bai' Tong Zhang 2:14 'School of Software Engineering, Xi'an Jiaotong University, China Schooo of Computer and Communication Sciences, EPFL, Switzerland 3 University of Chinese Academy of Sciences, Beijing, China
ABSTRACT
Contrastive Language-Image Pre-training (CLIP) excels in global alignment with language but exhibits limited sensitivity to spatial information, leading to strong performance in zero-shot classification tasks but underperformance in tasks quiring precise spatial understanding, Recent approaches have introduced Region- Language Alignment (RLA) enhance CLIP's performance in dense multimodal tasks by aligning regional visual representations with corresponding text inputs. However, we find that CLIP ViTs fine-tuned with RLA suffer from notable loss in spatial awareness, which is crucial for dense prediction tasks. To address this, we propose the Spatial Correlation Distillation (SCD) framework, which preserves CLIP's inherent spatial structure and mitigates above degradation. To further en- hance spatial correlations, we introduce a lightweight Refiner that extracts refined correlations directly from CLIP before feeding them into SCD, based on an in- triguing finding that CLIP naturally capture high-quality dense features. Together, these components form robust distillation framework that enables CLIP VITs integrate both visual-language and visual-centric improvements, achieving state-of- the-art art results across various open- -ocabullyy dense prediction benchmarks.
INTRODUCTION
CLIP models (Radford et al., 3021; Sun et al., 2023) have significantly advanced vision-language aligmment, achieving notable zero-shot classification and cross-modal retrieval performance. These models align image-level representations with text embeddings, enabling descriptions of wider categories through language. This capability has driven the development of Open- Vocabulary (OV) dense prediction, which aims to recognize broad range of visual concepts beyond predefined categories. Recent works (Liang et al., 2023; Xu et al., 2023a;b) have successfully extended CLIP's zero-shot abilities to OV dense prediction tasks using Vision Transformer (ViT) models (Dosovitskiy et al., 2021). However, CLIP's imaee-level pre-training limits its spatial precision dense cross- modal tasks (Minderer et al., 2022; Paiss et al., 2023). To address this, several approaches (Mukhoti et al., 2023; Zhong et al., 2022; Wu et al., 2023c;b) enhance CLIP's fine-grained cross cross-mode modal perception by aligning region-level visual representations with language supervision, a technique known as Region-LLangaae Alignment (RLA), extending CLIP's success to dense prediction tasks.
While acknowledging prior successes, we step back from RLA's focus language alignment to critically re-examine it from a visual-centric perspective by removing supervision from text. In dense prediction tasks, learning features with strong spatial awareness for localization and recognition is essential (Caron et al., 2021; Oquab et al., 2023; Wu et al., 2023d). Since OV dense prediction tasks extend their visual counterparts, we argue that spatial awareness in CLIP's image encoder is equally crucial. In Fig. ((a), we analyze the spatial structure of CLIP's dense features using I-SE (Van der Maaten & Hinton, 2008), and apply unsupervised segmentation with CAUSE (Kim et al., 2023d) as quantitative measure. Our preliminary findings indicate that RLA strategies, such as ReIIonCLIP (Zhong et al., 2022) and CLIPSelf (Wu et al., 2023b), result in a notable degradation
'oode will he available at https://congcciquu. hwtpnniiiugiithu..... io/Riini $ Corresponding author. It refers the understanding of the spatial relationships between visual concepts within image.

---

Published as a conference paper at ICLR 2025
Dense Feature
IE
Refiner
CLIP
CLIPSelf
Ursupeevvisd Segmembation 23.0
0
Language Supervision (RLA)
Asco Visual-centric Superviston
Our
CLUSS
RepsaCLIP
(a) Dense Feature Quality
(b) Refined-Spatial Spatial-Correlalllll guided RLA
Figure 1: (a) Evaluation of dense feature quality. We visualize the object-level dense features
of image encoder with t-SNE and present the unsupervised segmentation results. Existing Region- Language A Alignment methods lead to significant degradation of visual-centrie feature quality. (b) The framework of our fine-tuning structure. We design an additional visual-centric branch for RLA to enhance model's s spatial awareness.
in the visual-centric quality of dense features. We attribute it to the lack of spatial granularity in language supervision, which compromises the model's ability to rich visual-centric perception, rendering RLA methods suboptimal for OV dense prediction tasks. Given these insights, our objective is to improve models spatial awareness during the RLA process, enhancing OV dense prediction from both visual-centric and vision- languae perspectives.
In this paper, we propose Opatial-Correlation Correlatioo-guiieed Region-Language Alignment (SC-RLA) framework, designed to preserve the spatial awareness of CLIP ViTs during the RLA process. One key challenge is domain conflict, as the RLA process projects dense visual embeddings into a text-oriented domain, making them incompatible with visual-centric objectives. To address this, we extend the correlation distillation mechanism (Li et al., 2020; Zhang & Ma, 2023), which focuses preserving consistency of spatial relationships between visual concepts encoded by the dense features, to the cross modal domain, enabling the transfer of visual-centric spatial knowledge. Specifically, distill spatial correlations from the original CLIP ViT into the student model, enforcing consistency in spatial correlations during fine-tuning and thereby preserving the model's ' spatial awareness.
While our experiments validate the effectiveness of SC-RLA in preserving CLIP's spatial awareness, significant limitation persists: CLIP's native spatial awareness remains suboptimal (Wei al., 2023), which consequently constrains the full potential of SC-RLA. To mitigate this issue, we propose a self- supervised refinement mechanism aimed enhancing the spatial awareness of CLIP ViTs, thereby improving the supervision quality of SC-RLA. This approach is motivated by a key observation: CLIP ViTs exhibit strong inherent spatial awareness if irrelevant semantic contaminants of CLIP's feature map are filtered out. Building on this insight, we introduce a lightweight module, the Refiner, which generates siggltyulitty spatial refinements from the frozen CLIP ViTs. This process unlocks the dense perception capabilities of the model in a visual-centric manner, without requiring external supervision. By integrating the Refiner into the SC-RLA pipeline, we present R-SC-RLA, robust framework that enhances CLIP ViTs from both visual-centric and vision-language perspectives.
The effectiveness of our method is experimentally validated on the open-vocabulary dense prediction tasks, including object detection and image segmentation. With only few epochs of finetuning on small datasets like COCO (Lin et al., 2014), our method achieves non-trivial performance improvements when integrated with the recent RLA methods like CLIPSelf (Wu et al., 2023b) and RegionCLIIP (Zhong et al., 2022) for object detection tasks. For the segmentation benchmarks, our method also improves the performance of the recent state-of-the-art model Cat-Seg (Cho et al., 2023).
RELATED WORK
Open-vocabule vocabulary Dense Prediction. A rich body of research has focused on refining and transfer- ring the knowledge learned by CL.PP (Radford et al., 2021) to downstream tasks. Our approach targets
2

---

Published as a conference paper at ICLR 2025
two key areas within open- vocabulayyy dense prediction: object detection and image segmentation. In object detection, two primary strategies are commonly used: i) designing additional network struc- tures for object localization while utilizing the Vision-Language Model (VLM) encoder as a feature extractor for region-language alignment (Wu et al., 2023e; Minderer et al., 2022; Kuo et al., 2022), and ii) extending conventional detection models by learning from VI.M-provided region-language alignment signals through distillation (Du et al., 2022; Ma et al., 2022; Wang et al., 2023; Pham et al., 2024; Wu al., 2023a; Gu et al., 2021). Segmentation, which requires finer-grained cross-modal alignment, has advanced in parallel with object detection. Similar to detection strategies, segmen- tation can be addressed by generating alasstggosstic masks while leveraging VLM's vision-to to-text matching capabilities (Xu et al., 2023a; 2022; Yu et al., 2024; Ding et al., 2022), or by distilling cross-modal consistency knowledge into existing segmentation models (Chen et al., 2023a;b; Qin et al., 2023). Despite the success of these methods, they remain tailored to specific tasks. To enable broader applications, our approach focuses on fine-tuning the CLIP image encoder at the midstream stage to improve generalizability.
Region-Language alignment. Inspired by the success of language-image alignment (Radford et al., 2021; Kim et al., 2021; Li et al., 2022a), considerable attention has been directed toward facilitating RLA at various training stages. At the upstream pre-training stage, some studies introduce region-text alignment tasks using annotated visual grounding data (Li et al., 2022b; Liu et al., 2023), generate pseudo-region-level text annotations from image captions (Zhong et al., 2022). At the midstream stage, to avoid large-scale pre-training from scratch, several works (Mukhoti al., 2023; Wu et al., 2024a; Zhou et al., 2022a; Lin et al., 2023) refine image-level into form more suitable for dense-level tasks. This is achieved by training a lightweight RLA module (Mukhoti et al., 2023), extracting training- faieeng-free RLA signals (Zhou et al., 2022a). or fine- tuning the image encoder (Lin al., 2023; Wu et al., 2024a). The recent advance of CLIPSelf (Wu et al., 2023b) enhances RLA by directly aligning region representations with the text-oriented [CLS] token of the image encoder, eliminating the need for text. Since recent OV dense prediction models combines dense prediction with vision-text matching, improving the spatial awareness of the image encoder is as critical as enhancing its alignment with language signals an aspect seldom discussed in previous RLA research and key motivation for our work.
correspondence
Correlation Distillation. Correlation distillation(Gao et al., 2022a; Li et al., 2020; Zhang & Ma, 2023; Peng et al., 2019; 2023; Yang et al., 2022) is commonly utilized ensure consistency of structural correlations within feature representations between target and source feature sets. This approach typically employs correlation matrix, either within the same feature map (Peng al., 2023; Yang et al., 2022) or across different instances (Gao et al., 2022a; Li et al., 2020; Peng et al., 2019; Zhang& & Ma, 2023), to capture these structural dependencies, which are then used supervise the distillation process. In our work, we harness the spatial awareness of CLIP by leveraging spatial correlation to guide Region-Language Alignment. We demonstrate the feasibility and robustness of correlation as an effective tool for bridging the cross-modal gap, enabling vision-language models to benefit from a visual-centric perspective. Unlike conventional methods (Peng et al., 2023; Li et al., 2020; Peng et al., 2019; Zhang & Ma, 2023), our approach is unique in its multi-modal focus, utilizing spatial correlation to improve open-vocabulary dense prediction tasks.
METHODOLOGY
3.1 PRELIMINARY: REGION-LANGUAGE ALIGNMENT
Region-Language Alignment. Let CLIP's image encoder be denoted f1, with an input image X and a set of region proposals (b, | Region Legginn-annuage Alignment (RLA) methods fine-tune the student model fj. initialized from fi, to align region representations with corresponding language supervision. This alignment is achieved using the following loss function:
1 CRLA
B Σ
CAlign (RolPooling (X), T,),
(1)
where T; denotes the language supervision corresponding to region b, and LAliga represents an alignment loss, such as InfoNCE (Oord et al., 2018) or cosine similarity. As depicted in the top left of Fig. 2, we explore two key RLA mechanisms from RegionCLIP (Zhong et al., 2022) and CLIP. Self (Wu et al., 2023b). RegionCLIIP aligns region proposals with object nouns generate pseudo
3

---

Published as a conference paper at ICLR 2025
Feature Map
Regional Feature Map
Image Encoder
RolAlign
Image Encoder
LOOOO
RolAlign
Linguistic Supervision Region Embeddings
Spatial Correlation Distillation
AA
Pooling
Reglon-L.nngage Allgnment
Spatial Awareness Preservation
Figure 2: Overview of SC-RLA. The conventional RLA process (blue arrow) aligns the region representations of the student model with the corresponding language supervision signals generated by either CLIP's text encoder image encoder. We enhance this process by integrating Spatial Correlation Distillation (red arrow) to preserve the structural relationships between visual tokens.
region-text pairs, which are processed by the text encoder to obtain T, We adapt RegionCLIP's RLA process for fine-tuning following the approach of (Wu et al., 2023b), which we term RRegioText', In contrast, CLIPSelf leverages the inherent consistency between image encoder's [CLS] tokens and text embeddings, using the [CLS] tokens of cropped images defined by b, as the corresponding Tü
Limitation of RLA. As shown in Fig. (a)), RLA compromises the visual-centric quality of dense features for the alignment with the language domain (full results and technical details are provided in Appendix A). However, we argue that OV dense prediction requires a dual capability: strong consistency with language, and robust spatial awareness for dense prediction. Prioritizing only one dimension, as RLA does, is suboptimal. To address this, we propose a visual-centric solution that seamlessly integrates with RLA effectively balance both aspects.
3.2 SPATIAL-CORELLLIIONN
RLA
To enhance spatial awareness, one might consider integrating dense-level visual pre-training tech- niques (Wang al., 2021; Zhou al., 2022b) or aligning the dense features of the student and teacher models. However, these approaches conflict with RLA's goal, which projects visual-centric dense features into the language domain. To reconcile this, we introduce Spatial Correlation Distillation (SCD), inspired by correlation distillation methods (Li et al., 2020; Peng et al., 2019), shown in the bottom right of Fig. 2. To capture region-level semantics, we process the input image X through both the student model fj and teacher model f1. extracting regional features Zj Z R with sampled proposals (b. ) using RolAlign (He et al., 2017), where I. denotes the sequence length of the flattened dense features. This process is formulated as:
Z RoLAlign (fj (X), b:), Z
(rrAl)gn(ff(X)))
(2)
The spatial correlation matrices C₁₁, C₁ E RLXL are then computed as:
= (Z,", C = Z (Z
(3)
We normalize these matrices using softmax to highlight regional structural relationships: exp( cop(C (.))T)
exp(C (.) k. Σn exp ())))) (; EE exp (spp(Ct,
(4)
where 7 is temperature parameter, and Ci(j, k) is the element at coordinate k). To preserve spatial awareness of the student model, we minimize the cross-entropy loss between the student and teacher correlation matrices:
ESCd 1 Σ HCC: B
(5)
Since LSED focuses solely on spatial correlations without requiring cross-domain consistency, it integrates smoothly with RLA, guiding the fine-tuning process from visual-centric perspective. This leads to the SC-RLA objective:
ESC-RLA = CRLA + ALSCD.
(6)
where À is hyperparameter that balances the two losses.
4

---

Published as a conference paper at ICLR 2025
e (L MI
ZxpZ
Average
N=1
N
32
Zx.
Figure 3: training-free illustration of refining CLIP. We compute the average features from frozen CLIP model across diverse contexts to mitigate semantic contamination. As the number of aggregated images N increases, the model's spatial awareness improves progressively.
N Layers
K Layers
Crop
Init.
Multi-crop
A gggregation
Refining
Resize
Refiner
N Layers
55
RoTAlign
Figure 4: CLIP refining pipeline. The proposed pipeline enhances CLIP's dense representations using lightweight Refiner module. Initialized with the last K layers of CLIP's image encoder, this module aggregates corresponding tokens in global-to-loce dynamic, eliminating unnecessary contextual distortion and focusing on high-quality local semanties.
3.3 REFINING SPATIAL AWARENESS CLIP
As demonstrated in Sec.4, the SC-RLA objective significantly improves the OV dense prediction. However, CLIP's inherent spatial awareness remains limited al., 2023). To further enhance the SCD process, we propose to explicitly refine CLIP's spatial awareness.
Identifying CLIP's Dense-level Potential. Our approach driven by key observation: CLIP inherently provides robust dense representations for vision-centric perception tasks. To substantiate this, we conduct a training- free investigation, as illustrated in Fig. 3. Given a set randomly embed a predefined target image X, into each X; at random positions,
sampled images {X; J.N. producing modified images X, with X, serving as the context. These modified images are processed through CL.IP to extract the submap Z xx,, corresponding to X,, We then refine the target's features by averaging the submaps, yielding an aggregated feature map Zx
2x.
Z Xx.Xx. N Σ
In this setup, the target image X, remains constant across all X.M. with the only variation being the context provided by the different X, Compared to the direct output from CLIP, the aggregated feature map Zx,, especially for larger N. is more focused on fine nne-grained semanties. This finding reveals a critical insight: CLIP's dense features are subject to semantic contamination from con- textual information. By aggregating features from different contexts, can effectively mitigate these distortions. Further analysis, detailed in Appendix B, demonstrates that the refined features significantly enhance performance in dense prediction tasks.
Refining CLIP's Dense-level Representation. The analysis indicates that enhancing CLIP's spatial awareness in a visual-centric manner is achievable. However, aggregating large numbers of images is computationally expensive and impractical for inference. Therefore, to explicitly extract high-qualing dense features at once, we propose to train a lightweight Refiner module. It leverages the insight of the above analysis, but performs aggregation within the same image, as depicted in Fig. 4. For the frozen CLIP image encoder fi := ff o fit where f (ff² represent the final K ((nitiall N K) residual blocks of fı. we initialize the Refiner fn by cloning f Given an input image X and selected region b. fR outputs the refined feature map as:
2 = fR (RolAlign(ff(XX)))
(8)
Here, fR inherits the knowledge learned by SP and fine-tuned to extract spatially aware refinements from the output of the frozen f To train the Refiner, diverge from the common local-to-global
5

---

Published as a conference paper at ICLR 2025
Table 1: Zero-shot evaluation of dense representation. We report Top1 and Top5 mean accuracy.
Boxes Top1
Thing Masks Top1
Stuff
Masks
Backbone
Method
RPN Proposals
Top5
Top5
Top1
Top5
ViT-B/16
EVA-CLIP
18.2
33.2
20.6
36.5
18.4
43,5
ViT-B/16
CLIPSelf
X
72.1
91.3
74.4
91.8
46.8
80.2
ViT-B/16
R-SC-CLIPSelf
X
76.0
93.1
76.2
92.5
53.5
84.4
ViT-B/16
RegionText
V
71.1
90.7
73.7
91.4
34.2
68.6
VIT-B/16
TexC-ReeiooTTxt
V
72.0
91.3
74.3
91.6
41.6
73.3
ViT-B/16
CLIPSelf
V
74.0
92.6
76.3
92.8
36.8
75.0
VIT-B/16
R-SC-CLIPSelf
V
77.3
94.0
78.9
94.2
52.6
83.9
ViT-L/14
EVA-CLIP
56.7
78.0
59.0
79.8
20.8
41.9
ViT-L/14
CLIPSelf
X
77.1
93.3
78.7
93.7
44.4
78.3
ViT-L/14
R-SC-CLIPSeIF
X
82.9
96.0
82.8
95.6
57.8
86.5
ViT-L/14
CLIPSelf
V
77.8
94.0
80.4
94.5
34.0
71.8
ViT-L/14
R-SC-CLIPSeIF
81.7
95.8
82.9
95.9
52.5
83.9
approach in self-supervised learning (Zhang et al., 2022; Caron et al., 2021) and instead design global-to-local alignment mechanism. This eliminates unnecessary contextual distortion outside local region, enabling the network to focus high-quality fine- one-grained semantics, similar to the aggregation process Eq. Specifically, we randomly sample local region proposals (b() to generate C' local crops X from X. We then forward the global image X and the region by through Eq. to obtain refinements Z RLXD, and pass the context-free local crops X through fi to extract local feature maps Z I LxD, We align the corresponding tokens between Zi and Zi'. defining the Refining loss as:
1 C E
LReliner
Calign (2,, Z,),
where Latign denotes the alignment loss. In our implementation. we use InfoNCE (Oord al., 2018) for Lulign due to its robustness, treating other tokens within the same crop as negative samples. A detailed analysis of the alignment loss provided in Appendix C.2.
3.4 REFINED SPATIAL CORRELATION DISTILLATION
Overall Framework. To enhance CLIP's spatial awareness using the trained Refiner, we modify the target correlation matrix in Eq. 3 by replacing Z with the refined features Z: This allows to supervise the spatial correlations in the student model using the refined spatial structure. We refer to this process Refined Spatial Correlation Distillation (R-SCD), which forms the final R-SC-RLA framework. Notably, the refined model does not participate in the RLA branch, thereby preserving the integrity of the language supervision.
Visual-centric Application. The R-SCD process can also be applied independently to the student model, focusing solely on enhancing spatial awareness without language supervision. We call this approach Visual-centric R-SCD (R-SC-V).
4 EXPERIMENTAL RESULTS
4.1 IMPLEMENTATION DETAILS
Our full distillation consists of two stages: i) the refining of Refiner; and ii) CLIP fine-tuning stage. Although the two stages can be jointly trained in an end-to-end manner (Sec. 4.5), we first train i) to obtain a stable Refiner, then utlize the refinements to guide ii). Concretely, we use RTX 3090 GPUs for both stages with AdamW (Loshchilov & Hutter, 2017) optimizer. For the first stage, we set the learning rate to 1e - and train Refiner for epochs with the batch size as 16. For the second stage, set the learning rate 2e and perform CLIP fine-tuning for 6 epochs with the batch size as The proposals for RLA process are generated by a trained RPN, identical to (Wu et al., 2023b). Both stage are trained on COCO train2017 dataset (Lin et al., 2014). The experiments involves two

---

Published as a conference paper at ICLR 2025
Table Effects of Refiner. Comparison of distilled models with and without refining.
Boxes Top1
Thing Topl
Masks Top5
Stuff Masks
Backbone
Method
RPN Proposals
Top5
Top1
Top5
ViT-B/16 ViT-B/16
CLIPSelf
74.0
92.6
76.3
92.8
36.8
75.0
SC-CLIPSelf
76.0
93.5
77.9
93.9
49,4
82.6
ViT-B/16
R-SC-CLIPSelf
77.3
94.0
78.9
94.2
52.5
83,9
Input
CLIP
CLIPSelE
+SC
+R-SC
mioU
22.6
23.0
22.2
10.9
17.1
CLirawt
ABc
Rogion
R&CVV
(a) Point afitnttffinity Visualization
(b) Unsupervised Segmentation
Figure 5: Visual-centric analysis. (a) We visualize the affinity map w.r.t a selected query token embeddings (marked by the red dot) of the visual encoder. (b) Unsupervised segmentation evaluation with CAUSE on Cityscapes, where the mloU reported.
CLIP models: OpenAl CLIP (Radford et al., 2021) and EVA-CLIP (Sun et al., 2023). For our design specifics, Refiner is initialized with the weights of the last 4 blocks of the visual encoder for ViT-B and the last 6 blocks for ViT-L, with the early layers kept frozen. To optimize Refiner, we generate C crops per image at scale ratios between [0.3, 0.7]. During the stage of spatial correlation distillation, we set the temperature TT Ts = 0.2, with À 0.2 for ViT-B and À 0.4 for VIT-L. Further structural details of Refiner are deferred to the Appendix. C.
4.2 EVALUATION OF DENSE REPRESENTATION
Recognition Capability. We conduct dense-level zero-shot classification evaluate model recogni- tion capabilities. following the protocol in (Wu et al., 2023b). The region representations are extracted with three strategies: Boxes, which applies RolPooling COCO dataset object bounding boxes, ii) Thing Masks, and iii) Stuff Masks, both extracted via mask pooling (He et al., 2017) using COCO Panoptic dataset masks (Kirillov et al., 2019). The results are shown Tab. 1, where 'RegionTexx" refers to RegionCLIP's RLA process. Our method yields consistent and significant improvements across all settings. As shown Tab. 2, we further demonstrate the Refiner's necessity. Notably, R-SC-RLA achieves a 10% 20% improvement on COCO-Stuff using RPN proposals, where many objects are neglected by the RLA supervision. This indicates that SCD can still effectively transfer language supervision to tokens, even when they misaligned with the text.
Visual-centric Analysis. From a visual-centric perspective, we access the quality of the dense features both qualitatively and quantitatively to analyze the causes of above improvements. The visualization of point affinity maps is shown in Fig. 5, following the principle in (Bai et al., 2022) (full results provided in Appendix. F), where we calculate the cosine similarity map between selected token and the feature map. Additionally, we use CAUSE for unsupervised segmentation on Cityscapes (Cordts al., 2016) quantitative indicator. Both results demonstrate a significant improvement regarding to the quality of dense features, which is consistent with our motivation of enhancing model's spatial awareness.
4.3 OPEN-VOCABULARY DENSE PREDICTION
We evaluate the fine-tuned models via OV dense predction, including detection OV-COCO and OV-LVIS benchmarks following the protocol in (Wu et al., 2023b), and semantic segmentation following Cat-Seg (Cho et al., 2023). The corresponding details presented in the Appendix. D.
7

---

Published as conference paper at ICLR 2025
Table 3: Results on open-vocabulary object detection. We report 50 for OV-COCO and mAP, of the classes for OV-LVIS. 'SC-' denotes employing SC-RLA, and 'R-SC- denotes the full distillation strategy wtih the Refiner.
APpeedt of the novel classes
(a) OV-COCO benchmark
(b) OV-LVIS benchmark
Method
Backbone
APronl AP
Method
Backbone
mAP,
F-VLM (Kuo et al., 2022)
RNS0
28.0
BARON KD (Wi et al., 2023a)
RN50
22.6
BARON- KD (Wu
2023a)
RN50
34,0
OV-DETR (Zang el al,, 2022)
RN50
17.4
LP-OVVD (Pham al., 2024)
RNS0
40.5
(Zhou al., 2022c)
24.9
ViLD (Gu el al., 2021)
RN50
27.6
CORA+ (Wu al., 2023c)
RN50
28.1
Detic (Zhou et al., 2022c)
RN50
27.8
F-VLM (Kuo et al., 2022)
RN50x4
32.8
RegionCI.PP (Zhong al., 2022)
RN50x4
39.3
CORA (Wu al., 2023e)
RN50x4
41.7
VLDet
2022)
SwinB
26.3
CORA+ (Wu et al. 2023c)
RN50x4
43.1
Detic (Zhou al., 2022c)
SwinB
33.8
PromprOVD (Song & Bang. 2023)
VIT-B/16
30.6
PromptOVD (Song & Bang, 2023)
VIT-B/16
23.1
RO-VIT (Kim ct al., 2023b)
VIT-L/16
33.0
RO RO-ViT (Kim et al., 2023b)
VIT-B/16
28.4
CFM-VIT (Kim 2023a)
ViT-L/16
34.1
RO-VIT (Kim el al., 2023b)
VIT-L/16
32.4
DITO (Kim 2023c)
ViT-L/16
46.1
CFM-VIT (Kim
2023a)
VIT-B/16
28.8
CFM-VIT (Kim
2023a)
WT-J/16
33.9
RegionText
ViT-B/16
34.4
DITO (Kim al., 2023c)
VIT-L/16
38.4
SC-Region
VIT-B/16
35.8
CoDet (Ma et al., 2023)
VIT-L/14
37.0
R-SC-Region Text
VIT-B/16
37.0
CLIPSelf (Wu et al., 2023b)
VIT-B/16
Region
VIT-B/16
21.2
SC-CLIPSelf
ViT-B166
39.1
R-SC-RegionTex
VIT-B/16
23.6
R-SC-CLIPSelC
ViT-B/16
40.9
CLIPSelf (Wu et al., 2023b)
VIT-B/16
25.3
CLIPSelf (Wu et al., 2023b)
VIT-L/14
44.3
R-SC-CLIPSdI
VIT-B/16
27.5
SC-CLIPSelf
VIT-L/14
46.5
CLIPSelf (Wu al., 2023b)
VIT-L/14
34.9
R-SC-CLIPSelI
ViT-L/14
R-SC-CLIPSelf
VI--1/14
37.2
Table Results on open-vocabulary segmentation. We report the mloU performance. f denotes the vanilla version of Cat-Seg.
ADE-150
ADE-847
PASCAL Context
Method
VLM
mloU
MACC
mloU
mACC
mloU
mACC
SAN (Xu al., 2023b)
CLIP VIT-B/16
27.5
45.6
10.1
21.1
53.8
73.0
SAN (Xu al., 2023b)
CLIP ViT-L/14
32.1
50.7
12.4
25.2
57.7
77.6
SILC (Naeem et al., 2023)
SIL.C-CB/16
37.0
13.5
SILC (Naeem et al., 2023)
SILC-C-L/16
37.7
15.0
63.5
Cat-Segt
CLIP VIT-B/16
27.2
41.2
8.4
16.6
57.5
74.0
CCI-Segi+CIIPPSelf
CLIIP VIT-B/16
29.0
46.0
9.3
20.1
58.0
75.3
Cu--eggi+R-SC-CIIPSell
CLIP ViT-B/16
29.9
47.2
9.8
21.2
58.3
75.9
Cat-Seg
CLIP VIT-B/16
31.8
48.8
12.0
22.6
57.5
75.5
Cat-Seg Nog++Cl CLIPPeelf
CLIP VIT-B/16
30.8
48.4
11.9
21.9
56.3
75.0
- CLLISee
CLIP VIT-B/I6
32.0
48.9
12.2
22.0
57.2
75.3
S-----
CLIP ViT-B/16
32.7
49.7
12.3
22.6
58.0
76.0
Cat-Seg
CLIP ViT-L/14
37.9
55.7
16.0
28.7
63.3
80.0
CLIP VIT-L/14
38.4
56.0
16.6
29.2
63.6
80.2
Open-vocabulary Object Detection. Following (Wu et al., 2023b), we utilize a two-stage detector, F-VIT, which extracts multi-scale feature maps from the intermediate layers of the frozen EVA-CLIP model. We report the poel for novel classes on the OV-COCO dataset and mAPr for rare classes on the OV-LVIS dataset, with results presented in Tab. When combined with RLA method, such as CLIPSelf or RegionText, our SCD module consistently enhances performance, achieving a final improvement of 2% 4% across all benchmarks when further integrated with the Refiner.
Open- vocabulcyyy Semantic Segmentation. Cat-Seg, a state-of tfee----tthhrrr art model for open- vocabulayy semantic segmentation, leverages OpenAl's CLIP ViT's as its vision-lant languale backbone, followed by cost-aggregation module. We evaluate two variants: the original Cat-Seg with frozen text encoder, and an updated version with a fine-tuned text encoder. Trained on the ADE20K dataset (Zhou et al., 2017) and evaluated on ADE-847, ADE-150, and Pascal Context (Mottaghi et al., 2014), our distilled mondel-enhanced with R-SCD objective-aeeee consistently outperforms both Cat-Seg and CLIPSelf in the vanilla setup. shown in Tab. 4. siterestingly.ine-tuns tuning the text encoder in the updated Cat-Seg results a performance decline for CLIPSelf. We attribute this to the fine-tuned text encoder achieving more precise implicit agiounlang alignment, thus diminishing CLIPSelf's advantage.
8

---

Published as a conference paper at ICLR 2025
CLIPS
Figure 6: Off-the-shelf segmen-
tation with MaskCLIP.
PASCAL. Context
VLM Model
Stuff
OpenAT-CLIP
25.5
14.6
+CLIPSelf
26.4
16.1
+R-SC-CLIPSeI
27.9
17.5
DEN
29.4
18.6
+CLIPSelf
30.8
20.1
+R-SC-CLIPSeIf
32.1
21.2
Meta-CLIP
30,3
20.0
+CLIPSelf
30.1
19.7
+R SC CLIPSelf
33.6
22.0
EVA CLIP
22.8
15.6
Figure 7: Visualization of segmentation results. We visualize the segmentation results with MaskCLIP using different VLM backbones. Best viewed in color and zoomed in.
+CL
32.2
20.1
+R-SC-CLIPSeI
37.0
23.8
Table 5: Unsupervised segmentation with CAUSE. We report mloU and mACC results.
Input
DINO V2
+R-SCD
Dataset
Method
mIoU
mACC
DINO V2
29.9
89.8
Cityscapes
R-SC-V
31.8
90.5
Figure 8: Affinity map visualization of the given red point DINOv2 and DINOv2+R- SCD. Lighter regions indicate higher affinity.
DINO V2
43.0
76.9
COCO-Stuff
R-SC-V
44.1
77.4
Furthermore, CLIPSelf's limited spatial awareness contributes to this decline. To address these issues, we employ the R-SC-V objective, described in Sec. 3.3, visual-centric fine-tuning strategy, which leads to superior performance across all datasets.
Off-the-shelff Zero-shot Segmentation. We further apply our method to more CLIP's variants, including DFN (Fang et al., 2024) and Meta-CIIIP (Xu et al., 2024). We adopt the off-the-shelf segmentation protocol in MaskCLIP (Zhou et al., 2022a), which directly classifies each dense feature output by the frozen image encoder using cosine similarity with the corresponding category embedded by the text encoder. The mloU results reported in Tab, 6, showcasing the superiority and generalizability of our method. Visualization is provided in Fig. 7, with more examples in Fig. 17.
4.4 VISUAL-CENTRIC APPLICATION: ENHANCING DINO V2
DINO V2 (Oquab et al., 2023) is a self-supervised foundational model designed for vision-centric tasks. However, as highlighted by (Darcet et al., 2023), DINO V2 tends to produce dense feature artifacts, which impair its ability to capture fine- ane-grained details and result in abnormal representations dominated by global context. To address these shortcomings, we integrate R-SC-V as visual-centric enhancement module to fine-tune DINO V2. This enhancement consistently improves performance in unsupervised segmentation tasks, evidenced by results the Cityscapes (Cordts al., 2016) and COCO-Stuff (Caesar et al., 2018) datasets (see Tab. 5). Moreover, the failure cases observed in DINO V2, visualized in Fig. 8, are notably reduced after R-SC-V fine-tuning.
4.5 ABLATION STUDY
We dissect our framework and study the impact of each component to reveal the strengths of our designs. A more comprehensive investigation can found the Appendix. E.
Comparison with Correlation Distillation. We compare only the SCD method, excluding the Refiner, against several established techniques in correlation distillation (Li et al., 2020; Peng et al., 2019; 2023; Yang et al., 2022). To adjust the distillation objective, we replace the standard cross- entropy loss with the Frobenius norm of the correlation matrix, following the approach in (Yang al., 2022; Li et al., 2020), which denote as Lp. In terms of correlation matrix construction, explore two alternatives: Limer- which emphasizes maps (Peng al., 90;9); and LAun, which focuses attention values (Peng et al., 2023). Results in
correlations across various feature

---

Published as a conference paper at ICLR 2025
Table 6: Ablation on the design choices of R-SCD, We report Topl for zero-shot dense prediction and So for OV-COCO.
(a) Ablation on SCD
(b) Ablation on Refiner
Method
Boxes Topl
Stuff Topl
Thing Top1
OV-CDCO
Methoo
Boses Topl
Thing Topl
StufT
OV-COCO
CLIPSelf
74.0
76.3
36.8
37.6
CLIPSell
R-SCD
77.9
49.4
(Corelation distilaation derigns)
+C+
73.5
75.2
35.9
36.8
(Designs of Kepoer)
+Lme
73.4
75.4
37.2
37.2
+Random
76.7
78.3
50.8
39.1
74.3
76.2
36.6
37.9
+Exugumous
+PACI
76.8
78.2
514
39.9
Different vinual- centric constraints)
(Zhiining stranogles)
+Ca
65.1
67.6
29.6
27.4
w/ SCD (E2E)
76.8
78.4
51.9
40.0
+amm
75.9
w/ R-SCD (L2G)
75.2
76.9
43.2
38.0
+Lsco
76.0
77.9
49.4
39.1
+Ours
77.3
78.9
52.5
40.9
Tab. 6(a) indicate that our method, which prioritizes structural relationships within the same scene, is more effective at enhancing spatial awareness during RLA fine-tuning.
Comparison on Visual-centric Constraints. Previous work has utilized visual-centric self- supervised learning techniques (He et al., 2022: Chen al., 2020; Zhou al., 2022b) to improve the dense feature quality of CLIP (Dong et al., 2023; Li et al., 2023). However, these methods are limited image-language pre-training, where fine-grained language supervision is not concern. This raises the question of whether they are suitable for RLA fine-tuning, as discussed in Sec. 3.2. Following MaskCLIP(Dong et al., 2023), we incorporate an additional EMA model, updated via momentum from the student's weights provide visual supervision. We explore two types of constraints: (i) LMIM, which adopt masked image modeling objective as iBOT (Zhou et al., 2022b); and (ii) LaI, with dense-level contrastive loss in DenseCL (Wang et al., 2021). As shown in Tab. 6(a), these constraints fail to improve the performance, supporting our claim that typical visual-centric constraints may conflict with dense-level language supervision without non-trivial modifications.
Ablation on the Refiner's Structure. We evaluate different architectural designs for the Refiner: (i) Random Initialization, where weights are inherited from the final K attention blocks of CLIP; (ii) Exogenous, where a randomly initialized Refiner is applied on top of CLIP: and (iii) PACL, which integrates lightweight residual block (vision embedder) as proposed in PACL (Mukhoti et al., 2023). Table Tab. 6(b) demonstrates that all Refiner variants improve performance, highlighting the importance the refinement process. Our approach, which leverages the weights from the last K attention blocks of CLIP. achieves the best results, underscoring the benefit of inheriting pretrained knowledge from CLIP for the Refiner module.
Global-to-Local Refining Dynamics. We further investigate the impact of the global-to-local dynamics on training the Refiner. As illustrated in Table 6(b), a local-to-globee to-global (L2G) pipeline reversing the process in Fig. 4 leads to significant performance degradation, compared to SC-CLIPSelf without the Refiner. This confirms the necessity of our global-to-local design.
End-to-end Training. In Table 6(b), we present results from end-to-end (E2E) training, where both the Refiner and the student encoder are fine-tuned simultancously. Although the performance is slightly lower than that of the two-stage training approach, it still surpasses the CLIPSelf and SC-CLIPSelf baselines, demonstrating the flexibility of our framework. Nevertheless, we recommend the two-stage training method in practice for optimal performance.
CONCLUSION
In this paper, introduced the Spatial Correlation Distillation framework to address the issue of quality degradation in dense features when fine-tuning CLIP ViTs with Region-Language Alignment. Our approach preserves the spatial structural knowledge of the model and incorporates the Refiner module to further enhance CL.PP's spatial awareness, leading to notable performance gains on open- vocabulary dense prediction benchmarks. Our work highlights the critical role of spatial awareness in vision-language models from visual-centric perspective, extending beyond mere linguistic alignment. The experimental results demonstrate that our framework enables CLIP ViTs to integrate both vision-language and visual-centric enhancements, providing novel avenue for advancing dense-level perception in CLIP-based models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant No. 62376209 and 62472349.
REFERENCES
Yutong Bai, Xinlei Chen, Alexander Kirillov, Alan Yuille, and Alexander C Berg. Point-level region contrast for object detection pre-training. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16061-16070, 2022.
Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, and Fahad Shahbaz Khan. Bridging the gap between object and image-level representations for open- vocabulary detection. Advances in Neural Information Processing Systems, 35:33781-33794,
2022.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1209-1218, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervied vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Jun Chen, Deyao Zhu, Guocheng Qian, Bernard Ghanem, Zhicheng Yan, Chenchen Zhu, Fanyi Xiao, Sean Chang Culatana, and Mohamed Elhoseiny. Exploring open-vocabulary semantic segmen- tation from clip vision encoder distillation only. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 699-710, 2023a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020.
Xi Chen, Shuang Li, Ser-Nam Lim, Antonio Torralba, and Hengshuang Zhao. Open-vocabulary panoptic segmentation with embedding modulation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1411-1150, 2023b.
Seokju Cho, Heeseong Shin, Sunghwan Hong. Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. arXiv preprint arXiv:2303. 11797, 2023.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16655 2023,
Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11583-11592, 2022.
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskelip: Masked self-distillation advances contrastive language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10995-11000 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICI.R, 2021.
11

---

Published as a conference paper at ICLR 2025
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- apen-vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. [4084-14093, 2022.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: /oppnreviiiieeeeeeeeeoeeeeeeeeee
net/ (orumm?id=KAkk6g770555
Huan Gao, Jichang Guo, Guoli Wang, and Qian Zhang. Cross-domain correlation distillation for unsupervised domain adaptation in nighttime semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1119999112999 9923, 2022a.
Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding- box labels. In European Conference on Springer. 2022b.
Computer Vision, pp.
Xiuye Gu, Tsung- Yi Lin. Weicheng Kuo, and Yin Cui. Open- vocabuocy object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.1392) 2021.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings the IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Dahun Kim, Anelia Angelova, and Weicheng Kuo. Contrastive feature masking open-vocabulary vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15602-1560 15612, 2023a.
Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for open-vocabulary object detection with vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1 1114-111554 2023b.
Dahun Kim, Anelia Angelova, and Weicheng Kuo. Detection-oriented image-text pretraining for open-vocabulary detection. arXiv preprint arX1v:2310.00161, 2023c.
Junho Kim, Byung-Kwan Kwan Lee, and Yong Man Ro. Causal unsupervised semantic segmentation. arXiv preprint arX1::2310.07379, 2023d.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convo- lution region supervision. In International conference on machine learning, pp. 5583-5594. PMLR, 2021.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1399-64008 2019.
Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-ylm: Open-vocayulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209. 15639, 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping
pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zeang.een--Nee Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975, 2022b.
Xiaojie Li, Jianlong Wu, Hongyu Fang, Yue Liao, Fei Wang, and Chen Qian. Local correlation consistency for knowledge distillation. In European Conference on Computer Vision, pp. 18-33. Springer, 2020.
12

---

Published as a conference paper at ICLR 2025
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision, pp. 280-296. Springer, 2022c.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language- image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pp. 23390 234000, 2023.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kumpeng Li, Yinan Zhao, Hang Zhang. Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open- vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object detection. arXiv preprint wxXi::2211.14855 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Yuqi Lin, Minghao Chen, Wenxiao Wang. Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu, and Xiaofei He. Clip is also an efficient segmenter: A text-driven approach for weakly supervised semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15305-15314, 2023.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-trainin tee-traiinng for open-set object detection. arXiv preprint Siii::2303.05499 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1711v: 2017.
Chuofan Ma. Yi Jiang, Xin Wen, Zehuan Yuan, and XIAOJUAN QI. Codet: Co-occurrence guided region-word alignment for open-vocabulary object detection. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: http///ppeeeeeeeeeeeeeeeeee net/ forum? id (d-Tkjx41IP7a
Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. Open-vocabulary one-stage detection with hierarchical visual-language knowl- edge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14074-14083, 2022.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani. Zhuoran Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pp. 728-755. Springer, 2022.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-W Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
891-898, 2014.
Jishnu Mukhoti, Tsung- Yu Lin, Omid Poursaeed, Rui Wang, Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
19413-19423, 2023.
Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool, and Federico Tombari. Sile: Improving vision language pretraining with self-distillation. arXiv preprint arXiv:2310.13355, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: arXiv:/807.0223 87880033748 2018.
13

---

Published as a conference paper at ICLR 2025
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo. Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang- Bhang-Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip to count to ten. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3170-3180, 2023.
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5007-5016, 2019.
Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23641-23651 2023.
Chau Pham, Truong Vu, and Khoi Nguyen. Lp-ovod: Open-vocabulary object detection by linear probing. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 779-788, 2024.
Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, et al. Freeseg: Unified, universal and open-vocabulary image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19446-19999 19455, 2023.
Congpei Qiu, Tong Zhang, Wei Ke, Mathieu Salzmann, and Sabine Süsstrunk. De-coupling and de-positioning dense self-supervised learning, arXiv preprint arXiv:2303.16947 2023a.
Congpei Qiu, Tong Zhang, Yanhao Wu, Wei Ke, Mathieu Salzmann, and Sabine Süsstrunk. Mind your augmentation: The key decoupling dense self-supervised learning. In The Twelfth International Conference on Learning Representations, 2023b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp- 8748-8763. PMIR, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I:: Long Papers), pp. 2556-2565, 2018
Hwanjun Song and Jihwan Bang. Prompt-guided transformers for end-to-end open-vocabulary object detection. arXiv preprint arXiv:2303. /4386, 2023.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Luting Wang, Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi. Biaolong Chen, and Liu. Object-aware distillation pyramid for open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11186-11196 2023.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for supervude visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3024-3033 2021.
14

---

Published as a conference paper at ICLR 2025
Yixuan Wei, Han Hu, Zhenda Xie, Ze Liu, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Improving clip fine- tuning performance. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5439-549444 5449, 2023.
Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open vocabulayy object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 15254-15264, 2023a.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Xiangtai Li, Wentao Liu, and Chen Change Loy. Clipselff: Vision transformer distills itself for open- open-vocabulary dense prediction. arXiv preprint arXiv:2310.014000 403,, 2023b.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Wentao Liu, and Chen Change Loy. Clim: language-image mosaic for region representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38. pp. 6117-6125, 2024a.
Contrastive
Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 7031-7040, 2023c.
Yanhao Wu, Tong Zhang, Wei Ke, Sabine Süsstrunk, and Mathieu Salzmann. Spatiotemporal self- supervised learning for point clouds in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5251-5260, 2023d.
Yanhao Wu, Tong Zhang, Wei Ke, Congpei Qiu, Sabine Süsstrunk,, and Mathieu Salzmann. Mitigating object dependencies: Improving point cloud self-supervised learning through object exchange. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23052-23061 2024b.
Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, 2024. URL https: // //openrevien net/ www.mmmdddBBFFfEE...
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2955-2966, 2023a.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. simple baseline for open- gpecbuocayylary semantic segmentation with pre-trained vision-language model, In European Conference on Computer Vision, pp. 736-753. Springer, 2022.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2945-2954, 2023b.
Dongbao Yang, Yu Zhou, Aoting Zhang, Xurui Sun, Dayan Wu, Weiping Wang, and Qixiang Ye. Vilt correlation distillation for incremental object detection. Pattern Recognition, 131: 108863, 2022.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chich Chen. Convolutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In European Conference Computer Vision, pp. 106-122. Springer, 2022.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open- vocabularr object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14392-14402, 2021.
Linfeng Zhang and Kaisheng Ma. Structured knowledge distillation for accurate and efficient object detection. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
15

---

Published as a conference paper at ICLR 2025
Tong Zhang, Congpei Qiu, Wei Ke, Sabine Süsstrunk, and Mathieu Salzmann. Leverage your local and global representations: A new self-supervised learning strategy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16580-16589, 2022.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pre- training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16793-16803, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and patter recognition, pp. 633-641, 2017.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pp. 696-712. Springer, 2022a.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022b.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühhl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In European Conference on Computer Vision, pp. 350-368. Springer, 2022c.
16

---

Published as a conference paper at ICLR 2025
APPENDIX CONTENTS
The appendix is structured as follows:
Appendix A presents comprehensive experiments evaluating the dense features of various fine-tuned CLIP ViTs.
Appendix B provides analysis of the dense features from the original CLIP ViTs, serving as empirical evidence for our refining strategy.
Appendix C details the design of the Refiner, supplemented with ablation studies and further empirical study.
Appendix D outlines the implementation details for the open-vocabulary dense prediction tasks.
Appendix includes additional ablation studies for the overall framework.
Appendix F provides the implementation details for point-affinity visualization, along with further visual results.
17

---

Published as a conference paper at ICLR 2025
Input
N =
= 4
N = 32
mloU
25.2
24.5
24.5
22.6
CLIP
N-4
N-16
N-32 Refiner
(a) Visualization Results
(b) Unsupervised Segmentation
Figure 9: (a) point-affinity visualization with different number N of aggregated images. An increasing N tends to rendering dense features with better spatial awareness. Best viewed color and zoomed in. (b) When the semantic contamination of dense features is effectively eliminated with large N, unsupervised segmentation present significant performance improvement. 'Refiner' denotes utilzing the output of our trained Refiner for inference.
A VISUAL-CENTRIC EVALUATION OF DENSE FEATURES
A.1 UNSUPERVISED SEGMENTATION.
As Oquab al. (2023) argue, a powerful pre-trained visual encoder can produce dense features that are directly applicable to unsupervised segmentation, even surpassing the performance of fine-tuned methods. Building on this insight, we perform unsupervised segmentation using the state-of-the-ar CAUSE (Kim et al., 2023d) numerical indicator to assess the quality of the dense features generated by frozen visual encoder.
A.2 T-SNE OF DENSE FEATURES.
I-SNE (Van der Maaten & Hinton, 2008) is widely used technique for projecting high-dimensional embeddings into a lower-dimensional space for visualization. In our experiments, we first extract instance-level features by applying masked average pooling to the dense features generated by an image encoder, using the ground-truth segmentation masks to define the pooling regions. We then apply t-SNE to project the extracted object-level dense features into 2D space for visualization. To enhance clarity, we randomly sample 256 instances from each category and select 7 categories for each visualization. The images and corresponding annotations are taken from the COCO train2017 dataset. More visualizations are provided in Fig. 15.
B ANALYSIS ON DENSE-LEVEL POTENTIAL OF CLIP
B. EXTRACTING HIGH-QUALITY DENSE FEATURES FROM FROZEN CLIP
As an experimental complement, we present more visualization results in Fig. 9a), which presents clearer trend that when the number N modified images M Fig. 3 increases, the dense features tend to be more spatially aware and aligned with the object boundaries. For the qualitative evaluation, a large N as 32 yields 2% mloU improvement in the unsupervised segmentation without any training process. This observation demonstrates the dense-level potential of the CLIP image encoder once eliminate the irrelevant distractions hindering dense feature quality, the aggregation operation acts as average filter to filter out the semantic contamination.
18

---

Published as a conference paper at ICLR 2025
Input Image
CLIP
Noise
Aggregated
Refined
Figure 10: Point-aftinity visualization of dense features. From left right: CLIP's original feature map, semantic contamination, aggregated dense features, and the output of the trained Refiner.
B.2 EFFECTS OF REFINER
If consider the aggregated features Zx, in Eq. the target features, for each feature map Z directly output by CLIP image encoder, we define the noise pattern e := Z Zx, as the deviation from the target features. As in Fig. 10, the noise results in meaningless correlation, irrelevant to the fine-grained visual concepts. For the effects of our desinged Refiner, Shown in Fig. 9b), the trained Refiner exhibits more effectiveness in unsupervised segmentation than both the original dense features and the aggregated feature, demonstrating the necessity of Refiner.
C REFINER
C.I DESIGN CHOICES
Layers
Layers
Multi-crop Aggregation
Init.
[CLS]
N - K Layers
RolAlign
Refiner
Avg-pool
Internediate processor
[CLS]
Generator
Figure 11: The architecture of the proposed Refiner. The framework consists three components: a Refiner head, an Intermediate processer, and region-level (CLS) token generator.
The Refiner consists of three components: a Refiner head, intermediate processor, and region- level [C'LS] token generator. We here detail the design of the latter two components.
Region-level [CLS] generator. The [CLS] token of ViT contains the global information of the image input. As the [CLS] is directly bonded with the full image, to integrate with region-level features, the patch tokens in earlier layer output corresponding to region bounding box b, are fused with Rol pooling and subsequently forwarded to two-layer MLP with hidden size of 4096, which derived as:
[CLS|
FCa.s [RolPool (X), b:)]
(10)
Intermediate processer. To extract refined dense representations from earlier layers fA. instead of solely processing its final outputs, also utilize the output tokens from the 1, 12-th layer as the
19

---

Published as a conference paper at ICLR 2025
Table 8: OV-COCO detection results with different loss. We report the AP and A results.
Input
SCD-Cos
SCD-NCE
OV-COCO
Model
Method
A 50
AP
VIT-B/16
CLIPSelf
37.6
54.9
Figure 12: Affinity map with different loss.
ViT-B/16 VIT-B/16
SCD-Cos
34.5
50.1
We present the affinity map obtained with Re- finer for Cosine loss and InfoNCE loss.
SCD-NCE
40.9
54.7
intermediate auxiliary imput, i.e.:
Z. = fæ (RoIAlign( (f (X) + Inter b,)) Inter = FCinter [Concat (Zi,, Zr,),,
(11)
where the multi-scale processer FCinter TR2D - RDD is two-layer MI.P with hidden size of 4096. For the visual encoder of ViT-B, we set 1 = 4 and l2 7, and for ViT-L, set 4 9 and l2 14.
C.2 MORE ABLATION REFINER
Designs of Refiner. We dissect the components of the Refiner to investigate their contributions and present the results in Tab. 7. Both the Intermediate processer and the [CLS] generator contribute to the extraction high- quality refined spatial correlation, which is crucial for the distillation process, thus yielding performance improve- ment with both components enabled. Additionally, instead of local regions defined by the proposals, we also explore the 'Late" setting where we perform RolAlign on the out- put of Refiner, i.e.:
Table 7: Ablation on different compo- nents in Refiner. We report Apnovel on OV-COCO.
FC
FCcLS
Late
Apnovel
40.1
40.3
40.9
v
40.2
2, RoLAlign (fR(fA(X)) b.)
(12)
However, this setting leads to performance degradation, indicating the necessity to focus model's attention on the local regions.
Cosine vs. InfoNCE. Our original Refiner objective with the InfoNCE loss is derived as:
exp( 2 2(D)
LNCE -
Σ- log
(13)
Σ. exp 2 Z([A))*
j=1
To demonstrate the necessity of InfoNCE for training the Refiner, we conduct an additional experiment by replacing Eq. 9 with the cosine loss:
--
Ccos = E Σ cos (.11,,
(14)
We visualize the affinity map calculated with the dense features output by the Refiner in Fig. 12, where the selected token can entangled with its irrelevant surroundings. This phenomenon harms the Refiner for extracting high-quality refinements, leading to performance drop presented in Tab. 8. In contrast, the mura-feature-mar contrast in Eq. further filters out interference from irrelevant neighboring tokens, effectively tackling this issue.
C.3 SEMANTIC COUPLING CLIP
To further assess whether the Refiner's effects align with its design objectives, conduct a quan- titative analysis to evaluate the tendency of CLIP's dense features to become entangled with irrel- evant context, referred to semantic coupling in Fig. 13, which is also observed by recent works (Qiu et al., 2023a;b; Wu et al., 2024b). Specifically, we concatenate two independently sampled images X, and X, z side by side, denoted as X AB, which introduces context disturbance
20

---

Published as a conference paper at ICLR 2025
Semantic contamination
Cosine Similarity
IE
XAB
Z ZAIAB
Z
Coupling
concat
Feature Map
Ratio
Cosine
IE
Similarity
XA
X B
A
B
Figure 13: Measuring pipeline semantic coupling. We concatenate two independently sampled images Xa and XB to analyze the semantic contamination between them. The defined coupling ratio reflects the significance semantic coupling.
from n to XA. We forward XAB, XA XB the image encoder obtain regional feature map Z AJABB BJAB Zz BB. Finally, the coupling ratio is computed as:
cos Z AIAB [1. ZB AlA
CR = E
= arg max cos Z [AAAB(i), B
(15)
cos cos [ ZB(l)
where we identify the most similar token in ZB An to the token in ZalaR, and analyze whether this similarity arises from the entanglement of irrelevant semantics introduced by the concatenation operation. Ideally, the CR value is expected to be close to 1, as XA and Xx, possess independent semantics. By calculating the average CR value across COCO va/2017, we report the measured CR value in Tab. 9. The results indicate that both the original and LLIPSelf-finetuned CLIP models are significantly affected by semantic coupling. In contrast, our proposed Refiner effectively addresses this issue, demonstrating high consistency with its intended design goals of climinating semantic contamination.
Table 9; CR value of different models. We report the CR values with different finetuning strategies using EVA-CLIP.
Method
EVA-CLIP
w/ CLIPSelf
EVA-CLIP-Refiner w/ R-SC-CLIPSeel
CR
2.32
1.86
0.95
0.97
D IMPLEMENTATION DETAILS OF OPEN-VOCABULARY DENSE PREDICTION
D. OPEN-VOCABULARY OBJECT DETECTION.
We adopt F-VIT (Wu et al., 2023b) the open-vocabulary object detector, which replaces the simple Feature Pyramid Network (FPN) ViTDet (Li et al., 2022c) detector with standard FPN and utilizes the feature maps from multiple intermediate layers of the VIT. The entire visual encoder is keep frozen during the training process. The F-VIT model is trained for epochs for the OV-COCO benchmark and 48 epochs for the OV-LVIS benchmark. Following the common practice, the box AP with loU threshold of 0.5 on the novel classes reported for OV-COCO, and the mean mask AP is reported for OV-LVIS.
D.2 OPEN-VOCABULARY SEMANTIC SEGMENTATION.
We utilize two version of Cat-Seg (Cho et al., 2023) for the open-vocabulary semantic segmentation task. Both the vanilla and updated versions Cat-Seg fine-tune the attention weights the vision encoder and the additional cost aggregation module. The main difference at the level of VLM is that the vanilla version freezes the text encoder of CLIP, while the updated version fine-tunes the text encoder to implicitly align the vision and text representations. The model is trained the ADE20K (Zhou et al., 2017) dataset. We evaluate the model on three benchmarks: A-150 and A-847, which contain 150 and 847 classes respectively, and Pascal Context (Mottaghi et al., 2014) dataset
21

---

Published as a conference paper at ICLR 2025
Table 10: Full comparison on OV-COCO benchmark.
Method
Backbone
Pnovel
A APbuse Pbuse
A AP6o
OV-RCNN (Zareian et al., 2021)
RN50
17.5
41.0
34.9
RegionCLIP (Zhong al., 2022)
RN50
26.8
54.8
47.5
RegionCLIP (Zhong et al., 2022)
Rn50
31.4
57.1
50.4
RegionCLIP (Zhong al., 2022)
RN50x4
39.3
61.6
55.7
ViLD (Gu et al., 2021)
RN50
27.6
59.5
51.2
OV-DETR (Zang al., 2022)
RN50
29.4
61.0
52.7
PB-OVD (Gao et al., 2022b)
RN50
30.8
46.1
42.1
Detic (Zhou et al., 2022c)
RN50
27.8
51.1
45.0
OC-OVD (Bangalath et al., 2022)
RN50
36.6
54.0
49.4
VLDet (Lin al., 2022)
RN50
32.0
50.6
45,8
F-VLM (Kuo al., 2022)
RN50
28.0
39.6
BARON-Cap (Wu et al., 2023a)
RN50
33.1
54.8
49.1
BARON-KD (Wu al., 2023a)
RN50
34.0
60.4
53.5
BARON-Cap&KD (Wu et al., 2023a)
RN50
42.7
54.9
51.7
OADP (Wang et al., 2023)
RN50
35.6
55.8
50.5
CORA (Wu et al., 2023c)
RN50
35.1
35.5
35.4
CORA (Wu al., 2023c)
RN50x4
41.7
44.5
43.8
CORA+ (Wu et al., 2023c)
RN50x4
43.1
60.9
56.2
RO-ViT (Kim et al., 2023b)
VIT-B/16
30.2
41.5
RO-VIT (Kim et al., 2023b)
ViT-L/16
33.0
47.7
CFM-ViT (Kim et al., 2023a)
ViT-L/16
34.1
46.0
DITO (Kim et al., 2023c)
ViT-L/16
40.8
50.3
CLIPSelf (Wu et al., 2023b)
VIT-B/16
37.6
54.9
50.4
R-SC-CLIPSeIf
VIT-B/16
40.9
54.7
51.1
CLIPSelf (Wu al., 2023b)
ViT-I/14
44.3
64.1
59.0
R-SC-CLIPSeff
ViT-L/14
48.1
65.4
60.8
22
20
18
R.SC-CLIPSeI CLIPSei!
RUSC-CLIPSei CLiPSed
SCD Ratio
OV-COCO open-vocabulary object detection
Unsupervised segmentation on Cityscapes
Figure 14: Ablation on spatial correlation distillation. We control the loss ratio of SCD and report APponel OV-COCO detection and mloU Cityscapes segmentation.
with the PC-59 benchmark. The baseline of Cat-Seg is conducted by rerun the training process with the official released code.
E FURTHER ABLATION STUDIES
SCD Ratio A. We conduct an ablation study with various SCD ratios À to investigate the effects of the spatial correlation distillation. We evaluate the performance of the distilled model on two levels: i) the opca-vocabulary object detection on OV-COCO and ii) the unsupervised segmentation on Cityscapes (Cordts et al., 2016) with CAUSE (Kim et al., 2023d). All the models are fine-tuned on COCO train2017 dataset for epochs following the setting of CLIPSelf with proposals, except the
for 'R-SC-V" that focuses the visual-centtn fine-tuning. As the OVOD task weights vision-to-tess alignment capability, we additionally involve the unsupervised segmentation task to evaluate the quality of the dense representations. As depicted in Fig. 14(b), the alignment between
22

---

Published as a conference paper at ICLR 2025
Table 1I: Full comparison on OV-LVIS benchmark.
Method
Backbone
mAP,
mAP.
MAP,
mAP
RegionCIIIP (Zhong et al., 2022)
RN50
17.1
27.4
34.0
28.2
RegionCIIIP (Zhong et al., 2022)
RN50x4
22.0
32.1
36.9
32.3
Detic (Zhou al., 2022c)
RN50
24.9
32.4
Detic (Zhou et al., 2022c)
SwinB
33.8
47.0
VLDet (Lin et al., 2022)
RN50
21.7
29.8
34.3
30.1
VLDet (Lin et al., 2022)
SwinB
26.3
39.4
41.9
38.1
ViLD (Gu et al., 2021)
RN50
16.6
24.6
30.3
25.5
OV-DETR (Zang al., 2022)
RN50
17.4
25.0
32.5
26.6
DetPro (Du et al., 2022)
RN50
19.8
25.6
28.9
25.9
BARON-KD (Wu al., 2023a)
RN50
22.6
27.6
29.8
27.6
OADP (Wang et al., 2023)
RN50
21.7
26.3
29.0
26.6
OC-OVD (Bangalath et al., 2022)
RN50
21.1
25.0
29.1
25.9
F-VLM (Kuo et al., 2022)
RN50
18.6
24.2
F-VLM (Kuo et al., 2022)
RN50x4
26.3
28.5
R-VLM (Kuo et al., 2022)
RN50x16 16
30.4
32.1
F-VLM (Kuo et al., 2022)
RN50x64
3288
34.9
CORA (Wu et al., 2023c)
RN50x4
22.2
CORA+ (Wu et al., 2023c)
RN50x4
28.1
OWL-ViT (Kim al., 2023b)
ViT-L/14
25.6
34.7
RO-ViT (Kim et al., 2023b)
ViT-B/16
28.0
30.2
RO-VIT (Kim et al., 2023b)
VIT-L/16
32.1
34.0
RO-ViT (Kim et al., 2023b)
VÝT-H/16
34.1
35.1
CFM-ViT (Kim et al., 2023a)
ViT-L/16
33.9
36.6
DITO (Kim et al., 2023c)
ViT-L/16
38.4
37.7
CoDet (Ma al., 2023)
ViT-L/14
37.0
CLIPSelf (Wu et al., 2023b)
ViT-B/16
25.3
21.8
29.1
25.2
R-SC-CLIPSelf
VIT-B/16
27.5
22.7
29.8
26.3
CLIPSelf (Wu et al., 2023b)
ViT-L/14
34.9
34.6
35.6
35.1
R-SC-CLIPSelf
VIT-L/14
37.2
37.2
37.1
37.2
the visual and (CL.]] token presented by CLIPSelf causes the degradation of the segmentation performance, With the SCD loss that extracts and maintains the spatial correlation, the performance degradation is mitigated, achieving the balance between the vision-to-tex alignment and the dense- level understanding. Moreover, when applying the R-SC-V V loss, the performance is further improved with non-trivial margin. In addition, observed in Fig. 14(a), SCD loss significantly boosts the performance on OV-COCO, even for the R-SC-V model without a RLA branch, indicating the importance of the refined spatial awareness holds for the OVOD task.
Depth of the Refiner. We investigate the impact of the depth the Refiner on the performance of the distilled model. The depth will affect the distillation process from two aspects: the balance between the capacity of refining and preserving the original visual knowledge learned by the visual encoder, and ii) the computational efficiency the training process. We conduct experiments with different depths of the Refiner A deeper Refiner will increase the parameter size and the complexity of the fine-tuned model, but more difficult to perserve learned knowledge of the pre-trained model. As shown in Tab. 12, the model with 4-yerr Refiner achieves the best performance, obtaining balance between the refining capacity and the knowledge preservation.
Temperature of Spatial Correlation Distillation. We conduct ablation study on the temperature of the spatial correlation distillation as shown in Tab. 13. The temperature т, and T. of the student and teacher logits respectively control the softness of the spatial correlation distillation. Generally, a sharpening process with т₃ > Tı typically leads to higher performance than the distillation with Ts Tt. But find equal temperature setting of T, т 0.2 achieves the best performance. which indicates that the denoised spatial correlation stems from the intra-scene contrast loss already sharp enough.
23

---

Published as a conference paper at ICLR 2025
Table 12: Ablation on the depth of the Refiner. We report the Apnovel on OV-COCO and the Topl performance of zero-shot classification on COCO
OV-COCO S
Boxes Top1 Acc.
Thing Masks
Stuff Masks
Depth K
Top1 Acc.
Top1 Acc.
2
40.3
76.3
78.0
50.7
40.7
77.0
78.5
52.5
4
40.9
77.3
78.9
52.5
40.5
76.9
78.1
51.6
Table 13: Ablation on the temperature of the spatial correlation distillation. We report the APnowel on OV-COCO
OV-COCO
Ts
AP
APhase
AP50
0.1
0.1
39.2
54.0
50.2
0.15
0.15
39.6
53.5
49.9
0.2
0.15
39.2
53.3
49.6
0.2
0.2
40.9
54.7
51.1
0.2
0.3
38.5
54.5
50.4
0.25
0.25
39.8
53.9
50.2
Local vs. Global Distillation. For spatial correlation distillation, we utilize sampled bounding box define the region for distillation. Here investigate another setting that directly distills the spatial correlation of the entire image to the student model, i.e. defining the region bounding box as the whole image area. The results are presented in Tab. 14. Though still effective with performance improvement, global distillation significantly underperforms local distillation, which aligns with our intuition facilitate the model focus on the local.
Table 14: Comparison of local and global distillation strategy. We report the Apnovel on OV- COCO and the Topl performance of zero-shot classification on COCO
OV-COCO Apnovel
Boxes Topl Acc.
Thing Masks
Stuff Masks
Strategy
Topl Acc.
Topl Acc.
CLIPSelf
37.6
74.0
76.3
36.8
Local
40.9
77.3
78.9
52.5
Global
39.7
75.6
77.8
50.2
Training on larger-scale dataset. We further fine-tune the Refiner and train EVA-CLIP with R- SC-CLIPSelf on CC3M (Sharma et al., 2018) for one epoch, evaluating the performance using MaskCLIP. As presented in Tab. 15, model benefits from the larger-scale dataset, achieving improved multi-modal dense prediction performance.
F VISUALIZATION
F.1 AFFINITY MAP
As presented in Fig. 16, the query token marked with red dot, and the cosine similarity between the query token and the feature map is calculated for the visualization. We visualize the vanilla CLIP, CLIPSelf, R-SC-CLIPSelf, RegionText, and R-SC-Region Text respectively.
F.2 MASKCLIP SEGMENTATION
As presented in Fig. 17, adopt off-the-shelf zero-shot segmentation with MaskCLIP (Zhou et al., 2022a) and present the results of visualization with EVA-CLIP and Meta-CLIP backbones.
24

---

Published as a conference paper at ICLR 2025
Table 15: Off-the-shelf segmentation with MaskCLIP.
PASCAL Context
COCO
Method
Dataset
Stuff
R-SC-CLIPSelf R-SC-CLIPSeIf
COCO
37.0
23.8
CC3M
38.2
25.0
CLIP
CLIPSelf
E-SC-CLIPSell
CLIP
CLIPSelf
R-SC CLIPSelf
CLIP
CLIPSelf
R SC-CLiPSelf
Figure 15: Visualization of t-SNE. In each row, we visualize the dense features with the same set of
categories. We respectively present the results of vanilla CLIP, CLIPSelf, and R-SC-CLIPSelt
25

---

Published as a conference paper at ICLR 2025
Input
CLIP
CLIPSelf
+R-SCD
Region
+R-SCD
Figure 16: Visualization of affinity map. We present the affinity map obtained with the vanilla CLIP, CLIPSelf, R-SC-CLIPSelf RegionTexx, and R-SC-RegionTexx respectively. The query token is marked with a red dot.
26

---

Published as a conference paper at ICLR 2025
Input
EVA-CLIP
CLLIPSelf
R-SC-CLIPSeII
Meia-CLIP
+CLIPSelf
R-SC-CLIPSSe
MUU
CaU
RALLI
Figure 17: Visualization of segmentation results with MasKCLIP, We present the visualization results of MaskCLIP segmentation with EVA-CLIP and Meta-CLIP, Best viewed with color and zoomed in.
27


Reviews:
[{"review_id": "OXitIL2guS", "reviewer": "Reviewer_FR8o", "summary": "This paper aims to improve upon the Region-Language Alignment (RLA) approaches for Contrastive Language-Image Pre-training (CLIP) models. In order to not only promote the linguistic alignment but also preserve the spatial awareness, Spatial Correlation Distillation (SCD) is proposed to plug into the existing methods such as RegionCLIP and CLIPSelf. Refiner is also introduced to enhance the regional representation of teacher model. The experiments on the open-vocabulary dense prediction tasks demonstrate the effectiveness of the proposed method.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Motivated by the observation that RLA methods suffer from notable loss in spatial awareness for CLIP ViTs, SCD is specifically designed to capture the spatial structure in a region. The widespread experiments show the superior performance of the proposed method across multiple open-vocabulary dense prediction benchmarks.", "weaknesses": "The motivation to design Refiner has been implied in CLIPSelf. When K = N, the proposed Refiner is almost the same as CLIPSelf, which reduces the technical novelty. Also, the ablation study of K is absent in this work.\n\nThe application of the proposed method is restricted. Spatial Correlation Distillation (SCD) is auxiliary and is to preserve the spatial awareness when another optimization is applied (e.g. RLA). Therefore, it seems that SCD cannot be applied independently since in this case the weights of teacher model and student model are always equal. Besides, R-SC-V is only learned from the teacher model that is optimized by Refiner (similar to CLIPSelf), so it cannot be further applied to CLIPSelf based approach.\n\nIn order to showcase the generalizability and scalability of the proposed method, the experiments with data scaling up are expected to be provided, which is missing in the current version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VSsjLvVMsF", "reviewer": "Reviewer_SSGt", "summary": "This paper proposes a framework called Spatial Correlation Distillation(SCD) to refine the spatial awareness of CLIP. To recall the visual perception of vanilla CLIP loss in Region-Language Alignment (RLA) training, it proposes a Spatial Correlation Distillation training process and a light-weight refiner to distill the visual relationship from the vanilla CLIP. SCD can improve the existing OV methods and achieve the new sota on the ov tasks.", "questions": "1. In the SC-RLA, how to ensure the correspondence between the RoI regions from teacher and student models? The student model would be trained, so the number, order and attribute of the RoI regions would become different with the frozen image encoder.\n\n2. In the line 242 -258, would the contents of embeded sampled images {X_i} affect the SPATIAL AWARENESS? Could you provide some quantitative analysis like the cosine similarity between the {X_i} and X_t ?\n\n3. Since the motivation and module is to improve spatial awareness (which can also be seen from the visualization), are there more segmentation related visualizations? Qualitative results using segmentation would be more convincing (e.g. finer edges)", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Experiments. The experiments are sufficent and solied to prove the improvement of framework.\n\n2. Significance. The framework is a convience, plug and play, and effecitive pipeline to improve the existing methods of the OV tasks.\n\n3. Motivation. The motivation is clear. The Fig.1 and intro explain the loss of dense prediction ability of RLA process.", "weaknesses": "1. Novelty. The frame work is composed with SCD and a refiner and the novelty of the SCD is limited. The SCD is a common distillation module to reserve the similarity between the RoI features of student and teacher models. \n\n2. Lack of analysis. This paper does not provide a deeper analysis of the experimental results. For example, why R-SC can effectively improve the classification accuracy in Tab.1? In terms of motivation and structure, this module improves spatial perception and does not have much gain in classified task with the annotated masks..\n\n3. Typo. What is the Fig.2.o in the line 193?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PcChykRufL", "reviewer": "Reviewer_Mtwu", "summary": "This paper aims to enhance CLIP's spatial awareness. That is to say, increases the quality of dense features extracted by CLIP. It proposes the Spatial Correlation Distillation (SCD) framework, which preserves CLIP's inherent spatial structure and mitigates degradation for spatial awareness by Region-Languaeg Alignment. It also introduces a lightweight Refiner that extracts refined correlations directly from CLIP before feeding them into SCD, based on an intriguing finding that CLIP naturally captures high-quality dense features.", "questions": "1. It might be better to use \"language\" or \"text\" instead of \"linguistic\"\n\n2. How come methods specifically designed for CLIP dense prediction like CLIPSelf and RegionText work even worse than vanilla CLIP?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This pager reveals the potential problems of previous works, that \"the RLA process projects dense visual embeddings into a text-oriented domain, making them incompatible with visual-centric objectives\". To tackle this, the paper proposes to conduct Spatial-Correlation-guided Region-Language Alignment with Refiner to preserve spatial awareness. This design is novel and reasonable.\n\n2. The performance improvement is significant compared with baseline methods.\n\n3. The experiments and analysis are comprehensive.", "weaknesses": "1. The paper writing is not so rigorous. Authors should give clear definitions of each term they are discussing. Like what is the definition of \"spatial awareness\", what it means by a better dense feature, what is \"visual-centric\", what is \"intra-feature structural relationships\", etc.\n\nAs an example, spatial awareness here is (probably) defined as performance for tasks like localization and recognition, which I think, is equivalent to increasing the quality of dense features extracted by CLIP, which means different parts in the image with different semantics should be extracted with the features that are distinguishable from each other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)