Refining CLIP's Spatial Awareness: A Visual-Centric Perspective

Authors: Congpei Qiu, Yanhao Wu, Wei Ke, Xiuxiu Bai, Tong Zhang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-distillation; CLIP; Open-vocabulary dense prediction

Abstract:
Contrastive Language-Image Pre-training (CLIP) excels in global alignment with language but exhibits limited sensitivity to spatial information, leading to strong performance in zero-shot classification tasks but underperformance in tasks requiring precise spatial understanding. Recent approaches have introduced Region-Language Alignment (RLA) to enhance CLIP's performance in dense multimodal tasks by aligning regional visual representations with corresponding text inputs. However, we find that CLIP ViTs fine-tuned with RLA suffer from notable loss in spatial awareness, which is crucial for dense prediction tasks. To address this, we propose the Spatial Correlation Distillation (SCD) framework, which preserves CLIP's inherent spatial structure and mitigates above degradation. To further enhance spatial correlations, we introduce a lightweight Refiner that extracts refined correlations directly from CLIP before feeding them into SCD, based on an intriguring finding that CLIP naturally capture high-quality dense features. Together, these components form a robust distillation framework that enables CLIP ViTs to integrate both visual-language and visual-centric improvements, achieving state-of-the-art results across various open-vocabulary dense prediction benchmarks.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730400904762

thread_type:
official_review

<thread_object>
creator: reviewer_mtwu

<content>
summary:
This paper aims to enhance CLIP's spatial awareness. That is to say, increases the quality of dense features extracted by CLIP. It proposes the Spatial Correlation Distillation (SCD) framework, which preserves CLIP's inherent spatial structure and mitigates degradation for spatial awareness by Region-Languaeg Alignment. It also introduces a lightweight Refiner that extracts refined correlations directly from CLIP before feeding them into SCD, based on an intriguing finding that CLIP naturally captures high-quality dense features.

strengths:
1. This pager reveals the potential problems of previous works, that "the RLA process projects dense visual embeddings into a text-oriented domain, making them incompatible with visual-centric objectives". To tackle this, the paper proposes to conduct Spatial-Correlation-guided Region-Language Alignment with Refiner to preserve spatial awareness. This design is novel and reasonable.

2. The performance improvement is significant compared with baseline methods.

3. The experiments and analysis are comprehensive.

weaknesses:
1. The paper writing is not so rigorous. Authors should give clear definitions of each term they are discussing. Like what is the definition of "spatial awareness", what it means by a better dense feature, what is "visual-centric", what is "intra-feature structural relationships", etc.

As an example, spatial awareness here is (probably) defined as performance for tasks like localization and recognition, which I think, is equivalent to increasing the quality of dense features extracted by CLIP, which means different parts in the image with different semantics should be extracted with the features that are distinguishable from each other.

questions:
1. It might be better to use "language" or "text" instead of "linguistic"

2. How come methods specifically designed for CLIP dense prediction like CLIPSelf and RegionText work even worse than vanilla CLIP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730520002671

thread_type:
official_review

<thread_object>
creator: reviewer_ssgt

<content>
summary:
This paper proposes a framework called Spatial Correlation Distillation(SCD) to refine the spatial awareness of CLIP. To recall the visual perception of vanilla CLIP loss in Region-Language Alignment (RLA) training, it proposes a Spatial Correlation Distillation training process and a light-weight refiner to distill the visual relationship from the vanilla CLIP. SCD can improve the existing OV methods and achieve the new sota on the ov tasks.

strengths:
1. Experiments. The experiments are sufficent and solied to prove the improvement of framework.

2. Significance. The framework is a convience, plug and play, and effecitive pipeline to improve the existing methods of the OV tasks.

3. Motivation. The motivation is clear. The Fig.1 and intro explain the loss of dense prediction ability of RLA process.

weaknesses:
1. Novelty. The frame work is composed with SCD and a refiner and the novelty of the SCD is limited. The SCD is a common distillation module to reserve the similarity between the RoI features of student and teacher models. 

2. Lack of analysis. This paper does not provide a deeper analysis of the experimental results. For example, why R-SC can effectively improve the classification accuracy in Tab.1? In terms of motivation and structure, this module improves spatial perception and does not have much gain in classified task with the annotated masks..

3. Typo. What is the Fig.2.o in the line 193?

questions:
1. In the SC-RLA, how to ensure the correspondence between the RoI regions from teacher and student models? The student model would be trained, so the number, order and attribute of the RoI regions would become different with the frozen image encoder.

2. In the line 242 -258, would the contents of embeded sampled images {X_i} affect the SPATIAL AWARENESS? Could you provide some quantitative analysis like the cosine similarity between the {X_i} and X_t ?

3. Since the motivation and module is to improve spatial awareness (which can also be seen from the visualization), are there more segmentation related visualizations? Qualitative results using segmentation would be more convincing (e.g. finer edges)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721167412

thread_type:
official_review

<thread_object>
creator: reviewer_fr8o

<content>
summary:
This paper aims to improve upon the Region-Language Alignment (RLA) approaches for Contrastive Language-Image Pre-training (CLIP) models. In order to not only promote the linguistic alignment but also preserve the spatial awareness, Spatial Correlation Distillation (SCD) is proposed to plug into the existing methods such as RegionCLIP and CLIPSelf. Refiner is also introduced to enhance the regional representation of teacher model. The experiments on the open-vocabulary dense prediction tasks demonstrate the effectiveness of the proposed method.

strengths:
Motivated by the observation that RLA methods suffer from notable loss in spatial awareness for CLIP ViTs, SCD is specifically designed to capture the spatial structure in a region. The widespread experiments show the superior performance of the proposed method across multiple open-vocabulary dense prediction benchmarks.

weaknesses:
The motivation to design Refiner has been implied in CLIPSelf. When K = N, the proposed Refiner is almost the same as CLIPSelf, which reduces the technical novelty. Also, the ablation study of K is absent in this work.

The application of the proposed method is restricted. Spatial Correlation Distillation (SCD) is auxiliary and is to preserve the spatial awareness when another optimization is applied (e.g. RLA). Therefore, it seems that SCD cannot be applied independently since in this case the weights of teacher model and student model are always equal. Besides, R-SC-V is only learned from the teacher model that is optimized by Refiner (similar to CLIPSelf), so it cannot be further applied to CLIPSelf based approach.

In order to showcase the generalizability and scalability of the proposed method, the experiments with data scaling up are expected to be provided, which is missing in the current version.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732035537660

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank all the reviewers for their thoughtful and constructive feedback, which has been both encouraging and insightful. We are pleased to see that the reviewers appreciate the following aspects of our work:

- The effectiveness of our designs, demonstrating significant performance improvements (FR8o, SSGt, Mtwu)
- The solid and comprehensive nature of our experiments (SSGt, Mtwu)
- The novelty and soundness of our design, supported by thorough analysis (Mtwu)
- The framework’s convenience and plug-and-play capability (SSGt)

We have carefully considered the reviewers' comments and revised our manuscript accordingly. The updated content is highlighted in **magenta** for clarity. Below is a brief outline of the revisions:

- In response to Mtwu, we have provided clearer explanations of the crucial concepts in our paper
- Following MaskCLIP [1], we have added a new off-the-shelf zero-shot segmentation experiment in Sec. 4.3, including additional VLMs such as Meta-CLIP [2] and DFN [3], to demonstrate the generalizability of our method. We have also included corresponding visualizations following SSGt’s suggestion.
- At SSGt's suggestion, we have conducted further empirical analysis in Appendix C.3 to better interpret the positive effects of our model.

**Reference**

[1] Zhou et al. "Extract free dense labels from clip."  ECCV 2022.

[2] Fang et al. "Data filtering networks." ICLR 2024.

[3] Xu et al. "Demystifying clip data." ICLR 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents the Spatial Correlation Distillation (SCD) framework, aimed at enhancing CLIP’s spatial awareness to overcome its limitations in dense prediction tasks. The authors propose the addition of a lightweight Refiner module, which extracts and refines spatial correlations from CLIP’s inherent dense features, boosting performance in tasks requiring spatial precision, such as segmentation and object detection.

AC concurred with the reviewers on the significant contribution of improving CLIP’s spatial awareness and commended the authors’ efforts in addressing concerns, particularly in clarifying the technical methodology and enhancing the quality of the presentation. Reviewer Mtwu highlighted the lack of clear definitions for key terms in the paper and suggested that the authors provide more detailed explanations to further improve readability. AC encourages the authors to release the code to support the community and enhance reproducibility.