Diffusion Feedback Helps CLIP See Better
Authors: Wenxuan Wang
Quan Sun
Fan Zhang
Yepeng Tang
Jing Liu
Xinlong Wang
Venue: ICLR 2025 Poster (2025)
Keywords: CLIP Model
Diffusion Model
Generative Feedback
Representation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: In this work, we present a simple post-training approach for CLIP models, which largely overcomes its visual shortcomings via a self-supervised diffusion process.

Abstract: Contrastive Language-Image Pre-training (CLIP), which excels at abstracting open-world representations across domains and modalities, has become a foundation for a variety of vision and multimodal tasks. However, recent studies reveal that CLIP has severe visual shortcomings, such as which can hardly distinguish orientation, quantity, color, structure, etc. These visual shortcomings also limit the perception capabilities of multimodal large language models (MLLMs) built on CLIP. The main reason could be that the image-text pairs used to train CLIP are inherently biased, due to the lack of the distinctiveness of the text and the diversity of images. In this work, we present a simple post-training approach for CLIP models, which largely overcomes its visual shortcomings via a self-supervised diffusion process. We introduce DIVA, which uses the DIffusion model as a Visual Assistant for CLIP. Specifically, DIVA leverages generative feedback from text-to-image diffusion models to optimize CLIP representations, with only images (without corresponding text). We demonstrate that DIVA improves CLIP's performance on the challenging MMVP-VLM benchmark which assesses fine-grained visual abilities to a large extent (e.g., 3-7%), and enhances the performance of MLLMs and vision models on multimodal understanding and segmentation tasks. Extensive evaluation on 29 image classification and retrieval benchmarks confirms that our framework preserves CLIP's strong zero-shot capabilities. The code is publicly available at https://github.com/baaivision/DIVA.

Published as a conference paper at ICLR 2025
DIFFUSION FEEDBACK HELPS CLIP SEE BETTER
Wenxuan Wang!23. Quan Sun Fan Zhang' Yepeng Tang* Jing Liu Xinlong Wang"1
Institute of Automation, Chinese Academy of Sciences
2 School of Artificial Intelligence, University of Chinese Academy of Sciences
3 Beijing Academy of Artificial Intelligence
Institute of Information Science, Beijing Jiaotong University
Project Page: https://rubbce. wtw.:ics-xxaaniitu...be...
ABSTRACT
Contrastive Language-Inan Pre-training (CLIP). which excels at abstracting open- world representations across domains and modalities, has become a foundation for a variety of vision and multimodal tasks. However, recent studies reveal that CLIP has severe visual shortcomings, such as which can hardly distinguish orientation, quantity, color, structure, etc. These visual shortcomings also limit the perception capabilities of multimodal large language models (MLLMs) built on CLIP. The main reason could be that the image-text pairs used to train CLIP are inherently biased, due to the lack of the distinctiveness of the text and the diversity of images. In this work, we present a simple post-training approach for CLJP models, which largely overcomes its visual shortcomings via self-supervised diffusion process. We introduce DIVA, which uses the Diffusion model as a Visual Assistant for CLIP. Specifically, DIVA leverages generative feedback from text-to-image diffusion models to optimize CLIP representations, with only images (without corresponding text). We demonstrate that DIVA improves CLIP's performance on the challenging MMVP-VLM benchmark which assesses fine- grained visual abilities to large extent (e.g-, 3-7% 1), and enhances the performance of MI.LMs and vision models on multimodal understanding and segmentation tasks. Extensive evaluation on 29 image classification and retrieval benchmarks confirms that our framework preserves CLIP's strong zero-shot capabilities. The code is publicly available at https: / thub. www/ www.bbaaiiiiiiiooo vision/DIVA.
INTRODUCTION
Contrastive language-image pre-training (CLIP) (Radford et al., 2021b) has been widely applied to various multimodal understanding and generation tasks, including open-domain image classification (Sun et al., 2024d: Zhang et al., 2022; Zhu et al., 2023), text-to-imant Baldrati et al., 2022; Sain et al., 2023), visual grounding (Wang et al., 2022; Yu et al., 2023; Wang al., 2024b;a), and text-to-image generation (Frans al., 2022; Bar-Tal al., 2022; Rombach et al., 2022a; Crowson et al., 2022; Ramesh et al., 2022; Vinker et al., 2022). This widespread application is due to CLIP's excellent visual representation ability, learned from large-scale data. Thus, enhancing CLIP's ' representation and capabilities is crucial for advancing downstream tasks.
retrieval (Luo et al., 2023;
Since the introduction of CLIP (Radford al., 2021b), numerous subsequent studies on CLIP models have emerged in recent years. These studies have utilized training techniques such as pre-training (Sun al., 2023; 2024b; Fang et al.. 2023; Xu et al., 2023a; Zhai et al., 2023; Shi al., 2024) and fine-tuning (Wei et al., 2023b: Zhang et al., 2024) CLIP models, achieving improved performance and unlocking new abilities. However, these approaches still suffer from unavoidable limitations, as they heavily rely on image-text data pairs and cannot work on image-only data.
As noted by recent works (Kim al., 2023; Zeng et al., 2021; Zhang et al., 2024; Tong et al., 2024b;a), despite its excellent zero-shot performance, CLIP suffers from certain perceptual understanding limitations due to the contrastive learning paradigm and the noisy image-text pairs used in training.
Equul contribution. Correspondence wangxinlong @ baai.ac.cn.

---

Published as a conference paper at ICLR 2025
OpenAl CLIP
MetaCLIP
fie taclid
? ?i?
15.9
25.2
23.7
The away
V11-8-204
VIL-L-356
VIE-L-224
VIE-10:244
Diffosion Feedback
Beseline
DFN
the school her parked fazing Eron cumuera
37.8 36.5
31.8
31,8
The school Bea is palked facing
Figure I: Left:
The existing CLIP models mostly suffer from the inability to distinguish visual details. After enhancing the visual capabilities with our DIVA, the sensitivity of CLIP to visual details has greatly improved. Right: Our proposed DIVA consistently boosts the performance of various CLIP models (Radford et al., 2021b; Fang et al., 2023; Xu et al., 2023a; Zhai et al., 2023) on MMVP-VLM benchmark that evaluates the visual capabilities of vision-language models.
These limitations include an inability to accurately comprehend long texts and to perceive fine-grained differences similar images. While some studies have attempted to address the text comprehension issue (Zhang et al., 2024), research on improving CLIP's fine-graines visual perception remains underexplored. As studied in (Kim et al., 2023), the perceptual shortcomings of CLIP are closely tied to its visual encoder's 's inability to grasp fine-graine visual details, which is crucial for multimodal models. The lack of this capability in CLIP directly affects the performance of vision and multimodal models that CLIP a vision encoder (Tong et al., 2024b;a). As demonstrated in Fig. 1, existing CLIP models mostly struggle to distinguish subtle visual differences between similar images, highlighting the fundamental limitation in aineggriiedd visual perception. To mitigate this visual perception shortcoming of CLIP, lots of recent works such as (Tong et al., 2024b; Kar et al., 2025; Zong et al., 2024) have attempted to improve multimodal large language models (MLLMs) by incorporating extra vision-only self-supervised backbones, such as DINOv2 (Caron et al., 2021), 1), to enhance visual perception capability. However, these approaches still do not fundamentally address CLIP's visual perception limitations, and often introduce significant computational costs.
Thus, in this work, we aim to fundamentally address CLIP's inability to distinguish fine-grained visual details via SSL paradigm. Considering that diffusion models can naturally generate detailed and realistic images, text-to-image generation inherently requires the diffusion models to grasp fine-grained visual representations (Wei et al., 2023a). Motivated by this, we propose utilizing the generative feedback from diffusion models to enhance CLIP's s visual perception capabilities, which is solely driven by image data with manageable training cost. By conditioning diffusion models with CLIP's densely recapped visual features and applying reconstruction loss for CLIP's representation optimization, we leverage the DIffusion model as Visual Assistant for CLIP, thus the name of our approach, DIVA. Our results highlight that DIVA greatly enhances CLIP's performance on MMVP-VLM benchmark measuring visual abilities of vision-language (V-L) models, and improves MLLMs and vision models multimodal and vision understanding tasks. Besides, DIVA maintains CLIP's s excellent zero-shot performance on 29 image classification and retrieval benchmarks. The key advantage of DIVA lies in its ability to enhance CLIP's visual perception capabilities through resource-efficient post-training process, using only image data in self-supervised manner, without the need for expensive image-text pair data collection introducing additional visual encoders.
Our main contributions can be summarized as follows:
Concentrating on overcoming CLIP's visual shortcomings in perceiving fine- ine-ggained details, present the first work to exploit the potential of leveraging generative feedback from extttto-mmage diffusion models to optimize CLIP model's discriminative representations.
We propose a simple self-supervised framework DIVA for CLIP's representation optimization. Coupled with our visual dense recap scheme, DIVA conditions diffusion models with dense visual features from CLIP and incorporates image reconstruction loss for optimization.
Our DIVA greatly boosts CLIP's visual perception capability and improves its performance on MMVP-VI VLM benchmark, further enhancing MI.MMs and vision models on multimodal and visual understanding tasks. Meanwhile, our results on 29 image classification and retrieval benchmarks show that DIVA maintains CLIP's s original excellent zero-shot performance.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
CLIP Models & MILMs. The introduction of CLIP (Radford et al., 2021b) has significantly advanced multimodal learning. Since its debut, a series of CLIP models have emerged (Sun et al., 2023; Fang et al., 2023; Xu et al., 2023a; Zhai al., 2023), enhancing performance and unlocking new capabilities through improved pre-training techniques and model architectures. On this basis, CLIP has been widely adopted as a foundation model, serving as a backbone for various applications such image segmentation (Li al., 2022a; Xu al., 2022; Shan al., 2024; Xu et al., 2023e; Liang et al., 2023; Zhou et al., 2023), object detection (Gu et al., 2021; Li et al., 2022b; Subramanian et al., 2022) and video understanding (Bose et al., 2023; Lin al., 2022; Castro & Heilbron, 2022; Xu et al., 2021; Rasheed et al., 2023; Tang et al., 2024). Its ability to align language and vision has led to superior results these tasks compared to traditional methods. Moreover, CLIP has driven the development of MLLMs (Liu et al., 2024b;a; Sun et al., 2024c;a). Combining strong visual understanding with advanced large language models facilitates more sophisticated interactions between vision and language. Recent works have highlighted inherent visual flaws in the CLIP models and MI.LMMs using CL.PP as the visual encoder (Tong et al., 2024b;a). To address this, some research has incorporated multiple vision encoders to achieve more precise and comprehensive visual perception (Kar et al., 2024; Jiang et al., 2023; Tong et al., 2024b). However, this approach increases computational costs and memory usage. There has been research directly enhancing CLIP's visual perception capabilities to better serve MLLMs. Thus, the main focus of our work is to fundamentally overcome CLIP's visual perception shortcomings, directly benefiting both vision models and multimodal MLLMs that use CLIP as a backbone.
Diffusion Models for Representation Learning. Diffusion models (Ho al., 2020; Song al., 2020) have made remarkable progress in various generative tasks, such as image generation (Rombach al., 2022b; Saharia et al., 2022; Betker al., 2023; Zheng et al., 2024), video generation (Singer et al., 2022; Blattmann et al., 2023; Junhao Zhang et al., 2023; Ho et al., 2022), editing (Meng et al., 2021; Hu et al., 2024; Mou al., 2023), etc. Apart from research above, there are also many works focus on employing diffusion models for representation learning. Some of works leverage the intermediate activation of pre-trained diffusion models for different downstream tasks, including classification (Xiang et al., 2023), semantic segmentation (Baranchuk et al., 2021), panoptic segmentation (Xu et al., 2023b), depth estimation (Zhao et al., 2023), etc. Other works (Hudson et al., 2024; Pan et al., 2023) train their own diffusion models coupled with meticulously devised modules to further boosting the representation capabilities. Besides, Diffusion- TTA (Prabhudesai al., 2023) aims adapt pre-trained vision encoders samples in testing set using feedback from diffusion model. Additionally, some methods (Guo et al., 2024; Trabucco et al., 2023; Tian et al., 2024; Azizi et al., 2023) utilize diffusion models to generate synthetic data, which is then adopted to enhance the representation capabilities of corresponding models. In contrast, in our work, we mainly focus on exploring the potential of enhancing the original discriminative representations of CLIP models by directly utilizing generative feedback from the diffusion models. Additionally, we aim to leverage the diffusion models to break free from the constraints paired image-text data and construct a self-supervised framework to improve CLIP's visual perception capabilities.
3 ENHANCING CLIP'S REPRESENTATIONS VIA DIFFUSION FEEDBACK
In this section, we present our DIVA, an effective framework for boosting CLIP's visual perception capabilities with pre-trained conditional diffuusinn model. We first discuss CLIP's ' visual deficiencies
in perceiving details and generative diffusion models as preliminaries in Sec. 3. 1 and Sec. 3.1.2 respectively. Then the overall architecture of DIVA is illustrated in Sec. 3.2, followed by our well-designed visual dense recap strategy for better unleashing the power of DIVA in Sec. 3.3.
3.1 PRELIMINARIES
3.1.1 CLIP's VISUAL DEFICIENCIES
Thanks to the excellent representations learned through pre-training on massive data, CLIP (Radford et al., 2021b) exhibits outstanding generalization capabilities and is widely applied in the V-L understanding domain. However, CLIP is not without its flaws. As highlighted in the study (Tong et al., 2024b), CLIP struggles to distinguish detailed differences between two images that are obviously distinct human observers. This deficiency mainly stems from two aspects: Training
3

---

Published as a conference paper at ICLR 2025
The original contrastive learning strategy of CLIP aims to minimize the distance between positive pairs and maximize the distance between negative pairs of visual class tokens and textual semantics, resulting in visual perception bias that mainly focuses high-level semantic information while overlooking visual details such as orientation, quantity, color, and structure. Consequently, CLIP sometimes encodes visually different images into similar embeddings, making it difficult to differentiate these images' subtle variations. 2) Data Format: The text in the image-text pairs used train CLIP limited in length. As pointed out by (Zhang al., 2024), although the length of the text token is restricted to 77, CLIP's actual effective text length is less than 20. Therefore, the textual data in these image-text pairs inherently lacks descriptions of the visual details in the corresponding positive sample images. This fundamental limitation of the training data also leads to CLIP's inability to adequately perceive visual detail information.
Paradigm:
3.1.2 GENERATIVE DIFFUSION MODELS
Generative diffusion models have proven to be highly effective in modeling high-dimensional data and have become the de facto approach for visual generation tasks (Podell et al., 2023; Yang et al., 2024). They possess strong capabilities in capturing intricate visual details. A diffusion model reconstructs an image by gradually removing noise added during the forward process. Formally, given an image sample ng drawn from an underlying probability distribution p(r)). a forward diffusion process defines Markov chain to gradually add random Gaussian noise ét E (O, I) to the original sample Xp:
X --
+ ViG t= 1,,,..,,
Here, T denotes the number of diffusion steps, and 8, E (0. 1) is a predefined time-dependent variance schedule. As the T becomes large enough, XT is close to NO I). The transition equation can be reformulated as follows by leveraging the additive property of Gaussian distribution:
X vaxxo + a1-a,c, t=
in which 0ş 1 3r and ā₂ IT Otj. On this basis, the image sample To can be iteratively generated from random noise XT ~ NN by reversing the forward diffusion process:
1
X
X+
(xe; ()) oee, 1 L=
vI
at
é is a denoising neural network trained to predict in the forward diffusion process and o, is the posterior noise variance. A commonly used training objective for a diffusion model eø is:
L(Φ)= = Ex, [[|c ca (V + v1 - ā,c,t)ll
Besides, the diffusion models can be easily extended to conditional generation by incorporating condition into Eg. in which e can be a class label, text prompt, etc. Thus, the training objective should be modified to:
l(d) = Ex.. [lle E Va vI ā,e, āte,t, c)]l
(5)
In practice, modulation layers or cross- attention layers can be employed to integrate conditions into denoising neural network, guiding the denosing process.
3.2 OVERALL STRUCTURE OF DIVA
As illustrated in Fig. 2, DIVA mainly consists of two parts: the CLIP model to be enhanced in terms of visual perception capabilities, and the pre-trained text-to-image diffusion model providing generative feedback. Taking an original image as input, the CLIP model encodes the corresponding visual features, which will be combined with the empty text's embeddings (i.e., [BOS] & [FOS]) from the diffusion model's 's text encoder for diffusion's 's condition. Given the image with added noise, the diffusion model attempts to predict the noise added from the previous step to the current step with the aforementioned condition. This process needs to be repeated N times because, for each image, we will randomly select N states with a uniform distribution from the total steps (e.g., 1000 steps) of the diffusion model for optimization. The corresponding loss function can be represented as Equation 5. Keeping all parts' weights except the CLIP visual encoder frozen, the training objective is simply to minimize the reconstruction loss (i.e., generative guidance). In this manner, by constraining
4

---

Published as a conference paper at ICLR 2025
CLS Token
Patch Token
Condition
Feedback
Xp
x
Visual Encoder
Denoiser
Noise EG
Figure 2: Overall architecture of our DIVA. Given image To. the CLIP model 0 encodes the visual features as main part of condition c, then the generative diffusion model á predicts the added noise taking the noisy image 2, and condition as input. We optimize the CLIP's representation by maximizing the image likelihood with the diffusion loss via generative feedback.
the diffusion model to more accurately predict the added noise, the CLIP's original semantic-rich discriminative representations will be gradually optimized into representations with more visual details through diffusion feedback. On this basis, results in Sec. 4.4 demonstrate that our DIVA does not greatly damage the zero-shot performance of the original CLIP models. The pseudo code of the specific enhancement process be found at Algorithm 1 in Appendix.
3.3 DIFFUSION'S & VISUAL CONDITION DESIGN
In our DIVA, the diffusion model's condition design is pivotal, it sets the upper limit for enhancing CLIP's visual capabilities. Since our motivation for designing DIVA is to enhance CLIP's ability to capture fine-grained details through resource-efficient, purely image-driven self-supervised framework for post-training. we do not introduce aligned textual information as the diffusion condition. However, bridging the gap between pure visual features and text embeddings for the diffusion process crucial. To address this, we first pass the visual features through a vision- languaee projection layer map the high-level representations. Additionally, as part of the diffusion model's 's condition to further narrow this gap. In this way, our condition design resembles an empty text shell, but the actual content inside is akin to "visual sentence" obtained via vision-to-text mapping. This design more seamlessly fits the diffusion model's conditional input.
incorporate the embedding of empty text
Visual Dense Recap Scheme. Apart from the aforementioned empty text's embeddings from diffusion model's 's text encoder, we introduce simple yet effective strategy called visual dense recap for the designing of visual condition part. Unlike detailed ca-pttionnng of an image's caption in natural language, our approach performs re-captioning at the level of visual richness by incorporating features from local patch tokens along with the class token into the condition. When only the class token present, CLIP's visual features primarily contain strong semantic information, which is insufficient for reconstructing the original image. Consequently, the reconstruction task becomes challenging due to the lack of adequate information, and CLIP cannot learn significantly enhanced representations. By incorporating local patch features, the auxiliary function of the condition is significantly enhanced, allowing the generative feedback to effectively improve CLIP's visual perception capabilities. We conduct ablation studies in Sec. 4.5 to demonstrate the efficacy of visual dense recap.
Visual Recap Density. Although the visual dense recap scheme appears straightforward, the density of the recap is crucial. If the density is too high (i.e., introducing too many local tokens), the richness of the condition information approaches its maximum, greatly reducing the difficulty of reconstruction task. This results in CLIP's representation requiring minimal optimization to easily complete the reconstruction, limiting the upper bound of CLIP's optimized capabilities. Conversely, if the recap density is too low (i.e., retaining only class token or introducing few local tokens), CLIP's optimization process will struggle with the high difficulty of reconstruction, failing to adequately learn the expected detailed visual representations. This intuitive point is confirmed in Sec. 4.5.
In fact, our DIVA can enhance different CLIP models' fine-grained perception abilities through general and simple condition design principle, e.g., introducing all the local patch tokens along with the class token as the visual condition for diffusion model. However, we empirically believe that each baseline's requirement for visual density is actually different, which is also supported by our preliminary experiments. This is because different CLIP models are trained on vastly different data, leading to significant variations in their learned representations, Therefore, their visual density for various CLIP models needs to be specifically adjusted to maximize the learning of corresponding detailed representations through generative feedback. Specifically, ensuring the
5

---

Published as a conference paper at ICLR 2025
visual class token is always present in the condition, we introduce randomly selected local token features with approximately 15% and 30% probabilities for OpenAl CLIP (Radford et al., 2021b) at 224 and 336 resolutions. For the SigLIP ViT-SO-14 (Zhai et al., 2023) at 224 and 384 image sizes, we incorporate local token features obtained through 1D average pooling with local window sizes of 6 and 10, respectively. Except that introducing 50% randomly selected patch tokens into the condition for DFN ViT-H-14/377,, for the remaining baselines (Fang et al., 2023; Xu et al., 2023a), we include all local token features for the condition design. Apart from DFN ViT-H-14/222 and SigLIP (Zhai et al., 2023) only using visual class token, all other models incorporate local features consistent with the training stage conditions during inference, combining them with class token to fully leverage the detailed representations captured by the enhanced CLIP.
EXPERIMENTAL RESULTS
To evaluate the effectiveness of our DIVA and demonstrate its potential to enhance CLIP representa- tions, comprehensive experiments are conducted on multimodal understanding and visual perception tasks, which will be elaborated in the followings.
4.1 IMPLEMENTATION DETAILS
DIVA is trained on NVIDIA-A 80GB GPUs with a batch size of 640. We adopt Stochastic Gradient Descent (SGD) optimizer with a learning rate of le-4 and momentum of 0.9 to refine CLIPs' representations via generative feedback. We only optimize the CLIP models with relatively high-quality Conceptual-3M dataset (Sharma et al., 2018) for 4600 steps (i.e., nearly 1 epoch) during training, which can already boost CLIP's performance in a training-efficien all experiments, we adjust the parameters of the discriminative CLIP vision encoders and keep the pre- pre-trained diffusion models frozen through the training process. Besides, the specific dataset and evaluation details about the MMVP and MMVP- ALMVP-VLM benchmark can be found in Appendix.
manner. For
Regarding the diffusion sampling step N. our choice is made by balancing training cost and model performance gains. Specifically, we begin with the initial state of N=1 to enhance the representation quality of CLIP vision encoder. As N increases, the training cost rises significantly. When increasing N from 1 to 2 (meaning that each image undergo diffusion sampling twice provide two rounds of generative feedback for CLIP model optimization), performance gains are observed. However, further increasing N beyond not only greatly escalate training costs but also do not yield additional benefits for the CLIP model's 's representation learning. Therefore, N=2 is selected as the optimal sampling step to consistently improve performance across various baselines.
4.2 FINE-GRAINED VISUAL PERCEPTION EVALUATION
To validate that our DIVA can effectively mitigate the inherent visual capability deficiencies of CLIP models, we first conduct experiment on various existing CLIP models (Radford et al., 2021b; Fang et al., 2023; Xu et al., 2023a; Zhai et al., 2023). Despite the variations in image resolution, model size, training data and methodology among these CLIP models, our method consistently enhances their performances on the VLM benchmark. As presented in Table 1, framework achieves the best performance improvements (i.e., 14-7%) on OpenAI ViT-L-14 14 and MetaCLIP ViT-H-14, and even current perstreeff DEN ViT-H- 14 our framework realizes a performance gain of nearly 3-5%. This fully demonstrates that DIVA is both general and effective in enhancing the fine-grained visual perception capabilities of CLIP models. Notably, through the generative guidance provided by our self-supervised framework that is free from image-text constraints, the perceptual abilities of CLIP models on almost all visual patterns have the potential to be enhanced.
4.3 BACKBONE ENHANCEMENT PERFORMANCE EVALUATION
Next, with the help of our DIVA, we further evaluate the performance gains brought by the enhanced CLIP backbones for multimodal understanding and visual perception tasks.
Enhanced Vision Backbone for MILMs. Firstly, we adopt LLa (Liu et al., 2024a) as the baseline framework to explore the potential of improved visual encoders in MLLM. LLaVA employs a pre-trained CLIP vision encoder and trains a projector to semantically align visual tokens with textual tokens from large language model (LLM). To ensure fair comparisons, the same setting in LLaVA and evaluate model performance on various multimodal understanding benchmarks (i.e., MMVP (Tong et al., 2024b), POPE (Li et al., 2023), MME- MME--Prrceptio (Fu et al.,
train model with

---

Published as a conference paper at ICLR 2025
Table 1: Performance of CLIP based models on various visual patterns MMVP-VLM benchmark. Our framework greatly overcomes CLIP's original shortcomings in terms of perceiving visual details. Symbols for visual patterns as (Tong al., 2024b) are inherited:
Orientation and Direction, Q: Presence of Specific Features, O State and Condition, f:: Quantity and Count,
I: Positional and Relational Context, @ Color and Appearance, t Structural and Physical Characteristics, A: Texts, O Viewpoint and Perspective.
Method
Ours: Image Size Params (M).
A
o
Average
OpenAl VIT-L-14
224"
427.6
13.3
13.3
20.0
20.0
13.3
53.3
20.0
6.7
13.3
19.3
OpenAl VIT-L-14
224
427.6
[3.3
20.0
20.0
53.3
46.7
20.0
13.3
25,9 (6.00)
OpenAl V1T-L-14
336²
427.9
0.0
20.0
40.0
20.0
6.7 20.0
33.3
6.7
33.3
20.0
OpenAl V1T-L.-14
V
336²
427.9
!26.7 20.0 33.3 13.3
13.3 46.7
26.7
6.7
40.0
(15.)
MetaCLIP 14
224"
427.6
13.3
6.7 66.7 6.7 33.3 46.7 20.0
6.7 13.3
23.7
MetaCLIP ViT-L-14
V
224
427.6
6.7 6.7 60.0 0.0 26.7 66.7 20.0
20.0
40.0
27.4 (337)
MetaCLIP ViT-H-114
224
986.1
6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3
25.2
MetaCLIP ViT-HI-14
V
224
986.1
13.33 20.0 53.3 33.3 13.3 66.7 33.3 13.3 40.0 31.9 (1677
SigLIP ViT-SO-14
224
877.4
(26.7 20.0 53.3 40.0 20.0 66.7 40.0 20.0 53.3
37.8
SigLIP VIT-SO-14
224
877.4
113.3 26.7 60.0 46.7 13.3 73.3 53.3 26.7 53.3 40.7
SigLIP ViT-SO-14
384
878.0
220.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3
37.0
SigLIP VIT-SO-14
384
878.0
26.77 33.3 53.3 26.7 13.3 80.0 40.0 26.7 46.7
(8.5
DEN VIT-H-14
224"
986.1
(20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3
39.3
DEN VIT-H-14
224
986.1
20.0 20.0 80.0 40.0 46.7 66.7 46.7 20.0 53.3
43.7 (14.4)
DEN ViT-H-14
3782
986.7
13.3 20.0 53.3 33.3 26.7 66.7 40.0
20.0
40.0
34.8
DFN VIT-H-14
378
986.7
26.7 26.7 53.3 33.3 26.7 73.3 26.7 13.3 60.0
37.8 (+3.0)
people standing
tloes bread
repetablle splitee?
Yos.
No.
The lock locledd. os attached "M"
people running
the leck beked unbscied?
rubbic in middle
letter "Q"
Is the end of clephant's musk pointing Iamant
The and clepbasats polsting outward.
The the etrphant's mrakk pointing inward.
a rabblz on the left the eaps
letter "X" in
Are the buttorthy wingss dlieer to bring open or
The bntterllyy' wiags aee le cloned pootiion
The fires wiag: clisr to being open, it deserrbllll an butturtty with in wings ogea".
Baselime
Figure 3: Qualitative analysis on MMVP-VLM and MMVP benchmark. Left: The prediction results from the OpenAl ViT-L-14 CLIP before after incorporating DIVA. Right: The prediction results from LLaVA- 7B before & after using our DIVA. The results on both benchmarks show that our framework can greatly enhance CLIP models' fine- -he-grained visual perception capability and effectively alleviate the hallucination problem.
2023), MMBench (Liu et al., 2023), MMBench- CN (Liu et al., 2023), (Liu et al., 2024b)). It can be clearly from Table 2 that LLaVA's performance is greatly boosted by replacing the original CLIP vision encoder to ours. The big accuracy gains on these benchmarks (except MME) are all thanks to the significant enhancement in CLIP's visual perception capabilities brought by our DIVA paradigm utilizing generative feedback.
Wild
Table 2: Performance gains achieved by our enhanced CLIP visual backbone for MILM (i.e., LLaVA-1.5-7B and LLaVA-L.5-13BB) on various V-L understanding tasks. By refining the CLIP's representation with generative feedback, our method mitigates the visual deficiencies in MLLMS (i.e., LLaVA and improves original instruction following ability.
Method
Ouurs
LLM
Image
MMVP
'rand pop adv MME MMBench en
POPE
LLLVAA -ild
Size
en
LLaVA
Vicuna-7B
336"
24.7
87.3 86.1 84.2 1510.7 64.3 58.3"
65.4
LLaVA 1.5
Vicuna-7B 7B
336"
31.3
887.9 87.0 84.6, 1500.6, 66.4
60.6
66.3
LLaVA 5
Vicuna- 13cuna-13B
336"
30.7
(87.1
86.2 84.5 1531.3167.7
63.6
72.5
LLaVA 1.5
Vicuna-13B
3362
35.3
188.1
87.4 84.8 1522.9169.4
63.1
73.5

---

Published as a conference paper at ICLR 2025
Table 3: Performance gains achieved by our enhanced CLIP backbone with generative guidance on semantic segmentation task. denotes the re-implemented results. Boosting CLIP's ability to perceive fine-ge graiige visual details results in considerable benefits for visual dense prediction task.
Method
Backbone
ADE20K AADE220K-S Pascal Context-459, Pascal Context-59
SAN*
14/224
10.9
29.2
14.2
55.8
SAN
ViT-L-14/224
11.0
30.2
15.4
56.7
SAN*
ViT-L-14/336.
11.5
30.3
14.7
56.7
SAN
VIT-L--14/36-
V
11.5
31,8
15.7
57.8
Enhanced Vision Backbone for Fine-Grained Visual Perception. We also include segmentation task to evaluate the benefits brought by our enhanced CLIP backbones for visual dense prediction task. We adopt the recent state-of-the-art model in the open vocabulary semantic segmentation field, SAN (Xu et al., 2023c) with CLIP (Radford et al., 2021b) at both 224 and 336 image resolutions, as baselines. Four commonly used benchmarks (i.e., A-DE0K-847/156 (Zhou et al., 2017) and Pascal C5n/ext-459/59 (Mottaghi et al., 2014) are employed for performance evaluation. As shown in Table 3, with the benefit of our generative tuned CLIP backbone, the baseline models achieve considerable performance improvements on most of segmentation benchmarks and do not suffer performance degradation on the remaining one (i.e., ADE20K-847).
4.4 GENERALIZATION CAPABILITY EVALUATION
Table 4: Summary of zero-shot image classification performance on 27 datasets for evaluating model generalization capability. O-1 and M-1 separately represent OpenAl ViT-L--14/224 and MetaCLIP VIT-H-14/224, DIVA greatly improves CLIP's ability to perceive visual details, while preserving its outstanding generalization capabilities.
Dur ImugeNer-iK
Image V2
ImageNer-Ade
ImugeNet-Rewt
CIFAR-100 100
FDVC Aireraft
Country 211
Stanford Cars
Flowers: 102
Rendered SST2
avg. top-1 acc.
Method
Imugee
ObjectNer
CIFAR
MNST
Cabteh
SUN397
Birdsaup
Fiunsal
FER2013
Food-110
GTSRB
PCam
Resise45
STL.0
VOC2007
Pets
0-11 175.569.8 89 69. 75.8669888688888888 76.4 67.5 31
77 45
68 8 59.4599.777.4444
0-1 0-1 59. 69. 195
76,3 7
86.8
67.5 8 31.8
77.9
51.8
55. 60.2 900.448888999.00000...
93, 07. 64.5 99.3 77.03777.000 77.8169.
M-1 77..72..1 69.166666999188..
73.6 86.
89.1 74.1 48.2 34.7 87.2 68.
69.8
55.6
80.7
71.
M 121.114.771.998 69..9666999111166.99. 7998 86.481 89.1 74.3 47.0 34.7 67.2 55.0
74.3
80.7
92.44.....2...444 94,1 70.793.099. 77.777777977777777
Table 5: Summary of zero-shot image-to-text and text-to-image retrieval performance on Flickr30K (Young et al., 2014) and COCO (Lin et al., 2014) benchmark datasets for evaluating model generalization capability. DIVA significantly enhances CLIP's visual detail perception ability while maintaining its excellent generalization capabilities.
Image-to-Ters Text Retrieval
Text-to-lmage Retrieval
Method
Ours Image Size
Flickr30K
COCO
Flickr30K
COCO
R@ 1 RQ@RR@RR @@ 1 @ R@SRØ 10 RR@5 R@ @ R@SR@ 10
OpenAl V1T-L-14
224
185.1 97.3 99.0 '56.4 79.4 86.6 165.2 87.3 92.0 36.5 61.0 71.1
OpenAl VIT-L-14 14
224
185.3 97.3 99.0 56.77 79.7
87.0 64.4 86.9 92.0 36.6 61.0 71.3
MetaCLIP ViT-H-14,
224"
18955 98.8 99.7 165.5 85.2 91.1 76..8 93.9 96.6 148.2 72.3 81.1
MetaCLIP VIT-H- H-THH-14 14:
224
189.2 98.7 99.7 165.5 85.0 91.1 (77.3 93.8 96.7 148.4 72.4 81.1
After validation of DIVA'S 's ability boost CLIP models' fine-grained visual perception abilities, we conduct thorough assessment of CLIP model's 's original generalization ability. The details about all benchmarks can be found Table 9 in Appendix. Specifically, OpenAl ViT-L-14 (Radford et al., 2021b) and MetaCLIP ViT-H- 14 (Xu et al., 2023a), which are widely used and shows the greatest performance gains on MMVP-VLM benchmark, are adopted as our baselines. We present their zero-shot accuracies on 27 image classification benchmarks in Table 4. It is evident that our DIVA significantly enhances CLIP models' representations of fine-grained visual details without adversely affecting the generalization capabilities of the baselines to large extent. Furthermore. Table 5 illustrates the comparison of zero-shot rext-to-image and image-to-texi retrieval performance before and after incorporating DIVA. The quantitative results reaffirm that optimizing CLIP models'
8

---

Published as a conference paper at ICLR 2025
representations with DIVA preserves the original great generalization ability. Given that these tasks heavily rely on the CLIP visual backbone's 's global semantic understanding, it is reasonable that our generative guided CLIPs do not achieve much performance improvements these tasks.
4.5 ABLATION STUDY
By taking OpenAl ViT-L-14/224 as the baseline model, we conduct comprehensive ablation studies on MMVP-VLM regarding each pattern and average score, exploring the effect of condition designing, introduced data scale and adopted diffusion models for DIVA.
Table 6: Ablation study on the condition design for diffusion models. G and L denote visual class token and patch tokens. Compared to using semantic matching constraints from image-text pairs as condition, taking visual features as condition for representation-Iewe optimization is more effective. Furthermore, an appropriate degree of visual dense recap scheme is also crucial for DIVA.
Visual Condition
Textual Condition
t
A
Average
13.3 13.3 20.0
20.0
13.3
53.3
20.0
6.7
13.3 19.3
Real Caption
6.7 13.3 26.7 20.0
6.7
53.3
33.3
13.3
20.0
(422)
x
Free-Source Phrase
6.7
13.3 20.0 20.0
6.7
53.3
26.7 13.3
26.7,,0.7...
only G
Empty Caption
120.0 20.0 20.0 20.0
13.3
46.7 26.7 20.0
13.3 (+2.)
G part L
Empty Caption
'1333 20.0 40.0 6.7
20.0
53.3 46.7 20.0 13.3 (+6.0)
G L
Empty Caption
6.7 20.0 40.0 6.7 6.7 40.0 40.0
6.7 13.3
20.0
only G
Real Caption
13.3 26.7 33.3 13.3 6.7 53.3 26.7
13.3 6.7 2215 (122)
G part L
Real Caption
13..3 13.3 33.3
13.3
13.3
46.7 60.0
13.3 0.0 123.0 (1am)
+ L
Real Caption
113.3 6.7 26.7 13.3 13.3
46.7
26.7
6.7 13.31 (-0.)
Effect of Condition Design for Diffusion Models, We first probe into the influence of diffusion models' condition designing. As elaborated in Sec. 3.3, the condition design of diffusion models is vital as it directly determines the upper limit of CLIP models' enhanced representation quality. As shown in Table 6, we consider two condition settings: 1) using pure text embeddings as condition, which similar to Diffusion-TWe TTA (Prabhudesai et al., 2023) (rows 3-4); 2) incorporating densely recapped visual features and empty text's embeddings as condition (rows 5-7). For these two settings, the visual encoder is kept frozen for setting I, while the text encoder stays frozen for setting 2. Specifically, whether using real caption matched with images from CC-3M dataset or using free- source phrases about image details, guiding CLIP through image-text matching constraints can yield performance gains. However, since this manner does not originate from the representation level, the achieved gain is not significant. In contrast, our condition design introduces appropriately densified visual features, constructing framework that only uses images to achieve self-suuervised optimization of CLIP representations and detaching from image-text form constraints. DIVA helps CLIP achieve the best performance gains by introducing partial visual local features coupled with class tokens condition. Introducing too few many local tokens results in visual density being too low (row 5) or high (row 7), both of which reduce the achieved performance improvement. We also explore whether incorporating text descriptions could further enhance DIVA's potential. However, both our empirical intuitive and quantitative results indicate that the answer is negative. The main reason is that the introduction of text descriptions greatly reduces the difficulty of the reconstruction task, allowing the CLIP visual backbone's 's representations to easily accomplish the reconstruction without needing to optimize towards capturing more detailed representations. Therefore, it directly impairs the representation learning of the CLIP model's 's visual backbone through generative feedback.
Table 7: Ablation study on the data scaling property of our DIVA with different data scales. Training time is measured by # gpusxhours DIVA demonstrates great potential with data scaling properties, where the increase in data volume proportionally enlarges the performance gains.
Data Scale, Training Time.
A
Average
x
x
13.3
13.3
20.0
20.0
13.3
53.3
20.0
6.7
13.3
19.3
25%
16.8
6.7
13.3
20.0
13.3
6.7
46.7
53.3
13.3
13.3
20.7 (61.4)
50%
32.8
13.3
13.3
40.0
13.3
6.7
26.7
53.3
20.0
13.3
22.2 (429)
75%
49.6
6.7
26.7
40.0
13.3
6.7
53.3
53.3
6.7
6.7
23.7 (44.)
100%
66.4
113.3
20.0
40.0
6.7
20.0
53.3
46.7
20.0
13.3
25.9
(+6.6)

---

Published as a conference paper at ICLR 2025
Data Scaling Property. Then, we investigate the potential data scaling property of our DIVA. The corresponding results are presented in Table 7. It is obvious that the CLIP model's performance on MMVP- VLMM benchmark is consistently improved with more training samples. As the ratios of introduced samples continue to rise, there's no sign of diminishing gains in accuracy. suggesting that framework has great potential with continually scaled up training data. Noticeably, by integrating 100% training data, our method greatly boosts CLIP's ' visual perception capability (i.e., approximately 17%) with 66.4 # gpus hours, means if 8 gpus are available, DIVA only need 8.3 hours training time to realize considerable performance gains for CLIP models. Besides, this setting results in an adaptation time of roughly 0.01 seconds per sample, proving the efficacy of our method.
Table 8: Ablation study on the adopted generative diffusion models in our DIVA. Training time is measured by # gpus xhous. Our framework not sensitive to the version of stable diffusion models, consistently brings representation enhancement for CLIP models.
Method
Diffusion Resolution, Training Time,
Q
e
t:
A
Average
x
13.3 13.3 20.0 20.0 13.3 53.3 20.0
6.7 13.3 19.3
DIT-XL/2
1
512*
80.8
20.0 13.3 20.0 6.7 6.7 20.0 20.0 6.7 6.7 13.3 (-6.))
SD-1-4
5122
71.2
,2000 13.3 26.7 20.0 13.3 40.0 33.3 26.7
13.3 (43.7)
SD-2-I-base
512²
66.4
113.3 20.0 40.0 6.7 20.0 53.3 46.7 20.0 13.3/25.9 (16.6)
SDDxx--base-IO0
5122
90.4
120.0 20.0 26.7 26.7 6.7 46.7 33.3 6.7 26.7:23.7 (14.4)
Effeet of Diffusion Model Structures. At last, we explore the effect of incorporating various types of diffusion models for generative guidance. Specifically, two types of diffusion models are employed in our DIVA, including DiT (Peebles & Xie, 2023) and stable diffusion (SD) series (Rombach et al., 2022a). It is clear in Table 8 that our method achieves the biggest performance gain on MMVP- VLM by integrating SD-2-1-base- Furthermore, we observe that integrating DIT-X112 as generative guidance exacerbates the perceptual ability of original CLIP model in capturing visual details. We attribute this to DiT's relatively poor representation quality compared to SD models. For the included SD series, the quantitative results in Table 8 also demonstrate that DIVA is not sensitive version of SD models, which can consistently refine CLIP's feature representations within our framework.
5 CONCLUSION
In this work, we focus addressing the visual limitation of CLIP models that struggle with distinguishing fine-gre iinegrrined image details. We present the first work to explore leveraging generative feedback from text-to-image diffusion models to directly optimize CLIP models' representations. Specifically, by feeding dense visual features from CLIP as condition to the diffusion models and introducing the reconstruction loss from diffusion process onto the CLIP model's optimization, we establish a self-supervised framework DIVA. Notably, this architecture is simple and clean, requiring no additional plugins while demonstrating significant potential. Extensive evaluations demonstrate that our DIVA not only substantially enhances CLIP models' performance on the MMVP-VLM benchmark that measures visual abilities of vision- lsigun-a models, but also aids in improving the performance of MLLMs and vision networks respectively on multimodal and visual understanding tasks. Furthermore, experiments on 29 benchmarks evaluating generalization capabilities confirm that our self-supervised DIVA maintains the CLIP models' original excellent generalization capabilities.
6 LIMITATIONS AND FUTURE TOPICS
One potential limitation of this work is that the data scale for generative fine-tuning and the model capacity of our DIVA could be scaled up further to push better CLIP representations and performance. Moreover, this work mainly focuses on designing a simple but effective framework to enhance CLIP models with generative diffusion process, which means although our DIVA demonstrates the newly exploited potential of using generative diffusion models for better CLIP models' representation guidance, it can be integrated with finer-grained supervision scheme to further boost discriminative model capabilities, Exploring additional modalities beyond image-text data, e.g., video and audio, is also a promising direction for investigation. Furthermore, employing purely visual information when scaling up DIVA framework to further constrain the semantic alignment of CLIP's visual and text encoders is worthwhile avenue for exploration. Since this work is just a beginning in this direction, it opens up a future research perspective to develop a more general and powerful framework based on diffusion models that could enhance vision-language foundation models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Shekoofeh Azizi, Simon Kormblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint 0rXiv:2304.08466, 2023.
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining clip-based features. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 21466-21111 21474, 2022.
Omer Bar- Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European conference on computer vision, pp. 707-723. Springer, 2022.
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126, 2021.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurlPS, 2019.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle Alexander, David Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In CVPR, 2014.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai, com/papers/lalle-33 pdf, 2(3):8, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:2311.11122 2023.
Digbalay Bose, Rajat Hebbar, Krishna Somandepalli. Haoyang Zhang, Yin Cui, Kree Cole- McLaughlin, Huisheng Wang, and Shrikanth Narayanan. Movieclip: Visual scene recognition in movies. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2083-2092, 2023.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-min discriminative compo- nents with random forests. In ECCV, 2014.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Anmand Joulin. Emerging properties self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Santiago Castro and Fabian Caba Heilbron. Fitelip: Refining large-scale pretrained image-text models for shro video understanding tasks. arXiv preprini arXiv:2203. /3371, 2022.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 2017.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTAT, 201
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. In European Conference Computer Vision, pp. 88-105. Springer, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.

---

Published as a conference paper at ICLR 2025
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. "the PAS- CAL Visual Object Classes Challenge 2007 (VOC2007) Results. "http:/wwww.psccal 2007.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprini arXiv:2309. 17425, 2023.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, 2004.
Kevin Frans, Lisa Soros, and Olaf Witkowski. Clipdraw: Exploring text-to-drawing synthesis through tanguage-image encoders. Advances in Neural Information Processing Systems, 35:5207-5218, 2022.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learning: A report on three machine learning contests. In ICONIP, 2013.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921, 2021.
Jiayi Guo, Junhao Zhao, Chunjiang Ge, Chaoqun Du, Zanlin Ni, Shiji Song, Humphrey Shi, and Gao Huang. Everything to the synthetic: Diffusion-driven test-time adaptation via synthetic-domain alignment. arXiv preprint arXiv:2406. 04295, 2024.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ut-of-distiobution generalization. CVPR, 2021a.
Dan Hendryeks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-66 68500,68811111 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300: 2022.
Hexiang Hu, Kelvin CK Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, et al. Instruct-imagen: Image generation with multi- modal instruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4754-4763, 2024.
Drew A Hudson, Daniel Zoran, Mateusz Malinowski, Andrew K Lampinen, Andrew Jaegle, James McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottleneck diffusion models for representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23115-23127, 2024.
Dongsheng Jiang, Yuchen Liu, Songlin Liu, Xiaopeng Zhang, Jin Li, Hongkai Xiong, and Qi Tian. From clip to dino: Visual encoders shout in arXiv:2310.0883 08825, 2023,
large language models. arXiv preprint
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv e-prints, pp. arXiv-2309, 2023.
12

---

Published as a conference paper at ICLR 2025
Oğuzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision- -ainuala models. arXiv preprint arXiv:2404.07204, 2024.
Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Broadening the visual encoding of rision-language models. In European Conference on Computer Vision, pp. 113-132. Springer, 2025.
Tombari. Brave:
Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for open-vocabulary object detection with vision transformers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11144-11154, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 1998.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201 200:22000.3366,, 2022a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Nennn-NNn Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975, 2022b.
Yifan Li. Yifan Du Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object ballucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kumpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard De Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Conference Computer Vision, pp. 388-408 Springer, 2022.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: your multi-modal model an all-around player? 2023.
arXiv preprint ar
Ziyang Luo, Pu Zhao, Can Xu, Xiubo Geng, Tao Shen, Chongyang Tao, Jing Ma, Qingwei Lin, and language-image pre-training for large- scale image-text
Daxin Jiang. Lexlip: sparse retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
11206-11217, 2023.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 13X6v:1306.51:: 2013.
13

---

Published as a conference paper at ICLR 2025
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXhv:2108.01073, 2021.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 891-898, 2014.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation diffusion models. arXiv preprint XXXv:2307.022211 2023.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008
Zixuan Pan, Jianxu Chen, and Yiyu Shi. Masked diffusion as self-supervised representation learner. arXiv preprint M.Xi:2308.006955 2023.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Mihir Prabhudesai, Tsung- Wei Ke, Alexander Cong Li, Deepak Pathak, and Katerina Fragkiadaki. Diffusion-tta: Test-time adaptation of discriminative models via generative feedback. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In ICML, 2021a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 9748-8763. PMLR, 2021b.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6545-6554, 2023.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet?, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic exxteto-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1487, 2022.
14

---

Published as a conference paper at ICLR 2025
Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Clip for all things zero-shot sketch-based image retrieval, fine- fine- grained or not. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2765-2775 2023.
Xiangheng Shan, Dongyue Wu, Guilin Zhu, Yuanjie Shao, Nong Sang, and Changxin Gao. Open- vocabulary semantic segmentation with image embedding balancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 28412-28422 2024.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernyme,, image alt-text dataset for automatic image captioning. In ACL, 2018.
Bowen Shi, Peisen Zhao, Zichen Wang, Yuhang Zhang, Yaoming Wang, Jin Li, Wenrui Dai, Junni Zou, Hongkai Xiong, Qi Tian, et al. Umg-clip: A unified multi-granularity vision generalist for open-world understanding. arXiv preprint arXiv:2401.063397, 2024.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint wrXhv:2209.14792 2022.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. basee generative modeling through stochastic differential equations. arXiv preprint arXhv:2011.13455. 2020.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Bench- marking machine learning algorithms for traffic sign recognition. Neural networks, 2012.
Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot bascline for referring expression comprehension. arXiv preprint arXiv:2204.05999, 2022.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved training techniques for clip at scale. arXiv preprint 23t:::2333.13889 2023.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14398-14409, June 2024a.
Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-18b: Scaling clip to billion parameters. arXiv preprint 2xxx::2402.04252. 2024b.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.005: 052222 2024c.
Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: clip model focusing on wherever you want. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13019-13029, 2024d.
Yepeng Tang, Weining Wang, Chunjie Zhang, Jing Liu, and Yao Zhao. Learnable feature aug- mentation framework for temporal action localization. IEEE Transactions Image Processing, 2024.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36, 2024.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1 1: A fully open, vision-centric exploration of multimodal lmss. arXiv preprint arXiv:2406. 16860, 2024a.
15

---

Published as a conference paper at ICLR 2025
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9568 9578, 2024b.
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmenta- tion with diffusion models. arXiv preprini urXiv22302.07944, 2023.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivari- ant cnns for digital pathology. In MICCAI, 2018.
Yael Vinker, Ehsan Pajouheshgar, Jessica Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-aweer object sketching. ACM Transactions on Graphics (TOG), (11: 2022.
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. NeurlPS, 2019.
Wenxuan Wang, Xingjian He, Yisi Zhang, Longteng Guo, Jiachen Shen, Jiangyun Li, and Jing Liu. Cm-masksd: Cross-modality masked self-distillation for referring image segmentation. IEEE Transactions Multimedia, 2024a.
Wenxuan Wang, Tongtian Yue, Yisi Zhang, Longteng Guo, Xingjian He, Xinlong Wang, and Jing Liu. Unveiling parts beyond objects: Towards aner-granularity referring expression segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
12998-13008, 2024b.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11886-11695, 2022.
Chen Wei, Karttikeya Mangalam, Yao Huang. Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models as masked autoencoders. arXiv preprint arXiv:230.00388. 2023a.
Yixuan Wei, Han Hu, Zhenda Xie, Ze Liu, Zheng Zhang. Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Improving clip fine-tuning performance. In Proceedings of the IEÉF/CVF International Conference Computer Vision, pp. 5439-5449, 2023b.
Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-super learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15802-15812 2023.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint urXiv:2109. 14084, 2021.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes,
Vasu Sharma, Shang-Wer Wen Li, Gargi Ghosh, Luke Zeulemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. arXiv preprint 2rXiv:2309.1660 2023a.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18134-18144 2022.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2955-2966 2023b.
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open- vocabulary semantic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2945-2954, 2023c.
16

---

Published as a conference paper at ICLR 2025
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang. Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint 2400022400000072. 2024.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2014,
Seonghoon Yu, Paul Hongsuck Seo, and Jeany Son. Zero-shot referring image segmentation with global-local context features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19456-19465, 2023.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint 1.X1v:2111.08276,22 2021.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang. and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. arXiv preprint arXiv:2403 15378, 2024.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-aerpter Training-fre free adaption clip for few-shot classification, In European conference on computer vision, pp. 493-510. Springer, 2022.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text- to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5729-5739 2023.
Wendi Zheng, Jiayan Teng, Zhuoyi Yang. Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogview3: Finer and faster text-to-image generation via relay diffusion. arXiv preprint arXhv:2403.05122 2024.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 633-641,
Ziqin Zhou, Yinjie Lei, Bowen Zhang, Linggiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11175-1111 1188,, 2023.
Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang. Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2605-2615, 2023.
Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404.1 13046, 2024.
17

---

Published as a conference paper at ICLR 2025
APPENDIX
A DATASETS FOR GENERALIZATION CAPABILITY EVALUATION
Table 9: Datasets used to evaluate CLIP models's generalization capability.
Dataset
Classes
Test size Evaluation Metric
ImageNet-Ik 1K (Deng et al., 2009)
1000
50,000
accuracy
ImageNet- V2 (Recht al., 2019)
1000
10,000
accuracy
MaageNectAdversariin (Hendrycks et 2021b)
1000
7,500
accuracy
ImageNet-R(endition)) (Hendrycks et al., 2021a)
1000
30,000
accuracy
ImageNet-Sketer (Wang ct al., 2019)
1000
50,899
accuracy
ObjectNet (Barbu et al., 2019)
1000
50,273
accuracy
CIFAR-10 (Krizhevsky et al., 2009)
10
10,000
accuracy
CIFAR-100 (Krizhevsky et al., 2009)
100
10,000
accuracy
MNIST (LeCun et al., 1998)
10
10,000
accuracy
Caltech101 (Fei-Fei et al., 2004)
101
9144
accuracy
SUN397 (Xiao al., 2010)
397
108,754
accuracy
FGVC Aircraft (Maji al., 2013)
100
3,333
accuracy
Country-211 (Radford et al., 2021a)
21,100
accuracy
Stanford Cars (Krause ct al., 2013)
196
8,041
accuracy
Birdsnap (Berg al., 2014)
500
2,195
accuracy
Describable Textures (Cimpoi al., 2014)
47
1,880
accuracy
EureSAT( al., 2019)
10
27,000
accuracy
Facial Emotion Recognition 2013 (Goodfellow et al., 2013)
8
3,574
accuracy
Oxford Flowers 102 (Nilsback & Zisserman, 2008)
102
6,149
accuracy
Food- 101 (Bossard et al., 2014)
102
25,250
accuracy
GTSRB (Stallkamp al., 2012)
43
12,630
accuracy
PatchCamelyon (Veeling et al., 2018)
32,768
accuracy
Oxford-IIIT Pets (Parkhi et al., 2012)
37
3,669
accuracy
Rendered SST2 (Radford et al., 2021a)
2
1,821
accuracy
RESISC45 (Cheng et 2017)
45
31,500
accuracy
STL-10 (Coates al., 2011)
10
8000
accuracy
Pascal VOC 2007 Classification (Everingham et al., 2007)
20
4,952
accuracy
Flickr30K (Young et al., 2014)
1000
recall
COCO (Lin al., 2014)
5000
recall
B PSEUDO CODE FOR DIVA PIPELINE
Algorithm 1 DIVA
1: Input: Image x, CLIP model weights 0, diffusion model weights ø, representation optimization
steps N, batch size B, learning rate 7). optimized CLIP model weights 0*.
2: for optimization step E (I...., N) do
3: Compute current CLIP's visual features ff(x) as partial condition e
B 4: Sample timesteps {t. j= and noises {
5: Loss L(0, Φ)
Ee || Vaol
B
/T O0,e ci; C,4,) <e
6: Update CLIP weights 0* + V
Vo (0,0))
7: end for
8: return optimized CLIP weights 0*
C DATASETS FOR FINE-GRAINED VISUAL PERCEPTION EVALUATION
The MMVP-VLM dataset (Tong et al., 2024b) evaluates CLIP models' sensitivity to fine-grained visual patterns. It includes nine distinct visual patterns, each with 15 image-text pairs. In zero- shot setting, models must accurately match images to their corresponding textual descriptions. For each pattern, models align 15 image-text pairs from 30 images, with accuracy calculated as the
18

---

Published as a conference paper at ICLR 2025
percentage of correct matches. A sample is correct only if the model accurately matches both the two images and texts. Overall performance is the average accuracy across all nine patterns, reflecting the model''s 's understanding of various visual patterns. The MMVP dataset (Tong et al., 2024b) evaluates multimodal large language models' visual comprehension. comprises 300 test images, each with associated questions and correct answers for visual question answering. Models are assessed in shot setting, generating answers based on image content and comparing them to predefined correct answers. Accuracy calculated the percentage of exact matches over the total questions. Overall performance is the average accuracy across all test images, reflecting the model's 's understanding of various visual patterns.
19


Reviews:
[{"review_id": "kAM6inLj8a", "reviewer": "Reviewer_Vv7J", "summary": "The paper aims to improve some CLIP's shortcoming cases, e.g., understanding the quantity,color,structure of images, since CLIP is trained by focusing on the high level semantic features. Motivated by this issue, the paper proposed to leverage a diffusion model to assist the improvement of CLIPs and inject the CLIP features as a condition into the diffusion model. The design brought a significant performance gain on the MMVP-VLM benchmark as well as the Llava 1.5 and segmentation tasks without impairing the other benchmarks.", "questions": "See the weakness above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the paper is well written and easy to follow.\n2. The improvement on the MMVP-VLM benchmark is significant.\n3. The experiment is well designed and comprehensive.", "weaknesses": "1. The hypothesis of this paper is not well explained, and the solution is a little bit ad-hoc. In detail, the failure of CLIP on several typical cases mentioned in the paper should be better studied and analyzed instead of just citing a few papers and mentioning very shortly. Are the failure cases caused by the architecture design, training strategy or just the lack of training data is unknown. Only given such analysis, we can start to think of a solution to improve. On the contrary, the paper just proposed to use diffusion model to help CLIP output more detail or low-level features directly, which is less convincing to me.\n\n2. Several settings of this paper are also ad-hoc. For instance, in L225 - L241, the paper discussed the 'Visual Recap Density'. It seems for different CLIP variants, the density is way different, and I can hardly find any relationship across them. A more scientific way to decide the density should be considered. Similar thing for the times of the condition is used. The paper chooses 2 (N=2). I'm curious how this number is determined and how the performance will change for different numbers, e.g., N=10, 100.\n\n3. From my understanding, the paper used CC3M dataset to tune a pretrained CLIP. I wonder whether the gains on the benchmarks are just brought by tuning the CLIP with CC3M or the diffusion model?\n\n4. The rows 3-4 in Table 6 is confusing. When the textual condition is used, how the vision encoder can be trained?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8uFKMbsxW1", "reviewer": "Reviewer_PoSC", "summary": "This manuscript introduces DIVA, a fine-tuning framework to enhance CLIP's fine-grained visual capabilities using a text-to-image diffusion model. DIVA tunes CLIP through diffusion loss with a frozen diffusion model conditioned by the CLIP outputs. To be specific, DIVA introduces a heuristic design for CLIP output-conditioned diffusion model, which leverages randomly dropped CLIP output sequences. Throughout experiments, it demonstrates improvements in extensive benchmarks including MMVP-VLM.", "questions": "Please see the weakness section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Learning a good discriminative representation from image generation loss is now new, but I think it still has a big potential and should be studied. In this regard, the proposed idea of fine-tuning a pre-trained discriminative model (CLIP) via a pre-trained generative model (Diffusion) is interesting and demonstrates its potential.\n\n- The authors evaluate DIVA across various benchmarks, including multimodal understanding such as MMVP-VLM, the backbone of LLaVA, zero-shot classification, and segmentation.", "weaknesses": "- Overall, this manuscript lacks detailed motivations and explanations. It is unclear why this approach is needed - e.g., Why does CLIP need diffusion feedback and how does it help? Why this particular approach is superior to others?\n\n- The training process involves sampling multiple random states of the diffusion process for each image, which can be computationally expensive. There is insufficient discussion on whether this cost is justified and whether the gains are significant enough under these costs.\n\n- The proposed diffusion's condition design (section 3) is quite heuristic and overall improvements are not consistent and even not significant in many cases. For example, none of the cases in Tables 1, 2, 4, 5, 7, and 8 show consistent improvements in all metrics. Moreover, many metrics indicate the improvements are marginal so I am uncertain that the proposed method effectively works and can generalize well across different types of visual challenges..\n\n- There is a lack of analysis on potential trade-offs when using diffusion models as feedback. The paper does not clearly address whether the alignment between the CLIP's text encoder and visual encoder might degrade over time, potentially affecting performance, despite there being no consideration of the text encoder in the proposed training framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CULAS3HQGZ", "reviewer": "Reviewer_hLU8", "summary": "The paper utilizes the diffusion model to improve CLIP features' spatial and visual understanding. As CLIPs are only trained with global visual token, it lacks the ability to understand fine-grained visual concepts. At the same time, the diffusion model is trained to model the image distribution, thus it inherently has a better visual understanding. Given this idea, the paper tries to fine-tune CLIP model with a diffusion model guidance. \n\nIn details, the proposed method sends the global token and random-sampled local tokens as a condition to a diffusion model. The gradient back from the diffusion loss is used as guidance to optimize the CLIP model. Based on this, the paper presents detailed ablation studies and shows improvement for visual understanding.", "questions": "- The paper has highlighted that the CLIP is trained with limited length. Most diffusion models are also only trained with limited token lengths and have the same problem. Would it be possible that the golden fraction of local patch sampling rate (in Table 6) is just because of this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. The paper follows a clear intuition on how to improve the CLIP model with better visual understanding. The utilization of diffusion model to improve CLIP model is overall a nice idea.\n\n2. The paper presented a method to use both the global tokens and local tokens to improve the visual encoder. The paper also found that the percentage of used local tokens is important and conducted experiment to validate that. \n\n2. The paper analyzes the method on multiple tasks and thus the improvement is better shown.\n\n3. The paper shows ablation studies for key designs of the method (some are actually the analysis of the method).", "weaknesses": "Weaknesses: \n\n- The paper's clarity of the method needs to be improved. Usually, the presentation (i.e., writing) of the paper would not be directly treated as weakness and some of them are put in suggestions below. However, the ambiguity in current method description does affect the reader's understanding of the paper method, and affect the assessment of the paper during review. Thus the reviewer put it as a weakness to highlight it and hope that it can be improved during rebuttal. I list some points here:\n  \n   1. For Sec 3.3, does it an expansion of context for the \"visual features combined with the empty text’s embeddings\"? If so, please help specify it in the text.\n   2. From my current understanding, the visual features (patches) are feed as the text embeddings to the diffusion model pipeline. It would be better to have any analysis on the validity of doing this (at least have some discussions regarding this as it is not an natural operation). \n   3. What would be the positional embedding for the patches when feeding into the text encoder?\n   4. In Sec 3.3 (\"visual reacp density\"), it would be better to illustrate the reason why the random selection schemas are different for different methods. \n\n\n- The paper lacks the details of the evaluation protocol. As the detailed inference method can largely affect the results, it's crucial to have a detailed description of how the method is evaluated on the MMVP dataset. \n   1. Given this, it would be hard to evaluate the validity of the ablation studies, which also affect the review of the paper. \n\n- The analysis needs to be improved in presentation:\n   \n   1. For some comparisons, the difference between baseline and proposed methods are close (e.g., 0.4 in absolute value). For these small differences, it's hard to understand its improvement. Some variance or significances analysis would be appreciated. Also, highlighting these numbers with gray colors would be helpful to readers. \n   2. In Table 3, some baseline and the proposed methods have the same score, but only the proposed methods are bold. It would be confusing to readers. \n\nSuggestions (will not be treated as paper weakness but would request it in the revision):\n0. The algorithm description in the main text is not comprehensive enough for reproducing. It would be better to expand the descriptions. Even it sounds trivial, it would be better to extend these content for clarity. \n1. It would be better to refine the terminology in Algo 1 (Appendix B). E.g., \n   - The $\\theta^*$ and $\\theta$ are confusing. \n   - It's better to write CLIP feature as $f_\\theta(x)$ instead of \\theta(x)$\n\nComments:\n\nI overall would like to trust the validness and soundness of the paper given the detailed experiments. The current details provided in the paper are not convincing enough to the readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "spD9OMhwBK", "reviewer": "Reviewer_gbrY", "summary": "CLIP faces challenges related to fine-grained visual perception. To address this, the authors propose fine-tuning the CLIP image encoder by leveraging the spatial awareness of diffusion models. They use the visual tokens and the $[CLS]$ token from the CLIP image encoder as conditions for the diffusion models, backpropagating the gradients to update the CLIP encoder. The fine-tuned models are evaluated across several challenging benchmarks that require fine-grained visual perception, yielding promising results.", "questions": "- The proposed method significantly enhances the spatial awareness of CLIP. I wonder if this improvement extends to other multimodal tasks, such as open-vocabulary dense prediction tasks.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Utilizing diffusion models to enhance CLIP's dense representation presents a novel perspective that will likely interest the community.\n\n- The proposed method is intuitive and straightforward to implement, making it compatible with various existing vision-language models\n\n- The experiments are comprehensive, demonstrating significant improvements across various benchmarks.\n\n- The paper is well-written and easy to follow.", "weaknesses": "- Given that no explicit alignment constraint between the image and text encoders is applied during the fine-tuning process and only the image encoder is updated, I am concerned about the potential for misalignment between the image and text encoders. \n\n- The visual recap operation varies across different models, potentially necessitating careful tuning of hyperparameters. Is there a general strategy applicable to all models, or must it be tailored to each specific model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o9Ge8F1h2l", "reviewer": "Reviewer_UUiV", "summary": "The paper introduces DIVA, a self-supervised framework designed to enhance CLIP's ability to perceive fine-grained visual details by leveraging generative feedback from text-to-image diffusion models. The framework incorporates a novel visual dense recap scheme, conditioning the diffusion models with CLIP's dense visual features and using image reconstruction loss for optimization. The results show that DIVA significantly improves CLIP's visual perception and its performance on multimodal and visual understanding tasks, as demonstrated on the MMVP-VLM benchmark. Additionally, DIVA maintains CLIP’s strong zero-shot capabilities across 29 image classification and retrieval benchmarks, proving its effectiveness in optimizing CLIP's discriminative representations.", "questions": "1. It is crucial to also verify that the original cross-modal alignment capability of CLIP can also be preserved.\n\n2. What criteria were used to decide the proportion of local tokens? Could you provide an analysis or ablation study to support this choice?\n\n3. How does the varying proportion of local tokens across different datasets and between training and inference affect the method's generalizability?\n\n4. Is there any other local token selection strategy that might yield better results?\n\n5. Would incorporating image-associated text in the diffusion model conditions improve the performance observed in Table 6?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "--The paper presents an innovative approach by being the first to leverage generative feedback from text-to-image diffusion models to optimize the visual encoder of CLIP, aiming to addressing its limitation in fine-grained visual perception.\n\n--The experiment part is extensive, validating their approach using the MMVP-VLM benchmark and demonstrating robust improvement. The integration with MLLM shows its impact, while experiments across 29 image classification and retrieval benchmarks confirm that the original capabilities of CLIP are preserved.\n\n--The paper is clearly written, with well-explained methods and detailed descriptions of the proposed framework and experimental results.", "weaknesses": "--Disruption of Visual-Language Alignment: Optimizing CLIP’s visual encoder may disrupt the inherent visual-language alignment of CLIP. Although the authors validate that CLIP’s original capabilities are retained through experiments on 29 datasets, these tests are somewhat limited, as they only cover image classification and unimodal retrieval (text or image retrieval). Because the text encoder is not fine-tuned, the retrieval results for text remain unchanged, making text-retrieval experiments redundant to some degree. Moreover, the paper lacks experiments on tasks that rely heavily on cross-modal alignment, an area where CLIP excels, such as cross-modal retrieval. If such disruption occurs, it would be valuable to explore methods for preserving cross-modal alignment, such as applying a reconstruction loss to the visual class token to maintain the original feature.\n\n--Part Token Selection Strategy: The method involves inputting tokens from CLIP’s visual encoder, including the global (class) token and a subset of local tokens, as conditions for the diffusion model generation. Although the authors state that using only the global token or a combination of the global token with all local tokens yields suboptimal results than global + part local tokens, they do not clarify how the proportion of local tokens is determined. Additionally, this proportion appears to vary across different datasets and between training and inference, raising concerns about the method’s generalizability. Furthermore, the strategy for selecting part tokens could impact the results, and using a simple random selection might not be optimal.\n\n--Empty Text Conditions: In Table 6, the conditions for the diffusion model include an empty text condition. It would be interesting to explore whether incorporating text descriptions corresponding to the images could further enhance the results.\n\n--Minor typo: There is an extra word \"for\" at line 350.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)